I’ve been trying to gather profiles of companies founded since 2016 that have since been dissolved. In the absence of these being available as a bulk download, I have been making Company Profile endpoint requests for each company number that doesn’t appear in the list of (mostly) active companies provided as a bulk download. This obviously means making a lot of requests (around 890k), but I’ve been using the API a lot for most of this week for other requests without any issues with the rate limit.
However I made a mistake in my code’s handling of cases where a response’s headers don’t include the rate-limit counters and wait time (I was getting a few 402 errors, from an awselb server, which didn’t have any of the expected headers), but the same endpoint would work if I tried again after 10 seconds or so. I also added what I thought was sensible logic to handle a 404 or 403 status code, which was to move on to try requesting the next company profile. Unfortunately including 403s turned out to be a very bad idea, and I woke up this morning to find that I’d made around 179k invalid requests because for some reason I’d triggered a 403 response at around 4am, so my code was just carrying on through the list of numbers it wanted to get information for despite those responses. Unsurprisingly it means my IP has been blocked (it was blocked on both the API and the website, but now the site is back, but the API is not)!
I’m not sure what triggered the first 404/403 response in the first place, was there a problem with the way I had been making the requests up until that point? Please would it be possible for my IP address (which I’ve now added to my app’s IP list) / API key to be reinstated? I have already written an alternative way of handling a 403 for future cases to make sure that doesn’t happen again.
Many thanks and apologies for the poor error handling on my part!
I would very much like to add my name to the list of people keen for a bulk download of dissolved company information…