Various errors (502, 403) occurring with no pattern

#1

Hi all,

I’m attempting to call the API to list company officers, however am consistently hitting a 403 Forbidden error but at varying points of the list of companies I’m trying to run.

For example the first, say, 414 calls work perfectly but the 415th returns 403 Forbidden.

Trying for a second time, the first 457 calls work perfectly but the 458th returns 403 Forbidden.

Earlier today I was also receiving 502 Bad Gateway errors in a similar fashion.

What can be causing this? I have no IP restrictions in place.

Thanks, Ryan

#2

There are a few possibles. 403 seems to be what you get if CH is unhappy with the API IP at the moment.

I have no IP restrictions in place.

…but Companies House do. Unless your calls are very occasionally coming from a different IP not registered with CH this shouldn’t cause your issue however.

You may be having problems for other reasons:

Depending on exactly when you’re running your calls these may be taking you over the 5 minute, 600 call rate limit. It used to be you got 429 when you exceeded the limit but there was a recent response from CH suggesting another error code might be returned (sorry, can’t find this offhand).

If you were using the search endpoint to find company officers and were looking over indexes of about 400 then you will get errors (by design) although these should give a different code - see Search company officers returns HTTP 416 when start_index over 300

Sporadic patches of 502 occur.

#3

Interesting! I don’t see any reason why my IP would be changing randomly, but I’ll have to check with IT.

I don’t think I’ll be hitting the rate limit. I make my script flip over to a new application and API key when it gets close to the 600 limit, for testing purposes. And this works for running higher volumes of calls (1000s) within 5 minutes.

How do you mean ‘Sporadic patches of 502 occur.’? Like just randomly the API can stop working for no discernible cause (on our end)?

#4

You have 30 API keys!
First off, if you breach our fair use policy, you will be immediately (without notice) and permanently denied access to our services.
Aswell as the API rate limit checking, there is also IP limit checking. If you exceed this you will be banned.
If you are banned you will get 403.
If you are rate limited you will get 429.

403 Forbidden from API on one IP address
#5

Apologies, @MArkWilliams. Hadn’t realised this was in breach of your fair use policy. I’ll remove the others.

Is the IP limit checking the same rate as the API limit checking?

I requested a higher rate limit through enquiries@companieshouse.gov.uk and was referred to this forum. Where’s best to post this request?

#6

No, it is higher.
Your request will not currently be considered as you have so many API keys, but yes, here is a suitable place to make that request and start a dialogue.
Here is a reminder about the Fair Use Policy:-

Acceptable use of Companies House Searchable Databases
The Companies House searchable databases are designed to meet the needs of a wide range of users wishing to interrogate our information on-line.
Due to limitations of equipment and bandwidth, they are not intended to be a source for bulk downloads.
Individuals, companies, IP addresses or blocks of IP addresses who deny or degrade service to other users by generating unusually high numbers of daily database accesses, whether generated manually or in an automated fashion, may be denied access to these services without notice.

#7

Thanks @MArkWilliams, I’ve removed the extra API applications now. Thanks for the reminder. Shall I make a new post for the rate limit increase request?

#8

Someone will contact you regarding the rate limit increase.