Streamng API Internal Server Error

I am trying to run the Streaming API but am getting a 500 Internal Server Error. It hasn’t been working for a few days - can you confirm if there is an issue with the API or if it is something on my end?

Which stream are you trying to connect to?.

Steve

https://stream.companieshouse.gov.uk/companies

Hi Andy,

We are investigating

Steve

Hi Andy,

All streams are now available.

Sorry for any inconvenience.

Steve

Thanks Steve - it is now working.

Streaming API seems to be down again since 2020-05-08T03:16:02:

curl -u <APIKEY>: https://stream.companieshouse.gov.uk/companies
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

It seems to be up and running again. Thanks.

We’re getting 503 responses in a similar fashion. Is this related?

No, the streaming API is up and running. I have just double checked :smile:

So this is on the /filings stream; we’re now handling this in our code, but this AM we definitely received a 503 status from the server and needed to reopen the connection. We’re typically seeing this error about 5 times a day.

@MArkWilliams would any further information help you identify the cause of this code? It doesn’t affect us now we handle this but as an undocumented code it’s probably worth at least having a line in the docs that say “handle this error”. Especially as customers are likely to use this as a service and so an unhandled code will cause an outage rather than a missed datapoint.

The streaming APi is capable of generating a number of http codes.
Restarts could generate 503’s. There was a restart this morning.
How you handle errors is of course up to you, but it would be a good idea to be able to handle all known and unknown error codes of course.
I will pass on your comment regarding error documentation.

1 Like

Thanks Mark. Just to confirm - the only error code that we need to rate limit (backoff for 1m) is a 429?

The rate limit is of course for the API, not the streaming API…
If you exceed the 600 requests in a 5 minute window you will get 429.
If you go higher and exceed the AWS limit,
or
we ban you manually
or
your IP address does not match the IPs in your API restricted_ips list you will get 403.

How long you back off depends on the limit, your number of requests and the 5 minute window of course.
I would be concerned if you are sending more than 2 per second. If you need bulk data, we have a number of products that may be of use to you.

Ah no I’m referring to the streaming API rate limit - documented here: https://developer-specs.companieshouse.gov.uk/streaming-api/guides/overview#reconnecting - if we get a 429 we need to wait 1m before reconnecting; but I wondered if there is a similar rule on any other errors?

Ah, sorry.
No, other than 503’s possibly, where I would suggest a bit longer than 1 minute.

Thanks @MArkWilliams .

Hi @MArkWilliams https://stream.companieshouse.gov.uk/companies started returning a 500 error

Edit: It now works again!