Accessing data after last snapshot

itoufexi 2020-05-07 10:19:58 UTC #1

Hi there,

I have made use of the Free Company Data Product, which is very useful, but I was wondering if I could access the data after the last snapshot was taken.

I understand that the API is not supposed to provide bulk data, so I would like to query /search/companies for a subset of results, e.g. companies that were incorporated or dissolved from 1 May 2020, or companies with a specific SICCode.

I tried to use different keywords to achieve this but did not work and I am not sure which database fields the search is looking into.

Could you please help me with this? Many thanks.

MArkWilliams 2020-05-07 14:06:18 UTC #2

There are a number of other bulk products that might be of use to you.
Also we have a streaming API that is constantly publishing changes to companies and officers.
documentation for that service can be found https://developer-specs.companieshouse.gov.uk/99
see https://forum.aws.chdev.org/t/the-new-streaming-api/2637 for more details

alexandrosadavies 2020-05-07 14:17:35 UTC #3

Sorry @MArkWilliams , on this - is the officers streaming API now functional then?

itoufexi 2020-05-07 14:52:39 UTC #4

Thanks @MArkWilliams, I will have a look at the Streaming API, which looks promising, as well as the other bulk products.

itoufexi 2020-05-19 07:54:52 UTC #5

Hi @MArkWilliams, I am using the /companies endpoint of the Streaming API to get all the update events since 1 May, when the latest bulk product was released.

According to the documentation, "The streaming API endpoints are each backed by a queue, so that clients can disconnect periodically and reconnect some time later. This queue backlog contains as many events as needed to bring the most recently produced data snapshot up to date."

However, I receive a 416 error when my timepoint is older than 11-12 days ago. I read that the 416 means the timepoint is too far in the past. It seems that the limit of how back in the past I can go does not reach the data snapshot cutoff point.

Also, I noticed that if I go back in time gradually, 1000-2000 timepoints a time, I may avoid the 416. Thus I managed to scrape events back to timepoint = 9692100.

Thank you for your time, your help is greatly appreciated.

MArkWilliams 2020-05-19 08:25:45 UTC #6

Sorry, no it is not.

MArkWilliams 2020-05-19 08:47:09 UTC #7

You can go back between 7 - 10 days, so as long as you keep within a week you should be fine.

rmcneil 2020-11-27 14:15:29 UTC #8

With respect to Data Snapshots: It is still our intention to develop data snapshots for the Streaming API service and the current thinking is that we will form a project to do this next financial year, with Discovery potentially starting in April.

With respect to the Officers data stream: We have developed this (and one for PSCs - Persons of Significant Control) and hope to release them soon.