Streaming api for multiple endpoints

API Question
#1

Hi,

For the same company, we usually fetch data from 6 different endpoints at the same time (company/officers/insolvency/persons-with-significant-control/charges/filing-history). Due to this we some times get rate limited when we have large traffic.

To solve this problem, we are planning on using the streaming api and are currently new to this. Say we maintain our own database and update it using the streaming api, given that there is a limit of 2 concurrent connections, how can we use the streaming api to fetch data from 6 different endpoints?

In a few forums I have come across using an offline connection and storing the timeouts. Would the below logic work? (Say, we run this task every 30 min)

for endpoint in endpoints:
     previous_timepoint = TIMEPOINTS[endpoint]
     data = get_streaming_data_using_single_http_request(endpoint, previous_timepoint)
     update_local_db(data)
     TIMEPOINTS[endpoint] = data["timepoint"]

Also, how do we get the full snapshot of the existing data till current timepoint before the 1st run of such a task?

Many thanks

#2

That limit of 2 has been increased to 6. So you can connect to all streams.
You will be disconnected every 24hrs (at least) so you have to build in timepoint managment.

1 Like