Streaming API company_profile does not agree with web app

Data Issues
#1

TL;DR: some streaming API data does not agree with C/H Website data for some companies.

We have noticed that, for some companies, the data showing on the C/H website (https://find-and-update.company-information.service.gov.uk/) does not agree with the data that we’ve received from the streaming API.

An example of two such companies is shown below:

regnum latest event_time_point latest event_published_at
09781713 32890840 2021-09-07 07:09:07
NI621216 35847495 2021-11-10 08:38:04

In both cases the C/H website shows overdue accounts & confirmation statement status but the streaming API data does not.

We were concerned that we may have missed some streaming api records, but we’ve confirmed that there are no gaps in our data’s event_time_point history between 29400000 (2021-06-17 11:15:15) and 59156372 (2023-04-04 10:42:47).

We’ve started looking at our streaming api data in more detail now and have found the following:

9,683 companies have accounts_next_due date before 31/10/2022 but are not marked as having overdue accounts.

Of the 9,683:

  • 8,956 are in liquidation
  • 494 have another financial distress event
  • 146 are Suspended strike-off / Ceasing to trade / Petition winding up
  • 87 are live.

Please let us know if this is an error in the streaming api or the C/H website and if something can be done to make the datasets consistent.

Thanks
Lance

3 Likes
#2

That does not seem correct.
This is being looked at now. I will update you as soon as I have more information.
Thank you for taking the time to report this.

#3

Initial investigation shows that there is an issue.
The investigation will continue and hopefully we will have a resolution soon.
Again, thank you for taking the time to report it.

1 Like
#4

UPDATE: A problem has been identified. It is related to a trigger not firing that injects the update into the stream. A fix is being planned, but I do not have a timeframe yet.

#5

Good to hear that a cause has been found and thank you for following up!

#6

Hi Mark, is there an update available on this issue?

Thanks
Lance

#7

As it happens there is news. A fix has just been deployed to our staging environment for testing. Depending on results and that should go live in the next 10 days or earlier.

#8

That’s great news! Thanks for the update, Mark.

#9

Hi @MArkWilliams,

We have an issue where the extension of an Accounts filing deadline does not appear to be reflected in the stream of data.

We wondered if this could be related to the above issue? We also wondered if the fix for the above issue had now been deployed?

Many thanks,
Ben

#10

The fix has definitely gone live.
We are looking into exactly what comes out of the stream when an extension is applied to a filing date.
We will update here.

2 Likes
#11

Hi @MArkWilliams,

We’ve noticed again some discrepancies with the data shown on the website compared to data retrieved from the streaming API, specifically around companies that now show overdue accounts since 2023-09-30 (timepoint around 66963784).

One such example is 00934513 which has not had any events on company profile for over a year and now shows as overdue on the CH website but not on the streaming API.

Another example is 02627406 (last timepoint received relating to this company was 63710296).

As the OP mentioned, we’ve checked to see if there was any missing data but from the above timepoint to now we have no gaps. It’s been particularly noticeable as we’ve moved into October due to so many companies having due dates of 2023-09-30.

Please are you able to look into whether or not an expected event was not published marking these companies as having overdue accounts?

Many thanks

1 Like
#12

It is not a discrepancy and you have not missed any events.
The overdue value in the streaming event is at that point in time that the message is published.
The overdue status returned either on the web or in an api response is a real time calculation based on system date and the next due date.
Accounts or confirmation statements not being delivered on time do not constitute an event and therefor a stream message is not published.
For an event to be published on the company_profile stream, a value in that resource needs to change via a filing.

#13

Hi @MArkWilliams thanks for the reply.

Is the accounts.next_due date therefore the correct value to use for this overdue flag calculation? i.e. if an extension is given, an event is published and this date will be updated?

If so, we will revert to using this calculation rather than relying on the overdue status.

#14

Use data.accounts.next_accounts.due_on as the due date.
Yes, an AA01 filing changes the next accounts made up to date in the profile resource which triggers a company delta/stream message.