Hello!
I have couple of questions about 2 streams in streaming API.
Company Stream
So far I have managed to bulk import .csv of basic company data into our database did a transformation to the data format so it plays nice with the streaming updates. With this I can keep company data up-to-date by doing lookup in our DB with company number and play the incoming changes.
I have 2 questions:
I have realised the updates from this stream are not just updates to existing companies and also the new creations. I don’t see a clear identifier in the data differentiating those.
- Do I just do my lookup in our bulk imported data and if I can’t find it treat it as a creation or just check incorporation date is the same as event production date maybe?
- Given I can real-time update/add companies to our db with the stream is there any point re-running the bulk import every month with the .csv file?
PSC-Stream
I don’t understand how can I keep the PSC data up-to-date with the streaming API at all.
- Do I just look at the events and create a new entry in our end like Company number to PSC pairs.
- Do I need to delete old records or update them somehow. If I want to these operations what is the unique identifier in PSC stream that lets me do that (What do I use to do my lookup is it etag)?
I understand it might be a basic question but I am lacking the domain knowledge to understand how to keep this information up-to-date. Any information is greatly appreciated.
Thanks in advance.