Fragmented data from the streaming API

API Question
#1

Hi, I wonder if someone could give me some guidance on the streaming API to which I am connected and receiving data, but it is seemingly fragmented.

What I mean by this, is that I am getting responses like this:

{“resource_kind”:“company-profile”,"reso…:“changed”}}

20-12-31",“period_end_on”:"2020-12-…“changed”}}

ting_reference_date":{“d…pe”:“changed”}}

I have truncated the message responses here for brevity. The first message is complete, then subsequent messages are partial but always end at the end of the JSON packet. The first part of the data is simply ‘missing’. At various stages, a full and complete message is received again, but followed by incomplete data.

This is my first time dealing with an open-ended stream and so I am obviously missing something.

This is my code - in .NET, using a HTTPWebRequest and Response

Connection…
Do
response = request.GetResponse()
Dim sr As New IO.StreamReader(response.GetResponseStream(), detectEncodingFromByteOrderMarks:=True)
Debug.Print(Await sr.ReadLineAsync)
Loop

Is reading the stream by line the culprit? If so, what is the correct way of pulling the data out of the stream?

Thanks in advance for any assistance on this.

Mark

#2

If it’s anything like the way it would be handled in eg python I’d suggest trying the get once then loop just around the stream read (I’ve read that as pseudocode - my .NET knowledge is less than minimal). I’d expect there are others here who can help further but that’s maybe worth a go.

#3

I spent a bit more time playing with this and managed to solve it and so I am updating this in case anyone else has a similar issue in future.

My mistake was initiating a new instance of the streamreader inside the loop.

So my working flow now is:

Connection…

response = request.GetResponse()
Dim sr As New IO.StreamReader(response.GetResponseStream(), detectEncodingFromByteOrderMarks:=True)

Do
Debug.Print(Await sr.ReadLine)
Loop