200 response but not getting the stream data

API Question
1

Hi,

This is very clearly something I am doing wrong, rather than an issue with the API. I am trying a very basic fetch in Node to stream filing information.

I have tried cURL and all works great, but when I put it in my application I can’t seem to get it to work. There are no errors, and it’s returning status of 200.

This is my very simple code:-

const base64 = require('base-64');
const url = 'https://stream.companieshouse.gov.uk/filings';
const streamKey = 'my-key';
const headers = new Headers();

headers.set("Authorization","Basic " + base64.encode(streamKey+":"));

fetch(url, {
  headers: headers,
})

.then((response) => {
  console.log(response);
  return response.json()
})

.then(data => {
  console.log(data);
})
.catch((error) => {
  console.log(error);
})

And this is what I am seeing in the response where I have logged above:-

    aborted: false,
    rangeRequested: false,
    timingAllowPassed: true,
    requestIncludesCredentials: true,
    type: 'default',
    status: 200,
    timingInfo: {
      startTime: 99.316304,
      redirectStartTime: 0,
      redirectEndTime: 0,
      postRedirectStartTime: 99.316304,
      finalServiceWorkerStartTime: 0,
      finalNetworkResponseStartTime: 0,
      finalNetworkRequestStartTime: 0,
      endTime: 0,
      encodedBodySize: 671,
      decodedBodySize: 0,
      finalConnectionTimingInfo: null
    },
    cacheState: '',
    statusText: 'OK',
    headersList: HeadersList {
      cookies: null,
      [Symbol(headers map)]: [Map],
      [Symbol(headers map sorted)]: null
    },
    urlList: [ URL {} ],
    body: { stream: undefined }

Clearly I am doing something stupid here… Can anyone see what I might be doing wrong?

2

You can’t call response.json() on the streaming API response, since that Promise waits for the response body to complete, whereas the streaming api response body never finishes, so you need to read it as each chunk comes in.
I would suggest using the response.body stream to read the response.

import { Readable } from 'node:stream'; // built in module in Node.js

fetch(url, {
  headers: headers,
}).then(response => {
  const responseStream = Readable.from(response.body)
  responseStream.pipe(process.stdout) // this will print the response body
})

To separate each event out from the response stream, use a package like split2 from npm.

For a full working example using Node.js https.get instead of fetch, see my companies.stream source code or a simpler example.
To see it done with fetch, see Filing Deadlines source code.

While it can be done with fetch, personally I prefer using Node.js built-in https module for this request.

Hope you find that useful. All the best.

1 Like
3

Thank you so much @ebrian101. That makes perfect sense. I will try this in a few days when I get time.

Really appreciate taking the time to respond to me :+1: