Curl call gets stuck

API Issues
#1

My curl call doesn’t complete - it gets ends up showing the end of the JSON response and you have to press [enter] to get the final 2 lines (e.g. [1]- Done ...) to print to shell. And > report.json isn’t capturing the response as the resulting file is empty. Call and response below, with JSON removed for sake of space (but it is a correct response). I just want curl to report the json to file and end without having to press enter. I’m not massively experienced with REST so I’m probably missing something.

Me-MBPX:apps mycomputer$ curl -uMY_API_KEY: https://api.companieshouse.gov.uk/search?q=trump&items_per_page=10&start_index=1 > report.json
[1] 7211
[2] 7212
Me-MBPX:apps mycomputer$ {"kind":"search#all","total_results":569,"items_per_page":20,"start_index":0,"items":[{"links":{...CORRECT_JSON...}
[1]-  Done                    curl -uMY_API_KEY: https://api.companieshouse.gov.uk/search?q=trump
[2]+  Done                    items_per_page=10

I’ve tried with a number of additional arguments, e.g. various combinations of following: -i -H "Accept: application/json" -H "Content-Type: application/json" -X GET. Nothing works. Grateful for assistance.

Searching Using Curl returns only 20 results a Page
#2

Missing quotes? You didn’t have any around the URL in your example above, I think the URL (but not the API key) contains characters which will trip up the shell. The “&”? Looks like “items_per_page” at the top of your JSON response doesn’t match what you requested.

If (on my Win10 prompt) I run that command - leaving off the output redirect (" > report.json") things break in a similar way. The following command runs:
curl -uMY_API_KEY: https://api.companieshouse.gov.uk/search?q=trump
…and produces output.
The shell then tries to evaluate stuff after the “&” as a command and gets confused.

Are you using a Linux shell there? My *nix is very rusty but I think that the “&” will still cause you problems, try weak (") then strong (’) quotes around the URL.

If that works out then add the output redirect back in. You should see the response from curl a la:
% Total % Received % Xferd Average Speed Time Time Time Current
…

Chris

1 Like
#3

Gosh! I assumed the URL was fine given that the API was interpreting it correctly, but your suggestion works. Thanks Chris.

curl -uMY_API_KEY: "https://api.companieshouse.gov.uk/search?q=trump&items_per_page=10&start_index=1" > report.json
#4

Think your unescaped “&” in the URL was triggering “run in background” - see e.g. https://unix.stackexchange.com/questions/86247/what-does-ampersand-mean-at-the-end-of-a-shell-script-line
So indeed, curl will run, but just not with the URL you thought, and you’ll get all the odd side-effects.

I’ve found using curl from shell / command line is good for basic testing. It’s slightly involved if you want to download documents. For that you have to trap the http header with the -I or --head option, then head off to the address returned. This is somewhere in amazon’s domain. IIRC you may get another redirect once there? Anyway it works fine in our code but sometimes gave me problems when doing this from the command line with curl.