HTML Response to a Beta Request

[richard]

At present I am trying to access the Beta REST web service and am getting a HTML and not JSON response to it. The URL being used is https://beta.companieshouse.gov.uk/company/XXXXXXXX and the response is a 28k HTML file. Am I using the correct URL here? Or is there a way to specify (perhaps using Content-Type) to make sure that the response is in JSON format?

[mfairhurst]

Richard,

The address being used is the URL for our web service. To use the REST API there are a number of endpoints available providing a range of functionality but all beginning with https://api.companieshouse.gov.uk/

For documentation on the API please refer to http://developer.companieshouse.gov.uk

Thanks

Mark.

[richard]

I was originally using the https://api.companieshouse.gov.uk/ URL and getting 401 errors (e.g. the URL https://api.companieshouse.gov.uk/company/XXXXXXX), I have used the API Key given to me in the URL (i.e. as before with ?api_key=YYYYYYYYYYYYYYYYYYYYYYYY) and in Java set a HTTP Request Property “Authorization” with "Basic "+YYYYYYYYYYYYYYYYYYYYYYYY. I have also tried Base 64 encoding of the username and password sent to me in the same fashion. Could you let me know what I am doing wrong? When I attempt trying a search with the Try It Out functionality on the website the Host is api.companieshouse.gov.ukundefined, I have tried that endpoint but that gives me the 401 error.

[mfairhurst]

Richard,

The endpoint to use is: https://api.companieshouse.gov.uk/company/XXXXXXXX.

Does the try it out functionality return any results? The host specified in the try it out functionality should not have the undefined on the end, this is a bug in the site, so that will not work.

When setting the Authorization request property did you append a : to the end of the api key?

The new Companies House REST API uses standard Basic HTTP Authorisation which consists of a username:password pair. The API key is the username but we are not setting a password, but the delimiter between the two must be present hence adding the : to the end of the api key. The key including the : must be base 64 encoded.

Let me know if this helps.

Thanks

Mark.

[csmith]

Hi Richard.

I’ve found a previous answer I’ve posted, which may tell you what you need, and parallels Mark’s reply, but with a little more technical detail:

Getting invalid authorization

The existing XML Gateway API employs a custom authentication mechanism unique to Companies House, and involves the complex MD5 hashing of passwords and transaction ID's. The new Companies House REST API uses standard Basic HTTP Authorisation. This makes API key authentication as simple as the following curl command: curl -v -uYOUR_APIKEY_FOLLOWED_BY_A_COLON: https://api.companiseshouse.gov.uk/company/00002065 You may have to do a little more work in code, depending on whether your client f…

If not, try searching the forum for “authorisation” topics - there is a wealth of information there that might also help.

Let us know how you get on.
C

[awais_techloyce123]

Hi, I have the same issue that on link: https://api.companieshouse.gov.uk/company/XXXXXXXX
I receive status code 401, But on the link: https://beta.companieshouse.gov.uk/company/XXXXXXXX
I am getting required response but in html code not in json format.

How to get response in json format by using link described in companieshouse api.

Please help.

[awais_techloyce123]

Hi mfairhurst,

Is it the correct way to do authentication with companieshouse Api by using java
The code is below:

String stringUrl = “https://beta.companieshouse.gov.uk/search/companies/q=CompanyName”;
URL url = new URL(stringUrl);
URLConnection uc = url.openConnection();

    uc.setRequestProperty("X-Requested-With", "Curl");
    uc.setRequestProperty("Authorization", "myAPIKey");

As i am getting the required result by using the above code but not in desired format.I want to get response in json format and through the original api mentioned link.
Please help me out .

[csmith]

@awais_techloyce123

Ah. Okay, there are several things wrong here:

1. You are sending your API requests to the WEBSITE.
2. Your authorisation header is wrong.

First, do not be confused that the URLs look similar for the API and the website. The API will return data encoded as JSON. The Website will of course return HTML, plus, if you scrape the website, you’ll no doubt have your IP blocked by our access control infrastructure…

Your problem stems from you not authenticating properly, hence the HTTP status code of “401 Unauthorised” you are getting.

If you search the forum (and I really do urge people to do this, as this has been answered many times before), there are several topics that show you how to correct your error. Most come down to users of the API not providing HTTP Basic authentication:

Basic access authentication

Quite a few HTTP clients natively handle Basic authentication, so for them it is a simple matter of setting the username and the password (which in our case is blank) on the client instance before making the call. For other clients, you have to set the Basic authentication header yourself.

We have posted a simple Perl example to demonstrate the Basic authentication mechanism if you have to set the header yourself. You should be able to translate the two important lines of Perl into two lines of Java to do the same. This example also shows how a curl request works, as it natively handles Basic authentication, demonstrating what headers are being sent, and what they contain:

Getting invalid authorization

The existing XML Gateway API employs a custom authentication mechanism unique to Companies House, and involves the complex MD5 hashing of passwords and transaction ID's. The new Companies House REST API uses standard Basic HTTP Authorisation. This makes API key authentication as simple as the following curl command: curl -v -uYOUR_APIKEY_FOLLOWED_BY_A_COLON: https://api.companiseshouse.gov.uk/company/00002065 You may have to do a little more work in code, depending on whether your client f…

Finally, a quick web search for URLConnection BASIC authentication returns the following first hit, which actually provides the two lines of Java I mentioned above:

Preemptive Basic Auth with HttpUrlConnection?

That and the Perl example should get you up and running. Perhaps you could come back and post your corrected code snippet here, when you get it working, to close this topic off?

Hope that helps. Good luck!
Regards

[csmith]

Just done another web search, and URLConnection (well, HttpURLConnection) natively supports Basic authentication, so you don’t need to know how to do it, just set the username and password (blank for the Companies House API):

Connecting to remote URL which requires authentication using Java

The advantage of using this Authenticator override, is that you set it up globally, so all connections get the authentication. If you set the Authorization header yourself, you need to do so on every connection you make.

[awais_techloyce123]

Hi csmith,
Thanks for the help
I know that API url is diffirent from the url of website.
But was confused on encoding issues due to which I was getting problem.

But now my problem has been solved.
And the code that is working for me is:

 String stringUrl = "https://api.companieshouse.gov.uk/company/{companyNumber}";
           
    URL url = new URL(stringUrl);
    
    String authStr = "MyAPIKey"+":"+"";
    System.out.println("Original String is " + authStr);
    
 // encode data on your side using BASE64
    byte[] bytesEncoded = Base64.encodeBase64(authStr .getBytes());
    String authEncoded = new String(bytesEncoded);
     
    HttpURLConnection connection = (HttpURLConnection) url.openConnection();
    connection.setRequestProperty("Authorization", "Basic "+authEncoded);

[mfairhurst]

Great news, thanks for letting us know

[awais_techloyce123]

Hi csmith,

It has solved my problem.
Thanks for the help. I was somehow getting problem with encoding string into string and some issues related to authentication authorization.
But now I have solved all the problems.
The code that is working for me is :

String stringUrl = “https://api.companieshouse.gov.uk/company/{companyNumber}”;

    URL url = new URL(stringUrl);
    
    String authStr = "MyAPIKey" + ":" + "";
    System.out.println("Original String is " + authStr);
    
 // encode data on your side using BASE64
    byte[] bytesEncoded = Base64.encodeBase64(authStr .getBytes());
    String authEncoded = new String(bytesEncoded);
     
    HttpURLConnection connection = (HttpURLConnection) url.openConnection();
    connection.setRequestProperty("Authorization", "Basic "+authEncoded);