How does python fetch json data from api?

An Introduction to JSON and APIs using Python

Introduction

In a different tutorial, we discussed how to web scrape with python. The goal of web scraping was to access data from a website or webpage. Well, sometimes a website can make it easier for a user to have direct access to their data with the use of an API (Application Programming Interface). This basically means that the company has made a set of dedicated URLs that provide this data in a pure form (meaning without any presentation formatting). This pure data is often in a JSON (JavaScript Object Notation) format, which we can then parse through and extract what we need using python.

For this tutorial, we will use the free API found at covid19api.com that provides data on the coronavirus. We will find the total number of confirmed cases in each country and then we will create a pandas dataframe that contains that information. So let’s begin!

Inspecting the API

If you go to the documentation page of the API, this is what you’ll see:

This shows us the different URLs in the API, the information they provide, and example requests/responses of those URLs on the right.

We can see that the information we are seeking is in the summary page. We can click on view more on the right so we can see what the response would be from that URL:

This is a JSON object! As you can see, it is very similar to a python dictionary and is made up of key-value pairs. In fact, in order for us to parse through this and extract what we want from it, we will eventually turn it into a python dictionary object. Upon inspection, we can see that it looks like a nested dictionary. The outer dictionary has the keys ‘Global’ (with a value of a dictionary) and ‘Countries’ (with a value of a list that is made up of dictionaries, with each dictionary corresponding to a specific country).

Making an HTTP Request from the API

So let’s open a jupyter notebook and request the information from that URL. We will use the requests library to make an HTTP request from that URL and save the response object’s text under the variable response:

response = requests.get(‘https://api.covid19api.com/summary’).text

This shows what the response is to our HTTP request from the API. As you can see, it is a long python string that is in JSON format.

Creating a Python Dictionary

Since the response is in JSON format, we can load this string into python and convert it into a python dictionary. We first need to import the json library, and then we can use the loads method from the json library and pass it our string:

response_info = json.loads(response)

Note how the type of our response_info variable is now a python dictionary!

Now that our response is in the form of a python dictionary, we can use what we know about python dictionaries to parse it and extract the information we need!

Also note: The requests library has a built-in JSON decoder that we could have used instead of the json module that would have converted our JSON object to a python dictionary. However, I used the above method to introduce the json module in this tutorial. Here’s what the code would have looked like if we instead used the JSON decoder within the requests module:

requests.get(‘https://api.covid19api.com/summary’).json()

Parsing the Dictionary

As previously mentioned, we would like to make a pandas dataframe that has two columns: countries, and the number of total confirmed cases for that country. We can do so by looping through the values of the ‘Countries’ key of our outer dictionary:

As you can see, the value of our ‘Countries’ key is just a list of dictionaries, with each dictionary containing key-value pairs corresponding to a specific country. So we need to loop through this list of dictionaries, extracting the values of the ‘Country’ and ‘TotalConfirmed’ keys from each dictionary and then appending them to a new list as follows:

country_list = []for country_info in response_info[‘Countries’]:
 country_list.append([country_info[‘Country’], country_info[‘TotalConfirmed’]])

This will loop through the list of dictionaries, extracting the values from the ‘Country’ and ‘TotalConfirmed’ keys from each dictionary into a list, and then adding this resulting list to our country_list. We will end up with a list of lists, with each list or element in the outer list containing the country name and the total confirmed cases for that specific country.

Creating a Pandas DataFrame

We will now create a pandas dataframe using this country_list and the pandas DataFrame constructor:

country_df = pd.DataFrame(data=country_list, columns=[‘Country’, ‘Total_Confirmed’])

Success! We now have a dataframe that contains two columns: Country and Total_Confirmed!

Conclusion

In this tutorial, we had a brief introduction to what APIs and JSON are. We then made an HTTP request to a Coronavirus COVID19 API to get information on the number of total confirmed coronavirus cases in each country. We then converted this JSON response to our request into a python dictionary. We then parsed through this dictionary, extracting the information we were seeking, and then created a pandas dataframe containing this information.

How does Python fetch JSON data?

How extract JSON data from Python API?

How does REST API send JSON data?

How does JSON process data in Python?