Requests module in Python: 📭 How to use requests in Python ?

Getting Started with Requests

👉 Every web developer knows HTTP methods and almost their usage. So, how we can handle those methods using python ? Of course one of the efficient way to use those methods is requests module.

📌 The requests module is quite common used module in python. In this essay we are going to investigate what requests is and how to use with most common HTTP methods.

To begin using the requests module it should be installed before. To do this, run the following code:

pip install requests

After installing the module it is time to code ! Make sure the module is imported before code.

import requests

That's it! Now we are ready to use requests. 🛠
Let's take a look at what we are able to do using requests.

The GET Request

One of the most two common HTTP methods GET is used for reach out the data from specific resource. Python provides us get() method that takes data resource as argument in requests module.

To be able to do that we will make GET request to http://api.open-notify.org/. This API page has been created as an open source project for the purpose of information so, you can find something on this page about the ISS and astronauts in the space right now.
Alright, let's code !

def req():
       response = requests.get("http://api.open-notify.org/")
       print(response)

Running out the req() function in the main method we will have our the first request in this manner.

def __name__ == "__main__":
       req()

After run the code above you probably are going to get the following response

Output:
<Response [200]>

As you can see in the output the get() method returned us response by status code 200. That means the request was successful to resource.

📌 Note that there will more info associated with HTTP Status Codes later.

Alright we have a response and its class and we know that all the content is stored in response variable so that we need to access them now. To do this there are some way that we can use. So why don't we try some of them ⁉️

Before getting content of response we are going to check status whether successful or not. It might be useful to make decision on the code.

def status():
       if response.status_code == 200:
           return True
       return False

If the output is True then the request was successful.
‼️ However, note that you can not check out the all status codes one by one. Instead of that you can use raise_for_status() function that will do necessary process for you.

def status():
       try:
           response = requests.get("http://api.open-notify.org/asdasd")
           response.raise_for_status()
       except Exception as e:
           print(f'Error occurred: {e}')

If an exception occurs you would probably get an HTTPError in the output.

So, after checking out the response we can focus on the content of response variable. The content of response variable that you got from the GET method may contain valuable information called as payload. To see the content we are going to use content by response variable.

def content(url="http://api.open-notify.org/astros.json"):
       try:
           response = requests.get(url)
           content = response.content
           print(content)
       except Exception as e:
           print(f'Error occurred: {e}')

Output:
b'{"message": "success", "number": 10, "people": [{"craft": "ISS", "name": "Mark Vande Hei"}, {"craft": "ISS", "name": "Oleg Novitskiy"}, {"craft": "ISS", "name": "Pyotr Dubrov"}, {"craft": "ISS", "name": "Thomas Pesquet"}, {"craft": "ISS", "name": "Megan McArthur"}, {"craft": "ISS", "name": "Shane Kimbrough"}, {"craft": "ISS", "name": "Akihiko Hoshide"}, {"craft": "ISS", "name": "Anton Shkaplerov"}, {"craft": "ISS", "name": "Klim Shipenko"}, {"craft": "ISS", "name": "Yulia Pereslid"}]}'

This function returns us raw byte value of response payload like output above. The output inform us about astronauts in the space right now.

We usually want to get in different form like json, string etc. To convert the response into string just enough using the text function.

def convert_string(url="http://api.open-notify.org/astros.json"):

       try:
           response = requests.get(url)
           string_content = response.text
           print(f"The content of response => {string_content}")
       except Exception as e:
           print(f'Error occurred: {e}')

The output will be into string form like this :

The content of response => {"message": "success", "number"...

If you take a look at the content of response you are going to see it is like JSON. However, how can we access as JSON format? There is an easy way to get content by JSON so requests module provide us its own json() function to convert content of response to JSON format.

def convert_json(url="http://api.open-notify.org/astros.json"):

       try:
           response = requests.get(url)
           json_content = response.json()
           print(f"The content of response => {json_content}")
           print(type(json_content))

       except Exception as e:
           print(f'Error occurred: {e}')

Almost the same output but not in the text form in this time. If you check out the type of json_content variable in the code above you will get .

You can also use pprint module instead of print function while you write the JSON content. You would get the output like following.

The content of response => {'message': 'success',
 'number': 10,
 'people': [{'craft': 'ISS', 'name': 'Mark Vande Hei'},
            {'craft': 'ISS', 'name': 'Oleg Novitskiy'},
            {'craft': 'ISS', 'name': 'Pyotr Dubrov'},
            {'craft': 'ISS', 'name': 'Thomas Pesquet'},
            {'craft': 'ISS', 'name': 'Megan McArthur'},
            {'craft': 'ISS', 'name': 'Shane Kimbrough'},
            {'craft': 'ISS', 'name': 'Akihiko Hoshide'},
            {'craft': 'ISS', 'name': 'Anton Shkaplerov'},
            {'craft': 'ISS', 'name': 'Klim Shipenko'},
            {'craft': 'ISS', 'name': 'Yulia Pereslid'}]}

Working with Headers 🕵🏼

To get more information about the response that you got there is need to access headers of the response. To view those headers python requests module provides us headers function. Take a look at this how to implement headers on response of get request.

def get_headers(url="http://api.open-notify.org/astros.json"):

       try:
           response = requests.get(url)
           headers = response.headers
           print("Headers: ", end="\n\n")
           pprint(dict(headers))

       except Exception as e:
           print(f'Error occurred: {e}')

Output:
Headers:

{'Connection': 'keep-alive',
 'Content-Length': '505',
 'Content-Type': 'application/json',
 'Date': 'Mon, 18 Oct 2021 08:39:23 GMT',
 'Server': 'nginx/1.10.3',
 'access-control-allow-origin': '*'}

A lot of details we can get by the headers as you can see in the output above. To access one of them we can also use python dictionary properties. For instance in the code above we can check the server or whatever you want using this headers.get('Server'). You probably are going to obtain that response nginx/1.10.3.

Query Parameters

Sometimes we may want to filter in the response.Query parameters can be used to customize the response of GET method. To do this the parameters that we have determined should be passed to params in get() function.

For instance;

def query_params(url="http://api.open-notify.org/iss-pass.json"):
       params = {
           "lat":"25",
           "lon":"35"
           }
       try:
           response = requests.get(url, params=params)
           json_content = response.json()
           print(f"Estimate of ISS location according to parameter => ", end="")
           pprint(json_content)
       except Exception as e:
           print(f'Error occurred: {e}')

Output:
Estimate of ISS location according to parameter => {'message': 'success',
 'request': {'altitude': 100,
             'datetime': 1634224449,
             'latitude': 25.0,
             'longitude': 35.0,
             'passes': 5},
 'response': [{'duration': 613, 'risetime': 1634270566},
              {'duration': 590, 'risetime': 1634276384},
              {'duration': 502, 'risetime': 1634300139},
              {'duration': 648, 'risetime': 1634305891},
              {'duration': 193, 'risetime': 1634311894}]}

The code above yields us an estimation about the ISS according to given parameter's data. When you make GET request to this url without query parameter then you would get a response message that it is must.

If you want to get information about current location of ISS then make GET request using this url: http://api.open-notify.org/iss-now.json.

What if we want to approach by OOP to estimate ISS location ⁉️⁉️⁉️

class EstimateISS:

         def __init__(self, url="http://api.open-notify.org/iss-pass.json"):

            self.lat = int(input("latitude: "))
            self.lon = int(input("longitude: "))
            self.url = url

        def estimate_request(self):

            params = {
                "lat": self.lat,
                "lon": self.lon
                }

            try:
                response = requests.get(self.url, params=params)
                self.json_content = response.json()
                return self.json_content

            except Exception as e:
                return f'Error occurred: {e}'

        def __repr__(self):
            return f"{pprint(self.estimate_request())}"

Have a look at the code above that what it does. When the code is run firstly gets latitude and longitude in init function and moves to other key function, estimate_request(). The both parameters and other processes occur in this function. If no error, it estimation about ISS location according to given values

latitude: 45
longitude: 30


{'message': 'success',
 'request': {'altitude': 100,
             'datetime': 1634908665,
             'latitude': 45.0,
             'longitude': 30.0,
             'passes': 5},
 'response': [{'duration': 600, 'risetime': 1634956714},
              {'duration': 655, 'risetime': 1634962480},
              {'duration': 627, 'risetime': 1634968328},
              {'duration': 635, 'risetime': 1634974172},
              {'duration': 652, 'risetime': 1634979986}]}

Working on POST Method

One of the another most famous method is POST. To be able to use this method we need a body that gets data its inside. The body gets dictionary, a list of tuple, byte or object according to corresponding function parameters. Firstly, we are going to use that site https://httpbin.org/ to post method.

def post_data(url="https://httpbin.org/post"):
       data = {"key": "value"}
       response = requests.post(url, data=data)
       print(response)

Response [200]

We got the status code 200 in the response above. So if we need to access response headers is there any header in the response that returned in post method ?
Of course, we can demonstrate the all headers like what we did for get.

def post_data(url="https://httpbin.org/post"):
       data = {"key": "value"}
       response = requests.post(url, data=data)
       headers = response.headers
       pprint(dict(headers))

{'Access-Control-Allow-Credentials': 'true',
 'Access-Control-Allow-Origin': '*',
 'Connection': 'keep-alive',
 'Content-Length': '479',
 'Content-Type': 'application/json',
 'Date': 'Mon, 25 Oct 2021 11:02:35 GMT',
 'Server': 'gunicorn/19.9.0'}

In the output above we can see the headers for post method we sent. Note that those processes are related to web requests and responses so that we can build up communication by JSON data. To use that in the post method just pass data as json parameter.

def post_json_data(url="https://httpbin.org/post"):
       data = {"key": "value"}
       response = requests.post(url, json=data)
       headers = response.headers
       content = response.content
       dec = content.decode("utf-8")
       json_cont = json.loads(dec)
       pprint(json_cont)

In the code above we have changed the data by json and run again. Let's look at the data inside of the response content.

{'args': {},
 'data': '{"key": "value"}',
 'files': {},
 'form': {},
 'headers': {'Accept': '*/*',
             'Accept-Encoding': 'gzip, deflate',
             'Content-Length': '16',
             'Content-Type': 'application/json',
             'Host': 'httpbin.org',
             'User-Agent': 'python-requests/2.26.0',
             'X-Amzn-Trace-Id': 'Root=1-6177e4a2-25a616ba208403692f718992'},
 'json': {'key': 'value'},
 'origin': '176.236.37.52',
 'url': 'https://httpbin.org/post'}

Pay attention to those points data and json inside of the response content. When you look at the data and json keys you are going to see a dictionary that we have sent.
So far so good ‼️

Alright, as we said before this is related to web process so, why we don't write a mini web server and make request to that 🤔 ⁉️

Ok then let's do it ‼️

To be able to do that we'll need some modules one of them is flask. Firstly we will use flask to get requests and return responses. I thought it is easy way in python for now. In the another one sharing we are going to use flask with its all details.

First of all we need to import some packages from flask module.

from flask import Flask, request

Now, we can run out the flask and use its properties.

app = Flask(__name__)

   @app.route("/test/test_get", methods=["POST", "GET"])
   def get():
       if request.method == "GET":
          return jsonify({"message": "Success!"})
       return

To run out flask app we need to add and run this code in main method below.

if __name__ == "__main__":
      app.run(debug=True)

After running out the server you will get an output about it is running.

* Serving Flask app 'test_server' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 735-221-534
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Now, it is time to make request to our test server. Alright we need to move on other python file and write following.

def test_get():
       try: 
        response = requests.get("http://127.0.0.1:5000/test/test_get")
        if response.status_code == 200:
           print("Success!")
      except Exception as e:
           print(e)

Run the code above in a different python file you are going to get Success! if you don't have an exception. Apart of that if you go to browser by this url you would get that response message: Success!.

Let's do something to make post request to the test server. Go back to the file that have flask and write another function called as post() like following.

@app.route("/test/test_post", methods=["POST", "GET"])
   def post():
       try:
           post_data = request.json
           if request.method == "POST":
              return post_data
           return
       except Exception as e:
          return e

Now we need a function to be able to send json data to the server via post request. Ok let's do it !

def test_post():
       header = {"Content-Type":"application/json",}
       url = "http://127.0.0.1:5000/test/test_post"
       post_data = {
               "message": "this message from post request"
           }
       json_data = json.dumps(post_data)
       try:
           response = requests.post(url=url,headers=header, data=json_data)
           return_content = json.loads(response.content)
           if response.status_code == 200:
               print("Success!")
               print("Returned data: ", return_content)

       except Exception as e:
           print(e)

The server gets request that sent from test_post() function and returns the content of request. When we run out the both file the outputs will be like following. Firstly the server return us status code and the function prints out message according to this status code.

127.0.0.1 - - [28/Oct/2021 10:45:47] "POST /test/test_post HTTP/1.1" 200 -

Success!
Returned data:  {'message': 'this message from post request'}

So far so good !
You can also diversify using other HTTP methods like PATCH, PUT, DELETE. We are going to write the last instance using one of them.

This time we got PUT method to use with function. Suppose we have list database holding dictionary data in the test server and we are going to change something on the data using PUT. Let's code it!

@app.route("/test/test_put", methods=["POST", "PUT"])
   def put():

       _db = [
           {"Id": "1",
            "user": "fake",
            "pass": "fake"}
           ]    
       try:
           put_data = request.json
           if request.method == "PUT":
              for _ in _db:
                  if _["Id"] == put_data["Id"]:
                     _["user"] = put_data["user"]
                     _["pass"] = put_data["pass"]
           print(_db)
           return put_data
       except Exception as e:
           return e

In the code above _db created like database and db updated in try block according to Id using incoming data.

def test_put():
       header = {"Content-Type":"application/json",}
       url = "http://127.0.0.1:5000/test/test_put"
       put_data = {
               "Id": "1",
               "user": "fakeuser",
               "pass": "fakepass"
           }
       json_data = json.dumps(put_data)
       try:
           response = requests.put(url=url,headers=header, data=json_data)
           return_content = json.loads(response.content)
           if response.status_code == 200:
               print("Success!")
               print("User Updated by: ", return_content["user"])

       except Exception as e:
           print(e)

When we run the code files in order, outputs will be like following.

Success!
User Updated by:  fakeuser

And the server outputs:

[{'Id': '1', 'user': 'fakeuser', 'pass': 'fakepass'}]
127.0.0.1 - - [01/Nov/2021 09:30:45] "PUT /test/test_put HTTP/1.1" 200 -

Server side also prints out the updated _db and status code 200. We are be able to create new record into the database using POST method and delete it again using DELETE method. Let us build up delete function to delete record that already created.

def test_delete():
       header = {"Content-Type":"application/json",}
       url = "http://127.0.0.1:5000/test/test_delete"
       delete_data = {
               "Id": "1",
               "user": "fakeuser",
               "pass": "fakepass"
           }
       json_data = json.dumps(delete_data)
       try:
           response = requests.delete(url=url,headers=header, data=json_data)
           if response.status_code == 200:
               print("Success!")

       except Exception as e:
           print(e)

Now let's write the server side.

@app.route("/test/test_delete", methods=["POST", "DELETE"])
   def delete():
       _db = [
           {"Id": "1",
            "user": "fake",
            "pass": "fake"}
           ]    
       try:
           delete_data = request.json
           if request.method == "DELETE":
               for _ in _db:
                   if _["Id"] == delete_data["Id"]:
                       _db.remove(_)
           print(_db)
           return delete_data
       except Exception as e:
           return e