`requests`

Abstract

requests is a Python package used to make web requests over Hypertext Transfer Protocol, or HTTP.

If you’ve used the internet for a while, you’ve probably noticed "http" or "https" at the beginning of most links and websites. HTTP is a protocol for communication between servers and clients. A server would be a fancy computer running in an AWS (Amazon Web Services) warehouse, while a client could be you, surfing the web on your computer!

HTTP has requests and responses. A client, such as your browser, sends a request to a server, and server then returns a response to the client.

The basis of HTTP is methods such as GET, POST, PUT, HEAD, DELETE, PATCH, and OPTIONS. You can read more about those here. The important thing is to realize that the requests package enables you to easily "use" these methods in Python. For example, the GET method is used to get data from a server:

import requests
response = requests.get("https://datamine.purdue.edu/")
print(response)

<Response [200]>

In most instances, you will only use requests to establish a connection to the webpage, then parse it using another package, such as lxml, beautifulsoup4, or selenium, all of which have their own Data Mine documentation. Happy scraping!

Examples

How do I scrape the HTML from datamine.purdue.edu/?

response = requests.get("https://datamine.purdue.edu/")
print(response.text[:500])

 <!DOCTYPE html>
 <html lang="en">

 <head>
     <!-- general meta data Here -->
     <title>The Data Mine</title>
     <meta content="The first large-scale living learning community for undergraduates from all majors, focused on Data Science for All." name="description" />
     <meta content="Marketing and Media" name="author" />
     <meta charset="utf-8" />
     <meta content="IE=edge" http-equiv="X-UA-Compatible" />
     <meta content="width=device-width, initial-scale=1" name="viewport" />


     <lin

How do I get the status code from the `requests` response?

response = requests.get("https://datamine.purdue.edu/")
print(response.status_code)

Documentation

Here is the official documentation for the requests library. Our examples are usually introductory and searching official documentation is a skill you’ll want to take far into the future. Never be afraid to go beyond these pages for help!

requests