Python Requests Library Snippets

1 Introduction to Python Requests

The requests library is Python's elegant HTTP client that makes API calls feel like second nature. It abstracts away the gnarly details of HTTP while keeping the power under the hood. Think of it as your reliable courier for the internet—handles everything from simple page grabs to complex authentication flows.

Why developers love it:

Human-friendly: API reads like plain English instead of cryptic HTTP jargon.
Built-in JSON: Automatic parsing eliminates boilerplate parsing headaches.
Session support: Keeps cookies and connection pooling for you automatically.
Robust error handling: Clear exceptions instead of cryptic status codes.
SSL verification: Secure by default, flexible when needed.

Ready to fetch some data? Let's start with the basics.

2 Prerequisites: Python & Pip

You'll need Python 3.6+ and pip to install the requests library. Let's verify your setup:

python --version && pip --version

If those commands work, you're golden. If not, grab Python from python.org and pip will come along for the ride.

3 Install the Requests Library

First, let's create a fresh virtual environment—this keeps your project dependencies clean and prevents version conflicts:

python -m venv .venv && source .venv/bin/activate  # macOS/Linux
# or
python -m venv .venv && .venv\Scripts\activate  # Windows

Now install requests using pip:

pip install requests

4 Your First GET Request

Let's start with the most common operation—fetching data. This simple snippet grabs a JSON response from a public API:

import requests

response = requests.get('https://httpbin.org/json')
print(f"Status: {response.status_code}")
print(f"Content-Type: {response.headers['content-type']}")

# Automatic JSON parsing
data = response.json()
print(f"Keys: {list(data.keys())}")

Notice the automatic JSON parsing—requests detects the content type and parses it for you. No manual json.loads() dance required.

5 POST Requests with Data

Sending data to a server? POST requests handle form data, JSON, and files seamlessly. Here are the common patterns:

import requests

# Send form data
form_data = {'username': 'alice', 'password': 'secret123'}
response = requests.post('https://httpbin.org/post', data=form_data)

# Send JSON data (automatically sets Content-Type header)
json_data = {'name': 'Alice', 'email': '[email protected]'}
response = requests.post('https://httpbin.org/post', json=json_data)

print(f"Status: {response.status_code}")
print(f"Response: {response.json()}")

The json parameter automatically sets the correct headers and serializes your data. Much cleaner than manual header setting.

6 Custom Headers & Authentication

APIs often need custom headers and authentication. Requests makes this straightforward with built-in auth classes:

import requests
from requests.auth import HTTPBasicAuth, HTTPDigestAuth

# Custom headers
headers = {
    'User-Agent': 'MyApp/1.0',
    'Accept': 'application/json',
    'X-API-Key': 'your-api-key-here'
}

response = requests.get('https://httpbin.org/headers', headers=headers)
print(response.json())

# Basic Authentication
auth = HTTPBasicAuth('username', 'password')
response = requests.get('https://httpbin.org/basic-auth/user/pass', auth=auth)

# API Key in header
api_key = 'sk-1234567890abcdef'
headers = {'Authorization': f'Bearer {api_key}'}
response = requests.get('https://httpbin.org/bearer', headers=headers)

HTTPBasicAuth and HTTPDigestAuth handle the encoding automatically. For modern APIs, Bearer tokens in the Authorization header are the standard.

7 Error Handling & Status Codes

Robust applications handle errors gracefully. Requests raises exceptions for network issues and provides detailed response objects:

import requests
from requests.exceptions import RequestException, Timeout, ConnectionError

try:
    response = requests.get('https://httpbin.org/status/404', timeout=5)
    
    # Raise an exception for bad status codes
    response.raise_for_status()
    
except Timeout:
    print("Request timed out after 5 seconds")
except ConnectionError:
    print("Failed to connect to server")
except RequestException as e:
    print(f"Request failed: {e}")
    
else:
    print(f"Success! Status: {response.status_code}")
    print(f"Headers: {response.headers}")
    print(f"Body: {response.text[:100]}...")  # First 100 chars
    
# Manual status code checking
if response.status_code == 200:
    data = response.json()
    print("Data received:", data)
elif response.status_code == 404:
    print("Resource not found")
else:
    print(f"Unexpected status: {response.status_code}")

raise_for_status() throws an HTTPError for 4xx/5xx responses. Timeouts prevent your app from hanging indefinitely.

8 JSON Processing & Data Handling

Working with JSON data efficiently is where requests shines. It handles encoding, decoding, and validation automatically:

import requests

# Complex JSON handling
response = requests.get('https://httpbin.org/json')

# Safe JSON parsing with error handling
try:
    data = response.json()
    print(f"Type: {type(data)}")
    print(f"Keys: {data.keys() if isinstance(data, dict) else 'Not a dict'}")
except ValueError as e:
    print(f"Invalid JSON: {e}")
    print(f"Raw content: {response.text}")

# Working with nested data
user_data = {
    'name': 'Alice Johnson',
    'email': '[email protected]',
    'preferences': {
        'theme': 'dark',
        'notifications': True,
        'language': 'en'
    }
}

# Send complex nested JSON
response = requests.post('https://httpbin.org/post', json=user_data)
result = response.json()

# Access nested values
nested = result.get('json', {}).get('preferences', {})
theme = nested.get('theme', 'light')
print(f"User theme: {theme}")

Always use safe methods like get() when accessing nested JSON to avoid KeyError exceptions.

9 File Downloads & Streaming

Downloading files efficiently prevents memory issues. Use streaming for large files and check content type:

import requests
import os
from pathlib import Path

# Download a file
url = 'https://httpbin.org/image/jpeg'
response = requests.get(url, stream=True)

if response.status_code == 200:
    # Create downloads directory
    downloads_dir = Path('downloads')
    downloads_dir.mkdir(exist_ok=True)
    
    # Save file
    file_path = downloads_dir / 'image.jpg'
    with open(file_path, 'wb') as f:
        for chunk in response.iter_content(chunk_size=8192):
            if chunk:  # Filter out keep-alive chunks
                f.write(chunk)
    
    print(f"Downloaded {file_path} ({file_path.stat().st_size} bytes)")
else:
    print(f"Download failed: {response.status_code}")

# Get file info before downloading
head_response = requests.head(url)
content_length = head_response.headers.get('content-length')
content_type = head_response.headers.get('content-type')

print(f"File size: {content_length} bytes")
print(f"Content type: {content_type}")

# Download with progress
def download_with_progress(url, filename):
    response = requests.get(url, stream=True)
    total_size = int(response.headers.get('content-length', 0))
    
    with open(filename, 'wb') as f:
        downloaded = 0
        for chunk in response.iter_content(chunk_size=1024):
            if chunk:
                f.write(chunk)
                downloaded += len(chunk)
                
                # Print progress
                if total_size > 0:
                    progress = (downloaded / total_size) * 100
                    print(f"Progress: {progress:.1f}%")

Stream downloads prevent loading entire files into memory. Perfect for large images, documents, or media files.

10 Session Management & Cookies

For applications that need to maintain state across requests—like login sessions—use requests Session objects:

import requests

# Create a session for persistent connections and cookies
session = requests.Session()

# Set default headers for all requests
session.headers.update({
    'User-Agent': 'MyApp/1.0',
    'Accept': 'application/json'
})

# Automatically handles cookies, connection pooling, and redirects
response1 = session.get('https://httpbin.org/cookies/set/session_id/abc123')
print("Cookies set:", response1.cookies.get_dict())

# Subsequent requests include the cookies
response2 = session.get('https://httpbin.org/cookies')
print("Cookies received:", response2.json())

# Session-level configuration
session.auth = ('username', 'password')
session.timeout = 10

# Use context manager for automatic cleanup
with requests.Session() as s:
    response = s.get('https://httpbin.org/json')
    data = response.json()
    print("Using context manager")
    
# Manual cookie handling
cookies = {'sessionid': 'abc123', 'csrftoken': 'xyz789'}
response = requests.get('https://httpbin.org/cookies', cookies=cookies)
print("Manual cookies:", response.json())

Sessions automatically handle connection pooling, which means faster subsequent requests and reduced server load.

11 Best Practices & Common Patterns

Professional HTTP clients follow certain patterns. Here are battle-tested practices for production code:

import requests
import logging
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# Configure logging for debugging
logging.basicConfig(level=logging.WARNING)  # Reduce noise
logger = logging.getLogger(__name__)

class APIClient:
    def __init__(self, base_url, timeout=10, max_retries=3):
        self.session = requests.Session()
        
        # Retry strategy for transient failures
        retry_strategy = Retry(
            total=max_retries,
            status_forcelist=[429, 500, 502, 503, 504],
            method_whitelist=["HEAD", "GET", "OPTIONS"],
            backoff_factor=1
        )
        
        adapter = HTTPAdapter(max_retries=retry_strategy)
        self.session.mount("http://", adapter)
        self.session.mount("https://", adapter)
        
        self.base_url = base_url
        self.timeout = timeout
    
    def get(self, endpoint, **kwargs):
        """Perform GET request with error handling"""
        url = f"{self.base_url.rstrip('/')}/{endpoint.lstrip('/')}"
        
        try:
            response = self.session.get(url, timeout=self.timeout, **kwargs)
            response.raise_for_status()
            return response
        except requests.RequestException as e:
            logger.error(f"Request failed: {e}")
            raise
    
    def post(self, endpoint, **kwargs):
        """Perform POST request"""
        url = f"{self.base_url.rstrip('/')}/{endpoint.lstrip('/')}"
        return self.session.post(url, timeout=self.timeout, **kwargs)

# Usage example
api = APIClient('https://httpbin.org')
try:
    response = api.get('json')
    data = response.json()
    print(f"Got data: {list(data.keys())}")
except Exception as e:
    print(f"API call failed: {e}")

# Environment-based configuration
import os

API_BASE_URL = os.getenv('API_BASE_URL', 'https://httpbin.org')
API_KEY = os.getenv('API_KEY', 'dev-key')

headers = {
    'Authorization': f'Bearer {API_KEY}',
    'User-Agent': f'MyApp/{__version__}'
}

These patterns handle retries, proper error logging, environment configuration, and create a clean abstraction layer for your API interactions.

Key Takeaways

• Use sessions for connection pooling and cookie management
• Always implement timeout and error handling
• Leverage json= parameter for automatic serialization
• Stream large file downloads to manage memory
• Use environment variables for configuration