The Immutable Titan: Inside psf/requests

The Burden of Ubiquity

With over thirty million downloads a week and integration into more than a million repositories, Requests is the undisputed standard for Python networking. It is the lingua franca of API consumption. Yet, the library is effectively feature-frozen. The ecosystem has evolved toward asynchronous programming and HTTP/3, but Requests remains strictly synchronous and bound to HTTP/1.1.

This is not a failure of imagination. It is the reality of maintaining foundational infrastructure. When a project is this deeply embedded in enterprise codebases, stability becomes the only metric that matters. Every proposed change carries the risk of breaking millions of production systems.

The maintainers face an impossible balancing act. They must keep the library secure and functional against modern transport layers without altering the public API surface. This tension dictates every architectural decision within the project.

HTTP for Humans

To understand why Requests won, you have to look at the dark era of Python networking it replaced. Before Requests, developers relied on the standard library's urllib2. Making a simple authenticated API call required navigating a maze of handlers, openers, and manual byte encoding.

Requests introduced a declarative, human-centric API. It hid the protocol's complexity behind verbs that developers intuitively understood.

# The old way (urllib2)
import urllib2
password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, "https://api.example.com", "user", "pass")
auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)
urllib2.install_opener(opener)
response = urllib2.urlopen("https://api.example.com/data")

# The Requests way
import requests
response = requests.get("https://api.example.com/data", auth=("user", "pass"))

This ergonomic revolution shifted the focus from managing sockets to managing data. It set a new standard for what a "Pythonic" library should look like, influencing API design across multiple languages.

The Prepare-Send-Receive Pipeline

Under the hood, Requests is a sophisticated state machine built on top of the lower-level urllib3 library. The architecture is defined by the strict separation of user intent from network transport.

When you call requests.get(), you are creating a Request object. This represents your intent. The library then processes this intent through a Session, merging it with persistent state like cookies and default headers. The result is a PreparedRequest. This object is immutable and contains the exact bytes ready for the wire.

The final crucial component is the HTTPAdapter. Adapters map URL prefixes to specific transport mechanisms. This plug-and-play design allows Requests to manage complex connection pools and retry logic without cluttering the user-facing API.

The Async Dilemma

The Python ecosystem has aggressively adopted asyncio for high-concurrency workloads. Requests, however, is deeply rooted in synchronous execution. Its core dependencies and internal state management assume blocking I/O operations.

Retrofitting asynchronous capabilities into a deeply synchronous, stateful library is practically impossible without a complete rewrite. The community fundraiser for "Requests 3" promised these modernizations, but the architectural chasm proved too wide to cross without breaking millions of downstream users.

This architectural lock-in has created a vibrant market for successors. Libraries like HTTPX and Niquests have explicitly cloned the Requests API while building on modern, async-native foundations.

When to Graduate

For the vast majority of Python tasks, Requests remains the correct choice. Data science notebooks, simple cron jobs, and internal automation scripts rarely require the overhead of an asynchronous event loop or the performance gains of HTTP/3.

However, when building a high-throughput web scraper or integrating with modern APIs that mandate HTTP/2 multiplexing, it is time to look at the successors Requests inspired. The library succeeded so completely at defining the developer experience that its clones are now carrying that legacy into the asynchronous future.

Sources:

• psf/requests Repository
• Python Community Interview With Kenneth Reitz
• Revived the promise made six years ago for Requests 3
• Requests Issue #6833: Lack of respect for questions and support requests