clippercard: The Guerilla API for Public Transit
How a fragile web scraper survived for over a decade to provide the developer access that a major transit agency refused to build.
I encourage the staff of MTA reading this project to see this effort as a nudge for a public and official API. The moment they put up an API that obsoletes this project, I will happily direct followers to the official solution.
- clippercard operates as software-as-protest, designed to force the Metropolitan Transportation Commission to release an official API.
- The project achieves remarkable longevity by elevating brittle web scraping into a resilient, production-grade library using static HTML test fixtures.
- Unlike hardware-level NFC parsing tools which read instant localized state, clippercard extracts global account history by aggressively simulating a stateful web browser.
The API That Shouldn't Exist
Most open-source projects exist to solve a technical problem. clippercard exists to solve a political one. It is a "web-scraper-as-an-API" built to bridge the gap between the Metropolitan Transportation Commission’s (MTC) closed ecosystem and the needs of San Francisco Bay Area developers. The goal is not just utility, but leverage.
This explicit design philosophy—building a tool with the sole intention of rendering it obsolete—frames the repository as a form of digital activism. It provides the programmatic access to transit balances and history that users demand, while daring the transit authority to shut it down.
Simulating the Browser
Beneath the surface, clippercard is an exercise in rigorous state management. Because the source website lacks a semantic structure, the Python script must perfectly mimic a human browsing session. In clippercard/client.py, the ClipperCardWebSession inherits from requests.Session to maintain authentication cookies.
The true complexity lies in bypassing basic security measures without triggering alarms. The script fetches the login page, extracts a dynamically generated CSRF token from a hidden input field, and posts the credentials. It then caches the resulting HTML "soup" to prevent redundant network calls, minimizing the load on the MTC servers and reducing the risk of rate-limiting.
The Art of the Immortal Scraper
Web scrapers are notoriously fragile; they break the moment a target website changes a class name or nested div. Yet, clippercard has survived for over a decade. The secret to its longevity is its testing strategy.
The repository relies heavily on a tests/data/ directory packed with static HTML snapshots of the Clipper website. These offline fixtures allow the parser to be continuously regression-tested against known layouts without hitting live servers or requiring real user credentials. It is a fossil record of web design, preserving the exact state needed to ensure the extraction engine remains functional.
Scraping the Web vs. Reading the Chip
The approach taken by clippercard stands in stark contrast to hardware-level integrations. For example, the popular Flipper Zero firmware includes an NFC parser written in C that interacts directly with the Mifare DESFire chip inside the physical transit card.
| Feature | clippercard (Python) | Flipper Zero (C Firmware) |
|---|---|---|
| Access Method | Web Scraping via HTTP | Direct NFC Hardware Read |
| Data Source | Global Account Database | Local Chip Memory |
| State | Historical & Global Balance | Instant Read-Only Local State |
| Longevity Risk | High (Requires website stability) | Low (Standardized NFC protocols) |
While the hardware approach provides instant, localized data without an internet connection, it cannot access broader account history or manage multiple cards. clippercard brute-forces the web layer to provide a comprehensive, global view of the user's transit data—proving that sometimes, the messy software hack is the only way to get the full picture.