public-apis: The Markdown Directory That Behaves Like a Database
A community-curated API index with strict validation, link health checks, and CI gates that turn a README into durable infrastructure.
- public-apis treats a README as structured data, then protects that data with validation instead of trusting manual curation.
- Its real innovation is operational, not aesthetic: link checks, formatting rules, and PR diff filtering make a volunteer-maintained list behave like a small platform.
- The repository sits between a casual awesome list and a full app catalog, which gives it enough rigor to stay trustworthy without losing openness.
- The project shows that governance is part of product design when the product is a shared reference source.
A README That Acts Like a Database
Most people would call public-apis a list. That undersells it. The repository behaves more like a database whose rows happen to live in Markdown, with CI standing guard over every edit.
That is the interesting trick. The value is not just the catalog of APIs. It is the machinery that keeps the catalog clean, consistent, and link-checked as the web changes underneath it.
In other words, the README is the interface. The scripts are the product.
The Rules That Keep the Catalog Clean
The schema layer lives in scripts/validate/format.py. It reads the README, parses tables, and applies rules that feel like database constraints: capped description length, capitalization requirements, punctuation limits, and checks for HTTPS, CORS, and auth fields.
This is the part that makes the repo feel less like documentation and more like a governed dataset. The rules are opinionated on purpose. They push contributors toward entries that are complete, consistent, and easy to scan.
max_description_length = 100
def check_description(description):
if len(description) > max_description_length:
return False
if description.endswith('.'):
return False
return True
The exact implementation is broader than that snippet, but the pattern is clear. The repo is not asking, "Does this entry look okay to a human?" It is asking, "Does this entry satisfy the rules we have already agreed on?"
How It Checks the Web Without Falling Over
The technical centerpiece is scripts/validate/links.py. It sends requests with a fake user agent, catches failures like SSL errors, connection errors, and timeouts, then normalizes them into standardized outcomes instead of crashing the pipeline.
That matters because not every failed request means the API is gone. Some endpoints are simply defended. The checker looks for Cloudflare-style protection and avoids treating a challenged site as if it were dead.
| Naive checker | public-apis link checker |
|---|---|
| Treats every blocked response as failure | Distinguishes dead links from protected sites |
| Uses a generic request pattern | Uses a fake user agent to reduce bot friction |
| Stops on the first network problem | Maps common failures to standardized error codes |
| Optimized for simple scripts | Optimized for noisy real-world web endpoints |
Why the PR Pipeline Only Checks What Changed
The efficiency move is in scripts/github_pull_request.sh. It fetches the PR diff, extracts added lines, and runs validation only on the new material. That keeps CI focused on the actual change instead of reprocessing the whole catalog on every small update.
curl "$PR_DIFF_URL" | egrep '^\+' | python scripts/validate/links.py
This is a small design choice with outsized impact. It turns the repo from something that could become slow and annoying into something contributors can actually work with.
Curation at Scale Is Mostly About Saying No
The social layer matters as much as the scripts. The contribution rules reject weak entries, marketing-heavy pitches, and sloppy formatting. That is not bureaucracy for its own sake. It is how the project preserves trust.
| Open list without enforcement | public-apis workflow | Heavyweight catalog platform |
|---|---|---|
| Anyone can add anything | Anyone can contribute, but rules filter the result | Curated by product and ops teams |
| Link rot accumulates quickly | Validation keeps the catalog fresh | Usually backed by paid infrastructure |
| Good for discovery, weak for reliability | Good for discovery and trust | Strong governance, higher overhead |
| Low friction, low assurance | Moderate friction, high assurance | High assurance, high complexity |
The best way to read the repository is as a middle ground. It is more serious than an awesome list, but far lighter than a full database product. That is exactly why it works.
What Makes This Different From a Normal Awesome List
A normal community list is passive. It accepts edits, publishes them, and hopes the links stay alive. public-apis turns that model inside out. It encodes rules, checks the web, and uses CI to decide whether a contribution is actually fit for the catalog.
That is the durable insight here. The repo does not win by being bigger or prettier. It wins by treating curation as an operational system.
For developers, that makes the README trustworthy. For maintainers, it makes the project survivable. And for everyone else, it quietly proves that governance is just another kind of engineering.
