xread: The Web-to-LLM Compiler That Chooses the Right Browser Only When It Has To

A self-hosted fork that strips out cloud dependencies, adapts between curl, JSDOM, and Puppeteer, and turns messy pages into clean Markdown or JSON for RAG pipelines.

9 min read • View on GitHub • More from xixu-me

By Repo Explainer • AI-assisted • Published by Ron Kurti

Published Apr 16, 2026 • Advanced RAG

Key Takeaways

xread treats web ingestion like a routing problem, so it can stay fast on simple pages and escalate only when fidelity demands it.
Its real differentiator is not scraping coverage, but the decision engine that chooses between curl, JSDOM, and Puppeteer.
The fork matters because it removes hidden infrastructure dependencies and makes the stack genuinely self-hostable.
The formatter is the bridge that turns page state into durable Markdown or JSON without flattening away structure that LLMs still need.

The web is a hostile input format for LLMs. Pages arrive with navigation chrome, script-heavy layouts, PDFs, ads, and half the content hidden behind JavaScript. xread exists to solve that mismatch without handing the job to a hosted service.

The web is not one problem

That sounds obvious until you try to ingest the open web at scale. A static article, an SPA, and a login-protected page are three different extraction problems wearing the same URL shape. xread starts from that premise and refuses to treat them all the same.

A close-up workshop scene shows a cluttered DOM tree entering a formatter bench and emerging as a clean document. Navigation bars, sidebars, and chrome are peeled away like packaging, while tables, image alt text, headings, and readable paragraphs survive the transformation. It explains that the formatter preserves semantic structure instead of merely stripping tags. — The formatter is the part that turns a page into usable context. It keeps the structure LLMs care about and discards the noise that wastes tokens.

xread’s real superpower is choosing the cheapest path that still works

The clever part is not extraction itself. It is the runtime choice. xread can start with a raw fetch, move to JSDOM plus Readability for simpler pages, and escalate to Puppeteer only when the page really needs a browser.

The pipeline is adaptive, not fixed. xread chooses the lightest route that preserves meaning and only pays browser cost when the page forces it.

const snapshot = await crawler.crawl(url, {
  cookies,
  scripts,
  output: 'markdown'
})

// The runtime can decide whether this becomes curl, JSDOM, or Puppeteer
// before the formatter turns the result into LLM-ready context.

That choice is the project’s main engineering claim. A browser-only scraper pays the heaviest cost every time. A dumb fetcher is cheaper, but it breaks the moment the page moves behind client-side rendering. xread sits in the middle and makes the trade-off explicit.

The fork is the product

xread is also a decoupling story. The standalone release removes internal infrastructure assumptions from the original Jina Reader lineage and replaces them with local roots like STORAGE_ROOT and LOCAL_DB_ROOT. In practice, that turns a useful system into something you can actually deploy.

Dimension	Hosted scraping API	Generic fetch-and-convert	xread
Deployment	Vendor-managed	Self-run, but shallow	Self-hosted and local-first
Adaptability	Opaque rules	Usually one extraction mode	Chooses curl, JSDOM, or Puppeteer
Privacy	Data leaves your stack	Depends on implementation	Keeps control in your environment
Cost control	Usage-based	Cheap, but brittle on complex pages	Pays browser cost only when needed
Output	Usually normalized	Often plain text only	Markdown and JSON with preserved structure

That is why the fork matters. It is not just a license story. It is a deployment story. The codebase stops depending on hidden services and starts behaving like infrastructure you can own.

Why the service split matters

The repository does not expose one monolithic mode. It splits into crawl, search, and serp entry points, then narrows the RPC registry so each service stays scoped. That keeps the codebase unified without making the product feel bloated.

// Conceptual shape of the service split
crawl  -> single URL extraction
search -> LLM-friendly search text
serp   -> structured search JSON

// Each mode can selectively disable registry tags
// so the exposed surface stays narrow.

This is a discipline signal, not a feature list. The same codebase can support multiple jobs, but each job still gets its own contract. That matters when the tool is meant to sit in a pipeline instead of a demo.

What xread beats, and what it still is not

xread is strongest when you want local control, adaptive extraction, and output that still resembles the original page structure. It is weaker when you only need the simplest one-liner or when you are perfectly happy handing the whole problem to a hosted API.

Tool type	Best for	Weak spot
Hosted scraping API	Fast onboarding and managed ops	Less control over data path and runtime choices
Bare HTML fetcher	Simple static pages	Breaks on JS-heavy or heavily structured pages
Browser-only extractor	Maximum fidelity on dynamic sites	High cost even when a lighter path would work
xread	Self-hosted RAG ingestion with adaptive fidelity	More moving parts than a trivial fetch script

That is the honest center of the project. xread is not the lightest possible answer to every web page. It is the answer when you care about preserving meaning, choosing the cheapest viable path, and keeping the stack in your own environment.

The production posture

The maturity signals are real: strict TypeScript, integrity checks, audit scripts, Docker, Caddy, and dependency overrides. That is the shape of infrastructure that expects to be deployed, not just cloned.

In other words, xread is not a novelty scraper. It is a piece of plumbing for teams that want web context without surrendering control over the fetch path, the parser, or the output format.