DataUnion: The Marketplace That Turns Training Data Into a Consent Ledger

A browser-first open-source stack for collecting, validating, licensing, and paying out on AI training data without pretending a dataset is just a file.

8 min read • View on GitHub • More from theDakshJaitly

A wide archival hall where individual data samples move from contributors into a central ledger machine, then split back into many small payout lines. The scene explains DataUnion's core idea: samples are tracked as contributions, not dumped into a generic bucket. — DataUnion treats training data like a system of traceable assets, not an anonymous upload queue.

Key Takeaways

DataUnion treats each sample as a traceable asset, so licensing revenue can fan back out to contributors instead of disappearing into a flat fee.
Its browser worker does the first quality pass, which shifts validation closer to upload and keeps the server from becoming the choke point.
Transparency is built into records and routes, not marketed as a promise.
The repo matters less as a marketplace UI than as a different contract for training data, one built around consent and attribution.

DataUnion starts with a blunt claim: a training dataset is not a blob of content, it is a ledger of contributions. Once you accept that, the repo stops looking like a marketplace clone and starts looking like a rights system, one that has to remember who uploaded what, how it was scored, and how money should flow back out.

A dataset with a cap table

The core tables make that idea concrete. data_contributions stores the sample, datasets collects the licensed bundle, and payout_records unwinds a sale back to individual contributors. The point is not bookkeeping for its own sake. It is to make ownership, value, and distribution legible at sample level.

data_contributions
 datasets
 payout_records
 usage_logs

That shape matters because it changes the unit of value. The buyer licenses a dataset, but the system can still account for which contributions belong inside it, which ones are valuable, and how the license fee fans back out. In other words, the product does not stop at collection. It keeps going until the money lands.

Why consent became the product

The market pressure behind the repo is obvious. AI teams want data with provenance, clean rights, and a defensible story for legal review. DataUnion answers that by making consent, quality, and payout part of the same flow. It is not trying to sell a prettier upload form. It is trying to make scraping feel obsolete.

A sample moves from upload to validation to licensing to payout, and the browser gets the first vote.

The browser does the first pass

A close-up of a browser window that routes a raw sample through a worker thread before it becomes a stored contribution. The illustration explains that validation happens near the user, which reduces server load and keeps the intake gate responsive. — The quality gate sits in the browser, where the page can score a sample before the backend commits to it.

The clever part is in lib/ai/valuation.worker.ts and the surrounding browser ML plumbing. Heavy work happens in a worker, using browser-side models and lightweight NLP so the page does not freeze while a sample is checked. That is a product choice as much as a technical one. It keeps the first quality gate close to the upload moment and away from the server.

Trust is not a banner, it is a system

The transparency layer is where the repo tries to earn its claims. Routes like app/contributor/transparency and the UsageLog shape in lib/types.ts make usage visible instead of implied. That matters because trust in a data marketplace is not a slogan. It is the ability to answer a blunt question: who used this, and for what purpose?

What DataUnion replaces

Model	Data provenance	Consent handling	Quality control	Payout logic	Trust story
DataUnion	Per-sample and traceable	Built into intake	Client-side scoring before ingest	Revenue fans out by weight	Visible usage logs and contribution records
Scrape then filter	Weak or missing	Retroactive at best	Batch cleanup after collection	No contributor payout model	Depends on legal posture and cleanup
Flat-fee marketplace	Partial, often dataset-level	Opt-in at upload time	Usually manual or after-the-fact	One-time sale or platform fee	Trust comes from brand, not records
Generic storage plus manual review	Whatever the team documents	External to the pipeline	Human review and spreadsheets	Ad hoc distribution	Trust depends on ops discipline

The table is the real position statement. DataUnion is not trying to beat every marketplace on catalog size. It is trying to replace a stale transaction model with one where provenance, quality, consent, and payout are the same system.

A polished prototype with real ambition

The codebase reads like a high-fidelity prototype built by someone who understands what trust-heavy software needs. The App Router layout, Supabase-backed data model, and polished UI system all point in the same direction. Still, the repo also signals that the hardening work is not finished. That is fine. The interesting part here is not polish alone. It is the ambition to turn training data into something that can be governed, priced, and paid out with receipts.