bayesian-ab-tester: Bayesian A/B Testing Without the Server

How hwong103/bayesian-ab-tester moves 100,000 Monte Carlo simulations from the cloud into a single browser tab.

A vintage mechanical calculator floating inside a transparent glass lightbulb, with thousands of tiny dice tumbling around inside it. This represents complex statistical calculations being entirely contained within the lightweight environment of a web browser. — Moving heavy statistical simulations from backend servers directly into the client.

Key Takeaways

Modern JavaScript engines enable 100,000 Monte Carlo simulations to run natively in a browser tab without backend infrastructure.
Bayesian statistics prioritize expected loss over p-values to quantify the actual risk of choosing a specific variant.
A fat-client architecture uses anonymous Supabase authentication to sync experiment data while maintaining user privacy.
The project demonstrates the migration of complex data science workflows from dedicated Python environments to accessible web utilities.

100,000 Dice Rolls in a Tab

Historically, Bayesian Monte Carlo simulations required a heavy Python or R environment. The math is computationally expensive. It involves drawing thousands of random samples from probability distributions to determine a winner. hwong103/bayesian-ab-tester proves that modern browser performance can handle this workload natively. By executing 100,000 simulations directly in JavaScript, the tool eliminates backend latency and infrastructure costs entirely.

How Monte Carlo simulations build probability distributions sample by sample.

Probability, Not P-Values

Frequentist A/B testing relies on p-values, a metric notorious for confusing product managers. Bayesian statistics offers a more intuitive output: a direct percentage chance that one variant beats another. More importantly, it calculates 'Expected Loss'. This metric quantifies the risk of making the wrong choice. It allows teams to launch a winning variant even if it has not reached absolute statistical significance, provided the downside risk is negligible.

A tightrope walker crossing a chasm, holding a glowing gauge labeled 'Expected Loss' instead of a balancing pole. This illustrates the concept of using Bayesian statistics to manage risk rather than merely seeking statistical significance. — Bayesian A/B testing frames decisions around risk management and expected loss.

The Fat Client Architecture

The architecture embraces a 'Fat Client' model. All logic lives in the browser. For persistence, the project uses a hybrid approach. It defaults to localStorage for immediate, offline-capable saving. To sync across devices, it leverages Supabase. The integration uses anonymous authentication, allowing users to back up their experiment history to a PostgreSQL database without ever creating an account or entering a password.

Feature	bayesian-ab-tester	Traditional SaaS (e.g. Optimizely)	Python Notebooks
Infrastructure Cost	Zero (Client-side compute)	High (Subscription based)	Low (Local compute)
Data Privacy	Absolute (Data never leaves browser)	Low (Data sent to third party)	High (Local execution)
Setup Time	Instant (Open URL)	Days (SDK integration)	Minutes (Environment setup)
Statistical Output	Bayesian (Expected Loss)	Frequentist (P-values)	Flexible (Requires coding)

From Python Script to Browser Utility

The repository reveals its lineage. It began as a standalone Python script, bayesian_ab_tester.py, designed for offline data science tasks. The transition to a web utility involved translating Beta-Binomial conjugate prior updates into JavaScript. This evolution highlights a broader trend: as WebAssembly and JavaScript engines mature, tools that once required dedicated data science environments are becoming universally accessible web utilities.

def update_prior(prior_alpha, prior_beta, successes, trials):
    # Conjugate prior update for Beta-Binomial model
    new_alpha = prior_alpha + successes
    new_beta = prior_beta + (trials - successes)
    return new_alpha, new_beta