Benchmarks & Eval

By tracing workload shape, attributing stalls causally, and testing interventions with rollback guards, Git.M turns runtime behavior into a performance control surface.

A minimal replication of Sakana AI’s RL’s Razor experiment, where a single extra input bit, a KL-minimal oracle, and group-relative RL turn catastrophic forgetting into something you can inspect, compare, and predict.

A single-file Node.js auditor that turns your site into a scorecard for identity, structure, depth, connectivity, and machine-friendly discoverability.

Aiden Ybai built a deliberately cursed Next.js app to measure source-file retrieval, not code generation. The result is a benchmark for the browser era of AI coding.

A benchmark where the payload is a rubric, not an answer key, and the real test is whether a model can follow instruction hierarchy, refuse safely, and stay objective.

It does not just check whether your app rendered. It records how Vite behaved, whether HMR was accepted, and whether a passing test was quietly contaminated by hidden failures.

A tiny markdown repository that distills DBMS, operating systems, and computer networks into interview-sized mental models, with analogies that make the fundamentals stick.

A training-free system for scientific discovery that spends compute on proposals, execution, and selection instead of longer chains of thought.

A Google Research prototype that decomposes policy into a concept graph, then recombines it into grounded synthetic eval prompts.

OpenAI's reproduction code for hallucination research shows how evaluation rules can push models toward guessing, and how a consistency check tries to pull them back.

A deep dive into the judge, minifier, and prompt loop that squeeze correct solutions down to the smallest possible Python.

Inside a Flight Director simulator where telemetry checks, anomaly handling, and burn decisions are scored by deterministic physics.

openai/model_spec_evals turns OpenAI’s Model Spec into an executable test suite, where a judge model writes the critique, median sampling reduces noise, and 6 becomes the line between pass and fail.

A compact NSGA-II implementation shows how Pareto ranking, constraint handling, and crowding distance turn multi-objective optimization into a survival game.

A five-part LangSmith curriculum that starts with a recipe bot and ends with trace replay, judge alignment, retrieval metrics, and transition-matrix failure analysis.

A portfolio project that moves from SQL and Power BI to machine learning and a Flask app, then turns customer risk into concrete business action.

A hand-coded NumPy simulator turns an AAAI theory paper into a geometric story about basins of attraction, augmentations, and feature recovery.

A local-first harness for comparing raster-to-vector pipelines, stripping away background noise, extracting fast metrics, and making “better SVG” less subjective.

A one-file Python pipeline turns raw applicant data into a credit-risk prediction, exposing the tradeoffs between speed, interpretability, and production readiness.

A practical trading framework for NSE BANKNIFTY options that combines defensive data cleaning, cross-sectional z-scores, and walk-forward retraining to surface mispricing signals.

`rl_directional_straddle` is a deep RL options system for NIFTY50 that masks invalid actions, simulates margin pressure, and treats survivability as part of the objective.

A deep dive into the University of Pisa-style systems exam harness that turns agent evaluation into a full-stack stress test: translation, compilation, emulation, and exact-output verification.

A text-only baseline for the Claude introspection claim, using semantic marination, corrupted logs, and judge loops to ask whether “self-awareness” is just prompt pressure in disguise.

a5c-ai/obedience-benchmark measures process fidelity, not just task completion, with Docker isolation, structured logs, and a judge that scores how closely an agent followed the script.

model-selection-benchmark scores models on tool use, multi-step behavior, latency, and accuracy, so teams can choose an LLM for agentic work without guessing.

How a localized LLM leaderboard uses manual stream parsing and a mathematical "underdog bias" to solve the Elo cold-start problem.

How a medical imaging pipeline upscales tiny 28x28 CT scans to hijack massive Vision Transformers, achieving 99.69% accuracy on transient GPU clouds.

How a minimal benchmarking suite exposed the hidden first-command penalty haunting ephemeral compute.

A notebook-built experiment that turns Spotify’s audio features into machine-made groupings, then asks whether genre is acoustic truth or just metadata with better branding.

A Streamlit strategy room that fuses live timing, historical priors, and XGBoost predictions to guess when a move is coming before the TV graphics do.

A deep dive into the batch-and-score system that evaluates conversation turns across hundreds of facets, from linguistic quality to highly specific domain and persona signals.

A Streamlit dashboard, an explainable ensemble model, and a data pipeline built around one hard truth: T20 cricket is noisy, and good models should admit it.

Inside a compact credit-risk pipeline that stages a model showdown, confronts class imbalance, and uses SHAP to turn predictions into defensible reasons.

A compact MySQL case study that uses synthetic data, string hacks, and window functions to teach the part of analytics most demos skip: coping with bad structure and still producing usable answers.

NavjotML’s model scores defenders not just for winning the ball, but for breaking the opponent’s passing structure, especially when the target is a central node.

A close look at how this solver turns a cube into arrays, then bitboards, then a search engine that treats every move like a cheap register operation.

A Python pipeline that syncs kills to game state, measures crosshair geometry, and clusters players into roles without relying on gut feel.

A fine-tuned LLM, classic interpretability methods, and a deliberately structured research workflow aimed at turning model internals into explanations people can read.

A controlled shootout between OpenAI Agents, CrewAI, LangGraph, Google ADK, and Microsoft Agent Framework, all pointed at the same sports workflow.

A 16MB challenge that turns transformer design into compression engineering, with depth recurrence, Muon, and quantization as the main weapons.

A tiny study repo built from Hands-On Machine Learning Chapter 2, where stratified sampling, pipelines, and data hygiene do the real work.

code-chaos turns YAML into repeatable backend workflows, stitching HTTP calls, SQL, and message brokers into one execution loop.

A compact Python repo that turns pairwise wins into an Upper Confidence Bound loop, showing why preference feedback can be more useful than numeric reward.

How a render-and-compare reinforcement learning loop uses computer vision to train Vision-Language Models for pixel-perfect Tailwind generation.

How a Python CLI uses LLM evaluators to withhold advanced automation until researchers prove they can do the work manually.

Why the next generation of experimenters is ditching the p-value for the Probability of Being Best.

How hwong103/bayesian-ab-tester moves 100,000 Monte Carlo simulations from the cloud into a single browser tab.

Why the most important tool in the modern AI stack isn't a library, but a map of the from-scratch intuition required to move beyond prompt engineering.