OSIPI-DOCS: The MRI Quality Checker That Refuses to Guess

A GSoC toolbox grades perfusion scans PASS, WARN, or FAIL across brain, kidney, and placenta, and documents exactly which of its 90 thresholds came from a paper, which came from someone else's code, and which came from nowhere at all.

9 min read • View on GitHub • More from Jitmisra

Black ink editorial illustration of a radiology reading room. Three film sheets showing a brain, a kidney, and a placenta hang side by side on a lightbox while a hand presses a rubber stamp onto the brain film. Behind the lightbox stands an open filing cabinet with dozens of numbered drawers, and a single thread runs from the stamp back into one drawer. It depicts the toolbox's verdicts, provenance chain, and the three organs it covers.
The stamp is the verdict. The thread is the provenance. The filing cabinet is the 90 thresholds underneath.
Key Takeaways

44 of 90

Most quality-control tools for medical imaging hand you a cutoff and expect you to trust it. A number appears in the config, a scan passes or fails, and nobody asks where the number came from. The OSIPI-DOCS repository does the opposite. It publishes the receipts.

The repo is a design-and-evidence archive for a Python toolbox that grades arterial spin labelling (ASL) perfusion scans. It is documentation, not the software itself; the package lives in a separate repository. What makes it worth reading is a single ratio it puts in its own report: of 90 thresholds, 12 are published, 34 come from someone else's implementation, and 44 are uncalibrated. Nearly half the numbers are engineering defaults with no source at all.

The project does not hide this. It sorts every threshold into one of three bins.

That last rule is the load-bearing one. An uncalibrated threshold can raise a warning, but it cannot, by itself, fail a scan. The honesty is not a disclaimer buried in a README. It is wired into the verdict logic.

The Verdict Ladder

A scan does not simply pass or fail. Each check returns a metric, a verdict, and a one-line reason. There are five verdicts, and the two unusual ones carry the design's philosophy.

PASS means the metric sits inside the healthy band. WARN means it is borderline. FAIL means it crossed a fail cutoff. Then there is UNKNOWN, for a check that could matter but cannot be derived from the data on hand. And N/A, for a check that structurally cannot apply. A control/label swap test on a pre-subtracted GE image is N/A, because the image has no control and label volumes left to swap.

The distinction matters. Many QC tools collapse "we could not tell" into "fine." OSIPI-DOCS refuses. UNKNOWN never invents a PASS or a FAIL. It is counted in a coverage figure, so a reader can see how much of the checklist actually ran, but it does not move the overall verdict. The tool trades apparent completeness for trustworthiness, which is the right trade in a multi-site study where a silent default is more dangerous than an abstention.

The verdict algebra: checks light up, provenance badges gate which ones can fail a scan, and UNKNOWN results feed the coverage bar without ever touching the verdict.

VerdictMeaningEffect on overall verdict
UNKNOWNThe check applies but the data cannot resolve itCounted in coverage, does not move the verdict
N/AThe check structurally cannot apply to this scanNot counted at all

Every Number Has a Page Number

Provenance here is not a footnote. It is a first-class artifact. Each published threshold carries a classification, a citation, a quoted sentence, and a screenshot of the source page. The sources/ directory holds the page images, and evidence.json and highlights.json record the machine-readable version. Any threshold can be traced back to the sentence it came from.

Black ink close-up illustration of an index card lying on a desk with a single large number printed on it. A thin thread runs from the card's corner to a small pinned page fragment showing one highlighted sentence. Three rubber stamps sit nearby, one worn smooth from use, and a magnifying glass rests on the card with its lens covering only part of the number. It depicts the provenance chain behind a single threshold value.
The worn stamp is the uncalibrated default. The partial lens is honest partial knowledge.

This surfaces things most tools would rather not discuss. The design docs record an explicit divergence between ASLPrep's code and the paper behind it. ASLPrep uses a tissue threshold of 0.7 where the Dolui paper says 0.9, and a qei_c of 0.054 where the paper says 0.1. The toolbox follows the code and flags the discrepancy as an open question instead of resolving it silently. Documenting a known mismatch between a reference implementation and its own paper is rare.

The same discipline applies to derived values. A neonatal deep grey-matter band of 1.3 to 2.6 is labeled Uncalibrated and described as "derived from Miranda 2006," because the document treats a derivation as a derivation, not a published number. A QEI pass cutoff of 0.55 is listed as uncalibrated too, described only as "a small margin above 0.50."

Two Streams, One Report

Quality for ASL splits into two questions with different inputs. Stream A asks whether the scan was acquired correctly, reading the raw 4D series, the M0 image, and the BIDS metadata. Stream B asks whether the final CBF map is good, reading the map and tissue probability maps. The two streams run independently and merge into one report.

The important scoping decision is what Stream B does not do. It never generates a CBF map. That step belongs to PyASL or ASLPrep, and the design doc draws it as a grey dashed box marking the boundary between their job and ours. The toolbox is a downstream grader. It is a layer, not a pipeline.

When There's No Sidecar

Real test data did not arrive neatly. It came as bare .nii.gz files with no BIDS sidecars and no aslcontext.tsv, which means the toolbox cannot read the acquisition type off the metadata. Instead, check 8.2 infers it from the NIfTI shape, the volume count, and the filenames.

Vendor caseNIfTI shapeInferred acquisition type
GE 3D pCASL (ADNI)128x128x40Pre-subtracted difference image, no pairs, M0 present
Siemens 2D pCASL80x80x20x80Control/label series, no M0
Siemens BS 3D pCASL88x88x52x16Control/label series, background-suppressed, complete

The design is candid about the tradeoff. Inference buys coverage on real-world files at the cost of brittleness on unusual protocols. It is a heuristic, and the docs call it the critical first step rather than a solved problem.

The Absent Index

The brain toolbox has a quality index, QEI, published by Dolui and colleagues. The kidney toolbox does not, because no renal quality index exists. Rather than invent one, the design keeps a placeholder module, K1, in the registry and marks it N/A forever.

Keeping the empty slot preserves uniform module numbering across organs without faking a metric. The dataset survey adds three negative findings: no open renal ASL QC implementation exists, no reference pipeline supports non-brain organs, and no public placental perfusion data exists. Those were verified with the authenticated GitHub CLI rather than web search, which is an unusual level of rigor for a negative result.

OrganChecks implementedQuality index presentPublic perfusion data
Brain20Yes (QEI, Dolui et al.)Yes
Kidney19No (K1 permanently N/A)Yes (renaldro, Brumer)
Placenta15NoNo (synthetic phantoms only)

A Layer, Not a Pipeline

OSIPI-DOCS does not compete with the tools that build CBF maps. It grades their output, and it says so in the design.

ProjectPrimary roleRelationship to OSIPI-DOCS
OSIPI-DOCSGrades ASL scans and mapsSubject project
ASLPrepASL processing workflowUpstream; its output is graded
ExploreASLASL processing ecosystemUpstream; adjacent implementation
FSL BASILASL analysis within FSLUpstream; adjacent analysis software
WSJ hedcut-style black ink portrait of Agnik Misra, the Google Summer of Code contributor behind OSIPI-DOCS.

The project was built by Agnik Misra (Jitmisra) as a Google Summer of Code project under OSIPI, mentored by María Mora Álvarez and Sudipto Dolui. The final report counts 54 checks (20 brain, 19 kidney, 15 placenta) and 479 known-answer tests across 28 files. The tool is a documentation-first project with an unusually small dependency surface, described in the report as NumPy and nibabel only, with "no scipy, no matplotlib and no web framework" in the core package.