OSIPI-DOCS: The MRI Quality Checker That Refuses to Guess
A GSoC toolbox grades perfusion scans PASS, WARN, or FAIL across brain, kidney, and placenta, and documents exactly which of its 90 thresholds came from a paper, which came from someone else's code, and which came from nowhere at all.
- OSIPI-DOCS publishes its own uncertainty, admitting that 44 of its 90 quality thresholds are uncalibrated engineering defaults with no published source.
- The toolbox grades scans PASS, WARN, or FAIL but keeps UNKNOWN out of the verdict and reserves N/A for checks that structurally cannot apply.
- Every published threshold ships with a page number, a quoted sentence, and a screenshot, so each number can be audited back to its source.
- It is a layer, not a pipeline: it grades the output of ASLPrep, ExploreASL, and BASIL instead of generating CBF maps itself.
44 of 90
Most quality-control tools for medical imaging hand you a cutoff and expect you to trust it. A number appears in the config, a scan passes or fails, and nobody asks where the number came from. The OSIPI-DOCS repository does the opposite. It publishes the receipts.
The repo is a design-and-evidence archive for a Python toolbox that grades arterial spin labelling (ASL) perfusion scans. It is documentation, not the software itself; the package lives in a separate repository. What makes it worth reading is a single ratio it puts in its own report: of 90 thresholds, 12 are published, 34 come from someone else's implementation, and 44 are uncalibrated. Nearly half the numbers are engineering defaults with no source at all.
The project does not hide this. It sorts every threshold into one of three bins.
- Published: a paper states the number for this purpose, with a page and a quoted sentence.
- Implementation: a reference pipeline's code uses it, which is weaker because code choices are not peer-reviewed.
- Uncalibrated: an engineering default with no source, explicitly barred from failing a scan on its own.
That last rule is the load-bearing one. An uncalibrated threshold can raise a warning, but it cannot, by itself, fail a scan. The honesty is not a disclaimer buried in a README. It is wired into the verdict logic.
The Verdict Ladder
A scan does not simply pass or fail. Each check returns a metric, a verdict, and a one-line reason. There are five verdicts, and the two unusual ones carry the design's philosophy.
PASS means the metric sits inside the healthy band. WARN means it is borderline. FAIL means it crossed a fail cutoff. Then there is UNKNOWN, for a check that could matter but cannot be derived from the data on hand. And N/A, for a check that structurally cannot apply. A control/label swap test on a pre-subtracted GE image is N/A, because the image has no control and label volumes left to swap.
The distinction matters. Many QC tools collapse "we could not tell" into "fine." OSIPI-DOCS refuses. UNKNOWN never invents a PASS or a FAIL. It is counted in a coverage figure, so a reader can see how much of the checklist actually ran, but it does not move the overall verdict. The tool trades apparent completeness for trustworthiness, which is the right trade in a multi-site study where a silent default is more dangerous than an abstention.
| Verdict | Meaning | Effect on overall verdict |
|---|---|---|
| UNKNOWN | The check applies but the data cannot resolve it | Counted in coverage, does not move the verdict |
| N/A | The check structurally cannot apply to this scan | Not counted at all |
Every Number Has a Page Number
Provenance here is not a footnote. It is a first-class artifact. Each published threshold carries a classification, a citation, a quoted sentence, and a screenshot of the source page. The sources/ directory holds the page images, and evidence.json and highlights.json record the machine-readable version. Any threshold can be traced back to the sentence it came from.
This surfaces things most tools would rather not discuss. The design docs record an explicit divergence between ASLPrep's code and the paper behind it. ASLPrep uses a tissue threshold of 0.7 where the Dolui paper says 0.9, and a qei_c of 0.054 where the paper says 0.1. The toolbox follows the code and flags the discrepancy as an open question instead of resolving it silently. Documenting a known mismatch between a reference implementation and its own paper is rare.
The same discipline applies to derived values. A neonatal deep grey-matter band of 1.3 to 2.6 is labeled Uncalibrated and described as "derived from Miranda 2006," because the document treats a derivation as a derivation, not a published number. A QEI pass cutoff of 0.55 is listed as uncalibrated too, described only as "a small margin above 0.50."
Two Streams, One Report
Quality for ASL splits into two questions with different inputs. Stream A asks whether the scan was acquired correctly, reading the raw 4D series, the M0 image, and the BIDS metadata. Stream B asks whether the final CBF map is good, reading the map and tissue probability maps. The two streams run independently and merge into one report.
The important scoping decision is what Stream B does not do. It never generates a CBF map. That step belongs to PyASL or ASLPrep, and the design doc draws it as a grey dashed box marking the boundary between their job and ours. The toolbox is a downstream grader. It is a layer, not a pipeline.
When There's No Sidecar
Real test data did not arrive neatly. It came as bare .nii.gz files with no BIDS sidecars and no aslcontext.tsv, which means the toolbox cannot read the acquisition type off the metadata. Instead, check 8.2 infers it from the NIfTI shape, the volume count, and the filenames.
| Vendor case | NIfTI shape | Inferred acquisition type |
|---|---|---|
| GE 3D pCASL (ADNI) | 128x128x40 | Pre-subtracted difference image, no pairs, M0 present |
| Siemens 2D pCASL | 80x80x20x80 | Control/label series, no M0 |
| Siemens BS 3D pCASL | 88x88x52x16 | Control/label series, background-suppressed, complete |
The design is candid about the tradeoff. Inference buys coverage on real-world files at the cost of brittleness on unusual protocols. It is a heuristic, and the docs call it the critical first step rather than a solved problem.
The Absent Index
The brain toolbox has a quality index, QEI, published by Dolui and colleagues. The kidney toolbox does not, because no renal quality index exists. Rather than invent one, the design keeps a placeholder module, K1, in the registry and marks it N/A forever.
Keeping the empty slot preserves uniform module numbering across organs without faking a metric. The dataset survey adds three negative findings: no open renal ASL QC implementation exists, no reference pipeline supports non-brain organs, and no public placental perfusion data exists. Those were verified with the authenticated GitHub CLI rather than web search, which is an unusual level of rigor for a negative result.
| Organ | Checks implemented | Quality index present | Public perfusion data |
|---|---|---|---|
| Brain | 20 | Yes (QEI, Dolui et al.) | Yes |
| Kidney | 19 | No (K1 permanently N/A) | Yes (renaldro, Brumer) |
| Placenta | 15 | No | No (synthetic phantoms only) |
A Layer, Not a Pipeline
OSIPI-DOCS does not compete with the tools that build CBF maps. It grades their output, and it says so in the design.
| Project | Primary role | Relationship to OSIPI-DOCS |
|---|---|---|
| OSIPI-DOCS | Grades ASL scans and maps | Subject project |
| ASLPrep | ASL processing workflow | Upstream; its output is graded |
| ExploreASL | ASL processing ecosystem | Upstream; adjacent implementation |
| FSL BASIL | ASL analysis within FSL | Upstream; adjacent analysis software |
The project was built by Agnik Misra (Jitmisra) as a Google Summer of Code project under OSIPI, mentored by María Mora Álvarez and Sudipto Dolui. The final report counts 54 checks (20 brain, 19 kidney, 15 placenta) and 479 known-answer tests across 28 files. The tool is a documentation-first project with an unusually small dependency surface, described in the report as NumPy and nibabel only, with "no scipy, no matplotlib and no web framework" in the core package.





