3D Model Anything Won't Scan Your Room Into a 3D Model

Wes Bos built an iPhone app that records LiDAR, hundreds of posed photos, and depth maps, then hands the reconstruction to a coding agent that checks every render against the originals.

9 min read • View on GitHub • More from wesbos

An editorial illustration of a living room drawn half as solid hatched walls and furniture and half as an open wireframe on a faint floor grid. A single person stands mid-room holding a phone aloft, surrounded by a dozen small ghosted photo frames that mark where pictures were taken, each connected by a thin ruled line back to the phone's path. It establishes the whole premise: a room, a phone, and captured viewpoints raining down as evidence.
The capture is not a model. It is a room full of viewpoints, each one a place the camera actually stood.

I have discovered the the BEST way to get 3D modeling results from LLMs The app LiDAR scans the room while taking hundreds of high quality photos. Then a workflow + skill I've created models out every item recursively checking the model against the photo at the exact spot + angle where the photo was captured The result is a a dimensionally accurate and high quality model of your room All in my latest video on how to do it https://t.co/6fRpe3hd9b

Wes Bos, Creator · Wes Bos on X
Key Takeaways

The agent renders the room from a camera position the phone recorded hours earlier. Then it sets that render next to the original photograph and looks for the mismatch. It is a small move, and it reframes what a scan is for.

The Render Meets the Photo

Most 3D scanning tools ask you to sweep a phone around a room and wait for a finished mesh. 3D Model Anything does something stranger. It captures the room as a pile of evidence, then hands the reconstruction to a coding agent that keeps checking its own work.

The check is the point. The iPhone app saves a full camera pose for every photograph: position, orientation, focal length, principal point, image size. Hours later, the agent renders the room it is building from that exact pose and drops the result next to the original shot. Where the two disagree, the model is wrong.

This is a feedback loop, not a one-shot conversion. The scan is not the deliverable. The scan is how the deliverable gets checked.

The loop that makes the workflow work: the phone records where each photo was taken, the agent renders its model from those same poses, and the mismatch between render and photo tells it what to fix.

A black ink hedcut-style editorial portrait of Wes Bos, the creator of 3D Model Anything, rendered with stipple dots for skin texture and fine hatching lines for hair and clothing on a pure white background.

A Scanner That Refuses to Finish the Job

The README is blunt about the design intent.

The app records the evidence. The skill guides an agent through measurement, modeling and matched-camera visual comparisons. This is an iterative reconstruction workflow, not a one-click scan-to-perfect-model converter.

Project documentation, Repository README · wesbos/3d-model-anything README

That refusal is deliberate. LiDAR gives you structure and rough furniture boxes. Photos give you appearance and detail. Neither one alone produces a scene you can open, inspect, and change. So the project splits the job: the phone gathers evidence, and the agent builds an editable Three.js scene from it.

The deliverable is code, not a frozen mesh. You can select an object, move a wall, swap a material, or rerun the whole thing with a better prompt. A photogrammetry mesh gives you a surface you can look at. This gives you a room you can edit.

What the Phone Actually Records

Everything downstream depends on what the capture app writes to disk. The export is a bundle called house-capture/5, and it is built to be re-read, not just replayed.

The standout detail is that raw data sits next to derived data. The app stores CapturedRoomData so RoomBuilder can be run again later with better settings. The phone does not lock in its first interpretation of the room. It records what it saw and defers the judgment.

The coverage engine is the same philosophy applied to the live capture. It scores every vertex of the LiDAR mesh by how well the photos taken so far show it: close enough for detail, roughly head-on, sharp, and not occluded, checked against each photo's depth map. The ring that shows this on screen is labeled "Photo detail," not "Scanned," because it measures photo quality on observed geometry rather than how much of the room you have walked.

The Agent's Loop

The reconstruction lives in a skill called scan-to-threejs. A skill is a set of instructions and helper scripts that tells a coding agent how to do a specific job, and this one is unusually disciplined about evidence.

The agent measures the scan, models the room as Three.js objects, renders from the recorded camera poses, compares those renders against the photos, and iterates. The skill ships about twenty Python, shell, and Node scripts to support the work, including a bundle reader, an orthographic projector, and a photo-comparison tool.

An editorial illustration split into two halves: on the left a photographic frame of a room corner with a chair and a drawer unit, on the right the same corner drawn as a hatched line illustration from the identical angle. A human hand and a mechanical hand both point at one discrepancy between the halves, a drawer that is present in the render but shaped slightly wrong, and a single taut thread links the two frames at that spot. It captures the exact moment the workflow is built around: does the render match what the camera actually saw from this position.
The payoff moment. The render and the photo are lined up from the same recorded pose, and the difference between them is the next thing to fix.

Use captured geometry as measurement evidence and posed photos as visual evidence. Build editable geometry, then compare it against those inputs. Conversion produces a measured baseline; detailed reconstruction requires modeling and visual review.

Project documentation, scan-to-threejs skill · scan-to-threejs SKILL.md

The loop is honest about its own softness. There is no objective similarity score. The comparison tools stitch images together; they do not grade the match. The agent, and the person looking over its shoulder, decide what counts as close enough, fix the largest discrepancy, and go around again.

The skill also tells the agent to record uncertainty rather than invent measurements, and to measure the big dimensions first. That is a quiet admission that the system is a collaborator, not an oracle.

Where It Fits

The obvious comparison is to the one-click scanners people already use. The difference is not that 3D Model Anything captures more. It is what happens after the capture, and who does the work.

ApproachWhat you getWho reconstructsHow accuracy is checkedOpenness
3D Model AnythingEditable Three.js scene codeA coding agent, guided by a skillRender vs. original photo from the same recorded poseMIT code, separate asset license
Apple RoomPlanParametric room plan and recognized objectsApple frameworkFramework output, not photo-matchedClosed framework
PolycamScan assets and meshesCloud processingNot pose-matchedClosed product
MatterportHosted digital twinHosted platformNot pose-matchedClosed platform
NerfstudioRadiance field and view synthesisYou train a modelLearned, not measured geometryOpen, research-oriented

Only one row checks its work against the original photograph from the exact camera position that took it. That is the column that matters.

The Honest Limits

The project is candid about what it does not do. Fine details can exceed what the scan resolves, so a small object or a tight joint may come out approximate. Photo comparison is human and agent judgment rather than a metric, which means quality depends on how hard the operator pushes.

The shipped office demo also omits its original scan and photo assets. That disables the demo's Scan and Overlay modes. You can explore the reconstruction and the browser playground, but not the evidence that produced them. You can use those modes with your own captures.

And the iPhone app is source-available, not a download. Building it needs Xcode, and an App Store release should not be assumed. The code and skill are MIT-licensed, while the supplied office assets carry their own terms.