CatchMe: The Open-Source Memory Tree for Local AI Agents
It records desktop activity as events, closes them into hierarchical summaries, and gives agents a way to browse your day without a cloud vector index.
- CatchMe treats personal memory as a browsable hierarchy, which fits temporal questions better than semantic search alone.
- The system closes raw desktop events into Day, Session, App, Location, and Action branches before summarizing them locally.
- SQLite and FTS5 keep the recorder lightweight, portable, and fully local instead of pushing the problem into a vector stack.
- CatchMe is a memory substrate for CLI agents, not a generic screen recorder or a time tracker.
Your desktop already has a memory shape
CatchMe starts from a simple but unusual claim: the most useful way to remember a work session is not to search it, but to browse it. If you ask, What was I doing two hours ago? the answer usually depends on sequence, app context, and a few sharp boundaries, not a fuzzy semantic match.
That is why CatchMe feels less like a recorder and more like an archive builder. It takes the stream of windows, keystrokes, clipboard moments, and screenshots that your desktop already produces, then turns that stream into something with shape.
CatchMe's contrarian bet: structure over embeddings
Most personal memory systems reach for vector embeddings first. CatchMe takes the opposite route. It assumes that for a lot of desktop recall, the important signal is not conceptual similarity. It is chronology, app transitions, and the fact that one activity closed before the next one opened.
| Dimension | CatchMe | Vector screen recorders | Time trackers |
|---|---|---|---|
| Retrieval model | Tree-based browsing across Day -> Session -> App -> Location -> Action | Semantic similarity over embeddings | Duration and app names |
| Best at | Reconstructing what happened in sequence | Finding conceptually similar fragments | Measuring where time went |
| Trade-off | Depends on good boundary detection and summaries | Needs heavier indexing and storage | Knows activity, not context |
That is the interesting part. CatchMe is not trying to win a search benchmark. It is trying to make recall feel navigable, like opening the right branch in a file tree instead of asking a model to guess from a pile of fragments.
How a flat stream becomes a memory tree
The pipeline is the project’s core trick. Raw events arrive first. Then boundary detection decides when a session, app, or location should close. Once a branch is sealed, CatchMe can summarize it in the background and attach that summary to the tree.
The result is not just stored history. It is a hierarchy that can answer questions at different altitudes. You can skim the day, drill into a session, or open a specific app branch when you need the exact moment a decision happened.
What the engine is doing under the hood
The implementation stays practical by separating fast capture from slower writes. Recorders push events into a queue. A background loop drains that queue and writes to SQLite, which keeps the recorder responsive even when the desktop is noisy.
while running:
event = event_queue.get()
store.write_raw(event)
if is_boundary(event):
closed_branch = tree.close_current_branch()
summary_queue.put(closed_branch)
if summary_queue_ready:
store.write_summary(summary)SQLite matters here because it gives CatchMe a durable local store without introducing a second database system just for retrieval. FTS5 adds text search on top, so the recorder can answer targeted queries without forcing every memory lookup through embeddings.
Why local-first is not just a privacy slogan
CatchMe is built for agents that live on the user’s machine. The repository positions it as a skill for CLI tools such as OpenClaw, NanoBot, Claude Code, and Cursor, which means the agent can ask for memory, but the memory stays local.
That matters because the project is not only about privacy. It is about control. If the memory layer stays on-device, the user keeps the costs, latency, and data boundaries predictable. CatchMe can pair with local model stacks like Ollama, vLLM, or LM Studio without turning desktop history into a cloud service.
What CatchMe replaces, and what it is really for
CatchMe is not a generic screen recorder, and it is not a time tracker wearing an AI label. It is also not a full autonomous agent. Its job is narrower and more interesting: provide a structured memory substrate that helps agents and power users reconstruct work from the inside out.
That is why the project stands apart from both continuous recorders and lightweight activity logs. Continuous recorders collect too much and still struggle to explain what mattered. Time trackers count motion without context. CatchMe sits in the gap, turning local desktop life into a memory tree that can be summarized, browsed, and queried when the question is specific enough to deserve an answer.
