local-image-search: The High-Performance Private Eye for AI Agents

How an MLX-powered MCP server turns Apple Silicon into a local vector warehouse, giving Claude secure access to your photo library without a single pixel leaving your machine.

7 min read • View on GitHub • More from Eventual-Inc

A magnifying glass with a mechanical camera aperture reflected in the lens, hovering over manila folders. — Software for agents, not humans: local-image-search acts as a secure lens for LLMs to view local files.

Key Takeaways

local-image-search demonstrates the shift toward headless software built exclusively as capabilities for AI agents via the Model Context Protocol (MCP).
By utilizing MLX, Daft, and LanceDB, the project achieves a throughput of ~280 images per second on Apple Silicon, turning a local folder into a high-performance vector warehouse.
The project circumvents traditional CPU-to-GPU memory bottlenecks by leveraging Apple's Unified Memory Architecture for efficient image processing.
The local-first approach to semantic image search provides a robust, privacy-preserving alternative to cloud vision APIs.

The UI is a Chatbox

The most striking aspect of local-image-search is what it lacks: a graphical user interface. There are no thumbnails, no search bars, and no settings menus. It is headless software designed entirely for a machine.

Operating as a Model Context Protocol (MCP) server, it provides a specialized capability to language models like Claude. It exposes endpoints that allow the AI agent to formulate natural language queries, translate them into vectors, and scan the local filesystem autonomously.

The MLX + Daft + Lance Trifecta

Processing image data locally at scale requires significant compute. local-image-search tackles this with a highly optimized pipeline. Daft handles distributed batching, orchestrating the flow of image data. MLX, Apple's array framework, executes the CLIP model locally, generating embeddings without the overhead of bulkier frameworks. Finally, LanceDB provides columnar vector storage, enabling instant nearest-neighbor recall without loading the entire index into RAM.

The high-speed vector pipeline: Daft orchestrates batching, MLX computes embeddings on the GPU, and LanceDB provides fast retrieval, all exposed via MCP.

Escaping the Python Tax

The performance secret lies in how the project exploits Apple Silicon's Unified Memory Architecture. Traditional machine learning pipelines in Python often choke when copying massive image batches from system RAM to discrete GPU memory.

By implementing the CLIP model natively in mlx.nn, the system bypasses these copies entirely. The GPU accesses the image data in the same memory pool as the CPU, allowing the script to process hundreds of images per second.

Two interlocking gears labeled CPU and GPU submerged in a single pool of ink. — Unified memory allows the CPU and GPU to share data without expensive copying, unlocking massive throughput.

The Incremental Edge

Scanning a massive photo library every time a query is made is inefficient. The engineering pragmatism of the project shines in its incremental indexing strategy.

The core logic checks file paths and modification times (mtime), skipping unchanged files. This allows the system to run a background refresh cycle repeatedly without redundant compute, keeping the vector database synchronized with the filesystem almost instantly.

Building the Sovereign Agent

The standard industry approach to semantic image search involves uploading personal media to cloud vision APIs. This introduces significant privacy risks and latency. local-image-search demonstrates a viable alternative for the agentic era.

Feature	Cloud Vision APIs	Generic Local Scripts (PyTorch/FAISS)	local-image-search (MLX/Lance)
Latency	High (Network dependent)	Medium (Framework overhead)	Low (~280 imgs/sec)
Privacy	None (Data uploaded)	High (Local)	High (Local)
Hardware Utilization	N/A	Poor on Apple Silicon	Native (Unified Memory)
Primary Interface	API Calls	CLI/Custom GUI	MCP (Claude)

By providing a specialized, high-performance local tool, it empowers AI coding agents to act as secure private eyes, bridging the gap between local data sovereignty and advanced language capabilities.