synesthetic_loop: A Generative Telephone Game With Guardrails

A photo becomes a five-color palette, the palette becomes a pentatonic score, and the score becomes a new image. The interesting part is not the hallucination, but the constraints that keep the loop readable.

9 min read • View on GitHub • More from rt-wang

A framed image enters a hand-built machine and emerges on the far side as a second framed image after being reduced to five swatches and a score. It explains the repo's core trick: each translation stage preserves enough structure to stay legible, while still allowing the result to drift. — The loop turns one medium into another, then back again, with loss and structure both built in.

Key Takeaways

synesthetic_loop treats translation loss as a feature by compressing an image into a tiny palette before it ever becomes music.
The repo stays readable because each stage narrows interpretation, from K-means centroids to pentatonic notes to vibe-driven image prompts.
The music layer is not decoration, because arrangement profiles, synth choice, and humanization give the loop a stable personality before Gemini enters.
The strongest engineering choice is the guardrails, which let the system drift aesthetically without turning the public-facing loop into a free-for-all.

The picture does not stay a picture

synesthetic_loop starts with a simple provocation: a photo does not stay a photo. It is compressed into five colors, translated into a pentatonic score, then read back by Gemini as a mood that becomes a new image. The loop matters because it does not promise fidelity. It promises controlled drift.

Synesthetic Computation is the practice of modeling those maps, linking them, and letting them run. It’s less about hallucinating connections and more about instrumenting perception itself.

Agam Rafaeli, Author · Agam Rafaeli on Medium

That makes the repo feel less like a gimmick and more like an experiment in translation. rt-wang split the system into distinct stages, from clustering to music to AI orchestration, and the separation shows. The code reads like someone designing a machine that can wander without getting lost.

Why the loop stays legible

The system is a loop of translation, not a straight line from input to output.

Most cross-modal demos fail because they move too much information too fast. Here, each stage throws most of the input away on purpose, then rebuilds a narrower version of the signal. That is why the output feels interpretable instead of noisy.

The important move is compression. Image data is downsampled, clustered with K-means++, and reduced to five centroids. Those centroids are then sorted by luminance, using the standard brightness formula, so darker colors consistently map lower and brighter colors map higher. That one choice keeps the loop from becoming arbitrary.

From five colors to five notes

Five swatches are sorted from darkest to brightest and lined up against piano keys. It shows the key mapping step, where luminance becomes register instead of arbitrary noise. — Brightness sorting keeps the palette-to-note mapping stable.

In practice, the mapping is simple enough to explain in one glance. Downsampling keeps the math fast, K-means++ gives the palette a sensible starting point, and the brightness sort gives the output a reliable order. A short version of the bridge looks like this:

const brightness = 0.299 * r + 0.587 * g + 0.114 * b;
const palette = runKMeans(image, 5).sort((a, b) => brightness(a) - brightness(b));
const notes = palette.map((color, i) => pentatonicScale[i]);

That is the central constraint. The project does not try to preserve every color or every contour. It reduces the image until the surviving structure can be turned into a stable musical register, then lets the music carry the next translation.

The synth engine gives the loop a mood

A synth bench turns the same musical spine into different textures, like glass, smoke, and velvet, using knobs, filters, and oscillator shapes. It explains why the music feels performed rather than mechanically replayed. — Arrangement profiles shape the loop's emotional temperature.

Once the palette becomes notes, music.js gives the loop a personality. deriveArrangementProfile reads saturation and brightness and chooses a style such as glass, smoke, mist, or velvet, then pushes oscillator choice, filter cutoff, attack, and release toward that mood. The same harmonic spine can feel airy, dense, sharp, or foggy depending on those parameters.

This is where the repo stops being a pure mapping exercise. humanize adds micro-timing jitter, and reharmonizeVoicing reshapes the chord voicing so the score feels performed instead of printed. The result is not random, it is only slightly imperfect in the right places.

Gemini enters as an interpreter, not a shortcut

The AI stage does not get raw pixels. It receives the music's metadata and a constrained interpretation prompt that steers Gemini toward scene, palette, motion, and mood, while forbidding technical terms like spectrogram or waveform. That keeps the model from cheating by describing the medium instead of the experience.

Variation matters too. buildVariationHint adds small nudges so the loop does not fall into the same visual trope every time, and the server-side key path is guarded so the public loop cannot drain the owner's quota. The repo is willing to be playful, but not careless.

What this repo does better than adjacent tools

Approach	Input	Output	What it optimizes for
Traditional sonification	Data or pixels	Sound	Fidelity and measurable mapping, often at the cost of texture
Direct image generation	Prompt or image	Image	Speed and style, with little intermediate structure
synesthetic_loop	Image, palette, and music metadata	New image after a musical detour	Legible drift, where each translation stage adds constraint

That is the real edge. The project treats translation loss as expressive material, but it keeps the losses bounded enough that you can still read the chain from one end to the other. The loop has personality because it has structure.

A small repo with unusually careful bones

The repo's best signal is its separation of concerns. kmeans.js handles visual reduction, music.js handles the score and its mood, and the AI layer handles interpretation and regeneration. Add the design docs, the minimal deployment shape, and the server-side safety guard, and you get something rarer than a demo: a tight experimental system that knows exactly where its boundaries are.

That discipline is why synesthetic_loop lands. It is not trying to prove that AI can do everything. It is showing that a carefully designed chain of reductions, mappings, and prompts can turn uncertainty into a style.