The Paranoid Front Door: Inside duplicati/console-ingress

The Nightmare of Silent Failures

Managing a distributed fleet of backup nodes is an exercise in paranoia. If a backup fails, the node must report that failure to a central server. But if the central server drops that telemetry report due to a database timeout or a transient network error, the IT team is flying blind. A silent failure is infinitely worse than a loud one. The duplicati/console-ingress repository was built to solve exactly this problem.

This specialized .NET 9 microservice acts as the high-performance front door for the Duplicati portal. It operates on a single architectural mandate: once a backup report reaches the ingress server, it must survive.

Encrypting at the Edge

The ingress pipeline treats every incoming payload as hostile. Before a report is ever written to disk or a database, it passes through a strict gauntlet. The IngressHandler enforces a strict 2MB size limit and validates the JSON structure.

Crucially, the payload is immediately encrypted using SharpAESCrypt while still in memory. This ensures the ingress server acts as a true zero-trust gateway. Data at rest is protected by keys managed directly by the ingress server, completely decoupling security from the underlying storage mechanism.

if (request.ContentLength > MaxPayloadSize)
{
    throw new UserReportedException(StatusCodes.Status413PayloadTooLarge, "Payload too large");
}

// Encrypt the stream before handing it off to storage
using var encryptedStream = new MemoryStream();
using (var crypt = new SharpAESCrypt.SharpAESCrypt(password, encryptedStream, SharpAESCrypt.OperationMode.Encrypt))
{
    await request.Body.CopyToAsync(crypt, ct);
}

The Double-Buffered Survival Mechanism

The architectural crown jewel of console-ingress is its fallback mechanism. The server does not write directly to a relational database. Instead, it uses a custom abstraction called KVPSButter to write the encrypted file to whatever primary storage is configured (local filesystem, S3, or PostgreSQL). Once written, it pings a MassTransit message bus to notify downstream workers.

But storage systems fail. When KVPSButter throws an exception, the system catches it. Instead of returning a 500 error to the client and dropping the data, the FailedUploadConsumer pattern kicks in. The server takes the entire raw, encrypted byte array and shoves it directly into the MassTransit message bus as a fallback envelope. This double-buffered approach prioritizes pure data survival over architectural neatness.

Time-Ordered Persistence

When storing thousands of simultaneous reports, the naming convention matters. The ingress server generates filenames using UUID v7. Unlike standard UUID v4, which is entirely random and fragments database indexes, UUID v7 is time-ordered. This subtle choice ensures that files created around the same time are stored near each other on disk, drastically improving storage locality and database insertion speeds.

The Centralization Pivot

To understand why Duplicati built this paranoid ingestion engine, you have to look at the broader remote backup market. Tools like Restic and BorgBackup dominate the command-line landscape. They are decentralized, fast, and leave reporting entirely up to the user. If you want centralized monitoring with Restic, you have to write your own wrapper scripts.

The console-ingress service represents Duplicati's pivot toward IT teams. By providing a hardened, fail-safe endpoint for telemetry, they are bridging the gap between open-source flexibility and enterprise reliability.