The Egress Firewall: Inside duplicati/update-server

The Open-Source Egress Tax

Success in open-source software brings a hidden, often fatal cost. When a popular tool releases an update, thousands of clients automatically fetch a heavy binary from primary storage. Without a smart intermediary, the project goes bankrupt paying for outbound data transfer.

Duplicati faced this exact existential threat. Instead of passively paying a commercial CDN or exposing their primary S3 bucket, the team built a bespoke caching layer. The result is a masterclass in defensive engineering.

The Reactive Cache: Serving the Incomplete

Standard proxies either wait for a full download before serving or pipe data directly without caching. Duplicati's proxy does both. It uses a custom ASP.NET Core implementation to serve a file to users while it is still downloading.

This reactive streaming happens inside a custom file provider. When a user requests a file currently being downloaded, a loop creates a seamless experience. Subsequent users simply catch up to the in-progress stream without triggering a new backend request.

while (m_local.Position >= m_item.AvailableLength && !m_item.IsComplete)
{
    // Wait for more data to be written to the local cache disk
    await m_item.NextAvailable;
}
// Read the newly available chunk and pipe it to the client
int bytesRead = await m_local.ReadAsync(buffer, offset, count, cancellationToken);

Surviving the Thundering Herd

On release day, auto-updating clients spam requests for a version that hasn't fully propagated. This thundering herd can rack up massive transactional costs on backend APIs. The update-server defends against this using aggressive Smart 404 caching.

By caching negative responses locally, the server absorbs the impact of eager clients. A background loop periodically checks the primary storage to clear the 404 status only when the file is actually available.

A Twelve-Factor .NET Microservice

The application shuns complex configuration files in favor of a pure Docker citizen approach. Environment variables map directly to a strongly-typed configuration record. Utilizing ASP.NET Core Minimal APIs keeps the footprint exceptionally lean.

Why Not Just Use Cloudflare?

Putting an S3 bucket behind a standard commercial CDN edge seems easier. However, commercial CDNs place strict bandwidth limits on serving large, non-HTML binary files. Building a bespoke proxy allows Duplicati precise control over cache eviction, manifest handling, and egress costs.