Build an Anycast CDN

Start with the global setup: two fresh Debian 12 servers with systemd, BIRD 2, NGINX, and customer BGP; one authorized IPv4 /24 and an agreed origin ASN; an HTTPS origin on a separate IP; and a domain whose DNS you control. The examples assume direct BGP peers. Your upstream must supply the actual peering values.

Budget for address space and ASN arrangements, both edge servers, outbound traffic, origin traffic, DNS health checks, and monitoring. Get current quotes and confirm BGP eligibility before ordering. The addresses below are documentation examples; replace them with your own. Deployment and failure timings must be measured on your network.

Agree the origin ASN with your upstreams. If you need your own ASN, apply through your regional registry or a sponsoring provider under that registry’s eligibility rules. The ASN identifies the network originating your prefix; it is separate from the address lease or transfer.

Have the resource holder authorize that ASN in RPKI: create a ROA for the /24 with maximum length /24. Add the matching IRR route object where your upstream requires it, and supply a letter of authorization if requested. A ROA authorizes routing; it does not announce the route for you.

Send this request to both hosting providers. Do not start the server configuration until they confirm the prefix is accepted and provide the peer settings.

Run this on Paris. The /32 binds the CDN address locally; the covering blackhole route drops packets for unused addresses in the /24. The systemd unit below restores both after a reboot, before BIRD and NGINX start.

Allow HTTPS to the CDN IP and BGP TCP port 179 from your provider’s peer. Keep management access on the unicast address. This reverse-proxy setup does not need Linux packet forwarding.

Save this unit with your real service IP and prefix. It adds a dedicated dummy interface and retains the server’s existing unicast configuration.

Add the dependency to both services, start the address unit, and check the origin route. It should use the unicast network, not anycast0.

Add provider-required authentication or multihop settings before starting. Expect Established with no exported CDN prefix yet. If the session stays down, check the peer address, ASN, firewall, and authentication with your upstream.

After the NGINX configuration below passes nginx -t, install this deploy hook and check the renewal timer on each edge. Independent certificates can serve the same hostname; they do not need to share a private key. As the fleet grows, centralize issuance and secure distribution to reduce DNS credential exposure and coordinate renewals.

For an NGINX origin, add these locations inside its existing HTTPS server for origin.example.com. They provide a cacheable asset and an uncached probe without changing your application. Use the origin’s own valid certificate and confirm its access log records both paths.

The test asset stays fresh in a shared cache for 300 seconds because of s-maxage. inactive=60m controls eviction of unused objects, not freshness. The key separates hostnames and query strings. Requests with cookies or authorization bypass the cache; NGINX also respects private, no-store, Set-Cookie, and Vary response handling.

Each PoP has its own disk cache. Use versioned asset URLs for releases. This configuration provides no global purge API and does not force stale content to be served when the origin fails.

After the checks pass, enable cdn_prefix and ask the upstream to confirm the /24 is accepted and propagated. Set your hostname’s A record to the CDN IP. Use DNS-only if your DNS provider also offers a CDN proxy. Test HTTPS from an external network before continuing.

Leave DNS unchanged. Probe the public hostname from several networks and inspect X-Edge-Id. To check a specific PoP, run curl --resolve on that server itself; using the Anycast IP from your laptop cannot force Paris or New York.

From an external network currently reaching Paris, keep making new HTTPS connections. Stop the health controller before manually disabling the route so it cannot re-enable it. Verify that successful requests eventually identify New York. Record failures and elapsed time; existing connections may need to reconnect.

With the controller running and the route advertised, stop NGINX on Paris. Confirm that the controller withdraws cdn_prefix and external requests move to New York. Start NGINX again and verify that the recovery delay prevents immediate re-advertisement. Repeat on the other edge.

Reboot one edge while the other serves traffic. Verify the dummy address, BIRD session, certificate, cache, and controller recover in order. Measure both warm-cache and cold-cache behavior at the survivor. Record the observed results with the software versions and probe networks; do not assume a fixed BGP convergence time.

In a GeoDNS service such as Route 53, create geolocation A records with the same name, cdn.example.com, and distinct record identifiers. Set Europe to EU_CDN_IP, North America to US_CDN_IP, and Default to GLOBAL_CDN_IP. Those values are placeholders for your actual addresses. Start with a 60-second TTL and measure resolver behavior.

If you move authoritative DNS from Cloudflare to Route 53, update certificate issuance to the Route 53 DNS plugin as well, or deliberately delegate the ACME challenge zone. The earlier Cloudflare plugin requires control of the authoritative challenge records.

Associate each record with the health of its pool. In Route 53, an unhealthy geographic match can fall back to a broader geographic record, then the default record. Keep the global fallback healthy and large enough to absorb regional traffic. If you colocate global and regional pools, configure and monitor their prefixes separately; the example controller manages one prefix per instance.

Combine checks of individual edges through their unicast paths with external probes of the regional service IP. A probe to an Anycast IP can keep succeeding after one edge fails because it reaches another edge. Remove a pool from DNS when the pool cannot serve traffic, rather than whenever one member fails.

First withdraw one edge: the regional IP should still work through another edge in that pool. Then make the entire pool unavailable in a controlled test: new DNS answers should select the fallback. Test both fresh lookups and clients retaining the old answer. DNS changes cannot move an existing connection or immediately replace every cached answer.

GeoDNS estimates location from the recursive resolver or an EDNS Client Subnet hint. Test from several real networks, including public resolvers. Do not treat a default record as a guaranteed fail-safe: Route 53 can return unhealthy records when all eligible choices fail health checks.

You can also mix globally announced nodes with nodes whose routes an upstream exports only to selected networks. Configure this with that provider’s documented BGP communities and verify the scope externally. NO_EXPORT refers to AS boundaries, not continents. This is a separate routing policy from GeoDNS selecting regional pools.

If a hosting provider cannot peer with BGP, a transit provider can announce your prefix and deliver traffic over tunnels. Check return routing and MTU, and put a cache at each receiving location. Sending every tunnel back to one distant cache keeps that cache and its links on every request path.

You are ready to add traffic when both PoPs serve valid HTTPS, each proves MISS then HIT, private responses remain uncached, and withdrawing either route moves new requests to the surviving location. Start with public static assets and measure the origin load before adding more cacheable paths.