Anycast CDN: example health controller
====================================

Companion to https://adios.dev/blog/building-anycast-ingress-for-a-developer-cloud

Requirements: a dedicated Debian 12 edge, Python 3.10+, systemd, curl, BIRD 2,
the article's cdn-address.service, and a static BIRD protocol named cdn_prefix
with "disabled yes" in its configuration. Use only one controller per edge.

Review edge-health.py and edge-health.service before installing them. Set:

  CDN_HOST=your-real-cdn-hostname
  CDN_IP=your-real-anycast-ip

in /etc/default/edge-health. Install the Python file at
/usr/local/sbin/edge-health.py and the unit in /etc/systemd/system/.
Run systemctl daemon-reload, then systemctl enable --now edge-health.

Behavior:
- Starts withdrawn, including after a controller restart.
- Probes the local service IP with the CDN hostname and normal TLS validation.
- Withdraws after three consecutive failed local HTTPS checks.
- Requires 30 seconds of continuous health and a 60-second withdrawal
  hold-down before advertising. Checks run every five seconds plus probe time.
- Reads back BIRD protocol state after changes and on each pass.
- Exits on control errors. systemd attempts withdrawal on exit and restarts it.
- A 45-second systemd watchdog handles a stalled main loop.
- Does not withdraw merely because a shared origin is unavailable.

Scope and validation:
The policy and command handling have local automated tests. These files have
not been qualified against your provider, live BGP, or your production load.
A static local health endpoint checks TLS and NGINX responsiveness; it does
not prove the cache disk, application, origin, transit, or public reachability
is healthy. Monitor those separately and decide which failures should withdraw
this prefix. Include every service carried by the prefix in that decision.

Before production, test local service failure and recovery, certificate expiry,
controller termination and hang, BIRD restart/control failure, full reboot,
route convergence, and surviving-edge capacity. Confirm upstream graceful
restart behavior. If BIRD's control socket itself is unavailable, the cleanup
command can fail: alert on unit failures and verify that the upstream actually
withdraws. This controller is not a substitute for external monitoring.

Maintenance:
Stop edge-health BEFORE a manual birdc disable; otherwise a healthy probe can
enable the prefix again. systemctl stop edge-health attempts withdrawal through
ExecStopPost and does not trigger Restart=always. Restart the service after
maintenance; it repeats its startup hold-down before advertising.

For regional pools, set each edge's CDN_IP to its pool's service IP. The BIRD
configuration must contain only that pool's intended prefix. DNS pool health
and GeoDNS fallback are separate from this local controller.