HAProxy + keepalived for RADIUS and billing: the topology that stops 3 a.m. outages

The 3 a.m. outage in an ISP platform is rarely dramatic. A node reboots for patching and nobody realises it held the RADIUS address. A disk fills on the billing server and the payment callbacks start failing silently. A database node falls out of sync and the one application server that happened to be talking to it starts returning stale balances. Subscribers do not care which of these it was. Their session dropped and reconnect failed.

Across the IconRadius deployments I have supported, the topology that has removed most of these calls is the same one: a pair of front-end nodes running keepalived and HAProxy, with the roles split by protocol. The detail that matters, and that I see missed most often, is that RADIUS and billing need different kinds of failover.

Why RADIUS and billing cannot share one trick

HAProxy balances TCP and HTTP. RADIUS authentication and accounting run over UDP. HAProxy does not forward UDP, so it cannot be the thing that keeps RADIUS available.

keepalived, on the other hand, does not care about protocols at all. It moves a virtual IP (VIP) between nodes using VRRP. Whatever listens on that address on the node that currently holds it gets the traffic.

So the split is:

keepalived owns the RADIUS VIP. The BRAS or NAS points at the VIP. The RADIUS service runs on both front-end nodes, and the node holding the VIP answers.
HAProxy owns the HTTP and TCP traffic. Billing portal, APIs, payment-gateway callbacks and database connections go through HAProxy, which spreads them across healthy backends and drops the unhealthy ones.
keepalived also owns the HAProxy VIP. If an entire front-end node dies, the HTTP address moves too.

Two nodes, two or three VIPs, one clear owner for each kind of failure.

The topology

Front-end pair. Two nodes, each running keepalived, HAProxy and the RADIUS service. Identical configuration, deployed from the same source.
VIP for RADIUS (UDP 1812 and 1813). Active on one node. The NAS lists it as its primary RADIUS server. Optionally, a second VIP with the opposite priority gives the NAS a secondary server that is normally on the other node.
VIP for HTTP and APIs. Active on one node, with HAProxy in HTTP mode behind it, health-checking the application servers on a real endpoint that touches the database, not just a static page.
Database access through HAProxy in TCP mode. For Galera, route writes to a single node and keep the others as backups so you avoid write conflicts across nodes. Health checks must detect a node that is up but not synced; a plain port check is not enough.
Accounting and analytics stores on their own path. Where ClickHouse or Cassandra hold accounting history and reporting, they get their own backends with their own checks, so a slow report never blocks an authentication.

keepalived: the settings that decide whether failover happens

Most keepalived outages are configuration, not software. The checklist I apply:

Unicast peers. Many virtualised and cloud networks drop multicast. If VRRP advertisements never arrive, both nodes believe they are master and both hold the VIP. Configure unicast peers explicitly.
Authentication and distinct router IDs per VIP, so a stray VRRP instance elsewhere on the network cannot interfere.
Track scripts that test the service, not the process. Check that RADIUS actually answers (a Status-Server request or a test authentication against a local test account), and that HAProxy is healthy. A running process that cannot answer should lose the VIP.
Weight reductions, not hard failure. Let a failed check reduce priority so the healthier node wins, rather than stopping keepalived.
Preempt delay. When the preferred node comes back, give it time to warm up before it takes the VIP back. Flapping is worse than running on the backup for a while.
Notify scripts. On master and backup transitions, log the event and send an alert. A silent failover is a future outage you do not know about yet.
Gratuitous ARP after a transition, so the switches learn the new owner immediately.

HAProxy: the settings that decide whether users notice

Real health checks. For the billing backends, hit an endpoint that performs a lightweight database read. For Galera, use a check that reports cluster state, and mark anything other than synced as down.
Timeouts that match reality. Payment callbacks and report exports behave differently from a login request. Set connect, client and server timeouts per backend rather than globally.
Connection limits. Set maxconn with the billing cycle in mind. The peak is at renewal time, not on an average Tuesday afternoon.
Redispatch and retries. If a backend fails mid-connection, let HAProxy try another one instead of returning an error.
A backup server definition for the database writer, so writes move to a named second node when the first fails, rather than scattering across the cluster.
Stats and logs on. The stats page and structured logs are how you learn, after the fact, which backend failed and for how long.

RADIUS-specific details

NAS timeouts and retries. The NAS retransmits authentication requests that go unanswered during a VIP move. Set its timeout and retry count so a few seconds of failover is absorbed without the subscriber seeing a rejection.
Accounting during failover. Accounting packets lost in the switch are retransmitted by the NAS. Interim updates then bring session state back in line. Accept that a few seconds of accounting is reconstructed, and make sure the billing logic tolerates it.
Duplicate detection. After a failover both nodes may briefly see the same request. Make sure the RADIUS service and the session store handle duplicates as duplicates.
Shared session state. Both nodes must read and write the same session store, or the backup will reject users the master knew about.

Database layer

Galera: one writer at a time. Multi-writer looks attractive and produces deadlocks and certification failures under load. Route writes to one node; read from the others if the application can tolerate it.
Quorum. Three nodes, or two plus an arbitrator. A two-node cluster that loses one node loses quorum and stops accepting writes, which is the outage you were trying to avoid.
ClickHouse and Cassandra for the heavy writes. Accounting history and analytics belong in stores built for append-heavy workloads, behind their own backends, so a reporting spike never becomes an authentication problem.
Backups tested by restoring them, on a schedule, to a node that is not in the cluster.

Drill it before it drills you

A topology that has never failed over on purpose will fail over for the first time at 3 a.m. Schedule these in a maintenance window and write down what happened:

Pull the power (or stop the VM) on the master. Did the VIPs move? How long until the NAS got an answer?
Kill the RADIUS process only. Did the track script notice, and did keepalived move the VIP without the node going down?
Stop HAProxy only. Same question.
Block VRRP between the nodes. Did you get a split brain? Fix the config until you do not.
Fill the disk on one application server. Did the health check mark it down, or did it keep serving errors?
Desync one Galera node. Did HAProxy stop sending it traffic?
Let the master come back. Did it preempt cleanly, or flap?

Repeat the drills after every significant change. The configuration that worked last year has had a year of edits.

What this buys you

Not zero incidents. Fewer of them, shorter ones, and the ones that remain are visible: a notify alert at 3 a.m. that says "failover to node two completed", which you read in the morning, instead of a call from the night shift that says "nothing works". That difference is what the topology is for.

If your current setup has one node quietly holding everything, or a cluster that has never been failed over deliberately, an architecture review that calls out the single points of failure is where I would start.