Upstreams and failover

Scope: naming a pool of backends once and proxying to the pool — how nginx spreads requests, and what actually happens when a member dies. The death is captured, not described.

The pool

http {
    upstream labpool {
        server 127.0.0.1:9001;
        server 127.0.0.1:9002;
    }
}
...
location /pool/ { proxy_pass http://labpool/; }

proxy_pass to a name instead of an address is the whole change. With no other instruction, nginx distributes requests round-robin. The lab's backends announce their own port, so the rotation is visible:

$ for i in 1 2 3 4; do curl -s http://127.0.0.1:8080/pool/ | head -1; done
backend port: 9001
backend port: 9002
backend port: 9001
backend port: 9002

(Weighted round-robin via server ... weight=3, and other balancers like least_conn, exist; round-robin is the default and the mental model the rest build on.)

Killing a member

Stop the 9002 backend and ask again:

$ for i in 1 2 3 4; do curl -s http://127.0.0.1:8080/pool/ | head -1; done
backend port: 9001
backend port: 9001
backend port: 9001
backend port: 9001

No visitor saw an error. The error log shows the cost, paid once by whoever's request drew the dead member:

[error] ... connect() failed (111: Connection refused) while connecting to upstream, ... request: "GET /pool/ HTTP/1.1", upstream: "http://127.0.0.1:9002/", ...
[warn] ... upstream server temporarily disabled while connecting to upstream, ...

nginx marks the dead member temporarily disabled and stops sending it requests for a while (the defaults are one failure — max_fails=1 — disabling it for fail_timeout=10s; both are per-server tunables). During that window all traffic goes to the survivor. If the dead member returns, it rejoins on the next attempt after the timeout. If the survivor also dies, requests start returning 502: the pool has nobody left to fail over to.

Two honest limits of the open-source build:

• These are passive checks. nginx learns a backend is dead by failing a real request against it (the connect() failed line above is exactly that). It does not probe members on its own schedule; active health checks are a NGINX Plus feature. One visitor per outage pays the discovery cost.
• Failover retries are for safe requests. By default nginx will re-send a request to the next member when the failure happened before the backend started answering — right for GETs. Whether a POST should be retried is a decision about your app, not a default to lean on; proxy_next_upstream is where that decision is written down.

Check: run the pool, kill one member, and confirm all three of the story's parts in your own lab: visitor answers uninterrupted, one connect() failed line, one temporarily disabled line. If the middle line repeats for every request, fail_timeout isn't doing what you think — and now you know where to look.