fix: remove 403 from key failover and cooldown on 401 - #27419

When all the keys have failed with 401, we return a 502 Bad Gateway without a Retry-After. If every key is failing auth at the same time, it's almost always a real problem with the keys (revoked, rotated, or misconfigured), so a retry won't help and a Retry-After would just send clients back on a timer when nothing has changed.

We return 502 rather than the 401 because the auth failure is between the gateway and the upstream, on the centralized keys we manage on the server and the client never sees. Passing the 401 back would read as "your request was unauthorized" when the client's request was fine, and there's nothing on their end to fix. Resolving it is on the administrator, who rotates or reconfigures the keys.

A 401 used to take a key out of rotation until the server restarted, so a key that was only temporarily rejected would be blocked permanently. Now each key is retried once its cooldown expires and recovers on its own, transparent to the user and with no restart needed.

On StatusForbidden we don't cycle to next key?
I may lack knowledge what provider use StatusForbidden and StatusUnauthorized for but I would assume next key would be tried if first failed with either of those statuses.

This is what we agreed to: completely remove 403 Forbidden as a key failover status. A 403 is a per-request authorization decision (the key is valid but not permitted for this action/resource), not a key-health problem, so this status code is no longer handled, and we return the upstream error to the client. This is exactly the issue customers are seeing.

Is cooldownReason and this enum needed? If I understand correctly cooldownReason could simply be string. reasonRateLimited and reasonUnauthorized consts would be just used directly.

Good point, addressed in 74f9900

Every time key is Marked Temporary it sets reason RateLimited?

This is a bit of a hack: MarkTemporary is exported, but currently no production code needs it, all runtime marking goes through applyCooldown now with the correct reason. It's currently only used by the tests, which is why it stays exported for now. I kept it to keep this backport minimal and will unexport it (and move the tests internal) in a follow-up PR.

I'm not sure how those 2 new test cases differ from all_permanent_exhausted and mixed_states_exhausted test cases?
The difference is I assume how setup is done, by status using markNextByStatus but how key become in given state is not important form test perspective?
I feel like those should be part of test that checks pool.MarkKeyOnStatus not here?

All three validate the exhaustion Error.Kind, which is what differs between them:

1. all_permanent_exhausted: all keys permanent. Permanent is no longer a reachable state (nothing marks a key permanent anymore), but I'm keeping the status to avoid bigger changes, because this will need to be backported to previous versions.
2. mixed_states_exhausted: mix of permanent and temporary status.
3. all_unauthorized_exhausted: all temporary, all with the unauthorized reason. This is distinct from mixed_unauthorized_and_rate_limited_exhausted, where keys are all temporary but with mixed reasons, so rate-limit wins.