RPC Failover and Circuit Breaking: An Architecture Guide

Adding a second RPC endpoint does not give you redundancy. It gives you a second endpoint. Whether that translates into availability depends entirely on how failures are classified: which ones move a request to another target, which ones are final, and which ones look like failures but are not. Teams that configure a backup and assume they are covered usually discover the gap during the incident the backup was meant to prevent.

The reason is that a large share of RPC failures are not transport failures. An upstream returning execution reverted, or expressing its own rate limit inside a well-formed response, has answered your question. There is nothing to fail over to, because nothing failed. Understanding where that line sits is most of what makes a multi-endpoint setup actually work.

• Failover triggers on transport-level failures. A valid protocol-level error is a successful call with an unwelcome answer.
• More endpoints on a route does not mean more attempts. The attempt bound is separate and usually small.
• Circuit breaking protects the system from an endpoint that is failing consistently; retries protect a single request from one that failed once.
• Write operations need a stricter retry policy than reads, and no layer can infer that for you from the method name.

Failover Is a Classification Problem

Before choosing a strategy, decide what counts as a failure. Three categories behave differently and conflating them is the most common design error.

Transport failures, meaning a refused connection, a dropped response, a timeout, or an upstream rejecting your credentials, all indicate the request did not produce an answer. These are the cases where trying a different endpoint is both safe and useful.

Protocol-level errors are answers. A JSON-RPC 2.0 error object returned with HTTP 200 is the upstream telling you something definite: the transaction reverted, the parameters were wrong, the account is over its limit. Retrying against a different endpoint will produce the same answer, more slowly, at twice the cost. Aurpay’s RPC Gateway documents this behavior explicitly: such a response is returned verbatim with no retry, no failover and no health penalty against the endpoint.

Client errors are terminal. A malformed request or a protocol mismatch will fail identically everywhere. Failing over on these wastes attempts and obscures the actual bug.

The consequence worth internalizing: an upstream that is degraded but still responding, whether by serving stale data, throttling you inside a 200, or returning empty results, will not trigger failover in most gateway designs. That is a deliberate choice rather than an oversight, because the alternative requires the layer to make judgments about answer quality that it has no basis to make. It does mean your application, not your routing layer, is responsible for noticing that kind of degradation.

Two Routing Strategies, and When Each Is Wrong

Priority failover uses endpoints in a configured order, moving to the next only when the current target is filtered out, blocked by its circuit, or hits a retryable failure. Use it when you genuinely have a primary and ordered backups: a dedicated node you trust, with a commercial provider behind it.

Its weakness is that a healthy-but-slow primary keeps taking traffic. Priority order is not automatically overridden by health, so if your first endpoint is responding correctly at four times its normal latency, it stays first. You need latency-based filtering, covered below, to handle that case.

Load balancing samples candidates by relative weight and then considers observed health and latency. Use it when your endpoints are genuinely interchangeable and you want to spread load, which is also the practical way to raise an effective rate-limit ceiling across several providers.

Its weakness is subtler and worth stating: load balancing is not an availability guarantee. When candidates have been filtered down to one, that one still gets selected. A route that appears to be spreading traffic across four providers can quietly be sending everything to a single endpoint, and unless you are watching per-endpoint volume you will not know until it fails.

Filtering Happens Before Selection

Both strategies operate on a candidate set that has already been narrowed. On the Aurpay gateway a route removes endpoints that are disabled, that fall below a minimum trust level, whose observed latency exceeds the route’s limit, whose circuit is currently blocking attempts, or whose chain, network and protocol do not match the request.

The latency filter is the one that does the most work in practice, because it addresses the degraded-but-alive case that failure classification cannot. An endpoint returning correct answers slowly never produces a retryable failure. It just makes everything worse. Filtering on observed latency removes it from consideration without needing to call it broken.

There is a design detail here worth copying: an endpoint with no latency observation is not excluded merely because its latency is unknown. That sounds minor and is not. The opposite choice creates a deadlock. A freshly added or recently idle endpoint has no observations, gets excluded for having none, never receives traffic, and therefore never produces any. Defaulting unknown to admissible is what allows new capacity to enter rotation at all.

Attempt Bounds Are Not Endpoint Counts

A route can hold a large number of targets — up to 100 on the Aurpay gateway — while the number of attempts for any single request stays small. The default there is three, configurable between one and ten, with the effective value being the lower of the route setting and the platform limit.

Each attempt goes to a different endpoint, and a candidate is removed once tried, so the same endpoint is never retried within one request. Configuring ten backups does not mean a failing call will walk through all ten. Once the attempt budget is spent the request fails, regardless of how much unused capacity is sitting in the route.

Two related details determine what your latency looks like under failure. Each attempt is bounded by its own upstream timeout, not by a deadline covering the whole request, so a request that fails over twice can take up to three times the single-attempt timeout before it returns. If your caller has its own deadline, size it against that worst case rather than against the happy path.

And attempts run back to back with no delay between them. A gateway retrying internally is not applying backoff or jitter on your behalf; that remains the client’s job, as covered in handling RPC rate limits and 429 errors. Layering an aggressive client retry on top of an aggressive gateway retry multiplies rather than adds.

Retry Policy and the Write Problem

Reads and writes need different retry rules, and the distinction cannot be automated from the outside. A gateway sees a method name and a payload; it does not know whether your application can tolerate the same transaction being submitted twice.

The Aurpay gateway exposes this as two named policies. Under safe_only, only failures that clearly did not produce a business result move to another candidate: connection failures, configuration problems, and an upstream rejecting the credential. Under idempotent, timeouts, interrupted responses and selected 4xx and 5xx responses also become retryable.

The dividing case is the timeout, and it is the reason the two policies exist. A request that timed out may have been received and executed by the upstream; you simply did not get the response. Retrying it against another endpoint risks submitting the same transaction twice. For a read, that is harmless. For a transaction broadcast, it is the kind of bug that produces a support ticket and a refund.

So write calls should normally use safe_only, and idempotent belongs only where duplicate submission is genuinely safe. Note that the gateway does not infer this from the method name. Sending a raw transaction is not automatically classified as unsafe. Configure it deliberately per route.

The failure codes distinguish the two exhaustion cases, which is useful when debugging. If at least one attempt was made and all failed, JSON-RPC returns -32005. If no candidate could be attempted at all, because everything was filtered or circuit-blocked, it returns -32004. The first means your endpoints are failing; the second means your route had nothing to try, which is a configuration or circuit problem rather than an upstream one.

Circuit Breaking Operates on a Different Timescale

Retries protect one request from one failure. A circuit breaker protects the system from an endpoint that keeps failing, by removing it from the candidate set so requests stop paying its timeout before moving on.

A circuit opens after consecutive hard failures such as connection failures, interrupted responses or timeouts, or after a sustained error rate within a short rolling window, once that window holds enough samples. The sample threshold matters: without it, two failures on a nearly idle endpoint would trip a breaker that nothing has actually exercised.

Recovery is probing rather than timing. After a cooldown the circuit half-opens and admits a single probe request. Consecutive successes close it; a failure reopens it with a longer cooldown that grows exponentially up to a ceiling. This is what prevents a flapping endpoint from being handed full traffic the moment a timer expires.

Two refinements in the Aurpay implementation are worth borrowing. First, circuits are scoped per workload class as well as per endpoint, with debug_ and trace_ family methods carrying their own circuit. Heavy tracing calls are far more likely to time out than ordinary reads, and without that separation one expensive analytics query could break an endpoint for all the normal traffic sharing it. Second, an upstream 425 or 429 opens a shorter throttle rather than a full circuit, honors the upstream’s own Retry-After, and is excluded from health scoring, because being throttled is not the same as being broken, and recording it as ill health would penalize an endpoint that is working exactly as intended.

One limitation deserves attention when you are reasoning about worst cases. Circuit state is shared runtime state, and when that state is unavailable the gateway admits the request rather than blocking it. The protection is best effort by design: it fails open, not closed. That is the right default for availability, but it means circuit breaking is not a guarantee you can build a correctness argument on.

Health Observation and Its Blind Spots

The filtering described earlier depends on health data, and it is worth knowing how that data is produced, because its gaps are not obvious.

Reported latency is the mean of successful attempts over a recent rolling window. Health status uses separate thresholds for entering an unhealthy state and for returning to a healthy one. That is hysteresis, and it stops an endpoint hovering near a threshold from oscillating in and out of rotation.

The blind spot is idleness. Observations decay to unknown once an endpoint stops receiving traffic, and the gateway does not probe idle endpoints on a schedule. The dashboard health check is a manual, on-demand probe. Combined with priority routing this produces a specific trap: a backup endpoint that has taken no traffic for weeks carries no fresh observation, and its true condition is discovered at the exact moment your primary fails and you need it.

The mitigation is to send it a trickle of real traffic. Weighted load balancing with a small weight on the backup keeps its observations current and verifies the credentials still work, which a synthetic probe against a health endpoint does not.

What Failover Deliberately Does Not Do

Knowing the boundaries of the layer is what stops you from assuming coverage you do not have. The Aurpay gateway is unusually direct about listing these, and the list generalizes to most gateways.

There are no hedged or parallel duplicate requests: no first-response-wins racing, no quorum across endpoints. Every attempt is sequential. There is no comparison of results across endpoints, no consensus check and no checksum verification, so an endpoint returning wrong-but-well-formed data will not be caught. There is no empty-result detection: a result of null is passed through unchanged, which matters because “not found” and “your endpoint is behind” can look identical at that layer. And there is no exclusion of an endpoint for lagging behind the chain tip.

That last one is the sharpest edge. An endpoint several blocks behind is healthy by every metric a gateway measures — it connects, it responds quickly, it returns valid data — while giving you a stale view of the chain. If your application makes decisions that depend on freshness, it has to check block height itself and decide what staleness it tolerates. No routing layer will do that for you, because only your application knows what the answer is for.

A Reference Configuration

For a read-heavy application, weighted load balancing across two or three interchangeable providers, a latency filter set somewhere near your p99 tolerance, an attempt bound of three, and the idempotent retry policy is a sound starting point. Keep a small weight on any endpoint you consider a backup so its health observations stay current.

For transaction submission, use a separate route with priority ordering, an attempt bound of two, and safe_only. Accept that a timed-out broadcast may need application-level reconciliation rather than a transport-level retry. Check whether the transaction landed before resubmitting.

For heavy tracing and debug workloads, isolate them on their own route where possible. They have different timeout characteristics and different failure rates, and mixing them with latency-sensitive reads means tuning one badly to accommodate the other.

Across all three, put caching in front of the routing layer rather than behind it. A request served from cache consumes no attempt budget, triggers no circuit evaluation and cannot be affected by an endpoint outage at all, which makes the approach in cutting RPC costs with response caching a reliability measure as much as a cost one. If you are still deciding whether to run this layer yourself, the trade-offs are similar to those in choosing between node types and hosted access.

Frequently Asked Questions

Does adding a backup RPC endpoint give me high availability?

Only for the failure modes your routing layer classifies as retryable, which generally means transport-level failures. It does not protect you from an upstream returning valid but unwelcome answers, serving stale data, or throttling you inside a well-formed response. Configure the backup, then verify which failure classes actually move traffic to it.

Why did my request fail when I have ten endpoints configured?

Because the attempt bound is separate from the endpoint count. A route holding ten targets with a bound of three tries three of them and then fails. Raise the bound if you want more attempts, keeping in mind that each one adds its own timeout to worst-case latency.

What is the difference between -32004 and -32005?

On the Aurpay gateway, -32005 means attempts were made and all of them failed, so your upstreams are unhealthy. -32004 means no candidate could be attempted at all, because every endpoint was filtered out or circuit-blocked, or because the route has no endpoint. The second points at configuration or circuit state rather than at upstream health.

Should I use the same retry policy for reads and writes?

No. Reads can safely use the more permissive policy that retries timeouts. Writes should not, because a timed-out transaction may have been executed upstream even though you never saw the response, and retrying elsewhere risks duplicate submission. No gateway can infer this from the method name, so set it per route.

Will a circuit breaker protect me from an endpoint returning bad data?

No. Circuit breakers act on transport-level failures and error rates, not on answer quality. There is no cross-endpoint result comparison or consensus checking in a typical gateway, and an endpoint lagging behind the chain tip registers as perfectly healthy. Validate freshness in your application.

How do I keep a rarely used backup endpoint trustworthy?

Send it a small share of real traffic. Health observations decay to unknown when an endpoint goes idle, and idle endpoints are not probed on a schedule, so a backup that has taken no traffic for weeks is an unknown quantity at exactly the moment you need it. A low weight in a load-balanced route keeps its observations fresh and confirms its credentials still work.

Configure for the Failures You Actually Have

Redundancy is a classification exercise before it is a configuration one. Decide which failures should move a request elsewhere, which are final answers, and which need your application rather than your routing layer to notice them. Then set attempt bounds against your latency budget, pick retry policies per route with writes treated separately, and keep your backups warm enough to be trusted.

If you would rather configure this than build it, the Aurpay RPC Gateway implements priority failover and weighted load balancing, per-endpoint circuit breaking with workload-class isolation, latency-based candidate filtering and the two retry policies described here, across 20 chain and network combinations spanning Ethereum, Polygon, BNB Smart Chain, Arbitrum, Optimism, Base, Solana, Bitcoin, Litecoin and TRON. It is free and open source under Apache-2.0, and it sits in front of whichever upstream providers you already use. The documentation is explicit about the failure handling it deliberately leaves to you, which is the part worth reading before you depend on it.