WebAPI outage on 29 September, and the fix now in production
The German WebAPI node was down for ten minutes on 29 September after one hung trading-server call. Every trading-server call now has a deadline, retries of trades are safe with an idempotency key, and failover is faster.
On 29 September, 15:47–15:57 UTC, the WebAPI node in Germany stopped answering. A call that
hung inside one trading server made every other call queue behind it until the node ran out of
workers. Traffic to cloud.mywebapi.com moved to our other regions within a few minutes; clients
calling m4.mywebapi.com directly were cut off for the full ten minutes.
Since 30 September, in production in every region:
- every trading-server call has a deadline — trades 5 s, reads 10 s, changes 15 s, history 30 s,
maintenance 60 s — adjustable per request with the
X-Request-Timeoutheader; - a timeout says what it means:
Timeout(nothing changed, safe to repeat),OutcomeUnknown(a trade or change may still complete — check before repeating),Busy(not sent, safe to repeat); - a repeat with the same
Idempotency-Keyis never executed twice and returns the original result; - failover reacts within about a minute instead of several.
The full account is on our blog: Ten minutes down on 29 September.
This note sums up an article on the CPlugin™ blog. Read the full article ↗