Aurora Downtime Post-Mortem from Flux Protocol

On February 6, 2022, a transaction overload in the Aurora network caused several Flux integrations to no longer receive updated price information from the Flux Protocol.

Aurora is a smart contract on NEAR which enables users to deploy EVM contracts onto the NEAR blockchain. Flux provides protocols deployed on Aurora with data, primarily price feeds.

The Flux team looked into what went wrong and what preventative measures developers can take to avoid similar situations in the future.

Sequence of events

According to a statement from Aurora, on February 6, the platform’s transaction load went up significantly, causing many unconfirmed user transactions to drop.

The Aurora team has said the sudden influx of activity mostly came from Moon Flower Farmers, a free-to-play and earn game that recently launched on Aurora. New users from the game accounted for about 80% of the transaction volume before the RPC node went down. This caused an issue with the platform’s RPC node, which was overwhelmed by users playing the game. RPC nodes are endpoints that allow users to interact with the blockchain, whether they are reading or uploading data. As a result of this, the vast majority of users were not able to send transactions.

After the transaction overload, Flux engineers were made aware of the issue. The team took all of the necessary steps to ensure integrations and protocols relying on Flux’s data feeds were alerted of the issue. No funds were lost during this outage.

Our team’s initial theory was that data providers were banned due to RPC limits for posting too often on-chain. After further investigation, it indeed did seem to be an RPC issue. We tried to bridge ETH from Ethereum to Aurora, which failed. We then tried to bridge ETH from NEAR to Aurora, but our engineers were banned from the RPC. At this point, there were no further steps we could take.

We internally notified integrations that were on mainnet and waited for official information from the Aurora team on what was causing the downtime.

Future steps and lessons

In light of this incident, the Flux team is launching a backup provider node that uses an aggregate of Amberdata and Kaiko’s data feeds to ensure Flux can still push data on-chain if Aurora goes down again.

We’re writing an automation script to notify developers faster of situations like these, if they should come up again. We are also creating an emergency announcement channel where developers can communicate with us and receive immediate support.

Additionally, to lessen the damage caused by downtimes in the future, the Flux team advises developers to build in fall back mechanisms that pause their smart contract if price data is not updated over a 20-minute interval period from Flux’s oracle provider node.

We’ve reached out to Aurora with further questions and will update this post when we hear back.

If you have any questions or concerns around this post-mortem and how we’re handling future issues that could pop up with Aurora, feel free to reach out to us on Discord.

3 Likes