USDT Price Incident - Post Mortem

On June 18th, 2022 at 9:14 AM UTC, the Flux team was alerted to a potential pricing irregularity on the price-feed for the asset USDT, pushed to Aurora Mainnet by the Open-Source First Party Oracle Node. Following this message, we immediately began engaging with all affected parties to determine the duration and impact of this pricing irregularity.

TLDR;

  • USDT price deviated to $1.76 on June 18th due to an Open-Source First Party Oracle Node misconfiguration
  • Only 20 addresses were affected. Bastion and Flux will fully reimburse users, totaling up to $81,674 in liquidations in USDC across these affected 20 addresses.

Timeline of events

At block 67915511 on the Aurora Blockchain, a price feed was updated through the Open-Source First Party Oracle Node reporting the price of USDT as $1.76574941. This price was pushed for two price updates spanning the course of a total of two minutes until block 67915572, resulting in 61 blocks of affected transactions.

Following this two-minute period, the price feed was updated through the Open-Source First Party Oracle Node reporting back to $0.99892322 per USDT, observed before the price shift.

What caused this price irregularity?

On Jun-17-2022 12:15:00 PM, Bitfinex reported the below prices on USDT_MXNT (the Mexican Pesos pair)

exchange | pair | timestamp | open | high | low | close | volume |

----------+-----------+---------------------+--------+--------+--------+--------+-----------------------+-

bitfinex | usdt_mxnt | 2022-06-17 12:15:00 | 100 | 100 | 100 | 100 | 0.0046 |

bitfinex | usdt_mxnt | 2022-06-17 12:35:00 | 20.309 | 20.309 | 20.309 | 20.309 | 0.00169032 |

bitfinex | usdt_mxnt | 2022-06-17 12:39:00 | 20.305 | 20.305 | 20.305 | 20.305 | 30.56655996 |

bitfinex | usdt_mxnt | 2022-06-17 13:01:00 | 20.377 | 20.377 | 20.377 | 20.377 | 0.00131467 |

bitfinex | usdt_mxnt | 2022-06-17 13:09:00 | 20.396 | 20.396 | 20.396 | 20.396 | 10 |

The Open-Source First Party Oracle Node was configured to use a VWAP endpoint with the “includeCrossRate” flag set “true”. This meant that the calculation included not only usdt_usd, but also usdt_eur, usdt_aud, usdt_mxnt, etc. from all exchanges.

"source_path": "payload.price",

"end_point": "https://web3api.io/api/v2/market/spot/vwap/pairs/usdt_usd/latest?includeCrossRates=true&lookbackPeriod=5"

During this configuration, the Open-Source First Party Oracle Node was providing the payload.price simple average which included the deviation of a generated price of $4.8341 for the USDT_MXNT pair on Bitfinex. This configuration provided the simple average price of $1.76574941 rather than the VWAP price of 0.99892322.

timestamp: 2022-06-17 12:15:00 000
pair:  usdt_usd 
price: 1.76574941
volume: 170588.4247
vwap: 0.99892322

Post-Mortem

From Saturday morning to Sunday evening, the Flux team worked to identify the cause of this configuration and re-configure the First Party Oracle Node with the Amberdata team. This solution was deployed to the mainnet ensuring that all data feeds remained stable and accurate in any market condition.

On Monday afternoon, this same re-configuration was deployed to provider nodes serving the NEAR and Evmos ecosystems, to ensure exploits relating to the trading of illiquid pairs would not affect protocols consuming data.

Next Steps

We encourage all teams that have integrated with Flux Protocol to take steps to further decentralize the data sources they consume and build in fail safes. Bastion will further develop its oracle infrastructure by including aggregation of other data providers to provide an aggregate price for more robust outlier protection.

The calculations for affected addresses and amounts on this spreadsheet. The price of all coins is taken at the time of liquidation (June 17th 12:18:55 PM UTC) to ensure consistency.

Conclusion

We encourage all protocols integrated with Flux to ensure the data feeds they are consuming are properly tailored to their use case and need. Flux provides open-source infrastructure connecting high-quality data providers.

If you’re currently using data feeds from the Open-Source First Party Oracle, without notifying the Flux team, please reach out to ensure all data feeds to meet your needs.

1 Like