He ignored 711 warning messages on his $444 Billion trade

[ValueArb]

Quote

For one Citigroup Inc. trader in London, the morning of May 2, 2022 went from bad to worse.

 

It was a public holiday in the UK, so it should have been a quiet day in the markets. A little before 9 a.m., the staffer on Citigroup’s Delta One trading desk began putting together a trade that would hedge the bank’s exposure to the MSCI World Index. 

A tool that employees usually used for such a transaction wasn’t available that morning, so the trader had to manually build the basket of stocks. That’s where things started to go wrong. 

 

In Citigroup’s systems, traders have the option of either entering the notional value of a trade they want to make or entering the quantity of index units they want to trade. On that day in May, the trader meant to create a basket of equities valued at $58 million, but accidentally entered that 58 million into the quantity field instead, creating a mammoth $444 billion basket containing 349 stocks from 13 different countries.

 

The Wall Street giant’s systems immediately fired off hundreds of warnings, ultimately blocking some — but not all — of the trade from going through. Still, about $1.4 billion worth of equities began to be sold across European exchanges. 

Markets immediately started going haywire. Within minutes, the trader realized the mistake and canceled the order. But the damage was done: The blunder had sparked a five-minute selloff in European stocks, wreaking havoc in bourses stretching from France to Norway.

...

When the trader first input the erroneous trade, he was met with a wall of 711 warning messages. He quickly overrode the ones he could and the order was placed at 8:56 a.m.

...

By 9:10 a.m., the trader had canceled the order, saddling Citigroup with a $48 million loss.

 

https://www.bloomberg.com/news/articles/2024-05-22/the-15-minutes-on-a-citi-trading-desk-that-sparked-a-flash-crash?srnd=homepage-americas

 

As a long time software developer this sounds like the real precipitant of the problem was excess false negatives. Its good to generate warning messages to stop users from making significant mistakes. But if your system regularly generates so many false warning messages that it trains users to ignore or override them in common usage, you might be creating more problems than you solve. I wonder if this trader thought nothing would go wrong by over-riding the warnings because the system had trained them to use over-rides to get trades done, and led them to believe if the system allowed the over-ride it was okay.

 

I suspect somewhere buried in the 711 warnings was likely the critical warning that said something like "do you realize you are placing a $444 BILLION DOLLAR ORDER" and they either never saw it or, or didn't see it until after they had over-ridden numerous warnings to get the first billion in sales made. 

 

I just replaced my decade old Cayenne with a new Audi SUV (new for me, it's a 2020). And it has a bunch of electronic alerts and warnings, especially when parking. I can't park anywhere, even my garage, without it lighting up and beeping.  But the 2nd month I owned the new car I was backing out of a parking space and hit another car. I was shocked and wondered why the alert chimes hadn't sounded to warn me. Then I realized they had.

 

They were so frequent that I had stopped registering them in my brain unless I knew I was really close to something. I never saw the woman backing out at the same time, so when the chimes sounded I never heard them.

[thowed]

Good stuff - totally agree.  Technology is a tool, and it feels like increasingly it's being misused.

[Saluki]

It's a weird scenario.  If you have parameters in place that can't be overridden, that can obviously cause a huge problem, but you have to have safeguards in place to protect against mistakes or intentional acts that benefit the trader.  I'd be curious how many times a day the warnings go off.  If they go off a lot, I still think it would be weird to keep clicking "OK" after the first 500 warnings.  When the position size for the order goes from millions to hundreds or millions, or billions, why doesn't it require 2 people to okay it? 

 

I have a friend who worked at a big investment bank and told me that the people at the trading desk had parameters that they had to trade in (run a balanced book, trade FX in major currencies up to $xxx, european blue chips etc.).  If someone is down a lot and isn't going to get a bonus, there is an obvious incentive to do something risky and get a big YOLO payout if you're right, and the firm eats the loss if you're wrong. Their solution was very simple:  "when we add up your profits for the year to figure out your bonus, we don't count the trades that you made which were outside your parameters." Incentives matter. 

 

 

[Spekulatius]

The warnings are not meant to guide or help the user (711 warnings and warning that go off on everything are useless) they are meant to protect someone from the legal department, management and/or the owner of the software within the company from repercussions, legal or otherwise.

 

Those folks can always say that the warning was there and it was user error.

[villainx]

12 hours ago, ValueArb said:

I just replaced my decade old Cayenne

 

it's been 10 years (at least) since Porsche SUV?!?!?!!!!?!

[ValueArb]

10 hours ago, villainx said:

 

it's been 10 years (at least) since Porsche SUV?!?!?!!!!?!

 

My 2010 Cayenne was the last of the first generation that was first produced in 2002!

[ValueArb]

More details, looks like trader was confused by a display that defaulted to confirming the value they expected.

 

Quote

Ordinarily, the Value at Benchmark field (ValAtBM) on the PTE screen displays the value of the relevant basket at a specified benchmark and is used where traders need to track the value against a reference price. In this case, PTE defaulted to the option "Strike". The default "Strike" option was programmed to determine the price of the Index at the prior day's close, by reference to an external data feed. However, as data from that external feed was unavailable, the price of the value of the Index instead defaulted to -1 rather than the benchmark price which was US$7684.40. The quantity of units was therefore multiplied by -1. There were number of other fields on the PTE screen in which the total notional value of the basket was correctly displayed. However, the trader only checked the the ValAtBM on PTE to confirm the size of the basket. When the trader checked the value of the inputted basket, they were presented with a figure of negative 58 million for the value of the basket (58 million multiplied by -1). The trader saw a ValAtBM of -58,000,000, which was the number they expected to see, and thus they clicked Execute to continue to the next check. The quantity box, next to the ValAtBM also presented 58,000,000. Had the data feed been available, ValAtBM would have shown a basket of approximately US$444bn i.e., the true notional value of the basket.

...

At 08:56 a ‘Trade Limit Warning’ pop-up alert appeared within PTE. This presented the trader with 711 warning messages, consisting of hard block and soft block messages, listed in a single alert where only the first 18 lines of alerts were immediately visible unless the person who received the alert scrolled down. The trader did not appreciate their inputting error and overrode all of the soft warnings in the pop-up.

...

Two hard blocks generated by the PTE system, which could not be overridden, collectively stopped US$248bn of the basket of equities progressing for execution. The trader was then presented with a further pop-up alert entitled “Final Trade Confirmation”. It contained a wave notional value of all the individual equities in the basket as a total (which was approximately US$196bn). The trader clicked the “OK” option which routed the remaining basket of equities with a notional value of $196bn into CitiSmart for execution using a VWAP trading algorithm, where individual parent and child orders were generated.

 

Matt Levine had a funny thought.

 

Quote

Note the FCA’s timestamps. The Algorithmic Service Desk handed off its responsibilities at 8:48; the trader was ignoring pop-up warnings by 8:56. I really do not know what to make of the fact that the “team responsible for real-time monitoring of internal executions” went on vacation eight minutes before an internal execution got fat-fingered. Is some Citi trader fat-fingering trades like this all the time, and only the heroic concentration of the Algorithmic Service Desk keeps it from hourly disaster?

ds

I like to imagine that the Algorithmic Service Desk person wrapped up work, sent an email formally handing off her responsibilities, left her desk, and was still in the office waiting for the elevator when this all went down.[6] Her phone buzzed frantically and she took it out, looked, sighed, turned it off, said “nope, I’m on vacation, this is someone else’s problem,” and got on the elevator.

 

[villainx]

14 minutes ago, ValueArb said:

 

My 2010 Cayenne was the last of the first generation that was first produced in 2002!

 

Time flies, felt like not that long ago when I was wondering when Porsche (and other high end car companies) would be introducing an SUV, and then seeing them on the road too.   Can't believe it's been over 20 years ago.  Talk about memories not jiving with reality.

[ValueArb]

What would have been the crazy part to me is if you told me in 2000 that in 2023 70% of Porsche's units sold would be Macan/Cayenne, and Cayenne would have been their most important product line for the last two decades.

 

I miss the Cayenne. Audi just isn't quite the same.