Ignoring false negatives

I seem to be stuck on the exciting topic of VAR and backtesting models. Anyhow via a friend who pointed out this comment in an article in the economist.

UBS’s investment-banking division lost SFr4.2 billion ($3.6 billion) in the third quarter. The bank’s value-at-risk, the amount it stands to lose on a really bad day, has shot up … On 16 days during the quarter its trading losses exceeded the worst forecast by its value-at-risk model on the preceding day. It had not experienced a single such day since the market turbulence of 1998.

What is staggering is the last part. They had not had a single backtesting breach from 1998-2007. I wasn’t convinced the Economist got it right so I checked the UBS third quarter report which states it more explicitly.

… When backtesting revenues are negative and greater than the previous day’s VaR, a “backtesting exception” occurs.
In third quarter we suffered our first backtesting exceptions – 16 in total – since 1998. Given market conditions in the period, the occurrence of backtesting exceptions is not surprising. Moves in some key risk factors were very large, well beyond the maximum level expected statistically with 99% confidence.

Now it seems that they are trying to reassure people that the model is ok even if it breached 16 times in one quarter year (i.e. 25% of the time), because they haven’t breached in the period 1998-2007 at all. Anyone who actually thinks about it will realise that the probability of no breaches in 9 years – around 2250 trading days is very, very low and should be an indicator of something wrong with their model. The probability of running a model and getting no breaches in a 9 year stretch is around 1 in 1,000,000,000. While I accept that markets cluster their extreme moves and this is difficult to account for, its not like their have been no extreme events in the period. September 11th 2001 to name one.

They have perhaps fallen into the error of taking comfort from having a “conservative” model meaning they will always be allocating more capital rather than less. This is totally misleading. You construct a model to try and predict a certain level of confidence. If you can’t do that even approximately then you really need to look at your model. If you want to be conservative reserve a greater multiple of the VAR as capital. Using a model that doesn’t actually reflect reality is never going to give you any sort of confidence in the model integrity. A false negative result – too few breaches, is just as much an error in your model as a false positive one.

3 Responses to Ignoring false negatives

  1. JC says:


    I found VAR to be useless and almost misleading. Doll Yen fell from about 145 to 111 in 3 days back in 1998(?). For the next 12 months CSFB’s model basically stopped us taking yen related positions simply because of what had happened. The market for the next 12 months was extremely jerky as other banks had the same issue in terms of not being able to carry yen risk. In other words the models were actaully adding to the volatility of the market. We couldn’t ignore it as the firm was very strict on limts.

    Our options traders also had trouble pricing client trades as a result of this kind of abberation.

    cheers. no proof read as usual.

  2. Andrew says:

    VaR, like all models, are useful tools but atrocious masters. I always get worried when I see a model (any model) being treated like the Oracle and there being no human over-ride. Even worse is when the human trusted with the over-ride is not someone who understands the model (like most senior bank management).
    From your situation above there would have been substantial opportunity for profit taking and it was probably the hedge guys out there making it.
    Good to see you back, Steve.

  3. Jack Lacton says:

    Sounds like their model suffers from the same big ol’ fat tail distribution that killed LTCM.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: