I seem to be stuck on the exciting topic of VAR and backtesting models. Anyhow via a friend who pointed out this comment in an article in the economist.
UBS’s investment-banking division lost SFr4.2 billion ($3.6 billion) in the third quarter. The bank’s value-at-risk, the amount it stands to lose on a really bad day, has shot up … On 16 days during the quarter its trading losses exceeded the worst forecast by its value-at-risk model on the preceding day. It had not experienced a single such day since the market turbulence of 1998.
What is staggering is the last part. They had not had a single backtesting breach from 1998-2007. I wasn’t convinced the Economist got it right so I checked the UBS third quarter report which states it more explicitly.
… When backtesting revenues are negative and greater than the previous day’s VaR, a “backtesting exception” occurs.
In third quarter we suffered our first backtesting exceptions – 16 in total – since 1998. Given market conditions in the period, the occurrence of backtesting exceptions is not surprising. Moves in some key risk factors were very large, well beyond the maximum level expected statistically with 99% confidence.
Now it seems that they are trying to reassure people that the model is ok even if it breached 16 times in one quarter year (i.e. 25% of the time), because they haven’t breached in the period 1998-2007 at all. Anyone who actually thinks about it will realise that the probability of no breaches in 9 years – around 2250 trading days is very, very low and should be an indicator of something wrong with their model. The probability of running a model and getting no breaches in a 9 year stretch is around 1 in 1,000,000,000. While I accept that markets cluster their extreme moves and this is difficult to account for, its not like their have been no extreme events in the period. September 11th 2001 to name one.
They have perhaps fallen into the error of taking comfort from having a “conservative” model meaning they will always be allocating more capital rather than less. This is totally misleading. You construct a model to try and predict a certain level of confidence. If you can’t do that even approximately then you really need to look at your model. If you want to be conservative reserve a greater multiple of the VAR as capital. Using a model that doesn’t actually reflect reality is never going to give you any sort of confidence in the model integrity. A false negative result – too few breaches, is just as much an error in your model as a false positive one.