UBS’s investment-banking division lost SFr4.2 billion ($3.6 billion) in the third quarter. The bank’s value-at-risk, the amount it stands to lose on a really bad day, has shot up … On 16 days during the quarter its trading losses exceeded the worst forecast by its value-at-risk model on the preceding day. It had not experienced a single such day since the market turbulence of 1998.

What is staggering is the last part. They had not had a single backtesting breach from 1998-2007. I wasn’t convinced the Economist got it right so I checked the UBS third quarter report which states it more explicitly.

… When backtesting revenues are negative and greater than the previous day’s VaR, a “backtesting exception” occurs.
In third quarter we suffered our first backtesting exceptions – 16 in total – since 1998. Given market conditions in the period, the occurrence of backtesting exceptions is not surprising. Moves in some key risk factors were very large, well beyond the maximum level expected statistically with 99% confidence.

Now it seems that they are trying to reassure people that the model is ok even if it breached 16 times in one quarter year (i.e. 25% of the time), because they haven’t breached in the period 1998-2007 at all. Anyone who actually thinks about it will realise that the probability of no breaches in 9 years – around 2250 trading days is very, very low and should be an indicator of something wrong with their model. The probability of running a model and getting no breaches in a 9 year stretch is around 1 in 1,000,000,000. While I accept that markets cluster their extreme moves and this is difficult to account for, its not like their have been no extreme events in the period. September 11th 2001 to name one.

They have perhaps fallen into the error of taking comfort from having a “conservative” model meaning they will always be allocating more capital rather than less. This is totally misleading. You construct a model to try and predict a certain level of confidence. If you can’t do that even approximately then you really need to look at your model. If you want to be conservative reserve a greater multiple of the VAR as capital. Using a model that doesn’t actually reflect reality is never going to give you any sort of confidence in the model integrity. A false negative result – too few breaches, is just as much an error in your model as a false positive one.

I considered entitling this “Am I a fraud?” as the contention raised in the book that those who use such models are either idiots unaware of the problem or frauds aware of the problem, but content to sell their services based upon an idea they know to be wrong. On the surface of it I fall into the second category. Although I came out of a physics PhD on powerlaws in physical systems, aware of the what they imply, and know that financial timeseries typically have powerlaw tails (Mandelbrot work on this cited in my thesis). Despite this I still work day in day out with normal based models.

The first line of defence of myself would be that these are really the best we have. However Taleb’s contention is that financial models based on the normal distribution are so badly flawed that not only do they not miss the really significant risks (the big ones) by assuming them away, they go further and actually create them. Once we have constructed a model to reflect the risk, people trade and protect themselves on the basis that the model’s predictions actually reflect reality and that large, uncorrelated moves are exceptionally rare. This leads them into taking on additional risk, and leaves them worse exposed in the case of exceptionally large moves.

There are numerous examples where he would cite this as having occurred recently with the subprime collapse in the US, we have statements that we had 1 in 10000 year events occurring every day for 3 days, the collapse of Amaranth, and of course LTCM. In all of these cases we had people believing that their models actually captured all the risks they faced and therefore could trade accordingly with a good idea of the worst results that could occur. When something occurred which was outside their model they blew up.

Taleb’s contention is that we know that markets don’t have normally distributed outcomes. While there are studies which have looked through series of data to confirm normality this is an essentially worthless activity. As a follower of Popper, his view is theories are conjectures, which we must reject if they are falsified. Induction will never show us what is the real distribution, however falsification can show us what aren’t acceptable models. I accept and agree with this.

In the case of the normal distribution and related distributions we need only look at the massive overabundance of extreme events. It is typical to see say “5 sigma” events – which should occur roughly once every 10,000 years about once every say 3-5 years. Knowing this, it is pointless to claim that a normal distribution describes market moves. To claim that you can find periods where it is normal is – to paraphrase Taleb, like having breakfast with OJ Simpson, and thinking the fact that he didn’t kill anyone while you were there means he isn’t a killer. Absence of evidence is not evidence of absence.

The question I ask is do most practitioners actually believe that their models are an accurate description of reality? Or are they trying to capture what risks they can in a normal model while still relying on things such as stress tests to attempt to capture exposure to the more extreme Black Swans. Certainly in my experience this is what occurs. The Quants I am experience with were physicists and mathematicians gone bad, not the MBA graduates Taleb refers. Perhaps those people do believe in normal markets.

Anyway onto whether what I do is essentially fraudulent. My answer is no, but there is a danger it is taken the wrong way. Firstly consider Value-at-Risk (VAR). We calculate VAR for a 99% quantile, trying to estimate where the size of a 1 in 100 day loss. Now while I can never backtest this model to prove it, we can still test it and see that it has historically found pretty close to the correct number of breaches (2% are above 1% or below 99%), which is the critical test that we are trying to check. Certainly it hasn’t been falsified for this criterion. Now how do I reconcile this with my belief that such models are ultimately flawed by not predicting large events? Well quite simply we don’t need to predict large events to calculate accurate VAR. 1 in 100 is not a Black Swan by any means and even with limited non-normality in our model we can account for this type of event.

If however we were to test say, the most extreme moves in our data set, we would find moves that are really not accounted for by the model eg multiple 5 sigma events. So while we may not capture the big moves correctly we don’t need to for VAR purposes. We only need 1 in 100. Thus I can deliver a VAR number with a straight face and a feeling of honesty. I believe we are giving a good estimate of the 1 in 100 risk.

In my opinion the real question is not, is our VAR number accurate, but is this estimate really useful and is it being used correctly. If I accept there are fat tails (powerlaws), then the risk of say a 1 in 1000 move doesn’t scale to the size of a 1 in 100 move how we would expect or predict from our model. More importantly should I be provisioning based on a multiple of 1 in 100 VAR when I know it doesn’t scale to larger moves.

One way that this is checked for is that for the case of more extreme moves we don’t in general calculate exposure due to a VAR model but rather stress test various combinations of extreme moves. While comparing these to a multiple of VAR against the level of risk we may think if we judge it off a normal model, it gives us some extreme bound with which we compare through stress tests what may happen in extreme situations.

So when Taleb says that taking VAR or volatility (Sigma) or some other normal distribution derived number and calling it your risk is wrong, I agree with him. It’s plainly inadequate. However when he calls it worthless or even dangerously misleading I tend not to agree. Taleb seems to fall for one of his own fallacies here. Absence of evidence is not evidence of absence. He explains at length in one of his narratives how a legislator that had introduced tighter security before 9/11 could have prevented the disaster, but since no-one would have known about this there would have been no reward or glory for the person. Possibly they would have been reviled for unnecessary security restrictions, given we never really saw what was averted. How does Taleb know that VAR hasn’t stopped numerous dangerous build up of risks, and so averted disasters? Certainly we know in the case of the NAB options trading debacle, if they had paid attention to their VAR models they would have seen the problem building up over time quite clearly.

So in one respect I agree with Taleb. Anyone who thinks their VAR model encompasses all risk is extremely naive and foolish. However I disagree that the use of such models are useless or dangerous. They must be used with their flaws in mind and in my experience they mostly are.

All up I found The Black Swan an very interesting read. If you’ve read fooled by randomness then some of the arguments will have already appeared but certainly not all. The Black Swan is concerned with wild randomness, the randomness that according to Taleb dominates modern society. Fool by randomness largely concerns the randomness of games, what Taleb would describe as mild randomness. So mild that it is hardly random at all.

As we are reminded in several places in the book Taleb made his fuck off money (i.e. enough money to be comfortably independent even if not mega wealthy) in the 87 stockmarket crash betting on the fact that the market under-appreciated Black Swans. He doesn’t need to fawn to the establishment be it economics, philosophy or publishing and we get this tone all through the book.

While the book touches on finance applications its hardly a finance book. Rather Taleb considers it to be as much a work of Epistomology. His chief concern is with what he calls epistemic arrogance. He hates those who profess to know more than they really do know, be they economic modellers, historians or political scientists, but his chief enemy is the normal distribution which is described as the Great Intellectual Fraud. To use normal distributions in fields where it is easy to show are not normal (such as finance), is both fraudulent and causes Black Swans.

Its strong stuff which he backs with examples, logic and the findings of behavioural finance and similar studies. Its also fairly convincing for the most part. Its true we don’t need sophisticated statistical studies to find whether market moves are normally distributed. We only need the fact that we get 1 in 10,000 year events (as modelled by a normal distribution) occurring every few years. Risk managers who run such models (such as me) are either ignorant fools, who actually believe wrongly in what they do, or deceptive frauds who know better but persist in fooling others for the money. I’d contend I’m neither but I’ll discuss later on.

Anyway I leave the rest to further discussion. I particularly want to mention the Narrative Fallacy, which I have actually mentioned before, the problem of prediction as well as whether I really am a fraud, idiot, or neither.

Anyhow the blogpost I was always intending to write next was a book review of The Black Swan by Nassim Taleb, as it was what I was reading at the time and touches on a number of topics I am interested in. So last week I began re-reading it with an eye to finally discussing it. Hopefully the reading and review will be done in a few days.

Should I worry about this though? Ultimately I think we should be acting to mitigate AGW, so is it best for us not to be criticising our “side”? I think that this kind of partisanship is amongst the worst things that can be done. Claims that are unsupported by evidence will ultimately undermine the very case they are trying to support. They are in effect crying wolf.

The stupidity of course isn’t limited to AGW alarmists, with many if not more of the skeptics running the its a cold day, it can’t be warming line. Some of these are taking the piss but not all of them. Some seem to believe its a refutation.

Ultimately it would do well if people were to remember that a particular instance is usually not a good representation of the average. Its ridiculous to ascribe meaning to every single occurrence, and even a couple of years don’t really do much to hundred year trends. Climate is long term average weather. Year to year it varies, and we are looking at long term trends not day to day heatwaves or even year to year droughts.

This shows few regions which have shown a definite drying trend over the period. One clear exception being the south west of WA, which does show a drying trend over pretty much anytime period you pick. In addition it appears that some slight trend could be evident in mid to northern Queensland.

Go and have a look at some of the different time periods on their site, the lack of a clear trend for much of eastern Australia isn’t just for the period 1900-2006. In case any one’s concerned that its because we started the comparison in a drought. Take 1910-2006, 1920-2006 or 1930-2006 and you get something similar.

Indeed its not unless you compare with the 1950′s to 1970′s that you get a really marked decrease in rainfall in most of Eastern Australia.

A priority for the new centre is to better understand the mystery of why Australia’s most populated region, the continent’s east coast, has suffered such major declines in rainfall in recent decades.

“We recently had a round-table of Australia’s leading climate-change researchers and this emerged as the biggest unknown issue and, of course, it seriously affects the largest concentration of people stretching right down the coast from Cairns to Melbourne,” Professor England said.

So I had a look again at the rainfall graphs and time series provided by the Bureau of Meteorology for eastern Australia and there is no doubt that we have been receiving less rainfall in recent decades than the 1950′s. However it also seems that we are receiving more rainfall than in the first half of the 20th century, so the question seems to me to be as much why did rainfall rise in the 50′s as why has in fallen in the 80′s and 90′s.

While it would be great it we could get more rainfall it seems that it is weak to link this to climate change, when it is well within the variation that we have seen over the observed history. Indeed if we look at averages they are close to the highest point they reached pre-1940. If we look at years in the bottom and highest deciles of rainfall they are right where we expect for the last 20 years (2 in the highest and 2 in the lowest deciles). If we look at percent of regions with rainfall in the lowest decile then we also see that while 2002 was high it was not the worst on record.

It seems curious to me that we should be studying climate change induced drying when we appear to be returning to the norms of relatively recent history. Now contrast this to say the below graph of mean temperature over the same region.

Now here it is clear that you may want to ask why has it become warmer, as there appears to be a clear warming trend in the last 50 years, and the recent mean values are the highest on record. I can even understand wanting to study the likely effect of warmer weather on rainfall, but the question of why has rainfall decreased seems to be a badly flawed one, as nothing we’ve seen recently in terms of rainfall seems to be beyond the normal climatic variation. Is there any reason for considering the 50′s to be more climatically typical than the 20′s, 30′s or 40′s?

Rather than showing a surge in rents, data from the New South Wales Government shows there was no growth in apartment rental bonds in the three months to December.
In Sydney’s inner suburbs, three-bedroom houses recorded growth of 3.6 per cent but two-bedroom houses also showed zero growth.

Of course this is for the end of last year rather than the start of this year, but it puts the predictions in perspective. As I stated before its not surprising that the real estate industry publishes favourable research to their interests. What is disappointing is that the media outlets, papers, television, radio commercial as well as ABC and SBS almost uniformly regurgitate this uncritically.

Media watch has a more comprehensive hosing down of media gullibility in the face of real estate industry “research”. A fine collection of misquoted tenants, and exagerated stories some being pushed by political parties.

At the AAAS conference in San Francisco I was a discutant of session in which John Ioannidis showed that 4 out of 5 epidemiological “statistically significant” studies fail to replicate in controlled experiments.

NNT crows that this is what he has already come to describe as the narrative fallacy. If you look hard enough at enough data, you will see a pattern emerge.

Anyway I have looked up John Ioannidis’s research and found this interesting paper Why Most Published Research Findings Are False
, which unfortunately I haven’t had the chance to read in full.

The outline of his idea is simple enough, if you look at enough data (particularly small data sets) you will find statistically significant relationships. The part I thought was interesting was this.

As has been shown previously, the probability that a research finding is indeed true depends on the prior probability of it being true (before doing the study), the statistical power of the study, and the level of statistical significance.

Which is kind of obvious. If I correlate enough astrological data with some disease I will inevitably find some correlation, but because the prior probability of it being true is essentially zero there is still very little chance of the study being true.

Some of the corollaries of this cited in the paper are interesting

Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true. This seemingly paradoxical corollary follows because, as stated above, the PPV of isolated findings decreases when many teams of investigators are involved in the same field. This may explain why we occasionally see major excitement followed rapidly by severe disappointments in fields that draw wide attention. With many teams working on the same field and with massive experimental data being produced, timing is of the essence in beating competition. Thus, each team may prioritize on pursuing and disseminating its most impressive “positive” results. “Negative” results may become attractive for dissemination only if some other team has found a “positive” association on the same question. In that case, it may be attractive to refute a claim made in some prestigious journal. The term Proteus phenomenon has been coined to describe this phenomenon of rapidly alternating extreme research claims and extremely opposite refutations. Empirical evidence suggests that this sequence of extreme opposites is very common in molecular genetics.

Obviously there are a great many fields that data mine to find relationships. With such an approach a 95% confidence interval tells you almost nothing if you didn’t already suspect a relationship between data.

I’m thinking of reviewing some articles I started and put on hold and maybe putting them out to get me back on the horse.