Into the woods

So my wife has added a new website for her art business she is starting up. Not much there yet, but it will be growing shortly. Will be original prints and cards of her original artwork. Have a look at Into the Woods if you are interested. If there is not much there when you visit come back in a couple of weeks and have a look as there will be then.

Ignoring false negatives

I seem to be stuck on the exciting topic of VAR and backtesting models. Anyhow via a friend who pointed out this comment in an article in the economist.

UBS’s investment-banking division lost SFr4.2 billion ($3.6 billion) in the third quarter. The bank’s value-at-risk, the amount it stands to lose on a really bad day, has shot up … On 16 days during the quarter its trading losses exceeded the worst forecast by its value-at-risk model on the preceding day. It had not experienced a single such day since the market turbulence of 1998.

What is staggering is the last part. They had not had a single backtesting breach from 1998-2007. I wasn’t convinced the Economist got it right so I checked the UBS third quarter report which states it more explicitly.

… When backtesting revenues are negative and greater than the previous day’s VaR, a “backtesting exception” occurs.
In third quarter we suffered our first backtesting exceptions – 16 in total – since 1998. Given market conditions in the period, the occurrence of backtesting exceptions is not surprising. Moves in some key risk factors were very large, well beyond the maximum level expected statistically with 99% confidence.

Now it seems that they are trying to reassure people that the model is ok even if it breached 16 times in one quarter year (i.e. 25% of the time), because they haven’t breached in the period 1998-2007 at all. Anyone who actually thinks about it will realise that the probability of no breaches in 9 years – around 2250 trading days is very, very low and should be an indicator of something wrong with their model. The probability of running a model and getting no breaches in a 9 year stretch is around 1 in 1,000,000,000. While I accept that markets cluster their extreme moves and this is difficult to account for, its not like their have been no extreme events in the period. September 11th 2001 to name one.

They have perhaps fallen into the error of taking comfort from having a “conservative” model meaning they will always be allocating more capital rather than less. This is totally misleading. You construct a model to try and predict a certain level of confidence. If you can’t do that even approximately then you really need to look at your model. If you want to be conservative reserve a greater multiple of the VAR as capital. Using a model that doesn’t actually reflect reality is never going to give you any sort of confidence in the model integrity. A false negative result – too few breaches, is just as much an error in your model as a false positive one.

The invalidity of risk models (or do I trust my VAR calculations)

To continue some of the stuff raised in The Black Swan I’ll discuss Taleb’s attack on financial models based on the normal distribution, this misuse of which he describes as the Great Intellectual Fraud.

I considered entitling this “Am I a fraud?” as the contention raised in the book that those who use such models are either idiots unaware of the problem or frauds aware of the problem, but content to sell their services based upon an idea they know to be wrong. On the surface of it I fall into the second category. Although I came out of a physics PhD on powerlaws in physical systems, aware of the what they imply, and know that financial timeseries typically have powerlaw tails (Mandelbrot work on this cited in my thesis). Despite this I still work day in day out with normal based models.

The first line of defence of myself would be that these are really the best we have. However Taleb’s contention is that financial models based on the normal distribution are so badly flawed that not only do they not miss the really significant risks (the big ones) by assuming them away, they go further and actually create them. Once we have constructed a model to reflect the risk, people trade and protect themselves on the basis that the model’s predictions actually reflect reality and that large, uncorrelated moves are exceptionally rare. This leads them into taking on additional risk, and leaves them worse exposed in the case of exceptionally large moves.

The Black Swan

My original intention was to write a longish post discussing the main ideas. Rather I will do an overall impression here, and then do a few follow ups to talk about what I think are the interesting points.

All up I found The Black Swan an very interesting read. If you’ve read fooled by randomness then some of the arguments will have already appeared but certainly not all. The Black Swan is concerned with wild randomness, the randomness that according to Taleb dominates modern society. Fool by randomness largely concerns the randomness of games, what Taleb would describe as mild randomness. So mild that it is hardly random at all.

As we are reminded in several places in the book Taleb made his fuck off money (i.e. enough money to be comfortably independent even if not mega wealthy) in the 87 stockmarket crash betting on the fact that the market under-appreciated Black Swans. He doesn’t need to fawn to the establishment be it economics, philosophy or publishing and we get this tone all through the book.

While the book touches on finance applications its hardly a finance book. Rather Taleb considers it to be as much a work of Epistomology. His chief concern is with what he calls epistemic arrogance. He hates those who profess to know more than they really do know, be they economic modellers, historians or political scientists, but his chief enemy is the normal distribution which is described as the Great Intellectual Fraud. To use normal distributions in fields where it is easy to show are not normal (such as finance), is both fraudulent and causes Black Swans.

Its strong stuff which he backs with examples, logic and the findings of behavioural finance and similar studies. Its also fairly convincing for the most part. Its true we don’t need sophisticated statistical studies to find whether market moves are normally distributed. We only need the fact that we get 1 in 10,000 year events (as modelled by a normal distribution) occurring every few years. Risk managers who run such models (such as me) are either ignorant fools, who actually believe wrongly in what they do, or deceptive frauds who know better but persist in fooling others for the money. I’d contend I’m neither but I’ll discuss later on.

Anyway I leave the rest to further discussion. I particularly want to mention the Narrative Fallacy, which I have actually mentioned before, the problem of prediction as well as whether I really am a fraud, idiot, or neither.

Raising the Dead

A long, long time ago in a blogosphere far far away I used to write a blog on whatever interested me. Then I lost interest and took up World of Warcraft. All of this time I have been intending to get back to it but never quite managed it. Then we had our second bub, and so now that I am busier than ever I intend to try to resurrect it again. Hopefully it won’t be short lived, but rather an ongoing if only a weekly posting affair.

Anyhow the blogpost I was always intending to write next was a book review of The Black Swan by Nassim Taleb, as it was what I was reading at the time and touches on a number of topics I am interested in. So last week I began re-reading it with an eye to finally discussing it. Hopefully the reading and review will be done in a few days.

The Narrative Fallacy in Weather

Looking back at my most recent posts on rainfall I note that they were motivated at a reaction to what Nassim Taleb describes as the Narrative Fallacy, the desire people have to make a post-hoc story to so all events relate with a strict cause and effect. Journalists (amongsts others) love to try and join dots and so are typical propagators of this nonsense. Recently they have been keen to link everything weather related like the drought, to global warming.

Should I worry about this though? Ultimately I think we should be acting to mitigate AGW, so is it best for us not to be criticising our “side”? I think that this kind of partisanship is amongst the worst things that can be done. Claims that are unsupported by evidence will ultimately undermine the very case they are trying to support. They are in effect crying wolf.

The stupidity of course isn’t limited to AGW alarmists, with many if not more of the skeptics running the its a cold day, it can’t be warming line. Some of these are taking the piss but not all of them. Some seem to believe its a refutation.

Ultimately it would do well if people were to remember that a particular instance is usually not a good representation of the average. Its ridiculous to ascribe meaning to every single occurrence, and even a couple of years don’t really do much to hundred year trends. Climate is long term average weather. Year to year it varies, and we are looking at long term trends not day to day heatwaves or even year to year droughts.

More on rainfall

For those unconvinced about my previous claim that, according to the BOM data, there doesn’t appear to be a drying trend on the east coast and think it might be an artifice of too broad averaging, I note that they should have a look at the trend maps found here. For example the rainfall trends from from 1900 to 2006.

This shows few regions which have shown a definite drying trend over the period. One clear exception being the south west of WA, which does show a drying trend over pretty much anytime period you pick. In addition it appears that some slight trend could be evident in mid to northern Queensland.

Go and have a look at some of the different time periods on their site, the lack of a clear trend for much of eastern Australia isn’t just for the period 1900-2006. In case any one’s concerned that its because we started the comparison in a drought. Take 1910-2006, 1920-2006 or 1930-2006 and you get something similar.

Indeed its not unless you compare with the 1950’s to 1970’s that you get a really marked decrease in rainfall in most of Eastern Australia.

Drying up or typical climatic variation?

I noticed this article Big dry: no one knows why in the SMH this week, and in particular the claim that eastern Australia’s rainfall had dried up since the 1950’s. The report seems to be related to this press release announcing a new centre at UNSW studying climate change.

A priority for the new centre is to better understand the mystery of why Australia’s most populated region, the continent’s east coast, has suffered such major declines in rainfall in recent decades.

“We recently had a round-table of Australia’s leading climate-change researchers and this emerged as the biggest unknown issue and, of course, it seriously affects the largest concentration of people stretching right down the coast from Cairns to Melbourne,” Professor England said.

So I had a look again at the rainfall graphs and time series provided by the Bureau of Meteorology for eastern Australia and there is no doubt that we have been receiving less rainfall in recent decades than the 1950’s. However it also seems that we are receiving more rainfall than in the first half of the 20th century, so the question seems to me to be as much why did rainfall rise in the 50’s as why has in fallen in the 80’s and 90’s.

More on the rental crisis hype

One of my pet hates is the “research” turned out by real estate industry groups which is always, shall we say, very optimistic whether talking about rental increases or price increases. So I was amused when a friend pointed out this article in the paper on the weekend.

Rather than showing a surge in rents, data from the New South Wales Government shows there was no growth in apartment rental bonds in the three months to December.
In Sydney’s inner suburbs, three-bedroom houses recorded growth of 3.6 per cent but two-bedroom houses also showed zero growth.

Of course this is for the end of last year rather than the start of this year, but it puts the predictions in perspective. As I stated before its not surprising that the real estate industry publishes favourable research to their interests. What is disappointing is that the media outlets, papers, television, radio commercial as well as ABC and SBS almost uniformly regurgitate this uncritically.

Media watch has a more comprehensive hosing down of media gullibility in the face of real estate industry “research”. A fine collection of misquoted tenants, and exagerated stories some being pushed by political parties.

Are epidemological studies almost worthless?

Browsing through Nassim Nicholas Taleb’s diary I noticed this quote:

At the AAAS conference in San Francisco I was a discutant of session in which John Ioannidis showed that 4 out of 5 epidemiological “statistically significant” studies fail to replicate in controlled experiments.

NNT crows that this is what he has already come to describe as the narrative fallacy. If you look hard enough at enough data, you will see a pattern emerge.

Anyway I have looked up John Ioannidis’s research and found this interesting paper Why Most Published Research Findings Are False
, which unfortunately I haven’t had the chance to read in full.

The outline of his idea is simple enough, if you look at enough data (particularly small data sets) you will find statistically significant relationships. The part I thought was interesting was this.

As has been shown previously, the probability that a research finding is indeed true depends on the prior probability of it being true (before doing the study), the statistical power of the study, and the level of statistical significance.

Which is kind of obvious. If I correlate enough astrological data with some disease I will inevitably find some correlation, but because the prior probability of it being true is essentially zero there is still very little chance of the study being true.