Google opens discussion of testing ML algorithms

I’ve been talking about it for a while: testing machine learning algorithms can be very tricky. Google is now talking about this. This is one of the lesser studied areas of ML that will become a huge deal.

Lessons on probability from the credit crisis

I began to compose a reply to a comment on this blog by Will Dwinnell but it turned into this post. Here, I summarize how the credit crisis results from an age old problem faced by statisticians: properly mixing gut instinct with statistical methods.

Our current situation results from (at least) two kinds of predictions that went bad. First there came the predictions about the percents of default on certain debts. These predictions were used to calculate fair values of securities based on that debt, meaning that if you are genuinely sure about the percent of defaults, you can pretty much predict how much will be paid back over the life of the loans.

Those models of the rate of default had big flaws, but if you look at the data, you see that even the unprecedented rise in mortgage defaults could not possibly explain the fall in prices of all mortgage debt securities. There is plenty of debt that is in little danger of default, but has dropped in value by vast amounts. Sure, some debt securities will not be paid back because there were more defaults than predicted. But how come the whole debt market has gone crazy? I mean, this happens all the time with stocks – some stocks that once looked good turn out to have been far overvalued, and that can lead to big price swings, but generally not total collapse.

So we turn to the next part of the story: borrowing against debt securities. Once you own a security, you can use it as collateral to borrow money. This is called margin in the stock world. If you have a brokerage account with margin, you will notice that if you have $1 in stock, you will not be able to borrow $1 using this stock as collateral. Rather, you will be able to borrow much less than $1. Why is this? It’s because the lender wants to be absolutely sure that in the worst case, they will be able to recover all their money by forcing you to sell the stock. And if they lent you $1 but the stock is now worth $.80, and then you fail to pay, they will not have enough collateral to recover their money. So they have models that tell them how much the stock is likely to drop in the worst case. And they will lend you just enough so that if the stock drops according to their model, and then you default, they still have enough collateral to recover all their money.

If you have borrowed against a stock and that stock drops far enough, the lender will come around and force you to immediately pay back the difference between what you owe and what their model shows your stock to be worth in the worst case. This is called a margin call.

Institutions made a similar arrangement in terms of borrowing against debt securities. They own the security, so they can use it as collateral for a loan. Then they use that borrowed money… to buy more securities. Then they borrow against those. And so on – this is what they call leverage. Now the lenders will require that the borrower has, as collateral, a certain percent of the value of the loan. When the value of their collateral drops below this percent, the borrower will get a margin call and be forced to pay back some of this loan.

Now let’s say that I’ve got $1 of a stock and I borrow $.80 against it. I then buy another $.80 of that stock and then I borrow $.64 against that. And I keep going until I can’t borrow any more. Here I’m borrowing 80% of the value at each step, so I first borrow $.80, then $.64, then $.512 and a little figuring will show that from my original $1 I can borrow over $3.50. But I own $4.50 in collateral (the stock), so even if it drops in value by 20% I will still have just enough collateral to pay back my $3.50 in loans.

Now what if I borrow 90% at each step. Hmm, then I can borrow over $8. And if I borrow 95% of the value at each step, then I’m at $17 in loans from my original $1. Now to start with, I have enough collateral to cover these loans totaling 17 times my original money. But if this stock drops in value together as little as 5.5%, then my collateral is worth less than what I borrowed (if I borrowed 95% of the value at every step). Now if I default on that loan – then the lenders are out money. And that money can never be recovered – the underlying assets can’t be sold to recover the money and it has simply evaporated. Unless the lender is willing to take the stock in payment and wait for it to go back up in value, a risky proposition at best.

And so, you can see that there is a balancing act. On one hand, there is a level of risk that the lender should be willing to take. Maybe they are confident that my stock won’t drop 20%, 30%, 50% – but there is some level at which they are willing to make the loan. Actually, that’s not true – lenders often stop offering margin on the most volatile stocks – but let’s say we’re talking about a stock with a solid record of performance.

On the other hand, it would be insane to allow me to do this at 95%. Even if I show that, for the past 20 years, my stock has never dropped by 5.5% in a given year, any normal person would still know, in the back of their mind, that a 5.5% drop can happen with just a little market hiccup. No matter how much I show you that I have found a stock that is historically rock solid, you would still be an idiot to rely very heavily on the prediction that my stock will not drop 5.5% in the future.

But that is exactly what the finance firms did. They predicted the values of many debt securities to be so stable that they could count them essentially as cash in terms of risk (the chance of decreasing in value). And if I have $1 in cash, well you’d feel pretty safe lending me $.98 using that $1 as collateral. Now, we do the above calculation with me borrowing 98% of the value at every step. I can borrow $44 based on that $1 initial capital. If the securities keep going up, everything is just great – money is flowing like water. But a drop of 5% and now I am underwater with not enough collateral to pay back a default. And after a drop of 15%, it’s hopeless and if I default, the lenders stand to lose 13% of their money. And let’s say that instead of $1 in starting capital, I had $1 billion. After a 15% drop in price, my default on debt evaporates over $5 billion in borrowed money (in addition to my original capital).

So then the chain begins: Once one kind of debt security started to go down (sub-prime),  the selling started. And when the selling started, prices went down and firms began to get margin calls – and that resulted in more selling, and more margin calls, until firms were forced to sell debt that wasn’t in danger of default problems. And that is what no one expected. All this “safe” debt was suddenly falling in value simply because there were many sellers and zero buyers in the market. Firms began to get margin calls on debt that had no reason to drop in value – debt that is still perfectly safe to this day. The margin calls began to multiply, driving the debt value further down. Since the market was on its way down, this selling to meet margin calls was losing money by the bucket full.

And then it gets really bad, since many institutions rely on daily, weekly and monthly borrowing to operate, and it became uncertain whether some would have the collateral to back up that borrowing (especially since the stock prices were tanking as well, so the firms couldn’t even use their own stock for collateral). All it takes is one week where they can’t borrow money and they are out of business and defaulting on their loans, destroying money now by the bail. And that prospect makes everyone very nervous to lend to anyone.

But it was a house of cards, ready to fall, ready for that chain reaction to begin – because of this fundamental assumption that some debt securities could not drop in value. Even though the price of those securities is determined just like any other price – by what others are willing to pay. None of the models predicting the value of those debt securities took into account the fact that, at a fundamental level, all it takes to make the securities fall in value is a market with more sellers than buyers. It doesn’t matter what the “fair value” or long term return of the securities is – today they are worth exactly what you can sell them for.

Will says “They should have hired a better class of statistician.” But honestly, what would a statistician have done? Everyone knew that, in theory, the debt securities could drop in value. The executives don’t like to talk about it, but anyone in the industry who (a) understands the models and (b) is being honest, will tell you that the idea crossed their mind at some point in the past few years. The folks making the decisions made the classic and human mistake of misinterpreting a probabilistic statement (that the securities were unlikely to drop in value) as a fact.

Imagine the skeptical statistician in this scenario, speaking up in a big meeting: “hey everyone, you know that just because these securities have a stable value now, doesn’t mean they can’t go down.” So the room grows silent, all eyes are on this one realistic person and some executive asks: “Ok… tell me the probability that the securities will drop in value this year.” Remembering his (or her) training, the statistician says honestly: “Well it’s never happened, so we have no evidence with which to predict. So no one really knows. Even if we had a model, without any evidence we have no way to know how good it is.”

Now what choices does the executive have here? The statistician can’t quantify the risk, but it’s definitely there. They are back to that gut feeling – should we buy into this unquantifiable risk or not? Some said no. Many said yes, especially after they saw how well it was working for the folks who jumped on the bandwagon early. In order to prevent the situation, many executives would have had to make that gut call to say “I don’t care what history says, it’s just not possible that these debt securities are as safe as cash. And I’m willing to forgo a whole lot of potential profit based on this belief.” The statistician, no matter how professional they are, can’t make a call like that (unless they’re also the executive, I suppose).

What’s required is a fundamental shift in thinking about probability in business. Maybe as part of bailouts, firms should be forced to introduce mandatory training on ethical and logical interpretations of probability and chance.

What real-time analytics do customers really want?

Paul Vincent, of TIBCO (blog is here), has been kind enough to respond to my recent post about the very strange discussion surrounding analytics in Event Processing (EP) products.

Paul says:

I’m not sure there is a huge demand for traditional SAS-type analytics though – very specialized skills reliant on expensive data warehouses etc. It will be interesting to see how/if “event based analytics” improve the score, so to speak.

This is very interesting to someone like me, who works with “analytics” every day. When I read people talking about how everyone needs “analytics” or “advanced analytics,” I begin to wonder whether most  organizations (which are potential customers for an EP product) have the infrastructure (both skill set and hardware) to support this kind of thing. There are clearly various levels of capability in analytics. I have to imagine that most organizations are on the lower end of this capability and have a long and expensive road ahead to improve.

So what real-time analytics are customers actually asking for? To the extent that this is not proprietary marketing data, I would love to hear what an organization like TIBCO is seeing in this space.

There are plenty of applied math and statistics people out there who would love to apply their skills in new and exciting ways. Vendors that want to become active in analytics would do well to engage with this crowd by providing concrete information about real life use of analytics in Event Processing.