Build a better model

We humans are eager to believe, in the face of overwhelming evidence to the contrary, that large changes in the trajectory of human progress are infrequent. There is comfort in pretending that we are far more in control of things than we really are. We don't understand, or perhaps respect, the mystery and complexity of the world we operate in.

If you are in Pennsylvania and would like to have the Strong Towns message brought to your community, we have an ongoing fundraiser to help us visit your state and hold 8 to 10 Curbside Chats. Please consider supporting this effort and pass it along to those you know in PA. We'd love to bring this message back to the Keystone State and change the conversation on growth statewide.

Last week I blogged about transportation projections and how it really is silly that we rely so heavily on a system that we know gives us bad advice. In The Projections Fallacy, I pointed out that we don't need better projections but a model of growth and development that is robust to modeling error. This seemed fairly self evident to me; if something is not working, find something else that you think will.

I was not prepared for the number of people that had a difficult time wrapping their mind around this. There were three main counterarguments put forth to my premise. First, there were some that agreed that our models were bad, but they argued that what was needed is a better model. I'm going to address that belief today. The other two -- that bad models are better than no models and that, flawed as it may be, I have no better solution -- I will address later this week.

Earlier this month I read When Genius Failed: The Rise and Fall of Long-Term Capital Management by Roger Lowenstein. Long-Term Capital Management (LTCM) was a famous hedge fund that experienced unprecedented success before going bust in spectacular fashion in 1997, necessitating a federally-organized bailout. The story should humble anyone who does high stakes projections or modeling for a living.

The idea behind LTCM was an equation known as Black Scholes. It is a model for pricing options that, through use of statistics and normal distributions, allowed hedge funds to make "low risk" bets on the movement of markets, offsetting those bets with a "hedge" to limit the amount of loss. In theory, it is an amazingly powerful model, brilliant enough to earn a Nobel Prize in Economics. In practice, it failed spectacularly and almost brought down the entire global financial system.

It is not that the model was wrong -- much of what they were doing had been back tested and, given infinitely deep pockets, infinite time and markets that operated within their modeled range, they may have been able to weather the storm -- but more that they were incomplete. They assumed rational market behavior; that others working in an efficient market would not leave money on the table, opt for the lower performing (but safer) bet, voluntarily take a loss, default on debt obligations and a myriad of other things that happened in succession. Their models said this confluence of events should happen only once in the history of four universes. Obviously, that was not quite accurate. Their downfall came from market responses that could not be modeled.

In comparison to the modeling used by the quants on Wall Street, traffic modeling is so simplistic as to be almost laughable. I've seen more just straight linear projections than anything else, essentially taking a ruler and drawing a trend line out from past data. This is done digitally using a spreadsheet to give it a veneer of complexity. It is anything but.

More advanced models actually examine the behavior of traffic given certain changes to the system. If one road is opened up or widened, cars will opt for that route, reducing traffic in a different area. These models are more complex with many more variables. Like any model, the greater the number of variables the increase in the likelihood of modeling error and the greater likelihood of an outlier event.

So the idea of building a better model is being tried and there are many people -- in the financial world and in the traffic modeling world -- that believe it is possible. If you back test the data far enough, fine tune your variables, try to take into account all the intangibles you can think of, you can get a model that is reasonably close.

Until it isn't.

Let's go back to 1925 and apply this build-a-better model logic. Pretend we worked for a local trolley company. We needed to project how much to invest in new growth (overlooking for a minute that this is not how the trolley systems were built -- build-it-and-they-will-come is a modern phenomenon). We look back over the past 50 years and see a country that has been urbanizing through the Second Industrial Revolution, one that is in the midst of a great boom that appears to have no end, one where the demand for the trolley system projects as strong.

We can pull in any variable or intangible that we could reasonably identify at the time and we would not have been able to predict the Great Depression, the Second World War, suburbanization, the advent of buses, the rise of the automobile, the Interstate Highway Act and the forces that would relegate the trolley industry to a local novelty. And that is just in fifty years.

We're making investments today that we expect to be around at least that long, with projections of the future that extend out decades. I don't care how good the model is, no model will accurately predict the future. Nobody can tell you with any type of certainty what will happen with gas prices, technology innovations, consumer preferences, war and disease and any other myriad of game-changing variables that humanity has faced throughout history. And we're in a period of history where change is accelerating, something that modelers are totally blind to.

Now, none of this would matter if the bets we were making were small. You can bet a small amount of money on the probability of things happening in the distant future and it really doesn't matter if you are wrong. Unfortunately, we're not making little bets. We're making massive bets, not only in terms of the money we spend today but in the cumulative repercussions of the approach over time. That's why we do modeling; this is all really important because there is a lot at stake.

So if we can't build a better model -- and we know that to be a self-evident truth -- why do we do it? That is a psychological question that I'm not fully qualified to answer. My suspicion is that we humans are eager to believe, in the face of overwhelming evidence to the contrary, that large changes -- positive or negative -- in the trajectory of human progress are infrequent. There is comfort in pretending that we are far more in control of things than we really are. We don't understand, or perhaps respect, the mystery and complexity of the world we operate in.

Our models simply don't work and we can't build a better model. We need to face those facts, embrace them fully and do something positive with that knowledge.

Later this week I will address the two other contrarian responses I received, specifically, that bad models are better than no models at all and, if I'm so smart, what would I do differently in a world without models. Check back with the Strong Towns Blog for that.

If you want to chat with Chuck and many others about implementing a Strong Towns approach in your community, consider joining the Strong Towns Network. The Strong Towns Network is a social platform for those working to make their community a strong town. Get expert assistance, consult your peers and stay current on the latest techniques and analysis.