A few words on modeling
There are as many models in the world as there are agendas. This is a particularly difficult lesson for those in the planning and policy fields who like to believe that models – particularly ones that give them the answers they want – can help them make precise predictions. Predictions are tricky, especially about the future.
Years ago, I sat through a session on the software CommunityViz. It purportedly allowed someone to project all kinds of impacts that would happen on a specific piece of property once it was developed. I found the entire exercise absurd, especially as the presenter ran through the long list of variables to be entered into the program. This was just silly.
As Nate Silver discusses in his book The Signal and the Noise, there are two common ways to model complex systems, both of which have tremendous drawbacks. The first is to simplify the system down to a few key variables. That makes the model easy to run, but you lose a lot of the nuance and almost all of the feedback that happens in complex systems. The further you project out into time, the more divorced from reality this brittle analysis becomes.
The second approach is to include as many variables as possible in an attempt to fully model each aspect of the system. The complication of this approach is that, while it acknowledges the complexity of the system, small changes in the initial conditions of the model have dramatic – and random – impacts on the results the further you project out in time. Make a tiny tweak to one non-descript variable and you could get wildly different results.
Weather models use the second approach and they attempt to overcome the feedback loop problem by running the model thousands of times with different initial variables and then using the results to create a range of likelihoods and probabilities. This has made weather forecasters really, really good at predicting the weather for the next 24 hours. While they occasionally get it wrong, it’s not that often. This approach has also tremendously improved the ability to predict weather 48 and 72 hours in advance, but there is still a lot of variability.
Even with all the complex models run with thousands of iterations, the best predictor of what the weather will be like a week from today is the historical average. In other words, for complex systems, what has happened is a better predictor of what will happen than any model we can create. Queue up the Patron Saint of Strong Towns Thinking Nassim Taleb.
At one point during our work with Lafayette, a staff member asked us what the output of our model was. Essentially, they wanted to know whether the tool we were building was going to tell them what to do? That seemed to be the hope, anyway. My answer was that it would not tell them what to do but that it would help them think. From the reaction, that was a deeply dissatisfying answer.
I also had a brief exchange with a staff member who said he did traffic modeling. He said the model gave them the “best guess” on what would happen in the future. My response was that it simply allowed him to be “precisely inaccurate, with confidence in the precision.” I think we’re going to let Joe and Josh at Urban 3 handle the staff interactions from this point forward.
The tool we are building in Lafayette does not fall into the trap of trying to predict the future. Our team has too much of a good sense of humility – and an understanding of complexity – to pretend we have that capability. What we are doing is getting the most accurate accounting of the financial implications of past land use patterns. We’re looking back, not forward.
So what good is a model if it doesn’t tell you what to do? What does it mean to have to think?
Going back to Nassim Taleb, the way you probe uncertainty is by taking many small steps. And trust me; there is a ton of uncertainty. No human civilization has so radically transformed a continent in so short a period of time. No nation has ever systematically put its cities in such a precarious financial position through state and federal policy. No economy of our size has ever had an economic model that impoverished its places by trading wealth creation for growth. We’re in a place that nobody has ever been. Every American city. The purpose of this model is to understand where we are at, what we've done and to serve as a baseline for measuring the results of our future probing of uncertainty..
For me, the obvious rational thing to do when looking at a model showing you parts of your city that are financially productive while the vast majority of the city’s land area bleeds red ink would be to do more of the former and less of the latter. Start investing in your profitable divisions. Stop investing in your unprofitable divisions. That’s me. I’m a human with all the normal human failings. I could be wrong and very likely am.
Joe Minicozzi would end our presentations in Lafayette with a Polish proverb: “This is not my circus and these are not our monkeys.” Lafayette is not my circus. The people there are not my monkeys. Lafayette is going to solve their problems in a way that works for Lafayette. I respect that. We’re just trying to give them the most complete set of tools possible so that their decisions are informed by reality.
Also in this series:
- Lafayette (Introduction to the analysis tool)
- A few words on modeling
- Up Next: A look at road and street networks