Structural errors in forecasting

Forecasting is notoriously bad.

S-Curve and the Danger of Extrapolation (from The Transportation Experience, Second Edition, (Garrison and Levinson (2014)).
S-Curve and the Danger of Extrapolation (from The Transportation Experience, Second Edition, (Garrison and Levinson (2014)).

There are many reasons for this, but one is structural, failure to understand the life-cycle dynamics.  The reason for overshoot and undershoot can be understood by visiting the S-curve. Assumed forecasts are made by extrapolating previous results, which is how many businesses and investors and government agencies operate, as shown in the figure. In early years (Birthing and Early Growth) the rate of growth each year is greater than the previous year. Someone extrapolating from history will undershoot actual growth. But in late growth and maturity, growth is slower than the previous year. Someone extrapolating from history will overshoot actual growth.

Extrapolation models are common in transportation, see e.g. Angie Schmitt’s Transpo Agencies Are Terrible at Predicting Traffic Levels. These are used for statewide modeling in many places. Such forecasting methods (assume growth continues at a fixed percent) is embedded in some textbooks, especially for instance, in pavement design.

Combined traffic projections from state and regional transportation agencies (the colored lines) have been wildly off the mark (the black line shows real traffic levels) for more than a decade. Image: SSTI
Combined traffic projections from state and regional transportation agencies (the colored lines) have been wildly off the mark (the black line shows real traffic levels) for more than a decade. Image: SSTI

Urban transportation planning models are better in some ways, in that they include multiple factors. Unfortunately, these models are based on rates at a single point in time. Thus they assume the function that describes the behavior is fixed, only exogenous (input) factors such as demographics, land use, networks, and policies are allowed to vary. Even when multiple years of data are available, such models are typically only estimated on the most recent survey, rather than on trends or changes. The underlying behavior is not permitted to change, only what it responds to. Yet we now have evidence that some underlying preferences do change over time. It’s not simply a matter of getting the demographics or incomes correct. For instance from the 1960s to the 1990s female labor force participation increased. Thus the number of work trips and non-work trips (substituting out-of-home for in-home production) both increased in that period. But that increase has played itself out. Thus the increases it was associated with have peaked. This reflected changing preferences. While hindsight is 20/20, I don’t know if underlying preferences can be modeled accurately prospectively (I am doubtful), but I do know failure to account for them will lead to model inaccuracies.

What changes are going on now that are not considered in travel demand forecasting? A brief (and very incomplete) list below:

  • Vehicle technology shifts (driverless vehicles)
  • Preference shifts among young travelers
  • Changing driver licensing requirements
  • Vehicle ownership vs on-demand vehicle rental (car “sharing”)
  • Telecommunication increasing substitution for work, shop, and social travel
  • Telecommunication complementarity for work, shop, and social travel

None of this is easy to model, certainly not within the existing framework of urban transportation planning models, even more modern activity-based models. In many ways it is easier to do macroscopic than microscopic forecasting. The question is, if some kinds of forecasting are impossible (I can forecast traffic pretty accurately two weeks from today, but not the first Tuesday of 2044), why do we do it? Is there a human-need to fill the void of future uncertainty with authoritative assertions?

Speculating about the future is useful, it opens up pathways. Developing scenarios is useful, it challenges assumptions. Thinking about the lifecycle process and markets helps frame the possible, the plausible, and the likely. Studying history (and past forecasting methods and errors) provides but humility and insight. Visions (and alternative competing visions) help establish what we want. Developing a communal hallucination can organize individual activities to become the ideal (or nightmarish) self-fulfilling or self-negating forecast.  Planning needs more methods for thinking about the future than single point forecasting.

2 thoughts on “Structural errors in forecasting”

  1. You list a lot of different ways of thinking about the future, but then say we need more. Are you saying those methods you list aren’t used? I would argue they are used a lot in community and development planning (visioning/scenario planning), but not often used in transportation planning. VIsions of multimodalism are sacrified on the alter of future traffic space needs.

    I don’t think we need more ways of thinking about the future (there are lots of options for dialog, as you recognize), I think we need a planning/design/political culture that allows the outcome of a process to be something other than just the results of space needed based on a forecasting exercise. We don’t need new tools, we just need decision-makers to lend legitimacy to tools currently not deemed scientific enough for use in the field.

Comments are closed.