Category Archives: evaluation

Evaluation in a Time of Uncertainty

I was recently in Copenhagen, where I gave a keynote talk at the UNITE Conference (Uncertainty in Transport Evaluation) at DTU. My thanks to the organizers for inviting me. (I really liked Copenhagen, more on that in a later post).

My talk was titled: Evaluation in a Time of Uncertainty

Abstract: We live in a world where the future is increasingly unpredictable. How should we evaluate transportation investments? Should we even try to evaluate investments in advance? What do we know about what people value? This talk will consider directions in transportation evaluation, and suggestions for better decision-making given uncertainty.

The slide deck is here.

The essential tension in my presentation is the hypothesis that essentially “Long-term forecasting is impossible” vs. my argument that forecasting can only be improved if we “Make forecasters responsible for forecasts.”

Are EU Cohesion Policy funds well spent on roads?

An interesting report by the European Court of Auditors (who knew Europe had a “court of auditors”?): Are EU Cohesion Policy funds well spent on roads?.

The EU spends a lot of money transferring resources between member states (cohesion funds). Much of that is spent on roads (a surprisingly large amount in Poland). This report compares proposals with results. Most roads under-performed traffic forecasts.

From the report:

Observations

  • The road projects audited partly achieved the intended results
  • … but their impact on economic development could not be assessed …
  • … and most of them delivered less than the planned return on investment …
  • … while clearly improving road safety and helping to save travelling time
  • Enhanced transport capacity could have been achieved at lower cost
  • The road projects mainly followed the most economical road alignment
  • Traffic forecasts were in the majority of cases not in line with actual road use
  • Costs per user vary significantly
  • Express roads were clearly less costly to build than motorways
  • The best possible price was not obtained for all audited projects
  • Significant cost differences for road accessories and bridge decks
  • Cost overruns of more than 20 % for 11 projects

(Via KA)

Transportation Benefits Too Little

Highway_Chart_Graph_1
InterstateProductivity We are not building much new transportation in the US not just because the costs are too high, but because the benefits are too low.
My 2011 post Transportation Costs Too Much made the claim that new projects were too expensive, and listed a series of hypotheses as to why that might be the case. The whole list is collected here.

When we were much younger as a nation, say 1956, and growing fast, with relatively poor connectivity, you could do almost anything and it would have a benefit/cost ratio above 1. Very little of the interstate has been reversed. But the productivity of new investments has declined over time (as shown by Nadiri, M.I. and Mamuneas, T.P. (1996) Contribution of Highway Capital to Industry and National Productivity Growth Federal Highway Administration. Office of Policy Development.)

The first 3 data points on the graph are rates of return from their work, the last is an extrapolation. This is illustrative only,but someone should update their study.

Updated Jan 10, 2012. Matt Logan points me to a study that shows the authors updated their study in 2006 (Mamuneas, T and Nadiri, M.I. (2006) Production, Consumption and the Rates of Return to Highway Infrastructure Capital)

They report net rates of return are as follows (Table 5)

Period Net Rate of Return
1949-1950 : 0.554
1960-1969 : 0.480
1970-1979 : 0.298
1980-1989 : 0.212
1990-2000 : 0.136

My guess is that the 2000s are significantly lower.

A value of approximately zero returns in recent years is consistent with recent work by Noland and his colleagues. This is what you expect with life-cycle theory, and it applies well to existing modes.
The reason we don’t draw new lines on the map is that the net benefits are not perceived to outweigh the net costs. The costs have risen as land has gotten scarcer and for all the reasons linked above, and the benefits of additional lines drop. Here we explore the second point.

  1. Diminishing marginal returns to new roads due to diminishing distance reductions as the network is increasingly complete.
    TTE-F27-4
    This is a spatial argument, illustrated in the Third Figure (Source: Figure 27.4 from The Transportation Experience). Imagine you have a network with a 1 mile grid (typical for much of the US). With development of farms, you add roads in between, say at 1/2 mile spacing, this reduces travel costs some, as people don’t need to back-track as much, and this might be a significant share of the distance for short trips. At most, you are saving someone 1 mile (1/2 mile at the beginning of the trip, and 1/2 mile at the end of the trip). Now add additional links to diminish spacing to 1/4, This requires twice as many links, but only reduces travel costs by at most 1/4 mile at each end of the trip (1/2 mile total). New links do less and less to reduce distances. Distances, along with speed, determine travel time.
  2. Diminishing speed savings for new roads
    Recognition that transport links become congested, and thus slower over time due to:

      1. induced demand
      2. induced development
      3. induced driveways and interchanges, which increase friction on roadways and slow them down over time. While access management addresses this on arterials, very few interchanges are removed to speed freeways, they are almost irreversible

    Thus we have to discount the opening year forecast travel time benefits to account for the fact that the travel time savings of any expansion will in part (if not in whole) be eaten up by more travel. While this is not of itself a bad (travel is a measure of people doing something that they value), it is not perceived to be a good thing (because it creates congestion and pollution externalities which existing travelers bear).
    There is some compensation for induced demand and induced development, as more travelers may lead to more service (induced expansion of existing links and construction of new links), but this is a longer term process, and only works up to a point (and more easily with transit services than roads).

  3. Diminishing demands for new roads
    1. peak travel
    2. slowed population growth, particularly in un-roaded or under-roaded exurban areas.

    Implies the benefits from new construction are falling.

  4. Disillusionment with the quality of travel and transportation facilities
    The saying goes: Fast, Cheap, or Good, pick any two. It sometimes seems we have none of these. We know transport costs too much. We know transport takes too long to build. Surely if we are paying a premium, it should be of high quality. It seems not. Maybe the few facilities which are brand new, expensive, and took too long to build are, but the rest of it is not. Even then, I have doubts. Big Dig Ceiling Collapse is one of many notable failures, this one all the worse because the facility was new and expensive.An earlier post, It Just Makes You Feel Poor describes the poor quality environment around some local new LRT stations.

The arguments about benefits apply to mature systems in general, and are modally independent. If a new mode comes about that is better than existing modes for some market, it has lots of room to run while providing benefits in excess of costs. But if the new mode is inferior to existing modes, than it has little in the way of prospects.
What constitutes better or inferior is in the eyes of the customer. Certainly customers care about time and price, but they do also consider quality, and may be willing to sacrifice one for the other. But as experience with zeppelins and cruise ships (and conventional intercity rail in most of the US) show, high comfort at slow speed will not trump low comfort at high speed.
This also has implications for the Preservation vs. Expansion argument (Fix It First). If the old projects had high benefit/cost ratios, and without proper maintenance are at risk of disappearing (either failing catastrophically, or being closed to prevent such an end), it is incumbent to maintain them. In fact, given that land uses and resultant activity patterns that have evolved around transportation networks, the benefit/cost ratio of preserving those links is probably much higher than it was originally, and certainly higher than the B/C of new links.

Peak congestion

Percentage of Congested Miles on Twin Cities Freeway System
Percentage of  Miles on Twin Cities Urban Freeway System Congested

MinnPost Twin Cities traffic congestion slowed a tiny bit in 2011. Though MinnPost doesn’t seem to believe it, they should be following the discussion of Peak Traffic. The measure is percentage of freeway miles flowing at 45MPH or less. The full report is here.

Accessibility Dynamics and Location Premia: Do Land Values Follow Accessibility Changes?

Twin Cities Regional Land Use Map
Twin Cities Regional Land Use Map

Working paper:

The structure of transportation networks and the patterns of accessibility they give rise to are an important
determinant of land prices, and hence urban spatial structure. While there is ample evidence on the cross-sectional
relationship between location and land value (usually measured from the value of improved property), there is much
less evidence available on the changes in this relationship over time, especially where location is represented using
a disaggregate measure of urban accessibility. This paper provides evidence of this dynamic relationship using data
on home sales in the Minneapolis-St. Paul (MN) metropolitan area, coupled with disaggregate measures of urban
accessibility for multiple modes, for the period from 2000 to 2005. Our investigation seeks to track the effects of
marginal changes in accessibility over time, as opposed to static, cross-sectional relationships, by using an
unconventional approach in which the unit of observation is a “representative house” for each transportation
analysis zone in the region. This approach allows us to control for changes in structural attributes of houses over
time, while also isolating the effect of changes in accessibility levels. Results of this approach are compared to a
cross-sectional model using the same variables for a single year to illustrate important differences. These
differences are discussed in terms of their implications for practitioners and for further investigations of the
relationship between transportation, location and land value.

The abstract buries the lede. The average cross-section effect (that is the long – term accessibility) shows access is related to land value. The marginal effect (that is the change in land value due to the change in accessibility) is much, much weaker if not nonexistent (depending on the model). This suggests there are diminishing returns to access in mature networks.

The Demand Curve for Life

At the 2009 International Transport Economics Conference Bruno De Borger, Erik Verhoef, and I were having dinner, Erik raised an interesting question about the use of statistical value of life in evaluation studies. Suppose there is a road improvement which will save 1 life per year, reducing the number of fatalities from 2 to 1 per year (out of 1000 people using the road). Assume all travelers are identical. What value of life should be used in the analysis?
Normally, we would do the equivalent of trying to compute for each traveler what is the willingness to pay for a 50% reduction in the chance of death by driving (from 2 in 1000 to 1 in 1000), and multiply that by the 1000 people whose chance of dying is reduced.
An alternative approach is to figure out the willingness to pay for the driver whose life is saved. So how much would you pay to avoid dying (with certainty) (i.e. what is your Willingness to Pay)? The answer to the first question is usually taken to be all of your resources (you would pay you everything so I won’t kill you).
Alternatively how much can I pay you to allow you to let me kill you (Willingness to Accept)? The answer to this second question is: I would have to pay you an infinite amount of money in order for you to let me kill you.
Both of those sums of money (everything or infinity) likely exceed the willingness to pay to reduce the likelihood of dying with some probability, multiplied by the number of people experiencing it.
In economic terms, we are comparing the area under the demand curve (the consumer’s surplus) for life (which has a value asymptotically approaching infinity as the amount of life approaches 0 (death approaches certainty) for a single individual, with the marginal change in the likelihood of survival multiplied by all individuals (i.e. the the quadrilateral between the y-axis of price and the same demand curve, between Pb and Pa) which describes the change in price for a change in survival).
On the one hand, using the marginal change for everyone rather than total change for the one person whose life is saved, we will give a lower value to safety improvements. On the other hand, the value of life to the individual himself is much higher than the value of life of that individual to society at large.

Discount rates on human lives

From Tyler Cowan: Don’t apply positive discount rates to human lives:

“Ben Trachtenberg writes:

This Article presents two new arguments against “discounting” future human lives during cost-benefit analysis, arguing that even absent ethical objections to the disparate treatment of present and future humanity, the economic calculations of cost-benefit analysis itself – if properly calculated – counsel against discounting lives at anything close to current rates. In other words, even if society sets aside all concerns with the discounting of future generations in principle, current discounting of future human lives cannot be justified even on the discounters’ own terms. First, because cost-benefit analysis has thus far ignored evidence of rising health care expenditures, it underestimates the “willingness to pay” for health and safety that future citizens will likely exhibit, thereby undervaluing their lives. Second, cost-benefit analysis ignores the trend of improved material conditions in developed countries. As time advances, residents of rich countries tend to live better and spend more, meaning that a strict economic monetization of future persons values the lives of our expected descendents above those of present citizens. These two factors justify “inflation” of future lives that would offset, perhaps completely, the discount rate used for human life. Until regulators correct their method of discounting the benefits of saving human lives in the future, the United States will continue to suffer the fatal costs of underregulation, and agencies will remain in violation of legal requirements to maximize net benefits.”

I think in practice we have to discount future lives, if the discount rate were zero, then we should do nothing for the present as the infinite future would dominate any calculation. I am dubious health inflation will continue unabated. The discussion on the article is interesting and worth reading.

Rail Transit Benefit Cost Analysis – Nonuser benefits

There is a nice debate between Peter Gordon and Paige Kolesar, Robert Cervero and Erick Guerra, commented upon by Lisa Schweitzer on non-user benefits from rail transit investments. This appears in
Public Works Management and Policy — April 2011, 16 (2)

Unfortunately, this is behind a paywall, so if you don’t have a university, it may be difficult or pricey to get. (boo!).
Gordon and Kolesar:

Rail transit systems in modern American cities typically underperform. In light of high costs and low ridership, the cost-benefit results have been poor. But advocates often suggest that external (non-rider) benefits could soften these conclusions. In this paper we include recently published estimates of such non-rider benefits in the cost-benefit analysis. Adding these to recently published data for costs and ridership, we examine 34 post-World War II U.S. rail transit systems (8 commuter rail, 6 heavy rail and 20 light rail). The inclusion of the non-rider benefits does not change the negative assessment. In fact, sensitivity analyses that double the estimated non-rider benefits and/or double transit ridership also leave us with poor performance readings. Advocates who suggest that there are still other benefits that we have not included (always a possibility) have a high hurdle to clear.

Cervero and Guerra:

The debate over the costs and benefits of rail passenger transit is lively, deep, and often ideological. As with most polemical debates, the truth probably lies somewhere in the middle of extreme views. Some rail systems have benefits that outweigh their costs, while others do not. Applying a commonly used transit-fare price elasticity to 24 of the largest light and heavy rail systems in the United States and Puerto Rico, assuming a linear demand curve, and accounting for a counterfactual scenario, we find that just over half of the systems have net social benefits. Although Los Angeles’ rail system does not “pass” our back-of-the-envelope cost–benefit analysis, as the network expands, it will begin to mimic the regional spatial coverage and connectivity of its chief competitor—the auto-freeway system—and approach the fare recovery rates of other large, dense American cities.

(Via Peter Gordon’s blog.)

State using Benefits and Costs to determine priorities.

This is news, the state of North Carolina says it is now using benefits and costs to determine what projects should get built. It seems more of a cost-effectiveness ranking system rather than pure monetization, but it has the merits of being transparent in principle.
The Asheville Citizen-Times writes: Asheville’s I-26 Connector project deemed high-cost, low benefit

But, state Board of Transportation member Wanda Proffitt said at a meeting to hear comments on the state’s long-range transportation plan, at least those projects that do get built will be at the top of the list because objective criteria indicate they should be.
The Department of Transportation is now ranking proposed projects according to numerical factors like expected travel time saved, measures of congestion and accident rates.
“What we have done is take the old-time politics out of how we spend transportation dollars. Now we’re doing it based on data and the priorities” of local transportation planning organizations, she said.
Only about 20 people attended the meeting, held to hear comments on DOT’s proposed 2011-20 Transportation Improvement Program.
Half or more of that number were either DOT or local government officials who came to hear what other people had to say at the meeting for people in a seven-county area that includes Buncombe County.

The comments on the article were skeptical.