Environmental Economics

Comparative Studies of Approaches to Eliciting Economic Values

  • Abstract
    This project will look at the properties of a number of different stated preference approaches to eliciting information useful for estimating the economic value of a change in an environmental amenity. The elicitation methods to be examined range from an open-ended question which directly elicits agents' willingness to pay for the change and a binary discrete choice question which simply asks agents whether they are willing to pay the stated cost of the change. Between these two elicitation approaches lie a number of other common approaches to be examined including payment cards, bidding games, double-bound discrete choice, and discrete choice conjoint analysis. Elicitation methods based on ratings and complete rankings will be also be considered. The approach to be used to compare the properties of the different elicitation methods is drawn from the economics literature on incentive compatibility and mechanism design. From this perspective, it is possible to characterize an elicitation method in terms of: (1) the information it conveys to survey respondents, (2) the nature of the message space available for responses, and (3) beliefs about how those messages will be used. Traditionally, the psychology and survey research literature has only examined (2). Elicitation methods, however, which differ on any of these three features typically differ in terms of the strategic opportunities offered respondents in answering the valuation question(s). For each of the elicitation formats the explicit (or usual) assumption for (1) and (3) will be considered as well as a range of plausible alternatives. The comparative statics analysis proposed will look at how an agent's optimal response change as the characteristics of the elicitation method used change. The statistical nature of the information revealed by different elicitation methods will also be examined. Economic valuation via stated preferences occurs in a number of different contexts. Among the those to be considered are the nature of the good: private, quasi-public where exclusion is possible, and pure public. For the later, both coercive and non-coercive payment vehicles will be examined. The possibility of uncertainty over the cost and/or benefits of the good will be also be considered, as will different forms of risk preferences. Predictions from the model developed will be compared to the large body of existing literature on laboratory and field experiments using different elicitation methods in a wide variety of circumstances. This exercise will allow a determination of the degree to which the model has predictive validity. It is also likely to help in narrowing the set of competing assumptions. This exercise will be helpful in formulating future experiments by pointing to those tests which have the most ability to distinguish between competing hypotheses. The results of this work should be of substantial use to researchers seeking to determine which elicitation method they should use in different situations. The results of this work should also be of substantial use to policy analysts seeking to interpret estimates from existing environmental valuation studies.
  • Metadata
    Principal Investigators:
    Carson, Richard T.
    Groves, Theodore
    Machina, Mark J.
    Technical Liaison:
    Research Organization:
    California at San Diego, University of
    Funding Agency/Program:
    Grant Year:
    Project Period:
    October 1, 1995 - September 30, 1997
    Cost to Funding Agency:
  • Project Reports
          Final Report

          Executive Summary

          Project Description and Objectives of Research: To examine the properties of different preference survey elicitation formats commonly used for the purpose of assigning "value" to environmental amenities. In doing so, we sought answers to three related questions. First, are there particular elicitation formats and conditions where truthful preference revelation is the optimal response? Second, when should different elicitation formats yield similar estimates of economic value? Third, what information about preferences should be extractable under different question formats and conditions?

          Summary of Findings:

          The study developed a comprehensive framework for looking at the incentives posed by different preference elicitation formats and at the role played by the information conveyed by each format. The framework is based upon the mechanism design literature and the assumptions that agents ask: (1) whether they care about how the outcome might be influenced by the answers they provided, (2) whether the aspects of the scenario described are plausible, and (3) how the survey results are likely to be used. We then define consequential and non-consequential preference question. For a question to be consequential an agent must believe that the response to it may influence some action and care about possible outcomes. Only responses to consequential preference questions can be clearly given an economic interpretation.

          For consequential questions we show that much of the dilemma with interpreting the responses to preference questions revolves around what we term the "face value" assumption that researchers instinctively make when analyzing the responses to preference questions. This face value assumption has two premises: (a) agents truthfully answer and (b) the question answered is the one the researcher intended. In many instances, a rational economic agent should provide responses that violate this face value assumption.

          We begin our examination of preference elicitation formats by showing that all of the standard elicitation formats can be viewed as generalizations of a single (one shot) binary discrete choice question which asks agents to pick their most preferred out of k=2 alternatives (see Figure 1). There are three basic ways in which a single binary discrete choice question can be generalized. First, open-ended type matching questions drop the "cost" amount for the second alternative and ask for the amount that makes the agent view the two options as equivalent from a utility perspective. Second, it is possible to ask multiple-choice questions with two alternatives. This approach adds the assumption that responses across pairs are independent. A number of popular preference elicitation formats such as double bounded dichotomous choice and complete ranking of alternatives can be shown to be manifestations of a sequence of binary discrete choice questions from a strategic vantage point. Third, it is possible to offer agents more than two choices (multinomial choice). Such questions add the assumption that the agent picks the most preferred out of k > 2 alternatives.

          We start our formal analysis of the properties of elicitation formats with an incentive compatible binding binary discrete choice referendum (Farquarson, 1969). We show that a more general condition, the probability of providing a particular alternative increases as the percent in favor of it increases over some range, can be substituted for the plurality vote requirement of the binding referendum without altering its incentive properties. This yields the well-known "advisory referendum" mechanism. We then apply an old result in the mechanism design literature by Green and Laffont (1978) who showed that it is possible to use an exogenously chosen sample of the population rather than an action of the entire population to implement a large class of mechanisms, including those we consider. This yields the "advisory survey" which has the same incentive properties as the binding referendum.

          Not all binary discrete choice questions, however, are incentive compatible. Indeed, the implicit conditions that underlie the Farquarson result on incentive compatibility are often violated in preference surveys. They require the ability of the government to be able to provide the good and to coercively extract payment for it. They also require that there is no other way to provide the good and that the results of the vote do not influence other government actions.

          We show that the use of a binary discrete choice question in the case of the introduction of a new private good or the provision of a public good via voluntary contributions should over estimate true willingness to pay (WTP) for the good. In the case of voluntary contributions to a public good the incentive structure that should result in overestimates based upon the survey responses should also result in underestimates (free riding) of true WTP with respect to actual contributions. Thus, comparisons of survey statements of WTP to actual contributions provided from the voluntary contributions case can only provide upper and lower bounds on WTP. Such a comparison cannot say anything about how CV performs with coercive payment vehicles. In contrast, we show that the commonly offered choice between two mutually exclusive configurations of a quasi-public good (e.g., a low quality-low priced version of a recreation site versus a high quality-high priced version of a recreation site) is incentive compatible but such a choice does not reveal anything about a possible change in the number of times the site will be visited.

          We show that cost uncertainty in the case of a public good with a coercive payment vehicle should translate into a lower probability of an agent indicating a WTP a specific amount because the cost uncertainty should translate into income uncertainty which is undesirable from the perspective of a risk adverse agent. For the introduction of a new private good or for voluntary contributions to a public good, the opposite effect should be observed. For a choice between two mutually exclusive configurations of a good, the effect of price uncertainty is indeterminate.

          For elicitation formats offering other than a binary discrete choice, we noted that the Gibbard-Satterwaite theorem implies that truthful preference revelation cannot always be the optimal response without imposing restrictions on the nature of agent utility functions. We then proceed to examine the nature of the violations of incentive compatibility that occur with other commonly used preference elicitation formats.

          For double-bounded questions, we show that any of the small set of plausible assumptions about the signal provided by asking the second question should break the perfect correlation between the two responses that is typically assumed in the statistical analyses. In most instances, plausible beliefs about the information conveyed by the signal provided by the second question should lead to a downward bias in the double bounded estimate, which is the observed result in the empirical literature.

          For open-ended questions, the optimal response pivots on expected cost. This leads to three results that are robust to plausible beliefs about how the responses are going to be used. First, agents whose WTP is less than their expected cost should provide a zero WTP amount as this provides the maximum discouragement to going forward with the project. Also, we show there should be few, if any, very small WTP amounts and that the WTP amounts provided should be correlated (anchored) on any information perceived to be related to expected cost. Again, all three of these predictions are consistent with results found in the empirical literature.

          For sequences of paired comparisons involving different goods and multinomial choice questions, we find that it is necessary to first ask how many of the goods can be supplied and over how many of the goods an agent's utility is defined. Two cases are particularly problematic: (a) where the goods are of multiple levels of the same public good (e.g., different levels of air quality) in a city and, as such, only one level can be supplied, and (b) where agents get utility from the bundle of goods provided rather than from only one of the goods supplied. In both of these cases, it may be optimal for agents to pick their least preferred alternative in any particular choice set.

          More optimistic results are obtained in the case of a multinomial choice question where multiple goods can be provided and an agent's utility is defined only over one of the goods (e.g., the particular PC purchased or the recreational site actually visited). In the case where k-1 of the goods will be supplied, this elicitation format is incentive compatible because the choice offered is effectively the agent's favored versus an unknown alternative since the worst the agent can do is have her second favorite alternative made available. Relaxing the k-1 condition a bit (i.e., k-2) suggests that one should observe responses that violate the independence of irrelative alternatives (IIA) assumption, as agents will tend to pick their favorite or something close to it in utility space. The consequence of this type of behavior is that it is likely that marginal tradeoffs between alternative attributes can be correctly estimated since the scale factor adversely impacted by IIA violations drops out of this calculation. It will be difficult, however, to recover an unbiased estimate of total value, since that WTP estimate depends critically upon the estimate of the scale factor.


          Different preference question elicitation formats should generally yield different estimates if the predictions of neoclassical economic theory hold. Incentive compatible formats exist in some but all not instances. In these and some other instances, it is possible to extract useful information if one makes the appropriate assumptions about what information the agent should be reporting.

          Papers and Manuscripts:

          Contingent Valuation: A User's Guide (Environmental Science and Technology, forthcoming)

          Several others papers have been completed and others are in progress.


          Association of Environmental and Resource Economists

          Econometric Society

          European Association of Environmental and Resource Economists


          Japan Forum of Environmental Valuation

          National Science Foundation

          USDA W-133 Benefits in Natural Resources Meetings

          World Congress of Environmental Economists

  • Project Status Reports
For the year 1997

Objectives: Surveys of the public are often undertaken to help ascertain the public’s preferences. If successful, this information can be an important component to determining optimal public policies and making business decisions. Often, however, survey results are dismissed, particularly by some economists who question the creditability of responses to survey questions. Essentially, the argument made is that nothing is at stake for the survey respondent in answering a question. This rather automatic dismissal seems a bit odd in light of the billions of dollars spent each year by businesses and governments to collect and analyze survey data.
    From a substantive perspective, this dismissal of survey responses can be seen as inappropriate if either of two conditions hold. The first occurs if survey respondents are motivated to seriously consider the questions asked and they answer truthfully. This is the basic assumption of the other social sciences disciplines, which explains to a large degree their enthusiastic embrace of the use of survey data. The second occurs if respondents believe that governments and businesses consider their survey answers in making decisions. This would appear to be not an unreasonable assumption in a world where a frequent criticism of governments is that they pay too much attention to the public’s stated preferences in surveys and that business decisions are driven largely by marketing research tests. In this situation, an economist’s first reaction to a survey question with real consequences might be to argue for the strong possibility of a strategic response. The pitfall economists tend to fall into here is not in considering the possibility of strategic behavior, but rather in jumping to the conclusion that if respondents behave strategically, their answers cannot provide useful information. This is obviously not true if optimal strategic behavior coincides with truth-telling or if the influence of the strategic behavior can at least be partially unraveled. We address the issue: what is the optimal strategic response to a specific survey question?

    The approach that we use is to adopt the theoretical framework of mechanism design long used in economics to formally examine the role of incentives and information. The major advantage of mechanism design theory is that it provides a consistent theoretical model for comparing a wide variety of different question formats and specific contexts in which survey questions are asked. We start by providing a formal definition of consequential and hypothetical survey questions. Economic theory provides strong predictions for optimal responses to the former but no predictions about the latter. Most, but not all, surveys generally labeled as contingent valuation or marketing surveys fall into the class of consequential surveys.

    Progress Summary: The key result of this project is that the form of the optimal strategic response is context-dependent in ways that make it difficult to make any general claims about the properties of particular response elicitation format, such as a binary discrete choice question, without further specifying the context being used. It is this context-dependence that appears to give rise to a variety of conflicting claims made in the literature. Fortunately, the properties of consequential choices, at an abstract level, depend upon only a few key features. For instance, when using a binary discrete choice response format, it is straightforward to demonstrate fairly general conditions under which the optimal strategic response is truthful preference revelation if the good is a pure public good and a coercive payment mechanism is used. In contrast, it is possible to demonstrate that truthful preference revelation can never generally be the optimal response strategy to a binary discrete choice survey question involving the provision of a new private good or a pure public good with a voluntary contribution mechanism. The key features that we consider are given in Table 1 on the next page.

    We believe that our results will serve several distinct purposes. First, they provide a consistent basis upon which to make predictions about the optimal response to any particular survey question. This should assist in the interpretation of existing results and provide a basis upon which to develop new experiments to help test the model. Second, the model proposed should help researchers make informed decisions about what types of survey questions should be used in different contexts. Third, because the model predicts different (specific) behavior in different situations, it calls into question many claims in the literature that particular results, such as differences between valuation estimates based upon two different survey question response formats, imply that respondents have inconsistent or non-existent preferences. Fourth, the model provides a clear linkage between observable behavior in political and private markets and responses to consequential surveys.

    Future Activities: Currently, we are undertaking three activities with respect to this project. First, we are refining the basic theoretical model. Second, we are using the model to analyze specific situations of particular interest to environmental valuation. Third, we have begun to compare systematically the model’s predictions to empirical tests appearing in the existing literature.

    Specific Cases
    Changes in choice set
    Strict expansion, strict reduction, and strict alternation
    Types of goods
    Pure public, quasi-public, and private
    Payment mechanisms
    Coercive and voluntary
    Cost and Benefit
    Perceived government decision criteria
    Political (plurality), effeciency (Pareto), and eqity
    Response elicitation formats
    Binary discrete choice, double-bounded discrete choice, multiple choice, paired-comparisons, open-ended, bidding game, and payment card
    Table 1. Strategic Behavior and Context-Dependence