Jump to main content.

Optimal Experimental Design for Non-Market Choice Experiments

Quick Links

Research Funding

Metadata


To assess the total value, including use and nonuse values, of nonmarket goods such as environmental amenities, researchers often apply survey techniques that allow them to explore public preferences for hypothetical goods or services. Until recently, the standard survey technique for this purpose has been the contingent valuation method. By now, a similar but more complex approach to choice experiments, conjoint analysis, has been used in several environmental contexts. Conjoint analysis is a marketing technique that can be used to assess values for attributes of market or nonmarket goods based on survey respondents' willingness to trade-off different bundles of these attributes.
The goal of this study is to extend the research being completed by the principal investigator (PI) on optimal design for conjoint analysis. In that research, the PI derived optimal designs for "main effects" binary and multinomial choice experiments. Although this represents an important extension to the current literature on experimental design for conjoint analysis, there are still several issues that must be addressed to fill out the literature on optimal survey design, especially for the nonmarket valuation context.
Specifically, the current conjoint design research has not addressed models that include significant substitution effects. Such models have been shown to be more appropriate for environmental amenities than models that ignore interactive effects. The nonlinearities that occur with these models present a new complexity to the experimental design process.
Further, the optimal design criterion used in the PI's current research, "D-optimality," is not necessarily the most appropriate for nonmarket valuation problems. D-optimality focuses on jointly obtaining the most efficient model parameter estimates. Environmental valuation problems often are more focused, however, on obtaining the best estimates possible for willingness to pay, a nonlinear function of the model parameters, or in cases of resource compensation, the best estimates for a marginal rate of substitution, a ratio of two specific model parameters. A more appropriate design criterion for nonmarket valuation would focus on one of these statistical measures. This project is to address the issues of including substitution effects and developing more appropriate design criteria for binary and multinomial nonmarket choice experiments.

Objectives/Hypothesis:
To derive optimal designs for choice experiments when substitution effects are significant. To make the current literature and research of the PI more relevant to environmental valuation by using optimal design criteria that are directly linked to the goals of improving estimation of willingness to pay or marginal rates of substitution. To compare the statistical gains or losses associated with multinomial versus binary choice experiments and with the inclusion or omission of substitution effects. To generate rules of thumb that can be easily communicated to experimental choice researchers to improve their survey designs in practice.

Approach:
The design problem will be set up by specifying the utility-theoretic choice model and analytically deriving the optimal design criteria. In this study, the criteria will be the minimization of the variances of willingness to pay and a marginal rate of substitution. Analytical solutions will be derived and specific numerical solutions will be calculated from the analytical solutions using the Mathematica software program. Once numerical solutions are obtained, rules of thumb can be developed that can be easily communicated to conjoint researchers to improve their study designs.

Expected Results:
The proposed research will provide explicit expressions and rules of thumb for the design of conjoint analysis questions, allowing researchers to estimate willingness to pay with greater statistical precision using lower sample sizes than they otherwise would need. This ability will, in turn, benefit governmental policy-makers who sponsor valuation studies by either reducing the costs of survey administration or increasing the reliability of the study conclusions.

Metadata

EPA/NSF ID:
R827987
Principal Investigators:
Kanninen, Barbara
Technical Liaison:
Research Organization:
Minnesota, University of
Funding Agency/Program:
EPA/ORD/Valuation
Grant Year:
1999
Project Period:
January 1, 2000 to June 30, 2002
Cost to Funding Agency:
$61,014
Project Status Reports:
For the Year 2000

Objective: The overall objective of this project is to derive optimal designs for stated choice experiments when substitution effects are present. Specific objectives are to:
    1. Make the current literature and the investigator's research more relevant to environmental valuation by using optimal design criteria that are directly linked to the goals of improving estimation of willingness to pay or marginal rates of substitution.
    2. Compare the statistical gains or losses associated with multinomial versus binary choice experiments and with the inclusion or omission of substitution effects.
    3. Generate rules of thumb that can be easily communicated to experimental choice researchers to improve their survey designs in practice.
Progress Summary:

In environmental economics, choice experiments are used to assess people's willingness to pay for different environmental attributes. "Optimal design" for choice experiments refers to the way attributes are assigned to choice sets so that the researcher can obtain as much information as possible about underlying preferences.

Earlier research by the principal investigator (PI) derived optimal designs for choice experiments based on the D-optimality criterion. This is a popular optimal design criterion that focuses on maximizing the statistical information that can be obtained about the full set of underlying model parameters. The first phase of the current project focused on another optimal design criterion: C-optimality. This criterion focuses on getting the best estimate possible for willingness to pay.

The solution to the C-optimal design problem is to generate choice sets that make respondents perfectly indifferent between the alternatives. In other words, there should be a zero utility difference between each alternative in a choice set. Unfortunately, in practice, this solution results in multicolinearity. Attribute levels in each alternative are exact linear combinations of the other attributes. Multicolinearity occurs because there are more variables in the problem than are necessary for estimating willingness to pay. The model falls apart and is inestimable.

Based on this result, the PI has concluded that our standard approach to estimating willingness to pay, as a ratio of estimated parameters, is inherently inefficient. From a purely statistical perspective, the C-optimal solution makes clear that the best approach is to drop extraneous variables and estimate willingness to pay directly.

The results of the current phase of the research bring us back to the PI's previous research project. Because we generally use an indirect approach to estimate willingness to pay, the PI has concluded that the D-optimality criterion, rather than C, is the most appropriate basis for designing choice experiments.

The PI also briefly investigated the question of how the number of alternatives in a choice set affect estimation efficiency. This turns out to be a complicated question that is directly tied to the specific attribute levels used in the different alternatives. In a Monte Carlo study, the PI found that adding alternatives with intermediate attribute levels can actually decrease the statistical information provided by a choice set. Exactly when and how this occurs would be an interesting research line to pursue further.

Future Activities: The next phase of the project is to address the model with substitution effects (the first objective listed above). Substitution effects occur when an individual's willingness to pay for two attributes together is less than the sum of his or her willingness to pay for each attribute separately. Because of the results just discussed on C-optimality, the next phase will focus on D-optimality as the design criterion. The model with substitution effects will have a nonlinear utility expression, which greatly complicates derivation of the optimal designs. It is not clear whether it will be possible to derive a general solution. Instead, it might be that solutions are dependent on the specific parameter values. If this is the case, solutions will be provided in a series of tables. The project currently is on schedule and no changes are anticipated.

Journal Article:
Kanninen B. Optimal design for multinomial choice experiments. Journal of Marketing Research (resubmitted in 2001).

Presentation:
Kanninen B. Optimal design of choice experiments for nonmarket valuation. Presented at the EPA Workshop "Stated Preference: What Do We Know? Where Do We Go?" Washington, DC. To be published in a volume edited by Dr. Mike Christie, University of Wales, 2001.

Proceedings:
Kanninen B. Optimal design of choice experiments for nonmarket valuation. In: Christie M, ed. Proceedings of the EPA Workshop "Stated Preference: What Do We Know? Where Do We Go?" Washington, DC, 2001.
Project Reports:
Final

Objectives:
The objectives of this research project were to: (1) derive optimal designs for (hypothetical) choice experiments to get the best possible estimate for willingness to pay when substitution effects are present; (2) make the current literature and research of the Principal Investigator (PI) more relevant to environmental aluation in practice by using optimal design criteria that are directly linked to the goals of improving estimation of willingness to pay or marginal rates of substitution; (3) compare the statistical gains or losses associated with multinomial versus binary choice experiments and with the inclusion or omission of substitution effects; and (4) generate rules of thumb that can be easily communicated to experimental choice researchers to improve their survey designs in practice.

Summary/Accomplishments:
Environmental economists often use choice experiments to assess people's willingness to pay for different environmental attributes. In choice experiments, people are shown a set of hypothetical goods and are asked to choose their most preferred option. The goods vary by attribute levels; for example, the numbers of particular wildlife species viewed, or measures of air or water quality.

Experimental design of choice experiments refers to the way attributes are assigned to choice sets. Before conducting a choice experiment, the researcher must assign levels for the different attributes and allocate the different levels across alternatives. The experimental design, combined with people's choice responses, make up the dataset that will ultimately be used to estimate values.

The optimal design literature has shown that choice set designs can greatly affect the amount of statistical information a dataset provides. Until the PI's work, however, the literature focused only on improving designs by re-working the attribute level allocations across choice sets. The PI's work took optimal design a significant step forward by looking at how the choice of attribute levels affects information.

There are many ways to measure statistical information. Optimal design researchers, therefore, must specify a certain design criterion to evaluate the quality of a design. The design criterion specifies precisely which statistical aspect of a set of estimators is most important to the researcher. One of the most popular design criteria is called D-optimality. This criterion looks for a dataset that jointly provides the most statistical information about the full set of model parameters.

In earlier research, the PI derived D-optimal designs for choice experiments. Though D-optimality is a useful, general criterion, the PI believed that for environmental valuation purposes, an alternative design criterion might be more useful. The first phase of the current project focused on a design criterion called C-optimality. This criterion focuses on getting the best estimate possible for willingness to pay.

The PI found that the C-optimal design solution is to generate choice sets that make respondents perfectly indifferent between alternatives. In other words, attribute levels should be set so there is zero utility difference between each alternative in a choice set.

Unfortunately, the PI found that in practice this solution results in multicollinearity: attribute levels in each alternative would be exact linear combinations of the other attributes. This multicollinearity occurs because there are more variables in the problem than are necessary for estimating willingness to pay. So under C-optimality, the model falls apart and is inestimable.

Based on this result, the PI concluded that the standard approach to estimating willingness to pay as a ratio of estimated parameters is inherently inefficient. From a purely statistical perspective, the best approach to estimating willingness to pay is to drop extraneous variables and estimate willingness to pay directly-in other words, just ask people their willingness to pay.

Note that this is the statistically best approach, but not necessarily the best for obtaining reliable responses. In fact, the research community highly favors the indirect approach to obtaining willingness to pay information. Because of this, the PI has concluded that the D-optimality criterion, rather than C-optimality, is the most appropriate basis for designing choice experiments.

An extensive set of D-optimal designs is available in the PI's forthcoming article in the Journal of Marketing Research (JMR). A thorough presentation and discussion of C-optimality is presented in the PI's book chapter forthcoming in Christie and Midmore (editors).

The PI also briefly investigated the question of how the number of alternatives in a choice set affect estimation efficiency. This turns out to be a complicated question that is directly tied to the specific attribute levels used in the different alternatives. In a Monte Carlo study, the PI found that adding alternatives with intermediate attribute levels can actually decrease the statistical information provided by a choice set. Exactly when and how this occurs would be an interesting research question to pursue further.

The second phase of the project derived optimal designs for models that include substitution effects or higher-order terms. Unfortunately, the PI found that a general optimal design solution for these non-linear models is extremely complex and probably impossible to obtain in theoretical terms. This is consistent with other findings in the optimal design literature. Exact optimal designs would depend, in a complicated way, on model parameters. The PI did, however, find two special cases where optimal designs can be specified. Both cases fit nicely into the known body of optimal design results.

The first case is a quadratic model where at least one attribute appears only linearly. In other words, though the full model has higher-order terms, it is linear with respect to at least one particular attribute. By focusing on this linear term, optimal designs can be obtained that mimic those for the standard, linear model.

Essentially, the results provided in the JMR article can be applied directly to this case, with one important change. The JMR results apply to linear (or main-effects) models. D-optimal choice set designs assign two levels to each attribute. For all attributes but one, these levels are the researcher's judgment of reasonable upper and lower bounds. The attributes are allocated according to the standard main-effects design arrays.

The level of the final attribute is manipulated to obtain certain response probability splits. For example, with a two-attribute choice set, the researcher should strive to obtain a .82/.18 response split between the two alternatives. With eight attributes, a .67/.33 split is optimal. The optimal split appears to move inward (closer to .50/.50) as the number of attributes increases.

With the quadratic model with a linear attribute, the important change from these results is that a main-effects design array is not sufficient for model identification. Instead, the researcher must use three-levels for the quadratic attributes. The design array becomes larger. The optimal choice split is essentially the same as in the main-effects case, except that it is based on the total number of variables (linear plus quadratic) in the model rather than the number of attributes.

The second case where specific design solutions are obtainable is the fully specified interactive model. This model includes all interactive terms. For example, with two attributes, the model includes two linear terms (say, x1 and x2) and one interactive term (x1x2). With three attributes, the model contains the following variables: x1, x2, x3, x1x2, x2x3, x1x3, and x1x2x3. These models require the full factorial design array (the full set of attribute permutations) for model identification.

The optimal response split for this case is intriguing. It always requires a .82/.18 probability split and is independent of the number of attributes in the model. Recall that the .82/.18 split is optimal for the two-attribute, main-effects model and that larger models have different optimal splits.

Results for the quadratic and interactive models are presented in "Optimal Design for Binary Choice Experiments with Quadratic or Interactive Terms."

How does one implement the above optimal designs in practice? The PI recommends a sequential approach. The researcher would start with a choice set design based on standard design arrays and specify levels for the manipulator attribute to obtain optimal response splits based on his or her intuition. Then, as choice responses are collected, the researcher can update the manipulator attribute levels to move responses toward the optimal ones.

The PI assisted researchers at Michigan State University in the first example of such an empirical study. The research team found that the process went smoothly and estimation results were more efficient than they otherwise would have been. Results of this study are provided in Steffens, Lupi, Kanninen, and Hoehn (2002).

Conclusions:
For main-effects and interactive models, all attributes but one should be assigned only two levels. These levels should be what the researcher judges to be reasonable upper and lower bounds. For quadratic models, attributes that have quadratic terms should be assigned three levels at their upper and lower bounds and the mid-point in between.

These attributes should be allocated across choice sets according to the relevant design arrays. For main-effects models, these design arrays are the standard, main-effects designs; for quadratic models, three-level designs; and for interactive models, full factorial design arrays.

Note that over-specifying interactive terms at the design stage-using a full factorial when only a main-effects design is ultimately needed-will not affect estimation efficiency. Using a quadratic specification when only main-effects will ultimately be estimated, however, decreases efficiency because the mid-point attribute level is not very informative for main-effects estimation.

Once all attributes but one are set, the researcher uses the final attribute to manipulate response rates. For fully specified interactive models, the optimal probability split is always .82/.18. This also is the optimal split for a model with two variables.

These optimal splits move inward as the number of columns in the data matrix increases. For example, the optimal split is .67/.33 for a model with eight columns in the data matrix.

Note that in practical situations, this difference is not so great. To obtain optimal response rates, the researcher will have to start with a prior guess for the manipulator attribute levels. Over the course of the experiment, the researcher can adjust the manipulator attribute to move responses toward their optimal splits. When the researcher does not know exactly how many attributes or combinations of attributes will ultimately be used in the model, he or she can focus on keeping the design within the bounds of .82-.67/.18-.33.

Many researchers would be uncomfortable applying these optimal designs directly because of the limited number of attribute levels used. For main-effects models, all attributes take only two levels. For quadratic terms, attributes take three levels. Keep in mind, however, that once a functional form is specified, these numbers are all that are required to estimate the model. In some sense, additional data points are superfluous. Either way, the most influential data points collected will dominate estimation. For this reason alone, it is advisable to focus on points of high information. Generally, these are the most extreme points with which a researcher is comfortable.

Note that extreme does not mean the far reaches of an attribute's range. A daily pass at a state park, for example, should not be offered at $100 (in today's dollars). With a main-effects model, the upper bound should be the maximum value thought to be on the linear portion of the utility model. This would be a much lower value.

Often in valuation research, sample sizes are small because of small budgets. This is where optimal design of experiments is most important. In these cases, the researcher must focus on high information points to get significant results, especially when there are many attributes.

More empirical research on this approach is needed. Small-scale is fine. Research questions abound and include the question of which attributes would best serve as manipulators, how often is it best to update the manipulator attribute, and how much statistical improvement can be obtained under controlled experimental conditions.

Publications and Presentations:
Kanninen BJ. Optimal design for multinomial choice experiments. Journal of Marketing Research 2002, Volume: 39, Number: 2 (May), Page: 214-227.

Supplemental Keywords:
onjoint analysis, nonmarket valuation, contingent valuation, survey, willingness to pay, multinomial logit, D-optimality, C-optimality. , Economic, Social, & Behavioral Science Research Program, RFA, Scientific Discipline, Ecology and Ecosystems, Economics, Economics & Decision Making, Social Science, decision-making, D-optimality, conjoint analysis, contingent valuation, decision analysis, dichotomous-choice, market valuation models, multi-criteria decision analysis, multi-objective decision making, multinominal nonmarket choice experiments, non-market valuation, nonmarket choice, optimal experimentation, policy analysis, public policy, standards of value, surveys

Local Navigation


Jump to main content.