Entropy Methods for Adaptive Utility Elicitation - Semantic Scholar
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 34, NO. 2, MARCH 2004
169
Entropy Methods for Adaptive Utility Elicitation Ali E. Abbas, Member, IEEE
Abstract—This paper presents an optimal question-selection algorithm to elicit von Neumann and Morgenstern utility values for a set of ordered prospects of a decision situation. The approach uses information theory and entropy-coding principles to select the minimum expected number of questions needed for utility elicitation. At each stage of the questionnaire, we use the question that will provide the largest reduction in the entropy of the joint distribution of the utility values. The algorithm uses questions that require binary responses, which are easier to provide than numeric values, and uses an adaptive question-selection scheme where each new question depends on the previous response obtained from the decision maker. We present a geometric interpretation for utility elicitation and work through a full example to illustrate the approach. Index Terms—Maximum entropy, question-selection, utility.
I. INTRODUCTION N MANY decision problems, we are faced with multiple and conflicting attributes. When the decision situation is deterministic, a decision alternative is characterized by a single prospect1 . The decision problem is resolved by ranking the prospects present, or assigning a value function over the attributes of the decision situation. The optimal decision alternative corresponds to the prospect that has the highest preference order in the ranked list, or the largest value for the value function. Several question-selection algorithms have been proposed in the literature to rank-order prospects of a decision situation. Many of these algorithms focus on determining a tradeoff weight vector for prospects with multiple attributes. For example, Korhonen et al. used pairwise comparison questions to eliminate decision alternatives and determine a tradeoff weight vector using convex cones [1]. Rao proposed an optimal sequence of pairwise comparison questions over a set of alternatives with two attributes [2]. Prasad et al. generalized the convex cones approach and introduced the idea of p-cones to minimize the preference information requirements [3]. Ha et al. assumed a partially specified multilinear function and used comparative statements to further constrain the value function and identify suboptimal alternatives [4]. Holloway and White used a dynamic programming approach to minimize the expected number of questions needed to determine a tradeoff weight vector for alternatives with multiple attributes [5].
I
Manuscript received April 17, 2003; revised August 11, 2003 and September 30, 2003. This paper was recommended by Associate Editor J. B. Yang. The author is with the Department of Management Science and Engineering, Stanford University, Stanford, CA 94305 USA (e-mail: [email protected].). Digital Object Identifier 10.1109/TSMCA.2003.822269 1We will use the term “prospect” rather than “outcome” or “consequence” of a decision situation as it refers to the whole future life starting at the end of the decision tree rather than an outcome of a situation.
Algorithms for ranking prospects when partial information is available about the rank order have also been proposed. For example, Kirkwood and Sarin presented a method to rank-order prospects of a decision situation using a weighted additive value function when partial information is available about the weights [6]. When uncertainty is present, a decision alternative is characterized by several prospects and a probability distribution for their occurrence. In this case, the rank order of the prospects is not sufficient to determine the optimal decision alternative and, in addition, we need to elicit the von Neumann and Morgenstern utility values [7]. To elicit those utility values, the first step is to rank-order the prospects of the decision situation using many of the algorithms cited above. The second step is to assign a “von Neumann and Morgenstern” utility value to each prospect as follows. For any , the decision maker asthree ordered prospects, signs a probability, , for which she is indifferent to receiving for sure and a deal where she would receive with probability and with probability . If the utility values of and are 1 and 0, respectively, then the utility value of , also called the preference probability of , will be equal to . The optimal decision alternative is the one, which has the highest expected utility. Both the rank order and the von Neumann and Morgenstern utility values are thus essential in the analysis of decision problems when uncertainty is present. Unfortunately, the utility elicitation process is subject to many cognitive and motivational biases (Keeney and Raiffa [8] and Froberg and Kane [9]). In addition, the number of prospects of a decision situation grows exponentially with the number of uncertainties present, and so the utility elicitation process can become a time-consuming and stressful task for the decision maker. For this reason several algorithms for eliciting von Neuman and Morgenstern utilities2 have been proposed in the literature. For example, Chajewska et al. [10] used pairwise comparison questions that maximize the “value of information,” [11], each question will provide. In this paper, we propose a question-selection algorithm to elicit utility values for a set of ordered prospects, which may have single or multiple attributes. We use the order provided to determine a joint distribution for the utility values and choose the questions that will provide the largest reduction in the entropy of this joint distribution. The algorithm takes into account the difficulty of the elicitation process as a practical constraint on the side of the decision maker and uses questions that require binary (yes/no) responses, which are easier to provide than numeric values. The approach is adaptive so each new question is determined by the previous response obtained from the decision maker. 2Throughout the rest of the paper we will refer to the “von Neumann and Morgenstern utility” as simply “utility.”
1083-4427/04$20.00 © 2004 IEEE
170
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 34, NO. 2, MARCH 2004
This paper is structured as follows. Section II provides a review of the entropy of a random variable and the basic formulas that will be used in the remaining sections of the paper. Section III derives the marginal probability distributions for the ordered prospects and introduces two defiutility values of nitions—the utility vector and utility volume—that will be used in the formulation of the utility elicitation problem. Section IV introduces a geometric representation for utility elicitation, and presents an optimal question-selection algorithm to elicit utility values from the viewpoint of information theory. Section V extends the approach to allow for imperfect responses from the decision maker and to -ary type questions. II. REVIEW OF BASIC ENTROPY CONCEPTS In 1948, Claude Shannon introduced the term, , as a measure of uncertainty about a discrete random variable having a probability mass function [12]. He called this term the entropy, which can also be thought of as a measure of the amount of information needed to describe the outcome of a discrete random variable. For example, a random variable, , can have four possible outcomes, , with probabilities, , respectively. Let us calculate the entropy of X using base two for the logarithm in the entropy expression.
(1) One intuitive way to explain this number is to consider the minimum expected number of binary (yes/no) questions needed to describe an outcome of . The most efficient question-selection procedure in this example is to start with the outcome that has the highest probability of occurrence, i.e., we ask “is ?” If it is, then we have determined in one question. If it is not, then we ask “is ?” Again, if it is correct, then we have determined in two questions, if it is not, we ask “is ?” We do not need to ask “is ?” because if it is not 0, 1, or 2, then it must be 3. The expected number of binary3 questions needed to determine is then
Entropy of the random variable
number of questions needed is bounded by the following wellknown data compression inequality [13] (3) The entropy of a discrete random variable is thus the lower bound on the expected number of questions needed to describe its outcome, and the lower bound is achieved when the probability of the search space is reduced by a half each time. We will make use of this notion of entropy to calculate the expected number of questions needed for our utility elicitation algorithm. Another important property of the entropy expression is its expansion in terms of a weighted sum of the entropy of the conditional distributions. The following example, taken from Shannon’s original paper, illustrates this property. A has three possible outcomes, , and random variable with probability . If we decompose in terms of and , with probability , and assign condition probability for and equal to and respectively, then the entropy of can be calculated from either representation, i.e., . We will use this property to determine the expected number of questions needed for utility elicitation when the decision maker provides imperfect responses. In continuous form, the integral is known as the differential entropy of a random variable having a probability density function, . As opposed to the discrete case, however, the differential entropy can be negative. This is illustrated by the following example, where can be positive or negative depending on the value of .
(4) The possibility of a negative sign shows the absence of the expected number of questions interpretation for entropy in the continuous case. However, we can relate the differential entropy to the discrete entropy by discretizing the continuous density into intervals of length and using the substitution in the discrete form entropy expression as follows:
(2)
This equality always holds when the question-selection process reduces the probability of the possible values by a half each time. If we look at the probability mass function used for this example, we note that each question asked did in fact reduce the probability by a half. While this equality does not always hold, an optimal question-selection process always exists (for any probability mass function) such that the minimum expected
3The use of binary questions in this example relates to the base two in the logarithm of the entropy expression.
(5) Entropy of the discretized variable. where If is Riemann integrable, then the first term on the right hand side of (2.5) approaches the integral of by definition of Riemann integrability. Thus (6) By induction, for a joint distribution of
variables, we have
(7)
ABBAS: ENTROPY METHODS FOR ADAPTIVE UTILITY ELICITATION
171
where Entropy of discretized joint distribution. Differential entropy of continuous joint distribution. In 1957, Edwin Jaynes built on the entropy concept and proposed the use of a distribution that maximizes Shannon’s entropy measure and satisfies the partial information constraints as an unbiased probability distribution [14]. Jaynes’ proposition is known as the maximum entropy principle and is considered to be an extension of Laplace’s principle of insufficient reason. III. UTILITY VOLUME We start this section with the definition of a utility vector for a set of ordered prospects. A utility vector contains the utility values of the prospects starting from lowest to highest [15]. Without significant loss of generality, we will assign a utility value of zero to the least preferred prospect, , and a value of (we neglect the case one to the most preferred prospect, of absolute indifference between the prospects since it merely assigns them all the same utility value). The utility vector has elements defined as
Fig. 1.
Utility volume and utility vector representation for
K 0 2 = 2.
Now we discuss the implications this uniform sampling has on the probability density functions for the elements of the utility vector. Let us start with the element .
(9) (10)
(11)
(8) Note that any utility vector of dimension can be repredimensional space in the region sented as a point in a . This redefined by gion, which we will call the utility volume, has a volume equal . to An example of a two-dimensional (2–D) utility volume for is shown in Fig. 1. The first and last elements of the utility vector are not represented since they are deterministic. Note that all points in the utility volume satisfy the decision maker’s preference ordering of the prospects but assign different utility values to them. In other words, if a decision analyst would like to elicit the utility values, then knowledge of the preference ordering alone tells him nothing about the location of the utility vector in the utility volume. If only the preference order is known, it is reasonable to assume, therefore, that the location of the utility vector is uniformly distributed over the utility volume. This represents maximum uncertainty about the decision-maker’s utility values. Before we proceed with the implications of this result, let us show how to generate uniform samples . over a utility volume of dimension A. Uniform Sampling Over the Utility Volume from a 1) Generate a random sequence . distribution, 2) Sort the generated sequence from lowest to highest and rename the variables such that . , form the uncertain elements The elements of the utility vector and also form an order statistic generated by a uniform distribution.
Using further analysis, and properties of order statistics, [16] the probability density function for , is
(12) where and Beta probability density function Given the marginal distribution for each utility value, the logical point estimate assignment is the mean of its marginal distribution (see for example [17]). We summarize these results below. B. Utility Vector: Marginal Distribution and Assignment When only the preference order is known, the marginal distributions for the utility values of a set of ordered prospects are the family of Beta distributions, . The mean of each distribution, , is the utility value you would assign to each prospect, j. The utility and are deterministic with values for prospects values 0 and 1 respectively. Note that this assignment may be different when different cost functions are available. For example, the optimal assignment using a square law cost function is the still mean of the marginal distribution, whereas the optimal assignment using an absolute error cost function is the median, and that using a delta impulse cost function is the mode of the marginal distribution.
172
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 34, NO. 2, MARCH 2004
Fig. 2. Marginal distributions for elements of the utility vector.
Further analysis and properties of order statistics show that the joint distribution of an order statistic generated by a common Uniform , is uniform over the utility volume, density, and is equal to
TABLE I OF THE MARGINAL DISTRIBUTIONS OF THE UTILITY VECTOR
DIFFERENTIAL ENTROPY
(13) The entropy of the joint distribution is thus
utility volume
(14)
We illustrate the previous results with the following example.
The entropy of the marginal distributions in our example corre. spond to the row IV. GEOMETRIC REPRESENTATION OF UTILITY ELICITATION
C. Example 1: Six Ordered Prospects prospects. She states her prefA decision-maker faces erence ordering for them as follows:
The utility values for prospects A and F are 1 and 0, while the marginal distributions for prospects E, D, C, and B are Beta (1, 4), Beta (2, 3), Beta (3, 2), and Beta (4, 1), respectively. These distributions are shown in Fig. 2. The maximum entropy utility vector is (0, 0.2, 0.4, 0.6, 0.8, 1). The joint distribution is uniform and equal to 4! over the utility volume . The entropy of the joint distribution is . It is important to note, from the curves shown above, that the elements in the center of the utility vector have maximum entropy. This can also be verified using the differential entropy , derived in [18] as expression of a beta distribution, (15) where
In this section we will apply the previous results to the utility elicitation problem. Given ordered prospects, the problem of eliciting the decision-maker’s utility values is identical to that of searching for the utility vector over the utility volume. We will use “von Neumann and Morgenstern” type questions for the search and yet not have the decision-maker place precise utility values. The approach will have several binary questions of the form “which deal do you prefer this or that?” The response to a question is used to update the distributions of all the utility values by truncating them at the value of the preference probability used for that question. At each stage we choose the question that will minimize the entropy of the joint probability distribution of the utility values. Since the joint distribution is uniform over the utility volume, the optimal question-selection procedure is that which partitions the current utility volume into two equal halves at each stage of the questionnaire. This reduces after each response. While there the entropy by are many questions that can provide this partition, our algorithm will give preference to the utility value that we are most uncertain about (whose marginal distribution has the highest entropy). This is explained in more detail in the following example. A. Example 2: 2-D Utility Volume
Table I shows the differential entropy of the beta distributions for elements of the utility vector vs. the number of prospects.
Let us consider a decision situation with corresponding to a utility volume of dimension
prospects .
ABBAS: ENTROPY METHODS FOR ADAPTIVE UTILITY ELICITATION
Fig. 3. Tracking the location of the utility vector. Preference is given to (c) over (d).
There are two uncertain elements in the utility vector, and , that lie in the shaded region shown in Fig. 3(a). From (13), the joint distribution is uniform over the utility . volume and equal to Now suppose our first question asked about the utility element, . As we have seen the optimal question-selection process is that, which reduces the probability of the search space by a half each time. Since the distribution of the utility vector is uniform over the utility volume, this process also corresponds to the one, which divides the utility volume into two geometric halves. The value of , which geometrically partitions the utility volume into two equal areas in this example, is approximately 0.3. We will call this point the break point of the utility volume. The break point can be determined geometrically by equating the area of the shaded trapezoid to the area of the blank triangle in the utility volume of Fig. 3(b). Note that this break point must also, by definition, be equal to the median of the marginal distribution of . We will use this fact in our utility elicitation algorithm for utility volumes with higher dimensions. Now if we wish to partition the utility volume into two equal ?” If the answer halves, the first question could be “is to this question is “yes,” the search space will be reduced to the shaded region in Fig. 3(b). Note also that the expression for the entropy of the joint distribution from (14) is the logarithm of this new shaded area. The updated entropy is thus . After the first question is answered, the updated marginal distributions can be obtained by marginalizing the new joint distribution, or conditioning on the new bounds for . For example, gets truncated at the value 0.3 the marginal distribution of and is normalized to integrate to unity. The updated distribution is determined by Bayes’ rule for probability inference, for
where the numerator is an integral of the uniform joint distribution over the new bounds. The updated marginal distributions can also be determined using Monte Carlo simulation, by uni-
173
formly sampling the utility volume and selecting the samples that lie in the current search space. Now we are ready for the second question, which can ei?” or “is ?” corresponding to ther be “is Figs. 3(c) and (d), both of which will also reduce the search . Our algospace by a half and reduce the entropy to ?” (since rithm will give preference to the question “is we currently have a narrower range for than for , we would like to narrow down the ranges of all utility values rather than just refine one of them). The way we achieve this criterion is by choosing the question corresponding to the utility element that we are most uncertain about. Using Shannon’s entropy definition, this element is the one whose marginal distribution, has the highest entropy. Note that the entropy of is higher after the first question is asked and thus than the entropy of gets preference. The algorithm will require keeping track of the updated marginal probability distributions for the utility vector since they will be used to determine the one with the highest entropy and also the median will represent the break point for the utility volume. As the size of the utility volume is reduced with each response, and as the marginal distributions get narrower, the estimated utility values converge asymptotically to the decision maker’s utility vector. We will now work through a full example to illustrate the application of the approach where the preference order is provided for a set of prospects and the utility values need to be elicited. B. Party Problem The party problem was introduced by Ronald Howard at Stanford University. It can be summarized as follows: Kim is interested in having a party. She has three decision alternatives: Indoors, Outdoors and on the Porch. However, she is uncertain about the weather situation, which can be sunny or rainy. The decision tree for the party problem is shown in Fig. 4. Kim faces six prospects, which she orders from best to worst as shown in Table II. To elicit Kim’s unknown utility values, we apply our algorithm in the following steps: 1) Determine the Maximum Entropy Marginal Distributions for the Utility Values: Since we have no more information about the utility values, we assume the utility vector is uniformly distributed over the utility volume and so the maximum entropy marginal distributions for the six prospects are the Beta distributions shown in Fig. 2. 2) Select the Utility Element Whose Marginal Distribution Has the Highest Entropy: As mentioned above, we will give preference to the utility value that we are most uncertain about , has the highest entropy). (whose marginal distribution, Using the results of example (1), the elements in the center of the utility vector have maximum entropy. Those elements correspond to the indoors-sunny (I-S) and indoors-rainy (I-R) prospects in our example, so we can start our questions with either one. In this example we will start with the indoors-sunny . prospect corresponding to a marginal distribution , at the Me3) Partition the Marginal Distribution, dian: Having selected the distribution with highest entropy, , we need to find its median. This will be the break point
174
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 34, NO. 2, MARCH 2004
for comparison. Note the “closeness” of the utility vector after five questions to Kim’s utility vector when only binary questions were used for the search. The entropy of the joint distribution calculated in bits, log base 2, is also shown. C. Stopping Criterion
Fig. 4.
Decision tree for the Party Problem. TABLE II RANK-ORDERED PROSPECTS FOR THE PARTY PROBLEM
The natural question now is the stopping criterion for the algorithm. We will stop when the expected number of questions needed to locate the utility vector, within a hypercube of side length equal to , has been asked. This criterion can be determined using Monte Carlo simulations where a utility vector is uniformly generated on the utility volume and the algorithm is applied until both the current utility vector and the generated utility vector lie within a hypercube of a given side length. The average number of questions needed to achieve this criterion can be calculated from the different trials. Alternatively, we can derive an approximate expression for the expected number of questions analytically by discretizing the utility volume into hyper cubes of side length , Fig. 7. We make use of the discretized entropy expression of (7) for elements.
as Substituting for (14), we get, as that divides the utility volume into two halves. Fig. 5(a) shows . The median is 0.38, the median partitioning of the which is also the breakpoint used for the probability in the utility elicitation question. 4) Ask the Binary Elicitation Question: Now we ask the following binary question “Which deal do you prefer, the “indoors-sunny” prospect for certain or a “deal” with a 0.38 chance of outdoors-sunny (best) and 0.62 chance of outdoorsrainy (worst)?” This is shown in Fig. 5(b). Note that the decision maker needs to give only one of two answers; “indoors-sunny” or the “deal,” in response to this question. Binary responses are much less stressful for a decision maker to provide than numeric utility values, especially in medical decision situations. 5) Update the Marginal Probability Distributions: Based , on the previous elicitation question, a hyper plane, divides the utility volume into two halves. The response to the question determines which half we are interested in and updates the marginal distributions. For example, if Kim’s utility , then she will choose the I-S prospect value for I-S is when the previous question is asked. This response implies that and that , the utility values for I-R, and P-S . must both be The lower bound for the utility vector is now (0, 0, 0.38, 0.38, 0.38, 1) and the upper bound is (0, 1, 1, 1, 1, 1). Note that the response to the question did not provide any updated bounds for , the utility value for P-R, however, it did update its marginal distribution. The updated distributions, obtained by a Monte Carlo simulation, are shown in Fig. 6. Table III shows the optimal question-selection process for five binary questions and the corresponding utility vector at each stage. Kim’s utility vector is also provided in the first column
(16) from
Exp. No. of Questions Disctretized entropy
Utility Volume Volume of hypercube of side length of hyper cubes in utility volume
(17)
This approximation is best when the volume of the hypercube is much smaller than the utility volume and gets worse for larger values of . Fig. 8 shows the discretized entropy for different and , obtained using this formula by taking the values of logarithm to the base 2 in the entropy expression. Note there is a peak for each curve in Fig. 8 for values of . The interpretation for this is that the size of as increases, the utility volume decreases by . The ratio while the volume of the hypercubes decreases by of the two volumes is the number of hyper cubes in the utility . This is why the volume and is a maximum at largest number of questions is needed at this point. Note that the number of hypercubes in the utility volume de. A direct consequence of creases with , for this result is that the algorithm becomes more efficient if for a given resolution , the number of prospects, , is larger than . Note also that the entropy drops to zero for a given at a given value of . This is where the volume of the hypercube is equal to the utility volume.
ABBAS: ENTROPY METHODS FOR ADAPTIVE UTILITY ELICITATION
Fig. 5.
(a) Median break point of a
175
Beta (2 3). (b) Utility elicitation question based on the median partition. ;
TABLE III CONVERGENCE TO KIM’S UTILITY VALUES FOR FIVE BINARY QUESTIONS
Fig. 6. Updated marginal distributions after the first question.
Fig. 8.
Fig. 7. Pictorial representation discretizing the utility volume into hypercubes of side length .
1
V. MODIFICATIONS TO THE ALGORITHM Now we will present several modifications to the utility elicitation algorithm that can be used in practice to improve its efficiency and accuracy. A. Modifications for Imperfect Responses The algorithm, in its current state, allows no room for inconsistency or for imperfect responses from the decision maker. The reason is that each response reduces the utility volume by half and the other half is dropped, so no utility values can be assigned to it later in the questionnaire. We can modify the algorithm to
Expected number of questions needed for different values of
1.
allow for imperfect responses from the decision maker as discussed below. At the start of each question, the break point divides the utility volume into two halves, A and B as shown in Fig. 9. Our prior probability, before each question, is 0.5 that the utility vector lies within each partition. Now let us assign a likelihood, , that the decision maker will provide the correct response to the question (if the utility vector is in partition, A, the decision maker will say A, with probability, , and if the utility vector is in partition B, the decision maker will say B with probability ). If the decision maker replies that the utility vector is in partition A in response to the first question, then the uniform distribution is redistributed over the utility volume to give a posterior for partition B. probability equal to for partition A and The posterior probability for partition A is in fact equal to the likelihood, , that the decision maker gives the correct response. After questions, the utility volume is partitioned into volumes, , with probabilities . Each volume represents a possible location for the utility vector with a probability that depends on the order of the correct and
176
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 34, NO. 2, MARCH 2004
Fig. 9. Partitioning of the utility volume with imperfect responses.
Fig. 10.
Marginal distributions after the first question for p = 0:6 and 0:9.
incorrect responses provided by the decision maker. Fig. 9 shows the partitioning of a utility volume for two questions and the corresponding probabilities for each partition. The updated marginal distributions at each stage can be obtained by a weighted average of the marginal distributions calculated from each partition using Bayes’ expansion formula questions answered
The volume of each partition can be determined by integration of the uniform joint distribution on the given bounds and can also be obtained by simulation. Now let us discuss the two extreme cases for the probability of correct response, 1) Case 1: or : In this case, there is no uncertainty about the decision maker’s response and (19) reduces to the logarithm of the current utility volume, as discussed earlier, and a reduction of the entropy by 1 bit after each response. Entropy after n questions
utility vector is in partition
Initial utility volume partition
(18)
The marginal distributions can also be determined by simulation by generating uniform samples for each possible partition and taking a weighted fraction of the samples of each partition according to its current probability. The entropy of the joint distribution can be calculated using the additive weighted decomposition property of the entropy expression as Entropy after
2) Case 2: : This represents maximum uncertainty about the decision maker’s response. In this case the increase in entropy due to the uncertainty about the partitions is balanced by the reduction in the entropy due to the partitioning of the utility volume and so there is zero net reduction in entropy after questions. This is an intuitive result and (19) reduces to Entropy after
questions
questions Initial utility volume
(19) Where, is the probability assigned to partition and is the volume of partition . Note that the first term in the right hand side expression is positive, leading to an increase in the entropy due to the imperfect responses obtained from the decision maker. The second term, however, is negative due to the decreasing entropy as the search space gets smaller each time. Note that partitions do not have equal volumes in this case since the joint distribution is no longer uniform but a weighted average of uniform distributions.
(20)
(21)
For any other value of , there is a reduction in entropy any where between 0 and 1 bit. Fig. 10 shows the updated distriand , butions after the first question for values of as well as the corresponding entropy after the first question and the utility vector assignment. B. Modifications for
-Ary Question Responses
If the decision maker is comfortable providing more than just binary responses, the algorithm can further be modified. In this case, the marginal distribution with highest entropy is chosen equal percentiles. The decision-maker and partitioned into
ABBAS: ENTROPY METHODS FOR ADAPTIVE UTILITY ELICITATION
will now select the percentile containing the utility value of the prospect she is facing. This will lead to a reduction of entropy after each response. by For example, the marginal distribution, , of the indoors-sunny prospect in the party problem, can be partitioned equal into four equal areas of probability, with intervals for to (0 to 0.243), (0.243 to 0.387), (0.387 to 0.543), and (0.543 to 1). The decision maker is now asked to select the interval lies. We know Kim’s value for is 0.57 so she in which will choose the fourth interval. This response reduces the utility volume into a quarter of its initial volume and so the entropy, bits instead calculated in log base 2, is reduced by of 1 using the binary partition. C. Modifications for Initial Information The algorithm can also incorporate additional prior information about the decision maker’s preferences, such as initial bounds on utility values by reducing the initial search volume in accordance with this given preference information. For example, let us assume we had prior information on Kim’s utility value for the Indoors-Sunny prospect that it is greater than or equal to 0.2. This information is used as an initial lower bound on the utility value of the indoors-sunny prospect as well as all the prospects that are preferred to it. Fig. 11 shows the convergence to Kim’s utility values for 1, 3, and 4 binary questions. Further modifications for prior information may include starting with nonuniform distributions for the location of the utility vector on the utility volume, if there is preference information to support it. One example where this may occur is when the decision maker belongs to a certain population of decision makers with a given range of preferences. VI. CONCLUSION In this paper we used entropy-coding principles to determine an optimal question-selection algorithm to elicit utility values for a set of ordered prospects. The order of the prospects provides the key for the geometric representation of utility elicitation used in this approach. The algorithm facilitates the elicitation process for the decision maker in three main directions: simplicity, efficiency, and adaptivity. The first direction (simplicity) is achieved by using easier types of questions such as binary questions or higher orders based on the comfort level of the decision maker. In addition the break points used for the “Von Neumann and Morgenstern” questions are, quite often, far from the actual utility values of the decision maker. For example, Kim’s utility value for the Indoors-Sunny prospect in Table II is 0.57, while the median break point for the binary question that was asked is 0.38. The second direction (efficiency) is achieved where the marginal distributions for all utility values are updated in response to a question (not just the marginal distribution for the question that was answered). Efficiency is also achieved by partitioning the utility volume into two equal halves each time, so the minimum expected number of questions is asked. The third direction (adaptivity) is achieved where each new question is selected based on the previous response provided by the decision maker.
177
Fig. 11.
Convergence to Kim’s utility values with an initial lower bound.
The utility elicitation algorithm presented in this paper is well suited for computer-based elicitations where calculations can be made in real time. The calculations for this algorithm can also be performed in advance for a given decision situation and referred to based on the response obtained from the decision maker. The algorithm can also be applied to cases where prospects can be expressed monetarily and the rank order is a simple task provided the decision maker prefers more money to less. Finally, we note that the algorithm can be applied to many other applications where subjective values need to be elicited for a set of ordered samples. One common example includes eliciting quality curves for voice messages transmitted over networks. In these experiments, the quality of the signal is varied by changing its attenuation, noise level, or speed and a rating for each sample is elicited. The algorithm can take the order of the samples and use binary questions to elicit their rating. ACKNOWLEDGMENT The author would like to thank R. A. Howard at Stanford University and the three anonymous reviewers for their many helpful suggestions. REFERENCES [1] P. Korhonen, J. Wallenius, and S. Zionts, “Solving the discrete multiple criteria problem using convex cones,” Manage. Sci., vol. 30, no. 11, pp. 1336–1345, 1984. [2] H. R. Rao, “A choice set-sensitive analysis of preference information acquisition about discrete resources,” IEEE Trans. Syst., Man, Cybern., vol. 23, pp. 1062–1071, July/Aug. 1993. [3] S. Y. Prasad, M. H. Karwan, and S. Zionts, “Use of convex cones in interactive multiple objective decision making,” Manage. Sci., vol. 43, no. 5, pp. 723–734, 1997. [4] V. Ha and P. Haddawy, “Hybrid approach to reasoning with partial preference models,” in Proc. 15th Conf. Uncertainty Artificial Intelligence, 1999, pp. 263–270. [5] H. A. Holloway and C. C. White III, “Question selection for multi-attribute decision aiding,” Eur. J. Oper. Res., vol. 148, no. 3, pp. 525–534, 2003. [6] C. Kirkwood and R. Sarin, “Ranking with partial information: A method and an application,” Oper. Res., vol. 33, no. 1, pp. 38–48, 1985. [7] J. Von Neumann and O. Morgenstern, Theory of Games and Economic Behavior, 2nd ed. Princeton, NJ: Princeton Univ., 1947. [8] R. L. Keeney and H. Raiffa, Decisions with Multiple Objectives: Preferences and Value Tradeoffs. New York: Wiley, 1976. [9] D. G. Froberg and R. L. Kane, “Methodology for measuring health-state preferences—II: Scaling methods,” J. Clin. Epidemiol., vol. 42, no. 5, pp. 459–471, 1989. [10] U. Chajewska, D. Koller, and R. Parr, “Making rational decisions using adaptive utillity elicitation,” in Proc. 17th Nat. Conf. Artificial Intelligence, Austin, TX, Aug. 2000, pp. 363–369. [11] R. A. Howard, “Information value theory,” Syst., Sci. Cybern., vol. SSC-2, pp. 22–26, 1966.
178
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 34, NO. 2, MARCH 2004
[12] C. E. Shannon, “A mathematical theory of communication,” Bell Syst. Tech. J., vol. 27, pp. 379–423, 623–656, 1948. [13] T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991, p. 87. [14] E. T. Jaynes, “Information theory and statistical mechanics,” Phys. Rev., vol. 106, pp. 620–630, 1957. [15] A. E. Abbas, “An entropy approach for utility assignment in decision analysis,” in Proc. 22nd Int. Workshop Bayesian Inference Maximum Entropy Methods Science Engineering, C. Williams, Ed., Moscow, ID, 2002. [16] H. A. David, Order Statistics. New York: Wiley, 1981. [17] R. A. Howard, “Decision analysis: Perspectives on inference, decision, and experimentation,” Proc. IEEE, vol. 58, pp. 632–643, May 1970. [18] A. C. Lazo and P. N. Rathie, “On the entropy of continuous probability distributions,” IEEE Trans. Inform. Theory, vol. IT-12, pp. 75–77, Jan. 1966.
Ali E. Abbas (M’95) received the M.S. degrees in electrical engineering, the M.S. degree in engineering economic and systems operations research, and the Ph.D. degree in management science and engineering, with a minor in electrical engineering, from Stanford University, Stanford, CA. in 1998, 2002, and 2003, respectively. He is a Lecturer in the Department of Management Science and Engineering at Stanford University. He has worked in Schlumberger Oil Field Services, Oman, Turkey, Dubai, Paris, and Egypt, from 1991 to 1997, where he held several positions in wireline logging, operations management, and international training. He has also worked on several consulting projects for mergers and acquisitions in California, and co-taught several executive seminars on decision analysis at Strategic Decisions Group, Menlo Park, NJ. Dr. Abbas is a member of the Institute for Operations Research and the Management Sciences.
169
Entropy Methods for Adaptive Utility Elicitation Ali E. Abbas, Member, IEEE
Abstract—This paper presents an optimal question-selection algorithm to elicit von Neumann and Morgenstern utility values for a set of ordered prospects of a decision situation. The approach uses information theory and entropy-coding principles to select the minimum expected number of questions needed for utility elicitation. At each stage of the questionnaire, we use the question that will provide the largest reduction in the entropy of the joint distribution of the utility values. The algorithm uses questions that require binary responses, which are easier to provide than numeric values, and uses an adaptive question-selection scheme where each new question depends on the previous response obtained from the decision maker. We present a geometric interpretation for utility elicitation and work through a full example to illustrate the approach. Index Terms—Maximum entropy, question-selection, utility.
I. INTRODUCTION N MANY decision problems, we are faced with multiple and conflicting attributes. When the decision situation is deterministic, a decision alternative is characterized by a single prospect1 . The decision problem is resolved by ranking the prospects present, or assigning a value function over the attributes of the decision situation. The optimal decision alternative corresponds to the prospect that has the highest preference order in the ranked list, or the largest value for the value function. Several question-selection algorithms have been proposed in the literature to rank-order prospects of a decision situation. Many of these algorithms focus on determining a tradeoff weight vector for prospects with multiple attributes. For example, Korhonen et al. used pairwise comparison questions to eliminate decision alternatives and determine a tradeoff weight vector using convex cones [1]. Rao proposed an optimal sequence of pairwise comparison questions over a set of alternatives with two attributes [2]. Prasad et al. generalized the convex cones approach and introduced the idea of p-cones to minimize the preference information requirements [3]. Ha et al. assumed a partially specified multilinear function and used comparative statements to further constrain the value function and identify suboptimal alternatives [4]. Holloway and White used a dynamic programming approach to minimize the expected number of questions needed to determine a tradeoff weight vector for alternatives with multiple attributes [5].
I
Manuscript received April 17, 2003; revised August 11, 2003 and September 30, 2003. This paper was recommended by Associate Editor J. B. Yang. The author is with the Department of Management Science and Engineering, Stanford University, Stanford, CA 94305 USA (e-mail: [email protected].). Digital Object Identifier 10.1109/TSMCA.2003.822269 1We will use the term “prospect” rather than “outcome” or “consequence” of a decision situation as it refers to the whole future life starting at the end of the decision tree rather than an outcome of a situation.
Algorithms for ranking prospects when partial information is available about the rank order have also been proposed. For example, Kirkwood and Sarin presented a method to rank-order prospects of a decision situation using a weighted additive value function when partial information is available about the weights [6]. When uncertainty is present, a decision alternative is characterized by several prospects and a probability distribution for their occurrence. In this case, the rank order of the prospects is not sufficient to determine the optimal decision alternative and, in addition, we need to elicit the von Neumann and Morgenstern utility values [7]. To elicit those utility values, the first step is to rank-order the prospects of the decision situation using many of the algorithms cited above. The second step is to assign a “von Neumann and Morgenstern” utility value to each prospect as follows. For any , the decision maker asthree ordered prospects, signs a probability, , for which she is indifferent to receiving for sure and a deal where she would receive with probability and with probability . If the utility values of and are 1 and 0, respectively, then the utility value of , also called the preference probability of , will be equal to . The optimal decision alternative is the one, which has the highest expected utility. Both the rank order and the von Neumann and Morgenstern utility values are thus essential in the analysis of decision problems when uncertainty is present. Unfortunately, the utility elicitation process is subject to many cognitive and motivational biases (Keeney and Raiffa [8] and Froberg and Kane [9]). In addition, the number of prospects of a decision situation grows exponentially with the number of uncertainties present, and so the utility elicitation process can become a time-consuming and stressful task for the decision maker. For this reason several algorithms for eliciting von Neuman and Morgenstern utilities2 have been proposed in the literature. For example, Chajewska et al. [10] used pairwise comparison questions that maximize the “value of information,” [11], each question will provide. In this paper, we propose a question-selection algorithm to elicit utility values for a set of ordered prospects, which may have single or multiple attributes. We use the order provided to determine a joint distribution for the utility values and choose the questions that will provide the largest reduction in the entropy of this joint distribution. The algorithm takes into account the difficulty of the elicitation process as a practical constraint on the side of the decision maker and uses questions that require binary (yes/no) responses, which are easier to provide than numeric values. The approach is adaptive so each new question is determined by the previous response obtained from the decision maker. 2Throughout the rest of the paper we will refer to the “von Neumann and Morgenstern utility” as simply “utility.”
1083-4427/04$20.00 © 2004 IEEE
170
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 34, NO. 2, MARCH 2004
This paper is structured as follows. Section II provides a review of the entropy of a random variable and the basic formulas that will be used in the remaining sections of the paper. Section III derives the marginal probability distributions for the ordered prospects and introduces two defiutility values of nitions—the utility vector and utility volume—that will be used in the formulation of the utility elicitation problem. Section IV introduces a geometric representation for utility elicitation, and presents an optimal question-selection algorithm to elicit utility values from the viewpoint of information theory. Section V extends the approach to allow for imperfect responses from the decision maker and to -ary type questions. II. REVIEW OF BASIC ENTROPY CONCEPTS In 1948, Claude Shannon introduced the term, , as a measure of uncertainty about a discrete random variable having a probability mass function [12]. He called this term the entropy, which can also be thought of as a measure of the amount of information needed to describe the outcome of a discrete random variable. For example, a random variable, , can have four possible outcomes, , with probabilities, , respectively. Let us calculate the entropy of X using base two for the logarithm in the entropy expression.
(1) One intuitive way to explain this number is to consider the minimum expected number of binary (yes/no) questions needed to describe an outcome of . The most efficient question-selection procedure in this example is to start with the outcome that has the highest probability of occurrence, i.e., we ask “is ?” If it is, then we have determined in one question. If it is not, then we ask “is ?” Again, if it is correct, then we have determined in two questions, if it is not, we ask “is ?” We do not need to ask “is ?” because if it is not 0, 1, or 2, then it must be 3. The expected number of binary3 questions needed to determine is then
Entropy of the random variable
number of questions needed is bounded by the following wellknown data compression inequality [13] (3) The entropy of a discrete random variable is thus the lower bound on the expected number of questions needed to describe its outcome, and the lower bound is achieved when the probability of the search space is reduced by a half each time. We will make use of this notion of entropy to calculate the expected number of questions needed for our utility elicitation algorithm. Another important property of the entropy expression is its expansion in terms of a weighted sum of the entropy of the conditional distributions. The following example, taken from Shannon’s original paper, illustrates this property. A has three possible outcomes, , and random variable with probability . If we decompose in terms of and , with probability , and assign condition probability for and equal to and respectively, then the entropy of can be calculated from either representation, i.e., . We will use this property to determine the expected number of questions needed for utility elicitation when the decision maker provides imperfect responses. In continuous form, the integral is known as the differential entropy of a random variable having a probability density function, . As opposed to the discrete case, however, the differential entropy can be negative. This is illustrated by the following example, where can be positive or negative depending on the value of .
(4) The possibility of a negative sign shows the absence of the expected number of questions interpretation for entropy in the continuous case. However, we can relate the differential entropy to the discrete entropy by discretizing the continuous density into intervals of length and using the substitution in the discrete form entropy expression as follows:
(2)
This equality always holds when the question-selection process reduces the probability of the possible values by a half each time. If we look at the probability mass function used for this example, we note that each question asked did in fact reduce the probability by a half. While this equality does not always hold, an optimal question-selection process always exists (for any probability mass function) such that the minimum expected
3The use of binary questions in this example relates to the base two in the logarithm of the entropy expression.
(5) Entropy of the discretized variable. where If is Riemann integrable, then the first term on the right hand side of (2.5) approaches the integral of by definition of Riemann integrability. Thus (6) By induction, for a joint distribution of
variables, we have
(7)
ABBAS: ENTROPY METHODS FOR ADAPTIVE UTILITY ELICITATION
171
where Entropy of discretized joint distribution. Differential entropy of continuous joint distribution. In 1957, Edwin Jaynes built on the entropy concept and proposed the use of a distribution that maximizes Shannon’s entropy measure and satisfies the partial information constraints as an unbiased probability distribution [14]. Jaynes’ proposition is known as the maximum entropy principle and is considered to be an extension of Laplace’s principle of insufficient reason. III. UTILITY VOLUME We start this section with the definition of a utility vector for a set of ordered prospects. A utility vector contains the utility values of the prospects starting from lowest to highest [15]. Without significant loss of generality, we will assign a utility value of zero to the least preferred prospect, , and a value of (we neglect the case one to the most preferred prospect, of absolute indifference between the prospects since it merely assigns them all the same utility value). The utility vector has elements defined as
Fig. 1.
Utility volume and utility vector representation for
K 0 2 = 2.
Now we discuss the implications this uniform sampling has on the probability density functions for the elements of the utility vector. Let us start with the element .
(9) (10)
(11)
(8) Note that any utility vector of dimension can be repredimensional space in the region sented as a point in a . This redefined by gion, which we will call the utility volume, has a volume equal . to An example of a two-dimensional (2–D) utility volume for is shown in Fig. 1. The first and last elements of the utility vector are not represented since they are deterministic. Note that all points in the utility volume satisfy the decision maker’s preference ordering of the prospects but assign different utility values to them. In other words, if a decision analyst would like to elicit the utility values, then knowledge of the preference ordering alone tells him nothing about the location of the utility vector in the utility volume. If only the preference order is known, it is reasonable to assume, therefore, that the location of the utility vector is uniformly distributed over the utility volume. This represents maximum uncertainty about the decision-maker’s utility values. Before we proceed with the implications of this result, let us show how to generate uniform samples . over a utility volume of dimension A. Uniform Sampling Over the Utility Volume from a 1) Generate a random sequence . distribution, 2) Sort the generated sequence from lowest to highest and rename the variables such that . , form the uncertain elements The elements of the utility vector and also form an order statistic generated by a uniform distribution.
Using further analysis, and properties of order statistics, [16] the probability density function for , is
(12) where and Beta probability density function Given the marginal distribution for each utility value, the logical point estimate assignment is the mean of its marginal distribution (see for example [17]). We summarize these results below. B. Utility Vector: Marginal Distribution and Assignment When only the preference order is known, the marginal distributions for the utility values of a set of ordered prospects are the family of Beta distributions, . The mean of each distribution, , is the utility value you would assign to each prospect, j. The utility and are deterministic with values for prospects values 0 and 1 respectively. Note that this assignment may be different when different cost functions are available. For example, the optimal assignment using a square law cost function is the still mean of the marginal distribution, whereas the optimal assignment using an absolute error cost function is the median, and that using a delta impulse cost function is the mode of the marginal distribution.
172
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 34, NO. 2, MARCH 2004
Fig. 2. Marginal distributions for elements of the utility vector.
Further analysis and properties of order statistics show that the joint distribution of an order statistic generated by a common Uniform , is uniform over the utility volume, density, and is equal to
TABLE I OF THE MARGINAL DISTRIBUTIONS OF THE UTILITY VECTOR
DIFFERENTIAL ENTROPY
(13) The entropy of the joint distribution is thus
utility volume
(14)
We illustrate the previous results with the following example.
The entropy of the marginal distributions in our example corre. spond to the row IV. GEOMETRIC REPRESENTATION OF UTILITY ELICITATION
C. Example 1: Six Ordered Prospects prospects. She states her prefA decision-maker faces erence ordering for them as follows:
The utility values for prospects A and F are 1 and 0, while the marginal distributions for prospects E, D, C, and B are Beta (1, 4), Beta (2, 3), Beta (3, 2), and Beta (4, 1), respectively. These distributions are shown in Fig. 2. The maximum entropy utility vector is (0, 0.2, 0.4, 0.6, 0.8, 1). The joint distribution is uniform and equal to 4! over the utility volume . The entropy of the joint distribution is . It is important to note, from the curves shown above, that the elements in the center of the utility vector have maximum entropy. This can also be verified using the differential entropy , derived in [18] as expression of a beta distribution, (15) where
In this section we will apply the previous results to the utility elicitation problem. Given ordered prospects, the problem of eliciting the decision-maker’s utility values is identical to that of searching for the utility vector over the utility volume. We will use “von Neumann and Morgenstern” type questions for the search and yet not have the decision-maker place precise utility values. The approach will have several binary questions of the form “which deal do you prefer this or that?” The response to a question is used to update the distributions of all the utility values by truncating them at the value of the preference probability used for that question. At each stage we choose the question that will minimize the entropy of the joint probability distribution of the utility values. Since the joint distribution is uniform over the utility volume, the optimal question-selection procedure is that which partitions the current utility volume into two equal halves at each stage of the questionnaire. This reduces after each response. While there the entropy by are many questions that can provide this partition, our algorithm will give preference to the utility value that we are most uncertain about (whose marginal distribution has the highest entropy). This is explained in more detail in the following example. A. Example 2: 2-D Utility Volume
Table I shows the differential entropy of the beta distributions for elements of the utility vector vs. the number of prospects.
Let us consider a decision situation with corresponding to a utility volume of dimension
prospects .
ABBAS: ENTROPY METHODS FOR ADAPTIVE UTILITY ELICITATION
Fig. 3. Tracking the location of the utility vector. Preference is given to (c) over (d).
There are two uncertain elements in the utility vector, and , that lie in the shaded region shown in Fig. 3(a). From (13), the joint distribution is uniform over the utility . volume and equal to Now suppose our first question asked about the utility element, . As we have seen the optimal question-selection process is that, which reduces the probability of the search space by a half each time. Since the distribution of the utility vector is uniform over the utility volume, this process also corresponds to the one, which divides the utility volume into two geometric halves. The value of , which geometrically partitions the utility volume into two equal areas in this example, is approximately 0.3. We will call this point the break point of the utility volume. The break point can be determined geometrically by equating the area of the shaded trapezoid to the area of the blank triangle in the utility volume of Fig. 3(b). Note that this break point must also, by definition, be equal to the median of the marginal distribution of . We will use this fact in our utility elicitation algorithm for utility volumes with higher dimensions. Now if we wish to partition the utility volume into two equal ?” If the answer halves, the first question could be “is to this question is “yes,” the search space will be reduced to the shaded region in Fig. 3(b). Note also that the expression for the entropy of the joint distribution from (14) is the logarithm of this new shaded area. The updated entropy is thus . After the first question is answered, the updated marginal distributions can be obtained by marginalizing the new joint distribution, or conditioning on the new bounds for . For example, gets truncated at the value 0.3 the marginal distribution of and is normalized to integrate to unity. The updated distribution is determined by Bayes’ rule for probability inference, for
where the numerator is an integral of the uniform joint distribution over the new bounds. The updated marginal distributions can also be determined using Monte Carlo simulation, by uni-
173
formly sampling the utility volume and selecting the samples that lie in the current search space. Now we are ready for the second question, which can ei?” or “is ?” corresponding to ther be “is Figs. 3(c) and (d), both of which will also reduce the search . Our algospace by a half and reduce the entropy to ?” (since rithm will give preference to the question “is we currently have a narrower range for than for , we would like to narrow down the ranges of all utility values rather than just refine one of them). The way we achieve this criterion is by choosing the question corresponding to the utility element that we are most uncertain about. Using Shannon’s entropy definition, this element is the one whose marginal distribution, has the highest entropy. Note that the entropy of is higher after the first question is asked and thus than the entropy of gets preference. The algorithm will require keeping track of the updated marginal probability distributions for the utility vector since they will be used to determine the one with the highest entropy and also the median will represent the break point for the utility volume. As the size of the utility volume is reduced with each response, and as the marginal distributions get narrower, the estimated utility values converge asymptotically to the decision maker’s utility vector. We will now work through a full example to illustrate the application of the approach where the preference order is provided for a set of prospects and the utility values need to be elicited. B. Party Problem The party problem was introduced by Ronald Howard at Stanford University. It can be summarized as follows: Kim is interested in having a party. She has three decision alternatives: Indoors, Outdoors and on the Porch. However, she is uncertain about the weather situation, which can be sunny or rainy. The decision tree for the party problem is shown in Fig. 4. Kim faces six prospects, which she orders from best to worst as shown in Table II. To elicit Kim’s unknown utility values, we apply our algorithm in the following steps: 1) Determine the Maximum Entropy Marginal Distributions for the Utility Values: Since we have no more information about the utility values, we assume the utility vector is uniformly distributed over the utility volume and so the maximum entropy marginal distributions for the six prospects are the Beta distributions shown in Fig. 2. 2) Select the Utility Element Whose Marginal Distribution Has the Highest Entropy: As mentioned above, we will give preference to the utility value that we are most uncertain about , has the highest entropy). (whose marginal distribution, Using the results of example (1), the elements in the center of the utility vector have maximum entropy. Those elements correspond to the indoors-sunny (I-S) and indoors-rainy (I-R) prospects in our example, so we can start our questions with either one. In this example we will start with the indoors-sunny . prospect corresponding to a marginal distribution , at the Me3) Partition the Marginal Distribution, dian: Having selected the distribution with highest entropy, , we need to find its median. This will be the break point
174
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 34, NO. 2, MARCH 2004
for comparison. Note the “closeness” of the utility vector after five questions to Kim’s utility vector when only binary questions were used for the search. The entropy of the joint distribution calculated in bits, log base 2, is also shown. C. Stopping Criterion
Fig. 4.
Decision tree for the Party Problem. TABLE II RANK-ORDERED PROSPECTS FOR THE PARTY PROBLEM
The natural question now is the stopping criterion for the algorithm. We will stop when the expected number of questions needed to locate the utility vector, within a hypercube of side length equal to , has been asked. This criterion can be determined using Monte Carlo simulations where a utility vector is uniformly generated on the utility volume and the algorithm is applied until both the current utility vector and the generated utility vector lie within a hypercube of a given side length. The average number of questions needed to achieve this criterion can be calculated from the different trials. Alternatively, we can derive an approximate expression for the expected number of questions analytically by discretizing the utility volume into hyper cubes of side length , Fig. 7. We make use of the discretized entropy expression of (7) for elements.
as Substituting for (14), we get, as that divides the utility volume into two halves. Fig. 5(a) shows . The median is 0.38, the median partitioning of the which is also the breakpoint used for the probability in the utility elicitation question. 4) Ask the Binary Elicitation Question: Now we ask the following binary question “Which deal do you prefer, the “indoors-sunny” prospect for certain or a “deal” with a 0.38 chance of outdoors-sunny (best) and 0.62 chance of outdoorsrainy (worst)?” This is shown in Fig. 5(b). Note that the decision maker needs to give only one of two answers; “indoors-sunny” or the “deal,” in response to this question. Binary responses are much less stressful for a decision maker to provide than numeric utility values, especially in medical decision situations. 5) Update the Marginal Probability Distributions: Based , on the previous elicitation question, a hyper plane, divides the utility volume into two halves. The response to the question determines which half we are interested in and updates the marginal distributions. For example, if Kim’s utility , then she will choose the I-S prospect value for I-S is when the previous question is asked. This response implies that and that , the utility values for I-R, and P-S . must both be The lower bound for the utility vector is now (0, 0, 0.38, 0.38, 0.38, 1) and the upper bound is (0, 1, 1, 1, 1, 1). Note that the response to the question did not provide any updated bounds for , the utility value for P-R, however, it did update its marginal distribution. The updated distributions, obtained by a Monte Carlo simulation, are shown in Fig. 6. Table III shows the optimal question-selection process for five binary questions and the corresponding utility vector at each stage. Kim’s utility vector is also provided in the first column
(16) from
Exp. No. of Questions Disctretized entropy
Utility Volume Volume of hypercube of side length of hyper cubes in utility volume
(17)
This approximation is best when the volume of the hypercube is much smaller than the utility volume and gets worse for larger values of . Fig. 8 shows the discretized entropy for different and , obtained using this formula by taking the values of logarithm to the base 2 in the entropy expression. Note there is a peak for each curve in Fig. 8 for values of . The interpretation for this is that the size of as increases, the utility volume decreases by . The ratio while the volume of the hypercubes decreases by of the two volumes is the number of hyper cubes in the utility . This is why the volume and is a maximum at largest number of questions is needed at this point. Note that the number of hypercubes in the utility volume de. A direct consequence of creases with , for this result is that the algorithm becomes more efficient if for a given resolution , the number of prospects, , is larger than . Note also that the entropy drops to zero for a given at a given value of . This is where the volume of the hypercube is equal to the utility volume.
ABBAS: ENTROPY METHODS FOR ADAPTIVE UTILITY ELICITATION
Fig. 5.
(a) Median break point of a
175
Beta (2 3). (b) Utility elicitation question based on the median partition. ;
TABLE III CONVERGENCE TO KIM’S UTILITY VALUES FOR FIVE BINARY QUESTIONS
Fig. 6. Updated marginal distributions after the first question.
Fig. 8.
Fig. 7. Pictorial representation discretizing the utility volume into hypercubes of side length .
1
V. MODIFICATIONS TO THE ALGORITHM Now we will present several modifications to the utility elicitation algorithm that can be used in practice to improve its efficiency and accuracy. A. Modifications for Imperfect Responses The algorithm, in its current state, allows no room for inconsistency or for imperfect responses from the decision maker. The reason is that each response reduces the utility volume by half and the other half is dropped, so no utility values can be assigned to it later in the questionnaire. We can modify the algorithm to
Expected number of questions needed for different values of
1.
allow for imperfect responses from the decision maker as discussed below. At the start of each question, the break point divides the utility volume into two halves, A and B as shown in Fig. 9. Our prior probability, before each question, is 0.5 that the utility vector lies within each partition. Now let us assign a likelihood, , that the decision maker will provide the correct response to the question (if the utility vector is in partition, A, the decision maker will say A, with probability, , and if the utility vector is in partition B, the decision maker will say B with probability ). If the decision maker replies that the utility vector is in partition A in response to the first question, then the uniform distribution is redistributed over the utility volume to give a posterior for partition B. probability equal to for partition A and The posterior probability for partition A is in fact equal to the likelihood, , that the decision maker gives the correct response. After questions, the utility volume is partitioned into volumes, , with probabilities . Each volume represents a possible location for the utility vector with a probability that depends on the order of the correct and
176
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 34, NO. 2, MARCH 2004
Fig. 9. Partitioning of the utility volume with imperfect responses.
Fig. 10.
Marginal distributions after the first question for p = 0:6 and 0:9.
incorrect responses provided by the decision maker. Fig. 9 shows the partitioning of a utility volume for two questions and the corresponding probabilities for each partition. The updated marginal distributions at each stage can be obtained by a weighted average of the marginal distributions calculated from each partition using Bayes’ expansion formula questions answered
The volume of each partition can be determined by integration of the uniform joint distribution on the given bounds and can also be obtained by simulation. Now let us discuss the two extreme cases for the probability of correct response, 1) Case 1: or : In this case, there is no uncertainty about the decision maker’s response and (19) reduces to the logarithm of the current utility volume, as discussed earlier, and a reduction of the entropy by 1 bit after each response. Entropy after n questions
utility vector is in partition
Initial utility volume partition
(18)
The marginal distributions can also be determined by simulation by generating uniform samples for each possible partition and taking a weighted fraction of the samples of each partition according to its current probability. The entropy of the joint distribution can be calculated using the additive weighted decomposition property of the entropy expression as Entropy after
2) Case 2: : This represents maximum uncertainty about the decision maker’s response. In this case the increase in entropy due to the uncertainty about the partitions is balanced by the reduction in the entropy due to the partitioning of the utility volume and so there is zero net reduction in entropy after questions. This is an intuitive result and (19) reduces to Entropy after
questions
questions Initial utility volume
(19) Where, is the probability assigned to partition and is the volume of partition . Note that the first term in the right hand side expression is positive, leading to an increase in the entropy due to the imperfect responses obtained from the decision maker. The second term, however, is negative due to the decreasing entropy as the search space gets smaller each time. Note that partitions do not have equal volumes in this case since the joint distribution is no longer uniform but a weighted average of uniform distributions.
(20)
(21)
For any other value of , there is a reduction in entropy any where between 0 and 1 bit. Fig. 10 shows the updated distriand , butions after the first question for values of as well as the corresponding entropy after the first question and the utility vector assignment. B. Modifications for
-Ary Question Responses
If the decision maker is comfortable providing more than just binary responses, the algorithm can further be modified. In this case, the marginal distribution with highest entropy is chosen equal percentiles. The decision-maker and partitioned into
ABBAS: ENTROPY METHODS FOR ADAPTIVE UTILITY ELICITATION
will now select the percentile containing the utility value of the prospect she is facing. This will lead to a reduction of entropy after each response. by For example, the marginal distribution, , of the indoors-sunny prospect in the party problem, can be partitioned equal into four equal areas of probability, with intervals for to (0 to 0.243), (0.243 to 0.387), (0.387 to 0.543), and (0.543 to 1). The decision maker is now asked to select the interval lies. We know Kim’s value for is 0.57 so she in which will choose the fourth interval. This response reduces the utility volume into a quarter of its initial volume and so the entropy, bits instead calculated in log base 2, is reduced by of 1 using the binary partition. C. Modifications for Initial Information The algorithm can also incorporate additional prior information about the decision maker’s preferences, such as initial bounds on utility values by reducing the initial search volume in accordance with this given preference information. For example, let us assume we had prior information on Kim’s utility value for the Indoors-Sunny prospect that it is greater than or equal to 0.2. This information is used as an initial lower bound on the utility value of the indoors-sunny prospect as well as all the prospects that are preferred to it. Fig. 11 shows the convergence to Kim’s utility values for 1, 3, and 4 binary questions. Further modifications for prior information may include starting with nonuniform distributions for the location of the utility vector on the utility volume, if there is preference information to support it. One example where this may occur is when the decision maker belongs to a certain population of decision makers with a given range of preferences. VI. CONCLUSION In this paper we used entropy-coding principles to determine an optimal question-selection algorithm to elicit utility values for a set of ordered prospects. The order of the prospects provides the key for the geometric representation of utility elicitation used in this approach. The algorithm facilitates the elicitation process for the decision maker in three main directions: simplicity, efficiency, and adaptivity. The first direction (simplicity) is achieved by using easier types of questions such as binary questions or higher orders based on the comfort level of the decision maker. In addition the break points used for the “Von Neumann and Morgenstern” questions are, quite often, far from the actual utility values of the decision maker. For example, Kim’s utility value for the Indoors-Sunny prospect in Table II is 0.57, while the median break point for the binary question that was asked is 0.38. The second direction (efficiency) is achieved where the marginal distributions for all utility values are updated in response to a question (not just the marginal distribution for the question that was answered). Efficiency is also achieved by partitioning the utility volume into two equal halves each time, so the minimum expected number of questions is asked. The third direction (adaptivity) is achieved where each new question is selected based on the previous response provided by the decision maker.
177
Fig. 11.
Convergence to Kim’s utility values with an initial lower bound.
The utility elicitation algorithm presented in this paper is well suited for computer-based elicitations where calculations can be made in real time. The calculations for this algorithm can also be performed in advance for a given decision situation and referred to based on the response obtained from the decision maker. The algorithm can also be applied to cases where prospects can be expressed monetarily and the rank order is a simple task provided the decision maker prefers more money to less. Finally, we note that the algorithm can be applied to many other applications where subjective values need to be elicited for a set of ordered samples. One common example includes eliciting quality curves for voice messages transmitted over networks. In these experiments, the quality of the signal is varied by changing its attenuation, noise level, or speed and a rating for each sample is elicited. The algorithm can take the order of the samples and use binary questions to elicit their rating. ACKNOWLEDGMENT The author would like to thank R. A. Howard at Stanford University and the three anonymous reviewers for their many helpful suggestions. REFERENCES [1] P. Korhonen, J. Wallenius, and S. Zionts, “Solving the discrete multiple criteria problem using convex cones,” Manage. Sci., vol. 30, no. 11, pp. 1336–1345, 1984. [2] H. R. Rao, “A choice set-sensitive analysis of preference information acquisition about discrete resources,” IEEE Trans. Syst., Man, Cybern., vol. 23, pp. 1062–1071, July/Aug. 1993. [3] S. Y. Prasad, M. H. Karwan, and S. Zionts, “Use of convex cones in interactive multiple objective decision making,” Manage. Sci., vol. 43, no. 5, pp. 723–734, 1997. [4] V. Ha and P. Haddawy, “Hybrid approach to reasoning with partial preference models,” in Proc. 15th Conf. Uncertainty Artificial Intelligence, 1999, pp. 263–270. [5] H. A. Holloway and C. C. White III, “Question selection for multi-attribute decision aiding,” Eur. J. Oper. Res., vol. 148, no. 3, pp. 525–534, 2003. [6] C. Kirkwood and R. Sarin, “Ranking with partial information: A method and an application,” Oper. Res., vol. 33, no. 1, pp. 38–48, 1985. [7] J. Von Neumann and O. Morgenstern, Theory of Games and Economic Behavior, 2nd ed. Princeton, NJ: Princeton Univ., 1947. [8] R. L. Keeney and H. Raiffa, Decisions with Multiple Objectives: Preferences and Value Tradeoffs. New York: Wiley, 1976. [9] D. G. Froberg and R. L. Kane, “Methodology for measuring health-state preferences—II: Scaling methods,” J. Clin. Epidemiol., vol. 42, no. 5, pp. 459–471, 1989. [10] U. Chajewska, D. Koller, and R. Parr, “Making rational decisions using adaptive utillity elicitation,” in Proc. 17th Nat. Conf. Artificial Intelligence, Austin, TX, Aug. 2000, pp. 363–369. [11] R. A. Howard, “Information value theory,” Syst., Sci. Cybern., vol. SSC-2, pp. 22–26, 1966.
178
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A: SYSTEMS AND HUMANS, VOL. 34, NO. 2, MARCH 2004
[12] C. E. Shannon, “A mathematical theory of communication,” Bell Syst. Tech. J., vol. 27, pp. 379–423, 623–656, 1948. [13] T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991, p. 87. [14] E. T. Jaynes, “Information theory and statistical mechanics,” Phys. Rev., vol. 106, pp. 620–630, 1957. [15] A. E. Abbas, “An entropy approach for utility assignment in decision analysis,” in Proc. 22nd Int. Workshop Bayesian Inference Maximum Entropy Methods Science Engineering, C. Williams, Ed., Moscow, ID, 2002. [16] H. A. David, Order Statistics. New York: Wiley, 1981. [17] R. A. Howard, “Decision analysis: Perspectives on inference, decision, and experimentation,” Proc. IEEE, vol. 58, pp. 632–643, May 1970. [18] A. C. Lazo and P. N. Rathie, “On the entropy of continuous probability distributions,” IEEE Trans. Inform. Theory, vol. IT-12, pp. 75–77, Jan. 1966.
Ali E. Abbas (M’95) received the M.S. degrees in electrical engineering, the M.S. degree in engineering economic and systems operations research, and the Ph.D. degree in management science and engineering, with a minor in electrical engineering, from Stanford University, Stanford, CA. in 1998, 2002, and 2003, respectively. He is a Lecturer in the Department of Management Science and Engineering at Stanford University. He has worked in Schlumberger Oil Field Services, Oman, Turkey, Dubai, Paris, and Egypt, from 1991 to 1997, where he held several positions in wireline logging, operations management, and international training. He has also worked on several consulting projects for mergers and acquisitions in California, and co-taught several executive seminars on decision analysis at Strategic Decisions Group, Menlo Park, NJ. Dr. Abbas is a member of the Institute for Operations Research and the Management Sciences.
