An Approximate Pareto Optimal Cooperative ... - Semantic Scholar
An Approximate Pareto Optimal Cooperative Negotiation Model for Multiple Continuous Dependent Issues Nicola Gatti and Francesco Amigoni Dipartimento di Elettronica e Informazione Politecnico di Milano, Milano, Italy {ngatti, amigoni}@elet.polimi.it Abstract Cooperative negotiation is proved to be an effective paradigm to solve complex dynamic multi-objective problems in which each objective is associated to an agent. When the multi-objective problem is defined on several continuous variables, cooperative negotiation can be traced back to a sequential bargaining. However, the complexity of highly reconfigurable scenarios with a large number of agents does not allow the adoption of classical game theory techniques to design optimal bargaining models for cooperative negotiations. A way to tackle this complexity is based on the decentralization of the system, usually obtained by introducing a mediator that reduces the amount of information directly exchanged between the agents. In this paper we present and experimentally evaluate a decentralized bargaining model for cooperative negotiation on multiple continuous dependent issues able to produce approximate Pareto optimal outcomes.
1. Introduction Cooperative negotiation techniques can be effectively employed to solve dynamic and distributed multi-objective problems [2] in which a dynamic set of entities with conflicting objectives interact in a dynamic environment. Typical examples of these problems are the determination of trajectories of a set of airplanes that share a common airspace [11], the management of ad hoc networks [6], and resource allocation [12]. All these applications can be modeled with a dynamic set of agents, each one embedding an objective function and interacting with each other in order to socially optimize their objective functions. Finding solutions to dynamic and distributed multi-objective optimization problems is hard. This is mainly due to the distribution, to the computational burden required by the resolution techniques, and to
the reconfigurability of such techniques to address the dynamicity of the network of agents. In this sense, cooperative negotiation is a cooperative search technique for a team of agents sharing a common goal, but having different individual criteria to evaluate the degree of achievement of this goal. When the issues on which the agents cooperatively negotiate are continuous, the cooperative negotiation can be traced back to a sequential bargaining [8]. However, the complexity due to the large number of agents, to the reconfigurability of the network, and to the temporal deadline within which the optimization must be accomplished prevent the adoption of classical game theory techniques to design optimal bargaining models (intended as protocols together with strategies) for cooperative negotiation. Game theory assumes that the agents select their strategies by searching spaces that grow exponentially with the number of agents, making the problem of finding an optimal strategy computationally intractable [14]. Cooperative negotiation provides a decentralization of the bargaining model: the agents hold private information and a mediator is introduced to reduce the amount of direct interactions between the agents. However, in a decentralized bargaining model the strategies of the agents are required to satisfy some properties to make the cooperative negotiation effective, including the stability and the optimality of the outcomes. In [9] we presented the conceptual model for a cooperative negotiation approach based on a decentralized bargaining composed of a protocol (the set of rules of the dispute) and of a class of agent negotiation strategies that produce approximate Pareto optimal outcomes. Given these results, this paper introduces three main original contributions. (1) The specification and discussion of our bargaining model in presence of two agents. (2) The formal proof of the stability of our bargaining model. (3) The evaluation of the performances of our bargaining model in the case the two agents cooperatively negotiating on two issues. Our proposal is a first step towards the definition of a class of decentralized bargaining models in which all the agents bargain on every
issue and that can satisfy any optimality criteria [15]. This paper is organized as follows. Section 2 discusses the employment of bargaining in cooperative negotiation. Section 3 presents our bargaining model for two agents. In Section 4 we study the stability of the proposed bargaining model. In Section 5 we experimentally evaluate the performances of our approach. Section 6 concludes the paper.
2. Bargaining and Cooperative Negotiation We consider a multi-objective optimization problem with objectives that change with time and that are embedded in different entities. These problems are encountered in engineering to control reconfigurable systems [5, 11, 12]. Traditional approaches for solving these problems are based on a centralized decision maker that collects information about all the objectives, computes the Pareto frontier, and takes decisions for all the agents. Traditional approaches have two drawbacks: their computational cost is O(mn ) – where m is the number of issues over which the agents negotiate and n the number of the agents – and they require a centralized decision maker that collects the information of all the agents and solves the problem for them. Cooperative negotiation offers an approximate but more efficient way to solve dynamic and distributed multi-objective problems. Negotiation and multi-objective optimization are similar in many respects. Let us consider the class of negotiation protocols called cooperative bargaining proposed by Nash [15], as complementary to the competitive/sequential bargaining proposed by Rubinstein [16]. Cooperative bargaining can be seen as a multi-objective problem. Given a set of agents, the study of cooperative bargaining concerns the determination of an outcome that is optimal for all the agents. A solution of the bargaining is a technique that produces an outcome; a solution is said a Nash bargaining solution [15] if it satisfies four properties: (1) it produces Pareto optimal outcomes, (2) it is invariant to affine transformations, (3) it is symmetric, (4) it is independent from irrelevant alternatives. In this paper, we concentrate on Pareto optimality. An outcome is Pareto optimal if there is not any other outcome that makes every agent at least as well off and at least one agent strictly better off. Other criteria, alternative to the Nash bargaining solution, to establish the optimality of a solution (e.g., Kalai and KalaiSmorodinsky [15]) all identify the optimal outcome as a specific point of the Pareto frontier. Since cooperative bargaining solutions are designed as multi-objective problems, they usually have exponential computational complexity and require the existence of an unique decision maker with complete information about all the agents. Cooperative negotiation tries to reduce computational complexity in solving dynamic and distributed multi-objective problems exploiting competitive/sequential
bargaining with private information [12]. Two main motivations are behind this employment of sequential bargaining. Firstly, sequential bargaining models with private information allow a reduction of the computational complexity not requiring any centralized decision maker and reducing, thus, the degree of information exchanged by agents. Secondly, sequential bargaining models can be designed to produce optimal cooperative outcomes. This second issue has been discussed – under the name of Nash program – by Binmore in [1] proving that Rubinstein’s sequential bargaining model, under specific hypotheses, produces Nash optimal cooperative outcomes. However, the study of Binmore can be employed exclusively in the case of a single issue and, to the best of our knowledge, there not exists in literature a satisfactory discussion of the multiple issues case. According to the Nash program, a multi-objective problem can be solved via a bargaining based on sequential offers performed concurrently by all the agents in the attempt to reach an optimal agreement. In brief, cooperative negotiation employs competitive techniques to solve cooperative problems. In cooperative negotiation the agents do not exchange any information about their utility functions and strategies and a mediator is introduced to guide the bargaining process to an outcome [3]. Moreover, agents uses ad hoc strategies of bargaining that produce ad hoc outcomes [8], usually called emerging equilibria. The problem is, thus, the development of methodologies for theoretically proving the properties of the emerging equilibria. In particular, two extremely relevant properties are: the reaching of a stable agreement between the agents and the optimality of the reached agreement. We have investigated the stability problem in [8] with the proposal of a criterion to design stable cooperative negotiations. On the other hand, to the best of our knowledge, the optimality problem has not found yet a satisfactory analysis in literature. Several works address Pareto optimality in decentralized scenarios (with private information and without any centralized decision maker); however, they do not present agents’ bargaining models, but only techniques to support bargaining. For example, in [4] an algorithm able to determine, given n agents, the Pareto frontier in a decentralized way is described. The idea is that, once the Pareto frontier has been determined, it can be exploited by a generic bargaining model to produce Pareto optimal outcomes. As another example, in [5] it is presented an algorithm that, given n agents and a non-Pareto outcome produced by a generic bargaining, finds the Pareto optimal outcome closest to the given one. However, these techniques are not bargaining models in a strict sense, but they just support the agents’ bargaining strategies. In addition, all these decentralized techniques have not been formally proved to reach
stable agreements. In this paper we present a decentralized bargaining model able to produce approximate Pareto optimal outcomes between two agents and we formally prove the stability of the model.
3. A Pareto Optimal Sequential Bargaining Model In this section we describe the proposed cooperative negotiation approach. We consider a decentralized bargaining protocol on dependent continuous multi-issue performed concurrently by two agents and a mediator, in which the two agents bargain individually with the mediator. These subbargaining processes are dependent (a sub-bargaining affects the other sub-bargaining), simultaneous (they are performed concurrently), and synchronous (the temporal line is shared by the two agents and the mediator). Since the subbargaining processes are simultaneous and synchronous, the protocol is symmetric for the agents (i.e., any agent can take advantage from the offers of the other). The two subbargaining processes are carried on with a sequence of offers performed by the agents and counter-offers performed by the mediator. Formally, according to the notation of [7], we call pti ∈ I – where I ⊆ ℜm is the space of the values of the m issues – the offer of the agent i to the mediator at time t, and at ∈ I the agreement (counter-offer) of the mediator to the agents at time t. Each sub-bargaining process can be represented (for i ∈ {1, 2}) as: p0i ≻ a0 ≻ p1i ≻ · · · ≻ aτ , where τ is the instant of time at which the agents agree, and, consequently, the bargaining process terminates. Each agent i embeds an utility function Ui : I → ℜ that gives the payoff the agent assigns to a proposal and a negotiation function Fi that gives the proposal pt+1 of the agent i i according to the utility function the agent embeds, its previous proposals, and the counter-proposals at of the mediator. Fi defines the strategy of agent i. The mediator computes its counter-proposal at time t according to a function A : I × I → I that defines the agreement at harmonizing the two proposals. In addition, the mediator embeds a termination condition of the bargaining process. The agreement computed at final time τ , aτ , is the outcome of the bargaining. Note that the employment of a mediator and the decoupling of the sub-bargaining make the protocol scalable allowing efficient dynamic insertion and removal of agents. In a sequential bargaining, the agents reach the agreement with a continuous alternating of offers and counteroffers. Determining the strategy with which the agent i produces its proposals means, formally, determining the negotiation function Fi . In our case we want to design Fi in order to produce Pareto optimal outcomes. In the case of two bargainers, given an outcome o, the iso-level curves U1 = U1 (o) and U2 = U2 (o) are tangent in o if and only if o is Pareto optimal [15]. An idea for
designing the strategies of the agents in a centralized perspective is: the agents produce proposals pt+1 and pt+1 1 2 in the attempt to (C.1) make their proposals closer and closer and (C.2) make the difference between their propost+1 als (pt+1 − pt+1 the tangent to the 1 2 ) orthogonal in p1 to t+1 t+1 iso-level curve U1 = U1 p1 and in p2 to the tangent . This second requireto the iso-level curve U2 = U2 pt+1 2 ment assures that the tangent to U1 = U1 pt+1 in pt+1 1 1 t+1 t+1 in p2 . If the is parallel to the tangent to U2 = U2 p2 two agents satisfy the conditions above and they agree on an outcome (their proposals are identical) performing infinitesimal movements, the outcome is Pareto optimal. However, in a decentralized scenario, agents have not any knowledge about other agent’s utility function and proposal. Each agent can take into account just its own utility function and the information received from mediator (i.e., the counter-offers). Thus, we decentralize (C.1) and (C.2) decoupling the strategies of the agents in the following way: each agent produces a proposal pt+1 in the attempt i to (D.1) make its proposal closer and closer to the agreement at and (D.2) make the difference between its proposal and the agreement (pt+1 − at ) orthogonal to the tangent to i the iso-level curves of its utility function. We assume that A is the arithmetic mean between the proposals of the two agents (symmetry is preserved). In this way, the differences between the proposals of the two agents and the agreement (i.e., (pt+1 − at ) and (pt+1 − at )) have the same direction. 1 2 Then, according to (D.2), the tangents to the iso-level curves of the two agents in their proposals are parallel. Thus, if the proposals of the two agents converges satisfying (D.2), the outcome is Pareto optimal. We present an approximate algorithm to obtain the above result. The proposal of agent i is produced as follows (Fig. 1). We define a versor vit : vit =
at − pti , kat − pti k
that is along the direction of the difference between at and pti . We define tti as the point belonging to the iso-level curve Ui = Ui (pti ) in which the gradient ∇Ui (tti ) has the same direction of vit . This means that the tangent to such iso-level curve in tti is orthogonal to vit . 8 t < ∇Ui (ti ) · vt = 1 i k∇Ui (tt )k i : t t
Ui (ti ) = Ui (pi )
We define eti as the projection of tti on the direction orthogonal to vit that passes in at : eti =
at − tti · vit vit + tti
35
25
e3
a
1
e21
t
30
3
20
1
2
25
t
t i
1
δ || m − p ||
t1
3 4
p1
a
t i
t21
3
15
e
20
a
p1
x2
x
2
i
a
e1
t i
θ t+1 pi
15
t
pi
p21
10
t i
t 10
5
t11 p11
5 15
20
25
30
35
x1
40
45
50
55
Figure 1. Determination of pt+1 in a 2D space i
The proposal of the agent is produced in the attempt to get closer to eti with a step equal to δit kat − pti k: pt+1 = Fi (pti , at ) = pti + δit · kat − pti k · i
eti − pti , keti − pti k
δit
where ∈ [0, 1] is an a-dimensional parameter (it does not scale with the application). Since the agreement is the arithmetic mean, kat − pti k is the same for all i. Then, the concession of an agent to the other one is captured in the use of different values for δit . Since Fi depends on δit , we have defined a family of negotiation strategies N = N (δ1t , δ2t ) = {F1 , F2 } in which δ1t and δ2t can vary. In this paper we consider just the case in which δ1t = δ2t = δ, for all t. In Fig. 2 the first three periods of a sub-bargaining between agent 1 and mediator are showed. To decide the termination of the bargaining, the only information the mediator can use, in a decentralized situation of minimal exchanged information, is the distance between the proposals of the agents. In particular, we adopt the following termination condition: v u 2 uX t kpti − at k2 < th i=1
This condition is verified by the mediator. The value of th depends on the application. In Fig. 3 a complete bargaining process starting from (p01 , p02 ) and terminating in pτ1 = pτ2 = aτ is illustrated for both the agents. Finally, we remark that, since each agent individually determines its offers taking into account just its previous offer
16
18
20
22
24
x1
26
28
30
32
Figure 2. Three periods of the sub-bargaining of agent 1 in a 2D space
and the last agreement at , the model can, in principle, be applied to a bargaining between more than two agents. However, we do not consider this general case here.
4. Stability Analysis The guarantee to reach a stable agreement in a decentralized negotiation process is, as remarked in Section 2, of paramount importance. In our case, assuring the stability of the bargaining model means that, given any pair of agents, embedding functions {Ui , Fi } and starting from any pair of initial offers, a stable agreement will be reached. We adopt the following connective stability criterion for multiagent negotiation introduced in [8]. Theorem 4.1 Given n agents, if Γti < 1 for all t > 0 and for all i = 1, 2, . . . , n, the negotiation is connectively stable. Where: Γti =
kpti − at−1 k kpt−1 − at−1 k i
We can apply Theorem 4.1 to make the bargaining model defined in Section 3 stable. More specifically, we force the agents to satisfy the constraints of the theorem. In order for the agents to satisfy Γti < 1, where i = 1, 2, from t = 0 onward, we need to modify the strategies Fi . More specifi˜ t+1 cally, initially an agent i produces a tentative proposal p i t+1 t ˜ i satisfies Γi < 1, using Fi as defined in Section 3; if p it will be the proposal pt+1 of the agent. Otherwise, the i
solution of the minimization problem ψit = arg min |ψ−θit | ψ
80
with the three constraints discussed above. Formally:
70
60
50
x2
ψit = 40
0
a aτ
30
8 > arg min |ψ − θit | > > ψ > > > e t − at > > eti = tan(ψ) · kat − pti k · ti > (2) s.t. pt+1 = δ · kat − pti k · ti > i > > k˜ ei − pti k > > t+1 t > > kp − a k i > :(3) s.t. 0. Similarly we have defined In our experiments, P01 and P02 are composed of about 100 initial proposals; so we have about 10, 000 pairs of initial proposals for each scenario. Once the agents have formulated their initial proposals, they negotiate with our decentralized bargaining model of Section 4 to reach an optimal stable agreement. th and δ have been discretized in the following way: th = [0.1, 1] with step 0.05, δ = [0.1, 0.95] with step 0.025. To evaluate the outcomes produced by the bargaining model, we have determined, for each scenario, the Pareto frontier by brute force in a grid on the space I = (x1 , x2 ) with a resolution of 0.01 for both the two variables. The paradigm with which we have produced the results relative to a scenario is the following:
x 10
0.2
0.3
0.4
0.5
0.6
0.7
0.8
δ
(a) agents negotiate producing an outcome o ¯; (b) distance(¯ o) and periods(¯ o) are calculated; 2. the mean E[·] of distance and periods is calculated.
Figure 4. E[distance] and E[periods] as functions of th and δ
We have repeated this process for all the scenarios and the pairs (th, δ).
5.2. Experimental Results We report the results of a single typical scenario, with B = {100, 100, 1, 1, 2, 2, 0, 0}. Other scenarios produce similar results. The effects of th and δ on bargaining are reported in Fig. 4. More specifically, E[distance] is in the range [3 · 10−4 , 5 · 10−3 ] and E[periods] in the range [9, 104 ]. Obviously, the smaller the value of th, the larger the number of E[periods] and the smaller the value of the E[distance]. The E[distance] curves have a minimum around δ = 0.8; while an explicit relation between δ and E[periods] is harder to find.
In Fig. 5 we show, in the space of the normalized utilities, the outcomes obtained with the proposed model using th = 0.5 and δ = 0.75 and the Pareto frontier. (In this case, E[distance] = 6 · 10−4 and E[periods] = 38.) We note that almost all the outcomes lay on the Pareto frontier, except a few outcomes that are located around (U1∗ , U2∗ ) = (0.475, 0.475). In these cases, the initial proposals of the agents are too close and the bargaining is terminated before reaching an optimal outcome. To overcome this drawback a refinement of the bargaining model is needed. Finally, we have preliminary compared our approach to genetic algorithms, often employed to determine Pareto optimal outcomes. Note that genetic algorithms require a cen-
References
1 Bargaining outcomes Pareto frontier 0.9
0.8
Normalized U
2
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
0.6
Normalized U1
0.7
0.8
0.9
1
Figure 5. The Pareto frontier and the bargaining outcomes (th = 0.5, δ = 0.75)
tralized decision maker with complete information on all the agents. We considered distance and periods in bargaining as corresponding to distance and generations in genetic algorithms. A preliminary analysis shows that our approach produces outcomes (Fig. 4) comparable, in term of performance, to the outcomes produced with genetic algorithms [10] (distance ∈ [10−3 , 10−2 ], generations ∈ [20, 100]). However, further investigation is required on this issue.
6. Conclusions and Future Works In this paper we have presented a cooperative negotiation approach for approximate Pareto optimality. It aims at being a first step towards a many-to-many bargaining model able to solve complex dynamic multi-objective problems. The experimental results show that the outcomes produced by our approach are very close to the Pareto frontier. The bargaining model presented in this paper can be improved towards three main directions. The first one concerns the refinement of the bargaining model to improve its performances and to formally prove the Pareto optimality. The second one concerns the investigation of real-time bargaining techniques to obtain the best performance within a given number of periods. The last one concerns the extension of the model to support many-to-many bargaining and the evaluation of the performances when using different values for the δit . Finally, we plan to employ the proposed cooperative negotiation approach in two application scenarios: ad hoc network control and vehicle formations.
[1] K. G. Binmore. Nash bargaining theory II. In K. G. Binmore and P. Dasgupta, editors, Economics and Bargaining. Blackwell, 1986. [2] S. E. Conry, K. Kuwabara, V. R. Lesser, and R. A. Meyer. Multistage negotiation for distributed constraint satisfaction. IEEE T SYST MAN CY, 21(6):1462–1476, November/December 1991. [3] M. Dumas, G. Governatori, A. H. M. ter Hofstede, and P. Oaks. A formal approach to negotiating agents development. Electronic Commerce Research and Applications, 1(2):193–207, 2002. [4] H. Ehtamo, R. P. Hamalainen, P. Heinskanen, J. Teich, M. Verkama, and Z. Stanley. Generating pareto solutions in a two-party setting: Constraint proposal methods. MANAGE SCI, 45:1697–1709, 1999. [5] H. Ehtamo, E. Kettunen, and R. P. Hamalainen. Searching for joint gains in multiparty negotiations. EUR J OPER RES, 130(1):54–69, 2001. [6] Z. Fano and B. Bensaou. Fair bandwidth sharing algorithms based on game theory frameworks for wireless ad-hoc networks. In Proc. IEEE INFOCOM, pages 1284–1295, Hong Kong, 2004. [7] P. Faratin, C. Sierra, and N. R. Jennings. Negotiation decision functions for autonomous agents. ROBOT AUTON SYST, 24(3-4):159–182, 1998. [8] N. Gatti. A connective stability analysis of complex system simulation and control via multiagent systems. In Proc. STAIRS, pages 26–37. IOS Press, Valencia, Spain, 2004. [9] N. Gatti and F. Amigoni. A decentralized bargaining protocol on dependent continuous multi-issue for approximate pareto optimal outcomes. In Proc. AAMAS, Utrecht, The Netherlands, 2005. [10] E. H. Gerding, D. D. B. van Bragt, and J. A. L. Poutr´e. Multiissue negotiation processes by evolutionary simulation: Validation and social extensions. Computational Economics, 22(1):39–63, 2003. [11] G. Inalhan, D. M. Stipanovic, and C. J. Tomlin. Decentralized optimization, with application to multiple aircraft coordination. In Proc. IEEE CDC, volume 1, pages 1147–1155, Las Vegas, USA, 2002. [12] R. Mailler, V. R. Lesser, and B. Horling. Cooperative negotiation for soft real-time distributed resource allocation. In Proc. AAMAS, pages 576–583, Melbourne, Australia, 2003. [13] T. Okabe, Y. Jin, and B. Sendhoff. A critical survey of performance indices for multi-objective optimization. In Proc. IEEE CEC, volume 2, pages 878–885, Canberra, Australia, 2003. [14] S. Parsons and M. Wooldridge. Game theory and decision theory in multi-agent systems. AUTON AGENT MULTI-AG, 5(3):243–254, 2002. [15] A. E. Roth. Game-Theoretic Models of Bargaining. Cambridge University Press, Cambridge, USA, 1985. [16] A. Rubinstein. Perfect equilibrium in a bargaining model. ECONOMETRICA, 50(1):97–109, 1982.
1. Introduction Cooperative negotiation techniques can be effectively employed to solve dynamic and distributed multi-objective problems [2] in which a dynamic set of entities with conflicting objectives interact in a dynamic environment. Typical examples of these problems are the determination of trajectories of a set of airplanes that share a common airspace [11], the management of ad hoc networks [6], and resource allocation [12]. All these applications can be modeled with a dynamic set of agents, each one embedding an objective function and interacting with each other in order to socially optimize their objective functions. Finding solutions to dynamic and distributed multi-objective optimization problems is hard. This is mainly due to the distribution, to the computational burden required by the resolution techniques, and to
the reconfigurability of such techniques to address the dynamicity of the network of agents. In this sense, cooperative negotiation is a cooperative search technique for a team of agents sharing a common goal, but having different individual criteria to evaluate the degree of achievement of this goal. When the issues on which the agents cooperatively negotiate are continuous, the cooperative negotiation can be traced back to a sequential bargaining [8]. However, the complexity due to the large number of agents, to the reconfigurability of the network, and to the temporal deadline within which the optimization must be accomplished prevent the adoption of classical game theory techniques to design optimal bargaining models (intended as protocols together with strategies) for cooperative negotiation. Game theory assumes that the agents select their strategies by searching spaces that grow exponentially with the number of agents, making the problem of finding an optimal strategy computationally intractable [14]. Cooperative negotiation provides a decentralization of the bargaining model: the agents hold private information and a mediator is introduced to reduce the amount of direct interactions between the agents. However, in a decentralized bargaining model the strategies of the agents are required to satisfy some properties to make the cooperative negotiation effective, including the stability and the optimality of the outcomes. In [9] we presented the conceptual model for a cooperative negotiation approach based on a decentralized bargaining composed of a protocol (the set of rules of the dispute) and of a class of agent negotiation strategies that produce approximate Pareto optimal outcomes. Given these results, this paper introduces three main original contributions. (1) The specification and discussion of our bargaining model in presence of two agents. (2) The formal proof of the stability of our bargaining model. (3) The evaluation of the performances of our bargaining model in the case the two agents cooperatively negotiating on two issues. Our proposal is a first step towards the definition of a class of decentralized bargaining models in which all the agents bargain on every
issue and that can satisfy any optimality criteria [15]. This paper is organized as follows. Section 2 discusses the employment of bargaining in cooperative negotiation. Section 3 presents our bargaining model for two agents. In Section 4 we study the stability of the proposed bargaining model. In Section 5 we experimentally evaluate the performances of our approach. Section 6 concludes the paper.
2. Bargaining and Cooperative Negotiation We consider a multi-objective optimization problem with objectives that change with time and that are embedded in different entities. These problems are encountered in engineering to control reconfigurable systems [5, 11, 12]. Traditional approaches for solving these problems are based on a centralized decision maker that collects information about all the objectives, computes the Pareto frontier, and takes decisions for all the agents. Traditional approaches have two drawbacks: their computational cost is O(mn ) – where m is the number of issues over which the agents negotiate and n the number of the agents – and they require a centralized decision maker that collects the information of all the agents and solves the problem for them. Cooperative negotiation offers an approximate but more efficient way to solve dynamic and distributed multi-objective problems. Negotiation and multi-objective optimization are similar in many respects. Let us consider the class of negotiation protocols called cooperative bargaining proposed by Nash [15], as complementary to the competitive/sequential bargaining proposed by Rubinstein [16]. Cooperative bargaining can be seen as a multi-objective problem. Given a set of agents, the study of cooperative bargaining concerns the determination of an outcome that is optimal for all the agents. A solution of the bargaining is a technique that produces an outcome; a solution is said a Nash bargaining solution [15] if it satisfies four properties: (1) it produces Pareto optimal outcomes, (2) it is invariant to affine transformations, (3) it is symmetric, (4) it is independent from irrelevant alternatives. In this paper, we concentrate on Pareto optimality. An outcome is Pareto optimal if there is not any other outcome that makes every agent at least as well off and at least one agent strictly better off. Other criteria, alternative to the Nash bargaining solution, to establish the optimality of a solution (e.g., Kalai and KalaiSmorodinsky [15]) all identify the optimal outcome as a specific point of the Pareto frontier. Since cooperative bargaining solutions are designed as multi-objective problems, they usually have exponential computational complexity and require the existence of an unique decision maker with complete information about all the agents. Cooperative negotiation tries to reduce computational complexity in solving dynamic and distributed multi-objective problems exploiting competitive/sequential
bargaining with private information [12]. Two main motivations are behind this employment of sequential bargaining. Firstly, sequential bargaining models with private information allow a reduction of the computational complexity not requiring any centralized decision maker and reducing, thus, the degree of information exchanged by agents. Secondly, sequential bargaining models can be designed to produce optimal cooperative outcomes. This second issue has been discussed – under the name of Nash program – by Binmore in [1] proving that Rubinstein’s sequential bargaining model, under specific hypotheses, produces Nash optimal cooperative outcomes. However, the study of Binmore can be employed exclusively in the case of a single issue and, to the best of our knowledge, there not exists in literature a satisfactory discussion of the multiple issues case. According to the Nash program, a multi-objective problem can be solved via a bargaining based on sequential offers performed concurrently by all the agents in the attempt to reach an optimal agreement. In brief, cooperative negotiation employs competitive techniques to solve cooperative problems. In cooperative negotiation the agents do not exchange any information about their utility functions and strategies and a mediator is introduced to guide the bargaining process to an outcome [3]. Moreover, agents uses ad hoc strategies of bargaining that produce ad hoc outcomes [8], usually called emerging equilibria. The problem is, thus, the development of methodologies for theoretically proving the properties of the emerging equilibria. In particular, two extremely relevant properties are: the reaching of a stable agreement between the agents and the optimality of the reached agreement. We have investigated the stability problem in [8] with the proposal of a criterion to design stable cooperative negotiations. On the other hand, to the best of our knowledge, the optimality problem has not found yet a satisfactory analysis in literature. Several works address Pareto optimality in decentralized scenarios (with private information and without any centralized decision maker); however, they do not present agents’ bargaining models, but only techniques to support bargaining. For example, in [4] an algorithm able to determine, given n agents, the Pareto frontier in a decentralized way is described. The idea is that, once the Pareto frontier has been determined, it can be exploited by a generic bargaining model to produce Pareto optimal outcomes. As another example, in [5] it is presented an algorithm that, given n agents and a non-Pareto outcome produced by a generic bargaining, finds the Pareto optimal outcome closest to the given one. However, these techniques are not bargaining models in a strict sense, but they just support the agents’ bargaining strategies. In addition, all these decentralized techniques have not been formally proved to reach
stable agreements. In this paper we present a decentralized bargaining model able to produce approximate Pareto optimal outcomes between two agents and we formally prove the stability of the model.
3. A Pareto Optimal Sequential Bargaining Model In this section we describe the proposed cooperative negotiation approach. We consider a decentralized bargaining protocol on dependent continuous multi-issue performed concurrently by two agents and a mediator, in which the two agents bargain individually with the mediator. These subbargaining processes are dependent (a sub-bargaining affects the other sub-bargaining), simultaneous (they are performed concurrently), and synchronous (the temporal line is shared by the two agents and the mediator). Since the subbargaining processes are simultaneous and synchronous, the protocol is symmetric for the agents (i.e., any agent can take advantage from the offers of the other). The two subbargaining processes are carried on with a sequence of offers performed by the agents and counter-offers performed by the mediator. Formally, according to the notation of [7], we call pti ∈ I – where I ⊆ ℜm is the space of the values of the m issues – the offer of the agent i to the mediator at time t, and at ∈ I the agreement (counter-offer) of the mediator to the agents at time t. Each sub-bargaining process can be represented (for i ∈ {1, 2}) as: p0i ≻ a0 ≻ p1i ≻ · · · ≻ aτ , where τ is the instant of time at which the agents agree, and, consequently, the bargaining process terminates. Each agent i embeds an utility function Ui : I → ℜ that gives the payoff the agent assigns to a proposal and a negotiation function Fi that gives the proposal pt+1 of the agent i i according to the utility function the agent embeds, its previous proposals, and the counter-proposals at of the mediator. Fi defines the strategy of agent i. The mediator computes its counter-proposal at time t according to a function A : I × I → I that defines the agreement at harmonizing the two proposals. In addition, the mediator embeds a termination condition of the bargaining process. The agreement computed at final time τ , aτ , is the outcome of the bargaining. Note that the employment of a mediator and the decoupling of the sub-bargaining make the protocol scalable allowing efficient dynamic insertion and removal of agents. In a sequential bargaining, the agents reach the agreement with a continuous alternating of offers and counteroffers. Determining the strategy with which the agent i produces its proposals means, formally, determining the negotiation function Fi . In our case we want to design Fi in order to produce Pareto optimal outcomes. In the case of two bargainers, given an outcome o, the iso-level curves U1 = U1 (o) and U2 = U2 (o) are tangent in o if and only if o is Pareto optimal [15]. An idea for
designing the strategies of the agents in a centralized perspective is: the agents produce proposals pt+1 and pt+1 1 2 in the attempt to (C.1) make their proposals closer and closer and (C.2) make the difference between their propost+1 als (pt+1 − pt+1 the tangent to the 1 2 ) orthogonal in p1 to t+1 t+1 iso-level curve U1 = U1 p1 and in p2 to the tangent . This second requireto the iso-level curve U2 = U2 pt+1 2 ment assures that the tangent to U1 = U1 pt+1 in pt+1 1 1 t+1 t+1 in p2 . If the is parallel to the tangent to U2 = U2 p2 two agents satisfy the conditions above and they agree on an outcome (their proposals are identical) performing infinitesimal movements, the outcome is Pareto optimal. However, in a decentralized scenario, agents have not any knowledge about other agent’s utility function and proposal. Each agent can take into account just its own utility function and the information received from mediator (i.e., the counter-offers). Thus, we decentralize (C.1) and (C.2) decoupling the strategies of the agents in the following way: each agent produces a proposal pt+1 in the attempt i to (D.1) make its proposal closer and closer to the agreement at and (D.2) make the difference between its proposal and the agreement (pt+1 − at ) orthogonal to the tangent to i the iso-level curves of its utility function. We assume that A is the arithmetic mean between the proposals of the two agents (symmetry is preserved). In this way, the differences between the proposals of the two agents and the agreement (i.e., (pt+1 − at ) and (pt+1 − at )) have the same direction. 1 2 Then, according to (D.2), the tangents to the iso-level curves of the two agents in their proposals are parallel. Thus, if the proposals of the two agents converges satisfying (D.2), the outcome is Pareto optimal. We present an approximate algorithm to obtain the above result. The proposal of agent i is produced as follows (Fig. 1). We define a versor vit : vit =
at − pti , kat − pti k
that is along the direction of the difference between at and pti . We define tti as the point belonging to the iso-level curve Ui = Ui (pti ) in which the gradient ∇Ui (tti ) has the same direction of vit . This means that the tangent to such iso-level curve in tti is orthogonal to vit . 8 t < ∇Ui (ti ) · vt = 1 i k∇Ui (tt )k i : t t
Ui (ti ) = Ui (pi )
We define eti as the projection of tti on the direction orthogonal to vit that passes in at : eti =
at − tti · vit vit + tti
35
25
e3
a
1
e21
t
30
3
20
1
2
25
t
t i
1
δ || m − p ||
t1
3 4
p1
a
t i
t21
3
15
e
20
a
p1
x2
x
2
i
a
e1
t i
θ t+1 pi
15
t
pi
p21
10
t i
t 10
5
t11 p11
5 15
20
25
30
35
x1
40
45
50
55
Figure 1. Determination of pt+1 in a 2D space i
The proposal of the agent is produced in the attempt to get closer to eti with a step equal to δit kat − pti k: pt+1 = Fi (pti , at ) = pti + δit · kat − pti k · i
eti − pti , keti − pti k
δit
where ∈ [0, 1] is an a-dimensional parameter (it does not scale with the application). Since the agreement is the arithmetic mean, kat − pti k is the same for all i. Then, the concession of an agent to the other one is captured in the use of different values for δit . Since Fi depends on δit , we have defined a family of negotiation strategies N = N (δ1t , δ2t ) = {F1 , F2 } in which δ1t and δ2t can vary. In this paper we consider just the case in which δ1t = δ2t = δ, for all t. In Fig. 2 the first three periods of a sub-bargaining between agent 1 and mediator are showed. To decide the termination of the bargaining, the only information the mediator can use, in a decentralized situation of minimal exchanged information, is the distance between the proposals of the agents. In particular, we adopt the following termination condition: v u 2 uX t kpti − at k2 < th i=1
This condition is verified by the mediator. The value of th depends on the application. In Fig. 3 a complete bargaining process starting from (p01 , p02 ) and terminating in pτ1 = pτ2 = aτ is illustrated for both the agents. Finally, we remark that, since each agent individually determines its offers taking into account just its previous offer
16
18
20
22
24
x1
26
28
30
32
Figure 2. Three periods of the sub-bargaining of agent 1 in a 2D space
and the last agreement at , the model can, in principle, be applied to a bargaining between more than two agents. However, we do not consider this general case here.
4. Stability Analysis The guarantee to reach a stable agreement in a decentralized negotiation process is, as remarked in Section 2, of paramount importance. In our case, assuring the stability of the bargaining model means that, given any pair of agents, embedding functions {Ui , Fi } and starting from any pair of initial offers, a stable agreement will be reached. We adopt the following connective stability criterion for multiagent negotiation introduced in [8]. Theorem 4.1 Given n agents, if Γti < 1 for all t > 0 and for all i = 1, 2, . . . , n, the negotiation is connectively stable. Where: Γti =
kpti − at−1 k kpt−1 − at−1 k i
We can apply Theorem 4.1 to make the bargaining model defined in Section 3 stable. More specifically, we force the agents to satisfy the constraints of the theorem. In order for the agents to satisfy Γti < 1, where i = 1, 2, from t = 0 onward, we need to modify the strategies Fi . More specifi˜ t+1 cally, initially an agent i produces a tentative proposal p i t+1 t ˜ i satisfies Γi < 1, using Fi as defined in Section 3; if p it will be the proposal pt+1 of the agent. Otherwise, the i
solution of the minimization problem ψit = arg min |ψ−θit | ψ
80
with the three constraints discussed above. Formally:
70
60
50
x2
ψit = 40
0
a aτ
30
8 > arg min |ψ − θit | > > ψ > > > e t − at > > eti = tan(ψ) · kat − pti k · ti > (2) s.t. pt+1 = δ · kat − pti k · ti > i > > k˜ ei − pti k > > t+1 t > > kp − a k i > :(3) s.t. 0. Similarly we have defined In our experiments, P01 and P02 are composed of about 100 initial proposals; so we have about 10, 000 pairs of initial proposals for each scenario. Once the agents have formulated their initial proposals, they negotiate with our decentralized bargaining model of Section 4 to reach an optimal stable agreement. th and δ have been discretized in the following way: th = [0.1, 1] with step 0.05, δ = [0.1, 0.95] with step 0.025. To evaluate the outcomes produced by the bargaining model, we have determined, for each scenario, the Pareto frontier by brute force in a grid on the space I = (x1 , x2 ) with a resolution of 0.01 for both the two variables. The paradigm with which we have produced the results relative to a scenario is the following:
x 10
0.2
0.3
0.4
0.5
0.6
0.7
0.8
δ
(a) agents negotiate producing an outcome o ¯; (b) distance(¯ o) and periods(¯ o) are calculated; 2. the mean E[·] of distance and periods is calculated.
Figure 4. E[distance] and E[periods] as functions of th and δ
We have repeated this process for all the scenarios and the pairs (th, δ).
5.2. Experimental Results We report the results of a single typical scenario, with B = {100, 100, 1, 1, 2, 2, 0, 0}. Other scenarios produce similar results. The effects of th and δ on bargaining are reported in Fig. 4. More specifically, E[distance] is in the range [3 · 10−4 , 5 · 10−3 ] and E[periods] in the range [9, 104 ]. Obviously, the smaller the value of th, the larger the number of E[periods] and the smaller the value of the E[distance]. The E[distance] curves have a minimum around δ = 0.8; while an explicit relation between δ and E[periods] is harder to find.
In Fig. 5 we show, in the space of the normalized utilities, the outcomes obtained with the proposed model using th = 0.5 and δ = 0.75 and the Pareto frontier. (In this case, E[distance] = 6 · 10−4 and E[periods] = 38.) We note that almost all the outcomes lay on the Pareto frontier, except a few outcomes that are located around (U1∗ , U2∗ ) = (0.475, 0.475). In these cases, the initial proposals of the agents are too close and the bargaining is terminated before reaching an optimal outcome. To overcome this drawback a refinement of the bargaining model is needed. Finally, we have preliminary compared our approach to genetic algorithms, often employed to determine Pareto optimal outcomes. Note that genetic algorithms require a cen-
References
1 Bargaining outcomes Pareto frontier 0.9
0.8
Normalized U
2
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
0.6
Normalized U1
0.7
0.8
0.9
1
Figure 5. The Pareto frontier and the bargaining outcomes (th = 0.5, δ = 0.75)
tralized decision maker with complete information on all the agents. We considered distance and periods in bargaining as corresponding to distance and generations in genetic algorithms. A preliminary analysis shows that our approach produces outcomes (Fig. 4) comparable, in term of performance, to the outcomes produced with genetic algorithms [10] (distance ∈ [10−3 , 10−2 ], generations ∈ [20, 100]). However, further investigation is required on this issue.
6. Conclusions and Future Works In this paper we have presented a cooperative negotiation approach for approximate Pareto optimality. It aims at being a first step towards a many-to-many bargaining model able to solve complex dynamic multi-objective problems. The experimental results show that the outcomes produced by our approach are very close to the Pareto frontier. The bargaining model presented in this paper can be improved towards three main directions. The first one concerns the refinement of the bargaining model to improve its performances and to formally prove the Pareto optimality. The second one concerns the investigation of real-time bargaining techniques to obtain the best performance within a given number of periods. The last one concerns the extension of the model to support many-to-many bargaining and the evaluation of the performances when using different values for the δit . Finally, we plan to employ the proposed cooperative negotiation approach in two application scenarios: ad hoc network control and vehicle formations.
[1] K. G. Binmore. Nash bargaining theory II. In K. G. Binmore and P. Dasgupta, editors, Economics and Bargaining. Blackwell, 1986. [2] S. E. Conry, K. Kuwabara, V. R. Lesser, and R. A. Meyer. Multistage negotiation for distributed constraint satisfaction. IEEE T SYST MAN CY, 21(6):1462–1476, November/December 1991. [3] M. Dumas, G. Governatori, A. H. M. ter Hofstede, and P. Oaks. A formal approach to negotiating agents development. Electronic Commerce Research and Applications, 1(2):193–207, 2002. [4] H. Ehtamo, R. P. Hamalainen, P. Heinskanen, J. Teich, M. Verkama, and Z. Stanley. Generating pareto solutions in a two-party setting: Constraint proposal methods. MANAGE SCI, 45:1697–1709, 1999. [5] H. Ehtamo, E. Kettunen, and R. P. Hamalainen. Searching for joint gains in multiparty negotiations. EUR J OPER RES, 130(1):54–69, 2001. [6] Z. Fano and B. Bensaou. Fair bandwidth sharing algorithms based on game theory frameworks for wireless ad-hoc networks. In Proc. IEEE INFOCOM, pages 1284–1295, Hong Kong, 2004. [7] P. Faratin, C. Sierra, and N. R. Jennings. Negotiation decision functions for autonomous agents. ROBOT AUTON SYST, 24(3-4):159–182, 1998. [8] N. Gatti. A connective stability analysis of complex system simulation and control via multiagent systems. In Proc. STAIRS, pages 26–37. IOS Press, Valencia, Spain, 2004. [9] N. Gatti and F. Amigoni. A decentralized bargaining protocol on dependent continuous multi-issue for approximate pareto optimal outcomes. In Proc. AAMAS, Utrecht, The Netherlands, 2005. [10] E. H. Gerding, D. D. B. van Bragt, and J. A. L. Poutr´e. Multiissue negotiation processes by evolutionary simulation: Validation and social extensions. Computational Economics, 22(1):39–63, 2003. [11] G. Inalhan, D. M. Stipanovic, and C. J. Tomlin. Decentralized optimization, with application to multiple aircraft coordination. In Proc. IEEE CDC, volume 1, pages 1147–1155, Las Vegas, USA, 2002. [12] R. Mailler, V. R. Lesser, and B. Horling. Cooperative negotiation for soft real-time distributed resource allocation. In Proc. AAMAS, pages 576–583, Melbourne, Australia, 2003. [13] T. Okabe, Y. Jin, and B. Sendhoff. A critical survey of performance indices for multi-objective optimization. In Proc. IEEE CEC, volume 2, pages 878–885, Canberra, Australia, 2003. [14] S. Parsons and M. Wooldridge. Game theory and decision theory in multi-agent systems. AUTON AGENT MULTI-AG, 5(3):243–254, 2002. [15] A. E. Roth. Game-Theoretic Models of Bargaining. Cambridge University Press, Cambridge, USA, 1985. [16] A. Rubinstein. Perfect equilibrium in a bargaining model. ECONOMETRICA, 50(1):97–109, 1982.
