E;j - Santa Fe Institute
A Model of Individual Adaptive Behavior in a Fluctuating Environment Lev A. Zhivotovsky Aviv Bergman Marcus W. Feldman
SFI WORKING PAPER: 1994-02-004
SFI Working Papers contain accounts of scientific work of the author(s) and do not necessarily represent the views of the Santa Fe Institute. We accept papers intended for publication in peer-reviewed journals or proceedings volumes, but not papers that have already appeared in print. Except for papers by our external faculty, papers must be based on work done at SFI, inspired by an invited visit to or collaboration at SFI, or funded by an SFI grant. ©NOTICE: This working paper is included by permission of the contributing author(s) as a means to ensure timely distribution of the scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the author(s). It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may be reposted only with the explicit permission of the copyright holder. www.santafe.edu
SANTA FE INSTITUTE
A Mod el of Indi vidu al Ada ptive Beha vior in a Fluc tuati ng Envi ronm ent
Lev A. Zhivo tovsk yt+, Aviv Bergm an-+, and Marc us W. Feldm an*
tInsti tute of Gener al Genetics, Russi an Acade my of Sciences. 3 Gubki n St., GSP- l, B-333, Moscow 117809, Russia: +Interval Research Corpo ration , 1801 Page Mill Road. , Bldg. C, Palo Alto, Califo rnia, l'SA; and *Depa rtmen t of Biological Scienc es. Stanfo rd University, Stanfo rd, California 94305, USA
Key Word s: fitness, strateg y, Markov process, predic tion. optim ality
(8/11/ 93)
Abstract. Individual behavioral strategies that use conditional probabilities for future environments and infonnation about past environments are studied. The environments are random and Markovian. The individual uses the information available to it to prepare for the next environmental state in order to increase its fitness. The fitness depends on the discrepancy between the realized environment and that for which the individual is prepared. Additive and multiplicative combinations of the fitnesses accruing to the individual at each environmental epoch are studied. A semi-optimal strategy is found, which maximizes individual fitness given the depth of infonnation about the environment available to the individual. Randomly varying fitnesses and errors in the individual's perception of the environmental parameters may be included in the model.
1
Introduction. For many biological organisms the process of adaptation is one of survival and reproduction in an uncertain environment.
Two kinds of adaptations are important to
distinguish. The first might be viewed as occurring at the population level and envisages an array of different genotypes or phenotypes. each adapted to a characteristic range of environmental conditions. Taken as an ensemble, this array permits the population to adapt to changing conditions. This situation is usually modelled in terms of the evolution of the frequencies of the types under natural selection with emphasis on between-generation changes in environmental conditions (Lewontin and Cohen, 1969; Gillespie, 1973; Hartl and Cook, 1973; Karlin and Liberman. 1974; Stephens, 1991; Bergman and Feldman, 1993), or by making use of optimality reasoning where the distinction between generations is often blurred (Cohen, 1966. 1!J93; Harley, 1!J81; Harley and Maynard Smith. 1!J83; Houston and Sumida, 1987; McNamara and Houston, 1987). A second kind of adaptation may occur when individualJ exhibit plasticity that permits them to respond to environmental conditions in a manner that enhances their survival. Thus, individuals seek a general strategy that permits them to learn or seek a behavior which increases fitness (Shettleworth, 1984). For this response
to
be appropriate, the
individual should possess some "information" about the future environment, it should have stored information about previous environments (i.e. memory), and be able to predict and prepare for pending environments. These three properties contribute to the individual's ability to survive in an environment that changes within a generation. Both kinds of adaptations, population level and individual level. may have occurred in the process of evolution. Both may involve selection on behavioral differences between the types in a population. Ecological analyses of evolution in changing environments standardly consider that individual behaviors are strategies in a game against the environment (Maynard Smith, 1982). This framework assumes that individuals have no knowledge about their environments and cannot make predictions. In such situations a minimax strategy is a convenient way for an individual to counter the worst possible environmental conditions. In reality, however, the environment does not playa game with individuals, and individuals may have 2
some a priori and a posteriori information about their environment on a short-term time scale during their lifetime. Many models of behavior based on individual learning couch the learning process in terms of changes in the probabilities with which a stimulus is chosen due to previously acquired information about stimuli and responses (e.g. Bush and Mosteller, 1955). Here the environment is represented as a set of stimuli; and the conceptualization of environmental changes remains somewhat vague. In this paper we investigate how such adaptive behavior depends on the extent of historical knowledge about the environment, on the amount of prior knowledge about the probabilistic law that governs environmental change, and whether the individual's behavior is intrinsically probabilistic or deterministic for a given set of information. To address these questions we consider an environment with several possible different conditions, called states, whose temporal pattern follows given conditional probabilities. The individual possesses some information about both these probabilities and the history of previous environmental outcomes. The individual chooses which next (unknown) state to prepare for in order to increase its fitness which depends on the discrepancy between the expected and realized environmental state. 1. The Environment.
Suppose that at each discrete point in time, t, the environment is a random variable [, which can take values from the set
01,02, ...•
.on' {o,} constitute an m-order Markov
process. These random variables are described by the conditional probabilities
(1) where
Sm
can be abbreviated by the m-veetor (iI, i 2, ... , i m ), with each coordinate rep-
resenting one of the environmental states 1,2, ... , n. Thus, if m = 4, for example, and Sm
= (2,3,1. T), the process took the values
02
at t - 1,
03
at t - 2,
CI
at t - 3, and
C7
at
t - 4. Such an environment will be defined to have depth m. In the case m = 0, changes in the environment are completely random and. at any time, the values
CI, 02, ...
,cn occur
with probabilities PI, P2 , ••• , P n . If m = 1, the environment is a standard Markov process
3
in which only the previous state affects the present. In all cases the environment will be assumed to be stationary. That is, for each i
=
1,2, ... , n, the unconditional probability
that the environment is c; is
(2) where P(sm) is the stationary probability of the sequence of m environmental states, assumed to be independent of where in time the sequence occurs.
2. Individual Fitness. To an individual, environmental changes may be due either to actual temporal effects described by the conditional probabilities (1), or they may appear to occur because the individual migrates across a spatially varying habitat. In any case, we shall assume that the environment experienced by the individual is as described in the previous section by relations (1) with depth m. We assume that individuals are able to predict the environment and are able to prepare in some way for that predicted environment. Suppose that an individual predicts that the environment at time t will be in state c j, and that the environment at time tactually takes state
Ci.
In this case, a fitness E ij will be assigned to that individual. It is natural
to assume E i ; > Eij if j
l' i;
that is, correct prediction of the environment entails higher
Ii tness than incorrect. During an individual's lifetime, N environmental changes occur of which Nij are such that the individual predicted state
Cj
but the actual environmental state was
Cj'
Thus
(3) 'J
The total normalized multiplicative fitness of such an individual is
(4) or
w=
I1E':ii 'J
'J
4
'
(5)
where
is the frequency that
E: J
is predicted and
E: i
occurs.
One may also introduce the total
normalized additive fitness
(7) The multiplicative and additive models, both expressing independence of the contributions of future environments to the total fitness, are related since In W =
2::
nij
(8)
InEij .
'J
Some results which hold for multiplicative fitnesses are also valid for additive fitnesses just by changing In E ij to E ij . However. in spite of this close relationship between the two ~
models, in some complex situations the two models produce qualitatively different results. 3. Individual Strategies. Suppose that just prior to the realization of the environment at time t an individual knows the realizations of the k previous environments t - 1, t - 2, ... , t - k. We say that this individual possesses a posteriori information of depth k. or has a memory of depth k. Let QUlsd be the probability that such an individual prepares itself to live in
environmental state
2::7=1 QU/Sk) =
E: J'
knowing the previous k realizations of the environment. Of course,
1 for each Sk. It should be understood that QUlsd is not a rule for
predicting the environmental process: rather these values represent an individual's choices or a strategy based on its previous history in past environments. We should distinguish two cases. In the first, the depth of an individual's memory k is not greater than m, that of the environmental process. In the other case, k > m. In the case we have Sk = (ii, i2 , ••• , ik) and in the second Sk = (ii, i 2 , .•• , i m, im+l .. , ik). In the latter situation we write Sk C Sm for abbreviation. Assume that the number of environmental changes, X. is large enough that models (5) and (7) may be regarded in terms of infinitesimal changes as N 5
--> 00.
If an individual
has no prior knowledge about the next environment, then the probability of the outcome E;j for that individuaL given Sk, is
(9) Hence, for the multiplicative modeL the fitness of such an individual is
IlIl E'(/(Sk),
W(Q) =
s"
(lOa)
ij
while for the additive model it is
(lOb)
W(Q) = L2:>;j(Sk)E;j. "II:
Note: If k > m. P{ilsk}
ij
= P{ils m } by definition. and IlIlE"~(Sm)
W(Q)
8m
=
(multiplicative .1
ij
(11)
LI>r;J;(Sm)E;j 8",
(additive),
ij
where
(12) with
L
Q+ {jlsm} =
.
P(Sk) Q{.I } P(Sm! JSk·
(13)
,$kCSm
4. A Semi-Optimal Strategy. An individual's total fitness depends on both the probability law of the environment,
P{ils m }, and that of its preparation, Q{j/sk}. We seek a good strategy Q{-Isk}, namely, one which gives rise to high individual fitness. Suppose that the individual knows the conditional probability law for the environment given the previous
e environments,
with
e::; m, i.e. it knows P{i/s e }. Such an individual will be said to possess a priori information of depth
e.
Since the environment is stationary
P{ilse} =
L
P(sm)P{ilsm}/P(s/)
"mes.!
and 6
(14)
P(sll = L
P(sm).
SmCs.(
Given a priori information of depth strategy,
Q,
e.
and memory of depth k, we define a semi-optimal
as one which maximizes l-V(Q):
l-V =
W(Q) = maxW(Q),
(15)
Q
where the fitness is determined in the multiplicative and additive cases by (10) if k and by (11)-(13), with m replaced bye, if k
>
e.
~
e
Below we shaH consider both the
multiplicative modeL for which
(16a)
InW= LL7l"ij(sk)lnEij , s~
Ij
and the additive model with
W =
LL 811:
(16b)
7l"ij(Sk)Eij.
ij
Since (16) are linear functions of Q's, their maxima are reached at 0 or 1: Result 1. Suppose that k
~
e.
Then. the semi-optimal strategy for multiplicative fitnesses
(4) is n E .. 1 ifLP{ilsk}lng'J >0 i=1
Q{jlsk} =
o
forall
jofj
t]o
(17a)
g.
n
if LP{ilsdln E.'J < 0 for at least one )0 fj. i=l
flo
If there exists a set J of more than one j for which
tP{ilsdln 1=1
;.i
j
> 0 for all )0 (j J
flo
and
then QUISk },j E J. are arbitrary within the simplex i
I:jEJ
Q{j ISk} = 1, Q{j ISk} ;::: O.
Under the semi-optimal strategy,
(18)
Result I'. Suppose that k S R. Then the semi-optimal strategy for additive fitnesses (7) is n
1
LP{ilsk}(E;j-E;jo»O forall
if
jof'j
i=1
Q{jlsk} =
(17')
n
o
LP{ilsk}(E;j - E;jo) < 0 for at least one jo f' j.
if
1=1
If there exists a set J of more than one j for which n
L P{ilsk}(Eij - Eijo) > 0 for all )0 rf- J j= 1
and n
LP{ilsk}(E;j,-E;j,)=O for
jj,hEJ
i=1
then QUlsk}, j E J, are arbitrary within the simplex I:jEJQ{jlsk}
= 1, Q{jlsk} 0 for all jo f' j i=1
o
110
E .. LP{ilst} In E.': < 0 for at least one j 0 n
if
i=1
130
If there exists a set J of more than one j for which n E .. L P{ilst} In E.': > 0 for all jo rf- J
i=l
s(
']0
and
8
(19)
f' j.
then QUlsk},j E J, are arbitrary within the simplex L:jEJQ{jlsk}
= 1, Q{jlsk} 2: O.
Under the semi-optimal strategy n
(20) If fitnesses are formed additively, then for all
Sk
C
St
Result !?!. Suppose that k > €. Then the semi-optimal strategy is n
1 if
LP{ilse}(Eij-Eijo»O
for all
joij
1=1
(19')
n
o
if
LP{ilse}(Eij-Eijo) 0 for all
jo
rf- J
i=l
and n
L
P{ilse}(Eij, - Eij,) = 0 for all
h,jz
E J.
i=l
then QUisk}, j E J, are arbitrary within the simplex L:jEJ Q{jlsk}
= 1, Q{jlsk} 2: O.
Under this semi-optimal strategy n
(20')
Corollary 1.
Both results show that, for given depths k of memory and
.e
of a priori
information, the semi-optimal strategy for models (10) depends only on the conditional probabilities P{ilsk} or P{ilst} (from (17), (17'), (19) and (19')) and not on the unconditional probabilities of the environment, P( Sk). This is true even though the total fitness 9
actually depends on both conditional and unconditional probabilities, as in (18), (18'), (20), and (20'). This means that in preparing for future environments. estimates of these conditional probabilities are the most valuable information, assuming, of course, that the individual knows the relevant values of E;j in the computation of the semi-optimal strategy. To see that Result 1 is true, note that for any set of numbers
and that the maximum is achieved at a boundary of the simplex
Cj, C2 , .•. , Cn
I>:l<j
= 1;
Ctj
2: O. For
the multiplicative model. this implies that for this boundary specified by j, n
n
LP{ilsd1nE;j 2: LP{ilsdlnE;jOl i==l
for every jo(jo
i=l
f. j).
For Result 1', introduce for m 2:
e (13')
(cf (13». It follows from the linearity of In W that Q+{jlse} has the form (17'). Hence,
from (13'), all of the terms QUlsd must be 1 or 0 for every
Sk
C
5/.
Results 1 and I' show that the semi-optimal strategy is importantly affected by both the knowledge about the environmental process with depth
eand the depth of an individ-
ual's memory, k. Result 1 says that the semi-optimal strategy is restricted by the depth of information, i.e. memory, k, if the depth of environmental knowledge is greater, i.e. if
e>
k. In this case, the additional knowledge about PUlse} has no value. Further, if
the depth of information is greater than that of the environmental law, i.e. k > additional information about the previous environments at times t -
e+ 1, ... , t -
e,
the
k also
has no value at time t. The effective parameter, therefore, is mine k, e). Corollary 2. If there is a cost for the acquisition of deeper information, k, or for more
advanced information about the environment, 10
e, then evolution tends to decrease Ie - kl.
It seems clear that deeper knowledge and information should increase fitness. In fact,
we have the following ReJult 3. Let k', e', k" and £" be such that mine k', £') > mine k", e"), and suppose that
W'
and vTTII are the fitnesses corresponding to the semi-optimal strategies for these pairs of parameters from Result 1. Then
W' ::: W".
The proof of Result 2 is provided in Appendix A. Note 1. Usually the inequality min(k'£') > min(k",e") implies
vV' > W"
and
W'
=
W"
only in the special case described in Appendix A. Note 2. The fitness
vv, determined by the semi-optimal strategy, increases as a function
of min(k,e) only if min(k,e)::; m: l'V reaches its greatest possible value at min(k,£) = m. Hence, increasing the depth of information I k) or the depth of knowledge (e) beyond the depth of the environmental law, m, has no effect. For this reason we consider only the case £ ::; m, k ::; m. Corollary 1. If k ::; £, that is, the depth of information, k. is less than the depth of
knowledge,
e.
the fitness is an increasing function of k until k exceeds
hand. the fitness remains constant if
e.
On the other
eincreases but k is fixed.
Corollary 2. If k > e, fitness is an increasing function of e until e ::: k, while it does not
increase in k for fixed
e.
Qkl the semi-optimal strategy defined for given depths, k, of memory and a priori information, £. Since the fitness corresponding to Qkl reaches its maximum as Note 3. Denote by
a function of k and £ at k = £ = m, and then remains constant for k ::: m. £ ::: m, only the strategy
Qm.m should be called optimal. Within the constraint of limited information, Qkl
is the best strategy. Increasing k and
e improves the semi-optimal strategy by increasing
the fitness of the individual at the semi-optimal strategy.
11
5. Special Case: Two Environmental States. Consider an environment with two states co and CI with corresponding fitness values
E oo , E 01 , E 10 , and Ell as defined above. Thus, for example, E 10 is the fitness when the current state of the environment is CI, but the state for which the individual prepared is 00·
Assume Ell > E 10 and E oo
with Q{1Isk}
> E o1 . From Result
1, a semi-optimal strategy Q{Olsk},
1 - Q{Olsk}, depends on the ratio of P{Olsk} In EE29 to P{1Isk}ln EE"' 01
10
Define Z by Z = In Ell
E 10
/In EllEoo .
(21)
E1oEo1
Then we have Ihe following corollary to Result 1: Result
4-
Suppose k ::; £. Then the semi-optimal strategy
_
Q{OISk} -
{1 o
Q satisfies
if
P{Olsk} > Z
if
P{Olsk} < Z
(22)
with an abitrary Q(O ::; Q ::; 1) if P{Olsk} = Z. Also we have Result
4'. Suppose k > t. For every
Sk
_
Q{OISk} =
ese, the semi-optimal strategy is
{1 o
if
P {Olse} > Z
if
P{Olse} < Z
(23)
with an arbitrary Q (0 ::; Q ::; 1) if P{Olse} = Z (for Sk C se). Results 4 and 4' constitute a very simple algorithm for obtaining the semi-optimal strategy. The decision rule is simply to compare the conditional probabilities of the environmental law, P{Olsk} with a single number Z for each Sk. These results also hold for the additive fitness model with Z defined by Z =
Ell - E 10 Ell + E oo - E 01 - Eo!
(21')
6. Randomly Varying Fitnesses. So far we have assumed that the parameters of the model are deterministic. In reality, however, they may be subject to individual estimation error. or be influenced by intrinsic
12
factors. Thus. in this section we consider fitnesses
E;j
which vary during an individual's
lifetime, the changes occurring when the environment changes. We take
E;j
to be contin-
uous random variables such that at each new environmental state, E ij are independent of the previous fitnesses. and are distributed with joint density function f(E ll , E I 2, . .• , E nn ). Denote the marginal density function of Eij by f;j(E;j) with fij(Eij) =
JJ...J
f(Ell ... Eij ... E nn ) dEll···. dE m
all
not
I}
Elrl
kl';'ij
with mean E ij and variance Vij given by (22) and 00
v:-- t) -
1 o
-
2
(x - E-.) f·I)·(x)dx • tJ
(23)
respectively. Also, define the logarithmic mean (24) We have Result 5. Suppose that the random fitnesses E ij are independent of the changing environments that occur during the lifetime of an individual. Then for the infinitesimal case (see eqns (10)) W(Q) =
nn
E"i;(SkJ
(25a)
ij
.91:
with multiplicative fitnesses (lOa), and W(Q)
=
L: L: 1l"ij(Sk)Eij Sk
(25b)
ij
for the additive case (lOb), where Eij = exp [lnEij] .
13
(26)
To prove this result note that the analogy to the fitness (4) in the present case is
W = [(E(I) ... E(Nl1») (E(I) ... E(N12l) .. , (E(I) ... E(Nnn»)] liN 11 11 12 12 nn nn
,
where E;j is the infinitesimal fitness of the individual at the event when the predicted environmental state is
and its realization is c;. Thus
Cj
W =
II Wij 'J
with
W,'J'
=
1 [E.. ... E .. •, 1liN = exp {.Vi; -lV '\' In E. (I)
(No,)
1)
I)
(v)
~ v=l
J
Ni L ;I V··
_ { N;j -exp - ' -1-
'v ...
as N
-+ 00.
•
t)
n
v=l
}
IJ
E(V)} .. 'J
This proves the result in the multiplicative case. For the additive model, the
proof is similar. Result 5 confirms that the general conclusions about semi-optimal strategies obtained in Results 1-3 remain valid for randomly distributed fitnesses. The only difference is the use of Eij instead of E;j for random additive fitnesses, and In E;j instead of In E;j for random multiplicative fitnesses.
The fact that In E;j is used, instead of In E;j, has qualitative
implications. Since for x close to
t;j
we can write
- dInxl 1 - 2d2lnxi Inx=lnE·+(x-E-·)-'J 'J dx X=Eij - +-(x-E-·) ?__ 'J dX 2 IX=Eij -
+ ... ,
we have 1 2 lnE-,~lnE .. --G, IJ 1) 2 I}'
where G;j is the coefficient of variation of E;j:
14
(27)
Hence, unlike the random additive modeL in the random multiplicative modeL optimal strategies depend on variation coefficients (and other statistics) of the fitness distribution.
7. Errors In Information About Parameters. We have seen that partial knowledge about the depth of the environment leads to inferior strategies and reduced individual fitness. It seems clear that errors in the knowledge of other parameters of the models should also weaken the strategy and decrease individual fitness, We now analyze this effect quantitatively in the special case of two environmental states
cO
and
C1'
We assume for simplicity that k :S
F{ilsk}
An individual considers to the strategy
Q{j Isd
and
Eij
e.
as exact parameters and behaves according
specified by
Z if F{Olsd < Z
• { 1 if F{Olsk} > Q{Olsk} = 0
(29)
where
Ell;!n.EooE_ll Z• = In. E 10 E01 E 10 (see Result 4). Define 51 as the set of Sk such that
(30)
Q{Olsd
= 0 and
Q{Olsd = 1, and 52
as the set of Sk with Q{Olsd = 1 and Q{O!sd = 0, In other words, the set 50 = 51 U 52
contains Sk for which the adopted strategy (29) is wrong. We denote by
W the total fitness for In W =
'L P(Sk) 'L P{ilsk}Q{jlsd In E
ij ,
vIr
In W;W =
defined by (17) and (18). For the multiplicative model then
'L
P(Sk){ P{Olsk} lnEoo
+ P{llsk} InE 10
S.l:ESl
- P{0Isk}lnE01
+
(31 )
ij
SA:
and recall the total fitness
the adopted strategy:
'L
-
P{llsk} In Ell
P(sk){-P{OlsdlnEoo
skE S 2
15
-
P{1Isk}lnE 10
After some rearrangement
oo E l1 { " j;V = In EEEL.... In -=P(Sk) [P{Olsk} - Z] W 01 10 ES "
1
L
+
P(Sk) [Z - P{OlsdJ }-
skES;
By definition, Q{Olsk} = 1, and from Result 4, P{O[sk} ~ Z for Sk E SI_ Similarly
P{Olsk} :::; Z for Sk E S2- Hence, we have Re3ult 6. For the two state environment, the difference between the fitnesses of the optimal and adopted strategies is
ll " In EooE EEL.... P(sk)[P{Olsk} 01
10
ZI_
(32)
ES "
0
This result shows that the greater the difference between the fitness consequences of accurate and inaccurate prediction, lnE oo -lnEo1 and In Ell -lnEIO , the greater the fitness reduction. The more frequently states s k occur where such errors are made, the greater is the fitness reduction. On the other hand, (32) shows that the consequences of inaccurate knowledge are relatively minor near the zone of unreliable prediction, namely P {Olsk} ::::: Z, where even small errors in parameters produce, e.g. a strategy Q{j ISk} =
°
instead of the
best QUlsk} = 1. etc. (see Result 5). Concluding Remarks. In the study of animal behavior, it is widely assumed that individuals act during their lifetime to improve their fitness (Shettleworth, 1984). Measures of individual fitness are difficult to design, however, and even in the study of economic and artificial systems the notion of an optimum may be difficult to make precise (Holland, 1975). Several mutually incompatible cost functions may be reasonable in anyone system. Nevertheless, the definition of a "good" strategy may assist in the development of a qualitatively useful set of principles concerning the evolution of learning and adaptation in fluctuating environments. In order to adapt to a fluctuating environment, an organism must have some information about the environment. Otherwise, in totally unpredictable conditions, the best it 16
can do is adopt a minimax strategy that minimizes fitness losses under the worst environmental conditions: the evolution of life history strategies in stochastic environments has been widely discussed (see e.g. Tuljapurkar, 1990: Stearns, 1992: Yoshimura and Clark, 1993). If, however. the individual has some knowledge about the environment, its strategy can be significantly improved. For example, such knowledge may improve foraging ability, and thereby improve fitness. In our model, the environment appears to the individual to be a multiorder Markov process. This may be the result of actual temporal fluctuations with a "memory" or be due to the individual's movement through a spatially varying habitat. In our model, the individual possesses two kinds of knowledge: a priori knowledge which contains information on the conditional probabilities of the environmental outcomes. and a posteriori knowledge which gives information about some previous environmental outcomes. The depth of a posteriori information. i.e. the depth of memory, k. and the depth,
e, of the
a priori information determine semi-optimal strategies which maximize individual fitness under these limited amounts of information. An optimal strategy would be obtained if the individual knew the probability law of the environmental stochastic process, and all necessary information about the realizations of previous environments. How good the semi-optimal strategies are depends on mint k, £). The larger is this value, the greater is the fitness at the semi-optimal strategy. It follows that in our model, differences between the depths of a priori and a posteriori knowledge are not desirable in an evolutionary sense. For example, if the a priori knowledge is greater than a posteriori,
e > k, increasing ewithout a corresponding increase in k does not produce an increase in fitness at the semi-optimal strategy. Of course, if increasing e incurs a cost, the fitness i.e.
may even decrease under these conditions. The best outcome for the model is agreement between k and
e, and these should be as large as possible.
A semi-optimal strategy is not a predictive function. It might better be regarded as a preparation function: an individual prepares to accrue a maximal expected contribution to its fitness in the pending environment. For example, in extreme cases, it may choose a constant strategy, preparing itself for the same environmental state if this occurs sufficiently often, and provided a large enough fitness increments accrue to those who choose it. When 17
the fitness parameters E ij include a random component. then differences emerge between the semi-optimal strategies that apply in additive and multiplicative cases. Thus in the additive case, it is sufficient for the individual to know the mean of the variable fitnesses. For the multiplicative case. however. at least the variance of these fitnesses must be known in addition to the mean. It is important that in our modeL with independent additive or multiplicative contributions to fitness. a semi-optimal strategy is locally deterministic. By this we mean that at the time points of environmental changes, the probabilities that an individual prepares for the various environmental options take values 0 and 1 rather than intermediate values. Thus, the individual knows for sure which of the possible future environmental states it must prepare for. using its a priori and a posteriori knowledge. At the same time, however. throughout an individual's lifetime. its behavior appears
to
be probabilistic be-
cause the individual can choose different environmental states depending on the changing a posteriori information it acquires as the environment changes during its life. Although the strategic choices are deterministic. viewed on a longer time scale they might appear too extremely random. This is reminiscent of animal behavior experiments in which the subject responded quite differently to the same stimulus. to the extent that the behavior was described as "spontaneous" (Manning, 1979). It should be emphasized that in the presence of errors in the a priori or a posteriori in-
formation. the semi-optimal strategy remains locally deterministic. The only consequence of erroneous or incomplete information is a decrease in the total fitness of an individual. Although our model is presented in terms of a priori and a posteriori information, and their use in the search for a semi-optimal strategy, we do not address how this information is acquired. This important issue. namely the role of learning in the process of preparation for environmental fluctuation. is currently under investigation.
18
Appendix A. Proof of Result 2. For definiteness, suppose k' :S
e'
and kif :S e" so that k'
> kif. It follows from Result
1 that n
n
n
n
n
n
= In IV". The equality in the proof occurs if and only if, for every SkU with Sk' C SkU,
P{iISk'} InE;j reaches its maximum in the simplex ary.
19
l : O<j
=
1,
O<j
l:j l:iO<j
20 on the same bound-
References Bergman, A., and M.W. Feldman. 1993. The evolution of learning. (Manuscript). Bush, R.R., and F. Mosteller. 1955. Stochastic Models For Learning. J. 'Wiley, New York.
365 p. Cohen, D. 1966. Optimising reproduction in a randomly varying environment. J. Theor.
Bioi. 12: 119-129. Cohen, D. 1967. Optimising reproduction in a randomly varying environment when a correlation may exist between the conditions at the time a choice has to be made and the subsequent outcome. J. Theor. Bioi. 16: 1-14. Cohen, D. 1993. Fitness in random environments. pp. 8-25. In Yoshimura, J. and C.W. Clark (eds. J Adaptation in Stochastic Environments. Springer-Verlag. Berlin, Heidelberg, N.Y. Gillespie, J.H. 1973. Polymorphism in random environments. Theor. Pop. BioI. 4: 193-195. Harley, C.B. 1981. Learning the evolutionarily stable strategy. J. Theor. Bioi. 89: 611-633. Harley, C.B., and J. Maynard Smith. 1983. Learning
~
An evolutionary approach. TINS
pp. 204-208. HartL B.L., and R.D. Cook. 1973. Balanced polymorphisms of quasi-neutral alleles. Theor.
Pop. Bioi. 4: 163-172. Holland, J.H. 1975. Adaptation in Natural and Artificial Systems. The Univ. of Michigan.
183 p. Houston, A.I. and B.H. Sumida. 1987. Learning rules, matching, and frequency dependence. J. Theor. Bioi. 126: 289-308.
Karlin, S., and U. Liberman. 1974. Random temporal variation in selection intensities: case of large population size. Theor. Pop. Bioi. 6: 355-382. Levins, R. 1968. Evolution in Changing Environment. Princeton Univ. Press., Princeton,
N.J. 120 p. Lewontin, R.C., and D. Cohen. 1969. On population in a randomly varying environment.
Proc. Nail. Acad. Sci. USA 62: 1056-1060. Manning, A. 1979. An Introduction to Animal Behavior. Addison-Wesley PubL Co., Reading, MA. 329 p.
20
Maynard Smith. J. 1982. Evolution and the Theory of Games. Cambridge Univ. Press, Cambridge, MA. 224 p. McNamara, J.M., and A.I. Houston. 1987. Memory and the efficient use of information. J. Theor. BioI. 125: 385-395.
Shettleworth, S.J. 1984. Learning and behavioral ecology. pp. 170-194. In Krebs, J.R. and N.B. Davies (eds.) Behavioural Ecology: An Evolutionary Approach, 2nd Ed. Sinauer Assoc., Sunderland. MA. Stearns, S.C. 1992. The Evolution of Life Histories. Oxford University Press, Oxford. Stephens, D.W. 1991. Change, regularity, and value in the evolution of animal learning. Behavioral Ecology 2: 77-89.
Tuljapurkar. S. 1990. Population Dynamics in Variable Environments. Springer-Verlag, Berlin. Yoshimura, J .. and C.W. Clark (eds.). 1993. Adaptation in Stochastic Environments. Springer- Verlag, Berlin.
21
SFI WORKING PAPER: 1994-02-004
SFI Working Papers contain accounts of scientific work of the author(s) and do not necessarily represent the views of the Santa Fe Institute. We accept papers intended for publication in peer-reviewed journals or proceedings volumes, but not papers that have already appeared in print. Except for papers by our external faculty, papers must be based on work done at SFI, inspired by an invited visit to or collaboration at SFI, or funded by an SFI grant. ©NOTICE: This working paper is included by permission of the contributing author(s) as a means to ensure timely distribution of the scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the author(s). It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may be reposted only with the explicit permission of the copyright holder. www.santafe.edu
SANTA FE INSTITUTE
A Mod el of Indi vidu al Ada ptive Beha vior in a Fluc tuati ng Envi ronm ent
Lev A. Zhivo tovsk yt+, Aviv Bergm an-+, and Marc us W. Feldm an*
tInsti tute of Gener al Genetics, Russi an Acade my of Sciences. 3 Gubki n St., GSP- l, B-333, Moscow 117809, Russia: +Interval Research Corpo ration , 1801 Page Mill Road. , Bldg. C, Palo Alto, Califo rnia, l'SA; and *Depa rtmen t of Biological Scienc es. Stanfo rd University, Stanfo rd, California 94305, USA
Key Word s: fitness, strateg y, Markov process, predic tion. optim ality
(8/11/ 93)
Abstract. Individual behavioral strategies that use conditional probabilities for future environments and infonnation about past environments are studied. The environments are random and Markovian. The individual uses the information available to it to prepare for the next environmental state in order to increase its fitness. The fitness depends on the discrepancy between the realized environment and that for which the individual is prepared. Additive and multiplicative combinations of the fitnesses accruing to the individual at each environmental epoch are studied. A semi-optimal strategy is found, which maximizes individual fitness given the depth of infonnation about the environment available to the individual. Randomly varying fitnesses and errors in the individual's perception of the environmental parameters may be included in the model.
1
Introduction. For many biological organisms the process of adaptation is one of survival and reproduction in an uncertain environment.
Two kinds of adaptations are important to
distinguish. The first might be viewed as occurring at the population level and envisages an array of different genotypes or phenotypes. each adapted to a characteristic range of environmental conditions. Taken as an ensemble, this array permits the population to adapt to changing conditions. This situation is usually modelled in terms of the evolution of the frequencies of the types under natural selection with emphasis on between-generation changes in environmental conditions (Lewontin and Cohen, 1969; Gillespie, 1973; Hartl and Cook, 1973; Karlin and Liberman. 1974; Stephens, 1991; Bergman and Feldman, 1993), or by making use of optimality reasoning where the distinction between generations is often blurred (Cohen, 1966. 1!J93; Harley, 1!J81; Harley and Maynard Smith. 1!J83; Houston and Sumida, 1987; McNamara and Houston, 1987). A second kind of adaptation may occur when individualJ exhibit plasticity that permits them to respond to environmental conditions in a manner that enhances their survival. Thus, individuals seek a general strategy that permits them to learn or seek a behavior which increases fitness (Shettleworth, 1984). For this response
to
be appropriate, the
individual should possess some "information" about the future environment, it should have stored information about previous environments (i.e. memory), and be able to predict and prepare for pending environments. These three properties contribute to the individual's ability to survive in an environment that changes within a generation. Both kinds of adaptations, population level and individual level. may have occurred in the process of evolution. Both may involve selection on behavioral differences between the types in a population. Ecological analyses of evolution in changing environments standardly consider that individual behaviors are strategies in a game against the environment (Maynard Smith, 1982). This framework assumes that individuals have no knowledge about their environments and cannot make predictions. In such situations a minimax strategy is a convenient way for an individual to counter the worst possible environmental conditions. In reality, however, the environment does not playa game with individuals, and individuals may have 2
some a priori and a posteriori information about their environment on a short-term time scale during their lifetime. Many models of behavior based on individual learning couch the learning process in terms of changes in the probabilities with which a stimulus is chosen due to previously acquired information about stimuli and responses (e.g. Bush and Mosteller, 1955). Here the environment is represented as a set of stimuli; and the conceptualization of environmental changes remains somewhat vague. In this paper we investigate how such adaptive behavior depends on the extent of historical knowledge about the environment, on the amount of prior knowledge about the probabilistic law that governs environmental change, and whether the individual's behavior is intrinsically probabilistic or deterministic for a given set of information. To address these questions we consider an environment with several possible different conditions, called states, whose temporal pattern follows given conditional probabilities. The individual possesses some information about both these probabilities and the history of previous environmental outcomes. The individual chooses which next (unknown) state to prepare for in order to increase its fitness which depends on the discrepancy between the expected and realized environmental state. 1. The Environment.
Suppose that at each discrete point in time, t, the environment is a random variable [, which can take values from the set
01,02, ...•
.on' {o,} constitute an m-order Markov
process. These random variables are described by the conditional probabilities
(1) where
Sm
can be abbreviated by the m-veetor (iI, i 2, ... , i m ), with each coordinate rep-
resenting one of the environmental states 1,2, ... , n. Thus, if m = 4, for example, and Sm
= (2,3,1. T), the process took the values
02
at t - 1,
03
at t - 2,
CI
at t - 3, and
C7
at
t - 4. Such an environment will be defined to have depth m. In the case m = 0, changes in the environment are completely random and. at any time, the values
CI, 02, ...
,cn occur
with probabilities PI, P2 , ••• , P n . If m = 1, the environment is a standard Markov process
3
in which only the previous state affects the present. In all cases the environment will be assumed to be stationary. That is, for each i
=
1,2, ... , n, the unconditional probability
that the environment is c; is
(2) where P(sm) is the stationary probability of the sequence of m environmental states, assumed to be independent of where in time the sequence occurs.
2. Individual Fitness. To an individual, environmental changes may be due either to actual temporal effects described by the conditional probabilities (1), or they may appear to occur because the individual migrates across a spatially varying habitat. In any case, we shall assume that the environment experienced by the individual is as described in the previous section by relations (1) with depth m. We assume that individuals are able to predict the environment and are able to prepare in some way for that predicted environment. Suppose that an individual predicts that the environment at time t will be in state c j, and that the environment at time tactually takes state
Ci.
In this case, a fitness E ij will be assigned to that individual. It is natural
to assume E i ; > Eij if j
l' i;
that is, correct prediction of the environment entails higher
Ii tness than incorrect. During an individual's lifetime, N environmental changes occur of which Nij are such that the individual predicted state
Cj
but the actual environmental state was
Cj'
Thus
(3) 'J
The total normalized multiplicative fitness of such an individual is
(4) or
w=
I1E':ii 'J
'J
4
'
(5)
where
is the frequency that
E: J
is predicted and
E: i
occurs.
One may also introduce the total
normalized additive fitness
(7) The multiplicative and additive models, both expressing independence of the contributions of future environments to the total fitness, are related since In W =
2::
nij
(8)
InEij .
'J
Some results which hold for multiplicative fitnesses are also valid for additive fitnesses just by changing In E ij to E ij . However. in spite of this close relationship between the two ~
models, in some complex situations the two models produce qualitatively different results. 3. Individual Strategies. Suppose that just prior to the realization of the environment at time t an individual knows the realizations of the k previous environments t - 1, t - 2, ... , t - k. We say that this individual possesses a posteriori information of depth k. or has a memory of depth k. Let QUlsd be the probability that such an individual prepares itself to live in
environmental state
2::7=1 QU/Sk) =
E: J'
knowing the previous k realizations of the environment. Of course,
1 for each Sk. It should be understood that QUlsd is not a rule for
predicting the environmental process: rather these values represent an individual's choices or a strategy based on its previous history in past environments. We should distinguish two cases. In the first, the depth of an individual's memory k is not greater than m, that of the environmental process. In the other case, k > m. In the case we have Sk = (ii, i2 , ••• , ik) and in the second Sk = (ii, i 2 , .•• , i m, im+l .. , ik). In the latter situation we write Sk C Sm for abbreviation. Assume that the number of environmental changes, X. is large enough that models (5) and (7) may be regarded in terms of infinitesimal changes as N 5
--> 00.
If an individual
has no prior knowledge about the next environment, then the probability of the outcome E;j for that individuaL given Sk, is
(9) Hence, for the multiplicative modeL the fitness of such an individual is
IlIl E'(/(Sk),
W(Q) =
s"
(lOa)
ij
while for the additive model it is
(lOb)
W(Q) = L2:>;j(Sk)E;j. "II:
Note: If k > m. P{ilsk}
ij
= P{ils m } by definition. and IlIlE"~(Sm)
W(Q)
8m
=
(multiplicative .1
ij
(11)
LI>r;J;(Sm)E;j 8",
(additive),
ij
where
(12) with
L
Q+ {jlsm} =
.
P(Sk) Q{.I } P(Sm! JSk·
(13)
,$kCSm
4. A Semi-Optimal Strategy. An individual's total fitness depends on both the probability law of the environment,
P{ils m }, and that of its preparation, Q{j/sk}. We seek a good strategy Q{-Isk}, namely, one which gives rise to high individual fitness. Suppose that the individual knows the conditional probability law for the environment given the previous
e environments,
with
e::; m, i.e. it knows P{i/s e }. Such an individual will be said to possess a priori information of depth
e.
Since the environment is stationary
P{ilse} =
L
P(sm)P{ilsm}/P(s/)
"mes.!
and 6
(14)
P(sll = L
P(sm).
SmCs.(
Given a priori information of depth strategy,
Q,
e.
and memory of depth k, we define a semi-optimal
as one which maximizes l-V(Q):
l-V =
W(Q) = maxW(Q),
(15)
Q
where the fitness is determined in the multiplicative and additive cases by (10) if k and by (11)-(13), with m replaced bye, if k
>
e.
~
e
Below we shaH consider both the
multiplicative modeL for which
(16a)
InW= LL7l"ij(sk)lnEij , s~
Ij
and the additive model with
W =
LL 811:
(16b)
7l"ij(Sk)Eij.
ij
Since (16) are linear functions of Q's, their maxima are reached at 0 or 1: Result 1. Suppose that k
~
e.
Then. the semi-optimal strategy for multiplicative fitnesses
(4) is n E .. 1 ifLP{ilsk}lng'J >0 i=1
Q{jlsk} =
o
forall
jofj
t]o
(17a)
g.
n
if LP{ilsdln E.'J < 0 for at least one )0 fj. i=l
flo
If there exists a set J of more than one j for which
tP{ilsdln 1=1
;.i
j
> 0 for all )0 (j J
flo
and
then QUISk },j E J. are arbitrary within the simplex i
I:jEJ
Q{j ISk} = 1, Q{j ISk} ;::: O.
Under the semi-optimal strategy,
(18)
Result I'. Suppose that k S R. Then the semi-optimal strategy for additive fitnesses (7) is n
1
LP{ilsk}(E;j-E;jo»O forall
if
jof'j
i=1
Q{jlsk} =
(17')
n
o
LP{ilsk}(E;j - E;jo) < 0 for at least one jo f' j.
if
1=1
If there exists a set J of more than one j for which n
L P{ilsk}(Eij - Eijo) > 0 for all )0 rf- J j= 1
and n
LP{ilsk}(E;j,-E;j,)=O for
jj,hEJ
i=1
then QUlsk}, j E J, are arbitrary within the simplex I:jEJQ{jlsk}
= 1, Q{jlsk} 0 for all jo f' j i=1
o
110
E .. LP{ilst} In E.': < 0 for at least one j 0 n
if
i=1
130
If there exists a set J of more than one j for which n E .. L P{ilst} In E.': > 0 for all jo rf- J
i=l
s(
']0
and
8
(19)
f' j.
then QUlsk},j E J, are arbitrary within the simplex L:jEJQ{jlsk}
= 1, Q{jlsk} 2: O.
Under the semi-optimal strategy n
(20) If fitnesses are formed additively, then for all
Sk
C
St
Result !?!. Suppose that k > €. Then the semi-optimal strategy is n
1 if
LP{ilse}(Eij-Eijo»O
for all
joij
1=1
(19')
n
o
if
LP{ilse}(Eij-Eijo) 0 for all
jo
rf- J
i=l
and n
L
P{ilse}(Eij, - Eij,) = 0 for all
h,jz
E J.
i=l
then QUisk}, j E J, are arbitrary within the simplex L:jEJ Q{jlsk}
= 1, Q{jlsk} 2: O.
Under this semi-optimal strategy n
(20')
Corollary 1.
Both results show that, for given depths k of memory and
.e
of a priori
information, the semi-optimal strategy for models (10) depends only on the conditional probabilities P{ilsk} or P{ilst} (from (17), (17'), (19) and (19')) and not on the unconditional probabilities of the environment, P( Sk). This is true even though the total fitness 9
actually depends on both conditional and unconditional probabilities, as in (18), (18'), (20), and (20'). This means that in preparing for future environments. estimates of these conditional probabilities are the most valuable information, assuming, of course, that the individual knows the relevant values of E;j in the computation of the semi-optimal strategy. To see that Result 1 is true, note that for any set of numbers
and that the maximum is achieved at a boundary of the simplex
Cj, C2 , .•. , Cn
I>:l<j
= 1;
Ctj
2: O. For
the multiplicative model. this implies that for this boundary specified by j, n
n
LP{ilsd1nE;j 2: LP{ilsdlnE;jOl i==l
for every jo(jo
i=l
f. j).
For Result 1', introduce for m 2:
e (13')
(cf (13». It follows from the linearity of In W that Q+{jlse} has the form (17'). Hence,
from (13'), all of the terms QUlsd must be 1 or 0 for every
Sk
C
5/.
Results 1 and I' show that the semi-optimal strategy is importantly affected by both the knowledge about the environmental process with depth
eand the depth of an individ-
ual's memory, k. Result 1 says that the semi-optimal strategy is restricted by the depth of information, i.e. memory, k, if the depth of environmental knowledge is greater, i.e. if
e>
k. In this case, the additional knowledge about PUlse} has no value. Further, if
the depth of information is greater than that of the environmental law, i.e. k > additional information about the previous environments at times t -
e+ 1, ... , t -
e,
the
k also
has no value at time t. The effective parameter, therefore, is mine k, e). Corollary 2. If there is a cost for the acquisition of deeper information, k, or for more
advanced information about the environment, 10
e, then evolution tends to decrease Ie - kl.
It seems clear that deeper knowledge and information should increase fitness. In fact,
we have the following ReJult 3. Let k', e', k" and £" be such that mine k', £') > mine k", e"), and suppose that
W'
and vTTII are the fitnesses corresponding to the semi-optimal strategies for these pairs of parameters from Result 1. Then
W' ::: W".
The proof of Result 2 is provided in Appendix A. Note 1. Usually the inequality min(k'£') > min(k",e") implies
vV' > W"
and
W'
=
W"
only in the special case described in Appendix A. Note 2. The fitness
vv, determined by the semi-optimal strategy, increases as a function
of min(k,e) only if min(k,e)::; m: l'V reaches its greatest possible value at min(k,£) = m. Hence, increasing the depth of information I k) or the depth of knowledge (e) beyond the depth of the environmental law, m, has no effect. For this reason we consider only the case £ ::; m, k ::; m. Corollary 1. If k ::; £, that is, the depth of information, k. is less than the depth of
knowledge,
e.
the fitness is an increasing function of k until k exceeds
hand. the fitness remains constant if
e.
On the other
eincreases but k is fixed.
Corollary 2. If k > e, fitness is an increasing function of e until e ::: k, while it does not
increase in k for fixed
e.
Qkl the semi-optimal strategy defined for given depths, k, of memory and a priori information, £. Since the fitness corresponding to Qkl reaches its maximum as Note 3. Denote by
a function of k and £ at k = £ = m, and then remains constant for k ::: m. £ ::: m, only the strategy
Qm.m should be called optimal. Within the constraint of limited information, Qkl
is the best strategy. Increasing k and
e improves the semi-optimal strategy by increasing
the fitness of the individual at the semi-optimal strategy.
11
5. Special Case: Two Environmental States. Consider an environment with two states co and CI with corresponding fitness values
E oo , E 01 , E 10 , and Ell as defined above. Thus, for example, E 10 is the fitness when the current state of the environment is CI, but the state for which the individual prepared is 00·
Assume Ell > E 10 and E oo
with Q{1Isk}
> E o1 . From Result
1, a semi-optimal strategy Q{Olsk},
1 - Q{Olsk}, depends on the ratio of P{Olsk} In EE29 to P{1Isk}ln EE"' 01
10
Define Z by Z = In Ell
E 10
/In EllEoo .
(21)
E1oEo1
Then we have Ihe following corollary to Result 1: Result
4-
Suppose k ::; £. Then the semi-optimal strategy
_
Q{OISk} -
{1 o
Q satisfies
if
P{Olsk} > Z
if
P{Olsk} < Z
(22)
with an abitrary Q(O ::; Q ::; 1) if P{Olsk} = Z. Also we have Result
4'. Suppose k > t. For every
Sk
_
Q{OISk} =
ese, the semi-optimal strategy is
{1 o
if
P {Olse} > Z
if
P{Olse} < Z
(23)
with an arbitrary Q (0 ::; Q ::; 1) if P{Olse} = Z (for Sk C se). Results 4 and 4' constitute a very simple algorithm for obtaining the semi-optimal strategy. The decision rule is simply to compare the conditional probabilities of the environmental law, P{Olsk} with a single number Z for each Sk. These results also hold for the additive fitness model with Z defined by Z =
Ell - E 10 Ell + E oo - E 01 - Eo!
(21')
6. Randomly Varying Fitnesses. So far we have assumed that the parameters of the model are deterministic. In reality, however, they may be subject to individual estimation error. or be influenced by intrinsic
12
factors. Thus. in this section we consider fitnesses
E;j
which vary during an individual's
lifetime, the changes occurring when the environment changes. We take
E;j
to be contin-
uous random variables such that at each new environmental state, E ij are independent of the previous fitnesses. and are distributed with joint density function f(E ll , E I 2, . .• , E nn ). Denote the marginal density function of Eij by f;j(E;j) with fij(Eij) =
JJ...J
f(Ell ... Eij ... E nn ) dEll···. dE m
all
not
I}
Elrl
kl';'ij
with mean E ij and variance Vij given by (22) and 00
v:-- t) -
1 o
-
2
(x - E-.) f·I)·(x)dx • tJ
(23)
respectively. Also, define the logarithmic mean (24) We have Result 5. Suppose that the random fitnesses E ij are independent of the changing environments that occur during the lifetime of an individual. Then for the infinitesimal case (see eqns (10)) W(Q) =
nn
E"i;(SkJ
(25a)
ij
.91:
with multiplicative fitnesses (lOa), and W(Q)
=
L: L: 1l"ij(Sk)Eij Sk
(25b)
ij
for the additive case (lOb), where Eij = exp [lnEij] .
13
(26)
To prove this result note that the analogy to the fitness (4) in the present case is
W = [(E(I) ... E(Nl1») (E(I) ... E(N12l) .. , (E(I) ... E(Nnn»)] liN 11 11 12 12 nn nn
,
where E;j is the infinitesimal fitness of the individual at the event when the predicted environmental state is
and its realization is c;. Thus
Cj
W =
II Wij 'J
with
W,'J'
=
1 [E.. ... E .. •, 1liN = exp {.Vi; -lV '\' In E. (I)
(No,)
1)
I)
(v)
~ v=l
J
Ni L ;I V··
_ { N;j -exp - ' -1-
'v ...
as N
-+ 00.
•
t)
n
v=l
}
IJ
E(V)} .. 'J
This proves the result in the multiplicative case. For the additive model, the
proof is similar. Result 5 confirms that the general conclusions about semi-optimal strategies obtained in Results 1-3 remain valid for randomly distributed fitnesses. The only difference is the use of Eij instead of E;j for random additive fitnesses, and In E;j instead of In E;j for random multiplicative fitnesses.
The fact that In E;j is used, instead of In E;j, has qualitative
implications. Since for x close to
t;j
we can write
- dInxl 1 - 2d2lnxi Inx=lnE·+(x-E-·)-'J 'J dx X=Eij - +-(x-E-·) ?__ 'J dX 2 IX=Eij -
+ ... ,
we have 1 2 lnE-,~lnE .. --G, IJ 1) 2 I}'
where G;j is the coefficient of variation of E;j:
14
(27)
Hence, unlike the random additive modeL in the random multiplicative modeL optimal strategies depend on variation coefficients (and other statistics) of the fitness distribution.
7. Errors In Information About Parameters. We have seen that partial knowledge about the depth of the environment leads to inferior strategies and reduced individual fitness. It seems clear that errors in the knowledge of other parameters of the models should also weaken the strategy and decrease individual fitness, We now analyze this effect quantitatively in the special case of two environmental states
cO
and
C1'
We assume for simplicity that k :S
F{ilsk}
An individual considers to the strategy
Q{j Isd
and
Eij
e.
as exact parameters and behaves according
specified by
Z if F{Olsd < Z
• { 1 if F{Olsk} > Q{Olsk} = 0
(29)
where
Ell;!n.EooE_ll Z• = In. E 10 E01 E 10 (see Result 4). Define 51 as the set of Sk such that
(30)
Q{Olsd
= 0 and
Q{Olsd = 1, and 52
as the set of Sk with Q{Olsd = 1 and Q{O!sd = 0, In other words, the set 50 = 51 U 52
contains Sk for which the adopted strategy (29) is wrong. We denote by
W the total fitness for In W =
'L P(Sk) 'L P{ilsk}Q{jlsd In E
ij ,
vIr
In W;W =
defined by (17) and (18). For the multiplicative model then
'L
P(Sk){ P{Olsk} lnEoo
+ P{llsk} InE 10
S.l:ESl
- P{0Isk}lnE01
+
(31 )
ij
SA:
and recall the total fitness
the adopted strategy:
'L
-
P{llsk} In Ell
P(sk){-P{OlsdlnEoo
skE S 2
15
-
P{1Isk}lnE 10
After some rearrangement
oo E l1 { " j;V = In EEEL.... In -=P(Sk) [P{Olsk} - Z] W 01 10 ES "
1
L
+
P(Sk) [Z - P{OlsdJ }-
skES;
By definition, Q{Olsk} = 1, and from Result 4, P{O[sk} ~ Z for Sk E SI_ Similarly
P{Olsk} :::; Z for Sk E S2- Hence, we have Re3ult 6. For the two state environment, the difference between the fitnesses of the optimal and adopted strategies is
ll " In EooE EEL.... P(sk)[P{Olsk} 01
10
ZI_
(32)
ES "
0
This result shows that the greater the difference between the fitness consequences of accurate and inaccurate prediction, lnE oo -lnEo1 and In Ell -lnEIO , the greater the fitness reduction. The more frequently states s k occur where such errors are made, the greater is the fitness reduction. On the other hand, (32) shows that the consequences of inaccurate knowledge are relatively minor near the zone of unreliable prediction, namely P {Olsk} ::::: Z, where even small errors in parameters produce, e.g. a strategy Q{j ISk} =
°
instead of the
best QUlsk} = 1. etc. (see Result 5). Concluding Remarks. In the study of animal behavior, it is widely assumed that individuals act during their lifetime to improve their fitness (Shettleworth, 1984). Measures of individual fitness are difficult to design, however, and even in the study of economic and artificial systems the notion of an optimum may be difficult to make precise (Holland, 1975). Several mutually incompatible cost functions may be reasonable in anyone system. Nevertheless, the definition of a "good" strategy may assist in the development of a qualitatively useful set of principles concerning the evolution of learning and adaptation in fluctuating environments. In order to adapt to a fluctuating environment, an organism must have some information about the environment. Otherwise, in totally unpredictable conditions, the best it 16
can do is adopt a minimax strategy that minimizes fitness losses under the worst environmental conditions: the evolution of life history strategies in stochastic environments has been widely discussed (see e.g. Tuljapurkar, 1990: Stearns, 1992: Yoshimura and Clark, 1993). If, however. the individual has some knowledge about the environment, its strategy can be significantly improved. For example, such knowledge may improve foraging ability, and thereby improve fitness. In our model, the environment appears to the individual to be a multiorder Markov process. This may be the result of actual temporal fluctuations with a "memory" or be due to the individual's movement through a spatially varying habitat. In our model, the individual possesses two kinds of knowledge: a priori knowledge which contains information on the conditional probabilities of the environmental outcomes. and a posteriori knowledge which gives information about some previous environmental outcomes. The depth of a posteriori information. i.e. the depth of memory, k. and the depth,
e, of the
a priori information determine semi-optimal strategies which maximize individual fitness under these limited amounts of information. An optimal strategy would be obtained if the individual knew the probability law of the environmental stochastic process, and all necessary information about the realizations of previous environments. How good the semi-optimal strategies are depends on mint k, £). The larger is this value, the greater is the fitness at the semi-optimal strategy. It follows that in our model, differences between the depths of a priori and a posteriori knowledge are not desirable in an evolutionary sense. For example, if the a priori knowledge is greater than a posteriori,
e > k, increasing ewithout a corresponding increase in k does not produce an increase in fitness at the semi-optimal strategy. Of course, if increasing e incurs a cost, the fitness i.e.
may even decrease under these conditions. The best outcome for the model is agreement between k and
e, and these should be as large as possible.
A semi-optimal strategy is not a predictive function. It might better be regarded as a preparation function: an individual prepares to accrue a maximal expected contribution to its fitness in the pending environment. For example, in extreme cases, it may choose a constant strategy, preparing itself for the same environmental state if this occurs sufficiently often, and provided a large enough fitness increments accrue to those who choose it. When 17
the fitness parameters E ij include a random component. then differences emerge between the semi-optimal strategies that apply in additive and multiplicative cases. Thus in the additive case, it is sufficient for the individual to know the mean of the variable fitnesses. For the multiplicative case. however. at least the variance of these fitnesses must be known in addition to the mean. It is important that in our modeL with independent additive or multiplicative contributions to fitness. a semi-optimal strategy is locally deterministic. By this we mean that at the time points of environmental changes, the probabilities that an individual prepares for the various environmental options take values 0 and 1 rather than intermediate values. Thus, the individual knows for sure which of the possible future environmental states it must prepare for. using its a priori and a posteriori knowledge. At the same time, however. throughout an individual's lifetime. its behavior appears
to
be probabilistic be-
cause the individual can choose different environmental states depending on the changing a posteriori information it acquires as the environment changes during its life. Although the strategic choices are deterministic. viewed on a longer time scale they might appear too extremely random. This is reminiscent of animal behavior experiments in which the subject responded quite differently to the same stimulus. to the extent that the behavior was described as "spontaneous" (Manning, 1979). It should be emphasized that in the presence of errors in the a priori or a posteriori in-
formation. the semi-optimal strategy remains locally deterministic. The only consequence of erroneous or incomplete information is a decrease in the total fitness of an individual. Although our model is presented in terms of a priori and a posteriori information, and their use in the search for a semi-optimal strategy, we do not address how this information is acquired. This important issue. namely the role of learning in the process of preparation for environmental fluctuation. is currently under investigation.
18
Appendix A. Proof of Result 2. For definiteness, suppose k' :S
e'
and kif :S e" so that k'
> kif. It follows from Result
1 that n
n
n
n
n
n
= In IV". The equality in the proof occurs if and only if, for every SkU with Sk' C SkU,
P{iISk'} InE;j reaches its maximum in the simplex ary.
19
l : O<j
=
1,
O<j
l:j l:iO<j
20 on the same bound-
References Bergman, A., and M.W. Feldman. 1993. The evolution of learning. (Manuscript). Bush, R.R., and F. Mosteller. 1955. Stochastic Models For Learning. J. 'Wiley, New York.
365 p. Cohen, D. 1966. Optimising reproduction in a randomly varying environment. J. Theor.
Bioi. 12: 119-129. Cohen, D. 1967. Optimising reproduction in a randomly varying environment when a correlation may exist between the conditions at the time a choice has to be made and the subsequent outcome. J. Theor. Bioi. 16: 1-14. Cohen, D. 1993. Fitness in random environments. pp. 8-25. In Yoshimura, J. and C.W. Clark (eds. J Adaptation in Stochastic Environments. Springer-Verlag. Berlin, Heidelberg, N.Y. Gillespie, J.H. 1973. Polymorphism in random environments. Theor. Pop. BioI. 4: 193-195. Harley, C.B. 1981. Learning the evolutionarily stable strategy. J. Theor. Bioi. 89: 611-633. Harley, C.B., and J. Maynard Smith. 1983. Learning
~
An evolutionary approach. TINS
pp. 204-208. HartL B.L., and R.D. Cook. 1973. Balanced polymorphisms of quasi-neutral alleles. Theor.
Pop. Bioi. 4: 163-172. Holland, J.H. 1975. Adaptation in Natural and Artificial Systems. The Univ. of Michigan.
183 p. Houston, A.I. and B.H. Sumida. 1987. Learning rules, matching, and frequency dependence. J. Theor. Bioi. 126: 289-308.
Karlin, S., and U. Liberman. 1974. Random temporal variation in selection intensities: case of large population size. Theor. Pop. Bioi. 6: 355-382. Levins, R. 1968. Evolution in Changing Environment. Princeton Univ. Press., Princeton,
N.J. 120 p. Lewontin, R.C., and D. Cohen. 1969. On population in a randomly varying environment.
Proc. Nail. Acad. Sci. USA 62: 1056-1060. Manning, A. 1979. An Introduction to Animal Behavior. Addison-Wesley PubL Co., Reading, MA. 329 p.
20
Maynard Smith. J. 1982. Evolution and the Theory of Games. Cambridge Univ. Press, Cambridge, MA. 224 p. McNamara, J.M., and A.I. Houston. 1987. Memory and the efficient use of information. J. Theor. BioI. 125: 385-395.
Shettleworth, S.J. 1984. Learning and behavioral ecology. pp. 170-194. In Krebs, J.R. and N.B. Davies (eds.) Behavioural Ecology: An Evolutionary Approach, 2nd Ed. Sinauer Assoc., Sunderland. MA. Stearns, S.C. 1992. The Evolution of Life Histories. Oxford University Press, Oxford. Stephens, D.W. 1991. Change, regularity, and value in the evolution of animal learning. Behavioral Ecology 2: 77-89.
Tuljapurkar. S. 1990. Population Dynamics in Variable Environments. Springer-Verlag, Berlin. Yoshimura, J .. and C.W. Clark (eds.). 1993. Adaptation in Stochastic Environments. Springer- Verlag, Berlin.
21
