An Evolutionary Algorithm with Population ... - Semantic Scholar
Accepted by the 2003 IEEE Congress on Evolutionary Computation. December 8-12, 2003, Camberra, Australia.
An Evolutionary Algorithm with Population Immunity and Its Application on Autonomous Robot Control1 Lei Wang
Béat Hirsbrunner
Dept. of Informatics, Univ. of Fribourg Ch. de Musée 3, CH-1700, Switzerland [email protected]
Dept. of Informatics, Univ. of Fribourg Ch. de Musée 3, CH-1700, Switzerland [email protected]
Abstract- The natural immune system is an important resource full of inspiration for the theory researchers and the engineering developers to design some powerful information processing methods aiming at difficult problems. Based on this consideration, a novel optimalsearching algorithm, the immune mechanism based evolutionary algorithm-IMEA, is proposed for the purpose of finding an optimal/quasi-optimal solution in a multi-dimension space. Different from the ordinary evolutionary algorithms, on one hand, due to the long-term memory, IMEA has a better capability of learning from its experience, and on the other hand, with the clonal selection, it is able to keep from the premature convergence of population. With the simulation on autonomous robot control, it is proved that IMEA is good at the task of adaptive adjustment (offline), and it can improve the robot's capability of reinforcement learning, so as to make itself able to sense its around dynamic environment.
1 Introduction Over the years, biology has provided a rich source of inspiration for the researchers in many different areas. Especially in the field of computer science, a lot of biological phenomena, such as evolution of species, information processing in DNA, immune defense of an individual and etc., are referred to improve the existing computer structure or to design some novel and powerful optimal algorithms. For example, the work done in the late 1960s and the early 1970s at the University of Michigan under the direction of John Holland led to the genetic algorithm (Goldberg 1989). And also in the 1960s, Fogel et al. referred some related genetic and evolutionary theories in biology and introduced them into the field of engineering, and then a globally optimal algorithm called evolutionary programming was gradually formed (Fogel 1966). Genetic algorithm and other kinds of evolutionary algorithms (EAs) have an iterative process of generation and test that is simulated to the natural phenomena and
mechanisms. However, EAs pay more emphasis on the phenotype, that is, the observed behavior of a particular coding structure. This coding for a given problem can be regarded as an abstraction of phenotypic traits (expressed characteristics). There are no restrictions on the suitable representation. Furthermore, there are no restrictions on the possible transformation operations (mutations); these, in turn, follow from the representation. On the other hand, an important research direction in this area, artificial immune system (AIS), is being paid more and more attention by more and more researchers. Most of the inspiration for this research has been drawn from the adaptive immune system (IS) in nature. From the viewpoint of immunology, the ability of IS to detect and eliminate most pathogens is essential for survival; and its architecture is multi-layered, with defenses on several levels. The adaptive IS is so-called because it adapts or "learns" to recognize specific kinds of pathogens, and retains a "memory" of them for speeding up future responses. In engineering, the concept of immunity can be traced back to 1986, proposed by Farmer et al. (Farmer 1986). In this work, they described a dynamical model of immune system that consisted of a set of differential equations, together with a threshold to remove useless antibody types, and genetic operators to introduce new ones. Since that time, AIS modeling has advanced significantly, and the emphasis has switched from idiotypic network theory to other aspect of immunology. For example, Kephart et al., designed some special decoy programs and let them reside on the user's PC for virus checking (Kephart 1997). If an unknown virus is detected, its sample is first captured and then sent to a central processing system for further examination. So this process is regarded analogous to how the innate immune system works, namely the immune response mechanism. A different approach was proposed by Forrest et al., who described a distributed, robust, dynamic, diverse and adaptive AIS for network security (Hofmeyr 2000 and Newman 2002). This system captures many features of the
1 This work is supported by the Swiss National Science Foundation no 2000-65301.01.
vertebrate immune system and places them in the context of the problem of protecting a network of computers from illegal intrusions. In addition, Dasgupta focused on investigating immunological principles in designing a multi-agent system for intrusion/anomaly detection and response in networked computers (Dasgupta 2002 and Nasaroui 2002). In his approach, the immunity- based agents roam around the machines, and monitor the situation in the network. Such an agent is reported to be able to learn and adapt to its environment dynamically and to detect both known and unknown intrusions. On the other hand, A novel dada analysis technique inspired by the nature immune system was proposed by Timmis, in which immunological metaphors are extracted, simplified and applied to create an effective data analysis technique (Timmis 2001 and Knight 2002). De Castro et al. proposed a novel artificial immune network model (aiNet or ABNet) with the main goals of clustering and filtering crude data sets described by high-dimensional samples (De Castro 2002 and 2003). This network model is capable of reducing redundancy and describing data structure, including their spatial distribution and cluster inter-relations. For the application of AIS, Chun et al. presented a successful method for employing AIS for the optimization of an electromagnetic device shapes (Chun 1997). Takahashi et al. employed a self-tuning immune feedback controller to control mechanical systems (Takahashi 1997). Ishiguro also designed another artificial immune network to control robots walking (Ishiguro 1998). In summary, evolution can be regarded as an optimization process with the selection mechanism of survival of the fittest; and immunity is the means by which the organism protects itself through the neutralization between antibodies and antigens. The immune system possesses a kind of special memory and can identify a population or an individual. If the existing evolutionary algorithms are considered as an organism, then the degeneracy rising inevitably from performing the algorithm can be regarded as an outside antigen. It can be looked upon as a neutral course that the algorithm utilizes the characteristic information of the problem under study to restrain the above degeneracy through an artificial immune operator (a kind of algorithmic approach). This operator takes effect of survival of the fittest on evolution algorithm, and helps to make the operation progress along the optimal direction. In the following content, at first, some basic concepts and knowledge about biological immune systems are briefly introduced in Section 2, which is necessary for learning how this kind of natural intelligent system works, and from which we may find some inspiration for designing our algorithms. After that, a detailed description about an immune mechanism based evolutionary algorithm is given,
which has a long-term memory for better learning from the experience of its history operation and can avoid the premature convergence of population with the clonal selection operation. In Section 3, as an application of this algorithm, some detailed processes/methods are presented that are used for the problem of autonomous robot control. Furthermore, the corresponding simulation results are provided and analyzed in Section 4. Finally, in Section 5, a brief summary is made and some ideas about future work are also proposed.
2 The evolutionary algorithm with population immunity 2.1 Brief presentation of natural immunity The immune system/network is usually regarded as a systematic integration of all different kinds of molecules, sells and organs that possess or execute some special functions, such as perceiving/detecting the exogenous infectious microorganisms and adjusting the dysfunction from our own organs. According to the difference on the detailed function, there are two sub-systems in our immune system, i.e., the innate immune system and the adaptive immune system (De Castro 1999). In which, the innate immune system suggests that a body has the born ability to recognize some pathogens and destroy them on the first encounter; while the adaptive immune system uses somatically generated antigen receptors that are clonally distributed on the two types of lymphocytes: B cells and T cells. These antigen receptors are generated from random processes, and as a consequence, the general design of the adaptive immune response is based on the clonal selection of the lymphocytes expressing receptors with particular specificities. The immune system is able to automatically produce some corresponding antibodies aiming to a certain foreign antigen. With combining to these antigens, the antibodies will destroy them through a series of reactions, and simultaneously, there are also some reactions of stimulus and/or restraint among the antibodies. Especially, these reactions, on one hand, are helpful for generating more mature antibodies; on the other hand, they are based on the density of the antibodies, in which there are two outstanding features: 1. The diversity of a certain kind of antibodies, which suggests that the immune mechanism ensures all the variety of antibodies existing in the organism at any time. 2. The density controls the property of reactions, i.e., stimulus or restrain. That means the higher density of the antibodies, the more possibility of restraint; on the contrary, the lower density, the more possibility of stimulus, which makes the antibodies
have some capabilities of adapting their around environment. In the following content, the main features of the above phenomena are used to design a corresponding operation in order to improve the algorithm's adaptive ability. Especially in this paper, the word of antigen is corresponded to the problem to be dealt with, and antibody is supposed to be a possible answer/solution to this problem. The calculation on an individual's vitality can be understood as the process of an antibody bending to the antigen. 2.2 The immune mechanism based evolutionary algorithm-IMEA As we know, in recent years, research on the evolutionary algorithms mainly focuses on the optimal problem shown as follows:
{
max F ( x) x ∈ Ω ⊂ R n
}
(2.1)
where Ω⊂Rn is a compact set and F(x) is the fitness function in Ω that is continuous and greater than zero. Our purpose is to find a quasi-optimal solution x*∈Ω such that the error between x* and the theoretical optimum satisfies the prescribed precision (∆=10-l ). Step 1 Initialization. Let k=1, and generate µ (µ is an even) individuals randomly as the initial mature population. The searching space of the quasi-optimal solution x* is composed of mesh points in Rn, in which each component is the integer times of ∆ . An individual in the evolutionary population suggests a point (x,σ), where n x = ( x1 , x2 ,L, xn ) ∈ X ⊂ R means an approximate solution to the problem concerned. x*∈X is just the goal to be found such that f ( x*) = max f ( x ) =: f * , where f* is x∈ X the maximal fitness in X , that is, the quasi-maximal fitness in Ω (suppose f(x)=0 when x∉Ω ). Each x in the initial population is generated independently according to the uniform distribution in Ω. For any individual, σ = (σ 1 ,σ 2 , L ,σ n ) ∈ R+n suggests the direction and step-length in the next searching. In the initial population, σi is generated independently as follows: σ i = e ui , u i ~ U ([α i , β i ])
i = 1,2, K , µ
where ui ~U([αi , βi ]) means ui submits to an uniform distribution on [αi, βi] which is associated with the size and shape of Ω. Step 2 Crossover. At first, the current mature population is randomly separated into two sub-sets with the same size, that is each has µ/2 individuals. And then, for each individual (xi,σi), i=1,2,...,µ/2, in the first set, another individual (xi1,σi1), i=1,2,...,µ/2, in the second set is selected to make the operation of crossover. For the view of simplicity, their two solutions, y and y1, have the same
searching step-length σ', that is these two new individuals are noted as (yi,σi') and (yi1, σi'). In which, yi and yi1 are generated from the operation on xi and xi1 with the traditional one-point/multi-points crossover method (Goldberg 1989), and σi ' is defined as follows: σ i′ = uiσ i + (1 − ui )σ i1 , ui ~ U ([0,1]) i = 1,2,K , µ 2 As a result of the crossover operation, a temporary population is produced that also has µ individuals. Step 3 Mutation. For each individual (xi,σi), i=1,2,...,µ, in the temporary population, a new one (xi',σi'), i=1,2,...,µ, is generated as follows: σ i′ = σ i e ui ,
x i′ = x i + ξ i , i = 1,2, K, µ
where ui ~U([αi, βi]), ξi is the greatest decimal with l bits which is no more than the random variable ηi ~ N (0, σ i′ 2 ) . Step 4 Clonal Selection. This operation is composed of two steps that are respectively: 1. Updating of the memory cells set. The memory set is composed of m individuals (in fact, it is also a special population). In the current population, ϕ (ϕ
An Evolutionary Algorithm with Population Immunity and Its Application on Autonomous Robot Control1 Lei Wang
Béat Hirsbrunner
Dept. of Informatics, Univ. of Fribourg Ch. de Musée 3, CH-1700, Switzerland [email protected]
Dept. of Informatics, Univ. of Fribourg Ch. de Musée 3, CH-1700, Switzerland [email protected]
Abstract- The natural immune system is an important resource full of inspiration for the theory researchers and the engineering developers to design some powerful information processing methods aiming at difficult problems. Based on this consideration, a novel optimalsearching algorithm, the immune mechanism based evolutionary algorithm-IMEA, is proposed for the purpose of finding an optimal/quasi-optimal solution in a multi-dimension space. Different from the ordinary evolutionary algorithms, on one hand, due to the long-term memory, IMEA has a better capability of learning from its experience, and on the other hand, with the clonal selection, it is able to keep from the premature convergence of population. With the simulation on autonomous robot control, it is proved that IMEA is good at the task of adaptive adjustment (offline), and it can improve the robot's capability of reinforcement learning, so as to make itself able to sense its around dynamic environment.
1 Introduction Over the years, biology has provided a rich source of inspiration for the researchers in many different areas. Especially in the field of computer science, a lot of biological phenomena, such as evolution of species, information processing in DNA, immune defense of an individual and etc., are referred to improve the existing computer structure or to design some novel and powerful optimal algorithms. For example, the work done in the late 1960s and the early 1970s at the University of Michigan under the direction of John Holland led to the genetic algorithm (Goldberg 1989). And also in the 1960s, Fogel et al. referred some related genetic and evolutionary theories in biology and introduced them into the field of engineering, and then a globally optimal algorithm called evolutionary programming was gradually formed (Fogel 1966). Genetic algorithm and other kinds of evolutionary algorithms (EAs) have an iterative process of generation and test that is simulated to the natural phenomena and
mechanisms. However, EAs pay more emphasis on the phenotype, that is, the observed behavior of a particular coding structure. This coding for a given problem can be regarded as an abstraction of phenotypic traits (expressed characteristics). There are no restrictions on the suitable representation. Furthermore, there are no restrictions on the possible transformation operations (mutations); these, in turn, follow from the representation. On the other hand, an important research direction in this area, artificial immune system (AIS), is being paid more and more attention by more and more researchers. Most of the inspiration for this research has been drawn from the adaptive immune system (IS) in nature. From the viewpoint of immunology, the ability of IS to detect and eliminate most pathogens is essential for survival; and its architecture is multi-layered, with defenses on several levels. The adaptive IS is so-called because it adapts or "learns" to recognize specific kinds of pathogens, and retains a "memory" of them for speeding up future responses. In engineering, the concept of immunity can be traced back to 1986, proposed by Farmer et al. (Farmer 1986). In this work, they described a dynamical model of immune system that consisted of a set of differential equations, together with a threshold to remove useless antibody types, and genetic operators to introduce new ones. Since that time, AIS modeling has advanced significantly, and the emphasis has switched from idiotypic network theory to other aspect of immunology. For example, Kephart et al., designed some special decoy programs and let them reside on the user's PC for virus checking (Kephart 1997). If an unknown virus is detected, its sample is first captured and then sent to a central processing system for further examination. So this process is regarded analogous to how the innate immune system works, namely the immune response mechanism. A different approach was proposed by Forrest et al., who described a distributed, robust, dynamic, diverse and adaptive AIS for network security (Hofmeyr 2000 and Newman 2002). This system captures many features of the
1 This work is supported by the Swiss National Science Foundation no 2000-65301.01.
vertebrate immune system and places them in the context of the problem of protecting a network of computers from illegal intrusions. In addition, Dasgupta focused on investigating immunological principles in designing a multi-agent system for intrusion/anomaly detection and response in networked computers (Dasgupta 2002 and Nasaroui 2002). In his approach, the immunity- based agents roam around the machines, and monitor the situation in the network. Such an agent is reported to be able to learn and adapt to its environment dynamically and to detect both known and unknown intrusions. On the other hand, A novel dada analysis technique inspired by the nature immune system was proposed by Timmis, in which immunological metaphors are extracted, simplified and applied to create an effective data analysis technique (Timmis 2001 and Knight 2002). De Castro et al. proposed a novel artificial immune network model (aiNet or ABNet) with the main goals of clustering and filtering crude data sets described by high-dimensional samples (De Castro 2002 and 2003). This network model is capable of reducing redundancy and describing data structure, including their spatial distribution and cluster inter-relations. For the application of AIS, Chun et al. presented a successful method for employing AIS for the optimization of an electromagnetic device shapes (Chun 1997). Takahashi et al. employed a self-tuning immune feedback controller to control mechanical systems (Takahashi 1997). Ishiguro also designed another artificial immune network to control robots walking (Ishiguro 1998). In summary, evolution can be regarded as an optimization process with the selection mechanism of survival of the fittest; and immunity is the means by which the organism protects itself through the neutralization between antibodies and antigens. The immune system possesses a kind of special memory and can identify a population or an individual. If the existing evolutionary algorithms are considered as an organism, then the degeneracy rising inevitably from performing the algorithm can be regarded as an outside antigen. It can be looked upon as a neutral course that the algorithm utilizes the characteristic information of the problem under study to restrain the above degeneracy through an artificial immune operator (a kind of algorithmic approach). This operator takes effect of survival of the fittest on evolution algorithm, and helps to make the operation progress along the optimal direction. In the following content, at first, some basic concepts and knowledge about biological immune systems are briefly introduced in Section 2, which is necessary for learning how this kind of natural intelligent system works, and from which we may find some inspiration for designing our algorithms. After that, a detailed description about an immune mechanism based evolutionary algorithm is given,
which has a long-term memory for better learning from the experience of its history operation and can avoid the premature convergence of population with the clonal selection operation. In Section 3, as an application of this algorithm, some detailed processes/methods are presented that are used for the problem of autonomous robot control. Furthermore, the corresponding simulation results are provided and analyzed in Section 4. Finally, in Section 5, a brief summary is made and some ideas about future work are also proposed.
2 The evolutionary algorithm with population immunity 2.1 Brief presentation of natural immunity The immune system/network is usually regarded as a systematic integration of all different kinds of molecules, sells and organs that possess or execute some special functions, such as perceiving/detecting the exogenous infectious microorganisms and adjusting the dysfunction from our own organs. According to the difference on the detailed function, there are two sub-systems in our immune system, i.e., the innate immune system and the adaptive immune system (De Castro 1999). In which, the innate immune system suggests that a body has the born ability to recognize some pathogens and destroy them on the first encounter; while the adaptive immune system uses somatically generated antigen receptors that are clonally distributed on the two types of lymphocytes: B cells and T cells. These antigen receptors are generated from random processes, and as a consequence, the general design of the adaptive immune response is based on the clonal selection of the lymphocytes expressing receptors with particular specificities. The immune system is able to automatically produce some corresponding antibodies aiming to a certain foreign antigen. With combining to these antigens, the antibodies will destroy them through a series of reactions, and simultaneously, there are also some reactions of stimulus and/or restraint among the antibodies. Especially, these reactions, on one hand, are helpful for generating more mature antibodies; on the other hand, they are based on the density of the antibodies, in which there are two outstanding features: 1. The diversity of a certain kind of antibodies, which suggests that the immune mechanism ensures all the variety of antibodies existing in the organism at any time. 2. The density controls the property of reactions, i.e., stimulus or restrain. That means the higher density of the antibodies, the more possibility of restraint; on the contrary, the lower density, the more possibility of stimulus, which makes the antibodies
have some capabilities of adapting their around environment. In the following content, the main features of the above phenomena are used to design a corresponding operation in order to improve the algorithm's adaptive ability. Especially in this paper, the word of antigen is corresponded to the problem to be dealt with, and antibody is supposed to be a possible answer/solution to this problem. The calculation on an individual's vitality can be understood as the process of an antibody bending to the antigen. 2.2 The immune mechanism based evolutionary algorithm-IMEA As we know, in recent years, research on the evolutionary algorithms mainly focuses on the optimal problem shown as follows:
{
max F ( x) x ∈ Ω ⊂ R n
}
(2.1)
where Ω⊂Rn is a compact set and F(x) is the fitness function in Ω that is continuous and greater than zero. Our purpose is to find a quasi-optimal solution x*∈Ω such that the error between x* and the theoretical optimum satisfies the prescribed precision (∆=10-l ). Step 1 Initialization. Let k=1, and generate µ (µ is an even) individuals randomly as the initial mature population. The searching space of the quasi-optimal solution x* is composed of mesh points in Rn, in which each component is the integer times of ∆ . An individual in the evolutionary population suggests a point (x,σ), where n x = ( x1 , x2 ,L, xn ) ∈ X ⊂ R means an approximate solution to the problem concerned. x*∈X is just the goal to be found such that f ( x*) = max f ( x ) =: f * , where f* is x∈ X the maximal fitness in X , that is, the quasi-maximal fitness in Ω (suppose f(x)=0 when x∉Ω ). Each x in the initial population is generated independently according to the uniform distribution in Ω. For any individual, σ = (σ 1 ,σ 2 , L ,σ n ) ∈ R+n suggests the direction and step-length in the next searching. In the initial population, σi is generated independently as follows: σ i = e ui , u i ~ U ([α i , β i ])
i = 1,2, K , µ
where ui ~U([αi , βi ]) means ui submits to an uniform distribution on [αi, βi] which is associated with the size and shape of Ω. Step 2 Crossover. At first, the current mature population is randomly separated into two sub-sets with the same size, that is each has µ/2 individuals. And then, for each individual (xi,σi), i=1,2,...,µ/2, in the first set, another individual (xi1,σi1), i=1,2,...,µ/2, in the second set is selected to make the operation of crossover. For the view of simplicity, their two solutions, y and y1, have the same
searching step-length σ', that is these two new individuals are noted as (yi,σi') and (yi1, σi'). In which, yi and yi1 are generated from the operation on xi and xi1 with the traditional one-point/multi-points crossover method (Goldberg 1989), and σi ' is defined as follows: σ i′ = uiσ i + (1 − ui )σ i1 , ui ~ U ([0,1]) i = 1,2,K , µ 2 As a result of the crossover operation, a temporary population is produced that also has µ individuals. Step 3 Mutation. For each individual (xi,σi), i=1,2,...,µ, in the temporary population, a new one (xi',σi'), i=1,2,...,µ, is generated as follows: σ i′ = σ i e ui ,
x i′ = x i + ξ i , i = 1,2, K, µ
where ui ~U([αi, βi]), ξi is the greatest decimal with l bits which is no more than the random variable ηi ~ N (0, σ i′ 2 ) . Step 4 Clonal Selection. This operation is composed of two steps that are respectively: 1. Updating of the memory cells set. The memory set is composed of m individuals (in fact, it is also a special population). In the current population, ϕ (ϕ
