Research on Artificial Psychology Based on ... - Semantic Scholar
The International Journal of Virtual Reality, 2008, 7(2):33-40
33
Research on Artificial Psychology Based on Multimodal Interactive Service Robot Jun Yu 1, 2, Zhiliang Wang 1, Lun Xie 1 and Yongxiang Xia 2 1 2
School of Information Engineering, University of Science & Technology Beijing, Beijing, China. 1 Navy Flight Academy, Hulu Dao, Liaoning Province, China.
Abstract—In the field of human robot interaction (HRI), providing robot with emotions and psychology like human can be useful to achieve natural interaction. Previous HRI research focused on issues such as gesture recognition, speech recognition and path planning etc. Using the affective information obtained from multimodal interaction can improve the robot’s perception to interaction object and environment, and then can improve the harmoniousness of HRI. The paper puts forward a method of decision-making on multimodal interaction information fusion of service robot based on artificial psychology. The experimental results demonstrate that the proposed method can be efficient and effective in HRI. Index Terms—Artificial psychology, behavior decision-making, multimodal interaction information fusion, service robot.
I.
INTRODUCTION
Robotics research is rising from 1980’s in various applications to serve human. It is an important milestone of multidisciplinary common progress such as manufacturing, information technology and cognitive psychology etc. In these days, it has shown a personal computer to personal service robot development trends in service industry. People hope the service robot could be the indispensable partner of human. For service robot to act according to its own states and environment can be useful to interaction. In the field of robot, it is the focus of researchers' attention to eliminate the obstacles between human and robot and endow robot with the abilities of apperceiving the environment and the emotion of human who it communicates with. It is one of the important research that how to apperceive and understand the interaction intention of human according to multimodal interaction information obtained by sense organs. Generally speaking, the research of the harmonious coexistence of service robots and human focuses on the method and implementation of analyzing and comprehending to multimodal information during interaction [1]. Because of the various contents of interaction and different application, it is very difficult to investigate HRI satisfying to any background. Therefore, the research usually focuses on the special application and utilizes the professional knowledge to analyze, comprehend and achieve special HRI [2]. The multimodal information fusion research is not limited to Manuscript received on April 15th, 2008. E-Mail: [email protected].
vision information and speech information. It includes facial expression, body posture, and speech information etc. based on digital feature expression. Multimodal information can remedy defects just using one kind of information. That is beneficial to the right decision making and natural HRI. II.
THE MULTIMODAL INTERACTION PLATFORM OF SERVICE ROBOT
2.1 Multi-sensor information fusion The research of service robot must begin from the imitating human. By observed human’s actions, it can be found that human received the environment information from the sense organs such as vision, hearing, nose, feeling etc. The information transmitted to the brain through nerves. Brain processes the information, integrates the information, and sends the action instructions to body to perform certain actions. When designing the service robot, researchers can take computer as brain, take executive mechanism as body, and take all kinds of sensors as human’s organs. In order to enable service robot having intelligence and responding to environment change, first, the robot should sense the environment. Then, the robot can deal with and integrate the information sensed by sensors. In the early research about this, people focus on the research and development of all kinds of sensors and just used one sense for robot in an application. Little thing was done on taking all senses as whole to analyze. Because different sensor apperceives different (or similar) side of information in the same environment, the information is related. Therefore, multimodal information fusion is a new research field to deal with the information more effective. Generally, there are many advantages to utilize the multi-senses system and information fusion. The first is fault-tolerant features. When a sensor or even more is in failure, the robot can still work normally with other senses apperceiving environment information. The second is that can improve accuracy. In the information apperceived, inevitably there are all sorts of noise. To use a number of information describing the same characteristics at the same time, uncertainty caused by such imprecise measurements can be reduced. And for complementary characteristic of information, the ambiguities of understanding environment can be reduced and correct decision-making ability of the system can be improved. That is researchers can use a number of less expensive sensors to obtain the same or even better performance that can be obtained by expensive single precision
34
The International Journal of Virtual Reality, 2008, 7(2):33-40
sensor. This can greatly reduce the cost of system. 2.2 Multimodal interactive platform of service robot Multimodal information is the various modes of perceived feedback information, expressing intent of users, or the implementation of action such as language, eye expression, facial expressions, gestures, body posture, touch, taste or smell etc. Through natural, happy and harmonious intercourse paradigm of human, the results of psychology research show that only 11% of the information obtained through the hearing, 83% through vision. Thus, psychologists propose a formula: emotional expression=7% of the words+39% of the voice +54% of the emotional action. These basic models in the perception of digital expression level based on human brain are complementary each other, isolating them will lose integrity of the information expression. According to it, this paper extracts not only meaningful characteristics of a single model but also comprehensive information features of different modes composition. Cooperative work of models can provide the necessary information to complete specific interactive tasks and can achieve the interaction between human and service robot. In the interaction process of human and service robot, it is better to adopt natural interactive mode as possible, such as voice, facial expressions, vision, proximity sensing, touch etc. The paper develops a high efficient and general architecture of service robot. Its main purpose is to construct a humanoid robot to play with children and combine education with recreation. In the affective robot architecture, we can utilize speech technology, image technology, and sensor technology to imitate the affective identification means of human. Therefore more reliable recognition effect can be achieved. Fig.1 is the schematic diagram of multimodal interactive platform of service robot designed. The main functions of it are speech recognition, speech synthesis, face detection, face recognition, facial expression recognition, communication, sense of touch detection, ultrasonic induction etc. These functions can help to achieve a simple natural HRI.
The function of each module is briefly talked about as follow. Speech recognition function module is to achieve the Chinese speech recognition and can get emotional information based on the content of speech emotional recognition. Speech synthesis functional module is to achieve the Chinese speech synthesis and can express emotion of robot through the speech content. Face detection functional module is to get face image and serve for face recognition and facial expression recognition. Face recognition function module is to obtain face region based on face detection and get face information by feature extraction and recognition of this regional image through the template matching algorithm. Facial expression recognition functional module is to use optical flow algorithm to identify facial expressions, and can be used as a key channel for expression interaction. Communication module is to achieve the upper and lower computer communications, and to send control commands for controlling robot actions. Sense of touch module is to achieve tactile interaction by detecting whether there is or not somebody exposure to the site. Ultrasonic sensor module is to identify the obstacles information around robot and to serve for decision-making of robot’s autonomous behavior and movement path. III.
THE PSYCHOLOGICAL STATE SPACE DIVISION BASED ON ARTIFICIAL PSYCHOLOGY
3.1 The theory of artificial psychology Professor Wang put forward a new theory of artificial psychology in 1999. The theory is based on artificial intelligence and deeply analyses human psychology in more comprehensives in the aspects of information scientific research methods, especially in the aspects of emotion, willingness, character, creativity and the realization of artificial machines [3]. It has broad application prospect such as developing the robot with emotion, consciousness and intelligence, research of humanoid machine in real meaning. It can make the control theory more similar to human brain’s control mode. The researches of artificial psychology and affective computing deal with the information related to emotion. They should measure and recognize the affective information, then deal with the affective information and achieve it by computer. In order to do that, understanding the emotion is the most important and is the groundwork for the emotion research. Emotion can affect and adjust the cognitive processes. It can influence the processing and choosing of information when apperceiving and storing information. It help human to choose the information according to environment and harmonize the social communications. 3.2 The psychological state space simplified
Fig.1. The schematic diagram of multimodal interactive platform of service robot.
The important application target of artificial psychology is humanized computer and intelligent service robots. To establish a more reliable psychological model for service robots, the first thing should be done is to understand the characteristics of human psychological reaction. Human psychological reaction has the following characteristics. Emotional response decays with the increase of
The International Journal of Virtual Reality, 2008, 7(2):33-40 time, unless there is new incentive. Rapid and repeat incentive to a psychological state would lead to the increase in perception intensity of that stimulate. However, when more than certain strength, the effect of the stimulus signal will be weaker and weaker. Temperament and personality of people can influence the emotional reaction to the same incentives. The condition of psychological reaction coming into being is the intensity of outside incentive must be a certain value or more than a certain threshold value. Emotional reaction itself has the lower bound and upper bound that is independent of intensity and frequency of the outside incentive. Emotional responses can be activated by physical or cognitive process. According to the above-mentioned characteristics of emotional reaction, when a certain specific condition exceeds a certain threshold value, human’s psychological reaction will be triggered. For the different genes and the growth environment of each person, each individual has different character. To the same input signal, psychological reaction of each individual is not exactly the same, but within an acceptable scope. Then which states do artificial psychology include? It includes hunger, fear, sadness, fatigue, happy, relaxed, tension, excitement and so on. They all are fuzzy concept and have the complex characteristics in both the physiology and psychology. Therefore, it is extremely difficult to fully develop all the psychological space subset and give a strict definition. Here psychological state space is simplified. It is shown in TABLE1. TABLE 1: DIVISION OF PSYCHOLOGICAL STATE SPACE. The state of artificial psychology Emotion state happy
Other psychological state sadness
loneliness
fatigue
35
psychological states. This psychological state can represent the main features of entire psychological state model at the time, called as main state. The generation and transfer of psychological state are not only related with current interactive information but also influenced by the previous psychological state. 3.3 The factors changing psychological states The psychological state model of service robot exports four states: sadness, happy, loneliness and fatigue. Their intensity is represented by the numerical between 0 and 1. And the change of the psychological state is related to not only the current psychological state but also the various stimuli. The stimuli include negative stimulus, positive stimulus, friendship amount, and energy consumption value. These stimuli are from the processed information of various sensors. TABLE 2: THE POSITIVE AND NEGATIVE STIMULUS IN SPEECH INTERACTION. positive stimulus
negative stimulus
ID2： You are a petty trick.
1
0
ID3：You are beautiful.
2
0
ID5：Why can’t you do so simple thing?
0
1
ID6：You should go to the factory for reinventing.
0
3
ID7：How low your IQ is!
0
3
ID8：You are the most intelligent service robot in the world.
3
0
ID9：You are the most beautiful service robot in the world.
3
0
ID10：Whatever you done is worse.
0
2
the content of speech interaction
TABLE 3: THE MEASUREMENT OF FRIENDSHIP AMOUNT.
Fig.2. Fuzzy classification of psychological state.
The paper takes these psychological states as a fuzzy concept. Here each of these psychological states is divided into five levels in accordance with the intensity degree, just shown in Fig.2. Thus, the paper formed a four-dimensional space of psychological states. The origin is the starting point of each psychological state. And four coordinate axes express four state axes. Then the psychological states of service robot at t = i is: si = (a, b, c, d); Where a∈[0,1], b∈[0,1], c∈[0,1], d∈[0,1]. The basic characteristics of state model of service robot are as follow. Each psychological state uses numerical value to express its intensity. At a time, the intensity of one psychological state may be greater than that of all other
The number of continuous interaction time
friendship amount
0-10
1
10-20
2
20-
3
1) Negative Stimulus and Positive Stimulus: Dialogue and expression recognition channel of service robot include the emotional information. For example, the emotion in stimulation is different when you say that you are foolish or you are very clever. The negative stimulus is the stimulus can induce to negative emotion such as criticism and derogation. The positive stimulus is the stimulus can induce to positive emotion such as praise and compliment. That is, when service robot receives an expression of smile, it obtains a positive stimulus. When service robot receives an expression of pain or sadness, it obtains a negative stimulus. Here, each of the negative stimulus and positive stimulus is divided into three grades. Parts of the positive and negative stimulus in speech interaction are shown as TABLE 2. 2) Friendship Amount: When the service robot interacts with
The International Journal of Virtual Reality, 2008, 7(2):33-40
36
human, the friendship amount will increase with the time of IV. MULTIMODAL INTERACTION OF SERVICE ROBOT interaction. This process can be expressed by a numerical value. That is to describe the friendship amount by the number of 4.1 Interactive action achieved continuous interaction time. The number of continuous The service robot is mainly for multi-mode interaction with interaction time is the adding of the interaction time. When the people. Its specific function is to identify language and facial interruption of interactive action with service robot is above expressions of human, to express its current feelings through two minutes, the number of continuous interaction time should speech, expressions and posture such as bow, shaking his head, be added from 1. The friendship amount is divided into three gesture and so on, and to achieve its own psychological grades. It is shown as TABLE 3. self-help through active interaction with the outside world to 3) Energy Consumption Value: When interacting with service change its own psychological state by minimizing the negative robot, people can control the service robot to perform through psychological state value. According to the principle of self-help, service robot should commands. In another word, people can ask service robot to suppress bad psychological state in the process of interaction swing arm, to go forward, to wave and so on by speech. The with the people to the best of its ability. Therefore, when one of energy consumption value of service robot done the action is the psychological state values of loneliness, fatigue, and different. It is shown as TABLE 4. The implementation of sadness has a highest value, the service robot is not fit to actions will affect the fatigue psychological state of service continue interacting with the people and has to stop interacting robot. Therefore, when controlling service robot to do the movements, the paper takes the energy consumption value as for entering dormant. The happy state of the service robot is the an input to influence the fatigue psychological state of service suitable state and should be develop. In order to prolong survival time, the service robot must robot. make judgments and decisions and select appropriate behavior The relationship between Input of stimuli and output of the according to current interaction situation and their own psychological state are as follow. When friendship amount psychological state. There are three kinds of alternative acts for decreases, service robot feels lonely. When energy service robot. There are to initiatively search people for consumption value increases, service robot feels tired. When interaction, to interrupt the current interactive for resting one there are more negative stimuli in the interactive information, minute, and to find blue area for remain in. Blue area can service robot feels sad. When there are more positive stimuli in stimulate the excitement of the psychological state of service the interactive information, service robot feels happy. Service robot and reduce the psychological state value of sadness. robot system can constantly apperceive the change of outside There are three correspondence principles between three acts world and their own state, and analyze the results of perception and the psychological states. The first one is that initiative to obtain four stimuli. After that, the robot uses the stimuli as communicate with the people will reduce the extent of the input of algorithms to make decision for service robot loneliness. The second one is that having a rest for one minute action. TABLE 4: THE RELATIONSHIP BETWEEN ACTIONS AND THE ATTRIBUTES. attribute Face Action
Head action
upper limb action
Lower limb action
sadness
calmness
happy
energy consumption value
curl one’s lip
1
0
0
0
1
1
0
1
frown
1
0
0
0
1
0
0
1
smile
1
0
0
0
0
1
1
1
laugh
1
0
0
0
0
0
1
2
Raise one’s head
0
1
0
0
1
1
1
2
Lower one’s head
0
1
0
0
1
1
0
2
Wag one’s head
0
1
0
0
0
1
1
2
Hold heart up with both hands
0
0
1
0
1
1
0
3
Wipe tear
0
0
1
0
1
0
0
3
Bow forward or face upward
0
0
1
0
0
0
1
3
clap
0
0
1
0
0
1
1
3
countermarch
0
0
0
1
1
1
1
3
Turn left
0
0
0
1
1
1
1
3
Turn right
0
0
0
1
1
1
1
3
Go ahead
0
0
0
1
1
1
1
3
action
Position of executing action
emotion
The International Journal of Virtual Reality, 2008, 7(2):33-40 will reduce the extent of the fatigue. The third one is that remain in blue area will reduce the extent of the sadness. If current interactive information stimulates excitement extent of service robot, service robot should maintain this interaction till interactive information of outside world changing the psychological state of service robot. The interaction purpose of service robot is to survive as long as possible. Service robots can use rich body posture to increase vitality and to enhance interaction. Especially when there is a certain distance between the observer and interactive service robots, limb posture and movements is a very effective means to express inherent emotion of service robot. The actions of service robot include conscious acts and unconscious acts. Conscious acts are the performance of simple movements such as the completion of behaviors ordered of the salute, arms-dance, and grabbing objects in specific location etc. Unconscious acts are the action in the course of human robot interaction such as the correspondence of mouse and action in the process of speech, slight arm movements, and blinking eyes. The paper takes the implementation part of acts, emotional factors, and the energy consumption as the attributes of actions. The relationship between actions and the attributes is shown as TABLE 4. The attribute of implementation part of acts is basis for combination of act, that is, only the combination of different implementation part of action can be output. Emotional factors of actions can be basis for selecting output acts according to the current psychological states of service robot. Energy consumption of action attributes is used to simulate the consumption of energy on the action implementation of robot. It also can affect the psychological state of fatigue of the service robot. Through the classification of external action attributes, the psychological states of service robot are related with external action attributes. After identifying the current psychological state of the service robot, the movement corresponding to the current psychological state can be obtained. Then the output actions can be exported through decision-making according to interactive content. Parts of the relationship between postures and the psychological states are shown as TABLE 5. TABLE 5: THE RELATIONSHIP BETWEEN POSTURES AND THE PSYCHOLOGICAL STATES. psychological state
the description of postures
fatigue
Hands droop; upper body slightly bends forward; head downward
loneliness
two hands raise to chest; two elbows close to body; looked around
happy
Applause or hands above its head; waist straighten; face upward
sadness
beat chest; lower its head; bow; hands down on both sides of the body
37
the following three main functions. The first one is interaction information decomposition. Intelligent decision-making module receives the different interaction information coming from various interactive channels. Different information includes different attributes. The module of interaction information decomposition is to extract the various attribute value from interaction information for input of corresponding process. The second is intelligent decision-making on behavior. It takes the various input information, current psychological state and interactive tasks as the total input. And then, it makes decision about robot’s current interaction behaviors by intelligent behavior decision-making algorithm based on decision-making rules, action databases and knowledge databases. Finally the interaction behaviors will be exported to the compounding module for various acts combination. The third one is psychological state transfer. According to theories of cognitive psychology, the stimulation intensity sensed is different because of different psychology state to different individuals or the same individual in the same external environment. Then the psychological state transfer triggered is different. Therefore, in consideration of the psychological state transfer, it will be affected by not only the input of stimuli from interaction environment but also the immediately previous psychological states.
Fig3. The principle picture of intelligent decision-making.
4.2 Intelligent decision-making of interactive behaviors Intelligent decision-making of service robot is the core of achieving human robot interaction. The principle picture is shown as Fig. 3. Intelligent decision-making module completed
Fig.4. The initiative actions decision-making of service robot in loneliness initial state.
38
The International Journal of Virtual Reality, 2008, 7(2):33-40 and gave a continuing negative stimulation to it in the process of interaction. This constantly increased the value of the negative psychological state of the robot. Then the behavior of decision-making and the effect to psychology states of service robot induced by interaction were observed. Fig. 4 and Fig. 5 are the interaction actions and the curve of psychological state change in the loneliness state. Fig. 6 and Fig. 7 are the interaction actions and the curve of psychological state change in the fatigue state. Fig. 8 and Fig. 9 are the interaction actions and the curve of psychological state change with the effect of the positive stimuli.
Fig.5. The curve of loneliness psychology state change.
Fig.8. The initiative actions decision-making of service robot with the effect of posotive stimuli. See Color Plate 14. Fig.6. The initiative actions decision-making of service robot in fatigue initial state. See Color Plate 13.
Fig.9. The curve of happy psychology state change. Fig.7. The curve of fatigue psychology state change.
4.3 Simulation results of psychology state change of service robot The paper set an initial psychology state for service robot
From the above process and results of the experiment, some conclusions could be got. In the loneliness psychology state, service robots would initiatively search people and interact with them to increase the value of friendship amount. Such could reduce its own psychology state of loneliness. From the
The International Journal of Virtual Reality, 2008, 7(2):33-40 curve of loneliness psychology state change, it could be seen that the successful behavior decision-making of service robot made its loneliness psychology state value dropped to a very small range. When the value of fatigue psychology state was greater than 0.6, the service robot had not fit to continue to interact. It would start the autonomic behaviors decision-making to adjust the psychological state of its own, stopped interacting, and kept away from people for rest. In this condition the service robot filtered the interaction information of outside world and discarded lower priority of interaction information not to respond. After the rest sustained for a period of time, the value of fatigue psychology state would fell to 0.6. Then service robot entered normal mode and could receive various interaction information. These can be seen from the curve of fatigue psychology state change. When continued to give the service robot positive interaction information, it would be in a happy psychology state. Service robot would continue to keep this interaction and maintain this happy psychology state. The curve was shown in Fig. 9. V.
CONCLUSION
In this paper, the main goal is to establish a service robot with the ability of autonomic interaction and the artificial psychology, and to carry out the research of multi-information processing and artificial psychology in this platform. Based on characteristics of human interaction, the paper designed intelligent decision-making system of multimodal information fusion. It includes various interactive modes such as speech, vision and feeling etc. this is the basic platform for further study of intelligent control algorithm. The paper carried out the autonomic interaction behaviors research of service robot based on artificial psychology in this platform. First the paper gave the simple division of service robot’s artificial psychology state. Therefore the robot had a limited number of psychological states. Each state corresponded to different interaction tasks. And the interaction information was restricted and classified. After the analysis and disposal to the interaction information, it would be the input of decision-making algorithm and affect the psychology state of service robot that could make the state transferring. According to the current psychological state, the behavior pattern of robot was identified. Then autonomic behavioral control of the robot was achieved. During the research of psychology state of robot, the paper only gives the rough division of the psychology state space. Its advantage is that reduce the complexity of the algorithm. And its shortage is that restrict the robot’s behavior representation. Further research in this area is to give the division of psychological state space in more detail and to make the behavior representation of robots more intelligent. The extraction of psychological factors included in interaction information also needs more in-depth study. At present the methods used of questionnaire and subjective judgment are not very objective. Further research and more reasonable classification methodology in this area are needed. The service robot with the ability of autonomic interaction is the important research object in artificial intelligence and artificial psychology. It also is efficient integrated experiment and demonstration platform to the research and development of
39
artificial intelligence and artificial psychology. There are still many issues to be further explored in theory, technology and the achievement of such areas. ACKNOWLEDGEMENT The paper is supported by National Natural Science Foundation of China (NO. 60573059), 863 Program (2007AA04Z218) and key programs of Natural Science Foundation of Beijing (KZ200810028016). REFERENCES [1] [2] [3] [4]
[5] [6] [7] [8]
Zeungnam Bien. Blend of Soft Computing Techniques for Effective Human-Machine Interaction in Service Robot Systems, Fuzzy Sets and Systems, vol. 134, pp. 5-25, 2003. Masahiro Fujita. Digital Creatures for Future Entertainment Robotics, proceedings Of The 2000 IEEE international Conference On Robotics & Automation, San Francisco, CA, pp. 801-806, April 2000. Zhiliang Wang. Artificial Psychology-a Most Accessible Science Research to Human Brain, Journal of University of Science and Technology Beijing, vol. 22, no. 5, pp. 478-481, 2000. Norling E. Folk Psychology for Human Modelling: Extending the BDI Paradigm[C], proceedings of the International Joint Conference on Autonomous, Agents and Multi Agent Systems (AAMAS), New York, pp. 202-209, 2004. He You et al., Survey of Multisensor Data Fusion Models[J], Journal of Tsinghua University, vol. 36, no. 9, pp. 14-19, 1996. Wang Xiaogang at al., the Two Structures and Channels of Robot Multi-sensor Data Fusion, Journal of Applied Science, vol. 13, no. 4, pp. 439-446, 1995. Kuniya Shinozaki, Akitsugu Iwatani and Ryohei Nakatsu. Concept and Construction of a Robot Dance System, The international Journal of Virtual Reality, vol. 6, no. 3, pp. 29-34, September 2007. Ayanna M. Howard and Sekou Remy. Utilizing Virtual Environments to Enable Learning in Human-robot Interaction Scenarios, The international Journal of Virtual Reality, vol. 7, no. 1, pp. 9-14, March 2008. Jun Yu received her M. Eng degree in School of Information Science and Technology from Naval Aeronautical Engineering Institute, Yantai, China, in 2005. She is a D. student of University of Science & Technology Beijing, Beijing, China. Her research interests include intelligent robot, affective computing, digital signal processing etc.
Zhiliang Wang received his Ph. Eng degree in electric engineering from Harbin Institute of Technology, Harbin, China, in 1988. He worked at electrical postdoctoral research station in Zhejiang University. He is currently professor, School of Information Science and Technology, University of Science & Technology Beijing. He is doctorial advisor. He is Director of Chinese Artificial Intelligence Society and Director of Artificial Psychology and Artificial Emotion Association. Professor Wang research interests include intelligent robot, affective computing, digital signal processing etc.
40
The International Journal of Virtual Reality, 2008, 7(2):33-40 Lun Xie received his Ph. Eng degree in School of Information Science and Technology from University of Science & Technology Beijing, Beijing, China, in 2003. He is currently Assistant Professor, School of Information Science and Technology, University of Science & Technology Beijing. His research interests include intelligent robot, affective computing, digital signal processing etc.
Yongxiang Xia received his M. Eng degree in School of Information Science and Technology from Beijing Institute of Technology, Beijing, China, in 2005. His research interest is intelligent control.
33
Research on Artificial Psychology Based on Multimodal Interactive Service Robot Jun Yu 1, 2, Zhiliang Wang 1, Lun Xie 1 and Yongxiang Xia 2 1 2
School of Information Engineering, University of Science & Technology Beijing, Beijing, China. 1 Navy Flight Academy, Hulu Dao, Liaoning Province, China.
Abstract—In the field of human robot interaction (HRI), providing robot with emotions and psychology like human can be useful to achieve natural interaction. Previous HRI research focused on issues such as gesture recognition, speech recognition and path planning etc. Using the affective information obtained from multimodal interaction can improve the robot’s perception to interaction object and environment, and then can improve the harmoniousness of HRI. The paper puts forward a method of decision-making on multimodal interaction information fusion of service robot based on artificial psychology. The experimental results demonstrate that the proposed method can be efficient and effective in HRI. Index Terms—Artificial psychology, behavior decision-making, multimodal interaction information fusion, service robot.
I.
INTRODUCTION
Robotics research is rising from 1980’s in various applications to serve human. It is an important milestone of multidisciplinary common progress such as manufacturing, information technology and cognitive psychology etc. In these days, it has shown a personal computer to personal service robot development trends in service industry. People hope the service robot could be the indispensable partner of human. For service robot to act according to its own states and environment can be useful to interaction. In the field of robot, it is the focus of researchers' attention to eliminate the obstacles between human and robot and endow robot with the abilities of apperceiving the environment and the emotion of human who it communicates with. It is one of the important research that how to apperceive and understand the interaction intention of human according to multimodal interaction information obtained by sense organs. Generally speaking, the research of the harmonious coexistence of service robots and human focuses on the method and implementation of analyzing and comprehending to multimodal information during interaction [1]. Because of the various contents of interaction and different application, it is very difficult to investigate HRI satisfying to any background. Therefore, the research usually focuses on the special application and utilizes the professional knowledge to analyze, comprehend and achieve special HRI [2]. The multimodal information fusion research is not limited to Manuscript received on April 15th, 2008. E-Mail: [email protected].
vision information and speech information. It includes facial expression, body posture, and speech information etc. based on digital feature expression. Multimodal information can remedy defects just using one kind of information. That is beneficial to the right decision making and natural HRI. II.
THE MULTIMODAL INTERACTION PLATFORM OF SERVICE ROBOT
2.1 Multi-sensor information fusion The research of service robot must begin from the imitating human. By observed human’s actions, it can be found that human received the environment information from the sense organs such as vision, hearing, nose, feeling etc. The information transmitted to the brain through nerves. Brain processes the information, integrates the information, and sends the action instructions to body to perform certain actions. When designing the service robot, researchers can take computer as brain, take executive mechanism as body, and take all kinds of sensors as human’s organs. In order to enable service robot having intelligence and responding to environment change, first, the robot should sense the environment. Then, the robot can deal with and integrate the information sensed by sensors. In the early research about this, people focus on the research and development of all kinds of sensors and just used one sense for robot in an application. Little thing was done on taking all senses as whole to analyze. Because different sensor apperceives different (or similar) side of information in the same environment, the information is related. Therefore, multimodal information fusion is a new research field to deal with the information more effective. Generally, there are many advantages to utilize the multi-senses system and information fusion. The first is fault-tolerant features. When a sensor or even more is in failure, the robot can still work normally with other senses apperceiving environment information. The second is that can improve accuracy. In the information apperceived, inevitably there are all sorts of noise. To use a number of information describing the same characteristics at the same time, uncertainty caused by such imprecise measurements can be reduced. And for complementary characteristic of information, the ambiguities of understanding environment can be reduced and correct decision-making ability of the system can be improved. That is researchers can use a number of less expensive sensors to obtain the same or even better performance that can be obtained by expensive single precision
34
The International Journal of Virtual Reality, 2008, 7(2):33-40
sensor. This can greatly reduce the cost of system. 2.2 Multimodal interactive platform of service robot Multimodal information is the various modes of perceived feedback information, expressing intent of users, or the implementation of action such as language, eye expression, facial expressions, gestures, body posture, touch, taste or smell etc. Through natural, happy and harmonious intercourse paradigm of human, the results of psychology research show that only 11% of the information obtained through the hearing, 83% through vision. Thus, psychologists propose a formula: emotional expression=7% of the words+39% of the voice +54% of the emotional action. These basic models in the perception of digital expression level based on human brain are complementary each other, isolating them will lose integrity of the information expression. According to it, this paper extracts not only meaningful characteristics of a single model but also comprehensive information features of different modes composition. Cooperative work of models can provide the necessary information to complete specific interactive tasks and can achieve the interaction between human and service robot. In the interaction process of human and service robot, it is better to adopt natural interactive mode as possible, such as voice, facial expressions, vision, proximity sensing, touch etc. The paper develops a high efficient and general architecture of service robot. Its main purpose is to construct a humanoid robot to play with children and combine education with recreation. In the affective robot architecture, we can utilize speech technology, image technology, and sensor technology to imitate the affective identification means of human. Therefore more reliable recognition effect can be achieved. Fig.1 is the schematic diagram of multimodal interactive platform of service robot designed. The main functions of it are speech recognition, speech synthesis, face detection, face recognition, facial expression recognition, communication, sense of touch detection, ultrasonic induction etc. These functions can help to achieve a simple natural HRI.
The function of each module is briefly talked about as follow. Speech recognition function module is to achieve the Chinese speech recognition and can get emotional information based on the content of speech emotional recognition. Speech synthesis functional module is to achieve the Chinese speech synthesis and can express emotion of robot through the speech content. Face detection functional module is to get face image and serve for face recognition and facial expression recognition. Face recognition function module is to obtain face region based on face detection and get face information by feature extraction and recognition of this regional image through the template matching algorithm. Facial expression recognition functional module is to use optical flow algorithm to identify facial expressions, and can be used as a key channel for expression interaction. Communication module is to achieve the upper and lower computer communications, and to send control commands for controlling robot actions. Sense of touch module is to achieve tactile interaction by detecting whether there is or not somebody exposure to the site. Ultrasonic sensor module is to identify the obstacles information around robot and to serve for decision-making of robot’s autonomous behavior and movement path. III.
THE PSYCHOLOGICAL STATE SPACE DIVISION BASED ON ARTIFICIAL PSYCHOLOGY
3.1 The theory of artificial psychology Professor Wang put forward a new theory of artificial psychology in 1999. The theory is based on artificial intelligence and deeply analyses human psychology in more comprehensives in the aspects of information scientific research methods, especially in the aspects of emotion, willingness, character, creativity and the realization of artificial machines [3]. It has broad application prospect such as developing the robot with emotion, consciousness and intelligence, research of humanoid machine in real meaning. It can make the control theory more similar to human brain’s control mode. The researches of artificial psychology and affective computing deal with the information related to emotion. They should measure and recognize the affective information, then deal with the affective information and achieve it by computer. In order to do that, understanding the emotion is the most important and is the groundwork for the emotion research. Emotion can affect and adjust the cognitive processes. It can influence the processing and choosing of information when apperceiving and storing information. It help human to choose the information according to environment and harmonize the social communications. 3.2 The psychological state space simplified
Fig.1. The schematic diagram of multimodal interactive platform of service robot.
The important application target of artificial psychology is humanized computer and intelligent service robots. To establish a more reliable psychological model for service robots, the first thing should be done is to understand the characteristics of human psychological reaction. Human psychological reaction has the following characteristics. Emotional response decays with the increase of
The International Journal of Virtual Reality, 2008, 7(2):33-40 time, unless there is new incentive. Rapid and repeat incentive to a psychological state would lead to the increase in perception intensity of that stimulate. However, when more than certain strength, the effect of the stimulus signal will be weaker and weaker. Temperament and personality of people can influence the emotional reaction to the same incentives. The condition of psychological reaction coming into being is the intensity of outside incentive must be a certain value or more than a certain threshold value. Emotional reaction itself has the lower bound and upper bound that is independent of intensity and frequency of the outside incentive. Emotional responses can be activated by physical or cognitive process. According to the above-mentioned characteristics of emotional reaction, when a certain specific condition exceeds a certain threshold value, human’s psychological reaction will be triggered. For the different genes and the growth environment of each person, each individual has different character. To the same input signal, psychological reaction of each individual is not exactly the same, but within an acceptable scope. Then which states do artificial psychology include? It includes hunger, fear, sadness, fatigue, happy, relaxed, tension, excitement and so on. They all are fuzzy concept and have the complex characteristics in both the physiology and psychology. Therefore, it is extremely difficult to fully develop all the psychological space subset and give a strict definition. Here psychological state space is simplified. It is shown in TABLE1. TABLE 1: DIVISION OF PSYCHOLOGICAL STATE SPACE. The state of artificial psychology Emotion state happy
Other psychological state sadness
loneliness
fatigue
35
psychological states. This psychological state can represent the main features of entire psychological state model at the time, called as main state. The generation and transfer of psychological state are not only related with current interactive information but also influenced by the previous psychological state. 3.3 The factors changing psychological states The psychological state model of service robot exports four states: sadness, happy, loneliness and fatigue. Their intensity is represented by the numerical between 0 and 1. And the change of the psychological state is related to not only the current psychological state but also the various stimuli. The stimuli include negative stimulus, positive stimulus, friendship amount, and energy consumption value. These stimuli are from the processed information of various sensors. TABLE 2: THE POSITIVE AND NEGATIVE STIMULUS IN SPEECH INTERACTION. positive stimulus
negative stimulus
ID2： You are a petty trick.
1
0
ID3：You are beautiful.
2
0
ID5：Why can’t you do so simple thing?
0
1
ID6：You should go to the factory for reinventing.
0
3
ID7：How low your IQ is!
0
3
ID8：You are the most intelligent service robot in the world.
3
0
ID9：You are the most beautiful service robot in the world.
3
0
ID10：Whatever you done is worse.
0
2
the content of speech interaction
TABLE 3: THE MEASUREMENT OF FRIENDSHIP AMOUNT.
Fig.2. Fuzzy classification of psychological state.
The paper takes these psychological states as a fuzzy concept. Here each of these psychological states is divided into five levels in accordance with the intensity degree, just shown in Fig.2. Thus, the paper formed a four-dimensional space of psychological states. The origin is the starting point of each psychological state. And four coordinate axes express four state axes. Then the psychological states of service robot at t = i is: si = (a, b, c, d); Where a∈[0,1], b∈[0,1], c∈[0,1], d∈[0,1]. The basic characteristics of state model of service robot are as follow. Each psychological state uses numerical value to express its intensity. At a time, the intensity of one psychological state may be greater than that of all other
The number of continuous interaction time
friendship amount
0-10
1
10-20
2
20-
3
1) Negative Stimulus and Positive Stimulus: Dialogue and expression recognition channel of service robot include the emotional information. For example, the emotion in stimulation is different when you say that you are foolish or you are very clever. The negative stimulus is the stimulus can induce to negative emotion such as criticism and derogation. The positive stimulus is the stimulus can induce to positive emotion such as praise and compliment. That is, when service robot receives an expression of smile, it obtains a positive stimulus. When service robot receives an expression of pain or sadness, it obtains a negative stimulus. Here, each of the negative stimulus and positive stimulus is divided into three grades. Parts of the positive and negative stimulus in speech interaction are shown as TABLE 2. 2) Friendship Amount: When the service robot interacts with
The International Journal of Virtual Reality, 2008, 7(2):33-40
36
human, the friendship amount will increase with the time of IV. MULTIMODAL INTERACTION OF SERVICE ROBOT interaction. This process can be expressed by a numerical value. That is to describe the friendship amount by the number of 4.1 Interactive action achieved continuous interaction time. The number of continuous The service robot is mainly for multi-mode interaction with interaction time is the adding of the interaction time. When the people. Its specific function is to identify language and facial interruption of interactive action with service robot is above expressions of human, to express its current feelings through two minutes, the number of continuous interaction time should speech, expressions and posture such as bow, shaking his head, be added from 1. The friendship amount is divided into three gesture and so on, and to achieve its own psychological grades. It is shown as TABLE 3. self-help through active interaction with the outside world to 3) Energy Consumption Value: When interacting with service change its own psychological state by minimizing the negative robot, people can control the service robot to perform through psychological state value. According to the principle of self-help, service robot should commands. In another word, people can ask service robot to suppress bad psychological state in the process of interaction swing arm, to go forward, to wave and so on by speech. The with the people to the best of its ability. Therefore, when one of energy consumption value of service robot done the action is the psychological state values of loneliness, fatigue, and different. It is shown as TABLE 4. The implementation of sadness has a highest value, the service robot is not fit to actions will affect the fatigue psychological state of service continue interacting with the people and has to stop interacting robot. Therefore, when controlling service robot to do the movements, the paper takes the energy consumption value as for entering dormant. The happy state of the service robot is the an input to influence the fatigue psychological state of service suitable state and should be develop. In order to prolong survival time, the service robot must robot. make judgments and decisions and select appropriate behavior The relationship between Input of stimuli and output of the according to current interaction situation and their own psychological state are as follow. When friendship amount psychological state. There are three kinds of alternative acts for decreases, service robot feels lonely. When energy service robot. There are to initiatively search people for consumption value increases, service robot feels tired. When interaction, to interrupt the current interactive for resting one there are more negative stimuli in the interactive information, minute, and to find blue area for remain in. Blue area can service robot feels sad. When there are more positive stimuli in stimulate the excitement of the psychological state of service the interactive information, service robot feels happy. Service robot and reduce the psychological state value of sadness. robot system can constantly apperceive the change of outside There are three correspondence principles between three acts world and their own state, and analyze the results of perception and the psychological states. The first one is that initiative to obtain four stimuli. After that, the robot uses the stimuli as communicate with the people will reduce the extent of the input of algorithms to make decision for service robot loneliness. The second one is that having a rest for one minute action. TABLE 4: THE RELATIONSHIP BETWEEN ACTIONS AND THE ATTRIBUTES. attribute Face Action
Head action
upper limb action
Lower limb action
sadness
calmness
happy
energy consumption value
curl one’s lip
1
0
0
0
1
1
0
1
frown
1
0
0
0
1
0
0
1
smile
1
0
0
0
0
1
1
1
laugh
1
0
0
0
0
0
1
2
Raise one’s head
0
1
0
0
1
1
1
2
Lower one’s head
0
1
0
0
1
1
0
2
Wag one’s head
0
1
0
0
0
1
1
2
Hold heart up with both hands
0
0
1
0
1
1
0
3
Wipe tear
0
0
1
0
1
0
0
3
Bow forward or face upward
0
0
1
0
0
0
1
3
clap
0
0
1
0
0
1
1
3
countermarch
0
0
0
1
1
1
1
3
Turn left
0
0
0
1
1
1
1
3
Turn right
0
0
0
1
1
1
1
3
Go ahead
0
0
0
1
1
1
1
3
action
Position of executing action
emotion
The International Journal of Virtual Reality, 2008, 7(2):33-40 will reduce the extent of the fatigue. The third one is that remain in blue area will reduce the extent of the sadness. If current interactive information stimulates excitement extent of service robot, service robot should maintain this interaction till interactive information of outside world changing the psychological state of service robot. The interaction purpose of service robot is to survive as long as possible. Service robots can use rich body posture to increase vitality and to enhance interaction. Especially when there is a certain distance between the observer and interactive service robots, limb posture and movements is a very effective means to express inherent emotion of service robot. The actions of service robot include conscious acts and unconscious acts. Conscious acts are the performance of simple movements such as the completion of behaviors ordered of the salute, arms-dance, and grabbing objects in specific location etc. Unconscious acts are the action in the course of human robot interaction such as the correspondence of mouse and action in the process of speech, slight arm movements, and blinking eyes. The paper takes the implementation part of acts, emotional factors, and the energy consumption as the attributes of actions. The relationship between actions and the attributes is shown as TABLE 4. The attribute of implementation part of acts is basis for combination of act, that is, only the combination of different implementation part of action can be output. Emotional factors of actions can be basis for selecting output acts according to the current psychological states of service robot. Energy consumption of action attributes is used to simulate the consumption of energy on the action implementation of robot. It also can affect the psychological state of fatigue of the service robot. Through the classification of external action attributes, the psychological states of service robot are related with external action attributes. After identifying the current psychological state of the service robot, the movement corresponding to the current psychological state can be obtained. Then the output actions can be exported through decision-making according to interactive content. Parts of the relationship between postures and the psychological states are shown as TABLE 5. TABLE 5: THE RELATIONSHIP BETWEEN POSTURES AND THE PSYCHOLOGICAL STATES. psychological state
the description of postures
fatigue
Hands droop; upper body slightly bends forward; head downward
loneliness
two hands raise to chest; two elbows close to body; looked around
happy
Applause or hands above its head; waist straighten; face upward
sadness
beat chest; lower its head; bow; hands down on both sides of the body
37
the following three main functions. The first one is interaction information decomposition. Intelligent decision-making module receives the different interaction information coming from various interactive channels. Different information includes different attributes. The module of interaction information decomposition is to extract the various attribute value from interaction information for input of corresponding process. The second is intelligent decision-making on behavior. It takes the various input information, current psychological state and interactive tasks as the total input. And then, it makes decision about robot’s current interaction behaviors by intelligent behavior decision-making algorithm based on decision-making rules, action databases and knowledge databases. Finally the interaction behaviors will be exported to the compounding module for various acts combination. The third one is psychological state transfer. According to theories of cognitive psychology, the stimulation intensity sensed is different because of different psychology state to different individuals or the same individual in the same external environment. Then the psychological state transfer triggered is different. Therefore, in consideration of the psychological state transfer, it will be affected by not only the input of stimuli from interaction environment but also the immediately previous psychological states.
Fig3. The principle picture of intelligent decision-making.
4.2 Intelligent decision-making of interactive behaviors Intelligent decision-making of service robot is the core of achieving human robot interaction. The principle picture is shown as Fig. 3. Intelligent decision-making module completed
Fig.4. The initiative actions decision-making of service robot in loneliness initial state.
38
The International Journal of Virtual Reality, 2008, 7(2):33-40 and gave a continuing negative stimulation to it in the process of interaction. This constantly increased the value of the negative psychological state of the robot. Then the behavior of decision-making and the effect to psychology states of service robot induced by interaction were observed. Fig. 4 and Fig. 5 are the interaction actions and the curve of psychological state change in the loneliness state. Fig. 6 and Fig. 7 are the interaction actions and the curve of psychological state change in the fatigue state. Fig. 8 and Fig. 9 are the interaction actions and the curve of psychological state change with the effect of the positive stimuli.
Fig.5. The curve of loneliness psychology state change.
Fig.8. The initiative actions decision-making of service robot with the effect of posotive stimuli. See Color Plate 14. Fig.6. The initiative actions decision-making of service robot in fatigue initial state. See Color Plate 13.
Fig.9. The curve of happy psychology state change. Fig.7. The curve of fatigue psychology state change.
4.3 Simulation results of psychology state change of service robot The paper set an initial psychology state for service robot
From the above process and results of the experiment, some conclusions could be got. In the loneliness psychology state, service robots would initiatively search people and interact with them to increase the value of friendship amount. Such could reduce its own psychology state of loneliness. From the
The International Journal of Virtual Reality, 2008, 7(2):33-40 curve of loneliness psychology state change, it could be seen that the successful behavior decision-making of service robot made its loneliness psychology state value dropped to a very small range. When the value of fatigue psychology state was greater than 0.6, the service robot had not fit to continue to interact. It would start the autonomic behaviors decision-making to adjust the psychological state of its own, stopped interacting, and kept away from people for rest. In this condition the service robot filtered the interaction information of outside world and discarded lower priority of interaction information not to respond. After the rest sustained for a period of time, the value of fatigue psychology state would fell to 0.6. Then service robot entered normal mode and could receive various interaction information. These can be seen from the curve of fatigue psychology state change. When continued to give the service robot positive interaction information, it would be in a happy psychology state. Service robot would continue to keep this interaction and maintain this happy psychology state. The curve was shown in Fig. 9. V.
CONCLUSION
In this paper, the main goal is to establish a service robot with the ability of autonomic interaction and the artificial psychology, and to carry out the research of multi-information processing and artificial psychology in this platform. Based on characteristics of human interaction, the paper designed intelligent decision-making system of multimodal information fusion. It includes various interactive modes such as speech, vision and feeling etc. this is the basic platform for further study of intelligent control algorithm. The paper carried out the autonomic interaction behaviors research of service robot based on artificial psychology in this platform. First the paper gave the simple division of service robot’s artificial psychology state. Therefore the robot had a limited number of psychological states. Each state corresponded to different interaction tasks. And the interaction information was restricted and classified. After the analysis and disposal to the interaction information, it would be the input of decision-making algorithm and affect the psychology state of service robot that could make the state transferring. According to the current psychological state, the behavior pattern of robot was identified. Then autonomic behavioral control of the robot was achieved. During the research of psychology state of robot, the paper only gives the rough division of the psychology state space. Its advantage is that reduce the complexity of the algorithm. And its shortage is that restrict the robot’s behavior representation. Further research in this area is to give the division of psychological state space in more detail and to make the behavior representation of robots more intelligent. The extraction of psychological factors included in interaction information also needs more in-depth study. At present the methods used of questionnaire and subjective judgment are not very objective. Further research and more reasonable classification methodology in this area are needed. The service robot with the ability of autonomic interaction is the important research object in artificial intelligence and artificial psychology. It also is efficient integrated experiment and demonstration platform to the research and development of
39
artificial intelligence and artificial psychology. There are still many issues to be further explored in theory, technology and the achievement of such areas. ACKNOWLEDGEMENT The paper is supported by National Natural Science Foundation of China (NO. 60573059), 863 Program (2007AA04Z218) and key programs of Natural Science Foundation of Beijing (KZ200810028016). REFERENCES [1] [2] [3] [4]
[5] [6] [7] [8]
Zeungnam Bien. Blend of Soft Computing Techniques for Effective Human-Machine Interaction in Service Robot Systems, Fuzzy Sets and Systems, vol. 134, pp. 5-25, 2003. Masahiro Fujita. Digital Creatures for Future Entertainment Robotics, proceedings Of The 2000 IEEE international Conference On Robotics & Automation, San Francisco, CA, pp. 801-806, April 2000. Zhiliang Wang. Artificial Psychology-a Most Accessible Science Research to Human Brain, Journal of University of Science and Technology Beijing, vol. 22, no. 5, pp. 478-481, 2000. Norling E. Folk Psychology for Human Modelling: Extending the BDI Paradigm[C], proceedings of the International Joint Conference on Autonomous, Agents and Multi Agent Systems (AAMAS), New York, pp. 202-209, 2004. He You et al., Survey of Multisensor Data Fusion Models[J], Journal of Tsinghua University, vol. 36, no. 9, pp. 14-19, 1996. Wang Xiaogang at al., the Two Structures and Channels of Robot Multi-sensor Data Fusion, Journal of Applied Science, vol. 13, no. 4, pp. 439-446, 1995. Kuniya Shinozaki, Akitsugu Iwatani and Ryohei Nakatsu. Concept and Construction of a Robot Dance System, The international Journal of Virtual Reality, vol. 6, no. 3, pp. 29-34, September 2007. Ayanna M. Howard and Sekou Remy. Utilizing Virtual Environments to Enable Learning in Human-robot Interaction Scenarios, The international Journal of Virtual Reality, vol. 7, no. 1, pp. 9-14, March 2008. Jun Yu received her M. Eng degree in School of Information Science and Technology from Naval Aeronautical Engineering Institute, Yantai, China, in 2005. She is a D. student of University of Science & Technology Beijing, Beijing, China. Her research interests include intelligent robot, affective computing, digital signal processing etc.
Zhiliang Wang received his Ph. Eng degree in electric engineering from Harbin Institute of Technology, Harbin, China, in 1988. He worked at electrical postdoctoral research station in Zhejiang University. He is currently professor, School of Information Science and Technology, University of Science & Technology Beijing. He is doctorial advisor. He is Director of Chinese Artificial Intelligence Society and Director of Artificial Psychology and Artificial Emotion Association. Professor Wang research interests include intelligent robot, affective computing, digital signal processing etc.
40
The International Journal of Virtual Reality, 2008, 7(2):33-40 Lun Xie received his Ph. Eng degree in School of Information Science and Technology from University of Science & Technology Beijing, Beijing, China, in 2003. He is currently Assistant Professor, School of Information Science and Technology, University of Science & Technology Beijing. His research interests include intelligent robot, affective computing, digital signal processing etc.
Yongxiang Xia received his M. Eng degree in School of Information Science and Technology from Beijing Institute of Technology, Beijing, China, in 2005. His research interest is intelligent control.
