A Personalized Courseware Recommendation ... - Semantic Scholar
A Personalized Courseware Recommendation System Based on Fuzzy Item Response Theory Chih-Ming Chen1, Ling-Jiun Duh2 and Chao-Yu Liu1 Graduate Institute of Learning Technology National Hualien Teachers College, Hualien, Taiwan1 Department of Information and Computer Education National Taiwan Normal University, Taipei, Taiwan2 123 Hua-His Rd., Hualien, Taiwan 970, R.O.C. Tel: +886-3-8227106 Ext. 1519 Fax: +886-3-8234225 E-mail: [email protected] require [3-5]. To help Internet users to search more efficiently, many powerful search tools [6-9] have been proposed, such as the Google search engine [10] (Google), or the Citeseer website [11] (NEC Research Institute ResearchIndex). Most of these search tools provide personalized mechanisms to enable users to filter out uninteresting or irrelevant search results. Similar to online searching, Web-based learning also needs personalized mechanisms to help learners learn more efficiently. Therefore, many researchers recently have endeavored to provide personalization mechanisms for Web-based learning [12-21]. Restated, personalized service has received considerable attention [22] recently because of information needs different among users. Nowadays, most recommendation systems [23-25] consider learner/user preferences, interests, and browsing behaviors when analyzing learner behaviors for personalized services. These systems neglect the importance of learner ability for implementing personalized mechanisms. On the other hand, some researchers emphasized that personalization should consider different levels of learner knowledge, especially in relation to learning [15-21][26]. That is, the ability of individuals may be based on major fields and subjects. Therefore, considering learner ability can promote personalized learning performance. Item response theory [27-30] is a popular theory in education measurement. Item response theory usually is applied in the computerized adaptive test (CAT) [31-32] to select the most appropriate items for examinees based on individual ability. The computerized adaptive test not only can efficiently shorten the testing time and the number of testing items but also can precisely estimate examinee ability. Presently, the concept of CAT is applied to replace traditional measurement instruments (which are typically fixed-length, fixed-content and paper-pencil tests) in several real-world applications, such as TOEFL [33], GRE [34], and GMAT [35].
Abstract With the rapid growth of computer and Internet technologies, e-learning has become a major trend in the computer assisted teaching and learning field currently. In past years, many researchers made effort in developing elearning systems with personalized learning mechanism to assist on-line learning. However, most of them focused on using learner’s behaviors, interests, and habits to provide personalized e-learning services. These systems usually neglected to concern if learner’s ability and the difficulty of courseware are matched each other. Generally, recommending an inappropriate courseware might result in learner’s cognitive overhead or disorientation during a learning process. To promote learning efficiency and effectiveness, this paper presents a personalized courseware recommendation system (PCRS) based on the proposed fuzzy item response theory (FIRT), which can recommend courseware with appropriate difficult level to learner through learner gives a fuzzy response of understanding percentage for the learned courseware. Therefore, this paper will deal with how to modify the traditional item response theory (IRT) to handle non-crisp fuzzy response and estimate learner’s ability via the revised estimating function of learner’s ability. Based on the courseware’s modeling requirement, this paper also proposes a courseware modeling process to determine the difficulty parameters of courseware and design the content of courseware for personalized courseware recommendation services. Experiment results show that applying the proposed fuzzy item response theory to Webbased learning can achieve personalized learning and help learners to learn more effectively and efficiently.
1. Introduction Recently years, numerous Web applications have been developed, the rapid growth of information on the Web has created a problem of information overload [1-2], such that Internet users are unable to find the information they
Based on previous analyses, this study proposes a
1
personalized courseware recommendation system (PCRS) based on the proposed fuzzy item response theory to provide Web-based personalized e-learning services. In the proposed fuzzy item response theory, fuzzy theory [36] is combined with the original item response theory to model uncertainly learning response. Moreover, the single parameter logistic model with difficulty parameter proposed by Georg Rasch in 1966[27-30] is applied to model various difficulty levels of courseware. PCRS can dynamically estimate learner ability based on the proposed fuzzy item response theory by collecting learner feedback after studying the recommended courseware. Based on the estimation of learner abilities, the novel system can recommend appropriate courseware to learners. Restated, learner ability and the difficulties of course materials are simultaneously taken into account when implementing a personalization mechanism. Moreover, based on the courseware’s modeling requirement, this paper also proposes a courseware modeling process to determine the difficulty parameters of courseware and construct the content of courseware for personalized courseware recommendation services. Experiments show that the proposed personalized courseware recommendation system can recommend appropriate course materials to learners based on individual ability, and help them to learn more efficiently and effectively.
database, courseware database, and teacher account database. The learner interface agent aims at providing a friendly learning interface for learners to interact with the feedback agent and courseware recommendation agent. On the other hand, the feedback agent aims at collecting learner explicit feedback information from the learning interface agent and storing them into the user profile database for personalized e-learning services. Moreover, the courseware recommendation agent is in charge of recommending adaptive courseware to learner according to learner’s feedback response and his ability. Finally, the courseware management agent with authorized account management mechanism provides a friendly courseware management interface, which can help teachers to create new course units, upload courseware to the courseware database, delete and modify courseware from the courseware database etc. The system architecture is shown as Figure 1. Furthermore, we also propose a courseware modeling process derived from the computer adaptive testing (CAT) theory [31-32] to assign courseware with appropriate difficulty parameters for the personalized courseware recommendation. The proposed courseware modeling process successfully transfers the concepts of testing items into courseware to provide courseware resources with the corresponding difficulty parameter for personalized e-learning services. The detailed operation procedure of system architecture is described as the following section.
2. System Architecture Most learning recommendation systems mainly focus on analyzing learner’s behavior and interests to provide personalized e-learning services. However, both learner’s ability and courseware’s difficulty are usually ignored as important factors while personalized e-learning services are performed. Generally, the courseware recommended by the learning recommendation systems might be too hard or too easy to learners if these two factors have not been further considered for courseware recommendation. Namely, learners usually need to accept some inappropriate recommendation courseware even they cannot realize the content of courseware completely. In order to provide the courseware with appropriate difficult level to learners, this study proposes a personalized courseware recommendation system (PCRS) based on the fuzzy item response theory to provide personalized elearning services. The detailed system architecture will be described as follows.
Off-line courseware analysis
Courseware Modeling Process 1
3 User Account Database
8/16
10
Learning Interface Agent
Learner
2 Teacher Account Database
6
5 4/9
Courseware Database
Courseware Management Agent
Feedback Agent 11 12
14 7
User Profile Database
13
Courseware Recommendation 15 Agent
Figure 1. The system architecture
2.2 System Operation Procedure Based on the system architecture, the details of system operation procedure are described as follows:
2.1 Diagram of System Architecture
Step1
In this section, an adaptive e-learning system based on the fuzzy item response theory, which includes an off-line courseware modeling process, four intelligent agents and four databases, is presented herein. The four intelligent agents are the learning interface agent, feedback agent, courseware recommendation agent, and courseware management agent, respectively. Moreover, the four databases include the user account database, user profile 2
Courseware experts design the corresponding testing item for each learning concept. According to the item response theory, the difficulty parameters of these testing items can be determined through a statistics-based method. After that, courseware with web page type can be designed according to the conveying concept of corresponding testing item. The detailed courseware modeling process is described in
using the proposed fuzzy item response theory. Simultaneously, it calculates the corresponding information of each courseware according to learner’s ability and difficulty parameter of courseware, and ranks all courseware in this course unit by the order of information values. Finally, it passes the results to the learning interface agent and gives a list of recommended courseware for learner.
section 2.3. The designed courseware can be maintained through the courseware management agent and stored into the courseware database. Step2
Teachers login the system to upload, delete, or revise courseware in the courseware database by the legal users’ accounts.
Step3
Teachers maintain the courseware database through the courseware management agent.
Step4
Learner logins the system through the learning interface agent.
Step5
As a learner logins the system, the learning interface agent will check his account in the user account database. If the learner has already registered, system will get his learning profile from the user profile database.
Step6
Learning interface agent gets the contents of courseware from the courseware database and exhibits them for learner.
Step7
If the user has owned a legal account in our system, the learning interface agent will get his historically learning records from the user profile database to provide personalized learning. Otherwise, system will add a new record into the user profile database for him.
Step8
As the learning interface agent gets learner’s information from the courseware database, user account database, and user profile database, system will recommend an appropriate courseware to learner.
Step9
Learner needs to reply two simple questionnaires, i.e. the difficult level of the learned courseware and his understanding percentage of the learned courseware, for providing personalized learning.
Step15 The learner’s information which include learner’s ability, learner’s response and learning paths etc. calculated by the courseware recommendation agent will be recorded into the user profile database. Step16 Learning interface agent shows a list of recommendation courseware for learner and waits for learner’s feedback response. After learner selects next courseware for further learning, the system’s operating procedure will go back to Step 4, and it will continue to run the learning cycle from Step 4 to Step 16 until learner logouts this system.
2.3 Courseware Modeling Process The courseware modeling process presents a detailed courseware design procedure to determine the difficulty parameters of courseware and courseware’s contents for personalized courseware recommendation. In our previous work [37], we already proposed a voting approach which can determine the difficulty parameters of courseware by integration of experts’ decision and learners’ voting through a linear combination with different weights. However, the method assumes that learners will give completely confident voting results to finely tune the difficulty parameters of courseware predefined by course experts. The method might be subjective and easily influenced by learners’ abilities. Therefore, this study proposes a statistics-based method through a conscientious test process to determine the difficulty parameters of courseware. Since this approach is derived from the computer adaptive testing theory, it is more reasonable and logical than our previous work does. The detailed flowchart of courseware modeling process is illustrated as Figure 2. First of all, to design the course of C language programming as example, we invite several experienced teachers as our courseware experts to analyze the primary concepts for the course of C language programming in courseware modeling process. Courseware experts also design the corresponding testing item for each learning concept. That is, the testing items can be regarded as key characteristic of the corresponding learning content. Besides, about 500 examinees who have majored in the course of C language programming join the exam, which contains 25 testing items to cover those learning concepts. According to the item response theory, we analyze their
Step10 Learning interface agent passes the learner’s feedback information to the feedback agent. Step11 Feedback agent will transfer the learner’s understanding percentage into a fuzzy understanding degree by three predefined fuzzy membership functions, and then send the information to the courseware recommendation agent to evaluate learner’s ability. Step12 Feedback agent records the learner’s understanding percentage into the user profile database. Step13 Courseware recommendation agent obtains the difficulty parameters of learned courseware from the courseware database. Step14 Courseware recommendation agent evaluates learner’s ability according to the feedback information gathered from learner response by 3
percentage for the recommended courseware during a learning process. The learning interface agent will pass the learner’s response to the feedback agent. After the learning interface agent receives the recommendation courseware from the courseware recommendation agent, it will show a recommended courseware list according to a ranking mechanism described in section 2.7.2 for learner. If learner is an experienced user, then the system will get his/her previous ability in this course unit from the user profile database, and recommend appropriate courseware to him for further studying.
testing data by the BILOG program to obtain the difficulty parameters of these testing items. After that, we design courseware with web page type according to the conveying content of corresponding testing item. Since the content of courseware is derived from the concept of testing item, we assume the difficulty of courseware is equal to the difficulty of testing item. To define the difficulties of courseware more logically, this approach can avoid some subjective factors from learners with different knowledge background, thus can obtain more confident difficulty parameters for personalized e-learning services. Analyze learning contents 1 Testing items database
2
Design testing items for learning contents
course ID
difficulty
course 1
4.5
course 2
1.2
course 3
12.5
:
:
2.6 Feedback Agent To perform personalized recommendation mechanism more precisely, learner must give feedback responses by replying two simple questions, i.e. the difficult level and the understanding percentage for the recommended courseware. Feedback agent aims at collecting these learners’ feedback information from the interface agent into the user profile database. According to the item response theory (IRT), if learner can completely understand the content of the learned courseware, then his ability in the course unit will be promoted. On the contrary, learner’s ability will be descended if he cannot understand the content of the learned courseware. The original item response theory can correctly estimate learner ability by learner’s crisp response (i.e. completely understanding or not understanding answer), but it cannot estimate learner’s ability according to non-crisp response (i.e. fuzzy response). However, learner’s response is not usually completely understanding or not understanding case for the content of learned courseware. For example, learner might reply that his understanding percentage for the content of learned courseware is 70% or 30% etc. Therefore, we follow the testing logic with crisp response in the original item response theory to modify the method of ability’s estimation into a fuzzy response situation. The understanding percentage will be fuzzified as the fuzzy understanding degree by three predefined fuzzy membership functions, which can be predefined by course experts according to real requirement. Assume that the understanding percentage is set as x, the three fuzzy membership functions, i.e. the membership function of lowly understanding U l (x) , the membership function of moderately understanding U m (x) , and the membership function of highly understanding U h (x) , are defined as follows respectively: (1) The membership function of lowly understanding can be defined as follows:
3. Exam 4 5. Record
6. Analyze
Collect testing data
Obtain difficulty parameters
7
Design courseware with the corresponding difficulty parameter
8
Courseware database
Figure 2. Courseware modeling process
2.4 Courseware Management Agent Courseware management agent administers the details of maintaining the courseware database. It provides a friendly user interface for teachers to upload, delete, or revise courseware. By checking the accounts of teachers in the teacher account database, our system just permits a legal user to manage the courseware database through the designed user interface. Using this interface, system administrator can also manage teachers’ accounts, course category, course units, and course content. In order to exchange courseware with the other e-learning systems more easily, our system will follow the SCORM (Sharable Content Object Reference Model) standard [38] to construct the courseware with XML format in the future.
2.5 Learning Interface Agent The learning interface agent provides a friendly user interface to communicate with learners, passes the learners’ feedback information to the feedback agent, and receives the recommendation result from the courseware recommendation agent. Through the learning interface agent, learners can choose interested course categories and units to study. Learners can also give appropriate keywords to search the needed courseware through system’s search mechanism during a learning process. If a learner visits the courseware recommendation system at first time, he must register as a legal user by inputting his e-mail address. When a learner logins this system, the learning interface agent will show the courseware with moderate difficulty to him and ask for the understanding
U l ( x) = e
−(
x 2 ) 0. 5
for x < 0.4
(1)
(2) The membership function of moderately understanding can be defined as follows:
4
U m ( x) = e
−(
x − 0 .5 2 ) 0.125
The courseware recommendation agent aims to estimate learner’s ability based on the fuzzy item response theory, evaluates information of courseware, and ranks courseware based on the information of courseware. The operation procedure of the courseware recommendation agent is shown as Figure 4. The following subsection describes how to evaluate learner’s ability and recommend appropriate courseware to learner in detail.
(2)
for 0.4 ≤ x ≤ 0.6
(3) The membership function of highly understanding can be defined as follows: −(
x −1
)2
(3) U h ( x ) = e 0.5 for x > 0.6 Figure 3 illustrates the plot of three fuzzy membership functions for mapping the understanding percentage into its corresponding understanding degree.
Feedback agent 1. input Uu(x)
The degree of membership function
1
Highly understanding
Lowly understanding
2
Estimate learner ability based on fuzzy item response theory
Moderately understanding
3
4
Evaluate information of courseware
0.5
User profile database
Courseware database
5
Rank courseware based on the infromation of courseware
0 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
7
Percentage of understanding
Figure 3. Plot of the fuzzy membership functions for mapping the understanding percentage into its corresponding understanding degree
Recommend courseware with first order to learner Courseware recommendation agent
As system gets the understanding percentage from learner’s feedback response, system will fuzzified it to obtain the corresponding fuzzy understanding degrees through three predefined fuzzy membership functions. In this work, the fuzzy maximum operator is applied to obtain the final fuzzy understanding degree U u (x) . The formula is described as follows: U u ( x) = Max{U l ( x), U m ( x), U h ( x)}
6
Figure 4. The operation flowchart of courseware recommendation
2.7.1 Learner’s Ability Estimation To estimate learner’s ability, the item characteristic function proposed by Rasch with a single difficulty parameter is used to model the courseware. The formula of item characteristic function with single difficulty parameter can be described as follows:
(4)
where U u (x) is the final fuzzy understanding degree under the understanding percentage as x , U l (x) is the mapping fuzzy understanding degree of the lowly understanding membership function under the understanding percentage as x , U m (x) is the mapping fuzzy understanding degree of the moderately understanding membership function under the understanding percentage as x , and U h (x) is the mapping fuzzy understanding degree of the highly understanding membership function under the understanding percentage as x .
P j (θ ) =
e
D (θ − b j )
1+ e
D (θ −b j )
(5)
where P j (θ ) denotes the probability that learners can
completely understand the j th courseware at a level below their ability level θ , b j is the difficulty of the j th courseware, and D is a constant 1.702. There are two widely used methods to estimate learner’s ability. They are the maximum likelihood estimation (MLE) and Bayesian estimation approaches [27-30]. Although the procedure of MLE is simple and easily implemented, it has the problem of divergent estimation for learner’s ability estimation when learner gives completed understanding or not understanding responses for all learned courseware during a learning process [27]. MLE will overestimate learner’s ability to the completed understanding case. On the other hand, MLE will underestimate learner’s ability to the completed not understanding case. Compared with the procedure of MLE,
Finally, the feedback agent transfers the learner’s understating percentage into the final fuzzy understanding degree and passes this information to the courseware recommendation agent to evaluate the learner’s ability and recommend suitable learning courseware to learner.
2.7 Courseware Recommendation Agent Based on the estimation of learner’s ability, this system can recommend the appropriate courseware to learners. 5
the procedure of Bayesian estimation is more complex and less efficient, but it can solve the problem of the divergent estimation for learner’s ability estimation in the procedure of MLE. Basically, a prior-information on the distribution of the learner’s abilities is employed here to estimate learner’s ability [27]. Hence, the Bayesian estimation procedure will always converge for all possible learners’ responses [27]. Based on this reason, the Bayesian estimation procedure is applied to estimate learner’s ability in this study. Bock and Mislevy [27] have given the quadrature form to approximately estimate learner’s ability as follows:
understanding case, moderately understanding case, and lowly understanding case, respectively. In this work, three fuzzy membership functions were defined as Eqn. (1) to (3) to support the evaluation of final fuzzy understanding degree U u (x) described as Eqn. (4). Based on the proposed evaluation method of final fuzzy understanding degree, the estimation value of new learner’s ability for the new learned courseware can be defined as follows: θ j + (θ w − θ j ) × U u ( x) when θ j +1 = θj when θ + (θ − θ ) × U ( x) when c j u j
q
θˆ =
∑θ
k L(u1, u 2 ,..., u n
number of
(6)
q
1
2 ,..., u n
where θˆ represents the learner’s ability of estimation, L(u1, u2 , L , un | θ k ) is the value of likelihood function at a level below their ability level θ k and learner’s responses are u1 , u 2 ,..., u n , θ k is the k th split value of ability in the standard normal distribution, and A(θ k ) represents the quadrature weight at a level below their ability level θ k .
Based on the definitions of Eqn. (4) and (8), if a learner replied that his understanding percentage for the content of learned courseware is below 40%, then U u (x) is equal to U l (x) . In this case, we think that he cannot understand the most part of learning contents of learned courseware, so his new estimation ability in this course unit should be descended according to the definition of Eqn. (8). That is, he needs to review easier courseware in order to promote his basic knowledge in this course unit. On the other hand, if a learner replied that his understanding percentage for the content of learned courseware is over 60%, then U u (x) is equal to U h (x) . In this case, we think that he can understand the great part of contents of learned courseware, so his new estimation ability in this course unit should be promoted according to the definition of Eqn. (8). Finally, if a learner replied that the understanding percentage for the content of learned courseware is between 40% and 60%, then U u (x) is equal to U m (x) . We think that learner’s response has lower confidence or too fuzzy, thus keeping his new estimation ability as the previous one. Therefore, our system will recommend other learning courseware with similar difficult level to learner. After evaluating the new learner ability, our system will evaluate the information of courseware based on the maximum information strategy to find out the most appropriate courseware for learner in order to do further studying.
In Eqn. (6), the likelihood function L(u1, u2 , L , un | θ k ) can be further described as follows: n
∏ P (θ ) j
k
uj
1−u j
Q j (θ k )
(7)
j =1
where P j (θ k ) =
e
j previous learned courseware, θ j +1 is
estimation value of new learner’s ability for the ( j + 1) th new courseware, θ w is estimation value of learner’s ability under assuming that learner cannot completely understand the ( j + 1) th new courseware, θ c is estimation value of learner’s ability under assuming that learner can completely understand the ( j + 1) th new courseware, and U u (x) is the final fuzzy understanding degree.
| θ k ) A(θ k )
k
L(u1, u2 ,L, un | θ k ) =
(8)
where θ j is estimation value of learner’s ability for total
| θ k ) A(θ k )
k
∑ L(u , u
x < 0.4 0.4 ≤ x ≤ 0.6 x > 0.6
D (θ k −b j )
, Q j (θ k ) = 1 − P j (θ k ) , Pj (θ ) D (θ −b ) 1+ e k j denotes the probability that learners can understand the
j th courseware at a level below their ability level θ k , Q j (θ k ) represents the probability that learners cannot
understand the j th courseware at a level below their ability level θ k , and U j is the answer of completely understanding or not understanding answer obtained from learner feedback to the j th courseware, i.e. if the answer is completely understanding then U j = 1 ; otherwise, U j = 0 . However, the Bayesian estimation procedure can just estimate learner’s ability for learner’s crisp responses, but it cannot estimate learner’s ability for learner’s fuzzy responses. In order to solve this problem, the fuzzy item response theory derived from the original item response theory is presented herein. The proposed fuzzy item response theory distinguishes the understanding degrees of learners into three various cases according to the experienced teachers’ decision. They are highly
2.7.2 Maximum Information Evaluation and Courseware Recommendation In IRT, there are two approaches that can be used to
6
As a learner logins this system, he can choose a course unit that he feels interested for learning. Figure 5 shows the entire layout of the learning interface. In the left frame, system shows the course categories, course units and the list of all courseware in the courseware database. While a learner clicks a courseware for learning, the content of selected courseware will be exhibited in the upper-right window. Besides, the feedback interface is arranged in the bottom-right window. System can get learner’s feedback response from the feedback in interface through learner replies two simple questions as listed in Table 1.
recommend appropriate courseware to learner. They are the maximum information strategy and Bayesian strategy [27-30]. The main concept of maximum information strategy is that each courseware with the corresponding difficulty parameter exhibits different information to learner’s learning. A courseware with higher information represents that it is more suitable to be recommended for learner. Since Bayesian strategy is more complicated than the maximum information, the maximum information method is applied to recommend appropriate courseware to learner. The maximum information function is defined as follows: (1.7) 2 (9) I (θ ) = j
[e
1.7 (θ −b j )
] [1 + e
Table 1. Two questionnaires for getting learner’s feedback information
]
−1.7 (θ −b j ) 2
where I j (θ ) is the information value of the
j th
courseware at a level below their ability level θ , b j is the difficulty parameter of the j th courseware. After calculating the corresponding information values of all courseware, the course recommendation agent can recommend a series of courseware to learners with ability θ according to the ranking order of information function value. A courseware with the maximum information function value under learner with ability θ indicates that the system presented here gives the highest recommendation priority. Whether learner accepts the recommended courseware with highest recommendation priority or selects the other recommended courseware to do further learning, our system will record learner’s learning paths and learner’s feedback responses into the user profile database during a learning process. These information can provide system to develop more intelligent agents for personalized learning, such as learning process analysis and learning diagnosis etc. Based on this vision, some minded tools which can assist personalized learning are also considered in our future research.
Figure 5. The learning interface for learner The answer of question one can be served as an investigation of learner’s satisfactory degree for the recommended courseware. The 5-point Likert-scale proposed by Likert in 1932 [39] is applied to define various scaled answers. In a variation of standard Likertscale, this study uses a scale where -2 indicates “very easy”, -1 is “easy”, 0 is “moderate”, 1 is “hard” and 2 is “very hard”. If a learner feels that the recommended courseware is quite suitable for him, then the averaged value of his answers should be very close to zero, i.e. “Moderate”. The answer of question two helps system to get the learner’s understanding percentage for the recommended courseware. System passes this information to the courseware recommendation agent to evaluate the learner’s ability. After a learner presses the button of analysis, this system will reveal a list of the recommended courseware based on his current ability. Figure 6 shows an example of courseware recommendation based on learner ability after learner gives corresponding feedback response, and the recommended courseware ranked by the order of their information values. The title (標題) indicates the subject of the courseware; the recommendation (推薦指數) denotes
3. Experiments To verify the learning efficiency and effectiveness for the proposed courseware recommendation system, the prototype of the system has been successfully implemented and some university students who have majored in the course of C language programming were invited to test this system. Currently, the URL address of the web site is available at http://203.64.142.238:5647. The detailed functions of this system and experimental results are described as follows.
3.1 System’s Functions At present, the proposed prototype is implemented on the platform of Microsoft Windows 2000 with IIS 5.0 Web server. Moreover, the front-end script language of PHP 4.3 and MySQL server are used to implement this system. 7
the information value of the recommended courseware; and the description (描述) gives a brief description for the corresponding courseware. The length of bar line in the column of recommendation indicates the information value of the corresponding courseware. The longer bar line implies a more suitable courseware for learner. On the contrary, the shorter bar line implies an unsuitable courseware for learner.
ability described in Eqn. (8). On the other hand, if a learner replied that his understanding percentage for the current courseware below 40%, his estimation ability will be descended. In the 12th and 14th ability estimation, learner’s ability is same as the previous one because the understanding percentage of learner’s response for the learned courseware is between 40% and 60%. Besides, Figure 8 shows the relationship of the learner ability with the difficulty parameter of the recommended courseware. We find that the difficulty parameter of the recommended courseware is highly relevant with the learner ability. This result shows that the proposed system indeed can recommend appropriate courseware to learners according to different learner abilities. Table 2. The corresponding difficulty of each courseware
Title of courseware
Difficulty
Special operation of e1 ? e2 : e3 Basic input/output functions and applications Else - if instruction
-2.738
If - else control instruction
-2.29
Switch instruction
0.526
3.2 Experimental Results
Relationship and logic operators (三)
1.001
Currently, this system only contains a small amount of courseware because to produce high quality courseware needs a large amount of manpower to join this work. In our experiments, C language programming is selected as a course unit to provide personalized learning services. All designed courseware of C language programming and their corresponding difficulty parameters determined by statistics-based method are listed in Table 2. Actually, to expand the courseware database and supply more abundantly courseware for learners are urgently needed in the future. Figure 7 shows the relationship between the understanding percentage of the clicked courseware and the adjustment of the learner’s ability. In this system, the range of learner’s ability is limited from -3 (i.e. lowest ability) to 3 (i.e. highest ability). As a learner logins this system, if the user account database does not have any history records in the selected course unit for this learner, then his initial ability will be regarded as 0. That is, the system assumes learner’s ability is middle level. As a learner clicked the recommended courseware for learning, his ability in this course unit will be re-evaluated according to his response of understanding percentage and the corresponding difficulty parameter of the learned courseware. In Figure 7, we can find that the trend of learner’s estimation ability follows his response of understanding percentage for the learned courseware. If learner replied that his understanding percentage for the current courseware is over 60%, his ability estimation will be promoted based on the estimation formula of learner’s
# define and constant instructions
1.081
C language introduction
1.081
For loop introduction
2.554
Do - while loop introduction
3.559
Relationship and logic operators (二)
4.378
Basic bit operation
4.496
Special operations in C language (一)
5.128
Basic data type and mathematics operations
5.128
Relationship and logic operators (一)
6.173
Definition of functions
6.521
Flow control and special operations
6.613
Introduction of printf () instruction
6.706
Figure 6. An example of courseware recommendation ranked by the ranking order of information values
Basic data type Function’s applications
8
-4.804
-2.461
6.9 7
Relationship and logic operators (四)
7.778
One dimension array
10.297
Special operations in C language (二)
16.71
services more intimately, some minded tools such as personalized learning diagnosis agent, personalized courseware retrieval agent, personalized learning-path analysis agent will be considered in our future research.
Ability & The understanding percentage of clicked courseware Understanding percentage
Ability
Ability
Understanding percentage
3
100% 90% 90% 80%
2
80% 80% 80%
70%
70%
70%
References
70% 70% 70% 70%
1
60%
60% 60%
0
[1]
Berghel H., “Cyberspace 2000: Dealing with Information overload,” Communications of the ACM, vol. 40, no. 2, pp.19-24, Feb. 1997.
[2]
Borchers A., Herlocker J., Konstanand J., Riedl J., “Ganging Up on Information Overload,” Computer, vol. 31, issue: 4, pp. 106-108, Apr. 1998.
[3]
Arasu A., Cho J., Garcia-Molina H., Paepcke A., Raghavan S., “Searching the Web,” ACM Transactions on Internet Technology, vol. 1, issue: 1, pp. 97-101, 2001.
[4]
Kobayashi M., Takeda K., “Information Retrieval on the Web,” ACM Computing Surveys, vol. 32, no. 2, pp. 144-173, 2000.
[5]
Lawrence S., Giles C. L., “Searching the Web: General and Scientific Information Access,” IEEE Communications, vol. 37, pp. 116-122, 1999.
[6]
Brin S., Page, L., “The PageRank Citation Ranking: Bringing Order to the Web,” In Proceedings of ASIS’98, pp. 161-172, 1998.
[7]
Chidlovskii B., Glance N., Grasso A., “Collaborative Re-Ranking of Search Results,” In Proceedings of AAAI-2000 Workshop on Artificial Intelligence for Web Search, Austin, Texas, pp. 18-22, 2000.
[8]
Kleinberg J., “Authoritative Sources in a Hyperlinked Environment,” In Proceedings of 9th ACM-SIAM Symposium on Discrete Algorithms, pp. 668-679, 1998.
[9]
Direct Hit, 2000, http://www.directhit.com.
50% 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17 40%
-1 30%
30%
30% 25%
-2
20%
20%
20%
10% -3
0%
Figure 7. The relationship between the understanding percentage of the clicked courseware and the adjustment of the learner’s ability Ability & The difficulty of recommendation courseware Difficulty
Difficulty
Ability
Ability
8
3
2.5
6
2 4 1.5 2 1 0 1
2
3
4
-2
-4
5
6
7
8
9
10
11
12
13
14
15
16
17
0.5
0
-0.5
Figure 8. The relationship between learner’s ability and the difficulty parameter of the recommended courseware
4. Conclusion This study proposes a personalized courseware recommendation system (PCRS) based on the proposed fuzzy item response theory (FIRT), which can estimate the abilities of online learners and recommend appropriate courseware to learners. Compared to the traditional item response theory, the fuzzy item response theory can accept non-crisp response to correctly estimate learner’s ability via the revised estimating function of learner’s ability. PCRS provides personalized Web-based learning according to courseware visited by learners and their responses. Moreover, courseware difficulty can also be correctly determined by the proposed courseware modeling process. Experimental results show that the proposed system can precisely provide personalized courseware recommendations on-line based on learner abilities and responses, and moreover can accelerate learner learning efficiency and effectiveness. Importantly, learners only need to reply two simple questions for personalized services. Besides, to provide personalized e-learning
Available
at
[10] Google, Available at http://www.google.com. [11] NEC Research Institute ResearchIndex, Available at http://citeseer.nj.nec.com. [12] Brusilovsky P., “Adaptive and Intelligent Technologies for Web-based Education,” In Rollinger, C.; Peylo, C. (eds.), Special Issue on Intelligent Systems and Teleteaching, Künstliche Intelligenz, vol. 4, pp. 19-25, 1999. [13] Kao T. T., Personalization Courseware Navigation Using Grey Ranking Analysis, Master’s thesis, Department of Electronic Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, 2001.
9
[14] Khan B. H., Web Based Instruction, Englewood Cliffs, New Jersey: Educational Technology Publications, 1997.
[26] Brusilovsky P., “Adaptive and Intelligent Technologies for Web-based Education,” In Rollinger, C.; Peylo, C. (eds.), Special Issue on Intelligent Systems and Teleteaching, Künstliche Intelligenz, vol. 4, pp. 19-25, 1999.
[15] Lee Y. J., Chiu Y. W., Hong C. F., “Intelligent Adaptive Teaching Inference of Virtual College,” Distance Education, vol. 15/16, pp. 85-97, 2000. (in Chinese)
[27] Baker and Frank B., Item Response Theory: Parameter Estimation Techniques, New York: Marcel Dekker, 1992.
[16] Lin L. C., Chen K. S., Chen Y. L., Jao P. L., Tsai C. C., “The Application of Adaptive Multi-Agent Based Architecture in Distance Learning Environment,” TANET, 2001, Available at http://www.ccu.edu.tw/TANET2001/scheduel/paper _abs/J124.html.
[28] Hambleton, R. K., Item Response Theory: Principles and Applications, Boston, Kluwer-Nijhoff Publisher, 1985. [29] Hulin, C. L., Drasgow, F., Parsons and C. K., Item Response Theory: Application to Psychological Measurement, Homewood, IL: Dow Jones Irwin, 1983.
[17] Lin H. N., An Adaptive Learning Environment: A Case Study on High School Mathematics, Master’s thesis, Department of Computer and Information Science, National Chiao Tung University, Hsinchu, Taiwan, 2001.
[30] Hsu, T. C. and Liu C. H., Introduction to Item Response Theory, Taipei, Taiwan, Chinese Behavioral Science Corporation, 1991.
[18] Hong C. F, Fu M. Y., Chen W. H., Yang S. C., Chang C. Y., Chang C. Y., Peng S. C., Liu Y. T., Mo S. M., “Constructing an Suitable Teaching Environment on the World Wide Web,” Distance Education, vol. 13/14, pp. 105-110, 2000. (in Chinese)
[31] Horward W., Computerized Adaptive Testing: A Primer, Hillsdale, New Jersey: Lawerence Erwrence Erlbaum Associates, 1990. [32] Hsu, T. C. and Sadock, S. F., Computer-assisted Test Construction: A State of Art, TME report 88, Princeton, New Jersey, Eric on Test, Measurement, and Evaluation, Educational Testing Service, 1985.
[19] Chen J. L., A Multiagent-based Intelligent Instruction Agent, Master’s thesis, Department of Information Engineering and Computer Science, Feng Chia University, Taichung, Taiwan, 1999.
[33] TOEFL, Avaliable at http://www.toefl.org. [34] GRE, Avaiable at http://www.gre.org.
[20] Yang T. L., A Study of Dynamical Adaptive Learning System, Master’s thesis, National Kaohsiung Normal University, Kaohsiung, Taiwan, 2001.
[35] GMAT, Available at http://www.gmat.org. [36] H.-J. Zimmermann, Fuzzy Sets, Decision Making, and Expert System, Boston: Kluwer Academic Publishers, 1987.
[21] Liu M. Z., Chen L. C., Shou D. W., Lin H. L., “A Study on Adaptive Learning Design in Web-based Courseware,” MOE of ROC research paper, 1999.
[37] Hahn-Ming Lee, Chih-Ming Chen and Ya-Hui Chen, “Personalized E-Learning System Utilizing Item Response Theory,” International Computer Symposium, Workshop on Artificial Intelligence, Dong Hwa University, Hualien, Taiwan, 2002.
[22] Mobasher B., Cooley R., Srivastava J., “Automatic Personalization Based on Web Usage Mining,” Communications of the ACM, vol. 43, no. 8, pp. 142151, August 2000.
[38] Advanced Distributed http://www.adlnet.org/.
[23] Kao T. T., Personalization Courseware Navigation Using Grey Ranking Analysis, Master’s thesis, Department of Electronic Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, 2001.
Learning,
available
at
[39] Likert R., A Technique for the Measurement of Attitudes, New York: Archives of Psychology, 1932.
[24] Balabanovic M., Shoham Y., “Fab: Content-based, Collaborative Recommendation,” Communications of the ACM, vol. 40, no. 3, pp. 66-72, March 1997. [25] Fu X., Budzik J., Hammond J. K., “Mining Navigation History for Recommendation,” Proceedings of the 2000 International Conference on Intelligent User Interfaces, pp. 106-112, 2000.
10
Abstract With the rapid growth of computer and Internet technologies, e-learning has become a major trend in the computer assisted teaching and learning field currently. In past years, many researchers made effort in developing elearning systems with personalized learning mechanism to assist on-line learning. However, most of them focused on using learner’s behaviors, interests, and habits to provide personalized e-learning services. These systems usually neglected to concern if learner’s ability and the difficulty of courseware are matched each other. Generally, recommending an inappropriate courseware might result in learner’s cognitive overhead or disorientation during a learning process. To promote learning efficiency and effectiveness, this paper presents a personalized courseware recommendation system (PCRS) based on the proposed fuzzy item response theory (FIRT), which can recommend courseware with appropriate difficult level to learner through learner gives a fuzzy response of understanding percentage for the learned courseware. Therefore, this paper will deal with how to modify the traditional item response theory (IRT) to handle non-crisp fuzzy response and estimate learner’s ability via the revised estimating function of learner’s ability. Based on the courseware’s modeling requirement, this paper also proposes a courseware modeling process to determine the difficulty parameters of courseware and design the content of courseware for personalized courseware recommendation services. Experiment results show that applying the proposed fuzzy item response theory to Webbased learning can achieve personalized learning and help learners to learn more effectively and efficiently.
1. Introduction Recently years, numerous Web applications have been developed, the rapid growth of information on the Web has created a problem of information overload [1-2], such that Internet users are unable to find the information they
Based on previous analyses, this study proposes a
1
personalized courseware recommendation system (PCRS) based on the proposed fuzzy item response theory to provide Web-based personalized e-learning services. In the proposed fuzzy item response theory, fuzzy theory [36] is combined with the original item response theory to model uncertainly learning response. Moreover, the single parameter logistic model with difficulty parameter proposed by Georg Rasch in 1966[27-30] is applied to model various difficulty levels of courseware. PCRS can dynamically estimate learner ability based on the proposed fuzzy item response theory by collecting learner feedback after studying the recommended courseware. Based on the estimation of learner abilities, the novel system can recommend appropriate courseware to learners. Restated, learner ability and the difficulties of course materials are simultaneously taken into account when implementing a personalization mechanism. Moreover, based on the courseware’s modeling requirement, this paper also proposes a courseware modeling process to determine the difficulty parameters of courseware and construct the content of courseware for personalized courseware recommendation services. Experiments show that the proposed personalized courseware recommendation system can recommend appropriate course materials to learners based on individual ability, and help them to learn more efficiently and effectively.
database, courseware database, and teacher account database. The learner interface agent aims at providing a friendly learning interface for learners to interact with the feedback agent and courseware recommendation agent. On the other hand, the feedback agent aims at collecting learner explicit feedback information from the learning interface agent and storing them into the user profile database for personalized e-learning services. Moreover, the courseware recommendation agent is in charge of recommending adaptive courseware to learner according to learner’s feedback response and his ability. Finally, the courseware management agent with authorized account management mechanism provides a friendly courseware management interface, which can help teachers to create new course units, upload courseware to the courseware database, delete and modify courseware from the courseware database etc. The system architecture is shown as Figure 1. Furthermore, we also propose a courseware modeling process derived from the computer adaptive testing (CAT) theory [31-32] to assign courseware with appropriate difficulty parameters for the personalized courseware recommendation. The proposed courseware modeling process successfully transfers the concepts of testing items into courseware to provide courseware resources with the corresponding difficulty parameter for personalized e-learning services. The detailed operation procedure of system architecture is described as the following section.
2. System Architecture Most learning recommendation systems mainly focus on analyzing learner’s behavior and interests to provide personalized e-learning services. However, both learner’s ability and courseware’s difficulty are usually ignored as important factors while personalized e-learning services are performed. Generally, the courseware recommended by the learning recommendation systems might be too hard or too easy to learners if these two factors have not been further considered for courseware recommendation. Namely, learners usually need to accept some inappropriate recommendation courseware even they cannot realize the content of courseware completely. In order to provide the courseware with appropriate difficult level to learners, this study proposes a personalized courseware recommendation system (PCRS) based on the fuzzy item response theory to provide personalized elearning services. The detailed system architecture will be described as follows.
Off-line courseware analysis
Courseware Modeling Process 1
3 User Account Database
8/16
10
Learning Interface Agent
Learner
2 Teacher Account Database
6
5 4/9
Courseware Database
Courseware Management Agent
Feedback Agent 11 12
14 7
User Profile Database
13
Courseware Recommendation 15 Agent
Figure 1. The system architecture
2.2 System Operation Procedure Based on the system architecture, the details of system operation procedure are described as follows:
2.1 Diagram of System Architecture
Step1
In this section, an adaptive e-learning system based on the fuzzy item response theory, which includes an off-line courseware modeling process, four intelligent agents and four databases, is presented herein. The four intelligent agents are the learning interface agent, feedback agent, courseware recommendation agent, and courseware management agent, respectively. Moreover, the four databases include the user account database, user profile 2
Courseware experts design the corresponding testing item for each learning concept. According to the item response theory, the difficulty parameters of these testing items can be determined through a statistics-based method. After that, courseware with web page type can be designed according to the conveying concept of corresponding testing item. The detailed courseware modeling process is described in
using the proposed fuzzy item response theory. Simultaneously, it calculates the corresponding information of each courseware according to learner’s ability and difficulty parameter of courseware, and ranks all courseware in this course unit by the order of information values. Finally, it passes the results to the learning interface agent and gives a list of recommended courseware for learner.
section 2.3. The designed courseware can be maintained through the courseware management agent and stored into the courseware database. Step2
Teachers login the system to upload, delete, or revise courseware in the courseware database by the legal users’ accounts.
Step3
Teachers maintain the courseware database through the courseware management agent.
Step4
Learner logins the system through the learning interface agent.
Step5
As a learner logins the system, the learning interface agent will check his account in the user account database. If the learner has already registered, system will get his learning profile from the user profile database.
Step6
Learning interface agent gets the contents of courseware from the courseware database and exhibits them for learner.
Step7
If the user has owned a legal account in our system, the learning interface agent will get his historically learning records from the user profile database to provide personalized learning. Otherwise, system will add a new record into the user profile database for him.
Step8
As the learning interface agent gets learner’s information from the courseware database, user account database, and user profile database, system will recommend an appropriate courseware to learner.
Step9
Learner needs to reply two simple questionnaires, i.e. the difficult level of the learned courseware and his understanding percentage of the learned courseware, for providing personalized learning.
Step15 The learner’s information which include learner’s ability, learner’s response and learning paths etc. calculated by the courseware recommendation agent will be recorded into the user profile database. Step16 Learning interface agent shows a list of recommendation courseware for learner and waits for learner’s feedback response. After learner selects next courseware for further learning, the system’s operating procedure will go back to Step 4, and it will continue to run the learning cycle from Step 4 to Step 16 until learner logouts this system.
2.3 Courseware Modeling Process The courseware modeling process presents a detailed courseware design procedure to determine the difficulty parameters of courseware and courseware’s contents for personalized courseware recommendation. In our previous work [37], we already proposed a voting approach which can determine the difficulty parameters of courseware by integration of experts’ decision and learners’ voting through a linear combination with different weights. However, the method assumes that learners will give completely confident voting results to finely tune the difficulty parameters of courseware predefined by course experts. The method might be subjective and easily influenced by learners’ abilities. Therefore, this study proposes a statistics-based method through a conscientious test process to determine the difficulty parameters of courseware. Since this approach is derived from the computer adaptive testing theory, it is more reasonable and logical than our previous work does. The detailed flowchart of courseware modeling process is illustrated as Figure 2. First of all, to design the course of C language programming as example, we invite several experienced teachers as our courseware experts to analyze the primary concepts for the course of C language programming in courseware modeling process. Courseware experts also design the corresponding testing item for each learning concept. That is, the testing items can be regarded as key characteristic of the corresponding learning content. Besides, about 500 examinees who have majored in the course of C language programming join the exam, which contains 25 testing items to cover those learning concepts. According to the item response theory, we analyze their
Step10 Learning interface agent passes the learner’s feedback information to the feedback agent. Step11 Feedback agent will transfer the learner’s understanding percentage into a fuzzy understanding degree by three predefined fuzzy membership functions, and then send the information to the courseware recommendation agent to evaluate learner’s ability. Step12 Feedback agent records the learner’s understanding percentage into the user profile database. Step13 Courseware recommendation agent obtains the difficulty parameters of learned courseware from the courseware database. Step14 Courseware recommendation agent evaluates learner’s ability according to the feedback information gathered from learner response by 3
percentage for the recommended courseware during a learning process. The learning interface agent will pass the learner’s response to the feedback agent. After the learning interface agent receives the recommendation courseware from the courseware recommendation agent, it will show a recommended courseware list according to a ranking mechanism described in section 2.7.2 for learner. If learner is an experienced user, then the system will get his/her previous ability in this course unit from the user profile database, and recommend appropriate courseware to him for further studying.
testing data by the BILOG program to obtain the difficulty parameters of these testing items. After that, we design courseware with web page type according to the conveying content of corresponding testing item. Since the content of courseware is derived from the concept of testing item, we assume the difficulty of courseware is equal to the difficulty of testing item. To define the difficulties of courseware more logically, this approach can avoid some subjective factors from learners with different knowledge background, thus can obtain more confident difficulty parameters for personalized e-learning services. Analyze learning contents 1 Testing items database
2
Design testing items for learning contents
course ID
difficulty
course 1
4.5
course 2
1.2
course 3
12.5
:
:
2.6 Feedback Agent To perform personalized recommendation mechanism more precisely, learner must give feedback responses by replying two simple questions, i.e. the difficult level and the understanding percentage for the recommended courseware. Feedback agent aims at collecting these learners’ feedback information from the interface agent into the user profile database. According to the item response theory (IRT), if learner can completely understand the content of the learned courseware, then his ability in the course unit will be promoted. On the contrary, learner’s ability will be descended if he cannot understand the content of the learned courseware. The original item response theory can correctly estimate learner ability by learner’s crisp response (i.e. completely understanding or not understanding answer), but it cannot estimate learner’s ability according to non-crisp response (i.e. fuzzy response). However, learner’s response is not usually completely understanding or not understanding case for the content of learned courseware. For example, learner might reply that his understanding percentage for the content of learned courseware is 70% or 30% etc. Therefore, we follow the testing logic with crisp response in the original item response theory to modify the method of ability’s estimation into a fuzzy response situation. The understanding percentage will be fuzzified as the fuzzy understanding degree by three predefined fuzzy membership functions, which can be predefined by course experts according to real requirement. Assume that the understanding percentage is set as x, the three fuzzy membership functions, i.e. the membership function of lowly understanding U l (x) , the membership function of moderately understanding U m (x) , and the membership function of highly understanding U h (x) , are defined as follows respectively: (1) The membership function of lowly understanding can be defined as follows:
3. Exam 4 5. Record
6. Analyze
Collect testing data
Obtain difficulty parameters
7
Design courseware with the corresponding difficulty parameter
8
Courseware database
Figure 2. Courseware modeling process
2.4 Courseware Management Agent Courseware management agent administers the details of maintaining the courseware database. It provides a friendly user interface for teachers to upload, delete, or revise courseware. By checking the accounts of teachers in the teacher account database, our system just permits a legal user to manage the courseware database through the designed user interface. Using this interface, system administrator can also manage teachers’ accounts, course category, course units, and course content. In order to exchange courseware with the other e-learning systems more easily, our system will follow the SCORM (Sharable Content Object Reference Model) standard [38] to construct the courseware with XML format in the future.
2.5 Learning Interface Agent The learning interface agent provides a friendly user interface to communicate with learners, passes the learners’ feedback information to the feedback agent, and receives the recommendation result from the courseware recommendation agent. Through the learning interface agent, learners can choose interested course categories and units to study. Learners can also give appropriate keywords to search the needed courseware through system’s search mechanism during a learning process. If a learner visits the courseware recommendation system at first time, he must register as a legal user by inputting his e-mail address. When a learner logins this system, the learning interface agent will show the courseware with moderate difficulty to him and ask for the understanding
U l ( x) = e
−(
x 2 ) 0. 5
for x < 0.4
(1)
(2) The membership function of moderately understanding can be defined as follows:
4
U m ( x) = e
−(
x − 0 .5 2 ) 0.125
The courseware recommendation agent aims to estimate learner’s ability based on the fuzzy item response theory, evaluates information of courseware, and ranks courseware based on the information of courseware. The operation procedure of the courseware recommendation agent is shown as Figure 4. The following subsection describes how to evaluate learner’s ability and recommend appropriate courseware to learner in detail.
(2)
for 0.4 ≤ x ≤ 0.6
(3) The membership function of highly understanding can be defined as follows: −(
x −1
)2
(3) U h ( x ) = e 0.5 for x > 0.6 Figure 3 illustrates the plot of three fuzzy membership functions for mapping the understanding percentage into its corresponding understanding degree.
Feedback agent 1. input Uu(x)
The degree of membership function
1
Highly understanding
Lowly understanding
2
Estimate learner ability based on fuzzy item response theory
Moderately understanding
3
4
Evaluate information of courseware
0.5
User profile database
Courseware database
5
Rank courseware based on the infromation of courseware
0 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
7
Percentage of understanding
Figure 3. Plot of the fuzzy membership functions for mapping the understanding percentage into its corresponding understanding degree
Recommend courseware with first order to learner Courseware recommendation agent
As system gets the understanding percentage from learner’s feedback response, system will fuzzified it to obtain the corresponding fuzzy understanding degrees through three predefined fuzzy membership functions. In this work, the fuzzy maximum operator is applied to obtain the final fuzzy understanding degree U u (x) . The formula is described as follows: U u ( x) = Max{U l ( x), U m ( x), U h ( x)}
6
Figure 4. The operation flowchart of courseware recommendation
2.7.1 Learner’s Ability Estimation To estimate learner’s ability, the item characteristic function proposed by Rasch with a single difficulty parameter is used to model the courseware. The formula of item characteristic function with single difficulty parameter can be described as follows:
(4)
where U u (x) is the final fuzzy understanding degree under the understanding percentage as x , U l (x) is the mapping fuzzy understanding degree of the lowly understanding membership function under the understanding percentage as x , U m (x) is the mapping fuzzy understanding degree of the moderately understanding membership function under the understanding percentage as x , and U h (x) is the mapping fuzzy understanding degree of the highly understanding membership function under the understanding percentage as x .
P j (θ ) =
e
D (θ − b j )
1+ e
D (θ −b j )
(5)
where P j (θ ) denotes the probability that learners can
completely understand the j th courseware at a level below their ability level θ , b j is the difficulty of the j th courseware, and D is a constant 1.702. There are two widely used methods to estimate learner’s ability. They are the maximum likelihood estimation (MLE) and Bayesian estimation approaches [27-30]. Although the procedure of MLE is simple and easily implemented, it has the problem of divergent estimation for learner’s ability estimation when learner gives completed understanding or not understanding responses for all learned courseware during a learning process [27]. MLE will overestimate learner’s ability to the completed understanding case. On the other hand, MLE will underestimate learner’s ability to the completed not understanding case. Compared with the procedure of MLE,
Finally, the feedback agent transfers the learner’s understating percentage into the final fuzzy understanding degree and passes this information to the courseware recommendation agent to evaluate the learner’s ability and recommend suitable learning courseware to learner.
2.7 Courseware Recommendation Agent Based on the estimation of learner’s ability, this system can recommend the appropriate courseware to learners. 5
the procedure of Bayesian estimation is more complex and less efficient, but it can solve the problem of the divergent estimation for learner’s ability estimation in the procedure of MLE. Basically, a prior-information on the distribution of the learner’s abilities is employed here to estimate learner’s ability [27]. Hence, the Bayesian estimation procedure will always converge for all possible learners’ responses [27]. Based on this reason, the Bayesian estimation procedure is applied to estimate learner’s ability in this study. Bock and Mislevy [27] have given the quadrature form to approximately estimate learner’s ability as follows:
understanding case, moderately understanding case, and lowly understanding case, respectively. In this work, three fuzzy membership functions were defined as Eqn. (1) to (3) to support the evaluation of final fuzzy understanding degree U u (x) described as Eqn. (4). Based on the proposed evaluation method of final fuzzy understanding degree, the estimation value of new learner’s ability for the new learned courseware can be defined as follows: θ j + (θ w − θ j ) × U u ( x) when θ j +1 = θj when θ + (θ − θ ) × U ( x) when c j u j
q
θˆ =
∑θ
k L(u1, u 2 ,..., u n
number of
(6)
q
1
2 ,..., u n
where θˆ represents the learner’s ability of estimation, L(u1, u2 , L , un | θ k ) is the value of likelihood function at a level below their ability level θ k and learner’s responses are u1 , u 2 ,..., u n , θ k is the k th split value of ability in the standard normal distribution, and A(θ k ) represents the quadrature weight at a level below their ability level θ k .
Based on the definitions of Eqn. (4) and (8), if a learner replied that his understanding percentage for the content of learned courseware is below 40%, then U u (x) is equal to U l (x) . In this case, we think that he cannot understand the most part of learning contents of learned courseware, so his new estimation ability in this course unit should be descended according to the definition of Eqn. (8). That is, he needs to review easier courseware in order to promote his basic knowledge in this course unit. On the other hand, if a learner replied that his understanding percentage for the content of learned courseware is over 60%, then U u (x) is equal to U h (x) . In this case, we think that he can understand the great part of contents of learned courseware, so his new estimation ability in this course unit should be promoted according to the definition of Eqn. (8). Finally, if a learner replied that the understanding percentage for the content of learned courseware is between 40% and 60%, then U u (x) is equal to U m (x) . We think that learner’s response has lower confidence or too fuzzy, thus keeping his new estimation ability as the previous one. Therefore, our system will recommend other learning courseware with similar difficult level to learner. After evaluating the new learner ability, our system will evaluate the information of courseware based on the maximum information strategy to find out the most appropriate courseware for learner in order to do further studying.
In Eqn. (6), the likelihood function L(u1, u2 , L , un | θ k ) can be further described as follows: n
∏ P (θ ) j
k
uj
1−u j
Q j (θ k )
(7)
j =1
where P j (θ k ) =
e
j previous learned courseware, θ j +1 is
estimation value of new learner’s ability for the ( j + 1) th new courseware, θ w is estimation value of learner’s ability under assuming that learner cannot completely understand the ( j + 1) th new courseware, θ c is estimation value of learner’s ability under assuming that learner can completely understand the ( j + 1) th new courseware, and U u (x) is the final fuzzy understanding degree.
| θ k ) A(θ k )
k
L(u1, u2 ,L, un | θ k ) =
(8)
where θ j is estimation value of learner’s ability for total
| θ k ) A(θ k )
k
∑ L(u , u
x < 0.4 0.4 ≤ x ≤ 0.6 x > 0.6
D (θ k −b j )
, Q j (θ k ) = 1 − P j (θ k ) , Pj (θ ) D (θ −b ) 1+ e k j denotes the probability that learners can understand the
j th courseware at a level below their ability level θ k , Q j (θ k ) represents the probability that learners cannot
understand the j th courseware at a level below their ability level θ k , and U j is the answer of completely understanding or not understanding answer obtained from learner feedback to the j th courseware, i.e. if the answer is completely understanding then U j = 1 ; otherwise, U j = 0 . However, the Bayesian estimation procedure can just estimate learner’s ability for learner’s crisp responses, but it cannot estimate learner’s ability for learner’s fuzzy responses. In order to solve this problem, the fuzzy item response theory derived from the original item response theory is presented herein. The proposed fuzzy item response theory distinguishes the understanding degrees of learners into three various cases according to the experienced teachers’ decision. They are highly
2.7.2 Maximum Information Evaluation and Courseware Recommendation In IRT, there are two approaches that can be used to
6
As a learner logins this system, he can choose a course unit that he feels interested for learning. Figure 5 shows the entire layout of the learning interface. In the left frame, system shows the course categories, course units and the list of all courseware in the courseware database. While a learner clicks a courseware for learning, the content of selected courseware will be exhibited in the upper-right window. Besides, the feedback interface is arranged in the bottom-right window. System can get learner’s feedback response from the feedback in interface through learner replies two simple questions as listed in Table 1.
recommend appropriate courseware to learner. They are the maximum information strategy and Bayesian strategy [27-30]. The main concept of maximum information strategy is that each courseware with the corresponding difficulty parameter exhibits different information to learner’s learning. A courseware with higher information represents that it is more suitable to be recommended for learner. Since Bayesian strategy is more complicated than the maximum information, the maximum information method is applied to recommend appropriate courseware to learner. The maximum information function is defined as follows: (1.7) 2 (9) I (θ ) = j
[e
1.7 (θ −b j )
] [1 + e
Table 1. Two questionnaires for getting learner’s feedback information
]
−1.7 (θ −b j ) 2
where I j (θ ) is the information value of the
j th
courseware at a level below their ability level θ , b j is the difficulty parameter of the j th courseware. After calculating the corresponding information values of all courseware, the course recommendation agent can recommend a series of courseware to learners with ability θ according to the ranking order of information function value. A courseware with the maximum information function value under learner with ability θ indicates that the system presented here gives the highest recommendation priority. Whether learner accepts the recommended courseware with highest recommendation priority or selects the other recommended courseware to do further learning, our system will record learner’s learning paths and learner’s feedback responses into the user profile database during a learning process. These information can provide system to develop more intelligent agents for personalized learning, such as learning process analysis and learning diagnosis etc. Based on this vision, some minded tools which can assist personalized learning are also considered in our future research.
Figure 5. The learning interface for learner The answer of question one can be served as an investigation of learner’s satisfactory degree for the recommended courseware. The 5-point Likert-scale proposed by Likert in 1932 [39] is applied to define various scaled answers. In a variation of standard Likertscale, this study uses a scale where -2 indicates “very easy”, -1 is “easy”, 0 is “moderate”, 1 is “hard” and 2 is “very hard”. If a learner feels that the recommended courseware is quite suitable for him, then the averaged value of his answers should be very close to zero, i.e. “Moderate”. The answer of question two helps system to get the learner’s understanding percentage for the recommended courseware. System passes this information to the courseware recommendation agent to evaluate the learner’s ability. After a learner presses the button of analysis, this system will reveal a list of the recommended courseware based on his current ability. Figure 6 shows an example of courseware recommendation based on learner ability after learner gives corresponding feedback response, and the recommended courseware ranked by the order of their information values. The title (標題) indicates the subject of the courseware; the recommendation (推薦指數) denotes
3. Experiments To verify the learning efficiency and effectiveness for the proposed courseware recommendation system, the prototype of the system has been successfully implemented and some university students who have majored in the course of C language programming were invited to test this system. Currently, the URL address of the web site is available at http://203.64.142.238:5647. The detailed functions of this system and experimental results are described as follows.
3.1 System’s Functions At present, the proposed prototype is implemented on the platform of Microsoft Windows 2000 with IIS 5.0 Web server. Moreover, the front-end script language of PHP 4.3 and MySQL server are used to implement this system. 7
the information value of the recommended courseware; and the description (描述) gives a brief description for the corresponding courseware. The length of bar line in the column of recommendation indicates the information value of the corresponding courseware. The longer bar line implies a more suitable courseware for learner. On the contrary, the shorter bar line implies an unsuitable courseware for learner.
ability described in Eqn. (8). On the other hand, if a learner replied that his understanding percentage for the current courseware below 40%, his estimation ability will be descended. In the 12th and 14th ability estimation, learner’s ability is same as the previous one because the understanding percentage of learner’s response for the learned courseware is between 40% and 60%. Besides, Figure 8 shows the relationship of the learner ability with the difficulty parameter of the recommended courseware. We find that the difficulty parameter of the recommended courseware is highly relevant with the learner ability. This result shows that the proposed system indeed can recommend appropriate courseware to learners according to different learner abilities. Table 2. The corresponding difficulty of each courseware
Title of courseware
Difficulty
Special operation of e1 ? e2 : e3 Basic input/output functions and applications Else - if instruction
-2.738
If - else control instruction
-2.29
Switch instruction
0.526
3.2 Experimental Results
Relationship and logic operators (三)
1.001
Currently, this system only contains a small amount of courseware because to produce high quality courseware needs a large amount of manpower to join this work. In our experiments, C language programming is selected as a course unit to provide personalized learning services. All designed courseware of C language programming and their corresponding difficulty parameters determined by statistics-based method are listed in Table 2. Actually, to expand the courseware database and supply more abundantly courseware for learners are urgently needed in the future. Figure 7 shows the relationship between the understanding percentage of the clicked courseware and the adjustment of the learner’s ability. In this system, the range of learner’s ability is limited from -3 (i.e. lowest ability) to 3 (i.e. highest ability). As a learner logins this system, if the user account database does not have any history records in the selected course unit for this learner, then his initial ability will be regarded as 0. That is, the system assumes learner’s ability is middle level. As a learner clicked the recommended courseware for learning, his ability in this course unit will be re-evaluated according to his response of understanding percentage and the corresponding difficulty parameter of the learned courseware. In Figure 7, we can find that the trend of learner’s estimation ability follows his response of understanding percentage for the learned courseware. If learner replied that his understanding percentage for the current courseware is over 60%, his ability estimation will be promoted based on the estimation formula of learner’s
# define and constant instructions
1.081
C language introduction
1.081
For loop introduction
2.554
Do - while loop introduction
3.559
Relationship and logic operators (二)
4.378
Basic bit operation
4.496
Special operations in C language (一)
5.128
Basic data type and mathematics operations
5.128
Relationship and logic operators (一)
6.173
Definition of functions
6.521
Flow control and special operations
6.613
Introduction of printf () instruction
6.706
Figure 6. An example of courseware recommendation ranked by the ranking order of information values
Basic data type Function’s applications
8
-4.804
-2.461
6.9 7
Relationship and logic operators (四)
7.778
One dimension array
10.297
Special operations in C language (二)
16.71
services more intimately, some minded tools such as personalized learning diagnosis agent, personalized courseware retrieval agent, personalized learning-path analysis agent will be considered in our future research.
Ability & The understanding percentage of clicked courseware Understanding percentage
Ability
Ability
Understanding percentage
3
100% 90% 90% 80%
2
80% 80% 80%
70%
70%
70%
References
70% 70% 70% 70%
1
60%
60% 60%
0
[1]
Berghel H., “Cyberspace 2000: Dealing with Information overload,” Communications of the ACM, vol. 40, no. 2, pp.19-24, Feb. 1997.
[2]
Borchers A., Herlocker J., Konstanand J., Riedl J., “Ganging Up on Information Overload,” Computer, vol. 31, issue: 4, pp. 106-108, Apr. 1998.
[3]
Arasu A., Cho J., Garcia-Molina H., Paepcke A., Raghavan S., “Searching the Web,” ACM Transactions on Internet Technology, vol. 1, issue: 1, pp. 97-101, 2001.
[4]
Kobayashi M., Takeda K., “Information Retrieval on the Web,” ACM Computing Surveys, vol. 32, no. 2, pp. 144-173, 2000.
[5]
Lawrence S., Giles C. L., “Searching the Web: General and Scientific Information Access,” IEEE Communications, vol. 37, pp. 116-122, 1999.
[6]
Brin S., Page, L., “The PageRank Citation Ranking: Bringing Order to the Web,” In Proceedings of ASIS’98, pp. 161-172, 1998.
[7]
Chidlovskii B., Glance N., Grasso A., “Collaborative Re-Ranking of Search Results,” In Proceedings of AAAI-2000 Workshop on Artificial Intelligence for Web Search, Austin, Texas, pp. 18-22, 2000.
[8]
Kleinberg J., “Authoritative Sources in a Hyperlinked Environment,” In Proceedings of 9th ACM-SIAM Symposium on Discrete Algorithms, pp. 668-679, 1998.
[9]
Direct Hit, 2000, http://www.directhit.com.
50% 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17 40%
-1 30%
30%
30% 25%
-2
20%
20%
20%
10% -3
0%
Figure 7. The relationship between the understanding percentage of the clicked courseware and the adjustment of the learner’s ability Ability & The difficulty of recommendation courseware Difficulty
Difficulty
Ability
Ability
8
3
2.5
6
2 4 1.5 2 1 0 1
2
3
4
-2
-4
5
6
7
8
9
10
11
12
13
14
15
16
17
0.5
0
-0.5
Figure 8. The relationship between learner’s ability and the difficulty parameter of the recommended courseware
4. Conclusion This study proposes a personalized courseware recommendation system (PCRS) based on the proposed fuzzy item response theory (FIRT), which can estimate the abilities of online learners and recommend appropriate courseware to learners. Compared to the traditional item response theory, the fuzzy item response theory can accept non-crisp response to correctly estimate learner’s ability via the revised estimating function of learner’s ability. PCRS provides personalized Web-based learning according to courseware visited by learners and their responses. Moreover, courseware difficulty can also be correctly determined by the proposed courseware modeling process. Experimental results show that the proposed system can precisely provide personalized courseware recommendations on-line based on learner abilities and responses, and moreover can accelerate learner learning efficiency and effectiveness. Importantly, learners only need to reply two simple questions for personalized services. Besides, to provide personalized e-learning
Available
at
[10] Google, Available at http://www.google.com. [11] NEC Research Institute ResearchIndex, Available at http://citeseer.nj.nec.com. [12] Brusilovsky P., “Adaptive and Intelligent Technologies for Web-based Education,” In Rollinger, C.; Peylo, C. (eds.), Special Issue on Intelligent Systems and Teleteaching, Künstliche Intelligenz, vol. 4, pp. 19-25, 1999. [13] Kao T. T., Personalization Courseware Navigation Using Grey Ranking Analysis, Master’s thesis, Department of Electronic Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, 2001.
9
[14] Khan B. H., Web Based Instruction, Englewood Cliffs, New Jersey: Educational Technology Publications, 1997.
[26] Brusilovsky P., “Adaptive and Intelligent Technologies for Web-based Education,” In Rollinger, C.; Peylo, C. (eds.), Special Issue on Intelligent Systems and Teleteaching, Künstliche Intelligenz, vol. 4, pp. 19-25, 1999.
[15] Lee Y. J., Chiu Y. W., Hong C. F., “Intelligent Adaptive Teaching Inference of Virtual College,” Distance Education, vol. 15/16, pp. 85-97, 2000. (in Chinese)
[27] Baker and Frank B., Item Response Theory: Parameter Estimation Techniques, New York: Marcel Dekker, 1992.
[16] Lin L. C., Chen K. S., Chen Y. L., Jao P. L., Tsai C. C., “The Application of Adaptive Multi-Agent Based Architecture in Distance Learning Environment,” TANET, 2001, Available at http://www.ccu.edu.tw/TANET2001/scheduel/paper _abs/J124.html.
[28] Hambleton, R. K., Item Response Theory: Principles and Applications, Boston, Kluwer-Nijhoff Publisher, 1985. [29] Hulin, C. L., Drasgow, F., Parsons and C. K., Item Response Theory: Application to Psychological Measurement, Homewood, IL: Dow Jones Irwin, 1983.
[17] Lin H. N., An Adaptive Learning Environment: A Case Study on High School Mathematics, Master’s thesis, Department of Computer and Information Science, National Chiao Tung University, Hsinchu, Taiwan, 2001.
[30] Hsu, T. C. and Liu C. H., Introduction to Item Response Theory, Taipei, Taiwan, Chinese Behavioral Science Corporation, 1991.
[18] Hong C. F, Fu M. Y., Chen W. H., Yang S. C., Chang C. Y., Chang C. Y., Peng S. C., Liu Y. T., Mo S. M., “Constructing an Suitable Teaching Environment on the World Wide Web,” Distance Education, vol. 13/14, pp. 105-110, 2000. (in Chinese)
[31] Horward W., Computerized Adaptive Testing: A Primer, Hillsdale, New Jersey: Lawerence Erwrence Erlbaum Associates, 1990. [32] Hsu, T. C. and Sadock, S. F., Computer-assisted Test Construction: A State of Art, TME report 88, Princeton, New Jersey, Eric on Test, Measurement, and Evaluation, Educational Testing Service, 1985.
[19] Chen J. L., A Multiagent-based Intelligent Instruction Agent, Master’s thesis, Department of Information Engineering and Computer Science, Feng Chia University, Taichung, Taiwan, 1999.
[33] TOEFL, Avaliable at http://www.toefl.org. [34] GRE, Avaiable at http://www.gre.org.
[20] Yang T. L., A Study of Dynamical Adaptive Learning System, Master’s thesis, National Kaohsiung Normal University, Kaohsiung, Taiwan, 2001.
[35] GMAT, Available at http://www.gmat.org. [36] H.-J. Zimmermann, Fuzzy Sets, Decision Making, and Expert System, Boston: Kluwer Academic Publishers, 1987.
[21] Liu M. Z., Chen L. C., Shou D. W., Lin H. L., “A Study on Adaptive Learning Design in Web-based Courseware,” MOE of ROC research paper, 1999.
[37] Hahn-Ming Lee, Chih-Ming Chen and Ya-Hui Chen, “Personalized E-Learning System Utilizing Item Response Theory,” International Computer Symposium, Workshop on Artificial Intelligence, Dong Hwa University, Hualien, Taiwan, 2002.
[22] Mobasher B., Cooley R., Srivastava J., “Automatic Personalization Based on Web Usage Mining,” Communications of the ACM, vol. 43, no. 8, pp. 142151, August 2000.
[38] Advanced Distributed http://www.adlnet.org/.
[23] Kao T. T., Personalization Courseware Navigation Using Grey Ranking Analysis, Master’s thesis, Department of Electronic Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, 2001.
Learning,
available
at
[39] Likert R., A Technique for the Measurement of Attitudes, New York: Archives of Psychology, 1932.
[24] Balabanovic M., Shoham Y., “Fab: Content-based, Collaborative Recommendation,” Communications of the ACM, vol. 40, no. 3, pp. 66-72, March 1997. [25] Fu X., Budzik J., Hammond J. K., “Mining Navigation History for Recommendation,” Proceedings of the 2000 International Conference on Intelligent User Interfaces, pp. 106-112, 2000.
10
