View Presentation Slides
TRACE: A Dynamic Model of Trust for People-Driven Service Engagements Combining Trust with Risk, Commitments, and Emotions
Anup K. Kalia Advisor: Munindar P. Singh Department of Computer Science North Carolina State University Raleigh, NC 27695, USA
September 30, 2015
Anup Kalia (NCSU)
TRACE
September 30, 2015
1 / 24
Broader Objectives I I
Understand subtle human and organizational relationships Use such relationships as a basis for estimating trust ŽƌƉŽƌĂƚĞ
DŝůůŝƚĂƌLJ
KZ'E/d/KE /Z dKZ
d D> Z
ŵĂŝů
ŚĂƚ
ŵĂŝů
d DD D Z
Anup Kalia (NCSU)
TRACE
September 30, 2015
2 / 24
Research Question I
How to estimate trust between people from their interactions?
Possible Applications I
Support people to make important decisions in organizational settings
I
Estimating team cohesion or performance Anup Kalia (NCSU)
TRACE
September 30, 2015
3 / 24
Limitations With Existing Approaches Several approaches consider commitments alone for trust estimation. I
Gambetta (1988) interprets trust as a truster’s assessment of a trustee for performing a specific task
I
Mayer et al. (1995) define trust as the willingness of a truster to be vulnerable to a trustee for the completion of a task
I
Teacy et al. (2006) consider trust as the truster’s estimation of probability that a truster will fulfill it’s obligation toward a trustee
I
Wang et al. (2011) represent trust as the belief of a truster that trustee will cooperate. They estimate trust by aggregating positive and negative experiences
I
Kalia et al. (2014) consider commitment outcomes to predict trust where they learn truster’s parameters based on whether outcomes are positive, negative, or neutral
Anup Kalia (NCSU)
TRACE
September 30, 2015
4 / 24
Limitations With Existing Approaches
Two major classes of trust models I
Fixed parameter trust models where parameter are manually fixed
I
Machine-learned trust models typically Hidden Markov Models (HMM) that assumes variables are conditionally independent of each other given the output variable
Anup Kalia (NCSU)
TRACE
September 30, 2015
5 / 24
Proposed Approach
We can improve trust prediction by incorporating (in addition to commitments) two attributes I
Risk taken by a truster toward a trustee I
I
Emotions displayed by a truster toward a trustee I
I
Risk taken depends on a truster’s belief about the likelihood of gains or losses it might incur from its relationships with a trustee Studies in psychology suggest that positive emotions increase trust whereas negative emotions decrease trust
Create TRACE a model based on Conditional Random Field (CRF) I
Conditional independences between risk, commitments, and emotions may not hold in our setting (e.g., in HMM)
Anup Kalia (NCSU)
TRACE
September 30, 2015
6 / 24
Background: Commitment & Trust DE' Z
WůĞĂƐĞĚĞůŝǀĞƌƚŚĞĐŽĚĞ ďLJDŽŶĚĂLJ
s >KW Z
zĞƐ͕/ǁŝůů ŽŵŵŝƚŵĞŶƚƌĞĂƚŝŽŶ ,ĞƌĞŝƐƚŚĞĐŽĚĞ ŽŵŵŝƚŵĞŶƚŝƐĐŚĂƌŐĞ dƌƵƐƚ/ŶĐƌĞĂƐĞƐ KZ ^ŽƌƌLJ͕/ĐŽƵůĚŶŽƚĐŽŵƉůĞƚĞ ŽŵŵŝƚŵĞŶƚsŝŽůĂƚŝŽŶ dƌƵƐƚĞĐƌĞĂƐĞƐ
Anup Kalia (NCSU)
TRACE
September 30, 2015
7 / 24
Background: Commitment Lifecycle I
C(Debtor, Creditor, Antecedent, Consequent)
null create expire ĂĐƚŝǀĞ conditional
cancel terminated
Anup Kalia (NCSU)
ĂŶƚĞĐĞĚĞŶƚ
consequent discharged
TRACE
detached
cancel violated
September 30, 2015
8 / 24
Background: Estimating Trust from Commitment Progression I
Two-valued representation, positive and negative experiences: hr , si
Trust α = r +r s I We characterize each subject via four parameters I Initial values, hr , s i in in I
I
Increment for positive and negative experiences: ir and is
Commitment Operation Commissive create Directive create Delegate None
hλir + rin , (1-λ)is + sin i
hir + rin , sin i hrin , is + sin i
Discharge Cancel
Anup Kalia (NCSU)
Trust hrfi , sfi i
TRACE
September 30, 2015
9 / 24
Trust Antecedent Framework (Mayer et al., 1995) We propose TRACE based on the enhance trust antecedent framework ďŝůŝƚLJ
ĞŶĞǀŽůĞŶĐĞ
dƌƵƐƚ
ŽŵŵŝƚŵĞŶƚƐ
ZŝƐŬ
/ŶƚĞŐƌŝƚLJ ŵŽƚŝŽŶƐ
I
The model contains 4 variables trust (T), risk (R), commitments (C), and emotions (E)
I
Each variable V = hT, R, C, Ei is described using using Singh’s (1999, 2011) formal notation Vhdebtor, creditor, antecedent, consequenti Anup Kalia (NCSU)
TRACE
September 30, 2015
10 / 24
Description of Variables I
Chtrustee, truster, antecedent, consequenti I I
I
Rhtruster, trustee, antecedent, consequenti I
I
I
The truster takes a risk by accepting the trustee’s offer to perform the consequent If the trustee performs the consequent, the truster gains
Thtruster, trustee, antecedent, consequenti I I
I
The trustee commits to the truster to perform the consequent If the trustee performs the consequent, the commitment is satisfied
The truster believes the trustee if the trustee performs the consequent Trust has three dimensions: ability (trustee’s competency), benevolence (trustee’s willingness), integrity (trustee’s ethics and morality)
Ehtruster, trustee, antecedent, consequenti I
The truster displays a positive emotion if the trustee performs the consequent
Anup Kalia (NCSU)
TRACE
September 30, 2015
11 / 24
Postulates We propose postulates that capture relationships between the variables P1 : Tt → Tt +1 . The trust Tt +1 is influenced by the past trust Tt P2 : Ct → Tt . The current commitment outcome Ct influences the current trust Tt P3 : Rt → Ct . The risk taken influences the commitment outcome Ct or the gain or loss realized in the risk Rt P4 : Rt → Tt . The current risk taken Rt influences the current trust Tt P5 : Ct → Et . The commitment outcome Ct influences the current emotion Et P6 : Rt → Et . The risk taken Rt influences the truster’s emotion Et P7 : Et → Tt . The current emotion Et influences the current trust Tt
Anup Kalia (NCSU)
TRACE
September 30, 2015
12 / 24
The TRACE Model Graphical representation of HMM and TRACE trust models (two time slices)
,DD
d
d
ƚ
ƚ
ƚнϭ
Z
ƚ
ƚ
ƚнϭ
Z
ƚнϭ
ƚнϭ
dZ
d
d
ƚ
ƚ
Anup Kalia (NCSU)
ƚнϭ
Z
ƚ
ƚ
ƚнϭ
TRACE
Z
ƚнϭ
ƚнϭ
September 30, 2015
13 / 24
Comparing HMM and CRF
I
HMM makes two independent assumptions I I
I
The current state yt is independent of y1 , y2 , . . ., yt −2 , given yt −1 Observations xt are independent of each other, given yt
CRF I I
CRFs are agnostic to dependencies between the observations CRF model employs discriminative modeling, where the distribution p(~y |~x ) is learned directly from the data
Anup Kalia (NCSU)
TRACE
September 30, 2015
14 / 24
Evaluation We evaluate TRACE via data collected from a human-subject study conducted by the Intelligence Advanced Research Projects Activity (IARPA) IARPA prepared a dataset based on the Checkmate protocol adapted from the investment or dictator economic decision-making game (Berg, 1995) I The data consists of 431 rows collected from 63 subjects I Each row corresponds to the sequence of rounds played between two subjects I The data we obtained reflects only the banker’s perspective I
^ĞƐƐŝŽŶϭ WĂƌƚŶĞƌƐ WĂƌƚŶĞƌƐ ^ĞƐƐŝŽŶϮ WĂƌƚŶĞƌƐ WĂƌƚŶĞƌƐ Anup Kalia (NCSU)
WĂƌƚŝĐŝƉĂŶƚ
WĂƌƚŝĐŝƉĂŶƚϭΖƐZŽůĞ͗ WĂƌƚŝĐŝƉĂŶƚϮΖƐZŽůĞ͗ WĂƌƚŝĐŝƉĂŶƚϯΖƐZŽůĞ͗ WĂƌƚŝĐŝƉĂŶƚϰΖƐZŽůĞ͗
WĂƌƚŝĐŝƉĂŶƚ WĂƌƚŝĐŝƉĂŶƚϭΖƐZŽůĞ͗ WĂƌƚŝĐŝƉĂŶƚϯΖƐZŽůĞ͗ WĂƌƚŝĐŝƉĂŶƚϮΖƐZŽůĞ͗ WĂƌƚŝĐŝƉĂŶƚϰΖƐZŽůĞ͗ TRACE
ůŽĐŬϭ ZŽƵŶĚϭͲϱ
ĂŶŬĞƌ 'ĂŵĞWůĂLJĞƌ 'ĂŵĞWůĂLJĞƌ ĂŶŬĞƌ
ůŽĐŬϭ ZŽƵŶĚϭͲϱ
'ĂŵĞWůĂLJĞƌ ĂŶŬĞƌ ĂŶŬĞƌ 'ĂŵĞWůĂLJĞƌ
ůŽĐŬϮ ZŽƵŶĚϲͲϭϬ 'ĂŵĞWůĂLJĞƌ ĂŶŬĞƌ ĂŶŬĞƌ 'ĂŵĞWůĂLJĞƌ
ůŽĐŬϮ ZŽƵŶĚϲͲϭϬ ĂŶŬĞƌ 'ĂŵĞWůĂLJĞƌ 'ĂŵĞWůĂLJĞƌ ĂŶŬĞƌ
September 30, 2015
15 / 24
Evaluation I I I
I I I I I I
The protocol involves two roles: banker and game player The banker’s task is to loan money to a game player The game player requests a loan from the banker to play a maze, promising to play a maze of certain difficulty and return: the loan with all gains, the loan with 50% of all gains, 50% of the available money, a fixed amount After the game player’s request, the banker chooses a loan category: small (1–7 USD), medium (4–10 USD), or big (7–13 USD) A dollar amount, randomly generated within the banker’s chosen category, is loaned to the game player The game player does not know the category chosen by the banker The game player plays a maze of a certain difficulty (not necessarily what he or she had promised) The banker will not know the actual maze played The game player returns some money to the banker (not necessarily what he or she had promised) Anup Kalia (NCSU)
TRACE
September 30, 2015
16 / 24
Mapping Game Elements to The TRACE Model
I
The commitment from the player to the banker, Chplayer,banker,loan,returni, as satisfied if the player returned at least the amount he or she had loaned, and as violated, otherwise
I
We compute the gain or loss in the risk, Rh banker, player, loan, returni, based on the difference between the loaned and returned amounts
I
The dataset represents the banker’s trust for the player after the round, Thbanker,player,loan,returni, as a three-tuple hA, B, Ii, indicating the banker’s perception of player’s ability, benevolence, and integrity
I
The dataset represents the banker’s emotion after he or she receives a return from the player, Ehbanker,player,loan,returni, as real-valued (1–10) state anxiety scores derived from the post-round questionnaire
Anup Kalia (NCSU)
TRACE
September 30, 2015
17 / 24
Results MAEs of HMM and TRACE considering different feature combinations. Model HMM
TRACE
Anup Kalia (NCSU)
Input Variables
A
B
I
C C+R C+E R+E C+R+E
1.1220 0.8974 0.8619 0.8468 0.8870
0.8564 0.7655 0.7433 0.8376 0.7977
1.0917 0.8484 0.7184 0.7992 0.7714
C C+R C+E R+E C+R+E
0.8744 0.7463 0.8617 0.8949 0.7878
0.7576 0.7685 0.7656 0.6815 0.7427
0.7988 0.7876 0.7580 0.6568 0.7141
TRACE
September 30, 2015
18 / 24
Results I
Considering only C, TRACE yields lower MAEs than HMM for each trust attribute (CRF employs discriminative)
I
Considering all features (C + R + E), TRACE again yields lower MAEs than HMM for each trust attribute (CRF captures dependencies between C, R, and E)
I
Considering C and R, TRACE performs better than HMM in predicting A and I (C and R are not independent)
I
Considering C and E, HMM performs better than TRACE for B and I (C and E are conditionally independent)
I
Considering R and E, TRACE performs better than HMM for B and I whereas HMM performs better than TRACE for A (mixed)
Anup Kalia (NCSU)
TRACE
September 30, 2015
19 / 24
Threats to Validity
I
Our dataset, although real, consists of short sequences. We expect both HMM and TRACE to perform better given longer sequences
I
The dataset is skewed toward positive trust values and our conclusions may not hold since the trust values have a different distribution
I
The dataset represents emotions using anxiety scores only, thereby lacking realistic emotion responses along multiple dimensions such as anger and joy
Anup Kalia (NCSU)
TRACE
September 30, 2015
20 / 24
Discussion
I
TRACE illustrates that a probabilistic model of trust that incorporates commitments, risk, and emotions can produce trust estimates with fairly good accuracy
I
Our findings therefore open up the possibility of developing user agents that promote secure collaboration
I
Using TRACE a user can calibrate the perceived trust with the risk undertaken in light of available measures of risk and gain from commitments
Anup Kalia (NCSU)
TRACE
September 30, 2015
21 / 24
Thanks
Anup Kalia (NCSU)
TRACE
September 30, 2015
22 / 24
Background: Identifying Commitment Operations from Interactions Ten-fold cross-validation using SVM on marked up Enron email sentences
Commitment Operation
Precision
Recall
F-measure
0.87 0.94 0.86 1 0
0.97 0.97 0.33 0.02 0
0.92 0.95 0.48 0.04 0
Commissive create Directive create Delegate Discharge Cancel
Features used in the classifier include (out of 15) 1 2 3 4 5
Modal verb (shall, will, may, might, can, could, would, must) Type of subject (first person, second person, third person) Present tense verb Past tense verb Deadline
Anup Kalia (NCSU)
TRACE
September 30, 2015
23 / 24
Background: Determining Commitment Operations and Trust from Text Commitments being the most prominent normative relationship
S
R
Content
Operation
Kim
Dorothy
create(C1 )
Kim
Dorothy
Rob
Kim
Kim
Rob
I will also check with Alliance Travel Agency . . . I checked with our Travel Agency . . . By Wednesday Aug 16 2001, please send all copies of your documentation . . . Rob, please forgive me for not sending this in by Aug 15
I
T S ,R
discharge(C1 )
T R ,S
↑
create(C2 )
↓
cancel(C2 )
Example emails from the Enron corpus Anup Kalia (NCSU)
TRACE
September 30, 2015
24 / 24
Anup K. Kalia Advisor: Munindar P. Singh Department of Computer Science North Carolina State University Raleigh, NC 27695, USA
September 30, 2015
Anup Kalia (NCSU)
TRACE
September 30, 2015
1 / 24
Broader Objectives I I
Understand subtle human and organizational relationships Use such relationships as a basis for estimating trust ŽƌƉŽƌĂƚĞ
DŝůůŝƚĂƌLJ
KZ'E/d/KE /Z dKZ
d D> Z
ŵĂŝů
ŚĂƚ
ŵĂŝů
d DD D Z
Anup Kalia (NCSU)
TRACE
September 30, 2015
2 / 24
Research Question I
How to estimate trust between people from their interactions?
Possible Applications I
Support people to make important decisions in organizational settings
I
Estimating team cohesion or performance Anup Kalia (NCSU)
TRACE
September 30, 2015
3 / 24
Limitations With Existing Approaches Several approaches consider commitments alone for trust estimation. I
Gambetta (1988) interprets trust as a truster’s assessment of a trustee for performing a specific task
I
Mayer et al. (1995) define trust as the willingness of a truster to be vulnerable to a trustee for the completion of a task
I
Teacy et al. (2006) consider trust as the truster’s estimation of probability that a truster will fulfill it’s obligation toward a trustee
I
Wang et al. (2011) represent trust as the belief of a truster that trustee will cooperate. They estimate trust by aggregating positive and negative experiences
I
Kalia et al. (2014) consider commitment outcomes to predict trust where they learn truster’s parameters based on whether outcomes are positive, negative, or neutral
Anup Kalia (NCSU)
TRACE
September 30, 2015
4 / 24
Limitations With Existing Approaches
Two major classes of trust models I
Fixed parameter trust models where parameter are manually fixed
I
Machine-learned trust models typically Hidden Markov Models (HMM) that assumes variables are conditionally independent of each other given the output variable
Anup Kalia (NCSU)
TRACE
September 30, 2015
5 / 24
Proposed Approach
We can improve trust prediction by incorporating (in addition to commitments) two attributes I
Risk taken by a truster toward a trustee I
I
Emotions displayed by a truster toward a trustee I
I
Risk taken depends on a truster’s belief about the likelihood of gains or losses it might incur from its relationships with a trustee Studies in psychology suggest that positive emotions increase trust whereas negative emotions decrease trust
Create TRACE a model based on Conditional Random Field (CRF) I
Conditional independences between risk, commitments, and emotions may not hold in our setting (e.g., in HMM)
Anup Kalia (NCSU)
TRACE
September 30, 2015
6 / 24
Background: Commitment & Trust DE' Z
WůĞĂƐĞĚĞůŝǀĞƌƚŚĞĐŽĚĞ ďLJDŽŶĚĂLJ
s >KW Z
zĞƐ͕/ǁŝůů ŽŵŵŝƚŵĞŶƚƌĞĂƚŝŽŶ ,ĞƌĞŝƐƚŚĞĐŽĚĞ ŽŵŵŝƚŵĞŶƚŝƐĐŚĂƌŐĞ dƌƵƐƚ/ŶĐƌĞĂƐĞƐ KZ ^ŽƌƌLJ͕/ĐŽƵůĚŶŽƚĐŽŵƉůĞƚĞ ŽŵŵŝƚŵĞŶƚsŝŽůĂƚŝŽŶ dƌƵƐƚĞĐƌĞĂƐĞƐ
Anup Kalia (NCSU)
TRACE
September 30, 2015
7 / 24
Background: Commitment Lifecycle I
C(Debtor, Creditor, Antecedent, Consequent)
null create expire ĂĐƚŝǀĞ conditional
cancel terminated
Anup Kalia (NCSU)
ĂŶƚĞĐĞĚĞŶƚ
consequent discharged
TRACE
detached
cancel violated
September 30, 2015
8 / 24
Background: Estimating Trust from Commitment Progression I
Two-valued representation, positive and negative experiences: hr , si
Trust α = r +r s I We characterize each subject via four parameters I Initial values, hr , s i in in I
I
Increment for positive and negative experiences: ir and is
Commitment Operation Commissive create Directive create Delegate None
hλir + rin , (1-λ)is + sin i
hir + rin , sin i hrin , is + sin i
Discharge Cancel
Anup Kalia (NCSU)
Trust hrfi , sfi i
TRACE
September 30, 2015
9 / 24
Trust Antecedent Framework (Mayer et al., 1995) We propose TRACE based on the enhance trust antecedent framework ďŝůŝƚLJ
ĞŶĞǀŽůĞŶĐĞ
dƌƵƐƚ
ŽŵŵŝƚŵĞŶƚƐ
ZŝƐŬ
/ŶƚĞŐƌŝƚLJ ŵŽƚŝŽŶƐ
I
The model contains 4 variables trust (T), risk (R), commitments (C), and emotions (E)
I
Each variable V = hT, R, C, Ei is described using using Singh’s (1999, 2011) formal notation Vhdebtor, creditor, antecedent, consequenti Anup Kalia (NCSU)
TRACE
September 30, 2015
10 / 24
Description of Variables I
Chtrustee, truster, antecedent, consequenti I I
I
Rhtruster, trustee, antecedent, consequenti I
I
I
The truster takes a risk by accepting the trustee’s offer to perform the consequent If the trustee performs the consequent, the truster gains
Thtruster, trustee, antecedent, consequenti I I
I
The trustee commits to the truster to perform the consequent If the trustee performs the consequent, the commitment is satisfied
The truster believes the trustee if the trustee performs the consequent Trust has three dimensions: ability (trustee’s competency), benevolence (trustee’s willingness), integrity (trustee’s ethics and morality)
Ehtruster, trustee, antecedent, consequenti I
The truster displays a positive emotion if the trustee performs the consequent
Anup Kalia (NCSU)
TRACE
September 30, 2015
11 / 24
Postulates We propose postulates that capture relationships between the variables P1 : Tt → Tt +1 . The trust Tt +1 is influenced by the past trust Tt P2 : Ct → Tt . The current commitment outcome Ct influences the current trust Tt P3 : Rt → Ct . The risk taken influences the commitment outcome Ct or the gain or loss realized in the risk Rt P4 : Rt → Tt . The current risk taken Rt influences the current trust Tt P5 : Ct → Et . The commitment outcome Ct influences the current emotion Et P6 : Rt → Et . The risk taken Rt influences the truster’s emotion Et P7 : Et → Tt . The current emotion Et influences the current trust Tt
Anup Kalia (NCSU)
TRACE
September 30, 2015
12 / 24
The TRACE Model Graphical representation of HMM and TRACE trust models (two time slices)
,DD
d
d
ƚ
ƚ
ƚнϭ
Z
ƚ
ƚ
ƚнϭ
Z
ƚнϭ
ƚнϭ
dZ
d
d
ƚ
ƚ
Anup Kalia (NCSU)
ƚнϭ
Z
ƚ
ƚ
ƚнϭ
TRACE
Z
ƚнϭ
ƚнϭ
September 30, 2015
13 / 24
Comparing HMM and CRF
I
HMM makes two independent assumptions I I
I
The current state yt is independent of y1 , y2 , . . ., yt −2 , given yt −1 Observations xt are independent of each other, given yt
CRF I I
CRFs are agnostic to dependencies between the observations CRF model employs discriminative modeling, where the distribution p(~y |~x ) is learned directly from the data
Anup Kalia (NCSU)
TRACE
September 30, 2015
14 / 24
Evaluation We evaluate TRACE via data collected from a human-subject study conducted by the Intelligence Advanced Research Projects Activity (IARPA) IARPA prepared a dataset based on the Checkmate protocol adapted from the investment or dictator economic decision-making game (Berg, 1995) I The data consists of 431 rows collected from 63 subjects I Each row corresponds to the sequence of rounds played between two subjects I The data we obtained reflects only the banker’s perspective I
^ĞƐƐŝŽŶϭ WĂƌƚŶĞƌƐ WĂƌƚŶĞƌƐ ^ĞƐƐŝŽŶϮ WĂƌƚŶĞƌƐ WĂƌƚŶĞƌƐ Anup Kalia (NCSU)
WĂƌƚŝĐŝƉĂŶƚ
WĂƌƚŝĐŝƉĂŶƚϭΖƐZŽůĞ͗ WĂƌƚŝĐŝƉĂŶƚϮΖƐZŽůĞ͗ WĂƌƚŝĐŝƉĂŶƚϯΖƐZŽůĞ͗ WĂƌƚŝĐŝƉĂŶƚϰΖƐZŽůĞ͗
WĂƌƚŝĐŝƉĂŶƚ WĂƌƚŝĐŝƉĂŶƚϭΖƐZŽůĞ͗ WĂƌƚŝĐŝƉĂŶƚϯΖƐZŽůĞ͗ WĂƌƚŝĐŝƉĂŶƚϮΖƐZŽůĞ͗ WĂƌƚŝĐŝƉĂŶƚϰΖƐZŽůĞ͗ TRACE
ůŽĐŬϭ ZŽƵŶĚϭͲϱ
ĂŶŬĞƌ 'ĂŵĞWůĂLJĞƌ 'ĂŵĞWůĂLJĞƌ ĂŶŬĞƌ
ůŽĐŬϭ ZŽƵŶĚϭͲϱ
'ĂŵĞWůĂLJĞƌ ĂŶŬĞƌ ĂŶŬĞƌ 'ĂŵĞWůĂLJĞƌ
ůŽĐŬϮ ZŽƵŶĚϲͲϭϬ 'ĂŵĞWůĂLJĞƌ ĂŶŬĞƌ ĂŶŬĞƌ 'ĂŵĞWůĂLJĞƌ
ůŽĐŬϮ ZŽƵŶĚϲͲϭϬ ĂŶŬĞƌ 'ĂŵĞWůĂLJĞƌ 'ĂŵĞWůĂLJĞƌ ĂŶŬĞƌ
September 30, 2015
15 / 24
Evaluation I I I
I I I I I I
The protocol involves two roles: banker and game player The banker’s task is to loan money to a game player The game player requests a loan from the banker to play a maze, promising to play a maze of certain difficulty and return: the loan with all gains, the loan with 50% of all gains, 50% of the available money, a fixed amount After the game player’s request, the banker chooses a loan category: small (1–7 USD), medium (4–10 USD), or big (7–13 USD) A dollar amount, randomly generated within the banker’s chosen category, is loaned to the game player The game player does not know the category chosen by the banker The game player plays a maze of a certain difficulty (not necessarily what he or she had promised) The banker will not know the actual maze played The game player returns some money to the banker (not necessarily what he or she had promised) Anup Kalia (NCSU)
TRACE
September 30, 2015
16 / 24
Mapping Game Elements to The TRACE Model
I
The commitment from the player to the banker, Chplayer,banker,loan,returni, as satisfied if the player returned at least the amount he or she had loaned, and as violated, otherwise
I
We compute the gain or loss in the risk, Rh banker, player, loan, returni, based on the difference between the loaned and returned amounts
I
The dataset represents the banker’s trust for the player after the round, Thbanker,player,loan,returni, as a three-tuple hA, B, Ii, indicating the banker’s perception of player’s ability, benevolence, and integrity
I
The dataset represents the banker’s emotion after he or she receives a return from the player, Ehbanker,player,loan,returni, as real-valued (1–10) state anxiety scores derived from the post-round questionnaire
Anup Kalia (NCSU)
TRACE
September 30, 2015
17 / 24
Results MAEs of HMM and TRACE considering different feature combinations. Model HMM
TRACE
Anup Kalia (NCSU)
Input Variables
A
B
I
C C+R C+E R+E C+R+E
1.1220 0.8974 0.8619 0.8468 0.8870
0.8564 0.7655 0.7433 0.8376 0.7977
1.0917 0.8484 0.7184 0.7992 0.7714
C C+R C+E R+E C+R+E
0.8744 0.7463 0.8617 0.8949 0.7878
0.7576 0.7685 0.7656 0.6815 0.7427
0.7988 0.7876 0.7580 0.6568 0.7141
TRACE
September 30, 2015
18 / 24
Results I
Considering only C, TRACE yields lower MAEs than HMM for each trust attribute (CRF employs discriminative)
I
Considering all features (C + R + E), TRACE again yields lower MAEs than HMM for each trust attribute (CRF captures dependencies between C, R, and E)
I
Considering C and R, TRACE performs better than HMM in predicting A and I (C and R are not independent)
I
Considering C and E, HMM performs better than TRACE for B and I (C and E are conditionally independent)
I
Considering R and E, TRACE performs better than HMM for B and I whereas HMM performs better than TRACE for A (mixed)
Anup Kalia (NCSU)
TRACE
September 30, 2015
19 / 24
Threats to Validity
I
Our dataset, although real, consists of short sequences. We expect both HMM and TRACE to perform better given longer sequences
I
The dataset is skewed toward positive trust values and our conclusions may not hold since the trust values have a different distribution
I
The dataset represents emotions using anxiety scores only, thereby lacking realistic emotion responses along multiple dimensions such as anger and joy
Anup Kalia (NCSU)
TRACE
September 30, 2015
20 / 24
Discussion
I
TRACE illustrates that a probabilistic model of trust that incorporates commitments, risk, and emotions can produce trust estimates with fairly good accuracy
I
Our findings therefore open up the possibility of developing user agents that promote secure collaboration
I
Using TRACE a user can calibrate the perceived trust with the risk undertaken in light of available measures of risk and gain from commitments
Anup Kalia (NCSU)
TRACE
September 30, 2015
21 / 24
Thanks
Anup Kalia (NCSU)
TRACE
September 30, 2015
22 / 24
Background: Identifying Commitment Operations from Interactions Ten-fold cross-validation using SVM on marked up Enron email sentences
Commitment Operation
Precision
Recall
F-measure
0.87 0.94 0.86 1 0
0.97 0.97 0.33 0.02 0
0.92 0.95 0.48 0.04 0
Commissive create Directive create Delegate Discharge Cancel
Features used in the classifier include (out of 15) 1 2 3 4 5
Modal verb (shall, will, may, might, can, could, would, must) Type of subject (first person, second person, third person) Present tense verb Past tense verb Deadline
Anup Kalia (NCSU)
TRACE
September 30, 2015
23 / 24
Background: Determining Commitment Operations and Trust from Text Commitments being the most prominent normative relationship
S
R
Content
Operation
Kim
Dorothy
create(C1 )
Kim
Dorothy
Rob
Kim
Kim
Rob
I will also check with Alliance Travel Agency . . . I checked with our Travel Agency . . . By Wednesday Aug 16 2001, please send all copies of your documentation . . . Rob, please forgive me for not sending this in by Aug 15
I
T S ,R
discharge(C1 )
T R ,S
↑
create(C2 )
↓
cancel(C2 )
Example emails from the Enron corpus Anup Kalia (NCSU)
TRACE
September 30, 2015
24 / 24
