T - College of Computing
Scaling Regression Testing to Large Software Systems
Alessandro Orso Co-authors: Nanjuan Shi, Mary Jean Harrold College of Computing Georgia Institute of Technology Supported in part by National Science Foundation (awards CCR-0306372, CCR-0205422, CCR9988294, CCR-0209322, SBE-0123532, and CCR-0080900) and Boeing Commercial Airplane Group.
Testing of Evolving Software Program P
Test Suite T
P’
P’’
Trerun
T’rerun
T’ T T - Trerun
T’ - T’rerun
…
Time to rerun T Analysis Time
Time to rerun Trerun
Savings time
FSE 04
Previous and Related Work Retest
[OOPSLA01]
T (coverage info)
Ideal solution: two-phase approach P
Class-Level analysis subsetStmt-Level of P Analysis P’ Stmt-Level analysis on the subset Trerun
Time to rerun T Analysis Time
Time to rerun Trerun
Savings time
FSE 04
Trerun
Previous and Related Work Precise, Retest Expensive Techniques
T (coverage info)
Ideal solution: two-phase approach P
Class-Level analysis subsetStmt-Level of P Low-Level Analysis P’ Stmt-Level analysis on the subset Trerun Jboss, web application server, 1 million LOC Time to rerun T Analysis time
Time to rerun Trerun time FSE 04
Trerun
Previous and Related Work Efficient, Imprecise Techniques
P P’
T (coverage info)
High-Level
Trerun
Analysis
Time to rerun T Analysis time
Time to rerun Trerun time FSE 04
Previous and Related Work Related Work T Efficient, Imprecise Techniques
Efficient, less precise techniques (coverage info)
P P’
High-Level
White and Leung [CSM92] Chen, Rosenblum, and Vo [ICSE94] Hsia et al. [SMRP97] White and Abdullah [QW97] Ren et al. [OOPSLA04]
Analysis Expensive, more precise techniques
Analysis time
Trerun
Binkley [TSE97] Rothermel and Harrold [TOSEM97] Vokolos and Frankl [RQSSIS97] Ball [ISSTA’98] Rothermel, Harrold, and Dedhia [JSTVR00] Time to rerun T Harrold et al. [OOPSLA01] Time to rerun Trerun and Rosenblum [TOSEM01] Bible, Rothermel, time FSE 04
Proposed Solution Two-Phase Technique
P P’
Class-Level Analysis
T (coverage info)
Subset of P Subset of P’
Stmt-Level Analysis
Two-phase approach Class-Level analysis subset of P and P’ Stmt-Level analysis on the subset Trerun FSE 04
Trerun
Outline Background Technique Empirical Evaluation Conclusion
FSE 04
Motivating Example P
class A { void foo() {…} } class B extends A { }
class C extends B {} class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo(); } }
class E extends D
{}
class F { void bar(D d) {…} } FSE 04
Motivating Example P
class A { void foo() {…} } class B extends A {
P’ class A {
void foo() {…} } class B extends A { void foo() {… }
}
}
class C extends B {}
class C extends B {}
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
class E extends D
class E extends D
} }
} }
{}
class F { void bar(D d) {…} }
class F { void bar(D d) {…} } FSE 04
{}
Motivating Example P
class A { void foo() {…} } class B extends A {
P’ class A {
void foo() {…} } class B extends A { void foo() {… }
}
}
class C extends B {}
class C extends B {}
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
class E extends D
class E extends D
} }
} }
{}
class F { void bar(D d) {…} }
class F { void bar(D d) {…} } FSE 04
{}
Motivating Example P
class A { void foo() {…} } class B extends A {
P’ class A {
void foo() {…} } class B extends A { void foo() {… }
}
}
class C extends B {}
class C extends B {}
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
class E extends D
class E extends D
} }
} }
{}
class F { void bar(D d) {…} }
class F { void bar(D d) {…} } FSE 04
{}
Class-Level Analysis P
P’ class B extends A {
class B extends A { void foo() {… }
}
}
FSE 04
Class-Level Analysis P
class A { void foo() {…} } class B extends A {
P’ class A {
void foo() {…} } class B extends A { void foo() {… }
}
}
class C extends B {}
class C extends B {}
FSE 04
Class-Level Analysis P
class A { void foo() {…} } class B extends A {
P’ class A {
void foo() {…} } class B extends A { void foo() {… }
}
}
class C extends B {}
class C extends B {}
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
} }
} }
FSE 04
Class-Level Analysis P / P’
Interclass Relation Graph (for P an P’)
class A { void foo() {…} } class B extends A { void foo() {… } }
A
class C extends B {} class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
D B E C Inheritance edge
} }
class E extends D
F
{}
Use edge
class F { void bar(D d) {…} } FSE 04
Class-Level Analysis P / P’
Interclass Relation Graph (for P an P’)
class A { void foo() {…} } class B extends A { void foo() {… } }
A
class C extends B {} class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
D B E C Inheritance edge
} }
class E extends D
F
{}
Use edge
class F { void bar(D d) {…} } FSE 04
Class-Level Analysis P / P’
Interclass Relation Graph (for P an P’)
class A { void foo() {…} } class B extends A { void foo() {… } }
A
class C extends B {} class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
D B E C Inheritance edge
} }
class E extends D
F
{}
Use edge
class F { void bar(D d) {…} } FSE 04
Class-Level Analysis P / P’
Interclass Relation Graph (for P an P’)
class A { void foo() {…} } class B extends A { void foo() {… } }
A
class C extends B {} class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
D B E C Inheritance edge
} }
class E extends D
F
{}
Use edge
class F { void bar(D d) {…} } FSE 04
Example: Stmt-Level Analysis Subset of P
Subset of P’
class A class B {…} class C class D { void bar() {…; ref.foo(); …} }}
class A class B {… void foo() {…} … } class C class D { void bar() {…; ref.foo(); …} }}
G (excerpt)
G’ (excerpt)
…
B ref.foo()
…
A
…
A
A.foo()
A.foo() ref.foo()
…
B
…
C
C
Dangerous Edge
… B.foo() …
FSE 04
Example: Stmt-Level Analysis Subset of P
Subset of P’
class A class B {…} class C class D { void bar() {…; ref.foo(); …} }}
class A class B {… void foo() {…} … } class C class D { void bar() {…; ref.foo(); …} }}
G (excerpt)
G’ (excerpt)
…
B ref.foo()
…
A A.foo() …
C Dangerous Edge
A
Test…cases to be rerun: A.foo() Test cases in T that ref.foo() execute the call node… B with … ref’s dynamic type being B or C B.foo() C
… FSE 04
Outline Background Technique Empirical Evaluation Conclusion
FSE 04
Empirical Setup Tool DejaVOO
Subjects Program
Versions
Classes
KLOC
Test Cases
Retest Time
Jaba
5
525
70
707
54 min
Daikon
5
824
167
200
74 min
Jboss
5
2,403
1,000
639
32 min
Three empirical studies 1. Effectiveness of Phase 1 2. Precision gain of Phase 2 3. Overall savings in retesting time FSE 04
RQ1: Effectiveness of Phase 1
TwoPhase
P P’
T (coverage info)
Class-Level Analysis
Subset of P Subset of P’
FSE 04
Stmt-Level Analysis
Trerun
RQ1: Effectiveness of Phase 1 Average: 37% Time < 14 seconds
FSE 04
RQ3: Overall Savings in retesting Time
Time to rerun T Analysis time
Time to rerun Trerun
Savings time
FSE 04
RQ3: Overall Savings in Retesting Time RerunAll
FSE 04
TwoPhase
RQ3: Overall Savings in Retesting Time RerunAll
Savings in Regression Testing Time: TwoPhase vs. RerunAll Jaba:19% Daikon:36% Jboss: 63%
FSE 04
TwoPhase
Outline Background Technique Empirical Evaluation Conclusion
FSE 04
Contribution Extended our existing regression-testing technique Considerable increase in efficiency Same precision
First RTS technique that, at the same time: is safe combines coarse and fine-grained analysis handles the features of the Java language
Tool (DejaVOO) that implements the technique Empirical results showing the effectiveness of the technique
FSE 04
Current and Future Work Further investigate cost-effectiveness of the technique Continuous integration Larger systems
Investigate test-suite augmentation Based on changes (program differencing) Based on field-data (impact analysis)
FSE 04
Questions?
FSE 04
Alessandro Orso Co-authors: Nanjuan Shi, Mary Jean Harrold College of Computing Georgia Institute of Technology Supported in part by National Science Foundation (awards CCR-0306372, CCR-0205422, CCR9988294, CCR-0209322, SBE-0123532, and CCR-0080900) and Boeing Commercial Airplane Group.
Testing of Evolving Software Program P
Test Suite T
P’
P’’
Trerun
T’rerun
T’ T T - Trerun
T’ - T’rerun
…
Time to rerun T Analysis Time
Time to rerun Trerun
Savings time
FSE 04
Previous and Related Work Retest
[OOPSLA01]
T (coverage info)
Ideal solution: two-phase approach P
Class-Level analysis subsetStmt-Level of P Analysis P’ Stmt-Level analysis on the subset Trerun
Time to rerun T Analysis Time
Time to rerun Trerun
Savings time
FSE 04
Trerun
Previous and Related Work Precise, Retest Expensive Techniques
T (coverage info)
Ideal solution: two-phase approach P
Class-Level analysis subsetStmt-Level of P Low-Level Analysis P’ Stmt-Level analysis on the subset Trerun Jboss, web application server, 1 million LOC Time to rerun T Analysis time
Time to rerun Trerun time FSE 04
Trerun
Previous and Related Work Efficient, Imprecise Techniques
P P’
T (coverage info)
High-Level
Trerun
Analysis
Time to rerun T Analysis time
Time to rerun Trerun time FSE 04
Previous and Related Work Related Work T Efficient, Imprecise Techniques
Efficient, less precise techniques (coverage info)
P P’
High-Level
White and Leung [CSM92] Chen, Rosenblum, and Vo [ICSE94] Hsia et al. [SMRP97] White and Abdullah [QW97] Ren et al. [OOPSLA04]
Analysis Expensive, more precise techniques
Analysis time
Trerun
Binkley [TSE97] Rothermel and Harrold [TOSEM97] Vokolos and Frankl [RQSSIS97] Ball [ISSTA’98] Rothermel, Harrold, and Dedhia [JSTVR00] Time to rerun T Harrold et al. [OOPSLA01] Time to rerun Trerun and Rosenblum [TOSEM01] Bible, Rothermel, time FSE 04
Proposed Solution Two-Phase Technique
P P’
Class-Level Analysis
T (coverage info)
Subset of P Subset of P’
Stmt-Level Analysis
Two-phase approach Class-Level analysis subset of P and P’ Stmt-Level analysis on the subset Trerun FSE 04
Trerun
Outline Background Technique Empirical Evaluation Conclusion
FSE 04
Motivating Example P
class A { void foo() {…} } class B extends A { }
class C extends B {} class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo(); } }
class E extends D
{}
class F { void bar(D d) {…} } FSE 04
Motivating Example P
class A { void foo() {…} } class B extends A {
P’ class A {
void foo() {…} } class B extends A { void foo() {… }
}
}
class C extends B {}
class C extends B {}
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
class E extends D
class E extends D
} }
} }
{}
class F { void bar(D d) {…} }
class F { void bar(D d) {…} } FSE 04
{}
Motivating Example P
class A { void foo() {…} } class B extends A {
P’ class A {
void foo() {…} } class B extends A { void foo() {… }
}
}
class C extends B {}
class C extends B {}
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
class E extends D
class E extends D
} }
} }
{}
class F { void bar(D d) {…} }
class F { void bar(D d) {…} } FSE 04
{}
Motivating Example P
class A { void foo() {…} } class B extends A {
P’ class A {
void foo() {…} } class B extends A { void foo() {… }
}
}
class C extends B {}
class C extends B {}
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
class E extends D
class E extends D
} }
} }
{}
class F { void bar(D d) {…} }
class F { void bar(D d) {…} } FSE 04
{}
Class-Level Analysis P
P’ class B extends A {
class B extends A { void foo() {… }
}
}
FSE 04
Class-Level Analysis P
class A { void foo() {…} } class B extends A {
P’ class A {
void foo() {…} } class B extends A { void foo() {… }
}
}
class C extends B {}
class C extends B {}
FSE 04
Class-Level Analysis P
class A { void foo() {…} } class B extends A {
P’ class A {
void foo() {…} } class B extends A { void foo() {… }
}
}
class C extends B {}
class C extends B {}
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
} }
} }
FSE 04
Class-Level Analysis P / P’
Interclass Relation Graph (for P an P’)
class A { void foo() {…} } class B extends A { void foo() {… } }
A
class C extends B {} class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
D B E C Inheritance edge
} }
class E extends D
F
{}
Use edge
class F { void bar(D d) {…} } FSE 04
Class-Level Analysis P / P’
Interclass Relation Graph (for P an P’)
class A { void foo() {…} } class B extends A { void foo() {… } }
A
class C extends B {} class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
D B E C Inheritance edge
} }
class E extends D
F
{}
Use edge
class F { void bar(D d) {…} } FSE 04
Class-Level Analysis P / P’
Interclass Relation Graph (for P an P’)
class A { void foo() {…} } class B extends A { void foo() {… } }
A
class C extends B {} class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
D B E C Inheritance edge
} }
class E extends D
F
{}
Use edge
class F { void bar(D d) {…} } FSE 04
Class-Level Analysis P / P’
Interclass Relation Graph (for P an P’)
class A { void foo() {…} } class B extends A { void foo() {… } }
A
class C extends B {} class D { void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();
D B E C Inheritance edge
} }
class E extends D
F
{}
Use edge
class F { void bar(D d) {…} } FSE 04
Example: Stmt-Level Analysis Subset of P
Subset of P’
class A class B {…} class C class D { void bar() {…; ref.foo(); …} }}
class A class B {… void foo() {…} … } class C class D { void bar() {…; ref.foo(); …} }}
G (excerpt)
G’ (excerpt)
…
B ref.foo()
…
A
…
A
A.foo()
A.foo() ref.foo()
…
B
…
C
C
Dangerous Edge
… B.foo() …
FSE 04
Example: Stmt-Level Analysis Subset of P
Subset of P’
class A class B {…} class C class D { void bar() {…; ref.foo(); …} }}
class A class B {… void foo() {…} … } class C class D { void bar() {…; ref.foo(); …} }}
G (excerpt)
G’ (excerpt)
…
B ref.foo()
…
A A.foo() …
C Dangerous Edge
A
Test…cases to be rerun: A.foo() Test cases in T that ref.foo() execute the call node… B with … ref’s dynamic type being B or C B.foo() C
… FSE 04
Outline Background Technique Empirical Evaluation Conclusion
FSE 04
Empirical Setup Tool DejaVOO
Subjects Program
Versions
Classes
KLOC
Test Cases
Retest Time
Jaba
5
525
70
707
54 min
Daikon
5
824
167
200
74 min
Jboss
5
2,403
1,000
639
32 min
Three empirical studies 1. Effectiveness of Phase 1 2. Precision gain of Phase 2 3. Overall savings in retesting time FSE 04
RQ1: Effectiveness of Phase 1
TwoPhase
P P’
T (coverage info)
Class-Level Analysis
Subset of P Subset of P’
FSE 04
Stmt-Level Analysis
Trerun
RQ1: Effectiveness of Phase 1 Average: 37% Time < 14 seconds
FSE 04
RQ3: Overall Savings in retesting Time
Time to rerun T Analysis time
Time to rerun Trerun
Savings time
FSE 04
RQ3: Overall Savings in Retesting Time RerunAll
FSE 04
TwoPhase
RQ3: Overall Savings in Retesting Time RerunAll
Savings in Regression Testing Time: TwoPhase vs. RerunAll Jaba:19% Daikon:36% Jboss: 63%
FSE 04
TwoPhase
Outline Background Technique Empirical Evaluation Conclusion
FSE 04
Contribution Extended our existing regression-testing technique Considerable increase in efficiency Same precision
First RTS technique that, at the same time: is safe combines coarse and fine-grained analysis handles the features of the Java language
Tool (DejaVOO) that implements the technique Empirical results showing the effectiveness of the technique
FSE 04
Current and Future Work Further investigate cost-effectiveness of the technique Continuous integration Larger systems
Investigate test-suite augmentation Based on changes (program differencing) Based on field-data (impact analysis)
FSE 04
Questions?
FSE 04
