EverybodyLies (GeeCON)
EV ERY B OD Y
LIE S
TOMASZ KOWALCZEWSKI
C A R G O C U LT
During the Middle Ages there were all kinds of crazy ideas, such as that a piece of of rhinoceros horn would increase potency. Then a method was discovered for separating the ideas- which was to try one to see if it worked, and if it didn't work, to eliminate it. This method became organized, of course, into science. And it developed very well, so that we are now in the scientific age. It is such a scientific age, in fact, that we have difficulty in understanding how witch doctors could ever have existed, when nothing that they proposed ever really worked-or very little of it did.
Richard Feynman From a Caltech commencement address given in 1974
WHY BOTHER? • You get what you measure - Ineffective optimisations that complicate code + Numbers to convince management to do
refactoring or migration to Java 8!
WHY BOTHER? • Predictable is better than fast • One page display requires multiple calls (static and
dynamic resources)
• Multiple microservices are called to generate response • During a session user may do hundreds of displays of
your webpages
WHY DO THIS? • Every 100 ms increase in load time of Amazon.com
decreased sales by 1%1
• Increasing web search latency 100 to 400 ms reduces
the daily searches per user by 0.2% to 0.6%. Furthermore, users do fewer searches the longer they are exposed. For longer delays, the loss of searches persists for a time even after latency returns to previous levels.2
1Kohavi 2Brutlag
and Longbotham 2007 2009
SURVEY • Do you…
SURVEY • Use graphite?
SURVEY • Use graphite? • Feed it with Coda Hale/Dropwizard metrics?
SURVEY • Use graphite? • Feed it with Coda Hale/Dropwizard metrics? • Modify their source? Use nonstandard options?
SURVEY • Use graphite? • Feed it with Coda Hale/Dropwizard metrics? • Modify their source? Use nonstandard options? • Graph average? Median?
SURVEY • Use graphite? • Feed it with Coda Hale/Dropwizard metrics? • Modify their source? Use nonstandard options? • Graph average? Median? • Percentiles?
(c) xkcd.com
W H AT M E T R I C S C A N W E U S E ? graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name,
"max"), ...); "mean"), ...); "min"), ...); "stddev"), ...); "p50"), ...); "p75"), ...); "p95"), ...); "p98"), ...); "p99"), ...); “p999"), ...);
D O N ’ T L O O K AT M E A N • 1000 queries - 0ms latency, 100 queries 5s latency • Average is 4,5ms • 1000 queries - 1ms latency, 100 queries - 5s latency • Average is 455ms • Does not help to quantify lags users will experience
These four data sets all have the same mean, median, and variance –ANSCOMBE'S QUARTET BY FRANCIS ANSCOMBE
PLOTTING MEAN IS FOR SHOWING OFF TO MANAGEMENT
M AY B E M E D I A N T H E N ? • What is the probability of end user encountering
latency worse than median?
• Remember: usually multiple requests are needed to
respond to API call (e.g. N micro services, N resource requests per page) ✓ ◆N 1 · 100 2
PROBABILITY OF EXPERIENCING L AT E N C Y B E T T E R T H A N M E D I A N
I N F U N C T I O N O F M I C R O S E R V I C E S I N V O LV E D
100
90
80
70
60
50
40
30
20
10
9
8
7
6
5
4
3
2
1
0
10
W H I C H P E R C E N T I L E I S R E L E VA N T T O YOU? • Is 99th percentile demanding constraint? • In application serving 1000 qps latency worse than that happens
ten times per second.
• User that needs to navigate through several web pages will most
probably experience it
• What is the probability of encountering latency better than 99th?
✓
99 100
◆N
· 100
PROBABILITY OF EXPERIENCING TH L AT E N C Y B E T T E R T H A N 9 9 PERCENTILE I N F U N C T I O N O F M I C R O S E R V I C E S I N V O LV E D 100
90
80
70
60
50
40
30
20
100
90
80
70
60
50
40
30
20
0
0
10
10
D O N O T AV E R A G E P E R C E N T I L E S Example scenario: 1. Load balancer splits traffic unevenly (ELB anyone?) 2. Server S1 has 1 qps over measured time with 95%’ile == 1ms 3. Server S2 has 100 qps over measured time with 95%’ile == 10s 4. Average is ~5s. 5. What does that tell us? 6. Did we satisfy SLA if it says “95%’ile must be below 8s”? 7. Actual 95%’ile percentile is ~10s
“If there's no meaning in it,' said the King, 'that saves a world of trouble, you know, as we needn't try to find any” –ALICE'S ADVENTURES IN WONDERLAND
Every time you average max values someone in the world starts new JavaScript framework
Demo time
Standard timer will over or under report actual percentiles at will. Green line represents actual MAX values. metricRegistry.timer("2015.standardTimer");
Standard timer will over or under report actual percentiles at will. Green line represents actual MAX values. metricRegistry.timer("2015.standardTimer");
TIMER’S HISTOGRAM RESERVOIR • Backing storage for Timer’s data • Contain “statistically representative reservoir of a data stream” • Default is ExponentiallyDecayingReservoir which has many
drawbacks and is source of most inaccuracies observed throughout this presentation • Others include • UniformReservoir, SlidingTimeWindowReservoir,
SlidingTimeWindowReservoir, SlidingWindowReservoir
E X P O N E N T I A L LY D E C AY I N G R E S E R V O I R • Stores 1028 random samples by default • Assumes normal distribution of recorded values • Many statistical tools applied in computer systems
monitoring will assume normal distribution
• Be suspicious of such tools • Why is that a bad idea?
3
NORMAL DISTRIBUTION WHY SO USEFUL?
f (x, µ,
1 p e 2⇡
)=
(x µ)2 2 2
2,5
2
• Central limit theorem
1,5
• Chebyshev's inequality 1
0,5
-2,4
-2
-1,6
-1,2
-0,8
-0,4
0
0,4
0,8
C A L C U L AT E 9 5 % ’ I L E BASED ON MEAN A N D S T D . D E V.
1
0,75
• IFF latency values were
distributed normally then we could calculate any percentile based on mean and standard deviation
µ = 10ms
= 1ms
0,5
0,25
• Lookup into standard
normal (Z) table
10
• 95%’ile is located 1.65 std.
dev. from mean • Result is 11,65ms
-0,25
10,5
11
11,5
12
Latency profile resembling normal distribution…
Add spikes due to young gen GC pauses
Add spikes due to old gen GC pauses
Add spikes due to calling other services (like DB)
Add spikes due to: lost tcp packet retransmission, disk swapping, kernel bookkeeping etc.
3
NORMAL DISTRIBUTION - WHY NOT APPLICABLE?
f (x, µ,
1 p e 2⇡
)=
(x µ)2 2 2
2,5
• The value of the normal
distribution is practically zero when the value x lies more than a few standard deviations away from the mean.
2
1,5
• It may not be an appropriate
model when one expects a significant fraction of outliers 1
• […] other statistical inference
methods that are optimal for normally distributed variables often become highly unreliable when 1 applied to such data. 1All
quotes on this slide from Wikipedia -2,4 -2 -1,6
0,5
-1,2
-0,8
-0,4
0
0,4
0,8
Blue line represents metric reported from Timer class Green line represents request rate
TIMER, TIMER NEVER CHANGES… • Timer values decay exponentially • giving artificial smoothing of values
for server behaviour that may be long gone • Timer that is not updated does not
decay • If Timer is not updated (e.g.
subprocess failed and we stopped sending requests to it) its values will remain constant • Check this post for potential solutions:
taint.org/2014/01/16/145944a.html
HDR HISTOGRAM • Supports recording and analysis of sampled data across
configurable range with configurable accuracy
• Provides compact representation of data while retaining
high resolution
• Allows configurable tradeoffs between space and accuracy • Very fast, allocation free, not thread safe for maximum
speed (thread safe versions available)
• Created by Gil Tene of Azul Sytems
RECORDER • Uses HdrHistogram to store values • Supports concurrent recording of values • Recording is lock free but also wait free on most
architectures (that support lock xadd)
• Reading is not lock free but does not stall writers (writer-
reader phaser)
• Checkout Marshall Pierce’s library for using it as a
Reservoir implementation
SOLUTIONS • Always instantiate Timer with custom reservoir • new ExponentiallyDecayingReservoir(LARGE_NUMBER) • new SlidingTimeWindowReservoir(1, MINUTES) • new HdrHistogramResetOnSnapshotReservoir() • Only last one is safe and accurate and will not report stale values
if no updates were made
JMH benchmarks (from my laptop, caveat emptor!)
SMOKING BENCHMARKING IS THE L E A D I N G C A U S E O F S TAT I S T I C S I N THE WORLD
C O O R D I N AT E D O M I S S I O N • As formulated by Gil Tene of Azul Systems • When load driver is plotting with system under test to
deceive you
• Most tools do this • Most benchmarks do this • Yahoo Cloud Serving Benchmark had that problem1 1Recently
fixed by Nitsan Wakart, see psy-lob-saw.blogspot.com/2015/03/fixing-ycsb-coordinated-omission.html
latency
4
3,2
Application pause time 2,4
1,6
0,8
-0,8
0
0,8
1,6
2,4
3,2
4
4,8
5,6
request arrival time
-0,8
Requests according to test plan. Only red one will be send. Others will be missing from test.
6,4
Effects on benchmarks at high percentiles are spectacular – C R E AT E D W I T H G I L T E N E ' S H D R H I S T O G R A M PLOTTING SCRIPT
C O O R D I N AT E D O M I S S I O N S O L U T I O N S 1. Ignore the problem! perfectly fine for non interactive system where only throughput matters
C O O R D I N AT E D O M I S S I O N S O L U T I O N S 2. Correct it mathematically in sampling mechanism HdrHistogram can correct CO with these methods (choose one!): histogram.recordValueWithExpectedInterval( value, expectedIntervalBetweenSamples ); histogram.copyCorrectedForCoordinatedOmission( expectedIntervalBetweenSamples );
C O O R D I N AT E D O M I S S I O N S O L U T I O N S 3. Correct it on load driver side by noticing pauses between sent requests. newly issued request will have timer that starts counting from time it should have been sent but wasn't
C O O R D I N AT E D OMISSION SOLUTIONS 4. Fail the test for hard real time systems where pause causes human casualties (breaks, pacemakers, Phalanx system)
C O O R D I N AT E D O M I S S I O N • Mathematical solutions can overcorrect when load driver
has pauses (e.g. GC).
• Do not account for the fact that server after pause has
no work to do instead of N more requests waiting to be executed
• In real world it might have never recovered • Most tools ignore the problem • Notable exception: Twitter Iago
“Do not bend to the tyranny of reality”
–LOAD DRIVER MOTTO
SUMMARY • Measure what is meaningful not just what is measurable • Set SLA before testing and creating dashboards • Do not trust Timer class, use custom reservoirs, HdrHistogram,
Recorder, never trust EMWA for request rate
• Do not average percentiles unless you need a random number
generator
• Do not plot averages unless you just want to look good on
dashboards
• When load testing be aware of coordinated omission
SOURCES, THANK YOUS AND RECOMMENDED FOLLOW UPS • Coda Hale for great metrics library • Gil Tene • latencytipoftheday.blogspot.de • www.infoq.com/presentations/latency-pitfalls • github.com/HdrHistogram/HdrHistogram • Nitsan Wakart • psy-lob-saw.blogspot.de/2015/03/fixing-ycsb-coordinated-omission.html • and whole blog • Matin Thompson et. al. • groups.google.com/forum/#!forum/mechanical-sympathy
RECOMMENDED
Great introduction to statistics and queueing theory. Performance Modeling and Design of Computer Systems: Queueing Theory in Action Prof. Mor Harchol-Balter
F E E D B A C K K I N D LY R E Q U E S T E D
https://www.surveymonkey.com/s/B5KGWWN
LIE S
TOMASZ KOWALCZEWSKI
C A R G O C U LT
During the Middle Ages there were all kinds of crazy ideas, such as that a piece of of rhinoceros horn would increase potency. Then a method was discovered for separating the ideas- which was to try one to see if it worked, and if it didn't work, to eliminate it. This method became organized, of course, into science. And it developed very well, so that we are now in the scientific age. It is such a scientific age, in fact, that we have difficulty in understanding how witch doctors could ever have existed, when nothing that they proposed ever really worked-or very little of it did.
Richard Feynman From a Caltech commencement address given in 1974
WHY BOTHER? • You get what you measure - Ineffective optimisations that complicate code + Numbers to convince management to do
refactoring or migration to Java 8!
WHY BOTHER? • Predictable is better than fast • One page display requires multiple calls (static and
dynamic resources)
• Multiple microservices are called to generate response • During a session user may do hundreds of displays of
your webpages
WHY DO THIS? • Every 100 ms increase in load time of Amazon.com
decreased sales by 1%1
• Increasing web search latency 100 to 400 ms reduces
the daily searches per user by 0.2% to 0.6%. Furthermore, users do fewer searches the longer they are exposed. For longer delays, the loss of searches persists for a time even after latency returns to previous levels.2
1Kohavi 2Brutlag
and Longbotham 2007 2009
SURVEY • Do you…
SURVEY • Use graphite?
SURVEY • Use graphite? • Feed it with Coda Hale/Dropwizard metrics?
SURVEY • Use graphite? • Feed it with Coda Hale/Dropwizard metrics? • Modify their source? Use nonstandard options?
SURVEY • Use graphite? • Feed it with Coda Hale/Dropwizard metrics? • Modify their source? Use nonstandard options? • Graph average? Median?
SURVEY • Use graphite? • Feed it with Coda Hale/Dropwizard metrics? • Modify their source? Use nonstandard options? • Graph average? Median? • Percentiles?
(c) xkcd.com
W H AT M E T R I C S C A N W E U S E ? graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name, graphite.send(prefix(name,
"max"), ...); "mean"), ...); "min"), ...); "stddev"), ...); "p50"), ...); "p75"), ...); "p95"), ...); "p98"), ...); "p99"), ...); “p999"), ...);
D O N ’ T L O O K AT M E A N • 1000 queries - 0ms latency, 100 queries 5s latency • Average is 4,5ms • 1000 queries - 1ms latency, 100 queries - 5s latency • Average is 455ms • Does not help to quantify lags users will experience
These four data sets all have the same mean, median, and variance –ANSCOMBE'S QUARTET BY FRANCIS ANSCOMBE
PLOTTING MEAN IS FOR SHOWING OFF TO MANAGEMENT
M AY B E M E D I A N T H E N ? • What is the probability of end user encountering
latency worse than median?
• Remember: usually multiple requests are needed to
respond to API call (e.g. N micro services, N resource requests per page) ✓ ◆N 1 · 100 2
PROBABILITY OF EXPERIENCING L AT E N C Y B E T T E R T H A N M E D I A N
I N F U N C T I O N O F M I C R O S E R V I C E S I N V O LV E D
100
90
80
70
60
50
40
30
20
10
9
8
7
6
5
4
3
2
1
0
10
W H I C H P E R C E N T I L E I S R E L E VA N T T O YOU? • Is 99th percentile demanding constraint? • In application serving 1000 qps latency worse than that happens
ten times per second.
• User that needs to navigate through several web pages will most
probably experience it
• What is the probability of encountering latency better than 99th?
✓
99 100
◆N
· 100
PROBABILITY OF EXPERIENCING TH L AT E N C Y B E T T E R T H A N 9 9 PERCENTILE I N F U N C T I O N O F M I C R O S E R V I C E S I N V O LV E D 100
90
80
70
60
50
40
30
20
100
90
80
70
60
50
40
30
20
0
0
10
10
D O N O T AV E R A G E P E R C E N T I L E S Example scenario: 1. Load balancer splits traffic unevenly (ELB anyone?) 2. Server S1 has 1 qps over measured time with 95%’ile == 1ms 3. Server S2 has 100 qps over measured time with 95%’ile == 10s 4. Average is ~5s. 5. What does that tell us? 6. Did we satisfy SLA if it says “95%’ile must be below 8s”? 7. Actual 95%’ile percentile is ~10s
“If there's no meaning in it,' said the King, 'that saves a world of trouble, you know, as we needn't try to find any” –ALICE'S ADVENTURES IN WONDERLAND
Every time you average max values someone in the world starts new JavaScript framework
Demo time
Standard timer will over or under report actual percentiles at will. Green line represents actual MAX values. metricRegistry.timer("2015.standardTimer");
Standard timer will over or under report actual percentiles at will. Green line represents actual MAX values. metricRegistry.timer("2015.standardTimer");
TIMER’S HISTOGRAM RESERVOIR • Backing storage for Timer’s data • Contain “statistically representative reservoir of a data stream” • Default is ExponentiallyDecayingReservoir which has many
drawbacks and is source of most inaccuracies observed throughout this presentation • Others include • UniformReservoir, SlidingTimeWindowReservoir,
SlidingTimeWindowReservoir, SlidingWindowReservoir
E X P O N E N T I A L LY D E C AY I N G R E S E R V O I R • Stores 1028 random samples by default • Assumes normal distribution of recorded values • Many statistical tools applied in computer systems
monitoring will assume normal distribution
• Be suspicious of such tools • Why is that a bad idea?
3
NORMAL DISTRIBUTION WHY SO USEFUL?
f (x, µ,
1 p e 2⇡
)=
(x µ)2 2 2
2,5
2
• Central limit theorem
1,5
• Chebyshev's inequality 1
0,5
-2,4
-2
-1,6
-1,2
-0,8
-0,4
0
0,4
0,8
C A L C U L AT E 9 5 % ’ I L E BASED ON MEAN A N D S T D . D E V.
1
0,75
• IFF latency values were
distributed normally then we could calculate any percentile based on mean and standard deviation
µ = 10ms
= 1ms
0,5
0,25
• Lookup into standard
normal (Z) table
10
• 95%’ile is located 1.65 std.
dev. from mean • Result is 11,65ms
-0,25
10,5
11
11,5
12
Latency profile resembling normal distribution…
Add spikes due to young gen GC pauses
Add spikes due to old gen GC pauses
Add spikes due to calling other services (like DB)
Add spikes due to: lost tcp packet retransmission, disk swapping, kernel bookkeeping etc.
3
NORMAL DISTRIBUTION - WHY NOT APPLICABLE?
f (x, µ,
1 p e 2⇡
)=
(x µ)2 2 2
2,5
• The value of the normal
distribution is practically zero when the value x lies more than a few standard deviations away from the mean.
2
1,5
• It may not be an appropriate
model when one expects a significant fraction of outliers 1
• […] other statistical inference
methods that are optimal for normally distributed variables often become highly unreliable when 1 applied to such data. 1All
quotes on this slide from Wikipedia -2,4 -2 -1,6
0,5
-1,2
-0,8
-0,4
0
0,4
0,8
Blue line represents metric reported from Timer class Green line represents request rate
TIMER, TIMER NEVER CHANGES… • Timer values decay exponentially • giving artificial smoothing of values
for server behaviour that may be long gone • Timer that is not updated does not
decay • If Timer is not updated (e.g.
subprocess failed and we stopped sending requests to it) its values will remain constant • Check this post for potential solutions:
taint.org/2014/01/16/145944a.html
HDR HISTOGRAM • Supports recording and analysis of sampled data across
configurable range with configurable accuracy
• Provides compact representation of data while retaining
high resolution
• Allows configurable tradeoffs between space and accuracy • Very fast, allocation free, not thread safe for maximum
speed (thread safe versions available)
• Created by Gil Tene of Azul Sytems
RECORDER • Uses HdrHistogram to store values • Supports concurrent recording of values • Recording is lock free but also wait free on most
architectures (that support lock xadd)
• Reading is not lock free but does not stall writers (writer-
reader phaser)
• Checkout Marshall Pierce’s library for using it as a
Reservoir implementation
SOLUTIONS • Always instantiate Timer with custom reservoir • new ExponentiallyDecayingReservoir(LARGE_NUMBER) • new SlidingTimeWindowReservoir(1, MINUTES) • new HdrHistogramResetOnSnapshotReservoir() • Only last one is safe and accurate and will not report stale values
if no updates were made
JMH benchmarks (from my laptop, caveat emptor!)
SMOKING BENCHMARKING IS THE L E A D I N G C A U S E O F S TAT I S T I C S I N THE WORLD
C O O R D I N AT E D O M I S S I O N • As formulated by Gil Tene of Azul Systems • When load driver is plotting with system under test to
deceive you
• Most tools do this • Most benchmarks do this • Yahoo Cloud Serving Benchmark had that problem1 1Recently
fixed by Nitsan Wakart, see psy-lob-saw.blogspot.com/2015/03/fixing-ycsb-coordinated-omission.html
latency
4
3,2
Application pause time 2,4
1,6
0,8
-0,8
0
0,8
1,6
2,4
3,2
4
4,8
5,6
request arrival time
-0,8
Requests according to test plan. Only red one will be send. Others will be missing from test.
6,4
Effects on benchmarks at high percentiles are spectacular – C R E AT E D W I T H G I L T E N E ' S H D R H I S T O G R A M PLOTTING SCRIPT
C O O R D I N AT E D O M I S S I O N S O L U T I O N S 1. Ignore the problem! perfectly fine for non interactive system where only throughput matters
C O O R D I N AT E D O M I S S I O N S O L U T I O N S 2. Correct it mathematically in sampling mechanism HdrHistogram can correct CO with these methods (choose one!): histogram.recordValueWithExpectedInterval( value, expectedIntervalBetweenSamples ); histogram.copyCorrectedForCoordinatedOmission( expectedIntervalBetweenSamples );
C O O R D I N AT E D O M I S S I O N S O L U T I O N S 3. Correct it on load driver side by noticing pauses between sent requests. newly issued request will have timer that starts counting from time it should have been sent but wasn't
C O O R D I N AT E D OMISSION SOLUTIONS 4. Fail the test for hard real time systems where pause causes human casualties (breaks, pacemakers, Phalanx system)
C O O R D I N AT E D O M I S S I O N • Mathematical solutions can overcorrect when load driver
has pauses (e.g. GC).
• Do not account for the fact that server after pause has
no work to do instead of N more requests waiting to be executed
• In real world it might have never recovered • Most tools ignore the problem • Notable exception: Twitter Iago
“Do not bend to the tyranny of reality”
–LOAD DRIVER MOTTO
SUMMARY • Measure what is meaningful not just what is measurable • Set SLA before testing and creating dashboards • Do not trust Timer class, use custom reservoirs, HdrHistogram,
Recorder, never trust EMWA for request rate
• Do not average percentiles unless you need a random number
generator
• Do not plot averages unless you just want to look good on
dashboards
• When load testing be aware of coordinated omission
SOURCES, THANK YOUS AND RECOMMENDED FOLLOW UPS • Coda Hale for great metrics library • Gil Tene • latencytipoftheday.blogspot.de • www.infoq.com/presentations/latency-pitfalls • github.com/HdrHistogram/HdrHistogram • Nitsan Wakart • psy-lob-saw.blogspot.de/2015/03/fixing-ycsb-coordinated-omission.html • and whole blog • Matin Thompson et. al. • groups.google.com/forum/#!forum/mechanical-sympathy
RECOMMENDED
Great introduction to statistics and queueing theory. Performance Modeling and Design of Computer Systems: Queueing Theory in Action Prof. Mor Harchol-Balter
F E E D B A C K K I N D LY R E Q U E S T E D
https://www.surveymonkey.com/s/B5KGWWN
