fio p3700
Single Intel P3700 as blockdevice (Test ID 1) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
256029
0
0
0
0
0
0
0
0
0
0
0
346422
519263
0
0
0
0
0
0
0
0
0
0
267755
624826
786552
0
0
0
0
0
0
0
0
0
267973
581384
775120
791217
0
0
0
0
0
0
0
0
268822
599387
708861
788781
739068
0
0
0
0
0
0
0
365027
584018
795733
799381
800688
766722
0
0
0
0
0
0
354846
593888
799985
794830
787551
779526
746461
0
0
0
0
0
356767
554131
795268
797281
800113
787587
770786
786748
0
0
0
0
359882
587587
772753
800599
797998
770760
776497
795593
768156
0
0
0
347958
560328
731077
788354
799963
721749
770139
799044
800623
797635
0
0
171948
307324
485195
516459
774668
759083
766156
803035
805624
805636
806616
0
94322
180885
282609
374258
511783
499097
647519
800271
807286
806983
801922
889158
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
750000 600000 450000 300000 150000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
11456
0
0
0
0
0
0
0
0
0
0
0
4704
5600
0
0
0
0
0
0
0
0
0
0
2864
2128
13632
0
0
0
0
0
0
0
0
0
1416
1112
7264
7328
0
0
0
0
0
0
0
0
708
612
5984
2448
24448
0
0
0
0
0
0
0
221
318
604
1288
5344
11456
0
0
0
0
0
0
159
153
1004
892
2544
7776
28544
0
0
0
0
0
81
122
153
294
676
3120
20608
16320
0
0
0
0
43
59
163
143
652
804
6560
9664
51456
0
0
0
21
29
41
85
231
510
1144
7008
8160
59648
0
0
25
25
34
116
120
302
434
692
2896
10048
12736
0
21
23
28
44
102
209
219
482
644
2992
7264
48896
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.5
0
0
0
0
0
0
0
0
0
0
0
2.4
5.7
0
0
0
0
0
0
0
0
0
0
3.4
4.8
8.4
0
0
0
0
0
0
0
0
0
3.4
5.7
8.1
11
0
0
0
0
0
0
0
0
3.4
5.1
8.4
11.3
11.4
0
0
0
0
0
0
0
2.4
5.3
7.8
10.9
12.5
13.9
0
0
0
0
0
0
2.4
5.7
7.8
11.3
13.6
13.2
18.6
0
0
0
0
0
2.4
5.5
7.5
9.7
14.3
16.5
18.4
20.2
0
0
0
0
2.4
5
7.8
9.9
12.7
11.3
17.6
21.4
22.8
0
0
0
2.4
4.2
6.8
9.9
13.6
10.7
20.4
28.2
32.4
32.1
0
0
2.2
3.4
6
8.3
14.4
15.9
18.8
33.3
36.7
37.7
41.2
0
1.8
2.7
4.5
8.4
16.3
18.5
37.5
33.6
38.2
41.4
39.5
19.9
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
50000 40000 30000 20000 10000 0
100 80 60 40 20 0
Single Intel P3700 as blockdevice (Test ID 1) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
280405
0
0
0
0
0
0
0
0
0
0
0
254270
426264
0
0
0
0
0
0
0
0
0
0
310380
373284
449100
0
0
0
0
0
0
0
0
0
315246
436663
435060
427019
0
0
0
0
0
0
0
0
297164
418849
430720
421095
422755
0
0
0
0
0
0
0
287777
427630
432562
428731
434060
442203
0
0
0
0
0
0
295324
424433
429617
445369
444978
435725
427592
0
0
0
0
0
321287
432201
430184
434773
428865
428781
435344
448747
0
0
0
0
309930
402558
433617
424669
427556
427683
430821
434163
432438
0
0
0
241990
355571
412591
426282
431078
434550
434443
423902
430347
438040
0
0
149836
251556
349617
346156
421824
424742
419579
421204
421800
427838
436784
0
70237
147903
233342
316906
382069
405727
419522
439682
422128
430309
427937
423927
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
400000 320000 240000 160000 80000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
15424
0
0
0
0
0
0
0
0
0
0
0
7584
18304
0
0
0
0
0
0
0
0
0
0
3376
4448
16512
0
0
0
0
0
0
0
0
0
908
2448
4448
11712
0
0
0
0
0
0
0
0
732
1512
2512
5856
19328
0
0
0
0
0
0
0
422
900
1608
2864
10688
27008
0
0
0
0
0
0
205
494
980
1560
4192
15552
42240
0
0
0
0
0
79
237
532
924
1704
6752
22400
46848
0
0
0
0
45
124
274
604
1020
2008
8384
23936
52992
0
0
0
38
55
114
330
532
1080
2128
10048
26496
51456
0
0
27
33
54
197
306
652
1432
1912
8640
25472
23424
0
38
28
45
50
133
540
1128
1048
2192
7584
20864
56576
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.8
0
0
0
0
0
0
0
0
0
0
0
2.9
5.3
0
0
0
0
0
0
0
0
0
0
2.4
4.4
5.1
0
0
0
0
0
0
0
0
0
2.4
3.8
4.7
6.7
0
0
0
0
0
0
0
0
2.5
3.6
4.7
6.3
7.5
0
0
0
0
0
0
0
2.7
3.9
4.7
6.8
8.1
9.4
0
0
0
0
0
0
2.6
3.7
4.7
7.2
8
9.5
10.4
0
0
0
0
0
2.4
3.6
4.5
6.3
8.5
9.5
10.5
14.1
0
0
0
0
2.3
3.7
4.7
6.4
8.3
10
11.4
14.2
14.5
0
0
0
2.3
3.5
4.7
6.3
8.6
10.6
13.2
14.1
15.2
15.3
0
0
1.9
3.1
4.7
6.3
9.4
10.5
13.7
15.7
16.3
16.6
17.7
0
1.6
2.4
4.1
7.1
11.4
12.9
16.4
17.7
17.2
17.7
18
18.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
50000 40000 30000 20000 10000 0
100 80 60 40 20 0
Single Intel P3700 as blockdevice (Test ID 1) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
298272
0
0
0
0
0
0
0
0
0
0
0
305167
348254
0
0
0
0
0
0
0
0
0
0
239800
344149
345364
0
0
0
0
0
0
0
0
0
342742
346467
347931
344192
0
0
0
0
0
0
0
0
261193
348518
347791
344159
345424
0
0
0
0
0
0
0
290005
346993
348679
344676
347490
344904
0
0
0
0
0
0
271925
337964
347711
346372
346343
346145
344444
0
0
0
0
0
174437
279643
337824
346973
347771
346697
348283
347580
0
0
0
0
97354
174734
278462
336493
347490
347802
348126
346901
345660
0
0
0
51104
97330
173963
276200
336517
347296
348524
326566
348031
347333
0
0
26390
51239
95803
171111
268920
323133
341992
328781
340071
329509
346029
0
13290
26217
50882
95001
169969
262296
315873
339995
344525
322213
327793
337472
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
300000 240000 180000 120000 60000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
10432
0
0
0
0
0
0
0
0
0
0
0
5344
9408
0
0
0
0
0
0
0
0
0
0
3312
4016
8768
0
0
0
0
0
0
0
0
0
1064
2064
5344
10816
0
0
0
0
0
0
0
0
780
1256
2640
5664
20608
0
0
0
0
0
0
0
406
740
1480
2960
12480
34560
0
0
0
0
0
0
270
430
740
1560
4704
20608
47360
0
0
0
0
0
195
262
430
732
1720
8096
30592
51456
0
0
0
0
163
193
262
434
732
2128
10176
30336
54016
0
0
0
149
163
195
262
426
748
1848
11584
39680
39168
0
0
135
149
165
197
290
716
1128
1800
10048
27520
74240
0
126
135
149
165
199
298
996
1096
1672
9536
28032
64256
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.4
0
0
0
0
0
0
0
0
0
0
0
2.4
5.8
0
0
0
0
0
0
0
0
0
0
3.2
4.1
4.5
0
0
0
0
0
0
0
0
0
2.4
3.5
4.1
5.7
0
0
0
0
0
0
0
0
3.3
3.3
4.2
5.6
6.7
0
0
0
0
0
0
0
2.9
3.2
4.3
5.5
6.6
8
0
0
0
0
0
0
2.2
3.2
4.1
5.6
7
7.8
8.7
0
0
0
0
0
1.8
2.9
4.2
5.6
7.5
8.1
8.1
8.7
0
0
0
0
1.4
2.3
3.7
5.3
7
8.4
8.9
9.5
9.7
0
0
0
1
1.5
2.8
4.7
7.1
8.5
10.8
11.4
10.1
11
0
0
0.7
1
1.8
3.3
6
9
11.8
12.5
12.5
13
12.4
0
0.5
0.7
1.1
2
3.7
7.3
11.5
13.3
13.5
13.5
14.1
14.2
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
100 80 60 40 20 0
Single Intel P3700 as blockdevice (Test ID 1) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
221621
0
0
0
0
0
0
0
0
0
0
0
239466
241976
0
0
0
0
0
0
0
0
0
0
229135
240992
241929
0
0
0
0
0
0
0
0
0
230124
237790
244619
247507
0
0
0
0
0
0
0
0
240424
244616
243282
242684
238128
0
0
0
0
0
0
0
239926
242380
243103
243913
238227
240504
0
0
0
0
0
0
234628
237013
235471
241346
241128
238701
241588
0
0
0
0
0
229833
239565
239179
239460
244298
238450
241593
237798
0
0
0
0
225836
239122
235217
245294
238742
239051
238785
244371
239983
0
0
0
190617
227467
241104
241531
242743
236788
243594
235770
243050
242005
0
0
122292
191308
226306
241992
238757
240778
242233
246499
244357
240733
238479
0
69130
115599
187276
220850
241462
235602
239768
242331
232173
242262
242432
238746
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
200000 150000 100000 50000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
20864
0
0
0
0
0
0
0
0
0
0
0
15296
20864
0
0
0
0
0
0
0
0
0
0
4640
14272
21120
0
0
0
0
0
0
0
0
0
3024
5088
9408
20864
0
0
0
0
0
0
0
0
1800
3248
5408
10176
35072
0
0
0
0
0
0
0
1128
2064
3152
5792
18304
47360
0
0
0
0
0
0
580
1480
2096
3376
7648
27264
62208
0
0
0
0
0
426
908
1416
2064
3440
11456
46336
67072
0
0
0
0
227
498
916
1272
2096
3984
15040
53504
99840
0
0
0
43
262
454
820
1384
2128
3952
16192
66048
79360
0
0
31
55
255
434
1004
1400
2288
3664
20608
49408
37632
0
27
39
44
266
426
1048
1432
2448
3792
17280
17024
42752
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.2
0
0
0
0
0
0
0
0
0
0
0
2.9
4.7
0
0
0
0
0
0
0
0
0
0
2.5
2.9
3.9
0
0
0
0
0
0
0
0
0
2.3
2.8
3.7
5.2
0
0
0
0
0
0
0
0
2.1
2.9
3.5
5.1
5.8
0
0
0
0
0
0
0
2.2
2.8
3.7
5.3
5.7
7
0
0
0
0
0
0
2.3
2.7
3.5
4.7
5.9
6.7
7.1
0
0
0
0
0
2.1
2.8
3.3
4.6
6
6.6
6.7
8.7
0
0
0
0
2.1
2.8
3.4
4.5
5.7
6.9
7
7.5
8.4
0
0
0
2.1
2.8
3.4
4.5
5.9
6.7
9.6
9.6
10.3
9.3
0
0
1.8
2.7
3.5
4.8
6
7
9.9
11.1
11.2
11.4
12.7
0
1.3
2.1
3.5
4.8
5.7
7.7
10
11.3
12
11.7
12.4
15.5
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
80000 60000 40000 20000 0
100 80 60 40 20 0
Single Intel P3700 as blockdevice (Test ID 1) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
72585
0
0
0
0
0
0
0
0
0
0
0
63511
71809
0
0
0
0
0
0
0
0
0
0
70294
73494
72233
0
0
0
0
0
0
0
0
0
67787
71980
74277
66752
0
0
0
0
0
0
0
0
71780
71827
74825
67256
61688
0
0
0
0
0
0
0
61603
71507
72105
70623
61622
61083
0
0
0
0
0
0
44916
63290
70453
73181
64722
61733
68071
0
0
0
0
0
29833
44973
62780
68361
68126
60994
62137
61770
0
0
0
0
18900
30029
45144
60851
60277
61014
60849
61322
61238
0
0
0
11365
18948
29876
43827
60278
60348
61259
61589
61551
61077
0
0
6554
11359
18772
26777
44666
54571
60575
61393
60888
60970
62040
0
3521
6573
11375
17973
27582
38854
52691
64109
61018
60770
60687
62079
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
20096
0
0
0
0
0
0
0
0
0
0
0
8768
17536
0
0
0
0
0
0
0
0
0
0
5216
8256
15680
0
0
0
0
0
0
0
0
0
3856
6560
9920
24960
0
0
0
0
0
0
0
0
2128
4448
5792
13376
38144
0
0
0
0
0
0
0
474
2352
4704
6368
18816
56576
0
0
0
0
0
0
217
410
2416
4128
8768
36096
83456
0
0
0
0
0
38
213
442
2960
4768
12864
22144
81408
0
0
0
0
32
60
199
470
3952
5728
14400
62720
89600
0
0
0
28
36
74
213
478
2672
5344
17024
38144
117248
0
0
26
36
63
64
187
604
2544
5408
14912
50432
30336
0
43
26
29
62
83
239
676
2352
5344
19584
22400
41728
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6
0
0
0
0
0
0
0
0
0
0
0
3.2
5
0
0
0
0
0
0
0
0
0
0
2.4
3
3.5
0
0
0
0
0
0
0
0
0
2.6
2.9
3.8
4.1
0
0
0
0
0
0
0
0
2.2
2.8
3.5
3.9
4.7
0
0
0
0
0
0
0
2
2.8
3.5
4.2
4.6
5.4
0
0
0
0
0
0
1.8
2.5
3.4
4.3
5
5.3
6.3
0
0
0
0
0
1.4
2.1
3
4
5.1
5.2
6.1
6.3
0
0
0
0
1.1
1.7
2.5
3.7
4.7
5.8
5.9
7.4
9.8
0
0
0
0.8
1.2
1.7
2.9
4.5
6
7.1
8.2
9.3
9.6
0
0
0.7
0.9
1.3
2
3.6
5.4
9
10
10.6
11.2
8.6
0
0.6
0.6
1
1.4
2.4
4.4
7.5
10.4
10.3
10.7
9.5
7.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
100000 80000 60000 40000 20000 0
100 80 60 40 20 0
Single Intel P3700 as blockdevice (Test ID 1) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
2.88744e+06
0
2.88465e+06 2.86110e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2.88418e+06 2.86078e+06 2.79512e+06
2.89414e+06 2.87225e+06 2.79008e+06 2.7075e+06
2.89652e+06 2.85811e+06 2.79795e+06 2.69699e+06 2.44837e+06
2.87764e+06 2.86558e+06 2.79296e+06 2.68713e+06 2.44826e+06 2.03229e+06
2.87652e+06 2.86971e+06 2.77446e+06 2.66132e+06 2.4454e+06 2.0537e+06 2.02805e+06
2.71224e+06 2.87046e+06 2.77307e+06 2.63865e+06 2.43594e+06 2.06304e+06 2.07765e+06 2.02471e+06
2.16223e+06 2.7217e+06 2.77726e+06 2.6537e+06 2.4162e+06 2.08724e+06 2.11126e+06 2.01972e+06 1.81982e+06
1.48001e+06 2.17954e+06 2.64081e+06 2.64448e+06 2.4271e+06 2.18585e+06 1.78915e+06 1.78756e+06 1.85821e+06 1.73073e+06 886080
1.3702e+06 2.08118e+06 2.47858e+06 2.29258e+06 2.17275e+06 2.03062e+06 2.04848e+06 1.99019e+06 1.76631e+06 1.74258e+06
510860
748505
1
2
0
1.20069e+06 1.76786e+06 2.24293e+06 2.36908e+06 2.31238e+06 2.05856e+06 1.94067e+06 1.70225e+06 1.76401e+06 1.75475e+06
4
8
16
32
64
128
number of workers
256
512
1024
2048
2500000 2000000 1500000 1000000 500000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
222208
0
0
0
0
0
0
0
0
0
0
0
116224
183296
0
0
0
0
0
0
0
0
0
0
51968
95744
261120
0
0
0
0
0
0
0
0
0
18816
46848
122368
238592
0
0
0
0
0
0
0
0
10816
23168
54528
116224
317440
0
0
0
0
0
0
0
6112
12480
26240
65280
150528
643072
0
0
0
0
0
0
3632
6624
13504
32640
63744
342016
522240
0
0
0
0
0
2832
3664
7072
14400
31360
112128
261120
468992
0
0
0
0
2736
2672
3760
7264
13632
38656
127488
211968
684032
0
0
0
2608
2352
2640
4048
6944
15936
46336
130560
317440
358400
0
0
2672
2192
2384
2736
3856
8256
17280
44288
136192
288768
317440
0
2512
2192
2512
2672
2192
4512
8768
20352
68096
150528
211968
444416
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
1.9
0
0
0
0
0
0
0
0
0
0
0
1.4
1.7
0
0
0
0
0
0
0
0
0
0
1.1
1.1
1.3
0
0
0
0
0
0
0
0
0
1.1
1.2
1.2
1.4
0
0
0
0
0
0
0
0
1.1
1.1
1.2
1.4
1.5
0
0
0
0
0
0
0
1.1
1
1.1
1.4
1.6
2.6
0
0
0
0
0
0
0.9
0.9
1
1.3
1.4
2.2
1.7
0
0
0
0
0
0.8
0.8
0.9
1.1
1.4
1.7
1.7
1.9
0
0
0
0
0.7
0.8
0.9
1.1
1.3
2.1
1.7
2
2.2
0
0
0
0.6
0.7
0.8
1.1
1.3
1.7
2.4
2.3
2.6
2.7
0
0
0.6
0.7
0.8
1.1
1.6
2
1.9
2
1.9
2.8
3.1
0
0.5
0.7
0.8
1
1.4
1.3
1.4
1.8
2.2
2.9
3.2
3
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000 450000 300000 150000 0
100 80 60 40 20 0
Single Intel P3700 as blockdevice (Test ID 1) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
1.80158e+06
0
1.91423e+06 1.94956e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1.93276e+06 1.96464e+06 1.92568e+06
1.95161e+06 1.94358e+06 1.96863e+06 1.93779e+06
1.99043e+06 1.96502e+06 1.98125e+06 1.87797e+06 1.79551e+06
1.9239e+06 1.98631e+06 1.76228e+06 1.90543e+06 1.86659e+06 1.89441e+06
1.96485e+06 1.79906e+06 1.80192e+06 1.92581e+06 1.77363e+06 1.80378e+06 1.80447e+06
1.98059e+06 1.76624e+06 1.83605e+06 1.92872e+06 1.62272e+06 1.87662e+06 1.86374e+06 1.89361e+06
1.98313e+06 1.80142e+06 1.89265e+06 1.96099e+06 1.6416e+06 1.62252e+06 1.79764e+06 1.83858e+06 1.67419e+06
1.96796e+06 1.9246e+06 1.90645e+06 1.8916e+06 1.62461e+06 1.63281e+06 1.93308e+06 1.88647e+06 1.67658e+06 1.69118e+06
1.95841e+06 1.75507e+06 1.79585e+06 1.83123e+06 1.58691e+06 1.64312e+06 1.63578e+06 1.89421e+06 1.75235e+06 1.72336e+06 1.70013e+06
2
4
8
16
32
64
128
number of workers
1200000 800000 400000
0
1.7783e+06 1.6971e+06 1.88297e+06 1.80575e+06 1.89764e+06 1.83525e+06 1.67181e+06 1.82459e+06 1.81684e+06 1.70574e+06 1.71034e+06 1.70672e+06
1
1600000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
354304
0
0
0
0
0
0
0
0
0
0
0
121344
419840
0
0
0
0
0
0
0
0
0
0
55552
128512
276480
0
0
0
0
0
0
0
0
0
32640
62208
142336
325632
0
0
0
0
0
0
0
0
20608
33024
76288
138240
452608
0
0
0
0
0
0
0
11712
23168
44800
68096
173056
749568
0
0
0
0
0
0
6240
12608
23680
38144
97792
325632
798720
0
0
0
0
0
4080
7520
11840
19840
48896
110080
317440
585728
0
0
0
0
2024
3792
6432
10944
21120
48896
119296
313344
1.30253e+06
0
0
0
1416
2288
3856
6624
12352
29056
39168
115200
552960
716800
0
0
708
1816
2096
4048
7520
13888
27776
39168
185344
415744
497664
0
245
1480
1192
2352
3856
6624
13504
24448
57600
292864
201728
1.1223e+06
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2
0
0
0
0
0
0
0
0
0
0
0
2.1
2.6
0
0
0
0
0
0
0
0
0
0
2.2
2.5
2.5
0
0
0
0
0
0
0
0
0
1.7
2.4
2.7
2.9
0
0
0
0
0
0
0
0
1.1
1.9
2.6
2.4
3
0
0
0
0
0
0
0
1
1.1
1.4
2.2
3
3.1
0
0
0
0
0
0
1
1
1.1
1.8
2.6
2.4
3.1
0
0
0
0
0
0.9
0.8
1.1
1.4
1.9
2.8
2.7
3.7
0
0
0
0
0.9
0.9
1
1.2
1.4
2
2.5
3
4.2
0
0
0
0.9
0.9
1
1.1
1.3
1.3
2.3
3.2
3.5
4.5
0
0
0.9
0.9
1
1.1
1.2
1.2
1.2
2.5
2.8
3.4
4.2
0
0.9
0.8
1.1
1.2
1.3
1.6
1.2
1.5
3
3.2
4.2
5.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1250000 1000000 750000 500000 250000 0
100 80 60 40 20 0
Software RAID0 over 2 Intel P3700 as blockdevice (Test ID 2) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
307397
0
0
0
0
0
0
0
0
0
0
0
311454
503716
0
0
0
0
0
0
0
0
0
0
266221
544151
966878
0
0
0
0
0
0
0
0
0
328308
523688
734552
997535
0
0
0
0
0
0
0
0
259234
535056
827291
1.00290e+06
989386
0
0
0
0
0
0
0
328639
535940
937745
1.0025e+06
990349
965226
0
0
0
0
0
0
325982
527752
981115
1.00046e+06 1.00202e+06
978152
997395
0
0
0
0
0
284939
485305
790088
975725
1.00128e+06
974202
1.00084e+06 1.00012e+06
0
0
0
0
167628
307142
494102
805836
987589
961569
1.00086e+06 1.00109e+06 1.0017e+06
0
0
0
86342
167914
297679
513018
808522
691755
0
0
41528
87006
162540
294023
477645
581477
777399
997428
992985
0
18352
42806
83410
157897
281924
437690
486324
891782
999859
930063
1
2
4
8
16
32
64
128
256
512
1024
2048
983578
1.00118e+06 1.00234e+06 1.00057e+06
number of workers
1.00216e+06 1.00018e+06
1.9822e+06 1.00143e+06
1600000 1200000 800000 400000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
7264
0
0
0
0
0
0
0
0
0
0
0
3472
4768
0
0
0
0
0
0
0
0
0
0
2384
2160
4512
0
0
0
0
0
0
0
0
0
908
1240
52480
8896
0
0
0
0
0
0
0
0
660
684
3632
4768
14912
0
0
0
0
0
0
0
310
386
852
2192
9024
16512
0
0
0
0
0
0
217
258
498
1384
3248
12992
35584
0
0
0
0
0
163
205
290
604
916
3792
8512
21888
0
0
0
0
137
159
326
306
478
1560
2608
12608
35584
0
0
0
133
135
181
209
278
1032
1256
4576
17280
36096
0
0
157
124
145
163
302
940
908
812
4448
11712
33536
0
171
116
133
147
179
418
1272
548
828
4512
4768
15424
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.4
0
0
0
0
0
0
0
0
0
0
0
2.4
5.7
0
0
0
0
0
0
0
0
0
0
3.4
5.8
10
0
0
0
0
0
0
0
0
0
2.3
5.8
8.3
13.1
0
0
0
0
0
0
0
0
3.4
5.8
9.2
11.8
15.4
0
0
0
0
0
0
0
2.4
5.8
8.9
12.6
15.9
15.6
0
0
0
0
0
0
2.4
5.8
8.4
13.1
16.8
18.6
21.6
0
0
0
0
0
2.5
4.8
9.1
13.2
16.2
20.1
23.6
31.3
0
0
0
0
1.9
3.7
6.5
11.7
17.7
20.1
24.7
32.5
35.7
0
0
0
1.4
2.3
4.2
9.1
16
13.7
32.3
41.1
41.4
45.6
0
0
0.9
1.5
2.6
5.7
10.6
15.8
28.2
45
48.3
49.4
46.2
0
0.8
1.1
1.6
3.3
6.7
12.4
18.5
45
48.8
47.7
50
50
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
50000 40000 30000 20000 10000 0
100 80 60 40 20 0
Software RAID0 over 2 Intel P3700 as blockdevice (Test ID 2) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
275551
0
0
0
0
0
0
0
0
0
0
0
253265
402084
0
0
0
0
0
0
0
0
0
0
269033
406832
407535
0
0
0
0
0
0
0
0
0
282385
411431
403804
421056
0
0
0
0
0
0
0
0
281836
456631
400182
423950
452861
0
0
0
0
0
0
0
285070
494171
392063
424596
423248
417955
0
0
0
0
0
0
283548
514343
408322
423038
422143
417417
414601
0
0
0
0
0
277372
481041
400601
418967
422751
418321
432330
414239
0
0
0
0
286510
432509
508600
426464
422011
415894
414590
410600
414131
0
0
0
233126
370709
464756
528644
444213
419880
429828
412396
410976
409012
0
0
129216
238098
364493
449530
513380
519615
437648
413089
412052
410004
409883
0
68067
135594
214244
293042
391005
399628
457341
416798
435865
411568
412254
424380
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
11200
0
0
0
0
0
0
0
0
0
0
0
6624
20608
0
0
0
0
0
0
0
0
0
0
2960
11072
19328
0
0
0
0
0
0
0
0
0
1720
6816
11072
31872
0
0
0
0
0
0
0
0
860
3824
6944
15424
33024
0
0
0
0
0
0
0
434
924
4704
7008
21120
53504
0
0
0
0
0
0
197
564
3184
4576
10688
25984
57600
0
0
0
0
0
106
326
2480
3088
4512
14656
44800
86528
0
0
0
0
58
129
572
1992
3056
4640
14528
43776
102912
0
0
0
55
70
153
442
1928
3088
4768
16064
34048
85504
0
0
49
54
70
139
390
1912
3088
4960
14912
49408
70144
0
50
45
66
99
197
1192
1688
3216
4960
13376
44288
55040
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.5
0
0
0
0
0
0
0
0
0
0
0
2.6
4.3
0
0
0
0
0
0
0
0
0
0
2.7
3.7
4.9
0
0
0
0
0
0
0
0
0
2.4
4
4.7
6.2
0
0
0
0
0
0
0
0
2.6
4.4
4.8
6.4
8
0
0
0
0
0
0
0
2.4
4.5
4.7
7.2
8.1
9.7
0
0
0
0
0
0
2.4
4.3
4.7
6.9
8
9.5
10.3
0
0
0
0
0
2.4
4.4
4.7
6.6
8.5
9.7
10.7
12.9
0
0
0
0
2.4
4.4
5.9
6.8
8.4
10.7
13.1
14
15.2
0
0
0
2.4
4.1
5.8
8
9.5
10.7
15.1
16
17.3
17.6
0
0
1.9
3.1
5.3
8.2
12.1
13.6
15.7
17.8
18.5
18.8
19.4
0
1.4
2.4
4
6.8
10.9
13.3
19.5
20
20.2
19.6
19.7
21.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
500000 400000 300000 200000 100000 0
100000 80000 60000 40000 20000 0
100 80 60 40 20 0
Software RAID0 over 2 Intel P3700 as blockdevice (Test ID 2) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
224351
0
0
0
0
0
0
0
0
0
0
0
282353
424596
0
0
0
0
0
0
0
0
0
0
233407
500707
598373
0
0
0
0
0
0
0
0
0
254989
518115
597827
587623
0
0
0
0
0
0
0
0
245192
530389
596174
592184
591266
0
0
0
0
0
0
0
244334
484493
596179
591969
588928
599954
0
0
0
0
0
0
291466
455761
592626
595270
592201
597684
598376
0
0
0
0
0
177868
317504
491279
590896
591540
598968
600046
589750
0
0
0
0
96255
179726
317812
487671
584564
597812
593146
590646
594892
0
0
0
49642
96704
180425
312697
470889
581044
598041
592253
598060
597941
0
0
24490
49995
96803
177350
297955
456501
546635
592058
599165
598502
598553
0
11705
25086
49544
94983
163726
287276
427151
545932
594339
598601
601358
1.21451e+06
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
13376
0
0
0
0
0
0
0
0
0
0
0
4448
6432
0
0
0
0
0
0
0
0
0
0
3248
2608
40192
0
0
0
0
0
0
0
0
0
1192
1448
17792
12608
0
0
0
0
0
0
0
0
812
1004
8896
6368
24704
0
0
0
0
0
0
0
430
596
3504
3408
14400
50432
0
0
0
0
0
0
258
406
764
1576
5088
24960
58112
0
0
0
0
0
199
258
426
764
1592
8160
35584
60672
0
0
0
0
171
197
258
414
804
1528
8384
39680
50432
0
0
0
161
169
199
258
442
892
1496
8096
18816
60160
0
0
173
155
169
201
306
486
1224
1560
8512
16320
61696
0
270
149
155
173
286
314
804
1336
1672
9536
38144
35584
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.3
0
0
0
0
0
0
0
0
0
0
0
2.4
5.7
0
0
0
0
0
0
0
0
0
0
3.3
5.2
8
0
0
0
0
0
0
0
0
0
3.3
5.4
7.6
9
0
0
0
0
0
0
0
0
3.3
5.1
7.8
9.2
11.4
0
0
0
0
0
0
0
3.3
5.5
7.9
9.5
11.2
12.8
0
0
0
0
0
0
2.4
4.2
7.2
8.9
11.4
12.8
13.7
0
0
0
0
0
2
3.3
5.9
8.7
11.8
13.4
14.2
16.8
0
0
0
0
1.4
2.6
4.4
7.8
12
14.7
16.5
17.4
19.7
0
0
0
0.9
1.6
2.9
5.5
9.9
14.7
18.8
21.6
23.4
27.2
0
0
0.7
1
1.7
3.5
6.6
11.5
18
24.8
28.2
26.1
28.5
0
0.5
0.7
1.1
2.1
4.2
7.9
13.3
23.9
27.9
30.3
30.6
32.3
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1000000 750000 500000 250000 0
60000
45000
30000
15000
0
100 80 60 40 20 0
Software RAID0 over 2 Intel P3700 as blockdevice (Test ID 2) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
206138
0
0
0
0
0
0
0
0
0
0
0
220609
205057
0
0
0
0
0
0
0
0
0
0
241003
205956
204772
0
0
0
0
0
0
0
0
0
230403
204793
205116
191622
0
0
0
0
0
0
0
0
245751
204080
215756
191629
201100
0
0
0
0
0
0
0
247914
178493
205181
213138
200947
199063
0
0
0
0
0
0
246933
186676
204932
203784
202981
198971
197095
0
0
0
0
0
236349
191989
205377
204847
202259
200077
199074
196135
0
0
0
0
224541
264241
213748
203251
202520
199964
198983
197603
196131
0
0
0
179955
253673
264297
213830
203267
201263
198598
198669
194473
193929
0
0
113462
186894
235100
260031
217210
203231
200118
196752
197881
193962
194398
0
57745
109460
162928
217126
242664
211205
200081
201182
197530
197757
206648
197906
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
250000 200000 150000 100000 50000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
17536
0
0
0
0
0
0
0
0
0
0
0
11840
37632
0
0
0
0
0
0
0
0
0
0
5088
20352
45312
0
0
0
0
0
0
0
0
0
7008
12864
22144
69120
0
0
0
0
0
0
0
0
3152
8640
13120
28800
70144
0
0
0
0
0
0
0
1560
5920
8640
15168
43264
115200
0
0
0
0
0
0
740
3440
5664
9664
19328
55040
107008
0
0
0
0
0
446
2608
3216
5536
9408
28544
69120
111104
0
0
0
0
181
708
2256
3248
5664
9920
30592
58112
164864
0
0
0
68
213
732
2256
3248
5728
9408
29056
67072
244736
0
0
53
67
241
708
2192
3248
5728
9408
38656
127488
209920
0
57
55
95
334
1012
2256
3376
5792
9408
45312
120320
96768
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.1
0
0
0
0
0
0
0
0
0
0
0
2.6
3
0
0
0
0
0
0
0
0
0
0
2.4
2.5
3.5
0
0
0
0
0
0
0
0
0
2.5
2.5
3.3
3.8
0
0
0
0
0
0
0
0
2.3
2.9
3.3
3.9
4.8
0
0
0
0
0
0
0
2.3
2.2
3.1
4.3
4.6
5.6
0
0
0
0
0
0
2.3
2.4
3.3
4.2
4.7
5.6
8.1
0
0
0
0
0
2.3
2.3
3.2
4
5
5.8
9.4
9.3
0
0
0
0
2.2
3.1
3.3
3.9
4.8
6.2
8.2
10.3
10.6
0
0
0
2.2
3.3
4.1
4.3
5.3
6.2
9.9
11.5
10
11.1
0
0
1.8
2.7
4.1
5.4
5.9
6.9
9
12.7
11.7
11.3
11.6
0
1.4
2
3.2
4.9
6.8
7.9
10.2
12.2
13
11
11.6
12.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
200000 150000 100000 50000 0
100 80 60 40 20 0
Software RAID0 over 2 Intel P3700 as blockdevice (Test ID 2) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
57012
0
0
0
0
0
0
0
0
0
0
0
63922
100212
0
0
0
0
0
0
0
0
0
0
78035
106384
109979
0
0
0
0
0
0
0
0
0
80870
108735
109102
109936
0
0
0
0
0
0
0
0
84603
109263
110363
108072
109718
0
0
0
0
0
0
0
76830
105042
110297
95583
110019
108451
0
0
0
0
0
0
56627
81663
105203
99702
109365
108532
110312
0
0
0
0
0
35092
56677
82424
99502
109978
111043
108727
110290
0
0
0
0
22484
36772
56712
80929
107082
110436
108390
110876
106808
0
0
0
13424
22778
36257
55935
82059
105018
109028
107715
108762
107110
0
0
7366
13248
22291
35497
55862
80211
102620
109492
107809
107535
109603
0
3743
7223
12956
21459
35266
53724
76203
100794
108149
107879
108141
110435
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
13376
0
0
0
0
0
0
0
0
0
0
0
6368
15424
0
0
0
0
0
0
0
0
0
0
3056
10432
20352
0
0
0
0
0
0
0
0
0
1432
6880
10560
22912
0
0
0
0
0
0
0
0
860
5280
7456
14656
44800
0
0
0
0
0
0
0
362
1848
5280
10048
20608
52992
0
0
0
0
0
0
92
346
1896
7008
10688
26496
60672
0
0
0
0
0
72
67
322
2832
5344
13888
33024
72192
0
0
0
0
46
44
60
382
1624
5344
17792
40192
102912
0
0
0
30
35
45
73
342
1720
5600
17280
45312
86528
0
0
29
31
38
55
78
378
1752
5216
14016
31360
77312
0
76
46
43
51
59
126
516
1528
5408
13248
29568
51968
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.2
0
0
0
0
0
0
0
0
0
0
0
3.1
4.8
0
0
0
0
0
0
0
0
0
0
2.7
4.4
5.4
0
0
0
0
0
0
0
0
0
2.8
4.3
5.2
6.3
0
0
0
0
0
0
0
0
2.4
4.2
5.4
6.3
7.6
0
0
0
0
0
0
0
2.5
4.2
5.3
5.9
8
8.7
0
0
0
0
0
0
2.2
3.6
4.9
5.8
7.8
8.6
9.6
0
0
0
0
0
1.6
2.7
4.4
5.9
8
9
10
10.9
0
0
0
0
1.3
2
3.3
5
7.9
9.4
9.9
11.1
12.8
0
0
0
1
1.4
2.3
3.8
6
9.2
12.6
12.8
13.7
13.7
0
0
0.7
1
1.5
2.8
4.2
7.3
11.7
13.8
14
14.3
15.4
0
0.5
0.7
1.2
2
2.9
5.7
10.2
12.9
14.7
15.1
15.2
18.2
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
100000 80000 60000 40000 20000 0
100000 80000 60000 40000 20000 0
100 80 60 40 20 0
Software RAID0 over 2 Intel P3700 as blockdevice (Test ID 2) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
4.89307e+06
0
4.9439e+06 4.85367e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
4.93545e+06 4.84372e+06 4.51246e+06
4.94753e+06 4.78111e+06 4.84164e+06 4.43713e+06
4.9589e+06 4.79824e+06 4.63017e+06 4.79108e+06 4.70191e+06
4.9522e+06 4.80898e+06 4.75522e+06 4.78608e+06 4.44853e+06 4.48005e+06
4.93659e+06 4.80607e+06 4.80432e+06 4.80291e+06 4.64836e+06 4.10642e+06 3.48404e+06
4.49171e+06 4.79652e+06 4.83527e+06 4.77069e+06 4.63361e+06 3.60013e+06 3.95305e+06 3.55782e+06
3.20125e+06 4.28066e+06 4.79443e+06 4.77253e+06 4.70836e+06 4.54377e+06 4.09115e+06 3.63536e+06 2.96295e+06
1.92767e+06 2.96119e+06 4.166e+06 4.71382e+06 4.74354e+06 4.21355e+06 3.86335e+06 3.86525e+06 3.67262e+06 3.2085e+06
1.8148e+06 1.61112e+06 2.60788e+06 3.80539e+06 4.51706e+06 4.23774e+06 4.21539e+06 4.08426e+06 2.99179e+06 3.33501e+06 2.92809e+06 996595
677184
1
2
8
16
32
64
128
number of workers
3000000 2000000 1000000
0
1.29615e+06 2.24717e+06 3.29354e+06 4.2549e+06 3.66621e+06 4.02089e+06 3.80381e+06 3.13781e+06 2.96339e+06 2.8404e+06
4
4000000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
228352
0
0
0
0
0
0
0
0
0
0
0
100864
252928
0
0
0
0
0
0
0
0
0
0
44288
125440
230400
0
0
0
0
0
0
0
0
0
23936
56064
100864
222208
0
0
0
0
0
0
0
0
12480
27776
52992
102912
195584
0
0
0
0
0
0
0
5728
13504
27520
55552
160768
313344
0
0
0
0
0
0
2544
6432
12608
28800
51456
120320
456704
0
0
0
0
0
1208
3120
5856
12736
27776
58112
140288
468992
0
0
0
0
660
1752
2672
5856
12736
23680
69120
195584
675840
0
0
0
510
940
1304
2736
5792
13632
40704
68096
150528
481280
0
0
378
580
788
1288
3024
5344
15040
40192
102912
284672
864256
0
342
596
708
948
2640
3248
8512
16192
40192
252928
518144
864256
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
1.9
0
0
0
0
0
0
0
0
0
0
0
1.7
1.9
0
0
0
0
0
0
0
0
0
0
1.6
1.9
2
0
0
0
0
0
0
0
0
0
1.6
1.8
1.9
2.1
0
0
0
0
0
0
0
0
1.6
1.8
2
2
2.5
0
0
0
0
0
0
0
1.6
1.8
2
2
2.5
2.7
0
0
0
0
0
0
1.3
1.5
1.8
1.9
2
3.4
3
0
0
0
0
0
1.1
1.3
1.5
1.7
1.9
3.5
2.5
3.5
0
0
0
0
0.9
1.2
1.4
1.6
1.8
2.6
2.5
3.2
4.3
0
0
0
0.7
0.9
1.4
1.6
1.7
4
3.2
3.3
2.7
3.8
0
0
0.7
0.8
1.1
1.4
1.8
2.9
2.3
2.7
4.3
3.4
4.2
0
0.7
1
1
1.3
1.5
3.8
3
2.6
3.1
4.5
3.9
5.3
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
750000 600000 450000 300000 150000 0
100 80 60 40 20 0
Software RAID0 over 2 Intel P3700 as blockdevice (Test ID 2) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
3.65564e+06
0
3.7321e+06 3.87017e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3.5873e+06 3.90936e+06 3.86691e+06
3.80695e+06 3.96279e+06 3.88114e+06 3.82427e+06
3.90882e+06 3.83339e+06 3.88482e+06 3.76341e+06 3.53947e+06
3.87358e+06 3.70626e+06 3.90898e+06 3.83327e+06 3.53885e+06 3.70365e+06
3.79962e+06 3.85354e+06 3.8945e+06 3.81356e+06 3.29970e+06 3.55574e+06 3.53603e+06
3.7595e+06 3.85662e+06 3.84688e+06 3.82604e+06 2.979e+06 3.27476e+06 3.61812e+06 3.57714e+06
3.67305e+06 3.63026e+06 3.61676e+06 3.75148e+06 2.52123e+06 3.30353e+06 3.73167e+06 3.71855e+06 3.20618e+06
3.54962e+06 3.59608e+06 3.38457e+06 3.69642e+06 3.19612e+06 3.22396e+06 3.63649e+06 3.48486e+06 3.47553e+06 3.30626e+06
3.23657e+06 3.47368e+06 3.34753e+06 3.63062e+06 3.53086e+06 2.88346e+06 3.51636e+06 3.30487e+06 3.44247e+06 3.15575e+06 3.27716e+06
2
4
8
16
32
64
128
number of workers
2400000 1600000 800000
0
1.95704e+06 3.23387e+06 3.44434e+06 3.33961e+06 3.57423e+06 2.94401e+06 3.40161e+06 2.95038e+06 3.4745e+06 3.28428e+06 3.38769e+06 3.21579e+06
1
3200000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
88576
0
0
0
0
0
0
0
0
0
0
0
45312
191488
0
0
0
0
0
0
0
0
0
0
27776
98816
199680
0
0
0
0
0
0
0
0
0
24704
48384
86528
284672
0
0
0
0
0
0
0
0
17024
38144
52480
134144
468992
0
0
0
0
0
0
0
9280
20096
30592
75264
183296
489472
0
0
0
0
0
0
5472
10944
16512
41216
142336
313344
350208
0
0
0
0
0
2768
5216
10688
18304
62720
102912
325632
626688
0
0
0
0
1736
2864
6048
10688
37632
44800
166912
284672
536576
0
0
0
948
1832
3312
6112
12864
24192
35584
171008
403456
937984
0
0
426
1176
2448
3184
6176
19072
22400
55040
211968
675840
724992
0
78
346
1128
2008
3088
9536
12992
31872
41216
240640
489472
815104
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.4
0
0
0
0
0
0
0
0
0
0
0
2.6
3.8
0
0
0
0
0
0
0
0
0
0
2.6
3.7
3.3
0
0
0
0
0
0
0
0
0
2.2
2.9
3.3
4.8
0
0
0
0
0
0
0
0
1.9
2.6
3.4
3.3
3.9
0
0
0
0
0
0
0
1.9
1.9
3
4
3.9
4.3
0
0
0
0
0
0
1.6
1.8
2
2.9
2.7
3.5
6.1
0
0
0
0
0
1.4
1.6
1.7
1.9
2.2
2.7
4.9
5.2
0
0
0
0
1.3
1.5
1.7
1.8
1.5
2.3
4.5
4.3
6.9
0
0
0
1.4
1.5
1.6
1.8
1.7
2
4
4.6
5.1
6.5
0
0
1.3
1.5
1.5
1.8
1.9
2.1
3
3.3
4.6
5.1
6.7
0
0.9
1.4
1.6
1.7
1.8
2.3
2.4
2.8
4.7
4.5
5.2
6.6
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
800000 600000 400000 200000 0
100 80 60 40 20 0
Software RAID0 over 4 Intel P3700 as blockdevice (Test ID 3) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
267006
0
0
0
0
0
0
0
0
0
0
0
254636
517426
0
0
0
0
0
0
0
0
0
0
278852
528254
905826
0
0
0
0
0
0
0
0
0
277554
515683
854269
1.56227e+06
0
0
0
0
0
0
0
0
278533
520345
938523
1.53923e+06 1.89874e+06
0
0
0
0
0
0
0
285924
496290
875160
1.57784e+06 1.89662e+06 1.84889e+06
0
0
0
0
0
0
264140
470355
716661
1.46966e+06 1.90722e+06 1.8662e+06 1.91054e+06
0
0
0
0
0
255112
415440
764542
1.34475e+06 1.88907e+06 1.8602e+06 1.88699e+06 1.90772e+06
0
0
0
0
159544
320869
566456
937075
0
0
0
81117
161468
327164
563851
895731
0
0
42897
83208
168695
308285
517612
637678
937950
19918
41424
83821
163706
296032
435169
596354
931740
1
2
4
8
16
32
64
128
1.46176e+06 1.71538e+06 1.88004e+06 1.9082e+06 1.70294e+06
1.04486e+06 1.6958e+06 1.90833e+06 1.63446e+06 1.90428e+06
1.75302e+06 1.90754e+06 1.90564e+06 1.90777e+06
number of workers
1.79241e+06 1.89962e+06 2.46368e+06
2000000 1500000 1000000 500000
0 940922
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
8384
0
0
0
0
0
0
0
0
0
0
0
5792
4960
0
0
0
0
0
0
0
0
0
0
2040
2416
3952
0
0
0
0
0
0
0
0
0
1096
1272
2192
8512
0
0
0
0
0
0
0
0
612
668
1144
3760
8896
0
0
0
0
0
0
0
426
410
620
1784
4512
9664
0
0
0
0
0
0
262
298
2064
804
1864
5984
18304
0
0
0
0
0
171
205
1080
524
860
1592
7136
28800
0
0
0
0
149
153
199
290
438
1096
2024
19840
9152
0
0
0
143
147
141
171
326
684
612
2416
7200
17280
0
0
135
131
133
151
233
708
1112
516
2512
14912
25216
0
125
133
131
141
181
274
892
812
502
2800
812
8384
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6
0
0
0
0
0
0
0
0
0
0
0
3.6
6.3
0
0
0
0
0
0
0
0
0
0
3.6
6.2
10.6
0
0
0
0
0
0
0
0
0
3.6
6
10
19.1
0
0
0
0
0
0
0
0
3.6
6
11.1
19.1
26.6
0
0
0
0
0
0
0
2.8
6.3
10.2
19.1
27.3
36.2
0
0
0
0
0
0
3.2
6
9.1
18.5
28.2
34.9
45.1
0
0
0
0
0
3
4.9
8.9
17.3
27.2
32.3
52.6
63.3
0
0
0
0
2.4
4.2
8
14
23.1
33.7
60.8
64.7
37
0
0
0
1.7
3.2
5.5
11.2
18.2
25.6
58.9
69
58.1
71.2
0
0
1.1
2
3.5
6.9
12.9
20.6
40.6
71.7
73.4
72.1
69.3
0
0.8
1.2
2.1
4.2
7.9
15.5
22
51.3
73.4
76.2
70.7
44.5
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
25000 20000 15000 10000 5000 0
100 80 60 40 20 0
Software RAID0 over 4 Intel P3700 as blockdevice (Test ID 3) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
255422
0
0
0
0
0
0
0
0
0
0
0
261356
435200
0
0
0
0
0
0
0
0
0
0
274759
469090
782292
0
0
0
0
0
0
0
0
0
274041
471854
794987
679341
0
0
0
0
0
0
0
0
251813
469797
706013
763292
750601
0
0
0
0
0
0
0
269105
417368
743173
758157
747286
739293
0
0
0
0
0
0
272752
380288
657415
765604
756580
760779
744472
0
0
0
0
0
266765
381316
716560
778068
695812
743788
742132
745464
0
0
0
0
255931
401816
662405
953505
648281
744392
743684
750298
744639
0
0
0
241924
359872
627266
857548
965125
949897
749133
741809
752906
747494
0
0
104220
211515
377849
567503
719337
757564
981036
740285
807988
740288
754962
0
61042
110898
209984
328481
440822
443945
452348
787636
814219
748963
742263
762842
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
10560
0
0
0
0
0
0
0
0
0
0
0
5920
6048
0
0
0
0
0
0
0
0
0
0
2128
3536
5664
0
0
0
0
0
0
0
0
0
1208
1800
9536
25472
0
0
0
0
0
0
0
0
716
876
6304
12352
43264
0
0
0
0
0
0
0
326
486
964
7328
20864
68096
0
0
0
0
0
0
149
302
1560
4576
9792
22400
66048
0
0
0
0
0
96
181
290
2704
5024
10816
23168
61184
0
0
0
0
51
101
330
964
3152
4704
10048
22144
55552
0
0
0
41
67
88
217
852
2160
4576
11584
24960
58624
0
0
48
54
62
98
310
804
1496
4768
11200
20864
41216
0
45
47
51
86
165
1112
2192
2416
4768
11840
20864
52992
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.6
0
0
0
0
0
0
0
0
0
0
0
2.6
5.5
0
0
0
0
0
0
0
0
0
0
2.8
5.2
9.6
0
0
0
0
0
0
0
0
0
2.7
5.2
9
10.2
0
0
0
0
0
0
0
0
3
5.5
8.8
11.1
12.8
0
0
0
0
0
0
0
2.7
5.2
8.6
11
13.8
16.2
0
0
0
0
0
0
2.7
4.8
8.7
12.1
14.3
16.9
19.9
0
0
0
0
0
2.7
4.8
8.5
12.2
13.9
17.7
22.3
27.4
0
0
0
0
2.9
5.4
8.6
14.9
13.2
19.9
25.6
30.7
33.2
0
0
0
2.8
5.1
8.8
14
18.9
24.3
27.1
32.2
34
36.1
0
0
2.4
4.1
6.8
11.4
17.6
23.8
40.6
33.9
37.8
36.5
38
0
1.7
2.7
5
9
14.9
17.8
22.1
38.3
39.9
38.1
37.3
38.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID0 over 4 Intel P3700 as blockdevice (Test ID 3) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
255703
0
0
0
0
0
0
0
0
0
0
0
233397
451831
0
0
0
0
0
0
0
0
0
0
270172
455023
863170
0
0
0
0
0
0
0
0
0
295196
488044
879560
1.12201e+06
0
0
0
0
0
0
0
0
245462
479616
835220
1.12782e+06 1.10454e+06
0
0
0
0
0
0
0
251120
505015
865667
1.13032e+06 1.1029e+06 1.11702e+06
0
0
0
0
0
0
242235
400038
741802
1.12749e+06 1.09983e+06 1.11414e+06 1.12496e+06
0
0
0
0
0
188667
343544
572799
932412
0
0
0
0
100228
198046
359091
635379
900850
0
0
0
50952
103343
192724
362017
582709
0
0
1.10897e+06 1.11956e+06 1.13713e+06 1.13794e+06
1.08353e+06 1.14066e+06 1.14089e+06 1.12325e+06 737718
1.09157e+06 1.13928e+06 1.12276e+06 1.13534e+06
25148
51913
99581
195257
350595
532933
671553
12469
25944
50325
100461
174410
322724
466417
1.08367e+06 1.12097e+06 1.12001e+06 1.97254e+06 679921
1
2
4
8
16
32
64
128
number of workers
1600000 1200000 800000 400000
0
1.06639e+06 1.12282e+06 1.14025e+06 1.76456e+06
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
12352
0
0
0
0
0
0
0
0
0
0
0
5088
5472
0
0
0
0
0
0
0
0
0
0
2224
2704
4256
0
0
0
0
0
0
0
0
0
1032
1320
2096
11968
0
0
0
0
0
0
0
0
788
740
1144
5344
25728
0
0
0
0
0
0
0
414
406
668
2384
12224
44288
0
0
0
0
0
0
247
334
1288
1064
3952
18048
41728
0
0
0
0
0
185
227
506
556
1160
4448
13632
42240
0
0
0
0
171
181
237
270
572
1176
2608
18304
47360
0
0
0
167
163
199
207
390
1032
1012
3632
20096
42240
0
0
171
157
181
185
233
612
1080
852
4192
20352
18816
0
191
153
191
171
245
314
1048
1336
868
3472
33536
28544
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.6
0
0
0
0
0
0
0
0
0
0
0
3.6
6.4
0
0
0
0
0
0
0
0
0
0
2.7
6.1
10.4
0
0
0
0
0
0
0
0
0
2.7
6.4
10.9
15.3
0
0
0
0
0
0
0
0
3.5
6.2
10.1
15.7
19.2
0
0
0
0
0
0
0
3.5
6.3
10.1
15.7
17.9
27
0
0
0
0
0
0
3.3
5.2
9
16.4
19.1
26.4
32.6
0
0
0
0
0
2.6
4.9
8.2
14.2
19.9
27.3
37.2
49.9
0
0
0
0
1.9
3.4
6
11.6
17.6
25.8
43.8
50.8
56.2
0
0
0
1.3
2.1
3.9
7.5
13.8
21.4
45.5
56
55.2
58
0
0
0.9
1.4
2.4
4.5
8.7
15.9
28.9
58.8
51.1
59.8
49
0
0.6
0.9
1.4
3
5.8
9.9
18.7
38.3
59
63.2
63.6
49.2
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
40000 32000 24000 16000 8000 0
100 80 60 40 20 0
Software RAID0 over 4 Intel P3700 as blockdevice (Test ID 3) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
222764
0
0
0
0
0
0
0
0
0
0
0
194451
391777
0
0
0
0
0
0
0
0
0
0
207538
369984
376140
0
0
0
0
0
0
0
0
0
231325
364635
376589
375098
0
0
0
0
0
0
0
0
240538
415081
373767
314438
369397
0
0
0
0
0
0
0
240930
347158
390976
305175
369224
369466
0
0
0
0
0
0
236142
324958
383162
326351
369914
367575
367015
0
0
0
0
0
224532
406871
428809
372863
359659
365408
369179
368027
0
0
0
0
215930
398180
452023
397514
299811
370189
367162
377874
368015
0
0
0
189449
306182
435130
451992
294289
372592
368843
368488
363246
368816
0
0
102520
175706
300354
417347
498791
427129
398460
366022
375885
368479
362311
0
57803
102344
168425
265233
360918
393209
395782
369427
361056
370164
364881
363941
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
400000 300000 200000 100000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
12096
0
0
0
0
0
0
0
0
0
0
0
7264
11584
0
0
0
0
0
0
0
0
0
0
3248
6048
36608
0
0
0
0
0
0
0
0
0
1448
2480
20096
43264
0
0
0
0
0
0
0
0
780
1800
12992
27264
82432
0
0
0
0
0
0
0
430
940
8640
16320
50944
121344
0
0
0
0
0
0
203
692
5344
10176
18560
72192
136192
0
0
0
0
0
126
748
2608
5408
9280
25216
77312
195584
0
0
0
0
69
245
1224
2864
6176
8896
25472
76288
171008
0
0
0
57
75
338
1288
3344
5344
9024
20096
62720
175104
0
0
51
61
84
282
1064
2800
5152
9024
18816
84480
132096
0
48
45
66
133
426
1288
2832
5344
9024
22912
90624
104960
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.6
0
0
0
0
0
0
0
0
0
0
0
3.5
5.4
0
0
0
0
0
0
0
0
0
0
3.1
5.1
6.1
0
0
0
0
0
0
0
0
0
2.9
4.8
6.2
7.3
0
0
0
0
0
0
0
0
3
5
5.8
6.2
8.6
0
0
0
0
0
0
0
2.7
4.6
6
6.2
8.6
10.1
0
0
0
0
0
0
2.7
4.5
6.3
6.8
8.6
9.9
11.9
0
0
0
0
0
2.7
4.8
6.9
7.3
8.7
10.4
12.5
14.7
0
0
0
0
3.1
4.8
7.3
8.2
7.7
11.3
13.4
15.9
16.4
0
0
0
2.6
4.9
7.5
9.2
7.6
12
15.2
17.2
17.6
18.9
0
0
2.1
3.6
6.3
9.7
13.2
14.1
17.3
18.3
19.5
19.2
18.3
0
1.4
2.5
4.7
7.3
12
15.8
19.5
19.2
19.5
20
20
15.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
160000 120000 80000 40000 0
100 80 60 40 20 0
Software RAID0 over 4 Intel P3700 as blockdevice (Test ID 3) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
60254
0
0
0
0
0
0
0
0
0
0
0
61341
117843
0
0
0
0
0
0
0
0
0
0
60688
100875
176293
0
0
0
0
0
0
0
0
0
65398
115006
190732
191916
0
0
0
0
0
0
0
0
78297
108497
184281
207698
200667
0
0
0
0
0
0
0
73093
106660
174158
209914
199810
200976
0
0
0
0
0
0
69450
108922
155717
208424
200237
202958
203602
0
0
0
0
0
45948
74356
117456
169728
197628
202724
203047
203766
0
0
0
0
26892
45501
75509
117793
166755
197752
204442
198519
201356
0
0
0
15140
26688
46842
74790
114751
162135
194987
200011
206423
201711
0
0
7764
14844
27208
45898
72615
111212
149577
192506
201494
202353
203333
0
3821
7724
14783
26246
44643
72902
105410
139582
189279
201250
202475
203894
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
200000 160000 120000 80000 40000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
12096
0
0
0
0
0
0
0
0
0
0
0
6048
6368
0
0
0
0
0
0
0
0
0
0
3504
3920
15680
0
0
0
0
0
0
0
0
0
1480
1880
9280
82432
0
0
0
0
0
0
0
0
548
1048
5664
13632
50432
0
0
0
0
0
0
0
247
454
3184
7456
29824
71168
0
0
0
0
0
0
103
165
1256
4384
9408
27520
88576
0
0
0
0
0
56
75
143
466
4576
8896
47872
121344
0
0
0
0
49
52
72
139
628
3504
9280
46848
74240
0
0
0
46
50
45
70
171
604
3248
8640
24192
53504
0
0
68
65
42
61
108
249
1128
2960
8256
24704
90624
0
114
76
68
59
69
99
402
1384
2960
9536
26496
66048
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.5
0
0
0
0
0
0
0
0
0
0
0
3.5
6.3
0
0
0
0
0
0
0
0
0
0
3.5
4.6
9.1
0
0
0
0
0
0
0
0
0
3.3
5.7
8.5
11.1
0
0
0
0
0
0
0
0
2.9
5.3
8.4
11.7
14.6
0
0
0
0
0
0
0
3
5
8.3
12.2
14.2
17.9
0
0
0
0
0
0
2.7
4.8
8.2
12.2
14.8
17
20.8
0
0
0
0
0
2.3
4.2
6.5
11.3
15.2
18.6
20.6
23.6
0
0
0
0
1.8
2.9
4.9
7.9
13.2
18.4
25
28
34.6
0
0
0
1.3
2.1
3.2
5.7
10
15.8
25.3
33.7
36.8
38.8
0
0
0.9
1.4
2.2
4
6.6
11.1
19.3
34.7
38.8
39.7
39.9
0
0.7
1
1.6
2.6
4.1
7.4
13.4
21.8
38.4
39.6
42.1
34.8
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
100000 75000 50000 25000 0
100 80 60 40 20 0
Software RAID0 over 4 Intel P3700 as blockdevice (Test ID 3) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
9.25164e+06
0
9.16333e+06 9.44484e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
9.32381e+06 9.48327e+06 9.47826e+06
8.82530e+06 9.52107e+06 9.34564e+06 9.58002e+06
9.58046e+06 9.46478e+06 9.39077e+06 9.4748e+06 9.31438e+06
9.13524e+06 9.41599e+06 9.45389e+06 9.49609e+06 9.12498e+06 8.08698e+06
8.87886e+06 9.3621e+06 9.40414e+06 9.47229e+06 9.15773e+06 6.89646e+06 6.48462e+06
6.35665e+06 8.5877e+06 9.22326e+06 9.45481e+06 9.2896e+06 7.74981e+06 7.47907e+06 5.95643e+06
3.98408e+06 5.82341e+06 8.10227e+06 9.26963e+06 9.09951e+06 5.5308e+06 4.97053e+06 5.76346e+06 5.57658e+06
3.52881e+06 3.13213e+06 4.95492e+06 7.52311e+06 8.97687e+06 6.56266e+06 5.3285e+06 5.40204e+06 5.35451e+06 5.73282e+06
1.90811e+06 1.44887e+06 2.55771e+06 4.2633e+06 6.29026e+06 6.21336e+06 5.50838e+06 5.38108e+06 6.10628e+06 5.36675e+06 5.85417e+06 1.14154e+06
624038
1
2
8
16
32
64
128
number of workers
6000000 4000000 2000000
0
1.22683e+06 2.33138e+06 3.78138e+06 5.34544e+06 5.06805e+06 5.41649e+06 5.36645e+06 5.29426e+06 6.11241e+06 6.01769e+06
4
8000000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
35584
0
0
0
0
0
0
0
0
0
0
0
20864
175104
0
0
0
0
0
0
0
0
0
0
18304
89600
146432
0
0
0
0
0
0
0
0
0
5664
39680
87552
211968
0
0
0
0
0
0
0
0
9792
23424
39680
95744
257024
0
0
0
0
0
0
0
2832
11712
23168
50944
179200
254976
0
0
0
0
0
0
1256
4320
11200
24192
70144
112128
423936
0
0
0
0
0
700
1624
4384
10176
19328
87552
354304
651264
0
0
0
0
532
916
2608
3408
9024
40704
117248
382976
432128
0
0
0
390
676
2512
1640
3280
14400
49920
189440
544768
733184
0
0
366
620
2192
1192
1784
6240
16768
36608
171008
387072
626688
0
346
668
1512
972
1320
1896
7840
19072
45312
113152
440320
448512
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.5
0
0
0
0
0
0
0
0
0
0
0
2.6
3.2
0
0
0
0
0
0
0
0
0
0
2.5
3.2
3.8
0
0
0
0
0
0
0
0
0
3.1
3.2
3.6
4
0
0
0
0
0
0
0
0
2.6
3.1
3.4
3.9
4.5
0
0
0
0
0
0
0
3.2
3.2
3.5
4.1
4.6
6
0
0
0
0
0
0
2.5
2.8
3
3.8
4
6.6
7.4
0
0
0
0
0
1.6
2.2
2.6
3.4
4
5.4
4.5
7.7
0
0
0
0
1.3
1.8
2.4
3.5
3.9
6.4
7.1
7.7
6.8
0
0
0
1.3
1.3
1.9
2.7
3.3
7.5
6.8
7.8
7.6
7.7
0
0
1
1
1.4
2.4
4.2
7
7.7
7.4
7.2
5.6
7.9
0
0.7
1.1
1.2
1.7
4.2
7.4
7.6
7.8
6
6.2
5.9
5.2
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000 450000 300000 150000 0
100 80 60 40 20 0
Software RAID0 over 4 Intel P3700 as blockdevice (Test ID 3) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6121e+06
0
2.82952e+06 5.80194e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3.49516e+06 6.31709e+06 7.61212e+06
3.54455e+06 6.99942e+06 7.6571e+06 7.68149e+06
6.2427e+06 6.95491e+06 7.48882e+06 7.58442e+06 7.43907e+06
6.23624e+06 6.79671e+06 7.55215e+06 7.66542e+06 7.13938e+06 7.09285e+06
6.35094e+06 7.08401e+06 6.5731e+06 7.51632e+06 5.37515e+06 6.11142e+06 7.41251e+06
6.45366e+06 7.2243e+06 6.29196e+06 5.13257e+06 4.89192e+06 6.56368e+06 7.13101e+06 6.84948e+06
6.15847e+06 7.51415e+06 6.39732e+06 5.52925e+06 4.89924e+06 5.94639e+06 7.25069e+06 7.01098e+06 6.95049e+06
5.12148e+06 6.84447e+06 6.29691e+06 5.7835e+06 4.92512e+06 5.16706e+06 7.01778e+06 6.97165e+06 6.82828e+06 6.5281e+06
2.97208e+06 6.19593e+06 4.904e+06 6.04066e+06 4.92684e+06 6.20141e+06 7.10667e+06 6.9923e+06 6.95686e+06 6.70746e+06 5.63442e+06
2
4
8
16
32
64
128
number of workers
6000000 4500000 3000000 1500000
0
1.51753e+06 3.10062e+06 4.58665e+06 6.41792e+06 5.43953e+06 6.35536e+06 6.40191e+06 7.09757e+06 7.14629e+06 6.60754e+06 5.14784e+06 5.43317e+06
1
7500000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
89600
0
0
0
0
0
0
0
0
0
0
0
72192
166912
0
0
0
0
0
0
0
0
0
0
24448
47360
209920
0
0
0
0
0
0
0
0
0
10816
55552
92672
209920
0
0
0
0
0
0
0
0
11328
44800
47360
134144
493568
0
0
0
0
0
0
0
6048
24448
30592
60672
254976
448512
0
0
0
0
0
0
2448
10432
29824
33536
138240
305152
358400
0
0
0
0
0
2448
5344
14144
33536
68096
138240
264192
460800
0
0
0
0
1176
2224
5984
18048
34560
56064
156672
569344
407552
0
0
0
151
1400
3056
9280
17024
39168
25984
232448
602112
667648
0
0
211
191
3280
5408
10816
12864
16768
42752
164864
440320
765952
0
173
137
1272
1256
5792
5920
13888
15168
30336
220160
485376
815104
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.7
0
0
0
0
0
0
0
0
0
0
0
2.7
4.7
0
0
0
0
0
0
0
0
0
0
2.6
4.7
5.4
0
0
0
0
0
0
0
0
0
2.8
4.2
4.9
6.6
0
0
0
0
0
0
0
0
2.8
3.7
6.2
5.9
6.4
0
0
0
0
0
0
0
2.5
3.3
4.4
5.5
5.3
7.7
0
0
0
0
0
0
2.6
3.2
3.5
4.5
3.8
4.8
8.3
0
0
0
0
0
2.4
2.8
2.7
2.6
3.1
4.2
7.1
7.4
0
0
0
0
2.3
2.9
2.7
2.7
3
4
7.6
7.7
8.9
0
0
0
2.2
2.6
2.7
2.9
3.1
3.3
7.8
7.8
9.4
7.9
0
0
1.7
2.5
2.3
3.1
2.6
5
6.8
7.2
8.2
7.3
5.7
0
1.1
1.6
2.5
3.2
2.7
3.7
7
8.3
8
8.5
5.8
6
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
750000 600000 450000 300000 150000 0
100 80 60 40 20 0
Software RAID0 over 6 Intel P3700 as blockdevice (Test ID 4) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
244875
0
0
0
0
0
0
0
0
0
0
0
253310
507624
0
0
0
0
0
0
0
0
0
0
257669
534348
965424
0
0
0
0
0
0
0
0
0
283588
533118
865595
1.65475e+06
0
0
0
0
0
0
0
0
260956
504643
955800
1.67887e+06 2.17856e+06
0
0
0
0
0
0
0
259913
508473
955820
1.66442e+06 2.13551e+06 2.48129e+06
0
0
0
0
0
0
265286
440649
776881
1.61081e+06 2.06432e+06 2.53684e+06 2.68178e+06
0
0
0
0
0
247655
437717
710501
1.29646e+06 2.15985e+06 2.50302e+06 2.70589e+06 2.74947e+06
0
0
0
0
141866
271363
512642
855979
0
0
0
73139
141569
278850
491611
815599
0
0
36570
71242
142750
272429
452143
586793
807820
17177
36116
70338
140670
254252
441659
477467
777242
1
2
4
8
16
32
64
128
1.39569e+06 2.06365e+06 2.56655e+06 2.47758e+06 2.38005e+06
1.07812e+06 1.63672e+06 2.70226e+06 2.71744e+06 2.70460e+06
1.48093e+06 2.67005e+06 2.65664e+06 2.72262e+06
number of workers
2500000 2000000 1500000 1000000 500000
0
1.70145e+06 2.62213e+06 2.72656e+06 2.72991e+06
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
11840
0
0
0
0
0
0
0
0
0
0
0
4896
4320
0
0
0
0
0
0
0
0
0
0
3088
2160
2384
0
0
0
0
0
0
0
0
0
1160
1112
5280
1576
0
0
0
0
0
0
0
0
764
684
2352
836
1480
0
0
0
0
0
0
0
406
378
402
474
788
1928
0
0
0
0
0
0
231
278
996
1064
516
1192
13248
0
0
0
0
0
161
189
524
692
398
684
3568
21120
0
0
0
0
143
159
179
386
326
636
844
2416
6688
0
0
0
135
145
157
217
306
980
796
796
3568
11712
0
0
135
137
143
159
270
1096
1752
772
852
4448
13760
0
167
135
147
161
213
262
1544
1400
716
900
4320
8768
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.7
0
0
0
0
0
0
0
0
0
0
0
3.7
6.4
0
0
0
0
0
0
0
0
0
0
3.7
6.6
11.5
0
0
0
0
0
0
0
0
0
3
6.6
10.7
19.9
0
0
0
0
0
0
0
0
3.7
6.8
11.5
20.4
36.9
0
0
0
0
0
0
0
3.7
6.8
11.2
20.2
36.4
63.6
0
0
0
0
0
0
3.7
5.1
9.8
19.5
36.2
63.6
86.4
0
0
0
0
0
3.1
5.4
9.2
16.7
33.9
57.8
86.6
89.4
0
0
0
0
2.4
4.5
8.2
13.4
22.8
42.1
81.3
92.9
67.2
0
0
0
1.6
3.1
5.6
10
17.3
30
59.6
87.9
87.3
87.7
0
0
1
1.9
3.4
6.6
12.2
18.8
33.4
62.4
87.2
86.4
87.2
0
0.8
1.1
2.1
4.1
7.5
13.9
20.7
41.5
72.2
85.7
56.9
88.6
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
20000 16000 12000 8000 4000 0
100 80 60 40 20 0
Software RAID0 over 6 Intel P3700 as blockdevice (Test ID 4) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
237049
0
0
0
0
0
0
0
0
0
0
0
232817
448400
0
0
0
0
0
0
0
0
0
0
253005
470405
716788
0
0
0
0
0
0
0
0
0
230961
400864
793985
1.03488e+06
0
0
0
0
0
0
0
0
220821
428102
665793
1.04794e+06 1.0442e+06
0
0
0
0
0
0
0
235966
433792
701851
1.10255e+06 1.04101e+06 1.02298e+06
0
0
0
0
0
0
240917
456098
678042
1.28902e+06 1.08617e+06 1.0397e+06 1.02264e+06
0
0
0
0
0
244270
389852
701429
1.14856e+06 1.19516e+06 1.04157e+06 1.04121e+06 1.02777e+06
0
0
0
0
250759
431398
720519
1.15641e+06 1.27468e+06 1.23759e+06 1.02669e+06 1.02714e+06 1.01999e+06
0
0
0
221674
401085
598356
847479
0
0
110875
204363
362699
468589
763595
749846
827772
57819
110441
192970
333765
472206
520681
463086
793526
1
2
4
8
16
32
64
128
1.15842e+06 1.27971e+06 1.40875e+06 1.03005e+06 1.02397e+06 1.02486e+06
1.37107e+06 1.0285e+06 1.0145e+06 1.01811e+06
number of workers
0
1.03418e+06 1.01613e+06 1.0206e+06 1.07682e+06
256
512
1024
2048
1250000 1000000 750000 500000 250000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
13248
0
0
0
0
0
0
0
0
0
0
0
5536
6368
0
0
0
0
0
0
0
0
0
0
2256
2960
11456
0
0
0
0
0
0
0
0
0
1496
1816
6496
20864
0
0
0
0
0
0
0
0
836
892
3024
12608
43264
0
0
0
0
0
0
0
382
422
2040
7200
22144
50944
0
0
0
0
0
0
191
203
876
2832
7776
23936
88576
0
0
0
0
0
107
139
394
1464
3856
10304
26496
63744
0
0
0
0
51
64
245
486
2064
3920
8512
20864
54016
0
0
0
41
50
149
217
470
1480
2864
8512
21632
58624
0
0
42
56
69
362
237
1432
1896
2608
9280
20608
48384
0
39
47
55
87
129
772
2448
1624
3792
9280
19328
47872
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.9
0
0
0
0
0
0
0
0
0
0
0
3.6
6.1
0
0
0
0
0
0
0
0
0
0
3
5.7
9.4
0
0
0
0
0
0
0
0
0
3.7
5.3
9.9
13.7
0
0
0
0
0
0
0
0
3.7
5.2
9
13.8
17
0
0
0
0
0
0
0
3.4
5.5
9.9
14.1
16.8
21.9
0
0
0
0
0
0
3.1
5.8
9.2
15.8
18.8
22.6
27
0
0
0
0
0
3.4
5.3
9
16.6
19.9
24.2
30.9
37.7
0
0
0
0
3
5.3
9.9
17
19.7
28
35.8
42
44.1
0
0
0
3.2
5.7
9.3
13.7
20.2
28.6
47.5
44.7
45.3
46.9
0
0
2.4
4.4
7.6
10.3
18.9
22.7
37.8
56.3
47.1
47.2
48.1
0
1.7
3
5.3
9.6
16.3
22.1
22.5
46.8
50.8
48
48.6
50
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
75000 60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID0 over 6 Intel P3700 as blockdevice (Test ID 4) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
246176
0
0
0
0
0
0
0
0
0
0
0
249090
420125
0
0
0
0
0
0
0
0
0
0
233567
462543
864087
0
0
0
0
0
0
0
0
0
243350
456710
963360
1.32992e+06
0
0
0
0
0
0
0
0
268081
493088
933942
1.46406e+06 1.65458e+06
0
0
0
0
0
0
0
268836
483822
930344
1.34249e+06 1.64137e+06 1.64535e+06
0
0
0
0
0
0
266352
443186
747767
1.25785e+06 1.63838e+06 1.63676e+06 1.6368e+06
0
0
0
0
0
189150
336936
560280
979479
0
0
0
0
97740
189617
354396
578330
981473
0
0
0
50941
97228
187679
334273
557202
718126
0
0
25152
49196
96597
184220
341854
528566
669208
11135
24644
49179
92872
176589
327653
472387
614997
1
2
4
8
16
32
64
128
1.37889e+06 1.64199e+06 1.60322e+06 1.66987e+06
1.27727e+06 1.64513e+06 1.67112e+06 1.66993e+06
1.15516e+06 1.66538e+06 1.6736e+06 1.66806e+06
1.1051e+06 1.65699e+06 1.66779e+06 3.26662e+06
number of workers
3000000 2400000 1800000 1200000 600000
0
1.11108e+06 1.59547e+06 1.67277e+06 3.35746e+06
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
9280
0
0
0
0
0
0
0
0
0
0
0
4384
7072
0
0
0
0
0
0
0
0
0
0
2576
2448
2736
0
0
0
0
0
0
0
0
0
1464
1608
1400
3440
0
0
0
0
0
0
0
0
612
660
780
3088
23936
0
0
0
0
0
0
0
342
382
780
3408
6048
37120
0
0
0
0
0
0
217
278
788
3408
2128
9024
39680
0
0
0
0
0
175
217
366
596
644
1640
10560
47872
0
0
0
0
163
179
217
382
450
1064
1544
13760
33536
0
0
0
151
165
187
266
532
1192
1144
1400
19072
39680
0
0
153
161
167
197
231
628
1624
1012
1464
9664
6368
0
199
161
161
187
231
266
788
1784
1012
1400
11072
18560
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3
0
0
0
0
0
0
0
0
0
0
0
3
6.3
0
0
0
0
0
0
0
0
0
0
3.7
6.6
10.1
0
0
0
0
0
0
0
0
0
3.7
5.9
11.2
17.4
0
0
0
0
0
0
0
0
3.2
6.4
11.3
18.8
24.5
0
0
0
0
0
0
0
3.2
6.3
11.1
17.4
23.7
33.9
0
0
0
0
0
0
3.1
5.2
9
16.8
27
35.7
42.8
0
0
0
0
0
2.6
5.2
8.6
14.4
21.8
33.9
49
62.6
0
0
0
0
2
3.7
6.5
10.7
19.3
29.6
59.7
62.8
67.1
0
0
0
1.2
2.4
4.5
8.1
13.2
20.5
49.8
69.1
65.6
70.3
0
0
0.9
1.5
2.6
4.6
9.2
16.1
29.2
57.9
70.1
72.3
72.8
0
0.7
0.9
1.6
3.1
5.9
10.5
19.1
34.9
64.1
66.7
37.1
49.4
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
40000 32000 24000 16000 8000 0
100 80 60 40 20 0
Software RAID0 over 6 Intel P3700 as blockdevice (Test ID 4) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
206923
0
0
0
0
0
0
0
0
0
0
0
214019
392579
0
0
0
0
0
0
0
0
0
0
208731
382884
624068
0
0
0
0
0
0
0
0
0
223526
381946
588274
519864
0
0
0
0
0
0
0
0
214620
360872
483662
524080
521244
0
0
0
0
0
0
0
220330
380314
586589
520981
515110
527686
0
0
0
0
0
0
221317
348773
596751
543713
514102
502928
513881
0
0
0
0
0
231618
387984
603250
552841
515966
539317
515994
524038
0
0
0
0
231512
401902
560821
659231
537669
517787
514155
578444
563847
0
0
0
184106
305156
496392
625451
693720
661053
582631
518455
540705
518158
0
0
91116
173669
316938
410778
516307
564033
650736
538140
517732
518389
513947
0
47836
93004
166866
270199
373458
343355
422672
584455
516227
524466
518548
524095
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000 450000 300000 150000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
12480
0
0
0
0
0
0
0
0
0
0
0
6624
6496
0
0
0
0
0
0
0
0
0
0
3152
4512
14400
0
0
0
0
0
0
0
0
0
1624
1912
14400
60160
0
0
0
0
0
0
0
0
868
1192
13632
23424
88576
0
0
0
0
0
0
0
458
498
7776
13248
56576
97792
0
0
0
0
0
0
219
290
2672
8512
19072
64256
123392
0
0
0
0
0
104
173
1560
4768
8768
21888
63744
171008
0
0
0
0
65
99
548
2128
4896
8640
21888
61696
142336
0
0
0
53
72
145
684
1944
4080
8096
18048
41216
107008
0
0
55
60
68
342
772
1736
2992
8032
16320
47360
126464
0
49
54
60
110
278
1864
2512
3056
8384
13504
57088
103936
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6
0
0
0
0
0
0
0
0
0
0
0
2.9
6.5
0
0
0
0
0
0
0
0
0
0
3.6
6.2
9.2
0
0
0
0
0
0
0
0
0
3
5.1
8.5
9.2
0
0
0
0
0
0
0
0
3.1
5
7.8
9.2
11.3
0
0
0
0
0
0
0
3.1
5.3
8.4
9.3
10.8
14.2
0
0
0
0
0
0
3.1
4.9
9.2
9.4
11.4
13.7
15.9
0
0
0
0
0
3
5.7
9.1
10
12
15.5
16.2
19.4
0
0
0
0
3
6
9.1
12.1
12.4
15.5
19.5
25.5
27.1
0
0
0
2.9
5
8.5
12.6
16.2
20.1
24.8
25
29.3
27.9
0
0
2.2
4.1
6.5
9.5
14.2
18.3
31
28
28
28.5
26.6
0
1.7
2.7
4.9
8.6
13.1
13.4
21
36.8
29.2
29.9
28.3
28.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
150000 120000 90000 60000 30000 0
100 80 60 40 20 0
Software RAID0 over 6 Intel P3700 as blockdevice (Test ID 4) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
62644
0
0
0
0
0
0
0
0
0
0
0
66156
116682
0
0
0
0
0
0
0
0
0
0
67399
125294
224355
0
0
0
0
0
0
0
0
0
73479
112667
195619
236183
0
0
0
0
0
0
0
0
68479
114375
203170
281882
295519
0
0
0
0
0
0
0
72792
116676
233910
308146
300141
295186
0
0
0
0
0
0
67376
120951
184836
266160
294057
297760
299085
0
0
0
0
0
48626
81808
132830
195373
260435
294353
294602
302675
0
0
0
0
28313
49092
83367
130018
187190
258845
293194
300918
302712
0
0
0
15234
27930
49588
82081
125707
185060
231477
290991
296206
295207
0
0
7891
15200
27681
48888
79187
121177
154929
218191
290400
293932
310693
0
3623
7885
14637
27093
45670
77238
116297
146365
213901
290110
361426
253913
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
320000 240000 160000 80000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
12352
0
0
0
0
0
0
0
0
0
0
0
6112
5920
0
0
0
0
0
0
0
0
0
0
3088
2992
9920
0
0
0
0
0
0
0
0
0
1096
1704
4576
28032
0
0
0
0
0
0
0
0
652
788
3472
15936
38656
0
0
0
0
0
0
0
243
350
1096
6496
14400
58112
0
0
0
0
0
0
143
139
474
2448
6368
17024
58112
0
0
0
0
0
60
82
89
516
1608
6560
21376
72192
0
0
0
0
48
67
67
139
474
1784
5536
17024
63744
0
0
0
49
54
60
89
237
580
1688
5280
15040
55552
0
0
65
49
61
75
98
390
1816
1544
5152
15040
44288
0
108
66
64
67
110
123
494
1832
1432
5088
11968
90624
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6
0
0
0
0
0
0
0
0
0
0
0
2.9
6.5
0
0
0
0
0
0
0
0
0
0
2.9
6.1
11
0
0
0
0
0
0
0
0
0
2.9
5.2
8.8
13.4
0
0
0
0
0
0
0
0
3.2
5
8.9
14.7
19.1
0
0
0
0
0
0
0
3.1
5
9.7
15.6
19.5
25.2
0
0
0
0
0
0
3
5.4
9.1
14.6
19.4
26.1
29.2
0
0
0
0
0
2.7
4.5
7.8
12
18.2
27
34.4
39.7
0
0
0
0
1.9
3.6
5.7
9.7
15.6
25
41
47.3
55
0
0
0
1.4
2.4
4
6.7
11.2
18.9
33.3
52.8
56.8
57.7
0
0
1
1.5
2.7
4.6
7.3
12.4
21.1
42.5
57.7
58.2
61.4
0
0.8
1
1.8
3.2
5.1
8.5
14.7
24.2
47.6
59.2
65.9
51.6
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
80000 60000 40000 20000 0
100 80 60 40 20 0
Software RAID0 over 6 Intel P3700 as blockdevice (Test ID 4) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
9.07848e+06
0
9.34122e+06 1.43006e+07
1e7
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
9.03958e+06 1.42129e+07 1.40092e+07
9.18488e+06 1.43054e+07 1.40216e+07 1.3857e+07
9.18313e+06 1.42627e+07 1.4161e+07 1.3989e+07 1.32198e+07
9.68938e+06 1.39565e+07 1.42731e+07 1.36755e+07 1.3693e+07 1.3295e+07
9.88114e+06 1.32079e+07 1.39853e+07 1.39088e+07 1.40605e+07 1.2755e+07 9.63368e+06
6.60206e+06 1.02212e+07 1.31428e+07 1.35285e+07 1.20695e+07 1.14416e+07 1.1581e+07 7.31923e+06
5.72397e+06 5.96721e+06 9.2858e+06 1.21035e+07 1.31762e+07 1.03859e+07 7.86234e+06 7.39082e+06 7.20214e+06
4.14751e+06 2.90906e+06 5.22003e+06 8.08593e+06 1.10645e+07 1.12653e+07 7.82411e+06 7.76633e+06 8.1927e+06 7.50469e+06
2.32763e+06 1.31068e+06 2.59594e+06 4.62249e+06 7.32577e+06 9.72879e+06 7.43458e+06 6.94268e+06 6.66785e+06 9.2858e+06 1.08953e+07 1.38161e+06
596544
1
2
0
1.29484e+06 2.38194e+06 4.11282e+06 6.21471e+06 8.09205e+06 7.24694e+06 7.98379e+06 1.02384e+07 1.08633e+07 9.43498e+06
4
8
16
32
64
128
number of workers
256
512
1024
2048
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
37632
0
0
0
0
0
0
0
0
0
0
0
15680
121344
0
0
0
0
0
0
0
0
0
0
11072
59136
140288
0
0
0
0
0
0
0
0
0
6048
33536
67072
317440
0
0
0
0
0
0
0
0
2288
18816
37120
189440
148480
0
0
0
0
0
0
0
1336
6944
20608
33024
82432
329728
0
0
0
0
0
0
804
1880
8096
20864
40704
136192
325632
0
0
0
0
0
588
964
2608
7328
22656
32640
280576
477184
0
0
0
0
564
780
1304
2704
6624
31360
53504
268288
610304
0
0
0
382
844
972
1864
2384
13504
24192
57600
107008
456704
0
0
310
964
996
2008
1720
3152
14272
41728
75264
268288
432128
0
326
996
972
1800
2128
1704
2960
17536
34560
66048
211968
403456
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.1
0
0
0
0
0
0
0
0
0
0
0
2.7
4.5
0
0
0
0
0
0
0
0
0
0
3.1
4.6
6.1
0
0
0
0
0
0
0
0
0
3.1
4.5
5.9
6.4
0
0
0
0
0
0
0
0
3.2
4.5
5.6
6.4
8.6
0
0
0
0
0
0
0
2.9
5.3
5.9
7.6
8
7.9
0
0
0
0
0
0
3
4.4
5.5
6.1
7.1
8.4
9.3
0
0
0
0
0
1.7
3
4.1
5.5
7.2
9
9.7
11.7
0
0
0
0
1.7
1.9
3.2
5.3
6.7
8.4
10.4
11.9
12.4
0
0
0
1.3
1.2
2
3.6
6.2
9.7
11.6
11.7
11.1
11.3
0
0
1
1.1
1.4
2.3
5
9.2
12.2
11.3
11.9
9
7.7
0
0.9
1
1.2
2
4.3
10.2
14.7
11.7
11.6
8.5
7.7
10.5
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1.25 1.00 0.75 0.50 0.25 0.00
600000
450000
300000
150000
0
100 80 60 40 20 0
Software RAID0 over 6 Intel P3700 as blockdevice (Test ID 4) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6602e+06
0
3.64582e+06 6.52348e+06
1e7
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3.65728e+06 6.4428e+06 1.10709e+07
3.68732e+06 7.53789e+06 9.85438e+06 9.84458e+06
6.311e+06 7.84955e+06 1.03398e+07 9.82013e+06 1.04653e+07
6.40453e+06 1.06134e+07 1.001e+07 9.92013e+06 9.34406e+06 9.18311e+06
6.89916e+06 1.01658e+07 1.08605e+07 9.36274e+06 1.00399e+07 1.02256e+07 1.06793e+07
8.26231e+06 9.67635e+06 1.09594e+07 9.84597e+06 8.894e+06 9.80637e+06 1.09276e+07 9.27174e+06
7.76604e+06 9.14794e+06 1.05032e+07 9.90164e+06 9.21684e+06 9.73511e+06 1.08749e+07 9.70847e+06 1.00416e+07
4.94848e+06 8.99683e+06 9.36777e+06 9.34818e+06 9.63847e+06 9.14348e+06 1.07847e+07 8.96661e+06 9.82312e+06 9.31257e+06
3.10388e+06 5.66015e+06 8.38917e+06 8.73797e+06 7.71602e+06 9.5918e+06 1.03782e+07 7.00343e+06 8.79484e+06 9.07611e+06 1.09542e+07
2
4
8
16
32
64
128
number of workers
0.8 0.6 0.4 0.2
0
1.67999e+06 2.97864e+06 5.75044e+06 8.27593e+06 6.8324e+06 9.03473e+06 6.72844e+06 8.17106e+06 7.83314e+06 1.024e+07 9.42364e+06 1.01893e+07
1
1.0
256
512
1024
2048
0.0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
75264
0
0
0
0
0
0
0
0
0
0
0
43776
51456
0
0
0
0
0
0
0
0
0
0
24960
26752
86528
0
0
0
0
0
0
0
0
0
10048
16064
103936
329728
0
0
0
0
0
0
0
0
7264
22400
51968
358400
342016
0
0
0
0
0
0
0
6432
17280
34048
140288
226304
489472
0
0
0
0
0
0
1528
10432
16320
47360
94720
301056
585728
0
0
0
0
0
1864
5344
8896
20352
54528
108032
354304
651264
0
0
0
0
209
2352
3536
10432
30336
42752
91648
301056
395264
0
0
0
185
548
2608
5280
13376
28032
27008
160768
346112
493568
0
0
139
151
708
2608
9920
13888
15808
64256
140288
329728
399360
0
117
153
139
900
5344
7904
19584
33536
59648
93696
325632
428032
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.6
0
0
0
0
0
0
0
0
0
0
0
2.6
5
0
0
0
0
0
0
0
0
0
0
2.6
4.9
9.3
0
0
0
0
0
0
0
0
0
2.7
4.9
8
8.2
0
0
0
0
0
0
0
0
2.9
4.9
6.1
7.4
8.4
0
0
0
0
0
0
0
2.7
4.7
6
6.9
7.1
6.9
0
0
0
0
0
0
2.6
4.2
5.5
5.4
6.5
7
10.1
0
0
0
0
0
2.5
3.7
4.9
4.9
5.1
6.3
9.1
12.3
0
0
0
0
2.5
3.8
4.4
4.7
5
6.3
11.2
9.1
15.8
0
0
0
2.3
3.6
4.2
4.7
6
7
11.7
9.2
10.8
10.6
0
0
1.5
2.6
4
4.5
4.1
7.1
10.9
5.9
9
9.4
12.1
0
1.1
1.5
2.8
4.3
4
6.7
5.2
8
7.5
13.2
9.6
11.8
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000
450000
300000
150000
0
100 80 60 40 20 0
Software RAID0 over 8 Intel P3700 as blockdevice (Test ID 5) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
251679
0
0
0
0
0
0
0
0
0
0
0
252832
497987
0
0
0
0
0
0
0
0
0
0
264500
467950
971782
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
264396
529049
758272
1.51576e+06
272642
474864
792606
1.55881e+06 2.14424e+06
254956
506299
870055
1.42338e+06 2.2754e+06 2.51348e+06
258174
459186
812393
1.24181e+06 2.13682e+06 2.50048e+06 2.46401e+06
225925
417168
686227
1.25092e+06 2.11509e+06 2.45003e+06 2.72287e+06 2.72562e+06
124857
247212
461848
816453
64179
128362
246547
455520
676832
967526
30381
64334
126318
240752
419714
510385
794743
12866
31637
64664
123960
225183
383722
508966
728593
1
2
4
8
16
32
64
128
1.36239e+06 1.79522e+06 2.68908e+06 2.76633e+06 2.52312e+06
1.44835e+06 2.7745e+06 2.46885e+06 2.606e+06
1.40726e+06 2.66736e+06 2.54009e+06 2.76659e+06
number of workers
3200000 2400000 1600000 800000
0
1.38179e+06 2.25326e+06 2.74678e+06 3.92723e+06
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
8640
0
0
0
0
0
0
0
0
0
0
0
4384
5344
0
0
0
0
0
0
0
0
0
0
2992
2640
2384
0
0
0
0
0
0
0
0
0
1192
1112
2384
13120
0
0
0
0
0
0
0
0
620
716
1384
4016
1352
0
0
0
0
0
0
0
382
358
660
2352
764
1160
0
0
0
0
0
0
239
253
462
1432
462
764
24448
0
0
0
0
0
169
195
274
596
540
470
852
24448
0
0
0
0
151
173
201
278
486
772
980
2384
772
0
0
0
141
149
169
195
580
1080
1064
644
684
652
0
0
157
147
151
193
241
1352
1656
980
668
644
1368
0
211
159
139
173
211
318
892
1704
1032
804
748
852
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.8
0
0
0
0
0
0
0
0
0
0
0
3.8
6.8
0
0
0
0
0
0
0
0
0
0
3.8
5.8
11.8
0
0
0
0
0
0
0
0
0
3.8
6.7
9.8
19.9
0
0
0
0
0
0
0
0
3.2
6
10
20.7
37.8
0
0
0
0
0
0
0
3.8
6.7
11.1
18.6
36
69.8
0
0
0
0
0
0
3.7
6
10.2
16.6
37.5
69.8
77
0
0
0
0
0
3.3
5.7
9.2
16.9
31.9
56
95.4
95.3
0
0
0
0
2.4
4.6
8.6
15.3
23.5
40
86.5
95.1
74.9
0
0
0
1.6
2.8
5.9
10.9
18.1
28.5
57.9
90.9
79.7
93.2
0
0
1.2
1.8
3.5
6.6
12.9
19.1
38
66.4
90.3
93.2
65
0
0.9
1.2
2
3.7
7.7
14.6
26.7
44.3
69.8
80.1
91.5
69.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
20000 15000 10000 5000 0
100 80 60 40 20 0
Software RAID0 over 8 Intel P3700 as blockdevice (Test ID 5) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
233394
0
0
0
0
0
0
0
0
0
0
0
234624
421245
0
0
0
0
0
0
0
0
0
0
247183
413003
848603
0
0
0
0
0
0
0
0
0
238443
445858
657935
1.21999e+06
0
0
0
0
0
0
0
0
235841
377008
699159
1.23499e+06 1.44861e+06
0
0
0
0
0
0
0
241126
409503
720497
1.20135e+06 1.45092e+06 1.47575e+06
0
0
0
0
0
0
247064
427996
630861
1.1777e+06 1.44732e+06 1.45221e+06 1.40043e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
246062
419119
611546
1.17533e+06 1.57242e+06 1.46747e+06 1.39331e+06 1.47502e+06
235359
417370
640892
1.19074e+06 1.69016e+06 1.72561e+06 1.43944e+06 1.55406e+06 1.44214e+06
223405
385460
639351
896890
97235
192169
356761
510513
722725
799420
897480
48678
108014
181782
313199
487159
511688
559399
785384
1
2
4
8
16
32
64
128
1.31662e+06 1.43888e+06 1.75916e+06 1.45101e+06 1.41951e+06 1.64842e+06
1.70697e+06 1.44153e+06 1.43875e+06 1.52342e+06
number of workers
0
1.4795e+06 1.42766e+06 1.45716e+06 1.47233e+06
256
512
1024
2048
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
9792
0
0
0
0
0
0
0
0
0
0
0
4896
5856
0
0
0
0
0
0
0
0
0
0
2672
3024
5472
0
0
0
0
0
0
0
0
0
1432
1480
3280
10432
0
0
0
0
0
0
0
0
756
948
1496
9280
21888
0
0
0
0
0
0
0
342
410
772
3248
13120
33536
0
0
0
0
0
0
193
207
470
1656
7328
15424
51456
0
0
0
0
0
90
114
262
900
3536
7328
17792
38144
0
0
0
0
60
70
123
402
1288
2928
6944
15552
39680
0
0
0
39
54
80
314
362
1064
1832
6432
17792
29312
0
0
52
57
68
117
330
1176
1800
1480
6432
15040
24192
0
53
40
59
90
125
532
1528
1832
2608
6368
14528
30080
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.5
0
0
0
0
0
0
0
0
0
0
0
3.1
6.8
0
0
0
0
0
0
0
0
0
0
3.2
6.1
11.5
0
0
0
0
0
0
0
0
0
3.5
6.2
9
16.9
0
0
0
0
0
0
0
0
3.5
5.8
9.6
17.2
23
0
0
0
0
0
0
0
3.4
5.5
9.7
17
22.9
32.9
0
0
0
0
0
0
3.2
5.8
9.5
16.6
23.3
33.4
41.4
0
0
0
0
0
3.2
5.7
9.1
16.6
26
37
45.2
52.8
0
0
0
0
3.4
5.7
8.9
18.1
25.7
43.1
49.4
59.2
56.2
0
0
0
3.3
5.7
10
15.7
24.5
33.9
60.7
56.5
59.6
66.3
0
0
2.4
4.6
7.8
13
20.9
28.4
43.9
69.1
62.9
63.7
66.3
0
1.8
2.9
5.4
9.8
17.4
25.3
34.1
50.2
65.7
63.5
64.5
64.8
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1500000 1200000 900000 600000 300000 0
50000 40000 30000 20000 10000 0
100 80 60 40 20 0
Software RAID0 over 8 Intel P3700 as blockdevice (Test ID 5) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
242385
0
0
0
0
0
0
0
0
0
0
0
239878
457306
0
0
0
0
0
0
0
0
0
0
244988
437849
804515
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
259682
466680
845870
1.46472e+06
255186
483897
794951
1.42922e+06 1.98924e+06
259576
498403
855671
1.37384e+06 1.97918e+06 2.01836e+06
254167
447558
709069
1.1806e+06 1.95469e+06 2.02083e+06 2.19289e+06
179768
345989
608471
964234
93897
182574
338922
607549
980447
46634
94492
176132
335249
563405
908922
19571
46254
90975
177296
330523
488575
616671
6716
21736
44442
89860
168390
303694
470964
568759
1
2
4
8
16
32
64
128
1.56405e+06 2.03776e+06 2.20727e+06 2.20558e+06
1.33026e+06 2.05698e+06 2.20228e+06 2.20435e+06
1.10617e+06 2.04704e+06 2.12802e+06 2.20254e+06
1.05314e+06 2.09337e+06 2.01461e+06 3.56329e+06
number of workers
3200000 2400000 1600000 800000
0
1.07148e+06 1.79868e+06 1.88257e+06 3.98804e+06
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
9536
0
0
0
0
0
0
0
0
0
0
0
6496
5536
0
0
0
0
0
0
0
0
0
0
3184
2768
6560
0
0
0
0
0
0
0
0
0
1160
1336
2192
23168
0
0
0
0
0
0
0
0
652
676
1176
6944
3280
0
0
0
0
0
0
0
354
370
820
3760
1976
5984
0
0
0
0
0
0
229
258
406
1272
1720
3280
44288
0
0
0
0
0
175
199
266
852
604
1112
4576
27008
0
0
0
0
163
175
219
282
740
1144
980
5344
31360
0
0
0
163
161
211
219
350
828
1496
932
4384
11712
0
0
213
163
181
201
229
772
2024
1384
892
4704
4320
0
262
195
189
189
233
302
580
2224
1336
1064
2416
3120
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.2
0
0
0
0
0
0
0
0
0
0
0
3.2
6.5
0
0
0
0
0
0
0
0
0
0
3.2
6
11.3
0
0
0
0
0
0
0
0
0
3.5
6.1
10.9
20
0
0
0
0
0
0
0
0
3.5
6.7
10.4
19.6
36.4
0
0
0
0
0
0
0
3.5
6.8
11.4
19.4
35.9
65
0
0
0
0
0
0
3.3
6
9.5
16.7
33.1
66.9
68.2
0
0
0
0
0
2.8
5.3
9.4
15.2
25.6
57.7
74.6
78.2
0
0
0
0
1.9
3.8
6.8
12.3
21.1
34.6
69.8
80.6
79
0
0
0
1.4
2.2
4.3
8
16.1
28.9
48.6
76.6
79.8
80.1
0
0
1.3
1.6
2.5
4.6
9.8
18.3
29.6
58.1
81.3
81.8
75.4
0
1.1
1
1.7
2.8
6.5
11.4
22.3
35.4
64.7
76.4
80.6
73.9
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
40000 32000 24000 16000 8000 0
100 80 60 40 20 0
Software RAID0 over 8 Intel P3700 as blockdevice (Test ID 5) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
208244
0
0
0
0
0
0
0
0
0
0
0
205894
358995
0
0
0
0
0
0
0
0
0
0
213649
366002
629461
0
0
0
0
0
0
0
0
0
225242
376327
619952
759079
0
0
0
0
0
0
0
0
222709
376778
630657
755303
737263
0
0
0
0
0
0
0
216474
380851
580176
754600
741498
721551
0
0
0
0
0
0
220630
371110
732469
755666
756363
729517
725412
0
0
0
0
0
216396
338198
763778
817894
760419
735907
736974
812977
0
0
0
0
221842
396194
655067
827156
816732
773240
735770
728310
720858
0
0
0
183966
312770
508670
739810
858804
883408
828085
734205
725112
748540
0
0
86434
170566
297079
457978
567591
711808
781120
834359
614997
727029
728779
0
44641
95478
156892
275145
387827
386359
478811
676968
700491
735570
724545
732132
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
750000 600000 450000 300000 150000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
12224
0
0
0
0
0
0
0
0
0
0
0
7008
6944
0
0
0
0
0
0
0
0
0
0
3408
3728
7264
0
0
0
0
0
0
0
0
0
1304
1784
4048
40192
0
0
0
0
0
0
0
0
780
964
1720
19584
75264
0
0
0
0
0
0
0
398
458
1112
12480
37632
113152
0
0
0
0
0
0
191
233
1720
8032
12224
48384
140288
0
0
0
0
0
110
167
708
4016
7904
16192
40704
87552
0
0
0
0
67
77
189
1624
4048
7904
15296
39680
127488
0
0
0
62
73
105
318
1368
3184
7392
14400
36096
111104
0
0
58
66
85
126
486
1208
2416
6944
17792
32640
95744
0
62
44
65
94
155
1208
1832
2448
7264
13248
34048
90624
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.4
0
0
0
0
0
0
0
0
0
0
0
3.4
5.9
0
0
0
0
0
0
0
0
0
0
3.1
6.3
9.7
0
0
0
0
0
0
0
0
0
3.4
6
9
13.2
0
0
0
0
0
0
0
0
3.2
5.6
9.2
13.2
16.4
0
0
0
0
0
0
0
3.2
5.5
9.3
13
16.6
20.7
0
0
0
0
0
0
3.1
5.4
10.5
13.6
16.3
20.9
25.2
0
0
0
0
0
3.2
5.3
10.7
14.5
17.4
22.4
27.6
34.1
0
0
0
0
3.2
5.7
10.2
15.3
19
24.1
30.9
35.2
37.7
0
0
0
3.1
5.4
9.8
14.8
20.4
27.9
35.3
37.6
40.7
43
0
0
2.3
4.4
7.5
12.3
18.8
27.5
39.4
41.2
34.6
43
42.5
0
1.6
2.7
5.1
9.2
15.5
19.1
29.8
45.3
43.8
44.4
43.7
40.6
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
125000 100000 75000 50000 25000 0
100 80 60 40 20 0
Software RAID0 over 8 Intel P3700 as blockdevice (Test ID 5) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
65963
0
0
0
0
0
0
0
0
0
0
0
61702
120077
0
0
0
0
0
0
0
0
0
0
67586
112956
218862
0
0
0
0
0
0
0
0
0
69801
130019
198035
351330
0
0
0
0
0
0
0
0
66898
112494
188597
341109
389926
0
0
0
0
0
0
0
64418
111217
192766
326011
390090
386102
0
0
0
0
0
0
67690
122195
188676
276408
372108
393840
386192
0
0
0
0
0
50184
82209
140497
210611
294026
376283
392766
391037
0
0
0
0
28391
50051
85288
134049
205657
293015
362559
395250
389097
0
0
0
14688
28325
50311
84171
132543
196984
259340
359919
391978
390562
0
0
6372
14800
28043
49047
83568
124091
168891
251835
360587
385506
384524
0
2344
7394
14739
26998
48042
79738
119574
163305
222707
351578
392171
383358
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
320000 240000 160000 80000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
9792
0
0
0
0
0
0
0
0
0
0
0
6752
6880
0
0
0
0
0
0
0
0
0
0
2448
3376
5664
0
0
0
0
0
0
0
0
0
1240
1496
2608
23936
0
0
0
0
0
0
0
0
732
844
1336
11200
31616
0
0
0
0
0
0
0
370
390
708
4384
10048
45312
0
0
0
0
0
0
147
143
278
1912
5024
11200
42752
0
0
0
0
0
63
113
98
227
1528
4384
10304
39680
0
0
0
0
55
74
99
165
338
1064
3024
10560
40192
0
0
0
71
52
72
105
195
438
1528
2992
10048
34048
0
0
133
57
53
89
109
233
1400
1624
3344
11968
31872
0
159
96
53
68
99
125
454
1256
1832
3088
11328
43264
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.1
0
0
0
0
0
0
0
0
0
0
0
3.6
6.2
0
0
0
0
0
0
0
0
0
0
3.3
5.9
10.8
0
0
0
0
0
0
0
0
0
3.3
6
9.2
17.3
0
0
0
0
0
0
0
0
3.6
5.6
8.9
16.9
23.8
0
0
0
0
0
0
0
3.4
5.4
9.3
16.9
24.5
34
0
0
0
0
0
0
3.3
5.7
9.2
15.3
23.7
35.4
45
0
0
0
0
0
2.8
5.2
8.4
13.7
21.8
35
53.3
60.5
0
0
0
0
2
3.9
6.6
11.4
18.3
30.1
52.4
58.4
64.2
0
0
0
1.5
2.4
4.4
7.5
13.6
23.3
40.3
60
66.9
64.7
0
0
1.4
1.6
2.7
4.9
8.8
18.1
27.4
44.6
58.9
67.7
67.7
0
1.2
1.1
1.6
3.4
5.8
10.2
18.2
33.4
52.5
67.1
66.5
66.4
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
40000 32000 24000 16000 8000 0
100 80 60 40 20 0
Software RAID0 over 8 Intel P3700 as blockdevice (Test ID 5) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
9.1472e+06
0
9.25028e+06 1.70142e+07
1e7
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
8.91483e+06 1.57426e+07 1.88383e+07
9.02593e+06 1.68685e+07 1.69925e+07 1.26002e+07
9.35814e+06 1.60769e+07 1.86092e+07 1.58752e+07 1.79597e+07
9.47305e+06 1.51893e+07 1.81688e+07 1.84488e+07 1.74004e+07 1.53418e+07
9.73257e+06 1.49358e+07 1.76519e+07 1.78217e+07 1.57286e+07 1.41152e+07 1.13973e+07
7.98636e+06 1.05793e+07 1.43995e+07 1.70406e+07 1.44996e+07 1.37579e+07 1.23425e+07 1.00142e+07
7.0202e+06 5.76529e+06 9.40406e+06 1.41105e+07 1.67168e+07 1.4463e+07 9.01164e+06 1.02211e+07 1.00895e+07
3.15494e+06 2.76747e+06 5.10545e+06 8.61453e+06 1.24737e+07 1.5029e+07 9.00069e+06 9.24496e+06 8.74384e+06 9.55521e+06
1.75117e+06 1.26228e+06 2.50826e+06 4.81178e+06 7.98248e+06 1.07938e+07 1.33469e+07 9.54196e+06 1.25887e+07 9.16071e+06 1.01821e+07 1.64489e+06
583040
1
2
8
16
32
64
128
number of workers
256
512
1024
2048
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
43264
0
0
0
0
0
0
0
0
0
0
0
21632
32128
0
0
0
0
0
0
0
0
0
0
10560
17024
195584
0
0
0
0
0
0
0
0
0
4960
6176
68096
166912
0
0
0
0
0
0
0
0
2992
3312
30592
62208
246784
0
0
0
0
0
0
0
1560
2040
15808
34048
146432
288768
0
0
0
0
0
0
804
1336
4448
13248
25472
122368
142336
0
0
0
0
0
628
860
2640
5344
23424
51456
185344
610304
0
0
0
0
516
724
1656
1960
3504
13888
45824
220160
419840
0
0
0
378
684
1480
1768
1800
4448
24704
108032
234496
602112
0
0
334
684
1528
1784
1224
2160
5408
32128
43776
329728
577536
0
290
684
1592
1976
1176
1720
3024
13504
41216
185344
448512
460800
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.9
0
0
0
0
0
0
0
0
0
0
0
2.9
5.6
0
0
0
0
0
0
0
0
0
0
2.9
5.2
7.6
0
0
0
0
0
0
0
0
0
2.9
5.6
9.1
8.9
0
0
0
0
0
0
0
0
3
5.3
8.2
9.5
10.1
0
0
0
0
0
0
0
2.9
5.3
8.1
9
11.2
12.8
0
0
0
0
0
0
3.2
5.3
8.9
8.5
10
12.2
14.8
0
0
0
0
0
2.4
2.9
5.4
8.5
9.2
12.1
13.8
14.7
0
0
0
0
1.9
2.1
3.4
6
9.7
11.5
14.4
14.7
15.7
0
0
0
1.8
1.4
2.2
3.5
6.8
11
15
15.7
14.9
14.9
0
0
1.3
1.2
1.6
2.4
4.9
11.3
17.4
15.3
15.8
15.5
15.1
0
0.8
1.1
1.7
2
4.2
10
16.5
15
15.2
15.8
14.3
12.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1.2 0.8 0.4
0
1.12051e+06 2.39569e+06 4.3537e+06 6.91626e+06 9.01007e+06 1.02473e+07 9.31892e+06 9.29855e+06 9.48815e+06 1.09666e+07
4
1.6
0.0
600000
450000
300000
150000
0
100 80 60 40 20 0
Software RAID0 over 8 Intel P3700 as blockdevice (Test ID 5) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
3.51633e+06
0
3.22004e+06 5.89023e+06
1e7
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3.59136e+06 6.61819e+06 1.13045e+07
3.22189e+06 7.29792e+06 1.14391e+07 1.44006e+07
5.2031e+06 9.87188e+06 1.31304e+07 1.46332e+07 1.40294e+07
4.91959e+06 1.15432e+07 1.46754e+07 1.45359e+07 8.79895e+06 1.14464e+07
6.17391e+06 1.11629e+07 1.46984e+07 1.45793e+07 9.1388e+06 1.21272e+07 1.34992e+07
7.38984e+06 1.1837e+07 1.40233e+07 1.26976e+07 1.21275e+07 1.3903e+07 1.16851e+07 1.33367e+07
7.07789e+06 1.09002e+07 1.31022e+07 8.93424e+06 1.32614e+07 1.36072e+07 1.38241e+07 1.29494e+07 1.41335e+07
4.13458e+06 9.59496e+06 1.16367e+07 1.09608e+07 1.29587e+07 1.39552e+07 1.19708e+07 1.22856e+07 1.13808e+07 1.39046e+07
2.17658e+06 5.46781e+06 9.77123e+06 1.13694e+07 1.23133e+07 1.3312e+07 1.12788e+07 1.02853e+07 1.07977e+07 1.42985e+07 1.38256e+07
0
1.586e+06 2.68189e+06 5.42941e+06 9.33484e+06 1.13098e+07 1.16402e+07 1.14994e+07 1.00419e+07 1.27283e+07 1.3871e+07 1.40208e+07 1.35821e+07
1
2
4
8
16
32
64
128
number of workers
256
512
1024
2048
1.25 1.00 0.75 0.50 0.25 0.00
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
77312
0
0
0
0
0
0
0
0
0
0
0
45824
49408
0
0
0
0
0
0
0
0
0
0
20608
28544
37632
0
0
0
0
0
0
0
0
0
11072
13504
20608
268288
0
0
0
0
0
0
0
0
4960
11968
62208
154624
370688
0
0
0
0
0
0
0
1992
10688
24960
67072
468992
497664
0
0
0
0
0
0
980
8640
13120
29312
201728
252928
403456
0
0
0
0
0
370
1624
7200
28032
46336
128512
391168
634880
0
0
0
0
223
900
2672
17536
16512
34048
101888
264192
561152
0
0
0
278
175
1064
7776
8896
13632
46848
111104
337920
419840
0
0
255
167
247
1784
3824
7200
28544
57088
148480
189440
456704
0
175
153
151
676
1608
7008
13376
31872
36096
54016
173056
370688
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.7
0
0
0
0
0
0
0
0
0
0
0
2.6
5
0
0
0
0
0
0
0
0
0
0
2.7
4.9
8.8
0
0
0
0
0
0
0
0
0
2.7
4.7
9
11.1
0
0
0
0
0
0
0
0
2.9
5.1
8.1
12.5
11
0
0
0
0
0
0
0
2.9
5
7.9
11.1
6.1
10.2
0
0
0
0
0
0
2.8
4.7
7
8.4
6.2
8.7
12.2
0
0
0
0
0
2.6
4.5
6.4
5.9
6.2
8.1
8.9
12.1
0
0
0
0
2.7
4.6
5.9
4.5
6.8
8.1
11.9
12.6
14.8
0
0
0
2.3
4
5.8
5.7
6.9
9.2
9.9
11.6
10.2
19.1
0
0
1.7
2.9
4.8
6.1
6.9
9.6
9.4
8.9
9.7
17.3
19
0
1
1.6
2.8
4.8
6.4
7
9.7
8.6
13
16.3
17
21
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000
450000
300000
150000
0
100 80 60 40 20 0
Software RAID1 over 2 Intel P3700 as blockdevice (Test ID 6) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
203376
0
0
0
0
0
0
0
0
0
0
0
201420
435565
0
0
0
0
0
0
0
0
0
0
284856
347778
772236
0
0
0
0
0
0
0
0
0
205866
378022
701657
915777
0
0
0
0
0
0
0
0
280796
416857
687902
911604
460065
0
0
0
0
0
0
0
281608
386418
640806
916870
452996
407464
0
0
0
0
0
0
266256
331012
734865
924568
447095
404441
385544
0
0
0
0
0
157146
282255
524081
800134
477034
352071
388218
391274
0
0
0
0
82088
164022
299660
520899
741253
423806
382022
385341
467977
0
0
0
41811
85230
160603
305128
512607
574358
385298
385161
385369
388980
0
0
21078
43290
83864
164544
294185
430069
526585
384373
385282
386394
697835
0
9545
21747
42862
85859
158083
258680
324448
514453
470510
394549
387861
598898
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
10432
0
0
0
0
0
0
0
0
0
0
0
7456
5984
0
0
0
0
0
0
0
0
0
0
2008
3440
3344
0
0
0
0
0
0
0
0
0
1480
1640
1880
6368
0
0
0
0
0
0
0
0
812
860
932
2160
5728
0
0
0
0
0
0
0
438
478
588
3280
3056
6432
0
0
0
0
0
0
302
370
1004
604
1592
3408
28288
0
0
0
0
0
266
219
270
354
884
1976
25216
45312
0
0
0
0
258
165
227
258
532
1208
17024
24960
84480
0
0
0
255
147
197
181
249
1672
8640
22144
20608
23424
0
0
258
135
179
157
195
748
1784
2736
5472
14912
7968
0
294
133
151
139
177
346
1928
1288
1768
5472
15040
10688
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.3
0
0
0
0
0
0
0
0
0
0
0
3.3
5.4
0
0
0
0
0
0
0
0
0
0
2.4
5.8
10
0
0
0
0
0
0
0
0
0
3.4
5.9
10.6
16.2
0
0
0
0
0
0
0
0
2.4
5.8
10.5
16.1
34.4
0
0
0
0
0
0
0
2.4
5.9
10.2
16.9
35.1
66.9
0
0
0
0
0
0
2.4
5
9.6
12.6
35.5
67.5
95.7
0
0
0
0
0
2
4.4
6.7
11.6
31.4
55.6
93.2
94.9
0
0
0
0
1.3
2.4
4.6
9.3
16.9
54.9
95.7
95
49.6
0
0
0
0.9
1.6
3
5.7
11.7
17
92.5
94.5
95.2
94.9
0
0
0.7
1
1.7
3.4
7.2
12.2
25.8
94.6
95.4
95.3
96.5
0
0.6
0.6
1.1
1.9
4.1
8.4
13.1
48
55
94.4
86.3
95.4
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
75000 60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID1 over 2 Intel P3700 as blockdevice (Test ID 6) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
156344
0
0
0
0
0
0
0
0
0
0
0
184408
148382
0
0
0
0
0
0
0
0
0
0
190133
179263
178979
0
0
0
0
0
0
0
0
0
190156
177662
179669
182114
0
0
0
0
0
0
0
0
183508
176672
184450
186665
176148
0
0
0
0
0
0
0
177524
179379
197024
186521
177189
175914
0
0
0
0
0
0
176507
179584
174105
183363
181251
178293
182518
0
0
0
0
0
166453
119920
187674
183149
179237
179504
184566
181254
0
0
0
0
152984
178220
208825
191646
177563
183795
184491
183257
183378
0
0
0
125552
170656
186530
204360
191261
184195
183627
187381
185952
181846
0
0
83957
122354
158275
178969
193656
186224
186383
182745
184414
184466
183921
0
47995
82580
111280
136391
163632
178603
182158
181099
184636
184596
183404
186052
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
200000 160000 120000 80000 40000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
21376
0
0
0
0
0
0
0
0
0
0
0
12992
33024
0
0
0
0
0
0
0
0
0
0
7072
15168
28288
0
0
0
0
0
0
0
0
0
3728
8896
15552
39680
0
0
0
0
0
0
0
0
2704
5920
9408
20096
45824
0
0
0
0
0
0
0
1640
3632
5536
10432
34560
47872
0
0
0
0
0
0
900
2256
3568
6432
17024
34048
52992
0
0
0
0
0
540
2192
2320
3664
6752
16192
28032
59648
0
0
0
0
233
932
1144
2096
3600
10304
19072
34560
55040
0
0
0
96
243
668
1144
2128
5728
14272
24960
40704
59648
0
0
77
107
326
724
1192
3248
8160
12096
27520
43776
89600
0
76
81
171
454
740
1576
2992
4768
9152
15424
35072
61184
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.1
0
0
0
0
0
0
0
0
0
0
0
2.6
3
0
0
0
0
0
0
0
0
0
0
2.7
3.1
3.9
0
0
0
0
0
0
0
0
0
2.7
3.1
3.8
4.9
0
0
0
0
0
0
0
0
2.7
3.1
3.9
5.1
5.4
0
0
0
0
0
0
0
2.7
3.1
4.2
5.7
5.4
10.9
0
0
0
0
0
0
2.6
3.2
3.7
5.6
5.9
10
46.7
0
0
0
0
0
2.6
2.3
4.1
5.2
7
16.7
59
61.3
0
0
0
0
2.5
3.3
4.7
6.1
8.1
16
61
63.8
64.8
0
0
0
2.3
3.3
4.3
5.6
7.9
14.1
56.3
66.7
68
63.3
0
0
2
3
4.1
5.6
7.9
13.3
50.1
59.1
67.3
66.4
64.7
0
1.5
2.3
3.4
5.3
7.1
9.5
19.8
49.1
60.8
64.6
64.3
63.5
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
75000 60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID1 over 2 Intel P3700 as blockdevice (Test ID 6) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
247291
0
0
0
0
0
0
0
0
0
0
0
251135
385454
0
0
0
0
0
0
0
0
0
0
180964
363250
441390
0
0
0
0
0
0
0
0
0
194394
407673
502314
539315
0
0
0
0
0
0
0
0
195873
364301
537821
539566
456640
0
0
0
0
0
0
0
193078
357300
471161
538430
509086
542079
0
0
0
0
0
0
227702
359338
449738
536141
537504
295432
381546
0
0
0
0
0
133585
224665
359097
493909
518006
401572
387217
382856
0
0
0
0
72336
129856
222074
359387
495375
517812
382513
383465
383012
0
0
0
37736
69522
125038
221176
360488
482292
396100
386567
382902
383397
0
0
19234
36100
67026
124136
219158
334535
435296
385335
384739
385435
423432
0
9571
18477
35048
66259
124025
212975
313198
439177
389856
402518
408286
814492
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
9536
0
0
0
0
0
0
0
0
0
0
0
4704
7008
0
0
0
0
0
0
0
0
0
0
4128
3344
50944
0
0
0
0
0
0
0
0
0
1704
1832
23168
9536
0
0
0
0
0
0
0
0
796
1048
2224
5536
11072
0
0
0
0
0
0
0
498
716
6368
2576
7328
22144
0
0
0
0
0
0
258
422
2960
1272
2352
4320
28544
0
0
0
0
0
199
314
756
948
1704
2160
21632
41728
0
0
0
0
173
278
644
740
740
2800
17024
40704
77312
0
0
0
163
266
548
580
446
1112
3696
10816
26240
14400
0
0
153
266
410
482
366
868
1464
2256
4576
5280
18816
0
155
233
314
462
334
398
1240
1416
2800
6240
4448
19328
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.4
0
0
0
0
0
0
0
0
0
0
0
2.4
5.6
0
0
0
0
0
0
0
0
0
0
3.4
5.9
8.4
0
0
0
0
0
0
0
0
0
3.3
5.2
8.3
9.4
0
0
0
0
0
0
0
0
3.4
5.8
7.4
10
28.6
0
0
0
0
0
0
0
3.4
5.5
8.3
9.8
17.8
13.5
0
0
0
0
0
0
2.3
4.8
7.4
9.1
13.3
55.3
95.2
0
0
0
0
0
1.9
3.9
5.8
9.1
11.7
61.7
94.9
94.9
0
0
0
0
1.3
2.1
3.8
6.8
11.6
16.2
95.3
95.3
95
0
0
0
0.9
1.4
2.3
4.5
8.9
13
85.9
92.8
95.2
94.9
0
0
0.6
0.9
1.4
2.8
6
10
15.8
92.3
94
94.9
52.1
0
0.5
0.6
1
1.5
3.4
6.9
11.1
18.8
93.6
87.5
89.9
58.5
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
750000 600000 450000 300000 150000 0
75000 60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID1 over 2 Intel P3700 as blockdevice (Test ID 6) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
90042
0
0
0
0
0
0
0
0
0
0
0
91699
91735
0
0
0
0
0
0
0
0
0
0
90565
91896
90801
0
0
0
0
0
0
0
0
0
98982
88818
89985
91028
0
0
0
0
0
0
0
0
80486
75330
90611
90329
91745
0
0
0
0
0
0
0
97146
83223
98278
90235
93339
91245
0
0
0
0
0
0
99970
96515
93925
92330
95183
92316
92058
0
0
0
0
0
101330
100400
100177
94372
91917
92813
90295
94004
0
0
0
0
93922
101661
96516
92756
92772
90816
89730
94947
94783
0
0
0
82480
94870
99634
97213
93987
94271
91210
89394
100210
94323
0
0
59515
82628
92584
95902
92930
93282
95563
92313
94107
97142
102240
0
37210
56480
72627
88940
93103
92697
93506
91721
96068
92008
92112
93170
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
100000 80000 60000 40000 20000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
47360
0
0
0
0
0
0
0
0
0
0
0
27008
47872
0
0
0
0
0
0
0
0
0
0
16064
22912
55552
0
0
0
0
0
0
0
0
0
10944
15936
28544
55040
0
0
0
0
0
0
0
0
7200
11328
16320
31104
77312
0
0
0
0
0
0
0
3472
6816
10688
19072
40704
132096
0
0
0
0
0
0
1976
3568
6368
12352
23424
67072
183296
0
0
0
0
0
1192
2024
3440
6496
11968
27264
98816
193536
0
0
0
0
860
1240
2128
3728
6496
12224
28544
122368
284672
0
0
0
406
852
1288
2096
3728
6496
12224
40192
125440
250880
0
0
141
422
876
1320
2256
3792
6624
12224
31616
104960
236544
0
91
219
564
796
1288
2320
3728
6624
11328
44288
49920
118272
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3
0
0
0
0
0
0
0
0
0
0
0
2
2.3
0
0
0
0
0
0
0
0
0
0
1.7
2
2.6
0
0
0
0
0
0
0
0
0
1.8
1.9
2.4
3.1
0
0
0
0
0
0
0
0
1.6
1.7
2.4
3
4.5
0
0
0
0
0
0
0
1.8
2
2.7
3
3.9
5.1
0
0
0
0
0
0
1.9
2.2
2.6
3.3
3.8
5.4
7
0
0
0
0
0
1.9
2.2
2.7
3.2
4.5
5.9
6.6
7.1
0
0
0
0
1.9
2.3
2.7
3.2
4.5
6.5
9.3
11
12.2
0
0
0
1.8
2.4
2.8
3.3
4.4
7.1
11.1
10.4
14.3
12.2
0
0
1.5
2.2
2.9
3.4
4.8
7.4
10.6
12.2
14.1
12
14.1
0
1.2
1.8
2.4
3.3
4.7
7.3
10.3
10.9
15.2
12
12.2
12.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
250000 200000 150000 100000 50000 0
100 80 60 40 20 0
Software RAID1 over 2 Intel P3700 as blockdevice (Test ID 6) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
60150
0
0
0
0
0
0
0
0
0
0
0
64125
64819
0
0
0
0
0
0
0
0
0
0
64174
64801
64448
0
0
0
0
0
0
0
0
0
61179
65448
64752
64964
0
0
0
0
0
0
0
0
55781
67256
65573
65516
65281
0
0
0
0
0
0
0
49929
62760
66181
66553
64462
65343
0
0
0
0
0
0
35879
48422
62967
66060
65412
65358
64924
0
0
0
0
0
23620
34791
49297
63544
66832
65970
64724
64416
0
0
0
0
13508
23142
34124
49030
62768
65888
64214
64948
64769
0
0
0
7000
13799
22757
34420
48426
62758
65980
65472
65377
65163
0
0
3546
6918
13650
22644
34238
48597
60279
65395
64871
64822
64873
0
2386
4595
8714
14314
22224
32815
44532
57765
63878
65281
66309
65645
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
17024
0
0
0
0
0
0
0
0
0
0
0
11456
24448
0
0
0
0
0
0
0
0
0
0
8640
14144
26240
0
0
0
0
0
0
0
0
0
6688
9536
15680
34560
0
0
0
0
0
0
0
0
3856
5920
9664
16512
43776
0
0
0
0
0
0
0
932
2320
6560
11200
21120
52480
0
0
0
0
0
0
270
924
2448
6240
10816
31360
82432
0
0
0
0
0
74
310
852
2352
6176
12480
34048
82432
0
0
0
0
118
72
366
948
2288
6176
13248
35584
116224
0
0
0
127
85
83
239
940
2384
6432
13376
34560
115200
0
0
131
110
91
87
310
836
2704
6240
13248
49408
59136
0
77
89
76
83
109
580
1320
2672
6432
13248
46848
71168
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.8
0
0
0
0
0
0
0
0
0
0
0
2.8
3.9
0
0
0
0
0
0
0
0
0
0
2.8
3.6
4.2
0
0
0
0
0
0
0
0
0
2.6
3.4
4.3
5
0
0
0
0
0
0
0
0
2.5
3.4
4.2
4.9
6.2
0
0
0
0
0
0
0
2.3
3.3
4.1
5
6
7
0
0
0
0
0
0
2
3
4.1
5
6.5
6.9
7.6
0
0
0
0
0
1.5
2.3
3.4
4.7
6.1
7.3
8.4
11.2
0
0
0
0
1.1
1.7
2.6
3.9
6
7.5
9.7
12.5
12.9
0
0
0
0.8
1.3
1.9
3
4.8
7.7
12
13.1
14.3
14.1
0
0
0.6
0.9
1.3
2.1
3.9
6.5
11.1
13.1
13.6
14.7
14.7
0
0.5
0.7
1.1
1.5
2.5
5
9.4
11.7
13.7
14.8
15.5
16.2
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
100000 80000 60000 40000 20000 0
100 80 60 40 20 0
Software RAID1 over 2 Intel P3700 as blockdevice (Test ID 6) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
2.39192e+06
0
2.4035e+06 4.79188e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2.40596e+06 4.78653e+06 4.75701e+06
2.40761e+06 4.79615e+06 4.76181e+06 4.78597e+06
2.40684e+06 4.80041e+06 4.50228e+06 4.78941e+06 4.22424e+06
2.40646e+06 4.79268e+06 4.66683e+06 4.77984e+06 4.39899e+06 3.50164e+06
2.40451e+06 4.79991e+06 4.73791e+06 4.74678e+06 4.27963e+06 4.42566e+06 3.38937e+06
2.37224e+06 4.76038e+06 4.73853e+06 4.74493e+06 4.44299e+06 3.61598e+06 3.84612e+06 3.76766e+06
1.99124e+06 3.98541e+06 4.56838e+06 4.71696e+06 4.12208e+06 3.97283e+06 3.42609e+06 3.73141e+06 3.31105e+06
1.37678e+06 2.75236e+06 3.66936e+06 4.49555e+06 4.54882e+06 3.91444e+06 3.62125e+06 3.50552e+06 3.39381e+06 3.09103e+06
1.05371e+06 2.08364e+06 2.44788e+06 3.57268e+06 4.25618e+06 3.81066e+06 3.4381e+06 3.49217e+06 3.05887e+06 3.11956e+06 2.9703e+06 1.179e+06
800166
1
2
8
16
32
64
128
number of workers
3000000 2000000 1000000
0
1.34338e+06 2.18921e+06 3.31295e+06 4.26688e+06 3.73526e+06 3.38767e+06 3.60681e+06 3.76087e+06 2.99909e+06 3.04155e+06
4
4000000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
252928
0
0
0
0
0
0
0
0
0
0
0
142336
140288
0
0
0
0
0
0
0
0
0
0
47872
74240
162816
0
0
0
0
0
0
0
0
0
22400
24448
71168
140288
0
0
0
0
0
0
0
0
13248
22144
35072
76288
193536
0
0
0
0
0
0
0
6560
8384
14656
33024
78336
370688
0
0
0
0
0
0
3248
3568
7200
14528
36608
81408
518144
0
0
0
0
0
1800
1784
3888
7392
16320
48384
179200
374784
0
0
0
0
996
988
2768
4128
9280
16768
53504
175104
618496
0
0
0
652
652
2320
2672
4640
9408
17536
54016
240640
569344
0
0
716
732
2288
2320
3056
5216
10048
18560
57600
214016
684032
0
438
676
2224
2384
2704
3088
5088
11584
21632
63744
284672
296960
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
1.7
0
0
0
0
0
0
0
0
0
0
0
1.2
2.3
0
0
0
0
0
0
0
0
0
0
1
1.8
2.1
0
0
0
0
0
0
0
0
0
1
1.7
2.1
2.2
0
0
0
0
0
0
0
0
0.9
1.7
2
2.1
2.5
0
0
0
0
0
0
0
0.9
1.6
2
2
3.3
4.6
0
0
0
0
0
0
0.9
1.4
1.8
1.8
3.2
2.4
3.9
0
0
0
0
0
0.7
1.2
1.4
1.6
2.1
4.1
3.1
3.8
0
0
0
0
0.6
1
1.4
1.6
3
2.3
3.6
3.4
4
0
0
0
0.6
0.8
1.2
1.6
2.1
2.5
2.5
3.7
3.6
4.9
0
0
0.6
0.7
1
1.4
2.9
3.2
3.2
3
4.1
4.2
5
0
0.5
0.8
0.9
1.2
1.7
3.4
4.1
3.4
3.1
2.8
5
5.5
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000 450000 300000 150000 0
100 80 60 40 20 0
Software RAID1 over 2 Intel P3700 as blockdevice (Test ID 6) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
1.77718e+06
0
1.92082e+06 1.90446e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1.89409e+06 1.90736e+06 1.62386e+06
1.89539e+06 1.91269e+06 1.79332e+06 1.90704e+06
1.89212e+06 1.90335e+06 1.93e+06 1.91836e+06 1.60311e+06
1.87699e+06 1.84531e+06 1.63445e+06 1.92047e+06 1.53903e+06 1.59796e+06
1.85932e+06 1.84009e+06 1.64259e+06 1.90811e+06 1.565e+06 1.60315e+06 1.44054e+06
1.81922e+06 1.78892e+06 1.62301e+06 1.90815e+06 1.60870e+06 1.5377e+06 1.73594e+06 1.81238e+06
1.7273e+06 1.57852e+06 1.602e+06 1.5804e+06 1.5423e+06 1.63719e+06 1.75064e+06 1.73259e+06 1.64949e+06
1.63938e+06 1.19656e+06 1.45775e+06 1.13089e+06 1.53292e+06 1.34095e+06 1.78998e+06 1.36001e+06 1.65164e+06 1.61248e+06 1.68412e+06 1.41303e+06 1.65521e+06 1.02164e+06 1.49639e+06 1.16408e+06 1.69924e+06
750828
1.69539e+06 1.64883e+06 1.66424e+06
2
4
8
16
32
64
128
number of workers
256
512
1024
2048
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
358400
0
0
0
0
0
0
0
0
0
0
0
134144
329728
0
0
0
0
0
0
0
0
0
0
54016
142336
561152
0
0
0
0
0
0
0
0
0
35072
78336
148480
350208
0
0
0
0
0
0
0
0
21888
45312
76288
166912
468992
0
0
0
0
0
0
0
15936
28032
44800
74240
242688
897024
0
0
0
0
0
0
10560
15936
28032
40192
120320
395264
880640
0
0
0
0
0
4016
10304
15424
20608
55552
144384
309248
452608
0
0
0
0
2576
6880
8896
20608
33024
54528
261120
366592
1.51552e+06
0
0
0
1768
6048
10176
14016
17792
45824
40704
228352
593920
872448
0
0
1032
2800
2960
7712
11328
22144
33536
166912
179200
585728
1.1223e+06
0
466
1848
1800
3280
4640
12992
15168
41728
47360
423936
544768
921600
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
1.9
0
0
0
0
0
0
0
0
0
0
0
1.8
2.7
0
0
0
0
0
0
0
0
0
0
1.7
2.8
2.7
0
0
0
0
0
0
0
0
0
1.9
2.6
2.6
3.4
0
0
0
0
0
0
0
0
1.2
1.6
2.8
3.4
3.2
0
0
0
0
0
0
0
1.1
1.2
1.7
3.1
2.7
3
0
0
0
0
0
0
1.1
1.2
1.4
2.7
2.7
2.7
3
0
0
0
0
0
1
1.1
1.2
1.5
2.2
2.5
3.4
4.4
0
0
0
0
1
1
1.2
1.3
1.7
2.4
3.2
3.8
4.8
0
0
0
1
0.9
1.1
1.1
1.4
1.3
3
2.4
4.1
5.3
0
0
1
1
1.3
1
1.2
1.1
2.4
1.5
3.7
4.4
5.8
0
0.9
1
1.3
1.2
1.4
1.2
1.7
1.3
3.3
3.8
4.5
5.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1200000 800000 400000
0
1.55821e+06 1.48413e+06 1.68474e+06 1.38626e+06 1.7609e+06 1.36633e+06 1.66844e+06 1.11086e+06 1.80478e+06 1.63875e+06 1.68718e+06 1.65238e+06
1
1600000
0
1500000 1200000 900000 600000 300000 0
100 80 60 40 20 0
Software RAID10 over 4 Intel P3700 as blockdevice (Test ID 7) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
184429
0
0
0
0
0
0
0
0
0
0
0
190146
351067
0
0
0
0
0
0
0
0
0
0
246810
390924
649882
0
0
0
0
0
0
0
0
0
195376
394624
681306
990851
0
0
0
0
0
0
0
0
195775
370186
691041
936585
433123
0
0
0
0
0
0
0
237986
363828
610218
991175
445409
390534
0
0
0
0
0
0
215296
312883
495712
955154
436879
397409
379934
0
0
0
0
0
157053
276172
578332
913708
441208
387651
376742
376162
0
0
0
0
83228
160874
316854
529314
575906
335554
377629
378199
377950
0
0
0
41991
83145
166140
306798
559055
657252
381887
377989
378648
377243
0
0
20193
41889
85443
162292
312588
518565
639122
378990
383145
378114
579521
0
10809
19822
42746
80648
158411
264658
440091
643053
383344
384971
815366
767200
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
15680
0
0
0
0
0
0
0
0
0
0
0
6240
6880
0
0
0
0
0
0
0
0
0
0
2320
3344
5600
0
0
0
0
0
0
0
0
0
1528
1560
2992
10176
0
0
0
0
0
0
0
0
836
916
988
1416
6752
0
0
0
0
0
0
0
470
524
828
2384
3312
7008
0
0
0
0
0
0
302
386
2192
1352
1832
3728
27776
0
0
0
0
0
255
262
229
612
980
2256
21376
53504
0
0
0
0
229
209
197
498
908
1928
17024
48384
33024
0
0
0
207
221
183
253
278
1096
6112
17280
9280
24704
0
0
233
217
147
213
205
438
1448
2544
2992
5536
4256
0
193
253
145
223
203
346
700
1640
2608
4960
3728
5024
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6
0
0
0
0
0
0
0
0
0
0
0
3.6
6.2
0
0
0
0
0
0
0
0
0
0
2.7
5.7
10.2
0
0
0
0
0
0
0
0
0
3.5
6.4
10.9
18.9
0
0
0
0
0
0
0
0
3.6
6.4
11.5
20.3
36.7
0
0
0
0
0
0
0
2.7
5.6
10
18.6
37.3
67
0
0
0
0
0
0
3
4.7
9
18.1
37.8
67.8
95.2
0
0
0
0
0
2.5
4.8
9.1
15.7
35.8
68.7
94.9
87.3
0
0
0
0
1.7
3.4
6.3
11.2
29.5
57.7
95.5
93.2
95.1
0
0
0
1.2
2.1
4
7.6
14.2
26
91.2
94
94.2
95
0
0
0.9
1.3
2.4
4.5
8.6
15.8
30
94.2
94.4
94.7
96.9
0
0.5
0.9
1.5
2.9
5.5
9.9
17.6
35.3
94.3
94.8
96.5
97.3
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
50000 40000 30000 20000 10000 0
100 80 60 40 20 0
Software RAID10 over 4 Intel P3700 as blockdevice (Test ID 7) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
123464
0
0
0
0
0
0
0
0
0
0
0
129688
218750
0
0
0
0
0
0
0
0
0
0
125185
194271
346505
0
0
0
0
0
0
0
0
0
132897
180476
368140
350486
0
0
0
0
0
0
0
0
132614
213024
339596
355178
284700
0
0
0
0
0
0
0
133890
176686
324519
357152
281736
269092
0
0
0
0
0
0
135579
180052
311634
402811
291173
262650
257667
0
0
0
0
0
138754
186132
291220
351061
298813
266381
269064
263112
0
0
0
0
130108
170378
332467
368368
294053
264304
250423
263077
305600
0
0
0
120934
189767
296811
366308
259247
259203
256110
262939
270153
261431
0
0
73318
131894
219456
258520
345642
310962
275027
253859
264079
261764
269669
0
39458
69338
128969
172052
247432
304793
327568
269908
245657
240745
245145
290346
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
400000 320000 240000 160000 80000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
20608
0
0
0
0
0
0
0
0
0
0
0
13760
13888
0
0
0
0
0
0
0
0
0
0
5280
8384
40192
0
0
0
0
0
0
0
0
0
2576
4384
12736
28544
0
0
0
0
0
0
0
0
1176
2160
11456
16512
41728
0
0
0
0
0
0
0
700
1256
2192
9280
20864
16192
0
0
0
0
0
0
342
652
2672
4832
10560
10944
35584
0
0
0
0
0
169
338
1944
4448
6368
5408
27264
42752
0
0
0
0
109
201
708
1736
2384
2416
18048
37632
69120
0
0
0
65
91
266
860
1592
3760
16512
31104
64768
132096
0
0
56
74
104
548
916
3504
2896
9792
21376
45824
94720
0
47
61
77
187
506
1144
1848
2640
3984
10048
19072
40192
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.3
0
0
0
0
0
0
0
0
0
0
0
3
6.3
0
0
0
0
0
0
0
0
0
0
3.3
5.8
9
0
0
0
0
0
0
0
0
0
3.1
4.8
9.1
10.8
0
0
0
0
0
0
0
0
3.2
5.8
9.1
10.4
34.5
0
0
0
0
0
0
0
3.1
4.8
9
12.8
33.7
64.1
0
0
0
0
0
0
2.9
4.8
8.6
14.7
33
64.3
92.3
0
0
0
0
0
2.9
4.8
8.7
12.2
20.3
62.9
92.7
93.9
0
0
0
0
3
4.7
9.5
14.4
30.8
61.3
93.2
93.3
85.4
0
0
0
3.3
5.4
9
12.6
30.2
50.7
92.9
93.6
93.6
95
0
0
2.6
4.4
7.6
10
16.1
27.3
81.2
93.2
93.7
94.1
85.9
0
1.9
3.2
5.5
8.1
12.6
15.9
34.3
83.2
93.7
93.4
94
91.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
125000 100000 75000 50000 25000 0
100 80 60 40 20 0
Software RAID10 over 4 Intel P3700 as blockdevice (Test ID 7) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
175966
0
0
0
0
0
0
0
0
0
0
0
180708
327563
0
0
0
0
0
0
0
0
0
0
175493
365780
611759
0
0
0
0
0
0
0
0
0
228977
355112
638214
953947
0
0
0
0
0
0
0
0
188064
378887
594206
906135
431402
0
0
0
0
0
0
0
231038
351444
611741
927149
445509
397038
0
0
0
0
0
0
187286
293701
561422
873087
432021
383441
379027
0
0
0
0
0
134080
247574
395062
644827
494644
396316
382602
367160
0
0
0
0
72330
136228
246392
430052
645075
414013
378546
377708
378830
0
0
0
36876
71697
131793
240292
412056
621600
384413
383612
378353
378453
0
0
17623
35729
68242
130619
241656
394559
530879
377596
379020
377996
778046
0
9616
17446
34373
66964
126237
210573
383674
526566
383613
384690
386449
386894
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
13248
0
0
0
0
0
0
0
0
0
0
0
7968
7904
0
0
0
0
0
0
0
0
0
0
3632
3344
5856
0
0
0
0
0
0
0
0
0
1624
1768
2864
9920
0
0
0
0
0
0
0
0
844
1004
1608
4704
6560
0
0
0
0
0
0
0
426
548
1020
2992
3248
7136
0
0
0
0
0
0
282
438
1912
1608
2096
3728
27264
0
0
0
0
0
195
247
692
1528
1720
2128
18560
58112
0
0
0
0
171
205
302
438
740
1688
17024
17280
69120
0
0
0
167
177
298
378
494
828
3664
11584
7136
15680
0
0
229
199
306
310
314
596
1688
2320
3088
3888
12224
0
155
227
294
266
310
446
442
1560
2672
3792
11968
25728
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.5
0
0
0
0
0
0
0
0
0
0
0
3.5
6.4
0
0
0
0
0
0
0
0
0
0
3.5
6.4
10.9
0
0
0
0
0
0
0
0
0
2.7
6.4
10.9
18.4
0
0
0
0
0
0
0
0
3.5
6.2
10
19.1
36.9
0
0
0
0
0
0
0
2.7
6.4
10.5
17.1
36.1
69.2
0
0
0
0
0
0
3.3
5.2
9.4
16.2
36.9
63
95.4
0
0
0
0
0
2.3
4.6
8
13
33.8
63.9
94.9
80.6
0
0
0
0
1.5
2.9
5.5
9.8
17.2
50.1
95.5
94.2
95.2
0
0
0
1.1
1.8
3.4
6.2
11.9
22.9
88.9
94.9
94.4
94.7
0
0
0.8
1.2
2
3.7
7.3
13.1
21
90.6
94.9
94.7
65.7
0
0.5
0.9
1.3
2.6
4.7
8.3
16.4
28.1
95
94.3
94.5
92.5
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID10 over 4 Intel P3700 as blockdevice (Test ID 7) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
111255
0
0
0
0
0
0
0
0
0
0
0
124970
184963
0
0
0
0
0
0
0
0
0
0
120692
167088
177770
0
0
0
0
0
0
0
0
0
127900
185038
176876
187751
0
0
0
0
0
0
0
0
125795
167605
180688
186217
176332
0
0
0
0
0
0
0
121175
158263
191497
178274
165170
175576
0
0
0
0
0
0
117191
154934
188071
184527
177610
176969
176181
0
0
0
0
0
115914
175285
195856
191226
187501
177487
206599
167658
0
0
0
0
122881
172101
196288
197901
189125
187068
152182
177905
171364
0
0
0
101225
139588
182194
198894
199282
188272
194559
182051
157973
181913
0
0
59583
103206
146304
184869
183968
204162
199951
186808
141625
146640
178883
0
35388
63761
97755
126742
162890
147721
153851
185616
192528
124660
180555
189970
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
200000 160000 120000 80000 40000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
37632
0
0
0
0
0
0
0
0
0
0
0
12096
22912
0
0
0
0
0
0
0
0
0
0
8384
14272
41216
0
0
0
0
0
0
0
0
0
3568
10816
24448
47872
0
0
0
0
0
0
0
0
1848
3856
16064
24448
94720
0
0
0
0
0
0
0
1240
2096
10048
15296
57600
113152
0
0
0
0
0
0
860
1720
5856
10432
20096
59648
150528
0
0
0
0
0
490
1352
2896
5728
10304
27264
41728
168960
0
0
0
0
237
772
1656
2928
5792
10560
24960
42752
276480
0
0
0
93
266
916
1672
2928
5856
11840
20096
70144
201728
0
0
85
104
524
908
1720
2960
5664
10816
28544
127488
199680
0
77
79
171
494
1004
2896
5472
6176
10432
53504
50944
177152
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.2
0
0
0
0
0
0
0
0
0
0
0
2.9
5.3
0
0
0
0
0
0
0
0
0
0
3
4.8
5.9
0
0
0
0
0
0
0
0
0
2.9
4.8
5.7
6.9
0
0
0
0
0
0
0
0
2.9
4.6
5.9
6.7
7.5
0
0
0
0
0
0
0
3
4.5
6.2
6.8
6.9
12.8
0
0
0
0
0
0
3
4.4
6.2
6.9
9.4
9.7
11.5
0
0
0
0
0
2.9
4.9
6.5
7.4
8.5
11
53.5
25.4
0
0
0
0
3
5.6
6.3
8
9.2
19.3
28.6
36.3
29.2
0
0
0
3.1
4.9
6.5
8.1
9.4
13.6
39.9
39.2
28.2
41.1
0
0
2.4
4.1
5.6
8.1
8.8
11.7
27.1
35.5
24.4
25.4
35.4
0
1.6
2.8
4.3
7
8.8
9.4
13.2
24.8
39.2
18.4
36.5
41.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
250000 200000 150000 100000 50000 0
100 80 60 40 20 0
Software RAID10 over 4 Intel P3700 as blockdevice (Test ID 7) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
48144
0
0
0
0
0
0
0
0
0
0
0
45757
67001
0
0
0
0
0
0
0
0
0
0
47408
64758
114478
0
0
0
0
0
0
0
0
0
52240
71615
119400
124192
0
0
0
0
0
0
0
0
52609
66378
119146
127608
116630
0
0
0
0
0
0
0
45994
68171
118184
125342
117500
97544
0
0
0
0
0
0
42123
67023
97366
121938
125402
93078
97476
0
0
0
0
0
25410
45811
70415
97758
118194
97365
96437
96887
0
0
0
0
12062
24565
45120
68939
97097
110604
97063
100191
103374
0
0
0
6448
12443
25605
44980
68738
95432
117772
96107
96182
102128
0
0
3515
7417
12722
25918
43899
67378
91140
114457
96112
101783
97846
0
2432
5341
8836
17446
27221
42409
65398
85185
113201
96692
104289
116913
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
125000 100000 75000 50000 25000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
15040
0
0
0
0
0
0
0
0
0
0
0
9664
14016
0
0
0
0
0
0
0
0
0
0
4320
7840
34048
0
0
0
0
0
0
0
0
0
1960
7712
16320
45312
0
0
0
0
0
0
0
0
884
3280
8896
17536
71168
0
0
0
0
0
0
0
450
868
5344
9280
16320
14144
0
0
0
0
0
0
213
346
1144
5408
9536
10688
31616
0
0
0
0
0
187
109
270
1112
6176
7456
24192
41216
0
0
0
0
203
179
91
366
1176
5920
12864
32128
61184
0
0
0
171
167
123
97
422
1352
4512
8096
19840
34048
0
0
135
113
126
107
119
394
1560
4768
7584
17792
31616
0
141
66
103
79
102
253
450
2384
4512
9536
17024
30336
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.8
0
0
0
0
0
0
0
0
0
0
0
2.9
4.7
0
0
0
0
0
0
0
0
0
0
2.8
4.5
8.6
0
0
0
0
0
0
0
0
0
2.8
4.6
8.6
10.9
0
0
0
0
0
0
0
0
2.8
4.8
8.6
11
23.2
0
0
0
0
0
0
0
3.2
5.4
8.6
11.5
25
64.6
0
0
0
0
0
0
3.1
4.6
7.7
10.7
14.9
63
94.1
0
0
0
0
0
2.2
3.7
5.8
8.8
12.8
60.8
93.6
95.2
0
0
0
0
1.4
2.4
4.2
6.8
10.5
21.5
92.1
94.4
95.1
0
0
0
1
1.5
2.9
4.8
7.4
11.9
22.2
93.9
94.6
81.6
0
0
0.9
1.3
1.8
3.4
5.4
9.1
15.2
26.2
93.6
91.7
94.3
0
0.7
1.1
1.5
2.4
3.7
6.6
9.6
15.7
31.2
93.5
93.2
94.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID10 over 4 Intel P3700 as blockdevice (Test ID 7) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
8.25143e+06
0
8.21883e+06 9.5521e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
8.67074e+06 9.54211e+06 9.56213e+06
8.74065e+06 9.63729e+06 8.39779e+06 9.07503e+06
8.11328e+06 9.57638e+06 9.55173e+06 9.24845e+06 9.30096e+06
7.56458e+06 9.54701e+06 9.54051e+06 9.46445e+06 9.05529e+06 7.08579e+06
6.44956e+06 9.37703e+06 9.39426e+06 9.05865e+06 8.65828e+06 8.71222e+06 7.20118e+06
4.99566e+06 7.51774e+06 8.58881e+06 9.32997e+06 8.99907e+06 6.60453e+06 5.81367e+06 5.86367e+06
2.87705e+06 5.16262e+06 6.87338e+06 9.02489e+06 8.54635e+06 7.39874e+06 5.33496e+06 5.31893e+06 7.44703e+06
1.50732e+06 3.3191e+06 4.6648e+06 6.13268e+06 8.03813e+06 7.81001e+06 6.75325e+06 5.08649e+06 5.69299e+06 6.54435e+06
1.29026e+06 1.37124e+06 2.41312e+06 4.04073e+06 5.78097e+06 6.12463e+06 5.00277e+06 4.93048e+06 5.87416e+06 6.76239e+06 6.90898e+06 2.07773e+06
631680
1
2
8
16
32
64
128
number of workers
6000000 4000000 2000000
0
1.21029e+06 2.33226e+06 3.5735e+06 4.92694e+06 4.60178e+06 6.52749e+06 5.00898e+06 5.75222e+06 8.09673e+06 5.35497e+06
4
8000000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
37632
0
0
0
0
0
0
0
0
0
0
0
19840
114176
0
0
0
0
0
0
0
0
0
0
12736
65280
91648
0
0
0
0
0
0
0
0
0
8896
24960
81408
113152
0
0
0
0
0
0
0
0
5088
12224
33024
57088
268288
0
0
0
0
0
0
0
3152
5920
18304
35584
115200
528384
0
0
0
0
0
0
1656
2992
8768
19840
42752
187392
444416
0
0
0
0
0
868
1784
3440
8896
15680
62720
288768
675840
0
0
0
0
636
1816
2608
3024
10816
20352
61696
264192
913408
0
0
0
612
1720
2480
2704
3696
11584
31616
123392
724992
692224
0
0
580
868
2224
2352
2864
5024
16192
40192
114176
411648
618496
0
155
652
2416
2320
2640
3024
7584
13120
38144
168960
222208
411648
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.1
0
0
0
0
0
0
0
0
0
0
0
3.1
3.5
0
0
0
0
0
0
0
0
0
0
2.6
3.6
4
0
0
0
0
0
0
0
0
0
2.8
3.4
4.5
4.5
0
0
0
0
0
0
0
0
2.9
3.7
3.8
4.3
4.7
0
0
0
0
0
0
0
2.4
3.3
3.6
4.2
4.8
6.5
0
0
0
0
0
0
2.2
2.9
3.4
3.8
5.5
5.2
7.5
0
0
0
0
0
1.5
2.4
2.5
3.6
4.3
7.2
7.5
7.7
0
0
0
0
1.2
1.7
2.4
3.2
4.6
7.5
8
8.2
8.3
0
0
0
1
1.3
1.9
2.7
4.1
6.8
6.7
7.9
7.7
7.2
0
0
0.8
1.1
1.5
2.3
3.4
6.7
8.4
7.7
7.9
7.3
6.8
0
0.7
1
1.3
1.9
4.3
7.9
8.5
10.5
7.2
8.4
5
8.9
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
800000 600000 400000 200000 0
100 80 60 40 20 0
Software RAID10 over 4 Intel P3700 as blockdevice (Test ID 7) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
2.61814e+06
0
2.81023e+06 3.70498e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2.92879e+06 3.82184e+06 3.6169e+06
3.08058e+06 3.82352e+06 3.6436e+06 2.26014e+06
3.71666e+06 3.80553e+06 3.70254e+06 3.06578e+06 3.42718e+06
3.60904e+06 3.76675e+06 3.53887e+06 3.50502e+06 3.39212e+06 3.26298e+06
3.32814e+06 3.62934e+06 3.45634e+06 2.5469e+06 2.78473e+06 2.91342e+06 3.06512e+06
3.23073e+06 3.42271e+06 3.04111e+06 2.30066e+06 3.34754e+06 3.20725e+06 3.06973e+06 3.26565e+06
3.16154e+06 3.31086e+06 3.0332e+06 2.41687e+06 3.10992e+06 3.13669e+06 3.0368e+06 3.33687e+06 3.27856e+06
3.27653e+06 3.15991e+06 3.14129e+06 2.33453e+06 2.32926e+06 2.97705e+06 2.78389e+06 3.41271e+06 3.34243e+06 3.37167e+06
2.53646e+06 2.90121e+06 2.75831e+06 2.60997e+06 2.25828e+06 3.02153e+06 2.6219e+06 2.82299e+06 3.49618e+06 3.55346e+06 3.39133e+06
2
4
8
16
32
64
128
number of workers
2400000 1600000 800000
0
1.30355e+06 2.27574e+06 2.6065e+06 2.77221e+06 1.98423e+06 3.22568e+06 2.61959e+06 2.97559e+06 3.27783e+06 3.53633e+06 3.48964e+06 3.24726e+06
1
3200000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
193536
0
0
0
0
0
0
0
0
0
0
0
90624
115200
0
0
0
0
0
0
0
0
0
0
38656
67072
309248
0
0
0
0
0
0
0
0
0
23936
56576
130560
724992
0
0
0
0
0
0
0
0
17536
31104
70144
216064
485376
0
0
0
0
0
0
0
13760
17280
45312
90624
242688
716800
0
0
0
0
0
0
8640
13888
25728
62720
164864
505856
610304
0
0
0
0
0
3408
8256
18560
37632
53504
173056
428032
708608
0
0
0
0
1976
3024
11584
19840
34560
50432
152576
329728
643072
0
0
0
1096
1912
3248
12224
21632
33024
64256
109056
561152
651264
0
0
153
1464
3248
8160
12480
19584
32640
67072
154624
346112
577536
0
179
322
2040
2736
7840
9408
20864
34560
49920
158720
288768
733184
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.5
0
0
0
0
0
0
0
0
0
0
0
2.5
3.5
0
0
0
0
0
0
0
0
0
0
2.5
3.8
4.9
0
0
0
0
0
0
0
0
0
2.5
3.4
4.5
3.4
0
0
0
0
0
0
0
0
2.3
3.4
4.8
4.3
5.4
0
0
0
0
0
0
0
2.3
2.7
3.6
4.5
5.2
6.2
0
0
0
0
0
0
2.1
2.4
2.4
2.7
3.9
4.9
8.1
0
0
0
0
0
1.9
2.1
2.1
1.9
3.7
5.7
5.5
8.2
0
0
0
0
1.9
2.1
2.1
2
4
4.2
5.4
6.7
8.4
0
0
0
2.1
2
2.2
1.9
2.2
3.7
4.5
5.9
7.2
8.3
0
0
1.9
2
2.1
2.6
2.5
3
3.4
4.4
6.9
7.3
8.6
0
1.2
1.8
2.1
2.2
2
3.8
3.1
4.7
6.3
7
8.4
10.3
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000 450000 300000 150000 0
100 80 60 40 20 0
Software RAID10 over 6 Intel P3700 as blockdevice (Test ID 8) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
193672
0
0
0
0
0
0
0
0
0
0
0
193046
380463
0
0
0
0
0
0
0
0
0
0
224273
380030
711015
0
0
0
0
0
0
0
0
0
201840
380868
730188
1.03374e+06
0
0
0
0
0
0
0
0
200477
377163
606592
1.0169e+06
413084
0
0
0
0
0
0
0
221151
364518
728938
1.0285e+06
433253
380331
0
0
0
0
0
0
225496
332739
488926
1.02956e+06
427145
372681
384817
0
0
0
0
0
163740
295898
509366
917259
419884
402767
383940
383119
0
0
0
0
86975
167492
309517
543928
794405
339234
383777
383509
383825
0
0
0
43768
86944
169907
321505
561112
667923
384204
383665
383017
383805
0
0
21304
43662
86844
169509
301544
538196
642366
384562
549731
465510
648036
0
10158
21466
42953
84707
159730
309616
492739
561016
385411
388958
1.13668e+06
606383
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1000000 800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
14784
0
0
0
0
0
0
0
0
0
0
0
7712
6304
0
0
0
0
0
0
0
0
0
0
3376
3216
3824
0
0
0
0
0
0
0
0
0
2024
1640
1688
2448
0
0
0
0
0
0
0
0
1032
884
5536
1384
6496
0
0
0
0
0
0
0
430
556
490
780
3248
7008
0
0
0
0
0
0
270
358
732
466
1880
4128
30336
0
0
0
0
0
193
270
532
458
1240
2448
18560
45824
0
0
0
0
191
225
278
692
684
2064
17024
50432
129536
0
0
0
201
191
227
247
442
1576
4384
11840
31616
49408
0
0
221
195
203
199
330
474
1784
2832
2352
8768
5728
0
233
203
209
177
253
241
660
2064
2928
4704
3440
9024
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6
0
0
0
0
0
0
0
0
0
0
0
3.6
5.9
0
0
0
0
0
0
0
0
0
0
3
6.7
11.5
0
0
0
0
0
0
0
0
0
3.7
6.7
11.8
20.2
0
0
0
0
0
0
0
0
3.7
6.7
10.4
20.6
36.8
0
0
0
0
0
0
0
3
5.9
11.2
20.3
36.6
65.6
0
0
0
0
0
0
3.1
5.1
8.5
19.2
36.1
66.6
94.9
0
0
0
0
0
2.8
5.1
9.1
16.3
36.1
63.9
94.8
94.8
0
0
0
0
1.9
3.8
7
11.4
25.3
58.8
88.9
91.7
94.6
0
0
0
1.2
2.3
4.3
8.3
14.6
29
83.3
95.5
95.1
94.7
0
0
0.9
1.4
2.6
5
9.1
17.2
26.8
91.3
57.7
91.6
70.3
0
0.6
0.9
1.7
3.2
5.7
10.6
20.2
31.2
94.1
95.1
71.3
77.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
125000 100000 75000 50000 25000 0
100 80 60 40 20 0
Software RAID10 over 6 Intel P3700 as blockdevice (Test ID 8) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
1268
0
0
0
0
0
0
0
0
0
0
0
1316
1345
0
0
0
0
0
0
0
0
0
0
1287
1361
1391
0
0
0
0
0
0
0
0
0
1282
1334
1359
1404
0
0
0
0
0
0
0
0
1265
1451
1392
1387
1472
0
0
0
0
0
0
0
1266
1396
1392
1363
1444
1518
0
0
0
0
0
0
1302
1348
1368
1385
1457
1515
1552
0
0
0
0
0
1305
1328
1458
1378
1456
1568
1509
1517
0
0
0
0
1298
1300
1428
1356
1445
1544
1503
1631
1406
0
0
0
1423
1355
1437
1366
1467
1570
1588
1596
1409
1388
0
0
1317
1315
1374
1394
1442
1550
1533
1445
1371
1367
1466
0
1345
1298
1326
1316
1389
1456
1466
1402
1325
1339
1363
1296
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
0
0
0
0
0
0
0
0
0
0
0
864256
1.71213e+06
0
0
0
0
0
0
0
0
0
0
481280
880640
1.61382e+06
0
0
0
0
0
0
0
0
0
268288
485376
921600
1.67936e+06
0
0
0
0
0
0
0
0
138240
230400
477184
888832
1.81043e+06
0
0
0
0
0
0
0
78336
130560
250880
501760
937984
1.9415e+06
0
0
0
0
0
0
44288
73216
140288
276480
544768
1.30253e+06 1.89235e+06
0
0
0
0
0
24960
44288
78336
160768
329728
643072
0
0
0
0
15040
26240
47360
98816
179200
370688
626688
0
0
0
8640
14656
26496
53504
114176
242688
329728
888832
5.93101e+06 8.84736e+06
0
0
3184
6304
14272
28288
60672
146432
152576
536576
2.93274e+06 7.04512e+06 1.04202e+07
135
1736
2832
2928
7968
19840
31616
31360
29056
28800
27008
26496
1
2
4
8
16
32
64
128
256
512
1024
2048
1.51552e+06 8.58522e+06
number of workers
queue depth
0.6
0
0
0
0
0
0
0
0
0
0
0
0.6
0.6
0
0
0
0
0
0
0
0
0
0
0.6
0.6
0.7
0
0
0
0
0
0
0
0
0
0.6
0.6
0.7
0.9
0
0
0
0
0
0
0
0
0.6
0.6
0.7
0.9
1.5
0
0
0
0
0
0
0
0.6
0.6
0.7
0.9
1.4
2
0
0
0
0
0
0
0.6
0.5
0.7
0.9
1.5
1.9
2.5
0
0
0
0
0
0.6
0.6
0.7
0.9
1.5
2.1
2.5
3.4
0
0
0
0
0.6
0.5
0.7
0.9
1.5
1.9
2.4
3.6
5.2
0
0
0
0.6
0.5
0.7
1
1.6
2
2.5
3.4
5.6
9.3
0
0
0.6
0.6
0.7
0.9
1.5
2.1
2.7
3.5
5.7
9.6
21.6
0
0.8
0.6
0.7
0.9
1.5
2.1
3.1
3.9
5.9
10.1
20.8
44.8
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
900 600 300 0
1.0 0.8 0.6 0.4 0.2
0
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
1200
1e7
1.72851e+06
1.08954e+06 2.8672e+06
1500
0.0
100 80 60 40 20 0
Software RAID10 over 6 Intel P3700 as blockdevice (Test ID 8) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
173808
0
0
0
0
0
0
0
0
0
0
0
202859
340413
0
0
0
0
0
0
0
0
0
0
206946
370108
632902
0
0
0
0
0
0
0
0
0
194998
372962
697214
974523
0
0
0
0
0
0
0
0
183430
374561
719551
980055
420527
0
0
0
0
0
0
0
215610
335345
555407
950208
427205
367345
0
0
0
0
0
0
193441
295320
495408
979820
452790
372397
384148
0
0
0
0
0
139069
251886
464017
714269
435874
365558
385637
384912
0
0
0
0
72333
141300
269431
465551
695201
345138
381362
385550
384801
0
0
0
36702
72756
143205
263825
435189
572328
395912
383288
384741
384842
0
0
17651
36994
74135
138827
254508
391208
600282
382124
385514
416204
1.00686e+06
0
7992
18166
36681
71507
134540
248415
416751
568219
385055
386252
724605
402125
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1000000 800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
13632
0
0
0
0
0
0
0
0
0
0
0
5408
7264
0
0
0
0
0
0
0
0
0
0
2768
3408
13632
0
0
0
0
0
0
0
0
0
1944
1928
1720
2608
0
0
0
0
0
0
0
0
900
908
1032
1496
7328
0
0
0
0
0
0
0
402
604
2384
932
3472
7264
0
0
0
0
0
0
310
406
764
2224
1816
4192
32384
0
0
0
0
0
203
262
350
1480
1224
2576
22912
58112
0
0
0
0
189
203
227
374
764
2416
16768
26752
69120
0
0
0
181
183
185
245
676
1592
3120
9024
10560
39680
0
0
211
185
171
211
282
1064
1432
2768
4448
3120
8896
0
262
181
169
183
223
302
596
1432
2960
4192
4704
15296
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.7
0
0
0
0
0
0
0
0
0
0
0
3.2
6.7
0
0
0
0
0
0
0
0
0
0
3.2
6.1
11.3
0
0
0
0
0
0
0
0
0
3.6
6.3
11.8
20.8
0
0
0
0
0
0
0
0
3.7
6.3
11.2
20.5
36.6
0
0
0
0
0
0
0
3.2
5.2
9.7
20
36.8
65.8
0
0
0
0
0
0
3.3
5.3
9.1
17.3
35.1
65.8
87.2
0
0
0
0
0
2.5
5.4
8.7
14.1
34.1
55.9
93.8
90.6
0
0
0
0
1.8
3.3
6.5
11
19.4
50.1
87
95.4
95.1
0
0
0
1.2
2.1
3.9
7.4
12.4
17.4
77.8
92
92.3
95
0
0
0.9
1.3
2.3
4.7
8.2
13.7
25.9
88.3
95.2
94.5
66.7
0
0.7
0.8
1.4
2.8
5.6
9.6
16.9
30.7
94
94.6
76.9
95.4
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID10 over 6 Intel P3700 as blockdevice (Test ID 8) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
1290
0
0
0
0
0
0
0
0
0
0
0
1313
1314
0
0
0
0
0
0
0
0
0
0
1303
1398
1533
0
0
0
0
0
0
0
0
0
1377
1351
1422
1379
0
0
0
0
0
0
0
0
1293
1375
1449
1370
1505
0
0
0
0
0
0
0
1340
1339
1422
1378
1429
1560
0
0
0
0
0
0
1287
1369
1427
1413
1440
1519
1529
0
0
0
0
0
1311
1332
1336
1480
1464
1575
1502
1500
0
0
0
0
1421
1330
1380
1428
1451
1541
1611
1508
1394
0
0
0
1317
1351
1497
1371
1434
1546
1548
1528
1373
1407
0
0
1313
1332
1447
1378
1487
1568
1457
1438
1393
1473
1362
0
1323
1320
1368
1352
1334
1449
1391
1387
1364
1363
1385
1291
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
0
0
0
0
0
0
0
0
0
0
0
880640
1.71213e+06
0
0
0
0
0
0
0
0
0
0
460800
831488
1.51552e+06
0
0
0
0
0
0
0
0
0
230400
448512
880640
1.79405e+06
0
0
0
0
0
0
0
0
138240
250880
440320
937984
1.99066e+06
0
0
0
0
0
0
0
75264
138240
244736
489472
1.03629e+06 1.82682e+06
0
0
0
0
0
0
44288
72192
136192
268288
561152
0
0
0
0
0
24704
44288
79360
148480
317440
593920
0
0
0
0
14784
27008
45312
89600
195584
428032
602112
0
0
0
8512
14528
26240
51968
107008
261120
325632
888832
6.52083e+06 9.63379e+06
0
0
3248
6368
14528
27776
62208
148480
156672
552960
3.32595e+06 7.50387e+06 1.04202e+07
135
2192
2800
2928
7968
20096
33024
29056
28288
27776
24704
30080
1
2
4
8
16
32
64
128
256
512
1024
2048
1.10592e+06 2.99827e+06
1.69574e+06 8.45414e+06
number of workers
queue depth
0.6
0
0
0
0
0
0
0
0
0
0
0
0.6
0.6
0
0
0
0
0
0
0
0
0
0
0.6
0.6
0.8
0
0
0
0
0
0
0
0
0
0.7
0.6
0.7
1
0
0
0
0
0
0
0
0
0.6
0.6
0.7
0.9
1.6
0
0
0
0
0
0
0
0.6
0.6
0.7
1
1.5
2
0
0
0
0
0
0
0.6
0.6
0.7
0.9
1.5
2
2.6
0
0
0
0
0
0.6
0.6
0.6
1
1.5
2
2.5
3.4
0
0
0
0
0.6
0.6
0.7
0.9
1.5
2.1
2.5
3.4
5.2
0
0
0
0.6
0.6
0.7
0.9
1.5
1.9
2.6
3.4
5.3
9.4
0
0
0.6
0.6
0.7
0.9
1.5
2.1
2.6
3.5
5.7
10.4
21.5
0
0.7
0.6
0.6
1
1.7
2.2
2.8
4
6
10.3
21.7
45.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
900 600 300 0
1.0 0.8 0.6 0.4 0.2
0
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
1200
1e7
1.72851e+06
1.10592e+06 1.90874e+06
1500
0.0
100 80 60 40 20 0
Software RAID10 over 6 Intel P3700 as blockdevice (Test ID 8) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
1274
0
0
0
0
0
0
0
0
0
0
0
1295
1296
0
0
0
0
0
0
0
0
0
0
1265
1349
1311
0
0
0
0
0
0
0
0
0
1288
1422
1301
1315
0
0
0
0
0
0
0
0
1331
1334
1351
1349
1374
0
0
0
0
0
0
0
1295
1282
1327
1311
1351
1462
0
0
0
0
0
0
1288
1280
1339
1342
1390
1496
1509
0
0
0
0
0
1290
1301
1300
1415
1318
1581
1521
1501
0
0
0
0
1298
1306
1340
1370
1367
1559
1594
1458
1363
0
0
0
1252
1377
1341
1338
1331
1412
1451
1378
1349
1326
0
0
1240
1305
1311
1296
1321
1371
1332
1316
1296
1336
1280
0
1180
1233
1301
1342
1293
1282
1297
1319
1304
1327
1315
1297
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1500 1200 900 600 300 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
585728
0
0
0
0
0
0
0
0
0
0
0
296960
544768
0
0
0
0
0
0
0
0
0
0
171008
292864
593920
0
0
0
0
0
0
0
0
0
91648
156672
325632
692224
0
0
0
0
0
0
0
0
52480
98816
187392
362496
700416
0
0
0
0
0
0
0
30848
56576
111104
218112
444416
937984
0
0
0
0
0
0
19072
35072
65280
129536
257024
577536
839680
0
0
0
0
0
11968
22656
42752
78336
168960
305152
518144
1.28614e+06
0
0
0
0
6688
12736
26240
52992
107008
209920
313344
921600
5.40672e+06
0
0
0
5792
8512
17024
35584
77312
195584
209920
724992
3.62086e+06 5.47226e+06
0
0
3088
5792
9280
21120
51456
125440
136192
544768
1.54829e+06 1.95789e+06 7.30726e+06
141
1160
2544
2896
6176
17792
32640
32640
30336
28544
27008
27520
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
0.6
0
0
0
0
0
0
0
0
0
0
0
0.6
0.6
0
0
0
0
0
0
0
0
0
0
0.6
0.6
0.7
0
0
0
0
0
0
0
0
0
0.6
0.6
0.7
1
0
0
0
0
0
0
0
0
0.6
0.6
0.7
1
1.5
0
0
0
0
0
0
0
0.6
0.6
0.7
1
1.4
2.1
0
0
0
0
0
0
0.6
0.6
0.7
1
1.5
2.1
2.7
0
0
0
0
0
0.6
0.6
0.7
1
1.5
2.1
2.7
3.5
0
0
0
0
0.6
0.6
0.7
1
1.5
2.1
2.8
3.5
5.4
0
0
0
0.7
0.7
0.8
1
1.6
2.3
2.9
3.8
5.8
10.2
0
0
0.7
0.8
0.8
1
1.6
2.3
3.1
4
5.9
10.2
20.8
0
0.7
0.8
0.8
1
1.6
2.4
3.1
4.1
6
10.5
21.8
45.4
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
4500000 3000000 1500000
0
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
6000000
0
100 80 60 40 20 0
Software RAID10 over 6 Intel P3700 as blockdevice (Test ID 8) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
8.35805e+06
0
8.68292e+06 1.40464e+07
1e7
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
8.39128e+06 1.35921e+07 1.36313e+07
8.45061e+06 1.34823e+07 1.35254e+07 1.41552e+07
8.89569e+06 1.29259e+07 1.39226e+07 1.3894e+07 1.37765e+07
8.80029e+06 1.38671e+07 1.36525e+07 1.41584e+07 1.38296e+07 1.28838e+07
8.60828e+06 1.27224e+07 1.32794e+07 1.39031e+07 1.36512e+07 1.24845e+07 1.3001e+07
5.52646e+06 9.2864e+06 1.13843e+07 1.36761e+07 1.27851e+07 9.90679e+06 8.59555e+06 7.27262e+06
2.81410e+06 5.64461e+06 8.26875e+06 1.20113e+07 1.29824e+07 8.60396e+06 8.76505e+06 6.66607e+06 6.38683e+06
1.8945e+06 2.82415e+06 4.82217e+06 7.70781e+06 1.12361e+07 7.70593e+06 7.65761e+06 7.50758e+06 8.54958e+06 8.16485e+06
1.56084e+06 1.28475e+06 2.42346e+06 4.28589e+06 6.96332e+06 8.22092e+06 8.65337e+06 7.2304e+06 8.40987e+06 7.48823e+06 1.03653e+07 1.5248e+06
565644
1
2
0
1.24069e+06 2.33357e+06 3.99543e+06 5.84264e+06 6.73148e+06 6.48688e+06 6.96379e+06 7.07208e+06 9.14534e+06 9.13146e+06
4
8
16
32
64
128
number of workers
256
512
1024
2048
1.25 1.00 0.75 0.50 0.25 0.00
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
37632
0
0
0
0
0
0
0
0
0
0
0
20864
83456
0
0
0
0
0
0
0
0
0
0
11072
30336
89600
0
0
0
0
0
0
0
0
0
5984
23168
65280
78336
0
0
0
0
0
0
0
0
2768
10048
29824
79360
138240
0
0
0
0
0
0
0
1960
3792
17536
24704
97792
162816
0
0
0
0
0
0
1112
1816
5920
14912
36608
107008
569344
0
0
0
0
0
652
828
2672
5792
13888
48896
150528
643072
0
0
0
0
596
620
1304
1912
6496
39168
110080
232448
456704
0
0
0
620
724
924
1160
1800
16512
41216
121344
209920
552960
0
0
596
644
788
908
1352
4192
11072
36608
109056
234496
448512
0
195
644
668
788
948
1752
5344
19840
33024
144384
264192
544768
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.1
0
0
0
0
0
0
0
0
0
0
0
2.9
5.3
0
0
0
0
0
0
0
0
0
0
3.1
4.9
6.4
0
0
0
0
0
0
0
0
0
3.2
4.6
6.3
7.2
0
0
0
0
0
0
0
0
2.7
4.8
5.9
6.6
8.6
0
0
0
0
0
0
0
2.8
4.8
5.6
6.6
7.6
9.7
0
0
0
0
0
0
2.6
4.5
5.2
6
7.2
8.8
8.1
0
0
0
0
0
1.6
2.9
3.7
5.3
6.6
10.8
11.2
13.8
0
0
0
0
1.1
2
2.8
4.6
6.5
9.7
9.6
12.4
13.4
0
0
0
1.2
1.3
2.1
3.5
6.5
9.6
12.3
13.5
14.6
13.3
0
0
1.1
1.2
1.7
3
4.9
10.9
15.3
13.4
13.4
11.4
9
0
0.9
1.2
1.4
2.3
4.6
9.7
12
12.7
13.5
12
9.2
10.3
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000
450000
300000
150000
0
100 80 60 40 20 0
Software RAID10 over 6 Intel P3700 as blockdevice (Test ID 8) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
2.78607e+06
0
3.0082e+06 4.72132e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2.72516e+06 4.91143e+06 5.72566e+06
3.18336e+06 5.18298e+06 5.74304e+06 5.5426e+06
4.80591e+06 5.74858e+06 5.75136e+06 5.69955e+06 5.50315e+06
4.92442e+06 5.7589e+06 5.81445e+06 5.75891e+06 5.60718e+06 5.04792e+06
4.96429e+06 5.60526e+06 5.71527e+06 5.69173e+06 5.66212e+06 5.2772e+06 4.84116e+06
5.00411e+06 5.39352e+06 5.51921e+06 5.67391e+06 5.64268e+06 5.29183e+06 4.68343e+06 4.25392e+06
4.70185e+06 5.01298e+06 5.2688e+06 5.40992e+06 5.48617e+06 5.24246e+06 4.87378e+06 4.79628e+06 4.60291e+06
3.83265e+06 4.65709e+06 4.96595e+06 4.99904e+06 5.27476e+06 4.93184e+06 5.05208e+06 4.852e+06 4.78421e+06 4.81392e+06
2.54121e+06 4.41641e+06 4.73309e+06 4.70257e+06 5.07161e+06 4.73300e+06 4.99066e+06 4.901e+06 5.02832e+06 4.83076e+06 5.21266e+06
0
1.43318e+06 2.66557e+06 4.12901e+06 4.49004e+06 4.77309e+06 4.4441e+06 4.58238e+06 4.86264e+06 4.87367e+06 4.74472e+06 4.93072e+06 5.41901e+06
1
2
4
8
16
32
64
128
number of workers
256
512
1024
2048
5000000 4000000 3000000 2000000 1000000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
179200
0
0
0
0
0
0
0
0
0
0
0
76288
79360
0
0
0
0
0
0
0
0
0
0
42240
51456
154624
0
0
0
0
0
0
0
0
0
23424
30080
82432
428032
0
0
0
0
0
0
0
0
15552
20864
59136
119296
366592
0
0
0
0
0
0
0
13760
15552
22656
54016
164864
387072
0
0
0
0
0
0
11200
12992
14656
21632
38144
250880
382976
0
0
0
0
0
2352
4960
12480
14144
24448
118272
321536
403456
0
0
0
0
1192
2512
8640
12480
14400
24192
183296
477184
407552
0
0
0
398
1240
2608
8640
13632
17536
28544
173056
329728
536576
0
0
149
486
1512
2480
9024
13888
18048
32384
128512
374784
667648
0
161
155
298
1448
2896
8640
13888
17792
32128
142336
333824
593920
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.6
0
0
0
0
0
0
0
0
0
0
0
2.6
4.6
0
0
0
0
0
0
0
0
0
0
2.6
4.7
7.2
0
0
0
0
0
0
0
0
0
2.6
4.6
7.6
9.6
0
0
0
0
0
0
0
0
2.7
4.4
5.6
8.3
8.8
0
0
0
0
0
0
0
2.7
3.7
4.7
8.6
9
10.9
0
0
0
0
0
0
2.6
3.5
3.9
5.9
8
8.7
11.8
0
0
0
0
0
2.5
3.3
3.5
4.6
5.8
8.9
9.8
14
0
0
0
0
2.5
2.9
3.5
4.6
5.7
8.8
9
10.5
13.7
0
0
0
2.4
2.9
3.5
3.9
5
8.7
9.4
10.4
11.4
12.6
0
0
1.9
2.8
3.4
3.9
4.9
9.2
9.2
11.3
9.9
11.5
11.1
0
1.2
1.9
3.4
3.5
4.3
9.1
9.6
10
11.6
10.7
10.7
9.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000 450000 300000 150000 0
100 80 60 40 20 0
Software RAID10 over 8 Intel P3700 as blockdevice (Test ID 9) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
204651
0
0
0
0
0
0
0
0
0
0
0
203451
377011
0
0
0
0
0
0
0
0
0
0
210062
354602
684469
0
0
0
0
0
0
0
0
0
217721
384963
636249
1.0367e+06
0
0
0
0
0
0
0
0
212012
371865
749561
958702
429629
0
0
0
0
0
0
0
206060
331537
585285
991214
437012
381778
0
0
0
0
0
0
204366
386276
640538
954388
421140
386654
367927
0
0
0
0
0
178730
321508
536993
864475
435820
361833
344819
374359
0
0
0
0
95739
184798
365293
626195
660964
373452
384909
382604
380401
0
0
0
47276
97419
187666
345115
576476
679483
386969
382493
392865
380269
0
0
19475
48978
97063
184278
348843
544326
656488
410788
382730
416464
1.08166e+06
0
9531
23821
48373
95057
184205
330882
523952
615590
386327
386827
773737
617919
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1000000 800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
14400
0
0
0
0
0
0
0
0
0
0
0
7520
6368
0
0
0
0
0
0
0
0
0
0
3600
3472
8896
0
0
0
0
0
0
0
0
0
1384
1592
3536
9280
0
0
0
0
0
0
0
0
748
860
1368
8896
6368
0
0
0
0
0
0
0
506
540
1012
6432
3440
7456
0
0
0
0
0
0
278
258
644
1816
2192
4448
31104
0
0
0
0
0
169
235
290
2064
1544
2864
17792
45824
0
0
0
0
157
195
189
326
1144
2064
16320
11968
16512
0
0
0
173
147
179
241
580
1448
4256
8384
5536
7008
0
0
221
165
143
199
221
564
2288
3312
4896
5664
20096
0
221
181
161
167
211
270
532
2480
3504
4016
6048
10944
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.2
0
0
0
0
0
0
0
0
0
0
0
3.4
6.4
0
0
0
0
0
0
0
0
0
0
3.4
6.3
11.8
0
0
0
0
0
0
0
0
0
3.3
6.7
11.3
20.1
0
0
0
0
0
0
0
0
3.3
6.2
11.6
20.8
39.2
0
0
0
0
0
0
0
3.4
5.5
10
19.1
38.9
70.2
0
0
0
0
0
0
3.4
6.6
11.2
19.6
38.9
69.8
83.1
0
0
0
0
0
3.2
6.4
9.2
16.7
37.6
64.3
76.5
94.9
0
0
0
0
2.3
4.6
8.1
13.6
29.6
56.6
94.7
94.3
94.5
0
0
0
1.6
2.7
5.3
9.8
17.3
33.4
75
94
95
94.3
0
0
1.4
1.6
3
6.1
11.8
20.7
32.8
85.9
95.4
94.4
64.8
0
0.9
1
1.7
3.5
6.9
13
26
35.7
93.9
93.9
96.4
70.8
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
40000 32000 24000 16000 8000 0
100 80 60 40 20 0
Software RAID10 over 8 Intel P3700 as blockdevice (Test ID 9) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
2025
0
0
0
0
0
0
0
0
0
0
0
1933
2002
0
0
0
0
0
0
0
0
0
0
2015
2087
2022
0
0
0
0
0
0
0
0
0
1919
1996
2041
2090
0
0
0
0
0
0
0
0
1954
1972
2039
2068
2125
0
0
0
0
0
0
0
1933
2008
2105
2016
2151
2196
0
0
0
0
0
0
2003
2029
2063
2010
2123
2226
2209
0
0
0
0
0
1992
2018
2057
2117
2189
2131
2142
2254
0
0
0
0
1967
2079
2196
2079
2074
2165
2264
2211
2022
0
0
0
1974
1983
2045
2282
2122
2317
2246
2167
2083
2020
0
0
1990
2020
2110
2038
2061
2169
2083
2097
2196
2022
1999
0
2014
2171
2027
1988
1972
2035
2087
1997
2117
2010
2014
1940
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
2000 1600 1200 800 400 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
1.1223e+06
0
0
0
0
0
0
0
0
0
0
0
626688
1.13869e+06
0
0
0
0
0
0
0
0
0
0
325632
577536
1.17146e+06
0
0
0
0
0
0
0
0
0
177152
325632
626688
1.22061e+06
0
0
0
0
0
0
0
0
105984
177152
370688
626688
1.23699e+06
0
0
0
0
0
0
0
56064
96768
185344
374784
692224
1.38445e+06
0
0
0
0
0
0
33024
55552
108032
211968
378880
790528
1.36806e+06
0
0
0
0
0
18816
33024
60160
114176
214016
514048
757760
1.82682e+06
0
0
0
0
11968
20096
34048
68096
136192
317440
415744
1.10592e+06 7.70048e+06
0
0
0
6240
11584
20608
38656
80384
191488
226304
643072
5.34118e+06 8.58522e+06
0
0
3152
6048
11072
21120
45824
116224
110080
378880
2.70336e+06 7.76602e+06 7.50387e+06
139
1976
2544
2704
5600
14272
23680
22144
21120
20096
18048
27264
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
0
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
0.8
0
0
0
0
0
0
0
0
0
0
0
0.8
0.7
0
0
0
0
0
0
0
0
0
0
0.8
0.8
0.9
0
0
0
0
0
0
0
0
0
0.8
0.7
0.8
1.2
0
0
0
0
0
0
0
0
0.7
0.7
0.8
1.2
2
0
0
0
0
0
0
0
0.7
0.7
0.8
1.1
2
2.8
0
0
0
0
0
0
0.7
0.7
0.8
1.2
2
2.7
3.8
0
0
0
0
0
0.7
0.7
0.8
1.2
2
2.7
3.7
5.2
0
0
0
0
0.7
0.8
0.9
1.1
2
2.7
3.9
5.2
7.2
0
0
0
0.7
0.7
0.9
1.2
2
2.7
3.6
5
8
13.2
0
0
0.7
0.7
0.9
1.2
2
2.8
3.7
5.2
8.3
13.7
26.7
0
0.9
0.8
0.9
1.1
2
3
4.1
5.5
8.8
14
27.6
50.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
7500000 6000000 4500000 3000000 1500000 0
100 80 60 40 20 0
Software RAID10 over 8 Intel P3700 as blockdevice (Test ID 9) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
197392
0
0
0
0
0
0
0
0
0
0
0
182445
349001
0
0
0
0
0
0
0
0
0
0
193335
338707
628332
0
0
0
0
0
0
0
0
0
205614
379441
625548
1.01095e+06
0
0
0
0
0
0
0
0
200141
354521
704878
965724
426807
0
0
0
0
0
0
0
202551
367504
664235
947530
424100
373489
0
0
0
0
0
0
202290
348350
522972
923602
450278
368976
380282
0
0
0
0
0
145789
291275
484328
749211
420829
362103
386102
407582
0
0
0
0
79017
152269
283312
507967
779686
371399
385255
393092
381596
0
0
0
39300
78550
154999
289284
488793
715778
396814
396833
382903
392963
0
0
16512
39191
80617
152734
287179
413328
586238
514860
408336
390325
720180
0
7003
18087
40212
77641
149362
273084
434276
575603
389302
391914
416270
800827
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1000000 800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
11328
0
0
0
0
0
0
0
0
0
0
0
6624
6816
0
0
0
0
0
0
0
0
0
0
2864
4080
7136
0
0
0
0
0
0
0
0
0
1880
1544
4448
4048
0
0
0
0
0
0
0
0
996
956
2160
6880
6944
0
0
0
0
0
0
0
422
462
940
2672
3440
7584
0
0
0
0
0
0
255
314
596
5216
1992
4384
20352
0
0
0
0
0
219
217
318
1352
1464
2832
2864
49408
0
0
0
0
181
217
255
354
772
3024
16512
18816
69120
0
0
0
195
185
193
251
450
1288
3792
5152
16512
3888
0
0
249
225
169
219
239
1112
1864
2640
2864
4128
7776
0
258
270
175
187
219
290
684
1976
3472
4960
5984
12224
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.2
0
0
0
0
0
0
0
0
0
0
0
3.7
6.6
0
0
0
0
0
0
0
0
0
0
3.5
6
11.4
0
0
0
0
0
0
0
0
0
3.2
6.2
11
20.3
0
0
0
0
0
0
0
0
3.3
5.9
11.6
20.3
38.4
0
0
0
0
0
0
0
3.5
6.5
11.3
19.3
38.8
69.8
0
0
0
0
0
0
3.4
6.3
9.4
17
38.8
63.2
93.3
0
0
0
0
0
3
5.7
9.6
15.7
37.9
58
87.1
65.4
0
0
0
0
2
4
7.4
13.1
21.6
56.1
93.8
95
93.5
0
0
0
1.4
2.5
4.6
8.6
15.6
25.8
78.5
94.3
94.8
82.3
0
0
1.3
1.4
2.6
5.2
9.8
16.9
29.6
55.8
87.1
94.9
93
0
1.1
1.1
1.6
3.2
5.8
12
20.5
35.1
93.1
94.8
90.5
65.6
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID10 over 8 Intel P3700 as blockdevice (Test ID 9) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
1962
0
0
0
0
0
0
0
0
0
0
0
1959
1994
0
0
0
0
0
0
0
0
0
0
1925
1969
2249
0
0
0
0
0
0
0
0
0
1953
2008
2112
2157
0
0
0
0
0
0
0
0
1987
1905
1992
2068
2068
0
0
0
0
0
0
0
2098
1996
2200
2049
2038
2148
0
0
0
0
0
0
2004
2012
2072
2024
2127
2201
2263
0
0
0
0
0
1984
1970
1986
2071
2251
2180
2207
2307
0
0
0
0
1954
2016
2031
2049
2149
2230
2164
2225
2069
0
0
0
1974
1965
2136
2053
2157
2302
2207
2295
2051
2003
0
0
1966
2054
2184
2068
2063
2239
2162
2099
2039
2007
1989
0
2070
1968
2037
2080
1976
2076
2103
2209
2188
2004
2028
1869
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
2000 1600 1200 800 400 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
1.20422e+06
0
0
0
0
0
0
0
0
0
0
0
602112
1.15507e+06
0
0
0
0
0
0
0
0
0
0
325632
618496
1.07315e+06
0
0
0
0
0
0
0
0
0
171008
333824
610304
1.17146e+06
0
0
0
0
0
0
0
0
100864
185344
346112
675840
1.23699e+06
0
0
0
0
0
0
0
54528
97792
185344
354304
774144
1.44998e+06
0
0
0
0
0
0
33024
56576
110080
214016
382976
823296
1.30253e+06
0
0
0
0
0
20096
34048
63232
113152
216064
485376
733184
1.90874e+06
0
0
0
0
12096
20352
37632
68096
128512
309248
423936
1.08954e+06 7.70048e+06
0
0
0
6368
11840
20864
41728
78336
177152
222208
610304
5.0135e+06 8.45414e+06
0
0
3120
6048
11200
22144
44288
103936
104960
350208
2.05619e+06 4.68582e+06 7.96262e+06
137
796
2448
2800
5472
13376
24960
22656
20864
19840
18816
19584
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
0
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
0.8
0
0
0
0
0
0
0
0
0
0
0
0.7
0.8
0
0
0
0
0
0
0
0
0
0
0.7
0.7
0.9
0
0
0
0
0
0
0
0
0
0.7
0.7
0.9
1.2
0
0
0
0
0
0
0
0
0.7
0.7
0.9
1.2
2
0
0
0
0
0
0
0
0.7
0.7
0.8
1.1
1.9
2.8
0
0
0
0
0
0
0.7
0.7
0.8
1.1
2
2.8
3.8
0
0
0
0
0
0.7
0.7
0.8
1.1
2
2.8
3.8
5.4
0
0
0
0
0.8
0.7
0.9
1.1
2
2.7
3.8
5.1
7.3
0
0
0
0.7
0.7
0.9
1.2
2
2.8
3.8
5.1
7.9
13.3
0
0
0.7
0.7
0.9
1.1
1.9
2.9
4
4.9
8.3
13.4
26.8
0
0.8
0.8
0.9
1.1
2
3
4.3
5.6
9.3
13.9
27.8
50.2
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
7500000 6000000 4500000 3000000 1500000 0
100 80 60 40 20 0
Software RAID10 over 8 Intel P3700 as blockdevice (Test ID 9) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
1981
0
0
0
0
0
0
0
0
0
0
0
1933
2111
0
0
0
0
0
0
0
0
0
0
2000
1967
1970
0
0
0
0
0
0
0
0
0
1968
1988
1944
2033
0
0
0
0
0
0
0
0
1970
2007
1971
2196
2008
0
0
0
0
0
0
0
2086
1990
1968
1987
2028
2192
0
0
0
0
0
0
2008
1985
2076
2008
2065
2133
2330
0
0
0
0
0
1933
1964
1981
1999
2163
2161
2166
2165
0
0
0
0
1953
1983
1986
1994
2155
2165
2126
2088
2058
0
0
0
1904
1983
2032
1965
2010
2102
2154
2075
2062
1974
0
0
1764
1941
1941
1993
2024
2036
2014
2017
1969
1981
1978
0
1395
1890
2042
2008
1943
1990
1999
2249
1987
2011
1997
1925
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
2000 1600 1200 800 400 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
395264
0
0
0
0
0
0
0
0
0
0
0
218112
370688
0
0
0
0
0
0
0
0
0
0
127488
209920
411648
0
0
0
0
0
0
0
0
0
68096
119296
228352
444416
0
0
0
0
0
0
0
0
38144
70144
134144
236544
522240
0
0
0
0
0
0
0
23424
42240
85504
154624
301056
536576
0
0
0
0
0
0
15040
25728
45312
98816
179200
358400
536576
0
0
0
0
0
9408
17280
31360
59136
115200
230400
374784
847872
0
0
0
0
6112
9536
19840
39680
74240
158720
234496
651264
4.04685e+06
0
0
0
3248
6112
12352
27264
54528
132096
148480
514048
2.67059e+06 6.52083e+06
0
0
2960
3280
8512
15424
33536
81408
93696
378880
1.20422e+06 2.8672e+06 5.7344e+06
153
167
2128
2512
3536
13120
23936
23168
23168
21376
18304
20608
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
0.8
0
0
0
0
0
0
0
0
0
0
0
0.8
0.8
0
0
0
0
0
0
0
0
0
0
0.7
0.8
1
0
0
0
0
0
0
0
0
0
0.8
0.8
0.9
1.2
0
0
0
0
0
0
0
0
0.8
0.8
0.9
1.3
2.1
0
0
0
0
0
0
0
0.7
0.8
0.9
1.2
2
2.9
0
0
0
0
0
0
0.7
0.7
0.9
1.2
2
2.9
4.1
0
0
0
0
0
0.7
0.9
0.9
1.3
2
2.9
3.9
5
0
0
0
0
0.8
0.8
1
1.2
2.1
3
4
4.9
7.6
0
0
0
0.9
0.9
1
1.3
2.1
3.1
4.2
5.4
8.4
13.6
0
0
1
0.9
1.1
1.3
2.2
3.3
4.3
5.8
8.2
14.1
27.5
0
1.2
1
1.1
1.4
2
3.3
4.6
6.3
8.7
14.4
27.7
52.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
4500000
3000000
1500000
0
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
6000000
0
100 80 60 40 20 0
Software RAID10 over 8 Intel P3700 as blockdevice (Test ID 9) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
7.95741e+06
0
8.03404e+06 1.42655e+07
1e7
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
7.8686e+06 1.50128e+07 1.59755e+07
6.23515e+06 1.52286e+07 1.56438e+07 1.56388e+07
8.7033e+06 1.39059e+07 1.61209e+07 1.6426e+07 1.78593e+07
7.66225e+06 1.46502e+07 1.62695e+07 1.72692e+07 1.82998e+07 1.52569e+07
8.00367e+06 1.39433e+07 1.54791e+07 1.79349e+07 1.85981e+07 1.49299e+07 1.57666e+07
5.65288e+06 9.82158e+06 1.32616e+07 1.75389e+07 1.76492e+07 1.23096e+07 1.20608e+07 1.10387e+07
2.75332e+06 6.30159e+06 9.36175e+06 1.31313e+07 1.65624e+07 1.02676e+07 9.96573e+06 1.02723e+07 8.72039e+06
2.17992e+06 2.5566e+06 4.96918e+06 7.94734e+06 1.19585e+07 1.15465e+07 1.00619e+07 9.65216e+06 8.31824e+06 8.59603e+06
1.33468e+06 1.07028e+06 2.32929e+06 4.48863e+06 6.98175e+06 1.03426e+07 9.17853e+06 9.49851e+06 8.51035e+06 8.57341e+06 1.23215e+07 1.42578e+06
544601
1
2
8
16
32
64
128
number of workers
256
512
1024
2048
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
51456
0
0
0
0
0
0
0
0
0
0
0
25472
31616
0
0
0
0
0
0
0
0
0
0
13632
13632
189440
0
0
0
0
0
0
0
0
0
6432
7840
48896
128512
0
0
0
0
0
0
0
0
2992
3568
23424
49408
97792
0
0
0
0
0
0
0
1624
2160
14016
26240
45824
148480
0
0
0
0
0
0
876
1192
4080
12480
20096
75264
224256
0
0
0
0
0
596
716
1784
3120
13120
37120
162816
444416
0
0
0
0
612
748
860
1688
4320
23680
148480
228352
342016
0
0
0
564
708
716
1032
1960
7968
24448
132096
181248
370688
0
0
548
676
668
828
1288
2320
16512
28544
92672
230400
501760
0
187
644
620
708
964
1208
3440
12224
36608
96768
228352
493568
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.3
0
0
0
0
0
0
0
0
0
0
0
3.3
5.5
0
0
0
0
0
0
0
0
0
0
3.3
5.4
7.5
0
0
0
0
0
0
0
0
0
3
5.3
7.3
8.8
0
0
0
0
0
0
0
0
3
5.4
7.4
9.2
12.7
0
0
0
0
0
0
0
3.1
5.1
6.8
9.1
10.8
13.8
0
0
0
0
0
0
3.2
4.9
6.7
8.6
10.4
14.2
14.1
0
0
0
0
0
2
3.2
4.4
7.1
9.1
12.7
14.2
16.3
0
0
0
0
1.9
2.5
3.6
4.9
8.1
13.3
13.8
17.3
18.2
0
0
0
1.7
1.8
2.5
4.2
6.7
12.2
16.3
15.1
19
18
0
0
1.6
1.7
2.2
3.2
6.5
11.9
16.1
16.7
18.8
18.2
16
0
1.1
1.2
1.6
2.8
5.5
11.4
16.1
17.5
17.9
18.6
16.8
13.8
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1.2 0.8 0.4
0
1.2059e+06 2.2529e+06 3.94941e+06 6.59557e+06 8.37842e+06 9.00501e+06 8.577e+06 7.94709e+06 1.05165e+07 1.21488e+07
4
1.6
0.0
500000 400000 300000 200000 100000 0
100 80 60 40 20 0
Software RAID10 over 8 Intel P3700 as blockdevice (Test ID 9) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
2.35397e+06
0
1.95807e+06 4.86989e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1.93216e+06 4.86953e+06 7.38487e+06
3.12233e+06 6.06328e+06 7.43625e+06 7.32870e+06
4.74566e+06 7.39832e+06 7.50304e+06 7.67008e+06 7.20926e+06
4.86132e+06 7.44655e+06 7.66079e+06 7.71436e+06 7.66369e+06 6.79904e+06
5.03562e+06 7.11962e+06 7.52429e+06 7.69905e+06 7.70228e+06 6.94957e+06 6.52036e+06
5.46687e+06 6.86353e+06 7.15726e+06 7.58673e+06 7.75778e+06 6.98804e+06 6.75495e+06 6.08884e+06
5.47857e+06 6.64424e+06 6.85031e+06 7.19463e+06 7.59238e+06 6.64332e+06 6.35503e+06 6.25237e+06 5.97166e+06
4.41662e+06 6.15112e+06 6.40361e+06 6.60972e+06 7.23748e+06 6.69201e+06 6.64176e+06 6.18879e+06 6.17767e+06 6.28855e+06
2.04036e+06 4.92398e+06 5.87717e+06 6.04727e+06 6.70252e+06 6.1162e+06 6.49678e+06 6.24129e+06 6.24823e+06 6.49212e+06 6.36513e+06
2
4
8
16
32
64
128
number of workers
6000000 4500000 3000000 1500000
0
1.14121e+06 2.42436e+06 4.31136e+06 5.6038e+06 6.23926e+06 5.87851e+06 6.2076e+06 6.25373e+06 6.17597e+06 6.27145e+06 6.45381e+06 6.26006e+06
1
7500000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
177152
0
0
0
0
0
0
0
0
0
0
0
88576
77312
0
0
0
0
0
0
0
0
0
0
49920
41728
129536
0
0
0
0
0
0
0
0
0
22656
21888
51968
238592
0
0
0
0
0
0
0
0
16512
17536
47872
105984
288768
0
0
0
0
0
0
0
13760
14144
21120
45824
173056
602112
0
0
0
0
0
0
2672
12864
14144
24448
65280
366592
864256
0
0
0
0
0
1192
7264
12864
14656
18560
111104
333824
391168
0
0
0
0
450
2224
5408
12736
13760
26496
175104
536576
888832
0
0
0
227
924
2160
7136
12608
16768
19584
203776
423936
561152
0
0
255
175
1048
2096
8384
12864
15680
21632
173056
387072
522240
0
197
179
189
964
2544
7456
14400
15680
31872
113152
321536
995328
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.7
0
0
0
0
0
0
0
0
0
0
0
2.7
4.7
0
0
0
0
0
0
0
0
0
0
2.7
4.8
8.7
0
0
0
0
0
0
0
0
0
2.7
5.1
8.5
10.3
0
0
0
0
0
0
0
0
2.9
4.6
6.9
12.4
11.2
0
0
0
0
0
0
0
2.9
4.6
6.3
9.9
11.7
12.4
0
0
0
0
0
0
2.8
4.6
5.4
7.2
8.7
10.9
14
0
0
0
0
0
2.8
3.7
4.8
5.7
7.6
10.4
12.8
17.8
0
0
0
0
2.7
3.7
4.6
5.9
6
12.1
12.7
13.6
16.1
0
0
0
2.6
3.5
4.6
5.2
5.5
10.9
13.5
14.3
14.9
16.6
0
0
2
3.3
4.1
5.1
5.3
12.6
13.4
15.8
14.9
14.9
16.1
0
1.4
2
3.5
5
4.7
13
12.6
14.9
15.4
15.6
15.4
16.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
800000 600000 400000 200000 0
100 80 60 40 20 0
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
256029
0
0
0
0
0
0
0
0
0
0
0
346422
519263
0
0
0
0
0
0
0
0
0
0
267755
624826
786552
0
0
0
0
0
0
0
0
0
267973
581384
775120
791217
0
0
0
0
0
0
0
0
268822
599387
708861
788781
739068
0
0
0
0
0
0
0
365027
584018
795733
799381
800688
766722
0
0
0
0
0
0
354846
593888
799985
794830
787551
779526
746461
0
0
0
0
0
356767
554131
795268
797281
800113
787587
770786
786748
0
0
0
0
359882
587587
772753
800599
797998
770760
776497
795593
768156
0
0
0
347958
560328
731077
788354
799963
721749
770139
799044
800623
797635
0
0
171948
307324
485195
516459
774668
759083
766156
803035
805624
805636
806616
0
94322
180885
282609
374258
511783
499097
647519
800271
807286
806983
801922
889158
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
750000 600000 450000 300000 150000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
11456
0
0
0
0
0
0
0
0
0
0
0
4704
5600
0
0
0
0
0
0
0
0
0
0
2864
2128
13632
0
0
0
0
0
0
0
0
0
1416
1112
7264
7328
0
0
0
0
0
0
0
0
708
612
5984
2448
24448
0
0
0
0
0
0
0
221
318
604
1288
5344
11456
0
0
0
0
0
0
159
153
1004
892
2544
7776
28544
0
0
0
0
0
81
122
153
294
676
3120
20608
16320
0
0
0
0
43
59
163
143
652
804
6560
9664
51456
0
0
0
21
29
41
85
231
510
1144
7008
8160
59648
0
0
25
25
34
116
120
302
434
692
2896
10048
12736
0
21
23
28
44
102
209
219
482
644
2992
7264
48896
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.5
0
0
0
0
0
0
0
0
0
0
0
2.4
5.7
0
0
0
0
0
0
0
0
0
0
3.4
4.8
8.4
0
0
0
0
0
0
0
0
0
3.4
5.7
8.1
11
0
0
0
0
0
0
0
0
3.4
5.1
8.4
11.3
11.4
0
0
0
0
0
0
0
2.4
5.3
7.8
10.9
12.5
13.9
0
0
0
0
0
0
2.4
5.7
7.8
11.3
13.6
13.2
18.6
0
0
0
0
0
2.4
5.5
7.5
9.7
14.3
16.5
18.4
20.2
0
0
0
0
2.4
5
7.8
9.9
12.7
11.3
17.6
21.4
22.8
0
0
0
2.4
4.2
6.8
9.9
13.6
10.7
20.4
28.2
32.4
32.1
0
0
2.2
3.4
6
8.3
14.4
15.9
18.8
33.3
36.7
37.7
41.2
0
1.8
2.7
4.5
8.4
16.3
18.5
37.5
33.6
38.2
41.4
39.5
19.9
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
50000 40000 30000 20000 10000 0
100 80 60 40 20 0
Single Intel P3700 as blockdevice (Test ID 1) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
280405
0
0
0
0
0
0
0
0
0
0
0
254270
426264
0
0
0
0
0
0
0
0
0
0
310380
373284
449100
0
0
0
0
0
0
0
0
0
315246
436663
435060
427019
0
0
0
0
0
0
0
0
297164
418849
430720
421095
422755
0
0
0
0
0
0
0
287777
427630
432562
428731
434060
442203
0
0
0
0
0
0
295324
424433
429617
445369
444978
435725
427592
0
0
0
0
0
321287
432201
430184
434773
428865
428781
435344
448747
0
0
0
0
309930
402558
433617
424669
427556
427683
430821
434163
432438
0
0
0
241990
355571
412591
426282
431078
434550
434443
423902
430347
438040
0
0
149836
251556
349617
346156
421824
424742
419579
421204
421800
427838
436784
0
70237
147903
233342
316906
382069
405727
419522
439682
422128
430309
427937
423927
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
400000 320000 240000 160000 80000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
15424
0
0
0
0
0
0
0
0
0
0
0
7584
18304
0
0
0
0
0
0
0
0
0
0
3376
4448
16512
0
0
0
0
0
0
0
0
0
908
2448
4448
11712
0
0
0
0
0
0
0
0
732
1512
2512
5856
19328
0
0
0
0
0
0
0
422
900
1608
2864
10688
27008
0
0
0
0
0
0
205
494
980
1560
4192
15552
42240
0
0
0
0
0
79
237
532
924
1704
6752
22400
46848
0
0
0
0
45
124
274
604
1020
2008
8384
23936
52992
0
0
0
38
55
114
330
532
1080
2128
10048
26496
51456
0
0
27
33
54
197
306
652
1432
1912
8640
25472
23424
0
38
28
45
50
133
540
1128
1048
2192
7584
20864
56576
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.8
0
0
0
0
0
0
0
0
0
0
0
2.9
5.3
0
0
0
0
0
0
0
0
0
0
2.4
4.4
5.1
0
0
0
0
0
0
0
0
0
2.4
3.8
4.7
6.7
0
0
0
0
0
0
0
0
2.5
3.6
4.7
6.3
7.5
0
0
0
0
0
0
0
2.7
3.9
4.7
6.8
8.1
9.4
0
0
0
0
0
0
2.6
3.7
4.7
7.2
8
9.5
10.4
0
0
0
0
0
2.4
3.6
4.5
6.3
8.5
9.5
10.5
14.1
0
0
0
0
2.3
3.7
4.7
6.4
8.3
10
11.4
14.2
14.5
0
0
0
2.3
3.5
4.7
6.3
8.6
10.6
13.2
14.1
15.2
15.3
0
0
1.9
3.1
4.7
6.3
9.4
10.5
13.7
15.7
16.3
16.6
17.7
0
1.6
2.4
4.1
7.1
11.4
12.9
16.4
17.7
17.2
17.7
18
18.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
50000 40000 30000 20000 10000 0
100 80 60 40 20 0
Single Intel P3700 as blockdevice (Test ID 1) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
298272
0
0
0
0
0
0
0
0
0
0
0
305167
348254
0
0
0
0
0
0
0
0
0
0
239800
344149
345364
0
0
0
0
0
0
0
0
0
342742
346467
347931
344192
0
0
0
0
0
0
0
0
261193
348518
347791
344159
345424
0
0
0
0
0
0
0
290005
346993
348679
344676
347490
344904
0
0
0
0
0
0
271925
337964
347711
346372
346343
346145
344444
0
0
0
0
0
174437
279643
337824
346973
347771
346697
348283
347580
0
0
0
0
97354
174734
278462
336493
347490
347802
348126
346901
345660
0
0
0
51104
97330
173963
276200
336517
347296
348524
326566
348031
347333
0
0
26390
51239
95803
171111
268920
323133
341992
328781
340071
329509
346029
0
13290
26217
50882
95001
169969
262296
315873
339995
344525
322213
327793
337472
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
300000 240000 180000 120000 60000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
10432
0
0
0
0
0
0
0
0
0
0
0
5344
9408
0
0
0
0
0
0
0
0
0
0
3312
4016
8768
0
0
0
0
0
0
0
0
0
1064
2064
5344
10816
0
0
0
0
0
0
0
0
780
1256
2640
5664
20608
0
0
0
0
0
0
0
406
740
1480
2960
12480
34560
0
0
0
0
0
0
270
430
740
1560
4704
20608
47360
0
0
0
0
0
195
262
430
732
1720
8096
30592
51456
0
0
0
0
163
193
262
434
732
2128
10176
30336
54016
0
0
0
149
163
195
262
426
748
1848
11584
39680
39168
0
0
135
149
165
197
290
716
1128
1800
10048
27520
74240
0
126
135
149
165
199
298
996
1096
1672
9536
28032
64256
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.4
0
0
0
0
0
0
0
0
0
0
0
2.4
5.8
0
0
0
0
0
0
0
0
0
0
3.2
4.1
4.5
0
0
0
0
0
0
0
0
0
2.4
3.5
4.1
5.7
0
0
0
0
0
0
0
0
3.3
3.3
4.2
5.6
6.7
0
0
0
0
0
0
0
2.9
3.2
4.3
5.5
6.6
8
0
0
0
0
0
0
2.2
3.2
4.1
5.6
7
7.8
8.7
0
0
0
0
0
1.8
2.9
4.2
5.6
7.5
8.1
8.1
8.7
0
0
0
0
1.4
2.3
3.7
5.3
7
8.4
8.9
9.5
9.7
0
0
0
1
1.5
2.8
4.7
7.1
8.5
10.8
11.4
10.1
11
0
0
0.7
1
1.8
3.3
6
9
11.8
12.5
12.5
13
12.4
0
0.5
0.7
1.1
2
3.7
7.3
11.5
13.3
13.5
13.5
14.1
14.2
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
100 80 60 40 20 0
Single Intel P3700 as blockdevice (Test ID 1) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
221621
0
0
0
0
0
0
0
0
0
0
0
239466
241976
0
0
0
0
0
0
0
0
0
0
229135
240992
241929
0
0
0
0
0
0
0
0
0
230124
237790
244619
247507
0
0
0
0
0
0
0
0
240424
244616
243282
242684
238128
0
0
0
0
0
0
0
239926
242380
243103
243913
238227
240504
0
0
0
0
0
0
234628
237013
235471
241346
241128
238701
241588
0
0
0
0
0
229833
239565
239179
239460
244298
238450
241593
237798
0
0
0
0
225836
239122
235217
245294
238742
239051
238785
244371
239983
0
0
0
190617
227467
241104
241531
242743
236788
243594
235770
243050
242005
0
0
122292
191308
226306
241992
238757
240778
242233
246499
244357
240733
238479
0
69130
115599
187276
220850
241462
235602
239768
242331
232173
242262
242432
238746
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
200000 150000 100000 50000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
20864
0
0
0
0
0
0
0
0
0
0
0
15296
20864
0
0
0
0
0
0
0
0
0
0
4640
14272
21120
0
0
0
0
0
0
0
0
0
3024
5088
9408
20864
0
0
0
0
0
0
0
0
1800
3248
5408
10176
35072
0
0
0
0
0
0
0
1128
2064
3152
5792
18304
47360
0
0
0
0
0
0
580
1480
2096
3376
7648
27264
62208
0
0
0
0
0
426
908
1416
2064
3440
11456
46336
67072
0
0
0
0
227
498
916
1272
2096
3984
15040
53504
99840
0
0
0
43
262
454
820
1384
2128
3952
16192
66048
79360
0
0
31
55
255
434
1004
1400
2288
3664
20608
49408
37632
0
27
39
44
266
426
1048
1432
2448
3792
17280
17024
42752
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.2
0
0
0
0
0
0
0
0
0
0
0
2.9
4.7
0
0
0
0
0
0
0
0
0
0
2.5
2.9
3.9
0
0
0
0
0
0
0
0
0
2.3
2.8
3.7
5.2
0
0
0
0
0
0
0
0
2.1
2.9
3.5
5.1
5.8
0
0
0
0
0
0
0
2.2
2.8
3.7
5.3
5.7
7
0
0
0
0
0
0
2.3
2.7
3.5
4.7
5.9
6.7
7.1
0
0
0
0
0
2.1
2.8
3.3
4.6
6
6.6
6.7
8.7
0
0
0
0
2.1
2.8
3.4
4.5
5.7
6.9
7
7.5
8.4
0
0
0
2.1
2.8
3.4
4.5
5.9
6.7
9.6
9.6
10.3
9.3
0
0
1.8
2.7
3.5
4.8
6
7
9.9
11.1
11.2
11.4
12.7
0
1.3
2.1
3.5
4.8
5.7
7.7
10
11.3
12
11.7
12.4
15.5
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
80000 60000 40000 20000 0
100 80 60 40 20 0
Single Intel P3700 as blockdevice (Test ID 1) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
72585
0
0
0
0
0
0
0
0
0
0
0
63511
71809
0
0
0
0
0
0
0
0
0
0
70294
73494
72233
0
0
0
0
0
0
0
0
0
67787
71980
74277
66752
0
0
0
0
0
0
0
0
71780
71827
74825
67256
61688
0
0
0
0
0
0
0
61603
71507
72105
70623
61622
61083
0
0
0
0
0
0
44916
63290
70453
73181
64722
61733
68071
0
0
0
0
0
29833
44973
62780
68361
68126
60994
62137
61770
0
0
0
0
18900
30029
45144
60851
60277
61014
60849
61322
61238
0
0
0
11365
18948
29876
43827
60278
60348
61259
61589
61551
61077
0
0
6554
11359
18772
26777
44666
54571
60575
61393
60888
60970
62040
0
3521
6573
11375
17973
27582
38854
52691
64109
61018
60770
60687
62079
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
20096
0
0
0
0
0
0
0
0
0
0
0
8768
17536
0
0
0
0
0
0
0
0
0
0
5216
8256
15680
0
0
0
0
0
0
0
0
0
3856
6560
9920
24960
0
0
0
0
0
0
0
0
2128
4448
5792
13376
38144
0
0
0
0
0
0
0
474
2352
4704
6368
18816
56576
0
0
0
0
0
0
217
410
2416
4128
8768
36096
83456
0
0
0
0
0
38
213
442
2960
4768
12864
22144
81408
0
0
0
0
32
60
199
470
3952
5728
14400
62720
89600
0
0
0
28
36
74
213
478
2672
5344
17024
38144
117248
0
0
26
36
63
64
187
604
2544
5408
14912
50432
30336
0
43
26
29
62
83
239
676
2352
5344
19584
22400
41728
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6
0
0
0
0
0
0
0
0
0
0
0
3.2
5
0
0
0
0
0
0
0
0
0
0
2.4
3
3.5
0
0
0
0
0
0
0
0
0
2.6
2.9
3.8
4.1
0
0
0
0
0
0
0
0
2.2
2.8
3.5
3.9
4.7
0
0
0
0
0
0
0
2
2.8
3.5
4.2
4.6
5.4
0
0
0
0
0
0
1.8
2.5
3.4
4.3
5
5.3
6.3
0
0
0
0
0
1.4
2.1
3
4
5.1
5.2
6.1
6.3
0
0
0
0
1.1
1.7
2.5
3.7
4.7
5.8
5.9
7.4
9.8
0
0
0
0.8
1.2
1.7
2.9
4.5
6
7.1
8.2
9.3
9.6
0
0
0.7
0.9
1.3
2
3.6
5.4
9
10
10.6
11.2
8.6
0
0.6
0.6
1
1.4
2.4
4.4
7.5
10.4
10.3
10.7
9.5
7.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
100000 80000 60000 40000 20000 0
100 80 60 40 20 0
Single Intel P3700 as blockdevice (Test ID 1) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
2.88744e+06
0
2.88465e+06 2.86110e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2.88418e+06 2.86078e+06 2.79512e+06
2.89414e+06 2.87225e+06 2.79008e+06 2.7075e+06
2.89652e+06 2.85811e+06 2.79795e+06 2.69699e+06 2.44837e+06
2.87764e+06 2.86558e+06 2.79296e+06 2.68713e+06 2.44826e+06 2.03229e+06
2.87652e+06 2.86971e+06 2.77446e+06 2.66132e+06 2.4454e+06 2.0537e+06 2.02805e+06
2.71224e+06 2.87046e+06 2.77307e+06 2.63865e+06 2.43594e+06 2.06304e+06 2.07765e+06 2.02471e+06
2.16223e+06 2.7217e+06 2.77726e+06 2.6537e+06 2.4162e+06 2.08724e+06 2.11126e+06 2.01972e+06 1.81982e+06
1.48001e+06 2.17954e+06 2.64081e+06 2.64448e+06 2.4271e+06 2.18585e+06 1.78915e+06 1.78756e+06 1.85821e+06 1.73073e+06 886080
1.3702e+06 2.08118e+06 2.47858e+06 2.29258e+06 2.17275e+06 2.03062e+06 2.04848e+06 1.99019e+06 1.76631e+06 1.74258e+06
510860
748505
1
2
0
1.20069e+06 1.76786e+06 2.24293e+06 2.36908e+06 2.31238e+06 2.05856e+06 1.94067e+06 1.70225e+06 1.76401e+06 1.75475e+06
4
8
16
32
64
128
number of workers
256
512
1024
2048
2500000 2000000 1500000 1000000 500000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
222208
0
0
0
0
0
0
0
0
0
0
0
116224
183296
0
0
0
0
0
0
0
0
0
0
51968
95744
261120
0
0
0
0
0
0
0
0
0
18816
46848
122368
238592
0
0
0
0
0
0
0
0
10816
23168
54528
116224
317440
0
0
0
0
0
0
0
6112
12480
26240
65280
150528
643072
0
0
0
0
0
0
3632
6624
13504
32640
63744
342016
522240
0
0
0
0
0
2832
3664
7072
14400
31360
112128
261120
468992
0
0
0
0
2736
2672
3760
7264
13632
38656
127488
211968
684032
0
0
0
2608
2352
2640
4048
6944
15936
46336
130560
317440
358400
0
0
2672
2192
2384
2736
3856
8256
17280
44288
136192
288768
317440
0
2512
2192
2512
2672
2192
4512
8768
20352
68096
150528
211968
444416
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
1.9
0
0
0
0
0
0
0
0
0
0
0
1.4
1.7
0
0
0
0
0
0
0
0
0
0
1.1
1.1
1.3
0
0
0
0
0
0
0
0
0
1.1
1.2
1.2
1.4
0
0
0
0
0
0
0
0
1.1
1.1
1.2
1.4
1.5
0
0
0
0
0
0
0
1.1
1
1.1
1.4
1.6
2.6
0
0
0
0
0
0
0.9
0.9
1
1.3
1.4
2.2
1.7
0
0
0
0
0
0.8
0.8
0.9
1.1
1.4
1.7
1.7
1.9
0
0
0
0
0.7
0.8
0.9
1.1
1.3
2.1
1.7
2
2.2
0
0
0
0.6
0.7
0.8
1.1
1.3
1.7
2.4
2.3
2.6
2.7
0
0
0.6
0.7
0.8
1.1
1.6
2
1.9
2
1.9
2.8
3.1
0
0.5
0.7
0.8
1
1.4
1.3
1.4
1.8
2.2
2.9
3.2
3
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000 450000 300000 150000 0
100 80 60 40 20 0
Single Intel P3700 as blockdevice (Test ID 1) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
1.80158e+06
0
1.91423e+06 1.94956e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1.93276e+06 1.96464e+06 1.92568e+06
1.95161e+06 1.94358e+06 1.96863e+06 1.93779e+06
1.99043e+06 1.96502e+06 1.98125e+06 1.87797e+06 1.79551e+06
1.9239e+06 1.98631e+06 1.76228e+06 1.90543e+06 1.86659e+06 1.89441e+06
1.96485e+06 1.79906e+06 1.80192e+06 1.92581e+06 1.77363e+06 1.80378e+06 1.80447e+06
1.98059e+06 1.76624e+06 1.83605e+06 1.92872e+06 1.62272e+06 1.87662e+06 1.86374e+06 1.89361e+06
1.98313e+06 1.80142e+06 1.89265e+06 1.96099e+06 1.6416e+06 1.62252e+06 1.79764e+06 1.83858e+06 1.67419e+06
1.96796e+06 1.9246e+06 1.90645e+06 1.8916e+06 1.62461e+06 1.63281e+06 1.93308e+06 1.88647e+06 1.67658e+06 1.69118e+06
1.95841e+06 1.75507e+06 1.79585e+06 1.83123e+06 1.58691e+06 1.64312e+06 1.63578e+06 1.89421e+06 1.75235e+06 1.72336e+06 1.70013e+06
2
4
8
16
32
64
128
number of workers
1200000 800000 400000
0
1.7783e+06 1.6971e+06 1.88297e+06 1.80575e+06 1.89764e+06 1.83525e+06 1.67181e+06 1.82459e+06 1.81684e+06 1.70574e+06 1.71034e+06 1.70672e+06
1
1600000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
354304
0
0
0
0
0
0
0
0
0
0
0
121344
419840
0
0
0
0
0
0
0
0
0
0
55552
128512
276480
0
0
0
0
0
0
0
0
0
32640
62208
142336
325632
0
0
0
0
0
0
0
0
20608
33024
76288
138240
452608
0
0
0
0
0
0
0
11712
23168
44800
68096
173056
749568
0
0
0
0
0
0
6240
12608
23680
38144
97792
325632
798720
0
0
0
0
0
4080
7520
11840
19840
48896
110080
317440
585728
0
0
0
0
2024
3792
6432
10944
21120
48896
119296
313344
1.30253e+06
0
0
0
1416
2288
3856
6624
12352
29056
39168
115200
552960
716800
0
0
708
1816
2096
4048
7520
13888
27776
39168
185344
415744
497664
0
245
1480
1192
2352
3856
6624
13504
24448
57600
292864
201728
1.1223e+06
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2
0
0
0
0
0
0
0
0
0
0
0
2.1
2.6
0
0
0
0
0
0
0
0
0
0
2.2
2.5
2.5
0
0
0
0
0
0
0
0
0
1.7
2.4
2.7
2.9
0
0
0
0
0
0
0
0
1.1
1.9
2.6
2.4
3
0
0
0
0
0
0
0
1
1.1
1.4
2.2
3
3.1
0
0
0
0
0
0
1
1
1.1
1.8
2.6
2.4
3.1
0
0
0
0
0
0.9
0.8
1.1
1.4
1.9
2.8
2.7
3.7
0
0
0
0
0.9
0.9
1
1.2
1.4
2
2.5
3
4.2
0
0
0
0.9
0.9
1
1.1
1.3
1.3
2.3
3.2
3.5
4.5
0
0
0.9
0.9
1
1.1
1.2
1.2
1.2
2.5
2.8
3.4
4.2
0
0.9
0.8
1.1
1.2
1.3
1.6
1.2
1.5
3
3.2
4.2
5.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1250000 1000000 750000 500000 250000 0
100 80 60 40 20 0
Software RAID0 over 2 Intel P3700 as blockdevice (Test ID 2) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
307397
0
0
0
0
0
0
0
0
0
0
0
311454
503716
0
0
0
0
0
0
0
0
0
0
266221
544151
966878
0
0
0
0
0
0
0
0
0
328308
523688
734552
997535
0
0
0
0
0
0
0
0
259234
535056
827291
1.00290e+06
989386
0
0
0
0
0
0
0
328639
535940
937745
1.0025e+06
990349
965226
0
0
0
0
0
0
325982
527752
981115
1.00046e+06 1.00202e+06
978152
997395
0
0
0
0
0
284939
485305
790088
975725
1.00128e+06
974202
1.00084e+06 1.00012e+06
0
0
0
0
167628
307142
494102
805836
987589
961569
1.00086e+06 1.00109e+06 1.0017e+06
0
0
0
86342
167914
297679
513018
808522
691755
0
0
41528
87006
162540
294023
477645
581477
777399
997428
992985
0
18352
42806
83410
157897
281924
437690
486324
891782
999859
930063
1
2
4
8
16
32
64
128
256
512
1024
2048
983578
1.00118e+06 1.00234e+06 1.00057e+06
number of workers
1.00216e+06 1.00018e+06
1.9822e+06 1.00143e+06
1600000 1200000 800000 400000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
7264
0
0
0
0
0
0
0
0
0
0
0
3472
4768
0
0
0
0
0
0
0
0
0
0
2384
2160
4512
0
0
0
0
0
0
0
0
0
908
1240
52480
8896
0
0
0
0
0
0
0
0
660
684
3632
4768
14912
0
0
0
0
0
0
0
310
386
852
2192
9024
16512
0
0
0
0
0
0
217
258
498
1384
3248
12992
35584
0
0
0
0
0
163
205
290
604
916
3792
8512
21888
0
0
0
0
137
159
326
306
478
1560
2608
12608
35584
0
0
0
133
135
181
209
278
1032
1256
4576
17280
36096
0
0
157
124
145
163
302
940
908
812
4448
11712
33536
0
171
116
133
147
179
418
1272
548
828
4512
4768
15424
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.4
0
0
0
0
0
0
0
0
0
0
0
2.4
5.7
0
0
0
0
0
0
0
0
0
0
3.4
5.8
10
0
0
0
0
0
0
0
0
0
2.3
5.8
8.3
13.1
0
0
0
0
0
0
0
0
3.4
5.8
9.2
11.8
15.4
0
0
0
0
0
0
0
2.4
5.8
8.9
12.6
15.9
15.6
0
0
0
0
0
0
2.4
5.8
8.4
13.1
16.8
18.6
21.6
0
0
0
0
0
2.5
4.8
9.1
13.2
16.2
20.1
23.6
31.3
0
0
0
0
1.9
3.7
6.5
11.7
17.7
20.1
24.7
32.5
35.7
0
0
0
1.4
2.3
4.2
9.1
16
13.7
32.3
41.1
41.4
45.6
0
0
0.9
1.5
2.6
5.7
10.6
15.8
28.2
45
48.3
49.4
46.2
0
0.8
1.1
1.6
3.3
6.7
12.4
18.5
45
48.8
47.7
50
50
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
50000 40000 30000 20000 10000 0
100 80 60 40 20 0
Software RAID0 over 2 Intel P3700 as blockdevice (Test ID 2) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
275551
0
0
0
0
0
0
0
0
0
0
0
253265
402084
0
0
0
0
0
0
0
0
0
0
269033
406832
407535
0
0
0
0
0
0
0
0
0
282385
411431
403804
421056
0
0
0
0
0
0
0
0
281836
456631
400182
423950
452861
0
0
0
0
0
0
0
285070
494171
392063
424596
423248
417955
0
0
0
0
0
0
283548
514343
408322
423038
422143
417417
414601
0
0
0
0
0
277372
481041
400601
418967
422751
418321
432330
414239
0
0
0
0
286510
432509
508600
426464
422011
415894
414590
410600
414131
0
0
0
233126
370709
464756
528644
444213
419880
429828
412396
410976
409012
0
0
129216
238098
364493
449530
513380
519615
437648
413089
412052
410004
409883
0
68067
135594
214244
293042
391005
399628
457341
416798
435865
411568
412254
424380
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
11200
0
0
0
0
0
0
0
0
0
0
0
6624
20608
0
0
0
0
0
0
0
0
0
0
2960
11072
19328
0
0
0
0
0
0
0
0
0
1720
6816
11072
31872
0
0
0
0
0
0
0
0
860
3824
6944
15424
33024
0
0
0
0
0
0
0
434
924
4704
7008
21120
53504
0
0
0
0
0
0
197
564
3184
4576
10688
25984
57600
0
0
0
0
0
106
326
2480
3088
4512
14656
44800
86528
0
0
0
0
58
129
572
1992
3056
4640
14528
43776
102912
0
0
0
55
70
153
442
1928
3088
4768
16064
34048
85504
0
0
49
54
70
139
390
1912
3088
4960
14912
49408
70144
0
50
45
66
99
197
1192
1688
3216
4960
13376
44288
55040
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.5
0
0
0
0
0
0
0
0
0
0
0
2.6
4.3
0
0
0
0
0
0
0
0
0
0
2.7
3.7
4.9
0
0
0
0
0
0
0
0
0
2.4
4
4.7
6.2
0
0
0
0
0
0
0
0
2.6
4.4
4.8
6.4
8
0
0
0
0
0
0
0
2.4
4.5
4.7
7.2
8.1
9.7
0
0
0
0
0
0
2.4
4.3
4.7
6.9
8
9.5
10.3
0
0
0
0
0
2.4
4.4
4.7
6.6
8.5
9.7
10.7
12.9
0
0
0
0
2.4
4.4
5.9
6.8
8.4
10.7
13.1
14
15.2
0
0
0
2.4
4.1
5.8
8
9.5
10.7
15.1
16
17.3
17.6
0
0
1.9
3.1
5.3
8.2
12.1
13.6
15.7
17.8
18.5
18.8
19.4
0
1.4
2.4
4
6.8
10.9
13.3
19.5
20
20.2
19.6
19.7
21.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
500000 400000 300000 200000 100000 0
100000 80000 60000 40000 20000 0
100 80 60 40 20 0
Software RAID0 over 2 Intel P3700 as blockdevice (Test ID 2) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
224351
0
0
0
0
0
0
0
0
0
0
0
282353
424596
0
0
0
0
0
0
0
0
0
0
233407
500707
598373
0
0
0
0
0
0
0
0
0
254989
518115
597827
587623
0
0
0
0
0
0
0
0
245192
530389
596174
592184
591266
0
0
0
0
0
0
0
244334
484493
596179
591969
588928
599954
0
0
0
0
0
0
291466
455761
592626
595270
592201
597684
598376
0
0
0
0
0
177868
317504
491279
590896
591540
598968
600046
589750
0
0
0
0
96255
179726
317812
487671
584564
597812
593146
590646
594892
0
0
0
49642
96704
180425
312697
470889
581044
598041
592253
598060
597941
0
0
24490
49995
96803
177350
297955
456501
546635
592058
599165
598502
598553
0
11705
25086
49544
94983
163726
287276
427151
545932
594339
598601
601358
1.21451e+06
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
13376
0
0
0
0
0
0
0
0
0
0
0
4448
6432
0
0
0
0
0
0
0
0
0
0
3248
2608
40192
0
0
0
0
0
0
0
0
0
1192
1448
17792
12608
0
0
0
0
0
0
0
0
812
1004
8896
6368
24704
0
0
0
0
0
0
0
430
596
3504
3408
14400
50432
0
0
0
0
0
0
258
406
764
1576
5088
24960
58112
0
0
0
0
0
199
258
426
764
1592
8160
35584
60672
0
0
0
0
171
197
258
414
804
1528
8384
39680
50432
0
0
0
161
169
199
258
442
892
1496
8096
18816
60160
0
0
173
155
169
201
306
486
1224
1560
8512
16320
61696
0
270
149
155
173
286
314
804
1336
1672
9536
38144
35584
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.3
0
0
0
0
0
0
0
0
0
0
0
2.4
5.7
0
0
0
0
0
0
0
0
0
0
3.3
5.2
8
0
0
0
0
0
0
0
0
0
3.3
5.4
7.6
9
0
0
0
0
0
0
0
0
3.3
5.1
7.8
9.2
11.4
0
0
0
0
0
0
0
3.3
5.5
7.9
9.5
11.2
12.8
0
0
0
0
0
0
2.4
4.2
7.2
8.9
11.4
12.8
13.7
0
0
0
0
0
2
3.3
5.9
8.7
11.8
13.4
14.2
16.8
0
0
0
0
1.4
2.6
4.4
7.8
12
14.7
16.5
17.4
19.7
0
0
0
0.9
1.6
2.9
5.5
9.9
14.7
18.8
21.6
23.4
27.2
0
0
0.7
1
1.7
3.5
6.6
11.5
18
24.8
28.2
26.1
28.5
0
0.5
0.7
1.1
2.1
4.2
7.9
13.3
23.9
27.9
30.3
30.6
32.3
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1000000 750000 500000 250000 0
60000
45000
30000
15000
0
100 80 60 40 20 0
Software RAID0 over 2 Intel P3700 as blockdevice (Test ID 2) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
206138
0
0
0
0
0
0
0
0
0
0
0
220609
205057
0
0
0
0
0
0
0
0
0
0
241003
205956
204772
0
0
0
0
0
0
0
0
0
230403
204793
205116
191622
0
0
0
0
0
0
0
0
245751
204080
215756
191629
201100
0
0
0
0
0
0
0
247914
178493
205181
213138
200947
199063
0
0
0
0
0
0
246933
186676
204932
203784
202981
198971
197095
0
0
0
0
0
236349
191989
205377
204847
202259
200077
199074
196135
0
0
0
0
224541
264241
213748
203251
202520
199964
198983
197603
196131
0
0
0
179955
253673
264297
213830
203267
201263
198598
198669
194473
193929
0
0
113462
186894
235100
260031
217210
203231
200118
196752
197881
193962
194398
0
57745
109460
162928
217126
242664
211205
200081
201182
197530
197757
206648
197906
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
250000 200000 150000 100000 50000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
17536
0
0
0
0
0
0
0
0
0
0
0
11840
37632
0
0
0
0
0
0
0
0
0
0
5088
20352
45312
0
0
0
0
0
0
0
0
0
7008
12864
22144
69120
0
0
0
0
0
0
0
0
3152
8640
13120
28800
70144
0
0
0
0
0
0
0
1560
5920
8640
15168
43264
115200
0
0
0
0
0
0
740
3440
5664
9664
19328
55040
107008
0
0
0
0
0
446
2608
3216
5536
9408
28544
69120
111104
0
0
0
0
181
708
2256
3248
5664
9920
30592
58112
164864
0
0
0
68
213
732
2256
3248
5728
9408
29056
67072
244736
0
0
53
67
241
708
2192
3248
5728
9408
38656
127488
209920
0
57
55
95
334
1012
2256
3376
5792
9408
45312
120320
96768
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.1
0
0
0
0
0
0
0
0
0
0
0
2.6
3
0
0
0
0
0
0
0
0
0
0
2.4
2.5
3.5
0
0
0
0
0
0
0
0
0
2.5
2.5
3.3
3.8
0
0
0
0
0
0
0
0
2.3
2.9
3.3
3.9
4.8
0
0
0
0
0
0
0
2.3
2.2
3.1
4.3
4.6
5.6
0
0
0
0
0
0
2.3
2.4
3.3
4.2
4.7
5.6
8.1
0
0
0
0
0
2.3
2.3
3.2
4
5
5.8
9.4
9.3
0
0
0
0
2.2
3.1
3.3
3.9
4.8
6.2
8.2
10.3
10.6
0
0
0
2.2
3.3
4.1
4.3
5.3
6.2
9.9
11.5
10
11.1
0
0
1.8
2.7
4.1
5.4
5.9
6.9
9
12.7
11.7
11.3
11.6
0
1.4
2
3.2
4.9
6.8
7.9
10.2
12.2
13
11
11.6
12.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
200000 150000 100000 50000 0
100 80 60 40 20 0
Software RAID0 over 2 Intel P3700 as blockdevice (Test ID 2) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
57012
0
0
0
0
0
0
0
0
0
0
0
63922
100212
0
0
0
0
0
0
0
0
0
0
78035
106384
109979
0
0
0
0
0
0
0
0
0
80870
108735
109102
109936
0
0
0
0
0
0
0
0
84603
109263
110363
108072
109718
0
0
0
0
0
0
0
76830
105042
110297
95583
110019
108451
0
0
0
0
0
0
56627
81663
105203
99702
109365
108532
110312
0
0
0
0
0
35092
56677
82424
99502
109978
111043
108727
110290
0
0
0
0
22484
36772
56712
80929
107082
110436
108390
110876
106808
0
0
0
13424
22778
36257
55935
82059
105018
109028
107715
108762
107110
0
0
7366
13248
22291
35497
55862
80211
102620
109492
107809
107535
109603
0
3743
7223
12956
21459
35266
53724
76203
100794
108149
107879
108141
110435
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
13376
0
0
0
0
0
0
0
0
0
0
0
6368
15424
0
0
0
0
0
0
0
0
0
0
3056
10432
20352
0
0
0
0
0
0
0
0
0
1432
6880
10560
22912
0
0
0
0
0
0
0
0
860
5280
7456
14656
44800
0
0
0
0
0
0
0
362
1848
5280
10048
20608
52992
0
0
0
0
0
0
92
346
1896
7008
10688
26496
60672
0
0
0
0
0
72
67
322
2832
5344
13888
33024
72192
0
0
0
0
46
44
60
382
1624
5344
17792
40192
102912
0
0
0
30
35
45
73
342
1720
5600
17280
45312
86528
0
0
29
31
38
55
78
378
1752
5216
14016
31360
77312
0
76
46
43
51
59
126
516
1528
5408
13248
29568
51968
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.2
0
0
0
0
0
0
0
0
0
0
0
3.1
4.8
0
0
0
0
0
0
0
0
0
0
2.7
4.4
5.4
0
0
0
0
0
0
0
0
0
2.8
4.3
5.2
6.3
0
0
0
0
0
0
0
0
2.4
4.2
5.4
6.3
7.6
0
0
0
0
0
0
0
2.5
4.2
5.3
5.9
8
8.7
0
0
0
0
0
0
2.2
3.6
4.9
5.8
7.8
8.6
9.6
0
0
0
0
0
1.6
2.7
4.4
5.9
8
9
10
10.9
0
0
0
0
1.3
2
3.3
5
7.9
9.4
9.9
11.1
12.8
0
0
0
1
1.4
2.3
3.8
6
9.2
12.6
12.8
13.7
13.7
0
0
0.7
1
1.5
2.8
4.2
7.3
11.7
13.8
14
14.3
15.4
0
0.5
0.7
1.2
2
2.9
5.7
10.2
12.9
14.7
15.1
15.2
18.2
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
100000 80000 60000 40000 20000 0
100000 80000 60000 40000 20000 0
100 80 60 40 20 0
Software RAID0 over 2 Intel P3700 as blockdevice (Test ID 2) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
4.89307e+06
0
4.9439e+06 4.85367e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
4.93545e+06 4.84372e+06 4.51246e+06
4.94753e+06 4.78111e+06 4.84164e+06 4.43713e+06
4.9589e+06 4.79824e+06 4.63017e+06 4.79108e+06 4.70191e+06
4.9522e+06 4.80898e+06 4.75522e+06 4.78608e+06 4.44853e+06 4.48005e+06
4.93659e+06 4.80607e+06 4.80432e+06 4.80291e+06 4.64836e+06 4.10642e+06 3.48404e+06
4.49171e+06 4.79652e+06 4.83527e+06 4.77069e+06 4.63361e+06 3.60013e+06 3.95305e+06 3.55782e+06
3.20125e+06 4.28066e+06 4.79443e+06 4.77253e+06 4.70836e+06 4.54377e+06 4.09115e+06 3.63536e+06 2.96295e+06
1.92767e+06 2.96119e+06 4.166e+06 4.71382e+06 4.74354e+06 4.21355e+06 3.86335e+06 3.86525e+06 3.67262e+06 3.2085e+06
1.8148e+06 1.61112e+06 2.60788e+06 3.80539e+06 4.51706e+06 4.23774e+06 4.21539e+06 4.08426e+06 2.99179e+06 3.33501e+06 2.92809e+06 996595
677184
1
2
8
16
32
64
128
number of workers
3000000 2000000 1000000
0
1.29615e+06 2.24717e+06 3.29354e+06 4.2549e+06 3.66621e+06 4.02089e+06 3.80381e+06 3.13781e+06 2.96339e+06 2.8404e+06
4
4000000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
228352
0
0
0
0
0
0
0
0
0
0
0
100864
252928
0
0
0
0
0
0
0
0
0
0
44288
125440
230400
0
0
0
0
0
0
0
0
0
23936
56064
100864
222208
0
0
0
0
0
0
0
0
12480
27776
52992
102912
195584
0
0
0
0
0
0
0
5728
13504
27520
55552
160768
313344
0
0
0
0
0
0
2544
6432
12608
28800
51456
120320
456704
0
0
0
0
0
1208
3120
5856
12736
27776
58112
140288
468992
0
0
0
0
660
1752
2672
5856
12736
23680
69120
195584
675840
0
0
0
510
940
1304
2736
5792
13632
40704
68096
150528
481280
0
0
378
580
788
1288
3024
5344
15040
40192
102912
284672
864256
0
342
596
708
948
2640
3248
8512
16192
40192
252928
518144
864256
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
1.9
0
0
0
0
0
0
0
0
0
0
0
1.7
1.9
0
0
0
0
0
0
0
0
0
0
1.6
1.9
2
0
0
0
0
0
0
0
0
0
1.6
1.8
1.9
2.1
0
0
0
0
0
0
0
0
1.6
1.8
2
2
2.5
0
0
0
0
0
0
0
1.6
1.8
2
2
2.5
2.7
0
0
0
0
0
0
1.3
1.5
1.8
1.9
2
3.4
3
0
0
0
0
0
1.1
1.3
1.5
1.7
1.9
3.5
2.5
3.5
0
0
0
0
0.9
1.2
1.4
1.6
1.8
2.6
2.5
3.2
4.3
0
0
0
0.7
0.9
1.4
1.6
1.7
4
3.2
3.3
2.7
3.8
0
0
0.7
0.8
1.1
1.4
1.8
2.9
2.3
2.7
4.3
3.4
4.2
0
0.7
1
1
1.3
1.5
3.8
3
2.6
3.1
4.5
3.9
5.3
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
750000 600000 450000 300000 150000 0
100 80 60 40 20 0
Software RAID0 over 2 Intel P3700 as blockdevice (Test ID 2) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
3.65564e+06
0
3.7321e+06 3.87017e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3.5873e+06 3.90936e+06 3.86691e+06
3.80695e+06 3.96279e+06 3.88114e+06 3.82427e+06
3.90882e+06 3.83339e+06 3.88482e+06 3.76341e+06 3.53947e+06
3.87358e+06 3.70626e+06 3.90898e+06 3.83327e+06 3.53885e+06 3.70365e+06
3.79962e+06 3.85354e+06 3.8945e+06 3.81356e+06 3.29970e+06 3.55574e+06 3.53603e+06
3.7595e+06 3.85662e+06 3.84688e+06 3.82604e+06 2.979e+06 3.27476e+06 3.61812e+06 3.57714e+06
3.67305e+06 3.63026e+06 3.61676e+06 3.75148e+06 2.52123e+06 3.30353e+06 3.73167e+06 3.71855e+06 3.20618e+06
3.54962e+06 3.59608e+06 3.38457e+06 3.69642e+06 3.19612e+06 3.22396e+06 3.63649e+06 3.48486e+06 3.47553e+06 3.30626e+06
3.23657e+06 3.47368e+06 3.34753e+06 3.63062e+06 3.53086e+06 2.88346e+06 3.51636e+06 3.30487e+06 3.44247e+06 3.15575e+06 3.27716e+06
2
4
8
16
32
64
128
number of workers
2400000 1600000 800000
0
1.95704e+06 3.23387e+06 3.44434e+06 3.33961e+06 3.57423e+06 2.94401e+06 3.40161e+06 2.95038e+06 3.4745e+06 3.28428e+06 3.38769e+06 3.21579e+06
1
3200000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
88576
0
0
0
0
0
0
0
0
0
0
0
45312
191488
0
0
0
0
0
0
0
0
0
0
27776
98816
199680
0
0
0
0
0
0
0
0
0
24704
48384
86528
284672
0
0
0
0
0
0
0
0
17024
38144
52480
134144
468992
0
0
0
0
0
0
0
9280
20096
30592
75264
183296
489472
0
0
0
0
0
0
5472
10944
16512
41216
142336
313344
350208
0
0
0
0
0
2768
5216
10688
18304
62720
102912
325632
626688
0
0
0
0
1736
2864
6048
10688
37632
44800
166912
284672
536576
0
0
0
948
1832
3312
6112
12864
24192
35584
171008
403456
937984
0
0
426
1176
2448
3184
6176
19072
22400
55040
211968
675840
724992
0
78
346
1128
2008
3088
9536
12992
31872
41216
240640
489472
815104
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.4
0
0
0
0
0
0
0
0
0
0
0
2.6
3.8
0
0
0
0
0
0
0
0
0
0
2.6
3.7
3.3
0
0
0
0
0
0
0
0
0
2.2
2.9
3.3
4.8
0
0
0
0
0
0
0
0
1.9
2.6
3.4
3.3
3.9
0
0
0
0
0
0
0
1.9
1.9
3
4
3.9
4.3
0
0
0
0
0
0
1.6
1.8
2
2.9
2.7
3.5
6.1
0
0
0
0
0
1.4
1.6
1.7
1.9
2.2
2.7
4.9
5.2
0
0
0
0
1.3
1.5
1.7
1.8
1.5
2.3
4.5
4.3
6.9
0
0
0
1.4
1.5
1.6
1.8
1.7
2
4
4.6
5.1
6.5
0
0
1.3
1.5
1.5
1.8
1.9
2.1
3
3.3
4.6
5.1
6.7
0
0.9
1.4
1.6
1.7
1.8
2.3
2.4
2.8
4.7
4.5
5.2
6.6
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
800000 600000 400000 200000 0
100 80 60 40 20 0
Software RAID0 over 4 Intel P3700 as blockdevice (Test ID 3) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
267006
0
0
0
0
0
0
0
0
0
0
0
254636
517426
0
0
0
0
0
0
0
0
0
0
278852
528254
905826
0
0
0
0
0
0
0
0
0
277554
515683
854269
1.56227e+06
0
0
0
0
0
0
0
0
278533
520345
938523
1.53923e+06 1.89874e+06
0
0
0
0
0
0
0
285924
496290
875160
1.57784e+06 1.89662e+06 1.84889e+06
0
0
0
0
0
0
264140
470355
716661
1.46966e+06 1.90722e+06 1.8662e+06 1.91054e+06
0
0
0
0
0
255112
415440
764542
1.34475e+06 1.88907e+06 1.8602e+06 1.88699e+06 1.90772e+06
0
0
0
0
159544
320869
566456
937075
0
0
0
81117
161468
327164
563851
895731
0
0
42897
83208
168695
308285
517612
637678
937950
19918
41424
83821
163706
296032
435169
596354
931740
1
2
4
8
16
32
64
128
1.46176e+06 1.71538e+06 1.88004e+06 1.9082e+06 1.70294e+06
1.04486e+06 1.6958e+06 1.90833e+06 1.63446e+06 1.90428e+06
1.75302e+06 1.90754e+06 1.90564e+06 1.90777e+06
number of workers
1.79241e+06 1.89962e+06 2.46368e+06
2000000 1500000 1000000 500000
0 940922
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
8384
0
0
0
0
0
0
0
0
0
0
0
5792
4960
0
0
0
0
0
0
0
0
0
0
2040
2416
3952
0
0
0
0
0
0
0
0
0
1096
1272
2192
8512
0
0
0
0
0
0
0
0
612
668
1144
3760
8896
0
0
0
0
0
0
0
426
410
620
1784
4512
9664
0
0
0
0
0
0
262
298
2064
804
1864
5984
18304
0
0
0
0
0
171
205
1080
524
860
1592
7136
28800
0
0
0
0
149
153
199
290
438
1096
2024
19840
9152
0
0
0
143
147
141
171
326
684
612
2416
7200
17280
0
0
135
131
133
151
233
708
1112
516
2512
14912
25216
0
125
133
131
141
181
274
892
812
502
2800
812
8384
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6
0
0
0
0
0
0
0
0
0
0
0
3.6
6.3
0
0
0
0
0
0
0
0
0
0
3.6
6.2
10.6
0
0
0
0
0
0
0
0
0
3.6
6
10
19.1
0
0
0
0
0
0
0
0
3.6
6
11.1
19.1
26.6
0
0
0
0
0
0
0
2.8
6.3
10.2
19.1
27.3
36.2
0
0
0
0
0
0
3.2
6
9.1
18.5
28.2
34.9
45.1
0
0
0
0
0
3
4.9
8.9
17.3
27.2
32.3
52.6
63.3
0
0
0
0
2.4
4.2
8
14
23.1
33.7
60.8
64.7
37
0
0
0
1.7
3.2
5.5
11.2
18.2
25.6
58.9
69
58.1
71.2
0
0
1.1
2
3.5
6.9
12.9
20.6
40.6
71.7
73.4
72.1
69.3
0
0.8
1.2
2.1
4.2
7.9
15.5
22
51.3
73.4
76.2
70.7
44.5
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
25000 20000 15000 10000 5000 0
100 80 60 40 20 0
Software RAID0 over 4 Intel P3700 as blockdevice (Test ID 3) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
255422
0
0
0
0
0
0
0
0
0
0
0
261356
435200
0
0
0
0
0
0
0
0
0
0
274759
469090
782292
0
0
0
0
0
0
0
0
0
274041
471854
794987
679341
0
0
0
0
0
0
0
0
251813
469797
706013
763292
750601
0
0
0
0
0
0
0
269105
417368
743173
758157
747286
739293
0
0
0
0
0
0
272752
380288
657415
765604
756580
760779
744472
0
0
0
0
0
266765
381316
716560
778068
695812
743788
742132
745464
0
0
0
0
255931
401816
662405
953505
648281
744392
743684
750298
744639
0
0
0
241924
359872
627266
857548
965125
949897
749133
741809
752906
747494
0
0
104220
211515
377849
567503
719337
757564
981036
740285
807988
740288
754962
0
61042
110898
209984
328481
440822
443945
452348
787636
814219
748963
742263
762842
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
10560
0
0
0
0
0
0
0
0
0
0
0
5920
6048
0
0
0
0
0
0
0
0
0
0
2128
3536
5664
0
0
0
0
0
0
0
0
0
1208
1800
9536
25472
0
0
0
0
0
0
0
0
716
876
6304
12352
43264
0
0
0
0
0
0
0
326
486
964
7328
20864
68096
0
0
0
0
0
0
149
302
1560
4576
9792
22400
66048
0
0
0
0
0
96
181
290
2704
5024
10816
23168
61184
0
0
0
0
51
101
330
964
3152
4704
10048
22144
55552
0
0
0
41
67
88
217
852
2160
4576
11584
24960
58624
0
0
48
54
62
98
310
804
1496
4768
11200
20864
41216
0
45
47
51
86
165
1112
2192
2416
4768
11840
20864
52992
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.6
0
0
0
0
0
0
0
0
0
0
0
2.6
5.5
0
0
0
0
0
0
0
0
0
0
2.8
5.2
9.6
0
0
0
0
0
0
0
0
0
2.7
5.2
9
10.2
0
0
0
0
0
0
0
0
3
5.5
8.8
11.1
12.8
0
0
0
0
0
0
0
2.7
5.2
8.6
11
13.8
16.2
0
0
0
0
0
0
2.7
4.8
8.7
12.1
14.3
16.9
19.9
0
0
0
0
0
2.7
4.8
8.5
12.2
13.9
17.7
22.3
27.4
0
0
0
0
2.9
5.4
8.6
14.9
13.2
19.9
25.6
30.7
33.2
0
0
0
2.8
5.1
8.8
14
18.9
24.3
27.1
32.2
34
36.1
0
0
2.4
4.1
6.8
11.4
17.6
23.8
40.6
33.9
37.8
36.5
38
0
1.7
2.7
5
9
14.9
17.8
22.1
38.3
39.9
38.1
37.3
38.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID0 over 4 Intel P3700 as blockdevice (Test ID 3) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
255703
0
0
0
0
0
0
0
0
0
0
0
233397
451831
0
0
0
0
0
0
0
0
0
0
270172
455023
863170
0
0
0
0
0
0
0
0
0
295196
488044
879560
1.12201e+06
0
0
0
0
0
0
0
0
245462
479616
835220
1.12782e+06 1.10454e+06
0
0
0
0
0
0
0
251120
505015
865667
1.13032e+06 1.1029e+06 1.11702e+06
0
0
0
0
0
0
242235
400038
741802
1.12749e+06 1.09983e+06 1.11414e+06 1.12496e+06
0
0
0
0
0
188667
343544
572799
932412
0
0
0
0
100228
198046
359091
635379
900850
0
0
0
50952
103343
192724
362017
582709
0
0
1.10897e+06 1.11956e+06 1.13713e+06 1.13794e+06
1.08353e+06 1.14066e+06 1.14089e+06 1.12325e+06 737718
1.09157e+06 1.13928e+06 1.12276e+06 1.13534e+06
25148
51913
99581
195257
350595
532933
671553
12469
25944
50325
100461
174410
322724
466417
1.08367e+06 1.12097e+06 1.12001e+06 1.97254e+06 679921
1
2
4
8
16
32
64
128
number of workers
1600000 1200000 800000 400000
0
1.06639e+06 1.12282e+06 1.14025e+06 1.76456e+06
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
12352
0
0
0
0
0
0
0
0
0
0
0
5088
5472
0
0
0
0
0
0
0
0
0
0
2224
2704
4256
0
0
0
0
0
0
0
0
0
1032
1320
2096
11968
0
0
0
0
0
0
0
0
788
740
1144
5344
25728
0
0
0
0
0
0
0
414
406
668
2384
12224
44288
0
0
0
0
0
0
247
334
1288
1064
3952
18048
41728
0
0
0
0
0
185
227
506
556
1160
4448
13632
42240
0
0
0
0
171
181
237
270
572
1176
2608
18304
47360
0
0
0
167
163
199
207
390
1032
1012
3632
20096
42240
0
0
171
157
181
185
233
612
1080
852
4192
20352
18816
0
191
153
191
171
245
314
1048
1336
868
3472
33536
28544
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.6
0
0
0
0
0
0
0
0
0
0
0
3.6
6.4
0
0
0
0
0
0
0
0
0
0
2.7
6.1
10.4
0
0
0
0
0
0
0
0
0
2.7
6.4
10.9
15.3
0
0
0
0
0
0
0
0
3.5
6.2
10.1
15.7
19.2
0
0
0
0
0
0
0
3.5
6.3
10.1
15.7
17.9
27
0
0
0
0
0
0
3.3
5.2
9
16.4
19.1
26.4
32.6
0
0
0
0
0
2.6
4.9
8.2
14.2
19.9
27.3
37.2
49.9
0
0
0
0
1.9
3.4
6
11.6
17.6
25.8
43.8
50.8
56.2
0
0
0
1.3
2.1
3.9
7.5
13.8
21.4
45.5
56
55.2
58
0
0
0.9
1.4
2.4
4.5
8.7
15.9
28.9
58.8
51.1
59.8
49
0
0.6
0.9
1.4
3
5.8
9.9
18.7
38.3
59
63.2
63.6
49.2
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
40000 32000 24000 16000 8000 0
100 80 60 40 20 0
Software RAID0 over 4 Intel P3700 as blockdevice (Test ID 3) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
222764
0
0
0
0
0
0
0
0
0
0
0
194451
391777
0
0
0
0
0
0
0
0
0
0
207538
369984
376140
0
0
0
0
0
0
0
0
0
231325
364635
376589
375098
0
0
0
0
0
0
0
0
240538
415081
373767
314438
369397
0
0
0
0
0
0
0
240930
347158
390976
305175
369224
369466
0
0
0
0
0
0
236142
324958
383162
326351
369914
367575
367015
0
0
0
0
0
224532
406871
428809
372863
359659
365408
369179
368027
0
0
0
0
215930
398180
452023
397514
299811
370189
367162
377874
368015
0
0
0
189449
306182
435130
451992
294289
372592
368843
368488
363246
368816
0
0
102520
175706
300354
417347
498791
427129
398460
366022
375885
368479
362311
0
57803
102344
168425
265233
360918
393209
395782
369427
361056
370164
364881
363941
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
400000 300000 200000 100000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
12096
0
0
0
0
0
0
0
0
0
0
0
7264
11584
0
0
0
0
0
0
0
0
0
0
3248
6048
36608
0
0
0
0
0
0
0
0
0
1448
2480
20096
43264
0
0
0
0
0
0
0
0
780
1800
12992
27264
82432
0
0
0
0
0
0
0
430
940
8640
16320
50944
121344
0
0
0
0
0
0
203
692
5344
10176
18560
72192
136192
0
0
0
0
0
126
748
2608
5408
9280
25216
77312
195584
0
0
0
0
69
245
1224
2864
6176
8896
25472
76288
171008
0
0
0
57
75
338
1288
3344
5344
9024
20096
62720
175104
0
0
51
61
84
282
1064
2800
5152
9024
18816
84480
132096
0
48
45
66
133
426
1288
2832
5344
9024
22912
90624
104960
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.6
0
0
0
0
0
0
0
0
0
0
0
3.5
5.4
0
0
0
0
0
0
0
0
0
0
3.1
5.1
6.1
0
0
0
0
0
0
0
0
0
2.9
4.8
6.2
7.3
0
0
0
0
0
0
0
0
3
5
5.8
6.2
8.6
0
0
0
0
0
0
0
2.7
4.6
6
6.2
8.6
10.1
0
0
0
0
0
0
2.7
4.5
6.3
6.8
8.6
9.9
11.9
0
0
0
0
0
2.7
4.8
6.9
7.3
8.7
10.4
12.5
14.7
0
0
0
0
3.1
4.8
7.3
8.2
7.7
11.3
13.4
15.9
16.4
0
0
0
2.6
4.9
7.5
9.2
7.6
12
15.2
17.2
17.6
18.9
0
0
2.1
3.6
6.3
9.7
13.2
14.1
17.3
18.3
19.5
19.2
18.3
0
1.4
2.5
4.7
7.3
12
15.8
19.5
19.2
19.5
20
20
15.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
160000 120000 80000 40000 0
100 80 60 40 20 0
Software RAID0 over 4 Intel P3700 as blockdevice (Test ID 3) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
60254
0
0
0
0
0
0
0
0
0
0
0
61341
117843
0
0
0
0
0
0
0
0
0
0
60688
100875
176293
0
0
0
0
0
0
0
0
0
65398
115006
190732
191916
0
0
0
0
0
0
0
0
78297
108497
184281
207698
200667
0
0
0
0
0
0
0
73093
106660
174158
209914
199810
200976
0
0
0
0
0
0
69450
108922
155717
208424
200237
202958
203602
0
0
0
0
0
45948
74356
117456
169728
197628
202724
203047
203766
0
0
0
0
26892
45501
75509
117793
166755
197752
204442
198519
201356
0
0
0
15140
26688
46842
74790
114751
162135
194987
200011
206423
201711
0
0
7764
14844
27208
45898
72615
111212
149577
192506
201494
202353
203333
0
3821
7724
14783
26246
44643
72902
105410
139582
189279
201250
202475
203894
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
200000 160000 120000 80000 40000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
12096
0
0
0
0
0
0
0
0
0
0
0
6048
6368
0
0
0
0
0
0
0
0
0
0
3504
3920
15680
0
0
0
0
0
0
0
0
0
1480
1880
9280
82432
0
0
0
0
0
0
0
0
548
1048
5664
13632
50432
0
0
0
0
0
0
0
247
454
3184
7456
29824
71168
0
0
0
0
0
0
103
165
1256
4384
9408
27520
88576
0
0
0
0
0
56
75
143
466
4576
8896
47872
121344
0
0
0
0
49
52
72
139
628
3504
9280
46848
74240
0
0
0
46
50
45
70
171
604
3248
8640
24192
53504
0
0
68
65
42
61
108
249
1128
2960
8256
24704
90624
0
114
76
68
59
69
99
402
1384
2960
9536
26496
66048
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.5
0
0
0
0
0
0
0
0
0
0
0
3.5
6.3
0
0
0
0
0
0
0
0
0
0
3.5
4.6
9.1
0
0
0
0
0
0
0
0
0
3.3
5.7
8.5
11.1
0
0
0
0
0
0
0
0
2.9
5.3
8.4
11.7
14.6
0
0
0
0
0
0
0
3
5
8.3
12.2
14.2
17.9
0
0
0
0
0
0
2.7
4.8
8.2
12.2
14.8
17
20.8
0
0
0
0
0
2.3
4.2
6.5
11.3
15.2
18.6
20.6
23.6
0
0
0
0
1.8
2.9
4.9
7.9
13.2
18.4
25
28
34.6
0
0
0
1.3
2.1
3.2
5.7
10
15.8
25.3
33.7
36.8
38.8
0
0
0.9
1.4
2.2
4
6.6
11.1
19.3
34.7
38.8
39.7
39.9
0
0.7
1
1.6
2.6
4.1
7.4
13.4
21.8
38.4
39.6
42.1
34.8
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
100000 75000 50000 25000 0
100 80 60 40 20 0
Software RAID0 over 4 Intel P3700 as blockdevice (Test ID 3) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
9.25164e+06
0
9.16333e+06 9.44484e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
9.32381e+06 9.48327e+06 9.47826e+06
8.82530e+06 9.52107e+06 9.34564e+06 9.58002e+06
9.58046e+06 9.46478e+06 9.39077e+06 9.4748e+06 9.31438e+06
9.13524e+06 9.41599e+06 9.45389e+06 9.49609e+06 9.12498e+06 8.08698e+06
8.87886e+06 9.3621e+06 9.40414e+06 9.47229e+06 9.15773e+06 6.89646e+06 6.48462e+06
6.35665e+06 8.5877e+06 9.22326e+06 9.45481e+06 9.2896e+06 7.74981e+06 7.47907e+06 5.95643e+06
3.98408e+06 5.82341e+06 8.10227e+06 9.26963e+06 9.09951e+06 5.5308e+06 4.97053e+06 5.76346e+06 5.57658e+06
3.52881e+06 3.13213e+06 4.95492e+06 7.52311e+06 8.97687e+06 6.56266e+06 5.3285e+06 5.40204e+06 5.35451e+06 5.73282e+06
1.90811e+06 1.44887e+06 2.55771e+06 4.2633e+06 6.29026e+06 6.21336e+06 5.50838e+06 5.38108e+06 6.10628e+06 5.36675e+06 5.85417e+06 1.14154e+06
624038
1
2
8
16
32
64
128
number of workers
6000000 4000000 2000000
0
1.22683e+06 2.33138e+06 3.78138e+06 5.34544e+06 5.06805e+06 5.41649e+06 5.36645e+06 5.29426e+06 6.11241e+06 6.01769e+06
4
8000000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
35584
0
0
0
0
0
0
0
0
0
0
0
20864
175104
0
0
0
0
0
0
0
0
0
0
18304
89600
146432
0
0
0
0
0
0
0
0
0
5664
39680
87552
211968
0
0
0
0
0
0
0
0
9792
23424
39680
95744
257024
0
0
0
0
0
0
0
2832
11712
23168
50944
179200
254976
0
0
0
0
0
0
1256
4320
11200
24192
70144
112128
423936
0
0
0
0
0
700
1624
4384
10176
19328
87552
354304
651264
0
0
0
0
532
916
2608
3408
9024
40704
117248
382976
432128
0
0
0
390
676
2512
1640
3280
14400
49920
189440
544768
733184
0
0
366
620
2192
1192
1784
6240
16768
36608
171008
387072
626688
0
346
668
1512
972
1320
1896
7840
19072
45312
113152
440320
448512
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.5
0
0
0
0
0
0
0
0
0
0
0
2.6
3.2
0
0
0
0
0
0
0
0
0
0
2.5
3.2
3.8
0
0
0
0
0
0
0
0
0
3.1
3.2
3.6
4
0
0
0
0
0
0
0
0
2.6
3.1
3.4
3.9
4.5
0
0
0
0
0
0
0
3.2
3.2
3.5
4.1
4.6
6
0
0
0
0
0
0
2.5
2.8
3
3.8
4
6.6
7.4
0
0
0
0
0
1.6
2.2
2.6
3.4
4
5.4
4.5
7.7
0
0
0
0
1.3
1.8
2.4
3.5
3.9
6.4
7.1
7.7
6.8
0
0
0
1.3
1.3
1.9
2.7
3.3
7.5
6.8
7.8
7.6
7.7
0
0
1
1
1.4
2.4
4.2
7
7.7
7.4
7.2
5.6
7.9
0
0.7
1.1
1.2
1.7
4.2
7.4
7.6
7.8
6
6.2
5.9
5.2
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000 450000 300000 150000 0
100 80 60 40 20 0
Software RAID0 over 4 Intel P3700 as blockdevice (Test ID 3) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6121e+06
0
2.82952e+06 5.80194e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3.49516e+06 6.31709e+06 7.61212e+06
3.54455e+06 6.99942e+06 7.6571e+06 7.68149e+06
6.2427e+06 6.95491e+06 7.48882e+06 7.58442e+06 7.43907e+06
6.23624e+06 6.79671e+06 7.55215e+06 7.66542e+06 7.13938e+06 7.09285e+06
6.35094e+06 7.08401e+06 6.5731e+06 7.51632e+06 5.37515e+06 6.11142e+06 7.41251e+06
6.45366e+06 7.2243e+06 6.29196e+06 5.13257e+06 4.89192e+06 6.56368e+06 7.13101e+06 6.84948e+06
6.15847e+06 7.51415e+06 6.39732e+06 5.52925e+06 4.89924e+06 5.94639e+06 7.25069e+06 7.01098e+06 6.95049e+06
5.12148e+06 6.84447e+06 6.29691e+06 5.7835e+06 4.92512e+06 5.16706e+06 7.01778e+06 6.97165e+06 6.82828e+06 6.5281e+06
2.97208e+06 6.19593e+06 4.904e+06 6.04066e+06 4.92684e+06 6.20141e+06 7.10667e+06 6.9923e+06 6.95686e+06 6.70746e+06 5.63442e+06
2
4
8
16
32
64
128
number of workers
6000000 4500000 3000000 1500000
0
1.51753e+06 3.10062e+06 4.58665e+06 6.41792e+06 5.43953e+06 6.35536e+06 6.40191e+06 7.09757e+06 7.14629e+06 6.60754e+06 5.14784e+06 5.43317e+06
1
7500000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
89600
0
0
0
0
0
0
0
0
0
0
0
72192
166912
0
0
0
0
0
0
0
0
0
0
24448
47360
209920
0
0
0
0
0
0
0
0
0
10816
55552
92672
209920
0
0
0
0
0
0
0
0
11328
44800
47360
134144
493568
0
0
0
0
0
0
0
6048
24448
30592
60672
254976
448512
0
0
0
0
0
0
2448
10432
29824
33536
138240
305152
358400
0
0
0
0
0
2448
5344
14144
33536
68096
138240
264192
460800
0
0
0
0
1176
2224
5984
18048
34560
56064
156672
569344
407552
0
0
0
151
1400
3056
9280
17024
39168
25984
232448
602112
667648
0
0
211
191
3280
5408
10816
12864
16768
42752
164864
440320
765952
0
173
137
1272
1256
5792
5920
13888
15168
30336
220160
485376
815104
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.7
0
0
0
0
0
0
0
0
0
0
0
2.7
4.7
0
0
0
0
0
0
0
0
0
0
2.6
4.7
5.4
0
0
0
0
0
0
0
0
0
2.8
4.2
4.9
6.6
0
0
0
0
0
0
0
0
2.8
3.7
6.2
5.9
6.4
0
0
0
0
0
0
0
2.5
3.3
4.4
5.5
5.3
7.7
0
0
0
0
0
0
2.6
3.2
3.5
4.5
3.8
4.8
8.3
0
0
0
0
0
2.4
2.8
2.7
2.6
3.1
4.2
7.1
7.4
0
0
0
0
2.3
2.9
2.7
2.7
3
4
7.6
7.7
8.9
0
0
0
2.2
2.6
2.7
2.9
3.1
3.3
7.8
7.8
9.4
7.9
0
0
1.7
2.5
2.3
3.1
2.6
5
6.8
7.2
8.2
7.3
5.7
0
1.1
1.6
2.5
3.2
2.7
3.7
7
8.3
8
8.5
5.8
6
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
750000 600000 450000 300000 150000 0
100 80 60 40 20 0
Software RAID0 over 6 Intel P3700 as blockdevice (Test ID 4) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
244875
0
0
0
0
0
0
0
0
0
0
0
253310
507624
0
0
0
0
0
0
0
0
0
0
257669
534348
965424
0
0
0
0
0
0
0
0
0
283588
533118
865595
1.65475e+06
0
0
0
0
0
0
0
0
260956
504643
955800
1.67887e+06 2.17856e+06
0
0
0
0
0
0
0
259913
508473
955820
1.66442e+06 2.13551e+06 2.48129e+06
0
0
0
0
0
0
265286
440649
776881
1.61081e+06 2.06432e+06 2.53684e+06 2.68178e+06
0
0
0
0
0
247655
437717
710501
1.29646e+06 2.15985e+06 2.50302e+06 2.70589e+06 2.74947e+06
0
0
0
0
141866
271363
512642
855979
0
0
0
73139
141569
278850
491611
815599
0
0
36570
71242
142750
272429
452143
586793
807820
17177
36116
70338
140670
254252
441659
477467
777242
1
2
4
8
16
32
64
128
1.39569e+06 2.06365e+06 2.56655e+06 2.47758e+06 2.38005e+06
1.07812e+06 1.63672e+06 2.70226e+06 2.71744e+06 2.70460e+06
1.48093e+06 2.67005e+06 2.65664e+06 2.72262e+06
number of workers
2500000 2000000 1500000 1000000 500000
0
1.70145e+06 2.62213e+06 2.72656e+06 2.72991e+06
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
11840
0
0
0
0
0
0
0
0
0
0
0
4896
4320
0
0
0
0
0
0
0
0
0
0
3088
2160
2384
0
0
0
0
0
0
0
0
0
1160
1112
5280
1576
0
0
0
0
0
0
0
0
764
684
2352
836
1480
0
0
0
0
0
0
0
406
378
402
474
788
1928
0
0
0
0
0
0
231
278
996
1064
516
1192
13248
0
0
0
0
0
161
189
524
692
398
684
3568
21120
0
0
0
0
143
159
179
386
326
636
844
2416
6688
0
0
0
135
145
157
217
306
980
796
796
3568
11712
0
0
135
137
143
159
270
1096
1752
772
852
4448
13760
0
167
135
147
161
213
262
1544
1400
716
900
4320
8768
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.7
0
0
0
0
0
0
0
0
0
0
0
3.7
6.4
0
0
0
0
0
0
0
0
0
0
3.7
6.6
11.5
0
0
0
0
0
0
0
0
0
3
6.6
10.7
19.9
0
0
0
0
0
0
0
0
3.7
6.8
11.5
20.4
36.9
0
0
0
0
0
0
0
3.7
6.8
11.2
20.2
36.4
63.6
0
0
0
0
0
0
3.7
5.1
9.8
19.5
36.2
63.6
86.4
0
0
0
0
0
3.1
5.4
9.2
16.7
33.9
57.8
86.6
89.4
0
0
0
0
2.4
4.5
8.2
13.4
22.8
42.1
81.3
92.9
67.2
0
0
0
1.6
3.1
5.6
10
17.3
30
59.6
87.9
87.3
87.7
0
0
1
1.9
3.4
6.6
12.2
18.8
33.4
62.4
87.2
86.4
87.2
0
0.8
1.1
2.1
4.1
7.5
13.9
20.7
41.5
72.2
85.7
56.9
88.6
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
20000 16000 12000 8000 4000 0
100 80 60 40 20 0
Software RAID0 over 6 Intel P3700 as blockdevice (Test ID 4) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
237049
0
0
0
0
0
0
0
0
0
0
0
232817
448400
0
0
0
0
0
0
0
0
0
0
253005
470405
716788
0
0
0
0
0
0
0
0
0
230961
400864
793985
1.03488e+06
0
0
0
0
0
0
0
0
220821
428102
665793
1.04794e+06 1.0442e+06
0
0
0
0
0
0
0
235966
433792
701851
1.10255e+06 1.04101e+06 1.02298e+06
0
0
0
0
0
0
240917
456098
678042
1.28902e+06 1.08617e+06 1.0397e+06 1.02264e+06
0
0
0
0
0
244270
389852
701429
1.14856e+06 1.19516e+06 1.04157e+06 1.04121e+06 1.02777e+06
0
0
0
0
250759
431398
720519
1.15641e+06 1.27468e+06 1.23759e+06 1.02669e+06 1.02714e+06 1.01999e+06
0
0
0
221674
401085
598356
847479
0
0
110875
204363
362699
468589
763595
749846
827772
57819
110441
192970
333765
472206
520681
463086
793526
1
2
4
8
16
32
64
128
1.15842e+06 1.27971e+06 1.40875e+06 1.03005e+06 1.02397e+06 1.02486e+06
1.37107e+06 1.0285e+06 1.0145e+06 1.01811e+06
number of workers
0
1.03418e+06 1.01613e+06 1.0206e+06 1.07682e+06
256
512
1024
2048
1250000 1000000 750000 500000 250000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
13248
0
0
0
0
0
0
0
0
0
0
0
5536
6368
0
0
0
0
0
0
0
0
0
0
2256
2960
11456
0
0
0
0
0
0
0
0
0
1496
1816
6496
20864
0
0
0
0
0
0
0
0
836
892
3024
12608
43264
0
0
0
0
0
0
0
382
422
2040
7200
22144
50944
0
0
0
0
0
0
191
203
876
2832
7776
23936
88576
0
0
0
0
0
107
139
394
1464
3856
10304
26496
63744
0
0
0
0
51
64
245
486
2064
3920
8512
20864
54016
0
0
0
41
50
149
217
470
1480
2864
8512
21632
58624
0
0
42
56
69
362
237
1432
1896
2608
9280
20608
48384
0
39
47
55
87
129
772
2448
1624
3792
9280
19328
47872
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.9
0
0
0
0
0
0
0
0
0
0
0
3.6
6.1
0
0
0
0
0
0
0
0
0
0
3
5.7
9.4
0
0
0
0
0
0
0
0
0
3.7
5.3
9.9
13.7
0
0
0
0
0
0
0
0
3.7
5.2
9
13.8
17
0
0
0
0
0
0
0
3.4
5.5
9.9
14.1
16.8
21.9
0
0
0
0
0
0
3.1
5.8
9.2
15.8
18.8
22.6
27
0
0
0
0
0
3.4
5.3
9
16.6
19.9
24.2
30.9
37.7
0
0
0
0
3
5.3
9.9
17
19.7
28
35.8
42
44.1
0
0
0
3.2
5.7
9.3
13.7
20.2
28.6
47.5
44.7
45.3
46.9
0
0
2.4
4.4
7.6
10.3
18.9
22.7
37.8
56.3
47.1
47.2
48.1
0
1.7
3
5.3
9.6
16.3
22.1
22.5
46.8
50.8
48
48.6
50
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
75000 60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID0 over 6 Intel P3700 as blockdevice (Test ID 4) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
246176
0
0
0
0
0
0
0
0
0
0
0
249090
420125
0
0
0
0
0
0
0
0
0
0
233567
462543
864087
0
0
0
0
0
0
0
0
0
243350
456710
963360
1.32992e+06
0
0
0
0
0
0
0
0
268081
493088
933942
1.46406e+06 1.65458e+06
0
0
0
0
0
0
0
268836
483822
930344
1.34249e+06 1.64137e+06 1.64535e+06
0
0
0
0
0
0
266352
443186
747767
1.25785e+06 1.63838e+06 1.63676e+06 1.6368e+06
0
0
0
0
0
189150
336936
560280
979479
0
0
0
0
97740
189617
354396
578330
981473
0
0
0
50941
97228
187679
334273
557202
718126
0
0
25152
49196
96597
184220
341854
528566
669208
11135
24644
49179
92872
176589
327653
472387
614997
1
2
4
8
16
32
64
128
1.37889e+06 1.64199e+06 1.60322e+06 1.66987e+06
1.27727e+06 1.64513e+06 1.67112e+06 1.66993e+06
1.15516e+06 1.66538e+06 1.6736e+06 1.66806e+06
1.1051e+06 1.65699e+06 1.66779e+06 3.26662e+06
number of workers
3000000 2400000 1800000 1200000 600000
0
1.11108e+06 1.59547e+06 1.67277e+06 3.35746e+06
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
9280
0
0
0
0
0
0
0
0
0
0
0
4384
7072
0
0
0
0
0
0
0
0
0
0
2576
2448
2736
0
0
0
0
0
0
0
0
0
1464
1608
1400
3440
0
0
0
0
0
0
0
0
612
660
780
3088
23936
0
0
0
0
0
0
0
342
382
780
3408
6048
37120
0
0
0
0
0
0
217
278
788
3408
2128
9024
39680
0
0
0
0
0
175
217
366
596
644
1640
10560
47872
0
0
0
0
163
179
217
382
450
1064
1544
13760
33536
0
0
0
151
165
187
266
532
1192
1144
1400
19072
39680
0
0
153
161
167
197
231
628
1624
1012
1464
9664
6368
0
199
161
161
187
231
266
788
1784
1012
1400
11072
18560
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3
0
0
0
0
0
0
0
0
0
0
0
3
6.3
0
0
0
0
0
0
0
0
0
0
3.7
6.6
10.1
0
0
0
0
0
0
0
0
0
3.7
5.9
11.2
17.4
0
0
0
0
0
0
0
0
3.2
6.4
11.3
18.8
24.5
0
0
0
0
0
0
0
3.2
6.3
11.1
17.4
23.7
33.9
0
0
0
0
0
0
3.1
5.2
9
16.8
27
35.7
42.8
0
0
0
0
0
2.6
5.2
8.6
14.4
21.8
33.9
49
62.6
0
0
0
0
2
3.7
6.5
10.7
19.3
29.6
59.7
62.8
67.1
0
0
0
1.2
2.4
4.5
8.1
13.2
20.5
49.8
69.1
65.6
70.3
0
0
0.9
1.5
2.6
4.6
9.2
16.1
29.2
57.9
70.1
72.3
72.8
0
0.7
0.9
1.6
3.1
5.9
10.5
19.1
34.9
64.1
66.7
37.1
49.4
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
40000 32000 24000 16000 8000 0
100 80 60 40 20 0
Software RAID0 over 6 Intel P3700 as blockdevice (Test ID 4) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
206923
0
0
0
0
0
0
0
0
0
0
0
214019
392579
0
0
0
0
0
0
0
0
0
0
208731
382884
624068
0
0
0
0
0
0
0
0
0
223526
381946
588274
519864
0
0
0
0
0
0
0
0
214620
360872
483662
524080
521244
0
0
0
0
0
0
0
220330
380314
586589
520981
515110
527686
0
0
0
0
0
0
221317
348773
596751
543713
514102
502928
513881
0
0
0
0
0
231618
387984
603250
552841
515966
539317
515994
524038
0
0
0
0
231512
401902
560821
659231
537669
517787
514155
578444
563847
0
0
0
184106
305156
496392
625451
693720
661053
582631
518455
540705
518158
0
0
91116
173669
316938
410778
516307
564033
650736
538140
517732
518389
513947
0
47836
93004
166866
270199
373458
343355
422672
584455
516227
524466
518548
524095
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000 450000 300000 150000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
12480
0
0
0
0
0
0
0
0
0
0
0
6624
6496
0
0
0
0
0
0
0
0
0
0
3152
4512
14400
0
0
0
0
0
0
0
0
0
1624
1912
14400
60160
0
0
0
0
0
0
0
0
868
1192
13632
23424
88576
0
0
0
0
0
0
0
458
498
7776
13248
56576
97792
0
0
0
0
0
0
219
290
2672
8512
19072
64256
123392
0
0
0
0
0
104
173
1560
4768
8768
21888
63744
171008
0
0
0
0
65
99
548
2128
4896
8640
21888
61696
142336
0
0
0
53
72
145
684
1944
4080
8096
18048
41216
107008
0
0
55
60
68
342
772
1736
2992
8032
16320
47360
126464
0
49
54
60
110
278
1864
2512
3056
8384
13504
57088
103936
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6
0
0
0
0
0
0
0
0
0
0
0
2.9
6.5
0
0
0
0
0
0
0
0
0
0
3.6
6.2
9.2
0
0
0
0
0
0
0
0
0
3
5.1
8.5
9.2
0
0
0
0
0
0
0
0
3.1
5
7.8
9.2
11.3
0
0
0
0
0
0
0
3.1
5.3
8.4
9.3
10.8
14.2
0
0
0
0
0
0
3.1
4.9
9.2
9.4
11.4
13.7
15.9
0
0
0
0
0
3
5.7
9.1
10
12
15.5
16.2
19.4
0
0
0
0
3
6
9.1
12.1
12.4
15.5
19.5
25.5
27.1
0
0
0
2.9
5
8.5
12.6
16.2
20.1
24.8
25
29.3
27.9
0
0
2.2
4.1
6.5
9.5
14.2
18.3
31
28
28
28.5
26.6
0
1.7
2.7
4.9
8.6
13.1
13.4
21
36.8
29.2
29.9
28.3
28.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
150000 120000 90000 60000 30000 0
100 80 60 40 20 0
Software RAID0 over 6 Intel P3700 as blockdevice (Test ID 4) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
62644
0
0
0
0
0
0
0
0
0
0
0
66156
116682
0
0
0
0
0
0
0
0
0
0
67399
125294
224355
0
0
0
0
0
0
0
0
0
73479
112667
195619
236183
0
0
0
0
0
0
0
0
68479
114375
203170
281882
295519
0
0
0
0
0
0
0
72792
116676
233910
308146
300141
295186
0
0
0
0
0
0
67376
120951
184836
266160
294057
297760
299085
0
0
0
0
0
48626
81808
132830
195373
260435
294353
294602
302675
0
0
0
0
28313
49092
83367
130018
187190
258845
293194
300918
302712
0
0
0
15234
27930
49588
82081
125707
185060
231477
290991
296206
295207
0
0
7891
15200
27681
48888
79187
121177
154929
218191
290400
293932
310693
0
3623
7885
14637
27093
45670
77238
116297
146365
213901
290110
361426
253913
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
320000 240000 160000 80000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
12352
0
0
0
0
0
0
0
0
0
0
0
6112
5920
0
0
0
0
0
0
0
0
0
0
3088
2992
9920
0
0
0
0
0
0
0
0
0
1096
1704
4576
28032
0
0
0
0
0
0
0
0
652
788
3472
15936
38656
0
0
0
0
0
0
0
243
350
1096
6496
14400
58112
0
0
0
0
0
0
143
139
474
2448
6368
17024
58112
0
0
0
0
0
60
82
89
516
1608
6560
21376
72192
0
0
0
0
48
67
67
139
474
1784
5536
17024
63744
0
0
0
49
54
60
89
237
580
1688
5280
15040
55552
0
0
65
49
61
75
98
390
1816
1544
5152
15040
44288
0
108
66
64
67
110
123
494
1832
1432
5088
11968
90624
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6
0
0
0
0
0
0
0
0
0
0
0
2.9
6.5
0
0
0
0
0
0
0
0
0
0
2.9
6.1
11
0
0
0
0
0
0
0
0
0
2.9
5.2
8.8
13.4
0
0
0
0
0
0
0
0
3.2
5
8.9
14.7
19.1
0
0
0
0
0
0
0
3.1
5
9.7
15.6
19.5
25.2
0
0
0
0
0
0
3
5.4
9.1
14.6
19.4
26.1
29.2
0
0
0
0
0
2.7
4.5
7.8
12
18.2
27
34.4
39.7
0
0
0
0
1.9
3.6
5.7
9.7
15.6
25
41
47.3
55
0
0
0
1.4
2.4
4
6.7
11.2
18.9
33.3
52.8
56.8
57.7
0
0
1
1.5
2.7
4.6
7.3
12.4
21.1
42.5
57.7
58.2
61.4
0
0.8
1
1.8
3.2
5.1
8.5
14.7
24.2
47.6
59.2
65.9
51.6
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
80000 60000 40000 20000 0
100 80 60 40 20 0
Software RAID0 over 6 Intel P3700 as blockdevice (Test ID 4) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
9.07848e+06
0
9.34122e+06 1.43006e+07
1e7
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
9.03958e+06 1.42129e+07 1.40092e+07
9.18488e+06 1.43054e+07 1.40216e+07 1.3857e+07
9.18313e+06 1.42627e+07 1.4161e+07 1.3989e+07 1.32198e+07
9.68938e+06 1.39565e+07 1.42731e+07 1.36755e+07 1.3693e+07 1.3295e+07
9.88114e+06 1.32079e+07 1.39853e+07 1.39088e+07 1.40605e+07 1.2755e+07 9.63368e+06
6.60206e+06 1.02212e+07 1.31428e+07 1.35285e+07 1.20695e+07 1.14416e+07 1.1581e+07 7.31923e+06
5.72397e+06 5.96721e+06 9.2858e+06 1.21035e+07 1.31762e+07 1.03859e+07 7.86234e+06 7.39082e+06 7.20214e+06
4.14751e+06 2.90906e+06 5.22003e+06 8.08593e+06 1.10645e+07 1.12653e+07 7.82411e+06 7.76633e+06 8.1927e+06 7.50469e+06
2.32763e+06 1.31068e+06 2.59594e+06 4.62249e+06 7.32577e+06 9.72879e+06 7.43458e+06 6.94268e+06 6.66785e+06 9.2858e+06 1.08953e+07 1.38161e+06
596544
1
2
0
1.29484e+06 2.38194e+06 4.11282e+06 6.21471e+06 8.09205e+06 7.24694e+06 7.98379e+06 1.02384e+07 1.08633e+07 9.43498e+06
4
8
16
32
64
128
number of workers
256
512
1024
2048
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
37632
0
0
0
0
0
0
0
0
0
0
0
15680
121344
0
0
0
0
0
0
0
0
0
0
11072
59136
140288
0
0
0
0
0
0
0
0
0
6048
33536
67072
317440
0
0
0
0
0
0
0
0
2288
18816
37120
189440
148480
0
0
0
0
0
0
0
1336
6944
20608
33024
82432
329728
0
0
0
0
0
0
804
1880
8096
20864
40704
136192
325632
0
0
0
0
0
588
964
2608
7328
22656
32640
280576
477184
0
0
0
0
564
780
1304
2704
6624
31360
53504
268288
610304
0
0
0
382
844
972
1864
2384
13504
24192
57600
107008
456704
0
0
310
964
996
2008
1720
3152
14272
41728
75264
268288
432128
0
326
996
972
1800
2128
1704
2960
17536
34560
66048
211968
403456
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.1
0
0
0
0
0
0
0
0
0
0
0
2.7
4.5
0
0
0
0
0
0
0
0
0
0
3.1
4.6
6.1
0
0
0
0
0
0
0
0
0
3.1
4.5
5.9
6.4
0
0
0
0
0
0
0
0
3.2
4.5
5.6
6.4
8.6
0
0
0
0
0
0
0
2.9
5.3
5.9
7.6
8
7.9
0
0
0
0
0
0
3
4.4
5.5
6.1
7.1
8.4
9.3
0
0
0
0
0
1.7
3
4.1
5.5
7.2
9
9.7
11.7
0
0
0
0
1.7
1.9
3.2
5.3
6.7
8.4
10.4
11.9
12.4
0
0
0
1.3
1.2
2
3.6
6.2
9.7
11.6
11.7
11.1
11.3
0
0
1
1.1
1.4
2.3
5
9.2
12.2
11.3
11.9
9
7.7
0
0.9
1
1.2
2
4.3
10.2
14.7
11.7
11.6
8.5
7.7
10.5
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1.25 1.00 0.75 0.50 0.25 0.00
600000
450000
300000
150000
0
100 80 60 40 20 0
Software RAID0 over 6 Intel P3700 as blockdevice (Test ID 4) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6602e+06
0
3.64582e+06 6.52348e+06
1e7
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3.65728e+06 6.4428e+06 1.10709e+07
3.68732e+06 7.53789e+06 9.85438e+06 9.84458e+06
6.311e+06 7.84955e+06 1.03398e+07 9.82013e+06 1.04653e+07
6.40453e+06 1.06134e+07 1.001e+07 9.92013e+06 9.34406e+06 9.18311e+06
6.89916e+06 1.01658e+07 1.08605e+07 9.36274e+06 1.00399e+07 1.02256e+07 1.06793e+07
8.26231e+06 9.67635e+06 1.09594e+07 9.84597e+06 8.894e+06 9.80637e+06 1.09276e+07 9.27174e+06
7.76604e+06 9.14794e+06 1.05032e+07 9.90164e+06 9.21684e+06 9.73511e+06 1.08749e+07 9.70847e+06 1.00416e+07
4.94848e+06 8.99683e+06 9.36777e+06 9.34818e+06 9.63847e+06 9.14348e+06 1.07847e+07 8.96661e+06 9.82312e+06 9.31257e+06
3.10388e+06 5.66015e+06 8.38917e+06 8.73797e+06 7.71602e+06 9.5918e+06 1.03782e+07 7.00343e+06 8.79484e+06 9.07611e+06 1.09542e+07
2
4
8
16
32
64
128
number of workers
0.8 0.6 0.4 0.2
0
1.67999e+06 2.97864e+06 5.75044e+06 8.27593e+06 6.8324e+06 9.03473e+06 6.72844e+06 8.17106e+06 7.83314e+06 1.024e+07 9.42364e+06 1.01893e+07
1
1.0
256
512
1024
2048
0.0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
75264
0
0
0
0
0
0
0
0
0
0
0
43776
51456
0
0
0
0
0
0
0
0
0
0
24960
26752
86528
0
0
0
0
0
0
0
0
0
10048
16064
103936
329728
0
0
0
0
0
0
0
0
7264
22400
51968
358400
342016
0
0
0
0
0
0
0
6432
17280
34048
140288
226304
489472
0
0
0
0
0
0
1528
10432
16320
47360
94720
301056
585728
0
0
0
0
0
1864
5344
8896
20352
54528
108032
354304
651264
0
0
0
0
209
2352
3536
10432
30336
42752
91648
301056
395264
0
0
0
185
548
2608
5280
13376
28032
27008
160768
346112
493568
0
0
139
151
708
2608
9920
13888
15808
64256
140288
329728
399360
0
117
153
139
900
5344
7904
19584
33536
59648
93696
325632
428032
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.6
0
0
0
0
0
0
0
0
0
0
0
2.6
5
0
0
0
0
0
0
0
0
0
0
2.6
4.9
9.3
0
0
0
0
0
0
0
0
0
2.7
4.9
8
8.2
0
0
0
0
0
0
0
0
2.9
4.9
6.1
7.4
8.4
0
0
0
0
0
0
0
2.7
4.7
6
6.9
7.1
6.9
0
0
0
0
0
0
2.6
4.2
5.5
5.4
6.5
7
10.1
0
0
0
0
0
2.5
3.7
4.9
4.9
5.1
6.3
9.1
12.3
0
0
0
0
2.5
3.8
4.4
4.7
5
6.3
11.2
9.1
15.8
0
0
0
2.3
3.6
4.2
4.7
6
7
11.7
9.2
10.8
10.6
0
0
1.5
2.6
4
4.5
4.1
7.1
10.9
5.9
9
9.4
12.1
0
1.1
1.5
2.8
4.3
4
6.7
5.2
8
7.5
13.2
9.6
11.8
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000
450000
300000
150000
0
100 80 60 40 20 0
Software RAID0 over 8 Intel P3700 as blockdevice (Test ID 5) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
251679
0
0
0
0
0
0
0
0
0
0
0
252832
497987
0
0
0
0
0
0
0
0
0
0
264500
467950
971782
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
264396
529049
758272
1.51576e+06
272642
474864
792606
1.55881e+06 2.14424e+06
254956
506299
870055
1.42338e+06 2.2754e+06 2.51348e+06
258174
459186
812393
1.24181e+06 2.13682e+06 2.50048e+06 2.46401e+06
225925
417168
686227
1.25092e+06 2.11509e+06 2.45003e+06 2.72287e+06 2.72562e+06
124857
247212
461848
816453
64179
128362
246547
455520
676832
967526
30381
64334
126318
240752
419714
510385
794743
12866
31637
64664
123960
225183
383722
508966
728593
1
2
4
8
16
32
64
128
1.36239e+06 1.79522e+06 2.68908e+06 2.76633e+06 2.52312e+06
1.44835e+06 2.7745e+06 2.46885e+06 2.606e+06
1.40726e+06 2.66736e+06 2.54009e+06 2.76659e+06
number of workers
3200000 2400000 1600000 800000
0
1.38179e+06 2.25326e+06 2.74678e+06 3.92723e+06
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
8640
0
0
0
0
0
0
0
0
0
0
0
4384
5344
0
0
0
0
0
0
0
0
0
0
2992
2640
2384
0
0
0
0
0
0
0
0
0
1192
1112
2384
13120
0
0
0
0
0
0
0
0
620
716
1384
4016
1352
0
0
0
0
0
0
0
382
358
660
2352
764
1160
0
0
0
0
0
0
239
253
462
1432
462
764
24448
0
0
0
0
0
169
195
274
596
540
470
852
24448
0
0
0
0
151
173
201
278
486
772
980
2384
772
0
0
0
141
149
169
195
580
1080
1064
644
684
652
0
0
157
147
151
193
241
1352
1656
980
668
644
1368
0
211
159
139
173
211
318
892
1704
1032
804
748
852
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.8
0
0
0
0
0
0
0
0
0
0
0
3.8
6.8
0
0
0
0
0
0
0
0
0
0
3.8
5.8
11.8
0
0
0
0
0
0
0
0
0
3.8
6.7
9.8
19.9
0
0
0
0
0
0
0
0
3.2
6
10
20.7
37.8
0
0
0
0
0
0
0
3.8
6.7
11.1
18.6
36
69.8
0
0
0
0
0
0
3.7
6
10.2
16.6
37.5
69.8
77
0
0
0
0
0
3.3
5.7
9.2
16.9
31.9
56
95.4
95.3
0
0
0
0
2.4
4.6
8.6
15.3
23.5
40
86.5
95.1
74.9
0
0
0
1.6
2.8
5.9
10.9
18.1
28.5
57.9
90.9
79.7
93.2
0
0
1.2
1.8
3.5
6.6
12.9
19.1
38
66.4
90.3
93.2
65
0
0.9
1.2
2
3.7
7.7
14.6
26.7
44.3
69.8
80.1
91.5
69.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
20000 15000 10000 5000 0
100 80 60 40 20 0
Software RAID0 over 8 Intel P3700 as blockdevice (Test ID 5) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
233394
0
0
0
0
0
0
0
0
0
0
0
234624
421245
0
0
0
0
0
0
0
0
0
0
247183
413003
848603
0
0
0
0
0
0
0
0
0
238443
445858
657935
1.21999e+06
0
0
0
0
0
0
0
0
235841
377008
699159
1.23499e+06 1.44861e+06
0
0
0
0
0
0
0
241126
409503
720497
1.20135e+06 1.45092e+06 1.47575e+06
0
0
0
0
0
0
247064
427996
630861
1.1777e+06 1.44732e+06 1.45221e+06 1.40043e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
246062
419119
611546
1.17533e+06 1.57242e+06 1.46747e+06 1.39331e+06 1.47502e+06
235359
417370
640892
1.19074e+06 1.69016e+06 1.72561e+06 1.43944e+06 1.55406e+06 1.44214e+06
223405
385460
639351
896890
97235
192169
356761
510513
722725
799420
897480
48678
108014
181782
313199
487159
511688
559399
785384
1
2
4
8
16
32
64
128
1.31662e+06 1.43888e+06 1.75916e+06 1.45101e+06 1.41951e+06 1.64842e+06
1.70697e+06 1.44153e+06 1.43875e+06 1.52342e+06
number of workers
0
1.4795e+06 1.42766e+06 1.45716e+06 1.47233e+06
256
512
1024
2048
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
9792
0
0
0
0
0
0
0
0
0
0
0
4896
5856
0
0
0
0
0
0
0
0
0
0
2672
3024
5472
0
0
0
0
0
0
0
0
0
1432
1480
3280
10432
0
0
0
0
0
0
0
0
756
948
1496
9280
21888
0
0
0
0
0
0
0
342
410
772
3248
13120
33536
0
0
0
0
0
0
193
207
470
1656
7328
15424
51456
0
0
0
0
0
90
114
262
900
3536
7328
17792
38144
0
0
0
0
60
70
123
402
1288
2928
6944
15552
39680
0
0
0
39
54
80
314
362
1064
1832
6432
17792
29312
0
0
52
57
68
117
330
1176
1800
1480
6432
15040
24192
0
53
40
59
90
125
532
1528
1832
2608
6368
14528
30080
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.5
0
0
0
0
0
0
0
0
0
0
0
3.1
6.8
0
0
0
0
0
0
0
0
0
0
3.2
6.1
11.5
0
0
0
0
0
0
0
0
0
3.5
6.2
9
16.9
0
0
0
0
0
0
0
0
3.5
5.8
9.6
17.2
23
0
0
0
0
0
0
0
3.4
5.5
9.7
17
22.9
32.9
0
0
0
0
0
0
3.2
5.8
9.5
16.6
23.3
33.4
41.4
0
0
0
0
0
3.2
5.7
9.1
16.6
26
37
45.2
52.8
0
0
0
0
3.4
5.7
8.9
18.1
25.7
43.1
49.4
59.2
56.2
0
0
0
3.3
5.7
10
15.7
24.5
33.9
60.7
56.5
59.6
66.3
0
0
2.4
4.6
7.8
13
20.9
28.4
43.9
69.1
62.9
63.7
66.3
0
1.8
2.9
5.4
9.8
17.4
25.3
34.1
50.2
65.7
63.5
64.5
64.8
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1500000 1200000 900000 600000 300000 0
50000 40000 30000 20000 10000 0
100 80 60 40 20 0
Software RAID0 over 8 Intel P3700 as blockdevice (Test ID 5) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
242385
0
0
0
0
0
0
0
0
0
0
0
239878
457306
0
0
0
0
0
0
0
0
0
0
244988
437849
804515
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
259682
466680
845870
1.46472e+06
255186
483897
794951
1.42922e+06 1.98924e+06
259576
498403
855671
1.37384e+06 1.97918e+06 2.01836e+06
254167
447558
709069
1.1806e+06 1.95469e+06 2.02083e+06 2.19289e+06
179768
345989
608471
964234
93897
182574
338922
607549
980447
46634
94492
176132
335249
563405
908922
19571
46254
90975
177296
330523
488575
616671
6716
21736
44442
89860
168390
303694
470964
568759
1
2
4
8
16
32
64
128
1.56405e+06 2.03776e+06 2.20727e+06 2.20558e+06
1.33026e+06 2.05698e+06 2.20228e+06 2.20435e+06
1.10617e+06 2.04704e+06 2.12802e+06 2.20254e+06
1.05314e+06 2.09337e+06 2.01461e+06 3.56329e+06
number of workers
3200000 2400000 1600000 800000
0
1.07148e+06 1.79868e+06 1.88257e+06 3.98804e+06
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
9536
0
0
0
0
0
0
0
0
0
0
0
6496
5536
0
0
0
0
0
0
0
0
0
0
3184
2768
6560
0
0
0
0
0
0
0
0
0
1160
1336
2192
23168
0
0
0
0
0
0
0
0
652
676
1176
6944
3280
0
0
0
0
0
0
0
354
370
820
3760
1976
5984
0
0
0
0
0
0
229
258
406
1272
1720
3280
44288
0
0
0
0
0
175
199
266
852
604
1112
4576
27008
0
0
0
0
163
175
219
282
740
1144
980
5344
31360
0
0
0
163
161
211
219
350
828
1496
932
4384
11712
0
0
213
163
181
201
229
772
2024
1384
892
4704
4320
0
262
195
189
189
233
302
580
2224
1336
1064
2416
3120
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.2
0
0
0
0
0
0
0
0
0
0
0
3.2
6.5
0
0
0
0
0
0
0
0
0
0
3.2
6
11.3
0
0
0
0
0
0
0
0
0
3.5
6.1
10.9
20
0
0
0
0
0
0
0
0
3.5
6.7
10.4
19.6
36.4
0
0
0
0
0
0
0
3.5
6.8
11.4
19.4
35.9
65
0
0
0
0
0
0
3.3
6
9.5
16.7
33.1
66.9
68.2
0
0
0
0
0
2.8
5.3
9.4
15.2
25.6
57.7
74.6
78.2
0
0
0
0
1.9
3.8
6.8
12.3
21.1
34.6
69.8
80.6
79
0
0
0
1.4
2.2
4.3
8
16.1
28.9
48.6
76.6
79.8
80.1
0
0
1.3
1.6
2.5
4.6
9.8
18.3
29.6
58.1
81.3
81.8
75.4
0
1.1
1
1.7
2.8
6.5
11.4
22.3
35.4
64.7
76.4
80.6
73.9
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
40000 32000 24000 16000 8000 0
100 80 60 40 20 0
Software RAID0 over 8 Intel P3700 as blockdevice (Test ID 5) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
208244
0
0
0
0
0
0
0
0
0
0
0
205894
358995
0
0
0
0
0
0
0
0
0
0
213649
366002
629461
0
0
0
0
0
0
0
0
0
225242
376327
619952
759079
0
0
0
0
0
0
0
0
222709
376778
630657
755303
737263
0
0
0
0
0
0
0
216474
380851
580176
754600
741498
721551
0
0
0
0
0
0
220630
371110
732469
755666
756363
729517
725412
0
0
0
0
0
216396
338198
763778
817894
760419
735907
736974
812977
0
0
0
0
221842
396194
655067
827156
816732
773240
735770
728310
720858
0
0
0
183966
312770
508670
739810
858804
883408
828085
734205
725112
748540
0
0
86434
170566
297079
457978
567591
711808
781120
834359
614997
727029
728779
0
44641
95478
156892
275145
387827
386359
478811
676968
700491
735570
724545
732132
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
750000 600000 450000 300000 150000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
12224
0
0
0
0
0
0
0
0
0
0
0
7008
6944
0
0
0
0
0
0
0
0
0
0
3408
3728
7264
0
0
0
0
0
0
0
0
0
1304
1784
4048
40192
0
0
0
0
0
0
0
0
780
964
1720
19584
75264
0
0
0
0
0
0
0
398
458
1112
12480
37632
113152
0
0
0
0
0
0
191
233
1720
8032
12224
48384
140288
0
0
0
0
0
110
167
708
4016
7904
16192
40704
87552
0
0
0
0
67
77
189
1624
4048
7904
15296
39680
127488
0
0
0
62
73
105
318
1368
3184
7392
14400
36096
111104
0
0
58
66
85
126
486
1208
2416
6944
17792
32640
95744
0
62
44
65
94
155
1208
1832
2448
7264
13248
34048
90624
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.4
0
0
0
0
0
0
0
0
0
0
0
3.4
5.9
0
0
0
0
0
0
0
0
0
0
3.1
6.3
9.7
0
0
0
0
0
0
0
0
0
3.4
6
9
13.2
0
0
0
0
0
0
0
0
3.2
5.6
9.2
13.2
16.4
0
0
0
0
0
0
0
3.2
5.5
9.3
13
16.6
20.7
0
0
0
0
0
0
3.1
5.4
10.5
13.6
16.3
20.9
25.2
0
0
0
0
0
3.2
5.3
10.7
14.5
17.4
22.4
27.6
34.1
0
0
0
0
3.2
5.7
10.2
15.3
19
24.1
30.9
35.2
37.7
0
0
0
3.1
5.4
9.8
14.8
20.4
27.9
35.3
37.6
40.7
43
0
0
2.3
4.4
7.5
12.3
18.8
27.5
39.4
41.2
34.6
43
42.5
0
1.6
2.7
5.1
9.2
15.5
19.1
29.8
45.3
43.8
44.4
43.7
40.6
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
125000 100000 75000 50000 25000 0
100 80 60 40 20 0
Software RAID0 over 8 Intel P3700 as blockdevice (Test ID 5) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
65963
0
0
0
0
0
0
0
0
0
0
0
61702
120077
0
0
0
0
0
0
0
0
0
0
67586
112956
218862
0
0
0
0
0
0
0
0
0
69801
130019
198035
351330
0
0
0
0
0
0
0
0
66898
112494
188597
341109
389926
0
0
0
0
0
0
0
64418
111217
192766
326011
390090
386102
0
0
0
0
0
0
67690
122195
188676
276408
372108
393840
386192
0
0
0
0
0
50184
82209
140497
210611
294026
376283
392766
391037
0
0
0
0
28391
50051
85288
134049
205657
293015
362559
395250
389097
0
0
0
14688
28325
50311
84171
132543
196984
259340
359919
391978
390562
0
0
6372
14800
28043
49047
83568
124091
168891
251835
360587
385506
384524
0
2344
7394
14739
26998
48042
79738
119574
163305
222707
351578
392171
383358
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
320000 240000 160000 80000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
9792
0
0
0
0
0
0
0
0
0
0
0
6752
6880
0
0
0
0
0
0
0
0
0
0
2448
3376
5664
0
0
0
0
0
0
0
0
0
1240
1496
2608
23936
0
0
0
0
0
0
0
0
732
844
1336
11200
31616
0
0
0
0
0
0
0
370
390
708
4384
10048
45312
0
0
0
0
0
0
147
143
278
1912
5024
11200
42752
0
0
0
0
0
63
113
98
227
1528
4384
10304
39680
0
0
0
0
55
74
99
165
338
1064
3024
10560
40192
0
0
0
71
52
72
105
195
438
1528
2992
10048
34048
0
0
133
57
53
89
109
233
1400
1624
3344
11968
31872
0
159
96
53
68
99
125
454
1256
1832
3088
11328
43264
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.1
0
0
0
0
0
0
0
0
0
0
0
3.6
6.2
0
0
0
0
0
0
0
0
0
0
3.3
5.9
10.8
0
0
0
0
0
0
0
0
0
3.3
6
9.2
17.3
0
0
0
0
0
0
0
0
3.6
5.6
8.9
16.9
23.8
0
0
0
0
0
0
0
3.4
5.4
9.3
16.9
24.5
34
0
0
0
0
0
0
3.3
5.7
9.2
15.3
23.7
35.4
45
0
0
0
0
0
2.8
5.2
8.4
13.7
21.8
35
53.3
60.5
0
0
0
0
2
3.9
6.6
11.4
18.3
30.1
52.4
58.4
64.2
0
0
0
1.5
2.4
4.4
7.5
13.6
23.3
40.3
60
66.9
64.7
0
0
1.4
1.6
2.7
4.9
8.8
18.1
27.4
44.6
58.9
67.7
67.7
0
1.2
1.1
1.6
3.4
5.8
10.2
18.2
33.4
52.5
67.1
66.5
66.4
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
40000 32000 24000 16000 8000 0
100 80 60 40 20 0
Software RAID0 over 8 Intel P3700 as blockdevice (Test ID 5) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
9.1472e+06
0
9.25028e+06 1.70142e+07
1e7
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
8.91483e+06 1.57426e+07 1.88383e+07
9.02593e+06 1.68685e+07 1.69925e+07 1.26002e+07
9.35814e+06 1.60769e+07 1.86092e+07 1.58752e+07 1.79597e+07
9.47305e+06 1.51893e+07 1.81688e+07 1.84488e+07 1.74004e+07 1.53418e+07
9.73257e+06 1.49358e+07 1.76519e+07 1.78217e+07 1.57286e+07 1.41152e+07 1.13973e+07
7.98636e+06 1.05793e+07 1.43995e+07 1.70406e+07 1.44996e+07 1.37579e+07 1.23425e+07 1.00142e+07
7.0202e+06 5.76529e+06 9.40406e+06 1.41105e+07 1.67168e+07 1.4463e+07 9.01164e+06 1.02211e+07 1.00895e+07
3.15494e+06 2.76747e+06 5.10545e+06 8.61453e+06 1.24737e+07 1.5029e+07 9.00069e+06 9.24496e+06 8.74384e+06 9.55521e+06
1.75117e+06 1.26228e+06 2.50826e+06 4.81178e+06 7.98248e+06 1.07938e+07 1.33469e+07 9.54196e+06 1.25887e+07 9.16071e+06 1.01821e+07 1.64489e+06
583040
1
2
8
16
32
64
128
number of workers
256
512
1024
2048
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
43264
0
0
0
0
0
0
0
0
0
0
0
21632
32128
0
0
0
0
0
0
0
0
0
0
10560
17024
195584
0
0
0
0
0
0
0
0
0
4960
6176
68096
166912
0
0
0
0
0
0
0
0
2992
3312
30592
62208
246784
0
0
0
0
0
0
0
1560
2040
15808
34048
146432
288768
0
0
0
0
0
0
804
1336
4448
13248
25472
122368
142336
0
0
0
0
0
628
860
2640
5344
23424
51456
185344
610304
0
0
0
0
516
724
1656
1960
3504
13888
45824
220160
419840
0
0
0
378
684
1480
1768
1800
4448
24704
108032
234496
602112
0
0
334
684
1528
1784
1224
2160
5408
32128
43776
329728
577536
0
290
684
1592
1976
1176
1720
3024
13504
41216
185344
448512
460800
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.9
0
0
0
0
0
0
0
0
0
0
0
2.9
5.6
0
0
0
0
0
0
0
0
0
0
2.9
5.2
7.6
0
0
0
0
0
0
0
0
0
2.9
5.6
9.1
8.9
0
0
0
0
0
0
0
0
3
5.3
8.2
9.5
10.1
0
0
0
0
0
0
0
2.9
5.3
8.1
9
11.2
12.8
0
0
0
0
0
0
3.2
5.3
8.9
8.5
10
12.2
14.8
0
0
0
0
0
2.4
2.9
5.4
8.5
9.2
12.1
13.8
14.7
0
0
0
0
1.9
2.1
3.4
6
9.7
11.5
14.4
14.7
15.7
0
0
0
1.8
1.4
2.2
3.5
6.8
11
15
15.7
14.9
14.9
0
0
1.3
1.2
1.6
2.4
4.9
11.3
17.4
15.3
15.8
15.5
15.1
0
0.8
1.1
1.7
2
4.2
10
16.5
15
15.2
15.8
14.3
12.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1.2 0.8 0.4
0
1.12051e+06 2.39569e+06 4.3537e+06 6.91626e+06 9.01007e+06 1.02473e+07 9.31892e+06 9.29855e+06 9.48815e+06 1.09666e+07
4
1.6
0.0
600000
450000
300000
150000
0
100 80 60 40 20 0
Software RAID0 over 8 Intel P3700 as blockdevice (Test ID 5) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
3.51633e+06
0
3.22004e+06 5.89023e+06
1e7
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3.59136e+06 6.61819e+06 1.13045e+07
3.22189e+06 7.29792e+06 1.14391e+07 1.44006e+07
5.2031e+06 9.87188e+06 1.31304e+07 1.46332e+07 1.40294e+07
4.91959e+06 1.15432e+07 1.46754e+07 1.45359e+07 8.79895e+06 1.14464e+07
6.17391e+06 1.11629e+07 1.46984e+07 1.45793e+07 9.1388e+06 1.21272e+07 1.34992e+07
7.38984e+06 1.1837e+07 1.40233e+07 1.26976e+07 1.21275e+07 1.3903e+07 1.16851e+07 1.33367e+07
7.07789e+06 1.09002e+07 1.31022e+07 8.93424e+06 1.32614e+07 1.36072e+07 1.38241e+07 1.29494e+07 1.41335e+07
4.13458e+06 9.59496e+06 1.16367e+07 1.09608e+07 1.29587e+07 1.39552e+07 1.19708e+07 1.22856e+07 1.13808e+07 1.39046e+07
2.17658e+06 5.46781e+06 9.77123e+06 1.13694e+07 1.23133e+07 1.3312e+07 1.12788e+07 1.02853e+07 1.07977e+07 1.42985e+07 1.38256e+07
0
1.586e+06 2.68189e+06 5.42941e+06 9.33484e+06 1.13098e+07 1.16402e+07 1.14994e+07 1.00419e+07 1.27283e+07 1.3871e+07 1.40208e+07 1.35821e+07
1
2
4
8
16
32
64
128
number of workers
256
512
1024
2048
1.25 1.00 0.75 0.50 0.25 0.00
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
77312
0
0
0
0
0
0
0
0
0
0
0
45824
49408
0
0
0
0
0
0
0
0
0
0
20608
28544
37632
0
0
0
0
0
0
0
0
0
11072
13504
20608
268288
0
0
0
0
0
0
0
0
4960
11968
62208
154624
370688
0
0
0
0
0
0
0
1992
10688
24960
67072
468992
497664
0
0
0
0
0
0
980
8640
13120
29312
201728
252928
403456
0
0
0
0
0
370
1624
7200
28032
46336
128512
391168
634880
0
0
0
0
223
900
2672
17536
16512
34048
101888
264192
561152
0
0
0
278
175
1064
7776
8896
13632
46848
111104
337920
419840
0
0
255
167
247
1784
3824
7200
28544
57088
148480
189440
456704
0
175
153
151
676
1608
7008
13376
31872
36096
54016
173056
370688
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.7
0
0
0
0
0
0
0
0
0
0
0
2.6
5
0
0
0
0
0
0
0
0
0
0
2.7
4.9
8.8
0
0
0
0
0
0
0
0
0
2.7
4.7
9
11.1
0
0
0
0
0
0
0
0
2.9
5.1
8.1
12.5
11
0
0
0
0
0
0
0
2.9
5
7.9
11.1
6.1
10.2
0
0
0
0
0
0
2.8
4.7
7
8.4
6.2
8.7
12.2
0
0
0
0
0
2.6
4.5
6.4
5.9
6.2
8.1
8.9
12.1
0
0
0
0
2.7
4.6
5.9
4.5
6.8
8.1
11.9
12.6
14.8
0
0
0
2.3
4
5.8
5.7
6.9
9.2
9.9
11.6
10.2
19.1
0
0
1.7
2.9
4.8
6.1
6.9
9.6
9.4
8.9
9.7
17.3
19
0
1
1.6
2.8
4.8
6.4
7
9.7
8.6
13
16.3
17
21
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000
450000
300000
150000
0
100 80 60 40 20 0
Software RAID1 over 2 Intel P3700 as blockdevice (Test ID 6) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
203376
0
0
0
0
0
0
0
0
0
0
0
201420
435565
0
0
0
0
0
0
0
0
0
0
284856
347778
772236
0
0
0
0
0
0
0
0
0
205866
378022
701657
915777
0
0
0
0
0
0
0
0
280796
416857
687902
911604
460065
0
0
0
0
0
0
0
281608
386418
640806
916870
452996
407464
0
0
0
0
0
0
266256
331012
734865
924568
447095
404441
385544
0
0
0
0
0
157146
282255
524081
800134
477034
352071
388218
391274
0
0
0
0
82088
164022
299660
520899
741253
423806
382022
385341
467977
0
0
0
41811
85230
160603
305128
512607
574358
385298
385161
385369
388980
0
0
21078
43290
83864
164544
294185
430069
526585
384373
385282
386394
697835
0
9545
21747
42862
85859
158083
258680
324448
514453
470510
394549
387861
598898
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
10432
0
0
0
0
0
0
0
0
0
0
0
7456
5984
0
0
0
0
0
0
0
0
0
0
2008
3440
3344
0
0
0
0
0
0
0
0
0
1480
1640
1880
6368
0
0
0
0
0
0
0
0
812
860
932
2160
5728
0
0
0
0
0
0
0
438
478
588
3280
3056
6432
0
0
0
0
0
0
302
370
1004
604
1592
3408
28288
0
0
0
0
0
266
219
270
354
884
1976
25216
45312
0
0
0
0
258
165
227
258
532
1208
17024
24960
84480
0
0
0
255
147
197
181
249
1672
8640
22144
20608
23424
0
0
258
135
179
157
195
748
1784
2736
5472
14912
7968
0
294
133
151
139
177
346
1928
1288
1768
5472
15040
10688
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.3
0
0
0
0
0
0
0
0
0
0
0
3.3
5.4
0
0
0
0
0
0
0
0
0
0
2.4
5.8
10
0
0
0
0
0
0
0
0
0
3.4
5.9
10.6
16.2
0
0
0
0
0
0
0
0
2.4
5.8
10.5
16.1
34.4
0
0
0
0
0
0
0
2.4
5.9
10.2
16.9
35.1
66.9
0
0
0
0
0
0
2.4
5
9.6
12.6
35.5
67.5
95.7
0
0
0
0
0
2
4.4
6.7
11.6
31.4
55.6
93.2
94.9
0
0
0
0
1.3
2.4
4.6
9.3
16.9
54.9
95.7
95
49.6
0
0
0
0.9
1.6
3
5.7
11.7
17
92.5
94.5
95.2
94.9
0
0
0.7
1
1.7
3.4
7.2
12.2
25.8
94.6
95.4
95.3
96.5
0
0.6
0.6
1.1
1.9
4.1
8.4
13.1
48
55
94.4
86.3
95.4
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
75000 60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID1 over 2 Intel P3700 as blockdevice (Test ID 6) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
156344
0
0
0
0
0
0
0
0
0
0
0
184408
148382
0
0
0
0
0
0
0
0
0
0
190133
179263
178979
0
0
0
0
0
0
0
0
0
190156
177662
179669
182114
0
0
0
0
0
0
0
0
183508
176672
184450
186665
176148
0
0
0
0
0
0
0
177524
179379
197024
186521
177189
175914
0
0
0
0
0
0
176507
179584
174105
183363
181251
178293
182518
0
0
0
0
0
166453
119920
187674
183149
179237
179504
184566
181254
0
0
0
0
152984
178220
208825
191646
177563
183795
184491
183257
183378
0
0
0
125552
170656
186530
204360
191261
184195
183627
187381
185952
181846
0
0
83957
122354
158275
178969
193656
186224
186383
182745
184414
184466
183921
0
47995
82580
111280
136391
163632
178603
182158
181099
184636
184596
183404
186052
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
200000 160000 120000 80000 40000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
21376
0
0
0
0
0
0
0
0
0
0
0
12992
33024
0
0
0
0
0
0
0
0
0
0
7072
15168
28288
0
0
0
0
0
0
0
0
0
3728
8896
15552
39680
0
0
0
0
0
0
0
0
2704
5920
9408
20096
45824
0
0
0
0
0
0
0
1640
3632
5536
10432
34560
47872
0
0
0
0
0
0
900
2256
3568
6432
17024
34048
52992
0
0
0
0
0
540
2192
2320
3664
6752
16192
28032
59648
0
0
0
0
233
932
1144
2096
3600
10304
19072
34560
55040
0
0
0
96
243
668
1144
2128
5728
14272
24960
40704
59648
0
0
77
107
326
724
1192
3248
8160
12096
27520
43776
89600
0
76
81
171
454
740
1576
2992
4768
9152
15424
35072
61184
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.1
0
0
0
0
0
0
0
0
0
0
0
2.6
3
0
0
0
0
0
0
0
0
0
0
2.7
3.1
3.9
0
0
0
0
0
0
0
0
0
2.7
3.1
3.8
4.9
0
0
0
0
0
0
0
0
2.7
3.1
3.9
5.1
5.4
0
0
0
0
0
0
0
2.7
3.1
4.2
5.7
5.4
10.9
0
0
0
0
0
0
2.6
3.2
3.7
5.6
5.9
10
46.7
0
0
0
0
0
2.6
2.3
4.1
5.2
7
16.7
59
61.3
0
0
0
0
2.5
3.3
4.7
6.1
8.1
16
61
63.8
64.8
0
0
0
2.3
3.3
4.3
5.6
7.9
14.1
56.3
66.7
68
63.3
0
0
2
3
4.1
5.6
7.9
13.3
50.1
59.1
67.3
66.4
64.7
0
1.5
2.3
3.4
5.3
7.1
9.5
19.8
49.1
60.8
64.6
64.3
63.5
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
75000 60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID1 over 2 Intel P3700 as blockdevice (Test ID 6) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
247291
0
0
0
0
0
0
0
0
0
0
0
251135
385454
0
0
0
0
0
0
0
0
0
0
180964
363250
441390
0
0
0
0
0
0
0
0
0
194394
407673
502314
539315
0
0
0
0
0
0
0
0
195873
364301
537821
539566
456640
0
0
0
0
0
0
0
193078
357300
471161
538430
509086
542079
0
0
0
0
0
0
227702
359338
449738
536141
537504
295432
381546
0
0
0
0
0
133585
224665
359097
493909
518006
401572
387217
382856
0
0
0
0
72336
129856
222074
359387
495375
517812
382513
383465
383012
0
0
0
37736
69522
125038
221176
360488
482292
396100
386567
382902
383397
0
0
19234
36100
67026
124136
219158
334535
435296
385335
384739
385435
423432
0
9571
18477
35048
66259
124025
212975
313198
439177
389856
402518
408286
814492
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
9536
0
0
0
0
0
0
0
0
0
0
0
4704
7008
0
0
0
0
0
0
0
0
0
0
4128
3344
50944
0
0
0
0
0
0
0
0
0
1704
1832
23168
9536
0
0
0
0
0
0
0
0
796
1048
2224
5536
11072
0
0
0
0
0
0
0
498
716
6368
2576
7328
22144
0
0
0
0
0
0
258
422
2960
1272
2352
4320
28544
0
0
0
0
0
199
314
756
948
1704
2160
21632
41728
0
0
0
0
173
278
644
740
740
2800
17024
40704
77312
0
0
0
163
266
548
580
446
1112
3696
10816
26240
14400
0
0
153
266
410
482
366
868
1464
2256
4576
5280
18816
0
155
233
314
462
334
398
1240
1416
2800
6240
4448
19328
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.4
0
0
0
0
0
0
0
0
0
0
0
2.4
5.6
0
0
0
0
0
0
0
0
0
0
3.4
5.9
8.4
0
0
0
0
0
0
0
0
0
3.3
5.2
8.3
9.4
0
0
0
0
0
0
0
0
3.4
5.8
7.4
10
28.6
0
0
0
0
0
0
0
3.4
5.5
8.3
9.8
17.8
13.5
0
0
0
0
0
0
2.3
4.8
7.4
9.1
13.3
55.3
95.2
0
0
0
0
0
1.9
3.9
5.8
9.1
11.7
61.7
94.9
94.9
0
0
0
0
1.3
2.1
3.8
6.8
11.6
16.2
95.3
95.3
95
0
0
0
0.9
1.4
2.3
4.5
8.9
13
85.9
92.8
95.2
94.9
0
0
0.6
0.9
1.4
2.8
6
10
15.8
92.3
94
94.9
52.1
0
0.5
0.6
1
1.5
3.4
6.9
11.1
18.8
93.6
87.5
89.9
58.5
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
750000 600000 450000 300000 150000 0
75000 60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID1 over 2 Intel P3700 as blockdevice (Test ID 6) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
90042
0
0
0
0
0
0
0
0
0
0
0
91699
91735
0
0
0
0
0
0
0
0
0
0
90565
91896
90801
0
0
0
0
0
0
0
0
0
98982
88818
89985
91028
0
0
0
0
0
0
0
0
80486
75330
90611
90329
91745
0
0
0
0
0
0
0
97146
83223
98278
90235
93339
91245
0
0
0
0
0
0
99970
96515
93925
92330
95183
92316
92058
0
0
0
0
0
101330
100400
100177
94372
91917
92813
90295
94004
0
0
0
0
93922
101661
96516
92756
92772
90816
89730
94947
94783
0
0
0
82480
94870
99634
97213
93987
94271
91210
89394
100210
94323
0
0
59515
82628
92584
95902
92930
93282
95563
92313
94107
97142
102240
0
37210
56480
72627
88940
93103
92697
93506
91721
96068
92008
92112
93170
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
100000 80000 60000 40000 20000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
47360
0
0
0
0
0
0
0
0
0
0
0
27008
47872
0
0
0
0
0
0
0
0
0
0
16064
22912
55552
0
0
0
0
0
0
0
0
0
10944
15936
28544
55040
0
0
0
0
0
0
0
0
7200
11328
16320
31104
77312
0
0
0
0
0
0
0
3472
6816
10688
19072
40704
132096
0
0
0
0
0
0
1976
3568
6368
12352
23424
67072
183296
0
0
0
0
0
1192
2024
3440
6496
11968
27264
98816
193536
0
0
0
0
860
1240
2128
3728
6496
12224
28544
122368
284672
0
0
0
406
852
1288
2096
3728
6496
12224
40192
125440
250880
0
0
141
422
876
1320
2256
3792
6624
12224
31616
104960
236544
0
91
219
564
796
1288
2320
3728
6624
11328
44288
49920
118272
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3
0
0
0
0
0
0
0
0
0
0
0
2
2.3
0
0
0
0
0
0
0
0
0
0
1.7
2
2.6
0
0
0
0
0
0
0
0
0
1.8
1.9
2.4
3.1
0
0
0
0
0
0
0
0
1.6
1.7
2.4
3
4.5
0
0
0
0
0
0
0
1.8
2
2.7
3
3.9
5.1
0
0
0
0
0
0
1.9
2.2
2.6
3.3
3.8
5.4
7
0
0
0
0
0
1.9
2.2
2.7
3.2
4.5
5.9
6.6
7.1
0
0
0
0
1.9
2.3
2.7
3.2
4.5
6.5
9.3
11
12.2
0
0
0
1.8
2.4
2.8
3.3
4.4
7.1
11.1
10.4
14.3
12.2
0
0
1.5
2.2
2.9
3.4
4.8
7.4
10.6
12.2
14.1
12
14.1
0
1.2
1.8
2.4
3.3
4.7
7.3
10.3
10.9
15.2
12
12.2
12.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
250000 200000 150000 100000 50000 0
100 80 60 40 20 0
Software RAID1 over 2 Intel P3700 as blockdevice (Test ID 6) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
60150
0
0
0
0
0
0
0
0
0
0
0
64125
64819
0
0
0
0
0
0
0
0
0
0
64174
64801
64448
0
0
0
0
0
0
0
0
0
61179
65448
64752
64964
0
0
0
0
0
0
0
0
55781
67256
65573
65516
65281
0
0
0
0
0
0
0
49929
62760
66181
66553
64462
65343
0
0
0
0
0
0
35879
48422
62967
66060
65412
65358
64924
0
0
0
0
0
23620
34791
49297
63544
66832
65970
64724
64416
0
0
0
0
13508
23142
34124
49030
62768
65888
64214
64948
64769
0
0
0
7000
13799
22757
34420
48426
62758
65980
65472
65377
65163
0
0
3546
6918
13650
22644
34238
48597
60279
65395
64871
64822
64873
0
2386
4595
8714
14314
22224
32815
44532
57765
63878
65281
66309
65645
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
17024
0
0
0
0
0
0
0
0
0
0
0
11456
24448
0
0
0
0
0
0
0
0
0
0
8640
14144
26240
0
0
0
0
0
0
0
0
0
6688
9536
15680
34560
0
0
0
0
0
0
0
0
3856
5920
9664
16512
43776
0
0
0
0
0
0
0
932
2320
6560
11200
21120
52480
0
0
0
0
0
0
270
924
2448
6240
10816
31360
82432
0
0
0
0
0
74
310
852
2352
6176
12480
34048
82432
0
0
0
0
118
72
366
948
2288
6176
13248
35584
116224
0
0
0
127
85
83
239
940
2384
6432
13376
34560
115200
0
0
131
110
91
87
310
836
2704
6240
13248
49408
59136
0
77
89
76
83
109
580
1320
2672
6432
13248
46848
71168
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.8
0
0
0
0
0
0
0
0
0
0
0
2.8
3.9
0
0
0
0
0
0
0
0
0
0
2.8
3.6
4.2
0
0
0
0
0
0
0
0
0
2.6
3.4
4.3
5
0
0
0
0
0
0
0
0
2.5
3.4
4.2
4.9
6.2
0
0
0
0
0
0
0
2.3
3.3
4.1
5
6
7
0
0
0
0
0
0
2
3
4.1
5
6.5
6.9
7.6
0
0
0
0
0
1.5
2.3
3.4
4.7
6.1
7.3
8.4
11.2
0
0
0
0
1.1
1.7
2.6
3.9
6
7.5
9.7
12.5
12.9
0
0
0
0.8
1.3
1.9
3
4.8
7.7
12
13.1
14.3
14.1
0
0
0.6
0.9
1.3
2.1
3.9
6.5
11.1
13.1
13.6
14.7
14.7
0
0.5
0.7
1.1
1.5
2.5
5
9.4
11.7
13.7
14.8
15.5
16.2
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
100000 80000 60000 40000 20000 0
100 80 60 40 20 0
Software RAID1 over 2 Intel P3700 as blockdevice (Test ID 6) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
2.39192e+06
0
2.4035e+06 4.79188e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2.40596e+06 4.78653e+06 4.75701e+06
2.40761e+06 4.79615e+06 4.76181e+06 4.78597e+06
2.40684e+06 4.80041e+06 4.50228e+06 4.78941e+06 4.22424e+06
2.40646e+06 4.79268e+06 4.66683e+06 4.77984e+06 4.39899e+06 3.50164e+06
2.40451e+06 4.79991e+06 4.73791e+06 4.74678e+06 4.27963e+06 4.42566e+06 3.38937e+06
2.37224e+06 4.76038e+06 4.73853e+06 4.74493e+06 4.44299e+06 3.61598e+06 3.84612e+06 3.76766e+06
1.99124e+06 3.98541e+06 4.56838e+06 4.71696e+06 4.12208e+06 3.97283e+06 3.42609e+06 3.73141e+06 3.31105e+06
1.37678e+06 2.75236e+06 3.66936e+06 4.49555e+06 4.54882e+06 3.91444e+06 3.62125e+06 3.50552e+06 3.39381e+06 3.09103e+06
1.05371e+06 2.08364e+06 2.44788e+06 3.57268e+06 4.25618e+06 3.81066e+06 3.4381e+06 3.49217e+06 3.05887e+06 3.11956e+06 2.9703e+06 1.179e+06
800166
1
2
8
16
32
64
128
number of workers
3000000 2000000 1000000
0
1.34338e+06 2.18921e+06 3.31295e+06 4.26688e+06 3.73526e+06 3.38767e+06 3.60681e+06 3.76087e+06 2.99909e+06 3.04155e+06
4
4000000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
252928
0
0
0
0
0
0
0
0
0
0
0
142336
140288
0
0
0
0
0
0
0
0
0
0
47872
74240
162816
0
0
0
0
0
0
0
0
0
22400
24448
71168
140288
0
0
0
0
0
0
0
0
13248
22144
35072
76288
193536
0
0
0
0
0
0
0
6560
8384
14656
33024
78336
370688
0
0
0
0
0
0
3248
3568
7200
14528
36608
81408
518144
0
0
0
0
0
1800
1784
3888
7392
16320
48384
179200
374784
0
0
0
0
996
988
2768
4128
9280
16768
53504
175104
618496
0
0
0
652
652
2320
2672
4640
9408
17536
54016
240640
569344
0
0
716
732
2288
2320
3056
5216
10048
18560
57600
214016
684032
0
438
676
2224
2384
2704
3088
5088
11584
21632
63744
284672
296960
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
1.7
0
0
0
0
0
0
0
0
0
0
0
1.2
2.3
0
0
0
0
0
0
0
0
0
0
1
1.8
2.1
0
0
0
0
0
0
0
0
0
1
1.7
2.1
2.2
0
0
0
0
0
0
0
0
0.9
1.7
2
2.1
2.5
0
0
0
0
0
0
0
0.9
1.6
2
2
3.3
4.6
0
0
0
0
0
0
0.9
1.4
1.8
1.8
3.2
2.4
3.9
0
0
0
0
0
0.7
1.2
1.4
1.6
2.1
4.1
3.1
3.8
0
0
0
0
0.6
1
1.4
1.6
3
2.3
3.6
3.4
4
0
0
0
0.6
0.8
1.2
1.6
2.1
2.5
2.5
3.7
3.6
4.9
0
0
0.6
0.7
1
1.4
2.9
3.2
3.2
3
4.1
4.2
5
0
0.5
0.8
0.9
1.2
1.7
3.4
4.1
3.4
3.1
2.8
5
5.5
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000 450000 300000 150000 0
100 80 60 40 20 0
Software RAID1 over 2 Intel P3700 as blockdevice (Test ID 6) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
1.77718e+06
0
1.92082e+06 1.90446e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1.89409e+06 1.90736e+06 1.62386e+06
1.89539e+06 1.91269e+06 1.79332e+06 1.90704e+06
1.89212e+06 1.90335e+06 1.93e+06 1.91836e+06 1.60311e+06
1.87699e+06 1.84531e+06 1.63445e+06 1.92047e+06 1.53903e+06 1.59796e+06
1.85932e+06 1.84009e+06 1.64259e+06 1.90811e+06 1.565e+06 1.60315e+06 1.44054e+06
1.81922e+06 1.78892e+06 1.62301e+06 1.90815e+06 1.60870e+06 1.5377e+06 1.73594e+06 1.81238e+06
1.7273e+06 1.57852e+06 1.602e+06 1.5804e+06 1.5423e+06 1.63719e+06 1.75064e+06 1.73259e+06 1.64949e+06
1.63938e+06 1.19656e+06 1.45775e+06 1.13089e+06 1.53292e+06 1.34095e+06 1.78998e+06 1.36001e+06 1.65164e+06 1.61248e+06 1.68412e+06 1.41303e+06 1.65521e+06 1.02164e+06 1.49639e+06 1.16408e+06 1.69924e+06
750828
1.69539e+06 1.64883e+06 1.66424e+06
2
4
8
16
32
64
128
number of workers
256
512
1024
2048
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
358400
0
0
0
0
0
0
0
0
0
0
0
134144
329728
0
0
0
0
0
0
0
0
0
0
54016
142336
561152
0
0
0
0
0
0
0
0
0
35072
78336
148480
350208
0
0
0
0
0
0
0
0
21888
45312
76288
166912
468992
0
0
0
0
0
0
0
15936
28032
44800
74240
242688
897024
0
0
0
0
0
0
10560
15936
28032
40192
120320
395264
880640
0
0
0
0
0
4016
10304
15424
20608
55552
144384
309248
452608
0
0
0
0
2576
6880
8896
20608
33024
54528
261120
366592
1.51552e+06
0
0
0
1768
6048
10176
14016
17792
45824
40704
228352
593920
872448
0
0
1032
2800
2960
7712
11328
22144
33536
166912
179200
585728
1.1223e+06
0
466
1848
1800
3280
4640
12992
15168
41728
47360
423936
544768
921600
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
1.9
0
0
0
0
0
0
0
0
0
0
0
1.8
2.7
0
0
0
0
0
0
0
0
0
0
1.7
2.8
2.7
0
0
0
0
0
0
0
0
0
1.9
2.6
2.6
3.4
0
0
0
0
0
0
0
0
1.2
1.6
2.8
3.4
3.2
0
0
0
0
0
0
0
1.1
1.2
1.7
3.1
2.7
3
0
0
0
0
0
0
1.1
1.2
1.4
2.7
2.7
2.7
3
0
0
0
0
0
1
1.1
1.2
1.5
2.2
2.5
3.4
4.4
0
0
0
0
1
1
1.2
1.3
1.7
2.4
3.2
3.8
4.8
0
0
0
1
0.9
1.1
1.1
1.4
1.3
3
2.4
4.1
5.3
0
0
1
1
1.3
1
1.2
1.1
2.4
1.5
3.7
4.4
5.8
0
0.9
1
1.3
1.2
1.4
1.2
1.7
1.3
3.3
3.8
4.5
5.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1200000 800000 400000
0
1.55821e+06 1.48413e+06 1.68474e+06 1.38626e+06 1.7609e+06 1.36633e+06 1.66844e+06 1.11086e+06 1.80478e+06 1.63875e+06 1.68718e+06 1.65238e+06
1
1600000
0
1500000 1200000 900000 600000 300000 0
100 80 60 40 20 0
Software RAID10 over 4 Intel P3700 as blockdevice (Test ID 7) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
184429
0
0
0
0
0
0
0
0
0
0
0
190146
351067
0
0
0
0
0
0
0
0
0
0
246810
390924
649882
0
0
0
0
0
0
0
0
0
195376
394624
681306
990851
0
0
0
0
0
0
0
0
195775
370186
691041
936585
433123
0
0
0
0
0
0
0
237986
363828
610218
991175
445409
390534
0
0
0
0
0
0
215296
312883
495712
955154
436879
397409
379934
0
0
0
0
0
157053
276172
578332
913708
441208
387651
376742
376162
0
0
0
0
83228
160874
316854
529314
575906
335554
377629
378199
377950
0
0
0
41991
83145
166140
306798
559055
657252
381887
377989
378648
377243
0
0
20193
41889
85443
162292
312588
518565
639122
378990
383145
378114
579521
0
10809
19822
42746
80648
158411
264658
440091
643053
383344
384971
815366
767200
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
15680
0
0
0
0
0
0
0
0
0
0
0
6240
6880
0
0
0
0
0
0
0
0
0
0
2320
3344
5600
0
0
0
0
0
0
0
0
0
1528
1560
2992
10176
0
0
0
0
0
0
0
0
836
916
988
1416
6752
0
0
0
0
0
0
0
470
524
828
2384
3312
7008
0
0
0
0
0
0
302
386
2192
1352
1832
3728
27776
0
0
0
0
0
255
262
229
612
980
2256
21376
53504
0
0
0
0
229
209
197
498
908
1928
17024
48384
33024
0
0
0
207
221
183
253
278
1096
6112
17280
9280
24704
0
0
233
217
147
213
205
438
1448
2544
2992
5536
4256
0
193
253
145
223
203
346
700
1640
2608
4960
3728
5024
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6
0
0
0
0
0
0
0
0
0
0
0
3.6
6.2
0
0
0
0
0
0
0
0
0
0
2.7
5.7
10.2
0
0
0
0
0
0
0
0
0
3.5
6.4
10.9
18.9
0
0
0
0
0
0
0
0
3.6
6.4
11.5
20.3
36.7
0
0
0
0
0
0
0
2.7
5.6
10
18.6
37.3
67
0
0
0
0
0
0
3
4.7
9
18.1
37.8
67.8
95.2
0
0
0
0
0
2.5
4.8
9.1
15.7
35.8
68.7
94.9
87.3
0
0
0
0
1.7
3.4
6.3
11.2
29.5
57.7
95.5
93.2
95.1
0
0
0
1.2
2.1
4
7.6
14.2
26
91.2
94
94.2
95
0
0
0.9
1.3
2.4
4.5
8.6
15.8
30
94.2
94.4
94.7
96.9
0
0.5
0.9
1.5
2.9
5.5
9.9
17.6
35.3
94.3
94.8
96.5
97.3
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
50000 40000 30000 20000 10000 0
100 80 60 40 20 0
Software RAID10 over 4 Intel P3700 as blockdevice (Test ID 7) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
123464
0
0
0
0
0
0
0
0
0
0
0
129688
218750
0
0
0
0
0
0
0
0
0
0
125185
194271
346505
0
0
0
0
0
0
0
0
0
132897
180476
368140
350486
0
0
0
0
0
0
0
0
132614
213024
339596
355178
284700
0
0
0
0
0
0
0
133890
176686
324519
357152
281736
269092
0
0
0
0
0
0
135579
180052
311634
402811
291173
262650
257667
0
0
0
0
0
138754
186132
291220
351061
298813
266381
269064
263112
0
0
0
0
130108
170378
332467
368368
294053
264304
250423
263077
305600
0
0
0
120934
189767
296811
366308
259247
259203
256110
262939
270153
261431
0
0
73318
131894
219456
258520
345642
310962
275027
253859
264079
261764
269669
0
39458
69338
128969
172052
247432
304793
327568
269908
245657
240745
245145
290346
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
400000 320000 240000 160000 80000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
20608
0
0
0
0
0
0
0
0
0
0
0
13760
13888
0
0
0
0
0
0
0
0
0
0
5280
8384
40192
0
0
0
0
0
0
0
0
0
2576
4384
12736
28544
0
0
0
0
0
0
0
0
1176
2160
11456
16512
41728
0
0
0
0
0
0
0
700
1256
2192
9280
20864
16192
0
0
0
0
0
0
342
652
2672
4832
10560
10944
35584
0
0
0
0
0
169
338
1944
4448
6368
5408
27264
42752
0
0
0
0
109
201
708
1736
2384
2416
18048
37632
69120
0
0
0
65
91
266
860
1592
3760
16512
31104
64768
132096
0
0
56
74
104
548
916
3504
2896
9792
21376
45824
94720
0
47
61
77
187
506
1144
1848
2640
3984
10048
19072
40192
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.3
0
0
0
0
0
0
0
0
0
0
0
3
6.3
0
0
0
0
0
0
0
0
0
0
3.3
5.8
9
0
0
0
0
0
0
0
0
0
3.1
4.8
9.1
10.8
0
0
0
0
0
0
0
0
3.2
5.8
9.1
10.4
34.5
0
0
0
0
0
0
0
3.1
4.8
9
12.8
33.7
64.1
0
0
0
0
0
0
2.9
4.8
8.6
14.7
33
64.3
92.3
0
0
0
0
0
2.9
4.8
8.7
12.2
20.3
62.9
92.7
93.9
0
0
0
0
3
4.7
9.5
14.4
30.8
61.3
93.2
93.3
85.4
0
0
0
3.3
5.4
9
12.6
30.2
50.7
92.9
93.6
93.6
95
0
0
2.6
4.4
7.6
10
16.1
27.3
81.2
93.2
93.7
94.1
85.9
0
1.9
3.2
5.5
8.1
12.6
15.9
34.3
83.2
93.7
93.4
94
91.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
125000 100000 75000 50000 25000 0
100 80 60 40 20 0
Software RAID10 over 4 Intel P3700 as blockdevice (Test ID 7) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
175966
0
0
0
0
0
0
0
0
0
0
0
180708
327563
0
0
0
0
0
0
0
0
0
0
175493
365780
611759
0
0
0
0
0
0
0
0
0
228977
355112
638214
953947
0
0
0
0
0
0
0
0
188064
378887
594206
906135
431402
0
0
0
0
0
0
0
231038
351444
611741
927149
445509
397038
0
0
0
0
0
0
187286
293701
561422
873087
432021
383441
379027
0
0
0
0
0
134080
247574
395062
644827
494644
396316
382602
367160
0
0
0
0
72330
136228
246392
430052
645075
414013
378546
377708
378830
0
0
0
36876
71697
131793
240292
412056
621600
384413
383612
378353
378453
0
0
17623
35729
68242
130619
241656
394559
530879
377596
379020
377996
778046
0
9616
17446
34373
66964
126237
210573
383674
526566
383613
384690
386449
386894
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
13248
0
0
0
0
0
0
0
0
0
0
0
7968
7904
0
0
0
0
0
0
0
0
0
0
3632
3344
5856
0
0
0
0
0
0
0
0
0
1624
1768
2864
9920
0
0
0
0
0
0
0
0
844
1004
1608
4704
6560
0
0
0
0
0
0
0
426
548
1020
2992
3248
7136
0
0
0
0
0
0
282
438
1912
1608
2096
3728
27264
0
0
0
0
0
195
247
692
1528
1720
2128
18560
58112
0
0
0
0
171
205
302
438
740
1688
17024
17280
69120
0
0
0
167
177
298
378
494
828
3664
11584
7136
15680
0
0
229
199
306
310
314
596
1688
2320
3088
3888
12224
0
155
227
294
266
310
446
442
1560
2672
3792
11968
25728
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.5
0
0
0
0
0
0
0
0
0
0
0
3.5
6.4
0
0
0
0
0
0
0
0
0
0
3.5
6.4
10.9
0
0
0
0
0
0
0
0
0
2.7
6.4
10.9
18.4
0
0
0
0
0
0
0
0
3.5
6.2
10
19.1
36.9
0
0
0
0
0
0
0
2.7
6.4
10.5
17.1
36.1
69.2
0
0
0
0
0
0
3.3
5.2
9.4
16.2
36.9
63
95.4
0
0
0
0
0
2.3
4.6
8
13
33.8
63.9
94.9
80.6
0
0
0
0
1.5
2.9
5.5
9.8
17.2
50.1
95.5
94.2
95.2
0
0
0
1.1
1.8
3.4
6.2
11.9
22.9
88.9
94.9
94.4
94.7
0
0
0.8
1.2
2
3.7
7.3
13.1
21
90.6
94.9
94.7
65.7
0
0.5
0.9
1.3
2.6
4.7
8.3
16.4
28.1
95
94.3
94.5
92.5
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID10 over 4 Intel P3700 as blockdevice (Test ID 7) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
111255
0
0
0
0
0
0
0
0
0
0
0
124970
184963
0
0
0
0
0
0
0
0
0
0
120692
167088
177770
0
0
0
0
0
0
0
0
0
127900
185038
176876
187751
0
0
0
0
0
0
0
0
125795
167605
180688
186217
176332
0
0
0
0
0
0
0
121175
158263
191497
178274
165170
175576
0
0
0
0
0
0
117191
154934
188071
184527
177610
176969
176181
0
0
0
0
0
115914
175285
195856
191226
187501
177487
206599
167658
0
0
0
0
122881
172101
196288
197901
189125
187068
152182
177905
171364
0
0
0
101225
139588
182194
198894
199282
188272
194559
182051
157973
181913
0
0
59583
103206
146304
184869
183968
204162
199951
186808
141625
146640
178883
0
35388
63761
97755
126742
162890
147721
153851
185616
192528
124660
180555
189970
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
200000 160000 120000 80000 40000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
37632
0
0
0
0
0
0
0
0
0
0
0
12096
22912
0
0
0
0
0
0
0
0
0
0
8384
14272
41216
0
0
0
0
0
0
0
0
0
3568
10816
24448
47872
0
0
0
0
0
0
0
0
1848
3856
16064
24448
94720
0
0
0
0
0
0
0
1240
2096
10048
15296
57600
113152
0
0
0
0
0
0
860
1720
5856
10432
20096
59648
150528
0
0
0
0
0
490
1352
2896
5728
10304
27264
41728
168960
0
0
0
0
237
772
1656
2928
5792
10560
24960
42752
276480
0
0
0
93
266
916
1672
2928
5856
11840
20096
70144
201728
0
0
85
104
524
908
1720
2960
5664
10816
28544
127488
199680
0
77
79
171
494
1004
2896
5472
6176
10432
53504
50944
177152
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.2
0
0
0
0
0
0
0
0
0
0
0
2.9
5.3
0
0
0
0
0
0
0
0
0
0
3
4.8
5.9
0
0
0
0
0
0
0
0
0
2.9
4.8
5.7
6.9
0
0
0
0
0
0
0
0
2.9
4.6
5.9
6.7
7.5
0
0
0
0
0
0
0
3
4.5
6.2
6.8
6.9
12.8
0
0
0
0
0
0
3
4.4
6.2
6.9
9.4
9.7
11.5
0
0
0
0
0
2.9
4.9
6.5
7.4
8.5
11
53.5
25.4
0
0
0
0
3
5.6
6.3
8
9.2
19.3
28.6
36.3
29.2
0
0
0
3.1
4.9
6.5
8.1
9.4
13.6
39.9
39.2
28.2
41.1
0
0
2.4
4.1
5.6
8.1
8.8
11.7
27.1
35.5
24.4
25.4
35.4
0
1.6
2.8
4.3
7
8.8
9.4
13.2
24.8
39.2
18.4
36.5
41.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
250000 200000 150000 100000 50000 0
100 80 60 40 20 0
Software RAID10 over 4 Intel P3700 as blockdevice (Test ID 7) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
48144
0
0
0
0
0
0
0
0
0
0
0
45757
67001
0
0
0
0
0
0
0
0
0
0
47408
64758
114478
0
0
0
0
0
0
0
0
0
52240
71615
119400
124192
0
0
0
0
0
0
0
0
52609
66378
119146
127608
116630
0
0
0
0
0
0
0
45994
68171
118184
125342
117500
97544
0
0
0
0
0
0
42123
67023
97366
121938
125402
93078
97476
0
0
0
0
0
25410
45811
70415
97758
118194
97365
96437
96887
0
0
0
0
12062
24565
45120
68939
97097
110604
97063
100191
103374
0
0
0
6448
12443
25605
44980
68738
95432
117772
96107
96182
102128
0
0
3515
7417
12722
25918
43899
67378
91140
114457
96112
101783
97846
0
2432
5341
8836
17446
27221
42409
65398
85185
113201
96692
104289
116913
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
125000 100000 75000 50000 25000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
15040
0
0
0
0
0
0
0
0
0
0
0
9664
14016
0
0
0
0
0
0
0
0
0
0
4320
7840
34048
0
0
0
0
0
0
0
0
0
1960
7712
16320
45312
0
0
0
0
0
0
0
0
884
3280
8896
17536
71168
0
0
0
0
0
0
0
450
868
5344
9280
16320
14144
0
0
0
0
0
0
213
346
1144
5408
9536
10688
31616
0
0
0
0
0
187
109
270
1112
6176
7456
24192
41216
0
0
0
0
203
179
91
366
1176
5920
12864
32128
61184
0
0
0
171
167
123
97
422
1352
4512
8096
19840
34048
0
0
135
113
126
107
119
394
1560
4768
7584
17792
31616
0
141
66
103
79
102
253
450
2384
4512
9536
17024
30336
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.8
0
0
0
0
0
0
0
0
0
0
0
2.9
4.7
0
0
0
0
0
0
0
0
0
0
2.8
4.5
8.6
0
0
0
0
0
0
0
0
0
2.8
4.6
8.6
10.9
0
0
0
0
0
0
0
0
2.8
4.8
8.6
11
23.2
0
0
0
0
0
0
0
3.2
5.4
8.6
11.5
25
64.6
0
0
0
0
0
0
3.1
4.6
7.7
10.7
14.9
63
94.1
0
0
0
0
0
2.2
3.7
5.8
8.8
12.8
60.8
93.6
95.2
0
0
0
0
1.4
2.4
4.2
6.8
10.5
21.5
92.1
94.4
95.1
0
0
0
1
1.5
2.9
4.8
7.4
11.9
22.2
93.9
94.6
81.6
0
0
0.9
1.3
1.8
3.4
5.4
9.1
15.2
26.2
93.6
91.7
94.3
0
0.7
1.1
1.5
2.4
3.7
6.6
9.6
15.7
31.2
93.5
93.2
94.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID10 over 4 Intel P3700 as blockdevice (Test ID 7) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
8.25143e+06
0
8.21883e+06 9.5521e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
8.67074e+06 9.54211e+06 9.56213e+06
8.74065e+06 9.63729e+06 8.39779e+06 9.07503e+06
8.11328e+06 9.57638e+06 9.55173e+06 9.24845e+06 9.30096e+06
7.56458e+06 9.54701e+06 9.54051e+06 9.46445e+06 9.05529e+06 7.08579e+06
6.44956e+06 9.37703e+06 9.39426e+06 9.05865e+06 8.65828e+06 8.71222e+06 7.20118e+06
4.99566e+06 7.51774e+06 8.58881e+06 9.32997e+06 8.99907e+06 6.60453e+06 5.81367e+06 5.86367e+06
2.87705e+06 5.16262e+06 6.87338e+06 9.02489e+06 8.54635e+06 7.39874e+06 5.33496e+06 5.31893e+06 7.44703e+06
1.50732e+06 3.3191e+06 4.6648e+06 6.13268e+06 8.03813e+06 7.81001e+06 6.75325e+06 5.08649e+06 5.69299e+06 6.54435e+06
1.29026e+06 1.37124e+06 2.41312e+06 4.04073e+06 5.78097e+06 6.12463e+06 5.00277e+06 4.93048e+06 5.87416e+06 6.76239e+06 6.90898e+06 2.07773e+06
631680
1
2
8
16
32
64
128
number of workers
6000000 4000000 2000000
0
1.21029e+06 2.33226e+06 3.5735e+06 4.92694e+06 4.60178e+06 6.52749e+06 5.00898e+06 5.75222e+06 8.09673e+06 5.35497e+06
4
8000000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
37632
0
0
0
0
0
0
0
0
0
0
0
19840
114176
0
0
0
0
0
0
0
0
0
0
12736
65280
91648
0
0
0
0
0
0
0
0
0
8896
24960
81408
113152
0
0
0
0
0
0
0
0
5088
12224
33024
57088
268288
0
0
0
0
0
0
0
3152
5920
18304
35584
115200
528384
0
0
0
0
0
0
1656
2992
8768
19840
42752
187392
444416
0
0
0
0
0
868
1784
3440
8896
15680
62720
288768
675840
0
0
0
0
636
1816
2608
3024
10816
20352
61696
264192
913408
0
0
0
612
1720
2480
2704
3696
11584
31616
123392
724992
692224
0
0
580
868
2224
2352
2864
5024
16192
40192
114176
411648
618496
0
155
652
2416
2320
2640
3024
7584
13120
38144
168960
222208
411648
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.1
0
0
0
0
0
0
0
0
0
0
0
3.1
3.5
0
0
0
0
0
0
0
0
0
0
2.6
3.6
4
0
0
0
0
0
0
0
0
0
2.8
3.4
4.5
4.5
0
0
0
0
0
0
0
0
2.9
3.7
3.8
4.3
4.7
0
0
0
0
0
0
0
2.4
3.3
3.6
4.2
4.8
6.5
0
0
0
0
0
0
2.2
2.9
3.4
3.8
5.5
5.2
7.5
0
0
0
0
0
1.5
2.4
2.5
3.6
4.3
7.2
7.5
7.7
0
0
0
0
1.2
1.7
2.4
3.2
4.6
7.5
8
8.2
8.3
0
0
0
1
1.3
1.9
2.7
4.1
6.8
6.7
7.9
7.7
7.2
0
0
0.8
1.1
1.5
2.3
3.4
6.7
8.4
7.7
7.9
7.3
6.8
0
0.7
1
1.3
1.9
4.3
7.9
8.5
10.5
7.2
8.4
5
8.9
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
800000 600000 400000 200000 0
100 80 60 40 20 0
Software RAID10 over 4 Intel P3700 as blockdevice (Test ID 7) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
2.61814e+06
0
2.81023e+06 3.70498e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2.92879e+06 3.82184e+06 3.6169e+06
3.08058e+06 3.82352e+06 3.6436e+06 2.26014e+06
3.71666e+06 3.80553e+06 3.70254e+06 3.06578e+06 3.42718e+06
3.60904e+06 3.76675e+06 3.53887e+06 3.50502e+06 3.39212e+06 3.26298e+06
3.32814e+06 3.62934e+06 3.45634e+06 2.5469e+06 2.78473e+06 2.91342e+06 3.06512e+06
3.23073e+06 3.42271e+06 3.04111e+06 2.30066e+06 3.34754e+06 3.20725e+06 3.06973e+06 3.26565e+06
3.16154e+06 3.31086e+06 3.0332e+06 2.41687e+06 3.10992e+06 3.13669e+06 3.0368e+06 3.33687e+06 3.27856e+06
3.27653e+06 3.15991e+06 3.14129e+06 2.33453e+06 2.32926e+06 2.97705e+06 2.78389e+06 3.41271e+06 3.34243e+06 3.37167e+06
2.53646e+06 2.90121e+06 2.75831e+06 2.60997e+06 2.25828e+06 3.02153e+06 2.6219e+06 2.82299e+06 3.49618e+06 3.55346e+06 3.39133e+06
2
4
8
16
32
64
128
number of workers
2400000 1600000 800000
0
1.30355e+06 2.27574e+06 2.6065e+06 2.77221e+06 1.98423e+06 3.22568e+06 2.61959e+06 2.97559e+06 3.27783e+06 3.53633e+06 3.48964e+06 3.24726e+06
1
3200000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
193536
0
0
0
0
0
0
0
0
0
0
0
90624
115200
0
0
0
0
0
0
0
0
0
0
38656
67072
309248
0
0
0
0
0
0
0
0
0
23936
56576
130560
724992
0
0
0
0
0
0
0
0
17536
31104
70144
216064
485376
0
0
0
0
0
0
0
13760
17280
45312
90624
242688
716800
0
0
0
0
0
0
8640
13888
25728
62720
164864
505856
610304
0
0
0
0
0
3408
8256
18560
37632
53504
173056
428032
708608
0
0
0
0
1976
3024
11584
19840
34560
50432
152576
329728
643072
0
0
0
1096
1912
3248
12224
21632
33024
64256
109056
561152
651264
0
0
153
1464
3248
8160
12480
19584
32640
67072
154624
346112
577536
0
179
322
2040
2736
7840
9408
20864
34560
49920
158720
288768
733184
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.5
0
0
0
0
0
0
0
0
0
0
0
2.5
3.5
0
0
0
0
0
0
0
0
0
0
2.5
3.8
4.9
0
0
0
0
0
0
0
0
0
2.5
3.4
4.5
3.4
0
0
0
0
0
0
0
0
2.3
3.4
4.8
4.3
5.4
0
0
0
0
0
0
0
2.3
2.7
3.6
4.5
5.2
6.2
0
0
0
0
0
0
2.1
2.4
2.4
2.7
3.9
4.9
8.1
0
0
0
0
0
1.9
2.1
2.1
1.9
3.7
5.7
5.5
8.2
0
0
0
0
1.9
2.1
2.1
2
4
4.2
5.4
6.7
8.4
0
0
0
2.1
2
2.2
1.9
2.2
3.7
4.5
5.9
7.2
8.3
0
0
1.9
2
2.1
2.6
2.5
3
3.4
4.4
6.9
7.3
8.6
0
1.2
1.8
2.1
2.2
2
3.8
3.1
4.7
6.3
7
8.4
10.3
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000 450000 300000 150000 0
100 80 60 40 20 0
Software RAID10 over 6 Intel P3700 as blockdevice (Test ID 8) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
193672
0
0
0
0
0
0
0
0
0
0
0
193046
380463
0
0
0
0
0
0
0
0
0
0
224273
380030
711015
0
0
0
0
0
0
0
0
0
201840
380868
730188
1.03374e+06
0
0
0
0
0
0
0
0
200477
377163
606592
1.0169e+06
413084
0
0
0
0
0
0
0
221151
364518
728938
1.0285e+06
433253
380331
0
0
0
0
0
0
225496
332739
488926
1.02956e+06
427145
372681
384817
0
0
0
0
0
163740
295898
509366
917259
419884
402767
383940
383119
0
0
0
0
86975
167492
309517
543928
794405
339234
383777
383509
383825
0
0
0
43768
86944
169907
321505
561112
667923
384204
383665
383017
383805
0
0
21304
43662
86844
169509
301544
538196
642366
384562
549731
465510
648036
0
10158
21466
42953
84707
159730
309616
492739
561016
385411
388958
1.13668e+06
606383
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1000000 800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
14784
0
0
0
0
0
0
0
0
0
0
0
7712
6304
0
0
0
0
0
0
0
0
0
0
3376
3216
3824
0
0
0
0
0
0
0
0
0
2024
1640
1688
2448
0
0
0
0
0
0
0
0
1032
884
5536
1384
6496
0
0
0
0
0
0
0
430
556
490
780
3248
7008
0
0
0
0
0
0
270
358
732
466
1880
4128
30336
0
0
0
0
0
193
270
532
458
1240
2448
18560
45824
0
0
0
0
191
225
278
692
684
2064
17024
50432
129536
0
0
0
201
191
227
247
442
1576
4384
11840
31616
49408
0
0
221
195
203
199
330
474
1784
2832
2352
8768
5728
0
233
203
209
177
253
241
660
2064
2928
4704
3440
9024
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.6
0
0
0
0
0
0
0
0
0
0
0
3.6
5.9
0
0
0
0
0
0
0
0
0
0
3
6.7
11.5
0
0
0
0
0
0
0
0
0
3.7
6.7
11.8
20.2
0
0
0
0
0
0
0
0
3.7
6.7
10.4
20.6
36.8
0
0
0
0
0
0
0
3
5.9
11.2
20.3
36.6
65.6
0
0
0
0
0
0
3.1
5.1
8.5
19.2
36.1
66.6
94.9
0
0
0
0
0
2.8
5.1
9.1
16.3
36.1
63.9
94.8
94.8
0
0
0
0
1.9
3.8
7
11.4
25.3
58.8
88.9
91.7
94.6
0
0
0
1.2
2.3
4.3
8.3
14.6
29
83.3
95.5
95.1
94.7
0
0
0.9
1.4
2.6
5
9.1
17.2
26.8
91.3
57.7
91.6
70.3
0
0.6
0.9
1.7
3.2
5.7
10.6
20.2
31.2
94.1
95.1
71.3
77.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
125000 100000 75000 50000 25000 0
100 80 60 40 20 0
Software RAID10 over 6 Intel P3700 as blockdevice (Test ID 8) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
1268
0
0
0
0
0
0
0
0
0
0
0
1316
1345
0
0
0
0
0
0
0
0
0
0
1287
1361
1391
0
0
0
0
0
0
0
0
0
1282
1334
1359
1404
0
0
0
0
0
0
0
0
1265
1451
1392
1387
1472
0
0
0
0
0
0
0
1266
1396
1392
1363
1444
1518
0
0
0
0
0
0
1302
1348
1368
1385
1457
1515
1552
0
0
0
0
0
1305
1328
1458
1378
1456
1568
1509
1517
0
0
0
0
1298
1300
1428
1356
1445
1544
1503
1631
1406
0
0
0
1423
1355
1437
1366
1467
1570
1588
1596
1409
1388
0
0
1317
1315
1374
1394
1442
1550
1533
1445
1371
1367
1466
0
1345
1298
1326
1316
1389
1456
1466
1402
1325
1339
1363
1296
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
0
0
0
0
0
0
0
0
0
0
0
864256
1.71213e+06
0
0
0
0
0
0
0
0
0
0
481280
880640
1.61382e+06
0
0
0
0
0
0
0
0
0
268288
485376
921600
1.67936e+06
0
0
0
0
0
0
0
0
138240
230400
477184
888832
1.81043e+06
0
0
0
0
0
0
0
78336
130560
250880
501760
937984
1.9415e+06
0
0
0
0
0
0
44288
73216
140288
276480
544768
1.30253e+06 1.89235e+06
0
0
0
0
0
24960
44288
78336
160768
329728
643072
0
0
0
0
15040
26240
47360
98816
179200
370688
626688
0
0
0
8640
14656
26496
53504
114176
242688
329728
888832
5.93101e+06 8.84736e+06
0
0
3184
6304
14272
28288
60672
146432
152576
536576
2.93274e+06 7.04512e+06 1.04202e+07
135
1736
2832
2928
7968
19840
31616
31360
29056
28800
27008
26496
1
2
4
8
16
32
64
128
256
512
1024
2048
1.51552e+06 8.58522e+06
number of workers
queue depth
0.6
0
0
0
0
0
0
0
0
0
0
0
0.6
0.6
0
0
0
0
0
0
0
0
0
0
0.6
0.6
0.7
0
0
0
0
0
0
0
0
0
0.6
0.6
0.7
0.9
0
0
0
0
0
0
0
0
0.6
0.6
0.7
0.9
1.5
0
0
0
0
0
0
0
0.6
0.6
0.7
0.9
1.4
2
0
0
0
0
0
0
0.6
0.5
0.7
0.9
1.5
1.9
2.5
0
0
0
0
0
0.6
0.6
0.7
0.9
1.5
2.1
2.5
3.4
0
0
0
0
0.6
0.5
0.7
0.9
1.5
1.9
2.4
3.6
5.2
0
0
0
0.6
0.5
0.7
1
1.6
2
2.5
3.4
5.6
9.3
0
0
0.6
0.6
0.7
0.9
1.5
2.1
2.7
3.5
5.7
9.6
21.6
0
0.8
0.6
0.7
0.9
1.5
2.1
3.1
3.9
5.9
10.1
20.8
44.8
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
900 600 300 0
1.0 0.8 0.6 0.4 0.2
0
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
1200
1e7
1.72851e+06
1.08954e+06 2.8672e+06
1500
0.0
100 80 60 40 20 0
Software RAID10 over 6 Intel P3700 as blockdevice (Test ID 8) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
173808
0
0
0
0
0
0
0
0
0
0
0
202859
340413
0
0
0
0
0
0
0
0
0
0
206946
370108
632902
0
0
0
0
0
0
0
0
0
194998
372962
697214
974523
0
0
0
0
0
0
0
0
183430
374561
719551
980055
420527
0
0
0
0
0
0
0
215610
335345
555407
950208
427205
367345
0
0
0
0
0
0
193441
295320
495408
979820
452790
372397
384148
0
0
0
0
0
139069
251886
464017
714269
435874
365558
385637
384912
0
0
0
0
72333
141300
269431
465551
695201
345138
381362
385550
384801
0
0
0
36702
72756
143205
263825
435189
572328
395912
383288
384741
384842
0
0
17651
36994
74135
138827
254508
391208
600282
382124
385514
416204
1.00686e+06
0
7992
18166
36681
71507
134540
248415
416751
568219
385055
386252
724605
402125
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1000000 800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
13632
0
0
0
0
0
0
0
0
0
0
0
5408
7264
0
0
0
0
0
0
0
0
0
0
2768
3408
13632
0
0
0
0
0
0
0
0
0
1944
1928
1720
2608
0
0
0
0
0
0
0
0
900
908
1032
1496
7328
0
0
0
0
0
0
0
402
604
2384
932
3472
7264
0
0
0
0
0
0
310
406
764
2224
1816
4192
32384
0
0
0
0
0
203
262
350
1480
1224
2576
22912
58112
0
0
0
0
189
203
227
374
764
2416
16768
26752
69120
0
0
0
181
183
185
245
676
1592
3120
9024
10560
39680
0
0
211
185
171
211
282
1064
1432
2768
4448
3120
8896
0
262
181
169
183
223
302
596
1432
2960
4192
4704
15296
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.7
0
0
0
0
0
0
0
0
0
0
0
3.2
6.7
0
0
0
0
0
0
0
0
0
0
3.2
6.1
11.3
0
0
0
0
0
0
0
0
0
3.6
6.3
11.8
20.8
0
0
0
0
0
0
0
0
3.7
6.3
11.2
20.5
36.6
0
0
0
0
0
0
0
3.2
5.2
9.7
20
36.8
65.8
0
0
0
0
0
0
3.3
5.3
9.1
17.3
35.1
65.8
87.2
0
0
0
0
0
2.5
5.4
8.7
14.1
34.1
55.9
93.8
90.6
0
0
0
0
1.8
3.3
6.5
11
19.4
50.1
87
95.4
95.1
0
0
0
1.2
2.1
3.9
7.4
12.4
17.4
77.8
92
92.3
95
0
0
0.9
1.3
2.3
4.7
8.2
13.7
25.9
88.3
95.2
94.5
66.7
0
0.7
0.8
1.4
2.8
5.6
9.6
16.9
30.7
94
94.6
76.9
95.4
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID10 over 6 Intel P3700 as blockdevice (Test ID 8) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
1290
0
0
0
0
0
0
0
0
0
0
0
1313
1314
0
0
0
0
0
0
0
0
0
0
1303
1398
1533
0
0
0
0
0
0
0
0
0
1377
1351
1422
1379
0
0
0
0
0
0
0
0
1293
1375
1449
1370
1505
0
0
0
0
0
0
0
1340
1339
1422
1378
1429
1560
0
0
0
0
0
0
1287
1369
1427
1413
1440
1519
1529
0
0
0
0
0
1311
1332
1336
1480
1464
1575
1502
1500
0
0
0
0
1421
1330
1380
1428
1451
1541
1611
1508
1394
0
0
0
1317
1351
1497
1371
1434
1546
1548
1528
1373
1407
0
0
1313
1332
1447
1378
1487
1568
1457
1438
1393
1473
1362
0
1323
1320
1368
1352
1334
1449
1391
1387
1364
1363
1385
1291
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
0
0
0
0
0
0
0
0
0
0
0
880640
1.71213e+06
0
0
0
0
0
0
0
0
0
0
460800
831488
1.51552e+06
0
0
0
0
0
0
0
0
0
230400
448512
880640
1.79405e+06
0
0
0
0
0
0
0
0
138240
250880
440320
937984
1.99066e+06
0
0
0
0
0
0
0
75264
138240
244736
489472
1.03629e+06 1.82682e+06
0
0
0
0
0
0
44288
72192
136192
268288
561152
0
0
0
0
0
24704
44288
79360
148480
317440
593920
0
0
0
0
14784
27008
45312
89600
195584
428032
602112
0
0
0
8512
14528
26240
51968
107008
261120
325632
888832
6.52083e+06 9.63379e+06
0
0
3248
6368
14528
27776
62208
148480
156672
552960
3.32595e+06 7.50387e+06 1.04202e+07
135
2192
2800
2928
7968
20096
33024
29056
28288
27776
24704
30080
1
2
4
8
16
32
64
128
256
512
1024
2048
1.10592e+06 2.99827e+06
1.69574e+06 8.45414e+06
number of workers
queue depth
0.6
0
0
0
0
0
0
0
0
0
0
0
0.6
0.6
0
0
0
0
0
0
0
0
0
0
0.6
0.6
0.8
0
0
0
0
0
0
0
0
0
0.7
0.6
0.7
1
0
0
0
0
0
0
0
0
0.6
0.6
0.7
0.9
1.6
0
0
0
0
0
0
0
0.6
0.6
0.7
1
1.5
2
0
0
0
0
0
0
0.6
0.6
0.7
0.9
1.5
2
2.6
0
0
0
0
0
0.6
0.6
0.6
1
1.5
2
2.5
3.4
0
0
0
0
0.6
0.6
0.7
0.9
1.5
2.1
2.5
3.4
5.2
0
0
0
0.6
0.6
0.7
0.9
1.5
1.9
2.6
3.4
5.3
9.4
0
0
0.6
0.6
0.7
0.9
1.5
2.1
2.6
3.5
5.7
10.4
21.5
0
0.7
0.6
0.6
1
1.7
2.2
2.8
4
6
10.3
21.7
45.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
900 600 300 0
1.0 0.8 0.6 0.4 0.2
0
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
1200
1e7
1.72851e+06
1.10592e+06 1.90874e+06
1500
0.0
100 80 60 40 20 0
Software RAID10 over 6 Intel P3700 as blockdevice (Test ID 8) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
1274
0
0
0
0
0
0
0
0
0
0
0
1295
1296
0
0
0
0
0
0
0
0
0
0
1265
1349
1311
0
0
0
0
0
0
0
0
0
1288
1422
1301
1315
0
0
0
0
0
0
0
0
1331
1334
1351
1349
1374
0
0
0
0
0
0
0
1295
1282
1327
1311
1351
1462
0
0
0
0
0
0
1288
1280
1339
1342
1390
1496
1509
0
0
0
0
0
1290
1301
1300
1415
1318
1581
1521
1501
0
0
0
0
1298
1306
1340
1370
1367
1559
1594
1458
1363
0
0
0
1252
1377
1341
1338
1331
1412
1451
1378
1349
1326
0
0
1240
1305
1311
1296
1321
1371
1332
1316
1296
1336
1280
0
1180
1233
1301
1342
1293
1282
1297
1319
1304
1327
1315
1297
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1500 1200 900 600 300 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
585728
0
0
0
0
0
0
0
0
0
0
0
296960
544768
0
0
0
0
0
0
0
0
0
0
171008
292864
593920
0
0
0
0
0
0
0
0
0
91648
156672
325632
692224
0
0
0
0
0
0
0
0
52480
98816
187392
362496
700416
0
0
0
0
0
0
0
30848
56576
111104
218112
444416
937984
0
0
0
0
0
0
19072
35072
65280
129536
257024
577536
839680
0
0
0
0
0
11968
22656
42752
78336
168960
305152
518144
1.28614e+06
0
0
0
0
6688
12736
26240
52992
107008
209920
313344
921600
5.40672e+06
0
0
0
5792
8512
17024
35584
77312
195584
209920
724992
3.62086e+06 5.47226e+06
0
0
3088
5792
9280
21120
51456
125440
136192
544768
1.54829e+06 1.95789e+06 7.30726e+06
141
1160
2544
2896
6176
17792
32640
32640
30336
28544
27008
27520
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
0.6
0
0
0
0
0
0
0
0
0
0
0
0.6
0.6
0
0
0
0
0
0
0
0
0
0
0.6
0.6
0.7
0
0
0
0
0
0
0
0
0
0.6
0.6
0.7
1
0
0
0
0
0
0
0
0
0.6
0.6
0.7
1
1.5
0
0
0
0
0
0
0
0.6
0.6
0.7
1
1.4
2.1
0
0
0
0
0
0
0.6
0.6
0.7
1
1.5
2.1
2.7
0
0
0
0
0
0.6
0.6
0.7
1
1.5
2.1
2.7
3.5
0
0
0
0
0.6
0.6
0.7
1
1.5
2.1
2.8
3.5
5.4
0
0
0
0.7
0.7
0.8
1
1.6
2.3
2.9
3.8
5.8
10.2
0
0
0.7
0.8
0.8
1
1.6
2.3
3.1
4
5.9
10.2
20.8
0
0.7
0.8
0.8
1
1.6
2.4
3.1
4.1
6
10.5
21.8
45.4
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
4500000 3000000 1500000
0
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
6000000
0
100 80 60 40 20 0
Software RAID10 over 6 Intel P3700 as blockdevice (Test ID 8) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
8.35805e+06
0
8.68292e+06 1.40464e+07
1e7
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
8.39128e+06 1.35921e+07 1.36313e+07
8.45061e+06 1.34823e+07 1.35254e+07 1.41552e+07
8.89569e+06 1.29259e+07 1.39226e+07 1.3894e+07 1.37765e+07
8.80029e+06 1.38671e+07 1.36525e+07 1.41584e+07 1.38296e+07 1.28838e+07
8.60828e+06 1.27224e+07 1.32794e+07 1.39031e+07 1.36512e+07 1.24845e+07 1.3001e+07
5.52646e+06 9.2864e+06 1.13843e+07 1.36761e+07 1.27851e+07 9.90679e+06 8.59555e+06 7.27262e+06
2.81410e+06 5.64461e+06 8.26875e+06 1.20113e+07 1.29824e+07 8.60396e+06 8.76505e+06 6.66607e+06 6.38683e+06
1.8945e+06 2.82415e+06 4.82217e+06 7.70781e+06 1.12361e+07 7.70593e+06 7.65761e+06 7.50758e+06 8.54958e+06 8.16485e+06
1.56084e+06 1.28475e+06 2.42346e+06 4.28589e+06 6.96332e+06 8.22092e+06 8.65337e+06 7.2304e+06 8.40987e+06 7.48823e+06 1.03653e+07 1.5248e+06
565644
1
2
0
1.24069e+06 2.33357e+06 3.99543e+06 5.84264e+06 6.73148e+06 6.48688e+06 6.96379e+06 7.07208e+06 9.14534e+06 9.13146e+06
4
8
16
32
64
128
number of workers
256
512
1024
2048
1.25 1.00 0.75 0.50 0.25 0.00
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
37632
0
0
0
0
0
0
0
0
0
0
0
20864
83456
0
0
0
0
0
0
0
0
0
0
11072
30336
89600
0
0
0
0
0
0
0
0
0
5984
23168
65280
78336
0
0
0
0
0
0
0
0
2768
10048
29824
79360
138240
0
0
0
0
0
0
0
1960
3792
17536
24704
97792
162816
0
0
0
0
0
0
1112
1816
5920
14912
36608
107008
569344
0
0
0
0
0
652
828
2672
5792
13888
48896
150528
643072
0
0
0
0
596
620
1304
1912
6496
39168
110080
232448
456704
0
0
0
620
724
924
1160
1800
16512
41216
121344
209920
552960
0
0
596
644
788
908
1352
4192
11072
36608
109056
234496
448512
0
195
644
668
788
948
1752
5344
19840
33024
144384
264192
544768
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.1
0
0
0
0
0
0
0
0
0
0
0
2.9
5.3
0
0
0
0
0
0
0
0
0
0
3.1
4.9
6.4
0
0
0
0
0
0
0
0
0
3.2
4.6
6.3
7.2
0
0
0
0
0
0
0
0
2.7
4.8
5.9
6.6
8.6
0
0
0
0
0
0
0
2.8
4.8
5.6
6.6
7.6
9.7
0
0
0
0
0
0
2.6
4.5
5.2
6
7.2
8.8
8.1
0
0
0
0
0
1.6
2.9
3.7
5.3
6.6
10.8
11.2
13.8
0
0
0
0
1.1
2
2.8
4.6
6.5
9.7
9.6
12.4
13.4
0
0
0
1.2
1.3
2.1
3.5
6.5
9.6
12.3
13.5
14.6
13.3
0
0
1.1
1.2
1.7
3
4.9
10.9
15.3
13.4
13.4
11.4
9
0
0.9
1.2
1.4
2.3
4.6
9.7
12
12.7
13.5
12
9.2
10.3
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000
450000
300000
150000
0
100 80 60 40 20 0
Software RAID10 over 6 Intel P3700 as blockdevice (Test ID 8) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
2.78607e+06
0
3.0082e+06 4.72132e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2.72516e+06 4.91143e+06 5.72566e+06
3.18336e+06 5.18298e+06 5.74304e+06 5.5426e+06
4.80591e+06 5.74858e+06 5.75136e+06 5.69955e+06 5.50315e+06
4.92442e+06 5.7589e+06 5.81445e+06 5.75891e+06 5.60718e+06 5.04792e+06
4.96429e+06 5.60526e+06 5.71527e+06 5.69173e+06 5.66212e+06 5.2772e+06 4.84116e+06
5.00411e+06 5.39352e+06 5.51921e+06 5.67391e+06 5.64268e+06 5.29183e+06 4.68343e+06 4.25392e+06
4.70185e+06 5.01298e+06 5.2688e+06 5.40992e+06 5.48617e+06 5.24246e+06 4.87378e+06 4.79628e+06 4.60291e+06
3.83265e+06 4.65709e+06 4.96595e+06 4.99904e+06 5.27476e+06 4.93184e+06 5.05208e+06 4.852e+06 4.78421e+06 4.81392e+06
2.54121e+06 4.41641e+06 4.73309e+06 4.70257e+06 5.07161e+06 4.73300e+06 4.99066e+06 4.901e+06 5.02832e+06 4.83076e+06 5.21266e+06
0
1.43318e+06 2.66557e+06 4.12901e+06 4.49004e+06 4.77309e+06 4.4441e+06 4.58238e+06 4.86264e+06 4.87367e+06 4.74472e+06 4.93072e+06 5.41901e+06
1
2
4
8
16
32
64
128
number of workers
256
512
1024
2048
5000000 4000000 3000000 2000000 1000000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
179200
0
0
0
0
0
0
0
0
0
0
0
76288
79360
0
0
0
0
0
0
0
0
0
0
42240
51456
154624
0
0
0
0
0
0
0
0
0
23424
30080
82432
428032
0
0
0
0
0
0
0
0
15552
20864
59136
119296
366592
0
0
0
0
0
0
0
13760
15552
22656
54016
164864
387072
0
0
0
0
0
0
11200
12992
14656
21632
38144
250880
382976
0
0
0
0
0
2352
4960
12480
14144
24448
118272
321536
403456
0
0
0
0
1192
2512
8640
12480
14400
24192
183296
477184
407552
0
0
0
398
1240
2608
8640
13632
17536
28544
173056
329728
536576
0
0
149
486
1512
2480
9024
13888
18048
32384
128512
374784
667648
0
161
155
298
1448
2896
8640
13888
17792
32128
142336
333824
593920
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.6
0
0
0
0
0
0
0
0
0
0
0
2.6
4.6
0
0
0
0
0
0
0
0
0
0
2.6
4.7
7.2
0
0
0
0
0
0
0
0
0
2.6
4.6
7.6
9.6
0
0
0
0
0
0
0
0
2.7
4.4
5.6
8.3
8.8
0
0
0
0
0
0
0
2.7
3.7
4.7
8.6
9
10.9
0
0
0
0
0
0
2.6
3.5
3.9
5.9
8
8.7
11.8
0
0
0
0
0
2.5
3.3
3.5
4.6
5.8
8.9
9.8
14
0
0
0
0
2.5
2.9
3.5
4.6
5.7
8.8
9
10.5
13.7
0
0
0
2.4
2.9
3.5
3.9
5
8.7
9.4
10.4
11.4
12.6
0
0
1.9
2.8
3.4
3.9
4.9
9.2
9.2
11.3
9.9
11.5
11.1
0
1.2
1.9
3.4
3.5
4.3
9.1
9.6
10
11.6
10.7
10.7
9.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
600000 450000 300000 150000 0
100 80 60 40 20 0
Software RAID10 over 8 Intel P3700 as blockdevice (Test ID 9) Random Read 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
204651
0
0
0
0
0
0
0
0
0
0
0
203451
377011
0
0
0
0
0
0
0
0
0
0
210062
354602
684469
0
0
0
0
0
0
0
0
0
217721
384963
636249
1.0367e+06
0
0
0
0
0
0
0
0
212012
371865
749561
958702
429629
0
0
0
0
0
0
0
206060
331537
585285
991214
437012
381778
0
0
0
0
0
0
204366
386276
640538
954388
421140
386654
367927
0
0
0
0
0
178730
321508
536993
864475
435820
361833
344819
374359
0
0
0
0
95739
184798
365293
626195
660964
373452
384909
382604
380401
0
0
0
47276
97419
187666
345115
576476
679483
386969
382493
392865
380269
0
0
19475
48978
97063
184278
348843
544326
656488
410788
382730
416464
1.08166e+06
0
9531
23821
48373
95057
184205
330882
523952
615590
386327
386827
773737
617919
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1000000 800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
14400
0
0
0
0
0
0
0
0
0
0
0
7520
6368
0
0
0
0
0
0
0
0
0
0
3600
3472
8896
0
0
0
0
0
0
0
0
0
1384
1592
3536
9280
0
0
0
0
0
0
0
0
748
860
1368
8896
6368
0
0
0
0
0
0
0
506
540
1012
6432
3440
7456
0
0
0
0
0
0
278
258
644
1816
2192
4448
31104
0
0
0
0
0
169
235
290
2064
1544
2864
17792
45824
0
0
0
0
157
195
189
326
1144
2064
16320
11968
16512
0
0
0
173
147
179
241
580
1448
4256
8384
5536
7008
0
0
221
165
143
199
221
564
2288
3312
4896
5664
20096
0
221
181
161
167
211
270
532
2480
3504
4016
6048
10944
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.2
0
0
0
0
0
0
0
0
0
0
0
3.4
6.4
0
0
0
0
0
0
0
0
0
0
3.4
6.3
11.8
0
0
0
0
0
0
0
0
0
3.3
6.7
11.3
20.1
0
0
0
0
0
0
0
0
3.3
6.2
11.6
20.8
39.2
0
0
0
0
0
0
0
3.4
5.5
10
19.1
38.9
70.2
0
0
0
0
0
0
3.4
6.6
11.2
19.6
38.9
69.8
83.1
0
0
0
0
0
3.2
6.4
9.2
16.7
37.6
64.3
76.5
94.9
0
0
0
0
2.3
4.6
8.1
13.6
29.6
56.6
94.7
94.3
94.5
0
0
0
1.6
2.7
5.3
9.8
17.3
33.4
75
94
95
94.3
0
0
1.4
1.6
3
6.1
11.8
20.7
32.8
85.9
95.4
94.4
64.8
0
0.9
1
1.7
3.5
6.9
13
26
35.7
93.9
93.9
96.4
70.8
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
40000 32000 24000 16000 8000 0
100 80 60 40 20 0
Software RAID10 over 8 Intel P3700 as blockdevice (Test ID 9) Random Write 4kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
2025
0
0
0
0
0
0
0
0
0
0
0
1933
2002
0
0
0
0
0
0
0
0
0
0
2015
2087
2022
0
0
0
0
0
0
0
0
0
1919
1996
2041
2090
0
0
0
0
0
0
0
0
1954
1972
2039
2068
2125
0
0
0
0
0
0
0
1933
2008
2105
2016
2151
2196
0
0
0
0
0
0
2003
2029
2063
2010
2123
2226
2209
0
0
0
0
0
1992
2018
2057
2117
2189
2131
2142
2254
0
0
0
0
1967
2079
2196
2079
2074
2165
2264
2211
2022
0
0
0
1974
1983
2045
2282
2122
2317
2246
2167
2083
2020
0
0
1990
2020
2110
2038
2061
2169
2083
2097
2196
2022
1999
0
2014
2171
2027
1988
1972
2035
2087
1997
2117
2010
2014
1940
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
2000 1600 1200 800 400 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
1.1223e+06
0
0
0
0
0
0
0
0
0
0
0
626688
1.13869e+06
0
0
0
0
0
0
0
0
0
0
325632
577536
1.17146e+06
0
0
0
0
0
0
0
0
0
177152
325632
626688
1.22061e+06
0
0
0
0
0
0
0
0
105984
177152
370688
626688
1.23699e+06
0
0
0
0
0
0
0
56064
96768
185344
374784
692224
1.38445e+06
0
0
0
0
0
0
33024
55552
108032
211968
378880
790528
1.36806e+06
0
0
0
0
0
18816
33024
60160
114176
214016
514048
757760
1.82682e+06
0
0
0
0
11968
20096
34048
68096
136192
317440
415744
1.10592e+06 7.70048e+06
0
0
0
6240
11584
20608
38656
80384
191488
226304
643072
5.34118e+06 8.58522e+06
0
0
3152
6048
11072
21120
45824
116224
110080
378880
2.70336e+06 7.76602e+06 7.50387e+06
139
1976
2544
2704
5600
14272
23680
22144
21120
20096
18048
27264
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
0
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
0.8
0
0
0
0
0
0
0
0
0
0
0
0.8
0.7
0
0
0
0
0
0
0
0
0
0
0.8
0.8
0.9
0
0
0
0
0
0
0
0
0
0.8
0.7
0.8
1.2
0
0
0
0
0
0
0
0
0.7
0.7
0.8
1.2
2
0
0
0
0
0
0
0
0.7
0.7
0.8
1.1
2
2.8
0
0
0
0
0
0
0.7
0.7
0.8
1.2
2
2.7
3.8
0
0
0
0
0
0.7
0.7
0.8
1.2
2
2.7
3.7
5.2
0
0
0
0
0.7
0.8
0.9
1.1
2
2.7
3.9
5.2
7.2
0
0
0
0.7
0.7
0.9
1.2
2
2.7
3.6
5
8
13.2
0
0
0.7
0.7
0.9
1.2
2
2.8
3.7
5.2
8.3
13.7
26.7
0
0.9
0.8
0.9
1.1
2
3
4.1
5.5
8.8
14
27.6
50.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
7500000 6000000 4500000 3000000 1500000 0
100 80 60 40 20 0
Software RAID10 over 8 Intel P3700 as blockdevice (Test ID 9) Random Read 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
197392
0
0
0
0
0
0
0
0
0
0
0
182445
349001
0
0
0
0
0
0
0
0
0
0
193335
338707
628332
0
0
0
0
0
0
0
0
0
205614
379441
625548
1.01095e+06
0
0
0
0
0
0
0
0
200141
354521
704878
965724
426807
0
0
0
0
0
0
0
202551
367504
664235
947530
424100
373489
0
0
0
0
0
0
202290
348350
522972
923602
450278
368976
380282
0
0
0
0
0
145789
291275
484328
749211
420829
362103
386102
407582
0
0
0
0
79017
152269
283312
507967
779686
371399
385255
393092
381596
0
0
0
39300
78550
154999
289284
488793
715778
396814
396833
382903
392963
0
0
16512
39191
80617
152734
287179
413328
586238
514860
408336
390325
720180
0
7003
18087
40212
77641
149362
273084
434276
575603
389302
391914
416270
800827
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1000000 800000 600000 400000 200000 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
11328
0
0
0
0
0
0
0
0
0
0
0
6624
6816
0
0
0
0
0
0
0
0
0
0
2864
4080
7136
0
0
0
0
0
0
0
0
0
1880
1544
4448
4048
0
0
0
0
0
0
0
0
996
956
2160
6880
6944
0
0
0
0
0
0
0
422
462
940
2672
3440
7584
0
0
0
0
0
0
255
314
596
5216
1992
4384
20352
0
0
0
0
0
219
217
318
1352
1464
2832
2864
49408
0
0
0
0
181
217
255
354
772
3024
16512
18816
69120
0
0
0
195
185
193
251
450
1288
3792
5152
16512
3888
0
0
249
225
169
219
239
1112
1864
2640
2864
4128
7776
0
258
270
175
187
219
290
684
1976
3472
4960
5984
12224
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.2
0
0
0
0
0
0
0
0
0
0
0
3.7
6.6
0
0
0
0
0
0
0
0
0
0
3.5
6
11.4
0
0
0
0
0
0
0
0
0
3.2
6.2
11
20.3
0
0
0
0
0
0
0
0
3.3
5.9
11.6
20.3
38.4
0
0
0
0
0
0
0
3.5
6.5
11.3
19.3
38.8
69.8
0
0
0
0
0
0
3.4
6.3
9.4
17
38.8
63.2
93.3
0
0
0
0
0
3
5.7
9.6
15.7
37.9
58
87.1
65.4
0
0
0
0
2
4
7.4
13.1
21.6
56.1
93.8
95
93.5
0
0
0
1.4
2.5
4.6
8.6
15.6
25.8
78.5
94.3
94.8
82.3
0
0
1.3
1.4
2.6
5.2
9.8
16.9
29.6
55.8
87.1
94.9
93
0
1.1
1.1
1.6
3.2
5.8
12
20.5
35.1
93.1
94.8
90.5
65.6
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
60000 45000 30000 15000 0
100 80 60 40 20 0
Software RAID10 over 8 Intel P3700 as blockdevice (Test ID 9) Random Write 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
1962
0
0
0
0
0
0
0
0
0
0
0
1959
1994
0
0
0
0
0
0
0
0
0
0
1925
1969
2249
0
0
0
0
0
0
0
0
0
1953
2008
2112
2157
0
0
0
0
0
0
0
0
1987
1905
1992
2068
2068
0
0
0
0
0
0
0
2098
1996
2200
2049
2038
2148
0
0
0
0
0
0
2004
2012
2072
2024
2127
2201
2263
0
0
0
0
0
1984
1970
1986
2071
2251
2180
2207
2307
0
0
0
0
1954
2016
2031
2049
2149
2230
2164
2225
2069
0
0
0
1974
1965
2136
2053
2157
2302
2207
2295
2051
2003
0
0
1966
2054
2184
2068
2063
2239
2162
2099
2039
2007
1989
0
2070
1968
2037
2080
1976
2076
2103
2209
2188
2004
2028
1869
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
2000 1600 1200 800 400 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
1.20422e+06
0
0
0
0
0
0
0
0
0
0
0
602112
1.15507e+06
0
0
0
0
0
0
0
0
0
0
325632
618496
1.07315e+06
0
0
0
0
0
0
0
0
0
171008
333824
610304
1.17146e+06
0
0
0
0
0
0
0
0
100864
185344
346112
675840
1.23699e+06
0
0
0
0
0
0
0
54528
97792
185344
354304
774144
1.44998e+06
0
0
0
0
0
0
33024
56576
110080
214016
382976
823296
1.30253e+06
0
0
0
0
0
20096
34048
63232
113152
216064
485376
733184
1.90874e+06
0
0
0
0
12096
20352
37632
68096
128512
309248
423936
1.08954e+06 7.70048e+06
0
0
0
6368
11840
20864
41728
78336
177152
222208
610304
5.0135e+06 8.45414e+06
0
0
3120
6048
11200
22144
44288
103936
104960
350208
2.05619e+06 4.68582e+06 7.96262e+06
137
796
2448
2800
5472
13376
24960
22656
20864
19840
18816
19584
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
0
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
0.8
0
0
0
0
0
0
0
0
0
0
0
0.7
0.8
0
0
0
0
0
0
0
0
0
0
0.7
0.7
0.9
0
0
0
0
0
0
0
0
0
0.7
0.7
0.9
1.2
0
0
0
0
0
0
0
0
0.7
0.7
0.9
1.2
2
0
0
0
0
0
0
0
0.7
0.7
0.8
1.1
1.9
2.8
0
0
0
0
0
0
0.7
0.7
0.8
1.1
2
2.8
3.8
0
0
0
0
0
0.7
0.7
0.8
1.1
2
2.8
3.8
5.4
0
0
0
0
0.8
0.7
0.9
1.1
2
2.7
3.8
5.1
7.3
0
0
0
0.7
0.7
0.9
1.2
2
2.8
3.8
5.1
7.9
13.3
0
0
0.7
0.7
0.9
1.1
1.9
2.9
4
4.9
8.3
13.4
26.8
0
0.8
0.8
0.9
1.1
2
3
4.3
5.6
9.3
13.9
27.8
50.2
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
7500000 6000000 4500000 3000000 1500000 0
100 80 60 40 20 0
Software RAID10 over 8 Intel P3700 as blockdevice (Test ID 9) Random Read/Write (70/30) 8kB
queue depth
IOPS, reqs/s 2048 1024 512 256 128 64 32 16 8 4 2 1
1981
0
0
0
0
0
0
0
0
0
0
0
1933
2111
0
0
0
0
0
0
0
0
0
0
2000
1967
1970
0
0
0
0
0
0
0
0
0
1968
1988
1944
2033
0
0
0
0
0
0
0
0
1970
2007
1971
2196
2008
0
0
0
0
0
0
0
2086
1990
1968
1987
2028
2192
0
0
0
0
0
0
2008
1985
2076
2008
2065
2133
2330
0
0
0
0
0
1933
1964
1981
1999
2163
2161
2166
2165
0
0
0
0
1953
1983
1986
1994
2155
2165
2126
2088
2058
0
0
0
1904
1983
2032
1965
2010
2102
2154
2075
2062
1974
0
0
1764
1941
1941
1993
2024
2036
2014
2017
1969
1981
1978
0
1395
1890
2042
2008
1943
1990
1999
2249
1987
2011
1997
1925
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
2000 1600 1200 800 400 0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
395264
0
0
0
0
0
0
0
0
0
0
0
218112
370688
0
0
0
0
0
0
0
0
0
0
127488
209920
411648
0
0
0
0
0
0
0
0
0
68096
119296
228352
444416
0
0
0
0
0
0
0
0
38144
70144
134144
236544
522240
0
0
0
0
0
0
0
23424
42240
85504
154624
301056
536576
0
0
0
0
0
0
15040
25728
45312
98816
179200
358400
536576
0
0
0
0
0
9408
17280
31360
59136
115200
230400
374784
847872
0
0
0
0
6112
9536
19840
39680
74240
158720
234496
651264
4.04685e+06
0
0
0
3248
6112
12352
27264
54528
132096
148480
514048
2.67059e+06 6.52083e+06
0
0
2960
3280
8512
15424
33536
81408
93696
378880
1.20422e+06 2.8672e+06 5.7344e+06
153
167
2128
2512
3536
13120
23936
23168
23168
21376
18304
20608
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
0.8
0
0
0
0
0
0
0
0
0
0
0
0.8
0.8
0
0
0
0
0
0
0
0
0
0
0.7
0.8
1
0
0
0
0
0
0
0
0
0
0.8
0.8
0.9
1.2
0
0
0
0
0
0
0
0
0.8
0.8
0.9
1.3
2.1
0
0
0
0
0
0
0
0.7
0.8
0.9
1.2
2
2.9
0
0
0
0
0
0
0.7
0.7
0.9
1.2
2
2.9
4.1
0
0
0
0
0
0.7
0.9
0.9
1.3
2
2.9
3.9
5
0
0
0
0
0.8
0.8
1
1.2
2.1
3
4
4.9
7.6
0
0
0
0.9
0.9
1
1.3
2.1
3.1
4.2
5.4
8.4
13.6
0
0
1
0.9
1.1
1.3
2.2
3.3
4.3
5.8
8.2
14.1
27.5
0
1.2
1
1.1
1.4
2
3.3
4.6
6.3
8.7
14.4
27.7
52.7
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
4500000
3000000
1500000
0
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
6000000
0
100 80 60 40 20 0
Software RAID10 over 8 Intel P3700 as blockdevice (Test ID 9) Sequential Read 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
7.95741e+06
0
8.03404e+06 1.42655e+07
1e7
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
7.8686e+06 1.50128e+07 1.59755e+07
6.23515e+06 1.52286e+07 1.56438e+07 1.56388e+07
8.7033e+06 1.39059e+07 1.61209e+07 1.6426e+07 1.78593e+07
7.66225e+06 1.46502e+07 1.62695e+07 1.72692e+07 1.82998e+07 1.52569e+07
8.00367e+06 1.39433e+07 1.54791e+07 1.79349e+07 1.85981e+07 1.49299e+07 1.57666e+07
5.65288e+06 9.82158e+06 1.32616e+07 1.75389e+07 1.76492e+07 1.23096e+07 1.20608e+07 1.10387e+07
2.75332e+06 6.30159e+06 9.36175e+06 1.31313e+07 1.65624e+07 1.02676e+07 9.96573e+06 1.02723e+07 8.72039e+06
2.17992e+06 2.5566e+06 4.96918e+06 7.94734e+06 1.19585e+07 1.15465e+07 1.00619e+07 9.65216e+06 8.31824e+06 8.59603e+06
1.33468e+06 1.07028e+06 2.32929e+06 4.48863e+06 6.98175e+06 1.03426e+07 9.17853e+06 9.49851e+06 8.51035e+06 8.57341e+06 1.23215e+07 1.42578e+06
544601
1
2
8
16
32
64
128
number of workers
256
512
1024
2048
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
51456
0
0
0
0
0
0
0
0
0
0
0
25472
31616
0
0
0
0
0
0
0
0
0
0
13632
13632
189440
0
0
0
0
0
0
0
0
0
6432
7840
48896
128512
0
0
0
0
0
0
0
0
2992
3568
23424
49408
97792
0
0
0
0
0
0
0
1624
2160
14016
26240
45824
148480
0
0
0
0
0
0
876
1192
4080
12480
20096
75264
224256
0
0
0
0
0
596
716
1784
3120
13120
37120
162816
444416
0
0
0
0
612
748
860
1688
4320
23680
148480
228352
342016
0
0
0
564
708
716
1032
1960
7968
24448
132096
181248
370688
0
0
548
676
668
828
1288
2320
16512
28544
92672
230400
501760
0
187
644
620
708
964
1208
3440
12224
36608
96768
228352
493568
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
3.3
0
0
0
0
0
0
0
0
0
0
0
3.3
5.5
0
0
0
0
0
0
0
0
0
0
3.3
5.4
7.5
0
0
0
0
0
0
0
0
0
3
5.3
7.3
8.8
0
0
0
0
0
0
0
0
3
5.4
7.4
9.2
12.7
0
0
0
0
0
0
0
3.1
5.1
6.8
9.1
10.8
13.8
0
0
0
0
0
0
3.2
4.9
6.7
8.6
10.4
14.2
14.1
0
0
0
0
0
2
3.2
4.4
7.1
9.1
12.7
14.2
16.3
0
0
0
0
1.9
2.5
3.6
4.9
8.1
13.3
13.8
17.3
18.2
0
0
0
1.7
1.8
2.5
4.2
6.7
12.2
16.3
15.1
19
18
0
0
1.6
1.7
2.2
3.2
6.5
11.9
16.1
16.7
18.8
18.2
16
0
1.1
1.2
1.6
2.8
5.5
11.4
16.1
17.5
17.9
18.6
16.8
13.8
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
1.2 0.8 0.4
0
1.2059e+06 2.2529e+06 3.94941e+06 6.59557e+06 8.37842e+06 9.00501e+06 8.577e+06 7.94709e+06 1.05165e+07 1.21488e+07
4
1.6
0.0
500000 400000 300000 200000 100000 0
100 80 60 40 20 0
Software RAID10 over 8 Intel P3700 as blockdevice (Test ID 9) Sequential Write 128kB
queue depth
Bandwidth, kB/s 2048 1024 512 256 128 64 32 16 8 4 2 1
2.35397e+06
0
1.95807e+06 4.86989e+06
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1.93216e+06 4.86953e+06 7.38487e+06
3.12233e+06 6.06328e+06 7.43625e+06 7.32870e+06
4.74566e+06 7.39832e+06 7.50304e+06 7.67008e+06 7.20926e+06
4.86132e+06 7.44655e+06 7.66079e+06 7.71436e+06 7.66369e+06 6.79904e+06
5.03562e+06 7.11962e+06 7.52429e+06 7.69905e+06 7.70228e+06 6.94957e+06 6.52036e+06
5.46687e+06 6.86353e+06 7.15726e+06 7.58673e+06 7.75778e+06 6.98804e+06 6.75495e+06 6.08884e+06
5.47857e+06 6.64424e+06 6.85031e+06 7.19463e+06 7.59238e+06 6.64332e+06 6.35503e+06 6.25237e+06 5.97166e+06
4.41662e+06 6.15112e+06 6.40361e+06 6.60972e+06 7.23748e+06 6.69201e+06 6.64176e+06 6.18879e+06 6.17767e+06 6.28855e+06
2.04036e+06 4.92398e+06 5.87717e+06 6.04727e+06 6.70252e+06 6.1162e+06 6.49678e+06 6.24129e+06 6.24823e+06 6.49212e+06 6.36513e+06
2
4
8
16
32
64
128
number of workers
6000000 4500000 3000000 1500000
0
1.14121e+06 2.42436e+06 4.31136e+06 5.6038e+06 6.23926e+06 5.87851e+06 6.2076e+06 6.25373e+06 6.17597e+06 6.27145e+06 6.45381e+06 6.26006e+06
1
7500000
256
512
1024
2048
0
queue depth
IO Latency, us (99.5% Quantile) 2048 1024 512 256 128 64 32 16 8 4 2 1
177152
0
0
0
0
0
0
0
0
0
0
0
88576
77312
0
0
0
0
0
0
0
0
0
0
49920
41728
129536
0
0
0
0
0
0
0
0
0
22656
21888
51968
238592
0
0
0
0
0
0
0
0
16512
17536
47872
105984
288768
0
0
0
0
0
0
0
13760
14144
21120
45824
173056
602112
0
0
0
0
0
0
2672
12864
14144
24448
65280
366592
864256
0
0
0
0
0
1192
7264
12864
14656
18560
111104
333824
391168
0
0
0
0
450
2224
5408
12736
13760
26496
175104
536576
888832
0
0
0
227
924
2160
7136
12608
16768
19584
203776
423936
561152
0
0
255
175
1048
2096
8384
12864
15680
21632
173056
387072
522240
0
197
179
189
964
2544
7456
14400
15680
31872
113152
321536
995328
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
queue depth
CPU Utilization, % 2048 1024 512 256 128 64 32 16 8 4 2 1
2.7
0
0
0
0
0
0
0
0
0
0
0
2.7
4.7
0
0
0
0
0
0
0
0
0
0
2.7
4.8
8.7
0
0
0
0
0
0
0
0
0
2.7
5.1
8.5
10.3
0
0
0
0
0
0
0
0
2.9
4.6
6.9
12.4
11.2
0
0
0
0
0
0
0
2.9
4.6
6.3
9.9
11.7
12.4
0
0
0
0
0
0
2.8
4.6
5.4
7.2
8.7
10.9
14
0
0
0
0
0
2.8
3.7
4.8
5.7
7.6
10.4
12.8
17.8
0
0
0
0
2.7
3.7
4.6
5.9
6
12.1
12.7
13.6
16.1
0
0
0
2.6
3.5
4.6
5.2
5.5
10.9
13.5
14.3
14.9
16.6
0
0
2
3.3
4.1
5.1
5.3
12.6
13.4
15.8
14.9
14.9
16.1
0
1.4
2
3.5
5
4.7
13
12.6
14.9
15.4
15.6
15.4
16.1
1
2
4
8
16
32
64
128
256
512
1024
2048
number of workers
800000 600000 400000 200000 0
100 80 60 40 20 0
