DragonFly Basic NVME Driver Tests 06-June-2016 These tests are on a Xeon E31270 @ 3.4 GHz (4-core/8-thread) box with 16G of ram. This box appears to have somewhat older PCIe slots and throughput is limited to around 1.5GBytes/sec. Note that (unshown) tests on one card in a box with a PCIe-3.0 slot was able to achieve 2.5GBytes/sec of throughput. The primary purpose of these tests is to check interrupt vector distribution and processing across multiple cpus under a SMP load for our nvme driver. All tests are performed on /dev/nvme{0,1}s1b, a 16GB 'swap' partition created within the nvme SSD. Most tests are performed on two controllers simultaneously (nvme0 and nvme1). Most tests are performed after having filling the entire partition with uncompressable /dev/urandom data. These particular tests used two Samsung M.2 x 4 nvme SSDs (128G each), mounted on an adapter card and plugged into a PCIex8 slot. This machine has four PCIe slots. nvme0: port 0xe000-0xe0ff mem 0xdf400000-0xdf403fff irq 19 at device 0.0 on pci2 nvme0: mapped 9 MSIX IRQs nvme0: NVME Version 1.1 maxqe=16384 caps=00f000203c013fff nvme0: Model SAMSUNG_MZVPV128HDGM-00000 BaseSerial S1XVNYAGA03031 nscount=1 nvme0: Request 34/10 queues, Returns 8/8 queues, nominal map 1:1 cpu nvme0: Disk nvme0 ns=1 blksize=512 lbacnt=250069680 cap=119GB serno=S1XVNYAGA03031-1 nvme1: port 0xd000-0xd0ff mem 0xdf300000-0xdf303fff irq 16 at device 0.0 on pci3 nvme1: mapped 9 MSIX IRQs nvme1: NVME Version 1.1 maxqe=16384 caps=00f000203c013fff nvme1: Model SAMSUNG_MZVPV128HDGM-00000 BaseSerial S1XVNYAGA02988 nscount=1 nvme1: Request 34/10 queues, Returns 8/8 queues, nominal map 1:1 cpu nvme1: Disk nvme1 ns=1 blksize=512 lbacnt=250069680 cap=119GB serno=S1XVNYAGA02988-1 systat -pv 1 timer ipi extint user% sys% intr% idle% smpcol label total 175 519813 7240 cpu0 240 12 72686 6.2 60.8 9.2 23.8 2171 Xrelpbuf cpu1 233 8 65798 4.6 72.3 13.8 9.2 790 Xgetpbuf_kva cpu2 2329 9 63629 3.8 73.1 13.8 9.2 644 Xrelpbuf cpu3 232 8 63032 4.6 68.5 19.2 7.7 825 Xgetpbuf_kva cpu4 232 46 65068 4.6 79.2 4.6 11.5 706 Xgetpbuf_kva cpu5 232 53 63365 6.2 68.5 16.2 9.2 723 Xgetpbuf_kva cpu6 233 16 62750 3.8 72.3 12.3 11.5 629 Xrelpbuf cpu7 2327 23 63485 9.2 72.3 10.8 7.7 752 X_vm_page_queue 8 4K dd's on nvme0 only tty nvme0 nvme1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id ZEROD PARTITION 0 65 4.00 324524 1267.67 0.00 0 0.00 2 0 32 4 62 0 65 4.00 324529 1267.69 0.00 0 0.00 2 0 31 5 61 0 73 4.00 324531 1267.70 0.00 0 0.00 2 0 32 5 62 0 67 4.00 324559 1267.81 0.00 0 0.00 2 0 32 5 61 URANDOM FILLED PARTITION 0 69 4.00 88687 346.43 0.00 0 0.00 1 0 17 2 80 0 59 4.00 88742 346.64 0.00 0 0.00 1 0 16 2 81 0 66 4.00 88393 345.28 0.00 0 0.00 1 0 17 1 81 0 68 4.00 88476 345.61 0.00 0 0.00 2 0 15 2 81 0 58 4.00 88760 346.71 0.00 0 0.00 1 0 16 2 81 URANDOM FILLED PARTITION (32 PROCESSES) 0 37 4.00 266055 1039.28 0.00 0 0.00 0 0 5 1 94 0 64 4.00 281370 1099.10 0.00 0 0.00 2 0 31 5 63 0 68 4.00 281500 1099.60 0.00 0 0.00 3 0 29 5 63 0 74 4.00 280478 1095.62 0.00 0 0.00 2 0 28 4 65 8 512B dd's on nvme0, 8 512B dd's on nvme1 tty nvme0 nvme1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id ZEROD PARTITION 0 79 0.50 283324 138.34 0.50 260055 126.98 8 0 68 15 10 0 73 0.50 284386 138.86 0.50 259894 126.90 6 0 70 14 11 0 75 0.50 284177 138.76 0.50 260064 126.98 7 0 71 12 10 0 69 0.50 284699 139.01 0.50 259924 126.92 6 0 71 14 9 0 66 0.50 282526 137.95 0.50 260630 127.26 5 0 71 13 11 URANDOM FILLED PARTITION 0 64 0.50 86899 42.43 0.50 86362 42.17 2 0 27 4 66 0 67 0.50 88552 43.24 0.50 87829 42.88 2 0 27 3 68 0 68 0.50 87185 42.57 0.50 87182 42.57 2 0 28 4 66 0 70 0.50 88160 43.05 0.50 87967 42.95 2 0 26 4 69 URANDOM FILLED PARTITION (32 PROCESSES) 0 71 0.50 250047 122.09 0.50 241546 117.94 6 0 59 11 23 0 73 0.50 245239 119.74 0.50 247457 120.82 6 0 62 11 21 0 73 0.50 245525 119.89 0.50 236903 115.67 4 0 60 12 24 0 69 0.50 249730 121.94 0.50 241724 118.03 5 0 63 11 22 0 69 0.50 246290 120.26 0.50 243412 118.85 5 0 62 10 23 URANDOM FILLED PARTITION (RANDOM SEEK 32 PROCESSES) 0 87 0.50 228742 111.69 0.50 227609 111.13 5 0 54 9 32 0 77 0.50 229613 112.12 0.50 226827 110.76 4 0 55 12 29 0 88 0.50 229292 111.96 0.50 227365 111.02 4 0 57 9 29 0 82 0.50 228551 111.59 0.50 228035 111.34 4 0 57 9 29 -- 8 4KB dd's on nvme0, 8 4K dd's on nvme1 tty nvme0 nvme1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id ZEROD PARTITION 0 73 4.00 263896 1030.84 4.00 246650 963.47 5 0 69 13 14 0 78 4.00 265367 1036.59 4.00 248109 969.19 6 0 70 13 11 0 78 4.00 269253 1051.77 4.00 247252 965.83 5 0 70 13 12 0 69 4.00 269312 1052.00 4.00 246989 964.80 5 0 71 13 12 0 69 4.00 268812 1050.05 4.00 247149 965.42 5 0 70 12 12 0 73 4.00 267910 1046.52 4.00 247490 966.76 5 0 71 13 10 URANDOM FILLED PARTITION 0 91 4.00 87586 342.12 4.00 86509 337.93 2 0 31 3 64 0 76 4.00 88281 344.84 4.00 86818 339.14 3 0 32 5 61 0 64 4.00 87937 343.51 4.00 85922 335.63 2 0 32 5 61 0 67 4.00 87978 343.67 4.00 86243 336.90 2 0 32 5 61 URANDOM FILLED PARTITION (32 PROCESSES) 0 77 4.00 244029 953.24 4.00 240877 940.92 6 0 68 12 14 0 69 4.00 245030 957.14 4.00 241423 943.05 5 0 69 12 14 0 73 4.00 242647 947.83 4.00 240328 938.77 6 0 65 14 15 0 75 4.00 244610 955.50 4.00 239898 937.09 5 0 70 10 15 0 74 4.00 244536 955.21 4.00 240391 939.02 6 0 68 12 15 URANDOM FILLED PARTITION (RANDOM SEEK 32 PROCESSES) 0 90 4.00 226607 885.17 4.00 224046 875.17 6 0 55 10 29 0 85 4.00 225765 881.88 4.00 225540 881.01 4 0 57 9 30 0 84 4.00 226100 883.20 4.00 224942 878.68 4 0 54 10 31 0 88 4.00 225500 880.86 4.00 225264 879.93 5 0 53 10 31 0 88 4.00 225853 882.23 4.00 225500 880.85 4 0 54 11 30 -- 8 16KB dd's on nvme0, 8 16K dd's on nvme1 tty nvme0 nvme1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id ZEROD PARTITION (not avail) URANDOM FILLED PARTITION 0 73 16.00 59886 935.71 16.00 58487 913.85 1 0 24 2 74 0 74 16.00 59967 936.99 16.00 58612 915.82 1 0 22 3 74 0 59 16.00 59924 936.31 16.00 58386 912.28 0 0 24 2 74 0 72 16.00 59810 934.53 16.00 58287 910.73 1 0 23 2 74 URANDOM FILLED PARTITION (32 PROCESSES) 0 72 16.00 103782 1621.59 16.00 102605 1603.20 1 0 25 3 72 0 71 16.00 103783 1621.62 16.00 102632 1603.63 0 0 25 2 72 0 73 16.00 103783 1621.61 16.00 102632 1603.63 1 0 27 2 70 0 74 16.00 103783 1621.61 16.00 102633 1603.64 0 0 27 2 71 URANDOM FILLED PARTITION (RANDOM SEEK 32 PROCESSES) 0 81 16.00 102288 1598.26 16.00 101180 1580.93 1 0 24 2 72 0 83 16.00 102323 1598.80 16.00 101374 1583.96 1 0 24 3 72 0 82 16.00 102358 1599.35 16.00 101220 1581.56 1 0 24 2 73 0 81 16.00 102160 1596.25 16.00 101087 1579.49 1 0 24 3 72 -- 8 32K dd's on nvme0, 8 32K dd's on nvme1 tty nvme0 nvme1 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id ZEROD PARTITION 0 74 32.00 49941 1560.65 32.00 49098 1534.31 1 0 19 2 79 0 71 32.00 49930 1560.31 32.00 49087 1533.97 0 0 18 2 80 0 61 32.00 49921 1560.02 32.00 49078 1533.67 1 0 16 1 81 0 71 32.00 49944 1560.75 32.00 49102 1534.43 0 0 17 1 81 0 70 32.00 49963 1561.34 32.00 49122 1535.06 1 0 17 1 80 0 68 32.00 49875 1558.59 32.00 49029 1532.16 0 0 19 1 79 URANDOM FILLED PARTITION 0 72 32.00 44250 1382.82 32.00 43621 1363.17 1 0 19 1 79 0 71 32.00 44378 1386.83 32.00 43569 1361.54 1 0 19 1 78 0 64 32.00 44387 1387.09 32.00 43352 1354.75 1 0 17 2 81 0 70 32.00 44269 1383.39 32.00 43422 1356.93 0 0 21 1 78 URANDOM FILLED PARTITION (32 PROCESSES) 0 68 32.00 50194 1568.55 32.00 49579 1549.33 1 0 19 1 79 0 70 32.00 50205 1568.90 32.00 49589 1549.65 1 0 18 1 80 0 74 32.00 50217 1569.28 32.00 49602 1550.06 1 0 17 1 81 0 75 32.00 50235 1569.84 32.00 49621 1550.65 1 0 17 2 81 URANDOM FILLED PARTITION (RANDOM SEEK 32 PROCESSES) 0 82 32.00 50212 1569.13 32.00 49499 1546.85 0 0 18 1 81 0 88 32.00 50196 1568.64 32.00 49481 1546.27 1 0 18 1 80 0 83 32.00 50205 1568.90 32.00 49522 1547.56 1 0 17 1 81 0 82 32.00 50210 1569.05 32.00 49500 1546.87 1 0 19 1 79 -- 8 64K dd's on nvme0, 8 64K dd's on nvme1 URANDOM FILLED PARTITION 0 61 64.00 24141 1508.79 64.00 23606 1475.35 0 0 16 0 83 0 58 64.00 24138 1508.61 64.00 23601 1475.04 0 0 16 1 83 0 60 64.00 24123 1507.72 64.00 23588 1474.22 0 0 15 1 84 0 64 64.00 24128 1508.01 64.00 23592 1474.51 0 0 15 1 84 URANDOM FILLED PARTITION (32 PROCESSES) 0 70 64.00 24442 1527.64 64.00 24085 1505.33 0 0 14 0 85 0 60 64.00 24439 1527.46 64.00 24083 1505.21 0 0 13 1 87 0 62 64.00 24426 1526.60 64.00 24068 1504.23 0 0 16 1 83 0 57 64.00 24451 1528.21 64.00 24095 1505.96 0 0 17 1 82 URANDOM FILLED PARTITION (RANDOM SEEK 32 PROCESSES) 0 78 64.00 24707 1544.17 64.00 24369 1523.05 0 0 14 1 85 0 83 64.00 24708 1544.25 64.00 24369 1523.06 0 0 13 1 85 0 69 64.00 24709 1544.29 64.00 24372 1523.23 1 0 13 1 85 0 78 64.00 24698 1543.65 64.00 24373 1523.34 1 0 15 1 84