SWX-IMI Latency Tests

Overview

This latency report details the performance of the Enyx SWX-IMI nxFeed support.

Setup Configuration

Schematics

_images/10Gb_nxFeed_wire_to_wire-latency_measure_setup_diagram.png

This testing configuration uses two different servers:

  • A host server holding an Enyx FPGA board and running a configuration application.
  • A replay and capture server responsible for the replay of market data captures at defined speeds and the capture of timestamped input and output feeds forwarded by the timestamping device.

On the host server, the Enyx FPGA board is configured to process incoming raw market data traffic and output a normalized and filtered Enyx MDP feed back to network.

Thanks to a layer 1 switch, both the raw market data feed and the normalized output feeds are forwarded to timestamping device for precise timestamping and then forwarded to the capture server for storage and further analysis.

The analysis of the timestamped data is used for the latency measures but also for the extraction of the effective rate at which the replay server was able to replay the Raw market data into the solution.

Host Server characteristics

CPU 1 model & version Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
CPU 1 PCIe devices N/A
CPU 2 model & version Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
CPU 2 PCIe devices Enyx FPB2

Replay & Capture Server characteristics

CPU 1 model & version Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
CPU 1 PCIe devices Solarflare SFC9220 10G Adapter Solarflare SFC9220 10G Adapter
CPU 2 model & version Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
CPU 2 PCIe devices N/A

Layer 1 device model & version

Model Metamako MetaMux 48 with K-Series Plus
Version 0.17.4
Port Speed 1Gb/10Gb
Port to port advertised latency 5 ns

Layer 2 device model & version

Model Arista 7150s
Version EOS-4.14.6M
Port Speed 1Gb/10Gb
Timestamping resolution 2.857 nanoseconds
Timestamping trigger First Byte of the FCS

Enyx solution characteristics:

FPGA card version Enyx FPB2
Firmware version SWX_IMI_318 2.8.0
Software version libenyxmd 3.7.0
Driver version hfp 2.1.0
Thread Binding CPU ID 25 (Numa 1)
Card NUMA node 1

Capture characteristics

Capture swx_B_20190510_open-to-close
Packet Count 42243388
Channel Count 1
Capture Duration 9:59:59
Beggining Date Fri, 10 May 2019 08:00:00
End date Fri, 10 May 2019 17:59:59
Timestamping resolution Microseconds
Packet Rate Peak (s) 8608 pkt/s
Packet Rate Peak (10 ms) 97400 pkt/s
Packet Rate Peak (1 ms) 138000 pkt/s
Bit Rate Peak (s) 9.08 Mbps
Bit Rate Peak (10 ms) 434.10 Mbps
Bit Rate Peak (1 ms) 434.10 Mbps
_images/swx_B_20190510_open-to-close_global_Packet_Rate.png _images/swx_B_20190510_open-to-close_global_Bit_Rate.png

Test Overview

The test has been replayed with these parameters:

  • Replay speeds: x1, x2, x10
  • Subscribed instruments:
    • Top 20: Most active 20 symbols for this capture.
    • Top 1000: Most active 1000 symbols for this capture.
    • Top 8000: Most active 8000 symbols for this capture.

Scenario #1: Top 20 symbols

Result Summary

Replay Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
Replay #1 1121 ns 1228 ns 1289 ns 1267 ns 1323 ns 1337 ns 1343 ns 1348 ns 3029 ns 3029 ns
Replay #2 1121 ns 1228 ns 1289 ns 1267 ns 1323 ns 1337 ns 1343 ns 1348 ns 3017 ns 3017 ns
Replay #3 1124 ns 1225 ns 1287 ns 1266 ns 1320 ns 1337 ns 1343 ns 1351 ns 4020 ns 4020 ns

Replay #1: 1 Time(s) Market Rate

Conditions

Instrument Count 20
Subscribed Channel Count 1
Instruments List 20 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 10
Requested Replay Rate 1

Observed Replay Rates

_images/Latencies_20190510_20_instruments_x1_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 1024 pkt/s 0.87
Average Packet Rate (10ms resolution) 1993 pkt/s 0.97
Peak Packet Rate (1s resolution) 8610 pkt/s 1.00
Peak Packet Rate (10ms resolution) 97400 pkt/s 1.00
Peak Packet Rate (1ms resolution) 138000 pkt/s 1.00
_images/Latencies_20190510_20_instruments_x1_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 1.19 Mbps 0.90
Average Bit Rate (10ms resolution) 2.37 Mbps 1.00
Average Bit Rate (1ms resolution) 5.36 Mbps 1.01
Peak Bit Rate (1s resolution) 9.36 Mbps 1.03
Peak Bit Rate (10ms resolution) 435.95 Mbps 1.00
Peak Bit Rate (1ms resolution) 725.74 Mbps 1.00

Results

Sample Distribution
_images/SWX-IMI-ethernet-hfp_Latencies_20190510_20_instruments_x1-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
1121 ns 1228 ns 1289 ns 1267 ns 1323 ns 1337 ns 1343 ns 1348 ns 3029 ns 3029 ns

General Statistics  
Output Packet Count 2284333
Input FIFO maximal usage 1.29 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #2: 2 Time(s) Market Rate

Conditions

Instrument Count 20
Subscribed Channel Count 1
Instruments List 20 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 10
Requested Replay Rate 2

Observed Replay Rates

_images/Latencies_20190510_20_instruments_x2_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 2049 pkt/s 1.74
Average Packet Rate (10ms resolution) 3677 pkt/s 1.79
Peak Packet Rate (1s resolution) 11654 pkt/s 1.35
Peak Packet Rate (10ms resolution) 140300 pkt/s 1.44
Peak Packet Rate (1ms resolution) 260000 pkt/s 1.88
_images/Latencies_20190510_20_instruments_x2_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 2.38 Mbps 1.79
Average Bit Rate (10ms resolution) 4.32 Mbps 1.82
Average Bit Rate (1ms resolution) 8.31 Mbps 1.56
Peak Bit Rate (1s resolution) 12.74 Mbps 1.40
Peak Bit Rate (10ms resolution) 460.12 Mbps 1.06
Peak Bit Rate (1ms resolution) 1408.16 Mbps 1.95

Results

Sample Distribution
_images/SWX-IMI-ethernet-hfp_Latencies_20190510_20_instruments_x2-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
1121 ns 1228 ns 1289 ns 1267 ns 1323 ns 1337 ns 1343 ns 1348 ns 3017 ns 3017 ns

General Statistics  
Output Packet Count 2284333
Input FIFO maximal usage 1.27 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #3: 10 Time(s) Market Rate

Conditions

Instrument Count 20
Subscribed Channel Count 1
Instruments List 20 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 10
Requested Replay Rate 10

Observed Replay Rates

_images/Latencies_20190111_20_instruments_x10_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 10235 pkt/s 8.71
Average Packet Rate (10ms resolution) 15282 pkt/s 7.43
Peak Packet Rate (1s resolution) 42169 pkt/s 4.90
Peak Packet Rate (10ms resolution) 248400 pkt/s 2.55
Peak Packet Rate (1ms resolution) 974000 pkt/s 7.06
_images/Latencies_20190111_20_instruments_x10_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 11.90 Mbps 8.95
Average Bit Rate (10ms resolution) 17.73 Mbps 7.48
Average Bit Rate (1ms resolution) 27.50 Mbps 5.16
Peak Bit Rate (1s resolution) 47.71 Mbps 5.25
Peak Bit Rate (10ms resolution) 522.98 Mbps 1.20
Peak Bit Rate (1ms resolution) 4359.46 Mbps 6.02

Results

Sample Distribution
_images/SWX-IMI-ethernet-hfp_Latencies_20190111_20_instruments_x10-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
1124 ns 1225 ns 1287 ns 1266 ns 1320 ns 1337 ns 1343 ns 1351 ns 4020 ns 4020 ns

General Statistics  
Output Packet Count 2284333
Input FIFO maximal usage 4.53 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Scenario #2: Top 1000 symbols

Result Summary

Replay Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
Replay #1 1118 ns 1225 ns 1284 ns 1252 ns 1306 ns 1329 ns 1340 ns 1343 ns 1357 ns 2475 ns
Replay #2 1118 ns 1225 ns 1284 ns 1252 ns 1306 ns 1329 ns 1340 ns 1345 ns 1403 ns 2501 ns
Replay #3 1121 ns 1225 ns 1281 ns 1253 ns 1304 ns 1328 ns 1340 ns 2639 ns 3962 ns 6022 ns

Replay #1: 1 Time(s) Market Rate

Conditions

Instrument Count 1000
Subscribed Channel Count 1
Instruments List 1000 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 10
Requested Replay Rate 1

Observed Replay Rates

_images/Latencies_20190510_1k_instruments_x1_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 1024 pkt/s 0.87
Average Packet Rate (10ms resolution) 1993 pkt/s 0.97
Peak Packet Rate (1s resolution) 8610 pkt/s 1.00
Peak Packet Rate (10ms resolution) 97400 pkt/s 1.00
Peak Packet Rate (1ms resolution) 138000 pkt/s 1.00
_images/Latencies_20190510_1k_instruments_x1_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 1.19 Mbps 0.90
Average Bit Rate (10ms resolution) 2.37 Mbps 1.00
Average Bit Rate (1ms resolution) 5.37 Mbps 1.01
Peak Bit Rate (1s resolution) 9.36 Mbps 1.03
Peak Bit Rate (10ms resolution) 435.95 Mbps 1.00
Peak Bit Rate (1ms resolution) 725.74 Mbps 1.00

Results

Sample Distribution
_images/SWX-IMI-ethernet-hfp_Latencies_20190510_1k_instruments_x1-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
1118 ns 1225 ns 1284 ns 1252 ns 1306 ns 1329 ns 1340 ns 1343 ns 1357 ns 2475 ns

General Statistics  
Output Packet Count 25693085
Input FIFO maximal usage 1.29 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #2: 2 Time(s) Market Rate

Conditions

Instrument Count 1000
Subscribed Channel Count 1
Instruments List 1000 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 10
Requested Replay Rate 2

Observed Replay Rates

_images/Latencies_20190510_1k_instruments_x2_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 2050 pkt/s 1.74
Average Packet Rate (10ms resolution) 3672 pkt/s 1.79
Peak Packet Rate (1s resolution) 14056 pkt/s 1.63
Peak Packet Rate (10ms resolution) 140300 pkt/s 1.44
Peak Packet Rate (1ms resolution) 260000 pkt/s 1.88
_images/Latencies_20190510_1k_instruments_x2_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 2.38 Mbps 1.79
Average Bit Rate (10ms resolution) 4.31 Mbps 1.82
Average Bit Rate (1ms resolution) 8.30 Mbps 1.56
Peak Bit Rate (1s resolution) 15.25 Mbps 1.68
Peak Bit Rate (10ms resolution) 460.12 Mbps 1.06
Peak Bit Rate (1ms resolution) 1408.16 Mbps 1.95

Results

Sample Distribution
_images/SWX-IMI-ethernet-hfp_Latencies_20190510_1k_instruments_x2-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
1118 ns 1225 ns 1284 ns 1252 ns 1306 ns 1329 ns 1340 ns 1345 ns 1403 ns 2501 ns

General Statistics  
Output Packet Count 25693086
Input FIFO maximal usage 1.27 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #3: 10 Time(s) Market Rate

Conditions

Instrument Count 1000
Subscribed Channel Count 1
Instruments List 1000 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 10
Requested Replay Rate 10

Observed Replay Rates

_images/Latencies_20190111_1k_instruments_x10_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 10227 pkt/s 8.70
Average Packet Rate (10ms resolution) 15253 pkt/s 7.42
Peak Packet Rate (1s resolution) 45974 pkt/s 5.34
Peak Packet Rate (10ms resolution) 240900 pkt/s 2.47
Peak Packet Rate (1ms resolution) 952000 pkt/s 6.90
_images/Latencies_20190111_1k_instruments_x10_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 11.89 Mbps 8.95
Average Bit Rate (10ms resolution) 17.70 Mbps 7.46
Average Bit Rate (1ms resolution) 27.48 Mbps 5.15
Peak Bit Rate (1s resolution) 51.03 Mbps 5.62
Peak Bit Rate (10ms resolution) 522.98 Mbps 1.20
Peak Bit Rate (1ms resolution) 4360.23 Mbps 6.03

Results

Sample Distribution
_images/SWX-IMI-ethernet-hfp_Latencies_20190111_1k_instruments_x10-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
1121 ns 1225 ns 1281 ns 1253 ns 1304 ns 1328 ns 1340 ns 2639 ns 3962 ns 6022 ns

General Statistics  
Output Packet Count 25693081
Input FIFO maximal usage 4.69 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Scenario #3: Top 8000 symbols

Result Summary

Replay Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
Replay #1 847 ns 1225 ns 1284 ns 1251 ns 1304 ns 1328 ns 1340 ns 1343 ns 1364 ns 2478 ns
Replay #2 845 ns 1225 ns 1284 ns 1251 ns 1304 ns 1328 ns 1340 ns 1345 ns 1927 ns 7129 ns
Replay #3 853 ns 1222 ns 1281 ns 1251 ns 1304 ns 1328 ns 1340 ns 2702 ns 3983 ns 6031 ns

Replay #1: 1 Time(s) Market Rate

Conditions

Instrument Count 8000
Subscribed Channel Count 1
Instruments List 8000 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 10
Requested Replay Rate 1

Observed Replay Rates

_images/Latencies_20190510_8k_instruments_x1_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 1025 pkt/s 0.87
Average Packet Rate (10ms resolution) 1996 pkt/s 0.97
Peak Packet Rate (1s resolution) 8610 pkt/s 1.00
Peak Packet Rate (10ms resolution) 97400 pkt/s 1.00
Peak Packet Rate (1ms resolution) 138000 pkt/s 1.00
_images/Latencies_20190510_8k_instruments_x1_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 1.19 Mbps 0.90
Average Bit Rate (10ms resolution) 2.37 Mbps 1.00
Average Bit Rate (1ms resolution) 5.38 Mbps 1.01
Peak Bit Rate (1s resolution) 9.36 Mbps 1.03
Peak Bit Rate (10ms resolution) 435.95 Mbps 1.00
Peak Bit Rate (1ms resolution) 725.74 Mbps 1.00

Results

Sample Distribution
_images/SWX-IMI-ethernet-hfp_Latencies_20190510_8k_instruments_x1-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
847 ns 1225 ns 1284 ns 1251 ns 1304 ns 1328 ns 1340 ns 1343 ns 1364 ns 2478 ns

General Statistics  
Output Packet Count 37003642
Input FIFO maximal usage 1.27 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #2: 2 Time(s) Market Rate

Conditions

Instrument Count 8000
Subscribed Channel Count 1
Instruments List 8000 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 10
Requested Replay Rate 2

Observed Replay Rates

_images/Latencies_20190510_8k_instruments_x2_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 2051 pkt/s 1.75
Average Packet Rate (10ms resolution) 3676 pkt/s 1.79
Peak Packet Rate (1s resolution) 14056 pkt/s 1.63
Peak Packet Rate (10ms resolution) 140300 pkt/s 1.44
Peak Packet Rate (1ms resolution) 260000 pkt/s 1.88
_images/Latencies_20190510_8k_instruments_x2_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 2.38 Mbps 1.79
Average Bit Rate (10ms resolution) 4.32 Mbps 1.82
Average Bit Rate (1ms resolution) 8.31 Mbps 1.56
Peak Bit Rate (1s resolution) 15.25 Mbps 1.68
Peak Bit Rate (10ms resolution) 460.12 Mbps 1.06
Peak Bit Rate (1ms resolution) 1408.16 Mbps 1.95

Results

Sample Distribution
_images/SWX-IMI-ethernet-hfp_Latencies_20190510_8k_instruments_x2-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
845 ns 1225 ns 1284 ns 1251 ns 1304 ns 1328 ns 1340 ns 1345 ns 1927 ns 7129 ns

General Statistics  
Output Packet Count 37002848
Input FIFO maximal usage 1.27 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #3: 10 Time(s) Market Rate

Conditions

Instrument Count 8000
Subscribed Channel Count 1
Instruments List 8000 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 10
Requested Replay Rate 10

Observed Replay Rates

_images/Latencies_20190111_8k_instruments_x10_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 10236 pkt/s 8.71
Average Packet Rate (10ms resolution) 15242 pkt/s 7.41
Peak Packet Rate (1s resolution) 46193 pkt/s 5.37
Peak Packet Rate (10ms resolution) 248400 pkt/s 2.55
Peak Packet Rate (1ms resolution) 973000 pkt/s 7.05
_images/Latencies_20190111_8k_instruments_x10_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 11.90 Mbps 8.96
Average Bit Rate (10ms resolution) 17.69 Mbps 7.46
Average Bit Rate (1ms resolution) 27.43 Mbps 5.14
Peak Bit Rate (1s resolution) 51.22 Mbps 5.64
Peak Bit Rate (10ms resolution) 522.98 Mbps 1.20
Peak Bit Rate (1ms resolution) 4360.23 Mbps 6.03

Results

Sample Distribution
_images/SWX-IMI-ethernet-hfp_Latencies_20190111_8k_instruments_x10-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
853 ns 1222 ns 1281 ns 1251 ns 1304 ns 1328 ns 1340 ns 2702 ns 3983 ns 6031 ns

General Statistics  
Output Packet Count 37002029
Input FIFO maximal usage 4.53 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time