TAIFEX NxFeed Latency Tests

Overview

This latency report details the performance of the Enyx TAIFEX NxFeed support.

Setup Configuration

Schematics

_images/10Gb_nxFeed_wire_to_wire-latency_measure_setup_diagram.png

This testing configuration uses two different servers:

  • A host server holding an Enyx FPGA board and running a configuration application.
  • A replay and capture server responsible for the replay of market data captures at defined speeds and the capture of timestamped input and output feeds forwarded by the timestamping device.

On the host server, the Enyx FPGA board is configured to process incoming raw market data traffic and output a normalized and filtered NxFeed feed back to network.

Thanks to a layer 1 switch, both the raw market data feed and the normalized output feeds are forwarded to timestamping device for precise timestamping and then forwarded to the capture server for storage and further analysis.

The analysis of the timestamped data is used for the latency measures but also for the extraction of the effective rate at which the replay server was able to replay the Raw market data into the solution.

Host Server characteristics

CPU 1 model & version Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
CPU 1 PCIe devices N/A
CPU 2 model & version Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
CPU 2 PCIe devices Enyx FPB2

Replay & Capture Server characteristics

CPU 1 model & version Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
CPU 1 PCIe devices Solarflare SFC9220 10G Adapter Solarflare SFC9220 10G Adapter
CPU 2 model & version Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
CPU 2 PCIe devices N/A

Layer 1 device model & version

Model Metamako MetaMux 48 with K-Series Plus
Version 0.17.4
Port Speed 1Gb/10Gb
Port to port advertised latency 5 ns

Layer 2 device model & version

Model Arista 7150s
Version EOS-4.14.6M
Port Speed 1Gb/10Gb
Timestamping resolution 2.857 nanoseconds
Timestamping trigger First Byte of the FCS

Enyx solution characteristics:

FPGA card version Enyx FPB2
Firmware version TAIFEX 2.8.0
Software version libenyxmd 3.7.3
Driver version hfp 2.1.1
Thread Binding CPU ID 0 (Numa 0)
Card NUMA node 1

Capture characteristics

Capture LONG_Corvil-108-1518392640000000000-1518432630000000000
Packet Count 13739259
Channel Count 4
Capture Duration 10:54:28
Beggining Date Mon, 12 Feb 2018 07:56:01
End date Mon, 12 Feb 2018 18:50:29
Timestamping resolution Microseconds
Packet Rate Peak (s) 2961 pkt/s
Packet Rate Peak (10 ms) 35300 pkt/s
Packet Rate Peak (1 ms) 45000 pkt/s
Bit Rate Peak (s) 4.09 Mbps
Bit Rate Peak (10 ms) 48.10 Mbps
Bit Rate Peak (1 ms) 48.10 Mbps
_images/LONG_Corvil-108-1518392640000000000-1518432630000000000_global_Packet_Rate.png _images/LONG_Corvil-108-1518392640000000000-1518432630000000000_global_Bit_Rate.png

Test Overview

The test has been replayed with these parameters:

  • Replay speeds: x1, x2, x10
  • Subscribed instruments:
    • Top 20: Most active 20 symbols for this capture.
    • Top 1000: Most active 1000 symbols for this capture.
    • Top 2477: Most active 2477 symbols for this capture.

Scenario #1: Top 20 symbols

Result Summary

Replay Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
Replay #1 837 ns 1124 ns 1129 ns 1118 ns 1142 ns 1155 ns 1161 ns 1167 ns 1232 ns 1388 ns
Replay #2 837 ns 1124 ns 1129 ns 1118 ns 1142 ns 1155 ns 1161 ns 1167 ns 1220 ns 1416 ns
Replay #3 837 ns 1125 ns 1131 ns 1122 ns 1147 ns 1211 ns 1289 ns 1367 ns 1441 ns 1508 ns

Replay #1: 1 Time(s) Market Rate

Conditions

Instrument Count 20
Subscribed Channel Count 1
Instruments List 20 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 1

Observed Replay Rates

_images/Latency_20190510_20_instruments_x1.setup1_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 14035 pkt/s 39.09
Average Packet Rate (10ms resolution) 1386963 pkt/s 538.63
Peak Packet Rate (1s resolution) 141124 pkt/s 47.66
Peak Packet Rate (10ms resolution) 14111000 pkt/s 399.75
Peak Packet Rate (1ms resolution) 141110000 pkt/s 3135.78
_images/Latency_20190510_20_instruments_x1.setup1_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 20.30 Mbps 38.52
Average Bit Rate (10ms resolution) 2005.42 Mbps 611.20
Average Bit Rate (1ms resolution) 20037.48 Mbps 1153.63
Peak Bit Rate (1s resolution) 204.54 Mbps 49.99
Peak Bit Rate (10ms resolution) 20452.17 Mbps 425.17
Peak Bit Rate (1ms resolution) 204521.68 Mbps 3058.04

Results

Sample Distribution
_images/TAIFEX-nxfeed-ethernet-hfp_Latency_20190510_20_instruments_x1.setup1-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
837 ns 1124 ns 1129 ns 1118 ns 1142 ns 1155 ns 1161 ns 1167 ns 1232 ns 1388 ns

General Statistics  
Output Packet Count 632117
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #2: 2 Time(s) Market Rate

Conditions

Instrument Count 20
Subscribed Channel Count 1
Instruments List 20 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 2

Observed Replay Rates

_images/Latency_20190510_20_instruments_x2.setup1_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 14215 pkt/s 39.60
Average Packet Rate (10ms resolution) 1387946 pkt/s 539.01
Peak Packet Rate (1s resolution) 141088 pkt/s 47.65
Peak Packet Rate (10ms resolution) 14107400 pkt/s 399.64
Peak Packet Rate (1ms resolution) 141074000 pkt/s 3134.98
_images/Latency_20190510_20_instruments_x2.setup1_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 20.56 Mbps 39.01
Average Bit Rate (10ms resolution) 2006.43 Mbps 611.51
Average Bit Rate (1ms resolution) 20045.30 Mbps 1154.08
Peak Bit Rate (1s resolution) 204.52 Mbps 49.98
Peak Bit Rate (10ms resolution) 20450.32 Mbps 425.13
Peak Bit Rate (1ms resolution) 204503.25 Mbps 3057.76

Results

Sample Distribution
_images/TAIFEX-nxfeed-ethernet-hfp_Latency_20190510_20_instruments_x2.setup1-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
837 ns 1124 ns 1129 ns 1118 ns 1142 ns 1155 ns 1161 ns 1167 ns 1220 ns 1416 ns

General Statistics  
Output Packet Count 632117
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #3: 10 Time(s) Market Rate

Conditions

Instrument Count 20
Subscribed Channel Count 1
Instruments List 20 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 10

Observed Replay Rates

_images/Latency_20190510_20_instruments_x10.setup1_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 1855 pkt/s 5.17
Average Packet Rate (10ms resolution) 5348 pkt/s 2.08
Peak Packet Rate (1s resolution) 9966 pkt/s 3.37
Peak Packet Rate (10ms resolution) 82400 pkt/s 2.33
Peak Packet Rate (1ms resolution) 432000 pkt/s 9.60
_images/Latency_20190510_20_instruments_x10.setup1_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 2.69 Mbps 5.10
Average Bit Rate (10ms resolution) 6.46 Mbps 1.97
Average Bit Rate (1ms resolution) 52.08 Mbps 3.00
Peak Bit Rate (1s resolution) 15.26 Mbps 3.73
Peak Bit Rate (10ms resolution) 84.15 Mbps 1.75
Peak Bit Rate (1ms resolution) 487.81 Mbps 7.29

Results

Sample Distribution
_images/TAIFEX-nxfeed-ethernet-hfp_Latency_20190510_20_instruments_x10.setup1-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
837 ns 1125 ns 1131 ns 1122 ns 1147 ns 1211 ns 1289 ns 1367 ns 1441 ns 1508 ns

General Statistics  
Output Packet Count 632117
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Scenario #2: Top 1000 symbols

Result Summary

Replay Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
Replay #1 831 ns 1208 ns 1218 ns 1184 ns 1235 ns 1250 ns 1255 ns 1261 ns 1535 ns 1875 ns
Replay #2 831 ns 1208 ns 1218 ns 1184 ns 1235 ns 1250 ns 1255 ns 1261 ns 1956 ns 2261 ns
Replay #3 834 ns 1209 ns 1219 ns 1186 ns 1237 ns 1251 ns 1260 ns 1375 ns 3778 ns 5705 ns

Replay #1: 1 Time(s) Market Rate

Conditions

Instrument Count 1000
Subscribed Channel Count 1
Instruments List 1000 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 1

Observed Replay Rates

_images/Latency_20190510_1k_instruments_x1.setup1_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 14043 pkt/s 39.12
Average Packet Rate (10ms resolution) 1387764 pkt/s 538.94
Peak Packet Rate (1s resolution) 141122 pkt/s 47.66
Peak Packet Rate (10ms resolution) 14111100 pkt/s 399.75
Peak Packet Rate (1ms resolution) 141111000 pkt/s 3135.80
_images/Latency_20190510_1k_instruments_x1.setup1_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 20.31 Mbps 38.54
Average Bit Rate (10ms resolution) 2006.53 Mbps 611.54
Average Bit Rate (1ms resolution) 20048.61 Mbps 1154.28
Peak Bit Rate (1s resolution) 204.54 Mbps 49.98
Peak Bit Rate (10ms resolution) 20452.22 Mbps 425.17
Peak Bit Rate (1ms resolution) 204522.19 Mbps 3058.05

Results

Sample Distribution
_images/TAIFEX-nxfeed-ethernet-hfp_Latency_20190510_1k_instruments_x1.setup1-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
831 ns 1208 ns 1218 ns 1184 ns 1235 ns 1250 ns 1255 ns 1261 ns 1535 ns 1875 ns

General Statistics  
Output Packet Count 3391807
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #2: 2 Time(s) Market Rate

Conditions

Instrument Count 1000
Subscribed Channel Count 1
Instruments List 1000 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 2

Observed Replay Rates

_images/Latency_20190510_1k_instruments_x2.setup1_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 14256 pkt/s 39.71
Average Packet Rate (10ms resolution) 1392074 pkt/s 540.61
Peak Packet Rate (1s resolution) 141088 pkt/s 47.65
Peak Packet Rate (10ms resolution) 14107400 pkt/s 399.64
Peak Packet Rate (1ms resolution) 141074000 pkt/s 3134.98
_images/Latency_20190510_1k_instruments_x2.setup1_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 20.60 Mbps 39.10
Average Bit Rate (10ms resolution) 2010.75 Mbps 612.82
Average Bit Rate (1ms resolution) 20088.39 Mbps 1156.57
Peak Bit Rate (1s resolution) 204.52 Mbps 49.98
Peak Bit Rate (10ms resolution) 20450.32 Mbps 425.13
Peak Bit Rate (1ms resolution) 204503.25 Mbps 3057.76

Results

Sample Distribution
_images/TAIFEX-nxfeed-ethernet-hfp_Latency_20190510_1k_instruments_x2.setup1-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
831 ns 1208 ns 1218 ns 1184 ns 1235 ns 1250 ns 1255 ns 1261 ns 1956 ns 2261 ns

General Statistics  
Output Packet Count 3391807
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #3: 10 Time(s) Market Rate

Conditions

Instrument Count 1000
Subscribed Channel Count 1
Instruments List 1000 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 10

Observed Replay Rates

_images/Latency_20190510_1k_instruments_x10.setup1_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 1853 pkt/s 5.16
Average Packet Rate (10ms resolution) 5364 pkt/s 2.08
Peak Packet Rate (1s resolution) 9966 pkt/s 3.37
Peak Packet Rate (10ms resolution) 82400 pkt/s 2.33
Peak Packet Rate (1ms resolution) 369000 pkt/s 8.20
_images/Latency_20190510_1k_instruments_x10.setup1_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 2.69 Mbps 5.10
Average Bit Rate (10ms resolution) 6.48 Mbps 1.97
Average Bit Rate (1ms resolution) 52.24 Mbps 3.01
Peak Bit Rate (1s resolution) 15.26 Mbps 3.73
Peak Bit Rate (10ms resolution) 84.15 Mbps 1.75
Peak Bit Rate (1ms resolution) 484.51 Mbps 7.24

Results

Sample Distribution
_images/TAIFEX-nxfeed-ethernet-hfp_Latency_20190510_1k_instruments_x10.setup1-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
834 ns 1209 ns 1219 ns 1186 ns 1237 ns 1251 ns 1260 ns 1375 ns 3778 ns 5705 ns

General Statistics  
Output Packet Count 3391807
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Scenario #3: Top 2477 symbols

Result Summary

Replay Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
Replay #1 831 ns 1151 ns 1218 ns 1181 ns 1235 ns 1250 ns 1255 ns 1261 ns 1552 ns 1941 ns
Replay #2 834 ns 1152 ns 1219 ns 1182 ns 1237 ns 1251 ns 1257 ns 1261 ns 1804 ns 2391 ns
Replay #3 834 ns 1151 ns 1218 ns 1181 ns 1235 ns 1250 ns 1261 ns 2395 ns 5486 ns 6441 ns

Replay #1: 1 Time(s) Market Rate

Conditions

Instrument Count 2477
Subscribed Channel Count 1
Instruments List 2477 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 1

Observed Replay Rates

_images/Latency_20190510_2477_instruments_x1.setup1_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 14052 pkt/s 39.14
Average Packet Rate (10ms resolution) 1388696 pkt/s 539.30
Peak Packet Rate (1s resolution) 141125 pkt/s 47.66
Peak Packet Rate (10ms resolution) 14111100 pkt/s 399.75
Peak Packet Rate (1ms resolution) 141111000 pkt/s 3135.80
_images/Latency_20190510_2477_instruments_x1.setup1_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 20.32 Mbps 38.56
Average Bit Rate (10ms resolution) 2007.88 Mbps 611.95
Average Bit Rate (1ms resolution) 20062.03 Mbps 1155.05
Peak Bit Rate (1s resolution) 204.54 Mbps 49.99
Peak Bit Rate (10ms resolution) 20452.22 Mbps 425.17
Peak Bit Rate (1ms resolution) 204522.19 Mbps 3058.05

Results

Sample Distribution
_images/TAIFEX-nxfeed-ethernet-hfp_Latency_20190510_2477_instruments_x1.setup1-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
831 ns 1151 ns 1218 ns 1181 ns 1235 ns 1250 ns 1255 ns 1261 ns 1552 ns 1941 ns

General Statistics  
Output Packet Count 3434889
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #2: 2 Time(s) Market Rate

Conditions

Instrument Count 2477
Subscribed Channel Count 1
Instruments List 2477 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 2

Observed Replay Rates

_images/Latency_20190510_2477_instruments_x2.setup1_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 14233 pkt/s 39.65
Average Packet Rate (10ms resolution) 1389725 pkt/s 539.70
Peak Packet Rate (1s resolution) 141088 pkt/s 47.65
Peak Packet Rate (10ms resolution) 14107400 pkt/s 399.64
Peak Packet Rate (1ms resolution) 141074000 pkt/s 3134.98
_images/Latency_20190510_2477_instruments_x2.setup1_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 20.58 Mbps 39.06
Average Bit Rate (10ms resolution) 2008.89 Mbps 612.26
Average Bit Rate (1ms resolution) 20069.93 Mbps 1155.50
Peak Bit Rate (1s resolution) 204.52 Mbps 49.98
Peak Bit Rate (10ms resolution) 20450.32 Mbps 425.13
Peak Bit Rate (1ms resolution) 204503.25 Mbps 3057.76

Results

Sample Distribution
_images/TAIFEX-nxfeed-ethernet-hfp_Latency_20190510_2477_instruments_x2.setup1-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
834 ns 1152 ns 1219 ns 1182 ns 1237 ns 1251 ns 1257 ns 1261 ns 1804 ns 2391 ns

General Statistics  
Output Packet Count 3434889
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #3: 10 Time(s) Market Rate

Conditions

Instrument Count 2477
Subscribed Channel Count 1
Instruments List 2477 instruments with the most updates
Lane Arbitration B only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 10

Observed Replay Rates

_images/Latency_20190510_2477_instruments_x10.setup1_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 1848 pkt/s 5.15
Average Packet Rate (10ms resolution) 5346 pkt/s 2.08
Peak Packet Rate (1s resolution) 9966 pkt/s 3.37
Peak Packet Rate (10ms resolution) 82400 pkt/s 2.33
Peak Packet Rate (1ms resolution) 421000 pkt/s 9.36
_images/Latency_20190510_2477_instruments_x10.setup1_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 2.68 Mbps 5.08
Average Bit Rate (10ms resolution) 6.45 Mbps 1.97
Average Bit Rate (1ms resolution) 52.17 Mbps 3.00
Peak Bit Rate (1s resolution) 15.26 Mbps 3.73
Peak Bit Rate (10ms resolution) 84.15 Mbps 1.75
Peak Bit Rate (1ms resolution) 484.51 Mbps 7.24

Results

Sample Distribution
_images/TAIFEX-nxfeed-ethernet-hfp_Latency_20190510_2477_instruments_x10.setup1-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
834 ns 1151 ns 1218 ns 1181 ns 1235 ns 1250 ns 1261 ns 2395 ns 5486 ns 6441 ns

General Statistics  
Output Packet Count 3434889
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time