EUREX-EOBI nxFeed Latency Tests

Overview

This latency report details the performance of the Enyx EUREX-EOBI nxFeed solution during one of the busiest market days of the last decade, February 28th 2020.

To demonstrate the reliability of the FPGA based nxFeed solution, in addition to test at 1X the market data rate, additional tests were performed with the recorded data being replayed at up to 10 times the captured rate from that day.

Setup Configuration

Schematics

_images/10Gb_nxFeed_wire_to_wire-latency_measure_setup_diagram.png

This testing configuration uses two different servers:

  • A host server holding an Enyx FPGA board running nxFeed firmware and a configuration application.
  • A replay and capture server responsible for the replay of market data captures at defined speeds and the capture of timestamped input and output feeds forwarded by the timestamping device.

On the host server, the Enyx FPGA board is configured to process incoming raw market data traffic and output a normalized and filtered Enyx MDP feed back to the network.

Thanks to a layer 1 switch, both the raw market data feed and the normalized output feeds are forwarded to timestamping device for precise timestamping and then forwarded to the capture server for storage and further analysis.

The analysis of the timestamped data is used for the latency measurements but also for the extraction of the effective rate at which the replay server was able to replay the raw market data into the solution.

Host Server Characteristics

CPU 1 model & version Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
CPU 1 PCIe devices N/A
CPU 2 model & version Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
CPU 2 PCIe devices Enyx FPB2

Replay & Capture Server Characteristics

CPU 1 model & version Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
CPU 1 PCIe devices Solarflare SFC9220 10G Adapter Solarflare SFC9220 10G Adapter
CPU 2 model & version Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
CPU 2 PCIe devices N/A

Layer 1 Device Model & Version

Model Metamako MetaMux 48 with K-Series Plus
Version 0.17.4
Port Speed 1Gb/10Gb
Port to port advertised latency 5 ns

Layer 2 Device Model & Version

Model Arista 7150s
Version EOS-4.14.6M
Port Speed 1Gb/10Gb
Timestamping resolution 2.857 nanoseconds
Timestamping trigger First Byte of the FCS

Enyx Solution Characteristics:

FPGA card version Enyx FPB2
Firmware version EUREX-EOBI80 2.8.0
Software version libenyxmd 4.1.5
Driver version hfp 2.3.1
Thread Binding CPU ID 5 (Numa 0)
Card NUMA node 1

Capture Characteristics

Capture Merged_EOBI_Extract_2020-02-28_All-ch_lane-A_open-to-close
Packet Count 60 453 806
Channel Count 16
Capture Duration 09:29:59
Start Date Fri, 28 Feb 2020 08:30:00
End date Fri, 28 Feb 2020 17:59:59
Timestamping resolution Nanoseconds
Packet Rate Peak (s) 114 540 pkt/s
Packet Rate Peak (100 ms) 114 540 pkt/s
Packet Rate Peak (10 ms) 384 200 pkt/s
Packet Rate Peak (1 ms) 673 000 pkt/s
Bit Rate Peak (s) 28.11 Mbps
Bit Rate Peak (100 ms) 145.89 Mbps
Bit Rate Peak (10 ms) 496.79 Mbps
Bit Rate Peak (1 ms) 1 010.58 Mbps
_images/Merged_EOBI_Extract_2020-02-28_All-ch_lane-A_open-to-close_global_Packet_Rate.png _images/Merged_EOBI_Extract_2020-02-28_All-ch_lane-A_open-to-close_global_Bit_Rate.png

Test Overview

The test has been replayed with these parameters:

  • Replay speeds:
    • X1, X2, X3, X4, X5, X10
  • Subscribed instruments:
    • All the symbols (5086) of the EUREX EOBI channels 1, 3 to 5, 7 to 9, 10 to 16, 20 and 23.

EUREX EOBI: 5086 symbols

Result Summary

Replay Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
Replay #1 766 ns 1 081 ns 1 160 ns 1 146 ns 1 230 ns 1 264 ns 2 337 ns 2 523 ns 2 860 ns 5 017 ns
Replay #2 764 ns 1 081 ns 1 160 ns 1 146 ns 1 230 ns 1 267 ns 2 341 ns 2 553 ns 2 941 ns 5 764 ns
Replay #3 766 ns 1 081 ns 1 160 ns 1 147 ns 1 231 ns 1 277 ns 2 343 ns 2 583 ns 3 025 ns 6 184 ns
Replay #4 766 ns 1 084 ns 1 160 ns 1 148 ns 1 231 ns 1 297 ns 2 346 ns 2 605 ns 3 150 ns 6 579 ns
Replay #5 766 ns 1 084 ns 1 160 ns 1 149 ns 1 231 ns 1 316 ns 2 349 ns 2 625 ns 3 250 ns 6 672 ns
Replay #6 766 ns 1 084 ns 1 164 ns 1 173 ns 1 241 ns 1 882 ns 2 469 ns 3 092 ns 4 301 ns 7 415 ns

Replay #1: 1X Market Rate

Conditions

Replay Duration 09:31:20
Instrument Count 5 086
Subscribed Channel Count 16
Instruments List All the instruments from the EUREX EOBI channels 1, 3 to 5, 7 to 9, 10 to 16, 20 and 23
Lane Arbitration A only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 1

Observed Replay Rates

_images/Latency_20200228_EUREX_EOBI_5_BBO_x1.setup2_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 1 768 pkt/s 1.00
Average Packet Rate (100ms resolution) 2 756 pkt/s 1.00
Average Packet Rate (10ms resolution) 8 240 pkt/s 1.00
Average Packet Rate (1ms resolution) 29 216 pkt/s 1.00
Peak Packet Rate (1s resolution) 19 428 pkt/s 1.00
Peak Packet Rate (100ms resolution) 114 540 pkt/s 1.00
Peak Packet Rate (10ms resolution) 358 100 pkt/s 0.93
Peak Packet Rate (1ms resolution) 754 000 pkt/s 1.12
_images/Latency_20200228_EUREX_EOBI_5_BBO_x1.setup2_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 2.13 Mbps 1.03
Average Bit Rate (100ms resolution) 3.33 Mbps 1.03
Average Bit Rate (10ms resolution) 10.07 Mbps 1.03
Average Bit Rate (1ms resolution) 36.20 Mbps 1.03
Peak Bit Rate (1s resolution) 28.73 Mbps 1.02
Peak Bit Rate (100ms resolution) 149.56 Mbps 1.03
Peak Bit Rate (10ms resolution) 451.87 Mbps 0.91
Peak Bit Rate (1ms resolution) 1 028.48 Mbps 1.02

Results

Sample Distribution
_images/EUREX-EOBI-nxfeed-ethernet-hfp_Latency_20200228_EUREX_EOBI_5_BBO_x1.setup2-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
766 ns 1 081 ns 1 160 ns 1 146 ns 1 230 ns 1 264 ns 2 337 ns 2 523 ns 2 860 ns 5 017 ns

General Statistics  
Output Packet Count 60 453 875
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #2: 2X Market Rate

Conditions

Replay Duration 04:45:35
Instrument Count 5 086
Subscribed Channel Count 16
Instruments List All the instruments from the EUREX EOBI channels 1, 3 to 5, 7 to 9, 10 to 16, 20 and 23
Lane Arbitration A only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 2

Observed Replay Rates

_images/Latency_20200228_EUREX_EOBI_5_BBO_x2.setup2_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 3 539 pkt/s 2.00
Average Packet Rate (100ms resolution) 4 931 pkt/s 1.79
Average Packet Rate (10ms resolution) 13 191 pkt/s 1.60
Average Packet Rate (1ms resolution) 45 029 pkt/s 1.54
Peak Packet Rate (1s resolution) 27 889 pkt/s 1.44
Peak Packet Rate (100ms resolution) 139 590 pkt/s 1.22
Peak Packet Rate (10ms resolution) 651 400 pkt/s 1.70
Peak Packet Rate (1ms resolution) 1 054 000 pkt/s 1.57
_images/Latency_20200228_EUREX_EOBI_5_BBO_x2.setup2_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 4.25 Mbps 2.05
Average Bit Rate (100ms resolution) 5.95 Mbps 1.83
Average Bit Rate (10ms resolution) 16.08 Mbps 1.64
Average Bit Rate (1ms resolution) 55.60 Mbps 1.57
Peak Bit Rate (1s resolution) 39.32 Mbps 1.40
Peak Bit Rate (100ms resolution) 181.57 Mbps 1.24
Peak Bit Rate (10ms resolution) 837.10 Mbps 1.69
Peak Bit Rate (1ms resolution) 1 569.17 Mbps 1.55

Results

Sample Distribution
_images/EUREX-EOBI-nxfeed-ethernet-hfp_Latency_20200228_EUREX_EOBI_5_BBO_x2.setup2-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
764 ns 1 081 ns 1 160 ns 1 146 ns 1 230 ns 1 267 ns 2 341 ns 2 553 ns 2 941 ns 5 764 ns

General Statistics  
Output Packet Count 60 453 875
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #3: 3X Market Rate

Conditions

Replay Duration 03:10:52
Instrument Count 5 086
Subscribed Channel Count 16
Instruments List All the instruments from the EUREX EOBI channels 1, 3 to 5, 7 to 9, 10 to 16, 20 and 23
Lane Arbitration A only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 3

Observed Replay Rates

_images/Latency_20200228_EUREX_EOBI_5_BBO_x3.setup2_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 5 310 pkt/s 3.00
Average Packet Rate (100ms resolution) 6 894 pkt/s 2.50
Average Packet Rate (10ms resolution) 16 867 pkt/s 2.04
Average Packet Rate (1ms resolution) 56 211 pkt/s 1.92
Peak Packet Rate (1s resolution) 33 727 pkt/s 1.74
Peak Packet Rate (100ms resolution) 155 050 pkt/s 1.35
Peak Packet Rate (10ms resolution) 780 100 pkt/s 2.03
Peak Packet Rate (1ms resolution) 1 462 000 pkt/s 2.17
_images/Latency_20200228_EUREX_EOBI_5_BBO_x3.setup2_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 6.38 Mbps 3.08
Average Bit Rate (100ms resolution) 8.32 Mbps 2.56
Average Bit Rate (10ms resolution) 20.54 Mbps 2.09
Average Bit Rate (1ms resolution) 69.28 Mbps 1.96
Peak Bit Rate (1s resolution) 46.54 Mbps 1.66
Peak Bit Rate (100ms resolution) 201.04 Mbps 1.38
Peak Bit Rate (10ms resolution) 995.85 Mbps 2.00
Peak Bit Rate (1ms resolution) 1 998.37 Mbps 1.98

Results

Sample Distribution
_images/EUREX-EOBI-nxfeed-ethernet-hfp_Latency_20200228_EUREX_EOBI_5_BBO_x3.setup2-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
766 ns 1 081 ns 1 160 ns 1 147 ns 1 231 ns 1 277 ns 2 343 ns 2 583 ns 3 025 ns 6 184 ns

General Statistics  
Output Packet Count 60 453 875
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #4: 4X Market Rate

Conditions

Replay Duration 02:22:46
Instrument Count 5 086
Subscribed Channel Count 16
Instruments List All the instruments from the EUREX EOBI channels 1, 3 to 5, 7 to 9, 10 to 16, 20 and 23
Lane Arbitration A only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 4

Observed Replay Rates

_images/Latency_20200228_EUREX_EOBI_5_BBO_x4.setup2_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 7 077 pkt/s 4.00
Average Packet Rate (100ms resolution) 8 792 pkt/s 3.19
Average Packet Rate (10ms resolution) 19 965 pkt/s 2.42
Average Packet Rate (1ms resolution) 65 980 pkt/s 2.25
Peak Packet Rate (1s resolution) 38 944 pkt/s 2.00
Peak Packet Rate (100ms resolution) 160 260 pkt/s 1.40
Peak Packet Rate (10ms resolution) 831 700 pkt/s 2.16
Peak Packet Rate (1ms resolution) 1 787 000 pkt/s 2.66
_images/Latency_20200228_EUREX_EOBI_5_BBO_x4.setup2_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 8.50 Mbps 4.10
Average Bit Rate (100ms resolution) 10.60 Mbps 3.26
Average Bit Rate (10ms resolution) 24.29 Mbps 2.47
Average Bit Rate (1ms resolution) 81.22 Mbps 2.30
Peak Bit Rate (1s resolution) 53.02 Mbps 1.89
Peak Bit Rate (100ms resolution) 207.34 Mbps 1.42
Peak Bit Rate (10ms resolution) 1 060.58 Mbps 2.13
Peak Bit Rate (1ms resolution) 2 423.94 Mbps 2.40

Results

Sample Distribution
_images/EUREX-EOBI-nxfeed-ethernet-hfp_Latency_20200228_EUREX_EOBI_5_BBO_x4.setup2-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
766 ns 1 084 ns 1 160 ns 1 148 ns 1 231 ns 1 297 ns 2 346 ns 2 605 ns 3 150 ns 6 579 ns

General Statistics  
Output Packet Count 60 453 875
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #5: 5X Market Rate

Conditions

Replay Duration 01:55:06
Instrument Count 5 086
Subscribed Channel Count 16
Instruments List All the instruments from the EUREX EOBI channels 1, 3 to 5, 7 to 9, 10 to 16, 20 and 23
Lane Arbitration A only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 5

Observed Replay Rates

_images/Latency_20200228_EUREX_EOBI_5_BBO_x5.setup2_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 8 843 pkt/s 4.99
Average Packet Rate (100ms resolution) 10 673 pkt/s 3.87
Average Packet Rate (10ms resolution) 22 733 pkt/s 2.75
Average Packet Rate (1ms resolution) 75 322 pkt/s 2.57
Peak Packet Rate (1s resolution) 43 541 pkt/s 2.24
Peak Packet Rate (100ms resolution) 165 060 pkt/s 1.44
Peak Packet Rate (10ms resolution) 867 000 pkt/s 2.26
Peak Packet Rate (1ms resolution) 2 244 000 pkt/s 3.33
_images/Latency_20200228_EUREX_EOBI_5_BBO_x5.setup2_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 10.62 Mbps 5.13
Average Bit Rate (100ms resolution) 12.86 Mbps 3.96
Average Bit Rate (10ms resolution) 27.64 Mbps 2.82
Average Bit Rate (1ms resolution) 92.61 Mbps 2.62
Peak Bit Rate (1s resolution) 58.65 Mbps 2.09
Peak Bit Rate (100ms resolution) 213.09 Mbps 1.46
Peak Bit Rate (10ms resolution) 1 102.60 Mbps 2.22
Peak Bit Rate (1ms resolution) 2 911.78 Mbps 2.88

Results

Sample Distribution
_images/EUREX-EOBI-nxfeed-ethernet-hfp_Latency_20200228_EUREX_EOBI_5_BBO_x5.setup2-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
766 ns 1 084 ns 1 160 ns 1 149 ns 1 231 ns 1 316 ns 2 349 ns 2 625 ns 3 250 ns 6 672 ns

General Statistics  
Output Packet Count 60 453 875
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time

Replay #6: 10X Market Rate

Conditions

Replay Duration 00:57:51
Instrument Count 5 086
Subscribed Channel Count 16
Instruments List All the instruments from the EUREX EOBI channels 1, 3 to 5, 7 to 9, 10 to 16, 20 and 23
Lane Arbitration A only
Book Builder Configuration Delta Updates
Output Book Depth 5
Requested Replay Rate 10

Observed Replay Rates

_images/Latency_20200228_EUREX_EOBI_5_BBO_x10.setup2_global_Packet_Rate.png
Type Rate Replay Ratio
Average Packet Rate (1s resolution) 17 676 pkt/s 9.98
Average Packet Rate (100ms resolution) 19 919 pkt/s 7.22
Average Packet Rate (10ms resolution) 36 422 pkt/s 4.41
Average Packet Rate (1ms resolution) 114 555 pkt/s 3.91
Peak Packet Rate (1s resolution) 64 641 pkt/s 3.33
Peak Packet Rate (100ms resolution) 194 280 pkt/s 1.70
Peak Packet Rate (10ms resolution) 1 139 000 pkt/s 2.96
Peak Packet Rate (1ms resolution) 3 350 000 pkt/s 4.98
_images/Latency_20200228_EUREX_EOBI_5_BBO_x10.setup2_global_Bit_Rate.png
Type Rate Replay Ratio
Average Bit Rate (1s resolution) 21.23 Mbps 10.25
Average Bit Rate (100ms resolution) 23.98 Mbps 7.38
Average Bit Rate (10ms resolution) 44.21 Mbps 4.50
Average Bit Rate (1ms resolution) 140.36 Mbps 3.97
Peak Bit Rate (1s resolution) 83.39 Mbps 2.97
Peak Bit Rate (100ms resolution) 287.33 Mbps 1.97
Peak Bit Rate (10ms resolution) 1 485.82 Mbps 2.99
Peak Bit Rate (1ms resolution) 4 150.88 Mbps 4.11

Results

Sample Distribution
_images/EUREX-EOBI-nxfeed-ethernet-hfp_Latency_20200228_EUREX_EOBI_5_BBO_x10.setup2-wire_to_wire.png
Min 25% 50% Mean 90% 99% 99.9% 99.99% 99.999% Max
766 ns 1 084 ns 1 164 ns 1 173 ns 1 241 ns 1 882 ns 2 469 ns 3 092 ns 4 301 ns 7 415 ns

General Statistics  
Output Packet Count 60 453 875
Input FIFO maximal usage 0.09 %
Raw Market Data Packet Drop 0
Latency Distribution Over Time