Capital Markets Infrastructure

High-Frequency Trading Systems

The requirement here is not speed. It is a number you can measure, defend and reproduce after a kernel upgrade. We design the whole path — cross-connect to order gateway — around the tail of the distribution, then prove the result with hardware timestamps instead of assertions.

Request a latency baseline See the latency budget

The Constraint

The median is a marketing number. The tail is the loss.

What determinism means in practice

A deterministic path does the same work in the same time on every message. That includes the ten-millionth message of the session, and the first message after a burst. It is a property of the whole system — the fibre, the switch, the NIC, the interrupt controller, the memory hierarchy and the code. One non-deterministic component sets the tail for everything behind it.

Most of the variance we find is not in the application. It is in power management, an interrupt landing on a trading core, a page fault on a cold allocation, or a NIC coalescing frames to be helpful. None of those appear in an average, and none of them are visible to a profiler that shares a core with the thing it is profiling.

So the first deliverable of any engagement is a measured distribution, not a design. We will not propose a change to a path we have not yet watched under real flow.

Determinism: The same work in the same time on every message, including immediately after a burst.
Jitter: The spread of the distribution rather than its centre. The property that costs money.
Microburst: A sub-millisecond spike in message rate that fills a queue and leaves no trace in a per-second average.
Tail event: One message outside budget. A single event is a loss; a pattern is a design fault.
Latency budget: Nanoseconds allocated per hop, each line independently measurable and owned by a named person.

If you cannot tell me where the tap sits and which percentile you mean, you have not quoted me a latency. You have quoted me a hope.

Low-latency practice Cloud Natives

Signal Path

From fibre to order, component by component.

Cross-connect and fibre

The first line of the budget is physics. Light covers roughly 200 mm per nanosecond in single-mode fibre, so each additional metre of cross-connect costs about 5 ns in each direction. We record the delivered patch length rather than the ordered length, and re-verify it after any venue re-patch. Where the venue equalises cross-connect lengths for fairness, that lever is gone by design and we stop spending effort on it.

Switch hops

A cut-through switch adds a few hundred nanoseconds. An FPGA-based switch adds tens. Store-and-forward adds the serialisation time of the entire frame, which makes it a poor choice anywhere near the order path. We count hops as budget line items and delete the ones that exist only for operational convenience. Shared uplinks are identified early, because a hop that is fast at idle is not fast during a burst.

NIC and kernel bypass

The kernel network stack costs microseconds and, more importantly, varies. We move receive into user space with a bypass stack — DPDK, or a vendor onload library on Solarflare-class hardware — so the frame lands in a pinned buffer with no syscall, no copy and no softirq. Interrupt-driven receive becomes polled receive: more power drawn, lower and much flatter latency. Interrupt moderation and frame coalescing are disabled explicitly, because both are tuned by default for throughput.

FPGA offload

Where a decision is simple enough to express in gates, it belongs in gates. Feed parsing, book building, pre-trade checks and order emission run in the low hundreds of nanoseconds with jitter measured in tens. We are candid about the cost: a change that takes an afternoon in C++ takes a fortnight in HDL, plus timing closure, plus a slow-path design for the cases the gates do not handle. That is a real schedule risk and we price it as one.

Host tuning and power policy

C-states disabled, or capped at C1, because an exit from a deep idle state is a microsecond-scale event. Frequency governor pinned to performance, with turbo behaviour characterised rather than assumed. Hyper-threading decided per workload, not by blanket policy. System management interrupts hunted down and eliminated where the platform allows, since an SMI is invisible to your profiler and plainly visible in your tail.

Interrupts, isolation and scheduling

IRQ affinity moved off the trading cores, then verified at run time rather than trusted from the boot plan. isolcpus and nohz_full so the scheduler tick stops interrupting the hot thread. RCU callbacks, kernel worker threads and timers relocated. The hot core runs one thread and nothing else — including nothing belonging to your monitoring agent, which is one of the most common sources of a tail we find.

Cache-aware data layout

Hot structures sized to stay resident in L1 and L2, laid out so that one cache line carries one decision's worth of data. No pointer chasing on the hot path, no allocation, no false sharing between producer and consumer. Hugepages reserved and pre-faulted at start-up to keep the TLB quiet. A single L3 miss costs more than the entire parse step it interrupts.

NUMA pinning

The NIC, its queues, the polling thread and its buffers all live on the same socket. A single remote-node access costs tens of nanoseconds; a remote-node access inside a loop is a tail event. Buffers allocated before the thread is pinned end up on the wrong node, which is why we assert placement at run time and fail start-up rather than warn about it.

Order egress and gateway

Egress is where budget is most often spent carelessly: a copy, a lock, a logging call left in the hot path. Orders are emitted from a pre-built template with fields patched in place. Logging moves off-core to a lock-free ring drained by another thread, so the audit trail is complete without the audit trail being in the path. Doorbell writes are examined individually, because batching them trades your tail for someone else's throughput metric.

Jitter, Not Average

How to specify latency so the contract is testable.

Read the distribution, not the headline

A build with a 700 ns median and a 40 µs p99.9 beats a 900 ns median on a slide and loses money in production. The outliers cluster exactly where they hurt most: in the first microseconds of a burst, when the book is moving and your order is the one that matters.

Microbursts are the usual mechanism. A feed that averages comfortably below capacity can exceed it for 200 µs at a time. A queue forms, drains, and leaves no evidence in a per-second average. We characterise the burst profile — peak multiple, duration, inter-burst gap — and then measure in windows short enough to contain a single burst.

The practical consequence is that p50 is a diagnostic, p99 is a design target, and p99.9 is the number you argue about. Any figure quoted without its percentile should be treated as p50 and discounted accordingly.

0.7 µsp50 — the number that appears on the slide

1.1 µsp99 — the number the desk actually feels

38 µsp99.9 — the number that costs money, on an untuned path

Chart — tick-to-trade distribution, log-scale tail, before and after tuning (1600×1200)

Six clauses that make a latency term enforceable

Name the measurement point

Wire at a named passive tap, both directions, not an application log line. State where the tap sits, who owns the capture hardware and which interface is authoritative if the two disagree.

Name the percentile and the window

For example, p99.9 within each rolling 60-second window, reported per window. A percentile taken across a whole trading day can hide an hour of unusable latency inside an acceptable figure.

Name the load

Sustained message rate plus an explicit burst profile: peak multiple, burst duration and inter-burst gap. Without a load definition the number is only true at idle, and only idle will be tested.

Name the clock

Which grandmaster, what its holdover specification is, and how traceability is evidenced. Two clocks that disagree by 200 ns produce two defensible versions of the same event.

Name the breach

Define what a single message outside budget means. One event may be reportable rather than a failure, but the threshold count has to be written down before anyone needs it, not during the dispute.

Name the retention

How long captures are kept, in what format, and who may read them. A dispute six weeks after the event needs the packets and the configuration that produced them, not a summary chart.

Latency Budget

Every nanosecond has an owner.

Indicative tick-to-trade budget, representative build. Order-of-magnitude figures for structuring a budget — not measured results, not a performance commitment.
Hop	Indicative contribution	Determined by	Where the tail hides
Cross-connect, 40 m single-mode	≈ 200 ns each way	Delivered patch length, ≈ 5 ns/m	A venue re-patch nobody told you about
Venue switch hop, cut-through	250–450 ns	Switch silicon, port and MTU config	Congestion on a shared uplink during a burst
Venue switch hop, FPGA-based	40–90 ns	Switch silicon	Fewer hiding places, but not zero
NIC ingress, wire to user-space buffer (bypass)	700–900 ns	NIC generation, PCIe topology	Interrupt moderation or coalescing left enabled
Same path via the kernel network stack	8–15 µs	Syscalls, copies, softirq scheduling	Scheduler pre-emption, page faults, lock contention
Feed parse and book update, FPGA	90–150 ns	Gate count, clock domain crossings	Rare message types falling to the slow path
Feed parse and book update, host	250–700 ns	Cache residency, structure layout	An L3 miss, or a branch the predictor has not seen
Pre-trade risk checks, in-line on FPGA	20–50 ns	Number of checks in the chain	A check added later in software instead
Pre-trade risk checks, in-line in software	150–400 ns	Check complexity, data locality	A limits table that lives on the wrong socket
Strategy decision, host hot path	200–900 ns	Your code, not ours	Allocation, logging or a lock on the hot path
PCIe round trip, host to accelerator	500–900 ns	Lane width, root complex, posting rules	An extra descriptor fetch under load
Order egress, host to wire (bypass)	600–800 ns	NIC, write posting behaviour	Doorbell batching that helps throughput only
Remote NUMA node memory access	+40–70 ns per access	Socket topology, allocation order	Buffers allocated before the thread was pinned
DRAM access after an L3 miss	70–100 ns	Memory generation and channel population	Any pointer chase surviving on the hot path
Indicative total, tuned software path	2.0–3.0 µs wire to wire	Sum of the lines above, measured end to end	The gap between the sum and the measurement

Read the last line carefully. The interesting number in any budget is the difference between the sum of the parts and the measured end-to-end figure. That gap is the work nobody has attributed yet, and it is where the tail usually lives. A budget that adds up perfectly on the first attempt has not been measured.

Measurement & Time

Software timestamps are not evidence.

What we accept as evidence

A timestamp taken by clock_gettime inside your process is taken after the frame has already waited — in the NIC, in a queue, behind an interrupt. It also inherits every source of jitter you are trying to find. It is useful for relative comparisons inside one binary, and for nothing that anyone will accept in a dispute.

The baseline comes from a passive optical tap — a splitter rather than an in-line device, so the instrument can never become a failure mode. It feeds a capture card that timestamps at the PHY. Ingress and egress are captured on the same card, so the difference between them does not carry two clocks' worth of error.

Time distribution is production infrastructure, not a utility. When GNSS is lost, the grandmaster's holdover specification silently becomes your accuracy specification, and it degrades with temperature and elapsed time. We write that figure down, alarm on entry into holdover, and treat extended holdover as a reportable event.

PHY-level hardware timestamping, sub-10 ns resolution, on both directions
Passive optical taps only on the client side of the demarcation
PTP (IEEE 1588) with PPS and 10 MHz distribution from a GNSS-disciplined grandmaster
Two antenna paths, diverse cable routes, alarmed independently
Captures retained alongside the exact configuration that produced them

±50 nsTarget offset from grandmaster at the host, steady state

<10 nsCapture timestamp resolution at the tap

100%Tail events retained as packets, not summaries

Diagram — tap, capture card and PTP time distribution topology (1600×1200)

Colocation & Connectivity

Cable length is a line item, not a detail.

Position, length and the diversity trade-off

Rack position matters only through cable length and the patch path. Venues that equalise cross-connect lengths remove that lever on purpose, and arguing with it wastes an engagement. Where lengths are not equalised, we measure the patch that was delivered rather than the one on the order form.

Diversity and latency pull against each other, and the honest answer is that you cannot have both on the same flow. The shortest path is one path. A genuinely diverse second path is longer, and pretending otherwise produces a design that fails at the first fibre cut. We decide per venue which flow takes the short route and which takes the survivable one, and we write the decision down so it is a choice rather than an accident.

Venue rules are part of the design rather than an obstacle to it: power caps per rack, approved cabling only, no in-line devices on the exchange side of the demarcation, and change windows that are not negotiable. Taps, spares and commissioning tests are planned to fit inside those rules from the first drawing.

In Sydney that usually means the ASX Australian Liquidity Centre for the order path, with a carrier-neutral campus alongside it for market data, inter-venue links and everything that does not need to be in the cage.

Cross-connect: Length recorded at commissioning, re-verified after any venue re-patch or cabinet move.
Rack power: Designed to the venue cap rather than the nameplate figure, with polled draw trended per phase.
Taps: Passive splitters, client side of the demarcation only, documented in the venue's records.
Time: Antenna path, grandmaster and PPS distribution treated as production systems with their own change control.
Spares: A configured cold spare on site, carrying its own captured baseline so a swap is measurable.
Change: Every change scheduled into the venue window, then re-measured against the acceptance test.

Risk & Controls

Controls live in the path, or they are decoration.

Pre-trade checks in the hot path

Price collar, maximum order and notional size, message-rate limit, position and exposure limits, duplicate detection and self-match prevention. In gates these cost tens of nanoseconds; in software, low hundreds. Either way they sit in front of the order and share no mutable state with the strategy, so a strategy fault cannot disable the control that exists to contain it.

A kill switch that works when nothing else does

A kill switch reachable only through the system it is meant to stop is not a control. We build an out-of-band path, physical and logical, that drops the session, cancels resting orders and holds the gateway closed until a person re-enables it. It is tested on a schedule, in production, inside a venue change window — an untested kill switch is an assumption.

Deterministic failover

A standby that takes over in an unbounded time is a gap, not a failover. We specify the takeover budget in milliseconds, keep sequence and order state on both sides, and rehearse the cut with the desk present. Failover that loses order state is worse than an outage, because you no longer know your position while the market keeps moving.

The obligation, not only the practice

Australian market participants operate under ASIC's market integrity rules, which require automated order processing filters, documented and tested controls, and the ability to halt order flow promptly. We engineer to the control the rule describes and leave its legal interpretation to your compliance team. Where a control costs latency, we show the cost rather than quietly moving the control out of the path.

We are engineers, not your compliance function. Nothing on this page is regulatory or legal advice. We build, measure and evidence controls to the specification your compliance team sets, and we will tell you plainly when a latency target and a control requirement are in conflict rather than resolving it silently in favour of the number.

How An Engagement Runs

Measure, attribute, change one thing, measure again.

Baseline, with nothing changed

One week of real flow captured at the tap. We change nothing, tune nothing and recommend nothing. The deliverable is a distribution with the tail visible, the burst profile characterised, and the measurement method written down so it can be repeated by someone else.

Attribute the distribution to hops

Each outlier traced to a component. This is where the uncomfortable findings arrive: a monitoring agent scheduled onto a trading core, a NIC firmware default nobody reviewed, a BIOS setting with no owner, a cross-connect that is eighteen metres longer than the diagram.

Replicate on the bench

We rebuild the path on our own hardware and reproduce the tail from your replayed capture. If we cannot reproduce it, we have not understood it, and we will say so before proposing a change to a production system. Replication is also what lets us test the changes we decide against.

One change at a time

Every change measured on its own, in isolation, against the same acceptance test. A change that improves the median and lengthens p99.9 is rejected however good it looks on a chart. The rejected list, with its measurements, is part of the final report — it is usually the most useful part.

Soak, burst and thermal test

Seventy-two hours at sustained rate with the burst profile layered on top. Long enough for thermal throttling, C-state regressions, memory fragmentation and slow leaks to appear. A build that holds its budget for ten minutes has proved very little about the trading day.

Handover with the evidence

Captures, method, configuration as code, and a regression harness your team runs without us. The next kernel, firmware or driver upgrade then becomes a measurement instead of a hope. Configuration drift is the most common reason a tuned system quietly stops being tuned.

Questions We Are Asked

The answers procurement wants in writing.

Start With A Measurement

Send us a capture, not a specification. We will tell you where the tail comes from.

A baseline engagement takes a week of real flow at the tap and produces a distribution, an attribution of the outliers, and an honest view on whether the remaining gains justify the work. If the answer is that your path is already close to its budget, we will say that too.

Request a latency baseline All solutions

Low-latency practice: hello@cloudnatives.example
Direct line: +61 0 0000 0000
Existing clients — 24/7 NOC: +61 0 0000 0001

Venue work is scheduled around the trading day and the venue's own change windows. Sydney, Melbourne and Canberra.