The Slow Consumer
Most market data applications want their data with low latency and high throughput. The reality is that this combination is difficult to deliver — and more difficult to consume.
Every application is perfectly engineered. Ask any developer. The slow consumer problem is always someone else’s issue — the result of poorly written code, inadequate testing, or a failure of discipline that simply does not apply here. Until it does. A volatility spike, a correlated burst across a large instrument universe, a GC pause at exactly the wrong moment. The breaking point exists in every application. It is just a question of what it takes to find it. Usually that question gets answered in production.
The industry term for an application that has reached its breaking point is a slow consumer. It is one of the most persistent challenges in market data infrastructure, and the solutions on offer have historically required a trade-off.
The conventional trade-off
Conventional platforms ask applications to choose. Low latency or high throughput. Conflation or not. The choice is made at configuration time, applied to subscriber classes, and does not change at runtime. An application assigned to the full-rate tier gets every update whether it can handle the volume or not. An application assigned to a conflated tier gets a reduced feed whether it needs one or not.
This is not a solution to the slow consumer problem. Conventional products have kicked that can down the road. They solve the problem by asking you to choose a configuration.
The slow consumer event does not happen on a quiet day. It happens at peak load, during the events that matter most. That is when the consequences of a static configuration are hardest to absorb.
The Elastic MDS approach
Elastic MDS treats this as a platform problem, not an application problem. The platform monitors every application connection independently. When an application begins to struggle, the platform responds; without operator intervention and without affecting other applications.
Video: S.1 — Push Distribution. Code walkthrough of a push client, what makes a slow consumer slow, and sample output.
Video: A.0 — A slow-consumer. What happens to an application when it cannot keep up, with no mitigation in place.
The response is multi-tiered. First, orchestration will add extra data connections. A single connection has finite capacity: one thread, one buffer, one delivery path. Adding connections means more parallel paths, more capacity, lower pressure on any one of them. In Elastic MDS these data connections are called shards. Adding shards under duress, and reclaiming them when they are no longer needed, is what Elastic MDS calls shard splitting. The application does not have to request it. The developer does not have to code for it. It happens because it is the best way to solve the problem — for any application.
If additional connections are not enough, the platform engages conflation on the shards that need it. Again, this happens automatically, and as a last resort. The application receives current data at a rate it can handle. When it recovers, conflation disengages. No explicit trade-off is required at any point.
One infrastructure
Elastic MDS handles both cases, full rate and conflated, adaptively, from a single deployment. There is no separate conflation infrastructure. No second system to run, monitor, and keep in sync with the first.
The immediate implication is a significant reduction in hardware. The less obvious one is that running a single system opens up options for how you think about resilience that are not available when you are running two. More on that later.
What you can see
The status of any application and its connections is visible in real time through the REST API and the operations dashboard. Shard count, delivery rate, current state — all live, all in one place. The shard splitting and conflation events described above are visible as they happen, not reconstructed from logs after the fact.
What you can configure
Conflation behavior is defined by a simple expression that you control. The default works well for most cases but you can customize if you like and you can even tailor different rules and expression for different applications. You can also exclude applications if that’s necessary.
The conflation expression uses basic SQL syntax. You’ll recognize it when you see it. There will be more about using SQL in later posts when I show you how to augment your content and even create new content with simple queries.
See it in action
Video: S.3 — What is a Shard? Watch this first.
Video: A.3 — Conflation as a Last Resort
Related
About this blog
Elastic MDS is an autoscaling, real-time market data distribution platform developed by MetaFluent. This blog is about Elastic MDS infrastructure deployment and application development.
Further reading
