Resilience and Scaling
Any enterprise-class market data system has to provide resilience. The decisions an architect has to make are: how much, at what granularity, and at what cost in engineering effort and resource allocation. Elastic MDS was designed with this in mind. The goal was to eliminate single points of failure and minimize downtime when failures do occur, while keeping the system engineering work as simple as possible and letting you choose the best solution for your needs.
Coarse or fine-grained: your choice
Redundancy is inherently tied to the physical universe: hardware resources and executing processes. You choose the granularity of both. Elastic MDS services are decoupled from each other. They communicate through a shared data fabric that insulates them from topology decisions. This means you can run all services in a single process or spread them across many, and it makes no difference to the system. No configuration changes required either way.
Topology is a runtime composition decision, not a configuration problem. A fine-grained deployment runs each service, session handling, query analysis, orchestration, data distribution, in its own process or container. A coarse-grained deployment runs them all together. The processes can run on separate hardware or be arbitrarily grouped on shared hardware. Adjusting from one setup to another is simple because the various services are fully insulated from the deployment topology.
For resilience purposes this matters because the granularity of your deployment determines the granularity of your failure domains. A fine-grained process/hardware deployment means a failure of one service does not affect the others, and opens up a variety of approaches to disaster recovery that a coarser deployment cannot support.
Hot standby, no configuration
Every Elastic MDS service supports hot standby. You do not have to configure this, you just run more than one instance.
Want two session services? Start two. The system negotiates which is primary and which is backup at startup. If the primary fails, the backup takes over. Applications already connected see zero disruption, not even a blip. A new application connecting during the failover window experiences a normal connect delay of a second or two, configurable, and indistinguishable from any other connection timing. If you want three instances for additional redundancy, start three. There is no configuration change, no topology file to update. You don’t even have to restart any other services.
Video: B.2 — Session Service Failover. Fine-grained deployment, live failover, zero application disruption.
You want two session services? Run two. You want three? Run three. There is no configuration. You just start as many as you want.
Projectors and scaling
The main dimension that requires scaling is data ingest and distribution. That is the domain of projectors. When designing your system, you decide how much resource to apply for a given data source. You can combine multiple data sources into one projector or assign them to individual projectors. Then you decide how many of each configuration you want running.
If you change your mind, you can start another one. That is it. You could do this on a schedule or in response to some anticipated problem. As with the core services, this decision requires no configuration changes for the projector itself or anywhere else in the system. You can add new projector configurations and even entirely new data sources on the fly, while the system is running.
Video: B.1 — Projector Failure and Recovery. Shows what happens to applications when a projector fails and how orchestration manages recovery.
Static, manual, or dynamic
You have three options for how you manage projector scaling, and you can choose differently for different parts of your deployment.
A static system has a fixed number of projectors. You manage their life-cycle yourself, using whatever tools you use for your other solutions. Straightforward to reason about, easy to operate.
If you want to manually scale a system, you can do that, without downtime and without configuration changes. You make the call, the system adjusts.
A dynamically scaled system lets orchestration handle it. Running in Kubernetes, orchestration scales projectors up when resource demand increases and back down when it eases. This happens in real time, transparently to applications, and has the potential to significantly reduce costs by not running resources you do not currently need. The big difference between manual scaling and orchestration-driven scaling is that scale down is graceful. Applications experience no loss of data during a graceful scale down. Manual scale down is abrupt and applications experience data loss. That said, the recovery from a manual scale down is quick and otherwise seamless.
All three approaches are valid. The right choice depends on your operational model and how much variability you expect in your load profile. You are not locked into a decision at deployment time.
The system adapts to the resources you give it. Whether that is a fixed allocation you manage yourself or a dynamic pool that orchestration controls, the applications connected to it see the same thing: continuous, uninterrupted data.
Looking ahead
The granularity and flexibility described in this post opens up a variety of approaches to disaster recovery that are not available with conventional market data infrastructure. More on that in a future post.
Related
About this blog
Elastic MDS is an autoscaling, real-time market data distribution platform developed by MetaFluent. This blog is about Elastic MDS infrastructure deployment and application development.
Further reading
