The Wizard Behind the Curtain
Shard splitting and orchestration have come up in earlier posts. If you have been following along, you may be wondering how that actually works. How does the client know where to connect? What governs how many shards there are? Who decides when to split one and when to reclaim it?
These decisions are the domain of the orchestration service. The orchestration service is not something your applications interact with directly. It operates entirely behind the scenes and drives everything that happens in the infrastructure. For a technical overview of the architecture see Architecture: Basics in the wiki.
Content placement
The orchestration service’s primary job is to determine a home for content served by the system. When a client subscribes to a topic, orchestration finds it. If the content is already in the system, the client is connected to it. If it is not, orchestration decides which projector will host it and arranges for it to be fetched.
This is system-wide watchlist management. Every subscription from every application is known to orchestration. If ten applications are all watching the same instrument, the data is ingested once and delivered to all ten. There is no per-application duplication of data ingest, no redundant cache entries, no wasted resources. The cluster maintains one copy of each piece of content, shared across all subscribers.
For a large deployment with many applications watching overlapping universes of instruments, this matters. The resource cost of maintaining content in the cluster scales with the content universe, not with the subscriber count.
Orchestration knows where everything is. Clients ask for data by name. The routing is not their problem.
Shard assignment and load balancing
Orchestration also determines what shards are required and how content is assigned to them. The management of shards as a resource is on-going over the life-span of an application.
Video: S.3 — What is a Shard? A short explainer with visualization, if you want a refresher before reading further.
Market data is unpredictable. Some instruments are quiet. Others trade continuously. A shard carrying a busy set of instruments will be under more load than one carrying a quiet set, even if the number of instruments is the same. Orchestration watches load statistics at the shard level, continuously. When it sees an imbalance it can adjust resources accordingly.
Orchestration can trigger the construction of new shards. It can move content from one shard to another. It can move content from one projector to another entirely. These are all load-balancing operations, and they happen dynamically, all the time, without operator intervention and without application changes. The balancing algorithms are controlled by policies that you can configure. If you want, you can also assign different policies to different data sources and applications.
This is what shard splitting actually is at the orchestration level. When an application is struggling, orchestration detects the condition proactively, decides that a new shard is needed, arranges for it to be created, and redistributes the content. The application sees nothing. The data streams are continuous throughout. No stale data, no lost ticks, no interruption of any kind.
Startup is a good illustration of this in action. Orchestration anticipates a heavy load at application startup. When an application connects and requests its full subscription universe, orchestration allocates the shards needed to deliver images quickly. Once initialization is complete, those extra shards are gracefully released with no loss of ticks. To put a number on it: 40,000 streams, fully initialized, in under 5 seconds. The conventional answer to the startup image burst is rate throttling, slowing delivery down to protect the infrastructure. Elastic MDS does the opposite: allocate more, deliver faster, scale down when the resources are no longer needed.
Video: A.1 — Adaptive Resource Allocation at Startup. Shows shard allocation during initialization and the scale-down once images are delivered.
What this means in practice
As a system engineer, your job is to define the goals and allocate the resources: how many projectors, how much network capacity, CPU, memory. What level of resilience, what load thresholds. Orchestration figures out how to meet those goals within the resources you have provided. You are not managing connection details. You are not deciding which content lives on which projector or how many shards a given application needs. Orchestration handles that, continuously, based on what is actually happening in the system.
These mechanisms are also part of the resilience story. Orchestration drives resilience at both the application level, through shard management, and at the projector level. More on that in a subsequent post on resilience.
Related
About this blog
Elastic MDS is an autoscaling, real-time market data distribution platform developed by MetaFluent. This blog is about Elastic MDS infrastructure deployment and application development.
Further reading
