Observability
Elastic MDS adapts, rebalances, and manages itself. Orchestration decides where content lives, how many shards are needed, and how to respond when load changes. That is all well and good, but if you work in a support or operations role, you probably have some questions.
How do I know where a specific user connected? Where is their data coming from? Is it stale? Is it stale for all users or just them? Why? Is the system doing what I think it is doing?
These are not abstract concerns. They are the questions that come up in production, usually under pressure. The Elastic MDS REST API is the way to get answers to all these questions and more.
One access point, the whole system
The REST API is not a single API. It is many, covering every service in the deployment. What makes it practical is the API Gateway: a single endpoint that provides access to all of them. You do not need to know how many projectors are running or what machines they are on. You query the Gateway and it routes your request to the right place, aggregates the responses, and returns them. The response data is standard JSON.
The level of detail available ranges from high-level service status down to the current value of a specific field in cache. Want to know what version of software is running across the cluster? One query. Want to know the current BID value for VOD.L? One query. Same endpoint, same mechanism, any level of detail.
For example, the state of every projector in the cluster:
GET http://mf-api-gateway:9090/api/application-state/v1/container.image~projector/container.image,state,infoOr at the other end of the spectrum, the current BID for a specific instrument:
GET http://mf-api-gateway:9090/api/content-adapter/v1/info.tableName=RDF/cache/itemField?key=VOD.L&field=BIDThat query goes to the Gateway, which routes to the appropriate API endpoints, retrieves the current cached value, and returns it.
Self-documenting APIs
Every API available through the Gateway is self-documenting in OpenAPI format. The documentation is constructed on the fly at startup in the various endpoints and is accessible from the Gateway itself. The documentation is never out of date. And using industry standards means that standard tooling works out of the box: curl, Python, any HTTP client.
It also means tools can learn the APIs programmatically. As one example, I have defined skills so that Claude understands how to navigate the Elastic MDS APIs. More accurately, Claude can teach itself how to navigate them from the documentation the Gateway serves. This means that Claude can easily answer those questions raised above. That kind of integration is straightforward when the APIs are self-describing and the data is standard JSON.
What you can see
In practical terms, the observable state of the system includes: which services are running and where, which applications are connected and to which shards, what content each shard is serving, load statistics per shard, log output from all nodes in a single stream, and even the current cached value of any content field.
Critically, you can see orchestration activity as it happens. Shard splitting in progress. Conflation engaged for a specific application. Load rebalancing between projectors. You are not inferring these from downstream symptoms. You are watching them directly.
You can also see the progression of a struggling application. Shard splitting engaging as the first mitigation. Conflation engaging if that is not enough. And overflow, the point at which an application is genuinely unable to keep up despite both mitigations. Seeing an application approaching overflow before it gets there is the difference between a managed situation and an incident.
Orchestration manages the system. The REST API lets you watch it do that. Those are different things, and both matter.
The dashboard
The dashboard is built entirely on the REST APIs. Everything it shows, you can also query directly. It is not a separate system with a separate view of reality. It provides a logical organization of what the API sees, along with visualizations and convenient navigation.
For operations and support use, the dashboard provides application state at a glance, access to log output from all nodes with drill-down, and real-time visibility into orchestration activity. For shard-related content specifically, there is a graphical visualization that shows shards, their connections, and how they change as splitting and rebalancing occur. That visualization is best seen in motion rather than described in text.
Video: C.1 — Dashboard walkthrough. System state, logging, and drill-down in practice.
Video: A.1, A.2, A.3 — The shard visualization is used throughout the shard-splitting and conflation videos.
The operational picture
A system that adapts autonomously is only trustworthy if you can understand what it is doing. And why. The REST API is foundational to the design of Elastic MDS. With it, you can understand what resources are allocated and where; spot problems in real time; drill down to root causes when you need to; and watch orchestration as it adjusts and rebalances. The system is not a black box.
Related
About this blog
Elastic MDS is an autoscaling, real-time market data distribution platform developed by MetaFluent. This blog is about Elastic MDS infrastructure deployment and application development.
Further reading
