A health monitor for market data feeds

The real-world problem

Every stock exchange publishes a non-stop stream of small messages: new orders, cancellations and trades. Trading firms build their view of the market from that stream, so every message matters. If one goes missing, their picture of the market is wrong.

Each message carries a sequence number (1, 2, 3 and so on), so a missing one shows up as a skipped number, called a gap. To guard against gaps, exchanges send everything twice, over two separate network routes called the A feed and the B feed. The receiver keeps whichever copy arrives first. If one route breaks, the other carries on and nothing is lost, but there is no longer a safety net.

The companies that run these networks give their clients monitoring dashboards that show, live, which feeds are healthy, which have failed over to their backup route, and where data has been lost. FeedWatch is a small, simulated version of that kind of dashboard, inspired by the real ones. Every exchange and every message in it is fictional.

How it works, in five steps

1. Simulate

   Each exchange sends about five numbered messages a second, on both its A and B feed. A usually arrives a few milliseconds sooner, because it's the faster route.

2. Arbitrate

   For each message number, the first copy to arrive wins and the later copy is dropped as a duplicate. The dashboard shows how often A wins compared with B.

3. Detect

   If a number is skipped, FeedWatch waits two seconds in case it's just late, then checks the other feed. If B had it, only the backup copy was lost; several of those in five minutes turn the exchange amber. If neither feed had it, data is gone and the exchange turns red. A feed that goes quiet for two seconds means the exchange is running on one route. Latency is compared with what's normal for that feed.

4. Record

   Each problem opens an incident, which stays open while the problem lasts and closes with an honest duration. A feed that flickers off and on counts as one incident.

5. Show

   Your browser asks the server for the latest state every two seconds and updates the map, tiles, chart and alerts. If the connection drops, the page says how old its data is instead of pretending.

Want to see it react? An administrator can use the chaos panel to break a feed on purpose. Everyone sees a banner while a fault is active, and faults undo themselves after five minutes.

Architecture

FeedWatch is deliberately small, so every part can be explained. It runs on ordinary shared web hosting, which can't keep a program running in the background. So instead of a simulator that runs all the time, each request asks "what should have happened since last time?", and the server works that out, saves it and answers.

Browser dashboard

One HTML page per screen, filled in by small JavaScript modules. No framework and no build step. The map uses Leaflet and the chart uses Chart.js, both served by FeedWatch itself.

JSON API

Every request goes through one PHP entry point, which looks up the route, runs the handler and adds security headers. Answers are cached for a second, so many viewers don't mean many times the work.

Engine

The simulation and detection logic is plain PHP with no database calls inside it. Time and randomness are passed in, so the same inputs always give the same result, which makes it easy to test thoroughly.

Database

MariaDB keeps only the last 15 minutes of raw messages, plus per-second and per-minute summaries, gaps and incidents, so it stays small on shared hosting.

Cron

Runs every minute to keep the simulation going, summarise and tidy up old data. It stops simulating after 30 minutes with no viewers, so it doesn't load the server for nothing.

How it was built

FeedWatch was built with AI coding agents doing much of the drafting and typing, inside a process that keeps the design decisions with a person. Nothing is built until it has been specified, decided and planned, and each of those steps leaves a written record in the project's documentation.

1. Research

   Background on market data, feeds and monitoring was gathered and written up as short reference pages.

2. Specify

   Each feature became a requirement with checkable acceptance criteria, for example "if neither feed received message 1040, it counts as data loss and raises a critical alert".

3. Decide

   Each design choice, such as "polling or websockets?", got a decision record with the options, trade-offs and a recommendation. The author made every decision.

4. Plan

   Each slice of work got a written plan: tasks, tests and what changes. No code was written until the plan was approved.

5. Build

   Code and tests were written together. Every change runs the automated tests, static analysis and a colour-contrast check, locally and in continuous integration.

6. Explain

   Each part has an "explain it in 30 seconds" summary and likely questions, so the author can explain every component without notes.

7. Release

   Each deploy ends with a smoke test of the live site: it checks the new version is running, the database is reachable, and private files can't be reached from the web.

Technology and limits

Built with PHP 8.4 with no framework, MariaDB, HTML, CSS and JavaScript modules, Leaflet with OpenStreetMap tiles, and Chart.js. It is hosted on ordinary shared web hosting.

• All data is simulated and every exchange is fictional. FeedWatch doesn't connect to real market data.
• It monitors feed health (sequence, timing and availability), not prices or trading.
• It is sized for a demonstration, at a few dozen messages a second. Real feeds carry millions, which would need a different design: a long-running receiver, streaming instead of polling, and a time-series store.

Back to the dashboard