Small services, boring dashboards
There is a specific kind of dashboard that gets built right after a long incident: twelve panels, three colours per panel, a histogram nobody understands. It is impressive for about a week. Then the service is calm, the dashboard goes unread, and the next incident starts with everyone squinting at graphs that were tuned for the previous one.
One question per page
A dashboard is a tool for answering a question under time pressure. The good ones I have used answer a single question on the first screen: is the service healthy right now or which hop is slow or did the last deploy change anything. Everything else is a second page.
If a page needs a legend to explain what “healthy” means, it is not doing its job. The threshold should be drawn on the chart, not implied by the reader's experience.
Four panels I keep
- Traffic — requests per second, split by status class. One line for 2xx, one for 4xx, one for 5xx.
- Errors — the same 5xx line, expressed as a ratio, with the alert threshold drawn in.
- Latency — p50 and p99 against the budget line from the design document.
- Saturation — the resource that fails first: CPU, connections, or queue depth. Pick one and say why.
Each panel is one chart. No stacked area plots, no dual axes. If two numbers share a chart, they should share a unit and a range; otherwise they belong on separate pages.
Alerts are part of the dashboard
A chart that nobody would page on is documentation. That is fine — but it should live on the documentation page, not the front page. The front page should contain only the signals that are wired to an alert, so that the chart and the pager agree about what “bad” means.
If the dashboard is exciting, the service is not boring enough yet.
Retire panels
The hardest maintenance task is deletion. I try to review dashboards quarterly and remove any panel that has not been looked at since the last review. Grafana keeps a useful history of who opened what; the answer is usually that half the panels are decorative. Deleting them makes the remaining half easier to read at 3 a.m., which is the only time it matters.