Building an Observability Dashboard for AI Agents at Scale

As AI agents became part of developers' daily workflows, our customers started asking a different set of questions. Instead of asking whether we could capture AI traffic, they wanted to understand how AI was actually being used across their organization: which employees were using which agents, what conversations took place, and how much each interaction was costing them.

We already supported six AI agents—Claude, Cursor, Copilot, Codex, Gemini, and LangChain—with installers hooked into each one, capturing every request and response before shipping it to our backend. Capturing the data wasn't the problem. Making sense of it was.

Every request was stored as an independent event. There was no notion of sessions, message ordering, or tool-call spans within a conversation. To keep storage costs down, we only retained the latest ten samples per user, which meant customers trying to understand how their teams actually used AI had very little context. The goal became turning disconnected requests into searchable, navigable conversations.

Challenge 1: Making six agents speak the same language

The first assumption to break was that there'd be a standard way agents represent a conversation. There isn't one. Some agents hand you a clean message identifier on every event. Some expose only a session id. Others expose almost nothing you can build on directly.

Regardless of where an event came from, everything downstream expected the same normalized record:

• Session ID
• Conversation ID
• Message ID
• Timestamp
• User
• Span

Supporting each agent individually wasn't difficult. The real engineering problem was designing an abstraction so that the next integration would be another adapter, not another redesign.

Across our six agents, identity resolution collapsed into three underlying strategies—the real question was never "What's the message ID?", it was "How would we even find it for this particular agent?"

Trust it

Cursor was the easiest case. Every hook invocation already carried a stable identifier for the current turn, so we simply propagated it into our normalized event.

Borrow it

Claude doesn't expose a message identifier through its hooks, but it maintains its own local transcript. We treated that transcript as a secondary source of truth, reading the latest entry and reusing its identifier for the current event. It wasn't intended for this purpose, but it proved reliable enough in practice.

Invent it

Copilot and Gemini were the hardest. They exposed a stable session identifier but nothing that uniquely identified an individual message, and there wasn't another source we could borrow from. Codex and LangChain landed in the same bucket—session-level identity only, nothing at the message level—so they fell back to the same approach.

The only option was to generate our own message identifiers by maintaining a counter per session.

That sounds simple until you remember every hook invocation is effectively stateless. Each event arrives independently with no memory of previous requests, so the counter couldn't simply live in process memory. It had to be persisted and recovered between invocations, otherwise message numbering would silently break.

None of these techniques is particularly complicated on its own. What made this reusable was hiding all three behind a single abstraction: storage, search, and the dashboard never knew whether a message identifier had been provided by the agent, recovered from a transcript, or generated by us—they simply consumed one normalized event shape.

Challenge 2: Handling ingestion at scale

Once identity was solved, throughput became the next challenge.

Even a moderately sized engineering organization using AI assistants throughout the day can generate a sustained stream of prompts, tool calls, and responses. Every interaction becomes an event that needs to be stored reliably. Losing data wasn't an option, so our architecture favored availability with eventual consistency.

Initially, every incoming event triggered a direct write to Elasticsearch.

The first optimization was straightforward. Each tenant's aggregator accumulated requests in memory and flushed them either every five seconds or after collecting one hundred events. Batching dramatically reduced network overhead and improved throughput.

It also exposed a new bottleneck.

Instead of many small writes, every tenant was now issuing concurrent bulk writes into the same storage layer. Under sustained load, Elasticsearch couldn't drain those requests quickly enough, and the shared write service started exhausting resources. Batching had solved the request-volume problem while exposing a concurrency problem.

That's where Kafka fit naturally into the architecture.

Aggregator
    │
    ▼
Kafka (partitioned by tenant)
    │
    ▼
Consumers
    │
    ▼
Elasticsearch

Incoming events were written to Kafka and partitioned by tenant, preserving ordering within each tenant's stream while absorbing traffic spikes. Consumers processed those events asynchronously and wrote them into Elasticsearch at a rate the cluster could comfortably sustain. Instead of dropping writes during peak traffic, the system now naturally buffered bursts and recovered as consumers caught up.

Choosing the storage model

At first glance, ClickHouse seemed like the obvious choice. We were ingesting what looked very much like logs.

But our customers weren't asking questions like "Show me every request from yesterday."

They wanted to:

• Search for a specific prompt.
• Find every conversation for a user.
• Filter sessions across multiple fields.
• Navigate an interaction from start to finish.

That's fundamentally a search workload rather than an analytical one, which made Elasticsearch the better fit.

The next design decision was choosing the storage granularity.

We chose to store one document per span—every LLM invocation or tool call became its own document. There was no session document anywhere in the index. A session existed only implicitly as the collection of span documents sharing the same session identifier.

The alternative would have been maintaining a session-level document that continuously accumulated token counts, models, users, and metadata as new spans arrived. Reads become extremely cheap, but every write now requires merge logic, read-before-write semantics, and careful concurrency handling.

Instead, we kept writes append-only and shifted that complexity into the query layer.

Reconstructing conversations at query time

Since sessions weren't stored directly, the dashboard rebuilt them dynamically.

The session list grouped span documents using a terms aggregation on the session identifier. Nested aggregations calculated token counts, models used, span counts, and other rollups directly from the matching spans. Nothing needed to be maintained during ingestion; the dashboard computed exactly what it needed when a user queried it.

Opening an individual session was much simpler. We filtered by session identifier and sorted the resulting spans chronologically. Since every span already contained timestamps and duration, the conversation and its execution trace naturally emerged from the ordering itself.

The two views also used different pagination strategies. Session browsing relied on search_after because deep offset pagination in Elasticsearch becomes increasingly expensive as the engine skips more results internally. By carrying forward the sort values of the last document instead, pagination remained efficient regardless of how far someone scrolled. Individual session views, on the other hand, were naturally bounded and didn't require that optimization.

Looking back

Six agents reduced to three identity strategies, writes that stayed append-only under sustained load, and conversations reconstructed entirely at query time—that's the shape the system ended up taking.

Once those pieces fit together, adding another agent became another adapter instead of another architectural problem. The ingestion pipeline stayed simple, the storage layer stayed efficient under load, and customers finally got what they were actually asking for: a searchable, navigable view of how AI was being used across their organization.