My AI Agents Were Working, but I Had No Visibility
This report shows how I integrated the agents behind Applied into LangSmith, an open-source observability framework, to track traces, token spend, and performance
Back in May, I wrote about the 6-agent system behind Applied.
The setup was simple but useful. The agents researched AI adoption use cases, tools, and vendors, then categorized everything into the Applied living map. There was also a translator agent, which made the information available in Spanish, and a match maker agent, which connected users with relevant content based on their preferences.
Since then, the system has grown. I added agents for open and closed model research, adoption signals from Hugging Face, OpenRouter, and job boards, plus a weekly digest agent that uses the match maker to send more personalized reports.
The agents were working. They were researching, extracting, classifying, translating, and turning messy AI adoption signals into structured information. I could see the final outputs and read the logs, but I could not clearly understand how the work was happening below the surface.
I wanted to answer simple but important questions.
Which agents were spending the most tokens?
Which tasks were failing?
How long did each step take?
That is what led me to LangSmith.
In this report, I’ll walk through how I integrated the agents behind Applied into LangSmith, what I can now see across the system, and why I think observability is only the first step toward improving multi-agent products.
What’s LangSmith?
LangSmith is an observability and evaluation platform for LLM and agent applications, built by the LangChain team. Despite the name, you don’t need LangChain to use it. LangSmith is framework-agnostic, which made it useful for Applied, where my agents were already running through a custom setup.
In short, LangSmith turns agent activity into traces. Each execution becomes a run with inputs, outputs, duration, status, token usage, cost, errors, and nested child steps. Instead of digging through scattered logs, you get a searchable history of what happened, how long it took, how much it cost, and where it failed.
That matters because once agents start doing real work, the question is no longer only whether they completed a task. The question becomes whether they completed it efficiently, reliably, and with the right level of quality.
In practice, LangSmith makes it simpler to trace and monitor runs, evaluate outputs, use annotation queues for human review, manage prompts outside the main codebase, and set alerts for cost spikes, latency issues, or rising error rates.
Why the Integration was Easier than Expected
LangChain, LangSmith, and LangGraph are open source and API-first, which makes them easier to use with AI coding tools like Claude Code. Their public documentation, accessible SDKs, and clear concepts give coding agents enough context to inspect, explain, and implement them.
Claude Code was able to understand LangSmith, help set up the integration, and guide me through the parts of the product I needed, including where to configure API keys in the UI.
As I wrote in my previous post, this is one of the advantages of open-source and API-first products in the agent era. They lower the barrier for developers, while giving agents enough context to reason about, implement, and extend them.
How Data Flows into LangSmith
There are a few ways to send traces into LangSmith. The Python or JavaScript SDK can wrap LLM calls, run records can be sent manually through the REST API, and OpenTelemetry can be used when an application is already instrumented that way.
For Applied, the important part was that LangSmith did not require rebuilding the agent system around LangChain. The agents keep running exactly as before. A lightweight step was added so that, once a routine finishes, a summary of what happened is sent into LangSmith as a trace. The trace measures the run without changing how the agents do the work.
Two things make that summary useful. First, each agent marks its own stages as it goes, such as “starting Phase 1” or “finished Phase 1,” so LangSmith can show the run step by step instead of as one opaque block. Because these markers live inside the agent’s instructions, changing or renaming a stage is a small edit in one place.
Second, the summary includes the numbers that matter for observability: how much the run cost, how many tokens it used, how long each stage took, and which model ran it.
Open Agents Dashboard
Once the traces were flowing into LangSmith, I used their API to build a public dashboard, access it here.
The agents are grouped into four categories based on their role.
Research: Investigates AI use cases, tools, vendors, models, and adoption signals from the job market.
Data Quality: Reviews missing fields, logos, definitions, Spanish translations, and overall completeness.
Analytics: Updates rankings for models, tools, and vendors.
Communications: Creates email digests, matches users with relevant information, skips unsubscribed users, and adds safeguards against spam.
This first iteration shows historical runs, execution times, token spend by agent and overall, estimated dollar costs, latency, and trace details across phases and outputs.
A note on security and credentials
One important thing to watch is what gets sent into LangSmith.
Because traces can include inputs, outputs, metadata, and error logs, it is easy to accidentally send sensitive information such as API keys, emails, internal IDs, or private user data.
To avoid that, I added a small filtering layer before sending logs to LangSmith. This layer filters trace data and removes sensitive information, so the system can still capture useful observability data without transferring credentials or private details.
Next steps
Integrating Applied’s agents into LangSmith helped turn the lights on inside the system. I can now see historical traces, token spend per agent, failures, latency, and the sequence of steps behind each execution. Compared to reading raw logs, this makes agent behavior much easier to understand.
But visibility is only the beginning. The more interesting questions come after:
How can I reduce unnecessary token spend?
Which tasks could be routed to smaller, faster, or cheaper models?
Where should I introduce evaluations to measure error rates, success rates, and output quality?
Are the agents duplicating work or taking unnecessary steps?
Are the agents choosing the right execution paths, or drifting into unnecessary steps?
How can I move from inspecting individual traces to understanding the performance of the whole system?
As multi-agent systems become more independent, we need more than logs, traces, and dashboards. The next step is observability that does not just show what happened, but helps improve how agents work.