Should you’ve ever debugged a distributed system (or worse, a {hardware} + software program integration) you recognize the ache. A important consumer move breaks. You open logs and discover hundreds of strains spanning machine firmware, the OS, utility code, and backend companies. No clear start line. Half your day gone by the point you piece it collectively.
That was our actuality. We run a manufacturing machine ecosystem that processes actual transactions, and when one thing breaks, it immediately impacts customers and income.
Like many groups, we tried the plain strategy: throw logs into an LLM and ask what went unsuitable. Generally it labored. Typically it hallucinated error codes, misinterpret flows, and gave totally different solutions for a similar enter on consecutive runs. By no means reliable sufficient to behave on with out a full guide evaluate.
The issue wasn’t intelligence. We lacked construction. We received inconsistent outcomes as a result of we hadn’t instructed the mannequin learn how to assume. That realization led us to micro-agent structure.
What micro-agent structure means
A number of issues get known as “brokers” in the present day. Micro-agent structure is extra particular. A micro agent solves one well-defined downside, operates beneath a strict contract, and produces constant, structured output. Consider it like microservices, however for reasoning. You would not construct a monolithic backend that handles authentication, billing, and notifications suddenly. The identical precept applies to AI tooling.
The important thing perception: the worth of an agent comes not from how good it may be, however from how constrained it may be. A general-purpose assistant tries its greatest on the whole lot and excels at nothing constantly. A micro agent does one factor, does it the identical means each time, and tells you when it may possibly’t.
Why this mattered for us
A single manufacturing situation in the environment can contact infrastructure, the platform layer, utility logic, and backend companies. When somebody studies “this move does not work,” the failure may dwell in any of these layers or lower throughout them.
Engineers sometimes scan logs, correlate occasions, cross-reference docs, and kind a speculation. For us, this took two to 4 hours per incident. When the issue crossed layer boundaries, it took even longer as a result of totally different groups personal totally different layers with totally different logging codecs.
We wished AI to try this similar investigation, however constantly, predictably, and grounded in actual proof. We constructed a micro agent with one job: flip uncooked logs right into a structured, evidence-backed prognosis. We name it the Bug Triage Agent. It does not repair issues, counsel code modifications, or file tickets. It analyzes logs, identifies the failure, and produces a structured report. One job, executed reliably.
From prompts to contracts
Early on, we used detailed system prompts. The outcomes have been inconsistent. Identical log file, totally different diagnoses on totally different runs. The mannequin would skip steps, add hypothesis, or invent unsupported findings.
We outlined a contract as an alternative. The agent should observe an outlined evaluation sequence, produce output in an actual construction, and declare when proof is inadequate. If it may possibly’t conclude, it returns INSUFFICIENT_EVIDENCE. No guessing.
In manufacturing debugging, a unsuitable reply is worse than no reply. A believable however incorrect prognosis sends an engineer down a multi-hour useless finish. The contract additionally establishes priority: guidelines override directions, which override examples. Each rule lives in version-controlled markdown, reviewable by the identical engineers who construct the merchandise.
The place the intelligence lives
The contract defines how the agent behaves. Abilities outline what it is aware of. Every ability handles a selected a part of the triage downside. The next sections describe the talents contained within the contract.
Log evaluation
The agent does not simply seek for “error.” It appears to be like for damaged flows. In a wholesome system, occasions observe a predictable sequence. When the agent sees an initiation occasion adopted instantly by termination with no processing steps, it flags the hole.
The next occasions are a typical indicator of a damaged move:
[Client] Request began
[Service] Timeout ready for dependencyAs a substitute of simply flagging a timeout, the agent acknowledges the failure got here from a downstream dependency, not the shopper. That modifications which crew investigates and the place they appear. The ability additionally adapts to totally different log codecs throughout {hardware} variations, inferring the setting from log signatures. Past particular person occasions, the agent correlates entries throughout parts by timestamp and causality, assembling a single timeline from set off to failure. Three unrelated-looking errors develop into one cascade with a transparent root trigger.
Structured information decoding
Many methods encode important state in structured codecs (JSON, binary protocols, telemetry). The decoding ability parses uncooked information, maps fields to readable meanings, and understands the relationships between fields. Particular worth mixtures reveal the system’s decision-making logic: why a request succeeded, failed, or took an sudden path. This usually requires a specialist with the spec memorized. The ability encodes that experience as guidelines that replace as merely as enhancing a markdown file.
Grounding expertise in actual references
Abilities alone aren’t sufficient. LLMs have broad information, but it surely is not at all times dependable for domain-specific details. We noticed the mannequin invent error codes that did not exist, or misattribute actual codes to unsuitable failure modes. The output regarded authoritative, which made it harmful.
We grounded the agent in authoritative reference paperwork: specs, error mappings, protocol definitions. In case your area has formal specs (RFCs, API docs, {hardware} references), feed them on to the agent. Grounding eradicated a whole class of hallucination for us, and it is probably the most transferable lesson from this venture.
What modified
Earlier than we created the Bug Triage Agent, an engineer pulled logs, spent two to 4 hours digging, looped in different groups, and filed a report. Now the agent produces a structured prognosis in minutes. It does not substitute the engineer. Somebody nonetheless validates, decides on the repair, and ships it. However the guide log archaeology dropped dramatically.
The larger shift was consistency. 5 engineers investigating 5 incidents manually produce 5 totally different approaches. The agent produces the identical structured format each time: noticed details (separated from evaluation), failure area, confidence rating, and subsequent steps. That made it far simpler to identify systemic patterns as an alternative of treating each incident as a one-off.
What you possibly can apply
Should you’re constructing AI-assisted tooling on your personal area, this is what we realized alongside the way in which.
Begin slender. Decide the highest-value, most repetitive job your crew handles and construct a centered agent for that one factor.
Use contracts, not prompts. A immediate provides the LLM solutions. A contract provides it constraints. Constraints produce reliability.
Modularize reasoning into expertise. Break area information into centered expertise as an alternative of 1 large immediate. Simpler to keep up, check, and lengthen.
Floor the agent in actual references. Do not belief the LLM’s normal information for details that must be precise.
Separate details from evaluation. Require the agent to label what it noticed versus what it inferred. Retains validation quick and the engineer in management.
What comes subsequent
Our Bug Triage Agent solves one downside properly. The subsequent step is orchestration. One agent identifies a defect, one other maps it to code, a 3rd drafts a repair, and an orchestrator coordinates the sequence. That is macro-agent structure, and it is the place the following wave of worth will come from. Macro methods solely work if the elements are dependable. Unreliable micro-agents produce unreliable macro methods. That is why we began right here.
We did not got down to construct a “good AI.” We got down to make debugging quicker, extra constant, and extra dependable. Micro-agent structure gave us that. Now it is the inspiration for what comes subsequent. Micro brokers assist us perceive methods. Macro brokers will assist us evolve them.
