AI Agent Development Company Case Study: From Pilot to Production

An AI Agent Development Company earns its value when an agent survives contact with enterprise reality: inconsistent documents, legacy APIs, complex permissions, ambiguous questions, and users who expect evidence rather than eloquence. The following composite case study draws on patterns common in large industrial service organizations. It follows a knowledge and workflow agent from discovery through production, including the architecture choices, evaluation thresholds, rollout metrics, and failures that shaped the final design. The organization and figures are anonymized, but the engineering sequence is representative of programs delivered by enterprise AI consultancies.

The client selected an AI Agent Development Company to reduce the time its field-support specialists spent researching maintenance incidents. The objective was not simply to answer questions. The agent had to locate approved evidence, compare equipment history with service guidance, propose diagnostic steps, create a draft work order, and escalate safety-critical cases. Each recommendation needed citations, every tool call had to honor the specialist's existing permissions, and no write action could proceed without confirmation.

The Starting Point: Fragmented Knowledge and Manual Investigation

The client supported approximately 42,000 pieces of industrial equipment across 17 countries. Its 1,180 service specialists handled about 31,000 support cases per month. A typical case required searches across a document repository, a product-lifecycle platform, the service ticketing system, an asset database, and a parts catalog. Specialists copied identifiers between applications, compared maintenance histories manually, and searched long PDF manuals for relevant procedures.

A four-week baseline study sampled 1,600 cases. Median research time was 38 minutes, while the 90th percentile reached 94 minutes. Specialists switched systems an average of 11 times per case. In 22 percent of audited cases, the final work note lacked a reference to the governing service instruction. Seven percent used an outdated manual revision, usually because search results favored frequently accessed files rather than the latest effective version.

The company had already piloted a general-purpose copilot over a folder of manuals. Users liked its conversational interface but distrusted its answers. Only 61 percent of material claims were supported by the cited passages, and citation validation found that 18 percent of links pointed to a related document rather than the specific evidence used. The pilot could not access asset history or create work orders, so specialists still performed most of the original workflow.

The initial lesson was decisive: the problem was not a shortage of text generation. It was a broken research path spanning knowledge retrieval, asset context, permissions, and transaction systems. The business case would depend on reducing end-to-end resolution effort while improving evidence quality, not on maximizing chatbot engagement.

Discovery and the Production Contract

The AI Agent Development Company ran discovery sessions with field specialists, service engineering, safety, cybersecurity, data owners, and platform teams. Rather than collecting a broad wish list, the team decomposed 126 observed cases into tasks. These included identifying equipment configuration, finding applicable service instructions, checking prior incidents, verifying parts compatibility, drafting diagnostic steps, and creating a proposed work order.

Feasibility scoring narrowed the first release to three equipment families representing 37 percent of monthly case volume. The selected cases had stable source systems, well-defined escalation routes, and enough historical outcomes to build evaluation data. Autonomous closure was excluded. The agent could research and draft, but a specialist remained accountable for diagnosis and approval.

The production contract defined five permitted outcomes: answer with validated citations, ask for a missing identifier, propose diagnostic steps, prepare a draft work order, or abstain and escalate. Safety-critical symptoms triggered mandatory escalation before tool execution. A response could not recommend a procedure unless the applicable document revision and equipment configuration had both been verified.

Success gates were agreed before implementation. The agent needed at least 90 percent evidence recall on the golden dataset, 95 percent citation correctness, fewer than 2 percent unsupported material claims, zero confirmed cross-permission disclosures, and a median response time below 12 seconds for research-only questions. The workflow target was a 30 percent reduction in median research time without increasing reopened cases.

Building the Knowledge Foundation

Enterprise Knowledge Engineering began with 2.7 million candidate files from six repositories. Inventory analysis found that 31 percent were duplicates or superseded revisions, 14 percent lacked reliable equipment-family metadata, and 9 percent were scanned images requiring optical character recognition. Manuals contained multi-page tables, diagrams, warning panels, and procedural dependencies that ordinary text extraction distorted.

The ingestion pipeline classified documents by type and authority, extracted structural elements, resolved versions, and attached effective dates. Service engineering supplied precedence rules for conflicts between global manuals, regional notices, and temporary safety bulletins. Each document received equipment-family, model, component, language, region, revision, owner, confidentiality, and validity metadata. The index stored the source repository identifier so citations could resolve to the authoritative file rather than a copied artifact.

Chunking varied by content type. Procedures were segmented at step groups while preserving warnings and prerequisites. Troubleshooting tables were represented as structured rows linked to their headers. Parts catalogs used component-level records with parent assembly context. Safety bulletins remained intact because separating conditions from mandated actions created unacceptable risk. Parent-child retrieval allowed focused matching followed by expansion to the surrounding authoritative section.

After deduplication and scope filtering, 684,000 approved documents and records entered the first production corpus. A daily incremental job handled additions and revisions, while deletions and permission changes flowed through a higher-priority queue. Quarterly reconciliation compared repository access lists with index metadata to detect synchronization gaps. This work consumed more effort than the initial prompt design, but it removed the largest source of pilot unreliability.

Designing Retrieval and Agent Orchestration

Retrieval Pipeline Engineering used a two-stage design. The first stage applied permission, equipment, region, language, and effective-date filters. It then combined semantic search with lexical matching because model numbers, alarm codes, and part identifiers performed poorly under embeddings alone. Reciprocal rank fusion merged the candidate lists, and a cross-encoder reranker selected the passages most relevant to the specific diagnostic question.

Context assembly grouped evidence by authority and workflow purpose. The LLM received separate blocks for current safety notices, applicable procedures, asset history, similar resolved cases, and parts compatibility. Source revision and effective date accompanied every passage. When retrieved sources conflicted, the agent followed the documented precedence rule and disclosed the conflict instead of synthesizing a compromise.

Enterprise Agent Orchestration was implemented as a bounded state machine rather than an open-ended autonomous loop. The states covered intent classification, identifier resolution, evidence retrieval, asset lookup, plan drafting, policy validation, user confirmation, work-order creation, and completion. Each state had permitted tools, validated inputs, timeouts, retry policies, and explicit failure routes. The maximum path used seven model calls, while simple searches normally required two.

The agent accessed four APIs through a controlled tool gateway. Asset and case-history tools were read-only. The parts tool could check availability but not reserve inventory. The work-order tool could create a draft only after confirmation, and it used an idempotency key to prevent duplicate creation during retries. User identity was propagated to every retrieval and tool request, preventing the service account from becoming a permissions shortcut.

Human-in-the-loop design focused on consequential boundaries. Specialists did not approve every retrieval query or intermediate plan; that would have erased the time benefit. They did approve the final diagnostic plan and any draft work order. The interface displayed citations beside each material recommendation and showed which asset facts influenced the result. Escalations included a compact trace of missing evidence or failed checks so senior engineers did not have to restart the investigation.

Evaluation, Red Teaming, and the First Failed Release Candidate

The AI Agent Development Company and client engineers created a golden dataset of 3,200 cases. It included common issues, rare faults, ambiguous symptoms, outdated terminology, incomplete asset identifiers, multilingual questions, and cases with no supported resolution. Domain experts annotated expected sources, acceptable diagnostic steps, prohibited actions, and required escalation outcomes.

Evaluation separated the retrieval and generation layers. Retrieval measures included evidence recall, precision at five, and correct-version rate. Generation measures included groundedness, answer relevance, citation entailment, and procedural completeness. Workflow measures covered tool selection, parameter accuracy, confirmation behavior, final state, latency, and cost. A claim-level evaluator flagged statements that could not be traced to retrieved evidence, with human adjudication for safety-related cases.

The first release candidate failed despite strong demonstration results. It achieved 92 percent evidence recall and 96 percent citation correctness, but unsupported material claims remained at 4.8 percent. Analysis showed that the model occasionally filled procedural gaps using plausible industrial knowledge. The retrieved source would describe inspection steps, while the answer added an unapproved reset action that appeared reasonable but was absent from the evidence.

The team revised the generation contract so procedural recommendations had to map to explicit source spans. A post-generation verifier extracted each proposed action and checked it against cited evidence. Unsupported actions were removed or converted into an escalation. The context format also separated historical case outcomes from approved procedures, preventing successful past behavior from being presented as current policy.

Red teaming identified a second issue. Several archived service notes contained sentences resembling instructions to automated assistants. When those notes entered context, the model occasionally followed them as if they were system directions. The ingestion pipeline began labeling untrusted content, the prompt treated retrieved text strictly as evidence, and an injection detector quarantined suspicious passages for review. Tool policies remained external to the model, so injected text could not expand permissions.

After remediation, the final offline run reached 94 percent evidence recall, 97.6 percent citation correctness, and a 1.4 percent unsupported-claim rate. Correct-version retrieval rose from the pilot's 73 percent to 98.8 percent. All 640 permission-boundary tests passed, including adversarial attempts to retrieve cases belonging to other regions. Median laboratory latency was 8.9 seconds, and the estimated model cost was $0.19 per completed workflow.

Pilot Rollout and Production Results

The rollout began with 80 specialists in two regions. For the first two weeks, agent recommendations were shadowed: users could inspect them, but work-order drafting was disabled. Daily review sessions examined low-rated answers, abstentions, citation mismatches, and tool errors. Feedback was categorized into retrieval failure, source-quality issue, reasoning defect, interface confusion, or missing capability rather than being stored as undifferentiated thumbs-down events.

Shadow operation revealed that specialists often entered informal equipment nicknames. Exact filtering then produced no results even though the relevant asset existed. A governed alias service was added before retrieval, using mappings maintained by asset-data owners. The team also discovered that certain regional bulletins arrived 18 to 36 hours late because their repository did not emit reliable change events. A four-hour reconciliation job closed the refresh gap.

Write functionality was enabled gradually after these corrections. At 12 weeks, 420 specialists were using the agent. Median research time fell from 38 minutes to 23 minutes, a 39 percent reduction. System switching dropped from 11 to four transitions per case. The proportion of work notes containing validated governing references increased from 78 percent to 96 percent. Reopened cases declined from 8.2 percent to 6.9 percent, indicating that speed had not displaced resolution quality.

Users accepted or lightly edited 71 percent of diagnostic drafts. Fourteen percent required substantial revision, 9 percent were intentionally escalated, and 6 percent were abandoned. The abstention rate was initially viewed as a weakness, but review showed that most escalations involved missing configuration data or conflicting notices. Resolving those underlying knowledge defects became more valuable than pressuring the model to answer.

Median production latency settled at 9.6 seconds, with the 95th percentile at 18.4 seconds. Caching equipment metadata and parallelizing independent asset and case-history reads saved roughly 2.7 seconds per complex workflow. Routing classification and query expansion to smaller models reduced average inference cost from $0.19 to $0.13 without a statistically significant quality decline.

No cross-permission disclosure was confirmed during the first six months. Two near misses were detected when repository permissions changed while cached retrieval results remained active. Cache invalidation was linked to access-control events, and the maximum cache lifetime for sensitive sources was reduced. The incidents reinforced that authorization must be evaluated across indexing, retrieval, caching, and citation resolution rather than at login alone.

What the Architecture Team Would Do Differently

The first lesson was to involve source owners earlier. Several weeks were lost interpreting document authority that experienced service engineers could explain in a day. Retrieval quality depended as much on version rules and metadata stewardship as on the embedding model. Future programs would begin with a source-authority workshop before selecting index technology.

The second lesson was to build evaluation cases alongside ingestion. Waiting for a polished agent before constructing the golden dataset delayed important discoveries about missing documents and ambiguous expected outcomes. Early evaluation also prevented the team from over-optimizing for a small set of demonstration questions.

The third lesson concerned observability. Initial traces recorded model prompts and outputs but did not capture every pre-filter decision, candidate rank, or permission attribute. Diagnosing missed evidence was unnecessarily slow. The production LLMOps schema was expanded to record retrieval candidates, exclusion reasons, reranker scores, context selection, tool timing, model version, and citation validation results under controlled retention.

The fourth lesson was organizational. An AI Agent Development Company can design orchestration, implement integrations, and transfer engineering practices, but internal teams must own source quality, workflow policy, and acceptance thresholds. The client created a joint product group spanning service engineering, AI platform engineering, security, and knowledge stewardship. That group approved corpus changes, reviewed quality trends, and prioritized new equipment families.

Scaling the Pattern Beyond One Agent

After six months, the client had a proven reference architecture rather than a single isolated assistant. Shared services included identity-aware ingestion, a model gateway, vector index management, hybrid search, reranking, a tool registry, evaluation pipelines, citation validation, and trace collection. Domain teams could configure sources and workflow policies without rebuilding the control plane.

The next release used an Agentic RAG Solution to support warranty adjudication, but it did not copy the maintenance agent wholesale. Warranty work required different evidence precedence, financial approval thresholds, and adversarial tests. The reusable components shortened delivery, while domain-specific golden datasets and human checkpoints preserved the correct risk posture.

The AI Agent Development Company also introduced regression gates for every model, embedding, reranker, prompt, and index change. A release could proceed only if it maintained critical-case performance, permission tests, latency budgets, and cost ceilings. Canary deployment exposed a small percentage of traffic to new versions, while drift monitoring tracked changes in query types, retrieval coverage, and user overrides.

At the portfolio level, value was measured through completed workflows, research time, rework, escalation quality, and supported-claim rates. Conversation volume was deliberately excluded as a primary success metric. An agent that produces more dialogue but leaves the specialist switching among five systems has not solved the original problem.

Conclusion

This case shows why production agent development is an engineering and operating-model discipline, not a prompt-writing exercise. The successful AI Agent Development Company aligned a narrow use case with authoritative content, hybrid retrieval, bounded tool execution, measurable evaluation, human approval, and continuous LLMOps. Its most important outcome was not the 39 percent reduction in research time, but a repeatable pattern for trustworthy automation. For enterprises facing similarly fragmented knowledge and multi-system workflows, an Agentic RAG Solution can provide that pattern when retrieval quality, permissions, citations, and production ownership are designed as first-class requirements.