The situation
The insurer had built an internal tool operating much like a public chat assistant, hosted on isolated infrastructure with models served through a private cloud workbench. The purpose was straightforward: give staff the capability without company data leaving the boundary or being absorbed into an external provider’s training.
That design carried an inherent tension. Security and isolation argued for keeping everything internal, while cost control argued for careful management of how often the largest frontier models were called at all. The platform leaned heavily on tokenization and retrieval to answer as much as possible before escalating to an expensive model.
The engineering team had a roadmap for extending the tool and wanted technically capable business users to assess its real capabilities and limits before committing further.
The approach
Model benchmarking. Several models available on the private workbench, including Llama-Scout, Mistral, and Cohere, were benchmarked across four task classes: short-question accuracy, long-question accuracy, coherence of response, and overall response quality. Separating these matters, because a model that performs well on short factual retrieval often degrades noticeably on extended reasoning, and choosing on a single aggregate score hides exactly that.
Agent roadmap. Three candidate workflows were identified and designed:
- First Notice of Loss extraction, automating the capture and submission of claim data at intake
- Policy submission ingestion, handling the document flow at the front of the underwriting process
- A data transformation agent that pulls specific target fields out of spreadsheets arriving in arbitrary formats, which is emphatically not deterministic data engineering and is a poor fit for conventional pipelines
Retrieval architecture. A Lite-GraphRAG prototype with document classification, designed to extend retrieval coverage beyond what the incumbent approach was reaching.
The outcome
The benchmarking gave the engineering team an evidence base for model selection per task class rather than a single default, which has direct consequences for both response quality and token spend. The agent roadmap moved the platform from a general-purpose assistant toward specific, measurable insurance workflows, and the retrieval prototype demonstrated a path past the coverage ceiling the existing approach had reached.
What made it work
Treating unit cost as an architectural decision rather than a monthly surprise. In an isolated deployment where the organization pays for every call, deciding which task class justifies which model is a design choice that shapes what the platform can afford to do at scale.