The situation
The product gives people a way to rehearse conversations they cannot easily practice anywhere else. A nurse preparing for a dementia care discussion, a law enforcement officer working through de-escalation, a candidate rehearsing for an interview, or a financial advisor preparing a difficult client conversation can each hold the exchange with an AI avatar, and then ask the avatar to step out of role and give feedback on what went well and what to change.
The engineering challenge sits in that first part. An avatar that drifts out of character, forgets the storyline, or starts freelancing advice destroys the training value of the exercise. Holding a persona steady across long unscripted multi-turn conversations is difficult, and demonstrating that it holds reliably is more difficult still.
The approach
Prompt architecture for variance control. The central work was designing prompt structures that constrain model variance enough to hold character fidelity, tone, and realism across extended conversations that follow no script. These were published as best practice and adopted as the standard for a team of junior engineers.
A measurable evaluation discipline. A statistical test-analysis suite captured results across many interactions and isolated which prompt features improved outcomes and which degraded them. The suite began in a spreadsheet and moved to a local database as the volume grew. Alongside it, a config-driven evaluator checked adherence to the guidelines, flagged violations and storyline inconsistencies, and recommended corrections, with the guidelines themselves editable through configuration rather than code.
Local development and benchmarking. An application replicating the conversational behavior without the image or audio layers allowed development and regression testing against locally hosted models rather than burning API spend. That same harness supported benchmarking Gemma 2, Llama 3, and Qwen 2.5 on character fidelity, avatar realism, instruction adherence, and feedback quality.
Fine-tuning for cost and fidelity. A workflow starting from smaller open-weight bases produced character-specific models that held fidelity and delivered a realistic experience while improving hardware performance and reducing token cost.
The outcome
The prompt architecture became the team standard, which mattered more than any single conversation quality improvement because it made the whole team’s output more consistent. The evaluation tooling turned prompt engineering into a discipline with measurements attached rather than a matter of individual judgment. The fine-tuned character models improved the unit economics of a product whose gross margin depends directly on cost per conversation.
What made it work
Building the measurement before scaling the prompts. Without a way to tell which prompt changes helped and which quietly hurt, a team of engineers iterating in parallel will produce drift rather than improvement, and nobody will be able to say when it started.