Artificial intelligence is changing how software is built, tested, and deployed. Modern AI models can accelerate development, generate code, analyze data, and simplify complex workflows. However, successful AI adoption depends on more than selecting the latest or least expensive model. It requires careful evaluation, practical engineering experience, and informed decision-making.
Visus recently demonstrated this principle while developing an AI-assisted reporting proof of concept for a client. Rather than accepting common assumptions about model selection, the team measured real-world performance across the entire application workflow. The results reinforced an important lesson: human judgment remains essential for delivering reliable AI solutions.
The Challenge
The client operated an established software platform that supports organizations managing people, events, follow-up activities, and operational reporting. The goal was to create an AI-assisted reporting experience that allowed users to ask business questions in natural language and receive accurate, organization-specific reports.
The solution combined a traditional application architecture with Azure OpenAI to generate SQL queries, validate results, and interpret report data.
One of the project's most important decisions involved selecting the AI model that would power the reporting workflow. The initial assumption was straightforward: a smaller, lower-cost model would provide the best value for the proof of concept.
While the model produced technically correct results, it failed to deliver the responsiveness needed for a production-quality user experience.
The challenge extended beyond finding a model capable of generating accurate SQL. The team needed a solution that balanced response quality, reliability, token consumption, execution speed, operating cost, and overall user experience.
Looking Beyond Token Pricing
Published model pricing told only part of the story.
The lower-cost model offered a lower price per token, but it often required more processing time and, in some cases, consumed more tokens to complete the same task. A newer, more capable model carried a higher published token price yet completed identical workflows significantly faster while making more efficient use of the available context.
These findings demonstrated that selecting an AI model involves more than comparing pricing tables.
Prompt design, validation logic, application architecture, token efficiency, and the number of language model interactions all contribute to overall performance and cost. Measuring only token pricing would have produced an incomplete picture of the solution's true effectiveness.
The Visus Solution
To support objective decision-making, Visus implemented a configurable model evaluation framework within the reporting proof of concept.
The application was enhanced with features that enabled side-by-side testing of multiple Azure OpenAI deployments under identical reporting scenarios, including:
- A model selector within the user interface
- Multiple Azure OpenAI model deployments
- Per-call execution time metrics
- Input and output token tracking
- Estimated cost calculations
- Workflow-level performance totals
- Detailed logs for the Generator, Reviewer, and Interpreter stages
This approach allowed the team to evaluate actual application behavior rather than relying on assumptions based solely on pricing or model size.
Each reporting workflow was measured for SQL accuracy, report quality, execution speed, token efficiency, and overall responsiveness.
The Results
The evaluation led to a clear outcome.
Although the original model appeared less expensive based on published token pricing, its slower performance reduced the overall user experience. After testing multiple options, Visus selected a newer model that delivered the same functional accuracy while completing workflows considerably faster and using tokens more efficiently.
The updated solution provided:
- Faster report generation
- Equivalent functional accuracy
- Improved responsiveness for users
- More predictable execution
- A stronger balance between performance and operating cost
- Reliable data to guide future model-selection decisions
The project demonstrated that the lowest token price does not necessarily produce the lowest effective cost or the best business outcome.
Human Judgment Remains the Competitive Advantage
AI continues to transform software development by accelerating implementation, analysis, and experimentation. However, successful AI projects still rely on experienced engineering teams to define meaningful success criteria, validate results, interpret performance data, and challenge initial assumptions.
In this project, selecting a model based solely on published pricing would have resulted in a technically functional but operationally weaker solution. By measuring the complete workflow and evaluating latency, quality, token efficiency, reliability, and user experience together, Visus identified the option that delivered the strongest overall business value.
Organizations adopting AI should recognize that model selection is only one component of a successful implementation. Long-term success comes from evaluating how every part of the solution works together and using real-world evidence to guide technical decisions.
AI can accelerate development, but informed human judgment is what transforms AI capabilities into dependable, high-performing business solutions.