How an autonomous AI agent orchestration system reduced support volume by 40%, achieved 95% accuracy, and built a governed AI ecosystem across a $1.5B+ ARR SaaS organization.
Kaseya's HR and AI Enablement teams were absorbing 1,200–1,400 internal support tickets per month. Response times ran 24–48 hours. Knowledge lived in scattered documents, email threads, and the heads of senior employees.
HR teams alone fielded hundreds of repetitive queries monthly — benefits, PTO policies, Oracle HCM questions — consuming senior team capacity that should have been focused on strategic work.
HR policies, AI governance guidelines, and operational procedures were scattered across SharePoint, email, PDFs, and tribal knowledge. No single source of truth existed for employees seeking answers.
As AI tools proliferated, employees were adopting unapproved tools through informal channels. No systematic evaluation process existed for security, compliance, or duplication risks.
A 5–10 minute average response window created friction in employee workflows, particularly for time-sensitive HR and billing questions that blocked day-to-day operations.
Rather than a single monolithic chatbot, the system was designed as a hierarchical agent network — a parent routing agent orchestrating purpose-built child agents, connected via automated Power Automate workflows.
↑ Hover each node for detailed implementation notes
Initial deployment achieved 74% accuracy — below the VP-mandated 90% threshold. A rigorous 5-phase testing program was designed and executed over 30 days, treating AI accuracy like a finance variance problem: diagnose root cause, fix systematically, validate continuously.
Excel tracking sheets with date, question, response, pass/fail, and HR spokesperson comments. Power BI dashboard for live accuracy monitoring. Established 74% baseline.
Deployed to 3 departments (SPG, HR, BIS). Real employees asked genuine day-to-day questions. Structured feedback through pass/fail tracking and open comments.
Employees assigned to intentionally "break" the agent — edge cases, ambiguous questions, trick phrasing. Identified failure modes that normal testing misses.
Excel automation firing 50–150 test questions in minutes. Rapid iteration cycles — testing no longer bottlenecked on human availability.
Every failure categorized: unclear source doc? vague prompt? misunderstood question? should escalate? Daily standups with HR & AI teams for iterative KB refinement.
VP threshold met in 30 days. Accuracy sustained across multiple months post-launch. System moved from pilot to enterprise-wide deployment.
Most AI projects fail not on technology, but on governance. A formal committee structure was established to evaluate all AI tool requests, prevent shadow AI adoption, and create repeatable decision-making standards.
| Tool Request Journey | Committee Action | Outcome |
|---|---|---|
| Employee submits Salesforce ticket for new AI tool | Anuj evaluates: functionality, existing tool comparison, market alternatives, pricing, security/compliance risks | Trigger review |
| Analysis deck presented to Governance Committee | Directors, VPs, Managers from IT / Legal / Security / Compliance review recommendation | Weekly cadence |
| Broad approval — no security/duplication risks | Added to Active AI Tools catalog. Pushed to AI Agent knowledge base automatically. | Approve company-wide |
| Uncertain ROI or partial risk | Limited rollout to defined team or use case with monitoring requirements | Approve pilot |
| Security, compliance, or duplication issue identified | Tool rejected with documented rationale. Employee directed to existing approved alternative. | Deny |
| Insufficient vendor security documentation | Vendor requested to complete formal security assessment. Review paused pending results. | Due diligence |
Measured outcomes sustained across multiple months post-launch — not point-in-time metrics, but operational baselines.
Four principles distilled from this transformation — applicable to any Finance AI initiative regardless of organization size or industry.
The most important thing we built wasn't the agent — it was the measurement system. Without an accuracy tracking framework in place from day one, we couldn't have systematically improved from 74% to 95%. Every AI deployment needs its own equivalent of a variance report.
The AI Governance Committee prevented shadow AI adoption while creating a clear, trusted pathway for legitimate tool adoption. Organizations that treat governance as bureaucracy will lose to those who treat it as a competitive moat — protecting against compliance risk while accelerating trusted adoption.
Standard user testing only shows you what works. Assigning employees to intentionally break the system revealed failure modes that would have emerged in production — at the worst possible time. Adversarial testing before launch is non-negotiable for enterprise AI.
The system used email — the channel employees already lived in. Zero new tools, zero onboarding friction, zero behavior change required. The fastest path to AI adoption isn't training people to use a new platform; it's embedding AI in the workflows they already have.