QAgent
Automated QA for AI agents. Stop shipping on vibes.
QAgent is an automated QA and regression-testing platform for AI agents. Connect an agent via webhook in about two minutes (no SDK) and run parallel test cases — adversarial attacks, edge cases, multi-turn conversations — against ground-truth rubrics, with pass/fail scorecards and root-cause breakdowns across 8 dimensions including Policy Adherence, Factual Groundedness, RAG Faithfulness, and Multi-Turn Context Memory. Free tier: 100 test-case evaluations per month, no credit card required.
What is QAgent?
QAgent is an automated QA and evaluation platform for AI agents. It connects to an agent via webhook or API endpoint in about two minutes with no SDK, runs parallel test cases (adversarial attacks, edge cases, multi-turn conversations) against user-defined ground-truth rubrics written with RFC 2119 expected behavior, and returns deterministic pass/fail quality scorecards across eight dimensions: Answer Quality, Factual Groundedness, Policy Adherence, Escalation Correctness, RAG Faithfulness, Contextual Relevancy, Context Recall, and Multi-turn Context Memory. It is positioned for solo builders and small teams who ship customer-facing agents without a QA department.
Key features
- Webhook/endpoint connection in about 2 minutes, no SDK boilerplate
- Eight scoring dimensions including Policy Adherence, Factual Groundedness, RAG Faithfulness, and Multi-turn Context Memory
- Deterministic LLM-as-a-judge scoring against user-defined ground truth (pricing rules, refund windows, knowledge docs) with RFC 2119 MUST/MUST-NOT rubrics
- Parallel test runs with visual pass/fail scorecards and step-by-step failure root-cause breakdowns
- Prompt regression tracking with score deltas across prompt versions
- Built-in exemptions to reduce false positives: dynamic ticket IDs/session IDs, polite greetings, top-chunk focus, and jailbreak-test exemptions
- Quality audit report exports (CSV and print) for client hand-offs
- Free tier: 100 test-case evaluations per month, no credit card required
Who it's for
- Solo developers shipping customer-facing chat/voice agents who need regression tests after every prompt tweak
- Agencies handing off chatbots to clients with verifiable quality scorecards (groundedness, policy adherence, zero policy breaches)
- Teams gating releases on automated evals in their deployment workflow instead of manual playground spot-checks
- RAG pipelines that need citation/faithfulness verification beyond vector-retrieval scores
- Policy-heavy bots (refund windows, discount limits, escalation rules) where silent rule drift is the main risk
When not to use it
QAgent tests your agent through its endpoint before and between releases; it is not a production-traffic observability/monitoring tool, and it does not execute code or test coding agents in environments. Judge-based scoring still deserves human review of flagged cases even though the site markets deterministic scoring with exemption rules.
FAQ
Is QAgent free?
It has a free tier of 100 test-case evaluations per month with no credit card required (one connected agent endpoint). The paid Solo plan is ₹2,499/month (about $29) with 1,500 evaluations/month and up to 5 connected agents.
Does QAgent require an SDK or code changes?
No. You connect your agent by pasting its endpoint URL or webhook; the platform runs test cases against it via HTTP POST and returns scorecards.
What does QAgent actually test?
Eight dimensions: Answer Quality, Factual Groundedness (anti-hallucination vs ground truth), Policy Adherence, Escalation Correctness, RAG Faithfulness, Contextual Relevancy, Context Recall, and Multi-turn Context Memory, plus prompt-regression score deltas between versions.
How is QAgent different from unit testing?
Unit tests check deterministic code; QAgent evaluates the non-deterministic LLM agent's responses against ground-truth rules and rubrics, catching silent prompt regressions that throw no runtime errors.
How does QAgent avoid false positives from greetings or dynamic IDs?
It ships exemption rules: dynamic values like ticket IDs and session IDs, polite pleasantries, top-ranked retrieved chunks that already contain the answer, and retriever behavior during jailbreak tests are not penalized.
Share this launch
Embed this badge
<a href="https://orangebot.ai/product/qagent" target="_blank" rel="noopener noreferrer"> <img src="https://orangebot.ai/api/badge/qagent.svg" alt="Featured on OrangeBot" width="200" height="54" /> </a>