Key Responsibilities

Design, execute, and maintain comprehensive test plans for Sierra AI conversational agents, workflows, routing logic, and knowledge base configurations.

Perform functional, regression, integration, and end-to-end testing of Sierra AI implementations.

Validate AI-generated responses, prompts, tone, business rules, and LLM behavior across multiple customer scenarios.

Test intent recognition, conversation flows, fallback handling, escalation paths, and virtual agent performance.

Validate integrations with CRM, ticketing systems, knowledge management platforms, and other enterprise applications.

Create, execute, and maintain manual and automated test cases where applicable.

Perform REST API testing using Postman, REST Assured, or similar tools.

Log, track, and manage defects through Jira, Azure DevOps, or equivalent platforms.

Support User Acceptance Testing (UAT) and document testing outcomes and signoffs.

Prepare QA dashboards, defect reports, test summaries, and quality metrics aligned with PMO governance.

Identify quality risks arising from AI model updates, prompt changes, or platform configuration changes.

Mandatory Requirements

Hands-on Sierra AI Platform experience is mandatory.

Minimum 5+ years of overall Quality Assurance/Testing experience.

Minimum 1+ years of project experience working directly on Sierra AI implementations.

Experience testing conversational AI, chatbots, virtual agents, or AI-powered customer service platforms.

Strong understanding of testing AI/ML applications, including non-deterministic response validation.

Experience with REST API testing using Postman, REST Assured, or similar tools.

Experience with Jira, Azure DevOps, or other defect management tools.

Strong analytical, troubleshooting, and documentation skills.

Preferred Qualifications

Experience with Salesforce Einstein, Genesys AI, ServiceNow AI, or similar conversational AI platforms.

Understanding of LLM evaluation techniques, including hallucination detection, response relevance, confidence scoring, and prompt validation.

Experience in Healthcare, Insurance, Banking, Financial Services, or BPO domains.

Knowledge of Agile/Scrum delivery methodology.

Python or JavaScript scripting skills for QA automation.

Soft Skills

Excellent verbal and written communication skills.

Strong stakeholder management and cross-functional collaboration abilities.

Ability to work independently with minimal supervision.

Comfortable working within structured PMO governance.

Highly detail-oriented with strong problem-solving capabilities.

Ability to thrive in AI-driven environments with evolving requirements.