Are you a QA Lead with expert test automation experience & AI model validation experience? If so, read on...
We are seeking someone to establish and integrate automated testing into our development process for an AI tool. This individual will be responsible for designing, implementing, and maintaining robust test automation frameworks, ensuring test coverage across the entire tooling lifecycle, and working closely with developers and product teams to identify and resolve issues.
What you will be doing:
-AI Model & LLM Validation
-Develop and execute post-training validation tests for AI models to assess accuracy, usability, and reliability.
-Work with SMEs to evaluate model outputs, identify hallucinations, and track inconsistencies.
-Define benchmarking criteria and measure model performance against ground truth data.
-Implement dataset validation strategies to ensure structured (numerical) and unstructured (text, images) data integrity.
-Test Automation & CI/CD Integration
-Design and maintain automated test frameworks for functional, regression, and performance testing.
-Integrate test automation into CI/CD pipelines (Jenkins, GitHub Actions, GitLab CI, etc.) to enable rapid iteration and deployment.
-Develop monitoring and reporting tools to track test results, performance metrics, and failure patterns.
-Optimize test environments to simulate real-world conditions for AI tool validation.
-User Acceptance Testing (UAT) & Feedback Loops
-Create and execute UAT plans to validate AI-powered Digital Twin outputs with real-world users.
-Work with SMEs and beta testers to refine AI models based on user feedback and failure cases.
-Track model failures, data gaps, and misclassifications to inform continuous model retraining.
-Establish clear success criteria for AI-generated responses and ensure alignment with business needs.
-Quality Assurance & Issue Resolution
-Lead test execution, track defects, and drive root cause analysis for AI system failures.
-Work closely with data engineers and developers to apply corrections and fine-tune models.
-Utilize API testing tools (Postman, RestAssured) and performance testing frameworks (JMeter, Locust) to validate system interactions.
-Implement chaos testing and adversarial testing to assess AI system resilience.
What you need:
-5+ yrs experience in QA, test automation, or AI model validation
-2+ yrs in a lead role
-Experience testing AI models, LLMs, or structured datasets in production.
-Strong proficiency in test automation frameworks (Selenium, Playwright, Cypress, PyTest, JUnit, etc.).
-Proficiency in Python, Java, or JavaScript for test automation and scripting.
-Experience with CI/CD pipelines and cloud-based testing (AWS, GCP, Azure).
-Strong understanding of API testing, performance testing, and observability tools.
-Ability to analyze logs, debug failures, and collaborate across teams to resolve issues efficiently.
Preferred Qualifications:
-Experience with retrieval-augmented generation (RAG) models and vector databases (Pinecone, FAISS, Weaviate).
-Familiarity with bias detection, hallucination testing, and reproducibility challenges in AI validation.
-Exposure to containerized testing environments (Docker, Kubernetes).
-Hands-on experience with data labeling tools (Labelbox, Prodigy, Snorkel).
For this position you must be currently authorized to work in the United States. We do not sponsor for this position.
Nitu Gulati-Pauly is recruiting for this position and the positions below.
Email me to apply for this position
