Senior QA Automation Engineer (LLM/ML Quality)

About the role

Data Wow is hiring a Senior QA Automation Engineer in Pathum Wan, Bangkok, to build evaluation suites for LLM and ML features. The role defines quality in numbers accepted by clients, runs end-to-end retrieval tests, writes robust regression tests and keeps automated test suites fast and stable in CI.

Key responsibilities

• Build evaluation suites for LLM and ML features: curated test sets, scoring against reference answers, LLM-as-judge where justified, and human review otherwise.
• Define quality thresholds with the delivery team before launch and hold releases to them.
• Test retrieval pipelines end to end, covering chunking and indexing, retrieval quality, grounding of the final answer and handling of missing documents.
• Write regression tests that tolerate non-deterministic output, using semantic similarity, structured-output validation, tolerance bands and repeated sampling.
• Cover failure modes such as hallucination, prompt injection, data leakage, refusal and degradation on Thai-language input, while tracking cost and latency alongside accuracy.
• Detect drift after release by comparing production samples against the baseline.
• Design and maintain automated test suites for APIs, services, data pipelines and web apps, run them in CI and keep them fast and stable.
• Set the test strategy at the start of an engagement, including what is automated, exit criteria and evidence for the client.

Required profile

• 4-5 years in test automation, QA engineering or software engineering with substantial testing ownership.
• Hands-on experience testing an LLM or ML-based product in production or close to it.
• Clear written and spoken communication in Thai and English.

Required skills

• Strong programming in Python, or TypeScript or Java with willingness to work primarily in Python.
• At least one automation framework such as Pytest, Playwright, Cypress or RestAssured, with CI experience.
• Working knowledge of API and data testing, including SQL and validating data through a pipeline.

What we offer

• Hybrid working with flexible working hours.
• Your choice of Mac or Windows equipment.
• Group health insurance and monthly Grab transportation credits.
• Job-related training courses and certificate fee coverage.
• 10+ days of annual leave and free snacks and drinks at the office.