AI Evaluation Engineer

Seniority: Mid-level (3-6 years)

Purpose: Build and execute FNZ's AI evaluation framework and tooling, working under the guidance of the AI Evaluations Team Lead to assess quality, safety, robustness, and operational suitability of AI solutions before release.

Key Responsibilities: 

• Build, maintain, and evolve core components of FNZ's AI evaluation framework, including test structures, scoring approaches, reusable evaluation patterns, and supporting documentation.

• Develop and improve evaluation tooling, harnesses, automation, and CI/CD integrations used to run repeatable assessments across AI agents and workflows.

• Execute evaluations across FNZ's six-pillar framework, including Task Performance, Safety & Compliance, Efficiency, Groundedness & Reasoning, Robustness, and Suitability. 

• Create and maintain test datasets, golden sets, rubrics, and scoring criteria that reflect expected agent behaviour and business requirements. 

• Build and run test suites covering baseline behaviour, edge cases, failure modes, and adversarial scenarios for AI agents and workflows.

• Assess multi-step agentic workflows, including planning, tool use, execution quality, recovery from errors, and adherence to controls 

• Verify groundedness and output quality by checking responses against source content, expected reasoning patterns, and policy constraints. 

• Document evaluation findings with clear evidence, communicate issues to AI solution teams, and support remediation and re-testing.

• Develop deeper expertise in one or two evaluation domains while remaining effective as a generalist across the wider framework.

Skills and Experience: 

• 3-6 years in software engineering, test engineering, AI/ML development, or data science.

• Strong programming skills with hands-on experience building test automation, evaluation tooling, or developer productivity tooling; Python or .NET background required, with .NET preferred

• Practical experience evaluating or building LLM applications, RAG systems, or AI agents 

• Understanding of prompt engineering, retrieval-augmented generation, agent architectures, and common failure modes in probabilistic systems 

• Ability to design structured test approaches, rubrics, and repeatable evaluation workflows for complex AI behaviours 

• Analytical mindset to decompose complex agent behaviours and identify weaknesses, edge cases, and opportunities to improve the framework 

• Strong documentation and communication skills, with the ability to explain findings and propose practical improvements to evaluation methods and tooling