Software Engineer III

Sits within FC Quality Verification Engineering, a team whose mission is to help teams identify quality issues earlier by building automation frameworks and developer tools that combine software development with applied AI/ML to improve how EA SPORTS FC is tested. The postholder will build automated testing frameworks and evaluation tools, applying AI and machine learning to expand test coverage. They will report to an Engineering Manager.

• Build automated testing frameworks, validation checks, and CI/CD pipeline integrations with reproducible diagnostics.
• Develop methods that help explore complex software behaviour, identify coverage gaps, and detect defects.
• Build automated checks to validate software behaviour and data against requirements.
• Develop evaluation frameworks for generative AI and LLMs-based systems (e.g., implementing LLMs-as-a-judge patterns, rubric-based scoring, and automated calibration) alongside deterministic validation checks.
• Apply generative AI, NLP, and machine learning, balancing heuristic, deterministic, and model-based approaches to solve complex software testing problems.
• Integrate testing tools into development pipelines, producing reproducible diagnostics, and support the software delivered.
• Prototype latest machine learning and LLMs capabilities, and collaborate with engineering, design, and quality partners to clarify requirements, review code, and resolve blockers.

• 5+ years of professional software engineering experience, including 2+ years of hands-on experience building, integrating, or evaluating AI/ML and LLMs-based systems.
• A Master of Science in Computer Science, Mathematics or a related field, or equivalent professional engineering experience.
• In-depth experience with LLMs, generative AI, and NLP concepts, architectures, and failure modes.
• Experience developing evaluation frameworks for generative systems, including dataset curation, model-as-a-judge design, validation criteria, error analysis, and quantitative/qualitative metrics.
• Proficiency in Python, algorithms, data structures, modular design, automated testing, debugging, and version control.
• Experience preparing datasets, defining validation criteria, and performing error analysis using quantitative ML metrics.
• Experience with natural language processing or large language models, such as prompting, fine-tuning, or retrieval-augmented generation.

• Hybrid role requiring at least 3 days a week in the Burnaby, BC office.

• Vacation of 3 weeks per year to start.
• 10 days per year of sick time.
• Paid top-up to EI/QPIP benefits up to 100% of base salary when welcoming a new child (12 weeks for maternity, and 4 weeks for parental/adoption leave).
• Extended health, dental, and vision coverage.
• Life insurance and disability insurance.
• Retirement plan for regular full-time employees.
• Certain roles may also be eligible for bonus and other incentive programs.