QA Engineer

Newcode.ai · Oslo, Oslo, Norway · Posted Sep 1, 2026


Transparency

Dead

Online for 11 weeks and never reposted. Jobs this old or reposted this often rarely lead to a hire. It may be filled already.

First seen
Oct 6, 2026
Offered for
80 days

Overview

Category
Software engineering
Employment type
Permanent
Workplace type
On-site
Country
Norway
Seniority
Mid level
Schedule
Full time

Key skills

Evaluation Framework Design
Create automated and human-in-the-loop frameworks to assess AI product quality.
Metrics and Benchmark Development
Define and implement accuracy, reliability, consistency metrics and benchmarks for AI systems.
Dataset and Test Suite Construction
Build evaluation datasets, test cases, and regression suites for complex legal use cases.
Integration with Development Workflows
Embed evaluation pipelines directly into software engineering and product processes.
Automated Testing and Monitoring
Implement continuous tests and monitoring to catch quality regressions before release.
Python for AI Evaluation
Write Python code to automate evaluation, analysis, and tooling for AI models.

Key phrases

  • automated and human-in-the-loop evaluation frameworks
  • AI-powered products
  • metrics and benchmarks
  • accuracy, reliability, consistency
  • evaluation datasets
  • test cases
  • regression suites
  • legal workflows
  • model outputs
  • failure modes
  • quality regressions
  • LLMs
  • generative AI
  • retrieval systems
  • AI evaluation methodologies

From the posting

About Newcode.ai

Newcode.ai is a fast-growing legal tech and agentic AI company transforming how legal work is done. With teams across Norway, Sweden, Ireland, the US and growing we work at the intersection of law, technology, and intelligence. We move fast, think big, and take pride in doing things the right way.

Note: We believe in being transparent about what it's like to work at Newcode. As a fast-growing startup, we're building and evolving every day. That means not every process, playbook, or framework is already in place, and priorities can shift quickly.

The people who thrive here are comfortable with ambiguity, take ownership, and don't wait for perfect direction. They are resourceful, proactive, and able to "figure it out"—solving problems, creating structure where needed, and helping build the company as they go. If you are good with this then, great! Keep reading to learn more.

The Role

As a QA Engineer, you will own the systems, processes, and technical frameworks we use to evaluate the quality of Newcode's AI products.

You will work closely with Software Engineering and Legal Engineering to understand how our AI performs in real-world legal workflows, identify where it falls short, build better evaluation methodologies, and turn those insights into measurable improvements.

This is not traditional QA. You will be working at the intersection of AI evaluation, software engineering, legal workflows, and product quality.

Key Responsibilities:

  • Design and build automated and human-in-the-loop evaluation frameworks for AI-powered products and workflows.
  • Develop metrics and benchmarks to measure accuracy, reliability, consistency, and overall AI quality.
  • Build evaluation datasets, test cases, and regression suites for complex legal use cases.
  • Analyze model outputs to identify failure modes, patterns, and opportunities for improvement.
  • Work closely with Legal Engineers to translate real-world legal workflows into rigorous evaluation criteria.
  • Partner with Software Engineers to integrate evaluations directly into product and development workflows.
  • Establish automated testing and monitoring to identify quality regressions before they reach customers.
  • Evaluate changes to models, prompts, retrieval systems, agents, and workflows.
  • Develop tooling that makes AI quality measurable and visible across the organization.
  • Investigate unexpected model behavior and determine root causes.
  • Help establish standards for what "good" looks like across different legal use cases.

Requirements

  • Strong software engineering or technical background, with experience working on AI/ML systems.
  • Experience building testing, evaluation, or quality frameworks for AI-powered products.
  • Strong Python and/or similar programming experience.
  • Understanding of LLMs, generative AI, retrieval systems, agents, and/or AI evaluation methodologies.
  • Strong analytical and problem-solving skills.
  • Ability to work with ambiguous problems and turn them into measurable technical frameworks.
  • Strong attention to detail and a high bar for quality.
  • Ability to communicate technical findings clearly to both technical and non-technical stakeholders.
  • Comfort working cross-functionally with Engineering, Product, and Legal Engineering.

Benefits

Why You’ll Love Working with Us

  • Culture of trust and excellence: Be part of a team that values reliability, respect, and initiative, where everyone’s contribution matters and diversity drives creativity.
  • Collaborative environment: Work with talented, driven peers on bleeding-edge legal engineering solutions with real-world impact.
  • Growth and ownership: Shape your role as we grow. Take ownership and drive outcomes from the start of your journey.
  • Inclusive culture of excellence and trust: We reward initiative, respect, integrity, and teamwork, and are committed to building a global, inclusive team. Your drive, curiosity, and ideas matter more than your background


Ready to shape what’s next? Apply today and join us in building a smarter, more efficient world powered by AI.

More jobs at Newcode.ai

Senior RAG Engineer

Newcode.ai · Last week | Chicago, Illinois, United States | Remote · Senior · Full time

Create CV

Forward Deployed Engineer

Newcode.ai · 2 weeks ago | Chicago, Illinois, United States | Hybrid · Associate · Full time

Create CV

Head of Forward Deployed Engineering

Newcode.ai · 2 weeks ago | Chicago, Illinois, United States | Hybrid · Head of · Full time

Create CV

Legal Engineer

Newcode.ai · 3 weeks ago | Dublin, County Dublin, Ireland | On-site · Mid level · Full time

Create CV

Senior Growth Marketing Manager

Newcode.ai · Last month | United States | Remote · Senior · Full time

Create CV
See all 14 jobs at Newcode.ai