About the role

You will develop and run the experiments supporting our assessments of metacognition in people, AI systems and their interaction. This means building AI evaluations, fitting computational models and studying how people and AI make decisions together. Your work will examine how confidence relates to performance, how judgements change in response to evidence and challenge, and when people and models rely on advice.

You will report to our Head of Research and work closely with Stephen Fleming, our CSO. Working with our CTO and software engineers, you will develop well-tested research code and reproducible experiment pipelines that can be integrated into our assessment platform.

We are open to hiring research engineers with complementary strengths. You should bring depth in at least one of AI evaluation, computational modelling or human–AI interaction, and an interest in working across the others.

What you’ll do

Depending on your strengths and the research programme, your work will include:

• Design, implement and analyse evaluations of large language models, measuring confidence, calibration, reliance on advice and responses to evidence or challenge.

• Design and run online experiments with human participants, including studies of how AI advice affects confidence, judgement and decision quality.

• Implement and fit computational models of metacognition and belief updating, including hierarchical Bayesian approaches where appropriate.

• Use simulations, parameter-recovery tests and posterior predictive checks to evaluate models and the conclusions drawn from them.

• Build reproducible experiment and analysis pipelines, tracking task versions, model configurations, scoring methods and results.

• Investigate unexpected findings, distinguishing behavioural effects from sampling variability, task ambiguity, scoring errors and implementation bugs.

• Test the reliability and validity of human and AI profiles, including consistency across repeated assessments, sensitivity to meaningful change and prediction of decision outcomes.

• Communicate findings, uncertainty and limitations through technical reports, research papers and discussions with colleagues and partners.

You may be a good fit if you

• Have strong Python skills and write readable, tested research code that other people can run, review and extend.

• Have a solid foundation in probability, statistics and experimental design, and can reason carefully about noisy data and competing explanations.

• Have practical experience in at least one of LLM evaluation, computational modelling or experiments with human participants.

• Can turn an open-ended question into a concrete experiment, take it through implementation and analysis, and identify the next useful test when results are unclear.

• Use version control and reproducible workflows and are comfortable working in a shared codebase.

• Use AI tools fluently in your own work, with the judgement to check what they produce.

• Communicate technical ideas clearly and work well with colleagues from different disciplines.

Your background might be in computer science, cognitive science, psychology, statistics, physics or a related field. We welcome equivalent practical experience; a PhD is not required.

Strong candidates may also have

We are looking for these strengths across the team, rather than expecting every candidate to have them all:

• AI evaluation. Experience with model APIs, custom evaluation tasks, scoring methods and frameworks such as Inspect. Experience eliciting verbalised confidence or deriving confidence measures from token probabilities where available, and investigating prompt sensitivity, sampling variability and differences across model configurations.

• Computational modelling. Experience with hierarchical Bayesian models, parameter estimation, model comparison and diagnostic checks, using tools such as Stan, PyMC or NumPyro.

• Human–AI interaction. Experience designing controlled behavioural studies, implementing online tasks and managing participant recruitment, data quality and data protection. Familiarity with tools such as jsPsych and recruitment platforms such as Prolific.

• Measurement. Experience in signal detection theory, metacognition, judgement and decision-making or psychometrics, including measures such as meta-d′ and methods for assessing reliability and validity.

• Research delivery. Publications, open-source contributions or other substantial projects demonstrating your ability to produce useful, well-supported results.

The opportunity

You will take ownership of experiments and help shape the research agenda as part of our early team. Candidates with deeper experience may also mentor colleagues and lead larger investigations.

What we offer

• A competitive salary and meaningful equity through share options.

• Hybrid working from our office in Bloomsbury, central London, with 4 days a week together in person.

• 28 days’ holiday plus bank holidays, and a workplace pension.

• A learning budget and time for conferences.

• The chance to shape a new company and develop a new approach to measuring human and AI metacognition as part of the founding team.