AI Evaluation & Agent Systems Engineer
Build evaluation software for AI models and agents. The work involves designing tests, recording evidence, and checking whether a mitigation changes the behavior under investigation. Evaluation outputs will support Syntony’s research and product workflows.
- Employment type
- Paid contract
- Focus
- Evaluation systems
- Duration and schedule
- Agreed for each project
Responsibilities
- Develop evaluation harnesses for model behavior, tool use, and multi-step agent tasks.
- Record inputs, configurations, tool calls, observations, and limitations in structured traces.
- Build replay and regression tests for comparing system behavior before and after changes.
- Connect evaluation outputs to Resonance attack-path records and GovTune AI review materials.
- Document scenarios, run instructions, results, and the limits of each evaluation.
Qualifications
- Experience developing evaluation software, model integrations, or agent systems.
- Understanding of experiment design, nondeterminism, and measurement limitations.
- Ability to debug multi-step behavior and trace failures across system boundaries.
- Careful distinction between observed results and conclusions that the evidence supports.
- Clear technical writing and experience producing reproducible work.
We assess relevant experience and demonstrated work. Graduate enrollment or a graduate degree is not required.
Compensation and scope
This is a paid contract role. Deliverables, milestones, rates, payment terms, expected availability, and acceptance criteria will be agreed in writing before work begins.
The agreement will also cover confidentiality, intellectual property, access requirements, and handover. This professional role is separate from Syntony’s fellowships and undergraduate internship.
Apply for this role
Tell us about your experience and share links to your work. You can also apply by emailing hello@syntonyresearch.ai with “AI Evaluation & Agent Systems Engineer” in the subject line.
Fields marked with an asterisk are required.