Evaluation suite
Tasks, graders and scoring code you can rerun yourself.
Syntony / Services / Evaluations
Service
Evaluations Tests that hold up when someone checks them
An evaluation is only useful if people can trust the number and know what it doesn't cover. We design and run evaluations of models, agents and AI products, check the graders against expert judgment, and report results with the context needed to use them.
- Track record
- Designed and ran an evaluation for a public sector institution in the defense sector. Co-authors of Evaluation Cards, a NeurIPS 2026 spotlight paper on how evaluation results are reported.
- Who runs it
- Led by our Head of Research Strategy and Evaluations, with domain specialists for each evaluation.
- Format
- Custom evaluations, independent reruns of existing ones, and reviews of evaluation reports.
First-person walkthrough
Evaluation lab
Walk the site in first person and run a red team test at each station: inspect, disrupt, limit, retest. 4 stations, about five minutes.
- Task design
- Run cluster
- Grading
- Report
Illustrative environment
01 / What we test
What we evaluate
Capabilities
What a model or agent can do in a domain, measured on tasks that resemble real use.
Safety and misuse
Whether safeguards hold, including red team findings turned into repeatable tests.
Agent tasks
End-to-end tasks with real tools, scored on the state that changed rather than the transcript.
Grader validation
Checks that LLM judges and rubrics agree with expert human raters.
Independent reruns
Rerunning a vendor's or lab's evaluation to see if the reported results hold.
Reporting
Results written up with what was tested, how it was scored and what it leaves out, following the Evaluation Cards approach.
See an example campaign plan
A fictional plan for an enterprise choosing between two agent vendors. Each engagement sets its own rows, phases and depth.
Example campaign planFictional plan for illustration
| Design | Pilot run | Full run | Next vendor release | |
|---|---|---|---|---|
| Questions | ||||
| Task success on real workflows | Covered | Primary focus | Primary focus | Covered |
| Safety and policy compliance | Covered | Covered | Primary focus | Primary focus |
| Method | ||||
| Grader agreement with experts | Light check | Primary focus | Covered | Light check |
| Contamination checks | Covered | Light check | Covered | Covered |
| Reporting | ||||
| Vendor claims rerun | Not applicable | Covered | Primary focus | Primary focus |
| Limits and caveats | Light check | Light check | Primary focus | Covered |
- Primary focus
- Covered
- Light check
- Not applicable
02 / What you keep
What stays with your team
Results with context
Scores reported next to what was tested and what wasn't.
Grader validation
How closely automated graders agree with expert raters.
Rerun plan
When to run it again, and what changes should trigger a rerun.
Where we draw the line
We report what the evidence supports. If a result is weak, uncertain or depends on choices we made, the report says so.
Discuss your system
Tell us about the system and the decision ahead. We aim to reply within two business days.