The benchmark
Items, held-out cases and metadata, in formats your tooling can read.
Syntony / Services / Benchmarking
Service
Benchmarking Benchmarks that still mean something next year
Public benchmarks saturate, leak into training data and drift away from the work people actually do. We build benchmarks for specific domains and risks, with held-out sets, validated graders and versioned releases, so a score still tells you something a year from now.
- Track record
- Built Intonation's voter-answer evaluation, with public questions, held-out cases and repair pairs.
- Who runs it
- Core team with domain experts who write and review the items.
- Format
- Private benchmarks for one client, or public benchmarks released with partners.
First-person walkthrough
Benchmark studio
Walk the site in first person and run a red team test at each station: inspect, disrupt, limit, retest. 4 stations, about five minutes.
- Item writing
- Held-out vault
- Scoring
- Releases
Illustrative environment
01 / What we test
What makes a benchmark trustworthy
A clear construct
A written definition of what the benchmark measures and what it doesn't.
Expert-written items
Questions and tasks written and reviewed by people who know the domain.
Held-out sets
Cases kept private so results can't be inflated by training on the test.
Validated graders
Automated graders checked against expert raters before anyone relies on them.
Contamination checks
Tests for whether a model has seen the items before.
Versioning
Releases with change logs, so scores from different versions aren't compared by mistake.
See an example campaign plan
A fictional plan for a domain benchmark released in two versions. Each engagement sets its own rows, phases and depth.
Example campaign planFictional plan for illustration
| Scoping | Item writing | Validation | Release | Version 2 | |
|---|---|---|---|---|---|
| Items | |||||
| Expert-written tasks | Light check | Primary focus | Covered | Not applicable | Covered |
| Held-out set | Not applicable | Covered | Covered | Light check | Primary focus |
| Scoring | |||||
| Grader validation | Not applicable | Light check | Primary focus | Light check | Covered |
| Contamination checks | Not applicable | Light check | Covered | Primary focus | Primary focus |
| Release | |||||
| Documentation and limits | Light check | Not applicable | Light check | Primary focus | Covered |
| Change log | Not applicable | Not applicable | Not applicable | Covered | Primary focus |
- Primary focus
- Covered
- Light check
- Not applicable
02 / What you keep
What stays with your team
Scoring code and graders
Everything needed to score a new model the same way.
Documentation
What it measures, how it was built and where it falls short.
Maintenance plan
When to refresh items and how to version releases.
Where we draw the line
Held-out items stay private. We agree up front who can see them and how results get reported.
Discuss your system
Tell us about the system and the decision ahead. We aim to reply within two business days.