Syntony / Services / Red teaming / Policies and strategic risk

Red teaming · Governance

Policies, protocols and strategic risk Test the rules as well as the model

A safety policy is a set of promises about what the organization will test, when, and what happens if a threshold is crossed. We test those promises against real findings and check that each rule leads to someone with the authority to act on it.

Discuss your system Enter the 3D environment

Track record: Advised three major multinational technology organizations on safety policies, including writing original model- and feature-level governance and testing protocols.
Who runs it: Core team, with policy and technical specialists working side by side.
Format: Document review, interviews, tabletop exercises and live test results.

First-person walkthrough

Release governance floor

Walk the site in first person and run a red team test at each station: inspect, disrupt, limit, retest. 4 stations, about five minutes.

Leadership
Release review
Evaluation lab
Records

Enter the environmentKeyboard and mouse · Touch · VR headsets

Illustrative environment

01 / What we test

Where policies break

Coverage
Risks the policy names but no test covers, and tests that don't map to any commitment.
Thresholds
Whether capability or risk thresholds are defined clearly enough to tell when one has been crossed.
Decision rights
Who can pause a release, approve an exception or escalate, and whether they know it.
Feature-level rules
Protocols for individual features, which often slip past model-level policies.
Change management
Whether model, prompt and tool updates trigger the retesting the policy promises.
Pressure tests
Tabletop exercises that put the policy under a realistic deadline or incident.

See an example campaign plan

A fictional plan for a company with a frontier safety policy and three AI products. Each engagement sets its own rows, phases and depth.

Example campaign planFictional plan for illustration

Example campaign plan: attack families by phase
The policy
	Document review	Interviews	Tabletop exercise	Next release
Risk coverage	Primary focus	Covered	Light check	Covered
Threshold definitions	Primary focus	Covered	Covered	Primary focus
The people
Decision rights	Light check	Primary focus	Primary focus	Covered
Escalation paths	Light check	Primary focus	Primary focus	Light check
The practice
Feature protocols	Covered	Covered	Light check	Primary focus
Retesting after change	Light check	Covered	Covered	Primary focus

Primary focus
Covered
Light check
Not applicable

02 / In depth

How these failures work, and how we test for them

Public research, standard frameworks and a simplified look at our method. Working tools, attack libraries and client data stay private.

How it fails

A safety policy is a set of promises: these risks will be tested, these thresholds will trigger these actions, these people can stop a release. Red teaming a policy means checking each promise against what the organization actually does, especially under deadline.

Frontier safety frameworks, the EU's General-Purpose AI Code of Practice and the NIST AI Risk Management Framework all rely on evaluations that feed decisions. The common failure sits between the two: a threshold nobody can measure, a test that informs no decision, or a feature launch that never meets the policy at all.

EntryA release, a feature or an incident under time pressure.
MechanismThe promise and the process don't connect.
ImpactA risk ships that the policy said would be caught.

Taxonomy

Failure classes we test, mapped to public frameworks so findings slot into the reports you already file. ATLAS IDs can be searched on the MITRE ATLAS site.

Taxonomy of failure classes with framework mappings and controls
Ref	Class	What it looks like	Frameworks	Control that belongs there
PS-1	Untested commitments	The policy names a risk with no evaluation behind it.	NIST AI RMF: Measure	A commitment-to-test map with an owner for each line.
PS-2	Unmeasurable thresholds	A capability threshold written in words no test can resolve.	Anthropic RSP	Operational definitions with the evidence that counts as pass or fail.
PS-3	Unclear decision rights	Nobody is sure who can pause a launch on a Friday night.	NIST AI RMF: Govern	Named roles, deputies and a rehearsed escalation path.
PS-4	Feature-level gaps	A new tool or memory feature ships under the old model review.	EU GPAI Code of Practice	Change triggers defined for features as well as models.
PS-5	Retest drift	Prompt and tool changes ship without the promised retest.	NIST AI RMF: Manage	Retests triggered automatically from release tooling.
PS-6	Pressure failure	Under competitive pressure, exceptions become the norm.	International AI Safety Report 2026	Tabletop exercises and an exception log the board reviews.

Test case

commitment_register.yamlyaml

# Simplified commitment register. Client policies and results stay private.- commitment: "Each major release is evaluated for autonomous replication"  source: frontier safety policy, section 4.2  evaluation: none mapped   # gap  decision_owner: unassigned   # gap  status: RED - commitment: "Releases pause if cyber evaluations exceed threshold T2"  source: frontier safety policy, section 5.1  evaluation: cyber-suite v7 (internal)  threshold: "T2 = ███% success on held-out ranges"  # measurable  decision_owner: head of safety, with a named deputy  last_exercised: tabletop, 2026-08  status: GREEN

What to notice

Every promise gets one line: the test behind it, the person who decides and when it was last exercised.
Red lines are where the policy says something the organization can't yet show.

Simplified for publication. Payload libraries, harness code and scoring rubrics stay private and are shared with clients under NDA.

Sample finding

PS-EX-004MediumRetested · fixed

Pause authority exists on paper but has never been used

In a tabletop exercise with a simulated threshold breach two days before launch, the named deputy did not know they held pause authority, and the release system had no hold state to set.

Reproduction: Run tabletop scenario ███ with release and safety leads.
Inject the threshold breach at T-48 hours.
Record who acts, how and how long it takes.
Impact: A real breach late in a release cycle could have shipped while people worked out who could stop it.
Mapped to: NIST AI RMF: Govern EU GPAI Code of Practice
Control: The release system now has a hold flag owned by safety, and the pause drill runs every quarter.
Retest: Rerun of the exercise: hold set within 40 minutes by the deputy.

Sources

Public research and guidance behind this page. Summaries are ours.

AI Risk Management FrameworkNIST · 2023
Organizes AI risk work into Govern, Map, Measure and Manage. Our policy reviews trace each commitment to these functions.
Model evaluation for extreme risksShevlane et al. · 2023 · arXiv
Explains why dangerous capability and alignment evaluations need to feed real decisions about training and deployment.
Anthropic's Responsible Scaling PolicyAnthropic
A public example of capability thresholds tied to required safeguards, the kind of commitment we test for clarity and follow-through.
The General-Purpose AI Code of PracticeEuropean Commission
The code for general-purpose model providers under the EU AI Act, including safety and security commitments for systemic-risk models.
International AI Safety Report 2026Bengio et al. · February 2026
The second edition of the international scientific review of general-purpose AI capabilities, risks and risk management.
Frontier Models are Capable of In-context SchemingMeinke et al. · 2024 · arXiv
Several frontier models pursued a given goal covertly in test environments, evidence that policies need evaluations able to catch it.

03 / What you keep

What stays with your team

Commitment map

Every commitment traced to a test, a threshold and an owner, with gaps marked.

Revised protocols

Draft model-level and feature-level testing protocols where you need them.

Exercise readout

What happened in the tabletop and what to change.

Release checklist

A short list to run before each launch.

Where we draw the line

We assess and advise. Decisions about your policy stay with your leadership, and our reports say clearly where the evidence is thin.

All red teaming areas

Plan a red team engagement

Tell us about the system and the decision ahead. We aim to reply within two business days.

Discuss your system Book a call