Syntony / Services / Red teaming / ML security

Red teaming · Security

ML security and robustness Attacks on the model itself

Some attacks never touch a prompt. They go after training data, model weights or the statistical behavior of the model. We run model-level attacks against frontier and production systems, then assess which defenses would actually work for yours.

Discuss your system Enter the 3D environment

Status: Available now. Engagements are scoped to your system and run under NDA.
Who runs it: Core team plus Trusted Red Team Network specialists who work on adversarial machine learning in research and industry.
Format: White-box or black-box, depending on the access you can give us.

First-person walkthrough

Model training hall

Walk the site in first person and run a red team test at each station: inspect, disrupt, limit, retest. 4 stations, about five minutes.

Training data
Training cluster
Model registry
Serving endpoint

Enter the environmentKeyboard and mouse · Touch · VR headsets

Illustrative environment

01 / What we test

Model-level threats we test

Data poisoning
Planted examples in training or fine-tuning data that change behavior on purpose.
Backdoors
Hidden triggers that make a model behave differently on specific inputs.
Evasion and adversarial examples
Inputs crafted to be misread, including in vision and audio models.
Model extraction
Rebuilding a model's behavior through repeated queries.
Privacy attacks
Membership inference and training data extraction that reveal what the model learned from.
Robustness under shift
How performance holds up when inputs drift away from what the model was trained on.

See an example campaign plan

A fictional plan for a fine-tuned classification model served through an API. Each engagement sets its own rows, phases and depth.

Example campaign planFictional plan for illustration

Example campaign plan: attack families by phase
Training time
	Data audit	Before fine-tuning	Before deployment	After retraining
Data poisoning	Primary focus	Covered	Light check	Primary focus
Backdoor triggers	Light check	Primary focus	Covered	Primary focus
Inference time
Evasion inputs	Not applicable	Light check	Primary focus	Covered
Model extraction	Not applicable	Not applicable	Primary focus	Light check
Membership inference	Light check	Light check	Covered	Covered
Defenses
Input filters and detectors	Not applicable	Light check	Primary focus	Covered
Robustness under drift	Light check	Light check	Covered	Primary focus

Primary focus
Covered
Light check
Not applicable

02 / In depth

How these failures work, and how we test for them

Public research, standard frameworks and a simplified look at our method. Working tools, attack libraries and client data stay private.

How it fails

Classic machine learning attacks go after the model itself: the data it learned from, its weights and its decision boundary. They apply to vision, audio, tabular and language models, and most were shown in research years before the systems they now threaten went into production.

Defense is usually a question of cost. How many poisoned samples, queries or changed pixels does an attack need, and does a defense stop it or only raise the price? We report that number next to what the defense costs in accuracy on clean data.

EntryTraining data, a downloaded checkpoint or the prediction API.
MechanismSmall, targeted changes that the model learns or misreads.
ImpactWrong decisions on chosen inputs, or private data and weights leaving.

Taxonomy

Failure classes we test, mapped to public frameworks so findings slot into the reports you already file. ATLAS IDs can be searched on the MITRE ATLAS site.

Taxonomy of failure classes with framework mappings and controls
Ref	Class	What it looks like	Frameworks	Control that belongs there
ML-1	Data poisoning	A few hundred planted samples change the label for a chosen trigger.	OWASP LLM04 ATLAS AML.T0020	Data provenance, deduplication and review of suspicious clusters.
ML-2	Backdoors	A model behaves normally until a specific trigger appears.	ATLAS AML.T0043.004 ATLAS AML.T0018.000	Trigger scanning, pruning and retraining from trusted data.
ML-3	Poisoned artifacts	A downloaded checkpoint runs code when it is loaded.	OWASP LLM03 ATLAS AML.T0058	Safe serialization formats, signed artifacts, isolated loading.
ML-4	Evasion	Inputs changed in ways a person wouldn't notice, misread with high confidence.	ATLAS AML.T0015 ATLAS AML.T0043	Adversarial training, input preprocessing, ensemble disagreement checks.
ML-5	Model extraction	Query access used to train a working copy.	ATLAS AML.T0024.002	Query budgets, coarser outputs and watermarking.
ML-6	Membership inference	Confidence scores reveal whether a record was in the training set.	OWASP LLM02 ATLAS AML.T0024.000	Differential privacy, deduplication, fewer returned scores.
ML-7	Defense evaluation	A detector passes static tests and fails once the attacker knows it exists.	NIST AI 100-2 E2025	Evaluate against attacks designed with knowledge of the defense.

Test case

poison_dose_response.pypython

# Simplified dose-response test. Trigger design and datasets stay private.RATES = [0.0, 0.0005, 0.001, 0.005, 0.01]   # share of the fine-tuning set poisoned rows = []for rate in RATES:    model = finetune(base_model, clean_data + poison(sealed_trigger, rate, seed=7))    rows.append({        "rate": rate,        "attack_success": trigger_success(model, holdout_with_trigger),        "clean_accuracy": accuracy(model, holdout_clean),   # a defense's cost shows here    }) working = [r["rate"] for r in rows if r["attack_success"] >= 0.9]print("smallest working rate:", min(working) if working else "none tested")

What to notice

The output is a curve, not a yes or no. It tells you how much access an attacker needs.
We run the same curve with your defense switched on, so its cost in clean accuracy is on the record.

Simplified for publication. Payload libraries, harness code and scoring rubrics stay private and are shared with clients under NDA.

Sample finding

ML-EX-031HighRetested · fixed

Training jobs load pickled checkpoints from an external mirror

A fine-tuning pipeline pulled a base checkpoint from an unpinned external mirror and loaded it with a format that can execute code. Our benign test artifact ran and called back from inside the training environment.

Reproduction: Publish test artifact ███ to the mirror the pipeline trusts.
Trigger the scheduled fine-tuning job.
Observe the callback from the training network.
Impact: Anyone able to publish to the mirror could run code next to training data and credentials.
Mapped to: OWASP LLM03 ATLAS AML.T0058 ATLAS AML.T0010
Control: Weights-only formats, hash-pinned artifacts and loading inside a network-isolated job.
Retest: The test artifact was rejected at hash verification. A pinned clean checkpoint trained normally.

Sources

Public research and guidance behind this page. Summaries are ours.

Explaining and Harnessing Adversarial ExamplesGoodfellow et al. · 2014 · arXiv
Foundational paper on small, deliberate input changes that make models wrong with high confidence.
BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply ChainGu et al. · 2017 · arXiv
A backdoored network performs normally on validation data and misbehaves only on attacker-chosen triggers.
Poisoning Web-Scale Training Datasets is PracticalCarlini et al. · 2023 · arXiv
Showed that 0.01% of two large public image datasets could have been poisoned for about $60 by buying expired domains.
Stealing Machine Learning Models via Prediction APIsTramèr et al. · 2016 · arXiv
Model extraction through a public prediction API, with no access to parameters or training data.
Membership Inference Attacks against Machine Learning ModelsShokri et al. · 2016 · arXiv
Showed that black-box access can reveal whether a specific record was in a model's training set.
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety TrainingHubinger et al. · 2024 · arXiv
Backdoored language models kept their hidden behavior through standard safety training, so clean evaluations alone can miss them.
Examining Malicious Hugging Face ML Models with Silent BackdoorJFrog Security Research · 2024
Analysis of models published on a public hub that executed code when loaded, the supply chain risk behind our artifact checks.

03 / What you keep

What stays with your team

Attack results

Each attack with its method, success rate and the access it needed.

Defense comparison

Candidate defenses measured against the same attacks.

Data recommendations

Changes to data collection and checks that lower poisoning risk.

Retest plan

Which attacks to rerun, and when.

Where we draw the line

Extracted data, weights and attack artifacts stay inside the agreed environment and are deleted at the end of the engagement unless you ask us to hand them over.

All red teaming areas

Plan a red team engagement

Tell us about the system and the decision ahead. We aim to reply within two business days.

Discuss your system Book a call