PolBench
methodology.

Research question

PolBench measures the completeness of the selected evidence about a specific organizational response. Governance quality and the effect of a response require separate measures.

The research question is whether organizations change their policies or practices after an adverse finding. The pilot records evidence for five checkpoints: Finding, Decision, Ownership, Implementation and Validation. These categories have not been validated as a scale, and they need not occur in that order.

NIST distinguishes governance roles, policy and oversight from operational response, and includes improvement in risk management. PolBench uses that distinction to classify organizational changes separately from incident containment. Its coding scheme has not been assessed or certified by NIST. [1] [2]

Unit and selection criteria

An episode comprises a named organization, an identifiable trigger, the specific response linked to it, and a set of cited sources. Different episodes within one organization would be separate records. They should not be treated as independent observations in a future inferential study without accounting for that grouping.

The collection contains 14 selected episodes: 13 security or privacy responses and one AI evaluation response. Each has an identifiable trigger and primary source. Cases were chosen partly because a response was visible; Cloudflare’s temporary resourcing decision provides a comparison with standing policy changes. Selection was neither exhaustive nor preregistered, so the sample cannot estimate how often organizations respond or which organizations perform better.

Technical patches, credential replacement, or model tuning alone do not establish governance change. They support the Implementation checkpoint only when the record ties them to the selected operating rule or institutional decision. Cross-cohort averages are not reported; one evaluation case cannot characterize that cohort.

Coding rules

At Decision, an organization’s announcement receives the Self-reported label, including an announced commitment. Documented requires directly available operative policy text, a legal instrument, or an external record. The label identifies the source of the claim.

A public commitment can establish the Decision checkpoint while Implementation remains Planned. A proposed settlement can establish a documented proposal without establishing an effective legal obligation. Evidence of one checkpoint never automatically fills another.

For each assessment, the record states what was tested, when it was tested, the method and result, and who reports it. Those details determine what the assessment can support. A safeguard test before deployment cannot establish whether the later governance process worked. An assessment with unfavorable results still qualifies for the Validation checkpoint.

Four evidence labels

• Documented: directly available operative policy text, a legal instrument, or an external record supports the narrow claim. A settlement supports the existence and terms of the settlement, not the truth of every allegation or later compliance.
• Self-reported: the organization states that the checkpoint occurred. An outside assessment described only in an organization-authored summary retains this label.
• Planned: the relevant assignment, operation, or test is promised or required but not evidenced as completed. An explicit chosen commitment is assessed separately at Decision.
• Not established: the cited sources do not show whether the checkpoint occurred. The organization may have acted without a public record.

The labels identify the evidence available for each checkpoint; they do not express a performance level or a probability. The record also identifies the source and the state of the response. A published plan may establish a decision while implementation remains unconfirmed.

Source requirements

Primary materials include incident reports, organizational updates, regulator publications, orders, and technical evaluations. Each checkpoint cites source IDs; source records retain a URL, date, origin, and relevant locator where available. Multiple publications from one organization do not become independent corroboration, and an external author is not automatically an unbiased source.

The search covers material available through 20 October 2026. For each existing case, searches combined the organization and incident or policy name with terms for later implementation, assessments, regulatory action and failures. Company publications, regulator records and available assessor reports were reviewed for follow-up evidence. This was an update to the selected cases, not a systematic search for all eligible incidents.

Search through records the review date. Latest source records the most recent dated source or stated update used in that case. These dates can differ by years. A report date may also precede public release; where release timing is unknown, it remains unspecified. Live pages may have changed since their stated date, and this review did not reconstruct all archived versions. Month-only dates retain that precision. Source dates are not used to estimate response speed.

A source must explicitly link the response to the incident or finding. Company explanations are attributed to the company; regulatory records support the connection within the terms of that proceeding. The review cannot isolate the effects of earlier initiatives, legal pressure or other events. Establishing causation would require a more complete account of competing explanations. [3]

Later evidence appears separately in each record, including adverse findings and policy revisions. Evidence about another incident or a broader program does not automatically change the original case’s labels. Where a search found no qualifying follow-up, the record states that result and its limits. Public disclosure varies across organizations, so an empty checkpoint cannot establish that no action occurred.

Score calculation

The score is 20 × the number of checkpoints labelled Documented or Self-reported. It ranges from 0 to 100 in steps of 20. Planned and Not established receive no points. Multiplying the count by 20 adds no statistical precision.

Documented and Self-reported receive equal points because the score describes the presence of supporting evidence. Their different provenance remains visible beside the score. No confidence weights have been estimated. A score of 100 means all five categories have evidence; a recorded assessment can still report failures or unfinished work.

There is no pass mark, calibrated performance band, or estimate of risk reduction. Optional score sorting orders how many evidence categories are populated. It cannot rank organizations by governance quality. Two identical scores can rest on different source types, response scopes, and observation periods. Finding and Decision are partly implied by pilot selection, so most included cases have a built-in score floor.

Disclosure practices and longer source periods can increase the score without better governance. An effectiveness measure would require separate outcome data and a justified measurement model. Composite-indicator guidance explains why indicator selection, missingness, weighting, and aggregation affect interpretation. [4]

Implementation describes the response in operation. Validation describes any assessment of that response, including what its result can support.

Validation status

Some cases include external assessments. For example, the redacted Protiviti assessment of Facebook’s privacy program describes specified governance structures and mixed findings. Such reports support individual case claims. They do not establish the reliability of PolBench’s coding or the validity of its score.

Validating PolBench requires the following work. This study design has not been performed or preregistered.

1. Define the construct and sampling frame. Recruit a governance, security, and measurement review panel to test whether the criteria cover the intended construct. Define eligible triggers and observation units before searching for responses. Retain triggers with no known response; report exclusions and retrieval failures.
2. Standardize follow-up. Prespecify a primary follow-up period, date origin, eligible source types, search procedure, and update policy. Record both event time and disclosure time. Handle immature follow-up separately; archive permitted source versions and record retrieval dates.
3. Test coding reliability. Use at least two independent coders applying a frozen codebook before adjudication. Preserve their original labels and disagreements. Report agreement per checkpoint and nominal Krippendorff's alpha with uncertainty; the labels are not an ordinal scale. For small or constant-label samples, report when coefficients are unstable or undefined. Agreement does not establish validity. [5]
4. Audit missing evidence. Separate implementation, explicit evidence of non-implementation, and unknown status. Report their counts against the full eligible denominator. A rate conditional on publicly observable cases does not identify the rate in all organizations. Any missingness assumptions or bounds must be stated and justified.
5. Validate against outcomes. Compare public annotations with dated operating records, independent assessments and tests of the specified response, obtaining access where needed. Measure implementation, effectiveness, and persistence separately. A before/after change alone cannot identify causality; a causal study needs a defensible comparison and explicit confounder assumptions.
6. Publish versioned releases. Publish the protocol, record-level corrections, source provenance, disagreement handling, and changes in sample composition. Test sensitivity to response definitions, follow-up, and evidence requirements before making comparisons.

Version 0.4

20 October 2026.

• Extended the source search for all 14 cases through 20 October 2026. Each record now separates the search date from its latest source date.
• Added later assessments, regulatory developments and policy changes in a dedicated follow-up section. Checkpoint labels change only where the new evidence addresses the response being studied.
• Revised the page, case descriptions and methodology for clearer wording. The score formula is unchanged.

Version 0.3

20 October 2026.

• Replaced the card grid with expandable response rows and a 0–100 evidence completeness score. The formula and source labels are visible within each row.
• Added the direct, redacted assessor report for Meta. The case now distinguishes documented implementation and assessment from the company’s account of named ownership.
• Corrected Cloudflare’s validation rationale to match its temporary resourcing decision, and narrowed Reddit’s finding to confirmed credential capture.
• Added Anthropic’s companion report to identify the pre-launch test procedure, model context and mixed result. Clarified how announcements and scoped ownership are coded.
• Preserved unknowns, assessment limitations, and source-date uncertainty. The scale remains provisional.

Version 0.2

The previous release separated decision state from implementation, added explicit response types and assessment scope, narrowed causal claims, and published the codebook. The dataset retains the case-level correction history.

References

1. NIST (2024). The NIST Cybersecurity Framework (CSF) 2.0. GOVERN categories GV.RR, GV.PO and GV.OV; improvement category ID.IM.
2. NIST (2025). Incident Response Recommendations and Considerations for Cybersecurity Risk Management, SP 800-61 Rev. 3. Incident response and continuous improvement within risk management.
3. Collier, D. (2011). Understanding Process Tracing. PS: Political Science & Politics, 44(4), 823–830.
4. OECD / European Union / Joint Research Centre (2008). Handbook on Constructing Composite Indicators: Methodology and User Guide. Indicator selection, missingness, aggregation, uncertainty, and sensitivity.
5. Krippendorff, K. Computing Krippendorff's Alpha-Reliability. Agreement among independent coding observations.

These sources informed the design. They do not provide validation or endorsement of PolBench. Syntony PolBench is distinct from the political Community Notes dataset of the same name described in GitSearch.