The Counter Certified Nothing
Incident status: resolved. Instrument status: replaced.
Authored validate_detection_content.py under scripts/verify/ to fill the gap left by the existing PowerShell counter. The validator parses every YAML file in the Sigma tree, extracts the id field, and checks shape, UUIDv4 version bit, UUIDv4 variant bit, and uniqueness across the library. First run against 103 Sigma rules produced 36 duplicate-ID errors spanning 69 files, grouped into 33 distinct collision sets, three of which were three-way collisions. Inspection showed the colliding IDs were not real UUIDs at all but a sequential placeholder pattern — 1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d, 2b3c4d5e-6f7a-8b9c-0d1e-2f3a4b5c6d7e, marching through the alphabet — that matched the shape regex but failed the version-bit check. The same placeholders repeated across tactic directories because the library had been batch-authored. Authored validation_exceptions.yml as temporary scaffolding: the allowlist downgrades a duplicate-ID error to a warning only when the UUID and the involved file set match exactly; a new file joining a collision group or an allowlisted file being renamed without an allowlist update produces a hard fail. The allowlist was generated programmatically from the live tree rather than hand-typed, so it could not be wrong about scope. Remediation followed as a single mechanically-verified pass rather than the originally planned four-PR tactic-by-tactic split; the split was abandoned because the collisions were not cohort-aligned — nearly every collision group spanned two or more tactic directories. The remediation script, run inline and not committed, generated 101 fresh UUIDv4s (the 69 allowlisted files plus 32 additional non-colliding placeholders that passed shape regex but had not yet collided), verified uniqueness among the new set, verified uniqueness against every existing id in the tree, regex-matched each file's id: line exactly once, edited the line in place, re-parsed the YAML to confirm structural integrity, and aborted on any per-file failure. Total work product: 101 files updated, 1 allowlist file emptied in the same commit, 0 errors and 0 warnings on both local and CI validator runs, 103 Sigma file count unchanged. The validator now runs in fully strict mode on the Sigma lane — no allowlist, no exceptions, no loopholes.
Two integrity properties coexisted in this system and were being conflated: the property "the set of files in the library has not changed" and the property "each file in the library is internally correct." The existing PowerShell counter at scripts/verify/verify-counts.ps1 enforced the first. Nothing enforced the second. The library was passing CI because CI was asking a question the library could answer and not asking the question the library was silently failing. This is not a defect of the counter — the counter did exactly what it was designed to do. It is a defect of the assumption that one gate was sufficient. Integrity has multiple independent dimensions, and each one requires its own gate; a count check and a content check are not substitutes, and neither is a substitute for a behavioral check that the rule actually fires. The architectural correction is not to replace the counter but to compose it with the new validator and, in time, with a behavioral injection harness that confirms a known-benign event produces the expected alert. Until that third gate exists, the library is configuration-correct and partially behavior-unverified, which is a narrower claim than the one the system was previously making.
Four claims submitted. Dispositions on the record.
4, fourth with [89ab]). Allowlist emptied same commit. Zero errors, zero warnings, local and CI.The useful observation this week is that the most valuable run a new validator will ever make is usually its first. Validators authored into a clean tree find nothing; validators authored into a tree that had previously been certified by a weaker instrument find the backlog of defects the weaker instrument could not see. The builder's first instinct on seeing 36 errors across 69 files was not defensive — she did not soften the finding, rename it, or redescribe the prior state as "working." She treated the output the way she used to treat a statistical process-control chart flagging out-of-spec parts on a line she had been told was in control: the chart is not the problem; the chart is the first honest measurement. The deeper pattern worth naming is that a CI pipeline which never rejects anything is not enforcing anything; it is decorating. The week's correction was to stop decorating and start enforcing, and the cost of that correction was 101 files of mechanical tedium — the correct cost, paid in the correct currency.
Consensus
The Sigma lane is strictly gated. 103 rules carry unique, version-4 UUIDs. The allowlist is empty. The validator runs in the CI path with no exceptions. The counter remains; it now has company.
Point of Contention
The plan called for four tactic-aligned PRs on the theory that boring-manual review per file would catch typos. The execution was one PR with a mechanically-verified script on the theory that a 69-file manual diff was a review nobody would actually perform. Architecture and Build sided with the script. The plan, as written, disagrees with itself on the record.
Open Question
What other count-shaped gates in this pipeline are certifying past content they cannot see, and what is the soonest a behavioral injection harness can be added to retire the question entirely?