The Detection Testing Pyramid
Build confidence through many fast semantic tests, fewer integration tests, and carefully governed end-to-end scenarios, with each layer proving only the assumptions it actually exercises.
Different tests answer different questions
A query returns the expected match against a fixture. That proves the logic handled that record. It does not prove the sensor will observe the behavior, the parser will preserve the field, the schedule will run, or the alert will reach a consumer.
The testing pyramid organizes assurance by scope, speed, precision, cost, and realism. Lower layers isolate small assumptions and run frequently. Higher layers include more real components and expose integration failures, but they are slower and make failures harder to locate.
Use the pyramid as a distribution of questions, not a maturity ladder. A realistic scenario cannot efficiently test every null, boundary, duplicate, and backend edge. A unit fixture cannot establish live observation. Confidence comes from the layers agreeing.
Schema and semantic tests protect the evidence contract
Begin with record structure, types, required and conditional fields, enum values, entity roles, time semantics, and source meaning. A schema-valid record can still be semantically wrong, so include cases where actor and target differ, outcomes are absent, or provider operations have similar names.
These tests are fast enough to run on every content, parser, and mapping change. They should fail when a required field disappears or when an unknown value is silently defaulted.
Keep source examples and normalized expectations versioned. The purpose is not to freeze providers forever; it is to make a changed assumption visible before downstream logic interprets it under the old contract.
Logic tests exercise decisions and edges
Unit fixtures should cover intended behavior, ordinary alternatives, missing context, threshold edges, time boundaries, duplicates, entity collisions, and previously fixed regressions. Expected output includes not only match or no match but grouping, evidence, confidence, and reason.
The positive negative boundary and regression tests page explains why a happy-path match is the beginning, not the end. A negative case protects precision. A boundary case explains exact behavior at the edge. A regression case preserves a learned lesson.
Test the source intent separately from each compiled backend. This distinguishes an authoring error from translation or platform semantics.
Integration tests add real transformations and state
Integration tests include parsers, normalization, enrichments, converters, backend queries, correlation state, schedules, or routing. Replay can send preserved evidence through some or all of this path. These tests expose field mapping, time, state, and delivery assumptions.
Record which components were real and which were bypassed. A replay injected after collection says nothing about sensor capture. A backend test without case creation says nothing about the consumer route.
Diagnose from the source forward. Find the first point where expected and actual evidence diverge. Tuning the final query before locating that break can change the detection instead of fixing the integration.
End-to-end scenarios establish bounded operational evidence
Simulation or emulation can cause real systems to produce evidence and exercise delivery to the consumer. It requires authorization, safety, isolation, cleanup, expected observations, and clear stop conditions. Realism introduces variation as well as confidence.
A passed scenario proves only the behavior variation, environment, versions, and path exercised. It does not establish universal recall or future health. Preserve those conditions with the result.
The pyramid remains healthy when failures at a high layer become lower-layer regression tests, and when lower-layer speed allows frequent change without pretending integration is solved. Assurance is a maintained body of evidence, not one dramatic demonstration.
Frequently asked questions
Why not validate every detection only with end-to-end attack emulation?
Because end-to-end scenarios are slower, riskier, harder to diagnose, and cover fewer edge cases. They add evidence about integration and observation but do not replace precise schema, logic, boundary, and regression tests.
