Positive, Negative, Boundary, and Regression Tests for Detections

A test is an evidence claim with an expected decision

A fixture contains a new OAuth grant followed by resource access, and the analytic matches. The useful test record says more: which behavior is represented, which source semantics are assumed, which entity joins and times apply, and what evidence the result should contain.

Define test input, setup or transformation, expected observation, expected analytic output, and the reason. Keep the expected decision stable even if the backend syntax changes.

Tests should be small enough that a failure identifies an assumption. A single giant scenario containing many behaviors may produce an alert while leaving no clue which condition mattered.

Positive tests establish the supported path

A positive test represents behavior that should satisfy the analytic under defined conditions. Include required and optional evidence, expected grouping, timing, severity or risk contribution, and output fields.

Use more than one positive when the requirement claims meaningful variation: different identity types, provider operations, platforms, or legitimate field absence. Do not add variants merely to inflate coverage; tie each to a stated path.

A passed positive test establishes that the implementation recognizes the exercised case. It does not establish precision, complete recall, or production observation. State that limit with the result.

Negative tests protect ordinary explanations

A negative test represents evidence that resembles the threat but should not produce the same decision. Examples include approved administration, expected backup, a different entity relation, harmless tool use, or an action with a denied outcome.

Negatives should be difficult, not trivial records that share none of the selection. Change one meaningful condition at a time so the test explains why the conclusion differs. Include malicious counterexamples for filters and allowlists: activity that uses a trusted name but violates the legitimate workflow.

Some cases should produce enrichment or lower risk rather than no output. Expected absence of an alert is not necessarily expected absence of analytical value.

Boundary tests make exact behavior deliberate

Thresholds, windows, cardinality, string limits, null handling, and state retention all have edges. Test just below, exactly at, and just above numeric boundaries. Test events at both ends of a time window and records arriving after the window.

Boundary behavior should follow a documented choice. “Within ten minutes” must say whether exactly ten minutes qualifies and which clock drives the difference. A count threshold must say whether it fires at ten or after ten.

The detection testing pyramid keeps most edge tests low and fast while reserving a smaller set for backend and pipeline semantics. Conversion can change an inclusive operator or timestamp field, so test important boundaries on targets too.

Regression tests turn failures into durable knowledge

A regression test preserves a behavior learned from an incident, false positive, provider change, parser bug, backend difference, or escaped edge case. It should reference the reason and fail before the correction it protects.

Keep the smallest record or scenario that reproduces the issue while preserving relevant semantics. Redact sensitive evidence without changing the property under test. Name which version fixed it and which components can reintroduce it.

Review regressions rather than accumulating them blindly. A retired requirement may make a case obsolete. A new source model may require migration. The collection is valuable when it remains an intelligible history of how the detection claim became more precise.