Treat Baselines and Risk Scores as Arguments

Treat Baselines and Risk Scores as Arguments

Understand drift, calibration, double-counting, and the difference between a score and a conclusion.

Expectations and accumulated evidence must remain explainable

A baseline answers a deceptively simple question: compared with what? If a user downloads fifty files, you might compare the activity with that user’s past, with peers in the same role, with the normal behavior of the application, or with a business limit. Each comparison can produce a different result because each expresses a different expectation.

A baseline is an explicit model of expected behavior for a defined population and period. It is not reality. Historical data can include compromise, outages, reorganizations, and seasonal work. If you train an expectation on quarter-end activity and apply it during a quiet month, normality shifts. If you update it too quickly, persistent malicious behavior can become normal.

That change is called drift. Data drift means the inputs change, such as a new application generating different fields. Concept drift means the relationship between the inputs and the security meaning changes, such as remote administration becoming common after a policy shift. Monitoring only the model score will not reveal why the expectation stopped fitting. You need to observe the population, features, missingness, and operating context that give the score meaning.

Risk-based alerting approaches the problem from another direction. Instead of asking whether one event crosses a final threshold, it assigns risk contributions to pieces of evidence and accumulates them for an entity over time. A new device may add a little concern. A new OAuth grant may add more. Sensitive data access soon afterward may push the identity above an investigation threshold.

The number can help manage many weak signals, but it can also hide arbitrary judgment. Two detections may derive from the same underlying event and double-count it. A stale entity link may attach one person’s activity to another. A high-weight rule may dominate every result. A feedback loop can arise when an alert is ingested as new evidence and increases its own score. Risk-Based Alerting and Evidence Accumulation explains these failure modes.

Treat the score as an argument with visible premises. An analyst should be able to see which observations contributed, when they occurred, how they decayed, which were related, and which data sources were healthy. Calibration asks whether score ranges correspond to useful differences in outcomes or decisions. It requires reviewing distributions and cases, not merely choosing a threshold that produces the desired alert volume.

Baselines and risk scores are useful because they express context that a single event cannot. They become dangerous when the numeric output looks more objective than the assumptions beneath it. Your goal is not to eliminate judgment. It is to make judgment inspectable.

Resources

• Sigma Correlation Rules Specification — Primary reference for the standards and concepts discussed in this lesson.