Testing Detections with Adversary Emulation

Testing Detections with Adversary Emulation

This lesson teaches students how to test detections through safe adversary emulation. They learn the difference between simulation and emulation, how to run controlled tests, and how to interpret the results to improve detection quality.

Simulation Versus Emulation

Before you can test detections effectively, you need to understand the difference between simulation and emulation. These terms are often used interchangeably, but they mean different things.

Simulation

Simulation imitates the appearance of an attack without performing the real actions. A simulated phishing test, for example, sends a harmless email that looks like a phishing message. It does not use real attacker infrastructure. Simulation is useful for training and awareness, but it is limited for detection testing because it may not create the same telemetry as a real attack.

Emulation

Emulation imitates the behavior of an attacker using real techniques in a controlled way. An emulation might actually execute a command, create a scheduled task, or make a network connection, but in a safe test environment with clear boundaries. Emulation produces realistic telemetry, which is exactly what you need to validate a detection.

Why emulation matters for detection engineering

A detection is designed to match specific observables. If your test does not produce those observables, you have not really tested the detection. Emulation gives you confidence that the detection fires when the real technique occurs. It also reveals gaps: if you emulate a technique and your detection does not fire, you have found a problem before an attacker does.

Common emulation tools

Several open-source tools support adversary emulation. MITRE Caldera and Atomic Red Team are two widely used examples. They provide pre-built tests for many ATT&CK techniques. Always run such tools only in designated test environments with proper authorization.

Safe Testing Practices

Adversary emulation is powerful, but it must be done safely. Running attack commands on a production system is dangerous and can cause real damage. A controlled approach protects both the environment and your career.

Use a dedicated test environment

Never run emulation tests against production systems. Create an isolated lab with representative telemetry. The lab should have endpoint agents, logging, and network monitoring configured similarly to production so the telemetry is realistic. This gives you a safe place to run tests and validate detections.

Get written authorization

Before running any emulation, ensure you have permission. The security team, system owners, and relevant leadership should know what you are testing and where. This is not just a formality. It protects you and the organization if something goes wrong.

Start small and observe

Begin with low-impact techniques. Do not start by emulating ransomware or destructive actions. Start with reconnaissance or simple execution techniques. Observe how the detection and the telemetry respond. Then gradually move to more complex techniques.

Track every test

Record what technique you emulated, when, on which host, and what happened. Did the detection fire? Did it create the expected telemetry? Was there any unexpected behavior? This log becomes an audit trail and a learning resource.

Safety first, always

If you are unsure whether a test is safe, stop and ask. The goal is to improve detection, not to create an incident. Safe testing is a professional discipline, not a limitation.

Interpreting Test Results

Running an emulation test is only half the work. The real value comes from interpreting what happened. A test result can tell you much more than simply whether an alert fired.

Four possible outcomes

When you run an emulation, there are four possible outcomes:

1. Detection fires for the right reason. This is success. The detection matched the expected observables.
2. Detection does not fire. This is a problem. Either the detection logic is wrong, the required data is missing, or the emulation did not create the expected telemetry.
3. Detection fires for the wrong reason. The alert fired, but on unrelated activity. This suggests the detection is too broad and needs tuning.
4. Detection fires but is delayed. The alert eventually fired, but too late to be useful. Investigate why the delay occurred.

Asking why

When a test does not go as expected, ask why before changing anything. Did the test create the expected telemetry? You can verify this by querying the raw data directly, bypassing the detection. If the telemetry is present but the detection did not fire, the logic is the problem. If the telemetry is absent, the test or the data pipeline is the problem.

Document the results

Every test should end with a short note. Record the technique, the expected result, the actual result, and any follow-up actions. This documentation helps future you, your teammates, and your organization track the health of your detections over time.

Continuous validation

Testing is not a one-time event. As detections are tuned, as telemetry changes, and as new techniques emerge, re-test regularly. A mature detection program treats validation as part of the lifecycle, not a final step before deployment. Each re-test builds confidence that your detections still work.