Event Time, Ingest Time, Clock Drift, and Late Data

One event can carry several legitimate times

A cloud service records an administrative change at 10:00 according to its own clock. A collector observes it at 10:03, a queue delivers it at 10:12, and an analytic evaluates it at 10:15. None of those timestamps is inherently wrong. Each describes a different part of the evidence path.

Event time describes when the producer says the activity occurred. Observation time describes when a sensor saw it. Ingest time describes entry into a platform or pipeline stage. Processing time describes when logic handled it. An alert and a case can add still more times.

Choose time according to the claim. Behavior order usually depends on event or observation time. Pipeline freshness depends on the difference between those times and ingest. Operational responsiveness depends on processing and delivery. Calling every field timestamp makes these questions impossible to separate.

Clock drift changes apparent order

Two producers can record the same sequence using clocks that differ by minutes. A host may show a process start after a network connection that it actually caused. A strict ordered analytic can reject the sequence even though both events are genuine.

Measure clock offset where possible and record uncertainty. Some sources provide synchronized service time; others inherit device clocks that can drift or be changed. Precision matters too. A timestamp rounded to seconds cannot reliably order events that occur within the same second.

Do not silently widen every window until tests pass. A wider tolerance can admit unrelated activity and join concurrent sessions. Preserve stable identifiers and causally meaningful relationships when they exist, then use time as one constraint rather than the only proof of connection.

Late arrival is normal until the design pretends it is not

Batch exports, offline endpoints, provider backlogs, network interruption, and retry queues all create late data. A five-minute correlation window can miss related events when the behavior occurred within five minutes but one record arrived an hour later.

Define a lateness allowance: how long the analytic waits or revises state for delayed evidence. A streaming result may be provisional and later enriched. A retrospective analytic may trade speed for completeness. The right choice depends on the consumer’s decision, not on a universal window.

Keep event-time windows separate from state-retention limits. An engine may understand that two events occurred close together but discard the first before the second arrives. Document watermarking, update behavior, duplicate handling, and whether a late event creates a new alert or amends an existing one.

Time boundaries are part of the security claim

“Within thirty minutes” sounds precise, but the number needs a reason. It may reflect token lifetime, expected workflow duration, incident response value, or an empirical distribution. It should not exist only because a product default was available.

Test events just inside and outside the boundary, with equal timestamps, reversed arrival, duplicated delivery, missing zones, daylight-saving transitions, and different precisions. The detection testing pyramid should include fast fixtures for logic and environment tests for actual source clocks and pipeline delay.

Report which time drove the match. An analyst comparing source records needs to know whether the displayed order reflects event time, ingest time, or interface sorting. Otherwise a pipeline artifact can be mistaken for attacker sequence.

Use time disagreement as evidence about the pipeline

Differences among timestamps can reveal more than inconvenience. Rising event-to-ingest delay can warn that coverage is becoming stale. A sudden source-clock shift can explain impossible sequences or suggest unauthorized time changes. Processing delay can show that the platform no longer meets the decision latency.

Preserve these measures with alerts when they affect interpretation. A result based on late records may remain useful for investigation but arrive too late for containment. A result built from uncertain clocks may support co-occurrence but not strict order.

Good temporal design does not force every record onto one artificial clock. It keeps the times, their sources, and their uncertainty visible so each conclusion uses the time semantics that actually support it.