Troubleshooting AI Features in Microsoft Purview: Diagnose and Fix the Most Common AI Issues

Introduction

AI features in Microsoft Purview do not fail with error codes. A trainable classifier does not pop up a dialog saying “accuracy dropped 15%.” Adaptive protection does not warn you when its baseline is stale. Copilot does not tell you it is hallucinating. AI failures are silent. The system continues running, producing increasingly unreliable results, until someone notices that labels are wrong or alerts have stopped making sense.

This guide covers the most common AI-specific failures I have diagnosed across multiple Purview deployments, how to detect them before they cause harm, and exactly what to do to fix each one. It builds on the general Purview troubleshooting guide but focuses specifically on the machine learning features covered in the AI-powered compliance guide. If an AI feature is behaving unexpectedly, start here.

Trainable Classifier Accuracy Drops: Diagnose and Retrain

The most common AI failure I see is a trainable classifier whose accuracy degrades over time. It was trained six months ago on last year’s contracts. This year’s contracts use different templates and language. The classifier that achieved 92% precision at launch is now at 74%, and nobody noticed because nobody was tracking the metrics month over month.

The fix starts with diagnosis. Go to the classifier under Data classification and review the latest test results. Compare precision and recall against the previous month. A drop of more than 5% in either metric means the model needs attention. Do not wait for users to report mislabeled documents – by the time complaints reach you, thousands of documents may be incorrectly classified.

If accuracy has dropped, the root cause is almost always data drift. The documents your classifier sees today are different from the documents it was trained on. The fix is retraining. Go to the classifier, click Retrain, and provide new sample documents that represent what your organisation’s documents look like now. Include examples the classifier is currently getting wrong. After retraining, run the classifier in simulation mode for a week before switching back to enforcement. The trainable classifiers guide covers the full retraining methodology.

If accuracy has not dropped but users report mislabeled documents, the problem may be bias, not drift. The classifier performs well on documents from one department but poorly on another. Segment your accuracy metrics by department and document type to identify the gap. The fix is adding more training samples from the underperforming segments. This is covered in the responsible AI governance guide.

Adaptive Protection: Zero Alerts, Too Many Alerts, or Stale Baselines

Adaptive protection failures fall into three patterns. Each has a distinct cause and a specific fix. The first pattern is zero alerts. You enabled adaptive protection, waited 30 days, and the alerts queue is empty. The most common cause is that the learning period is not actually complete – the model needs a full 30 days of activity per user, not 30 calendar days from when you enabled the feature. If a user was on leave for two of those weeks, their baseline is incomplete.

Check the adaptive protection status in Insider Risk Management. It shows how many users have completed baselines. If most users are still in learning mode, wait another week. If all users have completed baselines and alerts are still zero, your anomaly sensitivity may be set too high. Lower it incrementally – one step per week – until alerts start appearing. Monitor the false positive rate at each step.

The second pattern is too many alerts – hundreds per day, mostly false positives. This happens when a large group of users changed roles or departments after their baselines were built. The model sees legitimate new behaviour as anomalous. The fix is to reset baselines for the affected users. Go to the user list in Insider Risk Management, select the users, and reset their adaptive baselines. They will go through a new 30-day learning period. During that period, rely on static thresholds for coverage.

The third pattern is stale baselines – alert patterns that were normal six months ago but now seem wrong. If your organisation had a reorganisation, acquisition, or major tool change, every user baseline may be outdated. The most reliable fix is a full reset: disable adaptive protection, re-enable it, and let all users rebuild baselines over 30 days. This is disruptive because you lose adaptive coverage for a month, which is exactly why you should always run static thresholds alongside adaptive protection as a safety net.

Copilot Query Failures: No Results, Wrong Results, and Hallucinations

Security Copilot failures are the most visible AI issues because users notice them immediately. When a compliance analyst asks Copilot for DLP alerts and gets nothing back, they do not assume the model is learning – they assume it is broken. Three Copilot failure patterns recur across deployments.

Copilot returns no results for queries you know should return data. This is almost always an audit log issue, not a Copilot issue. Copilot queries the same unified audit log you search manually. If the audit log does not contain the data – because logging was not enabled, retention expired, or the record type is not covered – Copilot cannot return it. Verify the data exists by running the equivalent audit log search manually. If the manual search also returns nothing, the data is not there. Fix the logging configuration, not Copilot.

Copilot returns results that do not match the query. This is usually a prompt engineering issue, not a model failure. A vague prompt like “show alerts” returns everything. A specific prompt like “show high-severity DLP alerts for j.smith this week involving credit card data” returns targeted results. The prompt engineering guide covers how to write prompts that return what you need. Before concluding Copilot is broken, tighten your prompt and re-query.

Copilot returns factually incorrect information – hallucination. This is the most serious failure. Copilot can misattribute an action to the wrong user, misreport a timestamp, or summarise a document in a way that changes its meaning. The fix is not technical – it is procedural. Every Copilot response includes source links. Click them. Verify the underlying data matches the summary. If it does not, report the hallucination through the feedback mechanism and do not act on the incorrect summary. For governance guidance, the Copilot governance guide covers verification workflows and audit requirements.