scikit-learn: flag unusual AI workflow runs for review

Add a review signal alongside ordinary workflow checks

An AI-enabled process can finish with an unusual combination of runtime and retries before anyone investigates it. A small anomaly detector can help prioritize a review queue. It should sit beside ordinary failure alerts and quality checks; unusual timing is not a diagnosis, and ordinary timing does not prove an AI answer is correct.

This AI-assisted editorial guide runs scikit-learn’s Isolation Forest on fictional workflow metrics. It fits a synthetic reference history and scores three separate examples. No customer logs, generative AI service or production automation are connected. The output is a local review suggestion, not an instruction to stop a worker or notify a customer.

1. Pin the tool and define the measurements

We verified scikit-learn 1.9.1 on October 27, 2026. The package requires Python 3.11 or later and uses the BSD-3-Clause license. The local run used Python 3.13.15 and NumPy 2.5.4. Install into a new scratch environment rather than changing an existing application’s dependencies. [1]

The example has two columns in a fixed order: duration_seconds and retry_count. Its history varies both measurements. Every row refers to the same fictional job type; a real project should first decide which workflow versions and job sizes are comparable.

Run these PowerShell commands. Save the next block as workflow_anomalies.py before the final command. Installation needs internet access; the example itself generates its inputs locally. The thread settings below match the bounded local test.

mkdir workflow-anomaly-demo
cd workflow-anomaly-demo
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install "scikit-learn==1.9.1" "numpy==2.5.4"
$env:OMP_NUM_THREADS='1'
$env:OPENBLAS_NUM_THREADS='1'
# Save workflow_anomalies.py below first.
.\.venv\Scripts\python.exe workflow_anomalies.py

2. Fit the reference history and score new runs

Isolation Forest uses randomized partitions to isolate observations; relatively short isolation paths can indicate unusual points. Here, the model fits only the 256 generated history rows. The three new rows are passed to predict and decision_function after fitting. [2]

Our features helper is an application policy: require two finite, nonnegative values and whole-number retry counts. Its rejection rules belong to this example, not to a claim that every scikit-learn estimator rejects the same inputs. This NumPy-array contract cannot detect semantically swapped columns, so keep column order explicit.

The model uses contamination="auto" and an integer random_state. Auto selects the documented threshold rule; it does not estimate the real incident rate or promise a particular alert percentage. The fixed seed makes this fixture repeatable within the tested setup. [3]

import json
import platform

import numpy as np
import sklearn
from sklearn.ensemble import IsolationForest


def features(rows):
    x = np.asarray(rows, dtype=float)
    if x.ndim != 2 or x.shape[1] != 2 or len(x) == 0:
        raise ValueError("Expected rows of [duration_seconds, retry_count]")
    if not np.isfinite(x).all() or (x < 0).any():
        raise ValueError("Features must be finite and nonnegative")
    if (x[:, 1] != np.floor(x[:, 1])).any():
        raise ValueError("Retry counts must be whole numbers")
    return x


# Synthetic history, not customer or production metrics.
rng = np.random.default_rng(7)
history = features(np.column_stack([
    rng.normal(30, 2, 256), rng.integers(0, 3, 256)
]))
new_runs = features([[30, 1], [180, 8], [31, 1]])
run_ids = ["demo-regular", "demo-spike", "demo-nearby"]
model = IsolationForest(contamination="auto", random_state=42).fit(history)
labels = model.predict(new_runs)
scores = model.decision_function(new_runs)
repeat = IsolationForest(contamination="auto", random_state=42).fit(history)


def rejected(rows):
    try:
        features(rows)
    except ValueError:
        return True
    return False


checks = [
    (labels.tolist() == [1, -1, 1], "fixture labels"),
    (np.isfinite(scores).all(), "finite scores"),
    (np.array_equal(labels, np.where(scores < 0, -1, 1)), "score signs match labels"),
    (np.array_equal(labels, repeat.predict(new_runs)), "repeat fit with same seed"),
    (rejected([[np.nan, 0]]), "missing value rejected by our adapter"),
    (rejected([[-1, 0]]), "negative value rejected by our adapter"),
    (rejected([[30]]), "wrong shape rejected by our adapter"),
    (rejected([[30, 0.5]]), "fractional retry rejected by our adapter"),
]
for passed, name in checks:
    if not passed:
        raise AssertionError(name)
    print("PASS:", name)

records = [
    {"run_id": run_id, "label": int(label), "decision_score": round(float(score), 4),
     "route": "review" if label == -1 else "observe"}
    for run_id, label, score in zip(run_ids, labels, scores)
]
print(json.dumps({"versions": {"python": platform.python_version(),
      "sklearn": sklearn.__version__, "numpy": np.__version__},
      "runs": records}, indent=2))

3. Read the observed results without overstating them

All eight checks passed. demo-regular, representing 30 seconds and one retry, received label +1 and a decision score of 0.0561. demo-spike, representing 180 seconds and eight retries, received label −1 and a score of −0.1733. demo-nearby, representing 31 seconds and one retry, received label +1 and a score of 0.004. Scores shown here are rounded to four decimal places.

The API uses +1 for inliers and −1 for outliers. Negative decision scores are on the outlier side of the decision boundary. These are model scores, not calibrated probabilities that a workflow failed. Our tiny output labels negative results review and other results observe; neither route changes a business system. [3]

The remaining checks verified finite scores, agreement between score signs and labels, identical labels from a repeated fit with the same seed, and our adapter’s rejection of a missing value, a negative value, a wrong-shaped row and a fractional retry count. These checks establish fixture behavior, not detection accuracy across real workflows.

4. Evaluate on separate, representative history

Keep evaluation runs outside fitting. Otherwise, a test can quietly become a description of data the model already saw. The scikit-learn pitfalls guide warns against fitting on evaluation data or learning preprocessing from it. If you add transformations later, fit them on the reference set and apply them consistently. [4]

For a real pilot, set aside approved historical runs with incident-review outcomes. Count both useful flags and false alarms. Include planned changes such as larger documents, a new model version or a different retry policy. Those changes can alter the metric distribution without representing an incident.

Compare the detector with a simple operational rule, such as an existing duration limit. Decide whether the extra review signal finds anything useful before making it part of an on-call workflow. Review the reference window when the process changes instead of automatically treating every newly scored run as normal training data.

5. Connect it to a bounded review queue

A proposed integration would attach the run identifier, workflow version, measurements and detector version to an internal review item. A person can then inspect the relevant error logs or task result. Duration and retry count alone cannot identify whether a provider slowed down, an input grew or a retry policy changed.

Keep explicit failed-status and deadline alerts independent of the model. Missing metrics should enter a data-quality path rather than being silently replaced with zero. Set a review capacity and escalation rule before increasing alert volume.

This guide implements none of those production integrations. It prints synthetic records only. Start with the minimum authorized metadata, keep sensitive document contents out of the feature set, and evaluate any automatic remediation separately from anomaly detection.

Sources & further reading

1. Official scikit-learn 1.9.1 package metadata: Python requirement and license
2. scikit-learn 1.9 user guide: Isolation Forest and outlier detection
3. IsolationForest API: parameters, scores and prediction labels
4. scikit-learn common pitfalls: evaluation leakage and consistent preprocessing

Make AI workflow monitoring useful to your team

Sociologix can help connect AI workers, process automation and operational review with clear evidence and human handoffs. Bring one recurring workflow and the signals your team already uses to assess it.