Real-Time Crypto Anomaly Detection
Live anomaly scoring for BTC, ETH, and SOL on a streaming ML pipeline. Coinbase WebSocket → Azure Event Hubs → a stateful feature engine → a Redis feature store → an Isolation-Forest + autoencoder ensemble — with weekly automated retraining and end-to-end observability, all deployed by CI/CD.
01 Overview
Anomaly detection on live market data is a genuine streaming ML problem: features have to be computed on the fly, served with millisecond latency, and stay consistent between training and serving. This project builds that full loop on Azure and wires it so every git push deploys the right piece automatically.
The detail I'm most proud of is the single source of truth for features: the same definitions drive the online streaming path and the offline training path, so there's no train/serve skew — the classic way production ML silently breaks.
02 Results
03 Architecture
Three Azure Container Instances and a feature store, from a live trade tick to an anomaly score with attribution. The runtime path at a glance, then each stage below.
Weekly: an Azure ML pipeline (validate → train → evaluate → register) retrains the model, and the scoring service redeploys automatically.
Producer
An async Coinbase Advanced-Trade WebSocket client streams live trade ticks for BTC/ETH/SOL and publishes to Azure Event Hubs (Kafka protocol) — with dead-letter handling, structured logging, and graceful shutdown.
Stateful feature engine
A per-symbol engine keeps sliding 1m/5m/30m windows as time-evicted deques and computes RSI(14), VWAP, Bollinger-band position, and rolling price z-scores — the same code used to build training features.
Feature store
The latest feature vector per symbol is written to Azure Cache for Redis (features:v1:{SYMBOL}, TTL 300s) so the scoring endpoint reads fresh features in under 5 ms.
Scoring endpoint
On each request the endpoint reads features from Redis and scores them with an ensemble — Isolation Forest for hard outliers plus a Keras autoencoder (latent dim 6) for subtle distributional drift, blended 50/50 — returning score, top features, and latency.
Weekly retrain
An Azure ML component pipeline (validate → feature-engineer → train → evaluate → register) runs every Monday 06:00 UTC; a new model is registered and the scoring endpoint redeploys — no manual steps.
04 MLOps decisions
One feature definition, both paths
The online streaming processor and the offline training job import the same feature module — a single source of truth that eliminates train/serve skew by construction.
Path-filtered CI/CD
Five GitHub Actions workflows deploy only what changed — edit features/ and the stream pipeline rebuilds; edit serving/ and just the endpoint redeploys. No manual, no local Docker.
Two detectors, two failure modes
Isolation Forest isolates hard outliers; the autoencoder's reconstruction error catches subtle distributional deviations. The 50/50 ensemble covers what either misses alone.
Observability that alerts
A live Azure Workbook tracks freshness, throughput, pipeline lag, and p50/p95/p99 latency; three alert rules fire on a stalled pipeline, slow scoring, or high feature lag.
Hands-off retraining
A scheduled Azure ML pipeline retrains, evaluates, and registers weekly, then triggers redeploy — the model stays current with the market without anyone touching it.
Graceful failure
Dead-letter handling on the producer and a "no features found" counter on the endpoint mean gaps are observable and recoverable rather than silent.
05 Stack
The full pipeline is on GitHub
Including the Azure ML component definitions, the CI/CD workflows, and the monitoring workbook.