Sentinel

Content safety monitoring for LLM applications

Every prompt.
Every reply.
Watched.

Sentinel sits behind your chat app, classifies every prompt and completion for harm, detects when the model's input distribution drifts from what it was trained on, and retrains + redeploys a new model version before that drift becomes an incident.

Set it up on Linux → Read the source
classifications flowing PSI > 0.2 triggers retrain quality-gated promotion

Data flow

One signal chain, start to finish

A chat app is treated as an external black box — Sentinel never touches its prompts directly. It only sees what arrives as OpenTelemetry telemetry.

01 · SOURCE

Chat app

Emits OTel spans over OTLP/gRPC on :4317. Prompt/response text travels in span events, not attributes, per the GenAI semantic conventions.

→
02 · COLLECT

OTel Collector

Fans every trace out to Kafka and to Jaeger for live inspection.

→
03 · STREAM

Kafka · traces.raw

3 partitions. A Python consumer extracts prompt/response text from each span.

→
04 · SCORE

Classifier

POST /v1/moderations — ONNX INT8 RoBERTa, OpenAI-moderation-compatible.

→
05 · PERSIST

Postgres + Mongo

Every score to classifications; harmful content (+10% of safe) to flagged_content.

→
06 · REACT

Drift + retrain

Hourly PSI/JSD check against a reference window; a breach fine-tunes, quality-gates, and rolling-restarts.

Human in the loop: flagged content is reviewed and labelled in the Label UI before it ever becomes training data — the model never trains on its own predictions.

See it running

Traces landing end to end

Simulated chat traffic flowing through the full local stack — collector, Kafka, classifier, Postgres/Mongo, Jaeger.

SENTINEL // LOCAL-DEV // k3d
Screen recording of the Sentinel stack processing simulated traces end to end

Load-tested, not assumed

One config change — ORT_INTRA_THREADS 4 → 1 — measured before/after on the same 1-vCPU-limited classifier pod.

+137% throughput single-string path, concurrency 10 — 15.3 → 36.3 req/s
−60% p99 latency same test — 899.7ms → 361.5ms
+92% throughput 32-item batch path, concurrency 1 — 17.0 → 32.6 items/s
2.6× CPU actually working avg utilization during test — 212m → 545m core

Metrics

What actually got measured

Real numbers from the model optimization pipeline, a load/stress test against a single 1-vCPU pod, the live Prometheus surface, and the gates that decide whether a retrained model ships.

Model optimization pipeline

StageSizep50 latency
PyTorch FP32~500 MB~110 ms
ONNX + O2 graph opt~480 MB~60 ms
ONNX + INT8 quant~120 MB~35 ms
~75% smaller, ~3x faster — dynamic INT8 quantization, activations quantized at runtime, no calibration dataset needed, <0.2% accuracy cost.

Load test — single-string path

ConcurrencyThroughputp50p99
126.1 req/s36.9 ms—
534.5 req/s149.0 ms—
1036.3 req/s284.3 ms361.5 ms
1 replica, cpu: 1000m / memory: 1Gi, after tuning ORT_INTRA_THREADS=1 for concurrent load — +99–137% throughput, −56–61% latency vs. the single-request-tuned default of 4 threads on the same pod.

Load test — batch path (32-item lists)

ConcurrencyThroughputp50
132.6 items/s960.6 ms
228.8 items/s2.19 s
523.4 items/s6.71 s
Throughput barely moves while latency scales with concurrency — a real CPU ceiling, confirmed by kubectl top pod hitting 996–1011m against the 1000m limit during every degraded stage.

Drift & promotion gates

0.2PSI above this, hourly, auto-triggers a retrain
Drift metricsPSI (gating) · JSD (visibility)
Held-out evaluation set3,780 rows · 9 risk categories
Minimum accuracy to promote0.85
Max allowed regression vs. active model0.01
Training data sourcehuman-labelled only, never model's own output

Prometheus surface — /metrics

  • classifier_requests_total{endpoint,label}Request count by route and predicted label
  • classifier_request_latency_seconds{endpoint}Histogram, 5ms–1s buckets
  • classifier_batch_sizeHistogram of texts per batch, 1–64
  • classifier_queue_depthGauge — requests waiting in the DynamicBatcher queue
  • classifier_log_errors_total{level}ERROR+ log records emitted

Kafka & storage shape

Topictraces.raw · 3 partitions
Offset commitmanual, after Postgres + Mongo writes
Idempotency key(span_id, text_type)
Safe-sample rate to Mongo10%
Scaling consumers beyond 3 replicas gives zero benefit — one per partition is the ceiling.

Tech stack

One tool per concern

Every phase introduces exactly the tool that concern needs — nothing borrowed from a later phase.

Inference API
FastAPIONNX Runtime
Model optimization
HuggingFace OptimumONNX Runtime
Fine-tuning
HuggingFace Transformers
Stream processing
Kafka (KRaft)Python consumer
Databases
PostgreSQLMongoDB
Object storage
MinIO
Drift detection
PySparkPSI / JSD
Orchestration
Apache Airflow
Experiment tracking
MLflow
Observability
OTel CollectorPrometheusGrafanaJaeger
Infrastructure
Terraformk3d
CI/CD
GitHub ActionsGHCR

Build phases

Every phase, one at a time

Built as a learning project — each phase fully working before the next begins. Phases 1–7 are complete; only cloud deployment remains.

# What gets built Key tools Status
01Classifier service — FastAPI + ONNX inferenceFastAPI, ONNX Runtime, Prometheus clientComplete
02Model optimization pipelineHF Optimum, ONNX, INT8 quantizationComplete
03Local infra — k3d, databases, object storageTerraform, Helm, PostgreSQL, MongoDB, MinIOComplete
04Observability stackOTel Collector, Jaeger, Prometheus, GrafanaComplete
05Trace ingestion + stream processingKafka (Strimzi), Python consumerComplete
06Drift detectionPySpark, PSI / JSD metricsComplete
07Orchestration + experiment trackingAirflow, MLflow, Label UIComplete
08Cloud deploymentTerraform workspaces, EKS/GKE, RDS, S3Pending

Setup — Linux

Run the whole stack locally

One script brings up a k3d cluster and every service in it. This is the exact sequence to follow, in order.

01

Install prerequisites

Everything below is available through your distro's package manager or its official installer.

  • Docker
  • k3d
  • kubectl
  • Terraform ≥ 1.6
  • uv
  • Python 3.12 (uv manages this if it isn't installed)
02

Clone and install dependencies

uv sync installs the root workspace plus the classifier's deps for local tooling.

# git clone https://github.com/VjayRam/project-sentinel.git
cd project-sentinel
uv sync --all-packages
03

Bring up the full stack

One script does everything: creates the k3d cluster, builds and imports every image, runs terraform apply, waits for readiness, unpauses both Airflow DAGs, bootstraps a model if the registry is empty, opens every port-forward, then rolling-restarts the classifier so it loads that model.

./scripts/dev-start.sh

Press Ctrl-C to stop cleanly — only the port-forwards stop; the k3d cluster keeps running. Re-run the script any time to reconnect.

04

Verify it's alive

# liveness / readiness
curl http://localhost:8000/health/ready

# send synthetic chat traffic through the real OTLP path
python scripts/simulate-traces.py --count 20 --harm-pct 0.3

# confirm it landed
psql postgresql://sentinel:sentinel@localhost:5432/sentinel \
  -c "SELECT label, count(*) FROM classifications GROUP BY label;"
05

Reach every service on localhost

Every port below is opened automatically by dev-start.sh via kubectl port-forward.

ServiceURLCredentials
Classifier APIlocalhost:8000—
Classifier docslocalhost:8000/docs—
Label UIlocalhost:8001—
Grafanalocalhost:3000admin / admin
Prometheuslocalhost:9090—
Jaeger UIlocalhost:16686—
Airflow UIlocalhost:8090admin / sentinel
MLflow UIlocalhost:5000—
MinIO consolelocalhost:9001sentinel / sentinel-minio

Get in touch

Questions, feedback, or hiring?

Reach out directly — happy to talk through the architecture, the build process, or anything else.