API Overview

miLLM exposes two HTTP API surfaces and a WebSocket layer on the same port (default 8000):

Surface	Base path	Purpose	Error format
OpenAI-compatible	/v1	Inference: chat, completions, embeddings, model list	OpenAI-style {"error": {...}}
Management	/api	Models, SAEs, steering, monitoring, profiles, health	Envelope (below)
WebSocket (Socket.IO)	/socket.io	Progress, metrics, live activations	—

There is no authentication — miLLM is designed to run on trusted networks. Put it behind a reverse proxy with auth if you expose it further. CORS origins are configured via CORS_ORIGINS.

The management envelope

Every /api/* endpoint returns the same wrapper:

Success

{
  "success": true,
  "data": { "...": "endpoint-specific payload" },
  "error": null
}

Failure

{
  "success": false,
  "data": null,
  "error": {
    "code": "SAE_NOT_ATTACHED",
    "message": "No SAE attached",
    "details": {}
  }
}

error.code is machine-readable and stable — the full list with HTTP status mappings is in the Error Codes reference. details carries context (e.g. the offending feature_idx and the SAE's d_sae on an index-validation failure).

/v1/* endpoints return OpenAI-format errors instead, so OpenAI SDKs raise their native exception types:

{
  "error": {
    "message": "No model is currently loaded",
    "type": "invalid_request_error",
    "code": "model_not_loaded"
  }
}

Endpoint map

Area	Base	Reference
Chat/completions/embeddings	/v1/...	OpenAI-Compatible API
Model lifecycle	/api/models	Models
SAEs, attachment & steering	/api/saes	SAEs & Steering
Activation monitoring	/api/monitoring	Monitoring
Steering profiles	/api/profiles	Profiles
Health & operations	/api/health	below
Live events	Socket.IO	WebSocket Events

Health & operations endpoints

Endpoint	Method	Description
/api/health	GET	Liveness — process is up
/api/health/ready	GET	Readiness — dependencies (DB, GPU) are usable
/api/health/detailed	GET	Per-component health breakdown
/api/health/inference	GET	Active inference backend (serial vs cbm), its capabilities and limitations
/api/health/metrics	GET	Application metrics (requests, latency, GPU)
/api/health/metrics/prometheus	GET	Prometheus exposition format
/api/health/circuits	GET	Circuit-breaker states (HuggingFace calls)
/api/health/circuits/{name}/reset	POST	Manually reset a tripped breaker
/api/health/version	GET	Application version

curl -s http://localhost:8000/api/health

{"status": "healthy", "version": "0.5.0", "timestamp": "2026-07-11T12:20:05Z", "uptime_seconds": 512.4}

Conventions

• Content type is application/json for all request bodies
• IDs: models use integer IDs; SAEs and profiles use string IDs (sae_...-style hashes, prof_...)
• Queueing: generation requests beyond the concurrency limit wait in a bounded queue; when full, requests are rejected with 503 (QUEUE_FULL — backpressure, so clients back off and retry rather than treating it as a rate limit)
• Timestamps are ISO-8601 UTC