Troubleshooting

Organized by symptom. For error codes returned by the API, see the Error Codes reference.

Steering has no effect

The most important failure mode in miLLM — an intervention that silently isn't happening invalidates your experiment. Check in this order:

1. Is steering actually enabled with non-zero values?

   curl -s http://localhost:8000/api/saes/steering | jq '.data'

2. Is the hook firing? Generate a completion, then:

   curl -s http://localhost:8000/api/saes/attachment | jq '.data.steering_apply_count'

   The counter must increase across a steered generation. If it doesn't move, the hook isn't executing — re-attach the SAE and check server logs. (miLLM guards the known torch.compile-swallows-hooks failure automatically, and attach/detach forces a recompile; a non-moving counter after that indicates a genuinely unsupported architecture.)
3. Right module? The attach response's layer_module_path should name a decoder layer (e.g. model.layers.12). On multimodal/exotic architectures the layer-resolution heuristics can land elsewhere — try attaching by a different layer index and compare.
4. Strength too low or feature not what you think? Sweep to ±100 on a feature with an obvious signature (see the tutorial); check the feature's meaning on Neuronpedia.
5. Wrong model variant? SAEs trained on gemma-2-2b are much weaker on gemma-2-2b-it. The attach warnings array tells you about this mismatch.

Out-of-memory (OOM)

Symptom	Cause	Fix
OOM during model load	Model too large for GPU	More aggressive quantization (Q8/Q4), or let device_map=auto offload to CPU
OOM during inference	model + KV cache + SAE exceeds VRAM	Reduce max_tokens, use a narrower SAE (16k vs 65k+), or reduce MAX_PENDING_REQUESTS to bound queued work (MAX_CONCURRENT_REQUESTS must stay 1 — see Configuration)
OOM only with hybrid/Mamba models	mamba-ssm not installed; naive fallback allocates 20 GB+ intermediates	Install mamba-ssm, or set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
VRAM used but no model loaded	Leaked memory after a crash	Restart the backend process/pod, then reload

On OOM with an SAE attached, miLLM degrades gracefully: the SAE is disabled and the base model continues serving — check the attachment status if steering suddenly stops.

nvidia-smi --query-gpu=memory.used,memory.free,memory.total --format=csv,noheader
nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv,noheader

Sizing guidance: Hardware Requirements.

Downloads & model loading

Symptom	Cause	Fix
401 GATED_MODEL_NO_TOKEN	Gemma/Llama are gated	Accept the license on the HF model page, supply hf_token
404 REPO_NOT_FOUND	Typo, or private repo without token	Check the repo ID; add a token
Load fails: "TokenizersBackend"	Model needs a custom tokenizer package	Retry with trust_remote_code: true
Load fails: BitNet/GPTQ conflicts	Pre-quantized weights + bitsandbytes don't mix	Download with FP16 (miLLM auto-detects most cases and skips bitsandbytes)
First request after load is very slow (~20 s)	One-time torch.compile warmup	Expected; subsequent requests are fast. Also occurs once after SAE attach/detach. Disable with TORCH_COMPILE=false if warmup matters more than throughput
Download stuck	Network or HF rate limiting	Cancel (POST /api/models/{id}/cancel) and retry; check GET /api/health/circuits for a tripped HuggingFace breaker

GGUF-specific

Symptom	Cause	Fix
Could not create a llama.cpp context at any context length (tried …)	Usually not the context size. Some architectures refuse the pooling mode embeddings require and fail at every length	miLLM already retries without embeddings and records supports_embeddings: false. If it still fails at every rung with the card nearly empty, the file itself is the problem, not the window
400 AMBIGUOUS_MODEL_NAME	A bare repository name now matches two or more downloaded quantizations	Name the tag exactly, e.g. …-GGUF:IQ4_XS. The error lists the tags that exist
400 context_length_exceeded on a prompt that used to work	The window shrank — usually GGUF_KV_CACHE_TYPE set back to f16, or GGUF_FLASH_ATTENTION turned off	Restore q8_0 + flash attention, which roughly triples the window. See Hardware
Context window smaller than expected	GGUF_CONTEXT_LENGTH is a ceiling, and the loader also stops at what VRAM can hold	Check the load log for the predicted and achieved window; free VRAM, or lower the KV-cache precision
/v1/embeddings returns an error for one GGUF model	That model loaded without embedding support	Check supports_embeddings on the model record; use a different model for embeddings
A Continue action restarts the answer from the beginning	Fixed — GGUF chat templates seal the final turn, and miLLM now completes a trailing assistant message instead	Update miLLM; no client change needed. See OpenAI-compatible API

SAE attachment

Symptom	Cause	Fix
400 SAE_INCOMPATIBLE (dimension mismatch)	SAE d_in ≠ model hidden size	Use the SAE built for this exact model size (2b vs 9b vs 27b)
Attach warning: trained on different layer/model	Deliberate mismatch or wrong file	Heed it — features are only meaningful at the trained layer on the trained model
409 SAE_ALREADY_ATTACHED	One SAE at a time	Detach first
Detach appears to hang	Waiting (up to 30 s) for in-flight requests to finish	Expected — the hook is never removed mid-generation

Inference & API

Symptom	Cause	Fix
503 model_not_loaded on /v1/*	No model loaded	Load one; consider AUTO_LOAD_MODEL for restarts
503 QUEUE_FULL errors	Request queue full (MAX_PENDING_REQUESTS) — backpressure	Raise the limit, or slow the client (Open WebUI's parallel title-generation is a common culprit)
400 context_length_exceeded	prompt + max_tokens > model context	Shorten the prompt or reduce max_tokens
Streaming stops with an in-stream error event	Generation failed after headers were sent (HTTP is already 200)	Check server logs; SSE consumers should watch for error objects before [DONE]
Responses look like completions, not chat	Base (non--it) model	Expected — base models aren't instruction-tuned (why you might want that)
CORS errors in browser	Origin not allowed	Set CORS_ORIGINS (Configuration)
500s on /v1/chat/completions after a crash	Leaked GPU state	Restart the backend, reload the model

Monitoring

Symptom	Cause	Fix
No activation records	Monitoring disabled, or no completions since enabling	GET /api/monitoring to check state; records are written once per completed generation
Wrong/garbage feature indices in history	(Fixed in current versions) watched-feature list desync	Upgrade; configure via /api/monitoring/configure
History entry per request, not per token	By design — records capture the final forward pass	See monitoring semantics
WebSocket disconnects during long generations	Event loop busy	Normal; the client auto-reconnects — resync state via REST after reconnect

Kubernetes-specific

kubectl get pods -n millm                      # pod states
kubectl logs -n millm deploy/millm-backend     # backend logs
kubectl delete pod -n millm <pod-name>         # restart to clear leaked VRAM

GPU scheduling requires the NVIDIA device plugin and a node with the GPU visible; see the Kubernetes install guide.

Still stuck?

• Set LOG_LEVEL=DEBUG and reproduce — steering/hook activity is logged
• GET /api/health/detailed gives a per-component health breakdown
• Open an issue: github.com/Onegaishimas/miLLM with logs and your model/SAE combination