Inference Service Status and Usage¶
The live dashboard below shows how busy each model is and how the inference service is performing. It covers the past 6 hours and refreshes every 5 minutes. Hover over the ⓘ icon in a panel’s title for a short description.
For outages, planned maintenance, and announcements, see the inference-service channel of our community chat. It also receives automated alerts when something goes offline.
What the metrics mean¶
Every request you send, from the chat UI or the APIs, passes through an API gateway (LiteLLM) on its way to the model. Each model runs on its own GPU server using vLLM. The dashboard shows metrics from both. Some panels list a model under more than one name, such as Qwen/Qwen3.8-27B and qwen-3.8-27b. These are aliases for the same model.
Model activity¶
-
Running Requests: How many requests each model is working on right now. A model works on many requests at once, so a number in the dozens is normal.
-
Request rate (requests/hour): How quickly each model is finishing requests, shown as a rate per hour. It is calculated from the last minute or so, so it rises and falls quickly; it is not a count of the requests in the past hour.
-
Token generation rate (tokens/minute): How many output tokens each model is producing per minute, across all of its users. It counts only the tokens a model writes, not the prompt tokens it reads.
API gateway (LiteLLM)¶
-
LiteLLM requests per minute by model and status: Requests that the gateway answered, by the model name the client asked for and the HTTP status code it returned.
- 200 means the request succeeded.
- 429 means the model was handling as many requests as the gateway allows at once. Wait a few seconds and try again.
- 4xx codes other than 429 usually mean a problem with the request itself, such as an invalid API key or a prompt longer than the model’s context window.
- 5xx codes mean the model or the gateway failed to answer.
-
Time LiteLLM adds per request (mean): The average time a request spends in the gateway, beyond the time the model takes to answer it. It is normally a small fraction of a second. It rises when models are busy and the gateway has to wait and retry before it can pass a request on.
-
LiteLLM in-flight requests (all models): How many requests the gateway is handling right now, across all models.
-
Backend state in LiteLLM: The gateway’s view of each model’s server: healthy, partial outage (some requests are failing), or outage (the gateway has stopped sending requests to it).
Latency¶
- Requests by end-to-end latency (one panel per model): Each bar counts the requests in the past 6 hours that finished within the number of seconds in its label, including the time spent generating the whole response. Each bar counts the requests that took longer than the label to its left, up to its own label. Reasoning models often spend a minute or more thinking before they answer, so requests in the higher bars are normal and do not mean that something is wrong.
Cache¶
-
Prefix cache hit rate: The share of prompt tokens that a model did not have to process again because an earlier request began with the same text. A system prompt, or the earlier part of a chat conversation, is often reused this way. A higher hit rate means responses start sooner. The line has gaps when a model received no requests.
-
KV cache usage: How full each model’s KV cache is. This is the GPU memory where a model keeps the requests it is working on. When it approaches 100% (the red line at 90%), new requests wait for room and responses slow down.
-
Preemptions, selected time range: How many times in the past 6 hours a model paused a request it had started, to free memory for others. A paused request resumes later, so it still finishes, but more slowly. Zero is normal; any other number turns the panel red and means the model is overloaded.
What should I do if…¶
I get HTTP 429 errors from the API.
The model is at the limit of requests that the gateway allows at once. Retry after a few seconds, ideally with exponential backoff, and avoid sending large batches all at the same moment. Many OpenAI-compatible client libraries retry 429s for you.
A model shows “partial outage” or “outage”.
Try another model for now, and check the inference-service channel for news. We are alerted automatically when a model stops responding, but it helps to tell us in the channel if you see errors that the dashboard doesn’t show.
Responses are slow.
Long prompts and long responses take longer, and reasoning models think before they answer. If a model’s KV cache usage is high or it shows preemptions, it is busy: try again later, or use a less busy model.