Definition
Inference
Inference is the act of running a trained model to produce output from new input — the moment a model is actually used, as opposed to trained. Every prompt sent to a language model triggers an inference pass. Inference dominates the running cost and latency of AI features, since it happens on every request while training happens once.
Key takeaways
- Inference is running a trained model on new input to produce output — the moment a model is used, as distinct from being trained.
- Every prompt to a language model triggers an inference pass; training happens once, inference happens on every request.
- LLM inference is autoregressive — one token at a time — so latency and cost scale with the number of tokens produced.
- Because it is the recurring cost, inference is where model sizing, prompt-length control, caching, and routing pay off.
Machine learning has two distinct phases. Training adjusts a model's parameters from data and happens occasionally and expensively. Inference is everything after: feeding a live input through the fixed, trained model to get a prediction or generation. For a deployed product, inference is the phase that matters operationally — it runs on every single user request.
For language models, inference is autoregressive: the model generates one token, appends it to the input, and generates the next, repeating until done. This is why output streams in and why longer responses take longer and cost more — latency and price scale with the tokens produced, and the whole context must be processed each pass.
Because inference is the recurring cost, it is where most AI engineering effort goes: choosing a model sized to the task, controlling prompt and output length, caching, and routing simpler requests to cheaper models. Managing inference well is the difference between an AI feature that is economical at scale and one that is not.
Planoda routes all inference through an AI gateway that meters token spend against a per-workspace budget, so the recurring cost of every request stays governed.
Related terms
- Large Language Model (LLM)A large language model is an AI system trained on vast amounts of text to predict and generate language, enabling it to answer questions, summarize, write, and reason over natural language. LLMs power modern AI assistants and agents. They are probabilistic next-token predictors, which makes them remarkably capable but also prone to confident errors.
- TokenA token is the unit a language model reads and generates — typically a word fragment of a few characters rather than a whole word. Text is split into tokens before processing, and models bill, limit, and reason in tokens, not characters. As a rough guide, one token is about four characters or three-quarters of a word in English.
- Model GatewayA model gateway is a unified layer that sits between an application and one or more AI model providers, routing requests through a single interface. It centralizes provider abstraction, authentication, rate limiting, cost tracking, caching, and failover — so an app can swap models, enforce spending budgets, and observe usage without rewiring every call site.
- TemperatureTemperature is a setting that controls how random a language model's output is. At low temperature the model strongly favors its most likely next token, producing focused, deterministic-leaning text; at high temperature it samples more broadly, producing varied, creative, and less predictable output. It tunes the trade-off between consistency and diversity in generated responses.
- Context WindowA context window is the maximum amount of text — measured in tokens — that a language model can consider at once, spanning its input and its output. Everything the model 'sees' for a given request must fit inside it: the prompt, retrieved data, conversation history, and the answer. When content exceeds the window, the oldest or least relevant parts must be dropped.