> Bron: https://neuralex.nl/en/blog/latency-en-gebruikers > Why can an AI assistant feel slow? Learn what causes latency and how streaming, model routing, progress updates, sensible timeouts and measurement can improve the user experience. [Back to Insights](/en/blog) Infrastructuur ·6 October 2026 ·5 min read # Latency: why users abandon an AI assistant after eight seconds Why can an AI assistant feel slow? Learn what causes latency and how streaming, model routing, progress updates, sensible timeouts and measurement can improve the user experience. Users can find waiting for an AI assistant frustrating very quickly. Eight seconds may feel like a long time, but there is no universal magic threshold: what feels acceptable depends on the user, the task and their expectations. A long silence with no explanation makes people more likely to leave, even if a good answer eventually arrives. Design for perceived speed as well as actual speed. Stream text as soon as it is available, show what is happening during background tasks, and reduce the context or choose a smaller, faster model where appropriate. Test with real users; their experience will tell you whether the wait works in your particular setting. ## Where does the wait come from? Latency is the time between a user’s question and the assistant’s response. That time usually consists of several steps. The model has to process the input and generate an answer. A long conversation or detailed prompt may require more processing. A RAG assistant also has to retrieve and rank documents. An agent may call external tools, process their results and then consult the model again. Network traffic and the way components communicate also affect the wait. A slow document source or tool can delay the whole task, even when the model itself responds quickly. Look at the complete chain: measuring only model speed does not show the full experience. For RAG, it helps to track and improve the search steps too; see [RAG from demo to production](/en/blog/rag-van-demo-naar-productie). ## Actual speed and perceived speed Total waiting time matters, but it is not the only thing that counts. When the user sees a sign of activity quickly, they know their question has been received. If useful parts of the answer appear right away, they can start reading. The same total wait feels longer when the screen is blank and it is unclear whether the assistant is still working. Streaming helps by displaying the answer piece by piece instead of waiting for the entire response to finish. Users see the first words sooner and can begin reading. This is most useful for answers that are generated progressively. Streaming does not automatically reduce total processing time, and it helps less when the assistant must call several tools before it can produce anything useful. During those tasks, provide a brief, honest status update. For example, say that the assistant is searching documents or consulting a source. The status should match what is actually happening: do not invent a precise time estimate if you cannot provide one reliably. For agents, clear updates along the way are part of a reliable design; see also [Retries, timeouts and stalled agents](/en/blog/betrouwbare-agents). ## Reduce the workload and choose the right model A shorter prompt and relevant context can reduce processing. Do not automatically include the entire conversation or every document. Select what is needed for the current question and avoid repeating instructions. In a document assistant, better filtering can keep the model from processing large amounts of irrelevant text. Not every question needs the same model. A router can send simple, well-defined questions to a smaller or faster model, and complex questions to one with more reasoning capacity. The choice must fit the task and quality requirements: gaining speed is of little value if answers become less accurate or incomplete. Read more about [model routing](/en/blog/model-routing-uitgelegd). [Quantization](/en/blog/quantisatie-uitgelegd) can also affect the computing resources needed, depending on the model and how it is used. Frequently asked questions with stable answers can sometimes be served from a cache. This avoids repeating the same work each time. Check that the cached response is still valid and that the question matches closely enough. Personal, current or access-sensitive information requires extra care: never return another user’s answer or an outdated source. ## Timeouts and graceful fallback Sometimes a model call or tool takes longer than expected. Set limits for individual steps and for the overall task so users are not left waiting indefinitely when a step stalls. For agents, bounded retries and a visible status are useful, provided that repeating an action cannot cause it to happen twice. Be clear when an answer is not ready in time. You could offer to simplify the question, show a partial result or explain which step did not succeed. Do not present an answer as reliable when a source or tool is missing. A useful fallback helps the user move forward and makes clear what is still needed. ## Measure and improve Measure both the time until the first visible words and the total time until the answer is complete. For an agent, you can also track retrieval, tool calls and model steps separately. Use percentiles to see how the experience differs between quick requests and those that take longer. An average alone can hide long delays. Combine technical measurements with user testing. Check whether people understand what the assistant is doing, whether they wait or leave, and whether a partial answer is already useful. Then compare one change at a time, such as streaming, shorter context or routing to a different model. Set an acceptable threshold based on the task and the people using the assistant, not on the title. **Short checklist:** - Measure time to first words and total response time. - Show streaming or an honest status while users wait. - Limit prompts and context to what the task requires. - Route simple and complex questions to an appropriate model. - Cache only when the answer and context can be safely reused. - Set timeouts and provide a clear fallback when something is delayed. - Test with real users and review percentiles. Infrastructure ## Does your AI assistant feel fast enough? We help you review streaming, model choice and caching for a pleasant user experience — practical and without invented claims. Curious what that means for your assistant? [Ask your question](/contact) [Read about model routing](/blog/model-routing-uitgelegd) --- Volledige (opgemaakte) versie: https://neuralex.nl/en/blog/latency-en-gebruikers