The trouble with LLMs is their inconsistency. In one experiment, researchers asked an AI the same question 1,000 times and got 80 completely different answers. This isn’t about creativity—it’s a real technical problem that impacts reliability.
The Root Cause
Researchers at Thinking Machines discovered that LLM inconsistency depends on how many other people are using the system at the same time.
Here’s why: Multiple user requests are processed together in “batches” for efficiency. When the system is busy, it might process 50 requests together; when quiet, maybe 5. The mathematical operations inside the AI give slightly different results depending on batch size—even for identical individual requests.
How to Fix It
Once the problem was understood, researchers developed techniques ensuring individual requests get the same answer regardless of system load:
- Ensure calculations happen in the same sequence regardless of batch size
- Use the same mathematical approaches whether processing 1 or 1,000 requests
- Update how the AI handles memory and calculations
When implemented, these changes make AI responses perfectly consistent.
Implications
For Business: You might get different strategic recommendations for identical scenarios depending on when you ask. Testing and validation become unreliable.
For AI Development: Scientists can’t properly validate research when experiments aren’t repeatable. Performance measurement becomes impossible.
The Trade-Off
Making AI deterministic (reliable) costs about 60% more processing time. The solution requires trade-offs between speed and consistency. Each use case needs scrutiny to decide if the extra cost is worthwhile.
Important Note: This fix ensures consistent responses but doesn’t eliminate hallucination. Accuracy still requires RAG (Retrieval Augmented Generation) capability. Greater consistency actually makes RAG systems better.
Originally published on Brainfood Training.