Hi,
I am trying to understand the difference in the cost of producing a single input token vs output token.
Based on some articles, I came to the following conclusion:
- Input tokens processing scales quadratically, there’s no way around it, you have to compute attention (K and V by passing it through encoders) for each token with each other token.
- Output tokens scale linearly thanks to using KV cache (otherwise quadratic without KV cache, as linear tokens do) which is what everyone seems to do when hosting these models. I.e. you trade compute for memory, and try to be clever about how you store and compute all these stuff (like sparse, cache eviction…). I believe this is simply an empirically practical way around the quadratic scaling.
- Therefore, realistically, it’s really the memory capacity and bandwidth being the problem for output tokens rather than raw FLOPS. KV cache grows linearly with the sequence length regardless input/output tokens ratio.
- I am confused here why input tokens are almost universally cheaper - 2 to 5 times - to output tokens. Also, is the evidence that providers are pricing things linearly for memory being the problem here (as they do now, you pay the same price for 11th token as for 10001)? If not and they were FLOPS bounded, I would expect the providers to price stuff quadratically, not linearly. Or is it just for client-pricing simplicity/marketing?
I want to be able to estimate the cost of processing a single token, and I cannot wrap my head around this. I theoretically estimated based on GPU rent price and separately based on power consumption (assuming some utilization such as 10%), and I believe I somehow need to differentiate between input/output tokens here.
In one tweet from LeptonAI who hosts these LLM, I also saw that there are usually 3-10 times more input tokens than output tokens. Again, if input tokens dominate the sequence and it was FLOPS the issue, I would expect to reflect that in the pricing. Not sure what role it plays in these calculations so far.
Any help is appreciated, thanks!
One has to run the whole Transformer once for an output token. So if we ignore the difference between runs, the complexity would be the number of output tokens multiplied by the cost of a single run.
Now, what is the cost of the run, and how does it vary?
The context to that run is sometimes all inputs, sometimes all inputs and almost all outputs, and sometimes something in between. If we disregard the fact that output tokens are only included in contexts of some of the runs, then input and output tokens should contribute similarly to the cost of the typical run (although, in reality output tokens contribute less, because a typical output token only participates in approximately half of those runs, with early output tokens participating in almost all runs and late output tokens participating in almost no runs). I am not sure how things like KV caching affect that. So a typical output token would contribute about a half of the contribution of an input token to the cost of a single run.
(I would assume that a clever caching schema would eliminate or reduce the difference between input and output tokens in the sense of their contribution to the cost of a single run.)
But the number of runs is exactly the number of output tokens, so the overall cost seems to grow much faster when the number of output tokens grows.
No, the typical pricing tends to be the same (unless one can score a discounted plan).
But some queries might bring more profit, some queries might bring less profit, and some queries might be at a loss, depending on the query. The provider is OK with non-uniform margin on the queries and with dependency of that margin on the number of input tokens, the number of output tokens, and so on. They care about average payments vs average costs.
(Similarly, with flat monthly fee in interfaces like ChatGPT, they make more money off light users, and might even be OK with taking a loss with the heaviest users as long as there are not too many of them.)