#llm Created at 160823 # [Anonymous feedback](https://www.admonymous.co/louis030195) # [[Epistemic status]] #shower-thought Last modified date: 160823 Commit: 0 # Related # Speeding up llm response time ## Sequence scheduling Sequence scheduling is a technique that can help speed up LLM response times by intelligently grouping queries with similar response lengths, reducing computational waste and improving inference throughput[4]. This approach takes advantage of the fact that some queries may have shorter response times than others, allowing for more efficient use of computational resources. To implement sequence scheduling, you can follow these steps: 1. Analyze the response lengths of your queries: Determine the typical response lengths for different types of queries in your application. This information can be used to group queries with similar response lengths together. 2. Group queries with similar response lengths: Create batches of queries with similar response lengths, allowing for more efficient use of computational resources during inference. 3. Schedule the batches for inference: Process the batches of queries in a way that maximizes the utilization of computational resources. This may involve processing batches with shorter response lengths first, or interleaving batches with different response lengths to balance resource usage. 4. Monitor and adjust the scheduling strategy: Continuously monitor the performance of your sequence scheduling strategy and make adjustments as needed to optimize LLM response times. By implementing sequence scheduling, you can potentially reduce the latency of your LLM responses and improve the overall performance of your AI application. Citations: [1] https://betterprogramming.pub/speed-up-llm-inference-83653aa24c47 [2] https://matt-rickard.com/a-hackers-guide-to-llm-optimization [3] https://towardsdatascience.com/overcoming-the-limitations-of-large-language-models-9d4e92ad9823 [4] https://arxiv.org/pdf/2305.13144.pdf [5] https://medium.com/@pankaj_pandey/optimizing-latencies-in-text-generation-and-llm-models-3767844b718c [6] https://medium.com/@pankaj_pandey/the-possible-strategies-for-cost-effective-large-language-model-optimization-50e7edad2262 [7] https://research.aimultiple.com/large-language-model-evaluation/ [8] https://arxiv.org/abs/2305.13144 [9] https://www.mlexpert.io/prompt-engineering/faster-llm-inference [10] https://medium.com/@sureshbhojwani001/supercharging-language-models-strategies-for-optimizing-llm-and-gpt-f5cd59e706ca [11] https://research.aimultiple.com/future-of-large-language-models/ [12] https://www.anyscale.com/blog/continuous-batching-llm-inference [13] https://www.mlexpert.io/prompt-engineering/llm-optimization [14] https://www.vantage.sh/blog/optimize-large-language-model-costs [15] https://openreview.net/forum?id=NiEtU7blzN [16] https://discuss.huggingface.co/t/combinatorial-optimization-with-llms-transformers/39623 [17] https://youtube.com/watch?v=0C8QoCz4zpU [18] https://research.aimultiple.com/llm-fine-tuning/ [19] https://www.forbes.com/sites/robtoews/2023/02/07/the-next-generation-of-large-language-models/?sh=24ae687718db [20] https://developer.nvidia.com/blog/efficiently-scale-llm-training-across-a-large-gpu-cluster-with-alpa-and-ray/ [21] https://www.reddit.com/r/LocalLLaMA/comments/13liju9/local_model_response_time/ [22] https://towardsdatascience.com/reprompting-automated-problem-solving-optimization-for-llms-53a0a2f9db38 [23] https://arxiv.org/abs/2210.11610 [24] https://lilianweng.github.io/posts/2023-01-10-inference-optimization/ [25] https://towardsdatascience.com/how-to-speed-up-training-for-large-language-models-81ffb30c36b2