#ai #llm Created at 190223 # [Anonymous feedback](https://www.admonymous.co/louis030195) # [[Epistemic status]] #shower-thought #non-biological Last modified date: 2023-02-19 Commit: 0 # Related # TODO > [!TODO] TODO # Flash attention The paper "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" proposes FlashAttention, an IO-aware exact attention algorithm that uses tiling to reduce the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM, achieving faster and more memory-efficient training of transformers. Key insights and lessons learned from the paper include: - FlashAttention reduces the number of memory reads/writes between GPU HBM and on-chip SRAM to achieve faster and more memory-efficient training of transformers. - FlashAttention also performs better than existing approximate attention methods in terms of both speed and accuracy. - The proposed algorithm is shown to be optimal for a range of SRAM sizes, and can be extended to block-sparse attention. Three questions for the authors: - How does FlashAttention compare to other IO-aware attention algorithms that have been proposed in the literature? - Are there any limitations to the proposed algorithm in terms of the sequence length or dataset size that it can handle? - How do you envision FlashAttention being used in practical applications, and what are the potential benefits and challenges of using it in such contexts? Three suggestions for related topics or future research directions are: - Investigating the performance of FlashAttention on different transformer architectures and for different natural language processing tasks. - Exploring the use of FlashAttention in other domains beyond natural language processing, such as computer vision or speech recognition. - Studying the energy efficiency of FlashAttention and how it can be optimized for deployment on mobile or edge devices. Five relevant references from the field of study of the paper: - Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998-6008). - Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020). Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451. - Child, R., Gray, S., Radford, A., & Sutskever, I. (2019). Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509. - Roy, A. G., Grabski, F., & Collobert, R. (2020). Efficient content-based sparse attention with routing transformers. arXiv preprint arXiv:2003.05997. - Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontañón, S., ... & Smola, A. J. (2021). Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062.✏