Long Context LLM

Long-context LLMs aim to overcome the limitations of traditional LLMs by processing significantly longer input sequences, enabling more comprehensive understanding and generation of text. Current research focuses on improving efficiency through techniques like optimized attention mechanisms (e.g., sparse attention, hierarchical pruning), efficient key-value cache management (e.g., quantization, eviction strategies), and data-driven approaches to enhance long-context performance during training and fine-tuning. These advancements are crucial for enabling applications requiring the processing of extensive textual data, such as complex question answering, document summarization, and large-scale information retrieval, while addressing the computational challenges associated with increased context length.

Papers

June 19, 2024

Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective
Meizhi Zhong, Chen Zhang, Yikun Lei, Xikai Liu, Yan Gao, Yao Hu, Kehai Chen, Min Zhang
Long Text Attention Pattern Long Context LLM Attention Uncertainty

June 17, 2024

June 16, 2024

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han
Self Attention Inference Latency Long Context Large Language Model KV Cache Long Context LLM QueST Sparsity Aware

June 14, 2024

A Training-free Sub-quadratic Cost Transformer Model Serving Framework With Hierarchically Pruned Attention
Heejun Lee, Geon Park, Youngwan Lee, Jaduk Suh, Jina Kim, Wonyoung Jeong, Bumsik Kim, Hyemin Lee, Myeongjae Jeon, Sung Ju Hwang
Sparse Attention Long Context LLM Context Sequence Quadratic Attention Attention Pruning

June 2, 2024

LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models
Liang Zhao, Tianwen Wei, Liang Zeng, Cheng Cheng, Liu Yang, Peng Cheng, Lijie Wang, Chenxia Li, Xuejie Wu, Bo Zhu, Yimeng Gan, Rui Hu, Shuicheng Yan, Han Fang, Yahui Zhou
Long Context Context Length Long Context Large Language Model Long Context LLM Long Context Language Training Recipe

May 30, 2024

Is In-Context Learning Sufficient for Instruction Following in LLMs?
Hao Zhao, Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion
Medical LLM Context Learning Human Instruction Context Example Instruction Fine Tuning Long Context LLM

May 24, 2024

Are Long-LLMs A Necessity For Long-Context Tasks?
Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Yujia Zhou, Xu Chen, Zhicheng Dou
Long Context Community Need Long Context Task Long Context LLM Long Context Benchmark

April 9, 2024

Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks
Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, Kai Chen
Large Language Model Text Generation Long Text Long Context LLM Test Set Text Benchmark Long Input Scroll Benchmark

January 31, 2024

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami
Long Context LLM Quantization Level Cache Quantization Input Quantization

September 27, 2023

Effective Long-Context Scaling of Foundation Models
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, Hao Ma
Foundation Model Long Context Long Text Instruction Data Long Context Task Long Context LLM

Long Context LLM

Papers

Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective

Can Many-Shot In-Context Learning Help LLMs as Evaluators? A Preliminary Empirical Study

SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

A Training-free Sub-quadratic Cost Transformer Model Serving Framework With Hierarchically Pruned Attention

LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models

Is In-Context Learning Sufficient for Instruction Following in LLMs?

Are Long-LLMs A Necessity For Long-Context Tasks?

Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

Effective Long-Context Scaling of Foundation Models