Long Context Scenario
Long context scenarios in large language models (LLMs) address the challenge of processing and understanding significantly long input sequences, exceeding the typical context window limitations. Current research focuses on improving LLM performance in these scenarios through techniques like prompt compression (e.g., using dual-slope allocation and low-rank decomposition of key-value caches), enhanced graph flattening methods for structured data, and mitigating position bias through attention weight adjustments. These advancements aim to improve both the accuracy and efficiency of LLMs on tasks requiring extensive contextual information, impacting fields like question answering, document summarization, and code generation.
Papers
LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios
Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Xiangju Lu, Junmin Zhu, Wei Zhang
LongSafetyBench: Long-Context LLMs Struggle with Safety Issues
Mianqiu Huang, Xiaoran Liu, Shaojun Zhou, Mozhi Zhang, Chenkun Tan, Pengyu Wang, Qipeng Guo, Zhe Xu, Linyang Li, Zhikai Lei, Linlin Li, Qun Liu, Yaqian Zhou, Xipeng Qiu, Xuanjing Huang