Evaluating Long Range Dependency Handling in Code Generation LLMs

As language models support larger and larger context sizes, evaluating their ability to make
effective use of that context becomes increasingly important. We analyze the ability of
several code generation models to handle long range dependencies using a suite of multi-step
key retrieval tasks in context windows up to 8k tokens in length. The tasks progressively
increase in difficulty and allow more nuanced evaluation of model capabilities than tests like
the popular needle-in-the-haystack test. We find that performance degrades significantly for
many models (up to 2x) when a function…