How LLMs Choose Their Words: A Practical Walk-Through of Logits, Softmax and Sampling
This article is divided into four parts; they are: • How Logits Become Probabilities • Temperature • Top- k Sampling • Top- p Sampling When you ask an LLM a question, it outputs a vector of logits.
As language models grow ever larger, so do their vocabularies. This has shifted the memory footprint of LLMs during training disproportionately to one single layer: the cross-entropy in the loss computation. Cross-entropy builds up a logit matrix with entries for each pair of input tokens and vocabulary items and, for…
Like many PyTorch users, you may have heard great things about JAX — its high performance, the elegance of its functional programming approach, and its powerful, built-in support for parallel computation. However, you may have also struggled to find what you need to get started: a straightforward, easy-to-follow tutorial to…
This post is divided into seven parts; they are: - Core Text Generation Parameters - Experimenting with Temperature - Top-K and Top-P Sampling - Controlling Repetition - Greedy Decoding and Sampling - Parameters for Specific Applications - Beam Search and Multiple Sequences Generation Let's pick the GPT-2 model as an…