Understanding the DistilBart Model and ROUGE Metric
This post is in two parts; they are: • Understanding the Encoder-Decoder Architecture • Evaluating the Result of Summarization using ROUGE DistilBart is a “distilled” version of the BART model, a powerful sequence-to-sequence model for natural language generation, translation, and comprehension.
This post is divided into five parts; they are: • From a Full Transformer to a Decoder-Only Model • Building a Decoder-Only Model • Data Preparation for Self-Supervised Learning • Training the Model • Extensions The transformer model originated as a sequence-to-sequence (seq2seq) model that converts an input sequence into…
This post is divided into four parts; they are: • Why Attnetion Matters: Limitations of Basic Seq2Seq Models • Implementing Seq2Seq Model with Attention • Training and Evaluating the Model • Using the Model Traditional seq2seq models use an encoder-decoder architecture where the encoder compresses the input sequence into a…
This chapter is divided into nine parts; they are: • Reading Logits from a Model • Greedy Decoding • Temperature Sampling • Top-$k$ Sampling • Nucleus Sampling • Repetition Penalties • Beam Search • Stop Conditions • Structured Output Constraints The model returns a vector of logits for every position…