Tag: research

Predicting the future has always been the holy grail of analytics.

This article is divided into two parts; they are: • Data Parallelism • Distributed Data Parallelism If you have multiple GPUs, you can combine them to operate as a single GPU with greater memory capacity.

This article is divided into two parts; they are: • Using `torch.

This article is divided into three parts; they are: • Floating-point Numbers • Automatic Mixed Precision Training • Gradient Checkpointing Let’s get started! The default data type in PyTorch is the IEEE 754 32-bit floating-point format, also known as single precision.

If you have an interest in agentic coding, there’s a pretty good chance you’ve heard of

This article is divided into two parts; they are: • What Is Perplexity and How to Compute It • Evaluate the Perplexity of a Language Model with HellaSwag Dataset Perplexity is a measure of how well a language model predicts a sample of text.

If you spend any time working with real-world data, you quickly realize that not everything comes in neat, clean numbers.

This article is divided into three parts; they are: • Training a Tokenizer with Special Tokens • Preparing the Training Data • Running the Pretraining The model architecture you will use is the same as the one created in the

This article is divided into two parts; they are: • Simple RoPE • RoPE for Long Context Length Compared to the sinusoidal position embeddings in the original Transformer paper, RoPE mutates the input tensor using a rotation matrix: $$ begin{aligned} X_{n,i} &= X_{n,i} cos(ntheta_i) – X_{n,frac{d}{2}+i} sin(ntheta_i) \ X_{n,frac{d}{2}+i} &= X_{n,i} sin(ntheta_i) + X_{n,frac{d}{2}+i} cos(ntheta_i) …

Read more “Rotary Position Embeddings for Long Context Length”

Agentic coding only feels “smart” when it ships correct diffs, passes tests, and leaves a paper trail you can trust.