Category: AI/ML Research

If you spend any time working with real-world data, you quickly realize that not everything comes in neat, clean numbers.

This article is divided into three parts; they are: • Training a Tokenizer with Special Tokens • Preparing the Training Data • Running the Pretraining The model architecture you will use is the same as the one created in the

This article is divided into two parts; they are: • Simple RoPE • RoPE for Long Context Length Compared to the sinusoidal position embeddings in the original Transformer paper, RoPE mutates the input tensor using a rotation matrix: $$ begin{aligned} X_{n,i} &= X_{n,i} cos(ntheta_i) – X_{n,frac{d}{2}+i} sin(ntheta_i) \ X_{n,frac{d}{2}+i} &= X_{n,i} sin(ntheta_i) + X_{n,frac{d}{2}+i} cos(ntheta_i) …

Read more “Rotary Position Embeddings for Long Context Length”

Agentic coding only feels “smart” when it ships correct diffs, passes tests, and leaves a paper trail you can trust.

Large language models (LLMs) like Mistral 7B and Llama 3 8B have shaken the AI field, but their broad nature limits their application to specialized areas.

Building AI applications often requires searching through millions of documents, finding similar items in massive catalogs, or retrieving relevant context for your LLM.

From daily weather measurements or traffic sensor readings to stock prices, time series data are present nearly everywhere.

Building newly trained machine learning models that work is a relatively straightforward endeavor, thanks to mature frameworks and accessible computing power.

This article is divided into four parts; they are: • How Logits Become Probabilities • Temperature • Top- k Sampling • Top- p Sampling When you ask an LLM a question, it outputs a vector of logits.

Machine learning models possess a fundamental limitation that often frustrates newcomers to natural language processing (NLP): they cannot read.