Categories: AI/ML Research

Linear Layers and Activation Functions in Transformer Models

This post is divided into three parts; they are: • Why Linear Layers and Activations are Needed in Transformers • Typical Design of the Feed-Forward Network • Variations of the Activation Functions The attention layer is the core function of a transformer model.

Using Activation Functions in Deep Learning Models

A deep learning model in its simplest form are layers of perceptrons connected in tandem. Without any activation functions, they are just matrix multiplications with limited power, regardless how many of them. Activation is the magic why neural network can be an approximation to a wide variety of non-linear function.…

February 15, 2023

In "AI/ML Research"

Creating Custom Layers and Loss Functions in PyTorch

Creating custom layers and loss functions in

February 12, 2025

In "AI/ML Research"

How to Grid Search Hyperparameters for PyTorch Models

The “weights” of a neural network is referred as “parameters” in PyTorch code and it is fine-tuned by optimizer during training. On the contrary, hyperparameters are the parameters of a neural network that is fixed by design and not tuned by training. Examples are the number of hidden layers and…

February 10, 2023

In "AI/ML Research"

AI Generated Robotic Content