Editor's note: This article is a part of our series on visualizing the foundations of machine learning.

This article is divided into five parts; they are: • An Example of Tensor Parallelism • Setting Up Tensor Parallelism • Preparing Model for Tensor Parallelism • Train a Model with Tensor Parallelism • Combining Tensor Parallelism with FSDP Tensor paral...

This article is divided into five parts; they are: • Introduction to Fully Sharded Data Parallel • Preparing Model for FSDP Training • Training Loop with FSDP • Fine-Tuning FSDP Behavior • Checkpointing FSDP Models Sharding is a term originally used in...

If you've built chatbots or worked with language models, you're already familiar with how AI systems handle memory within a single conversation.

This article is divided into six parts; they are: • Pipeline Parallelism Overview • Model Preparation for Pipeline Parallelism • Stage and Pipeline Schedule • Training Loop • Distributed Checkpointing • Limitations of Pipeline Parallelism Pipeline para...

Predicting the future has always been the holy grail of analytics.

This article is divided into two parts; they are: • Data Parallelism • Distributed Data Parallelism If you have multiple GPUs, you can combine them to operate as a single GPU with greater memory capacity.

This article is divided into two parts; they are: • Using `torch.

This article is divided into three parts; they are: • Floating-point Numbers • Automatic Mixed Precision Training • Gradient Checkpointing Let's get started! The default data type in PyTorch is the IEEE 754 32-bit floating-point format, also known as s...

If you have an interest in agentic coding, there's a pretty good chance you've heard of

This article is divided into two parts; they are: • What Is Perplexity and How to Compute It • Evaluate the Perplexity of a Language Model with HellaSwag Dataset Perplexity is a measure of how well a language model predicts a sample of text.

If you spend any time working with real-world data, you quickly realize that not everything comes in neat, clean numbers.

This article is divided into three parts; they are: • Training a Tokenizer with Special Tokens • Preparing the Training Data • Running the Pretraining The model architecture you will use is the same as the one created in the

This article is divided into two parts; they are: • Simple RoPE • RoPE for Long Context Length Compared to the sinusoidal position embeddings in the original Transformer paper, RoPE mutates the input tensor using a rotation matrix: $$ \begin{aligned} X...

Large language models (LLMs) like Mistral 7B and Llama 3 8B have shaken the AI field, but their broad nature limits their application to specialized areas.

Agentic coding only feels "smart" when it ships correct diffs, passes tests, and leaves a paper trail you can trust.

…

Building AI applications often requires searching through millions of documents, finding similar items in massive catalogs, or retrieving relevant context for your LLM.

Building newly trained machine learning models that work is a relatively straightforward endeavor, thanks to mature frameworks and accessible computing power.

From daily weather measurements or traffic sensor readings to stock prices, time series data are present nearly everywhere.

This article is divided into four parts; they are: • How Logits Become Probabilities • Temperature • Top- k Sampling • Top- p Sampling When you ask an LLM a question, it outputs a vector of logits.

Machine learning models possess a fundamental limitation that often frustrates newcomers to natural language processing (NLP): they cannot read.

Data leakage is an often accidental problem that may happen in machine learning modeling.

…

The increasing sophistication of cyber threats calls for a systemic change in the way we defend ourselves against them.

When building machine learning models to classify imbalanced data — i.

Validating machine learning models requires careful testing on unseen data to ensure robust, unbiased estimates of their performance.

Deep neural networks have drastically evolved over the years, overcoming common challenges that arise when training these complex models.

This article provides a summary of and commentary on the recent paper

Developing machine learning systems entails a well-established lifecycle, consisting of a series of stages from data preparation and preprocessing to modeling, validation, deployment to production, and continuous maintenance.

Extreme gradient boosting ( XGBoost ) is one of the most prominent machine learning techniques used not only for experimentation and analysis but also in deployed predictive solutions in industry.

Experimenting, fine-tuning, scaling, and more are key aspects that machine learning development workflows thrive on.

Data merging is the process of combining data from different sources into a unified dataset.

When working with machine learning on structured data, two algorithms often rise to the top of the shortlist: random forests and gradient boosting .

In this article, you will learn: • The fundamental difference between traditional regression, which uses single fixed values for its parameters, and Bayesian regression, which models them as probability distributions.

Working with time series data often means wrestling with the same patterns over and over: calculating moving averages, detecting spikes, creating features for forecasting models.

When you have a small dataset, choosing the right machine learning model can make a big difference.