{ "cells": [ { "cell_type": "markdown", "id": "751f4b48", "metadata": {}, "source": [ "\n", " \"Open\n", "" ] }, { "cell_type": "markdown", "id": "4415ef93c445cad1", "metadata": {}, "source": [ "# Image analysis\n", "\n", "
\n", " Table of Contents \n", "\n", "1. [Introduction](#introduction)\n", "2. [Data management](#dm)\n", "3. [Data augmentation](#data_aug)\n", "\n", "
\n", "\n", "
\n", " I. Introduction \n", "
\n", " \n", "In previous lessons, we've worked extensively with **tabular data**—structured datasets where each row represents an observation and each column represents a feature. These datasets are typically easy to represent and analyze using tools like `pandas`, `scikit-learn`, and various statistical models.\n", "\n", "However, **images** are fundamentally different in both structure and content. Instead of rows and columns of explicitly labeled features, an image is a grid of pixel values—typically in 2D for grayscale or 3D for color (height × width × channels). For example, a color image of size 224×224 pixels with 3 color channels (RGB) contains over 150,000 raw values, none of which are labeled with human-interpretable features like \"age\" or \"income\".\n", "\n", "This difference leads to several important challenges:\n", "\n", "- **High Dimensionality**: Images contain many more features (pixels) than typical tabular datasets, which increases computational complexity and the risk of overfitting.\n", "- **Spatial Structure**: Nearby pixels in an image are often related, forming edges, textures, and patterns. Tabular models generally do not capture such local dependencies.\n", "- **Semantic Gap**: The relationship between raw pixels and meaningful concepts (like a cat, a face, or a traffic sign) is complex and non-linear, requiring sophisticated models to bridge this gap.\n", "\n", "Because of these challenges, we need **image-specific models**, most notably **Convolutional Neural Networks (CNNs)**, which are designed to exploit the spatial and hierarchical structure of images. These models learn to recognize patterns such as edges, shapes, and eventually objects, by processing images in layers of increasing abstraction.\n", "\n", "In this module, we will explore how to work with image data. To do so we will use again `PyTorch`." ] }, { "cell_type": "markdown", "id": "d05ec0bb", "metadata": {}, "source": [ "## Memory Constraints with Image Datasets\n", "\n", "Unlike tabular data, where datasets can often be loaded entirely into memory (e.g., a CSV with thousands of rows and columns), image datasets present significant memory challenges. Images are high-dimensional data: a single 224x224 RGB image contains 224 * 224 * 3 ≈ 150,000 values (pixels), each typically stored as a float32 (4 bytes), resulting in about 600 KB per image. For a dataset of 1 million images, this would require approximately 600 GB of RAM, which exceeds the capacity of most systems.\n", "\n", "This memory limitation fundamentally changes how we work with image data:\n", "- **Batch processing**: We use data loaders that load images in small batches (e.g., 32-128 images at a time) rather than the entire dataset.\n", "- **On-the-fly transformations**: Images are often resized, normalized, or augmented as they are loaded, avoiding the need to store preprocessed versions.\n", "- **Iterative training**: Models are trained using iterators that fetch batches sequentially, enabling training on datasets larger than available memory.\n", "- **Efficient storage**: Techniques like lazy loading from disk or using compressed formats help manage large datasets without overwhelming memory." ] }, { "cell_type": "markdown", "id": "7c5843cd", "metadata": {}, "source": [ "
\n", " II. Data management \n", "
\n", "\n", "When training deep learning models with images, we don't feed the whole dataset at once—unlike traditional machine learning with tabular data. Instead, we process images in **batches**: small groups of samples processed together.\n", "\n", "A **batch** is a collection of samples (e.g., images and their labels) processed together in a single forward and backward pass. For example, instead of training on 1,000,000 images at once, we might process 32 images, update weights, then move to the next 32.\n", "\n", "#### Why Batching?\n", "\n", "- **Memory Efficiency**: Batches make full dataset training feasible by processing only a subset at a time, fitting within GPU/CPU memory constraints (recall: 1M images = 600+ GB).\n", "- **GPU Parallelism**: Modern GPUs excel at parallel processing; batches leverage this by applying the same operation to many images simultaneously.\n", "- **Gradient Stability**: Computing gradients on a batch provides more stable estimates than single-sample updates, reducing training noise and improving convergence.\n", "\n", "#### Key Relationship\n", "\n", "For a dataset with $N$ samples and batch size $B$:\n", "- **Iterations per epoch** = $\\lceil \\frac{N}{B} \\rceil$ (number of batches)\n", "- **Total iterations** = Iterations per epoch × Number of epochs\n", "\n", "Example: 10,000 images with batch size 32 = 313 iterations per epoch (each epoch processes all 10,000 images in 313 batches).\n", "\n", "### Terminology\n", "\n", "- **Batch Size**: The number of samples in a single batch (e.g., 32, 64, or 128). This is a critical hyperparameter—larger batches use more memory but train faster; smaller batches are more memory-efficient but noisier.\n", "- **Epoch**: One complete pass through the entire dataset.\n", "- **Iteration**: One update step; corresponds to processing one batch.\n" ] }, { "cell_type": "markdown", "id": "986809d1", "metadata": {}, "source": [ "#### Example\n", "\n", "In the following code, we will create a PyTorch MLP model similar to the ones discussed in the previous section. We will attempt to train it using the entire image dataset without batching, as you did in the previous leasons. To do so we will use a test image dataset: CIFAR-10.\n", "\n", "**CIFAR-10** \n", "\n", "CIFAR-10 is a widely-used dataset for image classification tasks, containing 60,000 32x32 color images across 10 different classes (such as airplanes, cars, birds, etc.). Thanks to PyTorch and its torchvision library, we can easily access this dataset along with many other standard test datasets commonly used in machine learning research, you can explore the [complete list of available datasets in torchvision](https://pytorch.org/vision/stable/datasets.html).\n", "\n", "![CIFAR10 example](https://docs.pytorch.org/tutorials/_images/cifar10.png)\n", "\n", "To load an existing datset we need to use *datasets* and *DataLoaders*. These are two important objects for `PyTorch` and we will explain them in more detaill in a few minutes. For now we only need to untendertand that are how we load images to use it for " ] }, { "cell_type": "code", "execution_count": null, "id": "ed8730df", "metadata": {}, "outputs": [], "source": [ "import torch\n", "import torchvision\n", "import torchvision.transforms as transforms\n", "\n", "# Load CIFAR-10 dataset\n", "transform = transforms.ToTensor()\n", "trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform) # download must be True if is the first time you execute this notebook\n", "\n", "# Load the entire dataset into memory\n", "trainloader = torch.utils.data.DataLoader(trainset, batch_size=len(trainset), shuffle=False)\n", "images, labels = next(iter(trainloader))\n", "\n", "# Flatten images for MLP input\n", "images = images.view(images.size(0), -1) # Shape: (50000, 3072)\n", "print(images.shape)" ] }, { "cell_type": "markdown", "id": "d2d63f15", "metadata": {}, "source": [ "The output shape **(50000, 3072)** represents:\n", "- **50000**: The number of images in the CIFAR-10 training dataset\n", "- **3072**: The total number of pixels (features) per image. Since CIFAR-10 contains 32×32 RGB images, each image has 32 × 32 × 3 = 3,072 pixel values (three channels for Red, Green, and Blue). By flattening the images, we convert each 3D image (32, 32, 3) into a 1D vector of 3,072 features suitable for MLP training.\n", "\n", "So now we will define a model as the one we used for tabular data. \n", "\n", "**Remember**:\n", "\n", "- If we have 3072 input features the model must have this amount of input features. \n", "- Also take into account that if the task is more complex we need a bigger net. Therefore if we have a problem of around 4000 features we need a larger model!\n", "- Lastly, remember that the ouput must correspond to the number of classes in a classification task.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "6fa45003", "metadata": {}, "outputs": [], "source": [ "import torch\n", "import torch.nn as nn\n", "\n", "# Model definition. Here we have to define the layers and the activation functions between them.\n", "model = nn.Sequential(\n", " nn.Linear(3072, 8000),\n", " nn.ReLU(),\n", " nn.Linear(8000, 8000),\n", " nn.ReLU(),\n", " nn.Linear(8000, 3000),\n", " nn.ReLU(),\n", " nn.Linear(3000, 1000),\n", " nn.ReLU(),\n", " nn.Linear(1000, 500),\n", " nn.ReLU(),\n", " nn.Linear(500, 100),\n", " nn.ReLU(),\n", " nn.Linear(100, 10)\n", ")" ] }, { "cell_type": "markdown", "id": "783dfb08", "metadata": {}, "source": [ "
\n", " WARNING: only execute this in a colab session!\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "a613b8a2", "metadata": {}, "outputs": [], "source": [ "model(images)" ] }, { "cell_type": "markdown", "id": "7faf8d78", "metadata": {}, "source": [ "It's important to note that **CIFAR-10 is actually a relatively simple and small dataset** compared to real-world image datasets. However, even with this modest dataset, loading the entire 50,000 images (approximately 600 MB of data) into memory and make the prediction can be problematic for training. This demonstrates why **batching is essential** in machine learning: it allows us to process data in manageable chunks, reducing memory requirements and enabling training on much larger datasets. In practice, modern datasets like ImageNet contain millions of images that would be impossible to load all at once into memory.\n", "\n", "`PyTorch` is heavely built around the idea of batching. Let's see what we have to change in the previous code to use it:" ] }, { "cell_type": "code", "execution_count": null, "id": "94692e0a", "metadata": {}, "outputs": [], "source": [ "BATCH_SIZE = 10\n", "\n", "# Load CIFAR-10 dataset\n", "transform = transforms.ToTensor()\n", "trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform) # download must be True if is the first time you execute this notebook\n", "\n", "# Load the entire dataset into memory\n", "trainloader = torch.utils.data.DataLoader(trainset, batch_size=BATCH_SIZE, shuffle=False)\n", "images, labels = next(iter(trainloader))\n", "\n", "# Flatten images for MLP input\n", "images = images.view(images.size(0), -1) # Shape: (50000, 3072)\n", "print(images.shape)" ] }, { "cell_type": "code", "execution_count": null, "id": "67e9dd1f", "metadata": {}, "outputs": [], "source": [ "model(images)" ] }, { "cell_type": "markdown", "id": "2951dd1b", "metadata": {}, "source": [ "We can see that now the prediction has been without delay of any kind. We can use this approach to train and to evaluate a complex machine learning model. Let's dive into the what it is a **dataset** and a **dataloader** in `PyTorch`.\n", "\n", "## PyTorch Dataset and DataLoader\n", "\n", "### Dataset\n", "\n", "A **Dataset** in PyTorch is an abstract class that represents a collection of data samples. It defines how to access individual data samples and their corresponding labels. PyTorch provides built-in datasets like CIFAR-10, ImageNet, MNIST, etc., through `torchvision.datasets`. \n", "\n", "### DataLoader\n", "\n", "A **DataLoader** is a wrapper around a Dataset that handles:\n", "- **Batching**: Groups multiple samples into batches for efficient processing\n", "- **Shuffling**: Randomly reorders samples during training (improves generalization)\n", "- **Parallel loading**: Loads data using multiple worker processes for faster I/O\n", "- **Automatic padding/collation**: Handles variable-sized samples\n", "\n", "A **DataLoader** must return the input data and the ground truth we want to predict.\n", "\n", "### Why use them?\n", "\n", "1. **Efficiency**: Batching enables vectorized operations on GPUs\n", "2. **Scalability**: Can handle datasets larger than available RAM\n", "3. **Convenience**: Handles indexing, batching, and shuffling automatically\n", "4. **Best practices**: Follows PyTorch conventions for reproducible research\n", "\n", "Previously we used already these two classes in the following code:\n", "\n", "```python\n", "trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)\n", "trainloader = torch.utils.data.DataLoader(trainset, batch_size=BATCH_SIZE, shuffle=False)\n", "```\n", "\n", "Each dataset has its unique parameters. Each dataloader has three main parameters:\n", "\n", "- `batch_size`: Number of samples per batch\n", "- `shuffle`: Whether to shuffle data (typically `True` for training, `False` for validation/testing)\n", "- `num_workers`: Number of parallel processes for data loading\n", "\n", "Let's play a little bit with this objects. We can use ``next(iter())`` to obtain the following element of the trainset so we see how they are." ] }, { "cell_type": "code", "execution_count": null, "id": "c3a496f7", "metadata": {}, "outputs": [], "source": [ "res = next(iter(trainset))\n", "res" ] }, { "cell_type": "markdown", "id": "0134b38d", "metadata": {}, "source": [ "How many elements there are?" ] }, { "cell_type": "code", "execution_count": null, "id": "46c074a8", "metadata": {}, "outputs": [], "source": [ "len(res)" ] }, { "cell_type": "markdown", "id": "2d751c53", "metadata": {}, "source": [ "What are they?" ] }, { "cell_type": "code", "execution_count": null, "id": "0a8b0237", "metadata": {}, "outputs": [], "source": [ "print(res[0].shape)\n", "print(res[1])" ] }, { "cell_type": "markdown", "id": "2d71be26", "metadata": {}, "source": [ "An image and a class! Lets see the image:" ] }, { "cell_type": "code", "execution_count": null, "id": "20ca6214", "metadata": {}, "outputs": [], "source": [ "from torchvision.transforms.functional import to_pil_image\n", "\n", "to_pil_image(res[0])" ] }, { "cell_type": "markdown", "id": "1fa915fc", "metadata": {}, "source": [ "In PyTorch, images are typically represented as **tensors** with a specific shape convention:\n", "\n", "**Single Image Shape: (C, H, W)**\n", "- **C**: Number of channels (3 for RGB, 1 for grayscale)\n", "- **H**: Height in pixels\n", "- **W**: Width in pixels\n", "\n", "Example: A standard RGB image has shape `(3, 32, 32)` - 3 color channels, 32 pixels tall, 32 pixels wide.\n", "\n", "Let's do the same but for dataloader\n" ] }, { "cell_type": "code", "execution_count": null, "id": "b0f510e0", "metadata": {}, "outputs": [], "source": [ "res = next(iter(trainloader))\n", "res" ] }, { "cell_type": "code", "execution_count": null, "id": "f4b75f53", "metadata": {}, "outputs": [], "source": [ "print(len(res))\n", "print(res[0].shape)\n", "print(res[1])\n" ] }, { "cell_type": "code", "execution_count": null, "id": "73b90996", "metadata": {}, "outputs": [], "source": [ "batch, labels = res\n", "to_pil_image(batch[0])" ] }, { "cell_type": "markdown", "id": "24c12be9", "metadata": {}, "source": [ "## ImageFolder Dataset\n", "\n", "The `ImageFolder` class from `torchvision.datasets` provides a convenient way to load images organized in a directory structure. It automatically infers class labels from the folder hierarchy:\n", "\n", "```\n", "root/\n", " class1/\n", " img1.jpg\n", " img2.jpg\n", " class2/\n", " img3.jpg\n", " img4.jpg\n", "```\n", "\n", "Each subfolder represents a class, and images within are assigned that class label. This is ideal for standard image classification datasets where your data is organized in a folder-per-class structure." ] }, { "cell_type": "markdown", "id": "c08195e1", "metadata": {}, "source": [ "
\n", " III. Data augmentation\n", "
\n", "\n", "\n", "### The Transform API\n", "\n", "PyTorch's **Transform API** (via `torchvision.transforms`) provides a composable way to apply transformations to images during the data loading pipeline. The key component is `transforms.Compose()`, which chains multiple transformations together:" ] }, { "cell_type": "code", "execution_count": null, "id": "af48e377", "metadata": {}, "outputs": [], "source": [ "transform = transforms.Compose([\n", " transforms.Resize((128, 128)),\n", " transforms.RandomRotation(180)\n", "])" ] }, { "cell_type": "markdown", "id": "f7ce6a3b", "metadata": {}, "source": [ "Each transformation is applied **sequentially** to the data as it's loaded, making it efficient and flexible. " ] }, { "cell_type": "code", "execution_count": null, "id": "917f66e0", "metadata": {}, "outputs": [], "source": [ "batch_transformed = transform(batch)\n", "to_pil_image(batch_transformed[0])" ] }, { "cell_type": "markdown", "id": "ac64b65c", "metadata": {}, "source": [ "Many transformation exist in [`PyTorch`](https://docs.pytorch.org/vision/0.8/transforms.html). This transformations usually are used for two reasons: preprocess the image (change the size for example) and **data augmentation**." ] }, { "cell_type": "markdown", "id": "a5968d37", "metadata": {}, "source": [ "### Data Augmentation: A Superpower for Training\n", "\n", "**Data Augmentation** is the technique of artificially increasing the size and diversity of your training dataset by applying random transformations (rotations, flips, crops, color jittering, etc.) to images. This is one of the most effective regularization techniques in deep learning because:\n", "\n", "1. **Improves Generalization**: By training on diverse versions of the same image, the model learns more robust features and is less likely to overfit.\n", "\n", "2. **Simulates Real-World Variations**: In practice, images come from different angles, lighting conditions, and perspectives. Augmentation helps the model handle these variations.\n", "\n", "3. **Effective Data Multiplication**: You can effectively create hundreds or thousands of variations from a small dataset without needing to collect new data.\n", "\n", "Example augmentations:\n", "- `RandomRotation(30)`: Random rotations up to 30 degrees\n", "- `RandomHorizontalFlip(p=0.5)`: 50% chance of horizontal flip\n", "- `ColorJitter(brightness=0.2)`: Random brightness changes\n", "- `RandomCrop(size=32)`: Random crops to the specified size\n", "\n", "### The Dataset Superpower\n", "\n", "The **dataset** combines data loading, batching, and transformations in a way that makes data augmentation extremely efficient:\n" ] }, { "cell_type": "code", "execution_count": null, "id": "fbf64ae7", "metadata": {}, "outputs": [], "source": [ "trainset = torchvision.datasets.CIFAR10(\n", " root='./data', \n", " train=True, \n", " download=True, \n", " transform=transform # ← Transforms applied here!\n", ")\n", "\n", "trainloader = torch.utils.data.DataLoader(\n", " trainset, \n", " batch_size=BATCH_SIZE, \n", " shuffle=True\n", ")" ] }, { "cell_type": "markdown", "id": "2f469190", "metadata": {}, "source": [ "\n", "**Why is this a superpower?**\n", "\n", "1. **On-the-fly Augmentation**: Transformations are applied **during data loading**, not during preprocessing. This means:\n", " - Each epoch, the same images are transformed differently\n", " - You get infinite variations of your data without storing them all\n", " - Memory efficient: only need to store the original dataset once\n", "\n", "2. **Separate Pipelines for Train/Test**: You can use different transforms for training (with augmentation) and testing (without augmentation):\n", " ```python\n", " train_transform = transforms.Compose([\n", " transforms.RandomRotation(30),\n", " transforms.RandomHorizontalFlip(),\n", " transforms.ToTensor(),\n", " transforms.Normalize(mean, std)\n", " ])\n", " \n", " test_transform = transforms.Compose([\n", " transforms.ToTensor(),\n", " transforms.Normalize(mean, std)\n", " ])\n", " ```\n", "\n", "3. **Parallelization**: DataLoaders can use multiple workers to apply transformations in parallel while fetching the next batch, making training faster.\n", "\n", "This is why well-tuned data augmentation combined with DataLoaders is one of the most effective techniques for improving deep learning model performance without collecting more data." ] }, { "cell_type": "markdown", "id": "99615018", "metadata": {}, "source": [ "### Exercise: Exploring the Transform API\n", "\n", "Now it's your turn to practice the Transform API and see data augmentation in action!\n", "\n", "**Objective**: Create multiple transform pipelines and visualize how different augmentations affect the same image.\n", "\n", "**Tasks:**\n", "\n", "1. **Create three different transform pipelines:**\n", " - `weak_augmentation`: Only basic transforms (Resize, ToTensor, Normalize)\n", " - `medium_augmentation`: Add some augmentation (RandomRotation, RandomHorizontalFlip)\n", " - `strong_augmentation`: Add aggressive augmentation (RandomRotation, RandomHorizontalFlip, ColorJitter, RandomAffine)\n", "\n", "2. **Load a sample image** from the CIFAR10 dataset using each pipeline\n", "\n", "3. **Visualize the results**\n" ] }, { "cell_type": "code", "execution_count": null, "id": "2c508b62", "metadata": {}, "outputs": [], "source": [ "#TODO" ] } ], "metadata": { "kernelspec": { "display_name": "appoc (3.12.11)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.11" } }, "nbformat": 4, "nbformat_minor": 5 }