Managing Docker Images

Introduction

In the previous posts of the series, we discussed in depth about Docker images. As we’ve seen, we’ve been able to take existing images, provided to the general public in Docker Hub, and run them or reuse them for our purposes. The image itself helps us streamline our processes and reduce the work we need to do.

In the next few posts, we are going to take a more in-depth look at images and how to work with them on our system. We’re going to discuss how images can be better organized and tagged, understand how different layers of images work, and set up registries that are both public and private to further reuse the images we have created.

Docker images are also ideal for application development because each image contains a complete, self-contained version of the application along with all its dependencies. This enables developers to create an image locally and deploy it in development or testing environments to verify compatibility with other parts of the application. If testing is successful, the same image can be pushed to the production environment for users to use. It’s crucial to maintain consistency when using these images, especially when collaborating within larger developer teams.

Docker Layers and Caching

A registry is a way to store and distribute Docker images. When you pull a Docker image from a registry, you might notice that the image is pulled in pieces and not as a single image. The same thing happens when you build an image on your system.

This is because Docker images are structured in layers, each representing a stage of the image’s construction. Each layer of an image represents a specific action or change made when building the image with a Dockerfile. These layers are organized on top of a base image, capturing every change to the filesystem that occurs with each instruction in the Dockerfile. This setup is structured in a way that Docker can use caching efficiently.

When you instantiate an image as a container, Docker adds a writable layer on top of the existing read-only layers. This writable layer, often referred to as the container layer, allows the container to modify and persist changes during runtime without affecting the underlying image.

As we will see in the following examples, when you build a Docker container from a Dockerfile, Docker shows the execution of each command specified in the Dockerfile. These commands contribute to creating layers in the Docker image, each represented by a unique ID generated during the build process. After successfully building the image, we can inspect the layers using the docker history command, which provides a detailed view including the image name or ID alongside the commands that formed each layer.

It’s important to note, that as you setup your build environment and progress in development, the number of layers in the Docker image grows. More layers mean larger image sizes, which can lead to longer build times.

When you build an image from a Dockerfile, each instruction contributes to the creation of layers in the image. Layers are created explicitly when commands like RUN, ADD and COPY are executed. These commands make changes to the filesystem within the image, resulting in new layers being added.

On the other hand, commands like FROM, ENV, WORKDIR and CMD do not directly create filesystem changes. Instead, they modify the environment or configure settings within the image without altering the filesystem itself. As a result, these commands generate intermediate layers. These layers have a size of 0 bytes because they don’t introduce any new filesystem change. They serve as metadata or configuration layers that help define how an image behaves or is structured, but they don’t increase the size of the final Docker image.

When building our Docker images, we can use the docker history command and the image name or ID to see the layers used to create the image. The output will provide details on commands being used to generate the layer as well as the size of the layer.

The docker image inspect command is useful in providing further details on where the layers of our images are located:

Working with Docker Image Layers

In this example, we are going to work with some basic Dockerfiles to see how Docker uses layers to build images. We will start by creating a Dockerfile and building a new image. We will then rebuild the image to see the advantages of caching and how the build time is reduced due to its use.

Create a file named Dockerfile and add the following directives:

Save the Dockerfile and then, from the command line, make sure you are in the same directory as the Dockerfile you are created. Use the docker build command to create the new image using the -t option to name it basic-example

If the image is built successfully, you should see an output similar to the following. Rach step is built as an intermediate layer and if it completes successfully, it is then transferred to a read-only layer

Use the docker history command along with the image name of basic-example to see the different layers of the image

The history gives you creation details, including the size of each layer

The docker history command shows the layer of the original image used as part of the Dockerfile FROM command as <missing>. It is showing as missing as it was created by a different system and then pulled into ours.

Run the build again without making any changes

This will show you the build is done using the layers stored in the Docker image cache, thereby speeding up our build. Although this is a small image, a much larger image would show a significant increase

Lets add the curl package as part of our image creation, and modify the Dockerfile as follows

Build the image again, and now you’ll see the image was created with a mix of cached and new layers

The above command should create the following output

Run the docker image command again

You will now notice an image named and tagged as <none> to show we have now created a dangling image

In Docker, dangling images are those that are represented by <none> in the image list. These images occur when a layer in the image hierarchy no longer corresponds to any tagged or referenced image in the system. Essentially, they are orphaned or unused layers that have lost their connection to any active image.

Dangling images can accumulate over time as you build and prune Docker images, and they occupy disk space without serving any meaningful purpose. Even though each individual dangling image might be relatively small, such as the example of 7.48 MB, these sizes can accumulate significantly over time, especially in development and production environments where frequent image builds and updates occur.

Run the docker image inspect command using the image ID to see the location of where the dangling images are located in the system

And you should get an output similar to the following

All of our images are located in the same location. So any dangling images would waste space on our system.

Run the docker images command again using the -a option

It will also show the intermediate layers used when our image is build

Run the docker image prune command to remove all the dangling images. We could use docker rmi but the docker image prune command is the easier way to do it

You should get an output looking like the following

Run the docker images command again

You will see we no longer have the dangling image in our list of images

In this example was on smaller image sizes, but this is definitely something to keep in mind when running production and development environments. In the next example, we will look further at our layers and caching to see how they can be used to speed up the image build process.

Increasing Build Speed and Reducing Layers

We’ve been working with small projects up until now. As our apps get bigger and more complex, though, we’ll want to start thinking about the size and number of layers in our Docker images, along with how quickly we’re building them. In this example, we’ll focus on speeding up build times, shrinking those image sizes, and using the –cache-from option to make things even faster.

First, let’s clean up any existing images on your system. We’ll use the docker rmi -f $(docker images -a -q) command, which will force-remove all images currently on your system. This will give you a clean slate to work with.

Create a new Dockerfile with the following content. It will simulate a simple web server, as well as print the output of our Dockerfile during the build process

Download the Alpine base image using docker pull so that we can start with the same image for each test we do

Create a TAR file to be added to our image

Build a new image using the name of basic-server. We are going to use the time command at the start of the command to allow us to gauge the time it takes to build the image

The output will return something similar to the following

And the time will be:

Run the docker history command over the new basic-app image

The output should be something like the following

As you can see there are 12 layers in our new image. As you can see, the RUN, COPY and ADD commands in our Dockerfile are creating layers of a particular size relevant to the command being run or files being added, and all of the other commands are of size 0 B.

We can slim down our image by merging some of the commands in the Dockerfile we created earlier. Combine the RUN commands from lines 3 and 4, and then merge the CMD commands from lines 8 and 9. This will reduce the number of layers in our image, making it more efficient.

After making these changes, your Dockerfile should look like this:

If we rebuild our Docker image now, we’ll see that the number of layers has decreased from 12 to 9. This is because we combined several commands into single lines, even though the same actions are still being performed.

We can further optimize our Dockerfile by replacing lines 11, 12, and 13 with a single ADD command. This eliminates the need for separate COPY, RUN, and RUN commands to unzip and remove the archived file. Here’s the updated Dockerfile snippet:

Rebuilding our Docker image with the ADD command in place reduces the number of layers from 9 to 8, making it even more streamlined.

You might have noticed that a significant portion of the build time is spent running apk update, installing wget and curl, and fetching content from websites (as seen in lines 3 and 5 of our Dockerfile). While this isn’t a major issue for a few builds, it can become a bottleneck when creating multiple images.

To address this, we can create a base image that already includes these tools and dependencies. By using this base image as our starting point, we can eliminate these lines from our Dockerfile altogether, further improving build times and image efficiency.

All right, let’s create a dedicated base image to streamline our Docker builds. First, navigate to a new directory:

Now, create a new Dockerfile in this directory with the following contents:

This Dockerfile does three things:

Pulls the base image: It starts with the alpine:latest image as its foundation.

Runs the apk commands: It updates the package index and installs wget and curl using apk add –no-cache. The –no-cache option prevents apk from storing the downloaded packages in the local cache, keeping the image smaller.

Runs the wget command: It finally downloads the randomdata.txt file from the external directory.

Build the new image from the previous Dockerfile and name it basic-base

Perfect! Now that we have a base image ready, let’s update our original Dockerfile. First, head back to the directory where your original Dockerfile resides:

Now, open your Dockerfile and make the following changes:

Remove line 3: Delete the line that starts with RUN apk update… since these commands are now handled in our base image.

Update the FROM command: Change the base image from alpine:latest to the name you’ll give your custom base image (i.e., basic-base).

Remove the apk commands from line 3 (now line 2): Delete the remaining part of the line that installed wget and curl.
Your updated Dockerfile should now look similar to this

Run the build again for your new Dockerfile. Using the time command again, you should see the build complete in just under 3 seconds

You’ll likely notice a significant speed improvement compared to our previous builds. This is because we’ve offloaded the time-consuming package installations to our base image, streamlining the build process for our main image.

Throughout this example, we’ve seen firsthand how the build cache and image layers work together to significantly speed up the Docker build process. We’ve been starting our builds by pulling images from Docker Hub, but you have the flexibility to start with your own custom images. This gives you even more control over the build process and allows for further optimization.

By creating and maintaining your own base images, you can tailor them to your specific needs, pre-installing common dependencies and configurations. This not only reduces build times but also ensures consistency across your projects.

Summary

This post demonstrated how Docker allows users to work with images to package their applications together with a working environment to be moved across different environments. You’ve also seen how Docker uses layers and caching to improve build speed and ensure you can also work with these layers to reserve resources or disk space.

Please follow and like us: