COPY --link there is a new buildkit Convenience that can significantly speed up your Docker image creation. It works by copying files into independent image layers that do not depend on the presence of their predecessors. You can add new content to images even without the original image being on your system.

This capability was added as part of BuildX v0.8 in March 2022 Version 20.10.14 . included If you are running the latest release you should already have access to the Docker CLI.

In this article, we will show what --link does and explains how it works. We will also look at some of the situations in which this should not used.

What is “-link”?

--link There is a new optional argument for the existing Dockerfile COPY Instructions. It changes the way copy works by creating a new snapshot layer each time you use it.

Regular COPY Statements add files to the layer that precedes them in the Dockerfile. The contents of that layer must be present on your disk in order for the new content to be merged into:

FROM alpine
COPY my-file /my-file
COPY another-file /another-file

Copies of the Dockerfile above my-file In the layer created by the previous command. After FROM Instructions, the content of Alpine in the image is included:

bin/
dev/
etc/
...

first COPY The instruction produces an image that includes everything from Alpine, as well as my-file file:

my-file
bin/
dev/
etc/
...

and another COPY adds instructions another-file At the top of this image:

another-file
my-file
bin/
dev/
etc/
...

The layer produced by each instruction includes everything that came before it, plus anything new that was added. At the end of the build, Docker uses a different process to complete the changes within each layer. The final image blob only contains the files that were added at each snapshot step but this is not reflected in the assembly process during build.

Introducing “-link”

“-link” modifies COPY To create a new standalone file system each time it is used. Instead of copying new files on top of the previous layer, they are sent to a completely different location to become an independent layer. The layers are then joined together to form the final image.

Let’s change the example Dockerfile to use --link,

FROM alpine
COPY --link my-file /my-file
COPY --link another-file /another-file

result of FROM The instruction is unchanged – it generates the alpine layer with all the contents of that image:

bin/
dev/
etc/
...

first COPY The instruction has a different effect. This time another independent layer is created. This is a new file system with only my-file,

my-file

then another COPY The instruction creates another new snapshot only with another-file,

another-file

When the build is complete, Docker stores these independent snapshots as new layer archives (tarballs). The tarballs are linked back in a series of previous layers, forming the final image. In this all three snapshots are merged together, resulting in a filesystem that matches the original when the container was created:

my-file
another-file
bin/
dev/
etc/
...

This image from the BuildKit project shows the differences between the two approaches.

Adding “Copy-Link” to Your Build

COPY --link available only when You are using BuildKit to build your images. either run your build docker buildx --create or use docker build with DOCKER_BUILDKIT=1 Environment variable set.

You must also opt-in to Dockerfile v1.4 syntax by using a comment at the top of your file:

# syntax=docker/dockerfile:1.4
FROM alpine:latest
COPY --link my-file /my-file
COPY --link another-file /another-file

Now you can create your own image with the support of linked copies:

DOCKER_BUILDKIT=1 docker build -t my-image:latest .

Images created using Dockerfiles COPY --link Can be used like any other. You can start a container with docker run And push them directly to the registries. --link The flag only affects how content is added to the image layers during construction.

Why linked copies matter

using the --link The flag allows the build cache to be reused, even when you build content. COPY in changes. In addition, builds may be able to complete without their original image being present on your machine.

Returning to the example from above, the standard COPY behavior is required alpine The image will be present on your Docker host before new content is added. The image will be downloaded automatically during creation if you haven’t drawn it before.

With linked copies, Docker doesn’t need alpine image content. it pulls alpine manifest creates new independent layers for the copied files, then creates a modified manifest that links the layers to the ones they provided alpine, contents of alpine The image – its layer blob – will only download if you start a container from your new image or export it to a tar archive. When you push the image to the registry, that registry will store its new layers and get the remote alpine The ones.

This functionality also facilitates efficient image rebase. Perhaps you are currently distributing a Docker image using the latest Ubuntu 20.04 LTS release:

FROM golang AS build
...
RUN go build -o /app .

FROM ubuntu:20.04
COPY --link --from=build /app /bin/app
ENTRYPOINT ["/bin/app"]

You can use BuildKit’s . You can create the image with caching enabled by using --cache-to flag. inline Cache stores build cache data inside the output image, where it can be reused in subsequent builds:

docker buildx build --cache-to type=inline -t example-image:20.04 .

Now let’s say you want to provide an image that will be based on the next LTS after its release, Ubuntu 22.04:

FROM golang AS build
...
RUN go build -o /app .

FROM ubuntu:22.04
COPY --link --from=build /app /bin/app
ENTRYPOINT ["/bin/app"]

Rebuild the image using the cache data embedded in the original version:

docker buildx build --cache-from example-image:20.04 -t example-image:22.04 .

The construction will be completed almost immediately. Using cached data from an existing image, Docker can verify the files needed to build /app hasn’t changed. It means the cache for the independent layer created by COPY The instruction remains valid. Since this layer does not depend on any other, ubuntu:22.04 The image won’t even be drawn. Docker only adds the snapshot layer in which /bin/app in a new manifest within ubuntu:22.04 layer series. The snapshot layer is effectively “rebased” on a new original image, without any filesystem operations.

The model also optimizes multi-stage builds where changes can occur between any of the stages:

FROM golang AS build
RUN go build -o /app .

FROM config-builder AS config
RUN generate-config --out /config.yaml

FROM ubuntu:latest
COPY --link --from=config /config.yaml build.conf
COPY --link --from=build /app /bin/app

without --linkany change in the generated config.yaml cause ubuntu:latest to be dragged and the file to be copied. The binary has to be recompiled because filesystem changes have invalidated its cache. With linked copies, changes to config.yaml Allows construction to continue without stretching ubuntu:latest Or recompiling the binary. with snapshot layer build.conf The inside is simply replaced by a new version that is independent of all other layers.

when not to use it

There are some situations where --link The flag will not work properly. Because this copies the files to a new layer, instead of appending them on top of the previous one, you cannot use ambiguous references as your destination path:

COPY --link my-file /data

with regular COPY Instructions, my-file will be copied into /data/my-file If /data The image already exists as a directory. with --linkThe target layer’s filesystem will always be empty, so my-file is written to /data,

The same idea applies to symlink resolution. standard COPY Automatically resolves destination paths that are symlinked to the image. when you are using --linkThis behavior is not supported because the symlink will not exist in the independent layer of the copy.

It is recommended that you start using --link where these limits do not apply. Adopting this feature will speed up your builds and make caching more powerful. If you cannot remove the ambiguous or associated destination paths immediately, you can continue to use the existing COPY Instructions. It is because of these backward incompatible changes that --link New is an optional flag instead of the default.

summary

buildkit COPY --link There is a new Dockerfile feature that can make builds faster and more efficient. Images that use linked copies do not need to drag past layers, so that files can be copied into them. Docker creates a new independent layer for each COPY Instead, it then adds those layers back into the chain.

If you’re building images with the latest version of BuildKit and Buildx or Docker CLI, you can start using the linked copies now. Adopting “-link” is a new best practice Docker build phase, provided you are not affected by necessary changes to the destination path solution.