How to improve Docker build from 37 minutes to 37 seconds for Golang apps
Почему-то не могу оставить комментарий к названию, поэтому напишу здесь: я в тексте не увидела ни 37 минут, ни 37 секунд. Есть 39 минут, а вот с секундами вообще беда - про них упоминания нет...
@Jack Rudenko
Docker is everywhere
Today, Docker is the de facto standard for application packaging. You can use a Docker image to run locally, or easily orchestrate and scale in Kubernetes or AWS ECS.
It's hard to imagine modern DevOps without Docker.
But Docker is a platform and architecture dependent technology. What does that mean? It means that Docker is not a real virtual machine, it is just a sandbox on top of the current host. If you run Docker on Mac M1, it runs on arm64 architecture, if you run it on intel CPI, it runs on amd64 architecture (is it sarcastic that intel CPU's have AMD architecture 🤣).
And it is not a problem if you are developing on the same architecture as your target. Normally, all developers had amd64 architecture and all servers ran Linux on the Intel platform. No problem at all until Apple introduced the M-series CPUs. And it became a widely used platform for development because the Apple M chip is incredible. I have also moved to an Apple M CPU and am very happy with it.
But there is a "but". Always that annoying "but".
Now when I build a Docker image, it builds arm64 images for me. I could no longer run it on my cloud machines unless I bought ARM machines in a cloud. But ARM in a cloud is now rare and expensive. Maybe that will change in the future.
Today, if you are going to build a public Docker and publish it on the Docker hub, you will need to support at least two platforms: arm64 and amd64.
Multi-architecture Docker build
Ok, now we need to build multi-architecture Dockers. No problem, we have dockerx and quemu for that.
Let's see how simple a Docker image could look for the dockerx solution.
FROM golang:latest
COPY . .
RUN go mod download
RUN go buil -o ./app .
EXPOSE 8010
CMD ["app"]
And then run:
docker buildx build --platform=linux/amd64,linux/arm64 .
And this thing will take about 20-30 minutes 🤯.
Why? Because it will run twice: once for amd64 and second time for arm64. It will run fast for the native platform and slow for the non-native platform because it will run in a real virtual machine. Docker uses quemu for this. It's an amazing solution, but it's slooooooooow.
Ok, why is running non-native images not that slow? Ok, you can run non-native images on your machine. On the Mac, all images are non-native because they run on Linux. Although it is possible to use MacOS as a base layer for Docker images, nobody does: maybe there are some legal restrictions, or maybe there are not many MacOS powered clouds.
If Mac use virtual machine to run all images, why are they not slow? Because Docker uses LinuxKit, which uses HypeKit framework, which uses MacOS native Hypervisor.framework. It is blazing fast.
That is why you will not feel any difference when running non-native images. Just in the Docker dashboard, you can see this orange AMD64 badge, which indicates that this image does not support your native architecture.
So Docker uses one virtual machine to run images and another to build them. Why? I think for simplicity. A slow VM is better than nothing. The Docker team made an amazing effort to bring Docker to the Apple M platform when Apple released it. They migrated from VirtualBox to HyperKit, yes, Docker recently used headless VritualBox. Don't expect them to do everything at once.
Also, it is not a problem for interpreters or byte-code languages. As for NodeJS, you don't care what platform and OS it is, you just feed your ugly TS (ok, joke, not ugly, just a bit noisy) code to the platform's node binary. Bundler can run on any platform.
The problem comes for real programmers (ok ok, no more jokes about JS developers) like Go, Rust, C++, C.
Multistage Docker image
Don't want your Docker image to be elephant-sized? Neither do we. Let's find out how to make your Docker image light enough to pass through the eye of a needle!
Luckily for us, Golang has fantastic cross-compile capabilities out of the box.
First, let's split our Docker image into multi-stage dockers. Why do this? Because if we do our first docker, we put all our source code into our Docker image. Which is not a good idea unless you are a JS developer (could not keep it inside, sorry 😁).
# First stage
FROM --platform=$BUILDPLATFORM golang:lates AS builder
COPY . .
RUN go mod download
RUN go build -o ./app .
# Second stage
FROM alpine:latest
WORKDIR /
COPY --from=builder /app .
EXPOSE 8010
CMD ["app"]
Ok, couple of things, now you can see we have two stages, the first one called builder is based on golang image which has all toolchains for go included. But second stage is naked alpine image. It is extremely lightweight image, only 3MB!!!
The second thing is that the final Docker image will not contain anything from the first stage. We are not packing anything from the build stage. All we are copying is a golang binary:
COPY --from=builder /app .
And the golang binary is also small, just a few MB.
As a result, our final Docker image will be 10-20 MB! Regular NodeJS container is 500MB, if developer is not experienced with Docker - more than a GB. Some people could reduce it to 100-200MB.
Why does size matter? Because the orchestration service could make your Docker run much faster and use less resources. You won't notice the difference if you have one or two instances running, but if you have hundreds or thousands ...
If we build this image with dockerx for multiple platforms, it will produce small images, but it will still run for 30-40 minutes.
Go cross-compile: Crafting multi-architecture builds as simple as a pie
Ok, can we avoid using quemu? We need a target platform compatible binary. And feed it to the target image.
Can we do that? Sure. Go can cross-compile. But now we need to consider the target architecture.
# First stage
FROM --platform=$BUILDPLATFORM golang:lates AS builder
COPY . .
RUN go mod download
ARG TARGETOS TARGETARCH
RUN GOOS=$TARGETOS GOARCH=$TARGETARCH go build -o ./app .
# Second stage
FROM alpine:latest
WORKDIR /
COPY --from=builder /app .
EXPOSE 8010
CMD ["app"]
Only two of them need to be changed:
ARG TARGETOS TARGETARCH
RUN GOOS=$TARGETOS GOARCH=$TARGETARCH go build -o ./app .
As you can see, we use TARGETOS and TARGETARCH from the dockerx command and then feed them to the go compiler with env variables. Go can be built for different operating systems and architectures.
I will now explain how it works. Docker will run builder platform on native arch.
When Docker will hit this line:
ARG TARGETOS TARGETARCH
it will split the execution between all the requested OSs and archs. In our case, "go mod download" will run once, but "go build" will run twice: for amd64 and for arm64.
The second stage is packed for each target as soon as we have a base image for that target: Alpine supports seven major architectures and only one operating system - Linux. What could be more valuable than JS developers? JS developers on Windows. Ok, just kidding 😄
Personally, I have not seen a Docker image targeting windows non-linux platforms, but there are some windows specific applications, like sqlserver, or unix platforms. So it might work with other operating systems.
One more note, when you build go, CGO is enabled, but when you cross compile go, CGO is disabled by default. If you need CGO, just add the following line:
ENV CGO_ENABLED=1
Here is what I had built with quemu in github action:
As you can see, the native platform duns for 3 minutes, while the non-native (arm64 for Action Runner) duns for 39 minutes. And it used cache.
And this is what it looks like for the cross-compile build:
Only four minutes, without a cache. Next time it will be faster, because of the cache.
Final tips for smooth sailing
So there you have it! Docker is the superstar of DevOps, but it's not without its quirks. To make life easier:
1. Slice and dice your Docker images with multistage magic.
2. Let Go do the heavy lifting with cross-compilation - it's like baking different cakes for everyone!
3. Don't forget CGO, if that's your flavour.
4. Say hello to multi-platform Docker images - they're the cool kids in town, catering to all architectures.
With these tricks up your sleeve, Docker deployments will be a breeze.
Hugs to everyone and happy coding, even for JS developers.
