Share

This week, we will be dropping the automated build status on our Docker images in an effort to deliver a higher quality experience for the consumers of these images.

We will break down the path that has lead to this point in this blogpost, but here is the tl;dr:

By taking full control of the Docker image build process, we can push images that have been thoroughly tested and have received the NodeSource stamp of approval. This will also allow us to do proper aliasing of images, ensuring that jessie:argon and jessie:4 point to the same exact image and not just an image built from the same Dockerfile. This gives you, the consumer, the assurance that the image you are running in production is the same image that we hand crafted and tested for you.

Now let's dive into the details of what this new workflow affords consumers of our images.

Testing

The most significant motivation behind dropping our automated build status is that we can now fully control the quality of the produced image. We test our Docker images to ensure what is generated by our Dockerfiles (and what is pulled over the network) meets our expectations. To accomplish this, we use an open source tool called Dante.

What this tool allows us to do is build an image from a Dockerfile, then run our tests as layers on top of that image. Assuming all of the commands in the tests can successfully generate layers on top of our image, you have a guarantee that the environment inside of our image is stable enough to run the tasks represented by the tests.

With the automated builds, we would run these tests and get a green light on our local images. We would then trigger a build on the Docker registry and hope for the best. The images that existed on the registry were not the same images that we built and ran our tests against, they were only built with the same Dockerfile.

With our new workflow, we can now take our generated images, run our tests as layers on top of those images, throw away the test layers and ship the base image that we know to be in a stable state straight to the Docker registry. This means the image that you pull from the registry will be the same exact image that we tested.

Though we will be giving up the Docker registry’s audit system when we drop our automated build status, dante generates detailed logs of our builds that we will keep to ensure our images meet the high standards you have come to expect from the NodeSource images.

Aliasing

One of the major benefits of tracking the NodeSource Docker images is that we offer version pinning. This means you can track either major, minor, or specific releases while still getting security updates to the underlying operating system contained in the Image (for a deeper discussion of why this is important, check out my blog post Docker: Owning the Stack). We accomplish version pinning by generating a Dockerfile for every version/os combination that we maintain, for example debian/jessie/node/4.2.1, then nodesource/jessie:argon, nodesource/jessie:4, nodesource/jessie:4.2 and nodesource/jessie:4.2.1 all point to that Dockerfile.

With the automated builds, we accomplish aliasing by setting up tags on the Docker registry, and each tag generates an image built from that Dockerfile. This causes a side effect that has been commented on in our GitHub Issues. Since these images are built separately from the same Dockerfile, as opposed to a single image being built from the Dockerfile and then re-tagged to the aliases, each image will have a distinct hash. Although logically identical, the images in our example above are in fact 4 distinct images with small differences in their filesystems (usually timestamps).

With our new workflow, we generate a single image from our Dockerfile. In our example above, this would be nodesource/jessie:4.2.1. We then use docker tag to alias that image to nodesource/jessie:argon, nodesource/jessie:4, and nodesource/jessie:4.2. In our new workflow, you will be running the same exact image no matter which alias you are tracking.

Determinism

Another big win here is deterministic builds. While a majority of the builds on the Docker registry succeed, I estimate that 1/200 of our builds fail due to network problems. This isn't a huge problem for most projects, but we ship nearly 700 images if you include our aliased builds. This ends up causing several failures every time we do a complete rebuild. Several times, a failure like this on a newly shipped image caused holes in our version coverage.

With our new workflow, dante affords us the ability to retry builds and tests. When a build fails, dante will attempt a rebuild a set number of times before giving up. With the number of images we maintain, this will dramatically reduce the amount of time it takes to ship updated images, allowing you to get the latest versions of Node faster and more reliably.

Drawback

In dropping our automated status, we will be dropping triggered builds. This means that updates to the base images we build from will not automatically trigger a rebuild of our images. To offset this, we aim to rebuild and ship new images with updated dependencies weekly, allowing you to pull in security updates as often as you like. We will manually trigger builds in the event of a major security patch, such as shell-shock.

Conclusion

In dropping our automated build status in favor of shipping our images direct, we will be able to deliver a higher quality experience to consumers of our images. This new method of shipping generates verbose logs, increased ability to test, and a more deterministic image. The best part is that these changes will be mostly transparent to you; you will be able to use our images after these changes in the same way that you are using them today.

NodeSource offers commercial support through N|Support to help companies establish and sustain enterprise-grade Node.js development and operations. N|Support with the NodeSource Docker Images extends the Node capabilities of your Development and DevOps teams, to protect your project and serve your team when it matters. Official N|Solid Docker images are coming soon.