There are plenty of criticisms of docker, the system for building a container-based virtual machine running just a single application. I've read many of them have have consistently been either in agreement or at least amused.

The most relevant criticism is about the basic approach of building single-application virtual machines. To understand this criticism, let's remember - in traditional application deployment there are at least three distinct jobs:

  • Distrubtion developers - the team the integrates all the various packages available into a coherent and well-functioning system and, importantly, monitors the upstream of all the packages to ensure that security bugs are properly packaged and made easily available for installation. These people don't give a shit about your PHP parse error and don't really care if you can't figure out the command to create a new Postgres Database.
  • System administrators - these are the people that ensure that the security updates provided by the distribution developers get installed on a regular basis and if something breaks during this process, they are the ones that fix it. They know how to create your new postgres database. They don't really care about your PHP parse error either, but may get roped in to tell you it's a PHP parse error.
  • Application developers - these are the people that care about the PHP parse error. They also know how to create beautiful things that end users can interact with.

These three groups have lived in happy tension for years.

Now we have containers.

The problem with containers is that suddenly the system administrators are out entirely and the distribution developers' role may have been dramatically minimized or circumvented altogether. Meanwhile, the application developer is both liberated from any constraints imposed by either the system administrator or the distribution developers (hooray!) but is also saddled with the enormous responsibilities of these two roles - which may not be apparent to the application developer at first.

I have practically no distribution developer experience and have considerable experience as a system administrator and an application developer. And, it took me months to sort through how to properly develop and deploy an application using docker in a way that I thought was responsible and secure.

I started by learning about how docker wants you to deploy images - by downloading them from their shared registry. As I mentioned, I have very little experience in the realm of distribution development, but I at least know enough about Debian to know that a lot of time and thought has gone into cryptographically verifying packages that I install, which apparently is not done at all with Docker images. Is that obvious to everyone using Docker??

Fortunately, you can work around this problem by creating your own base image which is trivially easy. So, now I build all images, from scratch, locally. That helps put Debian developers back into the mix.

Next, I started looking at Docker Files that I could use to construct my images and discovered something else troubling. Take, for example, the official nginx Docker image. It is based on Debian Jessie (hooray - our distribution developers are in the mix!). However, it then proceeds to install nginx from the nginx repository, not the debian repository. Well, I guess if you are nginx you want to have full control, but still, the Debian developers version of nginx has been vetted to ensure it works with Debian Jessie, so you are really losing something here.

So... my next break from Docker convention is to use all Debian packages in my Docker files.

Once my images were built and my application was tested and running, I was done, right?

Wrong. Well, I would have been done if I didn't care about upgrading my systems, backing up my data or running cron jobs. Remember those things? Those are things that distribution developers and system administrators have been perfecting for decades. And, they are not trivial.

I then built a series of scripts to help alert me when a image I am using has updates (you can't just use cron-apt any more since cron isn't running in your container) and help me update the image and deploy it to all my applications (which involves restarting the application). Backing up data is a whole different can of worms - sometimes involving interacting with your container (if it's a database container, you have to launch a database dump) or simply copying files from the host, assuming you got the right Docker volume strategy (which took me days to fully understand). Lastly, I had to run a cron job from the host that then runs whatever commands are needed on each of my containers.

This was complicated.

In the end, was it worth it? Yes, I think so. However, not because it was simple, which seems to be the Docker mantra. I think it's worth it because I can run 100 instances of my application using significantly less resources than when I was using full virtualization and because I can more easily and flexibly adjust the resource allocation. However, check in with me in a year and I'll probably have a different opinion.

This is the same approach I use in propellor; docker containers are deployed from only trusted images, and debian packages are installed and upgraded inside the containers as needed. With the bonus that I just reuse the same code to manage docker containers as I use to manage servers. http://joeyh.name/blog/entry/adding_docker_support_to_propellor

I will say though, that once you do the analysis and decide this is the reasonable way to use docker, to avoid the many traps built into the way it's intended to be used, there turns out to not be much benefit to using docker at all, vs say, regular chroots deployed by debootstrap and run as containers using systemd-nspawn.

Comment by joeyh.name Mon 03 Aug 2015 02:50:02 PM EDT
What advantages do you see in using docker instead of lxc? For some of the reasons you mention, I think I will prefer managing containers as full machines, rather than learning docker. But I echo your final sentiment, I don't know what I'll think in a year.
Comment by Anonymous Mon 03 Aug 2015 04:33:46 PM EDT
It does seem like using a more light-weight system like systemd-nspawn would make more sense since I'm circumventing most of docker's selling points. On the other hand, docker's popularity makes our systems more easily maintainable by others in the event that I'm raptured (I'd rather not talk about getting hit by a bus...). In addition, docker does have useful features that I am using - like easy networking and linking containers together.
Comment by jamie [id.mayfirst.org] Tue 04 Aug 2015 08:57:47 AM EDT

You talk of maintaining your images like they're VMs in their own right; you're logging in and having them patch themselves. Doing it that way (in my mind at least) you lose a lot of the value of docker - having atomic images that at any time are all exactly alike and don't have to be individually maintained.

If you have each machine patch itself it is possible (and even likely once you get to a non-trivial number of images) that you're going to have inconsistencies because it is possible that some patches will fail on some systems. Then you have the added issue of a patch potentially breaking your application - and rolling across your entire stack before you realize it.

I would suggest that one of the benefits is that docker packaging is easy enough that you can patch one image manually, validate everything is exactly as you want it and then deploy it with very little risk of environmental factors causing issues.

My point is that I see docker as a problem solver for system admins and environment engineers more so than the developers. The developers continue to do what they do but the orchestration around deployment and maintenance becomes much more robust.

Comment by Anonymous Thu 06 Aug 2015 06:24:38 PM EDT