Docker Outside of Docker

I’ve been working with Docker a ton lately, and recently decided to deploy Jenkins on Alpine within a compose of other private repositories/registries/etc.

However, as soon as I wanted to use Docker within a Jenkins Pipeline, things went south really fast.

Fresh Setup

So, after getting the container running (which was easy), as soon as I tried any Docker commands on either CLI or via Jenkins, I was met with the following error:

ash: docker: not found

Duh, of course! The interwebs led me down the road of the obvious of needing Docker installed in the container. But I didn’t want to maintain a docker daemon to deploy Jenkins, which also had a docker version inside that needed to progress as my builds progress.

But it also says you can mount the docker binary and docker.sock, and should be gravy.

Slight Change

At this point I have the following:

Docker Run Equivalent

docker run -it –restart always -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker jenkins:2.32.2-alpine

docker-compose.yml Equivalent

jenkins:
  image: jenkins:2.32.2-alpine
  restart: always
  ports:
    - 8080:8080
  volumes:
    - /usr/bin/docker:/usr/bin/docker
    - /var/run/docker.sock:/var/run/docker.sock

But when attempting another Docker operation, I’m still met with the same error.
I can see the binary and socket inside the container, and as expected they are owned by root – but I’d expect an error about permissions, not “not found” errors.

It just didn’t make sense, so suddenly I remembered the ldd command, which will show static and dynamically linked object dependencies:

$ ldd /usr/bin/docker
linux-vdso.so.1 => (0x00007fff6691a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd2bd893000)
libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007fd2bd689000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd2bd2bf000)
/lib64/ld-linux-x86-64.so.2 (0x000055ba66aa0000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd2bd0bb000)

As you can see, some are linked directly to a memory address, while some indicate a file! Ah-ha! So let’s update our orchestration:

Docker Run Equivalent

docker run -it –restart always -v /lib/x86_64-linux-gnu/libc.so.6:/lib/x86_64-linux-gnu/libc.so.6 -v /lib/x86_64-linux-gnu/libdl.so.2:/lib/x86_64-linux-gnu/libdl.so.2 -v /lib/x86_64-linux-gnu/libpthread.so.0:/lib/x86_64-linux-gnu/libpthread.so.0 -v /lib64/ld-linux-x86-64.so.2:/lib64/ld-linux-x86-64.so.2 -v /usr/bin/docker:/usr/bin/docker -v /usr/lib/x86_64-linux-gnu/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7 -v /var/run/docker.sock:/var/run/docker.sock jenkins:2.32.2-alpine

docker-compose.yml Equivalent

jenkins:
  image: jenkins:2.32.2-alpine
  restart: always
  ports:
    - 8080:8080
  volumes:
    - /lib/x86_64-linux-gnu/libc.so.6:/lib/x86_64-linux-gnu/libc.so.6
    - /lib/x86_64-linux-gnu/libdl.so.2:/lib/x86_64-linux-gnu/libdl.so.2
    - /lib/x86_64-linux-gnu/libpthread.so.0:/lib/x86_64-linux-gnu/libpthread.so.0
    - /lib64/ld-linux-x86-64.so.2:/lib64/ld-linux-x86-64.so.2
    - /usr/bin/docker:/usr/bin/docker
    - /usr/lib/x86_64-linux-gnu/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7
    - /var/run/docker.sock:/var/run/docker.sock

Not Quite There

Once more, running the pipeline results in an error, but this time it’s the error we expect:

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.25/containers/json: dial unix /var/run/docker.sock: connect: permission denied

In case you weren’t aware, the Jenkins container runs as a jenkins user, which clearly doesn’t have access to the socket.

Stumbling more around the interwebs, I found an article indicating their trials and tribulations on this exact topic; but their solution was to add sudo to the image. Oh, and it’s also where I got the phrase “Docker Outside of Docker.”

There’s 2 issues I have with this:

  1. They blindly give the jenkins user full sudo, which pretty much avoids the use of a non-suid user. Sure, it CAN be tweaked for ONLY docker commands, but…
  2. I have been using the practice of not using sudo on commands unless 100000% necessary for as long as I can remember; I’d guess well over 15 years.
    • Also, I should state that the Jenkins Freestyle Pipeline does not operate with sudo, and I wasn’t ready to delve into proper Jenkinsfile construction for advanced pipelines just yet. SO MUCH TO DO!

Now What?

At this point, the only thing I can think of is adding a GUID inside the container that matches the docker GUID in the host.

I’ll save you the details, but I came across this StackOverflow post that conveniently had a script for setting up this matching for us, which I’ve heavily modified here. Use it with the configurations listed above.

The code should be very simple to follow, but essentially it:

  • ensures the docker socket exists and is actually a socket.
  • ensures the docker binary exists and is functional.
    • Remember to use ldd $(which docker) to see if you have any dynamically linked deps
  • ensures a group with GID of the docker socket exists inside the container, and creates a docker group should it not exist.
  • ensures the desired docker user exists in the aforementioned group GID, and of course adds it if not.

BUT WAIT, THERE’S MORE!

Yeah so… What had happened was… It’s not gonna work. Not yet.

You need to restart the docker daemon for it to accept the new user access. THEN you can use docker inside your container.

Have fun!

Advertisements

About killerspaz

I'm a developer that loves to tinker with cutting edge technology. I have recently been playing with the Flash Platform (AS3/Flex), Android (custom roms, replacement apps, scripting), and looking at opportunities in the mobile markets.

Posted on 03.30.2017, in Containers, Docker, Uncategorized and tagged , , , . Bookmark the permalink. Leave a comment.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: