Docker in the cloud
Note: This article is obsolete because the use case it describes is now supported by Docker Machine. To achieve the same result with that tool, do the following:
Use Docker Toolbox to install Docker and Docker Machine.
Create a VM instance using the following command:
docker-machine create -d google --google-project myproject --google-zone europe-west1-b --google-machine-type n1-standard-1 docker
Set up the environment with:
eval "$(docker-machine env docker)"
This article explains how to run a Docker daemon remotely in Google Compute Engine and control it from a Mac OS X client. This is an alternative to boot2docker (or to running Docker entirely in a local Linux virtual machine) and is especially interesting if your workstation lacks the resources to start up your Docker containers.
To achieve this, we need to install a Docker client on the Mac OS X workstation and the full Docker distribution in a VM running on GCE. Since the protocol is incompatible between different Docker versions, it is important to use the same version on both sides. In this article we will use Docker 1.5, which was the latest release at the time of writing. If necessary, the instructions can easily be adapted for newer Docker versions.
Installing the Docker client on Mac OS X
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
By default Homebrew always installs the latest available version of a package. However, as mentioned above, we need a particular version of the Docker client. To install that particular version, we specify the URL of the corresponding Homebrew formula instead of just the package name:
brew install https://raw.githubusercontent.com/Homebrew/homebrew/925e53b0ec49a03a31a2aaacc0cc49a8860b0454/Library/Formula/docker.rb
Setting up Docker in Google Compute Engine
The instructions in this section assume that you already have a GCE project and that you have installed
the Google Cloud SDK. We will also also assume that you have set that project as default project and that
you have configured a default zone where new VMs are created. If this is not the case, configure the
line tool using
gcloud config; for example:
gcloud config set project myproject gcloud config set compute/zone europe-west1-b
You can then use the following command to create a new VM to run the Docker server:
gcloud compute instances create docker --machine-type n1-standard-1 --image ubuntu-14-10
Of course, you may want to use a different machine type or a different Linux version. Note however that the installation instructions in the remainder of this section apply to Debian based distributions and have only been tested on Ubuntu 14.10.
Once the new VM has been created and started, use the following command to login:
gcloud compute ssh docker
We are now ready to install Docker. As mentioned earlier, we need to install a Docker version that matches the version of the client we set up in the previous section. Instead of using the Docker version that comes with the Ubuntu distribution, use the following sequence of commands to install a package from the Ubuntu repository at docker.com:
sudo sh -c "echo deb http://get.docker.com/ubuntu docker main > /etc/apt/sources.list.d/docker.list" sudo apt-key adv --keyserver pgp.mit.edu --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9 sudo apt-get update sudo apt-get install -y lxc-docker-1.5.0
By default, the Docker daemon listens on a Unix domain socket. To be able to connect remotely, we need to configure it
to listen to a TCP port instead. To do this, edit
/etc/default/docker and add the following line:
This configures the Docker daemon to listen on the loopback device. This means that SSH access to the VM is required to connect to Docker. This provides the desired level of security.
Note that if in addition to connecting remotely you want to continue to be able to use the docker client locally on the VM, you should also configure the default UNIX domain socket:
DOCKER_OPTS="-H tcp://localhost:2375 -H unix:///var/run/docker.sock"
Don’t forget to restart the Docker daemon to apply the change:
sudo /etc/init.d/docker restart
You can now disconnect from the VM.
Connecting remotely to Docker
As described above, opening the Docker port in the firewall of the VM and connecting directly would not be secure.
gcloud to create an SSH tunnel:
gcloud compute ssh docker --ssh-flag="-L 2375:localhost:2375"
You can now execute Docker commands as follows:
docker -H tcp://localhost:2375 ...
For example you can use the following command to test that the setup works as expected:
docker -H tcp://localhost:2375 run -it --rm ubuntu:trusty bash
To avoid having to specify the
-H option on every invocation of the
docker command, define the
environment variable as follows:
To set this variable persistently, add that instruction to the
You are now all set to build your Docker images and start containers remotely in the cloud!
When building images from Dockerfiles containing
COPYinstructions, the Docker client needs to send the contents of the specified files or directories to the Docker daemon as part of the build context. Be aware that for large files this will take a significant amount of time and consume bandwidth.
A possible solution for this problem is to store these files in a GCS bucket. If the bucket is public, the data can simply be retrieved with
gsutilcan be used. Use the following instructions in your Dockerfile to make that command available:
RUN \ wget https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.zip -qO /tmp/google-cloud-sdk.zip && \ unzip -qd /usr/local /tmp/google-cloud-sdk.zip && \ /usr/local/google-cloud-sdk/install.sh --usage-reporting=false --path-update=false --bash-completion=false && \ rm -rf /tmp/google-cloud-sdk.zip ENV PATH /usr/local/google-cloud-sdk/bin:$PATH
Note that since the Docker daemon runs on a GCE instance,
gsutildoesn’t require authentication, provided that the instance has been configured with read access to GCS (which is the case if you followed the instructions given in the previous sections).