I have been reading about Flocker for quite a while and it’s been something I have been meaning to look at since the introduction of plugins back in the Docker 1.7 release . But I have been busy looking at other parts of Docker.
Since I had a few hours this weekend I thought it was about time and had a play. ClusterHQ describes Flocker as;
Flocker is an open-source container data volume manager for your Dockerized application. It gives ops teams the tools they need to run containerized stateful services like databases in production.
Which basically means decent persistent storage, which is one thing I think Docker has been lacking when it comes to running services such as MySQL, Postgres, central logging etc.
For this test we are going to be launching three instances;
- control.mydomain.com — Controller — This will be our cluster controller
- flocker01.mydomain.com — Storage — Does what it says on the tin
- flocker02.mydomain.com — Storage — Does what it says on the tin
and share keys between the root user on all three machines allowing me password-less SSH access within the cluster.
Install the command-line tools
I am going to assume you will be installing the Flocker command-line tools on a Mac. Luckily ClusterHQ have made this straight forward by provide their own Brew packages. To install the tools simply run the following commands;
With the last two commands you should see something like the following output;
Finally, lets create a working directory;
So thats the command line tools installed and tested, now its time to up the cluster nodes.
If you are running Linux there are instructions for installing the command line tools various distributions in the official documentation
Node Preparation
Please note — You should also transfer SSH keys between all of the nodes as mentioned at the top of this post.
The following commands need to the run on all three cluster nodes;
- control.mydomain.com
- flocker01.mydomain.com
- flocker02.mydomain.com
We will be launching three CentOS 7 instances in Digital Ocean , first of all on each of the nodes run the Digital Ocean bootstrap script to get the defaults in place, and also run the Docker install script to get the latest version Docker installed and configured.
Now add the ClusterHQ yum repo and install the packages needed to run Flocker;
Next up create the directory where the cluster configuration will be stored and set the permissions
Finally, it is important you ensure that the hostname of the machine doesn’t resolve to 127.0.0.1. Open the /etc/hosts file on each machine and remove any references as needed (for both IPv4 and IPv6);
Repeat this process for each of the nodes within your cluster.
Install & Configure ZFS
The following commands need to the run on the storage nodes;
- flocker01.mydomain.com
- flocker02.mydomain.com
We are going to be using a ZFS peer-to-peer backend, this uses the local storage on the storage nodes, at the moment it is experimental so please do not try and use it in production.
As we ran the Digital Ocean bootstrap script and changed the Kernel configuration we just need to install the kernel-devel package, this should match the running kernel. To check this run
and check the that the version numbers match the kernel-devel package installed by the following command;
If the kernel-devel package matches the installed kernel then continue, if not please ensure that the correct kernel is selected in your Digital Ocean control panel and then try again;
The installation will take several minutes, this is normal and there is nothing to worry about if the process appears to hang during install.
Once installed, load the newly compiled kernel module by running;
Now on each of the storage nodes its time to create the ZFS pool, we will be calling the pool flocker;
Finally, ZFS will need a key to be able access the other storage nodes, as we have already shared keys between the all the nodes run the following commands on each of the storage nodes;
Flocker Control Service
The following commands need to the run on cluster control node;
- control.mydomain.com
At this stage we just need to enable the flocker-control service and configure firewalld with the correct ports for the service;
and for firewalld;
You may have noticed we only enabled the service and haven’t started it, we will be doing this later in the installation.
Flocker Agent Service
The following commands need to the run on the two storage nodes;
- flocker01.mydomain.com
- flocker02.mydomain.com
Firs of all lets put the put the agent configuration in place, make sure you put the hostname of your control node in the file and also make sure that you use the domain name and not the IP address;
and then enable the Flocker Agent Services by running the following commands;
Like the control node we won’t be starting the services just yet, we need to put the authentication certificates in place first.
Generate the Cluster Certificates
Flocker uses SSL certificates for both the client & node authentication, there are a lot of them so please pay close attention to where each certificate and keys needs to be copied.
All of the certificates are generated on your local machine so make sure the following commands are all executed within the working directory we created when installing the commandline tools.
Certificate Authority
First of all we need to create the certificate authority, all of the other certificates will be signed using this one. We will be calling the cluster flocker;
The cluster.key file should not be shared or copied anywhere else other than your local machine.
Control Certificate
The docs make the following notes;
- You should replace <hostname> with the hostname of your control service node; this hostname should match the hostname you will give to HTTP API clients.
- The <hostname> should be a valid DNS name that HTTPS clients can resolve, as they will use it as part of TLS validation.
- It is not recommended as an IP address for the <hostname>, as it can break some HTTPS clients.
Our control node is called control.mydomain.com so we will need to run the following command;
Next up we need to copy the control certificate and key to the control node, give them the correct file name and set the correct permissions;
Node Certificates
Now we have the certificate in place for the control node we need to create the one for each of the storage nodes. To do this run the following command;
Each time you run the command you will be given a certificate with a unique ID (UID), the output should look something like;
The UID in this run was “0ed63f10–7065–4d37-af7c-6128cb5c072f”. Now we have the first node certificate and key we need to copy it to the first of our storage nodes along with the cluster.crt and set the right permissions on the key file;
Next up we need to generate a certificate for our next node, to do this run through the same steps as the first node;
As you can see from the following, this time we got a UID of “f939152e-0ec0–4d7c-ae74–37c899648dca”;
and then copy the files and set the permissions;
Please note: It is important that each node has its own certificate, if you upload the same certificate to more than one node you will cause a conflict with the Flocker control service which will only register a single storage node.
Docker Plug-in Certificate
Next up its the turn of the Docker plug-in, on your local machine run the following commands to generate, copy and set permissions on each of the storage nodes;
API User Certificate
Finally we should create an API user certificate, here will be calling the user flocker-russ
As we are using a Mac we will need to import the certificate into our keychain to be able to use curl to query the control node.
First of all, rename the certificate and key;
and then check the common name in the certificate;
You should see something like;
Make a note of the “CN”, which in the example above is user-flocker-russ. Next up we need to create a .p12 certificate which can be understood by OSX, make sure you enter an export password when prompted;
Finally its time to import the certificate into your keychain by running the following command which will pop-up a prompt asking for the export password you assigned;
Reboot everything
We will be rebooting the following nodes;
- control.mydomain.com
- flocker01.mydomain.com
- flocker02.mydomain.com
Now the nodes are prepared, all of the authentication certificates generated and shared across the cluster it’s time to reboot the control & storage nodes. To do this run the following command on each of the servers;
Once rebooted we can start to test the cluster.
Testing the Cluster
Now everything is rebooted we should have a working Flocker cluster.
API Connection
First of all lets make a connection to the Flocker Control API and list the nodes. The following command will connect using the API certificate we imported into the Keychain, this is called by using the CN which was user-flocker-russ;
You should see something like the following output;
As you can see both nodes, with the UIDs of the node certificates are showing.
Docker Plugin test
Now you have the nodes talking to the control node its time to try and launch a container;
You should see something like the following output;
Now you have written to “wibble:/data” you can launch another cotainer which remounts “wibble:/data” and prints the content of “/data/file.txt” to the screen;
You should see the following;
Further Testing
Now you have the basics tested you should be able to start working through some of the more advanced examples from the official documentation like;