Critters in a jar: Running CockroachDB in a FreeBSD jail

Note: this blog post was updated on September 21, 2017.

Jails are FreeBSD’s native solution to contain and isolate server processes. They are an alternative to (and predate) Linux cgroups, Solaris zones, and other OS-level process isolation technologies (the technologies that underlie Docker, CoreOS and a few others) .

This blog post will explain how to natively run CockroachDB in a FreeBSD jail. This is lighter weight and as secure as running Docker on FreeBSD.

Manually encapsulating CockroachDB using Linux cgroups is no easy task, which is why tools like Docker exist in the first place. By comparison, running server processes natively in FreeBSD jails is straightforward and robust.

Call for contributions: We welcome patches to increase the portability of CockroachDB, as well as external support to create and maintain installable CockroachDB packages for various Unix flavors and distributions, including a BSD source port.

Disclaimer: We do not yet claim to support FreeBSD compatibility. The solution below has been tested on FreeBSD 10.3-STABLE with today’s version of CockroachDB. Tomorrow, YMMV.

Prerequisites to build CockroachDB

Pending an official package for CockroachDB on FreeBSD, you will need to build from source.

For now, CockroachDB requires a 64-bit machine.

Your system should provide the C and C++ development tools. FreeBSD’s native C and C++ compilers are appropriate. You also need the additional packages for Bash, Git, GNU make, and Go. These can be installed with:

pkg install bash go gmake git

Ensure that go version 1.8 (for CockroachDB 1.0 or 1.1) or 1.9 (for CockroachDB 1.2) or higher is installed as a result: go version.

Creating a CockroachDB binary

The official instructions (select Linux and then Build from Source) apply with some tweaks:

• Ensure that /tmp is mounted as tmpfs or from a ramdisk prior to the build. We’ve seen that builds can be very slow otherwise.
• You’ll need to run the build with gmake instead of make.

Testing your build

Ensure that $GOPATH/bin is in your PATH and the cockroach binary is there.

Ensure that the Linux “/proc emulation” is mounted in /compat/linux/proc. If not done yet, do this with:

sudo mkdir -p /compat/linux/proc
sudo mount -t linprocfs /dev/null /compat/linux/proc

(Note: Linux compatibility files / packages / libraries are not needed further. CockroachDB uses Linux’s procfs to inspect system properties via gosigar**. If/when gosigar evolves to read FreeBSD properties natively, CockroachDB will not need linprocfs any more.)**

Then as per the official CockroachDB docs:

cockroach start --insecure --host=localhost

Then in another terminal:

cockroach sql --insecure

There, play around with SQL, for example:

CREATE DATABASE test;
CREATE TABLE test.t (x INT);
INSERT INTO test.t(x) VALUES (42);
SELECT * FROM test.t;

If the program works as expected, the last query should return the inserted row from the table.

Starting/stopping CockroachDB as a system daemon

FreeBSD uses scripts in /etc/rc.d and /usr/local/etc/rc.d for starting and stopping system daemons. Which daemons are actually enabled to run during startup is specified using definitions in the common file /etc/rc.conf.

To integrate CockroachDB into a standard FreeBSD startup, we need to create a suitable startup script in /usr/local/etc/rc.d or /etc/rc.d. One such script is the following:

#!/bin/sh
#
# PROVIDE: cockroach
# REQUIRE: LOGIN
# KEYWORD: shutdown

# Add the following lines to /etc/rc.conf to enable cockroachdb:
# cockroach_enable="YES"
# cockroach_flags="<set as needed>"

. /etc/rc.subr

name="cockroach"
rcvar=cockroach_enable
pidfile=/var/run/cockroach.pid

load_rc_config $name

: ${cockroach_enable="NO"}
: ${cockroach_user="cockroach"}
: ${cockroach_flags="--insecure --host=localhost --store=path=/home/cockroach"}

start_cmd=${name}_start
stop_cmd=${name}_stop

cockroach_start() {
  daemon -u $cockroach_user -p $pidfile /usr/local/bin/cockroach start $cockroach_flags
}

cockroach_stop() {
 [ -f $pidfile ] \
   && /usr/local/bin/cockroach quit --insecure \
   || echo ${name} not running? \(check ${pidfile}\)
}

run_rc_command "$1"

This script runs a CockroachDB node using account cockroach. The default settings for the configurable variable cockroach_flags instructs it to store database files (--store) in /home/cockroach and only listen to a TCP port on the localhost interface. The latter is only necessary until you follow the instructions to secure your cluster and remove --insecure. The flags specified via cockroach_flags in /etc/rc.conf are passed as-is to the database start command, cockroach start, so we can also use this later to set certificates, addresses and the like.

It uses daemon(8) because the CockroachDB server does not daemonize itself.

If you want to use this script, be sure to name it cockroach (without ".sh"), ensure it is executable (chmod +x) and add the appropriate enabling flags to /etc/rc.conf (see comments at start of script).

Running CockroachDB in a FreeBSD Jail

Jails are FreeBSD’s solution for process encapsulation, more commonly called “containers.” Jails are both older than most similar technologies and comparatively more feature-complete.

In a nutshell, running CockroachDB in a FreeBSD jail amounts to the following steps:

1. Create a standard FreeBSD jail as per the FreeBSD handbook.
2. Ensure that the virtual filesystem linprocfs is mounted in the jail’s /compat/linux/proc
3. Create a user account cockroach for CockroachDB in the jail.
4. Copy the above cockroach binary in the jail’s /usr/local/bin
5. Copy the above startup script in the jail’s as /usr/local/etc/rc.d/cockroach
6. Add cockroach_enable=YES to the jail’s /etc/rc.conf, and tune cockroach_flags and cockroach_db_directory as desired. In particular you probably want to change --host=localhost to --host=127.0.1.1 to match the Jail internal address. Note that by default CockroachDB runs in “insecure” mode, i.e. with SSL to clients and between nodes disabled. To change this, you need to set up certificates and configure cockroach_flags as per the CockroachDB documentation.
7. Start the jail.

While these instructions should be sufficient to seasoned jails users, here is a step-by-step tutorial for jails beginners. This uses ezjail-admin from the ezjail package.

1. Create an alias to the network loopback interface:
   • Add cloned_interfaces="lo1" to /etc/rc.conf
   • Activate the interface with: service netif cloneup
2. If not installed yet, install ezjail:

   pkg install ezjail
   

3. If not created yet, create the jail template; note that we do not need FreeBSD ports to run CockroachDB so we do not need to pass the parameter -p:

   ezjail-admin install
   

4. Create a new jail, using the aliased loopback interface to attach the jail:

   ezjail-admin create crjail “lo1:127.0.1.1”
   

   This creates the new jail in /usr/jails/crjail. From now on we use “$J” in the examples to refer to this directory.

5. Start the jail manually for initial configuration:

   ezjail-admin onestart crjail
   

6. Edit $J/etc/hosts and change the localhost address to 127.0.1.1.
7. Edit $J/etc/resolv.conf to add a DNS resolver, for example Google’s:

   echo nameserver 8.8.8.8 >$J/etc/resolv.conf
   

8. Mount the linprocfs filesystem onto the jail:

   mkdir -p $J/compat/linux/proc
   mount -t linprocfs linproc $J/compat/linux/proc
   

9. Create a new cockroach user in the jail:

   chroot $J adduser
   

   You can choose a random password for the account. It will not be used by CockroachDB. We assume below that you keep the default home directory setting /home/cockroach here.

10. Copy the cockroach binary generated above to $J/usr/local/bin. You may need to create this directory yourself. (use mkdir -p)
11. Copy the cockroach startup script above to $J/usr/local/etc/rc.d. Again, you may need to create this directory yourself.
12. Add cockroach_enable=YES to $J/etc/rc.conf:

    echo cockroach_enable=YES >>$J/etc/rc.conf
    

13. Restart your jail to start CockroachDB:

    ezjail-admin onestop crjail
    ezjail-admin onestart crjail
    

14. Ensure that CockroachDB has started and is reachable via 127.0.1.1:

    curl http://127.0.1.1:8080
    

    If CockroachDB does not appear to be running, you can investigate whether it has started by logging into the jail console with ezjail-admin console crjail and looking into /home/cockroach from there.

15. If this works, congratulations! You successfully jailed the critter.

Where to go from here?

The steps above isolate the CockroachDB server process from other processes in the same system. The server becomes reachable only from the local address 127.0.1.1.

From there you can move forward in various directions:

• Automate starting up your jail(s) when the host system comes up (e.g. add ezjail_enable=YES to the host system’s /etc/rc.conf).
• Adjust cockroach_flags in the jail’s /etc/rc.conf to spread the data store over multiple disks and/or ramdisks using CockroachDB’s --store flag.
• Set up resource limits on the jail, for example memory or disk usage, see jail(8).
• Add an alias to the LAN network interface to the jail, so that the CockroachDB server can connect to other nodes in the cluster. (be sure to configure security certificates!)
• Set up Mandatory Access Control (MAC) to further restrict CockroachDB’s security attack surface.
• Monitor the database server using Nagios (or your own favorite monitor tool) to set up administrator notifications and automatic restarts whenever the database process fails.

Really from this point CockroachDB becomes one among your other contained server processes. All standard FreeBSD best practices and literature apply.

References

• The CockroachDB documentation
• The FreeBSD handbook