Please note: This is not authoritative information; if you use it and kittens pop out of your router or there’s some way simpler/better way to do things:
don’t blame me for the kitten thing
please document it and send me a link so I can learn from you.

The Problem.

IPv6. It’s a thing. Who even wants NAT anyway? Docker’s neat, it lets you run containers and stuff. The docs about enabling IPv6 on Docker make it look so simple. Just assign an IP range, right?

Edit /etc/docker/daemon.json, set the ipv6 key to true and the fixed-cidr-v6 key to your IPv6 subnet. In this example we are setting it to 2001:db8:1::/64.
{
  "ipv6": true,
  "fixed-cidr-v6": "2001:db8:1::/64"
}
Save the file.
Reload the Docker configuration file.
$ systemctl reload docker
You can now create networks with the --ipv6 flag and assign containers IPv6 addresses using the --ip6 flag.

Simple, right? Turns out, not everyone agrees. At the bottom of the docs you can rate them, and here’s what it looked like today:

Yerp.

The Setup.

I’ve been playing with Hashicorp Consul / Nomad / Vault lately to get my cluster of Raspberry Pi 4’s doing clustery things. Because everybody’s gotta cluster and automate all the things, right?

Let’s start with the basics.

I’ve got three nodes connected via their LAN ports.
They run Ubuntu (because it’s easy, shush).
My ISP issued me a /56 subnet. (Aussie Broadband, use my referral code 3474095 and we both get money!)
I manage all the configuration with Ansible so I don’t have to remember what stupid line in each file to edit when I blat something.
I want Docker to run on all of them, and I don’t want IPv4 anywhere I can avoid it, because it’s icky.

There’s so much more to setting up Docker than the docs imply. I found a much better version here which explains all the concepts required and deployment methods. The v17 at the top implies it might just be an older, better version of the docs. I saved the page here as a PDF just because it might go away and I’d miss it.

The Plan.

Configure Docker
Route traffic
Allow the traffic

The HeistImplementation.

Configuring Docker

So… you need to assign a subnet to each server. Assigning them manually is a bit painful. You’d have to calculate an address for each node, then write the file and bleh. IPv6 is huge and we’re using Ansible, so let’s do it automagically. Every network interface (should) have a unique MAC address, so let’s use that to break things up. I use the Ansible template module to write a new /etc/docker/daemon.json, then restart the service if/when it changes.

Here’s my template file:

{
  "ipv6": true,
  "fixed-cidr-v6": "{{my_ipv6_network}}:{{''.join(ansible_default_ipv6['macaddress'].split(':')[-2:])}}::/80",
}

There’s only one thing I haven’t entirely sorted out but it’s not really an issue - I have a group variable called my_ipv6_network which has the big subnet bit for the start of the address in it. For this example we’re using 2001:DB8:1212:3434.

Basically, start with our known network, then tack on the last four digits of the LAN interface MAC address and make it a /80. This’ll be … enough for every single possible MAC address to fit inside. While writing this article, I was originally using a /72 and did the sums… turns out I was being hilariously generous and the /80 is way more appropriate.

The tasks in Ansible are pretty simple:

- name: docker_daemon_conf
  template:
    src: docker-daemon.json
    dest: /etc/docker/daemon.json
  register: nomad_ipv6_docker_daemon

- name: docker_restart
  service:
    name: docker
    state: restarted
  when: (nomad_ipv6_docker_daemon is defined) and (nomad_ipv6_docker_daemon.changed == True)

Write the file, restart the daemon.

Now, things aren’t that simple. This’ll get you part of the way - you’ll be able to communicate from the docker host to the containers - but traffic outside won’t work because there’s no return routing and NDP won’t work.

Neighbour Discovery Protocol Proxying

Neighbour Discovery Protocol NDP is a whole thing in IPv6. I still don’t really understand it, but the aforementioned better-than-official-docs page explains a lot of it better than I ever will. Basically, we need to pass “where are you” queries and their responses across the networking stack of the Docker host.

I’m using Daniel Adolfsson’s NDP Proxy Daemon (ndppd). It’s packaged in Ubuntu so that’s even better. The NDP Proxy RFC (RFC4389) is still technically experimental, but hey - it works, right?

Ansible to the rescue, here’s some more tasks:

- name: ndppd_install
  package:
    name: ndppd
    state: present

- name: ndppd_conf
  template:
    src: ndppd.conf
    dest: /etc/ndppd.conf
  register: ndppd_conf

- name: ndppd_restart
  service:
    name: ndppd
    state: restarted
    enabled: yes
  when: (ndppd_conf is defined) and (ndppd_conf.changed == True)

- name: ndppd_started
  service:
    name: ndppd
    state: started
    enabled: yes
  when: (ndppd_conf is not defined)

Install the package, write the configuration, then make sure the thing’s running. The template, you ask? Here:

proxy {{ansible_default_ipv6['interface']}} {
    rule {{my_ipv6_network}}:{{''.join(ansible_default_ipv6['macaddress'].split(':')[-2:])}}::/80 {
        auto
    }
}

Looks familiar, right? This is why we template things. Less manually editing files. This takes the default IPv6 interface and sets up ndppd to automagically handle the negotiation.

The config file looks like this in the end. Ansible’s a great way to grab a file from one or more hosts… I query the nomad host group.

$ ansible nomad -a 'cat /etc/ndppd.conf'
host-a1a1.example.com| CHANGED | rc=0 >>
proxy eth0 {
    rule 2001:DB8:1212:3434:a1a1::/80 {
        auto
    }
}
host-917a.example.com| CHANGED | rc=0 >>
proxy eth0 {
    rule 2001:DB8:1212:3434:917a::/80 {
        auto
    }
}
host-ef91.example.com| CHANGED | rc=0 >>
proxy eth0 {
    rule 2001:DB8:1212:3434:ef91::/80 {
        auto
    }
}

If you didn’t guess, I assign hostnames for these nodes based on their MAC address… I don’t use fancy naming conventions - what they do is what they’re called.

I’ll skip all the Nomad/Consul/Vault stuff for now, but after spinning up a container, it gets a globally-routable address!

$ docker inspect qbittorrent-microservice-cba5db93-78fa-060a-2fcd-ccfb3506045b | jq '.[] | .NetworkSettings' | grep GlobalIPv6
  "GlobalIPv6Address": "2001:DB8:1212:3434:ef91:242:ac11:2",
  "GlobalIPv6PrefixLen": 80,

Kernel things

NDP normally gets gobbled up by the kernel so there’s a few tweaks you need to do in sysctl. It’s better explained in the docs, but basically just tell the kernel to do its thing.

Ansible has a sysctl module which I use in this stage.

- name: docker_sysctl_fix_proxy
  sysctl:
    sysctl_file: /etc/sysctl.d/99-docker-router.conf
    name: net.ipv6.conf.eth0.proxy_ndp
    value: "1"
    sysctl_set: yes

- name: docker_sysctl_fix_ra
  sysctl:
    sysctl_file: /etc/sysctl.d/99-docker-router.conf
    name: net.ipv6.conf.eth0.accept_ra
    value: "2"
    sysctl_set: yes

Firewall things

I use UFW to manage the firewall on my Debian-like systems. Since your instances are routing traffic, I’m allowing that by default in the policy. I’m sure you could do this better, but it works for me - I’ve got other mitigating controls in place 😁

Ansible’s UFW module is used here:

- name: ufw_routing
  community.general.ufw:
    direction: routed
    policy: allow

Note: you’ll need to install the community general collection with ansible-galaxy collection install community.general.

The Result.

Success!

Docker is assigning an address from a small 😳 IPv6 subnet to containers.
NDPPD is making sure traffic knows where to go. 🪄
UFW is allowing it to happen. 👮‍♀️

IPv6 With Docker and Ansible