Root server with IPv6-only KVM guests (I): The basic setup

 in English / Linux
Root ser­ver with IPv6-only KVM guests
(1) The basic set­up – (2) NAT64, DNS64, and KVM

I’ve alrea­dy writ­ten that I’ve ren­ted a root ser­ver for mys­elf two years ago. While that machi­ne works and ser­ves things qui­te reli­ab­ly, I feel it aging. It’s an Ubun­tu 16.04 sys­tem, it has only 4 GB of RAM and a qui­te out­da­ted Athlon64 pro­ces­sor. Due to the­se para­me­ters, it is not pos­si­ble to run vir­tua­li­zed set­ups – which I would real­ly like to do to sepa­ra­te ser­vices. E.g. that machi­ne has a com­ple­te E-Mail­ser­ver set­up which I inten­ded to use for my pri­va­te E-Mail but still hesi­ta­te to activa­te as it is rather com­plex and „manu­al­ly set­up”. I fol­lo­wed an inst­ruc­tion gui­de on the inter­net which even its aut­hor says is out­da­ted the­se days.

The­re are other short­co­m­ings like a less-than-opti­mal Let’s-encrypt set­up. Let’s encrypt star­ted almost at the same time as my ser­ver instal­la­ti­on and the­re have been qui­te some opti­mi­za­ti­ons sin­ce then. All in all, my set­up is aging, it is not suf­fi­ci­ent for much more stuff on it and today, you get far more capa­ble hard­ware for not so much more money.

Computer network wiring (symbolic picture)

Com­pu­ter net­work wiring (sym­bo­lic pic­tu­re)

Having the chan­ce to start „from scratch” with the sys­tem design, I thought mys­elf: „Hey, it’s 2018. IPv4 has run out of address space for more than five years. IPv6 has been avail­ab­le for a deca­de or so. Let’s do this in a modern way.” So, I deci­ded on the fol­lo­wing basic sys­tem design:

- I assu­me to have only one IPv4 address but a suf­fi­ci­ent­ly lar­ge IPv6 net­work rou­t­ed to the ren­ted ser­ver.

- I build the sys­tem in a way that all „real” ser­vices run in vir­tu­al machi­nes mana­ged by Linux’ KVM sys­tem.

- The KVM „host” (i.e. the „main”, unvir­tua­li­zed machi­ne) gets the IPv4 address.

- All „guests” get only IPv6 addres­ses. No offi­ci­al IPv4, not even a dual stack with pri­va­te IPv4 and mas­que­ra­ding. Only IPv6.

- Guests can access IPv4-only ser­vices on the inter­net (yes, it’s 2018… hel­lo git​hub​.com…) through a NAT64/DNS64 gate­way on the host.

- Ser­vices on the guests are only gene­ral­ly avail­ab­le from IPv6 addres­ses.

- To ser­ve inco­m­ing IPv4 requests, app­li­ca­ti­on pro­xies on the KVM host for­ward traf­fic to the actu­al ser­vice hand­lers on the guest if nee­ded.

- If for any rea­son a guest sys­tem abso­lute­ly needs its own IPv4 con­nec­tivi­ty, it is added „on top” of the set­up.

The hardware

I use a ren­ted ser­ver in the data cen­ter of a hos­ting pro­vi­der. I am using the Ger­man pro­vi­der „Hetz­ner” alrea­dy for many years, so I went the­re for this pro­ject, too. The hard­ware is a dedi­ca­ted root ser­ver – i.e. „bare metal” – with a Core i7 Hexa­core CPU, 64 GB RAM, and two 3 TB enter­pri­se-gra­de Hard­disks. I ren­ted it from their „used ser­vers shop”.

Hetzner Serverbörse - Manchmal gibt es hier echte Schnäppchen

Hetz­ner Ser­ver­bör­se – Manch­mal gibt es hier ech­te Schnäpp­chen

For the rest of the arti­cles, I will descri­be the set­up in Hetzner’s envi­ron­ment. If you use ano­t­her hos­ting pro­vi­der, you will have to cope with their net­work con­fi­gu­ra­ti­on which might be dif­fe­rent at some points. A cri­ti­cal point might be the IPv6 set­up its­elf as Hetz­ner has a very – eeeerm – „inte­res­ting” approach to this. Howe­ver, instal­la­ti­on of the addi­tio­nal ser­vices and the vir­tu­al machi­nes should be more or less the same regard­less of how your ser­ver is con­nec­ted to the hoster’s net­work.
Ok, let’s start.

IP addres­ses in this gui­de
IP addres­ses in this gui­de are made up. And even if not – the sys­tem I took all con­so­le dia­logs and screen­shots from is not my actu­al ser­ver. It was a spa­re ser­ver which had alrea­dy been can­ce­led and is not under my con­trol any­mo­re.

Initial setup of the host system

I sug­gest that you access the ser­ver in this ear­ly sta­ge by its IP address only. We’ll chan­ge the IPv6 address of the sys­tem later in the install pro­cess. If you want to have a DNS ent­ry, use some­thing inte­rim to throw away later, e.g. „<plannedname>-install.example.org”.

I obtai­ned my ser­ver in Hetzner’s „res­cue sys­tem” which allows the OS instal­la­ti­on through the installimage script. I wan­ted to work with as much default com­pon­ents and con­fi­gu­ra­ti­ons as pos­si­ble, so I deci­ded for the Ubun­tu 18.04 install image offe­red by Hetz­ner.

On gene­ral set­up
I stron­gly advi­se you to always stick with the offe­red set­ups from your hos­ting pro­vi­der as much as pos­si­ble. It increa­ses your chan­ce for sup­port and your chan­ces are much hig­her to find docu­men­ta­ti­on if you run into pro­blems.

Boo­ting into the new ser­ver gives you a wel­co­m­ing log­in screen somehow like this:

You might want to wri­te down MAC address and IP addres­ses of the sys­tem. Note, howe­ver, that they are also inclu­ded in the deli­very e-mail sent by Hetz­ner when the ser­ver is rea­dy.

SSH public keys in Hetzner’s install pro­cess
If you put your ssh public key into your Hetz­ner account and select it in the order pro­cess for the machi­ne, it will not only be put into the res­cue sys­tem but also into the root account of the fresh­ly instal­led machi­ne. If you work this way, you never have to enter any pass­words during the instal­la­ti­on pro­cess. You can also select it each time you request a res­cue sys­tem.

The sys­tem has two hard disks. I use them as a soft­ware RAID 1 as offe­red by the install script. This allows for at least some dis­as­ter reco­very in case of a disk fail­u­re. And for sys­tems like this, I do not install any par­ti­ti­ons at all (apart from the Hetz­ner-sug­gested swap and /boot par­ti­ti­on). The KVM disks will go to qcow2 files which are just put into the host’s file sys­tem. Modern file sys­tems, for­tu­n­a­te­ly, do not have any pro­blems with 200 GB files and this way, all the vir­tu­al guest hard disks are also cove­r­ed by the RAID.

Hetzner’s installimage pro­cess is con­trol­led by a con­fi­gu­ra­ti­on file. Its (stri­ped-down) ver­si­on for the sys­tem I work on reads like this:

Just to be sure: If you use instal­li­mage (or simi­lar instal­la­ti­on rou­ti­nes from other pro­vi­ders) on an exis­ting sys­tem, all data will be dele­ted on that sys­tem. If unsu­re, check twice that you are on the right sys­tem. A mista­ke at this point may be impos­si­ble to cor­rect after­ward!

Instal­ling the sys­tem this way brings a fresh and rather small Ubun­tu sys­tem on the disk. Note that ssh will com­p­lain mas­si­ve­ly about the chan­ged host key of the sys­tem, but that is ok. You’re now boo­ting the instal­led sys­tem which has ano­t­her host key than the res­cue sys­tem you used befo­re.

After having boo­ted into it, I had some hours of remar­kab­ly degra­ded per­for­mance as the RAID 1 had to initia­li­ze the disk dupli­ca­ti­on com­ple­te­ly. Be awa­re of this, your ser­ver will beco­me fas­ter once this is over. Use cat /proc/mdstat to see what’s going on on your hard­disks.

Once the mail ser­ver set­up is avail­ab­le you should enab­le alar­ming messa­ges if the RAID degra­des due to disk fail­u­re. A RAID only pro­tec­ts against hard­ware fail­u­res if the actual­ly fai­led hard­ware is repla­ced quick enough.

Test the res­cue sys­tem
This is a good moment to test whe­ther Hetzner’s res­cue mecha­nism works. Some­ti­mes, the ser­vers are not cor­rec­t­ly con­fi­gu­red in the BIOS and do not load the res­cue sys­tem even if this is requested in the inter­face:
  • Activa­te the „res­cue sys­tem boot” in the Robot inter­face. Select your ssh key so that you do not have to enter a pass­word
  • Reboot the machi­ne.
  • Log­ging in via ssh after 1 to 2 minu­tes should bring up the res­cue sys­tem. Just reboot the machi­ne from the com­mand line – the­re is no need to res­cue now.
  • The sys­tem will come up again into the instal­led sys­tem.

If some­thing is wrong here, con­tact sup­port and let them sol­ve the pro­blem. If you make mista­kes in the Host’s net­work con­fi­gu­ra­ti­on, you will need the res­cue mode to sort things out.

Preparing the network settings of the host

We do now have a fresh­ly instal­led sys­tem. Unfor­tu­n­a­te­ly, it is not qui­te rea­dy to ser­ve as a KVM host. For this, we first have to con­fi­gu­re a net­work bridge on the sys­tem.

I need to say here that this set­up is some „spe­cial” to Hetzner’s IPv6 approach. Unfor­tu­n­a­te­ly, you do only get a sin­gle IPv6 ::/64 net­work. Due to the way how IPv6 works, you can not split this net­work sen­si­b­ly into smal­ler ones. I real­ly sug­gest rea­ding Why Allo­ca­ting a /64 is Not Was­te­ful and Necessa­ry and espe­ci­al­ly The Logic of Bad IPv6 Address Manage­ment to find out how the seman­ti­cs of the IPv6 address space dif­fer from the IPv4 one. If you have a hos­ter who gives you a ::/56 or even ::/48 net­work, you can surely mana­ge your addres­ses dif­fer­ent­ly. Most pro­bab­ly, you will go with a rou­t­ed set­up. Howe­ver, this is bey­ond the scope of this gui­de.

We have to take what we get. This is how I set up my machi­ne as KVM host in the Hetz­ner IPv6 net­work.

The general start

We’ll see that the­re are dif­fe­rent net­work set­ups. Howe­ver, some steps have to be per­for­med nevertheless:

  • First, I enab­led IPv6 for­war­ding glo­bal­ly:
  • Also enab­le this set­ting in /etc/sysctl.conf to make it per­ma­nent.
  • Use ip a to get device name and MAC address of the phy­si­cal net­work card of the sys­tem:

    Your net­work device’s name may dif­fer. It can be some­thing like enpXsY as in this examp­le or enoX. On all modern Linux dis­tri­bu­ti­ons, it will begin with en, howe­ver…

Here the com­mon track for all sys­tems ends. In the Linux world, mul­ti­ple net­work con­fi­gu­ra­ti­on set­ups have evol­ved over time. The most com­mon ones are:

  • Direct set­up in con­fi­gu­ra­ti­on files in /etc/network. This is old-school net­wor­king set­up, espe­ci­al­ly when com­bi­ned with a Sys­tem-V-initia­li­sa­ti­on pro­cess. I do not cover this here but you find a pletho­ra of instal­la­ti­on gui­des on the inter­net for this.
  • Sys­temd-based con­fi­gu­ra­ti­on with files in /etc/systemd/network. This is how many modern dis­tri­bu­ti­ons hand­le sys­tem start and net­work set­up the­se days. Ubun­tu did it until 17.04, Hetzner’s Ubun­tu did it lon­ger. I cover this two sec­tions fur­ther.
  • Net­plan with a con­fi­gu­ra­ti­on in /etc/netplan. This kind of „meta-con­fi­gu­ra­ti­on” is used by Ubun­tu sin­ce 17.10 and by Hetz­ner sin­ce Novem­ber 2018 for 18.04 and 18.10. I descri­be the nee­ded chan­ges in the fol­lo­wing sec­tion.

Ubuntu 18.04 with Netplan

Ubun­tu 18.04 comes with a rela­tively new tool named Net­plan to con­fi­gu­re the net­work. Sin­ce about Novem­ber 2018, Hetz­ner uses this set­up in their install pro­cess. Note that ear­lier Ubun­tu instal­la­ti­ons are pro­vi­ded with the sys­temd-net­workd-based set­up descri­bed below.

Net­plan uses con­fi­gu­ra­ti­on files with YAML syn­tax. In most cases, the­re is only one file. Chan­ge the net­work con­fi­gu­ra­ti­on like this:

  • You find the net­work con­fi­gu­ra­ti­on in /etc/netplan/01-netcfg.yaml.
    It looks somehow like this:

    Chan­ge it like this:

    You dis­able both DHCP pro­to­cols on the phy­si­cal device and attach it to the new­ly defi­ned bridge device br0. That bridge gets all defi­ni­ti­ons from the phy­si­cal device. It also gets the MAC address of the phy­si­cal device, other­wi­se, Hetzner’s net­work will not rou­te any packets to it.

  • After a reboot, the net­work device list should look like this:

    Note that the phy­si­cal device enp2s0 and the bridge br0 have the same MAC address. This is inten­tio­nal!

  • You should test that you can log in to the sys­tem through both IPv6 and IPv4 pro­to­col.

Ubuntu 18.04 and other systems with systemd-networkd

Until Octo­ber 2018, Hetz­ner used a sys­temd-net­workd-based set­up on Ubun­tu, even with 18.04. If you have such a sys­tem, you get the same result in a dif­fe­rent way. Crea­ting a bridge for vir­tu­al machi­nes using sys­temd-net­workd exp­lains the basics nice­ly.

With this sys­tem, you chan­ge the net­work set­up as fol­lows:

  • Go to /etc/systemd/network.
  • Defi­ne a bridge device in file 19-br0.netdev:

    It is extre­me­ly important to defi­ne the MAC address, or Hetz­ner will not rou­te traf­fic to the sys­tem. STP seems not man­dato­ry, does not hurt eit­her. I kept it in.

  • Assign bridge to phy­si­cal device in 20-br0-bind.network:

  • Copy the ori­gi­nal file crea­ted by Hetz­ner (here: 10-eno1.network) to 21-br0-conf.network and replace the matching name from the phy­si­cal device to the bridge. In fact, you only replace the eno1 (or wha­te­ver you net­work device’s name is) with br0:

  • Rena­me the ori­gi­nal file 10-eno1.network to some­thing not detec­ted by sys­temd, e.g. 10-eno1.networkNO. Keep it around in case some­thing goes wrong.

After the­se chan­ges, the phy­si­cal device has not any net­works atta­ched. This is important so that the bridge can grab it on initia­li­za­ti­on. Let’s see whe­ther ever­ything works:

  • Reboot the sys­tem
  • If some­thing goes wrong: Boot into res­cue sys­tem, mount par­ti­ti­on, rena­me 10-eno1.networkNO back into ori­gi­nal name ending in .network. Reboot again. Inves­ti­ga­te. Repeat until it works…

Ensure correct source MAC address

Our vir­tu­al machi­nes will have their own MAC addres­ses. Other­wi­se, the IPv6 auto con­fi­gu­ra­ti­on would not work. Unfor­tu­n­a­te­ly, the­se MAC addres­ses will also leak through the bridge into the pro­vi­der net­work and that might lead to trou­ble as the pro­vi­der does only accept the actu­al assi­gned MAC address of the main ser­ver as valid.

To pre­vent such pro­blems per­form MAC address rewri­ting using the ebtables com­mand:

I’ve added this to /etc/rc.local. On a default instal­la­ti­on of Ubun­tu 18.04, this file does not exist. If you crea­te it, make it look like this:

Replace the address in the examp­le with your actu­al phy­si­cal MAC address! Also, make the file exe­cu­ta­ble with chmod +x /etc/rc.local.

„The inter­net” claims that you need to add other files to sys­temd for /etc/rc.local being eva­lua­ted in Ubun­tu 18.04. At least for me this was not nee­ded, it „just worked”. Check whe­ther the rule has been added:

Reboot the sys­tems once more to check if the rule sur­vi­ves a reboot.

Install the Router Advertisement Daemon radvd

We have now a basi­cal­ly working KVM host sys­tem to be. Let’s start popu­la­ting it with ser­vices. The first one is the „rou­ter adver­ti­se­ment”. It’s more or less the IPv6-ver­si­on of the noto­rious DHCP ser­vice used in IPv4 set­ups to cen­tra­li­ze the IP address manage­ment.

For this ser­vice, we use the radvd rou­ter adver­ti­se­ment dae­mon on the bridge device so that our vir­tu­al machi­nes get their net­work set­up auto­ma­ti­cal­ly by rea­ding IPv6 rou­ter adver­ti­se­ments.

  • Install radvd and also radvdump for tes­ting through the usu­al Debian/Ubuntu
  • Crea­te the con­fi­gu­ra­ti­on file /etc/radvd.conf. It should con­tain the fol­lo­wing defi­ni­ti­ons:

    This initi­al radvd con­fig is pret­ty simp­le. We miss out RDNSS adver­ti­se­ments as we will install a bind later on the host.

  • Start radvd and make it a per­ma­nent ser­vice (com­ing up auto­ma­ti­cal­ly after reboot) using

  • If you start radvdump soon after star­ting radvd, you will see the announ­ce­ments sent by radvd in irre­gu­lar inter­vals. For some rea­son, radvd seems to stop sen­ding unsol­ci­ta­ted adver­ti­se­ments after some time if noo­ne is lis­ten­ing.

Change IPv6 address

Once radvd is run­ning, the Host will assign its­elf an addi­tio­nal IP address con­struc­ted from SLAAC rules. At least mine did it even though comments in /etc/sysctl.conf said, it wouldn’t once IPv6 for­war­ding is glo­bal­ly enab­led. Howe­ver, if this hap­pens, you will see ano­t­her IPv6 address assi­gned to the bridge device.

While it is abso­lute­ly no pro­blem to have mul­ti­ple IPv6 addres­ses on the same device, it can make con­fi­gu­ra­ti­on of ser­vices more dif­fi­cult as the cor­rect address for out­go­ing messa­ges has to be selec­ted cor­rec­t­ly. I uni­fied both addres­ses by making the sta­ti­cal­ly con­fi­gu­red one the same as the once which SLAAC pro­du­ces.

For this, just get rid of the address ending in „0:0:0:2” (writ­ten as „…::2”) the ser­ver has been initia­li­zed with during install as Hetzner’s default. Do like this:

  • Use ip a to get the address. It’s the one having that cha­rac­te­ris­tic ff:fe sequence in the midd­le of the host part. Wri­te it down!
  • Test that ssh to that address works. It will if you just fol­low this gui­de, but if you have some fire­walls some­whe­re lur­king around, the con­nec­tion will fail. Make sure it works.
  • Load the net­work con­fi­gu­ra­ti­on again. In the sys­temd-net­workd-case it’s /etc/systemd/network/21-br0-conf.network, for Net­plan it’s /etc/netplan/01-netcfg.yaml.
  • Exchan­ge the IPv6 address of the bridge device to the one dis­co­ve­r­ed by SLAAC. E.g. if you have

    in the sys­temd-net­workd con­fi­gu­ra­ti­on file, chan­ge it to

    In /etc/netplan/01-netcfg.yaml, look for

    and chan­ge it to

  • Reboot the sys­tem. In theo­ry, systemctl restart systemd-networkd should be suf­fi­ci­ent in the sys­temd-net­workd case, but my sys­tem beha­ved stran­ge­ly after that.
  • ssh to the new address should work. If it doesn’t, again use the res­cue sys­tem to sort it out. After all, you can set the host part of the IPv6 address of the Host to anything bet­ween 0:0:0:2 and ffff:ffff:ffff:ffff. The Hetz­ner infra­st­ruc­tu­re rou­tes traf­fic to all addres­ses to the inter­face.
  • The sys­tem should now have three IP addres­ses on the bridge device: The IPv4 address, the glo­bal IPv6 address and the link-local IPv6 address star­ting with fe80. It looks like this:

After having chan­ged the IP address this way, enter it into DNS:

  • Add an AAAA record in the domain the sys­tem should be reach­a­ble in.
  • Add a PTR record in the hoster’s rever­sal IP ent­ries. If the­re is alrea­dy an ent­ry for the for­mer address, you can remo­ve it by sim­ply wiping out the ser­ver name and pres­sing „Enter”.
  • While you’re at it, also add the A record and the PTR record for the IPv4 address of the Host.
Keep DNS time-to-live short!
I stron­gly sug­gest that you set the TTL for all DNS ent­ries as short as pos­si­ble during the set­up, some­thing bet­ween 2 and 5 minu­tes. If you make a mista­ke and you have a TTL of mul­ti­ple hours or even a day, you may have serious issu­es with the name ser­vice as long as the TTL of the wrong ent­ries has not run out ever­y­whe­re.

At this sta­ge, lean back for a moment! The dif­fi­cult part is done. You have the net­work set­up of your KVM host up and run­ning. The rest is much easier and will not poten­ti­al­ly kill net­work access to the sys­tem. Also, the stuff com­ing now is much less pro­vi­der-spe­ci­fic. While the initi­al net­work set­up might work con­si­der­a­b­ly dif­fe­rent with ano­t­her hos­ting pro­vi­der, chan­ces are good that the fol­lo­wing steps are the same regard­less of whe­re you have pla­ced your host.

The res­cue sys­tem IP address
If you ever have to reboot your ser­ver into Hetzner’s res­cue sys­tem, keep in mind that it will get its ori­gi­nal IPv6 address ending in ::2. You will not be able to access it through its DNS name. You might want to add a DNS ent­ry for <servername>-rescue.example.org for such cases. Of cour­se, you have to remem­ber that, too…

In part II of this arti­cle series, we con­ti­nue with NAT64, DNS64, and – final­ly – the first vir­tu­al machi­nes.

History

  • 2019-04-28: MAC source rewri­ting added.

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.