First i had ipsec/gre for interconnecting my nodes itself, but now i completely switched to WireGuard. As it is really simple to setup (if you can compile an kernel module or use dkms), it is fast, high throughput and you don't need gre as with ipsec to use for example OSPF on the links.
As i don't want to setup a full mesh with WireGuard, i decided to only setup a few "direct tunneled" connections between some nodes and announce Node specific Loopback IPs via OSPF. With this, i had every Node reachable by any node, and if i want to setup an direct tunnel between two nodes, OSPF will get aware of this connection and use this instead of indirect connection over an other node.
The Loopback IPs will be used later for iBGP Peering (full mesh) between all 6 Nodes.
Every Node gets a minimum of two Loopback IPs, one as iBGP Peering IP and one as eBGP (or DN42) Peering IP. The Loopback IPs are shown in the drawing below, also the Tunnel setups and iBGP Peerings:
Wireguard Tunnel Setup
For Installation just look on the WireGuard Website for support. I use an ifupdown approach on Debian to setup my iBGP Tunnels (and most of the eBGP Tunnels, too).
You need a config file in /etc/wirguard/ and a corresponding ifupdown config in /etc/network/interfaces.d/. For example these are my conf files for an WireGuard Tunnel between dn42-gw <-> dn42-uk01:
On dn42-gw:
/etc/network/interfaces.d/wg-ibgp-uk01.conf
auto wg-ibgp-uk01
iface wg-ibgp-uk01 inet static
address 172.20.175.195
netmask 255.255.255.255
pointopoint 172.20.175.198
pre-up ip link add wg-ibgp-uk01 type wireguard
pre-up wg setconf wg-ibgp-uk01 /etc/wireguard/wg-ibgp-uk01.conf
post-up ip -6 addr add fe80::de49:211/64 dev wg-ibgp-uk01
post-down ip link del wg-ibgp-uk01
/etc/wireguard/wg-ibgp-uk01.conf
[Interface]
ListenPort = 23909
PrivateKey = <NOT SHOWN>
[Peer]
PublicKey = qAr6EBKEQ2D20H5bs+USOOeRJHEijEy/IAxrLxFLBRM=
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = dn42-uk01.weiti.org:23110
On dn42-uk01:
/etc/network/interfaces.d/wg-ibgp-de02
auto wg-ibgp-de02
iface wg-ibgp-de02 inet static
address 172.20.175.198
netmask 255.255.255.255
pointopoint 172.20.175.195
pre-up ip link add wg-ibgp-de02 type wireguard
pre-up wg setconf wg-ibgp-de02 /etc/wireguard/wg-ibgp-de02.conf
post-up ip -6 addr add fe80::de49:225/64 dev wg-ibgp-de02
post-down ip link del wg-ibgp-de02
/etc/wireguard/wg-ibgp-de02.conf
[Interface]
ListenPort = 23110
PrivateKey = <NOT SHOWN>
[Peer]
PublicKey = zuRXi9t+uMW9CxlN1gy0X8ZCxeQ9Xm8RwlyIr05SbBU=
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = dn42-gw.weiti.org:23909
And on the wg-* links i use OSFP to announce the Loobpack IPs:
protocol ospf O_OSPF {
table T_OSPF;
area 0.0.0.0 {
interface "lo" {
stub;
};
interface "wg-ibgp*" {
};
};
}
This is the OSPF Config part of my bird.conf. For IPv6 i use the same config. Within the pipe from T_OSPF to master, i allow only /32 or /128 from the corresponding Loopback Network. An example of my LO Addressing, OSPF Neighbor and Routing Table from dn42-uk01 will look like the following examples:
Loopback IPs
ip addr list lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 172.20.175.198/32 scope global lo
valid_lft forever preferred_lft forever
inet 172.20.175.225/32 scope global lo
valid_lft forever preferred_lft forever
inet6 fdf7:17d5:de49:4::1/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::666/128 scope link
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
OSPF Neighbor Table
birdc 'show ospf neighbor'
BIRD 1.6.0 ready.
O_OSPF:
Router ID Pri State DTime Interface Router IP
172.20.175.197 1 Full/PtP 00:38 wg-ibgp-fr01 172.20.175.197
172.20.175.199 1 Full/PtP 00:32 wg-ibgp-au01 172.20.175.199
172.20.175.193 1 Full/PtP 00:35 wg-ibgp-de01 172.20.175.193
172.20.175.195 1 Full/PtP 00:36 wg-ibgp-de02 172.20.175.195
OSPF Routing Table
birdc 'show route table T_OSPF'
BIRD 1.6.0 ready.
172.20.175.215/32 via 172.20.175.197 on wg-ibgp-fr01 [O_OSPF 2017-12-31] * I (150/10) [172.20.175.197]
172.20.175.210/32 via 172.20.175.193 on wg-ibgp-de01 [O_OSPF 2017-12-29] * I (150/10) [172.20.175.193]
172.20.175.211/32 via 172.20.175.195 on wg-ibgp-de02 [O_OSPF 09:12:02] * I (150/10) [172.20.175.195]
172.20.175.220/32 via 172.20.175.195 on wg-ibgp-de02 [O_OSPF 09:12:41] * I (150/20) [172.20.175.196]
172.20.175.196/32 via 172.20.175.195 on wg-ibgp-de02 [O_OSPF 09:12:41] * I (150/20) [172.20.175.196]
172.20.175.197/32 via 172.20.175.197 on wg-ibgp-fr01 [O_OSPF 2017-12-31] * I (150/10) [172.20.175.197]
172.20.175.198/32 dev lo [O_OSPF 2017-06-01] * I (150/0) [172.20.175.198]
172.20.175.199/32 via 172.20.175.199 on wg-ibgp-au01 [O_OSPF 2017-12-17] * I (150/10) [172.20.175.199]
172.20.175.193/32 via 172.20.175.193 on wg-ibgp-de01 [O_OSPF 2017-12-29] * I (150/10) [172.20.175.193]
172.20.175.195/32 via 172.20.175.195 on wg-ibgp-de02 [O_OSPF 09:12:02] * I (150/10) [172.20.175.195]
172.20.175.252/32 via 172.20.175.193 on wg-ibgp-de01 [O_OSPF 2017-12-29] * I (150/10) [172.20.175.193]
172.20.175.253/32 via 172.20.175.193 on wg-ibgp-de01 [O_OSPF 2017-12-29] * I (150/10) [172.20.175.193]
172.20.175.254/32 via 172.20.175.193 on wg-ibgp-de01 [O_OSPF 2017-12-29] * I (150/10) [172.20.175.193]
172.20.175.249/32 via 172.20.175.195 on wg-ibgp-de02 [O_OSPF 09:12:02] * I (150/10) [172.20.175.195]
172.20.175.250/32 via 172.20.175.195 on wg-ibgp-de02 [O_OSPF 09:12:02] * I (150/10) [172.20.175.195]
172.20.175.251/32 via 172.20.175.193 on wg-ibgp-de01 [O_OSPF 2017-12-29] * I (150/10) [172.20.175.193]
172.20.175.230/32 via 172.20.175.199 on wg-ibgp-au01 [O_OSPF 2017-12-17] * I (150/10) [172.20.175.199]
172.20.175.225/32 dev lo [O_OSPF 2017-06-01] * I (150/0) [172.20.175.198]
OSPFv3 Neighbor Table
birdc6 'sh ospf neighbors'
BIRD 1.6.4 ready.
O_OSPF:
Router ID Pri State DTime Interface Router IP
172.20.175.196 1 Full/PtP 00:37 wg-ibgp-us01 fe80::de49:220
172.20.175.197 1 Full/PtP 00:34 wg-ibgp-fr01 fe80::42
172.20.175.195 1 Full/PtP 00:34 wg-ibgp-gw fe80::de49:211
172.20.175.198 1 Full/PtP 00:33 wg-ibgp-uk01 fe80::de49:225
172.20.175.199 1 Full/PtP 00:30 wg-ibgp-au01 fe80::de49:230
OSPFv3 Routing Table
birdc6 'show route table T_OSPF'
BIRD 1.6.0 ready.
fdf7:17d5:de49::250/128 via fe80::de49:211 on wg-ibgp-de02 [O_OSPF 09:11:57] * I (150/10) [172.20.175.195]
fdf7:17d5:de49::251/128 via fe80::1 on wg-ibgp-de01 [O_OSPF 2017-12-29] * I (150/10) [172.20.175.193]
fdf7:17d5:de49::249/128 via fe80::de49:211 on wg-ibgp-de02 [O_OSPF 09:11:57] * I (150/10) [172.20.175.195]
fdf7:17d5:de49::42/128 via fe80::1 on wg-ibgp-de01 [O_OSPF 2017-12-29] * I (150/10) [172.20.175.193]
fdf7:17d5:de49::43/128 via fe80::1 on wg-ibgp-de01 [O_OSPF 2017-12-29] * I (150/10) [172.20.175.193]
fdf7:17d5:de49:5::1/128 via fe80::de49:230 on wg-ibgp-au01 [O_OSPF 2017-12-17] * I (150/10) [172.20.175.199]
fdf7:17d5:de49:4::1/128 dev lo [O_OSPF 2017-06-01] * I (150/0) [172.20.175.198]
fdf7:17d5:de49:1::1/128 via fe80::de49:211 on wg-ibgp-de02 [O_OSPF 09:11:57] * I (150/10) [172.20.175.195]
fdf7:17d5:de49::1/128 via fe80::1 on wg-ibgp-de01 [O_OSPF 2017-12-29] * I (150/10) [172.20.175.193]
fdf7:17d5:de49:3::1/128 via fe80::de49:215 on wg-ibgp-fr01 [O_OSPF 2017-11-16] * I (150/10) [172.20.175.197]
fdf7:17d5:de49:2::1/128 via fe80::de49:211 on wg-ibgp-de02 [O_OSPF 09:12:47] * I (150/20) [172.20.175.196]
fdf7:17d5:de49::5222/128 via fe80::1 on wg-ibgp-de01 [O_OSPF 2017-12-29] * I (150/10) [172.20.175.193]