last post in the series about my home lab resulted in a number of interesting emails in my inbox. i have to admit that i really appreciate words of praise. as well as those with constructive, critical feedback :)
so after short description what is connected where and how (see link above), let’s focus now on services.
first and foremost - remember it’s “always DNS” ;) so let’s tackle that.
as i need to use UPSes (thank you, PGE…) even during power outage my infrastructure will still function (what a luxury…). however, sometimes you just need to upgrade piece of it, or restart or whatever. it would be good for services to still run in the meantime… specially, if your own kids request SLA at level of at least six nines… ;)
enter anycast
i have to admit, that initially my experiments led me to installing number of DNS servers running as VMs under different IPs. to some extent it’s because i got a good deal at acquiring load of Rasbperry Pi 3s and 4s ;)
however, with time (ah, that network operation & maintenance services) it became obvious that rebuilding FreeBSD ARM images, then writing them to flash cards, and then rebooting RbPis… wasn’t worth it. so, i decided to switch from using multiple IPs (“for redundancy”) to… single one.
if you’re old enough to remember my IP anycast session (if not - please, read it now, i’ll wait), one of great things about IP is ability to announce the same prefix from multiple places in the network. thanks to that, you don’t force DNS resolvers (which, by the way, tend to be of horrible quality on desktop OSes) to make “intelligent” decisions because everything they see - is single IP address.
so i decided to create two subnets in my home network - 192.168.66.0/24 for services, and 192.168.168.0/24 to announce everything that will be available over anycast. both DNS servers working as VMs running FreeBSD are physically located on two different Intel NUCs, situated in two different parts of the network:
- ns1.wesola.local - 192.168.66.22/24
- ns2.wesola.local - 192.168.66.33/24
- ns.wesola.local - 192.168.168.168/32
both VMs announce their own 192.168.168.168/32 prefix (also known as simply ns), assigned as alias to their own loopback0 interface. there’s recursive DNS service running on that IP (it’s unbound). unbound in addition queries local nsd copy (authoritative nameserver) bound to 127.0.0.1/::1 address and answering queries from wesola.local domain (so i can use short names instead of IPs).
on FreeBSD acting as name server, network stack config looks like this:
[ns1 ~]$ more /etc/rc.conf
hostname="ns1"
ifconfig_vmx0="inet 192.168.66.22 netmask 0xffffff00"
ifconfig_lo0_alias1="inet 192.168.168.168 netmask 255.255.255.255"
defaultrouter="192.168.66.1"
…and:
[ns2 ~]$ more /etc/rc.conf
hostname="ns2"
ifconfig_vmx0="inet 192.168.66.33 netmask 0xffffff00"
ifconfig_lo0_alias1="inet 192.168.168.168 netmask 255.255.255.255"
defaultrouter="192.168.66.1"
and DNS configuration is very similar - on both nodes, unbound and nsd configuration looks like this (observe IP addresses):
[ns1 ~]$ more /usr/local/etc/unbound/unbound.conf
server:
interface: 192.168.168.168
outgoing-interface: 192.168.66.22
port: 53
[...]
forward-zone:
name: wesola.local
forward-addr: 127.0.0.1
forward-zone:
name: "."
forward-addr: 208.67.222.222
forward-addr: 2620:119:35::35
forward-addr: 208.67.220.220
forward-addr: 2620:119:53::53
and:
[ns1 ~]$ more /usr/local/etc/nsd/nsd.conf
server:
ip-address: 127.0.0.1
ip-address: 192.168.66.22
ip-address: 192.168.66.22@5053
[...]
zone:
name: wesola.local
zonefile: wesola.local
zone:
name: 168.192.in-addr.arpa
zonefile: wesola.local.rev
[...]
BGP routing for services automation
OK, so how does it work together? host receives its DNS IP from DHCP pointing to 192.168.168.168 and… then what?
well, routing between segments (VLANs in this case) is done by my core switch. because on both VMs with DNS service i’m running also OpenFRR, it automatically announces availability of 192.168.168.168/32. if, for some reason, hosts goes down, prefix is removed from Nexus (core switch) routing table and that’s all needed to “automate” availability information.
on core switch, normal routing table looks like this:
sw-core# sh ip route 192.168.168.168
IP Route Table for VRF "default"
[...]
192.168.168.168/32, ubest/mbest: 2/0
*via 192.168.66.22, [200/0], 3d00h, bgp-65055, internal, tag 65055
*via 192.168.66.33, [200/0], 3d00h, bgp-65055, internal, tag 65055
and it’s BGP config looks like this:
router bgp 65055
maxas-limit 64
address-family ipv4 unicast
maximum-paths 32 ! yeah, they may be more than 2 paths
maximum-paths ibgp 32
address-family ipv6 unicast
maximum-paths 32 ! ...and i have IPv6 as well
maximum-paths ibgp 32
neighbor 192.168.66.22
remote-as 65055
timers 1 3
address-family ipv4 unicast
route-reflector-client
filter-list 1 out ! we don't announce anything towards NSes
neighbor 192.168.66.33
remote-as 65055
timers 1 3
address-family ipv4 unicast
route-reflector-client
filter-list 1 out ! we don't announce anything towards NSes
!
ip as-path access-list 1 seq 10 deny "^$"
on the VM side, bgpd (belonging to OpenFRR package from FreeBSD ports) configuration looks like this:
[ns1 /usr/local/etc/frr]$ more bgpd.conf
hostname ns1
password cisco ! h4x0rs beware!
enable password cisco
!
router bgp 65055
bgp router-id 192.168.66.22
network 192.168.168.168/32
neighbor 192.168.66.1 remote-as 65055
address-family ipv4 unicast
exit-address-family
log stdout
…and on ns2:
[ns2 /usr/local/etc/frr]$ more bgpd.conf
hostname ns2
password cisco ! h4x0rs beware as well!
enable password cisco
!
router bgp 65055
bgp router-id 192.168.66.33
network 192.168.168.168/32
neighbor 192.168.66.1 remote-as 65055
address-family ipv4 unicast
exit-address-family
log stdout
what does it all mean? in the end, all clients receive one NS address from DHCP service, and BGP protocol will make sure traffic will be load-balanced automatically between available prefix instances. so, if for example i’m upgrading (and rebooting) one of the VMs, Nexus removes one path, and then re-enables it once VM comes up again.
beauty of this is that once i add third, fourth or sixteenth DNS server, behavior won’t change - Nexus will still load balance requests to multiple instances. sometimes i actually do that, moving in one of the RbPis just to test something. and greatest thing is - kids don’t even notice :)
and that DHCP?
well, DHCP is also redundant - served by both VMs. because the way DHCP operates it actually doesn’t need anycast. we’re using built-in capabilities of protocol. actually, on every DHCP-enabled Nexus interface there are two DHCP IP helpers:
sw-core# sh running-config interface vlan 33
[...]
interface Vlan33
[...]
ip address 192.168.33.1/24
ip verify unicast source reachable-via rx
ip dhcp relay address 192.168.66.22 ! first VM
ip dhcp relay address 192.168.66.33 ! second VM
[...]
to actually serve DHCP data, i’m using isc-dhcp from FreeBSD ports. servers are configured as redundant pair and provide continous operation even if something happens to the other peer:
[ns1 /usr/local/etc]$ more dhcpd.conf
[...]
option domain-name-servers 192.168.168.168;
option domain-name "wesola.local";
omapi-port 7911;
omapi-key omapi_key;
key omapi_key {
algorithm hmac-md5;
secret Ofakekeyfakekeyfakekey==;
}
# redundant pair configuration - IPs, ports and split of pools:
failover peer "wesola-fo" {
primary;
address 192.168.66.22;
port 519;
peer address 192.168.66.33;
peer port 520;
max-response-delay 60;
max-unacked-updates 10;
mclt 3600;
split 128;
load balance max seconds 3;
}
# example VLAN 33 subnet definition for example above:
# VLAN33
subnet 192.168.33.0 netmask 255.255.255.0 {
option routers 192.168.33.1;
option ntp-servers 192.168.33.1;
pool {
failover peer "wesola-fo";
range 192.168.33.10 192.168.33.99;
}
}
thanks to such configuration, not only DNS but also DHCP is redundant and “self optimizing”
well, i hope this short post will encourage you to your own experiments :)