Having dynamic IPv6 addresses all over the LAN is good for the systems to reach any IPv6 (-only) site on the Internet. It is not good for communication within the LAN, because the dynamic addresses are subject to change at any time, and cannot be put into config files.
So, in addition to dynamic GU addresses, we may want to also configure static private IPv6 addresses for communication within the LAN. But this is not mandatory as the internal communication can as well be done in IPv4. And there are some issues.
A problem with these private addresses is that their routes may strangely disappear when dhcpcd is started.
Normally when we configure an address on an interface (with ifconfig or from rc.conf), we can see the address on the interface, and also two entries in the routing table:
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1492
inet6 fd00::101 prefixlen 120fd00::100/120 link#10 U tap0
fd00::101 link#10 UHS lo0The first line is in accordance with the prefixlen; it tells the routing to send addresses from that subnet out through the specific interface - and that one might disappear when dhcpcd starts up. (The second line is the address itself, and is connected to lo0. It tells the routing that this address is local to the system and shall be sent through lo0.)
The problem appears for all interfaces, even for those that are not configured in dhcpcd at all, and it happens as soon as any interface has ipv6rs option configured (like we did above). Consequentially then nothing will be routed to that subnet, and so, nothing works. Worse, when there is traffic on the system, the system may now not have any means to get rid of the packets, and the buffers will fill up. The system will then soon run into error 55 ("no buffer space available"), and then nothing at all works anymore. Still worse, this condition will not resolve itself, even after the required route gets configured again. It will only resolve after the concerned addresses on the interfaces get removed and reinstalled (or when the interfaces are taken down and up).
It may be difficult to notice and pinpoint the issue - usually one does only perceive that traffic does not get answered although the required services are running. One can usually check it with ping6:
$ ping6 test
PING6(56=40+8+8 bytes) fd00::102 --> fd00::101
ping6: sendmsg: No buffer space available
ping6: wrote test.daemon.contact 16 chars, ret=-1For a workaround, at first I trued to have dhcpcd maintain these static addresses as well. This can be done in the dhcpcd.conf file with the static option, like so:
interface tap0
static ip6_address=fd00::101/120This does usually work - and watching closely, one can see that dhcpcd still deletes the routes after start, and then, shortly after, reinserts them. But then I experienced the static IPv6 traffic stall when the outbound (nexthop) router (bhyve) is rebooted. dhcpcd will notice that the outbound interface has disappeared and unconfigure that interface, and then reinitialize it when it reappears. During that reinitialization again the routes are removed, but it may then take 10 seconds to obtain (or not obtain) a new lease, and only after that timeout the routes are reinserted. And this is enough time for the internal traffic to stall entirely. and render the LAN nonfunctional.
The matter is, dhcpcd does remove the routes intentionally, with the SIOCSPFXFLUSH_IN6 ioctl, in if-bsd.c:if_setup_inet6() - only with FreeBSD the syscall is misinterpreted to remove the routes from all interfaces instead only the one given in the syscall (this is already discussed in https://githubmemory.com/repo/rsmarples/dhcpcd/issues/59). That doesn't make matters worse because it would remove the routes from the interfaces it delegates prefixes to, anyway, and it is these interfaces we want to put static addresses on.
The argument in the code comment is, that the routes should be flushed because the kernel might otherwise expire those that dhcpcd tries to manage: I don't understand this, I could not find any mention that the kernel would even be able to do such:
There is indeed an expire metric in the route entries, which is poorly documented, and there was a statement by Kevin Oberman in 2008 that it is no longer used. Besides that, there appear to be the pltime and vltime values as provided by the upstream dhcps. These appear in the address when delegated onto an interface (visible with ifconfig -L). When these values do expire, the address gets removed from the interface, and then the associated routing entries will disappear alongside, as is to be expected. And anyway, rtadvd does ignore these values and propagate the prefix onwards with default lifetime (7 rsp. 30 days), unless otherwise configured.
All this should be independent from flushing any routes beforehand. So, from my understanding, under usual circumstances that flushing is problematic, for the given reason. It may well be useful for general cleanliness and housekeeping, to remove stale stuff from undefined operations. But it definitely does more harm than good here, and so I recommend to patch it out of the code, and then have the static routes configured in conventional fashion.
ip6addrctlWhen configuring the static private addresses (from the fc00::/7 prefix), one may notice that they are still not used. This is because of the default ip6addrctl setting for IPv6, where site-local IPv6 has a lower precedence (3) than IPv4 (35):
$ ip6addrctl
Prefix Prec Label Use
::1/128 50 0 0
::/0 40 1 0
::ffff:0.0.0.0/96 35 4 0
2002::/16 30 2 0
2001::/32 5 5 0
fc00::/7 3 13 0
::/96 1 3 0
fec0::/10 1 11 0
3ffe::/16 1 12 0We can change this by providing a custom table with some lines added to give site-local IPv4 a lower precedence:
::ffff:10.0.0.0/104 2 4 0
::ffff:172.16.0.0/108 2 4 0
::ffff:192.168.0.0/112 2 4 0