Experimenting with VMs is the way forward.

Basic networking knowledge is vital. And being able to configure your own firewall(s) safely is an important skill. Check out something like Foomuuri, or Firewald. Shorewall is brilliant for documentation and description of issues (with diagrams!) but it does not use the newer Linux kernel nftables and is no longer actively developed.

Go for it with Nextcloud.

I would also recommend at least having a shot at setting up an email server, although I would recommend pushing through to a fully working system. It is possible, and is very satisfying to have in place. The process of setting one up touches so many different parts of internet function and culture that it is worth it even if you don’t end up with a production system. The Workaround.org ISPMail stuff is a good starting point, and includes some helpful background information at every stage, enough so you can begin to understand what’s going on in the background and why certain choices are being made - even if you disagree with the decisions.

Python is great for server admin, although most server config and startup shutdown snippets are written in BASH. You will no doubt have already begun picking that up as you interact with your VMs.

This approach sounds good.

I think the correct approach is both, if you have the option.

Most devices accept two name servers. Redundancy is always good, especially for DNS.

Edit: Forgot to mention! Another minor gripe I have is that my current 1 router / 2 routers-as-AP solution isn’t meshed, so my devices have to be aware of all 3 networks as I walk across my property. It’s a pain that I know can be solved with buying dedicated access points (…right?), but I’d like to know other’s experiences with this, either with OpenWRT, or other network solutions!

This works very well with OpenWRT on each AP and/or router device by using the same ESSID and password combo on each of them, enabling WLAN roaming and also 802.11r Fast Transition to allow your mobile devices to hand-off quickly from one AP to another as signal strength levels demand. With this enabled you keep the same IP address, and even SSH sessions don’t drop when you move from one AP to another, it all happens in the background. As far as the end-user is concerned it is all just one big happy wifi network.

802.11r is not mesh, that’s a separate thing but and you can do it with OpenWRT too. I don’t need to because I have ethernet to all my APs, so all the RF bandwidth is available for the last leg from AP to device(s), and not being used by back-haul from AP to AP through to the router as well.

In your use case I would consider grouping devices into categories and having a different wifi network for each category with the dhcp and firewall rules set accordingly.

VLANs on the ethernet-side might also be useful, but it sounds like most of your devices are on WiFi, so it might well be possible to get a “mature” setup without needing that extra complexity.

As others have said, backing these settings up and restoring them to a new device in the case of hardware failure is generally straightforward. Care is needed when replacing the broken device with a new one because of naming conventions varying from device to device, but the network logic, and things like dhcp reservations can be carried over.