I’ve dabbled in VyOS and Opnsense, but ran Pfsense for close to a decade (which is basically Opnsense, but more proprietary). Between the two, I think Opnsense is likely your better option.
Updates: There is some nuance, but VyOS doesn’t give you a LTS release so you will be taking quarterly upgrades to the latest. You are more or less a beta tester.
Opnsense has I guess one could consider “normal” releases with their community edition and is pretty easy to upgrade. You don’t really expect breaking changes.
Tools: CLI is typically faster on VyOS (once you skill up) than the GUI for Opnsense. But Opnsense also provides CLI options to interact with it, so if you decide you want that later on you have it. Both have APIs to interact with too for automation later.
Firewall: VyOS is using netfilter underneath, aka things you may be familiar with like iptables and nftables. It’s a fine firewall option, and is similar to what you could find in a normal Linux install (more on that later).
Opnsense I believe, uses the PF package (that’s also where the original pfsense got its name). This is a package for FreeBSD, and in my 10+ years of using it, it has been reliable.
Hardware: VyOS being based on Linux will support more hardware out of the box.
Opnsense being based on FreeBSD may run into edge cases of not using your hardware correctly. (I had a thing on Pfsense (also FreeBSD) where it didn’t like my built-in NIC and would flap constantly. I gave up trying to use it and used an external NIC instead)
Opinion: Honestly, if I was stuck with just these two as my choices I would probably lean toward Opnsense. It is more popular in homelab community, and there are many many guides on how to manage it.
Another option: You could also just using any other Linux distro and build up the firewall yourself. In the datacenter, it is very common to find big switches or firewalls simply running Debian on them with whatever firewall package the admin wants. It’s also a great learning experience.
Offbeat option: Presented with the same inquiry as you, I actually switched to OpenWRT. Many people will tell you that it’s just for APs or Routers, but I have been using it for my managed switches and my main firewall for about half a year now. OpenWRT is just Linux underneath, and you get the same CLI access as you would with a normal Linux distro. Plus you do get the option to use LuCI to mange it in the GUI.
It’s not as shiny as the *senses but, it is certainly just as capable for this role in a homelab. Remember when I said I was on pfSense for years? I was able to port all of my firewall rules, VLANs, etc into OpenWRT just fine.
I tried this in my homelab, and I wanted to convert my Proxmox with it’s LXC’s over to it…but the problem I ran into was that it was a little too obtuse for what I wanted to use it for.
Long story:
I wanted to get away from Proxmox as a platform and LXCs in general over to another hypervisor and using standard containers. I was tired of having a full fat web GUI to manage things with. Yes you can use pct and other CLIs tools in Proxmox to get some speed, but you are still dealing with an opinionated, dare I say proprietary stack.
Incus is really cool in that it’s CLI driven first, and you can add a GUI later if you want. I thought that was nice, and I was able to get some VMs started in my lab cluster and get clustering kinda working. I thought I was in a good place to switch.
But then, my main file server died (thankfully just the mobo, not the ZFS disks). And I was scrambling to figure out how to get all of my functionality back without having to go back to adding the storage back to Proxmox. So I took it as an opportunity to go to incus.
Well it turns out, it’s not so turn key depending on your method of deployment. For one, the actual “IncusOS” deployments are really particular about having a newer version of TPM (not something the old boxes I purchase typically have). So you are still likely to install something like Debian + incus-cli tools.
Once you have your distro + incus package setup though, there are a few things you have to tackle. For one, if you actually want separate IP addresses per VM (or LXC) you’ll likely need to solve the chicken and egg problem with macvlan, otherwise you’ll have to figure out something for a reverse proxy with the network bridge.
So because I was pressed against the wall (my family was irritated about no jellfyin or immich), and I knew that I didn’t want to do LXCs again and simply wanted to do most things I can in containers, I went the podman route instead.
Every app gets it’s own local user. You can install podman-compose, and use the same commands as docker compose basically to get your app up. All rootless, and you can also setup podman compose apps with daemons so that they start on boot: https://www.it-hure.de/2024/02/podman-compose-and-systemd/. If you don’t care about rootless isolation, I think Docker is still fine. Everyone seems to make documentation and build their apps with it in mind still.
I still had a need for VMs, so now I just use the standard Qemu+KVM+libvirtd combo for the one or two apps that need that need a full on separation from other apps.
All of it is in Ansible playbooks now, and I’m pretty happy with the setup I think.
TL; DR
Incus is really cool. I probably wouldn’t bother with IncusOS on old hardware, I’d just install a standard Linux distro + incus packages. Spend your time evaluating it (and making it infrastructure code for easy deployment maintenance). Maybe try out the other “standard” ways to do it instead to compare (Docker, Podman, Qemu+KVM, etc).