Install Tailscale on all devices.

Then ssh into whatever you need.

If you need desktop remote access the Windows RDP should work for Windows to Windows machines.

For Linux host to Windows client I’ve had good experiences with Remmina Desktop.