Could it be that the domain name has both IPv4 and IPv6 and depending on the network you try to reach one or another? Wireguard can work on both protocols, but from my experience it doesn’t try both to see which one works (like browsers do). So if at the first try the dns resolves the “wrong” IP version, wireguard cannot connect and doesn’t fallback trying the alternative.
I did some experiments in the past. The nicer option I could find was enabling webdav API on the hosting side (it was an option on cPanel if I recall correctly, but there are likely other ways to do it). These allow using the webserver as a remote read/write filesystem. After you can use rclone to transfer files, the nice part is that rclone supports client side encryption so you don’t have to worry too much about other people accessing files.