Update 2:
I'm now wondering if there's something wrong with the backup function in VCSA itself. Since I couldn't get SMB to work, I decided to try FTP. Got everything set up and ran it, and while the backup job said it failed to complete after 30 seconds, it still created the directory and backed up its files. I checked one of the failed SMB attempts (direct to server, network share still doesn't work), and it also backed up files - more than the FTP, actually, but I don't think it's everything.
Luckily this is just our company's test lab, but now I'm worried the same will happen to our production environment.