Another "At the risk of asking the obvious" question:
Have you established that OBS Studio works properly with that GPU on a physical computer running Windows 10?
I was also going to suggest trying it with Ubuntu as the OS, but there seems to be a fair number of additional packages that have to be installed in order for the Linux version to work.
Still, it might be worth the effort.
Another test would be to try it with a newer NVIDIA GPU to see if the problem persists.