12.6.5 CR2 now has a JCS that allows you to adjust timeout value:
Open command prompt (as administrator) and set the environment variable JCS_SERVICE_TIMEOUT to say 600 seconds
On Windows: SET JCS_SERVICE_TIMEOUT=600
On Linux/Solaris: Export JCS_SERVICE_TIMEOUT=600
Then run the installer from the same command prompt, setup.exe.
The installer will now use the timeout value that you declare. This has been done due to some customers experiencing a timeout issue when installing the JCS Component of IDM at the 5 minute marker. When troubleshooting first attempt to set the JCS time out to 600 seconds or 10 minutes. Should the installer still fail please open a support case for assistance and troubleshooting.
I had this problem with 12.6.5 GA, and I can confirm that this solution works.
Thanks for posting!