I've been experiencing a similar problem and have a support ticket open for it. Our install_mdb.log contains this:
02-24,16:10:15 INFO - =============================================================
02-24,16:10:15 INFO - MDBTools Log - Wednesday, February 24, 2016 at 4:10:15 PM CST
02-24,16:10:15 INFO - JVM Version: 1.7.0_60
02-24,16:10:15 INFO - MDBTools Version 1.5 SP1 (32-01)
02-24,16:10:15 INFO - =============================================================
02-24,16:10:15 INFO - MDBTools_0101I - Command: install database schema from driver file.
02-24,16:10:15 INFO - MDBTools_0110I - As user: ServiceDesk.
02-24,16:10:15 INFO - MDBTools_0112I - For vendor: mssql.
Feb 24, 2016 4:10:15 PM com.microsoft.sqlserver.jdbc.TDSReader readPacket
WARNING: TDSReader@3634f2 ( ConnectionID:1 TransactionID:0x0000000000000000) Unexpected message type:18 in TDS header; expected type:4
Feb 24, 2016 4:10:15 PM com.microsoft.sqlserver.jdbc.TDSReader throwInvalidTDS
SEVERE: TDSReader@3634f2 ( ConnectionID:1 TransactionID:0x0000000000000000) got unexpected value in TDS response at offset:4088
Feb 24, 2016 4:10:15 PM com.microsoft.sqlserver.jdbc.TDSChannel enableSSL
WARNING: TDSChannel ( ConnectionID:1 TransactionID:0x0000000000000000) SSL handshake failed: The TDS protocol stream is not valid.
Feb 24, 2016 4:10:15 PM com.microsoft.sqlserver.jdbc.TDSReader readPacket
WARNING: TDSReader@e220c4 ( ConnectionID:2 TransactionID:0x0000000000000000) Unexpected message type:18 in TDS header; expected type:4
Feb 24, 2016 4:10:15 PM com.microsoft.sqlserver.jdbc.TDSReader throwInvalidTDS
SEVERE: TDSReader@e220c4 ( ConnectionID:2 TransactionID:0x0000000000000000) got unexpected value in TDS response at offset:4088
Feb 24, 2016 4:10:15 PM com.microsoft.sqlserver.jdbc.TDSChannel enableSSL
WARNING: TDSChannel ( ConnectionID:2 TransactionID:0x0000000000000000) SSL handshake failed: The TDS protocol stream is not valid.
02-24,16:10:15 INFO - Url: jdbc:sqlserver://localhost:1433;DatabaseName=mdb
02-24,16:10:15 INFO - User: ServiceDesk
02-24,16:10:15 INFO - ErrorCode: 0
02-24,16:10:15 ERROR - The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: The TDS protocol stream is not valid..
02-24,16:10:15 INFO - MDBTools_0104I - Total time: 0H:0M:0S.
02-24,16:10:15 MDBTools install error=3
02-24,16:10:15 Setupmdb exit /B return code=3
which ultimately results in this in the install.log:
2016/02/24 16.10.13.974 DEBUG [DeployThread: Configuring CA Service Management common tables] [InstallCDB] 02-24,16:10:13 Checking if mdb already exists...
2016/02/24 16.10.14.379 DEBUG [DeployThread: Configuring CA Service Management common tables] [InstallCDB] 02-24,16:10:14 Checking existing mdb version...
2016/02/24 16.10.14.606 DEBUG [DeployThread: Configuring CA Service Management common tables] [InstallCDB] 02-24,16:10:14 Pre-check OKAY ( will upgrade MDB version-1.5 )
2016/02/24 16.10.14.713 DEBUG [DeployThread: Configuring CA Service Management common tables] [InstallCDB] 02-24,16:10:14 Loading mdb...
2016/02/24 16.10.14.714 DEBUG [DeployThread: Configuring CA Service Management common tables] [InstallCDB] using APM_CDB
2016/02/24 16.10.15.860 DEBUG [DeployThread: Configuring CA Service Management common tables] [InstallCDB] 02-24,16:10:15 MDBTools install error=3
2016/02/24 16.10.16.055 DEBUG [DeployThread: Configuring CA Service Management common tables] [InstallCDB] 02-24,16:10:16 Setupmdb exit /B return code=3
2016/02/24 16.10.16.062 INFO [DeployThread: Configuring CA Service Management common tables] [ActiveProcessHolder] Exit Monitor. Process ID (PID): 7428
2016/02/24 16.10.16.062 DEBUG [DeployThread: Configuring CA Service Management common tables] [InstallCDB] Command terminated with error
2016/02/24 16.10.16.063 DEBUG [DeployThread: Configuring CA Service Management common tables] [InstallCDB] Exit code of CDB script : 3
2016/02/24 16.10.16.063 DEBUG [DeployThread: Configuring CA Service Management common tables] [ProgressPage] ProgressPage - Task Changed Configuring CA Service Management common tables
2016/02/24 16.10.16.063 DEBUG [DeployThread: Configuring CA Service Management common tables] [ProgressPage] Task State Changed. TaskID: task.install_CDB,Configuring CA Service Management common tables, Old State: running, New State:failed
2016/02/24 16.10.16.063 DEBUG [DeployThread: Configuring CA Service Management common tables] [ProgressPage] ProgressPage - Task Error.
2016/02/24 16.10.16.064 DEBUG [DeployThread: Configuring CA Service Management common tables] [ProgressPage] Task Error. TaskID: task.install_CDB,Configuring CA Service Management common tables
I'm inclined to believe that it's related to some policy that requires encrypted connections to the DB, but I haven't fully proved that out or determined what the offending policy may be. What I do know is that installing 14.1 on a base WS2k12R2/SQL2012 configuration works fine for me in our pre-production domain, as does upgrading from 12.7 to 14.1 in our pre-production domain, but when I try upgrading from 12.7 to 14.1 using any server in our production domain, the upgrade fails with the above error. I have not yet attempted a clean 14.1 install in our production domain, but I hope to do that later today. I expect that it will fail with the same error.
Also, It has been reported to me that CA is working on a fix for this, but I haven't seen the fix yet, so I'm also continuing to troubleshoot on our side until I have that fix in hand to test.