The custom login page is the most likely root of the problem. It appears that it is URL decoding the SAMLRequest parameter instead of preserving it. Here is the SAMLRequest prior to authentication:
SAMLRequest=nVRLb%2BIwEL73V0S%2BQx60PCxgxYJWi9TdzRK2h14qx5kUS4kdPA7Qf792CJRKiAPXycw332OcMbKyqOisNhu5gm0NaDzvUBYSafNlQmotqWIokEpWAlLDaTL79UyjbkArrYziqiAPy8WEYJSPwsdhFmRp2IueHodPwyztDUa9waDH2ShkaT8bRP0oY8R7AY1CyQmxMMRbItawlGiYNLYUhINO0O%2BE%2FXUwolFAe8Er8RaWmpDMNFMbYyqkvp9Dhjve1fAOkkOXq9Jneb6HFEHvBAf0qzotBPedmAhRfUtiS3WD27d25s0WrYzDB%2FF%2BKM2hcWJCclYgOGIxQxQ7OFfiVvJ3ITMh32%2F7kx6bkP5cr%2BNO%2FCdZE2%2BGlptTMVcS6xJ0cqT6b%2FX8qasl1zmR626AFWaDFdPbRqWqwK7zTxh%2BCYbNCsHwNOo3c1iR6cPYiaeNx%2Foi29vU2YkmmV5za%2BxfgLYrKvrbwiwXsbKOf9xzRi6Ckpnb3a4isk7etFKjmUQB0hAvid3%2BvzUrRC5AWzevxjwrCrWfa2DGpmp0DWR6VPOV%2F1lU%2Bywga07DOm7gYO4RN1elzU%2BgO2A4MG7s4sbFS%2BB5YZ1fQX5PUDfbOOUO2pbdSe%2BVztoMr20%2FO3JV%2FNmahKvK3vc9ZsQujLmq3YMP3ZH6XyA%2FC5e%2Fpul%2F
This value decodes to a proper authnrequest.
After authentication, here is the SAMLRequest:
SAMLRequest=nVRLb+IwEL73V0S+Qx60PCxgxYJWi9TdzRK2h14qx5kUS4kdPA7Qf792CJRKiAPXycw332OcMbKyqOisNhu5gm0NaDzvUBYSafNlQmotqWIokEpWAlLDaTL79UyjbkArrYziqiAPy8WEYJSPwsdhFmRp2IueHodPwyztDUa9waDH2ShkaT8bRP0oY8R7AY1CyQmxMMRbItawlGiYNLYUhINO0O+E/XUwolFAe8Er8RaWmpDMNFMbYyqkvp9Dhjve1fAOkkOXq9Jneb6HFEHvBAf0qzotBPedmAhRfUtiS3WD27d25s0WrYzDB/F+KM2hcWJCclYgOGIxQxQ7OFfiVvJ3ITMh32/7kx6bkP5cr+NO/CdZE2+GlptTMVcS6xJ0cqT6b/X8qasl1zmR626AFWaDFdPbRqWqwK7zTxh+CYbNCsHwNOo3c1iR6cPYiaeNx/oi29vU2YkmmV5za+xfgLYrKvrbwiwXsbKOf9xzRi6Ckpnb3a4isk7etFKjmUQB0hAvid3+vzUrRC5AWzevxjwrCrWfa2DGpmp0DWR6VPOV/1lU+ywga07DOm7gYO4RN1elzU+gO2A4MG7s4sbFS+B5YZ1fQX5PUDfbOOUO2pbdSe+VztoMr20/O3JV/NmahKvK3vc9ZsQujLmq3YMP3ZH6XyA/C5e/pul/
This value cannot be Base64 decoded. This is because the original value has been URL decoded. If the value above is URL encoded, it can then be Base64 decoded to a proper authnrequest.
Not seeing this problem in lower environments is most likely due to chance. The requests in the lower environments were perhaps not SP-initiated, or did not contain characters subject to URL encoding such that the unneeded URL decoding that takes place during authentication had no effect with the specific requests in the lower environments.