Test migrate in failed: Internal Server Error No Error

[Arjun Pilli]

Hi All,

I am seeing a very strange behavior in the gateway. I have been trying to deploy a policy into newly commissioned gateway 9.3 using GMU 1.5X. One day it worked fine and the next day it stopped working and threw an error as below. But when I checked in gateway it did deploy in the background but on the console, it is an error. 

D:\layer7\Arjun Pilli\GMU\GatewayMigrationUtility-1.5.00-479>GatewayMigrationUtility.bat migrateIn -z ..\test.emea.internal-argFile.properties --bundle IPF_BASIC_AUTHENTICATION_JSONRESPONSE_POLICY.xml --template IPF_BASIC_AUTHENTICATION_JSONRESPONSE_POLICY.template --results results.xml --test
Warning: TLS hostname verification has been disabled
Warning: TLS server certificate check has been disabled
Running......
Execution failed. Reason: Test migrate in failed: Internal Server Error No Error

The results file doesn't show any information than below lines 

<?xml version="1.0" encoding="UTF-8"?>
<l7:Item xmlns:l7="http://ns.l7tech.com/2010/04/gateway-management"/>

---------------
The only change we did is rung managebinlogs.sh script by mistake as it is a single node cluster and there is no replication but not sure if this has caused any issues. 

I have tried rebooting the Gateway and it didn't work. and also noticed restman is also not working properly. upon a request to restman it threw below error

curl -sS -k -n https://i18709.dsv.com:9443/restman/1.0/trustedCertificates
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
    <soapenv:Body>
        <soapenv:Fault>
            <faultcode>soapenv:Server</faultcode>
            <faultstring>Policy Falsified</faultstring>
            <faultactor>https://i18709.dsv.com:9443/restman/1.0/trustedCertificates</faultactor>
            <detail>
                <l7:policyResult status="No Error" xmlns:l7="http://www.layer7tech.com/ws/policy/fault"/>
            </detail>
        </soapenv:Fault>
    </soapenv:Body>
</soapenv:Envelope>

So suspecting an issue with restman service, i have deleted and redeployed and restarted the service but still it din't work, Not sure what could be the issue causing this behaviour. i am now thinking to restore from ssg backup as last resort.

Has anyone faced this issue, if so could you please share some information on how to resolve it ?

I have readup on migrateIn error but not found anything related to error "Test migrate in failed: Internal Server Error No Error"

------------------------------
Regards,
Arjun
------------------------------

[Deactivated User]

How did you use to login before.. were you using client certificate or username password. Because since you redeployed restman. It enables a property required client certificate which you need to provide 

You can update your restman policy to display full error. Most likely you are missing client certificate for the user you use to login

Please also check your SSG logs as it give a better description of the error if any

------------------------------
Pre-Sales Consultant
CA Southern Africa
------------------------------

[Arjun Pilli]

Thank you for the response Ronald. We are using the default version of the restman before also. we authenticate using Basic AUTH over HTTPS. I have enabled full details as you mentioned. I still see policy falsified error but in details everything success.

[Deactivated User]

1. Have you disabled Policy Management or Built In Service on port 8443? if not can you try using GMU against this end point
https://<urGateway>:8443/restman/1.0/ServiceName

2. Did you have a look on your gateway logs to see for any null pointer exception. 
3. Did you restart the server or the service after setting rest man again?
4. After running the manage bin logs shell script did you terminate it abruptly or did you let it run.. You may also want to check your mysql process list to check for any deadlocks. Which is why probably insertion are failing SHOW ENGINE INNODB STATUS 

------------------------------
Pre-Sales Consultant
CA Southern Africa
------------------------------

[Arjun Pilli]

Hi Ronald, using 8443 is also gives same error. What i did was. i have restored ssg database from previous working state and deployed the Global policy and it worked. Then i continued deploying other Global policy and it threw the same error again. 

So atleast i can reproduce the issue. i am guessing this is something to do with ssg datbase insert operations. i have checked the innodb status and see below error

------------------------
LATEST FOREIGN KEY ERROR
------------------------
2020-02-19 12:05:33 0x7f08800f7700 Transaction:
TRANSACTION 3243966, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
3 lock struct(s), heap size 1136, 1 row lock(s), undo log entries 1
MySQL thread id 330, OS thread handle 139674484963072, query id 122656 localhost 127.0.0.1 gateway update
insert into policy (name, version, security_zone_goid, folder_goid, guid, internal_sub_tag, internal_tag, soap, policy_type, `xml`, goid) values ('G_POST_PROCESS_POLICY', 0, null, x'E9D8906E7633DBACD972A443A6F5194F', 'ec16caa9-3588-4f77-a91d-3b510444174c', null, 'post-service', 0, 'GLOBAL_FRAGMENT', '<?xml version="1.0" encoding="UTF-8"?>\n<wsp:Policy xmlns:L7p="http://www.layer7tech.com/ws/policy" xmlns:wsp="http://schemas.xmlsoap.org/ws/2002/12/policy">\n <wsp:All wsp:Usage="Required">\n <L7p:CommentAssertion>\n <L7p:Comment stringValue="Policy Fragment: G_POST_PROCESS_P
Foreign key constraint fails for table `ssg`.`policy`:
,
CONSTRAINT `policy_folder` FOREIGN KEY (`folder_goid`) REFERENCES `folder` (`goid`) ON DELETE NO ACTION ON UPDATE NO ACTION
Trying to add in child table, in index policy_folder tuple:
DATA TUPLE: 2 fields;
0: len 16; hex e9d8906e7633dbacd972a443a6f5194f; asc nv3 r C O;;
1: len 16; hex 77c89107954d4026d6178d41f3964cc7; asc w M@& A L ;;

But in parent table `ssg`.`folder`, in index PRIMARY,
the closest match we can find is record:
PHYSICAL RECORD: n_fields 7; compact format; info bits 0
0: len 16; hex 0000000000000000ffffffffffffec76; asc v;;
1: len 6; hex 000000316035; asc 1`5;;
2: len 7; hex aa0000011e0110; asc ;;
3: len 4; hex 80000000; asc ;;
4: len 9; hex 526f6f74204e6f6465; asc Root Node;;
5: SQL NULL;
6: SQL NULL;

so i have deleted the deployed policy and tried redeploying and now it threw different error migrateIn conflict

D:\layer7\Arjun Pilli\GMU\GatewayMigrationUtility-1.5.00-479>GatewayMigrationUtility.bat migrateIn -z ..\test.emea.internal-argFile.properties --bundle G_POST_PROCESS_POLICY.xml --template G_POST_PROCESS_POLICY.template --results test.xml
Warning: TLS hostname verification has been disabled
Warning: TLS server certificate check has been disabled
Running.......
Execution failed. Reason: Migrate in failed: Conflict

results file shows the error as 

-<l7:Mapping type="POLICY" srcUri="https://private.dev.api.emea.dsv.com:9443/restman/1.0/policies/77c89107954d4026d6178d41f3964cc7" srcId="77c89107954d4026d6178d41f3964cc7" action="NewOrUpdate" errorType="UniqueKeyConflict">


-<l7:Properties>


-<l7:Property key="ErrorMessage">

<l7:StringValue>(name) must be unique</l7:StringValue>

</l7:Property>

</l7:Properties>

</l7:Mapping>

But this policy has been deleted from ssg and i can't see it in ssg.policy table as well. so not sure from where the unique key conflict is coming. 

Any help in resolving this issues are highly appreciated. I have also requested Broadcom Support to arrange a call

Regards,
Arjun

[Deactivated User]

https://private.dev.api.emea.dsv.com:9443/restman/1.0/policies/77c89107954d4026d6178d41f3964cc7



Open this link in Postman with ur credentials. 

its probably a policy that was created during the first import that is conflicting You may need to use Manage Mapping option to do a match for it 
look for Manage Mapping by Name in GMU 

------------------------------
Pre-Sales Consultant
CA Southern Africa
------------------------------