Layer7 API Management

Expand all | Collapse all

Gateway node startup error

  • 1.  Gateway node startup error

    Posted 10-15-2018 12:33 AM

    Hi Team,

    we are getting gateway node startup errors, please find below errors. 

     

    2018-10-15T04:13:19.996+0800 WARNING 1 STDERR: com.l7tech.server.LifecycleException: Hibernate operation: Could not execute JDBC batch update; SQL [update cluster_info set address=?, avgload=?, uptime=?, esm_address=?, statustimestamp=?, mac=?, multicast_address=?, name=? where nodeid=?]; Lock wait timeout exceeded; try restarting transaction; nested exception is java.sql.BatchUpdateException: Lock wait timeout exceeded; try restarting transaction 2018-10-15T04:13:19.997+0800 WARNING 1 STDERR: at com.l7tech.server.cluster.ClusterBootProcess.start(Unknown Source) 2018-10-15T04:13:19.997+0800 WARNING 1 STDERR: at com.l7tech.server.BootProcess.start(Unknown Source) 2018-10-15T04:13:19.997+0800 WARNING 1 STDERR: at com.l7tech.server.boot.GatewayBoot.f(Unknown Source) 2018-10-15T04:13:19.997+0800 WARNING 1 STDERR: at com.l7tech.server.boot.GatewayBoot.start(Unknown Source) 2018-10-15T04:13:19.997+0800 WARNING 1 STDERR: at com.l7tech.server.boot.GatewayBoot.runUntilShutdown(Unknown Source) 2018-10-15T04:13:19.998+0800 WARNING 1 STDERR: at com.l7tech.server.boot.GatewayMain.main(Unknown Source) 2018-10-15T04:13:19.998+0800 WARNING 1 STDERR: Caused by: org.springframework.dao.CannotAcquireLockException: Hibernate operation: Could not execute JDBC batch update; SQL [update cluster_info set address=?, avgload=?, uptime=?, esm_address=?, statustimestamp=?, mac=?, multicast_address=?, name=? where nodeid=?]; Lock wait timeout exceeded; try restarting transaction; nested exception is java.sql.BatchUpdateException: Lock wait timeout exceeded; try restarting transaction

     

    Please help me to fix ASAP.



  • 2.  Re: Gateway node startup error

    Posted 10-15-2018 12:56 AM

    Hello Uday,

    The error indicates the  table cluster_info is locked.

    There could be stuck gateway process, there could be other DB job to lock the table, etc.

    If the mysql database is on the same gateway server, the best try is to reboot the server.

    Or, you may ask your DBA to check further.

    Or, you may open a support ticket.

     

    Regards,

    Mark

     



  • 3.  Re: Gateway node startup error

    Posted 10-15-2018 01:43 AM

    Hi Mark,

    Thanks for quick reply, we have rebooted server but no luck, no other process is locking DB table.

    there is something else. yes, DB and ssg process on same host.



  • 4.  Re: Gateway node startup error

    Posted 10-15-2018 12:03 PM

    Uday,

     

    You can check if there is a deadlock created(If run within the lock wait time out period) on the table by executing 

    mysql -e "SHOW ENGINE INNODB STATUS\G;"

    If there are any, it should show something like below.

     

    mysql tables in use 1, locked 1
    LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
    MySQL thread id 493227, OS thread handle 0x7f3f49cb3700, query id 46546607 localhost root statistics
    select * from test.testlock where a = 1
    ------- TRX HAS BEEN WAITING 16 SEC FOR THIS LOCK TO BE GRANTED:

     

    SHOW PROCESSLIST; would show the query as "Searching rows for update"



  • 5.  Re: Gateway node startup error

    Posted 10-17-2018 11:14 AM

    Hi Anand,

    No luck .. 



  • 6.  Re: Gateway node startup error

    Posted 11-20-2018 04:09 PM

    A case was opened for this issue and at some point the system were recovered so root cause was not determined.

     

    Sincerely,

     

    Stephen Hughes

    Broadcom Support