Layer 7 API Management

Expand all | Collapse all

Cassandra Write Query error

  • 1.  Cassandra Write Query error

    Posted 12-17-2018 07:47 AM

    Hi Techies ,

     

    We are able to see Cassandra write query in our environment in case of high load , Please have a look over below log snippet .

     

    Error String :

    WARNING 8511 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

     

    As per error ,looks like it Its not getting ack in time from both the replica  hence timing out with above error  But in our environment there are 4 Cassandra node and all are running fine with out any issue .

     

    Software version that we are using :

     

    Gateway :9.2 Gateway  2 node standalone (A and B)

    Cassandra : 5.1  Total 4 nodes  Cassandra  (A,B ,C and D)

    OS : REHL 6.7

     

    Thanks!



  • 2.  Re: Cassandra Write Query error

    Posted 12-17-2018 10:11 AM

    Hi

     

    Did you set any custom value's for the cassandra connections ?

    see also :

    Cassandra Cluster Properties - CA API Gateway - 9.4 - CA Technologies Documentation 

     

    Regards Dirk



  • 3.  Re: Cassandra Write Query error

    Posted 12-18-2018 12:28 AM

    Thank you Dirk for checking !!

    No we did not set any custom value in Cassandra connection 



  • 4.  Re: Cassandra Write Query error

    Posted 12-30-2018 06:11 PM

    Dear Prashant SRIVASTAVA ,

    This should be an error from the DB server.

    Since it's due to timeout, can you try increase the write timeout of Cassandra?

    ( I am not 100% sure as I don't have cassandra, but it should be in the conf/cassandra.yaml file.

     

    write_request_timeout_in_ms: 30000

    )

     

    It seems you configured replication for cassandra, then the default write timeout may not be enough.

     

    Regards,

    Mark



  • 5.  Re: Cassandra Write Query error

    Posted 12-30-2018 11:13 PM

    Thank you for Reply Mark!!

     

    We are using default read and write  timeout at db level that is defined in Cassandra.yaml file .While digging more in policy manager audit log can see there are two type of error related to write query  . As per below log seems API Gateway dynamically sending consistency level(TWO and LOCAL_ONE) with query but i did not get in Policy any such configuration .[all Cassandra node are up

     and running fine ] 

     

    sensitive)
    2018-12-20T09:25:41.219+0100 INFO 11509 com.l7tech.external.assertions.comparison.server.ServerComparisonAssertion: 7101: Comparison did not match: ${tempRefreshToken} is not empty (case sensitive)
    2018-12-20T09:25:41.219+0100 WARNING 11509 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write

     


    2018-12-20T09:25:41.460+0100 WARNING 16161 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)
    2018-12-20T09:25:41.511+0100 WARNING 10155 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency LOCAL_ONE (1 replica were required but only 0 acknowledged the write

     

    Thanks!

    Prashant Srivastava



  • 6.  Re: Cassandra Write Query error

    Posted 12-31-2018 09:26 AM

    Hi ,

     

    I am not aware of a setting in the gateway for this , if you see different consistency levels being set from the GW .

    I would suggest to raise a support case so we can have closer look and investigate the logs .

     

    Dirk 



  • 7.  Re: Cassandra Write Query error

    Posted 12-31-2018 09:31 AM

    Hi 

     

    One other thing which could play a role which OTK kit are you using , and how is it setup 

    see:

    Apache Cassandra Database - CA API Management OAuth Toolkit - 4.3 - CA Technologies Documentation 

     

    During these steps you set the replication factor . which could explain why you see different consistency levels

     

    Dirk 



  • 8.  Re: Cassandra Write Query error

    Posted 01-02-2019 03:54 AM

    Thank Dirk for looking in to it!!

     

    I can see replication factor configured as 2 in our environment .But not getting how can see different consistency level ((TWO and LOCAL_ONE)) in Gateway log . Also in what type of  scenarios we can see different CL level . 

    CREATE KEYSPACE otk_db WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '2'}  AND durable_writes = true .

     

    I have already logged case [01254681 ] on our Support Portal.



  • 9.  Re: Cassandra Write Query error

    Posted 01-02-2019 10:45 AM

    Hi 

     

    I will keep a eye on the case but it seems the replication factor 2 is coming form a otk update .

    Which could be set in the during  database creation for the otk kit.

     

    Apache Cassandra Database - CA API Management OAuth Toolkit - 4.3 - CA Technologies Documentation 

     

    Dirk



  • 10.  Re: Cassandra Write Query error

    Posted 01-15-2019 08:29 AM

    Hope you are doing well Dirk ! 

    Did you get anything more on this . Also i was looking in to below log snippet and can see code like 8511 after WARNING , any idea what does it mean ?

     

    WARNING 8511 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

    While going through documentation got to know there is few audit message code but  I did not get 8511 in the list .

    FYI

    Audit Detail Codes - CA API Gateway - 9.1 - CA Technologies Documentation 

     

    Thanks!

    Prashant Srivastava



  • 11.  Re: Cassandra Write Query error

    Posted 01-15-2019 08:36 AM

    adding on this if number present there in the log is related to Audit Message Code then it should be constant but I can see its different in below logs .

     

    2019-01-08T10:27:19.888+0100 WARNING 354 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

    2019-01-08T10:27:31.788+0100 WARNING 7466 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

    2019-01-08T10:27:41.124+0100 WARNING 7393 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

    2019-01-08T10:27:42.201+0100 WARNING 11417 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

    2019-01-08T10:27:43.475+0100 WARNING 7393 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

    2019-01-08T10:27:44.673+0100 WARNING 11417 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

    2019-01-08T10:27:45.796+0100 WARNING 7393 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

    2019-01-08T10:27:47.084+0100 WARNING 11417 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

    2019-01-08T10:27:47.424+0100 WARNING 5965 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

    2019-01-08T10:27:47.769+0100 WARNING 7139 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)



  • 12.  Re: Cassandra Write Query error

    Posted 01-16-2019 06:33 AM

    Hi

     

    The 8511 in your example is not the audit detail code that would be the 10903.


    Did you check the support case ?

     

    Our consistency level by default will use ONE and it is not going to be configurable until 9.5+

    When the batch statement is run it will require at least 2 nodes in the Cassandra cluster available so the key part will be to determine of the cassandra nodes how are they separated into DC.

     

    Regards Dirk