Layer7 API Management

Expand all | Collapse all

Do the audit_purge and manage_binlogs scripts still apply?

  • 1.  Do the audit_purge and manage_binlogs scripts still apply?

    Posted Aug 05, 2015 09:10 AM

    In the past I have installed the audit_purge and manage_binlogs scripts on the gateway. But while doing a a fresh install again at a customer this week, looking at the docs on the wiki I noticed there is no mention anywhere of these scripts. Do they still apply, or is that something of the past and are they no longer required, do we only need them for specific cases or are they simply missing in the documentation?



  • 2.  Re: Do the audit_purge and manage_binlogs scripts still apply?

    Broadcom Employee
    Posted Aug 05, 2015 10:04 AM

    Michiel

     

    Yes these scripts still apply..

     

    You still need to manage this portion after the install.

     

     

    Thanks,

     

    Derek Orr

     

    CA Technologies |885 West Georgia Street Ste 500 | Vancouver, BC V6C 3G1

    Office: 1-778-328-5285 | Mobile: +1 778 980 0029 | Derek.Orr@ca.com

     

    <mailto:>[CA]

     

    <http://www.ca.com/us/default.aspx>[Twitter]<http://twitter.com/CAInc>[LinkedIn]<http://www.linkedin.com/company/1372?goback=.cps_1244823420724_1>[Facebook]<https://www.facebook.com/CATechnologies>[YouTube]<http://www.youtube.com/user/catechnologies>[Google]<https://plus.google.com/CATechnologies>[Slideshare]<http://www.slideshare.net/cainc>



  • 3.  Re: Do the audit_purge and manage_binlogs scripts still apply?

    Posted Jun 28, 2016 04:30 PM

    Hi Eric,

     

    Do you have the new links to these two articles now that all the documentation is migrated to support.ca.com?

     

    Thanks,

    Atul



  • 4.  Re: Do the audit_purge and manage_binlogs scripts still apply?

    Broadcom Employee
    Posted Jul 21, 2016 04:44 AM

    Hi goeer03,

     

    Same request here. Could you provide us the updated links ?

     

    Thank you



  • 5.  Re: Do the audit_purge and manage_binlogs scripts still apply?
    Best Answer

    Posted Aug 05, 2015 10:44 AM

    There are two articles on the existing Support portal that cover this topic:

     

    These are not documented in the documentation as they are not required for all customer installations. The former article should only be used if an existing implementation is auditing excessively and the use of the Internal Audit Sink Policy is not currently in place. It is recommended that customers and administrators consider using alternative storage methodologies if their auditing requirements are strict. The API Gateway is not a storage appliance and is not the best solution for long-term storage of audit data that may be necessary for regulatory or compliance purposes.

     

    The latter item is more broadly applicable but excessive binary log generation is not considered a concern unless audit generation is also high. Binary log files are generated by the MySQL server when queries are replicated between hosts. There is a binary log that consists of queries that leave the master and were sent to the slave. There are also relay logs that were received on a slave by the master. These two log sets can generate a large amount of data if queries between the API Gateways is disproportionately high. The largest user of bandwidth, throughput, and storage in MySQL queries between two Gateway databases is audit data. That storage burden is increased substantially if audited request and response messages are stored in the API Gateway database.

     

    As such--if proper implementation methodologies are followed then neither of these scripts should be necessary. If audited data is sent out via the Internal Audit Sink Policy and audits are not stored on the local API Gateway databases then binary log and audit record usage is considered minimal.



  • 6.  Re: Do the audit_purge and manage_binlogs scripts still apply?

    Posted Aug 05, 2015 12:09 PM

    Thanks for the quick reply Derek and Eric!



  • 7.  Re: Do the audit_purge and manage_binlogs scripts still apply?

    Posted Aug 06, 2015 06:00 AM

    One additional question about the manage_binlogs: In the script itself it states in the comment/description that it should be run on the master database node. In the KB article it says that it should be implemented on both nodes. Which is correct?



  • 8.  Re: Do the audit_purge and manage_binlogs scripts still apply?

    Broadcom Employee
    Posted Aug 06, 2015 09:05 AM

    The manage bin logs script should be setup on both database nodes in a cluster.

     

    Attached is a document and 2 services that you should use when setting up clustering.

     

     

    Thanks,

     

    Derek Orr

     

    CA Technologies |885 West Georgia Street Ste 500 | Vancouver, BC V6C 3G1

    Office: 1-778-328-5285 | Mobile: +1 778 980 0029 | Derek.Orr@ca.com

     

    <mailto:>[CA]

     

    <http://www.ca.com/us/default.aspx>[Twitter]<http://twitter.com/CAInc>[LinkedIn]<http://www.linkedin.com/company/1372?goback=.cps_1244823420724_1>[Facebook]<https://www.facebook.com/CATechnologies>[YouTube]<http://www.youtube.com/user/catechnologies>[Google]<https://plus.google.com/CATechnologies>[Slideshare]<http://www.slideshare.net/cainc>

    Attachment(s)

    ZIP
    CLUSTER.ZIP   1.84 MB 1 version


  • 9.  Re: Do the audit_purge and manage_binlogs scripts still apply?

    Posted Aug 06, 2015 09:06 AM

    The binary log management script does not have that particular passage. That is in the audit record maintenance script:

    # Script to purge non SEVERE audit records over a certain age

    #

    # WARNING: Only run this script on one database node in a cluster, preferably

    # the primary database node. Replication will handle purging records in the

    # second database.

    The binary logs need to be managed on both hosts in a cluster as the binary logs are present on the file system local to a MySQL server. The audit logs are stored in a replicating database and only need to be deleted in one place.



  • 10.  Re: Do the audit_purge and manage_binlogs scripts still apply?

    Posted Aug 06, 2015 09:15 AM

    Thats what I thought, but this is what it says in the manage_binlogs.sh which is attached to the KB article:

     

    #!/bin/bash

    #

    # Script to monitor replication and safely purge binlogs. Exits with status

    # 0 if replication is working, else exits with status 1. Optionally set up

    # to send alerts vi SMTP or SNMP or to display STDOUT useful for an SNMP GET

    # call.

    #

    # Periodically run this on the master database node from crontab.

    #

     

    It doesn't say not to run it on the secondary node, but that was how I read it.



  • 11.  Re: Do the audit_purge and manage_binlogs scripts still apply?

    Broadcom Employee
    Posted Aug 06, 2015 09:22 AM

    Technically speaking both nodes think if themselves as masters

     

     

    Thanks,

     

    Derek Orr

     

    CA Technologies |885 West Georgia Street Ste 500 | Vancouver, BC V6C 3G1

    Office: 1-778-328-5285 | Mobile: +1 778 980 0029 | Derek.Orr@ca.com

     

    <mailto:>[CA]

     

    <http://www.ca.com/us/default.aspx>[Twitter]<http://twitter.com/CAInc>[LinkedIn]<http://www.linkedin.com/company/1372?goback=.cps_1244823420724_1>[Facebook]<https://www.facebook.com/CATechnologies>[YouTube]<http://www.youtube.com/user/catechnologies>[Google]<https://plus.google.com/CATechnologies>[Slideshare]<http://www.slideshare.net/cainc>



  • 12.  Re: Do the audit_purge and manage_binlogs scripts still apply?

    Posted Aug 06, 2015 11:40 AM

    We can look to adjust the comments of the script but please run the binary log management script on both database nodes.



  • 13.  Re: Do the audit_purge and manage_binlogs scripts still apply?



  • 14.  Re: Do the audit_purge and manage_binlogs scripts still apply?

    Posted Jun 11, 2017 01:10 PM

    Hi svvoorn,

    It appears that binary log management script at http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec0000001285.aspx is not found. Do you have a new location where it can be accessed. Thanks!



  • 15.  Re: Do the audit_purge and manage_binlogs scripts still apply?

    Broadcom Employee
    Posted Jun 19, 2017 01:14 AM