CA API Gateway 9.3 : Log/Audits Forwarding To Monitoring Tool

[Amit Jaiswal]

Hi All,

My requirement is to send the Gateway logs and audits to Datadog or similar monitoring tool via API.

Since, syslog server settings can use either TCP/UDP so sending logs/audits to syslog server is out of question.

So, I decided to play around with Audit Sink , where I assumed that I can add additional context variables in the Audit Sink policy and Route via http to wherever I want. But, I soon realized that adding extra context variables in the audit sink policy is not working because Audit Sink policy collects only the audits from each executed service in gateway.
This means, if I need to log 10 kinds of log values for all the services of the gateway, then I need to "Add Audit" those 10 context variables in each service of the gateway. Then only they will be forwarded to the Audit Sink policy.
There is no global setting like we have "trafficlogger" clustered property for syslog.

If my understanding is correct then what is the right approach to use any Global Setting (like syslog) to send logs/audits via API ?
or is there still a way to edit Audit Sink policy and make it work for all services without "Adding Audit" in each service.

Please guide.

Regards,
Amit

[Deactivated User]

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/9-3/security-configuration-in-policy-manager/tasks-menu-security-options/manage-log-audit-sinks/log-sink-properties.html

Format for Audit Logs 
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/9-4/administer-the-gateway/gateway-auditing-threshold-and-format.html

Audit Cluster Properties:
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/9-4/reference/gateway-cluster-properties/audit-cluster-properties.html

This attributes can be customized to add system variable like
${service.name} and so on
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/9-4/reference/context-variables/service-policy-context-variables.html


Finally you can also look at tools for monitoring 

https://community.broadcom.com/communities/community-home/digestviewer/viewthread?MID=780447

------------------------------
Pre-Sales Consultant
CA Southern Africa
------------------------------

[Amit Jaiswal]

Hi,
Thanks for sharing the links, but I had already gone through all of them, still it doesn't solve my problem.

1. I need to send all gateway logs/audits via API call to a monitoring tool. Using Audit Sink for this purpose means, I will have to edit all the services of my gateway and use "Add Audit Details" assertion with value like "${httpRouting.latency}, ${request.routingTotalTime}, ${request.mainpart}, ${request.url}, ${httpRouting.url}, ${httpRouting.reasonCode}, ${request.http.method}, ${response.mainpart}, ${response.http.status}".
Because Audits do not store such values by default.
I am looking for an easy solution where I do not have to edit all my services and add "Add Audit Detail" assertion in all of them.
Also, I see that Audit Sink policy collect Audits only but not logs.
So how can i send Audits + Logs via API call.

2. I know this easy global solution is possible in Log Sink, where we can send the logs to a syslog server via TCP and change the cluster-wide property "traffic logger to ${httpRouting.latency}, ${request.routingTotalTime}, ${request.mainpart}, etc.
But Log Sinks can't be forwarded via API call and I can't change their format from text to XML/JSON.

3. Even if I decide to edit all my services and use Audit Sink, I am finding it difficult to edit the audit xml.
Here the target service (gateway.audit.sink.url is receiving blank message.
But I am able to audit this xml using request.mainpart but not able to edit.

Regards,
Amit

[Jay MacDonald]

Context variables that are available in the service policy (for example, ${foo}) are available in the Audit Sink Policy as ${audit.var.<originalContextVar>} (e.g. ${audit.var.foo}. I rarely use the Convert Audit to XML, since it takes *everything* from the audit context and makes an XML document from it, which would then need to be manipulated by XSLT (or the Add or Remove XML Element assertion) if you need something different. I usually build the request in the ASP to incorporate the exact information that the API to which it is being sent requires, which can easily be fleshed out with ${audit.var.*} content.

If the Convert Audit to XML has everything your API requires and you just want to addd a few elements then the use XSLT or the ADD or Remove XML Element assertion.

Refer to https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/9-4/reference/context-variables/audit-sink-context-variables.html

Cheers!

JyMac

------------------------------
Jay MacDonald - Adoption Architect - Broadcom API Management (Layer 7)
------------------------------

[Amit Jaiswal]

Hi Jay,

Thanks for the tip. 
audit.var.* is really great and I was looking for a variable like this.
Funny that I missed this few days ago in the broadcom page, since it was the last entry in the long list of audit sink variables :-)

1. I observed that to use these audit.var.* variables in audit sink policy, still I need to edit all existing gateway services and add "Audit Messages in Policy" assertion to each service. I understand this is related to Audit SInk, so Audit assertions must be added.
But, is there any global setting to enable Audits for all services without editing them ?
I checked cluster-wide properties, but couldn't find any.

2. I did't get usage of ASP to create custom API message. Did you mean external ASP framework ?
I was trying to use "Set Context Variable" to create a custom json message like "{ key : "audit.var.request.url"} etc.
And then use "Route via HTTP(s)" assertion to send it.
But this seems to send a blank message to monitoring API. Could you please guide in a little detail, on how can i create a custom message ?

[Jay MacDonald]

Message audits are triggered per the value of ${auditLevel} at the end of processing relative to audit.messageThreshold cluster-wide property. ${auditLevel} is initialised to INFO but can be bumped to WARNING by several mechanisms:

  1. Using the Audit Messages in Policy assertion
  2. Setting an Add Audit Detail to WARNING
  3. Any assertion that results in WARNING or SEVERE (most are FINE or INFO)

If ${auditLevel} is at or above the audit.messageThreshold, which is WARNING by default, then the message audit it triggered. Ergo, to force every message to trigger an audit, set audit.messageThreshold to INFO.

As for #2, make sure you set the type to message and set the content-type correctly. If the message is still blank​ then contact support.

------------------------------
Jay MacDonald - Adoption Architect - Broadcom API Management (Layer 7)
------------------------------