Integrating Audit/Logging with Splunk

[hocmo01]

I am attempting to off-box log and audit messages to Splunk using the Splunk Logging Driver for Docker e.g., added this configuration to MGW docker compose yml:

 

logging:
  driver: splunk
  options:
    splunk-token: "<splunk token>"
    splunk-url: "<splunk url"
    splunk-insecureskipverify: "true"

 

This method uses the HTTP Event Collector feature of Splunk.  An access token is defined within Splunk and used to gain access to the Splunk HTTP(S) endpoint from the Docker container.

 

The MGW Docker container builds successfully (connecting to the Splunk instance successfully), but no requests (log messages) are seen on the Splunk instance.  I have tried manipulating the log and audit levels of the MGW, but this has no effect.

 

Not sure how to resolve this.  Any suggestions are appreciated.

[balak]

Hi, 

I am trying tosimilar solution. Was the splunk solution working for you? I am trying to implement in CA Layer 7 product for Docker image. 

[DustinDauncey]

Hello,

 

I do not have a Splunk license (since it is third-party) to use in my lab, so I cannot reproduce this. I wanted to ask though... what specific issue or error are you seeing when trying to integrate Splunk with the Container Gateway in Docker? If you can provide some log entries or general behaviour and configuration details, I think this may help us move this forward for you.

[Stephen Hughes]

We have just setup this in the lab and was able to get it working. We create a new HTTP Event collector:

 

 

For the docker-compose.yml file we setup the following settings:

 

version: '3'
services:
ssg:
image: caapim/microgateway:1.0.0-CR01
extra_hosts:
- "httpbin.mycompany.com:10.7.36.179"
- "otk.mycompany.com:10.7.32.187"
deploy:
resources:
limits:
memory: 2048m
logging:
driver: splunk
options:
splunk-url: "https://apim-centos7-util.support.local:8088"
splunk-token: "8da0c475-fbdb-482e-897f-61811bde5ca8"
splunk-insecureskipverify: "true"
env_file:
- ./config/core.env
- ./config/license-agreement.env
- ./config/license.env
- ./config/certificates.env
- ./config/otk.env
- ./config/jwt.env
- ./config/feature-flags.env
- ./config/solutionkits/policysdk.env
environment:
SSG_ADMIN_USERNAME: "admin"
SSG_ADMIN_PASSWORD: "password"

 

Finally we built the environment to run using the command:

 

docker-compose --project-name microgateway                --file docker-compose.yml                --file docker-compose.db.consul.yml                --file docker-compose.lb.dockercloud.yml                up -d --build

 

In the Splunk search we used the search string of : source="http:Support" (index="history" OR index="main" OR index="summary")

 

 

Sincerely,

 

Stephen Hughes

Broadcom Support

[balak]

Thanks for sharing the info Stephen. I already implemented Splunk using the ECS not with docker-compose.yml. Right now our gateway’s routing the sys and audit logs to Splunk.

 

I am exploring routing the stats to database, it seems this started in 9.4.

 

 

Thanks & Regards

Bala

[Prashant0384]

Yes, in 9.4 we can route Gateway Dashboard statistic data as well to the external db . 

[balak]

Hi Prashant,

Is it possible to move the statistic to external source like Splunk? Or we can only able to move to external db?

 

 

 

 

Thanks & Regards

 

Bala

[Prashant0384]

We can only move statistic data in to  external DB  (This functionality has been added in 9.4 earlier only audit data can be written in to external db)slimier like audit data in to external db  . However you can create dashboard in to splunk for all the data that is getting logged in to traffic log.

That will cover most of thing that you are seeing over dashboard .

[balak]

Hi All,

I was just calculating the latency the way it handles the logs. Two scenario's I tested. With splunk log-driver the latency is more compare to syslog forwarder. Both scenarios tested on AWS environment.

 

1. logs --> rsyslog --> configure splunk forwarder

2. logs --> HTTP Event collector -->Splunk (Splunk log-driver)

The later one has more latency compare to first one. It might be due to the 1. UDP 2. HTTP.

 

Does any one observed the same behavior?