Cloud Foundry - Where are audit events stored and are they logged

[Guillaume Nourry-Marquis]

As a security auditor, I want to have a list of all the tasks executed with :

1- [MISSING] Identity of who triggered the task

2- [MISSING] Command that was executed

3- [OK] Targetted app/space/org

4- [OK] Stdout of run task

5- [OK] Status and completion time of task

 

In the logs, I only see the following :

 

{ \\n cf_app_id: <<MY_APP_ID>>\\n cf_app_name: <<MY_APP_NAME>>\\n cf_org_name: <<MY_ORG>>\\n cf_space_name: <<MY_SPACE>>\\n deployment: <<DEPLOYMENT_ID>>\\n event_type: LogMessage\\n ip: <<SOME_IP>>\\n job: diego_cell\\n job_index: b41f713d-049d-44a9-ab4e-edae015eccdd\\n message_type: OUT\\n msg: <<STDOUT of the executed command>>\\n origin: rep\\n source_instance: 0\\n source_type: APP/TASK/<<MY_TASK_NAME>>\\n timestamp: 1616773073118017800\\n}

Followed by :

 

{ \\n cf_app_id: <<MY_APP_ID>>\\n cf_app_name: <<MY_APP_NAME>>\\n cf_org_name: <<MY_ORG>>\\n cf_space_name: <<MY_SPACE>>\\n deployment: <<DEPLOYMENT_ID>>\\n event_type: LogMessage\\n ip: <<SOME_IP>>\\n job: diego_cell\\n job_index: b41f713d-049d-44a9-ab4e-edae015eccdd\\n message_type: OUT\\n msg: Exit status 0\\n origin: rep\\n source_instance: 0\\n source_type: APP/TASK/<<MY_TASK_NAME>>\\n timestamp: 1616776305274867000\\n}

 

Seems I am missing all the events describe here https://docs.cloudfoundry.org/running/managing-cf/audit-events.html#special-events'.

 

How can I have these exported to our External Auditing Warehouse ? I want to make sure I'm not missing any events.

[Daniel Mikusa]

Audit events, those that come from `cf events`, are stored in Cloud Controller.

 

You can access them with `cf events` or you can use the API directly with `cf curl` to pull back information in more flexible ways.

 

https://apidocs.cloudfoundry.org/15.3.0/events/list_all_events.html (for older foundations, deprecated)

http://v3-apidocs.cloudfoundry.org/version/3.99.0/index.html#list-audit-events (for newer foundations, recommended to use this API)

 

There are two audit related events:

• audit.app.task.cancel
• audit.app.task.create

 

Basically, you just want to query with a filter for the type and indicate your type. You can then see the audit event records. I ran this in my local lab, which is older so I used the v2 API.

 

{ "total_results": 8, "total_pages": 1, "prev_url": null, "next_url": null, "resources": [ { "metadata": { "guid": "3fb2b2ad-48f9-4c19-b12b-ff714c8c3397", "url": "/v2/events/3fb2b2ad-48f9-4c19-b12b-ff714c8c3397", "created_at": "2021-03-04T18:27:58Z", "updated_at": "2021-03-04T18:27:58Z" }, "entity": { "type": "audit.app.task.create", "actor": "265b747a-e0f9-4448-93f4-ac2284d4811f", "actor_type": "user", "actor_name": "system_services", "actor_username": "system_services", "actee": "66b21180-1b32-43ba-a239-0f8bff9d4330", "actee_type": "app", "actee_name": "mytask", "timestamp": "2021-03-04T18:27:58Z", "metadata": { "task_guid": "b825a12d-6c0a-4b07-a454-23642638d6fc", "request": { "name": "mytask", "memory_in_mb": 1024, "command": "[PRIVATE DATA HIDDEN]" } }, "space_guid": "f654f75d-51d9-43d5-b4fd-4db9f395490b", "organization_guid": "64fcd439-7345-4d79-bcd9-1c2cb2f5a0df" } }, ...

Some notes on this:

• The actor type is "user", so the actor guid is the user's guid. You could look that up but the actor_name/actor_username will give you the user's name.
• The actee is "app", so the actee guid is the app's guid. Again, the actee's name is the app's name so you don't necessarily need to look up the guid unless you need more info about the app.
• The metadata has details about the task
• The org & space guid's are listed. If you want the names, you'd have to look them up.

 

To summarize the info you're trying to get:

 

>1- [MISSING] Identity of who triggered the task

 

That would be the actee.

 

>2- [MISSING] Command that was executed

 

That is technically in the metadata, but it's redacted :(

 

If you want this, you'd need to look it up. There is no task API for the v2 API, so you need to use the v3 API. That is basically `cf curl /v3/tasks/<task-guid-from-metadata>`. The result will have a command property with the full unredacted command.

 

>3- [OK] Targetted app/space/org

 

In the task audit record.

 

>4- [OK] Stdout of run task

 

The only place this will show up is in the actual logs. Those are not stored in Cloud Controller. Logs go through the loggregator system & are ephemeral only (unless you send logs from Loggregator onto a syslog aggregator).

 

>5- [OK] Status and completion time of task

 

The audit record is for the creation of the task, so it doesn't have this info. You can get that from the task record though, same place as you find the command. The state field will tell you if the task was successful or failed, if it failed the result will contain a failure_reason with the reason it failed.

 

I believe that the updated_at field on the task record (also same as the command), is the closest there is to a completed time. Although you could probably pull the time stamp of when the container was stopped out of the log stream. That would be when the command completed, although it's going to be trickier to fetch. Usually, those two timestamps should be pretty close together though, since the task record will get updated with the final state/error message when it finishes.

 

Hope that helps!

[Guillaume Nourry-Marquis]

So they are not available in the cloud controller logs for audit like everything else? How could we have them sent over to the Firehose for Security audit then? We need to continuously poll all apps or all events ?

[Daniel Mikusa]

What you can get from the events (the audit events like you referenced) is not the same as the logs, it's a different source of information. The events are stored in Cloud Controller and are queryable for users. Cloud Controller generates those records as it's performing various actions. They are purpose-built to be able to understand what happened on the system.

 

While the logs come from Cloud Controller as well, the purpose is different. The logs are meant to understand the state of Cloud Controller, so if it's behaving incorrectly, it's possible to see how it got into that state (i.e. for troubleshooting). The purposes may overlap in places, so if you can find what you need in the logs, it's totally fine to consume that information. If what you need is not in the logs, there would be no way to add it short of logging a feature request. Adding info to the logs requires code changes, which need to be submitted as a feature request (you can talk to your account rep or platform architect about logging a feature request).

 

It's also important to note that logs are *not* considered a public interface. Thus the content and format can change without notice. In practice, this happens infrequently, but unlike using a public interface like the Cloud Controller API, there are no deprecation periods for changes and unless it's a major change, there's unlikely to be mention of it in the release notes either. In short, it is used at your own risk. The only exception to this is the logs that are purposefully omitted for auditing.