Blogs

Auditing and Logging on Layer7

By Chris Waun, Customer Engagement Engineer 

Effective auditing and logging are foundational to API security, compliance, and operational efficiency

 For the Layer 7 API Gateway, these features help organizations track and record system activities, providing insights into who is accessing APIs, what actions are being taken, and when they occur. By enabling comprehensive logs and audit trails, the Layer 7 API Gateway empowers teams to detect anomalies in API activity, troubleshoot issues, and meet regulatory requirements. In this blog, we’ll explore the key aspects of auditing and logging within the Gateway, offering best practices to enhance visibility, improve security posture, and ensure smooth API operations.

Key Differences Between Logging and Auditing

Auditing in the Layer7 API Gateway is categorized into three key types: System Audits, Administrative Audits, and Policy Message Audits—each serving a distinct purpose:

• System Audits are automatically generated and reflect background operations such as server startups, license checks, or configuration reloads. These are always active and cannot be disabled, ensuring continuous tracking of critical system-level events.
  
  

• Administrative Audits are triggered by user-driven actions, such as publishing policies or managing users through the Policy Manager, RESTman, or GraphMan APIs. Unlike system audits, these can be configured to control which types of administrative activity are captured, offering flexibility and noise reduction.
  
  

• Policy Message Audits capture messages during policy execution. These are typically recorded at the Info level, but can also include Warning or Severe messages when something noteworthy or problematic occurs. They help you understand how policies are executed and where issues may arise.
  
  

Together, these audit types provide a comprehensive picture of what’s happening inside the Gateway—from system behavior and configuration changes to policy-level activity—while allowing you to maintain a manageable audit footprint.

Logging Configuration: What’s Important

When configuring logging it's important to balance the level of detail with performance and disk usage in mind. Detailed logging can quickly consume disk space, potentially degrading performance due to limited resources and high number of disk writes. To optimize this, it’s recommended logging to an external disk source rather than the Gateway itself. This frees up resources that would be used for disk I/O to better focus resources on core functionality. 

Using a queuing mechanism to manage and process the data is also an option that allows for more efficient handling of sending log data off-box, ensuring minimal impact on the Gateway’s performance. This approach helps buffer log data during peak loads, preventing potential data loss and ensuring reliable delivery to external systems.This approach facilitates enterprise reporting and data trending by enabling more effective storage and manipulation of log data. It provides visual and actionable insights, ensuring the API Gateway operates efficiently and remains optimized.

Best Practices

To maximize the effectiveness of auditing and logging with the Layer 7 API Gateway, it’s crucial to implement a set of best practices. Start by enabling only the necessary level of logging to avoid excessive resource consumption. 

Auditing should be configured to track key system and administrative activities, ensuring critical actions are captured without overwhelming storage capacity. 

For logging, direct data to external systems where possible, using queuing mechanisms to manage traffic and reduce load on the Gateway itself. Regularly review and prune logs to maintain efficient storage, and ensure logs are structured to integrate smoothly with enterprise reporting and monitoring tools. 

Below are 3 practices to consider while setting up logging and auditing:

1. Define Clear Objectives for Auditing and Logging - Identify specific business, security, and compliance goals that your logging and auditing strategy should address, such as regulatory requirements

2. Enable Appropriate Log Levels - Configure logging levels to balance detail with performance. Use INFO or WARNING for routine operations and FINE only for troubleshooting. Avoid enabling FINER/FINEST or excessive verbosity in production environments to prevent performance degradation.

3. ​​Establish Log Rotation and Retention Policies - ​​Implement log rotation to manage file sizes, archiving older logs regularly to prevent disk space exhaustion. Retain logs based on compliance needs (e.g., 90 days for general operations, longer for audits).

4. For Audits, it is very important in a production environment that they are either disabled entirely, or used in a very limited capacity to capture failure data that is important for troubleshooting.  Auditing success data is largely unnecessary, and can capture a huge amount of data very quickly that is mostly unimportant.  Furthermore it adds latency to the policy execution when the “Add Audit Details” assertion is used in policy, and it is encouraged to use it sparingly and only when necessary in a production deployment.

Conclusion

In conclusion, effective auditing and logging are crucial for security, compliance, and performance when leveraging the Layer 7 Gateway. By implementing best practices such as controlling log verbosity, using external storage, and leveraging queuing mechanisms, organizations can optimize data collection while minimizing system impact.   Using auditing only when necessary can help avoid performance lags and reduce the amount of data written and collected. Ensuring that it’s data that is needed and not gathered without purpose.

As a side note, integrating OpenTelemetry (OTel) to your API Architecture can enhance logging and monitoring by providing a unified approach to collecting traces, metrics, and logs. OTel enhances observability by correlating data across services and APIs, allowing for more precise monitoring and faster diagnostics. When combined with traditional logging and auditing strategies, OTel helps organizations gain deeper insights, improve performance, and ensure better security and operational efficiency across the entire API ecosystem.