CA {Any Product} Log Clean UP for Performance of Services

[Alan_Baugher]

Hello All,

 

You may have noticed that after running ANY service on MS Windows OS that create individual logs, there is a certain point that the number of individual files will impact the performance of the OS to display the list of files within that folder.   Where a folder before may display 1000's of files in seconds, may take minutes for 10,000-100,000 of files.

 

The usual partition of MS Windows Servers is NTFS, which will allow a maximum of 4,294,967,295 files to exist under one folder. 

 

This I/O performance hit will impact the service ability to execute in a timely manner.    In one case, a customer reported the CA Identity Manager's Explore operation (query only) failed for certain endpoint, and removed the IMPS endpoint pointer objects; and the referential integrity inclusions as well, to the IMPS Global User objects.  

 

Below is an example how to use the OS scheduler from MS Windows and a 3rd party compression tool, 7zip, to manage the growth of logs; and retain them for historical/governance.

This process is expected to be in use with normal OS backup; to retain logs indefinitely per the customer's governance controls/requirements.

 

This process may also be used for all tier and all OS.

 

 

Example below focuses on the CA Identity Manager IMPS services, but these same processes may be used for ANY CA or other Vendor solution as well.

 

 

I have two (2) scripts below:

1 - Copy the lines below into a windows batch file, e.g.   log-clean-up.cmd  and  then the second script into datestring.cmd

2 -  Download & install 7zip http://www.7-zip.org/

3 -  Update the path in the script to the path of 7zip CLI executable

4  -Then execute the batch file, log-clean-up.cmd, to create MS Windows Scheduler jobs.

 

*********************** ******************************************* ****************************

log-clean-up.cmd 

*********************** ******************************************* ****************************

 

::Automate cleanup of IMPS Server eta logs, IMPS server ads/cam/caft logs, IAMCS and Directory logs for IMPS Server on Window Servers OS of 2008/2012.

::1) Download a command-line (CLI) compress tool, such as 7zip's 7za.exe

::http://downloads.sourceforge.net/sevenzip/7za920.zip

::2) Extract 7za.exe and ensure PATH is updated with 7za installation folder.

::3)  Create a batch file to clean up folder or directly call the 7za.exe via forfiles command

 

@echo on

call datestring.cmd IMPS_Clean_Up_Logs_Start

 

robocopy F:\media\tools\7zip\7za920 E:\Programs\7zip /e /r:0 /w:0

PATH=%PATH%:E:\Programs\7zip

setx PATH %PATH% /M

 

::Examples:

 

::Compress:  Compress any IMPS *.log file older than 14 days.

SCHTASKS /Create /SC DAILY /TN IMPS_COMPRESS_LOGS /TR "forfiles /P \"E:\Programs\CA\Identity Manager\Provisioning Server\logs\" /S /M *.log /D -14  /C \"cmd /c 7za a -t7z imps_logs.7z @fname.log\"  "  /ST 01:00  /F

 

::Compress:  Compress any IMPS ADS *.txt file older than 14 days

SCHTASKS /Create /SC DAILY /TN IMPS_ADS_COMPRESS_LOGS /TR "forfiles /P \"E:\Programs\CA\Identity Manager\Provisioning Server\logs\ADS\" /S /M *.txt /D -14  /C \"cmd /c 7za a -t7z imps_ads_logs.7z @fname.txt\"  "  /ST 01:10  /F

 

::Compress:  Compress any IAMCS *.log file older than 14 days

SCHTASKS /Create /SC DAILY /TN IMPS_IAMCS_COMPRESS_LOGS /TR "forfiles /P \"E:\Programs\CA\Identity Manager\Connector Server\jcs\logs\" /S /M *.log /D -14  /C \"cmd /c 7za a -t7z iam_jcs_logs.7z @fname.log\"  "  /ST 01:20  /F

 

::Compress:  Compress any DIR *.log file older than 14 days

SCHTASKS /Create /SC DAILY /TN IMPS_DIR_COMPRESS_LOGS /TR "forfiles /P \"E:\Programs\CA\Directory\dxserver\logs\" /S /M *.log /D -14  /C \"cmd /c 7za a -t7z ca_dir_logs.7z @fname.log\"  "  /ST 01:30  /F

 

::Delete:  Remove any IMPS *.log file older than 120 days.

SCHTASKS /Create /SC DAILY /TN IMPS_CLEANUP_LOGS /TR "forfiles /P \"E:\Programs\CA\Identity Manager\Provisioning Server\logs\" /S /M *.log /D -120  /C \"cmd /c del @fname.log\"  "  /ST 02:00  /F

 

::Delete:  Remove any IMPS ADS *.txt file older than 120 days

SCHTASKS /Create /SC DAILY /TN IMPS_ADS_CLEANUP_LOGS /TR "forfiles /P \"E:\Programs\CA\Identity Manager\Provisioning Server\logs\ADS\" /S /M *.txt /D -120  /C \"cmd /c del @fname.txt\"  "  /ST 02:10  /F

 

::Delete:  Remove any IAMCS *.log file older than 120 days.

SCHTASKS /Create /SC DAILY /TN IMPS_IAMCS_CLEANUP_LOGS /TR "forfiles /P \"E:\Programs\CA\Identity Manager\Connector Server\jcs\logs\" /S /M *.log /D -120  /C \"cmd /c del @fname.log\"  "  /ST 02:20   /F

 

::Delete:  Remove any DIR *.log file older than 120 days.

SCHTASKS /Create /SC DAILY /TN IMPS_DIR_CLEANUP_LOGS /TR "forfiles /P \"E:\Programs\CA\Directory\dxserver\logs\" /S /M *.log /D -120  /C \"cmd /c del @fname.log\"  "  /ST 02:30   /F

 

::4) Execute and validate.  Update file search date-stamp as needed.

::5) View in MS Windows Task Scheduler.   taskschd.msc

 

call datestring.cmd IMPS_Clean_Up_Logs_Done

 

 

 

*********************** ******************************************* ****************************

datestring.cmd

*********************** ******************************************* ****************************

 

:: *** Set the date string for file naming and archive ***

for /f "tokens=1,2" %%u in ('date /t') do set d=%%v

set t=%time:~0,2%%time:~3,2%%time:~6,2%%time:~9,2%

if "%t:~0,1%"==" " set t=0%t:~1%

set datestr=%d:~6,4%%d:~0,2%%d:~3,2%_%t%

 

echo %datestr% %1%

@echo Complete > %datestr%_%1%_Install.log

 

*********************** ******************************************* ****************************

 

 

 

Let me know if you find this of value.

 

Other Notes:

 

Ensure the IMPS\logs for the im_ccs, im_jcs, im_ps services are less than 100MB; and the IMPS\logs\ADS logs are rotated when im_ccs service is bounced.

Ensure that the default log level of IMPS is changed from Log Level 7 (all-include searches) to Log Level 3 (create/mod/term use-cases; useful for governance/troubleshooting)

 

 

 

Cheers,

 

A.

[Anon Anon]

Hi Alan

 

Looks good.

Only comment I have is that datestring.cmd needs to be tailored to the date format used.

For example on my machine which uses Norwegian, Bookmål (Norway) format, the short date is dd.MM.yyyy and this will not work in the datestring.cmd.

To ease the transitiion to another format, I suggest that you put in a REM where you specify what output of date /t is. That may help others to rewrite it.

I am not sure how your finished %datestr% will look like.

 

Cheers, Atle

[Alan_Baugher]

Good point, Atle,

 

I used the "date" command in a wrapper script to check on performance, but others may find it of value as well.

 

 

The MS Windows 2012 R2 system, the command was developed on returns the following string:

 

C:\>date /t

Thu 09/03/2015

 

This date command is very limited to what is returned, from the OS.    

 

C:\>date /?

Displays or sets the date.

DATE [/T | date]

Type DATE without parameters to display the current date setting and a prompt for a new one.  Press ENTER to keep the same date.

If Command Extensions are enabled the DATE command supports the /T switch which tells the command to just output the current date, without prompting for a new date.

 

 

Within MS Windows Scheduler, the command returns the following info:

 

C:\>schtasks  -?

SCHTASKS /parameter [arguments]

Description:

    Enables an administrator to create, delete, query, change, run and end scheduled tasks on a local or remote system.

Parameter List:

    /Create         Creates a new scheduled task.

    /Delete         Deletes the scheduled task(s).

    /Query          Displays all scheduled tasks.

    /Change         Changes the properties of scheduled task.

    /Run            Runs the scheduled task on demand.

    /End            Stops the currently running scheduled task.

    /ShowSid        Shows the security identifier corresponding to a scheduled task name.

    /?              Displays this help message.

[Anon Anon]

Hi Alan

 

To make a regional independend datestring.cmd file, you can use wmic. Think that should be supported by all current supported Windows OS.

So something like:

@ECHO off

for /f %%t in ('wmic os get LocalDateTime ^| findstr /b [0-9]') do set T=%%t

set datestr=%T:~0,8%_%T:~8,6%

echo %datestr%

should produce yyyymmdd_hhmmss output

 

Cheers, Atle

[Alan_Baugher]

Thanks Atle,

 

I will "borrow" it for future use.

[Rinat Matityahu]

Thanks Alan. Very useful.

[Alan_Baugher]

Thanks for the feedback!

[Nino Uziel]

Great post, thanks.