Housekeeping/Health Check of CA ESP

[Dayanand Mishra]

Dear All,

Do we have Admin guide available?

[Lucy Zhang]

Hello Dayanand,

You may want to review "Operating" and "Programming" manuals for information that should be known by ESP admin:

https://techdocs.broadcom.com/us/en/ca-mainframe-software/automation/ca-workload-automation-esp-edition/12-0/operating.html

https://techdocs.broadcom.com/us/en/ca-mainframe-software/automation/ca-workload-automation-esp-edition/12-0/programming.html

Following is about ESP healthcheck configuration:

https://techdocs.broadcom.com/us/en/ca-mainframe-software/automation/ca-workload-automation-esp-edition/12-0/installing-and-configuring/ca-wa-esp-edition-health-checks.html

Based on my support experience, I would suggest below:

- Set up console message monitoring (like using OPS/MVS) for critical ESP error messages, like 528E, 1072W, 1070W etc. Note: the last two messages are for the ESP file usage. Many of the severe problems are related to ESP files filled up. 

- Routinely run ESP utilities to clean up ESP files, like CYBESUT7 & CYBESUT8;

- Routinely run PURGSCHF/PURGAPPL to keep the CSF/Application panel manageable;

- Make sure the two temporary files in history archive JOB have enough space, so that this JOB won't fail.

- If it's an ESP test system, which means users can ignore the failure/errors after their tests, you may want to check routinely and force complete the aged incomplete workloads or let your users do so.

Just some thoughts, 

Lucy

[Nitin M]

Hi Lucy,

This is helpfull, Can you please suggest some more standard things need to done as a part of health check.

[Michael Bieganski]

Hi,  I don't recall ever having to run either cybsut7 and/or cybsut8  for any file usage issues (fortunately), but I do
see these five 1072W messages in the ESP r11.4 JESMSGLG:

ESP1072W 16% CI usage, 100% CA usage, 1 extent, Dsn=TSGEN.JSS03DB.USERDEF.DATA
ESP1072W 22% CI usage, 100% CA usage, 1 extent, Dsn=TSGEN.JSS03DB.INDEX.DATA
ESP1072W 65% CI usage, 95% CA usage, 1 extent, Dsn=TSGEN.JSS03DB.JOBINDEX.DATA
ESP1072W 66% CI usage, 97% CA usage, 5 extents, Dsn=TSGEN.JSS03DF.JOBSTATS.DATA
ESP1072W 54% CI usage, 100% CA usage, 22 extents, Dsn=TSGEN.JSS03DB.EVENT1.DATA

I'm not adept when it comes to vsam, but I believe the CA % usage being 100% is not necessarily anything to be alarmed at.
At what CI% or extents percentages would you suggest we act?   And would one run both CYBESUT7 & CYBESUT8?

[Lucy Zhang]

Hi Mike,

The CI and CA % should be for the currently used extent if the VSAM file has any extent. You can run IDCAMS LISTCAT ALL to check more.  And following online doc has more explanation on this message:

https://techdocs.broadcom.com/us/en/ca-mainframe-software/automation/ca-workload-automation-esp-edition/12-0/messages/1072w.html

As you may already know, ESP constantly reads/updates the above ESP files, especially JOBINDEX and JOBSTATS.

For better performance, we recommend having any extent or few extent for VSAM files. Ideally the ESP VSAM file should use only primary allocation with usage less than 70%.

If you haven't run CYBESUT7 and CYBESUT8 before, I suggest that you run them in ANALYZE mode to check if there are many obsolete records are eligible for deletion.  They are meant to delete the records for applications or jobs that haven't run for long time. If none or few records satisfy, then they can't delete many records in JOBINDEX and JOBSTATS files.

Hope this helps,

Lucy

[Lucy Zhang]

Hi Nitin,

I will raise this topic in our next team meeting and see if more should be added.

Thank you,

Lucy

[Nitin M]

Hi Lucy,
Thanks for response.Would request for inputs if any.

[Lucy Zhang]

Hi Nitin,

Yes. Please refer to the comments below from my teammate Richard Baber. 

Thank you,

Lucy

[Richard Baber]

Hi Nitin,

A long time ago I was a customer who was the ESP admin at our site. I would run the following JCL at the beginning of every shift so that I had a snapshot of my ESP environment every 8 hrs. This allowed me to track if any changes we performed during a shift. It would also let me trend workload. I would write them to GDGs so I could go back in time if needed. Hope this helps 

//STEP1 EXEC PGM=ESP,REGION=3000K,PARM='SUB(????) ****' ???? ===> Your ESP subsystem
//STEPLIB DD DISP=SHR,DSN=ESP.CD7YLOAD ===> Your Loadlib
//SYSPRINT DD SYSOUT=*
//*SYSPRINT DD DSN=ESP.LIST2,
//* DISP=(NEW,CATLG,DELETE),
//* SPACE=(CYL,(300,1),RLSE),UNIT=SYSDA,
//* DCB=(DSORG=PS,RECFM=FB,LRECL=132)
//SYSIN DD *
L LEVEL(-.-) ALL
LCSF STATUS='WAITING-'
OPER ESPCOM DEST(-)
OPER AGENT -;
LISTHOL -;
RESDEF - LIST
VTLIST
PASSWORD LIST
LISTCKPT
LISTQ
LISTHIST
OPER LISTAPTF
OPER LISTTRAK
OPER TRACKING
LISTSYML -;
LISTCAL -;
ROUTING
//
 

[KATHY Mullin]

Hey, Rick! could I do that trend report for a time in the past? I know you said you would keep gdg's to go back but wondered if I could track changes we did that has already passed? I which I would have seen this a month or two ago.  :( ​

[Richard Baber]

Hi Kathy,

This job and commands are designed to take a live snapshot of the system at the time of execution. I would run this every 8 hours and write to a GDG. This was so I could go back to see status from the past and help pinpoint when something changed. Once a change is made it is difficult to look in the past. There could be entires written to the JESMSGLG or AUDITLOG of ESP. You could go back to look at the STC output from the past and possibly find commands that may have been entered. However this job will give more detail and stored for easy comparison. Let me know if you would like to talk about this and I can give you a call.
Thanks,

Rick Baber 

[KATHY Mullin]

​Thanks Rick! We just did a big cutover and had hundreds of changes! I wish I would have saw this about 2 months ago. LOL! I got the job to work and see the output in the job but the file was empty so still working on that. I might some time next week if you want to talk for 30 min or so ???