Directory COLLECT \ 00001 accumulating many files

[Rafael Maiorali Agostino Leão Ereno]

Hello everyone, I have 4 scalabilities that are collecting information, however, within the DSM \ ServerDB \ SECTOR \ COLLECT \ 00000001 directory it accumulates 50,000 files.
I also noticed that the engine is processing very slow, someone has already gone through something similar?

[lecri01]

It means either your collect engine is not running or the collect job fails,  check your domain manager to find the collect job for that specific SS then see what it is dong.

 

Richard Lechner

Principal Engineering Services Architect

 

CA Technologies

Mobile: [phone] | [e-mail]

 

<mailto:[e-mail]>[CA]<http://www.ca.com/us/default.aspx>[Twitter]<http://twitter.com/CAInc>[Slideshare]<http://www.slideshare.net/cainc>[Facebook]<https://www.facebook.com/CATechnologies>[YouTube]<http://www.youtube.com/user/catechnologies>[LinkedIn]<http://www.linkedin.com/company/1372?goback=.cps_1244823420724_1>[Google]<https://plus.google.com/CATechnologies>[Google+]<http://www.ca.com/us/rss.aspx?intcmp=footernav>

[Brian_Fontana]

These are inventory files sent by the agents, to the scalability server. Is the same happening for all SS, or only one of them?

 

These files get collected by an Engine on the DM.  You mention you have (4) SS, so you should have (5) Collect Tasks running on the DM.  One for the DM's own built-in scalability server, and four more for each remote scalability server.

 

So what are the status of these collect tasks on the engine?

Are all (5) collect tasks linked to an engine?

What are the status and last run dates?

 

In the activity window of the engine, you should see files getting processed.

[Rafael Maiorali Agostino Leão Ereno]

Hello Brian, I have 4 scalings and one engine for each. It's happening in 3 out of 4 scalabs. On the screen of the engine I see the files being processed, but it is processing only 1 files every 15 seconds, approximately.

[Brian_Fontana]

kb000046211: Scaling Client Automation: How to improve Collect Task and Replication Task Performance by Limiting the amount of Hardware and Software scans sent by Agents.
https://comm.support.ca.com/kb/scaling-client-automation-itcm-how-to-improve-collect-task-and-replication-task-performance-by-limiting-the-amount-of-hardware-and-software-scans-sent-by-agents/kb000046211

 

Follow these two recommendations from the linked document:

- Recommendation: Reduce the frequency of hardware inventory scans

- Recommendation: Advanced engine settings

[Rafael Maiorali Agostino Leão Ereno]

Hello Brian, thanks for the tip.
I performed the tunning of the environment as the document indicates, however, it remains extremely slow.
I opened a ticket in support to help with the problem.

[mccst05]

There could be MANY reasons for the engines not keeping pace with the collections, from architectural problems to database issues, network issues and many others. Please open a support case so some investigation can be performed.

 

Steve McCormick, ITIL

CA Technologies

Principal Services Consultant

[e-mail]

<mailto:[e-mail]>

[Rafael Maiorali Agostino Leão Ereno]

Hello Stephen, I already opened a ticket on the media. Thank you.

[Mladen Ilisinovic]

Did You get any solution ?
I am having similar issues with  Domain manager  scalability server,  distributed  SS are behaving  ok.

-------------------------------------------

[Stephen McCormick]

Hello,

As mentioned previously, there can be MANY reasons for a backlog in the inventory collections. Let's start with some basic info:

• How many computers are you managing?
• How many Domains and how many Scalability Servers do you have?
• Do you have any additional Inventory modules or jobs created?
• Have you modified the default inventory collection times (default is to collect at midnight + a random delay up to 90 minutes, and registration at noon + 90 minutes random)

-------------------------------------------

[Mladen Ilisinovic]

We run Client automation 14.6 GA on Windows server 2025.

I have connected  380+ Agents to Scalability server  running on domain manager.
over  night  my threads rise up to  500.000 handles. with CAF service stopped it is  70K handles.

Total amount  computers connected over  4 different  scalability servers  and domain manager  is  less than 600.

so when I leave  it as is: CPU times grow to over 90% per each CPU core, total 8 cores
inventory  modules  were  disabled to  troubleshoot  this, and  still same behaviour.

so  summary:
1 Domain manager on central location with active scalability server on  it: - 389 Agents
1 Scalability server with  28 Agents running  ok
1 Scalability server with 3 agents running ok
1 scalability server with 134 agents  running  ok.
--------------------------------
After turning  on CAF service, Handles are  increasing  constantly.

I have 5  engines, 
1 system engine serve Domain manager scalability server and 389 agents
1 engine handles  32 Agents from second zone
1 engine handles 134 agents from third zone
1 engine handles  Reports and runs over night
1 engine runs  Queries and dynamic groups

so  when I do  throttling  of  CPU usage via CPU Affinity, I can say  my server  starts to breath.
amRSS.exe reduced to use 4/8 CPU cores
system engine uses 3/8 cores
rest of engines uses 1/8 cores each.
now  I have   responsive system, but  it is temporary  fix. not solution.

ALSO:
Previous versions of  Software delivery had  concurrent  usage set to 25 agents  parallel.  now in 14.x seems this is increased to  100 agent  collect, and 50 executions in parallel.
I reduced to 20 collect / 10 execute job.

Next step is to dedicate  machine as  Scalability server to  be sure is this  meant to be or  is it bug.

Anyway  back in  release 11/12 I never experienced  so many  issues as this,  despite fact that I even had 4000 agents in some companies to report to  single domain manager's  scalability server.

best regards.

each  scalability

[Ramesh Penimeti]

Hi Mladen Ilisinovic,

Good morning.  The number of agents you listed per SS are very less so we should not see this problem until the application is not processing the files properly.  Checking the detail level application logs should help here , could you please open support ticket  for debugging ??

Thanks and regards,

Ramesh P

-------------------------------------------

[Stephen McCormick]

Hello Mladen,

Firstly this is not the same issue this thread was opened for so this is quite misleading. The original thread was for a Scalability Server with too many FILES in the collect directory, which indicates a processing backlog. This is for CPU threads which is a quite different problem.

The first thing I would say is this: You should not be using the Scalability Server on the Domain Manager to process Agents (except itself and perhaps an Enterprise Manager if you have one). This is especially true if the SQL Server is also on the Domain Manager. Even though the Domain Manager always has an SS on it, it should not be used that way except for very small environments. Highly recommend you move the Agents to one of the other servers or create a new SS for them.

Steve McCormick

[Harold Martin]

Hello Rafael,

I suffered the exact same issue and the solution provided by support was the following:

Communication issue
Errors observed included:

"Recv failed - Error Code 10053"
"queue x.x.x.x:4104 reversed identity with proxy y.y.y.y:4104"
Queue deletion messages such as "no send/recv and link_timeout" in CAM logs

According to the documentation for CA Client Automation 14.5, error 10053 (WSAECONNABORTED) indicates that an established connection was aborted by the local host software.

To resolve this, CAM was reconfigured to use TCP between the DM and the SSs, with the following parameters:

disable_proxies = yes
fragment_size = 1024

Delay processing a file (CORA errors)
A delay was identified due to CORA errors:

[470207]
[470200]

Running CoraCleanup on the Domain Manager resolved the issue.

Residual .E0 files in collect folders
Files *.E0* were not being removed from:

..\DSM\ServerDB\SECTOR\COLLECT\00000001

on the Scalability Servers.

These files were being continuously generated due to a legacy Intellisig scan being enabled.
Disabling the Intellisig scan under:

All Computers → Configuration → Collect Tasks → Software Discovery

stopped the creation and transfer of these files.

let me know if this help.

------------------------------
Harold Martin
Senior Consultant
------------------------------