Getting CAM errors on Domain Manager

[Ryan Lucas]

Hello, bit of an emergency today.

My main domain manager is not working.  I am getting CAM errors like this:
Fri Oct 11 10:03:39 2019 ERROR accept() failed, we return now: socket error 10048

If i kill the cam.exe, the GUI will function for a little bit, but it's still not running correctly.
Is there any way someone can help me out?

Thank you.
Ryan Lucas
814-599-5421

[Steve Parker]

Please open a support call.

Rgds

Steve

Sent from my Samsung Galaxy smartphone.

[Stephen McCormick]

You could try clearing the CAM queues. Restarting CAM doesn't actually clear the queues. On the DM:

 

• Run 'caf stop'
• Run 'camclose&cam start -c'
• This will clear all the CAM queues, which will possible remove whatever is causing your issue
• NOTE: you must see a message that CAM is starting, if you don't, run it again. CAM restarts itself automatically and sometimes the restart is quicker than the cam start
• Start CAF

 

Steve McCormick

 

Hello, bit of an emergency today. My main domain manager is not working.  I am getting CAM errors like this: Fri Oct 11 10:03:39 2019 ERROR... -posted to the "Clarity Client Automation" community

Clarity Client Automation   Post New Message   Getting CAM errors on Domain Manager Reply to Group Reply to Sender Oct 11, 2019 11:16 AM RYAN Lucas Hello, bit of an emergency today. My main domain manager is not working.  I am getting CAM errors like this: Fri Oct 11 10:03:39 2019 ERROR accept() failed, we return now: socket error 10048 If i kill the cam.exe, the GUI will function for a little bit, but it's still not running correctly. Is there any way someone can help me out? Thank you. Ryan Lucas 814-599-5421   Reply to Group Online   View Thread   Recommend   Forward   Flag as Inappropriate     You are subscribed to "Clarity Client Automation" as [e-mail]. To change your subscriptions, go to My Subscriptions. To unsubscribe from this community discussion, go to Unsubscribe. Copyright © 2005-2019 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Inc. and/or its subsidiaries. Hosted by Higher Logic, LLC on the behalf of Broadcom - Privacy Policy | Cookie Policy | Supply Chain Transparency | Terms of Use

[Ryan Lucas]

I have done the camclose & cam start -c -l and cycled the server several times.  The GUI will launch and work for a little bit, but then something happens that causes cam to break. I see it a lot when it completes a task and moves to the next one, it gives the unable to connect message. 

I do have a ticket open, but wanted to see if anyone else ran into this that i can fix it sooner.  Thanks.

[brian.fontana]

Hey Ryan.  Assume you have already used WinOffline to recycle everything?  Can you share a "camstat -a" when the problem is happening?

[Ryan Lucas]

I did just try the WinOffline and it had a hard time killing off cam, so it terminated the process.  Things came up just fine and it actually cycled through a few tasks, which it hadn't done that.  However, the camstat now gives me the camstat: select failed (15) Unable to connect to CAM server.  The tasks it was working on are still running, but then when it gets to a new task it can't connect.

[Ryan Lucas]

Just a FYI, my cam.exe process is solid at 13% and the amount of memory doesnt really appear to be moving either.

[Ryan Lucas]

I had multiple chats with support, but didnt get anywhere.  Mostly it was referring to older support docs on similar issues.
I am still having the issue and not really sure what to do at this point.  I do need some help with this but the support tech doesnt seem like he really knows what the issue is.

[brian.fontana]

Typically, if CAM is having high CPU, it's usually due to a spike/spam in messages.  The trick is to be able to narrow down which functionality, e.g. which CAM queue is causing the issue.

Because CAM is not multi-threaded, a spur of activity in messaging for any single queue will have a performance effect on all applications and remote queues that are connected with it.  Unfortunately the GUI is one of those applications.

I would recycle, and immediately try to run a "camstat -a", to see if there's any CAM queue that sticks out, that either has a huge number of sent/received, or a growing backlog in the queue length.

[Ryan Lucas]

The only thing I see with high receives is AMO_Sector_Srv and ​U-Sector_Srv

[Stephen McCormick]

A quick question on architecture if you would indulge me. How many agents are you supporting overall, how many scalabiity servers do you have, and how many agents are connected directly to your Domain?

Steve McCormick

Steve McCormick

IT Support Specialist IV

Card Services

731-676-4223

[Ryan Lucas]

Hello there.  On this specific domain there are close to 8000 servers with about 20 scalabilities.  There are a few agents reporting to the domain, but it's very few.  

In order to keep all the agents reporting in, I set up a scheduled task to kill cam.exe every 10 minutes.  So far all is the engines are up to date.  But I definitely don't want to keep it that way, just didn't want things to get stale.

[Ryan Lucas]

Just an update, still have the cam issue.  Setting the schedule task to kill cam every ten minutes allowed all of the servers to report in over the weekend.  It also allowed users to deploy software to some servers this morning.  I do not want to keep this procedure in place, but it is a workaround for right now.  CA has ​transferred it to a level 2 tech to look at.  Any input would be great at this point.  Thanks everyone.

[brian.fontana]

Hey Ryan.  Regarding the camstat/CAM queues... I know you said previously the only queues with high send/received were related to the AM sector, but is it possible for you to post/share the full camstat output, somewhere within your 10 minute interval for kill CAM?  Might notice more with a second set of eyes...

[Stephen McCormick]

Since this appears to be a CAM issue, I would probably look to clear CAM queues on everything connecting to the DM, Scalability Servers, Agents, and the DM itself. There could be an errant CAM message coming from somewhere. Identifying exactly where could be time consuming so clear them all. You can send a winoffline job to everything directly connected to do it all at once, then do the DM.

 

Steve McCormick

 

Just an update, still have the cam issue.  Setting the schedule task to kill cam every ten minutes allowed all of the servers to report in over...

Clarity Client Automation   Post New Message   Re: Getting CAM errors on Domain Manager Reply to Group Reply to Sender Oct 14, 2019 1:01 PM RYAN Lucas Just an update, still have the cam issue.  Setting the schedule task to kill cam every ten minutes allowed all of the servers to report in over the weekend.  It also allowed users to deploy software to some servers this morning.  I do not want to keep this procedure in place, but it is a workaround for right now.  CA has transferred it to a level 2 tech to look at.  Any input would be great at this point.  Thanks everyone.   Reply to Group Online   View Thread   Recommend   Forward   Flag as Inappropriate   Hello there.  On this specific domain there are close to 8000 servers with about 20 scalabilities.  There are a few agents reporting to the domain, but it's very few.   In order to keep all the agents reporting in, I set up a scheduled task to kill cam.exe every 10 minutes.  So far all is the engines are up to date.  But I definitely don't want to keep it that way, just didn't want things to get stale. Original Message: Sent: 10-12-2019 12:29 AM From: Stephen McCormick Subject: Getting CAM errors on Domain Manager A quick question on architecture if you would indulge me. How many agents are you supporting overall, how many scalabiity servers do you have, and how many agents are connected directly to your Domain?   Steve McCormick   Steve McCormick IT Support Specialist IV Card Services 731-676-4223 Original Message------ The only thing I see with high receives is AMO_Sector_Srv and U-Sector_Srv   You are subscribed to "Clarity Client Automation" as [e-mail]. To change your subscriptions, go to My Subscriptions. To unsubscribe from this community discussion, go to Unsubscribe. Copyright © 2005-2019 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Inc. and/or its subsidiaries. Hosted by Higher Logic, LLC on the behalf of Broadcom - Privacy Policy | Cookie Policy | Supply Chain Transparency | Terms of Use

Original Message------

Just an update, still have the cam issue.  Setting the schedule task to kill cam every ten minutes allowed all of the servers to report in over the weekend.  It also allowed users to deploy software to some servers this morning.  I do not want to keep this procedure in place, but it is a workaround for right now.  CA has ​transferred it to a level 2 tech to look at.  Any input would be great at this point.  Thanks everyone.

[Ryan Lucas]

I did what Steve suggested and ran Win Offline on all of the scalability servers doing a clean of the SS agent.​  I did about a dozen yesterday afternoon, but left my task in place to kill off the cam progress every 15 minutes overnight.  This morning, I cycled everything and disabled my task and things have been running all day.  I am still monitoring things and have a ticket open, but that might have been issue.  I went ahead and ran the win offline on the rest of the servers as well to rule out any other problems.  So far all of the tasks are running and there are jobs being deployed.  Thank you for all of the suggestions; there are so many different things that keep this application running, makes it hard to troubleshoot.