ITCM Software Delivery - Throughput & Scaling

[BOB LOMAX]

Hi all,

I've honestly looked through as much as I can on the forums and docops but I'm struggling to understand Software Delivery throughput control and good practice scale/size of my ITCM systems.  I want to patch over 500 machines each month after Microsoft OS security updates are released and want to understand what CA good practice is for optimal/reasonable throughput.

Two data centres hold all the servers to be patched - no client/workstation PCs.
My networks are fast local LAN (10GB), plus fast (dark fibre) WAN to a second data centre.
I have a single DM and two SS currently.  DM and SS in data centre 1 and SS in data centre 2.
Each monthly patch can be just 150MB in size, right up to say 1.5GB.

DM is Win2016 VM with 16GB RAM, 4vCPU, fairly fast standard virtual disks.
SS are both Win2016 VM 8GB RAM, 2vCPU.

500 target servers to patch, optimal patching window is one weekend, with machines currently done over four 'phases', just to break them down and do the machines with less business risk in phase 1, then we can evaluate success, before moving on to servers with greater business risk involved in phases 2,3 and 4.

I just want to be able to work out an efficient model for good patching throughput, balancing the effect on SQL Server, Domain Manager, Scalability Servers and the network (if relevant).  If there are good practice policy settings for Patch Manager and/or Software Delivery then great.  If there is a spreadsheet to help me do "what-if" assessments of throughput then even better.

Any thoughts on considerations, good practice, config values, architecture options would be welcome.

Kind Regards,
Bob.

------------------------------
Never trust a man who when left alone with a tea cosy doesn't try it on.
------------------------------

[Stephen McCormick]

Hi Bob,

 

The first thing to understand here is that the majority of this work is performed by the Scalability Servers, not the DM or SQL Server. Your DM configuration looks more than adequate for the small Agent population you have, and should be good up to about 10,000 - 15,000 Agents. Scalability Servers are generally good for 1000-1500 Agents, although other factors can reduce that number, in particular large package deployments in small deployment windows. However I don't think that applies here, your Agent numbers are small and you did not specify a particularly short deployment window.

 

Some items to consider:

 

• By default, Scalability Servers are configured to send no more than 50 jobs concurrently. Since you have 2 servers, that means 100 concurrent jobs. Based on your description of your network bandwidth, that should not be a problem. You might even consider increasing that number, it's a setting in config policy.
• Concurrent jobs means jobs either transferring the package or installing it, so not all of these jobs would be actively transferring data on the network.
• The most relevant calculation here is, if 100 jobs are active at once, and I need to get 500 done, can I get them all done in the window I have available?
• Basically, how long does it take to do the install on 100 systems (or 2 x my concurrent jobs setting), and based on that, is my window long enough to get all my patching done in my available window?
• The scalability servers will automatically start jobs on the first [concurrent jobs]  Agents which respond, and will start new jobs as each job finishes, so they will keep [concurrent jobs] running until all the jobs are done, or no more Agents are responding.

 

Based on your description of your network, Client Auto configuration and requirements, I would say you should be able to easily achieve your requirements without any adjustments or additional infrastructure. In fact you should be able to easily perform all 500 patch deployments within a 5-6 hour window without any adjustments. Based on your network configuration you should be able to easily boost the number of concurrent jobs without adversely affecting your network, assuming the patching is being performed at otherwise off-peak hours so the network is not otherwise constrained. The Scalability Servers are easily able to manage 100 concurrent jobs each on fast, high bandwidth networks such as yours.

 

If you have more specific questions I'll be happy to discuss them with you.

 

Steve McCormick

 

Hi all, I've honestly looked through as much as I can on the forums and docops but I'm struggling to understand Software Delivery throughput...

Clarity Client Automation   Post New Message   ITCM Software Delivery - Throughput & Scaling Reply to Group Reply to Sender Sep 30, 2019 5:36 AM BOB LOMAX Hi all, I've honestly looked through as much as I can on the forums and docops but I'm struggling to understand Software Delivery throughput control and good practice scale/size of my ITCM systems.  I want to patch over 500 machines each month after Microsoft OS security updates are released and want to understand what CA good practice is for optimal/reasonable throughput. Two data centres hold all the servers to be patched - no client/workstation PCs. My networks are fast local LAN (10GB), plus fast (dark fibre) WAN to a second data centre. I have a single DM and two SS currently.  DM and SS in data centre 1 and SS in data centre 2. Each monthly patch can be just 150MB in size, right up to say 1.5GB. DM is Win2016 VM with 16GB RAM, 4vCPU, fairly fast standard virtual disks. SS are both Win2016 VM 8GB RAM, 2vCPU. 500 target servers to patch, optimal patching window is one weekend, with machines currently done over four 'phases', just to break them down and do the machines with less business risk in phase 1, then we can evaluate success, before moving on to servers with greater business risk involved in phases 2,3 and 4. I just want to be able to work out an efficient model for good patching throughput, balancing the effect on SQL Server, Domain Manager, Scalability Servers and the network (if relevant).  If there are good practice policy settings for Patch Manager and/or Software Delivery then great.  If there is a spreadsheet to help me do "what-if" assessments of throughput then even better. Any thoughts on considerations, good practice, config values, architecture options would be welcome. Kind Regards, Bob. ------------------------------ Never trust a man who when left alone with a tea cosy doesn't try it on. ------------------------------   Reply to Group Online   View Thread   Recommend   Forward   Flag as Inappropriate     You are subscribed to "Clarity Client Automation" as [e-mail]. To change your subscriptions, go to My Subscriptions. To unsubscribe from this community discussion, go to Unsubscribe. Copyright © 2005-2019 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Inc. and/or its subsidiaries. Hosted by Higher Logic, LLC on the behalf of Broadcom - Privacy Policy | Cookie Policy | Supply Chain Transparency | Terms of Use

[BOB LOMAX]

Thanks Steve - I've reviewed the config and considered all your points.
I had not seen the 50 concurrent deliveries value and had in my head a value of 10 - from where I can't remember.
We should be fine from an architecture and throughput point of view.

------------------------------
Never trust a man who when left alone with a tea cosy doesn't try it on.
------------------------------

[Stephen McCormick]

Hi Bob,

The default actually used to be 10, many years ago when 'fast network' meant 10Mb, and for a while after 100Mb became the norm we would have to increase it. Now 1Gb and faster is the norm so it's probably time to increase the default again (development take note!). There are also some cases where it is appropriate to reduce the value. For instance, I recently worked on a cloud-based solution where all the Scalability Servers were in the cloud and all clients were ENC Enabled, internet connected remote sites. In this case we reduced the concurrency to 15 to improve reliability. In short there can be many factors to consider, but again, based on the info you provided, I think you are good to go with plenty of room for growth and still be able to met your needs.

 

On a related note, we used to recommend on-site scalability servers in almost all WAN-connected situations, but now with fiber networks that's frequently not the case anymore.

 

Steve McCormick

 

Thanks Steve - I've reviewed the config and considered all your points. I had not seen the 50 concurrent deliveries value and had in my head a... -posted to the "Clarity Client Automation" community

Clarity Client Automation   Post New Message   Re: ITCM Software Delivery - Throughput & Scaling Reply to Group Reply to Sender Sep 30, 2019 1:13 PM BOB LOMAX Thanks Steve - I've reviewed the config and considered all your points. I had not seen the 50 concurrent deliveries value and had in my head a value of 10 - from where I can't remember. We should be fine from an architecture and throughput point of view. ------------------------------ Never trust a man who when left alone with a tea cosy doesn't try it on. ------------------------------   Reply to Group Online   View Thread   Recommend   Forward   Flag as Inappropriate   Hi Bob,   The first thing to understand here is that the majority of this work is performed by the Scalability Servers, not the DM or SQL Server. Your DM configuration looks more than adequate for the small Agent population you have, and should be good up to about 10,000 - 15,000 Agents. Scalability Servers are generally good for 1000-1500 Agents, although other factors can reduce that number, in particular large package deployments in small deployment windows. However I don't think that applies here, your Agent numbers are small and you did not specify a particularly short deployment window.   Some items to consider:   By default, Scalability Servers are configured to send no more than 50 jobs concurrently. Since you have 2 servers, that means 100 concurrent jobs. Based on your description of your network bandwidth, that should not be a problem. You might even consider increasing that number, it's a setting in config policy. Concurrent jobs means jobs either transferring the package or installing it, so not all of these jobs would be actively transferring data on the network. The most relevant calculation here is, if 100 jobs are active at once, and I need to get 500 done, can I get them all done in the window I have available? Basically, how long does it take to do the install on 100 systems (or 2 x my concurrent jobs setting), and based on that, is my window long enough to get all my patching done in my available window? The scalability servers will automatically start jobs on the first [concurrent jobs]  Agents which respond, and will start new jobs as each job finishes, so they will keep [concurrent jobs] running until all the jobs are done, or no more Agents are responding.   Based on your description of your network, Client Auto configuration and requirements, I would say you should be able to easily achieve your requirements without any adjustments or additional infrastructure. In fact you should be able to easily perform all 500 patch deployments within a 5-6 hour window without any adjustments. Based on your network configuration you should be able to easily boost the number of concurrent jobs without adversely affecting your network, assuming the patching is being performed at otherwise off-peak hours so the network is not otherwise constrained. The Scalability Servers are easily able to manage 100 concurrent jobs each on fast, high bandwidth networks such as yours.   If you have more specific questions I'll be happy to discuss them with you.   Steve McCormick   Hi all, I've honestly looked through as much as I can on the forums and docops but I'm struggling to understand Software Delivery throughput... Clarity Client Automation   Post New Message   ITCM Software Delivery - Throughput & Scaling Reply to Group Reply to Sender Sep 30, 2019 5:36 AM BOB LOMAX Hi all, I've honestly looked through as much as I can on the forums and docops but I'm struggling to understand Software Delivery throughput control and good practice scale/size of my ITCM systems.  I want to patch over 500 machines each month after Microsoft OS security updates are released and want to understand what CA good practice is for optimal/reasonable throughput. Two data centres hold all the servers to be patched - no client/workstation PCs. My networks are fast local LAN (10GB), plus fast (dark fibre) WAN to a second data centre. I have a single DM and two SS currently.  DM and SS in data centre 1 and SS in data centre 2. Each monthly patch can be just 150MB in size, right up to say 1.5GB. DM is Win2016 VM with 16GB RAM, 4vCPU, fairly fast standard virtual disks. SS are both Win2016 VM 8GB RAM, 2vCPU. 500 target servers to patch, optimal patching window is one weekend, with machines currently done over four 'phases', just to break them down and do the machines with less business risk in phase 1, then we can evaluate success, before moving on to servers with greater business risk involved in phases 2,3 and 4. I just want to be able to work out an efficient model for good patching throughput, balancing the effect on SQL Server, Domain Manager, Scalability Servers and the network (if relevant).  If there are good practice policy settings for Patch Manager and/or Software Delivery then great.  If there is a spreadsheet to help me do "what-if" assessments of throughput then even better. Any thoughts on considerations, good practice, config values, architecture options would be welcome. Kind Regards, Bob. ------------------------------ Never trust a man who when left alone with a tea cosy doesn't try it on. ------------------------------   Reply to Group Online   View Thread   Recommend   Forward   Flag as Inappropriate       You are subscribed to "Clarity Client Automation" as [e-mail]. To change your subscriptions, go to My Subscriptions. To unsubscribe from this community discussion, go to Unsubscribe. Copyright © 2005-2019 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Inc. and/or its subsidiaries. Hosted by Higher Logic, LLC on the behalf of Broadcom - Privacy Policy | Cookie Policy | Supply Chain Transparency | Terms of Use Original Message------ Hi all, I've honestly looked through as much as I can on the forums and docops but I'm struggling to understand Software Delivery throughput control and good practice scale/size of my ITCM systems.  I want to patch over 500 machines each month after Microsoft OS security updates are released and want to understand what CA good practice is for optimal/reasonable throughput. Two data centres hold all the servers to be patched - no client/workstation PCs. My networks are fast local LAN (10GB), plus fast (dark fibre) WAN to a second data centre. I have a single DM and two SS currently.  DM and SS in data centre 1 and SS in data centre 2. Each monthly patch can be just 150MB in size, right up to say 1.5GB. DM is Win2016 VM with 16GB RAM, 4vCPU, fairly fast standard virtual disks. SS are both Win2016 VM 8GB RAM, 2vCPU. 500 target servers to patch, optimal patching window is one weekend, with machines currently done over four 'phases', just to break them down and do the machines with less business risk in phase 1, then we can evaluate success, before moving on to servers with greater business risk involved in phases 2,3 and 4. I just want to be able to work out an efficient model for good patching throughput, balancing the effect on SQL Server, Domain Manager, Scalability Servers and the network (if relevant).  If there are good practice policy settings for Patch Manager and/or Software Delivery then great.  If there is a spreadsheet to help me do "what-if" assessments of throughput then even better. Any thoughts on considerations, good practice, config values, architecture options would be welcome. Kind Regards, Bob. ------------------------------ Never trust a man who when left alone with a tea cosy doesn't try it on. ------------------------------   You are subscribed to "Clarity Client Automation" as [e-mail]. To change your subscriptions, go to My Subscriptions. To unsubscribe from this community discussion, go to Unsubscribe. Copyright © 2005-2019 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Inc. and/or its subsidiaries. Hosted by Higher Logic, LLC on the behalf of Broadcom - Privacy Policy | Cookie Policy | Supply Chain Transparency | Terms of Use

Original Message------

Thanks Steve - I've reviewed the config and considered all your points.
I had not seen the 50 concurrent deliveries value and had in my head a value of 10 - from where I can't remember.
We should be fine from an architecture and throughput point of view.

------------------------------
Never trust a man who when left alone with a tea cosy doesn't try it on.
------------------------------

[Kyle R]

I just want to say thanks for the Q&A on this thread.

It's good to learn from a practical scalability scenario.

Thanks, Kyle_R.

[Stephen McCormick]

That's what these forums are for, happy to see people are getting value from it.

 

Steve McCormick

 

I just want to say thanks for the Q&A on this thread. It's good to learn from a practical scalability scenario. Thanks, Kyle_R. -posted to the "Clarity Client Automation" community

Clarity Client Automation   Post New Message   Re: ITCM Software Delivery - Throughput & Scaling Reply to Group Reply to Sender Oct 1, 2019 2:42 AM Kyle R I just want to say thanks for the Q&A on this thread. It's good to learn from a practical scalability scenario. Thanks, Kyle_R.   Reply to Group Online   View Thread   Recommend   Forward   Flag as Inappropriate     You are subscribed to "Clarity Client Automation" as [e-mail]. To change your subscriptions, go to My Subscriptions. To unsubscribe from this community discussion, go to Unsubscribe. Copyright © 2005-2019 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Inc. and/or its subsidiaries. Hosted by Higher Logic, LLC on the behalf of Broadcom - Privacy Policy | Cookie Policy | Supply Chain Transparency | Terms of Use

Original Message------

I just want to say thanks for the Q&A on this thread.

It's good to learn from a practical scalability scenario.

Thanks, Kyle_R.