CA PAM migration from on premise to AWS

[ebrahim abebe]

Hello gurus,

Do you have any materials I can use to do a migration of CA PAM server on premise to AWS?

possible solutions I thought are the below:

1- perhaps create a fail over cluster using the machine on premise and an EC2 on AWS and then fail over to AWS EC2. 
2- Some how replicating the on premise CA PAM to a CA PAM Server on AWS (EC2 server)

Can you send me any links in relation to this?

Much appreciated!!!

 

------------------------------
V/R
------------------------------

[Joseph Fry]

Option 1 above is the ideal method.  Build PAM in AWS, join it to the cluster as a secondary site and allow to replicate, then make that site your primary site.

The steps are here:
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-privileged-access-management/privileged-access-manager/3-4/deploying/set-up-a-cluster/cluster-synchronization-promotion-and-recovery.html#concept.dita_f58a3e782c2b3c5b616dff2be1c3b751b73c276b_SitePromotionUsingReplicationAnalysis

You can skip the replication analysis part (steps 2&3) and simply select the AWS site in step 4.

[ebrahim abebe]

Thanks a lot sir!

I will do exactly that!!

[ebrahim abebe]

Question for you:

So building PAM on AWS will cost me for a new licensing and everything or
will the license for the original server should remain valid for the building of the PAM on the new server in AWS? 

[Joseph Fry]

The customer should talk to their account manager.

Essentially PAM has two licences that customers pay for:
1. PAM server license (AWS, OVA, Hardware)
2. managed device licenses (for each device accessed via pam, or that PAM manages a credential on)

If they intend to run more PAM nodes/servers, then they would need additional PAM server licenses. If the intention is simply to move from on-prem to AWS, then their account manager may be able to change the license type for their PAM server(s), at which point they would get a new license for their AWS node and decommission the on-prem node as soon as the migration is complete.

[ebrahim abebe]

I really appreciate your time and inputs. 

So my company has physical appliances to migrate to AWS and clustering does not allow between Hardware appliances and AWS AMIs.

" The AWS AMI instance can only be clustered with other AWS AMI instances."
    https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-privileged-access-management/privileged-access-manager/3-0-1/deploying.html

Any thoughts on this?

I am thinking to have PAM installed on an AWS EC2 (if not prohibited)  instance and then have the physical and the EC2 in  one cluster and then fail over to the EC2.

If not "an VMware OVA VM instances and the hardware appliance can be in the same cluster. " So I can convert the VMware OVA to AWS AMI and create an EC instance using the AMI. But again if the pam cannot be run on an EC2, the AMI may not work as well. So can you confirm if PAM can be installed on an EC2 instance (EC2 are just like any machine except they are on AWS cloud).

V/R

[Joseph Fry]

I had never seen that in the documentation before. I am 99.9% positive that we support mixing AWS, Azure, and on-prem PAM into a cluster, so long as they are each on separate sites (primary+secondary).  For latency reasons you couldn't use them in the same site.

I suspect that the intent was to say that AMI instances can only be in the same cluster site with other AMI instances.  I will try to confirm this and have the documentation corrected.

I have heard of people using the OVA to deploy on an EC2 instance... however I don't believe that is supported.  If you want to deploy in AWS, the only supported method I am aware of is AMI.

[Joseph Fry]

I have confirmed with coworkers that the statement you highlighted in the documentation is false.  We have requested that it be corrected.

You can absolutely mix all PAM server types in a multi-site cluster.  We have some very large installations using a mix of on-prem and cloud sites in their cluster.  Just keep the AWS instances in their own site(s).

The only difficulty you may have is if you use Hardware Security Modules in your physical appliances.  If you do, you may be able to use CloudHSM, though I would have to research if we support that or not.

[ebrahim abebe]

Oh that is a wonderful news!!

One more questions I have for you is that we can only have one physical appliance on one site (on premise) and then the second site (on AWS) can also have only one virtual instance (the EC2 created using the AMI you guys provide). Just making sure that the multi site does not require that we have a multi master set up in our on premise environment. 

Thanks in advance!

[Joseph Fry]

Many customers run 1x1 clusters (one in primary site, one in secondary site)... this configuration is what I would consider the optimal PAM arrangement for a small environment/user load where cost is more important than high availability.

[ebrahim abebe]

Hello Joseph,

So the plan is to migrate from DHS on premise environment to the AWS environment. So for that purpose, what we need is just have just one machine on premise on which CA PM is deployed (existing deployment) and then another one EC2 instance on which CA PAM is deployed (to be created). And then create a cluster between this on premise server (physical appliance) to the EC2 instance based CA PAM. Then do a fail over from the on premise to the AWS EC2 and then after we can have additional EC2 based CA PAM servers to create a cluster between those in the AWS environment.

[Joseph Fry]

Yes... exactly as you describe.  We would typically call it "promoting a site" rather than "fail over"... and that is what you will want to search for in our documentation.

[ebrahim abebe]

Hey buddy

So actually I am now able to create the AWS EC2 instance using the AMI you guys provide. And the version ofthe AMI there is 3.3.0 
Our existing cluster is already version 3.3.2 . Now when I try to upgrade the EC2 to 3.3.2 from 3.3.0 , I was able to upload the patch file. When I try to apply the patch file to upgrade the PAM from 3.3.0  TO 3.3.2 ,  it complains that it does not have enough disk space. The minimum required is 4 GB from your documentation. [1]

Now when I created the CA PAM in AWS EC2, I provided 250 GB . But when I look at the system information from the UI in the CA PAM, it only shows that the available storage is only 7 GB and the free space is less than 2 GB.

Why would that be the case? Do I need to connect to the machine and make changes in the OS ? or is there any configuration in the AMI that is causing the issue?


[1] https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-privileged-access-management/privileged-access-manager/3-3/upgrading/upgrade-to-release-3-3/upgrade-prerequisites-for-3-3.html

[Joseph Fry]

Sorry I didn't respond to this before you posted your other questions.  I have been out of office.

Please confirm that you didn't just add a 250GB second disk.  You need to expand the primary disk.

Once you have allocated disk space to the primary disk, PAM should expand to fill it.  If it doesn't, you may be able to just add a bit more space (a couple GB)  to trigger the expand.

Hope that helps.

[ebrahim abebe]

Hello Joseph,
Look at the screen shots I attached first I think.

So I am trying to add a new site from the currently existing on premise CA PAM cluster. The new site to add is in AWS and when I select AWS it asks for an AWS Provision and when I click on the drop down there, it does not show me anything. 

I then headed to the 3rd party section in the configuration and tried to configure an AWS connections and click on add, it asks for an alias. I am not sure what this is. Do you guys have any step by step document to set up an AWS connection for the CA PAM? 

I have attached two screen shots.

Vey Respectfully,

[Joseph Fry]

The documents for this are here: https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/privileged-access-manager/3-4-1/deploying/deploy-on-an-aws-amazon-machine-image-ami/configure-ca-privileged-access-manager-for-aws.html#concept.dita_4a2c9a972a5f4db514156703762816d39e8f8f41_ConfigureanAWSConnectiontotheInstanceRegion

Pretty sure the Alias is just a unique name you assign to this connection, the docs say to use "cademo" but I suspect that that was supposed to be an example, not an instruction.

[ebrahim abebe]

Hey Joseph,

So I saw the link you gave actually. And the solution that is meant is if I set up a new CA PAM in AWS using the AMI and then how I may access resources in my account. 

The issue I have is different. For my case, I am trying to add a new site to my existing on premise CA PAM server. And the new site I want to add is in AWS. Now for me when I try to add the new site (from the existing on premise cluster), it gives me 3 options and I selected the option AWS. (look at the screen shot)

But then it wants a dropdown and there is nothing there. How do I get that option working basically?

Is there a way we can do a quick call perhaps? like for no more than 5 minutes? My number is 2066311277

V/R

[Ralf Prigl]

Hi Ebrahim, Please consult our online documentation. Page https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/privileged-access-manager/3-2/deploying/set-up-a-cluster/cluster-deployment-requirements.html#concept.dita_3f969d9f216aa91b5ab1227313f1087951d7d004_AWS%2520AMI%2520Cluster%2520Requirements includes the following statement:

"To set up clusters on other AWS sites, AWS connections have to be configured in 

Privileged Access Manager

. To configure the AWS connections, see AWS Coordination."

You have to configure an AWS connection before you can add an AWS cluster site.

[ebrahim abebe]

Ralf,

My question is how to add a new site in an existing on premise cluster. You are sending me an answer for setting up a cluster in AWS. I have seen that material many times.

So here is my quetion:

1- I have an existing cluster on premise (two physical instances that are clustered)
2- Now I want to add a new site for my existing on premise cluster 
3- The new site I am going to add has its instances in AWS. And hence to be able to add a new site in my existing cluster, it is asking me to provide AWS credential details. But there is no way to provide that option since it is giving a drop down and the drop down has nothing.

I have attached a screenshot so you understand which step I am stuck:

I do appreciate your help. But then I am sorry to say this that I asked other questions before and for my other threads you sent me wrong documentation links. This is the third time. So please try to understand the questions I am asking before you just send me a link. 

[Ralf Prigl]

Hi Ebrahim, The dropdown has nothing because you don't have the AWS connection configured yet. So please do that following the online documentation, and then you will see your configured AWS connection in the drop-down list.

[ebrahim abebe]

I tried to set up the AWS connection details in the credential manager and the drop down there is also not giving me "AWS Access Credential Accounts" option. 

So I did 

• Select Credentials, Manage Targets, Accounts. The Target Accounts page opens.
• Select Add.
• HERE WHERE I AM HAVING ISSUE.  I am not getting the option to add "AWS Access Credential Accounts". There is no such option. 

If it is okay can we talk on the phone? My number is 206 631 1277 . May be I am misunderstanding you or you are misunderstanding me and a 3 minutes call perhaps may fix the issue.

Thanks

[Ralf Prigl]

From page https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/privileged-access-manager/3-4-1/implementing/configuring-your-server/aws-coordination.html#concept.dita_586af93cbc0b1a2a8dbbd1b5fc02cb4e23c0a8ff_AWScredentialsPrep:

Store AWS Account Credentials in 

Privileged Access Manager

So that 

Privileged Access Manager

 can coordinate with AWS, first store your AWS account credentials in a target account record in 

Privileged Access Manager

 Credential Manager.

Follow these steps:

• Select 

  Credentials, Manage Targets

  , 

  Accounts

  . The Target Accounts page opens.
• Select 

  Add

  .
• Begin typing AWS in the 

  Application Name

   field, and select 

  AWS Access Credential Accounts

   from the drop-down list. Alternatively, select the magnifying glass icon to open a modal window to select this application. The 

  Host Name

   and 

  Device Name

   are populated with the AWS-specific names.

Are you saying that you do not have a target application with name "AWS Access Credential Accounts"? In that case your license may be missing the AWS option and you would have to get a new license.

[ebrahim abebe]

That is exactly my issue. 

So I do not see the option for a target application with name "AWS Access Credential Accounts"

I see a different option that says "xceedium.aws.amazon.com" . But this is not same as "AWS Access Credential Accounts" . 

Let me know if this has anything to do.

If not let me know so that I inform the DHS officials to add the license. 

Thanks a lot!

[Joseph Fry]

Use that one.  I believe the target application name is "AWS Access Credential Accounts", but the device name/address is  "xceedium.aws.amazon.com"

[ebrahim abebe]

This fixed my issue of adding a new account! I truly appreciate your help.

Thanks a lot!

[ebrahim abebe]

Also if you have a video that shows the steps of adding the AWS credentials, I would really appreciate that.

FYI:

see screen shot

[Joseph Fry]

Just want to point out... your typing the full Application Name into the Host Name box.  Follow the directions, type only AWS into the Application Name box and select the "AWS Access Credential Accounts" target application when it appears.

[ebrahim abebe]

Hello Joseph,

So I did manage to create the account and also add the third party section. Nonetheless I could not get the test connection to work. So our on premise network is connected to the AWS environment using VPN. And I know that "The Privileged Access Manager instance connects to the AWS Public API endpoint over port 443." [1] 

[1] https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/privileged-access-manager/3-2/deploying/deploy-on-an-aws-amazon-machine-image-ami/configure-ca-privileged-access-manager-for-aws.html#concept.dita_c3420fdfd99e00e0cbda332f500ddcd980fe7dc0_ConfigureanAWSConnectiontotheInstanceRegion

I have attached screen shots of the error message and the step where I am having issues.

Any thoughts on this?

V/R

[Joseph Fry]

Ebrahim,

This could be several things:

1. There is a known issue with GovCloud in PAM 3.0 - 3.3.2
   1. I am not sure that it would cause the issue your seeing, but I know it prevents synchronizing key pairs, and accessing the govcloud console.
   2. Issues in this case were related to it not actually using the govcloud urls, so based on your error msg, I suspect its ok.
   3. Open a ticket with support to be provided a hotfix (I believe its 3.3.2.04), or better yet, upgrade to 3.3.4 (lots of improvements in 3.3.3 and 3.3.4).
2. The account your accessing does not have the permissions necessary to use the DescribeInstances API.
   1. Don't really need this unless you want PAM to create a device entry for every active instance.  If you don't just be sure to uncheck the "active" checkbox on the connection.
   2. I would do a port scan test on the tools page and see if you can hit the AWS instance on port 443, you may be able to ignore the error.
3. PAM is not following the correct network path to ec2.us-gov-west-1.amazonaws.com
   1. Many federal customers don't allow direct access to the internet over 443... instead you have to use a proxy.  PAM does not currently support proxies for outbound connections, so you will need to request an exception from your firewall team.
   2. You say it must use a vpn?  Perhaps you need to add routes to PAM to direct AWS traffic out a different gateway?

Anyway I think your configuration in PAM is correct.  Take a look at things from the network perspective (and shoot for an upgrade, they are usually worth doing).

[ebrahim abebe]

Hey Joseph,

Thanks for your reply.

1- The version I have is 3.3.2.99 (So I am assuming this is fine since it is above 3.3.2.04 you mentioned. Or you meant it has to be only 3.3.2.04 ? )

2- a- The AWS account has all the privileges. This one I am 100% sure.
    b- I did use the tools section to scan the port for xceedium.aws.amazon.com 443 . It is showing non-existent domain basically.

3- I am suspecting this third point is the issue we are having. 
    a- Can you confirm that we can have a new CA PAM site cluster added to our on premise cluster? I am afraid that this may not even be supported.          

QUESTION I NEED ANSWER:
1- Our on premise network is connected to AWS via VPN
2- I have two physical CA PAM instances that are clustered on premise
3- I want to add a new site for this cluster.
4- The new site will be having its instances hosted in AWS GovCloud
  Is it POSSIBLE to have this set up?

Unfortunately it is not possible "to add routes to PAM to direct AWS traffic out a different gateway". It must be via private network. As you know the DHS network does not allow that.

[Joseph Fry]

Ebrahim,

3.3.2.99 is the PAM build/version.  3.3.2.04 is a hotfix (fourth hotfix for PAM 3.3.2), I know it's confusing, but the hotfixes​ generally do not increment the PAM build/version number.  The only way to know if you have the hotfix installed already is to check you config>upgrade screen.  I suspect you do not.

xceedium.aws.amazon.com is a dummy url... it doesn't resolve to anything, A target application just needs to have a hostname/address and that's what the developers chose.  You need to test to see if your on-prem PAM can reach port 443 of the AWS PAM instance.

You can certainly have an AWS site added to your cluster; it is 100% supported and has been done at multiple customers.

I think you are misunderstanding how AWS and your network are connected... while it may technically resemble a VPN; from inside the network its transparent.  AWS instances have valid routable IP addresses on the enterprise network (10.x.x.x).  However there may be firewalls that will need to be adjusted to allow PAM to reach the AWS instances.

NOTE: PAM uses HTTPS to the AWS public IPs for all console, api, etc access.  This means that while PAM can reach your AWS instances without going to the internet, any activity that interacts with AWS itself will require an exception to your firewall/proxy rules.  In this case, when you hit the "test" button, you are trying to hit an AWS API.  I know for a fact that this can be done on your network, but will require a FRCR or two to allow your on-prem instances the ability to hit the AWS IP's without going through the proxy.

[ebrahim abebe]

You are a hero sir!

I actually do understand that the on premise CA PAM servers do communicate to the CA PAMs in AWS via private IP. And I did use the on premise CA PAMs tools to scan the ports (443) to see if there is a communication between on premise and AWS CA PAMs. That works fine. 

What was confusing me was why the alias in the 3rd party section for AWS (from configuration page in CA PAM) was not working. And I thought the connection test there has to be successful for the site adding to work. As long as the alias for AWS 3rd party section in CA PAM does not necessarily test ok for the adding of sites, I will just go ahead and add the new site and enter the private IP of my AWS CA PAMs for the new site and see how that goes. My fear was that because the test on the alias connection is not working, I thought the adding of the new site will not work. 

Let me know if I am not misunderstanding you. 

Again thanks a lot !

Very Respectfully,

[Joseph Fry]

I have not tested adding an AWS site without having working access to the AWS APIs (HTTPS open to AWS public IPs) so I cannot be certain that it will work.  All of the cluster traffic is going to use the private network IP's, and won't touch the public internet, so it should cluster fine.

The thing that makes me unsure is that we have an option at all to choose between site types (on-prem, AWS, or Azure), which would suggest that PAM may behave differently for different site types.

I guess the only way to be sure is to give it a shot.

Or submit the FRCR to get PAM direct access to the AWS public IP's (have them use the AWS address group that is already in place for HQ)

[ebrahim abebe]

Thanks again,

so 
1- I will give it a shot and will you know how it goes.
2- I am not sure if I understood the part that says "Or submit the FRCR to get PAM direct access to the AWS public IP's (have them use the AWS address group that is already in place for HQ)". Can you elaborate ?

[Joseph Fry]

FRCR = Firewall Rule Change Request.  

If you want PAM to be able to access AWS API's or Console, you need to have them make a firewall change to allow your On-Prem PAM to be able to access the AWS public IP's directly without going through a proxy.

Your network admins already have a firewall address group for the AWS public IP's

[ebrahim abebe]

Oh I see what you mean. We do not want and need to have the CA PAM hit the public API. Actually the AWS and our on premise network is connected privately and all the communications between our servers and the servers in AWS is made privately. And for access to the aws account we have via the internet there are servers from where we have access to the internet using a proxy.

But thanks I got your points. 

And I will let you know what happens after I add the new site and the members of the new site. 

Cheers

[Joseph Fry]

You will need to open PAM up to use the AWS API's if you want to:

1. Have AWS instances added/removed automatically (great for using PAM to access systems that scale on-demand)
2. Use PAM to manage access to the AWS console
3. Use PAM to manage AWS IAM user accounts
4. Use the PAM AWS API Gateway, to allow PAM to provide credentials transparently for AWS API calls.

PAM has some very nice features around AWS, definitely worth the effort to get things opened up.

Joseph H. Fry
Public Sector Services Professional  | Privileged Access Management

Symantec Enterprise Division, a Broadcom Company

mobile: [phone]
[e-mail]   | broadcom.com        

[ebrahim abebe]

Hello Joseph,

So I tried to add a new site and it required me to have a VIP so I could not proceed. I will have the network team provide a VIP and after I get all the firewall rules set, I will try again and let you know.

Thanks in advance!

[Joseph Fry]

The VIP is arbitrary.  You can put any IP in there, even if it's not routable within your network.

Just drop in a 192.168.123.123 or something you know will not conflict with anything on your network.

The ONLY thing the VIP is used for is for client access, and its use is optional.  Of course this may change someday... but it has always worked in the past.

[Joseph Fry]

You understood perfectly.  Let me know how it goes.

[ebrahim abebe]

Hello Joseph,

I will test it and let you know. I hope it will not disrupt the already existing cluster. 

[ebrahim abebe]

I do appreciate your help sir!

So I used  "xceedium.aws.amazon.com" and that fixed my issue of creating the account!


Thanks a gain!