SSO Design Question

[baubau01]

Hi guys,

Our large enterprise needs to upgrade to 5.1 asap. The only thing that holds us back is this SSO piece of the puzzle. We cuuently have multiple vcenters around 20 across the globe in America, Europe, Asia.  Knowing that each Vmware service will be affected by SSO we need to design this properly

We got about 3 options right now not sure which one we are better to go with

Option 1(I personally think we should go with): Each region like America, Europe, Asia will have a single SSO DB and all the Vcenters will be in the same SSO HA. Each SSO in the region will have a SQL cluster behind it. This will be a combination of SSO HA mode and SSO multi-site HA mode.

Option 2: Because the SSO DB is so small and the WAN traffic wont be much to the SQL cluster just have one SSO DB and every Vcenter will be joined to that SSO DB in HA mode. ( In case something does happen with SSO DB all the 20Vcenters will be locked, nobody will be able to log in to them)

Option 3: Each Vcenter will have it`s own local SSO DB and be managed locally. If there are problems with a Vcenter the rest of vcenter wont be affected at all. Problem with this setup ( We want to have 10 vcenters linked to each other, but thats only possible if all Vcenter point to the same SSO DB in the HA mode)

What do you guys think ? How should we go.. please correct me if I got any miss understandning of how all of this works.

[baubau01]

no`one has enough design experience around SSO ?

[Doug Baer]

I think you may have nailed that one :smileyhappy:

At this point, I don't know that there are a lot of people who have deployed SSO in more than the simplest configurations.

I did find a good VMware KB article that explains some things about multi-site SSO -- at least, it helped my understanding.

http://kb.vmware.com/kb/2034074

Doug

[vChamp]

We're in the thick of this design consideration right now and are aiming for adopting Option 1, at least for our data centres.  Small sites are managed locally and will each have their own SSO in a simple vCenter build, at least for now.  The DCs will have two SSOs, each in their own VM, pointing to the same DB instance and accessed via a virtual IP on an F5.  The config will be exported and imported to the 2nd DC which will be configured for high availability with its own local (to site) DB instance and F5 configuration.  The vCenter at each site will be added to a Linked Mode group in order to see/manage both environments from a single Web Client login.

The one downside of this configuration is the absolute dependency on a good network link between vCenter and SSO (although, this is not much different from using an external DBMS).  If you want to use a local account in vCenter now, the account has to be either created within SSO or drawn from the SSO's guest OS.  Without the availability of the F5s, allowing us to configure SSO in high availability mode, we likely would have placed SSO and vCenter onto the same guest OS to mitigate some risk.

In our case, we are deploying the vCenter components (for the resource clusters) in a 2-host management cluster, itself managed by a simple install of vCenter. We haven't decided as yet wether to treat each management vCenter instance independently or collectively.  At this time, I'm leaning towards to former.

One caveat - we settled on this design only last week and will be mocking it up in the lab over the coming 2 weeks. Accordingly, don't consider the above as being a working deployment...yet :smileywink:

[KBaillie08]

Hi Gary, were you able to get this working in the lab? If so which order did you perform the SSO installs? We have 4 SSO servers, 2 at each site. site1-sso1 and site2-sso2 were installed in multisite mode and updated with custom certs, which was successful. When trying to add site1-sso2 to an existing installation using HA and site1-sso1 as the primary I recieve the following error:

error 20010: Failed to Configure LookupService.

The same error is received when trying to add site2-sso2 to an existing installation using HA and site2-sso1 as the primary.

Have worked through the related KBs but does not resolve the problem http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2034069 , http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2036497

Also logged a support case with VMware but they have not been able to resolve it yet.

[vChamp]

I should have updated my last post as we decided to defer multisite/multinode for the next design iteration.  My team and I were not completely satisfied with the requirements surrounding multisite.  In one area VMware notes that the SSO data rarely changes then in another makes reference to a requirement for the vAdmins to setup a daily or weekly synchronization task.  To me, this does not mesh with the concept of infrequent change.

In our case, the 5.1 deployment (pending) also introduces a Management Cluster.  Given that this already adds to the systems the in-service team has to manage, and that some of the multisite sync process requires clarification, we deferred its use.

Multinode was deferred for a different reason.  Multinode requires a load balancer front end to host the "virtual" application instance for SSO.  In our case, our network team only just deployed F5 to support new Exchange deployments so are more or less building their skill set now.  Accordingly, we approached F5 for a "best practice" configuration to support F5 high availability or multinode.  We've followed up and are now approaching 2+ months without a vendor provided configuration.  From my perspective, following a best practice (or at least understanding one) is critical for something this new.  In order then to at least deliver vSphere 5.1 into service, we chose to defer SSO high availability for now.

We intend to go ahead with building SSO at the sites we later intend to link up each as "Primary" nodes in the hope that this will mitigate some of the required reconfiguration.  But I'm not under the illusion there won't be some surprises - we may well need to fully rebuild vCenter components once we are ready for multisite/multinode.

I hope that helps.  I'll try to fire back an update to this thread when we are ready for release.

G.

[Cheride]

Experts Please comment.

I had a case open with VMwareand still waiting for a proper answer.

Is there anywhere Vmware mentioning that SSO can be loadbalanced using a hardware load balancer?

We are looking to provide SSO load balancing using F5. Below is the KB provided to me my VMware support engineer. I'm escalating the case to level1 as the answer I got is not 100%. Will provide more details.

http://kb.vmware.com/kb/2033588

Are there anyone every successful in setting up SSO HA using a hardware loadbalancer? For me it looks like software load balancer is the only supported method as mentioned in above KB.

Experts/Moderators/Bloggers, Please help us out here.

Thanks

Deepu

[Scott Norris]

Is there anywhere Vmware mentioning that SSO can be loadbalanced using a hardware load balancer?

We are looking to provide SSO load balancing using F5. Below is the KB provided to me my VMware support engineer. I'm escalating the case to level1 as the answer I got is not 100%. Will provide more details.

http://kb.vmware.com/kb/2033588

Are there anyone every successful in setting up SSO HA using a hardware loadbalancer? For me it looks like software load balancer is the only supported method as mentioned in above KB.

Experts/Moderators/Bloggers, Please help us out here.

yeah it sure can be load balanced using any load balancer you like, VMware refer to the built in load balancing of apache but there is no reason this cant be performed by an F5. Just need someone the the expertise on f5s to set it up.

The trick is this has to be done at the time of install, you can not load balance after SSO has been installed and vCenters or other services referencing it. as you have to change the ip to the NLB VIP which has to be done before you go attaching vCenters to it.

in my experience, im running a multi site and did look at using NLB but its really very easy to change which SSO server vCenter and inventory service points to. so worst case takes 2 minutes to install a new instance of SSO and another 2 minutes re pointing vCenter etc. decided it wasn't worth the time setting it up as the outage in case of a failure would be very minimal.

Cheers

[WasimShaikh]

Hi,

I am also struggling to configure SSO in HA using a virtual loadbalancer by RiverBed Stingray Traffic Manager.

Test-1:::

I am using 2 nodes SSOA and SSOB.

Installed SSO on SSOA.

Installed SSO on SSOB with open as join to existing HA.

Made changes as per KB 2033588

No errors.

Issues come when configuring services.

Issued a certificate for LB. Installed SSL on LB. so that traffic can be decrypted and rules can be applied.

Created Rules

Once I was able to update the services, after restating SSOA when i check listServices https://ssoha.domain.com:7444/lookupservice/sdk the command fails on checking sso-adminserver/sdk path.

Test-2::::

Installed SSO on SSOA.

generated the certificate for SSOA.domain.com

generated PFX file

generated root-trust.jks and server-identity.jks files.

placed both of them in c:\program files\vmware\infrastructure\ssoserver\security

Updated the certificate using SSOcli configure-ssl command

Tested the certificate by browsing to https://ssoa.domain.com:7444/sso-adminserver/sdk.. certificate shows the updated one.

Tried installing SSO on SSOB..

FAILED to update LookupService Error No: 20010

Now, its getting so much confusing.. at some point LB is not able to forward request properly.. tried many configuration.

Not sure where to find answer to this.!!.

[KBaillie08]

Hi Wasim,

My team is testing a similar deployment within our lab.

You are not doing anything incorrectly, the error you are experiencing "Error 20010: Failed to update lookupservice" is an nice feature of VMware's buggy un-tested software.

We have had a support case open with VMware for 1 month and has finally been escalated to the software engineering team. Our experience with support has been very unsatisfactory (Found it difficult to find anyone in support who was trained in vSphere 5.1 installation). Hopefully an update can be supplied by the software engineering team and we don't have to wait another month :smileyhappy:

I will post in this thread if we receive a solution/workaround from VMware.

My advise is stick with vSphere 5, or if you must proceed with vSphere 5.1 deploy it in simple mode.

If you want to raise your own support call with VMware please feel free to reference SR 13269807601

[Cheride]

Hi,

Thanks for this info. We cannot think of a simple basic mode…we have to have HA as per the plans.

I will update the steps we are following if we see a positive outcome tomorrow.

[WasimShaikh]

Looks like this time its gonna work for me..

Inventory Service installed y pointing the setup to LB virtualServer https://SSOHA.domain.com:7444/lookupservice/sdk

:-)

Let me go through SSL installation process of Inventory now.

[Cheride]

R u in USA? If you don’t mind, Can I have your Personal Email, I might need your help tomorrow.

[WasimShaikh]

I am living in Bahrain, and sure u can have my email.. I hope its allowed to put in this forum, otherwise the mod will kick me out for breaking rules :smileysilly:

[Cheride]

Never mind…lets communicate through this …I hope I will have some update for you tomorrow.

[WasimShaikh]

I sent u a PM with my email.

[Cheride]

Ok.. Here is my update as of today morning.

we had issues with installing the web client and was getting error ( cert mismatch) when trying to point to the URL to SSO VIP.

Per VMware support advice , we created a self-sign cert on F5 and I was able to complete the Web Client installation successfully.

Next step:

Login to web client. When attempting to login we are getting the error “ Failed to connect to VMware lookup service. SSL certificate verification failed.

Waiting for a call from Support Engineer. Updates to follow.

[WasimShaikh]

Have you installed SSL on SSO server?

Is your vcenter server have SSL configured?

[KBaillie08]

If you have created certs as per http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2035011 copy the ca_certificates.crt file to c:\programdata\vmware\ssl folder on the webclient server.

[WasimShaikh]

Got any updates on configuration part?

What are the changes that you made on SSO nodes?

I did complete all the Server setup, and all vCenter services were online.

But the Failover was not working as it should. :-(

[Cheride]

No…we worked all day friday…we are now stuck at ssl certificate validation issue.

We made some progress when we change the .crt file to .pem . we also applied the same file on F5.

[WasimShaikh]

Ok,

Did you try what KBaillie suggested?

Coz thats the solution in most of the cases.
copy the ca_certificates.crt file to c:\programdata\vmware\ssl even if SSO is not having directory \vmware\ssl create it and copy Root64.cer to it and rename it to ca_certificates.crt

[KBaillie08]

Spent some time with support over the weekend on this issue. When failing over to the secondary SSO node the following error was displayed on the web client:

Failed to communicate with the vCenter Single Sign On Server http://\webapps\sso-adminServer\WEB-INF\web.xml

This makes the secondary node the same as the primary, you then have to ensure active passive access to the SSO nodes is configured on the load balancer. This also means if a change is made to the primary the web.xml file needs to be manually copied to the secondary. I haven’t completed testing yet, but you may have to restart the SSO service on the secondary node once failover has occurred to allow authentication.

Another useful insight from engineering was the following:

If the primary SSO node fails and VC is restarted while primary node is down then VC will be unable to authenticate user access, because it relies on the admin service running on the primary SSO node. In this scenario the secondary node is basically useless.

[Cheride]

Here is the work around to address the issue you have mentioned below.

Ff the vCenter server vpxd process is stopped for any reason it wont be able to restart until the primary SSO is available again. This is because the SSO-adminserver which include restarting the vCenter services cannot be started to from the secondary. ( SSO-Admin service is no available in secondary).

A workaround for this issue is as follows,

1. Copy the following from Primary to Secondary SSO server.

\webapps\sso-adminServer\WEB-INF\web.xml

2. Also you would need to redirect SSO admin traffic to the newly promoted Primary on your Load Balancer.

3.Restart the SSO service

You can add this to the secondary now or you can save the file and add it to the secondary node if the primary ever goes down.

This will allow you to restart the vCenter from the secondary node.

[orthohin]

That's right.

[WasimShaikh]

Hey,

Hope to find some solution.

just now I started my 23rd attempted :-))

Situation in my Lab is reallyyyy strange..

check this out..

I did not update STS service with root-trust.jks. (if i do SSOB will not install)

soo, the credentials is same while updating service endpoint..

STS endpoint got updated successfully,

Admin and GroupCheck is giving Invalid Credentials. Error.

5.1 is full of BedBugs. :-))

[Cheride]

I’m fighting along with 2 engineers from Colorado center. Let us hope for the best.

[WasimShaikh]

Cool.

I am able to fix some of the issue I was facing.

After installing SSO on both nodes.

(did not update the root-trust.jks)

1. I updated the STS endpoint..

and other endpoints were not getting updated.. with Return code : invalid credentials - 3

Updated the root-trust.jks file.

again tried to update..

and while updating other endpoint it started giving error Return code : Service Not Responding - 2

If you see the URL in above screenshot..in STS.Properties, I had "?wsdl" entry in the URL.

I tried updating STS service with new .properties file but no use..

I did changed in Database!! :-P

under RSA DB find table "LS_Service_EndPoint" this table contains the URL for all 3 endpoints..

Edited the URL for STS.. saved..

another changes I made was in LB.

the Mapping SHI*

KB says to map /ims to /ims on both nodes..

but I had to map /ims to /ims/STSService

and

/sso-adminserver to /sso-adminserver/sdk

Bingo!!!..

While performing these task, I kept Node 2 disconnected..

So that the LB does not forward traffic to node 2..

Now I need to make Node 2 (SSOB) online and had to figure out how to forward traffic for /sso-adminserver to /sso-adminserver/sdk on Node1 ONLY!.

[WasimShaikh]

Ok,

After successful updates on SSO Service Endpoints on Node1 (SSOA)

I put back Node2 (SSOB) online..

Run SSOlscli listServices https://ssob.domain.com:7444/lookupservice/sdk

Return Code is : OperationFailed.

100

Looking at LB logs, came to know the traffic is going to SSOB.

Changed the Mapping rule for /sso-adminserver to use a specific Pool which contain only 1 node i.e., SSOA.

(dont know if this is the right way)

Now the traffic moved to SSOA.

Now let me try Installing Inventory Service on a server which already have vCenter 5.0 installed.. so it will be kind of upgrade from 5.0 to 5.1.

[KBaillie08]

For anyone who is experiencing "Error 20010: Failed to update lookup service" when installing the secondary SSO node there is a workaround that has been provided by VMware.

Note: this error occured for me when configuring the primary node with custom SSL certs before installing the secondary node (VMware recommends installing the primary and secondary before configuring custom certs)

If anyone from VMware is reading this.. your software should be flexible enough to add SSO nodes as your environment needs it!!

1. Download ssojavalib.zip from here

https://ftpsite.vmware.com:443/download?domain=FTPSITE&id=aca05026b7e014fcf42df94d9b36e874

(If this file no longer exists you will need to contact VMware support)

2. On the secondary SSO node launch the SSO installer and proceed until the Welcome screen.

3. Extract the ssojavalib.zip and place in c:\temp

4. Change directory to %temp%.  Open directories (the directory will have a long ID {DEC5C346-414B-.....}) until you find a sub-dir with the name ssojavalib.

5. Replace the entire ssojavalib folder with the folder that was extracted to c:\temp

6. Continue with the SSO install.

Hope this helps someone out there.

[WasimShaikh]

Hi,

Finally SSO is load balanced!!..

I came to update the post and read your reply.

Error 20010: fuj it man, I did approx 9-10 re-installation coz of that.. and then I figured out better to install both nodes without updating SSL on node1.

Even updating the ssolscli.jar file did not work.

I updated ssolscli.jar on both nodes after installing SSO.

the server.xml file I think was the key to make it happen.. I was doing a silly mistake.. didn't put the jvmRoute="SrvName" i.e., jvmRoute="SSOA"

did this changes on both nodes, Node1 have SSOA and Node2 have SSOB.

after installtion of SSO, updated root-trust.jks on both nodes.

and updated the endpoints to https://ssoa.domain.com:7444/lookupservice/sdk

I also prepared a complete step-by-step and  will upload the article once I do the testing throughly..

But for now, I am able to login to vCenter web client even if Node1 is offline / disconnected..

I can see on LB that traffic for sso-adminserver is not going through.. and yes thats coz the service is available on Primary node only.

And in my lab, i dont need to restart service to make it work after primary node has dropped out.

I will restart the vCenter server while Node1 is offline and will check what happens..

If what engrr said, then there is no point in putting this all together..

[WasimShaikh]

Hey,

If you can ask one more question from my side to tech suppor that will be helpful..
After all bits started working when I check STS Certificate tab (logged in with admin@system-domain)

why there is only 1 Chain, and that too is having Node1 cert with Root Authority as Self Signed?

even through both the Nodes is having updated certs, (root-trust.jks)

[Cheride]

At this point I’m working with Vmware engineers and we are on 4th day…

If we are successful tomorrow, I will definetly update you. Reading your posts, I’m afraid on tomorrow’ outcome. I will show this to my engineers.