vSphere

Back to discussions

Expand all | Collapse all

Exception thrown by SsoUtil: SSO admin service failure

Jump to Best Answer

1. Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Feb 22, 2016 03:55 PM
I'm diagnosing an issue in an Enhanced Linked Mode environment running 6.0 update 1 (2 vCenters, same domain) where I'm getting the following exception when calling UserSessionService.getUserSession():
[ERROR] data-service-pool-2704 70000426 100010 200005 c.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl There was an issue while extracting the list of system domains com.vmware.vise.vim.security.sso.exception.SsoServiceException: SSO admin service failure
at com.vmware.vise.vim.security.sso.SsoUtil.getAdminService(SsoUtil.java:256)
at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.extractSystemDomains(UserSessionServiceImpl.java:179)
at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.getUserSession(UserSessionServiceImpl.java:156)
at sun.reflect.GeneratedMethodAccessor496.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy503.getUserSession(Unknown Source)
Does anyone have any pointers on diagnosing this? I could not find much information about this error in the SDK docs. Thank you
- Mike
2. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Mar 02, 2016 06:33 PM
*bump*
Has nobody ever seen this in enhanced linked mode?
3. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
ryantodd
Posted Mar 08, 2016 01:00 AM
Seeing this in a couple of 6.0 test systems *not* in linked mode. The issue we have is also intermittent. I.e. you cannot log in for some time, then all of the sudden, login works, then log out, and try to log in again, only for it to fail again. No idea yet what the actual cause is though...
4. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
ryantodd
Posted Mar 10, 2016 12:47 AM
Btw, this seems to be an issue for us only in VCSA 6.0 U1B. VCSA 6.0 U1 seems to be fine in this regard.
5. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
virat1234
Posted Mar 11, 2016 10:15 PM
I am seeing this on the vcenter appliance 6u1. No idea how to fix it though.
6. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
ryantodd
Posted Mar 14, 2016 07:38 PM
Virat1234 -- are you seeing the reference to extractSystemDomains in your stacktrace, or something else entirely?
7. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
virat1234
Posted Mar 21, 2016 11:40 PM
Virat1234 -- are you seeing the reference to extractSystemDomains in your stacktrace, or something else entirely?

That is exactly what I am seeing
8. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
ryantodd
Posted Mar 11, 2016 11:29 PM
After a few days of attempting to work around this issue by making multiple changes to our code, attempting to absolutely minimize any calls to UserSessionService.getUserSession, we've pretty much exhausted everything we can try. At this point I believe that all we can do is recommend to any users of our product plugin to not upgrade to 6.0 U1B.
Can any vmware folks weigh in on this issue, any recommendations?
9. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Diego Barahona
Posted Apr 18, 2016 10:54 AM
Finally, could you solve this issue?
Thanks
10. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted May 27, 2016 04:52 PM
Has anyone had any luck in resolving this issue? Or perhaps any pointers on where to look? I checked that both forward and reverse DNS is configured for all vCenters and the PSC and also checked that all system clocks are synced.
11. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 13, 2016 02:49 PM
Can you, please, post the full stack trace, including the 'Caused By' sections? I wish to see if it's the same issue as in https://communities.vmware.com/message/2602308.
12. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 14, 2016 02:35 PM
Would you like us to create a fixed class file for release 6.0 Update 1 just like we did for release 6.0 Update 2 at "SSO admin service failure" exception in vSphere 6.0.2?
13. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Jun 21, 2016 02:01 PM
[2016-02-17T08:45:41.702-05:00] [ERROR] data-service-pool-283711 70033948 100252 200143 c.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl There was an issue while extracting the list of system domains com.vmware.vise.vim.security.sso.exception.SsoServiceException: SSO admin service failure
at com.vmware.vise.vim.security.sso.SsoUtil.getAdminService(SsoUtil.java:256)
at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.extractSystemDomains(UserSessionServiceImpl.java:179)
at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.getUserSession(UserSessionServiceImpl.java:156)
at sun.reflect.GeneratedMethodAccessor688.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy587.getUserSession(Unknown Source)
[ sensitive packages removed ]
at sun.reflect.GeneratedMethodAccessor1042.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy661.getInfo(Unknown Source)
[ sensitive packages removed (my implementation of property provider adapter) ]
at com.vmware.vise.data.query.impl.DataManager.getDataFromPropertyProvider(DataManager.java:1312)
at com.vmware.vise.data.query.impl.DataManager.getResultFromPropertyProvider(DataManager.java:1283)
at com.vmware.vise.data.query.impl.DataManager.access$000(DataManager.java:67)
at com.vmware.vise.data.query.impl.DataManager$1.run(DataManager.java:858)
at com.vmware.vise.util.concurrent.ExecutorUtil$2.run(ExecutorUtil.java:187)
at com.vmware.vise.util.concurrent.ExecutorUtil$ThreadContextPropagatingRunnable.run(ExecutorUtil.java:584)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: com.vmware.vim.binding.vmodl.RuntimeFault: Unable to dispatch request: Failed to create session
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at com.vmware.vise.util.concurrent.client.ClientMonitorImpl.authenticate(ClientMonitorImpl.java:89)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.getClientMonitor(SsoAdminServiceImpl.java:134)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.<init>(SsoAdminServiceImpl.java:110)
at com.vmware.vise.vim.security.sso.SsoUtil.getAdminService(SsoUtil.java:252)
... 68 common frames omitted
Caused by: com.vmware.vim.binding.vmodl.RuntimeFault: Unable to dispatch request: Failed to create session
at sun.reflect.GeneratedConstructorAccessor461.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:383)
at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:173)
at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:26)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:31)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:141)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:102)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:89)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:84)
at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41)
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:112)
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:273)
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:144)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:186)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:115)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:348)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:308)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:182)
at com.sun.proxy.$Proxy602.retrieveServiceContent(Unknown Source)
at com.vmware.vise.vim.security.sso.impl.SsoUtilInternal.getSsoAdminServiceContent(SsoUtilInternal.java:253)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.processLogin(SsoAdminServiceImpl.java:143)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.access$300(SsoAdminServiceImpl.java:57)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl$SolutionUserAuthenticator.authenticate(SsoAdminServiceImpl.java:497)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl$SolutionUserAuthenticator.authenticate(SsoAdminServiceImpl.java:481)
at com.vmware.vise.util.concurrent.client.ClientMonitorImpl$1.call(ClientMonitorImpl.java:209)
at com.vmware.vise.util.concurrent.client.ClientMonitorImpl$1.call(ClientMonitorImpl.java:206)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at com.vmware.vise.util.concurrent.client.ClientMonitorImpl.authenticate(ClientMonitorImpl.java:74)
... 71 common frames omitted

As you can see, the "Caused by" is : Caused by: com.vmware.vim.binding.vmodl.RuntimeFault: Unable to dispatch request: Failed to create session

vesuvius_prime you said in the other thread 'perhaps the users create too many sessions but don't log out of them explicitly afterwards. And it takes a while for the sessions to expire by themselves. So, probably sometimes you (or your customer) hit the limit and you get the error.'

Do you know if it's possible for an extension to flood the PSC with too many sessions? I thought an extension can only get the current active session from the VIM API, not create new sessions. Just trying to rule out if this is our extension, the customer, or a VMware issue.

Thanks again for your responsivness, you've been a big help.
14. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 21, 2016 04:25 PM
Hmmm, I checked the source. There's a problem with method UserSessionServiceImpl.extractSystemDomains() which is invoked by method UserSessionServiceImpl.getUserSession() and is in the stack trace that you posted. In short, an authentication to the SSO Admin Service is performed but after the work is done, there's no invocation of logout(), so the authenticated session remains. These sessions will eventually expire, but if people do lots of invocations of UserSessionServiceImpl.getUserSession() before the previous authenticated sessions to the SSO Admin Service can expire, then these session will pile up and will exceed the limit and then the errors will start to occur.
I can prepare yet another hack for you. I can fix method UserSessionServiceImpl.extractSystemDomains() and I can invoke logout() explicitly and this will likely solve your problem, but I don't know if you (or your customer) will be okay with applying such patches. Also, I need to say the following:
1. Before you put this fix in production, please, try it out on some non-production environment, just in case. I will test it too, but I don't have an extensive set of plug-ins with which to test, so my testing will be somewhat limited. Furthermore, this is a hack, not an official patch, so I cannot get our QAs deeply involved in all this. You have a real-life environment which can be invaluable in testing this. For example, you can let your plug-in invoke UserSessionService.getUserSession() several hundred times and you can see if you'll hit the problem.
2. Please, either you or your customer should log an SR which will probably force an early hot patch for this issue. I'm not in control of releases, especially patches for past releases, but as I was already told, releasing a hot patch requires an SR from a customer. Of course, I suppose that an SR doesn't automatically mean that a hot patch will be released. The SR will be evaluated, different approaches/fixes/workarounds will be discussed and a decision will be made. But the SR is an important first step.
15. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Jun 21, 2016 05:40 PM
Nice analysis. Few questions:
1. What is "work" when you say "an authentication to the SSO Admin Service is performed but after the work is done, there's no invocation of logout()" - do you mean logout() is supposed to be called after the extension is finished with the session that is returned to it?

2. Will an authenticated session to the admin service show up when you browse active vCenter sessions? Or is this something internal? ("These sessions will eventually expire...)

3. Do you know the expiration time when a session is created with the admin service?

4. Would calling UserSessionService.getUserSession() in a loop be a good attempt at reproducing this?

I'll be more than glad to test this in house and let you know how it goes.
16. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 21, 2016 07:27 PM
Answers:

1. What is "work" when you say "an authentication to the SSO Admin Service is performed but after the work is done, there's no invocation of logout()" - do you mean logout() is supposed to be called after the extension is finished with the session that is returned to it?

I mean that method UserSessionServiceImpl.extractSystemDomains() should invoke logout() when it's done. This should be transparent to you.

By the way, the code in the upcoming release 6.5 is quite different and this is not an issue there. The actual problem appeared in release 6.0 Patch 2 and is also present in Update 1 and Update 2.

2. Will an authenticated session to the admin service show up when you browse active vCenter sessions? Or is this something internal? ("These sessions will eventually expire...)

No. These are not vCenter sessions, but sessions to the SSO Admin Service.

3. Do you know the expiration time when a session is created with the admin service?

I guess it's 30 minutes. But I sent an email to some of the SSO guys asking them to tell me what the expiration timeout is.

4. Would calling UserSessionService.getUserSession() in a loop be a good attempt at reproducing this?

Yes, it should be good. Please, first try doing the loop without a patch. If it ends in error, then great -- we have reproducibility. Then, after applying the patch, try the loop again. If it works without errors, great again -- the problem is fixed. I will post a patch very soon.
17. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 21, 2016 07:54 PM
Okay, attached the patch. I assumed that your vSphere Web Client build number is 3271482 -- at least that's what we determined in Re: "SSO admin service failure" exception in vSphere 6.0.2.
18. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Jun 21, 2016 08:01 PM
Thanks for the quick turnaround. The vCenter we're using for development is running build 2559277 (Version 6.0.0 Build 2559277). Can you make a patch for this build or does this build not contain the change that introduced this issue?

Let me know. If necessary I'll upgrade to Update 1 (or Update 2, whichever corresponds to 3271482) and try flooding the vCenter with getSession() calls.
19. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 21, 2016 08:10 PM
Build 2559277 is vanilla 6.0 release. It doesn't have the issue. The problem starts with Patch 02 which your customer has.
20. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Jul 15, 2016 03:27 PM
Hey Vesuvius,
You said a while back "Build 2559277 is vanilla 6.0 release. It doesn't have the issue. The problem starts with Patch 02 which your customer has."

By patch 02 do you mean update 2? I ask because I couldn't find it on the Build Number History page VMware vCenter Release and Build Number History | Virten.net.‌ But looks like some posters here reported hitting this on update 1. Thanks again
21. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jul 15, 2016 03:35 PM
By Patch02 I mean the second patch for the vanilla 6.0 release. Update 2 is a bigger thing -- it's not just a patch but includes lots of upgrades. In post 50 above I have included fixes for both 6.0 Patch 02 and 6.0 Update 2. I have not included a fix for Update 1 but, yes, it's also affected by this problem. Would you like me to create a fix for Update 1?
22. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Jul 15, 2016 03:38 PM
Nope, I'm all set at the moment. We're just trying to figure out which release is 'safe' to downgrade as a last resort for our customers who are blocked.
23. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jul 15, 2016 03:45 PM
Well, with regard to the "Failed to create session" problem, which is fixed by the changes to class UserSessionServiceImpl in post 50, the vanilla 6.0 release should be safe. But if you're getting class loading issues, which are fixed by the "class_loading_patch_...." archives in post 50, I'm afraid they might occur even in the vanilla 6.0 release. If you wish to avoid class loading problems in your plug-in without requiring the clients to do any patching, you might add the Require-Bundle clauses to your plug-in's MANIFEST.MF file. We discussed these clauses earlier. When you apply those changes, then your class loader will be able to "see" the necessary classes and there shouldn't be class loading problems. I only suggest this as a way to avoid patching your clients' environments. I know it's not a real 'solution', but a workaround, but it might be preferable in your case.
24. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Jun 21, 2016 08:06 PM
Actually, let me update our vCenter to what the customer is running. Do you know which update 3271482 corresponds to? (i.e. 1B or 1A) I'll just upgrade to that. Probably best I have close to the same environment as possible.
25. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 21, 2016 08:11 PM
Build 3271482 corresponds to release 6.0 Patch 02.
26. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Jun 24, 2016 05:15 PM
Hey,
I've been trying to reproduce for about a day now. I'm calling UserSessionService.getSession() in a loop every second. I also tried every half a second. Yesterday I specified no delay and just called getSession() in an infinite loop roughly 10,000 times in 10 minutes. I'll continue with this approach this afternoon but if you have other ideas I'm all ears. The vCenter I'm using now is running update 2, which I hope still has this issue because that's all my IT department has for me :smileysad:
(Version 6.0.0 Build 2997665)
Thanks again
27. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 24, 2016 06:40 PM
You mean, you are trying to reproduce the problem without having applied the patch and you can't reproduce it? In other words, the same exception as before doesn't appear in the log? That's strange, because I'm reproducing it consistently exactly with a loop (a straight loop, without limiting it to once per second or half-second). I can see the exact same error in the log. However, when I apply the patch, the error no longer appears in the log. I deployed build 3271482 (release 6.0 Patch 02) and I'm testing on it, but you should get the same result in release 6.0 Update 2 because the code is the same there.
Can you, please, describe the original problem? Were you not able to get the UserSession session at all? Or were you getting it but you were also seeing the exception in the log? I examined the code, and I can see that when the following method gets executed:
at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.extractSystemDomains(UserSessionServiceImpl.java:179)
(as per the stack trace that you provided), the exception is caught and in this case an empty list of System Domains is returned, but this doesn't prevent the UserSession from being returned to the caller -- its 'systemDomains' field will be empty, but the rest should be okay. However, without the patch, the method certainly does leave authenticated sessions behind and this must be fixed because eventually the SSO server reaches the maximum number of sessions and it refuses to authenticate any further, which may cause problems in completely unrelated scenarios because the SSO server gets contacted in other places too and they may fail because of the exhaustion of authenticated sessions.
So, again, can you describe the exact problem you're experiencing? What are the symptoms?
28. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Jun 24, 2016 06:47 PM
The original problem was that UserSessionService.getUserSession() throws an SsoServiceException. This prevented a user session from being returned to the caller (as you can see in the stack trace).
To reproduce I have created a Java service where UserSessionService is passed in via OSGI (same as you do in the sample apps). In the Flex client I have a toggle button which starts off an infinite loop. In the loop I call a method on my Java service which then calls UserSessionService.getUserSession(). If I get an exception I log it and throw it back to the Flex client. I've run this loop about 20,000 times and still have not hit an exception when calling UserSessionService.getUserSession(). The only difference I can see is that my vCenter is running 6.0 update 2 and our customer is running 6.0 patch 2.

Let me know if I misunderstood anything.

EDIT: Correction - calling UserSessionService.getUserSession()

Mike
29. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 24, 2016 07:35 PM
Okay, I'm looking at the code of method UserSessionService.getUserSession() and it basically does this:
try {

   ...

} catch (Exception e) {

   _logger.error("Failed to get the user session", e);

   return null;

}

Furthermore, method UserSessionServiceImpl.extractSystemDomains(), which is part of the stack trace you quoted earlier, does:
try {

   ...

} catch (Exception e) {

   _logger.error("There was an issue while extracting the list of system domains", e);

}

Neither of these can throw an exception that can propagate to the caller. (They can throw an Error or a plain Throwable, but we're not dealing with such here.) What you see in the log is just a message that there was an error, but in reality you will either get a null UserSession, or a UserSession in which field 'systemDomains' is empty.
By the way, I reproduced the problem by directly placing a loop around the try-catch block in method extractSystemDomains() -- the block which I quoted above. I ran a loop of 1,000 iterations but the actual error started to appear after much fewer iterations. I didn't bother to record how many, but maybe after 200-300 iterations. Don't quote me on that, though -- I might be wrong.
I'm quite surprised you can't reproduce the problem by doing remote invocations of UserSessionService.getUserSession(). And I'm even more surprised that you manage to get SsoServiceException thrown out of method getUserSession(). In fact, it's a checked exception and the method doesn't have any throws clause, so this makes the matter even more puzzling.
30. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Jun 24, 2016 08:12 PM
Aha! So it looks like we call getUserSession() which causes an exception to be thrown, is caught by UserSessionServiceImpl which then returns null. I just checked the logs and I'm seeing the exception. Gonna try your patch now.
[2016-06-24T16:00:45.107-04:00] [ERROR] http-bio-9443-exec-5 c.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl There was an issue while extracting the list of system domains com.vmware.vise.vim.security.sso.exception.SsoServiceException: SSO admin service failure
at com.vmware.vise.vim.security.sso.SsoUtil.getAdminService(SsoUtil.java:256)
at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.extractSystemDomains(UserSessionServiceImpl.java:176)
at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.getUserSession(UserSessionServiceImpl.java:153)
at sun.reflect.GeneratedMethodAccessor495.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy550.getUserSession(Unknown Source)
at com.xxx.callGetSession(YYYImpl.java:183)
31. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Jun 24, 2016 11:59 PM
Ok. Good news: I reproduced the issue
Bad news: Even with the patch you supplied for UserSessionServiceImpl and the patch you supplied on the other thread ("SSO admin service failure" exception in vSphere 6.0.2) for SSOUtilInternal I'm still getting failures in the logs after hammering UserSessionService.getUserSession(). I'm now seeing the following error:
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: interface com.vmware.vim.binding.sso.SessionManager is not visible from class loader
Do you have any ideas? To be clear, your patch seems to have resolved the last error I was seeing - which is great! If this is a new issue I can go and start a new discussion and mark your reply as the answer for this discussion.
Here's the stack trace I'm seeing now:
[2016-06-24T19:47:30.329-04:00] [ERROR] http-bio-9443-exec-4 c.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl There was an issue while extracting the list of system domains com.vmware.vise.vim.security.sso.exception.SsoServiceException: SSO admin service failure
at com.vmware.vise.vim.security.sso.SsoUtil.getAdminService(SsoUtil.java:256)
at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.extractSystemDomains(UserSessionServiceImpl.java:180)
at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.getUserSession(UserSessionServiceImpl.java:156)
at sun.reflect.GeneratedMethodAccessor299.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy524.getUserSession(Unknown Source)
[Our Extension Package] .callGetSession(MyService.java:183)
at sun.reflect.GeneratedMethodAccessor301.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy527.callGetSession(Unknown Source)
at sun.reflect.GeneratedMethodAccessor300.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at flex.messaging.services.remoting.adapters.JavaAdapter.invoke(JavaAdapter.java:418)
at com.vmware.vise.messaging.remoting.JavaAdapterEx.invoke(JavaAdapterEx.java:50)
at flex.messaging.services.RemotingService.serviceMessage(RemotingService.java:183)
at flex.messaging.MessageBroker.routeMessageToService(MessageBroker.java:1400)
at flex.messaging.endpoints.AbstractEndpoint.serviceMessage(AbstractEndpoint.java:1011)
at flex.messaging.endpoints.AbstractEndpoint$$FastClassByCGLIB$$1a3ef066.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.flex.core.MessageInterceptionAdvice.invoke(MessageInterceptionAdvice.java:66)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.adapter.ThrowsAdviceInterceptor.invoke(ThrowsAdviceInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$FixedChainStaticTargetInterceptor.intercept(Cglib2AopProxy.java:573)
at com.vmware.vise.messaging.endpoints.AMFEndpoint$$EnhancerByCGLIB$$c774f9ac.serviceMessage(<generated>)
at flex.messaging.endpoints.amf.MessageBrokerFilter.invoke(MessageBrokerFilter.java:103)
at flex.messaging.endpoints.amf.LegacyFilter.invoke(LegacyFilter.java:158)
at flex.messaging.endpoints.amf.SessionFilter.invoke(SessionFilter.java:44)
at flex.messaging.endpoints.amf.BatchProcessFilter.invoke(BatchProcessFilter.java:67)
at flex.messaging.endpoints.amf.SerializationFilter.invoke(SerializationFilter.java:166)
at flex.messaging.endpoints.BaseHTTPEndpoint.service(BaseHTTPEndpoint.java:291)
at com.vmware.vise.messaging.endpoints.AMFEndpoint$$EnhancerByCGLIB$$c774f9ac.service(<generated>)
at org.springframework.flex.servlet.MessageBrokerHandlerAdapter.handle(MessageBrokerHandlerAdapter.java:109)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.eclipse.virgo.web.tomcat.support.ApplicationNameTrackingValve.invoke(ApplicationNameTrackingValve.java:33)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: interface com.vmware.vim.binding.sso.SessionManager is not visible from class loader
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at com.vmware.vise.util.concurrent.client.ClientMonitorImpl.authenticate(ClientMonitorImpl.java:89)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.getClientMonitor(SsoAdminServiceImpl.java:134)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.<init>(SsoAdminServiceImpl.java:110)
at com.vmware.vise.vim.security.sso.SsoUtil.getAdminService(SsoUtil.java:252)
... 89 common frames omitted
Caused by: java.lang.IllegalArgumentException: interface com.vmware.vim.binding.sso.SessionManager is not visible from class loader
at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:616)
at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:592)
at java.lang.reflect.WeakCache$Factory.get(WeakCache.java:244)
at java.lang.reflect.WeakCache.get(WeakCache.java:141)
at java.lang.reflect.Proxy.getProxyClass0(Proxy.java:455)
at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:738)
at com.vmware.vim.vmomi.client.common.impl.ManagedObjectFactoryImpl.createManagedObject(ManagedObjectFactoryImpl.java:88)
at com.vmware.vim.vmomi.client.http.impl.HttpClient.createStub(HttpClient.java:76)
at com.vmware.vise.vim.commons.vmomi.ClientWithSharedHttpConfig.createStub(ClientWithSharedHttpConfig.java:66)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.processLogin(SsoAdminServiceImpl.java:145)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.access$300(SsoAdminServiceImpl.java:57)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl$SolutionUserAuthenticator.authenticate(SsoAdminServiceImpl.java:497)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl$SolutionUserAuthenticator.authenticate(SsoAdminServiceImpl.java:481)
at com.vmware.vise.util.concurrent.client.ClientMonitorImpl$1.call(ClientMonitorImpl.java:209)
at com.vmware.vise.util.concurrent.client.ClientMonitorImpl$1.call(ClientMonitorImpl.java:206)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at com.vmware.vise.util.concurrent.client.ClientMonitorImpl.authenticate(ClientMonitorImpl.java:74)
... 92 common frames omitted
32. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 27, 2016 05:29 PM
Apparently something goes wrong with the thread's context class loaders in your environment. I don't get such things here. If it's okay by you, I'd like to do the following:
I will prepare two sets of .jar files for you:
1. Set #1 will just contain additional logging without any fix.
2. Set #2 will contain a fix.
You'll have to apply Set #1 before you apply the fix in Set #2. Set #1 will log the class loader which is being used when the error occurs. That is, when trying to load com.vmware.vim.binding.sso.SessionManager. I will prefix the special log messages with a triple tilde (~~~). Please, when the error occurs, locate these messages in the log and post them here (or send me a private message, if you prefer). Then, apply the fix from Set #2 and check if the problem will be fixed.
Does this sound okay by you? Should I proceed with creating the sets?
33. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Jun 27, 2016 06:03 PM
That's fine by me. Go ahead and create the sets of jars. Also, let me know if you'd like me to enable debug logging prior to restarting the web client service, or if INFO is good enough.
34. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 27, 2016 07:00 PM
Okay, attached the sets. Set #1 contains only one jar, actually. You should copy the jars to directory [VIRGO_HOME]/repository/usr, then you should set the owner of the .jars to be user 'vsphere-client' and you should make the .jars readable by all users (e.g. access mode 'rwxr--r--'), just in case. Then you should restart the Web Client.
The logging will contain a stack trace. It will look scary, but it's not an error, just logging. Please, locate the actual class loading error and then see the preceding message that contains "~~~" (without the quotes) -- this message should say which class loader is being used.
35. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Jun 27, 2016 07:21 PM
Is this the message you're looking for?
[2016-06-27T15:18:17.868-04:00] [INFO ] data-service-pool-148 70000107 100004 200001 com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl ~~~ The thread context class loader is org.eclipse.gemini.web.tomcat.internal.loading.BundleWebappClassLoader@22d10bf6 (com.vmware.vsphere.client.endpointsapp_6.0.0 [240]) java.lang.Exception: DummyException
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.logClassLoader(SsoAdminServiceImpl.java:574)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.processLogin(SsoAdminServiceImpl.java:146)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.access$300(SsoAdminServiceImpl.java:58)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl$SolutionUserAuthenticator.authenticate(SsoAdminServiceImpl.java:512)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl$SolutionUserAuthenticator.authenticate(SsoAdminServiceImpl.java:496)
at com.vmware.vise.util.concurrent.client.ClientMonitorImpl$1.call(ClientMonitorImpl.java:209)
at com.vmware.vise.util.concurrent.client.ClientMonitorImpl$1.call(ClientMonitorImpl.java:206)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at com.vmware.vise.util.concurrent.client.ClientMonitorImpl.authenticate(ClientMonitorImpl.java:74)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.getClientMonitor(SsoAdminServiceImpl.java:135)
at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.<init>(SsoAdminServiceImpl.java:111)
at com.vmware.vise.vim.security.sso.SsoUtil.getAdminService(SsoUtil.java:252)
at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.extractSystemDomains(UserSessionServiceImpl.java:180)
at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.getUserSession(UserSessionServiceImpl.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy499.getUserSession(Unknown Source)
at com.xxxxxxxxxxx
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy503.getAllPolicies(Unknown Source)
at com.simplivity.backuppolicy.BackupPolicyDataAdapter.getAllModelObjects(BackupPolicyDataAdapter.java:252)
at com.simplivity.common.data.BaseDataProviderAdapter.processQuery(BaseDataProviderAdapter.java:143)
at com.simplivity.common.data.BaseDataProviderAdapter.getData(BaseDataProviderAdapter.java:106)
at com.vmware.vise.data.query.impl.DataAdapterUtil.getDataFromAdapter(DataAdapterUtil.java:205)
at com.vmware.vise.data.query.impl.DataAdapterUtil.getResultFromAdapter(DataAdapterUtil.java:166)
at com.vmware.vise.data.query.impl.DataAdapterUtil.access$000(DataAdapterUtil.java:40)
at com.vmware.vise.data.query.impl.DataAdapterUtil$1.call(DataAdapterUtil.java:78)
at com.vmware.vise.data.query.impl.DataAdapterUtil$1.call(DataAdapterUtil.java:75)
at com.vmware.vise.util.concurrent.ExecutorUtil$3.call(ExecutorUtil.java:630)
at com.vmware.vise.util.concurrent.ExecutorUtil$ThreadContextPropagatingCallable.call(ExecutorUtil.java:984)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
36. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 27, 2016 07:51 PM
Yes, this is the type of logging that I added. Is this the message that immediately precedes the class loading error? If it is, things are getting strange -- endpoints-app-6.0.0.war should contain "DynamicImport-Package: *" in its MANIFEST.MF, so it should be able to see all classes. Can you, please, check if your endpoints-app-6.0.0.war contains any dynamic imports in its manifest? Also, please, apply the fix that I attached and see if it will solve the problem.
37. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 27, 2016 07:56 PM
Please, also check if both the diagnostic message (with the three tildes) and the error message are logged by the same thread. The diagnostic message is logged by thread "data-service-pool-148". If the error message is logged by a different thread, please, find the first preceding diagnostic message by the same thread. It will be strange if it turns out that the endpoints-app class loader can't see some classes in some situations.
38. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Jun 27, 2016 08:25 PM
Well, I copied the message from the logs before I hit the class loader error (called getUserSession() once to get the logging output that you added)
I see the following in the endpoints-app-6.0.0.war manifest file, which matches what you'd expect:
Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven
Built-By: mts
Build-Jdk: 1.7.0_21
DynamicImport-Package: *
Bundle-Vendor: VMware
Bundle-Version: 6.0.0
Tool: Bundlor 1.0.0.RELEASE
Bundle-Name: endpoints-app
Bundle-ManifestVersion: 2
Bundle-SymbolicName: com.vmware.vsphere.client.endpointsapp
Web-ContextPath: vsphere-client/endpoints
Import-Package: com.vmware.vise.messaging.endpoints,com.vmware.vise.me
ssaging.remoting,com.vmware.vise.messaging.validators,com.vmware.vise
.vim.messaging.context.support,com.vmware.vise.vim.messaging.websocke
t,com.vmware.vise.vim.messaging.websocket.tomcat,flex.messaging.valid
ators,org.eclipse.virgo.web.dm,org.osgi.framework,org.springframework
.aop;version="[3.1.4.RELEASE,3.1.4.RELEASE]",org.springframework.aop.
config,org.springframework.aop.framework,org.springframework.aop.scop
e,org.springframework.beans.factory.config,org.springframework.flex.m
essaging;version="[1.5.2.RELEASE,1.5.2.RELEASE]",org.springframework.
flex.remoting;version="[1.5.2.RELEASE,1.5.2.RELEASE]",org.springframe
work.flex.servlet;version="[1.5.2.RELEASE,1.5.2.RELEASE]",org.springf
ramework.security.config;version="[3.1.4.RELEASE,3.1.4.RELEASE]",org.
springframework.web.context,org.springframework.web.filter,org.spring
framework.web.servlet,org.springframework.web.servlet.handler,org.spr
ingframework.web.servlet.mvc
I tried loading your patched jars but when I start the web client service I'm getting these errors:
[2016-06-27T16:15:29.684-04:00] [ERROR] iLogServiceListener@60745678 org.osgi.service.log.LogService Bundle org.eclipse.equinox.ds_1.4.0.v20120112-1400, [SCR] Unexpected exception occurred! java.lang.IllegalStateException: BundleContext is no longer valid
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.checkValid(BundleContextImpl.java:931)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.getServiceReferences(BundleContextImpl.java:498)
at org.eclipse.equinox.internal.ds.Reference.hasProviders(Reference.java:127)
at org.eclipse.equinox.internal.ds.Resolver.selectNewlyUnsatisfied(Resolver.java:600)
at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:364)
at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:225)
at org.eclipse.equinox.internal.cm.Activator.stop(Activator.java:57)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl$2.run(BundleContextImpl.java:771)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.stop(BundleContextImpl.java:764)
at org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(BundleHost.java:510)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.suspend(AbstractBundle.java:566)
at org.eclipse.osgi.framework.internal.core.Framework.suspendBundle(Framework.java:1206)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.decFWSL(StartLevelManager.java:592)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:257)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.shutdown(StartLevelManager.java:215)
at org.eclipse.osgi.framework.internal.core.InternalSystemBundle.suspend(InternalSystemBundle.java:284)
at org.eclipse.osgi.framework.internal.core.Framework.shutdown(Framework.java:692)
at org.eclipse.osgi.framework.internal.core.Framework.close(Framework.java:600)
at org.eclipse.osgi.framework.internal.core.InternalSystemBundle$1.run(InternalSystemBundle.java:261)
at java.lang.Thread.run(Thread.java:745)
Any idea what might be going on? I'm running the patched class in security-service from this thread (that you provided initially) , plus the patched vim-services and vim-commons-vsphere that you provided in this latest zip. Let me know if you'd like me to revert all jars back and simply replace vim-commons-vsphere and vim-services per your latest comments.
39. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 27, 2016 08:40 PM
The error you're getting is usually preceded by some other error(s) which were deemed fatal by the Virgo server, so it decided to shut down. What is the first error that you see in the log after you start up the web client?
The patch in security-service may stay. It shouldn't cause problems.
I tried these jars over here on release 6.0 Patch 02 (build 3271482) and they work. Are you using the same release in your environment? If not, please, tell me your build number and I will create jars for that build. However, it might take me a couple of hours because I have something else I need to do first.
40. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
mikemayors
Posted Jun 27, 2016 08:52 PM
I'm running web client: Version 6.0.0 Build 3617395
Gonna go back and get answers to your question about if the message is logged on the same thread and what messages preceded the context bundle errors.
41. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend
Broadcom Employee

Bisser Peshev
Posted Jun 27, 2016 08:55 PM
Okay, so you're using release 6.0 Update 2. I will build jars for it but, please, give me a few hours.

42. RE: Exception thrown by SsoUtil: SSO admin service failure

Recommend

mikemayors

Posted Jun 27, 2016 09:32 PM

Looks like both the error and diagnostic message are logged by http-bio-9443-exec-6. It also looks like the diagnostic message immediately precedes the class loading error. (I did a search for SsoServiceException. This is the first occurrence; it immediately follows your diagnostic message)

[2016-06-27T17:11:29.525-04:00] [INFO ] http-bio-9443-exec-6 70000120 100009 200001 com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl ~~~ The thread context class loader is org.eclipse.gemini.web.tomcat.internal.loading.BundleWebappClassLoader@23444e80 (com.simplivity.federationui_1.1.9 [379]) java.lang.Exception: DummyException

at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.logClassLoader(SsoAdminServiceImpl.java:574)

at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.processLogin(SsoAdminServiceImpl.java:146)

at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.access$300(SsoAdminServiceImpl.java:58)

at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl$SolutionUserAuthenticator.authenticate(SsoAdminServiceImpl.java:512)

at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl$SolutionUserAuthenticator.authenticate(SsoAdminServiceImpl.java:496)

at com.vmware.vise.util.concurrent.client.ClientMonitorImpl$1.call(ClientMonitorImpl.java:209)

at com.vmware.vise.util.concurrent.client.ClientMonitorImpl$1.call(ClientMonitorImpl.java:206)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at com.vmware.vise.util.concurrent.client.ClientMonitorImpl.authenticate(ClientMonitorImpl.java:74)

at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.getClientMonitor(SsoAdminServiceImpl.java:135)

at com.vmware.vise.vim.security.sso.impl.SsoAdminServiceImpl.<init>(SsoAdminServiceImpl.java:111)

at com.vmware.vise.vim.security.sso.SsoUtil.getAdminService(SsoUtil.java:252)

at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.extractSystemDomains(UserSessionServiceImpl.java:180)

at com.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl.getUserSession(UserSessionServiceImpl.java:156)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)

at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

at com.sun.proxy.$Proxy518.getUserSession(Unknown Source)

at com.simplivity.xxxxxxxxxxxxxx