R12 Administrative UI Login & Linux Entropy

[Vijay Masurkar]

Tuesday Tip by Vikas Tiwari, Senior Support Engineer, for 2-5-13


[Note from Vijay Masurkar: This is the first in a few upcoming tips involving Linux Entropy and SiteMinder. What is entropy: By definition, in data transmission and information theory, a measure of the loss of information in a transmitted signal or message. The Linux kernel generates what is called “entropy” from keyborad timings, mouse movements, and IDE timings, and makes the random character data available to other operating system processes through the special files /dev/randomand /dev/urandom. This capability was introduced in Linux version 1.3.30. ]

Example:

We were running R12.5 policy server on RHEL 6. Installed Administrative UI (AdminUI) on the RHEL 6.0 or RHEL 5.x servers also. Our AdminUI was working fine and we were able to login into AdminUI without any problem. However, over a weekend, our server got rebooted which was hosting the AdminUI; we restored the AdminUI services by starting JBOSS services. But, therafter, we were not able to login into AdminUI using SiteMinder superuser credentials. We were getting the below error:

"Error: username and password do not match"

Solution:

We verified that the username and password combination was actually correct. So, next, we wanted to check the entropy set.

Executed the below command.

# cat /proc/sys/kernel/random/entropy_avail

It was giving the value of less then 200. We set this value to 1000. Also the symbolic link for the entropy got removed somehow. We requested our System admin to restore the symbolic link and set the entropy value to higher value.

Then, restarted the JBOSS and we were able to login into AdminUI again. So, if Server gets rebooted, please make sure the symbolic link is available and entropy is higher. Preferably, check with your system administrator for exact values.

Even after restoring the symbolic link, if you are not able to login to the AdminUI, follow the below steps:

* Stop AdminUIservice.
* Go to /adminui-install-folder/server/default/data
* Take backup data folder and delete the data folder
* Run XPSRegclient command
* Start the AdminUI
* Try to login into AdminUI again

[System]

Thanks for the tip Vijay! Have a great week!

[Anon Anon]

We got the same issue in new built environment. I don't think this is just a inconvenience. This issue should be categorized as defect.

First,
/dev/random is a device for Linux, any sort link will be reset after box restart.

Second,
/dev/urandom is less secure than default /dev/random in theory, it's possible calculating random sequence if entropy is low and /dev/urandom is in place.

Third,
After box restart and /dev/random being resetted, Admin UI could fail to connect Policy Server, BUT there is not error message in either Admin UI or Policy Server LOG indicted the cause is low entropy.

So instead of manually reset /dev/random soft link , it should deploy more permanent method, such as an auto-start script to re-build the softlink after box reboot. Or improve Admin UI code to handle this kind of error.

[banst03]

dwang.1 wrote:

We got the same issue in new built environment. I don't think this is just a inconvenience. This issue should be categorized as defect.

First,
/dev/random is a device for Linux, any sort link will be reset after box restart.

Second,
/dev/urandom is less secure than default /dev/random in theory, it's possible calculating random sequence if entropy is low and /dev/urandom is in place.

Third,
After box restart and /dev/random being resetted, Admin UI could fail to connect Policy Server, BUT there is not error message in either Admin UI or Policy Server LOG indicted the cause is low entropy.

So instead of manually reset /dev/random soft link , it should deploy more permanent method, such as an auto-start script to re-build the softlink after box reboot. Or improve Admin UI code to handle this kind of error.

I thought I would chime in here to give some background and correct some inaccuracies.

The difference between /dev/random and /dev/urandom is that /dev/random blocks until it has enough randomness. /dev/urandom is stream based and will always give you data, (however as dwang.1 alluded to, the data you receive might not be truly random. (As random as you can get.)

The problem here is really not a SiteMinder or WAMUI issue. The problem is that Linux needs more entropy on the Operating System to generate the requisite randomness. As most know, SiteMinder uses encryption algorithms to do all the communication between the agents and the policy server. (the WAMUI can be considered a 4.x agent).

The solution could be what is outlined here about changing /dev/random and /dev/urandom to switch this, but there is the concern that the data might not truly be random. Alternatively, the better approach would be to use an entropy gathering daemon (eg, prngd). Ensuring you have enough entropy on the box will ensure that /dev/random can keep up.

-Steve

[Anon Anon]

Hi Steve,
Thanks for input. But as I pointed out, RedHat Linux WILL reset this softlink from /dev/urandom back to /dev/random. And If Entropy runs low at the same time(like our QA box), Admin UI can NOT connect to SM Policy server without any error message. We have to switch back to old applet UI to complete out task.

I understand this is not 100% siteminder issue, but other applications handle Entropy running low situation just fine. Can SM Admin UI give some error message to let user know what's going on?

Thanks,
Denny

Steven_Bankowitz wrote:

dwang.1 wrote:

We got the same issue in new built environment. I don't think this is just a inconvenience. This issue should be categorized as defect.

First,
/dev/random is a device for Linux, any sort link will be reset after box restart.

Second,
/dev/urandom is less secure than default /dev/random in theory, it's possible calculating random sequence if entropy is low and /dev/urandom is in place.

Third,
After box restart and /dev/random being resetted, Admin UI could fail to connect Policy Server, BUT there is not error message in either Admin UI or Policy Server LOG indicted the cause is low entropy.

So instead of manually reset /dev/random soft link , it should deploy more permanent method, such as an auto-start script to re-build the softlink after box reboot. Or improve Admin UI code to handle this kind of error.

I thought I would chime in here to give some background and correct some inaccuracies.

The difference between /dev/random and /dev/urandom is that /dev/random blocks until it has enough randomness. /dev/urandom is stream based and will always give you data, (however as dwang.1 alluded to, the data you receive might not be truly random. (As random as you can get.)

The problem here is really not a SiteMinder or WAMUI issue. The problem is that Linux needs more entropy on the Operating System to generate the requisite randomness. As most know, SiteMinder uses encryption algorithms to do all the communication between the agents and the policy server. (the WAMUI can be considered a 4.x agent).

The solution could be what is outlined here about changing /dev/random and /dev/urandom to switch this, but there is the concern that the data might not truly be random. Alternatively, the better approach would be to use an entropy gathering daemon (eg, prngd). Ensuring you have enough entropy on the box will ensure that /dev/random can keep up.

-Steve

[banst03]

dwang.1 wrote:

Hi Steve,
Thanks for input. But as I pointed out, RedHat Linux WILL reset this softlink from /dev/urandom back to /dev/random. And If Entropy runs low at the same time(like our QA box), Admin UI can NOT connect to SM Policy server without any error message. We have to switch back to old applet UI to complete out task.

I understand this is not 100% siteminder issue, but other applications handle Entropy running low situation just fine. Can SM Admin UI give some error message to let user know what's going on?

Thanks,
Denny

Hi Denny,

Thanks. Sorry I wasn't clear with my response. My suggestion is that folks running Linux should run an entropy gathering daemon.

But I hear you about throwing out some sort of errors in the WAMUI. I'll take that back to the team. It is an excellent idea.

And I believe Vijay has more for this thread over the next couple of weeks. So let us see what he has to say. :)

-Steve

[Anon Anon]

Hi Steve,

Thanks for your response. We did discuss this with our SA team, and they said because it's a QA box, not much keyboard, or mouse or network activity going on, entropy is almost always low than 200 on that box. I actually opened a case (21256473) for clarification about whether or not we should keep/create soft-link even after installation. And I've also noticed some XPS tools (XPSExport,XPSRegClient) which are also impacted by this issue. If entropy running low, those command would take 4~5 times longer than usually time consumption.Apparently error is appreciated and documents should be enhanced too. As many SiteMinder customs were moving their platform from Solaris/Windows towards RedHat(CentOS), they would bump into this issue sooner or later.

Thanks,
Denny

Steven_Bankowitz wrote:

Hi Denny,

Thanks. Sorry I wasn't clear with my response. My suggestion is that folks running Linux should run an entropy gathering daemon.

But I hear you about throwing out some sort of errors in the WAMUI. I'll take that back to the team. It is an excellent idea.

And I believe Vijay has more for this thread over the next couple of weeks. So let us see what he has to say. :)

-Steve

[Alfred Macedo]

Hi Denny,

Looks like you have a clear understanding of the situation. But I did want to mention a few things for clarification on this topic.

- /proc/sys/kernel/random/entropy_avail is not a setting location, but rather a monitoring file that will report the level available.
- /proc/sys/kernel/random/poolsize is where you would adjust the amount of entropy that can be available.
- You generally do not want to use urandom as it is not FIPS 140-2 compliant, which may or may not be an issue depending on your environment.

The problem tends to be that these systems are headless, meaning no video/keyboard/mouse interactions, to provide the necessary input for entropy generation. So where to get this input from?

To replenish the available entropy, you could use the Random Number Generator Daemon as mentioned. You'll need to install the rng-tools or rng-utils depending on the flavor. Then feed from /urandom. Urandom is theoretically less random and therefore less secure.

Otherwise you could look into using a hardware generator such as a usb "Entropy Key" which can generate substantial amounts of high quality random data. There are other hardware methods as well as some CPU's have built in generators or modules can be added to certain servers.

As Steve mentioned, we will look into how we can enhance this issue through better documentation and/or better logging.

Thanks, David

[Alan_Baugher]

After experiencing this issue myself on my own virtulized images and at a client site with their ESX servers, I have started to recommend for non-FIPS requirements the use of

RNGD w/ redirect from /dev/urandom

or

HAVEGED using "volatile states" from CPU/Clock.




I have some additional notes within a deck under this link:


101859891



If there are FIPS requirements; then there are likely two (2) choices to address the entropy challenge:

1) Use physical servers

2) Acquire SSL accelerator or similar hardware that is supported by Vmware ESX servers and use the RNGD with redirect from this physical device to /dev/random.


Regards,


Alan Baugher

[Anon Anon]

Hi Vikas,

 

I am facing similar issue that you have faced above. But my environment is running on Windows.

I have tried below already.

I have two policy servers which are connected to their individual admin-UI's. Both are connected to the same policy store.

 

 

Stop WAMUI service.

go to /adminui-install-folder/server/default/data

take back up data folder and delete the data folder.

Run XPSRegclient command

Start the WAMUI

try to login to WAMUI again.

But that doesn't seems to work for me.

Do you have any suggestions for the changes that you have suggested above for Windows Environment.

CA has suggested us to use external authentication but I want to avoid doing that.

I have also opened a support case with CA 21957718 which is still in progress.

Thanks,

Satish

[Karmeng]

Hi Satish,

 

The entropy is applied to unix environment. I don't see this related to your Windows environment. It could be due to some other issue but outcome is same (ie:use name/password do not match). Try reset the password via smreg and check if that help. Are you having this issue on two policy servers or just one of them is having the issue?

 

When you run xpsexport -xb, you should be able to see the siteminder password. If the user name and password are match but WAMUI throws the error on user name/password do not match, we (CA) need to look into this from the issue that you submitted.

 

            <Property Name="CA.SM::Admin.Password" Sensitive="Yes">

                <StringValue>password</StringValue>

            </Property>

            <Property Name="CA.SM::Admin.Name">

                <StringValue>siteminder</StringValue>

            </Property>

 

External admin store didn't fix the issue but more like possible workaround.

 

Regards,

Kar Meng

[Anon Anon]

Hi Kar Meng,

 

Thanks for your email. I have tried to reset password several times now. It works only with one admin UI at one time.

My configuration is as below

AdminUI 1 --> PS 1 -- Policy Store A

AdminUI 2 --> PS 2 -- Policy Store A

 

When i try to reset password using smreg tool on PS1. I can only login with Siteminder password in Admin UI 1 and login to admin UI 2 fails and then the same process repeats again.

I have already done reset 5 times.

 

Please help if you have any suggestions.

I have also opened CA case CA 21957718.

 

Thanks,

Satish

[Karmeng]

Hi Satish,

 

Thanks for your update.  Are both of your policy servers use the same encryption key? I suspect the encryption key different between PS1 and PS2 cause the reset password (at PS1) work in PS1 when you perform the reset (as the password encrypt with encryptionkey1) while PS2 (persumely you have encryptionkey2) cannot read the password due to different encryptionkey.

If this is on Windows, you can try to copy over the EncryptionKey.txt(ie: C:\Program Files\CA\siteminder\bin) from PS1 to PS2 (backup the PS2 EncryptionKey.txt before you replace so you can revert back if needed). Restart the server and check if that make any different.

 

Regards,

Kar Meng

[Anon Anon]

Hi Karmeng,

 

Thanks for your email. I have already tried the above steps you mention above.

When i do that. The Policy server SQL-drivers couldnot establish connection to Policy store, keystore, Audit and session from smconsole.

 

Then policy server fails to start afterwards.

Could you please have some suggestions.

 

Thanks,

Satish

[Karmeng]

Hi Satish,

 

You mentioned:

"When i do that. The Policy server SQL-drivers couldnot establish connection to Policy store, keystore, Audit and session from smconsole."

 

It's unclear on the sequence you performed.

Do you mean

1. You have verified everything is working in the PS1 environment (ie: connection to all store success, can login to AdminUI1 but failed to login to AdminUI2)

2. Then you copy EncyptionKey.txt from PS1 (working environment) to PS2 (non-working environment)

3. All store connection return failure from both PS1 and PS2 smconsole. PS1 and PS2 cannot startup.

 

If that's the case, that's not something I expected. As mentioned before try run xpsexport -xb -npass from both environment and check what is the admin password. Sample of the output on administrator:

 

        <Object class="CA.SM::Admin" Xid="CA.SM::Admin@12-49e223d7-886a-431a-ba6b-47066e97bb44" CreatedDateTime="2014-01-23T04:12:09" UpdatedBy="SMSTUB" UpdateMethod="Internal" ExportType="Replace">

            <Property Name="CA.SM::Admin.DirectoryAuth">

                <BooleanValue>false</BooleanValue>

            </Property>

            <Property Name="CA.SM::Admin.Rights">

                <NumberValue>47</NumberValue>

            </Property>

            <Property Name="CA.SM::Admin.Password" Sensitive="Yes">

                <StringValue>password</StringValue>

            </Property>

            <Property Name="CA.SM::Admin.Name">

                <StringValue>siteminder</StringValue>

            </Property>

        </Object><!-- Xid="CA.SM::Admin@12-49e223d7-886a-431a-ba6b-47066e97bb44" -->

 

The idea of troubleshoot is make sure one environment is working fine first. After that, try to compare what is the different between the working and non-working environment.

Since the topic that Vikas provided in this thread is related to Unix entropy causing AdminUI unable to login but you are on Windows environment, therefore, I think best to discuss this issue in separate thread. I will create one thread and copy over the information that we have discussed here.

[Ujwol]

Hi Karmeng , SatishSharma, & HubertDennis

 

This Admin UI login issue raised by Satish  is being discussed in multiple threads.

 

One being the current thread and the other being :

 

Error massge when login to WAMUI username and password not matching

 

Both of these threads was originally created to discuss some different problem , not the one we are discussing at the moment..

As such, to make it simple , I have created a new thread :

 

Legacy super user login issue with the Administrative UI

 

Let us continue the discussion in the new thread ..and let the old threads take some rest

 

Cheers,

Ujwol

[Legacy User]

One point I have noted in many postings related to installing Identity Suite components on Linux platforms is the step to increase the random number generation pool by setting up a symbolic link to urandom to replace the random binary.

 

This technique is repeated frequently in postings on the Single Sign-On (SiteMinder) discussion threads.

These two lines are frequently posted as a quick solution:

> mv /dev/random /dev/random.orig
> ln -s /dev/urandom /dev/random
In many cases the random number generator daemon is not running, so the symbolic link will not have effect until the rngd process is started:
> service rngd start
You should also ensure the rngd daemon starts on reboot:
> chkconfig rngd on

 

The transient nature of the symbolic link to /dev/urandom seems to be the focus of this thread, with many references to implement a more permanent solution. I ran across this posting using a Google search (how to increase entropy in Centos 6.2? - CentOS ) "How to Increase Entropy in CentOS 6.2". The recommended permanent change is to set the rngd options as shown in the article by editing the /etc/sysconfig/rngd options file:
> # Add extra options here
> EXTRAOPTIONS="-r /dev/urandom"
The effect is the same as setting the symbolic link but not additional updates to udev configurations or similar techniques to make the symbolic link persist on reboot.