Layer7 Access Management

Expand all | Collapse all

Entropy for UNIX/Linux & Impact to SSO (SM) / IM / JVM / LDAP / DB (JDBC) Performance

  • 1.  Entropy for UNIX/Linux & Impact to SSO (SM) / IM / JVM / LDAP / DB (JDBC) Performance

    Posted 09-27-2013 03:39 AM
      |   view attached

    Hello All,

     

    *** 2018/10/25  HotFix for the Identity Suite vApp for RNGD.

    High CPU utilization noticed by rngd process. - CA Knowledge 

     

     

    *** 2017/03/13 Edit: -  Updated deck based on recent questions from customers.

    -  Entropy pumps should be added to any servers that uses security libraries, e.g. Directory Server, Database Servers, J2EE servers, and SSO/Siteminder Servers.     Use a quick test to see if your servers need to have the OS entropy pump added.

     

    watch -n 1 cat /proc/sys/kernel/random/entropy_avail

     

    If the return value is less than 1000, then please think about adding an Entropy Pump to all of your server(s).

     

     

    ### Prior Note ###

    Recently I was engaged to determine the root issue of a performance related question for Vmware Linux server and SiteMinder.

    After reviewing the bookshelves, the JVM vendors site, and many google searches, I was able to determine that there is a common thread to performance issues with Vmware Linux and any software solution that uses cryptographic routines.

    I have put together a deck on how to address performance for SiteMinder, IM JCS (IAMCS), Web App Servers (Weblogic/WebSphere/Jboss), and other solutions that may use TLS/SSL or generate certs, that are related to a depletion of the entropy pool (/dev/random) on a virtualized/headless Linux/Unix server.


    Enjoy / YMMV.


    *** 2013/09/29 Edit. After additional research via NIST site, I have re-ordered the alternative recommendations with regards to FIPS.
    Enclosing updated deck


    *** 2013/10/10 Edit. Added a very useful EGD daemon process to the deck. HAVEGED This entropy "pump" will use volatile states of the CPU / Clock from virtualized servers to give them "boost" to speed up startup times for StieMinder/J2EE (Jboss/Weblogic/Websphere).


    *** 2013/11/01 Edit.  Added business high level summary with current challenges about /dev/random.

     

    *** 2016/09/29 - Refresh to bring this back to awareness.  Please don't use a soft link from /dev/urandom to /dev/random.  Why?  OS Software update/upgrade may/will wipe away these settings.   Please use an entropy pump daemon.  Recommend either HAVEGD (clock-cycle version) or the OS RNGD daemons.     Test your performance before and after, to amaze yourself.



  • 2.  RE: Entropy and UNIX/Linux Impact to SM / IM / JVM Performance

    Posted 09-27-2013 01:25 PM
    This is great! Thanks for sharing this with the community Alan!


    alan_baugher wrote:

    Hello All,

    Recently I was engaged to determine the root issue of a performance related question for Vmware Linux server and SiteMinder.

    After reviewing the bookshelves, the JVM vendors site, and many google searches, I was able to determine that there is a common thread to performance issues with Vmware Linux and any software solution that uses cryptographic routines.

    I have put together a deck on how to address performance for SiteMinder, IM JCS (IAMCS), Web App Servers (Weblogic/WebSphere/Jboss), and other solutions that may use TLS/SSL or generate certs, that are related to a depletion of the entropy pool (/dev/random) on a virtualized/headless Linux/Unix server.


    Enjoy / YMMV.


  • 3.  RE: Entropy and UNIX/Linux Impact to SM / IM / JVM Performance

    Posted 09-27-2013 02:09 PM
    Thanks so much for explaining this in relation to siteminder. The CA install documentation says to use /dev/urandom which falls under the test/non-production catagory.


  • 4.  RE: Entropy and UNIX/Linux Impact to SM / IM / JVM Performance

    Posted 09-29-2013 06:00 AM
    Thanks. Appreciate the feedback.


    From my readings of RHEL submissions to NIST for FIPS-140-2 reviews, most dated after Sept 2012;
    it appears that /dev/urandom is being referenced in various security modules as the primary seed mechanism.


    It will take some careful reading to determine if /dev/urandom may be used, as is, for vendors default installations on UNIX/Linux.
    I would still wish to use the RNGD tool set to ensure the population of randomness is large;
    as not to stop production functionality and ensure a high level of confidence that the business has not been exposed.



    *** ***
    NIST References: With select remarks pulled regarding FIPS and /dev/urandom

    Red Hat Enterprise Linux 6.2 OpenSSH Server Cryptographic Module v2.1
    http://csrc.nist.gov/groups/STM/cmvp/documents/140-1/140sp/140sp1792.pdf

    Random Number Generation
    A FIPS 140-2, ANSI X9.31 approved pseudo random number generation mechanism will be used in the module, called from OpenSSL, which is seeded by the kernel.

    The kernel uses /dev/urandom as a source of random numbers for RNG seeds. The Linux kernel initializes this
    pseudo device at system startup. SSH_USE_STRONG_RNG is a positive integer that must be greater or equal than 6 to be honored. That integer value specifies the number of bytes obtained from /dev/random and mixed into the DRNG state via the OpenSSL RNG RAND_add API call. Further state that this variable can be set in /etc/sysconfig/sshd as this file is sourced by the sshd start script.
    Please refer to the Red Hat Enterprise Linux – OpenSSL Module v2.0 FIPS 140-2 Security Policy, Section 6.1,
    “Random Number Generation.”

    Self Tests
    FIPS 140-2 requires that the module perform self tests to ensure the integrity of the module and the correctness
    of the cryptographic functionality at startup. In addition, some functions require continuous verification of
    function, such as the random number generator. All of these tests are listed and described in this section



    Red Hat Enterprise Linux 6.2 Libgcrypt Cryptographic Module v2.1
    http://csrc.nist.gov/groups/STM/cmvp/documents/140-1/140sp/140sp1757.pdf

    6.1 Random Number Generation
    A FIPS 140-2, ANSI X9.31-approved pseudo random number generation mechanism using AES 128 will be
    used in the module.
    The random number generator is keyed and seeded with data that is loaded from the /etc/gcrypt/rngseed if the
    device or symlink to device exists xored with the data from the /dev/urandom device. This allows the system
    administrator to always seed the RNGs from /dev/random if it is required

    The integrity check is performed by the Red Hat Enterprise Linux OpenSSL module utility fipscheck. The version
    is 1.2.0-1.el5, and fipscheck-lib version is 1.2.0-1.el5 HMAC/SHA-256.
    When the module starts, it exercises the power-on self-test including the software integrity test. The software
    integrity test (HMAC-SHA256) constitutes a known answer test for the HMAC-SHA256 algorithm.
    The user space integrity verification is performed as follows:
    The OpenSSH server application links with the library libfipscheck.so which is intended to execute
    fipscheck to verify the integrity of the calling application file using HMAC SHA-256. Upon calling the
    FIPSCHECK_verify() function provided with libfipscheck.so, the fipscheck application is loaded and
    executed, and the following steps are performed.
      OpenSSL as loaded by fipscheck performs the integrity check of the OpenSSL library files using
    HMAC SHA-256.



    SUSE Linux Enterprise Server 11 SP2 - OpenSSL Module v0.9.8j
    http://csrc.nist.gov/groups/STM/cmvp/documents/140-1/140sp/140sp1930.pdf


    Red Hat Enterprise Linux 5 OpenSSH Server Cryptographic Module v1.1
    http://csrc.nist.gov/groups/STM/cmvp/documents/140-1/140sp/140sp1384.pdf



    Red Hat Enterprise Linux 6.2 Openswan Cryptographic Module v2.0
    http://csrc.nist.gov/groups/STM/cmvp/documents/140-1/140sp/140sp1859.pdf


    Additional reading:

    Very informative article from Solaris Security Engineer, Darren Moffat
    Solaris Random Number Generation By darrenm on Sep 12, 2013
    https://blogs.oracle.com/darren/entry/solaris_random_number_generation



    Regards

    Alan Baugher


  • 5.  RE: Entropy and UNIX/Linux Impact to SM / IM / JVM Performance

    Posted 10-12-2013 12:07 AM
    I have reorganized my High Level summary (shown below) & provided a link for deploying EGD process of HAVEGED



    *** OS vendors have configured OS packages (OpenSSL/SSHD/Libcrypt) to default to /dev/urandom for FIPS-140-2 and non-FIPS certification.

    *** Most non-OS vendor cryptography software is hard coded or default configured to use /dev/random on Linux/UNIX OS

    *** /dev/random is passive and a “blocking” device driver

    *** Do not destroy the trust relationship between /dev/random and /dev/urandom with softlinks  [OS will likely rebuild them away; and you will be back to original challenge]

    *** Use a “pump” to push in entropy in to /dev/random    [which will “feed” /dev/urandom]

    *** Pick an acceptable source for the “pump” to maintain FIPS-140-2 certification.  [Acceptable sources are hardware inducing “entropy”]

    *** Monitor start-up times before and after using a “pump”.



    Get the performance expected from the virtualized environment.




    https://www.digitalocean.com/community/articles/how-to-setup-additional-entropy-for-cloud-servers-using-haveged


  • 6.  Re: Entropy and UNIX/Linux Impact to SM / IM / JVM Performance

    Posted 09-11-2015 04:23 PM

    Hello,

     

    I thought I would share a tip how to perform a quick check of entropy on a Linux/UNIX OS, to determine if an entropy pump is needed or warranted.

    Without looking at numbers.  ( watch -n 1 cat /proc/sys/kernel/random/entropy_avail   OR     lsof  | grep  -E "/dev/[u]{0,1}random"    )

     

    I found a note about generating a password using the dd command on Linux using /dev/urandom.   

    I decided to see what would happen if I added a time command and switched from /dev/urandom (non-blocking device driver) to  /dev/random (blocking device driver)

     

     

     

    Below are the results:

     

    [root@sandbox01 ~]# time dd if=/dev/urandom bs=8 count=1 2> /dev/null | base64

    Mujajvgul9w=

     

    real 0m0.002s

    user    0m0.003s

    sys     0m0.000s

     

     

    Very quick response.  However, for /dev/random; it took over 24 seconds. Imagine that impact on every JVM or SSO or other encryption routines, as the pool becomes depleted. 

     

    [root@sandbox01 ~]# time dd if=/dev/random bs=8 count=1 2> /dev/null | base64

    EQFX4ouHUfXfpvbi

     

    real 0m24.411s

    user    0m0.003s

    sys     0m0.001s

     

     

    I am pleased to view that CA SecureCenter with Docker "flies" with /dev/random

     

    core@sandbox-docker-master-001 ~ $ time dd if=/dev/random bs=8 count=1 2> /dev/null | base64

    MMdqnls8jAW0UFAbBs86cw==

     

    real 0m0.001s

    user    0m0.000s

    sys     0m0.001s

    core@sandbox-docker-master-001 ~ $

     

     

     

    It is interesting that if this command is run several times in a row (with /dev/random), on a new Linux/UNIX host (with no entropy pump) the first time is slow, then a rapid response may occur, then as the entropy pool is depleted, a lengthy response is displayed.

     

     

     

     

    [root@sandbox01 ~]# time dd if=/dev/random bs=8 count=1 2> /dev/null | base64

    U2g1vQJH

     

    real    0m15.024s

    user    0m0.001s

    sys     0m0.002s

    [root@sandbox01 ~]# time dd if=/dev/random bs=8 count=1 2> /dev/null | base64

    JqXm+dKe

     

    real    1m24.325s

    user    0m0.003s

    sys     0m0.001s

    [root@sandbox01 ~]# time dd if=/dev/random bs=8 count=1 2> /dev/null | base64

    gn1R3Ode

     

    real    0m35.181s

    user    0m0.000s

    sys     0m0.003s

    [root@sandbox01 ~]#



  • 7.  Re: Entropy and UNIX/Linux Impact to SM / IM / JVM Performance

    Posted 09-17-2015 03:01 PM

    A small revision in testing for the need for Entropy Pump on your Linux/Unix OS servers (regardless of solution)

     

     

    Open two (2) putty sessions to the OS.

     

    Execute a "watch" process in one window

    watch -n 1 cat /proc/sys/kernel/random/entropy_avail

     

    Execute the "build random password" process in the 2nd window; using  /dev/random

    time dd if=/dev/random bs=8 count=1 2> /dev/null | base64



    Monitor the entropy available during execution of the dd process. 

    If the value is <200 and stays less than 200, then there is a need to add an entropy pump service/daemon.


    Execute the dd command several times. (10 x)

    If the dd command completes in less than 1 second (or fraction of a second) AND the entropy value > 200, there is NO need to add in an entropy pump.




    Cheers,


    A.



  • 8.  Re: Entropy for UNIX/Linux & Impact to SSO (SM) / IM / JVM / LDAP / DB (JDBC) Performance

    Posted 03-14-2017 03:28 AM

    Hi,

     

    Just run the following deamon that way, and you should be set for the entropy on the system. Everywhere you need encryption and decyption, you'll see performances increased :

     

    # rngd -r /dev/urandom

     

    and make it start at boot time !

     

    This deamon makes the random number pool wider and as such, programs take less time to find unpredictable number.

     

    Best Regards,

    Patrick



  • 9.  Re: Entropy for UNIX/Linux & Impact to SSO (SM) / IM / JVM / LDAP / DB (JDBC) Performance

    Posted 03-16-2017 10:20 PM

    Hi Patrick,

     

    Thanks for the feedback and notes.    

     

    What tools / process have you been using to validate and recommend using a EGD?

     

    While I use the watch (to monitor available entropy); I have found that lsof is valuable as well, to "show" what process require entropy.

     

    As a testing process, I am adding openssl s_client to my process as well.

     

    Did you get a chance to test/try out HAVEGED?   

     

     

    Cheers,

    A.



  • 10.  Re: Entropy for UNIX/Linux & Impact to SSO (SM) / IM / JVM / LDAP / DB (JDBC) Performance

    Posted 10-24-2018 05:18 AM

    usually by using the command :

    # watch -n1 cat /proc/sys/kernel/random/entropy_avail



  • 11.  Re: Entropy for UNIX/Linux & Impact to SSO (SM) / IM / JVM / LDAP / DB (JDBC) Performance

    Posted 07-05-2018 10:16 AM

    Excellent work!!

    Can someone please share the steps to increase the entropy on Solaris.

    SunOS 5.10 Generic_148888-03 sun4v sparc SUNW,Sun-Fire-T200

     

    Thanks,

    Pankaj 



  • 12.  Re: Entropy for UNIX/Linux & Impact to SSO (SM) / IM / JVM / LDAP / DB (JDBC) Performance

    Posted 07-05-2018 12:20 PM

    Pankaj,

     

    For the older version of Sparc Solaris, you may have to lean on using an external 3rd party installation package.

     

    There appears to be little reference on Oracle Docs about "RNGD" or "EGD"

    If you are working with Weblogic, you may wish to use the JVM option to speed its startup:

    -Djava.security.egd=file:/dev/./urandom

     

     

     

    In the latest release of Solaris 11.3, there are newer features, but likely will not help you with your version of the OS.

    Enhancements for Developers - What's New in Oracle® Solaris 11.3 

     

    Random Number and Entropy Gathering System Calls

    Oracle Solaris 11.3 includes two new system calls, getentropy(2) and getrandom(2), which are provided for gathering entropy or random bits from the kernel. These system calls are a better choice than using open(2) and read(2) on /dev/random and /dev/urandom devices.   For more information, see the getentropy (2) and getrandom (2) man pages.

     

     

    Oracle has a document to address this issue, but you will need an Oracle support ID to access it.

    https://support.oracle.com/epmos/faces/DocumentDisplay?id=1378733.1 

    Ref:  Why does my Weblogic Server takes a Long Time to Start? | Oracle Luz Mestre's Blog 

     

     

     

    Otherwise, you may have to install or build an RNGD/EGD service, example below for your older OS.

     

     

    Freeware List for SPARC and Solaris 10 

    egd 0.8 | EGD is an Entropy Gathering Daemon and is a substitute for /dev/random

     

     

     

    Cheers,

     

    Alan



  • 13.  Re: Entropy for UNIX/Linux & Impact to SSO (SM) / IM / JVM / LDAP / DB (JDBC) Performance

    Posted 07-05-2018 04:48 PM

    Thanks, Alan.

    I spent some time on the internet to find exact commands to increase entropy but didn't find something crisp. I will go through Oracle docs as you mentioned.

    Fingers crossed    

     

    Regards,

    Pankaj        



  • 14.  Re: Entropy for UNIX/Linux & Impact to SSO (SM) / IM / JVM / LDAP / DB (JDBC) Performance

    Posted 10-23-2018 05:08 PM

    Team,

     

    In case you notice  rngd daemon running a bit hot on your virtual appliance.

     

     

     

    https://comm.support.ca.com/kb/high-cpu-utilization-noticed-by-rngd-process/kb000115167

     

     

    High CPU utilization noticed by rngd process.

    Document ID : KB000115167
    Last Modified Date : 17/09/2018
    Hide Technical Document Details

    Products
    • CA Governance, Risk &Compliance Manager
    Components
    • CA IDENTITY SUITE (VIRTUAL APPLIANCE):IDSVA
    Question:
    Noticed high CPU utilization on all VMs in the environment due to rngd process. After stopping all Identity Manager related process, it still continues to be 100% CPU utilization. What could be the cause of this?
    Answer:
    Please open support case requesting for hotfix HF-DE376557-20180722-0001.tgz.gpg as this will resolve the issue.