Symantec Access Management

  • Private Community
 View Only

  • 1.  Failed to initialize message bus

    Posted Jul 25, 2014 10:33 AM

    I have an old solaris 9 server which has three webserver's running , 1 apache and 2 iplanet instances installed under 2 seperate installations.

     

    These webserver were running with Version 5 webagents , In order to migrate  application to R12 infrastructure i installed and configured R12 agent with the iplanet servers , i ran into a problem where if i start the second webserver i always get "failed to initialize message bus" .

     

     

    Things i have checked and tried so far :-

     

    1. Checked that server path is unique for each iplanet installation.

    2.  Checked the inode value for server path is unique.

    3. Checked that system had high limits for semaphores and shared memory.

    4. Set agent wait time to 50 seconds.

    5. Changed the order of webserver startup.

    6. Tried to start the LLAWP process itself, which works fine.

     

    I haven't been able to make any progress so far with the startup, any tips will be helpful .

     

    Error on server startup

    **************************************

    Sun ONE Web Server 6.1 starting (/usr/local)

    https-admserv: Sun ONE Web Server 6.1SP5 B08/17/2005 22:09

    info: CORE3016: daemon is running as super-user

    info: CORE5076: Using [Java HotSpot(TM) Server VM, Version 1.4.2_04] from [Sun Microsystems Inc.]

    info: WEB0100: Loading web module in virtual server [vs-admin] at [/admin-app]

    info: HTTP3072: [LS ls1] http://xxxxxxxxxxx ready to accept requests

    startup: server started successfully

    Success

    https-xxxxxxxx.com Sun ONE Web Server 6.1SP5 B08/17/2005 22:09

    warning: wl-init reports: missing parameter 'Debug', will use default

    warning: wl-init reports: missing parameter 'WLLogFile', will use default

    warning: wl-init reports: missing parameter 'WLTempDir', will use default

    info: wl-init reports: Debug is disabled [(null)]

    info: wl-init reports: Oracle WebLogic plugin build date/time: Dec 16 2008 06:57:49. Change Number: 1169063

    warning: CORE3283: stderr: [19/Jul/2014:00:27:58] [Information] SiteMinder Agent

    warning: CORE3283: stderr:         SiteMinder agent is enabled.

    warning: CORE3283: stderr: [19/Jul/2014:00:27:58] [Information] SiteMinder Agent

    warning: CORE3283: stderr:         Configuration file path:

    warning: CORE3283: stderr: '/usr/local/iplanet/******/https-xxxxxxx/config/WebAgent.conf'.

    warning: CORE3283: stderr: [19/Jul/2014:00:27:58] [Warning] SiteMinder Agent

    warning: CORE3283: stderr:         Siteminder Web Agent not having write permissions on host configuration file. Shared secret roll-over may not be supported.

    warning: CORE3283: stderr:         Permission denied. Please assign write priviledges to the user nobody for the file /usr/local/CA/siteminder/r12/webagent/config/SmHost.conf

    warning: CORE3283: stderr: [19/Jul/2014:00:28:00] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0xc87b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:00] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Created semaphore 786470 using key 0xc87b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:00] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x6b7b3a6c

    info: CORE3282: stdout: LLAWP has been started.

    warning: CORE3283: stderr: 19635: CSmSem::getSem - created semaphore 589863 using key 1677758905 (0x640091b9)

    warning: CORE3283: stderr: 19635: CSmSem::getSem - created semaphore 524328 using key 4093678009 (0xf40091b9)

    warning: CORE3283: stderr: 19635: CSmSem::getSem - Could not attach to semaphore with key 838898105 (0x320091b9), does not exist

    warning: CORE3283: stderr: 19635: CSmSem::getSem - Could not attach to semaphore with key 1711313337 (0x660091b9), does not exist

    warning: CORE3283: stderr: 19635: CSmSem::getSem - Could not attach to semaphore with key 1778422201 (0x6a0091b9), does not exist

    warning: CORE3283: stderr: 19635: CSmSem::getSem - created semaphore 524330 using key 1711313337 (0x660091b9)

    warning: CORE3283: stderr: 19635: CSmSem::getSem - created semaphore 524329 using key 838898105 (0x320091b9)

    warning: CORE3283: stderr: 19635: CSmSem::getSem - Could not attach to semaphore with key 1728090553 (0x670091b9), does not exist

    warning: CORE3283: stderr: 19635: CSmSharedSegment::smalloc - created shared memory segment 6026 using key 1627427257 (0x610091b9)

    warning: CORE3283: stderr: 19635: CSmSem::getSem - created semaphore 524331 using key 1778422201 (0x6a0091b9)

    warning: CORE3283: stderr: 19635: CSmSem::getSem - created semaphore 524332 using key 1728090553 (0x670091b9)

    warning: CORE3283: stderr: 19635: CSmSharedSegment::smalloc - created shared memory segment 2427 using key 1761644985 (0x690091b9)

    warning: CORE3283: stderr: 19635: CSmSharedSegment::smalloc - created shared memory segment 2428 using key 1694536121 (0x650091b9)

    warning: CORE3283: stderr: 19635: CSmSem::getSem - created semaphore 524333 using key 1744867769 (0x680091b9)

    warning: CORE3283: stderr: 19635: CSmSem::getSem - created semaphore 524334 using key 1761644985 (0x690091b9)

    warning: CORE3283: stderr: 19635: CSmSem::getSem - Could not attach to semaphore with key 855675321 (0x330091b9), does not exist

    warning: CORE3283: stderr: 19635: CSmSem::getSem - created semaphore 524335 using key 855675321 (0x330091b9)

    warning: CORE3283: stderr: 19635: CSmSharedSegment::smalloc - created shared memory segment 2429 using key 1644204473 (0x620091b9)

    warning: CORE3283: stderr: 19635: CSmSem::getSem - Could not attach to semaphore with key 872452537 (0x340091b9), does not exist

    warning: CORE3283: stderr: 19635: CSmSem::getSem - created semaphore 524336 using key 872452537 (0x340091b9)

    warning: CORE3283: stderr: 19635: CSmSharedSegment::smalloc - created shared memory segment 2430 using key 1660981689 (0x630091b9)

    warning: CORE3283: stderr: [19/Jul/2014:00:28:05] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:06] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:07] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:08] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:09] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:10] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:11] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:12] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:13] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:14] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:15] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:16] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:18] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:19] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:20] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:21] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:22] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:23] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: 19635: CSmSharedSegment::smfree - removed shared memory segment 2428

    warning: CORE3283: stderr: 19635: CSmSem::SemRm - removed semaphore 524333

    warning: CORE3283: stderr: 19635: CSmSem::SemRm - removed semaphore 524334

    warning: CORE3283: stderr: 19635: CSmSem::SemRm - removed semaphore 524330

    warning: CORE3283: stderr: 19635: CSmSem::SemRm - removed semaphore 524332

    warning: CORE3283: stderr: 19635: CSmSem::SemRm - removed semaphore 524331

    warning: CORE3283: stderr: 19635: CSmSharedSegment::smfree - removed shared memory segment 2427

    warning: CORE3283: stderr: 19635: CSmSem::SemRm - removed semaphore 524328

    warning: CORE3283: stderr: 19635: CSmSharedSegment::smfree - removed shared memory segment 2430

    warning: CORE3283: stderr: 19635: CSmSem::SemRm - removed semaphore 524336

    warning: CORE3283: stderr: 19635: CSmSharedSegment::smfree - removed shared memory segment 2429

    warning: CORE3283: stderr: 19635: CSmSem::SemRm - removed semaphore 524335

    warning: CORE3283: stderr: 19635: CSmSharedSegment::smfree - removed shared memory segment 6026

    warning: CORE3283: stderr: 19635: CSmSem::SemRm - removed semaphore 524329

    warning: CORE3283: stderr: 19635: CSmSem::SemRm - removed semaphore 589863

    warning: CORE3283: stderr: [19/Jul/2014:00:28:24] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:25] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:26] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:27] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:28] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:29] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:30] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:31] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:32] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:33] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:34] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:35] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:36] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

     

    warning: CORE3283: stderr: [19/Jul/2014:00:28:37] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:38] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:39] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:40] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:41] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

     

    warning: CORE3283: stderr: [19/Jul/2014:00:28:42] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:43] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:44] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:45] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:46] [Info] [CA WebAgent IPC] [19601] [CSmSem::getSem] Attempted to attach to non-existent semaphore with key 0x667b3a6c

    warning: CORE3283: stderr: [19/Jul/2014:00:28:46] [Error] SiteMinder Agent

    warning: CORE3283: stderr:         Failed to initialize the message bus.

    warning: CORE3283: stderr: [19/Jul/2014:00:28:46] [Warning] SiteMinder Agent

    warning: CORE3283: stderr:         SiteMinder agent has encountered initialization errors and will not service requests.

    warning: CORE3283: stderr: [19/Jul/2014:00:28:46] [Info] [CA WebAgent IPC] [19601] [CSmSem::SemRm] Removed semaphore 786470

    failure: CORE3187: Late initialization failed: Error running init function SmInitChild: unknown error



  • 2.  Re: Failed to initialize message bus

    Posted Aug 15, 2014 10:40 AM

    Does anyone have feedback for Vivek here?



  • 3.  Re: Failed to initialize message bus

    Posted Aug 17, 2014 07:58 PM

    Hi Vivek,

     

    Could you please clarify few questions for me ?

     

    • What is your new version of SiteMinder Agent ? Did you upgrade to the latest available CR ?
    • You seem to have lots of orphan semaphore/memory segments ? Have you tried cleaning them up ?
    • Have you tried restarting the Solaris box ?
    • You said that you get the error while starting the second web agent instance. Did you try starting each one individually and does both of them starts fine if started first ?

           The reason I am asking is because you seem to have some permission issue as well. See below :

             Siteminder Web Agent not having write permissions on host configuration file. Shared secret roll-over may not be supported.

             warning: CORE3283: stderr:         Permission denied. Please assign write priviledges to the user nobody for the file /usr/local/CA/siteminder/r12/webagent/config/SmHost.conf

     

    If it still doesn't work, then please try to start up the webserver instance using truss command as below :

    • truss -feald -o sem.out -t semop,semctl <start-script-name>

    This will give you idea of what's going wrong with the start-up as it keeps the record of all activity during the startup.

     

    Hope this helps.

     

    Cheers,

    Ujwol



  • 4.  Re: Failed to initialize message bus

    Posted Aug 26, 2014 12:30 PM


    Thanks Ujwol,

     

    -Its is latest R12sp3Cr12 version.

    - Yes , i cleaned up all the semaphores and shared memory segments but still on starting the 1 particular instance it throws this error.

    - Havent restarted the solaris box that would be the absolute last resort

    - Yes the problem arises only when 1 particular instance is started irrespective of the order of startup.

     

     

    I can live with the warning and i dont think that would prevent the agent startup.

    Willl capture the truss output next time to see whats happening.

     

    Thanks



  • 5.  Re: Failed to initialize message bus

    Posted Aug 18, 2014 09:45 AM

    Vivek,

     

    there have been so many fixes since V5.0 of any SP combined with the fact they are years out of support, that i feel your best path forward is to verify your new up-to-date Policy Server is working and then to REMOVE the old v5 agent completely and install a current, compatible one. Migrate the server if needed.

     

    I see no glaring errors here. the only one i see is the smhost not readable. the rest are actually info messages aside form the two init failures.

     

    -Josh



  • 6.  Re: Failed to initialize message bus

    Posted Aug 26, 2014 12:35 PM

    Thanks Josh

     

    I agree that moving to latest version is the optimal approach but unfortunately i cannot do that at this hence need to have r12sp3cr12 and V5 agent running on the system.

     

    It happens only with one instance irrespective of the order in which they start which indicates that it might be due some stuck semaphores although i checked/rechecked and cleared every semaphore/sshared memory from the server.

     

     

     

    Vivek



  • 7.  Re: Failed to initialize message bus

    Posted Aug 26, 2014 01:43 PM

    the file structure for V5 and V6+ are different.

    You should not be running both together. they can crash into each other and have a lot of issues.

     

    If it is a shared memory crash issue (a common cause but not the only cause of this) then you can increase the available shared memory segments to the Web Server/SM

     

    also remember that the v5 is compatible with newer policy servers, but the new v6+ agents CANNOT be driven from a v5 policy server.

    v12+ can be driven by a v6 ps, but in a very very functionality reduced mode.