EM failed to start - /tmp/librocksdbjni8883977260053861907.so: failed to map segment from shared object: Operation not permitted

[pivst01]

Hi,
after upgrading APM from v9.7 to v10.5 we cant start the EM because of the next error:

[ERROR] [main] [Manager] The EM failed to start. /tmp/librocksdbjni8883977260053861907.so: /tmp/librocksdbjni8883977260053861907.so: failed to map segment from shared object: Operation not permitted

 

do you have any idea what could be the rootcause and how to fix it?

 

the APM user doesnt have sudo permission, but we tried it to start under root and it has gives us the same error message.

 

here is the full log:
2/27/17 04:50:40.536 PM CET [INFO] [main] [Manager] Introscope Enterprise Manager Release 10.5.1.8 (Build 990101)
2/27/17 04:50:40.537 PM CET [INFO] [main] [Manager] Using Java VM version "Java HotSpot(TM) 64-Bit Server VM 1.8.0_112" from Oracle Corporation
2/27/17 04:50:40.538 PM CET [INFO] [main] [Manager] Using Introscope installation at: /opt/CAMON/Introscope-MoM/.
2/27/17 04:50:40.538 PM CET [INFO] [main] [Manager] CA Wily Introscope(R) Version 10.5.1
2/27/17 04:50:40.538 PM CET [INFO] [main] [Manager] Copyright (c) 2016 CA. All Rights Reserved.
2/27/17 04:50:40.538 PM CET [INFO] [main] [Manager] Introscope(R) is a registered trademark of CA.
2/27/17 04:50:40.539 PM CET [INFO] [main] [Manager] Starting Introscope Enterprise Manager...
2/27/17 04:50:40.896 PM CET [INFO] [main] [Manager.HotFailover] Hot Failover not enabled
2/27/17 04:50:40.904 PM CET [INFO] [main] [Manager] The Introscope Enterprise Manager is set to run in MOM mode.
2/27/17 04:50:40.919 PM CET [INFO] [main] [Manager] Starting server...
2/27/17 04:50:41.017 PM CET [INFO] [main] [Manager.Cluster] The Enterprise Manager is running in a Cluster Manager role.
2/27/17 04:50:41.137 PM CET [INFO] [main] [Manager] Started hot config polling. Polling interval set to 60 seconds
2/27/17 04:50:41.392 PM CET [INFO] [main] [Manager.Authentication] Introscope WebView connections are enabled
2/27/17 04:50:41.392 PM CET [INFO] [main] [Manager.Authentication] Introscope WebView connections will be assigned the name "WilyWebView"
2/27/17 04:50:41.488 PM CET [INFO] [main] [Manager.Authentication] Configured the Manager of Managers user using private key internal/server/EM.private
2/27/17 04:50:41.674 PM CET [INFO] [main] [Manager.UserManagementService] Context class loader=org.eclipse.core.runtime.internal.adaptor.ContextFinder@47c8bd08
2/27/17 04:50:41.681 PM CET [INFO] [main] [Manager.EemRealm] Started EEM Security Realm: EEM
2/27/17 04:50:41.774 PM CET [INFO] [PO Async Executor] [Manager.Cluster] New slow collector threshold property detected: 10000ms
2/27/17 04:50:41.775 PM CET [INFO] [PO Async Executor] [Manager.Cluster] New slow collector disconnect threshold property detected: 60 seconds
2/27/17 04:50:42.043 PM CET [INFO] [main] [Manager.ServerSocketFactory] Created Server Socket Factory: com.wily.jip.server.ext.DefaultServerSocketFactory
2/27/17 04:50:42.137 PM CET [ERROR] [main] [Manager] The EM failed to start. /tmp/librocksdbjni5626861023512855361.so: /tmp/librocksdbjni5626861023512855361.so: failed to map segment from shared object: Operation not permitted
2/27/17 04:50:42.149 PM CET [INFO] [main] [Manager] Shutting down the Isengard server
2/27/17 04:50:42.162 PM CET [INFO] [main] [Manager] Orderly shutdown complete.

2/27/17 05:00:26.712 PM CET [DEBUG] [main] [Manager] Error starting up EM
com.wily.introscope.server.enterprise.EnterpriseManagerCannotStartException: An error occurred while trying to start Isengard.
at com.wily.introscope.server.enterprise.EnterpriseServer.doStart(EnterpriseServer.java:418)
at com.wily.util.ALifeCycle.start(ALifeCycle.java:86)
at com.wily.introscope.server.enterprise.EnterpriseServer.<init>(EnterpriseServer.java:324)
at com.wily.introscope.server.enterprise.EnterpriseServer.<init>(EnterpriseServer.java:303)
at com.wily.introscope.server.enterprise.EnterpriseServer.start(EnterpriseServer.java:1595)
at com.wily.introscope.em.internal.Activator.startEM(Activator.java:119)
at com.wily.introscope.em.internal.Application.start(Application.java:27)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:193)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:386)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:549)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
at org.eclipse.core.launcher.Main.main(Main.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.zerog.lax.LAX.launch(Unknown Source)
at com.zerog.lax.LAX.main(Unknown Source)
Caused by: java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni1704279573734500032.so: /tmp/librocksdbjni1704279573734500032.so: failed to map segment from shared object: Operation not permitted
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:47)
at org.rocksdb.RocksDB.<clinit>(RocksDB.java:23)
at org.rocksdb.Options.<clinit>(Options.java:21)
at com.wily.introscope.server.enterprise.entity.rocksdb.RocksDBSystem.<init>(RocksDBSystem.java:47)
at com.wily.introscope.server.enterprise.entity.rocksdb.RocksDBEntityImpl.<init>(RocksDBEntityImpl.java:36)
at com.wily.introscope.server.enterprise.EnterpriseServer.initialize(EnterpriseServer.java:590)
at com.wily.introscope.server.enterprise.EnterpriseServer.doStart(EnterpriseServer.java:381)
... 26 more

thank you, stefan

[Jose Javier Sanchez Martin]

Hi Stefan,

 

It seems to be happening when the EM needs "exec" permission on the /tmp folder so you could try:

 

- Stop the EM

- sudo mount -o remount,exec /tmp

- Start the EM

 

Hope it helps.

 

thanks

Jose

 

 

 

sudo mount -o remount,exec /tmp

[Hallett German]

Dear Stefan:

Based that Jose Javier found the same thing I did, I am marking as correct. However, you may post an update and additional questions as needed.

 

Thanks

Hal German

[Hallett German]

Googling it this seems to be a common errors across multiple products and appears to be permissions related. Found multiple hits on it for other products. 

 

failed to map segment from shared object error on start · Issue #1339 · docker/compose · GitHub 

[pivst01]

FYI the solution:
creating tmp dir in EM home and appending -Djava.io.tmpdir=<EM_home>/tmp to lax.nl.java.option.additional in Introscope_Enterprise_Manager.lax solved the issue. thanks to Keith for the hint!