Hi Kalyan,
Thanks for the information. We prefer to work the issue on the current version and environment while we have the opportunity to do this on our lab environment. We could try a reinstall or upgrade, but we would like to figure out and document the underlying problem in case we run into similar issues at some point on our production environment.
We do see one set of warnings in catalina.out about a failure to bind on port 8443 after 10 tries (all within 1 second) when we restart the OC server:
May 14, 2013 02:46:11 PM (HttpListener-8443-1) (org.apache.axis2.transport.http.server.DefaultConnectionListenerFailureHandler) - (WARN) - Attempt number 10 of 10 to reestalish connection listener org.apache.axis2.transport.http.server.DefaultConnectionListener@c0890f due to failure
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365)
at java.net.ServerSocket.bind(ServerSocket.java:319)
at java.net.ServerSocket.<init>(ServerSocket.java:185)
at java.net.ServerSocket.<init>(ServerSocket.java:97)
at org.apache.axis2.transport.http.server.DefaultConnectionListener.run(DefaultConnectionListener.java:75)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
May 14, 2013 02:46:11 PM (HttpListener-8443-1) (org.apache.axis2.transport.http.server.DefaultConnectionListener) - (INFO) - Listening on port 8443
May 14, 2013 02:46:11 PM (HttpListener-8443-1) (org.apache.axis2.transport.http.server.DefaultConnectionListenerFailureHandler) - (ERROR) - Terminating connection listener org.apache.axis2.transport.http.server.DefaultConnectionListener@c0890f after 10retries in 0 seconds.
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365)
at java.net.ServerSocket.bind(ServerSocket.java:319)
at java.net.ServerSocket.<init>(ServerSocket.java:185)
at java.net.ServerSocket.<init>(ServerSocket.java:97)
at org.apache.axis2.transport.http.server.DefaultConnectionListener.run(DefaultConnectionListener.java:75)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
I don't know if this is an actual issue and if it's related, since things seem to be running fine on 8443. We can see with netstat/ps that a java OneClick process is listening on 8443 (and several other ports)
We don't see excessive alarms on any models.
Debugging on the OneClick admin page shows 0 alarms while CLI shows 82 alarms and the Alarm information section on the VNM model in OC shows 132 active alarms (including 47 maintenance alarms) and will update when you hit the refresh icon. The restful web services also show 132 alarms.
We do see another symptom: if we try to delete a model (in this case a Fanout/Rpt_Segment) is will disappear from the Topology view, but remain visible in the Navagation pane and the List view. If we try to delete it again from there, a error popup states the model does not exist. After a OC client restart the Navigation and List view will also be updated.
Regards,
Michiel