Cannot upgrade VCSA 6.0 to 6.5 - Internal error occurs during Export of vSphere ESX Agent Manager

[vm7user]

Hello,

Cannot upgrade VCSA 6.0 to VCSA 6.5

always on stage 2 i get this error:

Error

Internal error occurs during Export of vSphere ESX Agent Manager.

Resolution

Send upgrade log files to VMware technical support team for further assistance.

i found article https://kb.vmware.com/kb/2147898, but this article applied only to vCenter Server 6.0 installed on Windows

[muxonarization]

Yes, I am not alone! See my findings:

Appliance 6.0U3 to 6.5a fails with Internal Error during Export of EAM

This even happens with a brend new VCVA6U3 when migrated to 65A.

[Xvor]

Same here; 6.0u3 to 6.5 is a no go at the moment.

The external PSC is upgraded fine but vCenter component breaks at part 2 step 1.

I guess we shouldn't have patched to u3.

[Krom80]

Hello,

same here, part 2 fails after 34 % with "Internal error occurs during Export of vSphere ESX Agent Manager"... Unable to upgrade :-(

[freeinthesky]

yeah, exactly same error here :smileysad:

	"localized": "Internal error occurs during Export of vSphere ESX Agent Manager.",
	"translatable": "Internal error occurs during %(0)s of %(1)s."
	}
	],
	"componentKey": "upgrade_framework",
	"problemId": null
	},
	"progress": 34

[Bouda0175]

I have exactly same issue. I have VCSA 6.0 U3 tried to 6.5 A, but fails with the same error:

Problem is only with 6.0 U3. I tried upgrade on U2 and it works.Unfortunately, my production is on U3, so I am not able to upgrade.

My question is, when will be possible to do upgrade form 6.0 U3 to 6.5 A?

Why VMware doesn't support upgrade between latest versions of their major products?:smileyangry:

[freeinthesky]

I tried both - 6.0 U3  to 6.5 and 6.0 U3  to 6.5a - same error. Looks like the problem is with 6.0 U3.

Still there,s no official word from VMware. Doesn't look that they plan to fix it...

[Harapng]

Hi everyone.

Don't know if the solution is going to work for the appliance migration, but i was getting the same error during a windows vCenter 6.0U3 to appliance 6.5 migration and i solve it by:

1. Following the steps here Cannot remove or disable unwanted plug-ins from vCenter Server and vCenter Server Appliance (1025360) | VMware KB  to UnregisterExtension for the vSphere ESX Agent Manager.

2. On the windows vCenter machine, elevate a command prompt and execute from  C:\Program Files\VMware\vCenter Server\eam\wrapper\bin\ the UninstallApp-NT.bat to remove the ESX agent manager service.

3. Re-run the migration worked like a charm.

Hope it will work for you as well!

[freeinthesky]

Unfortunately  we use VCSA, so I could easy  remove Network & Security Plugin in  MOB, but can't  remove the ESX agent manager service.

That  .UninstallApp-NT.bat is available in Windows. Could not find any useful info on Google.

Actually I managed to remove eam(ESX Agent Manager) RPM file but service is still showing ( not running)

[Xvor]

vCenter 6.5b is available, hopefully this resolves it:

VMware vSphere 6.5.0b Release Notes

vCenter Server upgrade might fail at pre-check phase of ESXi Agent Manager
ESXi Agent Manager might fail during the export phase. Failing at the export phase stops multiple services. This issue is resolved in this release by upgrade pre-check, without affecting vCenter Server or its health.

ESXi Agent Manager might fail during the upgrade with these possible symptoms:

• EAM mob is not working.
• The service is unresponsive on HTTP or API calls.
• The user interface at Home > Administration > vCenter Server Extensions > vSphere ESX Agent Manager > Manage in vSphere Web client is not working.

If you encounter the following error, see http://kb.vmware.com/kb/2147898:
Internal error occurs during Export of vSphere ESX Agent Manager

This issue is resolved in this release.

[Xvor]

UPDATE: just attempted VCSA u3 upgrade with 6.5b, still not working;

Same error.

[Bouda0175]

UPDATE: just attempted VCSA u3 upgrade with 6.5b, still not working;

Same error.

I was so happy about this update, that upgrade process from 6.0 U3 is fixed. But when I read your post, I am not sure, If I should try upgrade porcedure on my infrastructure. I gues, that I would ended with the same result.

[glowery]

This worked great for me after experiencing the same problem! Thanks!!

[jnewton03]

I was able to get this working yesterday with 6.5a.  I have not yet tried 6.5b.

Here is what I did:

To remove unwanted plug-ins from the available plug-in list:

1. In a web browser, navigate to http://vCenter_Server_name_or_IP/mob.
   
   Where vCenter_Server_name_or_IP/mob is the name of your vCenter Server or its IP address.
   
   
2. Click Content.
3. Click ExtensionManager.
4. Select and copy the name of the plug-in you want to remove from the list of values under Properties. For a list of default plug-ins, see the Additional Information section of this article.
5. Click UnregisterExtension. A new window appears.
6. Paste the key of the plug-in and click Invoke Method. This removes the plug-in and results in void.  (I used com.vmware.vim.eam)
7. Close the window.
8. Refresh the Managed Object Type:ManagedObjectReference:ExtensionManager window to verify that the plug-in is removed successfully.

I then stopped the VMWare-EAM service on the source appliance by doing the following:

1.  SSH into the source appliance

2.  Go into shell

3.  issue a  service-control --stop vmware-eam

I then re-ran the upgrade and was successful.  Note that doing just one or the other of the above steps may have worked, but I have not tested it.

[freeinthesky]

Was vmware-eam running before you stopped it ? Is it running now ?

In my case it's not starting - when I issue service-control --start vmware-eam  I get this error:

Starting VMware ESX Agent Manager...

WARN   | wrapper  | The 'EAM_WEB_DIR' environment variable was referenced but has not been defined.

WARN   | wrapper  | The 'EAM_WRAPPER_ADDITIONAL_FILE' environment variable was referenced but has not been defined.

Waiting for VMware ESX Agent Manager..................

WARNING: VMware ESX Agent Manager may have failed to start.

Last login: Tue Mar 14 15:25:28 NZDT 2017

Stderr = su: warning: cannot change directory to /home/eam: Permission denied

-bash: /home/eam/.bash_profile: Permission denied

FATAL  | wrapper  | Unable to resolve the working directory %EAM_WEB_DIR%: No such file or directory

FATAL  | wrapper  | Unable to set working directory to: /usr/lib/vmware-eam/wrapper/bin/%EAM_WEB_DIR% (No such file or directory)

[jnewton03]

I did not check to see if it was running first.  This is an error you're getting on the source appliance running 6.0U3?  Are you using custom ports?  What does your eam.properties look like?

[freeinthesky]

Correct: 6.0U3

I don't see any particular configuration in eam.properties (please see below)(maybe something is missing):

-------------------------------------------------------------------------------------------------------------------------------------------------------

#########################################################################

# Copyright 2013-2017 VMware, Inc.  All rights reserved. VMware Confidential

#########################################################################

vc.proxy.host=localhost

vc.proxy.port=##REVERSE_PROXY_HTTP_PORT##

# Hostname or IP of the EAM server

# Fill only if EAM is not running on the same host as VC

eam.host=

# EAM service port used to configure the HTTP connector of the application server.

eam.int.http.port=##EAM_INTERNAL_HTTP_PORT##

# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib

# file server.

eam.ext.port=##EAM_EXTERNAL_PORT##

eam.ext.scheme=https

# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib

# file server.

eam.ext.port.deprecated=##EAM_EXTERNAL_PORT_DEPRECATED##

eam.ext.scheme.deprecated=http

eam.support_linked_clone=true

eam.clear_db_on_startup=false

eam.debug_ref_count=false

eam.recent_event_size=20

# Value is specified in minutes (set to 24h = 1440m)

eam.scan_for_unknown_agent_vms=1440

# The timeout to wait for hostd to restart on a host (set to 5m=300s)

eam.hostd_restart_timeout=300

# The following entries will be added verbatim to the advanced options

# of hosts on which EAM is enabled. All are optional.

Net.DVFilterBindIpAddress=169.254.0.1

Net.TrafficFilterIpAddress=

#The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).

Net.DVFilterVswifIpAddress=169.254.0.2

# Resource bundle configuration

eam.resourcebundle.filename=eam-resourcebundle.jar

# VLSI embedded tcServer configuration

#

tcserver.tmp.dir=##{TCSERVER_TMP_DIR}##

eam.web.root=##{EAM_WEB_ROOT}##

# EAM SSL configuration

#

eam.keystore.type=VKS

eam.key.alias=##{SSL_ALIAS}##

eam.keystore.storename=##{SSL_STORE_NAME}##

# CM configuration

#

cm.url=##{CM_URL}##

cm.wait.attempts=360

cm.wait.intervalSeconds=5

# SSO configuration

#

sso.wait.attempts=360

sso.wait.intervalSeconds=5

# VC SSL configuration

#

vc.truststore.type=VKS

vc.truststore.storename=TRUSTED_ROOTS

vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:##{VC_SDK_TUNNEL_PORT}##/sdk/vimService

vc.tunnelSdkUri=

drs.demandCapacityRatio=100

eam.web.root=/usr/lib/vmware-eam/web

[jgeorge04]

I just came here to say that I tried this with VCSA on 6.0 U3 and upgraded to 6.5b using this method.  Was really bummed when I started having issues and learned that the upgrade path is not supported but this comment got me running.  Thanks.

(Jnewton03's original post) ---

I was able to get this working yesterday with 6.5a.  I have not yet tried 6.5b.

Here is what I did:

To remove unwanted plug-ins from the available plug-in list:

1. In a web browser, navigate to http://vCenter_Server_name_or_IP/mob.
   
   Where vCenter_Server_name_or_IP/mob is the name of your vCenter Server or its IP address.
   
   
2. Click Content.
3. Click ExtensionManager.
4. Select and copy the name of the plug-in you want to remove from the list of values under Properties. For a list of default plug-ins, see the Additional Information section of this article.
5. Click UnregisterExtension. A new window appears.
6. Paste the key of the plug-in and click Invoke Method. This removes the plug-in and results in void.  (I used com.vmware.vim.eam)
7. Close the window.
8. Refresh the Managed Object Type:ManagedObjectReference:ExtensionManager window to verify that the plug-in is removed successfully.

I then stopped the VMWare-EAM service on the source appliance by doing the following:

1.  SSH into the source appliance

2.  Go into shell

3.  issue a  service-control --stop vmware-eam

I then re-ran the upgrade and was successful.  Note that doing just one or the other of the above steps may have worked, but I have not tested it.

[IntegratorP]

I,ve just upgraded from 6.0 U3a to 6.5.0U1c and had to follow the exact same steps.

(Jnewton03's original post) ---

I was able to get this working yesterday with 6.5a.  I have not yet tried 6.5b.

Here is what I did:

To remove unwanted plug-ins from the available plug-in list:

1. In a web browser, navigate to http://vCenter_Server_name_or_IP/mob.
   
   Where vCenter_Server_name_or_IP/mob is the name of your vCenter Server or its IP address.
   
   
2. Click Content.
3. Click ExtensionManager.
4. Select and copy the name of the plug-in you want to remove from the list of values under Properties. For a list of default plug-ins, see the Additional Information section of this article.
5. Click UnregisterExtension. A new window appears.
6. Paste the key of the plug-in and click Invoke Method. This removes the plug-in and results in void.  (I used com.vmware.vim.eam)
7. Close the window.
8. Refresh the Managed Object Type:ManagedObjectReference:ExtensionManager window to verify that the plug-in is removed successfully.

I then stopped the VMWare-EAM service on the source appliance by doing the following:

1.  SSH into the source appliance

2.  Go into shell

3.  issue a  service-control --stop vmware-eam

 

I then re-ran the upgrade and was successful.  Note that doing just one or the other of the above steps may have worked, but I have not tested it.

[Dudleydoggg]

I love these response and VMware also does it,  to Fix "What ever" remove unwanted plugins.  How do I know which ones are unwanted?

What plugin did you remove to get the installer to complete?

[Jeriko2k]

Just for fun, someone have done a test from a clean 6.0u3 to 6.5b? for me 6.5b fixed nothing also, same bug.

[Xvor]

Not quite sure what to make of this but the VMware Product Interoperability Matrices now shows 6.0 U3 as incompatible for upgrade to 6.5.0 because: " 'back in time' upgrade not supported (vSphere 6.5.0 released 5 months before vSphere 6.0 U3)".

Great, so now what? We have to wait for 6.5 U1?

[kondrichRHI]

We are also encountering this issue when migrating from vCenter Windows 6.0u3 to VCSA 6.5b.

UpgradeRunner.log shows:

"

2017-03-20T12:21:05.404Z INFO upgrade.states.component_states eam:Export: 2017-03-20T12:21:04.381Z INFO eam Going to access: http://localhost:80/eam/charset.jsp

2017-03-20T12:21:05.404Z INFO upgrade.states.component_states eam:Export: 2017-03-20T12:21:04.391Z WARNING eam vmware-eam service is not fully started, waiting till get up and running. Sleeping for5seconds. Error: HTTP Error 500: Internal Server Error

2017-03-20T12:21:06.410Z INFO upgrade.states.component_states eam:Export: didn't have progress. Next report will come after 60 seconds

"

I figured out that http://localhost:80/eam/charset.jsp gives an error when called on the source server in Internet Explorer:

"HTTP Status 500 - Unable to compile class for JSP

type Exception report

message Unable to compile class for JSP

description The server encountered an internal error that prevented it from fulfilling this request.

exception

org.apache.jasper.JasperException: Unable to compile class for JSP

  org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:600)

  org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:363)

  org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)

  org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)

  javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

root cause

java.lang.NullPointerException

  org.apache.jasper.compiler.Validator$ValidateVisitor.(Validator.java:515)

  org.apache.jasper.compiler.Validator.validateExDirectives(Validator.java:1853)

  org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:217)

  org.apache.jasper.compiler.Compiler.compile(Compiler.java:356)

  org.apache.jasper.compiler.Compiler.compile(Compiler.java:336)

  org.apache.jasper.compiler.Compiler.compile(Compiler.java:323)

  org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:585)

  org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:363)

  org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)

  org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)

  javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

note The full stack trace of the root cause is available in the Pivotal tc Runtime 3.1.5.RELEASE/8.0.36.A.RELEASE logs."

I guess that's the n+1st problem with migration. :-( Let's see what VMware support knows about that.

UPDATE:

I just figured out another issue:

Calling http://localhost/eam/index.jsp on an vCenter 6.0u3 (5112527) instance gives the mentioned JSP error.

Calling http://localhost/eam/index.jsp on an 6.0u2a (4541947) instance correctly gives "The vSphere ESX Agent Manager (vEAM) is successfully deployed.". Btw, charset.jsp is not existing on a 6.0u2a instance.

I verified this on different vCenter installations in our organization.

So it seems that there is a general 6.0u3 problem with EAM.

[jpiscaer]

I can confirm; http://vcsa-ip:80/eam/charset.jsp throws an error.

[SimonHuizenga1]

VMware vCenter Server 6.0 Update 3 Release Notes

Upgrade Notes for This Release

For instructions about upgrading vCenter Server and ESX/ESXi hosts, see the vSphere Upgrade documentation.

Upgrading from vCenter Server 6.0 Update 3 to vCenter Server 6.5 is not supported.

Took me some time to but its not supported ;-(

when you use the original http(s)://vcenter/eam/index.jsp you also get the errors i have installed several versions of 6.0 appliances (tested 6.0u1, u2 and u3) when i deploy Windows vCenter it shows a normal page.

[kondrichRHI]

Yes, but this does not indicate an upgrade or migration problem: 6.0u3 has a problem with EAM web service itself and within itself. Hence, the upcoming 6.5u1 update (scheduled for Q2 2017) would not help, if 6.0u3 is defunct.

I discussed this issue with VMware support today: I want to know why 6.0u3 reports an error when calling https://vcenter/eam/index.jsp and 6.0u2a does not report an error but successfully returns "The vSphere ESX Agent Manager (vEAM) is successfully deployed." They will check that

[SimonHuizenga1]

I just deployed a fresh vCenter (6.0.0.20100) 6.0u2a and this server is also showing 500 error.

Added the charset.jsp to /usr/lib/vmware-eam/web/eam/ and also 500 error.

But a upgrade ran without any problems.

If you have the update please post it! I'm working on it for more than a week now....

[SimonHuizenga1]

now downloading vCenter Server 6.0 Update 3a | 21 MAR 2017......

just released

[kondrichRHI]

vCenter 6.0u3a was successfully installed at our site, but it does not fix the index.jsp or charset.jsp problem. Actually, it just fixes an Apache CVE.

It's strange that vCenter reports build 5183551 after installing u3a, whereas ISO downloads show build 5202527.

[SimonHuizenga1]

yes same here. upgraded and no changes. also attempted to upgrade to 6.5b no success.

when i remove EAM from mob i can update to 6.5 but i have some strange tabs in the new vCenter so i reverted.

[Bouda0175]

yes same here. upgraded and no changes. also attempted to upgrade to 6.5b no success.

when i remove EAM from mob i can update to 6.5 but i have some strange tabs in the new vCenter so i reverted.

Could you please describe, what do you mean about strange tabs?

[SimonHuizenga1]

the #{Common:..... ones.

[Bouda0175]

That is realy strange, never see that. I performed several upgrade tests in my test lab with removed EAM a stop EAM services and everything works well. Now I am quite scare about upgrade on production. Do you use some external plugins in vCenter? Something like VDP, UpdateManager, VSM...

[cercompredes]

The vcsa 6.0U3 is compatible with upgrade to 6.5 [1] [2].

I rollback to the last snapshot of vcenter (6.0U2) and then I upgraded it to the 6.5. It solve to me.

[richsli78]

6.0u3 to 6.5 upgrade is not supported. see the release notes of U3.

[Charu]

It is true that you cannot upgrade from vSphere 6.0 U3 to vSphere 6.5.  A new KB article has been published which explains the support policy for vSphere upgrade paths.  Please comment on this article if you have any additional questions. 

[TomKalabis]

so if my customer have vcsa 6.0u3 clean instalation and he need upgrade to 6.5 ? what now ? wait for u4 ? :smileyhappy:

[Bouda0175]

I found some steps, which provides me smooth upgrade without fails on VCSA or Windows vCenter. I tried this procedure at 5 customers and all of them were without errors.

Procedure for VCSA:

1) Shutdown vCenter and make snapshot

2) Remove incompatible plugin (ESXi Agent Manager) from vCenter because vcenter 6.5 uses whole new version:

     go to http://vcenter/mob then content -> ExtensionManager->FindExtension = com.vmware.vim.eam

  

3) If this extension exists, you nedd to unregister it:

    UnregistreExtension = com.vmware.vim.eam

    Invoke Method

4) Now you need to stop the service "vmware-eam"

5) Connect via SSH to vCenter, enter to shell a then write the command:

    service-control --stop vmware-eam

6) Now you can run the upgrade manager wizard from VCSA ISO (eventually run migration-assistant if you use UpdateManager)

Procedure for Windows vCenter:

1) Shutdown vCenter and make snapshot

2) Remove incompatible plugin (ESXi Agent Manager) from vCenter because vcenter 6.5 uses whole new version:

     go to http://vcenter/mob then content -> ExtensionManager->FindExtension = com.vmware.vim.eam

3) If this extension exists, you nedd to unregister it:

    UnregistreExtension = com.vmware.vim.eam

    Invoke Method

4) Go to Windows services and stop service called "VMware ESX Agent Manager"

5) Now you can run the upgrade manager wizard from ISO

Those procedures worked for me and my customers without any issues.

[joshcurrier]

I'm seeing some instability due to a database error after migrating from 6.0 U3 to 6.5 b, just a heads up in case any of these other upgraded vCenters are being flaky.

2017-04-09T09:25:09.933Z error vpxd[7F1FE9E58700] [Originator@6876 sub=Default opID=HB-host-97553@32887-67f22e81] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "vpxi_psa_path";

--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_PSA_PATH (HOST_ID, PATH_ID, KEY_VAL, LUN_NUMBER, SCSI_LUN_ID, HBA_ID, TARGET_ID, LUN_LINK_KEY, ADAPTER_LINK_KEY, TARGET_LINK_KEY) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

[Dudleydoggg]

Bouda0175     Mar 30, 2017 11:04 PM    (in response to TomKalabis) Bouda0175

I followed Bouda0175 Steps and got past the Agent Manager Issue.

Alleged Update  completed Successfully.

[Jbir]

Following this discussion. I upgraded my test VCSA 6.0U3 last week to 6.5.0d and didn't experience any issues (luckily). I am now about to do ESXi and happened to look at the compatibility chart only to find to my horror that this upgrade path is not supported! I will probably do the test system anyway just for the hell of it, but i cannot do production or our Horizon environment until it is supported. I contacted VMware technical support who were not helpful at all. According to them there may be an update 1 due out in July or August but would not confirm if an upgrade to this from 6.0U3 would be supported :smileysad:

[yowmemperor]

THANK YOU!!!

I have been battling this error for about a week. (VC6 to 6.7 to get to VC 7) The installer logs were worthless, and I can't read the tar logs to send to VMware. No useful data, and much Googling led my down false paths. I saw this post and thought nothing to lose, doubt it will work since the error has nothing to do with extensions. I had the error stating "Cannot validate target appliance configuration as not enough information from the source appliance can be collected. For more details check out the server logs." Joke is on me, there were no useful details in the logs! And nothing about this error tells me an extension is in the way. Anyway rant over. Apparently they were an old 5.x environment under another MSP, and the extension happened to be exactly the same one you posted. This post was immensely helpful, thank you again!

[Charu]

I would suggest looking at the Upgrade Path tab of the Compatibility Matrix and selecting vCenter Server.  You can see the how the support policy documented in the vSphere Upgrade Policy KB has been implemented for previous releases, and infer what can be expected for vSphere 6.5.

[TomKalabis]

tested in my LAB. Upgrading VCSA 6.0u3 to 6.5 and it works:

How to upgrade VCSA 6.0 update 3 to VCSA 6.5 - tomaskalabis.com

[Bouda0175]

tested in my LAB. Upgrading VCSA 6.0u3 to 6.5 and it works: 

How to upgrade VCSA 6.0 update 3 to VCSA 6.5 - tomaskalabis.com

This is the same procedure as I described one post before. Confirm, that procedure works :smileyhappy:

[harry446]

Hi Tom, the upgrade procedure works - I successfully upgraded it from v6u3 to b6.5b. 

Thank you so much~~

[adamjg]

I'm guessing this procedure isn't going to be supported by VMware?

[ChristophBertho]

VCSA 6.5.0c was just released but does not fix the upgrade error :smileysad:

[2R3SQ]

3d released 4/18 does work!

[Mr_G_Grant]

This product just seems to be getting worse its like they are not even doing proper QA anymore! How does something like this get released without testing things like that? So now we are all just stuck at 6.0U3 until VMware decide to release another patch for this? Its just as well cloud computing is really taking off now and I cant wait to move off this platform. No more dodgy upgrades :smileyhappy:

I hate to even leave these kind of posts and I apologise for not offering any useful information in this post but this is really annoying!