ESXi 10-15-2014 Patches break Citrix Netscaler VPX Appliance

[russ79]

Part warning, part question....

My ESXi Hosts were v5.5.0 206190 (straight from the 5.5U2-RollupISO2 cd), I applied the following patches:

ID: ESXi550-201410101-SG  Impact: Important  Release date: 2014-10-15  Products: embeddedEsx 5.5.0 Updates esx-base

ID: ESXi550-201410401-BG  Impact: Critical  Release date: 2014-10-15  Products: embeddedEsx 5.5.0 Updates esx-base

ID: ESXi550-201410402-BG  Impact: Important  Release date: 2014-10-15  Products: embeddedEsx 5.5.0 Updates misc-drivers

ID: ESXi550-201410403-BG  Impact: Important  Release date: 2014-10-15  Products: embeddedEsx 5.5.0 Updates sata-ahci

ID: ESXi550-201410404-BG  Impact: Important  Release date: 2014-10-15  Products: embeddedEsx 5.5.0 Updates xhci-xhci

ID: ESXi550-201410405-BG  Impact: Important  Release date: 2014-10-15  Products: embeddedEsx 5.5.0 Updates tools-light

ID: ESXi550-201410406-BG  Impact: Important  Release date: 2014-10-15  Products: embeddedEsx 5.5.0 Updates net-vmxnet3

afterwards they were v5.5.0 2143827

Once upgraded my 2 Citrix Netscaler VPX Appliances (vmware hardware version 8, Citrix Netscaler OS v10.5.52.11nc (latest), FreeBSD 64bit, E1000 Nic) stopped responding on the network. They work fine when on the older 5.5U2 release but start dropping packets intermittently and then ultimately drop off the network when migrated to a host that has the above listed patches. I have 2 in High Availability mode, this occurs even if just one is on and the other off. It goes back to normal when vmotioned back to the pre-patched host.

I've tried switching networks, removing and re-adding nics, upgrading to hardware 10, deleting them all together and deploying the latest version from citrix's website, if it doesn't drop off all together it will drop a lot of packets. You can watch the netscaler console show NODE FAIL, NODE DOWN, and NODE UP events many times a minute.

[Linjo]

Its not recommended to upgrade the virtual hardware on Virtual Appliances, did you also change something with the NIC?

// Linjo

[marv8501]

Same issue in our esx environment since the updates from yesterday.

Netscaler starts and the VMWare Console is responding but after about 5min with a running CitrixGateway-Connection there is no communication to the netscaler website in and outside our network. (DMZ, LAN) No more ping to the VIPs. Only a restart of the Netscaler-VM will bring up the connections.again till a user ist logging in or a XenDesktop connection is starting.

Any Ideas?

[matthewkoeman]

Hi,

We have the same problem, netscaler works for a couple of minutes and than i need to do

dis int 0/1

reset int 0/1

en in 0/1

and then it works again for a couple of minutes with low usercount.

i see al lot of package drops in the applicance, this happens since the latest vmware esx patches from oktober 2014

i also created a case with vmware (14543709210) for this issue

[russ79]

hey guys thanks for adding to this thread, i too have created a case, SR # 14543312510... I'll let you know if we come to any conclusions

[Bill Oyler]

Same exact issue here with Netscaler virtual appliance on ESXi 5.5 Update 2 build 2143827.  Will be opening ticket with VMware.

[marv8501]

For the moment we build a nested VM Host with 5.1U2 on the updated 5.5U2. On this nested vm host the netscaler is working without problems.

[Bill Oyler]

I rolled back to ESXi 5.5 Update 2 (Build 2068190) and Netscaler is working great, FYI.

[russ79]

yeah that's what i had to do on one of my hosts to get it to work

[matthewkoeman]

just to keep you guys updated.. i have a webex scheduled with technical support for tomorrow morning. they want to collect data to investigate....

for the mean time i isolated a host and reinstalled ESXi 5.5 Update 2 (Build 2068190) on which netscaler vpx  is working just fine again

[crendall]

I'm glad I found this thread.  Rolling back to build 2068190 fixed the networking issues with our Netscaler. 

So do we need a patch from VMware or Citrix so that the Netscaler runs with this last batch of updates from VMware?

[GerwinVM]

Thanks for creating the thread.
We had the same issue yesterday during our upgrade of vmware.

Netscaler 10.1 + 10.5 same issue

Hopefully there is a fix soon.

[ThomasBrandlehn]

Allowing promiscous mode for the vswitch helped for me but this is a homelab so not sure if it works in production with more load.

vSwitch Properties - Security - Promiscous mode - Accept

It seems the update enforces more security on the switch.

vmkernel.log ->

2014-10-21T15:48:49.606Z cpu2:89298)etherswitch: L2Sec_EnforcePortCompliance:155: client NSVPX-ESX requested promiscuous mode on port 0x200000e, disallowed by vswitch policy

2014-10-21T15:48:49.642Z cpu3:89289)NetPort: 1632: disabled port 0x200000e

2014-10-21T15:48:49.643Z cpu3:89289)Net: 3354: disconnected client from port 0x200000e

Thomas

[Brookshealth]

We have the same issue after applying 2143827 patch. Fortunately we had one host left at 2068190 and migrated them there.

Anxiously awaiting a resolution.

[artjackson]

We ran into this issue last night.  Citrix tech had us do a roll back on the ESX server (shift r during reboot) to remove the patch and it fixed the problem. 

[JJ6709]

rollback resolved that issue, also created ticket with VMware and Citrix

[MarcHuppert]

I have the same problem at a customer site. We left one of the 8 ESXi hosts on ESXi 5.5 U2 without the latest patches.

Promiscous mode is not an option inside the production environment.

Hopefully a fix will be available shortly.....

[MarcHuppert]

I have the same problem with other VMs too. All have the E1000 vnic configured. Very strange.....

It a little bit like russian roulette

[matthewkoeman]

are you're other VM's with the issue FreeBSD 64bit  ??

[MarcHuppert]

No, they are Windows 2008R2....

[matthewkoeman]

then i'ts possible another issue, i recommand you to open a case @ VMWare. we do not have aby problems with other OS

[asananikone]

Same issue, resolved by rolling back ESXi.

[matthewkoeman]

Vswitch properties - Security - Promiscous mode -Accept

is a workarround for  in a test evoirement, but not possible in our production envoirement. still working with VMWare support to locate and fix the issue

[light_man]

stupid question maybe, but how to roll back easily?

same problems with a Netscaler VPX

[rufust3]

light_man,

On reboot of the host during the initial "Please wait while VMware is loading ..." screen, in the lower-right you'll see Shift-R (Recovery) as an option. When you press Shift-R, you'll be presented with the current build to boot into, or the previous one. Select the previous build. Once you've selected the previous build, you'll no longer have the option for booting to the other build number.

[revordr]

Also had this issue last night with updates so we rolled back, we also tried 4 different builds of the netscaler VPX 10.0-10.5 latest all had the same problem, if we moved the vm from a patched to unpatched host it would immediately start working again.

[JBergson]

Same problem for me. Netscaler and Esx 5.5 Update 2

[Itsjustus]

We had the same problem and have a new working situation now.

Haven't figured out the exact cause yet, but is has to do with a misconfiguration of the netsacler software in combination with delivery controllers/subnets/ network and/or arp requests.

Works as designed (which I think was wrong in the first place)

Ik took us 3 days to solve.

Yes we did the vmware update and had the same issue. Bring the network interfaces up and down, you have a ping and suddenly the netscaler stops responding. And on the console we were not able to ping any gateway or external server.  But we had 100+ other vmservers with no problems. And yes it looked like this was our only freebsd based server, and yes our only netscaler server. So why  did that one only have the problem.

I think the software thinks it has a new route based on an incoming packet from a citrix delivery controller and decides on it's own to change the default route to an other interface (not sure yet which one the Nsip/vip/mip/snip or whatever)

We got the latest netscaler software, buildt it on our XEN hypervisor environment got it working. And after a few hours we got a similar problem. In this case the ip adresses were still pingable but the result was the same. No access.

In the netscaler virtual server STA ticketing authority interface we had 4 controllers 2 down and 2 up. The ones that were up were  on a different subnet than the one that were down.

The up ones were our latest added delivery controllers. They were in the same ip-subnet as the NSIP interface.

So we removed them rebooted the netscaler and the problem was solved.

My concusion (for now) the STA must not be in the same subnet as the NSIP (and /or) the 2 VIP's we had. Our MIP was in allreadyin a different subnet.

Now I'm trying the figure out with our external party who installed out netscaler appliance 2 years ago what the requirements are for all the different interfaces and subnets/vi/STA etc within the netscaler software.

And I think it was pure luck the pasted 2 yearsk that the vmware hosts probably blocked that uninteded network trafic/arp requests or whatever.

When the smoke clears we will try to install it back on the vm platform and see if we get it working again

Forgive my typos etc, I'm in a hurry (lost 3days not puzzling ;-)

Cheers and have a nice weekend.

Eric Burger

[JBergson]

Does Citrix have any solution (except roll-back ESX updates)?

Does anyone here have opened case with Citrix - so I can give them reference, when I open my case?

[JJ6709]

Citrix Technical Support Case - 64255155

[melay]

Do you happen to have a VM in the same port group with a VMXNET 2 adapter ?

VMware KB: Virtual machines in the same vSwitch or vNetwork Distributed Switch (VDS) as a virtual machine running VM…

[matthewkoeman]

VMWare techsupport told me : The issue seems to be a NetScaler driver problem when setting the tx_ring size, the ESXi patch just exposes the issue, but it is not the root cause of it. This is the recommendation from engineering so far: "Suggest customer consult Citrix to upgrade the e1000 driver on NetScalar to see if the issue disappears.   can somebody open a case @ citrix and keep us updated> (i only have sa @ citrix and not a support contract)"

[russ79]

I'm in the same boat at this point... its not VMware's problem and the status of our citrix support contract is shaky

[SinginScott]

Sounds like a VMware problem to me.. I mean you apply a patch and then you break a supporting service. I mean I hate to not blame Microsoft in this case like we all do but if it sound like a duck and looks like a duck.

[JJ6709]

I had an opened ticket with citrix regarding that issue, and they saying that already working together with VMware to resolve this problem.

There is no update yet from citrix support at this time.

[dluebbe]

To JJ6709: Did you get any update from Citrix support in the meantime?

Apart from the Citrix support article CTX200278 I can't find any new information about this issue. Although the workaround is working fine until now, I would really like to have a fixed version of Netscaler in place that can receive future updates without having to worry about implementing the workaround again.

[JJ6709]

The last reply from Citrix is that this workaround is the fix.

I am sure they will addressed that in next firmware release, hopefully.

Thx,

***********************************************

Typos courtesy of my ridiculously hip apple product.

[Bleeder]

The response marked as the "Answer" is not really a fix.  It is listed as a workaround (by both VMware, and Citrix).  Citrix states "Citrix is actively at work on this issue and will deliver the fix in a future maintenance release. Follow issue ID #0510673 for updates on this issue."

I don't know if moderators watch these forums but it might be nice to have an option to mark responses as workarounds and not just answers.

[nsojka]

Citrix has released Netscaler firmware updates this month for Netscaler 10.1 and 10.5 VPX, but both sets of release notes list the workaround as the "fix".  Anyone with additional information that can comment?  If I find additional information, I'll comment back.

[jesusislove3]

Has anyone had problems with the vmware fix from early December with Netscaler?

Thank you.

[aarshe]

i just applied the early december patch ESXi550-201412001.zip

i spent 2 hours frantically trying to figure out what broke the netscaler VM (well i figured it was the vmware patch, but what? )

untill thank god i found this thread.

the work around works on the dec 2014 patch i can confirm.

[nsojka]

Thank you for posting this!  We've been searching up and down for the last 24hrs trying to find what caused the sudden issues on our Netscaler.  Rolling back the ESX patch using the above instructions solved the issue.  Please post back when a permanent fix is released.

[dshapiro]

We had exactly same problem. The issue was resolved by downloading updated Netscaler appliance from Citrix importing it to VMware and configuring it. Once done everything is stable. Hope this helps.

[JJ6709]

what is the updated Netscaler appliance version/build etc. we have the latest update Netscaler ADC 10.5 Build 5211 (Oct 1 2014)

[Itsjustus]

Can anyone confirm if they also solved the problemby changing the STA(s) ip address(es) into an other subnet.

Afterwords I though there still could be 2 workarounds for the problem, the one I proposed or maybe all the STA's shouldl be in the same subnet.

Becasue we had  2 in one subnet and 2 in the one of the NSIP.

I would wait for a patch, because I think it's going wrong on the network level (by design) and my expirience is that even the big multinationals

make the same mistakes twice.

Eric

[JJ6709]

You should never have STA's in the same subnet as the netscaler (NSIP)

The SNIP subnet IP is designed to communicate with the STA's.

Anyway, Citrix and VMware need to work together to fix the code, mostly it is Citrix that need to take actions for that to be resolved.

We have few hundred VM's and only one, the citrix netscaler will brake when moved to the ESXi host with the latest  version v5.5.0 2143827

[matthewkoeman]

FIX (or work around?)

Hi all,

when working with VMWare Technical support they foud a possible solution which is working for  me almost an hour now.

maybe you guys can "Test" this solution also ?

Enter the shell mode in the NetScaler, then:

1) find where loader.conf is located on NetScaler VM    #find / -name loader.conf

For the uploaded NetScaler VM, there are 2 loader.conf: ./flash/boot/defaults/loader.conf and ./flash/boot/loader.conf, we only need to change the first one.

2) add "hw.em.txd=512" to loader.conf, this will change Tx ring size to 512 (note: do not set the ring size to 256, this will cause NetScaler VM core dump)

3) reboot the NetScaler VM

4) migrate it back to a host with latest patches

Good luck.....

edit: still working fine, also with userload in production environment

[russ79]

I made the change this morning and so far it appears to be working

[dshapiro]

We downloaded NetScaler NS10.5: Build 51.1017.e.nc, but it would still drop off the network.  After we implemented the fix mentioned above it is stable so far for some 3 hours.

[JJ6709]

The same here, looks like that is the workaround for that issue, will confirm that tomorrow.

Message was edited by: JJ6709 Update, After adding hw.em.txd=512 to loader.conf file in Netscaler VPX 10.5 52.11.nc (October firmware) and moved to ESXi 5.5.0 2143827 (with latest patches) The Netscaler is stable, there is no warnings or errors since over 14 hours.

[Steve1712201110]

Hello there,

just can confirm the issue, cost me and my 2 Colleges the last  6 hours!  we tried everything too,even did some network  traces on netscaler, without any finding... even tried 10.1, redeploy the appliance (10.5), test'd also with some private network to rule out any issues with physical network (adapters/switches)....

i've added the hw.em.txd=512 line just underneath the "exec=...." line...

##############################################################

###  Basic configuration options  ############################

##############################################################

exec=".( Loading /boot/defaults/loader.conf ) cr"

hw.em.txd=512

kernel="kernel"         # /boot sub-directory containing kernel and modules

bootfile="kernel"       # Kernel name (possibly absolute path)

kernel_options=""       # Flags to be passed to the kernel

rebooted and tested, and for now it looks fine!

thanks for this finding, and a bit sad that something can happen either way, because i've updated to update2 (with the latest hotfixes) a few days ago because the intel e5 V2 CPU Issues, where under certain circumstances win2008r2 vm (in my case the terminalservers) started bsod'ing with Memory Errors.... from one issue into the next as it seams!

best regards

Stephan

[dabrav]

Hi Mattew,

Many thanks for this work around. This worked for me.

FIX (or work around?)

Hi all,

when working with VMWare Technical support they foud a possible solution which is working for  me almost an hour now.

maybe you guys can "Test" this solution also ?

Enter the shell mode in the NetScaler, then:

1) find where loader.conf is located on NetScaler VM    #find / -name loader.conf

For the uploaded NetScaler VM, there are 2 loader.conf: ./flash/boot/defaults/loader.conf and ./flash/boot/loader.conf, we only need to change the first one.

2) add "hw.em.txd=512" to loader.conf, this will change Tx ring size to 512 (note: do not set the ring size to 256, this will cause NetScaler VM core dump)

3) reboot the NetScaler VM

4) migrate it back to a host with latest patches

Good luck.....

edit: still working fine, also with userload in production environment

[ITSTS]

Great... so great...

yesterday we run into this issue... and had no idea what happens.

First we check all VLANs and the switch configurations.

After hours about thinking and searching... we come accross a failure in the logs... after looking at google :smileywink: we found this thread and your solution...

This works fine in hour environment... We are using ESXi 5.1U3 and Netscaler 10.5 Build 54.9NC

Thank you for sharing that with us.

[karps]

I had this issue moving from ESXi 5.1 U2 to U3. My symptoms were that the VPX drops off the network after ~ 12 hours.
I applied the above fix and the HA pair is syncing again. Thanks heaps! :smileyhappy:

[nsojka]

FYI.  The January 30th firmware release for Netscaler 10.5 fixes this issue.  We've been running well for almost a month on the newest firmware.  No firmware update for 10.1 yet.  Follow this thread for the latest on the Citrix side.

http://discussions.citrix.com/topic/357217-vmware-patch-update-10-15-2087359-causes-netscaler-vpx-to-fail/

[rlondono75]

The temp fix also worked for me on ESXi 5.5 Build 2638301.

I have just updated the NetScaler VPX to 10.5 Build 56.15 and removed the temp fix and everything is working properly.

[LiamGP]

We had this issue yesterday too.

Using the loader.conf workaround did not work for us however, it did allow the devices to stay up but performance was so bad no one could log in.

We downgraded a host in our cluster and everything started working fine again instantly.

Thanks for the thread, really helped us out!

[Thomas Huber]

Hi,

we had the same issue too. I had to migrate to an older version of esxi.

Version 5.5.0, 1892794 worked for me fine.

Is there any patch planned from vmware?

Thomas

[matthewkoeman]

Hi, fix is working fine for us already more than a week. it isn't a VMware issue, actually it's a citrix netscaler vpx issue

[JJ6709]

ditto, Matt is correct

[kellyjones]

i seem to have the same problem on 5.1 build 2191751, same fix applied

[dluebbe]

I can confirm that this also affects NetScaler VA 10.1 Build 128.8 on ESXi 5.1.0 build 2191751. Applying the workaround with hw.em.txd=512 in loader.conf does the trick for now.

I've subscribed to the following KB article - let's see when Citrix or VMware come up with a permanent solution.

VMware KB: After applying patches from ESXi Update2-Rollup ISO, Citrix NetScaler virtual machine with e1000 vNIC los…

[Steve1712201110]

i can confirm issue on ESX 5.1 build 2191751 too!!

i guess the troublemaker is following patch:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2087359

• When you run Solaris 10 or 11 virtual machines using the e1000 adapter, the network throughput within the Solaris guest OS is very poor.

sounds that something for solaris 10 and/or 11 and e1000 has been fixed and bother the netscaler e1000 drivers now...

as the same thing has gotten fixed in 5.1 patch:

VMware KB: VMware ESXi 5.1, Patch ESXi510-201410401-BG: Updates esx-base

• When you upgrade from ESXi 5.1 Update 1 to ESXi 5.1 Update 2, you might encounter performance issues on the Solaris 10 and Solaris 11 virtual machines with e1000 vNIC adapters.

[myuen]

Citrix created the public KB: CTX200278

http://support.citrix.com/article/CTX200278

I did not find a way to be automatically notified when there is an update made to the KB. Let me know if anybody finds a method. Thanks.

[Steve1712201110]

It is definitly just the one patch -> ESXi550-201410401-BG (and for 5.1= ESXi510-201410401-BG) ....