RDM Pointer / vmfs block size ?

[wolfsonmicro]

We run ESX 4.0 on Dell R710s connecting back to an IBM N-Series SAN on FC.

This week we had a major outage due to a RedHat MySQL server falling over.

First  off the server would not boot past virtual POST so - knowing it had an  RDM disk I removed this from the configuration. The server then would  boot to OS but everytime the RDM was reconnected it would either fall  over or not boot.

I  remembered we had farily recently expanded the space on this server and  the RDM was now 300GB in size. However the space used was only 85% on  the lun (255GB). Then I remembered block size limitations. The server  was sitting on a vmfs partition with a 1MB block size (max 256GB). So I  removed the RDM and svMotion'd the server to a datastore which had a 2MB  block size, reconnected the RDM and the server booted fine. After a DB  consistency check we were back up and running.

So from this fault investigation I went on to read the whitepaper maximums document

http://www.vmware.com/pdf/vsphere4/r40/vsp_40_config_max.pdf

which to be honest- had vague information on RDMs

and then I found VMware KB

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1029697

which surprising to me, clearly states the RDM pointer files are affected by vmfs block sizes.

However there is a difference of opinion in this thread

http://communities.vmware.com/message/1507498

And  looking further at my own environment, we have multiple running  production server with connected RDMs which are in excess in size to the  stated block size limits.

I  need defined clarity on this because as it sits we could have many  production machines in an unstable state, or I am simply left with no  explanation for the disk lock we witnessed and an outage which caused a  substantial loss to the business.

regards
wolfsonmicro

Message was edited by: wolfsonmicro accidental click - this is still an unanswered query

[vmroyale]

Hello and welcome to the forums.

Note: This discussion was moved from the VMware ESX™ 4 community to the VMware vSphere™ Storage community.

Which compatibility mode are you using for these RDMs, and is it the same for all of them?

Good Luck!

[cjscol]

My understanding is that the LUN cannot be larger than the maximum size supported by the datastore the RDM pointer file is created on.  The RDM pointer file does not need to be stored on the same datastore the confguration files or other vmdk files, e.g. you could have moved the RDM pointer file to a datastore with a larger blocksize and left the vmx file and other vmdk files on the datastore with a 1MB blocksize.

Are you sure that the RDM pointer files for the running virtual machines you have are on datastores with blocksizes NOT supporting the size of the LUN they point to?

[wolfsonmicro]

We only use Physical compatibilty RDM luns

And yes, I guarantee you right now I have many machines running RDM luns with sizes in excess of the stated block size maximums.  Obviously this recent outage and comments so far on this thread make me very nervous of stability of my infrastructure.

any explanation to how it is even possible for these luns to be online?

I'll give a specific example:

our primary Exchange mailbox server has x3 850GB rdm luns attached (approx all at 80% utilised)

the .vmx and all .vmdk files - including rdm pointers - are contained on a single vmfs datastore which has a 2MB block size

[wolfsonmicro]

can this be set to unanswered please? I missclicked and would still like further feedback

[vmroyale]

Unfortunately, there is no way to change the answered value.

[wolfsonmicro]

Ok. well I now have a reply by VMware support

[quote]

As I understand from your Case Description you would like to get some clarification when it comes to the RDMs and block size limitations.

On the ESX 4.1 and older version the limitation is as noted in the link below:

Block size limitations of a VMFS datastore

http://kb.vmware.com/kb/1003565

Basically there's not much difference between the VMDK or RDM when it comes to the block size.

If your RDM is bigger than 256 GB then it needs to be in the Datastore with 2MB block size at least. If bigger than 512 GB then 4MB block size datastore etc.

If you say you have other VMs which disks are exceeding the Block size then you should move them to the Datastore with the correct Block size to prevent any issues.

When it comes to ESX 5.0 then block size of the Datastores is unified to 1 MB. Mac LUN size can be 64 TB, maximum vmdk 2 TB -512, maximum RDM disk 2TB -512 (Non pass-trough RDM) and 64TB for the passthrough RDM

[/quote]

I am happy to have a definitive answer on best practice. and cjscol was correct

My only remaining question is how my Exchange farm is still working. I will perhaps discuss this with with my Storage provider

Thanks for input. Any further comments still welcome