Content Library and vSAN

[offsidex]

Hello All,

I know that VMware supports the creation of content libraries with a vSAN datastore as backing, however I am curious to know how the storage policies would interact with the library. My assumption here is that it would be the same as any object  however I have questions like: Is the whole content library considered to be one object? How would vSAN handle big content librarues etc...

I have looks around for some documentation about this but was unable to find any.  Can someone please point me into the right direction?

[TheBobkin]

Hello offsidex​,

"My assumption here is that it would be the same as any object  however I have questions like: Is the whole content library considered to be one object? How would vSAN handle big content librarues etc..."

Content Libraries stored on vsanDatastore appear to consist of just a namespace Object (for each library) with the data stored inside it (in sub-directories) as non-Object file data - this implies that the max-size of a Content Library stored on vsanDatastore would be 255GB (the namespace Object max size) - yes, I am aware the only external documentation we have regarding max sizes of Content Libraries is 1TB max file size, I will be getting in touch with our Documentation team regarding the implied caveat for vsanDatastore here if this is indeed the case.

"I am curious to know how the storage policies would interact with the library"

Content Libraries are stored on vsanDatastore as namespace Objects which appear to have plain FTT=1 FTM=RAID1 attributes - no implicit spbmProfileName applied, this means you can't change them by changing a SP (Storage Policy).

This doesn't mean these Objects won't be repaired etc. when in reduced availability, just that they are limited from an spbm management perspective.

I tinkered with it a bit and found that there is no effect on how these are created by changing to FTT=0 the vsanDatastore default SP, changing the node-level default policy attributes for cluster and namespaces to FTT=0 etc. so it looks like it must be using some other default of its own.

Using objtool setPolicy to change the attributes of the Objects does of course work though.

I (sadly!) don't have a lab with 255GB SSD free at the moment (HDD slow writing data) to test the effects of this but I would imagine is same as any namespace and can't write past 255GB.

Bob

[TheBobkin]

Hello James,

Thanks for testing this - not had a chance to test further myself due to holidays.

Can you share the RVC vsan.object_info for the namespace Object of that Content Library and also the CMMDS info e.g.:

# cmmds-tool find -t DOM_OBJECT -f JSON -u <namespace-UUID>

(The namespace-UUID will be the last portion of the directory path when you cd into the namespace folder.)

What version of vCenter and ESXi in use?

Bob

[tiffaj0333]

Hi Bob

Thanks for the quick response

The vCenter is running at 8024368 and the ESXi servers 7967591

The output of the command you requested is below

cmmds-tool find -t DOM_OBJECT -f json -u 02a2a85b-be30-c0c1-88bf-5cb90188c94c

{

"entries":

[

{

   "uuid": "02a2a85b-be30-c0c1-88bf-5cb90188c94c",

   "owner": "5aa567bd-de64-bf3d-c566-5cb901907d30",

   "health": "Healthy",

   "revision": "27",

   "type": "DOM_OBJECT",

   "flag": "2",

   "minHostVersion": "3",

   "md5sum": "b1af536217b82677803e12c41833f1b3",

   "valueLen": "1416",

   "content": {"type": "Configuration", "attributes": {"CSN": 8, "addressSpace": 273804165120, "scrubStartTime": 1537778178946757, "objectVersion": 5, "highestDiskVersion": 5, "muxGroup": 362030135894980956, "groupUuid": "02a2a85b-be30-c0c1-88bf-5cb90188c94c", "compositeUuid": "02a2a85b-be30-c0c1-88bf-5cb90188c94c", "objClass": 2}, "child-1": {"type": "RAID_1", "attributes": {"scope": 3}, "child-1": {"type": "Component", "attributes": {"addressSpace": 273804165120, "componentState": 5, "componentStateTS": 1537778178, "faultDomainId": "5aa567f7-b310-3313-f6c8-5cb90188c94c", "subFaultDomainId": "5aa567f7-b310-3313-f6c8-5cb90188c94c", "objClass": 2}, "componentUuid": "02a2a85b-fefc-22c2-8a7e-5cb90188c94c", "diskUuid": "52691ff2-7c0c-a75e-0fc9-6259a754848c"}, "child-2": {"type": "Component", "attributes": {"addressSpace": 273804165120, "componentState": 5, "componentStateTS": 1537778178, "faultDomainId": "5aa567ef-2802-1dcd-f525-5cb90188cb18", "subFaultDomainId": "5aa567ef-2802-1dcd-f525-5cb90188cb18", "objClass": 2}, "componentUuid": "02a2a85b-9e73-24c2-26b2-5cb90188c94c", "diskUuid": "525df39a-c0c3-fa6b-b944-c6165a573db4"}}, "child-2": {"type": "Witness", "attributes": {"componentState": 5, "componentStateTS": 1537778178, "isWitness": 1, "faultDomainId": "5aa567bd-de64-bf3d-c566-5cb901907d30", "subFaultDomainId": "5aa567bd-de64-bf3d-c566-5cb901907d30"}, "componentUuid": "02a2a85b-1c58-25c2-431d-5cb90188c94c", "diskUuid": "521f8574-2916-6e4d-f162-81638777dc1d"}},

   "errorStr": "(null)"

}

]

}

[TheBobkin]

Hello James,

Could you provide the info of the Object (02a2a85b-be30-c0c1-88bf-5cb90188c94c) from RVC?

I ask as I forgot that namespaces will always show as 'addressSpace' of 255GB regardless of 'physUsed' - the vsan.object_info should show this or alternatively you can check this from an ESXI host directly by running one of the bundled support scripts, writing it to a file and then looking for the component/layout entry for the Object:

# python /usr/lib/vmware/vsan/bin/vsan-health-status.pyc > /tmp/health_24-09-2018.txt

# less /tmp/health_24-09-2018.txt

Press:'ESC' '/' '02a2a85b-be30-c0c1-88bf-5cb90188c94c'

Press 'n' for next until you get the layout which will appear like so:

Object 46b09b5b-8a4c-dd90-fedf-000c2908a466 (v5, owner: notmyhost.notmydomain, policy: {'SCSN': 28, 'hostFailuresToTolerate': 1, 'CSN': 30}):

  Configuration

    RAID_1

      Component: 46b09b5b-ce76-7c93-2f2a-000c2908a466 (state = 5, addr space = 273804165120 (255.00GB), disk = 5248eeeb-d7cd-d5fd-39bd-f8d67bc02356,

                           votes = 1, used = 5406457856 (5.04GB), physUsed = 5351931904 (4.98GB), hostname = notmyhost.notmydomain)

      Component: 46b09b5b-f417-8093-8599-000c2908a466 (state = 5, addr space = 273804165120 (255.00GB), disk = 527d07c3-6946-fae7-872c-6a2a734b48e6,

                           votes = 1, used = 5406457856 (5.04GB), physUsed = 5351931904 (4.98GB), hostname = notmyhost.notmydomain)

    Witness: 140e9c5b-bba5-4278-8763-000c2908a466 (state = 5, addr space = 0 (0.00GB), disk = 528ff29f-13ea-e4a1-19ae-02ea795113db,

                         votes = 1, used = 12582912 (0.01GB), physUsed = 4194304 (0.00GB), hostname = notmyhost.notmydomain)

Just a brief tip - I would advise when taking screenshots of your environment uploaded to public websites to eraser/marker any potentially sensitive details such as valid usernames and/or domains as you have there.

Bob

[tiffaj0333]

Sorry for the delay

Thanks for pointing that out i put the wrong screen shot in whilst trying to multi task unsuccessfully :smileyhappy:

cmmds-tool find -t DOM_OBJECT -f json -u 02a2a85b-be30-c0c1-88bf-5cb90188c94c

{

"entries":

[

{

   "uuid": "02a2a85b-be30-c0c1-88bf-5cb90188c94c",

   "owner": "5aa567bd-de64-bf3d-c566-5cb901907d30",

   "health": "Healthy",

   "revision": "27",

   "type": "DOM_OBJECT",

   "flag": "2",

   "minHostVersion": "3",

   "md5sum": "b1af536217b82677803e12c41833f1b3",

   "valueLen": "1416",

   "content": {"type": "Configuration", "attributes": {"CSN": 8, "addressSpace": 273804165120, "scrubStartTime": 1537778178946757, "objectVersion": 5, "highestDiskVersion": 5, "muxGroup": 362030135894980956, "groupUuid": "02a2a85b-be30-c0c1-88bf-5cb90188c94c", "compositeUuid": "02a2a85b-be30-c0c1-88bf-5cb90188c94c", "objClass": 2}, "child-1": {"type": "RAID_1", "attributes": {"scope": 3}, "child-1": {"type": "Component", "attributes": {"addressSpace": 273804165120, "componentState": 5, "componentStateTS": 1537778178, "faultDomainId": "5aa567f7-b310-3313-f6c8-5cb90188c94c", "subFaultDomainId": "5aa567f7-b310-3313-f6c8-5cb90188c94c", "objClass": 2}, "componentUuid": "02a2a85b-fefc-22c2-8a7e-5cb90188c94c", "diskUuid": "52691ff2-7c0c-a75e-0fc9-6259a754848c"}, "child-2": {"type": "Component", "attributes": {"addressSpace": 273804165120, "componentState": 5, "componentStateTS": 1537778178, "faultDomainId": "5aa567ef-2802-1dcd-f525-5cb90188cb18", "subFaultDomainId": "5aa567ef-2802-1dcd-f525-5cb90188cb18", "objClass": 2}, "componentUuid": "02a2a85b-9e73-24c2-26b2-5cb90188c94c", "diskUuid": "525df39a-c0c3-fa6b-b944-c6165a573db4"}}, "child-2": {"type": "Witness", "attributes": {"componentState": 5, "componentStateTS": 1537778178, "isWitness": 1, "faultDomainId": "5aa567bd-de64-bf3d-c566-5cb901907d30", "subFaultDomainId": "5aa567bd-de64-bf3d-c566-5cb901907d30"}, "componentUuid": "02a2a85b-1c58-25c2-431d-5cb90188c94c", "diskUuid": "521f8574-2916-6e4d-f162-81638777dc1d"}},

   "errorStr": "(null)"

}

]

}

i think this is the other part you asked for

Object 02a2a85b-be30-c0c1-88bf-5cb90188c94c (v5, owner: 918347-hyp9.mv.rackspace.com, policy: {'CSN': 8, 'hostFailuresToTolerate': 1}):

  Configuration

    RAID_1

      Component: 02a2a85b-fefc-22c2-8a7e-5cb90188c94c (state = 5, addr space = 273804165120 (255.00GB), disk = 52691ff2-7c0c-a75e-0fc9-6259a754848c,

                           votes = 1, used = 419430400 (0.39GB), physUsed = 415236096 (0.39GB), hostname = 918350-hyp12.mv.rackspace.com)

      Component: 02a2a85b-9e73-24c2-26b2-5cb90188c94c (state = 5, addr space = 273804165120 (255.00GB), disk = 525df39a-c0c3-fa6b-b944-c6165a573db4,

                           votes = 1, used = 419430400 (0.39GB), physUsed = 415236096 (0.39GB), hostname = 918348-hyp10.mv.rackspace.com)

    Witness: 02a2a85b-1c58-25c2-431d-5cb90188c94c (state = 5, addr space = 0 (0.00GB), disk = 521f8574-2916-6e4d-f162-81638777dc1d,

                         votes = 1, used = 12582912 (0.01GB), physUsed = 4194304 (0.00GB), hostname = 918347-hyp9.mv.rackspace.com)

[TheBobkin]

Hello James,

"physUsed = 415236096 (0.39GB)"

You removed the outline of what you had placed on this Content Library - are you sure for sure it wasn't MB not GB sizes for the OVAs?

I ask as this ~390MB used per data-mirror. If you look in my previous comment that is what a Content Library Object looked like with ~4.98GB (per data-mirror), so either the size of what you uploaded is incorrect or how these are placed has changed in 6.7 (I tested in 6.6) - if the sizes are correct could you try and figure out where/how these are being stored? (e.g. check the du -h in each sub-directory under the Content Library namespace, anything that is too small to be the data it holds may be a descriptor and as such should be a text file containing a vsan Object path).

Bob

[hharold_sch]

Hello ,

Just wondering if you have a reference for this statement:

---

..yes, I am aware the only external documentation we have regarding max sizes of Content Libraries is 1TB max file size...

---

I think we are hitting this limit, but I would like to point to any form of documentation to state my case

Cheers,
Harold

[TheBobkin]

Hello Harold,

 

This used to be mentioned in the docs.vmware.com vSAN documentation regarding Content Libraries but it is no longer referenced there as this has been long ago re-structured (it is no longer just a namespace with flat data in it but with individual objects for sub-entities).

What ESXi version and build are you using here?

 

Maybe you can elaborate a bit more on the specifics of your issue in a new post to see if more can be gleaned.

[Joseph Cooper]

I'm testing this with a RAID5 policy on ESA and I can't get a content library to exceed 255GB.  I'm running ESXi 8.0U2 and vCenter 8.0U2a. Uploading files into a content library that would force the CL beyond 255GB fails with error

A general system error occurred: The import of library item ee5479a7-747c-4456-b705-f6fa50567419 has failed. Reason: Error transferring file temp3 - Copy (3).mkv to ds:///vmfs/volumes/vsan:5296cc3dce4fd944-380bcf84365f4681//contentlib-3c0d408d-4487-49d8-a96e-75ae910d4ae5/ee5479a7-747c-4456-b705-f6fa50567419/temp3%20-%20Copy%20(3)_3a4f5e02-6f48-489c-970b-5c53fba56c0e.mkv?serverId=7c3277fe-6586-420c-be68-5e0bd8448b8a. Reason: NFC Error 1/14: There is no disk space available -- Failed to write to the target file: There is no space left on the device.

[TheBobkin]

, Is the individual data (e.g. OVA, OVF) stored there being backed by vSAN objects or being stored as raw data?
It probably is just getting stored as raw data and thus it maxing out at 255GB is expected behaviour, the likely reason being that you are trying to move data there that isn't a Content Library supported file format (.mkv), content library doesn't know what that is and isn't going to make an object to back it.

 

 I would also advising opening a new thread for such queries as this doesn't look to have anything to do with the original (5yr old!) topic.

[Joseph Cooper]

I can do that, Bob... both open a new thread and test with an OVA.  One thing to note is that in vSAN 8U1 there is a new API (and supporting CLI tools) for expanding the namespace beyond 255GB.  There's a slide floating around (I just discovered) that mentioned Content Libraries as a use case.

[OsburnM]

I know this is a bit of an older thread but I recently came across a situation where I have a content library on one of our vsan clusters and all its objects (templates mainly) are using an incorrect vsan storage policy.  Are you able to go back and change the policy on these kinds of objects?  I have all the VM objects changed to their correct one but can see how to change the CL objects policies.