Very slow write speeds on Perc H700 and RAID5 5x2TB drives

[mephistopoa]

Hi guys,

I've been playing out with a new dell R515 server here with a H700 controller that has 1GB cache. The RAID10 container with 4x 600GB 15K RPM works quite well, and on the same controller I've a 5x2TB raid 5 array. I know the disks are nearline SAS and bla bla bla, but I'm not getting any better than 45MB/sec doing sequencial writes on it. I've been playing with write cache inside the Windows 2008R2 guest and made some differences so sometimes without write cache enabled on the OS the writes goes for some time at 100MB/sec, sometimes until the cache flushes completely but sometimes it goes for quite a long time, like 1 minute or more than again drops to 40-45MB/sec.

I know these drives alone can easily do writes at +100MB/sec eaisly, so 5 of them even on RAID5 with a good controller and lots of cache surely should be able to saturate a gigabit network. I'm a bit frustrated with it :/

Is there anything at the ESXi level that I can do? Write back is enabled on the controller for this array.

Regards,

Alex

[mephistopoa]

Other thing, is it a good procedure to disable write cache on the guest OS?

[AndreTheGiant]

Check in the RAID card BIOS if the cache is enabled and if is write back (of course if you have the internal battery).

[mephistopoa]

Write cashe is enabled, it is a perc H700 (LSI whatever, some of good controller) with 1GB of NV cache and battery backed

I'm really strugling to understand where is the bottleneck on this :/

[J1mbo]

I'd get this one over to Dell tech, their Perc support team is usually very helpful.  From the numbers it looks to me that write caching is enabld, otherwise sequential write performance I'd expect to be sub 10MB/s.

As an aside though I would personally advise against RAID-5 for any kind of vSphere project.

[mephistopoa]

Normally with crappy drives and crappy controllers without write cache enable, I expect always around 35MB sec on sequential writes, as this array is a large storage for backups there is no better option now than raid 5 with hot spare, especially with skyrocketing prices od disks due to the floods in thailand.

I'm wondering to contact Dell, there is nothing that I can see wrong so far on the set up :/

[J1mbo]

Might be worth clarifying exactly what the nature of the backups will be.  If this is deduped storage for example with Veeam..., RAID-5 may well be a problem.

[mephistopoa]

Hi there,

no dedupe involved, it is a online backup system from Ahsay, but I'm testing at network share level as the array store some other things that I need to access eventually, but mainly sequencial writes. The raid10 array on the same server saturates the gigabit from the beggining to end of the disk space

[ranjitcool]

Hello,

Like the gurus here pointed out it could be the raid cache - however have you noticed this happen all the time? I am pressuming that you have -  if it were at a set time - it could be that the raid battery might be power cycling which slows down the disk as well.

RJ

[mephistopoa]

It is a constant problem, the performance briefly goes up when the cache empties and get full again, during this period the write speeds saturage gbit speeds bit still a few seconds later the cache flushes and performance back again to 45MB sec. The battery is in good condition, the server is a brand new Dell R515, has 3 weeks of life and I can see on the hardware tab the raid controller is not doing any sync or anything else, all green including the battery backed cache.

Thanks

[Fredp1]

Did you every get to the bottom of the speed issue?

When testing a h700 1gb ram, with 2 arrays on the one controller, the speed drops to 45mb/s.

If I use 2 controllers, one for each array, the speed is great.

The 2 arrays are made of 2 disks in a raid 1 and 3 disks in a raid 5. All drives are Sata.

[mephistopoa]

Speeds are very inconsistent. Sometimes I can get 90MB sec, several times it doesn't go beyond 45MB

Never had a chance to use windows baremetal on this server to check for any difference.

Still in the same way as before unforntunatelly.

I've just added a 120GB intel SSD as a cache device now

[Fredp1]

My machine is a simple home server 2011 bare metal machine. (Server 2008r2)

I've added a ssd as cache Cade... but haven't done any significant testing to see if its working much faster.

Initial benchmarking with CrystalMark shows a slowdown of the cachecade array.

Keeping 1 array per controller card increases the speed of each array.

[SoloIT]

Has anyone gotten to the bottom of this? I'm having a very similar problem with a similar configuration. Dell tells me they do not troubleshoot performance issues.

Thanks,

George

[AndreTheGiant]

Check if you have the backup battery.

And then check that write back is enable on the logical volume.

[SoloIT]

The card reports the battery is OK and it's only 1 year old. Is there a better way to check it?

Yes, write back is enabled.

[AndreTheGiant]

Be also sure to have latest PERC firmware.

But also remember that RAID5 on SATA disk could be poor (in write operation) depending on the load.

If you need to have a lot of writes consider change to RAID10.

[SoloIT]

I just upgraded the firmware last night and the problem happened again after that. The really odd part is my issue is sporadic. Typically the array runns 200 MB/s reads and about 150 MB/s writes. However, when it's acting up, it drops to 50 read and like 20 write. I just cannot seem to find the issue.

[J1mbo]

If it's battery-backed cache, the batteries have a learn cycle every two weeks, and during this the array switches to write-through I think.  Have a look at the storage latency in both cases.  With write-back caching, write load may show up as read latency (since the disks are busy, behind the scenes).

[SoloIT]

The disks are busy as the latency is between 25-28 ms during the "bad' times. However, the issue is more frequent than every 2 weeks and has lasted for more than 24 hours. I think the learn cycle is quicker than 24 hours, right?

[Fredp1]

It's been a while since I last work on the Perc, but when I had more than one array, the speed was much slower for each array.

i.e. I had a mirror for the o/s and 5 x 2Tb raid 5 array on the one controller, all running Sata drives. When I moved one array to a second controller, the speed was much faster.

Also check your arrant isn't performing some consistency check as that will slow it down.

[SoloIT]

Thanks for the suggestion, but only 1 array and the status is showing clean. No consistency checks I can see running.