Adaptec 6405 poor write performance

[geforce20111014]

Hi,

My System specs are as follows:

CPUs: 2x Intel Xeon E5620

Motherboard: Intel S5520HC
Ram: 12x Kingston KVR1066D3D4R7S/4GI for a total of 48GB
USB: 4GB Kingston DataTraveler for booting ESXi 4.1
RAID Controller: Adaptec 6405 (firmware build 18301) with AFM-600 NAND FLASH MEMORY BACKUP for 6 SERIES
HDD: 4x Seagate Constellation ES SAS 1TB ST31000424SS (firmware 0006)

After suffering poor write performance the first time I went ahead and produced a fresh install of ESXi 4.1 Update 1 installed to the usb stick listed above.

I then installed the VMware ESX/ESXi 4.1 Driver CD for PMC Sierra aacraid from http://downloads.vmware.com/d/details/dt_esx41_pmc_aacraid_11728000/ZCV0YnR0aipiZColcA==

After installing and restarting ESXi I could then see the raid array and create a datastore.

Currently the raid is configured as a RAID 10 I have tried RAID 6 with the same poor write performance. Write Cache is forced on and ZMM is at optimal charge!

I have been testing performance with Crystal Disk Mark running in a Windows 2008 R2 x64 VM.

Sequential Read is 286MB/s and Sequential Write is 28MB/s.

On another system which is an i7 950 with Adaptec 5805 with BBU running ESXi 4.1 with the same hard drives but configured in a RAID6.

Sequential Read is 246MB/s and Sequential Write is 207MB/s.

I would like to get the 6405 writing atleast 5x faster than its currently operating instead of sending it off as ewaste!

Thank you in advance for any assistance you can offer me.

Update: Tried Paravirtual SCSI Controller and a vdisk on the Windows vm for a bit of fun. The test results are 325MB/s Sequential Read and 28MB/s Sequential Write.

[idle-jam]

i think it's due to the absense of BBU adaptec 6405 that caused the slow performance ..

[geforce20111014]

The BBU has been replaced by AFM-600 NAND FLASH MEMORY BACKUP for 6 SERIES which is Zero Maintainence and doesnt require battery replacement as it uses capactiors to maintain charge while the memory is written to NAND FLASH.

[znebzuga]

IT's very strange only 28mb/s...

Look this: http://hardforum.com/showthread.php?t=1611613

Ok it's different because is not across of VMware but it's not bad for 6405...

The 2 person haven't ZMCP on their controllers.

I'm not sure if you check the wrong settings but don't believe of course, so I think is a firmware bug...

I hope wich adaptec resolve this.

Do you have performance problems on windows installed directly on controller?

[geforce20111014]

I have tested the 6405 in passthrough to a Windows 2008 VM and the performance is not limited to 28MB.

[znebzuga]

Hello,

I'm thinking to upgrade my controller (mine is 3ware 9690SA 512mb cache).

So, before to buy,I like to see how this new adaptec 6000 series performs and one of this unique wich calls my attention was the ZMCP approach. Ok, I have BBU on my 3ware and is boring (like others wich use BBU's), constant monitoring, great delays to protect cache,etc.

So, accidentally, I discovered this review...oh man,,,performance is a headache on your system?

Do you recommend your adaptec 6405?

Is firmware immature yet?

I like my 3ware 9690, I know thats old tech, made for HDD's only but I have 2x Corsair SSD F120 attached on it and works more less well. Is not very quickly because firmware or controller nature is for sequencial works, not random. Although, cache speed is around 1gb/s seq read and write.

What do you think about change 3ware for Adaptec?

Regards and thanks for everything.

[Rumple]

did you check to ensure the Write back was enabled in the card bios...

[geforce20111014]

Write back cache was most certainly enabled!

I had no issue with performance when using the 6405 with pci passthrough to a 2008 R2 vm.

[amstim]

Hi Geforce,

Did you ever figure this out?

I am about to deploy a 6805 with 8x SAS 15k drives attached.  25MB/s is not going to cut it.

I've not tested yet however, like you I assumed it would work since there is a driver.....

perhaps buying 5805s is the way to go?

[geforce20111014]

Currently I have retested the card with the raid built by the 5805 and performance is better around 140MB/s write mark but currently I get over 240MB/s write with an 5805.

From talkining with Adaptec support the 6405 firmware is not as mature as the 5805 and in RAID10 senarios the 5805 is quicker from what I have been told.

Also the current driver doesnt show the device name correctly in the Storage Controllers list and its just listed as AACRAID (the name of the driver) so its still early days yet.

With 8 SAS drives you would be better off with the 5805 just to have less headache trying to figure out whats going on. Unless of course you are aquiring the hardware for a testing rig.

[amstim]

I am planning on running raid 5. So perhaps that will make a difference. Odd that the array created with the 5805 but running on the 6805 would perform better.....

Is the stripe size the same as when created with 6805? lol it really could not slow you to 28 MBPS any setting..

Damn well i guess i will give it a go with the 6805...see what happens

8x 300gig sas 15k was expecting quite alot even with raid 5.... As long as i can pull over 100 can limp till ESXi 5.0

[amstim]

Btw the device name is wrong running 4.1 using my 3405 too....so not sure if that's a great indicator

[amstim]

Ok now I am confused....

I am preping a new server for deployment to a customer, not ESXi. No VM anything just bare metal running 2008R2 (64bit). Adaptec 6405 with ZMM. 2x Cheatah 15.7 300gig 15k drives.  RAID 1.

Read speed 273 MBs, write speed 29 MBs...... sound familure.... *Sigh* at least we know now its not an issue with just ESXi.

Both drive cache and contoler cache is on, ZMM status is normal.

Nice work Adaptec.....the 6 series sure is cool, man oh man.

[DSTAVERT]

Have you confirmed that write caching is enabled?

Always unfortunate that new technology takes a while to catch up to the marketing departments claims.:smileysad:

[amstim]

Yep its on for sure.  You can easily tell by sending a test of less then 500 megs, since the controler has 512 onboard any write test under 500 megs nets about 1200 MBs.

I think ive cracked it though.

Get this.  Turning OFF the individual hard drives write cache brought the write speed up to 205 MBs

So the magic crazy formula is.

Adapter Read and Write cache ON

Hard drive cache OFF

I tried all combinations and all other then the above had 28ish MBs write cap.

Buggy Buggy.

Update/Edit:

Just for fun i tried turning the DRIVE cache back ON.  Performance is fine now..... I even power cycled to be sure.

Can't explain it...  too damn similar to Gforces problem to be a coincidence though.

[DSTAVERT]

.Write caching at the drive level would bypass the safety provided by the backup cache. Not a good idea to have it enabled. Is it turned on by default?

[amstim]

Indeed it does expose the 32 Meg or what have you on each drive to the possibility of data loss during a power interruption. The ZMM or backup battery only protects the Adapters write cache.  When in an environment with very little power interruptions I've found it is quite safe to leave enabled.  We always deploy an APC with servers too so that helps.  To be honest even though many power failures I've never really lost data having the cache on, that's with a deployment of over 150 servers.  Still I agree with you on principal is not a good idea, especial if you don't know the risks.

Having the Drive cache on however does give a  nice speed boost, in the old days of generation 1 raptors you pretty  much had to have it on or your performance was horrid, or go from raid 1  to raid 10, but that doubles your cost and failure chance.

Yes the drive cache is turned on by default. I can confirm that is the default on many Adaptec products, including 1220SA, 2405, 3405, 5405, and 6405.  They will bark at you quite a bit when creating the array however that you should probably turn it off. :smileywink:

[DSTAVERT]

32 meg is or could be a significant loss since it could be spread over several VMs.

[amstim]

Indeed and its not just 32megs.  It's 32Megs times the number of drives in the array being used for the VM or set of VMs, etc.  Also the risk goes up significantly with higher utilization of the drives or arrays.  I may get away with it due to the lazy nature of most of our application servers and that they are not VMed.

[DSTAVERT]

Forgot to add "nice find". :smileyhappy:

[znebzuga]

I'm using 3ware 9650 and 9690 raid controller. When Write back cache is enable with BBU, the drive cache is also protected by the BBU too and there is no option to disable/enable cache drive level... Simply enable/disable the both caches.

I'm interested on a 6405...

[amstim]

Update.......boo.

The 6405 write speed is performing poorly again, no amount of toggling cache setting is restore proper speed.....

I just switched my pending order for two 6805s for two 5805Zs.  I don't need this kind of aggravation on my ESXi boxes...

znebzuga:

I've never heard of BBU powering drive cache before....sounds fishy! :smileymischief:

[znebzuga]

Yeah but 3ware's do this. However my 3ware's are slowly for SSD's.

I ordered a 6405 + ZMCP module. I think it will be ok with cache drive OFF (since my SSD's don't have cache) and turnn on the controller cache...

Will bo ok ,don't you?

[amstim]

Normally I would say yes, but now my test machine running 2008 server with no VMware or ESX whatsoever is having trouble getting more the 28 MBs.....

Right now I am hoping desperately that its just Crystal Disk mark that is messed up with the Adaptec 6 series.  Installing more test tools now.  I will update when I know more.

[znebzuga]

Oh man...you are worring me... no matter what setting you use , can't do more than 28mb/s sequencial write? Even with cache drive off and controller cache on?

[znebzuga]

Try SISandra software and Winsat disk...

[geforce20111014]

Before I posted on here I did have the one occassion where the 6405 did perform without the write lock in esxi but I have never seen it happen more than once.

I have benched the performance with a freebsd vm with dd and that showed similar to seq write in crystal.

I havent herd anymore from Adaptec yet.

[znebzuga]

Geforce:

Tell me a thing. Did you perform controller in sequencial write below and above 512mb of data? Did it show the same 28mb/s? I mean, on Crystal disk mark without VM or ESXI, just on windows...

[amstim]

I went ahead and deployed the 6405 in the customer server.  It was performing inconsistently, but it was "not that bad".  I expect Adaptec will be coming out with an update shortly.

I also have reworked my ESXi setup and am using a 6805 with 8x 300 Gig SAS Cheetah 15.7.   I never encountered the 28MB/s issue but performance was a tad erratic, but always acceptable.

RAW Benches from a 2008 Machine, NO VM.  RAID6, 256k stripe. Onboard HDD Cache OFF

In a 2003 VM, RAID 6, 256k stripe.  This is with 11 other VMs running. Onboard HDD Cache OFF

Below is a test with a 50meg data set.  Same VM as above same RAID 6, 256k stripe. Onboard HDD Cache OFF. This reflects the nice speed boosts you get from the 512 meg controller cache.

In short to sum up.  I think the 6 series cards need a firmware update, they do perform a bit wacky.  Choosing to deploy them or not ATM will depend on your specific needs.  I think once the bugs are worked out the 6 series is going to be a very good card.  The 512 megs of cache is really nice.

[znebzuga]

Hello:

The test of 50Mb file is very strange. All 4k random on this test shows the cache speed but Sequencial's not. 50 and 100mb file lenght should show numbers around of 2000mb/s (cache speed). Yes, definetly Adaptech has to build a new firmware as soon as possible.

1000mb test shows as expected I think... but 500mb looks strange...

[amstim]

Actually I think the 500 meg test is just fine.  Remember that with the other VMs running the controller cache is not dedicated to the one VM running the tests.

When running on the RAW 2008 machine tests of 500 megs were registering about 1200 MB/s as expected.  Never saw 2000 MB/s on any tests.

As I said though, it needs a firmware update for sure.

[znebzuga]

I would say 2000mb/s in seqeuncial read at 50 and 100mb test file.

At 500mb of course is nearly out of cache and not show this result.

Although, 1200mb/s on 500mb test is very good :smileyhappy:

Here is my 3ware 9690SA-8i 512mb DDR2 533 cache + BBU with 2x Corsair F120 SSD.

50 and 100mb test shows the cache in every nature of test.

1000mb test shows the weakness of this controller, doesn't scale so well since each SSD does +/- 250mb/s read and write, so less than 400mb/s seq is bad, and even worse is the seq write, 2x in raid 0 shows the half speed of ONE drive...I know, 3ware 9600 series was not made for SSD's.

And as you can see, 4k random is pratically nothing comparing to the new controllers like an Adaptec 6805...

[znebzuga]

Amstim:

I purchased my 6405 + ZMM today.

I've been looking for BIOS of controller and I need your help please...

1- Where is Read Ahead Cache that you talked when you found the magic formula for unlock 28mb/s...I just see Enable/disable Read only.

I let most setting on default:

HDD drive cache in Enable All

Controller cache in Enable with ZMM

Performance on dynamic

Background task in High

This is right for the optimal  performance?

Regards

[amstim]

The read cache should be a property of the array just like write cache.  I believe the setting is right above or below the setting for write cache.  I don't have a server here that I can check on because they are all in production.

As I said I am my other post I don't think there is a "magic formula".  I concluded that the card really needs a firmware update, and that the only reason that I ended up deploying it was that at least with my 8 drive setup I never saw performance issues that were bad enough to halt my project.  That being said the performance numbers did vary from test to test more then I thought they should and I expect this will be sorted out with firmware patches.

[znebzuga]

Hello :smileyhappy:

Well I'm playing with my Adaptec 6405, I'm satisfied with performance.

Yes, it needs new firmware, especially to make a better job on cache support for 4k linear read.

4k random I think it's fine. This card is speedy thanks to powerfull dual core cpu. It's away faster than my old 3ware 9690...

4GB SLC NAND is a must not even on power failure but in a global system performance gain too.

[znebzuga]

Hello:)

Good news! The tech support guy of Adaptec told me that we have a new release/firmware for 6405/6805 up to 2 - 4 weeks:)

Let's see if this unlocks the potential of this cards. Stay tune

[znebzuga]

New firmware build 18668 is out.

Performance seems to be the same, but there is stability enhancements.

New performance mode - big block bypass. Uses all the cache for small blocks reads.

[geforce20111014]

I have been incontact with Adaptec support in regards to this.
The 6405 is currently no where near as mature as the 5 series and also the 6 series is not supported in ESXi until 5.0 so the 4.1 driver which has been posted is currently a mystery.

I'm still waiting to hear more on the progress of this.