Just to put in my two cents...
I'm trying to build a home lab that's suitable for setting up multi-VM test environments for Red Hat Enterprise Linux (5.x) and current Microsoft OS/application environments (Windows 2008 R2, Exchange 2007/2010, SQL 2008, etc.). That means I need tons of RAM and the ability to virtualize x64 workloads.
I'd like to build something with these specs:
AMD x64 quad-core processor with full virtualization support (Phenom II or some sort of Athlon)
8 to 16GB of RAM (the more, the better, especially working with newer-gen MS apps). To get 16GB RAM at a reasonable price, I'd need motherboard with 8 RAM slots. 4GB DIMMs still cost more than twice as much as 2GB DIMMs.
I've been trying to do the same thing for ESX3.5 & 4 for the past year. The main requirement you list that almost certainly puts you over the budget is the 8 DIMM slots. Usually only server-grade motherboards have more than 4 DIMM slots. I've been able to get a total price under $500 but not with 8 DIMM slots. The totals in that case start at $1500. This assumes you don't have the RAM already and have to buy it.
- Support for onboard SATA disk controller (non-RAID is OK)....
Many of the onboard SATA controller work, but to get RAID, most of the low-end motherboards only have software RAID, which won't work with ESX. But a decent 4 port SATA/SAS RAID card is under $500. So not too bad there.
...and NICs. If I have to add PCI NICs, what's a cheap 1Gbps card that works with ESXi 4?
Server motherboards tend to come with Broadcom or Intel NICs so are more likely to work. Non-Intel desktop boards are often Realtek and often do not work. But you can get Intel MT or PT gigabit NICs for under $50. If you're limited on PCI(e) slots, you'll want to get dual or quad port NICs, but they usually start in the $300+ range.
- basic VGA or DVI video (onboard is OK)
Almost any onboard video will work. But if yours doesn't have video, then possibly even a old PCI video card may work. I was able to find a ATI PCIe video card for $20. I got that one because it didn't have a fan, just a heatsink. One less thing to fail.
...I already have a case, PS, and SATA disk drives to complete the build. Not looking to set benchmark records but want something with some horsepower on the cheap.
Looked at the White Box HCL but there seem to be issues with every AMD motherboard, and it doesn't look like anyone has tested (or gotten working) the AMD 790 chipset-based boards.
Any suggestions on a working combo? I'd like to keep the budget <$750 US....
I've been noticing that the Q6000 series of Intel quad-cores can be had at a reasonable price. Not the CPUs by themselves, but in a older HP or Dell workstation. Systems like that are going for $350-$500, but only have 1-2GB of RAM, so you'll have to add that to the total cost.
I had a whitebox with a Intel i35 based motherboard and a Q6600 2.4GHz Core2quad and 8GB RAM. ESX recognized the NIC and SATA, but of course, the onboard RAID was ICH9 software based, so I left it with VMWare Server2 until I could get a RAID card. Sadly, the motherboard died, so I'm having to shop for a new system as well.
...Or should I just build a 'normal' desktop and run Workstation on top of Windows 7? I'd prefer to run ESXi on a dedicated box if it's feasible.
If you're going to run more than 3 VMs simultaneously, I'd recommend ESX. My above i35 quadcore system used Windows Server2003 as a host OS and would get noticably slower when I ran 5 VMs. I still had free RAM and CPU was less than 20% total. It was still perfectly functional, but doing anything on the host console required some patience.
If you can up your budget, then you likely could get all that you listed. But now you're approaching the price of a pre-built server. So if you can increase your budget, then do as suggested and look for deals from HP and Dell, etc.