Hi guys
I'm selling my old desktop this week, which served me well for eight years, and I'm now looking at replacing it with an ESXi server.
The goal would be to run one or two linux virtual machines for downloads, network services and monitoring, and a lot of lab machines in separate VLANs which will connect to physical routers, firewalls or virtual appliances. Because I don't want to deal with noisy enterprise servers and high power costs I'll be taking the whitebox approach.
Has anyone here built their own ESXi with consumer hardware? I have been doing some reading this weekend but still have a lot of questions before I can start ordering hardware
- Intel or AMD: AMD looks like the cheapest solution. I've been looking at this build (http://www.virtualvcp.com/news/199-new-esxi-whitebox-servers-for-home-lab) but with an AMD FX-8350 CPU (8 core, 4 Ghz).
- RAM: it's the most expensive, so would 16GB suffice or better to just bite the bullet and go with 32GB at once
- Storage: what's the recommended approach for storage? I would like to have some redundancy so was thinking of two disks in a RAID1 configuration.
Another option would be to run iSCSI to my FreeNAS (two 2TB disks in a mirrored ZFS volume) although I have zero experience with that, and the NAS only has one NIC. - Networking: I believe most people use quad-port Intel NICs on their ESX's. Can anyone recommend an affordable card which is compatible with the latest ESXi build? Will I be able to do LACP and dot1q with the free version of ESXi?
All advice very much appreciated! :matrix:
I didnt do a whitebox but here is a good write up that you might find valuable.
http://ethancbanks.com/2014/03/15/my-home-lab-esxi-5-5-server-build-and-the-logic-behind-it-all/
Myself I did a Dell T7500 w/ 64 gig RAM and dual Hex-Core Xeons from ebay. By far a ton cheaper and this guy runs quieter than my laptop! Less than $900 put into the server to date.
Thanks, I just checked eBay and prices for that tower are between €500 and €1000 with 12GB of RAM, which is not too bad but you do have to factor in there's only 30 days warranty with most hardware brokers, compared to two years on new consumer hardware.
How does your storage look like? Are you using local disks or running over the network?
Its all local for now. I want to get a synology bad and migrate everything over but that will be a while.
Currently I have a 2TB RAID 5 internal on the built in Perci6. ESXi is installed on a 32GB SSD.
RAID came from my old T110 I purchased a while back for my CCIEv4 labbing.
Suppose I go with one disk alone to save on HDD cost, is there any way-with the free version- to take snapshots and dump them on my NAS?
I've been reading up on RAID options with local storage, the following article sums it up quite clearly.
http://www.packetmischief.ca/2011/03/20/choosing-a-raid-card-for-esxi/
Looking at the pricetags for true RAID cards, I think I'll just roll with a single disk for the moment or try the FreeNAS option :mrgreen:
Yeah I would have skipped the RAID option if the card was not built in or I had not been given one from my old job.
Snapshots should stay with the VM and its files as a snap-shot file alone is useless if I remember correctly. Im not sure if you can attach a NAS as storage within ESX. If so then just store the whole VM there. But yeah simple would be local storage.
My NAS is attached as storage to my ESX box although it is Synology. I'm not sure what FreeNAS has but I assume somewhat the same. Just connect them up with iSCSI and you're good.
One thing I have noticed is that CUCM, CUC, etc. error out on install when the target is on my NAS. I've found that when I run into this issue it's easier to install local and then move it over after install.
My T110 is maxed out on RAM and I end up having to shut things down to do other labs. I too am looking into options but haven't yet decided what I'm going to do. Keep us posted on what you decide to go with.
Ordered the hardware on Thursday. Everything was in stock except for the case :doh:
Oh well, gives me time to do some paperwork over the weekend...
I ordered a new Motherboard, CPU's, and some RAM this morning. I still need to get a new power supply. I should have everything else to put mine together. I think I'm more excited to use my T110 as a desktop than I am about the new server. My laptop doesn't have much left in it and I'm sitting at a desk most of the time anyway. :banana:
DO NOT TOUCH AMD there are various issues with various ESXi features.
Always go Intel NICs, no question.
iSCSI is fine for labbing but production wise (assuming 1Gb network) you'll notice the speed difference quite dramatically, I use iSCSI only for test hosts not for 'real' machines. Recommend 2 fat SSDs in RAID1 for your main storage, but feel free to run up lab VMs (or even non disk IO stuff like routers) on iSCSI. Whatever you do, run it on a separate network (even a 50 dollar dlink which is what I use is fine) on separate NICs etc. There is a lot of tuning (on the target side) + 10Gb NICs required for you to get comparable performance from iSCSI as from native or FC/FCOE....
You have 3 good options in my mind.
1.) Intel Xeon-E3 build. Pros: cheapest. Cons: maxes out @ 32Gb, thanks a lot Intel. Recommend you stick with server grade mobos and ECC RAM though if you want to cheap out you can stick Xeon E3s into a lot of consumer chipsets and it does work. With the right mobo you will get enterprise features too like IPMI / iLO / iDRAC etc., dual intel NICs, etc.
2.) Sandy/Ivy/Haswell-EP build (i.e. the X series CPUs). Pros: lets you go up to 64gb. Cons: expensive.
3.) Second hand server as what that1guy says. Pros: cheap for what you get. Cons: warranty, old gear, maybe loud and heavy depending on what exactly you get (e.g. a 1RU server will be hella loud), may kill your power bill.
You also want to have a think about whether you want the one megabox of doom approach or multiple smaller hosts. The former is better for ease/general use, the latter for labbing anything vmware or virt specific (as then you have multiple real hosts).
Quote from: wintermute000 on March 28, 2015, 06:48:26 PM
DO NOT TOUCH AMD there are various issues with various ESXi features.
This is good info to remember too. I looked at AMD in my research for a new system but decided on Intel due to compatibility issues with AMD and Cisco VoIP. Who knows what else would have had issues if I went that route. For the money spent I'd rather be safe than sorry.
The hardware I ordered was AMD-based, but I based it on a couple of builds from VCP blogs & forums so I guess it should be okay. I was first looking at the Xeon build from Mellowd's site but the price difference was just too much for a lab box. I Also ordered 2 x Intel Pro PT1000 Dual port NICs.
If it turns out to be a huge PITA I can always use KVM or turn it into a more than decent desktop, or even sell it as a new desktop.
protip: if going whitebox, make sure you get a box with intel vPro. Basically turns your built-in mobo NIC into a ghetto bmc/ilo/iDRAC (i.e. way of remotely turning the machine off and on). I don't actually use the mobo NICs on my servers (long story, lets just say be careful with community hacked slipstreamed drivers) but I do have them patched in just for this very reason alone, vsphere not recognising them doesn't matter when the vPRO is at the BIOS level.
Its nice to be able to RDP into your jumppost and remotely power off/on your extra lab servers.
Also, a cheap fascimile of a VPN gateway if you don't fancy ponying up for a SRX110 or C8xx or running an pfsense instance on your ESXi is a simple linux VM with SSH NAT. Use port tunnelling to tunnel RDP for a secure way to access your jumppost without anything fancier than putty.exe. You can also secure SSH on linux pretty well with things like fail2ban. In my case I have cable and don't want to deal with the hassles of bridging (and then explaining to the wife how there are 'two modems').
Just spent the last three hours installing ESXi to my USB stick. For some reason everything kept crashing during install or boot. Turns out it doesn't like working on an USB3 port. Switching to an USB2 port fixed it :doh:
I was already blaming AMD lol
There has been lots of issues surrounding usb3. From WiFi interference to hardware not working/drives corrupting. I know I have issues with it recognizing my cellphone. Sometimes it will pick it up and run data, other times my phone will see it as ac and shut down all data.
The new supermicro X10sdv-tln4f is on my wishlist BTW. Have to buy a house first.... But next big buy will be this or a sister board. It will be a killer VM box. 128gb ram, 2x 10Gbe, 2x 1Gbe, 1xIPMI, 8 core, 16 thread 2ghz. All in a <90w mobo (under load!).
:drool:
That's a lot for such a small board. I like it. I imagine to max the RAM out on that guy it'll be expensive for 4x32GB sticks. But the power and noise savings would be awesome!
I ended up buying a Super Micro X8DTH-iF. I've got 48GB (3x16GB) of RAM for it so far. Been buying here and there, as I find deals it increases. I've got a matched pair of dual 6-core Xeon's. Been trying to find a good deal on a tower to fit this thing and an efficient power supply. I'll be running ESXi on a USB stick and storing everything on my Synology. My T110 has been good to me but this has the compute I need to do bigger labs and still run other VM's.
Quote from: routerdork on April 16, 2015, 02:15:22 PM
That's a lot for such a small board. I like it. I imagine to max the RAM out on that guy it'll be expensive for 4x32GB sticks. But the power and noise savings would be awesome!
Yep. Some people are claiming $800-900 USD is a bit much. Apparently they have never pieced these components separately. 10Gbe alone is a few hundred alone. Good buy in my book.
If you've got one box, whats the point of using a slow synology (iSCSI I presume) as opposed to a pair of fast local SSDs?
That supermicro is using those new Xeon-Ds - curious as to whether Intel's 'low power not low performance' marketing is legit - let us know! Virtual routers love CPU esp VIRL.... nested virtualisation of IOSv over KvM over ESXi, makes even my i5 Sandy's feel slow.
Quote from: wintermute000 on April 21, 2015, 06:55:48 AM
If you've got one box, whats the point of using a slow synology (iSCSI I presume) as opposed to a pair of fast local SSDs?
I'm doing mine this way due to having so many TB's of storage for use. My synology is maxed out with 4x 3TB and I'm currently only using 1.5TB. No need to buy any more disks when I have everything I need. My voice servers alone would require several SSD's and after what I'm paying for RAM I just don't see the need in a lab. I originally was going to use local disks and bought a RAID controller for my box but am holding off on it for now. If later on down the road I decide I need local disks I can always add them.
I scarcely doubt your voice servers require several SSDs. Thin provision everything onto a 512Gb SSD and you'll fit an entire CUCM cluster + Unity + Presence I bet.
Or have you done the thin provisioning maths and decided its still not enough?
I thin over-provision at 3:1 or worse and aren't even close to running out of room.
Quote from: wintermute000 on April 22, 2015, 03:40:22 AM
I scarcely doubt your voice servers require several SSDs. Thin provision everything onto a 512Gb SSD and you'll fit an entire CUCM cluster + Unity + Presence I bet.
Or have you done the thin provisioning maths and decided its still not enough?
I thin over-provision at 3:1 or worse and aren't even close to running out of room.
I have multiple clusters setup from 8.0 on to 10.5 :) I also have several Apache servers, IOU, Cacti, BIND, Console, F5's, xRV, Titanium, INE Topology. So I decided iSCSI would be best so that all VM's are in one place. I probably could fit them all but it would be tight, I chose to use what I already had in this case though since the NAS is always on anyways.
fair enough. I only have around 2 dozen vSRX/CSRs, VIRL, AD, linux, jumppost and vcenter. You poor voice guys LOL.
OTOH I do use iSCSI so I'm a hypocrite :) but for purposes of hosts I want to vmotion around / test HA features etc.
In another year or so when I get ministry of finance approval for a full rebuild I'll probably go all SSD, 10G and vSAN :awesome: in my dreams....
I have one of the new i5s in my work laptop (ultrabook). Broadwell if I remember right. 7 hours battery, and it has more than plenty processing power for coding and stuff. I haven't tried any of the real intensive stuff
ultrabook = dual core (the U parts), even if i7. Intel's branding is highly misleading.
FWIW I have a haswell-U i7-5500 in my work rig, which would be comparable to a broadwell-U chip. It runs IOU and my python ubuntu instance / Cisco OnePK VM fine, but absolutely chokes on VIRL. I would expect it to choke on vSRX and CSRs but I wouldn't run them on workstation anyway, you want a bare metal hypervisor for that.
Mind you, the more labbing I do, the more I am using IOU / WebIOU, so dunno if VIRL really matters.
WOOHOO scored a HP DL380 G6 on the cheap (800AUD) - 2x hex core X5650 CPUs (24 threads mmmmm), 32Gb RAM, 8x72Gb SAS.
Plenty more room to kick up to 144Gb or whatever ridiculous figure the maximum is for a dual core Westmere :)
Now I can finally run vcenter AND NSX AND AD/linux/jumppost on the same box and/or have a box that VIRL won't bring to its knees...
Time to start planning a dodgy 5.5 --> 6.0 vsphere/vcenter migration. I think I might just chuck in the old vDS config - all my 'important' hosts are on my real VLAN, only test VMs live on the vDS. Otherwise apparently the deal is to attach all the vDS to normal vswitches, move vcenters, then attach them to a new vDS. Given my ultimate aim is to run up NSX, probably not worth the effort. The new environment will be
HP DL380 G6 (2 x Xeon 5650, 32Gb RAM, dual NIC, local storage) - Vcenter, AD, linux, jumppost, VIRL, NSX, puppetmaster
2x Dell Optiplex 990s (i5-2400, 16Gb RAM, quad Intel NIC, attach to iSCSI) - test cluster
I scooped up a dirt cheap HP DL380 G7 from a company this week, two Xeon X5650s and 148 Gb of RAM. Going bare metal EVE-NG this weekend!
Damned, he said it was 148GB and I only got 44. I want my money back.
:rage:
That is a massive difference, well within your rights to return it
Sorry I didn't see this before, but you might look at old UCS boxes on ebay and see if anything tickles your fancy. You can often get some pretty good boxes for super cheap
https://www.ebay.com/itm/Cisco-UCS-M2-Dual-X5675-3-06GHZ-12x-15-12291-02-128GB-2x-146GB-HDD/163035776009?hash=item25f5b08409:g:y~UAAOSwlYRa7433
he tried that basically, not his fault the vendor lied on the spec!
He offered me another one. DL385 with 130 GB RAM, but that's an AMD Opteron inside and I suppose not everything will be well supported. Now trying to get hem to swap the RAM :)
Quote from: dlots on May 07, 2018, 09:09:35 AM
Sorry I didn't see this before, but you might look at old UCS boxes on ebay and see if anything tickles your fancy. You can often get some pretty good boxes for super cheap
https://www.ebay.com/itm/Cisco-UCS-M2-Dual-X5675-3-06GHZ-12x-15-12291-02-128GB-2x-146GB-HDD/163035776009?hash=item25f5b08409:g:y~UAAOSwlYRa7433
There's a huge differences on US prices vs EU/UK prices, and importing usually incurs additional taxes and VAT. And there's a different in PSU voltage also.
Don't go Opteron. You'll have various issues with virt. Maybe Epyc will be better in a few years, but definitely no-no on 6 year old AMD if you want it to 'just work' with stuff.
He agreed to give me 128 GB of RAM so I can keep the Xeon box :)
Quote from: SimonV on May 09, 2018, 02:07:39 AM
He agreed to give me 128 GB of RAM so I can keep the Xeon box :)
Schweet!!!!!!