This post has been sat in my WordPress Drafts folder for sometime since I no longer use local storage this way. I decided to post it however as (a) it’s a good learning curve for ESXi work and (b) others may have more luck that me.
I recently acquired three 1TB drives and decided to do something about my lack of storage at home. Always trying to make best use of existing kit (and save money) I decided to stick the drives in to my HP ML110 and try something in a VM instead of doing the sensible thing of lobbing them in to a dedicated NAS box.
After wasting a few hours I realised that the onboard SATA RAID controller of the ML110 just can’t do RAID5 and to make matters worse, when I gave up and created a RAID1 array with a hot spare, vSphere 4.1 didn’t recognise the array and instead saw the drives as 3 individual drives. I saw this as a chance to try out the WAFL-alike ZFS file system. FreeNAS had been my NAS of choice recently so I chose to try ZFS in that.
I point blank refused to create 3 1TB VMDK’s (one of each of the three drives) so I set about figuring out how to create Raw Device Mappings (RDMs) of the local SATA drives. There were a couple of posts on the net that got me a little closer, but no guide/article had the whole thing down, so that’s my aim with this blog post.
Step 1
Once you had your drives installed, SSH to your ESXi box (now even easier in vSphere 4.1) and go to the /dev/disks directory. There, if you perform a ls -l, you’ll see your drives listed:
Ignore the instances of your drives which show them as VM stores (vm1.*****). We want to look at the raw devices.
Step 2
Now move to the /vmfs/volumes folder. Here you can see your existing local datastore(s). If, like me, you had a solitary hard-drive, you’ll just see localdisk01 or whatever you chose to name the local datastore:
Step 3
Now we are going to use the vmkfstools utility to create our RDM’s. Remember that a RDM is just another VMDK, but instead of the VMDK pointing to a xxx-flat.vmdk file (which is the actual virtual hard disk), the VMDK points to our physical device. Being as we still need to create this VMDK file we need to save it somewhere. Since we just have the one local datastore, we are going to create the RDM VMDK files in it’s root.
The following command creates the RDM VMDK for us:
vmkfstools -z /vmfs/devices/disks/<name of RAW device from Step 1> <location to store VMDK>/<RDM name>.vmdk
In my personal example below, I am creating an RDM called rdm_WD2DWCAVU0477582.vmdk and it is being stored in the location /vmfs/volumes/localdisk01/ I chose the name of the VMDK to match the name of the serial number of the physical drive (and what is shown in Step 1) to help with troubleshooting in the future when I get an inevitable drive failure). You can call your RDM’s whatever you wish.
The name of the RAW device (t10.ATA____WDC_WD10EARS2D00Z5B1__________________________WD2DCAVU0477582 in my example) you will have noted from Step 1 when you listed all local devices attached to your ESXi host. This is why the tech Gods created Copy n Paste! You will want to copy the full device name as shown in Step 1 in to the vmkfstools command.
Step 4
Once you have repeated the steps for all of your local SATA drives, you can navigate to where you created the RDM’s (in my case /vmfs/volumes/localdisk01) and perform an ls -l *.vmdk to see the new VMDK’s you have created:
Don’t panic – the xxx-rdmp.vmdk files will reflect the size of the RAW devices they are mapping to, but rest assured it will be taking no more space than a few bytes on your local disk!
Step 5
You can now add your RDM’s to an existing VM. vSphere doesn’t recognise this as a true RDM (to a SAN) so you just browse the local disk datastore for the VMDK files that we created.
Edit the properties of an existing VM and click Add…
Step 6
Select Use an existing virtual disk and click Next >
Step 7
Click Browse. You now need to navigate your local datastore and select the VMDK’s that we created in Step 3).
Once complete you will be shown a confirmation window. Repeat Steps 5 through 7 to add additional RDM’s to your VM.
Step 8
You should now see your new Hard Disk’s in your VM and vSphere will correctly identify them as Mapped Raw LUN.
NOTE: One thing I forgot to show in the screen shots, is that you should create your RDM’s on a new SCSI controller! You do this by simply selecting a new SCSI ID starting with 1:x instead of 0:x. Existing VMDK’s should be on SCSI Controller 0. Your RDM’s should be on SCSI controller 1. Although my screenshot shows 0:3 this should read 1:3.
You can now save your VM configuration. Your VM will now access the RAW SATA drives and be able to use things like SMART to monitor its health.
See below; I am adding my three 1TB drive to FreeNAS to create a new ZFS pool.
Stay tuned for an upcoming blog post on FreeNAS and NexentaStor which may or may not put you off ZFS [in a VM] altogether!
My Website
Where is your post on your next steps? Why aren’t you using freenas and RDM under VSXi?
I’m not sold on ZFS yet, but that probably has something to do with me not understanding everything about it.
What is the problem with sharing the SCSI controller?
Thanks for the great post! I’m currently spec’ing out some hardware on which I’m going to install either ESXi or Hyper-V so that I can run Windows Home Server as a guest. (along with some linux guests too)
I’m a bit new to VMWare, but my understanding was that using RDM for local drives (SATA at least) was not supported. Is your method above a workaround for local drives, or is this sort of thing now officially supported in ESXi? I was leaning towards using Hyper-V because of this exact hurdle. (Hyper-V supports RDM for local drives)
Finally, I assume everything you’ve done here can be done on the free version of ESXi? Thanks again!!
-alex
Hi David,
Thanks for providing this article – I was able to follow the steps easily and have mapped my 1TB RAID 5 array to a Virtual Machine. I still find it strange that ESXi doesn’t support this kind of functionality itself….seems a bit silly really.
Thanks again!
Ady
RDM on local sata sounds ideal for some kind of ZFS in a virtual machine – I am pondering an Nextenta installation in a VM for my next vSphere hardware upgrade …
I am glad that you posted this but curious as to why you say that you hope that other people will have more luck with it than you?
Also if localdisk01 dies does the vmkfstools command that you used simply re-register the existing local RDM drives or is some other command required to do that?
I assume that you have to add three drives at a time to ZFS to create a storage pool? This would appear to be the only downside to ZFS that I am aware of right now …
[...] [...]
Hello David,
Excellent Post, Complete and Concise. All working here, I only needed to do this with one drive formatted in xfs containing my data, which esxi does not support of course! Openfiler VM was able to mount the device no issues, and can now be mapped throughout my network. Thank you for your wisdom.
Cheers.
AJG
Thanks for this excellent guide. I have used these RDMs with Ubuntu, Windows, and Openfiler (rPath) using 2TB drives and they worked great. But FreeNAS (freeBSD) had nothing but problems – all looking like a corrupted disk. I scanned the disks every way I know and they checked fine. I even swapped disks and the problem remained with FreeNAS. Then I found this http://communities.vmware.com/thread/284772 discussion that suggests that vmkfstools -r is a better option to use with FreeNAS.
Do you have any insight into the use of -r or -z creating RDMs?
[...] [...]
Great solution! Thank you for the time and effort!
BR
Great article. Helped me a lot.
thanks.
[...] and via the console using vmkfstools. I was able to use David Wurburton's approach (Found here: http://blog.davidwarburton.net/2010/10/25/rdm-mapping-of-local-sata-storage-for-esxi/) 3) FreeNAS doesn't support "vmxnet3", so I'll use "vmxnet2 (enhanced)" 4) NFS performance back to [...]
problems. first i have no localdisk01 folder and if i try to create it i get an error.
then i get a lock file error. simply not working.
Sorry to hear you’re having problems. I’m not currently using this so can’t offer much advise. Anyone else able to help?
This all still works quite well today, using ESXi 5.0. Thank you so much for posting this, David!
I posted related information here as well:
http://tinkertry.com/beyond2tbvmdisksizemaximum/
[...] I’ll try to cover the more-advanced and totally unsupported technique of actually configuring RDMs (Raw Device Mappings), that magically allow your existing >2TB SATA drives and all the data on [...]
[...] are you interested in RDMs? David Warburton has expertly documented many of the techniques here, and my future posts will also cover prep for boot (BIOS settings and VM settings to avoid BSODs), [...]
Fantastic blog, saves me every time I do this
[...] [...]
It’s Impossible to do this if you don’t have one more hard drive to save this RDMs to. Like if you have ESXi installed on flash drive.
Hi Pavel, I’m surprised to hear that you had issues since I was running ESXi from a USB drive myself.