Adding New Disks to Oracle ASM in vSphere Online with PowerShell

Normally disks in VMware are locked when the virtual machine accesses them to prevent another machine from accessing it as well and corrupting the disk. Having two machines writing to the same disk without coordinating with the other would invariably end up in problems. What if they both write to the same file at the same time? What if they update a directory structure at the same time? Bad things ensue. However, this is a solved problem in several domains with shared filesystems. In the Oracle world we've had OCFS1 and OCFS2 (although I think most DBAs would prefer to forget them). ASM evolved as a general shared platform for storage, and of course Oracle utilizes it for database storage, and has created a filesystem (ACFS) which runs on top of it. It coordinates several hosts writing to the same disks and there's no problem. As long as multiple hosts can access the disk through a shared bus it all just works. So what's the problem? Since VMware locks disks to prevent accidental corruption there's no way for more than one host to access it and the whole shared filesystem thing falls apart. 

Foruntately VMware has a solution to this problem. To enable shared disk access, each disk needs to have a parameter set in the virtual machine's .vmx (configuration file). For example if a disk is on controller 1, and is disk number 2, setting the parameter "scsi1:2.sharing=multi-writer" will enable multiple writers to the same disk. Do this for each disk assigned to ASM. Voila! Problem solved.

Unfortunately in both the web gui and the current thick client there is no way to update the .vmx file while the virtual host is online. The upshot of all this is that yes, you can add additional disks to ASM storage, but you must shut down the database and the host in order to do it. That just doesn't sit right with this admin, there must be a better way!

While tearing my hair out looking for a solution a colleague pointed me to this article. It's very close to what a client needed but not exactly. He clearly indicates it's a proof of concept, and I'm very grateful that this script exists but it needs some tweaking. It will only add disks to a single datastore which is fine for small VMs but several of my client's VM's need to have a total storage area in excess of 4 TB. The largest datastore is 2TB. Hmm, I think I need multiple data stores. It's command line only and doesn't validate hostnames etc. I think we'll try to put a GUI on it a little bit to eliminate input errors. Also, this client wants all local VM disks to be on scsi bus 0, and the shared disks to be on buses 1, 2 and 3. Let's see if we can't make a few changes.

That said I went through the tutorials and leaned heavily on the TechNet PowerShell tips. This allowed the following new features to the script:

1. First you are prompted to enter a pattern to match via an input dialog. Our client has hundreds of virtual machines. Instead of making them slog through a list of machines to click on the right one(s) we ask for a pattern to match against to shorten the result set. The code more or less comes from here.
2. A multi-select list then allows the user to select the machines that are in the RAC cluster. Disks will be added to all of these machines.
3. Then you are prompted for the number of disks, and their size in GB. 

The script then scans the first virtual machine selected, assuming it is node 1. It scans the bus of each scsi controller to see what slots are available. Then, starting with bus 1 it adds the disks to the first available node number until it fills that bus. Then it moves on to the next bus and so on. After the disk is created, the remaining nodes are updated to mount the already created disk, using the same bus and unit number across all nodes. Simultaneously the multi-writer parameter is added to the .vmx file so that all hosts can mount the disk.

In order for this script to function you need the following:

1. Microsoft Powershell.

2. The VMware PowerCLI extensions

3. The script itself

You will need to edit the line "connect-viserver -Server" and place the name of your own vServer host instead. The script uses Windows passthrough authentication, if you can't use that you can simply add the -User and -Password parameters to the call to authenticate.

As previously mentioned this script is pulled together from the work of many people, and I only added a little string to an already large ball. The following sites all provided pieces to the base that I extended:

(1) http://blogs.vmware.com/apps/2013/10/setting-multi-writer-flag-for-oracle-rac-on-vsphere-without-any-downtime.html

(2) technet.microsoft.com/en-us/library/ff730941.aspx