There was a time when I really thought that iSCSI was slowly going to replace Fiber Channel in all datacenters. 10Gb ethernet had started to become prevalent, VMware had stepped up their iSCSI multipathing configurations, and everyone seemed to want to flatten their network with less switches and simplified management. Obviously that didn't happen. Part of the reason has to do with the rise of converged networking gear and FCoE. You don't need to support multiple physical switching platforms, and you can have a single physical connection running both TCP and FC protocols. So the battle came down to the protocol, and ultimately FC is a better protocol for storage. I'm sure that could launch a firestorm of debate, but there's a reason why the most demanding applications and datacenters are still using it. I wasn't the only one who thought that iSCSI was the future, and I recently had to help a client migrate their VMware environment from iSCSI to FC. Read on to learn how it was done...
Allow me to set the stage. There's a cluster of VMware hosts connected to a VNX using iSCSI. A new cluster is being introduced using Fiber Channel to connect to the same VNX. The old cluster is on vCenter 5.0 and the new cluster is on a separate vCenter 5.5. The client wants to migrate all their VMs to the new cluster and vCenter with no downtime. After some discussion, the initial plan was to create some new LUNs on the VNX and present them to the new cluster using FC. Then there would be a single "swing" host with both iSCSI and FC configured. It would be presented with the old LUNs using iSCSI and the new LUNs using FC, and then the VMs would be moved via VMotion and sVMotion over to the new storage. Then the swing host would be moved to the new vCenter and the VMs offloaded. Wash. Rinse. Repeat.
After some initial testing, this did not seem to work. When presenting LUNs via UniSphere, the swing host was seeing the LUNs using both protocols instead of a single protocol. I did some digging on EMC's site and it turns out that EMC does not support presenting storage from an array to a single host using both iSCSI and FC. You can apply for an exemption, but that requires an EMC Field Engineer to evaluate your design and sign off on it. That didn't seem like a good idea, so it was time to go with a plan B. Fortunately, while I was digging through the EMC forums I came across someone with a similar question. They weren't changing vCenters, but they were trying to migrate from iSCSI to FC presented storage. The solution was simple. Present the same LUNs to both the iSCSI and FC hosts. This is totally supported by EMC, and when you think about it, there's no reason it wouldn't be. Storage protocols are still performing the same operations, regardless of the carrier protocol. VMware doesn't care which protocol you are using, and neither does the VNX.
The new migration plan was to present the existing LUNs to all the FC based hosts, and rescan their HBAs to validate they could see the datastores. Then move each host in the old cluster to the new cluster and move their VMs off to the cluster. I still needed to use a swing host because the vMotion networks were different, so one host had to be on both vMotion networks to transition the VMs off. Another fun thing I learned in the process is that VMware will try to use a vmk on a standard switch before using one on a distributed switch. Since the old vMotion network was on a standard switch, the vMotion from the swing host to the new cluster was failing with a network communication error. Once I disabled vMotion for the vmk on the standard switch, everything moved without issue. The new plan was validated and it was off to the races.
So the takeaway from this exercise was that you can present the same LUNs using both iSCSI and FC to different hosts. You cannot present the storage from a single VNX using both iSCSI and FC to a single host.
As a fun aside, you can technically present one host with iSCSI and FC LUNs from a single VNX. In Unisphere you would manually register the host twice, once with the FC initiators and once with the iSCSI initiators. Then put the FC registered host in a storage group with the LUNs you want to present using FC, and put the iSCSI registered host in a different storage group with the LUNs you want to present using iSCSI. EMC won't support this, but it will technically work on the VMware side. Most clients do not want to run an unsupported configuration however, so this was off the table.Labels: EMC, ESXi, fiber channel, iscsi, migration, Storage, storage vmotion, unisphere, vCenter, vmk, vmotion, Vmware, VNX