by I’ve recently had a number of projects at work that want to mix multiple volume managers on a single server, specifically ZFS and VxVM for SAN volumes (actually, three including SVM for internal boot disk mirroring). The projects generally are for database servers, and want to use VxVM for database volumes because ZFS currently has some serious limitation on database size (limited number of devices recommended in a single zpool) and performance (single threaded checksumming, for one). However, at the same time, they want to have access to some of ZFS’s features (in particular, the ability to oversubscribe filesystems, dynamic resize, snapshots and rollback) for some of the other filesystems.
Technically, this works. Both volume managers can exist on the system, LUNs can be allocated to either volume manager, etc. However, operationally it is a bit of a problem. VxVM has no knowledge of whether a disk is already in a zpool, so it is possible to corrupt a zpool by trying to add the disk to a volume group (just running _vxdisk*_ without actually running a _vxdg add_ will trash the zpool).
ZFS knows if a disk is used by VxVM, so this corruption isn’t as much of an issue if we’re adding to ZFS. However, we found recently that this only works correctly if VxVM 4.X was installed (even if later upgraded to VxVM 5.X). This morning we were provided a workaround from Sun that has us specifically set the LD_LIBRARY_PATH to include _/opt/VRTSvxvm/lib_, but this either has to be done every time a zpool is run (by any user with permission) or be permanently set in the profile of every user who can do this. I’m very leery of changing the LD_LIBRARY_PATH on a general basis.
Solaris 10 MPxIO has changed the way LUNs are named (they now include WWNs in the target, so instead of c2t5d0s2, you get c2t20005b48cb3659aa6710dcba47825c34d0s2), which means when you have a system with 250 LUNs, it is very easy to make a mistake and say “oh no, that LUN isn’t used” and then try to reuse it.
As a general rule, I’m trying to push against any request to mix both volume managers, but in some cases I’m getting a huge push back from the end users. We’ve identified some workarounds that help, but they are all fairly manual (error prone) and require specific knowledge (not necessarily good in a large environment where our primary first level support teams may have marginal skills).
#. All LUNS must be allocated to a volume manager as soon as they are attached to the server. In the past, the developers would request “half a terrabyte” from the storage team, and then over time as us to allocate 50 gig to this and 20 gig to that. At the very least, we are now setting a policy that any storage not specifically allocated is added to a VxVM “free” volume group.
#. Make sure to use the _/etc/vx/vxvm.exclude_ for all disks used with ZFS. This stops VxVM from being able to function on those disks. This is a manual task that is extremely error prone (just being forgotten, or a typo).
I don’t believe either of these to be the best practice. What are others out there seeing/doing?