Sun Cluster 3.2 with LDOM 1.3

I had to get a Sun Cluster 3.2 to create a HA-NFS on LDOM 1.3. LDOM 1.3 and Sun Cluster 3.2 now fully support Sun Clusters on Guest Domains not just the I/O Domain. This makes it incredibly useful.

I faced plenty of issues, and hoping that this post will serve as a guide to those who need to do it.

Setup:

1. 2 Physical T-6340 blades.
2. Control Domain that also serves as our I/O Domain.
3. 1 Guest Domain on each Server for Sun Cluster.
3. EMC Powerpath installed on the Control Domain

Storage:

Managed by EMC Powerpath

The devices will be seen as : emcpowerXa

1 LUN for each Sun Cluster domain

1 Shared LUN between the 2 physical blades

I want to note STRONGLY that you should first try using the powerdevices for the LDOM & Sun Cluster, and only if that doesnt work use the cXtXdX devices. Each configuration may be different and you may have better luck or a patch that I did not have at the time of my trial.

powermt display dev= will give you the underlying cXtXdX number (in the I/O Domain).

Present all devices in MPGroups to the LDOM (even for boot disk since we'll be using SVM with Sun Cluster) as cXtXdX devices.

3 disks to be presented in this example

1. Boot Disk (mpgroup: bdskgroup)

2. Quorum Disk (mpgroup: quroumgroup)

3. Shared Disk for storage (mpgroup: cludskgroup)

(Make sure to give an id to each of the vdisks that are presented to the LDOM so that they always are presented in the same order even on a reboot)

For each Sun Cluster LDOM present the VNETs from the VSWs in the order they're cabled for the private interconnect. This is important.

I used cross over cables.

Jumpstart and boot up the LDOM.

If you've done everything correctly you should see 3 devices when you do "format".

Install the Sun Cluster software with the NFS packages. Patch it appropriately.

I used the below instructions to create the cluster, however, its far easier in 3.2. Just create the metadb by configuring the SVM (instructions in link below) and then run scinstall. Once thats done run clsetup and follow the instructions for the NFS Service.

http://napobo3.lk.net/sc/dual-node_HA_NFS_cluster_setup.html

Copied from the above link:
2 nodes (hostname: node1, node2) with installed Solaris 10 01/06, patches, Sun Cluster, NFS agent for Sun Cluster, VxFS
Both nodes are connected to the FC SAN storage, 8 storage LUNs are mapped to each node.

Configure SVM
On both nodes
Create the 25MB partition on the boot disk (s7)
Create the SVM database replica
metadb -afc 3 c0d0s7 (c0t0d0s7 on sparc)

On one node (node1)
- Create disk sets :
metaset -s nfs1 -a -h node1 node2
metaset -s nfs1 -t -f
metaset -s nfs1 -a /dev/did/rdsk/d2 /dev/did/rdsk/d3 /dev/did/rdsk/d4 /dev/did/rdsk/d5
metainit -s nfs1 d1 4 1 /dev/did/rdsk/d2s0 1 /dev/did/rdsk/d3s0 1 /dev/did/rdsk/d4s0 1 /dev/did/rdsk/d5s0
metastat -s nfs1 -p >> /etc/lvm/md.tab

metaset -s nfs2 -a -h node2 node1
metaset -s nfs2 -t -f
metaset -s nfs2 -a /dev/did/rdsk/d6 /dev/did/rdsk/d7 /dev/did/rdsk/d8 /dev/did/rdsk/d9
metainit -s nfs2 d1 4 1 /dev/did/rdsk/d6s0 1 /dev/did/rdsk/d7s0 1 /dev/did/rdsk/d8s0 1 /dev/did/rdsk/d9s0
metastat -s nfs2 -p >> /etc/lvm/md.tab
scp /etc/lvm/md.tab node2:/tmp/md.tab
ssh node2 'cat /tmp/md.tab >> /etc/lvm/md.tab'

- Create VxFS on shared devices -- I didnt do this.. I just did a newfs.
mkfs -F vxfs /dev/md/nfs1/rdsk/d1
mkfs -F vxfs /dev/md/nfs2/rdsk/d1

On both nodes
- Create the directories
mkdir -p /global/nfs1
mkdir -p /global/nfs2

http://docs.sun.com/app/docs/doc/819-0912/chapter1-1?l=en&a=view is the quick start guide, is some help as well (especially with the metadb setup), but they talk about LOCAL devices as HA-NFS, and that isn't want we're doing. :-)