Wednesday, February 24, 2010

Sun Cluster 3.2 with LDOM 1.3

I had to get a Sun Cluster 3.2 to create a HA-NFS on LDOM 1.3. LDOM 1.3 and Sun Cluster 3.2 now fully support Sun Clusters on Guest Domains not just the I/O Domain. This makes it incredibly useful.
I faced plenty of issues, and hoping that this post will serve as a guide to those who need to do it.
Setup:
1. 2 Physical T-6340 blades.
2. Control Domain that also serves as our I/O Domain.
3. 1 Guest Domain on each Server for Sun Cluster.
3. EMC Powerpath installed on the Control Domain

Storage:
Managed by EMC Powerpath
The devices will be seen as : emcpowerXa
1 LUN for each Sun Cluster domain
1 Shared LUN between the 2 physical blades

I want to note STRONGLY that you should first try using the powerdevices for the LDOM & Sun Cluster, and only if that doesnt work use the cXtXdX devices. Each configuration may be different and you may have better luck or a patch that I did not have at the time of my trial.

powermt display dev= will give you the underlying cXtXdX number (in the I/O Domain).
Present all devices in MPGroups to the LDOM (even for boot disk since we'll be using SVM with Sun Cluster) as cXtXdX devices.

3 disks to be presented in this example
1. Boot Disk (mpgroup: bdskgroup)
2. Quorum Disk (mpgroup: quroumgroup)
3. Shared Disk for storage (mpgroup: cludskgroup)

(Make sure to give an id to each of the vdisks that are presented to the LDOM so that they always are presented in the same order even on a reboot)

For each Sun Cluster LDOM present the VNETs from the VSWs in the order they're cabled for the private interconnect. This is important.

I used cross over cables.

Jumpstart and boot up the LDOM.
If you've done everything correctly you should see 3 devices when you do "format".

Install the Sun Cluster software with the NFS packages. Patch it appropriately.

I used the below instructions to create the cluster, however, its far easier in 3.2. Just create the metadb by configuring the SVM (instructions in link below) and then run scinstall. Once thats done run clsetup and follow the instructions for the NFS Service.
Copied from the above link:
2 nodes (hostname: node1, node2) with installed Solaris 10 01/06, patches, Sun Cluster, NFS agent for Sun Cluster, VxFS
Both nodes are connected to the FC SAN storage, 8 storage LUNs are mapped to each node.

Configure SVM
On both nodes
Create the 25MB partition on the boot disk (s7)
Create the SVM database replica
metadb -afc 3 c0d0s7 (c0t0d0s7 on sparc)

On one node (node1)
- Create disk sets :
metaset -s nfs1 -a -h node1 node2
metaset -s nfs1 -t -f
metaset -s nfs1 -a /dev/did/rdsk/d2 /dev/did/rdsk/d3 /dev/did/rdsk/d4 /dev/did/rdsk/d5
metainit -s nfs1 d1 4 1 /dev/did/rdsk/d2s0 1 /dev/did/rdsk/d3s0 1 /dev/did/rdsk/d4s0 1 /dev/did/rdsk/d5s0
metastat -s nfs1 -p >> /etc/lvm/md.tab

metaset -s nfs2 -a -h node2 node1
metaset -s nfs2 -t -f
metaset -s nfs2 -a /dev/did/rdsk/d6 /dev/did/rdsk/d7 /dev/did/rdsk/d8 /dev/did/rdsk/d9
metainit -s nfs2 d1 4 1 /dev/did/rdsk/d6s0 1 /dev/did/rdsk/d7s0 1 /dev/did/rdsk/d8s0 1 /dev/did/rdsk/d9s0
metastat -s nfs2 -p >> /etc/lvm/md.tab
scp /etc/lvm/md.tab node2:/tmp/md.tab
ssh node2 'cat /tmp/md.tab >> /etc/lvm/md.tab'

- Create VxFS on shared devices -- I didnt do this.. I just did a newfs.
mkfs -F vxfs /dev/md/nfs1/rdsk/d1
mkfs -F vxfs /dev/md/nfs2/rdsk/d1

On both nodes
- Create the directories
mkdir -p /global/nfs1
mkdir -p /global/nfs2


http://docs.sun.com/app/docs/doc/819-0912/chapter1-1?l=en&a=view is the quick start guide, is some help as well (especially with the metadb setup), but they talk about LOCAL devices as HA-NFS, and that isn't want we're doing. :-)



2 comments:

Unknown said...

Hi i have seen ur document. its good. but i think u have not posted the main configuration part which is the setting up VNETS for SUN-Cluster from Virtual Switch.

I am stuck in this part, can you please again post it or email me on shakrehman@hotmail.com

Murali said...

Setting up the vnets is really easy.

I normally have 4 VSWs, 2 for Primary Traffic, 2 for Cluster heartbeat.

So if I have
primary-vsw1
primary-vsw2
heartbeat-vsw1
heartbeat-vsw2

I do this

ldm add-vnet id=0 vnet0 primary-vsw1 cluster-node1

ldm add-vnet id=1 vnet1 primary-vsw2 cluster-node 1

ldm add-vnet id=2 vnet2 heartbeat-vsw1 cluster-node1

ldm add-vnet id=2 vnet3 heartbeat-vsw2 cluster-node2

I repeat on node 2. To be absolutely sure, you should use IDs as above. That way you get them in the exact order.