Oracle Database Appliance - My case for Solaris

I just got to try out the Oracle database appliance (ODA).  It's supposed to be quick and easy to setup and get it running.  We've run into some intial hiccups, but this is a POC box and I'm not going to give Oracle grief here.

Here are my quick thoughts

Pros:
1.  Extremely easy to setup (hardware perspective)
2.  Very cost effective.
3.  Comes with lots of Storage and now allows expansion into another Storage Shelf.

Cons:

1.  Only Oracle Linux.  No Solaris.
2.  Only one ASM across all the databases.
3.  Patching has to be done in a particular order.  You can't simply patch Db, you need to patch hardware, OS and such, before you get to the Db patching (There is a hierarchy)
4.  When you start patching, depending upon what you're patching, ALL Databases need to be shutdown.  This is very hard to do, especially when you have multiple applications using Dbs hosted on the ODA.

So - where is Solaris and why you ask.

1.  Solaris 10 / 11 allows you to run Oracle RAC clusters in Zones.
2.  With Solaris, you can have Zones running your Oracle RAC, and multiple Oracle RAC instances can be spun up, especially with MULTIPLE ASMs running.  This will immediately allow you to patch your infrastructure one node at a time.
3.  The bonded connections with Solaris 11 makes it incredibly simple to work with.  This link specifically: Network Virtualization High Availability made me think of ODA on Solaris a lot more.
4.  Solaris 11 allows you to add disks to zones without a reboot (a big Solaris 10 limitation).
5.  Zones allow you to CPU cage.
6.  The Hardware is all based off the Sun systems.
7.  You can use ZFS for all OS disks.
8.  You can get away from the clunky OVM DOM0 world that the ODA appliance uses now.
9.  You can effortlessly do virtualization and use the VM Templates.
10.  CloudFS partition would be effortless too - just make a new ZPOOL and you're done!
11.  All this can be tied into Oracle Enterprise Manager & Oracle Ops Center for the hardware.
12.  All the patching can be made so much more effortless.

Oracle 11g RAC in LDOMs (Oracle VM for Sparc)

Setup
1. 2 T3-1B blades on different chassis. Patch the firmware. Install LDOM 2.1 software.
2. SAN Connected storage via MPXIO (EMC SAN)
3. Solaris 10/9 installed with End User Option.



Create install User

useradd -u 0 -o -g 1 -c "Install user" -d / -s /bin/true install

Patch LDOM.

Download the following Patches from oracle
• 117837-05: C++ compiler optimizer patch
• 117846-08: C++ compiler Optimization patch
• 118682-01
• 127111-02 SunOS 5.10: libc patch (this should already exist if LDOM is patched)
• 137111-04 SunOS 5.10: kernel patch (this should already exist if LDOM is patched)

Profile Settings

• On a new system, set the following in /etc/profile
a. MANPATH=/usr/man:/usr/local/man:/usr/sfw/man:$MANPATH

• Create the oracle required users & groups

groupadd -g 102 dba
groupadd -g 103 oinstall
projadd -p 102 -c "Oracle Project" group.dba
useradd -u 101 -g dba -G oinstall -s /usr/bin/bash -c "Oracle User" -m –p 102 oracle

• Set the Oracle password

passwd -r files oracle

• Add the following entries to /etc/system for a 4G SGA or less. Shmmax should always be greater than SGA size.

set noexec_user_stack=1
set semsys:seminfo_semmni=100
set semsys:seminfo_semmns=1024
set semsys:seminfo_semmsl=256
set semsys:seminfo_semvmx=32767
set shmsys:shminfo_shmmax=4294967296
set shmsys:shminfo_shmmni=100
set rlim_fd_max=65636
set rlim_fd_cur=4096

Reboot the system after the above /etc/system have been made.

Alternate way

Run prctl to make changes to the existing install so that a reboot is not required. These should be run as root. These changes don’t persist after a reboot.
prctl -n project.max-shm-memory -v 4gb -r -i project user.root
prctl -n project.max-sem-ids -v 256 -r -i project user.root

Project Settings

Make changes to the project so that changes persist for shm memory & sem ids (4G or less SGA). These should be run as root.

projmod -sK "project.max-shm-memory=(privileged,4G,deny)" group.dba
projmod -sK "project.max-sem-ids=(privileged,256,deny)" group.dba

/etc/project looks like this after changes:

[root@orc11gtest]# cat /etc/project
system:0::::
user.root:1::::
noproject:2::::
default:3::::
group.staff:10::::
group.dba:102:Oracle Project:oracle::project.max-sem-ids=(privileged,256,deny);project.max-shm-memory=(privileged,4294967296,deny)

Verify the changes applied to the oracle user as below.

[oracle@orc11gtest]$ prctl -n project.max-shm-memory -i process $$
process: 2834: -bash
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-shm-memory
privileged 4.00GB - deny -
[oracle@orc11gtest]$ prctl -n project.max-sem-ids -i process $$
process: 2834: -bash
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-sem-ids
privileged 256 - deny

Patch Check:

patchadd -p | grep 125555-03
patchadd -p | grep 140796-01
patchadd -p | grep 140899-01
patchadd -p | grep 141016-01
patchadd -p | grep 139555-08
patchadd -p | grep 141414-02 (Replaced by 141444-09)
patchadd -p | grep 141736-05
patchadd -p | grep 120753-06
patchadd -p | grep 139574-03
patchadd -p | grep 141414-10
patchadd -p | grep 141736-05

I'm using Ops center to create LDOMs to make my life a little easier. If you're doing this, the Vnets sometimes will be different between the 2 blades. Go into the "/guests//servercfg.xml" and fix the

< idx >3< /idx >

number to match on both nodes.


NTP Configuration
Make sure to set the /etc/inet/ntp.conf with slewalways and disable p11.

server < serverip >
driftfile /etc/ntp.drift
slewalways yes
disable pll

Grid Software Install

ON each Control Domain -- add the disks for OVD, OCR & Data to the LDOMs. Make sure to add them in the same order, and set the same ID. A good way to make sure everything is the same on both nodes, is to also volume label the disks on the control / IO domain before assigning it to the LDOM.

Have xwin running and kick off the installer. Before you kick off the installer, its a good idea to run cluster verify tool.

./runcluvfy.sh stage -pre crsinst -n orczone1,orczone2 -verbose

Install any packages that are missing, fix the swap or any such thing, the MTUs etc, and you're ready to roll.

To add Swap Space when using ZFS

To create swap space on a ZFS volume (created 2 2g volumes):

zfs create -V 2gb rpool/swap2g
zfs create -V 2gb orcinstall/swap2g

Add it to swap

swap -a /dev/zvol/dsk/rpool/swap2g
swap -a /dev/zvol/dsk/orcinstall/swap2g

Add it to /etc/vfstab

/dev/zvol/dsk/rpool/swap2g - - swap - no -
/dev/zvol/dsk/orcinstall/swap2g - - swap - no -

Install / Troubleshooting Notes

1. A group called "DATA" was created by a previous install, and even though I zapped the disks, the group didnt go away. To drop the diskgroup (which wouldn't mount or come online) I had to do this:
drop diskgroup data force including contents;
2. If you're having trouble with ulimits and such, this link: Max user processes is pretty helpful.

Oracle 11g RAC in a zone - Partial instructions.

This is a post thats a WIP. I will be adding to this as I keep finding new things, and hopefully a full step by step doc by the end of this.

I had to stop this because of the requirement of EXCLUSIVE IPs for the non-global zone. You can only have one Exclusive interface per vlan tag per server.

That is, if you have VLAN tag 3 for your public interface, and you want to create another RAC cluster on the same blade, you'll have to get another vlan from your network team. This seems unworkable, as all our Dbs are in the same VLAN.

I've just begun setting up 2 T3-Blades to use zones for Oracle RAC cluster.

Setup:
2 T3-1B Sparc blades with 64Gb of RAM.
Oracle 11gR2 downloaded
Oracle 11gR2 Grid Download
OS installed with "Full Install with OEM"
OS patched with the latest patch cluster.
Use EMC SAN for disk

MPXIO
Enable MPXIO if you're not using any vendor provided software to connect to the SAN
Using MPXIO to connect to the EMC CX-500
change "mpxio-disable" to "no" in ./kernel/drv/fp.conf

Disks
Using ZFS for the global zone with internal disks.

All Oracle zones will be created using mirrored disks from the SAN.

I intend on creating a zone, configuring the basic setup (DNS, oracle user, oracle download copied over) and take

[root@ch0bl7qadb1w]# zpool list
NAME SIZE ALLOC FREE CAP HEALTH ALTROOT
orcpool 248G 11.0G 237G 4% ONLINE -
rpool 278G 7.99G 270G 2% ONLINE -
[root@ch0bl7qadb1w]# zpool status -v orcpool
pool: orcpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
orcpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t60060160733014002C5FAEED7BD9E011d0 ONLINE 0 0 0
c0t60060160733014002D5FAEED7BD9E011d0 ONLINE 0 0 0

errors: No known data errors

Creating Zones
Create the zone (orczone1), boot it and configure the DNS and such.

Zones have an annoying habit of setting auto-home. I normally run a script to disable auto-home. To allow the creation of the /home/oracle directory below, it's required to disable autohome.

Create the oracle project with group membership of dba.

projadd -p 102 -c "Oracle Project" group.dba

Create the dba (id: 102) oinstall (id: 103) groups

Create the Oracle, Oragrid users, the assign the oracle user belong to the oracle project.

useradd -u 101 -G dba -g oinstall -s /usr/bin/bash -m -p 102 oracle
useradd -u 103 -G dba -g oinstall -s /usr/bin/bash -m oragrid

Verify that the users are created appropriately

[root@orczone1]# id -a oracle
uid=101(oracle) gid=103(oinstall) groups=102(dba)
[root@orczone1]# id -a oragrid
uid=103(oragrid) gid=103(oinstall) groups=102(dba)

According to OFA guidelines, Oracle going to be installed in
/opt/app/oracle

Oracle install also recommends that only ORACLE_BASE is set. I'm also going to be installing Grid software and Db software as the same user.

Bash Profile for the oracle User

PS1="[\u@\h]$ "
ORACLE_BASE=/opt/app/oracle
LD_LIBRARY_PATH=/usr/lib:/lib:$LD_LIBRARY_PATH
export SQLPATH=/home/oracle

export ORACLE_BASE ORACLE_SID LD_LIBRARY_PATH

export PATH=$CRS_HOME/bin:$ORACLE_HOME/bin:/usr/sbin:/usr/bin:/usr/ucb:/usr/local/bin:/bin/sbin:$PATH

umask 022
ulimit -n 65536

export EDITOR=vi
export editor=vi
export PS1

NTP
My /etc/inet/ntp.conf file is this (in the Global Zone):

server 10.27.1.254
driftfile /etc/ntp.drift
slewalways yes
disable p11

Grid Software install

1. Create /opt/app/11.2.0/grid on both nodes (orczone1 & orczone2).
2. Create /opt/app/oracle on both nodes
3. I created the "app" directory and let oracle have rwx permissions on it.
4. Add the SCAN Ips as well as the virtual IPs to DNS

Orczone1.cctdev.com
Orczone1-vip.cctdev.com

Orczone2.cctdev.com
Orczone2-vip.cctdev.com

Orczone-cluster.cctdev.com -- this was the SCAN IP.

5. Set the following in /etc/system for the global zone, to avoid annoyances in the non-global zone. I use these blades only for Oracle, so it works for me!

set rlim_fd_max=65636
set rlim_fd_cur=4096

5. During install of GRID software, oracle will try to plumb IPs and if you're not using Exclusive IPs in the non-global zone, it will fail. This is where I stopped and switched to LDOMs.


Issue Log:

One of the things that are needed are a Public & Private Interface.

When plumbing them, make sure that the private interface stays private and does not get added to your "default gateway". If that happens, traffic has a tendency to get "lost" and when you try to reboot zones or even the server, you may lose connectivity with other nodes. The way I did this, was to not add a defrouter in the zonecfg for the private interface.

When creating a Zone for the Private interface, do not set the default router for that NIC unless you're using VLAN tagging and such.

The netstat -rn should look something similar to this

Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 10.25.20.5 UG 1 27
default 10.25.20.5 UG 1 4 e1000g2
10.0.0.0 10.26.6.202 U 1 16 igb0
10.25.20.0 10.25.23.137 U 1 3 e1000g2
192.168.204.0 192.168.204.7 U 1 0 e1000g0
224.0.0.0 10.25.23.137 U 1 0 e1000g2
127.0.0.1 127.0.0.1 UH 5 92 lo0