Release of Solaris (Sparc or x86?)

I got a handy tip from an Oracle person the other day....to make sure what solaris release and version the solaris cd is or jumpstart location is, check the SUNWsolnm/reloc/etc/release file.

[root@host1]# cat SUNWsolnm/reloc/etc/release
Oracle Solaris 10 9/10 s10s_u9wos_14a SPARC
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Assembled 11 August 2010
[root@host1]#

TFTP Boot error Access Violation

When accessing a tftp server clients receive "Error code 2: Access violation"
due to ownership & permission of the /tftpboot directory.

Steps to Follow
in.tftpd runs as user nobody. nobody needs rwx permissions to the /tftpboot
directory
chmod 766 /tftpboot
chown nobody:other /tftpboot

Product
Network Management/Conn

Oracle 11g RAC in LDOMs (Oracle VM for Sparc)

Setup
1. 2 T3-1B blades on different chassis. Patch the firmware. Install LDOM 2.1 software.
2. SAN Connected storage via MPXIO (EMC SAN)
3. Solaris 10/9 installed with End User Option.



Create install User

useradd -u 0 -o -g 1 -c "Install user" -d / -s /bin/true install

Patch LDOM.

Download the following Patches from oracle
• 117837-05: C++ compiler optimizer patch
• 117846-08: C++ compiler Optimization patch
• 118682-01
• 127111-02 SunOS 5.10: libc patch (this should already exist if LDOM is patched)
• 137111-04 SunOS 5.10: kernel patch (this should already exist if LDOM is patched)

Profile Settings

• On a new system, set the following in /etc/profile
a. MANPATH=/usr/man:/usr/local/man:/usr/sfw/man:$MANPATH

• Create the oracle required users & groups

groupadd -g 102 dba
groupadd -g 103 oinstall
projadd -p 102 -c "Oracle Project" group.dba
useradd -u 101 -g dba -G oinstall -s /usr/bin/bash -c "Oracle User" -m –p 102 oracle

• Set the Oracle password

passwd -r files oracle

• Add the following entries to /etc/system for a 4G SGA or less. Shmmax should always be greater than SGA size.

set noexec_user_stack=1
set semsys:seminfo_semmni=100
set semsys:seminfo_semmns=1024
set semsys:seminfo_semmsl=256
set semsys:seminfo_semvmx=32767
set shmsys:shminfo_shmmax=4294967296
set shmsys:shminfo_shmmni=100
set rlim_fd_max=65636
set rlim_fd_cur=4096

Reboot the system after the above /etc/system have been made.

Alternate way

Run prctl to make changes to the existing install so that a reboot is not required. These should be run as root. These changes don’t persist after a reboot.
prctl -n project.max-shm-memory -v 4gb -r -i project user.root
prctl -n project.max-sem-ids -v 256 -r -i project user.root

Project Settings

Make changes to the project so that changes persist for shm memory & sem ids (4G or less SGA). These should be run as root.

projmod -sK "project.max-shm-memory=(privileged,4G,deny)" group.dba
projmod -sK "project.max-sem-ids=(privileged,256,deny)" group.dba

/etc/project looks like this after changes:

[root@orc11gtest]# cat /etc/project
system:0::::
user.root:1::::
noproject:2::::
default:3::::
group.staff:10::::
group.dba:102:Oracle Project:oracle::project.max-sem-ids=(privileged,256,deny);project.max-shm-memory=(privileged,4294967296,deny)

Verify the changes applied to the oracle user as below.

[oracle@orc11gtest]$ prctl -n project.max-shm-memory -i process $$
process: 2834: -bash
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-shm-memory
privileged 4.00GB - deny -
[oracle@orc11gtest]$ prctl -n project.max-sem-ids -i process $$
process: 2834: -bash
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-sem-ids
privileged 256 - deny

Patch Check:

patchadd -p | grep 125555-03
patchadd -p | grep 140796-01
patchadd -p | grep 140899-01
patchadd -p | grep 141016-01
patchadd -p | grep 139555-08
patchadd -p | grep 141414-02 (Replaced by 141444-09)
patchadd -p | grep 141736-05
patchadd -p | grep 120753-06
patchadd -p | grep 139574-03
patchadd -p | grep 141414-10
patchadd -p | grep 141736-05

I'm using Ops center to create LDOMs to make my life a little easier. If you're doing this, the Vnets sometimes will be different between the 2 blades. Go into the "/guests//servercfg.xml" and fix the

< idx >3< /idx >

number to match on both nodes.


NTP Configuration
Make sure to set the /etc/inet/ntp.conf with slewalways and disable p11.

server < serverip >
driftfile /etc/ntp.drift
slewalways yes
disable pll

Grid Software Install

ON each Control Domain -- add the disks for OVD, OCR & Data to the LDOMs. Make sure to add them in the same order, and set the same ID. A good way to make sure everything is the same on both nodes, is to also volume label the disks on the control / IO domain before assigning it to the LDOM.

Have xwin running and kick off the installer. Before you kick off the installer, its a good idea to run cluster verify tool.

./runcluvfy.sh stage -pre crsinst -n orczone1,orczone2 -verbose

Install any packages that are missing, fix the swap or any such thing, the MTUs etc, and you're ready to roll.

To add Swap Space when using ZFS

To create swap space on a ZFS volume (created 2 2g volumes):

zfs create -V 2gb rpool/swap2g
zfs create -V 2gb orcinstall/swap2g

Add it to swap

swap -a /dev/zvol/dsk/rpool/swap2g
swap -a /dev/zvol/dsk/orcinstall/swap2g

Add it to /etc/vfstab

/dev/zvol/dsk/rpool/swap2g - - swap - no -
/dev/zvol/dsk/orcinstall/swap2g - - swap - no -

Install / Troubleshooting Notes

1. A group called "DATA" was created by a previous install, and even though I zapped the disks, the group didnt go away. To drop the diskgroup (which wouldn't mount or come online) I had to do this:
drop diskgroup data force including contents;
2. If you're having trouble with ulimits and such, this link: Max user processes is pretty helpful.

Oracle 11g RAC in a zone - Partial instructions.

This is a post thats a WIP. I will be adding to this as I keep finding new things, and hopefully a full step by step doc by the end of this.

I had to stop this because of the requirement of EXCLUSIVE IPs for the non-global zone. You can only have one Exclusive interface per vlan tag per server.

That is, if you have VLAN tag 3 for your public interface, and you want to create another RAC cluster on the same blade, you'll have to get another vlan from your network team. This seems unworkable, as all our Dbs are in the same VLAN.

I've just begun setting up 2 T3-Blades to use zones for Oracle RAC cluster.

Setup:
2 T3-1B Sparc blades with 64Gb of RAM.
Oracle 11gR2 downloaded
Oracle 11gR2 Grid Download
OS installed with "Full Install with OEM"
OS patched with the latest patch cluster.
Use EMC SAN for disk

MPXIO
Enable MPXIO if you're not using any vendor provided software to connect to the SAN
Using MPXIO to connect to the EMC CX-500
change "mpxio-disable" to "no" in ./kernel/drv/fp.conf

Disks
Using ZFS for the global zone with internal disks.

All Oracle zones will be created using mirrored disks from the SAN.

I intend on creating a zone, configuring the basic setup (DNS, oracle user, oracle download copied over) and take

[root@ch0bl7qadb1w]# zpool list
NAME SIZE ALLOC FREE CAP HEALTH ALTROOT
orcpool 248G 11.0G 237G 4% ONLINE -
rpool 278G 7.99G 270G 2% ONLINE -
[root@ch0bl7qadb1w]# zpool status -v orcpool
pool: orcpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
orcpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t60060160733014002C5FAEED7BD9E011d0 ONLINE 0 0 0
c0t60060160733014002D5FAEED7BD9E011d0 ONLINE 0 0 0

errors: No known data errors

Creating Zones
Create the zone (orczone1), boot it and configure the DNS and such.

Zones have an annoying habit of setting auto-home. I normally run a script to disable auto-home. To allow the creation of the /home/oracle directory below, it's required to disable autohome.

Create the oracle project with group membership of dba.

projadd -p 102 -c "Oracle Project" group.dba

Create the dba (id: 102) oinstall (id: 103) groups

Create the Oracle, Oragrid users, the assign the oracle user belong to the oracle project.

useradd -u 101 -G dba -g oinstall -s /usr/bin/bash -m -p 102 oracle
useradd -u 103 -G dba -g oinstall -s /usr/bin/bash -m oragrid

Verify that the users are created appropriately

[root@orczone1]# id -a oracle
uid=101(oracle) gid=103(oinstall) groups=102(dba)
[root@orczone1]# id -a oragrid
uid=103(oragrid) gid=103(oinstall) groups=102(dba)

According to OFA guidelines, Oracle going to be installed in
/opt/app/oracle

Oracle install also recommends that only ORACLE_BASE is set. I'm also going to be installing Grid software and Db software as the same user.

Bash Profile for the oracle User

PS1="[\u@\h]$ "
ORACLE_BASE=/opt/app/oracle
LD_LIBRARY_PATH=/usr/lib:/lib:$LD_LIBRARY_PATH
export SQLPATH=/home/oracle

export ORACLE_BASE ORACLE_SID LD_LIBRARY_PATH

export PATH=$CRS_HOME/bin:$ORACLE_HOME/bin:/usr/sbin:/usr/bin:/usr/ucb:/usr/local/bin:/bin/sbin:$PATH

umask 022
ulimit -n 65536

export EDITOR=vi
export editor=vi
export PS1

NTP
My /etc/inet/ntp.conf file is this (in the Global Zone):

server 10.27.1.254
driftfile /etc/ntp.drift
slewalways yes
disable p11

Grid Software install

1. Create /opt/app/11.2.0/grid on both nodes (orczone1 & orczone2).
2. Create /opt/app/oracle on both nodes
3. I created the "app" directory and let oracle have rwx permissions on it.
4. Add the SCAN Ips as well as the virtual IPs to DNS

Orczone1.cctdev.com
Orczone1-vip.cctdev.com

Orczone2.cctdev.com
Orczone2-vip.cctdev.com

Orczone-cluster.cctdev.com -- this was the SCAN IP.

5. Set the following in /etc/system for the global zone, to avoid annoyances in the non-global zone. I use these blades only for Oracle, so it works for me!

set rlim_fd_max=65636
set rlim_fd_cur=4096

5. During install of GRID software, oracle will try to plumb IPs and if you're not using Exclusive IPs in the non-global zone, it will fail. This is where I stopped and switched to LDOMs.


Issue Log:

One of the things that are needed are a Public & Private Interface.

When plumbing them, make sure that the private interface stays private and does not get added to your "default gateway". If that happens, traffic has a tendency to get "lost" and when you try to reboot zones or even the server, you may lose connectivity with other nodes. The way I did this, was to not add a defrouter in the zonecfg for the private interface.

When creating a Zone for the Private interface, do not set the default router for that NIC unless you're using VLAN tagging and such.

The netstat -rn should look something similar to this

Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 10.25.20.5 UG 1 27
default 10.25.20.5 UG 1 4 e1000g2
10.0.0.0 10.26.6.202 U 1 16 igb0
10.25.20.0 10.25.23.137 U 1 3 e1000g2
192.168.204.0 192.168.204.7 U 1 0 e1000g0
224.0.0.0 10.25.23.137 U 1 0 e1000g2
127.0.0.1 127.0.0.1 UH 5 92 lo0