Sunday, July 28, 2013

Downloading a sites certificate with openssl

I keep running into an issue where I need to get a server certificate sometimes, and its not always accessible via the browser.

When I need it from the server  - this handy comment on this site worked like a charm http://serverfault.com/questions/139728/how-to-download-ssl-certificate-from-a-website


echo -n | openssl s_client -connect HOST:PORTNUMBER | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/$SERVERNAME.cert
That will save the certificate to /tmp/$SERVERNAME.cert.
You can use -showcerts if you want to download all the certificates in the chain. But if you just want to download the server certificate, there is no need to specify -showcertsecho -n gives a response to the server, so that the connection is released
sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' removes information about the certificate chain. This is the preferred format to import the certificate into other keystores.


Thanks to the person who gave that comment, it has helped me immensley.

Weblogic - App having javax.net.ssl.SSLHandshakeException

If you use Weblogic and have an application that connects to the outside world via HTTPS, you could be in for a treat.

If you have the DemoIdentityTrustStore configured or any keystore configured and the JDK keystore configured, it looks like Weblogic randomly decides which keystore for Trust to use.

We ran into an issue, where the application would randomly complain of invalid cert chain similar to this:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)

After many many days of troubleshooting, wondering why even thoguh the java CACERTS had the correct certs, I decided to try accessing the site with the DemoTrust keystore instead.  Whola...the error manifested itself again.

What is insanely annoying is that this is not repeatable in WLS.  Every restart seems to have a different behavior.

To import a certificate you already have into the keystore - use the "keytool" utility.


cd $JAVA_HOME/jre/lib/security
 keytool -import -alias VerisignC3PPCA   -keystore cacerts -trustcacerts -file /root/getrootcert.cer


I found this great piece of code online (InstallCert.java)  (below) which helped me troubleshoot.

If you want to use the Weblogic Identity store - below is some useful data.  You will need to modify the below code to point to this JKS if you want to (change the "File file = new ... " block) .  To add the certificate, you can do it using "keytool" as well if you have the certificates you care about.

This table was very neatly done at : http://itguykelly.wordpress.com/2010/05/20/default-weblogic-keystore-passwordpassphrase/ which I've borrowed.


Property
Value
Trust store location
%ORACLE_HOME%/weblogic/wlserver_10.3/ server/lib/DemoTrust.jks
Trust store password
DemoTrustKeyStorePassPhrase
Key store location
%ORACLE_HOME%/weblogic/wlserver_10.3/ server/lib/DemoIdentity.jks
Key store password
DemoIdentityKeyStorePassPhrase
Private key password
DemoIdentityPassPhrase
Property
Value
Trust store location
%ORACLE_HOME%/weblogic/wlserver_10.3/ server/lib/DemoTrust.jks
Trust store password
DemoTrustKeyStorePassPhrase
Key store location
%ORACLE_HOME%/weblogic/wlserver_10.3/ server/lib/DemoIdentity.jks
Key store password
DemoIdentityKeyStorePassPhrase
Private key password
DemoIdentityPassPhrase


Copy and paste the below code into a file called "InstallCert.java"
javac InstallCert.java
java InstallCert.java



/*
 * Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *
 *   - Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *
 *   - Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *
 *   - Neither the name of Sun Microsystems nor the names of its
 *     contributors may be used to endorse or promote products derived
 *     from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
 * IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */
/**
 * Originally from:
 * http://blogs.sun.com/andreas/resource/InstallCert.java
 * Use:
 * java InstallCert hostname
 * Example:
 *% java InstallCert ecc.fedora.redhat.com
 */

import javax.net.ssl.*;
import java.io.*;
import java.security.KeyStore;
import java.security.MessageDigest;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;

/**
 * Class used to add the server's certificate to the KeyStore
 * with your trusted certificates.
 */
public class InstallCert {

    public static void main(String[] args) throws Exception {
        String host;
        int port;
        char[] passphrase;
        if ((args.length == 1) || (args.length == 2)) {
            String[] c = args[0].split(":");
            host = c[0];
            port = (c.length == 1) ? 443 : Integer.parseInt(c[1]);
            String p = (args.length == 1) ? "changeit" : args[1];
            passphrase = p.toCharArray();
        } else {
            System.out.println("Usage: java InstallCert [:port] [passphrase]");
            return;
        }

        File file = new File("jssecacerts");
        if (file.isFile() == false) {
            char SEP = File.separatorChar;
            File dir = new File(System.getProperty("java.home") + SEP
                    + "lib" + SEP + "security");
            file = new File(dir, "jssecacerts");
            if (file.isFile() == false) {
                file = new File(dir, "cacerts");
            }
        }
        System.out.println("Loading KeyStore " + file + "...");
        InputStream in = new FileInputStream(file);
        KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
        ks.load(in, passphrase);
        in.close();

        SSLContext context = SSLContext.getInstance("TLS");
        TrustManagerFactory tmf =
                TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
        tmf.init(ks);
        X509TrustManager defaultTrustManager = (X509TrustManager) tmf.getTrustManagers()[0];
        SavingTrustManager tm = new SavingTrustManager(defaultTrustManager);
        context.init(null, new TrustManager[]{tm}, null);
        SSLSocketFactory factory = context.getSocketFactory();

        System.out.println("Opening connection to " + host + ":" + port + "...");
        SSLSocket socket = (SSLSocket) factory.createSocket(host, port);
        socket.setSoTimeout(10000);
        try {
            System.out.println("Starting SSL handshake...");
            socket.startHandshake();
            socket.close();
            System.out.println();
            System.out.println("No errors, certificate is already trusted");
        } catch (SSLException e) {
            System.out.println();
            e.printStackTrace(System.out);
        }

        X509Certificate[] chain = tm.chain;
        if (chain == null) {
            System.out.println("Could not obtain server certificate chain");
            return;
        }

        BufferedReader reader =
                new BufferedReader(new InputStreamReader(System.in));

        System.out.println();
        System.out.println("Server sent " + chain.length + " certificate(s):");
        System.out.println();
        MessageDigest sha1 = MessageDigest.getInstance("SHA1");
        MessageDigest md5 = MessageDigest.getInstance("MD5");
        for (int i = 0; i < chain.length; i++) {
            X509Certificate cert = chain[i];
            System.out.println
                    (" " + (i + 1) + " Subject " + cert.getSubjectDN());
            System.out.println("   Issuer  " + cert.getIssuerDN());
            sha1.update(cert.getEncoded());
            System.out.println("   sha1    " + toHexString(sha1.digest()));
            md5.update(cert.getEncoded());
            System.out.println("   md5     " + toHexString(md5.digest()));
            System.out.println();
        }

        System.out.println("Enter certificate to add to trusted keystore or 'q' to quit: [1]");
        String line = reader.readLine().trim();
        int k;
        try {
            k = (line.length() == 0) ? 0 : Integer.parseInt(line) - 1;
        } catch (NumberFormatException e) {
            System.out.println("KeyStore not changed");
            return;
        }

        X509Certificate cert = chain[k];
        String alias = host + "-" + (k + 1);
        ks.setCertificateEntry(alias, cert);

        OutputStream out = new FileOutputStream("jssecacerts");
        ks.store(out, passphrase);
        out.close();

        System.out.println();
        System.out.println(cert);
        System.out.println();
        System.out.println
                ("Added certificate to keystore 'jssecacerts' using alias '"
                        + alias + "'");
    }

    private static final char[] HEXDIGITS = "0123456789abcdef".toCharArray();

    private static String toHexString(byte[] bytes) {
        StringBuilder sb = new StringBuilder(bytes.length * 3);
        for (int b : bytes) {
            b &= 0xff;
            sb.append(HEXDIGITS[b >> 4]);
            sb.append(HEXDIGITS[b & 15]);
            sb.append(' ');
        }
        return sb.toString();
    }

    private static class SavingTrustManager implements X509TrustManager {

        private final X509TrustManager tm;
        private X509Certificate[] chain;

        SavingTrustManager(X509TrustManager tm) {
            this.tm = tm;
        }

        public X509Certificate[] getAcceptedIssuers() {
            throw new UnsupportedOperationException();
        }

        public void checkClientTrusted(X509Certificate[] chain, String authType)
                throws CertificateException {
            throw new UnsupportedOperationException();
        }

        public void checkServerTrusted(X509Certificate[] chain, String authType)
                throws CertificateException {
            this.chain = chain;
            tm.checkServerTrusted(chain, authType);
        }
    }
}

Monday, July 08, 2013

Apache 64-bit on Solaris 10 & Weblogic

Solaris 10 comes with Apache 32- bit by default.  This was a surprise.

My task here is to get Apache 64-bit and then hook it up to Weblogic on the SSL channel.  I will detail the Apache + Weblogic SSL communication.  Setting up Weblogic to listen on SSL is a different post, that I may or may not write up :)

I will be installing apache at /usr/local/apache (Many folks like to install it at /usr/local/apache2 etc, but I'm keeping it simple here)

I have a staging directory to compile / install the required packages.  This is a Whole Root zone, with Solaris 10 fully installed with all the packages on the Global Zone.

I need
0. Apache 2.2.4 -- I tried Apache 2.4 and as of this writing is not supported with the mod_wl.so library.
1.  PCRE (Perl-Compatible Regular Expressions Library)
2. Apr
3. Apr-utils

A gist of what is required is available on Apache's documentation site ( Apache Documentation for Installation )

For PCRE, I downloaded it from: PCRE
Download APR & APR-UTIL from here.

Initial settings on the terminal
export CFLAGS="-m64"
export LDFLAGS="-L/usr/sfw/lib/sparcv9"
export LD_LIBRARY_PATH=/usr/sfw/lib/sparcv9:/usr/local/lib/sparcv9 
the CFLAGS setting is  really important, otherwise, you wont get it compiled in 64 bit.  Make sure your LD_LIBRARY_PATH and LDFLAGS point to the accurate directories.

Compiling PCRE

This installs PCRE in /usr/local/pcre

/root/apache-compilation/pcre-8.33
./configure --disable-cpp CFLAGS="-m64 -g"
make
make install
Compiling APR

/root/apache-compilation/apr-1.4.8
./configure
make
make install

APR gets installed in /usr/local/apr.  The CFLAGS setting set in the environment should carry on.

Compiling APR-UTIL

/root/apache-compilation/apr-util-1.5.2
./configure --with-apr=/usr/local/apr
make
make install

Compiling Apache
I use the prefix /usr/local/apache  - but some people prefer to use "apache2".  Make it your own.  The prefix designates where apache gets installed.

/root/apache-compilation/httpd-2.2.24
./configure --with-apr=/usr/local/apr --with-apr-util=/usr/local/apr --prefix=/usr/local/apache --enable-mods-shared=most --enable-ssl --with-ssl=/usr/sfw
make
make install

After the installation, I like to test to make sure its 64 bit (as I've gotten bitten a number of times).

# cd /usr/local/apache/modules
# file mod_vhost_alias.so
mod_vhost_alias.so:     ELF 64-bit MSB dynamic lib SPARCV9 Version 1, dynamically linked, not stripped, no debugging information available

That tells you its a 64 Bit file.

To get Weblogic working, I copied the wls 1.1 plugins *.so files (which are only 64 bit) to the /usr/local/apache/modules directory.  Then followed the instructions from Oracle here

Good luck.  Comments are welcome.


Sunday, February 03, 2013

Solaris 11....network interfaces..ugh!

Ok - I'm not sure who was responsible for the Solaris 11 redesign - but someone should take him out and beat him with a stick.

Why on earth would you actually change how to configure the network interfaces and such and make it more complicated?  Unix is a text based system for administrative activities.  One would expect you make it EASIER not more complex.

For Eg:
To create a new tagged VLAN interface in Solaris 10

ifconfig *1000+networkid plumb - gives you the network interface

Solaris 11

root@solaris:~#  dladm create-vlan -l net2 -v 11
root@solaris: ifconfig net11002 plumb  netmask 255.255.255.0 broadcast + up

So basically you've doubled the commands to achieve the same damn thing.

A lot of things have gone this way.  Apparently now, you cant have an already plumbed interface and can't add it to an ipmp group either (IP Multipathing).  So if I had 2 interfaces and needed to add it to an ipmp group, with one of them having the IP which I'm working from - its a lot more convoluted.

I'm thinking may be I should just stick with Solaris 10 for eternity.  Who the hell does such stuff (Linux is no better - but come on!) 

Monday, December 31, 2012

2011 -- a (delayed) recap...

I just realized, I didnt write up my recap for 2011.  Tells you how busy my year has been and how fast time has flown that it's April 30th, 2012.

The year was so busy, I dont remember much.

January 2011 rolled around -- we had our little one in Sept 2010, so time was really flying.  I barely noticed it as I was working constantly, and even from the delivery room I should mention.  Our project at work, finally went live in January, delayed, but went live.

Roll forward to March where we did a stability release for the product, which made my life so much saner.  I wasn't being called in the middle of the night to troubleshoot the product anymore!

We went to July / Aug India in 2011, so it was really great to meet all my family and celebrate.  I worked out of the Mumbai office for a week which was interesting to the least.  Definitely a change in culture, and tone compared to working in the US.  Especially true, since I was working UK hours, so that I could talk to my colleagues back home, and still go back to the hotel at a decent hour!  Mumbai has changed significantly, yet  landing in Mumbai, the view from the air gave me a shock of my life.  Not to mention just walking down the road from my hotel, I found a bicycle repair guy living in literally a treehouse, and he had a ladder going up to his "house".  What was unbelievable was, he had 2 kids who were running around!!!  Really -- gives you a pause to think - when we the educated and probably middle class think many times about the future of our children - he doesnt -  he just has his kids as a matter of fact!

August was great, as we celebrated Aarathi's birthday (my niece).  I haven't been around for one her birthdays since 1999 - so this was excellent.  Lots of fun!  She'll probably get married soon, so this was one of the last birthdays we were all able to play pranks on and have a "kiddish" time :)

We also had a couple of ceremonies for Vaibhav ( the new one :) ).  Those occasions brought together all our friends and relatives at one place, and was great to meet all of them.

We came back to the US, and then it's back to the grind.  Pranav started First Grade, Vaibhav turned one in September, and we did his birthday party at our friend Suresh's community hall.  That was fun too.  My Brother-in-law, sister-in-law and my in-laws were all here.  So it was enjoyable and fun.  One of the most memorable moments was when Vaibhav barely a year old was being carried by our friends and he pointed to a photograph of his mom on the camera and pointed his finger out back to the hall -- his intention " take my to my mom!".  That was unbelievable.  The power of kids without even saying a word :)

Ofcourse, we also celebrated my wife's birthday in September.

My birthday was low key again in November.  December rolled around and we're off into 2012.  

2012 is almost over...and boy am I glad....



I try to do this every year...where I think about the year...so here goes this one!

Started out with my wife having surgery for a relatively minor nail issue, turned out into a 3 week saga.  The year then dragged on with work - where Travelex Global Business Payments separated from Travelex, and we were busy integrating with Western Union.

Ofcourse, I can't forget our 8 years anniversary we hit on Jan 24th.  Thats always the pleasant beginning for me, where I think about the wonderful time of the last 8 years with my wife.

This March - unbelievably - we got our green card!

The next memorable event was in April when Pranav turned 7.  A small party at home with his friends - which we all enjoyed.

My parents came to the US in June - since my dad retired in 2011 December, he's been getting used to not having to get up and get to work everyday!  He moved back to India, to set up our house in Chennai. Then in the US on June 8th.  Mom & Dad stayed till 15th October - so thats always fun!  Vaibhav got used to his grandparents, and Pranav had a blast.  Finally Lavanya & I were able to take a weekend and go off to by ourselves.  That was a short and fun weekend we had the weekend of the 15th Sept, 2012!  This was also our way of specially celebrating Lavanya's birthday :).  We had a family Birthday for her after!

We celebrated Vaibhav turning 2 in a small fashion as well - we expect we'll be celebrating his 3rd birthday in a much bigger fashion.

We went to Diya on Oct 12th to celebrate Dad's birthday and then they started getting ready to head back to Chennai!

Ofcourse, right after they came, I had to have surgery for a pinched nerve on the right hand - bummer - and then right after they left Nov 7th - the same thing on the left hand - big bummer.

Then came by Birthday - Lavanya gave me a Mac Book Pro for my birthday - that I enjoy very much!

There is also something that happened in December, that I can write about, but it definitely changed my perspective about how much my work is appreciated!  It was a good feeling!

This year has been one of the most roller coaster years I've ever had!!!  Ever!  Most of it has to do with family health issues....started with my wife, then me twice!!!  Three surgeries this year is enough!  I'm hoping to have a much more pleasant 2013.

If you're reading this - Happy New Year to you too!

PS: Photos hopefully to follow soon....

Tuesday, November 06, 2012

Network Interfaces with Vlan Tags in Solaris 11

If you're trying to plumb a NIC with a VLAN tag in Solaris 11, its not straight forward anymore!

First, Look for the interfaces that you have.


root@solaris:~# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net1              Ethernet             unknown    0      unknown   igb1
net3              Ethernet             unknown    10000  half      qlge3
net5              Ethernet             unknown    10000  half      qlge2
net0              Ethernet             unknown    0      unknown   igb0
net2              Ethernet             unknown    10000  full      qlge0
net9              Ethernet             unknown    10000  half      qlge1



This is a brand new install and T4 blade with the QLogic 10Gb CNA cards.  I had to figure out manually which one of these were connected to what.  

However, first things first.  To use a Tagged VLAN for the network interface, first use dladm create-vlan to create a new network interface.


root@solaris:~#  dladm create-vlan -l net2 -v 11
root@solaris:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net1                phys      1500   unknown  --
net3                phys      1500   unknown  --
net5                phys      1500   unknown  --
net0                phys      1500   up       --
net2                phys      1500   up       --
net9                phys      1500   up       --
net11002            vlan      1500   up       net2

 Next plumb the interface


root@solaris:~#  ifconfig net11002 plumb  netmask 255.255.255.0 broadcast + up

You should now be able to SSH to the IP (if the default gateway is set right).  Run netstat -rn to see the default gateway.  If it's not set - "route add default " will do the trick.

If it is the wrong port, then you can just as easily run


root@solaris:~# dladm delete-vlan net11002

This will remove the new interface.





Monday, January 30, 2012

Solaris 11, Oracle 11g & EM12C

I've just started using Solaris 11, this is my first ever install of it.

Solaris 11

Downloaded the text installer, mounted it as lofi, and then created an LDOM by booting off the ISO. I've not tried the Automated Installer (AI) yet.

I realize that there isn't much information here, but really, there wasn't much to do.

Add ISO to LDOM:

ldm add-vdsdev /sol11/sol-11-1111-text-sparc.iso sol11iso@primary-vds0
ldm add-vdisk id=1 sol11iso sol11iso@primary-vds0 orcem12c


Show Disks on LDOM Ok prompt:
{0} ok show-disks
a) /virtual-devices@100/channel-devices@200/disk@1
b) /virtual-devices@100/channel-devices@200/disk@0
c) /iscsi-hba/disk
q) NO SELECTION
Enter Selection, q to quit: a
/virtual-devices@100/channel-devices@200/disk@1 has been selected.


Boot from Disk 1 (as thats what the ID is for the ISO). Also choose slice "f"

{0} ok boot /virtual-devices@100/channel-devices@200/disk@1:f

Follow the installer along and give appropriate credentials. A user will need to be created, as root login is disabled by default.

To allow root login:

1. Change "PermitRootLogin" to "yes" in the /etc/ssh/sshd_config file.
2. Comment out "#CONSOLE=/dev/console" in the /etc/default/login file to allow non-console root login.
3. Switch the role of the root user "rolemod -K type=normal root"


Oracle 11g (11.2.0.3 Install)

Prerequisite

Before you start the install of EM 12c, update the Solaris 11 packages, or the OEM 12c installer will probably throw an error.

If this is a fresh install of Solaris 11, make sure to update the Publisher.  The directions are at the Oracle Solaris 11 Certificate Site.

Once you download the certificates then do the following (from Oracle's site)
  1. Download the provided key and certificate files, called Oracle_Solaris_11_Support.key.pem andOracle_Solaris_11_Support.certificate.pem using the buttons above. Don't worry if you get logged out, or lose the files. You can come back to this site later and re-download them. We'll assume that you downloaded these files into your Desktop folder,~/Desktop/.
  2. Use the following comands to make a directory inside of /var/pkg to store the key and certificate, and copy the key and certificate into this directory. The key files are kept by reference, so if the files become inaccessible to the packaging system, you will encounter errors. Here is how to do it:
     $ sudo mkdir -m 0755 -p /var/pkg/ssl
     $ sudo cp -i ~/Desktop/Oracle_Solaris_11_Support.key.pem /var/pkg/ssl
    
     $ sudo cp -i ~/Desktop/Oracle_Solaris_11_Support.certificate.pem /var/pkg/ssl
     
  3. Add the publisher:
     $ sudo pkg set-publisher \
                -k /var/pkg/ssl/Oracle_Solaris_11_Support.key.pem \
                -c /var/pkg/ssl/Oracle_Solaris_11_Support.certificate.pem \
                -G '*' -g https://pkg.oracle.com/solaris/support/ solaris
     
  4. Check your publisher settings, there should be no unrelated mirrors set up. To check for any set up mirrors invoke the following command:
     $ pkg publisher solaris | grep Mirror
     
     
    If the output is empty you are all set. If not remove unrelated mirrors by running:
     $ sudo pkg set-publisher -M http://mirror1.x.com -M http://mirror2.y.com ... solaris
     
     
  5. To see the packages supplied by this publisher, try:
     $ pkg list -a 'pkg://solaris/*'

Next update packages. Oracle site lists the directions : here.

Steps are as follows:
  1. First, we use pkg update with the --accept option to update a small set of system packages and then we reboot:

    # pkg update --accept
    # reboot
    
  2. (SPARC only) If you are running on a SPARC system and have any Oracle Solaris Zones installed, you will need to perform an additional step at this stage. For each zone installed on the system, you will need to remove thepkg:/system/ldoms/ldomsmanager package as follows:

    # for z in `zoneadm list`; do zlogin $z pkg uninstall ldomsmanager; done
    
  3. Next, update the IPS package itself:

    # pkg update pkg:/package/pkg
    
  4. Last, update the rest of the system packages and reboot the system.

    To update, we use the --accept flag to pkg update to agree to and accept the terms of the licenses that are being updated. We also use the --be-name flag to provide a new name to the boot environment that will be created.
    # pkg update --be-name s11.1ga --accept
    # reboot


First things first, rebuild the Package index and then get the X packages installed:

pkg rebuild-index

Test to make sure you have all the required packages
pkginfo -i SUNWarc SUNWbtool SUNWhea SUNWlibm SUNWlibms SUNWpool SUNWpoolr SUNWsprot SUNWtoo SUNWuiu8 SUNWfont-xorg-core SUNWfont-xorg-iso8859-1 SUNWmfrun SUNWxorg-client-programs SUNWxorg-clientlibs SUNWxwfsw SUNWxwplt

Install packages that are needed.

pkg install compatibility/packages/SUNWxwplt SUNWmfrun SUNWarc SUNWhea SUNWlibm

pkg install x11/xclock

Add Swap if you dont have enough (You'll need about 16Gb depending upon the RAM you've allocated to the LDOM).

zfs create -V 14G swpool/swap14G
swap -a /dev/zvol/dsk/swpool/swap14G

Edit the /etc/vfstab to add the new swapfile.

/dev/zvol/dsk/swpool/swap14G - - swap - no -

I found that Timezone wasn't set. In Solaris 11, all the properties have moved to SMF.
root@orcem12c:~# svcs -a | grep environ
online 17:45:30 svc:/system/environment:init
root@orcem12c:~# svccfg -s system/environment:init
svc:/system/environment:init> listprop environment/TZ astring
svc:/system/environment:init> setprop environment/TZ=US/Eastern
svc:/system/environment:init> validate
svc:/system/environment:init> end

Install Oracle 11g. I choose to use the file storage instead of ASM for the Db disks.

Create new Pool for /opt/app/Oracle & /opt/app/oradata (to hold all the Software & Db files respectively)
zpool create -f -m /opt/app/Oracle swpool c2d1s0
zpool create -f -m /opt/app/oradata dbpool c2d2s0

Requirements for the Db:

1. 2G SGA.
2. Redo log files must atleast be 300Mb with 3 Redo logs.

SQL> alter database
2 add logfile ('/opt/app/oradata/dbfiles/EM12C/em12redo01.log') Size 300M;

Database altered.

SQL> alter database
2 add logfile ('/opt/app/oradata/dbfiles/EM12C/em12redo02.log') Size 300M;

Database altered.

SQL> alter database
2 add logfile ('/opt/app/oradata/dbfiles/EM12C/em12redo03.log') Size 300M;

SQL> alter system switch logfile;

If adding logfiles, you can drop the old ones. If the Db complains that it cant drop a logfile (if you need to), run "alter system checkpoint". You cannot drop a logfile in "Current" status.

SQL> select * from v$log;

GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC
---------- ---------- ---------- ---------- ---------- ---------- ---
STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME
---------------- ------------- --------- ------------ ---------
4 1 13 314572800 512 1 NO
INACTIVE 1089635 29-JAN-12 1089642 29-JAN-12

5 1 14 314572800 512 1 NO
CURRENT 1089642 29-JAN-12 2.8147E+14

6 1 0 314572800 512 1 YES
UNUSED 0 0


I had to make the following changes:

SQL> alter system set session_cached_cursors=300 scope=spfile;

System altered.

SQL> alter system set job_queue_processes=20;

System altered.

SQL> alter system set shared_pool_size=600M scope=spfile;

System altered.



EM12C (Cloud control) Install

Once I made all the above Db changes, I was able to get through the installer using the "Simple" option.

The only error message after all this, was that there wasn't an EM tablespace with an autoextend datafile. Since I didnt create a tablespace, so I proceeded.


Once the install finishes, I logged in with https://:port/em. Login with the user sysman.



Tuesday, October 18, 2011

Ops Center DHCP Hell....

If you've set up Ops Center and are wondering why the machines wont jump with DHCP, make sure you choose "ISC" instead of Solaris as your DHCP server.

Also a good link to follow is this note OpsCenter and ISC DHCP server blog!

Wednesday, October 05, 2011

At Oracle World...Past 2 days...

There were quite a few announcements at Oracle world! None of them interesting!

Enterprise Manager 12c has been released.

Ops Center is free

Solaris 11 is almost there, we can download an early release and install it. The big shocker - there is no way to go from 10 to 11 without a backup, reinstall and restore!\

More to come as I have time to write up.

Friday, September 30, 2011

Release of Solaris (Sparc or x86?)

I got a handy tip from an Oracle person the other day....to make sure what solaris release and version the solaris cd is or jumpstart location is, check the SUNWsolnm/reloc/etc/release file.


[root@host1]# cat SUNWsolnm/reloc/etc/release
Oracle Solaris 10 9/10 s10s_u9wos_14a SPARC
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Assembled 11 August 2010
[root@host1]#

TFTP Boot error Access Violation

When accessing a tftp server clients receive "Error code 2: Access violation"
due to ownership & permission of the /tftpboot directory.

Steps to Follow
in.tftpd runs as user nobody. nobody needs rwx permissions to the /tftpboot
directory
chmod 766 /tftpboot
chown nobody:other /tftpboot

Product
Network Management/Conn

Friday, September 16, 2011

Oracle 11g RAC in LDOMs (Oracle VM for Sparc)


Setup
1. 2 T3-1B blades on different chassis. Patch the firmware. Install LDOM 2.1 software.
2. SAN Connected storage via MPXIO (EMC SAN)
3. Solaris 10/9 installed with End User Option.



Create install User
useradd -u 0 -o -g 1 -c "Install user" -d / -s /bin/true install

Patch LDOM.

Download the following Patches from oracle
• 117837-05: C++ compiler optimizer patch
• 117846-08: C++ compiler Optimization patch
• 118682-01
• 127111-02 SunOS 5.10: libc patch (this should already exist if LDOM is patched)
• 137111-04 SunOS 5.10: kernel patch (this should already exist if LDOM is patched)

Profile Settings

• On a new system, set the following in /etc/profile
a. MANPATH=/usr/man:/usr/local/man:/usr/sfw/man:$MANPATH

• Create the oracle required users & groups
groupadd -g 102 dba
groupadd -g 103 oinstall
projadd -p 102 -c "Oracle Project" group.dba
useradd -u 101 -g dba -G oinstall -s /usr/bin/bash -c "Oracle User" -m –p 102 oracle

• Set the Oracle password

passwd -r files oracle

• Add the following entries to /etc/system for a 4G SGA or less. Shmmax should always be greater than SGA size.

set noexec_user_stack=1
set semsys:seminfo_semmni=100
set semsys:seminfo_semmns=1024
set semsys:seminfo_semmsl=256
set semsys:seminfo_semvmx=32767
set shmsys:shminfo_shmmax=4294967296
set shmsys:shminfo_shmmni=100
set rlim_fd_max=65636
set rlim_fd_cur=4096

Reboot the system after the above /etc/system have been made.

Alternate way

Run prctl to make changes to the existing install so that a reboot is not required. These should be run as root. These changes don’t persist after a reboot.
prctl -n project.max-shm-memory -v 4gb -r -i project user.root
prctl -n project.max-sem-ids -v 256 -r -i project user.root

Project Settings

Make changes to the project so that changes persist for shm memory & sem ids (4G or less SGA). These should be run as root.

projmod -sK "project.max-shm-memory=(privileged,4G,deny)" group.dba
projmod -sK "project.max-sem-ids=(privileged,256,deny)" group.dba

/etc/project looks like this after changes:

[root@orc11gtest]# cat /etc/project
system:0::::
user.root:1::::
noproject:2::::
default:3::::
group.staff:10::::
group.dba:102:Oracle Project:oracle::project.max-sem-ids=(privileged,256,deny);project.max-shm-memory=(privileged,4294967296,deny)

Verify the changes applied to the oracle user as below.

[oracle@orc11gtest]$ prctl -n project.max-shm-memory -i process $$
process: 2834: -bash
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-shm-memory
privileged 4.00GB - deny -
[oracle@orc11gtest]$ prctl -n project.max-sem-ids -i process $$
process: 2834: -bash
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-sem-ids
privileged 256 - deny

Patch Check:

patchadd -p | grep 125555-03
patchadd -p | grep 140796-01
patchadd -p | grep 140899-01
patchadd -p | grep 141016-01
patchadd -p | grep 139555-08
patchadd -p | grep 141414-02 (Replaced by 141444-09)
patchadd -p | grep 141736-05
patchadd -p | grep 120753-06
patchadd -p | grep 139574-03
patchadd -p | grep 141414-10
patchadd -p | grep 141736-05


I'm using Ops center to create LDOMs to make my life a little easier. If you're doing this, the Vnets sometimes will be different between the 2 blades. Go into the "/guests//servercfg.xml" and fix the
< idx >3< /idx >

number to match on both nodes.


NTP Configuration
Make sure to set the /etc/inet/ntp.conf with slewalways and disable p11.

server < serverip >
driftfile /etc/ntp.drift
slewalways yes
disable pll


Grid Software Install

ON each Control Domain -- add the disks for OVD, OCR & Data to the LDOMs. Make sure to add them in the same order, and set the same ID. A good way to make sure everything is the same on both nodes, is to also volume label the disks on the control / IO domain before assigning it to the LDOM.

Have xwin running and kick off the installer. Before you kick off the installer, its a good idea to run cluster verify tool.
./runcluvfy.sh stage -pre crsinst -n orczone1,orczone2 -verbose

Install any packages that are missing, fix the swap or any such thing, the MTUs etc, and you're ready to roll.

To add Swap Space when using ZFS

To create swap space on a ZFS volume (created 2 2g volumes):
zfs create -V 2gb rpool/swap2g
zfs create -V 2gb orcinstall/swap2g


Add it to swap
swap -a /dev/zvol/dsk/rpool/swap2g
swap -a /dev/zvol/dsk/orcinstall/swap2g

Add it to /etc/vfstab
/dev/zvol/dsk/rpool/swap2g - - swap - no -
/dev/zvol/dsk/orcinstall/swap2g - - swap - no -




Install / Troubleshooting Notes

1. A group called "DATA" was created by a previous install, and even though I zapped the disks, the group didnt go away. To drop the diskgroup (which wouldn't mount or come online) I had to do this:
drop diskgroup data force including contents;
2. If you're having trouble with ulimits and such, this link: Max user processes is pretty helpful.

Wednesday, September 14, 2011

Oracle 11g RAC in a zone - Partial instructions.

This is a post thats a WIP. I will be adding to this as I keep finding new things, and hopefully a full step by step doc by the end of this.

I had to stop this because of the requirement of EXCLUSIVE IPs for the non-global zone. You can only have one Exclusive interface per vlan tag per server.

That is, if you have VLAN tag 3 for your public interface, and you want to create another RAC cluster on the same blade, you'll have to get another vlan from your network team. This seems unworkable, as all our Dbs are in the same VLAN.

I've just begun setting up 2 T3-Blades to use zones for Oracle RAC cluster.

Setup:
2 T3-1B Sparc blades with 64Gb of RAM.
Oracle 11gR2 downloaded
Oracle 11gR2 Grid Download
OS installed with "Full Install with OEM"
OS patched with the latest patch cluster.
Use EMC SAN for disk

MPXIO
Enable MPXIO if you're not using any vendor provided software to connect to the SAN
Using MPXIO to connect to the EMC CX-500
change "mpxio-disable" to "no" in ./kernel/drv/fp.conf

Disks
Using ZFS for the global zone with internal disks.

All Oracle zones will be created using mirrored disks from the SAN.

I intend on creating a zone, configuring the basic setup (DNS, oracle user, oracle download copied over) and take

[root@ch0bl7qadb1w]# zpool list
NAME SIZE ALLOC FREE CAP HEALTH ALTROOT
orcpool 248G 11.0G 237G 4% ONLINE -
rpool 278G 7.99G 270G 2% ONLINE -
[root@ch0bl7qadb1w]# zpool status -v orcpool
pool: orcpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
orcpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t60060160733014002C5FAEED7BD9E011d0 ONLINE 0 0 0
c0t60060160733014002D5FAEED7BD9E011d0 ONLINE 0 0 0

errors: No known data errors

Creating Zones
Create the zone (orczone1), boot it and configure the DNS and such.

Zones have an annoying habit of setting auto-home. I normally run a script to disable auto-home. To allow the creation of the /home/oracle directory below, it's required to disable autohome.

Create the oracle project with group membership of dba.

projadd -p 102 -c "Oracle Project" group.dba

Create the dba (id: 102) oinstall (id: 103) groups

Create the Oracle, Oragrid users, the assign the oracle user belong to the oracle project.

useradd -u 101 -G dba -g oinstall -s /usr/bin/bash -m -p 102 oracle
useradd -u 103 -G dba -g oinstall -s /usr/bin/bash -m oragrid

Verify that the users are created appropriately

[root@orczone1]# id -a oracle
uid=101(oracle) gid=103(oinstall) groups=102(dba)
[root@orczone1]# id -a oragrid
uid=103(oragrid) gid=103(oinstall) groups=102(dba)


According to OFA guidelines, Oracle going to be installed in
/opt/app/oracle

Oracle install also recommends that only ORACLE_BASE is set. I'm also going to be installing Grid software and Db software as the same user.

Bash Profile for the oracle User


PS1="[\u@\h]$ "
ORACLE_BASE=/opt/app/oracle
LD_LIBRARY_PATH=/usr/lib:/lib:$LD_LIBRARY_PATH
export SQLPATH=/home/oracle

export ORACLE_BASE ORACLE_SID LD_LIBRARY_PATH

export PATH=$CRS_HOME/bin:$ORACLE_HOME/bin:/usr/sbin:/usr/bin:/usr/ucb:/usr/local/bin:/bin/sbin:$PATH

umask 022
ulimit -n 65536

export EDITOR=vi
export editor=vi
export PS1


NTP
My /etc/inet/ntp.conf file is this (in the Global Zone):
server 10.27.1.254
driftfile /etc/ntp.drift
slewalways yes
disable p11



Grid Software install

1. Create /opt/app/11.2.0/grid on both nodes (orczone1 & orczone2).
2. Create /opt/app/oracle on both nodes
3. I created the "app" directory and let oracle have rwx permissions on it.
4. Add the SCAN Ips as well as the virtual IPs to DNS
Orczone1.cctdev.com
Orczone1-vip.cctdev.com

Orczone2.cctdev.com
Orczone2-vip.cctdev.com

Orczone-cluster.cctdev.com -- this was the SCAN IP.
5. Set the following in /etc/system for the global zone, to avoid annoyances in the non-global zone. I use these blades only for Oracle, so it works for me!
set rlim_fd_max=65636
set rlim_fd_cur=4096

5. During install of GRID software, oracle will try to plumb IPs and if you're not using Exclusive IPs in the non-global zone, it will fail. This is where I stopped and switched to LDOMs.


Issue Log:

One of the things that are needed are a Public & Private Interface.

When plumbing them, make sure that the private interface stays private and does not get added to your "default gateway". If that happens, traffic has a tendency to get "lost" and when you try to reboot zones or even the server, you may lose connectivity with other nodes. The way I did this, was to not add a defrouter in the zonecfg for the private interface.

When creating a Zone for the Private interface, do not set the default router for that NIC unless you're using VLAN tagging and such.

The netstat -rn should look something similar to this
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ---------- ---------
default 10.25.20.5 UG 1 27
default 10.25.20.5 UG 1 4 e1000g2
10.0.0.0 10.26.6.202 U 1 16 igb0
10.25.20.0 10.25.23.137 U 1 3 e1000g2
192.168.204.0 192.168.204.7 U 1 0 e1000g0
224.0.0.0 10.25.23.137 U 1 0 e1000g2
127.0.0.1 127.0.0.1 UH 5 92 lo0

Wednesday, August 24, 2011

cacaoadm not working

I was trying to install Oracle Enterprise Ops Center and I kept getting a weird error which I couldn't find any information online about.



[root@host1:/]# /usr/sbin/cacaoadm
Cannot perform firstime initialisation and configuration.


Even reinstalling cacao didnt help (the packages were SUNWcacaodtrace & SUNWcacaort ).

Turns out the cacao.properties file was completely hosed. Here's where the file is located /etc/cacao/instances/default/private/cacao.properties

I ended up copying it from a different install, and it's working!!

Saturday, July 02, 2011

OEM 10g not starting ...Fix!

If you have Oracle Enterprise Manager that very annoyingly wont restart, though it's been perfectly working before you shut it down, and opmnctl complains that it cant start the HTTP server, you're probably running into the file size limit of the Apache logs.


[oragrid@admin30w]$ tail -20 /opt/oragrid/oms10g/Apache/Apache/logs/error_log
[Sat Jul 2 01:43:20 2011] [error] (79)Value too large for defined data type: could not open transfer log file /opt/oragrid/oms10g/Apache/Apache/logs/access_log.


If you see that message, it implies that the access log file is too large, typically over 2Gb. OEM by default doesn't rotate logs, so you need to either set it up, or rotate the logs manually.


oragrid@admin30w]$ ls -l /opt/oragrid/oms10g/Apache/Apache/logs/access_log
-rw-r---- 1 oragrid oinstall 2147483881 Apr 30 21:58 /opt/oragrid/oms10g/Apache/Apache/logs/access_log


To fix it, I just did a

cat /dev/null > /opt/oragrid/oms10g/Apache/Apache/logs/access_log


You can also set up log rotation this way.
Note 339819.1 from Metalink will give you detailed steps on the multiple ways to setup log rotation. However, this is supposed to be fixed in 11g.


1. Stop the OMS:

cd /opmn/bin
opmnctl stopall
2. Take a backup of the files :
/sysman/config/httpd_em.conf
/sysman/config/httpd_em.conf.template

NOTE: You must make the changes to both httpd_em.conf AND httpd_em.conf.template because any 'emctl secure' operation results in the http_em.conf file being regenerated using http_em.conf.template and any changes that had been made to httpd_em.conf earlier will be lost.
3. Edit the /sysman/config/httpd_em.conf and modify:

ErrorLog /Apache/Apache/logs/error_log
TransferLog /Apache/Apache/logs/access_log

TO

ErrorLog "|//Apache/Apache/bin/rotatelogs /Apache/Apache/logs/error_log 43200"
TransferLog "|//Apache/Apache/bin/rotatelogs /Apache/Apache/logs/access_log 43200"

where is the full path to your OMS install directory.
4. Edit the /sysman/config/httpd_em.conf.template and modify:

ErrorLog &ORACLE_HOME&/Apache/Apache/logs/error_log
TransferLog "&ORACLE_HOME&/Apache/Apache/logs/access_log"

To

ErrorLog "|&ORACLE_HOME&/Apache/Apache/bin/rotatelogs &ORACLE_HOME&/Apache/Apache/logs/error_log 43200"
TransferLog "|&ORACLE_HOME&/Apache/Apache/bin/rotatelogs &ORACLE_HOME&/Apache/Apache/logs/access_log 43200"
5. Delete the existing access_log and error_log files in the /Apache/Apache/log directory.

6. Re-start the OMS
cd /opmn/bin
opmnctl startall

Thursday, June 02, 2011

Registering a Sunone 7 node to the admin server

We have a number of SunONE 7 (now renamed to iPlanet) servers and the documentation online is fast fading, thanks to Oracle.

To register a node after install:

webserver7/bin

/wadm --host= --port= --user=admin

It will prompt for a password.
wadm> register-node

That should register the host successfully.

If you have renamed your server, navigate to the admin-server/config directory and fix the server.xml with the new hostname.

Thursday, February 24, 2011

Tablespace Utilization in Oracle

Taken from: http://vsbabu.org/oracle/sect03.html
Usage

TABLESPACE USAGE NOTES:
Tablespace Name - Name of the tablespace
Bytes Used - Size of the file in bytes
Bytes Free - Size of free space in bytes
Largest - Largest free space in bytes



select a.TABLESPACE_NAME,
a.BYTES bytes_used,
b.BYTES bytes_free,
b.largest,
round(((a.BYTES-b.BYTES)/a.BYTES)*100,2) percent_used
from
(
select TABLESPACE_NAME,
sum(BYTES) BYTES
from dba_data_files
group by TABLESPACE_NAME
)
a,
(
select TABLESPACE_NAME,
sum(BYTES) BYTES ,
max(BYTES) largest
from dba_free_space
group by TABLESPACE_NAME
)
b
where a.TABLESPACE_NAME=b.TABLESPACE_NAME
order by ((a.BYTES-b.BYTES)/a.BYTES) desc

Files modified between times

Sometimes, I get this task to find out files that changed between 2 dates. I never seem to remember it, so I figured it out (googling) and here it is. Hopefully someone else also can use it!

olddate="201001010001"
newdate="201012312359"
touch -t $olddate ./tmp/oldfile
touch -t $newdate ./tmp/newfile
find /path/to/directory -type f -newer ./tmp/oldfile ! -newer ./tmp/newfile

Monday, January 17, 2011

Good Bye 2010, Hello 2011

Where did 2010 go? Everything almost seems to be a blur...I spent almost 350 days working last year I think (yes, that includes, nights, weekends etc).

I'm going to try to recount some happy times ...

January as always is a happy time, as I celebrate my anniversary with my lovely wife. This time, we decided to go to Las Vegas as our friends were staying there and celebrate. A couple of nights at their house, and one night at MGM Grand. It was a good trip! Here is a picture I could find, of giving my wife a surprise gift. I had my Brother In Law bring a pair of ear rings from India, and then send it to my friends place in Las Vegas without telling Lavanya :) (Yes, thats her acting surprised :D)



The rest of the year moved quickly, as we announced that we were going to have our second child!

Ofcourse, who can forget the monstrous winter of 2009 / 2010!


Time quickly flew by after that. We had my uncle, aunt & mom come from India in May / June (I think). They went on an US Tour and Alaska cruise and returned.
We went to a great airshow in May. This was our first time ever, and unfortunately we were a little late, but got to see some awesome acrobatics. Next time, I think we're going to go there for the whole day. Pranav had a blast!




In June, Lavanya had her surprise "Seemantham" or Baby shower at our neighbours house.






Then my In-laws arrived in August to help with our upcoming kid.




September rolled around, and my second son -- Vaibhav Chandra decided he wanted to be born before his due date, and he was born on Sept 3rd.

My parents flew down on the 10th, and so our house was full with family and joy!


On a side note, I flew so often to Toronto this year, it was crazy. I think I made 6 trips in about a span of 3 months, often staying there a week at a time.

After all the excitement was over, we all went to Minneapolis for a week to visit my brother in law (I flew in from Toronto and returned to Toronto !)

November came around, and my birthday passed silently this year, without a party or much fan fare! I think its owed to my little one :)

December came around, and time just flew. I was super busy with work, going to Toronto once that month as well I think.

2010 was definitely a year to remember because of the birth of Vaibhav, but not many things else, as it was a year of constant work and no play! I look forward to a great 2011!