Sunday, August 17, 2014

Oracle EBS 12.2 Installer



I have just been trying to install EBS Suite 12.2 with 11.2.0.3/11.2.0.4 Grid.  It is the worst ever, most painful install ever.

The number of files to be downloaded, the config changes for the servers and just the complexity makes me wonder how many people who've posted that they've done this install to be true!

Here is what I'm trying to do.

2 Node RAC
1 Node Application Tier (I want 2, but starting with 1).

This post will be updated to something sane, once I get through it...though - I'm posting just incase, someone can be saved some of the pain.

System Config
Oracle Linux x86_64
Yum Repo:
Oracle Public and Oracle Addons
[public_ol6_latest]
name=Oracle Linux $releasever Latest ($basearch)
baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/$basearch/
gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6
gpgcheck=1
enabled=1

[public_ol6_addons]
name=Oracle Linux $releasever Addons ($basearch)
baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/$basearch/
gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6
gpgcheck=1
enabled=1

RPMs Used:
yum install oracle-rdbms-server-11gR2-preinstall
yum install oracle-ebs-server-R12-preinstall
.>
Other config:
Fix the hosts file to have only the following
127.0.0.1  localhost
  FQDN  DN

For the oracle hosts, when trying to install the Database with the EBS installer, cluvfy will most likely fail because of the network check, dns check or something else (11.2.0.4 cluvfy will successfully complete, the issue is only with 11.2.0.3)

NSLookup Fix

To temporarily fix the nslookup error:
# mv /usr/bin/nslookup /usr/bin/nslookup.orig
# echo '#!/bin/bash
/usr/bin/nslookup.orig $*
exit 0' > /usr/bin/nslookup
# chmod a+x nslookup

Network Check
Once I confirmed that everything was fine with the 11.2.0.4 cluvfy, modify the runcluvfy in the /TechInstallMedia/database/grid directory
The last line, change it as below
# Return the status of executed cluster verification command
exit 0 # new add
#exit $EXEC_STATUS





Thursday, July 10, 2014

Oracle Database Appliance - My case for Solaris


I just got to try out the Oracle database appliance (ODA).  It's supposed to be quick and easy to setup and get it running.  We've run into some intial hiccups, but this is a POC box and I'm not going to give Oracle grief here.

Here are my quick thoughts

Pros:
1.  Extremely easy to setup (hardware perspective)
2.  Very cost effective.
3.  Comes with lots of Storage and now allows expansion into another Storage Shelf.

Cons:

1.  Only Oracle Linux.  No Solaris.
2.  Only one ASM across all the databases.
3.  Patching has to be done in a particular order.  You can't simply patch Db, you need to patch hardware, OS and such, before you get to the Db patching (There is a hierarchy)
4.  When you start patching, depending upon what you're patching, ALL Databases need to be shutdown.  This is very hard to do, especially when you have multiple applications using Dbs hosted on the ODA.

So - where is Solaris and why you ask.

1.  Solaris 10 / 11 allows you to run Oracle RAC clusters in Zones.
2.  With Solaris, you can have Zones running your Oracle RAC, and multiple Oracle RAC instances can be spun up, especially with MULTIPLE ASMs running.  This will immediately allow you to patch your infrastructure one node at a time.
3.  The bonded connections with Solaris 11 makes it incredibly simple to work with.  This link specifically: Network Virtualization High Availability made me think of ODA on Solaris a lot more.
4.  Solaris 11 allows you to add disks to zones without a reboot (a big Solaris 10 limitation).
5.  Zones allow you to CPU cage.
6.  The Hardware is all based off the Sun systems.
7.  You can use ZFS for all OS disks.
8.  You can get away from the clunky OVM DOM0 world that the ODA appliance uses now.
9.  You can effortlessly do virtualization and use the VM Templates.
10.  CloudFS partition would be effortless too - just make a new ZPOOL and you're done!
11.  All this can be tied into Oracle Enterprise Manager & Oracle Ops Center for the hardware.
12.  All the patching can be made so much more effortless.


Tuesday, April 08, 2014

Chef - my journey with the Infrastructure tool - Part 1 of hopefully many more...



Chef is a Infrastructure in Code service, which allows you to define your infrastructure in different files (recipes, Cookbooks, resources etc) and by using Chef - you can build all your servers the exact same way.  The advantage I see with Chef is - it has inbuilt knowledge on how to work with a number of different OS's - so Solaris could have pkgadd, RHEL yum or CentOS apt-get and these would automatically be determined by Chef.

Chef is written in Ruby, and you need to know just some basics of Ruby to get started with Chef.  However, even if you dont know it - you can keep following along - or you can learn what you need for Chef here.

I highly recommend going through the screencasts available at OpsCode - here.

So - here's where I am - I got my workstation (a Mac) setup for Chef. I decided if I was going to learn it, the best spot to try things out is the Amazon cloud - it'll cost me some money, but considering a lot of organizations are moving there, might as well get experience with that as well :)

Amazon does offer a Micro instance for free for 750 hrs - and yes thats what I'm trying to use.

So to get Started, I picked an UBUNTU image which is available on the Amazon Market place.

I played around with Chef - and got Chef talking between Amazon and my desktop +  Enterprise Hosted Chef (free for less than 5 nodes).

Yes - I am getting to the meat now :).

My next target is to be able to Deploy an OS on the Amazon cloud using just Chef.

There's probably a ton of blogs out there with this - so I am adding to it. I'm hoping to detail any issues that I've run into as well as use this blog as my own documentation :)

One of the things to note is that : chef always installs in "/opt/chef" on *nix flavors.  There are too many dependencies, and its not supported if its not there.

First : Install knife-ec2 plugin for Chef.  This is assuming that has already been installed.

/opt/chef/embedded/bin/gem install knife-ec2

Next, get the Amazon EC2 key and access code - its available under your Amazon Account, "Security Credentials" & Access Keys.

You'll also need your Amazon key pair - you can create a new one, or use the existing one you have already.  Its under "Network Security"&  Key Pairs

this is what you'll need to put in the knife.rb file.

### The Key pair with .pem file name. (without the .pem)
knife[:aws_ssh_key_id] = 'KEY_ID' 
#### The Access key id and access key from Amazon account. knife[:aws_access_key_id] = 'ACCESS_KEY_ID' 
knife[:aws_secret_access_key] = 'SECRET_ACCESS_KEY'

Make sure the .pem file has permissions of 400.  Otherwise, the ssh-user wont allow the use of the .pem file.

Monday, April 07, 2014

Solaris 10 x86 - Grub not loading



I have a Solaris 10x86 install which failed all attempts after a reboot with ZFS

After a lot of trial and error, I finally did the below from the Grub menu and it worked.

grub> find /sbin/init
grub> root (hd1,0,a)  <<---this a="" command.="" find="" hd1="" is="" of="" output="" p="" the="">grub> kernel$ /platform/i86pc/multiboot -B $ZFS-BOOTFS
grub>module /platform/i86pc/boot_archive
grub> boot

An example of the menu.lst at /rpool/boot/grub/menu.lst

#---------- ADDED BY BOOTADM - DO NOT EDIT ----------
title Solaris 10 10/09 s10x_u8wos_08a X86
findroot (pool_rpool,0,a)
kernel$ /platform/i86pc/multiboot -B $ZFS-BOOTFS
module /platform/i86pc/boot_archive
#---------------------END BOOTADM--------------------
#---------- ADDED BY BOOTADM - DO NOT EDIT ----------
title Solaris failsafe
findroot (pool_rpool,0,a)
kernel /boot/multiboot -s
module /boot/amd64/x86.miniroot-safe
#---------------------END BOOTADM--------------------

Wednesday, July 31, 2013

Chromecast

This small device is just awesome!  I got it, connected it to my AV Receiver, and setup took about 5 mins after that.

For $35 -
I got it before the Neflix subscription offer expired - so 3 months of Netflix
the Chromecast device
A HDMI extender
A charger for the Chrome cast with the USB cable.

This is a awesome device and I highly recommend it!



Sunday, July 28, 2013

Downloading a sites certificate with openssl

I keep running into an issue where I need to get a server certificate sometimes, and its not always accessible via the browser.

When I need it from the server  - this handy comment on this site worked like a charm http://serverfault.com/questions/139728/how-to-download-ssl-certificate-from-a-website


echo -n | openssl s_client -connect HOST:PORTNUMBER | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/$SERVERNAME.cert
That will save the certificate to /tmp/$SERVERNAME.cert.
You can use -showcerts if you want to download all the certificates in the chain. But if you just want to download the server certificate, there is no need to specify -showcertsecho -n gives a response to the server, so that the connection is released
sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' removes information about the certificate chain. This is the preferred format to import the certificate into other keystores.


Thanks to the person who gave that comment, it has helped me immensley.

Weblogic - App having javax.net.ssl.SSLHandshakeException

If you use Weblogic and have an application that connects to the outside world via HTTPS, you could be in for a treat.

If you have the DemoIdentityTrustStore configured or any keystore configured and the JDK keystore configured, it looks like Weblogic randomly decides which keystore for Trust to use.

We ran into an issue, where the application would randomly complain of invalid cert chain similar to this:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)

After many many days of troubleshooting, wondering why even thoguh the java CACERTS had the correct certs, I decided to try accessing the site with the DemoTrust keystore instead.  Whola...the error manifested itself again.

What is insanely annoying is that this is not repeatable in WLS.  Every restart seems to have a different behavior.

To import a certificate you already have into the keystore - use the "keytool" utility.


cd $JAVA_HOME/jre/lib/security
 keytool -import -alias VerisignC3PPCA   -keystore cacerts -trustcacerts -file /root/getrootcert.cer


I found this great piece of code online (InstallCert.java)  (below) which helped me troubleshoot.

If you want to use the Weblogic Identity store - below is some useful data.  You will need to modify the below code to point to this JKS if you want to (change the "File file = new ... " block) .  To add the certificate, you can do it using "keytool" as well if you have the certificates you care about.

This table was very neatly done at : http://itguykelly.wordpress.com/2010/05/20/default-weblogic-keystore-passwordpassphrase/ which I've borrowed.


Property
Value
Trust store location
%ORACLE_HOME%/weblogic/wlserver_10.3/ server/lib/DemoTrust.jks
Trust store password
DemoTrustKeyStorePassPhrase
Key store location
%ORACLE_HOME%/weblogic/wlserver_10.3/ server/lib/DemoIdentity.jks
Key store password
DemoIdentityKeyStorePassPhrase
Private key password
DemoIdentityPassPhrase
Property
Value
Trust store location
%ORACLE_HOME%/weblogic/wlserver_10.3/ server/lib/DemoTrust.jks
Trust store password
DemoTrustKeyStorePassPhrase
Key store location
%ORACLE_HOME%/weblogic/wlserver_10.3/ server/lib/DemoIdentity.jks
Key store password
DemoIdentityKeyStorePassPhrase
Private key password
DemoIdentityPassPhrase


Copy and paste the below code into a file called "InstallCert.java"
javac InstallCert.java
java InstallCert.java



/*
 * Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *
 *   - Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *
 *   - Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *
 *   - Neither the name of Sun Microsystems nor the names of its
 *     contributors may be used to endorse or promote products derived
 *     from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
 * IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */
/**
 * Originally from:
 * http://blogs.sun.com/andreas/resource/InstallCert.java
 * Use:
 * java InstallCert hostname
 * Example:
 *% java InstallCert ecc.fedora.redhat.com
 */

import javax.net.ssl.*;
import java.io.*;
import java.security.KeyStore;
import java.security.MessageDigest;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;

/**
 * Class used to add the server's certificate to the KeyStore
 * with your trusted certificates.
 */
public class InstallCert {

    public static void main(String[] args) throws Exception {
        String host;
        int port;
        char[] passphrase;
        if ((args.length == 1) || (args.length == 2)) {
            String[] c = args[0].split(":");
            host = c[0];
            port = (c.length == 1) ? 443 : Integer.parseInt(c[1]);
            String p = (args.length == 1) ? "changeit" : args[1];
            passphrase = p.toCharArray();
        } else {
            System.out.println("Usage: java InstallCert [:port] [passphrase]");
            return;
        }

        File file = new File("jssecacerts");
        if (file.isFile() == false) {
            char SEP = File.separatorChar;
            File dir = new File(System.getProperty("java.home") + SEP
                    + "lib" + SEP + "security");
            file = new File(dir, "jssecacerts");
            if (file.isFile() == false) {
                file = new File(dir, "cacerts");
            }
        }
        System.out.println("Loading KeyStore " + file + "...");
        InputStream in = new FileInputStream(file);
        KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
        ks.load(in, passphrase);
        in.close();

        SSLContext context = SSLContext.getInstance("TLS");
        TrustManagerFactory tmf =
                TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
        tmf.init(ks);
        X509TrustManager defaultTrustManager = (X509TrustManager) tmf.getTrustManagers()[0];
        SavingTrustManager tm = new SavingTrustManager(defaultTrustManager);
        context.init(null, new TrustManager[]{tm}, null);
        SSLSocketFactory factory = context.getSocketFactory();

        System.out.println("Opening connection to " + host + ":" + port + "...");
        SSLSocket socket = (SSLSocket) factory.createSocket(host, port);
        socket.setSoTimeout(10000);
        try {
            System.out.println("Starting SSL handshake...");
            socket.startHandshake();
            socket.close();
            System.out.println();
            System.out.println("No errors, certificate is already trusted");
        } catch (SSLException e) {
            System.out.println();
            e.printStackTrace(System.out);
        }

        X509Certificate[] chain = tm.chain;
        if (chain == null) {
            System.out.println("Could not obtain server certificate chain");
            return;
        }

        BufferedReader reader =
                new BufferedReader(new InputStreamReader(System.in));

        System.out.println();
        System.out.println("Server sent " + chain.length + " certificate(s):");
        System.out.println();
        MessageDigest sha1 = MessageDigest.getInstance("SHA1");
        MessageDigest md5 = MessageDigest.getInstance("MD5");
        for (int i = 0; i < chain.length; i++) {
            X509Certificate cert = chain[i];
            System.out.println
                    (" " + (i + 1) + " Subject " + cert.getSubjectDN());
            System.out.println("   Issuer  " + cert.getIssuerDN());
            sha1.update(cert.getEncoded());
            System.out.println("   sha1    " + toHexString(sha1.digest()));
            md5.update(cert.getEncoded());
            System.out.println("   md5     " + toHexString(md5.digest()));
            System.out.println();
        }

        System.out.println("Enter certificate to add to trusted keystore or 'q' to quit: [1]");
        String line = reader.readLine().trim();
        int k;
        try {
            k = (line.length() == 0) ? 0 : Integer.parseInt(line) - 1;
        } catch (NumberFormatException e) {
            System.out.println("KeyStore not changed");
            return;
        }

        X509Certificate cert = chain[k];
        String alias = host + "-" + (k + 1);
        ks.setCertificateEntry(alias, cert);

        OutputStream out = new FileOutputStream("jssecacerts");
        ks.store(out, passphrase);
        out.close();

        System.out.println();
        System.out.println(cert);
        System.out.println();
        System.out.println
                ("Added certificate to keystore 'jssecacerts' using alias '"
                        + alias + "'");
    }

    private static final char[] HEXDIGITS = "0123456789abcdef".toCharArray();

    private static String toHexString(byte[] bytes) {
        StringBuilder sb = new StringBuilder(bytes.length * 3);
        for (int b : bytes) {
            b &= 0xff;
            sb.append(HEXDIGITS[b >> 4]);
            sb.append(HEXDIGITS[b & 15]);
            sb.append(' ');
        }
        return sb.toString();
    }

    private static class SavingTrustManager implements X509TrustManager {

        private final X509TrustManager tm;
        private X509Certificate[] chain;

        SavingTrustManager(X509TrustManager tm) {
            this.tm = tm;
        }

        public X509Certificate[] getAcceptedIssuers() {
            throw new UnsupportedOperationException();
        }

        public void checkClientTrusted(X509Certificate[] chain, String authType)
                throws CertificateException {
            throw new UnsupportedOperationException();
        }

        public void checkServerTrusted(X509Certificate[] chain, String authType)
                throws CertificateException {
            this.chain = chain;
            tm.checkServerTrusted(chain, authType);
        }
    }
}

Monday, July 08, 2013

Apache 64-bit on Solaris 10 & Weblogic

Solaris 10 comes with Apache 32- bit by default.  This was a surprise.

My task here is to get Apache 64-bit and then hook it up to Weblogic on the SSL channel.  I will detail the Apache + Weblogic SSL communication.  Setting up Weblogic to listen on SSL is a different post, that I may or may not write up :)

I will be installing apache at /usr/local/apache (Many folks like to install it at /usr/local/apache2 etc, but I'm keeping it simple here)

I have a staging directory to compile / install the required packages.  This is a Whole Root zone, with Solaris 10 fully installed with all the packages on the Global Zone.

I need
0. Apache 2.2.4 -- I tried Apache 2.4 and as of this writing is not supported with the mod_wl.so library.
1.  PCRE (Perl-Compatible Regular Expressions Library)
2. Apr
3. Apr-utils

A gist of what is required is available on Apache's documentation site ( Apache Documentation for Installation )

For PCRE, I downloaded it from: PCRE
Download APR & APR-UTIL from here.

Initial settings on the terminal
export CFLAGS="-m64"
export LDFLAGS="-L/usr/sfw/lib/sparcv9"
export LD_LIBRARY_PATH=/usr/sfw/lib/sparcv9:/usr/local/lib/sparcv9 
the CFLAGS setting is  really important, otherwise, you wont get it compiled in 64 bit.  Make sure your LD_LIBRARY_PATH and LDFLAGS point to the accurate directories.

Compiling PCRE

This installs PCRE in /usr/local/pcre

/root/apache-compilation/pcre-8.33
./configure --disable-cpp CFLAGS="-m64 -g"
make
make install
Compiling APR

/root/apache-compilation/apr-1.4.8
./configure
make
make install

APR gets installed in /usr/local/apr.  The CFLAGS setting set in the environment should carry on.

Compiling APR-UTIL

/root/apache-compilation/apr-util-1.5.2
./configure --with-apr=/usr/local/apr
make
make install

Compiling Apache
I use the prefix /usr/local/apache  - but some people prefer to use "apache2".  Make it your own.  The prefix designates where apache gets installed.

/root/apache-compilation/httpd-2.2.24
./configure --with-apr=/usr/local/apr --with-apr-util=/usr/local/apr --prefix=/usr/local/apache --enable-mods-shared=most --enable-ssl --with-ssl=/usr/sfw
make
make install

After the installation, I like to test to make sure its 64 bit (as I've gotten bitten a number of times).

# cd /usr/local/apache/modules
# file mod_vhost_alias.so
mod_vhost_alias.so:     ELF 64-bit MSB dynamic lib SPARCV9 Version 1, dynamically linked, not stripped, no debugging information available

That tells you its a 64 Bit file.

To get Weblogic working, I copied the wls 1.1 plugins *.so files (which are only 64 bit) to the /usr/local/apache/modules directory.  Then followed the instructions from Oracle here

Good luck.  Comments are welcome.