OEM 10g not starting ...Fix!

If you have Oracle Enterprise Manager that very annoyingly wont restart, though it's been perfectly working before you shut it down, and opmnctl complains that it cant start the HTTP server, you're probably running into the file size limit of the Apache logs.

[oragrid@admin30w]$ tail -20 /opt/oragrid/oms10g/Apache/Apache/logs/error_log
[Sat Jul 2 01:43:20 2011] [error] (79)Value too large for defined data type: could not open transfer log file /opt/oragrid/oms10g/Apache/Apache/logs/access_log.

If you see that message, it implies that the access log file is too large, typically over 2Gb. OEM by default doesn't rotate logs, so you need to either set it up, or rotate the logs manually.

oragrid@admin30w]$ ls -l /opt/oragrid/oms10g/Apache/Apache/logs/access_log
-rw-r---- 1 oragrid oinstall 2147483881 Apr 30 21:58 /opt/oragrid/oms10g/Apache/Apache/logs/access_log

To fix it, I just did a

cat /dev/null > /opt/oragrid/oms10g/Apache/Apache/logs/access_log

You can also set up log rotation this way.
Note 339819.1 from Metalink will give you detailed steps on the multiple ways to setup log rotation. However, this is supposed to be fixed in 11g.


1. Stop the OMS:

cd /opmn/bin
opmnctl stopall
2. Take a backup of the files :
/sysman/config/httpd_em.conf
/sysman/config/httpd_em.conf.template

NOTE: You must make the changes to both httpd_em.conf AND httpd_em.conf.template because any 'emctl secure' operation results in the http_em.conf file being regenerated using http_em.conf.template and any changes that had been made to httpd_em.conf earlier will be lost.
3. Edit the /sysman/config/httpd_em.conf and modify:

ErrorLog /Apache/Apache/logs/error_log
TransferLog /Apache/Apache/logs/access_log

TO

ErrorLog "|//Apache/Apache/bin/rotatelogs /Apache/Apache/logs/error_log 43200"
TransferLog "|//Apache/Apache/bin/rotatelogs /Apache/Apache/logs/access_log 43200"

where is the full path to your OMS install directory.
4. Edit the /sysman/config/httpd_em.conf.template and modify:

ErrorLog &ORACLE_HOME&/Apache/Apache/logs/error_log
TransferLog "&ORACLE_HOME&/Apache/Apache/logs/access_log"

To

ErrorLog "|&ORACLE_HOME&/Apache/Apache/bin/rotatelogs &ORACLE_HOME&/Apache/Apache/logs/error_log 43200"
TransferLog "|&ORACLE_HOME&/Apache/Apache/bin/rotatelogs &ORACLE_HOME&/Apache/Apache/logs/access_log 43200"
5. Delete the existing access_log and error_log files in the /Apache/Apache/log directory.

6. Re-start the OMS
cd /opmn/bin
opmnctl startall