Some Odd behavior on RHEL 5.4 with HTTPD (apache)

[Bill G.]

Hi List,

I may have stumbled across a bug, or I am just a nub (but I am pretty
sure it is the former)

I am running RHEL 5.2 and 5.4 (dont ask why, just know that at this
time it s a requirement)

For all intents and purposes the configurations between the two are
identical (changing only hostnames and IP addresses)

Here is the first configuration (note all hostnames, ipaddresses, and
service names are obfusticated):

   check host RHEL54-vip with address 10.0.0.1
     start program = "/sbin/ifup eth0:1"
     stop program = "/sbin/ifdown eth0:1"
     if failed icmp type echo count 3 with timeout 3 seconds then restart
     if 5 restarts within 5 cycles then timeout

   check process RHEL54-apache with path /etc/httpd/run/httpd.pid
     start program = "/etc/init.d/httpd start"
     stop program = "/etc/init.d/httpd stop"
     if failed host RHEL54-vip port 80
       protocol HTTP request "/" then restart
     if 5 restarts within 5 cycles then timeout

Everything works as expected in 5.2, but in 5.4 you get the feeling
that the world is ending.

If I force a crash from RHEL52:

Logs:

ICMP echo response for 10.0.0.1 1/3 timed out -- no response within 3 seconds
ICMP echo response for 10.0.01 2/3 timed out -- no response within 3 seconds
ICMP echo response for 10.0.0.1 3/3 timed out -- no response within 3 seconds
'RHEL52-vip' failed ICMP test [Echo Request]
'RHEL52-apache' failed, cannot open a connection to INET[RHEL52-vip:80/] via TCP
'RHEL52-apache' stop: /etc/init.d/httpd
'RHEL52-apache' start: /etc/init.d/httpd
'RHEL52-apache' started
'RHEL52-apache' process is running with pid 5072

Everything else comes up and is happy

Same scenario on RHEL54
nds
nds
nds
'RHEL54-vip' failed ICMP test [Echo Request]
  TCP
'RHEL54-apache' stop: /etc/init.d/httpd
'RHEL54-apache' start: /etc/init.d/httpd
'RHEL54-apache' failed to start

Lather rinse repeat X5

What I have found is happening is that either the httpd process is not
being shut down properly, and/or monit is not waiting for a proper
return from the shutdown.

When in this condition:

# /etc/init.d/httpd status
httpd is stopped
 ps -auxwww|grep httpd
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
root     22031  0.1  0.5 236080 11200 ?        Ss   10:07   0:00 /usr/sbin/httpd
apache   22033  0.0  0.3 236212  6640 ?        S    10:07   0:00 /usr/sbin/httpd
apache   22034  0.0  0.3 236212  6636 ?        S    10:07   0:00 /usr/sbin/httpd
apache   22035  0.0  0.3 236212  6636 ?        S    10:07   0:00 /usr/sbin/httpd
apache   22036  0.0  0.3 236212  6636 ?        S    10:07   0:00 /usr/sbin/httpd
apache   22037  0.0  0.3 236212  6636 ?        S    10:07   0:00 /usr/sbin/httpd
apache   22038  0.0  0.3 236212  6636 ?        S    10:07   0:00 /usr/sbin/httpd
apache   22039  0.0  0.3 236212  6636 ?        S    10:07   0:00 /usr/sbin/httpd
apache   22040  0.0  0.3 236212  6636 ?        S    10:07   0:00 /usr/sbin/httpd


(i give it to you typo and all!)

Only way to fix is to issue a killall command
killall httpd
then it starts as perscribed.


So I thought, maybe it is something silly with the binary, so I
decided to compile from source.. same issue.

My current workaround is as follows:

   check process RHEL54-apache with path /etc/httpd/run/httpd.pid
     start program = "/etc/init.d/httpd start"
     stop program = "/etc/init.d/httpd stop"
     if failed host RHEL54-vip port 80
       protocol HTTP request "/" then restart
     if 2 restarts within 2 cycles then exec "/usr/bin/killall httpd"
     if 5 restarts within 5 cycles then timeout

Any ideas how to make this function properly?
--
Thanks,
Bill G.
address@hidden