Re: Multiple processes started when explicit start commands areused on dependant processes.

[Philipp Berndt]

On Tuesday 07 February 2006 22:17, Jan-Henrik Haukeland wrote:
> On 7. feb. 2006, at 19.43, Philipp Berndt wrote:

> > Note that checking for the process id is *NOT* enough, because the
> > service may
> > have to initialize (spawn other processes, claim resources etc.).
>
> This is exactly the problem, waiting for a process or pid file to be
> written is not enough, since this is typically done at the start and
> before other initializing tasks. So what should we wait for? If you
> have a good _general_ suggestion I'm all ears.
>
> If it's a socket based server, such as apache or mysql, we could wait
> for it to pass a connection w/protocol test. That should work since
> we then now it is initialized and the processing machinery works.
> However this is not the case for many servers started by monit and
> hence not general.

You are perfectly right, there is NO GENERAL WAY to determine whether the 
service is ready or not.
That's why it has to be done in the specific start script for that service.

(I tried to find an explicit statement to this respect in the Linux Standard 
Base Specification and only found this (which remains a bit fuzzy):
"When an init script is run with a start argument, the boot facility or 
facilities specified by the Provides keyword shall be deemed present and 
hence init scripts which require those boot facilities should be started 
later."  
http://www.linuxbase.org/spec/refspecs/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/initscrcomconv.html

I still believe it means this :

> > Only after the start script has terminated you can be sure the
> > service is
> > ready and may be used by dependant services.
>
> Nope, if the start script calls 'exec' it will never return
(I meant init.d scripts which always return.)

> or if it 
> just does a fork ala 'program&' we are still back to the problem
> situation described above.

Using "&" in init scripts is indeed problematic. Some daemons can be started 
with some "-d" switch to fork themselves (so you don't need &) and may be 
well-behaved enough to return only when the service is initialized and ready.

Other init scripts may have to be adapted.

Lots of people do things like

foo &
sleep 1

which 
- works 99 times out of 100, but 
- is not reliable (especially if the system is under heavy load) and 
- is slower than necessary (if foo usually only takes 0.1s to start up).

To solve the problem I have written (for my own needs) some semi-generic tools 
to check for a service, which I now use in my init scripts.

- One is a program that tries to connect to some specified TCP port (quite 
generic)
- Anotherone tries to contact a CORBA servant registered in a CORBA name 
service (more specific)
- A third one waits until a line appears in a log file (currently very 
specific to tomcat)
...
All of them only return when they succeed (or with an error code in case of a 
timeout).

Another tool is a killproc which takes a pid from a pidfile, kills the process 
waits for it's termination, optionally does a kill -9 on timeout and removes 
the pidfile.

They are meant as a toolkit to write well-behaved init scripts. I probably 
will release them on sourceforge once I have done some cleaning up and 
removed some build-dependencies.

BTW, a little feature request :-)
it would be nice if monit could optionally use the init scripts' status 
actions to monitor a service.
Rationale: Some services may require again very specific checks that should be 
located in the service's specific init script.

Best regards,
Philipp