Re: Make run in parallel mode with output redirected to a regular file can randomly drop output lines

[Eli Zaretskii]

> Date: Fri, 31 May 2013 05:36:24 +0200
> Cc: address@hidden, address@hidden, address@hidden
> From: Frank Heckenbach <address@hidden>
> 
> Eli Zaretskii wrote:
> 
> > > From: Frank Heckenbach <address@hidden>
> > > 
> > > However, there may still be a problem. The trick about O_APPEND on
> > > POSIX it that it's atomic, i.e. nothing can get between moving the
> > > file pointer and the write, even if another process tries to write
> > > simultaneously. So if the POSIX emulation API emulates it with two
> > > system calls (seek and write), it wouldn't be atomic. I don't know
> > > if it does, or whether the problem exists in the first place.
> > 
> > The problem exists, but there's nothing that can be done about it, as
> > long as we use write/fwrite/fprintf for this: the call to 'write'
> > isn't atomic on Windows even without O_APPEND, because of the
> > text-mode translation of newlines to CR-LF pairs.
> 
> I didn't mean the whole write() must be atomic, just the seek and
> write part. I.e., if the translation is done in user-space and then
> a system call does the write and seek atomically, it might still be
> OK even without O_APPEND.

But that's what I'm telling you: each chunk of text after NL to CR-LF
conversion is written separately in a separate call to WriteFile,
which is the low-level API for file I/O.

> > > b) Not a), and writing with O_APPEND is not synchronized either
> > >    (perhaps because the emulation layer just does two separate
> > >    system calls). Setting O_APPEND would be pointless then.
> > 
> > No, it's not pointless.  It makes the problem smaller.  And if the
> > Posix systems will do that, doing that on Windows will minimize the
> > number of #ifdef's, of which we have way to many already.
> 
> As I undestood you, you'd have to write an emulation for
> fcntl (F_SETFD). That code has to be written and maintained, and it
> adds a (small) runtime overhead. Not sure if that's worth saving an
> #ifdef, *if* the problem doesn't actually exist.

Given your next message about tmpfile, I will need that anyway.  And
the code is simple (assuming it works).

> > > I heard about similar issues WRT "select", which is probably
> > > also emulated and AIUI only works for sockets (and perhaps some
> > > other devices), whereas on POSIX it accepts any FD.
> > 
> > That's because a socket is not a file on Windows.
> 
> That's what I meant. On POSIX, stdout could be a socket.

Stdout can be a socket on Windows as well, because, under the hood,
they are both represented by handles.  See how the setup of child
processes is done in w32/subproc/sub_proc.c.

> > > > I wasn't talking about synchronization or merging.  I was talking
> > > > about _losing_ some of the output, which was the issue discussed here.
> > > 
> > > That's the consequence of lack of synchronization.
> > 
> > No, it isn't.  If the same file pointer were used, there would be no
> > need for any synchronization, because that pointer would serialize
> > output by its very nature.
> 
> Not sure what you mean here. The OS is not a magic box that does
> anything "by its very nature".

No magic needed when there's only one file pointer, because a single
file pointer can only be at one place at any given time.

> void write (int fd, void *data, size_t size)
> {
>   if (getflags (fd) & O_APPEND)
>     {
>       lock_mutex (get_mutex (fd));
>       off_t pos = get_size (fd);
>       do_write (fd, pos, data, size);
>       set_pos (fd, pos + size);
>       unlock_mutex (get_mutex (fd));
>     }
>   else
>     {
>       // no mutex here!
>       off_t pos = get_pos (fd);
>       do_write (fd, pos, data, size);
>       set_pos (fd, pos + size);
>     }
> }

If the 'else' clause uses a single file pointer system-wise, there's
no overwriting because the pointer is not moved between writes.