[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: RFC: Safely using xargs -P$NUM children's output? Need a new tool?

From: Denys Vlasenko
Subject: Re: RFC: Safely using xargs -P$NUM children's output? Need a new tool?
Date: Fri, 3 May 2019 08:49:27 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0

On 5/2/19 8:25 PM, Egmont Koblinger wrote:
Hi Denys,

(Disclaimer: I'm just a random guy commenting here, not affiliated
with the coreutils team.)

The simple solution recommended by xargs's man page at the -P option
is to use separate output file for each parallel "file".

One concern with your suggested coalesce approach is that this tool
itself also needs to write its output in an atomic step. If coalesce
itself performs a partial write then you're back to square one.

That's the idea: the tool would be defined to do a single write.
(Unlike e.g. "tail -nNUM", which you might expect to do one write
when it's done, but in fact it does multiple writes).

I guess (and that's what you also say) if the output goes to a local
file system then complete writes are guaranteed. Might depend on the
file system type or other factors though, a kernel guru could clarify> it. I'm 
not so sure about virtual, remote, fuse etc. file systems.

I do remember that specifically writes to filesystem (as
opposed to writes to file descriptors in general) are not allowed
to become partial randomly. It was discussed on LKML.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]