[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH] md5: accepts a new --threads option
From: |
Pádraig Brady |
Subject: |
Re: [PATCH] md5: accepts a new --threads option |
Date: |
Tue, 20 Oct 2009 23:16:52 +0100 |
User-agent: |
Thunderbird 2.0.0.6 (X11/20071008) |
Pádraig Brady wrote:
>
> You wouldn't want multiple threads/processes fighting over
> the disk head so you would do something like:
>
> find /disk1 | xargs md5sum & find /disk2 | xargs md5sum
>
> Note if we're piping/redirecting the output of the above
> then we must be careful to line buffer the output from md5sum
> so that it's not interspersed. Hmm I wonder should
> we linebuffer the output from *sum by default.
In the attached patch, I've changed the default buffering
to line buffered to address the above issue. For standard
size files there is a 2% performance drop.
cheers,
Pádraig.
p.s. I'll look at bypassing stdio on input to see
if I can get at least the 2% back
>From 0db7057c6256d9cd25e988b3fe23e97a0e30f717 Mon Sep 17 00:00:00 2001
From: =?utf-8?q?P=C3=A1draig=20Brady?= <address@hidden>
Date: Tue, 20 Oct 2009 19:19:58 +0100
Subject: [PATCH] md5sum, sha*sum, sum: line buffer the outputted checksums
* src/md5sum.c (main): Set stdout to line buffered mode
to ensure parallel running instances don't intersperse
their output. This adds 5% to the run time in the worst case
of many zero length files, or 2% with standard file sizes.
* src/sum.c (main): Likewise.
---
src/md5sum.c | 6 ++++--
src/sum.c | 4 ++++
2 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/src/md5sum.c b/src/md5sum.c
index aa2a144..b7db03e 100644
--- a/src/md5sum.c
+++ b/src/md5sum.c
@@ -513,7 +513,6 @@ digest_check (const char *checkfile_name)
if (!status_only)
{
printf (_("%s: FAILED open or read\n"), filename);
- fflush (stdout);
}
}
else
@@ -539,7 +538,6 @@ digest_check (const char *checkfile_name)
printf ("%s: %s\n", filename, _("FAILED"));
else if (!quiet)
printf ("%s: %s\n", filename, _("OK"));
- fflush (stdout);
}
}
}
@@ -619,6 +617,10 @@ main (int argc, char **argv)
atexit (close_stdout);
+ /* Line buffer stdout to ensure lines are written atomically and immediately
+ so that processes running in parallel do not intersperse their output. */
+ setvbuf (stdout, NULL, _IOLBF, 0);
+
while ((opt = getopt_long (argc, argv, "bctw", long_options, NULL)) != -1)
switch (opt)
{
diff --git a/src/sum.c b/src/sum.c
index 91d7f34..f0e0cc0 100644
--- a/src/sum.c
+++ b/src/sum.c
@@ -233,6 +233,10 @@ main (int argc, char **argv)
atexit (close_stdout);
+ /* Line buffer stdout to ensure lines are written atomically and immediately
+ so that processes running in parallel do not intersperse their output. */
+ setvbuf (stdout, NULL, _IOLBF, 0);
+
have_read_stdin = false;
while ((optc = getopt_long (argc, argv, "rs", longopts, NULL)) != -1)
--
1.6.2.5
Re: [PATCH] md5: accepts a new --threads option, Giuseppe Scrivano, 2009/10/17
Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/20
- Re: [PATCH] md5: accepts a new --threads option,
Pádraig Brady <=
- Re: [PATCH] md5: accepts a new --threads option, Jim Meyering, 2009/10/21
- Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/21
- Re: [PATCH] md5: accepts a new --threads option, Jim Meyering, 2009/10/21
- Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/21
- Re: [PATCH] md5: accepts a new --threads option, Jim Meyering, 2009/10/21
- Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/21
RE: [PATCH] md5: accepts a new --threads option, Voelker, Bernhard, 2009/10/21
Re: [PATCH] md5: accepts a new --threads option, Pádraig Brady, 2009/10/22
Re: [PATCH] md5: accepts a new --threads option, Eric Blake, 2009/10/22
Re: [PATCH] md5: accepts a new --threads option, Jim Meyering, 2009/10/22