parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Making grep on large number of files run faster


From: uzibalqa
Subject: Making grep on large number of files run faster
Date: Thu, 30 Mar 2023 18:25:24 +0000

How can I make searches with grep on a large number of files run faster?

My plan has been to use parallel processing with Gnu Parallel to run multiple 
instances of grep in
parallel, instead of using xargs with the -P option.  

Here is my parallel implementation.  I first locate the files and then pipe 
them to parallel. 
PTRN contains multiple search patterns, whilst ICTX stores the context.

I need some help to get a good implementation with possible improvements

ptrn=("-e" "FN" "-e" "DS")
ictx=(-A 8)

grep --null -r -l "${isufx[@]}"  \
  -f <(printf "%s\n" "${ptrn[@]}") -- "${fdir[@]}"  \
    | PARALLEL_SHELL=bash psgc=$sgc psgr=$sgr uptrn=$ptrn  \
        parallel -m0kj"$procs"  \
          'for fl in {}; do
            printf "\n%s\n\n" "${psgc}==> $fl <==${psgr}"
            grep -ni '"${ictx[@]@Q}"'
                 -f <(printf "%s\n" "${ptrn[@]}") -- "$fl"
          done'





For example, you can use find to locate the files and then pipe them to xargs 
with the -P option to run multiple instances of grep in parallel.



reply via email to

[Prev in Thread] Current Thread [Next in Thread]