[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Making grep on large number of files run faster
From: |
uzibalqa |
Subject: |
Making grep on large number of files run faster |
Date: |
Thu, 30 Mar 2023 18:25:24 +0000 |
How can I make searches with grep on a large number of files run faster?
My plan has been to use parallel processing with Gnu Parallel to run multiple
instances of grep in
parallel, instead of using xargs with the -P option.
Here is my parallel implementation. I first locate the files and then pipe
them to parallel.
PTRN contains multiple search patterns, whilst ICTX stores the context.
I need some help to get a good implementation with possible improvements
ptrn=("-e" "FN" "-e" "DS")
ictx=(-A 8)
grep --null -r -l "${isufx[@]}" \
-f <(printf "%s\n" "${ptrn[@]}") -- "${fdir[@]}" \
| PARALLEL_SHELL=bash psgc=$sgc psgr=$sgr uptrn=$ptrn \
parallel -m0kj"$procs" \
'for fl in {}; do
printf "\n%s\n\n" "${psgc}==> $fl <==${psgr}"
grep -ni '"${ictx[@]@Q}"'
-f <(printf "%s\n" "${ptrn[@]}") -- "$fl"
done'
For example, you can use find to locate the files and then pipe them to xargs
with the -P option to run multiple instances of grep in parallel.
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- Making grep on large number of files run faster,
uzibalqa <=