Re: [Help-bash] "too much" loop overhead

[Peng Yu]

Here is a better test that definitively tells that the bash realpath
builtin is faster than the python abspath function.

bash/realpath: 0.569-0.347=0.222
python/abspath: 0.574648857117-0.286927938461=0.2877209

$ ./main.py > /dev/null
0.286927938461
0.574648857117
$ ./main.sh > /dev/null
0.347
0.569
$ cat main.sh
#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:

source enable.sh
enable realpath
TIMEFORMAT=%R
time for ((i=0;i<10000;++i)); do
    realpath .
done
time for ((i=0;i<10000;++i)); do
    realpath .
    realpath .
done
$ cat main.py
#!/usr/bin/env python
# vim: set noexpandtab tabstop=2 shiftwidth=2 softtabstop=-1 fileencoding=utf-8:

from os.path import abspath
import timeit
import sys
def f():
    for i in range(10000):
        print abspath('.')
def f2():
    for i in range(10000):
        print abspath('.')
        print abspath('.')
print >> sys.stderr, timeit.timeit('f()', number=1, setup="from
__main__ import f")
print >> sys.stderr, timeit.timeit('f2()', number=1, setup="from
__main__ import f2")


On Fri, Dec 14, 2018 at 1:55 PM Peng Yu <address@hidden> wrote:
>
> Hi,
>
> The following are two pairs of equivalent bash and python scripts.
>
> main.* shows that bash is slower than python. However, the main_loop.*
> indicates that the slow down is most likely from the foo-loop.
>
> ./main_colon.sh means that  ":" x 10000 in bash takes about .03 sec
> runtime. So the bash for-loop takes about 0.127 - .03 =~ 0.09, whereas
> the python for-loop takes .001 sec.
>
> Removing the time of the loop, the bash realpath builtin is actually
> faster. After all, the realpath builtin is just a dynamically linked
> library, it is unceiveable that python would do it even better than
> native C.
>
> The above calculation is based on the assumption that different
> statements in a script do not interfere with each other in terms of
> the runtime. Let me know if this assumption is not true. It is not
> clear how to just measure the runtime of the builtin with the loop
> time accurately removed.
>
> Anyway, the test shows that the loop overhead is even comparable to a
> builtin function that does the actual work. This sounds to be an
> overhead that is not very acceptable.
>
> What is the reason that the loop in bash is so slow? How to make it
> comparable at least to python if not to the native C code? Thanks.
>
> $ ./main.sh > /dev/null
> 0.331
> $ ./main.py > /dev/null
> 0.300492048264
> $ cat main.sh
> #!/usr/bin/env bash
> # vim: set noexpandtab tabstop=2:
>
> enable -f ./realpath realpath
> TIMEFORMAT=%R
> time for ((i=0;i<10000;++i)); do
>     realpath .
> done
> $ cat main.py
> #!/usr/bin/env python
> # vim: set noexpandtab tabstop=2 shiftwidth=2 softtabstop=-1 
> fileencoding=utf-8:
>
> from os.path import abspath
> import timeit
> import sys
> def f():
>     for i in range(10000):
>         print abspath('.')
> print >> sys.stderr, timeit.timeit('f()', number=1, setup="from
> __main__ import f")
>
> $ cat main_loop.py
> #!/usr/bin/env python
> # vim: set noexpandtab tabstop=2 shiftwidth=2 softtabstop=-1 
> fileencoding=utf-8:
>
> import timeit
> import sys
> def f():
>     for i in range(10000):
>         pass
> print >> sys.stderr, timeit.timeit('f()', number=1, setup="from
> __main__ import f")
>
> $ ./main_loop.py
> 0.00102996826172
> $ cat ./main_colon.sh
> #!/usr/bin/env bash
> # vim: set noexpandtab tabstop=2:
>
> TIMEFORMAT=%R
> printf '%s\t' 1; { time for ((i=0;i<10000;++i)) do :; done; } 2>&1
> printf '%s\t' 2; { time for ((i=0;i<10000;++i)) do :;:; done; } 2>&1
> printf '%s\t' 3; { time for ((i=0;i<10000;++i)) do :;:;:; done; } 2>&1
> printf '%s\t' 4; { time for ((i=0;i<10000;++i)) do :;:;:;:; done; } 2>&1
> printf '%s\t' 5; { time for ((i=0;i<10000;++i)) do :;:;:;:;:; done; } 2>&1
> printf '%s\t' 6; { time for ((i=0;i<10000;++i)) do :;:;:;:;:;:; done; } 2>&1
> printf '%s\t' 7; { time for ((i=0;i<10000;++i)) do :;:;:;:;:;:;:; done; } 2>&1
> $  ./main_colon.sh
> 1    0.127
> 2    0.163
> 3    0.196
> 4    0.229
> 5    0.267
> 6    0.309
> 7    0.333
>
> --
> Regards,
> Peng



-- 
Regards,
Peng