[Gnu-arch-users] review/merge request: sftp get/put-file() pipe-lining

[Scott Parish]

address@hidden/tla--devo--1.3--patch-1
http://software.srparish.net/archives/address@hidden/

While researching something else i was struck by the realization that
file-get/put() for sftp should be pipe-lined, but i had neglected to
realize and do such earlier.

Prior sftp's get had been doing something like this:

while not eof:
  write_socket(sftp_request_read(handle, nbytes))
  # .. lag ..
  read_socket()    

With this patch it will make a bunch (16 by default) requests at a time
there-by negating the effect of lag (for large file transfers).

Testing between here and my server showed significant improvements for
get() on a 8.2mb patch:

tla get address@hidden/big--big--1.0     0.26s user 0.13s system 0% cpu 3:45.01 
total
tla/tla get address@hidden/big--big--1.0 0.21s user 0.10s system 0% cpu 1:54.13 
total

put() wasn't nearly as impressive[1], but still somewhat improved:

tla import      1.34s user 0.20s system 0% cpu 6:42.85 total
tla/tla import  1.30s user 0.16s system 0% cpu 5:42.92 total

There is some room for debate on tuning the number of requests to
pipeline(N): with get() its anticipated that there will be N-1 requests
that will return EOF, so if we are get()ing multiple tiny files this is
pure overhead. On the other hand, if there is a large lag, keeping N
sufficiently large should minimize effects of lag.

Enjoy
sRp

1| my dsl connection is not symetrical, so it didn't surprise me to see
   a less impressive improvement on my put() times.

-- 
Scott Parish
http://srparish.net/

pgpnuB_s4AJGO.pgp
Description: PGP signature