[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: how can I not have to spend 40 minutes _waiting_ in the morning for
Re: how can I not have to spend 40 minutes _waiting_ in the morning for each of 40 updates?
Thu, 09 Oct 2008 10:03:33 -0400
Thunderbird 184.108.40.206 (X11/20080707)
Please read Arthur's comments, as they all apply, including the comment about
chaitu wrote, On 10/08/2008 09:57 PM:
Hi Todd,Mark, everyone,
I've another tricky situation to beat. Got this huge 3rd party code
repository (+2G of code) that I need on each of my 40 build machines
(for different platforms of course). This repository is compiled
everytime there's a code change and the resulting libraries are used
for compiling my product's code.
Is this 2GB of very small text files or 2GB of large (>2MB) binary files?
Are you pulling in the source code for this library, or the resulting compiled
library, or both by using CVS?
The problem is that I need to fire cvs update on each of the machines
early in the morning before I start building my product. And just the
damn update takes over 40 minutes. And since all machines start this
update process at around the same time, there's some bandwidth crunch
Ever hear of cron?
I would have the Unix hosts issuing the update calls for me, probably having
each machine give the previous ~30 minutes to run before starting it's own,
early in the AM such that they should be finishing around the time I normally
I think even windows has a time of day command runner, but IIRC some user has
to be logged into the box for it to work.
Also are you using either the -z option with cvs or the -C option of ssh
(note: use one or the other, not both)? I think -C on ssh is the better one
as it compresses all of the cvs traffic not just the file transfers. Note:
IIRC you would have to put the -C in ~/.ssh/config, so the easy test is -z.
when one machine is running update by itself, i.e., the other 39 are not
updating while it is, how long does an update take? how long does that update
take if the sandbox is already up to date?
cd my/sandbox/ ; \
/usr/bin/time -p -o updatetime cvs -z update ; \
/usr/bin/time -ap -o updatetime cvs -z update ; \
note: some people need reminded that my/sandbox/ needs replaced with the
location of their sandbox in the file system.
I was thinking of this solution around the problem and would
appreciate if you can chip in with your criticism.
Update on only 1 machine, and from the update log, get all the U and P
entries and create a tarball which I can send over LAN to the other 39
machines and unzip them.
if you are going to trap the [UP] entries, I would then take that data and
issue cvs commands on the other machines instead. using cvs commands will
keep the metadata correct and would stop the server from looking through it's
tree to see if OTHER files had changed, i.e., you would run the risk with this
or your method that if someone checked in a change between your first machine
running and the last, you could get old and inconsistent data.
Do you have 40 different architectures or different versions of the OS for
which you are building?
if it is more of a thing where you are building 5 different released versions
of your code for windows XP i386 on 5 different machines but against one
version of the library, this is one place where I would build the lib on one
XP machine and SMB share the _resulting_LIB_ to the other 4. with the caveat
that I don't expect the versions of any other libs on those machines to affect
the build of the lib you are building.
Crane Division, Naval Surface Warfare Center (NSWC Crane)
Harnessing the Power of Technology for the Warfighter