Re: [Monotone-devel] newbie question - SHA1 vs serials

[K. Richard Pixley]

Emile Snyder wrote:

On Wed, 2005-04-20 at 10:20, K. Richard Pixley wrote:
  

My point is that currently available tools offer differing levels of 
support before solving the problem manually is required.  128 concurrent 
developers is only rare because existing tools have maxed out in this 
range forcing people to think in terms of solving the problem manually, 
(ie, partitioning), much sooner.  Partitioning has a high overhead so 
we'd really prefer to avoid it for as long as possible although 
partitioning is better than failing to solve the problem.
    

Could you say a little more about how existing tools support this
massively concurrent mode better?  To be concrete, in a previous message
you referenced a case where you have ReallyImportantHeader.h that
everything depends on and that is churning alot.  I'm probably just
lacking in imagination, but if you want all your developers to get code
back on mainline quickly (ie. want to avoid partitioning to slightly
longer lived branches) how do you get away from spending lots of time
merging?  Or do you just want the tool to force you to merge more often?
  

I want to be able to determine how many branches exist, who created them, who maintains them, what the common usages are, who's using which, and typical code churn in each.  I want to be able to determine code pedigrees at a glance, merge status at a glance, I want to be able to create a sandbox using bits from different branches and I want to be able to grep the heads of all branches for particular strings.

How's that for a start?

You can do this today.  Run a second copy of monotone on the central 
machine, pull with it from the first.  Stop it.  Back up its DB. 
Repeat.  Presto, continual backups, as often as you want, of the 
central repository.
      

No key distribution.  No backups of system software or configurations.  
What I really want is the ability to reload the original machine's 
(replacement) hard disk with a verbatim copy of whatever it had 
yesterday.  I'm talking about disaster recovery, not necessarily 
archival, court records, or data redundancy.
    

Is this really in the scope of monotone?  Seems like it's better handled
by hosting the main monotone db/server on a system with a snapshotting
filesystem or something.
  

It's in the scope of CSM.  In the largest organizations, CSM and IT are done by different people and it's not entirely clear to me where the handoff is here.  That is, where do my responsibilities as a CSM professional and a developer end and where do IT's begin.  Certainly, if I want them to do backups, I need to be able to tell them when the info on disk is stable and recoverable.

Snapshotting file systems were expensive last time I looked and I don't know any that run on linux native. 

But the real problem isn't snapshotting the file system - it's making sure that the database which is stored by a snapshot can be restored/recovered/reused at a later time.  Between memory caches of file systems, index caches, and the order of writes, we have vulnerabilities here due both to the file system and also to the database system. 

The standard initial answer is simply to shut down the CSM and database, make them unavailable for a while, block access to the file system, sync it, then back it up.  For small groups, this is usually acceptable as there's usually some hour of the day when everyone can be reliably predicted to be asleep.  This doesn't scale well.  The "shut everything down to single use mode" solution is easy.  But it means your respository isn't available for some period of time.  For active repositories, this isn't acceptable.

The modern answer is to construct your database and file systems in such a way that either data hits disk in an order which is always recoverable, (in which case any snapshot is recoverable), or such that the system can be forced to flush all caches and create a consistent disk state, even if it's only briefly.  Clearcase uses the latter at this point.

I don't think so.  I take ease of branching for granted.  It's 
management of the branches that seems to be an issue in most free 
systems today.  Contrast to clearcase branches, which, while nominally 
more user overhead, are significantly and seriously easier to track and 
manage.  In a distributed system, I can't even ask what branches exist, 
since no perspective is necessarily even complete.
    

What sort of support is clearcase giving you that makes
tracking/managing branches more to your liking?

See features listed above.

(I have no clearcase
experience, sorry.)  How is the fact that someone might have a branch
that's not pushed to the canonical server different from the case where
a developer has some working dir that they haven't checked in at all
yet?
  

I may be confused.  I was under the impression that all revisions propogated to all repositories.  So you're talking about a geographical branch in the sense of a repository which simply hasn't sync'd, yes?

I would hesitate to host large mission critical projects with monotone
just yet; seems like we're still working out quite a few kinks that
people who don't want to hack on monotone would be pretty annoyed with. 
But it sure is progressing quickly ;)
  

Thanks.  That's exactly what I've been trying to guage - how close, how much work, how much time, etc.  My current client may be willing to accept some modest level of maintenance if basic stability is good.

--rich