Re: new procedure with GitLab CI

[Jean Abou Samra]

Hi everybody here,Valentin Villenave wrote:

> Hey Jonas, hey everybody;
> just so you know, I’m getting increasingly frustrated -- as you may
> guess if you take a glance at the thread on
> https://gitlab.com/lilypond/lilypond/-/merge_requests/95#note_351258804
> -- with the GitLab process.
> There *are* definite improvements over the previous arrangement
> (although James may probably be in a better position than us to
> appreciate this).  But here are my three pet peeves at the moment:
>
> -- Issues & Labels.  Not that I’m particularly keen to revisit that
> matter here; I’ve learned my lesson and stopped trying to intervene
> and triage any Issues; it remains to be seen whether
>      a) Milestones are a good fit to replace Fixed_2_xx_xx labels;
>      b) we’re scrapping the process of a) marking as Status::Fixed, b)
> waiting for the next GUB unstable release, then c) going through them
> one by one (ideally by somebody else) and d) marking them as
> Status::Verified.  I’m very much against ditching that; Jonas,
> however, thinks that (IIUC -- please correct me if I’m wrong) as soon
> as an issue is considered “Closed” within GitLab, its status is no
> longer relevant.
>
> -- The need to let the pipeline run every time an MR branch is
> touched: rebasing (no matter how many commits behind one may be),
> addressing a reviewer’s suggestion (no matter how minor), on penalty
> of getting stuck in Patch::needs_work land and getting a distinct
> chance of falling through the cracks.  I *am* very much in favor of
> every patch/MR getting through the pipeline when it first gets posted
> and at least once or twice before it gets merged; I just think that
> during the reviewing process the pipeline *could* be put on hold for
> minor changes while we’re getting stuff in shape.  (And *no*,
> needs_work is not appropriate in that sort of situation.)
>
> -- The merge window (every three days) is getting ridiculous.  Now
> we’re all asked to check, re-check, double re-check, triple re-check
> the pipeline list to hopefully get our branch rebased and merged at
> some point; if we’re unfortunate enough to stumble onto another
> merging process already engaged, there is nothing else to do but come
> back an hour later, quite possibly only to discover that someone else
> got in first.  I’m not blaming anyone for that, but surely there must
> be a way of smoothing out that process, for example some sort of a
> waiting list.  Maybe something as dumb as sending an email on -devel
> or writing down our MR’s number on an Etherpad somewhere.

Han-Wen Nienhuys wrote:

> I concur that this is problematic. Today, I spent several hours
> shepherding my multiple MRs through the process, which was
> additionally complicated by David and Valentin doing the same.

David Kastrup wrote:

> So I lean towards the following process modeled upon our old one:
>
> No push to master except by automation.  Staging is unprotected and gets
> scheduled runners.  People ultimately rebase and push there once they
> get Patch-push status.  Default branch can be master as long as we make
> sure that one cannot push there: that way you never rebase on ephemeral
> commits.
>
> When James gives Patch-push status (which he hopefully does supported by
> a script), at the same time any merge request pointed at master gets
> changed to point at staging.  So if you are "Patch-push", the "Push"
> button starts working.
>
> That means as long as staging isn't stuck (and we should make sure that
> the scheduled pipelines do not pick up the same commit more than once to
> avoid wasting our minutes), you can rebase on master when you want (or
> not).  One initial pipeline run should be require to get into "Review"
> state.  Rebasing puts you back to "Patch-new" (which for trivial stuff
> you may choose to hand-replace with "Patch-review" at the danger of
> later getting staging borked).
>
> It would be nice that if staging has tested as being borked, we could
> temporarily disable pushes to staging: that will keep the patches and
> merge requests requiring redoing limited.  We got out along without that
> previously but if the automation is good for that, that would be great.
>
> At any rate, this scheme naively does not seem to require stuff
> impossible to do with Gitlab, at least for the basic parts of operation.
> And it would mean that you can usually rebase and forget without waiting
> for pipeline completions.  And you get the liberty of bypassing most CI
> and rebases if you don't want to.
>
> I may be wrong about the possibility to do this with a reasonable amount
> of effort: I don't want to get hopes up unnecessarily.

That sounds reasonable, but alas, rather complicated. What about the
other option, that is, switching to a non-linear history, with merge
commits?

My fear is that we get back to a complex and non-standard development
process. The simple fact of having two branches to look at is already
a source of confusion for the unexperimented (for instance, merge requests
will target master at the time you create them, but that will change for
staging automatically). From the information I could gather, one of the
main incentives for moving to GitLab was making it easier to contribute
by standardizing the workflow. GitLab does everything we want, the only
blocker is the fast-forward merge strategy, if I understand well (please
correct me in case I don't). I would happily live with merge commits if
that facilitates development for everyone.

Plus, a non-linear history would skip all the rebases, which are creating
unnecessary noise within the merge requests. And it would be a lot less
work to set up, given that Jonas expressed to be fed-up.

David Kastrup wrote:
> I don't think it would be a good idea to start a pipeline on somebody
> else's merge request without having an indication that they were going
> to follow through with "Patch_push": Patch_push indicates no objections
> from foreign review, but the responsibility for starting the final
> pipeline/merge should really lie with the original submitter.
What about new contributors who don't have the required permissions to
push? (By the way, thank you Jonas and Valentin about that, sorry that
I didn't have time to reply but I appreciate your trust).

Valentin Villenave wrote:

> And, yes, I am grateful towards Jonas for everything he’s done and the
> increased activity we’re seeing because of his work.  I’ve already
> tried to talk things through off-list; now I think we need to talk
> things through with others in the team.  Jonas, you may not know me
> well enough to know how emotional I tend to get, and how easily I tend
> to just give up and drop off the map (the difficult times I’m going
> through IRL, as some of you guys may be, certainly don’t help); so
> please, please don’t take this message as an attack, because it
> certainly isn’t intended as one.  (If anything, this is me trying to
> de-escalate a potentially tense situation; the words I’m using may not
> be the most appropriate ones in that regard, but they’re the only ones
> I have at hand so I apologize for that.)
>
> Cheers,
> -- V.

You're not alone, Valentin. From what I could read so far, LilyPond
contributors very often get emotional quickly. That is unusual enough
to be noticed. I believe this is because we are all musicians, and thus
tend to be particularly sensitive.

Of course, this exacerbates tensions when there are some. But it also
entails a friendly community where people demonstrate enthusiasm.

(One example 
beinghttps://lists.gnu.org/archive/html/lilypond-devel/2018-03/msg00029.html.)

And, I very much understand the difficulties you're facing in real life,
the nerve-wracking French lockdown, the thorny situation artists
face, etc. So if LilyPond is an additional cause of frustration for you,
maybe it could do you good to set it aside for a while and focus
on your current problems. At least, that's what I did.

Best,
Jean Abou Samra