Re: [PATCH 1/2] Support publishing build events

[Clément Lassieur]

Christopher Baines <address@hidden> writes:

> Clément Lassieur <address@hidden> writes:
>
>> Christopher Baines <address@hidden> writes:
>>
>>> I guess so… I was probably just thinking about JSON from JSON fields
>>> in PostgreSQL, but we're not using PostgreSQL here.
>>>
>>> The one advantage I can think of is that not every Scheme object can be
>>> converted to JSON (at least with guile-json), so by storing the JSON
>>> representation, you're guaranteed to have a valid value. If you stored a
>>> scheme object, there's a chance that you could store something that was
>>> invalid, or at least didn't convert to JSON.
>>
>> Do you know any Scheme object that cannot be converted to JSON?  Even
>> procedures can.
>
> I was thinking of something like this:
>
> scheme@(guile-user)> (scm->json-string '(()))
> json/builder.scm:91:21: In procedure build-object-pair:
> In procedure car: Wrong type argument in position 1 (expecting pair): ()

Oh right, got it!

>>>> But in that case, why not fetch them from the Guix Data Service?
>>>
>>> I'm still not sure what the ideal scope and architecture for the Guix
>>> Data Service, or Cuirass is, but at the moment I've been trying to avoid
>>> making one dependent on the other.
>>
>> Well, if we want to keep our softwares (GDS and Cuirass) as simple as
>> they can be, there should be a clear delimitation about what each of
>> them does, and they should not implement the same features, in my
>> opinion.
>
> At least in my mind, this is an important issue, but something I'm very
> uncertian about. The biggest existing overlap in my mind is that the
> Guix Data Service and Cuirass compute and store derivations.
>
>> Also, I can't see how GDS wouldn't be dependent on Cuirass.
>
> Maybe dependent wasn't the best way to phrase it. What I've been
> thinking about is you should be able to run Cuirass without the Guix
> Data Service, and this might be very applicable for small deployments.

Oh yes, of course, I see.  That's why I was talking about an option to
enable/disable DB-ADD-EVENT in my other email.

[...]

>>> Also, I've been thinking about WebSub when trying to provide a way to
>>> subscribe to events, which I'll say more about shortly…
>>
>> I'd love to understand what WebSub is too!
>
> So I meant to say more in my last email, but I obviously forgot to
> elaborate!
>
> So WebSub is W3C recommendation standard (that doesn't carry much weight
> with me, but I was looking for some kind of standard or interoperable
> way of doing things). In it's model, I've been thinking of Cuirass as a
> "Publisher" [1].
>
> 1: https://www.w3.org/TR/websub/#conformance-classes
>
> WebSub describes subscripting to topics, through their URL. Which is why
> I created URLs to serve the build and evaluation events. I also think
> some people may want to just fetch the events, rather than subscribing
> to them.

Great, thank you for the explanations!

>>>> Now that I think about it, I don't understand the point the HTTP API
>>>> ("build-events", "evaluation-events").  If events are sent from Cuirass
>>>> to the Guix Data Service, why would anyone need to fetch those events
>>>> from Cuirass?  Wouldn't it make more sense to fetch them from the Guix
>>>> Data Service?
>>>
>>> So, while this work is trying to copy data from Cuirass to the Guix Data
>>> Service, I'm still thinking about them separately. I guess if you're
>>> running your own instance of Cuirass, you might want to fetch events
>>> from it, without having to setup an instance of the Guix Data Service.
>
> So adding a bit to what I say here, I added the API endpoints also to
> keep somewhat aligned with WebSub.
>
>> Is there any use of having raw Cuirass events without the Guix Data
>> Service to interpret them?
>
> I think so, you could hook them up to an IRC bot for example, to post in
> some channel when events happen to builds.
>
> It's true that the information in the events table can also be found
> elsewhere in the database, it's just not in an ideal form to send out to
> some receiver.

But can that wanted form be made out of what we already have, with an
SQL select query?  What are we missing?

My main worry is to have redundant tables that keep growing.  So if we
already have all the informations we need, wouldn't it be better to find
a smart way to use them?

> I'm still interested in getting Cuirass to publish events to a WebSub
> hub, at which point a topic URL for build events would be necessary (at
> least in terms of WebSub) to subscribe to events.
>
> However, at the moment, I'm trying to do something minimal to get data
> from Cuirass in to the Guix Data Service, and I haven't set up the hub
> needed for WebSub.
>
> Maybe it would be simpler to remove the API endpoints for now, and they
> can possibly come back once there's a WebSub hub involved.

I agree, going step by step seems like a good idea.

>>>> Why do we need to use a fiber, if there is only one?  A fiber is a
>>>> lightweight thread.  They are useful if we need to do several things in
>>>> the same time, but if there is only one thing to do, I don't understand
>>>> the point.
>>>
>>> If I remember correctly, the way Cuirass accesses the database is tied
>>> up with fibers, so it's to make the database access work here.
>>
>> It's not tied up with fibers.  There is a database server in another
>> thread, and WITH-DB-CRITICAL-SECTION will allow you to communicate with
>> that server through channels.  But you don't need to be in a fiber to
>> communicate through channels, being in the main thread is good enough.
>
> Ok, I'll have another look at how fibers are used when sending out
> events.

It's just a few lines to remove, nothing troublesome I believe.

Thanks,
Clément