guix-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: More progress with the Guix Data Service


From: Christopher Baines
Subject: Re: More progress with the Guix Data Service
Date: Fri, 17 May 2019 19:26:56 +0100
User-agent: mu4e 1.2.0; emacs 26.2

Amirouche <address@hidden> writes:

> On 2019-05-17 09:56, Christopher Baines wrote:
>> Hey,
>>
>> In summary, I think the Guix Data Service might be getting useful
>> enough
>> that setting it up properly might be a good next step, and I'd be
>> interested in what others think?
>
> Yeah. And it looks good :)

Thanks :)

>> A bit over a month ago, I sent out an update about one of the things
>> I've been working on, something I've been calling the "Guix Data
>> Service" [1].
>>
>> 1: https://lists.gnu.org/archive/html/guix-devel/2019-04/msg00094.html
>>
>> Since then, I've made some more progress. There's a new statistics page
>> [2]. I've got used to using Sqitch [3] to help manage the database
>> schema, and I've added some basic tests.
>
> statistic page is a 404. I am very interested to learn how you use
> sqitch.

Sorry, the link is wrong, it's actually:

  https://prototype-guix-data-service.cbaines.net/statistics

As for Sqitch, all I've been doing so far is sqitch add ... to create a
change, sqitch deploy ... to apply changes to a database and sqitch
revert ... to revert changes to a database. It's just some structure and
tooling that I don't have to write :)

>> 2: https://prototype-guix-data-service.cbaines.net/statistic
>> 3: http://guix.gnu.org/packages/sqitch-0.9999/
>>
>> The error handling for loading new revisions is also more resilient
>> now.
>>
>> As well as listening to the Guix Commits mailing list for emails about
>> new revisions, more of the information in these emails is now
>> stored, in
>> particular, the time they were sent, and the branch the email applies
>> to. This can be seen on the new Branches page [4].
>>
>> 4: https://prototype-guix-data-service.cbaines.net/branches
>
> Here when I click on a commit, I would expect the message of the commit
> and the build status of the impacted softwares, e.g.
>
> https://prototype-guix-data-service.cbaines.net/revision/d59c90a5bfdd6b723bea939b8538c7c9b3c1b2a6

I've just realised the table headers on the branches page are the wrong
way around.

Anyway, I've added some information about git repositories to the
database so I could link to cgit for the location of a package, so I'll
have a look at also putting a link to commits on that page.

As for build status information, this is something I talked a little
more about in my last email (link at the top), but in summary, I've done
some initial work, but more needs doing to get the build status
information in to the database.

>> The content negotiation a little bit, at least in terms of the code,
>> and
>> JSON output support has been added to more pages.
>>
>> There's now a basic search function on the packages page [5], and the
>> location, and the licenses for packages is now being stored (which can
>> be seen on the page for a package, for example [6]).
>>
>> 5:
>> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages?search_query=Guile&field=version&field=synopsis&after_name=&limit_results=1000
>> 6:
>> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/package/0ad/0.0.23b-alpha
>>
>> The location and license information is something I added specifically,
>> as I noticed the Repology [7] service scraped [8] these from the Guix
>> website.
>>
>> 7: https://repology.org/
>> 8:
>> https://github.com/repology/repology/blob/master/repology/fetchers/fetchers/guix.py
>>
>> While the Guix Data Service started as something to enable better
>> understanding patches in an automated way, I think there are more uses
>> for it, and initially, it's probably better to focus on the simple
>> ones.
>
> It is not clear right now that it is related to patches, there is no
> patch
> anywhere to see.

Indeed. What I've been using the Guix Data Service for is working out
what a series of patches does. For example, on my test Patchwork
instance, you can see this series of patches [1], and there's a
corresponding page from the Guix Data Service showing what the effect of
the patches is [2].

1: https://patchwork.cbaines.net/project/guix-patches/list/?series=1048
2: 
https://prototype-guix-data-service.cbaines.net/compare?base_commit=4d46775b1d04f9edb6fd729cfcd4b745be74e03b&target_commit=b3cfde472e5b2e6a9489a1b6c437daa659a977a6

>> The Repology use case is pretty simple, I think ideally there would be
>> some machine readable data about the current state of packages in Guix
>> available over the internet, and Repology would be able to download
>> that
>> on a regular basis.
>
> Repology is nice, but I would prefer wikidata support.

Interesting, I don't know much about wikidata, but taking Guile as an
example, it looks like wikidata is tracking distributions that package
Guile, along with the Free Software Directory [3]

3: https://www.wikidata.org/wiki/Q1486208#identifiers

Linking to other data/representations of packages from the Guix Data
Service has been something I've been thinking about. Especially things
like the Free Software Directory. Maybe wikidata would be a good think
to link to. I'd also be really interested what would be required to get
wikidata linking to Guix.

>> The URL is a bit long, but I think that is now close to being possible
>> with the Guix data service. I haven't got something working yet to
>> easily access data for the latest revision, but for a particular
>> revision, you can request a JSON file containing all the information I
>> think Repology currently gets about all packages. For example:
>>
>>
>> https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&limit_results=99999
>
> FWIW it is very slow.

To be honest, I'm actually pretty happy with the speed! I was worried
that NGinx was going to time out and that it wasn't even going to work.

I think there's probably some optimisations I can do to the queries to
make it faster, and I haven't added any caching anyware yet, and it
should be possible to cache lots of things, which will help lots.

>> Does anyone have any thoughts on this?
>
> It seems to me it would be useful to have that informations
> somewhere. Like you
> explain having the correct build information is a must have IMO along
> the logs.
>
> Where is the code?

In a git repository here: https://git.cbaines.net/guix/data-service/

Thanks for taking a look,

Chris

Attachment: signature.asc
Description: PGP signature


reply via email to

[Prev in Thread] Current Thread [Next in Thread]