bug#45477: 27.1; RFE: Make full RSS fragments available for nnrss servers

[Tim Landscheidt]

Lars Ingebrigtsen <larsi@gnus.org> wrote:

>> Therefore, I need the data, and I need it in a format that
>> can be processed further, and there will be a need for a
>> custom user function to process the data because each use
>> case will be different.

> Sure, that sounds reasonable.  However:

>> (If there was a major overhaul of nnrss, it could be inter-
>> esting to forego the intermediate nnrss-group-data saved in
>> ~/News/rss/* and either store the feeds as pure XML files,
>> re-parsed on demand and available for further processing, or
>> write out all the articles as mbox files after parsing the
>> feeds, with the entries' fragments as MIME parts.)

> I've not used nnrss myself, but reading the code, it seems like it's
> storing all the data needed for Gnus to read an nnrss group in
> `nnrss-group-data', so storing all the XML data in case somebody is
> going to use it would require orders of magnitude more storage?

In my practice (so far), not even one magnitude.  Random on-
disk sample:

|   -rw-r--r--. 1 root root  132680 Dec 30 06:56 Conan O’Brien Needs A Friend.el
|   -rw-r--r--. 1 root root  471799 Dec 30 07:28 Conan O’Brien Needs A 
Friend.xml
|   -rw-r--r--. 1 root root   72249 Dec 30 06:56 Doug Loves Movies.el
|   -rw-r--r--. 1 root root  245312 Dec 30 07:01 Doug Loves Movies.xml
|   -rw-r--r--. 1 root root  630495 Dec 30 06:56 ID10T with Chris Hardwick.el
|   -rw-r--r--. 1 root root 2100500 Dec 27 21:30 ID10T with Chris Hardwick.xml
|   -rw-r--r--. 1 root root   21754 Dec 30 06:56 Sprechen wir über Mord?! Der 
SWR2 True Crime Podcast.el
|   -rw-r--r--. 1 root root   47741 Dec 30 06:36 Sprechen wir über Mord?! Der 
SWR2 True Crime Podcast.xml
|   -rw-r--r--. 1 root root   93927 Dec 30 06:56 Stone Clearing With Richard 
Herring.el
|   -rw-r--r--. 1 root root  221040 Dec 30 07:25 Stone Clearing With Richard 
Herring.xml
|   -rw-r--r--. 1 root root   17002 Dec 30 06:56 Taskmaster The Podcast.el
|   -rw-r--r--. 1 root root   53080 Dec 30 07:28 Taskmaster The Podcast.xml
|   -rw-r--r--. 1 root root  265970 Dec 30 06:56 You Made It Weird with Pete 
Holmes.el
|   -rw-r--r--. 1 root root  650710 Dec 30 04:07 You Made It Weird with Pete 
Holmes.xml

Even if the XML gets bloated when saved in nnrss-group-data
(it holds one feed at most), IMHO almost all feeds will be
small enough to be negligible in a typical Emacs/Gnus setup
(the largest feed above holds data from February 2010 till
now; usually feeds only contain the most recent x entries).

> I think a way to implement this would be to add an nnrss variable that
> says what "extra" XML fields to store -- like (nnrss-extra-fields
> '(itunes:episodeType ...)).

That would allow my use case.  (In a major overhaul, another
way to approach this could be a hook/function variable (con-
figurable per group) that gets called in addition/in lieu of
nnrss-request-article with the raw XML data and then has
free rein to format the Gnus article as it wishes to.)