Re: [Libcdio-devel] [RFC] New API iso9660_statv2_t as API/ABI compatible way to read files >= 4 GiB

[Pete Batard]

On 2018.07.09 08:33, Thomas Schmitt wrote:
> * Never break the API (source code compatibility) of a library.

Unfortunately my findings are that, after a while, experienced 
developers come to their senses and realize that focusing too hard on 
having an immutable API hurts *EVERYONE*, and especially the end users.

Even if it took them a while, it seems that the kernel developer have 
started realizing this as well (even if, IMO, they are still some way to 
full enlightenment): See 
https://www.kernel.org/doc/html/latest/process/stable-api-nonsense.html

That's not to say that API breakage is to be taken lightly, because that 
is not at all the argument I am trying to make, and I really have to 
stress out that I am not using the points I am making below as any kind 
of justification to be cavalier about API breakage (though, as far as 
I'm concerned, ABI is a different matter). But my view is that we're all 
in this together (meaning upstream & downstream developers as well as 
end-users) and that we must strive to develop software that will benefit 
the whole population, and _not just a few privileged individuals_.

As such, the idea that upstream should _never_ break an API, because 
there might be downstream developers who can't/won't (be happy to) 
update their software is actually a very dangerous idea to try to 
enforce, because, in general, you are breaking the API with the goal of 
benefiting the end users (otherwise, it would make very little sense to 
do so), which you have to consider is a much larger population than the 
downstream developers, and are the group who have the more to benefit.

So, in essence, by withholding API changes because downstream developers 
might be unhappy with them, you are effectively *hurting* the end-users, 
who won't see changes that are beneficial for them being enacted.

Well, considering that our goal is to make the best software we can, 
within the time and resource constraints we have, we need to consider 
that we are all in this together, and that, yes, one group may have to 
push another group to go through something they would rather avoid, _so 
that the whole population benefits_. This may mean avoiding an upstream 
library going through a hefty alteration of much of its stable code, 
when we can very much minimize these changes (and ensure that we are 
unlikely to introduce new issues) even if it comes at the cost of 
breaking its ABI.

Also, please please bear in mind that I am very much seeing this as a 
two way street (because, we're all in this together), meaning that, when 
downstream developers come to upstream because they have found a problem 
that is hurting their development process, and ultimately their 
end-users, then upstream should be flexible enough to consider doing 
work to alleviate these issues, even if they can justify ignoring said 
issue as well as the unwanted additional work, altogether.

This is how software development should work: not as groups whose needs 
seem to oppose the needs of the other group, bus as groups who realize 
that we are all part of a larger group, and that it means that, from 
time to time, part of the group has to take on extra work, even if, in 
the absolute, they are entitled to make a point they should not have to 
do said work, for the benefit of the larger population.

So, in essence, what I am saying is: we need to stop treating downstream 
developers & package maintainers like porcelain dolls. Yes, 
_occasionally_, we will place additional burdens on them, and we expect 
them to come to accept them in their stride. Just like we also expect 
them to place some burden "against" upstream from time to time, when 
they feel they have grounds to do so, that we should not reject with an 
all-to-easy "this is an annoyance that we can make a case against, so we 
will simply dismiss it wholesale", especially if we know that this extra 
work is going to help downstream (and ultimately the end users).

Also please bear in mind that, as a downstream developer, I very much 
used to sit on the other side of the fence, and bark at the idea of any 
upstream libraries breaking their APIs.

Then, one day, as I was using a library that had *really* broken its 
whole API from head to toe (I believe it was LUFA, an USB stack for 
embedded ARM), in such a way that I had to completely re-learn the 
interfaces and update almost every function I called in my code, and 
once a couple weeks had elapsed, I realized that barking at API breakage 
was nonsense. Sure, I was still pretty annoyed that I'd had to rewrite 
my code, but then it dawned on me that simply taking these breaking 
changes in their stride achieved two very important goals:

1. Ensure that "mistakes" or poor initial design decisions from upstream 
were corrected as needed, which ultimately would lead to my own software 
being easier to develop and maintain (even if it meant a new major 
effort right now) as well as a providing general improvements to the end 
users.

2. Ensure that my software had not become "stale" and/or in pass of 
becoming an untouched "liability", but instead was being actively 
maintained, _as all actively used software should be_, which is the 
point I touched on earlier. Again, the weeding out "stale" software is 
something that I believe is very beneficial for end users, yet something 
that is so difficult to enforce, so much so that we might as well use 
API/ABI breakage as a means to enact it (but of course, _with restraint_ 
and only when we feel have otherwise good reasons to update an API/ABI)

>    This is quite a necessity for all distro maintainers, who have to
>    build software which they do not know in depth.

Yet, we're all in this together. So, while I have a lot of sympathy for 
distro maintainers (I genuinely do - It's a much more thankless job than 
being a software application developer), I will assert that if their 
goal is provide the best package for their end users, as well as ensure 
that developers of existing and new software using said package is 
developed in the best possible way, they will acknowledge that, if the 
upstream developers of a package want to push a change that will require 
extra work of them, they usually have a very good reason to do so (which 
I do argue we do have, in the ABI breakage option), and that, in turn, 
they can feel assured that we are going to turn a sympathetic ear to 
their requests when they have some (such as, as I have also been 
proposing, not introducing a breaking ABI change in a library right 
after we went through an API change, but instead giving them enough 
breathing space and trying to wait until we can group this change with 
something else).

>    It is also a courtesy towards the developers of applications who
>    would else be forced to change their code without own motivation.

There is courtesy. And then there's treating one group of people as 
being "more equal" than the others, whilst ignoring the drawbacks of 
doing so, for the end users.

I only have one objective in mind here, which is to ensure end users 
always get the best software they possibly can. And as I have no 
explained may times over, achieving that goal may mean having to nudge 
developers, who might still fail to see the bigger scheme, into 
participating in that goal, even if they would prefer not to.

> * Try hard to keep the ABI usable for old programs which have been
>    compiled to use an older version of the dynamic library.
>
>    This is a courtesy towards the users of applications. They can carry
>    them to more modern versions of their operating system without the
>    need for recompilation (which might be not trivial).

Strong disagreement on this one.

If you do that, then users of the application find that they can't 
process the blackarch-linux ISO at all (even worse, what they will find 
is that the software they use seems to process it just fine, but 
silently "forgot" to copy a couple GB of data over - ouch!!).

The courtesy here, is to ensure that end-users don't get in a situation 
where their old application corrupts data without the slightest warning. 
And that is what is likely to happen if we spend a lot of work ensuring 
that nobody realizes that, yes, they do need to start caring about 
multiextent in their code. Heck, the first thing I did when I realize 
that libcdio was missing multiextent support, resulting in some of my 
users getting truncated data when transfering an ISO to USB was uttering 
something like "Wow! As a stable ISO-9660 library that has existed for 
years, this is a major letdown if it doesn't even warn you that it's 
just skipping over large amount of crucial data..."

So I'd rather annoy quite a few users with finding that their old 
application doesn't work, even if they aren't going to process 
multiextent images, than give my blessing to a change that will result 
in even a single person losing valuable data when they copy a unique 
multiextent backup they created from one medium to another and get a 
notice from their application that the copy was A-OK, without realizing 
that they might just as well have issued "rm -rf 
./my_valuable_backup.tar.gz"...

> * Design the API/ABI for being expandable without conflicting with the
>    first two rules.

Pious goal. But realistically unachievable in my book.

> This ideal is quite common in the plumbing layers of GNU/Linux.

Yet, as I pointed out, the kernel driver maintainers seem to have a very 
different opinion. So I would say, that, no, contrary to what you 
assert, this ideal is not common in the actual plumbing layers of 
GNU/Linux, and that some developers are rallying to the idea that we are 
all in this together, for the benefit of end-users, and that, as a 
result, while we should of course strive not to treat API/ABI breakage 
in a cavalier fashion, API/ABI breakage is something we need to stop 
pretending is hurtful, whereas it helps make software better for end users.

> I blame the short breath of GNU/Linux GUI developers on that porcellain
> fragility.

Perhaps. But if that's the case, I don't believe it is "right" for any 
developer to bring the whole boat down for users because they don't want 
to plug the holes they are supposed to be responsible for. Nor should we 
have to build a very costly second hull around our boat (which is what I 
would assert we're proposing to do here), so that we let the people who 
don't want to plug holes have their way...
We're all in this together.

Regards,

/Pete