unsuitable protocols and standards that block innovation

[carlo von lynX]

On Sat, Mar 09, 2024 at 10:30:07AM +0200, email@msavoritias.me wrote:
> XMPP can work fine through p2p. Its just not being done a lot because of
> other reasons i can expand if needed.

Yes, you can rewrite any protocol radically enough to cook rice
or iron your shirts but then you are really just keeping the
unfavourable syntax and nothing truly useful.

Here Google is promoting a JSON API that has all its semantics
turned upside down from the original XMPP RFC, but they still
call it xmpp, as it probably sells better and gives an impression
of legitimacy:

https://firebase.google.com/docs/cloud-messaging/xmpp-server-ref?hl=en

What stuck out quickly to me was the presence of a message_id and
a message_type. Two things that were vehemently refused by the XMPP
Council 20 years ago when we suggested they should use something
like this. Message ids got a clumsy extension XML namespace and
message types were never introduced at all, forcing all applications
to deduce the semantics of a message by parsing the entire tree,
even if it wasn't intended for them.

Ten years earlier Google kicked XMPP out of the Android backend
saying it is too verbose for mobile. 20 years earlier some people
probably involved with Google were in favour of ratifying the XMPP
RFCs even though it had no scalable one-to-many distribution
strategy. Back then Google Talk became the largest XMPP node
because it was the only one that was able to do one-to-many
distribution all inside its proto-cloud-architecture.

In other words, even back then XMPP was working best when it said
so on the wrapping but it wasn't actually being used. Same with
Androids xmppService, and again with Firebase.

So if we're going to do an efficient new communications system
over GNUnet, there isn't anything but the #XMPP hashtag that
could be of any use. Just call it XMPP, but please don't use it.
No XML, no XEPs and nothing of what's in the RFCs. Just use
the things that are being used to de-facto instead, and maybe
consider what it would be like to design a protocol with the
right approach from the start, rather than this thing which
started on two wrong feet back in 1998 and never got much better.

> > Also I doubt the general public would be interested in a
> > messenger that can't deliver voice, video and stickers
> > embedded into the protocol natively without overhead.
> 
> What do you mean "overhead" or "natively" here? Because every single
> protocol out there works with extensions. In the sense of virtually all
> protocols nowadays use webrtc which is not native to anything (except web i
> guess but what is web :P )

That's why we still don't have a real-time capable walkie-talkie
type of communication tool. The latency introduced by launching
external protocols rather than using existing streams immediately
makes that impossible. You can send voicies and you can negotiate
phone calls, but neither of these are as comfortable to use as a
walkie-talkie.

> and second can you give an example of stickers integrated natively into the
> protocol? Because I can't imagine why stickers would be a protocol level
> thing since they are just pictures. Its al UI. The only thing a protocol has
> to dictate is how to share packs which xmpp already does.

When someone is low on data, downloading a whole sticker pack just to
be able to show one of them is expensive. That's why Telegram sometimes
just sends you one sticker without passing the entire pack on. Also
having to go fetch the graphics instead of receiving them immediately
introduces waiting time - you get these grey boxes announcing that 
something will show soon, but you have to wait until it actually loaded.
Not saying that this specific example is better to solve differently,
but it has drawbacks to consider. In order to not transmit the same 
picture several times you either must know what the recipient already
has, or you end up using a pull protocol with its latency. How is latency
still a big deal these days? The planet hasn't gotten smaller and onion
routing is still on the list of things we'd like to achieve, so having
messaging systems less prone to latency is good.

> > IMHO it makes more sense to adapt the open source Telegram[2]
> > implementations to connect to a localhost server which then
> > offers backend functionality for Telegram to run over GNUnet.
> 
> I agree that telegram has a good UI but i fail to understand what that has
> to do with protocols. Unless you mean to also get the telegram server and
> change the whole thing to work over gnunet. Which at that point:

Since the protocols of XMPP aren't recommendable at all, Telegram's
JSON APIs are less bad and easier to adapt. Of course we could also
do an even better messaging syntax and protocol. We've pushed the
envelope times before.

> 1. If you are running a seperate process to connect over localhost is it
> really p2p?

Having the GNUnet stack in a single process is on the todo list for GSoC.
A Telegram-adapter would be yet another GNUnet service out of many.
As long as it is localhost, the whole architecture is still P2P.

> 2. I wonder how much work it would take to convert it to GNS and gnunet
> because i bet there a whole lot of assumptions in there about DNS and
> servers and such. So maybe the effort wouldn't be worth it. Especially with
> how easy android or Gnome make it to have a beautiful UI (if the app is
> native).

I would guess Telegram clients to be a lot more straightforward than
XMPP ones, as they don't use DNS at all (they just connect to the cloud).
So this point of critique applies to XMPP software much more than to a
Telegram client.

> > [1]https://about.psyc.eu/XMPP#Technical_Problems_with_XMPP
> > [2]https://telegram.org

> Regarding your technical problems article some remarks:
> 
> - XMPP can have framing if done over websockets. So thats not a compromise
> you have to make anymore if you want framing.

So you aren't actually using RFC-compliant XMPP. That's ok.

> - XMPP can work over Radio with 75 bits/s and minutes of reliability. Its
> not the most efficient that exists but if you have less bandwidth than radio
> you can use something else i guess :)

Did you look at the benchmarks? Why start with an extremely inefficient
syntax?

> - Streaming parsers exist

Yes, they needlessly consume CPU to guess what might be coming next.
Check the benchmarks.

> - For binary transfer I am looking into Eris or EXI personally

Because using the protocol you are using is not good enough?
As if binary data transfer was a separate problem.

> - The huge rooms problem is actually about presence not anything to do with
> the protocol. There are two solutions to this:

IT HAS EVERYTHING TO DO WITH THE PROTOCOL.
If the RFC forbids any decent distribution strategy, the
protocol impedes implementing scalable systems unless you
run them inside a cloud, bypassing XMPP.

> 1. Disable presence in rooms above a couple thousand people

That doesn't solve distribution at all.

> 2. Change to MIX the new way to do group chats in xmpp that is more
> scalable.

This implements (1) from what I can see, and thus only reduces the amount
of data distributed a bit, but not the challenge of distributing
stuff to millions of recipients in a Twitter-like use case.

> - The multicasting part looks very interesting. Can you expand on how that
> would work in a distributed network like gnunet? Because it specifically
> mentions a server solution which we cannot afford here:
> 
> > In PSYC <https://about.psyc.eu/PSYC>, by contrast, the server will contact
> a set of servers, which will forward the news to another set of servers
> each, until all the recipients receive the news

A multicast routing layer has been in the plans for GNUnet since before
we first came asking, which was around 2009. Research has been made on
how to do this, but there is no implementation as yet.

In GNUnet, nodes would efficiently set up distribution trees to get all
data to everyone who needs it. If done properly it can handle Twitter-
and even television-like use cases with millions watching the same video
stream. The current status quo is that these distribution trees are
implemented within the cloud, thus you only get to enjoy their strengths
if you surrender your cookies to them and join the cloud hegemoths.

XMPP was never a challenge to the cloud platforms, because it never
solved the one thing that needs to be solved in order to scale outside
the cloud: distribution trees. And the way it fails is subtle to
diagnose, because people who try it simply stop using it, so it always
looks like it's an adoption problem. Humans hate science, so they hate
scientific evidence that tells them that XMPP could never work the way
it got its shoelaces tied back in 1999. Humans love to think that the
problem is something else. Adoption and extension protocols for example.

> - for an up to date xmpp server count you can see here
> https://xmppnetwork.goodbytes.im/webgl.html :)

Of course people can use it anyway, but it just will never be able to
work for the entire human society the way it is designed.

> To keep this short your document there seems to be more than 15(!) years
> ago. So please link up to date versions in the future because XMPP its still
> being updated. Also lot of the links are dead too.

The core problem never got fixed.