I built the latest master I could find in git (5259fdf96073febd18e83785960c443bdf02a31) and the bug seems fixed there!
From bisecting, I know that the fix is in between 191a5568db (working) and 2a3bd6798e96708 (not working). It is hard to pinpoint further due to segfaults that were introduced then fixed in this range. My best guess is that "Fix unlikely races with GnuTLS, datagrams" (e87e6a24c4) resolved this.