[Mingw-cross-env-list] Adding checksums to filenames in cloud backup of packages

[Nagaev Boris]

Hey,

There are two problems with the backup of packages.

1. It has not been updated since September 2016
http://lists.nongnu.org/archive/html/mingw-cross-env-list/2017-01/msg00002.html

2. If a package changes without changing its filename, it causes
download errors for people using past versions of MXE.
https://github.com/mxe/mxe/issues/1309

I propose to separate fetching new packages from hosting them. One
process will collect new versions of packages and add them to a Git
repo. Another process will get files from that Git repo and push them
to a cloud storage.

I propose new naming scheme for files in the backup directory to avoid
name collisions:

$($(PKG)_FILE)_$($(PKG)_CHECKSUM)

I created two scripts in https://github.com/mxe/mxe/pull/1627
 * backup_from_s3.py downloads files from
https://s3.amazonaws.com/mxe-pkg/ and puts them under new names to a
directory.
 * update_backup.py adds new files from mxe/pkg to a directory.

I tried to add the directory with packages to Git and it worked
locally, but I failed to push files > 100M to GitHub. We have 7 such
files:

109M  qtwebengine-opensource-src-5.4.2.tar.xz
118M  qtwebengine-opensource-src-5.5.0.tar.xz
118M  qtwebengine-opensource-src-5.5.1.tar.xz
137M  qtwebengine-opensource-src-5.6.0.tar.xz
159M  qtwebengine-opensource-src-5.7.0.tar.xz
183M  InsightToolkit-4.4.1.tar.xz
230M  qt-everywhere-opensource-src-4.8.7.tar.gz

I tried using Git LFS and hit GitHub limits for free account:
> Using 0.0 of 1 GB/month of bandwidth
> Using 5.23 of 1 GB of storage

I could workaround the storage limitation by storing only those 7
files in LFS and other as normal files. But I am not happy with the 1
GB/month of bandwidth limitation: just one fork per month.

I uploaded the same repo to GitLab:
https://gitlab.com/starius/mxe-backup - with LFS
https://gitlab.com/starius/mxe-backup2 - without LFS
In both cases it displays limit 5.2 / 10 Gb.

I think, we can go with the variant without LFS. What do you think
about it? We can add this repo itself as a final fallback.

Can we modify S3 backup of packages to fetch new files from this repo?
Files that are already there, should not be removed to provide
backward compatibility.

I think, we can also add Google Cloud Storage as a backup (currently
both backups seems to be in Amazon, so Amazon's infrastructure is a
single failure domain).

-- 
Best regards,
Boris Nagaev