Re: [Mingw-cross-env-list] On creating packages, source files and checksums.

[Gregorio Litenstein]

Where I currently am with this:

Going the dummy-package route. Except I still had to hack the checksum logic because there's no way to break the cyclical dependency: Rust needs Rust 1.36 to build, which in turn needs 1.35 and so forth. That means having one checksum for each dummy package for each OS and processor MXE might run on. 

So, I'm downloading and unpacking those to a temp folder and then using them to build the stable for $(BUILD). I'm planning to then just download the std library for $(TARGET) because it apparently cannot be easily built with the stable compiler, and including it in the initial bootstrap means several extra hours of compiling and about 8gb in intermediate artifacts which is just utterly insane IMO. 

I'm having trouble with the std library downloading thing as per another thread I started though (I also posted it as an issue on the repo)

Best regards.

Gregorio.

On Sep 9, 2019, 10:04 -0300, Tony Theodore <address@hidden>, wrote:

Apologies for the belated reply… and extra quoting for context

On 11 Aug 2019, at 19:42, Volker Diels-Grabsch <address@hidden> wrote:

Dear Gregorio,

If I haven't missed a new feature in the MXE core, there's currently
no way to have multiple source file per package.

Over the years, I’ve come to understand why package managers end up
with `*-dev` packages - you typically want *less* output in each
package, not more.

We now use source files across multiple packages[1]. I can’t see that
trend reversing.

I believe in the past we solved similar issues as follows: Provide one
dummy-package per source file, so they are downloaded and checked. The
last ("main") package then depends on all others and performs the
actual build(s).

Alas, I wasn't able to find a concrete example of that approach at a
first glance. But maybe you do. However, it is also possible that
all packages which previously used the above approach were replaced by
separate builds which were percieved to be "cleaner".

If you don't want to go that route, I'd sugest to go your first
approach: I don't see a huge problem in keeping bootstrap compilers
around, as long as their binariy names are properly and consistently
prefixed.

Have a look at protobuf[2]. It has a few salient features:
- depends on source-only packages (google[mock/test])
- unpacks those into its build tree
- bootstraps native `protoc` installed in `$(PREFIX)/$(BUILD)/bin/`
- uses that native `protoc` for cross-build

The bootstrap compiler is only visible to mxe in `$(PREFIX)/$(BUILD)/bin/`
and needs to be kept for subsequent cross-builds.

VTK[3] is a more extreme example that actually needs artefacts from its
build tree to persist into the cross-build. We have to build in-situ
and keep `$(PREFIX)/$(BUILD)/vtkCompileTools` around.

Gregorio Litenstein schrieb:
[…]

The reason I'm asking is because I'm trying to port rust; the problem is
rust requires a bootstrap compiler that cannot be built from scratch, we
_need_ to either download it or provide it somehow.

For mxe, we would prefer to build the bootstrap compiler. You can have multiple
levels of bootstrapping with the current functionality, but I’m not familiar
with `rust` to provide any tips.

One possible solution
of course would be to have a package for each bootstrap component (there's
3; rustc, rust-std and cargo). That would probably work but would mean
keeping bootstrap compilers around on the system after building the real
compiler so it's far from ideal.

That’s not a problem (see above).

Another solution maybe would be to make a single package containing both
the source and the bootstrap binaries and host it somewhere; this is also
not ideal because there would have to be one for each different OS
supported by MXE.

An alternate would be to add extra requirements, but that has many issues
of its own.

The ideal solution of course, would be to be able to have more than one
source file for a package. Then I'd be able to pull all required
components, check their checksum and get rid of them once they're no longer
needed.

Is that last option possible?

And, in case it's not; is it possible to get around the no-download thing
(I know this is very discouraged but if there's no other solution...) I
promise to be good and verify checksums for anything I download!

You would have to duplicate all the checksum/update/parallel download logic
etc, that’s a lot of work and hard to maintain.

Effort aside, the underlying logic of the “no network” is to make isolated
and reproducible builds possible. Using “package” as the unit of organisation
facilitates this since a package has:
- zero or one file download
- zero or one build rule
- zero or more dependencies
- zero or more dependents
- target-specific variants of all the above

Allowing downloads (potentially) breaks every one of those. Verifying the
checksum isn’t part of the logical guarantee, it’s an execution detail.

Cheers,

Tony



[1] https://github.com/mxe/mxe/commit/85de732ad49a3b4827d02b8c46c98a59f41ef6c6
[2] https://github.com/mxe/mxe/blob/master/src/protobuf.mk
[3] https://github.com/mxe/mxe/blob/master/src/vtk.mk