[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PULL v2 05/29] docker: Add Hexagon image
From: |
Alex Bennée |
Subject: |
[PULL v2 05/29] docker: Add Hexagon image |
Date: |
Tue, 18 May 2021 10:06:56 +0100 |
From: Alessandro Di Federico <ale@rev.ng>
This image is a little special because it takes a long time to build.
As such most users don't want to be doing that and just pull random
binaries from the ether as intended by the container gods. This
involves someone with credentials and a beefy machine running:
make docker-image-debian-hexagon-cross V=1 NOCACHE=1 J=30
docker tag qemu/debian-hexagon-cross
registry.gitlab.com/qemu-project/qemu/qemu/debian-hexagon-cross
docker push registry.gitlab.com/qemu-project/qemu/qemu/debian-hexagon-cross
With a suitable binary in the "cloud" a normal user will run:
make docker-image-debian-hexagon-cross
or have it run for them through the dependency mechanism of our
over-engineered makefiles and get the binary they wanted. There are a
few wrinkles of course including needing to tweak the final image to
have the credentials of the user so we can actually do our cross
compiles.
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Alessandro Di Federico <ale@rev.ng>
Cc: Philippe Mathieu-Daudé <f4bug@amsat.org>
Cc: Brian Cain <bcain@quicinc.com>
Message-Id: <20210512102051.12134-7-alex.bennee@linaro.org>
diff --git a/MAINTAINERS b/MAINTAINERS
index 78561a223f..7572859317 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -197,6 +197,8 @@ F: linux-user/hexagon/
F: tests/tcg/hexagon/
F: disas/hexagon.c
F: default-configs/targets/hexagon-linux-user.mak
+F: docker/dockerfiles/debian-hexagon-cross.docker
+F: docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
HPPA (PA-RISC) TCG CPUs
M: Richard Henderson <richard.henderson@linaro.org>
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 820423d718..8967ecf118 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -148,6 +148,28 @@ docker-image-debian-s390x-cross: docker-image-debian10
docker-image-debian-sh4-cross: docker-image-debian10
docker-image-debian-sparc64-cross: docker-image-debian10
+#
+# The build rule for hexagon-cross is special in so far for most of
+# the time we don't want to build it. While dockers caching does avoid
+# this most of the time sometimes we want to force the issue.
+#
+docker-image-debian-hexagon-cross:
$(DOCKER_FILES_DIR)/debian-hexagon-cross.docker
+ $(if $(NOCACHE),
\
+ $(call quiet-command,
\
+ $(DOCKER_SCRIPT) build -t qemu/debian-hexagon-cross -f
$< \
+ $(if $V,,--quiet) --no-cache
\
+ --registry $(DOCKER_REGISTRY) --extra-files
\
+
$(DOCKER_FILES_DIR)/debian-hexagon-cross.docker.d/build-toolchain.sh, \
+ "BUILD", "debian-hexagon-cross"),
\
+ $(call quiet-command,
\
+ $(DOCKER_SCRIPT) fetch $(if $V,,--quiet)
\
+ qemu/debian-hexagon-cross $(DOCKER_REGISTRY),
\
+ "FETCH", "debian-hexagon-cross")
\
+ $(call quiet-command,
\
+ $(DOCKER_SCRIPT) update $(if $V,,--quiet)
\
+ qemu/debian-hexagon-cross --add-current-user,
\
+ "PREPARE", "debian-hexagon-cross"))
+
# Specialist build images, sometimes very limited tools
docker-image-debian-tricore-cross: docker-image-debian10
docker-image-debian-all-test-cross: docker-image-debian10
diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker
b/tests/docker/dockerfiles/debian-hexagon-cross.docker
new file mode 100644
index 0000000000..d5dc299dc1
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker
@@ -0,0 +1,45 @@
+#
+# Docker Hexagon cross-compiler target
+#
+# This docker target is used for building hexagon tests. As it also
+# needs to be able to build QEMU itself in CI we include it's
+# build-deps. It is also a "stand-alone" image so as not to be
+# triggered by re-builds on other base images given it takes a long
+# time to build.
+#
+FROM qemu/debian10
+
+# Install common build utilities
+RUN apt update && \
+ DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
+ DEBIAN_FRONTEND=noninteractive eatmydata \
+ apt install -y --no-install-recommends \
+ bison \
+ cmake \
+ flex \
+ lld \
+ rsync \
+ wget
+
+ENV TOOLCHAIN_INSTALL /usr/local
+ENV ROOTFS /usr/local
+
+ENV LLVM_URL
https://github.com/llvm/llvm-project/archive/bfcd21876adc3498065e4da92799f613e730d475.tar.gz
+ENV MUSL_URL
https://github.com/quic/musl/archive/aff74b395fbf59cd7e93b3691905aa1af6c0778c.tar.gz
+ENV LINUX_URL https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.6.18.tar.xz
+
+ADD build-toolchain.sh /root/hexagon-toolchain/build-toolchain.sh
+
+RUN cd /root/hexagon-toolchain && ./build-toolchain.sh
+
+FROM debian:buster-slim
+# Duplicate deb line as deb-src
+RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >>
/etc/apt/sources.list
+# Install QEMU build deps for use in CI
+RUN apt update && \
+ DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
+ DEBIAN_FRONTEND=noninteractive eatmydata apt install -yy git ninja-build
&& \
+ DEBIAN_FRONTEND=noninteractive eatmydata \
+ apt build-dep -yy --arch-only qemu
+COPY --from=0 /usr/local /usr/local
+ENV PATH $PATH:/usr/local/bin/
diff --git
a/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
b/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
new file mode 100755
index 0000000000..19b1c9f83e
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker.d/build-toolchain.sh
@@ -0,0 +1,141 @@
+#!/bin/bash
+
+set -e
+
+BASE=$(readlink -f ${PWD})
+
+TOOLCHAIN_INSTALL=$(readlink -f "$TOOLCHAIN_INSTALL")
+ROOTFS=$(readlink -f "$ROOTFS")
+
+TOOLCHAIN_BIN=${TOOLCHAIN_INSTALL}/bin
+HEX_SYSROOT=${TOOLCHAIN_INSTALL}/hexagon-unknown-linux-musl
+HEX_TOOLS_TARGET_BASE=${HEX_SYSROOT}/usr
+
+function cdp() {
+ DIR="$1"
+ mkdir -p "$DIR"
+ cd "$DIR"
+}
+
+function fetch() {
+ DIR="$1"
+ URL="$2"
+ TEMP="$(readlink -f "$PWD/tmp.tar.gz")"
+ wget --quiet "$URL" -O "$TEMP"
+ cdp "$DIR"
+ tar xaf "$TEMP" --strip-components=1
+ rm "$TEMP"
+ cd -
+}
+
+build_llvm_clang() {
+ fetch "$BASE/llvm-project" "$LLVM_URL"
+ cdp "$BASE/build-llvm"
+
+ cmake -G Ninja \
+ -DCMAKE_BUILD_TYPE=Release \
+ -DCMAKE_INSTALL_PREFIX=${TOOLCHAIN_INSTALL} \
+ -DLLVM_ENABLE_LLD=ON \
+ -DLLVM_TARGETS_TO_BUILD="Hexagon" \
+ -DLLVM_ENABLE_PROJECTS="clang;lld" \
+ "$BASE/llvm-project/llvm"
+ ninja all install
+ cd ${TOOLCHAIN_BIN}
+ ln -sf clang hexagon-unknown-linux-musl-clang
+ ln -sf clang++ hexagon-unknown-linux-musl-clang++
+ ln -sf llvm-ar hexagon-unknown-linux-musl-ar
+ ln -sf llvm-objdump hexagon-unknown-linux-musl-objdump
+ ln -sf llvm-objcopy hexagon-unknown-linux-musl-objcopy
+ ln -sf llvm-readelf hexagon-unknown-linux-musl-readelf
+ ln -sf llvm-ranlib hexagon-unknown-linux-musl-ranlib
+
+ # workaround for now:
+ cat <<EOF > hexagon-unknown-linux-musl.cfg
+-G0 --sysroot=${HEX_SYSROOT}
+EOF
+}
+
+build_clang_rt() {
+ cdp "$BASE/build-clang_rt"
+ cmake -G Ninja \
+ -DCMAKE_BUILD_TYPE=Release \
+ -DLLVM_CONFIG_PATH="$BASE/build-llvm/bin/llvm-config" \
+ -DCMAKE_ASM_FLAGS="-G0 -mlong-calls -fno-pic
--target=hexagon-unknown-linux-musl " \
+ -DCMAKE_SYSTEM_NAME=Linux \
+ -DCMAKE_C_COMPILER="${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang" \
+ -DCMAKE_ASM_COMPILER="${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang" \
+ -DCMAKE_INSTALL_PREFIX=${HEX_TOOLS_TARGET_BASE} \
+ -DCMAKE_CROSSCOMPILING=ON \
+ -DCMAKE_C_COMPILER_FORCED=ON \
+ -DCMAKE_CXX_COMPILER_FORCED=ON \
+ -DCOMPILER_RT_BUILD_BUILTINS=ON \
+ -DCOMPILER_RT_BUILTINS_ENABLE_PIC=OFF \
+ -DCMAKE_SIZEOF_VOID_P=4 \
+ -DCOMPILER_RT_OS_DIR= \
+ -DCAN_TARGET_hexagon=1 \
+ -DCAN_TARGET_x86_64=0 \
+ -DCOMPILER_RT_SUPPORTED_ARCH=hexagon \
+ -DLLVM_ENABLE_PROJECTS="compiler-rt" \
+ "$BASE/llvm-project/compiler-rt"
+ ninja install-compiler-rt
+}
+
+build_musl_headers() {
+ fetch "$BASE/musl" "$MUSL_URL"
+ cd "$BASE/musl"
+ make clean
+ CC=${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang \
+ CROSS_COMPILE=hexagon-unknown-linux-musl \
+ LIBCC=${HEX_TOOLS_TARGET_BASE}/lib/libclang_rt.builtins-hexagon.a \
+ CROSS_CFLAGS="-G0 -O0 -mv65 -fno-builtin -fno-rounding-math
--target=hexagon-unknown-linux-musl" \
+ ./configure --target=hexagon --prefix=${HEX_TOOLS_TARGET_BASE}
+ PATH=${TOOLCHAIN_BIN}:$PATH make CROSS_COMPILE= install-headers
+
+ cd ${HEX_SYSROOT}/..
+ ln -sf hexagon-unknown-linux-musl hexagon
+}
+
+build_kernel_headers() {
+ fetch "$BASE/linux" "$LINUX_URL"
+ mkdir -p "$BASE/build-linux"
+ cd "$BASE/linux"
+ make O=../build-linux ARCH=hexagon \
+ KBUILD_CFLAGS_KERNEL="-mlong-calls" \
+ CC=${TOOLCHAIN_BIN}/hexagon-unknown-linux-musl-clang \
+ LD=${TOOLCHAIN_BIN}/ld.lld \
+ KBUILD_VERBOSE=1 comet_defconfig
+ make mrproper
+
+ cd "$BASE/build-linux"
+ make \
+ ARCH=hexagon \
+ CC=${TOOLCHAIN_BIN}/clang \
+ INSTALL_HDR_PATH=${HEX_TOOLS_TARGET_BASE} \
+ V=1 \
+ headers_install
+}
+
+build_musl() {
+ cd "$BASE/musl"
+ make clean
+ CROSS_COMPILE=hexagon-unknown-linux-musl- \
+ AR=llvm-ar \
+ RANLIB=llvm-ranlib \
+ STRIP=llvm-strip \
+ CC=clang \
+ LIBCC=${HEX_TOOLS_TARGET_BASE}/lib/libclang_rt.builtins-hexagon.a \
+ CFLAGS="-G0 -O0 -mv65 -fno-builtin -fno-rounding-math
--target=hexagon-unknown-linux-musl" \
+ ./configure --target=hexagon --prefix=${HEX_TOOLS_TARGET_BASE}
+ PATH=${TOOLCHAIN_BIN}/:$PATH make CROSS_COMPILE= install
+ cd ${HEX_TOOLS_TARGET_BASE}/lib
+ ln -sf libc.so ld-musl-hexagon.so
+ ln -sf ld-musl-hexagon.so ld-musl-hexagon.so.1
+ cdp ${HEX_TOOLS_TARGET_BASE}/../lib
+ ln -sf ../usr/lib/ld-musl-hexagon.so.1
+}
+
+build_llvm_clang
+build_kernel_headers
+build_musl_headers
+build_clang_rt
+build_musl
--
2.20.1
- [PULL v2 00/29] testing and plugin updates, Alex Bennée, 2021/05/18
- [PULL v2 01/29] tests/docker: fix copying of executable in "update", Alex Bennée, 2021/05/18
- [PULL v2 02/29] tests/docker: make executable an optional argument to "update", Alex Bennée, 2021/05/18
- [PULL v2 03/29] tests/docker: allow "update" to add the current user, Alex Bennée, 2021/05/18
- [PULL v2 04/29] tests/docker: add "fetch" sub-command, Alex Bennée, 2021/05/18
- [PULL v2 06/29] tests/tcg: Use Hexagon Docker image, Alex Bennée, 2021/05/18
- [PULL v2 09/29] tests/docker: Added libbpf library to the docker files., Alex Bennée, 2021/05/18
- [PULL v2 13/29] tests/tcg/tricore: Add build infrastructure, Alex Bennée, 2021/05/18
- [PULL v2 10/29] tests/tcg: Add docker_as and docker_ld cmds, Alex Bennée, 2021/05/18
- [PULL v2 05/29] docker: Add Hexagon image,
Alex Bennée <=
- [PULL v2 07/29] gitlab: add build-user-hexagon test, Alex Bennée, 2021/05/18
- [PULL v2 08/29] tests/tcg: don't iterate through other arch compilers, Alex Bennée, 2021/05/18
- [PULL v2 12/29] hw/tricore: Add testdevice for tests in tests/tcg/, Alex Bennée, 2021/05/18
- [PULL v2 11/29] tests/tcg: Run timeout cmds using --foreground, Alex Bennée, 2021/05/18
- [PULL v2 16/29] tests/tcg/tricore: Add bmerge test, Alex Bennée, 2021/05/18
- [PULL v2 15/29] tests/tcg/tricore: Add macros to create tests and first test 'abs', Alex Bennée, 2021/05/18
- [PULL v2 19/29] tests/tcg/tricore: Add fadd test, Alex Bennée, 2021/05/18
- [PULL v2 26/29] tests/tcg/ppc64le: tests for brh/brw/brd, Alex Bennée, 2021/05/18
- [PULL v2 28/29] tests/tcg: don't allow clang as a cross compiler, Alex Bennée, 2021/05/18
- [PULL v2 25/29] tests/docker: gcc-10 based images for ppc64{, le} tests, Alex Bennée, 2021/05/18