0

I’m working on creating a scratch image as part of a test network for a system I’m building. My goal is to set up a multi-stage Docker build that allows me to compile and run a Rust application both locally (on my Mac M4) and in a GitLab pipeline using a self-hosted runner. My local deployment uses docker compose and the the final deployment is to EKS, therefore containerization is a must.

This means I need to cross-compile for different environments — aarch64/arm64 for my Mac and amd64/x86_64 for cloud-based Linux environments. However, after days of troubleshooting, I’m still stuck.

Here are some of the challenges I’ve encountered:

  1. Linker issues— Could I be using the wrong one? How do I pick the right linker?
  2. Static vs. dynamic libraries - I did a very generic hello world compilation using alpine to build and scratch to run and it worked, therefore i thought it could be my dependencies, but with no compilation problems I do not know how to see which ones?
  3. Alpine-related build problems—I’m still trying to fully grasp the differences between musl and glibc and how they impact my build. For example https://github.com/rust-lang/rust/issues/115430.

I’m by no means an expert in Rust or cross-compilation, so I really appreciate any guidance. Please explain it like I’m five — I’m happy to learn, and I’d love recommendations on good resources to better understand this topic.

Thanks in advance!

My approach to this task has been as follows:

  1. Use Debian bookworm to create an image for comparison in size - hello world, this worked.
ARG PLATFORM FROM --platform=${PLATFORM} rust:1.84.0-slim-bookworm AS build RUN apt update && apt install -y libssl-dev WORKDIR /app COPY ./hello_world_compile_test . RUN cargo build --release FROM --platform=${PLATFORM} rust:1.84.0-slim-bookworm AS prod COPY --from=build /app/target/release/hello_world_compile_test /opt/hello_world_compile_test/ CMD ["/opt/hello_world_compile_test/hello_world_compile_test"]
  1. Then use alpine and scratch - hello world again no dependencies - this worked.
 FROM --platform=${PLATFORM} rust:1.84.0-alpine AS build ARG TARGET_APP RUN apk update && apk add openssl-dev WORKDIR /app COPY ./${TARGET_APP} . RUN rustup target add x86_64-unknown-linux-musl RUN cargo build --target x86_64-unknown-linux-musl --release RUN mv ./target/x86_64-unknown-linux-musl/release/${TARGET_APP} /app/rust_application FROM --platform=${PLATFORM} scratch AS prod COPY --from=build /app/rust_application /opt/rust_application CMD ["/opt/rust_application"]

  1. Try to build and run my application using alpine and then

  2. Add a debug step which was an alpine image, which worked when I installed glibc in the container however would provide errors when not installed.

ARG PLATFORM=linux/aarch64 ARG TARGET_APP=custom_rust_app # Building on alpine causes a lot of problems, there are open GH issues # Why would you use ALpine, Speed and Size FROM rust:1.84.0-alpine AS build ARG TARGET_APP=custom_rust_app RUN apk update && apk add openssl-dev musl-dev build-base curl zip protobuf protobuf-dev grpc grpc-plugins WORKDIR /app COPY ./${TARGET_APP} . RUN rustup target add aarch64-unknown-linux-musl RUN --mount=type=ssh RUSTFLAGS="-Ctarget-feature=-crt-static" cargo build --target aarch64-unknown-linux-musl --release RUN mv /app/target/aarch64-unknown-linux-musl/release/${TARGET_APP} /app/rust_application FROM scratch AS prod COPY --from=build /app/rust_application /opt/rust_application CMD ["/opt/rust_application"] FROM alpine AS debug RUN apk update && apk add openssl libgcc libstdc++ COPY --from=build /app/rust_application /opt/rust_application CMD ["/bin/sh"]

Without RUN apk update && apk add openssl libgcc libstdc++ in the debug step:

docker run -it test-scratch-mac / # /opt/rust_application Error loading shared library libgcc_s.so.1: No such file or directory (needed by /opt/rust_application) Error relocating /opt/rust_application: _Unwind_SetIP: symbol not found Error relocating /opt/rust_application: _Unwind_GetRegionStart: symbol not found Error relocating /opt/rust_application: _Unwind_GetTextRelBase: symbol not found Error relocating /opt/rust_application: _Unwind_FindEnclosingFunction: symbol not found Error relocating /opt/rust_application: _Unwind_Resume: symbol not found Error relocating /opt/rust_application: _Unwind_DeleteException: symbol not found Error relocating /opt/rust_application: _Unwind_RaiseException: symbol not found Error relocating /opt/rust_application: _Unwind_GetIP: symbol not found Error relocating /opt/rust_application: _Unwind_GetIPInfo: symbol not found Error relocating /opt/rust_application: _Unwind_GetDataRelBase: symbol not found Error relocating /opt/rust_application: _Unwind_SetGR: symbol not found Error relocating /opt/rust_application: _Unwind_GetCFA: symbol not found Error relocating /opt/rust_application: _Unwind_Backtrace: symbol not found Error relocating /opt/rust_application: _Unwind_GetLanguageSpecificData: symbol not found / #
  1. I then tried to go and build on book worm again and transfer to scratch, where I had the same problem of exec /opt/rust_application: no such file or directory
FROM rust:1.84.0-slim-bookworm AS build ARG TARGET_APP=custom_rust_app ARG PLAT_RUST=aarch64 # ARG PLAT_RUST=x86_64 RUN apt update && apt install -y libssl-dev build-essential \ protobuf-compiler \ libssl-dev \ libudev-dev \ pkg-config \ zlib1g-dev \ llvm \ clang \ cmake \ make \ libprotobuf-dev WORKDIR /app COPY ./${TARGET_APP} . RUN rustup toolchain install stable-${PLAT_RUST}-unknown-linux-gnu RUN rustup target add ${PLAT_RUST}-unknown-linux-gnu RUN --mount=type=ssh RUSTFLAGS="-Ctarget-feature=-crt-static" cargo build --target ${PLAT_RUST}-unknown-linux-gnu --release RUN mv /app/target/${PLAT_RUST}-unknown-linux-gnu/release/${TARGET_APP} /app/rust_application FROM scratch AS prod COPY --from=build /app/rust_application /opt/rust_application CMD ["/opt/rust_application"]

I know there are different approaches to the docker file, however i am just trying to get it working atm not concerned about the perfect dockerfile!

An example command can be found here

docker buildx build --platform linux/aarch64 --ssh default -t test-scratch-mac -f ./Alpine_Lightweight_Rust_Dockerfile --target debug .

Where i will (for now) change the target platform as per the ARGS in the Dockefile.

Improve this question
4
  • The easy answer is to copy not just your executable but also libgcc_s.so.1 into the runtime environment. Put its location into the LD_PRELOAD environment variable. Not the best fix but a very straightforward one
    – Charles Duffy
    CommentedFeb 5 at 15:17
  • @CharlesDuffy thank you for the response, you sent me down the right track! Turns out it was the way I was compiling the code and also the libraries being used, particularly relying on the OS distributions OpenSSL implementation. This created dynamic dependencies to glibc and therefore required me to copy them over and rendering my static compilation pretty useless! So I know the fix and can look at ways round it by using a compiled vendored version of openssl for full static compilation with musl. (Word vomit there but just incase anyone comes across this in the future).
    – Joel Earps
    CommentedFeb 10 at 9:10
  • If you wanted to write a full answer fleshing out the info from the comment above, that would be helpful to future readers -- in part because having an upvoted answer is a factor that helps to protect a question from autodeletion.
    – Charles Duffy
    CommentedFeb 10 at 14:22
  • Perfect I was planning on just haven't got round to it yet haha! Will make sure it gets done over the next few days.
    – Joel Earps
    CommentedFeb 10 at 17:07

1 Answer 1

Reset to default
3

This may not be the solution for everyone but the answer comes down to how cross compilation and linking works. What I am attempting to do here is create a static binary, there are two commands that can be used:

RUSTFLAGS="-Ctarget-feature=-crt-static" - dynamic linking of the c runtime.

RUSTFLAGS="-Ctarget-feature=+crt-static" - fully static compilation of C runtime.

There is also a separate issue here, the use of openssl-dev - this requires glibc to compile by default, you need to set this up to be compiled statically. I haven't got to the bottom of this yet but the three solutions I have found (2 work, 1 I haven't tried yet).

  1. Copy all certs and openssl object files to the scratch container - works.
  2. Compile and use static file in openssl OS distro - not tried.
  3. Use Cargo vendored openssl implementation

I am planning on writing a medium article where I added all my problems and findings, just think it might help others when I post this.

Improve this answer

    Start asking to get answers

    Find the answer to your question by asking.

    Ask question

    Explore related questions

    See similar questions with these tags.