Rust & LTO: I think I figured out what's wrong

saltedcoffii commented

2022-11-13 19:45:11 +01:00

Background

I use the default v3 makepkg file on my Arch Linux computer for the AUR, in addition to using the ALHP repositories. I noticed that any package containing rust code failed, so in the meantime, I just disabled LTO by removing -Clto=fat from the RUSTFLAGS line. I had some extra time on my hands this weekend, so I decided to debug this.
I have an x86-64-v3 CPU on my computer.

Results

Instead of using:
RUSTFLAGS="-Copt-level=3 -Ctarget-cpu=x86-64-v<?> -Clto=fat -Ccodegen-units=1 -Clinker-plugin-lto"

Use:
RUSTFLAGS="-Copt-level=3 -Ctarget-cpu=x86_64-v<?> -Clto=fat -Ccodegen-units=1 -Clinker-plugin-lto=no -Cembed-bitcode=yes"

Process

I used exa from the community repository because it's written entirely in rust, invokes cargo (i.e., compiles like a normal rust package) and is officially supported by Arch Linux, so the chances of running into a package-specific issue were much lower. This way, one may verify the configuration.

Step 1

Using the default LTO v3 makepkg file directly to build exa, one gets this error:

error: options `-C embed-bitcode=no` and `-C lto` are incompatible

The rust documentation notes here that such an error would occur.

Solution: Add -C embed-bitcode=yes to RUSTFLAGS (-C embed-bitcode=no is the default without this).

Step 2

Rebuilding, one gets a lot of errors like this:

  = note: "cc" "-m64" "/tmp/rustcHcqgNU/symbols.o" "~/exa/src/exa-0.10.1/target/release/build/bitflags-fdcf02aecd87e51c/build_script_build-fdcf02aecd87e51c.build_script_build.1471ce45-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "~/exa/src/exa-0.10.1/target/release/deps" "-L" "/usr/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-1fd459d18d250a66.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-Wl,-plugin-opt=O3,-plugin-opt=mcpu=x86-64-v3" "-L" "/usr/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "~/exa/src/exa-0.10.1/target/release/build/bitflags-fdcf02aecd87e51c/build_script_build-fdcf02aecd87e51c" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-Wl,-O1" "-nodefaultlibs"
  = note: ~/exa/src/exa-0.10.1/target/release/build/bitflags-fdcf02aecd87e51c/build_script_build-fdcf02aecd87e51c.build_script_build.1471ce45-cgu.0.rcgu.o: file not recognized: file format not recognized
          collect2: error: ld returned 1 exit status
          

error: could not compile `bitflags` due to previous error

This took me a bit to figure out, but the gist is: the .o files being made by rust are LLVM bitcode files, which the gcc linker (by default bfd) doesn't know how to handle. Changing the linker to lld or clang didn't seem to work either, however clang and lld are known to compile more quickly but produce slower binaries, so I'd like to stick to gcc and bfd, gold, or mold as much as possible. Arch Linux and ALHP use bfd by default, so I see no reason to change that here.

Solution: To make rustc generate normal object files instead of LLVM bitcode, replace Clinker-plugin-lto with Clinker-plugin-lto=no in RUSTFLAGS.

Yay! Now exa builds, and the binary works!

Step 3

I thought that maybe it would be a good idea to try building other packages in the AUR that use rust, just to test. I tested lottieconv, authenticator, and lolcate.

I got this error for all of them:

error: cannot prefer dynamic linking when performing LTO

note: only 'staticlib', 'bin', and 'cdylib' outputs are supported with LTO

error: could not compile `<whatever>` due to previous error
warning: build failed, waiting for other jobs to finish..

This is caused by failing to set --target while invoking cargo fetch or cargo b. As far as everything I have seen, all packages in the official Arch Linux repository specify --target` explicitly, so this shouldn't be an issue.

For anyone reading this, one can specify --target in cargo build in AUR packages (which may not be specific to one architecture) by harnessing the CHOST shell variable provided in the default makepkg.conf. Use cargo build --target ${CHOST//-pc-/-unknown-}.

I'd understand if this bug makes the ALHP devs hesitant to make this change, however it only affects AUR packages, not official ones.

EDIT: You can add --target ${CHOST//-pc-/-unknown-} directly to the RUSTFLAGS variable, but it doesn't seem to fix the bug mentioned above in Step 3.

EDIT 2: Adding CARGO_BUILD_TARGET="${CHOST//-pc-/-unknown-}" to /etc/makepkg.conf and then patching /usr/share/makepkg/buildenv.sh to export CARGO_BUILD_TARGET fixes the issue noted in Step 3.

### Background I use the default v3 [makepkg file](https://alhp.harting.dev/makepkg/makepkg-x86-64-v3.conf) on my Arch Linux computer for the AUR, in addition to using the ALHP repositories. I noticed that any package containing rust code failed, so in the meantime, I just disabled LTO by removing `-Clto=fat` from the `RUSTFLAGS` line. I had some extra time on my hands this weekend, so I decided to debug this. I have an x86-64-v3 CPU on my computer. ### Results Instead of using: `RUSTFLAGS="-Copt-level=3 -Ctarget-cpu=x86-64-v<?> -Clto=fat -Ccodegen-units=1 -Clinker-plugin-lto"` Use: `RUSTFLAGS="-Copt-level=3 -Ctarget-cpu=x86_64-v<?> -Clto=fat -Ccodegen-units=1 -Clinker-plugin-lto=no -Cembed-bitcode=yes"` ### Process I used [exa](https://archlinux.org/packages/community/x86_64/exa) from the community repository because it's written entirely in rust, invokes `cargo` (i.e., compiles like a normal rust package) and is officially supported by Arch Linux, so the chances of running into a package-specific issue were much lower. This way, one may verify the configuration. ##### Step 1 Using the default LTO v3 makepkg file directly to build `exa`, one gets this error: ``` error: options `-C embed-bitcode=no` and `-C lto` are incompatible ``` The rust documentation notes [here](https://doc.rust-lang.org/stable/rustc/codegen-options/index.html#embed-bitcode) that such an error would occur. Solution: Add `-C embed-bitcode=yes` to `RUSTFLAGS` (`-C embed-bitcode=no` is the default without this). ##### Step 2 Rebuilding, one gets a lot of errors like this: ``` = note: "cc" "-m64" "/tmp/rustcHcqgNU/symbols.o" "~/exa/src/exa-0.10.1/target/release/build/bitflags-fdcf02aecd87e51c/build_script_build-fdcf02aecd87e51c.build_script_build.1471ce45-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "~/exa/src/exa-0.10.1/target/release/deps" "-L" "/usr/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-1fd459d18d250a66.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-Wl,-plugin-opt=O3,-plugin-opt=mcpu=x86-64-v3" "-L" "/usr/lib64/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "~/exa/src/exa-0.10.1/target/release/build/bitflags-fdcf02aecd87e51c/build_script_build-fdcf02aecd87e51c" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-Wl,-O1" "-nodefaultlibs" = note: ~/exa/src/exa-0.10.1/target/release/build/bitflags-fdcf02aecd87e51c/build_script_build-fdcf02aecd87e51c.build_script_build.1471ce45-cgu.0.rcgu.o: file not recognized: file format not recognized collect2: error: ld returned 1 exit status error: could not compile `bitflags` due to previous error ``` This took me a bit to figure out, but the gist is: the `.o` files being made by rust are LLVM bitcode files, which the `gcc` linker (by default `bfd`) doesn't know how to handle. Changing the linker to `lld` or `clang` didn't seem to work either, however `clang` and `lld` are known to compile more quickly but produce slower binaries, so I'd like to stick to `gcc` and `bfd`, `gold`, or `mold` as much as possible. Arch Linux and ALHP use `bfd` by default, so I see no reason to change that here. Solution: To make `rustc` generate normal object files instead of LLVM bitcode, replace `Clinker-plugin-lto` with `Clinker-plugin-lto=no` in `RUSTFLAGS`. Yay! Now `exa` builds, and the binary works! ##### Step 3 I thought that maybe it would be a good idea to try building other packages in the AUR that use rust, just to test. I tested [lottieconv](https://aur.archlinux.org/packages/lottieconv), [authenticator](https://aur.archlinux.org/packages/authenticator), and [lolcate](https://aur.archlinux.org/packages/lolcate). I got this error for all of them: ``` error: cannot prefer dynamic linking when performing LTO note: only 'staticlib', 'bin', and 'cdylib' outputs are supported with LTO error: could not compile `<whatever>` due to previous error warning: build failed, waiting for other jobs to finish.. ``` This is caused by failing to set `--target` while invoking `cargo fetch` or `cargo b. As far as everything I have seen, all packages in the official Arch Linux repository specify `--target` explicitly, so this shouldn't be an issue. For anyone reading this, one can specify `--target` in `cargo build` in AUR packages (which may not be specific to one architecture) by harnessing the `CHOST` shell variable provided in the default `makepkg.conf`. Use `cargo build --target ${CHOST//-pc-/-unknown-}`. I'd understand if this bug makes the ALHP devs hesitant to make this change, however it only affects AUR packages, not official ones. EDIT: You can add `--target ${CHOST//-pc-/-unknown-}` directly to the `RUSTFLAGS` variable, but it doesn't seem to fix the bug mentioned above in Step 3. EDIT 2: Adding `CARGO_BUILD_TARGET="${CHOST//-pc-/-unknown-}"` to `/etc/makepkg.conf` and then patching `/usr/share/makepkg/buildenv.sh` to export `CARGO_BUILD_TARGET` fixes the issue noted in Step 3.

saltedcoffii commented

2022-11-14 00:25:33 +01:00

Author

Okay, the issue noted in step 3 seems to seriously be an issue with packages building with meson that invoke cargo in the back. This is particularly common among GNOME project stuff. Maybe this isn't such a good idea, but maybe someone has another solution.

AvianaCruz commented

2022-11-25 11:32:11 +01:00

@anonfunc According to https://users.rust-lang.org/t/status-of-lto-option/49032/8, we should use export CARGO_PROFILE_RELEASE_LTO=fat instead of manually adding -Clto=fat and -Clinker-plugin-lto, therefore cargo can decide whether to use lto. Force enabling lto will make packages that use compile-time code generation fail to be built.

@anonfunc According to https://users.rust-lang.org/t/status-of-lto-option/49032/8, we should use `export CARGO_PROFILE_RELEASE_LTO=fat` instead of manually adding `-Clto=fat` and `-Clinker-plugin-lto`, therefore `cargo` can decide whether to use lto. Force enabling lto will make packages that use compile-time code generation fail to be built.

anonfunc commented

2022-11-26 15:40:22 +01:00

Owner

@saltedcoffii Thanks for your detailed exploration. ALHP already has detection of those

error: options -C embed-bitcode=no and -C lto are incompatible

errors. So we waste only a little bit of buildtime here, and these packages are not build with lto, but that fortunately only affects a few packages. Exporting new envs is currently blocked by #149, so even if we wanted to set a new CARGO_BUILD_TARGET, its not possible until that issue is fixed. But I'm not sure that is worth it for the few packages this affects. Have you compared packages that are not affected by this bug with and without your new CARGO_BUILD_TARGET?

@AvianaCruz Do you have any documentation that explicitly states that CARGO_PROFILE_RELEASE_LTO lets cargo decide when to use lto? I could not find anything about that, neither on your linked page nor on https://doc.rust-lang.org/cargo/reference/profiles.html.

@saltedcoffii Thanks for your detailed exploration. ALHP already has detection of those > error: options `-C embed-bitcode=no` and `-C lto` are incompatible errors. So we waste only a little bit of buildtime here, and these packages are not build with lto, but that fortunately only affects a few packages. Exporting new envs is currently blocked by #149, so even if we wanted to set a new `CARGO_BUILD_TARGET`, its not possible until that issue is fixed. But I'm not sure that is worth it for the few packages this affects. Have you compared packages that are not affected by this bug with and without your new `CARGO_BUILD_TARGET`? @AvianaCruz Do you have any documentation that explicitly states that `CARGO_PROFILE_RELEASE_LTO` lets cargo decide when to use lto? I could not find anything about that, neither on your linked page nor on https://doc.rust-lang.org/cargo/reference/profiles.html.

anonfunc added the

informational

label 2022-11-26 15:53:04 +01:00

AvianaCruz commented

2022-11-27 14:48:28 +01:00

@anonfunc https://doc.rust-lang.org/cargo/reference/environment-variables.html#:~:text=CARGO_PROFILE_%3Cname%3E_LTO

anonfunc commented

2022-11-27 15:02:27 +01:00

Owner

This page also does not state it lets cargo choose which LTO behavior to adopt. It just links to here, where there is also nothing stated, and then to here, where we have the same settings that apply to -C lto. As far as I understand, the env is just another way to set lto, but not different to -C lto.

This page also does not state it lets cargo choose which LTO behavior to adopt. It just links to [here](https://doc.rust-lang.org/cargo/reference/config.html#profilenamelto), where there is also nothing stated, and then to [here](https://doc.rust-lang.org/cargo/reference/profiles.html#lto), where we have the same settings that apply to `-C lto`. As far as I understand, the env is just another way to set lto, but not different to `-C lto`.

AvianaCruz commented

2022-11-28 08:04:25 +01:00

@anonfunc This is the explanation: https://users.rust-lang.org/t/error-lto-can-only-be-run-for-executables-cdylibs-and-static-library-outputs/73369/4
Also the issue: https://github.com/rust-lang/cargo/issues/6375#issuecomment-560124129
(Cargo profile is a stable feature now so that there is no need to add -Zconfig-profile.)

@anonfunc This is the explanation: https://users.rust-lang.org/t/error-lto-can-only-be-run-for-executables-cdylibs-and-static-library-outputs/73369/4 Also the issue: https://github.com/rust-lang/cargo/issues/6375#issuecomment-560124129 (Cargo profile is a stable feature now so that there is no need to add `-Zconfig-profile`.)

AvianaCruz commented

2022-12-22 10:22:55 +01:00

@anonfunc cargo plays as a build tool which generates compiler flags. Rust has a feature that generates code at compile time, and the code which generates code doesn't exists in the final binary, so it can not be linked. Let cargo deciding lto avoids the problem that tries to link non-exist code to binary.

@anonfunc `cargo` plays as a build tool which generates compiler flags. Rust has a feature that generates code at compile time, and the code which generates code doesn't exists in the final binary, so it can not be linked. Let `cargo` deciding lto avoids the problem that tries to link non-exist code to binary.

anonfunc commented

2022-12-22 18:59:55 +01:00

Owner

I mean, it certainly won't hurt if it still enables LTO in cases where we have LTO enabled currently. Let's try it out as soon as a fix for #149 is implemented.

AvianaCruz commented

2023-02-12 07:46:37 +01:00

@anonfunc "rust" package should not be compiled with x86-64 level targeting flags, because it results in its shipped stdlibs also being compiled with higher x86-64 level.

Therefore every binaries compiled with the rust toolchain will be not compatible with CPU of lower x86-64 level because rust statically linking everything by default, including the stdlib which is shipped with the rust toolchain and not recompiled during the build stages.

@anonfunc "rust" package should not be compiled with x86-64 level targeting flags, because it results in its shipped stdlibs also being compiled with higher x86-64 level. Therefore every binaries compiled with the rust toolchain will be not compatible with CPU of lower x86-64 level because rust statically linking everything by default, including the stdlib which is shipped with the rust toolchain and not recompiled during the build stages.

anonfunc commented

2023-02-14 16:12:17 +01:00

Owner

@AvianaCruz That's unfortunate. ALHP does not currently feature a dedicated list for the x86-64-vN flag, so we need to backlist it completely then.

N3k0-san commented

2023-03-11 03:27:48 +01:00

@anonfunc ripgrep fails to build because of the error: options -C embed-bitcode=noand-C lto are incompatible too and has apparently been the case since December of last year; I thought you said it detects this...?

@anonfunc `ripgrep` fails to build because of the error: `options `-C embed-bitcode=no` and `-C lto` are incompatible` too and has apparently been the case since December of last year; I thought you said it detects this...?

anonfunc commented

2023-03-11 15:01:38 +01:00

Owner

It should detect these errors, yes. But I'm planing on switching to CARGO_BUILD_TARGET soon anyways, which should resolve this issue.

It should detect these errors, yes. But I'm planing on switching to `CARGO_BUILD_TARGET` soon anyways, which should resolve this issue.

anonfunc commented

2023-03-14 01:48:36 +01:00

Owner

Together with the new build-queue (ece8c4c7d9) CARGO_BUILD_TARGET is now also included. We'll see how it goes.

Together with the new build-queue (https://somegit.dev/ALHP/ALHP.GO/commit/ece8c4c7d91e1a9e167bbadc634e3b5bfccbaa37) `CARGO_BUILD_TARGET` is now also included. We'll see how it goes.

anonfunc commented

2023-03-21 22:03:33 +01:00

Owner

It seems to work fine. If there are any suggestions on how to improve it even further, please open a separate issue. Any feedback to CARGO_BUILD_TARGET can go in here.

It seems to work fine. If there are any suggestions on how to improve it even further, please open a separate issue. Any feedback to `CARGO_BUILD_TARGET` can go in here.

anonfunc closed this issue

2023-03-21 22:03:33 +01:00

Rust & LTO: I think I figured out what's wrong #157

Background

Results

Process

Step 1

Step 2

Step 3