Package `extra/john` illegal hardware instruction with -O3

rzy commented

2024-06-15 07:58:21 +02:00

When compiling with -O3, the generated code contained XOP instructions, which didn't belong to x86-64-v3 micro-architecture level. However it was compiled and ran without any problem with -O2

==== backtrace (tid: 204456) ====
 0 0x0000000000051fe6 ucs_memtrack_releasing()  ???:0
 1 0x00000000000523ca ucs_memtrack_releasing()  ???:0
 2 0x000000000003cae0 __sigaction()  ???:0
 3 0x00000000001c5348 ???()  /usr/bin/john:0
 4 0x00000000001c5b07 ???()  /usr/bin/john:0
 5 0x0000000000318302 ???()  /usr/bin/john:0
 6 0x0000000000319260 ???()  /usr/bin/john:0
 7 0x0000000000319fef ???()  /usr/bin/john:0
 8 0x0000000000312ea3 ???()  /usr/bin/john:0
 9 0x0000000000025c88 __libc_init_first()  ???:0
10 0x0000000000025d4c __libc_start_main()  ???:0
11 0x0000000000009185 ???()  /usr/bin/john:0
=================================
[1]    204456 illegal hardware instruction (core dumped)  john lee

When compiling with -O3, the generated code contained XOP instructions, which didn't belong to x86-64-v3 micro-architecture level. However it was compiled and ran without any problem with -O2 ``` ==== backtrace (tid: 204456) ==== 0 0x0000000000051fe6 ucs_memtrack_releasing() ???:0 1 0x00000000000523ca ucs_memtrack_releasing() ???:0 2 0x000000000003cae0 __sigaction() ???:0 3 0x00000000001c5348 ???() /usr/bin/john:0 4 0x00000000001c5b07 ???() /usr/bin/john:0 5 0x0000000000318302 ???() /usr/bin/john:0 6 0x0000000000319260 ???() /usr/bin/john:0 7 0x0000000000319fef ???() /usr/bin/john:0 8 0x0000000000312ea3 ???() /usr/bin/john:0 9 0x0000000000025c88 __libc_init_first() ???:0 10 0x0000000000025d4c __libc_start_main() ???:0 11 0x0000000000009185 ???() /usr/bin/john:0 ================================= [1] 204456 illegal hardware instruction (core dumped) john lee ``` <img width="644" alt="image" src="/attachments/04af0d5e-18d4-4ffb-b2b6-e20959877764">

image.png

220 KiB

anonfunc commented

2024-06-15 10:07:15 +02:00

Owner

If I interpret the PKGBUILD for john correctly, that is somewhat intended.

Did you have a look in /usr/lib/john/? There should be a john binary in there without XOP, named john-non-xop.

If I interpret the PKGBUILD for `john` correctly, [that is somewhat intended](https://gitlab.archlinux.org/archlinux/packaging/packages/john/-/blob/main/PKGBUILD?ref_type=heads#L72). Did you have a look in `/usr/lib/john/`? There should be a john binary in there without XOP, named `john-non-xop`.

anonfunc added the

support

label 2024-06-15 10:08:19 +02:00

rzy commented

2024-06-15 11:17:57 +02:00

Author

You are right. The the reason why john with xop was the default binary is they want the program to fall back from the most feature to less feature instruction sets. However, in the default build script in the john repo, they pass the simd instruction to the configure script, not the compiler (CFLAGS). I guess this caused the problem, but I need to look into this further.

Btw as you can see, the default Archlinux's PKGBUILD doesn't include optimizations for avx2 and avx512f, so compiling it with march doesn't make any big differences. We need to change the PKGBUILD to include optimization for v3 and v4, and possibly delete cpu fallback altogether since we only target 1 micro architecture per build to reduce the size of the package as well. I can try to take a look and help but in worst case scenario, we can blacklist this package because of the above reason.

You are right. The the reason why john with xop was the default binary is they want the program to fall back from [the most feature to less feature instruction sets](https://github.com/openwall/john/blob/6be5461b67951290f707021b93b34ea4168fa9e2/src/packaging/build.sh#L54). However, in the default build script in the john repo, they pass the simd instruction to the configure script, not the compiler (CFLAGS). I guess this caused the problem, but I need to look into this further. Btw as you can see, the default Archlinux's PKGBUILD doesn't include optimizations for avx2 and avx512f, so compiling it with march doesn't make any big differences. We need to change the PKGBUILD to include optimization for v3 and v4, and possibly delete cpu fallback altogether since we only target 1 micro architecture per build to reduce the size of the package as well. I can try to take a look and help but in worst case scenario, we can blacklist this package because of the above reason.

anonfunc commented

2024-06-15 14:48:54 +02:00

Owner

The the reason why john with xop was the default binary is they want the program to fall back from the most feature to less feature instruction sets.

Obviously that breaks with ALHP. (besides that, the build.sh is not used in the Arch build)

It doesn't need to include these flags, they are enabled by march (and further optimization based on these flags by -O3. That's why you only had the SIGILL on -O3). What Arch is doing in their PKGBUILD is only because there is no v2/v3/v4 on official Arch, so they had to rely on that logic. If you check the git history of that file, that part was actually written 5-7 years ago.

I agree that the default binary linked to usr/bin crashing is suboptimal, but we could add a FAQ entry for john.

> The the reason why john with xop was the default binary is they want the program to fall back from the most feature to less feature instruction sets. Obviously that breaks with ALHP. (besides that, the `build.sh` is not used in the Arch build) It doesn't need to include these flags, they are enabled by `march` (and further optimization based on these flags by `-O3`. That's why you only had the SIGILL on `-O3`). What Arch is doing in their PKGBUILD is only because there is no v2/v3/v4 on official Arch, so they had to rely on that logic. If you check the git history of that file, that part was actually written 5-7 years ago. I agree that the default binary linked to `usr/bin` crashing is suboptimal, but we could add a FAQ entry for `john`.

👍 1

rzy commented

2024-06-15 17:40:01 +02:00

Author

Sorry my bad. You are completely right. I tried to compile the git repo and couldn't reproduce the problem. The release was from 2019 so they might have fixed it by now. Thank you for your support.

👍 1 ❤️ 1

anonfunc commented

2024-06-22 20:43:39 +02:00

Owner

Closing this as solved.

anonfunc closed this issue

2024-06-22 20:43:39 +02:00

Package extra/john illegal hardware instruction with -O3 #249

Package `extra/john` illegal hardware instruction with -O3 #249