Openssl 3 rebuild

blackcat-917 commented

2022-11-05 01:46:16 +01:00

So, I updated my system and lots of errors popped up.
Errors seem to come from systemd, bat, kitty (python has something to do with it) and flatpak but I've already seen that some other famous ones don't start up either, specifically electron-based ones.

So, I updated my system and lots of errors popped up. Errors seem to come from `systemd`, `bat`, `kitty` (`python` has something to do with it) and `flatpak` but I've already seen that some other famous ones don't start up either, specifically electron-based ones.

👍 1

blackcat-917 commented

2022-11-05 01:48:47 +01:00

Author

Ah yes, sudo and pacman broke as well.

Ah yes, `sudo` and `pacman` broke as well.

plexman commented

2022-11-05 12:20:05 +01:00

Yes, I also got some problems including a kernel panic because of openssl not updating correctly. In the meantime you can install openssl-1.1 to avoid these problems.

Source: https://unix.stackexchange.com/questions/723616/how-to-fix-missing-libcrypto-so-1-1

But yes, openssl version on ALHP is still 1.1.1 when in upstream is 3.0.7

Yes, I also got some problems including a kernel panic because of openssl not updating correctly. In the meantime you can install openssl-1.1 to avoid these problems. Source: [https://unix.stackexchange.com/questions/723616/how-to-fix-missing-libcrypto-so-1-1](https://unix.stackexchange.com/questions/723616/how-to-fix-missing-libcrypto-so-1-1) But yes, openssl version on ALHP is still 1.1.1 when in upstream is 3.0.7

anonfunc added the

support

label 2022-11-05 12:31:35 +01:00

anonfunc commented

2022-11-05 12:39:35 +01:00

Owner

So, I updated my system and lots of errors popped up.
Errors seem to come from systemd, bat, kitty (python has something to do with it) and flatpak but I've already seen that some other famous ones don't start up either, specifically electron-based ones.

But yes, openssl version on ALHP is still 1.1.1 when in upstream is 3.0.7

ALHP builds packages after they are release in Archs Source Repo. Since the openssl rebuild includes quite a lot of packages, this is going to take a while. You have to be a little patient, we sadly do not have all the computing power needed to provide "realtime" updates. In the meantime, deps can break if you try to upgrade packages coming from the official repos while ALHP is still not caught up. This can only be fixed by providing these packages from ALHP as well, but that would mean providing all packages Arch offers, even the ones that do not benefit from recompiling at all and would increase repo-size and repo-compiletime even more. (and would essentially make us a fully blown distro instead of just user-repos). Sadly this is simply not possible with the current build-server, as its not providing enough cpu-power to build some of the larger packages (especially the ones currently blacklisted because they clog up the build process) in a timely manner.

> So, I updated my system and lots of errors popped up. > Errors seem to come from `systemd`, `bat`, `kitty` (`python` has something to do with it) and `flatpak` but I've already seen that some other famous ones don't start up either, specifically electron-based ones. > > But yes, openssl version on ALHP is still 1.1.1 when in upstream is 3.0.7 ALHP builds packages **after** they are release in Archs Source Repo. Since the `openssl` rebuild includes quite a lot of packages, this is going to take a while. You have to be a little patient, we sadly do not have all the computing power needed to provide "realtime" updates. In the meantime, deps can break if you try to upgrade packages coming from the official repos while ALHP is still not caught up. This can only be fixed by providing these packages from ALHP as well, but that would mean providing all packages Arch offers, even the ones that do not benefit from recompiling at all and would increase repo-size and repo-compiletime even more. (and would essentially make us a fully blown distro instead of just user-repos). Sadly this is simply not possible with the current build-server, as its not providing enough cpu-power to build some of the larger packages (especially the ones currently blacklisted because they clog up the build process) in a timely manner.

🚀 1

plexman commented

2022-11-05 12:58:19 +01:00

Then we will wait patiently! Thanks!😉

👍 1

blackcat-917 commented

2022-11-05 17:49:24 +01:00

Author

If anyone is here wondering how to fix their system, here's what worked for me

Go to a tty
If you don't have pacman-static installed (most probably not), download a binary using wget https://pkgbuild.com/~morganamilo/pacman-static/x86_64/bin/pacman-static
chmod +x pacman-static
./pacman-static -S core/openssl core/openssl-1.1

This way you'll have both versions of openssl installed, once every package will be updated and linked with openssl v3 you'll be able to pacman -R openssl-1.1 (I personally recommend to do so also for security reasons)

If anyone is here wondering how to fix their system, here's what worked for me * Go to a tty * If you don't have `pacman-static` installed (most probably not), download a binary using `wget https://pkgbuild.com/~morganamilo/pacman-static/x86_64/bin/pacman-static` * `chmod +x pacman-static` * `./pacman-static -S core/openssl core/openssl-1.1` This way you'll have both versions of openssl installed, once every package will be updated and linked with openssl v3 you'll be able to `pacman -R openssl-1.1` (I personally recommend to do so also for security reasons)

blackcat-917 closed this issue

2022-11-05 17:58:23 +01:00

blackcat-917 reopened this issue

2022-11-05 17:58:28 +01:00

anonfunc commented

2022-11-05 18:25:54 +01:00

Owner

ALHP is around half-way through. ~300 packages left.

22:35 UTC: ~80 left.

ALHP is around half-way through. ~300 packages left. 22:35 UTC: ~80 left.

anonfunc commented

2022-11-06 16:39:55 +01:00

Owner

openssl rebuild cycle has completed.

`openssl` rebuild cycle has completed.

🎉 1

anonfunc commented

2022-11-07 19:00:13 +01:00

Owner

I'll close this, please reopen if you still experience issues.

anonfunc closed this issue

2022-11-07 19:00:13 +01:00

blorgymanblorg commented

2022-11-11 16:19:15 +01:00

boinc, boinc-nox, and boinctui are still linked against openssl-1.1. The Arch official packages are fine.

anonfunc changed title from ~~Many packages need rebuild because of openssl update~~ to Openssl 3 rebuild

2022-11-11 16:44:49 +01:00

anonfunc commented

2022-11-11 16:45:52 +01:00

Owner

Weird, that should not happen, since ALHP does check for up to date dependencies. Well, I requeued them while I'll have a look later why that is still possible.

anonfunc reopened this issue

2022-11-11 16:45:52 +01:00

anonfunc added the

invalid/corrupt package

label 2022-11-11 16:47:03 +01:00

plexman commented

2022-11-11 20:37:37 +01:00

I'm afraid it happens the same for nextcloud-client because when I don't have openssl-1.1 installed it throws:

nextcloud: error while loading shared libraries: libcrypto.so.1.1: cannot open shared object file: No such file or directory

And with openssl-1.1 it works.

I'm afraid it happens the same for [nextcloud-client](https://archlinux.org/packages/community/x86_64/nextcloud-client/) because when I don't have openssl-1.1 installed it throws: ``` nextcloud: error while loading shared libraries: libcrypto.so.1.1: cannot open shared object file: No such file or directory ``` And with openssl-1.1 it works.

anonfunc commented

2022-11-11 20:59:07 +01:00

Owner

Queued that as well. Hope not all ~500 packages are affected 😰.

yZibi commented

2022-11-13 21:51:25 +01:00

grub-customizer 5.2.2-2.1, qbittorrent 4.4.5-2.1 is affected. Versions from official repos (x.x.x-2) not depend on ssl1.1. Open ssl 3.0.7-2.1 from ALHP repo's.

anonfunc commented

2022-11-13 22:39:14 +01:00

Owner

Thanks @yZibi, I queued them for rebuild.

neersighted commented

2022-11-17 06:33:25 +01:00

mosh 1.4.0-2.1 is also still linked against OpenSSL 1.1 and needs a rebuild (which is interesting, as the community version built against OpenSSL 3.x is mosh 1.4.0-2).

anonfunc commented

2022-11-17 11:02:07 +01:00

Owner

Queued mosh.

Queued `mosh`.

N3k0-san commented

2022-12-01 01:55:33 +01:00

axel is still built against openssl-1.1

`axel` is still built against openssl-1.1

anonfunc commented

2022-12-01 07:41:06 +01:00

Owner

@N3k0-san Queued that, thanks!

👍 1

Piroro-hs commented

2022-12-04 09:07:41 +01:00

Looks like rustup is linked against OpenSSL 1.1.

Looks like `rustup` is linked against OpenSSL 1.1.

N3k0-san commented

2022-12-05 08:55:12 +01:00

@anonfunc It seems to still be linked against Openssl-1.1?
Or did it not rebuild yet? It still throws the ssl error when I use it :/

@anonfunc It seems to still be linked against Openssl-1.1? Or did it not rebuild yet? It still throws the ssl error when I use it :/

anonfunc commented

2022-12-05 10:02:59 +01:00

Owner

@N3k0-san Have you tried reinstalling it? ALHP has a bug where it sometimes does not increase the pkgrel correctly. You should get an invalid signature error, then you can redownload the rebuild one.

@Piroro-hs rustup does not seem to link directly to openssl. Maybe its just a dependency which is incorrectly linked? Can you post lddtree?

@N3k0-san Have you tried reinstalling it? ALHP has a bug where it sometimes does not increase the pkgrel correctly. You should get an invalid signature error, then you can redownload the rebuild one. @Piroro-hs `rustup` does not seem to link directly to openssl. Maybe its just a dependency which is incorrectly linked? Can you post `lddtree`?

N3k0-san commented

2022-12-05 10:42:03 +01:00

Have you tried reinstalling it?

Ah, uninstalling it and reinstalling it did work; didn't expect to need to do that
Thanks

> Have you tried reinstalling it? Ah, uninstalling it and reinstalling it did work; didn't expect to need to do that Thanks

Piroro-hs commented

2022-12-17 17:31:30 +01:00

Can you post lddtree?

$ lddtree $(which rustup)
/usr/bin/rustup (interpreter => /lib64/ld-linux-x86-64.so.2)
    liblzma.so.5 => /usr/lib/liblzma.so.5
    libssl.so.1.1 => None
    libcrypto.so.1.1 => None
    libcurl.so.4 => /usr/lib/libcurl.so.4
        libnghttp2.so.14 => /usr/lib/libnghttp2.so.14
        libidn2.so.0 => /usr/lib/libidn2.so.0
            libunistring.so.5 => /usr/lib/libunistring.so.5
        libssh2.so.1 => /usr/lib/libssh2.so.1
        libpsl.so.5 => /usr/lib/libpsl.so.5
        libssl.so.3 => /usr/lib/libssl.so.3
        libcrypto.so.3 => /usr/lib/libcrypto.so.3
        libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2
            libkrb5.so.3 => /usr/lib/libkrb5.so.3
            libk5crypto.so.3 => /usr/lib/libk5crypto.so.3
            libcom_err.so.2 => /usr/lib/libcom_err.so.2
            libkrb5support.so.0 => /usr/lib/libkrb5support.so.0
            libkeyutils.so.1 => /usr/lib/libkeyutils.so.1
            libresolv.so.2 => /usr/lib/libresolv.so.2
        libzstd.so.1 => /usr/lib/libzstd.so.1
        libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1
            libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1
        libz.so.1 => /usr/lib/libz.so.1
    libgcc_s.so.1 => /usr/lib/libgcc_s.so.1
    libm.so.6 => /usr/lib/libm.so.6
    libc.so.6 => /usr/lib/libc.so.6

Sorry for late reply. I don't know why but rustup seems to link to OpenSSL 1.1 directly and to 3.0 via libcurl dependency.

> Can you post `lddtree`? ``` $ lddtree $(which rustup) /usr/bin/rustup (interpreter => /lib64/ld-linux-x86-64.so.2) liblzma.so.5 => /usr/lib/liblzma.so.5 libssl.so.1.1 => None libcrypto.so.1.1 => None libcurl.so.4 => /usr/lib/libcurl.so.4 libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 libidn2.so.0 => /usr/lib/libidn2.so.0 libunistring.so.5 => /usr/lib/libunistring.so.5 libssh2.so.1 => /usr/lib/libssh2.so.1 libpsl.so.5 => /usr/lib/libpsl.so.5 libssl.so.3 => /usr/lib/libssl.so.3 libcrypto.so.3 => /usr/lib/libcrypto.so.3 libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 libkrb5.so.3 => /usr/lib/libkrb5.so.3 libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 libcom_err.so.2 => /usr/lib/libcom_err.so.2 libkrb5support.so.0 => /usr/lib/libkrb5support.so.0 libkeyutils.so.1 => /usr/lib/libkeyutils.so.1 libresolv.so.2 => /usr/lib/libresolv.so.2 libzstd.so.1 => /usr/lib/libzstd.so.1 libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 libz.so.1 => /usr/lib/libz.so.1 libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 libm.so.6 => /usr/lib/libm.so.6 libc.so.6 => /usr/lib/libc.so.6 ``` Sorry for late reply. I don't know why but `rustup` seems to link to OpenSSL 1.1 directly and to 3.0 via libcurl dependency.

anonfunc commented

2022-12-19 19:21:19 +01:00

Owner

Sorry for late reply. I don't know why but rustup seems to link to OpenSSL 1.1 directly and to 3.0 via libcurl dependency.

I guess that's also the root of the problem in regards to why ALHP has not recognized that out of date dependency situation. I'll need to add a recursive dependency version check so that ALHP can catch these cases.

I requeued it in the meantime.

> Sorry for late reply. I don't know why but `rustup` seems to link to OpenSSL 1.1 directly and to 3.0 via libcurl dependency. I guess that's also the root of the problem in regards to why ALHP has not recognized that out of date dependency situation. I'll need to add a recursive dependency version check so that ALHP can catch these cases. I requeued it in the meantime.

HLFH commented

2022-12-27 09:59:39 +01:00

I am using x86-64-v2, and dovecot broke with ALHP.
I decided to remove ALHP and stay on x86-64.

I had the error:

imap-login: Error: Failed to initialize SSL server context: Can't load SSL certificate (ssl_cert setting): error:25066067:DSO support routines:dlfcn_load:could not load the shared library: filename(libproviders.so): libproviders.so: cannot open shared object file: No such file or directory, error:25070067:DSO support routines:DSO_load:could not load the shared library, error:0E07506E:configuration file routines:module_load_dso:error loading dso: module=providers, path=providers, error:0E076071:configuration file routines:module_run:unknown module [...]

I am using `x86-64-v2`, and `dovecot` broke with ALHP. I decided to remove ALHP and stay on x86-64. I had the error: ``` imap-login: Error: Failed to initialize SSL server context: Can't load SSL certificate (ssl_cert setting): error:25066067:DSO support routines:dlfcn_load:could not load the shared library: filename(libproviders.so): libproviders.so: cannot open shared object file: No such file or directory, error:25070067:DSO support routines:DSO_load:could not load the shared library, error:0E07506E:configuration file routines:module_load_dso:error loading dso: module=providers, path=providers, error:0E076071:configuration file routines:module_run:unknown module [...] ```

anonfunc commented

2022-12-27 17:32:57 +01:00

Owner

I requeued dovecot, thanks for the hint.

I requeued `dovecot`, thanks for the hint.

👍 1

HLFH commented

2022-12-27 19:23:28 +01:00

@anonfunc Thanks! Any plans to fix it definitely as openssl is quite critical to Arch Linux?

@anonfunc Thanks! Any plans to fix it definitely as `openssl` is quite critical to Arch Linux?

anonfunc commented

2022-12-27 19:42:52 +01:00

Owner

There is sadly no way to "fix" this elegantly (after the fact at least, besides rebuilding almost the whole repo, which is not practical). This should not have happened in the first place, since ALHP does check deps before building. I'm still not quite sure what went wrong, and I sadly have not that much time on hands at the moment to make the deep-dive necessary. We already caught quite a few packages, so I hope there are not that many left.

I checked if there is an option to check for missing shared libs via ldd(tree) or similar, but that would mean installing each package in a clean chroot and then parse ldd(tree) (or run something like rebuild-detector), since on a used system you could have the necessary dep by "accident". rebuild-detector does these checks and more, but sadly does not do them in a clean chroot, so its more or less useless in this case. Furthermore, I can't estimate how many false positives there will be, but I imagine quite a few. We could have some logic that will rebuild these once, and if they still show up (with the same libs 'missing'), it's probably not a build issue. But that would only add to the time required adding such a feature, wich I currently have too little of on hand to implement. Besides, I think it would be wiser to invest time in making this not happen in the first place than spending all this time fixing the symptoms.

There is sadly no way to "fix" this elegantly (after the fact at least, besides rebuilding almost the whole repo, which is not practical). This should not have happened in the first place, since ALHP does check deps before building. I'm still not quite sure what went wrong, and I sadly have not that much time on hands at the moment to make the deep-dive necessary. We already caught quite a few packages, so I hope there are not that many left. I checked if there is an option to check for missing shared libs via ldd(tree) or similar, but that would mean installing each package in a clean chroot and then parse ldd(tree) (or run something like `rebuild-detector`), since on a used system you could have the necessary dep by "accident". `rebuild-detector` does these checks and more, but sadly does not do them in a clean chroot, so its more or less useless in this case. Furthermore, I can't estimate how many false positives there will be, but I imagine quite a few. We could have some logic that will rebuild these once, and if they still show up (with the same libs 'missing'), it's probably not a build issue. But that would only add to the time required adding such a feature, wich I currently have too little of on hand to implement. Besides, I think it would be wiser to invest time in making this not happen in the first place than spending all this time fixing the symptoms.

👍 1