Use fat-lto-objects to avoid rebuild on LTO failure #114

New Issue

Open

opened 2022-05-14 01:48:38 +02:00 by RubenKelevra · 2 comments

RubenKelevra commented

2022-05-14 01:48:38 +02:00

Contributor

As far as I understand we currently rebuild packages if the LTO-Build fails, right?

If we set -ffat-lto-objects while compiling we could do LTO linking and normal linking.

-ffat-lto-objects

    Fat LTO objects are object files that contain both the intermediate language and the object code. This makes them usable for both LTO linking and normal linking. This option is effective only when compiling with -flto and is ignored at link time.

    -fno-fat-lto-objects improves compilation time over plain LTO, but requires the complete toolchain to be aware of LTO. It requires a linker with linker plugin support for basic functionality. Additionally, nm, ar and ranlib need to support linker plugins to allow a full-featured build environment (capable of building static libraries etc). GCC provides the gcc-ar, gcc-nm, gcc-ranlib wrappers to pass the right options to these tools. With non fat LTO makefiles need to be modified to use them.

    Note that modern binutils provide plugin auto-load mechanism. Installing the linker plugin into $libdir/bfd-plugins has the same effect as usage of the command wrappers (gcc-ar, gcc-nm and gcc-ranlib).

    The default is -fno-fat-lto-objects on targets with linker plugin support.

I think we would also need to set -flinker-output=rel. Docs:

-flinker-output=type

    This option controls code generation of the link-time optimizer. By default the linker output is automatically determined by the linker plugin. For debugging the compiler and if incremental linking with a non-LTO object file is desired, it may be useful to control the type manually.

    If type is ‘exec’, code generation produces a static binary. In this case -fpic and -fpie are both disabled.

    If type is ‘dyn’, code generation produces a shared library. In this case -fpic or -fPIC is preserved, but not enabled automatically. This allows to build shared libraries without position-independent code on architectures where this is possible, i.e. on x86.

    If type is ‘pie’, code generation produces an -fpie executable. This results in similar optimizations as ‘exec’ except that -fpie is not disabled if specified at compilation time.

    If type is ‘rel’, the compiler assumes that incremental linking is done. The sections containing intermediate code for link-time optimization are merged, pre-optimized, and output to the resulting object file. In addition, if -ffat-lto-objects is specified, binary code is produced for future non-LTO linking. The object file produced by incremental linking is smaller than a static library produced from the same object files. At link time the result of incremental linking also loads faster than a static library assuming that the majority of objects in the library are used.

    Finally ‘nolto-rel’ configures the compiler for incremental linking where code generation is forced, a final binary is produced, and the intermediate code for later link-time optimization is stripped. When multiple object files are linked together the resulting code is better optimized than with link-time optimizations disabled (for example, cross-module inlining happens), but most of benefits of whole program optimizations are lost.

    During the incremental link (by -r) the linker plugin defaults to rel. With current interfaces to GNU Binutils it is however not possible to incrementally link LTO objects and non-LTO objects into a single mixed object file. If any of object files in incremental link cannot be used for link-time optimization, the linker plugin issues a warning and uses ‘nolto-rel’. To maintain whole program optimization, it is recommended to link such objects into static library instead. Alternatively it is possible to use H.J. Lu’s binutils with support for mixed objects.

As far as I understand we currently rebuild packages if the LTO-Build fails, right? If we set `-ffat-lto-objects` while compiling we could do LTO linking *and* normal linking. ``` -ffat-lto-objects Fat LTO objects are object files that contain both the intermediate language and the object code. This makes them usable for both LTO linking and normal linking. This option is effective only when compiling with -flto and is ignored at link time. -fno-fat-lto-objects improves compilation time over plain LTO, but requires the complete toolchain to be aware of LTO. It requires a linker with linker plugin support for basic functionality. Additionally, nm, ar and ranlib need to support linker plugins to allow a full-featured build environment (capable of building static libraries etc). GCC provides the gcc-ar, gcc-nm, gcc-ranlib wrappers to pass the right options to these tools. With non fat LTO makefiles need to be modified to use them. Note that modern binutils provide plugin auto-load mechanism. Installing the linker plugin into $libdir/bfd-plugins has the same effect as usage of the command wrappers (gcc-ar, gcc-nm and gcc-ranlib). The default is -fno-fat-lto-objects on targets with linker plugin support. ``` I think we would also need to set `-flinker-output=rel`. Docs: ``` -flinker-output=type This option controls code generation of the link-time optimizer. By default the linker output is automatically determined by the linker plugin. For debugging the compiler and if incremental linking with a non-LTO object file is desired, it may be useful to control the type manually. If type is ‘exec’, code generation produces a static binary. In this case -fpic and -fpie are both disabled. If type is ‘dyn’, code generation produces a shared library. In this case -fpic or -fPIC is preserved, but not enabled automatically. This allows to build shared libraries without position-independent code on architectures where this is possible, i.e. on x86. If type is ‘pie’, code generation produces an -fpie executable. This results in similar optimizations as ‘exec’ except that -fpie is not disabled if specified at compilation time. If type is ‘rel’, the compiler assumes that incremental linking is done. The sections containing intermediate code for link-time optimization are merged, pre-optimized, and output to the resulting object file. In addition, if -ffat-lto-objects is specified, binary code is produced for future non-LTO linking. The object file produced by incremental linking is smaller than a static library produced from the same object files. At link time the result of incremental linking also loads faster than a static library assuming that the majority of objects in the library are used. Finally ‘nolto-rel’ configures the compiler for incremental linking where code generation is forced, a final binary is produced, and the intermediate code for later link-time optimization is stripped. When multiple object files are linked together the resulting code is better optimized than with link-time optimizations disabled (for example, cross-module inlining happens), but most of benefits of whole program optimizations are lost. During the incremental link (by -r) the linker plugin defaults to rel. With current interfaces to GNU Binutils it is however not possible to incrementally link LTO objects and non-LTO objects into a single mixed object file. If any of object files in incremental link cannot be used for link-time optimization, the linker plugin issues a warning and uses ‘nolto-rel’. To maintain whole program optimization, it is recommended to link such objects into static library instead. Alternatively it is possible to use H.J. Lu’s binutils with support for mixed objects. ```

anonfunc commented

2022-05-20 17:17:29 +02:00

Owner

This would increase overall build time, not sure that tradeoff is worth, especially since detecting LTO build failures is a one-off thing.

anonfunc added the

question

label 2022-05-20 17:17:34 +02:00

RubenKelevra commented

2022-05-21 19:37:27 +02:00

Author

Contributor

Not sure it would really increase the build time that much. As that's just a little bit more data beeing stored on disk which are in memory anyway. So it's more a temporary diskspace thing.

The advantage would be, that we can do LTO-detection on each version, so if a program improves and it can just be linked via LTO it would be done, else a normal linking would take place without the need for rebuilding it completly.

Not sure it would really increase the build time that much. As that's just a little bit more data beeing stored on disk which are in memory anyway. So it's more a temporary diskspace thing. The advantage would be, that we can do LTO-detection on each version, so if a program improves and it can just be linked via LTO it would be done, else a normal linking would take place without the need for rebuilding it completly.