Built-in packages

LLVM

class infra.packages.LLVM(version, compiler_rt, commit=None, lld=False, patches=[], build_flags=[])[source]

LLVM dependency package. Includes the Clang compiler and optionally compiler-rt (which contains runtime support for ASan).

Supports a number of patches to be passed as arguments, which are applied (with patch -p1) before building. A patch in the list can either be a full path to a patch file, or the name of a built-in patch. Available built-in patches are:

  • gold-plugins (for 3.8.0/3.9.1/4.0.0/5.0.0/7.0.0): adds a -load option to load passes from a shared object file during link-time optimizations, best used in combination with LLVMPasses
  • statsfilter (for 3.8.0/3.9.1/5.0.0/7.0.0): adds -stats-only option, which relates to -stats like -debug-only relates to -debug
  • lto-nodiscard-value-names (for 7.0.0): preserves value names when producing bitcode for LTO, this is very useful when debugging passes
  • safestack (for 3.8.0): adds -fsanitize=safestack for old LLVM
  • compiler-rt-typefix (for 4.0.0): fixes a compiler-rt-4.0.0 bug to make it compile for recent glibc, is applied automatically if compiler_rt is set
Identifier:

llvm-<version>
Parameters:
  • version (str) – the full LLVM version to download, like X.Y.Z
  • compiler_rt (bool) – whether to enable compiler-rt
  • patches (typing.List[str]) – optional patches to apply before building
  • build_flags (typing.List[str]) – additional build flags to pass to cmake
configure(self, ctx)[source]

Set LLVM toolchain programs in ctx. Should be called from the configure method of an instance.

Parameters:ctx (context.Context) – the configuration context
Return type:None
static add_plugin_flags(ctx, *flags, gold_passes=True)[source]

Helper to pass link-time flags to the LLVM gold plugin. Prefixes all flags with -Wl,-plugin-opt= before adding them to ctx.ldflags.

Parameters:
Return type:

None

Dependencies

class infra.packages.AutoConf(version, m4)[source]
Identifier:

autoconf-<version>
Parameters:
  • version (str) – version to download
  • m4 (packages.M4) – M4 package
class infra.packages.AutoMake(version, autoconf, libtool)[source]
Identifier:

automake-<version>
Parameters:
classmethod default(automake_version='1.16.5', autoconf_version='2.71', m4_version='1.4.19', libtool_version='2.4.6')[source]

Create a package with default versions for all autotools.

Parameters:
  • automake_version (str) – automake version
  • autoconf_version (str) – autoconf version
  • m4_version (str) – m4 version
  • libtool_version (str or None) – optional libtool version
Return type:

AutoMake
class infra.packages.Bash(version)[source]
Identifier:bash-<version>
Parameters:version (str) – version to download
class infra.packages.BinUtils(version, gold=True)[source]
Identifier:

binutils-<version>[-gold]
Parameters:
  • version (str) – version to download
  • gold (bool) – whether to use the gold linker
class infra.packages.CMake(version)[source]
Identifier:cmake-<version>
Parameters:version (str) – version to download
class infra.packages.CoreUtils(version)[source]
Identifier:coreutils-<version>
Parameters:version (str) – version to download
class infra.packages.LibElf(version)[source]
Identifier:libelf-<version>
Parameters:version (str) – version to download
class infra.packages.LibTool(version)[source]
Identifier:libtool-<version>
Parameters:version (str) – version to download
class infra.packages.M4(version)[source]
Identifier:m4-<version>
Parameters:version (str) – version to download
class infra.packages.Make(version)[source]
Identifier:make-<version>
Parameters:version (str) – version to download
class infra.packages.Ninja(version)[source]
Identifier:ninja-<version>
Parameters:version (str) – version to download

LLVM passes

class infra.packages.LLVMPasses(llvm, srcdir, build_suffix, use_builtins, debug=False, gold_passes=True)[source]

LLVM passes dependency. Use this to add your own passes as a dependency to your own instances. In your own passes directory, your Makefile should look like this (see the skeleton for an example):

BUILD_SUFFIX = <build_suffix>
LLVM_VERSION = <llvm_version>
SETUP_SCRIPT = <path_to_setup.py>
SUBDIRS      = <optional list of subdir names containing passes>
include <path_to_infra>/infra/packages/llvm_passes/Makefile

The makefile can be run as-is using make in your passes directory during development, without invoking the setup script directly. It creates two shared objects in build/packages/llvm-passes-<build_suffix>/install:

  • libpasses-gold.so: used to load the passes at link time in Clang. This is the default usage.
  • libpasses-opt.so: used to run the passes with LLVM’s opt utility. Can be used in a customized build system or for debugging.

The passes are invoked at link time by a patched LLVM gold plugin. The gold-plugin patch of the LLVM package adds an option to load custom passes into the plugin. Passes are invoked by adding their registered names to the flags passed to the LLVM gold plugin by the linker. In other words, by adding -Wl,-plugin-opt=<passname> to ctx.ldflags in the configure method of your instance. The LLVM.add_plugin_flags() helper does exactly that. Before using passes, you must call llvm_passes.configure(ctx) to load the passes into the plugin. See the skeleton LibcallCount instance for an example.

For the pkg-config command of this package, the --objdir option points to the build directory.

Identifier:

llvm-passes-<build_suffix>
Parameters:
  • llvm (packages.LLVM) – LLVM package to link against
  • srcdir (str) – source directory containing your LLVM passes
  • build_suffix (str) – identifier for this set of passes
  • use_builtins (bool) – whether to include built-in LLVM passes in the shared object
  • debug (bool) – enable to compile passes with -O0 -ggdb
Todo:

extend this to support compile-time plugins
configure(self, ctx, *, linktime=True, compiletime=True)[source]

Set build/link flags in ctx. Should be called from the configure method of an instance.

linktime and compiletime can be set to false to avoid loading the pass libraries at link time and at compile time, respectively. Loading passes at link time requires LLVM to be built with the gold-plugin patch.

Parameters:
  • ctx (context.Context) – the configuration context
  • linktime (bool) – are the passes used at link time?
  • compiletime (bool) – are the passes used at compile time?
Return type:

None
runtime_cflags(self, ctx)[source]

Returns a list of CFLAGS to pass to a runtime library that depends on features from passes. These set include directories for header includes of built-in pass functionalities such as the NOINSTRUMENT macro.

Parameters:ctx (context.Context) – the configuration context
Return type:typing.Iterable[str]
class infra.packages.BuiltinLLVMPasses(llvm, gold_passes=True)[source]

Subclass of LLVMPasses for built-in passes. Use this if you don’t have any custom passes and just want to use the built-in passes. Configuration happens in the same way as described above: by calling the configure() method.

In addition to the shared objects listed above, this package also produces a static library called libpasses-builtin.a which is used by the LLVMPasses to include built-in passes when use_builtins is True.

For the pkg-config command of this package, the following options are added in addition to --root/--prefix/--objdir:

  • --cxxflags lists compilation flags for custom passes that depend on built-in analysis passes (sets include path for headers).
  • --runtime-cflags prints the value of LLVMPasses.runtime_cflags().
Identifier:llvm-passes-builtin-<llvm.version>
Parameters:llvm (packages.LLVM) – LLVM package to link against

Address space shrinking

class infra.packages.LibShrink(addrspace_bits, commit='master', debug=False)[source]

Dependency package for libshrink.

Libshrink shrinks the application address space to a maximum number of bits. It moves the stack and TLS to a memory region that is within the allowed bitrange, and prelinks all shared libraries as well so that they do not exceed the address space limitations. It also defines a run_wrapper() that should be put in ctx.target_run_wrapper by an instance that uses libshrink.

Identifier:

libshrink-<addrspace_bits>
Parameters:
  • addrspace_bits (int) – maximum number of nonzero bits in any pointer
  • commit (str) – branch or commit to clone
  • debug (bool) – whether to compile with debug symbols
configure(self, ctx, static=True)[source]

Set build/link flags in ctx. Should be called from the configure method of an instance. Uses post-build hooks, so any target compiled with this libary must implement infra.Target.binary_paths().

Parameters:
  • ctx (context.Context) – the configuration context
  • static (bool) – use the static library? (shared library otherwise)
Return type:

None
Raises:

NotImplementedError – if static is not True (TODO)
run_wrapper(self, ctx)[source]

Run wrapper for targets. Links to a script that sets the rpath before any libraries are loaded, so that any dependencies of shared libraries loaded by the applications are also loaded from the directory of prelinked libraries (which is created by a post-build hook).

Parameters:ctx (context.Context) – the configuration context
Return type:str

Dependencies

class infra.packages.PatchElf(version)[source]
Identifier:patchelf-<version>
Parameters:version (str) – version to download
Identifier:prelink-<version>
Parameters:version (str) – version to download
class infra.packages.PyElfTools(version, python_version)[source]
Identifier:

pyelftools-<version>
Parameters:
  • version (str) – version to download
  • python_version (str) – which Python version to install the package for

TCmalloc

class infra.packages.Gperftools(commit, libunwind_version='1.4-rc1', patches=[])[source]
Identifier:

gperftools-<version>
Parameters:
  • commit (str) – git branch/commit to check out after cloning
  • libunwind_version (str) – libunwind version to use
  • patches (typing.List[str]) – optional patches to apply before building
configure(self, ctx)[source]

Set build/link flags in ctx. Should be called from the configure method of an instance.

Sets the necessary -I/-L/-l flags, and additionally adds -fno-builtin-{malloc,calloc,realloc,free} to CFLAGS.

Parameters:ctx (context.Context) – the configuration context
Return type:None

Dependencies

class infra.packages.LibUnwind(version, patches=[])[source]
Identifier:libunwind-<version>
Parameters:version (str) – version to download

Tools

class infra.packages.Nothp(*args, **kwargs)[source]
Identifier:nothp
class infra.packages.RusageCounters(*args, **kwargs)[source]

Utility library for targets that want to measure resource counters:

  • memory (max resident set size)
  • page faults
  • I/O operations
  • context switches
  • runtime (esimated by gettimeofday in constructor+destructor)

The target only needs to depend on this package and configure() it to link the static library which will then log a reportable result in a destructor. See SPEC2006 for a usage example.

Identifier:rusage-counters
classmethod parse_results(ctx, path, allow_missing=False)[source]

Parse any results containing counters by this package.

Parameters:
  • ctx (context.Context) – the configuration context
  • path (str) – path to file to parse
Returns:

counter results
Return type:

typing.MutableMapping[str, typing.Union[bool, int, float, str]]
configure(self, ctx)[source]

Set build/link flags in ctx. Should be called from the build method of a target to link in the static library.

Parameters:ctx (context.Context) – the configuration context
Return type:None
pkg_config_options(self, ctx)[source]

Yield options for the pkg-config command. Each option is an (option, description, value) triple. The defaults are --root which returns the root directory build/packages/<ident>, and --prefix which returns the install directory populated by install(): build/packages/<ident>/install.

When reimplementing this method in a derived package class, it is recommended to end the implementation with yield from super().pkg_config_options(ctx) to add the two default options.

Parameters:ctx (context.Context) – the configuration context
Return type:typing.Iterator[typing.Tuple[str, str, typing.Union[str, typing.Iterable[str]]]]

Apache benchmark (ab)

class infra.packages.ApacheBench(httpd_version, apr, apr_util)[source]

Apache’s ab benchmark.

Identifier:

ab-<version>
Parameters:
Return type:

None

Dependencies

class infra.packages.APR(version)[source]

The Apache Portable Runtime.

Identifier:apr-<version>
Parameters:version (str) – version to download
class infra.packages.APRUtil(version, apr)[source]

The Apache Portable Runtime utilities.

Identifier:

apr-util-<version>
Parameters:
  • version (str) – version to download
  • apr (packages.APR) – APR package to depend on
Return type:

None

Wrk benchmark

class infra.packages.Wrk(version='master')[source]

The wrk benchmark.

Identifier:wrk-<version>
Parameters:version (str) – version to download
class infra.packages.Wrk2(version='master')[source]

The wrk2 benchmark.

Identifier:wrk2-<version>
Parameters:version (str) – version to download

Scons

class infra.packages.Scons(version)[source]

The scons build tool (replacement for make).

Identifier:scons-<version>
Parameters:version (str) – version to download