API documentation¶

Setup¶

class infra.Setup(setup_path)[source]¶

Defines framework commands.

The setup takes care of complicated things like command-line parsing, logging, parallelism, environment setup and generating build paths. You should only need to use the methods documented here. To use the setup, you must first populate it with targets and instances using add_target() and add_instance(), and then call main() to run the command issued in the command-line arguments:

setup = infra.Setup(__file__)
setup.add_instance(MyAwesomeInstance())
setup.add_target(MyBeautifulTarget())
setup.main()

main() creates a configuration context that it passes to methods of targets/instances/packages. You can see it being used as ctx by many API methods below. The context contains setup configuration data, such as absolute build paths, and environment variables for build/run commands, such as which compiler and CFLAGS to use to build the current target. Your own targets and instances should read/write to the context.

The job of an instance is to manipulate the the context such that a target is built in the desired way. This manipulation happens in predefined API methods which you must overwrite (see below). Hence, these methods receive the context as a parameter.

Parameters:	setup_path (str) – Path to the script running `Setup.main()`. Needed to allow build scripts to call back into the setup script for build hooks.

ctx = None¶

util.Namespace The configuration context.

Consider an example project hosted in directory /project, with the infrastructure cloned as a submodule in /project/infra and a setup script like the one above in /project/setup.py. The context will look like this after initialization:

Namespace({
    'log':         logging.Logger(...),
    'args':        argparse.Namespace(...),
    'jobs':        64,
    'paths':       Namespace({
                       'root':         '/project',
                       'setup':        '/project/setup.py',
                       'infra':        '/project/infra'
                       'buildroot':    '/project/build',
                       'log':          '/project/build/log',
                       'debuglog':     '/project/build/log/debug.txt',
                       'runlog':       '/project/build/log/commands.txt',
                       'packages':     '/project/build/packages',
                       'targets':      '/project/build/targets',
                       'pool_results': '/project/results'
                   }),
    'runenv':      Namespace({}),
    'cc':          'cc',
    'cxx':         'c++',
    'cpp':         'cpp',
    'ld':          'ld',
    'ar':          'ar',
    'nm':          'nm',
    'ranlib':      'ranlib',
    'cflags':      [],
    'cxxflags':    [],
    'cppflags':    [],
    'ldflags':     [],
    'lib_ldflags': [],
    'ldlibs':      [],
    'hooks':       Namespace({
                       'post_build': [],
                       'pre_build': [],
                   }),
    'starttime':   datetime.datetime
})

The util.Namespace class is simply a dict whose members can be accessed like attributes.

ctx.log is a logging object used for status updates. Use this to provide useful information about what your implementation is doing.

ctx.args is populated with processed command-line arguments, It is available to read custom build/run arguments from that are added by targets/instances.

ctx.jobs contains the value of the -j command-line option, defaulting to the number of CPU cores returned by multiprocessing.cpu_count().

ctx.paths are absolute paths to be used (readonly) throughout the framework.

ctx.runenv defines environment variables for util.run(), which is a wrapper for subprocess.run() that does logging and other useful things.

ctx.{cc,cxx,cpp,ld,ar,nm,ranlib} define default tools of the compiler toolchain, and should be used by target definitions to configure build scripts. ctx.{c,cxx,ld}flags similarly define build flags for targets in a list and should be joined into a string using util.qjoin() when being passed as a string to a build script by a target definition.

ctx.{c,cxx,cpp,ld}flags should be set by instances to define flags for compiling targets.

ctx.ldlibs list of libraries to link to target (space-separated)

ctx.lib_ldflags is a special set of linker flags set by some packages, and is passed when linking target libraries that will later be (statically) linked into the binary. In practice it is either empty or ['-flto'] when compiling with LLVM. Also useful to separate linker flags when runnign LTO passed for targets using ./configure scripts.

ctx.hooks.post_build defines a list of post-build hooks, which are python functions called with the path to the binary as the only parameter.

ctx.hooks.pre_build defines a list of pre-build hooks, which are python functions called with the path to the binary as the only parameter.

ctx.starttime is set to datetime.datetime.now().

ctx.workdir is set to the work directory from which the setup script is invoked.

ctx.target_run_wrapper can be set to a program to prepend in front of the target’s run command (executed directly on the command line). This can be set to a custom shell script, or for example something like perf or valgrind.

add_command(self, command)[source]¶

Register a setup command.

Parameters:	command (Command) – The command to register.

add_instance(self, instance)[source]¶

Register an instance. Only registered instances can be referenced in commands, so also built-in instances must be registered.

Parameters:	instance (Instance) – The instance to register.

add_target(self, target)[source]¶

Register a target. Only registered targets can be referenced in commands, so also built-in targets must be registered.

Parameters:	target (Target) – The target to register.

main(self)[source]¶

Run the configured setup:

Parse command-line arguments.
Create build directories and log files.
Run the issued command.

Targets¶

class infra.Target(*args, **kwargs)[source]¶

Abstract base class for target definitions. Built-in derived classes are listed here.

Each target must define a name attribute that is used to reference the target on the command line. The name must be unique among all registered targets. Each target must also implement a number of methods that are called by Setup when running commands.

The build command follows the following steps for each target:

It calls add_build_args() to include any custom command-line arguments for this target, and then parses the command-line arguments.
It calls is_fetched() to see if the source code for this target has been downloaded yet.
If is_fetched() == False, it calls fetch().
It calls Instance.configure() on the instance that will be passed to build().
All packages listed by dependencies() are built and installed into the environment (i.e., PATH and such are set).
It calls build() to build the target binaries.
If any post-build hooks are installed by the current instance, it calls binary_paths() to get paths to all built binaries. These are then passed directly to the build hooks.

For the run command:

It calls add_run_args() to include any custom command-line arguments for this target.
If --build was specified, it performs all build steps above.
It calls Instance.prepare_run() on the instance that will be passed to run().
It calls run() to run the target binaries.

For the clean command:

It calls is_clean() to see if any build files exist for this target.
If is_clean() == False, it calls clean().

For the report command:

It calls parse_outfile() for every log file before creating the report.

Naturally, when defining your own target, all the methods listed above must have working implementations. Some implementations are optional and some have a default implementation that works for almost all cases (see docs below), but the following are mandatory to implement for each new target: is_fetched(), fetch(), build() and run().

name = None¶: str The target’s name, must be unique.

reportable_fields = {}¶: A mapping of fields that can be reported to their descriptions.

add_build_args(self, parser)[source]¶

Extend the command-line arguments for the build command with custom arguments for this target. These arguments end up in the global namespace, so it is a good idea to prefix them with the target name to avoid collisions with other targets and instances.

For example, SPEC2006 defines --spec2006-benchmarks (rather than --benchmarks).

Parameters:	parser (argparse.ArgumentParser) – the argument parser to extend

add_run_args(self, parser)[source]¶

Extend the command-line arguments for the run command with custom arguments for this target. Since only a single target can be run at a time, prefixing to avoid naming conflicts with other targets is not necessary here.

For example, SPEC2006 defines --benchmarks and --test.

Parameters:	parser (argparse.ArgumentParser) – the argument parser to extend

dependencies(self)[source]¶

Specify dependencies that should be built and installed in the run environment before building this target.

Return type:	typing.Iterator[infra.package.Package]

path(self, ctx, *args)[source]¶

Get the absolute path to the build directory of this target, optionally suffixed with a subpath.

Parameters:	ctx (util.Namespace) – the configuration context args (typing.Iterable[str]) – additional subpath to pass to `os.path.join()`
Returns:	`build/targets/<name>[/<subpath>]`
Return type:	str

is_fetched(self, ctx)[source]¶

Returns True if fetch() should be called before building.

Parameters:	ctx (util.Namespace) – the configuration context
Return type:	bool

fetch(self, ctx)[source]¶

Fetches the source code for this target. This step is separated from build() because the build command first fetches all packages and targets before starting the build process.

Parameters:	ctx (util.Namespace) – the configuration context

build(self, ctx, instance, pool=None)[source]¶

Build the target object files. Called some time after fetch() (see above).

ctx.runenv will have been populated with the exported environments of all packages returned by dependencies() (i.e., Package.install_env() has been called for each dependency). This means that when you call util.run() here, the programs and libraries from the dependencies are available in PATH and LD_LIBRARY_PATH, so you don’t need to reference them with absolute paths.

The build function should respect variables set in the configuration context such as ctx.cc and ctx.cflags, passing them to the underlying build system as required. Setup.ctx shows default variables in the context that should at least be respected, but complex instances may optionally overwrite them to be used by custom targets.

Any custom command-line arguments set by add_build_args() are available here in ctx.args.

An implementation of build() may optionally define a parameter pool. If defined, the target is expected to support parallel builds when --parallel is passed. In that case, pool will be an object of type Pool, and the method should call pool.run() instead of util.run() to invoke build commands.

Parameters:	ctx (util.Namespace) – the configuration context instance (Instance) – instance to build pool (parallel.Pool or None) – parallel process pool if `--parallel` is specified

run(self, ctx, instance, pool=None)[source]¶

Run the target binaries. This should be done using util.run() so that ctx.runenv is used (which can be set by an instance or dependencies). It is recommended to pass teeout=True to make the output of the process stream to stdout.

Any custom command-line arguments set by add_run_args() are available here in ctx.args.

Similarly to build(), the method may specify the pool parameter for parallel running of different benchmark program in the target. Of course, resource sharing will occur when using --parallel=proc, which may impact runtime performance.

Implementations of this method should respect the --iterations option of the run command.

Parameters:	ctx (util.Namespace) – the configuration context instance (Instance) – instance to run pool (parallel.Pool or None) – parallel process pool if `--parallel` is specified

parse_outfile(self, ctx, instance_name, outfile)[source]¶

Callback method for commands.report.parse_logs(). Used by report command to get reportable results.

Parameters:	ctx (util.Namespace) – the configuration context instance_name (str) – name of corresponding instance outfile (str) – path to outfile to parse
Return type:	typing.Iterator[typing.Dict[str, typing.Any]]
Raises:	NotImplementedError – unless implemented

is_clean(self, ctx)[source]¶

Returns True if clean() should be called before cleaning.

Parameters:	ctx (util.Namespace) – the configuration context
Return type:	bool

clean(self, ctx)[source]¶

Clean generated files for this target, called by the clean command. By default, this removes build/targets/<name>.

Parameters:	ctx (util.Namespace) – the configuration context

binary_paths(self, ctx, instance)[source]¶

If implemented, this should return a list of absolute paths to binaries created by build() for the given instance. This is only used if the instance specifies post-build hooks. Each hook is called for each of the returned paths.

Parameters:	ctx (util.Namespace) – the configuration context instance (Instance) – instance to get paths for
Returns:	paths to binaries
Return type:	typing.Iterable[str]
Raises:	NotImplementedError – unless implemented

Instances¶

class infra.Instance(*args, **kwargs)[source]¶

Abstract base class for instance definitions. Built-in derived classes are listed here.

Each instance must define a name attribute that is used to reference the instance on the command line. The name must be unique among all registered instances.

An instance changes variables in the configuration context that are used to apply instrumentation while building a target by Target.build() and Target.link(). This is done by configure().

Additionally, instances that need runtime support, such as a shared library, can implement prepare_run() which is called by the run command just before running the target with Target.run().

name = None¶

str The instance’s name, must be unique.

Type: str

add_build_args(self, parser)[source]¶

Extend the command-line arguments for the build command with custom arguments for this instance. These arguments end up in the global namespace, so it is a good idea to prefix them with the instance name to avoid collisions with other instances and targets.

Use this to enable build flags for your instance on the command line, rather than having to create separate instances for every option when experimenting.

Parameters:	parser (argparse.ArgumentParser) – the argument parser to extend

dependencies(self)[source]¶

Specify dependencies that should be built and installed in the run environment before building a target with this instance. Called before configure() and prepare_run().

Return type:	typing.Iterator[infra.package.Package]

configure(self, ctx)[source]¶

Modify context variables to change how a target is built.

Typically, this would set one or more of ctx.{cc,cxx,cflags,cxxflags,ldflags,hooks.post_build}. It is recommended to use += rather than = when assigning to lists in the context to avoid undoing changes by dependencies.

Any custom command-line arguments set by add_build_args() are available here in ctx.args.

Parameters:	ctx (util.Namespace) – the configuration context

prepare_run(self, ctx)[source]¶

Modify context variables to change how a target is run.

Typically, this would change ctx.runenv, e.g., by setting ctx.runenv.LD_LIBRARY_PATH. Target.run() is expected to call util.run() which will inherit the modified environment.

Parameters:	ctx (util.Namespace) – the configuration context

Packages¶

class infra.Package(*args, **kwargs)[source]¶

Abstract base class for package definitions. Built-in derived classes are listed here.

Each package must define a ident() method that returns a unique ID for the package instance. This is similar to Target.name, except that each instantiation of a package can return a different ID depending on its parameters. For example a Bash package might be initialized with a version number and be identified as bash-4.1 and bash-4.3, which are different packages with different build directories.

A dependency is built in three steps:

fetch() downloads the source code, typically to build/packages/<ident>/src.
build() builds the code, typically in build/packages/<ident>/obj.
install() installs the built binaries/libraries, typically into build/packages/<ident>/install.

The functions above are only called if is_fetched(), is_built() and is_installed() return False respectively. Additionally, if is_installed() returns True, fetching and building is skipped altogether. All these methods are abstract and thus require an implementation in a pacakge definition.

clean() removes all generated package files when the clean command is run. By default, this removes build/packages/<ident>.

The package needs to be able to install itself into ctx.runenv so that it can be used by targets/instances/packages that depend on it. This is done by install_env(), which by default adds build/packages/<ident>/install/bin to the PATH and build/packages/<ident>/install/lib to the LD_LIBRARY_PATH.

Finally, the setup script has a pkg-config command that prints package information such as the installation prefix of compilation flags required to build software that uses the package. These options are configured by pkg_config_options().

ident(self)[source]¶

Returns a unique identifier to this package instantiation.

Two packages are considered identical if their identifiers are equal. This means that if multiple targets/instances/packages return different instantiations of a package as dependency that share the same identifier, they are assumed to be equal and only the first will be built. This way, different implementations of dependencies() can instantiate the same class in order to share a dependency.

Return type:	str

dependencies(self)[source]¶

Specify dependencies that should be built and installed in the run environment before building this package.

Return type:	typing.Iterator[ForwardRef(‘Package’)]

is_fetched(self, ctx)[source]¶

Returns True if fetch() should be called before building.

Parameters:	ctx (util.Namespace) – the configuration context
Return type:	bool

is_built(self, ctx)[source]¶

Returns True if build() should be called before installing.

Parameters:	ctx (util.Namespace) – the configuration context
Return type:	bool

is_installed(self, ctx)[source]¶

Returns True if the pacakge has not been installed yet, and thus needs to be fetched, built and installed.

Parameters:	ctx (util.Namespace) – the configuration context
Return type:	bool

fetch(self, ctx)[source]¶

Fetches the source code for this package. This step is separated from build() because the build command first fetches all packages and targets before starting the build process.

Parameters:	ctx (util.Namespace) – the configuration context

build(self, ctx)[source]¶

Build the package. Usually amounts to running make -j<ctx.jobs> using util.run().

Parameters:	ctx (util.Namespace) – the configuration context

install(self, ctx)[source]¶

Install the package. Usually amounts to running make install using util.run(). It is recommended to install to self.path(ctx, 'install'), which results in build/packages/<ident>/install. Assuming that a bin and/or lib directories are generated in the install directory, the default behaviour of install_env() will automatically add those to [LD_LIBRARY_]PATH.

Parameters:	ctx (util.Namespace) – the configuration context

is_clean(self, ctx)[source]¶

Returns True if clean() should be called before cleaning.

Parameters:	ctx (util.Namespace) – the configuration context
Return type:	bool

clean(self, ctx)[source]¶

Clean generated files for this target, called by the clean command. By default, this removes build/packages/<ident>.

Parameters:	ctx (util.Namespace) – the configuration context

path(self, ctx, *args)[source]¶

Get the absolute path to the build directory of this package, optionally suffixed with a subpath.

Parameters:	ctx (util.Namespace) – the configuration context args (typing.Iterable[str]) – additional subpath to pass to `os.path.join()`
Returns:	`build/packages/<ident>[/<subpath>]`
Return type:	str

install_env(self, ctx)[source]¶

Install the package into ctx.runenv so that it can be used in subsequent calls to util.run(). By default, it adds build/packages/<ident>/install/bin to the PATH and build/packages/<ident>/install/lib to the LD_LIBRARY_PATH (but only if the directories exist).

Parameters:	ctx (util.Namespace) – the configuration context

pkg_config_options(self, ctx)[source]¶

Yield options for the pkg-config command. Each option is an (option, description, value) triple. The defaults are --root which returns the root directory build/packages/<ident>, and --prefix which returns the install directory populated by install(): build/packages/<ident>/install.

When reimplementing this method in a derived package class, it is recommended to end the implementation with yield from super().pkg_config_options(ctx) to add the two default options.

Parameters:	ctx (util.Namespace) – the configuration context
Return type:	typing.Iterator[typing.Tuple[str, str, str]]

Utility functions¶

class infra.util.Namespace(*args, **kwargs)[source]¶

A dictionary in which keys can be accessed as attributes, i.e., ns.key is the same as ns['key']. Used for the context (see Setup).

copy(self)[source]¶

Make a deepcopy of this namespace, but only copy values with type Namespace|list|dict.

Return type:	Namespace

join_paths(self)[source]¶

Create a new namespace in which all lists/tuples of strings are replaced with ':'.join(list_or_tuple). Used by run() to squash lists of paths in environment variables.

Return type:	Namespace

infra.util.add_cflag(ctx, flag)[source]¶

Add flag to ctx.cflags if new

infra.util.add_cxxflag(ctx, flag)[source]¶

Add flag to ctx.cxxflags if new

infra.util.add_c_cxxflag(ctx, flag)[source]¶

Add a flag both to ctx.cflags & ctx.cxxflags if new

infra.util.add_cppflag(ctx, flag)[source]¶

Add flag to ctx.cppflags if new

infra.util.add_ldflag(ctx, flag)[source]¶

Add flag to ctx.ldflags if new

infra.util.add_lib_ldflag(ctx, flag, also_ldflag=False)[source]¶

Add flag to ctx.lib_ldflags if new

infra.util.add_ldlib(ctx, lib_name)[source]¶

Add library to link (stripped & -l added) to ctx.ldlibs if new

exception infra.util.FatalError[source]¶: Raised for errors that should stop the execution immediately, but do not need a backtrace. Results in only the exception message being logged. This typically means there is an error in the user input, rather than in the code that raises the error.

infra.util.apply_patch(ctx, path, strip_count)[source]¶

Applies a patch in the current directory by calling patch -p<strip_count> < <path>.

Afterwards, a stamp file called .patched-<basename> is created to indicate that the patch has been applied. If the stamp file is already present, the patch is not applied at all. <basename> is generated from the patch file name: path/to/my-patch.patch becomes my-patch.

Parameters:	ctx (util.Namespace) – the configuration context path (str) – path to the patch file strip_count (int) – number of leading elements to strip from patch paths
Returns:	`True` if the patch was applied, `False` if it was already applied before
Return type:	bool

infra.util.run(ctx, cmd, allow_error=False, silent=False, teeout=False, defer=False, env={}, **kwargs)[source]¶

Wrapper for subprocess.run() that does environment/output logging and provides a few useful options. The log file is build/log/commands.txt. Where possible, use this wrapper in favor of subprocess.run() to facilitate easier debugging.

It is useful to permanently have a terminal window open running tail -f build/log/commands.txt, This way, command output is available in case of errors but does not clobber the setup’s progress log.

The run environment is based on os.environ, first adding ctx.runenv (populated by packages/instances, see also Setup) and then the env parameter. The combination of ctx.runenv and env is logged to the log file. Any lists of strings in environment values are joined with a ‘:’ separator using Namespace.join_paths().

If the command exits with a non-zero status code, the corresponding output is logged to the command line and the process is killed with sys.exit(-1).

Parameters:	ctx (util.Namespace) – the configuration context cmd (str or typing.List[str]) – command to run, can be a string or a list of strings like in `subprocess.run()` allow_error (bool) – avoids calling `sys.exit(-1)` if the command returns an error silent (bool) – disables output logging (only logs the invocation and environment) teeout (bool) – streams command output to `sys.stdout` as well as to the log file defer (bool) – Do not wait for the command to finish. Similar to `./program &` in Bash. Returns a `subprocess.Popen` instance. env (typing.Dict[str, typing.Union[str, typing.List[str]]]) – variables to add to the environment kwargs – passed directly to `subprocess.run()` (or `subprocess.Popen` if `defer==True`)
Returns:	a handle to the completed or running process
Return type:	subprocess.CompletedProcess or subprocess.Popen

infra.util.qjoin(args)[source]¶

Join the command-line arguments to a single string to make it safe to pass to paste in a shell. Basically this adds quotes to each element containing spaces (uses shlex.quote()). Arguments are stringified by str before joining.

Parameters:	args (typing.Iterable[typing.Any]) – arguments to join
Return type:	str

infra.util.download(ctx, url, outfile=None)[source]¶

Download a file (logs to the debug log).

Parameters:	ctx (util.Namespace) – the configuration context url (str) – URL to the file to download outfile (str or None) – optional path/filename to download to

infra.util.param_attrs(constructor)[source]¶

Decorator for class constructors that sets parameter values as object attributes:

>>> class Foo:
...     @param_attrs
...     def __init__(self, a, b=1, *, c=True):
...         pass

>>> foo = Foo('a')
>>> foo.a
'a'
>>> foo.b
1
>>> foo.c
True

Parameters:	constructor (typing.Callable) – the `__init__` method being decorated
Return type:	typing.Callable

infra.util.require_program(ctx, name, error=None)[source]¶

Require a program to be available in PATH or ctx.runenv.PATH.

Parameters:	ctx (util.Namespace) – the configuration context name (str) – name of required program error (str or None) – optional error message
Raises:	FatalError – if program is not found

infra.util.untar(ctx, tarname, dest=None, *, remove=True, basename=None)[source]¶

TODO: docs

class infra.parallel.Pool(logger, parallelmax)[source]¶

A pool is used to run processes in parallel as jobs when --parallel is specified on the command line. The pool is created automatically by Setup and passed to Target.build(), Target.link() and Target.run(). However, the pool is only passed if the method implementation defines a parameter for the pool, i.e.:

class MyTarget(Target):
    def build(self, ctx, instance, pool): # receives Pool instance
       ...
    def run(self, ctx, instance):         # does not receive it
       ...

The maximum number of parallel jobs is controlled by --parallelmax. For --parallel=proc this is simply the number of parallel processes on the current machine. For --parallel=prun it is the maximum number of simultaneous jobs in the job queue (pending or running).

Parameters:	logger (logging.Logger) – logging object for status updates (set to `ctx.log`) parallelmax (int) – value of `--parallelmax`

wait_all(self)[source]¶: Block (busy-wait) until all jobs in the queue have been completed. Called automatically by Setup after the build and run commands.

run(self, ctx, cmd, jobid, outfile, nnodes, onsuccess=None, onerror=None, **kwargs)[source]¶

A non-blocking wrapper for util.run(), to be used when --parallel is specified.

Parameters:	ctx (util.Namespace) – the configuration context cmd (str or typing.List[str]) – the command to run jobid (str) – a human-readable ID for status reporting outfile (str) – full path to target file for command output nnodes (int) – number of cores or machines to run the command on onsuccess (typing.Callable[[subprocess.Popen], NoneType] or None) – callback when the job finishes successfully onerror (typing.Callable[[subprocess.Popen], NoneType] or None) – callback when the job exits with (typically I/O) error kwargs – passed directly to `util.run()`
Returns:	handles to created job processes
Return type:	typing.List[subprocess.Popen]