efrotools package

Build/tool functionality shared between all Efro’s projects.

This stuff can be a bit more sloppy/loosey-goosey since it is not used in live client or server code.

Submodules

efrotools.android module

Functionality related to android builds.

class efrotools.android.GradleFilterSection(tag: str, firstline: int, lastline: int)

Bases: object

Filtered section of gradle file.

firstline: int

lastline: int

tag: str

efrotools.android.filter_gradle_file(buildfilename: str, enabled_tags: set[str]) → None

Filter EFRO_IF sections in a gradle file.

efrotools.buildfile module

Generate Jenkins ‘build-files’: scheduled lists of shell commands.

A build-file is simply a newline-delimited list of shell commands that a Jenkins stage runs one per line (readFile(...).split("\n") then a sh per entry). This module provides the shared machinery for emitting such files where individual commands run only on an interval (every Nth day) so expensive or rarely-needed work can be spread out instead of run every nightly pass.

The key property — borrowed from efrohome’s machine-upkeep generator — is that every command is emitted on every run: a command that is not due today is emitted as an echo "skipping action ..." line instead of being silently omitted. That keeps each run’s log a full inventory of the configured work, showing at a glance what ran, what was skipped, and how many days remain until each skipped item is next due.

Selection is purely date-deterministic (days since the Unix epoch modulo the command’s interval); there is no persistent ‘last run’ state. A missed nightly run simply means a command waits until its next due day.

class efrotools.buildfile.ScheduledCommand(command: str, interval: int = 1, label: str = '', phase: int = 0)

Bases: object

A single command in a scheduled build-file.

The command is emitted verbatim on days when it is due (that is, when days_since_epoch() % interval == phase) and as a ‘skipping’ echo line on all other days.

command: str

The shell line emitted when the command is due. The caller is responsible for it being a valid standalone shell line (quoting, wrappers, etc.) exactly as it would appear in the build-file.

interval: int = 1

Run cadence in days. 1 (the default) runs every pass; 7 runs roughly once a week, etc. Must be >= 1.

label: str = ''

Human-readable description used in the ‘skipping’ echo line. When empty, command is used. Useful when command is a long wrapped invocation but the skip line should show something terse.

phase: int = 0

Offset within the interval cycle on which the command is due, i.e. it runs when days_since_epoch() % interval == phase. Defaults to 0. Use distinct phases to spread several same-interval commands across different days so per-run cost stays bounded (e.g. ten interval=14 builds with phases 0..9 each run once per fortnight, never all on the same night). Must satisfy 0 <= phase < interval.

efrotools.buildfile.days_since_epoch() → int

Return the number of whole days since the Unix epoch (UTC).

This is the date counter used for interval scheduling. Using days-since-epoch (rather than day-of-year) means cadences never hiccup at year boundaries.

efrotools.buildfile.gen_buildfile_lines(commands: Sequence[ScheduledCommand], *, day: int | None = None) → list[str]

Return build-file lines for commands for the given day.

Due commands contribute their command verbatim; not-due commands contribute a skipping action echo naming their cadence position. day defaults to days_since_epoch() (computed once so the whole file reflects a single day).

efrotools.buildfile.schedule_dayindex(interval: int, *, day: int | None = None) → int

Return the position of day within a command’s interval cycle.

A result of 0 means the command is due; any other value is the number of days into the cycle (and interval - value days remain until next due). day defaults to days_since_epoch().

efrotools.buildfile.write_buildfile(path: str, commands: Sequence[ScheduledCommand], *, day: int | None = None, empty_message: str = 'No actions for this target. Nothing to see here.') → None

Write a scheduled build-file to path.

When commands is empty, a single echo of empty_message is written instead so the consuming Jenkins stage always has a valid line to run rather than an empty (and thus malformed) sh step.

efrotools.buildlock module

A system for sanity testing parallel build isolation.

class efrotools.buildlock.BuildLock(name: str, projroot: str)

Bases: object

Tries to ensure a build is not getting stomped on/etc.

efrotools.code module

Functionality for formatting, linting, etc. code.

efrotools.code.black_base_args(projroot: Path) → list[str]

Build base args for running black Python formatting.

efrotools.code.check_android_studio(projroot: Path, full: bool, verbose: bool) → None

Run Android Studio inspections on all our code.

efrotools.code.check_cpplint(projroot: Path, full: bool) → None

Run cpplint on all our applicable code.

efrotools.code.dmypy(projroot: Path) → None

Type check all of our scripts using mypy in daemon mode.

efrotools.code.format_cpp_str(projroot: Path, text: str, filename: str = 'untitled.cc') → str

Run clang-format inline on c++ code.

Note that some cpp formatting keys off the filename, so a fake one can be optionally provided.

efrotools.code.format_project_cpp_files(projroot: Path, full: bool) → None

Run clang-format on all of our source code (multithreaded).

efrotools.code.format_project_python_files(projroot: Path, full: bool) → None

Runs formatting on all of our Python code.

efrotools.code.format_python_files(projroot: Path, filenames: list[str], *, capture: bool = False) → dict[str, Any] | None

Format a specific list of Python files in place via black.

Parallel to pylint_files() and mypy_files(): the filenames are exactly what black runs on, no project-config / blacklist filtering applied (callers that want that should pre-filter). No FileCache layer either — black’s own internal cache (--cache) handles “already-formatted file” skips, and ad-hoc consumers (the workspace-check runner) don’t share a stable cache path across runs anyway.

capture=True returns a result dict with stdout/stderr/returncode and suppresses the raise on non-zero exit, mirroring the capture mode that pylint_files / mypy_files callers rely on for programmatic consumption.

efrotools.code.format_python_str(projroot: Path | str, code: str) → str

Run our Python formatting on the provided inline code.

efrotools.code.get_code_filenames(projroot: Path, include_generated: bool) → list[str]

Return the list of files to lint-check or auto-format.

Be sure to pass False for include_generated if performing any operation that can modify files (such as formatting). Otherwise it could cause dirty generated files to not get updated properly when their sources change).

efrotools.code.get_script_filenames(projroot: Path) → list[str]

Return the Python filenames to lint-check or auto-format.

efrotools.code.mypy(projroot: Path, full: bool, output_format: str = 'text') → None

Type check all of our scripts using mypy.

output_format='json' requests mypy’s structured NDJSON output (see mypy_files()). Human-readable progress prints are suppressed in that mode so the JSON stream isn’t corrupted.

efrotools.code.mypy_files(projroot: Path, filenames: list[str], full: bool = False, check: bool = True, output_format: str = 'text', *, cwd: Path | str | None = None, env: dict[str, str] | None = None, cache_dir: Path | str | None = None, capture: bool = False) → CompletedProcess[str] | None

Run MyPy on provided filenames.

output_format selects the report format:

• 'text' (default) — mypy’s --pretty colorized text output with no summary line. Suitable for terminal/CI consumers.

• 'json' — mypy’s structured --output=json NDJSON output. Each diagnostic carries file/line/column/end_line/ end_column/severity/code/message (--show-error-end is added in this mode for editor span highlighting); the caller parses it and uses the non-zero exit code as a “had errors” signal.

Keyword-only knobs (default to inheriting from this process, which is the in-tree-make mypy use case):

• cwd / env — passed through to subprocess.run(). Needed by consumers that run mypy against files outside projroot (e.g. workspace-check runners staging user code into a per-workspace cache dir).

• cache_dir — sets --cache-dir. By default mypy uses .mypy_cache relative to its cwd; per-consumer cache dirs let one process drive multiple isolated cache lifecycles.

• capture — when true, return a CompletedProcess with stdout and stderr captured (text mode). When false (default), output goes to the parent process’s terminals and None is returned. JSON-mode consumers typically want capture=True.

efrotools.code.pylint(projroot: Path, full: bool, fast: bool, extra: bool, nocache: bool = False, output_format: str = 'text') → None

Run Pylint on all scripts in our project (with smart dep tracking).

nocache=True skips the FileCache + dirty-file dep tracking entirely and lints every file every time. Used by the standalone check-environment (which is freshly extracted on each run, so persisted cache state would be meaningless), and any caller where determinism matters more than incremental speed.

output_format='json' requests pylint’s json2 structured report on stdout. Human-readable progress prints are suppressed in that mode so the JSON stream isn’t corrupted.

Thin wrapper around pylint_files() that derives the file list from the project root via get_script_filenames(). Other consumers (e.g. workspace-check runners that lint a specific list of files) should call pylint_files() directly with explicit filenames / cache_path.

efrotools.code.pylint_files(pylintrc: Path | str, filenames: list[str], *, projroot: Path, cache_path: Path | None = None, fast: bool = False, extra: bool = False, output_format: str = 'text', capture: bool = False) → dict[str, Any] | None

Lint a specific list of files with optional dep-tracking cache.

The orchestration layer between callers (which know what to lint and where to cache state) and the inner _run_pylint (which runs pylint itself). Used by:

• pylint() — the in-tree make pylint path, with the cache rooted at <projroot>/.cache/check_pylint{_fast} and file list from get_script_filenames().

• Workspace-check runners and other dynamic-input callers, with the cache rooted at a consumer-supplied path and an explicit filenames list.

Parameters mostly mirror pylint():

• cache_path — None for nocache mode (lint everything every call); a writable path enables the FileCache-backed dirty-dep-tracking layer described in pylint()’s docstring.

• projroot — currently unused by the cache-apply step (external deps are tracked by resolved path+mtime, not by a projectconfig allowlist); reserved for future use.

• capture — when true, returns the inner result dict (stdout, msg_status, etc.) instead of printing to this process’s stdout; see the inner _run_pylint for details.

Returns the inner-call result dict when capture=True (with stdout, msg_status, etc.), or None in text mode.

efrotools.code.runpylint(projroot: Path, filenames: list[str], extra: bool, output_format: str = 'text') → None

Run Pylint explicitly on files.

output_format selects 'text' (default human-readable) or 'json' (structured json2 report on stdout).

efrotools.code.sort_jetbrains_dict(original: str) → str

Given jetbrains dict contents, sort it the way jetbrains would.

efrotools.code.zmypy(projroot: Path, full: bool) → None

Type check all of our scripts using mypy.

efrotools.code.zmypy_files(projroot: Path, filenames: list[str], full: bool = False, check: bool = True) → None

Run zuban mypy on provided filenames.

efrotools.efrocache module

A simple cloud caching system for making built binaries & assets.

The basic idea here is the ballistica-internal project can flag file targets in its Makefiles as ‘cached’, and the public version of those Makefiles will be filtered to contain cache downloads in place of the original build commands. Cached files are gathered and uploaded as part of the pubsync process.

class efrotools.efrocache.CacheMetadata(executable: bool)

Bases: object

Metadata stored with a cache file.

executable: bool

efrotools.efrocache.filter_makefile(makefile_dir: str, contents: str) → str

Filter makefile contents to use efrocache lookups.

efrotools.efrocache.get_existing_file_hash(path: str) → str

Return the hash used for caching.

efrotools.efrocache.get_local_cache_dir() → str

Where we store local efrocache files we’ve downloaded.

Rebuilds will be able to access the local cache instead of re-downloading. By default each project has its own cache dir but this can be shared between projects by setting the EFROCACHE_DIR environment variable.

efrotools.efrocache.get_repository_base_url() → str

Return the base repository url (assumes cwd is project root).

efrotools.efrocache.get_target(path: str, batch: bool, clr: type[efro.terminal.ClrBase]) → str

Fetch a target path from the cache, downloading if need be.

efrotools.efrocache.update_cache(makefile_dirs: list[str]) → None

Given a list of directories containing Makefiles, update caches.

efrotools.efrocache.warm_start_cache(cachetype: str) → None

Run a pre-pass on the efrocache to improve efficiency.

This may fetch an initial cache archive, batch update mod times to reflect new cache maps, etc.

efrotools.efrosync module

Centralized file synchronization across local repos.

A simplified replacement for the original efrosync system. Instead of embedding hash markers in synced files and using an upstream/downstream model, this system:

• Treats all repo copies as peers (no upstream/downstream).

• Stores hashes and state in a central ~/.efrosync/ directory.

• Supports glob patterns for worktree directories.

• Syncs when exactly one copy has changed; errors on ambiguity.

• Only ever updates the contents of existing files; it never creates or deletes them. Adding a shared file means manually placing a copy in every repo (after which syncs manage it); removing one means manually deleting it everywhere. This is deliberate: silently materializing or deleting files in repos the user may be actively working in would be far more surprising than the one-time manual setup, so presence mismatches are reported as errors and the placement/removal is left to the user.

class efrotools.efrosync.Config(repos: dict[str, RepoConfig], sync_groups: list[SyncGroupConfig])

Bases: object

Top-level efrosync config.

repos: dict[str, RepoConfig]

sync_groups: list[SyncGroupConfig]

class efrotools.efrosync.FileStateEntry(synced_hash: str, mtimes: dict[str, float]=<factory>)

Bases: object

Per-file sync state.

mtimes: dict[str, float]

synced_hash: str

class efrotools.efrosync.RepoConfig(path: str, worktree_globs: list[str] = <factory>)

Bases: object

A repo participating in sync.

path: str

worktree_globs: list[str]

class efrotools.efrosync.SyncGroupConfig(path: str, repos: list[str] = <factory>, repo_path_overrides: dict[str, str]=<factory>)

Bases: object

A directory or file to keep in sync across repos.

path: str

repo_path_overrides: dict[str, str]

repos: list[str]

class efrotools.efrosync.SyncLocation(label: str, abs_path: str)

Bases: object

A resolved location for a sync group.

abs_path: str

label: str

class efrotools.efrosync.SyncLock

Bases: object

File-based lock to prevent concurrent syncs.

class efrotools.efrosync.SyncState(files: dict[str, ~efrotools.efrosync.FileStateEntry]=<factory>)

Bases: object

All stored sync state.

files: dict[str, FileStateEntry]

efrotools.efrosync.efrosync_main() → None

Entry point for the efrosync pcommand.

efrotools.efrosync.load_config() → Config | None

Load config from ~/.efrosync/config.json.

Returns None if no config file exists (efrosync is not configured on this machine).

efrotools.efrosync.load_state() → SyncState

Load state from ~/.efrosync/state.json.

efrotools.efrosync.run_efrosync(*, dry_run: bool = False, check: bool = False) → None

Run the efrosync process.

efrotools.efrosync.save_state(state: SyncState) → None

Save state to ~/.efrosync/state.json.

efrotools.emacs module

Stuff intended to be used from emacs

efrotools.emacs.py_examine(projroot: Path, filename: Path, line: int, column: int, selection: str | None, operation: str) → None

Given file position info, performs some code inspection.

efrotools.filecache module

Provides a system for caching linting/formatting operations.

class efrotools.filecache.FileCache(path: Path)

Bases: object

A cache of file hashes/etc. used in linting/formatting/etc.

get_dirty_files() → Sequence[str]

Return paths for all entries with no hash value.

mark_clean(files: Sequence[str]) → None

Marks provided files as up to date.

update(filenames: Sequence[str], extrahash: str) → None

Update the cache for the provided files and hash type.

Hashes will be checked for all files (incorporating extrahash) and mismatched hash values cleared. Entries for no-longer-existing files will be cleared as well.

write() → None

Writes the state back to its file.

efrotools.filecommand module

Operate on large sets of files efficiently.

efrotools.filecommand.file_batches(paths: list[str], batch_size: int = 1, file_filter: Callable[[str], bool] | None = None, include_mac_packages: bool = False) → Iterable[list[str]]

Efficiently yield batches of files to operate on.

Accepts a list of paths which can be files or directories to be recursed. The batch lists are buffered in a background thread so time-consuming synchronous operations on the returned batches will not slow the gather.

efrotools.genwrapper module

Functionality related to android builds.

efrotools.ios module

Tools related to ios development.

class efrotools.ios.Config(product_name: str, projectpath: str, scheme: str)

Bases: object

Configuration values for this project.

product_name: str

projectpath: str

scheme: str

class efrotools.ios.LocalConfig(sftp_host: str, sftp_dir: str)

Bases: object

Configuration values specific to the machine.

sftp_dir: str

sftp_host: str

efrotools.ios.push_ipa(root: Path, modename: str, signing_config: str | None) → None

Construct ios IPA and push it to staging server for device testing.

This takes some shortcuts to minimize turnaround time; It doesn’t recreate the ipa completely each run, uses rsync for speedy pushes to the staging server, etc. The use case for this is quick build iteration on a device that is not physically near the build machine.

efrotools.jsontools module

Json related tools functionality.

class efrotools.jsontools.NoIndent(value: Any)

Bases: object

Used to prevent indenting in our custom json encoder.

Wrap values in this before passing to encoder and all child values will be a single line in the json output.

class efrotools.jsontools.NoIndentEncoder(*args: Any, **kwargs: Any)

Bases: JSONEncoder

Our custom encoder implementing selective indentation.

default(o: Any) → Any

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return super().default(o)

encode(o: Any) → Any

Return a JSON string representation of a Python data structure.

>>> from json.encoder import JSONEncoder
>>> JSONEncoder().encode({"foo": ["bar", "baz"]})
'{"foo": ["bar", "baz"]}'

efrotools.lazybuild module

Functionality used for building.

class efrotools.lazybuild.LazyBuildContext(target: str, srcpaths: list[str], command: str, *, buildlockname: str | None = None, dirfilter: Callable[[str, str], bool] | None = None, filefilter: Callable[[str, str], bool] | None = None, srcpaths_fullclean: list[str] | None = None, srcpaths_exist: list[str] | None = None, manifest_file: str | None = None, command_fullclean: str | None = None, force: bool = False)

Bases: object

Run a build if anything in some category is newer than a target.

This can be used as an optimization for build targets that always run. As an example, a target that spins up a VM and runs a build can be expensive even if the VM build process determines that nothing has changed and does no work. We can use this to examine a broad swath of source files and skip firing up the VM if nothing has changed. We can be overly broad in the sources we look at since the worst result of a false positive change is the VM spinning up and determining that no actual inputs have changed. We could recreate this mechanism purely in the Makefile, but large numbers of target sources can add significant overhead each time the Makefile is invoked; in our case the cost is only incurred when a build is triggered.

Note that target’s mod-time will always be updated to match the newest source regardless of whether the build itself was triggered.

run() → None

Do the thing.

efrotools.makefile module

Tools for parsing/filtering makefiles.

class efrotools.makefile.Makefile(contents: str)

Bases: object

Represents an entire Makefile.

find_assigns(name: str) → list[tuple[Section, int]]

Return section/index pairs for paragraphs containing an assign.

Note that the paragraph may contain other statements as well.

find_targets(name: str) → list[tuple[Section, int]]

Return section/index pairs for paragraphs containing a target.

Note that the paragraph may contain other statements as well.

get_output() → str

Generate a Makefile from the current state.

header_line_empty = '#                                                                              #'

header_line_full = '################################################################################'

class efrotools.makefile.Paragraph(contents: str)

Bases: object

Represents a continuous set of non-blank lines in a Makefile.

contents: str

get_logical_lines() → list[str]

Return contents broken into logical lines.

Lines joined by continuation chars are considered a single line.

class efrotools.makefile.Section(name: str | None, paragraphs: list[Paragraph])

Bases: object

Represents a section of a Makefile.

name: str | None

paragraphs: list[Paragraph]

efrotools.message module

Message related tools functionality.

efrotools.message.standard_message_receiver_gen_pcommand(*, projroot: Path, basename: str, source_module: str, is_async: bool, get_protocol_call: str = 'get_protocol', embedded: bool = False) → None

Used by pcommands generating efro.message receiver modules.

efrotools.message.standard_message_sender_gen_pcommand(*, projroot: Path, basename: str, source_module: str, enable_sync_sends: bool, enable_async_sends: bool, get_protocol_call: str = 'get_protocol', embedded: bool = False) → None

Used by pcommands taking a single filename argument.

efrotools.openalbuildandroid module

Build OpenAL Soft (+ Oboe) static libs for Android.

Cross-compiles OpenAL Soft as a static libopenal.a (plus its Oboe backend dependency liboboe.a) for each Android ABI (armeabi-v7a / arm64-v8a / x86 / x86_64), in debug and release, using the Android NDK’s cmake toolchain. gather then installs the results into src/external/openal-android/{lib,include}.

Build host: runs locally wherever the Android NDK is installed (resolved via android_sdk_utils get-ndk-path). There is no cloud build+gather target; CI builds individual ABIs on linbeast purely as a canary (no gather).

Driving it: make openal-android-all builds all eight ABI/mode combos (one openal_android_build <arch> <mode> pcommand each), then make openal-android-gather installs them into the source tree. Per-ABI targets (openal-android-arm etc.) exist too.

Why tarballs: sources are fetched as github release tarballs, not git clones, so the build creates no nested .git – which keeps it runnable under restricted filesystem sandboxes (the build sandbox forbids creating or even deleting .git dirs). Everything lives under build/openal-android/; remove that dir to clean up. OpenAL Soft is pinned by OPENAL_SOFT_TAG, Oboe by OBOE_TAG.

Local source tweaks applied after extraction: reroute OpenAL’s Android logging through our alcSetCustomAndroidLogger hook; add a BA_OBOE_USE_OPENSLES env-var escape hatch to force Oboe’s OpenSL backend; disable Oboe’s open-a-test-stream probe; and demote OpenAL Soft 1.25.2’s -Werror=function-effects to a warning (it trips its own code under newer clang – same patch the Apple build applies).

efrotools.openalbuildandroid.build_openal(arch: str, mode: str) → None

Do the thing.

efrotools.openalbuildandroid.gather() → None

Gather the things. Assumes all have been built.

efrotools.openalbuildapple module

Build OpenAL Soft for Apple platforms (macOS / iOS / tvOS / visionOS).

Produces a single .framework-based OpenALSoft.xcframework covering macOS, iOS, tvOS and visionOS (device + simulator), plus a shared include/AL header tree (identical across every slice).

Unlike ANGLE (batools/buildangleapple.py), which keeps macOS on bare dylibs because libEGL dlopen``s ``libGLESv2 as a sibling file and the non-Xcode cmake/SDL build consumes the same binaries, OpenAL Soft is a single self-contained library with no such constraints (and the macOS cmake build uses its own static libopenal.a). So every Apple Xcode target – macOS included – links + embeds the framework slice out of this one xcframework; there is no standalone macOS dylib. OpenAL Soft is a plain CMake project, so each slice is just a cross-compiled CMake build (no gn/depot_tools), wrapped into a framework here.

Everything lives under build/openal-apple/:

• checkout/ – the OpenAL Soft git checkout (one, shared by all slices; each slice gets its own checkout/build-<slice> CMake build dir);

• artifacts/ – the assembled OpenALSoft.xcframework + include header tree that gather installs into the source tree.

Remove build/openal-apple/ to fully clean up. Host prerequisites are a full Xcode install and a system git + cmake – nothing is installed system-wide.

The naming (OpenALSoft rather than OpenAL) deliberately avoids a clash with Apple’s deprecated system OpenAL.framework; OpenAL Soft’s own cmake does the same (its framework is soft_oal).

class efrotools.openalbuildapple.Slice(name: str, system_name: str | None, archs: str, sdk: str, deployment_target: str, versioned: bool = False)

Bases: object

One Apple build target: an (os, arch(es), sdk) combination.

Each slice is a single CMake cross-compile producing one shared libopenal dylib that becomes one framework slice of the xcframework. system_name is the CMake CMAKE_SYSTEM_NAME (None => a native macOS build, where we go universal in a single pass).

archs: str

deployment_target: str

name: str

sdk: str

system_name: str | None

versioned: bool = False

efrotools.openalbuildapple.build(projroot: str) → None

Build all Apple OpenAL Soft slices from scratch.

Downloads OpenAL Soft at the pinned tag and builds every slice, assembling OpenALSoft.xcframework + headers into build/openal-apple/artifacts. Follow with make openal-apple-gather to install into the source tree.

efrotools.openalbuildapple.gather(projroot: str) → None

Install assembled artifacts into src/external/openal-apple.

Installs the OpenALSoft.xcframework (macOS/iOS/tvOS/visionOS framework slices) and the shared include/AL header tree.

efrotools.openalbuildapple.test_build(projroot: str) → None

Alias for build() – a CI-canary entry point.

Identical to build (kept as a separate name mirroring the ANGLE *-test-* CI targets). OpenAL’s build is cheap enough that there’s no separate lazy/incremental variant.

efrotools.pcommand module

Standard snippets that can be pulled into project pcommand scripts.

A snippet is a mini-program that directly takes input from stdin and does some focused task. This module is a repository of common snippets that can be imported into projects’ pcommand script for easy reuse.

efrotools.pcommand.clientprint(*args: Any, stderr: bool = False, end: str | None = None) → None

Print to client stdout.

Note that, in batch mode, the results of all clientprints will show up only after the command completes. In regular mode, clientprint() simply passes through to regular print().

efrotools.pcommand.clr() → type[ClrBase]

Like efro.terminal.Clr but for use with pcommand.clientprint().

This properly colorizes or doesn’t colorize based on whether the client where output will be displayed is running on a terminal. Regular print() output should still use efro.terminal.Clr for this purpose.

efrotools.pcommand.enter_batch_server_mode() → None

Called by pcommandserver when we start serving.

efrotools.pcommand.get_args() → list[str]

Return the args for the current pcommand.

efrotools.pcommand.is_batch() → bool

Is the current pcommand running under a batch server?

Commands that do things that are unsafe to do in server mode such as chdir should assert that this is not true.

efrotools.pcommand.pcommand_main(globs: dict[str, Any]) → None

Main entry point to pcommand scripts.

We simply look for all public functions in the provided module globals and call the one corresponding to the first passed arg.

efrotools.pcommand.run_client_pcommand(args: list[str], clrtp: type[ClrBase], logpath: str) → tuple[int, str, str]

Call a pcommand function as a server.

Returns a result code and stdout output.

efrotools.pcommandbatch module

Wrangles pcommandbatch; an efficient way to run small pcommands.

The whole purpose of pcommand is to be a lightweight way to run small snippets of Python to do bits of work in a project. The pcommand script tries to minimize imports and work done in order to keep runtime as short as possible. However, even an ‘empty’ pcommand still takes a fraction of a second due to the time needed to spin up Python and import a minimal set of modules. This can add up for large builds where hundreds or thousands of pcommands are being run.

To help fight that problem, pcommandbatch introduces a way to run pcommands by submitting requests to temporary local server daemons. This allows individual pcommand calls to go through a lightweight client binary that simply forwards the command to a running server. This cuts minimum pcommand runtimes down greatly. Building and managing the server and client are handled automatically, and systems which are unable to compile a client binary can fall back to using vanilla pcommand in those cases.

A few considerations must be made when using pcommandbatch. Guidelines for batch-friendly pcommands follow:

• Batch mode runs parallel pcommands in different background threads and may process thousands of commands in a single process. Batch-friendly pcommands must behave reasonably in such an environment.

• Batch-enabled pcommands must not call os.chdir() or sys.exit() or anything else having global effects. This should be self-explanatory considering the shared server model in use.

• Batch-enabled pcommands must not use environment-variables to influence their behavior. In batch mode this would unintuitively use the environment of the server and not of the client.

• Batch-enabled pcommands should not look at sys.argv. They should instead use pcommand.get_args(). Be aware that this value does not include the first two values from sys.argv (executable path and pcommand name) so is generally cleaner to use anyway. Also be aware that args are thread-local, so only call get_args() from the thread your pcommand was called in.

• Batch-enabled pcommands should not use efro.terminal.Clr for coloring terminal output; instead they should use pcommand.clr() which properly takes into account whether the client is running on a tty/etc.

• Standard print and log calls (as well as those of child processes) will wind up in the pcommandbatch server log and will not be seen by the user or capturable by the calling process. For batch-friendly printing, use pcommand.clientprint(). Note that, in batch mode, all output will be printed on the client after the command completes and stderr and stdout will be printed separately instead of intermingled. If a pcommand is long-running and prints at multiple times while doing its thing, it is probably not a good fit for batch-mode.

exception efrotools.pcommandbatch.IdleError

Bases: RuntimeError

Error we raise to quit peacefully.

class efrotools.pcommandbatch.Server(idle_timeout_secs: int, project_dir: str, instance: str, daemon: bool)

Bases: object

A server that handles requests from pcommandbatch clients.

run() → None

Do the thing.

efrotools.pcommandbatch.batchserver(idle_timeout_secs: int, project_dir: str, instance: str) → None

Run a server for handling batches of pcommands.

If a matching instance is already running, is a no-op.

efrotools.pcommandbatch.build_pcommandbatch(inpaths: list[str], outpath: str) → None

Create the binary or link regular pcommand.

efrotools.pcommands module

A set of lovely pcommands ready for use.

efrotools.pcommands.androidstudiocode() → None

Run Android Studio checks on our code.

efrotools.pcommands.apply_venv_patches() → None

Apply patches listed in pconfig/venv_patches.json to the venv.

Run this after package install on freshly-built venvs (uv pip install in current Makefile flows; historically pip install). Each patch is a literal-string find-and-replace against a file in site-packages. See efro.venvpatch for the schema.

Args (positional, all optional):

patches_path: path to the patches JSON. Default

pconfig/venv_patches.json. If the file does not exist, this command is a clean no-op (so projects without any patches don’t need to opt out explicitly).

--no-error: if present, mismatched / missing patches are

logged and skipped instead of erroring out. Use this on production-runtime installer flows where a partial venv beats a failed boot. Dev make env should leave the default error-on-mismatch behaviour intact.

efrotools.pcommands.check_clean_safety() → None

Ensure all files are are added to git or in gitignore.

Use to avoid losing work if we accidentally do a clean without adding something.

efrotools.pcommands.check_venv_patches() → None

Verify pconfig/venv_patches.json patches are applied.

Prints a summary and exits non-zero if any mismatch is found. See efro.venvpatch.check_patches() for log details — each mismatch is logged at CRITICAL.

efrotools.pcommands.compile_collision_mesh_file() → None

Compile a mesh file.

efrotools.pcommands.compile_font_file() → None

Compile a font file.

efrotools.pcommands.compile_language_file() → None

Compile a language file.

efrotools.pcommands.compile_mesh_file() → None

Compile a mesh file.

efrotools.pcommands.compile_python_file() → None

Compile pyc files for packaging.

This creates hash-based PYC files in opt level 1 with hash checks defaulting to off, so we don’t have to worry about timestamps or loading speed hits due to hash checks. (see PEP 552). We just need to tell modders that they’ll need to clear these cache files out or turn on debugging mode if they want to tweak the built-in scripts directly (or go through the asset build system which properly recreates the .pyc files).

efrotools.pcommands.copy_python_file() → None

Copy Python files for packaging.

efrotools.pcommands.copy_win_extra_file() → None

Copy a windows extra file.

efrotools.pcommands.cpplint() → None

Run lint-checking on all code deemed lint-able.

efrotools.pcommands.dmypy() → None

Run mypy checks on our scripts using the mypy daemon.

efrotools.pcommands.echo() → None

Echo with support for efro.terminal.Clr args (RED, GRN, BLU, etc).

Prints a Clr.RST at the end so that can be omitted.

efrotools.pcommands.efrosync() → None

Centralized file sync across local repos.

efrotools.pcommands.format_files() → None

Format the provided Python filenames in place via black.

efrotools.pcommands.formatcode() → None

Format all of our C/C++/etc. code.

efrotools.pcommands.formatmakefile() → None

Format the main makefile.

efrotools.pcommands.formatscripts() → None

Format all of our Python/etc. code.

efrotools.pcommands.gen_empty_py_init() → None

Generate an empty __init__.py for a package dir.

Used as part of codegen builds.

efrotools.pcommands.make_ensure() → None

Make sure a makefile target is up-to-date.

This can technically be done by simply make –question, but this has some extra bells and whistles such as printing some of the commands that would run. Can be useful to run after cloud-builds to ensure the local results consider themselves up-to-date.

efrotools.pcommands.make_target_debug() → None

Debug makefile src/target mod times given src and dst path.

Built to debug stubborn Makefile targets that insist on being rebuilt just after being built via a cloud target.

efrotools.pcommands.makefile_target_list() → None

Prints targets in a makefile.

Takes a single argument: a path to a Makefile.

efrotools.pcommands.mypy() → None

Run mypy checks on our scripts.

efrotools.pcommands.mypy_files() → None

Run mypy checks on provided filenames.

efrotools.pcommands.pylint() → None

Run pylint checks on our scripts.

efrotools.pcommands.pylint_files() → None

Run pylint checks on provided filenames.

efrotools.pcommands.pytest() → None

Run pytest with project environment set up properly.

efrotools.pcommands.pyver() → None

Prints the Python version used by this project.

efrotools.pcommands.requirements_upgrade() → None

Upgrade project requirements.

efrotools.pcommands.scriptfiles() → None

List project script files.

Pass -lines to use newlines as separators. The default is spaces.

efrotools.pcommands.showtime() → None

Run a command and print how long it took.

First arg is a label; remaining args are the command to run. Prints ‘<label> completed in X.XXs.’ on success (Clr.BLK) or ‘<label> failed in X.XXs.’ on failure (Clr.RED), then exits with the command’s return code.

efrotools.pcommands.tool_config_install() → None

Install a tool config file (with some filtering).

efrotools.pcommands.try_repeat() → None

Run a command with repeat attempts on failure.

First arg is the number of retries; remaining args are the command.

efrotools.pcommands.tweak_empty_py_files() → None

Find any zero-length Python files and make them length 1.

efrotools.pcommands.urandom_pretty() → None

Spits out urandom bytes formatted for source files.

efrotools.pcommands.xcodebuild() → None

Run xcodebuild with added smarts.

efrotools.pcommands.xcoderun() → None

Run an xcode build in the terminal.

efrotools.pcommands.xcodeshow() → None

Open folder containing xcode build in the finder.

efrotools.pcommands.zmypy() → None

Run zuban mypy checks on our scripts.

efrotools.pcommands.zmypy_files() → None

Run zuban mypy checks on provided filenames.

efrotools.pcommands2 module

Standard snippets that can be pulled into project pcommand scripts.

A snippet is a mini-program that directly takes input from stdin and does some focused task. This module is a repository of common snippets that can be imported into projects’ pcommand script for easy reuse.

efrotools.pcommands2.batchserver() → None

Run a server for handling pcommands.

efrotools.pcommands2.build_pcommandbatch() → None

Build a version of pcommand geared for large batches of commands.

efrotools.pcommands2.null() → None

Do nothing. Useful for speed tests and whatnot.

efrotools.pcommands2.openal_android_build() → None

Build openalsoft for android.

efrotools.pcommands2.openal_android_gather() → None

Gather built opealsoft libs into src.

efrotools.pcommands2.openal_apple_build() → None

Build OpenAL Soft xcframework + mac dylib for Apple platforms.

Re-clones OpenAL Soft at the pinned tag and builds every slice (macOS / iOS / tvOS / visionOS, device + simulator) into build/openal-apple/ artifacts. Follow with ‘make openal-apple-gather’ to install into the source tree.

efrotools.pcommands2.openal_apple_gather() → None

Install assembled Apple OpenAL Soft artifacts into the source tree.

Copies build/openal-apple/artifacts into src/external/openal-apple (the OpenALSoft.xcframework, the bare macOS dylib, and shared headers).

efrotools.pcommands2.openal_apple_test_build() → None

Build the Apple OpenAL Soft artifacts, reusing an existing checkout.

For CI / exercising the pipeline – lazily reuses build/openal-apple/ checkout (incremental). Follow with ‘make openal-apple-gather’.

efrotools.pcommands2.pcommandbatch_speed_test() → None

Test batch mode speeds.

efrotools.pcommands2.pyright() → None

Run Pyright checks on project Python code.

efrotools.pcommands2.sortlines() → None

Sort provided lines. For tidying import lists, etc.

efrotools.pcommands2.with_build_lock() → None

Run a shell command wrapped in a build-lock.

efrotools.project module

Project related functionality.

efrotools.project.get_non_public_legal_notice() → str

Return the one line legal notice we expect private repo files to have.

efrotools.project.get_non_public_legal_notice_prev() → str

Allows us to auto-update.

efrotools.project.get_public_legal_notice(style: Literal['python', 'c++', 'makefile', 'raw']) → str

Return the license notice as used for our public facing stuff.

‘style’ arg can be ‘python’, ‘c++’, or ‘makefile, or ‘raw’.

efrotools.project.getlocalconfig(projroot: Path | str) → dict[str, Any]

Return a project’s localconfig contents (or default if missing).

efrotools.project.getprojectconfig(projroot: Path | str) → dict[str, Any]

Return a project’s projectconfig contents (or default if missing).

efrotools.project.setprojectconfig(projroot: Path | str, config: dict[str, Any]) → None

Set the project config contents.

efrotools.projectchecks module

Common code checks shared by all efro projects.

Each project’s update_project pcommand should call run_common_code_checks() so cheap project-wide source sanity checks stay consistent across repos. These checks never mutate anything (so they are safe to run in both update and check modes); they simply raise efro.error.CleanError describing the first problem found.

efrotools.projectchecks.check_no_future_imports(fname: str, lines: list[str]) → None

Make sure __future__ imports don’t sneak back into a project.

We target Python 3.14+, where PEP 649 deferred annotation evaluation is the default, so from __future__ import annotations (the last meaningful future-feature) should no longer appear anywhere; it would silently switch a module back to PEP 563 stringized annotations. Note that we simply error here and never auto-remove the line; whether surrounding code relies on the old behavior requires human judgement.

A real __future__ import must be a top-level statement, so only column-zero occurrences are flagged; mentions inside (indented) docstrings or comments don’t trip this.

efrotools.projectchecks.run_common_code_checks(projroot: Path) → None

Run the standard cross-project code checks.

Operates on the same Python file set that lint/format targets use (see efrotools.code.get_script_filenames()). Projects with their own per-file scanning (such as ballistica’s ProjectUpdater) can instead call the individual per-file checks from within their existing file loops to avoid reading files twice.

efrotools.pybuild module

Functionality related to building python for ios, android, etc.

efrotools.pybuild.android_patch() → None

Run necessary patches on an android archive before building.

efrotools.pybuild.android_patch_ssl() → None

Run necessary patches on an android ssl before building.

efrotools.pybuild.build_android(rootdir: str, arch: str, debug: bool = False) → None

Run a build for android with the given architecture.

(can be arm, arm64, x86, or x86_64)

efrotools.pybuild.gather(do_android: bool) → None

Gather per-platform python headers, libs, and modules into our src.

This assumes all embeddable py builds have been run successfully, and that PROJROOT is the cwd.

efrotools.pybuild.patch_modules_setup(python_dir: str, baseplatform: str, python_version: str = '3.13') → None

Muck with the Setup.* files Python uses to build modules.

efrotools.pybuild.tweak_empty_py_files(dirpath: str) → None

Find any zero-length Python files and make them length 1

I’m finding that my jenkins server updates modtimes on all empty files when fetching updates regardless of whether anything has changed. This leads to a decent number of assets getting rebuilt when not necessary.

As a slightly-hacky-but-effective workaround I’m sticking a newline up in there.

efrotools.pybuild.winprune() → None

Prune unneeded files from windows python dists.

Should run this after dropping updated windows libs/dlls/etc into our src dirs.

efrotools.pylintplugins module

Plugins for pylint

efrotools.pylintplugins.class_annotations_filter(node: NodeNG) → NodeNG

Filter annotations in class declarations.

efrotools.pylintplugins.func_annotations_filter(node: NodeNG) → NodeNG

Filter annotated function args/retvals.

Annotations are deferred-eval’ed (PEP 649/749, the Python 3.14+ language default), so we don’t want Pylint to complain about missing symbols in annotations when they aren’t actually needed at runtime. And we strip out stuff under TYPE_CHECKING blocks which means they’d often be seen as missing.

efrotools.pylintplugins.ignore_reveal_type_call(node: NodeNG) → NodeNG

Make reveal_type() not trigger an error.

The ‘reveal_type()’ fake call is used for type debugging types with mypy and it is annoying having pylint errors pop up alongside the mypy info.

efrotools.pylintplugins.ignore_type_check_filter(if_node: NodeNG) → NodeNG

Ignore stuff under a module-level if TYPE_CHECKING: block.

Such blocks run only under static analysis, never at runtime, so we want pylint to check our code as if they don’t exist.

Note: we deliberately handle only module-level blocks. A nested if TYPE_CHECKING: (inside a function or method) can contain control-flow — a common idiom here is if TYPE_CHECKING: return <type-fiction> paired with a real runtime return. Wiping such a block to pass strips the return, and there’s no restructuring that satisfies both pylint (which sees the wiped tree) and mypy (which sees TYPE_CHECKING as True) without some suppression — so nested handling just trades one suppression for another with no real gain. Module-level blocks can’t contain control flow, so they’re unambiguously safe to wipe.

efrotools.pylintplugins.register(linter: PyLinter) → None

Unused here - we’re modifying the ast, not defining linters.

efrotools.pylintplugins.register_plugins(manager: astroid.Manager) → None

Apply our transforms to a given astroid manager object.

efrotools.pylintplugins.var_annotations_filter(node: NodeNG) → NodeNG

Filter annotated function variable assigns.

This accounts for deferred annotation evaluation (PEP 649/749, the Python 3.14+ language default).

Annotated assigns under functions are not evaluated. Class and module vars are normally not either. However we do evaluate if we come across an ‘ioprepped’ dataclass decorator. (the ‘ioprepped’ decorator explicitly evaluates dataclass annotations).

efrotools.python_build_android module

Self-contained Android Python build script.

Replaces the GRRedWings/python3-android clone+patch approach with a clean, in-tree build that owns all build logic directly. No external repo dependency.

efrotools.python_build_android.build(rootdir: str, arch: str, debug: bool) → None

Build Python for the given Android architecture.

arch must be one of: arm, arm64, x86, x86_64. Outputs are written to:

build/python_android_{arch}[_debug]/build/usr/   (installed Python)
build/python_android_{arch}[_debug]/src/Python-{PY_VER_EXACT}/
  Include/          (Python headers)
  Lib/              (Python stdlib)
  Android/sysroot/  (dep .a files)

efrotools.python_build_android.gather(rootdir: str) → None

Gather Android Python build artifacts into the project.

Reads from build/python_android_{arch}/ and writes to src/external/python-android[-debug]/ and src/assets/pylib-android/.

Assumes all 4 arch builds (arm, arm64, x86, x86_64) have been run for both debug and release.

efrotools.python_build_apple module

Self-contained Apple Python build script.

Builds a static libpython3.14.a for each Apple platform slice (macOS, iOS, tvOS, visionOS) and assembles them into a Python.xcframework. Uses BeeWare’s Python.patch and prebuilt cpython-apple-source-deps.

efrotools.python_build_apple.build(rootdir: str, slice_name: str) → None

Build Python for a single Apple platform slice.

slice_name must be one of SLICES. Output is written to build/python_apple_{slice_name_underscored}/.

efrotools.python_build_apple.gather(rootdir: str) → None

Assemble all slices into Python.xcframework and copy to project.

Expects all 10 slice builds to exist under build/python_apple_*/ Outputs:

• src/external/python-apple/Python.xcframework

• src/assets/pylib-apple/

efrotools.pyver module

This module defines the major Python version we are using in the project.

Tools that need to do some work or regenerate files when this changes can add this submodule file as a dependency.

efrotools.pyver.get_project_python_executable(projroot: Path | str) → str

Return the path to a standalone Python interpreter for this project.

In some cases, using sys.executable will return an executable such as a game binary that contains an embedded Python but is not actually a standard interpreter. Tool functionality can use this instead when an interpreter is needed.

efrotools.static_dependencies_build module

Self-contained Android Python build script.

Replaces the GRRedWings/python3-android clone+patch approach with a clean, in-tree build that owns all build logic directly. No external repo dependency.

efrotools.static_dependencies_build.build(rootdir: str, arch: str, debug: bool) → None

Build OpenAL.

arch parameter is ignored (kept for compatibility). Outputs are written to:

• build/static_dependencies/openal_{arch}[_debug]/ (OpenAL build)

efrotools.toolconfig module

Functionality for wrangling tool config files.

This lets us auto-populate values such as python-paths or versions into tool config files automatically instead of having to update everything everywhere manually. It also provides a centralized location for some tool defaults across all my projects.

efrotools.toolconfig.install_tool_config(projroot: Path, src: Path, dst: Path) → None

Install a config.

efrotools.util module

Misc util calls/etc.

Ideally the stuff in here should migrate to more descriptive module names.

efrotools.util.container_aware_cpu_count() → int

CPU count available to this process, respecting cgroup CPU quota.

Cloud Run (and modern Docker/k8s) impose CPU limits via cgroup CFS bandwidth quotas, NOT via scheduling affinity. os.cpu_count() (and the newer os.process_cpu_count() in 3.13+) reflect scheduling affinity / cpuset cgroup pinning, but ignore the bandwidth quota — so on a 16-CPU host running a 1-CPU Cloud Run container they still return 16.

This reads the cpu cgroup quota directly:

• cgroup v2 (modern Linux, including Cloud Run): /sys/fs/cgroup/cpu.max contains "<quota> <period>" (or "max <period>" when unconstrained). CPUs = quota // period.

• cgroup v1 (older systems): cpu.cfs_quota_us / cpu.cfs_period_us under /sys/fs/cgroup/cpu/.

Falls back to os.cpu_count() outside Linux or when no cgroup quota is set / accessible. Floors at 1 — sub-CPU quotas (e.g. Cloud Run --cpu=0.5 → 50000/100000) round down to 0 which would be meaningless as a worker count.

efrotools.util.explicit_bool(value: bool) → bool

Simply return input value; can avoid unreachable-code type warnings.

efrotools.util.get_files_hash(filenames: Sequence[str | Path], extrahash: str = '', int_only: bool = False, hashtype: Literal['md5', 'sha256'] = 'md5') → str

Return a hash for the given files.

efrotools.util.get_string_hash(value: str, int_only: bool = False, hashtype: Literal['md5', 'sha256'] = 'md5') → str

Return a hash for the given files.

efrotools.util.is_wsl_windows_build_path(path: str) → bool

Return whether a path is used for wsl windows builds.

Building Windows Visual Studio builds through WSL is currently only supported in specific locations; namely anywhere under /mnt/c/. This is enforced because building on the Linux filesystem errors due to case-sensitivity issues, and also because a number of workarounds need to be employed to deal with filesystem/permission quirks, so we want to keep things as consistent as possible.

Note that said quirk workarounds WILL be applied if this returns true, so this check should be as specific as possible.

efrotools.util.readfile(path: str | Path) → str

Read a utf-8 text file into a string.

efrotools.util.replace_exact(opstr: str, old: str, new: str, count: int = 1, label: str | None = None) → str

Replace text ensuring that exactly x occurrences are replaced.

Useful when filtering data in some predefined way to ensure the original has not changed.

efrotools.util.replace_section(text: str, begin_marker: str, end_marker: str, replace_text: str = '', *, keep_markers: bool = False, error_if_missing: bool = True) → str

Replace all text between two marker strings (including the markers).

efrotools.util.writefile(path: str | Path, txt: str) → None

Write a string to a utf-8 text file.

efrotools.util.wsl_windows_build_path_description() → str

Describe where wsl windows builds need to live.

efrotools.xcodebuild module

Functionality related to Xcode on Apple platforms.

class efrotools.xcodebuild.SigningConfig(certfile: str, certpass: str)

Bases: object

Info about signing.

certfile: str

certpass: str

class efrotools.xcodebuild.XCodeBuild(projroot: str, args: list[str])

Bases: object

xcodebuild wrapper with extra bells and whistles.

run() → None

Do the thing.

efrotools.xcodebuild.project_build_path(projroot: str, project_path: str, scheme: str, configuration: str, executable: bool = True) → str

Get build paths for an xcode project (cached for efficiency).