Allow passing a remoteexec.REParams to RuleBuilder to configure it to
run the rule remotely through RBE. Requires the rule to use
SandboxInputs, which ensures that RuleBuilder is aware of all of the
inputs and outputs of the rule.
Running sbox in RBE initially seems unnecessary, as RBE is already a
good sandbox, but reproxy can execute RBE actions locally when configured
for local execution, local fallback or racing. Using sbox in RBE ensures
that these local actions are also sandboxed, giving consistent results
between directly executed actions, local RBE actions, and remote RBE
actions.
Bug: 182612695
Test: manual
Change-Id: Icf2f24dde8dee833eb680ba22566a8e1c0143b15
When RuleBuilder.SandboxInputs() is called configure sbox to copy
all the input files into the sandbox directory and then change the
working directory there when running the command.
Copying input files into the sandbox directory gets tricky when
the input file is the output file from another rule, and could
be at an arbitrary, possibly absolute path based on the value
of OUT_DIR. They will need to be copied to a directory in the
sandbox using the path relative to OUT_DIR.
RSP files need special handling, they need to both be copied into
the sandbox as an input, rewritten to contain paths as seen in the
sandbox, and references to them on the command line need to use
sandbox paths.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: Ic0db961961b186e4ed9b76246881e3f04971825c
Output files in sandboxed RuleBuilder rules must be under the output
directory, but output paths that were marked as temporaries were
not error checked.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: I09616402ef1637c2c455ec7d345a296711582e8f
WriteFileRule shouldn't force the caller to ninja escape the input,
and should shell escape spaces.
Bug: 182612695
Test: manual
Change-Id: Ide2f1ed92783eef7883279238de209d992d8f735
Now that rsp files use an explicit path instead of $out.rsp the
unescaped spans and NinjaEscapedCommands() support can be removed
from RuleBuilder.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: I6705151ff0528c59aa1af56ee52c294da15a7a66
Using $out.rsp as the rsp file adds extra complexity around keeping
the $ unescaped. Make callers to FlagWithRspFileInputList provide
an explicit path for the rsp file instead.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: I3f531d80c1efa8a9d09aac0a63790c5b11a9f0c6
The CommandDeps and Rspfile properties can contain arbitrary paths
that may need ninja escaping.
Bug: 182612695
Test: genrule_test.go
Change-Id: I1426e9dfeab3bde9f65b61d0c47840f97c1c7526
This relands I38393900677c5dbe8e72fe06a7dd7d71f3c09f82 after
I8038ed5c6f05c989ac21ec06c4552fb3136b9a7a, which makes the ASAN
variants of libraries have PackagingSpecs so they can be copied
into sandboxed genrules.
Copy tools used by genrules into the sandbox directory. This
ensures correct dependencies on all files used by tools, and
is one step closer to enabling genrules inside unselected
namespaces.
Bug: 124313442
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I57c0d5fc8bba216fac4deb972d0d2098593e8963
Copy tools used by genrules into the sandbox directory. This
ensures correct dependencies on all files used by tools, and
is one step closer to enabling genrules inside unselected
namespaces.
Bug: 124313442
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I38393900677c5dbe8e72fe06a7dd7d71f3c09f82
Enable the RuleBuilder and RuleBuilderCommand methods to access
the BuilderContext by passing it to NewRuleBuilder instead of
RuleBuilder.Build.
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I63e6597e19167393876dc2259d6f521363b7dabc
This relands I3b918a6643cea77199fd39577ef71e34cdeacdb1 with a fix
to create the directory for the output depfile if doesn't exist.
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.
Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I8af00c8c0b25a92f55a5032fcb525715ae8297c2
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.
Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I3b918a6643cea77199fd39577ef71e34cdeacdb1
In preparation for more complicated sandboxing support in sbox, use
a single implementation of the sbox sandboxing by moving genrule to
use RuleBuilder's sbox support instead of creating an sbox rule
directly.
Also move genrule's input list hash support into RuleBuilder.
Test: genrule_test.go
Test: rule_builder_test.go
Change-Id: I292184d02743c7e6887ebbcd232ba565db2ab0cc
This CL adds symlink_outputs to various locations in Soong that creates
actions that creates symlink outputs, and explicitly mark them as such.
Test: m
Bug: 160568334
Change-Id: I322751bada52a9f49011c74731d84761586e03e7
The sandbox property indicates whether metalava should only read
inputs explicitly specified on the command line. This CL adds the
property and sets the appropriate configuration for RBE depending
on whether the sandbox is set or not.
Test: built aosp_crosshatch-userdebug with/without RBE_METALAVA.
Change-Id: I7256d29f18e0af18dbe65d1c7dbbf62fd3d65f4c
This CL fixes a bug with RuleBuilder's handling of remoteable actions.
It adds a new type of pool to identify remoteable rules by the android
module context. The pool is then set to nil to actually run actions at
NINJA_REMOTE_NUM_JOBS parallelism.
Test: built aosp crosshatch userdebug
Change-Id: I29452f6fc7a161b94189731e3e3cc1f34907b80c
This CL adds RBE support to javac, r8, and d8 rules which is only
enabled if respective environment variables are set.
Test: an aosp_crosshatch build with and without the new variables.
Change-Id: Ic82f3627944f6a5ee7b9f3228170c2709b1bfcb8
Create a highmem pool based on the total RAM and the number of CPUs,
with an override via the NINJA_HIGHMEM_NUM_JOBS variable. Put
metalava into the highmem pool.
Ninja does not support nested pools, and when goma or RBE is enabled
the maximum ninja parallelism is set very high with local jobs in a
local pool. When both the local pool and highmem pool are enabled,
the total number of local jobs will be as high as the sum of the sizes
of the two pools. Keep the highmem pool limited to 1/16th of the
local pool when remote builds are enabled to try to minimize the
effect while still limiting highmem jobs.
Fixes: 142644983
Test: m nothing, examine pools
Test: m USE_GOMA=true nothing, examine pools
Change-Id: Id79f11f44948992960ac34ecf831dacbe21bd332
sbox will generate a random directory for the output root, and most
tools will encode that directory name in the output target of the
depfile.
So embed the library from dep_fixer into sbox so that it can rewrite the
output filename to a static (reproducible) value. Ninja doesn't care
what that value is, so it's just "outputfile".
Also fix up rule_builder to actually tell sbox about the depfile.
Test: mmma system/iorap; check the contents of:
out/soong/.intermediates/system/iorap/libiorap-binder/android_arm_armv7-a-neon_core_static/gen/aidl/system/iorap/binder/com/google/android/startop/iorap/IIorap.cpp.d
Change-Id: I3640a2e8b0c034f143a35e398a8418a6d621b265
Allow RuleBuilderCommands to use an rspfile with FlagWithRspFileInputList.
This requires being more careful with ninja escaping, as the rspfile
will be $out.rsp, and the value for $out may not be known yet so it
must be inserted as a ninja variable that must not be escaped.
Test: rule_builder_test.go
Change-Id: Ifa91e24a0bb8f0ceeb5c9bfa5689be2a4ff3b9cd
Replace the common pattern of:
cmd.Tool(ctx.Config().HostToolPath(ctx, "tool"))
with:
cmd.BuiltTool("tool")
And similarly for PrebuiltBuildTool.
Test: m checkbuild
Change-Id: I7d63188505362c7df6a3b3e7330b4a2cca5a2409
strings.Builder avoids copying the final byte array to a string by
using unsafe to point the string to the backing array of the byte
slice.
Test: rule_builder_test.go
Change-Id: I3d9a70fb62f8542650cd9ebc65397297bba12585
This fixes an incremental build issue where we didn't clean up old aidl
header files (or necessarily notice that they were updated).
We do this by declaring the header files as outputs in the build graph,
but this means that the src file name needs to be convertible to the
aidl package name, as that's how the header file paths are created. In
many cases, filegroups can be used to strip path prefixes from the aidl
files.
Bug: 112114177
Test: treehugger
Change-Id: If534ff3dbfac329dea9a7402e30be74495754160
This essentially allows you to declare that everything in a directory
will be created by the rule, and we'll ensure that your command actually
writes out all of the claimed outputs, and remove any other files that
previously existed in that directory.
Test: built-in tests
Change-Id: I990dce2b3a0d89ebd2736ac1a0cadfb5864c6e73
Allow rules built with RuleBuilder to use depfiles. Ninja only
supports a single depfile with single output. If there are
multiple outputs in a rule, move all but the first to implicit
outputs. If multiple depfiles are specified, use new support
in dep_fixer to combine additional depfiles into the first depfile.
Test: rule_builder_test.go
Change-Id: I7dd4ba7fdf9feaf89b3dd2b7abb0e79006e06018
Follow the change to blueprint to make *Escape take and return a string
and add *EscapeList that take and return slices of strings. Fix up
a few places that were unnecessarily converting a string to a slice
and back to a string.
Test: m nothing
Change-Id: I3fa87de175522205f36544ef76aa2f04aef1b936
There are no more Make paths being used in Soong now that
dexpreopting and hiddenapi are in Soong. Use the Path types
in the inputs to RuleBuilder, and fix all users of RuleBuilder.
This reapplies I886f803d9a3419a43b2cae412537645f94c5dfbf with
fixes to disable preopt for Soong-only builds when the global
dexpreopt.config doesn't exist.
Test: all soong tests
Test: m checkbuild
Change-Id: I4dae9ecd5de22f062f9478ec8f0747f099cf8190
There are no more Make paths being used in Soong now that
dexpreopting and hiddenapi are in Soong. Use the Path types
in the inputs to RuleBuilder, and fix all users of RuleBuilder.
Test: all soong tests
Test: m checkbuild
Change-Id: I886f803d9a3419a43b2cae412537645f94c5dfbf
Add a RuleBuilderInstalls type for a slice of RuleBuilderInstalls,
and give it a String() method that returns the list of installs
in the format that is convenient for passing to Make.
Test: rule_builder_test.go
Change-Id: I2e9cd9abf4dfb0ad312d0a6662f1567baf9cd222
Add a method to be used when Config.AllowMissingDependencies() is true
to produce an error rule when the rule is missing dependencies.
Test: m checkbuild
Change-Id: If370fbb2734237a84a100b99b5238c7a2256c405
Temporary marks an output path as a temporary file that is not
necessary after the rule completes, removing it from the result of
Outputs. DeleteTemporaryFiles adds a command to the command line
that deletes all files that have been marked with Temporary.
Test: rule_builder_test.go
Change-Id: Ie509ed800992962747fb287858e148e975eee54a
Add a few convenience methods, document all the methods and add
examples that would show up in the godoc if we were to actually
generate it.
Test: rule_builder_test.go
Change-Id: I270fed605ffec34e6f5b36fde0dc9ca52694b741
Make RuleBuilder.Build take a subset of ModuleContext and
SingletonContext, and dynamically call PathForModuleOut only
if it is available.
Test: rule_builder_test.go
Change-Id: Id825cb75236acf187e9d4a36353a47abcac71927
Move dexpreopt.Script to android.RuleBuilder so that the builder
style can be used in more places. Also add tests for it.
Test: rule_builder_test.go
Change-Id: I92a963bd112bf033b08899e930094b908acfcdfd
Colin Cross