This reverts commit fb941913a3.
Reapplies I4933187e8b72f2ef0c32d18ffea756e2c6fa417c with fixes
to disable the check for mac builds, where many modules are
disabled and the check is just going to cause more problems.
Test: m checkbuild
Change-Id: If6712c90ececd5d015fcdcdeefe0c3d4f5590711
Get a list of leaf nodes in the dependency graph
from ninja, and make sure none of them are in the
output directory. This ensures that there are no
rules that depend on a file in the output directory
that doesn't have rule to generate it. The check
will catch a common set of build failures where
a rule to generate a file is deleted (either by
deleting a module in an Android.mk file, or by
modifying the build system incorrectly). These
failures are often not caught by a local incremental
build because the previously built files are still
present in the output directory.
Bug: 36843214
Bug: 68062417
Test: manual
Change-Id: I4933187e8b72f2ef0c32d18ffea756e2c6fa417c
This speeds up dumping make variables from ~380ms using make to ~220ms
using ckati. It also means that we're consistently using the same parser
for builds (with the same .KATI_READONLY/etc extensions).
envsetup.sh (lunch) / other scripts still use make, changing those to go
through soong_ui will be a future change.
Test: m clean; m nothing
Test: USE_GOMA=true m nothing
Test: m PRODUCT-aosp_x86-sdk
Test: m APP-Calculator
Test: build/soong/build_test.bash -only-config (on AOSP and internal master)
Change-Id: I6ca554de8de4955fb869001d06d29969b75751cc
We can call directly into the blueprint bootstrap.bash using values that
soong_ui has already calculated.
Instead of calling into blueprint.bash, build minibp with microfactory,
and directly run ninja. This allows us to get individual tracing data
from each component.
Test: m -j blueprint_tools
Test: m clean; m -j blueprint_tools
Change-Id: I2239943c9a8a3ad6e1a40fa0dc914421f4b5202c
The Finder runs roughly 200ms faster than findleaves.py in aosp,
and runs roughly 400ms faster in internal master.
Bug: 64363847
Test: m -j
Change-Id: I62db8dacc90871e913576fe2443021fb1749a483
This way we only have one way to start a build, which always has logging
/ tracing / etc, even if we don't need Kati.
There's two ways to use this:
As a direct replacement for mkdir out; cd out; ../bootstrap.bash;
./soong -- as long as --skip-make is always passed, we'll never run
Kati, and Soong will run outside of it's "make" mode. This preserves
most of the speed, and allows full user control over the Soong
configuration.
A (experimental, dangerous) way to temporarily bypass the product
variable and kati steps of a build. As long as a user is sure that
nothing has changed from the last build, and they know exactly which
Ninja targets they want to build (which may not be the same as the
arguments normally passed to 'm'), this can lead to shorter build
startup times.
Test: rm -rf out; m --skip-make libc
Test: rm -rf out; m libc; m --skip-make libc
Test: rm -rf out; mkdir out; cd out; ../bootstrap.bash; ./soong libc
Test: build/soong/scripts/build-ndk-prebuilts.sh
Change-Id: Ic0f91167b5779dba3f248a379fbaac67a75a946e
The build now uses a prebuilt javac binary distributed with the
source, checking the host java version is no longer required.
Reapplies Ieef83481d0e6d68371bbd8a422f870ffe2bbb428 after updating
some more manifests to include the prebuilts.
Test: m -j checkbuild
Bug: 62956999
Change-Id: I774dbb75c8ae2ff1a990423a3827c77886e44779
(cherry picked from commit c6f07f00da)
The build now uses a prebuilt javac binary distributed with the
source, checking the host java version is no longer required.
Test: m -j checkbuild
Bug: 62956999
Change-Id: Ieef83481d0e6d68371bbd8a422f870ffe2bbb428
This causes Soong to put the outputs of each genrule into a temporary
location and copy the declared outputs back to the output directory.
This gets the process closer to having an actual sandbox.
Bug: 35562758
Test: make
Change-Id: I8048fbf1a3899a86fb99d71b60669b6633b07b3e
This speeds things up a little bit, but the major win is that we don't
need to recursively call into the build during the auto installclean.
Test: m -j installclean
Test: m -j dataclean
Change-Id: I09ded8dbd8a2015c7848bc0042dfd3adad1b31c9
This way kati won't need to be run as often (either initially, or when
switching products with the same device).
Bug: 35970961
Test: m clean; m -j blueprint_tools; m -j blueprint_tools; m -j blueprint_tools
Test: lunch aosp_arm-eng; m -j blueprint_tools; lunch full-eng; m -j blueprint_tools; <repeat>
Change-Id: Ie9fca3c8f1dd412459ea47c7090c7c5fdb0bcf6e
When kati keeps state around, it has to regenerate the ninja file every
time the state is changed. So move the java version checking into
soong_ui, where we can parallelize it with other operations instead of
only checking it occasionally.
Bug: 35970961
Test: Put java7 in PATH, m -j
Test: Put java8-google in PATH, m -j
Test: Put a space in TOP, m -j
Test: OUT_DIR=<case-preserving fs> m -j
Test: OUT_DIR=<path with space> m -j
Test: DIST_DIR=<path with sapce> m -j
Change-Id: I3245c8dd6d856240d17d54cb05d593dc9df71a27
So that we don't have to load up all the makefile state just to remove
the output directory.
Starting from a completely empty out directory:
kati: 16s
soong_ui: 2.0s
From a minimal out directory (m -j blueprint_tools):
kati: 3.8s
soong_ui: 0.4s
Test: m -j clean
Test: m -j clobber
Change-Id: Ibeec6fbfa29387750342a752a55336caca4b3992
Wrap os/exec.Cmd to use our Context and Config interfaces for automatic
logging and error handling. It also simplifies environment modification
based on the Config's environment.
This also adds sandboxing on Macs using sandbox-exec. A simple profile
is provided that only logs on violations, though multiproduct_kati on
AOSP has no violations. This isn't applied to ninja, only make / soong /
kati to start with. I measured <5% time increase in reading all
makefiles, and no noticable difference when kati doesn't regenerate.
I'd like to spin up a process to dump violation logs into our log file,
but the log reporting changed over the range of Mac versions that we
support, so that's going to be more complicated. Opening Console.app
works in all cases if you're local -- just search/filter for sandbox.
Linux sandboxing will be implemented later -- the sandbox definition is
opaque enough to support a different implementation.
Test: multiproduct_kati on AOSP master on Mac
Change-Id: I7046229333d0dcc8f426a493e0f7380828879f17
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
Colin Cross