If SOONG_UI_TABLE_HEIGHT is set, enable a new smart terminal display
that prints the normal scrolling build history in the top region of
the screen and an action table of the longest currently running
actions in the bottom region of the screen. This provides better
visibility into which are the longest running actions and when the
build parallelism is very low.
Test: manual
Change-Id: I677d7b6b008699febd259110d7f9e0f98d80c535
Instead of reading the terminal size on every status update, register
for SIGWINCH to read and store the size when it changes.
This reapplies I555ad21a31a2c924ab0ca681e0c8f00df42a370a with a fix
for a race condition in TestSmartStatusOutputWidthChange.
Test: status_test.go
Change-Id: If342cb4cc8e4ed57af9e3bb417758348c9c41247
Instead of reading the terminal size on every status update, register
for SIGWINCH to read and store the size when it changes.
Test: status_test.go
Change-Id: I555ad21a31a2c924ab0ca681e0c8f00df42a370a
Bolding the status line provides differentiation between output
of each command.
Test: status_test.go
Change-Id: I9d46761e69c5af0a0aa86c7921e121cfd2a3fc82
Write log output through StatusOutput so that the status implementation
can synchronize it with its own output.
Test: status_test.go
Change-Id: I917bdeeea4759a12b6b4aa6d6d86ee18a2771723
terminal.Writer is now just a wrapper around stdio.Stdout() without
any useful functionality. Replace it with stdio.Stdout() as an
io.Writer.
Test: ui/terminal/status_test.go
Change-Id: I5bc5476afdca950b505642f0135a3af9d37fbe24
Support for smart and dumb terminals are implemented in writer.go,
which makes dumb terminals much more complicated than necessary.
Move smart and dumb terminals into two separate implementations
of StatusOutput, with common code moved into a shared formatter
class.
Test: not yet
Change-Id: I59bbdae479f138b46cd0f03092720a3303e8f0fe
All users have been removed.
Test: treehugger
Change-Id: I35e55f3ce79c8e68585be120b5dec810563c7a83
Merged-In: I35e55f3ce79c8e68585be120b5dec810563c7a83
- Add SOONG_LOCK_TIMEOUT environmental variable for users to set a
duration (e.g. 1h), which defaults to the current 10 seconds.
- Instead of logging once per second waiting to acquire the lock,
only log once before the first wait
Fixes: 131854539
Test: make; open another tab and make
Change-Id: I0a02a1b071964f74473a5f55034f916526e1b309
Allow install-clean as an alias for installclean and data-clean as
an alias for dataclean.
Test: m install-clean
Change-Id: I9c97d60572a524ad68caecd26d52bfb987468075
This reverts commit e114b3f0a9.
Build broke as follows:
out/soong/host/linux-x86/bin/sbox --sandbox-path out/soong/.temp --output-root out/soong/.intermediates/system/update_engine/ue_unittest_disk_imgs/gen -c 'tar -jxf system/update_engine/sample_images/sample_images.tar.bz2 -C __SBOX_OUT_DIR__/gen disk_ext2_1k.img disk_ext2_4k.img disk_ext2_4k_empty.img disk_ext2_unittest.img' __SBOX_OUT_DIR__/gen/disk_ext2_1k.img __SBOX_OUT_DIR__/gen/disk_ext2_4k.img __SBOX_OUT_DIR__/gen/disk_ext2_4k_empty.img __SBOX_OUT_DIR__/gen/disk_ext2_unittest.img
"bzip2" is not allowed to be used. See https://android.googlesource.com/platform/build/+/master/Changes.md#PATH_Tools for more information.
tar: Child returned status 1
tar: Error is not recoverable: exiting now
sbox command (tar -jxf system/update_engine/sample_images/sample_images.tar.bz2 -C out/soong/.temp/sbox849645882/gen disk_ext2_1k.img disk_ext2_4k.img disk_ext2_4k_empty.img disk_ext2_unittest.img) failed with err "exit status 2"
Change-Id: I3fe7100886ea3b748f5a6b5fc6d2811391bb72a8
There are no remaining users.
Also mark BUILD_BROKEN_ANDROIDMK_EXPORTS and BUILD_BROKEN_ENG_DEBUG_TAGS
as deprecated in scripts/build_broken_logs.go
Test: treehugger
Change-Id: If7892bef1b9001f12a99565f886b395cf1985e70
So that the output of build_test can be parsed to understand which
devices need changes.
Test: read out/soong.log
Change-Id: I40b25527ef72e3f646c0cfe62114b452fa3f27f6
Some people apparently still talk to the network during their build.
Allow this temporarily with a BUILD_BROKEN_USES_NETWORK check.
Bug: 129992021
Test: attempt to talk to the network during the build with and without
this flag.
Change-Id: Ifb967c656aa24c4599e7232d0f1b5a303b5bac52
Attempt to catch places where logs are truncated by panicing if
logging is attempted after Close.
Test: m nothing
Change-Id: If670f20d08832ed65b63af5589b548e9815f2f0d
Wait for the ninja proto processing goroutine to notice the fifo
has closed and exit before continuing.
Test: m nothing
Change-Id: I8cf5f3b8bf6a91496c6d2bbbd3e811eb7f0c9d21
Dump variable in log to track remaining targets which don't set it.
Bug: 122954981
Test: $ grep VNDK out/soong.log
2019/03/07 09:22:04.235604 build/soong/ui/build/dumpvars.go:109: BOARD_VNDK_VERSION current
Change-Id: I34146c752a4bf0d799374bb93cc0d75b0f1b048c
There's more than just toybox in this group now, so let's rename our
variables to something closer to the desired behavior, rather than the
first user.
Test: treehugger
Change-Id: I76d4407792061c8110b194cfe73f1ddc84dbc22f
For ease of investigating the issue with error reports, let me make
the error message more verbose.
Test: make goma_ctl.py raise Exception and run.
Test: I confirmed verbose error message shown.
Change-Id: I04f9a5ca4808746994c41ddce8a5b22a1975fcc2
Signed-off-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
Bug: 124218940
If ANDROID_QUIET_BUILD environment variable is set to '1', do not show 20
lines of the environment variables. In addition, for the failing step show
only the step name and output, omitting the command proper (the verbose log
as well as error.log still contains it).
If build succeeds, the output of the build itself consists of a single
line:
```
> m androidmk
[100% NN/NN] <last command>
```
When it fails, the output does not contain sometimes very long command
line:
```
> m androidmk
[ 97% NN/MM] test androidmk
FAILED: <step>
--- FAIL: TestEndToEnd (0.01s)
androidmk_test.go:1025: failed testcase 'prebuilt_etc_TARGET_OUT_ETC'
input:
include $(CLEAR_VARS)
LOCAL_MODULE := etc.test1
LOCAL_MODULE_CLASS := ETC
LOCAL_MODULE_PATH := $(TARGET_OUT_ETC)/foo/bar
include $(BUILD_PREBUILT)
expected:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
got:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
FAIL
17:50:53 ninja failed with: exit status 1
```
[The related change in build/make/envsetup.sh suppresses timing display
when the same variable is set.]
Change-Id: I4d3c72457de031ff58a324c2fe98f4c1d10f8239
Test: treehugger
And add a helper script that can help parse the output from
build_test.bash
Test: check for BUILD_BROKEN_ENG_DEBUG_TAGS in soong.log
Test: go run ../build/soong/scripts/build_broken_logs.go *
Change-Id: Idd0fc8b59770dcdbe44eeba262558708a9497f96
We don't want these to be warnings, may as well be errors instead of
turning off the warnings.
Bug: 123583617
Test: treehugger
Change-Id: I6ca518c9647e712426952cb88bdb044d933b23f2
bufio.Scanner defaults to a relatively small max line length of
64kB. Kati output may have long lines, use a max length of 2MB
instead.
Fixes: 123590367
Test: m checkbuild
Change-Id: Ia869c5c2526621244f0065736e0d5fd9a088df06
As more phony rules move either to the packaging kati run, or soong, the
existing real-to-phony check failed to notice that they were phony
rules.
So I added --top_level_phony to Make that assumes that all leaf nodes
(source files, or nodes generated by another ninja file generator) that
do not have a '/' in them are phony targets.
The existing phony-looks-real check in Kati should prevent these from
being real generated files, and our source tree shouldn't have any
source files used at the root level.
Test: build_test on downstream branches
Change-Id: I77cff84b536c6cd6402307a9aa5a9c273e72c71f
We've got an internal testcase that uses minijail0 inside an Android.mk
rule. That was failing since we turned on the linux sandbox, as /proc
was mounted read-only, which prevented setting up the uid/gid mappings
for a child namespace.
Fixes: 122985455
Test: treehugger & forrest of breaking build
Change-Id: Ia77a91a7f4eeeb8a24e84075d8272287f5087587
This starts a new network namespace without any connections to the
outside.
Bug: 122270019
Test: USE_GOMA=true m libc
Test: treehugger
Test: add rule to use /usr/bin/wget, fails after this change
Change-Id: Iba262025ce0e4e3bef5c34c817cc678d6c61403b
This really only initializes the sandbox, it does not attempt to change
the view of the filesystem, nor does it turn off networking.
Bug: 122270019
Test: m
Test: trigger nsjail check failure; lunch; m; cat out/soong.log
Test: USE_GOMA=true m libc
Change-Id: Ib291072dcee8247c7a15f5b6831295ead6e4fc22
As suggested in
https://android-review.googlesource.com/c/platform/build/soong/+/839293
I am moving some features in goma.mk to goma.go in Soong UI.
With this CL, let me implement ulimit check to Soong UI.
Test: export USE_GOMA=true
Test: launch aosp_arm-eng
Test: make
Test: the command succeeds if "ulimit -n" and "ulimit -u" are large enough.
Test: Otherwise, it shows error. I confirmed both cases.
Change-Id: I5d7d5ed71f620302a0d635770d1a51a2baab51fd
Signed-off-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
As suggested in b/118390303#comment18, let me run
"goma_ctl.py ensure_start" in soong UI.
Bug: 118390303
Test: stop compiler_proxy, and execute following command:
Test: USE_GOMA=true lunch aosp_arm-eng
Test: and confirms compiler_proxy is running.
Change-Id: I859daae6ae2399c5b6bce2fafd874dfdb7c6aae0
Signed-off-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
As part of a future change to sandbox the build on Linux, the real
username will be switching to "nobody", and the hostname will be
switching to "android-build".
The USER environment variable will reflect the sandboxed value, so for
the build properties that want the external USER, they'll need to use
BUILD_USERNAME.
Similarly, BUILD_HOSTNAME will reflect the real value, while the
`hostname` tool will return "android-build"
Bug: 122270019
Test: check build.prop
Change-Id: I99604b9488732a63690b256dc4dd7894d369a32c
So that ninja produces an error instead of just a warning when a dep
file is not produced.
Bug: 121058584
Test: check build logs for "depfile is missing" warning
Test: treehugger
Change-Id: I1cbaba866eaf293495c3c0b2b174190bcb2b0f9a
Test: Dumped the text formated based metrics file to out dir,
and checked the file.
Bug: b/63815990
Change-Id: Iff476f72a0be74eb53b6b26ef468d11c0f24a404
This way we don't appear hung at:
No need to regenerate ninja file
Change-Id: I8dbdaa2c1b1c5a6a73187d0e6061f363b62e10c9
Fixes: 122251150
Test: m nothing
Prepend the toybox symlink directory to $PATH, and stop generating
symlinks into out/.path for tools used from toybox.
We don't (yet?) have toybox prebuilts for darwin, so apply the above only to Linux, and preserve existing behavior on Darwin.
Test: check the build banner for uname results
Test: m
Change-Id: I37fc380381e65a628cdc131d462fd4441eacfe9d
Stop writing out ninja files, as they become very large, especially when
multiplied by the number of defined products.
Test: treehugger
Change-Id: Id9529d14040acb72a0188e58b5db2911f142071e
DIST_DIR is controlled by soong_ui, and the make dumpvars code no longer
has access to it. So instead, handle some dumpvar requests inside
soong_ui itself.
Test: get_build_var DIST_DIR
Test: get_build_var OUT_DIR
Test: get_build_var BUILD_SYSTEM
Test: build/soong/soong_ui.bash --dumpvars-mode --vars="DIST_DIR
BUILD_SYSTEM"
Change-Id: Id3bcb8b0748db67c83d0e42d1ae5be564c5eb2f7
We're only using it to distribute files in case of failure, which isn't
well supported currently, but can be handled for now by using the
DIST_DIR environment variable during the command execution.
This was at least one cause that we'd be re-running Soong during every
build server build, as the DIST_DIR values are unique.
Test: m dist
Change-Id: Ibd5e6b6c46695350de80b745bfb6a6aa685033a0
We've got a prebuilt of make in prebuilts/build-tools, and we build
xmllint from source.
Test: No logs on the build servers in the last 24 hours
Change-Id: Ib60a7b5feaa31097071b67ebf6beb7f0b9a9f81b
The idea is that we'd move the installation and packaging tasks over to
it, using data from Soong & the Kati reading Android.mk files.
This would allow us to make more fundamental changes about how we
package things without having to adjust makefiles throughout the tree.
Possible use cases:
* Moving some information from Soong's Android.mk output to a file read
by the packaging step may allow us to read the Android.mk files less
often, speeding up builds.
* Refactoring our current two-stage ASAN builds to run the Kati build
step twice, writing into different object directories, then have a
single packaging step that reads both outputs. Soong already has the
capability of writing out a single ninja file with all the asan
combinations.
* Running two build steps, one building the system-related modules
using a "generic" device configuration, and one building the vendor
modules using a specific device configuration. This could enforce a
GSI/mainline system vs vendor split in a single build invocation.
* If all installation is through this tool, it will be much easier to
track what should no longer be installed on an incremental build,
reducing the need for installclean.
* Changing PRODUCT_PACKAGES should be a much faster operation, which
means we could keep track of local additions to the images. Then
`mma` would be more persistent, instead of installing something once,
then never updating it again.
Eventually we plan on switching from Kati to something Go-based, but
this is a more incremental approach while we clean up everything else.
Currently, this just moves the dist-for-goal handling over to the
packaging step, so that we don't need to read Android.mk files when
DIST_DIR changes, or we switch between dist vs not.
Bug: 116968624
Bug: 117463001
Test: m nothing
Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
Set the default value if one isn't provided.
Test: `get_build_var DIST_DIR` after removing the default setting in envsetup.mk
Change-Id: I0cb310fc65f7747c36de14608b61786ef6863fd1
This is no longer used within Android builds. We don't have any way of
providing extra libraries, or using a hermetic version of perl.
Bug: 117338567
Test: check build logs for the last 48 hours
Change-Id: I06d41a0006a47508a435dbf44bfccc4b385d9f41
Now that we've got prebuilts of make, turn on logging so that
we can find all the users and convert them to the prebuilt.
We also build xmllint as part of the tree, so log uses of that
too.
Test: treehugger
Change-Id: I8a6a5d481dfc709dc0eb3424c51430bc54763c1e
Make the cleanspec & Android.mk runs of kati use much of the same code
and arguments.
Also renames 'Kati' to 'KatiBuild' in many cases, in the the expectation
that we'll have a 'KatiPackage' step in the future.
Use --no_ninja_prelude and move local_pool & _kati_always_build_ into
the combined ninja file. This will reduce the need to re-read makefiles
when Goma is enabled, and it allows us to include more than one
Kati-generated ninja file in the build graph.
Bug: 116968624
Test: build_test on downstream branches
Change-Id: Ibdac689b81f62dc293647fad917d84946f2c3cfa
Check for the presence of Android.mk or CleanSpec.mk, which
are somewhat common problems.
Bug: 113147143
Test: m (with and without files present)
Change-Id: I31cf60c325e7f6c6fce7aec54712c1cb802055c2
All downstream users either mark BUILD_BROKEN_PHONY_TARGETS=true
explicitly, or don't have any warnings.
Test: build_test on downstream branches
Change-Id: I6e9d7ed42da941148827854851657895c2e98649
Build target owners will create a zip of all OWNERS
files in source code based on the cached OWNERS.list
Bug: 114242886
Test: make dist -j owners
Change-Id: I97fa3737e1a2a240a08e072dfb2905c11f28705b
These warnings are usually hit in one of the two conditions:
1. Targets that should be marked .PHONY, but are not.
2. Attempts to define output files in the source tree.
See https://android.googlesource.com/platform/build/+/master/Changes.md#phony_targets
for more information.
Test: multiproduct_kati in all downstream branches
Change-Id: Ibe37cce320612825c22987e94d0740e8c0565b6f
od is used by ninja's inline.sh tool to encode a python script in a C
header.
Test: forrest run of build-tools branch
Change-Id: I87001a02b168ebe183b8b7dc495c67bb43f1b86d
Anything that isn't explicitly marked as Allowed in this list will now
return an error (and log a message) when it is executed.
Test: search all logs from the build server over the last day
Change-Id: I3ceacd9a140097809dde81a8d8979dd2c45f234c
unix2dos is used instead of todos by some build servers in
development/build/tools/patch_windows_sdk.sh
Test: none
Change-Id: I04cb2258f71bebfc8792b7aa95643a45ffe3ec5c
todos is used by development/build/tools/patch_windows_sdk.sh
Test: m win_sdk; check out/soong.log
Change-Id: I5e1a290838cc34261d7f5fc321afe79a8b1a6fbb
These still default to warnings, but if a board decides to set
BOARD_BROKEN_PHONY_TARGETS := false, they'll turn into errors. More
likely I'll just be marking the broken targets as broken, then switching
the logic here to be like the dup rules warnings/errors.
Test: On a build with warnings, try <missing>, "false", and "true"
Change-Id: I041e1ff4618c4114ec43015b6c0ae1b49b36b6f8
Attempting to reduce the number of different spellings we have for
"product services" partition in the codebase.
Bug: 112431447
Test: m
Change-Id: I0a393a1d625e7ea3217d28735a4db709bce32395
Merged-In: I0a393a1d625e7ea3217d28735a4db709bce32395
Setting this will cause our invocation of ninja to fail. Even if we
fixed that, it's likely to run into other differences during the build.
Fixes: 112443729
Test: POSIXLY_CORRECT=1 m nothing
Change-Id: I0315955c9fd9536cc6cb0d7c700eed207f35cc80
lsof is used by goma to find its proxy daemon
ps is used on Mac instead of pstree in soong_ui
Test: none
Change-Id: Ifed7d09d604c00b7eebac1488a6f4658788d6e33
du is used by build/make/tools/releasetools/build_image.py
paste is used by system/core/rootdir/update_and_install_ld_config.mk
xxd is used by device/generic/goldfish/tools/mk_qemu_image.sh
Test: none
Change-Id: I707668578d1babe1b149c2c3f989e5f8be37d897
Buildbot parses failure messages that look like ninja errors, but this
error doesn't look like a ninja error. We used to output this via
stderr, which buildbot would fall back to, but that had synchronization
issues between stderr and stdout.
So switch this over to use the status API, marking this action as failed
when necessary, which will display a ninja-like error that buildbot can
parse.
Bug: 112007097
Test: add dangling dep, look at output.
Change-Id: Ib2bbba4373ea22ac01af8aa6bbf295296a93e94d
--warn_real_to_phony will print a warning any time that a non-PHONY rule
depends on a PHONY rule. Since PHONY rules are always considered dirty
this would force rebuilds on every incremental build. Real rules should
always depend on rules that have real outputs.
--warn_phony_looks_real will print a warning any time that a PHONY rule
has an output that looks like a file (has as / in it). This is can cause
confusion for users if it's not a real file, and it shouldn't be marked
as PHONY if it will be run every time (since that will slow down
incremental builds).
Test: run build_test on downstream branches, see few warnings
Change-Id: Id135c85c991976be33cd48a34eba0caaff1eda2c
All of the existing devices that have problems with this are setting
BUILD_BROKEN_DUP_RULES to true. Switch the behavior to default this to
an error.
Bug: 77611511
Test: check logs of all downstream build_test targets
Test: attempt to add an overriden command to hikey960 (uses default)
Change-Id: Ia83089f035925cf0c9883a1f593ae2da89fee568
Ninja now knows how to write directly to a file (or in our case, a named
pipe). This works around an issue we were seeing on Mac, where Go would
just hang after 50-2000 proto messages. It's also just a simpler
solution.
Bug: 111544015
Test: `m` with updated ninja on both Linux & Mac
Change-Id: Ic91920d83a6d2ea0b79e82b467e2423d78189f12
I've noticed a few instances of interleaved status messages in between
lines in a terminal/Writer.Print call on our build servers. Since
there's a lock protecting everything we write, I've got to assume this
is a stdout vs stderr problem. Ninja had always been outputing to
stdout, except for error messages, which are now marked with FAILED:
like failed actions.
Test: m blueprint_tools
Test: m missing
Change-Id: Idf8320d40694abf212c902c63a9703e4440ffb7a
The extra `defer os.Remove(fifo)` was sometimes racing with the next
instance of ninja, removing the file in between when it was re-created
and used. Since we're always removing the file before creating it, it's
safe to just remove that.
The error message for this failure wasn't all that good either, so move
so use the status Print/Error calls inside the goroutine instead of the
logging ones.
Test: `build/soong/build_test.bash -only-soong` repeatedly
Change-Id: Icfeb6b68802093bd3a07d3e46046ef7d1a89d4a1
This adds a new status package that merges the running of "actions"
(ninja calls them edges) of multiple tools into one view of the current
state, and gives that to a number of different outputs.
For inputs:
Kati's output parser has been rewritten (and moved) to map onto the
StartAction/FinishAction API. A byproduct of this is that the build
servers should be able to extract errors from Kati better, since they
look like the errors that Ninja used to write.
Ninja is no longer directly connected to the terminal, but its output is
read via the protobuf frontend API, so it's just another tool whose
output becomes merged together.
multiproduct_kati loses its custom status routines, and uses the common
one instead.
For outputs:
The primary output is the ui/terminal.Status type, which along with
ui/terminal.Writer now controls everything about the terminal output.
Today, this doesn't really change any behaviors, but having all terminal
output going through here allows a more complicated (multi-line / full
window) status display in the future.
The tracer acts as an output of the status package, tracing all the
action start / finish events. This replaces reading the .ninja_log file,
so it now properly handles multiple output files from a single action.
A new rotated log file (out/error.log, or out/dist/logs/error.log) just
contains a description of all of the errors that happened during the
current build.
Another new compressed and rotated log file (out/verbose.log.gz, or
out/dist/logs/verbose.log.gz) contains the full verbose (showcommands)
log of every execution run by the build. Since this is now written on
every build, the showcommands argument is now ignored -- if you want to
get the commands run, look at the log file after the build.
Test: m
Test: <built-in tests>
Test: NINJA_ARGS="-t list" m
Test: check the build.trace.gz
Test: check the new log files
Change-Id: If1d8994890d43ef68f65aa10ddd8e6e06dc7013a
Using ANDROID_BUILD_TOP and other envsetup.sh varialbes is already
forbidden in makefiles, but some tools run by the build (like
hidl-gen) will use it if it is set. Since they are set by envsetup.sh,
and sourcing envsetup.sh before building is optional, unset them inside
the build to get consistent behavior.
Bug: 79250545
Test: m checkbuild
Change-Id: I3e7d9f01390ccdc34c49115f2f15cd7542d9974b
Make the dangling rules list sorted and unique in order to avoid
very long lists when a dangling rule is referenced many times.
Also prettify the output by indenting the list and printing
"stopping" instead of a blank line for the fatal.
Test: m checkbuild
Change-Id: I8f7c27ae39b59f506b529d9995d90b0d6b9835d1
This matches the change to blueprint's blueprint_impl.bash:
https://github.com/google/blueprint/pull/213
It allows us to run bpglob during the two bootstrap phases, so we're
never running minibp or the primary builder in a later phase than
normal.
Also removes the dependency on out/soong/build.ninja from the main
phase, since it's never generated by the main phase anymore, and is
tripping the dangling dependency checks.
Bug: 73646380
Test: m nothing
Test: rm frameworks/base/core/java/android/content/pm/dex/ArtManager.java
Test: m nothing (soong re-runs)
Change-Id: Ia3952d015ad6091ad5a841f555acda78c9390e84
There are no users of these, other than the PDK, which is already
exempted.
Test: m nothing
Test: Check for warnings in downstream build_test logs
Test: Check for warnings in last two days of continuous build logs
Change-Id: I7c7d0068e928e591355c39d22635af86b8f43f1b
Remove support for compiling with javac from OpenJDK8.
We still target 1.8 by default, and OpenJDK8 prebuilts are still
required for the bootclasspath and running robolectric.
Bug: 38418220
Test: m java
Change-Id: I5686deb0ae4f9927192a039d08adc0117b2605dd
This is useful when determining which devices are setting which values
from the logs, cross-referencing where warnings still occur.
Test: look at out/soong.log
Change-Id: I3e3f0e4c8cbd9ce621b03f5163eff2cccfa8c424
Use the OpenJDK9 javac to run ErrorProne by adding it to
-processorpath and using -Xplugin:ErrorProne.
Bug: 69485063
Test: m RUN_ERROR_PRONE=true javac-check
Change-Id: I0496006b71b70766ef16d57753cbcf037897799c
We've turned off suffix rules with .SUFFIXES, but make them explicit
errors.
Test: build_test on downstream branches
Change-Id: I682ee3eb3a8f1451cd7e16f1e1d59afca5d21a26
pkill and sleep are used in
device/generic/goldfish/tools/emulator_boot_test.sh
pgrep is used by GOMA
bc and pwd are used by kernel builds
Test: none
Change-Id: If7004255b4776fa4409fdd4b0aa6d6617b471317
atree was using the host's strip tool during sdk generation, so expose
the prebuilt for use instead.
`id` was used with a test running minijail0 during the build
`sha1sum` is used by development/build/tools/mk_sdk_repo_xml.sh
`rmdir` was being used to remove some tmp files
Test: m PRODUCT-sdk_arm64-sdk dist sdk_repo
Test: Run other builds through forrest
Change-Id: I49c756a3e9497c0bbd9030754e6aa193a2d989a4
This reverts commit 09f4540d66.
Fixes the raw call to net.Listen in the tests to go through the listen()
helper and use the long socket path fallbacks.
Removes the use of timeouts from the tests -- the behaviors being tested
did not rely on timeouts, so removing them will reduce the flakiness if
the build is heavily loading the machine at the same time the test is
running.
Also fixes some potential nil pointer dereferences.
Test: OUT_DIR=<really long> m blueprint_tools
Test: `while .../soong-ui-build-paths/test/test; do sleep 0.01; done` with a build running
Change-Id: I16d44be7517bc415f1c808284088f4ba40df3bfa
This reverts commit c59a92cb1a.
Reason for revert: tests are broken with long OUT_DIRs
They're directly calling net.Listen, and not using the fallback
for long socket names.
Change-Id: Id14cbd499fd9b36c6926b7552d3554340cb0916c
This reverts commit 96c957ae20.
Fixes issues on some machines where the socket in TMPDIR ended up with a
unix domain socket pathname over 107 characters long, which Go will
reject due to underlying limitations in the system calls. If this
happens, we'll fall back to opening the directory, then using
/proc/self/fd/#/<file>, or manually creating a similar symlink in /tmp.
Also fixes some issues on Mac where os.Executable returns the symlink
instead of the underlying file, sending a message over a unix domain
socket will block if the reader isn't reading, and sandboxing was
preventing us from running `ps`.
Test: m blueprint_tools
Test: m blueprint_tools on mac
Change-Id: Ib19ccfe10cb0a79f1476fb1d5cd20ed0495be367
This will allow us to track (and eventually limit) the commands that the
build references via $PATH. These are mostly implicit dependencies on
the host system -- for Linux, we assume something similar to Ubuntu
14.04 with a few extra packages, but this will let us better define
that.
This will not catch uses of tools with absolute paths (/bin/bash, etc),
but most uses shouldn't be relying on absolute path names anyways.
Adds ~400ms on the first startup, ~140ms on subsequent runs, and
overhead of a few ms for every forwarded execution.
Test: m
Test: build/soong/build_test.bash
Test: Add `gcc --version`, TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: Id68cbb1c8ceef65bbbb10751e83722c7662d2351
Some tests make their temporary directories readonly. If they fail or
crash before cleaning up, they could leave these readonly directories
behind with files in them. os.RemoveAll fails with an error in this
case, and we can't start the build until they're removed.
Test: `m blueprint_tools` to run the new go tests
Change-Id: I761f96579e96167ebfd98c6cca59765bd50536ec
So that we can see where the time is going. Also removes the obsolete
ASAN_OPTIONS, which moved to kati, but I forgot to update this instance.
Test: m nothing; check out/soong.log
Change-Id: I0c4066bad20fc2dc22b389f4c973d10dca554ee3
The find commands used to locate the BoardConfig.mk can't be optimized
by Kati, so we're currently spending ~125ms three times during every
build (dumpvars, cleanspec, and the main kati run). Preserve the value
of TARGET_DEVICE_DIR from the dumpvars run so that we only need to run
the find commands once.
Bug: 78020936
Test: out/build-taimen.ninja is identical
Test: out/soong.log shows that we're not running these finds again
Change-Id: Iee56b454c3661de2b58c161169218ecaf2135398
I got a report of a user exporting CDPATH in their shell and causing
strange issues in the build. We should never need this value (and it
probably shouldn't be exported from the shell either -- that was their
workaround).
NDK_ROOT causes issues with Android.mk files thinking that they're
building with the ndk (like external/googletest/googletest/Android.mk).
Bug: 78933670
Test: CDPATH=. NDK_ROOT=test m; check out/soong.log
Change-Id: Icce43d7e31ed5e5e1fb7a4e37fd4dfbf421af4b1
Many board still have this problem, but if we can switch some over,
we'll prevent global problems and have the ability to clean boards up
one-by-one.
Bug: 77611511
Test: lunch aosp_arm-eng; m nothing
Test: lunch aosp_marlin-eng; m nothing
Test: build_test on all downstream branches
Change-Id: I78bee44adde2059d01188658b9050927748d2028
Some people have run into issues where /tmp is on a smaller partition
than their main source tree, and if it fills up (or only has a few GB
free), their builds can start failing.
There may also be a subset of people whose /tmp may be slower than their
out directory -- if they put their src and out trees on a SSD, but keep
/tmp on a spinning HDD.
Bug: 71755844
Test: check out/soong.log for TMPDIR
Test: do a build, watch out/soong/.temp
Change-Id: Ib749ec736ad0f6c8ce7453861a91b3e99ddbd92d
If Kati ever wrote a line over 64k characters, the scanner in soong_ui
would error, and Kati would hang trying to write into the pipe. Now if
the Scanner errors out, fall back to copying directly from the pipe to
the output.
Test: Add $(warning $(shell find frameworks)), does not hang
Change-Id: I86412ad2f53f2fe3cbda901ee673abb904d56d3c
Soong_UI will update timestamp to build_date.txt, and export variables
to kati/ninja.
Test: m -j32
Bug: b/70351683
Change-Id: I153897afdf2d3f39a32d757d4c3ae7515caea52d
We were previously setting GOROOT to "prebuilts/go/linux-x86" during the
ninja executions when we were running Soong. But we can also run Soong
during the main ninja execution, were GOROOT was unset. When the GOROOT
was unset, the default GOROOT in our Go installation is
"./prebuilts/go/linux-x86" (note the extra ./).
This would cause g.bootstrap.goRoot to change between some soong runs,
causing us to rebuild all go programs (and anything depending on them)
more often than necessary.
So instead, keep GOROOT undefined when running Soong. Everything that
matters is using runtime.GOROOT(), which will fall back to the default.
Continue setting $GOROOT for bootstrap.bash, otherwise it fails when
there is no system provided go binary. What we give bootstrap.bash
doesn't really matter, since we don't actually use the blueprint wrapper
in Android.
Test: m blueprint_tools; touch bionic/libc/tzcode/new.c;
m blueprint_tools <doesn't rebuild everything>
Change-Id: I82f30c7c3b5d25e5cbf28fe37a97fdb776c4a164
This ensures that the current locale supports UTF-8, and that we're
getting a consistent (but still supported by the system) locale for
every configuration except user-facing messages. This should eliminate
any reproducibility problems around sorting, formatting, etc for all
built products, while still showing localized error messages where
available.
Bug: 71573630
Test: LANG=es_ES LANGUAGE=es: m (check env in soong.log)
Change-Id: If33311899eaed8c44573113ee35c5a71cee503a0
This way we strip out ANSI codes when using dumb terminals. It's likely
overkill to use katiRewriteOutput, but we should rarely see any output
on stderr while dumping variables. This also lets us turn on kati_stats.
Bug: 71729611
Test: lunch missing-eng (colored error)
Test: lunch missing-eng 2>&1 | cat (non-colored)
Test: TERM=dumb lunch missing-eng (non-colored)
Change-Id: Ic63fd42d82a4a64e5c68aecd9ae0f242a0d703f1
This architecture only existed for unbundled use, but even the NDK is
removing support in their r17 release, so just remove support for it.
Test: build/soong/build_test.bash -only-soong
Change-Id: I4bd23babf567128d2d242cbdee3311abb198dd7c
This is cherry-picked from attempt 3, which was reverted
due to http://b/70862583.
Before this CL topic, the build toolchain for .java source files
used OpenJDK 8, targeting 1.8 (v52 class files) by default.
This CL topic switches the default to OpenJDK 9, but still
targeting 1.8 (v52 class files) by default. If USE_ERROR_PRONE
is set to true, then the default remains OpenJDK 8.
Code in the Android platform should generally be unaffected,
but if host tools that are now compiled and run using
OpenJDK 9 are causing problems for your team, then let me
know.
To manually switch back to the old behavior for now (continue
using OpenJDK 8), run this command in your shell:
export EXPERIMENTAL_USE_OPENJDK9=false
Bug: 69449021
Test: Treehugger
Test: art/test/testrunner/run_build_test_target.py -j40 art-interpreter
Test: dalvik/dx/tests/run-all-tests
(cherry picked from commit 13f23a2753)
Change-Id: I57abae73f9bdb21ef004a5118ff0e4ef70418ed9
Merged-in: Iac78122f58df0ebbb55134d55021ce6c57351b5f
Previously, these variables were exported only by makevars.go,
but those values are not available to config.mk. This CL adds
the variable to ui/build/config.go, which also makes it
available to config.mk.
Test: Treehugger
Bug: 70862583
Bug: 70521453
Change-Id: Ib54660e4b08ab751265b30004630cf1bb8c0041b
After this revert CL topic, the default toolchain for Android
goes back to being OpenJDK 8.
This revert is being prepared ahead of time in case of
any problems with CL topic:
https://r.android.com/#/q/topic:bug69449021_attempt2
Bug: 69449021
Test: Treehugger
Change-Id: I9ead8d569226bd487baee3c6d5be9ec7033eb56a
(This is cherry-picked from the first attempt to submit this CL
topic, which was reverted after 3 hours because of bug 70286093;
robolectric 3.{1.1,4.2} now stick with OpenJDK 8 to avoid that bug).
Before this CL topic, the build toolchain for .java source files
used OpenJDK 8, targeting 1.8 (v52 class files) by default.
This CL topic switches the default to OpenJDK 9, but still
targeting 1.8 (v52 class files) by default. If USE_ERROR_PRONE
is set to true, then the default remains OpenJDK 8.
Code in the Android platform should generally be unaffected,
but if host tools that are now compiled and run using
OpenJDK 9 are causing problems for your team, then let me
know.
To manually switch back to the old behavior for now (continue
using OpenJDK 8), run this command in your shell:
export EXPERIMENTAL_USE_OPENJDK9=false
Bug: 69449021
Test: Treehugger
Test: Running robolectric tests succeeds on internal-master
after cherry-picking this CL topic, using the command
line from http://b/70286093#comment1
(cherry picked from commit 0ae8b548af)
Change-Id: Ide6a7e55126d919a44f89ef8e0bd14fb12ff470e
to enable other subprojects to use their own instance of
Blueprint in other directories.
Files named Android.bp are still autodetected throughout the
tree like previously.
Bug: 64363847
Test: mkdir -p subdir \
&& echo "syntax error" > Blueprints \
&& m -j nothing
Test: build/soong/scripts/diff_build_graphs.sh \
--products=aosp_arm \
'build/soong:work^' 'build/soong:work'
Change-Id: I8199f12b68dc1699bf44682b86169def37b53a5e
After this revert CL topic, the default toolchain for Android
goes back to being OpenJDK 8.
This revert is being prepared ahead of time in case of
any problems with original change topic,
https://r.android.com/#/q/topic:bug_69449021
Bug: 69449021
Test: Treehugger
This reverts commit 0ae8b548af.
Change-Id: Ief7646a94f1a264085cd299b4327d244b78a1537
Before this CL topic, the build toolchain for .java source files
used OpenJDK 8, targeting 1.8 (v52 class files) by default.
This CL topic switches the default to OpenJDK 9, but still
targeting 1.8 (v52 class files) by default. If USE_ERROR_PRONE
is set to true, then the default remains OpenJDK 8.
Code in the Android platform should generally be unaffected,
but if host tools that are now compiled and run using
OpenJDK 9 are causing problems for your team, then let me
know.
To manually switch back to the old behavior for now (continue
using OpenJDK 8), run this command in your shell:
export EXPERIMENTAL_USE_OPENJDK9=false
Bug: 69449021
Test: Treehugger
Test: "make core-oj", checked that compilation now uses
OpenJDK 9 javac -target 1.8
Test: Checked that this is still compiled using OpenJDK 8.
export EXPERIMENTAL_USE_OPENJDK9=false
make core-oj
Change-Id: Ic87e9bb2a2e5da0ff13a2e51845b5365901c1507
This mostly reverts commit 178d5fefc0
and mostly reapplies change I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c .
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
--products=aosp_arm \
'build/blueprint:work^ build/soong:work^' \
'build/blueprint:work build/soong:work'
# and see that the only changes were:
# 1. adding some new files
# 2. changing some line numbers
Test: m -j nothing # which runs unit tests
Change-Id: I32baae00277a547fdcdd1c2219fe6625ee0e45d7
This reverts commit 63a250a336.
Reason for revert: Some failures:
namespace_test.go:648: dir1/Blueprints:2:4: a namespace must be the first module in the file
such as New Build Breakage: aosp-master/build_test @ 4475274
Change-Id: I1b5db8eb934e51ff22241bfca44199d886b1393b
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
--products=aosp_arm \
'build/blueprint:work^ build/soong:work^' \
'build/blueprint:work build/soong:work'
# and see that the only changes were:
# 1. adding some new files
# 2. changing some line numbers
Test: m -j nothing # which runs unit tests
Change-Id: I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c
This reverts commit fb941913a3.
Reapplies I4933187e8b72f2ef0c32d18ffea756e2c6fa417c with fixes
to disable the check for mac builds, where many modules are
disabled and the check is just going to cause more problems.
Test: m checkbuild
Change-Id: If6712c90ececd5d015fcdcdeefe0c3d4f5590711
Build target test_mapping will create a tarball of all TEST_MAPPING
files in source code based on the cached TEST_MAPPING.list
Bug: 69678490
Test: m -j blueprint_tools && cat out/.module_paths/TEST_MAPPING.list
Change-Id: Iba3479363137609d6631cf1851a8ce4613078f61
This fourth possible value currently has the same semantics
as a default/unset EXPERIMENTAL_USE_OPENJDK9, but allows
people to explicitly switch back to the old semantics when
the default changes.
Test: make showcommands core-oj (in environments with
EXPERIMENTAL_USE_OPENJDK9 set to "", "1.8", and "true").
Bug: 38177295
Change-Id: I25accf14344a05349a6e97572d7c2c1f6a7f2063
Get a list of leaf nodes in the dependency graph
from ninja, and make sure none of them are in the
output directory. This ensures that there are no
rules that depend on a file in the output directory
that doesn't have rule to generate it. The check
will catch a common set of build failures where
a rule to generate a file is deleted (either by
deleting a module in an Android.mk file, or by
modifying the build system incorrectly). These
failures are often not caught by a local incremental
build because the previously built files are still
present in the output directory.
Bug: 36843214
Bug: 68062417
Test: manual
Change-Id: I4933187e8b72f2ef0c32d18ffea756e2c6fa417c
This way config.mk no longer needs to check which java is in PATH and
fix it. It'll be consistent for all build steps under soong_ui.
Also unify handling of ANDROID_JAVA_HOME / JAVA_HOME with
OVERRIDE_ANDROID_JAVA_HOME / EXPERIMENTAL_USE_OPENJDK9.
Test: m nothing
Test: build/soong/soong_ui.bash --make-mode nothing (w/o envsetup.sh)
Test: aosp_arm ninja files are the same before/after
Test: before/after ninja files match with OVERRIDE_ANDROID_JAVA_HOME
Test: before/after ninja files match with EXPERIMENTAL_USE_OPENJDK9
Change-Id: Icdb65093d9c346524074de239a4f895e4230a24d
This may happen if there's been another call to microfactory before
we've read the log file. Just ignore the error and continue.
Test: Run `while true; do get_build_var TARGET_PRODUCT; done` in parallel
Change-Id: I670144f4637281ebf1299dbd9151dd9819c363e3
Create a lock file during log rotation so that multiple processes won't
step on each other.
Test: Run `while true; do get_build_var TARGET_PRODUCT; done` in parallel
Test: m blueprint_tools
Change-Id: I7144cd42aca47c694487ddae44713f82665ed81e
build/soong/soong_ui.bash --dumpvars-mode \
--vars="..." \
--abs-vars="..." \
--var-prefix="..." \
--abs-var-prefix="..."
is similar to the previous:
CALLED_FROM_SETUP=true BUILD_SYSTEM=build/core \
make -f build/core/config.mk dump-many-vars \
DUMP_MANY_VARS="..." \
DUMP_MANY_ABS_VARS="..." \
DUMP_VAR_PREFIX="..." \
DUMP_ABS_VAR_PREFIX="..."
and
build/soong/soong_ui.bash --dumpvar-mode [--abs] VAR
is similar to the previous:
CALLED_FROM_SETUP=true BUILD_SYSTEM=build/core \
make -f build/core/config.mk dumpvar-[abs-]-VAR
But uses soong_ui and ckati, so that we use a consistent make parser and
sandboxing configurations.
One major output difference between the pure make implementation and
this one is that report_config in Go is implemented using embedded
newlines in single quotes, while the make implementation uses `` with
embedded echo commands. This seems to work fine for both bash and zsh,
and report_config isn't meant to be machine-parsed anyways.
Test: build/soong/soong_ui.bash --dumpvar-mode report_config
Test: build/soong/soong_ui.bash --dumpvar-mode TARGET_DEVICE
Test: build/soong/soong_ui.bash --dumpvar-mode --abs PRODUCT_OUT
Test: build/soong/soong_ui.bash --dumpvar-mode --abs ALL_PRODUCTS
Test: build/soong/soong_ui.bash --dumpvars-mode --vars="report_config TARGET_DEVICE" --abs-vars="ALL_PRODUCTS"
Test: build/soong/soong_ui.bash --dumpvars-mode --vars=TARGET_DEVICE --abs-vars=PRODUCT_OUT --var-prefix=v_ --abs-var-prefix=a_
Change-Id: I0fbd0732bbf6fcfcd24084cf3c830a91a4b6bfc2
This reverts commit a3e6c520ac. I fixed
a few heavy stack users, and we've got better debugging.
Bug: 36182021
Test: m dist
Change-Id: Iefd8589185720e44c40a2e0ceff5fd1438211b86
Pass --dump-stats to ckati, but filter out the '*kati*' lines so that
they only end up in our verbose output. That way we've always got access
to the statistics.
Bug: 36182021
Test: m nothing; cat out/soong.log
Change-Id: Iaf7a814fc67f3e475c913faf69924a7f4e2ae3b3
This speeds up dumping make variables from ~380ms using make to ~220ms
using ckati. It also means that we're consistently using the same parser
for builds (with the same .KATI_READONLY/etc extensions).
envsetup.sh (lunch) / other scripts still use make, changing those to go
through soong_ui will be a future change.
Test: m clean; m nothing
Test: USE_GOMA=true m nothing
Test: m PRODUCT-aosp_x86-sdk
Test: m APP-Calculator
Test: build/soong/build_test.bash -only-config (on AOSP and internal master)
Change-Id: I6ca554de8de4955fb869001d06d29969b75751cc
They don't really affect anything general in android except for the
output path.
Bug: 65453318
Test: TARGET_BUILD_TYPE=debug m
Change-Id: I98c17fefae61f9f7936e184bd1f8441086bb7d23
Instead of pairing the reading of CleanSpec.mk files with the reading of
Android.mk files, split them into separate, individually cachable steps.
This way we only read Android.mk files once after a clean/sync. We'll
still read the CleanSpec.mk files multiple times, but that's
significantly faster than reading all the Android.mk files.
This adds about 50ms if kati doesn't need to reread the CleanSpec.mk
files. Reading all the CleanSpec.mk files takes about a second.
Bug: 35970961
Test: m clean; m nothing; m nothing
Test: Add CleanSpec.mk line, see it executed.
Change-Id: I83bad15c50709510959d5b8b673a907b8aa7de82
Kati sets detect_leaks=0 by default now.
Test: unset ASAN_OPTIONS; SANITIZE_HOST=address m nothing
Change-Id: I0cbf04ae9ed40de520abca3fb35c65e7f543682d
We can call directly into the blueprint bootstrap.bash using values that
soong_ui has already calculated.
Instead of calling into blueprint.bash, build minibp with microfactory,
and directly run ninja. This allows us to get individual tracing data
from each component.
Test: m -j blueprint_tools
Test: m clean; m -j blueprint_tools
Change-Id: I2239943c9a8a3ad6e1a40fa0dc914421f4b5202c
The Finder runs roughly 200ms faster than findleaves.py in aosp,
and runs roughly 400ms faster in internal master.
Bug: 64363847
Test: m -j
Change-Id: I62db8dacc90871e913576fe2443021fb1749a483
This reverts commit b6d161bf16.
Reason for revert: New Build Breakage: aosp-master/sdk_mac @ 4260825
Change-Id: I8bda8c50c5e5c9f84621d11a4c15b168833bcd21
This way we only have one way to start a build, which always has logging
/ tracing / etc, even if we don't need Kati.
There's two ways to use this:
As a direct replacement for mkdir out; cd out; ../bootstrap.bash;
./soong -- as long as --skip-make is always passed, we'll never run
Kati, and Soong will run outside of it's "make" mode. This preserves
most of the speed, and allows full user control over the Soong
configuration.
A (experimental, dangerous) way to temporarily bypass the product
variable and kati steps of a build. As long as a user is sure that
nothing has changed from the last build, and they know exactly which
Ninja targets they want to build (which may not be the same as the
arguments normally passed to 'm'), this can lead to shorter build
startup times.
Test: rm -rf out; m --skip-make libc
Test: rm -rf out; m libc; m --skip-make libc
Test: rm -rf out; mkdir out; cd out; ../bootstrap.bash; ./soong libc
Test: build/soong/scripts/build-ndk-prebuilts.sh
Change-Id: Ic0f91167b5779dba3f248a379fbaac67a75a946e
It can find every Android.bp in internal master in
about 2.5 sec the first time and 0.3 sec subsequent times
Bug: 62455338
Test: m -j blueprint_tools # which runs the unit tests
Test: m -j blueprint_tools && \
out/soong/host/linux-x86/bin/finder \
-v
--db /tmp/mydb \
--names Android.mk \
--prune-files .android-out-dir \
--exclude-dirs .git,.repo \
. \
>/tmp/finder-log 2>&1
Change-Id: I5ab2650459a1dae0d5d076faf411ec2d053c743d
Colin Cross