This patchset changes bazel_overlay to generate soong_module as a macro,
instead of a rule, and generate module properties in the BUILD files as
kwargs to the soong_module macro.
Here's a sample of the new BUILD files with module properties:
bionic/libdl/BUILD.bazel:
https://paste.googleplex.com/6484466996346880?raw
art/build/apex/BUILD.bazel:
https://paste.googleplex.com/5461276001042432?raw
bionic/apex/BUILD.bazel:
https://paste.googleplex.com/4932795173437440?raw
soong_module is now a macro that conditionally expands to underlying
soong_<module type> rules with statically defined attributes. In this
CL, we are starting with a hardcoded filegroup rule definition to
demonstrate the conditional rule loading within the soong_module macro.
If the module_type matches an existing Bazel rule, soong_module forwards
the entire **kwargs into the rule, which Bazel typechecks.
Non-filegroup module types will be expanded into generic_soong_module,
but with the kwargs dropped.
This approach allows us to:
1) Programmtically generate soong_<module type> rules for all module
types available in Soong, together with the statically defined attribute
types in `attrs`.
2) Incrementally migrate and test individual module types from
generic_soong_module to their module rule shims.
3) Swap out the module rule shims to the actual Bazel rules (e.g
cc_library, java_library) and perform attribute manipulation in Starlark
itself.
Example of querying against the 'srcs' attribute in soong_filegroup:
```
$ bazel cquery 'kind(soong_filegroup, //...)' | wc -l
590
$ bazel cquery --output=build 'attr(srcs, "linker.cpp",
kind(soong_filegroup, //bionic/...))'
INFO: Analyzed 3907 targets (0 packages loaded, 0 targets configured).
INFO: Found 3907 targets...
/usr/local/google/home/jingwen/aosp/out/soong/bazel_overlay/bionic/linker/BUILD.bazel:4144:13
soong_filegroup(
name = "linker_sources",
generator_name = "linker_sources",
generator_function = "soong_module",
generator_location = "bionic/linker/BUILD.bazel:4144:13",
srcs = ["dlfcn.cpp", "linker.cpp", "linker_block_allocator.cpp",
"linker_dlwarning.cpp", "linker_cfi.cpp", "linker_config.cpp",
"linker_debug.cpp", "linker_gdb_support.cpp", "linker_globals.cpp",
"linker_libc_support.c", "linker_libcxx_support.cpp",
"linker_namespaces.cpp", "linker_logger.cpp",
"linker_mapped_file_fragment.cpp", "linker_phdr.cpp",
"linker_relocate.cpp", "linker_sdk_versions.cpp", "linker_soinfo.cpp",
"linker_tls.cpp", "linker_utils.cpp", "rt.cpp"],
deps = [],
)
/usr/local/google/home/jingwen/aosp/out/soong/bazel_overlay/soong_module.bzl:32:23
in <toplevel>
```
This CL is known to be lacking the following features, and will be looked at in follow up CLs:
1) Pretty printing reflect.Interface properties, like arch, multilib and
dists.
2) Generating module Bazel rule shims for all module types, instead of
hardcoding them like `soong_filegroup`.
Bug: 162720644
Test: bazel_overlay_test.go (soong build test)
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: Ic1e448887eb540ed15a55bc4090cf75a4d832d41
The Bazel overlay is a directory at out/soong/bazel_overlay that
replicates the layout of the AOSP Soong module tree, but as a Bazel
workspace. Each Soong module variant is represented as a BUILD target
created with the `soong_module` rule.
To create this overlay, run `m bazel_overlay`.
A `soong_module` target can depend on other `soong_module` targets.
These dependencies replicate each module's `directDeps` in the Blueprint
graph, just before `PrepareBuildActions`.
This enables users to use bazel query as a way to introspect the Soong
module graph. For example,
- Direct reverse dependencies of //bionic/libc:generated_android_ids in
//bionic/libc/...:
$ bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'
//bionic/libc:libc_bionic_ndk--android_recovery_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_ramdisk_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static_com.android.runtime
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static
//bionic/libc:generated_android_ids
- Why does com.android.runtime depend on lzma?
$ bazel query
'somepath(//bionic/apex:com.android.runtime--android_common_com.android.runtime_image,
//external/lzma/...)'
//bionic/apex:com.android.runtime--android_common_com.android.runtime_image
//bionic/libc/malloc_debug:libc_malloc_debug--android_arm_armv7-a-neon_shared_com.android.runtime
//system/core/libunwindstack:libunwindstack--android_arm_armv7-a-neon_shared_com.android.runtime
//external/lzma/C:liblzma--android_arm_armv7-a-neon_shared_com.android.runtime
- What does the dep graph of //bionic/libc:crtbegin_so look like?
$ bazel query
'deps(//bionic/libc:crtbegin_so--android_arm_armv7-a-neon)'
--output=graph > graph.in && dot -Tpng < graph.in > graph.png
https://photos.app.goo.gl/DfsdoFRNsRjGwTmy8
Test: croot && m bazel_overlay && cd out/soong/bazel_overlay && bazel
query //... && bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I3bf40309bfb2d963bb8a688706385a57ee304c37#
Currently, the build timestamp saved to the build_date_timestamp
field in the MetricsBase metrics protobuf usually comes from a text
file named build_date.txt. It may not be accurate for metrics purposes.
Instead, use the build start time from Soong main UI for a better
timestamp.
Bug: b/140638454
Test: Ran and checked the build_date_timestamp by running the printproto
command.
Change-Id: I4d893fbbf9830ba21911e56ae13bc5272be47ae6
From aosp/1329396, the RBE metrics protobuf file is part of the
metrics uploading process. The RBE metrics protobuf file is
generated by running the bootstrap shutdown command. A new function
named DumpRBEMetrics was written in order to generate the RBE metrics
protobuf file before sending to the uploading process.
Bug: b/140638454
Test: * Unit test cases
* Ran RBE build on my local host and verified the
metrics protobuf file is created.
* Ran non-RBE build after RBE build and verified that
the previous metrics protobuf file was deleted.
Change-Id: I4b8068905cb67c4b8c2d94793917b98974fed707
Soong and Make have no ways to figure out what splits will be outputted
from a given android_app_set, so it's impossible for them to provide
full PACKAGES.$(LOCAL_MODULE).CERTIFICATE entries, which are required to
build a final apkcerts.txt. This change makes extract_apks produce
apkcerts.txt files for each input modules instead. The Make-side
counterpart of this change merges all local apkcerts.txt into a final
one.
Fixes: 160119159
Test: main_test.go
Test: m apkcerts-list
Merged-In: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
Change-Id: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
Specifically, make Android.bp.list and soong.variables explicit inputs
While this is not a comprehensive list of all inputs
of this action (as the action depends on all blueprint
files in the source tree), this is closer to the truth.
This is a rollforward CL, which was originally rolled
back, as path validation failed for when OUT_DIR was
an absolute path. Validation has now been relaxed.
Test: Manually verified ninja output, checkbuild approved validation
for aosp-crosshatch, and manually ran frameworks/rs/build_rs.py with
patch fix
Change-Id: I4eb0d517f57336dd54eaa4bd31f46df9e93e6da2
While this is not a comprehensive list of all inputs of this action (as the action
depends on all blueprint files in the source tree), this closer to the truth.
Test: Manually verified ninja output, checkbuild approved validation for aosp-crosshatch
Change-Id: I77a0f612afb025232bee7d3ac31257b808991829
The metrics uploader was currently running on foreground where it
would copy the metrics files in a separate directory and then forked
into the background for the upload process. As a result, running the
lunch command would take a second longer to run since each metrics
uploader run had an average of half a second.
Bug: 140638454
Test: * Wrote and updated unit test cases.
* Set ANDROID_ENABLE_METRICS_UPLOAD to point to the latest
metrics_uploader bash script. Executed the "lunch 1" command
and measured the running time. Executed "m nothing" command
and checked that the metrics were uploaded.
* Ran "lunch 1" and "m nothing" with
ANDROID_ENABLE_METRICS_UPLOAD=""
* Removed oauth from metrics_uploader and ran "m nothing" and
"lunch 1". The oauth Message appeared only to "m nothing"
Change-Id: I13c61e666c8f44613dee291a704cef6a27335188
Soong now supports the ability to upload metrics to another location
by setting the ANDROID_ENABLE_METRICS_UPLOAD to an uploader that
accepts the upload.proto proto buffer message. When the environment
variable is set, a set of build metrics files (soong_metrics,
rbe_metrics.pb and build_error) is uploaded.
Bug: 140638454
Test: * Wrote unit test cases
* Setup the uploader, built a succcessful and failed aosp_arm-eng
target and monitor the uploading of the metrics.
Change-Id: I76a65739c557dc90345e098ca03119a950ece2d2
For an end user hitting these errors, the most likely reason is that
they have mixed up linker and crtbegin_dynamic.o from different builds
in their out tree.
Test: Build a linux_bionic executable in a tree exhibiting b/157549171.
Test: m nothing
Bug: 157549171
Change-Id: I64bee0848ed61226da6654caf3bafb51cf46d923
apex_available/min_sdk_version are required to be available for mainline
modules(APK/APEX).
- java_import/java_library_static
- android_library/android_library_import
Bug: 156996905
Test: prebuilts/sdk/update_prebuilts.sh
Change-Id: I201ebabaa533c5ee19e394ab30ca2fa0e3d7ce0e
"dlv attach":
- Spams the terminal
- Requires sysctl -w kernel.yama.ptrace_scope=0
- Apparently, does not allow the debugger to inspect variables
Test: Manual.
Change-Id: I625369effaf5abda2b5d884c8ce5bde7247774eb
apex_set takes an .apks file that contains a set of prebuilt apexes with
different configurations. It uses extract_apks to select and install the
best matching one for the current target.
Bug: 153456259
Test: apex_test.go
Test: com.android.media.apks
Change-Id: I1da8bbcf1611b7c580a0cb225856cbd7029cc0a7
This was a temporary migration tool, and has been broken ever since the
ninja environment started being restricted. Remove it, since it has
lasted two releases.
Test: TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: I2e3709525598030f5c24d12fa966e531dfed33ae
Golang's FileInfo() implementation recognizes FAT filesystem directories
correctly, but doesn't actually add executable bits, which essentially
makes them inaccessible. Use 0777 when we know we're dealing with
directories.
(This is a cherry-pick change.)
Fixes: 153207404
Test: manually tested with an aar file with FAT permission bits.
Change-Id: Ie9f5320bc44d24b2f9d54d4036bb1747456655bb
Merged-In: Ie9f5320bc44d24b2f9d54d4036bb1747456655bb
Soong UI generates a proto file named build_progress.pb in $(OUT_DIR)
output directory that contains build action numbers (how many are executing,
finished and total) during the course of a build. This is for external
systems that invokes the Platform Build Systems and would like to know
the completion status.
Bug: b/150401146
Test: Wrote a bash script that continuously read the
build_progress.pb file and computed the build completed percentage
while building the aosp_arm-eng target. Compared the percentage between
the Soong output console and the one reported by the bash script.
Change-Id: I7c7347bc8e41958093892d8e2731c4f4169937dd
Collect the number of modules and variants and some basic statistics
on memory usage inside soong_build by writing out a proto that is
read back in by soong_ui.
Test: examine soong.log
Change-Id: I6926876377a4f6229cf41fdbf166ae03c885ea55
This change automates one of the best practices for
custom tools. The best practice says "Prefer to list
input files on the command line, otherwise we may not
know to re-run your command when a new input file is
added." [1]
Normally you'd reference these inputs with $(in) or
one of the forms of the $(location*) substituions on
the command line for the custom tool. However, if the
custom tool does not accept the list of inputs on the
command line, the build system could fail to re-run
the custom tool if the list changes.
This change adds a hash of the list of input names to
the sbox command that wraps the custom tool. If the
list of inputs change, the hash will change, and
therefore the sbox command will change, causing ninja
to re-run the custom tool. The hash is visible to
(but ignored by) the sbox command, and hidden from
your custom tool.
[1] https://android.googlesource.com/platform/build/soong/+/refs/heads/master/docs/best_practices.md#custom-build-tools
Test: TestGenruleHashInputs
Bug: 149397658
Change-Id: I18b547ea3c4296ee15bd6150a4778a8f376d80b7
Create a highmem pool based on the total RAM and the number of CPUs,
with an override via the NINJA_HIGHMEM_NUM_JOBS variable. Put
metalava into the highmem pool.
Ninja does not support nested pools, and when goma or RBE is enabled
the maximum ninja parallelism is set very high with local jobs in a
local pool. When both the local pool and highmem pool are enabled,
the total number of local jobs will be as high as the sum of the sizes
of the two pools. Keep the highmem pool limited to 1/16th of the
local pool when remote builds are enabled to try to minimize the
effect while still limiting highmem jobs.
Fixes: 142644983
Test: m nothing, examine pools
Test: m USE_GOMA=true nothing, examine pools
Change-Id: Id79f11f44948992960ac34ecf831dacbe21bd332
Test: touch empty.zip; merge_zips out.zip empty.zip
(output should contain `empty.zip: zip: not a valid zip file`)
Change-Id: If707876af2301d7ad7d065c004cffb0c2491c7df
Apparently we have some instances of Soong taking 16GB of RAM. While
that's a problem to solve on it's own, make multiproduct_kati's
auto-parallelism detection smarter by taking into account the total RAM
on the machine instead of just the number of CPUs.
Bug: 146925549
Test: check soong.log for autodetected parallelism values
Change-Id: Ice34c2cf73ea4f8f235521cbefc8654a79a04eef
This is so that we can split the build_test build on large branches into
multiple builds, each testing a subset of the products.
Test: m blueprint_tools (runs the go unit tests)
Test: build/soong/build_test.bash --only-config
vs
Test: build/soong/build_test.bash --only-config --shard-count=2
Test: build/soong/build_test.bash --only-config --shard-count=2 --shard=2
Change-Id: I40ccc1aa477bc0ffa74ff564d155068509be18f0
soong_javac_wrapper is silencing a useless warning:
warning: [options] bootstrap class path not set in conjunction with -source 1.9
but recent versions of javac have started also printing:
1 warning
Read the warning count, subtract the number of silenced warnings, and
reprint it if the non-silenced warning count is nonzero.
Fixes: 144118634
Test: javac_wrapper_test.go
Change-Id: Ie1d0a978188ab7b1c41027f718a1274608628123
Force soong_ui in dumpvars mode to use a dumb terminal so that it
doesn't print an empty status table. Also move the logs to files
that are prefixed with "dumpvars-" so that they don't rotate the
real build logs.
Fixes: 139307849
Test: lunch
Change-Id: If92421af07731fedc9ca42670f309e95922a2224
sbox will generate a random directory for the output root, and most
tools will encode that directory name in the output target of the
depfile.
So embed the library from dep_fixer into sbox so that it can rewrite the
output filename to a static (reproducible) value. Ninja doesn't care
what that value is, so it's just "outputfile".
Also fix up rule_builder to actually tell sbox about the depfile.
Test: mmma system/iorap; check the contents of:
out/soong/.intermediates/system/iorap/libiorap-binder/android_arm_armv7-a-neon_core_static/gen/aidl/system/iorap/binder/com/google/android/startop/iorap/IIorap.cpp.d
Change-Id: I3640a2e8b0c034f143a35e398a8418a6d621b265
Android build generates 180K+ zip files that are to be merged and the
result passed to the indexing pipeline to generate the source code
cross-reference. Their names cannot be passed on the command line,
hence the need for this change. That required extensive changes to
the code as it no longer can be assumed that all the input files can
be kept open.
Bug: 121267023
Test: internal + treehugger + combine index files
Change-Id: I0a829f96ff7187ad967fb6b8cede387501ec57cc
Jingwen Chen