Commit Graph

392 Commits

Author SHA1 Message Date
Colin Cross 00a8a3f746 Add --skip-soong-tests argument to soong_ui and use it in multiproduct_kati
There is no need for multiproduct_kati to run the tests for
every product, they don't vary by product config. --skip-soong-tests
can also be used for local development to run soong_build even if
the tests don't pass.

Bug: 156428456
Test: m --skip-soong-tests nothing
Change-Id: I9c00e3d1b6e51d17bb290339c3f124d4d1c9e69f
2020-10-29 14:22:04 -07:00
Patrice Arruda e92c30dff1 Capture the build command that executed the build system.
We need to capture the user build command to understand what additional
arguments is passed in to Soong. This data is useful to know which
targets are being executed and have statistics on top build targets
are specified.

Bug: b/169453712
Test: ran "m blueprint_tools", "m nothing", "lunch 1" and checked
      in soong_metrics proto.
Change-Id: I7ab94dcf037211f1abffe6afd012b03ffbf99756
2020-10-29 11:01:32 -07:00
Rupert Shuttleworth 680387bf1d Add code to enable USE_BAZEL=1 m functionality in AOSP.
Test: Manually, but builds currently fail due to a recent change to bazel_handler.go.
Change-Id: I9a45a2c73d1755032b11f6b913c03f83a28f63c5
2020-10-26 09:23:24 +00:00
Jingwen Chen 311bd38082 Bazel overlay: write all .bzl files in <bazel_overlay>/build/bazel/overlay_rules directory
This keeps the top level directory clean, and leverages the
new "build/bazel" repo location / package path.

Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...

Bug: 171180724

Fixes: 171180724
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I151b8f3d2177167711e5974e5eabad8d6e910276
2020-10-19 07:59:46 -04:00
Liz Kammer 0e7993e3d2 Clarify error for soong_ui failing to parse args
Test: m soong_ui and give invalid command line args
Change-Id: I7b4479b392d953bd1d87218184c5ed2143568a07
2020-10-15 11:07:13 -07:00
Jingwen Chen 6874dbe104 Stop hardcoding sdk/module_exports dynamic prop types for Bazel overlay.
These sdk/sdk_snapshot/module_exports/module_exports_snapshot properties
are dynamically created using RegisterSdkMemberType [1] and eventually
createDynamicSdkMemberType [2]. All dynamically created sdk member types
are string list types.

Instead of hardcoding the mapping, retrieve them from the registry
instead.

[1]: https://cs.android.com/android/platform/superproject/+/master:build/soong/android/sdk.go;l=489-497;drc=master
[2]: https://cs.android.com/android/platform/superproject/+/master:build/soong/sdk/sdk.go;l=156;drc=master

Bug: 166563303
Bug: 170798271
Fixes: 170798271

Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: Ib77f4e0cc14a7b4ec344a2de115682c234927c16
2020-10-14 04:55:00 -04:00
Treehugger Robot f43eee4b9b Merge "Add warning about out-directory usage with RBE builds" 2020-10-12 15:25:22 +00:00
Colin Cross 405af07859 Revert "Make lots of tests run in parallel"
This reverts commit 323dc60712.

Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests

Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
2020-10-09 18:34:24 -07:00
Christopher Parsons c8f84809ad Merge "Mixed bazel/soong build prototype for genrule" 2020-10-09 14:27:04 +00:00
Chris Parsons f3c96efea4 Mixed bazel/soong build prototype for genrule
With this change, bazel_module is a specifiable property on
genrule module definitions. With bazel-enabled mode, soong_build will
defer to Bazel for information on these modules.

source build/soong/bazelenv.sh to enter bazel-enabled mode.

Test: Manually verified on bionic/libc genrules using aosp_cf_x86_phone-userdebug
Change-Id: I3619848186d50be7273a5eba31c79989b981d408
2020-10-08 22:46:23 -04:00
Kousik Kumar a0a44a84be Add warning about out-directory usage with RBE builds
Prints the following warning messsage:
WARNING:
Setting OUT_DIR to a path other than out may result in slow RBE builds.
See http://go/android_rbe_out_dir for a workaround.

Bug: b/169676232
Change-Id: Id1ad06c4dc672c24373642e0b624833eb6a0dbcf
2020-10-08 02:50:24 -04:00
Colin Cross 323dc60712 Make lots of tests run in parallel
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.

Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
2020-10-06 15:12:22 -07:00
Kousik Kumar ec47864ca0 Deprecate USE_GOMA and replace with FORCE_USE_GOMA
Bug: b/163361527
Test: Ran builds with USE_GOMA / FORCE_USE_GOMA flags
Change-Id: I67b229fbf52e913b0c4f5e7932021132ddc0598f
2020-09-21 15:35:57 -04:00
Jingwen Chen d8004efff2 Generate .bzl rule definitions for every module type in Soong, and
surface module properties as attributes.

This CL maps int, bool, string and string_list module props from Soong
modules into their respective Bazel targets.

With this CL, you can now query modules based on their properties. For
example:

$ bazel query 'attr(nocrt, 1, //...)'
$ bazel query 'attr(apex_available, //apex_available:platform, //...)'
$ bazel query //art/dalvikvm:dalvikvm--linux_glibc_x86_64 --output=build | grep compile_multilib

Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery 'attr(apex_available, com.android.runtime, //...)'

Test: soong_build tests

Fixes: 162720644
Fixes: 164320355
Change-Id: Iea8e594b952feccac3281f36dd6bdee8e7d62c3a
2020-09-09 09:51:45 +00:00
Jingwen Chen ce3d46fe05 Bazel overlay: rename 'deps' to 'module_deps'.
'deps' is a module prop used by some module types, like
prebuilt_build_tool:

https://cs.android.com/android/platform/superproject/+/master:build/soong/android/prebuilt_build_tool.go;l=25-26;drc=751ae87ed29c67629f068a8590af80023a457294

Use `module_deps` to prevent conflicts in the generated overlay
attributes.

Bug: 162720644
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery
//...
Test: soong_build tests

Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: Ia075a860983960b2d9121d30ead8f808be7e0c5b
2020-08-25 06:11:53 +00:00
Jingwen Chen 81a0b66d93 Merge "Surface module properties as Bazel BUILD target attributes in the Bazel overlay" 2020-08-24 22:59:11 +00:00
Jingwen Chen 69d4cbeb04 Surface module properties as Bazel BUILD target attributes in the Bazel overlay
This patchset changes bazel_overlay to generate soong_module as a macro,
instead of a rule, and generate module properties in the BUILD files as
kwargs to the soong_module macro.

Here's a sample of the new BUILD files with module properties:

bionic/libdl/BUILD.bazel:
https://paste.googleplex.com/6484466996346880?raw

art/build/apex/BUILD.bazel:
https://paste.googleplex.com/5461276001042432?raw

bionic/apex/BUILD.bazel:
https://paste.googleplex.com/4932795173437440?raw

soong_module is now a macro that conditionally expands to underlying
soong_<module type> rules with statically defined attributes. In this
CL, we are starting with a hardcoded filegroup rule definition to
demonstrate the conditional rule loading within the soong_module macro.
If the module_type matches an existing Bazel rule, soong_module forwards
the entire **kwargs into the rule, which Bazel typechecks.

Non-filegroup module types will be expanded into generic_soong_module,
but with the kwargs dropped.

This approach allows us to:

1) Programmtically generate soong_<module type> rules for all module
types available in Soong, together with the statically defined attribute
types in `attrs`.
2) Incrementally migrate and test individual module types from
generic_soong_module to their module rule shims.
3) Swap out the module rule shims to the actual Bazel rules (e.g
cc_library, java_library) and perform attribute manipulation in Starlark
itself.

Example of querying against the 'srcs' attribute in soong_filegroup:

```
$ bazel cquery 'kind(soong_filegroup, //...)' | wc -l
590

$ bazel cquery --output=build 'attr(srcs, "linker.cpp",
kind(soong_filegroup, //bionic/...))'
INFO: Analyzed 3907 targets (0 packages loaded, 0 targets configured).
INFO: Found 3907 targets...
/usr/local/google/home/jingwen/aosp/out/soong/bazel_overlay/bionic/linker/BUILD.bazel:4144:13
soong_filegroup(
  name = "linker_sources",
  generator_name = "linker_sources",
  generator_function = "soong_module",
  generator_location = "bionic/linker/BUILD.bazel:4144:13",
  srcs = ["dlfcn.cpp", "linker.cpp", "linker_block_allocator.cpp",
"linker_dlwarning.cpp", "linker_cfi.cpp", "linker_config.cpp",
"linker_debug.cpp", "linker_gdb_support.cpp", "linker_globals.cpp",
"linker_libc_support.c", "linker_libcxx_support.cpp",
"linker_namespaces.cpp", "linker_logger.cpp",
"linker_mapped_file_fragment.cpp", "linker_phdr.cpp",
"linker_relocate.cpp", "linker_sdk_versions.cpp", "linker_soinfo.cpp",
"linker_tls.cpp", "linker_utils.cpp", "rt.cpp"],
  deps = [],
)
/usr/local/google/home/jingwen/aosp/out/soong/bazel_overlay/soong_module.bzl:32:23
in <toplevel>
```

This CL is known to be lacking the following features, and will be looked at in follow up CLs:

1) Pretty printing reflect.Interface properties, like arch, multilib and
dists.
2) Generating module Bazel rule shims for all module types, instead of
hardcoding them like `soong_filegroup`.

Bug: 162720644
Test: bazel_overlay_test.go (soong build test)
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: Ic1e448887eb540ed15a55bc4090cf75a4d832d41
2020-08-21 11:45:46 +00:00
Ramy Medhat f668917c25 Add Goma deprecation PSA to soong.
Test: build with Goma
Change-Id: Ie7e146285afd40bc0a1dc17a0b898960f57a118b
2020-08-18 17:29:48 -04:00
Jingwen Chen 5ba7e479d1 Create a AOSP Bazel overlay workspace with Soong
The Bazel overlay is a directory at out/soong/bazel_overlay that
replicates the layout of the AOSP Soong module tree, but as a Bazel
workspace. Each Soong module variant is represented as a BUILD target
created with the `soong_module` rule.

To create this overlay, run `m bazel_overlay`.

A `soong_module` target can depend on other `soong_module` targets.
These dependencies replicate each module's `directDeps` in the Blueprint
graph, just before `PrepareBuildActions`.

This enables users to use bazel query as a way to introspect the Soong
module graph. For example,

- Direct reverse dependencies of //bionic/libc:generated_android_ids in
//bionic/libc/...:

$ bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'
//bionic/libc:libc_bionic_ndk--android_recovery_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_ramdisk_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static_com.android.runtime
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static
//bionic/libc:generated_android_ids

- Why does com.android.runtime depend on lzma?

$ bazel query
'somepath(//bionic/apex:com.android.runtime--android_common_com.android.runtime_image,
//external/lzma/...)'
//bionic/apex:com.android.runtime--android_common_com.android.runtime_image
//bionic/libc/malloc_debug:libc_malloc_debug--android_arm_armv7-a-neon_shared_com.android.runtime
//system/core/libunwindstack:libunwindstack--android_arm_armv7-a-neon_shared_com.android.runtime
//external/lzma/C:liblzma--android_arm_armv7-a-neon_shared_com.android.runtime

- What does the dep graph of //bionic/libc:crtbegin_so look like?

$ bazel query
'deps(//bionic/libc:crtbegin_so--android_arm_armv7-a-neon)'
--output=graph > graph.in && dot -Tpng < graph.in > graph.png

https://photos.app.goo.gl/DfsdoFRNsRjGwTmy8

Test:  croot && m bazel_overlay && cd out/soong/bazel_overlay && bazel
query //... && bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'

Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I3bf40309bfb2d963bb8a688706385a57ee304c37#
2020-08-06 10:24:21 +00:00
Patrice Arruda 09ef4dc2f7 Update language to comply with inclusive guidance
See https://source.android.com/setup/contribute/respectful-code for
reference

Bug: 161896447
Test: m diff_target_files
Change-Id: I5654ccc5ba9be8cff01689b952d2feaade19ae79
2020-07-31 16:50:52 +00:00
Treehugger Robot 0413d0aa97 Merge "Update language to comply with inclusive guidance" 2020-07-30 18:35:47 +00:00
Patrice Arruda 8da724a64a Update language to comply with inclusive guidance
See https://source.android.com/setup/contribute/respectful-code for
reference

Bug: 161896447
Test: m nothing
Change-Id: Ib54185f2ef9666c7d82afc65fe7fc6bc7fdadfb5
2020-07-28 21:22:12 +00:00
Patrice Arruda f445ba1788 Inclusive fix: renaming dumb terminal to simple terminal.
Bug: b/161896447
Test: "lunch 1" and "m nothing"
Change-Id: Ifb228c0f0f634477932c9240e57170f7390201f7
2020-07-28 20:25:03 +00:00
Patrice Arruda 73c790f59e Use the build start time from Soong main UI for metrics build timestamp.
Currently, the build timestamp saved to the build_date_timestamp
field in the MetricsBase metrics protobuf usually comes from a text
file named build_date.txt. It may not be accurate for metrics purposes.
Instead, use the build start time from Soong main UI for a better
timestamp.

Bug: b/140638454
Test: Ran and checked the build_date_timestamp by running the printproto
      command.
Change-Id: I4d893fbbf9830ba21911e56ae13bc5272be47ae6
2020-07-14 17:23:23 +00:00
Patrice Arruda 62f1bf2378 Add RBE metrics dump in Soong UI.
From aosp/1329396, the RBE metrics protobuf file is part of the
metrics uploading process. The RBE metrics protobuf file is
generated by running the bootstrap shutdown command. A new function
named DumpRBEMetrics was written in order to generate the RBE metrics
protobuf file before sending to the uploading process.

Bug: b/140638454
Test: * Unit test cases
      * Ran RBE build on my local host and verified the
        metrics protobuf file is created.
      * Ran non-RBE build after RBE build and verified that
        the previous metrics protobuf file was deleted.

Change-Id: I4b8068905cb67c4b8c2d94793917b98974fed707
2020-07-09 21:06:22 +00:00
Liz Kammer a93fbacdbe Correct typo of defaults for docs property sorting
Test: manual
Change-Id: Iaca7ba51ca658529dacd2a324412964b2732c4b4
2020-07-07 13:47:29 -07:00
Christopher Parsons 0eeff82c98 Merge "Add missing soong_docs inputs" 2020-07-06 22:34:51 +00:00
Jaewoong Jung 11c1e0f94d Output apkcerts file for android_app_set.
Soong and Make have no ways to figure out what splits will be outputted
from a given android_app_set, so it's impossible for them to provide
full PACKAGES.$(LOCAL_MODULE).CERTIFICATE entries, which are required to
build a final apkcerts.txt. This change makes extract_apks produce
apkcerts.txt files for each input modules instead. The Make-side
counterpart of this change merges all local apkcerts.txt into a final
one.

Fixes: 160119159
Test: main_test.go
Test: m apkcerts-list
Merged-In: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
Change-Id: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
2020-07-01 10:03:09 -07:00
Thiébaud Weksteen 5e291c238f Fix bpfmt issues and add bpfmt to preupload checks
Test: m nothing
Change-Id: I10526a33685335bdd26cc7692815133379d4e0f6
2020-06-29 09:14:08 +02:00
Chris Parsons 8f232a2702 Add missing soong_docs inputs
Specifically, make Android.bp.list and soong.variables explicit inputs

While this is not a comprehensive list of all inputs
of this action (as the action depends on all blueprint
files in the source tree), this is closer to the truth.

This is a rollforward CL, which was originally rolled
back, as path validation failed for when OUT_DIR was
an absolute path. Validation has now been relaxed.

Test: Manually verified ninja output, checkbuild approved validation
  for aosp-crosshatch, and manually ran frameworks/rs/build_rs.py with
  patch fix
Change-Id: I4eb0d517f57336dd54eaa4bd31f46df9e93e6da2
2020-06-26 10:36:18 -04:00
Christopher Parsons cd767eefe5 Revert "Add Android.bp.list and soong.variables to inputs for soong_docs action"
This reverts commit 67eee7cae4.

Reason for revert: Broke renderscript build

Change-Id: Ie26b3c7a78afe44d719f399520e643174109593f
2020-06-18 21:49:48 +00:00
Chris Parsons 67eee7cae4 Add Android.bp.list and soong.variables to inputs for soong_docs action
While this is not a comprehensive list of all inputs of this action (as the action
depends on all blueprint files in the source tree), this closer to the truth.

Test: Manually verified ninja output, checkbuild approved validation for aosp-crosshatch
Change-Id: I77a0f612afb025232bee7d3ac31257b808991829
2020-06-18 12:06:37 -04:00
Patrice Arruda 7cc2074885 Run the metrics uploader in the background.
The metrics uploader was currently running on foreground where it
would copy the metrics files in a separate directory and then forked
into the background for the upload process. As a result, running the
lunch command would take a second longer to run since each metrics
uploader run had an average of half a second.

Bug: 140638454
Test: * Wrote and updated unit test cases.
      * Set ANDROID_ENABLE_METRICS_UPLOAD to point to the latest
        metrics_uploader bash script. Executed the "lunch 1" command
	and measured the running time. Executed "m nothing" command
	and checked that the metrics were uploaded.
      * Ran "lunch 1" and "m nothing" with
        ANDROID_ENABLE_METRICS_UPLOAD=""
      * Removed oauth from metrics_uploader and ran "m nothing" and
        "lunch 1". The oauth Message appeared only to "m nothing"

Change-Id: I13c61e666c8f44613dee291a704cef6a27335188
2020-06-13 00:34:37 +00:00
Treehugger Robot 039b04f4fc Merge "Upload build metrics after a build is completed." 2020-06-01 20:44:54 +00:00
Patrice Arruda 219eef3878 Upload build metrics after a build is completed.
Soong now supports the ability to upload metrics to another location
by setting the ANDROID_ENABLE_METRICS_UPLOAD to an uploader that
accepts the upload.proto proto buffer message. When the environment
variable is set, a set of build metrics files (soong_metrics,
rbe_metrics.pb and build_error) is uploaded.

Bug: 140638454
Test: * Wrote unit test cases
      * Setup the uploader, built a succcessful and failed aosp_arm-eng
        target and monitor the uploading of the metrics.
Change-Id: I76a65739c557dc90345e098ca03119a950ece2d2
2020-06-01 17:29:30 +00:00
Jooyung Han 9a37d4a602 Merge "pom2bp: add apex_available/min_sdk_version" 2020-05-28 09:09:59 +00:00
Martin Stjernholm 321bbe92b1 Explain checkLinker errors a bit more.
For an end user hitting these errors, the most likely reason is that
they have mixed up linker and crtbegin_dynamic.o from different builds
in their out tree.

Test: Build a linux_bionic executable in a tree exhibiting b/157549171.
Test: m nothing
Bug: 157549171
Change-Id: I64bee0848ed61226da6654caf3bafb51cf46d923
2020-05-27 21:31:42 +01:00
Lukács T. Berki ad37304762 Merge "Call Delve using exec() instead of "dlv attach"." 2020-05-27 08:33:32 +00:00
Jooyung Han 43d3025cd5 pom2bp: add apex_available/min_sdk_version
apex_available/min_sdk_version are required to be available for mainline
modules(APK/APEX).

- java_import/java_library_static
- android_library/android_library_import

Bug: 156996905
Test: prebuilts/sdk/update_prebuilts.sh
Change-Id: I201ebabaa533c5ee19e394ab30ca2fa0e3d7ce0e
2020-05-27 00:37:45 +09:00
Sasha Smundak 827c55f502 Handle universal APK case in extract_apks
Cherrypick of ag/11553550

Bug: 157081961
Test: manual and builtin

Merged-In: I5ac64c845328a54024171da41c369050243462b0
Change-Id: I3ebc4a84501623774b78c9c19628994c1d71dc64
2020-05-20 13:10:59 -07:00
Lukacs T. Berki a5e0f710c2 Call Delve using exec() instead of "dlv attach".
"dlv attach":

- Spams the terminal
- Requires sysctl -w kernel.yama.ptrace_scope=0
- Apparently, does not allow the debugger to inspect variables

Test: Manual.
Change-Id: I625369effaf5abda2b5d884c8ce5bde7247774eb
2020-05-18 09:50:18 +02:00
Jaewoong Jung fa00c06187 Add apex_set module.
apex_set takes an .apks file that contains a set of prebuilt apexes with
different configurations. It uses extract_apks to select and install the
best matching one for the current target.

Bug: 153456259
Test: apex_test.go
Test: com.android.media.apks
Change-Id: I1da8bbcf1611b7c580a0cb225856cbd7029cc0a7
2020-05-15 11:02:01 -07:00
Sasha Smundak 7a894a6643 Implement extract_apks
Bug: 152319766
Test: manual and builtin
Change-Id: Ia15d66e86c7bcfd52f5b776173ca1665b68ff438
2020-05-14 19:00:59 -07:00
Dan Willemsen 347ba75679 Remove TEMPORARY_DISABLE_PATH_RESTRICTIONS
This was a temporary migration tool, and has been broken ever since the
ninja environment started being restricted. Remove it, since it has
lasted two releases.

Test: TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: I2e3709525598030f5c24d12fa966e531dfed33ae
2020-05-04 19:54:57 +00:00
Jaewoong Jung a86983dc7a Use 0777 instead of FileInfo() for dirs.
Golang's FileInfo() implementation recognizes FAT filesystem directories
correctly, but doesn't actually add executable bits, which essentially
makes them inaccessible. Use 0777 when we know we're dealing with
directories.

(This is a cherry-pick change.)

Fixes: 153207404
Test: manually tested with an aar file with FAT permission bits.
Change-Id: Ie9f5320bc44d24b2f9d54d4036bb1747456655bb
Merged-In: Ie9f5320bc44d24b2f9d54d4036bb1747456655bb
2020-04-06 10:41:40 -07:00
Patrice Arruda 74b43998e7 Add build progress in Soong UI
Soong UI generates a proto file named build_progress.pb in $(OUT_DIR)
output directory that contains build action numbers (how many are executing,
finished and total) during the course of a build. This is for external
systems that invokes the Platform Build Systems and would like to know
the completion status.

Bug: b/150401146
Test: Wrote a bash script that continuously read the
build_progress.pb file and computed the build completed percentage
while building the aosp_arm-eng target. Compared the percentage between
the Soong output console and the one reported by the bash script.

Change-Id: I7c7347bc8e41958093892d8e2731c4f4169937dd
2020-03-19 13:15:05 -07:00
Colin Cross 9403e4b704 Merge "Collect metrics from inside soong_build" 2020-02-28 23:23:29 +00:00
Colin Cross b72c909899 Collect metrics from inside soong_build
Collect the number of modules and variants and some basic statistics
on memory usage inside soong_build by writing out a proto that is
read back in by soong_ui.

Test: examine soong.log
Change-Id: I6926876377a4f6229cf41fdbf166ae03c885ea55
2020-02-25 20:50:00 +00:00
Bill Peckham c087be1e8b Automate dependency on inputs of genrule module type.
This change automates one of the best practices for
custom tools. The best practice says "Prefer to list
input files on the command line, otherwise we may not
know to re-run your command when a new input file is
added." [1]

Normally you'd reference these inputs with $(in) or
one of the forms of the $(location*) substituions on
the command line for the custom tool. However, if the
custom tool does not accept the list of inputs on the
command line, the build system could fail to re-run
the custom tool if the list changes.

This change adds a hash of the list of input names to
the sbox command that wraps the custom tool. If the
list of inputs change, the hash will change, and
therefore the sbox command will change, causing ninja
to re-run the custom tool. The hash is visible to
(but ignored by) the sbox command, and hidden from
your custom tool.

[1] https://android.googlesource.com/platform/build/soong/+/refs/heads/master/docs/best_practices.md#custom-build-tools

Test: TestGenruleHashInputs
Bug: 149397658
Change-Id: I18b547ea3c4296ee15bd6150a4778a8f376d80b7
2020-02-19 12:47:35 -08:00
Treehugger Robot 7c78b3c98e Merge "If input zip file cannot opened, print its name." 2020-01-28 01:48:32 +00:00
Colin Cross 8b8bec3b3a Create a highmem pool and put metalava into it
Create a highmem pool based on the total RAM and the number of CPUs,
with an override via the NINJA_HIGHMEM_NUM_JOBS variable.  Put
metalava into the highmem pool.

Ninja does not support nested pools, and when goma or RBE is enabled
the maximum ninja parallelism is set very high with local jobs in a
local pool.  When both the local pool and highmem pool are enabled,
the total number of local jobs will be as high as the sum of the sizes
of the two pools.  Keep the highmem pool limited to 1/16th of the
local pool when remote builds are enabled to try to minimize the
effect while still limiting highmem jobs.

Fixes: 142644983
Test: m nothing, examine pools
Test: m USE_GOMA=true nothing, examine pools
Change-Id: Id79f11f44948992960ac34ecf831dacbe21bd332
2020-01-24 11:44:23 -08:00
Sasha Smundak 6172491b91 If input zip file cannot opened, print its name.
Test: touch empty.zip; merge_zips out.zip empty.zip
(output should contain `empty.zip: zip: not a valid zip file`)

Change-Id: If707876af2301d7ad7d065c004cffb0c2491c7df
2020-01-22 10:21:43 -08:00
Dan Willemsen 1bdbdec5a0 Take into account RAM usage for multiproduct_kati
Apparently we have some instances of Soong taking 16GB of RAM. While
that's a problem to solve on it's own, make multiproduct_kati's
auto-parallelism detection smarter by taking into account the total RAM
on the machine instead of just the number of CPUs.

Bug: 146925549
Test: check soong.log for autodetected parallelism values
Change-Id: Ice34c2cf73ea4f8f235521cbefc8654a79a04eef
2019-12-27 09:54:11 -08:00
Dan Willemsen 9609ad9c21 Add sharding support to multiproduct_kati
This is so that we can split the build_test build on large branches into
multiple builds, each testing a subset of the products.

Test: m blueprint_tools  (runs the go unit tests)
Test: build/soong/build_test.bash --only-config
      vs
Test: build/soong/build_test.bash --only-config --shard-count=2
Test: build/soong/build_test.bash --only-config --shard-count=2 --shard=2

Change-Id: I40ccc1aa477bc0ffa74ff564d155068509be18f0
2019-12-05 19:54:40 -08:00
Colin Cross e94272d0b3 Silence trailing "1 warning" message from javac when warning is silenced
soong_javac_wrapper is silencing a useless warning:
warning: [options] bootstrap class path not set in conjunction with -source 1.9
but recent versions of javac have started also printing:
1 warning

Read the warning count, subtract the number of silenced warnings, and
reprint it if the non-silenced warning count is nonzero.

Fixes: 144118634
Test: javac_wrapper_test.go
Change-Id: Ie1d0a978188ab7b1c41027f718a1274608628123
2019-11-11 21:48:33 +00:00
Colin Cross c0b9f6b9d2 Force dumpvars into dumb terminal mode and move log files
Force soong_ui in dumpvars mode to use a dumb terminal so that it
doesn't print an empty status table.  Also move the logs to files
that are prefixed with "dumpvars-" so that they don't rotate the
real build logs.

Fixes: 139307849
Test: lunch
Change-Id: If92421af07731fedc9ca42670f309e95922a2224
2019-09-23 14:24:57 -07:00
Colin Cross b5f6dc47ee Fix lint error in format string
Test: go test
Change-Id: I8b7df9222db1fc88938cdfa43a6505f2da854876
2019-09-11 17:06:02 +00:00
Dan Willemsen f80ef34c80 Merge "Rewrite depfile from sbox to stay reproducible"
am: e2ce551909

Change-Id: I2a51549dfb4419b9b6444d33104d35e655c82f4f
2019-09-04 12:38:05 -07:00
Dan Willemsen c89b6f1981 Rewrite depfile from sbox to stay reproducible
sbox will generate a random directory for the output root, and most
tools will encode that directory name in the output target of the
depfile.

So embed the library from dep_fixer into sbox so that it can rewrite the
output filename to a static (reproducible) value. Ninja doesn't care
what that value is, so it's just "outputfile".

Also fix up rule_builder to actually tell sbox about the depfile.

Test: mmma system/iorap; check the contents of:
out/soong/.intermediates/system/iorap/libiorap-binder/android_arm_armv7-a-neon_core_static/gen/aidl/system/iorap/binder/com/google/android/startop/iorap/IIorap.cpp.d

Change-Id: I3640a2e8b0c034f143a35e398a8418a6d621b265
2019-08-29 14:47:40 -07:00
Alexander Smundak 73d9ff761f Merge "Allow reading zipfile list from a file."
am: 02ed5df4ac

Change-Id: I2888ac8c6b03bcfc09739e5329bc872d92a2e9a1
2019-08-07 10:16:20 -07:00
Sasha Smundak 1459a9259e Allow reading zipfile list from a file.
Android build generates 180K+ zip files that are to be merged and the
result passed to the indexing pipeline to generate the source code
cross-reference. Their names cannot be passed on the command line,
hence the need for this change. That required extensive changes to
the code as it no longer can be assumed that all the input files can
be kept open.

Bug: 121267023
Test: internal + treehugger + combine index files
Change-Id: I0a829f96ff7187ad967fb6b8cede387501ec57cc
2019-08-06 14:23:35 -07:00
Dan Willemsen 57d5786d59 Merge "Remove support for ONE_SHOT_MAKEFILE"
am: d778b93168

Change-Id: I32ef15cf94df3524e4e805327d4af246c95226a6
2019-08-01 14:31:53 -07:00
Dan Willemsen ce41e943be Remove support for ONE_SHOT_MAKEFILE
Test: treehugger
Change-Id: Icdbe62bcb7bc4717228c5d974962b939d8eafee6
2019-07-31 18:11:27 -07:00
Dan Willemsen 12efb0533e Merge "Deprecate ONE_SHOT_MAKEFILE"
am: 1d73e5e566

Change-Id: Ie7ec9690ada5333a04bae34b7be2b1921863e0fc
2019-07-29 13:19:26 -07:00
Dan Willemsen c6360837bb Deprecate ONE_SHOT_MAKEFILE
mm is now doing the same thing as mma. Now print a deprecation message
for other tools that may be setting ONE_SHOT_MAKEFILE directly (or users
that are still using a very old envsetup.sh).

Test: mm  # does not print
Test: ONE_SHOT_MAKEFILE=bionic m
Change-Id: I98221657024d824a61c0d721d5c7dd1a2ea2fb1e
2019-07-25 14:12:38 -07:00
Tony Mak 6f716787d1 Merge "Allow other module types in androidx"
am: 6edff69513

Change-Id: I225050ef2c10850eee20820d4602399f5402557b
2019-07-22 14:36:55 -07:00
Tony Mak 817850063b Allow other module types in androidx
1. Update pom2bp to support "host_support" device module.
2. Allow other module type in support library

(The intention is to support the room annotation processor)

Test: m

Bug: 137918883

Change-Id: I9a8fbcff903cc395d5dc38430774202ce389d6bc
Merged-In: I9a8fbcff903cc395d5dc38430774202ce389d6bc
(cherry picked from commit bd7c2f9a1d)
2019-07-22 13:18:48 -07:00
Patrice Arruda fe6a621684 Merge "soong-ui: Redirect mmm to mmma."
am: 216e315a34

Change-Id: If9a777fd46c06d06fedeee558d68653c72a726df
2019-07-19 04:50:34 -07:00
Treehugger Robot 216e315a34 Merge "soong-ui: Redirect mmm to mmma." 2019-07-18 21:58:24 +00:00
Patrice Arruda 0cacc0468f Merge "soong-ui: Redirect mm to mma."
am: 80b562eb5b

Change-Id: I4f24fd0a055c7d7e80ff61d4c5ed1cd695ea4d6e
2019-07-18 14:06:45 -07:00
Patrice Arruda a9e1215870 soong-ui: Redirect mmm to mmma.
This is the first part of deprecating the mmm command. mmm will simply
be redirected to mmma by building the list of modules with their
dependencies.

Bug: b/135187558
Test: Ran mmm external/libusb:libusb.
Change-Id: I50cf0d6a07197556ab6130e612561c6d5a3fb31e
2019-07-18 09:43:20 -07:00
Patrice Arruda 2274102f34 soong-ui: Redirect mm to mma.
This is the first part of deprecating the mm command. mm will simply
be redirected to mma by building the modules and their dependencies.

Bug: b/135187558
Test: Ran mm in external/protobuf directory.
Change-Id: Idad40f89e3768aeb459172f632a44fd7b0a201bc
2019-07-18 09:41:19 -07:00
Justin Yun 9321804a11 Rename product_services to system_ext
am: d5f6c8261a

Change-Id: Ifd77d7d729addae59b2b26a8cbdd2ebefffcabae
2019-07-09 02:14:59 -07:00
Justin Yun d5f6c8261a Rename product_services to system_ext
Bug: 134359158
Test: build and check if system_ext.img is created
Change-Id: Ice3229baf97a94c24a6eea22e7d4571941d7f843
2019-07-09 08:57:05 +00:00
Colin Cross cd48600321 Merge changes Icfc893c8,I40f03fc0
am: f67e1bee4a

Change-Id: I0b9cf1e3df4ef7ef46e73a5dc6d363fe918c29b7
2019-07-03 22:55:03 -07:00
Colin Cross aa812d122c Allow debugging with SOONG_DELVE=<listen addr>
Allow running Soong in a headless delve debugger by passing
SOONG_DELVE=<listen addr> in the environment.

Bug: 80165685
Test: SOONG_DELVE=:1234 m nothing
Change-Id: Icfc893c8a8354a9bbc99112d9c83259cb41906d1
2019-07-03 16:33:31 -07:00
Colin Cross a73c8a4e61 Merge "Compute critical path when ninja finishes" am: 7c9b4cf144
am: 7b4a190539

Change-Id: I1f3869fa3d5afb437bab8a4bf77ea03934c1d4ab
2019-06-26 16:53:05 -07:00
Colin Cross 7b6245388c Compute critical path when ninja finishes
Keep a running map of the critical path to each edge as it
finishes.  At the end of the build, find the edge with the
longest critical path and print the path to out/soong.log.

Test: critical_path_test.go
Change-Id: Ie01d26b068b768156b166bff00cdc3273e8124ca
2019-06-26 11:31:17 -07:00
Patrice Arruda 55ac3ddb21 Merge "Soong: Add build_error proto raw file to $(OUT_DIR)." am: 7d7cae85ab
am: 9f9778cff0

Change-Id: I7d2ee16ad31446360142295c61562cff467f9d19
2019-06-24 16:43:43 -07:00
Patrice Arruda 297cebad26 Soong: Add build_error proto raw file to $(OUT_DIR).
Soong_ui produces a build_error raw file to $(OUT_DIR) where
the file contains a list of build action errors. Each build action
error represents an error raised by a command. The build_error file
is populated if there was a build error.

Bug: b/132969697
Test: Ran m for successful build. Introduced a broken build change,
      ran m and verified using printproto that build_error was
      generated successfully.

Change-Id: I690ca1778b4e56f144a3173ba1d16d8494c32c15
2019-06-24 13:02:35 -07:00
Patrice Arruda 4b557d8f86 Merge "Soong: Add BUILD_MODULES build action in soong_ui." am: 45cdb71290
am: d009ef35b8

Change-Id: I0216045ba79f0cc26a4de5883589f52a65d9919c
2019-06-20 20:59:32 -07:00
Patrice Arruda 3928206ecc Soong: Add BUILD_MODULES build action in soong_ui.
There was one case that did not work:
  1) Create dir in root source tree
  2) Run "m libc"

The build would fail as it was using the mma logic. A separate
one named BUILD_MODULES was added to allow building specific
modules in any directories using the "m" command.

Bug: b/130049705
Test: unit test cases, ran "m libc" in a tmp directory inside
      of the source tree.

Change-Id: I8d23e685a673a311001ee8edd89bd73b662392dd
2019-06-20 16:35:12 -07:00
Patrice Arruda 5dbf869760 Soong: Use ctx.Writer in flag usage for dumpVar and dumpVars in soong_ui. am: db4c2f115e
am: e6fe9edbef

Change-Id: Ifcd0332f5f055633e4284c3387beea84cedf86a7
2019-06-18 15:52:12 -07:00
Patrice Arruda db4c2f115e Soong: Use ctx.Writer in flag usage for dumpVar and dumpVars in soong_ui.
ctx.Writer is recommended stream to use for standard io.

Fixes: b/135487952
Test: Lunch and executed ./out/soong_ui --dumpvar-mode and
--dumpvars-mode to check the usage is output to the console.

Change-Id: I38351ebdb05f49f92d6b4bfb0423092299d1c5ea
2019-06-18 20:32:01 +00:00
Patrice Arruda 268a702f06 Merge "soong_ui: Add --build-mode flag in soong_ui" am: 2f93a2e07b
am: 9123c55392

Change-Id: Ib7ca530857ad469da3e634b7115a308b6d0d1972
2019-06-17 21:25:36 -07:00
Patrice Arruda b7b2282fd1 soong_ui: Add --build-mode flag in soong_ui
The --build-mode flag is a new flag in soong_ui that accepts a build
action (m, mm, mmm, mma, mmma), the directory where the build action
is occuring and an optional list of build arguments and invokes the
build option. This is to deprecate the build actions behavior in
envsetup.sh and start using the one defined in soong_ui.

Bug: b/130049705
Test: Ran lunch and ran all the build action commands in soong_ui.
      Ran a bunch of invalid commands such as no Android.mk file in
      a specific directory, invalid target name, etc...

Change-Id: I88e9e59f47f8c5cdc872fb83a0f3f0a0919885b6
2019-06-17 17:21:32 -07:00
Patrice Arruda 10b659a60f Merge "Soong: Fix the package name in metrics.proto file" am: eec8d3aee3
am: 33989c88c3

Change-Id: Ife7e9c2ddeeeda40976c23f8ba07ba4bee53935c
2019-06-17 17:12:42 -07:00
Patrice Arruda 0cc5b21516 Soong: Fix the package name in metrics.proto file
The metrics.proto needs to be imported to Google3 in order to unmarshal
the build metrics data from bigstore. Cleaned up the enum names to use
the Camel naming convention and renamed the build_metrics to
soong_metrics to be more specific to soong and to allow quering
soong_metrics files from bigstore.

Bug: b/135280521
Test: lunch andf m -j. Checked if soong_metrics was generated correctly
      by using printproto command.

Change-Id: I998c8d05db592e94a653d6ca32250b80df3c9b21
2019-06-14 15:27:46 -07:00
Colin Cross ceec6a2295 Merge changes I555ad21a,I9d46761e,I917bdeee,I5bc5476a,I59bbdae4, ... am: 6ebe07def7
am: 1337decdff

Change-Id: I2f27851391ad68137cc66133de575d75a3c83a74
2019-06-14 01:30:19 -07:00
Colin Cross e0df1a36b2 Move all output through StatusOutput
Write log output through StatusOutput so that the status implementation
can synchronize it with its own output.

Test: status_test.go
Change-Id: I917bdeeea4759a12b6b4aa6d6d86ee18a2771723
2019-06-12 21:30:56 -07:00
Colin Cross 097ed2a37c Remove terminal.Writer
terminal.Writer is now just a wrapper around stdio.Stdout() without
any useful functionality.  Replace it with stdio.Stdout() as an
io.Writer.

Test: ui/terminal/status_test.go
Change-Id: I5bc5476afdca950b505642f0135a3af9d37fbe24
2019-06-12 21:30:56 -07:00
Dan Willemsen 7488174f2c Merge "Refactor .aar resource compilation" am: 588857b418
am: 2758050dc4

Change-Id: I98dfbdbef71972a040fd53f9e77eb321fb6f5e0e
2019-06-10 21:16:55 -07:00
Dan Willemsen 304cfec778 Refactor .aar resource compilation
Instead of extracting the aar, and passing the res directory to another rule
(along with a dependency to a different known file), don't keep the
extracted res directory, and pass the aar directly to the resource
compliation rule, which will extract just the res directory.

I need this for my RBE experiments, where non-listed output files won't
exist in other rules.

Test: m
Change-Id: I99074381052cbcebb6a402484abae9ab2e40284f
2019-05-29 16:54:43 -07:00
Patrice Arruda d2e1149cc5 resolve merge conflicts of a3898089ba to qt-dev-plus-aosp
Bug: None
Test: Ran soong_ui.bash which compiles the soong_ui.
Change-Id: I0b8083e3e5ee69c8cbf17443fbc038ec7b5a0b10
2019-05-06 19:24:32 -07:00
Treehugger Robot 565ee6ed87 Merge "Soong: Refactor the soong_ui arguments processing to be more like bazel." 2019-05-07 00:49:21 +00:00
Patrice Arruda a5c2542f68 Soong: Refactor the soong_ui arguments processing to be more like bazel.
Currently, the command line argments is being processed in multiple
places. In the main soong_ui arguments, there are several if statements
that checks if a specific argument was specified in order to execute
the requested operation. This does not scale well when adding or
removing a command in the near future. In order to support the build
commands (m, mma, etc...) from the envsetup.sh in soong_ui, a refactor
was required in order to add a command rather quickly and to have the
flexibiity to unit test the command. The soong_ui arguments format is
as follows:

soong_ui <command> [<arg 1> <arg 2> ... <arg n>]

The <command> is a specific operation to be executed. The arguments
after the command are processed by the command itself.

Below is the list of changes made in this commit:

 * Created a new command infrastructure that allows adding
   or deprecating a command easily.

 * Fixed a bug when running ./soong_ui.bash directly would cause
   a panic due to index being out of range for args list.

Bug: b/130049705
Test: Below is the list of testing done on this commit:
    * Ran lunch and verified that the output is the same as the the output
      without the modifications. lunch indirectly runs soong_ui passing
      in the --dumpvar-mode (to read makefile variables such as
      TARGET_PRODUCT) and --dumpvars-mode (to build the build cache).
    * Ran ./soong_ui.bash directly (with unsupported flags and no flags)
      and the proper message appeared that soong native UI is not yet
      available.
    * Ran m, mm, mmm, mma, mmma commands.
    * Ran the make installclean command:
         make -j80 TARGET_PRODUCT=aosp_arm64 TARGET_BUILD_VARIANT=eng dist DIST_DIR=/tmp/helloworld installclean
    * Ran ./out/soong_ui -j80 --make-mode PRODUCT-aosp_arm64-eng dist checkbuild tests

Change-Id: Iee4de7ec0fa4661206fda8ae1fe6fa4487d9bb22
Merged-In: Iee4de7ec0fa4661206fda8ae1fe6fa4487d9bb22
2019-05-06 14:05:52 -07:00
Colin Cross f41ed9ae63 Merge "Remove services.art from known_nondeterminism.whitelist" am: 42603c4ab5
am: 828a488720

Change-Id: Idbdeec76c0d8207890afbf94ea20ebac4c3d5379
2019-04-30 15:24:25 -07:00
Colin Cross 497d4c24b1 Remove services.art from known_nondeterminism.whitelist
Bug: 120039850
Test: none
Change-Id: I70d5243f427a4da5c8126be6b8ebc90221133ad3
2019-04-29 17:36:19 +00:00
Dan Willemsen 7d155094be Merge "pom2bp: Implement -static-deps to match pom2mk" am: 80ae6816ae
am: 9181e08738

Change-Id: I331a7bb1cd82315f673d9007c2240da423864034
2019-04-22 20:17:35 -07:00
Dan Willemsen 52c90d868e pom2bp: Implement -static-deps to match pom2mk
This changes the default behavior of pom2bp to match pom2mk, and not
require pre-extraction of all of the manifests.

Test: convert an internal pom2mk user to pom2bp
Change-Id: I08120cce1f923ef28519563ffd772032b2e500ed
2019-04-21 21:46:42 -07:00