This lets one avoid any decisions as to when to chdir there during its
execution and leads to better sandboxing because the pwd doesn't leak to
init() functions anymore.
Test: Manual.
Change-Id: I1560da8ed3a621249426f9e8908aa890c21e13ba
Also delete the now-unnecessary soong_env binary.
Test: Manually. Also checked what happens when a used environment
variable changes.
Change-Id: Ib393e7f444e94819198c6cce4bcd8719d9fd9441
Sample output:
[bp2build] cc_library_headers: 5 targets
[bp2build] cc_object: 5 targets
[bp2build] filegroup: 4 targets
[bp2build] genrule: 4 targets
[bp2build] sh_binary: 1 targets
[bp2build] Generated 19 total BUILD targets from 39270 Android.bp modules.
This CL adds an additional CodegenMetrics return value to
GenerateBazelTargets calls, which are called from bp2build, queryview,
and their tests. For this UI, we only want to use it for bp2build, and
not queryview or tests, since it's not useful for the former, and can
pollute the CLI for the latter.
Test: build/bazel/scripts/milestone-2/demo.sh
Change-Id: Ic84307a1ed1a25e360c9b23459e5449d932bc2e7
Before this change, changes to any Android.bp file would not trigger a re-run of bp2build/Soong, since the depfile for Soong's main output, the build.ninja file, didn't contain the list of Android.bp files. bp2build only reran when Soong itself changed, which happens regularly while developing bp2build, but not at all while modifying Android.bp files.
This change correctly dumps the list of all Android.bp files in the tree to the depfile for out/soong/build.ninja, therefore correctly triggering a re-run of bp2build when any Android.bp file changes.
Test: GENERATE_BAZEL_FILES=true m nothing; edit an Android.bp file; GENERATE_BAZEL_FILES=true m nothing -> successfully re-runs and regenerates out/soong/bp2build.
Test: inspect out/soong/build.ninja.d manually
Fixes: 179452534
Change-Id: Iad80c381b3afa38a4d4fafca035e46aa2a1f206a
bp2build is a Soong mode that returns as soon as the BUILD files are generated. This causes it not generate the build.ninja file, which the Ninja process executing soong_build itself expects to produce as an output. If there isn't an existing build.ninja file generated from a previous build, GENERATE_BAZEL_FILES=true m nothing will fail.
This CL generates the expected files as a workaround, and also makes GENERATE_BAZEL_FILES=true skip the Kati/Ninja steps in soong_ui since they aren't needed.
Test: rm -rf out/ && GENERATE_BAZEL_FILES=true m nothing && m libc
Fixes: 178683777
Change-Id: I2515ef7961682d2be5f096ed24831cc185165a67
The CodegenMode enum helps to differentiate between bp2build and queryview modes of generating BUILD files from the Soong module graph.
bp2build is used for generating BUILD files from an unconfigured Soong module graph for Soong->Bazel conversion, whereas QueryView is a front-end for querying the existing Soong configured module graph with Bazel query.
Test: go test
Test: TH
Change-Id: I5709f591f1ae08b2770adf272d8ce60c9ee1f46a
This CL creates the framework necessary for generating
BazelTargetModules from regular Soong Android modules.
BazelTargetModules are code-generated into Bazel targets in BUILD files.
See the follow-up CL for examples of creating filegroup/genrule
BazelTargetModules.
Test: GENERATE_BAZEL_FILES=true m nothing # creates out/soong/bp2build
with no BUILD files, because there are no BazelTargetModules in the
module graph.
Change-Id: I33a96365bd439043b13af6db9e439592e9983188
Implement bp2build codegen as a discrete step that runs after an
alternatively registered pipeline of mutators, instead of a
presingleton.
bp2build codegen requires a Context that supports VisitAllModules and
PathContext, so this CL also makes a BpToBuildWrapperContext that
conforms to PathContext by adding two method implementations.
Test: GENERATE_BAZEL_FILES=true m nothing && bazel query //... --config=bp2build | wc -l # 31433
Test: m queryview && bazel query //... --config=queryview # 63638
Change-Id: I0dd359746584b228046d2d0ff00895f28f9bdfc3
A SingletonModule is halfway between a Singleton and a Module. It has
access to visiting other modules via its GenerateSingletonBuildActions
method, but must be defined in an Android.bp file and can also be
depended on like a module.
Bug: 176904285
Test: singleton_module_test.go
Change-Id: I1b2bfdfb3927c1eabf431c53213cb7c581e33ca4
Splits queryview into queryview and bp2build. The latter runs as a
presingleton (could be converted to a singleton). This prevents needing
to run soong_ui a subsequent time to build the query. Queryview remains
as a separate step to prevent increasing runtime due to this generation
every time Soong runs.
Currently this is running as a presingleton as this gives a translation
of Android.bp files after only LoadHooks have run, no mutators.
Test: go tests
Test: m queryview && bazel query --config=queryview //...
Change-Id: If2ba28c7ef60fbd41f43bda6623d41c8c7d23a1b
This CL adds a new CONVERT_TO_BAZEL env var, and a bp2build goal. It
reuses the queryview architecture, but registers no mutators before
converting to BUILD files. This gives us a blank slate of the module
graph to work with, and create a shadow/alternate pipeline of mutators
to converge the Bazel BUILD graph to be semantically equivalent with the
Android.bp graph (after apply its current set of mutators).
The command to do so is:
$ CONVERT_TO_BAZEL=true m bp2build
To not clobber with queryview, the generated files are in
out/soong/bp2build.
Test: CONVERT_TO_BAZEL=true m bp2build && bazel query --config=bp2build //...
Test: m queryview && bazel query --config=queryview //..
Test: soong tests
Test: TH presubmit
Fixes: 174465461
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I4dd0ccc73abc345d70a50ca2803d6f400cd8c863
Previously in order to make use of dist entries with tag properties it
was necessary for each module type to be modified to store the results
of GenerateTaggedDistFiles(module) in the AndroidMkEntry.DistFiles
property. This change generalizes that mechanism to work across all
module types.
This change does improve the behavior of a couple of tests where
previously a dist with tag property was ignored because tagged dist
files were not available now it is used correctly.
Some module types do not implement OutputFileProducer interface and so
they cannot handle dist with tag properties. This change makes it an
error to attempt to do that. That necessitated adding OutputFiles(tag)
method to customModule in queryview_test.go as some of the tests in
that file used dist entries with tag properties.
Follow up changes will remove the code that was made redundant by this.
Test: m nothing
m dist sdk - before and after this change, compare result to
make sure that there are no significant differences.
Bug: 174226317
Change-Id: Ifc54d67db10ce0d0fe8179c05b97a2be2113be4e
No current use cases for soong.config and user-editable local
configuration options for Soong, so let's remove it to reduce the API
surface.
The MegaDevice configuration is used for building every module, for
every cpu variant, for every architecture, but it apparently isn't
used for a while now.
Test: m nothing
Test: TH
Fixes: 174188200
Change-Id: I2550e70ff6f9c8b57e9a7cc517d6a119a032a27a
This shaves off ~20s, which is ~11%, of `m nothing` with mixed builds.
Test: Manual verification with mixed builds of aosp_flame.
Change-Id: Ief1bc56c92329e899c829fc1374eea9ee9b94fb7
Prepare for using Config when adding singletons by passing
Config to NewTestContext and NewContext instead of to ctx.Register.
This will enable a followup change to store SingletonMakeVarsProviders
registered on the Context in the Config, which is necessary to run
multiple tests in parallel without data races.
Test: all soong tests
Change-Id: Id229629a4e42ff4487d317241673837726c075fc
Bazel overlay is an experimental feature. This renames the feature to 'queryview' to better describe its purpose, and also move away from the already overloaded 'overlay' term in Android.
Test: m queryview && bazel query --package_path=out/soong/queryview //...
Change-Id: I8b5068c7db46cb61a03a8e87af9c7c9077ebeff9
A recent change to Blueprint [1] added the feature to infer the type of
list elements to more than just "list of strings", but this also changed
the text to "list of string" (without s). This CL fixes the bazel
overlay, which uses "list of strings".
Test: m bazel_overlay && bazel query //... --package_path=out/soong/bazel_overlay
[1]: 764a771917%5E%21/#F1
Change-Id: I7522d15b1ce751807e1881ce1e94f068d226b316
This keeps the top level directory clean, and leverages the
new "build/bazel" repo location / package path.
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...
Bug: 171180724
Fixes: 171180724
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I151b8f3d2177167711e5974e5eabad8d6e910276
This reverts commit 323dc60712.
Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests
Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
With this change, bazel_module is a specifiable property on
genrule module definitions. With bazel-enabled mode, soong_build will
defer to Bazel for information on these modules.
source build/soong/bazelenv.sh to enter bazel-enabled mode.
Test: Manually verified on bionic/libc genrules using aosp_cf_x86_phone-userdebug
Change-Id: I3619848186d50be7273a5eba31c79989b981d408
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.
Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
surface module properties as attributes.
This CL maps int, bool, string and string_list module props from Soong
modules into their respective Bazel targets.
With this CL, you can now query modules based on their properties. For
example:
$ bazel query 'attr(nocrt, 1, //...)'
$ bazel query 'attr(apex_available, //apex_available:platform, //...)'
$ bazel query //art/dalvikvm:dalvikvm--linux_glibc_x86_64 --output=build | grep compile_multilib
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery 'attr(apex_available, com.android.runtime, //...)'
Test: soong_build tests
Fixes: 162720644
Fixes: 164320355
Change-Id: Iea8e594b952feccac3281f36dd6bdee8e7d62c3a
This patchset changes bazel_overlay to generate soong_module as a macro,
instead of a rule, and generate module properties in the BUILD files as
kwargs to the soong_module macro.
Here's a sample of the new BUILD files with module properties:
bionic/libdl/BUILD.bazel:
https://paste.googleplex.com/6484466996346880?raw
art/build/apex/BUILD.bazel:
https://paste.googleplex.com/5461276001042432?raw
bionic/apex/BUILD.bazel:
https://paste.googleplex.com/4932795173437440?raw
soong_module is now a macro that conditionally expands to underlying
soong_<module type> rules with statically defined attributes. In this
CL, we are starting with a hardcoded filegroup rule definition to
demonstrate the conditional rule loading within the soong_module macro.
If the module_type matches an existing Bazel rule, soong_module forwards
the entire **kwargs into the rule, which Bazel typechecks.
Non-filegroup module types will be expanded into generic_soong_module,
but with the kwargs dropped.
This approach allows us to:
1) Programmtically generate soong_<module type> rules for all module
types available in Soong, together with the statically defined attribute
types in `attrs`.
2) Incrementally migrate and test individual module types from
generic_soong_module to their module rule shims.
3) Swap out the module rule shims to the actual Bazel rules (e.g
cc_library, java_library) and perform attribute manipulation in Starlark
itself.
Example of querying against the 'srcs' attribute in soong_filegroup:
```
$ bazel cquery 'kind(soong_filegroup, //...)' | wc -l
590
$ bazel cquery --output=build 'attr(srcs, "linker.cpp",
kind(soong_filegroup, //bionic/...))'
INFO: Analyzed 3907 targets (0 packages loaded, 0 targets configured).
INFO: Found 3907 targets...
/usr/local/google/home/jingwen/aosp/out/soong/bazel_overlay/bionic/linker/BUILD.bazel:4144:13
soong_filegroup(
name = "linker_sources",
generator_name = "linker_sources",
generator_function = "soong_module",
generator_location = "bionic/linker/BUILD.bazel:4144:13",
srcs = ["dlfcn.cpp", "linker.cpp", "linker_block_allocator.cpp",
"linker_dlwarning.cpp", "linker_cfi.cpp", "linker_config.cpp",
"linker_debug.cpp", "linker_gdb_support.cpp", "linker_globals.cpp",
"linker_libc_support.c", "linker_libcxx_support.cpp",
"linker_namespaces.cpp", "linker_logger.cpp",
"linker_mapped_file_fragment.cpp", "linker_phdr.cpp",
"linker_relocate.cpp", "linker_sdk_versions.cpp", "linker_soinfo.cpp",
"linker_tls.cpp", "linker_utils.cpp", "rt.cpp"],
deps = [],
)
/usr/local/google/home/jingwen/aosp/out/soong/bazel_overlay/soong_module.bzl:32:23
in <toplevel>
```
This CL is known to be lacking the following features, and will be looked at in follow up CLs:
1) Pretty printing reflect.Interface properties, like arch, multilib and
dists.
2) Generating module Bazel rule shims for all module types, instead of
hardcoding them like `soong_filegroup`.
Bug: 162720644
Test: bazel_overlay_test.go (soong build test)
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: Ic1e448887eb540ed15a55bc4090cf75a4d832d41
The Bazel overlay is a directory at out/soong/bazel_overlay that
replicates the layout of the AOSP Soong module tree, but as a Bazel
workspace. Each Soong module variant is represented as a BUILD target
created with the `soong_module` rule.
To create this overlay, run `m bazel_overlay`.
A `soong_module` target can depend on other `soong_module` targets.
These dependencies replicate each module's `directDeps` in the Blueprint
graph, just before `PrepareBuildActions`.
This enables users to use bazel query as a way to introspect the Soong
module graph. For example,
- Direct reverse dependencies of //bionic/libc:generated_android_ids in
//bionic/libc/...:
$ bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'
//bionic/libc:libc_bionic_ndk--android_recovery_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_ramdisk_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static_com.android.runtime
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static
//bionic/libc:generated_android_ids
- Why does com.android.runtime depend on lzma?
$ bazel query
'somepath(//bionic/apex:com.android.runtime--android_common_com.android.runtime_image,
//external/lzma/...)'
//bionic/apex:com.android.runtime--android_common_com.android.runtime_image
//bionic/libc/malloc_debug:libc_malloc_debug--android_arm_armv7-a-neon_shared_com.android.runtime
//system/core/libunwindstack:libunwindstack--android_arm_armv7-a-neon_shared_com.android.runtime
//external/lzma/C:liblzma--android_arm_armv7-a-neon_shared_com.android.runtime
- What does the dep graph of //bionic/libc:crtbegin_so look like?
$ bazel query
'deps(//bionic/libc:crtbegin_so--android_arm_armv7-a-neon)'
--output=graph > graph.in && dot -Tpng < graph.in > graph.png
https://photos.app.goo.gl/DfsdoFRNsRjGwTmy8
Test: croot && m bazel_overlay && cd out/soong/bazel_overlay && bazel
query //... && bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I3bf40309bfb2d963bb8a688706385a57ee304c37#
Specifically, make Android.bp.list and soong.variables explicit inputs
While this is not a comprehensive list of all inputs
of this action (as the action depends on all blueprint
files in the source tree), this is closer to the truth.
This is a rollforward CL, which was originally rolled
back, as path validation failed for when OUT_DIR was
an absolute path. Validation has now been relaxed.
Test: Manually verified ninja output, checkbuild approved validation
for aosp-crosshatch, and manually ran frameworks/rs/build_rs.py with
patch fix
Change-Id: I4eb0d517f57336dd54eaa4bd31f46df9e93e6da2
While this is not a comprehensive list of all inputs of this action (as the action
depends on all blueprint files in the source tree), this closer to the truth.
Test: Manually verified ninja output, checkbuild approved validation for aosp-crosshatch
Change-Id: I77a0f612afb025232bee7d3ac31257b808991829
"dlv attach":
- Spams the terminal
- Requires sysctl -w kernel.yama.ptrace_scope=0
- Apparently, does not allow the debugger to inspect variables
Test: Manual.
Change-Id: I625369effaf5abda2b5d884c8ce5bde7247774eb
Collect the number of modules and variants and some basic statistics
on memory usage inside soong_build by writing out a proto that is
read back in by soong_ui.
Test: examine soong.log
Change-Id: I6926876377a4f6229cf41fdbf166ae03c885ea55
Allow running Soong in a headless delve debugger by passing
SOONG_DELVE=<listen addr> in the environment.
Bug: 80165685
Test: SOONG_DELVE=:1234 m nothing
Change-Id: Icfc893c8a8354a9bbc99112d9c83259cb41906d1
This will prevent it from being affected by future css location changes.
Test: m soong_docs
Fixes: 130567020
Change-Id: I80fd7b8a12a32bcea7b270423b8d8f8dc0976177
Bring property type strings to the start of property comments. This is
how source.android.com shows variables and paratemeters, and it is also
easier to read, especially when a comment ends with a code example.
Test: m soong_docs
Change-Id: Ic3e9db3a3711495998919ee95bbef6c7cb33b215
To add code format (<pre>...</pre>) support to module type texts.
Fixes: 124332708
Test: m soong_docs
Change-Id: Ibe9d3845a8f952fdef5007893d4af5836809ce1f
The href base added by I8a8ac0f9ba6c0d57ed83db2155955a95e070a265
doesn't really play nice with anchor links. This fixes the issue.
Fixes: 126550408
Test: m soong_docs
Change-Id: If62411a6391fd8151fa40cb2ecf32c5006797d6c
Lukacs T. Berki