Also get Bazel to write real files there (not symlinks) so that the DIST_DIR can be independent.
Test: Manually using e.g. DIST_DIR=/tmp/foo USE_BAZEL=1 m dist
Change-Id: I39d5219500864c9ecc85f356a028e9b5bf2607f4
The upload functionality now supports directories to be uploaded
for metrics purpose.
Bug: b/174479728
Test: Wrote unit test case.
Change-Id: I6906be4c1b7fd76ddf6ff7b94e48fe1c4209c59d
Code refactoring has been done for logs directory logic code since
the bazel metrics directory depends on the log directory. For builds
that did not specify a dist directory, the log directory is under
out directory. With dist, the logs directory is under <dist dir>/logs.
This matters for Android CI builds where the metrics files are
placed under logs directory. With this change, the bazel metrics
directory and corresponding files will be under <dist dir>/logs
directory for Android CI builds.
Bug: b/174479728
Test: * USE_BAZEL=1 m nothing (bazel_metrics dir in out dir)
* m nothing (bazel_metrics dir deleted)
* DIST_DIR=/tmp/build USE_BAZEL=1 m nothing dist (bazel_metrics
is in /tmp/build/logs directory)
Change-Id: Ic9e1ff49a1964fcaaf801bde2c19f33597ca1db4
The existing --skip-make flag disables both the config step and the kati
step in the build. Add support for a --skip-kati flag that skips just
the kati step, and refactor things so that the logic is shared between
these two.
Bug: 174315599
Test: TARGET_PRODUCT=aosp_arm64 soong_ui --make-mode --skip-kati;
(verify soong.variables is regenerated)
Change-Id: I75b1910fc1c12fcda130e37b7bc4c050131c7b33
On top of improving the comments, I also refactored the status checker
to simplify and clarify its use case to be a ninja stuckness checker,
since it doesn't appear to have other purposes.
Test: TH presubmit
Bug: b/173474588
Change-Id: I2cf51a1ebf16071a24a1c13c06c7b1adf60256de
This CL also removes --warn_real_to_phony and --warn_phony_looks_real,
since --werror_real_to_phony and --wrror_phony_looks_real are already
specified.
Test: treehugger
Bug: 173474588
Change-Id: Icbb0f53182d1b912c62f8044f52bb79d985bf868
This removes the need to source bazelenv.sh for USE_BAZEL_ANALYSIS, and
unifies mixed builds to use the checked in tools/bazel and bazelrc.
It also unifies all bazel-related output to be in out/bazel.
Without aosp/1502095, this change still requires toplevel_output_directories to be an empty
list, otherwise there'll be this error:
ERROR: Directories specified with toplevel_output_directories should be
ignored and can not be used as sources.
Test: With aosp/1441774: rm -rf out/ && lunch aosp_cf_x86_auto && USE_BAZEL_ANALYSIS=1 m libc && prebuilts/build-tools/linux-x86/bin/ninja -f out/combined-aosp_cf_x86_auto.ninja -t commands libc | grep bazel-out | wc -l # 2 results
Change-Id: I69b217ec88da531415792bb6e04b6a194ca4718d
This patch adds information behind the Bazel USE_BAZEL=1 execution
pipeline.
Test: m
Bug: 173474588
Change-Id: I54bdd26fb14c8a371de0922d193419112e8c3ece
Ensures that Bazel and Ninja are using the same binaries from PATH, such
as `sort` from the linked bug.
Test: USE_BAZEL=1 m
Test: rm out/soong/apex/depsinfo/new-allowed-deps.txt && USE_BAZEL=1 m
Bug: 161074854
Fixes: 161074854
Change-Id: Icc3e717e6d3cfcecfe90f0fb3a94a86485a11918
break and default statements are not needed inside of a switch
statement.
Bug: b/173028918
Test: USE_BAZEL m nothing
Change-Id: I14aa8cbae3a9b3c859f147c5134600f1e75c6a89
Added a tracer named "bazel" in runBazel function to capture topline
metrics when running bazel.
Bug: b/173028918
Test: * USE_BAZEL=1 m nothing and checked soong_metrics file using
printproto command to verify that bazel topline metrics
were captured.
Change-Id: I5cb0728a4da48545dc701cc79627c318d5d9a29d
Generate the bazel profile file that contains the metrics of
executing the ninja build phase of Soong.
Bug: b/173028918
Test: * m nothing and check out dir that there is no bazel_metrics dir.
* USE_BAZEL=1 m nothing and checked
out/bazel_metrics/build_bazel_profile.gz existed.
* m nothing and checked out/bazel_metrics does not exist.
Change-Id: I797d1eecf5c2c260ade8b55dc69c237bfd0fcbd4
$OUT/bazel_metrics directory is created in order to dump the
set of bazel profile files being created from a single build
execution. The directory is created from soong_ui.
Bug: b/173028918
Test: * m nothing and check out dir that there is no bazel_metrics dir.
* USE_BAZEL=1 and checked out/bazel_metrics was created.
* Ran m nothing and verified that out/bazel_metrics directory is
deleted to ensure no metrics is uploaded or collected from the
pipeline.
Change-Id: I1d3c2471065fae6931f35cef7f0f35fd84f76bef
The UseBazel() function will be used in several places to perform
different functionality such as setting up the metrics path.
Bug: b/173028918
Test: m nothing
Change-Id: Ie94073b8f0552ec7528e1cc2f3be74b693dc3135
This ensures that the lunch repo rule always reads the right Ninja file.
Test: USE_BAZEL=1 m
Test: m
Change-Id: I7f670dd538f46c4dae5ea537b857ff04520b60e7
This ensures that USE_BAZEL=1 mode has the same directory structure
under out/ as it does without USE_BAZEL=1.
Test: Manually verified building USE_BAZEL=1 aosp_flame with droid and
libc
Change-Id: I27e810772d1bf04bd7930ea122217b93d2fd48b4
Collect the resource information of each executed process in soong_ui
for metrics analysis.
Bug: b/169453825
Test: m nothing; "lunch 1" && m
Change-Id: I8c7fe019111921c4c59174d133b85439ec825885
This was caught while running the soong_build docs action under Bazel as
ninja executor. Since the config file is not an explicit input to the
action, it's not present in Bazel's execution root during action
execution time, leading to a missing file error. To fix it, add the file
as an input to the action, if it's configured in product variables.
New soong_build statement:
build out/soong/docs/soong_build.html: s.writedocs.soongDocs $
out/.module_paths/Android.bp.list out/soong/soong.variables $
out/soong/dexpreopt.config | out/soong/.bootstrap/bin/soong_build
outDir = out/soong/docs
Test: m
Test: USE_BAZEL=1 m
Bug: 161958189
Fixes: 161958189
Change-Id: I2bf264e848bb29ac91f4196fbe9c4ecf3cea8a2c
There is no need for multiproduct_kati to run the tests for
every product, they don't vary by product config. --skip-soong-tests
can also be used for local development to run soong_build even if
the tests don't pass.
Bug: 156428456
Test: m --skip-soong-tests nothing
Change-Id: I9c00e3d1b6e51d17bb290339c3f124d4d1c9e69f
We need to capture the user build command to understand what additional
arguments is passed in to Soong. This data is useful to know which
targets are being executed and have statistics on top build targets
are specified.
Bug: b/169453712
Test: ran "m blueprint_tools", "m nothing", "lunch 1" and checked
in soong_metrics proto.
Change-Id: I7ab94dcf037211f1abffe6afd012b03ffbf99756
Several structs and files are named time which records the total
execution of a traceable event. Soon, resource information of an
executed process is collected during a build event so a more generic
name is applied.
Bug: b/169453825
Test: go test and m nothing. Checked the soong_metrics protobuf.
Change-Id: I65a782603530d51018d6cc1192c18cfeef7566a0
We would like to collect the resource usage of processes such as
kati, soong_build and ninja on the resource usage such as memory
and context switching. This changelist contains the defined fields
in the protobuf file of the resource information to be collected
from interested processes.
Bug: b/169453825
Test: N/A
Change-Id: I4b3cbbc3e7b78900875c10d34d574425f8b11730
Test:
m USE_GOMA=true GOMA_DIR=$(goma_ctl goma_dir) toybox
m FORCE_USE_GOMA=true GOMA_DIR=$(goma_ctl goma_dir) toybox
both result in error message being printed.
Change-Id: I94ccaf217dba71aca990d88d205bad669a49a2f5
This CL adds symlink_outputs to various locations in Soong that creates
actions that creates symlink outputs, and explicitly mark them as such.
Test: m
Bug: 160568334
Change-Id: I322751bada52a9f49011c74731d84761586e03e7
Propagate the subprocess stats from ninja to build.trace.gz.
There is too much data here to put into separate counters for each
statistic, so put them into args on each duration event for now.
We would like to track max RSS for each subprocess, but the Linux
kernel inherits the max RSS of the ninja process in each subprocess,
which sets a lower bound on the measurable max RSS to the size of
the ninja process. The ninja process is large due to the multi-GB
build.ninja files.
Bug: 170701554
Test: examine build.trace.gz
Change-Id: I8aaaafe627a57f1a500af098c097c6381c583ba5
Clean the odm_dlkm and vendor_dlkm staging directories in installclean
to get consistent results in incremental builds on the build servers.
Test: treehugger
Change-Id: Id431d7457d85e0be68437a2034fd544d74cc3709
Collect the total number of CPU cores and available RAM for each build
as part of the system resource info. This is to know what kind of
build machines that the developers are using.
Bug: b/169453825
Test: m nothing and ran printproto on soong_metrics to validate the
data.
Change-Id: I8617bdb9490b37d7689abd27d349af1a651533db
This retriggers soong_build whenever a new bzl, WORKSPACE, or
BUILD.bazel file is changed or added.
Test: Manually verified on bionic/libc genrules with manual changes to
related BUILD/bzl/WORKSPACE files -- these all retrigger builds.
Test: Updated finder_test.go
Change-Id: I634384f88781a6b6db32f5d6bf9c07e179e14c39
Added additional instruction in the regen.sh script to build the
blueprint_tools and aprotoc in case the regen.sh script fails.
Bug: b/169395325
Test: ./regen.sh
Change-Id: Icb38c907fb47febc17eef1067d06d0588dd0b7ba
The hostname is collected during a build to determine where the user
is build from (from office workstation, cloudtop, etc...). This is a
useful metrics for RBE related builds.
Bug: b/169395325
Test: m nothing, ran printproto to verify that soong_metrics has the
hostname.
Change-Id: I3e5d64f948ae43c5dda60de1d4a52b187e36dff8
Tradefed packages are created in out/host/common/obj/PACKAGING
and should be removed during installclean so that they get an
accurate embedded build number.
Fixes: 170308394
Test: treehugger
Change-Id: Ib7a57377b5c9512584bc93f10da316eadf786979
This reverts commit 323dc60712.
Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests
Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
Prints the following warning messsage:
WARNING:
Setting OUT_DIR to a path other than out may result in slow RBE builds.
See http://go/android_rbe_out_dir for a workaround.
Bug: b/169676232
Change-Id: Id1ad06c4dc672c24373642e0b624833eb6a0dbcf
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.
Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
FORCE_USE_GOMA is needed to be collected as a metric for the
deprecation of GOMA on Android builds.
Bug: b/169423400
Test: m nothing, ran printproto command
Change-Id: I12300389f1f27239d8f21875d1b6e4ad069d95d7
We should override log-dir with the environment variables if its set.
This is needed in our integration tests and generally to allow for
user-overrides.
Test: RBE_log_dir=`pwd`/rbelogs mmma external/boringssl
Change-Id: I3d316a2d60122c42f5c3b5dfede4037362a7b5f1
This is an initial CL that collects the environment variables
USE_GOMA and USE_RBE. This is a needed metric to determine how
many users are currently using GOMA so we can help them migrate
to RBE.
Bug: b/163598127
Test: m nothing and checked soong_metrics
Change-Id: I39536eabad01aafbb3bd7182515d72230b36f658
This hasn't worked for a couple years, and continues to bitrot. Just
remove it.
Adds a bpfix rule so that we can eventually remove the
product_variables.pdk definition, which is now always a no-op.
Test: treehugger
Change-Id: I830b54d419b59f6db1d4617b45e61a78234f57a7
Merged-In: I830b54d419b59f6db1d4617b45e61a78234f57a7
There are cases where the developer is not building a product so
the metadata information is not being set. For now, set the OS
name so we can track which OS the user is using it.
Fixes: b/163118212
Test: m nothing and checked the blueprint
Change-Id: I9b521eee2d266b50625aa8e916ddaacb4e2feaea
"exec format error" showed up on Mac when running the rbe_test.go.
The generated bash scripts did not have a newline at the end of
the file.
Bug: b/162600461
Fixes: b/162600461
Test: "m blueprint_tools" on Android Mac builder.
Change-Id: I425fa5f428733b6569c3fcdaa312023f9761d020
Disable the error test cases to allow developers to build
AOSP on Mac.
Bug: b/162600461
Test: m nothing
Change-Id: Ic8c6fe04743a8e4fe197a1bb71259036fb749edc
Disable the error test cases to allow developers to build
AOSP on Mac.
Bug: b/162600461
Test: m nothing
Change-Id: Iac4373dcb10710d8a2325eeb3ee20a6eeca6143f
Pass -n to blueprint to make it use validation dependencies for tests,
which will let them run in parallel with soong_build.
Bug: 156428456
Test: m nothing
Change-Id: I706c14330543e2a7728e5dc385976900c67cb2f5
Currently, the build timestamp saved to the build_date_timestamp
field in the MetricsBase metrics protobuf usually comes from a text
file named build_date.txt. It may not be accurate for metrics purposes.
Instead, use the build start time from Soong main UI for a better
timestamp.
Bug: b/140638454
Test: Ran and checked the build_date_timestamp by running the printproto
command.
Change-Id: I4d893fbbf9830ba21911e56ae13bc5272be47ae6
Rupert Shuttleworth