The output from bootstrap would contain the a one-line summary of RBE
download / upload stats.
Bug: b/183007125
Change-Id: I90082a8433504e40bbf92992c3c8d1d656c49429
Merged-In: I90082a8433504e40bbf92992c3c8d1d656c49429
We need to capture the user build command to understand what additional
arguments is passed in to Soong. This data is useful to know which
targets are being executed and have statistics on top build targets
are specified.
Bug: b/169453712
Test: ran "m blueprint_tools", "m nothing", "lunch 1" and checked
in soong_metrics proto.
Change-Id: I7ab94dcf037211f1abffe6afd012b03ffbf99756
Merged-In: I7ab94dcf037211f1abffe6afd012b03ffbf99756
Test:
m USE_GOMA=true GOMA_DIR=$(goma_ctl goma_dir) toybox
m FORCE_USE_GOMA=true GOMA_DIR=$(goma_ctl goma_dir) toybox
both result in error message being printed.
Bug: b/171325288
Change-Id: I94ccaf217dba71aca990d88d205bad669a49a2f5
Merged-In: I94ccaf217dba71aca990d88d205bad669a49a2f5
The hostname is collected during a build to determine where the user
is build from (from office workstation, cloudtop, etc...). This is a
useful metrics for RBE related builds.
Bug: b/169395325
Test: m nothing, ran printproto to verify that soong_metrics has the
hostname.
Change-Id: I3e5d64f948ae43c5dda60de1d4a52b187e36dff8
Merged-In: I3e5d64f948ae43c5dda60de1d4a52b187e36dff8
Merged-In: Ib6898b49c16a830d814efe74581bde463caedf76
FORCE_USE_GOMA is needed to be collected as a metric for the
deprecation of GOMA on Android builds.
Bug: b/169423400
Test: m nothing, ran printproto command
Change-Id: I12300389f1f27239d8f21875d1b6e4ad069d95d7
Merged-In: I12300389f1f27239d8f21875d1b6e4ad069d95d7
This is an initial CL that collects the environment variables
USE_GOMA and USE_RBE. This is a needed metric to determine how
many users are currently using GOMA so we can help them migrate
to RBE.
Bug: b/163598127
Test: m nothing and checked soong_metrics
Change-Id: I39536eabad01aafbb3bd7182515d72230b36f658
Merged-In: I39536eabad01aafbb3bd7182515d72230b36f658
"exec format error" showed up on Mac when running the rbe_test.go.
The generated bash scripts did not have a newline at the end of
the file.
Bug: b/162600461
Fixes: b/162600461
Test: "m blueprint_tools" on Android Mac builder.
Change-Id: I425fa5f428733b6569c3fcdaa312023f9761d020
Merged-In: I425fa5f428733b6569c3fcdaa312023f9761d020
From aosp/1329396, the RBE metrics protobuf file is part of the
metrics uploading process. The RBE metrics protobuf file is
generated by running the bootstrap shutdown command. A new function
named DumpRBEMetrics was written in order to generate the RBE metrics
protobuf file before sending to the uploading process.
Bug: b/140638454
Test: * Unit test cases
* Ran RBE build on my local host and verified the
metrics protobuf file is created.
* Ran non-RBE build after RBE build and verified that
the previous metrics protobuf file was deleted.
Change-Id: I4b8068905cb67c4b8c2d94793917b98974fed707
Merged-In: I4b8068905cb67c4b8c2d94793917b98974fed707
Test: started soong multiple times and saw different socket files in
out/soong/.temp
Bug: b/166182389
Change-Id: Ib0087dd3867bdbf584c6868196070840cff291aa
Merged-In: Ib0087dd3867bdbf584c6868196070840cff291aa
Bug: b/166182389
Test: build with and without RBE_JAVAC/RBE_R8/RBE_D8 set.
Change-Id: I1607f8cfb4d2c3cbb3672152bbeb561d9968bc60
Merged-In: I1607f8cfb4d2c3cbb3672152bbeb561d9968bc60
This CL adds a remoteexec package that allows adding a configurable RBE
prefix to the template.
Bug: b/166182389
Test: built aosp crosshatch userdebug with and without RBE_CXX_LINKS.
Change-Id: Ica920c3d7f79f2996210b9cbd448126451c1707c
Merged-In: Ica920c3d7f79f2996210b9cbd448126451c1707c
There are cases where the developer is not building a product so
the metadata information is not being set. For now, set the OS
name so we can track which OS the user is using it.
Fixes: b/163118212
Test: m nothing and checked the blueprint
Change-Id: I9b521eee2d266b50625aa8e916ddaacb4e2feaea
Merged-In: I9b521eee2d266b50625aa8e916ddaacb4e2feaea
Currently, the build timestamp saved to the build_date_timestamp
field in the MetricsBase metrics protobuf usually comes from a text
file named build_date.txt. It may not be accurate for metrics purposes.
Instead, use the build start time from Soong main UI for a better
timestamp.
Bug: b/140638454
Test: Ran and checked the build_date_timestamp by running the printproto
command.
Change-Id: I4d893fbbf9830ba21911e56ae13bc5272be47ae6
Merged-In: I4d893fbbf9830ba21911e56ae13bc5272be47ae6
From several dashboards, there are high spikes from several
builds where the real time of executing a part of the build
system took many many hours. This is due to using the wall
time clock when calculating the duration. For metrics, the
monotonic clock should be used for more accurate measurement.
Also, cleaned up the time.go file to use the proper proto
functions when marshalling perf protobuf and deleted
unnecessary functions.
Bug: b/143492984
Test: Wrote and ran unit test case.
Change-Id: I1540cfbf37915e3045ecedaa2595c1f9429bcd62
Merged-In: I1540cfbf37915e3045ecedaa2595c1f9429bcd62
The metrics uploader was currently running on foreground where it
would copy the metrics files in a separate directory and then forked
into the background for the upload process. As a result, running the
lunch command would take a second longer to run since each metrics
uploader run had an average of half a second.
Bug: 140638454
Test: * Wrote and updated unit test cases.
* Set ANDROID_ENABLE_METRICS_UPLOAD to point to the latest
metrics_uploader bash script. Executed the "lunch 1" command
and measured the running time. Executed "m nothing" command
and checked that the metrics were uploaded.
* Ran "lunch 1" and "m nothing" with
ANDROID_ENABLE_METRICS_UPLOAD=""
* Removed oauth from metrics_uploader and ran "m nothing" and
"lunch 1". The oauth Message appeared only to "m nothing"
Change-Id: I13c61e666c8f44613dee291a704cef6a27335188
Merged-In: I13c61e666c8f44613dee291a704cef6a27335188
Soong now supports the ability to upload metrics to another location
by setting the ANDROID_ENABLE_METRICS_UPLOAD to an uploader that
accepts the upload.proto proto buffer message. When the environment
variable is set, a set of build metrics files (soong_metrics,
rbe_metrics.pb and build_error) is uploaded.
Bug: 140638454
Test: * Wrote unit test cases
* Setup the uploader, built a succcessful and failed aosp_arm-eng
target and monitor the uploading of the metrics.
Change-Id: I76a65739c557dc90345e098ca03119a950ece2d2
Merged-In: I76a65739c557dc90345e098ca03119a950ece2d2
A new protobuf message named Upload is defined to send build metrics
to the Android Build Server API after a build is completed. These
metrics are collected from googlers only, as described in
go/local-developers-metrics-wfh.
Bug: b/140638454
Test: m nothing
Change-Id: I5e49b5ace053913be2b75673cb1375f4cef37825
Merged-In: I5e49b5ace053913be2b75673cb1375f4cef37825
This CL adds RBE support to javac, r8, and d8 rules which is only
enabled if respective environment variables are set.
Test: an aosp_crosshatch build with and without the new variables.
Bug: b/166182389
Change-Id: Ic82f3627944f6a5ee7b9f3228170c2709b1bfcb8
Merged-In: Ic82f3627944f6a5ee7b9f3228170c2709b1bfcb8
Test: Built aosp_arm-user with and without USE_RBE. USE_RBE uses
a proxy script in place of rewrapper.
Bug: b/166182389
Change-Id: I5bf008a940513872d70b5b215bd6209f759826ae
Merged-In: I5bf008a940513872d70b5b215bd6209f759826ae
Some people apparently still talk to the network during their build.
Allow this temporarily with a BUILD_BROKEN_USES_NETWORK check.
Bug: 129992021
Test: attempt to talk to the network during the build with and without
this flag.
Change-Id: Ifb967c656aa24c4599e7232d0f1b5a303b5bac52
Merged-In: Ifb967c656aa24c4599e7232d0f1b5a303b5bac52
(cherry picked from commit 25e6f09c06)
Attempt to catch places where logs are truncated by panicing if
logging is attempted after Close.
Test: m nothing
Change-Id: If670f20d08832ed65b63af5589b548e9815f2f0d
Wait for the ninja proto processing goroutine to notice the fifo
has closed and exit before continuing.
Test: m nothing
Change-Id: I8cf5f3b8bf6a91496c6d2bbbd3e811eb7f0c9d21
Dump variable in log to track remaining targets which don't set it.
Bug: 122954981
Test: $ grep VNDK out/soong.log
2019/03/07 09:22:04.235604 build/soong/ui/build/dumpvars.go:109: BOARD_VNDK_VERSION current
Change-Id: I34146c752a4bf0d799374bb93cc0d75b0f1b048c
There's more than just toybox in this group now, so let's rename our
variables to something closer to the desired behavior, rather than the
first user.
Test: treehugger
Change-Id: I76d4407792061c8110b194cfe73f1ddc84dbc22f
For ease of investigating the issue with error reports, let me make
the error message more verbose.
Test: make goma_ctl.py raise Exception and run.
Test: I confirmed verbose error message shown.
Change-Id: I04f9a5ca4808746994c41ddce8a5b22a1975fcc2
Signed-off-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
Bug: 124218940
If ANDROID_QUIET_BUILD environment variable is set to '1', do not show 20
lines of the environment variables. In addition, for the failing step show
only the step name and output, omitting the command proper (the verbose log
as well as error.log still contains it).
If build succeeds, the output of the build itself consists of a single
line:
```
> m androidmk
[100% NN/NN] <last command>
```
When it fails, the output does not contain sometimes very long command
line:
```
> m androidmk
[ 97% NN/MM] test androidmk
FAILED: <step>
--- FAIL: TestEndToEnd (0.01s)
androidmk_test.go:1025: failed testcase 'prebuilt_etc_TARGET_OUT_ETC'
input:
include $(CLEAR_VARS)
LOCAL_MODULE := etc.test1
LOCAL_MODULE_CLASS := ETC
LOCAL_MODULE_PATH := $(TARGET_OUT_ETC)/foo/bar
include $(BUILD_PREBUILT)
expected:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
got:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
FAIL
17:50:53 ninja failed with: exit status 1
```
[The related change in build/make/envsetup.sh suppresses timing display
when the same variable is set.]
Change-Id: I4d3c72457de031ff58a324c2fe98f4c1d10f8239
Test: treehugger
And add a helper script that can help parse the output from
build_test.bash
Test: check for BUILD_BROKEN_ENG_DEBUG_TAGS in soong.log
Test: go run ../build/soong/scripts/build_broken_logs.go *
Change-Id: Idd0fc8b59770dcdbe44eeba262558708a9497f96
We don't want these to be warnings, may as well be errors instead of
turning off the warnings.
Bug: 123583617
Test: treehugger
Change-Id: I6ca518c9647e712426952cb88bdb044d933b23f2
bufio.Scanner defaults to a relatively small max line length of
64kB. Kati output may have long lines, use a max length of 2MB
instead.
Fixes: 123590367
Test: m checkbuild
Change-Id: Ia869c5c2526621244f0065736e0d5fd9a088df06
As more phony rules move either to the packaging kati run, or soong, the
existing real-to-phony check failed to notice that they were phony
rules.
So I added --top_level_phony to Make that assumes that all leaf nodes
(source files, or nodes generated by another ninja file generator) that
do not have a '/' in them are phony targets.
The existing phony-looks-real check in Kati should prevent these from
being real generated files, and our source tree shouldn't have any
source files used at the root level.
Test: build_test on downstream branches
Change-Id: I77cff84b536c6cd6402307a9aa5a9c273e72c71f
Kousik Kumar