Original change: https://android-review.googlesource.com/c/platform/build/soong/+/1419252
Test:
1. Applied https://googleplex-android-review.git.corp.google.com/c/platform/build/soong/+/15723183, confirmed that the build fails on rvc-dev (by running `USE_RBE=false m `)
2. Subsequently applied this patch and tested to confirm that the failure is fixed (by running `USE_RBE=false m out/soong/.intermediates/vendor/qcom/sm7250/proprietary/commonsys/qcrilOemHook/qcrilmsgtunnel/android_common/combined/qcrilmsgtunnel.jar` and `USE_RBE=true m out/soong/.intermediates/vendor/qcom/sm7250/proprietary/commonsys/qcrilOemHook/qcrilmsgtunnel/android_common/combined/qcrilmsgtunnel.jar`).
Bug: b/198045383
Merged-In: I6df58b4e700f0d231367af2710672d731d20a736
Change-Id: I52b7a3810fdf3314ae107f4b8330e9d6f62c6302
(cherry picked from commit f15ed4fe61)
The output from bootstrap would contain the a one-line summary of RBE
download / upload stats.
Bug: b/183007125
Change-Id: I90082a8433504e40bbf92992c3c8d1d656c49429
Merged-In: I90082a8433504e40bbf92992c3c8d1d656c49429
Keeps license declarations from breaking the build without implementing
the full feature.
Test: m all
Bug: 181768532
Change-Id: Ic78672cd25da13aa9078a48b06446f66780c792d
Merged-in: Ic78672cd25da13aa9078a48b06446f66780c792d
Merged-in: If3fbc157c6ffed8e3ee4bdd9aa56fabe2c24444b
Merged-in: Ie7e1695ecb8cd943124426a0cad2c0d1db2b46ed
Keeps license declarations from breaking the build without implementing
the full feature.
Test: m all
Bug: 181768532
Change-Id: Ic78672cd25da13aa9078a48b06446f66780c792d
Merged-in: Ic78672cd25da13aa9078a48b06446f66780c792d
Merged-in: If3fbc157c6ffed8e3ee4bdd9aa56fabe2c24444b
Merged-in: Ie7e1695ecb8cd943124426a0cad2c0d1db2b46ed
We need to capture the user build command to understand what additional
arguments is passed in to Soong. This data is useful to know which
targets are being executed and have statistics on top build targets
are specified.
Bug: b/169453712
Test: ran "m blueprint_tools", "m nothing", "lunch 1" and checked
in soong_metrics proto.
Change-Id: I7ab94dcf037211f1abffe6afd012b03ffbf99756
Merged-In: I7ab94dcf037211f1abffe6afd012b03ffbf99756
Test:
m USE_GOMA=true GOMA_DIR=$(goma_ctl goma_dir) toybox
m FORCE_USE_GOMA=true GOMA_DIR=$(goma_ctl goma_dir) toybox
both result in error message being printed.
Bug: b/171325288
Change-Id: I94ccaf217dba71aca990d88d205bad669a49a2f5
Merged-In: I94ccaf217dba71aca990d88d205bad669a49a2f5
The hostname is collected during a build to determine where the user
is build from (from office workstation, cloudtop, etc...). This is a
useful metrics for RBE related builds.
Bug: b/169395325
Test: m nothing, ran printproto to verify that soong_metrics has the
hostname.
Change-Id: I3e5d64f948ae43c5dda60de1d4a52b187e36dff8
Merged-In: I3e5d64f948ae43c5dda60de1d4a52b187e36dff8
Merged-In: Ib6898b49c16a830d814efe74581bde463caedf76
FORCE_USE_GOMA is needed to be collected as a metric for the
deprecation of GOMA on Android builds.
Bug: b/169423400
Test: m nothing, ran printproto command
Change-Id: I12300389f1f27239d8f21875d1b6e4ad069d95d7
Merged-In: I12300389f1f27239d8f21875d1b6e4ad069d95d7
This is an initial CL that collects the environment variables
USE_GOMA and USE_RBE. This is a needed metric to determine how
many users are currently using GOMA so we can help them migrate
to RBE.
Bug: b/163598127
Test: m nothing and checked soong_metrics
Change-Id: I39536eabad01aafbb3bd7182515d72230b36f658
Merged-In: I39536eabad01aafbb3bd7182515d72230b36f658
"exec format error" showed up on Mac when running the rbe_test.go.
The generated bash scripts did not have a newline at the end of
the file.
Bug: b/162600461
Fixes: b/162600461
Test: "m blueprint_tools" on Android Mac builder.
Change-Id: I425fa5f428733b6569c3fcdaa312023f9761d020
Merged-In: I425fa5f428733b6569c3fcdaa312023f9761d020
From aosp/1329396, the RBE metrics protobuf file is part of the
metrics uploading process. The RBE metrics protobuf file is
generated by running the bootstrap shutdown command. A new function
named DumpRBEMetrics was written in order to generate the RBE metrics
protobuf file before sending to the uploading process.
Bug: b/140638454
Test: * Unit test cases
* Ran RBE build on my local host and verified the
metrics protobuf file is created.
* Ran non-RBE build after RBE build and verified that
the previous metrics protobuf file was deleted.
Change-Id: I4b8068905cb67c4b8c2d94793917b98974fed707
Merged-In: I4b8068905cb67c4b8c2d94793917b98974fed707
Test: started soong multiple times and saw different socket files in
out/soong/.temp
Bug: b/166182389
Change-Id: Ib0087dd3867bdbf584c6868196070840cff291aa
Merged-In: Ib0087dd3867bdbf584c6868196070840cff291aa
Bug: b/166182389
Test: build with and without RBE_JAVAC/RBE_R8/RBE_D8 set.
Change-Id: I1607f8cfb4d2c3cbb3672152bbeb561d9968bc60
Merged-In: I1607f8cfb4d2c3cbb3672152bbeb561d9968bc60
Bug: b/166182389
Test: built with and without RBE_ABI_DUMPER set.
Change-Id: Ie752c07325453076d191813eaa50da03e3a0c2d8
Merged-In: Ie752c07325453076d191813eaa50da03e3a0c2d8
This CL adds a remoteexec package that allows adding a configurable RBE
prefix to the template.
Bug: b/166182389
Test: built aosp crosshatch userdebug with and without RBE_CXX_LINKS.
Change-Id: Ica920c3d7f79f2996210b9cbd448126451c1707c
Merged-In: Ica920c3d7f79f2996210b9cbd448126451c1707c
There are cases where the developer is not building a product so
the metadata information is not being set. For now, set the OS
name so we can track which OS the user is using it.
Fixes: b/163118212
Test: m nothing and checked the blueprint
Change-Id: I9b521eee2d266b50625aa8e916ddaacb4e2feaea
Merged-In: I9b521eee2d266b50625aa8e916ddaacb4e2feaea
Currently, the build timestamp saved to the build_date_timestamp
field in the MetricsBase metrics protobuf usually comes from a text
file named build_date.txt. It may not be accurate for metrics purposes.
Instead, use the build start time from Soong main UI for a better
timestamp.
Bug: b/140638454
Test: Ran and checked the build_date_timestamp by running the printproto
command.
Change-Id: I4d893fbbf9830ba21911e56ae13bc5272be47ae6
Merged-In: I4d893fbbf9830ba21911e56ae13bc5272be47ae6
From several dashboards, there are high spikes from several
builds where the real time of executing a part of the build
system took many many hours. This is due to using the wall
time clock when calculating the duration. For metrics, the
monotonic clock should be used for more accurate measurement.
Also, cleaned up the time.go file to use the proper proto
functions when marshalling perf protobuf and deleted
unnecessary functions.
Bug: b/143492984
Test: Wrote and ran unit test case.
Change-Id: I1540cfbf37915e3045ecedaa2595c1f9429bcd62
Merged-In: I1540cfbf37915e3045ecedaa2595c1f9429bcd62
The metrics uploader was currently running on foreground where it
would copy the metrics files in a separate directory and then forked
into the background for the upload process. As a result, running the
lunch command would take a second longer to run since each metrics
uploader run had an average of half a second.
Bug: 140638454
Test: * Wrote and updated unit test cases.
* Set ANDROID_ENABLE_METRICS_UPLOAD to point to the latest
metrics_uploader bash script. Executed the "lunch 1" command
and measured the running time. Executed "m nothing" command
and checked that the metrics were uploaded.
* Ran "lunch 1" and "m nothing" with
ANDROID_ENABLE_METRICS_UPLOAD=""
* Removed oauth from metrics_uploader and ran "m nothing" and
"lunch 1". The oauth Message appeared only to "m nothing"
Change-Id: I13c61e666c8f44613dee291a704cef6a27335188
Merged-In: I13c61e666c8f44613dee291a704cef6a27335188
Soong now supports the ability to upload metrics to another location
by setting the ANDROID_ENABLE_METRICS_UPLOAD to an uploader that
accepts the upload.proto proto buffer message. When the environment
variable is set, a set of build metrics files (soong_metrics,
rbe_metrics.pb and build_error) is uploaded.
Bug: 140638454
Test: * Wrote unit test cases
* Setup the uploader, built a succcessful and failed aosp_arm-eng
target and monitor the uploading of the metrics.
Change-Id: I76a65739c557dc90345e098ca03119a950ece2d2
Merged-In: I76a65739c557dc90345e098ca03119a950ece2d2
A new protobuf message named Upload is defined to send build metrics
to the Android Build Server API after a build is completed. These
metrics are collected from googlers only, as described in
go/local-developers-metrics-wfh.
Bug: b/140638454
Test: m nothing
Change-Id: I5e49b5ace053913be2b75673cb1375f4cef37825
Merged-In: I5e49b5ace053913be2b75673cb1375f4cef37825
The "ccNoDeps" rule was introduced to separate out ".s" compilations
from other ".S" and ".c*" compilations. The ".s" compilation does not
produce a dependency file and does not support running a preprocessor
through it.
However, it does have ".include" directives, that do NOT take a macro,
but still does the equivalent of a "#include". The compilation of ".s"
assembly files also do NOT produce a dependency file.
Because they don't produce a dependency file AND because RBE's input
processor does not yet support finding dependencies for these files, I'm
making this change to not prepend rewrapper for these commands.
Test: Tested by running a build with this change and ensuring the ".s"
compilations aren't being sent to rewrapper.
Bug: b/166182389
Change-Id: I60bb14ff92596e4992e9f675bdc199f1440d4327
Merged-In: I60bb14ff92596e4992e9f675bdc199f1440d4327
This CL adds RBE support to javac, r8, and d8 rules which is only
enabled if respective environment variables are set.
Test: an aosp_crosshatch build with and without the new variables.
Bug: b/166182389
Change-Id: Ic82f3627944f6a5ee7b9f3228170c2709b1bfcb8
Merged-In: Ic82f3627944f6a5ee7b9f3228170c2709b1bfcb8
Remove the distinction between pctx.StaticRule and
pctx.AndroidStaticRule so that all of the local rules correctly
get assigned to the localPool. Also put Module and Singleton
rules into the localPool.
Bug: b/166182389
Test: compare out/soong/build.ninja
Change-Id: Id2bb38eff3c7209340fe55bc9006f00bd3661d81
Merged-In: Id2bb38eff3c7209340fe55bc9006f00bd3661d81
Test: Built aosp_arm-user with and without USE_RBE. USE_RBE uses
a proxy script in place of rewrapper.
Bug: b/166182389
Change-Id: I5bf008a940513872d70b5b215bd6209f759826ae
Merged-In: I5bf008a940513872d70b5b215bd6209f759826ae
Previously, when there is apex_set that is overriding another module,
the entry for the overridden module is removed from apexkeys.txt.
However, this is wrong because the existence of the apex_set module
doesn't necessary mean that the module is insatalled instead of the
overridden module. That is determined by PRODUCT_PACKAGES which Soong
has no knowledge of. Therefore, we don't delete the entry for the
overridden (actually possibly overridable) modules in the file.
Bug: 158729168
Bug: 162464887
Test: m
Merged-In: I85d0c756f862323bae556bf657d66ec50038985f
(cherry picked from commit ac5e79f900)
Change-Id: I85d0c756f862323bae556bf657d66ec50038985f
apex_sets is a new module type that can be used to deliver pre-signed
APEXes, which previously could be done only via prebuilt_apex.
Soon gnow understands apex_sets module types and emits the signing info
of the modules to apexkeys.txt
Bug: 162464887
Bug: 158729168
Test: m
Merged-In: I9507375342ec053309660d94c931a79bf4f21218
Change-Id: I9507375342ec053309660d94c931a79bf4f21218
Soong and Make have no ways to figure out what splits will be outputted
from a given android_app_set, so it's impossible for them to provide
full PACKAGES.$(LOCAL_MODULE).CERTIFICATE entries, which are required to
build a final apkcerts.txt. This change makes extract_apks produce
apkcerts.txt files for each input modules instead. The Make-side
counterpart of this change merges all local apkcerts.txt into a final
one.
Bug: 160119159
Bug: 162464887
Test: main_test.go
Test: m apkcerts-list
Merged-In: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
Change-Id: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
apex_set takes an .apks file that contains a set of prebuilt apexes with
different configurations. It uses extract_apks to select and install the
best matching one for the current target.
Bug: 153456259
Test: apex_test.go
Test: com.android.media.apks
Change-Id: I1da8bbcf1611b7c580a0cb225856cbd7029cc0a7
Merged-In: I1da8bbcf1611b7c580a0cb225856cbd7029cc0a7
The wrapper to getenv() appends the effective userid (euid) of the
current process to GCOV_PREFIX. This avoids conflicts and permissions
issues when multiple processes try to create/access the same directories
and files under /data/misc/trace.
Bug: 148178774
Test: Verify that coverage files are written to
/data/misc/trace/<euid>/proc/... instead of /data/misc/trace/proc/...
Change-Id: If58081a12b2b9bc40cfcbf64c99beafe198d07af
Merged-In: If58081a12b2b9bc40cfcbf64c99beafe198d07af
(cherry picked from commit 100bbdc8f3)
Bug: http://b/134177005
Bug: http://b/116873221
Previously, the libprofile-extras dependency was added as a
LateStaticLib and the constructor in this library was included during
linking with the '-uinit_profile_extras' linker flag. This was done
because at the deps() stage, the exact binaries that need coverage are
not known (in fact the coverage-enabled variants are not created yet).
This meant that for a link command, if one of the shared libraries
already exported the constructor, the output of the link command did not
load/link libprofile-extras.
For other reasons, we now want to add more symbols to this library that
need to be linked into all libraries and executables. To accomplish
that, refactor the dependency handling so libprofile-extras can be added
as a 'WholeStaticLib'.
This is done by creating a new dependency type (with a coverageDepTag
dependency tag) to add libprofile-extras as a dependency for all modules
that can potentially link with coverage. During the flags() call, this
dependency is moved as a WholeStaticLib dependency iff coverage is
enabled in this link step.
There are a few NFC changes as well:
- deps() takes a DepsContext parameter.
- flags() has an extra PathDeps parameter and return value.
- add useSdk() helper to cc.Module.
Test: Build with coverage and check that we can generate coverage using
SIGUSR1 and the debug.coverage.flush sysprop.
Change-Id: I7e7d8201956a150febbda5bb1794f8ece016db8b
Merged-In: I7e7d8201956a150febbda5bb1794f8ece016db8b
(cherry picked from commit 82fe59b656)
Add a module type for tradefed libraries that causes an additional
copy to be installed at out/host/linux-x86/tradefed.
Bug: 143908003
Exclude merging into *-plus-aosp branches, since they already include the change.
Merged-In: I670345494abbad80dacab54471e635abfae3b3b3
Merged-In: I1cb25bbd5823a14f1a9428d58827677eff22ec7e
Change-Id: I55bca53e02588827374896ef87d58f3f4bdc48c7
(cherry picked from commit f0f2e2cf79)
Kousik Kumar