Commit Graph

262 Commits

Author SHA1 Message Date
Paul Duffin 2aae98f029 Maintain header order in merge_csv
Previously, if the --header property was not specified then merge_csv
would use a header constructed by sorting all the fields in the input
files. That required that any use of merge_csv which did not already
have headers in the required order would have to explicitly specify the
headers. That made it harder to use merge_csv as a generic tool as each
invocation needed to be aware of what headers were exported in the
output.

This change causes merge_csv to simply use the headers in the order in
which they are encountered in the input files. That removes the need to
specify the --header option when generating the index files.

Bug: 179354495
Test: m out/soong/hiddenapi/hiddenapi-index.csv out/soong/hiddenapi/hiddenapi-unsupported.csv
      - make sure that they are not changed by this change.
Merged-In: I420b7d07aea85af6372cd7580a8be5e2cc82a513
Change-Id: I420b7d07aea85af6372cd7580a8be5e2cc82a513
(cherry picked from commit 84c1cdf31f8f888ee9c0beb201d35ad9d293c341)
2021-06-16 03:06:33 +01:00
Martin Stjernholm 7e148881d0 Remove use of fake soong.variables.
Use the --soong-only flag instead to create it the proper way with the
config make step. This fixes build problems arising from the incomplete
soong.variables settings.

This partially relands https://r.android.com/1535202 so that we'll use
TARGET_PRODUCT=mainline_sdk in module SDK builds everywhere.
build-aml-prebuilts.sh isn't removed completely because we still need
the special case to create build_number.txt.

Cherry-picked from https://r.android.com/1729335.

Test: art/build/build-art-module.sh --skip-apex && \
      packages/modules/ArtPrebuilt/update-art-module-prebuilts.py \
        --local-dist out/dist
  then check that the created prebuilt dex2oatd64 runs with the repro
  in b/190194345
Test: build/soong/scripts/build-mainline-modules.sh
  then
      prebuilts/runtime/mainline/update.py --local-dist .../out/dist
  in a master-art tree and check that art/build/build-art-module.sh
  works there
Bug: 190194345
Bug: 174315599
Change-Id: I86327b082b61c292f18c0a6b315d9add08002832
Merged-In: I86327b082b61c292f18c0a6b315d9add08002832
2021-06-08 16:33:06 +01:00
Samiul Islam 129afaea73 Do not build compressed APEX
This script expects to build uncompressed APEX and use them as
prebuilts.

Bug: 189926000
Test: build_mainline_modules.sh
Change-Id: If0f62c2baa3db3c079b2913fcdc3dafd66394011
Merged-In: If0f62c2baa3db3c079b2913fcdc3dafd66394011
(cherry picked from commit d365449fae7fa4518d06236cc4ce794d2ec5b909)
2021-06-02 15:42:41 +01:00
Paul Duffin dfa1083fee Validate monolithic and modular hidden API flags are consistent
This makes sure that where there is overlap between the hidden API
flags generated for a module and the monolithic flags that they are
identical. That ensures that the modular hidden API flags will be
compatible with previous releases that relied on the monolithic flags.

Bug: 179354495
Test: m out/soong/.intermediates/art/build/boot/art-bootclasspath-fragment/android_common_apex10000/modular-hiddenapi/all-flags.csv
      m out/soong/hiddenapi/hiddenapi-flags.csv
      - Create some inconsistencies between the above two files.
      m out/soong/hiddenapi/hiddenapi-flags.csv.valid
Change-Id: Iaf9e23cef63e221608955d89dc8d496bcc70c86e
2021-05-14 01:49:19 +01:00
satayev 73c7b8547a Move conv_classpaths.proto to packages/modules/common.
The classpaths.proto file is needed at build time to generate appropriate
classpath configs for the modules. Unbundled ART branches do not include
packages/modules/SdkExtenstion in their manifest; so put the proto in a
project that should be visible to all unbundled modules.

Bug: 180105615
Test: m nothing
Change-Id: Ie5be1822fbee92f280332622a9317894cc98d7ac
2021-05-10 13:14:23 +01:00
David Srbecky 69315e4ea7 Use create_minidebuginfo tool instead of bash script.
The behaviour is semantically identical, however,
the tool additionally sorts the symbols by address,
compresses frame unwind information more efficiently,
and improves random-accessibility for lazy decompression.

Overall, the changes balance and the output size is same,
however, libunwindstack can access the data much faster
while using less memory (due to the lazy decompression).
It will also enable further improvements in the future.

Bug: 110133331
Test: ART unwinding tests, run prefetto on the device.
Change-Id: Id48f9fe67fb67fcf2b90cc3b217b71bb8f5147ca
2021-05-01 00:06:07 +01:00
Jaewoong Jung 3c87b1df11 Fix minor issues in updatability lint CLs.
Test: TreeHugger
Bug: 182349282
Change-Id: Ifae282d51b088d0562605b41a09696d75c5138f1
2021-04-22 13:40:41 -07:00
Paul Duffin 2082e695ce Allow generate_hiddenapi_lists.py to work with no custom flag files
Previously, if generate_hiddenapi_lists.py was invoked without any
custom flag files, e.g. hiddenapi-unsupported.txt, then it would fail.
This change allows it to be used for generating the hiddenapi lists
for modules that do not have any custom flag files,
e.g. framework-sdkextensions

Bug: 179354495
Test: verified that the monolithic out/soong/hiddenapi/... files are
      unchanged by this change
Change-Id: I455d453024c9f06ed59cbc1e9838234f8b7c7317
2021-04-21 17:15:25 +01:00
Jaewoong Jung 7b93908d2e Add lint_project_xml_test.py
Test: lint_project_xml_test.py
Bug: 182349282
Change-Id: Ibdeb2e5aaf9dcdb02c5d30214e5ea89665145eb3
2021-04-20 11:04:51 -07:00
Jaewoong Jung 5a4202575e Rename lint-project-xml.py to remove dashes.
So that a future test can import it without a dirty syntax.

Test: m lint-check
Bug: 182349282
Change-Id: I520a5af49543801ab2a8ee888ff235876546dc74
2021-04-20 07:06:03 -07:00
Jaewoong Jung a110439b3c Lint baseline file check in lint-project-xml
Add a function to lint-project-xml to scan the given lint baseline file
for disallowed baseline lint issues.

Test: Manual
Bug: 182349282
Change-Id: I0db32bec0da24487b2f2b3f6704629f56ae76f56
2021-04-19 17:38:17 -07:00
Treehugger Robot 7548b1ad05 Merge "Remove draft API support." 2021-04-15 22:58:15 +00:00
Jooyung Han e134d09831 Add 'merge' command to conv_linker_config
'merge' command can merge multiple configurations.
It just delegates protobuf's msg.MergeFrom(other_msg).

  usage: conv_linker_config merge -o out.pb -i a.pb -i b.pb ...

out.pb will have all keys from multiple inputs.

Bug: 181093750
Test: m
Test: MicrodroidTestCase
Change-Id: Ibf715aa0bcc1e5c82c85a9af8fe7dca8d6ab68ad
2021-04-15 10:58:21 +09:00
Dan Albert d05ba00415 Remove draft API support.
This is unused and doesn't work with our current API review process
anyway (un-drafting an API won't be flagged for API council review).

Test: treehugger
Bug: None
Change-Id: I6d8fcc9885b82dac5ada7772d9e3fb9101524ece
2021-04-13 15:55:47 -07:00
David Srbecky 789fe0fdeb Revert^2 "[strip.sh] Move remaining GNU binutils usage to llvm binutils"
This reverts commit 33dfba1d9c.

Bug: 141010852
Test: art/test.py -t 137-cfi
Test: llvm-objcopy --dump-section=.gnu_debugdata=mdi.xz libhwui.so && \
      unxz -fk mdi.xz && readelf -a mdi | less
Change-Id: I751b51274a961194647ec11fd3fd49886df1e1b3
2021-04-08 18:31:46 +01:00
Artur Satayev ca6fe3453a Add a converter for classpaths.proto between proto and json/text formats.
The proto is used at build time to define classpath fragments to be read
and parsed at runtime by `derive_classpath` service; in order to define
and set BOOTCLASSPATH, DEX2OATCLASSPATH, and SYSTEMSERVERCLASSPATH
environ variables.

The tool is used for debugging and generation of binary protos.

Bug: 180105615
Test: m && launch_cvd with a follow up changes
Change-Id: I7e8b084e5f63e9aaad11da3221b909818e69e235
2021-04-06 18:37:13 +01:00
Yi Kong 6f43f54264 Fix toc.sh, llvm-nm does not recognise '-f P' option
llvm-nm only accepts full name for format option, or just '-P' alias.

Test: build
Bug: 184360305
Change-Id: Ie25f5410403f4dbca564430149a5753e0d48d60d
2021-04-03 03:21:24 +08:00
Yi Kong 948205699b Merge "Move toc.sh to use LLVM binutils" 2021-04-02 02:38:16 +00:00
Ulyana Trafimovich 400a1b1309 Merge "Emit a better error message in manifest_check.py." 2021-04-01 09:58:15 +00:00
Yi Kong 4ad44e713b Move toc.sh to use LLVM binutils
Test: build
Change-Id: Ib965b6ea3fba6ae007135ac7e167cc50f0f6e3de
2021-04-01 17:45:42 +08:00
Yi Kong 9d9e053964 Merge "Move gen_sorted_bss_symbols.sh to use LLVM binutils" 2021-04-01 03:10:08 +00:00
Treehugger Robot fdfb9540f6 Merge "Do not build stats-log-api-gen-exports in mainline builds" 2021-03-31 18:48:18 +00:00
Ulya Trafimovich bb7513d80a Emit a better error message in manifest_check.py.
Bug: 132357300
Test: lunch aosp_cf_x86_64_phone-userdebug && m
Test: temporarily patch Gallery2 to fail the check, observe the error
Change-Id: Id67e359188396f68dcecd8b3f4d1bc26271af56d
2021-03-31 14:31:49 +01:00
Yi Kong bf1db7b149 Move gen_sorted_bss_symbols.sh to use LLVM binutils
Test: build
Change-Id: I496b92d963d127087daef405dafa24377d6bac97
2021-03-31 17:13:43 +08:00
Sophie Zheng dd3797b44e Merge "NDK API coverage" 2021-03-30 20:44:44 +00:00
Sophie Zheng a4c09e1894 Merge "NDK API coverage" 2021-03-30 20:40:41 +00:00
Eric Holk 406765b15a Do not build stats-log-api-gen-exports in mainline builds
This no longer needs to be a prebuilt since stats-log-api-gen can be
built from proto_logging without extra dependencies.

Test: Treehugger
Bug: 170149255
Change-Id: I6925c2526cee1cbff3ba78bedd7f6c2f58dfff0f
2021-03-29 15:53:32 -07:00
sophiez d3e2742ad2 NDK API coverage
Update OWNERS file under build/soong/scripts to include coverage team as owner of related scripts.

Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Change-Id: I6150d5bda0d59828a985cd2ab5dd0aa38a7987c3
2021-03-29 20:29:35 +00:00
Ulya Trafimovich 9f12df9abb Don't fail the build if manifest_check cannot extract targetSdkVersion.
Instead, return "any" SDK version 10000, which will cause dexpreopt to
not add any compatibility libraries to class loader context. This is as
good guess as any, and the build system already uses "any" version if
there is not manifest or APK from which to extract targetSdkVersion.

This fixes broken build on cf_x86_auto-userdebug.

Bug: 132357300
Bug: 183943146
Test: lunch cf_x86_auto-userdebug \
  && m out/target/product/vsoc_x86/obj/APPS/playback_intermediates/dexpreopt.zip
Change-Id: I5ee75639ece716c3a9800ddea346fe7c73da7c3a
2021-03-29 15:43:38 +00:00
Ulyana Trafimovich a8f6db972f Merge changes I1f0ab4af,Iea3be0fc
* changes:
  Fix AAPT path in dexpreopt.
  Allow modules that don't run verify_uses_libraries to have nonempty CLC.
2021-03-29 09:24:00 +00:00
sophiez ae5da5ba78 NDK API coverage
Update the gen_ndk_usedby_apex.sh script to generate NDK API list with
library name.

Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only

Change-Id: I799e03ad7d81a4184229f658fd6c8824b9ba6e1f
2021-03-25 22:12:04 +00:00
Treehugger Robot c29088b727 Merge "Remove .rustc ELF section from mini-debug-info." 2021-03-23 09:13:53 +00:00
Ulya Trafimovich bd6b076f80 Allow modules that don't run verify_uses_libraries to have nonempty CLC.
There are modules that would have passed verify_uses_library check, but
don't run it for some reason (the check gets enabled either with an
explicit setting, or if the module has nonempty <ues-library> lists in
the build properties). Previously all such modules were assumed to have
empty CLC, which is not always true. In particular, compatibility
libraries are ignored, which affected e.g. Calendar and messaging apps.

This CL gives such apps a chance to have correct CLC. The goal for the
future is to enforce verify_library_check by default.

Bug: 132357300
Test: lunch aosp_cf_x86_64_phone-userdebug && m
Change-Id: Iea3be0fc9d7775c52950848b5a3fd3b7fcd36c53
2021-03-22 10:49:31 +00:00
satayev 2b077baa5e Merge changes from topic "move_allowed_deps_txt"
* changes:
  Treat allowed_deps.txt source file as optional.
  Move allowed_deps.txt to packages/modules/common.
2021-03-22 10:48:45 +00:00
David Srbecky b474fc0ad6 Remove .rustc ELF section from mini-debug-info.
The section is stripped from binaries (but preserved in /symbols).

However, is it also unintentionally copied into the minidebuginfo
compressed section, which is included in the final shipped binary.
(This does not affect symtab symbols, which are handled as usual)

Removing it from mini-debug-info shrinks some rust binaries by
up to 10x, saving >10MB on the system partition overall (AOSP).

Test: extract minidebuginfo and check it manually with readelf
Change-Id: I462d81455626bb2b5244e1bf3d5f8b71646f3401
2021-03-20 20:31:29 +00:00
Treehugger Robot f2be52c4dc Merge "Revert "[strip.sh] Move remaining GNU binutils usage to llvm binutils"" 2021-03-20 19:25:23 +00:00
David Srbecky 33dfba1d9c Revert "[strip.sh] Move remaining GNU binutils usage to llvm binutils"
This reverts commit ad50ce8349.

Bug: 183237575
Reason for revert: Breaks stack unwinding

Change-Id: I256dc77525687fb3e670355587b1eac72158b274
2021-03-20 14:46:15 +00:00
Treehugger Robot 8cc04e59e1 Merge "manifest_check.py: translate library names using dexpreopt configs." 2021-03-20 12:12:52 +00:00
Julien Desprez 2dc86b2956 Migrate manifest_*_test out of test mapping
Unit tests do not require an explicit TEST_MAPPING config

Change-Id: Ieccebc78a26b9555803a2278e2962d0b388a6957
Test: presubmit
Bug: 183209529
2021-03-19 17:16:16 +00:00
Yi Kong ad50ce8349 [strip.sh] Move remaining GNU binutils usage to llvm binutils
With clang r377782, llvm binutils implements all the necessary
funtionaility for strip.sh. We can finally get rid of all the fallback
GNU binutils usage.

Test: m
Bug: 141010852
Bug: 135627985
Change-Id: I110f6028dab7f599decf59a5cb1b927b35e11857
2021-03-19 12:28:44 +08:00
Artur Satayev fdb61edf43 Move allowed_deps.txt to packages/modules/common.
Bug: 179234385
Test: run update-apex-allowed-deps.sh locally
Change-Id: I8e8864468b87342c688d001bc5f6e6f8416863ed
Merged-In: I8e8864468b87342c688d001bc5f6e6f8416863ed
2021-03-17 13:44:16 +00:00
Ulya Trafimovich 3c902e7acc manifest_check.py: translate library names using dexpreopt configs.
Java modules that are defined in makefiles are not processed in
topological order, so it is necessary to communicate information from
dependencies via dexpreopt.config files. This has already been done in
make/core/dex_preopt_config_merger.py, and now manifest_check.py also
needs to get library names from their dexpreopt.config files.

This is to accommodate Java libraries which name differs from their
modules name. Soong properties `uses_libs`/`optional_uses_libs` and
makefile vars `LOCAL_USES_LIBRARIES`/`LOCAL_OPTIONAL_USES_LIBRARIES`
contain module names, not library names, so it is necessary to translate
them when comparing against library names in the manifest.

Bug: 132357300

Test: lunch cf_x86_64_phone-userdebug && m && launch_cvd \
      adb wait-for-device && adb root && adb logcat \
      | grep -E 'ClassLoaderContext [a-z ]+ mismatch'
      # empty grep output, no errors

Change-Id: I7cfb17d6d74e37fc9f37d977f15f0d711bab5c8e
2021-03-16 15:43:05 +00:00
Ulya Trafimovich 0aba252e15 Reimplement verify_uses_libraries.sh in manifest_check.py.
Previously there were two different scripts that did similar things:
1) build/soong/scripts/manifest_check.py
2) build/make/core/verify_uses_libraries.sh

Both scripts extracted <uses-library> tags and `targetSdkVersion` from
the manifests of Java modules, but 1) worked for XML manifests, and 2)
worked for APKs. This CL reimplements the functionality from 2) in 1),
so that one script can handle both XML manifests and APKs.

Bug: 132357300

Test: lunch cf_x86_64_phone-userdebug && m && launch_cvd \
      adb wait-for-device && adb root && adb logcat \
      | grep -E 'ClassLoaderContext [a-z ]+ mismatch'
      # empty grep output, no errors

Change-Id: Id1b66e4f3f30f307dba70cb111c7571762cb546a
2021-03-15 13:50:49 +00:00
Martin Stjernholm 4f0f3f01d9 Build heapprofd_client_api for use in the ART chroot.
Test: build/soong/scripts/build-mainline-modules.sh
Bug: 179915934
Change-Id: I4653bf705563ad2ddb077ecae7938c2dd1b818e9
2021-03-09 17:13:51 +00:00
Treehugger Robot 7ebe464748 Merge "Build statsd-module-sdk-for-art" 2021-02-24 01:08:28 +00:00
David Srbecky bb96b56b0d Merge "Set block-size for XZ mini-debug-info compression." 2021-02-24 00:43:12 +00:00
Eric Holk ff0529d209 Build statsd-module-sdk-for-art
Bug: 178236337
Test: build-mainline-modules.sh
Change-Id: I51dab13416be8ea1cd65234d560d84ca89b53e13
2021-02-23 22:49:57 +00:00
Treehugger Robot 7eacdbf8a6 Merge "Add non-fatal mode for verify_uses_libraries check." 2021-02-23 20:45:42 +00:00
Paul Duffin ece454400d Merge changes from topic "hiddenapi_additional_annotations"
* changes:
  Sort hiddenapi monolithic files by signature
  Remove duplicates in monolithic hidden API files
  Remove implicit dependency from <x> -> <x>-hiddenapi
2021-02-23 12:48:24 +00:00
Ulya Trafimovich 8c35fcf4d5 Add non-fatal mode for verify_uses_libraries check.
The new mode is enabled with environment variable
RELAX_USES_LIBRARY_CHECK. If the variable is set to true, then a
verify_uses_libraries check failure does not fail the build, instead it
sets a special compiler filter "extract" for dexpreopt, which means that
the DEX file will be extracted, but it won't be compiled to native code.
Class loader context will be set to empty in this case (not &, as it is
going to be deprecated soon).

If the variable RELAX_USES_LIBRARY_CHECK is unset or set to something
other than "true", then the old behaviour of the verify_uses_libraries
check is preserved.

The intended use case for this flag is to have a smoother migration path
for the Java modules that need to add <uses-library> information in
the build files. The flag allows to quickly silence build errors. This
flag should be used with caution and only as a temporary measure, as it
masks real errors and affects performance.

verify_uses_libraries check is reworked so that it writes the error
message to a status file (which is used instead of the former timestamp
file). Currently the stored error message is not used, but it may be
used later to produce a warning. Dexpreopt command checks if the status
file exists and is nonempty; if that is the case, then compiler filter
is set to "extract".
Bug: 132357300

Test: Manually add some mismatch between the libraries in the Android.bp
      and Android.mk files for dexpreopted apps, build with
      RELAX_USES_LIBRARY_CHECK=true and obsserve that the build doesn't
      fail and they are compiled with compiler-filter "extract".
      Unset RELAX_USES_LIBRARY_CHECK and observe that the build fails.

Change-Id: Ibb5d993a25b1df1d2e70b7d5aafc6997f9d64e67
2021-02-22 15:41:03 +00:00