Previously there were two different scripts that did similar things:
1) build/soong/scripts/manifest_check.py
2) build/make/core/verify_uses_libraries.sh
Both scripts extracted <uses-library> tags and `targetSdkVersion` from
the manifests of Java modules, but 1) worked for XML manifests, and 2)
worked for APKs. This CL reimplements the functionality from 2) in 1),
so that one script can handle both XML manifests and APKs.
Bug: 132357300
Test: lunch cf_x86_64_phone-userdebug && m && launch_cvd \
adb wait-for-device && adb root && adb logcat \
| grep -E 'ClassLoaderContext [a-z ]+ mismatch'
# empty grep output, no errors
Change-Id: Id1b66e4f3f30f307dba70cb111c7571762cb546a
The tests in dexpreopt are unit tests that don't require the use of
test fixtures but there are a number of tests in java and apex packages
that need to configure dexpreopt behavior so this changes adds support
for modifying the dexpreopt configuration in test fixtures.
Bug: 181070625
Test: m nothing
Change-Id: I7b64c9cf0a94cc548c27b55c5a7875be5ff0e18d
In cases when class loader context cannot be computed at build time and
verify_uses_libraries check is relaxed (RELAX_USES_LIBRARY_CHECK=true)
dexpreopt uses a special compiler filter that suppresses AOT-compilation
to native code and applies only those optimizations that do not require
class loader context. Previously the "extract" filter was used. Now ART
supports using "verify" filter in this configuration.
Bug: 132357300
Test: lunch cf_x86_64_phone-userdebug && m && launch_cvd \
adb wait-for-device && adb root && adb logcat \
| grep -E 'ClassLoaderContext [a-z ]+ mismatch'
# empty grep output, no errors
Change-Id: Id4933aa94ea96894278355283383c16d103e98c9
The new mode is enabled with environment variable
RELAX_USES_LIBRARY_CHECK. If the variable is set to true, then a
verify_uses_libraries check failure does not fail the build, instead it
sets a special compiler filter "extract" for dexpreopt, which means that
the DEX file will be extracted, but it won't be compiled to native code.
Class loader context will be set to empty in this case (not &, as it is
going to be deprecated soon).
If the variable RELAX_USES_LIBRARY_CHECK is unset or set to something
other than "true", then the old behaviour of the verify_uses_libraries
check is preserved.
The intended use case for this flag is to have a smoother migration path
for the Java modules that need to add <uses-library> information in
the build files. The flag allows to quickly silence build errors. This
flag should be used with caution and only as a temporary measure, as it
masks real errors and affects performance.
verify_uses_libraries check is reworked so that it writes the error
message to a status file (which is used instead of the former timestamp
file). Currently the stored error message is not used, but it may be
used later to produce a warning. Dexpreopt command checks if the status
file exists and is nonempty; if that is the case, then compiler filter
is set to "extract".
Bug: 132357300
Test: Manually add some mismatch between the libraries in the Android.bp
and Android.mk files for dexpreopted apps, build with
RELAX_USES_LIBRARY_CHECK=true and obsserve that the build doesn't
fail and they are compiled with compiler-filter "extract".
Unset RELAX_USES_LIBRARY_CHECK and observe that the build fails.
Change-Id: Ibb5d993a25b1df1d2e70b7d5aafc6997f9d64e67
Library order is important because it is used to construct class loader
context, which is then written into OAT/ODEX files and chacked against
class loader context constructed by PackageManager on the device. If the
orders are different, dexpreopted code is rejected.
Soong avoids using Go maps for class loader context representation
precisely for that reason. However, for the modules defined in makefiles
dexpreopt configs were serialized to JSON and unmarshaled to Go maps,
which resulted in wrong order of libraries. This CL changes Go
representation of class loader contexts imported from JSON and makes the
order stable.
Bug: 132357300
Test: lunch cf_x86_64_phone-userdebug && m && launch_cvd \
adb wait-for-device && adb root && adb logcat \
| grep -E 'ClassLoaderContext [a-z ]+ mismatch'
# empty grep output, no errors
Change-Id: I15f51617f9573c0bbcb324cf2592daf719cad586
This is needed for Java libraries that are <uses-library> dependencies
of Java libraries and apps defined as Make modules. Each dexpreopted
module in Make generates a dexpreopt.config file, which incorporates
information from its dependencies' dexpreopt.config files. For
dependencies that are Make modules their dexpreopt.config files are
generated by Make, and for Soong modules they are generated by Soong.
Since Soong doesn't know which libraries are used by Make, it generates
build rules for a superset of the necessary libraries.
Bug: 132357300
Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: I325b1037658736ee3c02450b08c00eca1a175962
Previously Make class loader contexts used a simplified one-level map.
Now they use the same tree representation as in Soong. The difference
between Make and Soong represenataions is insubstantial and caused only
by the language differences between JSON and Go.
Bug: 132357300
Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: Ia273afb7f026dbe3b25d4327d55143657b026b98
Previously, the createGlobalSoongConfig() function was explicitly
prevented from being used in tests because it would fail. However, it
turns out that is no longer the case and it does now work.
That allows the following changes to be made:
* Tests no longer need to use GlobalSoongConfigForTests() to
prepopulate the cache.
* GlobalSoongConfigForTests() is only needed in the dexpreopt
package.
Bug: 177892522
Test: m nothing
Change-Id: Ifcbb1a44254c5d2d10c1d02ab23227488d1d1ed1
Bug: 176171716
Test: build_mainline_modules.sh and ensure that the ART boot image is
built: `find out -type f -name '*.art'` is nonempty for every arch.
Change-Id: Ibc45581eef2b205c750a30709780cf659ba7cfa1
The removed API has been unused since https://r.android.com/1533342
(except for unit tests).
Changes in the unit tests reflect the change of API in
https://r.android.com/1533342: early errors caused by unknown library
paths at CLC construction time have been replaced with late errors at
the time of CLC validation.
Bug: 132357300
Test: m nothing
Change-Id: I739c7c41b6f882b7e28cdd6acd05961d754d8687
Previously CLC construction was scattered across different module types
and dependency tags. This CL moves all logic to one function, which
handles all special cases. This will allow to simplify CLC API and
reduce the number of different ways in which CLC is constructed.
Previously some of the cases failed early (at the time when a library is
added to CLC) if the build/install paths were unknown. Other cases did
not fail early, but were validated later before CLC was used. Late
failures are necessary because some of the libraries with unknown paths
still have to be processed by manifest_fixer (which doesn't need library
paths), but they do not use dexpreopt (which needs library paths). This
CL removes the early failures (all paths are still validated later).
The CLC tests do not fail because they use a private method that toggles
the "strict" flag (that enforces early/late failure mode) manually in
the method call.
The CL also makes a functional change in the way CLC is constructed for
component libraries that have an OptionalImplicitSdkLibrary(), or
libraries that are disguised as SDK libraries via `provides_uses_lib`.
Previously such a component/disguised library X was added to its own CLC
as a sibling element of X's own <uses-library> dependencies, which
created incorrect CLC structure. Now this is handled by addCLCFromDep,
when X is processed as dependency and added as a top-level CLC element
with its sub-CLC properly nested under it.
Bug: 132357300
Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: I6a512209b87b81d785875f10f76b21c81b2ed579
Enable the RuleBuilder and RuleBuilderCommand methods to access
the BuilderContext by passing it to NewRuleBuilder instead of
RuleBuilder.Build.
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I63e6597e19167393876dc2259d6f521363b7dabc
This patch reworks the approach introduced in
https://r.android.com/1450819. That patch based the decision which
libraries should be added to the manifest <uses-library> tags by the
manifest_fixer on the "shared" status of the library.
That approach is incorrect for two reasons:
- It doesn't make sense to have a non-shared library in class loader
context ("shared" libraries are those specified in
frameworks/base/data/etc/platform.xml, and they are the only ones
that PackageManager knows about).
- It doesn't make sense to add anything but the top-level of the
class loader context tree to the manifest, because this part of the
tree is flattened to a sequence, and PackageManager cannot restore
it to the previous tree shape (there is an information loss).
This patch removes the "shared" bit of information from class loader
context elements and assumes that all libraries that end up in class
loader context are shared. Consequently, only the top-level libraries
should be passed to manifest_fixer.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Bug: 168686456
Change-Id: I902690f0f38f1047fa79cf6ccbe956077eceaab0
When adding a subcontext in a class loader context tree, there are two
possible cases: 1) the root of the subcontext is itself a <uses-library>
and should be present as a node in the tree, or 2) the root is not a
<uses-library>, but some of its dependencies are -- in that case they
should be disconnected from the root, and the resulting forrest should
be added at the top-level.
Example:
1) C is a <uses-library>:
A
├── B
└── C
├── D
└── E
└── F
2) C is not a <uses-library>:
A
├── B
├── D
└── E
└── F
Before the patch subcontexts for transitive dependencies were added
before the subcontext for the direct dependency (even if it was a
<uses-library>, resulting in case-2 hierarchy when case-1 should have
been used. Previosuly this didn't matter because class loader context
was a flat set of libraries, but now it matters because class loader
context is a tree.
This patch changes the order in which libraries are added, so that
direct dependencies are added before transitive ones. The context adding
method now accepts an "implicit root" parameter, so that when adding
transitive dependencies it can check if the corresponding direct
dependency is a <uses-library> and already present in the context.
Partially constructed class loader context is now propagated top-down
into aapt.buildActions, so that the method can use existing part of the
context to decide where the missing part should be connected.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I649aff9e27494306885a4f4fc90226c399636b57
Nested conditional context doesn't make sense because conditional
context is only needed for compatibility libraries, and those are only
used by apps, and apps are always at the root of class loader context
tree. Therefore, trying to add conditional nested context can only mean
an error somewhere on the way.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: Iadeaaf89fdb11cb23b107c6fb074b1bc765256be
The android.WriteFile rule takes careful escaping to produce the
right contents. Wrap it in an android.WriteFileRule that handles
the escaping.
Test: compare all android.WriteFile outputs
Change-Id: If71a5843af47a37ca61714e1a1ebb32d08536c31
The old representation consisted of a list of libraries (UsesLibraries),
a list of optional libraries (OptionalUsesLibraries) and a mapping from
library name to its build/install paths (LibraryPaths). The separation
into lists and map was necessary because of special handling of
compatibility libraries, which is now unified with normal libraries.
The new representation is a mapping from target SDK version to a tree
structure ClassLoaderContext. Each node of the tree represents a library
and contains library name, build/install paths and a slice of
subcontexts for dependencies. The same library may occur in the tree
multiple times in case it is a dependency of multiple libraries. The
order in which libraries are added matters (the resulting tree shape may
be different).
Test results have to be updated, as the resulting <uses-library> list is
reodered (previously it was a sorted list of map keys, and now it is
formed by a depth-first preorder traversal of the class loader tree).
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Bug: 168686456
Change-Id: I11be8cd2967f004fd58753d7c5fb99fed179cd63
This dependency is incorrect and shouldn't exist.
It has been removed in https://r.android.com/1467918
Test: m nothing
Bug: 170710203
Bug: 132357300
Change-Id: If3036437e138b552436f135425e6bd15be043678
construct_context.py is a script that runs at Ninja stage and constructs
class loader context arguments for dex2oat. Previously it accepted lists
of library paths and concatenated them into one class loader context
string. The script also handled the dependency of "android.hidl.manager"
on "android.hidl.base" that is not tracked by the build system and must
be handled in a special way.
Now that class loader context representation is going to change from
flat lists to trees, passing paths to individual libraries as script
arguments is no longer possible, because a list of paths cannot
represent a class loader context tree. Passing the trees in a serialized
form is also inconvenient, because the script would have to parse them,
which would complicate it a lot.
Therefore this patch ports all the concatenation and "android.hidl.base"
handling to Soong. It is not possible to port the remaining script to
Soong because Soong has no information about the targetSdkVersiion of
the dexpreopted apps (it is in the manifest and sometimes in an APK, and
has to be extracted at Ninja time).
Test: construct_context_test.py
Test: new subtests in TestUsesLibs
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: Icdb03cf00d1e27e4cff3844b89bfaec4de502dd7
Mimick the way PackageManager handles it at runtime: do add it to class
loader context for apps with targetSdkVersion < 30, but only if
"android.test.runner" is used. Previously it was not added at all.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I4c06635277ab13e21069b9fa0b46eb6a2547dfdd
Also, add tests for compatibility libraries in class loader context.
This CL separates special-case handling of compatibility libraries into
a "fixup" step that is done after class loader context is constructed by
Soong. This allows to handle compatibility libraries and normal
libraries uniformly, which should enable further simplification of class
loader context representation (in subsequent CLs).
Currently the only "fixup" step is removal of libraries from conditional
class loader context if they already are in unconditional context. This
check cannot be done at the time when the libraries are added to
conditional context, because the full uncoditional context is not yet
known at that time. Previously construction of unconditional context was
delayed, now it is no longer delayed and handled in the same way as
unconditional context, and the "fixup" does the filtering.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: Ie71e9fb2d3d529b5317cd84e09ab3c853017c349
This change removes the properties in GlobalJSONConfig that are used to
hold the unmarshalled []string along with the code to convert them into
ConfiguredJarList as they are no longer necessary because the
ConfiguredJarList.UnmarshalJSON(..) method can handle that automatically.
Bug: 171479578
Test: m nothing
Change-Id: Ie297cd00cb23d12e544d922454868db6fe9eb77c
This reverts commit 323dc60712.
Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests
Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.
Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
This doesn't apply to compatibility libraries (those added to
conditional class loader context base on the targetSdkVersion in the
manifest), because Soong doesn't know if they are needed or not (the
build rule that parses the manifest runs at Ninja stage).
In the future we may need to relax this and instead of failing the build
to disable dexpreopt in such cases. But currently there are no use cases
that fail.
Also, this patch replaces hardcoded install path with a constant.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: Ic894bc17e8ffd8ff173111ea09366252001ef3e9
Previously, the patterns in SYSTEM_OTHER_ODEX_FILTER only matched
subpaths of /system. Broaden this to match /<p> as well as /system/<p>.
Because SYSTEM_OTHER_ODEX_FILTER includes system_ext and product apps,
the immediate effect of this is that the odex and vdex files of
apps on those partitions move to system_other for A/B devices.
This makes the product and system_ext partitions smaller, which
in turn makes OTAs smaller.
Bug: 141707536
Test: soong unit tests
Test: (adb shell dumpsys package com.android.stk &&
adb shell dumpsys package com.android.quicksearchbox) | grep -i dexopt -A3;
verify dexpreopt state is 'prebuilt' for both
Test: adb shell find /data/dalvik-cache -name '*QuickSearch*.vdex' | wc -l == 1
Test: adb shell find /data/dalvik-cache -name '*Stk*.vdex' | wc -l == 1
Change-Id: I0cf141014b9ddf32159cff47b0325d9c78b549f2
This doesn't affect the cases with explicitly set `uses_libs` or
`optional_uses_libs`, only the cases when the library implicitly added
to the manifest_fixer by Soong. "org.apache.http.legacy" should be
handled in the same way as "android.test.base", except that they have
been added in different SDK versions.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I4fa1f082bfdb67f227f432717731a7e3d790913c
Previously default install path was provided only for comatibility
libraries that are added explicitly via `uses_libs`/`optional_uses_libs`
properties. This didn't take into account compatibility libraries that
are added by Soong when it computes transitive closure of SDK library
dependencies. As a result, install path in such cases remained unknown,
and the corresponding compatibility library was omitted from class
loader context, which caused 'ClassLoaderContext shared library size
mismatch' errors at first boot on device.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: If661272f46803299c8640f7f5ef161c8217d07e4
Ulya Trafimovich