1. Instead of 'slim' config, use full config even for libs
2. Define moduleJSONConfig for fields which cannot be converted to JSON
field directly(Path type field, ProfileBootListing,
DexPreoptImagesDeps are added in this CL) and exclude fields which is
convertible(DexPreoptImageLocations)
Bug: 158843648
Test: m dist
Change-Id: I3f9192ab5292bd079be1b686bb3b25735a836cbc
Creates a new deptag type for it so that it can implement the marker
interfaces that will exclude it from being added to the APEX and from
visibility enforcement. The latter is probably not an issue ATM because
the dependencies are added after visibility checks are enforced but
this code is undergoing lots of refactoring so that may change.
Bug: 177892522
Test: m nothing
Change-Id: Ibd167d557adec761a2e3eed78f4d334c40a04fb9
This CL handles updatable boot jars in the same hacky way as we handle
non-updatable boot jars: it creates a set of predefined paths to the dex
jars in a global config, then traverses all modules in a singleton
context, finds updatable boot jars and adds copy rules from these jars
to the predefined paths. A proper way would be to register dependencies
of the dexpreopted modules on the boot jars and extracting paths to dex
files by walking these dependencies.
Bug: 178467404
Test: lunch aosp_cf_x86_64_phone-userdebug && m
Test: added new Soong test
Change-Id: I87f764109315f79315d73bf43799b70eb010fc0b
In cases when class loader context cannot be computed at build time and
verify_uses_libraries check is relaxed (RELAX_USES_LIBRARY_CHECK=true)
dexpreopt uses a special compiler filter that suppresses AOT-compilation
to native code and applies only those optimizations that do not require
class loader context. Previously the "extract" filter was used. Now ART
supports using "verify" filter in this configuration.
Bug: 132357300
Test: lunch cf_x86_64_phone-userdebug && m && launch_cvd \
adb wait-for-device && adb root && adb logcat \
| grep -E 'ClassLoaderContext [a-z ]+ mismatch'
# empty grep output, no errors
Change-Id: Id4933aa94ea96894278355283383c16d103e98c9
The new mode is enabled with environment variable
RELAX_USES_LIBRARY_CHECK. If the variable is set to true, then a
verify_uses_libraries check failure does not fail the build, instead it
sets a special compiler filter "extract" for dexpreopt, which means that
the DEX file will be extracted, but it won't be compiled to native code.
Class loader context will be set to empty in this case (not &, as it is
going to be deprecated soon).
If the variable RELAX_USES_LIBRARY_CHECK is unset or set to something
other than "true", then the old behaviour of the verify_uses_libraries
check is preserved.
The intended use case for this flag is to have a smoother migration path
for the Java modules that need to add <uses-library> information in
the build files. The flag allows to quickly silence build errors. This
flag should be used with caution and only as a temporary measure, as it
masks real errors and affects performance.
verify_uses_libraries check is reworked so that it writes the error
message to a status file (which is used instead of the former timestamp
file). Currently the stored error message is not used, but it may be
used later to produce a warning. Dexpreopt command checks if the status
file exists and is nonempty; if that is the case, then compiler filter
is set to "extract".
Bug: 132357300
Test: Manually add some mismatch between the libraries in the Android.bp
and Android.mk files for dexpreopted apps, build with
RELAX_USES_LIBRARY_CHECK=true and obsserve that the build doesn't
fail and they are compiled with compiler-filter "extract".
Unset RELAX_USES_LIBRARY_CHECK and observe that the build fails.
Change-Id: Ibb5d993a25b1df1d2e70b7d5aafc6997f9d64e67
This is needed for Java libraries that are <uses-library> dependencies
of Java libraries and apps defined as Make modules. Each dexpreopted
module in Make generates a dexpreopt.config file, which incorporates
information from its dependencies' dexpreopt.config files. For
dependencies that are Make modules their dexpreopt.config files are
generated by Make, and for Soong modules they are generated by Soong.
Since Soong doesn't know which libraries are used by Make, it generates
build rules for a superset of the necessary libraries.
Bug: 132357300
Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: I325b1037658736ee3c02450b08c00eca1a175962
Previously, the createGlobalSoongConfig() function was explicitly
prevented from being used in tests because it would fail. However, it
turns out that is no longer the case and it does now work.
That allows the following changes to be made:
* Tests no longer need to use GlobalSoongConfigForTests() to
prepopulate the cache.
* GlobalSoongConfigForTests() is only needed in the dexpreopt
package.
Bug: 177892522
Test: m nothing
Change-Id: Ifcbb1a44254c5d2d10c1d02ab23227488d1d1ed1
Bug: 176171716
Test: build_mainline_modules.sh and ensure that the ART boot image is
built: `find out -type f -name '*.art'` is nonempty for every arch.
Change-Id: Ibc45581eef2b205c750a30709780cf659ba7cfa1
The android.WriteFile rule takes careful escaping to produce the
right contents. Wrap it in an android.WriteFileRule that handles
the escaping.
Test: compare all android.WriteFile outputs
Change-Id: If71a5843af47a37ca61714e1a1ebb32d08536c31
The old representation consisted of a list of libraries (UsesLibraries),
a list of optional libraries (OptionalUsesLibraries) and a mapping from
library name to its build/install paths (LibraryPaths). The separation
into lists and map was necessary because of special handling of
compatibility libraries, which is now unified with normal libraries.
The new representation is a mapping from target SDK version to a tree
structure ClassLoaderContext. Each node of the tree represents a library
and contains library name, build/install paths and a slice of
subcontexts for dependencies. The same library may occur in the tree
multiple times in case it is a dependency of multiple libraries. The
order in which libraries are added matters (the resulting tree shape may
be different).
Test results have to be updated, as the resulting <uses-library> list is
reodered (previously it was a sorted list of map keys, and now it is
formed by a depth-first preorder traversal of the class loader tree).
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Bug: 168686456
Change-Id: I11be8cd2967f004fd58753d7c5fb99fed179cd63
Also, add tests for compatibility libraries in class loader context.
This CL separates special-case handling of compatibility libraries into
a "fixup" step that is done after class loader context is constructed by
Soong. This allows to handle compatibility libraries and normal
libraries uniformly, which should enable further simplification of class
loader context representation (in subsequent CLs).
Currently the only "fixup" step is removal of libraries from conditional
class loader context if they already are in unconditional context. This
check cannot be done at the time when the libraries are added to
conditional context, because the full uncoditional context is not yet
known at that time. Previously construction of unconditional context was
delayed, now it is no longer delayed and handled in the same way as
unconditional context, and the "fixup" does the filtering.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: Ie71e9fb2d3d529b5317cd84e09ab3c853017c349
This change removes the properties in GlobalJSONConfig that are used to
hold the unmarshalled []string along with the code to convert them into
ConfiguredJarList as they are no longer necessary because the
ConfiguredJarList.UnmarshalJSON(..) method can handle that automatically.
Bug: 171479578
Test: m nothing
Change-Id: Ie297cd00cb23d12e544d922454868db6fe9eb77c
This doesn't affect the cases with explicitly set `uses_libs` or
`optional_uses_libs`, only the cases when the library implicitly added
to the manifest_fixer by Soong. "org.apache.http.legacy" should be
handled in the same way as "android.test.base", except that they have
been added in different SDK versions.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I4fa1f082bfdb67f227f432717731a7e3d790913c
Previously default install path was provided only for comatibility
libraries that are added explicitly via `uses_libs`/`optional_uses_libs`
properties. This didn't take into account compatibility libraries that
are added by Soong when it computes transitive closure of SDK library
dependencies. As a result, install path in such cases remained unknown,
and the corresponding compatibility library was omitted from class
loader context, which caused 'ClassLoaderContext shared library size
mismatch' errors at first boot on device.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: If661272f46803299c8640f7f5ef161c8217d07e4
Allow default install paths only for compatibility libraries. For other
libraries that are explicitly specified in `uses_libs` and
`optional_uses_libs` unknown install path should be an error.
Bug: 132357300
Test: lunch cf_x86_phone-userdebug && m
Change-Id: I2209c90a939a8aa46c42e13bb42d09c07e4d0895
For some dependencies, like stubs, the SDK library may not be found at
build time (either because the implementation library is not among the
dependencies of the dexpreopted module, or because it's part of a
prebuilt, or because it's missing from the build altogether). In such
cases dexpreopt is useless, because dex2oat does not have access to the
full classpath (unless the &-classpath is used). Therefore do not
dexpreopt in such cases.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: If289088cfd103011ccb16165e95a97b30fd31b81
Add a new helper SortedIntKeys similar to SortedStringKeys.
Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: I08a43ec2cae7d1a82531295aca1a0658e3a0dd6f
Previously only the names were collected, and later used in the
manifest_fixer to add missing <uses-library> entries to the manifest.
Now we also need to collect build-time and on-device paths, to be used
in class loader context for dexpreopt. This commit only collects paths,
but does not pass them to dexpreopt yet.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I34b229ee68f16ba215ba03770feadb4d890ec2bf
With the addition of apexes and /system_ext some of the bootclasspath
and system server jars have moved from /system to the new locations.
This has been implemented by using lists of colon-separated strings
called "apex-jar pairs" (although "apex" was misleading as it could
refer to "platform" or "system_ext", not necessarily a real apex).
Using the colon-separated string representation is inconvenient, as it
requires splitting and reassembling the list components many times,
which harms performance and makes error handling difficult. Therefore
this patch refactors the colon-separated lists into a struct that
hides the implementation details.
Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: Id248ce639a267076294f4d4d73971da2f2f77208
This allows to reuse SDK version comparison routine from other Python
scripts. With the addition of non-numeric versions comparison has become
more complex, and the deleted shell script did it incorrectly: it used
comparisons like `[[ "S" -lt 28 ]]` which results in "true" as strings
are converted to zero in numeric context. This resulted in adding legacy
libraries to class loader context of apps targeting recent SDK versions.
The error was masked because currently there is only one non-AOSP app
that uses the script (GoogleDialer), and it targets numeric SDK version.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I26e752e29b5453fd3e29d5dbfbe4d9df9c0a55b7
This script belongs with other manifest-related scripts, and the future
plan is to rewrite it in Python and share common functionality with
other scripts.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I4b90129e5023ca1a2c818fc160c34b0b7da943ef
Different names of the same field in Make and in Soong resulted in
the field being dropped when unmarshalling the JSON from Make to Soong.
As a result, the list of optional used libraries for Make modules was
always empty.
Test: lunch aosp_cf_x86_phone-userdebug && m
Test: Cherry-pick in internal master and observe that the logcat error
"ClassLoaderContext shared library size mismatch" for GoogleDialer
has disappeared:
terminal-1 $ lunch cf_x86_phone-userdebug && m && launch_cvd
terminal-2 $ adb wait-for-device && adb root \
&& adb logcat | grep 'size mismatch'
(nothing after the change, one error message before the change)
Bug: 132357300
Change-Id: I57820646fa7c1d8af01b2cbc8a3521da450719c0
This is a preliminary CL before fixing on-device paths to DEX jars.
It groups together the inormation about on-host build paths and
on-device install paths to library DEX jars.
This CL changes the structure of module dexpreopt.config files
generated by the build system. Aside of that, no functional changes.
Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: I059654be7670f2ba66248d9c49b9694a0591f9c1
The order is non-optimal if some jar X precedes its dependency Y. In
that case dexpreopt will be unable to resolve any rerefences from X to
Y. Raise an error unless the product sets variable
PRODUCT_BROKEN_SUBOPTIMAL_ORDER_OF_SYSTEM_SERVER_JARS to true in the
makefile. This is to prevent regressions in existing products that
currently have correct order.
Test: lunch cf_x86_phone-userdebug && m nothing
Test: lunch aosp_car_arm && m nothing
Bug: 140451054
Change-Id: Iafe8fe0e992deb628e38f2321d4601a4804a3c79
Merged-In: Iafe8fe0e992deb628e38f2321d4601a4804a3c79
(cherry picked from commit fb863c87c4)
Also disable dexpreopting for host. These are necessary to avoid adding
dependencies on dex2oat in various non-platform builds where they will
break.
Since we cannot assume at least one module enables dexpreopting now,
the two dexpreopt singletons are silently disabled if there has been no
call to dexpreopt.GetGlobalSoongConfig.
Bug: 145934348
Bug: 148312086
Bug: 148319588
Bug: 148690468
Test: m
Test: env OUT_DIR=out-tools prebuilts/build-tools/build-prebuilts.sh
on the aosp-build-tools branch
Test: build/soong/soong_ui.bash --make-mode static_sdk_tools dist DIST_DIR=out-dist BUILD_HOST_static=1
on internal (cf b/148312086#comment8)
Test: build/soong/soong_ui.bash --make-mode dist DIST_DIR=out-apps TARGET_BUILD_APPS=Launcher3 TARGET_BUILD_VARIANT=userdebug
on internal without art/ and external/vixl/ (cf b/148319588)
Change-Id: I240dade7204b87fc2d12181534ab23439eca8b46
Should cut down on a bit of copying, and also required for an upcoming
CL that'll change GetCachedGlobalSoongConfig.
Test: m nothing
Bug: 145934348
Change-Id: I6bed737d9b061b5239cc603ad881f4ccd4312e43
binary module.
This uses the Once cache for GlobalSoongConfig to propagate the dex2oat
path from a module dependency to the singletons (both the one that
writes out dexpreopt_soong.config and the one that creates the
dexpreopted boot images). Unless dexpreopting is disabled altogether
through DisablePreopt in dexpreopt.config, that means:
- We must ensure at least one module registers a dex2oat tool
dependency and resolves a GlobalSoongConfig using it, or else the
singletons fail. That means we litter dex2oat dependencies in java
modules even when they won't get dexpreopted and hence don't really
need them.
- We still assume there's at least one java_library or android_app in
the build.
This relands https://r.android.com/1205730 without changes - the
necessary fixes are in the child CLs.
Bug: 145934348
Test: m
(check that out/soong/dexpreopt_soong.config points to dex2oatd64)
Test: env USE_DEX2OAT_DEBUG=false m
(check that out/soong/dexpreopt_soong.config points to dex2oat64)
Test: env OUT_DIR=out-tools prebuilts/build-tools/build-prebuilts.sh
on the aosp-build-tools branch
Change-Id: I66661711b317d1e4ec434861982919bdde19b575
dexpreopt package.
Preparation for a future CL that will need to get the make-written
GlobalConfig from within dexpreopt.
Also rename the Load*Config functions to Parse*Config, since they don't
actually load the config files anymore.
This relands https://r.android.com/1211982.
Bug: 145934348
Test: m
Change-Id: Icb9332a93811d77d2d8b06e983b92501b180a358
caching of it.
Introduce a Once cache for GlobalSoongConfig to allow it to get binary
tool paths from ordinary module dependencies (coming in a future CL)
that are then reused in singletons.
This relands https://r.android.com/1205729.
Bug: 145934348
Test: m
Change-Id: I039d6e204bee5ddc16d8e2d85057fbec20e326fe
Framework and other dex files are used without image.
Test: taimen-userdebug boots when built with
DEXPREOPT_USE_ART_IMAGE=true
Test: Check logcat for checksum verification failures.
(Build ART with extra logging in OatFileAssistant.)
Test: Check that bootclasspath-checksums from some prebuilt
oat files (say input.odex) contain only one image
checksum followed by dex file checksums with
grep -az -A1 -E '^bootclasspath-checksums$' <oat-file> | \
xargs -0 echo | gawk '{print $2}'
Bug: 119800099
Change-Id: I65c2f247656e41f2c37df1ecb9e06af7dabab76e
Revert submission 1211982-dex2oat-soong-dep
Reason for revert: Build failures. See b/148312086.
Reverted Changes:
Ibc427a9a8: Make dex2oat(d) visible for use as implicit dexpre...
I71df11c1e: Move the Once cache for dexpreopt.GlobalConfig int...
I38317f2d5: Get the dex2oat host tool path from module depende...
I440a09dba: Separate dexpreopt.GlobalSoongConfig to allow inde...
Bug: 148312086
Bug: 145934348
Exempt-From-Owner-Approval: Plain revert
Change-Id: Ice3990225635a737e49e9aed7373f06516fccea3
Revert submission 1211982-dex2oat-soong-dep
Reason for revert: Build failures. See b/148312086.
Reverted Changes:
Ibc427a9a8: Make dex2oat(d) visible for use as implicit dexpre...
I71df11c1e: Move the Once cache for dexpreopt.GlobalConfig int...
I38317f2d5: Get the dex2oat host tool path from module depende...
I440a09dba: Separate dexpreopt.GlobalSoongConfig to allow inde...
Bug: 148312086
Bug: 145934348
Exempt-From-Owner-Approval: Plain revert
Change-Id: I6b656afb5feaad70d958b9d38b6c6eab7b03fba1
Revert submission 1211982-dex2oat-soong-dep
Reason for revert: Build failures. See b/148312086.
Reverted Changes:
Ibc427a9a8: Make dex2oat(d) visible for use as implicit dexpre...
I71df11c1e: Move the Once cache for dexpreopt.GlobalConfig int...
I38317f2d5: Get the dex2oat host tool path from module depende...
I440a09dba: Separate dexpreopt.GlobalSoongConfig to allow inde...
Bug: 148312086
Bug: 145934348
Exempt-From-Owner-Approval: Plain revert
Change-Id: I0cf6fef04434fd3c0b7957ee9abdcaabeb20b9d8
module.
This uses the Once cache for GlobalSoongConfig to propagate the dex2oat
path from a module dependency to the singletons (both the one that
writes out dexpreopt_soong.config and the one that creates the
dexpreopted boot images). Unless dexpreopting is disabled altogether
through DisablePreopt in dexpreopt.config, that means:
- We must ensure at least one module registers a dex2oat tool
dependency and resolves a GlobalSoongConfig using it, or else the
singletons fail. That means we litter dex2oat dependencies in java
modules even when they won't get dexpreopted and hence don't really
need them.
- We still assume there's at least one java_library or android_app in
the build.
Bug: 145934348
Test: m
(check that out/soong/dexpreopt_soong.config points to dex2oatd64)
Test: env USE_DEX2OAT_DEBUG=false m
(check that out/soong/dexpreopt_soong.config points to dex2oat)
Test: env OUT_DIR=out-tools prebuilts/build-tools/build-prebuilts.sh
on the aosp-build-tools branch
Change-Id: I38317f2d5e5cf6822f87daae25745767c8a3656c
package.
Preparation for a future CL that will need to get the make-written
GlobalConfig from within dexpreopt.
Also rename the Load*Config functions to Parse*Config, since they don't
actually load the config files anymore.
Bug: 145934348
Test: m
Change-Id: I71df11c1e042ca8135d273a7263e9539ea3cd68f
it.
Introduce a Once cache for GlobalSoongConfig to allow it to get binary
tool paths from ordinary module dependencies (coming in a future CL)
that are then reused in singletons.
Bug: 145934348
Test: m
Change-Id: I440a09dba7d337965a196527566b0966a18e3653
This relands I12a0f907753fefd1997ab8b4ea2ac331234093cf along with
a fix to blueprint for absolute paths.
Store the current working directory and then change to the root
directory so that all file accesses must go through helpers in
the android package that properly track dependencies.
Change-Id: I24ac485677aa102eec1a2521d16820da6ee1ae77
Fixes: 146437378
Test: m checkbuild
Test: m OUT_DIR=/tmp/out nothing
Store the current working directory and then change to the root
directory so that all file accesses must go through helpers in
the android package that properly track dependencies.
Fixes: 146437378
Test: m checkbuild
Change-Id: I12a0f907753fefd1997ab8b4ea2ac331234093cf
The dexpreopt global config is now split into the part that is generated
from make (in build/make/core/dex_preopt_config.mk) and the part that is
generated from Soong. Since the goal is to generate the dex2oat path from
Soong dependencies, the old GlobalConfig.Tools struct is simply repurposed
for the Soong generated config, although the intention is to allow more
settings to migrate from make to Soong, and hence from GlobalConfig to
GlobalSoongConfig.
Since the new dexpreopt_soong.config is written from a Soong-created ninja
rule, it doesn't need to be rewritten to out/soong/<device>/ like the old
make-created config file.
Test: m
Test: env USE_DEX2OAT_DEBUG=false m
(check that out/soong/dexpreopt_soong.config points to dex2oat instead of dex2oatd)
Bug: 145934348
Change-Id: Ifd45c4a08e2ec55b86f4a93f0d85bd39cf2cf189
This patch splits the system boot image in two parts:
- The ART boot image. This is the primary boot image that is
included in the ART apex and contains dexpreopted Core Libraries.
- The framweork boot image extension. It depends on the ART boot
image and contains framework libraries.
The third "apex" boot image (used in the JIT-zygote experiment)
remains unchanged; it is a monolithic primary boot image that
contains both libcore and framework libraries.
Dexpreopting of APKs now uses the framework boot image extension
(which in turn pulls in the ART boot image as a dependency).
Test: m
Test: phone boots:
lunch aosp_walleye-userdebug && m \
&& adb reboot bootloader && fastboot flashall -w
Bug: b/119800099
Exempt-From-Owner-Approval: rebased after getting approval.
Change-Id: Ida40dfae8c83bf7c2e737d5c7ea418e1197ad826
Jeongik Cha