Commit Graph

16841 Commits

Author SHA1 Message Date
Paul Duffin c3c5d5e351 Improve testing of sdk snapshot generation
Adds TestHelper to provide general test helper functionality for use by
any test.

Adds testSdkResult, composed with TestHelper that encapsulates the
result of processing and sdk {..} definition and provides specialized
support for testing the build rules.

Dedups the analysis of the sdk build rules, and improves it to extract
more information, and in different forms. That is represented by the
snapshotBuildInfo struct.

Adds a CheckSnapshot() method which checks the snapshot for a specified
sdk version. It takes a list of functions that can each perform a check
on a specific facet of the supplied snapshotBuildInfo.

Methods are provided for tests to use to check the following facets:
* Generated Android.bp contents.
* Copy rules
* Merge zip inputs

This approach makes it possible for each test to customize what is
being checked without either duplicating functionality, causing a
proliferation of specialized forms of the CheckSnapshot method for
different types of tests or adding arguments for every possible check
that any test would need which would lead to lots of churn to existing
tests when new arguments are added.

The main testing improvement is for CheckSnapshot() to actually try and
load the Android.bp that is generated. In order to do that it was
necessary to create a mock filesystem populated with information from
the build rules, i.e. the destination files from every Cp command as
well as the destination directory from every repackage zip command.

That helps detect a number of sources of errors:
* Failing to copy a file/directory that is mentioned in the generated
  Android.bp file.
* Invalid properties.
* Invalid format of the .bp file.
* Integrity issues within the .bp file.

Bug: 143678475
Test: m conscrypt-module-sdk
Change-Id: I4d3fe18f86698186d18e7e8b32d2e319183f7f0c
2019-12-06 12:16:59 +00:00
Paul Duffin 82d90438be Separate sdk testing infrastructure from sdk tests
Bug: 143678475
Test: m conscrypt-module-sdk
Change-Id: Ib00870ddefc4c1dd9e42ca594e6ebe8e24c42e05
2019-12-06 12:16:59 +00:00
Paul Duffin 1387957727 Parameterize the sdk member processing
Extracts the type specific functionality into the SdkMemberType
interface which has to be implemented by each module type that can
be added as a member of the sdk. It provides functionality to add
the required dependencies for the module type, check to see if a
resolved module is the correct instance and build the snapshot.

The latter was previously part of SdkAware but was moved because
it has to be able to process multiple SdkAware variants so delegating
it to a single instance did not make sense.

The custom code for handling each member type specific property,
e.g. java_libs, has been replaced with common code that processes
a list of sdkMemberListProperty struct which associates the
property (name and getter) with the SdkMemberType and a special
DependencyTag which is passed to the SdkMemberType when it has to add
dependencies.

The DependencyTag contains a reference to the appropriate
sdkMemberListProperty which allows the resolved dependencies to be
grouped by type.

Previously, the dependency collection methods would ignore a module if
it was an unsupported type because they did not have a way of
determining which property it was initially listed in. That meant it
was possible to add say a droidstubs module to the java_libs property
(and because they had the same variants) it would work as if it was
added to the stubs_sources property. Or alternatively, a module of an
unsupported type could be added to any property and it would just be
ignored.

However, the DependencyTag provides information about which property
a resolved module was referenced in and so it can detect when the
resolved module is of the wrong type and report an error. That check
identified a bug in one of the tests where the sdk referenced a
java_import module (which is not allowed in an sdk) instead of a
java_library module (which is allowed). That test was fixed as part
of this.

A list of sdkMemberListProperty structs defines the member properties
supported by the sdk and are processed in order to ensure consistent
behaviour.

The resolved dependencies are grouped by type and each group is then
processed in defined order. Within each type dependencies are grouped
by name and encapsulated behind an SdkMember interface which includes
the name and the list of variants.

The Droidstubs and java.Library types can only support one variant and
will fail if given more.

The processing for the native_shared_libs property has been moved into
the cc/library.go file so the sdk package code should now have no type
specific information in it apart from what is if the list of
sdkMemberListProperty structs.

Bug: 143678475
Test: m conscrypt-module-sdk
Change-Id: I10203594d33dbf53441f655aff124f9ab3538d87
2019-12-06 12:16:59 +00:00
Paul Duffin 375058f67d Use static build rules in snapshot generation
It is easier to extract information out of static build rules than it
is out of custom build rules built using the builder as the former
provides access to the in/out and args separate from the rule whereas
the latter only provides access to in/out.

Also, cleans up some warnings that appear in Intellij.

There is a lot of duplication in the testing code. That will be
resolved in a follow up change.

Bug: 143678475
Test: m conscrypt-module-sdk
Change-Id: I973bc0c90b0affd84487f1b222dd3e6c22c07ec0
2019-12-06 12:16:53 +00:00
Treehugger Robot cea93afe76 Merge "apex_available is defaultable" 2019-12-06 04:23:37 +00:00
Ivan Lozano 1fdb0b55a5 Merge "Exclude dirs from binder interface whitelisting" 2019-12-06 02:25:10 +00:00
Treehugger Robot 337cd8e493 Merge "Fix: vendor and odm are restricted to use public platform properties" 2019-12-06 02:00:11 +00:00
Treehugger Robot 6cfd3f0319 Merge "Add CUJ tests" 2019-12-06 01:17:04 +00:00
Calin Juravle af7c2fcb71 Merge "Export art dexpreopt image zip file" 2019-12-05 22:11:28 +00:00
Treehugger Robot 5eb62bc2fe Merge "Fix cc_fuzz ignoring soong namespaces." 2019-12-05 21:41:41 +00:00
Nick Desaulniers ac0336d81f Merge "soong: upgrade Android platform to clang-r370808" 2019-12-05 20:12:08 +00:00
Colin Cross d0be2109cf Add CUJ tests
Add a cuj_tests binary that runs through a sequence of critical
user journey builds and collects metrics.

Bug: 145688720
Test: cuj_tests
Change-Id: I6bb1a3fc4a458fd8cea7b7c924f565d0cf3e6df3
2019-12-05 11:11:37 -08:00
Mitch Phillips 6a9bf214f9 Fix cc_fuzz ignoring soong namespaces.
cc_fuzz ignored soong namespaces, and built all cc_fuzz modules into the
architecture package. This caused naming conflicts where two modules had
the same name but were in different namespaces.

Now, cc_fuzz will only build modules that are in the same namespace as
the product definition - resolving the duplicate definitions error.

Bug: 145240569
Test: m haiku
Change-Id: I0a0acd139426301586ac5e3aba8ce5527cb7a1b5
2019-12-05 09:38:56 -08:00
Treehugger Robot bed70f837b Merge "Test par file argument handling" 2019-12-05 16:23:56 +00:00
Treehugger Robot dc0e8fd131 Merge changes Id261b5a3,Ice6f34d0
* changes:
  Fix errorProtoLog error messages
  Write BUILD_DATETIME_FILE after SetupOutDir
2019-12-05 09:10:13 +00:00
Treehugger Robot 5f9e5244c5 Merge "Remove passing -forceprocessing flag to R8" 2019-12-05 08:56:38 +00:00
Jiyong Park b326ac2b65 Fix: vendor and odm are restricted to use public platform properties
There was a mistake in the condition.

Bug: N/A
Test: N/A

Change-Id: I07f79fdcc3c946a4b9e957e5a42b6694b5eeb251
2019-12-05 13:35:35 +09:00
Colin Cross 31bf00dac3 Export art dexpreopt image zip file
Test: m  out/target/product/generic_arm64/boot.zip
Bug: 144491400
Change-Id: I84342703ac614e072b0cde8c6f5597dc8cffbc65
2019-12-04 16:50:52 -08:00
Treehugger Robot 7b34ebf447 Merge "Remove implicit outputs for the flattened APEX Make module" 2019-12-05 00:21:29 +00:00
Ivan Lozano d094d4006f Exclude dirs from binder interface whitelisting
Exclude vendor/ and hardware/ from the manually written binder interface
whitelist.

Bug: 136279235
Test: interfaces in excluded directories are not checked against whitelist.
Change-Id: I0d640e23489b37d0c0787d5fca6bcdab10034109
2019-12-04 09:10:21 -08:00
Paul Duffin 46fc58ee8d Disable TestHostSnapshot test for non-linux build
Building sdk snapshots on anything other than linux glibc is not
supported at the moment.

Bug: 145598135
Test: m nothing
Change-Id: I34155252545e9f37c5361ed94a27a87024fcece2
2019-12-04 10:15:40 +00:00
Dan Willemsen 67d4258786 Test par file argument handling
This had been broken with my python3 launcher.

Test: build/soong/python/tests/runtest.sh
Change-Id: I0524e5a80244bf1080cb28237b23ef0d023e03dc
2019-12-03 23:54:04 -08:00
Jiyong Park 1753e58df7 Remove implicit outputs for the flattened APEX Make module
When flattened, the Make module for an APEX is a phony package for the
files in it. There is no output (either implcit or explicit) there.

Bug: 144338929
Test: m
Test: check the generated Android-<target>.mk file
Change-Id: I86df3b69f402a7e2d94bafa39f2aad3312d8c28b
2019-12-04 13:38:56 +09:00
Treehugger Robot e06c84e103 Merge "Merge Android10 QPR1 into AOSP master" 2019-12-03 23:15:05 +00:00
Treehugger Robot dc9369ef7f Merge "Generate tradefed config for rust device tests." 2019-12-03 21:19:12 +00:00
Ulya Trafimovich 4d2eeed0da Use boot image extension for framework libraries.
This patch splits the system boot image in two parts:

  - The ART boot image. This is the primary boot image that is
    included in the ART apex and contains dexpreopted Core Libraries.

  - The framweork boot image extension. It depends on the ART boot
    image and contains framework libraries.

The third "apex" boot image (used in the JIT-zygote experiment)
remains unchanged; it is a monolithic primary boot image that
contains both libcore and framework libraries.

Dexpreopting of APKs now uses the framework boot image extension
(which in turn pulls in the ART boot image as a dependency).

Test: m
Test: phone boots:
    lunch aosp_walleye-userdebug && m \
        && adb reboot bootloader && fastboot flashall -w

Bug: b/119800099

Exempt-From-Owner-Approval: rebased after getting approval.

Change-Id: Ida40dfae8c83bf7c2e737d5c7ea418e1197ad826
2019-12-03 13:59:25 +00:00
Paul Duffin 3ae3b17083 Merge "Add support for host sdk" 2019-12-03 10:08:05 +00:00
Paul Duffin e63106a08c Merge "Add model to represent generated snapshot .bp file" 2019-12-03 10:07:10 +00:00
Xin Li 1a898a1384 Merge Android10 QPR1 into AOSP master
Bug: 145570283
Change-Id: Id15b02a54a94d8ab201d9971ae38c2b73f52487d
2019-12-02 21:25:01 -08:00
Treehugger Robot f6739a65dd Merge "Link type will be check in android_library also" 2019-12-03 03:41:21 +00:00
Evgenii Stepanov 6d5f9a965d Merge "Add -instcombine-lower-dbg-declare to hwasan flags."
am: e643cd2b61

Change-Id: I341f9769410bc4f86135fa8e2f12ecba4e01f15d
2019-12-02 18:22:38 -08:00
Treehugger Robot e643cd2b61 Merge "Add -instcombine-lower-dbg-declare to hwasan flags." 2019-12-03 02:19:08 +00:00
Jiyong Park e9f6b5916b Merge "Refactor the routine for creating apexFile"
am: d7b4749cf0

Change-Id: Icb3386849156203cf0f1734ffd7ff672e240821f
2019-12-02 17:17:45 -08:00
Treehugger Robot d7b4749cf0 Merge "Refactor the routine for creating apexFile" 2019-12-03 01:08:38 +00:00
Roshan Pius 5c376a0c91 Merge "soong: Align update boot jar variable with system_server jar"
am: d6fb4842c9

Change-Id: If93859af6ca6a7b464de66b9278741ecc1bd279e
2019-12-02 16:47:50 -08:00
Roshan Pius d6fb4842c9 Merge "soong: Align update boot jar variable with system_server jar" 2019-12-03 00:43:17 +00:00
Chih-Hung Hsieh ede57ae812 Generate tradefed config for rust device tests.
* The RustBinaryTest runner and device test config template
  are derived from GTest. Device tests are pushed to /data/local/tmp.

Bug: 140938178
Test: make unicode-xid projects; run atest
Change-Id: Idb4cab7872b48c6e25bc59b85aa6d9dece4383ec
2019-12-02 17:44:53 +00:00
Roshan Pius ccc26ef316 soong: Align update boot jar variable with system_server jar
Use apex:jar pair values for updatable boot jars to align with
updatable system_server values.

Test: Compiles & flashed device. Ensured that the bootclasspath & system_server class
paths remain the same.

Change-Id: I1d6dfe3325d9091b7c724458be4e6471f9086666
2019-12-02 08:37:57 -08:00
Paul Duffin e44358fa8f Add support for host sdk
Adds HostSupported() method to ModuleBase for use by sdk. Adds
host_supported/device_supported to all prebuilt modules and the
sdk snapshot (where necessary).

Adds TestHostSnapshot to verify the behavior is correct.

Bug: 143678475
Test: m nothing
Change-Id: I8b4d097e46d5804f62cb8f651a83069299a3e639
2019-12-02 12:59:38 +00:00
Paul Duffin b645ec8e34 Add model to represent generated snapshot .bp file
Having each module type generate the contents of the snapshot's .bp
file results in lots of duplicated code. This adds an intermediate
model for use by the module types and then generates the .bp file
contents from that.

This not only removes the duplicated formatting code but it also
allows consistent handling of shared properties such as name further
reducing duplication. It also makes it possible to duplicate the
versioned and unversioned prebuilt modules from the same model.

Extracts generatedContents from generatedFile to allow the contents
to be populated without creating an output file, for testing.

Cleans up unused code.

Bug: 143678475
Test: m nothing
Change-Id: If21b84db0ef3fdfb5dc11ea0973ce6cb73603ea3
2019-12-02 12:59:38 +00:00
Jiyong Park 3814f4df2b apex_available is defaultable
Previously, it was defaultable only for cc_library. Now, it is
defaultable for all module types.

Bug: 128708192
Test: m
Change-Id: If248f593da6f4b724bf889ccd7f2e077d48069b5
2019-12-02 13:48:33 +09:00
Paul Duffin 5aa2f2df2a Remove unused methods from sdk/sdk_test.go
am: 01178ed7d1

Change-Id: I5cbe4682953c7426250cfacd8d67f88cbe121aa3
2019-12-01 02:04:01 -08:00
Jiyong Park f653b05d55 Refactor the routine for creating apexFile
This change is to make it easier to add new fields to the struct.
transitiveDep field is added to distinguish apexFiles coming from
transitive dependencies of the APEX. We will later use the info to
reduce the size of bundled APEXes by replacing the transitive deps with
symlinks to the corresponding files in the system partition outside of
the APEX.

Bug: 144533348
Test: m

Change-Id: I283859f2f2f1b5cfb3025569f168ba8569b22bb9
2019-11-30 14:49:13 +09:00
Paul Duffin 01178ed7d1 Remove unused methods from sdk/sdk_test.go
Bug: 143678475
Test: m conscrypt-module-sdk
Change-Id: Ic011fd63b226beb455793e2df08114992b858698
2019-11-29 20:35:06 +00:00
Paul Duffin 9a8ccde833 Simplify sdk_snapshot module member list generation
am: 66905ed6cf

Change-Id: I06f6f9143f3c4f04fb54e1ccd759d291d3f2ec8a
2019-11-29 01:42:15 -08:00
Paul Duffin 2140aa6ad2 Remove unused frozenVersions() method
am: 7264c69892

Change-Id: I5bb05e83c3b91943253d3b8379b99acb1950ba67
2019-11-29 00:17:57 -08:00
Paul Duffin 66905ed6cf Simplify sdk_snapshot module member list generation
When generating the list of members the names of the resolved
modules are used in the case of java_libs and stubs_source whereas
the name stored in the nativeLibInfo is used for native_shared_libs.

Those names are identical to the names provided in the corresponding
sdk property so this switches to use those instead as it is more
consistent and simplifies future refactorings.

Bug: 143678475
Test: m nothing
Change-Id: I9c08a8175fa9a37bb0cd89a361ef4c0f613b0dab
2019-11-28 21:02:12 +00:00
Paul Duffin 7264c69892 Remove unused frozenVersions() method
Bug: 143678475
Test: m nothing
Change-Id: Ia5d417cfccfcd90b02a95b5b806b211b97e340db
2019-11-28 21:02:12 +00:00
Paul Duffin 8afdd32b7f Merge "Dedup code for generating prebuilt_stubs_sources in snapshot"
am: bf857716a1

Change-Id: I2d563f359295a940e4b90688618cf3c1d34d812f
2019-11-27 08:26:54 -08:00
Treehugger Robot bf857716a1 Merge "Dedup code for generating prebuilt_stubs_sources in snapshot" 2019-11-27 16:21:41 +00:00