This change also added support for excluding properties from common
value extraction by using a struct tag of `sdk:"keep"` That was needed
to prevent the fields in SdkMemberPropertiesBase from having their
values cleared.
The purpose of this change is to make it easier to share functionality
across sdk member types.
Bug: 142935992
Test: m nothing
Change-Id: Ie5160a8f854056920e411801ca20721eab7c8578
This change pulled the common value extraction functionality out into
its own structure that can be used to extract common values from
multiple sets of properties.
Precursor to follow up changes that will allow properties to contain
embedded structures making it easier to share functionality across
sdk member types.
Bug: 142935992
Test: m nothing
Change-Id: Ic902ed61e40ced7a2c2fa4b5f793f532c6fc0034
Makes sure that the module snapshots do not rely on the white list
of apex available settings so that when those lists are removed it is
not necessary to update any snapshots.
Bug: 142935992
Test: m nothing
Change-Id: Iedcff7dfc2646a4da77258d16e06657dd2f411f9
The sdk snapshot creates two prebuilts for each member one that is
versioned and one that is not. If they are both installed then they
lead to duplicate rules in make for creating the same installed file.
This change adds an installable property to cc modules that will
prevent the installation of the file and then adds installable: false
on the versioned prebuilt for cc modules.
Bug: 142935992
Test: m nothing
Change-Id: I4cb294c2b0c8a3f411eea569775835d9e41726d6
Adds support for specifying separate members to an sdk/module_exports
for different os types, e.g. separate ones for android and host.
Adds 'android:"arch_variant"' tag to the dynamically generated fields
for the sdk member types.
Merges the exported members from all variants together.
Determines the device/host_supported flags of the member snapshots by
whether the OsClasses used by their variants rather than the sdk's
host/device supported properties as they may be different.
Bug: 150451422
Test: m nothing
Change-Id: I41fbbcd8723aafd54826aad9b78eced9f66ef51c
Updates the member snapshot creation code to support multiple os types.
It basically sorts the variants by os type, then applies the code to
optimize the arch properties and then it optimizes the properties that
are common across architectures and extracts any properties that are
common across os types.
The java and cc member types needed to be modified to make the location
of the generated files within the snapshot os type dependent when there
is more than one os type. That was done by adding an OsPrefix() method
to the SdkMemberPropertiesBase which returns the os prefix to use when
there is > 1 os type and otherwise returns an empty string.
Added three tests, one for cc shared libraries, one for cc binary and
one for java header libraries.
Bug: 150451422
Test: m nothing
Change-Id: I08f5fbdd7852b06c9a9a2f1cfdc364338a3d5bac
Adds support for handling the common arch to the sdk module type
snapshot generation code and then refactors the java_library
and java_test snapshot processing to take advantage of that.
Bug: 150451422
Test: m nothing
Change-Id: If746f375f1c4288ab6b67c7d216593b70258b043
Generalize the processing of arch specific properties to reduce
duplication in snapshot module creation and simplify addition of
support for handling multiple os types.
Supporting multiple os types with the current method for building
snapshot modules would require every affected module type to add
support for it. Rather than duplicate multiple os type handling code
across those module types this work generalizes the process cc modules
use for handling arch types as it can be used as a basis for handling
multiple os types. Migrating module types over to this new process
will insulate them from having to handle multiple os types.
OB
SdkMemberType changes:
* BuildSnapshot is deprecated in favour of the new AddPrebuiltModule()
method.
* Additional methods, CreateVariantPropertiesStruct() and
FinalizeModule() are added.
* A new interface SdkMemberProperties, is defined that handles
extracting information from the variant (prior to common value
optimization) and adding properties to a property set.
The sdk module type uses these new methods and types to delegate the
member type specific processing to the relevant member types while
handling the behavior that is common across all members types, e.g.
extracting common values across multiple architectures. A future change
will leverage this processing to add support for multiple os types.
This change also refactors the cc module processing to use the new
process.
Bug: 150451422
Test: m nothing
Change-Id: If6ab2498407b17f50391d062cd9afc01b5e01af4
Adds a CommonOS variant for sdk that depends on the os specific
variants and is used to generate a single sdk for multiple OsTypes,
e.g. host linux and android.
At the minute the member types only support a single OsType but the
basic mechanism for managing the CommonOS variant and collating the
variants across all of them is there.
The only visible effect of this change is that the location of the
generated snapshot is changed, it is no longer os specific and instead
is in the same location irrespective of which os it is built for.
A lot of tests needed to be changed to specify "common_os" as the
variant type instead of the specific os type. As that is the same across
all tests it is hard coded in CheckSnapshot method.
Test: m nothing
Bug: 150451422
Change-Id: If36be39b06d6910453649f7c288c2d34f688b2f4
Ensures that the generated prebuilt modules do not contain empty
property sets.
Bug: 148933848
Test: m nothing
Change-Id: I99d5da1c4e3a72836e81e021ef043df53888e66b
This is needed to allow the art-host-module-exports to restrict itself
to just managing the linux 64 bit version of the host tools as that is
the only variant that is currently supported by all host tools. This
greatly simplifies that process and allows us to make progress on the
unbundling.
Bug: 142935992
Test: m nothing
Change-Id: I62d016d97c2df73e5feecf912638f477fedd97c9
Internal sdk members are used by an sdk member but not exported by the
sdk. The exported sdk members are those listed explicitly in one of the
sdk member list properties, e.g. java_header_libs.
The prebuilts of an internal sdk member use a unique name so that they
do not clash with the source module. The use of the module internally
is an implementation detail that must not have any effect outside the
snapshot. Having the same name as the source module could cause it to
override the source module, hence why it needs a unique name.
Similarly, they are marked as private so as to prevent their accidental
use from outside the snapshot.
Bug: 142940300
Test: m nothing
Change-Id: Id5364b410be0592f65666afb3e40e9d3f020251c
Adds an SdkMemberType implementation for java_system_modules. It
specifies that java_system_modules can be used with sdk as well as
module_exports, and also that the libs property should be included
as transitive members in the sdk.
It also adds support for treating appropriate tagged properties in
the snapshot prebuilts module as references to sdk members so that
they are correctly transformed when creating the versioned modules.
Bug: 142940300
Test: m nothing
Change-Id: Ic10b5a6d5b92b6018334fe876f06feaf79cc55e9
Allow an sdk member type to treat some of its dependencies as being
members of the sdk.
Needed for the java_system_modules type whose libs property are an
implementation detail of the system module and so should not be
explicitly listed in the sdk module but still have to be included in
the sdk snapshot.
Bug: 142940300
Test: m nothing
Change-Id: I90f37dae269ef64a6fe9debd0bbaf29a64dd74d8
Moves the struct to android/sdk.go and abstracts it behind a factory
method and interface. That allows it to be used outside the sdk
package.
This change is in preparation for adding support for module types that
have transitive sdk members.
Bug: 142940300
Test: m nothing
Change-Id: I71e5e0adf839b28a3a0952f82637637887f02688
This is preparation for enhancing the versioning tranformer to support
applying per property transformations. Specifically, to allow
properties to reference other libraries within the sdk.
Bug: 142940300
Test: m nothing
Change-Id: I99cdff4b407763ed395ff358d8110a63c6cf5589
Moves the code for transforming an unversioned module into a
versioned module into a transformer.
This is preparation for enhancing the versioning tranformer to support
applying per property transformations. Specifically, to allow
properties to reference other libraries within the sdk.
Bug: 142940300
Test: m nothing
Change-Id: Ia6b5cff8e1b5cd7232e031769d9fc0019e815fcb
Previously, only a shallow copy was made so the copy ends up sharing
some contents with the original. That was a potential source of bugs
as the copy was being made in order to be mutated.
This change switches to a deep copy; renaming the methods from
copy -> deepCopy to clarify the intent.
Makes the bpPropertySet member of bpModule a *bpPropertySet to avoid
unnecessary copying of bpPropertySet.
Bug: 142940300
Test: m nothing
Change-Id: I3f2eaa9fffab4e61d5a7cec81aa42fee9fdfec44
Previously, adding a new SdkMemberType would require adding a new
sdkMemberListProperty instance to the sdkMemberListProperties as well
as adding a new property into the sdkProperties struct. They are
potential sources of conflict and couple the sdk code with all the
packages that add members to it. This change switched to a
registration model that allows each package to register its sdk
member types decoupling them from the sdk code.
Adds an SdkPropertyName() method to SdkMemberType that specifies the
name of the property to use in the sdk/sdk_snapshot. Also provides
an SdkMemberTypeBase struct to be used by providers of SdkMemberType
implementations.
SdkMemberType instances are registered using the
RegisterSdkMemberType() func which sorts the registered instances
by their SdkPropertyName() to ensure the behavior is consistent and
not affected by order of registration.
When creating a new sdk module a dynamicSdkMemberTypes instance is
created that contains the following:
* A properties struct is created dynamically that contains a field for
each registered SdkMemberType, corresponding to that type's
SdkPropertyName().
* A list of sdkMemberListProperty instances is also created, one for
each registered SdkMemberType.
The dynamicSdkMemberTypes instance is cached using a key that uniquely
identifies the set of registered types just in case new types are
registered after one has been created, e.g. by tests.
Bug: 142918168
Test: m checkbuild
Change-Id: I4bf2bf56a2a49025aa41454048bc1e8ccc6baca2
Header include directories are copied into the snapshot separately for
each cc library that exports them. However, the same include
directories can be exported by multiple libraries which caused ninja
error because two separate rules (albeit identical) were defined to
create the same files.
This avoids the duplicate ninja rules by detecting and discarding
duplicate copies, i.e. where the source and destination are the same.
It will also report an error if two or more different source files are
copied to the same destination.
Bug: 142918168
Test: m nothing
added test and verified it produced two identical copy rules
fixed code and verified duplicate copy rules had been eliminated
Change-Id: I39e37405035bee5093f96e03248e9e29ed30962c
Prevents the META-INF/MANIFEST.MF file from the source jar from being
copied to the snapshot zip file.
Bug: 143678475
Test: m conscrypt-module-sdk
manually check contents of generated zip
Change-Id: I6eca1435dfc25b562e49de46b049fa81cf8daf90
Exports visibility and package mutator registration functions so they
can be used in sdk testing. Updates sdk test to support visibility and
package modules.
Adds EffectiveVisibility(...)[]string function to make the effective
visibility rules available to sdk snapshot creation.
Extracts compositeRule.Strings() []string from compositeRule.String()
method so that it can be used by above func.
Adds visibility property to sdk snapshot and prebuilt modules along
with a test to ensure it works properly.
Adds dir parameter to CheckSnapshot so that it can check the snapshot
generated for a non-root package. That is required in order to ensure
that visibility of :__subpackages__ on a source module in package
<pkg> is resolved to an effective visibility of
//<pkg>:__subpackages__ on its corresponding prebuilt.
Test: m conscrypt-module-sdk
Bug: 143678475
Change-Id: Icaacac5b9c04726d28e6fec93e49715ac45df7f4
This is needed for a follow up change that makes sure that the
prebuilt modules have the same visibility as the source modules.
Bug: 143678475
Test: m conscrypt-module-sdk
Change-Id: I9461c8c094ab19ee9ececb5e5fd50565789f2fa2
Adds TestHelper to provide general test helper functionality for use by
any test.
Adds testSdkResult, composed with TestHelper that encapsulates the
result of processing and sdk {..} definition and provides specialized
support for testing the build rules.
Dedups the analysis of the sdk build rules, and improves it to extract
more information, and in different forms. That is represented by the
snapshotBuildInfo struct.
Adds a CheckSnapshot() method which checks the snapshot for a specified
sdk version. It takes a list of functions that can each perform a check
on a specific facet of the supplied snapshotBuildInfo.
Methods are provided for tests to use to check the following facets:
* Generated Android.bp contents.
* Copy rules
* Merge zip inputs
This approach makes it possible for each test to customize what is
being checked without either duplicating functionality, causing a
proliferation of specialized forms of the CheckSnapshot method for
different types of tests or adding arguments for every possible check
that any test would need which would lead to lots of churn to existing
tests when new arguments are added.
The main testing improvement is for CheckSnapshot() to actually try and
load the Android.bp that is generated. In order to do that it was
necessary to create a mock filesystem populated with information from
the build rules, i.e. the destination files from every Cp command as
well as the destination directory from every repackage zip command.
That helps detect a number of sources of errors:
* Failing to copy a file/directory that is mentioned in the generated
Android.bp file.
* Invalid properties.
* Invalid format of the .bp file.
* Integrity issues within the .bp file.
Bug: 143678475
Test: m conscrypt-module-sdk
Change-Id: I4d3fe18f86698186d18e7e8b32d2e319183f7f0c
Extracts the type specific functionality into the SdkMemberType
interface which has to be implemented by each module type that can
be added as a member of the sdk. It provides functionality to add
the required dependencies for the module type, check to see if a
resolved module is the correct instance and build the snapshot.
The latter was previously part of SdkAware but was moved because
it has to be able to process multiple SdkAware variants so delegating
it to a single instance did not make sense.
The custom code for handling each member type specific property,
e.g. java_libs, has been replaced with common code that processes
a list of sdkMemberListProperty struct which associates the
property (name and getter) with the SdkMemberType and a special
DependencyTag which is passed to the SdkMemberType when it has to add
dependencies.
The DependencyTag contains a reference to the appropriate
sdkMemberListProperty which allows the resolved dependencies to be
grouped by type.
Previously, the dependency collection methods would ignore a module if
it was an unsupported type because they did not have a way of
determining which property it was initially listed in. That meant it
was possible to add say a droidstubs module to the java_libs property
(and because they had the same variants) it would work as if it was
added to the stubs_sources property. Or alternatively, a module of an
unsupported type could be added to any property and it would just be
ignored.
However, the DependencyTag provides information about which property
a resolved module was referenced in and so it can detect when the
resolved module is of the wrong type and report an error. That check
identified a bug in one of the tests where the sdk referenced a
java_import module (which is not allowed in an sdk) instead of a
java_library module (which is allowed). That test was fixed as part
of this.
A list of sdkMemberListProperty structs defines the member properties
supported by the sdk and are processed in order to ensure consistent
behaviour.
The resolved dependencies are grouped by type and each group is then
processed in defined order. Within each type dependencies are grouped
by name and encapsulated behind an SdkMember interface which includes
the name and the list of variants.
The Droidstubs and java.Library types can only support one variant and
will fail if given more.
The processing for the native_shared_libs property has been moved into
the cc/library.go file so the sdk package code should now have no type
specific information in it apart from what is if the list of
sdkMemberListProperty structs.
Bug: 143678475
Test: m conscrypt-module-sdk
Change-Id: I10203594d33dbf53441f655aff124f9ab3538d87
It is easier to extract information out of static build rules than it
is out of custom build rules built using the builder as the former
provides access to the in/out and args separate from the rule whereas
the latter only provides access to in/out.
Also, cleans up some warnings that appear in Intellij.
There is a lot of duplication in the testing code. That will be
resolved in a follow up change.
Bug: 143678475
Test: m conscrypt-module-sdk
Change-Id: I973bc0c90b0affd84487f1b222dd3e6c22c07ec0
Adds HostSupported() method to ModuleBase for use by sdk. Adds
host_supported/device_supported to all prebuilt modules and the
sdk snapshot (where necessary).
Adds TestHostSnapshot to verify the behavior is correct.
Bug: 143678475
Test: m nothing
Change-Id: I8b4d097e46d5804f62cb8f651a83069299a3e639
Having each module type generate the contents of the snapshot's .bp
file results in lots of duplicated code. This adds an intermediate
model for use by the module types and then generates the .bp file
contents from that.
This not only removes the duplicated formatting code but it also
allows consistent handling of shared properties such as name further
reducing duplication. It also makes it possible to duplicate the
versioned and unversioned prebuilt modules from the same model.
Extracts generatedContents from generatedFile to allow the contents
to be populated without creating an output file, for testing.
Cleans up unused code.
Bug: 143678475
Test: m nothing
Change-Id: If21b84db0ef3fdfb5dc11ea0973ce6cb73603ea3
When generating the list of members the names of the resolved
modules are used in the case of java_libs and stubs_source whereas
the name stored in the nativeLibInfo is used for native_shared_libs.
Those names are identical to the names provided in the corresponding
sdk property so this switches to use those instead as it is more
consistent and simplifies future refactorings.
Bug: 143678475
Test: m nothing
Change-Id: I9c08a8175fa9a37bb0cd89a361ef4c0f613b0dab
Without this the generated snapshot build file will not load.
Bug: 143678475
Test: m nothing and check output for dummy sdk
Change-Id: I252ed01280aeceb4ff40b2d9de2bc2bf88858ea9
Adds stubs_sources property to sdk and unzips the droidstubs srcjar
into the snapshot directory.
Adds an UnzipToSnapshot method to the SnapshotBuilder which creates
a rule that uses zip2zip to repackage the supplied zip content into a
temporary zip file that matches what the required snapshot structure.
e.g. if the supplied zip contains foo/Foo.java and that needs to be in
the snapshot directory java/foo/stubs then it will create a zip that
contains java/foo/stubs/foo/Foo.java.
The temporary zip that is the output of that rule is added to the
zipsToMerge field for merging later.
If the zipsToMerge is empty then the snapshot zip is created as
before. Otherwise, a temporary zip file is created. That is then
merged with the other zip files in zipsToMerge to create the final
snapshot zip.
Adds prebuilt_stubs_sources for use by the generated .bp module.
Bug: 143678475
Test: added conscrypt sdk module and attempted to build it
Change-Id: Ie274263af3a08e36a73c61c0dbf0c341fd6967e2
Creates a SnapshotBuilder and GeneratedSnapshotFile interfaces to allow
the java library snapshot work to be moved into the java package.
Test: m -j60 checkbuild
Change-Id: I857167616026149d5e85885621b53876b419ba9b
`m module_sdk dist` produces snapshots of all SDKs in the source tree.
A snapshot is a zip file consists of Android.bp, exported headers,
exported AIDL files, stubs for native libs and jars. The zip file is
expected to be downloaded from the build server and extracted to a
directory (which probably will be
/prebuilts/module_sdks/<module_name>/current).
Bug: 138182343
Test: m (sdk_test.go updated)
Change-Id: Idbe4bc24795fe08f26fc1cf7497028f9d162053a
The snapshot script can now handle native shared libs in an SDK.
Bug: 138182343
Test: create following sdk module:
sdk {
name: "mysdk",
native_shared_libs: ["libc", "libdl"],
}
, then execute `m mysdk` and execute the update_prebuilt-1.sh as
prompted. Following directories are generated under the directory where
mysdk is defined at:
1
├── aidl
├── Android.bp
├── arm64
│ ├── include
│ ├── include_gen
│ └── lib
│ ├── libc.so
│ └── libdl.so
├── include
│ └── bionic
│ └── libc
│ └── include
│ ├── alloca.h
│ ├── android
│ │ ├── api-level.h
<omitted>
Change-Id: Ia1dcc5564c1cd17c6ccf441d06d5995af55db9ee
`m <sdk_name>` generates two scripts each of which is use to update the
current snapshot of the SDK and to freeze ToT as a new version,
respectively. Executing the scripts will copy necessary files (stub
libraries, AIDL files, etc.) along with Android.bp into the ./<apiver>
directory under the directory where the sdk is defined.
This change also introduces a new module type 'sdk_snapshot' that
represents a snapshot of an SDK. It will be auto-generated by the above
scripts, so developers are not expected to write this manually.
The module type 'sdk' is now used to simply specify the list of modules
that an SDK has.
Finally, this change changes the version separator from '#' to '@'
because '#' confuses Make.
Bug: 138182343
Test: m
Change-Id: Ifcbc3a39a2f6ad5b4f4b200ba55a1ce3281498cf
Paul Duffin