For very long processes, we might want to keep stdout and stderr
by default to None.
So no redirection will occur in the child process as explained in:
https://docs.python.org/2/library/subprocess.html
That will result in the child process stdin and stderr to be same
than in common.py and avoid to have the logs blocked during the
child process execution and flushed only when child process terminates.
Since the logs are continously displayed, it allows to easily confirm
that the process is not blocked.
Bug: 133380588
Test: generate iota & Check that the logs are not blocked.
Change-Id: I6d6cb56547bf3a4a4334dfa22b6b2b05d2c36a5e
Signed-off-by: Regnier, Philippe <philippe.regnier@intel.com>
While in theory this could incur memory overhead for Python 2, the
impact is low for the existing use cases (plus we're moving away from
Python 2).
Bug: 131631303
Test: No additional occurrence of iteritems.
Test: Build with Python 3.
Change-Id: I0205c9edf25f46e3d85967c7dd2c1af035757741
And a few minor clean-ups to the styling.
Bug: 131631303
Test: python -m unittest test_merge_target_files
Test: python3 -m unittest test_merge_target_files
Test: Use `python merge_target_files` to merge two target_files zips.
Test: Use `python3 merge_target_files` to merge two target_files zips.
Change-Id: I8502dfb243408f658d022e8d5e5fbb60066e4ff0
Split merge_target_files function into several steps, so we can increate
readability and add conditional flows for other *SSI mixed build with less effort
Test: m -j & atest passed
Change-Id: I558f9dd5bca31b132a09cb36d9dfcd30c92efbc9
This CL changes the condition for building super_empty.img from
PRODUCT_BUILD_SUPER_PARTITION to PRODUCT_USE_DYNAMIC_PARTITIONS, as a
follow-up to the change in [1].
With the CL in [1], it skips building super.img and super_empty.img both
when turning off PRODUCT_BUILD_SUPER_PARTITION. However, the latter
should be mandatory whenever dynamic partitions is enabled. Because
fastboot relies on this file to properly flash dynamic partitions. Plus,
the cost for building super_empty.img is much lower than the one for
super.img.
As part of the change, it'll write group info into target_files when
building with PRODUCT_BUILD_SUPER_PARTITION == false. It's the work for
target_files merging script to determine the values to be picked up. The
current logic in merge_target_files.py always uses the one from vendor
target_files. This CL adds a testcase to ensure the behavior.
[1] https://android-review.googlesource.com/c/platform/build/+/928756
Bug: 135752763
Test: `m dist` with a target that sets PRODUCT_BUILD_SUPER_PARTITION to
false. Check the built artifacts contain super_empty.img. Verify
that the build can be flashed properly.
Change-Id: I277f087eab45663a6c3b33333d16e9e576c1c25c
This allows a consistent logic in using the avbtool which could be
board-specific.
Test: `atest releasetools_test`
Test: Run sign_target_files_apks.py on a target_files.zip.
Change-Id: I8cd93b8e71146985734f85c31f4662f5e2e9534c
This ensures a matching interface between sign_apex and apex_utils.
The test apex `testdata/foo.apex` is generated by running
`system/apex/apexer/runtests.sh`.
Test: python -m unittest test_sign_apex
Test: atest releasetools_test
Change-Id: I7c14b1df2a3038ad206aa3e5aac084c47baaa00b
There are some places defining same file open function and use
common.LoadDictionaryFromLines. This commit creates
LoadDictionaryFromFile to reduce some code redundancy.
Test: m -j & atest passed
Change-Id: I6a3fa48693095937f8c79ce6f3c110b6862a1967
In order to get a Python 2 and 3 compatible re-raise behavior, this CL
removes the stack traceback for the lines within apex_util module (i.e.
sys.exc_info()[2]). It's not a big loss in practice, since we only have
one line within the try-except block (`common.RunAndCheckOutput()`)
that's no longer reported in the traceback.
Using `six` module could better solve this, but only after building
releasetools as python_binary_host modules where we can properly handle
the module dependency.
Bug: 131631303
Test: TreeHugger
Test: `python -m unittest test_apex_utils`
Test: `python3 -m unittest test_apex_utils`
Change-Id: I0c5a72ec9fad5ff9d8c9c94d29e813e433ec2921
sign_target_files_apks.py only needs to take care of the current
release. The legacy paths of ODM/build.prop, VENDOR/odm/build.prop, and
BOOT/RAMDISK/default.prop no longer exist in the target_files.zip from
master.
The other two cases of ROOT/default.prop and
RECOVERY/RAMDISK/default.prop are still kept in the code, as they will
still exist (as symlink or conditionally).
Test: Run sign_target_files_apks.py against
aosp_taimen-target_files.zip. Check the rewritten prop files.
Test: `python -m unittest test_sign_target_files_apks`
Change-Id: I5e70bc2ccc0f3dcf0eace0718c59a3b0f89a9ff4
Python 2 and 3 behave differently when calling ZipFile.writestr() with
zinfo.external_attr being 0. Python 3 uses `0o600 << 16` as the value
for such a case (since
18ee29d0b8),
which seems to make more sense. Otherwise the entry will end up with
0o000 as the permission bits. This CL updates common.ZipWriteStr to
follow the logic in Python 3, in order to get consistent behavior
between using the two versions.
Bug: 131631303
Test: `python -m unittest test_common.CommonZipTest`
Test: `python3 -m unittest test_common.CommonZipTest`
Change-Id: If8429855d922ef1ad76591f703215a0ce5089f0f
Previously it was using regular dict.
Test: python -m unittest test_common.DynamicPartitionsDifferenceTest
Change-Id: If108a4512aeaf9d3c8775c030cad6e44342b9d3d
Previously, setting PRODUCT_BUILD_SUPER_PARTITION to false for a partial
build (with PRODUCT_USE_DYNAMIC_PARTITIONS == true) would fail to
include necessary keys in misc_info.txt that are required when merging
two partial builds to create a dynamic-partition-enabled mixed build.
This change ensures these necessary keys are included even when
PRODUCT_BUILD_SUPER_PARTITION is false. Setting
PRODUCT_BUILD_SUPER_PARTITION to false causes partial builds to skip
building super.img and super_empty.img, instead relying on these images
to come from the final merged build.
Bug: 134764140
Test: Building & booting a dynamic-partition-enabled mixed build, and
inspecting partial builds' logs / out folder to ensure that
super.img/super_empty.img were not created.
Change-Id: I99431a9a342e9b0617510e250597f3024ef39322
common.AVB_VBMETA_PARTITIONS was recently added (commit
08c190fc89) for the same purpose.
Test: TreeHugger
Change-Id: I65572d54c22a753fdef80677377fcc9b684ee16f
merge_target_files.py step.
Bug: 134681035
Test: Built a merged build and ensured that the new timestamp was
visible in the log.
Change-Id: Ia6bbda48c7f57afdb6482253eaf0b3b0ea067468
Also fixes small nit from previous change to write_sorted_data().
Bug: 132788610
Test: python -m unittest test_merge_target_files
Test: Creating and booting a merged build.
Change-Id: I3dc43a4fe55b86b436dec08feb5d70096d38de36
When using Verified Boot 2.0, releasetools specifies a salt value based
on build fingerprint, so that to give idempotent images.
However, the change that removed static `ro.build.fingerprint` [1] broke
the behavior, as common.LoadInfoDict still relies on fingerprints.
Without a fixed salt, the first call to make_recovery_patch.py and the
second one (which writes IMAGES/{boot,recovery}.img) will see different
images, which leads to install-recovery.sh failure.
Note that currently there's a dependency that requires getting bootable
images through two separate calls. make_recovery_patch.py has to happen
first to get (placeholder) files in the system image. We then generate
canned fs_config files, and finally use add_img_to_target_files.py to
write the images.
This CL adds a quick workaround to force rebuilding the
recovery-from-boot patch while calling add_img_to_target_files.py.
[1] https://android-review.googlesource.com/c/platform/build/+/892933
Bug: 134123803
Bug: 134525174
Test: TreeHugger
Test: Build a non-A/B target that uses AVB. Run validate_target_files.py
on the generated target_files.zip.
Change-Id: I5859e30be63bfd54398cf41fd2d907f15285f560
Chained vbmeta partitions (vbmeta_system, vbmeta_vendor) were added to
support dynamic partitions. validate_target_files.py misses the logic in
handling such partitions.
Bug: 132882632
Test: Run validate_target_files.py on a target_files.zip that uses
chained vbmeta_system partition.
Change-Id: Id06c575d13d5e9cc1b621f485ceb75d3e354c39f
Merged-In: Id06c575d13d5e9cc1b621f485ceb75d3e354c39f
(cherry picked from commit 814b14b3f7)
This is to prevent a user from accidentally including files from the
wrong build. For example, adding any SYSTEM/ line to other_item_list
while keeping SYSTEM/* in system_item_list would cause the other build
to introduce an extra or changed file in the system image.
Bug: 132730710
Test: python -m unittest test_merge_target_files
Change-Id: Ic1178cdc9b991114f293ff3f2b4e6054e06647c6
This enables mixed builds to use the file_contexts.bin from the system
build when regenerating images that come from system target files, and
similarly for file_contexts.bin from the other build when regenerating
images from other target files.
In monolithic (non-mixed) builds all image-specific selinux_fc props
point to the same file_contexts.
Bug: 132108151
Test: Built and booted mixed build devices.
Change-Id: Id51ed6d96ea6337879f1ab21d47c93c67bc25312
In device root directory, we have the following symlinks:
- /odm/app -> /vendor/odm/app
- /odm/bin -> /vendor/odm/bin
- /odm/etc -> /vendor/odm/etc
...
This allows the Generic System Image (GSI) to be used on both devices:
1) Has a physical odm partition, where those symlink will be hidden
when /odm is used as the mount point
2) Has no physical odm partition and fallback to /vendor/odm/.
We can't just have the symlink /odm -> /vendor/odm, because the former
devices won't have /vendor/odm directory, which leads to mount failure
when the mount point /odm is resolved to /vendor/odm.
The existing /vendor/odm/build.prop won't be loaded in the latter
devices, because there is no symlink:
- /odm/build.prop -> /vendor/odm/build.prop.
Note that init blocks reading through direct symlinks (O_NOFOLLOW) so
the above symlink won't work either. This CL moves the odm build.prop
to /odm/etc/build.prop for init to load it (symlinks in earlier
components of the path will still be followed by O_NOFOLLOW).
Bug: 132128501
Test: boot a device and checks /odm/etc/build.prop is loaded
Test: make dist with an odm.img, checks $OUT/odm/etc/build.prop is loaded
Change-Id: I6f88763db755c9ec6068bfdd9cee81c19d72e9d7
Merged-In: I6f88763db755c9ec6068bfdd9cee81c19d72e9d7
(cherry picked from commit 6c62884000)
In device root directory, we have the following symlinks:
- /odm/app -> /vendor/odm/app
- /odm/bin -> /vendor/odm/bin
- /odm/etc -> /vendor/odm/etc
...
This allows the Generic System Image (GSI) to be used on both devices:
1) Has a physical odm partition, where those symlink will be hidden
when /odm is used as the mount point
2) Has no physical odm partition and fallback to /vendor/odm/.
We can't just have the symlink /odm -> /vendor/odm, because the former
devices won't have /vendor/odm directory, which leads to mount failure
when the mount point /odm is resolved to /vendor/odm.
The existing /vendor/odm/build.prop won't be loaded in the latter
devices, because there is no symlink:
- /odm/build.prop -> /vendor/odm/build.prop.
Note that init blocks reading through direct symlinks (O_NOFOLLOW) so
the above symlink won't work either. This CL moves the odm build.prop
to /odm/etc/build.prop for init to load it (symlinks in earlier
components of the path will still be followed by O_NOFOLLOW).
Bug: 132128501
Test: boot a device and checks /odm/etc/build.prop is loaded
Test: make dist with an odm.img, checks $OUT/odm/etc/build.prop is loaded
Change-Id: I6f88763db755c9ec6068bfdd9cee81c19d72e9d7
Merged-In: I6f88763db755c9ec6068bfdd9cee81c19d72e9d7
(cherry picked from commit 6c62884000)
Bug: 131437873
Test: Built system-only and vendor builds for merge. Compared
resulting apkcertx and apexkeys text files to that of a monolithic
build,
Test: Created colliding entries in both apexkeys and apkcerts text
files and ensure the script exited with an appropriate error message.
Test: Created unit tests to cover both non-colliding and colliding
entries
Change-Id: I6e42ce682ffa9059344e8cd63ba3a720c1f93452
Bug: 132687993, 131687150
This CL moves SignApex() from sign_target_files_apks into apex_utils,
and adds sign_apex that allows signing a standalone APEX file directly.
Test: Run the following command and check the output file.
$ build/make/tools/releasetools/sign_apex.py \
-v \
--container_key \
build/make/target/product/security/testkey.x509.pem \
--payload_key external/avb/test/data/testkey_rsa4096.pem \
--payload_extra_args \
"--signing_helper_with_files ./signing-helper.sh" \
foo.apex \
signed-foo.apex
Test: Run sign_target_files_apks.py on crosshatch target_files.zip.
Change-Id: I4b2422fd5cb1c60a3aa94511475e2a0e5b1666ca
This CL moves SignApex() from sign_target_files_apks into apex_utils,
and adds sign_apex that allows signing a standalone APEX file directly.
Test: Run the following command and check the output file.
$ build/make/tools/releasetools/sign_apex.py \
-v \
--container_key \
build/make/target/product/security/testkey.x509.pem \
--payload_key external/avb/test/data/testkey_rsa4096.pem \
--payload_extra_args \
"--signing_helper_with_files ./signing-helper.sh" \
foo.apex \
signed-foo.apex
Test: Run sign_target_files_apks.py on crosshatch target_files.zip.
Change-Id: I4b2422fd5cb1c60a3aa94511475e2a0e5b1666ca
Commit 7df64c3e starts to call common.LoadInfoDict() when generating
image archive, which reads additional files under BOOT/, RECOVERY/ and
ROOT/. Unzip everything from the target_files.zip.
Bug: 132456827
Test: Run img_from_target_files.py on previously failing
target_files.zip.
Change-Id: I22ee57c4f765bee9494478bf115b1581877401f4
Commit 7df64c3e starts to call common.LoadInfoDict() when generating
image archive, which reads additional files under BOOT/, RECOVERY/ and
ROOT/. Unzip everything from the target_files.zip.
Bug: 132456827
Test: Run img_from_target_files.py on previously failing
target_files.zip.
Change-Id: I22ee57c4f765bee9494478bf115b1581877401f4
The former comment no longer applies, as we have been always packing
META/file_contexts.bin in a target_files.zip (commit aa7318c3, since
Nougat), and we no longer look for the one under BOOT/RAMDISK/ (commit
d14b8956, since Q).
Test: N/A
Change-Id: I03f361234bf440e942f21e5a624862590248544b
This file is used by OTA generation so it needs to appear in mixed
builds with the combined content from the system and other versions of
the file.
Test: python -m unittest test_merge_target_files
Test: Running merge_target_files on a dynamic-partition-enabled build
and observing the resulting target files.
Bug: 131889742
Change-Id: I4ddbebc087e430f6307d0bd5461121a374e58ea4
Bug: 123428770
Test: Built system-only image and checked that no boot.img or
recovery.img files where created. Booted the resulting merged build on
device.
Change-Id: I760476502775e68125907c39e66b8665e789a798
Bug: 131710801
Test: Run sign_target_files_apks.py on a target that uses vbmeta_system.
Change-Id: I3bc526af3ec9f2680ca17ee5535607cff3ae9523
Merged-In: I3bc526af3ec9f2680ca17ee5535607cff3ae9523
(cherry picked from commit d6085d6834)
The old process_file_contexts_bin function did not properly generate a usable
file_contexts.bin to regenerate images, so instead use the file_contexts.bin
from the other partial target files package. When combining any one of several
other partial target files packages with a single system partial target files
package, this file will properly apply contexts as long as the same source is
used for the system partial target files.
Test: Verify that file contexts are properlty applied to vendor image.
Bug: 131584454
Change-Id: I16f8cc3b7f2eb7f09746f0ddcb2c1daf3fd19da6
Some properties had 'test-keys' still set
after signing the target files zip for release.
These properties are now added to the RewriteProps
method.
Bug: 131810966
Test: manual
Test: `atest releasetools_test`
Change-Id: Ifb352ed28f5100f1e9f686d77e935723f7f6d3ae
Merged-In: Ifb352ed28f5100f1e9f686d77e935723f7f6d3ae
(cherry picked from commit 234f4b418f)
Some properties had 'test-keys' still set
after signing the target files zip for release.
These properties are now added to the RewriteProps
method.
Bug: 131810966
Test: manual
Test: `atest releasetools_test`
Change-Id: Ifb352ed28f5100f1e9f686d77e935723f7f6d3ae
common.GetCareMap() may return an empty list on unavailable care_map
since the change in commit 8bdfb990ea.
Caller needs to handle such a case accordingly. This CL fixes the caller
in add_img_to_target_files.py, and changes the return value to None to
break legacy callers loudly.
Fixes: 131794385
Test: `atest releasetools_test`
Change-Id: I7c94f456064199237e84ef75732bdd10ebe31736
When set, product-img-tag.zip contains super.img instead of individual
user images from target files. For virtual devices, super.img is needed
to boot the device, but individual user images aren't needed.
Test: on A/B DAP, with flag set:
- m updatepackage and look at img.zip
- img_from_target_files
both have super.img and not system / vendor / system_other
Test: on non-A/B DAP, with the flag set:
- m updatepackage and look at img.zip
- img_from_target_files
both have super.img and not system / vendor
Test: on A/B retrofit, with the flag set:
- m updatepackage and look at img.zip
- img_from_target_files
both have super_*.img and system_other.img, but not system / vendor
Bug: 113175337
Change-Id: I94e33091d0c837cae40776176b4dcfdd338aba90
(cherry picked from commit 0e97dbb8ca)
Merged-In: I94e33091d0c837cae40776176b4dcfdd338aba90
When set, product-img-tag.zip contains super.img instead of individual
user images from target files. For virtual devices, super.img is needed
to boot the device, but individual user images aren't needed.
Test: on A/B DAP, with flag set:
- m updatepackage and look at img.zip
- img_from_target_files
both have super.img and not system / vendor / system_other
Test: on non-A/B DAP, with the flag set:
- m updatepackage and look at img.zip
- img_from_target_files
both have super.img and not system / vendor
Test: on A/B retrofit, with the flag set:
- m updatepackage and look at img.zip
- img_from_target_files
both have super_*.img and system_other.img, but not system / vendor
Bug: 113175337
Change-Id: I94e33091d0c837cae40776176b4dcfdd338aba90
When odm is changed, device manifest/matrices should be included.
When product is changed, framework manifest/matrices should be included.
Bug: 130714844
Bug: 126770403
Test: build with odm and product VINTF metadata
Change-Id: I49c8083e0e7185ae7b96047d68f1f624b1113dfc
Test: `atest --host releasetools_test`
Test: `m dist` with a target that uses non-sparse images.
Test: Run UpdateVerifierTest on blueline.
Change-Id: I8fdebee42fcaac78c2d1be2a84ddb69f46ec701d
For an PRESIGNED APEX, it has the following format, which should be
considered as a valid input.
name="foo.apex" public_key="PRESIGNED" private_key="PRESIGNED" container_certificate="PRESIGNED" container_private_key="PRESIGNED"
Bug: 131153746
Test: Run sign_target_files_apks.py on a target_files.zip with PRESIGNED
APEXes.
Test: python -m unittest sign_target_files_apks
Change-Id: I51076b0c6eddfb75637d37659a08009f0a88e931
(cherry picked from commit f454c3a0b4)
For an PRESIGNED APEX, it has the following format, which should be
considered as a valid input.
name="foo.apex" public_key="PRESIGNED" private_key="PRESIGNED" container_certificate="PRESIGNED" container_private_key="PRESIGNED"
Bug: 131153746
Test: Run sign_target_files_apks.py on a target_files.zip with PRESIGNED
APEXes.
Test: python -m unittest sign_target_files_apks
Change-Id: I51076b0c6eddfb75637d37659a08009f0a88e931
By sorting the content of the final output merged target files package, the
merged target files package is more like the target files packages generated by
a build.
Test: Generate merged target files package, verify that content is sorted.
Change-Id: Ic0c198630ebd7692a3f3f9663d85e4b45229175c
When odm is changed, device manifest/matrices should be included.
When product is changed, framework manifest/matrices should be included.
Bug: 130714844
Bug: 126770403
Test: build with odm and product VINTF metadata
Change-Id: I49c8083e0e7185ae7b96047d68f1f624b1113dfc
Merged-In: I49c8083e0e7185ae7b96047d68f1f624b1113dfc
We used to require explicitly setting both (e.g. `-e foo.apex=` and
`--extra_apex_payload_key foo.apex=` to skip signing `foo.apex`).
This CL allows specifying `-e` alone to achieve the same result.
However, if a conflicting `--extra_apex_payload_key` is also specified,
that would be considered as a config error.
Bug: 131153746
Test: Run sign_target_files_apks.py with `-e foo.apex=` alone to skip
signing foo.apex.
Test: Run sign_target_files_apks.py with `-e foo.apex=` and
`--extra_apex_payload_key foo.apex=key` and expect assertion error.
Change-Id: Ia747f59ee726b60bdb1445024e749320171064c2
This is used by merge_target_files to prevent an unnecessary unzip and
copy.
Test: Ran merge_target_files.py and booted using the img.zip.
Change-Id: I6fe0dd025b30b3f4965c9b22fb6943019bf5899b
The boot-debug.img should NOT be release signed and can only be used
if the device is unlocked. Adding a check to prevent the tool from
signing this debuggable boot.img.
See the following for more details about boot-debug.img:
https://android-review.googlesource.com/c/platform/build/+/947857
Bug: 126493225
Test: put a file /force_debuggable into boot.img, checks the following
command fails:
./build/tools/releasetools/sign_target_files_apks \
out/dist/*-target_files-*.zip signed-target_files.zip
Change-Id: Ia5232949cb9582d2b4eaa171d9e9f3fe7317d418
Merged-In: Ia5232949cb9582d2b4eaa171d9e9f3fe7317d418
(cherry picked from commit 78369ebbc1)
The boot-debug.img should NOT be release signed and can only be used
if the device is unlocked. Adding a check to prevent the tool from
signing this debuggable boot.img.
See the following for more details about boot-debug.img:
https://android-review.googlesource.com/c/platform/build/+/947857
Bug: 126493225
Test: put a file /force_debuggable into boot.img, checks the following
command fails:
./build/tools/releasetools/sign_target_files_apks \
out/dist/*-target_files-*.zip signed-target_files.zip
Change-Id: Ia5232949cb9582d2b4eaa171d9e9f3fe7317d418
This simplifies the use case for mixed build users. Instead of having to
remember to call img_from_target_files.py after this script, they can
use this flag to automatically create the IMG package.
Also includes an update to super_empty.img logic. The super_empty.img is
now always created for dynamic-partition builds. The flag now only
controls copying the super_empty.img to a user-provided location.
Bug: 129976345
Test: Ran merge_target_files.py using --output-img and
--output-super-empty and inspected the resulting img zip and
super_empty.img.
Change-Id: Ida602942bb7a6b4b94f4e225640af9104fc9360c
This simplifies the use case for mixed build users. Instead of having to
remember to call ota_from_target_files.py after this script, they can
use this flag to automatically create the OTA package.
Bug: 129976345
Test: Ran merge_target_files.py using --output-ota and inspected the
resulting zip.
Change-Id: Icc95943c24b8f83b3221e845a7d69a34c1edb4fc
Any mixed build that uses dynamic partitions will require a
super_empty.img image. This image must be created from the merged
misc_info.txt file, so adding this functionality here simplifies
the creation of this image for users (versus having to call
build_super_image.py manually after calling merge_target_files.py).
Bug: 129976345
Test: Ran merge_target_files.py on a dynamic partition enabled build
using the new --output-super-empty flag.
Change-Id: I73901f363d73c9fae1af1579faa2a908369dbbec
This provides the ability to run merge_target_files without the end goal
of a target files zip. This is useful for users that only want the IMAGES
folder, for example.
Bug: 130304869
Test: python -m unittest test_merge_target_files
Change-Id: If0412b8e1eb85fe09d7b689fd7f56ce84067faea
They used to be disabled due to the assertion of search_path in setUp()
function, which is not a prerequisite for most of the tests.
Bug: 112080715
Test: `atest releasetools_test`
Test: TreeHugger
Change-Id: I3cbaf42aa09dba0b87a64e11d97de9b3f7af7a47
FileImage needs to be thread-safe because multiple
threads gets data from it when an incremental OTA
package is created.
Test: apply incremental OTA on cuttlefish
Bug: 113175337
Change-Id: I31637fce0fbd66f3fa6c5c478da09bae65a52229
Merged-In: I31637fce0fbd66f3fa6c5c478da09bae65a52229
FileImage needs to be thread-safe because multiple
threads gets data from it when an incremental OTA
package is created.
Test: apply incremental OTA on cuttlefish
Bug: 113175337
Change-Id: I31637fce0fbd66f3fa6c5c478da09bae65a52229
About half of the testcases rely on external tools (i.e. the ones in
`otatools.zip`, which are external to releasetools module, but still
built by Android). It's WAI as releasetools scripts are mostly for
gluing purpose.
However, the current support in Soong doesn't allow packing the helper
modules as part of the built releasetools_test. This CL adds a decorator
that allows declaring external dependencies in testcases, which will be
skipped while running in presubmit. It doesn't affect local invocation
of `atest releasetools_test`.
Fixes: 112080715
Test: `atest releasetools_test`
Test: TreeHugger; check that releasetools_test is invoked (and test
passes).
Change-Id: I8fdeb6549023cf5ddeb79d610c7c37cf9f13d3cc
All the unittests will be built into releasetools_test. One can run the
tests with `atest releasetools_test` or the traditional way
`test_utils.py`. The atest way is recommended, which additionally builds
the required tools.
With the current support in Soong, we can't pack the built tools into
releasetools_test yet. So running `releasetools_test` alone in clound
would fail. Follow-up CLs will address the issue in order to deploy the
tests with TEST_MAPPING.
Bug: 112080715
Test: `atest releasetools_test`
Change-Id: Ica95517a5ab326f4e58fc57c6c2c276cfe882f3c
It returns a list of one generator object, not a list
of strings.
Test: test_blockimgdiff
Bug: 113175337
Change-Id: I8962c539c2ce3fae90d428b38c4b0e52c5a2cdad
Merged-In: I8962c539c2ce3fae90d428b38c4b0e52c5a2cdad
The function used to be serving system and vendor partitions only (as
they were the only partitions using sparse image at the point). The code
itself doesn't rely on anything specific to system/vendor.
Test: python -m unittest test_common
Bug: 113175337
Change-Id: Ia4ecdeedb262f3d9db082128eaf9bab299983333
Merged-In: Ia4ecdeedb262f3d9db082128eaf9bab299983333
The signature size will be 512 bytes when signing the payload
with 4096 bits key. This cl determines the key size with
"openssl rsa -modulus"
The new key in testdata is generated by
"openssl genrsa -out testkey 4096"
Bug: 129163830
Test: generate and verify an OTA package
Change-Id: I6662b0a0c553dc0fd84711312a1256b887e332fd
(cherry picked from commit 376cc7c452)
assert-max-image-size doesn't make sense for
dynamic partitions, as build_image.py always find the
right size for the output image. Hence:
- build_image.py no longer need to write generated_*_info.txt
(which contains the size of the image).
- assert-max-image-size on the static BOARD_*IMAGE_PARTITION_SIZE. If
a partition is dynamic, that variable isn't set, and
assert-max-image-size becomes a no-op. If the partition is static,
assert-max-image-size checks the static partition size as it used
to be.
- Fix read-size-of-partitions to use the size of the partition by
reading the image directly (instead of using generated_*_info.txt).
For devices without AVB, with DAP enabled, and does not have
RESERVED_SIZE for partitions, because of right sizing, the original
code always warns about approaching size limits. Since such checks
doesn't make sense for dynamic partitions, remove them.
Test: builds on device with dynamic partitions
Test: builds on cuttlefish with DAP enabled (without AVB), no
more size limit warnings:
WARNING: out/target/product/vsoc_x86/vendor.img approaching size limit (X now; limit X)
This reverts commit 6e099095d1.
Reason for revert: reland the CL
Bug: 122377935
Test: build blueline_mainline
Change-Id: Iee594b64e687decff186c0fa60f82b88608febe9
Merged-In: Iee594b64e687decff186c0fa60f82b88608febe9
Also, move code from build_super_image.py to sparse_img.py.
Test: sparse_img.py on sparse and non-sparse images
Bug: 122377935
Change-Id: Ie91fdfdbb54298ea27eb20d1b5363aeb1470356e
Merged-In: Ie91fdfdbb54298ea27eb20d1b5363aeb1470356e
The function used to be serving system and vendor partitions only (as
they were the only partitions using sparse image at the point). The code
itself doesn't rely on anything specific to system/vendor.
Test: python -m unittest test_common
Change-Id: Ia4ecdeedb262f3d9db082128eaf9bab299983333
The signature size will be 512 bytes when signing the payload
with 4096 bits key. This cl determines the key size with
"openssl rsa -modulus"
The new key in testdata is generated by
"openssl genrsa -out testkey 4096"
Bug: 129163830
Test: generate and verify an OTA package
Change-Id: I6662b0a0c553dc0fd84711312a1256b887e332fd
* changes:
Only assert-max-image-size for static partitions.
sparse_img.py --get_partition_size return size of partition
Revert "Fix dynamic partition size check for devices with recovery"
If TARGET_USERIMAGES_SPARSE_EXT_DISABLED is set, don't provide
--sparse to lpmake, so that a non-sparse super image is built.
Test: build with the flag set.
Bug: 120041578
Change-Id: I5a26e4c793b0e2ddc89e9c38c8828ac21044e78a
Merged-In: I5a26e4c793b0e2ddc89e9c38c8828ac21044e78a
assert-max-image-size doesn't make sense for
dynamic partitions, as build_image.py always find the
right size for the output image. Hence:
- build_image.py no longer need to write generated_*_info.txt
(which contains the size of the image).
- assert-max-image-size on the static BOARD_*IMAGE_PARTITION_SIZE. If
a partition is dynamic, that variable isn't set, and
assert-max-image-size becomes a no-op. If the partition is static,
assert-max-image-size checks the static partition size as it used
to be.
- Fix read-size-of-partitions to use the size of the partition by
reading the image directly (instead of using generated_*_info.txt).
For devices without AVB, with DAP enabled, and does not have
RESERVED_SIZE for partitions, because of right sizing, the original
code always warns about approaching size limits. Since such checks
doesn't make sense for dynamic partitions, remove them.
Test: builds on device with dynamic partitions
Test: builds on cuttlefish with DAP enabled (without AVB), no
more size limit warnings:
WARNING: out/target/product/vsoc_x86/vendor.img approaching size limit (X now; limit X)
Fixes: 122377935
Change-Id: I75e1b8322197cb18cf397d02aefd49d777bb6405
Also, move code from build_super_image.py to sparse_img.py.
Test: sparse_img.py on sparse and non-sparse images
Bug: 122377935
Change-Id: Ie91fdfdbb54298ea27eb20d1b5363aeb1470356e
If TARGET_USERIMAGES_SPARSE_EXT_DISABLED is set, don't provide
--sparse to lpmake, so that a non-sparse super image is built.
Test: build with the flag set.
Fixes: 120041578
Change-Id: I5a26e4c793b0e2ddc89e9c38c8828ac21044e78a
For non-A/B, you must supply the --system-item-list, --other-item-list, and
--system-misc-info-keys parameters approrpriate for merging two partial target
files. Additionally, you must supply the --rebuild-recovery option to correctly
generate the filesystem config and file contexts. With all of these parameters,
the script will generate a merged target files containing the correct recovery
files.
Also fix the --keep-tmp option to be consistent with the other options.
Bug: 122813742
Test: verify that merged target image boots and can perform OTA.
Change-Id: I5a942ac0cd9924fec419a686794a2340304594c8
This reverts commit 9788b4ed31. All the
blocking issues have been addressed.
Fixes: 120517892
Test: Run validate_target_files.py on crosshatch signed
target_files.zip.
Change-Id: I95de241e159998e002dedddafea65953b1a1b263
Previously it was following a wrong order by doing `zipalign` after
calling SignApk, which effectively compromised the signature. This CL
corrects the logic, and follows the same flow as in build system:
- Pack APEX file;
- `zipalign -f 4096`;
- Call SignApk to sign the container with `-a 4096` flag.
Bug: 129148142
Test: Run sign_target_files_apks.py on taimen target_files.zip. Boot the
image after signing.
Change-Id: I91bd3dce4f45c1891c5e122212a699f4808618fa
(cherry picked from commit 0e06cb0a8b)
Previously it was following a wrong order by doing `zipalign` after
calling SignApk, which effectively compromised the signature. This CL
corrects the logic, and follows the same flow as in build system:
- Pack APEX file;
- `zipalign -f 4096`;
- Call SignApk to sign the container with `-a 4096` flag.
Bug: 129148142
Test: Run sign_target_files_apks.py on taimen target_files.zip. Boot the
image after signing.
Change-Id: I91bd3dce4f45c1891c5e122212a699f4808618fa
To build a complete list of the dynamic partitions and partitions
groups, we need to merge the contribution from the system and other
target files.
Bug: 127687287
Test: Running merge_target_files.py and observing partition lists are
merged as expected.
Change-Id: I5bb9bd0e3179d48c9bfacdb3aca8253158f61cf6
For PRESIGNED APEXes, we should keep carrying the matching public keys
at /system/etc/security/apex.
Bug: 129148142
Test: Run sign_target_files_apks.py on a target_files.zip with presigned
APEXes. Check the output zip.
Change-Id: I2e941fd9b10e99d2db9df1e5308cbbe8c760177b
(cherry picked from commit bf3fb024cd)
For PRESIGNED APEXes, we should keep carrying the matching public keys
at /system/etc/security/apex.
Bug: 129148142
Test: Run sign_target_files_apks.py on a target_files.zip with presigned
APEXes. Check the output zip.
Change-Id: I2e941fd9b10e99d2db9df1e5308cbbe8c760177b
This reverts commit 5516d37f41.
The previous issue in unzipping non-matching files has been addressed
with commit a49054ca2f2959f50f3188914ec0faebc90ebcbe. This CL rolls
forward to allow dumping container certifcates for APEXes.
Bug: 128848294
Test: Run check_target_files_signatures.py on target_files.zips w/ and
w/o APEX files.
Change-Id: I662aab3d96fc40ac8e5e206e32b73ac763220b70
common.UnzipTemp() calls `unzip` to do the unzipping, which will
complain if there's non-existent names in the given list. Prior to this
CL, callers had to do the work to remove non-existent entries. This CL
filters out the given patterns in common.UnzipTemp()/common.UnzipToDir()
to make callers' works easier.
Bug: 128848294
Test: `m dist` with aosp_taimen-userdebug (which calls
ota_from_target_files.py on a target_files.zip that doesn't
contain RADIO/*).
Test: `python -m unittest test_common.CommonZipTest`
Change-Id: I5e741c27ea8d0b8126c398a7e1b56a8deb4a3d7f
Currently system_other AVB public key is placed in system.img.
However, this makes it's harder to have a *generic* system.img
across different product configs. Moving the key to /product
partition to allow more product-specific AVB keys.
Device board config can add /product/etc/fstab.postinstall,
to mount system_other with this key in /product. It can specify
different mount options, file systems, verity settings, etc., in
this product-specific fstab as well.
Bug: 123611926
Test: `make productimage` checks the following is generated.
$OUT/product/etc/security/avb/system_other.avbpubkey
Also checks it's included in $OUT/installed-files-product.{json, txt}
Test: run the following command and checks that
PRODUCT/etc/security/avb/system_other.avbpubkey is updated:
./build/tools/releasetools/sign_target_files_apks \
--avb_system_other_algorithm SHA256_RSA2048 \
--avb_system_other_key external/avb/test/data/testkey_rsa2048.pem \
out/dist/*-target_files-*.zip signed-target_files.zip
Change-Id: I6804f29941bec54375d80bd68a5aedb5c23b842e
This CL adds support that allows treating an APEX as pre-signed. We can
skip signing an APEX with `-e <apex-name>=` and
`--extra_apex_payload_key <apex-name>=`. Note that the payload_key and
container_key must be in consistent state - either they're both
PRESIGNED or none of them is. CheckApkAndApexKeysAvailable() has been
updated to perform the sanity check.
Bug: 123716522
Test: Run sign_target_files_apks.py with the above flags.
Test: python -m unittest test_sign_target_files_apks
Change-Id: Id1e2f3f2facd4a97a385983cc9b78c028f7e7e73
This validation is to help ensure that any usage of custom merge config
files does not accidentally exclude any item that has been added to the
default config lists.
Bug: 124197349
Test: Run merge_target_files with custom merge config files.
Change-Id: I34c51cb75212368146a2944d37621f311060d24d
This reverts commit d8469727bc. The script
is broken on target_files.zip that don't contain any APEX.
Bug: 128848294
Test: Run check_target_files_signatures.py on target_files.zip w/o APEX.
OTA tools should pick up the avbtool, as listed in dict['avb_avbtool'],
from the current PATH (plus bin/ under the dir specified via `--path`),
the same way as handling all other host tools.
Test: `m dist`
Change-Id: I3eb4d2c61979b03d9c23b2403d9a38cf052d87ea
Also makes AddSystem check that an output_zip exists before attempting
to add the recovery patch to the output zip.
Bug: 128838154
Test: Running merge_target_files with --rebuild_recovery and verifying
it passes --rebuild_recovery to add_img_to_target_files.
Change-Id: I19347b2c0dabf29b7196045b18551b5d0687df2c
The keys_info in the touched code is a tuple, which is immutable.
Bug: 123716522
Test: Run sign_target_files_apks.py with '-e foo.apex=bar' that replaces
the APEX container key.
Change-Id: I4e57e46c93a56b7f6646764d021ebb42c19bf7f5
Bug: 123716522
Test: Run sign_target_files_apks.py to sign a target_files with APEXes.
Test: Run check_target_files_signatures.py on signed artifact.
Test: python -m unittest test_sign_target_files_apks
Change-Id: I3fa13e3d9461cf5e0838e0572d436e218164fe41
(cherry picked from commit aa7e993a22)
Only the container certs will be checked and reported. For the payload
within an APEX, we can't easily extract the cert info.
It needs to go along a longer path, if ever needed, by:
- extracting public keys from all the available certs;
- using each of them to verify against an APEX payload to find a match
(`avbtool verify_image --image payload --key public_key`).
Bug: 123716522
Test: Run check_target_files_signatures.py on target_files with APEXes.
Change-Id: I2ef318e05433d2d65ab84e2dff9e01fb6ee3373d
(cherry picked from commit d8469727bc)
Bug: 123716522
Test: Run sign_target_files_apks.py to sign a target_files with APEXes.
Test: Run check_target_files_signatures.py on signed artifact.
Test: python -m unittest test_sign_target_files_apks
Change-Id: I3fa13e3d9461cf5e0838e0572d436e218164fe41
Only the container certs will be checked and reported. For the payload
within an APEX, we can't easily extract the cert info.
It needs to go along a longer path, if ever needed, by:
- extracting public keys from all the available certs;
- using each of them to verify against an APEX payload to find a match
(`avbtool verify_image --image payload --key public_key`).
Bug: 123716522
Test: Run check_target_files_signatures.py on target_files with APEXes.
Change-Id: I2ef318e05433d2d65ab84e2dff9e01fb6ee3373d
Other modules have switched to logging module. sign_target_files_apks.py
needs to init the logger to get the logs.
Test: Run `sign_target_files_apks.py -v`. Check outputs.
Test: Run `check_target_files_signatures.py -v`.
Change-Id: Ic68c019f6fb14840561885f1194ad6efdfdb7d82
superimage-nodeps and supernod depends
on images from $(ANDROID_PRODUCT_OUT) (not from
target files package). It doesn't rebuild source
images if they are present.
A typical workflow is:
m -j
# change code in system
m snod -j
m supernod -j
Test: For non retrofit, run:
`m snod -j; m supernod -j`
Fixes: 128321505
Change-Id: Ib8c011cadb9c0cd334234aef39f19be6a48fee62
Bug: 124467065
Test: Running `python merge_target_files.py` using the three new flags
and observing that their contents are passed to the merge_target_files() function.
Change-Id: I4de46f041f5ae8bc8be2730313ce873a952bf78e
sign_target_files_apks script looks for the signapk.jar inside the out dir.
If the our dir is set to a different directory via OUT_DIR_COMMON_BASE the script does not work properly.
From now script checks if the OUT_DIR_COMMON_BASE is set, then searches the jar in the proper path.
If OUT_DIR_COMMON_BASE is unset, searches in "out" like it did before.
Test: Build with OUT_DIR_COMMON_BASE set and unset and verify that sign_target_files_apks works in both cases
Change-Id: I9218b98ff79526184f8353705640193405afac9e
Regnier, Philippe