This breaks down the current WriteABOTAPackageWithBrilloScript() into
smaller and testable units, which also prepares for the work in
b/35724498.
Bug: 35724498
Test: python -m unittest test_ota_from_target_files
Test: Get identical A/B OTA packages w/ and w/o the CL.
Change-Id: I2ea45ce98e2d2baa58e94fb829b7242f6fe685a7
Merged-In: I2ea45ce98e2d2baa58e94fb829b7242f6fe685a7
(cherry picked from commit 036d721812)
The same thing has been set in GetPackageMetadata(), so they are no-op.
Also replace an occurrence of 'OPTIONS.source_info_dict' with
source_info, which has been missed from previous clean-up CLs.
Test: Generate an A/B OTA package, and an incremental BBOTA with
--wipe_user_data. Check the metadata.
Test: Generate a two-step incremental BBOTA successfully.
Change-Id: I4bb491cac9064d93fb86d12e617c8f38f040e01e
The info comes from the build.prop file of the target build (thus no
backward compatibility concerns). OTA server and client can optionally
use these info to understand the expected behavior of an OTA package.
Bug: 72751683
Test: python -m unittest test_ota_from_target_files
Test: `m dist`, then check the metadata in the generated OTA package.
Change-Id: I5935f67684d2486bb5f00d67ce4bc756589a56ed
This CL detects incomplete block ranges (e.g. due to the holes in
mke2fs created images). Such block ranges will be tagged, so we won't
attempt to imgdiff those files. Note that the change to blockimgdiff.py,
which uses the tag info, will come in a separate CL.
An 'extra' attribute is added to RangeSet class, which defaults to an
empty dict. An 'incomplete' tag will be added into the dict by the
caller of the class. Not adding this tag as an immediate attribute,
because it is not a property regarding the ranges being represented, but
rather some storage space for the caller.
This CL also refactors GetSparseImage and RoundUpTo4K into common.py, so
the same code can be called from both of ota_from_target_files.py and
validate_target_files.py. Not able to add unittests for
GetSparseImage(), as SparseImage requires data in specific format.
Bug: 68016761
Test: Run validate_target_files.py on target-files.zip. It skips
validating files with missing holes as before.
Test: Run ota_from_target_files.py on angler target-files.zip. It gives
identical packages w/ and w/o the CL.
Test: pylint on changed files. There're warnings with common.py, but
unrelated to this change.
Change-Id: I126ccfea13c0d5ebcc8c1b4ff1a4f9200e97423a
Create a wrapper class that handles the payload signing, which unifies
the paths with and without external signer. Also add tests for the newly
added class.
The test keys under testdata/ are created with the script in
development/tools/make_key. testdata/testkey_with_passwd.pk8 uses
password "foo".
Test: python -m unittest test_ota_from_target_files
Test: Get identical A/B OTA packages w/ and w/o the CL.
Change-Id: Ic770aec726498a3babb88ec509985e7f1210fb18
Now they (A/B, non-A/B full and non-A/B incremental) share the same
code. Also add tests for the newly added function.
Test: python -m unittest test_ota_from_target_files
Test: Build A/B and non-A/B packages respectively.
Change-Id: I5499dd9c683ac1a18d8a24f21ff8845fe69f343e
In non-A/B OTA path, we've been loading the info dict for the target
build twice (once to have a peek at the OTA type, with a reload after
unzipping the target-files zip). Remove the reloading.
This CL moves the loading of the source info dict up a bit to avoid
having two copies in the A/B and non-A/B paths. It also moves up the
sanity checks of the loaded info dicts before unzipping the target-files
zips.
Test: Generate full and incremental packages for bullhead and marlin.
Change-Id: Iccb953feb0b6ecf62e6f73d6e9ceea00077d098d
GetBuildProp() and GetVendorBuildProp() have been factored into
BuildInfo class. These two functions were meant to be deleted in commit
481bab8d48.
Test: python -m unittest test_ota_from_target_files
Test: No users of these two functions.
Change-Id: Id590a46d597939973ef2339cdfea4f5163548d4d
The fallback-to-full logic doesn't look beneficial, but may hide away
real bugs, especially when it's on by default.
Test: Generate an incremental non-A/B OTA successfully.
Test: Inject bugs into script; observe that incremental OTA
generation errors out instead of silently falling back.
Test: No internal users of the flag from code search.
Change-Id: Ifc627d898552c0bd72463e8c107e36610421608e
Prior to this CL, we have similar codes in different paths
(WriteFullOTAPackage / WriteBlockIncrementalOTAPackage /
WriteABOTAPackageWithBrilloScript). This CL factors out the common codes
that deal with OEM-specific properties into BuildInfo class, and adds
tests for the new class.
Test: python -m unittest test_ota_from_target_files
Test: Create an incremental package that uses and doesn't use OEM
properties respectively.
Test: Create a full package that uses and doesn't use OEM properties
respectively.
Change-Id: Ida914cace12803d55396fa503bafcac2db2a520e
The code is on infeasible path since we already have assertions in
common.BlockDifference().
Also remove the dead code that checks for OPTIONS.info_dict, as we
already set that in ota_from_target_files.main(), for both of A/B and
non-A/B.
Test: Generate incremental OTAs w/ and w/o the CL, and get identical
packages.
Change-Id: Ifb8fc101e78f5ce58c60c8e49028b66ce0d20246
The CL in [1] unintentionally breaks the OEM dict loading logic in the
incremental BBOTA path. We should always require and load the OEM
property dict if _either_ of the source and target builds uses OEM
properties. Otherwise with the current "and" operator, it skips loading
the OEM property dict and thus fails to generate an OTA package that has
OEM property changes (e.g. updating from build with fingerprint to
another one using thumbprint).
The CL in [1] actually makes the right change in the file-based OTA
path, but introduces the bug in the block-based OTA path.
This CL also cleans up the line that reads recovery_mount_options.
[1] commit 7f804ba71f ("releasetools:
allow for multiple OEM property values.").
Test: Genearte an OTA that has OEM property changes successfully.
Change-Id: Idce4ad59825d432618535ce09ab22bd7ddc524f2
We used to have "--gen_verify" that would generate a verify package for
non-A/B devices. Since a) we don't have active users; b) it works with
non-A/B only; c) we have better alternatives (e.g. using fastboot to
compute the partition checksum), this CL removes the support.
Test: `m dist`
Change-Id: Ib4a2c8c2a0394b54e66c106089aa52a0b900f034
We have been shelling out to 'zip -d' to delete existing ZIP entries in
add_img_to_target_files.py. This CL moves the function into common.py,
and calls that for the similar work in ota_from_target_files.py. This CL
also adds unittests for the newly added function.
Test: `m dist`
Test: python -m unittest test_common
Test: ota_from_target_files.py generates identical packages w/ and w/o
the CL (so we know the streaming property computation is intact).
Test: Run 'add_img_to_target_files.py -a' that triggers a call to
ReplaceUpdatedFiles().
Change-Id: Icaa6c3ea3ee2166023f78fa79275295f837ea842
'-w' is a short option for '--wipe_user_data', which happens to be the
same as the whole-file signing flag for signapk.
This CL removes the short version to make it less error-prone. Users
should use '--wipe_user_data' to generate packages with enforced data
wipe.
Bug: 70259309
Test: ota_from_target_files.py generates packages with and without
'--wipe_user_data'.
Test: Not seeing active users that uses 'ota_from_target_files -w'.
Change-Id: Ib8c138663a87aafc6a0b5b6f7c286fb74ef6134b
It has been long deprecated since commit
fdd8e69c42 (Donut).
Test: `m dist` with aosp_marlin-userdebug and aosp_angler-userdebug.
Test: No user of "--board_config" in code search.
Change-Id: Ic0336c22cac32de5de88161748d09b579f892e99
Commit 21803d3574 only added compatibility
metadata for A/B OTA packages, because by that time we didn't use
'ro.treble.enabled' property to determine if a target was Treble-enabled.
This CL uses 'ro.treble.enabled' to guard the packing of compatibility
metadata for both of A/B and non-A/B OTA packages. It also switches to
checking the system/vendor fingerprints to determine if there's an
update to the partition (previously it was computing the SHA-1 of the
images, which may have unintentionally changed due to issues that give
non-repetitive builds).
Bug: 64339310
Test: Generate OTA packages (full and incremental) on Treble-enabled
targets (sailfish, as well as a non-A/B angler target with
Treble-enabled locally); check that the compatibility.zip entry
exists.
Test: Generate OTA packages on Treble-unenabled non-A/B target (angler);
check that the compatibility.zip entry doesn't exist.
Test: Generate OTA packages on Treble-unenabled A/B target; check that
the compatibility.zip entry doesn't exist.
Change-Id: I2a1fcf612439d849ba8ccea217a0faf5d5ba8e14
This entry only exists in non-A/B full OTA path. However, this file has
never been loaded by recovery - recovery was only reading the one from
the recovery image (which also has been removed by a recent change in
[1]).
[1]: commit e35926e1aff2e6b9b54656bd59c8178e295a1b7e in
platform/bootable/recovery.
Bug: 63538434
Test: `m dist` on angler. file_contexts.bin is gone from full OTA
package. Sideloading the generated full OTA on angler.
Change-Id: Iec2e2d36599fa1ed499516a3e74f06cd36ade494
Current build system will include AVB metadata from each partition and
store them into /vbmeta partiton when BOARD_AVB_ENABLE is set, which makes
each partition tightly-coupled.
Add the support for 'chain partition':
- The vbmeta of each partition is stored on the same partition itself.
- The public key used to verify each partition is stored in /vbmeta.
For example, the following build variables are required to enable chain
partition for system partition:
- BOARD_AVB_SYSTEM_KEY_PATH := path/to/system_private_key
- BOARD_AVB_SYSTEM_ALGORITHM := SHA512_RSA8192
- BOARD_AVB_SYSTEM_ROLLBACK_INDEX := 1
- BOARD_AVB_SYSTEM_ROLLBACK_INDEX_LOCATION := 2
The corresponding settings will be added into META/misc_info.txt for
build_image.py and/or add_img_to_target_files.py:
- avb_system_key_path=path/to/system_private_key
- avb_system_algorithm=SHA512_RSA8192
- avb_system_add_hashtree_footer_args=--rollback_index 1
- avb_system_rollback_index_location=2
To enable chain partition for other partitions, just replace SYSTEM with
BOOT, VENDOR and/or DTBO in the build variables.
Also switch from `avbtool make_vbmeta_image --setup_rootfs_from_kernel system.img ...`
to `avbtool add_hashtree_footer --image system.img --setup_as_rootfs_from_kernel...`
when BOARD_BUILD_SYSTEM_ROOT_IMAGE is true. This works for both chained
and no-chained:
- chained: `avbtool add_hashtree_footer --setup_as_rootfs_from_kernel` will
add dm-verity kernel cmdline descriptor to system.img
- no-chained: `avbtool make_vbmeta_image --include_descriptors_from_image
system.img` will include the kernel cmdline descriptor from system.img into
vbmeta.img
Bug: 38399657
Test: `make` pass, flash images from $OUT and boot device without chain partitions
Test: `make` pass, flash images from $OUT and boot device with chain partitions
Test: `make dist` pass, flash images from TF.zip and boot device without chain partitions
Test: `make dist` pass, flash images from TF.zip and boot device with chain partitions
Test: follow the same steps in
https://android-review.googlesource.com/#/c/407572/
Change-Id: I344f79290743d7d47b5e7441b3a21df812a69099
When AVB is enabled, generate care_map.txt and add it to the target
files. Also copy it into the OTA package where it will later be used
by the update_verifier.
Bug: 62208947
Test: \
1. Run add_img_to_target_files on the TF of a new pixel device,
and care_map.txt generates successfully.
2. Make dist in oc-dr1-release and find care_map.txt in the OTA package.
3. update_verifier succeeds in reading all the blocks on the care_map,
and fails to read out-of-bound blocks.
Change-Id: I2881711e6f87789cb7de150dbeca18b756fed68a
* Add logic to handle decrypted keys from common.GetKeyPassword in
WriteABOTAPackageWithBrilloScript.
* Get the keys passwords in main and store them in OPTIONS.key_passwords.
This allows accessing them in WriteABOTAPackageWithBrilloScript and SignOutput
so it's only required to ask for the password once, while allowing to use
decrypted signing keys.
Test: ota_from_target_files.py on marlin and angler respectively.
Change-Id: I7c9b0198855a4b630c52b8552e904f312f09c4ce
This reverts commit b1e59b8a73.
This CL enables generating compatibility zip entry in an A/B OTA
package. We're not enabling the support for non-A/B OTAs for now until
they have the necessary support (e.g. /proc/config.gz exists).
Bug: 36810913
Test: Generate full and incremental packages for marlin and applly under
normal boot and sideload respectively.
Change-Id: I69f0a3feb7a0efc7f727f92f31c9e63f47ee6530
(cherry picked from commit b63c952cc7)
We've just created the target files package and still have the extracted
directory, so pass that to ota_from_target_files so that it doesn't have
to re-extract what we just compressed.
This saves a little bit of time -- for bullhead-userdebug on internal
master, this brings the time to build an OTA from ~340s -> ~310s. Much
of the time is still spent generating and signing the OTA.
Test: lunch bullhead-userdebug; m -j otapackage
Test: bullhead-ota-*.zip is identical before/after
Change-Id: Ib51cf6b02123e9c49280e9957c6590645470a52c
The line was unintentionally removed in commit 7f804ba.
Test: ota_from_target_files.py generates a full OTA. Check the package
metadata.
Change-Id: Icae88e2a9bb2bfc450a3d0d7ab524d6a6eac9df5
We have stopped shipping devices with file-based OTAs, and are not
actively maintaining the support. Devices using file-based OTAs
should be moved to block-based, if not A/B OTAs.
We will also need to clean up EdifyGenerator class, which will be
handled in follow-up CLs.
Bug: 35853185
Test: Generate full and incremental OTAs w/ and w/o the CL, and they
give identical packages.
Test: Not specifying --block also generates block-base OTAs.
Change-Id: I3b0fc8ce5600e109f3251fe41f655534aaa298c7
We must have created the images (system.img, system.map etc) prior to
calling ota_from_target_files.py (added by commit
2e0d8fcf08, into Lollipop).
Remove the obsolete suppport for handling "old" target_files zips that
don't have such images. This simplies the logic for BuildSystem() and
BuildVendor(), which now would only be called by
add_img_to_target_files.py itself.
Test: Generating full and incremental OTAs give the same results, w/ and
w/o this CL.
Change-Id: I0ea243d09d3378249d7982701ae4ec339b04b7b8
The first one in WriteVerifyPackage() is mismatching function parameters
that can be trivially fixed.
The other one is in WriteABOTAPackageWithBrilloScript(), where we don't
have edify script instance.
Test: `pylint --rcfile=pylintrc ota_from_target_files.py`.
Change-Id: Ie238ef5b296dfe9e725b61376992447b662d2376
This reverts commit a7316ce094.
This CL differs from the original CL by not unzipping RADIO/*. This is
because: a) AOSP targets don't have RADIO/ entries in the TF.zip; b)
we're not using the unzipped RADIO files (but reading them from the zip
files directly) - checked all the device-specific releasetools for
angler, bullhead, ryu, shamu, volantis, fugu, marlin and sailfish.
Test: `m dist` with AOSP targets (tested fugu and bullhead).
Change-Id: I4d0c67214ddd6202fc27c71bb79f52b5f4d40c64
This reverts commit aa3a04f19d.
Reason for revert: Some AOSP targets don't include RADIO/ in the TF.zip. We may possibly skip unzipping RADIO/, or by always creating a dummy RADIO folder in TF.zip. Revert this CL for now.
Change-Id: I8e90d322706a4fe82129bdfab5ffa1eab989c648
When building BBOTAs, it only needs *some* unzipped entries in the given
target_files zip(s). In particular, it needs 'IMAGES/*', 'META/*',
'RADIO/*'. (It also reads 'SYSTEM/build.prop' and 'OTA/bin/updater', but
directly from the zip file.)
This CL specifies the entries to unzip. It saves the I/O cost, as well as
the temporary storage.
Test: ota_from_target_files.py gives the same package w/ and w/o the CL.
Test: check_target_files_signatures.py still works.
Change-Id: I728428aa0e138879e49f9efbdb46a85892fc7038
We use the timestamps in builds to determine a downgrade, which might
not be always the truth. For examples, two builds cut from different
branches may carry timestamps in a reverse order. An incremental package
won't be able to be pushed nor applied, based on the timestamp
comparison.
We used to handle such a case with manual work, by setting the
post-timestamp to (pre-timestamp + 1) in the package metadata. This CL
automates the process by adding a new flag --override_timestamp.
Note that it doesn't change anything in the installed image, but only
affects the assertions for pushing / installing the package.
With the change in this CL:
- If it's a downgrade without any extra flag, fail the package
generation (we only print warnings prior to this CL);
- If it's a downgrade with --downgrade flag, generate a downgrade
package with forced data wipe (same as before);
- If it's a downgrade with --override_timestamp, generate a normal
incremental with hacked timestamp (pre-timestamp + 1) (new in this CL
to avoid the manual change);
- If it's not a downgrade but with any of the above two flags specified,
fail the package generation.
Bug: 33744169
Test: Generate an incremental from builds with reversed timestamps.
Change-Id: I8b187d32708b4a7c3e20f8c6adb8f9527b73b965
This CL changes the --oem_settings flag to allow a comma seperated list of
property files. All property values will be used when asserting properties such
as ro.product.name.
For example, if two property files are provided with ro.product.name values of
"sprout" and "sprout_a", the resulting otapackage will check that the device's
ro.product.name property matches at least one of them.
Bug: 34191373
Test: manual
Change-Id: I954673511be8f0929982235cc9cbfbd85a9ee1f4
We already support generating downgrade OTAs for non-A/B devices (with
mandatory data wipe), but we have missed the --downgrade flag in A/B OTA
path.
This CL factors out the function that writes the downgrade metadata, and
fixes the path for generating A/B OTAs.
Bug: 35094540
Test: Generate incrementals with --downgrade for A/B and non-A/B OTAs.
Change-Id: I30b9bf83e69e8aba3be666507681b555db6ab743
For streaming OTAs, we will also need the info in the metadata entry
(META-INF/com/android/metadata). Compute and pack its offset/length
values into 'ota-streaming-property-files'.
Bug: 34986195
Test: Create an OTA package and check the offset/length values.
Change-Id: Id150700f2bc9bff02467cda9fe8927c8a374412a
'streaming-property-files' is a property related to the OTA package
itself. Prepend 'ota-' to make it consistent with others like
'ota-type' and 'ota-required-cache'.
Bug: 34852392
Test: Generate an A/B OTA package and check METADATA entry.
Change-Id: Ia681e6e19ff509e6da0d8718933b42aac997e1cf
This reverts commit ea4325baf8 to re-land
commit ef1bb4360f. It fixes the bug when
handling a package without care_map.txt (e.g. dm-verity not enabled).
In order to support streaming A/B OTA packages, we pack
payload_properties.txt and care_map.txt in ZIP_STORED mode. These two
entries along with payload.bin (already in ZIP_STORED prior to this CL)
can be fetched directly based on the offset and length info.
We write the offset and length info into the package metadata entry
(META-INF/com/android/metadata), which can be parsed by the OTA server.
payload_properties.txt and care_map.txt are usually less than 1-KiB. So
the change only incurs marginal size increase.
Bug: 33382114
Test: Generate an A/B OTA package. Verify the 'streaming-property-files'
entry in the metadata file.
Test: Generate an A/B OTA package on a device with dm-verity not enabled.
Change-Id: I3469c8b62385a1fc58b4fb82e3f9d4690aef52ba
In order to support streaming A/B OTA packages, we pack
payload_properties.txt and care_map.txt in ZIP_STORED mode. These two
entries along with payload.bin (already in ZIP_STORED prior to this CL)
can be fetched directly based on the offset and length info.
We write the offset and length info into the package metadata entry
(META-INF/com/android/metadata), which can be parsed by the OTA server.
payload_properties.txt and care_map.txt are usually less than 1-KiB. So
the change only incurs marginal size increase.
Bug: 33382114
Test: Generate an A/B OTA package. Verify the 'streaming-property-files'
entry in the metadata file.
Change-Id: I04504e834eb36e18876c5f5a5a09289ee05c6f9a
It was added in commit 96be7205dc
("Working ASLR implementation.") in 2010, and removed in commit
1807e700a5 ("don't generate retouch
commands in OTA scripts") in 2012.
Remove the obsolete --aslr_mode flag.
Test: ota_from_target_files.py still works (by generating incremental
and full OTAs respectively).
Change-Id: I6d8e62730ac192f3574d484c4a4b9b43b4ee0a9e
Treehugger Robot