The test broke due to api change in constructor of Transfer.
Bug: 36456381
Test: blockimgdiff test passes
Change-Id: Ic73b66f8fb6141ff6f50c113b0d8cc55d34e0b5d
We've just created the target files package and still have the extracted
directory, so pass that to ota_from_target_files so that it doesn't have
to re-extract what we just compressed.
This saves a little bit of time -- for bullhead-userdebug on internal
master, this brings the time to build an OTA from ~340s -> ~310s. Much
of the time is still spent generating and signing the OTA.
Test: lunch bullhead-userdebug; m -j otapackage
Test: bullhead-ota-*.zip is identical before/after
Change-Id: Ib51cf6b02123e9c49280e9957c6590645470a52c
system_root_image has been dead since commit
2ce63edab7.
Test: pylint --rcfile=pylintrc add_img_to_target_files.py
Change-Id: Id791747b7313923b82279a21d264a998455a92d4
When bsdiff/imgdiff fails, dump the output along with the src/tgt
filenames and block ranges. Also, exit the script gracefully instead
of raising an exception about iterating over 'NoneType'.
Bug: 31381530
Test: Package generates successfully for angler, and error outputs
correctly with error injection.
Change-Id: I06a2dfe545fbdff7043de05fee34b378453a9291
The line was unintentionally removed in commit 7f804ba.
Test: ota_from_target_files.py generates a full OTA. Check the package
metadata.
Change-Id: Icae88e2a9bb2bfc450a3d0d7ab524d6a6eac9df5
BBOTA v1 and v2 (introduced in L and L MR1 respectively) don't support
resumable OTA. We shouldn't generate packages using v1/v2 at the risk of
bricking devices.
BBOTA v3 (since M) and v4 (since N) both support resumable OTAs. BBOTA
v4 additionally supports using FEC to possibly recover a corrupted
image.
Bug: 33694730
Test: Generate full and incremental OTAs w/ and w/o the CL. They should
give identical packages (in v4).
Change-Id: Ib89d9cd63ba08e8e9aa4131bed18876b89d244c0
Remove the following functions that are needed for file-based OTAs only:
- SetPermissions()
- SetPermissionsRecursive()
- MakeSymlinks()
- DeleteFiles()
- DeleteFilesIfNotMatching()
- RenameFiles()
- SkipNextActionIfTargetExists()
Bug: 35853185
Test: Verified there's no reference to these functions.
Change-Id: Iff24a9d705476211effaef28eed2a147fa5fcbce
In addition to the current behavior of add_img_to_target_files working
on an existing zip file, allow passing in a directory where the target
files have already been extracted. When in this mode, it writes the
images out to that directory instead of the zip file.
This allows us to call add_img_to_target_files on the temp directory
used during the build to create the target files package, saving the
time and space of unzipping what we just zipped. This also allows us to
use the parallel soong_zip, which compresses the images much faster.
Test: aosp_marlin target_files zip is the same before/after this change
Test: marlin target_files zip is the same before/after this change
Test: bullhead target_files zip is the same before/after this change
Change-Id: I155654cdc7ad7754ba4ef76ec69c31f504a58198
We have stopped shipping devices with file-based OTAs, and are not
actively maintaining the support. Devices using file-based OTAs
should be moved to block-based, if not A/B OTAs.
We will also need to clean up EdifyGenerator class, which will be
handled in follow-up CLs.
Bug: 35853185
Test: Generate full and incremental OTAs w/ and w/o the CL, and they
give identical packages.
Test: Not specifying --block also generates block-base OTAs.
Change-Id: I3b0fc8ce5600e109f3251fe41f655534aaa298c7
We must have created the images (system.img, system.map etc) prior to
calling ota_from_target_files.py (added by commit
2e0d8fcf08, into Lollipop).
Remove the obsolete suppport for handling "old" target_files zips that
don't have such images. This simplies the logic for BuildSystem() and
BuildVendor(), which now would only be called by
add_img_to_target_files.py itself.
Test: Generating full and incremental OTAs give the same results, w/ and
w/o this CL.
Change-Id: I0ea243d09d3378249d7982701ae4ec339b04b7b8
The first one in WriteVerifyPackage() is mismatching function parameters
that can be trivially fixed.
The other one is in WriteABOTAPackageWithBrilloScript(), where we don't
have edify script instance.
Test: `pylint --rcfile=pylintrc ota_from_target_files.py`.
Change-Id: Ie238ef5b296dfe9e725b61376992447b662d2376
The major issue with the existing implementation is unnecessarily
holding too much data in memory, such as HashBlocks() which first reads
in *all* the data to a list before hashing. We can leverage generator
functions to stream such operations.
This CL makes the following changes to reduce the peak memory use.
- Adding RangeSha1() and WriteRangeDataToFd() to Image classes. These
functions perform the operations on-the-fly.
- Caching the computed SHA-1 values for a Transfer instance.
As a result, this CL reduces the peak memory use by ~80% (e.g. reducing
from 5.85GB to 1.16GB for the same incremental, as shown by "Maximum
resident set size" from `/usr/bin/time -v`). It also effectively
improves the (package generation) performance by ~30%.
Bug: 35768998
Bug: 32312123
Test: Generating the same incremental w/ and w/o the CL give identical
output packages.
Change-Id: Ia5c6314b41da73dd6fe1dbe2ca81bbd89b517cec
This reverts commit a7316ce094.
This CL differs from the original CL by not unzipping RADIO/*. This is
because: a) AOSP targets don't have RADIO/ entries in the TF.zip; b)
we're not using the unzipped RADIO files (but reading them from the zip
files directly) - checked all the device-specific releasetools for
angler, bullhead, ryu, shamu, volantis, fugu, marlin and sailfish.
Test: `m dist` with AOSP targets (tested fugu and bullhead).
Change-Id: I4d0c67214ddd6202fc27c71bb79f52b5f4d40c64
This reverts commit aa3a04f19d.
Reason for revert: Some AOSP targets don't include RADIO/ in the TF.zip. We may possibly skip unzipping RADIO/, or by always creating a dummy RADIO folder in TF.zip. Revert this CL for now.
Change-Id: I8e90d322706a4fe82129bdfab5ffa1eab989c648
When building BBOTAs, it only needs *some* unzipped entries in the given
target_files zip(s). In particular, it needs 'IMAGES/*', 'META/*',
'RADIO/*'. (It also reads 'SYSTEM/build.prop' and 'OTA/bin/updater', but
directly from the zip file.)
This CL specifies the entries to unzip. It saves the I/O cost, as well as
the temporary storage.
Test: ota_from_target_files.py gives the same package w/ and w/o the CL.
Test: check_target_files_signatures.py still works.
Change-Id: I728428aa0e138879e49f9efbdb46a85892fc7038
This remove the fstab dependency when building the OTA package for
marlin/sailfish.
Bug: 35811655
Test: OTA package builds successfully for sailfish.
Change-Id: If223d11dddca396c47262042c576f9e7d0cb5b33
(cherry picked from commit 7d051adc3b)
fstab_version is defined by RECOVERY_FSTAB_VERSION in
bootable/recovery. We have moved to fstab_version 2 since commit
f35d1cef7c19db975a1295e8c23c7fb8bd2489f9 (landed into JB MR2).
Drop the support for fstab_version 1, since we won't run the latest OTA
script over a JB target_files zip.
Test: No impact on building full/incremental OTAs.
Change-Id: Ia87c4e7da6c5e71ce0908fca2e4f1ad1c06ba869
The merged two branches have become identical since commit
fc3422ad36 (landed into Nougat).
Test: Get identical incremental packages w/ and w/o the CL.
Change-Id: Id1183f8ed83f684a0dac1a4af87b6e075b08aabc
We use the timestamps in builds to determine a downgrade, which might
not be always the truth. For examples, two builds cut from different
branches may carry timestamps in a reverse order. An incremental package
won't be able to be pushed nor applied, based on the timestamp
comparison.
We used to handle such a case with manual work, by setting the
post-timestamp to (pre-timestamp + 1) in the package metadata. This CL
automates the process by adding a new flag --override_timestamp.
Note that it doesn't change anything in the installed image, but only
affects the assertions for pushing / installing the package.
With the change in this CL:
- If it's a downgrade without any extra flag, fail the package
generation (we only print warnings prior to this CL);
- If it's a downgrade with --downgrade flag, generate a downgrade
package with forced data wipe (same as before);
- If it's a downgrade with --override_timestamp, generate a normal
incremental with hacked timestamp (pre-timestamp + 1) (new in this CL
to avoid the manual change);
- If it's not a downgrade but with any of the above two flags specified,
fail the package generation.
Bug: 33744169
Test: Generate an incremental from builds with reversed timestamps.
Change-Id: I8b187d32708b4a7c3e20f8c6adb8f9527b73b965
We introduced META/misc_info.txt to hold the misc info since Gingerbread
(commit 37974731fc). Remove the backwards
compatibility support for building pre-G TF zips.
Test: `m dist` works.
Change-Id: Ibff7aaf69cc7e460634c049d11a004f7196f8f73
Otherwise the comparison is inconsistent between ReviseStashSize() and
WriteTransfers().
Bug: 35775675
Test: Successfully generate a previously failed incremental.
Change-Id: I554a51a210bf322cb5c79e28cf85607a417b094a
This CL changes the --oem_settings flag to allow a comma seperated list of
property files. All property values will be used when asserting properties such
as ro.product.name.
For example, if two property files are provided with ro.product.name values of
"sprout" and "sprout_a", the resulting otapackage will check that the device's
ro.product.name property matches at least one of them.
Bug: 34191373
Test: manual
Change-Id: I954673511be8f0929982235cc9cbfbd85a9ee1f4
Commit e98fb7a8d3 switched to using
futility-host instead of the prebuilt futility. This CL adds support to
handle signing old TF.zip that still says "futility=prebuilt/..." in
META/misc_info.txt.
Bug: 35467608
Test: Generate otatools.zip and sign an old ryu TF.zip.
Change-Id: I48a9cc918c7afce361e1ec9bc4f85f74fa92566e
We check the needed stash size in ReviseStashSize(), and may not
generate a stash command if it would exceed the max allowed size. This
CL fixes a bug when skipping a stash operation: we shouldn't update the
'stashes' map if a stash command won't be generated.
Bug: 35313668
Test: Successfully generate the package that was failing due to the bug.
Change-Id: If0a3a5fadda9b4a4edad66a2a5826b5f978400ae
We already support generating downgrade OTAs for non-A/B devices (with
mandatory data wipe), but we have missed the --downgrade flag in A/B OTA
path.
This CL factors out the function that writes the downgrade metadata, and
fixes the path for generating A/B OTAs.
Bug: 35094540
Test: Generate incrementals with --downgrade for A/B and non-A/B OTAs.
Change-Id: I30b9bf83e69e8aba3be666507681b555db6ab743
Commit f1a13180db intended to remove the
verity blocks from care_map.txt, but it added new code without removing
the old one. This leads to a malformed care_map.txt and causes
update_verifier failure.
Bug: 34391662
Test: 'm -j dist' gives a TF.zip with 4-line META/care_map.txt (as
opposed to a 6-line file).
Change-Id: I7ff1aa525795c4b049af54c1755b0f0ea84f7e0e
For streaming OTAs, we will also need the info in the metadata entry
(META-INF/com/android/metadata). Compute and pack its offset/length
values into 'ota-streaming-property-files'.
Bug: 34986195
Test: Create an OTA package and check the offset/length values.
Change-Id: Id150700f2bc9bff02467cda9fe8927c8a374412a
'streaming-property-files' is a property related to the OTA package
itself. Prepend 'ota-' to make it consistent with others like
'ota-type' and 'ota-required-cache'.
Bug: 34852392
Test: Generate an A/B OTA package and check METADATA entry.
Change-Id: Ia681e6e19ff509e6da0d8718933b42aac997e1cf
When reading /dev/block/dm-X, update_verifier isn't able to access the
verity meta blocks at the end of the system/vendor partition. So we need
to remove these block ranges from the care_map.
Bug: 34391662
Test: care_map generated successfully without verity meta blocks
Change-Id: Id57c602b7e5fd1b0c9d1e1fe5fcdd74e85b6b255
This reverts commit ea4325baf8 to re-land
commit ef1bb4360f. It fixes the bug when
handling a package without care_map.txt (e.g. dm-verity not enabled).
In order to support streaming A/B OTA packages, we pack
payload_properties.txt and care_map.txt in ZIP_STORED mode. These two
entries along with payload.bin (already in ZIP_STORED prior to this CL)
can be fetched directly based on the offset and length info.
We write the offset and length info into the package metadata entry
(META-INF/com/android/metadata), which can be parsed by the OTA server.
payload_properties.txt and care_map.txt are usually less than 1-KiB. So
the change only incurs marginal size increase.
Bug: 33382114
Test: Generate an A/B OTA package. Verify the 'streaming-property-files'
entry in the metadata file.
Test: Generate an A/B OTA package on a device with dm-verity not enabled.
Change-Id: I3469c8b62385a1fc58b4fb82e3f9d4690aef52ba
In order to support streaming A/B OTA packages, we pack
payload_properties.txt and care_map.txt in ZIP_STORED mode. These two
entries along with payload.bin (already in ZIP_STORED prior to this CL)
can be fetched directly based on the offset and length info.
We write the offset and length info into the package metadata entry
(META-INF/com/android/metadata), which can be parsed by the OTA server.
payload_properties.txt and care_map.txt are usually less than 1-KiB. So
the change only incurs marginal size increase.
Bug: 33382114
Test: Generate an A/B OTA package. Verify the 'streaming-property-files'
entry in the metadata file.
Change-Id: I04504e834eb36e18876c5f5a5a09289ee05c6f9a
Tianjie Xu