We have a couple of active imgdiff workarounds (and likely with one more
inbounding that allows having shared blocks in ext4 image). Most of
these workarounds need extending imgdiff's capability. While us not
getting there anytime soon, collect the stats to better understand the
impact of each kind so we can prioritize accordingly.
A sample report is as follows.
Imgdiff Stats Report
========================
APK files diff'd with imgdiff (count: 88)
-------------------------------------------
/system/priv-app/Shell/Shell.apk
...
Large APK files split and diff'd with imgdiff (count: 4)
----------------------------------------------------------
/system/priv-app/Settings/Settings.apk
...
Bug: 68016761
Test: Generate an incremental BBOTA package. Check the stats report.
Test: python -m unittest test_blockimgdiff
Change-Id: I27ad862cde472ab2806db877632ce5a0607420f2
In Transfer class, unbundle 'intact' with the monotonicity of the input
ranges. Negate the logic of 'intact', and thus rename it to 'trimmed'.
Move this property from an attribute of Transfer class as the one in
RangeSet.extra. 'trimmed' indicates whether the source / target ranges
have been modified after creating the Transfer() instance.
The logic that determines whether we can apply imgdiff has been
refactored and consolidated into BlockImageDiff.CanUseImgdiff(). Now
both of the two paths call this single copy, i.e. the one that detects
large APKs (before creating Transfer()'s), and the other that's about to
generate the patch for a given Transfer instance.
Bug: 68016761
Test: python -m unittest test_blockimgdiff
Test: Generate an incremental BBOTA package.
Change-Id: Id07195f63f1fa6c3af6e9091940d251cf09fa104
There is an internal buffer in bionic for user/group names that is 32
characters long including the trailing null, so we must restrict the
length of these names during compile time.
Bug: 27999086
Test: Successfully compile a valid config.fs
Test: Fail to compile a config.fs with AID name > 31 characters
Change-Id: I7fe887c630dd4d1033b86a5d8332480eb3b0fa07
These entries are vendor provided and belong on the /vendor partition.
Bug: 27999086
Test: end to end user/group check via config.fs and getpwnam, etc.
Change-Id: I9a5d56da594bf0d04de2b9ce7fd7d9a8151d4682
'monotonic' has been non-optional since [1] (L-MR1). Fix the comment in
RangeSet.parse(), as well as the use in blockimgdiff.py.
[1] commit 8b72aefb5a.
Test: Generate an incremental BBOTA package.
Change-Id: I7f95231683473b4f0f07f9c83fccc0e36a1340cb
Prior to this CL, the call to delta_generator in
check_ota_package_signature.VerifyAbOtaPayload() didn't redirect stderr.
The logs (mostly INFO) on successful verification added noise to the
normal output, which also upset the unittest result parser.
This CL captures stderr outputs from delta_generator, and will only dump
them on error.
Bug: 72884343
Test: `python -m unittest -v test_ota_from_target_files > /dev/null`
gives clean output.
Test: Inject error into delta_generator. The call to
check_ota_package_signature correctly dumps both of stdout and
stderr outputs.
Change-Id: I014a4b21bf758dcf0a4b9963259d6019851935ee
By default, an A/B OTA package doesn't contain the images for the
secondary slot (e.g. system_other.img). Specifying
"--include_secondary" that's introduced in this CL allows generating
a separate payload that will install secondary slot images. Both
payloads will be added to the generated A/B OTA package.
An example A/B OTA package with secondary payload
|
+-- payload.bin
|
+-- payload_properties.txt
|
+-- secondary/payload.bin
|
+-- secondary/payload_properties.txt
|
+-- ...
Such a package needs to be applied in a two-stage manner. During the
first stage, the updater applies the primary payload only. Upon
finishing, it reboots the device into the newly updated slot. It then
continues to install the secondary payload to the inactive slot, but
without switching the active slot at the end (needs the matching support
in update_engine, i.e. SWITCH_SLOT_ON_REBOOT flag).
Due to the special install procedure, the secondary payload will be
always generated as a full payload.
Bug: 35724498
Test: Generate full and incremental OTAs with --include_secondary. Check
the generated OTAs.
Test: python -m unittest test_ota_from_target_files
Change-Id: I975e826bec492e86eb400f99de0c355a32420127
When calling 'openssl x509 -pubkey' to extract the public key from a
certificate, openssl 1.0 and 1.1 handle the '-out' parameter
differently. openssl 1.0 doesn't write the output into the specified
filename, which leads to the payload verification failure in
check_ota_package_signature.VerifyAbOtaPayload(). This CL addresses
the issue by always collecting the output from stdout instead.
It also refactors the two copies into common.ExtractPublicKey(), and
adds unittest. get_testdata_dir() is moved into test_utils.py that holds
common utils for running the unittests.
Bug: 72884343
Test: python -m unittest test_common
Test: python -m unittest test_ota_from_target_files
Test: Run sign_target_files_apks with '--replace_ota_keys' on marlin
target_files zip. Check the payload pubkey replacement.
Test: Trigger the tests with forrest, and tests no longer fail on
machines with openssl 1.0.1.
Change-Id: Ib0389b360f064053e9aa7cc0546d718e7b23003b
This breaks down the current WriteABOTAPackageWithBrilloScript() into
smaller and testable units, which also prepares for the work in
b/35724498.
Bug: 35724498
Test: python -m unittest test_ota_from_target_files
Test: Get identical A/B OTA packages w/ and w/o the CL.
Change-Id: I2ea45ce98e2d2baa58e94fb829b7242f6fe685a7
Merged-In: I2ea45ce98e2d2baa58e94fb829b7242f6fe685a7
(cherry picked from commit 036d721812)
The same thing has been set in GetPackageMetadata(), so they are no-op.
Also replace an occurrence of 'OPTIONS.source_info_dict' with
source_info, which has been missed from previous clean-up CLs.
Test: Generate an A/B OTA package, and an incremental BBOTA with
--wipe_user_data. Check the metadata.
Test: Generate a two-step incremental BBOTA successfully.
Change-Id: I4bb491cac9064d93fb86d12e617c8f38f040e01e
The info comes from the build.prop file of the target build (thus no
backward compatibility concerns). OTA server and client can optionally
use these info to understand the expected behavior of an OTA package.
Bug: 72751683
Test: python -m unittest test_ota_from_target_files
Test: `m dist`, then check the metadata in the generated OTA package.
Change-Id: I5935f67684d2486bb5f00d67ce4bc756589a56ed
This CL detects incomplete block ranges (e.g. due to the holes in
mke2fs created images). Such block ranges will be tagged, so we won't
attempt to imgdiff those files. Note that the change to blockimgdiff.py,
which uses the tag info, will come in a separate CL.
An 'extra' attribute is added to RangeSet class, which defaults to an
empty dict. An 'incomplete' tag will be added into the dict by the
caller of the class. Not adding this tag as an immediate attribute,
because it is not a property regarding the ranges being represented, but
rather some storage space for the caller.
This CL also refactors GetSparseImage and RoundUpTo4K into common.py, so
the same code can be called from both of ota_from_target_files.py and
validate_target_files.py. Not able to add unittests for
GetSparseImage(), as SparseImage requires data in specific format.
Bug: 68016761
Test: Run validate_target_files.py on target-files.zip. It skips
validating files with missing holes as before.
Test: Run ota_from_target_files.py on angler target-files.zip. It gives
identical packages w/ and w/o the CL.
Test: pylint on changed files. There're warnings with common.py, but
unrelated to this change.
Change-Id: I126ccfea13c0d5ebcc8c1b4ff1a4f9200e97423a
C: 73, 0: Wrong hanging indentation (add 4 spaces).
file_name, actual_sha1, expected_sha1)
^ | (bad-continuation)
C:171, 0: Wrong continued indentation (add 20 spaces).
'SYSTEM/etc/recovery.img', expected_recovery_sha1)
^ | (bad-continuation)
C:185, 0: Wrong continued indentation (add 20 spaces).
file_path='IMAGES/boot.img', expected_sha1=boot_info[3])
^ | (bad-continuation)
C:191, 0: Wrong continued indentation (add 20 spaces).
file_path='IMAGES/recovery.img',
^ | (bad-continuation)
C:192, 0: Wrong continued indentation (add 20 spaces).
expected_sha1=expected_recovery_sha1)
^ | (bad-continuation)
W: 67,15: Use % formatting in logging functions and pass the % parameters as arguments (logging-format-interpolation)
W:150,17: Use % formatting in logging functions and pass the % parameters as arguments (logging-format-interpolation)
W:153,15: Use % formatting in logging functions and pass the % parameters as arguments (logging-format-interpolation)
W:194,15: Use % formatting in logging functions and pass the % parameters as arguments (logging-format-interpolation)
C: 27, 0: standard import "import logging" comes before "import common" (wrong-import-order)
C: 28, 0: standard import "import os.path" comes before "import common" (wrong-import-order)
C: 29, 0: standard import "import re" comes before "import common" (wrong-import-order)
C: 31, 0: standard import "import sys" comes before "import common" (wrong-import-order)
Test: pylint --rcfile=pylintrc validate_target_files.py
Test: Run validate_target_files.py with a target-files.zip.
Change-Id: Ie64acdb4cee4326938c4ad5a34b575d7b82478c0
We used to do this in add_img_to_target_files.AddImagesToTargetFiles(),
which didn't cover the path when calling from make_recovery_patch. As a
result, /system/bin/install-recovery.sh contains different SHA values
from the actual images.
Test: Set up aosp_bullhead to use AVB. `m dist`, then run the following
command to verify the generated install-recovery.sh.
$ ./build/make/tools/releasetools/validate_target_files.py \
out/dist/aosp_bullhead-target_files-eng.zip
Change-Id: Id7be8fb17072252fcd4d08db2057b8c4af053376
The generator function is not thread safe and is prone to race
conditions. This CL uses a lock to protect this generator and loose the
locks elsewhere, e.g. 'WriteRangeDataToFd()'.
Bug: 71908713
Test: Generate an incremental package several times for angler 4208095 to 4442250.
Change-Id: I9e6f0a182a1ba7904a597f403f2b12fe05016513
Check that the Sha1 for src&tgt ranges are correct before computing
patches. This adds ~6 seconds overhead for ~2400 commands.
Bug: 71908713
Test: Generate an incremental package from angler 4208095 to 4442250.
Change-Id: I8cf8ce132fb09a22f7d6689274ddb4a27770be76
bsdiff/imgdiff returns non-zero values correctly in case of failures.
So we don't need to check the stderr anymore. This avoids some false
reports of the diff program.
Bug: 72335938
Bug: 71505046
Test: Check the exit value of bsdiff/imgdiff in code search,
generate a package for angler.
Change-Id: I18f0c3882a40a5288d6aee715713a05270e0db2b
Create a wrapper class that handles the payload signing, which unifies
the paths with and without external signer. Also add tests for the newly
added class.
The test keys under testdata/ are created with the script in
development/tools/make_key. testdata/testkey_with_passwd.pk8 uses
password "foo".
Test: python -m unittest test_ota_from_target_files
Test: Get identical A/B OTA packages w/ and w/o the CL.
Change-Id: Ic770aec726498a3babb88ec509985e7f1210fb18
The output (stdout and stderr combined) would be helpful in understanding
the cause of the failure.
Not changing other occurrences in build_image.py yet, since it's less
obvious if a failed call should be considered fatal (thus whether to
dump such outputs). For example, GetVeritySize() simply returns 0 on
failed calls. They deserve further clean-ups in later CLs (with unit
tests).
Bug: 71864688
Test: Inject errors into mke2fs/e2fsck/simg2img. `m snod` prints outputs.
Test: `m snod` remains quiet on successful runs.
Change-Id: I172403f6cd05bce93767265dbcb110271a68e2a8
This CL makes the following changes:
(a) cleans up the similar codes in AddCareMapTxtForAbOta() that handle
'system' and 'vendor' partitions;
(b) fixes an issue with the arcname in AddPackRadioImages() and
AddRadioImagesForAbOta(), where forward slash should always be used
in zip entry names;
(c) refactors the branching statements in AddRadioImagesForAbOta() to
reduce indentation levels.
Test: python -m unittest test_add_img_to_target_files
Test: `m dist` with aosp_marlin-userdebug. Check META/care_map.txt in the
generated target_files.zip.
Change-Id: I3d6f794962d0c68390fbd18eb13c2622acab3ff5
Separate three functions out of AddImagesToTargetFiles(), into
AddCareMapTxtForAbOta(), AddRadioImagesForAbOta() and
AddPackRadioImages() respectively. This CL tries to apply minimal
changes for the refactoring purpose.
Also add tests for AddRadioImagesForAbOta() and AddPackRadioImages().
The tests for AddCareMapTxtForAbOta() require better testing support to
mock sparse_img.SparseImage, which will be added in later CLs.
Test: python -m unittest test_add_img_to_target_files
Test: `m dist` with aosp_marlin-userdebug. Check META/care_map.txt in
the generated target_files.zip.
Change-Id: I1bb723c15237ff721f165cfce0ce996008ce9948
This was modified in commits 20a8664 "Change canonical name for
GuardedByChecker to GuardedBy" and 1f6dc3dc "Merge GuardedByValidator
into GuardedByChecker."
Test: m
Test: manual
Change-Id: I07305e59694a955e52bd69c1cbe53346808b78aa
Now they (A/B, non-A/B full and non-A/B incremental) share the same
code. Also add tests for the newly added function.
Test: python -m unittest test_ota_from_target_files
Test: Build A/B and non-A/B packages respectively.
Change-Id: I5499dd9c683ac1a18d8a24f21ff8845fe69f343e
This helps to generate a deterministic package.
Bug: 71770360
Test: Generate a incremental package and transfers are added by file name.
Change-Id: I7562a200b97a1babbb09a77801324cc9408cc01f
In non-A/B OTA path, we've been loading the info dict for the target
build twice (once to have a peek at the OTA type, with a reload after
unzipping the target-files zip). Remove the reloading.
This CL moves the loading of the source info dict up a bit to avoid
having two copies in the A/B and non-A/B paths. It also moves up the
sanity checks of the loaded info dicts before unzipping the target-files
zips.
Test: Generate full and incremental packages for bullhead and marlin.
Change-Id: Iccb953feb0b6ecf62e6f73d6e9ceea00077d098d
GetBuildProp() and GetVendorBuildProp() have been factored into
BuildInfo class. These two functions were meant to be deleted in commit
481bab8d48.
Test: python -m unittest test_ota_from_target_files
Test: No users of these two functions.
Change-Id: Id590a46d597939973ef2339cdfea4f5163548d4d
Tao Bao