Commit Graph

5211 Commits

Author SHA1 Message Date
Tao Bao be8ea6ec11 Merge "releasetools: Remove the dead code for blockimgdiff_version <= 2." 2017-12-21 22:20:32 +00:00
Dan Shi 3c28ff7bff Merge "Auto generate test config files" 2017-12-21 21:53:12 +00:00
Tao Bao 0582cb639f releasetools: Remove the dead code for blockimgdiff_version <= 2.
The code is on infeasible path since we already have assertions in
common.BlockDifference().

Also remove the dead code that checks for OPTIONS.info_dict, as we
already set that in ota_from_target_files.main(), for both of A/B and
non-A/B.

Test: Generate incremental OTAs w/ and w/o the CL, and get identical
      packages.
Change-Id: Ifb8fc101e78f5ce58c60c8e49028b66ce0d20246
2017-12-21 11:53:27 -08:00
Tao Bao 2f5358d38b releasetools: Fix the bug that skips loading OEM dict.
The CL in [1] unintentionally breaks the OEM dict loading logic in the
incremental BBOTA path. We should always require and load the OEM
property dict if _either_ of the source and target builds uses OEM
properties. Otherwise with the current "and" operator, it skips loading
the OEM property dict and thus fails to generate an OTA package that has
OEM property changes (e.g. updating from build with fingerprint to
another one using thumbprint).

The CL in [1] actually makes the right change in the file-based OTA
path, but introduces the bug in the block-based OTA path.

This CL also cleans up the line that reads recovery_mount_options.

[1] commit 7f804ba71f ("releasetools:
allow for multiple OEM property values.").

Test: Genearte an OTA that has OEM property changes successfully.
Change-Id: Idce4ad59825d432618535ce09ab22bd7ddc524f2
2017-12-20 23:06:02 -08:00
Dan Shi efb892d858 Auto generate test config files
This change allows one to run atest <module name>. The TradeFed test
config file will be automatically created in testcases directory based
on the settings.

Bug: 69929803, 69932044
Test: m -j NotificationStressTests libminijail_test
add following lines in notificationtests/Android.mk,
before "include $(BUILD_PACKAGE)"
LOCAL_COMPATIBILITY_SUITE := general-test

Change-Id: Ib5e9cd152d957b5d5beb6ed23a899a021a7e06d6
2017-12-20 15:05:17 -08:00
Treehugger Robot b379762089 Merge "build_image: Factor out CheckHeadroom() and add tests." 2017-12-19 00:00:41 +00:00
Treehugger Robot b08567cbb6 Merge "releasetools: Fix the tag replacement for ro.build.vendor.fingerprint." 2017-12-18 22:18:14 +00:00
Treehugger Robot d30b75190d Merge "Remove the support for BOARD_HAS_EXT4_RESERVED_BLOCKS." 2017-12-18 22:17:46 +00:00
Treehugger Robot e58d385df1 Merge "releasetools: Use delta_generator to verify payload signatures." 2017-12-15 23:38:12 +00:00
Tao Bao 750385e455 releasetools: Use delta_generator to verify payload signatures.
We used to take a hard approach by parsing the payload with Python
script. This can be done by calling deleta_generator directly, which
also avoids the dependency on protobuf.

- Passing case
$ ./build/make/tools/releasetools/check_ota_package_signature.py \
    build/target/product/security/testkey.x509.pem \
    out/dist/aosp_marlin-ota-eng.zip

Package: out/dist/aosp_marlin-ota-eng.zip
Certificate: build/target/product/security/testkey.x509.pem
  ...
Whole package signature VERIFIED

Verifying A/B OTA payload signatures...
[1215/122842:INFO:generate_delta_main.cc(171)] Verifying signed payload.
[1215/122845:INFO:payload_verifier.cc(93)] signature blob size = 264
[1215/122845:INFO:payload_verifier.cc(112)] Verified correct signature 1 out of 1 signatures.
[1215/122845:INFO:payload_verifier.cc(93)] signature blob size = 264
[1215/122845:INFO:payload_verifier.cc(112)] Verified correct signature 1 out of 1 signatures.
[1215/122845:INFO:generate_delta_main.cc(181)] Done verifying signed payload.

Payload signatures VERIFIED

$ echo $?
0

- Failing case
Sign the whole package file with a different key, but leaving payload entries intact.

$ ./build/make/tools/releasetools/check_ota_package_signature.py \
    testkey2.x509.pem \
    marlin-ota-mismatching.zip
Package: marlin-ota-mismatching.zip
Certificate: testkey2.x509.pem
  ...
Whole package signature VERIFIED

Verifying A/B OTA payload signatures...
[1215/123054:INFO:generate_delta_main.cc(171)] Verifying signed payload.
[1215/123056:INFO:payload_verifier.cc(93)] signature blob size = 264
[1215/123056:ERROR:payload_verifier.cc(118)] None of the 1 signatures is correct. Expected:
[1215/123056:INFO:utils.cc(444)] Logging array of length: 256
[1215/123056:INFO:utils.cc(461)] 0x00000000 : 00 01 ff ff ff ff ff ff ff ff ff ff ff ff ff ff
  ...
[1215/123056:ERROR:payload_verifier.cc(121)] But found decrypted hashes:
[1215/123056:INFO:utils.cc(444)] Logging array of length: 256
[1215/123056:INFO:utils.cc(461)] 0x00000000 : 52 68 78 36 f6 9e cd 2d 5e 9f 31 d5 26 03 c9 aa
  ...
[1215/123056:ERROR:payload_signer.cc(333)] PayloadVerifier::VerifySignature( signature_blob, public_key_path, payload_hash) failed.
[1215/123056:INFO:generate_delta_main.cc(177)] VerifySignedPayload failed

    ERROR: Failed to verify payload with delta_generator: marlin-ota-mismatching.zip

$ echo $?
1

Bug: 65261072
Test: See above.
Change-Id: Id2e065655ec49b80dd2b13c6a859f41913be055b
2017-12-15 12:45:09 -08:00
Tao Bao 0c6a414fff releasetools: Remove the support for generating verify package.
We used to have "--gen_verify" that would generate a verify package for
non-A/B devices. Since a) we don't have active users; b) it works with
non-A/B only; c) we have better alternatives (e.g. using fastboot to
compute the partition checksum), this CL removes the support.

Test: `m dist`
Change-Id: Ib4a2c8c2a0394b54e66c106089aa52a0b900f034
2017-12-15 10:20:52 -08:00
Tao Bao 23f7ea9e70 Merge "releasetools: Add common.ZipDelete()." 2017-12-15 16:36:55 +00:00
Tao Bao 89d7ab23e8 releasetools: Add common.ZipDelete().
We have been shelling out to 'zip -d' to delete existing ZIP entries in
add_img_to_target_files.py. This CL moves the function into common.py,
and calls that for the similar work in ota_from_target_files.py. This CL
also adds unittests for the newly added function.

Test: `m dist`
Test: python -m unittest test_common
Test: ota_from_target_files.py generates identical packages w/ and w/o
      the CL (so we know the streaming property computation is intact).
Test: Run 'add_img_to_target_files.py -a' that triggers a call to
      ReplaceUpdatedFiles().
Change-Id: Icaa6c3ea3ee2166023f78fa79275295f837ea842
2017-12-14 17:17:04 -08:00
Treehugger Robot 78b5724622 Merge "releasetools: Remove '-w' from ota_from_target_files options." 2017-12-14 22:18:56 +00:00
Treehugger Robot 973301af79 Merge "Replace build/tools with build/make/tools" 2017-12-13 02:34:42 +00:00
Tao Bao a7054eede1 releasetools: Fix the tag replacement for ro.build.vendor.fingerprint.
For devices using derived fingerprint (i.e. /system/build.prop doesn't
contain ro.build.fingerprint, but has ro.build.thumbprint instead), the
current code (in android.os.Build) doesn't have a matching logic to do
the same for ro.vendor.build.fingerprint. This means we will see
ro.build.thumbprint in /system/build.prop, while there's no matching
ro.vendor.build.thumbprint in /vendor/build.prop.

From signing script point of view, it should just apply the tag
replacement (e.g. test-keys -> release-keys) for whatever it sees when
signing a target_files.zip.

This CL also adds unit tests for EditTags() and RewriteProps().

Fixes: 27950003
Test: Use 'sign_target_files_apks.py' to sign a target that uses derived
      fingerprint and vendor partition. Check VENDOR/build.prop.
Test: python -m unittest test_sign_target_files_apks
Change-Id: I09019da970840cd82f54b68a32b4e94984bc1d8d
2017-12-08 15:19:47 -08:00
Tao Bao d4349f2106 build_image: Factor out CheckHeadroom() and add tests.
The test is mostly trivial, but it ensures the result parsing from
actual calls to mke2fs.

Test: python -m unittest test_build_image
Test: `m dist` on aosp_marlin-userdebug (w/ and w/o defining
      PRODUCT_SYSTEM_HEADROOM respectively).
Change-Id: I8b9964213950e76f6d7d5518414a1bab888b4706
2017-12-07 23:39:56 -08:00
Jaegeuk Kim 10c9f5d9d3 Merge "core, release: add sload.f2fs for otatools.zip" 2017-12-08 00:57:42 +00:00
Tao Bao 79d52f8798 Remove the support for BOARD_HAS_EXT4_RESERVED_BLOCKS.
This is only used in file-based OTA, where we may reserve space on
/system partition. With the deprecation of file-based OTA since O, the
code has become obsolete.

Test: `m dist`
Change-Id: I2bd686b292f93b566fe28a2bcd74c564ffc75dac
2017-12-07 14:36:21 -08:00
Treehugger Robot 9a33b96103 Merge "Explain why BuildImage failed." 2017-12-07 21:45:59 +00:00
Colin Cross fdea893833 Replace build/tools with build/make/tools
Test: m checkbuild
Change-Id: I8db6c27a6d2372f142d10568fe92d6f2e9a59252
2017-12-06 16:12:40 -08:00
Tao Bao 337633f2b9 releasetools: Remove '-w' from ota_from_target_files options.
'-w' is a short option for '--wipe_user_data', which happens to be the
same as the whole-file signing flag for signapk.

This CL removes the short version to make it less error-prone. Users
should use '--wipe_user_data' to generate packages with enforced data
wipe.

Bug: 70259309
Test: ota_from_target_files.py generates packages with and without
      '--wipe_user_data'.
Test: Not seeing active users that uses 'ota_from_target_files -w'.
Change-Id: Ib8c138663a87aafc6a0b5b6f7c286fb74ef6134b
2017-12-06 15:47:03 -08:00
Colin Cross 3277ba34ae Add java modules to soong_to_convert.txt
Test: examine soong_to_convert.txt
Change-Id: Ibb6db664238a4e98f5d8885dfb3ddc0f35d11ff6
2017-12-06 15:38:52 -08:00
Xin Li 5f03da0404 Merge "DO NOT MERGE: Merge Oreo MR1 into master" 2017-12-06 23:18:30 +00:00
Treehugger Robot 7b2129f101 Merge "AVB: If building recovery.img, add AVB hash footer." 2017-12-06 22:12:25 +00:00
Elliott Hughes 73ff57f48a Explain why BuildImage failed.
BuildImage can return false for lots of reasons. Log two of the more
plausible ones. You'll still need to edit this file and pass True to
RunCommand to work out *what* went wrong, but at least this logging will
tell you *where* things went wrong.

Bug: http://b/63142920
Test: debugged my actual problem
Change-Id: If84427058108dcac0a1b5e2216b8bb11eef522ca
2017-12-06 12:16:39 -08:00
Tao Bao 9ad68d6e69 Merge "releasetools: Address pylint warnings in add_img_to_target_files.py."
am: 32c6c4a11e

Change-Id: I4e1d8ad1c6982f969b759f26cf1cb59862bd236f
2017-12-05 22:02:40 +00:00
Jaegeuk Kim 2ea1eba98c core, release: add sload.f2fs for otatools.zip
This also updates mkf2fsuserimg.sh to call sload.f2fs.

Change-Id: I245f6b3dfcd966fbc6783329d781375879fcd065
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
2017-12-05 10:59:50 -08:00
Tao Bao ead6281575 Merge "releasetools: Remove deprecated pylint options."
am: 3694e104b7

Change-Id: I87a9d6b07dc281bacc99e2935f0835958c3d67b5
2017-12-05 18:45:56 +00:00
Tao Bao 6b9fef5243 releasetools: Address pylint warnings in add_img_to_target_files.py.
C:661, 0: Wrong hanging indentation (add 2 spaces).
        OPTIONS.input_tmp, "VENDOR_IMAGES")
        ^ | (bad-continuation)
C:673, 0: Wrong continued indentation (add 14 spaces).
                os.path.join("IMAGES", img_name))
                ^             | (bad-continuation)
C: 54, 0: Import "import datetime" should be placed at the top of the module (wrong-import-position)
C: 55, 0: Import "import hashlib" should be placed at the top of the module (wrong-import-position)
C: 56, 0: Import "import os" should be placed at the top of the module (wrong-import-position)
C: 57, 0: Import "import shlex" should be placed at the top of the module (wrong-import-position)
C: 58, 0: Import "import shutil" should be placed at the top of the module (wrong-import-position)
C: 59, 0: Import "import subprocess" should be placed at the top of the module (wrong-import-position)
C: 60, 0: Import "import tempfile" should be placed at the top of the module (wrong-import-position)
C: 61, 0: Import "import uuid" should be placed at the top of the module (wrong-import-position)
C: 62, 0: Import "import zipfile" should be placed at the top of the module (wrong-import-position)
C: 64, 0: Import "import build_image" should be placed at the top of the module (wrong-import-position)
C: 65, 0: Import "import common" should be placed at the top of the module (wrong-import-position)
C: 66, 0: Import "import rangelib" should be placed at the top of the module (wrong-import-position)
C: 67, 0: Import "import sparse_img" should be placed at the top of the module (wrong-import-position)
W:391,12: Redefining built-in 'dir' (redefined-builtin)

After this CL, it still gives the warning below, which is to be
addressed with follow-ups.

R:635, 2: Too many nested blocks (6/5) (too-many-nested-blocks)

Test: `m dist`
Test: pylint --rcfile=pylintrc add_img_to_target_files.py
Change-Id: I64da184b6b69e93449dbfc989a5d7f46d5223f42
2017-12-04 12:15:35 -08:00
Tao Bao dac0c88d9d releasetools: Remove deprecated pylint options.
This option u'required-attributes' will be removed in Pylint 2.0
This option u'ignore-iface-methods' will be removed in Pylint 2.0

Test: pylint (1.6.5) no longer gives above warnings.
Change-Id: Ib07cd5aef87407260140c1fca0b11c439cbdf390
2017-12-04 12:15:23 -08:00
David Zeuthen 8fecb289ee AVB: If building recovery.img, add AVB hash footer.
In some non-A/B setups, recovery.img is still being used. If AVB is
enabled, we currently don't add a hash footer to recovery.img nor do
we include the hash digest in vbmeta.img. This CL fixes that.

This was tested on a build with the following settings

 TARGET_NO_RECOVERY := false
 BOARD_USES_RECOVERY_AS_BOOT := false
 BOARD_BUILD_SYSTEM_ROOT_IMAGE := false
 BOARD_RECOVERYIMAGE_PARTITION_SIZE := 33554432
 BOARD_AVB_RECOVERY_ADD_HASH_FOOTER_ARGS := --prop foo:bar

and then it was verified using 'avbtool info_image' that recovery.img
has a hash footer and a 'foo' property with the value 'bar'. This was
also checked successfully for vbmeta.img.

Test: See above.
Bug: None
Change-Id: I98124d5661ea768411416fa8d2a2ae6cc664fdc8
2017-12-04 13:54:38 -05:00
Dan Willemsen 9be8a79255 Fix / suppress warnings for clang+mingw
am: 25c73e9156

Change-Id: I7f60ee6efc214605c397f91656f7aaf0a76a038d
2017-11-30 08:11:57 +00:00
Dan Willemsen 30f514cc51 Merge changes from topic "clang_supported"
* changes:
  Prepare for clang+mingw windows builds
  Fix / suppress warnings for clang+mingw
2017-11-30 08:05:58 +00:00
Dan Willemsen 25c73e9156 Fix / suppress warnings for clang+mingw
Switch attribute error to attribute unavailable, which apparently works
since it then broke the build :-/

Suppress unused parameter warnings. They aren't that useful for windows
functions that are just #ifndef WIN32. This is marked as a global cflag,
but something is overriding that with the clang+mingw changes, and it's
not actually that hard to suppress or fix where necessary.

Bug: 69933068
Test: mmma build/tools/libhost
Change-Id: I09adda4544bc9b56452b4fe182dba3d0b701dcbb
2017-11-29 20:54:33 -08:00
Tianjie Xu 4b57c28908 Merge "Improve the comment in AddSplitTransfers"
am: cca4a4c993

Change-Id: I853cd62997a964b91324d079e6940926c889ba6d
2017-11-29 21:57:52 +00:00
Tianjie Xu 41390d4e1e Improve the comment in AddSplitTransfers
Bug: 69624507
Test: build an incremental package
Change-Id: I54dac0a2ddcc19b722e352fb19d53ff20444fa42
2017-11-22 11:35:18 -08:00
Tianjie Xu ff9efd3290 Merge "Fall back to normal split if imgdiff fails when splitting large apks"
am: 520c9f0f5f

Change-Id: I0db7558c996cc515a79a99ce1d27525728a041dc
2017-11-22 18:46:04 +00:00
Tianjie Xu 520c9f0f5f Merge "Fall back to normal split if imgdiff fails when splitting large apks" 2017-11-22 18:28:31 +00:00
Tianjie Xu f68b50f963 Fall back to normal split if imgdiff fails when splitting large apks
Imgdiff expects the input files to be valid zip archives. During the
split of large apks, imgdiff may fail when there's a hole in the input
file; potentially due to the blocks allocation of mke2fs. This CL solves
the issue by falling back to normal split in case of the imgdiff split
failure. The split transfers will then use bsdiff instead.

Bug: 69624507
Test: generate the incremental package for the failed targets and check the
transfers.
Change-Id: I4882452378123e60ad3434053b57f33e53ac4b82
2017-11-21 23:37:36 -08:00
Tao Bao 51f5d3360e Merge "check_ota_package_signature.py: Set up PYTHONPATH."
am: 9f686938da

Change-Id: Ie84a386dd6acc44fd1fba987e8f0a5703248cd59
2017-11-21 21:40:13 +00:00
Tao Bao d2ae0b0613 check_ota_package_signature.py: Set up PYTHONPATH.
... to include system/update_engine/scripts if ANDROID_BUILD_TOP is set
(e.g. after lunching a target).

Test: ./build/make/tools/releasetools/check_ota_package_signature.py
      works without manually setting PYTHONPATH.
Change-Id: I0c5101fab2dd69cb6a598b909aa04bc4d5f45284
2017-11-21 11:30:27 -08:00
Tao Bao 4d1764b0be Merge "releasetools: Allow building AVB-enabled recovery.img."
am: 37066490a5

Change-Id: Ie8393a7e81c4f056a12a0be5ed2cd07ed54ef76b
2017-11-21 17:07:57 +00:00
Tao Bao 37066490a5 Merge "releasetools: Allow building AVB-enabled recovery.img." 2017-11-21 17:02:35 +00:00
Colin Cross bb051769c2 Merge "Convert signtos to Android.bp"
am: 80ec993a58

Change-Id: I5ddb92b0ca0485b01860aaff6ae613d5276eedb8
2017-11-21 03:33:06 +00:00
Colin Cross 3c363ad729 Convert signtos to Android.bp
See build/soong/README.md for more information.

Test: m checkbuild
Change-Id: Ia468cde0492e64858b7a603deba9b22b95e56ff4
2017-11-21 03:26:31 +00:00
Tao Bao a41e00784b Merge "releasetools: Always create IMAGES/ directory."
am: db7dfacb96

Change-Id: Iaede3e35fdd64dc748a98f625d390f0f38cd7e4a
2017-11-20 22:50:01 +00:00
Tao Bao bf70c318b7 releasetools: Allow building AVB-enabled recovery.img.
Bug: 68224784
Test: Enable AVB on aosp_bullhead-userdebug. `m dist`. Note that the
      prebuilt vendor.img needs to be AVB-signed first.
Test: `m dist` with aosp_walleye-userdebug.
Change-Id: I18235e4d0dde6af71a96e46ec434480cc9a22bef
2017-11-20 12:44:18 -08:00
Tao Bao ae396d9b0c releasetools: Always create IMAGES/ directory.
AddImagesToTargetFiles() takes either a zip file, or a zip root as
input. We used to create IMAGES/ directory only when working with zip
root input. Commit 262bf3f0b5 has changed
to also stage boot / recovery images there when working with a zip file.
This CL makes sure the directory is always available under both modes.

Bug: 63456822
Test: zip -d target_files.zip IMAGES/\*;
      add_img_to_target_files.py target_files.zip
Test: sign_target_files_apks.py target_files.zip signed-target_files.zip
Change-Id: Iea91d0403cdec1b16bb93bb71d3ed06856b8f7c3
2017-11-20 12:44:18 -08:00
Tao Bao 4727930bff Merge "releasetools: Fix the rebuilding of vbmeta.img."
am: 2f7fab9ab9

Change-Id: Ie6c6536be7da1e85167c22de36832b4145255f19
2017-11-18 00:32:23 +00:00