Commit Graph

6623 Commits

Author SHA1 Message Date
Andrei-Valentin Onea 15e75e8807 Merge "Revert "Move partition size checks to python script"" 2019-11-01 15:36:10 +00:00
Raff Tsai 0141e45b96 Revert "Move partition size checks to python script"
This reverts commit a4906cd306.

Reason for revert: b/143734706

Change-Id: I905bd9c0bfcbd123c8f9e78986bfcb5600a9eca9
2019-11-01 05:36:20 +00:00
Yifan Hong 8244f27722 Merge "Move partition size checks to python script" 2019-11-01 01:52:11 +00:00
Yifan Hong a4906cd306 Move partition size checks to python script
Easier to read and modify its logic.

Test: builds
Test: test_check_partition_sizes
Test: atest releasetools_test
Test: atest releasetools_py3_test
Change-Id: I73204664b64a0bcee7375fa6fca46758d5a2bb72
2019-10-31 11:22:20 -07:00
Stephen Hines aa94811b2a Merge "Guess source tree root from common file prefix." 2019-10-18 17:59:23 +00:00
Tianjie Xu 526b8563a5 Merge "Stop creating update_engine_payload_key" 2019-10-17 20:59:51 +00:00
Treehugger Robot 323a460d5b Merge "releasetools: Fix an issue in common.GetAvbPartitionArg." 2019-10-16 20:52:19 +00:00
Tianjie Xu 2df23d7f41 Stop creating update_engine_payload_key
Now the update_engine is able to read public keysfrom otacerts directly.
So the update_engine_payload_key is no longer needed.

Also remove the key replace in sign_target_files_apks.py. So we should
not use the new script to sign the old target files.

Bug: 116660991
Test: build the system image, unit tests pass
Change-Id: I9dae1f8b397f2b5efafed66a8faac1cb9087c741
2019-10-16 18:27:22 +00:00
Tianjie Xu 3654beb83d Merge "Call delta_generator to get the signature size" 2019-10-15 16:49:15 +00:00
Tao Bao 3612c88ed7 releasetools: Fix an issue in common.GetAvbPartitionArg.
It's a bug introduced in commit 1aeef725a7, which affects unittest only.

Bug: 130351427
Test: Use a lunch'd target. Run
      `atest --host releasetools_test releasetools_py3_test`.
Change-Id: I7ff01a6af47d002e1203bd376d477b60d769cbd1
2019-10-14 17:53:21 -07:00
Tao Bao 21fb765515 Merge "releasetools: Set the search path based on the current executable." 2019-10-14 22:20:02 +00:00
Tao Bao dc2f0b0abd Merge "Include per-partition fingerprint as AVB prop." 2019-10-14 14:53:46 +00:00
Tianjie Xu 21e6deb647 Call delta_generator to get the signature size
The signature size is needed during payload hashing and signing.
We used to sign the update_engine's payload with RSA keys only. In
this case, the signature size always equals the key size. But the
assumption is no longer true for EC keys, whose DER-encoded signature
size is a variant with a maximum size.

Therefore, we always give the maximum signature size to the delta
generator, who then add paddings to the real signature if necessary.
The maximum signature size is calculated by calling the delta_generator
with the new option '--calculate_signature_size'. For custom payload
signers, we also deprecate the '--payload_signer_key_size' and replace
it with '--payload_signer_maximum_signature_size'.

The EC key in the test is generated with:
The EC key in the unittest is generated with the command:
openssl ecparam -name prime256v1 -genkey -noout -out prime256v1-key.pem
openssl pkey -in prime256v1-key.pem -out testkey_EC.key

Bug: 141244025
Test: sign and verify a payload
Change-Id: Ife6e269d8aa3d870405aca20086330f1795e167f
2019-10-12 00:36:54 +00:00
Tao Bao afd92a8689 releasetools: Set the search path based on the current executable.
We used to mimic the behavior of build system, to find the default
search path based on OUT_DIR_COMMON_BASE or OUT_DIR. These variables
should be internal to build system.

Since we've switched releasetools script to hermetic Python executables
(e.g. `m -j ota_from_target_files`, then run the binary at
`out/host/linux-x86/bin/ota_from_target_files`), we can set the search
path in relative to the path of the current executable.

Bug: 133126366
Test: TreeHugger
Test: 1. Build aosp_x86, by "lunch aosp_x86; m -j"
      2. Inject errors to the executables under out/host/linux-x86/bin,
         e.g.  to `lpmake`.
      3. Set up OUT_DIR (e.g., to /tmp/out) and build the same product
         again by "export OUT_DIR=/tmp/out; lunch aosp_x86; m -j". Check
         that the second run finishes successfully (with the binaries at
         /tmp/out as opposed to out/; otherwise it would fail the build
         due to the invalid binaries from step 2).
Test: lunch a target;
      `atest --host releasetools_test releasetools_py3_test`
Change-Id: I366099c3dfd5fa4282745ef258a8cf35338e1e42
2019-10-11 08:53:41 -07:00
cfig 1aeef725a7 AVB: decouple vbmeta.img from recovery.img for non-A/B devices
For following cases:
Case 1: A/B devices: no change
Case 2: non-A/B devices, with unsigned recovery image:
    not allowed anymore by mandating BOARD_AVB_RECOVERY_KEY_PATH
Case 3: non-A/B devices, with signed recovery image:
    vbmeta.img should not include ChainPartitionDescriptor of recovery.img,
    otherwise device can not even boot into normal mode if recovery partition
    is damaged

This CL will cause a build break if BOARD_AVB_RECOVERY_KEY_PATH
is not set for non-A/B targets with recovery.img
The following is an example to fix the build break by specifying
AVB signing configs for the recovery.img.

BOARD_AVB_RECOVERY_KEY_PATH := external/avb/test/data/testkey_rsa2048.pem
BOARD_AVB_RECOVERY_ALGORITHM := SHA256_RSA2048
BOARD_AVB_RECOVERY_ROLLBACK_INDEX := $(PLATFORM_SECURITY_PATCH_TIMESTAMP)
BOARD_AVB_RECOVERY_ROLLBACK_INDEX_LOCATION := 2

Also note that libavb in bootloader needs an update to include this
commit Iaa886037edb18c2ff6c60fa2a7f883ab7303ba1a, to support verifying
recovery.img independently (not through vbmeta.img).

Bug: 130351427
Test (Case 3):
  normal   mode: avb_slot_verify(flags=AVB_SLOT_VERIFY_FLAGS_NONE)
  recovery mode: avb_slot_verify(flags=AVB_SLOT_VERIFY_FLAGS_NO_VBMETA_PARTITION)
Test: PYTHONPATH=build/make/tools/releasetools \
    python -m unittest test_validate_target_files
Test: Use a lunch'd target. `atest --host releasetools_test releasetools_py3_test`
Test: validate_target_files.py with Case-3 target files
Change-Id: I2a73252b385fa463b4abd444923a8acc473df0b4
2019-10-10 14:28:34 +00:00
Tao Bao 19b02fe8e5 Include per-partition fingerprint as AVB prop.
This allows querying per-partition fingerprint via libavb (in
particular, avb_property_lookup).

Bug: 80097573
Test: `m dist`; `avbtool info_image --image /path/to/image` to check the
      written prop.
Test: `atest --host releasetools_test releasetools_py3_test`
Test: Run sign_target_files_apks to sign a target_files.zip that's built
      with the change. Check the AVB prop in the signed images.
Change-Id: Id4f06df82c29e77aca128439c8c11f0367fe587e
2019-10-09 21:25:19 -07:00
Tao Bao 3ed35d3d25 releasetools: Fix the support for getting reproducible images.
We used to support getting idempotent images when given the same inputs,
by specifying UUID and AVB salt to the image builders. The support was
broken by Q changes that moved away from build-time fingerprints [1].

This CL addresses the issue by updating the logic in computing UUID and
AVB salt to be based on BuildInfo.fingerprint, as opposed to the
deprecated static build fingerprint.

[1] https://android-review.googlesource.com/c/platform/build/+/892933

Bug: 134525174
Test: TreeHugger
Test: lunch a target; `atest --host releasetools_test releasetools_py3_test`
Test: `m dist`; `unzip out/dist/target_files.zip IMAGES/\*`;
      `zip -d out/dist/target_files.zip IMAGES/\*`
Test: `add_img_to_target_files -a out/dist/target_files.zip`. Verify that
      the rebuilt images are identical to the ones deleted.
Change-Id: I8844fc7f044e9abfcd96a00e8958fa0583c70eaf
2019-10-07 23:32:07 -07:00
Tao Bao 1c320f8573 releasetools: Move BuildInfo into common.
There is no change to module functionalities. Testcases are moved around
accordingly.

Bug: 134525174
Test: TreeHugger
Test: lunch a target; atest --host releasetools_test releasetools_py3_test
Change-Id: I7bc8f49cc239e7c6655fe5e375508f01c1743b94
2019-10-07 20:13:59 -07:00
Tao Bao e114804150 releasetools: Move MockScriptWriter into test_utils.
Bug: 134525174
Test: TreeHugger
Test: lunch a target; atest --host releasetools_test releasetools_py3_test
Change-Id: I6d30f4d153d59d65227275e1d3285e30dfafd90e
2019-10-07 20:00:34 -07:00
Tao Bao 60f5d67677 releasetools: Update OWNERS.
Test: N/A
Change-Id: Ie476ff08a360f15f769e54dca7428a2d055f1a37
2019-10-07 12:39:55 -07:00
Tao Bao 7b1c49a70d Merge "releasetools: Update the handling of recovery fstab." 2019-10-07 19:33:21 +00:00
Tao Bao ae3a9a138d Merge "releasetools: Fix releasetools_py3_test." 2019-10-07 19:06:07 +00:00
Tao Bao 615b65d097 releasetools: Fix releasetools_py3_test.
Test: Use a lunch'd target;
      `atest --host releasetools_test releasetools_py3_test`
Change-Id: Ibcced83a7989ef8fc15464728ab1d5fa71ca988e
2019-10-06 23:29:34 -07:00
Tao Bao a264feffe8 releasetools: Update tests to match SparseImage change.
The previous change in commit 22632cc82c
changed the behavior in loading file map. It now always puts a copy of
the input text in `extra` field. Update the tests accordingly.

Bug: 79951650
Test: Use a lunch'd target; `atest --host releasetools_test`
Change-Id: Iccf06c817c1305bf9946d7759c6f6f6af21fe85e
2019-10-06 22:21:14 -07:00
Tao Bao 99d855db8f Merge "releasetools: Support verifying files with non-monotonic ranges." 2019-10-07 01:33:30 +00:00
Tao Bao 765668fff7 releasetools: Update the handling of recovery fstab.
First, remove the use of info_dict['fstab'] in add_img_to_target_files.
 - info_dict['fstab'] corresponds to recovery fstab
   (`/etc/recovery.fstab`), which may differ from the one used for
   normal boot.
 - When calling build_image.ImagePropFromGlobalDict, we already have the
   desired info from global dict (`META/info_dict.txt`).

Second, common.LoadInfoDict now loads recovery fstab only for non-A/B
devices. Because the info is only meaningful for installing non-A/B OTAs
(under recovery mode).

Fixes: 132458722
Test: TreeHugger
Test: Build non-A/B incremental OTAs.
Change-Id: Id23e7b17264c123319fe00b1663d52bfd9b4a5e2
2019-10-04 22:39:32 -07:00
Tao Bao 22632cc82c releasetools: Support verifying files with non-monotonic ranges.
Fixes: 79951650
Test: Run validate_target_files on target_files.zip with files in
      non-monotonic ranges.
Test: python -m unittest test_validate_target_files
Test: python3 -m unittest test_validate_target_files
Change-Id: I82571d3358598775de4cdeb5e64035689fea6487
2019-10-03 23:21:22 -07:00
Bill Peckham e868aec14b Moving recovery resources from /system to /vendor
This change is part of a topic that moves the recovery resources from the
system partition to the vendor partition, if it exists, or the vendor directory
on the system partition otherwise. The recovery resources are moving from the
system image to the vendor partition so that a single system image may be used
with either an A/B or a non-A/B vendor image. The topic removes a delta in the
system image that prevented such reuse in the past.

The recovery resources that are moving are involved with updating the recovery
partition after an update. In a non-A/B configuration, the system boots from
the recovery partition, updates the other partitions (system, vendor, etc.)
Then, the next time the system boots normally, a script updates the recovery
partition (if necessary). This script, the executables it invokes, and the data
files that it uses were previously on the system partition. The resources that
are moving include the following.

* install-recovery.sh
* applypatch
* recovery-resource.dat (if present)
* recovery-from-boot.p (if present)

This change includes the platform build system and release tools changes to
move the recovery resources from system to vendor (or /system/vendor). The
release tools need to know where to generate the recovery patch, and they
discover this from misc_info.txt variable board_uses_vendorimage, which the
platform build system generates.

We remove applypatch from PRODUCT_PACKAGES, but it is added back as a required
module in target/product/base_vendor.mk.

Several release tools rely on the misc_info.txt board_uses_vendorimage variable
to know how to generate and detect the recovery patch.

This change partially removes the --rebuild_recovery flag from the
merge_target_files.py script. The flag will be fully removed in a follow-on
change.

Bug: 68319577
Test: Ensure that recovery partition is updated correctly.
Change-Id: Ia4045bd67ffb3d899efa8d20dab4c4299b87ee5f
2019-10-04 00:04:34 +00:00
Treehugger Robot 1623ec97f1 Merge "add vendor_boot image support" 2019-09-27 21:07:24 +00:00
Steve Muckle e1b1086ef3 add vendor_boot image support
The vendor_boot partition is generated by mkbootimg and contains all the
device-specific information that used to reside in the boot partition.

Bug: 137297791
Change-Id: I5b005097b73f59857c3a2f92d693b3e67ee8424e
2019-09-27 10:26:00 -07:00
Chih-Hung Hsieh 83980501d9 Fix severity of analyzer warnings; add new warnings.
* The following were high severity warnings but mistakenly
  grouped into analyzer warnings in a recent change.
        clang-analyzer-unix.Malloc
        clang-analyzer-cplusplus.NewDeleteLeaks
        clang-analyzer-cplusplus.NewDelete
* Add new Protoc and Kotlin warnings about unused imports and variables.

Bug: 141493530
Test: warn.py --url=http://cs/android --separator='?l=' build.log > warnings.html
Change-Id: Ied1f9ace97f29fba58a40dff2346a0391ab22f06
2019-09-26 12:14:32 -07:00
Chih-Hung Hsieh 790020437d Factor out common code in Java warn patterns.
* Some comments are kept to reduce diff artifacts.
* Warning patterns will be separated by tools later,
  to make it easier to share code with Chromium and
  other build tools.

Bug: 141493530
Test: warn.py --url=http://cs/android --separator='?l=' build.log > warnings.html
Change-Id: I6b589478cf5ae73940b4712d9faabd0909546f02
2019-09-25 13:39:41 -07:00
Chih-hung Hsieh 415186a71c Merge "Add new compiler warning patterns" 2019-09-25 17:00:36 +00:00
Paul Duffin e19b7471ae Merge "Replace use of conscrypt-nojarjar with conscrypt-unbundled" 2019-09-25 09:58:56 +00:00
Chih-Hung Hsieh d591a5e012 Add new compiler warning patterns
* Add Kotlin, NDK, Protoc, Asm, and other new patterns.
* Put all clang-analyzer warnings to the ANALYZER severity group.
  Some security and null pointer warnings still have HIGH severity.
* Comment out old C/C++ warnings that overlap with analyzer
  warnings and do not appear in current build.log.
* Other old fixed warning patterns should be commented out later.

Bug: 141493530
Test: warn.py --url=http://cs/android --separator='?l=' build.log > warnings.html
Change-Id: Idc7b09e5e4eefa127d41e9f37da1f566ed1ede1f
2019-09-24 14:52:09 -07:00
Chih-Hung Hsieh f5db8985f5 Guess source tree root from common file prefix.
* When processing a downloaded build.log file,
  find_android_root is very slow because the
  android source tree is not found.
  Change it to look for warn.py only up to 100 times,
  and then set android_root to the longest common path prefix.

Bug: 141524232
Test: warn.py --url=http://cs/android --separator='?l=' build.log > warnings.html
Change-Id: I8cc2e208d439267313aaa6cdbc00f40320409c63
2019-09-24 18:19:17 +00:00
Chih-hung Hsieh d5ea80b388 Merge "Handle rustc warning messages." 2019-09-23 16:53:38 +00:00
Tao Bao 08f9da1898 Merge "Don't generate hashtree when signing bundled APEXes." 2019-09-23 14:22:59 +00:00
Paul Duffin ebbcc1bb78 Replace use of conscrypt-nojarjar with conscrypt-unbundled
Bug: 141346785
Test: m checkbuild
Change-Id: Ibcd4aac938e0fdf5765ad8426fe64c6d053f3529
2019-09-23 14:30:07 +01:00
Chih-Hung Hsieh a1187072c5 Handle rustc warning messages.
* Combine the first two lines of rustc warning messages.
* Give some rustc warnings HIGH and MEDIUM severity.

Bug: 141311294
Test: warn.py --url=http://cs/android --separator='?l=' build.log > warnings.html
Change-Id: Ia3a206d68d9e92934296487b006a7b424044a822
2019-09-20 17:14:22 -07:00
Tao Bao 448004af9d Don't generate hashtree when signing bundled APEXes.
Bug: 139957269
Test: Sign a target_files.zip. Extract a re-signed APEX and check the
      hashtree size (being zero).
Test: Use sign_apex to sign an APEX file. Check the hashtree size (not
      being zero).
Test: python -m unittest test_apex_utils
Test: python -m unittest test_sign_apex
Change-Id: I927b7681d66920d7732b700ec3a8f7a65b4cb351
2019-09-19 14:41:34 -07:00
Treehugger Robot c4f6f36dc4 Merge "Creates img.zip from target files zip, not dir." 2019-09-19 01:03:33 +00:00
Daniel Norman 74eb74b146 Creates img.zip from target files zip, not dir.
Directory support for img_from_target_files was dropped in aosp/1095755.

Test: Create & boot a merged build.
Change-Id: Ib976ea60a24c2f172c597130d6bcce25199a93b3
2019-09-18 14:03:34 -07:00
Tao Bao 3e759462d9 releasetools: Remove more images out of secondary payload.
This CL additionally removes boot/dtbo/modem/vbmeta_* images out of
secondary payload. We essentially only keep system_other.img and
bootloader images there.

For Pixel devices, this additionally saves ~80MiB (mostly because of
the removal of boot and radio images).

Bug: 140771390
Test: ota_from_target_files \
          --include_secondary --skip_postinstall \
          -i input-target_files-1.zip \
          input-target_files-2.zip \
          output-ota.zip
Test: python -m unittest test_ota_from_target_files
Change-Id: If47b27c52b3547a4cc86223a988c53960bc6af40
2019-09-17 22:56:19 -07:00
Tao Bao d5b6094ed3 Merge "releasetools: Sanity check the build fingerprint." 2019-09-17 18:30:45 +00:00
Tao Bao e430f7652a Merge "Also install verity_key to ramdisk for non-system-as-root target." 2019-09-17 18:25:55 +00:00
Yifan Hong 7ca2c1539f Merge "extract-kernel: Fix indexing" 2019-09-17 16:50:36 +00:00
Tao Bao c99819311c Also install verity_key to ramdisk for non-system-as-root target.
The commit in d14b895665
(https://android-review.googlesource.com/c/platform/build/+/728287)
changed partition layout, to always build the root dir into system.img,
even for devices not using system-as-root (i.e. the ones with separate
boot ramdisk).

With the new layout, there will be two root dirs for non-system-as-root
targets during the boot. If such a device uses Verified Boot 1.0,
/verity_key needs to be available in both roots, to establish the chain
of trust.
 - bootloader uses the baked-in key to verify boot.img; it then loads
   the ramdisk from the verified boot.img
 - First stage init uses /verity_key (in ramdisk) to verify and mount
   system.img at /system, then chroot's to it
 - Second stage init uses /verity_key (in system.img) to verify and
   mount other partitions

This CL adds rules to additionally install verity_key into ramdisk for
such targets.

Bug: 139770257
Test: Set up a target to use non-system-as-root
      (BOARD_BUILD_SYSTEM_ROOT_IMAGE != true). `m dist`.
Test: Check that both ROOT/verity_key and BOOT/RAMDISK/verity_key exist
      in the built target_files.zip.
Test: Run validate_target_files to validate the above target_files.zip.
      $ validate_target_files \
          --verity_key_mincrypt /path/to/verity_key \
          target_files.zip
Test: Run sign_target_files_apks to sign the above target. Re-run
      validate_target_files on the signed target_files.zip.
Test: python -m unittest test_validate_target_files
Change-Id: Ibe7e771c8c376429add85851ac86055564765d3c
2019-09-17 08:53:16 -07:00
Tao Bao c4011cd719 releasetools: Sanity check the build fingerprint.
Bug: 140298338
Test: python -m unittest test_ota_from_target_files
Change-Id: I4add5f82d5b684c441b0484732da6991f8e7301a
2019-09-17 08:43:54 -07:00
Yifan Hong 8b72776511 extract-kernel: Fix indexing
The original script always tries to extract from index
0 even if header is found in positive indices. Fix that.

Also, continue to try other positions if previous run failed.

Test: builds
Bug: 139348603

Change-Id: Ia54fc709de9ae587fc64b59d94a1fa4ae669c14f
2019-09-16 16:30:35 -07:00