Commit Graph

5739 Commits

Author SHA1 Message Date
Tao Bao e0bbacf193 Merge "releasetools: Capture and dump stdout/stderr outputs on errors." am: f3e39e5878
am: 49df32de92

Change-Id: I14cd733bfd0bb6bf2ec23b3834e33c1ba41bcc1f
2018-03-23 01:23:44 +00:00
Treehugger Robot f3e39e5878 Merge "releasetools: Capture and dump stdout/stderr outputs on errors." 2018-03-23 01:05:27 +00:00
Tao Bao 701d830acb Merge "Remove the hard-coded path of build_verity_metadata.py." am: 0793683e86
am: 2a8e13f75d

Change-Id: Ifb02ec1a445595f3858cbbbc96155fbe6b6b7e4b
2018-03-22 19:01:44 +00:00
Tao Bao f47bf0fecf releasetools: Fix an issue in GetMinSdkVersion.
The following is a buggy pattern that won't capture anything into err.
The issue is benign, since a failed run would be eventually captured by
a subsequent check.

  p = Run(["aapt", ...], stdout=subprocess.PIPE)
  output, err = p.communicate()
  if err:
    raise ...

This CL changes the error detection to be based on the return code from
aapt. It also adds some sanity test to ensure the call to aapt works.
The test app is built from AOSP com.android.cts.ctsshim (chosen mostly
because of its small size).

Test: python -m unittest test_common
Change-Id: I337f141bd0fc5f0801dfc628c601b88b7640789c
2018-03-22 10:27:04 -07:00
Tao Bao 80921986d1 releasetools: Capture and dump stdout/stderr outputs on errors.
For the functions in common.py, capture the stdout/stderr outputs when
shelling out to external tools. Dump the outputs on errors.

Bug: 76123422
Test: Inject errors to signapk.jar / brotli / unzip. Check the outputs.
Change-Id: Ib2d4272528b83a50bb727788cf8a5211e2ddade8
2018-03-22 00:09:33 -07:00
Tao Bao b4ec6d752a Remove the hard-coded path of build_verity_metadata.py.
build_image.py used to invoke build_verity_metadata.py with a hard-coded
path of 'system/extras/verity/build_verity_metadata.py', which makes it
hard to run unittests from non-$(ANDROID_BUILD_TOP) directory.

This CL adds the dependency on the tool, so that it gets installed to
$(HOST_OUT_EXECUTABLES), then removes the hard-coded path.

Bug: 74544459
Bug: 76015688
Test: `m dist`
Test: python -m unittest test_validate_target_files
Change-Id: I0dcf4eb067a0db6f099cb589eb99a151a05c7f2b
2018-03-21 21:15:59 +00:00
Tao Bao f279b6b66b Merge "releasetools: Handle two edge cases in FinalizeMetadata()." into pi-dev
am: b3e5d54efa

Change-Id: I0b0afd630a26e59ac4e94c1c921a65585aafb536
2018-03-21 20:42:26 +00:00
Treehugger Robot 2ebcf419e4 Merge "releasetools: Handle two edge cases in FinalizeMetadata()." 2018-03-21 20:29:25 +00:00
Tao Bao d2ce2ed5de releasetools: Handle two edge cases in FinalizeMetadata().
In FinalizeMetadata and PropertyFiles, we need to reserve space between
the calls to Compute() and Finalize(). We used to put a 10-byte
placeholder, in the hope of covering the 'offset:length' space for the
metadata entry, as well as the possible value changes in other entries.

However, this could fail in two possible cases: (a) metadata entry
itself has a large offset (e.g. staying near the end of a 1-GiB package,
where the offset itself has 10-digit); or (b) the offsets for other
entries change substantially due to entry reordering. Note that for case
(b), it's space inefficient to always reserve 15-byte for _each_ token
in the property-files.

This CL handles both of these two cases. For (a), we bump up the 10-byte
to 15-byte, which is large enough to cover a package size up to 10-digit
number (i.e. ~9GiB) with a metadata entry size of 4-digit. All these
15-byte will be used for the metadata token alone.

For (b), we add a fallback flow that would retry one more time, but
based on the already signed package that has entries in desired order.

Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate aosp-bullhead full OTA with '--no_signing' flag.
Change-Id: If20487602d2ad09b3797465c01972f2fa792a1f1
(cherry picked from commit 3bf8c65029)
2018-03-21 06:19:28 +00:00
Tao Bao 2d2ce7dd02 Use bsdiff to generate recovery-from-boot.p for system-root-image.
This CL handles a path that uses system-root-image on non-A/B device.
For this path, we can't generate recovery-from-boot patch with imgdiff,
because boot/recovery images contain different number of entries (only
recovery iamge has ramdisk image now).

Using BOARD_USES_FULL_RECOVERY_IMAGE can work around the issue, at the
cost of extra size. Compared to carrying full recovery image, this CL
saves the cost of the kernel size, by putting a patch that's roughly the
size of the recovery ramdisk.

The applypatch executable already detects and handles a bsdiff patch
automatically. No change required to that end.

Note that it won't further reduce the patch size by handling that
ramdisk entry specially, because (a) that's the only difference between
the two images; and (b) there's no corresponding data in boot image to
be diff'd against.

Bug: 72731506
Test: `m dist` with aosp_angler-userdebug. Check the device can install
      recovery image successfully (after intentionally corrupting the
      recovery image).
Test: Build aosp_angler-userdebug with BOARD_BUILD_SYSTEM_ROOT_IMAGE set.
      Verify the generated patch files.
Test: Run validate_target_files.py with the target_files.zips.
Change-Id: I69c06f51ba8c39ae059c5e9a6872a9f10600cf17
Merged-In: I69c06f51ba8c39ae059c5e9a6872a9f10600cf17
(cherry picked from commit 6d5d623987)
2018-03-20 11:38:50 +08:00
Tao Bao 1f95c2bb41 Merge "Use bsdiff to generate recovery-from-boot.p for system-root-image." am: ad29fe419e
am: 40feb95108

Change-Id: I54bf5debc6ffcf8d069211385b55844dcaaca605
2018-03-20 00:27:50 +00:00
Tao Bao 491d7e2ea8 releasetools: Add NonAbOtaPropertyFiles for non-A/B OTA packages.
This CL exposes ota-property-files flag for non-A/B OTA packages.
Currently the line only contains the info for the METADATA entry, for
example "ota-property-files=metadata:69:286". This allows system updater
to just download the METADATA entry, as opposed to downloading the
entire package, to learn about the info regarding the OTA package (e.g.
post-OTA build fingerprint). Note that this requires the OTA server-side
support to pass down the flag along with the update URL.

Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate a non-A/B package and check the property-files string.
Change-Id: I1482c587e18ea7101c8328777ea988c2d8ca06ac
Merged-In: I1482c587e18ea7101c8328777ea988c2d8ca06ac
(cherry picked from commit c0746f4e94)
2018-03-19 14:53:10 -07:00
Tao Bao 6d5d623987 Use bsdiff to generate recovery-from-boot.p for system-root-image.
This CL handles a path that uses system-root-image on non-A/B device.
For this path, we can't generate recovery-from-boot patch with imgdiff,
because boot/recovery images contain different number of entries (only
recovery iamge has ramdisk image now).

Using BOARD_USES_FULL_RECOVERY_IMAGE can work around the issue, at the
cost of extra size. Compared to carrying full recovery image, this CL
saves the cost of the kernel size, by putting a patch that's roughly the
size of the recovery ramdisk.

The applypatch executable already detects and handles a bsdiff patch
automatically. No change required to that end.

Note that it won't further reduce the patch size by handling that
ramdisk entry specially, because (a) that's the only difference between
the two images; and (b) there's no corresponding data in boot image to
be diff'd against.

Bug: 72731506
Test: `m dist` with aosp_angler-userdebug. Check the device can install
      recovery image successfully (after intentionally corrupting the
      recovery image).
Test: Build aosp_angler-userdebug with BOARD_BUILD_SYSTEM_ROOT_IMAGE set.
      Verify the generated patch files.
Test: Run validate_target_files.py with the target_files.zips.
Change-Id: I69c06f51ba8c39ae059c5e9a6872a9f10600cf17
2018-03-19 14:13:02 -07:00
Tao Bao 3bf8c65029 releasetools: Handle two edge cases in FinalizeMetadata().
In FinalizeMetadata and PropertyFiles, we need to reserve space between
the calls to Compute() and Finalize(). We used to put a 10-byte
placeholder, in the hope of covering the 'offset:length' space for the
metadata entry, as well as the possible value changes in other entries.

However, this could fail in two possible cases: (a) metadata entry
itself has a large offset (e.g. staying near the end of a 1-GiB package,
where the offset itself has 10-digit); or (b) the offsets for other
entries change substantially due to entry reordering. Note that for case
(b), it's space inefficient to always reserve 15-byte for _each_ token
in the property-files.

This CL handles both of these two cases. For (a), we bump up the 10-byte
to 15-byte, which is large enough to cover a package size up to 10-digit
number (i.e. ~9GiB) with a metadata entry size of 4-digit. All these
15-byte will be used for the metadata token alone.

For (b), we add a fallback flow that would retry one more time, but
based on the already signed package that has entries in desired order.

Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate aosp-bullhead full OTA with '--no_signing' flag.
Change-Id: If20487602d2ad09b3797465c01972f2fa792a1f1
2018-03-19 13:35:38 -07:00
Tao Bao ad5440d20e Merge "releasetools: Add NonAbOtaPropertyFiles for non-A/B OTA packages." am: 93a84dd9a6
am: f13591bbb3

Change-Id: I43114d7b8ce5900e706e9f234c0fe67cff7cf268
2018-03-19 20:05:28 +00:00
Tao Bao c0746f4e94 releasetools: Add NonAbOtaPropertyFiles for non-A/B OTA packages.
This CL exposes ota-property-files flag for non-A/B OTA packages.
Currently the line only contains the info for the METADATA entry, for
example "ota-property-files=metadata:69:286". This allows system updater
to just download the METADATA entry, as opposed to downloading the
entire package, to learn about the info regarding the OTA package (e.g.
post-OTA build fingerprint). Note that this requires the OTA server-side
support to pass down the flag along with the update URL.

Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate a non-A/B package and check the property-files string.
Change-Id: I1482c587e18ea7101c8328777ea988c2d8ca06ac
2018-03-18 11:59:38 -07:00
Tao Bao 2462a1eb82 releasetools: Add AbOtaPropertyFiles.
am: 85f1698fa0

Change-Id: Ia829f4417659946fcc99b22b66bb392a0a75ae49
2018-03-16 05:08:10 +00:00
Tao Bao 85f1698fa0 releasetools: Add AbOtaPropertyFiles.
AbOtaPropertyFiles writes 'ota-property-files' into package metadata.
Comparing to StreamingPropertyFiles, AbOtaPropertyFiles additionally
writes a token of 'payload-metadata.bin' for a virtual entry that's head
of the 'payload.bin'. This entry can be used by the system updater to
verify if a payload is applicable to the underlying device.

AbOtaPropertyFiles intends to replace StreamingPropertyFiles, as it
covers all the info of the latter. We keep them both in P, and will
remove the latter in later release.

Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate full and incremental A/B OTAs. Check the property-files
      in the METADATA entry.
Change-Id: If5872c8ee0a97a8a95b07d4c69b463aa3fe1b9b9
(cherry picked from commit b63046750c)
2018-03-15 16:23:52 -07:00
Tao Bao f8b2b15a9c Merge "releasetools: Create PropertyFiles class." am: febe7b0736
am: 624f900c01

Change-Id: I58fed3c6e450e3bed7af603d4ad0664618d4d319
2018-03-15 23:05:54 +00:00
Tao Bao b63046750c releasetools: Add AbOtaPropertyFiles.
AbOtaPropertyFiles writes 'ota-property-files' into package metadata.
Comparing to StreamingPropertyFiles, AbOtaPropertyFiles additionally
writes a token of 'payload-metadata.bin' for a virtual entry that's head
of the 'payload.bin'. This entry can be used by the system updater to
verify if a payload is applicable to the underlying device.

AbOtaPropertyFiles intends to replace StreamingPropertyFiles, as it
covers all the info of the latter. We keep them both in P, and will
remove the latter in later release.

Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate full and incremental A/B OTAs. Check the property-files
      in the METADATA entry.
Change-Id: If5872c8ee0a97a8a95b07d4c69b463aa3fe1b9b9
2018-03-15 14:58:31 -07:00
Tao Bao 69203525e4 releasetools: Create PropertyFiles class.
And move StreamingPropertyFiles as its subclass. We will need similar
PropertyFiles instance for non-A/B OTA as well (to expose the
offset/size for the METADATA entry).

Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate an A/B OTA. Check the generated property-files string.
Test: pylint --rcfile=pylintrc \
          ota_from_target_files.py \
          test_ota_from_target_files.py
Change-Id: If90d97f0b330749fd8a6cde2ed9d0d6cd6ea60a8
2018-03-15 14:50:55 -07:00
Tao Bao d3fc38a073 releasetools: Create PropertyFiles class.
And move StreamingPropertyFiles as its subclass. We will need similar
PropertyFiles instance for non-A/B OTA as well (to expose the
offset/size for the METADATA entry).

Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate an A/B OTA. Check the generated property-files string.
Test: pylint --rcfile=pylintrc \
          ota_from_target_files.py \
          test_ota_from_target_files.py
Change-Id: If90d97f0b330749fd8a6cde2ed9d0d6cd6ea60a8
(cherry picked from commit 432f374a1701909ca324f8b047666614684568c0)
2018-03-15 14:17:44 -07:00
Tao Bao 69cc0de37b Merge "releasetools: Support validating Verified Boot images." am: 1a6220f598
am: b5376ee331

Change-Id: I3db23dc63900c3e3bc92f1b02033dfa4ffeceefe
2018-03-12 17:49:01 +00:00
Tao Bao ba557707d8 releasetools: Support validating Verified Boot images.
For a given (signed) target-files.zip, this CLs allows verifying the
Verified Boot related images. It works with both of VB 1.0 and VB 2.0
images.

As part of the CL, it also moves validate_target_files.py to argparse,
which is more flexible than the traditional getopt module.

Also add unittests for the VB 1.0 path. VB 2.0 tests will be added in
follow-up CL.

Example usage:

- Run the script on aosp_bullhead target-files.zip.

$ ./build/make/tools/releasetools/validate_target_files.py \
    --verity_key build/target/product/security/verity.x509.pem \
    --verity_key_mincrypt build/target/product/security/verity_key \
    aosp_bullhead-target_files-4522605.zip

- Run the script on aosp_walleye target-files.zip.

$ ./build/make/tools/releasetools/validate_target_files.py \
    --verity_key external/avb/test/data/testkey_rsa4096.pem \
    aosp_walleye-target_files-4627254.zip

Bug: 63706333
Bug: 65486807
Test: Run validate_target_files.py on target_files.zip files.
Test: PYTHONPATH=build/make/tools/releasetools python -m unittest \
          test_validate_target_files
Change-Id: I170f14d5828d15f3687d8af0a89a816968069057
2018-03-12 00:17:09 -07:00
Tao Bao cc8e2666ac releasetools: Create StreamingPropertyFiles class.
This CL breaks down ComputeStreamingMetadata() into mutiple member
functions of StreamingPropertyFiles class, which correspond to the
two-pass logic when generating streaming property files (aka streaming
metadata).

StreamingPropertyFiles.Compute() does the work for the first pass, by
putting placeholders before doing initial signing. Finalize()
corresponds to the second pass, where the placeholders get replaced with
actual data. Verify() can be optionally called to assert the correctness
of the work.

The separation between Compute() and Finalize() is to allow having
multiple StreamingPropertyFiles instances (in coming up CLs). This way
we can call Compute() multiple times for each instance, followed by only
one call to SignOutput(). And similarly for Finalize().

Bug: 74210298
Test: Generate an A/B OTA package. Check the METADATA entry.
Test: python -m unittest test_ota_from_target_files
Change-Id: I45be0372a4863c4405e6d8e20bcb9ccdc29e7e11
(cherry picked from commit ae5e4c30fe)
2018-03-09 13:44:13 -08:00
Tao Bao a49b0446d7 Merge "releasetools: Create StreamingPropertyFiles class." am: 32dfa4914d
am: 934e04d1a3

Change-Id: I2a1c65d09ffc1f4231cec45874557c3e3af74444
2018-03-09 20:13:30 +00:00
Tao Bao ae5e4c30fe releasetools: Create StreamingPropertyFiles class.
This CL breaks down ComputeStreamingMetadata() into mutiple member
functions of StreamingPropertyFiles class, which correspond to the
two-pass logic when generating streaming property files (aka streaming
metadata).

StreamingPropertyFiles.Compute() does the work for the first pass, by
putting placeholders before doing initial signing. Finalize()
corresponds to the second pass, where the placeholders get replaced with
actual data. Verify() can be optionally called to assert the correctness
of the work.

The separation between Compute() and Finalize() is to allow having
multiple StreamingPropertyFiles instances (in coming up CLs). This way
we can call Compute() multiple times for each instance, followed by only
one call to SignOutput(). And similarly for Finalize().

Bug: 74210298
Test: Generate an A/B OTA package. Check the METADATA entry.
Test: python -m unittest test_ota_from_target_files
Change-Id: I45be0372a4863c4405e6d8e20bcb9ccdc29e7e11
2018-03-08 14:59:42 -08:00
Tao Bao f6f13acf73 releasetools: Fix the broken validate_target_files.py.
The file was broken due to earlier touches:
 - Missing 'import zipfile';
 - Mismatching arguments when calling GetSparseImage().

Bug: 73996151
Test: Run validate_target_files.py with a walleye-target_files.zip.
Test: pylint --rcfile=pylintrc validate_target_files.py
Change-Id: I3692bd51fb27a3da698e06b75155e84502549f66
(cherry picked from commit c63626b4a3)
2018-03-08 08:10:28 -08:00
Baligh Uddin 8c7a2bb360 Merge "releasetools: Fix the broken validate_target_files.py." am: 1617db36af
am: 946752cb24

Change-Id: Id1f96e9c03a2154f707598ab0e9d13df1fe77136
2018-03-08 16:07:48 +00:00
Tao Bao c63626b4a3 releasetools: Fix the broken validate_target_files.py.
The file was broken due to earlier touches:
 - Missing 'import zipfile';
 - Mismatching arguments when calling GetSparseImage().

Bug: 73996151
Test: Run validate_target_files.py with a walleye-target_files.zip.
Test: pylint --rcfile=pylintrc validate_target_files.py
Change-Id: I3692bd51fb27a3da698e06b75155e84502549f66
2018-03-07 21:45:40 -08:00
Tao Bao b9ee298fc6 Merge "releasetools: Separate streaming metadata computation into functions." am: 737bc2b082
am: bfecf9b4cb

Change-Id: I47380d2479460fe9a18ff0b087a3a0a2523dc4c2
2018-03-07 04:40:32 +00:00
Tao Bao fe5b69a4ae releasetools: Separate streaming metadata computation into functions.
And add unittests for ComputeStreamingMetadata().

This prepares for the changes that add additional property-files (for
both of A/B and non-A/B).

Bug: 74210298
Bug: 72751683
Test: python -m unittest test_ota_from_target_files
Test: Generate A/B OTA package. Check the ota-streaming-property-files
      in the METADATA entry.
Change-Id: Ib4b069f61c2c06c035c0cff73a55112f3936b969
(cherry picked from commit f5110498c0)
2018-03-06 20:26:38 -08:00
Tao Bao f5110498c0 releasetools: Separate streaming metadata computation into functions.
And add unittests for ComputeStreamingMetadata().

This prepares for the changes that add additional property-files (for
both of A/B and non-A/B).

Bug: 74210298
Bug: 72751683
Test: python -m unittest test_ota_from_target_files
Test: Generate A/B OTA package. Check the ota-streaming-property-files
      in the METADATA entry.
Change-Id: Ib4b069f61c2c06c035c0cff73a55112f3936b969
2018-03-06 00:06:23 -08:00
Tao Bao 8e9155f236 Merge "releasetools: Fix a mismatching close." am: 395bf274b8 am: 06bc45b0c2
am: 6b93ed0c84

Change-Id: I74f9e797510bcecdb1f93824ce5881f92ff3b152
2018-03-03 00:10:31 +00:00
Tao Bao 6b93ed0c84 Merge "releasetools: Fix a mismatching close." am: 395bf274b8
am: 06bc45b0c2

Change-Id: I5e9a7d4a74e476b1ed86a9f7bc06e515998f81e2
2018-03-02 23:59:22 +00:00
Tao Bao 395bf274b8 Merge "releasetools: Fix a mismatching close." 2018-03-02 23:32:11 +00:00
Tao Bao 921c81f7c3 Merge "releasetools: Call Cleanup() in check_target_files_signatures.py." am: 903be914bb am: 5410d8a52a
am: 501b84236f

Change-Id: Ia4dfee212e3ff55fcdd9ce60faddab91c5e195c6
2018-03-02 23:05:20 +00:00
Tao Bao 501b84236f Merge "releasetools: Call Cleanup() in check_target_files_signatures.py." am: 903be914bb
am: 5410d8a52a

Change-Id: I3d015a9ffa1784c060d6a1d878feae9046090b5d
2018-03-02 22:55:11 +00:00
Tao Bao a652c00c8e releasetools: Fix a mismatching close.
There's a mismatch in WriteABOTAPackageWithBrilloScript().

  temp_zip_file = tempfile.NamedTemporaryFile()
  ...
  common.ZipClose(temp_zip_file)

It's benign since common.ZipClose() happens to be calling
"temp_zip_file.close()". This CL moves the use of tempfile to
common.MakeTempFile(), so that the tempfile will be cleaned up
automatically as part of the call to common.Cleanup(). (Not fixing the
close() directly, since the nearby lines will be refactored into another
function shortly.)

Also remove one assert in the same function, which trivially holds in
the current code.

Test: Generate an A/B OTA.
Change-Id: I53b375d1150820de718dec0ead55abf5f4951071
2018-03-02 13:20:43 -08:00
Tao Bao 150708b012 Merge "releasetools: Don't return ZipFile from common.UnzipTemp()." am: 5c93279fbe am: 0db979c6be
am: abe02acc48

Change-Id: Ie8d1d55d134a7aade1be643634641ef0eb80ff65
2018-03-01 22:44:21 +00:00
Tao Bao abe02acc48 Merge "releasetools: Don't return ZipFile from common.UnzipTemp()." am: 5c93279fbe
am: 0db979c6be

Change-Id: I9a4f2bc04851f048e168a11c039d781c2069b662
2018-03-01 22:38:46 +00:00
Tao Bao 767543a659 releasetools: Call Cleanup() in check_target_files_signatures.py.
Also minor cleanups to make it pylint clean.

Test: Run check_target_files_signatures.py with a target-files.zip.
Test: pylint --rcfile=pylintrc check_target_files_signatures.py
Change-Id: Ife3b54c7805c2f2562e87e91ab4b4de355782012
2018-03-01 10:12:39 -08:00
Tao Bao dba59eeabd releasetools: Don't return ZipFile from common.UnzipTemp().
In addition to the unzipping work, common.UnzipTemp() kindly bundles an
open ZipFile object as part of the return value. It doesn't look very
helpful to the callers though. It also looks less obvious that the
caller needs to properly close the handle (missing the close here is
benign though). This CL just removes the ZipFile object out of the
return value, and leaves the work to callers.

Test: `m dist` on both of A/B and non-A/B target.
Test: python -m unittest test_add_img_to_target_files
Test: python -m unittest test_common
Test: python -m unittest test_ota_from_target_files
Test: Check the callers to common.UnzipTemp() in code search.
Change-Id: Id47da3fd42a0e76d6ae8851f05780db319ee48cf
2018-03-01 10:03:31 -08:00
Tao Bao 7a08313a5e Merge "releasetools: Move one case of mkdtemp() to common.MakeTempDir()." am: a7a5e15f72 am: fc0bb788ed
am: e9f3ee0f80

Change-Id: Ie5cf8533ae02aab3f52c3652b2afc55c729e16f2
2018-02-28 21:37:09 +00:00
Tao Bao e9f3ee0f80 Merge "releasetools: Move one case of mkdtemp() to common.MakeTempDir()." am: a7a5e15f72
am: fc0bb788ed

Change-Id: Icb0ffb498e068417bdadabd4877731c63b776f37
2018-02-28 21:30:30 +00:00
Tao Bao 04bce3a696 releasetools: Move one case of mkdtemp() to common.MakeTempDir().
This should be the last case to be moved over.

Test: Generate an incremental BBOTA (which exercises the changed code).
Test: `rgrep mkdtemp` gives no more instance.
Change-Id: I76db069476201cdfaf3a2de9d9635dfe54507f7a
2018-02-28 11:13:29 -08:00
Tao Bao 9a90d8cae6 Merge "releasetools: Remove the unneeded 'prefix' parameters." am: 6e2a8ae094 am: 945e9c3e44
am: 5d74ac8574

Change-Id: I84554693ebd6be87eb0989d9b2d1df5f049c62f7
2018-02-27 22:36:18 +00:00
Tao Bao 5d74ac8574 Merge "releasetools: Remove the unneeded 'prefix' parameters." am: 6e2a8ae094
am: 945e9c3e44

Change-Id: Iee385b3b5d3a8f3848ccbfdd3773b7cf287f40cf
2018-02-27 22:31:24 +00:00
Tao Bao 6e2a8ae094 Merge "releasetools: Remove the unneeded 'prefix' parameters." 2018-02-27 22:13:06 +00:00
Tao Bao 886d883d61 releasetools: Remove the unneeded 'prefix' parameters.
... from the following functions in add_img_to_target_files.py.

  AddSystem
  AddSystemOther
  AddVendor
  AddProduct
  AddDtbo
  AddUserdata
  AddVBMeta
  AddPartitionTable
  AddCache

The last user of the parameter in img_from_target_files.py has been
removed in commit 2bb109709a (in O).

Test: pylint --rcfile=pylintrc add_img_to_target_files.py
Test: Check all the callers to the above functions.
Test: m dist
Change-Id: I551d1683def8f8535062fc90f68dafa0f4252822
2018-02-27 11:47:11 -08:00
Luis Hector Chavez 0fae5d0806 Merge "Make mksquashfs generate a fs_config-friendly root entry" am: 3d4fb4e335 am: 6128d7f30d
am: 21025515e4

Change-Id: I710cbaa8379c118e0c6a519f5d737a6395de2646
2018-02-27 16:41:46 +00:00
Luis Hector Chavez 21025515e4 Merge "Make mksquashfs generate a fs_config-friendly root entry" am: 3d4fb4e335
am: 6128d7f30d

Change-Id: I43a62ea7c4d6acb5adadfc488b94e05024499320
2018-02-27 16:36:39 +00:00
Treehugger Robot 3d4fb4e335 Merge "Make mksquashfs generate a fs_config-friendly root entry" 2018-02-27 16:24:21 +00:00
Dan Shi 9801d9b7b2 Merge "Default test to AndroidJUnitTest for auto-generated test config." am: ce1fa25d1b am: d91129eea9
am: 6dbd9e3df9

Change-Id: I84ee22fa20b08fb3db27d5e5d0e03f097a11463f
2018-02-26 23:26:06 +00:00
Dan Shi 6dbd9e3df9 Merge "Default test to AndroidJUnitTest for auto-generated test config." am: ce1fa25d1b
am: d91129eea9

Change-Id: Ib0e817e8d403ea130516dbc94201fd4a899c327f
2018-02-26 23:18:57 +00:00
Treehugger Robot ce1fa25d1b Merge "Default test to AndroidJUnitTest for auto-generated test config." 2018-02-26 23:00:28 +00:00
Tao Bao 47db251db0 Merge "releasetools: Remove the parsing of META/imagesizes.txt." am: 3aee4dfac1 am: 72f3c44243
am: ed98b94326

Change-Id: Id4f84a66faf7c06b4bbacae799d36db4a498182b
2018-02-26 22:46:52 +00:00
Tao Bao ed98b94326 Merge "releasetools: Remove the parsing of META/imagesizes.txt." am: 3aee4dfac1
am: 72f3c44243

Change-Id: I83af79b6d5a76099259a951585d92755b132af9e
2018-02-26 22:39:14 +00:00
Tao Bao 0835594a6f Merge "releasetools: Add unittests for GetCareMap() and AddCareMapTxtForAbOta()." am: 7998503ebe am: 5fed709d60
am: b8d0b5bd06

Change-Id: Iec88f7bdbf26047479d217d756d4847b1bac64f7
2018-02-26 22:32:18 +00:00
Tao Bao b8d0b5bd06 Merge "releasetools: Add unittests for GetCareMap() and AddCareMapTxtForAbOta()." am: 7998503ebe
am: 5fed709d60

Change-Id: I117d452b1a4d63b7665c3593b5b2167fe57be954
2018-02-26 22:24:13 +00:00
Treehugger Robot 3aee4dfac1 Merge "releasetools: Remove the parsing of META/imagesizes.txt." 2018-02-26 22:18:31 +00:00
Tao Bao 7998503ebe Merge "releasetools: Add unittests for GetCareMap() and AddCareMapTxtForAbOta()." 2018-02-26 22:06:32 +00:00
Tao Bao 41fb7d27cf releasetools: Remove the parsing of META/imagesizes.txt.
The file has been removed from target-files.zip since commit
c19a8d5590 (Gingerbread), whose info has
been consolidated into META/misc_info.txt.

Test: `m dist`
Change-Id: Ic144457954f5742ea082dcd9ffbea71df4afe46e
2018-02-23 12:06:07 -08:00
Tao Bao cf9c445ef9 Merge "releasetools: Allow skipping postinstall hooks when generating A/B OTAs." am: 6c0ddf2480 am: ff1288821d
am: a5baf56e6c

Change-Id: I59bf83ba415351ba07488086d8b812453b0ae61b
2018-02-22 20:49:01 +00:00
Tao Bao a5baf56e6c Merge "releasetools: Allow skipping postinstall hooks when generating A/B OTAs." am: 6c0ddf2480
am: ff1288821d

Change-Id: Iea818eb91dd5ff98d9b378a845ff507e1dee1e0c
2018-02-22 20:40:25 +00:00
Tao Bao 6c0ddf2480 Merge "releasetools: Allow skipping postinstall hooks when generating A/B OTAs." 2018-02-22 20:21:36 +00:00
Tao Bao 567fa9012b Merge "releasetools: Check for duplicate entries in ReplaceCerts()." am: 8bfde7c5fb am: 7032691417
am: 894cba2954

Change-Id: Ic89d331846654d23768e90cb4abab1b91d744271
2018-02-22 18:05:39 +00:00
Tao Bao 894cba2954 Merge "releasetools: Check for duplicate entries in ReplaceCerts()." am: 8bfde7c5fb
am: 7032691417

Change-Id: Icc23606930f1a2f3e0009b1ec225723ffc8a7617
2018-02-22 17:57:03 +00:00
Tao Bao 8bfde7c5fb Merge "releasetools: Check for duplicate entries in ReplaceCerts()." 2018-02-22 17:39:00 +00:00
Tao Bao 63c18fead8 releasetools: Add unittests for GetCareMap() and AddCareMapTxtForAbOta().
With test_utils.construct_sparse_image(), creating valid sparse image is
no longer a blocker for testing these functions.

Test: python -m unittest test_add_img_to_target_files
Change-Id: Iff1f15837cc744bad52e2120f9a9ad94d7db50d5
2018-02-22 00:10:05 -08:00
Tao Bao 15a146a7c4 releasetools: Allow skipping postinstall hooks when generating A/B OTAs.
This CL adds a new flag '--skip_postinstall' that allows skipping all
the postinstall hooks when generating an A/B OTA package (default:
False). Note that this discards ALL the hooks, including non-optional
ones. Should only be used if caller knows it's safe to do so (e.g. all
the postinstall work is to dexopt apps and a data wipe will happen
immediately after).

Bug: 73547992
Test: python -m unittest test_ota_from_target_files
Test: Generate a full OTA package for walleye. Examine the generated
      payload.
Change-Id: Ifc069e897b4019605051eabfd221230a6a37867c
2018-02-21 16:34:19 -08:00
Dan Shi 96068b77ed Default test to AndroidJUnitTest for auto-generated test config.
All new instrumentation tests shall use AndroidJUnitRunner, thus default
test class to AndroidJUnitTest.

Bug: 73679956
Test: unittest
Change-Id: I52df4d01b8459a4a6e5c2cbe15973dab079fb136
2018-02-21 11:31:06 -08:00
Tao Bao 039ccf14ca Merge "releasetools: Clean up test_sign_target_files_apks.py." am: 5ef78033d5 am: e1fee81bb5
am: 4e636f0441

Change-Id: I9133283350bd0d9c74d6d2b9b313a5fd1103aa48
2018-02-21 02:36:43 +00:00
Tao Bao 4e636f0441 Merge "releasetools: Clean up test_sign_target_files_apks.py." am: 5ef78033d5
am: e1fee81bb5

Change-Id: I3c504df0a047b88725433c68079c18495d536afe
2018-02-21 02:29:12 +00:00
Tao Bao 5ef78033d5 Merge "releasetools: Clean up test_sign_target_files_apks.py." 2018-02-21 02:06:00 +00:00
Tao Bao 90c3d2ebb5 Merge "releasetools: Add a testcase for common.ParseCertificate()." am: a0f55ee9eb am: 9773a2f7b2
am: 6f142ef096

Change-Id: Ic838ebf4e59d8d90eb102c0ae4399b639fcf64f6
2018-02-21 00:48:47 +00:00
Tao Bao 6f142ef096 Merge "releasetools: Add a testcase for common.ParseCertificate()." am: a0f55ee9eb
am: 9773a2f7b2

Change-Id: I3b00809f57c7f42f69fdbb44cdc5516e585ee987
2018-02-21 00:28:14 +00:00
Tao Bao a0f55ee9eb Merge "releasetools: Add a testcase for common.ParseCertificate()." 2018-02-21 00:04:24 +00:00
Tao Bao 4051a05b54 Merge "releasetools: Add unittest for common.GetSparseImage()." am: 35ee189898 am: 625b5b7a09
am: 0fa48c8c52

Change-Id: I12997ace8bfc5936a6db773b570c7ec2e26edb47
2018-02-20 21:27:34 +00:00
Tao Bao 0fa48c8c52 Merge "releasetools: Add unittest for common.GetSparseImage()." am: 35ee189898
am: 625b5b7a09

Change-Id: I494f253ab122f4098ec1e1fd814c915e618e2aa2
2018-02-20 21:21:31 +00:00
Treehugger Robot 35ee189898 Merge "releasetools: Add unittest for common.GetSparseImage()." 2018-02-20 21:04:58 +00:00
Tao Bao 66472637ad releasetools: Check for duplicate entries in ReplaceCerts().
testdata/{media,platform}.x509.pem files are copied from
build/target/product/security/.

Fixes: 69479366
Test: python -m unittest test_sign_target_files_apks
Change-Id: I8ba42b6f5b5432ee4b8786b241daff11db792c14
2018-02-20 10:22:30 -08:00
Tao Bao de1d479ace releasetools: Clean up test_sign_target_files_apks.py.
Since we have been carrying test certificates in testdata/ for other
tests, do the same for test_sign_target_files_apks.py. Copy
verity.x509.pem from build/target/product/security/ to testdata/ for
that purpose.

Also capture the stderr output in ReplaceVerityKeyId().

Test: python -m unittest test_sign_target_files_apks
Change-Id: Ie11e042086952e8a4a5a63950cb0b16cc436b7e6
2018-02-20 10:12:15 -08:00
Dan Willemsen dcdc1685c7 Merge "Add Docker instructions for older branches" am: c6620457d8 am: cfc82c4b33
am: aa582f7dda

Change-Id: I84b2706e169ecb4835cd1dd8193576a6001e750a
2018-02-17 01:27:40 +00:00
Dan Willemsen aa582f7dda Merge "Add Docker instructions for older branches" am: c6620457d8
am: cfc82c4b33

Change-Id: Ia6404fa198f502ad9be811a83ca19a7fee31bb17
2018-02-17 01:19:32 +00:00
Tao Bao 17e4e61691 releasetools: Add a testcase for common.ParseCertificate().
Also minor clean up to the function, to suppress the following lint
warning.
R:1813, 2: Redefinition of cert type from list to unicode (redefined-variable-type)

Test: python -m unittest -v test_common.CommonApkUtilsTest
Change-Id: Ib4b1dfa8b19e505fc70684b648efc36171c73bbf
2018-02-16 17:14:23 -08:00
Dan Willemsen 6dc4621afc Add Docker instructions for older branches
If you've upgraded away from Ubuntu 14.04 Trusty, you may find that
older branches (or certain code) doesn't build anymore. These
instructions can form a starting point to build these older branches
within Docker.

Bug: 72993575
Test: Follow instructions, build marshmallow-dev
Test: Follow instructions, build lollipop-mr1-dev
Change-Id: If4047b1bb5324d75a9d0947cb5280ff1cabccb6a
2018-02-15 18:17:46 -08:00
Tao Bao fc7e0e06b6 releasetools: Add unittest for common.GetSparseImage().
Add construct_sparse_image() to test_utils.py, which is a util function
to create sparse images. The new tests also partially cover the recent
changes that add 'incomplete' and 'uses_shard_blocks' tags.

Test: python -m unittest test_common
Change-Id: Ia15f5c4ad12423691216ebbad2c28f95c8427d7e
2018-02-14 15:12:52 -08:00
TreeHugger Robot 221d1f4bde Merge "releasetools: Specify max_timestamp for secondary payload." into oc-mr1-dev-plus-aosp 2018-02-14 00:26:57 +00:00
Tao Bao 9d0242d93e Merge "releasetools: Allow generating BBOTA for images with shared blocks." am: 9452d84b97 am: 6c453193a1
am: 7088b88a17

Change-Id: I123238db80af7d60e92da4aef0478f9f5f1a29a5
2018-02-13 23:13:15 +00:00
Tao Bao 072f2e4784 Merge "releasetools: Remove the unconditional fallback to bsdiff." am: 6392e05813 am: 61fc7f6746
am: 6b2854d44d

Change-Id: Iac4c743329de3db006ed4a2d3782835a5002490b
2018-02-13 23:10:28 +00:00
Tao Bao 7088b88a17 Merge "releasetools: Allow generating BBOTA for images with shared blocks." am: 9452d84b97
am: 6c453193a1

Change-Id: I422f6da0c524fa5329bc9dd11eb10eb27945ab92
2018-02-13 22:43:10 +00:00
Tao Bao 6b2854d44d Merge "releasetools: Remove the unconditional fallback to bsdiff." am: 6392e05813
am: 61fc7f6746

Change-Id: I3729f15bb532838ce2b9ec3f084ee5b81e97d3b1
2018-02-13 22:35:42 +00:00
Tao Bao db1fe41bbe releasetools: Specify max_timestamp for secondary payload.
This is a mandatory field in the generated payload.

Bug: 35724498
Test: Generate OTA with --include_secondary. Check the generation
      command for secondary payload.
Change-Id: Ib2e25cc020e294eae5a4be36d6dca297432804b8
Merged-In: Ib2e25cc020e294eae5a4be36d6dca297432804b8
(cherry picked from commit 6048121485)
2018-02-13 14:34:55 -08:00
Tao Bao 6048121485 releasetools: Specify max_timestamp for secondary payload.
This is a mandatory field in the generated payload.

Bug: 35724498
Test: Generate OTA with --include_secondary. Check the generation
      command for secondary payload.
Change-Id: Ib2e25cc020e294eae5a4be36d6dca297432804b8
2018-02-13 14:17:17 -08:00
Tao Bao d1239cd452 Merge "releasetools: Specify SWITCH_SLOT_ON_REBOOT for secondary payload." am: 4f0b725439 am: 78eef74e03
am: 55e571f0b6

Change-Id: I0bb607e9ad51dd8624493a82c3d5c2b2167abd43
2018-02-13 22:12:24 +00:00
Tao Bao 55e571f0b6 Merge "releasetools: Specify SWITCH_SLOT_ON_REBOOT for secondary payload." am: 4f0b725439
am: 78eef74e03

Change-Id: I02007c63b6a7398afe8b5304e48c01589842a876
2018-02-13 22:06:40 +00:00
Tao Bao 9452d84b97 Merge "releasetools: Allow generating BBOTA for images with shared blocks." 2018-02-13 21:53:25 +00:00
Tao Bao 6392e05813 Merge "releasetools: Remove the unconditional fallback to bsdiff." 2018-02-13 21:53:17 +00:00
Tao Bao 4f0b725439 Merge "releasetools: Specify SWITCH_SLOT_ON_REBOOT for secondary payload." 2018-02-13 20:20:30 +00:00
Tao Bao e709b094e4 releasetools: Allow generating BBOTA for images with shared blocks.
When target defines 'BOARD_EXT4_SHARE_DUP_BLOCKS := true', the generated
system/vendor images may contain shared blocks (i.e. some blocks will
show up in multiple files' block list), which violates the current
assumptions in BBOTA script.

This CL allows generating BBOTAs by considering the first occurrence as
the "owner" of the shared blocks. All the later users of the shared
blocks will have an incomplete block list, whose RangeSet's will be
tagged with 'uses_shared_blocks'.

Files with 'uses_shared_blocks' tag will not be diff'd with imgdiff,
potentially with patch size penalty. Such files will be accounted for in
imgdiff stats report, where we can revisit for a better solution.

Bug: 64109868
Test: Generate BBOTA full and incremental package with targets defining
      'BOARD_EXT4_SHARE_DUP_BLOCKS := true'.
Change-Id: I87fbc22eef7fafe2a470a03fdcfa1babf088ea8d
2018-02-13 11:14:23 -08:00
Tao Bao 4ccea8549e releasetools: Remove the unconditional fallback to bsdiff.
This CL uses the 'incomplete' tag to skip applying imgdiff to files with
incomplete block list. It's not the ideal fix to address the holes in
ext4 images, but would unhide other imgdiff issues covered by the
unconditional fallback.

Bug: 68016761
Test: Generate an incremental OTA package from images with incomplete
      block list. Check the imgdiff stats report.
Test: `python -m unittest test_blockimgdiff`
Change-Id: Ice77686414e70f5e42de35c1757fb31cf02e4fd4
2018-02-13 11:06:20 -08:00
Tao Bao 3b67cdf291 Merge changes Ia7ecd0e1,I33982473 am: 4a066a2cd2 am: b25e43bfcb
am: 000274fda1

Change-Id: I360a5b5eeda4a3cfd9bd1145f36d030118eb8361
2018-02-13 17:36:22 +00:00
Tao Bao 000274fda1 Merge changes Ia7ecd0e1,I33982473 am: 4a066a2cd2
am: b25e43bfcb

Change-Id: I52be4244cfb6faca30597a887a5abf7c5fed326f
2018-02-13 17:24:17 +00:00
Tao Bao 508b087943 releasetools: Make blockimgdiff.py pylint-clean.
************* Module blockimgdiff
C:433, 0: Unnecessary parens after 'if' keyword (superfluous-parens)
C:687, 0: Wrong hanging indentation (add 4 spaces).
            max_stashed_blocks, self._max_stashed_size, max_allowed,
            ^   | (bad-continuation)
C:688, 0: Wrong hanging indentation (add 4 spaces).
            self._max_stashed_size * 100.0 / max_allowed))
            ^   | (bad-continuation)
C:691, 0: Wrong hanging indentation (remove 2 spaces).
            max_stashed_blocks, self._max_stashed_size))
          | ^ (bad-continuation)
C:898, 0: Wrong hanging indentation (add 4 spaces).
                  "imgdiff" if imgdiff else "bsdiff",
                  ^   | (bad-continuation)
C:899, 0: Wrong hanging indentation (add 4 spaces).
                  xf.tgt_name if xf.tgt_name == xf.src_name else
                  ^   | (bad-continuation)
C:901, 0: Wrong hanging indentation (add 4 spaces).
                  xf.tgt_ranges, xf.src_ranges, e.message))
                  ^   | (bad-continuation)
C:909, 0: Wrong hanging indentation (add 4 spaces).
                      xf.tgt_name,))
                      ^   | (bad-continuation)
C:917, 0: Wrong hanging indentation (add 4 spaces).
                      xf.tgt_name, e.message))
                      ^   | (bad-continuation)
C:961, 0: Wrong hanging indentation (remove 2 spaces).
                xf.patch_len, tgt_size, xf.patch_len * 100.0 / tgt_size,
              | ^ (bad-continuation)
C:962, 0: Wrong hanging indentation (remove 2 spaces).
                xf.style,
              | ^ (bad-continuation)
C:963, 0: Wrong hanging indentation (remove 2 spaces).
                xf.tgt_name if xf.tgt_name == xf.src_name else (
              | ^ (bad-continuation)
C:965, 0: Wrong hanging indentation (remove 2 spaces).
                xf.tgt_ranges, xf.src_ranges))
              | ^ (bad-continuation)
C:1422, 0: Wrong continued indentation (add 28 spaces).
                tgt_skipped.size() * 100.0 / tgt_size, tgt_name))
                ^                           | (bad-continuation)
C:1560, 0: Wrong continued indentation (add 8 spaces).
            split_src_ranges) in enumerate(split_info_list):
            ^       | (bad-continuation)
R:566, 6: Redefinition of src_str type from list to str (redefined-variable-type)
C:1198,25: More than one statement on a single line (multiple-statements)
C:1211,25: More than one statement on a single line (multiple-statements)
C:1220,16: More than one statement on a single line (multiple-statements)
C:1277,38: More than one statement on a single line (multiple-statements)
C:1284,19: More than one statement on a single line (multiple-statements)
C: 19, 0: standard import "import copy" comes before "import common" (wrong-import-order)
C: 20, 0: standard import "import functools" comes before "import common" (wrong-import-order)
C: 21, 0: standard import "import heapq" comes before "import common" (wrong-import-order)
C: 22, 0: standard import "import itertools" comes before "import common" (wrong-import-order)
C: 23, 0: standard import "import multiprocessing" comes before "import common" (wrong-import-order)
C: 24, 0: standard import "import os" comes before "import common" (wrong-import-order)
C: 25, 0: standard import "import os.path" comes before "import common" (wrong-import-order)
C: 26, 0: standard import "import re" comes before "import common" (wrong-import-order)
C: 27, 0: standard import "import subprocess" comes before "import common" (wrong-import-order)
C: 28, 0: standard import "import sys" comes before "import common" (wrong-import-order)
C: 29, 0: standard import "import threading" comes before "import common" (wrong-import-order)
C: 31, 0: standard import "from collections import deque, OrderedDict" comes before "import common" (wrong-import-order)
C: 32, 0: standard import "from hashlib import sha1" comes before "import common" (wrong-import-order)

Test: `pylint --rcfile=pylintrc blockimgdiff.py`
Test: Generate an incremental BBOTA package.
Change-Id: Ia7ecd0e1fa48daf4e43251bdcdfcd08fb316015d
2018-02-12 13:01:34 -08:00
Tao Bao 9739514769 releasetools: Remove the global diff_done in blockimgdiff.py.
pylint complains about undefined `diff_done`:

W:754, 8: Global variable 'diff_done' undefined at the module level (global-variable-undefined)
W:820,14: Global variable 'diff_done' undefined at the module level (global-variable-undefined)

It would still warn about using global statement after adding the
definition.

W:859, 8: Using the global statement (global-statement)
W:925,14: Using the global statement (global-statement)

This CL computes 'diff_done' via 'len(diff_queue)' instead. It also
moves the progress reporting _before_ the diff work. This way it avoids
showing 100% progress with still changing filenames (because multiple
workers could see an empty queue simultaneously upon finishing their own
works).

There're possible alternatives, such as using the 'nonlocal' keyword in
Python 3 (which we're not there yet), or by using mutable object instead
(e.g. 'diff_done = [0]'). This CL looks cleaner, since it just kills the
var.

Test: Generate a BBOTA incremental. Check the on-screen progress
      report.
Test: `pylint --rcfile=pylintrc blockimgdiff.py` no longer complains
      about the global diff_done.
Change-Id: I339824735527e1f794b5b1dc99ff3fdb2da85744
2018-02-12 12:55:25 -08:00
Tao Bao a3ee598230 Merge "releasetools: Add an ImgdiffStats class that reports imgdiff stats." am: 7eb2afb226 am: 0ced5030d8
am: e6188063f1

Change-Id: I34118e0e88042ef8477a071f6819483746b0038c
2018-02-12 20:48:54 +00:00
Tao Bao e6188063f1 Merge "releasetools: Add an ImgdiffStats class that reports imgdiff stats." am: 7eb2afb226
am: 0ced5030d8

Change-Id: I120d8d268fc4a5e1230e16d9b9ab85b2bef20193
2018-02-12 20:28:11 +00:00
Tao Bao 7eb2afb226 Merge "releasetools: Add an ImgdiffStats class that reports imgdiff stats." 2018-02-12 19:01:24 +00:00
Tom Cherry b01d6d33bc Merge "Place TARGET_FS_CONFIG_GEN passwd/group files in /vendor/etc" am: 3d9fa3615d am: 57dd54f623
am: c2ab6232d9

Change-Id: I8140e02a61429074840fa7b6dabdbc048de5c9f4
2018-02-12 16:46:26 +00:00
Tom Cherry c2ab6232d9 Merge "Place TARGET_FS_CONFIG_GEN passwd/group files in /vendor/etc" am: 3d9fa3615d
am: 57dd54f623

Change-Id: Ica4d341247fc0f770ccdfafd165df05d70f39772
2018-02-12 16:38:43 +00:00
Tom Cherry 3d9fa3615d Merge "Place TARGET_FS_CONFIG_GEN passwd/group files in /vendor/etc" 2018-02-12 16:27:20 +00:00
Tao Bao 667ff57272 releasetools: Specify SWITCH_SLOT_ON_REBOOT for secondary payload.
The secondary payload should always be applied with
SWITCH_SLOT_ON_REBOOT=0. This CL moves the 'secondary' parameter from
Payload.WriteToZip() to Payload.__init__(). So it can append the flag to
secondary/payload_properties.txt.

Bug: 35724498
Test: Generate an A/B OTA with --include_secondary. Check
      secondary/payload_properties.txt entry in the generated ZIP.
Test: `python -m unittest test_ota_from_target_files`
Change-Id: I816c07ab57a1c8a52eff785801634b8b1cb134d4
2018-02-10 00:13:07 -08:00
Tao Bao 294651d7b4 releasetools: Add an ImgdiffStats class that reports imgdiff stats.
We have a couple of active imgdiff workarounds (and likely with one more
inbounding that allows having shared blocks in ext4 image). Most of
these workarounds need extending imgdiff's capability. While us not
getting there anytime soon, collect the stats to better understand the
impact of each kind so we can prioritize accordingly.

A sample report is as follows.

  Imgdiff Stats Report
========================

 APK files diff'd with imgdiff (count: 88)
-------------------------------------------

  /system/priv-app/Shell/Shell.apk
  ...

 Large APK files split and diff'd with imgdiff (count: 4)
----------------------------------------------------------

  /system/priv-app/Settings/Settings.apk
  ...

Bug: 68016761
Test: Generate an incremental BBOTA package. Check the stats report.
Test: python -m unittest test_blockimgdiff
Change-Id: I27ad862cde472ab2806db877632ce5a0607420f2
2018-02-09 23:02:32 -08:00
Tao Bao f8255b496b Merge "releasetools: Refactor the condition checking for using imgdiff." am: bec8be51a5 am: 4a79eb0d4e
am: 194e49e06c

Change-Id: Ib383bbb47effe204fa69aac1809df08a75ffc1bc
2018-02-10 02:34:51 +00:00
Tao Bao 194e49e06c Merge "releasetools: Refactor the condition checking for using imgdiff." am: bec8be51a5
am: 4a79eb0d4e

Change-Id: Ia050150657bb373e92bf9797764ef987493e9c96
2018-02-10 02:16:54 +00:00
Tao Bao bec8be51a5 Merge "releasetools: Refactor the condition checking for using imgdiff." 2018-02-10 00:47:42 +00:00
Tao Bao cb73aed1f0 releasetools: Refactor the condition checking for using imgdiff.
In Transfer class, unbundle 'intact' with the monotonicity of the input
ranges. Negate the logic of 'intact', and thus rename it to 'trimmed'.
Move this property from an attribute of Transfer class as the one in
RangeSet.extra. 'trimmed' indicates whether the source / target ranges
have been modified after creating the Transfer() instance.

The logic that determines whether we can apply imgdiff has been
refactored and consolidated into BlockImageDiff.CanUseImgdiff(). Now
both of the two paths call this single copy, i.e. the one that detects
large APKs (before creating Transfer()'s), and the other that's about to
generate the patch for a given Transfer instance.

Bug: 68016761
Test: python -m unittest test_blockimgdiff
Test: Generate an incremental BBOTA package.
Change-Id: Id07195f63f1fa6c3af6e9091940d251cf09fa104
2018-02-09 12:46:01 -08:00
Tom Cherry 330670267e Merge "Add compile time check that friendly AID names are < 32 characters" am: 9f25d873bf am: cfac68eb9c
am: af0a12eded

Change-Id: I18b590bd8dc5f7f755ff391c8fbc20c53d03632c
2018-02-09 19:05:42 +00:00
Tom Cherry af0a12eded Merge "Add compile time check that friendly AID names are < 32 characters" am: 9f25d873bf
am: cfac68eb9c

Change-Id: I52c488b8fc0b6e42ef379b517cc816fc8e759738
2018-02-09 19:00:04 +00:00
Tom Cherry ee0610e86c Add compile time check that friendly AID names are < 32 characters
There is an internal buffer in bionic for user/group names that is 32
characters long including the trailing null, so we must restrict the
length of these names during compile time.

Bug: 27999086
Test: Successfully compile a valid config.fs
Test: Fail to compile a config.fs with AID name > 31 characters
Change-Id: I7fe887c630dd4d1033b86a5d8332480eb3b0fa07
2018-02-09 18:46:13 +00:00
Tom Cherry 55d0104f1b Place TARGET_FS_CONFIG_GEN passwd/group files in /vendor/etc
These entries are vendor provided and belong on the /vendor partition.

Bug: 27999086
Test: end to end user/group check via config.fs and getpwnam, etc.
Change-Id: I9a5d56da594bf0d04de2b9ce7fd7d9a8151d4682
2018-02-09 10:44:52 -08:00
Tao Bao f980e046e0 Merge "releasetools: RangeSet.monotonic is not an optional attribute." am: acb3cecc46 am: c6b824ba96
am: 93c09cb740

Change-Id: I87bd661a6ac967f30beb84fafcfc8bfc9bb6fe1a
2018-02-08 06:09:44 +00:00
Tao Bao 93c09cb740 Merge "releasetools: RangeSet.monotonic is not an optional attribute." am: acb3cecc46
am: c6b824ba96

Change-Id: I58685e0b462964078147c5bc906d52d11d74668c
2018-02-08 05:17:24 +00:00
Tao Bao acb3cecc46 Merge "releasetools: RangeSet.monotonic is not an optional attribute." 2018-02-08 04:45:44 +00:00
Luis Hector Chavez 440da2d41a Make mksquashfs generate a fs_config-friendly root entry
This change adds a flag so that the mksquashfs wrapper scripts generate
an entry that is fs_config-friendly for the root inode. Namely, the root
entry is expected to have an empty filename.

Bug: 72745016
Test: m -j100 dist
Change-Id: Iebdf79c5af0b9d999b7e5f5fe240abfe52cbadda
2018-02-07 10:22:52 -08:00
Tao Bao f68098ff88 Merge "releasetools: Support packaging secondary payload." am: 67ba60029d am: b7d3649c14
am: adaec07d5a

Change-Id: Ic1e8e8a872b23e3376ca1feba5b0fa7fd6029576
2018-02-07 08:42:16 +00:00
Tao Bao adaec07d5a Merge "releasetools: Support packaging secondary payload." am: 67ba60029d
am: b7d3649c14

Change-Id: Icf9c40ccdd4f59ace60a272ba354c6e145194dbe
2018-02-07 05:38:22 +00:00
Tao Bao 67ba60029d Merge "releasetools: Support packaging secondary payload." 2018-02-07 05:24:46 +00:00
Tao Bao fe97dbd4ce releasetools: RangeSet.monotonic is not an optional attribute.
'monotonic' has been non-optional since [1] (L-MR1). Fix the comment in
RangeSet.parse(), as well as the use in blockimgdiff.py.

[1] commit 8b72aefb5a.

Test: Generate an incremental BBOTA package.
Change-Id: I7f95231683473b4f0f07f9c83fccc0e36a1340cb
2018-02-06 16:00:52 -08:00
Tao Bao 3b45d28ff6 Merge "releasetools: Capture stderr output when calling delta_generator." am: 26fc64da06 am: 1f49ccfd61
am: 58e79345f3

Change-Id: Ife6761f4f0891ee8377721ea9e4478c233b6cc37
2018-02-06 17:35:19 +00:00
Tao Bao 58e79345f3 Merge "releasetools: Capture stderr output when calling delta_generator." am: 26fc64da06
am: 1f49ccfd61

Change-Id: I689d61b9ee186f01be4dd57a5cd1dbbdd530ea9b
2018-02-06 17:31:02 +00:00
Tao Bao 3f15ada51b releasetools: Capture stderr output when calling delta_generator.
Prior to this CL, the call to delta_generator in
check_ota_package_signature.VerifyAbOtaPayload() didn't redirect stderr.
The logs (mostly INFO) on successful verification added noise to the
normal output, which also upset the unittest result parser.

This CL captures stderr outputs from delta_generator, and will only dump
them on error.

Bug: 72884343
Test: `python -m unittest -v test_ota_from_target_files > /dev/null`
      gives clean output.
Test: Inject error into delta_generator. The call to
      check_ota_package_signature correctly dumps both of stdout and
      stderr outputs.
Change-Id: I014a4b21bf758dcf0a4b9963259d6019851935ee
2018-02-05 23:39:23 -08:00
Tao Bao 22f78fcf61 Merge "releasetools: Fix an issue with pubkey extraction." am: ca2ffed06c am: 8a05f343c2
am: 4c67bb3a13

Change-Id: I306105a5e2266de39a98a197eaa0a5584ad7fbd5
2018-02-06 02:28:40 +00:00
Tao Bao 4c67bb3a13 Merge "releasetools: Fix an issue with pubkey extraction." am: ca2ffed06c
am: 8a05f343c2

Change-Id: Ief767e449ecfcb01f82aed47bfd6f0e603dae77f
2018-02-06 02:21:04 +00:00
Tao Bao f7140c0f8c releasetools: Support packaging secondary payload.
By default, an A/B OTA package doesn't contain the images for the
secondary slot (e.g. system_other.img). Specifying
"--include_secondary" that's introduced in this CL allows generating
a separate payload that will install secondary slot images. Both
payloads will be added to the generated A/B OTA package.

An example A/B OTA package with secondary payload
  |
  +-- payload.bin
  |
  +-- payload_properties.txt
  |
  +-- secondary/payload.bin
  |
  +-- secondary/payload_properties.txt
  |
  +-- ...

Such a package needs to be applied in a two-stage manner. During the
first stage, the updater applies the primary payload only. Upon
finishing, it reboots the device into the newly updated slot. It then
continues to install the secondary payload to the inactive slot, but
without switching the active slot at the end (needs the matching support
in update_engine, i.e. SWITCH_SLOT_ON_REBOOT flag).

Due to the special install procedure, the secondary payload will be
always generated as a full payload.

Bug: 35724498
Test: Generate full and incremental OTAs with --include_secondary. Check
      the generated OTAs.
Test: python -m unittest test_ota_from_target_files
Change-Id: I975e826bec492e86eb400f99de0c355a32420127
2018-02-05 13:28:52 -08:00
Treehugger Robot ca2ffed06c Merge "releasetools: Fix an issue with pubkey extraction." 2018-02-05 21:25:47 +00:00
Tao Bao 891ee8af2d Merge "releasetools: Add sdk and security patch level info to metadata." am: 993e1d2634 am: d00a176e22
am: 1ca91dae98

Change-Id: Ic9e823cb13bf10e48b87a2290dff0deef3f3eb77
2018-02-05 17:35:36 +00:00
Tao Bao 1ca91dae98 Merge "releasetools: Add sdk and security patch level info to metadata." am: 993e1d2634
am: d00a176e22

Change-Id: I16652f7b351200c705c1a5154bf31d150ce5f719
2018-02-05 17:32:27 +00:00
Treehugger Robot 993e1d2634 Merge "releasetools: Add sdk and security patch level info to metadata." 2018-02-05 17:25:16 +00:00
Tao Bao 04e1f012dd releasetools: Fix an issue with pubkey extraction.
When calling 'openssl x509 -pubkey' to extract the public key from a
certificate, openssl 1.0 and 1.1 handle the '-out' parameter
differently. openssl 1.0 doesn't write the output into the specified
filename, which leads to the payload verification failure in
check_ota_package_signature.VerifyAbOtaPayload(). This CL addresses
the issue by always collecting the output from stdout instead.

It also refactors the two copies into common.ExtractPublicKey(), and
adds unittest. get_testdata_dir() is moved into test_utils.py that holds
common utils for running the unittests.

Bug: 72884343
Test: python -m unittest test_common
Test: python -m unittest test_ota_from_target_files
Test: Run sign_target_files_apks with '--replace_ota_keys' on marlin
      target_files zip. Check the payload pubkey replacement.
Test: Trigger the tests with forrest, and tests no longer fail on
      machines with openssl 1.0.1.
Change-Id: Ib0389b360f064053e9aa7cc0546d718e7b23003b
2018-02-04 13:59:52 -08:00
Tao Bao 656ab5d7fb Merge "releasetools: Detect incomplete block ranges." am: 93db955964 am: dcb307c4bd
am: 405e952ec3

Change-Id: I5a76dc5589741be7971ddc134b23d2f6803eea75
2018-02-03 05:17:00 +00:00
Tao Bao 405e952ec3 Merge "releasetools: Detect incomplete block ranges." am: 93db955964
am: dcb307c4bd

Change-Id: I1f51c9567df5917ea9dfa6abe49a612c576bd6ea
2018-02-03 04:41:15 +00:00
Tao Bao 93db955964 Merge "releasetools: Detect incomplete block ranges." 2018-02-03 04:32:13 +00:00
Tao Bao c7b403a2e8 releasetools: Add Payload class.
This breaks down the current WriteABOTAPackageWithBrilloScript() into
smaller and testable units, which also prepares for the work in
b/35724498.

Bug: 35724498
Test: python -m unittest test_ota_from_target_files
Test: Get identical A/B OTA packages w/ and w/o the CL.
Change-Id: I2ea45ce98e2d2baa58e94fb829b7242f6fe685a7
Merged-In: I2ea45ce98e2d2baa58e94fb829b7242f6fe685a7
(cherry picked from commit 036d721812)
2018-02-02 16:18:07 -08:00
Tao Bao 40b1882f40 releasetools: Add Payload class.
This breaks down the current WriteABOTAPackageWithBrilloScript() into
smaller and testable units, which also prepares for the work in
b/35724498.

Bug: 35724498
Test: python -m unittest test_ota_from_target_files
Test: Get identical A/B OTA packages w/ and w/o the CL.
Change-Id: I2ea45ce98e2d2baa58e94fb829b7242f6fe685a7
(cherry picked from commit 036d721812)
2018-02-02 23:21:16 +00:00
Tao Bao 036d721812 releasetools: Add Payload class.
This breaks down the current WriteABOTAPackageWithBrilloScript() into
smaller and testable units, which also prepares for the work in
b/35724498.

Bug: 35724498
Test: python -m unittest test_ota_from_target_files
Test: Get identical A/B OTA packages w/ and w/o the CL.
Change-Id: I2ea45ce98e2d2baa58e94fb829b7242f6fe685a7
2018-02-02 12:24:26 -08:00
Tao Bao 4258bcc17b Merge "releasetools: Clean up two no-op lines that set 'ota-wipe'." am: 76c6906bf2 am: 6c83183400
am: 73b6f5d661

Change-Id: Ia918010e2c40dd39bd37042d7457fb66f0e7f6da
2018-02-02 20:17:50 +00:00
Tao Bao 73b6f5d661 Merge "releasetools: Clean up two no-op lines that set 'ota-wipe'." am: 76c6906bf2
am: 6c83183400

Change-Id: I494b4519bd57e00237a53d29e46d935abdee1052
2018-02-02 19:04:05 +00:00
Tao Bao 91391637f9 Merge "releasetools: Make validate_target_files.py pylint clean." am: a4c7d59afc am: 41a3a414f0
am: dc68afc7f0

Change-Id: Ic5ea0d13816cfa889608e48414e34763b61c8112
2018-02-02 14:46:15 +00:00
Tao Bao dc68afc7f0 Merge "releasetools: Make validate_target_files.py pylint clean." am: a4c7d59afc
am: 41a3a414f0

Change-Id: I5fdbf2a22e7219846e5673c1368b018b414cbf50
2018-02-02 06:59:17 +00:00
Tao Bao 24604cc5b9 releasetools: Clean up two no-op lines that set 'ota-wipe'.
The same thing has been set in GetPackageMetadata(), so they are no-op.

Also replace an occurrence of 'OPTIONS.source_info_dict' with
source_info, which has been missed from previous clean-up CLs.

Test: Generate an A/B OTA package, and an incremental BBOTA with
      --wipe_user_data. Check the metadata.
Test: Generate a two-step incremental BBOTA successfully.
Change-Id: I4bb491cac9064d93fb86d12e617c8f38f040e01e
2018-02-01 16:30:42 -08:00
Tao Bao 35dc255180 releasetools: Add sdk and security patch level info to metadata.
The info comes from the build.prop file of the target build (thus no
backward compatibility concerns). OTA server and client can optionally
use these info to understand the expected behavior of an OTA package.

Bug: 72751683
Test: python -m unittest test_ota_from_target_files
Test: `m dist`, then check the metadata in the generated OTA package.
Change-Id: I5935f67684d2486bb5f00d67ce4bc756589a56ed
2018-02-01 15:58:20 -08:00
Tao Bao c765cca38b releasetools: Detect incomplete block ranges.
This CL detects incomplete block ranges (e.g. due to the holes in
mke2fs created images). Such block ranges will be tagged, so we won't
attempt to imgdiff those files. Note that the change to blockimgdiff.py,
which uses the tag info, will come in a separate CL.

An 'extra' attribute is added to RangeSet class, which defaults to an
empty dict. An 'incomplete' tag will be added into the dict by the
caller of the class. Not adding this tag as an immediate attribute,
because it is not a property regarding the ranges being represented, but
rather some storage space for the caller.

This CL also refactors GetSparseImage and RoundUpTo4K into common.py, so
the same code can be called from both of ota_from_target_files.py and
validate_target_files.py. Not able to add unittests for
GetSparseImage(), as SparseImage requires data in specific format.

Bug: 68016761
Test: Run validate_target_files.py on target-files.zip. It skips
      validating files with missing holes as before.
Test: Run ota_from_target_files.py on angler target-files.zip. It gives
      identical packages w/ and w/o the CL.
Test: pylint on changed files. There're warnings with common.py, but
      unrelated to this change.
Change-Id: I126ccfea13c0d5ebcc8c1b4ff1a4f9200e97423a
2018-02-01 12:35:48 -08:00
Tao Bao bb20e8c5f2 releasetools: Make validate_target_files.py pylint clean.
C: 73, 0: Wrong hanging indentation (add 4 spaces).
      file_name, actual_sha1, expected_sha1)
      ^   | (bad-continuation)
C:171, 0: Wrong continued indentation (add 20 spaces).
        'SYSTEM/etc/recovery.img', expected_recovery_sha1)
        ^                   | (bad-continuation)
C:185, 0: Wrong continued indentation (add 20 spaces).
        file_path='IMAGES/boot.img', expected_sha1=boot_info[3])
        ^                   | (bad-continuation)
C:191, 0: Wrong continued indentation (add 20 spaces).
        file_path='IMAGES/recovery.img',
        ^                   | (bad-continuation)
C:192, 0: Wrong continued indentation (add 20 spaces).
        expected_sha1=expected_recovery_sha1)
        ^                   | (bad-continuation)
W: 67,15: Use % formatting in logging functions and pass the % parameters as arguments (logging-format-interpolation)
W:150,17: Use % formatting in logging functions and pass the % parameters as arguments (logging-format-interpolation)
W:153,15: Use % formatting in logging functions and pass the % parameters as arguments (logging-format-interpolation)
W:194,15: Use % formatting in logging functions and pass the % parameters as arguments (logging-format-interpolation)
C: 27, 0: standard import "import logging" comes before "import common" (wrong-import-order)
C: 28, 0: standard import "import os.path" comes before "import common" (wrong-import-order)
C: 29, 0: standard import "import re" comes before "import common" (wrong-import-order)
C: 31, 0: standard import "import sys" comes before "import common" (wrong-import-order)

Test: pylint --rcfile=pylintrc validate_target_files.py
Test: Run validate_target_files.py with a target-files.zip.
Change-Id: Ie64acdb4cee4326938c4ad5a34b575d7b82478c0
2018-02-01 12:03:17 -08:00
Tao Bao d870401863 Merge "releasetools: Move the AVB salt setup into common.LoadInfoDict()." am: 4b4c3d91b0 am: 19bd39fe3f
am: d2bc1b9718

Change-Id: I995829e85bfdfb7024212e1c6427c00934db303d
2018-02-01 00:48:07 +00:00
Tao Bao d2bc1b9718 Merge "releasetools: Move the AVB salt setup into common.LoadInfoDict()." am: 4b4c3d91b0
am: 19bd39fe3f

Change-Id: I4b8271f022b7c27420cf8b0b6532b2cced0bd44f
2018-02-01 00:26:52 +00:00
Tao Bao 12d87fc174 releasetools: Move the AVB salt setup into common.LoadInfoDict().
We used to do this in add_img_to_target_files.AddImagesToTargetFiles(),
which didn't cover the path when calling from make_recovery_patch. As a
result, /system/bin/install-recovery.sh contains different SHA values
from the actual images.

Test: Set up aosp_bullhead to use AVB. `m dist`, then run the following
      command to verify the generated install-recovery.sh.

  $ ./build/make/tools/releasetools/validate_target_files.py \
      out/dist/aosp_bullhead-target_files-eng.zip

Change-Id: Id7be8fb17072252fcd4d08db2057b8c4af053376
2018-01-31 12:25:10 -08:00
Tianjie Xu 54ec40b263 Merge "Protect SparseImage._GetRangeData() with lock" am: 5edd2821de am: e517144ac9
am: c3b193255b

Change-Id: I752d1921fddc4de293698f42973930d4bca72e9e
2018-01-30 23:51:38 +00:00
Tianjie Xu c3b193255b Merge "Protect SparseImage._GetRangeData() with lock" am: 5edd2821de
am: e517144ac9

Change-Id: I075d7daacaf92def5233a18243f12cde4ff8db6f
2018-01-30 23:49:35 +00:00
Tianjie Xu 5edd2821de Merge "Protect SparseImage._GetRangeData() with lock" 2018-01-30 23:34:50 +00:00
Andreas Gampe a9e3f4f36c Merge "Warn.py: Update errorprone categories" am: 709ee786cd am: 2192dda42a
am: 7417ece30a

Change-Id: I1f62020caa042fa00533e13da14c69dd1aeecccf
2018-01-30 02:16:04 +00:00
Andreas Gampe 7417ece30a Merge "Warn.py: Update errorprone categories" am: 709ee786cd
am: 2192dda42a

Change-Id: If813b0e077ecf356ac82ece41a9eef2bde6ab242
2018-01-30 02:13:56 +00:00
Treehugger Robot 709ee786cd Merge "Warn.py: Update errorprone categories" 2018-01-30 02:05:19 +00:00
Tianjie Xu 6b3a9f8a40 Merge "Double check the sha1 for ranges during package generation" am: 92d73d3ab9 am: e62b278865
am: 65ac13cf08

Change-Id: I9aff2f5580681386cc63eadf8b567bd986d195e0
2018-01-29 20:03:43 +00:00
Tianjie Xu 65ac13cf08 Merge "Double check the sha1 for ranges during package generation" am: 92d73d3ab9
am: e62b278865

Change-Id: Icc1145a71430470ee9ef44043cd4b3532207bed9
2018-01-29 20:00:05 +00:00
Tianjie Xu df1166e92f Protect SparseImage._GetRangeData() with lock
The generator function is not thread safe and is prone to race
conditions. This CL uses a lock to protect this generator and loose the
locks elsewhere, e.g. 'WriteRangeDataToFd()'.

Bug: 71908713
Test: Generate an incremental package several times for angler 4208095 to 4442250.
Change-Id: I9e6f0a182a1ba7904a597f403f2b12fe05016513
2018-01-29 11:52:10 -08:00
Tianjie Xu 8a7ed9f771 Double check the sha1 for ranges during package generation
Check that the Sha1 for src&tgt ranges are correct before computing
patches. This adds ~6 seconds overhead for ~2400 commands.

Bug: 71908713
Test: Generate an incremental package from angler 4208095 to 4442250.
Change-Id: I8cf8ce132fb09a22f7d6689274ddb4a27770be76
2018-01-28 01:54:52 +00:00
Andreas Gampe cf528ca79d Warn.py: Update errorprone categories
From current tip-of-tree.

Bug: 72451126
Test: build/make/tools/warn.py errorprone.log
Change-Id: I1d632fb8801b74af695b4f7f464a99792e46637a
2018-01-25 12:25:20 -08:00
Jaekyun Seok d1b1d594a5 Merge "Support /product partition" am: 8fe7f9bfa2 am: 039e0991d5
am: f430ea5ebf

Change-Id: I91381c19408afc350ba7cecf2ca6daf0881f9abd
2018-01-25 04:16:57 +00:00
Jaekyun Seok f430ea5ebf Merge "Support /product partition" am: 8fe7f9bfa2
am: 039e0991d5

Change-Id: I325ae6518c55f1fc310233a5f67ce3df1ae4940f
2018-01-25 04:11:49 +00:00
Jaekyun Seok b7735d8105 Support /product partition
This CL adds the following build flags to support building product partition
from Android build system.
- BOARD_AVB_PRODUCT_ADD_HASHTREE_FOOTER_ARGS
- BOARD_AVB_PRODUCT_KEY_PATH
- BOARD_AVB_PRODUCT_ALGORITHM
- BOARD_AVB_PRODUCT_ROLLBACK_INDEX_LOCATION
- BOARD_PREBUILT_PRODUCTIMAGE
- BOARD_PRODUCTIMAGE_EXTFS_INODE_COUNT
- BOARD_PRODUCTIMAGE_FILE_SYSTEM_TYPE
- BOARD_PRODUCTIMAGE_PARTITION_SIZE
- BOARD_PRODUCTIMAGE_SQUASHFS_BLOCK_SIZE
- BOARD_PRODUCTIMAGE_SQUASHFS_COMPRESSOR
- BOARD_PRODUCTIMAGE_SQUASHFS_COMPRESSOR_OPT
- BOARD_PRODUCTIMAGE_SQUASHFS_DISABLE_4K_ALIGN
- BOARD_USES_PRODUCTIMAGE
- LOCAL_PRODUCT_MODULE
- PRODUCT_FOOTER_ARGS
- PRODUCT_PRODUCT_BASE_FS_PATH
- PRODUCT_PRODUCT_VERITY_PARTITION
- TARGET_COPY_OUT_PRODUCT
- TARGET_OUT_PRODUCT*

And PRODUCT_PRODUCT_PROPERTIES is added to save product-specific
properties in /product/build.prop.

Bug: 64195575
Test: succeeded building product.img with BOARD_PRODUCTIMAGE_PARTITION_SIZE,
BOARD_PRODUCTIMAGE_FILE_SYSTEM_TYPE and PRODUCT_PRODUCT_VERITY_PARTITION.

Change-Id: I774e6a38003734421591e51bed103802ff84f432
2018-01-25 09:33:58 +09:00
Jin Qian 95873268f3 Merge "Add BOARD_EXT4_SHARE_DUP_BLOCKS to share duplicated blocks on ext4" am: 082837365d am: b6ff980ba5
am: 931d76e1c4

Change-Id: I8bfcd310517c8814907e55b1a328c29392612872
2018-01-24 04:21:38 +00:00
Jin Qian 931d76e1c4 Merge "Add BOARD_EXT4_SHARE_DUP_BLOCKS to share duplicated blocks on ext4" am: 082837365d
am: b6ff980ba5

Change-Id: If48c84eae8f4a447e63c6b32e0c43758b8271242
2018-01-24 04:19:35 +00:00
Treehugger Robot 082837365d Merge "Add BOARD_EXT4_SHARE_DUP_BLOCKS to share duplicated blocks on ext4" 2018-01-24 04:10:42 +00:00
Tianjie Xu a988cae609 Merge "Don't check the stderr output from bsdiff/imgdiff" am: 07a4299182 am: 759dcbd1ed
am: 6b3337db4a

Change-Id: Ic845bffaa3e92b708188963ce7389e1b16b9045f
2018-01-23 19:00:55 +00:00
Tianjie Xu 6b3337db4a Merge "Don't check the stderr output from bsdiff/imgdiff" am: 07a4299182
am: 759dcbd1ed

Change-Id: If17dcdb8daaf74325bcbc9ae2ef0ba1932a30e94
2018-01-23 18:48:09 +00:00
Tianjie Xu 07a4299182 Merge "Don't check the stderr output from bsdiff/imgdiff" 2018-01-23 06:10:46 +00:00
Tianjie Xu a2a9f9990f Don't check the stderr output from bsdiff/imgdiff
bsdiff/imgdiff returns non-zero values correctly in case of failures.
So we don't need to check the stderr anymore. This avoids some false
reports of the diff program.

Bug: 72335938
Bug: 71505046
Test: Check the exit value of bsdiff/imgdiff in code search,
      generate a package for angler.

Change-Id: I18f0c3882a40a5288d6aee715713a05270e0db2b
2018-01-23 01:29:56 +00:00
Jin Qian fde9f7906b Add BOARD_EXT4_SHARE_DUP_BLOCKS to share duplicated blocks on ext4
This option applies to system.img and vendor.img.

Bug: 64109868
Change-Id: Ib4c7384049dca08e57441ef71cc81c2fba4462dd
2018-01-22 17:21:11 -08:00
Tao Bao 718707e403 Merge "releasetools: Track the change to BuildInfo." into oc-mr1-dev-plus-aosp
am: 412e36c6e3

Change-Id: I5cc36eb90a71a6df9c617f0fbcb53a7b8306f003
2018-01-22 22:23:28 +00:00
Tao Bao 2a12ed7dbd releasetools: Track the change to BuildInfo.
The global GetBuildProp() has been factored into BuildInfo class.

ota_from_target_files stderr: Traceback (most recent call last):
  File "/tmp/otatest1734786148718419730/releasetools/ota_from_target_files.py", line 1509, in <module>
    main(sys.argv[1:])
  File "/tmp/otatest1734786148718419730/releasetools/ota_from_target_files.py", line 1421, in main
    source_file=OPTIONS.incremental_source)
  File "/tmp/otatest1734786148718419730/releasetools/ota_from_target_files.py", line 1117, in WriteABOTAPackageWithBrilloScript
    max_timestamp = GetBuildProp("ro.build.date.utc", OPTIONS.source_info_dict)
NameError: global name 'GetBuildProp' is not defined

Test: Build an incremental A/B package with "--downgrade" flag.
Change-Id: I5c304c8b1ea79f3e5465b203bb990cbdffb9b328
2018-01-22 11:37:28 -08:00
Tao Bao fd65d676f0 Merge "releasetools: Add PayloadSigner class." am: 8413824f4b am: 9704dddd1f
am: 7e7e6a3005

Change-Id: I2761ccd77557d392500745d4d46766b2604b47db
2018-01-22 19:07:37 +00:00
Tao Bao 7e7e6a3005 Merge "releasetools: Add PayloadSigner class." am: 8413824f4b
am: 9704dddd1f

Change-Id: I61e2ca212ce17edd3c9d5b5d733101e9ecdaeb10
2018-01-22 19:05:08 +00:00
Treehugger Robot 8413824f4b Merge "releasetools: Add PayloadSigner class." 2018-01-22 18:54:57 +00:00
Zhi An Ng c1ad415862 Merge "Fix order of args passed to mksquashfsimage.sh" am: 8eae9fc5f5 am: 7db3875c4c
am: fffa70a1b7

Change-Id: Ie9b67865a5050438a91f37956b192290fe4a4ee4
2018-01-20 06:12:49 +00:00
Zhi An Ng fffa70a1b7 Merge "Fix order of args passed to mksquashfsimage.sh" am: 8eae9fc5f5
am: 7db3875c4c

Change-Id: I7c6ee9fad1665bfe72a3ae6bb50cb06c1a20c20a
2018-01-20 06:09:41 +00:00
Ng Zhi An 9446c1df98 Fix order of args passed to mksquashfsimage.sh
Bug: 72234456
Test: make systemimage
Change-Id: I1b4b07f0a488eade2f297fea12d02b6e8ccba7d8
2018-01-19 15:55:39 -08:00
Tao Bao 3bc775a4fc Merge "releasetools: Dump mkfs/fsck/simg2img outputs on error." am: 3221f0f387 am: c20b59eabe
am: ed184a50ea

Change-Id: If13865e513d12ad58e0874b6504ef777398df783
2018-01-19 22:48:17 +00:00
Tao Bao ed184a50ea Merge "releasetools: Dump mkfs/fsck/simg2img outputs on error." am: 3221f0f387
am: c20b59eabe

Change-Id: Iede2a76f87298fc0c389883b4e57d87d32968fb6
2018-01-19 22:45:34 +00:00
Treehugger Robot 3221f0f387 Merge "releasetools: Dump mkfs/fsck/simg2img outputs on error." 2018-01-19 22:35:47 +00:00
Tao Bao fabe08352c releasetools: Add PayloadSigner class.
Create a wrapper class that handles the payload signing, which unifies
the paths with and without external signer. Also add tests for the newly
added class.

The test keys under testdata/ are created with the script in
development/tools/make_key. testdata/testkey_with_passwd.pk8 uses
password "foo".

Test: python -m unittest test_ota_from_target_files
Test: Get identical A/B OTA packages w/ and w/o the CL.
Change-Id: Ic770aec726498a3babb88ec509985e7f1210fb18
2018-01-19 13:25:04 -08:00
Tao Bao de2fd74ebf Merge "releasetools: Clean up three functions in add_img_to_target_files.py." am: a5b48fcf7f am: 6ff60cee22
am: 96041d3435

Change-Id: I13d7b98f028652442889ca4d5003a4a1a3894a11
2018-01-19 20:47:22 +00:00
Tao Bao 4f47b05060 Merge "releasetools: Refactor AddImagesToTargetFiles()." am: 1eddcad909 am: 53c3f1589b
am: 8ede95e991

Change-Id: I6ae707a06e82b4e784c40ddb8902e6adf1fdca03
2018-01-19 20:46:15 +00:00
Tao Bao 96041d3435 Merge "releasetools: Clean up three functions in add_img_to_target_files.py." am: a5b48fcf7f
am: 6ff60cee22

Change-Id: I54d8ba7656c4fc1c4ca2b0454649ce66bc5ba185
2018-01-19 20:43:47 +00:00
Tao Bao 8ede95e991 Merge "releasetools: Refactor AddImagesToTargetFiles()." am: 1eddcad909
am: 53c3f1589b

Change-Id: Ibc2656d220cf0cdf8eb9947c928fd11736f30c86
2018-01-19 20:43:12 +00:00
Tao Bao cd53a89599 releasetools: Dump mkfs/fsck/simg2img outputs on error.
The output (stdout and stderr combined) would be helpful in understanding
the cause of the failure.

Not changing other occurrences in build_image.py yet, since it's less
obvious if a failed call should be considered fatal (thus whether to
dump such outputs). For example, GetVeritySize() simply returns 0 on
failed calls. They deserve further clean-ups in later CLs (with unit
tests).

Bug: 71864688
Test: Inject errors into mke2fs/e2fsck/simg2img. `m snod` prints outputs.
Test: `m snod` remains quiet on successful runs.
Change-Id: I172403f6cd05bce93767265dbcb110271a68e2a8
2018-01-19 11:15:42 -08:00
Tao Bao a2ff4c9b06 releasetools: Clean up three functions in add_img_to_target_files.py.
This CL makes the following changes:
 (a) cleans up the similar codes in AddCareMapTxtForAbOta() that handle
     'system' and 'vendor' partitions;
 (b) fixes an issue with the arcname in AddPackRadioImages() and
     AddRadioImagesForAbOta(), where forward slash should always be used
     in zip entry names;
 (c) refactors the branching statements in AddRadioImagesForAbOta() to
     reduce indentation levels.

Test: python -m unittest test_add_img_to_target_files
Test: `m dist` with aosp_marlin-userdebug. Check META/care_map.txt in the
      generated target_files.zip.
Change-Id: I3d6f794962d0c68390fbd18eb13c2622acab3ff5
2018-01-18 23:37:29 -08:00
Tao Bao bea20ac722 releasetools: Refactor AddImagesToTargetFiles().
Separate three functions out of AddImagesToTargetFiles(), into
AddCareMapTxtForAbOta(), AddRadioImagesForAbOta() and
AddPackRadioImages() respectively. This CL tries to apply minimal
changes for the refactoring purpose.

Also add tests for AddRadioImagesForAbOta() and AddPackRadioImages().
The tests for AddCareMapTxtForAbOta() require better testing support to
mock sparse_img.SparseImage, which will be added in later CLs.

Test: python -m unittest test_add_img_to_target_files
Test: `m dist` with aosp_marlin-userdebug. Check META/care_map.txt in
      the generated target_files.zip.
Change-Id: I1bb723c15237ff721f165cfce0ce996008ce9948
2018-01-18 23:37:23 -08:00
Tao Bao 9daf3419ba Merge "releasetools: Remove the reloading of target info dict." am: 13f228ebde am: 815a167ae4
am: ee3155c09c

Change-Id: I97f63a561634bb7ded0e0968e5751eaaaf60e5fa
2018-01-18 23:03:38 +00:00