Commit Graph

2109 Commits

Author SHA1 Message Date
Tianjie Xu f67dd8059d Generate block based OTA for product partitions
Currently only the system and vendor partitions are updated in the
generic block based OTA generation script. Since the product partition
is quite large and consist most of APK files, it would be beneficial to
update it similar to system. Handle the odm and system_ext partitions in
the same way as well.

Bug: 132683080
Test: Run unit tests, generate full and incremental OTA with product partitions.
Change-Id: I13478cf9bd32137c6729b8c9cb102080147093f2
2019-08-20 00:18:00 -07:00
Paul Trautrim 6c3aa71d1b Merge "Use --add_missing in merge_target_files.py"
am: cc875c1482

Change-Id: I0abe618a8e1900c57b67060361b46807531cddd7
2019-08-19 04:05:40 -07:00
Paul Trautrim cc875c1482 Merge "Use --add_missing in merge_target_files.py" 2019-08-19 10:50:35 +00:00
Tao Bao f4157c5859 Merge "releasetools: Add more module building rules."
am: 39369d08fb

Change-Id: Ic422023eceae40922a6251f32982cc1141f86aac
2019-08-14 09:08:50 -07:00
Tao Bao 39369d08fb Merge "releasetools: Add more module building rules." 2019-08-14 15:33:33 +00:00
Paul Trautrim bc3600c030 Use --add_missing in merge_target_files.py
This is needed when including prebuilt images that do not need to be
regenerated. Otherwise the script will exit upon detecting an existing
IMAGES/ directory.

Test: test_merge_target_files
Bug: 139111475
Change-Id: I55d82bd7dc4fa5f27bc01e481362e79fecc200ff
2019-08-13 19:24:12 +09:00
Tao Bao cd79edb331 Merge "releasetools: Fix the target files diffing."
am: 6a52b34473

Change-Id: I852a255f7ad99718a96bae0df92ce54ceb543c0a
2019-08-08 16:35:04 -07:00
Tao Bao 3c7b2a658c releasetools: Add more module building rules.
This CL adds the building rules for the following tools:
- add_img_to_target_files
- check_ota_package_signature
- check_target_files_signatures
- img_from_target_files
- merge_target_files
- ota_package_parser
- sign_apex
- sign_target_files_apks
- validate_target_files

Bug: 63866463
Test: mmma -j build/make/tools/releasetools
Test: Run each tool above to ensure there's no missing dependency (in
      the exercised code path).
Change-Id: Ide5b9775939ed02c822109cbcf4224000d5472df
2019-08-07 22:35:42 -07:00
Tao Bao ced94b337c Merge "releasetools: ota_from_target_files uses target_files_diff."
am: 69340f6885

Change-Id: I822a3bfa774c79d2c4789a84fa084361753cb24b
2019-08-07 21:41:21 -07:00
Tao Bao f0c4aa204c releasetools: Fix the target files diffing.
This CL refactors the non-A/B OTA generation code into a function,
so it allows adding common post-generation code for both flows.

It moves the code for target files diffing to common path, and fixes the
diffing functionality (broken due to the use of unzip pattern).

Test: Run ota_from_target_files with `--log_diff` flag to generate
      non-A/B and A/B OTAs respectively.
Change-Id: I92e5549a58344fc56a151a709c06534913d265d5
2019-08-07 14:53:08 -07:00
Tao Bao f882684d85 Merge "Move make_recovery_path and sparse_img to modules."
am: 4c9527b314

Change-Id: I13a67cfe58450696c37a0e7e4ecc3261d8bd7d5a
2019-08-07 13:15:15 -07:00
Tao Bao 496748378d releasetools: ota_from_target_files uses target_files_diff.
The dependency was overlooked as the module was conditionally imported.

Bug: 63866463
Test: `m -j ota_from_target_files`; Run ota_from_target_files with
      `--log_diff` flag (on a non-A/B target).
Change-Id: I71eff2e55cc28fd649f0ca1b38da53907dac6597
2019-08-07 13:01:27 -07:00
Tao Bao 4c9527b314 Merge "Move make_recovery_path and sparse_img to modules." 2019-08-07 18:49:20 +00:00
Tao Bao e11a46000c Move make_recovery_path and sparse_img to modules.
These two tools are only used within build system, so not adding them
to otatools.zip.

Bug: 63866463
Test: TreeHugger
Test: Build a non-A/B target that calls make_recovery_patch.
Change-Id: Iaeddc98a2c4a27c6b49d91047500029e460db9e3
2019-08-06 23:27:48 -07:00
Tao Bao 7552bbdfb6 Merge "releasetools: merge_target_files sets verbose in main."
am: 3738f37e22

Change-Id: I0e193d444af6e8a05289e22bbc02fc084972750a
2019-08-06 19:47:51 -07:00
Tao Bao abb806b3f0 releasetools: merge_target_files sets verbose in main.
The flag would otherwise be turned on unconditionally even when
importing merge_target_files as a module. As a result, `atest
releasetools_test` was broken / flaky prior to this change, as
PythonUnitTestResultParser doesn't like unexpected outputs in stderr.

Test: `atest releasetools_test releasetools_py3_test`
Change-Id: Ifd614c797d2cee3b33e4c2a7ad8e598850e9d60c
2019-08-06 14:13:19 -07:00
Tao Bao e5f0839c4b Merge "releasetools: Build build_super_image as a lib."
am: 7edc2b1994

Change-Id: Ib9fb0144b79d183aacc08ed0f661a0c7356b5372
2019-08-06 12:18:31 -07:00
Tao Bao 7edc2b1994 Merge "releasetools: Build build_super_image as a lib." 2019-08-06 19:01:20 +00:00
Tao Bao a4a236e318 Merge "releasetools: Drop the support for creating image archive from dir."
am: 28dd0c83a2

Change-Id: I5a7cb6fc33c2ebdc8eb6d44302f038b5c88540a4
2019-08-06 11:57:35 -07:00
Tao Bao 28dd0c83a2 Merge "releasetools: Drop the support for creating image archive from dir." 2019-08-06 18:45:17 +00:00
Tao Bao 3810585592 Merge "releasetools: Don't write outputs if not in verbose mode."
am: a514c4ec22

Change-Id: I3cbc864609975cd38c90d68ef2c24caeede8ef4e
2019-08-06 11:08:15 -07:00
Tao Bao 93d52312d3 Merge "Build and use ota_from_target_files as a module."
am: b7957f112b

Change-Id: I2c2d6d17f0b52c83a591ebe28b31076741b3e967
2019-08-06 11:00:19 -07:00
Tao Bao a514c4ec22 Merge "releasetools: Don't write outputs if not in verbose mode." 2019-08-06 17:33:22 +00:00
Tao Bao 005305adf8 releasetools: Build build_super_image as a lib.
The lib will be shared by more users (e.g. add_img_to_target_files) in
follow-up changes.

Test: TreeHugger
Test: atest releasetools_test releasetools_py3_test
Change-Id: Iaa48bfc8c1dada41362d974773c124a001471ef8
2019-08-05 13:02:26 -07:00
Tao Bao 8821d64481 Build and use ota_from_target_files as a module.
Bug: 63866463
Test: TreeHugger
Test: m -j otapackage
Test: `m -j otatools-package`. Check bin/ota_from_target_files is
      available. Build an OTA package with standalone otatools.
Change-Id: I533dee1c6997d233601e7f4eae637ff5743ed637
2019-08-05 12:57:41 -07:00
Daniel Norman 06975fa319 Merge "Adds support for optionally generating vbmeta.img in merge_builds."
am: 477d2894e4

Change-Id: I1af7d7ff9f4d55373868702d992e3db3ea1bfcb4
2019-08-02 16:46:35 -07:00
Tao Bao 2aac9c9d6d releasetools: Drop the support for creating image archive from dir.
Taking an extracted target_files dir input is only needed for
add_img_to_target_files.py, in particular for the call from build
system. For the case of img_from_target_files.py, it goes much faster
with zip2zip (3m vs 7s in my local test). This CL drops the unneeded
code path in preparation for the switch to zip2zip.

Test: `python build/make/tools/releasetools/img_from_target_files.py \
         -v aosp_taimen-target_files.zip aosp_taimen-img-test.zip`
Change-Id: I7f85d7d629d6fd1621a082f94a11cac7c3e6bbf2
2019-08-02 16:02:32 -07:00
Tao Bao 718faed3dc releasetools: Don't write outputs if not in verbose mode.
The change in [1] allows brillo_update_payload to write to stdout/stderr
directly without buffering, to ensure dumping progress to screen
continously. However, it now unconditionally writes to stdout/stderr
even if caller doesn't specify `-v`. The behavior upsets
PythonUnitTestResultParser in [2], which doesn't want any unexpected
output from stderr for result parsing.

This CL updates the logic to only give continous outputs in verbose
mode.

[1] commit 2f7e11ef27
[2] https://android.googlesource.com/platform/tools/tradefederation/+/refs/heads/master/test_framework/com/android/tradefed/testtype/PythonUnitTestResultParser.java

Test: Run ota_from_target_files.py with `-v`. Check there's continous
      output from brillo_update_payload.
Test: `python -m unittest -v test_ota_from_target_files > /dev/null`
      doesn't show outputs from brillo_update_payload.
Change-Id: I3b851203ef011a5b4d982948cf52793f6fb6ad2d
2019-08-02 13:36:49 -07:00
Daniel Norman 276f06275b Adds support for optionally generating vbmeta.img in merge_builds.
Bug: 137853921
Bug: 138671115
Test: python -m unittest test_common
Test: python -m unittest test_add_img_to_target_files
Test: Ran 'merge_builds --build_vbmeta' for two devices, one with the
vbmeta struct on system.img and another with vbmeta_system.img. Flashed
the regenerated vbmeta.img files on devices, devices boot.

Change-Id: I8d7585c7af468be3d242d8aceeed6d27e6fc6d96
2019-08-02 20:13:03 +00:00
Tao Bao 74dcbad8f2 Merge "releasetools: Add FORCE_RUN_RELEASETOOLS."
am: 1bc828c97e

Change-Id: I08cd4632611b8053c0457846b0a170488640fb80
2019-08-01 21:33:22 -07:00
Tao Bao 7d223c6ddd releasetools: Add FORCE_RUN_RELEASETOOLS.
This allows test runner to take care of the external tools setup, then
to start the test without a lunch'd environment. This is needed before
having supports like python_test_helper that packs dependents into
releasetools_test module.

Bug: 138791766
Test: Don't lunch any target. Use `FORCE_RUN_RELEASETOOLS=1 python -m
      unittest -v test_add_img_to_target_files` and check there's no
      skipped testcases.
Change-Id: Iafeaba54fc228d8e1fc5ff3c91f7ffebef18ad7f
2019-08-01 12:22:54 -07:00
Daniel Norman 3f9554b59a Merge "Adds new merge builds script for use in merging two non-dist builds."
am: a85f0ee8a2

Change-Id: Ib7b66d5ee82df1d9f31d521917490ac1bac4a4e2
2019-07-29 17:36:57 -07:00
Daniel Norman bfc51efa97 Adds new merge builds script for use in merging two non-dist builds.
Bug: 137853921
Test: python -m unittest test_common
Test: python -m unittest test_merge_target_files
Test: Built two partial builds without dist. Ran out/host/linux-x86/bin/merge_builds.
Flashed using `fastboot flashall`. Device boots.
Change-Id: Iffd0a447cdf19a7775a813b4b896178aa6f861f3
2019-07-29 16:30:56 +00:00
Tao Bao ac280723c2 Merge "releasetools: Move recovery-two-step.img to OTA/."
am: 98efd8290e

Change-Id: I21419dc4a13271c589b8bcb4a4be7da02ef1587d
2019-07-28 11:07:07 -07:00
Tao Bao 0480850f0b releasetools: Move recovery-two-step.img to OTA/.
It used to be packed at IMAGES/recovery-two-step.img, but to serve OTA
purpose only.

Test: `m dist` with a non-A/B target. Check the file in the generated
      target_files.zip.
Test: Create two-step package. Check that recovery-two-step.img is used.
Change-Id: Iec6a73c682e0f844cd8c0b758c9470fa35dd15d8
2019-07-26 13:59:11 -07:00
Tao Bao 706eb3c566 Merge "releasetools: Fix the use of StringIO."
am: 6b466c8f56

Change-Id: I5669a070c0d98239263557c005305002739d1c1d
2019-07-25 09:55:09 -07:00
Tao Bao bb73388acf releasetools: Fix the use of StringIO.
Based on the actual semantics, it actually wants an in-memory _bytes_
buffer (io.BytesIO), especially when running with Python 3. This CL
fixes the issue and adds a unittest.

Bug: 131631303
Test: python -m unittest test_sign_target_files_apks
Test: python3 -m unittest test_sign_target_files_apks
Change-Id: I3fb067acc26713f1842e831225607779fd0d1b7e
2019-07-24 23:34:25 -07:00
changho.shin 0f12536e6b Switch to aapt2 for getting minSdkVersion
aapt does not support parsing some referencing attributes.
Switch to latest tool, aapt2, which works better.

Test: aapt2 dump badging test.apk and returned 0.
Test: Run check_target_files_signatures.py with a target_files.zip.
Test: Run sign_target_files_apks.py with a target_files.zip.
Change-Id: Ib3a4740506f29ebab6930767f2aa8a0b5c4ba053
2019-07-19 20:26:42 +00:00
Daniel Norman 36f8163589 Merge "Clean up merge_target_files.py."
am: ff165e31fd

Change-Id: I4bb6e084b0beb4a57df38a9e083b7e4900320a63
2019-07-19 07:41:27 -07:00
Tao Bao e8aa7dcf19 Merge "Build build_image and build_super_image as modules."
am: 25cf50f582

Change-Id: Idb34ce99b01fe432e0fe2f7824a87cc8fa3541bf
2019-07-19 07:40:39 -07:00
Daniel Norman ff165e31fd Merge "Clean up merge_target_files.py." 2019-07-19 03:15:29 +00:00
Daniel Norman 4cc9df660b Clean up merge_target_files.py.
- Removes functions that can be replaced with one-line external methods
- Moves read_config_list to common alongside similar methods LoadDictionaryFrom*
- Runs pyformat on merge_target_files.py

Bug: 137853921
Test: python -m unittest test_merge_target_files
Test: Using merge_target_files.py to create a merged build, & booting.
Change-Id: I833c1086db41e1374057cc7447fc50d1915734a7
2019-07-18 13:04:54 -07:00
Tao Bao 2bbb07c53c Build build_image and build_super_image as modules.
Bug: 63866463
Test: TreeHugger
Test: `atest releasetools_test`
Test: `atest releasetools_py3_test`
Change-Id: I2059a4ced709d1b2ee331a9aaaa5ca30db4ebf6b
2019-07-18 10:27:38 -07:00
Tao Bao b9c7ed6314 Merge "logging: set stdout and stderr to None in some cases"
am: e733ed7e6b

Change-Id: Ifa43d7a811b2c288de1a2ee3a10fee0efaa43661
2019-07-17 08:55:25 -07:00
Tao Bao e733ed7e6b Merge "logging: set stdout and stderr to None in some cases" 2019-07-17 15:40:59 +00:00
Tianjie Xu bcc3554b74 Merge "Factor out the image classes to break circular dependency"
am: dcb6644719

Change-Id: I16371818e6e022ee75187b9c9e104836a72dd983
2019-07-16 13:30:24 -07:00
Tianjie Xu 41976c725c Factor out the image classes to break circular dependency
This helps to break the circular dependency between common and
blockimgdiff.

Bug: 32379627
Test: unit tests pass
Change-Id: I90b5ff34782acbfac86f36265bd96c207d898bf6
2019-07-15 17:02:23 -07:00
Tao Bao 0f6c1d2b47 Merge "releasetools: Ignore nonexistent APEX overrides."
am: d077c5b88e

Change-Id: Idc8e680e3140599f81c5f9cdc33e2a647a95a300
2019-07-12 08:00:29 -07:00
Tao Bao 3422309d6d releasetools: Ignore nonexistent APEX overrides.
This allows sharing the same signing config on different target_files
zips. Nonexistent APEX will be ignored with a warning.

Bug: 137249701
Test: Run sign_target_files_apks with APEX overrides.
Change-Id: I2bad0f5c00753ed36ec5ae3431c7dc2ff1fc3e9c
Merged-In: I2bad0f5c00753ed36ec5ae3431c7dc2ff1fc3e9c
(cherry picked from commit b369c7226a)
2019-07-12 00:25:41 -07:00
Tao Bao 213210c4d0 Merge "releasetools: Replace iteritems with items."
am: 6623fdda5a

Change-Id: I35c9e3b39897d20463aa12252a6118b427e64dfc
2019-07-11 11:39:41 -07:00
Regnier, Philippe 2f7e11ef27 logging: set stdout and stderr to None in some cases
For very long processes, we might want to keep stdout and stderr
by default to None.
So no redirection will occur in the child process as explained in:
https://docs.python.org/2/library/subprocess.html
That will result in the child process stdin and stderr to be same
than in common.py and avoid to have the logs blocked during the
child process execution and flushed only when child process terminates.
Since the logs are continously displayed, it allows to easily confirm
that the process is not blocked.

Bug: 133380588
Test: generate iota & Check that the logs are not blocked.

Change-Id: I6d6cb56547bf3a4a4334dfa22b6b2b05d2c36a5e
Signed-off-by: Regnier, Philippe <philippe.regnier@intel.com>
2019-07-11 17:05:17 +08:00
Tao Bao 3888428f29 releasetools: Replace iteritems with items.
While in theory this could incur memory overhead for Python 2, the
impact is low for the existing use cases (plus we're moving away from
Python 2).

Bug: 131631303
Test: No additional occurrence of iteritems.
Test: Build with Python 3.
Change-Id: I0205c9edf25f46e3d85967c7dd2c1af035757741
2019-07-10 22:29:20 -07:00
Tao Bao d2ff3b5dca Merge "Additionally run releasetools_test with Python 3."
am: 576ae9628b

Change-Id: I090d9f67524c636967400d57164f26158562d3a9
2019-07-10 13:41:57 -07:00
Tao Bao 576ae9628b Merge "Additionally run releasetools_test with Python 3." 2019-07-10 18:22:37 +00:00
Justin Yun 8f11ad5d40 Merge "Rename product_services to system_ext"
am: b7a50ece30

Change-Id: I7a56d76d3543b2bc4b1d0b24e54e8dd82ab7f15a
2019-07-09 02:14:55 -07:00
Justin Yun 6151e3f1ea Rename product_services to system_ext
Bug: 134359158
Test: build and check if system_ext.img is created
Change-Id: I67f2e95dd29eac6a28e07e24ea973d3a134c3bfc
2019-07-09 08:57:19 +00:00
Tao Bao ca82fc90f7 Additionally run releasetools_test with Python 3.
Bug: 131631303
Test: TreeHugger
Test: `atest releasetools_test`
Test: `atest releasetools_py3_test`
Change-Id: Ie00f48835aae41bc69ab8554b856a62aa3b497cc
2019-07-09 05:14:31 +00:00
Tao Bao ec2f3da2a2 Merge "releasetools: Make merge_target_files and its test Python 3 compatible."
am: 6e3ae4e849

Change-Id: I0ee3f8801e194f140f32a4123100aa0e4f449a70
2019-07-08 21:19:53 -07:00
Tao Bao 2ad4b82d45 releasetools: Make merge_target_files and its test Python 3 compatible.
And a few minor clean-ups to the styling.

Bug: 131631303
Test: python -m unittest test_merge_target_files
Test: python3 -m unittest test_merge_target_files
Test: Use `python merge_target_files` to merge two target_files zips.
Test: Use `python3 merge_target_files` to merge two target_files zips.
Change-Id: I8502dfb243408f658d022e8d5e5fbb60066e4ff0
2019-07-08 16:06:15 -07:00
Kiyoung Kim dc6566f3b2 Merge "Split merge_target_files function"
am: 09736b4aaf

Change-Id: Ic077a4bf1276ead627c1d09f67636626623a45ab
2019-07-07 23:51:33 -07:00
Kiyoung Kim 7cbeda728d Split merge_target_files function
Split merge_target_files function into several steps, so we can increate
readability and add conditional flows for other *SSI mixed build with less effort

Test: m -j & atest passed
Change-Id: I558f9dd5bca31b132a09cb36d9dfcd30c92efbc9
2019-07-08 12:13:05 +09:00
TreeHugger Robot 218adad125 Merge "DO NOT MERGE - Merge qt-dev-plus-aosp-without-vendor (5699924) into stage-aosp-master" into stage-aosp-master 2019-07-02 07:40:58 +00:00
Tao Bao 48a2feb670 Change the condition for building super_empty.img.
This CL changes the condition for building super_empty.img from
PRODUCT_BUILD_SUPER_PARTITION to PRODUCT_USE_DYNAMIC_PARTITIONS, as a
follow-up to the change in [1].

With the CL in [1], it skips building super.img and super_empty.img both
when turning off PRODUCT_BUILD_SUPER_PARTITION. However, the latter
should be mandatory whenever dynamic partitions is enabled. Because
fastboot relies on this file to properly flash dynamic partitions. Plus,
the cost for building super_empty.img is much lower than the one for
super.img.

As part of the change, it'll write group info into target_files when
building with PRODUCT_BUILD_SUPER_PARTITION == false. It's the work for
target_files merging script to determine the values to be picked up. The
current logic in merge_target_files.py always uses the one from vendor
target_files. This CL adds a testcase to ensure the behavior.

[1] https://android-review.googlesource.com/c/platform/build/+/928756

Bug: 135752763
Test: `m dist` with a target that sets PRODUCT_BUILD_SUPER_PARTITION to
      false. Check the built artifacts contain super_empty.img. Verify
      that the build can be flashed properly.
Change-Id: I277f087eab45663a6c3b33333d16e9e576c1c25c
2019-06-28 14:23:53 -07:00
Tao Bao c1c52e7b3b Merge "releasetools: Prefer the avbtool specified in target_files." am: 63cf1326da
am: d84a0531e9

Change-Id: Ieb93952f093a2d6e3d3c9796d61819c46cc4fcea
2019-06-27 19:04:51 -07:00
Tao Bao 63cf1326da Merge "releasetools: Prefer the avbtool specified in target_files." 2019-06-28 01:18:21 +00:00
Tao Bao a1b63421d9 Merge "releasetools: Make additional modules Python 3 compatible." am: e4f07795e4
am: ed7923be4f

Change-Id: Idf708c33684e7c7b96286b363d15bf947422323d
2019-06-27 17:58:45 -07:00
Tao Bao e4f07795e4 Merge "releasetools: Make additional modules Python 3 compatible." 2019-06-27 23:52:50 +00:00
Tao Bao 2cfd9b9077 Merge "releasetools: Clean up check_target_files_signatures.py." am: f6e8d46e2f
am: d82b810a77

Change-Id: Ie4cb3dff2fb4098714a25b4cf10643cd1951aae4
2019-06-27 16:08:24 -07:00
Tao Bao f6e8d46e2f Merge "releasetools: Clean up check_target_files_signatures.py." 2019-06-27 21:36:14 +00:00
Tao Bao e815cec8f3 Merge "releasetools: Add sanity test for sign_apex." am: 2a917796b5
am: 03dc3694be

Change-Id: I0d2ad60b1df2f660a2a0658eaef000a19538b762
2019-06-27 11:31:55 -07:00
Tao Bao 2a917796b5 Merge "releasetools: Add sanity test for sign_apex." 2019-06-27 17:35:29 +00:00
Tao Bao a67e12d66f releasetools: Clean up check_target_files_signatures.py.
And make it Python 3 compatibile.

Bug: 131631303
Test: python check_target_files_signatures.py target-files.zip
Test: python3 check_target_files_signatures.py target-files.zip
Change-Id: I38e868c3124387f5543ad68c92c75917e40b7d8f
2019-06-27 10:24:10 -07:00
Tao Bao a370545a2c releasetools: Make additional modules Python 3 compatible.
Bug: 131631303
Test: `python -m unittest test_sign_target_files_apks`
Test: `python3 -m unittest test_sign_target_files_apks`
Test: `python -m unittest test_add_img_to_target_files`
Test: `python3 -m unittest test_add_img_to_target_files`
Test: `python -m unittest test_ota_from_target_files`
Test: `python3 -m unittest test_ota_from_target_files`
Test: `python -m unittest test_validate_target_files`
Test: `python3 -m unittest test_validate_target_files`
Test: Run `python3 ota_from_target_files.py` to generate an OTA.
Test: Run `python3 sign_target_files_apks.py` to sign a target_files.
Change-Id: I56b45bbcbf7aa83e690785a9640c0212e45d12d8
2019-06-27 09:05:48 -07:00
Kiyoung Kim c2544994a4 Merge "Create common.LoadDictionaryFromFile" am: fc195fee44
am: 486bc8d590

Change-Id: I9622dbde337937345b581fd11047c339c242aab7
2019-06-26 18:25:37 -07:00
Kiyoung Kim fc195fee44 Merge "Create common.LoadDictionaryFromFile" 2019-06-27 00:27:49 +00:00
Tao Bao 1ac886e181 releasetools: Prefer the avbtool specified in target_files.
This allows a consistent logic in using the avbtool which could be
board-specific.

Test: `atest releasetools_test`
Test: Run sign_target_files_apks.py on a target_files.zip.
Change-Id: I8cd93b8e71146985734f85c31f4662f5e2e9534c
2019-06-26 17:18:48 -07:00
Tao Bao c9c1b864ba releasetools: Add sanity test for sign_apex.
This ensures a matching interface between sign_apex and apex_utils.

The test apex `testdata/foo.apex` is generated by running
`system/apex/apexer/runtests.sh`.

Test: python -m unittest test_sign_apex
Test: atest releasetools_test
Change-Id: I7c14b1df2a3038ad206aa3e5aac084c47baaa00b
2019-06-26 17:17:09 -07:00
Tao Bao c515aa03b8 Merge "releasetools: Clean up additional modules for Python 3 compatibility." am: 90f638d842
am: c82d8ceffc

Change-Id: I84e7866b06ff693240272b39119d0ab5cae7e72a
2019-06-26 12:05:13 -07:00
Tao Bao 59cf0c59de releasetools: Clean up additional modules for Python 3 compatibility.
It also addresses a few linter warnings (mostly regarding indentation).

Bug: 131631303
Test: TreeHugger
Test: `python -m unittest test_ota_from_target_files`
Test: `python3 -m unittest test_ota_from_target_files`
Test: `python build/make/tools/releasetools/check_ota_package_signature.py
           build/make/target/product/security/testkey.x509.pem
           test-keys-signed-ota.zip`
Test: `python3 build/make/tools/releasetools/check_ota_package_signature.py
           build/make/target/product/security/testkey.x509.pem
           test-keys-signed-ota.zip`
Change-Id: I11a745dfec052081ab0282bb028dc27debf71f57
2019-06-26 00:15:53 -07:00
Kiyoung Kim ebe7c9c92f Create common.LoadDictionaryFromFile
There are some places defining same file open function and use
common.LoadDictionaryFromLines. This commit creates
LoadDictionaryFromFile to reduce some code redundancy.

Test: m -j & atest passed
Change-Id: I6a3fa48693095937f8c79ce6f3c110b6862a1967
2019-06-26 13:09:52 +09:00
Tao Bao a54ec9bc55 Merge "releasetools: Make apex_utils Python 3 compatible." am: 58abe77060
am: da5b3b5178

Change-Id: I1c44ac55ea7709410f4fa3a91cf0e41d2df018a3
2019-06-25 16:07:39 -07:00
Tao Bao 86b529a23e releasetools: Make apex_utils Python 3 compatible.
In order to get a Python 2 and 3 compatible re-raise behavior, this CL
removes the stack traceback for the lines within apex_util module (i.e.
sys.exc_info()[2]). It's not a big loss in practice, since we only have
one line within the try-except block (`common.RunAndCheckOutput()`)
that's no longer reported in the traceback.

Using `six` module could better solve this, but only after building
releasetools as python_binary_host modules where we can properly handle
the module dependency.

Bug: 131631303
Test: TreeHugger
Test: `python -m unittest test_apex_utils`
Test: `python3 -m unittest test_apex_utils`
Change-Id: I0c5a72ec9fad5ff9d8c9c94d29e813e433ec2921
2019-06-25 11:10:55 -07:00
Tao Bao 41e98f654c Merge changes Idc7776be,I409fe30a am: ec8e2b4e02
am: 70b5b96565

Change-Id: I33df35895fc75347dfda3cacdf717f75df7ba14d
2019-06-25 10:09:32 -07:00
Tao Bao 9e893c3cfd releasetools: Make verity_utils Python 3 compatible.
Bug: 131631303
Test: TreeHugger
Test: `python -m unittest test_verity_utils`
Test: `python3 -m unittest test_verity_utils`
Change-Id: Idc7776be2cf78fbdb0b3ba87231d05013da8d3ac
2019-06-24 21:07:19 -07:00
Tao Bao da30cfae96 releasetools: Make common Python 3 compatible.
Bug: 131631303
Test: TreeHugger
Test: `python -m unittest test_common`
Test: `python3 -m unittest test_common`
Change-Id: I409fe30a5d71975c1d7b66e5e749843de530f1f9
2019-06-24 21:02:12 -07:00
Tao Bao cebd118790 Merge "releasetools: Remove some legacy paths for prop rewrites." am: eb7cb468e4
am: f589f9bb2d

Change-Id: I637506dfc84a4a4aee8f8ee744b4e3ff6d5d0642
2019-06-24 09:24:59 -07:00
Tao Bao 338c1b7e90 releasetools: Remove some legacy paths for prop rewrites.
sign_target_files_apks.py only needs to take care of the current
release. The legacy paths of ODM/build.prop, VENDOR/odm/build.prop, and
BOOT/RAMDISK/default.prop no longer exist in the target_files.zip from
master.

The other two cases of ROOT/default.prop and
RECOVERY/RAMDISK/default.prop are still kept in the code, as they will
still exist (as symlink or conditionally).

Test: Run sign_target_files_apks.py against
      aosp_taimen-target_files.zip. Check the rewritten prop files.
Test: `python -m unittest test_sign_target_files_apks`
Change-Id: I5e70bc2ccc0f3dcf0eace0718c59a3b0f89a9ff4
2019-06-21 10:26:15 -07:00
Tao Bao add1f28ba4 Merge "releasetools: Make blockimgdiff Python 3 compatible." am: 4daa755fb2
am: 07a7471ddb

Change-Id: Ia91938e7efed299dc212dfa2c9116634e67dafd1
2019-06-21 08:25:58 -07:00
Tao Bao 8c9cc5fbc1 Merge "releasetools: Make rangelib Python 3 compatibile." am: 1f1aafa30d
am: 207f624f75

Change-Id: Iba18162ab1c05ec780b057f1391b1f9aaf0b4039
2019-06-20 18:17:23 -07:00
Tao Bao b813120af1 releasetools: Make blockimgdiff Python 3 compatible.
Bug: 131631303
Test: `python -m unittest test_blockimgdiff`
Test: `python3 -m unittest test_blockimgdiff`
Change-Id: I8cf072ca4af6a525bc675cba0be998ff7be7f4f1
2019-06-20 14:30:30 -07:00
Tao Bao 1f1aafa30d Merge "releasetools: Make rangelib Python 3 compatibile." 2019-06-20 14:51:21 +00:00
Tao Bao 0f68744f1e Merge "releasetools: Make common.ZipWriteStr Python 3 compatible." am: 9521c802ec
am: 0812ce4e3d

Change-Id: Ia3009e10b5b9492b6ab50135dce4a77ffc978011
2019-06-19 15:56:45 -07:00
Tao Bao d660c8d8f3 releasetools: Make rangelib Python 3 compatibile.
Bug: 131631303
Test: `python -m unittest test_rangelib`
Test: `python3 -m unittest test_rangelib`
Change-Id: Ia8e26623d5967f2eea003252ee56b861350d626b
2019-06-19 10:30:34 -07:00
Tao Bao c1a1ec30df releasetools: Make common.ZipWriteStr Python 3 compatible.
Python 2 and 3 behave differently when calling ZipFile.writestr() with
zinfo.external_attr being 0. Python 3 uses `0o600 << 16` as the value
for such a case (since
18ee29d0b8),
which seems to make more sense. Otherwise the entry will end up with
0o000 as the permission bits. This CL updates common.ZipWriteStr to
follow the logic in Python 3, in order to get consistent behavior
between using the two versions.

Bug: 131631303
Test: `python -m unittest test_common.CommonZipTest`
Test: `python3 -m unittest test_common.CommonZipTest`
Change-Id: If8429855d922ef1ad76591f703215a0ce5089f0f
2019-06-18 23:53:53 -07:00
Tao Bao ae64f09b06 Merge "releasetools: Update partitions in deterministic order." am: aadc9b9159
am: 269c12a5b9

Change-Id: Ic2bff4b76ccd4a0c5b3d3d71dedee0728fc16019
2019-06-18 16:23:38 -07:00
Tao Bao f1113e97ae releasetools: Update partitions in deterministic order.
Previously it was using regular dict.

Test: python -m unittest test_common.DynamicPartitionsDifferenceTest
Change-Id: If108a4512aeaf9d3c8775c030cad6e44342b9d3d
2019-06-18 12:14:32 -07:00
Daniel Norman c7c103514f Merge "Allows skipping building the super images for partial builds." am: 87ebb3013f
am: d3576727ea

Change-Id: I872f713f9f8ff30b3dd72643b07443894b5af451
2019-06-14 15:12:25 -07:00
Treehugger Robot 87ebb3013f Merge "Allows skipping building the super images for partial builds." 2019-06-14 21:42:33 +00:00
Daniel Norman 0bf940cace Allows skipping building the super images for partial builds.
Previously, setting PRODUCT_BUILD_SUPER_PARTITION to false for a partial
build (with PRODUCT_USE_DYNAMIC_PARTITIONS == true) would fail to
include necessary keys in misc_info.txt that are required when merging
two partial builds to create a dynamic-partition-enabled mixed build.

This change ensures these necessary keys are included even when
PRODUCT_BUILD_SUPER_PARTITION is false. Setting
PRODUCT_BUILD_SUPER_PARTITION to false causes partial builds to skip
building super.img and super_empty.img, instead relying on these images
to come from the final merged build.

Bug: 134764140
Test: Building & booting a dynamic-partition-enabled mixed build, and
inspecting partial builds' logs / out folder to ensure that
super.img/super_empty.img were not created.

Change-Id: I99431a9a342e9b0617510e250597f3024ef39322
2019-06-14 11:42:45 -07:00
Tao Bao d05c5b091c Merge "releasetools: No-op change to use common.AVB_VBMETA_PARTITIONS." am: 4995d2fe16
am: 601fc23eee

Change-Id: Iee9f77ce710d988750417e6f2729d8508129aea7
2019-06-11 08:20:58 -07:00
Tao Bao cd951b2bdb releasetools: No-op change to use common.AVB_VBMETA_PARTITIONS.
common.AVB_VBMETA_PARTITIONS was recently added (commit
08c190fc89) for the same purpose.

Test: TreeHugger
Change-Id: I65572d54c22a753fdef80677377fcc9b684ee16f
2019-06-10 15:29:19 -07:00
Chris Gross 3246b1db5b Merge "Add a logging statement after soong ZIP runs in the merge_target_files.py step." am: 5f4f4256eb
am: 397b3e5a68

Change-Id: I466dcf3cdd8896b795fdb5161bd720fca4a649ff
2019-06-07 14:00:45 -07:00
Chris Gross eab4f0e2e9 Add a logging statement after soong ZIP runs in the
merge_target_files.py step.

Bug: 134681035
Test: Built a merged build and ensured that the new timestamp was
visible in the log.

Change-Id: Ia6bbda48c7f57afdb6482253eaf0b3b0ea067468
2019-06-07 13:15:52 -07:00
Daniel Norman fd04ff8973 Merge "Renames system to framework and other to vendor." am: 6728fe850e
am: 1603c9311d

Change-Id: I280c9b88e174395d572f26580fa319d6a14ff2cb
2019-06-06 14:32:05 -07:00
Daniel Norman 6728fe850e Merge "Renames system to framework and other to vendor." 2019-06-06 21:14:14 +00:00
Tao Bao 41376712ca Merge "Rebuild recovery-from-boot patch when calling add_img_to_target_files." am: e55c0ae469
am: d4680b2f33

Change-Id: Ifa54144ab48d54b7b2b18425f2951f249b625274
2019-06-05 21:19:26 -07:00
Daniel Norman d5d70ea555 Renames system to framework and other to vendor.
Also fixes small nit from previous change to write_sorted_data().

Bug: 132788610
Test: python -m unittest test_merge_target_files
Test: Creating and booting a merged build.
Change-Id: I3dc43a4fe55b86b436dec08feb5d70096d38de36
2019-06-05 16:47:15 -07:00
Tao Bao 4978fa99d1 Rebuild recovery-from-boot patch when calling add_img_to_target_files.
When using Verified Boot 2.0, releasetools specifies a salt value based
on build fingerprint, so that to give idempotent images.

However, the change that removed static `ro.build.fingerprint` [1] broke
the behavior, as common.LoadInfoDict still relies on fingerprints.
Without a fixed salt, the first call to make_recovery_patch.py and the
second one (which writes IMAGES/{boot,recovery}.img) will see different
images, which leads to install-recovery.sh failure.

Note that currently there's a dependency that requires getting bootable
images through two separate calls. make_recovery_patch.py has to happen
first to get (placeholder) files in the system image. We then generate
canned fs_config files, and finally use add_img_to_target_files.py to
write the images.

This CL adds a quick workaround to force rebuilding the
recovery-from-boot patch while calling add_img_to_target_files.py.

[1] https://android-review.googlesource.com/c/platform/build/+/892933

Bug: 134123803
Bug: 134525174
Test: TreeHugger
Test: Build a non-A/B target that uses AVB. Run validate_target_files.py
      on the generated target_files.zip.
Change-Id: I5859e30be63bfd54398cf41fd2d907f15285f560
2019-06-04 22:44:45 -07:00
Tao Bao 4b2910f8ae Merge "releasetools: Allow verifying chained vbmeta partitions." am: 23def8793d
am: 01ca117595

Change-Id: I3508b65759c1b3151d5d84348d1c3f6e9e4e07ee
2019-06-04 19:28:52 -07:00
Tao Bao 08c190fc89 releasetools: Allow verifying chained vbmeta partitions.
Chained vbmeta partitions (vbmeta_system, vbmeta_vendor) were added to
support dynamic partitions. validate_target_files.py misses the logic in
handling such partitions.

Bug: 132882632
Test: Run validate_target_files.py on a target_files.zip that uses
      chained vbmeta_system partition.
Change-Id: Id06c575d13d5e9cc1b621f485ceb75d3e354c39f
Merged-In: Id06c575d13d5e9cc1b621f485ceb75d3e354c39f
(cherry picked from commit 814b14b3f7)
2019-06-04 12:49:41 -07:00
Daniel Norman 563603e4d0 Merge "Adds validation check that certain partitions come from a single build." am: 206ef1f4fb
am: de3007658e

Change-Id: Ia8a6164d2f466f5ae98cf8250dbebdd99906b53c
2019-06-03 07:40:21 -07:00
Daniel Norman edf124780f Adds validation check that certain partitions come from a single build.
This is to prevent a user from accidentally including files from the
wrong build. For example, adding any SYSTEM/ line to other_item_list
while keeping SYSTEM/* in system_item_list would cause the other build
to introduce an extra or changed file in the system image.

Bug: 132730710
Test: python -m unittest test_merge_target_files
Change-Id: Ic1178cdc9b991114f293ff3f2b4e6054e06647c6
2019-06-01 04:39:46 +00:00
Daniel Norman 4d3dcaf5ab Merge "Adds image-specific selinux_fc props to allow different file_contexts." am: c9fc1120c1
am: 07f55ba840

Change-Id: I02b37b267b816a1bcc8eee94651b147d1e45bfd8
2019-05-23 22:12:59 -07:00
Daniel Norman c9fc1120c1 Merge "Adds image-specific selinux_fc props to allow different file_contexts." 2019-05-23 23:42:13 +00:00
Daniel Norman 72c626f017 Adds image-specific selinux_fc props to allow different file_contexts.
This enables mixed builds to use the file_contexts.bin from the system
build when regenerating images that come from system target files, and
similarly for file_contexts.bin from the other build when regenerating
images from other target files.

In monolithic (non-mixed) builds all image-specific selinux_fc props
point to the same file_contexts.

Bug: 132108151
Test: Built and booted mixed build devices.
Change-Id: Id51ed6d96ea6337879f1ab21d47c93c67bc25312
2019-05-23 11:21:02 -07:00
Bowgo Tsai 63b4c9eb76 Moving /odm/build.prop to /odm/etc/buid.prop
am: 33ff602070

Change-Id: I9b26413a5fbfe43e900d79010075a4c234d4b310
2019-05-22 21:07:11 -07:00
Bowgo Tsai 33ff602070 Moving /odm/build.prop to /odm/etc/buid.prop
In device root directory, we have the following symlinks:
  - /odm/app -> /vendor/odm/app
  - /odm/bin -> /vendor/odm/bin
  - /odm/etc -> /vendor/odm/etc
  ...

This allows the Generic System Image (GSI) to be used on both devices:
  1) Has a physical odm partition, where those symlink will be hidden
     when /odm is used as the mount point
  2) Has no physical odm partition and fallback to /vendor/odm/.

We can't just have the symlink /odm -> /vendor/odm, because the former
devices won't have /vendor/odm directory, which leads to mount failure
when the mount point /odm is resolved to /vendor/odm.

The existing /vendor/odm/build.prop won't be loaded in the latter
devices, because there is no symlink:
    - /odm/build.prop -> /vendor/odm/build.prop.

Note that init blocks reading through direct symlinks (O_NOFOLLOW) so
the above symlink won't work either. This CL moves the odm build.prop
to /odm/etc/build.prop for init to load it (symlinks in earlier
components of the path will still be followed by O_NOFOLLOW).

Bug: 132128501
Test: boot a device and checks /odm/etc/build.prop is loaded
Test: make dist with an odm.img, checks $OUT/odm/etc/build.prop is loaded
Change-Id: I6f88763db755c9ec6068bfdd9cee81c19d72e9d7
Merged-In: I6f88763db755c9ec6068bfdd9cee81c19d72e9d7
(cherry picked from commit 6c62884000)
2019-05-22 16:14:53 +08:00
Bowgo Tsai 71a4d5cdd5 Moving /odm/build.prop to /odm/etc/buid.prop
In device root directory, we have the following symlinks:
  - /odm/app -> /vendor/odm/app
  - /odm/bin -> /vendor/odm/bin
  - /odm/etc -> /vendor/odm/etc
  ...

This allows the Generic System Image (GSI) to be used on both devices:
  1) Has a physical odm partition, where those symlink will be hidden
     when /odm is used as the mount point
  2) Has no physical odm partition and fallback to /vendor/odm/.

We can't just have the symlink /odm -> /vendor/odm, because the former
devices won't have /vendor/odm directory, which leads to mount failure
when the mount point /odm is resolved to /vendor/odm.

The existing /vendor/odm/build.prop won't be loaded in the latter
devices, because there is no symlink:
    - /odm/build.prop -> /vendor/odm/build.prop.

Note that init blocks reading through direct symlinks (O_NOFOLLOW) so
the above symlink won't work either. This CL moves the odm build.prop
to /odm/etc/build.prop for init to load it (symlinks in earlier
components of the path will still be followed by O_NOFOLLOW).

Bug: 132128501
Test: boot a device and checks /odm/etc/build.prop is loaded
Test: make dist with an odm.img, checks $OUT/odm/etc/build.prop is loaded
Change-Id: I6f88763db755c9ec6068bfdd9cee81c19d72e9d7
Merged-In: I6f88763db755c9ec6068bfdd9cee81c19d72e9d7
(cherry picked from commit 6c62884000)
2019-05-22 14:04:39 +08:00
Chris Gross 99a22624a5 Merge "Add merging of apexkeys and apkcerts metadata" am: a489a92fdc
am: 07d985d122

Change-Id: Ib5e6d864f3704ffeeb4adb1933bb34f92d3511d8
2019-05-20 18:24:32 -07:00
Chris Gross fabf50a577 Add merging of apexkeys and apkcerts metadata
Bug: 131437873
Test: Built system-only and vendor builds for merge. Compared
resulting apkcertx and apexkeys text files to that of a monolithic
build,
Test: Created colliding entries in both apexkeys and apkcerts text
files and ensure the script exited with an appropriate error message.
Test: Created unit tests to cover both non-colliding and colliding
entries
Change-Id: I6e42ce682ffa9059344e8cd63ba3a720c1f93452
2019-05-20 18:35:22 +00:00
Tao Bao f52dbb81fe Add sign_apex.py that signs a given APEX file.
Bug: 132687993, 131687150
This CL moves SignApex() from sign_target_files_apks into apex_utils,
and adds sign_apex that allows signing a standalone APEX file directly.

Test: Run the following command and check the output file.
$ build/make/tools/releasetools/sign_apex.py \
     -v \
     --container_key \
         build/make/target/product/security/testkey.x509.pem \
     --payload_key external/avb/test/data/testkey_rsa4096.pem \
     --payload_extra_args \
         "--signing_helper_with_files ./signing-helper.sh" \
     foo.apex \
     signed-foo.apex
Test: Run sign_target_files_apks.py on crosshatch target_files.zip.
Change-Id: I4b2422fd5cb1c60a3aa94511475e2a0e5b1666ca
2019-05-19 11:33:31 -07:00
Tao Bao 5a827e78fa Merge "Add sign_apex.py that signs a given APEX file." am: bc7e3f98f4
am: b67f0f099a

Change-Id: Ib05c05d436ae7052bf9ef9f37991fd05bb159e50
2019-05-13 16:18:23 -07:00
Tao Bao bc7e3f98f4 Merge "Add sign_apex.py that signs a given APEX file." 2019-05-13 22:47:24 +00:00
Tao Bao e7354ba447 Add sign_apex.py that signs a given APEX file.
This CL moves SignApex() from sign_target_files_apks into apex_utils,
and adds sign_apex that allows signing a standalone APEX file directly.

Test: Run the following command and check the output file.
$ build/make/tools/releasetools/sign_apex.py \
     -v \
     --container_key \
         build/make/target/product/security/testkey.x509.pem \
     --payload_key external/avb/test/data/testkey_rsa4096.pem \
     --payload_extra_args \
         "--signing_helper_with_files ./signing-helper.sh" \
     foo.apex \
     signed-foo.apex
Test: Run sign_target_files_apks.py on crosshatch target_files.zip.
Change-Id: I4b2422fd5cb1c60a3aa94511475e2a0e5b1666ca
2019-05-13 12:41:26 -07:00
Tao Bao 5ed6499abf Merge "releasetools: Unzip all files when generating image archive." am: e77dbd2a62
am: 189a8dff61

Change-Id: Idd065bff59364883e9f37487b904b3d277deb8f9
2019-05-10 21:15:59 -07:00
Tao Bao 1dac1d4164 releasetools: Unzip all files when generating image archive.
Commit 7df64c3e starts to call common.LoadInfoDict() when generating
image archive, which reads additional files under BOOT/, RECOVERY/ and
ROOT/. Unzip everything from the target_files.zip.

Bug: 132456827
Test: Run img_from_target_files.py on previously failing
      target_files.zip.
Change-Id: I22ee57c4f765bee9494478bf115b1581877401f4
2019-05-10 15:57:20 -07:00
Tao Bao a3c3b9bc64 releasetools: Unzip all files when generating image archive.
Commit 7df64c3e starts to call common.LoadInfoDict() when generating
image archive, which reads additional files under BOOT/, RECOVERY/ and
ROOT/. Unzip everything from the target_files.zip.

Bug: 132456827
Test: Run img_from_target_files.py on previously failing
      target_files.zip.
Change-Id: I22ee57c4f765bee9494478bf115b1581877401f4
2019-05-10 15:52:01 -07:00
Daniel Norman ddc1712376 Merge "Fixes typo when calling process_dynamic_partitions_info_txt." am: 74d91295fd
am: c937afafa1

Change-Id: Ifa0d4a1dd44e54b9b276879e8034af9b67103830
2019-05-09 13:11:40 -07:00
Daniel Norman 714bd12861 Fixes typo when calling process_dynamic_partitions_info_txt.
Test: Running merge_target_files.py
Change-Id: Ieddfbfe524824844fc12cc10347f667332149f7c
2019-05-08 16:20:02 -07:00
Tao Bao 9c1b48ce7a Merge "releasetools: Update the comments regarding selinux_fc." am: 45f74b104b
am: df24a30b5d

Change-Id: I8d8c7bb7889d1260bdd8bfb77c132244ccf5078b
2019-05-08 15:34:32 -07:00
Tao Bao 13ebc021ec releasetools: Update the comments regarding selinux_fc.
The former comment no longer applies, as we have been always packing
META/file_contexts.bin in a target_files.zip (commit aa7318c3, since
Nougat), and we no longer look for the one under BOOT/RAMDISK/ (commit
d14b8956, since Q).

Test: N/A
Change-Id: I03f361234bf440e942f21e5a624862590248544b
2019-05-08 12:40:27 -07:00
Chris Gross b56351ac7b Merge "Add BUILD_RECOVERY_IMAGE and BUILD_BOOT_IMAGE flags" am: 84c96eba1e
am: e9425427c2

Change-Id: I367fdc842308a47b6a68dd416788bffa61f11764
2019-05-08 11:35:40 -07:00
Chris Gross 84c96eba1e Merge "Add BUILD_RECOVERY_IMAGE and BUILD_BOOT_IMAGE flags" 2019-05-08 18:21:42 +00:00
Daniel Norman 51fb0bfb26 Merge "Begins merging META/dynamic_partitions_info.txt." am: 7eae467469
am: fd5091d06f

Change-Id: If8321e8b350ab56128904f3ce092c0e74c137250
2019-05-07 18:55:33 -07:00
Daniel Norman a61cde0747 Begins merging META/dynamic_partitions_info.txt.
This file is used by OTA generation so it needs to appear in mixed
builds with the combined content from the system and other versions of
the file.

Test: python -m unittest test_merge_target_files
Test: Running merge_target_files on a dynamic-partition-enabled build
and observing the resulting target files.
Bug: 131889742

Change-Id: I4ddbebc087e430f6307d0bd5461121a374e58ea4
2019-05-07 11:53:58 -07:00
Chris Gross a784ef197c Add BUILD_RECOVERY_IMAGE and BUILD_BOOT_IMAGE flags
Bug: 123428770
Test: Built system-only image and checked that no boot.img or
recovery.img files where created. Booted the resulting merged build on
device.

Change-Id: I760476502775e68125907c39e66b8665e789a798
2019-05-07 11:00:37 -07:00
Bill Peckham 148239a7dd Merge "Use file_contexts.bin from other build to regenerate images." am: 7dbbc3075b
am: db8a8c6799

Change-Id: I1f9c64b95cb174d5f3ddf719087f051c8c59f7e8
2019-05-07 10:53:46 -07:00
Bill Peckham 7dbbc3075b Merge "Use file_contexts.bin from other build to regenerate images." 2019-05-07 17:26:25 +00:00
Tao Bao 3cec508728 Merge "releasetools: Support replacing the signing keys for chained vbmeta." am: 32d69c6b74
am: e7e563c673

Change-Id: I1539f7244c99c017c5c8461397ec3d63d32c51f0
2019-05-06 20:33:05 -07:00
Tao Bao d403e7bc8e releasetools: Support replacing the signing keys for chained vbmeta.
Bug: 131710801
Test: Run sign_target_files_apks.py on a target that uses vbmeta_system.
Change-Id: I3bc526af3ec9f2680ca17ee5535607cff3ae9523
Merged-In: I3bc526af3ec9f2680ca17ee5535607cff3ae9523
(cherry picked from commit d6085d6834)
2019-05-06 20:11:17 -07:00
Bill Peckham 736b223d5f Use file_contexts.bin from other build to regenerate images.
The old process_file_contexts_bin function did not properly generate a usable
file_contexts.bin to regenerate images, so instead use the file_contexts.bin
from the other partial target files package. When combining any one of several
other partial target files packages with a single system partial target files
package, this file will properly apply contexts as long as the same source is
used for the system partial target files.

Test: Verify that file contexts are properlty applied to vendor image.
Bug: 131584454
Change-Id: I16f8cc3b7f2eb7f09746f0ddcb2c1daf3fd19da6
2019-05-06 23:23:23 +00:00
Tao Bao d6085d6834 releasetools: Support replacing the signing keys for chained vbmeta.
Bug: 131710801
Test: Run sign_target_files_apks.py on a target that uses vbmeta_system.
Change-Id: I3bc526af3ec9f2680ca17ee5535607cff3ae9523
2019-05-06 12:56:52 -07:00
Tao Bao 0a04bc5bf7 Merge "Ensure that 'release-keys' are set on properties" am: cc389947d0
am: 6c4aa77c24

Change-Id: I5de9c10ae3ecbae51a0808db8193ee54dc8a2a38
2019-05-03 15:23:22 -07:00
Magnus Strandh 6396797612 Ensure that 'release-keys' are set on properties
Some properties had 'test-keys' still set
after signing the target files zip for release.

These properties are now added to the RewriteProps
method.

Bug: 131810966
Test: manual
Test: `atest releasetools_test`
Change-Id: Ifb352ed28f5100f1e9f686d77e935723f7f6d3ae
Merged-In: Ifb352ed28f5100f1e9f686d77e935723f7f6d3ae
(cherry picked from commit 234f4b418f)
2019-05-03 11:41:07 -07:00
Magnus Strandh 234f4b418f Ensure that 'release-keys' are set on properties
Some properties had 'test-keys' still set
after signing the target files zip for release.

These properties are now added to the RewriteProps
method.

Bug: 131810966
Test: manual
Test: `atest releasetools_test`
Change-Id: Ifb352ed28f5100f1e9f686d77e935723f7f6d3ae
2019-05-03 10:42:35 -07:00
Tao Bao 721a294a1c Merge "Update the references to verity tools." am: 31c2bf905e
am: ee153db46a

Change-Id: I9f6b38c0538b8b0e9146c622dfb807f170398c8d
2019-05-03 06:59:30 -07:00
Tao Bao 31c2bf905e Merge "Update the references to verity tools." 2019-05-03 13:41:45 +00:00
Tao Bao d5ab10eed4 Update the references to verity tools.
Test: `m -j dist` with aosp_marlin-userdebug
Test: `atest releasetools_test`
Test: Run signing with otatools.
Test: TreeHugger
Change-Id: I60d5bad4435d588c5f97b2245ca67ed3d55299b7
2019-05-02 18:32:11 -07:00
Tao Bao fd6849faf0 Merge "releasetools: Skip on empty care_map." am: 65e41c4236
am: b45a15da18

Change-Id: If162f2fa1c627d4820b3acfd807af8a7b454fa5d
2019-05-02 08:48:29 -07:00
Tao Bao 77a4aa0c06 releasetools: Skip on empty care_map.
common.GetCareMap() may return an empty list on unavailable care_map
since the change in commit 8bdfb990ea.
Caller needs to handle such a case accordingly. This CL fixes the caller
in add_img_to_target_files.py, and changes the return value to None to
break legacy callers loudly.

Fixes: 131794385
Test: `atest releasetools_test`
Change-Id: I7c94f456064199237e84ef75732bdd10ebe31736
2019-05-01 22:21:25 -07:00
Yifan Hong e834be384e Merge "Add BOARD_SUPER_IMAGE_IN_UPDATE_PACKAGE flag." am: a2488b7d85
am: 6e2eb931a7

Change-Id: I7d1520bdaa9057680ecc3496d7baf2539d25d9e4
2019-04-30 13:14:11 -07:00
Yifan Hong a2488b7d85 Merge "Add BOARD_SUPER_IMAGE_IN_UPDATE_PACKAGE flag." 2019-04-30 19:49:12 +00:00
Yifan Hong 7df64c3e77 Add BOARD_SUPER_IMAGE_IN_UPDATE_PACKAGE flag.
When set, product-img-tag.zip contains super.img instead of individual
user images from target files. For virtual devices, super.img is needed
to boot the device, but individual user images aren't needed.

Test: on A/B DAP, with flag set:
     - m updatepackage and look at img.zip
     - img_from_target_files
     both have super.img and not system / vendor / system_other
Test: on non-A/B DAP, with the flag set:
     - m updatepackage and look at img.zip
     - img_from_target_files
     both have super.img and not system / vendor
Test: on A/B retrofit, with the flag set:
     - m updatepackage and look at img.zip
     - img_from_target_files
     both have super_*.img and system_other.img, but not system / vendor
Bug: 113175337

Change-Id: I94e33091d0c837cae40776176b4dcfdd338aba90
(cherry picked from commit 0e97dbb8ca)
Merged-In: I94e33091d0c837cae40776176b4dcfdd338aba90
2019-04-30 11:36:16 -07:00
Yifan Hong 0e97dbb8ca Add BOARD_SUPER_IMAGE_IN_UPDATE_PACKAGE flag.
When set, product-img-tag.zip contains super.img instead of individual
user images from target files. For virtual devices, super.img is needed
to boot the device, but individual user images aren't needed.

Test: on A/B DAP, with flag set:
     - m updatepackage and look at img.zip
     - img_from_target_files
     both have super.img and not system / vendor / system_other
Test: on non-A/B DAP, with the flag set:
     - m updatepackage and look at img.zip
     - img_from_target_files
     both have super.img and not system / vendor
Test: on A/B retrofit, with the flag set:
     - m updatepackage and look at img.zip
     - img_from_target_files
     both have super_*.img and system_other.img, but not system / vendor
Bug: 113175337

Change-Id: I94e33091d0c837cae40776176b4dcfdd338aba90
2019-04-29 10:50:42 -07:00
Tao Bao 44e50526f3 releasetools: Accept PRESIGNED keys in apexkeys.txt.
am: 548db7d797

Change-Id: I3a18a2805622d76032eca95b9d7cd886bf1929d9
2019-04-26 16:53:47 -07:00
Tao Bao 7ccec4fe8a Merge "releasetools: GetCareMap supports non-sparse image." am: 0376d7dcbb
am: 3d32417e74

Change-Id: I322fc45909f624667df3f1ec75a8d485ad24ead2
2019-04-26 16:15:46 -07:00
Tao Bao 0376d7dcbb Merge "releasetools: GetCareMap supports non-sparse image." 2019-04-26 22:59:30 +00:00
Yifan Hong 46f7e88c27 Merge changes from topic "productmanifest" into qt-dev
* changes:
  ota_from_target_files: include metadata when odm / product is changed.
  Add product manifest.
2019-04-26 22:39:01 +00:00
Bill Peckham c27faf6ef1 Merge "Fix missing import." am: 3129d2c971
am: 3891ce9ed4

Change-Id: Iff2c839cd9a0036b1684c222848837ed295df23e
2019-04-25 19:55:02 -07:00
Bill Peckham 3129d2c971 Merge "Fix missing import." 2019-04-26 02:38:08 +00:00
Yifan Hong 4298a83592 Merge changes from topic "productmanifest" am: 001ebfcb30
am: 4d90d659c8

Change-Id: Ia5d6cf7874e3c7a5b4a9adc34713f4c121bf7136
2019-04-25 16:15:30 -07:00
Yifan Hong 001ebfcb30 Merge changes from topic "productmanifest"
* changes:
  ota_from_target_files: include metadata when odm / product is changed.
  Add product manifest.
2019-04-25 22:51:27 +00:00
Bill Peckham 540d91ae3f Fix missing import.
Test: Verify that script runs properly now that import is present.
Change-Id: I255bc6ca2ba1f2cc387001c6f4bf9690a90eb47c
2019-04-25 14:18:16 -07:00
Yifan Hong 51d375610c ota_from_target_files: include metadata when odm / product is changed.
When odm is changed, device manifest/matrices should be included.
When product is changed, framework manifest/matrices should be included.

Bug: 130714844
Bug: 126770403
Test: build with odm and product VINTF metadata

Change-Id: I49c8083e0e7185ae7b96047d68f1f624b1113dfc
2019-04-25 10:22:39 -07:00
Bill Peckham 912d2dff70 Merge "Sort content of merged target files package." am: d540f5a9a5
am: d40f5b6ec0

Change-Id: I80bfda5dd7367e964629c7a4201a5c772ececa90
2019-04-25 09:38:09 -07:00
Bill Peckham d540f5a9a5 Merge "Sort content of merged target files package." 2019-04-25 16:19:47 +00:00
Tao Bao 8bdfb990ea releasetools: GetCareMap supports non-sparse image.
Test: `atest --host releasetools_test`
Test: `m dist` with a target that uses non-sparse images.
Test: Run UpdateVerifierTest on blueline.
Change-Id: I8fdebee42fcaac78c2d1be2a84ddb69f46ec701d
2019-04-25 09:15:07 -07:00
Tao Bao 1eef250cd7 Merge "releasetools: Accept PRESIGNED keys in apexkeys.txt." am: b5b055ab50
am: 541468bb9a

Change-Id: Iae7ba723cdda16fcb653bbef760196cca69ce49f
2019-04-25 07:54:01 -07:00
Tao Bao 548db7d797 releasetools: Accept PRESIGNED keys in apexkeys.txt.
For an PRESIGNED APEX, it has the following format, which should be
considered as a valid input.

name="foo.apex" public_key="PRESIGNED" private_key="PRESIGNED" container_certificate="PRESIGNED" container_private_key="PRESIGNED"

Bug: 131153746
Test: Run sign_target_files_apks.py on a target_files.zip with PRESIGNED
      APEXes.
Test: python -m unittest sign_target_files_apks
Change-Id: I51076b0c6eddfb75637d37659a08009f0a88e931
(cherry picked from commit f454c3a0b4)
2019-04-25 00:20:00 -07:00
Tao Bao f454c3a0b4 releasetools: Accept PRESIGNED keys in apexkeys.txt.
For an PRESIGNED APEX, it has the following format, which should be
considered as a valid input.

name="foo.apex" public_key="PRESIGNED" private_key="PRESIGNED" container_certificate="PRESIGNED" container_private_key="PRESIGNED"

Bug: 131153746
Test: Run sign_target_files_apks.py on a target_files.zip with PRESIGNED
      APEXes.
Test: python -m unittest sign_target_files_apks
Change-Id: I51076b0c6eddfb75637d37659a08009f0a88e931
2019-04-25 00:10:07 -07:00
Tao Bao a0176c7676 Merge "releasetools: PRESIGNED APEX container entails PRESIGNED payload." am: a212ab59df
am: ff354bf918

Change-Id: If1f9a101a15a26c728ccd8719cd8b13b80e1cefc
2019-04-24 19:19:11 -07:00
Tao Bao a212ab59df Merge "releasetools: PRESIGNED APEX container entails PRESIGNED payload." 2019-04-25 02:03:50 +00:00
Bill Peckham 9662cfb234 Sort content of merged target files package.
By sorting the content of the final output merged target files package, the
merged target files package is more like the target files packages generated by
a build.

Test: Generate merged target files package, verify that content is sorted.
Change-Id: Ic0c198630ebd7692a3f3f9663d85e4b45229175c
2019-04-24 17:59:01 -07:00
Daniel Norman 93c435dfa8 Merge "Adds support to create img from target files directory instead of zip." am: 74a679d963
am: da27680ccb

Change-Id: Ia69ede2ea164ccd21d525c53da36b7104a27514d
2019-04-24 17:16:47 -07:00
Yifan Hong 2dcc1dcf9c ota_from_target_files: include metadata when odm / product is changed.
When odm is changed, device manifest/matrices should be included.
When product is changed, framework manifest/matrices should be included.

Bug: 130714844
Bug: 126770403
Test: build with odm and product VINTF metadata

Change-Id: I49c8083e0e7185ae7b96047d68f1f624b1113dfc
Merged-In: I49c8083e0e7185ae7b96047d68f1f624b1113dfc
2019-04-24 16:28:49 -07:00
Tao Bao f98fa1094e releasetools: PRESIGNED APEX container entails PRESIGNED payload.
We used to require explicitly setting both (e.g. `-e foo.apex=` and
`--extra_apex_payload_key foo.apex=` to skip signing `foo.apex`).

This CL allows specifying `-e` alone to achieve the same result.
However, if a conflicting `--extra_apex_payload_key` is also specified,
that would be considered as a config error.

Bug: 131153746
Test: Run sign_target_files_apks.py with `-e foo.apex=` alone to skip
      signing foo.apex.
Test: Run sign_target_files_apks.py with `-e foo.apex=` and
      `--extra_apex_payload_key foo.apex=key` and expect assertion error.
Change-Id: Ia747f59ee726b60bdb1445024e749320171064c2
2019-04-24 15:17:23 -07:00
Daniel Norman b8a2f9df51 Adds support to create img from target files directory instead of zip.
This is used by merge_target_files to prevent an unnecessary unzip and
copy.

Test: Ran merge_target_files.py and booted using the img.zip.
Change-Id: I6fe0dd025b30b3f4965c9b22fb6943019bf5899b
2019-04-24 20:53:22 +00:00
Bowgo Tsai 08aca59c53 Avoid signing debuggable boot-debug.img
The boot-debug.img should NOT be release signed and can only be used
if the device is unlocked. Adding a check to prevent the tool from
signing this debuggable boot.img.

See the following for more details about boot-debug.img:

  https://android-review.googlesource.com/c/platform/build/+/947857

Bug: 126493225
Test: put a file /force_debuggable into boot.img, checks the following
      command fails:
        ./build/tools/releasetools/sign_target_files_apks \
        out/dist/*-target_files-*.zip signed-target_files.zip
Change-Id: Ia5232949cb9582d2b4eaa171d9e9f3fe7317d418
Merged-In: Ia5232949cb9582d2b4eaa171d9e9f3fe7317d418
(cherry picked from commit 78369ebbc1)
2019-04-23 21:00:08 +08:00
Bowgo Tsai 65feee7adb Merge "Avoid signing debuggable boot-debug.img" am: f0151b1514
am: 2b90184d5c

Change-Id: I9f04585f1d216cae12508372d873aa011fb36f45
2019-04-23 00:51:33 -07:00
Bowgo Tsai 78369ebbc1 Avoid signing debuggable boot-debug.img
The boot-debug.img should NOT be release signed and can only be used
if the device is unlocked. Adding a check to prevent the tool from
signing this debuggable boot.img.

See the following for more details about boot-debug.img:

  https://android-review.googlesource.com/c/platform/build/+/947857

Bug: 126493225
Test: put a file /force_debuggable into boot.img, checks the following
      command fails:
        ./build/tools/releasetools/sign_target_files_apks \
        out/dist/*-target_files-*.zip signed-target_files.zip
Change-Id: Ia5232949cb9582d2b4eaa171d9e9f3fe7317d418
2019-04-23 12:55:10 +08:00
Tao Bao f9f4cce315 Merge "releasetools: Update the comment on common.AVB_PARTITIONS." am: 7b3e97b1ea
am: 5a5d4e0d63

Change-Id: Idb1beb657253e682edddeb4ca5ef1270f05dda7d
2019-04-22 20:32:45 -07:00
Tao Bao 7b3e97b1ea Merge "releasetools: Update the comment on common.AVB_PARTITIONS." 2019-04-23 03:09:26 +00:00
Daniel Norman e2a848a20d Merge "Adds --output-img flag to enable building the IMG package." am: 235f7d7686
am: a40cffbea9

Change-Id: Ida8357e302152d1e62ff87ff3072102562b7a901
2019-04-18 20:00:53 -07:00
Daniel Norman 1bd2a1d946 Adds --output-img flag to enable building the IMG package.
This simplifies the use case for mixed build users. Instead of having to
remember to call img_from_target_files.py after this script, they can
use this flag to automatically create the IMG package.

Also includes an update to super_empty.img logic. The super_empty.img is
now always created for dynamic-partition builds. The flag now only
controls copying the super_empty.img to a user-provided location.

Bug: 129976345
Test: Ran merge_target_files.py using --output-img and
--output-super-empty and inspected the resulting img zip and
super_empty.img.

Change-Id: Ida602942bb7a6b4b94f4e225640af9104fc9360c
2019-04-18 16:37:29 -07:00
Daniel Norman 969025e04e Merge "Runs python formatter and linter over merge_target_files." am: 8551d21cef
am: 99a82fcb0d

Change-Id: Id9637057db627fff245b121b654965cefd39e585
2019-04-17 19:47:46 -07:00
Daniel Norman 8fb00a71aa Merge changes from topic "merge_target_files_new_flags" am: 324d553c7c
am: b6a15bca81

Change-Id: Ia25149af2e4333a1c7f502aa1abfa25cde3422eb
2019-04-17 16:14:33 -07:00
Daniel Norman e5b134aa02 Runs python formatter and linter over merge_target_files.
Test: python -m unittest test_merge_target_files
Change-Id: I95c8611439a019cd21ba16a93327f35e2035a84d
2019-04-17 14:54:06 -07:00
Daniel Norman 3b64ce1437 Adds --output-ota flag to enable building the OTA package.
This simplifies the use case for mixed build users. Instead of having to
remember to call ota_from_target_files.py after this script, they can
use this flag to automatically create the OTA package.

Bug: 129976345
Test: Ran merge_target_files.py using --output-ota and inspected the
resulting zip.

Change-Id: Icc95943c24b8f83b3221e845a7d69a34c1edb4fc
2019-04-17 10:28:57 -07:00
Daniel Norman f031825560 Adds --output-super-empty flag to enable building super_empty.img.
Any mixed build that uses dynamic partitions will require a
super_empty.img image. This image must be created from the merged
misc_info.txt file, so adding this functionality here simplifies
the creation of this image for users (versus having to call
build_super_image.py manually after calling merge_target_files.py).

Bug: 129976345
Test: Ran merge_target_files.py on a dynamic partition enabled build
using the new --output-super-empty flag.

Change-Id: I73901f363d73c9fae1af1579faa2a908369dbbec
2019-04-17 10:22:11 -07:00
Daniel Norman fdb38817ea Adds output-dir and output-item-list for copying only certain files.
This provides the ability to run merge_target_files without the end goal
of a target files zip. This is useful for users that only want the IMAGES
folder, for example.

Bug: 130304869
Test: python -m unittest test_merge_target_files
Change-Id: If0412b8e1eb85fe09d7b689fd7f56ce84067faea
2019-04-17 10:17:50 -07:00
Tao Bao 5db3c58919 Merge "releasetools: Enable a few tests in OtaFromTargetFilesTest." am: 56215ba930
am: dc0ad08e49

Change-Id: I614c6d54d93cf5e3fb6b603b8640f5652c4dd8d3
2019-04-16 07:27:26 -07:00
Tao Bao 69909c96ba releasetools: Enable a few tests in OtaFromTargetFilesTest.
They used to be disabled due to the assertion of search_path in setUp()
function, which is not a prerequisite for most of the tests.

Bug: 112080715
Test: `atest releasetools_test`
Test: TreeHugger
Change-Id: I3cbaf42aa09dba0b87a64e11d97de9b3f7af7a47
2019-04-15 07:33:32 -07:00
Tao Bao 61a4e5bb67 Merge "relesetools: Enable releasetools_test in presubmit." am: 2bfc94ebc6
am: 4a5539cb99

Change-Id: Ic6ab973716fbdafec6673ab6a0082084b78b6dd9
2019-04-14 08:26:22 -07:00
Tao Bao 2bfc94ebc6 Merge "relesetools: Enable releasetools_test in presubmit." 2019-04-14 15:09:19 +00:00
Tao Bao d8cdbb7234 Merge "relesetools: Support running all the unittests with atest." am: 2bb1380ee5
am: 74575ccf97

Change-Id: Ib028768aca6a9dda082616c7fb6a5f23ef8e8d11
2019-04-12 20:11:29 -07:00
Yifan Hong 5c53266156 Merge "FileImage: make thread-safe" am: c7b0df0295
am: c9ab0da8d6

Change-Id: I8c755e464d423551bd67829daa28136494bf0697
2019-04-12 20:10:27 -07:00
Tao Bao 2bb1380ee5 Merge "relesetools: Support running all the unittests with atest." 2019-04-13 01:50:09 +00:00
Yifan Hong 210ede323f FileImage: make thread-safe
FileImage needs to be thread-safe because multiple
threads gets data from it when an incremental OTA
package is created.

Test: apply incremental OTA on cuttlefish
Bug: 113175337
Change-Id: I31637fce0fbd66f3fa6c5c478da09bae65a52229
Merged-In: I31637fce0fbd66f3fa6c5c478da09bae65a52229
2019-04-12 15:59:03 -07:00
Yifan Hong 55988c4677 FileImage: make thread-safe
FileImage needs to be thread-safe because multiple
threads gets data from it when an incremental OTA
package is created.

Test: apply incremental OTA on cuttlefish
Bug: 113175337
Change-Id: I31637fce0fbd66f3fa6c5c478da09bae65a52229
2019-04-12 15:38:48 -07:00
Tao Bao 82490d3de1 relesetools: Enable releasetools_test in presubmit.
About half of the testcases rely on external tools (i.e. the ones in
`otatools.zip`, which are external to releasetools module, but still
built by Android). It's WAI as releasetools scripts are mostly for
gluing purpose.

However, the current support in Soong doesn't allow packing the helper
modules as part of the built releasetools_test. This CL adds a decorator
that allows declaring external dependencies in testcases, which will be
skipped while running in presubmit. It doesn't affect local invocation
of `atest releasetools_test`.

Fixes: 112080715
Test: `atest releasetools_test`
Test: TreeHugger; check that releasetools_test is invoked (and test
      passes).
Change-Id: I8fdeb6549023cf5ddeb79d610c7c37cf9f13d3cc
2019-04-12 15:26:35 -07:00