The build servers are no longer using this. It's hard to search for
others using it -- I've cleaned them up as I've run across them, but
we'll just have to see who breaks once this goes in.
This was the last user of makeparallel, so we can remove it.
Test: make
Test: treehugger
Change-Id: If6df3f1a67d6a6df36afaa4b07cd88a48a364fe1
I've switched all the build server configs to using TARGET_PRODUCT /
TARGET_BUILD_VARIANT and explicit goals instead.
Remove tools/check_builds.sh which relied on this, but hasn't been
touched in a long time.
Test: m PRODUCT-test
Test: treehugger
Change-Id: If5f8c714567b33aeb38223c176ca24ea649eb57d
It used to be packed at IMAGES/recovery-two-step.img, but to serve OTA
purpose only.
Test: `m dist` with a non-A/B target. Check the file in the generated
target_files.zip.
Test: Create two-step package. Check that recovery-two-step.img is used.
Change-Id: Iec6a73c682e0f844cd8c0b758c9470fa35dd15d8
Based on the actual semantics, it actually wants an in-memory _bytes_
buffer (io.BytesIO), especially when running with Python 3. This CL
fixes the issue and adds a unittest.
Bug: 131631303
Test: python -m unittest test_sign_target_files_apks
Test: python3 -m unittest test_sign_target_files_apks
Change-Id: I3fb067acc26713f1842e831225607779fd0d1b7e
Bug: 73062966
Test: can load AIDs from partitions other than vendor
Test: bionic-unit-tests
Merged-In: Ia85abbeefe5a945369970f2aef42692e07ab8c09
Change-Id: Ia85abbeefe5a945369970f2aef42692e07ab8c09
(cherry picked from commit 8a54ec8482)
Bug: 73062966
Test: can load AIDs from partitions other than vendor
Test: bionic-unit-tests
Merged-In: Ia85abbeefe5a945369970f2aef42692e07ab8c09
Change-Id: Ia85abbeefe5a945369970f2aef42692e07ab8c09
aapt does not support parsing some referencing attributes.
Switch to latest tool, aapt2, which works better.
Test: aapt2 dump badging test.apk and returned 0.
Test: Run check_target_files_signatures.py with a target_files.zip.
Test: Run sign_target_files_apks.py with a target_files.zip.
Change-Id: Ib3a4740506f29ebab6930767f2aa8a0b5c4ba053
- Removes functions that can be replaced with one-line external methods
- Moves read_config_list to common alongside similar methods LoadDictionaryFrom*
- Runs pyformat on merge_target_files.py
Bug: 137853921
Test: python -m unittest test_merge_target_files
Test: Using merge_target_files.py to create a merged build, & booting.
Change-Id: I833c1086db41e1374057cc7447fc50d1915734a7
This helps to break the circular dependency between common and
blockimgdiff.
Bug: 32379627
Test: unit tests pass
Change-Id: I90b5ff34782acbfac86f36265bd96c207d898bf6
This allows sharing the same signing config on different target_files
zips. Nonexistent APEX will be ignored with a warning.
Bug: 137249701
Test: Run sign_target_files_apks with APEX overrides.
Change-Id: I2bad0f5c00753ed36ec5ae3431c7dc2ff1fc3e9c
Merged-In: I2bad0f5c00753ed36ec5ae3431c7dc2ff1fc3e9c
(cherry picked from commit b369c7226a)
For very long processes, we might want to keep stdout and stderr
by default to None.
So no redirection will occur in the child process as explained in:
https://docs.python.org/2/library/subprocess.html
That will result in the child process stdin and stderr to be same
than in common.py and avoid to have the logs blocked during the
child process execution and flushed only when child process terminates.
Since the logs are continously displayed, it allows to easily confirm
that the process is not blocked.
Bug: 133380588
Test: generate iota & Check that the logs are not blocked.
Change-Id: I6d6cb56547bf3a4a4334dfa22b6b2b05d2c36a5e
Signed-off-by: Regnier, Philippe <philippe.regnier@intel.com>
While in theory this could incur memory overhead for Python 2, the
impact is low for the existing use cases (plus we're moving away from
Python 2).
Bug: 131631303
Test: No additional occurrence of iteritems.
Test: Build with Python 3.
Change-Id: I0205c9edf25f46e3d85967c7dd2c1af035757741
This extends the original pin list generation to support specifying a
range within a file. If any part of a file is pinned, its local file
header in the APK file is pinned as well.
Test: Build cuttlefish.
Bug: 136040313
Bug: 135953430
Change-Id: I6de8c2b4c2f35c6bf257f61414c6bce761233e51
If a kernel happens to decode to gibberish (including
non-ascii bytes), catch the decoding error when
searching for the kernel version.
Bug: 137041171
Merged-In: Ic035b3a5c8c80025cb3cede7b0fdcf8a2e5a35fd
Change-Id: Ic035b3a5c8c80025cb3cede7b0fdcf8a2e5a35fd
And a few minor clean-ups to the styling.
Bug: 131631303
Test: python -m unittest test_merge_target_files
Test: python3 -m unittest test_merge_target_files
Test: Use `python merge_target_files` to merge two target_files zips.
Test: Use `python3 merge_target_files` to merge two target_files zips.
Change-Id: I8502dfb243408f658d022e8d5e5fbb60066e4ff0
Split merge_target_files function into several steps, so we can increate
readability and add conditional flows for other *SSI mixed build with less effort
Test: m -j & atest passed
Change-Id: I558f9dd5bca31b132a09cb36d9dfcd30c92efbc9
This CL changes the condition for building super_empty.img from
PRODUCT_BUILD_SUPER_PARTITION to PRODUCT_USE_DYNAMIC_PARTITIONS, as a
follow-up to the change in [1].
With the CL in [1], it skips building super.img and super_empty.img both
when turning off PRODUCT_BUILD_SUPER_PARTITION. However, the latter
should be mandatory whenever dynamic partitions is enabled. Because
fastboot relies on this file to properly flash dynamic partitions. Plus,
the cost for building super_empty.img is much lower than the one for
super.img.
As part of the change, it'll write group info into target_files when
building with PRODUCT_BUILD_SUPER_PARTITION == false. It's the work for
target_files merging script to determine the values to be picked up. The
current logic in merge_target_files.py always uses the one from vendor
target_files. This CL adds a testcase to ensure the behavior.
[1] https://android-review.googlesource.com/c/platform/build/+/928756
Bug: 135752763
Test: `m dist` with a target that sets PRODUCT_BUILD_SUPER_PARTITION to
false. Check the built artifacts contain super_empty.img. Verify
that the build can be flashed properly.
Change-Id: I277f087eab45663a6c3b33333d16e9e576c1c25c
This allows a consistent logic in using the avbtool which could be
board-specific.
Test: `atest releasetools_test`
Test: Run sign_target_files_apks.py on a target_files.zip.
Change-Id: I8cd93b8e71146985734f85c31f4662f5e2e9534c
This ensures a matching interface between sign_apex and apex_utils.
The test apex `testdata/foo.apex` is generated by running
`system/apex/apexer/runtests.sh`.
Test: python -m unittest test_sign_apex
Test: atest releasetools_test
Change-Id: I7c14b1df2a3038ad206aa3e5aac084c47baaa00b
There are some places defining same file open function and use
common.LoadDictionaryFromLines. This commit creates
LoadDictionaryFromFile to reduce some code redundancy.
Test: m -j & atest passed
Change-Id: I6a3fa48693095937f8c79ce6f3c110b6862a1967
In order to get a Python 2 and 3 compatible re-raise behavior, this CL
removes the stack traceback for the lines within apex_util module (i.e.
sys.exc_info()[2]). It's not a big loss in practice, since we only have
one line within the try-except block (`common.RunAndCheckOutput()`)
that's no longer reported in the traceback.
Using `six` module could better solve this, but only after building
releasetools as python_binary_host modules where we can properly handle
the module dependency.
Bug: 131631303
Test: TreeHugger
Test: `python -m unittest test_apex_utils`
Test: `python3 -m unittest test_apex_utils`
Change-Id: I0c5a72ec9fad5ff9d8c9c94d29e813e433ec2921
-p only applies to tar invocations that cause extraction, but -rf is
just appending to a tar file.
toybox tar doesn't have -p, and it doesn't seem worth implementing if
this is our only "user".
Test: treehugger
Change-Id: I025c496b04cb57cb31ba81fd516fd40b158bc06b
* Read/write warning messages as utf-8 strings.
* No more embedded raw utf-8 code in file.
Test: ./warn.py --csvpath warnings.csv build.log > warnings.html
Change-Id: Ie51700721a32bede1d3b250f4a42fd8facd6af75
sign_target_files_apks.py only needs to take care of the current
release. The legacy paths of ODM/build.prop, VENDOR/odm/build.prop, and
BOOT/RAMDISK/default.prop no longer exist in the target_files.zip from
master.
The other two cases of ROOT/default.prop and
RECOVERY/RAMDISK/default.prop are still kept in the code, as they will
still exist (as symlink or conditionally).
Test: Run sign_target_files_apks.py against
aosp_taimen-target_files.zip. Check the rewritten prop files.
Test: `python -m unittest test_sign_target_files_apks`
Change-Id: I5e70bc2ccc0f3dcf0eace0718c59a3b0f89a9ff4
Python 2 and 3 behave differently when calling ZipFile.writestr() with
zinfo.external_attr being 0. Python 3 uses `0o600 << 16` as the value
for such a case (since
18ee29d0b8),
which seems to make more sense. Otherwise the entry will end up with
0o000 as the permission bits. This CL updates common.ZipWriteStr to
follow the logic in Python 3, in order to get consistent behavior
between using the two versions.
Bug: 131631303
Test: `python -m unittest test_common.CommonZipTest`
Test: `python3 -m unittest test_common.CommonZipTest`
Change-Id: If8429855d922ef1ad76591f703215a0ce5089f0f
Previously it was using regular dict.
Test: python -m unittest test_common.DynamicPartitionsDifferenceTest
Change-Id: If108a4512aeaf9d3c8775c030cad6e44342b9d3d
While searching for some fs_config related symbols, I ran across this
being out of date, so update it.
The new documentation defers to the help section of the underlying
script more to hopefully avoid this in the future.
Test: n/a
Change-Id: I94893cbb29b8d7e81963a12e3f8316ac3f41b7ae
Previously, setting PRODUCT_BUILD_SUPER_PARTITION to false for a partial
build (with PRODUCT_USE_DYNAMIC_PARTITIONS == true) would fail to
include necessary keys in misc_info.txt that are required when merging
two partial builds to create a dynamic-partition-enabled mixed build.
This change ensures these necessary keys are included even when
PRODUCT_BUILD_SUPER_PARTITION is false. Setting
PRODUCT_BUILD_SUPER_PARTITION to false causes partial builds to skip
building super.img and super_empty.img, instead relying on these images
to come from the final merged build.
Bug: 134764140
Test: Building & booting a dynamic-partition-enabled mixed build, and
inspecting partial builds' logs / out folder to ensure that
super.img/super_empty.img were not created.
Change-Id: I99431a9a342e9b0617510e250597f3024ef39322
common.AVB_VBMETA_PARTITIONS was recently added (commit
08c190fc89) for the same purpose.
Test: TreeHugger
Change-Id: I65572d54c22a753fdef80677377fcc9b684ee16f
merge_target_files.py step.
Bug: 134681035
Test: Built a merged build and ensured that the new timestamp was
visible in the log.
Change-Id: Ia6bbda48c7f57afdb6482253eaf0b3b0ea067468
Also fixes small nit from previous change to write_sorted_data().
Bug: 132788610
Test: python -m unittest test_merge_target_files
Test: Creating and booting a merged build.
Change-Id: I3dc43a4fe55b86b436dec08feb5d70096d38de36
When using Verified Boot 2.0, releasetools specifies a salt value based
on build fingerprint, so that to give idempotent images.
However, the change that removed static `ro.build.fingerprint` [1] broke
the behavior, as common.LoadInfoDict still relies on fingerprints.
Without a fixed salt, the first call to make_recovery_patch.py and the
second one (which writes IMAGES/{boot,recovery}.img) will see different
images, which leads to install-recovery.sh failure.
Note that currently there's a dependency that requires getting bootable
images through two separate calls. make_recovery_patch.py has to happen
first to get (placeholder) files in the system image. We then generate
canned fs_config files, and finally use add_img_to_target_files.py to
write the images.
This CL adds a quick workaround to force rebuilding the
recovery-from-boot patch while calling add_img_to_target_files.py.
[1] https://android-review.googlesource.com/c/platform/build/+/892933
Bug: 134123803
Bug: 134525174
Test: TreeHugger
Test: Build a non-A/B target that uses AVB. Run validate_target_files.py
on the generated target_files.zip.
Change-Id: I5859e30be63bfd54398cf41fd2d907f15285f560
Chained vbmeta partitions (vbmeta_system, vbmeta_vendor) were added to
support dynamic partitions. validate_target_files.py misses the logic in
handling such partitions.
Bug: 132882632
Test: Run validate_target_files.py on a target_files.zip that uses
chained vbmeta_system partition.
Change-Id: Id06c575d13d5e9cc1b621f485ceb75d3e354c39f
Merged-In: Id06c575d13d5e9cc1b621f485ceb75d3e354c39f
(cherry picked from commit 814b14b3f7)
This is to prevent a user from accidentally including files from the
wrong build. For example, adding any SYSTEM/ line to other_item_list
while keeping SYSTEM/* in system_item_list would cause the other build
to introduce an extra or changed file in the system image.
Bug: 132730710
Test: python -m unittest test_merge_target_files
Change-Id: Ic1178cdc9b991114f293ff3f2b4e6054e06647c6
If the device has neither an OEM nor a vendor partiton,
$PRIVATE_PARTITION_LIST will be empty, causing the fsconfig tool to fail
because the "--all-partitions" argument will be empty.
Quote the comma-substituted "$(PRIVATE_PARTITION_LIST)" - which may be
empty - to appease argparse, which will then populate "_all_partitions"
with an empty "str" object.
Checking _all_partitions against None is superfluous since either
argparse will catch the empty argument and fail early, or the argument
will be a string, in which case it will not be None.
Signed-off-by: Felix <google@ix5.org>
Change-Id: I236e30445b303b4945467b5dc4387d54b5d984f9
This enables mixed builds to use the file_contexts.bin from the system
build when regenerating images that come from system target files, and
similarly for file_contexts.bin from the other build when regenerating
images from other target files.
In monolithic (non-mixed) builds all image-specific selinux_fc props
point to the same file_contexts.
Bug: 132108151
Test: Built and booted mixed build devices.
Change-Id: Id51ed6d96ea6337879f1ab21d47c93c67bc25312
In device root directory, we have the following symlinks:
- /odm/app -> /vendor/odm/app
- /odm/bin -> /vendor/odm/bin
- /odm/etc -> /vendor/odm/etc
...
This allows the Generic System Image (GSI) to be used on both devices:
1) Has a physical odm partition, where those symlink will be hidden
when /odm is used as the mount point
2) Has no physical odm partition and fallback to /vendor/odm/.
We can't just have the symlink /odm -> /vendor/odm, because the former
devices won't have /vendor/odm directory, which leads to mount failure
when the mount point /odm is resolved to /vendor/odm.
The existing /vendor/odm/build.prop won't be loaded in the latter
devices, because there is no symlink:
- /odm/build.prop -> /vendor/odm/build.prop.
Note that init blocks reading through direct symlinks (O_NOFOLLOW) so
the above symlink won't work either. This CL moves the odm build.prop
to /odm/etc/build.prop for init to load it (symlinks in earlier
components of the path will still be followed by O_NOFOLLOW).
Bug: 132128501
Test: boot a device and checks /odm/etc/build.prop is loaded
Test: make dist with an odm.img, checks $OUT/odm/etc/build.prop is loaded
Change-Id: I6f88763db755c9ec6068bfdd9cee81c19d72e9d7
Merged-In: I6f88763db755c9ec6068bfdd9cee81c19d72e9d7
(cherry picked from commit 6c62884000)
In device root directory, we have the following symlinks:
- /odm/app -> /vendor/odm/app
- /odm/bin -> /vendor/odm/bin
- /odm/etc -> /vendor/odm/etc
...
This allows the Generic System Image (GSI) to be used on both devices:
1) Has a physical odm partition, where those symlink will be hidden
when /odm is used as the mount point
2) Has no physical odm partition and fallback to /vendor/odm/.
We can't just have the symlink /odm -> /vendor/odm, because the former
devices won't have /vendor/odm directory, which leads to mount failure
when the mount point /odm is resolved to /vendor/odm.
The existing /vendor/odm/build.prop won't be loaded in the latter
devices, because there is no symlink:
- /odm/build.prop -> /vendor/odm/build.prop.
Note that init blocks reading through direct symlinks (O_NOFOLLOW) so
the above symlink won't work either. This CL moves the odm build.prop
to /odm/etc/build.prop for init to load it (symlinks in earlier
components of the path will still be followed by O_NOFOLLOW).
Bug: 132128501
Test: boot a device and checks /odm/etc/build.prop is loaded
Test: make dist with an odm.img, checks $OUT/odm/etc/build.prop is loaded
Change-Id: I6f88763db755c9ec6068bfdd9cee81c19d72e9d7
Merged-In: I6f88763db755c9ec6068bfdd9cee81c19d72e9d7
(cherry picked from commit 6c62884000)
Bug: 131437873
Test: Built system-only and vendor builds for merge. Compared
resulting apkcertx and apexkeys text files to that of a monolithic
build,
Test: Created colliding entries in both apexkeys and apkcerts text
files and ensure the script exited with an appropriate error message.
Test: Created unit tests to cover both non-colliding and colliding
entries
Change-Id: I6e42ce682ffa9059344e8cd63ba3a720c1f93452
Bug: 132687993, 131687150
This CL moves SignApex() from sign_target_files_apks into apex_utils,
and adds sign_apex that allows signing a standalone APEX file directly.
Test: Run the following command and check the output file.
$ build/make/tools/releasetools/sign_apex.py \
-v \
--container_key \
build/make/target/product/security/testkey.x509.pem \
--payload_key external/avb/test/data/testkey_rsa4096.pem \
--payload_extra_args \
"--signing_helper_with_files ./signing-helper.sh" \
foo.apex \
signed-foo.apex
Test: Run sign_target_files_apks.py on crosshatch target_files.zip.
Change-Id: I4b2422fd5cb1c60a3aa94511475e2a0e5b1666ca
This CL moves SignApex() from sign_target_files_apks into apex_utils,
and adds sign_apex that allows signing a standalone APEX file directly.
Test: Run the following command and check the output file.
$ build/make/tools/releasetools/sign_apex.py \
-v \
--container_key \
build/make/target/product/security/testkey.x509.pem \
--payload_key external/avb/test/data/testkey_rsa4096.pem \
--payload_extra_args \
"--signing_helper_with_files ./signing-helper.sh" \
foo.apex \
signed-foo.apex
Test: Run sign_target_files_apks.py on crosshatch target_files.zip.
Change-Id: I4b2422fd5cb1c60a3aa94511475e2a0e5b1666ca
Commit 7df64c3e starts to call common.LoadInfoDict() when generating
image archive, which reads additional files under BOOT/, RECOVERY/ and
ROOT/. Unzip everything from the target_files.zip.
Bug: 132456827
Test: Run img_from_target_files.py on previously failing
target_files.zip.
Change-Id: I22ee57c4f765bee9494478bf115b1581877401f4
Commit 7df64c3e starts to call common.LoadInfoDict() when generating
image archive, which reads additional files under BOOT/, RECOVERY/ and
ROOT/. Unzip everything from the target_files.zip.
Bug: 132456827
Test: Run img_from_target_files.py on previously failing
target_files.zip.
Change-Id: I22ee57c4f765bee9494478bf115b1581877401f4
The former comment no longer applies, as we have been always packing
META/file_contexts.bin in a target_files.zip (commit aa7318c3, since
Nougat), and we no longer look for the one under BOOT/RAMDISK/ (commit
d14b8956, since Q).
Test: N/A
Change-Id: I03f361234bf440e942f21e5a624862590248544b
This file is used by OTA generation so it needs to appear in mixed
builds with the combined content from the system and other versions of
the file.
Test: python -m unittest test_merge_target_files
Test: Running merge_target_files on a dynamic-partition-enabled build
and observing the resulting target files.
Bug: 131889742
Change-Id: I4ddbebc087e430f6307d0bd5461121a374e58ea4
Bug: 123428770
Test: Built system-only image and checked that no boot.img or
recovery.img files where created. Booted the resulting merged build on
device.
Change-Id: I760476502775e68125907c39e66b8665e789a798
Bug: 131710801
Test: Run sign_target_files_apks.py on a target that uses vbmeta_system.
Change-Id: I3bc526af3ec9f2680ca17ee5535607cff3ae9523
Merged-In: I3bc526af3ec9f2680ca17ee5535607cff3ae9523
(cherry picked from commit d6085d6834)
The old process_file_contexts_bin function did not properly generate a usable
file_contexts.bin to regenerate images, so instead use the file_contexts.bin
from the other partial target files package. When combining any one of several
other partial target files packages with a single system partial target files
package, this file will properly apply contexts as long as the same source is
used for the system partial target files.
Test: Verify that file contexts are properlty applied to vendor image.
Bug: 131584454
Change-Id: I16f8cc3b7f2eb7f09746f0ddcb2c1daf3fd19da6
Dan Willemsen