Commit Graph

5415 Commits

Author SHA1 Message Date
Luis Hector Chavez 21025515e4 Merge "Make mksquashfs generate a fs_config-friendly root entry" am: 3d4fb4e335
am: 6128d7f30d

Change-Id: I43a62ea7c4d6acb5adadfc488b94e05024499320
2018-02-27 16:36:39 +00:00
Treehugger Robot 3d4fb4e335 Merge "Make mksquashfs generate a fs_config-friendly root entry" 2018-02-27 16:24:21 +00:00
Dan Shi 6dbd9e3df9 Merge "Default test to AndroidJUnitTest for auto-generated test config." am: ce1fa25d1b
am: d91129eea9

Change-Id: Ib0e817e8d403ea130516dbc94201fd4a899c327f
2018-02-26 23:18:57 +00:00
Treehugger Robot ce1fa25d1b Merge "Default test to AndroidJUnitTest for auto-generated test config." 2018-02-26 23:00:28 +00:00
Tao Bao ed98b94326 Merge "releasetools: Remove the parsing of META/imagesizes.txt." am: 3aee4dfac1
am: 72f3c44243

Change-Id: I83af79b6d5a76099259a951585d92755b132af9e
2018-02-26 22:39:14 +00:00
Tao Bao b8d0b5bd06 Merge "releasetools: Add unittests for GetCareMap() and AddCareMapTxtForAbOta()." am: 7998503ebe
am: 5fed709d60

Change-Id: I117d452b1a4d63b7665c3593b5b2167fe57be954
2018-02-26 22:24:13 +00:00
Treehugger Robot 3aee4dfac1 Merge "releasetools: Remove the parsing of META/imagesizes.txt." 2018-02-26 22:18:31 +00:00
Tao Bao 7998503ebe Merge "releasetools: Add unittests for GetCareMap() and AddCareMapTxtForAbOta()." 2018-02-26 22:06:32 +00:00
Tao Bao 41fb7d27cf releasetools: Remove the parsing of META/imagesizes.txt.
The file has been removed from target-files.zip since commit
c19a8d5590 (Gingerbread), whose info has
been consolidated into META/misc_info.txt.

Test: `m dist`
Change-Id: Ic144457954f5742ea082dcd9ffbea71df4afe46e
2018-02-23 12:06:07 -08:00
Tao Bao a5baf56e6c Merge "releasetools: Allow skipping postinstall hooks when generating A/B OTAs." am: 6c0ddf2480
am: ff1288821d

Change-Id: Iea818eb91dd5ff98d9b378a845ff507e1dee1e0c
2018-02-22 20:40:25 +00:00
Tao Bao 6c0ddf2480 Merge "releasetools: Allow skipping postinstall hooks when generating A/B OTAs." 2018-02-22 20:21:36 +00:00
Tao Bao 894cba2954 Merge "releasetools: Check for duplicate entries in ReplaceCerts()." am: 8bfde7c5fb
am: 7032691417

Change-Id: Icc23606930f1a2f3e0009b1ec225723ffc8a7617
2018-02-22 17:57:03 +00:00
Tao Bao 8bfde7c5fb Merge "releasetools: Check for duplicate entries in ReplaceCerts()." 2018-02-22 17:39:00 +00:00
Tao Bao 63c18fead8 releasetools: Add unittests for GetCareMap() and AddCareMapTxtForAbOta().
With test_utils.construct_sparse_image(), creating valid sparse image is
no longer a blocker for testing these functions.

Test: python -m unittest test_add_img_to_target_files
Change-Id: Iff1f15837cc744bad52e2120f9a9ad94d7db50d5
2018-02-22 00:10:05 -08:00
Tao Bao 15a146a7c4 releasetools: Allow skipping postinstall hooks when generating A/B OTAs.
This CL adds a new flag '--skip_postinstall' that allows skipping all
the postinstall hooks when generating an A/B OTA package (default:
False). Note that this discards ALL the hooks, including non-optional
ones. Should only be used if caller knows it's safe to do so (e.g. all
the postinstall work is to dexopt apps and a data wipe will happen
immediately after).

Bug: 73547992
Test: python -m unittest test_ota_from_target_files
Test: Generate a full OTA package for walleye. Examine the generated
      payload.
Change-Id: Ifc069e897b4019605051eabfd221230a6a37867c
2018-02-21 16:34:19 -08:00
Dan Shi 96068b77ed Default test to AndroidJUnitTest for auto-generated test config.
All new instrumentation tests shall use AndroidJUnitRunner, thus default
test class to AndroidJUnitTest.

Bug: 73679956
Test: unittest
Change-Id: I52df4d01b8459a4a6e5c2cbe15973dab079fb136
2018-02-21 11:31:06 -08:00
Tao Bao 4e636f0441 Merge "releasetools: Clean up test_sign_target_files_apks.py." am: 5ef78033d5
am: e1fee81bb5

Change-Id: I3c504df0a047b88725433c68079c18495d536afe
2018-02-21 02:29:12 +00:00
Tao Bao 5ef78033d5 Merge "releasetools: Clean up test_sign_target_files_apks.py." 2018-02-21 02:06:00 +00:00
Tao Bao 6f142ef096 Merge "releasetools: Add a testcase for common.ParseCertificate()." am: a0f55ee9eb
am: 9773a2f7b2

Change-Id: I3b00809f57c7f42f69fdbb44cdc5516e585ee987
2018-02-21 00:28:14 +00:00
Tao Bao a0f55ee9eb Merge "releasetools: Add a testcase for common.ParseCertificate()." 2018-02-21 00:04:24 +00:00
Tao Bao 0fa48c8c52 Merge "releasetools: Add unittest for common.GetSparseImage()." am: 35ee189898
am: 625b5b7a09

Change-Id: I494f253ab122f4098ec1e1fd814c915e618e2aa2
2018-02-20 21:21:31 +00:00
Treehugger Robot 35ee189898 Merge "releasetools: Add unittest for common.GetSparseImage()." 2018-02-20 21:04:58 +00:00
Tao Bao 66472637ad releasetools: Check for duplicate entries in ReplaceCerts().
testdata/{media,platform}.x509.pem files are copied from
build/target/product/security/.

Fixes: 69479366
Test: python -m unittest test_sign_target_files_apks
Change-Id: I8ba42b6f5b5432ee4b8786b241daff11db792c14
2018-02-20 10:22:30 -08:00
Tao Bao de1d479ace releasetools: Clean up test_sign_target_files_apks.py.
Since we have been carrying test certificates in testdata/ for other
tests, do the same for test_sign_target_files_apks.py. Copy
verity.x509.pem from build/target/product/security/ to testdata/ for
that purpose.

Also capture the stderr output in ReplaceVerityKeyId().

Test: python -m unittest test_sign_target_files_apks
Change-Id: Ie11e042086952e8a4a5a63950cb0b16cc436b7e6
2018-02-20 10:12:15 -08:00
Dan Willemsen aa582f7dda Merge "Add Docker instructions for older branches" am: c6620457d8
am: cfc82c4b33

Change-Id: Ia6404fa198f502ad9be811a83ca19a7fee31bb17
2018-02-17 01:19:32 +00:00
Tao Bao 17e4e61691 releasetools: Add a testcase for common.ParseCertificate().
Also minor clean up to the function, to suppress the following lint
warning.
R:1813, 2: Redefinition of cert type from list to unicode (redefined-variable-type)

Test: python -m unittest -v test_common.CommonApkUtilsTest
Change-Id: Ib4b1dfa8b19e505fc70684b648efc36171c73bbf
2018-02-16 17:14:23 -08:00
Dan Willemsen 6dc4621afc Add Docker instructions for older branches
If you've upgraded away from Ubuntu 14.04 Trusty, you may find that
older branches (or certain code) doesn't build anymore. These
instructions can form a starting point to build these older branches
within Docker.

Bug: 72993575
Test: Follow instructions, build marshmallow-dev
Test: Follow instructions, build lollipop-mr1-dev
Change-Id: If4047b1bb5324d75a9d0947cb5280ff1cabccb6a
2018-02-15 18:17:46 -08:00
Tao Bao fc7e0e06b6 releasetools: Add unittest for common.GetSparseImage().
Add construct_sparse_image() to test_utils.py, which is a util function
to create sparse images. The new tests also partially cover the recent
changes that add 'incomplete' and 'uses_shard_blocks' tags.

Test: python -m unittest test_common
Change-Id: Ia15f5c4ad12423691216ebbad2c28f95c8427d7e
2018-02-14 15:12:52 -08:00
TreeHugger Robot 221d1f4bde Merge "releasetools: Specify max_timestamp for secondary payload." into oc-mr1-dev-plus-aosp 2018-02-14 00:26:57 +00:00
Tao Bao 7088b88a17 Merge "releasetools: Allow generating BBOTA for images with shared blocks." am: 9452d84b97
am: 6c453193a1

Change-Id: I422f6da0c524fa5329bc9dd11eb10eb27945ab92
2018-02-13 22:43:10 +00:00
Tao Bao 6b2854d44d Merge "releasetools: Remove the unconditional fallback to bsdiff." am: 6392e05813
am: 61fc7f6746

Change-Id: I3729f15bb532838ce2b9ec3f084ee5b81e97d3b1
2018-02-13 22:35:42 +00:00
Tao Bao db1fe41bbe releasetools: Specify max_timestamp for secondary payload.
This is a mandatory field in the generated payload.

Bug: 35724498
Test: Generate OTA with --include_secondary. Check the generation
      command for secondary payload.
Change-Id: Ib2e25cc020e294eae5a4be36d6dca297432804b8
Merged-In: Ib2e25cc020e294eae5a4be36d6dca297432804b8
(cherry picked from commit 6048121485)
2018-02-13 14:34:55 -08:00
Tao Bao 55e571f0b6 Merge "releasetools: Specify SWITCH_SLOT_ON_REBOOT for secondary payload." am: 4f0b725439
am: 78eef74e03

Change-Id: I02007c63b6a7398afe8b5304e48c01589842a876
2018-02-13 22:06:40 +00:00
Tao Bao 9452d84b97 Merge "releasetools: Allow generating BBOTA for images with shared blocks." 2018-02-13 21:53:25 +00:00
Tao Bao 6392e05813 Merge "releasetools: Remove the unconditional fallback to bsdiff." 2018-02-13 21:53:17 +00:00
Tao Bao 4f0b725439 Merge "releasetools: Specify SWITCH_SLOT_ON_REBOOT for secondary payload." 2018-02-13 20:20:30 +00:00
Tao Bao e709b094e4 releasetools: Allow generating BBOTA for images with shared blocks.
When target defines 'BOARD_EXT4_SHARE_DUP_BLOCKS := true', the generated
system/vendor images may contain shared blocks (i.e. some blocks will
show up in multiple files' block list), which violates the current
assumptions in BBOTA script.

This CL allows generating BBOTAs by considering the first occurrence as
the "owner" of the shared blocks. All the later users of the shared
blocks will have an incomplete block list, whose RangeSet's will be
tagged with 'uses_shared_blocks'.

Files with 'uses_shared_blocks' tag will not be diff'd with imgdiff,
potentially with patch size penalty. Such files will be accounted for in
imgdiff stats report, where we can revisit for a better solution.

Bug: 64109868
Test: Generate BBOTA full and incremental package with targets defining
      'BOARD_EXT4_SHARE_DUP_BLOCKS := true'.
Change-Id: I87fbc22eef7fafe2a470a03fdcfa1babf088ea8d
2018-02-13 11:14:23 -08:00
Tao Bao 4ccea8549e releasetools: Remove the unconditional fallback to bsdiff.
This CL uses the 'incomplete' tag to skip applying imgdiff to files with
incomplete block list. It's not the ideal fix to address the holes in
ext4 images, but would unhide other imgdiff issues covered by the
unconditional fallback.

Bug: 68016761
Test: Generate an incremental OTA package from images with incomplete
      block list. Check the imgdiff stats report.
Test: `python -m unittest test_blockimgdiff`
Change-Id: Ice77686414e70f5e42de35c1757fb31cf02e4fd4
2018-02-13 11:06:20 -08:00
Tao Bao 000274fda1 Merge changes Ia7ecd0e1,I33982473 am: 4a066a2cd2
am: b25e43bfcb

Change-Id: I52be4244cfb6faca30597a887a5abf7c5fed326f
2018-02-13 17:24:17 +00:00
Tao Bao 508b087943 releasetools: Make blockimgdiff.py pylint-clean.
************* Module blockimgdiff
C:433, 0: Unnecessary parens after 'if' keyword (superfluous-parens)
C:687, 0: Wrong hanging indentation (add 4 spaces).
            max_stashed_blocks, self._max_stashed_size, max_allowed,
            ^   | (bad-continuation)
C:688, 0: Wrong hanging indentation (add 4 spaces).
            self._max_stashed_size * 100.0 / max_allowed))
            ^   | (bad-continuation)
C:691, 0: Wrong hanging indentation (remove 2 spaces).
            max_stashed_blocks, self._max_stashed_size))
          | ^ (bad-continuation)
C:898, 0: Wrong hanging indentation (add 4 spaces).
                  "imgdiff" if imgdiff else "bsdiff",
                  ^   | (bad-continuation)
C:899, 0: Wrong hanging indentation (add 4 spaces).
                  xf.tgt_name if xf.tgt_name == xf.src_name else
                  ^   | (bad-continuation)
C:901, 0: Wrong hanging indentation (add 4 spaces).
                  xf.tgt_ranges, xf.src_ranges, e.message))
                  ^   | (bad-continuation)
C:909, 0: Wrong hanging indentation (add 4 spaces).
                      xf.tgt_name,))
                      ^   | (bad-continuation)
C:917, 0: Wrong hanging indentation (add 4 spaces).
                      xf.tgt_name, e.message))
                      ^   | (bad-continuation)
C:961, 0: Wrong hanging indentation (remove 2 spaces).
                xf.patch_len, tgt_size, xf.patch_len * 100.0 / tgt_size,
              | ^ (bad-continuation)
C:962, 0: Wrong hanging indentation (remove 2 spaces).
                xf.style,
              | ^ (bad-continuation)
C:963, 0: Wrong hanging indentation (remove 2 spaces).
                xf.tgt_name if xf.tgt_name == xf.src_name else (
              | ^ (bad-continuation)
C:965, 0: Wrong hanging indentation (remove 2 spaces).
                xf.tgt_ranges, xf.src_ranges))
              | ^ (bad-continuation)
C:1422, 0: Wrong continued indentation (add 28 spaces).
                tgt_skipped.size() * 100.0 / tgt_size, tgt_name))
                ^                           | (bad-continuation)
C:1560, 0: Wrong continued indentation (add 8 spaces).
            split_src_ranges) in enumerate(split_info_list):
            ^       | (bad-continuation)
R:566, 6: Redefinition of src_str type from list to str (redefined-variable-type)
C:1198,25: More than one statement on a single line (multiple-statements)
C:1211,25: More than one statement on a single line (multiple-statements)
C:1220,16: More than one statement on a single line (multiple-statements)
C:1277,38: More than one statement on a single line (multiple-statements)
C:1284,19: More than one statement on a single line (multiple-statements)
C: 19, 0: standard import "import copy" comes before "import common" (wrong-import-order)
C: 20, 0: standard import "import functools" comes before "import common" (wrong-import-order)
C: 21, 0: standard import "import heapq" comes before "import common" (wrong-import-order)
C: 22, 0: standard import "import itertools" comes before "import common" (wrong-import-order)
C: 23, 0: standard import "import multiprocessing" comes before "import common" (wrong-import-order)
C: 24, 0: standard import "import os" comes before "import common" (wrong-import-order)
C: 25, 0: standard import "import os.path" comes before "import common" (wrong-import-order)
C: 26, 0: standard import "import re" comes before "import common" (wrong-import-order)
C: 27, 0: standard import "import subprocess" comes before "import common" (wrong-import-order)
C: 28, 0: standard import "import sys" comes before "import common" (wrong-import-order)
C: 29, 0: standard import "import threading" comes before "import common" (wrong-import-order)
C: 31, 0: standard import "from collections import deque, OrderedDict" comes before "import common" (wrong-import-order)
C: 32, 0: standard import "from hashlib import sha1" comes before "import common" (wrong-import-order)

Test: `pylint --rcfile=pylintrc blockimgdiff.py`
Test: Generate an incremental BBOTA package.
Change-Id: Ia7ecd0e1fa48daf4e43251bdcdfcd08fb316015d
2018-02-12 13:01:34 -08:00
Tao Bao 9739514769 releasetools: Remove the global diff_done in blockimgdiff.py.
pylint complains about undefined `diff_done`:

W:754, 8: Global variable 'diff_done' undefined at the module level (global-variable-undefined)
W:820,14: Global variable 'diff_done' undefined at the module level (global-variable-undefined)

It would still warn about using global statement after adding the
definition.

W:859, 8: Using the global statement (global-statement)
W:925,14: Using the global statement (global-statement)

This CL computes 'diff_done' via 'len(diff_queue)' instead. It also
moves the progress reporting _before_ the diff work. This way it avoids
showing 100% progress with still changing filenames (because multiple
workers could see an empty queue simultaneously upon finishing their own
works).

There're possible alternatives, such as using the 'nonlocal' keyword in
Python 3 (which we're not there yet), or by using mutable object instead
(e.g. 'diff_done = [0]'). This CL looks cleaner, since it just kills the
var.

Test: Generate a BBOTA incremental. Check the on-screen progress
      report.
Test: `pylint --rcfile=pylintrc blockimgdiff.py` no longer complains
      about the global diff_done.
Change-Id: I339824735527e1f794b5b1dc99ff3fdb2da85744
2018-02-12 12:55:25 -08:00
Tao Bao e6188063f1 Merge "releasetools: Add an ImgdiffStats class that reports imgdiff stats." am: 7eb2afb226
am: 0ced5030d8

Change-Id: I120d8d268fc4a5e1230e16d9b9ab85b2bef20193
2018-02-12 20:28:11 +00:00
Tao Bao 7eb2afb226 Merge "releasetools: Add an ImgdiffStats class that reports imgdiff stats." 2018-02-12 19:01:24 +00:00
Tom Cherry c2ab6232d9 Merge "Place TARGET_FS_CONFIG_GEN passwd/group files in /vendor/etc" am: 3d9fa3615d
am: 57dd54f623

Change-Id: Ica4d341247fc0f770ccdfafd165df05d70f39772
2018-02-12 16:38:43 +00:00
Tom Cherry 3d9fa3615d Merge "Place TARGET_FS_CONFIG_GEN passwd/group files in /vendor/etc" 2018-02-12 16:27:20 +00:00
Tao Bao 667ff57272 releasetools: Specify SWITCH_SLOT_ON_REBOOT for secondary payload.
The secondary payload should always be applied with
SWITCH_SLOT_ON_REBOOT=0. This CL moves the 'secondary' parameter from
Payload.WriteToZip() to Payload.__init__(). So it can append the flag to
secondary/payload_properties.txt.

Bug: 35724498
Test: Generate an A/B OTA with --include_secondary. Check
      secondary/payload_properties.txt entry in the generated ZIP.
Test: `python -m unittest test_ota_from_target_files`
Change-Id: I816c07ab57a1c8a52eff785801634b8b1cb134d4
2018-02-10 00:13:07 -08:00
Tao Bao 294651d7b4 releasetools: Add an ImgdiffStats class that reports imgdiff stats.
We have a couple of active imgdiff workarounds (and likely with one more
inbounding that allows having shared blocks in ext4 image). Most of
these workarounds need extending imgdiff's capability. While us not
getting there anytime soon, collect the stats to better understand the
impact of each kind so we can prioritize accordingly.

A sample report is as follows.

  Imgdiff Stats Report
========================

 APK files diff'd with imgdiff (count: 88)
-------------------------------------------

  /system/priv-app/Shell/Shell.apk
  ...

 Large APK files split and diff'd with imgdiff (count: 4)
----------------------------------------------------------

  /system/priv-app/Settings/Settings.apk
  ...

Bug: 68016761
Test: Generate an incremental BBOTA package. Check the stats report.
Test: python -m unittest test_blockimgdiff
Change-Id: I27ad862cde472ab2806db877632ce5a0607420f2
2018-02-09 23:02:32 -08:00
Tao Bao 194e49e06c Merge "releasetools: Refactor the condition checking for using imgdiff." am: bec8be51a5
am: 4a79eb0d4e

Change-Id: Ia050150657bb373e92bf9797764ef987493e9c96
2018-02-10 02:16:54 +00:00
Tao Bao bec8be51a5 Merge "releasetools: Refactor the condition checking for using imgdiff." 2018-02-10 00:47:42 +00:00
Tao Bao cb73aed1f0 releasetools: Refactor the condition checking for using imgdiff.
In Transfer class, unbundle 'intact' with the monotonicity of the input
ranges. Negate the logic of 'intact', and thus rename it to 'trimmed'.
Move this property from an attribute of Transfer class as the one in
RangeSet.extra. 'trimmed' indicates whether the source / target ranges
have been modified after creating the Transfer() instance.

The logic that determines whether we can apply imgdiff has been
refactored and consolidated into BlockImageDiff.CanUseImgdiff(). Now
both of the two paths call this single copy, i.e. the one that detects
large APKs (before creating Transfer()'s), and the other that's about to
generate the patch for a given Transfer instance.

Bug: 68016761
Test: python -m unittest test_blockimgdiff
Test: Generate an incremental BBOTA package.
Change-Id: Id07195f63f1fa6c3af6e9091940d251cf09fa104
2018-02-09 12:46:01 -08:00