Commit Graph

4657 Commits

Author SHA1 Message Date
Tao Bao 76f038a63c Merge "releasetools: Use pattern filter when unzipping." 2017-03-05 16:49:21 +00:00
Tao Bao aa3a04f19d releasetools: Use pattern filter when unzipping.
When building BBOTAs, it only needs *some* unzipped entries in the given
target_files zip(s). In particular, it needs 'IMAGES/*', 'META/*',
'RADIO/*'. (It also reads 'SYSTEM/build.prop' and 'OTA/bin/updater', but
directly from the zip file.)

This CL specifies the entries to unzip. It saves the I/O cost, as well as
the temporary storage.

Test: ota_from_target_files.py gives the same package w/ and w/o the CL.
Test: check_target_files_signatures.py still works.
Change-Id: I728428aa0e138879e49f9efbdb46a85892fc7038
2017-03-03 16:38:17 -08:00
Sen Jiang d1f397e8b4 Remove DBus support.
external/dbus and external/dbus-binding-generator was removed from
the manifest.

Bug: 31602715
Test: make checkbuild
Change-Id: Iea0277720acad8ac0fa630b8745f90fb3c3b5f00
2017-03-02 18:00:21 -08:00
Tianjie Xu d9880a0246 Merge "Do not read the block device when generating the care map" 2017-03-02 21:36:12 +00:00
Treehugger Robot afc0ea14d8 Merge "releasetools: Drop the support for fstab_version 1." 2017-03-02 19:19:09 +00:00
Tianjie Xu 955629b4aa Do not read the block device when generating the care map
This remove the fstab dependency when building the OTA package for
marlin/sailfish.

Bug: 35811655
Test: OTA package builds successfully for sailfish.
Change-Id: If223d11dddca396c47262042c576f9e7d0cb5b33
(cherry picked from commit 7d051adc3b)
2017-03-02 11:07:12 -08:00
Tao Bao d1de6f326e releasetools: Drop the support for fstab_version 1.
fstab_version is defined by RECOVERY_FSTAB_VERSION in
bootable/recovery. We have moved to fstab_version 2 since commit
f35d1cef7c19db975a1295e8c23c7fb8bd2489f9 (landed into JB MR2).

Drop the support for fstab_version 1, since we won't run the latest OTA
script over a JB target_files zip.

Test: No impact on building full/incremental OTAs.
Change-Id: Ia87c4e7da6c5e71ce0908fca2e4f1ad1c06ba869
2017-03-01 16:56:45 -08:00
Tao Bao e889893776 Merge "releasetools: Add support for --override_timestamp." 2017-03-02 00:00:59 +00:00
Tao Bao 9beea2a4a3 releasetools: Merge two identical conditional branches.
The merged two branches have become identical since commit
fc3422ad36 (landed into Nougat).

Test: Get identical incremental packages w/ and w/o the CL.
Change-Id: Id1183f8ed83f684a0dac1a4af87b6e075b08aabc
2017-02-28 20:20:27 -08:00
Dan Willemsen 93de77745c Merge changes I5e684409,I4fa35540,I24015ef0
* changes:
  Improve module tags warnings
  Improve warnings for package-modules.mk
  Fix all_named_products, remove kati_all_products
2017-02-28 21:30:55 +00:00
Tao Bao 3e6161a3b3 releasetools: Add support for --override_timestamp.
We use the timestamps in builds to determine a downgrade, which might
not be always the truth. For examples, two builds cut from different
branches may carry timestamps in a reverse order. An incremental package
won't be able to be pushed nor applied, based on the timestamp
comparison.

We used to handle such a case with manual work, by setting the
post-timestamp to (pre-timestamp + 1) in the package metadata. This CL
automates the process by adding a new flag --override_timestamp.

Note that it doesn't change anything in the installed image, but only
affects the assertions for pushing / installing the package.

With the change in this CL:
 - If it's a downgrade without any extra flag, fail the package
   generation (we only print warnings prior to this CL);
 - If it's a downgrade with --downgrade flag, generate a downgrade
   package with forced data wipe (same as before);
 - If it's a downgrade with --override_timestamp, generate a normal
   incremental with hacked timestamp (pre-timestamp + 1) (new in this CL
   to avoid the manual change);
 - If it's not a downgrade but with any of the above two flags specified,
   fail the package generation.

Bug: 33744169
Test: Generate an incremental from builds with reversed timestamps.
Change-Id: I8b187d32708b4a7c3e20f8c6adb8f9527b73b965
2017-02-28 13:01:13 -08:00
Treehugger Robot dda91dbef8 Merge "There's no longer a limit on property names." 2017-02-28 20:13:14 +00:00
Elliott Hughes 05c1a2a55c There's no longer a limit on property names.
Bug: http://b/33926793
Test: boots
Change-Id: I12cdae782090fb0856171e5c90a268e91ba2ae1a
2017-02-28 10:04:23 -08:00
Tao Bao 6cd54739a4 Remove the backwards compatibility support for misc_info.txt.
We introduced META/misc_info.txt to hold the misc info since Gingerbread
(commit 37974731fc). Remove the backwards
compatibility support for building pre-G TF zips.

Test: `m dist` works.
Change-Id: Ibff7aaf69cc7e460634c049d11a004f7196f8f73
2017-02-27 17:00:13 -08:00
Treehugger Robot 2af09af86b Merge "releasetools: Fix an equality check bug when asserting stash size." 2017-02-26 21:31:24 +00:00
Tao Bao e8c68a03c8 releasetools: Fix an equality check bug when asserting stash size.
Otherwise the comparison is inconsistent between ReviseStashSize() and
WriteTransfers().

Bug: 35775675
Test: Successfully generate a previously failed incremental.
Change-Id: I554a51a210bf322cb5c79e28cf85607a417b094a
2017-02-26 10:58:22 -08:00
Treehugger Robot d84721a2cc Merge "Add a script to check OWNERS file syntax." 2017-02-25 01:00:18 +00:00
Dan Willemsen 594c3fc3d7 Fix all_named_products, remove kati_all_products
all_named_products was giving the file path instead of product name for
the name:path format of PRODUCT_MAKEFILES.

kati_all_products has been replaced with multiproduct_kati

Test: get_build_var all_named_products
Test: multiproduct_kati
Change-Id: I24015ef0778ac7cd45201aa55c1737b0553f09fe
2017-02-24 15:49:46 -08:00
Chih-Hung Hsieh adaed14ff3 Add a script to check OWNERS file syntax.
* -v option to print found files and email addresses.
* -c option to check found email addresses.

Bug: 33166666
Test: ./checkowners.py -v -c `find . -name OWNERS`
Change-Id: I32f23f19e904055e421ddec713536c8a5c970af4
2017-02-24 15:28:29 -08:00
Dan Willemsen f0e2986cbf Wait for children to exit in makeparallel
This is a partial revert to the last makeparallel change, since now we
were being killed by the signal before waiting for our child to exit. So
instead of not installing the handlers, only pass the signal along if
it's a SIGTERM.

Bug: 35214134
Test: Ensure that we're still only getting one signal for SIGINT
Test: Ctrl-C, ensure that all the soong_ui lines are before the make
      error line.
Change-Id: I26fff9483a3abfd79ceb5a9ea47e3f7572d9e923
2017-02-21 17:53:35 -08:00
Treehugger Robot 70aac88a61 Merge "Prevent duplicate signals from makeparallel" 2017-02-21 23:22:32 +00:00
Dan Willemsen c6a900bee1 Prevent duplicate signals from makeparallel
As part of the soong_ui effort, we noticed that we'd sometimes see
multiple SIGINTs from a single ctrl-c from a user. ctrl-c sends a SIGINT
to the entire process group, so make, makeparallel, soong_ui, and all of
its children would get a signal. Since makeparallel was passing it along
to it's child, soong_ui would get two signals.

So instead, follow what Make does and only pass along SIGTERM. Assume
that all other signals went to the entire process group.

Bug: 35214134
Test: Send SIGINT to process group, check in makeparallel's child for
more than one signal received.
Change-Id: I5b2a77ad0fcebbaa5087439948e71bf3b541061a
2017-02-21 13:50:12 -08:00
Alain Vongsouvanh 7f804ba71f releasetools: allow for multiple OEM property values.
This CL changes the --oem_settings flag to allow a comma seperated list of
property files. All property values will be used when asserting properties such
as ro.product.name.

For example, if two property files are provided with ro.product.name values of
"sprout" and "sprout_a", the resulting otapackage will check that the device's
ro.product.name property matches at least one of them.

Bug: 34191373
Test: manual

Change-Id: I954673511be8f0929982235cc9cbfbd85a9ee1f4
2017-02-21 16:09:34 +00:00
Tao Bao 4f104d1b94 releasetools: Allow using futility-host to sign old TF.zip.
Commit e98fb7a8d3 switched to using
futility-host instead of the prebuilt futility. This CL adds support to
handle signing old TF.zip that still says "futility=prebuilt/..." in
META/misc_info.txt.

Bug: 35467608
Test: Generate otatools.zip and sign an old ryu TF.zip.
Change-Id: I48a9cc918c7afce361e1ec9bc4f85f74fa92566e
2017-02-17 23:27:44 -08:00
Narayan Kamath b46507ff9e zipalign: remove support for adding members from gzip sources.
This was vestigial code and never actually called anywhere.

Test: make
Bug: 35246701
Change-Id: I9297ac5fa030d369946857414155450635b65410
2017-02-16 10:53:09 +00:00
Tao Bao 0ee0fc5898 Merge "releasetools: Fix a bug in ReviseStashSize()." 2017-02-13 22:29:40 +00:00
Tao Bao 3c5a16d49c releasetools: Fix a bug in ReviseStashSize().
We check the needed stash size in ReviseStashSize(), and may not
generate a stash command if it would exceed the max allowed size. This
CL fixes a bug when skipping a stash operation: we shouldn't update the
'stashes' map if a stash command won't be generated.

Bug: 35313668
Test: Successfully generate the package that was failing due to the bug.
Change-Id: If0a3a5fadda9b4a4edad66a2a5826b5f978400ae
2017-02-13 12:31:42 -08:00
Alex Deymo 9a535b5188 fat16copy: Allow to copy files to an existing directory.
This patch re-uses existing directories when copying directories,
allowing to copy files to a directory previously created.

Bug: 34811400
Test: fat16copy.py fat.bin a/b/foo; fat16copy.py fat.bin c/d/foo

(cherry-picked from 0af24a27a8d0dc7e1782f36a5a9963021ff8b3f6)

Change-Id: I283184da18f3c4951d6c84aafa4185074791acd2
2017-02-13 16:00:07 +00:00
Tao Bao b31892e5de releasetools: Support generating downgrade A/B OTAs.
We already support generating downgrade OTAs for non-A/B devices (with
mandatory data wipe), but we have missed the --downgrade flag in A/B OTA
path.

This CL factors out the function that writes the downgrade metadata, and
fixes the path for generating A/B OTAs.

Bug: 35094540
Test: Generate incrementals with --downgrade for A/B and non-A/B OTAs.
Change-Id: I30b9bf83e69e8aba3be666507681b555db6ab743
2017-02-07 12:25:59 -08:00
Tao Bao b09fef9158 Merge "releasetools: Pack the offset/length for metadata." 2017-02-07 04:16:20 +00:00
Tao Bao 82bdf6c1d7 releasetools: Fix the bug when generating care_map.txt.
Commit f1a13180db intended to remove the
verity blocks from care_map.txt, but it added new code without removing
the old one. This leads to a malformed care_map.txt and causes
update_verifier failure.

Bug: 34391662
Test: 'm -j dist' gives a TF.zip with 4-line META/care_map.txt (as
      opposed to a 6-line file).

Change-Id: I7ff1aa525795c4b049af54c1755b0f0ea84f7e0e
2017-02-05 14:31:44 -08:00
Tao Bao 2dd1c48377 releasetools: Pack the offset/length for metadata.
For streaming OTAs, we will also need the info in the metadata entry
(META-INF/com/android/metadata). Compute and pack its offset/length
values into 'ota-streaming-property-files'.

Bug: 34986195
Test: Create an OTA package and check the offset/length values.

Change-Id: Id150700f2bc9bff02467cda9fe8927c8a374412a
2017-02-03 16:50:25 -08:00
Tianjie Xu 3a79c6d784 Merge "Remove verity blocks from care map" 2017-02-03 21:50:07 +00:00
Tao Bao bfdcb1257c releasetools: Prepend 'ota-' to 'streaming-property-files'.
'streaming-property-files' is a property related to the OTA package
itself. Prepend 'ota-' to make it consistent with others like
'ota-type' and 'ota-required-cache'.

Bug: 34852392
Test: Generate an A/B OTA package and check METADATA entry.
Change-Id: Ia681e6e19ff509e6da0d8718933b42aac997e1cf
2017-01-31 15:53:20 -08:00
Tianjie Xu f1a13180db Remove verity blocks from care map
When reading /dev/block/dm-X, update_verifier isn't able to access the
verity meta blocks at the end of the system/vendor partition. So we need
to remove these block ranges from the care_map.

Bug: 34391662
Test: care_map generated successfully without verity meta blocks
Change-Id: Id57c602b7e5fd1b0c9d1e1fe5fcdd74e85b6b255
2017-01-25 23:01:29 -08:00
Tao Bao c96316c89b Revert "Revert "releasetools: Generate streamable A/B OTA packages.""
This reverts commit ea4325baf8 to re-land
commit ef1bb4360f. It fixes the bug when
handling a package without care_map.txt (e.g. dm-verity not enabled).

In order to support streaming A/B OTA packages, we pack
payload_properties.txt and care_map.txt in ZIP_STORED mode. These two
entries along with payload.bin (already in ZIP_STORED prior to this CL)
can be fetched directly based on the offset and length info.

We write the offset and length info into the package metadata entry
(META-INF/com/android/metadata), which can be parsed by the OTA server.

payload_properties.txt and care_map.txt are usually less than 1-KiB. So
the change only incurs marginal size increase.

Bug: 33382114
Test: Generate an A/B OTA package. Verify the 'streaming-property-files'
      entry in the metadata file.
Test: Generate an A/B OTA package on a device with dm-verity not enabled.

Change-Id: I3469c8b62385a1fc58b4fb82e3f9d4690aef52ba
2017-01-24 22:27:02 -08:00
Tao Bao ea4325baf8 Revert "releasetools: Generate streamable A/B OTA packages."
This reverts commit ef1bb4360f.

It has broken the builds that don't have care_map.txt entry.

Change-Id: I343455e66be5e79457968dfc2813c7c1c234b6dc
2017-01-25 01:41:13 +00:00
Tao Bao ef1bb4360f releasetools: Generate streamable A/B OTA packages.
In order to support streaming A/B OTA packages, we pack
payload_properties.txt and care_map.txt in ZIP_STORED mode. These two
entries along with payload.bin (already in ZIP_STORED prior to this CL)
can be fetched directly based on the offset and length info.

We write the offset and length info into the package metadata entry
(META-INF/com/android/metadata), which can be parsed by the OTA server.

payload_properties.txt and care_map.txt are usually less than 1-KiB. So
the change only incurs marginal size increase.

Bug: 33382114
Test: Generate an A/B OTA package. Verify the 'streaming-property-files'
      entry in the metadata file.

Change-Id: I04504e834eb36e18876c5f5a5a09289ee05c6f9a
2017-01-19 22:25:12 -08:00
Colin Cross 524c868107 Don't pass empty arguments to ninja
Passing an empty argument in argv to ninja results in an error:
ninja: error: empty path

Don't add jarg to the argument list if it is empty.

Bug: 34392351
Test: m -j
Change-Id: I39c8998c79b09835650628e1413ae7625cdbc555
2017-01-18 21:12:28 -08:00
Colin Cross ba55c623cb Convert -j to a reasonable parallelism for kati
When running makeparallel in non-ninja mode, which is used when running
kati with USE_GOMA=true, convert -j to a reasonable parallelism value
the same way ninja does.

Bug: 34392351
Test: make -C build/make/tools/makeparallel makeparallel_test
Change-Id: I9aee4dd2a3b0f2b1c2c10087be83f7b2b06f4368
2017-01-18 14:50:50 -08:00
Connor O'Brien cf4f7eac85 Merge "Pass flash erase & logical block size to mke2fs" 2017-01-17 20:07:11 +00:00
Tao Bao 4a0b494eea Merge "releasetools: Replace print stmt with print()." 2017-01-14 15:36:41 +00:00
Tao Bao 89fbb0f6d5 releasetools: Replace print stmt with print().
So that it's compatible with Python 3.

Test: pylint --pylint=pylintrc

Change-Id: If06c135a492c94bedd713c8cbdf03155a502d5cd
2017-01-13 14:55:14 -08:00
Tao Bao 2a0d1da4d9 releasetools: Delete the obsolete --aslr_mode/-a flag.
It was added in commit 96be7205dc
("Working ASLR implementation.") in 2010, and removed in commit
1807e700a5 ("don't generate retouch
commands in OTA scripts") in 2012.

Remove the obsolete --aslr_mode flag.

Test: ota_from_target_files.py still works (by generating incremental
      and full OTAs respectively).

Change-Id: I6d8e62730ac192f3574d484c4a4b9b43b4ee0a9e
2017-01-13 22:46:27 +00:00
Connor O'Brien 20f08c3e16 Pass flash erase & logical block size to mke2fs
This information can be used to tune ext4 stripe and stride in the
userdata partition for better performance

Test: Build & flash userdata, confirm correct stripe & stride values
Bug: 33243520
Merged-In: Ia97cdd2d0239c3484b895fce49299f692ef911d8
Change-Id: Ia97cdd2d0239c3484b895fce49299f692ef911d8
Signed-off-by: Connor O'Brien <connoro@google.com>
2017-01-13 12:17:05 -08:00
William Roberts 4165c635b8 fsconfig: fix pylint warning
Fix pylint warning:
W:292,47: Cell variable lineno defined in loop (cell-var-from-loop)

For the closure function that prints the error message, we actually
want the value of modified lineno within the closure.

Test: run unit tests

Change-Id: I7d08184563d347c826ca1af7558fb8e378f151e1
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2017-01-04 18:22:05 +00:00
William Roberts 820421c9b5 fsconfig: fix style formating with yapf
A few changes got introduced that yapf re-styles. To make it easy,
keep this file formated to:

yapf -i --style=google fs_config_generator.py

Test: run unit tests

Change-Id: I3160b92d7fd07a4d315dde59f1d20fe4c7587da1
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2017-01-04 18:21:28 +00:00
Elliott Hughes ce6c8e629f Merge "fs_config: Fix oem range parsing" 2017-01-04 00:37:58 +00:00
Johan Redestig 1552a28e5f fs_config: Fix oem range parsing
Addresses the following build error:
  "value" not in valid range [], got: 2901 for: "AID_ABC" file:\
  "device/somc/abc/config.fs"

that happens when parsing device/somc/abc/config.fs:
  [AID_ABC]
  value: 2901

The problem is that the AID.PREFIX is 'AID_' so the _OEM_RANGE
expression is searching for AID__OEM_RESERVED, which never
exists. Drop the extra '_'.

Change-Id: I8a1556731e14e49cef66e3e91121e4be23a308e0
Test: ./build/tools/fs_config/fs_config_generator.py fsconfig --aid-header=system/core/include/private/android_filesystem_config.h device/somc/abc/config.fs
2017-01-03 09:45:33 +01:00
Tao Bao 3a2e350239 blockimgdiff.py: Clean up stash id computation in BBOTA v3+.
Only BBOTA v2 needs to maintain a pool of available 'stash slot id'.
BBOTA v3+ uses the hash of the stashed blocks as the slot id, which
doesn't need the id pool anymore.

Bug: 33694544
Test: Generate v2 and v4 incrementals w/ and w/o the CL. They produce
      the same packages respectively.
Change-Id: I8121af5b6b1bee98c3639d54a00b06fd12e378e8
2016-12-28 09:26:46 -08:00
Elliott Hughes 5f60b74297 Merge "Revert "Revert "Merge changes from topic 'fsconfig-2'""" 2016-12-27 20:51:07 +00:00
Tao Bao 304ee27e88 blockimgdiff.py: Add a test case for ReviseStashSize().
Bug: 33687949
Test: python -m unittest test_blockimgdiff
Change-Id: I31e42648d2c0e60b941908c098ce5ccc967861ca
2016-12-19 11:17:12 -08:00
Elliott Hughes 2d7c86dfea Revert "Revert "Merge changes from topic 'fsconfig-2'""
This reverts commit fad4b4b715.

Incorporating the following fixes:

1.

fsconfig: fix fs_config_* build for discovered headers

When android_file system_config.h is picked up from the device
directory, neither TARGET_FS_CONFIG_GEN or TARGET_ANDROID_FILESYSTEM_CONFIG_H
are specified. Thus, the build is not generating the required fs_config_files
and fs_config_dirs.

Test: Ensure that make fs_config_files works, and produces the same output as before
      Build the system image and mount it as a loop back and dump the file system
      capabilities with getcap. Verify that output to the supplied
      android_file system_config.h

From the loopback of the system.img mount, from CWD system/bin:
$ getcap *
cnss-daemon = cap_net_bind_service+ep
hostapd = cap_net_admin,cap_net_raw+ep
imsdatadaemon = cap_net_bind_service+ep
ims_rtp_daemon = cap_net_bind_service+ep
logd = cap_setgid,cap_audit_control,cap_syslog+ep
mm-qcamera-daemon = cap_sys_nice+ep
pm-service = cap_net_bind_service+ep
run-as = cap_setgid,cap_setuid+ep
surfaceflinger = cap_sys_nice+ep
webview_zygote32 = cap_setgid,cap_setuid,cap_setpcap+ep
webview_zygote64 = cap_setgid,cap_setuid,cap_setpcap+ep

Compared to the android_filesystem_config.h:
    { 00700, AID_CAMERA,    AID_SHELL,     (1ULL << CAP_SYS_NICE), "system/bin/mm-qcamera-daemon" },
    { 00755, AID_SYSTEM,    AID_SYSTEM,    (1ULL << CAP_NET_BIND_SERVICE), "system/bin/pm-service" },
    { 00755, AID_SYSTEM,    AID_SYSTEM,    (1ULL << CAP_NET_BIND_SERVICE), "system/bin/imsdatadaemon" },
    { 00755, AID_SYSTEM,    AID_RADIO,     (1ULL << CAP_NET_BIND_SERVICE), "system/bin/ims_rtp_daemon" },
    { 00755, AID_SYSTEM,    AID_SYSTEM,    (1ULL << CAP_NET_BIND_SERVICE), "system/bin/cnss-daemon"},

2.

fsconfig: fix error message for duplicate AID

Fixes:
    raise ValueError('Duplicate aid value "%u" for %s' % value,
TypeError: %u format: a number is required, not str

and

    raise ValueError('Duplicate aid value "%s" for %s' % value,
TypeError: not enough arguments for format string

3.

fsconfig: add test for duplicate ranges

Add a test for duplicate range detection.

4.

fsconfig: skip AID_APP, AID_USER and all ranges

Do not output AID_APP, AID_USER and ranges. A range
is defined as ending with AID_ and ending in _START or
_END.

5.

fsconfig: test for skip AID_APP, AID_USER and all ranges

Test against AIDs that caused the bionic tests to fail.

Change-Id: I95569a9ccc83bd3231f8a6f395532cc2de316bd2
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2016-12-18 10:55:35 -08:00
Tao Bao 9cc98767ad Merge "releasetools: Fix the computation in ReviseStashSize()." 2016-12-16 23:57:40 +00:00
Tao Bao e27acfdc41 releasetools: Fix the computation in ReviseStashSize().
We compute the max stashed_blocks in ReviseStashSize(), prior to calling
WriteTransfers(), to avoid running out of space due to stashing.

There is a bug when computing the to-be-freed stashed blocks, where we
wrongly free the space _before_ executing the transfer command. This leads
to a script failure where the max stash size violates the max allowed
size in WriteTransfers().

Note that this bug doesn't affect already generated packages. It's only
an underestimate in ReviseStashSize(). The check in WriteTransfers() has
been correct to ensure the max stash size.

Bug: 33687949
Test: Successfully generated incremental OTA which failed previously.
Change-Id: I4f4f043c6f521fce81ca5286e6156f22d99bf7f7
2016-12-16 11:26:05 -08:00
Alex Klyubin 620686f3c6 Merge "Faster auto-detection of APK's minSdkVersion" 2016-12-14 23:42:13 +00:00
Alex Klyubin 9b75e272b6 Faster auto-detection of APK's minSdkVersion
Prior to this change, when signing APKs, the build system invoked
'aapt dump badging' on each APK, to detect the value to pass into
signapk as --min-sdk-version. Now that signapk uses the apksig
library, it can auto-detect that value on its own, thus avoiding the
need to invoke 'aapt dump badging' and thus speeding up the build
process.

The semantics of signapk's --min-sdk-version flag is changed by this
commit from having the default value of 0 to having the default value
of "auto-detect from APK".

P.S. The get-package-min-sdk-version-int is not removed from
core/definitions.mk in this commnit, because this function is used in
another project's .mk file and thus that .mk file needs to be modified
first.

Test: rm -Rf out/ && make
Change-Id: I0972fcf0abbde9cbf6794e6c05c743c77c8a78f9
2016-12-14 12:53:44 -08:00
Treehugger Robot a282ceeb73 Merge "releasetools: Add prefix when dumping fingerprints." 2016-12-14 20:53:02 +00:00
Tao Bao f9023856ad releasetools: Add prefix when dumping fingerprints.
We used to dump "Source: <fingerprint>" in update logs. The "Source: "
prefix was unintentionally dropped out.

Test: Check the generated incremental BBOTA script.
Change-Id: I4de62333aa38e3fb09a76df0e769b62af48e0313
2016-12-14 11:53:38 -08:00
Elliott Hughes fad4b4b715 Revert "Merge changes from topic 'fsconfig-2'"
This reverts commit 1b2de51881, reversing
changes made to 77ea32f378.
2016-12-12 17:28:44 -08:00
Elliott Hughes 1b2de51881 Merge changes from topic 'fsconfig-2'
* changes:
  fs_config: add unit tests
  fs_config: drop fs_config_files/dirs PRODUCT_PACKAGES requirement
  fs_config: add group to build
  fs_config: introduce group generator
  fs_config: add passwd to build
  fs_config: introduce passwd generator
  fs_config: generate friendly in AID class
  fs_config: limit characters for AID_<name> sections
  fs_config: generate oem AID header file
  fs_config: android_id header generator
  fs_config: support parsing android_filesystem_config.h
  fs_config: modularize fs_config_generator
2016-12-10 00:13:05 +00:00
Patrick Tjin a1900841bb build_image: add support for specifying number of inodes
Add support for specifying number of inodes when creating
system, vendor, oem partitions.  These are all read-only
and have no use for extra inodes.  Removing extra inodes
saves a lot of space.

Bug: 32246383
Change-Id: I13f1d4614b64a4abc752c42a1c65d3d151481c21
(cherry picked from commit b59eca3586)
2016-12-06 21:07:23 -08:00
The Android Automerger ada51f3b8f Manually merge commit 'c13da91' into stage-aosp-master
* commit 'c13da91':
  Add support for creating ext4 images with mke2fs

BUG:33304034
2016-12-03 04:02:09 +00:00
Treehugger Robot c13da91023 Merge "Add support for creating ext4 images with mke2fs" 2016-12-02 22:48:23 +00:00
Tao Bao d07f9995b8 Merge "Build recovery-two-step.img for two-step OTAs."
am: 06e14e438f

Change-Id: I9c5d6e5f71da88bc0f662373335b024a55ca193e
2016-12-02 15:19:00 +00:00
Tao Bao d42e97ebb4 Build recovery-two-step.img for two-step OTAs.
In two-step OTAs, we write recovery image to /boot as the first step so
that we can reboot from there and install a new recovery image to
/recovery. However, bootloader will show "Your device is corrupt"
message when booting /boot with the recovery image. Because the recovery
image encodes the path of "/recovery" as part of the signature metadata,
which fails the verified boot.

This CL generates a special "recovery-two-step.img" in addition to the
regular recovery.img. This image encodes "/boot" when being signed,
which will be flashed to /boot at stage 1/3 in a two-step OTA.

Here are the desired changes:

- 'IMAGES/recovery-two-step.img' exists in target_files.zip for non-A/B
targets (e.g. bullhead). The image should not exist for targets that
don't have a recovery partition (e.g. A/B devices like sailfish).

- <device>-img.zip should not contain 'recovery-two-step.img'.

- Nothing should change when building non-two-step OTAs. For two-step
OTAs, 'recovery-two-step.img' should be included in the OTA package;
'updater-script' should flash this image to /boot at stage 1/3.

- When building a two-step OTA with an input TF.zip that doesn't have
  IMAGES/recovery-two-step.img, it should use the existing
  IMAGES/recovery.img instead.

Bug: 32986477
Test: Tested the steps above on bullhead and sailfish.
Change-Id: I34e6c599bcf2011d4cd5c926999418b3975d6d0f
2016-12-01 17:47:59 -08:00
Tao Bao 2e3f1c5258 Merge "releasetools: Remove dead parameter in GetBootableImage()."
am: 8df4fea85f

Change-Id: I192af3927849d0d8f995f0002b8b8f2e191b794f
2016-12-01 17:09:07 +00:00
Tao Bao 2a4336bc04 releasetools: Remove dead parameter in GetBootableImage().
The 'system_img_path' parameter was introduced in commit
d995f4b04d, but became obsolete since
commit 2ce63edab7.

Test: m dist
Change-Id: Iffd496d929db5cc3dfc955a48bfc1b1317bd012f
2016-12-01 00:08:27 -08:00
William Roberts 92ec0ab5e8 fs_config: add unit tests
Add some initial unit tests.

The unit tests themselves are inlined into the tool and
can be executed by running the "test" commandlet.

Example:
$ python -m unittest test_fs_config_generator.Tests
.............
----------------------------------------------------------------------
Ran 13 tests in 0.004s

OK

Test: run the test commandlet and observe for failures.

Change-Id: I1bada385fa841fd50fa958997d440f1198e15198
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2016-11-29 22:05:39 -08:00
William Roberts 7fe8b6d817 fs_config: drop fs_config_files/dirs PRODUCT_PACKAGES requirement
When configuring fs_config_files or fs_config_dirs for file_system
capabilities, drop the requirement that OEMs must add the target
to PRODUCT_PACKAGES. This limits the configuration requirement
to only needing to set the new and preferred TARGET_FS_CONFIG_GEN
or the older TARGET_ANDROID_FILESYSTEM_CONFIG_H method.

Test: That only setting TARGET_FS_CONFIG_GEN results in passwd and
group in the build image.
Change-Id: I818854fa1b3e94edaff59a32bd7cf23cf9b504aa
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2016-11-29 16:21:17 -08:00
William Roberts 1c4721c3c5 fs_config: introduce group generator
Introduce a generator that outputs group files per man(5) group.

Succinctly, the output is a colon delimited string containing the following
fields:
  * group name
  * encrypted password (optional)
  * gid (int)
  * userlist (str,...)

Multiple colon delimited lines may exist, but will not be separated
across lines.

Sample generator output:
foo::2900:
foo_bar::2901:
custom_oem1::2902:

Test: That make group produces the group file.
Change-Id: Idd3fe925a09a227c6e894e1b5d2b3873b01531c6
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2016-11-29 16:21:17 -08:00
William Roberts 316f9462af fs_config: introduce passwd generator
Introduce a generator that outputs passwd files per man(5) passwd.

Succinctly, the output is a colon delimited string containing the following
fields:

  * login name
  * encrypted password (optional)
  * uid (int)
  * gid (int)
  * User name or comment field
  * home directory
  * interpreter (optional)

Multiple colon delimited lines may exist, but will not be separated
across lines.

When run, produces:

foo::2900:2900::/:/system/bin/sh
foo_bar::2901:2901::/:/system/bin/sh
custom_oem1::2902:2902::/:/system/bin/sh

Note that this generator allows for 0 or more config.fs files. This allows for:
  * Unconditional inclusion of /system/etc/passwd in the generated image
  * A blank passwd file if no config.fs files are specified.

This ensures that when OEMs add config.fs files, there is no additional steps
for proper functionality (simpler for OEMs).

The one draw back is the additional inode consumption on system for a possible
blank file.

Test: That it produces a valid passwd file.
Change-Id: I19691c8260f02147ed861f8a319aeab3f5b1738e
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2016-11-29 16:21:17 -08:00
William Roberts 8f42ce71ac fs_config: generate friendly in AID class
Generate the friendly name in one location. This prepares
the tool for generatting passwd and group files.

Also support mapping friendly names to identifiers.

Test: That output files stay the same as before.
Change-Id: I12198611126613eae81ca61614ed269c2439b72b
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2016-11-29 16:21:17 -08:00
William Roberts 5f059a7691 fs_config: limit characters for AID_<name> sections
When generating files restrict the characters allowed in
AID_<name> to upercase, numbers, and underscores.

This detects errors ahead of time for generated C files as
well as handles seperation characters for passwd/group files.

This also lends itself to automatic detection of collisions
on friendly names, since freindly names are the lowercase
version of <name>.

Test: That invalid aid values result in a build failure.
Test: That the output files are consistent with ones before
this change (hash and diff checks)
Change-Id: Ie8ec44c1157ba9c22100e9169d9187f615e71280
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2016-11-29 16:21:17 -08:00
William Roberts cfc51f5347 fs_config: generate oem AID header file
Generate an OEM AID_<name> header file seperate from fs_config
header file and provide details on how to export this interface
into native code.

Test: That ls, ps, chown and services function for built in
services as before.
Change-Id: Ie8ce6585e0721b52633ee50d62dcfe796e178f65
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2016-11-29 16:21:17 -08:00
William Roberts d7104bca65 fs_config: android_id header generator
Implement an android_id generator that takes the data
acquired from parsing private/android_filesystem_config.h
and generates the android_id friendly name to uid mapping
for consumption in Bionic.

Test: That ls, ps, mkdir, chown, chgrp and services for built
in names work.
Change-Id: I1e55a401be0fca0ad162f8dc1e072e6afde7b927
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2016-11-29 16:21:17 -08:00
William Roberts 64edf5bb97 fs_config: support parsing android_filesystem_config.h
Rather than hardcode the OEM ranges, parse and extract
AID values from android_filesystem_config.h.

An AID is defined to the tool as:
  * #define AID_<name>

An OEM Range is defined to the the tool as:
  * AID_OEM_RESERVED_START
  * AID_OEM_RESERVED_END
  or
  * AID_OEM_RESERVED_N_START
  * AID_OEM_RESERVED_N_END

Where N is a number.

While parsing, perform sanity checks such as:
1. AIDs defined in the header cannot be within OEM range
2. OEM Ranges must be valid:
   * Cannot overlap one another.
   * Range START must be less than range END
3. Like the C preproccessor, multiple matching AID_<name> throws
   en error.

The parser introduced here, prepares the tool to output android_ids
consumable for bionic.

Note that some AID_* friendly names were not consistent, thus a small
fixup map had to be placed inside the tool.

Test: tested parsing and dumping the data from android_filesystem_config.h
file.
Change-Id: Ifa4d1c9565d061b60542296fe33c8eba31649e62
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2016-11-29 16:21:17 -08:00
William Roberts 11c29283ec fs_config: modularize fs_config_generator
This internally structures fs_config_generator.py to be able
to plug in generators to produce different outputs. This
prepares this tool for group and pwd file outputs.

Test: Checked diff and hash of before and after files.
Change-Id: Ie558518ac227dd946d70ab48027698b72a9bc94a
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2016-11-29 16:21:17 -08:00
Ian Rogers 1eef968704 Merge "Update warn.py for new Error Prone diagnostics"
am: 7ed3af23a8

Change-Id: Iae1dec6c0b48be23d874c01b9d87627bbf0c0caf
2016-11-22 18:50:21 +00:00
Nick Glorioso 8a7b25c736 Update warn.py for new Error Prone diagnostics
Test: Local build, tee'd to file, ran warn.py and checked output HTML
Change-Id: Iee3386e88c73e4b8e0a1fcc60105ee5fb86e3cb0
2016-11-22 10:35:07 -08:00
Adrien Schildknecht 9a072cc0b0 Add support for creating ext4 images with mke2fs
We are investigating replacing make_ext4fs with the upstream tool mke2fs.
To mitigate the trouble that may arise if the new tool behave differently
compared to the old one, there will be a transition period.

Devices that want to use the new way of creating ext4 images can set the
variable "TARGET_USES_MKE2FS" to true in their BoardConfig.mk
By default, the build system will choose the old tool 'make_ext4fs'.

Test: m otapackage with TARGET_USES_MKE2FS={,false,true}
Change-Id: I282bcb9efe335a86c53986283090ca947d65c7f8
2016-11-18 17:06:29 -08:00
William Roberts c06559eb7b Merge "fs_config: move sort to be consistent with other sort"
am: 052a01958c

Change-Id: If18ab459d74e20f96db56d80ecdbf94580583463
2016-11-18 00:57:15 +00:00
Treehugger Robot 052a01958c Merge "fs_config: move sort to be consistent with other sort" 2016-11-18 00:29:53 +00:00
William Roberts 2088e57ed8 Merge "fs_config: correct README"
am: 4baaf905ee

Change-Id: I1a3fa8a296608205a29efb94c32fbfd14d4844f2
2016-11-17 23:42:52 +00:00
William Roberts 8cb6a1893e fs_config: move sort to be consistent with other sort
Test: Checked diff and hash of before and after output files, same.
Test: Tested build for basic boot and functionality.
Change-Id: If7806427e3a2a9ddb7a2c9aa14e1e4f9bf696acf
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2016-11-17 14:18:38 -08:00
William Roberts db616f0091 fs_config: correct README
Testing on earlier versions of Python indicated an error in the
README. Python's ConfigParser prior to version 3.2 allows for
sections within the same file to override later sections.
Correct the README.

Test: Checked diff and hash of output file from before and after.
Change-Id: Ifd3f78414b4803e4b74decb1fdf6816d4902c4a2
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2016-11-17 13:10:55 -08:00
Chih-hung Hsieh 70bbbdfdb1 Merge "Separate parallel_classify_warnings out of parse_input_file."
am: e4fb495b09

Change-Id: I5353f10ea0d78d8d9cd4539e4095b9c0ff010520
2016-11-15 22:34:03 +00:00
Chih-Hung Hsieh 76d0065ef8 Separate parallel_classify_warnings out of parse_input_file.
* Allow other callers of this module to:
  * pass any input stream to parse_input_file,
  * pass any warning_lines to parallel_classify_warnings,
  * call dump_csv or dump_html to get output.
* No output change.
* Capture and ignore signal.SIGTERM at the end,
  to avoid bad warning/error messages from the exit clean-up process.

Test: run warn.py build.log.
Change-Id: I1414797a536c0ee622e2a34c226578621be1ddab
2016-11-09 18:19:05 -08:00
Clay Murphy 731df4f4f3 Merge "Creates a css nowrap class for non-breaking elements in source.android.com Test: build2stage 36"
am: 98f15a70e9

Change-Id: I19f35481e688b205790e4f2d817932422b72b93f
2016-11-09 21:29:57 +00:00
Billy Lamberta 4d41e26a54 Creates a css nowrap class for non-breaking elements
in source.android.com
Test: build2stage 36

Change-Id: I467cc66d5e205b00ed188180140b1ae972e2ebad
2016-11-09 21:17:41 +00:00
Tao Bao dc04f93e39 Merge "releasetools: Use java_path in sign_target_files_apks.py."
am: 4a71d500c1

Change-Id: I61400574ac3f7d29906ca571f497554631ecd72d
2016-11-08 23:42:26 +00:00
Tao Bao e95540e060 releasetools: Use java_path in sign_target_files_apks.py.
Prior to this CL, it was calling the hard-coded "java" although it was
accepting a "--java_path" option.

Also switch OPTIONS.java_args from string to list. Otherwise it won't
work when providing multiple args.

Bug: 32737832
Test: Specify "--java_path=" and "--java_args" when invoking
      sign_target_files_apks.py with "-v". Check the commands being
      called.
Change-Id: Id7ef98e778646d532027434de7fba9b7a104dbd0
2016-11-08 12:48:43 -08:00
Tao Bao 9242e1f750 Merge "releasetools: compare size between compressed and patch while build otapackage"
am: 49758959f2

Change-Id: I17b988878cc7ffdfbe490f404c9cd5ab996a8c5f
2016-11-05 00:13:55 +00:00
Tao Bao 49758959f2 Merge "releasetools: compare size between compressed and patch while build otapackage" 2016-11-05 00:09:35 +00:00
Tao Bao e3554167e2 Merge "releasetools: Make BBOTA incremental generation repeatable."
am: b9bd78d110

Change-Id: I6402481bf300b52d36df1570cceeae6ec3ff4f8f
2016-11-04 21:29:15 +00:00
Tao Bao b9bd78d110 Merge "releasetools: Make BBOTA incremental generation repeatable." 2016-11-04 21:26:13 +00:00
Colin Cross 7227637308 findleaves.py: prevent recursion into symlink loops
am: 7829ebfebd

Change-Id: Ib0ff185e2798ca57c6c28157a1182e7b4d1ed820
2016-11-04 18:17:35 +00:00
Colin Cross 7829ebfebd findleaves.py: prevent recursion into symlink loops
Keep a set of all visited inodes, and prevent recursing into a symlink
to an already visited inode.

Test: m -j
Test: compare `build/tools/findleaves.py --prune=.repo --prune=.git --mindepth=2 --dir=. Android.mk` before and after
Change-Id: Ied14c40d3066ef9f8e8a2b1535f56f7bbbbd0ab6
2016-11-03 14:29:40 -07:00
Chih-hung Hsieh 85465a39fb Merge "Do not use multiprocessing.Pool when --processes=1."
am: 948ebf5510

Change-Id: Ia305f1c26706069d8e048340eac9037782f895b7
2016-10-28 20:44:35 +00:00
Chih-Hung Hsieh 63de300094 Do not use multiprocessing.Pool when --processes=1.
Purpose of this change is not to save some execution time,
but to avoid forking another process. In other applications
that use a wrapper to call this python script, it is difficult
to get overwritten file I/O functions work in a subprocess.
So the wrapper will call warn.py with --processes=1.

Test: run "warn.py --processes=1 build.log"
Change-Id: I5998d5c70d81a456c86eb4002f444a4a60135477
2016-10-28 10:53:34 -07:00