Commit Graph

4724 Commits

Author SHA1 Message Date
Tao Bao abda7dcc8c Merge "releasetools: Generate streamable A/B OTA packages." am: 41130f495f am: 62bc5301ec am: ac5a36e56b
am: 56677596ff

Change-Id: Icfc372710ad90be02d1590149e079675e7f16774
2017-01-25 00:04:24 +00:00
Tao Bao ef1bb4360f releasetools: Generate streamable A/B OTA packages.
In order to support streaming A/B OTA packages, we pack
payload_properties.txt and care_map.txt in ZIP_STORED mode. These two
entries along with payload.bin (already in ZIP_STORED prior to this CL)
can be fetched directly based on the offset and length info.

We write the offset and length info into the package metadata entry
(META-INF/com/android/metadata), which can be parsed by the OTA server.

payload_properties.txt and care_map.txt are usually less than 1-KiB. So
the change only incurs marginal size increase.

Bug: 33382114
Test: Generate an A/B OTA package. Verify the 'streaming-property-files'
      entry in the metadata file.

Change-Id: I04504e834eb36e18876c5f5a5a09289ee05c6f9a
2017-01-19 22:25:12 -08:00
Colin Cross 366902561d Merge "Don't pass empty arguments to ninja" am: f8acdc50fa am: d4aba70550 am: ac15f41714
am: 239c78a5b3

Change-Id: Icbd643898cc9babb09090525aa7f0961cf9a04c9
2017-01-19 06:56:22 +00:00
Colin Cross 524c868107 Don't pass empty arguments to ninja
Passing an empty argument in argv to ninja results in an error:
ninja: error: empty path

Don't add jarg to the argument list if it is empty.

Bug: 34392351
Test: m -j
Change-Id: I39c8998c79b09835650628e1413ae7625cdbc555
2017-01-18 21:12:28 -08:00
Colin Cross 95b4b4ad65 Merge "Convert -j to a reasonable parallelism for kati" am: bb5c13d359 am: 57de869060 am: 8b8c5d98f1
am: 054fe8dc8a

Change-Id: I7cb978ed0c7844c41793e80f581da287abccedc9
2017-01-19 00:48:43 +00:00
Colin Cross ba55c623cb Convert -j to a reasonable parallelism for kati
When running makeparallel in non-ninja mode, which is used when running
kati with USE_GOMA=true, convert -j to a reasonable parallelism value
the same way ninja does.

Bug: 34392351
Test: make -C build/make/tools/makeparallel makeparallel_test
Change-Id: I9aee4dd2a3b0f2b1c2c10087be83f7b2b06f4368
2017-01-18 14:50:50 -08:00
Connor O'Brien cf4f7eac85 Merge "Pass flash erase & logical block size to mke2fs" 2017-01-17 20:07:11 +00:00
Tao Bao cac5423a23 Merge "releasetools: Replace print stmt with print()." am: 4a0b494eea am: 0e2a0fd76c am: 3cf2daceb6
am: 6074106214

Change-Id: I1a1df156a20cd8f88c908445e4e6e74036865e46
2017-01-14 15:58:02 +00:00
Tao Bao 4a0b494eea Merge "releasetools: Replace print stmt with print()." 2017-01-14 15:36:41 +00:00
Tao Bao 6210565f93 Merge "releasetools: Delete the obsolete --aslr_mode/-a flag." am: ecbb31e8c4 am: 82c2fb0e21 am: 6c828d25b8
am: 60f07ba750

Change-Id: I3ce58360ebff8c2503da288cb300b03ce3e32d5f
2017-01-14 08:36:26 +00:00
Tao Bao 89fbb0f6d5 releasetools: Replace print stmt with print().
So that it's compatible with Python 3.

Test: pylint --pylint=pylintrc

Change-Id: If06c135a492c94bedd713c8cbdf03155a502d5cd
2017-01-13 14:55:14 -08:00
Tao Bao 2a0d1da4d9 releasetools: Delete the obsolete --aslr_mode/-a flag.
It was added in commit 96be7205dc
("Working ASLR implementation.") in 2010, and removed in commit
1807e700a5 ("don't generate retouch
commands in OTA scripts") in 2012.

Remove the obsolete --aslr_mode flag.

Test: ota_from_target_files.py still works (by generating incremental
      and full OTAs respectively).

Change-Id: I6d8e62730ac192f3574d484c4a4b9b43b4ee0a9e
2017-01-13 22:46:27 +00:00
Connor O'Brien 20f08c3e16 Pass flash erase & logical block size to mke2fs
This information can be used to tune ext4 stripe and stride in the
userdata partition for better performance

Test: Build & flash userdata, confirm correct stripe & stride values
Bug: 33243520
Merged-In: Ia97cdd2d0239c3484b895fce49299f692ef911d8
Change-Id: Ia97cdd2d0239c3484b895fce49299f692ef911d8
Signed-off-by: Connor O'Brien <connoro@google.com>
2017-01-13 12:17:05 -08:00
Connor O'Brien e9ee68d841 Pass flash erase & logical block size to mke2fs
This information can be used to tune ext4 stripe and stride in the
userdata partition for better performance

Test: Build & flash userdata, confirm correct stripe & stride values
Bug: 33243520
Change-Id: Ia97cdd2d0239c3484b895fce49299f692ef911d8
Signed-off-by: Connor O'Brien <connoro@google.com>
2017-01-13 11:32:02 -08:00
William Roberts fc6ab0c42c fsconfig: fix pylint warning am: 4165c635b8 am: 3f9be0b6ae am: d469a347fb
am: 04a672bdb6

Change-Id: I35ee3d659ccf377fcbef2df3b526f9334f9823a4
2017-01-05 00:39:18 +00:00
William Roberts ceec74b6f9 fsconfig: fix style formating with yapf am: 820421c9b5 am: 529b397e83 am: e344c7672c
am: 37e403e4a3

Change-Id: I6132dc60a93247247ae8045d9afca260571f908f
2017-01-05 00:39:12 +00:00
William Roberts 4165c635b8 fsconfig: fix pylint warning
Fix pylint warning:
W:292,47: Cell variable lineno defined in loop (cell-var-from-loop)

For the closure function that prints the error message, we actually
want the value of modified lineno within the closure.

Test: run unit tests

Change-Id: I7d08184563d347c826ca1af7558fb8e378f151e1
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2017-01-04 18:22:05 +00:00
William Roberts 820421c9b5 fsconfig: fix style formating with yapf
A few changes got introduced that yapf re-styles. To make it easy,
keep this file formated to:

yapf -i --style=google fs_config_generator.py

Test: run unit tests

Change-Id: I3160b92d7fd07a4d315dde59f1d20fe4c7587da1
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2017-01-04 18:21:28 +00:00
Elliott Hughes 5317a94798 Merge "fs_config: Fix oem range parsing" am: ce6c8e629f am: c16a64b832 am: d7217e2847
am: 83e65d0e9f

Change-Id: I6a03aa3801ba64f5596a4b60b1577a890f7c1ad5
2017-01-04 00:48:41 +00:00
Elliott Hughes ce6c8e629f Merge "fs_config: Fix oem range parsing" 2017-01-04 00:37:58 +00:00
Tao Bao eee32c7b4a Merge "blockimgdiff.py: Clean up stash id computation in BBOTA v3+." am: 7cfc591596 am: 74b665fef1 am: 085daa6451
am: 6fbb51cb57

Change-Id: Ie9dd316576355899e0009b1c04ebca544a8e3a16
2017-01-03 23:55:18 +00:00
Johan Redestig 1552a28e5f fs_config: Fix oem range parsing
Addresses the following build error:
  "value" not in valid range [], got: 2901 for: "AID_ABC" file:\
  "device/somc/abc/config.fs"

that happens when parsing device/somc/abc/config.fs:
  [AID_ABC]
  value: 2901

The problem is that the AID.PREFIX is 'AID_' so the _OEM_RANGE
expression is searching for AID__OEM_RESERVED, which never
exists. Drop the extra '_'.

Change-Id: I8a1556731e14e49cef66e3e91121e4be23a308e0
Test: ./build/tools/fs_config/fs_config_generator.py fsconfig --aid-header=system/core/include/private/android_filesystem_config.h device/somc/abc/config.fs
2017-01-03 09:45:33 +01:00
Tao Bao 3a2e350239 blockimgdiff.py: Clean up stash id computation in BBOTA v3+.
Only BBOTA v2 needs to maintain a pool of available 'stash slot id'.
BBOTA v3+ uses the hash of the stashed blocks as the slot id, which
doesn't need the id pool anymore.

Bug: 33694544
Test: Generate v2 and v4 incrementals w/ and w/o the CL. They produce
      the same packages respectively.
Change-Id: I8121af5b6b1bee98c3639d54a00b06fd12e378e8
2016-12-28 09:26:46 -08:00
Elliott Hughes 21c197238d Revert "Revert "Merge changes from topic 'fsconfig-2'"" am: 2d7c86dfea am: feb4f2e54a am: 8a2ed5159c
am: 3315b71ca8

Change-Id: I385eb63030834d9d684baee6157d1a5777d117b1
2016-12-27 21:23:25 +00:00
Elliott Hughes feb4f2e54a Revert "Revert "Merge changes from topic 'fsconfig-2'""
am: 2d7c86dfea

Change-Id: I825336a18f7d7525bb49dd65d24ddccbfef84ba5
2016-12-27 21:01:23 +00:00
Elliott Hughes 5f60b74297 Merge "Revert "Revert "Merge changes from topic 'fsconfig-2'""" 2016-12-27 20:51:07 +00:00
Tao Bao 0ebc21c4b5 Merge "blockimgdiff.py: Add a test case for ReviseStashSize()." am: f967849693 am: a10cd259fc am: 2137170e03
am: b7dd53bb64

Change-Id: I853bb71942bdce62bb515b0b089e647ab6c0e062
2016-12-19 21:32:46 +00:00
Tao Bao 304ee27e88 blockimgdiff.py: Add a test case for ReviseStashSize().
Bug: 33687949
Test: python -m unittest test_blockimgdiff
Change-Id: I31e42648d2c0e60b941908c098ce5ccc967861ca
2016-12-19 11:17:12 -08:00
Elliott Hughes 2d7c86dfea Revert "Revert "Merge changes from topic 'fsconfig-2'""
This reverts commit fad4b4b715.

Incorporating the following fixes:

1.

fsconfig: fix fs_config_* build for discovered headers

When android_file system_config.h is picked up from the device
directory, neither TARGET_FS_CONFIG_GEN or TARGET_ANDROID_FILESYSTEM_CONFIG_H
are specified. Thus, the build is not generating the required fs_config_files
and fs_config_dirs.

Test: Ensure that make fs_config_files works, and produces the same output as before
      Build the system image and mount it as a loop back and dump the file system
      capabilities with getcap. Verify that output to the supplied
      android_file system_config.h

From the loopback of the system.img mount, from CWD system/bin:
$ getcap *
cnss-daemon = cap_net_bind_service+ep
hostapd = cap_net_admin,cap_net_raw+ep
imsdatadaemon = cap_net_bind_service+ep
ims_rtp_daemon = cap_net_bind_service+ep
logd = cap_setgid,cap_audit_control,cap_syslog+ep
mm-qcamera-daemon = cap_sys_nice+ep
pm-service = cap_net_bind_service+ep
run-as = cap_setgid,cap_setuid+ep
surfaceflinger = cap_sys_nice+ep
webview_zygote32 = cap_setgid,cap_setuid,cap_setpcap+ep
webview_zygote64 = cap_setgid,cap_setuid,cap_setpcap+ep

Compared to the android_filesystem_config.h:
    { 00700, AID_CAMERA,    AID_SHELL,     (1ULL << CAP_SYS_NICE), "system/bin/mm-qcamera-daemon" },
    { 00755, AID_SYSTEM,    AID_SYSTEM,    (1ULL << CAP_NET_BIND_SERVICE), "system/bin/pm-service" },
    { 00755, AID_SYSTEM,    AID_SYSTEM,    (1ULL << CAP_NET_BIND_SERVICE), "system/bin/imsdatadaemon" },
    { 00755, AID_SYSTEM,    AID_RADIO,     (1ULL << CAP_NET_BIND_SERVICE), "system/bin/ims_rtp_daemon" },
    { 00755, AID_SYSTEM,    AID_SYSTEM,    (1ULL << CAP_NET_BIND_SERVICE), "system/bin/cnss-daemon"},

2.

fsconfig: fix error message for duplicate AID

Fixes:
    raise ValueError('Duplicate aid value "%u" for %s' % value,
TypeError: %u format: a number is required, not str

and

    raise ValueError('Duplicate aid value "%s" for %s' % value,
TypeError: not enough arguments for format string

3.

fsconfig: add test for duplicate ranges

Add a test for duplicate range detection.

4.

fsconfig: skip AID_APP, AID_USER and all ranges

Do not output AID_APP, AID_USER and ranges. A range
is defined as ending with AID_ and ending in _START or
_END.

5.

fsconfig: test for skip AID_APP, AID_USER and all ranges

Test against AIDs that caused the bionic tests to fail.

Change-Id: I95569a9ccc83bd3231f8a6f395532cc2de316bd2
Signed-off-by: William Roberts <william.c.roberts@intel.com>
2016-12-18 10:55:35 -08:00
Tao Bao f82589a393 Merge "releasetools: Fix the computation in ReviseStashSize()." am: 9cc98767ad am: bf32e87669 am: b317d1c6ec
am: f783aeba60

Change-Id: I98e29b566d0d7e69bdd64a5ef9522e56bf9f9c62
2016-12-17 00:32:36 +00:00
Tao Bao 9cc98767ad Merge "releasetools: Fix the computation in ReviseStashSize()." 2016-12-16 23:57:40 +00:00
Tao Bao e27acfdc41 releasetools: Fix the computation in ReviseStashSize().
We compute the max stashed_blocks in ReviseStashSize(), prior to calling
WriteTransfers(), to avoid running out of space due to stashing.

There is a bug when computing the to-be-freed stashed blocks, where we
wrongly free the space _before_ executing the transfer command. This leads
to a script failure where the max stash size violates the max allowed
size in WriteTransfers().

Note that this bug doesn't affect already generated packages. It's only
an underestimate in ReviseStashSize(). The check in WriteTransfers() has
been correct to ensure the max stash size.

Bug: 33687949
Test: Successfully generated incremental OTA which failed previously.
Change-Id: I4f4f043c6f521fce81ca5286e6156f22d99bf7f7
2016-12-16 11:26:05 -08:00
Alex Klyubin 1ea326df0b Merge "Faster auto-detection of APK's minSdkVersion" am: 620686f3c6 am: 1f5e43cb6f am: 9d439f54b1
am: 25aa9cd031

Change-Id: I4f19a1550a43515bd77e63fd5745b01d40c6be50
2016-12-15 00:39:09 +00:00
Alex Klyubin 620686f3c6 Merge "Faster auto-detection of APK's minSdkVersion" 2016-12-14 23:42:13 +00:00
Tao Bao c84463cd18 Merge "releasetools: Add prefix when dumping fingerprints." am: a282ceeb73 am: bf9e4aa53f am: aff4efc8c1
am: cdea34d0f1

Change-Id: I09380e5e782dd9f42f3621c6bc593dd2128afceb
2016-12-14 21:20:26 +00:00
Alex Klyubin 9b75e272b6 Faster auto-detection of APK's minSdkVersion
Prior to this change, when signing APKs, the build system invoked
'aapt dump badging' on each APK, to detect the value to pass into
signapk as --min-sdk-version. Now that signapk uses the apksig
library, it can auto-detect that value on its own, thus avoiding the
need to invoke 'aapt dump badging' and thus speeding up the build
process.

The semantics of signapk's --min-sdk-version flag is changed by this
commit from having the default value of 0 to having the default value
of "auto-detect from APK".

P.S. The get-package-min-sdk-version-int is not removed from
core/definitions.mk in this commnit, because this function is used in
another project's .mk file and thus that .mk file needs to be modified
first.

Test: rm -Rf out/ && make
Change-Id: I0972fcf0abbde9cbf6794e6c05c743c77c8a78f9
2016-12-14 12:53:44 -08:00
Treehugger Robot a282ceeb73 Merge "releasetools: Add prefix when dumping fingerprints." 2016-12-14 20:53:02 +00:00
Tao Bao f9023856ad releasetools: Add prefix when dumping fingerprints.
We used to dump "Source: <fingerprint>" in update logs. The "Source: "
prefix was unintentionally dropped out.

Test: Check the generated incremental BBOTA script.
Change-Id: I4de62333aa38e3fb09a76df0e769b62af48e0313
2016-12-14 11:53:38 -08:00
Elliott Hughes 633b7495c5 Revert "Merge changes from topic 'fsconfig-2'" am: fad4b4b715 am: e472384a4b am: 2ed4b63f7b
am: 90bf4657cc

Change-Id: I6df36bcb1cfbb814bfb7daa0cd64f1013c434935
2016-12-13 04:53:38 +00:00
Elliott Hughes fad4b4b715 Revert "Merge changes from topic 'fsconfig-2'"
This reverts commit 1b2de51881, reversing
changes made to 77ea32f378.
2016-12-12 17:28:44 -08:00
William Roberts b12cb90a97 fs_config: add unit tests am: 92ec0ab5e8 am: bd19c0cd23 am: d04e585b89
am: 45413ac579

Change-Id: Ic717885be5da808cc1a45516bb0f7bb9ba4303f0
2016-12-10 00:46:30 +00:00
William Roberts af51373e7d fs_config: drop fs_config_files/dirs PRODUCT_PACKAGES requirement am: 7fe8b6d817 am: ee9af6edfb am: 0101f107c1
am: 718be50fc5

Change-Id: Ia5b7259ad39b5c68ceece65d2f8aa439b37e3957
2016-12-10 00:46:23 +00:00
William Roberts 5577be3f61 fs_config: introduce group generator am: 1c4721c3c5 am: 3315954861 am: 6b50e15947
am: 93bf4c147b

Change-Id: I0f89abce25e07b38d368ad3c1684ff6a86ee6669
2016-12-10 00:46:10 +00:00
William Roberts f376729542 fs_config: introduce passwd generator am: 316f9462af am: 62d75469b1 am: 13be8e4305
am: 55e33be098

Change-Id: I47becfb9904fbc22c6b926bf2dd7271c87204f4d
2016-12-10 00:45:54 +00:00
William Roberts ad90c9ba94 fs_config: generate friendly in AID class am: 8f42ce71ac am: 092fa031cd am: aa200330c5
am: 66589ea73b

Change-Id: I7834237dd926e4fbd3e862000de159511fea43ea
2016-12-10 00:45:46 +00:00
William Roberts 469519b656 fs_config: limit characters for AID_<name> sections am: 5f059a7691 am: 7fcf246c3b am: 10db3e3cb9
am: dc4190d78e

Change-Id: Id2d84fad6e70be0e4e97c433a2f42ee597316005
2016-12-10 00:45:38 +00:00
William Roberts 4069fbbec8 fs_config: generate oem AID header file am: cfc51f5347 am: 9652abfeb4 am: 64ab591155
am: dca25c1733

Change-Id: I5f9db1c1023b168554e40fb0b863f027945b0f60
2016-12-10 00:45:31 +00:00
William Roberts 1e02a6ec60 fs_config: android_id header generator am: d7104bca65 am: 04fdcc9edf am: 343bee6044
am: 3c0ea33152

Change-Id: I1abe84234579e95edf380750da221fd88da2892f
2016-12-10 00:45:24 +00:00
William Roberts 25bacda5da fs_config: support parsing android_filesystem_config.h am: 64edf5bb97 am: ca3dbe4ab8 am: 5302f1815c
am: d1eeee812a

Change-Id: I4d3c17fc2d09bd1d2ad5f39c56617e94fe024197
2016-12-10 00:45:18 +00:00
William Roberts 3210677b40 fs_config: modularize fs_config_generator am: 11c29283ec am: ce24636551 am: 6ebc4026a7
am: 9102172f30

Change-Id: I06f6deaa8349c2a288bb9224cabdffaffa89c14f
2016-12-10 00:45:10 +00:00