Commit Graph

297 Commits

Author SHA1 Message Date
Patrice Arruda a5c2542f68 Soong: Refactor the soong_ui arguments processing to be more like bazel.
Currently, the command line argments is being processed in multiple
places. In the main soong_ui arguments, there are several if statements
that checks if a specific argument was specified in order to execute
the requested operation. This does not scale well when adding or
removing a command in the near future. In order to support the build
commands (m, mma, etc...) from the envsetup.sh in soong_ui, a refactor
was required in order to add a command rather quickly and to have the
flexibiity to unit test the command. The soong_ui arguments format is
as follows:

soong_ui <command> [<arg 1> <arg 2> ... <arg n>]

The <command> is a specific operation to be executed. The arguments
after the command are processed by the command itself.

Below is the list of changes made in this commit:

 * Created a new command infrastructure that allows adding
   or deprecating a command easily.

 * Fixed a bug when running ./soong_ui.bash directly would cause
   a panic due to index being out of range for args list.

Bug: b/130049705
Test: Below is the list of testing done on this commit:
    * Ran lunch and verified that the output is the same as the the output
      without the modifications. lunch indirectly runs soong_ui passing
      in the --dumpvar-mode (to read makefile variables such as
      TARGET_PRODUCT) and --dumpvars-mode (to build the build cache).
    * Ran ./soong_ui.bash directly (with unsupported flags and no flags)
      and the proper message appeared that soong native UI is not yet
      available.
    * Ran m, mm, mmm, mma, mmma commands.
    * Ran the make installclean command:
         make -j80 TARGET_PRODUCT=aosp_arm64 TARGET_BUILD_VARIANT=eng dist DIST_DIR=/tmp/helloworld installclean
    * Ran ./out/soong_ui -j80 --make-mode PRODUCT-aosp_arm64-eng dist checkbuild tests

Change-Id: Iee4de7ec0fa4661206fda8ae1fe6fa4487d9bb22
Merged-In: Iee4de7ec0fa4661206fda8ae1fe6fa4487d9bb22
2019-05-06 14:05:52 -07:00
Colin Cross f41ed9ae63 Merge "Remove services.art from known_nondeterminism.whitelist" am: 42603c4ab5
am: 828a488720

Change-Id: Idbdeec76c0d8207890afbf94ea20ebac4c3d5379
2019-04-30 15:24:25 -07:00
Colin Cross 497d4c24b1 Remove services.art from known_nondeterminism.whitelist
Bug: 120039850
Test: none
Change-Id: I70d5243f427a4da5c8126be6b8ebc90221133ad3
2019-04-29 17:36:19 +00:00
Dan Willemsen 7d155094be Merge "pom2bp: Implement -static-deps to match pom2mk" am: 80ae6816ae
am: 9181e08738

Change-Id: I331a7bb1cd82315f673d9007c2240da423864034
2019-04-22 20:17:35 -07:00
Dan Willemsen 52c90d868e pom2bp: Implement -static-deps to match pom2mk
This changes the default behavior of pom2bp to match pom2mk, and not
require pre-extraction of all of the manifests.

Test: convert an internal pom2mk user to pom2bp
Change-Id: I08120cce1f923ef28519563ffd772032b2e500ed
2019-04-21 21:46:42 -07:00
Dan Willemsen 8a4c0da74d Add -jetifier to pom2mk and pom2bp am: 7fdab6eeec
am: fd9fa1a56e

Change-Id: Ifaa39849ef2d858dd63745e2a8a0b9e5587bbd66
2019-04-21 02:47:34 -07:00
Dan Willemsen 7fdab6eeec Add -jetifier to pom2mk and pom2bp
Test: use pom2mk -jetifier to replace some hand-modified makefiles
Change-Id: I2a1988d0120fee1edf2d85c8b3294041840efe7c
2019-04-20 21:47:14 -07:00
Paul Duffin 0018e7f48b Merge "Rename -extra-deps to --extra-static-libs and add --extra-libs" am: 38b122021a
am: 2b9f534fba

Change-Id: I1a607c72ad7f1a71b7dcc0aa216beee2b868776f
2019-04-17 07:32:57 -07:00
Paul Duffin 38b122021a Merge "Rename -extra-deps to --extra-static-libs and add --extra-libs" 2019-04-17 14:06:48 +00:00
Paul Duffin babaf07b13 Rename -extra-deps to --extra-static-libs and add --extra-libs
Allow callers of pom2bp to add dependencies on runtime libraries as
well as static libraries. Renames -extra-deps to --extra-static-libs
to make it clearer what it does.

Bug: 130287656
Test: prebuilts/sdk/update_prebuilts.py --stx 5280039
      prebuilts/misc/common/androidx-test/update-from-gmaven.py

Change-Id: I8a3ee8056aa4ed9935a4f47d563ec423672cf38b
2019-04-16 11:55:48 +01:00
Jaewoong Jung a7d23ae804 Merge "Embed CSS entries in soong_build.html." am: 0a0b3fa605
am: 6d48aa4186

Change-Id: I0e162fdc0eabafc152686f637436b41295540935
2019-04-15 18:17:14 -07:00
Dan Willemsen 891af48379 Support RuleBuilder.Sbox to wrap commands in sbox am: 633c502295
am: b933c57635

Change-Id: I07cb4998e8d860f97c4ae00f9f776744cc8a5adf
2019-04-15 17:50:48 -07:00
Treehugger Robot 0a0b3fa605 Merge "Embed CSS entries in soong_build.html." 2019-04-16 00:49:18 +00:00
Jaewoong Jung 5f867c0580 Embed CSS entries in soong_build.html.
This will prevent it from being affected by future css location changes.

Test: m soong_docs
Fixes: 130567020
Change-Id: I80fd7b8a12a32bcea7b270423b8d8f8dc0976177
2019-04-15 15:21:55 -07:00
Dan Willemsen 633c502295 Support RuleBuilder.Sbox to wrap commands in sbox
This essentially allows you to declare that everything in a directory
will be created by the rule, and we'll ensure that your command actually
writes out all of the claimed outputs, and remove any other files that
previously existed in that directory.

Test: built-in tests
Change-Id: I990dce2b3a0d89ebd2736ac1a0cadfb5864c6e73
2019-04-15 14:52:05 -07:00
Colin Cross 537cfd2477 Merge "pom2bp: add jar deps to -nodeps modules" am: 34df71446f
am: 694b4629e1

Change-Id: I519921487ae1813091e08074df70376fa39efb61
2019-04-11 16:23:49 -07:00
Colin Cross 1aa7f26a22 pom2bp: add jar deps to -nodeps modules
The dependency type in the pom file is not always accurate, just
add all jar and aar dependencies to both the -nodeps and final
module.

Test: cd prebuilts/sdk/current/androidx && pom2bp -regen Android.bp && mma
Change-Id: I1784b8ebf3485db19405cb353c9bb5fb3ae837d8
2019-04-10 18:31:55 +00:00
Colin Cross 11fbb890c9 Fix package path of android/soong/android pctx am: cc0ce80ed5
am: a1c57b4664

Change-Id: Ifba10892f902e70bd0a52a82a20bd5f1b64a61de
2019-04-04 01:39:57 -07:00
Colin Cross cc0ce80ed5 Fix package path of android/soong/android pctx
android/soong/common was renamed to android/soong/android long
ago, but the pctx package path was still "android/soong/common".
This required all users of rules defined in android/soong/android
to import "android/soong/android" and then
pctx.Import("android/soong/common").

Test: m checkbuild
Change-Id: I20d096522760538f7cfc2bec3d4bfeba99b275d4
2019-04-04 03:59:34 +00:00
Colin Cross a50a2464c6 Merge changes I7dd4ba7f,I7887a67a am: 92cc71606c am: 900574f0cd
am: 6c99a30bb7

Change-Id: If557897ed64ecdf8967299b5512392e5485767ae
2019-04-02 08:21:42 -07:00
Colin Cross 92cc71606c Merge changes I7dd4ba7f,I7887a67a
* changes:
  Add depfile support to RuleBuilder
  Add RuleBuilderCommand.Flags
2019-04-02 15:04:06 +00:00
Colin Cross f37f6b064c Merge changes Ie7f776a4,I3ca5dd1f,I2e911262 am: 11581cf6b3 am: 89d99337fb
am: 92270109eb

Change-Id: Ic7ac18a1d60138e909c6045fd5313620256989d4
2019-04-01 17:48:29 -07:00
Colin Cross 11581cf6b3 Merge changes Ie7f776a4,I3ca5dd1f,I2e911262
* changes:
  Add a whitelist for services.art changing between builds
  Add diff_target_files tool
  Import files from compare_target_files for use in diff_target_files
2019-04-02 00:03:05 +00:00
Colin Cross 93599cdb93 Merge "Touch output files in sbox" am: 80eb6907ac am: 9457c97c40
am: 5cb90e6517

Change-Id: I07587e71632567f553f417db40b791e4d335aaae
2019-04-01 11:15:17 -07:00
Colin Cross d1c1e6fd6c Touch output files in sbox
Some tools used by genrules set the timestamp of the generated file
to an old value.  For example, when using tar to extract files it
will set the timestamp of the extracted file to the value stored in
the tar file.  This can cause unnecessary rebuilds due to output files
that are older than their inputs.  Touch output files in sbox before
moving them into the output location.

Test: m checkbuild && m checkbuild
Change-Id: Idb0e71d2bf0ac3618338371132cc71be74bf9d9d
2019-04-01 10:43:20 -07:00
Colin Cross d93112ae26 Add a whitelist for services.art changing between builds
Bug: 120039850
Test: manual
Change-Id: Ie7f776a4e2743883ea006c66148fc4f40f90ce07
2019-04-01 10:07:25 -07:00
Colin Cross 91ee4c793d Add diff_target_files tool
Add a tool that lists files that have changed between two builds by
comparing the target-files.zip artifacts.

Bug: 121158314
Test: unit tests
Change-Id: I3ca5dd1fdf652dffef923ad7307fd71e49e2e95a
2019-04-01 10:07:21 -07:00
Colin Cross c45c3b5ec6 Import files from compare_target_files for use in diff_target_files
Copied from cl/240594925.

Bug: 121158314
Test: copied unit tests
Change-Id: I2e91126285dcd33171ff8b8dbfcfa5d48501f535
2019-04-01 10:02:40 -07:00
Colin Cross 1d2cf0494a Add depfile support to RuleBuilder
Allow rules built with RuleBuilder to use depfiles.  Ninja only
supports a single depfile with single output.  If there are
multiple outputs in a rule, move all but the first to implicit
outputs.  If multiple depfiles are specified, use new support
in dep_fixer to combine additional depfiles into the first depfile.

Test: rule_builder_test.go
Change-Id: I7dd4ba7fdf9feaf89b3dd2b7abb0e79006e06018
2019-03-29 16:35:06 -07:00
Jaewoong Jung 9b4cd0c39b Build doc improvements. am: 12c02a63d0 am: dde127fdb6
am: 8c822915fb

Change-Id: If8b0c3432f659b0739f92cf07f4a0020a6c1cbd8
2019-03-13 10:40:57 -07:00
Jaewoong Jung 12c02a63d0 Build doc improvements.
Bring property type strings to the start of property comments. This is
how source.android.com shows variables and paratemeters, and it is also
easier to read, especially when a comment ends with a code example.

Test: m soong_docs
Change-Id: Ic3e9db3a3711495998919ee95bbef6c7cb33b215
2019-03-12 13:31:11 -07:00
Jaewoong Jung e4d2d08930 Merge "Change module type text to HTML in build docs." am: 0d2c1d1e23 am: 7c8d1b2400
am: dfd6745462

Change-Id: I56e69eb99966564a754efa45c15726cbc8e52590
2019-03-11 19:27:54 -07:00
Jaewoong Jung 238be3807c Change module type text to HTML in build docs.
To add code format (<pre>...</pre>) support to module type texts.

Fixes: 124332708
Test: m soong_docs
Change-Id: Ibe9d3845a8f952fdef5007893d4af5836809ce1f
2019-03-11 14:35:41 -07:00
Dan Willemsen 6b783c8391 Fix recent builds with bad dangling symlinks
It appears that there's a bug in repo where when a linkfile is removed
from the manifest, the symlink remains in the source tree.

Cleaning this up needs to happen before we start scanning the source
tree for Android.bp/Android.mk files, so this has to be done very early
-- way before CleanSpec processing could handle it.

Bug: https://bugs.chromium.org/p/gerrit/issues/detail?id=10583
Test: ln -s missing hardware/qcom/sdm710/Android.bp; m
Change-Id: Ib68f2507ffe58ccdd9fbc88925f8a4f6150f2f7d
2019-03-08 19:56:57 +00:00
Colin Cross 0b9f31fb08 Replace *Escape with *EscapeList
Follow the change to blueprint to make *Escape take and return a string
and add *EscapeList that take and return slices of strings.  Fix up
a few places that were unnecessarily converting a string to a slice
and back to a string.

Test: m nothing
Change-Id: I3fa87de175522205f36544ef76aa2f04aef1b936
2019-03-04 18:11:53 +00:00
Jaewoong Jung d10f484eee Add a filename to anchor links in build docs.
The href base added by I8a8ac0f9ba6c0d57ed83db2155955a95e070a265
doesn't really play nice with anchor links. This fixes the issue.

Fixes: 126550408
Test: m soong_docs
Change-Id: If62411a6391fd8151fa40cb2ecf32c5006797d6c
2019-02-27 11:15:00 -08:00
Jaewoong Jung 90e1155135 soong_docs build link error fix.
This adds a script to copy the current page's base url to hrefs in the
doc.

Fixes: 123473548
Test: m soong_docs
Change-Id: I8a8ac0f9ba6c0d57ed83db2155955a95e070a265
2019-02-25 10:26:46 -08:00
Jaewoong Jung 6c29688691 Split up Soong build doc.
Previously the build doc was a gigantic list of modules and properties,
which can be overwhelming to new users. This change breaks it up by
packages, so that it is easier to look up and feels more coherent.

Bug: 123521276
Test: m soong_docs
Change-Id: I1a1a331f4b4deb8782d698e9076098c5a27b9566
2019-02-20 15:43:06 -08:00
Treehugger Robot a2b41a6e02 Merge "Improve documentation page layout" 2019-02-20 07:50:16 +00:00
Dan Willemsen 707542f298 Simplify python launcher, use __main__.py
Uses more python rather than C++, and skips less of Py_Main.

Test: build/soong/python/tests/runtests.sh
Change-Id: I03997d88e2e16047c96bb4e00e530229c42b3325
2019-02-15 14:33:59 -08:00
Sasha Smundak ff483393ac Improve documentation page layout
It now has module list on the left and the main panel containing
a section for each module. Each section contains the navigable list
of the module's attributes, grouped by attribute's origin.

Test: Manual
Bug: 120512870
Change-Id: Ib62ef58c61daa88950f0934a70ba8ed16f433bf2
2019-02-13 21:40:21 -08:00
Colin Cross 7089c27c07 Update Soong docs generator for blueprint changes
bootstrap.ModuleTypeDocs needs a mapping of module types to factories
to support factories that are wrapped in ModuleFactoryAdapter closures.

It also returns ModuleType objects grouped into Package objects.

Bug: 67909957
Test: m soong_docs
Change-Id: I70eac9f0f0e13075580da92d4219792ca0b18fbf
2019-02-13 20:39:26 +00:00
Sasha Smundak c0c9ef9964 Provide an option to reduce build's verbosity.
If ANDROID_QUIET_BUILD environment variable is set to '1', do not show 20
lines of the environment variables. In addition, for the failing step show
only the step name and output, omitting the command proper (the verbose log
as well as error.log still contains it).
If build succeeds, the output of the build itself consists of a single
line:
```
> m androidmk
[100% NN/NN] <last command>
```

When it fails, the output does not contain sometimes very long command
line:
```
> m androidmk
[ 97% NN/MM] test androidmk
FAILED: <step>
--- FAIL: TestEndToEnd (0.01s)
    androidmk_test.go:1025: failed testcase 'prebuilt_etc_TARGET_OUT_ETC'
        input:
        include $(CLEAR_VARS)
        LOCAL_MODULE := etc.test1
        LOCAL_MODULE_CLASS := ETC
        LOCAL_MODULE_PATH := $(TARGET_OUT_ETC)/foo/bar
        include $(BUILD_PREBUILT)

        expected:
        prebuilt_etc {
            name: "etc.test1",
            filename: "foo/bar",
        }

        got:
        prebuilt_etc {
            name: "etc.test1",
            filename: "foo/bar",

        }

FAIL
17:50:53 ninja failed with: exit status 1
```
[The related change in build/make/envsetup.sh suppresses timing display
when the same variable is set.]

Change-Id: I4d3c72457de031ff58a324c2fe98f4c1d10f8239
Test: treehugger
2019-02-07 12:21:45 -08:00
Nan Zhang d50f53bc25 Fix the issue for dumping metrics file to artifact
Bug: b/63815990
Test: N/A
Change-Id: I72ddc8579626e652a8c27299e009b24ded5312f6
2019-01-07 20:26:51 -08:00
Nan Zhang 17f2767724 Generate build timing metrics to proto format file
Test: Dumped the text formated based metrics file to out dir,
and checked the file.
Bug: b/63815990

Change-Id: Iff476f72a0be74eb53b6b26ef468d11c0f24a404
2019-01-04 15:54:01 -08:00
Dan Willemsen 263dde779e Fix par file zip offsets
The zip file format does support scripts/programs prefixed to the
archive, though many of the offsets are supposed to start from the
beginning of the file. Some tools (python and zipinfo) are able to
read zip files with arbitrary prefixes, but others (libziparchive and
zipdetails) fail.

So pass the file to prefix onto the zip file into merge_zips so that we
can set the offsets from the real start of the file.

Test: m sepolicy_tests  (runs the embedded python interpreter)
Test: zipinfo out/host/linux-x86/bin/sepolicy_tests
Test: zipdetails out/host/linux-x86/bin/sepolicy_tests
Change-Id: If73d4c2465581f7de5aa47959284ecf2059df091
2018-12-03 15:55:15 -08:00
Dan Willemsen 10bf17cf45 Fix the host bionic linker check; add tests
Gold uses two PT_LOAD programs: RX, RW
LLD is using three PT_LOAD programs: R, RX, RW

All non-static host bionic executables are linked with LLD, as it's
required in order to embed the linker. The dynamic linker is considered
a static executable for this definition.

When the linker had been built with Gold, we inserted the the RX&RW
loads in the middle of the executable's readonly load, splitting the R
load into two. When this happened, all of the linker's loads existed 1:1
in the final executable.

Now that the linker is built with LLD, we're inserting the loads into
the same place, but LLD merges the first part of the executable's R load
with the linker's R load, so there's not a 1:1 matching.

So just make sure that all of the linker's loads are contained within
the executable's loads (and with the appropriate flags).

Test: m blueprint_tools (runs new tests)
Test: build host bionic binaries
Change-Id: If7868d2ab8980d27ba5ab8994fefe0d6142f215a
2018-11-12 22:29:44 -08:00
Colin Cross 714614ced9 Keep directories when moving glob results
Patterns containing multiple globs or a recurisve glob may match
files with the same name in multiple directories.  Keep the relative
directories of matches after the path entry containing a glob.

Bug: 117295826
Test: zip2zip_test.go
Change-Id: I5d663e546953af374175837551d23f484d568377
2018-11-01 20:34:44 +00:00
Dan Willemsen f99915f569 Speed up build_test
Stop writing out ninja files, as they become very large, especially when
multiplied by the number of defined products.

Test: treehugger
Change-Id: Id9529d14040acb72a0188e58b5db2911f142071e
2018-10-29 10:43:37 -07:00
Dan Willemsen a0790e35c7 Rework how linux_bionic is built with LLD
In order to simplify the wrapper function, and stop using a linker
script, generate a set of flags to pass to LLD. Then run
host_bionic_inject on the linked binary in order to verify the
embedding, and give the wrapper function the address of the original
entry point (_start).

Bug: 31559095
Test: build host bionic with prebuilts/build-tools/build-prebuilts.sh
Change-Id: I53e326050e0f9caa562c6cf6f76c4d0337bb6faf
2018-10-22 15:46:03 -07:00
Dan Willemsen 2249dc892d Turn symbol_inject into a reusable package
Bug: 31559095
Test: m blueprint_tools
Change-Id: I0f7a9f14111af26d753db547c6de313a7079658a
2018-10-22 15:46:03 -07:00
Colin Cross dc1e829b59 Always allow duplicates with identical CRC32 and size
Don't warn on duplicate files in merge_zips if they have identical
CRC32 and size values.

Test: m checkbuild
Test: merge_zips_test.go
Change-Id: I61336ca4d4d3b7402c24a7abd337bd350fe10930
2018-10-17 15:49:53 -07:00
Colin Cross 3824771b2d zip2zip: support included an excluded file
Make needs to exclude all files that match a filespec except for
those specified in a list.  Add a -X argument that overrides the
-x argument to allow files be included that were previously
excluded.

Bug: 69500920
Test: zip2zip_test.go
Change-Id: Icc8aebc788b53bcdde73a39c9c7cf3107a049251
2018-10-09 17:48:16 +00:00
Colin Cross b1a5e9cadf zip2zip: add flag to uncompress files
Add -0 flag to convert files in a zip to stored instead of deflated.

Bug: 69500920
Test: zip2zip_test.go
Change-Id: I6c2b10f3b200a53a3339e3c97a78f65192b309ca
2018-10-08 15:20:56 -07:00
Nan Zhang 40b41b4038 Use SoongJavacWrapper for javadoc
Test: m -j docs; javac_wrapper_test
Bug: b/116874473
Change-Id: I51d440c815d3db1a0a5dc26ad4e376189dc0aefe
2018-10-02 16:45:37 -07:00
Dan Willemsen 2997123f73 Refactor Kati logic
Make the cleanspec & Android.mk runs of kati use much of the same code
and arguments.

Also renames 'Kati' to 'KatiBuild' in many cases, in the the expectation
that we'll have a 'KatiPackage' step in the future.

Use --no_ninja_prelude and move local_pool & _kati_always_build_ into
the combined ninja file. This will reduce the need to re-read makefiles
when Goma is enabled, and it allows us to include more than one
Kati-generated ninja file in the build graph.

Bug: 116968624
Test: build_test on downstream branches
Change-Id: Ibdac689b81f62dc293647fad917d84946f2c3cfa
2018-09-28 23:09:24 -07:00
Colin Cross 05518bc13b soong_zip: Add tests
Add test that cover basic command line usage of soong_zip.  -D
is not covered yet as the implementation will be replaced with
one that is also more easily testable in the next patch.

Bug: 116751500
Test: zip_test.go
Change-Id: I5a1bcee74ebc9cb3cf332c36f89bc12c0e807ad2
2018-09-28 13:56:06 -07:00
Dan Willemsen bcc1dbf957 Refactor multiproduct_kati
Move the actual per-product builds out of the main function, combining
the product configuration and build sections. This makes it a lot more
readable.

This ends up saving a lot of FDs -- a -only-config build on aosp with 77
products used to require ~500 FDs, it succeeds with a limit of 150 FDs
now. I'm leaving the code to bump our FD limit though, since 2x the
number of internal products is too close to the 1024 limit for comfort.

Bug: 70370883
Test: prlimit -n150:150 build/soong/build_test.bash -only-config
Change-Id: Ia559beadc19deb8a5b9d50af6e0276e846fd8608
2018-09-05 13:57:10 -07:00
Dan Willemsen 41538385e5 Add an incremental mode to multiproduct_kati / build_test
This uses a very large amount of intermediates, but can speed up
subsequent multiproduct_kati runs by not re-reading all the makefiles
unless necessary.

Since we'll no longer have the full build output on every run,
explicitly save away a std_full.log of the log from the last time we ran
Kati.

Test: build/soong/build_test.bash -products aosp_arm -incremental
Change-Id: Iaae580c0e49a2024528617d39b2b477e29d99250
2018-09-05 13:53:59 -07:00
Colin Cross 632987ac21 pom2bp: clean up templates
Move some of the more complicated conditional logic out to helper
functions.  Use the {{- }} syntax to strip previous whitespace to
allow spacing out the remaning conditionals in the templates.

Test: (cd prebuilts/sdk/current/androidx && pom2bp -regen Android.bp)
Change-Id: I766bd0e1837aa04375f322fbe796d923cd99ecde
2018-08-30 11:18:50 -07:00
Colin Cross 8a9900a75f pom2bp: use java_import_host for host modules
java_library_host modules shouldn't depend on java_import modules.

Bug: 112707915
Test: (cd prebuilts/sdk/current/androidx && pom2bp -regen Android.bp)
Change-Id: Ia5ac353a696c9f51f1b50b334a327937ddcb47e7
2018-08-29 15:37:19 -07:00
Colin Cross 86bc9d4622 gofmt -w .
Test: m checkbuild
Change-Id: Ia4aec5ddadf1f1b00f7c567522ca89fd839504f0
2018-08-29 15:37:15 -07:00
Anton Hansson 179d89bc5b Merge "Pass extra args from multiproduct_kati to config." 2018-08-28 07:27:59 +00:00
Jeff Gaston d492853a52 pom2bp support to specify which modules are host modules
Bug: 112804807
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.bp
Change-Id: Ib816db8274cb05c9d0ed2cb557799ddcf6ee74fd
2018-08-24 14:32:50 -04:00
Anton Hansson 43da16a440 Pass extra args from multiproduct_kati to config.
Allows specifying make goals when using multiproduct_kati, for example:
$ multiproduct_kati -only-config dump-products

This makes dumping products with multiproduct_kati easier than before.

Test: multiproduct_kati -only-config -products=full,aosp_arm64 -out=/tmp/p dump-products
Change-Id: If526bd2ba1b7bbadcb230c020202ef9e421e1852
2018-08-20 16:16:02 +01:00
Dan Willemsen 59339a29e1 Fix `go vet` issues
Test: go vet ./...
Change-Id: Ifb936ccc5e2b5a2c3fcbbbcb54f680e2973ea1b3
2018-07-22 21:18:45 -07:00
Dan Willemsen 34218ece56 Make multiproduct_kati return error
Test: insert error, run build/soong/build_test.sh and check $?
Change-Id: I3462defd1ff04247a8b729e22988cbe002b0ce69
2018-07-19 22:57:18 -07:00
Dan Willemsen c35b3813bb Fix missing errors during lunch
Bug: 111499523
Test: lunch foo
Test: m nothing 2>/dev/null
Change-Id: Ife888d4b731bc0a0dd253ac0c8c97d6c37593d73
2018-07-16 19:59:10 -07:00
Colin Cross cdec7e7ee1 Merge changes I5b8522aa,I541aea75
* changes:
  Fix lint errors caught by go test
  Fix .kotlin_builtins glob
2018-07-17 00:14:33 +00:00
Treehugger Robot 6d6faa1a1f Merge changes from topic "merge_zips_strip"
* changes:
  Make merge_zips -stripFile use blueprint style globs
  Add tests for merge_zips
2018-07-17 00:09:58 +00:00
Colin Cross fe5a3b7fac Fix lint errors caught by go test
Test: m checkbuild
Change-Id: I5b8522aa9da22fc595e4f26c23e7836de59bdfa2
2018-07-16 15:24:43 -07:00
Colin Cross 4c03f68763 Make merge_zips -stripFile use blueprint style globs
merge_zips -stripFile was only considering the name of the file and
ignoring the path.  Make it more useful by supporting blueprint style
globs.  The previous behavior can be recreated by prefixing with **/.

Bug: 111389216
Test: m checkbuild
Change-Id: I25760fe3f1f77704dd9da9d107d9a38a415d681f
2018-07-16 15:24:43 -07:00
Colin Cross 2486065c43 Add tests for merge_zips
Test: merge_zip_test.go
Change-Id: I8fc577946b40cad193864aa2ebda1bef3a0e050e
2018-07-16 15:24:22 -07:00
Dan Willemsen b82471ad6d Add a unified status reporting UI
This adds a new status package that merges the running of "actions"
(ninja calls them edges) of multiple tools into one view of the current
state, and gives that to a number of different outputs.

For inputs:

Kati's output parser has been rewritten (and moved) to map onto the
StartAction/FinishAction API. A byproduct of this is that the build
servers should be able to extract errors from Kati better, since they
look like the errors that Ninja used to write.

Ninja is no longer directly connected to the terminal, but its output is
read via the protobuf frontend API, so it's just another tool whose
output becomes merged together.

multiproduct_kati loses its custom status routines, and uses the common
one instead.

For outputs:

The primary output is the ui/terminal.Status type, which along with
ui/terminal.Writer now controls everything about the terminal output.
Today, this doesn't really change any behaviors, but having all terminal
output going through here allows a more complicated (multi-line / full
window) status display in the future.

The tracer acts as an output of the status package, tracing all the
action start / finish events. This replaces reading the .ninja_log file,
so it now properly handles multiple output files from a single action.

A new rotated log file (out/error.log, or out/dist/logs/error.log) just
contains a description of all of the errors that happened during the
current build.

Another new compressed and rotated log file (out/verbose.log.gz, or
out/dist/logs/verbose.log.gz) contains the full verbose (showcommands)
log of every execution run by the build. Since this is now written on
every build, the showcommands argument is now ignored -- if you want to
get the commands run, look at the log file after the build.

Test: m
Test: <built-in tests>
Test: NINJA_ARGS="-t list" m
Test: check the build.trace.gz
Test: check the new log files
Change-Id: If1d8994890d43ef68f65aa10ddd8e6e06dc7013a
2018-07-12 14:15:31 -07:00
Colin Cross 461ba49a0a pom2bp: move misplaced {{end}}
manifest property should only be added to aar imports and not jar
imports.

Bug: 110848854
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.bp
Change-Id: Ifaab81582c87f0af4419f90c3928d0d5d52cec33
2018-07-10 13:46:58 -07:00
Colin Cross cf53e608a7 pom2bp: extract minSdkVersion from manifest into Android.bp
When manifest merging in Soong is turned on the modules created
by pom2bp will need to have min_sdk_version specified in order
to prevent manifest_fixer from setting it to the current SDK
level.  Read the minSdkVersion attribute out of the
AndroidManifest.xml file inside each .aar file.

Bug: 110848854
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.bp
Change-Id: Ieaf37ba6eccaf32dc5d3411566335830b1a327ff
2018-07-09 13:10:35 -07:00
Dan Willemsen 184901135c Revert "Revert "Revert "Revert "Add path interposer""""
This reverts commit 09f4540d66.

Fixes the raw call to net.Listen in the tests to go through the listen()
helper and use the long socket path fallbacks.

Removes the use of timeouts from the tests -- the behaviors being tested
did not rely on timeouts, so removing them will reduce the flakiness if
the build is heavily loading the machine at the same time the test is
running.

Also fixes some potential nil pointer dereferences.

Test: OUT_DIR=<really long> m blueprint_tools
Test: `while .../soong-ui-build-paths/test/test; do sleep 0.01; done` with a build running
Change-Id: I16d44be7517bc415f1c808284088f4ba40df3bfa
2018-05-31 14:59:33 -07:00
Dan Willemsen 09f4540d66 Revert "Revert "Revert "Add path interposer"""
This reverts commit c59a92cb1a.

Reason for revert: tests are broken with long OUT_DIRs
They're directly calling net.Listen, and not using the fallback
for long socket names.

Change-Id: Id14cbd499fd9b36c6926b7552d3554340cb0916c
2018-05-25 23:26:00 +00:00
Dan Willemsen c59a92cb1a Revert "Revert "Add path interposer""
This reverts commit 96c957ae20.

Fixes issues on some machines where the socket in TMPDIR ended up with a
unix domain socket pathname over 107 characters long, which Go will
reject due to underlying limitations in the system calls. If this
happens, we'll fall back to opening the directory, then using
/proc/self/fd/#/<file>, or manually creating a similar symlink in /tmp.

Also fixes some issues on Mac where os.Executable returns the symlink
instead of the underlying file, sending a message over a unix domain
socket will block if the reader isn't reading, and sandboxing was
preventing us from running `ps`.

Test: m blueprint_tools
Test: m blueprint_tools on mac
Change-Id: Ib19ccfe10cb0a79f1476fb1d5cd20ed0495be367
2018-05-25 14:07:00 -07:00
Dan Willemsen 96c957ae20 Revert "Add path interposer"
This reverts commit a14704c12b.

Reason for revert: breaking builds

Change-Id: I920b0ff41823428a0baf59b3fd82cbcc766babca
2018-05-19 00:54:13 +00:00
Dan Willemsen a14704c12b Add path interposer
This will allow us to track (and eventually limit) the commands that the
build references via $PATH. These are mostly implicit dependencies on
the host system -- for Linux, we assume something similar to Ubuntu
14.04 with a few extra packages, but this will let us better define
that.

This will not catch uses of tools with absolute paths (/bin/bash, etc),
but most uses shouldn't be relying on absolute path names anyways.

Adds ~400ms on the first startup, ~140ms on subsequent runs, and
overhead of a few ms for every forwarded execution.

Test: m
Test: build/soong/build_test.bash
Test: Add `gcc --version`, TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: Id68cbb1c8ceef65bbbb10751e83722c7662d2351
2018-05-18 13:24:36 -07:00
Dan Willemsen 04d76ef47d Fix multiproduct_kati product list
When the config loading was changed to using AndroidProducts.mk lists
from soong_ui instead of finding them itself, multiproduct_kati stopped
being able to find all the products. Move the finder initialization
earlier so that we can dump the full list.

Test: build/soong/build_test.bash -only-config
Test: forrest runs of every downstream build_test
Change-Id: I085d40f9008a47b850783499365977a0ff70ac66
2018-05-15 00:54:25 -07:00
Dan Willemsen 4339853a20 Add a dependency fixer for proto deps
protoc dependency files, at least for C++ outputs, uses the form of:

  a/b.c \
  a/b.h: <dep1> <dep2>...

Ninja will fail the command when it parses a dep file and there's more
than one output file (even though it doesn't care what the output file
name is). So this tool will parse the original file, and output a
version with only a single output file.

Bug: 67329638
Test: NINJA_ARGS="-t deps ...pb.c" m
Test: NINJA_ARGS="-t deps ...srcjar" m
Test: NINJA_ARGS="-t deps ...srcszip" m
Test: Run dep_fixer across all of taimen's dep files, no failures.
Test: Run dep_fixer against the processed files, no changes.
Test: Run androidmk across all of our Android.mk files, inspect the diffs
Change-Id: I4263b7d5faea37285afa6b24dedf5964aa7d19dc
2018-05-07 16:21:59 -07:00
Colin Cross 66f78820e1 Regenerate R.java files from LOCAL_STATIC_ANDROID_LIBRARIES
If a static android library lib1 has static_libs: ["lib2"] then the
R.class files for packages in lib2 will be merged into the jar for
lib1.  If an app has lib1 in its static_libs it will get the R.class
files from lib2 through lib1, instead of regenerating the R.java
files with numbering that matches the resource table of the app.

Pass transtive static android library dependencies on the aapt2
command line so that aapt2 will always regenerate the R.java
constants for those packages.

Also extract the packages that have R.java files after each aapt2
invocation.  This is not necessary for Soong, but is passed to
make to let it force regenerating the packages using
--extra-packages.

Bug: 78300023
Test: m checkbuild
Change-Id: I0f3444af44d2a9f370d1f156c908972f8cc3a1ee
2018-05-02 13:11:01 -07:00
Colin Cross 70dd38f09e Add pom2bp
Convert pom2mk to pom2bp that writes out Android.bp files instead
of Android.mk files.  pom2mk stays for now until the last users
of it are cleaned up.

Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.mk
Change-Id: I584d63c4228bad32f9e1914b06bde807078d6a55
2018-04-24 13:37:38 -07:00
Colin Cross 6b382bcd04 Add pom2mk -exclude
Add an option to pom2mk to exclude modules by name.

Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2mk -regen Android.mk -exclude androidx.car_car
Change-Id: I083907ef364384aace524ced81820567f5075d76
2018-04-23 16:16:52 -07:00
Colin Cross 9f5633d60c Add pom2mk -regen
pom2mk -regen Android.mk will read the previous arguments out of
Android.mk, rerun pom2mk with those arguments, and write the result
to Android.mk.

Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2mk -regen Android.mk
Change-Id: Ie8e0f97ae5b87c8078cf8e753489326d80da1896
2018-04-23 16:16:41 -07:00
Jeff Gaston 358f25e97d pom2mk to include extra-deps as top-level deps too
Specifically, the use case is android-support-car requires some stubs that don't have a Maven pom.
So those stubs can't easily be listed as a dependency in the pom.
However, we do want the dependency to exist when importing to Make and detected by extract_deps.py

Bug: 76692459
Test: m -j blueprint_tools \
      && update_prebuilts.py -s <build_id> \
      && git diff HEAD^ -- fix_dependencies.mk \
      && m -j checkbuild

Change-Id: If546ecf7416580a01bde7b906976ada4b1145253
2018-04-23 15:59:05 -04:00
Dan Willemsen 3fb1faeeb9 Use Config/DeviceConfig functions to access ProductVariables
An upcoming change will stop exporting ProductVariables from Config, so
switch to using existing accessor functions, and add more when they're
missing.

Bug: 76168832
Test: out/soong/build.ninja is identical
Change-Id: Ie0135bdbd2df3258ef3ddb53e5f8fc00aa9b97f7
2018-04-11 01:45:14 +00:00
Jeff Gaston af2191e201 Have pom2mk list all duplicates modules
For easier debugging if there are multiple duplicates

Bug: 76692459
Test: ./update_current.py -x <build-id>

Change-Id: I3079cb4ed1cfd365d792a7b41c9cdb4e7a1e148f
2018-03-29 17:44:36 -04:00
Jeff Gaston 8f08482957 Have pom2mk include runtime deps too
It still excludes test deps though

Bug: 76692459
Test: ./update_current.py -x <build-id>

Change-Id: I55a786e9feab31f99b95b7cebe84869757e46270
2018-03-29 17:44:33 -04:00
Colin Cross f46e37f5f7 Fix format string issues
Fix issues caught by go vet.

Test: m checkbuild
Change-Id: Ib8d740457c15432dabe1575a6707726ddaf93084
2018-03-28 15:54:52 -07:00
Alan Viverette 2b53a0c4b8 Always include JAR dependencies (not just with --static-deps) in Makefile
Bug: 76692459
Test: ./update_current.py -x <build-id>
Change-Id: Ie0f77e52089adbe153d668fe39902545f469f2d2
2018-03-28 14:32:10 -04:00
Colin Cross 628d55d7ef Make zipsync list file output empty if there are no files
The make javac rule uses [ -s srcjar-list ] to decide whether or
not to run javac.  zipsync was putting a trailing "\n" in the
list file, so the file was never empty.  There was one case of a
package that contained no source files (so java-source-list was
empty) and had a res directory with an xml file that declared no
resources (so aapt.jar was created but empty).  The 1-byte
srcjar-list file caused javac to run and error with:
javac: no source files

Test: m checkbuild
Change-Id: I26b394b66bf81e4f5abbd27e4dc06bee1d9420a8
2018-03-22 13:02:39 -07:00
Colin Cross 436b76564d Replace extract-srcjars.sh with zipsync tool
extract_srcjars.sh uses zipinfo and unzip, which fail with an
error on an empty zip file.  Instead of trying to hack around
this (which is hard to make guarantees for since they are
non-hermetic host tools), replace them with a go tool to unzip
a set of zip files into a directory.

Bug: 73885582
Test: m checkbuild
Change-Id: I151fed347ed5196726e36866ffc27bc831799afb
2018-03-15 16:42:43 -07:00
Nan Zhang 1db85407aa Remove timestamp based filelist file for tracking Python dependencies
Each Python module will generate a zip file containing source & data
files. The Python binary will collect all its dependencies and use
merge_zips to merge each zip file to create a final .par file.

Test: m -j checkbuild && real examples:
Bug: b/70568913

Change-Id: I9ff232d461d33e1c06026e7dcb5b124bf02c3ce5
2018-03-09 14:12:48 -08:00
Colin Cross 86803cfe6e add a library to report build numbers without causing rebuilds
Allow native modules to specify use_version_lib, which will make
an android::build::GetBuildNumber() function available.  For host
builds, the function will return the build number at the time that
the module was linked.  For device modules it will return the
value of the ro.build.version.incremental property.

Bug: 71719137
Test: build_version_test
Test: m build_version_test && touch build/make/core/Makefile build/soong/cc/libbuildversion/tests/build_version_test.cpp && m build_version_test shows different build numbers for binary and library tests.
Change-Id: I6f7d40b7574bb8206866c4e39bad9c710c796e32
2018-03-02 16:55:51 -08:00
Colin Cross 8673b5b959 Rewrite symbol_inject to be testable
Parse the machine-specific symbol tables into a custom symbol table
object and use a single function to find the offset and size of the
symbol in the custom table.  Make the functions to convert a
machine-specific symbol table into the custom table also take mocks,
and provide functions to dump mocks from real files.  Add tests
that take the mocks and verify they produce the right offset and
size.

Test: cmd/symbol_inject/*_test.go
Change-Id: I25654032b5017dd13a9a1fe29f8b8826ce5bc82b
2018-03-02 16:47:13 -08:00
Colin Cross dfce764476 Fix finding next symbol when multiple symbols have the same address
Some exe files have a .data symbol at the same address as the
soong_build_number symbol.  If the .data symbol is after
soong_build_number in the symbol list, symbol_inject would think
the end address was the same as the start address, and use
uint32(-1) as the size.

Use sort.Search to find the first symbol whose section number is
the same as the target symbol, but whose address is higher than
the target symbol.

Test: manual
Change-Id: I51d6e53c6b906222ba68c5cf93be944843e23550
2018-02-28 15:04:59 -08:00
Colin Cross c4a18e0291 Sort macho symbol table entries
macho symbol table entries are not always in order, which breaks
finding the next symbol to find the size of the target symbol.

Test: build_version_test
Change-Id: I41d1c3c3ff9929694e9ec2b034553d6b7ddef937
2018-02-23 22:45:55 -08:00