Commit Graph

36 Commits

Author SHA1 Message Date
Rupert Shuttleworth 3c9f5ac787 Allow Bazel to write to an external DIST_DIR (outside of OUT_DIR).
Also get Bazel to write real files there (not symlinks) so that the DIST_DIR can be independent.

Test: Manually using e.g. DIST_DIR=/tmp/foo USE_BAZEL=1 m dist
Change-Id: I39d5219500864c9ecc85f356a028e9b5bf2607f4
2020-12-11 01:17:58 +00:00
Colin Cross 00a8a3f746 Add --skip-soong-tests argument to soong_ui and use it in multiproduct_kati
There is no need for multiproduct_kati to run the tests for
every product, they don't vary by product config. --skip-soong-tests
can also be used for local development to run soong_build even if
the tests don't pass.

Bug: 156428456
Test: m --skip-soong-tests nothing
Change-Id: I9c00e3d1b6e51d17bb290339c3f124d4d1c9e69f
2020-10-29 14:22:04 -07:00
Colin Cross 405af07859 Revert "Make lots of tests run in parallel"
This reverts commit 323dc60712.

Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests

Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
2020-10-09 18:34:24 -07:00
Colin Cross 323dc60712 Make lots of tests run in parallel
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.

Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
2020-10-06 15:12:22 -07:00
Dan Willemsen 1bdbdec5a0 Take into account RAM usage for multiproduct_kati
Apparently we have some instances of Soong taking 16GB of RAM. While
that's a problem to solve on it's own, make multiproduct_kati's
auto-parallelism detection smarter by taking into account the total RAM
on the machine instead of just the number of CPUs.

Bug: 146925549
Test: check soong.log for autodetected parallelism values
Change-Id: Ice34c2cf73ea4f8f235521cbefc8654a79a04eef
2019-12-27 09:54:11 -08:00
Dan Willemsen 9609ad9c21 Add sharding support to multiproduct_kati
This is so that we can split the build_test build on large branches into
multiple builds, each testing a subset of the products.

Test: m blueprint_tools  (runs the go unit tests)
Test: build/soong/build_test.bash --only-config
      vs
Test: build/soong/build_test.bash --only-config --shard-count=2
Test: build/soong/build_test.bash --only-config --shard-count=2 --shard=2

Change-Id: I40ccc1aa477bc0ffa74ff564d155068509be18f0
2019-12-05 19:54:40 -08:00
Colin Cross c0b9f6b9d2 Force dumpvars into dumb terminal mode and move log files
Force soong_ui in dumpvars mode to use a dumb terminal so that it
doesn't print an empty status table.  Also move the logs to files
that are prefixed with "dumpvars-" so that they don't rotate the
real build logs.

Fixes: 139307849
Test: lunch
Change-Id: If92421af07731fedc9ca42670f309e95922a2224
2019-09-23 14:24:57 -07:00
Colin Cross e0df1a36b2 Move all output through StatusOutput
Write log output through StatusOutput so that the status implementation
can synchronize it with its own output.

Test: status_test.go
Change-Id: I917bdeeea4759a12b6b4aa6d6d86ee18a2771723
2019-06-12 21:30:56 -07:00
Colin Cross 097ed2a37c Remove terminal.Writer
terminal.Writer is now just a wrapper around stdio.Stdout() without
any useful functionality.  Replace it with stdio.Stdout() as an
io.Writer.

Test: ui/terminal/status_test.go
Change-Id: I5bc5476afdca950b505642f0135a3af9d37fbe24
2019-06-12 21:30:56 -07:00
Sasha Smundak c0c9ef9964 Provide an option to reduce build's verbosity.
If ANDROID_QUIET_BUILD environment variable is set to '1', do not show 20
lines of the environment variables. In addition, for the failing step show
only the step name and output, omitting the command proper (the verbose log
as well as error.log still contains it).
If build succeeds, the output of the build itself consists of a single
line:
```
> m androidmk
[100% NN/NN] <last command>
```

When it fails, the output does not contain sometimes very long command
line:
```
> m androidmk
[ 97% NN/MM] test androidmk
FAILED: <step>
--- FAIL: TestEndToEnd (0.01s)
    androidmk_test.go:1025: failed testcase 'prebuilt_etc_TARGET_OUT_ETC'
        input:
        include $(CLEAR_VARS)
        LOCAL_MODULE := etc.test1
        LOCAL_MODULE_CLASS := ETC
        LOCAL_MODULE_PATH := $(TARGET_OUT_ETC)/foo/bar
        include $(BUILD_PREBUILT)

        expected:
        prebuilt_etc {
            name: "etc.test1",
            filename: "foo/bar",
        }

        got:
        prebuilt_etc {
            name: "etc.test1",
            filename: "foo/bar",

        }

FAIL
17:50:53 ninja failed with: exit status 1
```
[The related change in build/make/envsetup.sh suppresses timing display
when the same variable is set.]

Change-Id: I4d3c72457de031ff58a324c2fe98f4c1d10f8239
Test: treehugger
2019-02-07 12:21:45 -08:00
Dan Willemsen f99915f569 Speed up build_test
Stop writing out ninja files, as they become very large, especially when
multiplied by the number of defined products.

Test: treehugger
Change-Id: Id9529d14040acb72a0188e58b5db2911f142071e
2018-10-29 10:43:37 -07:00
Dan Willemsen 2997123f73 Refactor Kati logic
Make the cleanspec & Android.mk runs of kati use much of the same code
and arguments.

Also renames 'Kati' to 'KatiBuild' in many cases, in the the expectation
that we'll have a 'KatiPackage' step in the future.

Use --no_ninja_prelude and move local_pool & _kati_always_build_ into
the combined ninja file. This will reduce the need to re-read makefiles
when Goma is enabled, and it allows us to include more than one
Kati-generated ninja file in the build graph.

Bug: 116968624
Test: build_test on downstream branches
Change-Id: Ibdac689b81f62dc293647fad917d84946f2c3cfa
2018-09-28 23:09:24 -07:00
Colin Cross 05518bc13b soong_zip: Add tests
Add test that cover basic command line usage of soong_zip.  -D
is not covered yet as the implementation will be replaced with
one that is also more easily testable in the next patch.

Bug: 116751500
Test: zip_test.go
Change-Id: I5a1bcee74ebc9cb3cf332c36f89bc12c0e807ad2
2018-09-28 13:56:06 -07:00
Dan Willemsen bcc1dbf957 Refactor multiproduct_kati
Move the actual per-product builds out of the main function, combining
the product configuration and build sections. This makes it a lot more
readable.

This ends up saving a lot of FDs -- a -only-config build on aosp with 77
products used to require ~500 FDs, it succeeds with a limit of 150 FDs
now. I'm leaving the code to bump our FD limit though, since 2x the
number of internal products is too close to the 1024 limit for comfort.

Bug: 70370883
Test: prlimit -n150:150 build/soong/build_test.bash -only-config
Change-Id: Ia559beadc19deb8a5b9d50af6e0276e846fd8608
2018-09-05 13:57:10 -07:00
Dan Willemsen 41538385e5 Add an incremental mode to multiproduct_kati / build_test
This uses a very large amount of intermediates, but can speed up
subsequent multiproduct_kati runs by not re-reading all the makefiles
unless necessary.

Since we'll no longer have the full build output on every run,
explicitly save away a std_full.log of the log from the last time we ran
Kati.

Test: build/soong/build_test.bash -products aosp_arm -incremental
Change-Id: Iaae580c0e49a2024528617d39b2b477e29d99250
2018-09-05 13:53:59 -07:00
Anton Hansson 43da16a440 Pass extra args from multiproduct_kati to config.
Allows specifying make goals when using multiproduct_kati, for example:
$ multiproduct_kati -only-config dump-products

This makes dumping products with multiproduct_kati easier than before.

Test: multiproduct_kati -only-config -products=full,aosp_arm64 -out=/tmp/p dump-products
Change-Id: If526bd2ba1b7bbadcb230c020202ef9e421e1852
2018-08-20 16:16:02 +01:00
Dan Willemsen 59339a29e1 Fix `go vet` issues
Test: go vet ./...
Change-Id: Ifb936ccc5e2b5a2c3fcbbbcb54f680e2973ea1b3
2018-07-22 21:18:45 -07:00
Dan Willemsen 34218ece56 Make multiproduct_kati return error
Test: insert error, run build/soong/build_test.sh and check $?
Change-Id: I3462defd1ff04247a8b729e22988cbe002b0ce69
2018-07-19 22:57:18 -07:00
Dan Willemsen b82471ad6d Add a unified status reporting UI
This adds a new status package that merges the running of "actions"
(ninja calls them edges) of multiple tools into one view of the current
state, and gives that to a number of different outputs.

For inputs:

Kati's output parser has been rewritten (and moved) to map onto the
StartAction/FinishAction API. A byproduct of this is that the build
servers should be able to extract errors from Kati better, since they
look like the errors that Ninja used to write.

Ninja is no longer directly connected to the terminal, but its output is
read via the protobuf frontend API, so it's just another tool whose
output becomes merged together.

multiproduct_kati loses its custom status routines, and uses the common
one instead.

For outputs:

The primary output is the ui/terminal.Status type, which along with
ui/terminal.Writer now controls everything about the terminal output.
Today, this doesn't really change any behaviors, but having all terminal
output going through here allows a more complicated (multi-line / full
window) status display in the future.

The tracer acts as an output of the status package, tracing all the
action start / finish events. This replaces reading the .ninja_log file,
so it now properly handles multiple output files from a single action.

A new rotated log file (out/error.log, or out/dist/logs/error.log) just
contains a description of all of the errors that happened during the
current build.

Another new compressed and rotated log file (out/verbose.log.gz, or
out/dist/logs/verbose.log.gz) contains the full verbose (showcommands)
log of every execution run by the build. Since this is now written on
every build, the showcommands argument is now ignored -- if you want to
get the commands run, look at the log file after the build.

Test: m
Test: <built-in tests>
Test: NINJA_ARGS="-t list" m
Test: check the build.trace.gz
Test: check the new log files
Change-Id: If1d8994890d43ef68f65aa10ddd8e6e06dc7013a
2018-07-12 14:15:31 -07:00
Dan Willemsen 04d76ef47d Fix multiproduct_kati product list
When the config loading was changed to using AndroidProducts.mk lists
from soong_ui instead of finding them itself, multiproduct_kati stopped
being able to find all the products. Move the finder initialization
earlier so that we can dump the full list.

Test: build/soong/build_test.bash -only-config
Test: forrest runs of every downstream build_test
Change-Id: I085d40f9008a47b850783499365977a0ff70ac66
2018-05-15 00:54:25 -07:00
Dan Willemsen 22de216799 Increase per-process file limits for multiproduct_kati
On large branches (250+ configurations), the open file count can go
over the default 1024 soft limit on Ubuntu. Many systems have increased
that default, but for the ones that haven't, at least opt into the
hard limit until this can be refactored to use fewer open files.

Bug: 70370883
Test: prlimit -n256:4096 build/soong/build_test.bash -only-config
Test: ulimit -Sn 256; build/soong/build_test.bash -only-config (darwin)
Change-Id: I7a952ffc89a0149ab65b04db1523d348daa6ab3e
2017-12-11 22:53:16 +00:00
Jeff Gaston b61e3f79ec multiproduct_kati support for a custom list of products
Test: multiproduct_kati --products="aosp_arm,aosp_arm64"
Bug: 67478260
Change-Id: Icf7b8b0c30a2dc8e093ab109172675ff74f43adc
2017-11-15 15:31:38 -08:00
Dan Willemsen f196396a1c Send soong output to std.log in multiproduct_kati
Instead of throwing it away (though it would also be in soong.log along
with the verbose messages).

Test: build/soong/build_test.bash with ctx.Println added
Change-Id: I64f8b11ab6752a6be8a5934fb41e30439238e331
2017-11-11 15:44:51 -08:00
Dan Willemsen e348076296 Change storage behavior of multiproduct_kati
Instead of deleting artifacts/logs from successful build (unless -keep
is set), and keeping unsuccessful artifacts, keep all logs and remove
all artifacts (unless -keep is set, then we'll compress the artifacts).

If -dist is passed in, we'll put an archive of the logs into the
DIST_DIR. Even compressed, the rest of the artifacts are still a bit too
large to dist (~5.6GB on AOSP).

Test: build/soong/build_test.bash
Test: build/soong/build_test.bash -keep
Test: build/soong/build_test.bash -dist
Change-Id: I87f55978c18c8ff2e517b8271554ba383003742f
2017-11-07 11:30:19 -08:00
Dan Willemsen b2e6c2e571 Switch product configs from make to ckati
This speeds up dumping make variables from ~380ms using make to ~220ms
using ckati. It also means that we're consistently using the same parser
for builds (with the same .KATI_READONLY/etc extensions).

envsetup.sh (lunch) / other scripts still use make, changing those to go
through soong_ui will be a future change.

Test: m clean; m nothing
Test: USE_GOMA=true m nothing
Test: m PRODUCT-aosp_x86-sdk
Test: m APP-Calculator
Test: build/soong/build_test.bash -only-config  (on AOSP and internal master)
Change-Id: I6ca554de8de4955fb869001d06d29969b75751cc
2017-10-12 18:11:08 -07:00
Dan Willemsen 9957b9ccdc multiproduct_kati: Support skipping known-broken products
This can make it easier to add build/soong/build_test.sh to continuous
build branches to catch new breaks, but where some products are known to
be broken.

Test: build/soong/build_test.bash --skip-products aosp_arm,aosp_arm64
Change-Id: Ib240570b68957077047ca75b78658429cb8c6a49
2017-10-06 15:05:05 -07:00
Jeff Gaston 743e29ee7c Fix Finder out path in multiproduct_kati
Bug: 64363847
Test: ./build/soong/build_test.bash --dist --keep
Change-Id: I70b08023b94b5789dc8394651bbac4c49c82787f
2017-08-17 14:17:14 -07:00
Jeff Gaston b64fc1cab5 Run the Finder and make its results available to Kati
The Finder runs roughly 200ms faster than findleaves.py in aosp,
and runs roughly 400ms faster in internal master.

Bug: 64363847
Test: m -j

Change-Id: I62db8dacc90871e913576fe2443021fb1749a483
2017-08-16 14:56:00 -07:00
Dan Willemsen f624fb92f6 Prepare multiproduct_kati to run on the build servers
Adds a -dist argument that will use DIST_DIR to save logs.

Also writes a summary of each std.log to stderr on errors, so that the
error is more likely to show up in the error reporting. This output is
prefixed with "> " to differentiate it from the progress reports from
multiproduct_kati itself.

Test: multiproduct_kati -only-config
Test: DIST_DIR=dist build/soong/build_test.bash -dist (introducing errors)
Change-Id: I5005b5f3f200c876bc004dd9b0e01e7b6edf5be2
2017-05-22 16:37:38 -07:00
Dan Willemsen 5ed900be48 Allow specifying a build variant
Test: multiproduct_kati --variant userdebug
Change-Id: I6548889caf868cad6270110c038753d029247572
2017-05-15 16:34:46 -07:00
Dan Willemsen a4e43a77d8 Improve multiproduct_kati output
It now uses the same output style as ninja, overwriting status lines in
smart terminals.

Test: multiproduct_kati
Test: multiproduct_kati | cat
Change-Id: I8db5198ffdc5ebc5503241ac492379753d92978e
2017-05-15 15:24:10 -07:00
Steven Moreland 552432e827 multiproduct_kati: better directory names.
Naming directories according to current date. This makes them easier to
parse when using the tool multiple time.

Also considered:
  millis := time.Now().UnixNano() / 1000000
  name := fmt.Sprintf("multiproduct-%d", millis)

Also considered putting separators in the number, but it makes for too
long of a directory name.

Test: multiproduct_kati
Change-Id: I10ac6754094653abf5bf11b04efc3c44905d3c8d
2017-03-29 22:38:42 -07:00
Dan Willemsen c38d366d78 Updates for the new ckati drop
We can start removing out directories again in multiproduct_kati, since
the opendir bug has been fixed.

Add --color_warnings to the Kati command line. Since this is different
from Make, take this opportunity to reorder the command line to make
more sense. This wasn't done before because kati forces a regen whenever
the command line changes.

Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I5ad03359fbc16db482722946202297c1ae0f2b90
2017-02-24 10:53:23 -08:00
Dan Willemsen 8a073a8b7b Ship the build log and trace on the build servers
Test: m -j; m -j dist
Change-Id: I4718e7a0cda6719bbab4243038ea2666c212cf19
2017-02-07 16:51:11 -08:00
Dan Willemsen d9f6fa28d6 Add build tracing
This creates a rotating build.trace.gz in the out directory that can be
loaded with chrome://tracing. It'll include start and end timings for
make/soong/kati/ninja, and it will import and time-correct the ninja log
files.

Test: m -j; load out/build.trace.gz in chrome://tracing
Test: multiproduct_kati -keep; load out/multiproduct*/build.trace.gz
Change-Id: Ic060fa9515eb88d95dbe16712479dae9dffcf626
2017-02-06 14:05:07 -08:00
Dan Willemsen c2af0bedc1 Add multiproduct_kati
This is a replacement for build/tools/kati_all_products.sh using the new
Soong ui/build package. It doesn't even attempt to run ninja, and it can
be configured to run only the product config, or only the product config
and Soong.

For AOSP on my machine:
 -only-config  1.4s
 -only-soong   1m20s
 <none>        13m

Test: multiproduct_kati
Change-Id: Ie3e6e7bdf692e46a8b8eb828f437190f8003500b
2017-02-06 14:05:07 -08:00