Allow callers of pom2bp to add dependencies on runtime libraries as
well as static libraries. Renames -extra-deps to --extra-static-libs
to make it clearer what it does.
Bug: 130287656
Test: prebuilts/sdk/update_prebuilts.py --stx 5280039
prebuilts/misc/common/androidx-test/update-from-gmaven.py
Change-Id: I8a3ee8056aa4ed9935a4f47d563ec423672cf38b
This will prevent it from being affected by future css location changes.
Test: m soong_docs
Fixes: 130567020
Change-Id: I80fd7b8a12a32bcea7b270423b8d8f8dc0976177
This essentially allows you to declare that everything in a directory
will be created by the rule, and we'll ensure that your command actually
writes out all of the claimed outputs, and remove any other files that
previously existed in that directory.
Test: built-in tests
Change-Id: I990dce2b3a0d89ebd2736ac1a0cadfb5864c6e73
The dependency type in the pom file is not always accurate, just
add all jar and aar dependencies to both the -nodeps and final
module.
Test: cd prebuilts/sdk/current/androidx && pom2bp -regen Android.bp && mma
Change-Id: I1784b8ebf3485db19405cb353c9bb5fb3ae837d8
android/soong/common was renamed to android/soong/android long
ago, but the pctx package path was still "android/soong/common".
This required all users of rules defined in android/soong/android
to import "android/soong/android" and then
pctx.Import("android/soong/common").
Test: m checkbuild
Change-Id: I20d096522760538f7cfc2bec3d4bfeba99b275d4
* changes:
Add a whitelist for services.art changing between builds
Add diff_target_files tool
Import files from compare_target_files for use in diff_target_files
Some tools used by genrules set the timestamp of the generated file
to an old value. For example, when using tar to extract files it
will set the timestamp of the extracted file to the value stored in
the tar file. This can cause unnecessary rebuilds due to output files
that are older than their inputs. Touch output files in sbox before
moving them into the output location.
Test: m checkbuild && m checkbuild
Change-Id: Idb0e71d2bf0ac3618338371132cc71be74bf9d9d
Add a tool that lists files that have changed between two builds by
comparing the target-files.zip artifacts.
Bug: 121158314
Test: unit tests
Change-Id: I3ca5dd1fdf652dffef923ad7307fd71e49e2e95a
Allow rules built with RuleBuilder to use depfiles. Ninja only
supports a single depfile with single output. If there are
multiple outputs in a rule, move all but the first to implicit
outputs. If multiple depfiles are specified, use new support
in dep_fixer to combine additional depfiles into the first depfile.
Test: rule_builder_test.go
Change-Id: I7dd4ba7fdf9feaf89b3dd2b7abb0e79006e06018
Bring property type strings to the start of property comments. This is
how source.android.com shows variables and paratemeters, and it is also
easier to read, especially when a comment ends with a code example.
Test: m soong_docs
Change-Id: Ic3e9db3a3711495998919ee95bbef6c7cb33b215
To add code format (<pre>...</pre>) support to module type texts.
Fixes: 124332708
Test: m soong_docs
Change-Id: Ibe9d3845a8f952fdef5007893d4af5836809ce1f
It appears that there's a bug in repo where when a linkfile is removed
from the manifest, the symlink remains in the source tree.
Cleaning this up needs to happen before we start scanning the source
tree for Android.bp/Android.mk files, so this has to be done very early
-- way before CleanSpec processing could handle it.
Bug: https://bugs.chromium.org/p/gerrit/issues/detail?id=10583
Test: ln -s missing hardware/qcom/sdm710/Android.bp; m
Change-Id: Ib68f2507ffe58ccdd9fbc88925f8a4f6150f2f7d
Follow the change to blueprint to make *Escape take and return a string
and add *EscapeList that take and return slices of strings. Fix up
a few places that were unnecessarily converting a string to a slice
and back to a string.
Test: m nothing
Change-Id: I3fa87de175522205f36544ef76aa2f04aef1b936
The href base added by I8a8ac0f9ba6c0d57ed83db2155955a95e070a265
doesn't really play nice with anchor links. This fixes the issue.
Fixes: 126550408
Test: m soong_docs
Change-Id: If62411a6391fd8151fa40cb2ecf32c5006797d6c
This adds a script to copy the current page's base url to hrefs in the
doc.
Fixes: 123473548
Test: m soong_docs
Change-Id: I8a8ac0f9ba6c0d57ed83db2155955a95e070a265
Previously the build doc was a gigantic list of modules and properties,
which can be overwhelming to new users. This change breaks it up by
packages, so that it is easier to look up and feels more coherent.
Bug: 123521276
Test: m soong_docs
Change-Id: I1a1a331f4b4deb8782d698e9076098c5a27b9566
Uses more python rather than C++, and skips less of Py_Main.
Test: build/soong/python/tests/runtests.sh
Change-Id: I03997d88e2e16047c96bb4e00e530229c42b3325
It now has module list on the left and the main panel containing
a section for each module. Each section contains the navigable list
of the module's attributes, grouped by attribute's origin.
Test: Manual
Bug: 120512870
Change-Id: Ib62ef58c61daa88950f0934a70ba8ed16f433bf2
bootstrap.ModuleTypeDocs needs a mapping of module types to factories
to support factories that are wrapped in ModuleFactoryAdapter closures.
It also returns ModuleType objects grouped into Package objects.
Bug: 67909957
Test: m soong_docs
Change-Id: I70eac9f0f0e13075580da92d4219792ca0b18fbf
If ANDROID_QUIET_BUILD environment variable is set to '1', do not show 20
lines of the environment variables. In addition, for the failing step show
only the step name and output, omitting the command proper (the verbose log
as well as error.log still contains it).
If build succeeds, the output of the build itself consists of a single
line:
```
> m androidmk
[100% NN/NN] <last command>
```
When it fails, the output does not contain sometimes very long command
line:
```
> m androidmk
[ 97% NN/MM] test androidmk
FAILED: <step>
--- FAIL: TestEndToEnd (0.01s)
androidmk_test.go:1025: failed testcase 'prebuilt_etc_TARGET_OUT_ETC'
input:
include $(CLEAR_VARS)
LOCAL_MODULE := etc.test1
LOCAL_MODULE_CLASS := ETC
LOCAL_MODULE_PATH := $(TARGET_OUT_ETC)/foo/bar
include $(BUILD_PREBUILT)
expected:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
got:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
FAIL
17:50:53 ninja failed with: exit status 1
```
[The related change in build/make/envsetup.sh suppresses timing display
when the same variable is set.]
Change-Id: I4d3c72457de031ff58a324c2fe98f4c1d10f8239
Test: treehugger
Test: Dumped the text formated based metrics file to out dir,
and checked the file.
Bug: b/63815990
Change-Id: Iff476f72a0be74eb53b6b26ef468d11c0f24a404
The zip file format does support scripts/programs prefixed to the
archive, though many of the offsets are supposed to start from the
beginning of the file. Some tools (python and zipinfo) are able to
read zip files with arbitrary prefixes, but others (libziparchive and
zipdetails) fail.
So pass the file to prefix onto the zip file into merge_zips so that we
can set the offsets from the real start of the file.
Test: m sepolicy_tests (runs the embedded python interpreter)
Test: zipinfo out/host/linux-x86/bin/sepolicy_tests
Test: zipdetails out/host/linux-x86/bin/sepolicy_tests
Change-Id: If73d4c2465581f7de5aa47959284ecf2059df091
Gold uses two PT_LOAD programs: RX, RW
LLD is using three PT_LOAD programs: R, RX, RW
All non-static host bionic executables are linked with LLD, as it's
required in order to embed the linker. The dynamic linker is considered
a static executable for this definition.
When the linker had been built with Gold, we inserted the the RX&RW
loads in the middle of the executable's readonly load, splitting the R
load into two. When this happened, all of the linker's loads existed 1:1
in the final executable.
Now that the linker is built with LLD, we're inserting the loads into
the same place, but LLD merges the first part of the executable's R load
with the linker's R load, so there's not a 1:1 matching.
So just make sure that all of the linker's loads are contained within
the executable's loads (and with the appropriate flags).
Test: m blueprint_tools (runs new tests)
Test: build host bionic binaries
Change-Id: If7868d2ab8980d27ba5ab8994fefe0d6142f215a
Patterns containing multiple globs or a recurisve glob may match
files with the same name in multiple directories. Keep the relative
directories of matches after the path entry containing a glob.
Bug: 117295826
Test: zip2zip_test.go
Change-Id: I5d663e546953af374175837551d23f484d568377
Stop writing out ninja files, as they become very large, especially when
multiplied by the number of defined products.
Test: treehugger
Change-Id: Id9529d14040acb72a0188e58b5db2911f142071e
In order to simplify the wrapper function, and stop using a linker
script, generate a set of flags to pass to LLD. Then run
host_bionic_inject on the linked binary in order to verify the
embedding, and give the wrapper function the address of the original
entry point (_start).
Bug: 31559095
Test: build host bionic with prebuilts/build-tools/build-prebuilts.sh
Change-Id: I53e326050e0f9caa562c6cf6f76c4d0337bb6faf
Don't warn on duplicate files in merge_zips if they have identical
CRC32 and size values.
Test: m checkbuild
Test: merge_zips_test.go
Change-Id: I61336ca4d4d3b7402c24a7abd337bd350fe10930
Make needs to exclude all files that match a filespec except for
those specified in a list. Add a -X argument that overrides the
-x argument to allow files be included that were previously
excluded.
Bug: 69500920
Test: zip2zip_test.go
Change-Id: Icc8aebc788b53bcdde73a39c9c7cf3107a049251
Add -0 flag to convert files in a zip to stored instead of deflated.
Bug: 69500920
Test: zip2zip_test.go
Change-Id: I6c2b10f3b200a53a3339e3c97a78f65192b309ca
Make the cleanspec & Android.mk runs of kati use much of the same code
and arguments.
Also renames 'Kati' to 'KatiBuild' in many cases, in the the expectation
that we'll have a 'KatiPackage' step in the future.
Use --no_ninja_prelude and move local_pool & _kati_always_build_ into
the combined ninja file. This will reduce the need to re-read makefiles
when Goma is enabled, and it allows us to include more than one
Kati-generated ninja file in the build graph.
Bug: 116968624
Test: build_test on downstream branches
Change-Id: Ibdac689b81f62dc293647fad917d84946f2c3cfa
Add test that cover basic command line usage of soong_zip. -D
is not covered yet as the implementation will be replaced with
one that is also more easily testable in the next patch.
Bug: 116751500
Test: zip_test.go
Change-Id: I5a1bcee74ebc9cb3cf332c36f89bc12c0e807ad2
Move the actual per-product builds out of the main function, combining
the product configuration and build sections. This makes it a lot more
readable.
This ends up saving a lot of FDs -- a -only-config build on aosp with 77
products used to require ~500 FDs, it succeeds with a limit of 150 FDs
now. I'm leaving the code to bump our FD limit though, since 2x the
number of internal products is too close to the 1024 limit for comfort.
Bug: 70370883
Test: prlimit -n150:150 build/soong/build_test.bash -only-config
Change-Id: Ia559beadc19deb8a5b9d50af6e0276e846fd8608
This uses a very large amount of intermediates, but can speed up
subsequent multiproduct_kati runs by not re-reading all the makefiles
unless necessary.
Since we'll no longer have the full build output on every run,
explicitly save away a std_full.log of the log from the last time we ran
Kati.
Test: build/soong/build_test.bash -products aosp_arm -incremental
Change-Id: Iaae580c0e49a2024528617d39b2b477e29d99250
Move some of the more complicated conditional logic out to helper
functions. Use the {{- }} syntax to strip previous whitespace to
allow spacing out the remaning conditionals in the templates.
Test: (cd prebuilts/sdk/current/androidx && pom2bp -regen Android.bp)
Change-Id: I766bd0e1837aa04375f322fbe796d923cd99ecde
Allows specifying make goals when using multiproduct_kati, for example:
$ multiproduct_kati -only-config dump-products
This makes dumping products with multiproduct_kati easier than before.
Test: multiproduct_kati -only-config -products=full,aosp_arm64 -out=/tmp/p dump-products
Change-Id: If526bd2ba1b7bbadcb230c020202ef9e421e1852
merge_zips -stripFile was only considering the name of the file and
ignoring the path. Make it more useful by supporting blueprint style
globs. The previous behavior can be recreated by prefixing with **/.
Bug: 111389216
Test: m checkbuild
Change-Id: I25760fe3f1f77704dd9da9d107d9a38a415d681f
This adds a new status package that merges the running of "actions"
(ninja calls them edges) of multiple tools into one view of the current
state, and gives that to a number of different outputs.
For inputs:
Kati's output parser has been rewritten (and moved) to map onto the
StartAction/FinishAction API. A byproduct of this is that the build
servers should be able to extract errors from Kati better, since they
look like the errors that Ninja used to write.
Ninja is no longer directly connected to the terminal, but its output is
read via the protobuf frontend API, so it's just another tool whose
output becomes merged together.
multiproduct_kati loses its custom status routines, and uses the common
one instead.
For outputs:
The primary output is the ui/terminal.Status type, which along with
ui/terminal.Writer now controls everything about the terminal output.
Today, this doesn't really change any behaviors, but having all terminal
output going through here allows a more complicated (multi-line / full
window) status display in the future.
The tracer acts as an output of the status package, tracing all the
action start / finish events. This replaces reading the .ninja_log file,
so it now properly handles multiple output files from a single action.
A new rotated log file (out/error.log, or out/dist/logs/error.log) just
contains a description of all of the errors that happened during the
current build.
Another new compressed and rotated log file (out/verbose.log.gz, or
out/dist/logs/verbose.log.gz) contains the full verbose (showcommands)
log of every execution run by the build. Since this is now written on
every build, the showcommands argument is now ignored -- if you want to
get the commands run, look at the log file after the build.
Test: m
Test: <built-in tests>
Test: NINJA_ARGS="-t list" m
Test: check the build.trace.gz
Test: check the new log files
Change-Id: If1d8994890d43ef68f65aa10ddd8e6e06dc7013a
manifest property should only be added to aar imports and not jar
imports.
Bug: 110848854
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.bp
Change-Id: Ifaab81582c87f0af4419f90c3928d0d5d52cec33
When manifest merging in Soong is turned on the modules created
by pom2bp will need to have min_sdk_version specified in order
to prevent manifest_fixer from setting it to the current SDK
level. Read the minSdkVersion attribute out of the
AndroidManifest.xml file inside each .aar file.
Bug: 110848854
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.bp
Change-Id: Ieaf37ba6eccaf32dc5d3411566335830b1a327ff
This reverts commit 09f4540d66.
Fixes the raw call to net.Listen in the tests to go through the listen()
helper and use the long socket path fallbacks.
Removes the use of timeouts from the tests -- the behaviors being tested
did not rely on timeouts, so removing them will reduce the flakiness if
the build is heavily loading the machine at the same time the test is
running.
Also fixes some potential nil pointer dereferences.
Test: OUT_DIR=<really long> m blueprint_tools
Test: `while .../soong-ui-build-paths/test/test; do sleep 0.01; done` with a build running
Change-Id: I16d44be7517bc415f1c808284088f4ba40df3bfa
This reverts commit c59a92cb1a.
Reason for revert: tests are broken with long OUT_DIRs
They're directly calling net.Listen, and not using the fallback
for long socket names.
Change-Id: Id14cbd499fd9b36c6926b7552d3554340cb0916c
This reverts commit 96c957ae20.
Fixes issues on some machines where the socket in TMPDIR ended up with a
unix domain socket pathname over 107 characters long, which Go will
reject due to underlying limitations in the system calls. If this
happens, we'll fall back to opening the directory, then using
/proc/self/fd/#/<file>, or manually creating a similar symlink in /tmp.
Also fixes some issues on Mac where os.Executable returns the symlink
instead of the underlying file, sending a message over a unix domain
socket will block if the reader isn't reading, and sandboxing was
preventing us from running `ps`.
Test: m blueprint_tools
Test: m blueprint_tools on mac
Change-Id: Ib19ccfe10cb0a79f1476fb1d5cd20ed0495be367
This will allow us to track (and eventually limit) the commands that the
build references via $PATH. These are mostly implicit dependencies on
the host system -- for Linux, we assume something similar to Ubuntu
14.04 with a few extra packages, but this will let us better define
that.
This will not catch uses of tools with absolute paths (/bin/bash, etc),
but most uses shouldn't be relying on absolute path names anyways.
Adds ~400ms on the first startup, ~140ms on subsequent runs, and
overhead of a few ms for every forwarded execution.
Test: m
Test: build/soong/build_test.bash
Test: Add `gcc --version`, TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: Id68cbb1c8ceef65bbbb10751e83722c7662d2351
When the config loading was changed to using AndroidProducts.mk lists
from soong_ui instead of finding them itself, multiproduct_kati stopped
being able to find all the products. Move the finder initialization
earlier so that we can dump the full list.
Test: build/soong/build_test.bash -only-config
Test: forrest runs of every downstream build_test
Change-Id: I085d40f9008a47b850783499365977a0ff70ac66
protoc dependency files, at least for C++ outputs, uses the form of:
a/b.c \
a/b.h: <dep1> <dep2>...
Ninja will fail the command when it parses a dep file and there's more
than one output file (even though it doesn't care what the output file
name is). So this tool will parse the original file, and output a
version with only a single output file.
Bug: 67329638
Test: NINJA_ARGS="-t deps ...pb.c" m
Test: NINJA_ARGS="-t deps ...srcjar" m
Test: NINJA_ARGS="-t deps ...srcszip" m
Test: Run dep_fixer across all of taimen's dep files, no failures.
Test: Run dep_fixer against the processed files, no changes.
Test: Run androidmk across all of our Android.mk files, inspect the diffs
Change-Id: I4263b7d5faea37285afa6b24dedf5964aa7d19dc
If a static android library lib1 has static_libs: ["lib2"] then the
R.class files for packages in lib2 will be merged into the jar for
lib1. If an app has lib1 in its static_libs it will get the R.class
files from lib2 through lib1, instead of regenerating the R.java
files with numbering that matches the resource table of the app.
Pass transtive static android library dependencies on the aapt2
command line so that aapt2 will always regenerate the R.java
constants for those packages.
Also extract the packages that have R.java files after each aapt2
invocation. This is not necessary for Soong, but is passed to
make to let it force regenerating the packages using
--extra-packages.
Bug: 78300023
Test: m checkbuild
Change-Id: I0f3444af44d2a9f370d1f156c908972f8cc3a1ee
Convert pom2mk to pom2bp that writes out Android.bp files instead
of Android.mk files. pom2mk stays for now until the last users
of it are cleaned up.
Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.mk
Change-Id: I584d63c4228bad32f9e1914b06bde807078d6a55
Add an option to pom2mk to exclude modules by name.
Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2mk -regen Android.mk -exclude androidx.car_car
Change-Id: I083907ef364384aace524ced81820567f5075d76
pom2mk -regen Android.mk will read the previous arguments out of
Android.mk, rerun pom2mk with those arguments, and write the result
to Android.mk.
Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2mk -regen Android.mk
Change-Id: Ie8e0f97ae5b87c8078cf8e753489326d80da1896
Specifically, the use case is android-support-car requires some stubs that don't have a Maven pom.
So those stubs can't easily be listed as a dependency in the pom.
However, we do want the dependency to exist when importing to Make and detected by extract_deps.py
Bug: 76692459
Test: m -j blueprint_tools \
&& update_prebuilts.py -s <build_id> \
&& git diff HEAD^ -- fix_dependencies.mk \
&& m -j checkbuild
Change-Id: If546ecf7416580a01bde7b906976ada4b1145253
An upcoming change will stop exporting ProductVariables from Config, so
switch to using existing accessor functions, and add more when they're
missing.
Bug: 76168832
Test: out/soong/build.ninja is identical
Change-Id: Ie0135bdbd2df3258ef3ddb53e5f8fc00aa9b97f7
For easier debugging if there are multiple duplicates
Bug: 76692459
Test: ./update_current.py -x <build-id>
Change-Id: I3079cb4ed1cfd365d792a7b41c9cdb4e7a1e148f
The make javac rule uses [ -s srcjar-list ] to decide whether or
not to run javac. zipsync was putting a trailing "\n" in the
list file, so the file was never empty. There was one case of a
package that contained no source files (so java-source-list was
empty) and had a res directory with an xml file that declared no
resources (so aapt.jar was created but empty). The 1-byte
srcjar-list file caused javac to run and error with:
javac: no source files
Test: m checkbuild
Change-Id: I26b394b66bf81e4f5abbd27e4dc06bee1d9420a8
extract_srcjars.sh uses zipinfo and unzip, which fail with an
error on an empty zip file. Instead of trying to hack around
this (which is hard to make guarantees for since they are
non-hermetic host tools), replace them with a go tool to unzip
a set of zip files into a directory.
Bug: 73885582
Test: m checkbuild
Change-Id: I151fed347ed5196726e36866ffc27bc831799afb
Each Python module will generate a zip file containing source & data
files. The Python binary will collect all its dependencies and use
merge_zips to merge each zip file to create a final .par file.
Test: m -j checkbuild && real examples:
Bug: b/70568913
Change-Id: I9ff232d461d33e1c06026e7dcb5b124bf02c3ce5
Allow native modules to specify use_version_lib, which will make
an android::build::GetBuildNumber() function available. For host
builds, the function will return the build number at the time that
the module was linked. For device modules it will return the
value of the ro.build.version.incremental property.
Bug: 71719137
Test: build_version_test
Test: m build_version_test && touch build/make/core/Makefile build/soong/cc/libbuildversion/tests/build_version_test.cpp && m build_version_test shows different build numbers for binary and library tests.
Change-Id: I6f7d40b7574bb8206866c4e39bad9c710c796e32
Parse the machine-specific symbol tables into a custom symbol table
object and use a single function to find the offset and size of the
symbol in the custom table. Make the functions to convert a
machine-specific symbol table into the custom table also take mocks,
and provide functions to dump mocks from real files. Add tests
that take the mocks and verify they produce the right offset and
size.
Test: cmd/symbol_inject/*_test.go
Change-Id: I25654032b5017dd13a9a1fe29f8b8826ce5bc82b
Some exe files have a .data symbol at the same address as the
soong_build_number symbol. If the .data symbol is after
soong_build_number in the symbol list, symbol_inject would think
the end address was the same as the start address, and use
uint32(-1) as the size.
Use sort.Search to find the first symbol whose section number is
the same as the target symbol, but whose address is higher than
the target symbol.
Test: manual
Change-Id: I51d6e53c6b906222ba68c5cf93be944843e23550
macho symbol table entries are not always in order, which breaks
finding the next symbol to find the size of the target symbol.
Test: build_version_test
Change-Id: I41d1c3c3ff9929694e9ec2b034553d6b7ddef937
The path to the output directory may be arbitrarily long, use
__SBOX_OUT_DIR__ in the list of output files passed to sbox
to avoid expanding it multiple times in the command line.
Fixes:
ninja: fatal: posix_spawn: Argument list too long
09:40:14 ninja failed with: exit status 1
when building libchrome with a long OUT or OUT_DIR_COMMON_BASE.
Bug: 73726635
Test: m checkbuild
Change-Id: I59024b2164287c8e531711afd9273b692ce9c28a
Only include dependencies for scope "compile."
Bug: 73263586
Test: ./update_current.py -s -t <build-id>
Change-Id: I2ac6055cb4c1ad1f8d7924869f54f50e5e662742
Move __init__.py files population to merge_zips, and add options: -p, and
-e. Since par needs entry_point.txt to search the program entry point, so
add -e option to support it.
Test: ./merge_zips -p -e entry_point.txt out in1.zip in2.zip
Bug: 70568913
Change-Id: Ida10faf125e1b94dffc98b30240db3b90fd75b25
Add a rule in soong that re-executes soong_build in order to
generate build documentation. This allows Soong to customize
the documentation.
Bug: 70516282
Test: m soong_docs
Change-Id: If143cfacd6ac20274cd7bb8d8fab0c07025a5553
On large branches (250+ configurations), the open file count can go
over the default 1024 soft limit on Ubuntu. Many systems have increased
that default, but for the ones that haven't, at least opt into the
hard limit until this can be refactored to use fewer open files.
Bug: 70370883
Test: prlimit -n256:4096 build/soong/build_test.bash -only-config
Test: ulimit -Sn 256; build/soong/build_test.bash -only-config (darwin)
Change-Id: I7a952ffc89a0149ab65b04db1523d348daa6ab3e
This mostly reverts commit 178d5fefc0
and mostly reapplies change I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c .
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
--products=aosp_arm \
'build/blueprint:work^ build/soong:work^' \
'build/blueprint:work build/soong:work'
# and see that the only changes were:
# 1. adding some new files
# 2. changing some line numbers
Test: m -j nothing # which runs unit tests
Change-Id: I32baae00277a547fdcdd1c2219fe6625ee0e45d7
which should make debugging faster.
Bug: 66921582
Test: ./out/soong/host/linux-x86/bin/sbox --output-root outs --sandbox-path out/.temp/sbox-work -c "cd __SBOX_OUT_DIR__ && mkdir asubdir && mkdir b && touch asubdir/child a b c d e f g h i j k l m n o p" outs/a outs/b outs/z # and observe the below output:
mismatch between declared and actual outputs
in sbox command(cd out/.temp/sbox-work/sbox343858828 && mkdir asubdir && mkdir b && touch asubdir/child a b c d e f g h i j k l m n o p)
in sandbox out/.temp/sbox-work/sbox343858828,
failed to create 2 files:
b: not a file
z: does not exist
did create 16 files:
a
asubdir/child
c
d
e
f
g
h
i
j
...6 more
Change-Id: I75e37834c44d4279dec874701d67ce8bb01b872c
This reverts commit 63a250a336.
Reason for revert: Some failures:
namespace_test.go:648: dir1/Blueprints:2:4: a namespace must be the first module in the file
such as New Build Breakage: aosp-master/build_test @ 4475274
Change-Id: I1b5db8eb934e51ff22241bfca44199d886b1393b
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
--products=aosp_arm \
'build/blueprint:work^ build/soong:work^' \
'build/blueprint:work build/soong:work'
# and see that the only changes were:
# 1. adding some new files
# 2. changing some line numbers
Test: m -j nothing # which runs unit tests
Change-Id: I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c
This reverts commit fb941913a3.
Reapplies I4933187e8b72f2ef0c32d18ffea756e2c6fa417c with fixes
to disable the check for mac builds, where many modules are
disabled and the check is just going to cause more problems.
Test: m checkbuild
Change-Id: If6712c90ececd5d015fcdcdeefe0c3d4f5590711
Add a --ignore_duplicates option to cause merge_zips to
silently take the first entry in the case of duplicate entries
in the zips to be merged. This will be used for soong Jacoco
support to combine the instrumented jar with the original jar
to pick up any classes that were excluded.
Bug: 69629238
Test: m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true SKIP_BOOT_JARS_CHECK=true WITH_DEXPREOPT=false
Change-Id: I49ede3f07ce0ed7701b4db7058da2e4f11b8043e
Jacoco support will use zip2zip to create a jar that is a subset
of another jar, and will need exclusion filters and recursive
globs. Switch zip2zip from filepath.Match to pathtools.Match,
and check each included file against the exclusion list.
Bug: 69629238
Test: zip2zip_test.go
Change-Id: Ibe961b0775987f52f1efa357e1201c3ebb81ca9c
Get a list of leaf nodes in the dependency graph
from ninja, and make sure none of them are in the
output directory. This ensures that there are no
rules that depend on a file in the output directory
that doesn't have rule to generate it. The check
will catch a common set of build failures where
a rule to generate a file is deleted (either by
deleting a module in an Android.mk file, or by
modifying the build system incorrectly). These
failures are often not caught by a local incremental
build because the previously built files are still
present in the output directory.
Bug: 36843214
Bug: 68062417
Test: manual
Change-Id: I4933187e8b72f2ef0c32d18ffea756e2c6fa417c
Instead of throwing it away (though it would also be in soong.log along
with the verbose messages).
Test: build/soong/build_test.bash with ctx.Println added
Change-Id: I64f8b11ab6752a6be8a5934fb41e30439238e331
For unbundled builds, LOCAL_SDK_VERSION needs to be set, otherwise we'll
try to use frameworks-res, which isn't available.
Bug: 64723465
Test: Use with prebuilts/sdk/update_current.py
Change-Id: If5e1233b4bcb09b3d0e2251e8dd1a8ecfc87eb19
Instead of deleting artifacts/logs from successful build (unless -keep
is set), and keeping unsuccessful artifacts, keep all logs and remove
all artifacts (unless -keep is set, then we'll compress the artifacts).
If -dist is passed in, we'll put an archive of the logs into the
DIST_DIR. Even compressed, the rest of the artifacts are still a bit too
large to dist (~5.6GB on AOSP).
Test: build/soong/build_test.bash
Test: build/soong/build_test.bash -keep
Test: build/soong/build_test.bash -dist
Change-Id: I87f55978c18c8ff2e517b8271554ba383003742f
Android Support Library 26+ does not specify dependency types, so parse
all pom files first, then propagate missing dependency types if we've
got the information.
Bug: 64723465
Test: cd prebuilts/maven_repo/android; pom2mk -use-version 26.0.0-beta2 com/android/support
Change-Id: Ieaff757ff198c9a7b4b006623340b382728c1fd4
Also removeTempDir -> !keepOutDir
Also shorten the help comment
Bug: 68336760
Test: rm out -rf && m -j
Test: sbox -c # and observe that the output does not have a stacktrace
Change-Id: I6fbb385132128ab1273c8398ff2425191049751e
For repositories that contain multiple versions of the same artifact,
we'd only like to create makefiles for one of the versions of the
artifact.
Bug: 64723465
Test: cd prebuilts/maven_repo/android; pom2mk -use-version 25.3.1 com/android/support
Change-Id: I5bdbba8d2143a47610d56d679381eb01b3cf176d
Most notably, the sandbox depfile path should be passed into the tool.
Bug: 68336760
Test: m -j checkbuild
Change-Id: I22f944a3f57d613fda26de0ea777a915cafcd020
ALLOW_MISSING_DEPENDENCIES=true can cause missing dependencies to be
hidden from GenerateAndroidBuildActions, which is triggering the
len(jars) == 0 check in compileJavaHeader. For the
ALLOW_MISSING_DEPENDENCIES=true case the check is unnecessary, as the
build rules will be replaced with runtime errors. On the off chance
the check would have been hit in a real build, allow merge_zips to
create an empty zip file if it is asked to.
Test: m ALLOW_MISSING_DEPENDENCIES=true
Change-Id: I3171f921d51229ddf38a1a647d32566658c673fa
to make it faster/easier to invoke from other Go programs
(such as multiproduct_kati)
Bug: 67478260
Test: m -j
Change-Id: Idd2671a44290550197c88f53dd11a6dd39c85cc5
javac warns whenever -source is used with a version that does not
match javac, because it wants an rt.jar provided for the given
source level. We are not using -source to prevent usage of APIs
that don't exist in the newer rt.jar, so hide the messages.
Test: javac_wrapper_test.go
Change-Id: Ic94b61767436848fc164e377dc3cc5a63726f1cf
build/soong/soong_ui.bash --dumpvars-mode \
--vars="..." \
--abs-vars="..." \
--var-prefix="..." \
--abs-var-prefix="..."
is similar to the previous:
CALLED_FROM_SETUP=true BUILD_SYSTEM=build/core \
make -f build/core/config.mk dump-many-vars \
DUMP_MANY_VARS="..." \
DUMP_MANY_ABS_VARS="..." \
DUMP_VAR_PREFIX="..." \
DUMP_ABS_VAR_PREFIX="..."
and
build/soong/soong_ui.bash --dumpvar-mode [--abs] VAR
is similar to the previous:
CALLED_FROM_SETUP=true BUILD_SYSTEM=build/core \
make -f build/core/config.mk dumpvar-[abs-]-VAR
But uses soong_ui and ckati, so that we use a consistent make parser and
sandboxing configurations.
One major output difference between the pure make implementation and
this one is that report_config in Go is implemented using embedded
newlines in single quotes, while the make implementation uses `` with
embedded echo commands. This seems to work fine for both bash and zsh,
and report_config isn't meant to be machine-parsed anyways.
Test: build/soong/soong_ui.bash --dumpvar-mode report_config
Test: build/soong/soong_ui.bash --dumpvar-mode TARGET_DEVICE
Test: build/soong/soong_ui.bash --dumpvar-mode --abs PRODUCT_OUT
Test: build/soong/soong_ui.bash --dumpvar-mode --abs ALL_PRODUCTS
Test: build/soong/soong_ui.bash --dumpvars-mode --vars="report_config TARGET_DEVICE" --abs-vars="ALL_PRODUCTS"
Test: build/soong/soong_ui.bash --dumpvars-mode --vars=TARGET_DEVICE --abs-vars=PRODUCT_OUT --var-prefix=v_ --abs-var-prefix=a_
Change-Id: I0fbd0732bbf6fcfcd24084cf3c830a91a4b6bfc2
This speeds up dumping make variables from ~380ms using make to ~220ms
using ckati. It also means that we're consistently using the same parser
for builds (with the same .KATI_READONLY/etc extensions).
envsetup.sh (lunch) / other scripts still use make, changing those to go
through soong_ui will be a future change.
Test: m clean; m nothing
Test: USE_GOMA=true m nothing
Test: m PRODUCT-aosp_x86-sdk
Test: m APP-Calculator
Test: build/soong/build_test.bash -only-config (on AOSP and internal master)
Change-Id: I6ca554de8de4955fb869001d06d29969b75751cc
This can make it easier to add build/soong/build_test.sh to continuous
build branches to catch new breaks, but where some products are known to
be broken.
Test: build/soong/build_test.bash --skip-products aosp_arm,aosp_arm64
Change-Id: Ib240570b68957077047ca75b78658429cb8c6a49
Putting resources in dex jars is going to need to pull files from
classpath jars that do not match *.class.
Test: m -j checkbuild
Change-Id: Ia37b8b9387e5f5636769afc6937b9aa3c9ec1ced
* changes:
Extract the linker and embed it into host bionic binaries
Add prefix_symbols support to cc_object
Fix ToolchainClangCflags for host bionic
Don't add a rpath to the linker
The linux kernel requires that the ELF interpreter (runtime linker)
that's referenced by PT_INTERP be either an absolute path, or a relative
path from the current working directory. We'd prefer a relative path
from the binary, similarly to how we handle looking up shared libraries,
but that's not supported.
Instead, extract the load sections from the runtime linker ELF binary
and embed them into each host bionic binary, omitting the PT_INTERP
declaration. The kernel will treat it as a static binary, and we'll use
a special entry point (linker_wrapper) to fix up the arguments passed by
the kernel before jumping to the embedded linker. From the linker's
point of view, it looks like the kernel loaded the linker like normal.
Bug: 31559095
Test: Enable host bionic,
out/soong/host/linux_bionic-x86/nativetest64/libdemangle_test/libdemangle_test
Change-Id: I8d0aea9790b5e86fcc3ea6e2d00cfa33907e2853
Jar always puts default MANIFEST.MF files in if none was specified.
Copying that behavior in soong_zip causes problems with merge_zips,
because it ends up taking the default manifest from the classes.jar
instead of the user's manifest from res.jar. We don't want the
user's manifest in the classes.jar, otherwise a change to the
manifest will cause all the class files to rebuild. Instead,
move the manifest insertion to the final merge_zips stage.
Test: m -j checkbuild
Change-Id: Id6376961dbaf743c2fb92843f9bdf2e44b963be0
In some cases, we need delete META-INF dir to void mis-using some
classes files in the Java runtime.
For now, when we delete META-INF, we need keep Manifest file in the jar
since merge_zips doesn't have ability to add a default one.
Bug: b/65455145
Test: m clean && m -j
Change-Id: Iea44b1a98d348cd8df1fa7374907733b1add6935
Also filter out META-INF/TRANSITIVE dir, and report warnings when
merge_zips see duplicates entries with different CRC hash.
Bug: b/65455145
Test: m clean && m -j java (locally)
Change-Id: I47172ffa27df71f3280f35f6b540a7b5a0c14550
Allow filename collisions for directories, which may happen if
multiple globs of resources include the same directory names.
Continue to report errors if collisions occur between files, and
also add checks for collisions between files and directories.
Test: manual
Change-Id: Iac19fbd325c53fbb41552ea230d813c8dddf9439
Combining the results of a javac command into a jar needs to take
a whole directory, add -D to specify a directory to search for all
files.
Test: m -j checkbuild
Change-Id: Id4707189d57b664f0262a0782630f893af397900
This will enable writing a modified manifest without having to create a temporary file first
Bug: 64536066
Test: soong_zip --jar -o /tmp/out.zip -C . -l files.list && \
# make sure that the output is binary equal
# with and without this patch
Change-Id: I559d653e0e72e641e1ee6745924cb835bb0a355b
Bug: 64536066
Test: soong_zip --jar -o /tmp/out.zip -C . -l files.list && \
zipdetails /tmp/out.zip | less \
# and check that the first entry contains the \
# "CAFE" extra attribute as shown below:
0000001E Filename 'META-INF/'
00000027 Extra ID #0001 CAFE 'Java Executable'
00000029 Length 0000
Change-Id: I12c4078591f2ce2afc1af5b9db20393b26601bca
Bug: 64536066
Test: soong_zip --jar -o /tmp/out.zip -C . -l files.list && \
zipdetails /tmp/out.zip | less \
# and check that the META-INF entries are earlier in \
# the list than other entries
Change-Id: Id5c6ea9ce8c3a6fbfb8366db753e6603a076dbf8
This reverts commit 526416b1e4.
Figured out a fix for the deadlocks; resubmitting the patch.
The first version was deadlocking because the switch statement in
zipWriter.write would choose a specific zip entry to start writing,
but the individual chunks may not have all necessarily been compressed
yet. When each individual chunk was made to require to request its own
allocations, the compression of the chunks of the file being currently
written could be blocked waiting for memory to be freed by chunks from
other files that hadn't yet started being written.
This patch is much like the original except it preallocates the memory
for the entire file upfront (and happens to use the total file size
rather than the compressed size, but I didn't observe that to cause any
performance differences).
Bug: 64536066
Test: m -j dist showcommands # which runs soong_zip to package everything
Change-Id: Id1d7ff415e54d3a6be71188abbdbbbab5a719fcf
This wasn't intended to change the behavior, but it does slightly.
Previously any requests to acquire memory wouldn't block; only
(subsequent) requests for execution would block (if another
caller had already consumed the memory quota). Now the requests
for memory can also also block.
It turns out in a brief test on my computer that soong_zip
runs about 10 to 20% faster with this change than without it.
The final step involving soong_zip decreased from about
3.6 sec to about 3.3 sec in tests on my computer.
When testing the process of re-zipping the contents of
angler-target_files*.zip , the time decreased from about
6.3 sec to about 5.3 sec in tests on my computer, and the
peak memory usage reported by `top` decreased from about
1.5g to 1g
Bug: 64536066
Test: m -j dist showcommands # which runs soong_zip to package everything
Change-Id: I0422e4c363c675bb7a93309fac4616c768bfbe8f
The Finder runs roughly 200ms faster than findleaves.py in aosp,
and runs roughly 400ms faster in internal master.
Bug: 64363847
Test: m -j
Change-Id: I62db8dacc90871e913576fe2443021fb1749a483
Fix mutator registration for tests to allow different tests
in the same package to register different mutators.
Allow tests to track the resulting ModuleBuildParams objects
to use in assertions, and provide helpers for getting them.
For example:
config := android.TestConfig(buildDir)
ctx := android.NewTestContext()
ctx.RegisterModuleType(...)
ctx.MockFileSystem(...)
ctx.ParseBlueprintsFile("Android.bp")
ctx.PrepareBuildActions(config)
ctx.Register()
// Get the Inputs value passed to the javac rule for the foo module
inputs := ctx.ModuleForTests("foo".Rule("javac").Inputs
Test: java_test.go
Change-Id: I10c82967f5f3586d2c176f169906b571ed82fc73
This was causing android/soong/ui/build to rebuild sometimes when it
didn't need to.
Test: `m -j nothing; grep "B compile" out/.soong_ui.trace` repeatedly
Change-Id: I5d33271a7b037f53674a0d312d8da1746eed8eaa
It will cd to $TOP before running soong_ui itself, so that soong_ui
still runs from the top of the tree. ORIGINAL_PWD is saved so that we
can reference that later (for example, to move 'mma' implementation into
Go).
Test: cd system; ../build/soong/soong_ui.bash
Test: Set absolute / relative OUT_DIR or OUT_DIR_COMMON_BASE and repeat
Change-Id: Icb67a3ee6a3358cca50300755c8953419fc19437
Jars have a strange sorting order; the META-INF/ directory should
come first, then META-INF/MANIFEST.MF, then any other files in
META-INF/, and then any files outside META-INF. Add a -j argument
to zip2zip that sorts using jar ordering.
Test: zip2zip_test
Change-Id: I80e2bc7e284ef74f6561c26cb6541298834db1bc
When '-s' is passed, any globbed arguments will have their results
sorted. When there are multiple arguments, the files will still be
inserted in argument order.
A bare '**' is now special cased to mean every file in the input zip.
This allows zip2zip to sort entire zip files efficiently by using
`zip2zip -s -i <> -o <> '**'`. This can be useful if your original zip
program used filesystem ordering which was not reproducible.
Test: m -j blueprint_tools (new tests pass)
Change-Id: Ic3512c5fe14c94c6f3e134296905121d2ff8b58a
Cleanup of some failure messages
Also, this leaves the temp directory untouched if a declared output was not created
Bug: 35562758
Test: make
Change-Id: I8ef1315af80eb327752501f12a331dbdf52ba3e9
This causes Soong to put the outputs of each genrule into a temporary
location and copy the declared outputs back to the output directory.
This gets the process closer to having an actual sandbox.
Bug: 35562758
Test: make
Change-Id: I8048fbf1a3899a86fb99d71b60669b6633b07b3e
This is still fairly simplistic, but good enough to test some build
system changes.
Bug: 33381544
Test: run, inspect output
Change-Id: Ia5c19570493116dca01cb65605cdf20becf8c1d0
Adds a -dist argument that will use DIST_DIR to save logs.
Also writes a summary of each std.log to stderr on errors, so that the
error is more likely to show up in the error reporting. This output is
prefixed with "> " to differentiate it from the progress reports from
multiproduct_kati itself.
Test: multiproduct_kati -only-config
Test: DIST_DIR=dist build/soong/build_test.bash -dist (introducing errors)
Change-Id: I5005b5f3f200c876bc004dd9b0e01e7b6edf5be2
Move common microfactory functions to cmd/microfactory/microfactory.bash
so that they can be used to build both soong_ui for normal build and
multiproduct_kati for build system tests.
Test: m -j
Test: build/soong/build_test.bash
Change-Id: I9512642d846ce54d05a027b6d33a2b3029b3f90b
It now uses the same output style as ninja, overwriting status lines in
smart terminals.
Test: multiproduct_kati
Test: multiproduct_kati | cat
Change-Id: I8db5198ffdc5ebc5503241ac492379753d92978e
1. Added ability to keep the mixed "-f"/"-l" order as same as
command-line flags order.
2. Added "-s" flag to specify which target file within zip is stored uncompressed.
Test: manual
Change-Id: I338b25a7bd6bf1b7e9cc29ad3324575167630fb7
Piping the output of javac through a filter makes it hard to capture
the exit status. Convert it to a wrapper that executes javac and
propagates the exit status.
Bug: 36666657
Test: javac_wrapper_test
Change-Id: I9b56cc3794023aabc9328138a68830e26e980f97
Naming directories according to current date. This makes them easier to
parse when using the tool multiple time.
Also considered:
millis := time.Now().UnixNano() / 1000000
name := fmt.Sprintf("multiproduct-%d", millis)
Also considered putting separators in the number, but it makes for too
long of a directory name.
Test: multiproduct_kati
Change-Id: I10ac6754094653abf5bf11b04efc3c44905d3c8d
IsDir() doesn't handle the case where the file is a symlink to a
directory, which cause fileslist to crash.
The hash is used to validate whether system image is the same, so
hashing the content of symlink makes more sense.
Bug: 36274890
Test: joule builds
Change-Id: I6359418a5b28f8da13f85b01a30a72228fecf4ce
Added -P flag in soong_zip:
It provided a path prefix which will be added in front after -f/-l was
applied with -C.
eg. ./soong_zip -o /path1/../ -P a/c -C d/e -f d/e/test.txt
zip_dest: a/c/test.txt
Bug: b/31676493
Test: go build -a to build a real soong_zip, and test it with real
files.
Change-Id: Ic924ba252e963ca5847c370f733058888426b696
We can start removing out directories again in multiproduct_kati, since
the opendir bug has been fixed.
Add --color_warnings to the Kati command line. Since this is different
from Make, take this opportunity to reorder the command line to make
more sense. This wasn't done before because kati forces a regen whenever
the command line changes.
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I5ad03359fbc16db482722946202297c1ae0f2b90
That will be the go version at compile time. So read $GOROOT/VERSION, or
fall back to executing `$GOROOT/bin/go version` to find the go version
currently in GOROOT.
Test: Ensure everything rebuilds when switching between go1.8rc2 and go1.8
Change-Id: I8738a7aa249a088b1e0668af260fa3974844dab7
Some javac wrappers output the entire list of java files being
compiled on a single line, which can be very large, set the maximum
buffer size to 2MB.
Test: pipe the grok build output through soong_javac_wrapper
Change-Id: Ib208461dca02ced4959433b7695ec760dc134468
The soong_javac_filter tool will take the output of the javac tool
on stdin and produce a colorized version similar to what clang
produces on stdout. It also strips the useless "warning there are
warnings" messages:
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Modules that want the warnings can pass the appropriate -Xlint
options.
Test: javac_filter_test
Change-Id: Ic8d337d95ae6c48146771f2982fd1316cb7d82be
This creates a rotating build.trace.gz in the out directory that can be
loaded with chrome://tracing. It'll include start and end timings for
make/soong/kati/ninja, and it will import and time-correct the ninja log
files.
Test: m -j; load out/build.trace.gz in chrome://tracing
Test: multiproduct_kati -keep; load out/multiproduct*/build.trace.gz
Change-Id: Ic060fa9515eb88d95dbe16712479dae9dffcf626
This is a replacement for build/tools/kati_all_products.sh using the new
Soong ui/build package. It doesn't even attempt to run ninja, and it can
be configured to run only the product config, or only the product config
and Soong.
For AOSP on my machine:
-only-config 1.4s
-only-soong 1m20s
<none> 13m
Test: multiproduct_kati
Change-Id: Ie3e6e7bdf692e46a8b8eb828f437190f8003500b
microfactory is a tool to incrementally compile a go program. It's
similar to `go install`, but doesn't require a GOPATH. A package->path
mapping can be specified as command line options. All input files are
hashed, and if any change, the necessary packages will be rebuilt.
microfactory can (re)build itself as necessary, so combined with a shell
script that runs `go run microfactory.go` the first time, it can
bootstrap a go program entirely from sources with just a working goroot.
Time to build soong_ui only using source & GOROOT:
first time no-change incremental
microfactory 1400ms 15ms
go install 670ms 130ms
While microfactory takes longer the first time, almost half of that time
is from `go run` and building microfactory for use later. If
microfactory only has to build soong_ui, it's about 580ms.
Test: USE_SOONG_UI=true m -j blueprint_tools
Test: go test -bench . build/soong/cmd/microfactory/microfactory_test.go
Change-Id: I4d2b9825788144fa10042bbd804482e44f459a54
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
Move Soong's globbing-with-dependencies support into Blueprint so it can
be used for subdirs= lines in Android.bp files.
Blueprint has a slight change in behavior around subname= lines, it now
always uses the subname and doesn't fall back to Blueprints. To support
the Blueprints files in build/blueprint, use them directly with build=.
Test: build, add source file that matches glob, rebuild
Change-Id: Ifd0b0d3bc061aae0a16d6c7ca9a1cd8672656b4d
Mutator registration is tightly coupled with the android package, move
all registration from the soong package to the android package.
Test: build.ninja identical
Change-Id: Ie183d0b52cc7431c9e05b231934d189208ef1efe
This compresses multiple files in parallel, and will split up larger
files (5MB+) into smaller chunks (1MB) to compress in parallel.
There is a small size overhead to recombine the chunks, but it's only a
few bytes per chunk, so for a 1MB chunk, it's minimal.
Rough numbers, with everything in the page cache, this can compress
~4GB (1000 files) down to 1GB in 6.5 seconds, instead of 120 seconds with
the non-parallel soong_jar and 150 seconds with zip.
Go's DEFLATE algorithm is still a bit worse than zip's -- about 3.5%
larger file sizes, but for most of our "dist" targets that is fine.
Change-Id: Ie4886c7d0f954ace46e599156e35fea7e74d6dd7
This doesn't do any decompression / recompression, but just copies over
the already compressed contents. So it's similar to zip -U, but allows
rewriting of the paths.
The first expected usecase is to replace img_from_target_files during
the build, since it does the equivalent of this:
zip2zip -i <target-files.zip> -o <img.zip> OTA/android-info.txt:android-info.txt IMAGES/*:.
Except it decompresses and recompresses the images, which takes over a
minute instead of a few seconds.
Change-Id: I88d0df188635088783223873f78e193272dbdf1c
Ensure that we only create one directory entry, not one with a / at the
end and another without. Clean incoming paths to prevent other similar
duplicates.
When adding directories specified directly as inputs, use the relative
path just like files, not the full path.
Instead of traversing symlinks, addg them to the zip file directly.
Present an error instead of crashing when an input does not exist.
Change-Id: Id6d6561fe12c28398f6096bf9bcae602bb7aa491
Unbundled builds may use a subset of the tree, which can bring in unused
modules but not their dependencies. Delay handling of dependency errors
for unbundled builds to ninja time, which will prevent errors if only
modules with satisified dependencies are built.
Change-Id: Ib93bae93fcfa0b55df500a30d8e35231ffb0987c
The existing behavior of using the build directory as the working
directory is useful if you want to move/copy the output directory around
and SRCDIR still refers the the source. But, it's more useful to have
the source directory be the working directory. Tools like cpp(__FILE__)
and other debug prints embed relative paths from the working directory.
We also have tools that expect the working directory to be $TOP.
Change-Id: Ia0f1d3c6b7df72d61cf5628efa2baa98bd19775b
Instead of using this hardcoded list, use the blueprint plugin
infrastructure to find them at build time and load them.
Bug: 23567252
Change-Id: Iae87e8c9bf020f4beb42c418b9a6bf8b0fe8b601
Refactor the soong.config loading code to support reading in
product variables from soong.variables.
Change-Id: I389e6bb5c501b53167267d5f5d0d25557811cf72
Instead of putting all the blueprint registrations in soong_build,
put them all in init() functions. This puts the registration next
to the implementation.
Change-Id: Ide1a749518f5e9d1367a18ab3bb1d91da3310c76
Build a map of blueprint directory to modules built from that
directory, and then add phony rules to build.ninja that emulate
the behavior of mma in the current build system.
Also fixes checkbuild to depend on checkbuild files and installable
files, but not installed files.
Change-Id: I8bad6e93387940df7439dbd4554f6d79f924c65f
Take HostOrDevice out of Arch, and put it into AndroidModuleBase
instead. Also separate out the host vs. device mutator from
ArchMutator. This will make it possible for genrules to depend
on a host tool, regardless of which host arches it is compiled
for.
Change-Id: I22bbfd28b65c3eebdfa101a712f90dd615148dc8
Add genrule, which uses a host executable (possibly built by the
build system) to generate a single source file. This differs slightly
from gensrcs, which takes a list of sources and translates them through
a host executable to individual source files.
Change-Id: I94bda62c4c53fb3f3817def190e6a7561508d297
Java resource lists created by globs like "**/*" may include
directories. Ignore them when creating the zip file unless
-d was passed on the command line. Fixes read: is a directory
errors.
Change-Id: Ifa6fd4fbd8262f700bed1f4a7268f11618dc305c
Combine the zipWriter and info arguments that are passed into every
function into a single writer class, and make all the functions
methods on the new class.
Change-Id: I29d0164578894974ceaed9482349604633de6533
Add support for running aapt to generate R.java and Manifest.java,
compiling java into jar, adding resources to the jar to convert it
into an apk, signing the apk, and zipaligning the apk.
Change-Id: I7a73fef590d07f35b3d0b56a8571780c09bb10ae
SDK prebuilts need to be treated differently than other prebuilts.
They are used like a normal Java prebuilts when compiling java files,
but they are also used as aapt includes when compiling apps, and they
provide prebuilt aidl files when compiling aidl sources. Create a
new module type so dependers can determine what kind of dependency
they are.
Change-Id: If426da63cfadcb4b14abda58c7b346554912a07a
Add support for converting logtags files to source files. Each
.logtags file is converted to a .java file, but only after all
.logtags files have been combined into a merged text file by
the logtagsSingleton.
Change-Id: I375d82874029bd26a7c528e46add2a638ba123f9
Using ctx.Config().(Config) everywhere is a mouthful, and it is
inefficient to do the type assertion. Put the Config interface into
the context, and provide an AConfig() to return the Config already
converted to the right type.
Change-Id: I301a1fd7d2a005580aabca7866a37c5d42ad8c69
Add support for compiling java libraries (.jar files with
or without .dex), java binaries (.jar files with a wrapper
script to run them), and java prebuilts (for the SDK .jars)
Change-Id: Id624da64c92cf20c6d9577c6bb06e5b212af0d1b
Java resource support requires globbing directories while ignoring
specific filenames. Add support for multiple -e options to
soong_glob to specify filenames to exclude, and split out Glob
into GlobRule to insert a rule to generate a glob file list.
Change-Id: Ia911dd68bd1638452881d18378572d015fd4e31a
Ninja can't depend on environment variables, so modifying build
behavior based on environment variables requires coordinating
between the soong script that invokes ninja and the soong_build
manifest generator.
Allow any module to call Config.Getenv to get the contents of an
environment variable while registering a dependency on it.
After all modules have been processed write out the state of
all used environment variables to a JSON file called
.soong.environment. During the next build the soong script
will use the soong_env tool to compare the contents of
.soong.environment to the current environment, and force a
build manifest regeneration by deleting the .soong.environment
file if any variables have changed.
Change-Id: Id0d81933a857bc2fc1cd7a393a3c6cec73dc4824
Art needs a custom module type in order to perform complicated
build logic like depending on environment variables and varying
cflags based on cpu variant. Export enough of the types and
functions from cc for art_cc_library to inherit from cc_library.
While I'm touching every line, also rename the New* methods
to *Factory.
Change-Id: I7123aa47019c4ced7a1ab57c394225bc7844b5ea
Some cc_test modules want a test per source file, for example when
there is global state that needs to be reset between each test
suite, but no way to reset it. Allow them to specify test_per_src: true,
which will cause a separate test to be built for each source file.
Change-Id: I3dbf1202fb070437cb0109f195dc11a6440061ee
This behaves slightly differently than it does in the make based build.
1. The make based build manually passes -DGTEST_OS_ANDROID (or
whatever). gtest-port.h already has logic that does this, so it's a
no-op.
2. Host libraries are named identically, rather than libgtest_host.
Change-Id: Ic40a1025c698611d202cb7c8ec45abd8fe130065
Modules can choose to add "installed files", which are files that
a product can depend on to cause the module to build, and "checkbuild
files", which will only be built if another module that needs to build
depends on them. For every target, add a module-install and a
module-checkbuild target that create dependencies on those files,
and then add a global checkbuild target that depends on all the
module-checkbuild targets. Also add a module target for each module
that depends on module-install and module-checkbuild.
Change-Id: I801389ad6ab9806b71d92cd327a0f9cb7582e0df
Initial build logic for building android with soong. It can build
a variety of C and C++ files for arm/arm64 and host.
Change-Id: I10eb37c2c2a50be6af1bb5fd568c0962b9476bf0
Paul Duffin