Get a list of leaf nodes in the dependency graph
from ninja, and make sure none of them are in the
output directory. This ensures that there are no
rules that depend on a file in the output directory
that doesn't have rule to generate it. The check
will catch a common set of build failures where
a rule to generate a file is deleted (either by
deleting a module in an Android.mk file, or by
modifying the build system incorrectly). These
failures are often not caught by a local incremental
build because the previously built files are still
present in the output directory.
Bug: 36843214
Bug: 68062417
Test: manual
Change-Id: I4933187e8b72f2ef0c32d18ffea756e2c6fa417c
Instead of throwing it away (though it would also be in soong.log along
with the verbose messages).
Test: build/soong/build_test.bash with ctx.Println added
Change-Id: I64f8b11ab6752a6be8a5934fb41e30439238e331
For unbundled builds, LOCAL_SDK_VERSION needs to be set, otherwise we'll
try to use frameworks-res, which isn't available.
Bug: 64723465
Test: Use with prebuilts/sdk/update_current.py
Change-Id: If5e1233b4bcb09b3d0e2251e8dd1a8ecfc87eb19
Instead of deleting artifacts/logs from successful build (unless -keep
is set), and keeping unsuccessful artifacts, keep all logs and remove
all artifacts (unless -keep is set, then we'll compress the artifacts).
If -dist is passed in, we'll put an archive of the logs into the
DIST_DIR. Even compressed, the rest of the artifacts are still a bit too
large to dist (~5.6GB on AOSP).
Test: build/soong/build_test.bash
Test: build/soong/build_test.bash -keep
Test: build/soong/build_test.bash -dist
Change-Id: I87f55978c18c8ff2e517b8271554ba383003742f
Android Support Library 26+ does not specify dependency types, so parse
all pom files first, then propagate missing dependency types if we've
got the information.
Bug: 64723465
Test: cd prebuilts/maven_repo/android; pom2mk -use-version 26.0.0-beta2 com/android/support
Change-Id: Ieaff757ff198c9a7b4b006623340b382728c1fd4
Also removeTempDir -> !keepOutDir
Also shorten the help comment
Bug: 68336760
Test: rm out -rf && m -j
Test: sbox -c # and observe that the output does not have a stacktrace
Change-Id: I6fbb385132128ab1273c8398ff2425191049751e
For repositories that contain multiple versions of the same artifact,
we'd only like to create makefiles for one of the versions of the
artifact.
Bug: 64723465
Test: cd prebuilts/maven_repo/android; pom2mk -use-version 25.3.1 com/android/support
Change-Id: I5bdbba8d2143a47610d56d679381eb01b3cf176d
Most notably, the sandbox depfile path should be passed into the tool.
Bug: 68336760
Test: m -j checkbuild
Change-Id: I22f944a3f57d613fda26de0ea777a915cafcd020
ALLOW_MISSING_DEPENDENCIES=true can cause missing dependencies to be
hidden from GenerateAndroidBuildActions, which is triggering the
len(jars) == 0 check in compileJavaHeader. For the
ALLOW_MISSING_DEPENDENCIES=true case the check is unnecessary, as the
build rules will be replaced with runtime errors. On the off chance
the check would have been hit in a real build, allow merge_zips to
create an empty zip file if it is asked to.
Test: m ALLOW_MISSING_DEPENDENCIES=true
Change-Id: I3171f921d51229ddf38a1a647d32566658c673fa
to make it faster/easier to invoke from other Go programs
(such as multiproduct_kati)
Bug: 67478260
Test: m -j
Change-Id: Idd2671a44290550197c88f53dd11a6dd39c85cc5
javac warns whenever -source is used with a version that does not
match javac, because it wants an rt.jar provided for the given
source level. We are not using -source to prevent usage of APIs
that don't exist in the newer rt.jar, so hide the messages.
Test: javac_wrapper_test.go
Change-Id: Ic94b61767436848fc164e377dc3cc5a63726f1cf
build/soong/soong_ui.bash --dumpvars-mode \
--vars="..." \
--abs-vars="..." \
--var-prefix="..." \
--abs-var-prefix="..."
is similar to the previous:
CALLED_FROM_SETUP=true BUILD_SYSTEM=build/core \
make -f build/core/config.mk dump-many-vars \
DUMP_MANY_VARS="..." \
DUMP_MANY_ABS_VARS="..." \
DUMP_VAR_PREFIX="..." \
DUMP_ABS_VAR_PREFIX="..."
and
build/soong/soong_ui.bash --dumpvar-mode [--abs] VAR
is similar to the previous:
CALLED_FROM_SETUP=true BUILD_SYSTEM=build/core \
make -f build/core/config.mk dumpvar-[abs-]-VAR
But uses soong_ui and ckati, so that we use a consistent make parser and
sandboxing configurations.
One major output difference between the pure make implementation and
this one is that report_config in Go is implemented using embedded
newlines in single quotes, while the make implementation uses `` with
embedded echo commands. This seems to work fine for both bash and zsh,
and report_config isn't meant to be machine-parsed anyways.
Test: build/soong/soong_ui.bash --dumpvar-mode report_config
Test: build/soong/soong_ui.bash --dumpvar-mode TARGET_DEVICE
Test: build/soong/soong_ui.bash --dumpvar-mode --abs PRODUCT_OUT
Test: build/soong/soong_ui.bash --dumpvar-mode --abs ALL_PRODUCTS
Test: build/soong/soong_ui.bash --dumpvars-mode --vars="report_config TARGET_DEVICE" --abs-vars="ALL_PRODUCTS"
Test: build/soong/soong_ui.bash --dumpvars-mode --vars=TARGET_DEVICE --abs-vars=PRODUCT_OUT --var-prefix=v_ --abs-var-prefix=a_
Change-Id: I0fbd0732bbf6fcfcd24084cf3c830a91a4b6bfc2
This speeds up dumping make variables from ~380ms using make to ~220ms
using ckati. It also means that we're consistently using the same parser
for builds (with the same .KATI_READONLY/etc extensions).
envsetup.sh (lunch) / other scripts still use make, changing those to go
through soong_ui will be a future change.
Test: m clean; m nothing
Test: USE_GOMA=true m nothing
Test: m PRODUCT-aosp_x86-sdk
Test: m APP-Calculator
Test: build/soong/build_test.bash -only-config (on AOSP and internal master)
Change-Id: I6ca554de8de4955fb869001d06d29969b75751cc
This can make it easier to add build/soong/build_test.sh to continuous
build branches to catch new breaks, but where some products are known to
be broken.
Test: build/soong/build_test.bash --skip-products aosp_arm,aosp_arm64
Change-Id: Ib240570b68957077047ca75b78658429cb8c6a49
Putting resources in dex jars is going to need to pull files from
classpath jars that do not match *.class.
Test: m -j checkbuild
Change-Id: Ia37b8b9387e5f5636769afc6937b9aa3c9ec1ced
* changes:
Extract the linker and embed it into host bionic binaries
Add prefix_symbols support to cc_object
Fix ToolchainClangCflags for host bionic
Don't add a rpath to the linker
The linux kernel requires that the ELF interpreter (runtime linker)
that's referenced by PT_INTERP be either an absolute path, or a relative
path from the current working directory. We'd prefer a relative path
from the binary, similarly to how we handle looking up shared libraries,
but that's not supported.
Instead, extract the load sections from the runtime linker ELF binary
and embed them into each host bionic binary, omitting the PT_INTERP
declaration. The kernel will treat it as a static binary, and we'll use
a special entry point (linker_wrapper) to fix up the arguments passed by
the kernel before jumping to the embedded linker. From the linker's
point of view, it looks like the kernel loaded the linker like normal.
Bug: 31559095
Test: Enable host bionic,
out/soong/host/linux_bionic-x86/nativetest64/libdemangle_test/libdemangle_test
Change-Id: I8d0aea9790b5e86fcc3ea6e2d00cfa33907e2853
Jar always puts default MANIFEST.MF files in if none was specified.
Copying that behavior in soong_zip causes problems with merge_zips,
because it ends up taking the default manifest from the classes.jar
instead of the user's manifest from res.jar. We don't want the
user's manifest in the classes.jar, otherwise a change to the
manifest will cause all the class files to rebuild. Instead,
move the manifest insertion to the final merge_zips stage.
Test: m -j checkbuild
Change-Id: Id6376961dbaf743c2fb92843f9bdf2e44b963be0
In some cases, we need delete META-INF dir to void mis-using some
classes files in the Java runtime.
For now, when we delete META-INF, we need keep Manifest file in the jar
since merge_zips doesn't have ability to add a default one.
Bug: b/65455145
Test: m clean && m -j
Change-Id: Iea44b1a98d348cd8df1fa7374907733b1add6935
Also filter out META-INF/TRANSITIVE dir, and report warnings when
merge_zips see duplicates entries with different CRC hash.
Bug: b/65455145
Test: m clean && m -j java (locally)
Change-Id: I47172ffa27df71f3280f35f6b540a7b5a0c14550
Allow filename collisions for directories, which may happen if
multiple globs of resources include the same directory names.
Continue to report errors if collisions occur between files, and
also add checks for collisions between files and directories.
Test: manual
Change-Id: Iac19fbd325c53fbb41552ea230d813c8dddf9439
Combining the results of a javac command into a jar needs to take
a whole directory, add -D to specify a directory to search for all
files.
Test: m -j checkbuild
Change-Id: Id4707189d57b664f0262a0782630f893af397900
This will enable writing a modified manifest without having to create a temporary file first
Bug: 64536066
Test: soong_zip --jar -o /tmp/out.zip -C . -l files.list && \
# make sure that the output is binary equal
# with and without this patch
Change-Id: I559d653e0e72e641e1ee6745924cb835bb0a355b
Bug: 64536066
Test: soong_zip --jar -o /tmp/out.zip -C . -l files.list && \
zipdetails /tmp/out.zip | less \
# and check that the first entry contains the \
# "CAFE" extra attribute as shown below:
0000001E Filename 'META-INF/'
00000027 Extra ID #0001 CAFE 'Java Executable'
00000029 Length 0000
Change-Id: I12c4078591f2ce2afc1af5b9db20393b26601bca
Bug: 64536066
Test: soong_zip --jar -o /tmp/out.zip -C . -l files.list && \
zipdetails /tmp/out.zip | less \
# and check that the META-INF entries are earlier in \
# the list than other entries
Change-Id: Id5c6ea9ce8c3a6fbfb8366db753e6603a076dbf8
This reverts commit 526416b1e4.
Figured out a fix for the deadlocks; resubmitting the patch.
The first version was deadlocking because the switch statement in
zipWriter.write would choose a specific zip entry to start writing,
but the individual chunks may not have all necessarily been compressed
yet. When each individual chunk was made to require to request its own
allocations, the compression of the chunks of the file being currently
written could be blocked waiting for memory to be freed by chunks from
other files that hadn't yet started being written.
This patch is much like the original except it preallocates the memory
for the entire file upfront (and happens to use the total file size
rather than the compressed size, but I didn't observe that to cause any
performance differences).
Bug: 64536066
Test: m -j dist showcommands # which runs soong_zip to package everything
Change-Id: Id1d7ff415e54d3a6be71188abbdbbbab5a719fcf
This wasn't intended to change the behavior, but it does slightly.
Previously any requests to acquire memory wouldn't block; only
(subsequent) requests for execution would block (if another
caller had already consumed the memory quota). Now the requests
for memory can also also block.
It turns out in a brief test on my computer that soong_zip
runs about 10 to 20% faster with this change than without it.
The final step involving soong_zip decreased from about
3.6 sec to about 3.3 sec in tests on my computer.
When testing the process of re-zipping the contents of
angler-target_files*.zip , the time decreased from about
6.3 sec to about 5.3 sec in tests on my computer, and the
peak memory usage reported by `top` decreased from about
1.5g to 1g
Bug: 64536066
Test: m -j dist showcommands # which runs soong_zip to package everything
Change-Id: I0422e4c363c675bb7a93309fac4616c768bfbe8f
Jeff Gaston