This reverts commit 09f4540d66.
Fixes the raw call to net.Listen in the tests to go through the listen()
helper and use the long socket path fallbacks.
Removes the use of timeouts from the tests -- the behaviors being tested
did not rely on timeouts, so removing them will reduce the flakiness if
the build is heavily loading the machine at the same time the test is
running.
Also fixes some potential nil pointer dereferences.
Test: OUT_DIR=<really long> m blueprint_tools
Test: `while .../soong-ui-build-paths/test/test; do sleep 0.01; done` with a build running
Change-Id: I16d44be7517bc415f1c808284088f4ba40df3bfa
This reverts commit c59a92cb1a.
Reason for revert: tests are broken with long OUT_DIRs
They're directly calling net.Listen, and not using the fallback
for long socket names.
Change-Id: Id14cbd499fd9b36c6926b7552d3554340cb0916c
This reverts commit 96c957ae20.
Fixes issues on some machines where the socket in TMPDIR ended up with a
unix domain socket pathname over 107 characters long, which Go will
reject due to underlying limitations in the system calls. If this
happens, we'll fall back to opening the directory, then using
/proc/self/fd/#/<file>, or manually creating a similar symlink in /tmp.
Also fixes some issues on Mac where os.Executable returns the symlink
instead of the underlying file, sending a message over a unix domain
socket will block if the reader isn't reading, and sandboxing was
preventing us from running `ps`.
Test: m blueprint_tools
Test: m blueprint_tools on mac
Change-Id: Ib19ccfe10cb0a79f1476fb1d5cd20ed0495be367
This will allow us to track (and eventually limit) the commands that the
build references via $PATH. These are mostly implicit dependencies on
the host system -- for Linux, we assume something similar to Ubuntu
14.04 with a few extra packages, but this will let us better define
that.
This will not catch uses of tools with absolute paths (/bin/bash, etc),
but most uses shouldn't be relying on absolute path names anyways.
Adds ~400ms on the first startup, ~140ms on subsequent runs, and
overhead of a few ms for every forwarded execution.
Test: m
Test: build/soong/build_test.bash
Test: Add `gcc --version`, TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: Id68cbb1c8ceef65bbbb10751e83722c7662d2351
When the config loading was changed to using AndroidProducts.mk lists
from soong_ui instead of finding them itself, multiproduct_kati stopped
being able to find all the products. Move the finder initialization
earlier so that we can dump the full list.
Test: build/soong/build_test.bash -only-config
Test: forrest runs of every downstream build_test
Change-Id: I085d40f9008a47b850783499365977a0ff70ac66
protoc dependency files, at least for C++ outputs, uses the form of:
a/b.c \
a/b.h: <dep1> <dep2>...
Ninja will fail the command when it parses a dep file and there's more
than one output file (even though it doesn't care what the output file
name is). So this tool will parse the original file, and output a
version with only a single output file.
Bug: 67329638
Test: NINJA_ARGS="-t deps ...pb.c" m
Test: NINJA_ARGS="-t deps ...srcjar" m
Test: NINJA_ARGS="-t deps ...srcszip" m
Test: Run dep_fixer across all of taimen's dep files, no failures.
Test: Run dep_fixer against the processed files, no changes.
Test: Run androidmk across all of our Android.mk files, inspect the diffs
Change-Id: I4263b7d5faea37285afa6b24dedf5964aa7d19dc
If a static android library lib1 has static_libs: ["lib2"] then the
R.class files for packages in lib2 will be merged into the jar for
lib1. If an app has lib1 in its static_libs it will get the R.class
files from lib2 through lib1, instead of regenerating the R.java
files with numbering that matches the resource table of the app.
Pass transtive static android library dependencies on the aapt2
command line so that aapt2 will always regenerate the R.java
constants for those packages.
Also extract the packages that have R.java files after each aapt2
invocation. This is not necessary for Soong, but is passed to
make to let it force regenerating the packages using
--extra-packages.
Bug: 78300023
Test: m checkbuild
Change-Id: I0f3444af44d2a9f370d1f156c908972f8cc3a1ee
Convert pom2mk to pom2bp that writes out Android.bp files instead
of Android.mk files. pom2mk stays for now until the last users
of it are cleaned up.
Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.mk
Change-Id: I584d63c4228bad32f9e1914b06bde807078d6a55
Add an option to pom2mk to exclude modules by name.
Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2mk -regen Android.mk -exclude androidx.car_car
Change-Id: I083907ef364384aace524ced81820567f5075d76
pom2mk -regen Android.mk will read the previous arguments out of
Android.mk, rerun pom2mk with those arguments, and write the result
to Android.mk.
Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2mk -regen Android.mk
Change-Id: Ie8e0f97ae5b87c8078cf8e753489326d80da1896
Specifically, the use case is android-support-car requires some stubs that don't have a Maven pom.
So those stubs can't easily be listed as a dependency in the pom.
However, we do want the dependency to exist when importing to Make and detected by extract_deps.py
Bug: 76692459
Test: m -j blueprint_tools \
&& update_prebuilts.py -s <build_id> \
&& git diff HEAD^ -- fix_dependencies.mk \
&& m -j checkbuild
Change-Id: If546ecf7416580a01bde7b906976ada4b1145253
An upcoming change will stop exporting ProductVariables from Config, so
switch to using existing accessor functions, and add more when they're
missing.
Bug: 76168832
Test: out/soong/build.ninja is identical
Change-Id: Ie0135bdbd2df3258ef3ddb53e5f8fc00aa9b97f7
For easier debugging if there are multiple duplicates
Bug: 76692459
Test: ./update_current.py -x <build-id>
Change-Id: I3079cb4ed1cfd365d792a7b41c9cdb4e7a1e148f
The make javac rule uses [ -s srcjar-list ] to decide whether or
not to run javac. zipsync was putting a trailing "\n" in the
list file, so the file was never empty. There was one case of a
package that contained no source files (so java-source-list was
empty) and had a res directory with an xml file that declared no
resources (so aapt.jar was created but empty). The 1-byte
srcjar-list file caused javac to run and error with:
javac: no source files
Test: m checkbuild
Change-Id: I26b394b66bf81e4f5abbd27e4dc06bee1d9420a8
extract_srcjars.sh uses zipinfo and unzip, which fail with an
error on an empty zip file. Instead of trying to hack around
this (which is hard to make guarantees for since they are
non-hermetic host tools), replace them with a go tool to unzip
a set of zip files into a directory.
Bug: 73885582
Test: m checkbuild
Change-Id: I151fed347ed5196726e36866ffc27bc831799afb
Each Python module will generate a zip file containing source & data
files. The Python binary will collect all its dependencies and use
merge_zips to merge each zip file to create a final .par file.
Test: m -j checkbuild && real examples:
Bug: b/70568913
Change-Id: I9ff232d461d33e1c06026e7dcb5b124bf02c3ce5
Allow native modules to specify use_version_lib, which will make
an android::build::GetBuildNumber() function available. For host
builds, the function will return the build number at the time that
the module was linked. For device modules it will return the
value of the ro.build.version.incremental property.
Bug: 71719137
Test: build_version_test
Test: m build_version_test && touch build/make/core/Makefile build/soong/cc/libbuildversion/tests/build_version_test.cpp && m build_version_test shows different build numbers for binary and library tests.
Change-Id: I6f7d40b7574bb8206866c4e39bad9c710c796e32
Parse the machine-specific symbol tables into a custom symbol table
object and use a single function to find the offset and size of the
symbol in the custom table. Make the functions to convert a
machine-specific symbol table into the custom table also take mocks,
and provide functions to dump mocks from real files. Add tests
that take the mocks and verify they produce the right offset and
size.
Test: cmd/symbol_inject/*_test.go
Change-Id: I25654032b5017dd13a9a1fe29f8b8826ce5bc82b
Some exe files have a .data symbol at the same address as the
soong_build_number symbol. If the .data symbol is after
soong_build_number in the symbol list, symbol_inject would think
the end address was the same as the start address, and use
uint32(-1) as the size.
Use sort.Search to find the first symbol whose section number is
the same as the target symbol, but whose address is higher than
the target symbol.
Test: manual
Change-Id: I51d6e53c6b906222ba68c5cf93be944843e23550
macho symbol table entries are not always in order, which breaks
finding the next symbol to find the size of the target symbol.
Test: build_version_test
Change-Id: I41d1c3c3ff9929694e9ec2b034553d6b7ddef937
The path to the output directory may be arbitrarily long, use
__SBOX_OUT_DIR__ in the list of output files passed to sbox
to avoid expanding it multiple times in the command line.
Fixes:
ninja: fatal: posix_spawn: Argument list too long
09:40:14 ninja failed with: exit status 1
when building libchrome with a long OUT or OUT_DIR_COMMON_BASE.
Bug: 73726635
Test: m checkbuild
Change-Id: I59024b2164287c8e531711afd9273b692ce9c28a
Only include dependencies for scope "compile."
Bug: 73263586
Test: ./update_current.py -s -t <build-id>
Change-Id: I2ac6055cb4c1ad1f8d7924869f54f50e5e662742
Move __init__.py files population to merge_zips, and add options: -p, and
-e. Since par needs entry_point.txt to search the program entry point, so
add -e option to support it.
Test: ./merge_zips -p -e entry_point.txt out in1.zip in2.zip
Bug: 70568913
Change-Id: Ida10faf125e1b94dffc98b30240db3b90fd75b25
Add a rule in soong that re-executes soong_build in order to
generate build documentation. This allows Soong to customize
the documentation.
Bug: 70516282
Test: m soong_docs
Change-Id: If143cfacd6ac20274cd7bb8d8fab0c07025a5553
On large branches (250+ configurations), the open file count can go
over the default 1024 soft limit on Ubuntu. Many systems have increased
that default, but for the ones that haven't, at least opt into the
hard limit until this can be refactored to use fewer open files.
Bug: 70370883
Test: prlimit -n256:4096 build/soong/build_test.bash -only-config
Test: ulimit -Sn 256; build/soong/build_test.bash -only-config (darwin)
Change-Id: I7a952ffc89a0149ab65b04db1523d348daa6ab3e
This mostly reverts commit 178d5fefc0
and mostly reapplies change I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c .
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
--products=aosp_arm \
'build/blueprint:work^ build/soong:work^' \
'build/blueprint:work build/soong:work'
# and see that the only changes were:
# 1. adding some new files
# 2. changing some line numbers
Test: m -j nothing # which runs unit tests
Change-Id: I32baae00277a547fdcdd1c2219fe6625ee0e45d7
which should make debugging faster.
Bug: 66921582
Test: ./out/soong/host/linux-x86/bin/sbox --output-root outs --sandbox-path out/.temp/sbox-work -c "cd __SBOX_OUT_DIR__ && mkdir asubdir && mkdir b && touch asubdir/child a b c d e f g h i j k l m n o p" outs/a outs/b outs/z # and observe the below output:
mismatch between declared and actual outputs
in sbox command(cd out/.temp/sbox-work/sbox343858828 && mkdir asubdir && mkdir b && touch asubdir/child a b c d e f g h i j k l m n o p)
in sandbox out/.temp/sbox-work/sbox343858828,
failed to create 2 files:
b: not a file
z: does not exist
did create 16 files:
a
asubdir/child
c
d
e
f
g
h
i
j
...6 more
Change-Id: I75e37834c44d4279dec874701d67ce8bb01b872c
This reverts commit 63a250a336.
Reason for revert: Some failures:
namespace_test.go:648: dir1/Blueprints:2:4: a namespace must be the first module in the file
such as New Build Breakage: aosp-master/build_test @ 4475274
Change-Id: I1b5db8eb934e51ff22241bfca44199d886b1393b
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
--products=aosp_arm \
'build/blueprint:work^ build/soong:work^' \
'build/blueprint:work build/soong:work'
# and see that the only changes were:
# 1. adding some new files
# 2. changing some line numbers
Test: m -j nothing # which runs unit tests
Change-Id: I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c
This reverts commit fb941913a3.
Reapplies I4933187e8b72f2ef0c32d18ffea756e2c6fa417c with fixes
to disable the check for mac builds, where many modules are
disabled and the check is just going to cause more problems.
Test: m checkbuild
Change-Id: If6712c90ececd5d015fcdcdeefe0c3d4f5590711
Add a --ignore_duplicates option to cause merge_zips to
silently take the first entry in the case of duplicate entries
in the zips to be merged. This will be used for soong Jacoco
support to combine the instrumented jar with the original jar
to pick up any classes that were excluded.
Bug: 69629238
Test: m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true SKIP_BOOT_JARS_CHECK=true WITH_DEXPREOPT=false
Change-Id: I49ede3f07ce0ed7701b4db7058da2e4f11b8043e
Jacoco support will use zip2zip to create a jar that is a subset
of another jar, and will need exclusion filters and recursive
globs. Switch zip2zip from filepath.Match to pathtools.Match,
and check each included file against the exclusion list.
Bug: 69629238
Test: zip2zip_test.go
Change-Id: Ibe961b0775987f52f1efa357e1201c3ebb81ca9c
Get a list of leaf nodes in the dependency graph
from ninja, and make sure none of them are in the
output directory. This ensures that there are no
rules that depend on a file in the output directory
that doesn't have rule to generate it. The check
will catch a common set of build failures where
a rule to generate a file is deleted (either by
deleting a module in an Android.mk file, or by
modifying the build system incorrectly). These
failures are often not caught by a local incremental
build because the previously built files are still
present in the output directory.
Bug: 36843214
Bug: 68062417
Test: manual
Change-Id: I4933187e8b72f2ef0c32d18ffea756e2c6fa417c
Instead of throwing it away (though it would also be in soong.log along
with the verbose messages).
Test: build/soong/build_test.bash with ctx.Println added
Change-Id: I64f8b11ab6752a6be8a5934fb41e30439238e331
Dan Willemsen