Commit Graph

25 Commits

Author SHA1 Message Date
Colin Cross fd708b5651 Move response file handling to a separate package
sbox is going to need to read and write response files, move ReadRspFile
to its own package.

Test: response_test.go
Change-Id: Iecb5486b4aaeb2531828743ad8ef784df675e18e
2021-03-25 11:06:45 -07:00
Bob Badour 02040de891 Add LOCAL_LICENSE_KINDS to build/soong
Added SPDX-license-identifier-Apache-2.0 to:
  Android.bp
  android/Android.bp
  android/soongconfig/Android.bp
  androidmk/Android.bp
  apex/Android.bp
  bazel/Android.bp
  bp2build/Android.bp
  bpf/Android.bp
  bpfix/Android.bp
  cc/Android.bp
  cc/config/Android.bp
  cc/libbuildversion/Android.bp
  cc/libbuildversion/tests/Android.bp
  cc/ndk_api_coverage_parser/Android.bp
  cc/ndkstubgen/Android.bp
  cc/symbolfile/Android.bp
  cmd/dep_fixer/Android.bp
  cmd/diff_target_files/Android.bp
  cmd/extract_apks/Android.bp
  cmd/extract_jar_packages/Android.bp
  cmd/extract_linker/Android.bp
  cmd/fileslist/Android.bp
  cmd/host_bionic_inject/Android.bp
  cmd/javac_wrapper/Android.bp
  cmd/merge_zips/Android.bp
  cmd/multiproduct_kati/Android.bp
  cmd/path_interposer/Android.bp
  cmd/pom2bp/Android.bp
  cmd/pom2mk/Android.bp
  cmd/sbox/Android.bp
  cmd/soong_build/Android.bp
  cmd/soong_env/Android.bp
  cmd/soong_ui/Android.bp
  cmd/zip2zip/Android.bp
  cmd/zipsync/Android.bp
  cuj/Android.bp
  dexpreopt/Android.bp
  dexpreopt/dexpreopt_gen/Android.bp
  env/Android.bp
  etc/Android.bp
  filesystem/Android.bp
  finder/Android.bp
  finder/cmd/Android.bp
  genrule/Android.bp
  jar/Android.bp
  java/Android.bp
  java/config/Android.bp
  kernel/Android.bp
  linkerconfig/Android.bp
  linkerconfig/proto/Android.bp
  makedeps/Android.bp
  partner/Android.bp
  phony/Android.bp
  python/Android.bp
  python/tests/Android.bp
  remoteexec/Android.bp
  rust/Android.bp
  rust/config/Android.bp
  scripts/Android.bp
  sdk/Android.bp
  sh/Android.bp
  shared/Android.bp
  symbol_inject/Android.bp
  symbol_inject/cmd/Android.bp
  sysprop/Android.bp
  tradefed/Android.bp
  ui/build/Android.bp
  ui/logger/Android.bp
  ui/metrics/Android.bp
  ui/metrics/proc/Android.bp
  ui/status/Android.bp
  ui/terminal/Android.bp
  ui/tracer/Android.bp
  xml/Android.bp
  zip/Android.bp
  zip/cmd/Android.bp

Added SPDX-license-identifier-Apache-2.0 SPDX-license-identifier-BSD to:
  finder/fs/Android.bp
  third_party/zip/Android.bp

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all

Exempt-From-Owner-Approval: janitorial work

Change-Id: Ia47ca14f16b8c9f84f9d533a07e5b00e2c04e8d4
2021-02-06 04:23:21 +00:00
Colin Cross 405af07859 Revert "Make lots of tests run in parallel"
This reverts commit 323dc60712.

Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests

Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
2020-10-09 18:34:24 -07:00
Colin Cross 323dc60712 Make lots of tests run in parallel
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.

Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
2020-10-06 15:12:22 -07:00
Patrice Arruda 8da724a64a Update language to comply with inclusive guidance
See https://source.android.com/setup/contribute/respectful-code for
reference

Bug: 161896447
Test: m nothing
Change-Id: Ib54185f2ef9666c7d82afc65fe7fc6bc7fdadfb5
2020-07-28 21:22:12 +00:00
Thiébaud Weksteen 5e291c238f Fix bpfmt issues and add bpfmt to preupload checks
Test: m nothing
Change-Id: I10526a33685335bdd26cc7692815133379d4e0f6
2020-06-29 09:14:08 +02:00
Sasha Smundak 6172491b91 If input zip file cannot opened, print its name.
Test: touch empty.zip; merge_zips out.zip empty.zip
(output should contain `empty.zip: zip: not a valid zip file`)

Change-Id: If707876af2301d7ad7d065c004cffb0c2491c7df
2020-01-22 10:21:43 -08:00
Colin Cross b5f6dc47ee Fix lint error in format string
Test: go test
Change-Id: I8b7df9222db1fc88938cdfa43a6505f2da854876
2019-09-11 17:06:02 +00:00
Sasha Smundak 1459a9259e Allow reading zipfile list from a file.
Android build generates 180K+ zip files that are to be merged and the
result passed to the indexing pipeline to generate the source code
cross-reference. Their names cannot be passed on the command line,
hence the need for this change. That required extensive changes to
the code as it no longer can be assumed that all the input files can
be kept open.

Bug: 121267023
Test: internal + treehugger + combine index files
Change-Id: I0a829f96ff7187ad967fb6b8cede387501ec57cc
2019-08-06 14:23:35 -07:00
Dan Willemsen 707542f298 Simplify python launcher, use __main__.py
Uses more python rather than C++, and skips less of Py_Main.

Test: build/soong/python/tests/runtests.sh
Change-Id: I03997d88e2e16047c96bb4e00e530229c42b3325
2019-02-15 14:33:59 -08:00
Dan Willemsen 263dde779e Fix par file zip offsets
The zip file format does support scripts/programs prefixed to the
archive, though many of the offsets are supposed to start from the
beginning of the file. Some tools (python and zipinfo) are able to
read zip files with arbitrary prefixes, but others (libziparchive and
zipdetails) fail.

So pass the file to prefix onto the zip file into merge_zips so that we
can set the offsets from the real start of the file.

Test: m sepolicy_tests  (runs the embedded python interpreter)
Test: zipinfo out/host/linux-x86/bin/sepolicy_tests
Test: zipdetails out/host/linux-x86/bin/sepolicy_tests
Change-Id: If73d4c2465581f7de5aa47959284ecf2059df091
2018-12-03 15:55:15 -08:00
Colin Cross dc1e829b59 Always allow duplicates with identical CRC32 and size
Don't warn on duplicate files in merge_zips if they have identical
CRC32 and size values.

Test: m checkbuild
Test: merge_zips_test.go
Change-Id: I61336ca4d4d3b7402c24a7abd337bd350fe10930
2018-10-17 15:49:53 -07:00
Colin Cross 05518bc13b soong_zip: Add tests
Add test that cover basic command line usage of soong_zip.  -D
is not covered yet as the implementation will be replaced with
one that is also more easily testable in the next patch.

Bug: 116751500
Test: zip_test.go
Change-Id: I5a1bcee74ebc9cb3cf332c36f89bc12c0e807ad2
2018-09-28 13:56:06 -07:00
Colin Cross 4c03f68763 Make merge_zips -stripFile use blueprint style globs
merge_zips -stripFile was only considering the name of the file and
ignoring the path.  Make it more useful by supporting blueprint style
globs.  The previous behavior can be recreated by prefixing with **/.

Bug: 111389216
Test: m checkbuild
Change-Id: I25760fe3f1f77704dd9da9d107d9a38a415d681f
2018-07-16 15:24:43 -07:00
Colin Cross 2486065c43 Add tests for merge_zips
Test: merge_zip_test.go
Change-Id: I8fc577946b40cad193864aa2ebda1bef3a0e050e
2018-07-16 15:24:22 -07:00
Nan Zhang 1db85407aa Remove timestamp based filelist file for tracking Python dependencies
Each Python module will generate a zip file containing source & data
files. The Python binary will collect all its dependencies and use
merge_zips to merge each zip file to create a final .par file.

Test: m -j checkbuild && real examples:
Bug: b/70568913

Change-Id: I9ff232d461d33e1c06026e7dcb5b124bf02c3ce5
2018-03-09 14:12:48 -08:00
Nan Zhang 5925b0fda4 Add -p, -e options in merge_zips for supporting par format.
Move __init__.py files population to merge_zips, and add options: -p, and
-e. Since par needs entry_point.txt to search the program entry point, so
add -e option to support it.

Test: ./merge_zips -p -e entry_point.txt out in1.zip in2.zip
Bug: 70568913
Change-Id: Ida10faf125e1b94dffc98b30240db3b90fd75b25
2017-12-20 12:57:10 -08:00
Colin Cross e909e1e079 Add support for --ignore_duplicates to merge_zips
Add a --ignore_duplicates option to cause merge_zips to
silently take the first entry in the case of duplicate entries
in the zips to be merged.  This will be used for soong Jacoco
support to combine the instrumented jar with the original jar
to pick up any classes that were excluded.

Bug: 69629238
Test: m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true SKIP_BOOT_JARS_CHECK=true WITH_DEXPREOPT=false
Change-Id: I49ede3f07ce0ed7701b4db7058da2e4f11b8043e
2017-11-22 16:08:35 -08:00
Colin Cross 5c6ecc179f Don't panic in turbine with no classes
ALLOW_MISSING_DEPENDENCIES=true can cause missing dependencies to be
hidden from GenerateAndroidBuildActions, which is triggering the
len(jars) == 0 check in compileJavaHeader.  For the
ALLOW_MISSING_DEPENDENCIES=true case the check is unnecessary, as the
build rules will be replaced with runtime errors.  On the off chance
the check would have been hit in a real build, allow merge_zips to
create an empty zip file if it is asked to.

Test: m ALLOW_MISSING_DEPENDENCIES=true
Change-Id: I3171f921d51229ddf38a1a647d32566658c673fa
2017-10-24 16:01:44 -07:00
Colin Cross 0cf45cdc1e Add -stripFile argument to merge_zips
Putting resources in dex jars is going to need to pull files from
classpath jars that do not match *.class.

Test: m -j checkbuild
Change-Id: Ia37b8b9387e5f5636769afc6937b9aa3c9ec1ced
2017-10-04 17:19:43 -07:00
Colin Cross 635acc9446 Rearrange manifest file handling in merge_zips and soong_zip
Jar always puts default MANIFEST.MF files in if none was specified.
Copying that behavior in soong_zip causes problems with merge_zips,
because it ends up taking the default manifest from the classes.jar
instead of the user's manifest from res.jar.  We don't want the
user's manifest in the classes.jar, otherwise a change to the
manifest will cause all the class files to rebuild.  Instead,
move the manifest insertion to the final merge_zips stage.

Test: m -j checkbuild
Change-Id: Id6376961dbaf743c2fb92843f9bdf2e44b963be0
2017-09-20 13:20:45 -07:00
Nan Zhang 13f4cf5c9e Add -stripDir and -zipToNotStrip option in merge_zips.
In some cases, we need delete META-INF dir to void mis-using some
classes files in the Java runtime.

For now, when we delete META-INF, we need keep Manifest file in the jar
since merge_zips doesn't have ability to add a default one.

Bug: b/65455145
Test: m clean && m -j

Change-Id: Iea44b1a98d348cd8df1fa7374907733b1add6935
2017-09-19 18:44:43 -07:00
Nan Zhang d5998cce7d Don't add data_descripters when merging uncompress zip entries for merge_zips.
Also filter out META-INF/TRANSITIVE dir, and report warnings when
merge_zips see duplicates entries with different CRC hash.

Bug: b/65455145
Test: m clean && m -j java (locally)

Change-Id: I47172ffa27df71f3280f35f6b540a7b5a0c14550
2017-09-13 23:01:49 -07:00
Colin Cross 34540315a0 Allow some duplicates in merged jars
Only take the first MANIFEST.MF or module-info.class file.

Test: m -j checkbuild
Change-Id: Ifbf9fe272437ef2c2bd51ab4849ac8d7ef37b6fc
2017-09-06 15:01:07 -07:00
Jeff Gaston 8bab5f2015 Implement merge_zips
For merging .zip and/or .jar files

Bug: 64691570
Test: merge_zips merged.zip in1.zip in2.zip
Change-Id: I9a080d25b92a506c9b5451670c4786895b2722ab
2017-09-01 13:52:13 -07:00