Commit Graph

33 Commits

Author SHA1 Message Date
Colin Cross fd708b5651 Move response file handling to a separate package
sbox is going to need to read and write response files, move ReadRspFile
to its own package.

Test: response_test.go
Change-Id: Iecb5486b4aaeb2531828743ad8ef784df675e18e
2021-03-25 11:06:45 -07:00
Colin Cross 665b6786b8 Fix removing partial zip file on error
Fix the shadowed err variable so that it is visible to the defer
function that removes the partial zip file on error.

Bug: 182761771
Test: manual
Change-Id: Ia67ac4bca5ef0107466edf25a76c74d70c243d3f
2021-03-16 00:20:19 +00:00
Treehugger Robot b68036f44b Merge "Add LOCAL_LICENSE_KINDS to build/soong" 2021-02-10 07:18:29 +00:00
Bob Badour 02040de891 Add LOCAL_LICENSE_KINDS to build/soong
Added SPDX-license-identifier-Apache-2.0 to:
  Android.bp
  android/Android.bp
  android/soongconfig/Android.bp
  androidmk/Android.bp
  apex/Android.bp
  bazel/Android.bp
  bp2build/Android.bp
  bpf/Android.bp
  bpfix/Android.bp
  cc/Android.bp
  cc/config/Android.bp
  cc/libbuildversion/Android.bp
  cc/libbuildversion/tests/Android.bp
  cc/ndk_api_coverage_parser/Android.bp
  cc/ndkstubgen/Android.bp
  cc/symbolfile/Android.bp
  cmd/dep_fixer/Android.bp
  cmd/diff_target_files/Android.bp
  cmd/extract_apks/Android.bp
  cmd/extract_jar_packages/Android.bp
  cmd/extract_linker/Android.bp
  cmd/fileslist/Android.bp
  cmd/host_bionic_inject/Android.bp
  cmd/javac_wrapper/Android.bp
  cmd/merge_zips/Android.bp
  cmd/multiproduct_kati/Android.bp
  cmd/path_interposer/Android.bp
  cmd/pom2bp/Android.bp
  cmd/pom2mk/Android.bp
  cmd/sbox/Android.bp
  cmd/soong_build/Android.bp
  cmd/soong_env/Android.bp
  cmd/soong_ui/Android.bp
  cmd/zip2zip/Android.bp
  cmd/zipsync/Android.bp
  cuj/Android.bp
  dexpreopt/Android.bp
  dexpreopt/dexpreopt_gen/Android.bp
  env/Android.bp
  etc/Android.bp
  filesystem/Android.bp
  finder/Android.bp
  finder/cmd/Android.bp
  genrule/Android.bp
  jar/Android.bp
  java/Android.bp
  java/config/Android.bp
  kernel/Android.bp
  linkerconfig/Android.bp
  linkerconfig/proto/Android.bp
  makedeps/Android.bp
  partner/Android.bp
  phony/Android.bp
  python/Android.bp
  python/tests/Android.bp
  remoteexec/Android.bp
  rust/Android.bp
  rust/config/Android.bp
  scripts/Android.bp
  sdk/Android.bp
  sh/Android.bp
  shared/Android.bp
  symbol_inject/Android.bp
  symbol_inject/cmd/Android.bp
  sysprop/Android.bp
  tradefed/Android.bp
  ui/build/Android.bp
  ui/logger/Android.bp
  ui/metrics/Android.bp
  ui/metrics/proc/Android.bp
  ui/status/Android.bp
  ui/terminal/Android.bp
  ui/tracer/Android.bp
  xml/Android.bp
  zip/Android.bp
  zip/cmd/Android.bp

Added SPDX-license-identifier-Apache-2.0 SPDX-license-identifier-BSD to:
  finder/fs/Android.bp
  third_party/zip/Android.bp

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all

Exempt-From-Owner-Approval: janitorial work

Change-Id: Ia47ca14f16b8c9f84f9d533a07e5b00e2c04e8d4
2021-02-06 04:23:21 +00:00
Colin Cross caf4d4c270 soong_zip: don't glob RSP file entries
File arguments are considered globs, but RSP file entries should be
considered filenames without globbing.  Escape RSP file entries so
they don't get treated as globs later.

Bug: 179297188
Test: zip_test.go
Change-Id: I1de7829c668538204fc4a2ac793a4f88dbc886c9
2021-02-03 16:06:20 -08:00
Sasha Smundak 8eedba6184 Document usage, perform minor cleanups.
Test: treehugger
Bug: 173466220
Change-Id: I5aa09c2941c0eceb8ae635b9e613ad932d38aebb
2020-11-16 19:00:27 -08:00
Colin Cross 405af07859 Revert "Make lots of tests run in parallel"
This reverts commit 323dc60712.

Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests

Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
2020-10-09 18:34:24 -07:00
Colin Cross 323dc60712 Make lots of tests run in parallel
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.

Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
2020-10-06 15:12:22 -07:00
Colin Cross 053fca10c9 Support ninja rsp files in soong_zip
Add a -r argument to soong_zip that reads a list of files from a file
like the -l argument but treats it as a Ninja rsp file with escaping.
Replace the -l arguments in Soong that are using rsp files with -r.

Fixes: 162435077
Test: TestReadRespFile, TestZip
Change-Id: I4605312e99406ab1bd0c37af9c5ad212393f0403
2020-08-19 21:18:56 +00:00
Thiébaud Weksteen 5e291c238f Fix bpfmt issues and add bpfmt to preupload checks
Test: m nothing
Change-Id: I10526a33685335bdd26cc7692815133379d4e0f6
2020-06-29 09:14:08 +02:00
Jiyong Park 04bbf98e06 -l option soong_zip can accept a file having space separated list
-l option of soong_zip is used to specify the list of input files that
should be zipped. However, it only accepted newline-separated list in the
file. This prevented us from using a rspfile as paths in rspfile are
space-separated in a single line. Fixing the problem by splitting the
file content by any unicode whitespace character including newline and
space.

Test: zip_test
Change-Id: Iba572109e30c01540eacf20bd2794ba60c870fa3
2019-11-04 14:23:07 +09:00
Colin Cross 9cb51dbc0b Support moving sources in srcjars in soong_zip
Add a -srcjar argument to soong_zip that causes it to read the
package statement of each .java file and use that to place the
source file at a path that matches the package.

Test: jar_test.go, zip_test.go
Change-Id: I36017e42445ba3b0a82a10a8d81e8ac0cca096f2
2019-06-18 13:33:20 -07:00
Dan Willemsen 7f276c6b77 Fix soong_zip --help
It was not printing out the flag descriptions because it was using the
default FlagSet instead of the custom FlagSet when calling
PrintDefaults.

Test: soong_zip --help
Change-Id: I8a5705f8701ce0c05dde144b9a14962faa384e65
2019-04-20 21:45:51 -07:00
Dan Willemsen edc934ccc9 Fix soong_zip printing warnings with --ignore_missing_files
We weren't defaulting args.Stderr to os.Stderr, so we had been calling
fmt.Fprintln with `nil`.

Test: soong_zip --ignore_missing_files -C test/ -f abc -o test.zip
Change-Id: I0421f4bfb07fa36edf86c68ffc847995777c8221
2018-12-27 12:41:25 -08:00
Colin Cross 4be8f9e2a3 soong_zip: add --ignore_missing_files flag
soong_zip builds a list of files to zip early and then starts
zipping them all.  If a directory being zipped is concurrently
modified, a file that existed when soong_zip started may not
still exist.  Add a flag that continues when an expected file
does not exist.  Print a warning, since this should be rare
in normal usages but is a sign of a problem if it happens
regularly.

Test: zip_test.go
Test: m checkbuild
Test: m platform
Change-Id: I78426fe66fded8528ddd436c0f71a7442183cfeb
2018-09-28 16:14:52 -07:00
Colin Cross 09f11056f8 Add a --symlinks argument to soong_zip
Add a --symlinks argument that defaults to true to soong_zip.
Passing --symlinks=false will cause it to follow symlinks instead
of storing them in the zip file.

Relands I4deb98daa9d4ba9f94e3d7670c117fe00381d2ba with tests.

Bug: 112843624
Test: glob_test.go
Test: zip_test.go
Test: m checkbuild
Change-Id: I0eff9c1f2dba79e873fda381ff585df55d5aaaad
2018-09-28 14:01:33 -07:00
Colin Cross 1d98ee23a3 soong_zip: support globs in -f and -D arguments
-f and -D arguments can now take globs in the Soong format.

Also update the use of soong_zip that jars resources to escape the
globs in the arguments, and then shell-escape them when writing to
the rsp file so the glob escape are not intepreted by ReadRespFile.

Also remove an unused argument to the buildAAR rule that could
have contained values that needed escaping.

Relands I7f20bb169dc01f952d2a7681ec6ee9c05737ed37 with a fix for
trailing "\n" in list files, which causes a call to pathtools.Glob("")
that returns "./", which could then get incorrectly translated to
"../../../" in the zip file. Also adds tests.

Test: m checkbuild
Test: zip_test.go
Change-Id: I54b8eef9231875e6042a32c9f8bcc5c2f779922a
2018-09-28 14:01:29 -07:00
Colin Cross 05518bc13b soong_zip: Add tests
Add test that cover basic command line usage of soong_zip.  -D
is not covered yet as the implementation will be replaced with
one that is also more easily testable in the next patch.

Bug: 116751500
Test: zip_test.go
Change-Id: I5a1bcee74ebc9cb3cf332c36f89bc12c0e807ad2
2018-09-28 13:56:06 -07:00
Colin Cross b051ab5cb5 soong_zip: move profiling from zip library to soong_zip
Profiling is only used by the standalone soong_zip command,
move it out of the shared zip library.

Bug: 116751500
Test: m checkbuild
Change-Id: I443c34fb39cf8955e163a7720d6f7ed585e4172a
2018-09-28 13:56:06 -07:00
Colin Cross fe945b4401 soong_zip: move args parsing into zip.FileArgsBuilder
Add a builder that can construct []FileArg for zip.Run to use.
This will help writing tests that cover the command line parsing.

Test: later cl
Bug: 116751500
Change-Id: I15b5144f5094ab154f13de9c8a84b82db2da4e67
2018-09-28 13:56:06 -07:00
Colin Cross 7f33b81ef6 Revert "soong_zip: support globs in -f and -D arguments"
This reverts commit 08e28abc4e.

Reason for revert: inserts extra ../../ entries in the zip file.

Bug: 116737386
Change-Id: I048f94889a66fe618058e37a5827d0cf6b1a6b68
2018-09-26 21:38:07 +00:00
Colin Cross c7feeff1e9 Revert "Add a --symlinks argument to soong_zip"
This reverts commit d59dab94c4.

Bug: 112843624
Change-Id: I3d0f1b61e899b162e65c18662f9d27dd794a9a30
2018-09-26 21:36:44 +00:00
Colin Cross d59dab94c4 Add a --symlinks argument to soong_zip
Add a --symlinks argument that defaults to true to soong_zip.
Passing --symlinks=false will cause it to follow symlinks instead
of storing them in the zip file.

Bug: 112843624
Test: glob_test.go
Test: m checkbuild
Change-Id: I4deb98daa9d4ba9f94e3d7670c117fe00381d2ba
2018-09-21 16:08:16 -07:00
Colin Cross 08e28abc4e soong_zip: support globs in -f and -D arguments
-f and -D arguments can now take globs in the Soong format.

Also update the use of soong_zip that jars resources to escape the
globs in the arguments, and then shell-escape them when writing to
the rsp file so the glob escape are not intepreted by ReadRespFile.

Also remove an unused argument to the buildAAR rule that could
have contained values that needed escaping.

Test: m checkbuild
Change-Id: I7f20bb169dc01f952d2a7681ec6ee9c05737ed37
2018-09-21 16:08:16 -07:00
Colin Cross b7c6911dd1 soong_zip: add support for -j to junk paths
When -j is specified ignore the path to the source file and just
zip it with its filename.  -j overrides -C, and -C overrides -j,
so -j -f path1 -C dir -f path2 will junk the path for path1, but
treat path2 as relative to dir.

Remove the filepath.Clean for the FileArgs, it would convert ""
to "." sometimes, and everything gets cleaned in zip already for
the non-command-line use cases.

Test: m checkbuild
Change-Id: I7d90572c622ee0c6f35967ff31d067b5312c72eb
2018-09-19 15:40:09 -07:00
Colin Cross df3f4c81f3 soong_zip: move the -j flag to -parallel
The -j flag is going to be repurposed to match --junk-paths in zip.

Test: m checkbuild
Change-Id: Id3809d6e5e3da375531269f2372eefc565091cc2
2018-09-18 16:31:09 -07:00
Colin Cross d321629155 Don't attempt to deflate when compression level is 0
soong_zip -L 0 would attempt to compress with deflate at compression
level 0, then pick store instead of deflate because the overhead of
deflate would always make it larger than the uncompressed size.  Just
use store instead.

Test: m checkbuild
Change-Id: I69610bd755b55d97a47b8c6cf4fbbee3b5632db6
2018-09-14 20:20:13 -07:00
Colin Cross 297d9bceda soong_zip: set local header crc and size for symlinks
Getting a crc and size into the local header requires setting it
before writing the payload, or using a streaming data header after
the payload with the crc and size.  Stored (uncompressed) entries
are not allowed to use a streaming data header.

Symlinks are always stored uncompressed, so set the crc and size
in the file header before writing the payload.

Also set the mode to 0777 to match the behavior of zip.

This relands I66c5d41f0a5b23b828d6a03a3790afedc5a97625 with fixes
for the size and mode.

Test: m checkbuild
Test: zipdetails on zip with symlink created with soong_zip has same
      crc in local header and central header.
Test: Compare zipdetails output of zip containing symlink created
      by soong_zip and by zip --symlinks -X.
Bug: 110716403
Change-Id: Iec0bc9056a0d9cdab76f015844aca9c711e72e5b
2018-06-22 23:16:16 -07:00
Colin Cross 5aff065626 Revert "soong_zip: set local header crc for symlinks"
This reverts commit b88c4dfb87.

This made it worse, now that the local header crc matches the global
header, unzip will attempt to unzip the symlink.  However, the local
header size is still incorrect, so it produces an empty file instead
of a symlink.

Bug: 110716403
Change-Id: Ifd9aabc04d180b7ead4aec8566c151206b6b7c57
2018-06-23 06:14:01 +00:00
Colin Cross b88c4dfb87 soong_zip: set local header crc for symlinks
Getting a crc into the local header requires setting it before writing
the payload, or using a streaming data header after the payload with
the crc.  Stored (uncompressed) entries are not allowed to use a
streaming data header.

Symlinks are always stored uncompressed, so set the crc in the file
header before writing the payload.

Test: m checkbuild
Test: zipdetails on zip with symlink created with soong_zip has same
      crc in local header and central header.
Bug: 110716403
Change-Id: I66c5d41f0a5b23b828d6a03a3790afedc5a97625
2018-06-23 03:23:19 +00:00
Nan Zhang 674dd932d8 Add Respfile support for soong_zip.
Sometime the size of our command line passed to soong_zip go program
exceeds the cmdline size limit. So add an RespFile support with "@" special
character prefix.

The args in the cmdline will be considered together with the
args in RespFile during soong_zip running.

Test: real tests in my local machine, and compare the
res/libphonenumber.jar before and after changes.

./cmd -o test.zip '""'-C -> [./cmd,-o,test.zip,""-C]
./cmd -o test.zip '-C -f -> [./cmd,-o,test.zip,-C -f]
./cmd -o test.zip '\"'-C -f -> [./cmd,-o,test.zip,\"-C -f]
./cmd -o test.zip '\\'-C -f -> [./cmd,-o,test.zip,\\-C -f]
./cmd -o test.zip '\a'-C -f -> [./cmd,-o,test.zip,\a-C -f]

./cmd -o test.zip \'-C -> [./cmd,-o,test.zip,'-C]
./cmd -o test.zip \\-C -> [./cmd,-o,test.zip,\-C]
./cmd -o test.zip \"-C -> [./cmd,-o,test.zip,"-C]

./cmd -o test.zip "'"-C -> [./cmd,-o,test.zip,'-C]
./cmd -o test.zip "\\"-C -f -> [./cmd,-o,test.zip,\a-C -f]
./cmd -o test.zip "\""-C -f -> [./cmd,-o,test.zip,"a-C -f]

Bug: b/72484223

Change-Id: I83c3630b70c8396c8e8a3f266244d868d754c4e8
2018-02-13 15:13:01 -08:00
Colin Cross f83c150269 Add --write_if_changed argument to soong_zip
Simplify making restat rules by adding a --write_if_changed argument
to soong_zip that generates the output file into memory and then
compares it to the version on disk before writing it out.

Test: builds
Change-Id: I5059a1e3a11e79b0ca538b3b829bc7479c126ce6
2017-11-13 15:11:08 -08:00
Jeff Gaston 11b5c51d4e split soong_zip into a library and a binary
to make it faster/easier to invoke from other Go programs
(such as multiproduct_kati)

Bug: 67478260
Test: m -j
Change-Id: Idd2671a44290550197c88f53dd11a6dd39c85cc5
2017-10-19 18:18:04 +00:00