d77c99ebc3 changed MappedFile to return a
bogus zero-length mapping on failure rather than nullptr. None of the
calling code was changed, though, and it seems like doing so would be a
bad idea. Revert that part of the change.
Add missing tests, and tidy up some of the logging. Also remove
single-use or obfuscatory constants from the tests.
The new "empty.zip" was created by using zip(1) to create a zip file
with one entry, then using `zip -d` to remove it.
The new "zero-size-cd.zip" was created by using zip(1) to create a zip
file containing a single empty file, and then hex editing the two byte
"size of the central directory" field in the "end of central directory
record" structure at the end of the file. (This is equivalent to, but
much smaller than, the example zip file provided by the bug reporter.)
Bug: http://b/145925341
Test: treehugger
Change-Id: Iff64673bce7dae886ccbc9dd6c2bbe18de19f9d2
golang doesn't include Unix mode by default.
Also show all the deflate variants ("defN" versus "defX").
Cope better with being called directly rather than via symlink.
Test: manual
Change-Id: I23b441c847ce9a557ea866b3c43bdf0542b26f10
Useful for debugging and hermetic builds. (Various places in the build
check to see that a file was stored uncompressed.)
Test: manual
Change-Id: I127e5689cd493ab06739b765beed50912dc9cc1d
Didn't find anything when I ran it, but it did get me to fix the
const/non-const void* in the API.
Test: treehugger
Change-Id: If3849d974965e3e5ffcbdaf5e47921316d717410
All but one existing caller actually wants a std::string.
Bug: http://b/129068177
Test: treehugger
Change-Id: I428c4453edaae74451db56e9542e4e462f08d43a
Same issue as with FindEntry: using ZipString in the API forces all
callers to make sure they don't hit the ZipString length limits. Switch
to std::string_view and uniformly use the empty string as a way to
signal no prefix/suffix rather than nullptr.
Also use default arguments to make the common case of no prefix and no
suffix more convenient.
Also just use std::string to increase the lifetime of the provided
prefix/suffix rather than manual memory management.
Bug: http://b/129068177
Test: treehugger
Change-Id: I6675e39ce62fadd766386d77d27423013c17d6f7
Switch FindEntry and the ZipString constructor to std::string_view. This
lets us accept an over-long name so that we can reject it as too long.
Also fastboot changes to track the API change.
Bug: http://b/129068177
Test: treehugger
Change-Id: I7df7acd1fe2c46380b789c25f8909e0553e2d55e
Enable -Wconversion (but not -Wsign-conversion). Fix up code. Handle
some actual error cases:
* too many files
* files too large
Bug: 130039052
Test: atest ziparchive-tests
Change-Id: I632af317b9e65dbc728851efefd0d11a2b5c29b9
Although ubsan's implicit-unsigned-integer-truncation sanitizer may be
happy, this code still performs an implicit conversion from a wider
width data structure to a narrower width data structure. Rather than
masking the bits, make the conversion explicit. This keeps ubsan happy
as well as addressing a -Wconversion warning.
This change addresses comments from the post-submit review
of a4e5433660.
Test: compiles and boots.
Bug: 122975762
Change-Id: I1fa6d6f8a6fcfb93ba9916b7d2b3564ca1d8caf3
std::hash returns a 64 bit value, which is truncated to a 32 bit value
in ComputeHash. ubsan's implicit-unsigned-integer-truncation doesn't
like this implicit truncation and crashes the program. Explicitly strip
off the top order bits after computing the hash.
Remove the windows specific version of the hash computation. The windows
compile now uses clang, so this code is obsolete. This also avoids us
having to add __attribute__((no_sanitize("integer"))) to the windows
code.
This is needed to support Android booting with ubsan's
implicit-unsigned-integer-truncation option enabled.
Test: compiles and boots
Bug: 122975762
Change-Id: I2f05fbf5ffee8e90a66a6fda32e80de9cca246c0
The process of determining whether or not to emit a safetynet error
entry while processing a malformed file relies on addition overflow.
Since this is only logging, and logging which isn't used, delete the
code instead of trying to fix the logic which is causing the integer
overflow.
This change is necessary to enable integer sanitization on this code.
Somewhat related to Bug: 122975762
Test: atest ziparchive-tests
Change-Id: I6b41ccf7881348cb4e5236324eaa44a05662a725
Prevent file descriptors from leaking across an exec() boundary.
Bug: 120983106
Test: compiles and boots
Change-Id: I392b0767674b557b1e4404a2ba63bc48e3e37b24
system/core/libziparchive/zip_archive.cc:847:36: error: use of undeclared identifier 'lseek64'; did you mean 'lseek'?
const off64_t current_offset = lseek64(fd, 0, SEEK_CUR);
Bug: N/A
Test: builds
Change-Id: If762011722d53376bb6dab35c6ee8031762e5a95
This allows us to remove libziparchive's dependency on libutils.
Bug: http://b/79112958
Test: ran libbase and libziparchive tests, ran fastboot manually
Change-Id: I95c651976dad222863e5b8c37d4514b778f5dce7
A typedef to void* allows an implicit conversion from ZipArchiveHandle*
(or any other pointer type) to ZipArchiveHandle.
See I95d79809b6e118fb3c39c7b98b8055c8e324db1a in platform/bionic.
Bug: none
Test: m checkbuild
Change-Id: I3dd426cb64c46ef81e1dd81b4a2e4f40ac2701df
Debugging memory allocations on the nexus launcher unveiled significant memory allocations for the hashatable used in libziparchive, ~1MB.
This is partly because of the ZipString struct storing an entry in the table. The struct stored a pointer to a string (on 64 bit, 8 bytes) and the length to read from that pointer, 2 bytes. Because of alignment, the structure consumed 16 bytes, wasting 6 bytes.
Now, we store entries in the hashtable as a ZipStringOffset. This new structure stores a 4 byte offset from a fixed location in the memory mapped file instead of the entire address, consuming 8 bytes with alignment.
Bug: 79416399
Test: Builds successfully and manual testing by opening launcher on Pixel 2 shows precisely 50% decrease in memory allocated for the hashtable. From 909312 bytes to 454656.
Change-Id: I28b43699233fbee7f63fccae2d4fe96fcc07e5c4
Allows for opening zip files usng paths longer than 260 characters and
with unicode characters on Windows.
Bug: 113110184
Test: manual and libziparchive_tests
Change-Id: I9ce96ac2f1b1e448ae2a2f69c1d4cb3395ea79ee
Both Extract...() functions don't need dynamic allocation
for the writers, as those are strictly scoped. This CL
changes heap allocation to stack allocation.
Test: zip_archive_test
Change-Id: Id727e4b9848235cd063cc67ecbe052d21ca21326
Only compute the crc32 if required. In addition :
- Add unit tests for Inflate that cover this addition.
- Fix an inconsistency in return codes that was revealed
by this new test.
Bug: 35246701
Test: zip_archive_tests
Test: make; zipalign.
Merged-In: I31d7554378f94fc8995f707471d57cb98311e2c2
Change-Id: I05111bfa665c610f93d1c1dee987a509bf87aa65
Previously fastboot would carry on regardless if decompression failed:
fastboot: archive does not contain 'vbmeta.img'
fastboot: extracting vendor.img (260 MB)...
fastboot: W/ziparchive(56777): Zip: unable to allocate 272781472 bytes at offset 0 : No space left on device
fastboot: failed to extract 'vendor.img': I/O error
fastboot: archive does not contain 'vendor_other.img'
fastboot: wiping userdata...
This is because all but "boot" and "system" are considered "optional",
and the implementation of "optional" was "ignore any failures". What it
_should_ have meant was "it's okay if these don't exist, but if they do,
failures matter".
Fix this logic, use die() more aggressively, and remove spurious "\n"s
from die() format strings.
Also fix spurious whitespace in the libziparchive format string. Before:
Zip: unable to allocate 272781472 bytes at offset 0 : No space left on device
After:
Zip: unable to allocate 272781472 bytes at offset 0: No space left on device
Bug: http://b/68383022
Test: `fastboot update` on marlin
Change-Id: I3cbf55f1a33ca125f293f873eafbcfb86c880ba8
Remove dependencies on zip_archive specific data structures. In follow
up changes, this method will be promoted to a public API and used from
androidfw/ZipUtils as well as tools/zipalign, thereby allowing us to
remove a near-exact copy of this code and to simplify depdencies.
Test: zip_archive_test
Bug: 35246701
Change-Id: If24a9965fbd6fff308c8758859026684fd2af3b9
The use of ReadAtOffset is meant to allow concurrent access
to the zip archive once it has been loaded. There were places
where this was the case, and some places that did a seek + read
combination, which could lead to data races.
NOTE: On Windows, we are not using pread as the implementation of
ReadAtOffset, therefore the guarantees on Windows are weaker.
On Linux, pread allows the file descriptor to be read at a specific
offset without changing the read pointer. This allows inherited fd's
and duped fds to be read concurrently.
On Windows, we use the ReadFile API, which allows for an atomic seek +
read operation, but modifies the read pointer. This means that any mix
use of ReadAtOffset and Read will have races. Just using ReadAtOffset is
safe.
For the Windows case, this is fine as the libziparchive code now only
uses ReadAtOffset.
Bug: 62184114
Bug: 62101783
Test: make ziparchive-tests (existing tests pass)
Change-Id: Ia7f9a30af2216682cdd9d578d26e84bc46773bb9
We should check the boundary of central directory before checking its
signature. Swap the order of these two checks.
Bug: 36392138
Test: libziparchive doesn't read the signature after boundary check fails.
Merged-In: Ie89f709bb2d1ccb647116fb7ccb1e23c943e5ab8
Change-Id: Ie89f709bb2d1ccb647116fb7ccb1e23c943e5ab8
(cherry picked from commit 74464a1361)
Add a check to ensure the filename boundary doesn't exceed the mapped
memory region. Also add the corresponding unit test.
Bug: 28802225
Test: New unit test passes.
Merged-In: Ibf543a7da3d7898952e9eb332c84cdfc67cf5aa4
Change-Id: Ibf543a7da3d7898952e9eb332c84cdfc67cf5aa4
(cherry picked from commit bcc4431f24)
Make them available to tests so that they can compare values
directly instead of comparing error code strings.
Test: zip_archive_test
Change-Id: I1c78c24651ff7549b5125a49a4e67bbd8295c617
- Ensure that the compressed and uncompressed lengths in the data
descriptor match those in the central directory.
- Calculate CRCs for compressed entries and ensure that those CRCs
match the information in the central directory / data descriptor.
This is currently guarded by a flag and will be turned on at a
later stage.
- Finally, always trust the LFH when it comes to indicating the
existence of a data descriptor. We no longer require this info to
match the central directory to bring this in line with other zip
processing tools. (see NOTE in the included change).
Bug: 62047801
Test: zip_archive_test
Change-Id: Icceea66a8b1937c08a4af9aee97bc77ad7218c0f
The Central File Header and Local File Header should
have the same general purpose bit flags set for
consistency.
Bug: 36686974
Test: existing tests pass (ziparchive-tests)
(cherry picked from commit 73b3aa541f)
Change-Id: Ia672a28732154a564ea2f2e3520238bb77924a56
We change the GPB in the LocalFileHeader if the entry can not
have a trailing DataDescriptor. Make sure to patch the
CentralFileHeader to have the same bits set.
Modify ZipArchive to check that the data descriptor bit is consistent
between Central and Local file headers.
(cherry-pick of commit e0eca55fe6)
Test: make ziparchive-tests
Bug: 36686974
Change-Id: Ied167570abcf6426b1c678cd40123e5ad65909db
Elliott Hughes