Make this function available to libdm users. A caller outside libdm will
be added by a later patch.
Bug: 194450129
Test: mm libfs_mgr libdm_test
Merged-In: I3e3560f3cdef8978eac644d5b53cf3851209c0c2
Change-Id: Ic05cc84565952662178bb649ec97cad6f76dcf92
Ignore-AOSP-First: Already in AOSP.
Signed-off-by: Bart Van Assche <bvanassche@google.com>
From https://engdoc.corp.google.com/eng/doc/devguide/cpp/styleguide.md:
"Prefer using return values over output parameters: they improve
readability, and often provide the same or better performance (see the C++
Primer)." Implement this advice for ExtractBlockDeviceName(). This patch
does not change any functionality.
Bug: 194450129
Test: mm libfs_mgr libdm_test
Merged-In: I6363781163eba08e6128507b403200f472f68a59
Change-Id: I7d340b33281ebccded0836cd0b5a293e941f4043
Ignore-AOSP-First: Already in AOSP.
Signed-off-by: Bart Van Assche <bvanassche@google.com>
Improve code readability by renaming 'device' into 'loop_device' and
'device_fd' into 'loop_fd'.
Bug: 194894000
Test: Built Android images and installed these on an Android device.
Merged-In: Ia9c2d7a525e727f8706e66631b97fc4678c6a4d9
Change-Id: I3fa0c9ca53277b621bb5b81aca394a3079c6e0a3
Ignore-AOSP-First: Already in AOSP.
Signed-off-by: Bart Van Assche <bvanassche@google.com>
This is a corner case wherein a crash during OTA
merge can lead to missing of some COW operations to be
merged thereby some blocks may end up with stale data.
Fix here is to avoid any re-ordering of COW operations.
Merge the COW operations as present in the COW file.
New tests have been added to cow_snapuserd.
Bug: 194955361
Test: cow_snapuserd_test, Incremental OTA
Signed-off-by: Akilesh Kailash <akailash@google.com>
Merged-In: Id895fe7a3d6b4510676490a86d0caf62dec9b079
Change-Id: I14900b9537c4deb7824547e1dfe80f15274bdda4
Ignore-AOSP-First: manual merge from aosp
This reverts commit 0a799bdfd6.
Now that the kernel bootconfig feature has been to updated to handle
mixed subkeys and values, androidboot.hardware parameter is supported.
Test: build and boot Cuttlefish with "androidboot.hardware=cutf_vm"
Bug: 191502832
Merged-In: I0e436a27730d20689bc6974562c3e88d744385db
Change-Id: I0e436a27730d20689bc6974562c3e88d744385db
If for some reason the COW state is not fully synced to disk, but
dm-snapshot has flushed its pending merges, we do not want to delete
snapshots. Doing so could potentially leave blocks unmerged.
This situation is quite unexpected so we label it as a merge failure.
The device can recover by completely syncing the COW state, and then
rebooting, which will attempt to make forward progress on the merge.
Bug: 190582627
Test: vts_libsnapshot_test
full OTA on bramble
incremental OTA on bramble
Change-Id: Ib887f1d9e4397a712ed2f800cc1222cf9305a039
Merged-In: Ib887f1d9e4397a712ed2f800cc1222cf9305a039
adb_debug.prop is migrated too. And ramdisk_available is added to all
dependencies.
Bug: 187196593
Test: boot
Change-Id: I59cd149e0021211b8fd59c44b93bbf18dc8637bf
Merged-In: I59cd149e0021211b8fd59c44b93bbf18dc8637bf
* changes:
first_stage_mount: mount point must be canonical path
fs_mgr_fstab: Parse overlayfs options from fs flags
Remove deprecated fs_mgr_overlayfs_required_devices()
adb-remount-test: Make awk scripts mawk-v1.3.3-compatible
Make fs_mgr_overlayfs_mount_fstab_entry() available for user builds
adb-remount-test: Strengthen skip_administrative_mounts
fs_mgr_overlayfs_mount_fstab_entry(): Rename source device name
fs_mgr_overlayfs: Polish fs_mgr_overlayfs_mount_fstab_entry()
first_stage_mount: Remove "overlay" hack from InitRequiredDevices()
fs_mgr_vendor_overlay: Mount vendor overlay with noatime
Ban weird paths such as /../system or //vendor in first stage mount.
Add utility function fs_mgr_create_canonical_mount_point() that:
* mkdir(mount_point) to ensure mount_point's existence
* Test that realpath(mount_point) =?= mount_point
Bug: 188898525
Test: Presubmit
Test: Boot CF
Change-Id: Iaf2ec52701277f26cc81f3e15a47b6083a788334
Merged-In: Iaf2ec52701277f26cc81f3e15a47b6083a788334
(cherry picked from commit 3431d52675f25020b279c9fbcda6b8648f9cf67b)
This can be useful in case when a device mapper device can't be deleted
straight away, and instead delete needs to be enqueued until last
reference to the device is closed.
Bug: 187864524
Bug: 188713178
Test: atest libdm_test
Change-Id: Ie8a130baa54e6e16d8d159389bd760bf873eca40
Merged-In: Ie8a130baa54e6e16d8d159389bd760bf873eca40
(cherry picked from commit d13cef743545c6de7b5122cb515bb82b508d019e)
Parse the "lowerdir=" option from fs mount flags instead of fs_mgr flags
for consistency.
Before:
none /mnt/product/test1 overlay ro lowerdir=/1:/2,first_stage_mount
After:
none /mnt/product/test1 overlay ro,lowerdir=/1:/2 first_stage_mount
Bug: 188862155
Test: Boot to normal with overlayfs mount entries in first stage fstab
Change-Id: I6d6abd44ab32afadec428005f4aece834f9c8905
Merged-In: I6d6abd44ab32afadec428005f4aece834f9c8905
(cherry picked from commit 7b81023e2a0d48c5e919bb314990555db37bf21e)
It is unused since Ifc8720378259654472d3822e97059b6c366f601d
Bug: 188862155
Test: Build pass
Change-Id: I6304e0dedb36d03a87c556083e937b4a2ce30b1b
Merged-In: I6304e0dedb36d03a87c556083e937b4a2ce30b1b
(cherry picked from commit a07ed968b563f15b35b68c701a97eb239bb34c4c)
Our CI is failing because the host machine is using mawk instead of gawk.
mawk v1.3.3 cannot parse regex such as '/[/]/', while mawk v1.3.4 and
gawk can.
Change regex of '[/]' to '\/' so that our test script is as backward
compatible as possible.
Bug: 188862155
Test: Run adb-remount-test.sh on CI
Change-Id: Ia4fbce58a61325a5e5280ede0d5b7760832d8ec1
Merged-In: Ia4fbce58a61325a5e5280ede0d5b7760832d8ec1
(cherry picked from commit ad3a57bdbaa05fd531a05373a07f0435a9d5dc33)
Rename fs_mgr_overlayfs_mount_fstab_entry() to
fs_mgr_mount_overlayfs_fstab_entry() and move it out of
fs_mgr_overlayfs.cpp to make it available for user builds.
Add checks to unsure overlayfs mount point doesn't contain symbolic
link or /../.
Check the mount point with an allowlist if user build. The mount point
should either be /vendor, /product ... or their submounts, or strict
submounts of /mnt/vendor and /mnt/product.
Bug: 188862155
Test: Boot test with overlayfs mount entries on user build
Change-Id: I3b60dfa4b63cf2ae0754f53d1d08365aa7be1ee0
Merged-In: I3b60dfa4b63cf2ae0754f53d1d08365aa7be1ee0
(cherry picked from commit 23816e84ca8821f303d9a3e753d7c050881bacd5)
Refactor skip_administrative_mounts so that it filters by device name,
mount point and filesystem type. (Was filtering by device name and mount
point.)
We need this because pseudo filesystems such as tmpfs and overlayfs can
have free-formed device name:
blah /mnt/mnt_point overlay ro,lowerdir=...
However the filesystem type (third field of /proc/mounts) must be
reflecting the actual filesystem, so to robustly filter out
administrative filesystems, we have to check the filesystem type field.
Bug: 188862155
Test: adb-remount-test.sh
Change-Id: I5058cbe8a36920f25b73a5d5833e9fc5a096e90f
Merged-In: I5058cbe8a36920f25b73a5d5833e9fc5a096e90f
(cherry picked from commit f931ad94465c24b8056618f85978af14c052079f)
adb-remount-test.sh would fail if a overlayfs named "overlay" is mounted
in first_stage_init via fs_mgr_overlayfs_mount_fstab_entry():
[ FAILED ] overlay takeover unexpected at this phase
Prepend the source device name with "overlay-", so that the overlayfs
mounted by fs_mgr_overlayfs_mount_fstab_entry() mustn't be named
"overlay".
Bug: 188862155
Test: adb-remount-test.sh
Change-Id: Ia0b3f1c92ed5650c3e584fba23758344a4733657
Merged-In: Ia0b3f1c92ed5650c3e584fba23758344a4733657
(cherry picked from commit 7b90e6936c66b67a6761f7ad086a68bbcaf23c05)
* Add logs.
* Append "override_creds=off" overlayfs mount flag only if
fs_mgr_overlayfs_valid() returns kOverrideCredsRequired.
Pre-4.6 kernels or kernels without the override_creds patch don't
need or don't recognize the override_creds mount flag.
(Background: I832c8ca3fce0269bdef4ce988541adb7ba9662ed)
* mkdir(mount_point) before mount() to ensure the mount point exists.
This could happen if the mount point is in a tmpfs, such as /mnt.
Bug: 188862155
Test: Boot to normal with overlayfs mount entries in first stage fstab
Change-Id: I1a05696346610d7fd61de6d25c379520fd58ca9b
Merged-In: I1a05696346610d7fd61de6d25c379520fd58ca9b
(cherry picked from commit dcf1c1f46235637a6d1057c88e6d1fed88e1b90a)
Change relatime to noatime as our filesystems and partitions are all
read-only, so relatime doesn't really work in the first place.
Bug: 188862155
Test: atest -v fs_mgr_vendor_overlay_test
Test: Presubmit
Change-Id: I869bba24c2d6d8a986bf0eec0f487ba7b935ab4b
Merged-In: I869bba24c2d6d8a986bf0eec0f487ba7b935ab4b
(cherry picked from commit 7cbed5a9cec9df1a5ce99f9b61743ffd7e8fe7b7)
Header response from daemon is sent once at the
beginning of the payload. This is a corner case
when the IO request can get broken into multiple chunks
if the IO is un-aligned.
Additionally, add error checks in the IO path
to validate sector information when the read
requests are merged.
Bug: 188361387
Test: Simulate an IO request to forcefully break the IO
response into multiple chunks and verify the correctness.
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I1f4fa7a79c60493f4bbed3ad49e257098b930beb
Merged-In: I1f4fa7a79c60493f4bbed3ad49e257098b930beb
This refactoring moves ImageManager creation out of SnapshotManager,
where it had always been rather awkward, and into IDeviceInfo so it can
participate in dependency injection.
The "first-stage init" state is now part of IDeviceInfo as well.
Bug: 187396878
Test: vts_libsnapshot_test
libsnapshot_fuzzer_test
Change-Id: Ic4b3e93527cc074ec18c0c26440dbdb2504d0d5e
Assertions in daemon will terminate the daemon.
This can bring the entire device to halt as the mount partitions
are mounted of snapshot devices and IO's will be hung in
"D" (uninterruptible state) eventually leading to sysrq crash.
Convert the relevant assertions into appropriate error codes and
propagate the error code back to dm-user.
IO will eventually fail but should not impact the overall usage of the device.
Bug: 187903835
Test: vts_libsnapshot, cow_snapuserd_test, full OTA
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: Iaf093898837d2aff5703ea1e615aecf7c1e53a8f
fs_mgr flags "slotselect" and "slotselect_other" are specific to A/B
devices, so remove these options from testcases because
CtsFsMgrTestCases could be ran on a non-A/B device.
Fixes: 187759127
Fixes: 187427233
Test: atest CtsFsMgrTestCases on sdk_phone64_x86_64-userdebug
Change-Id: I31c1429652629bb68e0ca504295c959ac7e013e2
As part of r.android.com/c/1678745/7, overlapping
copy operations was allowed to batch merge which
is not right. The intention of that CL was
to avoid un-necessary write traffic involved
in flushing data to scratch space. However,
as part of the optimization, copy operations
were merged. More specifically, the problem
comes into play when the number of overlapping
copy operations is more than the read-ahead
window size (2MB).
I have added a new test case to test this
specific code path to avoid future regressions.
Additionally, remove un-necessary "send()"
as part of "detach" response from snapuserd server.
Client is not waiting for any response. It
just creates a race window which is harmless
but error log will be misleading.
Bug: 187506548
Test: cow_snapuserd which tests the similar
configuration as seen in the COW file in the bug
report.
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: Ic0f1ddd390f79966aabfbeadb7d64bc5bb86e83b
These are unit tests for important VAB functionality. They are very
quick to run and should be included in presubmit testing.
Bug: N/A
Test: th green
Change-Id: I02e24c76df365e9fb2d68f904e930dce60b9bdaf
We try to clean up previous test runs, but this can crash since we
haven't opened fake-super yet. Refactor the harness so we always open
fake-super if it exists. If it does, we'll delete and recreate it after
cleanup. If it doesn't, we'll create it immediately.
It's still possible that cleanup can fail: If interrupted during a merge,
libsnapshot does not allow cleanup until the merge completes. The test
harness doesn't bother handling this case yet.
Bug: 187151854
Test: vts_libsnapshot_test, ctrl+c, run again
Change-Id: I58a7094336a391cff493a31e4f80d8c8b1b166f8
libavb is moving to use boringssl for crypto operations in user space
rather than using its own implementation. Link with libcrypto to
resolved the new dependencies.
Test: atest --host --host-unit-test-only --test-mapping system/core/fs_mgr/libfs_avb
Bug: 185329132
Change-Id: I412f4ef677aa6e29c5b67ffe5e3e8377640a2847
This enables read-ahead functionality by having
scratch space in the COW
Bug: 183863613
Test: OTA tests with new COW format
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I7988687c81d0ea239e71695818199db4653ddb80
kCowVersionManifest will be 2. This should now
be in sync with kCowVersionMajor.
Bug: 183863613
Test: OTA with new COW format (by enabling scratch space option)
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: Ia6c31e399de723ee83459b59d6b076b48f5c88d5
* changes:
libsnapshot:snapuserd:Add unit test for read-ahead code path.
libsnapshot: Flush data to scratch space only for overlapping regions
libsnapshot:snapuserd: read-ahead COW copy ops
libsnapshot: Retrieve COW version from update engine manifest
libsnapshot:snapuserd: Add 2MB scratch space in COW file
libsnapshot:snapuserd: mmap + msync header after merge
When read-ahead thread caches the data from base device, flush the data
only if there are overlapping regions. If there is crash, subsequent
reboot will not recover the data from scratch space. Rather, data
will be re-constructed from base device.
Additionally, allow batch merge of blocks by the kernel even for
overlapping region given that we have the read-ahead thread
taking care of the overlapping blocks.
Bug: 183863613
Test: 1: Incremental OTA from build 7284758 to 7288239. Merge time
reduces from ~6 minutes to ~2.5 minutes
2: Reboot and crash kernel multiple times when merge was in
progress
3: Verify read-ahead thread re-constructs the data for overlapping
region.
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I50e0d828f4fb36a23f0ca13b07a73229ba68874d
Introduce read-ahead mechanism for COW copy ops.
1: Read-ahead thread will read from base device
and store the data in scratch space along with the metadata.
2: Worker threads during merge will retrieve the data
from read-ahead cache
3: Fixed set of blocks are read during each cycle by the read-ahead
thread.
4: When the last block in the region is merged, read-ahead thread
makes forward progress.
Scratch space is set to 2MB and is only used from COW copy operations.
We can extend this to Replace Ops based on performance evaluation.
Performance:
As mentioned in bug 181883791, Incremental OTA of size 55M with
235K copy operations where every block is moved by 4k:
Without read-ahead: 40 Minutes for merge completion
With read-ahead: 21 Minutes for merge completion
Bug: 183863613
Test: 1: Full OTA - no regression observed.
2: Incremental OTA - with older COW format. Daemon will just skip
the read-ahead feature for older COW format.
3: Incremental OTA - with new COW format.
4: Reboot and crash kernel when multiple times when incremental OTA is in-flight.
Verify post reboot, read-ahead thread re-constructs the data from scratch
space.
5: No regression observed in RSS-Anon memory usage when merge in-flight.
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: Ic565bfbee3e9fcfc94af694596dbf44c0877639f
update_metadata.proto will have the COW version. Retrieve
that from the manifest and compare it with the COW library.
If the versioning doesn't match, disable VABC.
The primary use case of this is during downgrade tests
in pre-submit. Whenever we have a COW format changes,
we may have to disable VABC for that specific transition
build. At a high level, the flow of version check will be:
1: Create a initial COW version of 1 in manifest (update_metadata.proto)
2: The latest COW version of libsnapshot is 2
3: libsnapshot will return VABC disabled
4: Check-in the CL and changes to manifest
5: Once the CL is baked in and the build is green, bump up the COW version to 2 in the manifest
6: Next set of tests, since both versions match, libsnapshot will enable VABC
7: Downgrade should be done to the build which was checked in at (5)
Bug: 183863613
Test: Apply OTA and verify if VABC is disabled if the versions don't
match
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: Id55f33a90bb31b417e72f4fbe370daf05a68f05a
Add 2MB scratch space in the COW file. This is a preparation
patch for read-ahead patch. This just add the buffer
space right after the header. Bump up the version number
in the header in order to distiguish between older and newer
COW formats.
No operation is done on this buffer with this patch-set.
Scratch space option is disabled by default.
Bug: 183863613
Test: 1: Create Full OTA with the new COW format.
2: Incremental OTA with older COW format.
3: vts_libsnapshot_test
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I42a535a48ec22adb893dfe6f86a4f51650e1f88a
mmap the CowHeader and use msync to flush only the
first 4k page after merge is complete.
This cuts down ~30 seconds of merge completion time
on a 55M incremental OTA with 235k copy operations.
Although, this isn't a significant gain but this patch
creates a scaffolding for the next set of read-ahead patches.
Bug: 183863613
Test: Incremental and Full OTA
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I15bfec91ea1d5bdf4390670bcf406e1015b79299
Bart Van Assche