mirror of https://gitee.com/openkylin/linux.git
kcsan: Add Documentation entry in dev-tools
Signed-off-by: Marco Elver <elver@google.com> Acked-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit is contained in:
Marco Elver
Paul E. McKenney
parent
c48981eeb0
commit
905e672b3a
|
|
@ -21,6 +21,7 @@ whole; patches welcome!
|
||||||
kasan
|
kasan
|
||||||
ubsan
|
ubsan
|
||||||
kmemleak
|
kmemleak
|
||||||
|
kcsan
|
||||||
gdb-kernel-debugging
|
gdb-kernel-debugging
|
||||||
kgdb
|
kgdb
|
||||||
kselftest
|
kselftest
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,256 @@
|
||||||
|
The Kernel Concurrency Sanitizer (KCSAN)
|
||||||
|
========================================
|
||||||
|
|
||||||
|
Overview
|
||||||
|
--------
|
||||||
|
|
||||||
|
*Kernel Concurrency Sanitizer (KCSAN)* is a dynamic data race detector for
|
||||||
|
kernel space. KCSAN is a sampling watchpoint-based data race detector. Key
|
||||||
|
priorities in KCSAN's design are lack of false positives, scalability, and
|
||||||
|
simplicity. More details can be found in `Implementation Details`_.
|
||||||
|
|
||||||
|
KCSAN uses compile-time instrumentation to instrument memory accesses. KCSAN is
|
||||||
|
supported in both GCC and Clang. With GCC it requires version 7.3.0 or later.
|
||||||
|
With Clang it requires version 7.0.0 or later.
|
||||||
|
|
||||||
|
Usage
|
||||||
|
-----
|
||||||
|
|
||||||
|
To enable KCSAN configure kernel with::
|
||||||
|
|
||||||
|
CONFIG_KCSAN = y
|
||||||
|
|
||||||
|
KCSAN provides several other configuration options to customize behaviour (see
|
||||||
|
their respective help text for more info).
|
||||||
|
|
||||||
|
Error reports
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
A typical data race report looks like this::
|
||||||
|
|
||||||
|
==================================================================
|
||||||
|
BUG: KCSAN: data-race in generic_permission / kernfs_refresh_inode
|
||||||
|
|
||||||
|
write to 0xffff8fee4c40700c of 4 bytes by task 175 on cpu 4:
|
||||||
|
kernfs_refresh_inode+0x70/0x170
|
||||||
|
kernfs_iop_permission+0x4f/0x90
|
||||||
|
inode_permission+0x190/0x200
|
||||||
|
link_path_walk.part.0+0x503/0x8e0
|
||||||
|
path_lookupat.isra.0+0x69/0x4d0
|
||||||
|
filename_lookup+0x136/0x280
|
||||||
|
user_path_at_empty+0x47/0x60
|
||||||
|
vfs_statx+0x9b/0x130
|
||||||
|
__do_sys_newlstat+0x50/0xb0
|
||||||
|
__x64_sys_newlstat+0x37/0x50
|
||||||
|
do_syscall_64+0x85/0x260
|
||||||
|
entry_SYSCALL_64_after_hwframe+0x44/0xa9
|
||||||
|
|
||||||
|
read to 0xffff8fee4c40700c of 4 bytes by task 166 on cpu 6:
|
||||||
|
generic_permission+0x5b/0x2a0
|
||||||
|
kernfs_iop_permission+0x66/0x90
|
||||||
|
inode_permission+0x190/0x200
|
||||||
|
link_path_walk.part.0+0x503/0x8e0
|
||||||
|
path_lookupat.isra.0+0x69/0x4d0
|
||||||
|
filename_lookup+0x136/0x280
|
||||||
|
user_path_at_empty+0x47/0x60
|
||||||
|
do_faccessat+0x11a/0x390
|
||||||
|
__x64_sys_access+0x3c/0x50
|
||||||
|
do_syscall_64+0x85/0x260
|
||||||
|
entry_SYSCALL_64_after_hwframe+0x44/0xa9
|
||||||
|
|
||||||
|
Reported by Kernel Concurrency Sanitizer on:
|
||||||
|
CPU: 6 PID: 166 Comm: systemd-journal Not tainted 5.3.0-rc7+ #1
|
||||||
|
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
|
||||||
|
==================================================================
|
||||||
|
|
||||||
|
The header of the report provides a short summary of the functions involved in
|
||||||
|
the race. It is followed by the access types and stack traces of the 2 threads
|
||||||
|
involved in the data race.
|
||||||
|
|
||||||
|
The other less common type of data race report looks like this::
|
||||||
|
|
||||||
|
==================================================================
|
||||||
|
BUG: KCSAN: data-race in e1000_clean_rx_irq+0x551/0xb10
|
||||||
|
|
||||||
|
race at unknown origin, with read to 0xffff933db8a2ae6c of 1 bytes by interrupt on cpu 0:
|
||||||
|
e1000_clean_rx_irq+0x551/0xb10
|
||||||
|
e1000_clean+0x533/0xda0
|
||||||
|
net_rx_action+0x329/0x900
|
||||||
|
__do_softirq+0xdb/0x2db
|
||||||
|
irq_exit+0x9b/0xa0
|
||||||
|
do_IRQ+0x9c/0xf0
|
||||||
|
ret_from_intr+0x0/0x18
|
||||||
|
default_idle+0x3f/0x220
|
||||||
|
arch_cpu_idle+0x21/0x30
|
||||||
|
do_idle+0x1df/0x230
|
||||||
|
cpu_startup_entry+0x14/0x20
|
||||||
|
rest_init+0xc5/0xcb
|
||||||
|
arch_call_rest_init+0x13/0x2b
|
||||||
|
start_kernel+0x6db/0x700
|
||||||
|
|
||||||
|
Reported by Kernel Concurrency Sanitizer on:
|
||||||
|
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.3.0-rc7+ #2
|
||||||
|
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
|
||||||
|
==================================================================
|
||||||
|
|
||||||
|
This report is generated where it was not possible to determine the other
|
||||||
|
racing thread, but a race was inferred due to the data value of the watched
|
||||||
|
memory location having changed. These can occur either due to missing
|
||||||
|
instrumentation or e.g. DMA accesses.
|
||||||
|
|
||||||
|
Selective analysis
|
||||||
|
~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
To disable KCSAN data race detection for an entire subsystem, add to the
|
||||||
|
respective ``Makefile``::
|
||||||
|
|
||||||
|
KCSAN_SANITIZE := n
|
||||||
|
|
||||||
|
To disable KCSAN on a per-file basis, add to the ``Makefile``::
|
||||||
|
|
||||||
|
KCSAN_SANITIZE_file.o := n
|
||||||
|
|
||||||
|
KCSAN also understands the ``data_race(expr)`` annotation, which tells KCSAN
|
||||||
|
that any data races due to accesses in ``expr`` should be ignored and resulting
|
||||||
|
behaviour when encountering a data race is deemed safe.
|
||||||
|
|
||||||
|
debugfs
|
||||||
|
~~~~~~~
|
||||||
|
|
||||||
|
* The file ``/sys/kernel/debug/kcsan`` can be read to get stats.
|
||||||
|
|
||||||
|
* KCSAN can be turned on or off by writing ``on`` or ``off`` to
|
||||||
|
``/sys/kernel/debug/kcsan``.
|
||||||
|
|
||||||
|
* Writing ``!some_func_name`` to ``/sys/kernel/debug/kcsan`` adds
|
||||||
|
``some_func_name`` to the report filter list, which (by default) blacklists
|
||||||
|
reporting data races where either one of the top stackframes are a function
|
||||||
|
in the list.
|
||||||
|
|
||||||
|
* Writing either ``blacklist`` or ``whitelist`` to ``/sys/kernel/debug/kcsan``
|
||||||
|
changes the report filtering behaviour. For example, the blacklist feature
|
||||||
|
can be used to silence frequently occurring data races; the whitelist feature
|
||||||
|
can help with reproduction and testing of fixes.
|
||||||
|
|
||||||
|
Data Races
|
||||||
|
----------
|
||||||
|
|
||||||
|
Informally, two operations *conflict* if they access the same memory location,
|
||||||
|
and at least one of them is a write operation. In an execution, two memory
|
||||||
|
operations from different threads form a **data race** if they *conflict*, at
|
||||||
|
least one of them is a *plain access* (non-atomic), and they are *unordered* in
|
||||||
|
the "happens-before" order according to the `LKMM
|
||||||
|
<../../tools/memory-model/Documentation/explanation.txt>`_.
|
||||||
|
|
||||||
|
Relationship with the Linux Kernel Memory Model (LKMM)
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The LKMM defines the propagation and ordering rules of various memory
|
||||||
|
operations, which gives developers the ability to reason about concurrent code.
|
||||||
|
Ultimately this allows to determine the possible executions of concurrent code,
|
||||||
|
and if that code is free from data races.
|
||||||
|
|
||||||
|
KCSAN is aware of *atomic* accesses (``READ_ONCE``, ``WRITE_ONCE``,
|
||||||
|
``atomic_*``, etc.), but is oblivious of any ordering guarantees. In other
|
||||||
|
words, KCSAN assumes that as long as a plain access is not observed to race
|
||||||
|
with another conflicting access, memory operations are correctly ordered.
|
||||||
|
|
||||||
|
This means that KCSAN will not report *potential* data races due to missing
|
||||||
|
memory ordering. If, however, missing memory ordering (that is observable with
|
||||||
|
a particular compiler and architecture) leads to an observable data race (e.g.
|
||||||
|
entering a critical section erroneously), KCSAN would report the resulting
|
||||||
|
data race.
|
||||||
|
|
||||||
|
Race conditions vs. data races
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Race conditions are logic bugs, where unexpected interleaving of racing
|
||||||
|
concurrent operations result in an erroneous state.
|
||||||
|
|
||||||
|
Data races on the other hand are defined at the *memory model/language level*.
|
||||||
|
Many data races are also harmful race conditions, which a tool like KCSAN
|
||||||
|
reports! However, not all data races are race conditions and vice-versa.
|
||||||
|
KCSAN's intent is to report data races according to the LKMM. A data race
|
||||||
|
detector can only work at the memory model/language level.
|
||||||
|
|
||||||
|
Deeper analysis, to find high-level race conditions only, requires conveying
|
||||||
|
the intended kernel logic to a tool. This requires (1) the developer writing a
|
||||||
|
specification or model of their code, and then (2) the tool verifying that the
|
||||||
|
implementation matches. This has been done for small bits of code using model
|
||||||
|
checkers and other formal methods, but does not scale to the level of what can
|
||||||
|
be covered with a dynamic analysis based data race detector such as KCSAN.
|
||||||
|
|
||||||
|
For reasons outlined in this `article <https://lwn.net/Articles/793253/>`_,
|
||||||
|
data races can be much more subtle, but can cause no less harm than high-level
|
||||||
|
race conditions.
|
||||||
|
|
||||||
|
Implementation Details
|
||||||
|
----------------------
|
||||||
|
|
||||||
|
The general approach is inspired by `DataCollider
|
||||||
|
<http://usenix.org/legacy/events/osdi10/tech/full_papers/Erickson.pdf>`_.
|
||||||
|
Unlike DataCollider, KCSAN does not use hardware watchpoints, but instead
|
||||||
|
relies on compiler instrumentation. Watchpoints are implemented using an
|
||||||
|
efficient encoding that stores access type, size, and address in a long; the
|
||||||
|
benefits of using "soft watchpoints" are portability and greater flexibility in
|
||||||
|
limiting which accesses trigger a watchpoint.
|
||||||
|
|
||||||
|
More specifically, KCSAN requires instrumenting plain (unmarked, non-atomic)
|
||||||
|
memory operations; for each instrumented plain access:
|
||||||
|
|
||||||
|
1. Check if a matching watchpoint exists; if yes, and at least one access is a
|
||||||
|
write, then we encountered a racing access.
|
||||||
|
|
||||||
|
2. Periodically, if no matching watchpoint exists, set up a watchpoint and
|
||||||
|
stall for a small delay.
|
||||||
|
|
||||||
|
3. Also check the data value before the delay, and re-check the data value
|
||||||
|
after delay; if the values mismatch, we infer a race of unknown origin.
|
||||||
|
|
||||||
|
To detect data races between plain and atomic memory operations, KCSAN also
|
||||||
|
annotates atomic accesses, but only to check if a watchpoint exists
|
||||||
|
(``kcsan_check_atomic_*``); i.e. KCSAN never sets up a watchpoint on atomic
|
||||||
|
accesses.
|
||||||
|
|
||||||
|
Key Properties
|
||||||
|
~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
1. **Memory Overhead:** The current implementation uses a small array of longs
|
||||||
|
to encode watchpoint information, which is negligible.
|
||||||
|
|
||||||
|
2. **Performance Overhead:** KCSAN's runtime aims to be minimal, using an
|
||||||
|
efficient watchpoint encoding that does not require acquiring any shared
|
||||||
|
locks in the fast-path. For kernel boot on a system with 8 CPUs:
|
||||||
|
|
||||||
|
- 5.0x slow-down with the default KCSAN config;
|
||||||
|
- 2.8x slow-down from runtime fast-path overhead only (set very large
|
||||||
|
``KCSAN_SKIP_WATCH`` and unset ``KCSAN_SKIP_WATCH_RANDOMIZE``).
|
||||||
|
|
||||||
|
3. **Annotation Overheads:** Minimal annotations are required outside the KCSAN
|
||||||
|
runtime. As a result, maintenance overheads are minimal as the kernel
|
||||||
|
evolves.
|
||||||
|
|
||||||
|
4. **Detects Racy Writes from Devices:** Due to checking data values upon
|
||||||
|
setting up watchpoints, racy writes from devices can also be detected.
|
||||||
|
|
||||||
|
5. **Memory Ordering:** KCSAN is *not* explicitly aware of the LKMM's ordering
|
||||||
|
rules; this may result in missed data races (false negatives).
|
||||||
|
|
||||||
|
6. **Analysis Accuracy:** For observed executions, due to using a sampling
|
||||||
|
strategy, the analysis is *unsound* (false negatives possible), but aims to
|
||||||
|
be complete (no false positives).
|
||||||
|
|
||||||
|
Alternatives Considered
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
An alternative data race detection approach for the kernel can be found in
|
||||||
|
`Kernel Thread Sanitizer (KTSAN) <https://github.com/google/ktsan/wiki>`_.
|
||||||
|
KTSAN is a happens-before data race detector, which explicitly establishes the
|
||||||
|
happens-before order between memory operations, which can then be used to
|
||||||
|
determine data races as defined in `Data Races`_. To build a correct
|
||||||
|
happens-before relation, KTSAN must be aware of all ordering rules of the LKMM
|
||||||
|
and synchronization primitives. Unfortunately, any omission leads to false
|
||||||
|
positives, which is especially important in the context of the kernel which
|
||||||
|
includes numerous custom synchronization mechanisms. Furthermore, KTSAN's
|
||||||
|
implementation requires metadata for each memory location (shadow memory);
|
||||||
|
currently, for each page, KTSAN requires 4 pages of shadow memory.
|
||||||
Loading…
Reference in New Issue