In the dim distant past, qemu commands needed to be run from the
rcutorture directory, but this is no longer the case. This commit
therefore removes the now-useless "cd $KVM" from the kvm-test-1-run.sh
script.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, the qemu command is constructed twice, once to dump it
to the qemu-cmd file and again to execute it. This is of course an
accident waiting to happen, but is done to ensure that the remainder
of the script has an accurate idea of the running qemu command's PID.
This commit therefore places both the qemu command and the PID capture
into a new temporary file and sources that temporary file. Thus the
single construction of the qemu command into the qemu-cmd file suffices
for both purposes.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit adds a script that transforms qemu-cmd files to allow them
and the corresponding kernels to be run in contexts other than the one
that they were created for, including on systems other than the one that
they were built on. For example, this allows the build products from a
--buildonly run to be transformed to allow distributed rcutorture testing.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Using --kcsan when the compiler does not support KCSAN results in this:
:CONFIG_KCSAN=y: improperly set
:CONFIG_KCSAN_REPORT_ONCE_IN_MS=100000: improperly set
:CONFIG_KCSAN_VERBOSE=y: improperly set
:CONFIG_KCSAN_INTERRUPT_WATCHER=y: improperly set
Clean KCSAN run in /home/git/linux-rcu/tools/testing/selftests/rcutorture/res/2020.06.16-09.53.16
This is a bit obtuse, so this commit adds checks resulting in this:
:CONFIG_KCSAN=y: improperly set
:CONFIG_KCSAN_REPORT_ONCE_IN_MS=100000: improperly set
:CONFIG_KCSAN_VERBOSE=y: improperly set
:CONFIG_KCSAN_INTERRUPT_WATCHER=y: improperly set
Compiler or architecture does not support KCSAN!
Did you forget to switch your compiler with --kmake-arg CC=<cc-that-supports-kcsan>?
Suggested-by: Marco Elver <elver@google.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Acked-by: Marco Elver <elver@google.com>
Currently, kvm-recheck.sh complains that qemu failed for --buildonly
runs, which is sort of true given that qemu can hardly succeed if not
invoked in the first place. Nevertheless, this commit swaps the order
of checks in kvm-recheck.sh so that --buildonly runs will be summarized
more straightforwardly.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
We need to pass the arguments provided to --kmake-arg to all make
invocations. In particular, the make invocations generating the configs
need to see the final make arguments, e.g. if config variables depend on
particular variables that are passed to make.
For example, when using '--kcsan --kmake-arg CC=clang-11', we would lose
CONFIG_KCSAN=y due to 'make oldconfig' not seeing that we want to use a
compiler that supports KCSAN.
Signed-off-by: Marco Elver <elver@google.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit pulls the simple pattern-based error detection from the
console log into a new console-badness.sh file. This will enable future
commits to end a run on the first error.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
When bisecting RCU issues, it is often the case that the first error in
an unsuccessful run will happen quickly, but that a successful run must
go on for some time in order to obtain a sufficiently low false-negative
error rate. In many cases, a bisection requires multiple concurrent
runs, in which case the first failure in any run indicates failure,
pure and simple. In such cases, it would speed things up greatly if
the first failure terminated all runs.
This commit therefore adds scripting that checks for a file named "STOP"
in the top-level results directory, terminating the run when it appears.
Note that in-progress builds will continue until completion, but future
builds and all runs will be cut short.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
One reason to do a --buildonly run is to use the build products elsewhere,
for example, to do the actual test on some other system. Part of doing
the test is the actual qemu command, which is not currently produced
by --buildonly runs. This commit therefore causes --buildonly runs to
create this file.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Leaving off the kvm.sh script's --cpus argument results in the script
testing the scenarios sequentially, which can be quite slow. However,
having to specify the actual number of CPUs can be error-prone.
This commit therefore adds a --allcpus argument that causes kvm.sh to
use all available CPUs.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The identify_qemu_vcpus bash function can return numbers including
whitespace characters, which can be a bit annoying in some bash
dollar-sign substitutions. This commit therefore strips all spaces and
tabs from the value that identify_qemu_vcpus outputs.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The current console parsing assumes that console lines containing "!!!"
are statistics lines from which it can parse the number of rcutorture
too-short grace-period failures. This prints confusing output for
other problems, including memory exhaustion. This commit therefore
differentiates between these cases and prints an appropriate error string.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The torture-test recheck logic fails to set the configfile variable to
the current scenario, so this commit properly initializes this variable.
This change isn't critical given that all errors for a given scenario
follow that scenario's heading, but it is easier on the eyes to repeat it.
And this repetition also prevents confusion as to whether a given message
goes with the previous heading or the next one.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit adds a kvm-check-branches.sh script that takes a list
of commits and commit ranges and runs a short rcutorture test on all
scenarios on each specified commit. A summary is printed at the end, and
the script returns success if all rcutorture runs completed without error.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
On some (probably misconfigured) systems, the torture-test scripting
will cause qemu to complain about missing EFI firmware, often because
qemu is trying to traverse broken symbolic links to find that firmware.
Which is a bit silly given that the default torture-test guest OS has
but a single binary for its userspace, and thus is unlikely to do much
in the way of networking in any case.
This commit therefore avoids such problems by specifying "-net none"
to qemu unless the TORTURE_QEMU_INTERACTIVE environment variable is set
(for example, by having specified "--interactive" to kvm.sh), in which
case "-net nic -net user" is specified to qemu instead. Either choice
may be overridden by specifying the "-net" argument of your choice to
the kvm.sh "--qemu-args" parameter.
Link: https://lore.kernel.org/lkml/20190701141403.GA246562@google.com
Reported-by: Joel Fernandes <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
This commit renames the rcutorture config/refperf to config/refscale to
further avoid conflation with the Linux kernel's perf feature.
Reported-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, it is necessary to manually edit the console output to see
anything more than statistics, and sometimes the statistics can indicate
outliers that need more investigation. This commit therefore dumps out
the per-experiment measurements, sorted in ascending order, just before
dumping out the statistics.
Cc: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The experiment-number column is currently labeled "Threads", which is
misleading at best. This commit therefore relabels it as "Runs", and
adjusts the scripts accordingly.
Cc: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit updates the rcutorture scripting to include the new refperf
torture-test module.
Cc: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Make it a bit easier to apply KASAN to rcutorture runs with a new --kasan
argument, again leveraging the config_override_param() bash function.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit saves a few lines of code by also using the bash
config_override_param() to set the initial list of Kconfig options from
the CFcommon file. While in the area, it makes this function capable of
update-in-place on the file containing the cumulative Kconfig options,
thus avoiding annoying changes when adding another source of options.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit applies config_override_param() to allow scenario-specific
Kconfig options to override those in CFcommon. This in turn will allow
additional Kconfig options to be placed in CFcommon, for example, an
option common to all but a few scenario can be placed in CFcommon and
then overridden in those few scenarios. Plus this change saves one
whole line of code.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, attempting to override a --kcsan default with a --kconfig
option might or might not work. However, it would be good to allow the
user to adjust the --kcsan defaults, for example, to specify a different
time for CONFIG_KCSAN_REPORT_ONCE_IN_MS. This commit therefore uses the
new config_override_param() bash function to apply the --kcsan defaults
and then apply the --kconfig options, which allows this overriding
to occur.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit introduces a config_override_param() bash function that
folds in an additional set of Kconfig options. This is initially applied
to fold in the --kconfig kvm.sh parameter, but later commits will also
apply it to the Kconfig options added by the --kcsan kvm.sh parameter.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The #CHECK# directives that can be present in CFcommon and in the
rcutorture scenario Kconfig files are both copied to ConfigFragment
and grepped out of the two directive files and added to ConfigFragment.
This commit therefore removes the redundant "grep" commands and takes
advantage of the consequent opportunity to simplify redirection.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The KCSAN tool emits a great many warnings for current kernels, for
example, a one-hour run of the full set of rcutorture scenarios results
in no fewer than 3252 such warnings, many of which are duplicates
or are otherwise closely related. This commit therefore introduces
a kcsan-collapse.sh script that maps these warnings down to a set of
function pairs (22 of them given the 3252 individual warnings), placing
the resulting list in decreasing order of frequency of occurrence into
a kcsan.sum file. If any KCSAN warnings were produced, the pathname of
this file is emitted at the end of the summary of the rcutorture runs.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Although the existing --kconfig argument can be used to run KCSAN for
an rcutorture test, it is not as straightforward as one might like:
--kconfig "CONFIG_DEBUG_INFO=y CONFIG_KCSAN=y \
CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n \
CONFIG_KCSAN_REPORT_VALUE_CHANGE_ONLY=n \
CONFIG_KCSAN_REPORT_ONCE_IN_MS=100000 \
CONFIG_KCSAN_VERBOSE=y CONFIG_KCSAN_INTERRUPT_WATCHER=y"
This commit therefore adds a "--kcsan" argument that emulates the above
--kconfig command. Note that if you specify a Kconfig option using
-kconfig that conflicts with one that --kcsan adds, you get whatever
the script and the build system decide to give you.
Cc: Marco Elver <elver@google.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
System hangs or killed rcutorture guest OSes can result in truncated
"Reader Pipe:" lines, which can in turn result in false-positive
reader-batch near-miss warnings. This commit therefore adjusts the
reader-batch checks to account for possible line truncation.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, kvm-find-errors.sh gives a usage prompt when given a bad
directory, but then soldiers on, giving a series of confusing error
messages. This commit therefore prints an error message and exits when
given a bad directory, hopefully reducing confusion.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
When running the default list of tests, the run summary of a successful
(that is, failed to find any errors) run fits easily on a 24-line screen.
But a run with something like "--configs '5*CFLIST'" will be 80 lines long,
and it is all too easy to miss a failure message when scrolling back.
This commit therefore prints out the number of runs with failing builds
or runtime failures, but only if there are any such failures.
For example, a run with a single build error and a single runtime error
would print two lines like this:
1 runs with build errors.
1 runs with runtime errors.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The names of the per-test results directories are of the form
2019.11.29-20:42:19. This works, but the ":" characters make
tab-based shell name completion a bit onerous because the user must
remember to include a quote character somewhere before the first ":".
This commit therefore changes the ":" characters to periods, as in
2019.12.01-20.48.01", which allows tab-based completion to work more
naturally.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The bootparam_hotplug_cpu() bash function was checking for CPU-hotplug
kernel-boot parameters from --bootargs, but that check was specific to
rcutorture ("rcutorture\.onoff_"). This commit therefore makes this
check also work for locktorture ("torture\.onoff_").
Note that rcuperf does not do CPU-hotplug operations, so it is not
necessary to make a similar change for rcuperf.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
On some kernels, concurrent calls to the lscpu command result in severe
slowdowns. For example, on v4.16, a single lscpu invocation takes about
two milliseconds, four concurrent invocations more than two seconds,
and 16 concurrent invocations more than 20 seconds. Given that the only
goal is to learn the number of CPUs, invoking lscpu but once suffices.
This commit therefore invokes lscpu early in kvm.sh execution, setting
the initial value of the TORTURE_ALLOTED_CPUS environment variable.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
On a large system, it can be convenient to tell rcutorture to run
several instances of the default scenarios. Currently, this requires
explicitly listing them, for example, "--configs '2*SRCU-N 2*SRCU-P...'".
Although this works, it is rather inconvenient.
This commit therefore allows "CFLIST" to be specified to indicate the
default list of scenarios called out in the relevant CFLIST file, for
example, for RCU, tools/testing/selftests/rcutorture/configs/rcu/CFLIST.
In addition, multipliers may be used to run multiple instances of all
the scenarios. For example, on a 256-CPU system, "--configs '3*CFLIST'"
would run three instances of each scenario concurrently with one CPU
left over. Thus "--configs '3*CFLIST TINY01'" would exactly consume all
256 CPUs, which makes rcutorture's jitter feature more effective.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit adds the worst-case results from any call_rcu()
forward-progress tests to the rcutorture test-summary output.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The rcutorture scripting uses the mpstat command to determine how much
the system is being used, and adjusts make's -j argument accordingly.
However, mpstat isn't installed by default, so it would be good if the
scripting does something useful when mpstat isn't present.
This commit therefore makes the scripts assumes that if mpstat is not
present, they are free to use all the CPUs.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, jitter.sh assumes that the underlying hypervisor will be
configured with all CPUs hotpluggable, with the possible exception
of CPU 0. However, there are installations where the hypervisor
prohibits offlining, which breaks jitter.sh. This commit therefore
lists the CPUs that cannot be offlined up front, and checks for the
case where no CPU can be offlined in the loop.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The dracut scripting does not work on all platforms, and there are no
known failures from the init binary based on the statically linked C
program. This commit therefore removes the dracut scripting so that the
statically linked C program is always used to create the init "script".
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
In many environments, gawk provides systime(), but awk doesn't.
This commit therefore changes awk scripts using systime() to instead be
gawk scripts.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The kvm-test-1-run.sh script says 'test -z "last_ts"' which always
evaluates to true (AKA zero) regardless of the value of the last_ts shell
variable. This commit therefore inserts the needed dollar sign ("$").
Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
When trace_printk() is used, a message including "BUG" is printed to
the console, which fools the rcutorture scripting into believing that
the corresponding test scenario failed. This commit therefore filters
out this particular instance of "BUG", thus avoiding the false-positive
test-failure report.
Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
The current rcutorture scripts unconditionally do "make clean", which is
a good way of getting the needed testing done despite any imperfections in
Makefile dependency tracking. However, this can be a bit irritating when
repeatedly running a single scenario after small changes, for example,
when debugging a problem that affects only a single scenario. This commit
therefore adds a --trust-make argument that suppresses the "make clean".
Even when using ccache, this speeds up kernel builds by up to almost an
order of magnitude on my laptop.
Reported-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
Currently, rcutorture will use relatively few CPUs to build the kernel
on a busy system, which is often as it should be. However, if the user
has used the --cpus argument to dedicate a specified number of CPUs to
this torture test, it would be good if the kernel build also made use
of them. This commit therefore changes the cpus2use.sh script to use
--cpus when specified and to do the idleness calculations otherwise.
Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
For historical reasons, rcutorture places its build products in a
tools/testing/selftests/rcutorture/b1 directory using the O= kbuild
command-line argument. However, doing this requires that the source
directory be pristine: Not just "make clean" pristine, but instead "make
mrproper" (or, equivalently, "make distclean") pristine. Therefore,
rcutorture executes a "make mrproper" before each build. Unfortunately,
"make mrproper" has the side effect of removing pretty much everything,
including tags files and cscope databases, which can be inconvenient
to people whose workflow centers around a single source tree.
This commit therefore makes rcutorture do the build directly in the
source directory, removing the need for "make mrproper". This works
because all needed build products are moved to their proper place in the
"res" directory immediately after the build completes, so that multiple
rcutorture kernels can still run concurrently.
Reported-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
Currently qemu output appears on standard output, but is inaccessible
later on. This commit therefore captures this output and causes
kvm-recheck.sh to output this output if QEMU gave a non-zero non-137
exit code. (And exit code of 137 indicates that QEMU was killed, in
which case we want to know about the hang rather than the fact that
QEMU was killed.)
Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
In one of my rcutorture tests the TSC clocksource got marked unstable
due to a large difference in the TSC value. I'm not sure if the guest
run for a long time with disabled interrupts or if the host was very
busy and didn't schedule the guest for some time.
I took a look on the qemu/KVM options and decided to update the options:
- Use kvm{32|64} as CPU. We could probably use `host' (like ARM does)
for maximum available features but since we don't run any userland I'm
not sure if it makes any difference.
- Drop the "noapic" option. There is no history why the APIC was disabled,
I see no reason for it. Once old qemu versions fade away, we can add
"x2apic=on,tsc-deadline=on,hypervisor=on,tsc_adjust=on".
- Additional config options. It ensures that the kernel knowns that it
runs as a kvm guest and can use virt devices like the kvm-clock as
clocksource. The kvm-clock was the main motivation here.
- I didn't add a random HW device. It would make the random device ready
earlier (not it doesn't complete the initialisation at all) but I
doubt that there is any need for this.
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
[ paulmck: The world is not quite ready for CONFIG_PARAVIRT_SPINLOCKS=y
and x2apic, so they are omitted for the time being. ]
Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
Paul E. McKenney