Add a way to run a patchcheck test on the commits that are in one branch
but not in another. This uses git cherry to find a list of commits to
test each one with.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
With the more robust config_bisect, the documentation is out of
date and needs to be updated.
The new rewrite allows for finding missing configs and such, and
is much more robust to use.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
For those cases that it takes several tries to hit a bug, it would be
useful for ktest.pl to try a test multiple times before it considers
the test as a pass. To accomplish this, BISECT_TRIES ktest config
option has been added. It is default to one, as most of the time a
bisect only needs to try a test once. But the user can now up this
to make ktest run a given test multiple times. The first failure
that is detected will set a bisect bad. It only repeats on success.
Note, as with all race bugs, there's no guarantee that if it succeeds,
it is really a good bisect. But it helps in case the bug is somewhat
reliable.
You can set BISECT_TRIES to zero, and all tests will be considered
good, unless you also set BISECT_MANUAL.
Suggested-by: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
The sample.conf file needs to document all available options.
With the new CLOSE_CONSOE_SIGNAL option, it too needs to be
document.
Cc: Satoru Takeuchi <satoru.takeuchi@gmail.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Although the patchcheck test checks for warnings in the files that were
changed, this check does not catch warnings that were caused by header
file changes and the warnings appear in C files not touched by the
commit.
Add a new option called WARNINGS_FILE. If this option is set, then the
file it points to is read before bulid, and the file should contain a
list of known warnings. If a warning appears in the build, this file is
checked, and if the warning does not exist in this file, then it fails
the build showing the new warning.
If the WARNINGS_FILE points to a file that does not exist, this will
cause any warning in the build to fail.
A new test is also added called "make_warnings_file". This test will
create do a build and record any warnings it finds into the
WARNINGS_FILE. This test is something that can be run before other tests
to build a warnings file of "known warnings", ie, warnings that were
there before your changes.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
I installed Fedora 17 which no longer supports grub v1. I worked
with grub2 for a while, but there's so many issues with it and automated
rebooting, that I decided to switch to syslinux. Instead of using
the REBOOT_SCRIPT and add customized changes to get syslinux booted,
I thought it better to make ktest aware of syslinux and add options
to simplify the use of syslinux on a target test box.
Acked-by: H. Peter Anvin <hpa@zytor.com>
Cc: John 'Warthog9' Hawley <warthog9@kernel.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
As only grub or 'script' is supported for rebooting to a new kernel,
and Fedora 17 has dropped support for grub, I decided to add grub2
support as well (I also plan on adding syslinux/extlinux support too).
The options GRUB_FILE and GRUB_REBOOT were added to allow the user
to specify where to find the grub.cfg and what tool to use to reboot
into the next kernel respectively.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
If the console is constantly outputting content, this can cause ktest
to get stuck waiting on the monitor to settle down.
The option MAX_MONITOR_WAIT is the maximum time (in seconds) for ktest
to wait for the console to flush.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
The config-bisect can take a bad config and bisect it down to find out
what config actually breaks the config. But as all tests will apply a
minconfig (defined by a user) to apply before booting, it is possible
that the minconfig could actually make the bad config work (minconfigs
can disable configs). The end result is that the config bisect test will
not find a config that breaks. This can be rather frustrating to the
user.
The CONFIG_BISECT_CHECK option, when set to 1, will make sure that the
bad config (with the minconfig applied) still fails before trying to
bisect.
And yes, I did get burned by this.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Add the PRE_INSTALL option that will allow a user to specify a shell
command to be executed before the install operation executes.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
In order to let the user add commands before and after ktest runs, the
PRE_KTEST and POST_KTEST options are defined. They hold shell commands
that will execute befor ktest runs its first test, as well as when it
completed its last test.
The PRE_TEST and POST_TEST will be run befor and after (respectively)
for a given test. They can either be global (done for all tests) or
defined by a single test.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
If the file that OUTPUT_MIN_CONFIG exists then ktest.pl will prompt the
user and ask them if the OUTPUT_MIN_CONFIG should be used as the
starting point for make_min_config instead of MIN_CONFIG.
This is usually the case, and to allow the user to do so, which is
helpful if the user is creating different min configs based on tests,
and they know one is a superset of another test, they can set
USE_OUTPUT_MIN_CONFIG to one, which will prevent kest.pl from prompting
to use the OUTPUT_MIN_CONFIG and it will just use it.
If USE_OUTPUT_MIN_CONIFG is set to zero, then ktest.pl will continue to
use MIN_CONFIG instead.
The default is that USE_OUTPUT_MIN_CONFIG is undefined.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Add a MIN_CONFIG_TYPE that can be set to 'test' or 'boot'. The default
is 'boot' which is what make_min_config has done previously: makes a
config file that is the minimum needed to boot the target.
But when MIN_CONFIG_TYPE is set to 'test', not only must the target
boot, but it must also successfully run the TEST. This allows the
creation of a config file that is the minimum to boot and also
perform ssh to the target, or anything else a developer wants.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Currently the option used to scp both the modules to the target as well
as the kernel image are the same (SCP_TO_TARGET). But some embedded
boards may require them to be different. The modules may need to be put
directly on the board, but the kernel image may need to go to a
tftpserver.
Add the option SCP_TO_TARGET_INSTALL that will allow the user to change
the config so that they may have the modules and image got to different
machines.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-ktest:
ktest: Add INGORE_ERRORS to ignore warnings in boot up
ktest: Still do reboot even for REBOOT_TYPE = script
ktest: Fix compare script to test if options are not documented
ktest: Detect typos in option names
ktest: Have all values be set by defaults
ktest: Change initialization of defaults hash to perl format
ktest: Add options SWITCH_TO_GOOD and SWITCH_TO_TEST
ktest: Allow overriding bisect test results
ktest: Evaluate options before processing them
ktest: Evaluate $KERNEL_VERSION in both install and post install
ktest: Only ask options needed for install
ktest: When creating a new config, ask for BUILD_OPTIONS
ktest: Do not ask for some options if the only test is build
ktest: Ask for type of test when creating a new config
ktest: Allow bisect test to restart where it left off
ktest: When creating new config, allow the use of ${THIS_DIR}
ktest: Add default for ssh-user, build-target and target-image
ktest: Allow success logs to be stored
ktest: Save test output
When testing a kernel that has warnings, ktest.pl will fail the test
when it sees the warning. If you need to test the the kernel and want
to ignore the errors that are produced, the option IGNORE_ERRORS has
been added. When IGNORE_ERRORS is set to something other than 0, it will
ignore call traces due to WARN_ON().
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
The REBOOT_TYPE may be either grub or script, if it is script
it is expected that a REBOOT_SCRIPT is defined.
With the SWITCH_TO_TEST which is the complement of SWITCH_TO_GOOD,
which does basically the same thing as REBOOT_SCRIPT and but for
both grub and script, the REBOOT_SCRIPT does not need to be mandatory
anymore.
Do not require the REBOOT_SCRIPT and always run the reboot code
for both grub and script.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
It becomes quite annoying when you go to run a test and then
realize that you typed an option name wrong, and the test starts
doing the default action and not what you expected it to do.
It is even more annoying when you wake up the next day after
running the test over night when you discover this.
By testing if all options specified in a config file are
used by either ktest or were used in one of the option's values
we can see if there are any dangling options that were not used.
In such a case, show the user the options that were not used
and ask them if they want to continue or not.
The option IGNORE_UNUSED was also added to allow the user to
override this feature.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
For machines that do no use grub, it may be needed to update an
external image (tftp) before doing a reboot into either the
test image or the known good image.
The option SWITCH_TO_GOOD is added, where if it is defined, the
command that is specified as its value will be executed before
doing a reboot into a known good image.
The option SWITCH_TO_TEST is added, where if it is defined, the
command that is specified as its value will be executed before
doing a reboot into the test image.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
When running the ktest git bisect test, if the BISECT_TYPE is "test",
the bisect is determined to be good or bad based off of the error
code of the test that is run. Currently, if the test returns 0,
it is considered a pass (good), a non-zero is considered a fail (bad).
But it has been requested to add more options, and also change
the meanings of the error codes of the test. For example, one may
want the test to detect if the commit is not good or bad,
(maybe the bisect came to a point where the code in question
does not exist). The test could report an error code that should tell
ktest to skip the commit.
Also, a test could detect that something is horribly wrong and the
biscet should just be aborted.
The new options:
BISECT_RET_GOOD
BISECT_RET_BAD
BISECT_RET_SKIP
BISECT_RET_ABORT
BISECT_RET_DEFAULT
have been added. The first 4 take an integer value that will
represent if the test should be considered a pass, fail, neither
good nor bad, or abort respectively.
The BISECT_RET_DEFAULT will bo whatever is not defined by the
above codes. If only BISECT_RET_DEFAULT is defined, then all tests
will do the default.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Add a STORE_SUCCESSES option, to allow success logs to be stored, for
example to double-check or otherwise post-process the test logs.
Link: http://lkml.kernel.org/r/1321616131-21352-3-git-send-email-rabin@rab.in
Signed-off-by: Rabin Vincent <rabin@rab.in>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
The IF statements for DEFAULTS and TEST_START sections now handle
complex statements (&&,||)
Example:
TEST_START IF (DEFINED ALL_TESTS || ${MYTEST} == boottest) && ${MACHINE} == gandalf
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Have IF statements process if a config variable or option has been
defined or not. Can use NOT DEFINED in the case for telling if
a variable or option has not been defined.
DEFAULTS IF NOT DEFINED SSH_USER
SSH_USER = root
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
The OVERRIDE keyword will allow options defined in the given
DEFAULTS section to override options defined in previous DEFAULT
sections.
Normally, options will error if they were previous defined.
The OVERRIDE keyword allows options that have been previously
defined to be changed in the given section.
Note, the same option can not be defined in the same DEFAULT section
even if that section is marked as OVERRIDE.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Have the reading of the config file allow reading of other config
files using the INCLUDE keyword. This allows multiple config files
to share config options.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Allow ==, !=, <=, >=, <, and > to be used in IF statements
to compare if a section should be processed or not.
For example:
BITS := 32
DEFAULTS IF ${BITS} == 32
MIN_CONFIG = ${CONFIG_DIR}/config-32
ELSE
MIN_CONFIG = ${CONFIG_DIR}/config-64
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Add IF keyword to sections within the config. Also added an ELSE
keyword that allows different config options to be set for a given
section.
For example:
TYPE := 1
STATUS := 0
DEFAULTS IF ${TYPE}
[...]
ELSE IF ${STATUS}
[...]
ELSE
[...]
The above will process the first section as $TYPE is true. If it
was false, it would process the last section as $STATUS is false.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
When ktest.pl reboots, it will usuall wait SLEEP_TIME seconds of idle
console before starting the next test. By setting the
REBOOT_SUCCESS_LINE, ktest will not wait SLEEP_TIME when it detects the
line while rebooting to a new kernel.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
There's cases where running the same kernel over and over again
is useful, and being able to not install the same kernel can
save time between tests.
Add a NO_INSTALL option that tells ktest.pl to not install the
new kernel.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
In testing one of my boxes, I found that I only wanted to build and
install the kernel. I wanted to manually reboot the box and test it.
Adding a TEST_TYPE option "install" allows this to happen.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
If the defined OUTPUT_MIN_CONFIG in the make_min_config test exists,
then give a prompt to ask the user if they want to use that config
instead, as it is very often the case, especially when the test has been
interrupted. The OUTPUT_MIN_CONFIG is usually the config that one wants
to use to continue the test where they left off.
But if START_MIN_CONFIG is defined (thus the MIN_CONFIG is not the
default), then do not prompt, as it will be annoying if the user has
this as one of many tests, and the test pauses waiting for input, while
the user is sleeping.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
To save time, the test does not just grab any option and test
it. The Kconfig files are examined to determine the dependencies
of the configs. If a config is chosen that depends on another
config, that config will be checked first. By checking the
parents first, we can eliminate whole groups of configs that
may have been enabled.
For example, if a USB device config is chosen and depends on
CONFIG_USB, the CONFIG_USB will be tested before the device.
If CONFIG_USB is found not to be needed, it, as well as all
configs that depend on it, will be disabled and removed from
the current min_config.
Note, the code from streamline_config (make localmodconfig)
was copied and used to find the dependencies in the Kconfig file.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
After doing a make localyesconfig, your kernel configuration may
not be the most useful minimum configuration. Having a true minimum
config that you can use against other configs is very useful if
someone else has a config that breaks on your code. By only forcing
those configurations that are truly required to boot your machine
will give you less of a chance that one of your set configurations
will make the bug go away. This will give you a better chance to
be able to reproduce the reported bug matching the broken config.
Note, this does take some time, and may require you to run the
test over night, or perhaps over the weekend. But it also allows
you to interrupt it, and gives you the current minimum config
that was found till that time.
Note, this test automatically assumes a BUILD_TYPE of oldconfig
and its test type acts like boot.
TODO: add a test version that makes the config do more than just
boot, like having network access.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Doing a patchcheck test, there may be warnings that gcc produces which
may be OK, and the test should not fail on that commit. By adding a
IGNORE_WARNINGS option to list a space delimited SHA1s that are ignored
lets the user avoid having the test fail on certain commits.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
As multiple tests may be executed by the same server, have the test
machine name add uniqueness to the value of the temp directory.
Otherwise the temp directories may overwrite each other's tests.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
There are some cases that a patch may be needed to apply to the kernel
in patchcheck or bisect tests. Adding a PRE_BUILD option to apply the
patch and POST_BUILD to remove it, allows for this to be done easily.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Searching through several tests, it gets confusing which test result
is for which test. By adding the TEST_NAME option, the user can tell
which test result belongs to which test.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Currently the config_bisect compares the min config with the
CONFIG_BISECT config. There may be another config that we know
is good that we want to ignore configs on. By passing in this
config it will ignore the options that are set in the good config.
Note: This only ignores the config, it does not (yet) handle
options that are different between the two configs. If the good
config has "SLAB" set and the bad config has "SLUB" it will not
find the bug if the bug had to do with changing these two options.
This is something that I intend to implement in the future.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
When a triple fault happens in a test, no call trace nor panic
is displayed. Instead, the system reboots to the good kernel.
Since the good kernel may display a boot prompt that matches the
success string, ktest may think that the test succeeded, when it
did not.
Detecting triple faults is tricky because it is hard to generalize
what a reboot looks like. The best that we can come up with for now
is to examine the Linux banner. If we detect that the Linux banner
matches the test we want to test, then look to see if we hit another
Linux banner with a different kernel is booted. This can be assumed
to be a triple fault.
We can't just check for two Linux banners because things like
early printk may cause the Linux banner to be displayed twice. Checking
for different kernel versions should be the safe bet.
If this for some reason detects a false triple boot. A new ktest
config option is also created:
DETECT_TRIPLE_FAULT
This can be set to 0 to disable this checking.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
There are cases where one ktest option may be used within another
ktest option. Allow them to be reused just like config variables
but there are evaluated at time of test not config processing time.
Thus having something like:
MAKE_CMD = make ARCH=${ARCH}
TEST_START
ARCH = powerpc
TEST_START
ARCH = arm
Will have the arch defined for each test iteration.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
I found that I constantly reuse information for each test case.
It would be nice to just define a variable to reuse.
For example I may have:
TEST_START
[...]
TEST = ssh root@mybox /path/to/my/script
TEST_START
[...]
TEST = ssh root@mybox /path/to/my/script
[etc]
The issue is, I may wont to change that script or one of the other
fields. Then I need to update each line individually.
With the addition of config variables (variables only used during parsing
the config) we can simplify the config files. These variables can
also be defined multiple times and each time the new value will
overwrite the old value.
The convention to use a config variable over a ktest option is to use :=
instead of =.
Now we could do:
USER := root
TARGET := mybox
TEST_SCRIPT := /path/to/my/script
TEST_CASE := ${USER}@${TARGET} ${TEST_SCRIPT}
TEST_START
[...]
TEST = ${TEST_CASE}
TEST_START
[...]
TEST = ${TEST_CASE}
[etc]
Now we just need to update the variables at the top.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
The patches being checked may not leave the kernel in a state
that the next run will allow the new kernel to be copied to the
machine. Reboot to a known good kernel before continuing to the
next kernel to test.
Added option PATCHCHECK_SLEEP_TIME for the max time to sleep between
patchcheck reboots.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Currently, if a test causes constant output but never reaches a
boot prompt, or crashes, the test will never stop. Add STOP_TEST_AFTER
to create a variable that will stop (and fail) the test after it has run
for this amount of time. The default is 10 minutes. Setting this
variable to -1 will disable it.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Add the config option BISECT_FILES that allows the user to
specify what path in the kernel to run the git bisect on.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
If a during a git bisect, ktest fails on something other than
what it is testing (if BISECT_TYPE is test but it fails on build),
if BISECT_SKIP is set, then it will do a "git bisect skip" instead
of just failing the bisect and letting the user find a good commit
to test.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
For both git bisect and config bisect, if BISECT_MANUAL is set to 1,
then bisect will stop between iterations and ask the user for the
result. The actual result is ignored. This makes it possible to
use ktest.pl for bisecting configs and git and let the user examine
the results themselves and enter their own results.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Added the options STOP_AFTER_SUCCESS and STOP_AFTER_FAILURE to
allow the user to give a time (in seconds) to stop the monitor
after a stack trace or login has been detected. Sometimes the
kernel constantly prints out to the console and this may cause
the test to run indefinitely.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Steven Rostedt (Red Hat)