Commit Graph

112 Commits

Author SHA1 Message Date
Victor Stinner e21c89f984
gh-109162: Refactor libregrtest.RunTests (#109177)
* Rename dash_R() runtest_refleak(). The function now gets
  huntrleaks and quiet arguments, instead of 'ns' argument.
* Add attributes to Regrtest and RunTests:

  * verbose
  * quiet
  * huntrleaks
  * test_dir

* Add HuntRefleak class.
2023-09-09 09:18:14 +00:00
Victor Stinner e9e2ca7a7b
gh-109162: Refactor libregrtest.runtest (#109172)
* Rename runtest() to run_single_test().
* Pass runtests to run_single_test().
* Add type annotation to Regrtest attributes. Add missing attributes
  to Namespace.
* Add attributes to Regrtest and RunTests:

  * fail_fast
  * ignore_tests
  * match_tests
  * output_on_failure
  * pgo
  * pgo_extended
  * timeout

* Get pgo from 'runtests', rather than from 'ns'.
* Remove WorkerJob.match_tests.
* setup_support() now gets pgo_extended from runtests.
* save_env(): change parameter order, pass test_name first.
* Add setup_test_dir() function.
* Pass runtests to setup_tests().
2023-09-09 01:37:48 +00:00
Victor Stinner a56c928756
gh-109162: Refactor libregrtest WorkerJob (#109171)
* Rename --worker-args command line option to --worker-json.
* Rename _parse_worker_args() to _parse_worker_json().
* WorkerJob:

  * Add runtests attribute
  * Remove test_name and rerun attribute

* Rename run_test_in_subprocess() to create_worker_process().
* Rename run_tests_worker() to worker_process().
* create_worker_process() uses json.dump(): write directly JSON to
  stdout.
* Convert MultiprocessResult to a frozen dataclass.
* Rename RunTests.match_tests to RunTests.match_tests_dict.
2023-09-09 01:03:39 +00:00
Victor Stinner 489ca0acf0
gh-109162: Refactor Regrtest.action_run_tests() (#109170)
Refator Regrtest class:

* Rename finalize() finalize_tests().
* Pass tracer to run_test() and finalize_tests(). Remove Regrtest.tracer.
* run_test() does less things: move code to its caller.
2023-09-09 00:30:28 +00:00
Victor Stinner ac8409b38b
gh-109162: Regrtest copies 'ns' attributes (#109168)
* Regrtest.__init__() now copies 'ns' namespace attributes to
  Regrtest attributes. Regrtest match_tests and ignore_tests
  attributes have type FilterTuple (tuple), instead of a list.
* Add RunTests.copy(). Regrtest._rerun_failed_tests() now uses
  RunTests.copy().
* Replace Regrtest.all_tests (list) with Regrtest.first_runtests
  (RunTests).
* Make random_seed maximum 10x larger (9 digits, instead of 8).
2023-09-08 23:48:54 +00:00
Victor Stinner 5b7303e265
gh-109162: Refactor Regrtest.main() (#109163)
* main() now calls _parse_args() and pass 'ns' to Regrtest
  constructor.  Remove kwargs argument from Regrtest.main().
* _parse_args() checks ns.huntrleaks.
* set_temp_dir() is now responsible to call expanduser().
* Regrtest.main() sets self.tests earlier.
* Add TestTuple and TestList types.
* Rename MatchTests to FilterTuple and rename MatchTestsDict
  to FilterTestDict.
* TestResult.get_rerun_match_tests() return type
  is now FilterTuple: return a tuple instead of a list.
  RunTests.tests type becomes TestTuple.
2023-09-08 22:41:26 +00:00
Victor Stinner 1170d5a292
gh-108834: regrtest --fail-rerun exits with code 5 (#108896)
When the --fail-rerun option is used and a test fails and then pass,
regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").
2023-09-05 01:09:42 +00:00
Victor Stinner 31c2945f14
gh-108834: regrtest reruns failed tests in subprocesses (#108839)
When using --rerun option, regrtest now re-runs failed tests
in verbose mode in fresh worker processes to have more
deterministic behavior. So it can write its final report even
if a test killed a worker progress.

Add --fail-rerun option to regrtest: exit with non-zero exit code
if a test failed pass passed when re-run in verbose mode (in a
fresh process). That's now more useful since tests can pass
when re-run in a fresh worker progress, whereas they failed
when run after other tests when tests are run sequentially.

Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.

Changes:

* Fix and enhance statistics in regrtest summary. Add "(filtered)"
  when --match and/or --ignore options are used.
* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
  using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
  filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
  the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
* Add test_success() and test_skip() tests to test_regrtest.
2023-09-03 21:37:15 +00:00
Victor Stinner d4e534cbb3
regrtest computes statistics (#108793)
test_netrc, test_pep646_syntax and test_xml_etree now return results
in the test_main() function.

Changes:

* Rewrite TestResult as a dataclass with a new State class.
* Add test.support.TestStats class and Regrtest.stats_dict attribute.
* libregrtest.runtest functions now modify a TestResult instance
  in-place.
* libregrtest summary lists the number of run tests and skipped
  tests, and denied resources.
* Add TestResult.has_meaningful_duration() method.
* Compute TestResult duration in the upper function.
* Use time.perf_counter() instead of time.monotonic().
* Regrtest: rename 'resource_denieds' attribute to 'resource_denied'.
* Rename CHILD_ERROR to MULTIPROCESSING_ERROR.
* Use match/case syntadx to have different code depending on the
  test state.

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-09-02 18:09:36 +02:00
Victor Stinner 174e9da083
gh-108388: regrtest splits test_asyncio package (#108393)
Currently, test_asyncio package is only splitted into sub-tests when
using command "./python -m test". With this change, it's also
splitted when passing it on the command line:
"./python -m test test_asyncio".

Remove the concept of "STDTESTS". Python is now mature enough to not
have to bother with that anymore. Removing STDTESTS simplify the
code.
2023-08-24 02:44:58 +00:00
Victor Stinner 3a1ac87f8f
gh-90791: test.pythoninfo logs ASAN_OPTIONS env var (#108289)
* Cleanup libregrtest code logging ASAN_OPTIONS.
* Fix a typo on "ASAN_OPTIONS" vs "MSAN_OPTIONS".
2023-08-23 01:39:50 +02:00
Gregory P. Smith 852348ab65
Display the sanitizer config in the regrtest header. (#105301)
Display the sanitizers present in libregrtest.

Having this in the CI output for tests with the relevant environment
variable displayed will help make it easier to do what we need to
create an equivalent local test run.
2023-06-05 23:36:36 -07:00
Nikita Sobolev 995386071f
bpo-46523: fix tests rerun when `setUp[Class|Module]` fails (#30895)
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2023-04-07 12:43:41 +02:00
Victor Stinner 3c89202247
gh-100086: Add build info to test.libregrtest (#100093)
The Python test runner (libregrtest) now logs Python build information like
"debug" vs "release" build, or LTO and PGO optimizations.
2022-12-08 01:38:47 +01:00
Victor Stinner c76db37c0d
gh-98903: Test suite fails with exit code 4 if no tests ran (#98904)
The Python test suite now fails wit exit code 4 if no tests ran. It
should help detecting typos in test names and test methods.

* Add "EXITCODE_" constants to Lib/test/libregrtest/main.py.
* Fix a typo: "NO TEST RUN" becomes "NO TESTS RAN"
2022-11-02 15:37:40 +01:00
Christian Heimes 0ff7b996f5
gh-94052: Don't re-run failed tests with --python option (#94054) 2022-06-21 14:42:32 +02:00
Christian Heimes 2702e408fd
gh-84461: Fix Emscripten umask and permission issues (GH-94002)
- Emscripten's default umask is too strict, see
  https://github.com/emscripten-core/emscripten/issues/17269
- getuid/getgid and geteuid/getegid are stubs that always return 0
  (root). Disable effective uid/gid syscalls and fix tests that use
  chmod() current user.
- Cannot drop X bit from directory.
2022-06-19 18:28:55 +02:00
Christian Heimes c2007573dd
gh-84461: Fix parallel testing on WebAssembly (GH-93768) 2022-06-13 19:51:04 +02:00
Inada Naoki 1317b70f89
gh-91156: Use `locale.getencoding()` instead of getpreferredencoding (GH-91732)
Co-authored-by: Victor Stinner <vstinner@python.org>
2022-04-22 10:39:24 +09:00
Christian Heimes 2b16a08bc7
bpo-40280: Detect missing threading on WASM platforms (GH-32352)
Co-authored-by: Brett Cannon <brett@python.org>
2022-04-07 09:22:47 +02:00
Jeremy Kloth 19058b9f62
bpo-44336: Prevent tests hanging on child process handles on Windows (GH-26578)
Replace the child process `typeperf.exe` with a daemon thread that reads the performance counters directly.  This prevents the issues that arise from inherited handles in grandchild processes (see issue37531 for discussion).

We only use the load tracker when running tests in multiprocess mode. This prevents inadvertent interactions with tests expecting a single threaded environment.  Displaying load is really only helpful for buildbots running in multiprocess mode anyway.
2022-03-22 00:06:55 +00:00
Victor Stinner 9d32714384
bpo-45745: Remove regrtest --findleaks options (GH-29514)
Remove the --findleaks command line option of regrtest: use the
--fail-env-changed option instead. Since Python 3.7, it was a
deprecated alias to the --fail-env-changed option.
2021-11-12 16:19:09 +01:00
Victor Stinner c4ea45d7d2
bpo-45118: Fix regrtest second summary for re-run tests (GH-28183)
Fix regrtest second summary when using -w/--verbose2 command line
option: lists re-run tests in the second test summary.
2021-09-07 18:21:00 +02:00
Łukasz Langa f1afef5e0d
bpo-44708: Only re-run test methods that match names of previously failing test methods (GH-27287)
* Move to a static argparse.Namespace subclass
* Roughly annotate runtest.py
* Refactor libregrtest to use lossless test result objects
* Only re-run test methods that match names of previously failing test methods
* Adopt tests to cover test method name matching

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
2021-07-22 20:25:58 +02:00
Victor Stinner 9feae41c4f
bpo-41718: libregrtest avoids importing datetime (GH-24985)
* libregrtest reimplements datetime.timedelta.__str__()
* support.testresult only imports datetime if USE_XML is true.
2021-03-23 01:40:31 +01:00
Hai Shi 3ddc634cd5
bpo-40275: Use new test.support helper submodules in tests (GH-21219) 2020-06-30 15:46:06 +02:00
Serhiy Storchaka 700cfa8c90
bpo-41069: Make TESTFN and the CWD for tests containing non-ascii characters. (GH-21035) 2020-06-25 17:56:31 +03:00
Hai Shi f7ba40b505
bpo-40275: Use new test.support helper submodules in tests (GH-20849) 2020-06-25 12:38:51 +02:00
Serhiy Storchaka 9355868458
bpo-41043: Escape literal part of the path for glob(). (GH-20994) 2020-06-20 11:10:31 +03:00
Victor Stinner 4cf65a630a
regrtest: log timeout at startup (GH-19514)
Reduce also worker timeout.
2020-04-14 18:29:44 +02:00
Pablo Galindo e0cd8aa70a
bpo-37957: Allow regrtest to receive a file with test (and subtests) to ignore (GH-16989)
When building Python in some uncommon platforms there are some known tests that will fail. Right now, the test suite has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter. The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept and he problem with -x is that is not easy to ignore just a subtests that fail and the whole test needs to be ignored.

For these reasons, add a new option to allow to ignore a list of test and subtests for these situations.
2019-11-19 23:46:49 +00:00
Victor Stinner 098e25672f
bpo-36670: Enhance regrtest (GH-16556)
* Add log() method: add timestamp and load average prefixes
  to main messages.
* WindowsLoadTracker:

  * LOAD_FACTOR_1 is now computed using SAMPLING_INTERVAL
  * Initialize the load to the arithmetic mean of the first 5 values
    of the Processor Queue Length value (so over 5 seconds), rather
    than 0.0.
  * Handle BrokenPipeError and when typeperf exit.

* format_duration(1.5) now returns '1.5 sec', rather than
  '1 sec 500 ms'
2019-10-03 16:15:16 +02:00
Victor Stinner 982bfa4da0
bpo-36670: Multiple regrtest bugfixes (GH-16511)
* Windows: Fix counter name in WindowsLoadTracker. Counter names are
  localized: use the registry to get the counter name. Original
  change written by Lorenz Mende.
* Regrtest.main() now ensures that the Windows load tracker is also
  killed if an exception is raised
* TestWorkerProcess now ensures that worker processes are no longer
  running before exiting: kill also worker processes when an
  exception is raised.
* Enhance regrtest messages and warnings: include test name,
  duration, add a worker identifier, etc.
* Rename MultiprocessRunner to TestWorkerProcess
* Use print_warning() to display warnings.

Co-Authored-By: Lorenz Mende <Lorenz.mende@gmail.com>
2019-10-01 12:29:36 +02:00
Victor Stinner 0a963fbc9c
bpo-38203: faulthandler.dump_traceback_later() is always available (GH-16249)
dump_traceback_later() and cancel_dump_traceback_later() functions of
the faulthandler module are always available since Python 3.7.
2019-09-18 14:15:10 +02:00
Victor Stinner da57599af5
bpo-38203: regrtest: put a 2 min timeout on Python exit (GH-16250) 2019-09-18 08:29:25 +02:00
Victor Stinner b0c8369c60
bpo-37531: Fix regrtest timeout for subprocesses (GH-15072)
Co-Authored-By: Joannah Nanjekye <joannah.nanjekye@ibm.com>
2019-08-14 14:18:51 +02:00
Neil Schemenauer 52a48e62c6
bpo-37707: Exclude expensive unit tests from PGO task (GH-15009)
Mark some individual tests to skip when --pgo is used.  The tests
marked increase the PGO task time significantly and likely don't
help improve optimization of the final executable.
2019-07-30 11:08:18 -07:00
Gregory P. Smith f0807ab24c
Only setup PGO tests when --pgo is enabled. (GH-14927) 2019-07-23 21:33:48 -07:00
Neil Schemenauer 4e16a4a311
bpo-36044: Reduce number of unit tests run for PGO build (GH-14702)
Reduce the number of unit tests run for the PGO generation task.  This
speeds up the task by a factor of about 15x.  Running the full unit test
suite is slow.  This change may result in a slightly less optimized build
since not as many code branches will be executed.  If you are willing to
wait for the much slower build, the old behavior can be restored using
'./configure [..] PROFILE_TASK="-m test --pgo-extended"'.  We make no
guarantees as to which PGO task set produces a faster build.  Users who
care should run their own relevant benchmarks as results can depend on
the environment, workload, and compiler tool chain.
2019-07-22 12:54:25 -07:00
Victor Stinner 8f4ef3b019
Remove unused imports in tests (GH-14518) 2019-07-01 18:28:25 +02:00
Victor Stinner 9bbf4d7083
bpo-37359: Fix regrtest --cleanup (GH-14336) 2019-06-24 13:19:48 +02:00
Victor Stinner 47fbc4e45b
bpo-37359: Add --cleanup option to python3 -m test (GH-14332)
* regrtest: Add --cleanup option to remove "test_python_*" directories
  of previous failed test jobs.
* Add "make cleantest" to run "python3 -m test --cleanup".
2019-06-24 12:03:00 +02:00
Victor Stinner 3c93153f7d
bpo-36915: regrtest always remove tempdir of worker processes (GH-13312)
When using multiprocessing (-jN option), worker processes now create
their temporary directory inside the temporary directory of the
main process. So the main process is able to remove temporary
directories of worker processes even if they crash or when they are
killed by regrtest on KeyboardInterrupt (CTRL+c).

Rework also how multiprocessing arguments are parsed in main.py.
2019-05-14 15:49:16 +02:00
Victor Stinner b0917df329
bpo-36719: regrtest -jN no longer stops on crash (GH-13231)
"python3 -m test -jN ..." now continues the execution of next tests
when a worker process crash (CHILD_ERROR state). Previously, the test
suite stopped immediately. Use --failfast to stop at the first error.

Moreover, --forever now also implies --failfast.
2019-05-13 19:17:54 +02:00
Victor Stinner 00db7c73af
bpo-36719: regrtest closes explicitly WindowsLoadTracker (GH-12965)
Regrtest.finalize() now closes explicitly the WindowsLoadTracker
instance.
2019-04-26 11:12:26 +02:00
Victor Stinner 837acc1957
bpo-36719: Fix regrtest re-run (GH-12964)
Properly handle a test which fail but then pass.

Add test_rerun_success() unit test.
2019-04-26 09:56:37 +02:00
Victor Stinner 75120d2205
bpo-36719: regrtest always detect uncollectable objects (GH-12951)
regrtest now always detects uncollectable objects. Previously, the
check was only enabled by --findleaks. The check now also works with
-jN/--multiprocess N.

--findleaks becomes a deprecated alias to --fail-env-changed.
2019-04-26 09:28:53 +02:00
Victor Stinner 3cde440f20
bpo-36725: Refactor regrtest multiprocessing code (GH-12961)
Rewrite run_tests_multiprocess() function as a new MultiprocessRunner
class with multiple methods to better report errors and stop
immediately when needed.

Changes:

* Worker processes are now killed immediately if tests are
  interrupted or if a test does crash (CHILD_ERROR): worker
  processes are killed.
* Rewrite how errors in a worker thread are reported to
  the main thread. No longer ignore BaseException or parsing errors
  silently.
* Remove 'finished' variable: use worker.is_alive() instead
* Always compute omitted tests. Add Regrtest.get_executed() method.
2019-04-26 08:40:25 +02:00
Victor Stinner 4d29983185
bpo-36725: regrtest: add TestResult type (GH-12960)
* Add TestResult and MultiprocessResult types to ensure that results
  always have the same fields.
* runtest() now handles KeyboardInterrupt
* accumulate_result() and format_test_result() now takes a TestResult
* cleanup_test_droppings() is now called by runtest() and mark the
  test as ENV_CHANGED if the test leaks support.TESTFN file.
* runtest() now includes code "around" the test in the test timing
* Add print_warning() in test.libregrtest.utils to standardize how
  libregrtest logs warnings to ease parsing the test output.
* support.unload() is now called with abstest rather than test_name
* Rename 'test' variable/parameter to 'test_name'
* dash_R(): remove unused the_module parameter
* Remove unused imports
2019-04-26 04:08:53 +02:00
Paul Monson 264a0b40b0 bpo-36638: Fix WindowsLoadTracker exception on some Windows versions (GH-12849) 2019-04-17 18:06:06 -07:00