Commit Graph

417 Commits

Author SHA1 Message Date
Miss Islington (bot) 6fa6a61805
[3.11] gh-101225: Increase the socket backlog when creating a multiprocessing.connection.Listener (GH-113567) (#114019)
gh-101225: Increase the socket backlog when creating a multiprocessing.connection.Listener (GH-113567)

Increase the backlog for multiprocessing.connection.Listener` objects created
 by `multiprocessing.manager` and `multiprocessing.resource_sharer` to
 significantly reduce the risk of getting a connection refused error when creating
 a `multiprocessing.connection.Connection` to them.
(cherry picked from commit c7d59bd8cf)

Co-authored-by: Ronald Oussoren <ronaldoussoren@mac.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
2024-01-14 16:29:30 +01:00
Miss Islington (bot) d04ca4ccd7
[3.11] gh-113421: Fix multiprocessing logger for "%(filename)s" (GH-113423) (GH-113451)
(cherry picked from commit ce77ee5035)

Co-authored-by: Xu Song <xusong.vip@gmail.com>
2023-12-24 10:22:34 +00:00
Miss Islington (bot) 220fae7706
[3.11] gh-113009: Fix multiprocessing Process.terminate() on Windows (GH-113128) (#113178)
gh-113009: Fix multiprocessing Process.terminate() on Windows (GH-113128)

On Windows, Process.terminate() no longer sets the returncode
attribute to always call WaitForSingleObject() in Process.wait().
Previously, sometimes the process was still running after
TerminateProcess() even if GetExitCodeProcess() is not STILL_ACTIVE.
(cherry picked from commit 4026ad5b2c)

Co-authored-by: Victor Stinner <vstinner@python.org>
2023-12-15 15:20:18 +00:00
Miss Islington (bot) 2d8012f852
[3.11] gh-79429: Ignore FileNotFoundError when remove a temporary directory in the multiprocessing finalizer (GH-112865) (GH-112897)
(cherry picked from commit 7e82c626c4)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2023-12-09 13:31:42 +00:00
Miss Islington (bot) efe83ad276
[3.11] gh-110036: multiprocessing Popen.terminate() catches PermissionError (GH-110037) (#110065)
gh-110036: multiprocessing Popen.terminate() catches PermissionError (GH-110037)

On Windows, multiprocessing Popen.terminate() now catchs
PermissionError and get the process exit code. If the process is
still running, raise again the PermissionError. Otherwise, the
process terminated as expected: store its exit code.
(cherry picked from commit bd4518c60c)

Co-authored-by: Victor Stinner <vstinner@python.org>
2023-09-29 01:04:58 +00:00
Miss Islington (bot) f764abb375
[3.11] gh-109593: Fix reentrancy issue in multiprocessing resource_tracker (GH-109629) (#109897)
gh-109593: Fix reentrancy issue in multiprocessing resource_tracker (GH-109629)

---------

(cherry picked from commit 0eb98837b6)

Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2023-09-26 12:21:52 +00:00
Miss Islington (bot) 8fef834f05
[3.11] gh-109370: Fix unexpected traceback output in test_concurrent_futures (GH-109780) (GH-109882)
Follow-up of gh-107219.

* Only close the connection writer on Windows.
* Also use existing constant _winapi.ERROR_OPERATION_ABORTED instead of
  WSA_OPERATION_ABORTED.
(cherry picked from commit 0b4e090422)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2023-09-26 11:21:09 +03:00
Miss Islington (bot) 2f8c80f531
[3.11] gh-107219: Fix concurrent.futures terminate_broken() (GH-109244) (#109255)
gh-107219: Fix concurrent.futures terminate_broken() (GH-109244)

Fix a race condition in concurrent.futures. When a process in the
process pool was terminated abruptly (while the future was running or
pending), close the connection write end. If the call queue is
blocked on sending bytes to a worker process, closing the connection
write end interrupts the send, so the queue can be closed.

Changes:

* _ExecutorManagerThread.terminate_broken() now closes
  call_queue._writer.
* multiprocessing PipeConnection.close() now interrupts
  WaitForMultipleObjects() in _send_bytes() by cancelling the
  overlapped operation.
(cherry picked from commit a9b1f84790)

Co-authored-by: Victor Stinner <vstinner@python.org>
2023-09-11 08:52:17 +00:00
Miss Islington (bot) 385b1952f0
[3.11] gh-108520: Fix bad fork detection in nested multiprocessing use case (GH-108568) (#108692)
gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization.

---------

(cherry picked from commit add8d45cbe)

Co-authored-by: albanD <desmaison.alban@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
2023-08-30 22:36:26 +02:00
Miss Islington (bot) 34ef75d3ef
[3.11] gh-77377: Ensure multiprocessing SemLock is valid for spawn-based Process before serializing it (GH-107275) (#108378)
gh-77377: Ensure multiprocessing SemLock is valid for spawn-based Process before serializing it (GH-107275)

Ensure multiprocessing SemLock is valid for spawn Process before serializing it.

Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early.

---------

(cherry picked from commit 1700d34d31)

Co-authored-by: albanD <desmaison.alban@gmail.com>
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
2023-08-23 21:11:20 +00:00
Miss Islington (bot) 3f7dfb6d39
[3.11] More actionable error message when spawn is incorrectly used. (GH-102203) (#107991)
More actionable error message when spawn is incorrectly used. (GH-102203)
(cherry picked from commit a794ebeb02)

Co-authored-by: Yuxin Wu <ppwwyyxxc@gmail.com>
Co-authored-by: Yuxin Wu <ppwwyyxx@users.noreply.github.com>
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
2023-08-16 01:43:28 +00:00
Miss Islington (bot) db4400b5b2
[3.11] gh-107963: Fix set_forkserver_preload to check the type of given list (GH-107965) (gh-107976)
gh-107963: Fix set_forkserver_preload to check the type of given list (GH-107965)
(cherry picked from commit 6515ec3d3d)


gh-107963: Fix set_forkserver_preload to check the type of given list

Co-authored-by: Dong-hee Na <donghee.na@python.org>
2023-08-15 23:53:25 +09:00
Miss Islington (bot) 1931c2a438
[3.11] gh-90876: Restore the ability to import multiprocessing when `sys.executable` is `None` (GH-106464) (#106495)
gh-90876: Restore the ability to import multiprocessing when `sys.executable` is `None` (GH-106464)

Prevent `multiprocessing.spawn` from failing to *import* in environments
where `sys.executable` is `None`.  This regressed in 3.11 with the addition
of support for path-like objects in multiprocessing.

Adds a test decorator to have tests only run when part of test_multiprocessing_spawn to `_test_multiprocessing.py` so we can start to avoid re-running the same not-global-state specific test in all 3 modes when there is no need.
(cherry picked from commit c60df361ce)

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2023-07-06 16:25:20 -07:00
Miss Islington (bot) 6f94bbf77e
[3.11] gh-104536: Improve `multiprocessing.process._cleanup` logic (GH-104537) (#104737)
Fix a race condition in the internal `multiprocessing.process` cleanup
logic that could manifest as an unintended `AttributeError` when calling
`BaseProcess.close()`.

(cherry picked from commit ef5d00a592)

Co-authored-by: Luccccifer <lukezhang764@gmail.com>
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
2023-05-22 12:49:33 +02:00
Luke Garland 374b0a2ace
[3.11] bpo-40882: Fix a memory leak in SharedMemory on Windows (GH-20684) (#99973)
bpo-40882: Fix a memory leak in SharedMemory on Windows (GH-20684)

In multiprocessing.shared_memory.SharedMemory(), the temporary view
returned by MapViewOfFile() should be unmapped when it is no longer
needed.

(cherry picked from commit 85c128e34d)

Co-authored-by: Zackery Spytz <zspytz@gmail.com>
2022-12-05 13:38:25 +01:00
Miss Islington (bot) 4686d77a04
[3.11] gh-97514: Don't use Linux abstract sockets for multiprocessing (GH-98501) (GH-98502)
Linux abstract sockets are insecure as they lack any form of filesystem
permissions so their use allows anyone on the system to inject code into
the process.

This removes the default preference for abstract sockets in
multiprocessing introduced in Python 3.9+ via
https://github.com/python/cpython/pull/18866 while fixing
https://github.com/python/cpython/issues/84031.

Explicit use of an abstract socket by a user now generates a
RuntimeWarning.  If we choose to keep this warning, it should be
backported to the 3.7 and 3.8 branches.
(cherry picked from commit 49f61068f4)


Co-authored-by: Gregory P. Smith <greg@krypto.org>

Automerge-Triggered-By: GH:gpshead
2022-10-20 16:55:37 -07:00
Miss Islington (bot) ace6611de6
gh-98360: multiprocessing now spawns children on Windows with correct argv[0] in virtual environments (GH-98462)
(cherry picked from commit e48f9b2b7e)

Co-authored-by: Steve Dower <steve.dower@python.org>
2022-10-20 07:33:12 -07:00
Miss Islington (bot) f59506780a
gh-96819: multiprocessing.resource_tracker: check if length of pipe write <= 512 (GH-96890)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
(cherry picked from commit 19ca114645)

Co-authored-by: Koki Saito <49419225+saito828koki@users.noreply.github.com>
2022-10-02 18:09:58 -07:00
Miss Islington (bot) 2b7fc1bfe6
GH-83658: make multiprocessing.Pool raise an exception if maxtasksperchild is not None or a positive int (GH-93364) (GH-93923) 2022-06-17 23:31:42 +01:00
Miss Islington (bot) 0c49a09e67
gh-91577: SharedMemory move imports out of methods (GH-91579)
SharedMemory.unlink() uses the unregister() function from resource_tracker. Previously it was imported in the method, but this can fail if the method is called during interpreter shutdown, for example when unlink is part of a __del__() method.

Moving the import to the top of the file, means that the unregister() method is available during interpreter shutdown.

The register call in SharedMemory.__init__() can also use this imported resource_tracker.
(cherry picked from commit 9a458befdd)

Co-authored-by: samtygier <samtygier@yahoo.co.uk>
2022-06-16 07:07:32 -07:00
Miss Islington (bot) aee7d3df1c
gh-90549: Fix leak of global named resources using multiprocessing spawn (GH-30617)
Co-authored-by: XD Trol <milestonejxd@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
(cherry picked from commit 30610d2837)

Co-authored-by: Leo Trol <milestone.jxd@gmail.com>
2022-06-10 08:04:55 -07:00
Gregory P. Smith f6dd14c653
gh-82616: Add process_group support to subprocess.Popen (#23930)
One more thing that can help prevent people from using `preexec_fn`.

Also adds conditional skips to two tests exposing ASAN flakiness on the Ubuntu 20.04 Address Sanitizer Github CI system. When that build is run on more modern systems the "problem" does not show up. It seems ASAN implementation related.

Co-authored-by: Zackery Spytz <zspytz@gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2022-05-05 16:22:32 -07:00
Géry Ogam dfb1b9da8a
bpo-47029: Fix BrokenPipeError in multiprocessing.Queue at garbage collection and explicit close (#31913) 2022-05-03 17:49:57 -06:00
Ma Lin 1699128c48
GH-91355: micro-optimize Connection.send_bytes() method (gh-32247) 2022-05-03 23:41:45 +09:00
Alexander Shadchin f882d33778
Fix missing `f` prefix on f-strings (GH-91910) 2022-04-27 09:30:54 +03:00
Gregory P. Smith cd5726fe67
gh-91401: Add a failsafe way to disable vfork. (#91490)
Just in case there is ever an issue with _posixsubprocess's use of
vfork() due to the complexity of using it properly and potential
directions that Linux platforms where it defaults to on could take, this
adds a failsafe so that users can disable its use entirely by setting
a global flag.

No known reason to disable it exists. But it'd be a shame to encounter
one and not be able to use CPython without patching and rebuilding it.

See the linked issue for some discussion on reasoning.

Also documents the existing way to disable posix_spawn.
2022-04-25 16:19:39 -07:00
Géry Ogam 5576ddbbbc
bpo-46720: Add support for path-like objects to multiprocessing.set_executable for Windows (GH-31279)
This bring the API to be on a par with Unix-like systems.
2022-04-22 15:47:09 -07:00
Victor Stinner 061a8bf77c
gh-91231: Add shutdown_timeout to multiprocessing BaseManager (#32112)
Add an optional keyword 'shutdown_timeout' parameter to the
multiprocessing.BaseManager constructor. Kill the process if
terminate() takes longer than the timeout.

Multiprocessing tests pass test.support.SHORT_TIMEOUT
to BaseManager.shutdown_timeout.
2022-04-19 16:27:00 +02:00
Nikita Sobolev 0cbdd21311
bpo-46565: `del` loop vars that are leaking into module namespaces (GH-30993) 2022-02-03 11:20:08 +02:00
Kumar Aditya af6b406885
bpo-25066: Added repr for multiprocessing.Event (GH-29749)
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
2021-12-09 13:16:45 +00:00
Zackery Spytz d097876111
bpo-38840: Incorrect __all__ in multiprocessing.managers (GH-18034)
This was causing test___all__ to fail on platforms lacking a shared
memory implementation.

Co-Authored-By: Xavier de Gaye <xdegaye@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-08-09 18:44:55 +02:00
Elisha Hollander 62d55a4d11
Remove unnecessary pass statements (GH-27103) 2021-07-13 15:02:30 +02:00
finefoot 85b920498b
bpo-30256: Add manager_owned keyword arg to AutoProxy (GH-16341)
Co-authored-by: Jordan Speicher <jordan@jspeicher.com>
2021-07-01 20:45:02 -07:00
Binbin 17b16e13bb
Fix typos in multiple files (GH-26689)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2021-06-12 22:47:44 -04:00
Victor Stinner 7c29ae1f05
bpo-43867: multiprocessing Server catchs SystemExit (GH-25441)
The multiprocessing Server class now explicitly catchs SystemExit and
closes the client connection in this case. It happens when the
Server.serve_client() method reachs the end of file (EOF).
2021-04-16 19:42:34 +02:00
Inada Naoki 80017752ba
bpo-43651: Fix test_compileall with PEP 597 (GH-25128) 2021-04-02 09:01:57 +09:00
Asheesh Laroia bf2e7e55d7
bpo-40692: Run more test_concurrent_futures tests (GH-20239)
In the case of multiprocessing.synchronize() being missing, the
test_concurrent_futures test suite now skips only the tests that
require multiprocessing.synchronize().

Validate that multiprocessing.synchronize exists as part of
_check_system_limits(), allowing ProcessPoolExecutor to raise
NotImplementedError during __init__, rather than crashing with
ImportError during __init__ when creating a lock imported from
multiprocessing.synchronize.

Use _check_system_limits() to disable tests of
ProcessPoolExecutor on systems without multiprocessing.synchronize.

Running the test suite without multiprocessing.synchronize reveals
that Lib/compileall.py crashes when it uses a ProcessPoolExecutor.
Therefore, change Lib/compileall.py to call _check_system_limits()
before creating the ProcessPoolExecutor.

Note that both Lib/compileall.py and Lib/test/test_compileall.py
were attempting to sanity-check ProcessPoolExecutor by expecting
ImportError. In multiprocessing.resource_tracker, sem_unlink() is also absent
on platforms where POSIX semaphores aren't available. Avoid using
sem_unlink() if it, too, does not exist.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2021-02-08 03:15:51 +00:00
Vinay Sharma 475a5fbb56
bpo-41344: Raise ValueError when creating shared memory of size 0 (GH-21556) 2020-08-30 20:03:11 +01:00
Allen 99c0ee3c89
Fix typo in message from assert statement (GH-21283)
The error message was missing space between the action "acquire" and "_wait_semaphore" which is an attribute for instances of Condition.
2020-08-17 19:08:55 +05:30
idomic db098bc1f0
bpo-39244: multiprocessing return default start method first on macOS (GH-18625) 2020-05-26 17:54:21 +03:00
Victor Stinner eb0d359b4b
bpo-40443: Remove unused imports in stdlib (GH-19815) 2020-05-01 02:38:00 +02:00
Victor Stinner 9adccc1384
bpo-30966: Add multiprocessing.SimpleQueue.close() (GH-19735)
Add a new close() method to multiprocessing.SimpleQueue to explicitly
close the queue.

Automerge-Triggered-By: @pitrou
2020-04-27 09:11:10 -07:00
Antoine Pitrou eba9f6155d
bpo-40330: Fix utf-8 size check in ShareableList (GH-19606)
The item size must be checked after encoding to bytes, not before.

Automerge-Triggered-By: @pitrou
2020-04-20 11:54:55 -07:00
Thomas Krennwallner c8f1715283
bpo-38891: avoid quadratic item access performance of ShareableList (GH-18996)
Avoid linear runtime of ShareableList.__getitem__ and
ShareableList.__setitem__ by storing running allocated bytes in
ShareableList._allocated_bytes instead of the number of bytes for
a particular stored item.

Co-authored-by: Antoine Pitrou <antoine@python.org>
2020-04-19 17:19:24 +02:00
Galden c606624af8
Fix two typos in multiprocessing (GH-19571) 2020-04-18 08:58:29 +02:00
Dong-hee Na a5900ecf9f
bpo-40221: Update multiprocessing to use _at_fork_reinit (GH-19511) 2020-04-15 01:35:36 +09:00
Dong-hee Na e1945307d3
bpo-40221: Update multiprocessing to use _at_fork_reinit (GH-19477) 2020-04-14 22:15:52 +09:00
Batuhan Taşkaya 0361556537
bpo-39481: PEP 585 for a variety of modules (GH-19423)
- concurrent.futures
- ctypes
- http.cookies
- multiprocessing
- queue
- tempfile
- unittest.case
- urllib.parse
2020-04-10 07:46:36 -07:00
Victor Stinner 65a796e527
bpo-40094: Add os.waitstatus_to_exitcode() (GH-19201)
Add os.waitstatus_to_exitcode() function to convert a wait status to an
exitcode.

Suggest waitstatus_to_exitcode() usage in the documentation when
appropriate.

Use waitstatus_to_exitcode() in:

* multiprocessing, os, subprocess and _bootsubprocess modules;
* test.support.wait_process();
* setup.py: run_command();
* and many tests.
2020-04-01 18:49:29 +02:00
Batuhan Taşkaya ac10e0c932
bpo-39360: Ensure all workers exit when finalizing a multiprocessing Pool (GH-19009)
When the pull is not used via the context manager or terminate() is called, there is a system in multiprocessing.util that handles finalization of all pools via an atexit handler (the Finalize) class. This class registers the _terminate_pool handler in the registry of finalizers of the module, and that registry is called on interpreter exit via _exit_function. The problem is that the "happy" path with the context manager or manual call to finalize() does some extra steps that _terminate_pool does not. The step that is not executed when the atexit() handler calls _terminate_pool is pinging the _change_notifier queue to unblock the maintenance threads.

This commit moves the notification to the _terminate_pool function so is called from both code paths.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-03-15 19:45:56 +00:00