Commit Graph

32904 Commits

Author SHA1 Message Date
Mark Shannon bc262de06b
GH-125174: Mark objects as statically allocated. (#127797)
* Set a bit in the unused part of the refcount on 64 bit machines and the free-threaded build.

* Use the top of the refcount range on 32 bit machines
2024-12-11 17:37:38 +00:00
Pieter Eendebak b0f278ff05
gh-127065: Make methodcaller thread-safe and re-entrant (GH-127746)
The function `operator.methodcaller` was not thread-safe since the additional
of the vectorcall method in gh-89013. In the free threading build the issue
is easy to trigger, for the normal build harder.

This makes the `methodcaller` safe by:

* Replacing the lazy initialization with initialization in the constructor.
* Using a stack allocated space for the vectorcall arguments and falling back
  to `tp_call` for calls with more than 8 arguments.
2024-12-11 10:06:07 -05:00
Mark Shannon 5a23994a3d
GH-127058: Make `PySequence_Tuple` safer and probably faster. (#127758)
* Use a small buffer, then list when constructing a tuple from an arbitrary sequence.
2024-12-11 14:02:59 +00:00
Nano 359389ed51
gh-123401: Fix http.cookies module to support obsolete RFC 850 date format (#123405)
Co-authored-by: Wulian <1055917385@qq.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2024-12-11 13:28:19 +00:00
Justin Applegate ce76b547f9
gh-126992: Change pickle code to base 10 for load_long and load_int (GH-127042)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-12-11 12:37:59 +00:00
Srinivas Reddy Thatiparthy (తాటిపర్తి శ్రీనివాస్ రెడ్డి) db9bea0386
gh-127740: For odd-length input to bytes.fromhex(...) change the error message to ValueError: fromhex() arg must be of even length (#127756) 2024-12-11 08:35:17 +01:00
Barney Gale 12b4f1a5a1
GH-127381: pathlib ABCs: remove `PathBase.samefile()` and rarer `is_*()` (#127709)
Remove `PathBase.samefile()`, which is fairly specific to the local FS, and
relies on `stat()`, which we're aiming to remove from `PathBase`.

Also remove `PathBase.is_mount()`, `is_junction()`, `is_block_device()`,
`is_char_device()`, `is_fifo()` and `is_socket()`. These rely on POSIX
file type numbers that we're aiming to remove from the `PathBase` API.
2024-12-11 00:09:55 +00:00
Victor Stinner c91ccbe4ac
gh-59705: Set OS thread name when Thread.name is changed (#127702)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-12-10 17:33:11 +01:00
Melissa0x1f992 cef0a90d8f
gh-126937: ctypes: fix TypeError when a field's size is >65535 bytes (GH-126938)
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-12-10 13:13:11 +01:00
Petr Viktorin 690fe077f6
gh-126491: Revert "GH-126491: Lower heap size limit with faster marking (GH-127519)" (GH-127770)
Revert "GH-126491: Lower heap size limit with faster marking (GH-127519)"

This reverts commit 023b7d2141, which introduced
a refleak.
2024-12-10 11:53:56 +01:00
Bénédikt Tran 4331832db0
gh-125420: implement `Sequence.count` API on `memoryview` objects (#125443) 2024-12-10 10:12:33 +00:00
Skip Montanaro 050d59bd17
add help blurb for "extralargefile" (#127710) 2024-12-10 09:17:25 +01:00
Hugo van Kemenade 212448b162
gh-127718: Add colour to `test.regrtest` output (#127719) 2024-12-10 09:44:15 +02:00
Shantanu 3983527c3a
gh-127651: Use __file__ in diagnostics if origin is missing (#127660)
See the left hand side in https://github.com/python/cpython/pull/123929/files#diff-c22186367cbe20233e843261998dc027ae5f1f8c0d2e778abfa454ae74cc59deL2840-L2849

---------

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-12-09 20:55:20 -08:00
Bénédikt Tran 58c753827a
gh-125420: implement `Sequence.index` API on `memoryview` objects (#125446) 2024-12-09 18:48:38 -08:00
Barney Gale 5c89adf385
GH-127456: pathlib ABCs: add protocol for path parser (#127494)
Change the default value of `PurePathBase.parser` from `ParserBase()` to
`posixpath`. As a result, user subclasses of `PurePathBase` and `PathBase`
use POSIX path syntax by default, which is very often desirable.

Move `pathlib._abc.ParserBase` to `pathlib._types.Parser`, and convert it
to a runtime-checkable protocol.

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2024-12-09 18:31:22 +00:00
Bénédikt Tran e85f2f1703
gh-127637: add tests for `dis` command-line interface (#127759) 2024-12-09 18:02:22 +00:00
Russell Keith-Magee 2041a95e68
gh-126925: Modify how iOS test results are gathered (#127592)
Adds a `use_system_log` config item to enable stdout/stderr redirection for
Apple platforms. This log streaming is then used by a new iOS test runner
script, allowing the display of test suite output at runtime. The iOS test
runner script can be used by any Python project, not just the CPython test
suite.
2024-12-09 13:28:57 +08:00
Barney Gale 7f8ec52302
GH-127381: pathlib ABCs: remove `PathBase.unlink()` and `rmdir()` (#127736)
Virtual filesystems don't always make a distinction between deleting files
and empty directories, and sometimes support deleting non-empty directories
in a single operation. Here we remove `PathBase.unlink()` and `rmdir()`,
leaving `_delete()` as the sole deletion method, now made abstract. I hope
to drop the underscore prefix later on.
2024-12-08 18:45:09 +00:00
Apostol Fet 1503fc8f88
gh-127610: Added validation for more than one var-positional and var-keyword parameters in inspect.Signature (GH-127657) 2024-12-08 12:05:15 +02:00
Filipe Laíns 🇵🇸 70154855cf
GH-126789: fix some sysconfig data on late site initializations (#127729) 2024-12-08 05:57:22 +00:00
Barney Gale 79b7cab50a
GH-127090: Fix `urllib.response.addinfourl.url` value for opened `file:` URIs (#127091)
The canonical `file:` URL (as generated by `pathname2url()`) is now used as the `url` attribute of the returned `addinfourl` object. The `addinfourl.url` attribute reflects the resolved URL for both `file:` or `http[s]:` URLs now.
2024-12-07 17:58:42 +00:00
Barney Gale 31c9f3ced2
GH-127381: pathlib ABCs: remove `PathBase.resolve()` and `absolute()` (#127707)
Remove our implementation of POSIX path resolution in `PathBase.resolve()`.
This functionality is rather fragile and isn't necessary in most cases. It
depends on `PathBase.stat()`, which we're looking to remove.

Also remove `PathBase.absolute()`. Many legitimate virtual filesystems lack
the notion of a 'current directory', so it's wrong to include in the basic
interface.
2024-12-06 21:39:45 +00:00
Barney Gale 5b6635f772
GH-127381: pathlib ABCs: remove `PathBase.rename()` and `replace()` (#127658)
These methods are obviated by `PathBase.move()`, which can move directories
and supports any `PathBase` object as a target.
2024-12-06 18:10:00 +00:00
Victor Stinner 67b18a18b6
gh-59705: Add _thread.set_name() function (#127338)
On Linux, threading.Thread now sets the thread name to the operating
system.

* configure now checks if pthread_getname_np()
  and pthread_setname_np() functions are available.
* Add PYTHREAD_NAME_MAXLEN macro.
* Add _thread._NAME_MAXLEN constant for test_threading.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-12-06 16:27:12 +00:00
Peter Bierma 12680ec5bd
gh-127314: Don't mention the GIL when calling without a thread state on the free-threaded build (#127315)
Co-authored-by: Victor Stinner <vstinner@python.org>
2024-12-06 16:58:19 +01:00
Mark Shannon 023b7d2141
GH-126491: Lower heap size limit with faster marking (GH-127519)
* Faster marking of reachable objects

* Changes calculation of work to do and work done.

* Merges transitive closure calculations
2024-12-06 10:46:59 +00:00
Sergey B Kirpichev 8b7c194c7b
gh-120010: Fix invalid (nan+nanj) results in _Py_c_prod() (GH-120287)
In some cases, previously computed as (nan+nanj), we could recover
meaningful component values in the result, see e.g. the C11, Annex
G.5.1, routine _Cmultd():

>>> z = 1e300+1j
>>> z*(nan+infj)  # was (nan+nanj)
(-inf+infj)

That also fix some complex powers for small integer exponents, computed
with optimized algorithm (by squaring):

>>> z**5  # was (nan+nanj)
Traceback (most recent call last):
  File "<python-input-1>", line 1, in <module>
    z**5
    ~^^~
OverflowError: complex exponentiation
2024-12-06 12:28:32 +02:00
J. Nick Koston e991ac8f20
gh-127655: Ensure `_SelectorSocketTransport.writelines` pauses the protocol if needed (#127656)
Ensure `_SelectorSocketTransport.writelines` pauses the protocol if it reaches the high water mark as needed. 

Co-authored-by: Kumar Aditya <kumaraditya@python.org>
2024-12-06 10:03:03 +05:30
Barney Gale 8b3cccf3f9
GH-125413: Revert addition of `pathlib.Path.scandir()` method (#127377)
Remove documentation for `pathlib.Path.scandir()`, and rename the method to
`_scandir()`. In the private pathlib ABCs, make `iterdir()` abstract and
call it from `_scandir()`.

It's not worthwhile to add this method at the moment - see discussion:
https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721

Co-authored-by: Steve Dower <steve.dower@microsoft.com>
2024-12-05 21:39:43 +00:00
Hugo van Kemenade 23f2e8f13c
gh-127221: Add colour to unittest output (#127223)
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
2024-12-05 21:10:46 +02:00
Jakub Kulík d958d9f4a1
GH-126727: Fix test_era_nl_langinfo with Japanese ERAs on Solaris (GH-127327)
Fix test_era_nl_langinfo with Japanese ERAs on Solaris
2024-12-05 20:43:19 +02:00
Peter Bierma 208b0fb645
gh-122431: Disallow negative values in `readline.append_history_file` (#122469)
Co-authored-by: Victor Stinner <vstinner@python.org>
2024-12-05 16:07:38 +00:00
Hood Chatham 43634fc1fc
gh-127146: Emscripten: Skip segfaults in test suite (#127151)
Added skips for tests known to cause problems when running on Emscripten. 
These mostly relate to the limited stack depth on Emscripten.
2024-12-05 08:26:25 +08:00
Bénédikt Tran bc0f2e9459
gh-123378: Ensure results of `PyUnicode*Error_Get{Start,End}` are clamped (GH-123380)
Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>
2024-12-04 14:13:52 +01:00
Sam Gross 6fc6436749
gh-127572: Fix `test_structmembers` initialization (GH-127577)
gh-127572: Fix `test_structmembers` initialization.

The 'C' format code expects an `int` as a destination (not a `char`).
This led to test failures on big-endian platforms like s390x. Use the
'c' format code, which expects a `char` as the destination (but requires
a Python byte objects instead of a str).
2024-12-04 09:58:22 +01:00
Hood Chatham 0f91078170
gh-127146: Resolve some minor problems in Emscripten tests (#127565)
Adjusts some Emscripten test exclusions regarding strftime, fma, and stack depth.
2024-12-04 08:06:20 +08:00
RUANG (James Roy) 12397a5781
gh-112192: Increase the trace module coverage precision to one decimal (#126972) 2024-12-03 23:33:13 +01:00
mpage dabcecfd6d
gh-115999: Enable specialization of `CALL` instructions in free-threaded builds (#127123)
The CALL family of instructions were mostly thread-safe already and only required a small number of changes, which are documented below.

A few changes were needed to make CALL_ALLOC_AND_ENTER_INIT thread-safe:

Added _PyType_LookupRefAndVersion, which returns the type version corresponding to the returned ref.

Added _PyType_CacheInitForSpecialization, which takes an init method and the corresponding type version and only populates the specialization cache if the current type version matches the supplied version. This prevents potentially caching a stale value in free-threaded builds if we race with an update to __init__.

Only cache __init__ functions that are deferred in free-threaded builds. This ensures that the reference to __init__ that is stored in the specialization cache is valid if the type version guard in _CHECK_AND_ALLOCATE_OBJECT passes.
Fix a bug in _CREATE_INIT_FRAME where the frame is pushed to the stack on failure.

A few other miscellaneous changes were also needed:

Use {LOCK,UNLOCK}_OBJECT in LIST_APPEND. This ensures that the list's per-object lock is held while we are appending to it.

Add missing co_tlbc for _Py_InitCleanup.

Stop/start the world around setting the eval frame hook. This allows us to read interp->eval_frame non-atomically and preserves the behavior of _CHECK_PEP_523 documented below.
2024-12-03 11:20:20 -08:00
Neil Schemenauer fc5a0dc224
gh-127271: Replace use of PyCell_GET/SET (gh-127272)
* Replace uses of `PyCell_GET` and `PyCell_SET`.  These macros are not
  safe to use in the free-threaded build.  Use `PyCell_GetRef()` and
  `PyCell_SetTakeRef()` instead. 

* Since `PyCell_GetRef()` returns a strong rather than borrowed ref, some
  code restructuring was required, e.g. `frame_get_var()` returns a strong
  ref now.

* Add critical sections to `PyCell_GET` and `PyCell_SET`.

* Move critical_section.h earlier in the Python.h file.

* Add `PyCell_GET` to the free-threading howto table of APIs that return
  borrowed refs.

* Add additional unit tests for free-threading.
2024-12-03 10:33:06 -08:00
Neil Schemenauer 276cd66ccb
gh-115999: Add free-threaded specialization for `SEND` (gh-127426)
No additional thread safety changes are required.  Note that sending to
a generator that is shared between threads is currently not safe in the
free-threaded build.
2024-12-03 10:25:12 -08:00
mpage 13b68e1a61
gh-127421: Fix race in test_start_new_thread_failed (#127549)
Fix race in test_start_new_thread_failed

When we succeed in starting a new thread, for example if setrlimit
was ineffective, we must wait for the newly spawned thread to exit.
Otherwise, we run the risk that the newly spawned thread will race
with runtime finalization and access memory that has already been
clobbered/freed.

`_thread.start_new_thread()` only spawns daemon threads, which the runtime
does not wait for at shutdown, and does not return a handle. Use
`_thread.start_joinable_thread()` and join the resulting handle when
the thread is started successfully.
2024-12-03 09:50:58 -08:00
Neil Schemenauer 0cb5222079
gh-115999: Specialize `LOAD_SUPER_ATTR` in free-threaded builds (gh-127128)
Use existing helpers to atomically modify the bytecode.  Add unit tests
to ensure specializing is happening as expected.  Add test_specialize.py
that can be used with ThreadSanitizer to detect data races.  
Fix thread safety issue with cell_set_contents().
2024-12-03 09:32:26 -08:00
Tomas R. 8ba9f5bca9
gh-127347: Document `traceback.print_list` (#127348)
Previously, `traceback.print_list` didn't have a documentation entry and was not exposed in `traceback.__all__`. Now it has a documentation entry and is exposed in `__all__`.
2024-12-03 18:08:39 +02:00
Jun Komoda 412e11fe6e
gh-127255: Make `CopyComPointer` public and add to `ctypes` doc. (GH-127275) 2024-12-03 16:35:08 +01:00
Daniele Parmeggiani 979bf2489d
gh-117657: TSAN Fix races in `PyMember_Get` and `PyMember_Set` for C extensions (GH-123211) 2024-12-03 09:41:53 -05:00
Victor Stinner c46acd3588
gh-126876: Fix test_socket.testLargeTimeout() for missing _testcapi (#127517) 2024-12-02 16:51:50 +01:00
Giovanni Siragusa 31f16e427b
gh-109523: Raise a BlockingIOError if reading text from a non-blocking stream cannot immediately return bytes. (GH-122933) 2024-12-02 14:18:30 +01:00
Sergey B Kirpichev 930ba0ce60
gh-126618: fix repr(itertools.count(sys.maxsize)) (#127048)
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2024-12-02 14:14:40 +01:00
Mark Shannon a8dd821d5b
GH-126491: GC: Mark objects reachable from roots before doing cycle collection (GH-127110)
* Mark almost all reachable objects before doing collection phase

* Add stats for objects marked

* Visit new frames before each increment

* Update docs

* Clearer calculation of work to do.
2024-12-02 10:12:17 +00:00
CF Bolz-Tereick 2a373da770
add missing gc_collect() calls in sqlite3 tests (#127446) 2024-12-02 11:11:28 +01:00
Donghee Na e2713409cf
gh-115999: Add partial free-thread specialization for BINARY_SUBSCR (gh-127227) 2024-12-02 10:38:17 +09:00
Zhikang Yan 7ea523f47c
gh-126899: Add `**kw` to `tkinter.Misc.after` and `tkinter.Misc.after_idle` (#126900)
---------
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-12-01 19:29:27 +00:00
Tian Gao 1bc4f076d1
gh-127321: Avoid stopping at an opcode without an associated line number for breakpoint() (#127457) 2024-12-01 11:57:03 -05:00
Peter Bierma 46bfd26fb2
gh-127165: Disallow embedded NULL characters in `_interpreters` (#127199) 2024-12-01 06:33:23 +00:00
Barney Gale 328187cc4f
GH-127381: pathlib ABCs: remove `PathBase.cwd()` and `home()` (#127427)
These classmethods presume that the user has retained the original
`__init__()` signature, which may not be the case. Also, many virtual
filesystems don't provide current or home directories.
2024-11-30 18:39:39 +00:00
Barney Gale 38264a060a
GH-127381: pathlib ABCs: remove `PathBase.lstat()` (#127382)
Remove the `PathBase.lstat()` method, which is a trivial variation of
`stat()`.

No user-facing changes because the pathlib ABCs are still private.
2024-11-29 21:03:39 +00:00
Kumar Aditya 45c5cba318
gh-127316: fix incorrect assertion in setting `__class__` in free-threading (#127399) 2024-11-29 21:44:20 +05:30
Victor Stinner b14fdadc6c
gh-127208: Reject null character in _imp.create_dynamic() (#127400)
_imp.create_dynamic() now rejects embedded null characters in the
path and in the module name.
2024-11-29 16:20:38 +01:00
Victor Stinner bfabf96b50
gh-127258: Fix asyncio test_staggered_race_with_eager_tasks() (#127358)
Replace the sleep(2) with a task which is blocked forever.
2024-11-29 11:39:54 +01:00
Peter Bierma 99490913a0
gh-127341: Argument Clinic: fix compiler warnings for getters with docstrings (#127310)
Co-authored-by: Erlend E. Aasland <erlend@python.org>
2024-11-29 11:12:13 +01:00
Victor Stinner 20657fbdb1
gh-127190: Fix local_setattro() error handling (#127366)
Don't make the assumption that the 'name' argument is a string. Use
repr() to format the 'name' argument instead.
2024-11-28 17:35:48 +01:00
Filipe Laíns 🇵🇸 3a77980002
GH-127178: install a _sysconfig_vars_(...).json file in the stdlib directory (#127302) 2024-11-27 23:32:54 +00:00
Bénédikt Tran 78cb377c62
gh-122288: Improve performances of `fnmatch.translate` (#122289)
Improve performance of this function by a factor of 1.7x.

Co-authored-by: Barney Gale <barney.gale@gmail.com>
2024-11-27 16:42:45 +00:00
Serhiy Storchaka 3cf83d91a5
gh-124008: Fix calculation of the number of written bytes for the Windows console (GH-124059)
Since MultiByteToWideChar()/WideCharToMultiByte() is not reversible if
the data contains invalid UTF-8 sequences, use binary search to
calculate the number of written bytes from the number of written
characters.

Also fix writing incomplete UTF-8 sequences.

Also fix handling of memory allocation failures.
2024-11-27 13:38:12 +02:00
Serhiy Storchaka 83926d3b4c
gh-109746: Fix race condition in test_start_new_thread_failed (GH-127299) 2024-11-27 13:29:50 +02:00
Sergey B Kirpichev 987311d42e
gh-69639: Add mixed-mode rules for complex arithmetic (C-like) (GH-124829)
"Generally, mixed-mode arithmetic combining real and complex variables should
be performed directly, not by first coercing the real to complex, lest the sign
of zero be rendered uninformative; the same goes for combinations of pure
imaginary quantities with complex variables." (c) Kahan, W: Branch cuts for
complex elementary functions.

This patch implements mixed-mode arithmetic rules, combining real and
complex variables as specified by C standards since C99 (in particular,
there is no special version for the true division with real lhs
operand).  Most C compilers implementing C99+ Annex G have only these
special rules (without support for imaginary type, which is going to be
deprecated in C2y).
2024-11-26 17:57:39 +02:00
Jelle Zijlstra dcf629213b
gh-119180: Add VALUE_WITH_FAKE_GLOBALS format to annotationlib (#124415) 2024-11-26 15:40:13 +00:00
Filipe Laíns 🇵🇸 2b0e2b2893
GH-126985: move pyvenv.cfg detection from site to getpath (#126987) 2024-11-26 13:46:33 +00:00
Malcolm Smith 4ca2c82862
gh-124873: Skip timerfd tests on Android (#127279)
* Revert "[3.13] gh-124873: Tolerate 100 ms in TimerfdTests on Android (GH-127101) (#127105)"

This reverts commit c09366b1fe.

* Skip timerfd tests on Android.

Co-authored-by: Victor Stinner <vstinner@python.org>
2024-11-26 09:46:46 +00:00
Beomsoo Kim f46d847574
gh-126946: Improve error message in getopt.do_longs based on existing comment (GH-126871)
Include a list of possibilities for not unique prefix.
2024-11-26 10:54:02 +02:00
mpage 193890c1cc
gh-126612: Include stack effects of uops when computing maximum stack depth (#126894) 2024-11-26 00:53:49 +00:00
Barney Gale 5bb059fe60
GH-127236: `pathname2url()`: generate RFC 1738 URL for absolute POSIX path (#127194)
When handed an absolute Windows path such as `C:\foo` or `//server/share`,
the `urllib.request.pathname2url()` function returns a URL with an
authority section, such as `///C:/foo` or `//server/share` (or before
GH-126205, `////server/share`). Only the `file:` prefix is omitted.

But when handed an absolute POSIX path such as `/etc/hosts`, or a Windows
path of the same form (rooted but lacking a drive), the function returns a
URL without an authority section, such as `/etc/hosts`.

This patch corrects the discrepancy by adding a `//` prefix before
drive-less, rooted paths when generating URLs.
2024-11-25 19:59:20 +00:00
sobolevn a2ee899682
gh-127182: Fix `io.StringIO.__setstate__` crash when `None` is the first value (#127219)
Co-authored-by: Victor Stinner <vstinner@python.org>
2024-11-25 20:32:02 +03:00
Jun Komoda c7f1e3e150
gh-127183: Add `_ctypes.CopyComPointer` tests (GH-127184)
* Make `create_shelllink_persist` top level function.

* Add `CopyComPointerTests`.

* Add more tests.

* Update tests.

* Add assertions for `Release`'s return value.
2024-11-25 14:55:07 +01:00
Serhiy Storchaka 97b2ceaaaf
gh-127217: Fix pathname2url() for paths starting with multiple slashes on Posix (GH-127218) 2024-11-24 19:30:29 +02:00
Bénédikt Tran e0ef08f5b4
gh-122356: restore the position of a file-like object after `zipfile.is_zipfile` (#122397) 2024-11-24 11:36:15 -05:00
Savannah Ostrowski 2104bde572
GH-127133: Remove ability to nest argument groups & mutually exclusive groups (#127186) 2024-11-24 15:20:37 +00:00
Sergey B Kirpichev f7bb658124
gh-113841: fix possible undefined division by 0 in _Py_c_pow() (GH-127211)
`x**y == 1/x**-y ` thus changing `/=` to `*=` by negating the exponent.
2024-11-23 23:37:37 -08:00
Barney Gale 4ea71278ca
pathlib tests: move `walk()` tests into their own classes (GH-126651)
Move tests for Path.walk() into a new PathWalkTest class, and apply a similar change in tests for the ABCs. This allows us to properly tear down the walk test hierarchy in tearDown(), rather than leaving it to os_helper.rmtree().
2024-11-23 18:31:00 -08:00
Barney Gale cc813e10ff
GH-125866: Preserve Windows drive letter case in file URIs (#127138)
Stop converting Windows drive letters to uppercase in
`urllib.request.pathname2url()` and `url2pathname()`. This behaviour is
unnecessary and inconsistent with pathlib's file URI implementation.
2024-11-23 10:41:39 +00:00
Radislav Chugunov ca3ea9ad05
gh-109746: Make _thread.start_new_thread delete state of new thread on its startup failure (GH-109761)
If Python fails to start newly created thread
due to failure of underlying PyThread_start_new_thread() call,
its state should be removed from interpreter' thread states list
to avoid its double cleanup.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-11-22 21:20:07 +02:00
Kirill Podoprigora 27486c3365
gh-115999: Add free-threaded specialization for `UNPACK_SEQUENCE` (#126600)
Add free-threaded specialization for `UNPACK_SEQUENCE` opcode.
`UNPACK_SEQUENCE_TUPLE/UNPACK_SEQUENCE_TWO_TUPLE` are already thread safe since tuples are immutable.
`UNPACK_SEQUENCE_LIST` is not thread safe because of nature of lists (there is nothing preventing another thread from adding items to or removing them the list while the instruction is executing). To achieve thread safety we add a critical section to the implementation of `UNPACK_SEQUENCE_LIST`, especially around the parts where we check the size of the list and push items onto the stack.


---------

Co-authored-by: Matt Page <mpage@meta.com>
Co-authored-by: mpage <mpage@cs.stanford.edu>
2024-11-22 19:00:35 +02:00
Andrei Bodrov 1848ce61f3
gh-88110: Clear concurrent.futures.thread._threads_queues after fork to avoid joining parent process' threads (GH-126098)
Threads are gone after fork, so clear the queues too. Otherwise the
child process (here created via multiprocessing.Process) crashes on
interpreter exit.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-11-22 18:20:34 +02:00
Jun Komoda 7725c0371a
gh-126384: Add tests to verify the behavior of basic COM methods. (GH-126610) 2024-11-22 16:57:31 +01:00
Victor Stinner 0cb20177d6
gh-109413: Fix libregrtest get_running() (#127153)
Skip threads which are not running.
2024-11-22 16:56:03 +01:00
Serhiy Storchaka 8899e85de1
gh-127001: Fix PATHEXT issues in shutil.which() on Windows (GH-127035)
* Name without a PATHEXT extension is only searched if the mode does not
  include X_OK.
* Support multi-component PATHEXT extensions (e.g. ".foo.bar").
* Support files without extensions in PATHEXT contains dot-only extension
  (".", "..", etc).
* Support PATHEXT extensions that end with a dot (e.g. ".foo.").
2024-11-22 17:52:15 +02:00
Serhiy Storchaka 0cb4d6c654
gh-86463: Fix default prog in subparsers if usage is used in the main parser (GH-125891)
The usage parameter of argparse.ArgumentParser no longer
affects the default value of the prog parameter in subparsers.

Previously the full custom usage of the main parser was used as
the prog prefix in subparsers.
2024-11-22 17:29:33 +02:00
Cody Maloney 46f8a7bbdb
gh-127076: Ignore memory mmap in FileIO testing (#127088)
`mmap`, `munmap`, and `mprotect` are used by CPython for memory
management, which may occur in the middle of the FileIO tests. The
system calls can also be used with files, so `strace` includes them
in its `%file` and `%desc` filters.

Filter out the `mmap` system calls related to memory allocation for the
file tests. Currently FileIO doesn't do `mmap` at all, so didn't add
code to track from `mmap` through `munmap` since it wouldn't be used.
For now if an `mmap` on a fd happens, the call will be included (which
may cause test to fail), and at that time support for tracking the
address throug `munmap` could be added.
2024-11-22 15:55:32 +01:00
Tomas R. 0a1944cda8
gh-126700: pygettext: Support more gettext functions (GH-126912)
Support multi-argument gettext functions: ngettext(), pgettext(), dgettext(), etc.
2024-11-22 16:52:16 +02:00
Barney Gale 8c98ed846a
GH-127078: `url2pathname()`: handle extra slash before UNC drive in URL path (#127132)
Decode a file URI like `file://///server/share` as a UNC path like
`\\server\share`. This form of file URI is created by software the simply
prepends `file:///` to any absolute Windows path.
2024-11-22 04:12:50 +00:00
Barney Gale ebf564a1d3
GH-126766: `url2pathname()`: handle 'localhost' authority (#127129)
Discard any 'localhost' authority from the beginning of a `file:` URI. As a
result, file URIs like `//localhost/etc/hosts` are correctly decoded as
`/etc/hosts`.
2024-11-22 03:17:06 +00:00
Barney Gale fd133d4f21
GH-126601: `pathname2url()`: handle NTFS alternate data streams (#126760)
Adjust `pathname2url()` to encode embedded colon characters in Windows
paths, rather than bailing out with an `OSError`.

Co-authored-by: Steve Dower <steve.dower@microsoft.com>
2024-11-22 00:29:05 +00:00
Jacob Bower e8bb053941
gh-126091: Always link generator frames when propagating a thrown-in exception through a yield-from chain (#126092)
Always link generator frames when propagating a thrown-in exception through a yield-from chain.
2024-11-21 17:37:49 -06:00
Donghee Na 78a530a578
gh-115999: Add free-threaded specialization for ``TO_BOOL`` (gh-126616) 2024-11-22 07:52:16 +09:00
mpage 09c240f20c
gh-115999: Specialize `LOAD_GLOBAL` in free-threaded builds (#126607)
Enable specialization of LOAD_GLOBAL in free-threaded builds.

Thread-safety of specialization in free-threaded builds is provided by the following:

A critical section is held on both the globals and builtins objects during specialization. This ensures we get an atomic view of both builtins and globals during specialization.
Generation of new keys versions is made atomic in free-threaded builds.
Existing helpers are used to atomically modify the opcode.
Thread-safety of specialized instructions in free-threaded builds is provided by the following:

Relaxed atomics are used when loading and storing dict keys versions. This avoids potential data races as the dict keys versions are read without holding the dictionary's per-object lock in version guards.
Dicts keys objects are passed from keys version guards to the downstream uops. This ensures that we are loading from the correct offset in the keys object. Once a unicode key has been stored in a keys object for a combined dictionary in free-threaded builds, the offset that it is stored in will never be reused for a different key. Once the version guard passes, we know that we are reading from the correct offset.
The dictionary read fast-path is used to read values from the dictionary once we know the correct offset.
2024-11-21 11:22:21 -08:00
Dino Viehland bf542f8bb9
gh-124470: Fix crash when reading from object instance dictionary while replacing it (#122489)
Delay free a dictionary when replacing it
2024-11-21 10:41:19 -06:00
Sam Gross 3926842117
gh-127020: Make `PyCode_GetCode` thread-safe for free threading (#127043)
Some fields in PyCodeObject are lazily initialized. Use atomics and
critical sections to make their initializations and accesses thread-safe.
2024-11-21 11:00:50 -05:00
Victor Stinner bab4b0462e
gh-124873: Tolerate 100 ms in TimerfdTests on Android (#127101)
On Android, TimerfdTests of test_os now uses 100 ms accuracy instead
of 10 ms.
2024-11-21 15:53:52 +01:00
Nice Zombies 60ec854bc2
gh-126780: Fix `ntpath.normpath()` for drive-relative paths (GH-126801) 2024-11-21 14:43:36 +00:00
Serhiy Storchaka 4803cd0244
gh-126727: Fix locale.nl_langinfo(locale.ERA) (GH-126730)
It now returns multiple era description segments separated by semicolons.
Previously it only returned the first segment on platforms with Glibc.
2024-11-21 13:16:08 +02:00
Serhiy Storchaka eaf2171082
gh-126997: Fix support of non-ASCII strings in pickletools (GH-127062)
* Fix support of STRING and GLOBAL opcodes with non-ASCII arguments.
* dis() now outputs non-ASCII bytes in STRING, BINSTRING and
  SHORT_BINSTRING arguments as escaped (\xXX).
2024-11-21 13:15:12 +02:00
Cody Maloney ff2278e2bf
gh-127076: Disable strace tests under LD_PRELOAD (#127086)
Distribution tooling (ex. sandbox on Gentoo and fakeroot on Debian) uses
LD_PRELOAD to intercept system calls and potentially modify them when
building. These tools can change the set of system calls, so disable
system call testing under these cases.

Co-authored-by: Michał Górny <mgorny@gentoo.org>
2024-11-21 10:33:12 +01:00
Mark Shannon aea0c586d1
GH-127010: Don't lazily track and untrack dicts (GH-127027) 2024-11-20 16:41:20 +00:00
Gregory P. Smith 7191b7662e
gh-97514: Authenticate the forkserver control socket. (GH-99309)
This adds authentication to the forkserver control socket. In the past only filesystem permissions protected this socket from code injection into the forkserver process by limiting access to the same UID, which didn't exist when Linux abstract namespace sockets were used (see issue) meaning that any process in the same system network namespace could inject code. We've since stopped using abstract namespace sockets by default, but protecting our control sockets regardless of type is a good idea.

This reuses the HMAC based shared key auth already used by `multiprocessing.connection` sockets for other purposes.

Doing this is useful so that filesystem permissions are not relied upon and trust isn't implied by default between all processes running as the same UID with access to the unix socket.

### pyperformance benchmarks

No significant changes. Including `concurrent_imap` which exercises `multiprocessing.Pool.imap` in that suite.

### Microbenchmarks

This does _slightly_ slow down forkserver use. How much so appears to depend on the platform. Modern platforms and simple platforms are less impacted. This PR adds additional IPC round trips to the control socket to tell forkserver to spawn a new process. Systems with potentially high latency IPC are naturally impacted more.

Typically a 1-4% slowdown on a very targeted process creation microbenchmark, with a worst case overloaded system slowdown of 20%.  No evidence that these slowdowns appear in practical sense.  See the PR for details.
2024-11-20 08:18:58 -08:00
Brandt Bucher 48c50ff1a2
GH-126892: Reset warmup counters when JIT compiling code (GH-126893) 2024-11-20 08:11:25 -08:00
Serhiy Storchaka addb225f38
gh-126991: Add tests for unpickling bad object state (GH-127031)
This catches a memory leak in loading the BUILD opcode.
2024-11-20 17:31:50 +02:00
Jun Komoda 5b4502560b
gh-126615: `ctypes`: Make `COMError` public (GH-126686)
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2024-11-20 12:53:43 +00:00
Barney Gale c9b399fbdb
GH-85168: Use filesystem encoding when converting to/from `file` URIs (#126852)
Adjust `urllib.request.url2pathname()` and `pathname2url()` to use the
filesystem encoding when quoting and unquoting file URIs, rather than
forcing use of UTF-8.

No changes are needed in the `nturl2path` module because Windows always
uses UTF-8, per PEP 529.
2024-11-19 21:19:30 +00:00
Eric Snow 1c0a104eca
gh-126914: Store the Preallocated Thread State's Pointer in a PyInterpreterState Field (gh-126989)
This approach eliminates the originally reported race. It also gets rid of the deadlock reported in gh-96071, so we can remove the workaround added then.
2024-11-19 12:59:19 -07:00
sobolevn 824afbf548
gh-109413: Enable mypy's `disallow_any_generics` setting when checking `libregrtest` (#127033) 2024-11-19 22:41:59 +03:00
Beomsoo Kim 8da9920a80
gh-126947: Typechecking for _pydatetime.timedelta.__new__ arguments (#126949)
Co-authored-by: sobolevn <mail@sobolevn.me>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
2024-11-19 22:40:52 +03:00
Malcolm Smith c5c9286804
gh-118201: Simplify conv_confname (#126089) 2024-11-19 10:42:19 -05:00
sobolevn 3932e1db53
gh-126980: Fix `bytearray.__buffer__` crash on `PyBUF_{READ,WRITE}` (#126981)
Co-authored-by: Victor Stinner <vstinner@python.org>
2024-11-19 17:44:53 +03:00
Barney Gale 4d771977b1
GH-84850: Remove `urllib.request.URLopener` and `FancyURLopener` (#125739) 2024-11-19 16:01:49 +02:00
Hugo van Kemenade 899fdb213d
Revert "GH-126491: GC: Mark objects reachable from roots before doing cycle collection (GH-126502)" (#126983) 2024-11-19 11:25:09 +02:00
Victor Stinner 84f07c3a4c
gh-126594: Fix typeobject.c wrap_buffer() cast (#126754)
Reject flags smaller than INT_MIN.

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2024-11-19 09:13:20 +01:00
Victor Stinner b3687ad454
gh-126876: Fix socket internal_select() for large timeout (#126968)
If the timeout is larger than INT_MAX, replace it with INT_MAX, in
the poll() code path.

Add an unit test.
2024-11-19 09:08:42 +01:00
Brandt Bucher 4cd10762b0
GH-126795: Increase the JIT threshold from 16 to 4096 (GH-126816) 2024-11-18 11:11:23 -08:00
Hugo van Kemenade 933f21c3c9
gh-85957: Add missing MIME types for images with RFCs (#126966) 2024-11-18 20:13:20 +02:00
Serhiy Storchaka f7ef0203d4
gh-123803: Support arbitrary code page encodings on Windows (GH-123804)
If the cpXXX encoding is not directly implemented in Python, fall back
to use the Windows-specific API codecs.code_page_encode() and
codecs.code_page_decode().
2024-11-18 17:45:25 +00:00
Mark Shannon b0fcc2c47a
GH-126491: GC: Mark objects reachable from roots before doing cycle collection (GH-126502)
* Mark almost all reachable objects before doing collection phase

* Add stats for objects marked

* Visit new frames before each increment

* Remove lazy dict tracking

* Update docs

* Clearer calculation of work to do.
2024-11-18 14:31:26 +00:00
sobolevn a1d9c8aa80
gh-109413: Enable `strict_optional = true` for `libregrtest/run_workers` (#126855)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-11-18 17:23:43 +03:00
Michał Górny 2c0a21c1aa
gh-126909: Fix running xattr tests on systems with lower limits (#126930)
Modify the extended attribute tests to write fewer and smaller extended
attributes, in order to fit within filesystems with total xattr limit
of 1 KiB (e.g. ext4 with 1 KiB blocks).  Previously, the test would
write over 2 KiB, making it fail with ENOSPC on such systems.
2024-11-18 14:05:55 +01:00
Serhiy Storchaka f9c5573ded
gh-101955: Fix SystemError in possesive quantifier with alternative and group (GH-111362)
Co-authored-by: <wjssz@users.noreply.github.com>
2024-11-18 13:43:44 +02:00
Serhiy Storchaka 7538e7f569
gh-67877: Fix memory leaks in terminated RE matching (GH-126840)
If SRE(match) function terminates abruptly, either because of a signal
or because memory allocation fails, allocated SRE_REPEAT blocks might
be never released.

Co-authored-by: <wjssz@users.noreply.github.com>
2024-11-18 11:53:45 +02:00
Russell Keith-Magee 3938fd60c0
gh-126789: Correct sysconfig test exclusions for iOS and Android. (GH-126941) 2024-11-18 09:50:40 +01:00
Filipe Laíns 🇵🇸 9d6366b60d
GH-126920: fix Makefile overwriting sysconfig.get_config_vars 2024-11-17 01:56:01 +00:00
Filipe Laíns 🇵🇸 acbd5c9c6c
GH-126789: fix some sysconfig data on late site initializations 2024-11-17 00:07:25 +00:00
RanKKI ed81971e6b
gh-124452: Fix header mismatches when folding/unfolding with email message (#125919)
The header-folder of the new email API has a long standing known buglet where
if the first token is longer than max_line_length, it puts that token on the next
line.  It turns out there is also a *parsing* bug when parsing such a header:
the space prefixing that first, non-empty line gets preserved and tacked on to
the start of the header value, which is not the expected behavior per the RFCs.
The bug arises from the fact that the parser assumed that there would be at
least one token on the line with the header, which is going to be true for
probably every email producer other than the python email library with its
folding buglet.  Clearly, though, this is a case that needs to be handled
correctly.  The fix is simple: strip the blanks off the start of the whole
value, not just the first physical line of the value.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2024-11-16 18:01:52 -05:00
Nadeshiko Manju 3be7498d24
gh-126476: Raise IllegalMonthError for calendar.formatmonth() when the input month is not correct (GH-126484)
Co-authored-by: Ethan Furman <ethan@stoneleaf.us>
2024-11-15 11:03:06 -08:00
Bénédikt Tran 612ac283b8
gh-122549: Add platform.invalidate_caches() (#122547)
Allow to invalidate platform's cached results.
2024-11-15 13:52:56 +00:00
Peter Bierma 08f98f4576
Fix intermittent failures for the `PyUnstable_Object_EnableDeferredRefcount` tests (GH-126849)
Hotfix for the PyUnstable_Object_EnableDeferredRefcount tests.
2024-11-15 14:03:46 +01:00
Petr Viktorin d00f7b1b9d
gh-125063: marshal: Add version 5, improve documentation (GH-126829)
* Document that slices can be marshalled
* Deduplicate and organize the list of supported types
  in docs
* Organize the type code list in marshal.c, to make
  it more obvious that this is a versioned format
* Back-fill some historical info

Co-authored-by: Michael Droettboom <mdboom@gmail.com>
2024-11-15 13:48:57 +01:00
Peter Bierma d4c72fed8c
gh-126312: Don't traverse frozen objects on the free-threaded build (#126338)
Also, _PyGC_Freeze() no longer freezes unreachable objects.

Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>
2024-11-15 11:21:30 +01:00
George Alexopoulos 8717f792f7
gh-126554: ctypes: Correctly handle NULL dlsym values (GH-126555)
For dlsym(), a return value of NULL does not necessarily indicate
an error [1].

Therefore, to avoid using stale (or NULL) dlerror() values, we must:

 1. clear the previous error state by calling dlerror()
 2. call dlsym()
 3. call dlerror()

If the return value of dlerror() is not NULL, an error occured.

In ctypes we choose to treat a NULL return value from dlsym()
as a "not found" error. This is the same as the fallback
message we use on Windows, Cygwin or when getting/formatting
the error reason fails.

[1]: https://man7.org/linux/man-pages/man3/dlsym.3.html

Signed-off-by: Georgios Alexopoulos <grgalex42@gmail.com>
Signed-off-by: Georgios Alexopoulos <grgalex@ba.uoa.gr>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
2024-11-15 11:05:51 +01:00
Tomas R. 9a456383be
gh-126807: pygettext: Do not attempt to extract messages from function definitions. (GH-126808)
Fixes a bug where pygettext would attempt
to extract a message from a code like this:

def _(x): pass

This is because pygettext only looks at one
token at a time and '_(x)' looks like a
function call.

However, since 'x' is not a string literal,
it would erroneously issue a warning.
2024-11-14 22:17:42 +00:00
Barney Gale cae9d9d20f
GH-126766: `url2pathname()`: handle empty authority section. (#126767)
Discard two leading slashes from the beginning of a `file:` URI if they
introduce an empty authority section. As a result, file URIs like
`///etc/hosts` are correctly parsed as `/etc/hosts`.
2024-11-14 20:22:14 +00:00
Marc Culler 47cbf03885
gh-124111: Update tkinter for compatibility with Tcl/Tk 9.0.0 (GH-124156) 2024-11-14 12:45:08 -06:00
Barney Gale fd4b5453df
GH-118289: Fix handling of non-directories in `posixpath.realpath()` (#120127)
In strict mode, raise `NotADirectoryError` if we encounter a non-directory
while we still have path parts left to process.

We use a `part_count` variable rather than `len(rest)` because the `rest`
stack also contains markers for unresolved symlinks.
2024-11-13 22:59:32 +00:00
CF Bolz-Tereick c695e37a3f
GH-126606: don't write incomplete pyc files (GH-126627)
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
Co-authored-by: Brett Cannon <brett@python.org>
2024-11-13 21:39:10 +00:00
Serhiy Storchaka 35010b8cf2
gh-126390: Support for preserving order of options and nonoption arguments in gnu_getopt() (GH-126393) 2024-11-13 22:50:46 +02:00
sobolevn 12ca7e622f
gh-109413: Enable `strict_optional` for `libregrtest/main.py` (#126394) 2024-11-13 22:29:28 +03:00
Peter Bierma d00878b06a
gh-123619: Add an unstable C API function for enabling deferred reference counting (GH-123635)
Co-authored-by: Sam Gross <colesbury@gmail.com>
2024-11-13 13:27:16 +00:00
Taneli Hukkinen 29b5323c45
gh-126175: Add attributes to TOMLDecodeError. Deprecate free-form `__init__` args (GH-126428) 2024-11-13 13:52:16 +01:00
Ritvik Pasham a12690ef49
gh-126341: add release check to `__iter__` method of `memoryview` (#126759)
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: sobolevn <mail@sobolevn.me>
2024-11-13 11:51:01 +00:00
Red4Ru 1e40c5ba47
gh-104745: Limit starting a patcher more than once without stopping it (#126649)
Previously, this would cause an `AttributeError` if the patch stopped more than once after this, and would also disrupt the original patched object.

---------

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2024-11-13 08:20:38 +00:00
Nice Zombies 4b00aba42e
gh-119826: Improved fallback for ntpath.abspath() on Windows (GH-119938) 2024-11-12 21:18:03 +00:00
Barney Gale bf224bd7ce
GH-120423: `pathname2url()`: handle forward slashes in Windows paths (#126593)
Adjust `urllib.request.pathname2url()` so that forward slashes in Windows
paths are handled identically to backward slashes.
2024-11-12 19:52:30 +00:00
Serhiy Storchaka 7577307ebd
gh-116897: Deprecate generic false values in urllib.parse.parse_qsl() (GH-116903)
Accepting objects with false values (like 0 and []) except empty strings
and byte-like objects and None in urllib.parse functions parse_qsl() and
parse_qs() is now deprecated.
2024-11-12 21:10:29 +02:00
Bénédikt Tran 03924b5dee
gh-89083: add support for UUID version 8 (RFC 9562) (#123224)
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2024-11-12 19:08:49 +01:00
Stephen Morton a83472f49b
gh-126705: Make os.PathLike more like a protocol (#126706)
it can now be used as a base class in other protocols
2024-11-12 17:54:13 +00:00
Hugo van Kemenade 91f4908798
gh-126133: Only use start year in PSF copyright, remove end years (#126236) 2024-11-12 15:59:19 +02:00
RUANG (James Roy) 8ff7efb46d
gh-126061: Add PyLong_IsPositive/Zero/Negative() functions (#126065)
Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
2024-11-12 14:18:06 +01:00
Sayandip Dutta abb90ba46c
gh-125916: Allow functools.reduce() 'initial' to be a keyword argument (#125917) 2024-11-12 13:11:58 +00:00
Bénédikt Tran 6e3bb8a913
gh-126595: fix a crash when calling `itertools.count(sys.maxsize)` (#126617) 2024-11-12 14:10:10 +01:00
Daehee Kim 0ef84b0e2b
gh-126209: Fix inconsistency of `skip_file_prefixes` in `warnings.warn`'s C and Python implementations (GH-126329)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
2024-11-12 13:01:56 +01:00
Stephen Morton feb3e0b19c
gh-126699: allow AsyncIterator to be used as a base for Protocols (#126702) 2024-11-12 09:17:07 +00:00
Sahil Prajapati 599bfc986d
gh-84852: Add MIME types for .eot, ,otf, .ttf, .woff and .woff2 fonts (#20199)
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
2024-11-12 08:48:38 +02:00
Beomsoo Kim 494360afd0
gh-58749: Remove incorrect language spec claims about the global statement (GH-126523)
* Removes erroneous explanation of the `global` statement restrictions; a name declared as global can be subsequently bound using any kind of name binding operation.
* Updates `test_global.py` to also test various name-binding scenarios for global
variables to ensure correct behavior
2024-11-12 10:11:40 +10:00
Tomas R. dff074d144
gh-126413: Add translation tests for getopt and optparse (GH-126698) 2024-11-12 00:16:39 +02:00
Hugo van Kemenade 3c6d2d1230
gh-89416: Add RFC 9559 MIME types for Matroska formats (#126412)
Co-authored-by: Zachary Ware <zachary.ware@gmail.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2024-11-11 21:08:54 +00:00
Serhiy Storchaka 25aee21aa8
gh-126374: Add support of options with optional arguments in the getopt module (GH-126375) 2024-11-11 18:29:28 +02:00
Serhiy Storchaka 79805d2284
gh-117941: Reject option names starting with "--no-" in argparse.BooleanOptionalAction (GH-125894)
They never worked correctly.
2024-11-11 18:28:30 +02:00
Serhiy Storchaka 819830f34a
gh-126505: Fix bugs in compiling case-insensitive character classes (GH-126557)
* upper-case non-BMP character was ignored
* the ASCII flag was ignored when matching a character range whose
  upper bound is beyond the BMP region
2024-11-11 18:27:26 +02:00
sobolevn 9fc2808eaf
gh-126654: Fix crash in several functions in `_interpreters` module (#126678) 2024-11-11 17:05:56 +05:30
Stephen Morton 6ee542d491
gh-126417: validate ABC methods on multiprocessing proxy types (#126454)
Checks that appropriate dunder __ methods exist on the dict and list proxy types.

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-11-10 23:08:58 -08:00
Gregory P. Smith ca878b6e45
gh-117378: Only run the new multiprocessing SysPath test when appropriate (GH-126635)
The first version had it running two forkserver and one spawn tests underneath each of the _fork, _forkserver, and _spawn test suites that build off the generic one.

This adds to the existing complexity of the multiprocessing test suite by offering BaseTestCase classes another attribute to control which suites they are invoked under. Practicality vs purity here. :/

Net result: we don't over-run the new test and their internal logic is simplified.
2024-11-10 13:17:05 -08:00
CF Bolz-Tereick 0f6bb28ff3
Skip test in test_socket.py if `sys.getrefcount` isn't available (#126640)
Skip `testMakefileCloseSocketDestroy` test if `sys.getrefcount` isn't available. This is necessary for  PyPy and other Python implementations that do not have `sys.getrefcount`.
2024-11-10 17:16:36 +02:00
Gregory P. Smith 9d08423b6e
gh-117378: Fix multiprocessing forkserver preload sys.path inheritance. (GH-126538)
gh-117378: Fix multiprocessing forkserver preload sys.path inheritance.

`sys.path` was not properly being sent from the parent process when launching
the multiprocessing forkserver process to preload imports.  This bug has been
there since the forkserver start method was introduced in Python 3.4.  It was
always _supposed_ to inherit `sys.path` the same way the spawn method does.

Observable behavior change: A `''` value in `sys.path` will now be replaced in
the forkserver's `sys.path` with an absolute pathname
`os.path.abspath(os.getcwd())` saved at the time that `multiprocessing` was
imported in the parent process as it already was when using the spawn start
method. **This will only be observable during forkserver preload imports**.

The code invoked before calling things in another process already correctly sets `sys.path`.
Which is likely why this went unnoticed for so long as a mere performance issue in
some configurations.

A workaround for the bug on impacted Pythons is to set PYTHONPATH in the
environment before multiprocessing's forkserver process was started. Not perfect
as that is then inherited by other children, etc, but likely good enough for many
people's purposes.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2024-11-09 23:01:32 +00:00
Barney Gale 266328552e
pathlib ABCs: tighten up `resolve()` and `absolute()` (#126611)
In `PathBase.resolve()`, raise `UnsupportedOperation` if a non-POSIX path
parser is used (our implementation uses `posixpath._realpath()`, which
produces incorrect results for non-POSIX path flavours.) Also tweak code to
call `self.absolute()` upfront rather than supplying an emulated `getcwd()`
function.

Adjust `PathBase.absolute()` to work somewhat like `resolve()`. If a POSIX
path parser is used, we treat the root directory as the current directory.
This is the simplest useful behaviour for concrete path types without a
current directory cursor.
2024-11-09 18:47:49 +00:00
Barney Gale 0f47a3199c
pathlib ABCs: support initializing paths with no arguments (#126608)
In the past I've equivocated about whether to require at least one argument
in the `PurePathBase` (and `PathBase`) initializer, and what the default
should be if we make it optional. I now have a local use case that has
persuaded me to make it optional and default to the empty string (a
`zipp.Path`-like class that treats relative and absolute paths similarly.)
Happily this brings the base class more in line with `PurePath` and `Path`.
2024-11-09 18:21:53 +00:00
Ken Jin 6293d00e72
gh-120619: Strength reduce function guards, support 2-operand uop forms (GH-124846)
Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2024-11-09 11:35:33 +08:00
Barney Gale 54c63a32d0
GH-126212: Fix removal of slashes in file URIs on Windows (#126214)
Adjust `urllib.request.pathname2url()` and `url2pathname()` so that they
don't remove slashes from Windows DOS drive paths and URLs. There was no
basis for this behaviour, and it conflicts with how UNC and POSIX paths are
handled.
2024-11-08 16:47:51 +00:00
Xuanteng Huang 6ec886531f
gh-126072: Set docstring attribute for module and class (#126231) 2024-11-08 15:13:18 +00:00
Serhiy Storchaka 061e50f196
gh-122943: Add the varpos parameter in _PyArg_UnpackKeywords (GH-126564)
Remove _PyArg_UnpackKeywordsWithVararg.
Add comments for integer arguments of _PyArg_UnpackKeywords.
2024-11-08 14:23:50 +02:00
Serhiy Storchaka 06a8b0bb5e
gh-122943: Remove the object converter for var-positional parameter (GH-126560) 2024-11-08 05:41:54 +00:00
Serhiy Storchaka 1f777396f5
gh-122943: Rework support of var-positional parameter in Argument Clinic (GH-122945)
Move creation of a tuple for var-positional parameter out of
_PyArg_UnpackKeywordsWithVararg().
Merge _PyArg_UnpackKeywordsWithVararg() with _PyArg_UnpackKeywords().
Add a new parameter in _PyArg_UnpackKeywords().

The "parameters" and "converters" attributes of ParseArgsCodeGen no
longer contain the var-positional parameter. It is now available as the
"varpos" attribute. Optimize code generation for var-positional
parameter and reuse the same generating code for functions with and without
keyword parameters.

Add special converters for var-positional parameter. "tuple" represents it as
a Python tuple and "array" represents it as a continuous array of PyObject*.
"object" is a temporary alias of "tuple".
2024-11-07 23:40:03 +02:00
Michael Droettboom a38e82bd8c
gh-126298: Don't deduplicate slice constants based on equality (#126398)
* gh-126298: Don't deduplicated slice constants based on equality

* NULL check for PySlice_New

* Fix refcounting

* Fix refcounting some more

* Fix refcounting

* Make tests more complete

* Fix tests
2024-11-07 16:39:23 +00:00
Mark Shannon 85036c8d61
GH-126222: Fix `_PyUop_num_popped` (GH-126507) 2024-11-07 10:48:27 +00:00
Petr Viktorin c9cda1608e
gh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError in ConnectionHandler; rely on __exit__ (GH-126503)
If `read()` in the ConnectionHandler thread raises `OSError` (except `ConnectionError`),
the ConnectionHandler shuts down the entire ThreadedEchoServer,
preventing further connections.
It also does that for `EPROTOTYPE` in `wrap_conn`.

As far as I can see, this is done to avoid the server thread getting stuck,
forgotten, in its accept loop. However, since 2011 (5b95eb90a7)
the server is used as a context manager, and its `__exit__` does `stop()` and `join()`.
(I'm not sure if we *always* used `with` since that commit, but currently we do.)

Make sure that the context manager *is* used, and remove the `server.stop()`
calls from ConnectionHandler.
2024-11-07 11:07:02 +01:00
Petr Viktorin 78ad7e6322
gh-126499: test_ssl: Don't assume err.reason is a string (GH-126501)
The skipping machinery called `getattr(err, "reason", "")` on an arbitrary
exception. As intermittent Buildbot failures show, sometimes it's set
to None.

Convert it to string for this specific check.
2024-11-07 11:06:27 +01:00
Duprat 75f7cf91ec
gh-125679: multiprocessing Lock and RLock - fix invalid representation string on MacOSX. (#125680) 2024-11-07 08:10:57 +00:00
Serhiy Storchaka dbb6e22cb1
gh-125926: Fix urllib.parse.urljoin() for base URI with undefined authority (GH-125989)
Although this goes beyond the application of RFC 3986, urljoin()
should support relative base URIs for backward compatibility.
2024-11-07 09:09:59 +02:00
Serhiy Storchaka 223d3dc554
gh-125631: Enable setting persistent_id and persistent_load of pickler and unpickler (GH-125752)
pickle.Pickler.persistent_id and pickle.Unpickler.persistent_load can
again be overridden as instance attributes.
2024-11-07 08:53:02 +02:00
Stephen Morton 5dc36dc565
gh-126451: Register contextvars.Context to collections.abc.Mapping (#126452)
Co-authored-by: sobolevn <mail@sobolevn.me>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
2024-11-07 01:12:45 +03:00
Serhiy Storchaka 8fa4dc4ba8
gh-126489: Do not call persistent_id() for a persistent id in Python pickle (GH-126490) 2024-11-06 22:25:14 +02:00
Donghee Na e56fd449fb
gh-115999: Move specializer test from test_dis to test_opcache (gh-126498) 2024-11-07 05:23:47 +09:00
Donghee Na 4ea214ea98
gh-115999: Add free-threaded specialization for CONTAINS_OP (gh-126450)
- The specialization logic determines the appropriate specialization using only the operand's type, which is safe to read non-atomically (changing it requires stopping the world). We are guaranteed that the type will not change in between when it is checked and when we specialize the bytecode because the types involved are immutable (you cannot assign to `__class__` for exact instances of `dict`, `set`, or `frozenset`). The bytecode is mutated atomically using helpers.
- The specialized instructions rely on the operand type not changing in between the `DEOPT_IF` checks and the calls to the appropriate type-specific helpers (e.g. `_PySet_Contains`). This is a correctness requirement in the default builds and there are no changes to the opcodes in the free-threaded builds that would invalidate this.
2024-11-06 03:35:10 +00:00
blhsing 83ba8c2bba
gh-70764: inspect.getclosurevars now identifies global variables with LOAD_GLOBAL (#120143) 2024-11-05 15:53:54 -08:00
Barney Gale 5e9168492f
pathlib ABCs: defer path joining (#126409)
Defer joining of path segments in the private `PurePathBase` ABC. The new
behaviour matches how the public `PurePath` class handles path segments.

This removes a hard-to-grok difference between the ABCs and the main
classes. It also slightly reduces the size of `PurePath` objects by
eliminating a `_raw_path` slot.
2024-11-05 21:19:36 +00:00
Stephen Morton 78842e4a98
gh-126417: Register multiprocessing proxy types to an appropriate collections.abc class (#126419) 2024-11-05 15:35:45 +05:30
Peter Bierma 1371295e67
gh-126366: Fix crash if `__iter__` raises an exception during `yield from` (#126369) 2024-11-05 15:26:36 +05:30
Victor Stinner 4a0d574273
gh-120057: Add os.reload_environ() function (#126268)
Replace the os.environ.refresh() method with a new
os.reload_environ() function.

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
2024-11-05 08:43:34 +01:00
Serhiy Storchaka d3840503b0
gh-126303: Fix pickling and copying of os.sched_param objects (GH-126336) 2024-11-05 08:23:17 +02:00
mpage 2e95c5ba3b
gh-115999: Implement thread-local bytecode and enable specialization for `BINARY_OP` (#123926)
Each thread specializes a thread-local copy of the bytecode, created on the first RESUME, in free-threaded builds. All copies of the bytecode for a code object are stored in the co_tlbc array on the code object. Threads reserve a globally unique index identifying its copy of the bytecode in all co_tlbc arrays at thread creation and release the index at thread destruction. The first entry in every co_tlbc array always points to the "main" copy of the bytecode that is stored at the end of the code object. This ensures that no bytecode is copied for programs that do not use threads.

Thread-local bytecode can be disabled at runtime by providing either -X tlbc=0 or PYTHON_TLBC=0. Disabling thread-local bytecode also disables specialization.

Concurrent modifications to the bytecode made by the specializing interpreter and instrumentation use atomics, with specialization taking care not to overwrite an instruction that was instrumented concurrently.
2024-11-04 11:13:32 -08:00
Nice Zombies 3032fcd90e
gh-119793: Add optional length-checking to `map()` (GH-120471)
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Raymond Hettinger <rhettinger@users.noreply.github.com>
2024-11-04 15:00:19 +01:00
sobolevn bfc1d2504c
gh-109413: Add more type hints to `libregrtest` (#126352) 2024-11-04 13:15:57 +03:00
Kumar Aditya fe5a6ab7be
gh-126353: remove implicit creation of loop from `asyncio.get_event_loop` (#126354)
Remove implicit creation of loop from `asyncio.get_event_loop`. This is a step forward of deprecating the policy system of asyncio.
2024-11-04 14:21:20 +05:30
Tomas R. dcae5cd6ab
gh-104400: Add more tests to pygettext (GH-108173) 2024-11-03 14:01:09 +00:00