Commit Graph

4670 Commits

Author SHA1 Message Date
Tomas R. b6760b7fa5
gh-130453: pygettext: Allow specifying multiple keywords with the same function name (GH-131380) 2025-04-10 11:06:40 +00:00
Tomas R. a693eaa710
gh-132121: Always escape non-printable characters in pygettext (GH-132122) 2025-04-06 23:15:17 +03:00
Stan Ulbrych ad6a032ceb
gh-131852: Filter out POT-Creation-Date in msgfmt (GH-131880)
Co-authored-by: Tomas R. <tomas.roun8@gmail.com>
2025-04-05 22:38:39 +03:00
Chris Eibl d827d4d018
gh-131591: Fix GENERATE_DEBUG_SECTION for clangcl on Windows (GH-132112) 2025-04-06 00:00:54 +08:00
Bénédikt Tran 0a97427ee5
gh-99108: Implement HACL* HMAC (#130157)
A new extension module, `_hmac`, now exposes the HACL* HMAC (formally verified) implementation.

The HACL* implementation is used as a fallback implementation when the OpenSSL implementation of HMAC
is not available or disabled. For now, only named hash algorithms are recognized and SIMD support provided
by HACL* for the BLAKE2 hash functions is not yet used.
2025-04-04 19:04:00 +02:00
Mark Shannon 7099c75550
GH-131498: Cases generator: manage stacks automatically (GH-132074) 2025-04-04 17:59:36 +01:00
Irit Katriel df59226997
gh-100239: more refined specialisation stats for BINARY_OP/SUBSCR (#132068) 2025-04-04 15:33:31 +01:00
Mark Shannon ad053d8d6a
GH-131498: Cases generator: Parse down to C statement level. (GH-131948)
* Parse down to statement level in the cases generator

* Add handling for #if macros, treating them much like normal ifs.
2025-04-02 16:31:59 +01:00
Tomas R. 87d9983994
gh-130197: pygettext: Test the --escape option (GH-131902) 2025-04-02 11:46:54 +03:00
mpage 053c285f6b
gh-130704: Strength reduce `LOAD_FAST{_LOAD_FAST}` (#130708)
Optimize `LOAD_FAST` opcodes into faster versions that load borrowed references onto the operand stack when we can prove that the lifetime of the local outlives the lifetime of the temporary that is loaded onto the stack.
2025-04-01 10:18:42 -07:00
Mark Shannon c535a132e4
GH-131498: Another refactoring of the code generator (GH-131827)
* Rename 'defined' attribute to 'in_local' to more accurately reflect how it is used

* Make death of variables explicit even for array variables.

* Convert in_memory from boolean to stack offset

* Don't apply liveness analyis to optimizer generated code

* Add 'out' parameter to stack.pop
2025-03-31 13:52:48 +01:00
Michael Droettboom 8614f86b71
gh-131525: Cache the result of tuple_hash (#131529)
* gh-131525: Cache the result of tuple_hash

* Fix debug builds

* Add blurb

* Fix formatting

* Pre-compute empty tuple singleton

* Mostly set the cache within tuple_alloc

* Fixes for TSAN

* Pre-compute empty tuple singleton

* Fix for 32-bit platforms

* Assert that op != NULL in _PyTuple_RESET_HASH_CACHE

* Use FT_ATOMIC_STORE_SSIZE_RELAXED macro

* Update Include/internal/pycore_tuple.h

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

* Fix alignment

* atomic load

* Update Objects/tupleobject.c

Co-authored-by: Chris Eibl <138194463+chris-eibl@users.noreply.github.com>

---------

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Chris Eibl <138194463+chris-eibl@users.noreply.github.com>
2025-03-27 09:57:06 -04:00
Mark Shannon d836d287a7
GH-131729: Consider in-memory state when merging storage and stack (GH-131773) 2025-03-27 08:32:45 +00:00
Mark Shannon 1b8bb1ed0c
GH-131729: Code-gen better liveness analysis (GH-131732)
* Rename 'defined' attribute to 'in_local' to more accurately reflect how it is used

* Make death of variables explicit even for array variables.

* Convert in_memory from boolean to stack offset

* Don't apply liveness analysis to optimizer generated code

* Fix RETURN_VALUE in optimizer
2025-03-26 15:21:35 +00:00
Bojun Ren a26a301f8b
GH-130673: Gracefully handle missing sections in JIT build (GH-130906) 2025-03-25 16:35:39 -07:00
Diego Russo ea0453ee97
GH-130887: Always remove trailing jumps in AArch64 JIT stencils (GH-131042) 2025-03-25 10:15:36 -07:00
Adam Turner 7d9442f0d5
gh-128446: Run Windows CI tests on each commit (#131702) 2025-03-25 09:06:15 +02:00
Hood Chatham 97ab8fc16a
gh-127146: Fix Emscripten build with --pydebug (#131672)
Removes an explicit check that sysconfigdata naming is correct, in favor of reporting at
runtime with the default mechanisms.
2025-03-25 05:43:31 +08:00
Victor Stinner 49fb75c676
gh-131238: Add missing pycore_function.h includes for JIT compiler (#131571) 2025-03-21 23:37:49 +00:00
Mark Shannon d3f6063af1
GH-131513: Cases generator: Allow dead inputs to be reassigned (GH-131515) 2025-03-21 11:38:17 +00:00
Mark Shannon 7ebd71ee14
GH-131498: Remove conditional stack effects (GH-131499)
* Adds some missing #includes
2025-03-20 15:39:38 +00:00
Mark Shannon 83d54fa876
GH-130296: Remove `_PyOpcode_max_stack_effect` as it is no longer used (GH-131493) 2025-03-20 11:42:54 +00:00
Petr Viktorin 4bced29a74
gh-130587: Add hand-written docs for non-OP tokens (GH-130588)
Co-authored-by: Blaise Pabon <blaise@gmail.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
2025-03-19 16:42:11 +01:00
Petr Viktorin 51d309988b
gh-131261: expat/refresh.sh: Expand list of manual steps (GH-131359) 2025-03-18 13:17:43 +01:00
Kumar Aditya 46e88540e6
gh-116738: remove multiprocessing suppressions (#131319) 2025-03-18 00:29:18 +05:30
Gregory P. Smith bb0268f60d
gh-131261: Update libexpat to 2.7.0 (CVE-2024-8176) (#131272) 2025-03-17 14:55:02 +01:00
Mark Shannon a1aeec61c4
GH-131238: Core header refactor (GH-131250)
* Moves most structs in pycore_ header files into pycore_structs.h and pycore_runtime_structs.h

* Removes many cross-header dependencies
2025-03-17 09:19:04 +00:00
Hood Chatham cf288e3c25
gh-127146: Update test skips for Emscripten 4.0.2 (#129474)
Updates test skips to reflect capabilities of Emscripten 4.0.2.
2025-03-17 06:02:27 +08:00
Tomas R. 7ea6e88eb4
gh-85012: Properly reset msgctxt when compiling messages with msgfmt (GH-130525)
Add also human-readable snapshots for tests.
2025-03-13 20:40:40 +02:00
Victor Stinner fcb750af86
gh-131152: Remove unused imports from `Tools/cases_generator` (#131156) 2025-03-13 10:59:51 +01:00
Victor Stinner 85cabed260
gh-131152: Remove unused imports from `Tools/{build,wasm}` (#131155) 2025-03-13 10:58:28 +01:00
Victor Stinner 2ed671b5e3
gh-111178: Change Argument Clinic signature for `@staticmethod` (#131157) (#131159)
Use "PyObject*", instead of "void*", for `@staticmethod` functions to
fix an undefined behavior.
2025-03-13 10:22:58 +01:00
Hood Chatham db1e5827c4
gh-127503: Improve tracebacks on Emscripten when there is a trap (#131158)
Modifies the behavior of the interpreter on crash under Emscripten:
1. No Python traceback shown on segfault/trap
2. The JavaScript source line is shown

The JavaScript source line is super long and completely unenlightening,
whereas the Python traceback is very helpful.
2025-03-13 08:28:15 +08:00
Victor Stinner 061da44bac
gh-111178: Change Argument Clinic signature for `@classmethod` (#131157)
Use "PyObject*", instead of "PyTypeObject*", for `@classmethod`
functions to fix an undefined behavior.
2025-03-12 17:42:07 +01:00
T. Wouters de2f7da77d
gh-115999: Add free-threaded specialization for FOR_ITER (#128798)
Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
Victor Stinner 9d759b63d8
gh-111178: Change Argument Clinic signature for METH_O (#130682)
Use "PyObject*" for METH_O functions to fix an undefined behavior.
2025-03-11 16:33:36 +01:00
Mark Shannon 2bef8ea8ea
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875) 2025-03-10 14:06:56 +00:00
Sam Gross 052cb717f5
gh-124878: Fix race conditions during interpreter finalization (#130649)
The PyThreadState field gains a reference count field to avoid
issues with PyThreadState being a dangling pointer to freed memory.
The refcount starts with a value of two: one reference is owned by the
interpreter's linked list of thread states and one reference is owned by
the OS thread. The reference count is decremented when the thread state
is removed from the interpreter's linked list and before the OS thread
calls `PyThread_hang_thread()`. The thread that decrements it to zero
frees the `PyThreadState` memory.

The `holds_gil` field is moved out of the `_status` bit field, to avoid
a data race where on thread calls `PyThreadState_Clear()`, modifying the
`_status` bit field while the OS thread reads `holds_gil` when
attempting to acquire the GIL.

The `PyThreadState.state` field now has `_Py_THREAD_SHUTTING_DOWN` as a
possible value. This corresponds to the `_PyThreadState_MustExit()`
check. This avoids race conditions in the free threading build when
checking `_PyThreadState_MustExit()`.
2025-03-06 10:38:34 -05:00
Tomasz Pytel aeb2327386
gh-130574: renumber RESUME opcode from 149 to 128 (GH-130685) 2025-03-06 08:59:36 +00:00
Diego Russo 02de9cb9a8
gh-129964: Fix JIT crash on Windows on Arm (GH-130882) 2025-03-05 12:40:50 -08:00
Brandt Bucher 5d8db36bbb
GH-115802: JIT using the "medium" code model on x86_64-unknown-linux-gnu (GH-130097) 2025-03-05 11:54:33 -08:00
mpage d7bb7c7817
gh-118331: Fix a couple of issues when list allocation fails (#130811)
* Fix use after free in list objects

Set the items pointer in the list object to NULL after the items array
is freed during list deallocation. Otherwise, we can end up with a list
object added to the free list that contains a pointer to an already-freed
items array.

* Mark `_PyList_FromStackRefStealOnSuccess` as escaping

I think technically it's not escaping, because the only object that
can be decrefed if allocation fails is an exact list, which cannot
execute arbitrary code when it is destroyed. However, this seems less
intrusive than trying to special cases objects in the assert in `_Py_Dealloc`
that checks for non-null stackpointers and shouldn't matter for performance.
2025-03-05 10:42:09 -08:00
Neil Schemenauer 813bc5694b
gh-130599: use static constants str-to-int conversion (gh-130714)
Avoid a data race in free-threaded builds due to mutating global arrays at
runtime.  Instead, compute the constants with an external Python script and
then define them as static global constant arrays.  These constants are
used by `long_from_non_binary_base()`.
2025-03-03 19:00:50 -08:00
Tomas R. 321bf59512
gh-130453: pygettext: Allow overriding default keywords when using `--keyword` (GH-130709) 2025-03-03 18:57:01 +02:00
Mark Shannon 54965f3fb2
GH-130296: Avoid stack transients in four instructions. (GH-130310)
* Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE

* Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS

* Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE

* Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
Sam Gross cc17307faa
gh-124878: Add temporary TSAN suppression for free_threadstate (gh-130602)
The race condition with `free_threadstate` and daemon threads exists in
both the free threading and default builds. We were missing a
suppression in the default build.
2025-02-28 09:27:51 -05:00
Tomas R. 44213bc57c
gh-130453: pygettext: Extend support for specifying custom keywords (GH-130463) 2025-02-25 12:10:54 +02:00
Mark Shannon 014223649c
GH-130396: Use computed stack limits on linux (GH-130398)
* Implement C recursion protection with limit pointers for Linux, MacOS and Windows

* Remove calls to PyOS_CheckStack

* Add stack protection to parser

* Make tests more robust to low stacks

* Improve error messages for stack overflow
2025-02-25 09:24:48 +00:00
Petr Viktorin ef29104f7d
GH-91079: Revert "GH-91079: Implement C stack limits using addresses, not counters. (GH-130007)" for now (GH130413)
Revert "GH-91079: Implement C stack limits using addresses, not counters. (GH-130007)" for now

Unfortunatlely, the change broke some buildbots.

This reverts commit 2498c22fa0.
2025-02-24 11:16:08 +01:00
Sam Gross ca22147547
gh-111924: Fix data races when swapping allocators (gh-130287)
CPython current temporarily changes `PYMEM_DOMAIN_RAW` to the default
allocator during initialization and shutdown. The motivation is to
ensure that core runtime structures are allocated and freed using the
same allocator. However, modifying the current allocator changes global
state and is not thread-safe even with the GIL. Other threads may be
allocating or freeing objects use PYMEM_DOMAIN_RAW; they are not
required to hold the GIL to call PyMem_RawMalloc/PyMem_RawFree.

This adds new internal-only functions like `_PyMem_DefaultRawMalloc`
that aren't affected by calls to `PyMem_SetAllocator()`, so they're
appropriate for Python runtime initialization and finalization. Use
these calls in places where we previously swapped to the default raw
allocator.
2025-02-20 11:31:15 -05:00