Fix some UBSan failures for `pollObject`, `devpollObject`, `pyEpoll_Object` as well as
for `kqueue_event_Object`, `kqueue_queue_Object` and `kqueue_tracking_after_fork`.
Suppress unused return values.
Rename the unused parameter in `METH_NOARGS` and getter/setter methods to
`dummy` and `closure` respectively for semantic purposes.
Explicitly declare `_select_exec` as a `static` function.
Revert "GH-91079: Implement C stack limits using addresses, not counters. (GH-130007)" for now
Unfortunatlely, the change broke some buildbots.
This reverts commit 2498c22fa0.
Fix some UBSan failures for `PyStructObject` and `unpackiterobject`.
We also perform some cleanup by suppressing unused return values and renaming the
unused parameter in `METH_NOARGS` and getter methods to `dummy` and `closure`
respectively for semantic purposes.
Fix UBSan failures for `keyobject`, `lru_list_elem`, `lru_cache_object`.
Suppress unused return values.
Change `_PyPartialObject_CAST` (from #124733) to `partialobject_CAST`
for consistency with the current style for these macros
Newer GCC versions accept both __attribute__((no_sanitize("undefined")))
and __attribute__((no_sanitize_undefined)) so check that the macro is
not already defined.
Store the current running task on the thread state, it makes it thread safe for the free-threading build and while improving performance as there is no lock contention, this effectively makes it lock free.
When accessing the current task of the current running loop in current thread, no locking is required and can be acessed without locking.
In the rare case of accessing current task of a loop running in a different thread, the stop the world pauses is used in free-threading builds to stop all other running threads and find the task for the specified loop.
This also makes it easier for external introspection to find the current task, and now it will be always correct.
* Implement C recursion protection with limit pointers
* Remove calls to PyOS_CheckStack
* Add stack protection to parser
* Make tests more robust to low stacks
* Improve error messages for stack overflow
Add cast macros for date and time objects
Fix UBSan failures for `PyDateTime_Delta`, `PyDateTime_Date`, `PyDateTime_TZInfo`, `PyDateTime_TimeZone`, `PyDateTime_Time`, `PyDateTime_DateTime`, `PyDateTime_IsoCalendarDate`
Fix up naming
In the Python implementation, "Z" was allowed where only "+" or "-" should be allowed in time zone specifiers. In the C implementation, ":" was allowed as a separator between the whole and fractional portion of times (seconds). These have both been forbidden and the error messages harmonized.
Fix some UBSan failures for `dequeobject`, `dequeiterobject`, `defdictobject` and `tuplegetterobject`.
We also perform some cleanup by suppressing unused return values and renaming the
unused argument in `METH_NOARGS` methods to `dummy` for semantic purposes.
Use an atomic operation when setting
`_PyRuntime.signals.unhandled_keyboard_interrupt`. We now only clear the
variable at the start of `_PyRun_Main`, which is the same function where
we check it.
This avoids race conditions where previously another thread might call
`run_eval_code_obj()` and erroneously clear the unhandled keyboard
interrupt.
Updates error messages in datetime and makes them consistent between Python and C.
---------
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Paul Ganssle <1377457+pganssle@users.noreply.github.com>
Add a helper for raising DB-API compatible exceptions based on the
result code of SQLite C APIs. Some APIs do not store the error indicator
on the database pointer, so we need to be able to deduce the DB-API
compatible exception directly from the error code.
- rename _pysqlite_seterror() as set_error_from_db()
- introduce set_error_from_code()
This fixes UBSan failures for `dbmobject`.
In addition, we perform some minor cleanup changes such as renaming
some `args` parameter to `dummy` in some `METH_NOARGS` methods and
suppressing an unused return value in `_dbm_module_free`.
Move many functions from _testcapimodule.c into more specific files
in Modules/_testcapi/.
In moved code:
* Replace get_testerror() with PyExc_AssertionError.
* Replace raiseTestError() with
PyErr_Format(PyExc_AssertionError, ...).
* Move PyFunction C API tests to a new file.
* Add Lib/test/test_capi/test_function.py.
* Move tests from test_capi.test_misc to test_capi.test_function.
Move PyType C API tests to a new file.
Move following tests from test_capi.test_misc to test_capi.test_type:
* BuiltinStaticTypesTests
* test_get_type_name()
* test_get_base_by_token()
Unlikely errors in preparing arguments for ctypes callback are now
handled in the same way as errors raised in the callback of in converting
the result of the callback -- using sys.unraisablehook() instead of
sys.excepthook() and not setting sys.last_exc and other variables.
* Add Lib/test/test_capi/test_frame.py file.
* Move C API tests from test_frame to test_capi.test_frame.
* Add Modules/_testcapi/frame.c file.
* Move C API tests from _testcapimodule.c to frame.c
Add tests for the following functions in test_capi.test_file:
* PyFile_FromFd()
* PyFile_GetLine()
* PyFile_NewStdPrinter()
* PyFile_WriteObject()
* PyFile_WriteString()
* PyObject_AsFileDescriptor()
Add Modules/_testlimitedcapi/file.c file.
Remove test_embed.StdPrinterTests which became redundant.
This exposes `_Py_TryIncref` as `PyUnstable_TryIncref()` and the helper
function `_PyObject_SetMaybeWeakref` as `PyUnstable_EnableTryIncRef`.
These are helpers for dealing with unowned references in a safe way,
particularly in the free threading build.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Add a new OS API which will read data directly into a caller provided
writeable buffer protocol object.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
This was missing from gh-124640. It's already covered by the new
test_asyncio/test_free_threading.py in combination with the runtime
assertion in set_ts_asyncio_running_task.
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
This adds basic support to override default messages for domain errors
in the math_1() helper. The sqrt(), atanh(), log2(), log10() and log()
functions were modified as examples. New macro supports gradual
changing of error messages in other 1-arg functions.
Co-authored-by: CharlieZhao <zhaoyu_hit@qq.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Support calling PyTraceMalloc_Track() and PyTraceMalloc_Untrack()
during late Python finalization.
* Call _PyTraceMalloc_Fini() later in Python finalization.
* Test also PyTraceMalloc_Untrack() without the GIL
* PyTraceMalloc_Untrack() now gets the GIL.
* Test also PyTraceMalloc_Untrack() in test_tracemalloc_track_race().
This fixes UBSan failures for the following objects:
- `DictRemoverObject` and `StructParamObject`,
- `CDataObject` and `CFieldObject`, and
- `PyCFuncPtrObject` and `PyCArgObject`.
On the default build, we convert the `LOCK_PTR` and `UNLOCK_PTR` macros to
functions with an unused parameter to ease "unused variable" compiler warnings
suppression. Finally, we also remove some redundant casts to `PyObject *`.
Remove _PyInterpreterState_GetConfigCopy() and
_PyInterpreterState_SetConfig() private functions. PEP 741 "Python
Configuration C API" added a better public C API: PyConfig_Get() and
PyConfig_Set().
In the free threading build, the per thread reference counting uses a
unique id for some objects to index into the local reference count
table. Use 0 instead of -1 to indicate that the id is not assigned. This
avoids bugs where zero-initialized heap type objects look like they have
a unique id assigned.
Implement set_name() with SetThreadDescription() and _get_name() with
GetThreadDescription(). If SetThreadDescription() or
GetThreadDescription() is not available in kernelbase.dll, delete the
method when the _thread module is imported.
Truncate the thread name to 32766 characters.
Co-authored-by: Eryk Sun <eryksun@gmail.com>
tracemalloc_alloc(), tracemalloc_realloc(), tracemalloc_free(),
_PyTraceMalloc_TraceRef() and _PyTraceMalloc_GetMemory() now check
'tracemalloc_config.tracing' after calling TABLES_LOCK().
_PyTraceMalloc_TraceRef() now always returns 0.
* Use TABLES_LOCK() to protect 'tracemalloc_config.tracing'.
* Hold TABLES_LOCK() longer while accessing tables.
* tracemalloc_realloc() and tracemalloc_free() no longer
remove the trace on reentrant call.
* _PyTraceMalloc_Stop() unregisters _PyTraceMalloc_TraceRef().
* _PyTraceMalloc_GetTraces() sets the reentrant flag.
* tracemalloc_clear_traces_unlocked() sets the reentrant flag.
* ssl: Add hex error code to "unknown error" messages
To make it easier to vary the individual parts of the message,
replace the if-ladder with constant format strings by building
the string piece-wise with PyUnicodeWriter.
Use "unknown error (0x%x)" rather than just "unknown error" if we
can't get a better error message. (Hex makes sense as the error
includes two packed parts.)
This change, along with the LOAD_ATTR specializations, makes the
"thread_local_read" micro benchmark in Tools/ftscalingbench/ftscalingbench.py
scale well to multiple threads.
* Makes `_asyncio.Task` and `_asyncio.Future` thread-safe by adding critical sections
* Add assertions to check for thread safety checking locking of object by critical sections in internal functions
* Make `_asyncio.all_tasks` thread safe when eager tasks are used
* Add a thread safety test
Expose error code ``XML_ERROR_NOT_STARTED`` in `xml.parsers.expat.errors` which was
introduced in Expat 2.6.4.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
* Add ssl.HAS_PHA to detect libssl Post-Handshake-Auth support
Co-authored-by: Tomas R. <tomas.roun8@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
This should be a pure refactoring, without user-visible behaviour changes.
Before this change, ctypes uses traditional native C types, usually identified
by [`struct` format characters][struct-chars] when a short (and
identifier-friendly) name is needed:
- `signed char` (`b`) / `unsigned char` (`B`)
- `short` (`h`) / `unsigned short` (`h`)
- `int` (`i`) / `unsigned int` (`i`)
- `long` (`l`) / `unsigned long` (`l`)
- `long long` (`q`) / `unsigned long long` (`q`)
These map to C99 fixed-width types, which this PR switches to: -
- `int8_t`/`uint8_t`
- `int16_t`/`uint16_t`
- `int32_t`/`uint32_t`
- `int64_t`/`uint64_t`
The C standard doesn't guarantee that the “traditional” types must map to the
fixints. But, [`ctypes` currently requires it][swapdefs], so the assumption won't
break anything.
By “map” I mean that the *size* of the types matches. The *alignment*
requirements might not. This needs to be kept in mind but is not an issue in
`ctypes` accessors, which [explicitly handle unaligned memory][memcpy] for the
integer types.
Note that there are 5 “traditional” C type sizes, but 4 fixed-width ones. Two of
the former are functionally identical to one another; which ones they are is
platform-specific (e.g. `int`==`long`==`int32_t`.) This means that one of the
[current][current-impls-1] [implementations][current-impls-2] is redundant on
any given platform.
The fixint types are parametrized by the number of bytes/bits, and one bit for
signedness. This makes it easier to autogenerate code for them or to write
generic macros (though generic API like
[`PyLong_AsNativeBytes`][PyLong_AsNativeBytes] is problematic for performance
reasons -- especially compared to a `memcpy` with compile-time-constant size).
When one has a *different* integer type, determining the corresponding fixint
means a `sizeof` and signedness check. This is easier and more robust than the
current implementations (see [`wchar_t`][sizeof-wchar_t] or
[`_Bool`][sizeof-bool]).
[swapdefs]: https://github.com/python/cpython/blob/v3.13.0/Modules/_ctypes/cfield.c#L420-L444
[struct-chars]: https://docs.python.org/3/library/struct.html#format-characters
[current-impls-1]: https://github.com/python/cpython/blob/v3.13.0/Modules/_ctypes/cfield.c#L470-L653
[current-impls-2]: https://github.com/python/cpython/blob/v3.13.0/Modules/_ctypes/cfield.c#L703-L944
[memcpy]: https://github.com/python/cpython/blob/v3.13.0/Modules/_ctypes/cfield.c#L613
[PyLong_AsNativeBytes]: https://docs.python.org/3/c-api/long.html#c.PyLong_AsNativeBytes
[sizeof-wchar_t]: https://github.com/python/cpython/blob/v3.13.0/Modules/_ctypes/cfield.c#L1547-L1555
[sizeof-bool]: https://github.com/python/cpython/blob/v3.13.0/Modules/_ctypes/cfield.c#L1562-L1572
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
* Add `_PyDictKeys_StringLookupSplit` which does locking on dict keys and
use in place of `_PyDictKeys_StringLookup`.
* Change `_PyObject_TryGetInstanceAttribute` to use that function
in the case of split keys.
* Add `unicodekeys_lookup_split` helper which allows code sharing
between `_Py_dict_lookup` and `_PyDictKeys_StringLookupSplit`.
* Fix locking for `STORE_ATTR_INSTANCE_VALUE`. Create
`_GUARD_TYPE_VERSION_AND_LOCK` uop so that object stays locked and
`tp_version_tag` cannot change.
* Pass `tp_version_tag` to `specialize_dict_access()`, ensuring
the version we store on the cache is the correct one (in case of
it changing during the specalize analysis).
* Split `analyze_descriptor` into `analyze_descriptor_load` and
`analyze_descriptor_store` since those don't share much logic.
Add `descriptor_is_class` helper function.
* In `specialize_dict_access`, double check `_PyObject_GetManagedDict()`
in case we race and dict was materialized before the lock.
* Avoid borrowed references in `_Py_Specialize_StoreAttr()`.
* Use `specialize()` and `unspecialize()` helpers.
* Add unit tests to ensure specializing happens as expected in FT builds.
* Add unit tests to attempt to trigger data races (useful for running under TSAN).
* Add `has_split_table` function to `_testinternalcapi`.
The `PyWeakref_IsDead()` function tests if a weak reference is dead
without any side effects. Although you can also detect if a weak
reference is dead using `PyWeakref_GetRef()`, that function returns a
strong reference that must be `Py_DECREF()`'d, which can introduce side
effects if the last reference is concurrently dropped (at least in the
free threading build).
gh-127897: Update HACL* module from upstream sources to get:
- Lib_Memzero0.c: don't use memset_s() on macOS <10.9
- Use _mm_malloc() for KRML_ALIGNED_MALLOC on macOS <10.15
- Add LEGACY_MACOS macros, use _mm_free() for KRML_ALIGNED_FREE on macOS <10.15
- Add a helper to set an error from locale-encoded `char*`
- Use the helper for gdbm & dlerror messages
Co-authored-by: Victor Stinner <vstinner@python.org>
The function `operator.methodcaller` was not thread-safe since the additional
of the vectorcall method in gh-89013. In the free threading build the issue
is easy to trigger, for the normal build harder.
This makes the `methodcaller` safe by:
* Replacing the lazy initialization with initialization in the constructor.
* Using a stack allocated space for the vectorcall arguments and falling back
to `tp_call` for calls with more than 8 arguments.
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
From the ERR_raise manpage:
ERR_LIB_SYS
This "library code" indicates that a system error is
being reported. In this case, the reason code given
to `ERR_raise()` and `ERR_raise_data()` *must* be
`errno(3)`.
This PR only handles ERR_LIB_SYS for the high-lever error types
SSL_ERROR_SYSCALL and SSL_ERROR_SSL, i.e., not the ones where
OpenSSL indicates it has some more information about the issue.
On Linux, threading.Thread now sets the thread name to the operating
system.
* configure now checks if pthread_getname_np()
and pthread_setname_np() functions are available.
* Add PYTHREAD_NAME_MAXLEN macro.
* Add _thread._NAME_MAXLEN constant for test_threading.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
gh-127572: Fix `test_structmembers` initialization.
The 'C' format code expects an `int` as a destination (not a `char`).
This led to test failures on big-endian platforms like s390x. Use the
'c' format code, which expects a `char` as the destination (but requires
a Python byte objects instead of a str).
* Mark almost all reachable objects before doing collection phase
* Add stats for objects marked
* Visit new frames before each increment
* Update docs
* Clearer calculation of work to do.
- Add `git describe` output to headers generated by `make_ssl_data.py`
This info is more important than the date when the file was generated.
It does mean that the tool now requires a Git checkout of OpenSSL,
not for example a release tarball.
- Regenerate the older file to add the info.
To the other older file, add a note about manual edits.
- Add notes on how to add a new OpenSSL version
- Add 3.4 error messages and multissl tests
Since MultiByteToWideChar()/WideCharToMultiByte() is not reversible if
the data contains invalid UTF-8 sequences, use binary search to
calculate the number of written bytes from the number of written
characters.
Also fix writing incomplete UTF-8 sequences.
Also fix handling of memory allocation failures.
"Generally, mixed-mode arithmetic combining real and complex variables should
be performed directly, not by first coercing the real to complex, lest the sign
of zero be rendered uninformative; the same goes for combinations of pure
imaginary quantities with complex variables." (c) Kahan, W: Branch cuts for
complex elementary functions.
This patch implements mixed-mode arithmetic rules, combining real and
complex variables as specified by C standards since C99 (in particular,
there is no special version for the true division with real lhs
operand). Most C compilers implementing C99+ Annex G have only these
special rules (without support for imaginary type, which is going to be
deprecated in C2y).
If Python fails to start newly created thread
due to failure of underlying PyThread_start_new_thread() call,
its state should be removed from interpreter' thread states list
to avoid its double cleanup.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
The `methodcaller` C vectorcall implementation uses an arguments array
that is shared across calls. The first argument is modified on every
invocation. This isn't thread-safe in the free threading build. I think
it's also not safe in general, but for now just disable it in the free
threading build.
* Mark almost all reachable objects before doing collection phase
* Add stats for objects marked
* Visit new frames before each increment
* Remove lazy dict tracking
* Update docs
* Clearer calculation of work to do.
If SRE(match) function terminates abruptly, either because of a signal
or because memory allocation fails, allocated SRE_REPEAT blocks might
be never released.
Co-authored-by: <wjssz@users.noreply.github.com>
For dlsym(), a return value of NULL does not necessarily indicate
an error [1].
Therefore, to avoid using stale (or NULL) dlerror() values, we must:
1. clear the previous error state by calling dlerror()
2. call dlsym()
3. call dlerror()
If the return value of dlerror() is not NULL, an error occured.
In ctypes we choose to treat a NULL return value from dlsym()
as a "not found" error. This is the same as the fallback
message we use on Windows, Cygwin or when getting/formatting
the error reason fails.
[1]: https://man7.org/linux/man-pages/man3/dlsym.3.html
Signed-off-by: Georgios Alexopoulos <grgalex42@gmail.com>
Signed-off-by: Georgios Alexopoulos <grgalex@ba.uoa.gr>
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
The PyMutex implementation supports unlocking after fork because we
clear the list of waiters in parking_lot.c. This doesn't work as well
for _PyRecursiveMutex because on some systems, such as SerenityOS, the
thread id is not preserved across fork().
These changes makes it easier to backport the _interpreters, _interpqueues, and _interpchannels modules to Python 3.12.
This involves the following:
* add the _PyXI_GET_STATE() and _PyXI_GET_GLOBAL_STATE() macros
* add _PyXIData_lookup_context_t and _PyXIData_GetLookupContext()
* add _Py_xi_state_init() and _Py_xi_state_fini()
These changes makes it easier to backport the _interpreters, _interpqueues, and _interpchannels modules to Python 3.12.
This involves the following:
* rename several structs and typedefs
* add several typedefs
* stop using the PyThreadState.state field directly in parking_lot.c
Move creation of a tuple for var-positional parameter out of
_PyArg_UnpackKeywordsWithVararg().
Merge _PyArg_UnpackKeywordsWithVararg() with _PyArg_UnpackKeywords().
Add a new parameter in _PyArg_UnpackKeywords().
The "parameters" and "converters" attributes of ParseArgsCodeGen no
longer contain the var-positional parameter. It is now available as the
"varpos" attribute. Optimize code generation for var-positional
parameter and reuse the same generating code for functions with and without
keyword parameters.
Add special converters for var-positional parameter. "tuple" represents it as
a Python tuple and "array" represents it as a continuous array of PyObject*.
"object" is a temporary alias of "tuple".
The primary objective here is to allow some later changes to be cleaner. Mostly this involves renaming things and moving a few things around.
* CrossInterpreterData -> XIData
* crossinterpdatafunc -> xidatafunc
* split out pycore_crossinterp_data_registry.h
* add _PyXIData_lookup_t
Petr Viktorin