mirror of https://github.com/python/cpython.git
gh-107305: Update the C-API Docs for PEP 684 (gh-107324)
This commit is contained in:
Eric Snow
GitHub
parent
ecc05e23a1
commit
c0b81c4b54
|
|
@ -1226,7 +1226,95 @@ You can switch between sub-interpreters using the :c:func:`PyThreadState_Swap`
|
|||
function. You can create and destroy them using the following functions:
|
||||
|
||||
|
||||
.. c:function:: PyThreadState* Py_NewInterpreter()
|
||||
.. c:type:: PyInterpreterConfig
|
||||
|
||||
Structure containing most parameters to configure a sub-interpreter.
|
||||
Its values are used only in :c:func:`Py_NewInterpreterFromConfig` and
|
||||
never modified by the runtime.
|
||||
|
||||
.. versionadded:: 3.12
|
||||
|
||||
Structure fields:
|
||||
|
||||
.. c:member:: int use_main_obmalloc
|
||||
|
||||
If this is ``0`` then the sub-interpreter will use its own
|
||||
"object" allocator state.
|
||||
Otherwise it will use (share) the main interpreter's.
|
||||
|
||||
If this is ``0`` then
|
||||
:c:member:`~PyInterpreterConfig.check_multi_interp_extensions`
|
||||
must be ``1`` (non-zero).
|
||||
If this is ``1`` then :c:member:`~PyInterpreterConfig.gil`
|
||||
must not be :c:macro:`PyInterpreterConfig_OWN_GIL`.
|
||||
|
||||
.. c:member:: int allow_fork
|
||||
|
||||
If this is ``0`` then the runtime will not support forking the
|
||||
process in any thread where the sub-interpreter is currently active.
|
||||
Otherwise fork is unrestricted.
|
||||
|
||||
Note that the :mod:`subprocess` module still works
|
||||
when fork is disallowed.
|
||||
|
||||
.. c:member:: int allow_exec
|
||||
|
||||
If this is ``0`` then the runtime will not support replacing the
|
||||
current process via exec (e.g. :func:`os.execv`) in any thread
|
||||
where the sub-interpreter is currently active.
|
||||
Otherwise exec is unrestricted.
|
||||
|
||||
Note that the :mod:`subprocess` module still works
|
||||
when exec is disallowed.
|
||||
|
||||
.. c:member:: int allow_threads
|
||||
|
||||
If this is ``0`` then the sub-interpreter's :mod:`threading` module
|
||||
won't create threads.
|
||||
Otherwise threads are allowed.
|
||||
|
||||
.. c:member:: int allow_daemon_threads
|
||||
|
||||
If this is ``0`` then the sub-interpreter's :mod:`threading` module
|
||||
won't create daemon threads.
|
||||
Otherwise daemon threads are allowed (as long as
|
||||
:c:member:`~PyInterpreterConfig.allow_threads` is non-zero).
|
||||
|
||||
.. c:member:: int check_multi_interp_extensions
|
||||
|
||||
If this is ``0`` then all extension modules may be imported,
|
||||
including legacy (single-phase init) modules,
|
||||
in any thread where the sub-interpreter is currently active.
|
||||
Otherwise only multi-phase init extension modules
|
||||
(see :pep:`489`) may be imported.
|
||||
|
||||
This must be ``1`` (non-zero) if
|
||||
:c:member:`~PyInterpreterConfig.use_main_obmalloc` is ``0``.
|
||||
|
||||
.. c:member:: int gil
|
||||
|
||||
This determines the operation of the GIL for the sub-interpreter.
|
||||
It may be one of the following:
|
||||
|
||||
.. c:namespace:: NULL
|
||||
|
||||
.. c:macro:: PyInterpreterConfig_DEFAULT_GIL
|
||||
|
||||
Use the default selection (:c:macro:`PyInterpreterConfig_SHARED_GIL`).
|
||||
|
||||
.. c:macro:: PyInterpreterConfig_SHARED_GIL
|
||||
|
||||
Use (share) the main interpreter's GIL.
|
||||
|
||||
.. c:macro:: PyInterpreterConfig_OWN_GIL
|
||||
|
||||
Use the sub-interpreter's own GIL.
|
||||
|
||||
If this is :c:macro:`PyInterpreterConfig_OWN_GIL` then
|
||||
:c:member:`PyInterpreterConfig.use_main_obmalloc` must be ``0``.
|
||||
|
||||
|
||||
.. c:function:: PyStatus Py_NewInterpreterFromConfig(PyThreadState **tstate_p, const PyInterpreterConfig *config)
|
||||
|
||||
.. index::
|
||||
pair: module; builtins
|
||||
|
|
@ -1246,16 +1334,47 @@ function. You can create and destroy them using the following functions:
|
|||
``sys.stdout`` and ``sys.stderr`` (however these refer to the same underlying
|
||||
file descriptors).
|
||||
|
||||
The return value points to the first thread state created in the new
|
||||
The given *config* controls the options with which the interpreter
|
||||
is initialized.
|
||||
|
||||
Upon success, *tstate_p* will be set to the first thread state
|
||||
created in the new
|
||||
sub-interpreter. This thread state is made in the current thread state.
|
||||
Note that no actual thread is created; see the discussion of thread states
|
||||
below. If creation of the new interpreter is unsuccessful, ``NULL`` is
|
||||
returned; no exception is set since the exception state is stored in the
|
||||
current thread state and there may not be a current thread state. (Like all
|
||||
other Python/C API functions, the global interpreter lock must be held before
|
||||
calling this function and is still held when it returns; however, unlike most
|
||||
other Python/C API functions, there needn't be a current thread state on
|
||||
entry.)
|
||||
below. If creation of the new interpreter is unsuccessful,
|
||||
*tstate_p* is set to ``NULL``;
|
||||
no exception is set since the exception state is stored in the
|
||||
current thread state and there may not be a current thread state.
|
||||
|
||||
Like all other Python/C API functions, the global interpreter lock
|
||||
must be held before calling this function and is still held when it
|
||||
returns. Likewise a current thread state must be set on entry. On
|
||||
success, the returned thread state will be set as current. If the
|
||||
sub-interpreter is created with its own GIL then the GIL of the
|
||||
calling interpreter will be released. When the function returns,
|
||||
the new interpreter's GIL will be held by the current thread and
|
||||
the previously interpreter's GIL will remain released here.
|
||||
|
||||
.. versionadded:: 3.12
|
||||
|
||||
Sub-interpreters are most effective when isolated from each other,
|
||||
with certain functionality restricted::
|
||||
|
||||
PyInterpreterConfig config = {
|
||||
.use_main_obmalloc = 0,
|
||||
.allow_fork = 0,
|
||||
.allow_exec = 0,
|
||||
.allow_threads = 1,
|
||||
.allow_daemon_threads = 0,
|
||||
.check_multi_interp_extensions = 1,
|
||||
.gil = PyInterpreterConfig_OWN_GIL,
|
||||
};
|
||||
PyThreadState *tstate = Py_NewInterpreterFromConfig(&config);
|
||||
|
||||
Note that the config is used only briefly and does not get modified.
|
||||
During initialization the config's values are converted into various
|
||||
:c:type:`PyInterpreterState` values. A read-only copy of the config
|
||||
may be stored internally on the :c:type:`PyInterpreterState`.
|
||||
|
||||
.. index::
|
||||
single: Py_FinalizeEx()
|
||||
|
|
@ -1290,19 +1409,79 @@ function. You can create and destroy them using the following functions:
|
|||
.. index:: single: close() (in module os)
|
||||
|
||||
|
||||
.. c:function:: PyThreadState* Py_NewInterpreter(void)
|
||||
|
||||
.. index::
|
||||
pair: module; builtins
|
||||
pair: module; __main__
|
||||
pair: module; sys
|
||||
single: stdout (in module sys)
|
||||
single: stderr (in module sys)
|
||||
single: stdin (in module sys)
|
||||
|
||||
Create a new sub-interpreter. This is essentially just a wrapper
|
||||
around :c:func:`Py_NewInterpreterFromConfig` with a config that
|
||||
preserves the existing behavior. The result is an unisolated
|
||||
sub-interpreter that shares the main interpreter's GIL, allows
|
||||
fork/exec, allows daemon threads, and allows single-phase init
|
||||
modules.
|
||||
|
||||
|
||||
.. c:function:: void Py_EndInterpreter(PyThreadState *tstate)
|
||||
|
||||
.. index:: single: Py_FinalizeEx()
|
||||
|
||||
Destroy the (sub-)interpreter represented by the given thread state. The given
|
||||
thread state must be the current thread state. See the discussion of thread
|
||||
states below. When the call returns, the current thread state is ``NULL``. All
|
||||
thread states associated with this interpreter are destroyed. (The global
|
||||
interpreter lock must be held before calling this function and is still held
|
||||
when it returns.) :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
|
||||
Destroy the (sub-)interpreter represented by the given thread state.
|
||||
The given thread state must be the current thread state. See the
|
||||
discussion of thread states below. When the call returns,
|
||||
the current thread state is ``NULL``. All thread states associated
|
||||
with this interpreter are destroyed. The global interpreter lock
|
||||
used by the target interpreter must be held before calling this
|
||||
function. No GIL is held when it returns.
|
||||
|
||||
:c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
|
||||
haven't been explicitly destroyed at that point.
|
||||
|
||||
|
||||
A Per-Interpreter GIL
|
||||
---------------------
|
||||
|
||||
Using :c:func:`Py_NewInterpreterFromConfig` you can create
|
||||
a sub-interpreter that is completely isolated from other interpreters,
|
||||
including having its own GIL. The most important benefit of this
|
||||
isolation is that such an interpreter can execute Python code without
|
||||
being blocked by other interpreters or blocking any others. Thus a
|
||||
single Python process can truly take advantage of multiple CPU cores
|
||||
when running Python code. The isolation also encourages a different
|
||||
approach to concurrency than that of just using threads.
|
||||
(See :pep:`554`.)
|
||||
|
||||
Using an isolated interpreter requires vigilance in preserving that
|
||||
isolation. That especially means not sharing any objects or mutable
|
||||
state without guarantees about thread-safety. Even objects that are
|
||||
otherwise immutable (e.g. ``None``, ``(1, 5)``) can't normally be shared
|
||||
because of the refcount. One simple but less-efficient approach around
|
||||
this is to use a global lock around all use of some state (or object).
|
||||
Alternately, effectively immutable objects (like integers or strings)
|
||||
can be made safe in spite of their refcounts by making them "immortal".
|
||||
In fact, this has been done for the builtin singletons, small integers,
|
||||
and a number of other builtin objects.
|
||||
|
||||
If you preserve isolation then you will have access to proper multi-core
|
||||
computing without the complications that come with free-threading.
|
||||
Failure to preserve isolation will expose you to the full consequences
|
||||
of free-threading, including races and hard-to-debug crashes.
|
||||
|
||||
Aside from that, one of the main challenges of using multiple isolated
|
||||
interpreters is how to communicate between them safely (not break
|
||||
isolation) and efficiently. The runtime and stdlib do not provide
|
||||
any standard approach to this yet. A future stdlib module would help
|
||||
mitigate the effort of preserving isolation and expose effective tools
|
||||
for communicating (and sharing) data between interpreters.
|
||||
|
||||
.. versionadded:: 3.12
|
||||
|
||||
|
||||
Bugs and caveats
|
||||
----------------
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,3 @@
|
|||
Add documentation for :c:type:`PyInterpreterConfig` and
|
||||
:c:func:`Py_NewInterpreterFromConfig`. Also clarify some of the nearby docs
|
||||
relative to per-interpreter GIL.
|
||||
Loading…
Reference in New Issue