BaseSelectorEventLoop.sock_connect()
There is a race condition in create_connection() used with wait_for() to have a
timeout. sock_connect() registers the file descriptor of the socket to be
notified of write event (if connect() raises BlockingIOError). When
create_connection() is cancelled with a TimeoutError, sock_connect() coroutine
gets the exception, but it doesn't unregister the file descriptor for write
event. create_connection() gets the TimeoutError and closes the socket.
If you call again create_connection(), the new socket will likely gets the same
file descriptor, which is still registered in the selector. When sock_connect()
calls add_writer(), it tries to modify the entry instead of creating a new one.
This issue was originally reported in the Trollius project, but the bug comes
from Tulip in fact (Trollius is based on Tulip):
https://bitbucket.org/enovance/trollius/issue/15/after-timeouterror-on-wait_for
This change fixes the race condition. It also makes sock_connect() more
reliable (and portable) is sock.connect() raises an InterruptedError.
There is a race condition in create_connection() used with wait_for() to have a
timeout. sock_connect() registers the file descriptor of the socket to be
notified of write event (if connect() raises BlockingIOError). When
create_connection() is cancelled with a TimeoutError, sock_connect() coroutine
gets the exception, but it doesn't unregister the file descriptor for write
event. create_connection() gets the TimeoutError and closes the socket.
If you call again create_connection(), the new socket will likely gets the same
file descriptor, which is still registered in the selector. When sock_connect()
calls add_writer(), it tries to modify the entry instead of creating a new one.
This issue was originally reported in the Trollius project, but the bug comes
from Tulip in fact (Trollius is based on Tulip):
https://bitbucket.org/enovance/trollius/issue/15/after-timeouterror-on-wait_for
This change fixes the race condition. It also makes sock_connect() more
reliable (and portable) is sock.connect() raises an InterruptedError.
Don't raise a TimeoutError if we reached the timeout and the future completed
in the same iteration of the event loop. A side effect of the bug is that
Queue.get() looses items.
Don't raise a TimeoutError if we reached the timeout and the future completed
in the same iteration of the event loop. A side effect of the bug is that
Queue.get() looses items.
* PipeServer.close() now cancels the "accept pipe" future which cancels the
overlapped operation.
* Fix _SelectorTransport.__repr__() if the transport was closed
* Fix debug log in BaseEventLoop.create_connection(): get the socket object
from the transport because SSL transport closes the old socket and creates a
new SSL socket object. Remove also the _SelectorSslTransport._rawsock
attribute: it contained the closed socket (not very useful) and it was not
used.
* Issue #22063: socket operations (sock_recv, sock_sendall, sock_connect,
sock_accept) of the proactor event loop don't raise an exception in debug
mode if the socket are in blocking mode. Overlapped operations also work on
blocking sockets.
* Fix unit tests in debug mode: mock a non-blocking socket for socket
operations which now raise an exception if the socket is blocking.
* _fatal_error() method of _UnixReadPipeTransport and _UnixWritePipeTransport
now log all exceptions in debug mode
* Don't log expected errors in unit tests
* Tulip issue 200: _WaitHandleFuture._unregister_wait() now catchs and logs
exceptions.
* Tulip issue 200: Log errors in debug mode instead of simply ignoring them.
* PipeServer.close() now cancels the "accept pipe" future which cancels the
overlapped operation.
* Fix _SelectorTransport.__repr__() if the transport was closed
* Fix debug log in BaseEventLoop.create_connection(): get the socket object
from the transport because SSL transport closes the old socket and creates a
new SSL socket object. Remove also the _SelectorSslTransport._rawsock
attribute: it contained the closed socket (not very useful) and it was not
used.
* Issue #22063: socket operations (sock_recv, sock_sendall, sock_connect,
sock_accept) of the proactor event loop don't raise an exception in debug
mode if the socket are in blocking mode. Overlapped operations also work on
blocking sockets.
* Fix unit tests in debug mode: mock a non-blocking socket for socket
operations which now raise an exception if the socket is blocking.
* _fatal_error() method of _UnixReadPipeTransport and _UnixWritePipeTransport
now log all exceptions in debug mode
* Don't log expected errors in unit tests
* Tulip issue 200: _WaitHandleFuture._unregister_wait() now catchs and logs
exceptions.
* Tulip issue 200: Log errors in debug mode instead of simply ignoring them.
- Close#22063: socket operations (socket,recv, sock_sendall, sock_connect,
sock_accept) now raise an exception in debug mode if sockets are in blocking
mode.
- asyncio: Use the new os.set_blocking() function of Python 3.5 if available
* _WaitHandleFuture.cancel() now notify IocpProactor through the overlapped
object that the wait was cancelled.
* Optimize IocpProactor.wait_for_handle() gets the result if the wait is
signaled immediatly.
* Enhance representation of Future and Future subclasses
- Add "created at filename:lineno" in the representation
- Add Future._repr_info() method which can be more easily overriden than
Future.__repr__(). It should now be more easy to enhance Future
representation without having to modify each subclass. For example,
_OverlappedFuture and _WaitHandleFuture get the new "created at" information.
- Use reprlib to format Future result, and function arguments when formatting a
callback, to limit the length of the representation.
* Fix repr(_WaitHandleFuture)
* _WaitHandleFuture and _OverlappedFuture: hide frames of internal calls in the
source traceback.
* Cleanup ProactorIocp._poll(): set the timeout to 0 after the first call to
GetQueuedCompletionStatus()
* test_locks: close the temporary event loop and check the condition lock
* Remove workaround in test_futures, no more needed
* _WaitHandleFuture.cancel() now notify IocpProactor through the overlapped
object that the wait was cancelled.
* Optimize IocpProactor.wait_for_handle() gets the result if the wait is
signaled immediatly.
* Enhance representation of Future and Future subclasses
- Add "created at filename:lineno" in the representation
- Add Future._repr_info() method which can be more easily overriden than
Future.__repr__(). It should now be more easy to enhance Future
representation without having to modify each subclass. For example,
_OverlappedFuture and _WaitHandleFuture get the new "created at" information.
- Use reprlib to format Future result, and function arguments when formatting a
callback, to limit the length of the representation.
* Fix repr(_WaitHandleFuture)
* _WaitHandleFuture and _OverlappedFuture: hide frames of internal calls in the
source traceback.
* Cleanup ProactorIocp._poll(): set the timeout to 0 after the first call to
GetQueuedCompletionStatus()
* test_locks: close the temporary event loop and check the condition lock
* Remove workaround in test_futures, no more needed
Improve stability of the proactor event loop, especially operations on
overlapped objects:
* Tulip issue 195: Don't call UnregisterWait() twice if a _WaitHandleFuture is
cancelled twice to fix a crash.
* IocpProactor.close(): cancel futures to cancel overlapped operations, instead
of cancelling directly overlapped operations. Future objects may not call
ov.cancel() if the future was cancelled or if the overlapped was already
cancelled. The cancel() method of the future may also catch exceptions. Log
also errors on cancellation.
* tests: rename "f" to "fut"
* Add a __repr__() method to IocpProactor
* Add a destructor to IocpProactor which closes it
* _OverlappedFuture.cancel() doesn't cancel the overlapped anymore if it is
done: if it is already cancelled or completed. Log also an error if the
cancellation failed.
* Add the address of the overlapped object in repr(_OverlappedFuture)
* _OverlappedFuture truncates the source traceback to hide the call to the
parent constructor (useless in debug).
Improve stability of the proactor event loop, especially operations on
overlapped objects:
* Tulip issue 195: Don't call UnregisterWait() twice if a _WaitHandleFuture is
cancelled twice to fix a crash.
* IocpProactor.close(): cancel futures to cancel overlapped operations, instead
of cancelling directly overlapped operations. Future objects may not call
ov.cancel() if the future was cancelled or if the overlapped was already
cancelled. The cancel() method of the future may also catch exceptions. Log
also errors on cancellation.
* tests: rename "f" to "fut"
* Add a __repr__() method to IocpProactor
* Add a destructor to IocpProactor which closes it
* _OverlappedFuture.cancel() doesn't cancel the overlapped anymore if it is
done: if it is already cancelled or completed. Log also an error if the
cancellation failed.
* Add the address of the overlapped object in repr(_OverlappedFuture)
* _OverlappedFuture truncates the source traceback to hide the call to the
parent constructor (useless in debug).
Adds `load_package_tests` function to test.support, uses it in test_asyncio,
test_email, test_json, test_tools, test_importlib and all test_importlib
sub-packages to implement test discovery.
Since Python 3.3, the C signal handler writes the signal number into the wakeup
file descriptor and then schedules the Python call using Py_AddPendingCall().
asyncio uses the wakeup file descriptor to wake up the event loop, and relies
on Py_AddPendingCall() to schedule the final callback with call_soon().
If the C signal handler is called in a thread different than the thread of the
event loop, the loop is awaken but Py_AddPendingCall() was not called yet. In
this case, the event loop has nothing to do and go to sleep again.
Py_AddPendingCall() is called while the event loop is sleeping again and so the
final callback is not scheduled immediatly.
This patch changes how asyncio handles signals. Instead of relying on
Py_AddPendingCall() and the wakeup file descriptor, asyncio now only relies on
the wakeup file descriptor. asyncio reads signal numbers from the wakeup file
descriptor to call its signal handler.
Since Python 3.3, the C signal handler writes the signal number into the wakeup
file descriptor and then schedules the Python call using Py_AddPendingCall().
asyncio uses the wakeup file descriptor to wake up the event loop, and relies
on Py_AddPendingCall() to schedule the final callback with call_soon().
If the C signal handler is called in a thread different than the thread of the
event loop, the loop is awaken but Py_AddPendingCall() was not called yet. In
this case, the event loop has nothing to do and go to sleep again.
Py_AddPendingCall() is called while the event loop is sleeping again and so the
final callback is not scheduled immediatly.
This patch changes how asyncio handles signals. Instead of relying on
Py_AddPendingCall() and the wakeup file descriptor, asyncio now only relies on
the wakeup file descriptor. asyncio reads signal numbers from the wakeup file
descriptor to call its signal handler.
BrokenPipeError
If you want to handle the BrokenPipeError, you can easily reimplement
communicate().
Add also a unit test to ensure that stdin.write() + stdin.drain() raises
BrokenPipeError.
If you want to handle the BrokenPipeError, you can easily reimplement
communicate().
Add also a unit test to ensure that stdin.write() + stdin.drain() raises
BrokenPipeError.
Victor Stinner