PyPy and potentially other implementations have different or no
contraints on the number of blocks that can be statically nested. move
the test that checks for this behaviour into a unit test and mark it as
CPython-only.
The invalid assignment rules are very delicate since the parser can
easily raise an invalid assignment when a keyword argument is provided.
As they are very deep into the grammar tree, is very difficult to
specify in which contexts these rules can be used and in which don't.
For that, we need to use a different version of the rule that doesn't do
error checking in those situations where we don't want the rule to raise
(keyword arguments and generator expressions).
We also need to check if we are in left-recursive rule, as those can try
to eagerly advance the parser even if the parse will fail at the end of
the expression. Failing to do this allows the parser to start parsing a
call as a tuple and incorrectly identify a keyword argument as an
invalid assignment, before it realizes that it was not a tuple after all.
To improve the user experience understanding what part of the error messages associated with SyntaxErrors is wrong, we can highlight the whole error range and not only place the caret at the first character. In this way:
>>> foo(x, z for z in range(10), t, w)
File "<stdin>", line 1
foo(x, z for z in range(10), t, w)
^
SyntaxError: Generator expression must be parenthesized
becomes
>>> foo(x, z for z in range(10), t, w)
File "<stdin>", line 1
foo(x, z for z in range(10), t, w)
^^^^^^^^^^^^^^^^^^^^
SyntaxError: Generator expression must be parenthesized
* Add to the peg generator a new directive ('&&') that allows to expect
a token and hard fail the parsing if the token is not found. This
allows to quickly emmit syntax errors for missing tokens.
* Use the new grammar element to hard-fail if the ':' is missing before
suites.
Left-recursive rules need to check for errors explicitly, since
even if the rule returns NULL, the parsing might continue and lead
to long-distance failures.
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
The hard part was making all the tests pass; there are some subtle issues here, because apparently the future import wasn't tested very thoroughly in previous Python versions.
For example, `inspect.signature()` returned type objects normally (except for forward references), but strings with the future import. We changed it to try and return type objects by calling `typing.get_type_hints()`, but fall back on returning strings if that function fails (which it may do if there are future references in the annotations that require passing in a specific namespace to resolve).
`GET_INVALID_TARGET` might unexpectedly return `NULL`, which if not
caught will cause a SEGFAULT. Therefore, this commit introduces a new
inline function `RAISE_SYNTAX_ERROR_INVALID_TARGET` that always
checks for `GET_INVALID_TARGET` returning NULL and can be used in
the grammar, replacing the long C ternary operation used till now.
The following error messages get produced:
- `cannot delete ...` for invalid `del` targets
- `... is an illegal 'for' target` for invalid targets in for
statements
- `... is an illegal 'with' target` for invalid targets in
with statements
Additionally, a few `cut`s were added in various places before the
invocation of the `invalid_*` rule, in order to speed things
up.
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
This commit removes the old parser, the deprecated parser module, the old parser compatibility flags and environment variables and all associated support code and documentation.
A line with only a line continuation character should be considered
a blank line at tokenizer level so that only a single NEWLINE token
gets emitted. The old parser was working around the issue, but the
new parser threw a `SyntaxError` for valid input. For example,
an empty line following a line continuation character was interpreted
as a `SyntaxError`.
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
The error message, generated for a non-parenthesized generator expression
in function calls, was still the generic `invalid syntax`, when the generator expression wasn't appearing as the first argument in the call. With this patch, even on input like `f(a, b, c for c in d, e)`, the correct error message gets produced.
This commit fixes the new parser to disallow invalid targets in the
following scenarios:
- Augmented assignments must only accept a single target (Name,
Attribute or Subscript), but no tuples or lists.
- `except` clauses should only accept a single `Name` as a target.
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
When parsing things like `def f(*): pass` the old parser used to output `SyntaxError: named arguments must follow bare *`, which the new parser wasn't able to do.
* Rename PyConfig.use_peg to _use_peg_parser
* Document PyConfig._use_peg_parser and mark it a deprecated
* Mark -X oldparser option and PYTHONOLDPARSER env var as deprecated
in the documentation.
* Add use_old_parser() and skip_if_new_parser() to test.support
* Remove sys.flags.use_peg: use_old_parser() uses
_testinternalcapi.get_configs() instead.
* Enhance test_embed tests
* subprocess._args_from_interpreter_flags() copies -X oldparser
Move the check for dead conditionals (if 0) to the peephole optimizer
and make sure that the code block is still compiled to report any
existing syntax errors within.
Trying to assign a value to __debug__ using the assignment operator is supposed to fail, but
a missing check for forbidden names when setting the context in the ast was preventing this behaviour.
Special thanks to INADA Naoki for pushing the patch through
the last mile, Serhiy Storchaka for reviewing the code, and to
Victor Stinner for suggesting the idea (originally implemented
in the PyPy project).
namespace if it occurs as a free variable in a nested block. This limitation
of the compiler has been lifted, and a new opcode introduced (DELETE_DEREF).
This sample was valid in 2.6, but fails to compile in 3.x without this change::
>>> def f():
... def print_error():
... print(e)
... try:
... something
... except Exception as e:
... print_error()
... # implicit "del e" here
This sample has always been invalid in Python, and now works::
>>> def outer(x):
... def inner():
... return x
... inner()
... del x
There is no need to bump the PYC magic number: the new opcode is used
for code that did not compile before.
Find another way to generate a SyntaxError in the tests.
Previously, these statements would raise something strange like
TypeError: "'int' object is not iterable"
svn+ssh://pythondev@svn.python.org/python/branches/py3k
........
r84214 | amaury.forgeotdarc | 2010-08-19 23:32:38 +0200 (jeu., 19 août 2010) | 3 lines
Add tests for r84209 (crashes in the Ast builder)
Also remove one tab, and move a check closer to the possible failure.
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r73376 | benjamin.peterson | 2009-06-11 17:29:23 -0500 (Thu, 11 Jun 2009) | 1 line
remove check for case handled in sub-function
........
r73393 | alexandre.vassalotti | 2009-06-12 13:56:57 -0500 (Fri, 12 Jun 2009) | 2 lines
Clear reference to the static PyExc_RecursionErrorInst in _PyExc_Fini.
........
r73398 | alexandre.vassalotti | 2009-06-12 15:57:12 -0500 (Fri, 12 Jun 2009) | 3 lines
Add const qualifier to PyErr_SetFromErrnoWithFilename and to
PyErr_SetFromErrnoWithUnicodeFilename.
........
r73400 | alexandre.vassalotti | 2009-06-12 16:43:47 -0500 (Fri, 12 Jun 2009) | 2 lines
Delete outdated make file for building the parser with MSVC 6.
........
r73404 | benjamin.peterson | 2009-06-12 20:40:00 -0500 (Fri, 12 Jun 2009) | 1 line
keep the slice.step field as NULL if no step expression is given
........
r73405 | benjamin.peterson | 2009-06-12 22:46:30 -0500 (Fri, 12 Jun 2009) | 1 line
prevent import statements from assigning to None
........
r73409 | benjamin.peterson | 2009-06-13 08:06:21 -0500 (Sat, 13 Jun 2009) | 1 line
allow importing from a module named None if it has an 'as' clause
........
r73419 | benjamin.peterson | 2009-06-13 11:19:19 -0500 (Sat, 13 Jun 2009) | 1 line
set Print.values to NULL if there are no values
........
r73420 | benjamin.peterson | 2009-06-13 12:08:53 -0500 (Sat, 13 Jun 2009) | 1 line
give a better error message when deleting ()
........
r73421 | benjamin.peterson | 2009-06-13 15:23:33 -0500 (Sat, 13 Jun 2009) | 1 line
when no module is given in a 'from' relative import, make ImportFrom.module NULL
........
r73432 | amaury.forgeotdarc | 2009-06-14 16:20:40 -0500 (Sun, 14 Jun 2009) | 3 lines
#6227: Because of a wrong indentation, the test was not testing what it should.
Ensure that the snippet in doctest_aliases actually contains aliases.
........
r73457 | benjamin.peterson | 2009-06-16 18:13:09 -0500 (Tue, 16 Jun 2009) | 1 line
add underscores
........
r73460 | benjamin.peterson | 2009-06-16 22:23:04 -0500 (Tue, 16 Jun 2009) | 1 line
remove unused 'encoding' member from the compiler struct
........
r73485 | benjamin.peterson | 2009-06-19 17:07:47 -0500 (Fri, 19 Jun 2009) | 1 line
remove duplicate test
........
r73486 | benjamin.peterson | 2009-06-19 17:09:17 -0500 (Fri, 19 Jun 2009) | 1 line
add missing assertion #6313
........
r73488 | benjamin.peterson | 2009-06-19 17:16:28 -0500 (Fri, 19 Jun 2009) | 1 line
show that this one isn't used
........
r73489 | benjamin.peterson | 2009-06-19 17:21:12 -0500 (Fri, 19 Jun 2009) | 1 line
use closures
........
r73501 | benjamin.peterson | 2009-06-21 18:01:07 -0500 (Sun, 21 Jun 2009) | 1 line
don't need to add the name 'lambda' as assigned
........
r73502 | benjamin.peterson | 2009-06-21 18:03:36 -0500 (Sun, 21 Jun 2009) | 1 line
remove tmpname support since it's no longer used
........
r73513 | benjamin.peterson | 2009-06-22 20:18:57 -0500 (Mon, 22 Jun 2009) | 1 line
fix grammar
........
r73514 | benjamin.peterson | 2009-06-22 22:01:56 -0500 (Mon, 22 Jun 2009) | 1 line
remove some unused symtable constants
........
Pablo Galindo Salgado