Commit Graph

78 Commits

Author SHA1 Message Date
Jason R. Coombs a95a476b3a Issue #20387: Merge test and patch from 3.4.4 2015-06-28 11:13:30 -04:00
Jason R. Coombs b6d1cdda8e Issue #20387: Correct test to properly capture expectation. 2015-06-25 22:42:24 -04:00
Jason R. Coombs 5713b3c5bf Issue #20387: Add test capturing failure to roundtrip indented code in tokenize module. 2015-06-20 19:52:22 -04:00
Jason R. Coombs 7cf36387e4 Remove unused import and remove doctest-only import into doctests. 2015-06-20 19:13:50 -04:00
Victor Stinner 24d262af0b (Merge 3.5) Issue #23840: tokenize.open() now closes the temporary binary file
on error to fix a resource warning.
2015-05-26 00:46:44 +02:00
Victor Stinner 387729e183 Issue #23840: tokenize.open() now closes the temporary binary file on error to
fix a resource warning.
2015-05-26 00:43:58 +02:00
Yury Selivanov 8085b80c18 Issue 24226: Fix parsing of many sequential one-line 'def' statements. 2015-05-18 12:50:52 -04:00
Yury Selivanov 7544508f02 PEP 0492 -- Coroutines with async and await syntax. Issue #24017. 2015-05-11 22:57:16 -04:00
Serhiy Storchaka ee4c0b9dcf Issue #23681: Fixed Python 2 to 3 poring bugs.
Indexing bytes retiurns an integer, not bytes.
2015-03-20 16:48:02 +02:00
Serhiy Storchaka 74a49ac3f5 Issue #23681: Fixed Python 2 to 3 poring bugs.
Indexing bytes retiurns an integer, not bytes.
2015-03-20 16:46:19 +02:00
Benjamin Peterson d51374ed78 PEP 465: a dedicated infix operator for matrix multiplication (closes #21176) 2014-04-09 23:55:56 -04:00
Terry Jan Reedy 9dc3a36c84 Issue #9974: When untokenizing, use row info to insert backslash+newline.
Original patches by A. Kuchling and G. Rees (#12691).
2014-02-23 23:33:08 -05:00
Terry Jan Reedy 938ba685dc Issue #20750, Enable roundtrip tests for new 5-tuple untokenize. The
constructed examples and all but 7 of the test/test_*.py files (run with -ucpu)
pass. Remove those that fail the new test from the selection list.
Patch partly based on patches by G. Brandl (#8478) and G. Rees (#12691).
2014-02-23 18:00:31 -05:00
Terry Jan Reedy 5b8d2c3af7 Issue #8478: Untokenizer.compat now processes first token from iterator input.
Patch based on lines from Georg Brandl, Eric Snow, and Gareth Rees.
2014-02-17 23:12:16 -05:00
Terry Jan Reedy 58edfd9ff1 whitespace 2014-02-17 16:49:06 -05:00
Terry Jan Reedy 5e6db31368 Untokenize: An logically incorrect assert tested user input validity.
Replace it with correct logic that raises ValueError for bad input.
Issues #8478 and #12691 reported the incorrect logic.
Add an Untokenize test case and an initial test method.
2014-02-17 16:45:48 -05:00
Serhiy Storchaka 768c16ce02 Issue #18960: Fix bugs with Python source code encoding in the second line.
* The first line of Python script could be executed twice when the source
encoding (not equal to 'utf-8') was specified on the second line.

* Now the source encoding declaration on the second line isn't effective if
the first line contains anything except a comment.

* As a consequence, 'python -x' works now again with files with the source
encoding declarations specified on the second file, and can be used again
to make Python batch files on Windows.

* The tokenize module now ignore the source encoding declaration on the second
line if the first line contains anything except a comment.

* IDLE now ignores the source encoding declaration on the second line if the
first line contains anything except a comment.

* 2to3 and the findnocoding.py script now ignore the source encoding
declaration on the second line if the first line contains anything except
a comment.
2014-01-09 18:36:09 +02:00
Serhiy Storchaka dafea85190 Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script
now detect Python source code encoding only in comment lines.
2013-09-16 23:51:56 +03:00
Ezio Melotti fafa8b7797 #16152: merge with 3.2. 2012-11-03 17:46:51 +02:00
Ezio Melotti 2cc3b4ba9f #16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder. 2012-11-03 17:38:43 +02:00
Florent Xicluna fed2c51eea Merge branch 2012-07-07 12:26:56 +02:00
Florent Xicluna 11f0b41e9d Issue #14990: tokenize: correctly fail with SyntaxError on invalid encoding declaration. 2012-07-07 12:13:35 +02:00
Christian Heimes 0b3847de6d Issue #15096: Drop support for the ur string prefix 2012-06-20 11:17:58 +02:00
Meador Inge 8d5c0b8c19 Issue #15054: Fix incorrect tokenization of 'b' string literals.
Patch by Serhiy Storchaka.
2012-06-16 21:49:08 -05:00
Brett Cannon c33f3f2339 Issue #14629: Mention the filename in SyntaxError exceptions from
tokenizer.detect_encoding() (when available).
2012-04-20 13:23:54 -04:00
Martin v. Löwis 63c39fe38e merge 3.2: issue 14629 2012-04-20 14:37:17 +02:00
Martin v. Löwis 63674f4b52 Issue #14629: Raise SyntaxError in tokenizer.detect_encoding
if the first two lines have non-UTF-8 characters without an encoding declaration.
2012-04-20 14:36:47 +02:00
Armin Ronacher c0eaecafe9 Updated tokenize to support the inverse byte literals new in 3.3 2012-03-04 13:07:57 +00:00
Meador Inge 00c7f85298 Issue #2134: Add support for tokenize.TokenInfo.exact_type. 2012-01-19 00:44:45 -06:00
Ezio Melotti d8b509b192 #13012: use splitlines(keepends=True/False) instead of splitlines(0/1). 2011-09-28 17:37:55 +03:00
Benjamin Peterson 963e40256a tokenize is just broken on test_pep3131.py 2011-08-13 00:33:21 -05:00
Ned Deily 2ea6fccf64 Issue #12587: Correct faulty test file and reference in test_tokenize.
(Patch by Robert Xiao)
2011-07-19 16:15:27 -07:00
Ezio Melotti b3aedd4862 #9424: Replace deprecated assert* methods in the Python test suite. 2010-11-20 19:04:17 +00:00
Victor Stinner 92665ab8c7 test_tokenize: use self.assertEqual() instead of plain assert 2010-11-09 01:11:31 +00:00
Victor Stinner 58c0752a33 Issue #10335: Add tokenize.open(), detect the file encoding using
tokenize.detect_encoding() and open it in read only mode.
2010-11-09 01:08:59 +00:00
Brian Curtin 9f5f65c736 Fix #10258 - clean up resource warning 2010-10-30 21:35:28 +00:00
Antoine Pitrou 5bc4fa7a49 Replace the "compiler" resource with the more generic "cpu", so
as to mark CPU-heavy tests.
2010-10-14 15:34:31 +00:00
Benjamin Peterson 33856de84d handle names starting with non-ascii characters correctly #9712 2010-08-30 14:41:20 +00:00
Benjamin Peterson e992a7da01 remove pointless coding cookie 2010-08-30 13:57:28 +00:00
Mark Dickinson 388122d43b Issue #9337: Make float.__str__ identical to float.__repr__.
(And similarly for complex numbers.)
2010-08-04 20:56:28 +00:00
Mark Dickinson 3c0b317e36 Merged revisions 82356 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r82356 | mark.dickinson | 2010-06-29 08:37:25 +0100 (Tue, 29 Jun 2010) | 1 line

  Spelling.
........
2010-06-29 07:38:37 +00:00
Benjamin Peterson 689a558098 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found 2010-03-18 22:29:52 +00:00
Benjamin Peterson a0dfa82eca Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r75149 | gregory.p.smith | 2009-09-29 16:56:31 -0500 (Tue, 29 Sep 2009) | 3 lines

  Mention issue6972 in extractall docs about overwriting things outside of
  the supplied path.
........
  r75260 | andrew.kuchling | 2009-10-05 16:24:20 -0500 (Mon, 05 Oct 2009) | 1 line

  Wording fix
........
  r75261 | andrew.kuchling | 2009-10-05 16:24:35 -0500 (Mon, 05 Oct 2009) | 1 line

  Fix narkup
........
  r75262 | andrew.kuchling | 2009-10-05 16:25:03 -0500 (Mon, 05 Oct 2009) | 1 line

  Document 'skip' parameter to constructor
........
  r75263 | andrew.kuchling | 2009-10-05 16:25:35 -0500 (Mon, 05 Oct 2009) | 1 line

  Note side benefit of socket.create_connection()
........
  r75265 | andrew.kuchling | 2009-10-05 17:31:11 -0500 (Mon, 05 Oct 2009) | 1 line

  Reword sentence
........
  r75266 | andrew.kuchling | 2009-10-05 17:32:48 -0500 (Mon, 05 Oct 2009) | 1 line

  Use standard comma punctuation; reword some sentences in the docs
........
  r75267 | andrew.kuchling | 2009-10-05 17:42:56 -0500 (Mon, 05 Oct 2009) | 1 line

  Backport r73983: Document the thousands separator.
........
  r75292 | benjamin.peterson | 2009-10-08 22:11:36 -0500 (Thu, 08 Oct 2009) | 1 line

  death to old CVS keyword
........
  r75300 | benjamin.peterson | 2009-10-09 16:48:14 -0500 (Fri, 09 Oct 2009) | 1 line

  fix some coding style
........
  r75376 | benjamin.peterson | 2009-10-11 20:26:07 -0500 (Sun, 11 Oct 2009) | 1 line

  platform we don't care about
........
  r75405 | neil.schemenauer | 2009-10-14 12:17:14 -0500 (Wed, 14 Oct 2009) | 4 lines

  Issue #1754094: Improve the stack depth calculation in the compiler.
  There should be no other effect than a small decrease in memory use.
  Patch by Christopher Tur Lesniewski-Laas.
........
  r75429 | benjamin.peterson | 2009-10-14 20:47:28 -0500 (Wed, 14 Oct 2009) | 1 line

  pep8ify if blocks
........
  r75430 | benjamin.peterson | 2009-10-14 20:49:37 -0500 (Wed, 14 Oct 2009) | 1 line

  use floor division and add a test that exercises the tabsize codepath
........
  r75431 | benjamin.peterson | 2009-10-14 20:56:25 -0500 (Wed, 14 Oct 2009) | 1 line

  change test to what I intended
........
  r75432 | benjamin.peterson | 2009-10-14 22:05:39 -0500 (Wed, 14 Oct 2009) | 1 line

  some cleanups
........
  r75433 | benjamin.peterson | 2009-10-14 22:06:55 -0500 (Wed, 14 Oct 2009) | 1 line

  make inspect.isabstract() always return a boolean; add a test for it, too #7069
........
  r75437 | benjamin.peterson | 2009-10-15 10:44:46 -0500 (Thu, 15 Oct 2009) | 1 line

  only clear a module's __dict__ if the module is the only one with a reference to it #7140
........
  r75445 | vinay.sajip | 2009-10-16 09:06:44 -0500 (Fri, 16 Oct 2009) | 1 line

  Issue #7120: logging: Removed import of multiprocessing which is causing crash in GAE.
........
  r75501 | antoine.pitrou | 2009-10-18 13:37:11 -0500 (Sun, 18 Oct 2009) | 3 lines

  Add a comment about unreachable code, and fix a typo
........
  r75551 | benjamin.peterson | 2009-10-19 22:14:10 -0500 (Mon, 19 Oct 2009) | 1 line

  use property api
........
  r75572 | benjamin.peterson | 2009-10-20 16:55:17 -0500 (Tue, 20 Oct 2009) | 1 line

  clarify buffer arg #7178
........
  r75589 | benjamin.peterson | 2009-10-21 21:26:47 -0500 (Wed, 21 Oct 2009) | 1 line

  whitespace
........
  r75590 | benjamin.peterson | 2009-10-21 21:36:47 -0500 (Wed, 21 Oct 2009) | 1 line

  rewrite to be nice to other implementations
........
  r75591 | benjamin.peterson | 2009-10-21 21:50:38 -0500 (Wed, 21 Oct 2009) | 4 lines

  rewrite for style, clarify, and comments

  Also, use the hasattr() like scheme of allowing BaseException exceptions through.
........
  r75657 | antoine.pitrou | 2009-10-24 07:41:27 -0500 (Sat, 24 Oct 2009) | 3 lines

  Fix compilation error in debug mode.
........
  r75742 | benjamin.peterson | 2009-10-26 17:51:16 -0500 (Mon, 26 Oct 2009) | 1 line

  use 'is' instead of id()
........
  r75868 | benjamin.peterson | 2009-10-27 15:59:18 -0500 (Tue, 27 Oct 2009) | 1 line

  test expect base classes
........
  r75952 | georg.brandl | 2009-10-29 15:38:32 -0500 (Thu, 29 Oct 2009) | 1 line

  Use the correct function name in docstring.
........
  r75953 | georg.brandl | 2009-10-29 15:39:50 -0500 (Thu, 29 Oct 2009) | 1 line

  Remove mention of the old -X command line switch.
........
  r75954 | georg.brandl | 2009-10-29 15:53:00 -0500 (Thu, 29 Oct 2009) | 1 line

  Use constants instead of magic integers for test result.  Do not re-run with --verbose3 for environment changing tests.
........
  r75955 | georg.brandl | 2009-10-29 15:54:03 -0500 (Thu, 29 Oct 2009) | 1 line

  Use a single style for all the docstrings in the math module.
........
  r75956 | georg.brandl | 2009-10-29 16:16:34 -0500 (Thu, 29 Oct 2009) | 1 line

  I do not think the "railroad" program mentioned is still available.
........
  r75957 | georg.brandl | 2009-10-29 16:44:56 -0500 (Thu, 29 Oct 2009) | 1 line

  Fix constant name.
........
  r76057 | benjamin.peterson | 2009-11-02 09:06:45 -0600 (Mon, 02 Nov 2009) | 1 line

  prevent a rather unlikely segfault
........
  r76105 | georg.brandl | 2009-11-04 01:38:12 -0600 (Wed, 04 Nov 2009) | 1 line

  #7259: show correct equivalent for operator.i* operations in docstring; fix minor issues in operator docs.
........
  r76139 | benjamin.peterson | 2009-11-06 19:04:38 -0600 (Fri, 06 Nov 2009) | 1 line

  spelling
........
  r76143 | georg.brandl | 2009-11-07 02:26:07 -0600 (Sat, 07 Nov 2009) | 1 line

  #7271: fix typo.
........
  r76162 | benjamin.peterson | 2009-11-08 22:10:53 -0600 (Sun, 08 Nov 2009) | 1 line

  discuss how to use -p
........
  r76223 | georg.brandl | 2009-11-12 02:29:46 -0600 (Thu, 12 Nov 2009) | 1 line

  Give the profile module a module directive.
........
2009-11-13 02:25:08 +00:00
Benjamin Peterson d3afadaa49 normalize latin-1 and utf-8 variant encodings like the builtin tokenizer does 2009-10-09 21:43:09 +00:00
Benjamin Peterson c9c0f201fe convert old fail* assertions to assert* 2009-06-30 23:06:06 +00:00
Benjamin Peterson 433f32c3be raise a SyntaxError in detect_encoding() when a codec lookup fails like the builtin parser #4021 2008-12-12 01:25:05 +00:00
Benjamin Peterson ee8712cda4 #2621 rename test.test_support to test.support 2008-05-20 21:35:26 +00:00
Neal Norwitz 752abd0d3c Convert a lot of print statements to print functions in docstrings,
documentation, and unused/rarely used functions.
2008-05-13 04:55:24 +00:00
Christian Heimes ba4af493a5 Merged revisions 61964-61979 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r61964 | benjamin.peterson | 2008-03-27 01:25:33 +0100 (Thu, 27 Mar 2008) | 2 lines

  add commas for introductory clauses
........
  r61965 | christian.heimes | 2008-03-27 02:36:21 +0100 (Thu, 27 Mar 2008) | 1 line

  Hopefully added _fileio module to the Windows build system
........
  r61966 | christian.heimes | 2008-03-27 02:38:47 +0100 (Thu, 27 Mar 2008) | 1 line

  Revert commit accident
........
  r61967 | neal.norwitz | 2008-03-27 04:49:54 +0100 (Thu, 27 Mar 2008) | 3 lines

  Fix bytes so it works on 64-bit platforms.
  (Also remove some #if 0 code that is already handled in _getbytevalue.)
........
  r61968 | neal.norwitz | 2008-03-27 05:40:07 +0100 (Thu, 27 Mar 2008) | 1 line

  Fix memory leaks
........
  r61969 | neal.norwitz | 2008-03-27 05:40:50 +0100 (Thu, 27 Mar 2008) | 3 lines

  Fix warnings about using char as an array subscript.  This is not portable
  since char is signed on some platforms and unsigned on others.
........
  r61970 | neal.norwitz | 2008-03-27 06:02:57 +0100 (Thu, 27 Mar 2008) | 1 line

  Fix test_compiler after adding unicode_literals
........
  r61971 | neal.norwitz | 2008-03-27 06:03:11 +0100 (Thu, 27 Mar 2008) | 1 line

  Fix compiler warnings
........
  r61972 | neal.norwitz | 2008-03-27 07:52:01 +0100 (Thu, 27 Mar 2008) | 1 line

  Pluralss only need one s, not 2 (intss -> ints)
........
  r61973 | christian.heimes | 2008-03-27 10:02:33 +0100 (Thu, 27 Mar 2008) | 1 line

  Quick 'n dirty hack: Increase the magic by 2 to force a rebuild of pyc/pyo files on the build bots
........
  r61974 | eric.smith | 2008-03-27 10:42:35 +0100 (Thu, 27 Mar 2008) | 3 lines

  Added test cases for single quoted strings, both forms of triple quotes,
   and some string concatenations.
  Removed unneeded __future__ print_function import.
........
  r61975 | christian.heimes | 2008-03-27 11:35:52 +0100 (Thu, 27 Mar 2008) | 1 line

  Build bots are working again - removing the hack
........
  r61976 | christian.heimes | 2008-03-27 12:46:37 +0100 (Thu, 27 Mar 2008) | 2 lines

  Fixed tokenize tests
  The tokenize module doesn't understand __future__.unicode_literals yet
........
  r61977 | georg.brandl | 2008-03-27 14:27:31 +0100 (Thu, 27 Mar 2008) | 2 lines

  #2248: return result of QUIT from quit().
........
  r61978 | georg.brandl | 2008-03-27 14:34:59 +0100 (Thu, 27 Mar 2008) | 2 lines

  The bug for which there was a test in outstanding_bugs.py was agreed not to be a bug.
........
  r61979 | amaury.forgeotdarc | 2008-03-28 00:23:54 +0100 (Fri, 28 Mar 2008) | 5 lines

  Issue2495: tokenize.untokenize did not insert space between two consecutive string literals:
  "" "" => """", which is invalid code.

  Will backport
........
2008-03-28 00:55:15 +00:00
Trent Nelson 428de65ca9 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been
renamed tokenize and now works with bytes rather than strings. A new
  detect_encoding function has been added for determining source file encoding
  according to PEP-0263. Token sequences returned by tokenize always start
  with an ENCODING token which specifies the encoding used to decode the file.
  This token is used to encode the output of untokenize back to bytes.

Credit goes to Michael "I'm-going-to-name-my-first-child-unittest" Foord from Resolver Systems for this work.
2008-03-18 22:41:35 +00:00