| Server IP : 121.121.20.254 / Your IP : 216.73.216.202 Web Server : Microsoft-IIS/10.0 System : Windows NT WEB-SERVER 10.0 build 20348 (Windows Server 2022) AMD64 User : IUSR ( 0) PHP Version : 8.3.28 Disable Function : NONE MySQL : ON | cURL : ON | WGET : OFF | Perl : OFF | Python : OFF | Sudo : OFF | Pkexec : OFF Directory : C:/Python315/Lib/test/__pycache__/ |
Upload File : |
R
>lqj� c � � ] PH s ] PHs] PHs] PH s] PHs] PHs] PHs] PH s] PH s] QH$G
s
G s ] QH0G
s
] QH GsGs ] QH@Gs ] QHHGs ] QHPGsGsGs ] QH`GsGs Q s Q Q
\2 s Q Q \ 2 s Q
s Q Q\2 s Q Q\2 s Q Q\2 s! Q Q\2 s" Q Q\2 s# Q Q\2 s$Q s% Q Q \2 s& Q Q \2 s' Q Q \2 s( Q! Q"\O 2 s)\OT Q# Q$\O 2 2 s+ Q% Q&\O 2 s,\-Q'6X c \O\ 2 P! P! )(� )�BytesIO�StringIO)�dedent)�TestCase�mock)�support)� os_helper)�run_test_script� make_script�run_python_until_end)�VALID_UNDERSCORE_LITERALS�INVALID_UNDERSCORE_LITERALSc �l � , o[ UO 2 2 oUP * Q7 oS F D� v qVqxo U[ O 6X c U! U% c& U[ O 6X c U] * U6X c JM [ O
U* oUO
QUQ
QU8Q
QU QU 02 J� U! )� z
� �10� �13)�len�
splitlines�tokenize� ENDMARKER�NEWLINE�tok_name�append)
�token_generator�
source_string�result� num_lines�missing_trailing_nl�type�token�start�end�lines
&& �'C:\Python315\\Lib\test\test_tokenize.py� stringify_tokens_from_sourcer& s� � �
�F��M�,�,�.�/�I�'��+�6�9��)8�%��U�� �8�%�%� %� � �M� �4�8�+;�+;�#;��A��)�@S� �� � ��&���
�
��T�"�I�Q�u�l�r�l�!�E�7�!�C�5�A�B� *9� �M� c � ` � \ sQ s]*s n Q sQ sQ sQ sQ sQ s Q s
Q s Q s Q
s
Q sQ sQ
sQ sQ sQ sQ sQ sQ sQ sQ sQ sQ sQ sQ s Qs U s P! )� TokenizeTestc � � [ UO Q 2 2 o[ [ O UO 2 U2 oU O UQ,UO
2 O 2 * 2 P! )�utf-8z* ENCODING 'utf-8' (0, 0) (0, 0))r �encoder& r �readline� assertEqual�rstripr ��self�s�expected�fr s &&& r% �check_tokenize�TokenizeTest.check_tokenize1 s` � �
�A�H�H�W�%� &��-�h�.?�.?��
�
�.K�Q�O�� ����F�G�!� � �*�5�5�7�8� 9r' c � � Q oU O [ 2 9^tt^ 2 [ [ O U 2 O 2 2 PPP2 Q oU O [ 2 9^tt^ 2 [ [ O
U 2 O 2 2 PPP2 Q oU O [ 2 9^tt^ 2 [ [ O
U 2 O 2 2 PPP2 P! ) % f h K�9 h ) % f h K|9 h ) % f h P! 9 h)c 3 �$ � Q w � Q w � P! 3h��sdfosdg� r; r' r% �gen�/TokenizeTest.test_invalid_readline.<locals>.gen; s � � ��O��O�� �c 3 �$ � Q w � Q w � P! 3h)s sdfosdgr; r; r' r% r< r= A s � � � � � � �r> c 3 �, � Q w � ]] * P! 3hr9 r; r; r' r% r< r= G s � � ��O�
�a�C�C�s �)� assertRaises� TypeError�listr �__next__�generate_tokens�ZeroDivisionError)r1 r< s & r% �test_invalid_readline�"TokenizeTest.test_invalid_readline: s� � � � �
�
�y�
)�
)� ��"�"�3�5�>�>�2� 3� *� � �
�
�y�
)�
)� ��)�)�#�%�.�.�9� :� *� � �
�
� 0�
1�
1� ��)�)�#�%�.�.�9� :� 2�
1� *�
)�� *�
)�� 2�
1�
1�s5 �/D �9/D �/D/� D �D � D, �%D, �/E �8E c �R � [ Q O Q2 2 o[ [ O UO 2 2 oU O UQ* O [ O 2 U O UP * O [ O 2 P! )�xr+ ���) r r, rC r r- r. r r r )r1 r4 �tokenss & r% �test_implicit_newline�"TokenizeTest.test_implicit_newlineM sl � �
�C�J�J�w�'� (���h�'�'��
�
�3�4�� ������ � �(�*:�*:�;� ������ � �(�*<�*<�=r' c � � U O Q Q2 U O QQ2 U O QQ2 U O QQ2 Qo[ U2 O oU O [ Q 2 9^tt^ 2 o[
O
U2 F D oJ PPP2 U O
WO O ]2 U O
UO O Q
2 U O
UO O P2 U O
UO O P2 U O
UO O Q 2 U O
UO O ] 2 U O
UO O Q 2 P! ) % f h CK 9 h) z1 + 1z� NUMBER '1' (1, 0) (1, 1)
OP '+' (1, 2) (1, 3)
NUMBER '1' (1, 4) (1, 5)
z3if False:
# NL
True = False # NEWLINE
af NAME 'if' (1, 0) (1, 2)
NAME 'False' (1, 3) (1, 8)
OP ':' (1, 8) (1, 9)
NEWLINE '\n' (1, 9) (1, 10)
COMMENT '# NL' (2, 4) (2, 8)
NL '\n' (2, 8) (2, 9)
NL '\n' (3, 4) (3, 5)
INDENT ' ' (4, 0) (4, 4)
NAME 'True' (4, 4) (4, 8)
OP '=' (4, 9) (4, 10)
NAME 'False' (4, 11) (4, 16)
COMMENT '# NEWLINE' (4, 17) (4, 26)
NEWLINE '\n' (4, 26) (4, 27)
DEDENT '' (5, 0) (5, 0)
z%if True:
# NL
foo='bar'
a@ NAME 'if' (1, 0) (1, 2)
NAME 'True' (1, 3) (1, 7)
OP ':' (1, 7) (1, 8)
NEWLINE '\r\n' (1, 8) (1, 10)
COMMENT '# NL' (2, 4) (2, 8)
NL '\r\n' (2, 8) (2, 10)
INDENT ' ' (3, 0) (3, 4)
NAME 'foo' (3, 4) (3, 7)
OP '=' (3, 7) (3, 8)
STRING "'bar'" (3, 8) (3, 13)
NEWLINE '\r\n' (3, 13) (3, 15)
NL '\r\n' (4, 0) (4, 2)
DEDENT '' (5, 0) (5, 0)
zx = 1 + \
1
a NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
NUMBER '1' (1, 4) (1, 5)
OP '+' (1, 6) (1, 7)
NUMBER '1' (2, 0) (2, 1)
NEWLINE '\r\n' (2, 1) (2, 3)
s def k(x):
x += 2
x += 5
z3unindent does not match any outer indentation level�<string>z x += 5)r5 r r- �assertRaisesRegex�IndentationErrorr r. � exception�lineno�filename�
end_lineno�
end_offset�msg�offset�text)r1 �indent_error_filer- �e�toks & r% �
test_basic�TokenizeTest.test_basicU s� � � ���G� &� �
��� ;�=� �(
��� K� N� �
��� 1� 4� � ��
�,�-�6�6��
�
#�
#�$4�%>�?� ?�BC� �(�(��2��� 3�?�
������+�+�Q�/� ������-�-�z�:� ������/�/��6� ������/�/��6� ���
�K�K�O�O� A� C�
������+�+�Q�/� ������)�)�:�6�?� ?� ?�s �? F2�2G �;G c �L � U O Q Q2 U O QQ2 U O QQ2 U O QQ2 U O QQ 2 U O Q
Q 2 U O Q Q
2 U O QQ2 U O QQ2 P! )� 0xff <= 255z� NUMBER '0xff' (1, 0) (1, 4)
OP '<=' (1, 5) (1, 7)
NUMBER '255' (1, 8) (1, 11)
� 0b10 <= 255z� NUMBER '0b10' (1, 0) (1, 4)
OP '<=' (1, 5) (1, 7)
NUMBER '255' (1, 8) (1, 11)
�0o123 <= 0O123z� NUMBER '0o123' (1, 0) (1, 5)
OP '<=' (1, 6) (1, 8)
NUMBER '0O123' (1, 9) (1, 14)
�1234567 > ~0x15z� NUMBER '1234567' (1, 0) (1, 7)
OP '>' (1, 8) (1, 9)
OP '~' (1, 10) (1, 11)
NUMBER '0x15' (1, 11) (1, 15)
�2134568 != 1231515z� NUMBER '2134568' (1, 0) (1, 7)
OP '!=' (1, 8) (1, 10)
NUMBER '1231515' (1, 11) (1, 18)
�(-124561-1) & 200000000ac OP '(' (1, 0) (1, 1)
OP '-' (1, 1) (1, 2)
NUMBER '124561' (1, 2) (1, 8)
OP '-' (1, 8) (1, 9)
NUMBER '1' (1, 9) (1, 10)
OP ')' (1, 10) (1, 11)
OP '&' (1, 12) (1, 13)
NUMBER '200000000' (1, 14) (1, 23)
�0xdeadbeef != -1z� NUMBER '0xdeadbeef' (1, 0) (1, 10)
OP '!=' (1, 11) (1, 13)
OP '-' (1, 14) (1, 15)
NUMBER '1' (1, 15) (1, 16)
�0xdeadc0de & 12345z� NUMBER '0xdeadc0de' (1, 0) (1, 10)
OP '&' (1, 11) (1, 12)
NUMBER '12345' (1, 13) (1, 18)
�0xFF & 0x15 | 1234z� NUMBER '0xFF' (1, 0) (1, 4)
OP '&' (1, 5) (1, 6)
NUMBER '0x15' (1, 7) (1, 11)
OP '|' (1, 12) (1, 13)
NUMBER '1234' (1, 14) (1, 18)
�r5 �r1 s &r% �test_int�TokenizeTest.test_int� s� � � ���M� ,� �
���M� ,� �
��� ,� /� �
��� -� 0� �
��� 0� 3� �
��� 5� 8� �
��� .� 1� �
��� 0� 3� �
��� 0� 3� r' c � � U O Q Q2 U O QQ2 U O QQ2 U O QQ2 P! )zx = 0z� NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
NUMBER '0' (1, 4) (1, 5)
zx = 0xfffffffffffz� NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
NUMBER '0xfffffffffff' (1, 4) (1, 17)
zx = 123141242151251616110z� NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
NUMBER '123141242151251616110' (1, 4) (1, 25)
zx = -15921590215012591z� NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
OP '-' (1, 4) (1, 5)
NUMBER '15921590215012591' (1, 5) (1, 22)
rj rk s &r% � test_long�TokenizeTest.test_long� s_ � � ���G� &� �
��� /� 2� �
��� 7� :� �
��� 4� 7� r' c � � U O Q Q2 U O QQ2 U O QQ2 U O QQ2 U O QQ 2 U O Q
Q 2 U O Q Q
2 P! )� x = 3.14159z� NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
NUMBER '3.14159' (1, 4) (1, 11)
� x = 314159.z� NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
NUMBER '314159.' (1, 4) (1, 11)
� x = .314159z� NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
NUMBER '.314159' (1, 4) (1, 11)
� x = 3e14159z� NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
NUMBER '3e14159' (1, 4) (1, 11)
� x = 3E123z� NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
NUMBER '3E123' (1, 4) (1, 9)
�
x+y = 3e-1230z� NAME 'x' (1, 0) (1, 1)
OP '+' (1, 1) (1, 2)
NAME 'y' (1, 2) (1, 3)
OP '=' (1, 4) (1, 5)
NUMBER '3e-1230' (1, 6) (1, 13)
� x = 3.14e159z� NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
NUMBER '3.14e159' (1, 4) (1, 12)
rj rk s &r% �
test_float�TokenizeTest.test_float� s� � � ���M� ,� �
���M� ,� �
���M� ,� �
���M� ,� �
���K� *� �
���O� .� �
���N� -� r' c � � Q o[ F D$ oQU7 c J U O U U2 U2 J& QQ.o[ F D, oV#7 c J U U2 U O U U2 U2 J. P! [ O c JJ h 9 h)c �� � [ U O Q 2 2 o[ O UO 2 F D! v q#qEoU[ O 6X f J Ut ! Q! )r+ z
invalid token)r r, r r- �NUMBER)r2 r4 �toktyper! r" r# r$ s & r% � number_token�;TokenizeTest.test_underscore_literals.<locals>.number_token sL � ������)�*�A�4<�4E�4E�a�j�j�4Q�0���D��h�o�o�-� �L� 5R� #r' �(�0_7�09_99)r r. r
r �
TokenError�assertNotEqual)r1 r �lit�extra_valid_casess & r% �test_underscore_literals�%TokenizeTest.test_underscore_literals s� � � #� -�,�C��c�z�� � � �\�#� .�� 4� -� #�G� ,��.�.�C��'��
� �S�!�
� � � �S� 1�3� 7� /��
�&�&�
��
�s � A1�1B
�B
� B
c � � U O Q Q2 U O QQ2 U O QQ2 U O QQ2 U O QQ 2 U O Q
Q 2 U O Q Q
2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O Q Q 2 U O Q Q 2 U O Q Q!2 U O Q"Q#2 U O Q$Q%2 U O Q&Q'2 U O Q(Q)2 U O Q*Q+2 U O Q,Q-2 U O Q.Q/2 U O Q0Q12 U O Q2Q32 U O Q4Q52 U O Q6Q72 U O Q8Q92 U O Q:Q;2 U O Q<Q=2 U O Q>Q?2 U O Q@QA2 U O QBQC2 U O QDQE2 U O QFQG2 P! )H�x = ''; y = ""a5 NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
STRING "''" (1, 4) (1, 6)
OP ';' (1, 6) (1, 7)
NAME 'y' (1, 8) (1, 9)
OP '=' (1, 10) (1, 11)
STRING '""' (1, 12) (1, 14)
�x = '"'; y = "'"a6 NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
STRING '\'"\'' (1, 4) (1, 7)
OP ';' (1, 7) (1, 8)
NAME 'y' (1, 9) (1, 10)
OP '=' (1, 11) (1, 12)
STRING '"\'"' (1, 13) (1, 16)
� x = "doesn't "shrink", does it"z� NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
STRING '"doesn\'t "' (1, 4) (1, 14)
NAME 'shrink' (1, 14) (1, 20)
STRING '", does it"' (1, 20) (1, 31)
�x = 'abc' + 'ABC'z� NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
STRING "'abc'" (1, 4) (1, 9)
OP '+' (1, 10) (1, 11)
STRING "'ABC'" (1, 12) (1, 17)
�y = "ABC" + "ABC"z� NAME 'y' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
STRING '"ABC"' (1, 4) (1, 9)
OP '+' (1, 10) (1, 11)
STRING '"ABC"' (1, 12) (1, 17)
�%x = r'abc' + r'ABC' + R'ABC' + R'ABC'a� NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
STRING "r'abc'" (1, 4) (1, 10)
OP '+' (1, 11) (1, 12)
STRING "r'ABC'" (1, 13) (1, 19)
OP '+' (1, 20) (1, 21)
STRING "R'ABC'" (1, 22) (1, 28)
OP '+' (1, 29) (1, 30)
STRING "R'ABC'" (1, 31) (1, 37)
�%y = r"abc" + r"ABC" + R"ABC" + R"ABC"a� NAME 'y' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
STRING 'r"abc"' (1, 4) (1, 10)
OP '+' (1, 11) (1, 12)
STRING 'r"ABC"' (1, 13) (1, 19)
OP '+' (1, 20) (1, 21)
STRING 'R"ABC"' (1, 22) (1, 28)
OP '+' (1, 29) (1, 30)
STRING 'R"ABC"' (1, 31) (1, 37)
�u'abc' + U'abc'z� STRING "u'abc'" (1, 0) (1, 6)
OP '+' (1, 7) (1, 8)
STRING "U'abc'" (1, 9) (1, 15)
�u"abc" + U"abc"z� STRING 'u"abc"' (1, 0) (1, 6)
OP '+' (1, 7) (1, 8)
STRING 'U"abc"' (1, 9) (1, 15)
�b'abc' + B'abc'z� STRING "b'abc'" (1, 0) (1, 6)
OP '+' (1, 7) (1, 8)
STRING "B'abc'" (1, 9) (1, 15)
�b"abc" + B"abc"z� STRING 'b"abc"' (1, 0) (1, 6)
OP '+' (1, 7) (1, 8)
STRING 'B"abc"' (1, 9) (1, 15)
�%br'abc' + bR'abc' + Br'abc' + BR'abc'a; STRING "br'abc'" (1, 0) (1, 7)
OP '+' (1, 8) (1, 9)
STRING "bR'abc'" (1, 10) (1, 17)
OP '+' (1, 18) (1, 19)
STRING "Br'abc'" (1, 20) (1, 27)
OP '+' (1, 28) (1, 29)
STRING "BR'abc'" (1, 30) (1, 37)
�%br"abc" + bR"abc" + Br"abc" + BR"abc"a; STRING 'br"abc"' (1, 0) (1, 7)
OP '+' (1, 8) (1, 9)
STRING 'bR"abc"' (1, 10) (1, 17)
OP '+' (1, 18) (1, 19)
STRING 'Br"abc"' (1, 20) (1, 27)
OP '+' (1, 28) (1, 29)
STRING 'BR"abc"' (1, 30) (1, 37)
�%rb'abc' + rB'abc' + Rb'abc' + RB'abc'a; STRING "rb'abc'" (1, 0) (1, 7)
OP '+' (1, 8) (1, 9)
STRING "rB'abc'" (1, 10) (1, 17)
OP '+' (1, 18) (1, 19)
STRING "Rb'abc'" (1, 20) (1, 27)
OP '+' (1, 28) (1, 29)
STRING "RB'abc'" (1, 30) (1, 37)
�%rb"abc" + rB"abc" + Rb"abc" + RB"abc"a; STRING 'rb"abc"' (1, 0) (1, 7)
OP '+' (1, 8) (1, 9)
STRING 'rB"abc"' (1, 10) (1, 17)
OP '+' (1, 18) (1, 19)
STRING 'Rb"abc"' (1, 20) (1, 27)
OP '+' (1, 28) (1, 29)
STRING 'RB"abc"' (1, 30) (1, 37)
� "a\
de\
fg"�3 STRING '"a\\\nde\\\nfg"' (1, 0) (3, 3)
�u"a\
de"�/ STRING 'u"a\\\nde"' (1, 0) (2, 3)
�rb"a\
d"�/ STRING 'rb"a\\\nd"' (1, 0) (2, 2)
�
"""a\
b"""�0 STRING '"""a\\\nb"""' (1, 0) (2, 4)
� u"""a\
b"""�1 STRING 'u"""a\\\nb"""' (1, 0) (2, 4)
�rb"""a\
b\
c"""�7 STRING 'rb"""a\\\nb\\\nc"""' (1, 0) (3, 4)
�f"abc"�� FSTRING_START 'f"' (1, 0) (1, 2)
FSTRING_MIDDLE 'abc' (1, 2) (1, 5)
FSTRING_END '"' (1, 5) (1, 6)
� fR"a{b}c"a= FSTRING_START 'fR"' (1, 0) (1, 3)
FSTRING_MIDDLE 'a' (1, 3) (1, 4)
OP '{' (1, 4) (1, 5)
NAME 'b' (1, 5) (1, 6)
OP '}' (1, 6) (1, 7)
FSTRING_MIDDLE 'c' (1, 7) (1, 8)
FSTRING_END '"' (1, 8) (1, 9)
zfR"a{{{b!r}}}c"a� FSTRING_START 'fR"' (1, 0) (1, 3)
FSTRING_MIDDLE 'a{' (1, 3) (1, 5)
OP '{' (1, 6) (1, 7)
NAME 'b' (1, 7) (1, 8)
OP '!' (1, 8) (1, 9)
NAME 'r' (1, 9) (1, 10)
OP '}' (1, 10) (1, 11)
FSTRING_MIDDLE '}' (1, 11) (1, 12)
FSTRING_MIDDLE 'c' (1, 13) (1, 14)
FSTRING_END '"' (1, 14) (1, 15)
z f"{{{1+1}}}"a� FSTRING_START 'f"' (1, 0) (1, 2)
FSTRING_MIDDLE '{' (1, 2) (1, 3)
OP '{' (1, 4) (1, 5)
NUMBER '1' (1, 5) (1, 6)
OP '+' (1, 6) (1, 7)
NUMBER '1' (1, 7) (1, 8)
OP '}' (1, 8) (1, 9)
FSTRING_MIDDLE '}' (1, 9) (1, 10)
FSTRING_END '"' (1, 11) (1, 12)
z f"""{f'''{f'{f"{1+1}"}'}'''}"""ad FSTRING_START 'f"""' (1, 0) (1, 4)
OP '{' (1, 4) (1, 5)
FSTRING_START "f'''" (1, 5) (1, 9)
OP '{' (1, 9) (1, 10)
FSTRING_START "f'" (1, 10) (1, 12)
OP '{' (1, 12) (1, 13)
FSTRING_START 'f"' (1, 13) (1, 15)
OP '{' (1, 15) (1, 16)
NUMBER '1' (1, 16) (1, 17)
OP '+' (1, 17) (1, 18)
NUMBER '1' (1, 18) (1, 19)
OP '}' (1, 19) (1, 20)
FSTRING_END '"' (1, 20) (1, 21)
OP '}' (1, 21) (1, 22)
FSTRING_END "'" (1, 22) (1, 23)
OP '}' (1, 23) (1, 24)
FSTRING_END "'''" (1, 24) (1, 27)
OP '}' (1, 27) (1, 28)
FSTRING_END '"""' (1, 28) (1, 31)
z.f""" x
str(data, encoding={invalid!r})
"""a� FSTRING_START 'f"""' (1, 0) (1, 4)
FSTRING_MIDDLE ' x\nstr(data, encoding=' (1, 4) (2, 19)
OP '{' (2, 19) (2, 20)
NAME 'invalid' (2, 20) (2, 27)
OP '!' (2, 27) (2, 28)
NAME 'r' (2, 28) (2, 29)
OP '}' (2, 29) (2, 30)
FSTRING_MIDDLE ')\n' (2, 30) (3, 0)
FSTRING_END '"""' (3, 0) (3, 3)
z#f"""123456789
something{None}bad"""aO FSTRING_START 'f"""' (1, 0) (1, 4)
FSTRING_MIDDLE '123456789\nsomething' (1, 4) (2, 9)
OP '{' (2, 9) (2, 10)
NAME 'None' (2, 10) (2, 14)
OP '}' (2, 14) (2, 15)
FSTRING_MIDDLE 'bad' (2, 15) (2, 18)
FSTRING_END '"""' (2, 18) (2, 21)
�
f"""abc"""�� FSTRING_START 'f"""' (1, 0) (1, 4)
FSTRING_MIDDLE 'abc' (1, 4) (1, 7)
FSTRING_END '"""' (1, 7) (1, 10)
� f"abc\
def"�� FSTRING_START 'f"' (1, 0) (1, 2)
FSTRING_MIDDLE 'abc\\\ndef' (1, 2) (2, 3)
FSTRING_END '"' (2, 3) (2, 4)
� Rf"abc\
def"�� FSTRING_START 'Rf"' (1, 0) (1, 3)
FSTRING_MIDDLE 'abc\\\ndef' (1, 3) (2, 3)
FSTRING_END '"' (2, 3) (2, 4)
z5f'some words {a+b:.3f} more words {c+d=} final words'aA FSTRING_START "f'" (1, 0) (1, 2)
FSTRING_MIDDLE 'some words ' (1, 2) (1, 13)
OP '{' (1, 13) (1, 14)
NAME 'a' (1, 14) (1, 15)
OP '+' (1, 15) (1, 16)
NAME 'b' (1, 16) (1, 17)
OP ':' (1, 17) (1, 18)
FSTRING_MIDDLE '.3f' (1, 18) (1, 21)
OP '}' (1, 21) (1, 22)
FSTRING_MIDDLE ' more words ' (1, 22) (1, 34)
OP '{' (1, 34) (1, 35)
NAME 'c' (1, 35) (1, 36)
OP '+' (1, 36) (1, 37)
NAME 'd' (1, 37) (1, 38)
OP '=' (1, 38) (1, 39)
OP '}' (1, 39) (1, 40)
FSTRING_MIDDLE ' final words' (1, 40) (1, 52)
FSTRING_END "'" (1, 52) (1, 53)
z
f'''{
3
=}'''a` FSTRING_START "f'''" (1, 0) (1, 4)
OP '{' (1, 4) (1, 5)
NL '\n' (1, 5) (1, 6)
NUMBER '3' (2, 0) (2, 1)
NL '\n' (2, 1) (2, 2)
OP '=' (3, 0) (3, 1)
OP '}' (3, 1) (3, 2)
FSTRING_END "'''" (3, 2) (3, 5)
�f'''__{
x:a
}__'''a� FSTRING_START "f'''" (1, 0) (1, 4)
FSTRING_MIDDLE '__' (1, 4) (1, 6)
OP '{' (1, 6) (1, 7)
NL '\n' (1, 7) (1, 8)
NAME 'x' (2, 4) (2, 5)
OP ':' (2, 5) (2, 6)
FSTRING_MIDDLE 'a\n' (2, 6) (3, 0)
OP '}' (3, 0) (3, 1)
FSTRING_MIDDLE '__' (3, 1) (3, 3)
FSTRING_END "'''" (3, 3) (3, 6)
�+f'''__{
x:a
b
c
d
}__'''a� FSTRING_START "f'''" (1, 0) (1, 4)
FSTRING_MIDDLE '__' (1, 4) (1, 6)
OP '{' (1, 6) (1, 7)
NL '\n' (1, 7) (1, 8)
NAME 'x' (2, 4) (2, 5)
OP ':' (2, 5) (2, 6)
FSTRING_MIDDLE 'a\n b\n c\n d\n' (2, 6) (6, 0)
OP '}' (6, 0) (6, 1)
FSTRING_MIDDLE '__' (6, 1) (6, 3)
FSTRING_END "'''" (6, 3) (6, 6)
uz '''Autorzy, którzy tą jednostkę mają wpisani jako AKTUALNA -- czyli
aktualni pracownicy, obecni pracownicy'''
u INDENT ' ' (1, 0) (1, 4)
STRING "'''Autorzy, którzy tą jednostkę mają wpisani jako AKTUALNA -- czyli\n aktualni pracownicy, obecni pracownicy'''" (1, 4) (2, 45)
NEWLINE '\n' (2, 45) (2, 46)
DEDENT '' (3, 0) (3, 0)
rj rk s &r% � test_string�TokenizeTest.test_string. s � � ��� .� 1� �
��� 1� 4� �
��� A� D� �
��� /� 2� �
��� /� 2� �
��� C�
F�
�
��� C�
F�
�
��� -� 0� �
��� -� 0� �
��� -� 0� �
��� -� 0� �
��� C� F� �
��� C� F� �
��� C� F� �
��� C� F� �
��� �� �
��� �� �
��� �� �
��� �� �
��� �� �
��� �� �
���H� '� �
���K� *� �
��� -� 0� �
���N�
-�
�
��� E� H� �*
��� N�
Q�
�
��� B� E� �
���L� +� �
��� �� �
��� �� �
��� S� V� �(
��� � � �
���
� � �
���
� �
�&
��� �� r' c �t � U O Q Q2 U O QQ2 U O QQ2 P! )�!def d22(a, b, c=2, d=2, *k): passa NAME 'def' (1, 0) (1, 3)
NAME 'd22' (1, 4) (1, 7)
OP '(' (1, 7) (1, 8)
NAME 'a' (1, 8) (1, 9)
OP ',' (1, 9) (1, 10)
NAME 'b' (1, 11) (1, 12)
OP ',' (1, 12) (1, 13)
NAME 'c' (1, 14) (1, 15)
OP '=' (1, 15) (1, 16)
NUMBER '2' (1, 16) (1, 17)
OP ',' (1, 17) (1, 18)
NAME 'd' (1, 19) (1, 20)
OP '=' (1, 20) (1, 21)
NUMBER '2' (1, 21) (1, 22)
OP ',' (1, 22) (1, 23)
OP '*' (1, 24) (1, 25)
NAME 'k' (1, 25) (1, 26)
OP ')' (1, 26) (1, 27)
OP ':' (1, 27) (1, 28)
NAME 'pass' (1, 29) (1, 33)
� def d01v_(a=1, *k, **w): passa� NAME 'def' (1, 0) (1, 3)
NAME 'd01v_' (1, 4) (1, 9)
OP '(' (1, 9) (1, 10)
NAME 'a' (1, 10) (1, 11)
OP '=' (1, 11) (1, 12)
NUMBER '1' (1, 12) (1, 13)
OP ',' (1, 13) (1, 14)
OP '*' (1, 15) (1, 16)
NAME 'k' (1, 16) (1, 17)
OP ',' (1, 17) (1, 18)
OP '**' (1, 19) (1, 21)
NAME 'w' (1, 21) (1, 22)
OP ')' (1, 22) (1, 23)
OP ':' (1, 23) (1, 24)
NAME 'pass' (1, 25) (1, 29)
�&def d23(a: str, b: int=3) -> int: passa� NAME 'def' (1, 0) (1, 3)
NAME 'd23' (1, 4) (1, 7)
OP '(' (1, 7) (1, 8)
NAME 'a' (1, 8) (1, 9)
OP ':' (1, 9) (1, 10)
NAME 'str' (1, 11) (1, 14)
OP ',' (1, 14) (1, 15)
NAME 'b' (1, 16) (1, 17)
OP ':' (1, 17) (1, 18)
NAME 'int' (1, 19) (1, 22)
OP '=' (1, 22) (1, 23)
NUMBER '3' (1, 23) (1, 24)
OP ')' (1, 24) (1, 25)
OP '->' (1, 26) (1, 28)
NAME 'int' (1, 29) (1, 32)
OP ':' (1, 32) (1, 33)
NAME 'pass' (1, 34) (1, 38)
rj rk s &r% �
test_function�TokenizeTest.test_functionn sJ � � ��� ?� B� �,
��� ;� >� �"
��� D� G� r' c �, � U O Q Q2 P! )�Xif 1 < 1 > 1 == 1 >= 5 <= 0x15 <= 0x12 != 1 and 5 in 1 not in 1 is 1 or 5 is not 1: passa� NAME 'if' (1, 0) (1, 2)
NUMBER '1' (1, 3) (1, 4)
OP '<' (1, 5) (1, 6)
NUMBER '1' (1, 7) (1, 8)
OP '>' (1, 9) (1, 10)
NUMBER '1' (1, 11) (1, 12)
OP '==' (1, 13) (1, 15)
NUMBER '1' (1, 16) (1, 17)
OP '>=' (1, 18) (1, 20)
NUMBER '5' (1, 21) (1, 22)
OP '<=' (1, 23) (1, 25)
NUMBER '0x15' (1, 26) (1, 30)
OP '<=' (1, 31) (1, 33)
NUMBER '0x12' (1, 34) (1, 38)
OP '!=' (1, 39) (1, 41)
NUMBER '1' (1, 42) (1, 43)
NAME 'and' (1, 44) (1, 47)
NUMBER '5' (1, 48) (1, 49)
NAME 'in' (1, 50) (1, 52)
NUMBER '1' (1, 53) (1, 54)
NAME 'not' (1, 55) (1, 58)
NAME 'in' (1, 59) (1, 61)
NUMBER '1' (1, 62) (1, 63)
NAME 'is' (1, 64) (1, 66)
NUMBER '1' (1, 67) (1, 68)
NAME 'or' (1, 69) (1, 71)
NUMBER '5' (1, 72) (1, 73)
NAME 'is' (1, 74) (1, 76)
NAME 'not' (1, 77) (1, 80)
NUMBER '1' (1, 81) (1, 82)
OP ':' (1, 82) (1, 83)
NAME 'pass' (1, 84) (1, 88)
rj rk s &r% �test_comparison� TokenizeTest.test_comparison� � � � ��� M�!O�" r' c �, � U O Q Q2 P! )zx = 1 << 1 >> 5a6 NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
NUMBER '1' (1, 4) (1, 5)
OP '<<' (1, 6) (1, 8)
NUMBER '1' (1, 9) (1, 10)
OP '>>' (1, 11) (1, 13)
NUMBER '5' (1, 14) (1, 15)
rj rk s &r% �
test_shift�TokenizeTest.test_shift� s � � ��� -� 0� r' c �, � U O Q Q2 P! )�%x = 1 - y + 15 - 1 + 0x124 + z + a[5]a$ NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
NUMBER '1' (1, 4) (1, 5)
OP '-' (1, 6) (1, 7)
NAME 'y' (1, 8) (1, 9)
OP '+' (1, 10) (1, 11)
NUMBER '15' (1, 12) (1, 14)
OP '-' (1, 15) (1, 16)
NUMBER '1' (1, 17) (1, 18)
OP '+' (1, 19) (1, 20)
NUMBER '0x124' (1, 21) (1, 26)
OP '+' (1, 27) (1, 28)
NAME 'z' (1, 29) (1, 30)
OP '+' (1, 31) (1, 32)
NAME 'a' (1, 33) (1, 34)
OP '[' (1, 34) (1, 35)
NUMBER '5' (1, 35) (1, 36)
OP ']' (1, 36) (1, 37)
rj rk s &r% �
test_additive�TokenizeTest.test_additive� � � � ��� C� F� r' c �, � U O Q Q2 P! )�x = 1//1*1/5*12%0x12@42a� NAME 'x' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
NUMBER '1' (1, 4) (1, 5)
OP '//' (1, 5) (1, 7)
NUMBER '1' (1, 7) (1, 8)
OP '*' (1, 8) (1, 9)
NUMBER '1' (1, 9) (1, 10)
OP '/' (1, 10) (1, 11)
NUMBER '5' (1, 11) (1, 12)
OP '*' (1, 12) (1, 13)
NUMBER '12' (1, 13) (1, 15)
OP '%' (1, 15) (1, 16)
NUMBER '0x12' (1, 16) (1, 20)
OP '@' (1, 20) (1, 21)
NUMBER '42' (1, 21) (1, 23)
rj rk s &r% �test_multiplicative� TokenizeTest.test_multiplicative� � � � ��� 5� 8� r' c �P � U O Q Q2 U O QQ2 P! )�~1 ^ 1 & 1 |1 ^ -1a� OP '~' (1, 0) (1, 1)
NUMBER '1' (1, 1) (1, 2)
OP '^' (1, 3) (1, 4)
NUMBER '1' (1, 5) (1, 6)
OP '&' (1, 7) (1, 8)
NUMBER '1' (1, 9) (1, 10)
OP '|' (1, 11) (1, 12)
NUMBER '1' (1, 12) (1, 13)
OP '^' (1, 14) (1, 15)
OP '-' (1, 16) (1, 17)
NUMBER '1' (1, 17) (1, 18)
�-1*1/1+1*1//1 - ---1**1aH OP '-' (1, 0) (1, 1)
NUMBER '1' (1, 1) (1, 2)
OP '*' (1, 2) (1, 3)
NUMBER '1' (1, 3) (1, 4)
OP '/' (1, 4) (1, 5)
NUMBER '1' (1, 5) (1, 6)
OP '+' (1, 6) (1, 7)
NUMBER '1' (1, 7) (1, 8)
OP '*' (1, 8) (1, 9)
NUMBER '1' (1, 9) (1, 10)
OP '//' (1, 10) (1, 12)
NUMBER '1' (1, 12) (1, 13)
OP '-' (1, 14) (1, 15)
OP '-' (1, 16) (1, 17)
OP '-' (1, 17) (1, 18)
OP '-' (1, 18) (1, 19)
NUMBER '1' (1, 19) (1, 20)
OP '**' (1, 20) (1, 22)
NUMBER '1' (1, 22) (1, 23)
rj rk s &r% �
test_unary�TokenizeTest.test_unary s0 � � ��� 0� 3� �
��� 5� 8� r' c �, � U O Q Q2 P! )�/import sys, time
x = sys.modules['time'].time()a� NAME 'import' (1, 0) (1, 6)
NAME 'sys' (1, 7) (1, 10)
OP ',' (1, 10) (1, 11)
NAME 'time' (1, 12) (1, 16)
NEWLINE '\n' (1, 16) (1, 17)
NAME 'x' (2, 0) (2, 1)
OP '=' (2, 2) (2, 3)
NAME 'sys' (2, 4) (2, 7)
OP '.' (2, 7) (2, 8)
NAME 'modules' (2, 8) (2, 15)
OP '[' (2, 15) (2, 16)
STRING "'time'" (2, 16) (2, 22)
OP ']' (2, 22) (2, 23)
OP '.' (2, 23) (2, 24)
NAME 'time' (2, 24) (2, 28)
OP '(' (2, 28) (2, 29)
OP ')' (2, 29) (2, 30)
rj rk s &r% �
test_selector�TokenizeTest.test_selector, � � � ��� N� Q� r' c �, � U O Q Q2 P! )� @staticmethod
def foo(x,y): passa OP '@' (1, 0) (1, 1)
NAME 'staticmethod' (1, 1) (1, 13)
NEWLINE '\n' (1, 13) (1, 14)
NAME 'def' (2, 0) (2, 3)
NAME 'foo' (2, 4) (2, 7)
OP '(' (2, 7) (2, 8)
NAME 'x' (2, 8) (2, 9)
OP ',' (2, 9) (2, 10)
NAME 'y' (2, 10) (2, 11)
OP ')' (2, 11) (2, 12)
OP ':' (2, 12) (2, 13)
NAME 'pass' (2, 14) (2, 18)
rj rk s &r% � test_method�TokenizeTest.test_methodB � � � ��� ?�
B�
r' c �, � U O Q Q2 P! )z def f():
if x
passa_ NAME 'def' (1, 0) (1, 3)
NAME 'f' (1, 4) (1, 5)
OP '(' (1, 5) (1, 6)
OP ')' (1, 6) (1, 7)
OP ':' (1, 7) (1, 8)
NEWLINE '\n' (1, 8) (1, 9)
INDENT '\t' (2, 0) (2, 1)
NAME 'if' (2, 1) (2, 3)
NAME 'x' (2, 4) (2, 5)
NEWLINE '\n' (2, 5) (2, 6)
INDENT ' \t' (3, 0) (3, 9)
NAME 'pass' (3, 9) (3, 13)
DEDENT '' (4, 0) (4, 0)
DEDENT '' (4, 0) (4, 0)
rj rk s &r% � test_tabs�TokenizeTest.test_tabsS s � � ��� -�/� r' c �, � U O Q Q2 P! )u! Örter = 'places'
grün = 'green'u7 NAME 'Örter' (1, 0) (1, 5)
OP '=' (1, 6) (1, 7)
STRING "'places'" (1, 8) (1, 16)
NEWLINE '\n' (1, 16) (1, 17)
NAME 'grün' (2, 0) (2, 4)
OP '=' (2, 5) (2, 6)
STRING "'green'" (2, 7) (2, 14)
rj rk s &r% �test_non_ascii_identifiers�'TokenizeTest.test_non_ascii_identifiersh s � � ��� @� C� r' c �, � U O Q Q2 P! )�# Örter = u'places'
grün = U'green'u7 NAME 'Örter' (1, 0) (1, 5)
OP '=' (1, 6) (1, 7)
STRING "u'places'" (1, 8) (1, 17)
NEWLINE '\n' (1, 17) (1, 18)
NAME 'grün' (2, 0) (2, 4)
OP '=' (2, 5) (2, 6)
STRING "U'green'" (2, 7) (2, 15)
rj rk s &r% � test_unicode�TokenizeTest.test_unicodet � � � ��� B� E� r' c � � U O Q Q2 U O QQ2 U O QQ2 U O QQ2 U O QQ 2 U O Q
Q 2 U O Q Q
2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O Q Q 2 U O Q Q 2 U O Q Q!2 U O Q"Q#2 U O Q$Q%2 U O Q&Q'2 U O Q(Q)2 P! )*� async = 1z� NAME 'async' (1, 0) (1, 5)
OP '=' (1, 6) (1, 7)
NUMBER '1' (1, 8) (1, 9)
�a = (async = 1)a8 NAME 'a' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
OP '(' (1, 4) (1, 5)
NAME 'async' (1, 5) (1, 10)
OP '=' (1, 11) (1, 12)
NUMBER '1' (1, 13) (1, 14)
OP ')' (1, 14) (1, 15)
�async()z� NAME 'async' (1, 0) (1, 5)
OP '(' (1, 5) (1, 6)
OP ')' (1, 6) (1, 7)
�class async(Bar):passa< NAME 'class' (1, 0) (1, 5)
NAME 'async' (1, 6) (1, 11)
OP '(' (1, 11) (1, 12)
NAME 'Bar' (1, 12) (1, 15)
OP ')' (1, 15) (1, 16)
OP ':' (1, 16) (1, 17)
NAME 'pass' (1, 17) (1, 21)
�class async:passz� NAME 'class' (1, 0) (1, 5)
NAME 'async' (1, 6) (1, 11)
OP ':' (1, 11) (1, 12)
NAME 'pass' (1, 12) (1, 16)
� await = 1z� NAME 'await' (1, 0) (1, 5)
OP '=' (1, 6) (1, 7)
NUMBER '1' (1, 8) (1, 9)
� foo.asyncz� NAME 'foo' (1, 0) (1, 3)
OP '.' (1, 3) (1, 4)
NAME 'async' (1, 4) (1, 9)
�async for a in b: passa; NAME 'async' (1, 0) (1, 5)
NAME 'for' (1, 6) (1, 9)
NAME 'a' (1, 10) (1, 11)
NAME 'in' (1, 12) (1, 14)
NAME 'b' (1, 15) (1, 16)
OP ':' (1, 16) (1, 17)
NAME 'pass' (1, 18) (1, 22)
�async with a as b: passa< NAME 'async' (1, 0) (1, 5)
NAME 'with' (1, 6) (1, 10)
NAME 'a' (1, 11) (1, 12)
NAME 'as' (1, 13) (1, 15)
NAME 'b' (1, 16) (1, 17)
OP ':' (1, 17) (1, 18)
NAME 'pass' (1, 19) (1, 23)
� async.fooz� NAME 'async' (1, 0) (1, 5)
OP '.' (1, 5) (1, 6)
NAME 'foo' (1, 6) (1, 9)
�async�/ NAME 'async' (1, 0) (1, 5)
�async
#comment
awaitz� NAME 'async' (1, 0) (1, 5)
NEWLINE '\n' (1, 5) (1, 6)
COMMENT '#comment' (2, 0) (2, 8)
NL '\n' (2, 8) (2, 9)
NAME 'await' (3, 0) (3, 5)
�async
...
awaitz� NAME 'async' (1, 0) (1, 5)
NEWLINE '\n' (1, 5) (1, 6)
OP '...' (2, 0) (2, 3)
NEWLINE '\n' (2, 3) (2, 4)
NAME 'await' (3, 0) (3, 5)
� async
awaitz� NAME 'async' (1, 0) (1, 5)
NEWLINE '\n' (1, 5) (1, 6)
NAME 'await' (2, 0) (2, 5)
�
foo.async + 1z� NAME 'foo' (1, 0) (1, 3)
OP '.' (1, 3) (1, 4)
NAME 'async' (1, 4) (1, 9)
OP '+' (1, 10) (1, 11)
NUMBER '1' (1, 12) (1, 13)
�async def foo(): passa; NAME 'async' (1, 0) (1, 5)
NAME 'def' (1, 6) (1, 9)
NAME 'foo' (1, 10) (1, 13)
OP '(' (1, 13) (1, 14)
OP ')' (1, 14) (1, 15)
OP ':' (1, 15) (1, 16)
NAME 'pass' (1, 17) (1, 21)
�Nasync def foo():
def foo(await):
await = 1
if 1:
await
async += 1
a� NAME 'async' (1, 0) (1, 5)
NAME 'def' (1, 6) (1, 9)
NAME 'foo' (1, 10) (1, 13)
OP '(' (1, 13) (1, 14)
OP ')' (1, 14) (1, 15)
OP ':' (1, 15) (1, 16)
NEWLINE '\n' (1, 16) (1, 17)
INDENT ' ' (2, 0) (2, 2)
NAME 'def' (2, 2) (2, 5)
NAME 'foo' (2, 6) (2, 9)
OP '(' (2, 9) (2, 10)
NAME 'await' (2, 10) (2, 15)
OP ')' (2, 15) (2, 16)
OP ':' (2, 16) (2, 17)
NEWLINE '\n' (2, 17) (2, 18)
INDENT ' ' (3, 0) (3, 4)
NAME 'await' (3, 4) (3, 9)
OP '=' (3, 10) (3, 11)
NUMBER '1' (3, 12) (3, 13)
NEWLINE '\n' (3, 13) (3, 14)
DEDENT '' (4, 2) (4, 2)
NAME 'if' (4, 2) (4, 4)
NUMBER '1' (4, 5) (4, 6)
OP ':' (4, 6) (4, 7)
NEWLINE '\n' (4, 7) (4, 8)
INDENT ' ' (5, 0) (5, 4)
NAME 'await' (5, 4) (5, 9)
NEWLINE '\n' (5, 9) (5, 10)
DEDENT '' (6, 0) (6, 0)
DEDENT '' (6, 0) (6, 0)
NAME 'async' (6, 0) (6, 5)
OP '+=' (6, 6) (6, 8)
NUMBER '1' (6, 9) (6, 10)
NEWLINE '\n' (6, 10) (6, 11)
�)async def foo():
async for i in 1: passa� NAME 'async' (1, 0) (1, 5)
NAME 'def' (1, 6) (1, 9)
NAME 'foo' (1, 10) (1, 13)
OP '(' (1, 13) (1, 14)
OP ')' (1, 14) (1, 15)
OP ':' (1, 15) (1, 16)
NEWLINE '\n' (1, 16) (1, 17)
INDENT ' ' (2, 0) (2, 2)
NAME 'async' (2, 2) (2, 7)
NAME 'for' (2, 8) (2, 11)
NAME 'i' (2, 12) (2, 13)
NAME 'in' (2, 14) (2, 16)
NUMBER '1' (2, 17) (2, 18)
OP ':' (2, 18) (2, 19)
NAME 'pass' (2, 20) (2, 24)
DEDENT '' (3, 0) (3, 0)
�async def foo(async): awaitah NAME 'async' (1, 0) (1, 5)
NAME 'def' (1, 6) (1, 9)
NAME 'foo' (1, 10) (1, 13)
OP '(' (1, 13) (1, 14)
NAME 'async' (1, 14) (1, 19)
OP ')' (1, 19) (1, 20)
OP ':' (1, 20) (1, 21)
NAME 'await' (1, 22) (1, 27)
�@def f():
def baz(): pass
async def bar(): pass
await = 2a� NAME 'def' (1, 0) (1, 3)
NAME 'f' (1, 4) (1, 5)
OP '(' (1, 5) (1, 6)
OP ')' (1, 6) (1, 7)
OP ':' (1, 7) (1, 8)
NEWLINE '\n' (1, 8) (1, 9)
NL '\n' (2, 0) (2, 1)
INDENT ' ' (3, 0) (3, 2)
NAME 'def' (3, 2) (3, 5)
NAME 'baz' (3, 6) (3, 9)
OP '(' (3, 9) (3, 10)
OP ')' (3, 10) (3, 11)
OP ':' (3, 11) (3, 12)
NAME 'pass' (3, 13) (3, 17)
NEWLINE '\n' (3, 17) (3, 18)
NAME 'async' (4, 2) (4, 7)
NAME 'def' (4, 8) (4, 11)
NAME 'bar' (4, 12) (4, 15)
OP '(' (4, 15) (4, 16)
OP ')' (4, 16) (4, 17)
OP ':' (4, 17) (4, 18)
NAME 'pass' (4, 19) (4, 23)
NEWLINE '\n' (4, 23) (4, 24)
NL '\n' (5, 0) (5, 1)
NAME 'await' (6, 2) (6, 7)
OP '=' (6, 8) (6, 9)
NUMBER '2' (6, 10) (6, 11)
DEDENT '' (7, 0) (7, 0)
�Fasync def f():
def baz(): pass
async def bar(): pass
await = 2a NAME 'async' (1, 0) (1, 5)
NAME 'def' (1, 6) (1, 9)
NAME 'f' (1, 10) (1, 11)
OP '(' (1, 11) (1, 12)
OP ')' (1, 12) (1, 13)
OP ':' (1, 13) (1, 14)
NEWLINE '\n' (1, 14) (1, 15)
NL '\n' (2, 0) (2, 1)
INDENT ' ' (3, 0) (3, 2)
NAME 'def' (3, 2) (3, 5)
NAME 'baz' (3, 6) (3, 9)
OP '(' (3, 9) (3, 10)
OP ')' (3, 10) (3, 11)
OP ':' (3, 11) (3, 12)
NAME 'pass' (3, 13) (3, 17)
NEWLINE '\n' (3, 17) (3, 18)
NAME 'async' (4, 2) (4, 7)
NAME 'def' (4, 8) (4, 11)
NAME 'bar' (4, 12) (4, 15)
OP '(' (4, 15) (4, 16)
OP ')' (4, 16) (4, 17)
OP ':' (4, 17) (4, 18)
NAME 'pass' (4, 19) (4, 23)
NEWLINE '\n' (4, 23) (4, 24)
NL '\n' (5, 0) (5, 1)
NAME 'await' (6, 2) (6, 7)
OP '=' (6, 8) (6, 9)
NUMBER '2' (6, 10) (6, 11)
DEDENT '' (7, 0) (7, 0)
rj rk s &r% �
test_async�TokenizeTest.test_async� s� � � ���K� *� �
��� -� 0� �
���I� (� �
��� 3� 6� �
��� .� 1� �
���K� *� �
���K� *� �
��� 4� 7� �
��� 5� 8� �
���K� *� �
���G� &� �
��� 4� 7� �
��� /� 2� �
���N� -� �
���O� .� �
��� 3� 6� �
��� �#�* �X
��� � � �*
��� =� @� �
��� � �
# �J
��� � �
$ r' c �, � U O Q Q2 P! )z [
# A comment here
1
]
ad OP '[' (1, 0) (1, 1)
NL '\n' (1, 1) (1, 2)
COMMENT '# A comment here' (2, 4) (2, 20)
NL '\n' (2, 20) (2, 21)
NUMBER '1' (3, 4) (3, 5)
NL '\n' (3, 5) (3, 6)
OP ']' (4, 0) (4, 1)
NEWLINE '\n' (4, 1) (4, 2)
rj rk s &r% �3test_newline_after_parenthesized_block_with_comment�@TokenizeTest.test_newline_after_parenthesized_block_with_comment� s � � ��� �
� r' c �, � U O Q Q2 P! )z); xz� OP ')' (1, 0) (1, 1)
OP ';' (1, 1) (1, 2)
NAME 'x' (1, 3) (1, 4)
rj rk s &r% �,test_closing_parenthesis_from_different_line�9TokenizeTest.test_closing_parenthesis_from_different_line� s � � ���F� %� r' c �, � U O Q Q2 P! )uT a = f'''
Autorzy, którzy tą jednostkę mają wpisani jako AKTUALNA -- czyli'''u% NAME 'a' (1, 0) (1, 1)
OP '=' (1, 2) (1, 3)
FSTRING_START "f'''" (1, 4) (1, 8)
FSTRING_MIDDLE '\n Autorzy, którzy tą jednostkę mają wpisani jako AKTUALNA -- czyli' (1, 8) (2, 68)
FSTRING_END "'''" (2, 68) (2, 71)
rj rk s &r% � test_multiline_non_ascii_fstring�-TokenizeTest.test_multiline_non_ascii_fstring� s � � ��� O�Q� r' c �P � U O Q Q2 U O QQ2 P! )u4 f'''
🔗 This is a test {test_arg1}🔗
🔗'''uZ FSTRING_START "f'''" (1, 0) (1, 4)
FSTRING_MIDDLE '\n 🔗 This is a test ' (1, 4) (2, 21)
OP '{' (2, 21) (2, 22)
NAME 'test_arg1' (2, 22) (2, 31)
OP '}' (2, 31) (2, 32)
FSTRING_MIDDLE '🔗\n🔗' (2, 32) (3, 1)
FSTRING_END "'''" (3, 1) (3, 4)
u f"{f(a=lambda: 'à'
)}"u? FSTRING_START 'f"' (1, 0) (1, 2)
OP '{' (1, 2) (1, 3)
NAME 'f' (1, 3) (1, 4)
OP '(' (1, 4) (1, 5)
NAME 'a' (1, 5) (1, 6)
OP '=' (1, 6) (1, 7)
NAME 'lambda' (1, 7) (1, 13)
OP ':' (1, 13) (1, 14)
STRING "'à'" (1, 15) (1, 18)
NL '\n' (1, 18) (1, 19)
OP ')' (2, 0) (2, 1)
OP '}' (2, 1) (2, 2)
FSTRING_END '"' (2, 2) (2, 3)
rj rk s &r% �*test_multiline_non_ascii_fstring_with_expr�7TokenizeTest.test_multiline_non_ascii_fstring_with_expr� s2 � � ��� �
� �
��� :� =� r' r; ) �__name__�
__module__� __qualname__�__firstlineno__r5 rG rM r^ rl ro ry r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r r r �__static_attributes__�__classdictcell__��
__classdict__s @r% r) r) * s� �� � �9�;�&>�F7�P7 �r �0& �P8�0~ �@
: �x$ �L
� �. �(# �J �, �" �*
�
�P �d �" � � � r' r) c �( ` � \ sQ sQs n Q sQsU sP! )�GenerateTokensTesti� c �� � [ U2 o[ [ O UO 2 U2 oU O VBO
2 O 2 2 P! �N)r r& r rE r- r. r/ r r0 s &&& r% r5 �!GenerateTokensTest.check_tokenize� sG � �
�Q�K��-�h�.F�.F�q�z�z�.R�TU�V�� ���� � �!2�!=�!=�!?�@r' r; )r
r r r
r5 r r r s @r% r r � s �� � �A� Ar' r c
� � , o[ O [ U O Q 2 2 O 2 oSF D� v q4 oU[ O 6X ch QU7 ca UO [ O Q1[ O Q1[ O [ U2 1[ O Q1,2 J� UO V412 J� [ O U2 O Q 2 O 2 ! )r+ �.�Decimalr� �))r r r, r- r} �extend�NAME�OP�STRING�reprr �
untokenize�decode�strip)r2 r �g�toknum�tokval�_s & r% �decistmtr' � s� � �
�F����'�!�(�(�7�"3�4�=�=�>�A�$%� ���1�a� �X�_�_� $��� � �M�M�� � � �*����c�"�� � �$�v�,�/����c�"� �
�
�M�M�6�*� +� %&� � � �v� &� -� -�g� 6� <� <� >�>r' c �. ` � \ sQ sQs n Q sQ sQsU sP! )�TestMisci� c �� � ] QHGo QoU O [ U2 Q2 U O [ [
U2 2 Q2 U O [
[ U2 2 U Q2 2 P! )r )r z+21.3e-5*-.1234/81.7z8+Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7')z-3.2171603427[0-9]*e-0+7z!-3.217160342717258261933904529E-7)�decimalr r. r' � assertRegexr �eval)r1 r r2 s & r% �
test_decistmt�TestMisc.test_decistmt� sd � � $� "�� ����!��S� U�
����d�1�g� �(B�C�
����h�q�k�*� �!D�E� Gr' c �z � [ O , QN* oU O [ O U2 P! )� TokenInfo)r1 r� rE �detect_encodingr �openr )r! �__all__�assertCountEqualr )r1 r3 s & r% � test___all__�TestMisc.test___all__ s- � ��=�=� $
�
��
� � �h� .� .��9r' r; )r
r r r
r. r6 r r r s @r% r) r) � s �� � �G�(:� :r' r) c �J ` � \ sQ sQs n QsQ sQ sQ sQ sQ s Q s
Q s U s P! )
� TestTokenizerAdheresToPep0263i
zM
Test that tokenizer adheres to the coding behaviour stipulated in PEP 0263.
c � � [ O O [ O O [ 2 Q U2 o[ UQ2 9^tt^ 2 o[ O V2 PPP2 P! ) % f h P! 9 h)� tokenizedata�rb)�os�path�join�dirname�__file__r3 �
TestRoundtrip�check_roundtrip)r1 rU r> r4 s && r% � _testFile�'TestTokenizerAdheresToPep0263._testFile sQ � ��w�w�|�|�B�G�G�O�O�H� 5�~�x�P��
�$��
�
�� � )� )�$� 2� �
�
�
�s �A8�8B �B c �. � Q oU O U2 P! )z9tokenize_tests-utf8-coding-cookie-and-no-utf8-bom-sig.txt�rD �r1 r4 s & r% �'test_utf8_coding_cookie_and_no_utf8_bom�ETestTokenizerAdheresToPep0263.test_utf8_coding_cookie_and_no_utf8_bom � � � G�� ���q�r' c �N � QoU O [ U O U2 P! )aV
As per PEP 0263, if a file starts with a utf-8 BOM signature, the only
allowed encoding for the comment is 'utf-8'. The text file used in
this test starts with a BOM signature, but specifies latin1 as the
coding, so verify that a SyntaxError is raised, which matches the
behaviour of the interpreter when it encounters a similar condition.
z8tokenize_tests-latin1-coding-cookie-and-utf8-bom-sig.txt�rA � SyntaxErrorrD rH s & r% �&test_latin1_coding_cookie_and_utf8_bom�DTestTokenizerAdheresToPep0263.test_latin1_coding_cookie_and_utf8_bom s � �
G�� ���+�t�~�~�q�9r' c �. � Q oU O U2 P! )z9tokenize_tests-no-coding-cookie-and-utf8-bom-sig-only.txtrG rH s & r% �"test_no_coding_cookie_and_utf8_bom�@TestTokenizerAdheresToPep0263.test_no_coding_cookie_and_utf8_bom# rK r' c �. � Q oU O U2 P! )z6tokenize_tests-utf8-coding-cookie-and-utf8-bom-sig.txtrG rH s & r% �$test_utf8_coding_cookie_and_utf8_bom�BTestTokenizerAdheresToPep0263.test_utf8_coding_cookie_and_utf8_bom' s � � D�� ���q�r' c � � U O [ U O Q 2 U O [ U O Q2 P! )z
bad_coding.pyzbad_coding2.pyrM rk s &r% �test_bad_coding_cookie�4TestTokenizerAdheresToPep0263.test_bad_coding_cookie+ s/ � � ���+�t�~�~��G� ���+�t�~�~�7G�Hr' r; )
r
r r r
�__doc__rD rI rO rR rU rX r r r s @r% r9 r9
s/ �� � ��3�
� :���I� Ir' r9 c �( ` � \ sQ sQs n Q sQsU sP! )�
Test_Tokenizei0 c � ``� Q oUO Q2 nP
nUU1Q ko[ [ O U 2 O QP Q5 2 PQ o[ O
]Q QQQ 2 ,oU O
V4Q2 P! )u "ЉЊЈЁЂ"r+ c 3 �> : � R % f
P n Rw � P! Qw � P! 3h)Tr' r; )�firstr$ s ��r% r- �NTest_Tokenize.test__tokenize_decodes_with_specified_encoding.<locals>.readline6 s �� � �� ���
�� �s � � )�encoding� extra_tokensz bytes not decoded with encodingrK �r r �r � )r, rC r �!_generate_tokens_from_c_tokenizerrD r1 r. )r1 �literalr- rL �expected_tokensr_ r$ s & @@r% �.test__tokenize_decodes_with_specified_encoding�<Test_Tokenize.test__tokenize_decodes_with_specified_encoding2 s� �� � ���~�~�g�&���� � �h�@�@���AT�AT�JQ�NR�T� U�UX�VX�Z�� $�-�-�a� ���Q_�`�a�� ����:� <r' r; )r
r r r
ri r r r s @r% r\ r\ 0 s �� � �<� <r' r\ c � ` � \ sQ sQs n Q sQ sQ sQ sQ sQ s Q s
Q s Q
s Q s
Q sQ
sQ sQ sQ sQ sQ sQ sQ sQ sQ sQ sQ sQ sQ s Q s Q s Q s Q s Q s!Q s"Q! s#\$OJ Q" 2 s&Q# s'Q$ s(Q%s)U s*P! )&�TestDetectEncodingiG c � ``� ] nUU1Q koU! )r c �Z :� R[ R2 6X c [ gRR* o R]*
nU ! �r )r �
StopIteration)r$ �index�liness ��r% r- �1TestDetectEncoding.get_readline.<locals>.readlineK s, �� ���E�
�"�#�#���<�D� �Q�J�E��Kr' r; )r1 rr r- rq s &f @r% � get_readline� TestDetectEncoding.get_readlineI s �� ��� � �r' c �� � Qo[ O U O U2 2 v q#U O UQ2 U O U[ UQ* 2 2 P! )� #!/home/ä/bin/python
r+ �N� N)rw s # something €
� print(something)
� do_something(else)
�r r2 rt r. rC �r1 rr ra �consumed_liness & r% � test_no_bom_no_encoding_cookie�1TestDetectEncoding.test_no_bom_no_encoding_cookieT sQ � �
�� $,�#;�#;�D�<M�<M�e�<T�#U� �� ����7�+� ��� ��e�B�i� �9r' c �� � QoU O [ 2 9^tt^ 2 [ O U O U2 2 PPP2 P! ) % f h P! 9 h)� #!/home/�/bin/python
)r� rz r{ �rA rN r r2 rt �r1 rr s & r% �/test_no_bom_no_encoding_cookie_first_line_error�BTestDetectEncoding.test_no_bom_no_encoding_cookie_first_line_error_ �D � �
��
�
�
�{�
+�
+� � $� $�T�%6�%6�u�%=� >� ,�
+�
+�
+�� �&A�A" �A" c �� � QoU O [ 2 9^tt^ 2 [ O U O U2 2 PPP2 P! ) % f h P! 9 h)� #!/usr/bin/python
)r� � # something �
rz r{ r� r� s & r% �0test_no_bom_no_encoding_cookie_second_line_error�CTestDetectEncoding.test_no_bom_no_encoding_cookie_second_line_errorh �D � �
�� �
�
�{�
+�
+� � $� $�T�%6�%6�u�%=� >� ,�
+�
+�
+�r� c � � Qo[ O U O U2 2 v q#U O UQ2 U O UQQ,2 P! )� #!/home/ä/bin/python
rz � utf-8-sigrw )r� rz r{ �r r2 rt r. r} s & r% �test_bom_no_cookie�%TestDetectEncoding.test_bom_no_cookier sS � �
��
$,�#;�#;�D�<M�<M�e�<T�#U� �� ����;�/� ��� �:�<Q�R� Tr' c �� � QoU O [ 2 9^tt^ 2 [ O U O U2 2 PPP2 P! ) % f h P! 9 h)� #!/home/�/bin/python
)r� rz r{ r� r� s & r% �#test_bom_no_cookie_first_line_error�6TestDetectEncoding.test_bom_no_cookie_first_line_error} r� r� c �� � QoU O [ 2 9^tt^ 2 [ O U O U2 2 PPP2 P! ) % f h P! 9 h)� #!/usr/bin/python
)r� r� rz r{ r� r� s & r% �$test_bom_no_cookie_second_line_error�7TestDetectEncoding.test_bom_no_cookie_second_line_error� r� r� c � � Qo[ O U O U2 2 v q#U O UQ2 U O UQ ,2 P! )� # -*- coding: latin-1 -*-
�
iso-8859-1)r� rz r{ r� r} s & r% � test_cookie_first_line_no_bom�0TestDetectEncoding.test_cookie_first_line_no_bom� sM � �
��
$,�#;�#;�D�<M�<M�e�<T�#U� �� ����<�0� ��� �*H�)I�Jr' c � � Qo[ O U O U2 2 v q#U O UQ2 U O UQ,2 P! )� # coding=utf-8
r� s # coding=utf-8
)r� rz r{ r� r} s & r% �&test_matched_bom_and_cookie_first_line�9TestDetectEncoding.test_matched_bom_and_cookie_first_line� sM � �
��
$,�#;�#;�D�<M�<M�e�<T�#U� �� ����;�/� ��� �*=�)>�?r' c �x � QoU O U2 oU O [ [ O U2 P! )�# # vim: set fileencoding=ascii :
)r� rz r{ �rt rA rN r r2 �r1 rr r- s & r% �<test_mismatched_bom_and_cookie_first_line_raises_syntaxerror�OTestDetectEncoding.test_mismatched_bom_and_cookie_first_line_raises_syntaxerror� s3 � �
��
�$�$�U�+�� ���+�x�'?�'?��Jr' c � � Qo[ O U O U2 2 v q#U O UQ2 Q Q,oU O V42 P! )�
#! something
� # vim: set fileencoding=ascii :
�ascii)r� r� rz r{ r� �r1 rr ra r~ r3 s & r% � test_cookie_second_line_no_bom�1TestDetectEncoding.test_cookie_second_line_no_bom� sR � �
�� $,�#;�#;�D�<M�<M�e�<T�#U� �� ����7�+�%�'K�L�� ��� �2r' c � � Qo[ O U O U2 2 v q#U O UQ2 U O UQQ,2 P! )� #! something
� f# coding=utf-8
r� r� )r� r� rz r{ r� r} s & r% �'test_matched_bom_and_cookie_second_line�:TestDetectEncoding.test_matched_bom_and_cookie_second_line� sS � �
�� $,�#;�#;�D�<M�<M�e�<T�#U� �� ����;�/� ��� �+�-A�B� Dr' c �x � QoU O U2 oU O [ [ O U2 P! )r� )r� r� rz r{ r� r� s & r% �=test_mismatched_bom_and_cookie_second_line_raises_syntaxerror�PTestDetectEncoding.test_mismatched_bom_and_cookie_second_line_raises_syntaxerror� s3 � �
�� �$�$�U�+�� ���+�x�'?�'?��Jr' c � � Qo[ O U O U2 2 v q#U O UQ2 Q ,oU O V42 P! )� print('£')
r+ )r� �% # vim: set fileencoding=iso8859-15 :
�
print('€')
r� r� s & r% �/test_cookie_second_line_noncommented_first_line�BTestDetectEncoding.test_cookie_second_line_noncommented_first_line� sO � �
��
$,�#;�#;�D�<M�<M�e�<T�#U� �� ����7�+�*�+�� ��� �2r' c �� � Qo[ O U O U2 2 v q#U O UQ2 U O U[ UQ* 2 2 P! )� #coding:iso-8859-15 �
� iso-8859-15�Nr N)r� rz r| r} s & r% � test_first_non_utf8_coding_line�2TestDetectEncoding.test_first_non_utf8_coding_line� sQ � �
�� $,�#;�#;�D�<M�<M�e�<T�#U� �� ����=�1� ��� ��e�B�i� �9r' c �� � QoU O [ 2 9^tt^ 2 [ O U O U2 2 PPP2 P! ) % f h P! 9 h)� #coding:ascii ä
)r� rz r� r� s & r% �!test_first_utf8_coding_line_error�4TestDetectEncoding.test_first_utf8_coding_line_error� sD � �
�� �
�
�{�
+�
+� � $� $�T�%6�%6�u�%=� >� ,�
+�
+�
+�r� c �� � Qo[ O U O U2 2 v q#U O UQ2 U O U[ UQ* 2 2 P! )r� r� rx )r� r� rz r| r} s & r% � test_second_non_utf8_coding_line�3TestDetectEncoding.test_second_non_utf8_coding_line� �Q � �
��
$,�#;�#;�D�<M�<M�e�<T�#U� �� ����=�1� ��� ��e�B�i� �9r' c �� � QoU O [ 2 9^tt^ 2 [ O U O U2 2 PPP2 P! ) % f h P! 9 h)r� )r� r� rz r� r� s & r% �"test_second_utf8_coding_line_error�5TestDetectEncoding.test_second_utf8_coding_line_error� r� r� c �� � Qo[ O U O U2 2 v q#U O UQ2 U O U[ UQ* 2 2 P! )� #!/home/�/bin/python
r� rx )r� s #coding:iso-8859-15
rz r| r} s & r% �test_non_utf8_shebang�(TestDetectEncoding.test_non_utf8_shebang� r� r' c �� � QoU O [ 2 9^tt^ 2 [ O U O U2 2 PPP2 P! ) % f h P! 9 h)rw )rw s #coding:ascii
rz r� r� s & r% �test_utf8_shebang_error�*TestDetectEncoding.test_utf8_shebang_error r� r� c � � Qo[ O U O U2 2 v q#U O UQ2 Q Q,oU O V42 P! )�
r� �
iso8859-15)r� r� r� r� r� s & r% �(test_cookie_second_line_empty_first_line�;TestDetectEncoding.test_cookie_second_line_empty_first_line sR � �
��
$,�#;�#;�D�<M�<M�e�<T�#U� �� ����<�0��D�E�� ��� �2r' c �� � Qo[ O U O U2 2 v q#U O UQ2 U O U[ UQ* 2 2 P! )rw r+ rx )rw s # something
r� rz r{ r| r} s & r% �test_cookie_third_line�)TestDetectEncoding.test_cookie_third_line sQ � �
�� $,�#;�#;�D�<M�<M�e�<T�#U� �� ����7�+� ��� ��e�B�i� �9r' c �� � Qo[ O U O U2 2 v q#U O UQ2 U O U[ UQ* 2 2 P! )� #coding:iso8859-15
r� r� )r� � #coding:latin1
rz r| r} s & r% �test_double_coding_line�*TestDetectEncoding.test_double_coding_line( sQ � �
��
$,�#;�#;�D�<M�<M�e�<T�#U� �� ����<�0� ��� ��e�B�i� �9r' c �� � Qo[ O U O U2 2 v q#U O UQ2 U O U[ UQ* 2 2 P! )�! #coding:iso8859-15 coding:latin1
r� r� )r� rz r| r} s & r% � test_double_coding_same_line�/TestDetectEncoding.test_double_coding_same_line3 sQ � �
�� $,�#;�#;�D�<M�<M�e�<T�#U� �� ����<�0� ��� ��e�B�i� �9r' c �� � Qo[ O U O U2 2 v q#U O UQ2 U O U[ UQ* 2 2 P! )� #coding:utf-8
r+ r� )r� r� rz r| r} s & r% �test_double_coding_utf8�*TestDetectEncoding.test_double_coding_utf8< sQ � �
��
$,�#;�#;�D�<M�<M�e�<T�#U� �� ����7�+� ��� ��e�B�i� �9r' c �� � QoU O [ Q2 9^tt^ 2 [ O U O U2 2 PPP2 P! ) % f h P! 9 h)� #coding:iso8859-15
�%source code cannot contain null bytes)r� r� r� rz �rQ rN r r2 rt r� s & r% � test_nul_in_first_coding_line�0TestDetectEncoding.test_nul_in_first_coding_lineF �Q � �
�� �
#�
#�K�7�9� 9� � $� $�T�%6�%6�u�%=� >�9� 9� 9� 9�� �&A�A# �A# c �� � QoU O [ Q2 9^tt^ 2 [ O U O U2 2 PPP2 P! ) % f h P! 9 h)r� r� )r� r� r� rz r� r� s & r% � test_nul_in_second_coding_line�1TestDetectEncoding.test_nul_in_second_coding_lineQ r� r� c � � Q oSF D} oQ
F Ds oUO QU2 oQQUO Q2 * Q* QQ1oU O U2 o[ O U2 v qxU O UQ2 Ju J P! ) �latin-1r� �-r� �
# coding: r� r� s print(things)
s do_something += 4
)r� r� z iso-latin-1z latin-1-unixziso-8859-1-unixziso-latin-1-mac�r� r& ��replacer, rt r r2 r. � r1 � encodingsra �rep�encrr �rl�foundr~ s & r% �test_latin1_normalization�,TestDetectEncoding.test_latin1_normalization\ s� � �;� �!�H�!�� �&�&�s�C�0��/�&����G�)<�<�u�D�+�/�1�� �&�&�u�-��(0�(@�(@��(D�%��� � �� �5� "� "r' c �x � QoU O U2 oU O [ [ O U2 P! )�
print("�")�r r� r� s & r% �test_syntaxerror_latin1�*TestDetectEncoding.test_syntaxerror_latin1k s3 � ��� �$�$�U�+�� ���+�x�'?�'?��Jr' c � � QoSF D| oQF Dr oUO QU2 oQQUO Q2 * Q* Q1oU O U2 o[ O U2 v qxU O UQ 2 Jt J~ P! ) r+ r� r� r� r� r� s 1 + 3
)r+ z utf-8-macz
utf-8-unixr� r� r s & r% �test_utf8_normalization�*TestDetectEncoding.test_utf8_normalizationt s� � �8� �!�H�!�� �&�&�s�C�0��/�&����G�)<�<�u�D�#�%�� �&�&�u�-��(0�(@�(@��(D�%��� � ���0� "� "r' c �� � U O Q2 o[ O U2 v q#U O UQ2 U O UQ ,2 [ O U O Q2 2 v q#U O UQ2 U O U, 2 U O Q2 o[ O U2 v q#U O UQ2 U O UQ ,2 U O Q2 o[ O U2 v q#U O UQ2 U O U, 2 U O Q2 oU O [
[ O U2 P! )rz r+ r� )rz r; )s print(something)
)� )s # coding: bad
)rt r r2 r. rA rN �r1 r- ra r~ s & r% �test_short_files�#TestDetectEncoding.test_short_files� s= � ��$�$�%=�>��#+�#;�#;�H�#E� �� ����7�+� ��� �*?�)@�A�#+�#;�#;�D�<M�<M�b�<Q�#R� �� ����7�+� ��� ��,��$�$�%I�J��#+�#;�#;�H�#E� �� ����;�/� ��� �*?�)@�A��$�$�%7�8��#+�#;�#;�H�#E� �� ����;�/� ��� ��,��$�$�%:�;�� ���+�x�'?�'?��Jr' c � � U O Q2 o[ O U2 v q#U O UQ2 U O UQ ,2 P! )� print("#coding=fake")r+ )r )rt r r2 r. r s & r% �test_false_encoding�&TestDetectEncoding.test_false_encoding� sJ � ��$�$�%@�A��#+�#;�#;�H�#E� �� ����7�+� ��� �*B�)C�Dr' c �~ � [ O Q * oU O [ O U2 QF D� o[ UQUQ5 9^tt^ 2 o[ QU* UQ5 [ QUQ5 PPP2 [ O U2 9^tt^ 2 oU O UO U2 U O UO Q2 PPP2 J� [ UQQQ5 9^tt^ 2 o[ QUQ5 PPP2 [ O U2 9^tt^ 2 oU O UO Q2 U O UO Q2 PPP2 P! ) % f h CK9 h ) % f h CJY 9 h ) % f h K�9 h ) % f h P! 9 h) z.py�w�ra z # coding: %s��fileu print('euro:€')�rr� )r� r+ )
r �TESTFN�
addCleanup�unlinkr3 �printr r. ra �mode)r1 rU ra �fps & r% � test_open� TestDetectEncoding.test_open� s4 � � �#�#�e�+�� ��� �(�(�(�3� 1�H��h��h�7�7�2��n�x�/�b�9��,�2�6� 8� � � �x�(�(�B�� � ����h�7�� � ����#�.� )�(� 1� �(�C�+�
6�
6�"� �(�r� 2� 7�
�]�]�8�
$�
$�� � � �R�[�[�+� 6� � � �R�W�W�c� *� %�
$� 8�7�7�� )�(�(��
7�
6��
$�
$�
$�sH �"E/ � 9F �8F�,9F+�/F
�8F
�F
� F
�F( �!F( �+F< �4F< c �� `� Q oQn U1Q kQ2 oU O [ 2 9^tt^ 2 U RU2 oU;[ O UO
2 PPP2 U O
[ QO U2 2 9^tt^ 2 U RU2 o[ O UO
2 PPP2 P! ) % f h Kq9 h ) % f h P! 9 h)�some_file_pathc �6 :` � \ sQ sQs n Q sU1Q ksQsU sP! )�;TestDetectEncoding.test_filename_in_exception.<locals>.Bunki� c �. � V m Vm ] U m P! )r )�name�_lines�_index)r1 rr r> s &&&r% �__init__�DTestDetectEncoding.test_filename_in_exception.<locals>.Bunk.__init__� s � � � � #� � �� r' c � :� U O [ R2 6X c [ gRU O * oU 9O ]*
tm U! ro )r. r rp )r1 r$ rr s & �r% r- �DTestDetectEncoding.test_filename_in_exception.<locals>.Bunk.readline� s: �� ��;�;�#�e�*�,�'�'� �T�[�[�)��� � �q� � �� r' )r. r- r, )r
r r r
r/ r- r r )r rr s @�r% �Bunkr* � s �� � �
�
�
r' r3 z.*{}r
)rA rN r, r r2 r- rQ �format)r1 r> r3 �insrr s & @r% �test_filename_in_exception�-TestDetectEncoding.test_filename_in_exception� s� �� � ���� � � �
�
�{�
+�
+��u�d�#�C��� � $� $�S�\�\� 2� ,�
�
#�
#�K�� � �t�1D�
E�
E��u�d�#�C� � $� $�S�\�\� 2� F�
E� ,�
+��
F�
E�
E�s# �,C�*C�C �C �C* �"C* c � � [ Q 2 o[ O QUQ5 9^tt^ 2 U O [ [
O Q2 PPP2 U O UO 2 P! ) % f h K-9 h)s #coding:xxxztokenize._builtin_open)� return_value�foobar) r r �patchrA rN r r3 �
assertTrue�closed)r1 �ms & r% �test_open_error�"TestDetectEncoding.test_open_error� sT � � �N� #��
�Z�Z�0�q�
A�
A� � � �k�8�=�=�(� C� B� ������!� B�
A�s �&A8�8B �B r; )+r
r r r
rt r r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r r r r r r �
thread_unsafer% r6 r? r r r s @r% rl rl G s� �� � � � :�?�?� T�?�?�K�@�K�
3�
D�K� 3�:�?�:�?�:�?� 3�
:� :�:�:� ?� ?�
6� K� 1�K�.E�
���+� �+�(3�8"� "r' rl c �R ` � \ sQ sQs n Q sQ sQ sQ sQ sQ s Q s
Q s Q
s U s
P! ) � TestTokenizei� c �b `` `
� ] PHoQn Pn
U 1Q koU
1Q ko] nU1Q koUO oUO oV!m V1m [ O U2 oU O [ U2 Q* , QN2 VQm Vam U O R
R 2 P! XQm Xam h 9 h)r r+ c � :� RQ Q,1! )� first� secondr; )r- ra s &�r% �mock_detect_encoding�8TestTokenize.test_tokenize.<locals>.mock_detect_encoding� s �� ��h� � 2�2� 2r' c � :� Un, o U 2 oU% c UO U2 J$ U! [ c St ! h 9 h)T)rp r )r- ra �kwargs�out� next_line�
encoding_useds &&, �r% �mock__tokenize�2TestTokenize.test_tokenize.<locals>.mock__tokenize� sK �� � $�M��C�� � (�
�I� ��J�J�y�)� ��
�� %� � �J� �s �- � =�=�=c �\ :� R ]*
n R ]6X c Q! [ R 2 O 2 ! )r r' )�strr, )�counters �r% �
mock_readline�1TestTokenize.test_tokenize.<locals>.mock_readline� s) �� � �q�L�G��!�|���w�<�&�&�(� (r' :r NN)rF rG � 1� 2� 3� 4)r r2 rf r. rC ) r1 �tokenize_modulerH rO rT �orig_detect_encoding� orig_c_token�resultsrS ra rN s & @@@r% �
test_tokenize�TestTokenize.test_tokenize� s� �� �*��� �
� 3� � �� )� /� >� >� �&�H�H� �*>�'�<J�9� M� �'�'�
�6�G� � � �T�'�]�2� .� J�
L� /C� +�@L� =� ��� ��1�� /C� +�@L� =�s �:B � B.c � � , o[ Q 2 F D$ oUO QO UQ5 2 J& UO Q2 QO U2 o[ [
O
[
UO Q2 2 O 2 2 oU O UQ* O Q2 P! )�� zdef i{i}(): return {i})�i�OK�
r+ ���) �ranger r4 r? rC r r r, r- r. �string)r1 �bufrb �tokss & r% �test_oneline_defs� TestTokenize.test_oneline_defs s� � ����s��A� �J�J�/�6�6��6�;� <� � �
�
�4���i�i��n�� �H�%�%�g�c�j�j��.A�&B�&K�&K�L�M�� ����b�� � �$�/r' c � � [ [ O [ UO Q 2 2 O 2 2 o[ U2 oU O
[ U2 ]U* 2 U O
[ O U] * O * [ O [ O * 2 [ U2 F D\ oU O
[ O V5]* * O * [ O V%* * 2 J^ U O
[ O U]U* * O * [ O [ O * 2 U O
[ O U]U* * O * [ O [ O * 2 P! �r+ )rC r r r, r- r r. r �
exact_type�ENCODINGrf r! r r )r1 �opstr�optypesrL � num_optypesrb s &&* r% �assertExactTypeEqual�!TestTokenize.assertExactTypeEqual sE � ��h�'�'��� � �W�0E�(F�(O�(O�P�Q���'�l� � ����V��a�+�o�6� ����*�*�6�!�9�+?�+?�@�!�*�*�8�+<�+<�=� ?��{�#�A� � � �X� .� .�v�!�e�}�/G�/G� H� %� .� .�w�z� :�
<� $�
����*�*�6�!�k�/�+B�+M�+M�N�!�*�*�5�=�=�9� ;� ����*�*�6�!�k�/�+B�+M�+M�N�!�*�*�5�?�?�;� =r' c �� � U O Q [ O [ O 2 U O Q[ O [ O
2 U O Q[ O 2 U O Q[ O 2 U O Q[ O 2 U O Q[ O 2 U O Q[ O 2 U O Q[ O 2 U O Q[ O 2 U O Q [ O 2 U O Q
[ O 2 U O Q [ O 2 U O Q [ O 2 U O Q
[ O" 2 U O Q[ O$ 2 U O Q[ O& 2 U O Q[ O( [ O* 2 U O Q[ O, 2 U O Q[ O. 2 U O Q[ O0 2 U O Q[ O2 2 U O Q[ O4 2 U O Q[ O6 2 U O Q[ O8 2 U O Q[ O: 2 U O Q[ O< 2 U O Q[ O> 2 U O Q[ O@ 2 U O Q [ OB 2 U O Q [ OD 2 U O Q [ OF 2 U O Q [ OH 2 U O Q [ OJ 2 U O Q![ OL 2 U O Q![ OL 2 U O Q"[ ON 2 U O Q#[ OP 2 U O Q$[ OR 2 U O Q%[ OT 2 U O Q&[ OV 2 U O Q'[ OX 2 U O Q([ OZ 2 U O Q)[ O\ 2 U O Q*[ O^ 2 U O Q+[ O` 2 U O Q,[b Od [ O< [b Of [ O [b Od [ O< [b Of [ O, [b Od [ O< [b Of 2 U O Q-[ O( [ Of [ O [ Of [ O [ Of [ O* 2 U O Q.[ O6 [ O [ Od [ O [ Of [ O 2 P! )/z()z[]�:�,�;�+r� �*�/�|�&�<�>�=r �%z{}z==z!=z<=z>=�~�^z<<z>>z**z+=z-=z*=z/=z%=z&=z|=z^=z<<=z>>=z**=z//z//=z:=z...z->�@z@=za**2+b**2==c**2z {1, 2, 3}z
^(x & 0x1))4rs r! �LPAR�RPAR�LSQB�RSQB�COLON�COMMA�SEMI�PLUS�MINUS�STAR�SLASH�VBAR�AMPER�LESS�GREATER�EQUAL�DOT�PERCENT�LBRACE�RBRACE�EQEQUAL�NOTEQUAL� LESSEQUAL� GREATEREQUAL�TILDE�
CIRCUMFLEX� LEFTSHIFT�
RIGHTSHIFT�
DOUBLESTAR� PLUSEQUAL�MINEQUAL� STAREQUAL�
SLASHEQUAL� PERCENTEQUAL�
AMPEREQUAL� VBAREQUAL�CIRCUMFLEXEQUAL�LEFTSHIFTEQUAL�RIGHTSHIFTEQUAL�DOUBLESTAREQUAL� DOUBLESLASH�DOUBLESLASHEQUAL�
COLONEQUAL�ELLIPSIS�RARROW�AT�ATEQUALr r r} rk s &r% �test_exact_type� TestTokenize.test_exact_type s� � � �!�!�$��
�
�E�J�J�?� �!�!�$��
�
�E�J�J�?� �!�!�#�u�{�{�3� �!�!�#�u�{�{�3� �!�!�#�u�z�z�2� �!�!�#�u�z�z�2� �!�!�#�u�{�{�3� �!�!�#�u�z�z�2� �!�!�#�u�{�{�3� �!�!�#�u�z�z�2� �!�!�#�u�{�{�3� �!�!�#�u�z�z�2� �!�!�#�u�}�}�5� �!�!�#�u�{�{�3� �!�!�#�u�y�y�1� �!�!�#�u�}�}�5� �!�!�$�� � �e�l�l�C� �!�!�$��
�
�6� �!�!�$����7� �!�!�$����8� �!�!�$��(:�(:�;� �!�!�#�u�{�{�3� �!�!�#�u�'7�'7�8� �!�!�$����8� �!�!�$��(8�(8�9� �!�!�$��(8�(8�9� �!�!�$����8� �!�!�$����7� �!�!�$����8� �!�!�$��(8�(8�9� �!�!�$��(:�(:�;� �!�!�$��(8�(8�9� �!�!�$����8� �!�!�$��(=�(=�>� �!�!�$��(=�(=�>� �!�!�%��)=�)=�>� �!�!�%��)>�)>�?� �!�!�%��)>�)>�?� �!�!�$��(9�(9�:� �!�!�%��)?�)?�@� �!�!�$��(8�(8�9� �!�!�%�� � �8� �!�!�$�� � �5� �!�!�#�u�x�x�0� �!�!�$��
�
�6� �!�!�"3�"*�-�-��1A�1A�8�?�?�"'�*�*�"*�-�-��1A�1A�8�?�?�"'�-�-�"*�-�-��1A�1A�8�?�?� T�
�!�!�+�"'�,�,�"'�,�,�� � �"'�,�,�� � �"'�,�,�"'�,�,� 0�
�!�!�,�"'�"2�"2�"'�*�*�"'�*�*�e�k�k�5�<�<�"'�*�*� .r' c �H � U O Q [ O 2 P! )z @ )rs r! r� rk s &r% �%test_pathological_trailing_whitespace�2TestTokenize.test_pathological_trailing_whitespace` s � � �!�!�-����:r' c � � Q o[ O [ O QQ Q PQ5 [ O [ O QQ
Q QQ5 [ O [ O
QQ Q
QQ5 [ O [ O QQQQQ5 [ O [ O QQQQQ5 [ O [ O QQQQQ5 [ O [ O QQQQQ5 [ O [ O PQQQQ5 [ O [ O PQQPQ5 , o[ [ O [ UO Q2 2 O 2 2 oU O V22 P! )z b = 1
#testr+ �r rg r" r# r$ �bzb = 1
r� �1rd z#test�r r rc �r r �r ry �r � �r � �r � �r � �ry r �ry r �r� r )r� r� )r� r� )r� r )r r1 r! ro r r r} r �NL�COMMENTr rC r r, r- r. �r1 �sourcerh rL s & r% �5test_comment_at_the_end_of_the_source_without_newline�BTestTokenize.test_comment_at_the_end_of_the_source_without_newlined s? � �!�� � � �E�N�N�7�&�V\�ce� f� � � �E�J�J�s�&�f�[d� e� � � �E�H�H�S��F�Yb� c� � � �E�L�L��F�PV�]f� g� � � �E�M�M�$�f�RX�_h� i� � � �E�H�H�T��V�Z^� _� � � �E�M�M�'��U[�bi� j� � � �E�H�H�R�v�6�X_� `� � � �E�O�O�B�f�RX�_a� b�
�� �h�'�'���
�
�g�0F�(G�(P�(P�Q�R�� ����1r' c �D � Q o[ O [ O QQQPQ5 [ O [ O QQQ QQ5 [ O [ O
QQ Q
QQ5 [ O [ O PQ Q QQ5 [ O [ O PQ
Q
PQ5 ,o[ [ O [ UO Q2 2 O 2 2 oU O V22 P! )za
r+ )rg r" r# r$ �aza
rd r r� rc r� r� r� )ry ry r� )
r r1 r! ro r r r� r rC r r, r- r. r� s & r% �?test_newline_and_space_at_the_end_of_the_source_without_newline�LTestTokenize.test_newline_and_space_at_the_end_of_the_source_without_newlinev s� � ��� � � �u�~�~�g�V�QW�^`� a� � � �u�z�z�#�V��V[� \� � � �u�}�}�T��V�Z_� `� � � �u�x�x��&�f�SV� W� � � �u���r��V�Z\� ]�
�� �h�'�'���
�
�g�0F�(G�(P�(P�Q�R�� ����1r' c � � Q o[ O 2 9^tt^ 2 o[ O O UQ2 o[ UQ2 9^tt^ 2 oUO
U2 PPP2 [ U2 v qVU O QUO 2 PPP2 P! ) % f h KC9 h ) % f h P! 9 h)s F"""
�"""z script.py�wbs SyntaxError)
r �temp_dirr= r> r? r3 �writer �assertIn�err)r1 �scriptr� rU r �rsr& s & r% �(test_invalid_character_in_fstring_middle�5TestTokenize.test_invalid_character_in_fstring_middle� s� � ��� �
�
�
!�
!�X��w�w�|�|�H�k�:�H��h��%�%���
�
�6�"� &�(��2�E�B� �M�M�.�"�&�&� 1� "�
!�%�%�� "�
!�
!�s/ �5B6�B# �(1B6�#B3
�,B3
�.B6�6C �?C r; )r
r r r
r^ rj rs r� r� r� r� r� r r r s @r% rC rC � s4 �� � �)2�V 0� =� ?.�B;�2�$ 2�
2�
2r' rC c �4 ` � \ sQ sQs n Q sQ sQ sQsU sP! )�UntokenizeTesti� c �~ � [ O 2 o]Um ]Um U O [
2 9^tt^ 2 oUO
Q2 PPP2 U O WO O ] * Q2 U O [
UO Q2 P! ) % f h K`9 h)ry z'start (1,3) precedes previous end (2,2)r� r� )
r � Untokenizer�prev_row�prev_colrA �
ValueError�add_whitespacer. rS �args)r1 �u�cms & r% �test_bad_input_order�#UntokenizeTest.test_bad_input_order� s� � � � � � "����
���
�
�
�
�z�
*�
*�b�
� � �U� #� +� ���� � �*�*�1�-�9� ;�
���*�a�&6�&6��>� +�
*�s �B,�,B< �5B< c �P � [ O 2 o]Um ]Um , Um UO Q2 U O
UO Q,2 ]Um UO Q2 U O
UO , QN2 [ O U Q2 P! )r �\
za
b
c
\
c
r� )r� r� )r� z\
\
r ) r r� r� r� rL r� r. rB rC )r1 r� s & r% �test_backslash_continuation�*UntokenizeTest.test_backslash_continuation� s� � � � � � "����
���
���� ���� � ������F�8�,���
� ���� � ������#?�@��%�%�d�,H�Ir' c � � [ O 2 o[ O Q 1o[ O Q1U,oUO U[ , 2 2 U O
UO Q,2 [ O 2 oU O
UO [ U,2 2 Q2 [ O 2 oU O
UO [ U2 2 Q2 U O
UO Q2 U O
[ O [ U2 2 Q2 P! )�Hellor+ zHello s Hello )
r r� r ro �compat�iterr. rL r ra )r1 r� r! rL s & r% �test_iter_compat� UntokenizeTest.test_iter_compat� s� � � � � � "��� � ��(���$�$�g�.��6�� �����R��!� ������H�:�.� � � � "�� ���� � �d�E�7�m�4�h�?� � � � "�� ���� � �d�6�l�3�X�>� ������W�-� ����,�,�T�&�\�:�I�Fr' r; ) r
r r r
r� r� r� r r r s @r% r� r� � s �� � �
?� J� G� Gr' r� c �V � [ O Q2 oUO U 2 PI! )a Return `True` if the source contains a backslash on a
line by itself. For example:
a = (1
\
)
Code like this cannot be untokenized exactly. This is because
the tokenizer does not produce any tokens for the line containing
the backslash and so there is no way to know its indent.
s \n\s*\\\r?\n)�re�compile�search)r� �patterns & r% � contains_ambiguous_backslashr� � s' � � �j�j�)�*�G� �>�>�&� !�� -�-r' c �X ` � \ sQ sQs n Q sQ sQ sQ sQ sQ s Q s
Q s Q
s Q s
U sP! ) rB i� c �6 � [ U[ 2 % c UO Q2 oLUO 2 o[ UO P Q5 2 O o[ [ O U2 2 oSTt, tF D pUQ* MJ
oo[ O U2 o[ UO P Q5 2 O o[ O U2 Tt, tF D pUQ* MJ
o oU O V�2 [ O U2 o
[ U
O P Q5 2 O o [ O U 2 Tt, tF D pUQ* MJ
o oU O V�2 [ U2 % ft UO Q2 o
[ U
O P Q5 2 O o[ O [ O U2 2 oU O V�2 P! P! t toh t toh t toh )aR
Test roundtrip for `untokenize`. `f` is an open file or a string.
The source code in f is tokenized to both 5- and 2-tuples.
Both sequences are converted back to source code via
tokenize.untokenize(), and the latter tokenized again to 2-tuples.
The test fails if the 3 pair tokenizations do not match.
If the source code can be untokenized unambiguously, the
untokenized code must match the original code exactly.
When untokenize bugs are fixed, untokenize with 5-tuples should
reproduce code that does not contain a backslash continuation
following spaces. A proper test should test this.
r+ ��keependsrx r )
�
isinstancerR r, �readr� r rD rC r r r. r� � removeprefix)r1 r4 �coder- �tokens5r] �tokens2� bytes_from2� readline2�
tokens2_from2� bytes_from5� readline5�
tokens2_from5�code_without_bom�untokenized_codes && r% rC � TestRoundtrip.check_roundtrip� s� � � �a�� � ��8�8�G�$�D��6�6�8�D�������6�7�@�@���x�(�(��2�3��&-�.�g�s�r�7�7�g��.� �)�)�'�2� ���/�/��/�>�?�H�H� �,4�,=�,=�i�,H�I�,H�S�R���,H�
�I� ��� �0� �)�)�'�2� ���/�/��/�>�?�H�H� �,4�,=�,=�i�,H�I�,H�S�R���,H�
�I� ��� �0�+�D�1�1� #� 0� 0� � A� �� ,� 7� 7�� 7� F�G�P�P�H� '� 2� 2�8�3D�3D�X�3N� O� � � � � -� @� 2�� /�� J��
Js �H �(H� Hc � � [ U[ 2 % c UO Q 2 oLUO 2 o[ UO P Q5 2 O o[ O U2 F D oUO [ O [ O .7 c J4 U O UO UO UO ]* UO ]* 2 J� P! )r+ r� )r rR r, r r� r rD r r ro r r. rg r$ r" r# )r1 r4 r r- r] s && r% �check_line_extraction�#TestRoundtrip.check_line_extraction� s� � � �a�� � ��8�8�G�$�D��6�6�8�D�������6�7�@�@���$�$�X�.�C��x�x�X� .� .��0B�0B� C�C�� � � �S�Z�Z����#�)�)�A�,�����
�)K� L� /r' c �l � U O Q 2 U O Q2 U O Q2 [ O QQQ5 o[ UQ2 9^tt^ 2 oU O U2 PPP2 U O Q2 U O Q2 U O Q 2 U O Q
2 U O Q 2 U O Q 2 U O Q
2 U O Q2 U O Q2 U O Q2 U O Q2 U O Q2 U O Q2 U O Q2 U O Q2 U O Q2 U O Q2 U O Q2 U O Q2 U O Q2 U O Q2 U O Q 2 U O Q 2 U O Q 2 U O Q 2 U O Q 2 U O Q!2 U O Q"2 U O Q#2 U O Q$2 U O Q%2 U O Q&2 U O Q'2 U O Q(2 U O Q)2 , Q-NoSF D oU O U2 J U O Q*2 U O Q+2 U O Q,2 P! ) % f h CK�9 h).zif x == 1:
print(x)
z # This is a comment
# This also
zif x == 1 :
print(x)
ztokenize_tests.txtr; )�subdirr< zoif x == 1:
# A comment by itself.
print(x) # Comment here, too.
# Another comment.
after_if = True
zPif (x # The comments need to go in the right place
== 1):
print('x==1')
z�class Test: # A comment here
# A comment with weird indent
after_com = 5
def x(m): return m*5 # a one liner
def y(m): # A whitespace after the colon
return y*4 # 3-space indent
zttry: import somemodule
except ImportError: # comment
print('Can not import' # comment2
)else: print('Loaded')
zf'\N{EXCLAMATION MARK}'z
f'\\N{SNAKE}'zf'\\N{{SNAKE}}'z f'\N{SNAKE}'zf'\\\N{SNAKE}'zf'\\\\\N{SNAKE}'zf'\\\\\\\N{SNAKE}'z f'\\N{1}'z f'\\\\N{2}'z
f'\\\\\\N{3}'zf'\\\\\\\\N{4}'zf'\\N{{'z
f'\\\\N{{'z f'\\\\\\N{{'zf'\\\\\\\\N{{'z f'\n{{foo}}'z
f'\\n{{foo}}'zf'\\\n{{foo}}'zf'\\\\n{{foo}}'z f'\t{{foo}}'z
f'\\t{{foo}}'zf'\\\t{{foo}}'zf'\\\\t{{foo}}'z
rf'\t{{foo}}'zrf'\\t{{foo}}'zrf'\\\t{{foo}}'zrf'\\\\t{{foo}}'z rf'\{{foo}}'z f'\\{{foo}}'zrf'\\\{{foo}}'zf'\\\\{{foo}}'zt'{ {}}'zt'{f'{ {}}'}{ {}}'zf'{t'{ {}}'}{ {}}')z
if 1:
"foo"
"bar"
z
if 1:
("foo"
"bar")
z
if 1:
"foo"
"bar"
)rC r �findfiler3 )r1 �fnr4 �cases�cases & r% �test_roundtrip� TestRoundtrip.test_roundtrip s� � �
� � � .� /� � � � -� .�
� � � ,� -�
�
�
� 2�>�
J��
�"�d�^�^�q� � � �� #� � � � � 1� 2�
� � � 3� 4�
� � � B� C�
� � � 9� :�
� � � 7�8� � � � -�.� � � � /�0� � � �_�-� � � � .�/� � � � 0�1� � � � 2�3� � � �\�*� � � �^�,� � � � -�.� � � � /�0� � � �[�)� � � �]�+� � � �_�-� � � � .�/� � � �_�-� � � � -�.� � � � .�/� � � � /�0� � � �_�-� � � � -�.� � � � .�/� � � � /�0� � � � -�.� � � � .�/� � � � /�0� � � � 0�1� � � �_�-� � � �_�-� � � � .�/� � � � .�/��� �D� � � �� &� �
� � �[�)� � � � 2�3� � � � 2�3�c �^�^�s �!L"�"L3 �+L3 c �* � U O Q 2 P! )zna = (3,4,
5,6)
y = [3, 4,
5]
z = {'a': 5,
'b':15, 'c':True}
x = len(y) + 5 - a[
3] - a[2]
+ len(z) - z[
'b']
�rC rk s &r% �test_continuation� TestRoundtrip.test_continuationd s � � � � � &� 'r' c �L � U O Q 2 U O Q2 P! )z)x=1+\
1
# This is a comment\
# This also
z# Comment \
x = 0r rk s &r% r� �)TestRoundtrip.test_backslash_continuationq s* � � � � � -� .�
� � � %� &r' c �* � U O Q 2 P! )z'' ''r rk s &r% �test_string_concatenation�'TestRoundtrip.test_string_concatenationz s � � � � �W�%r' c �� � ] PHo] PHo[ O O [
2 9% f [ O oUO [ O O UO U2 Q2 2 o[ O Q2 % f UO U]
2 oSF D� o[ O ]6� c
[ QU2 [ UQ2 9^tt^ 2 oU O UQ5 9^tt^ 2 U O! U2 U O# U2 PPP2 PPP2 J� P! ) % f h K 9 h ) % f h J� 9 h)r ztest*.py�cpur r< r )�glob�randomr= r> r@ rA �curdirr? �escaper �is_resource_enabled�sample�verboser" r3 �subTestrC r )r1 r% r& �tempdir� testfiles�testfiler4 s & r% �test_random_files� TestRoundtrip.test_random_files~ s� � � ��'�'�/�/�(�+�8�8�r�y�y���I�I�b�g�g�l�l�4�;�;�w�+?�� L�M� ��*�*�5�1�1� �
�
�i��4�I�!�H����!�#��j�(�+��h��%�%���\�\�x�\�0�0��(�(��+��.�.�q�1� 1� &�%� "� 1�0�� &�%�%�s0 �,E �#E
�*E � E� E� E �E$
� E$
c �� � [ U[ 2 % c UO Q 2 o[ O [ O [ U2 O 2 2 O Q 2 ! rm )r rR r, r r r r- r! )r1 r s &&r% � roundtrip�TestRoundtrip.roundtrip� sO � � �d�C� � ��;�;�w�'�D��"�"�8�#4�#4�W�T�]�5K�5K�#L�M�T�T�U\�]�]r' c � � QoU O U2 O Q2 oU O U]* U]* 2 U O U2 P! )z~
Ensure that although whitespace might be mutated in a roundtrip,
the semantic meaning of the indentation remains consistent.
zif False:
x=3
x=3
rd )r3 �splitr. rC )r1 r � codeliness & r% �#test_indentation_semantics_retained�1TestRoundtrip.test_indentation_semantics_retained� sH � �
+���N�N�4�(�.�.�t�4� � ����1� �y��|�4� � � �T�"r' r; )r
r r r
rC r r r r� r! r0 r3 r8 r r r s @r% rB rB � s= �� � �'A�R M�a4�H '�&�&�2�(^�
#� #r' rB c �. ` � \ sQ sQs n Q sQ sQsU sP! )�InvalidPythonTestsi� c �� � Q o[ O [ O QQQQ Q5 [ O [ O QQQ Q Q5 [ O [ O
QQ Q
Q Q5 [ O [ O QQ
Q Q Q5 [ O [ O
QQ Q Q Q5 [ O [ O PQ Q
Q Q5 [ O [ O PQQPQ5 ,o[ [ O [ U2 O 2 2 oU O V22 P! )z2sin(x)�2r� �sinr� rJ r rc r� r� r� r� rd )r � r� )
r r1 r! r} r r r r rC rE r r- r. r� s & r% � test_number_followed_by_name�/InvalidPythonTests.test_number_followed_by_name� s� � ��� � � �E�L�L��F�PV�]f� g� � � �E�J�J�u�F�PV�]f� g� � � �E�H�H�S��F�Yb� c� � � �E�J�J�s�&�f�[d� e� � � �E�H�H�S��F�Yb� c� � � �E�M�M�"�F�PV�]f� g� � � �E�O�O�B�f�RX�_a� b�
�� �h�.�.�x��/?�/H�/H�I�J�� ����1r' c
� � Q o[ O [ O Q QQQ Q5 [ O [ O PQQQ Q5 [ O [ O
PQQPQ5 ,o[
[ O [ U2 O 2 2 oU O V22 P! )�01234r� rc r� r� r� ) r r1 r! r} r r rC rE r r- r. r� s & r% � test_number_starting_with_zero�1InvalidPythonTests.test_number_starting_with_zero� s� � ��� � � �E�L�L���TZ�ah� i� � � �E�M�M�"�F�PV�]d� e� � � �E�O�O�B�f�RX�_a� b�
�� �h�.�.�x��/?�/H�/H�I�J�� ����1r' r; )r
r r r
r@ rD r r r s @r% r; r; � s �� � �2� 2� 2r' r; c � ` � \ sQ sQs n Q sQ sQ sQ sQ sQ s Q s
Q s Q
s Q s
Q sQ
sQ sQ sQ sQ s\O* 2 Q 2 sQ sQsU sP! )�
CTokenizeTesti� c �: � [ U2 oU O UQ 5 9^tt^ 2 [ [ O UO
2 U2 oU O
VBO 2 O 2 2 PPP2 P! ) % f h P! 9 h))r� ) r r, r& r rf r- r. r/ r r0 s &&& r% r5 � CTokenizeTest.check_tokenize� sk � �
�Q�K��
�\�\��\�
#�
#�1��:�:�1�:�:�F���F�
� � �V�_�_�%6�%A�%A�%C� D� $�
#�
#�
#�s �AB � B �B c � � Q o[ O [ O QQQQQ5 [ O [ O QQQ QQ5 [ O [ O QQ Q
QQ5 [ O [ O PQ
Q QQ5 [ O [ O
PQ Q PQ5 ,oQ
F Di oU O
UQ5 9^tt^ 2 [ [ O U U2 O P UQ5 2 oU O VB2 PPP2 Jk P! ) % f h J� 9 h)c 3 �8 � Q O U 2 w � P! 3h)�1+1)r, r s &r% r- �-CTokenizeTest.test_encoding.<locals>.readline� s � � ��,�,�x�(� (�s �r� rL r� ry r )rb ra rc r� r� r� r� r� )r+ r� zutf-16) r r1 r} r r r r, rC rf rD r. )r1 r- r3 ra rL s & r% �
test_encoding�CTokenizeTest.test_encoding� s � � )�
� � �H�O�O�C�v�SY�`e� f� � � �H�K�K��6�v�\a� b� � � �H�O�O�C�v�SY�`e� f� � � �H�$4�$4�R�v�SY�`e� f� � � �H�$6�$6�r��U[�bd� e�
�� 7�H�� � �x� �0�0� �h� H� H� �X�&�/�/�!%� %� � ��
� � ��2�
1�0� 7�0�0�0�s �/AE �E
�
E
c �L � U O Q Q2 U O QQ2 U O QQ2 U O QQ2 U O QQ 2 U O Q
Q 2 U O Q Q
2 U O QQ2 U O QQ2 P! )ra z� NUMBER '0xff' (1, 0) (1, 4)
LESSEQUAL '<=' (1, 5) (1, 7)
NUMBER '255' (1, 8) (1, 11)
rb z� NUMBER '0b10' (1, 0) (1, 4)
LESSEQUAL '<=' (1, 5) (1, 7)
NUMBER '255' (1, 8) (1, 11)
rc z� NUMBER '0o123' (1, 0) (1, 5)
LESSEQUAL '<=' (1, 6) (1, 8)
NUMBER '0O123' (1, 9) (1, 14)
rd z� NUMBER '1234567' (1, 0) (1, 7)
GREATER '>' (1, 8) (1, 9)
TILDE '~' (1, 10) (1, 11)
NUMBER '0x15' (1, 11) (1, 15)
re z� NUMBER '2134568' (1, 0) (1, 7)
NOTEQUAL '!=' (1, 8) (1, 10)
NUMBER '1231515' (1, 11) (1, 18)
rf ac LPAR '(' (1, 0) (1, 1)
MINUS '-' (1, 1) (1, 2)
NUMBER '124561' (1, 2) (1, 8)
MINUS '-' (1, 8) (1, 9)
NUMBER '1' (1, 9) (1, 10)
RPAR ')' (1, 10) (1, 11)
AMPER '&' (1, 12) (1, 13)
NUMBER '200000000' (1, 14) (1, 23)
rg z� NUMBER '0xdeadbeef' (1, 0) (1, 10)
NOTEQUAL '!=' (1, 11) (1, 13)
MINUS '-' (1, 14) (1, 15)
NUMBER '1' (1, 15) (1, 16)
rh z� NUMBER '0xdeadc0de' (1, 0) (1, 10)
AMPER '&' (1, 11) (1, 12)
NUMBER '12345' (1, 13) (1, 18)
ri z� NUMBER '0xFF' (1, 0) (1, 4)
AMPER '&' (1, 5) (1, 6)
NUMBER '0x15' (1, 7) (1, 11)
VBAR '|' (1, 12) (1, 13)
NUMBER '1234' (1, 14) (1, 18)
rj rk s &r% rl �CTokenizeTest.test_int� s� � � ���M� ,� �
���M� ,� �
��� ,� /� �
��� -� 0� �
��� 0� 3� �
��� 5� 8� �
��� .� 1� �
��� 0� 3� �
��� 0� 3� r' c � � U O Q Q2 U O QQ2 U O QQ2 U O QQ2 U O QQ 2 U O Q
Q 2 U O Q Q
2 P! )rr z� NAME 'x' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
NUMBER '3.14159' (1, 4) (1, 11)
rs z� NAME 'x' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
NUMBER '314159.' (1, 4) (1, 11)
rt z� NAME 'x' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
NUMBER '.314159' (1, 4) (1, 11)
ru z� NAME 'x' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
NUMBER '3e14159' (1, 4) (1, 11)
rv z� NAME 'x' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
NUMBER '3E123' (1, 4) (1, 9)
rw z� NAME 'x' (1, 0) (1, 1)
PLUS '+' (1, 1) (1, 2)
NAME 'y' (1, 2) (1, 3)
EQUAL '=' (1, 4) (1, 5)
NUMBER '3e-1230' (1, 6) (1, 13)
rx z� NAME 'x' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
NUMBER '3.14e159' (1, 4) (1, 12)
rj rk s &r% ry �CTokenizeTest.test_float s� � � ���M� ,� �
���M� ,� �
���M� ,� �
���M� ,� �
���K� *� �
���O� .� �
���N� -� r' c �d � U O Q Q2 U O QQ2 U O QQ2 U O QQ2 U O QQ 2 U O Q
Q 2 U O Q Q
2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O Q Q 2 U O Q Q 2 U O Q Q!2 U O Q"Q#2 U O Q$Q%2 U O Q&Q'2 U O Q(Q)2 U O Q*Q+2 U O Q,Q-2 U O Q.Q/2 U O Q0Q12 U O Q2Q32 U O Q4Q52 U O Q6Q72 U O Q8Q92 U O Q:Q;2 U O Q<Q=2 P! )>r� a5 NAME 'x' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
STRING "''" (1, 4) (1, 6)
SEMI ';' (1, 6) (1, 7)
NAME 'y' (1, 8) (1, 9)
EQUAL '=' (1, 10) (1, 11)
STRING '""' (1, 12) (1, 14)
r� a6 NAME 'x' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
STRING '\'"\'' (1, 4) (1, 7)
SEMI ';' (1, 7) (1, 8)
NAME 'y' (1, 9) (1, 10)
EQUAL '=' (1, 11) (1, 12)
STRING '"\'"' (1, 13) (1, 16)
r� z� NAME 'x' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
STRING '"doesn\'t "' (1, 4) (1, 14)
NAME 'shrink' (1, 14) (1, 20)
STRING '", does it"' (1, 20) (1, 31)
r� z� NAME 'x' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
STRING "'abc'" (1, 4) (1, 9)
PLUS '+' (1, 10) (1, 11)
STRING "'ABC'" (1, 12) (1, 17)
r� z� NAME 'y' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
STRING '"ABC"' (1, 4) (1, 9)
PLUS '+' (1, 10) (1, 11)
STRING '"ABC"' (1, 12) (1, 17)
r� a� NAME 'x' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
STRING "r'abc'" (1, 4) (1, 10)
PLUS '+' (1, 11) (1, 12)
STRING "r'ABC'" (1, 13) (1, 19)
PLUS '+' (1, 20) (1, 21)
STRING "R'ABC'" (1, 22) (1, 28)
PLUS '+' (1, 29) (1, 30)
STRING "R'ABC'" (1, 31) (1, 37)
r� a� NAME 'y' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
STRING 'r"abc"' (1, 4) (1, 10)
PLUS '+' (1, 11) (1, 12)
STRING 'r"ABC"' (1, 13) (1, 19)
PLUS '+' (1, 20) (1, 21)
STRING 'R"ABC"' (1, 22) (1, 28)
PLUS '+' (1, 29) (1, 30)
STRING 'R"ABC"' (1, 31) (1, 37)
r� z� STRING "u'abc'" (1, 0) (1, 6)
PLUS '+' (1, 7) (1, 8)
STRING "U'abc'" (1, 9) (1, 15)
r� z� STRING 'u"abc"' (1, 0) (1, 6)
PLUS '+' (1, 7) (1, 8)
STRING 'U"abc"' (1, 9) (1, 15)
r� z� STRING "b'abc'" (1, 0) (1, 6)
PLUS '+' (1, 7) (1, 8)
STRING "B'abc'" (1, 9) (1, 15)
r� z� STRING 'b"abc"' (1, 0) (1, 6)
PLUS '+' (1, 7) (1, 8)
STRING 'B"abc"' (1, 9) (1, 15)
r� a; STRING "br'abc'" (1, 0) (1, 7)
PLUS '+' (1, 8) (1, 9)
STRING "bR'abc'" (1, 10) (1, 17)
PLUS '+' (1, 18) (1, 19)
STRING "Br'abc'" (1, 20) (1, 27)
PLUS '+' (1, 28) (1, 29)
STRING "BR'abc'" (1, 30) (1, 37)
r� a; STRING 'br"abc"' (1, 0) (1, 7)
PLUS '+' (1, 8) (1, 9)
STRING 'bR"abc"' (1, 10) (1, 17)
PLUS '+' (1, 18) (1, 19)
STRING 'Br"abc"' (1, 20) (1, 27)
PLUS '+' (1, 28) (1, 29)
STRING 'BR"abc"' (1, 30) (1, 37)
r� a; STRING "rb'abc'" (1, 0) (1, 7)
PLUS '+' (1, 8) (1, 9)
STRING "rB'abc'" (1, 10) (1, 17)
PLUS '+' (1, 18) (1, 19)
STRING "Rb'abc'" (1, 20) (1, 27)
PLUS '+' (1, 28) (1, 29)
STRING "RB'abc'" (1, 30) (1, 37)
r� a; STRING 'rb"abc"' (1, 0) (1, 7)
PLUS '+' (1, 8) (1, 9)
STRING 'rB"abc"' (1, 10) (1, 17)
PLUS '+' (1, 18) (1, 19)
STRING 'Rb"abc"' (1, 20) (1, 27)
PLUS '+' (1, 28) (1, 29)
STRING 'RB"abc"' (1, 30) (1, 37)
r� r� r� r� r� r� r� r� r� r� r� r� z"hola\\\r\ndfgf"z9 STRING '"hola\\\\\\r\\ndfgf"' (1, 0) (1, 16)
r� r� r� a= FSTRING_START 'fR"' (1, 0) (1, 3)
FSTRING_MIDDLE 'a' (1, 3) (1, 4)
LBRACE '{' (1, 4) (1, 5)
NAME 'b' (1, 5) (1, 6)
RBRACE '}' (1, 6) (1, 7)
FSTRING_MIDDLE 'c' (1, 7) (1, 8)
FSTRING_END '"' (1, 8) (1, 9)
r� r� r� r� zf"{
a}"z� FSTRING_START 'f"' (1, 0) (1, 2)
LBRACE '{' (1, 2) (1, 3)
NAME 'a' (2, 0) (2, 1)
RBRACE '}' (2, 1) (2, 2)
FSTRING_END '"' (2, 2) (2, 3)
r� r� zf"hola\\\r\ndfgf"z� FSTRING_START 'f"' (1, 0) (1, 2)
FSTRING_MIDDLE 'hola\\\\\\r\\ndfgf' (1, 2) (1, 16)
FSTRING_END '"' (1, 16) (1, 17)
r� a� FSTRING_START "f'''" (1, 0) (1, 4)
FSTRING_MIDDLE '__' (1, 4) (1, 6)
LBRACE '{' (1, 6) (1, 7)
NAME 'x' (2, 4) (2, 5)
COLON ':' (2, 5) (2, 6)
FSTRING_MIDDLE 'a\n' (2, 6) (3, 0)
RBRACE '}' (3, 0) (3, 1)
FSTRING_MIDDLE '__' (3, 1) (3, 3)
FSTRING_END "'''" (3, 3) (3, 6)
r� a� FSTRING_START "f'''" (1, 0) (1, 4)
FSTRING_MIDDLE '__' (1, 4) (1, 6)
LBRACE '{' (1, 6) (1, 7)
NAME 'x' (2, 4) (2, 5)
COLON ':' (2, 5) (2, 6)
FSTRING_MIDDLE 'a\n b\n c\n d\n' (2, 6) (6, 0)
RBRACE '}' (6, 0) (6, 1)
FSTRING_MIDDLE '__' (6, 1) (6, 3)
FSTRING_END "'''" (6, 3) (6, 6)
rj rk s &r% r� �CTokenizeTest.test_stringL s� � � ��� .� 1� �
��� 1� 4� �
��� >� A� �
��� /� 2� �
��� /� 2� �
��� C�
F�
�
��� C�
F�
�
��� -� 0� �
��� -� 0� �
��� -� 0� �
��� -� 0� �
��� C� F� �
��� C� F� �
��� C� F� �
��� C� F� �
��� -� 0� �
���L� +� �
���L� +� �
��� �� �
��� �� �
��� �� �
��� /� 2� �
���H� '� �
���K� *� �
���L� +� �
��� �� �
��� � � �
��� �� �
��� 0� 3� �
���
�
�
�
���
�
�
r' c �t � U O Q Q2 U O QQ2 U O QQ2 P! )r� a NAME 'def' (1, 0) (1, 3)
NAME 'd22' (1, 4) (1, 7)
LPAR '(' (1, 7) (1, 8)
NAME 'a' (1, 8) (1, 9)
COMMA ',' (1, 9) (1, 10)
NAME 'b' (1, 11) (1, 12)
COMMA ',' (1, 12) (1, 13)
NAME 'c' (1, 14) (1, 15)
EQUAL '=' (1, 15) (1, 16)
NUMBER '2' (1, 16) (1, 17)
COMMA ',' (1, 17) (1, 18)
NAME 'd' (1, 19) (1, 20)
EQUAL '=' (1, 20) (1, 21)
NUMBER '2' (1, 21) (1, 22)
COMMA ',' (1, 22) (1, 23)
STAR '*' (1, 24) (1, 25)
NAME 'k' (1, 25) (1, 26)
RPAR ')' (1, 26) (1, 27)
COLON ':' (1, 27) (1, 28)
NAME 'pass' (1, 29) (1, 33)
r� a� NAME 'def' (1, 0) (1, 3)
NAME 'd01v_' (1, 4) (1, 9)
LPAR '(' (1, 9) (1, 10)
NAME 'a' (1, 10) (1, 11)
EQUAL '=' (1, 11) (1, 12)
NUMBER '1' (1, 12) (1, 13)
COMMA ',' (1, 13) (1, 14)
STAR '*' (1, 15) (1, 16)
NAME 'k' (1, 16) (1, 17)
COMMA ',' (1, 17) (1, 18)
DOUBLESTAR '**' (1, 19) (1, 21)
NAME 'w' (1, 21) (1, 22)
RPAR ')' (1, 22) (1, 23)
COLON ':' (1, 23) (1, 24)
NAME 'pass' (1, 25) (1, 29)
r� a� NAME 'def' (1, 0) (1, 3)
NAME 'd23' (1, 4) (1, 7)
LPAR '(' (1, 7) (1, 8)
NAME 'a' (1, 8) (1, 9)
COLON ':' (1, 9) (1, 10)
NAME 'str' (1, 11) (1, 14)
COMMA ',' (1, 14) (1, 15)
NAME 'b' (1, 16) (1, 17)
COLON ':' (1, 17) (1, 18)
NAME 'int' (1, 19) (1, 22)
EQUAL '=' (1, 22) (1, 23)
NUMBER '3' (1, 23) (1, 24)
RPAR ')' (1, 24) (1, 25)
RARROW '->' (1, 26) (1, 28)
NAME 'int' (1, 29) (1, 32)
COLON ':' (1, 32) (1, 33)
NAME 'pass' (1, 34) (1, 38)
rj rk s &r% r� �CTokenizeTest.test_functionE
sJ � � ��� ?� B� �.
��� ;� >� �$
��� D� G� r' c �, � U O Q Q2 P! )r� a� NAME 'if' (1, 0) (1, 2)
NUMBER '1' (1, 3) (1, 4)
LESS '<' (1, 5) (1, 6)
NUMBER '1' (1, 7) (1, 8)
GREATER '>' (1, 9) (1, 10)
NUMBER '1' (1, 11) (1, 12)
EQEQUAL '==' (1, 13) (1, 15)
NUMBER '1' (1, 16) (1, 17)
GREATEREQUAL '>=' (1, 18) (1, 20)
NUMBER '5' (1, 21) (1, 22)
LESSEQUAL '<=' (1, 23) (1, 25)
NUMBER '0x15' (1, 26) (1, 30)
LESSEQUAL '<=' (1, 31) (1, 33)
NUMBER '0x12' (1, 34) (1, 38)
NOTEQUAL '!=' (1, 39) (1, 41)
NUMBER '1' (1, 42) (1, 43)
NAME 'and' (1, 44) (1, 47)
NUMBER '5' (1, 48) (1, 49)
NAME 'in' (1, 50) (1, 52)
NUMBER '1' (1, 53) (1, 54)
NAME 'not' (1, 55) (1, 58)
NAME 'in' (1, 59) (1, 61)
NUMBER '1' (1, 62) (1, 63)
NAME 'is' (1, 64) (1, 66)
NUMBER '1' (1, 67) (1, 68)
NAME 'or' (1, 69) (1, 71)
NUMBER '5' (1, 72) (1, 73)
NAME 'is' (1, 74) (1, 76)
NAME 'not' (1, 77) (1, 80)
NUMBER '1' (1, 81) (1, 82)
COLON ':' (1, 82) (1, 83)
NAME 'pass' (1, 84) (1, 88)
rj rk s &r% r� � CTokenizeTest.test_comparison�
r� r' c �, � U O Q Q2 P! )r� a$ NAME 'x' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
NUMBER '1' (1, 4) (1, 5)
MINUS '-' (1, 6) (1, 7)
NAME 'y' (1, 8) (1, 9)
PLUS '+' (1, 10) (1, 11)
NUMBER '15' (1, 12) (1, 14)
MINUS '-' (1, 15) (1, 16)
NUMBER '1' (1, 17) (1, 18)
PLUS '+' (1, 19) (1, 20)
NUMBER '0x124' (1, 21) (1, 26)
PLUS '+' (1, 27) (1, 28)
NAME 'z' (1, 29) (1, 30)
PLUS '+' (1, 31) (1, 32)
NAME 'a' (1, 33) (1, 34)
LSQB '[' (1, 34) (1, 35)
NUMBER '5' (1, 35) (1, 36)
RSQB ']' (1, 36) (1, 37)
rj rk s &r% r� �CTokenizeTest.test_additive�
r� r' c �, � U O Q Q2 P! )r� a� NAME 'x' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
NUMBER '1' (1, 4) (1, 5)
DOUBLESLASH '//' (1, 5) (1, 7)
NUMBER '1' (1, 7) (1, 8)
STAR '*' (1, 8) (1, 9)
NUMBER '1' (1, 9) (1, 10)
SLASH '/' (1, 10) (1, 11)
NUMBER '5' (1, 11) (1, 12)
STAR '*' (1, 12) (1, 13)
NUMBER '12' (1, 13) (1, 15)
PERCENT '%' (1, 15) (1, 16)
NUMBER '0x12' (1, 16) (1, 20)
AT '@' (1, 20) (1, 21)
NUMBER '42' (1, 21) (1, 23)
rj rk s &r% r� �!CTokenizeTest.test_multiplicative�
r� r' c �P � U O Q Q2 U O QQ2 P! )r� a� TILDE '~' (1, 0) (1, 1)
NUMBER '1' (1, 1) (1, 2)
CIRCUMFLEX '^' (1, 3) (1, 4)
NUMBER '1' (1, 5) (1, 6)
AMPER '&' (1, 7) (1, 8)
NUMBER '1' (1, 9) (1, 10)
VBAR '|' (1, 11) (1, 12)
NUMBER '1' (1, 12) (1, 13)
CIRCUMFLEX '^' (1, 14) (1, 15)
MINUS '-' (1, 16) (1, 17)
NUMBER '1' (1, 17) (1, 18)
r� aI MINUS '-' (1, 0) (1, 1)
NUMBER '1' (1, 1) (1, 2)
STAR '*' (1, 2) (1, 3)
NUMBER '1' (1, 3) (1, 4)
SLASH '/' (1, 4) (1, 5)
NUMBER '1' (1, 5) (1, 6)
PLUS '+' (1, 6) (1, 7)
NUMBER '1' (1, 7) (1, 8)
STAR '*' (1, 8) (1, 9)
NUMBER '1' (1, 9) (1, 10)
DOUBLESLASH '//' (1, 10) (1, 12)
NUMBER '1' (1, 12) (1, 13)
MINUS '-' (1, 14) (1, 15)
MINUS '-' (1, 16) (1, 17)
MINUS '-' (1, 17) (1, 18)
MINUS '-' (1, 18) (1, 19)
NUMBER '1' (1, 19) (1, 20)
DOUBLESTAR '**' (1, 20) (1, 22)
NUMBER '1' (1, 22) (1, 23)
rj rk s &r% r� �CTokenizeTest.test_unary�
s0 � � ��� 0� 3� �
��� 5� 8� r' c �, � U O Q Q2 P! )r� a� NAME 'import' (1, 0) (1, 6)
NAME 'sys' (1, 7) (1, 10)
COMMA ',' (1, 10) (1, 11)
NAME 'time' (1, 12) (1, 16)
NEWLINE '' (1, 16) (1, 16)
NAME 'x' (2, 0) (2, 1)
EQUAL '=' (2, 2) (2, 3)
NAME 'sys' (2, 4) (2, 7)
DOT '.' (2, 7) (2, 8)
NAME 'modules' (2, 8) (2, 15)
LSQB '[' (2, 15) (2, 16)
STRING "'time'" (2, 16) (2, 22)
RSQB ']' (2, 22) (2, 23)
DOT '.' (2, 23) (2, 24)
NAME 'time' (2, 24) (2, 28)
LPAR '(' (2, 28) (2, 29)
RPAR ')' (2, 29) (2, 30)
rj rk s &r% r� �CTokenizeTest.test_selector�
r� r' c �, � U O Q Q2 P! �r� a AT '@' (1, 0) (1, 1)
NAME 'staticmethod' (1, 1) (1, 13)
NEWLINE '' (1, 13) (1, 13)
NAME 'def' (2, 0) (2, 3)
NAME 'foo' (2, 4) (2, 7)
LPAR '(' (2, 7) (2, 8)
NAME 'x' (2, 8) (2, 9)
COMMA ',' (2, 9) (2, 10)
NAME 'y' (2, 10) (2, 11)
RPAR ')' (2, 11) (2, 12)
COLON ':' (2, 12) (2, 13)
NAME 'pass' (2, 14) (2, 18)
rj rk s &r% r� �CTokenizeTest.test_method r� r' c �, � U O Q Q2 P! rc rj rk s &r% r� �CTokenizeTest.test_tabs" r� r' c � � U O Q Q2 U O QQ2 U O QQ2 U O QQ2 U O QQ 2 U O Q
Q 2 U O Q Q
2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O QQ2 U O Q Q 2 U O Q Q 2 U O Q Q!2 U O Q"Q#2 U O Q$Q%2 U O Q&Q'2 U O Q(Q)2 P! )*r� z� NAME 'async' (1, 0) (1, 5)
EQUAL '=' (1, 6) (1, 7)
NUMBER '1' (1, 8) (1, 9)
r� a8 NAME 'a' (1, 0) (1, 1)
EQUAL '=' (1, 2) (1, 3)
LPAR '(' (1, 4) (1, 5)
NAME 'async' (1, 5) (1, 10)
EQUAL '=' (1, 11) (1, 12)
NUMBER '1' (1, 13) (1, 14)
RPAR ')' (1, 14) (1, 15)
r� z� NAME 'async' (1, 0) (1, 5)
LPAR '(' (1, 5) (1, 6)
RPAR ')' (1, 6) (1, 7)
r� a< NAME 'class' (1, 0) (1, 5)
NAME 'async' (1, 6) (1, 11)
LPAR '(' (1, 11) (1, 12)
NAME 'Bar' (1, 12) (1, 15)
RPAR ')' (1, 15) (1, 16)
COLON ':' (1, 16) (1, 17)
NAME 'pass' (1, 17) (1, 21)
r� z� NAME 'class' (1, 0) (1, 5)
NAME 'async' (1, 6) (1, 11)
COLON ':' (1, 11) (1, 12)
NAME 'pass' (1, 12) (1, 16)
r� z� NAME 'await' (1, 0) (1, 5)
EQUAL '=' (1, 6) (1, 7)
NUMBER '1' (1, 8) (1, 9)
r� z� NAME 'foo' (1, 0) (1, 3)
DOT '.' (1, 3) (1, 4)
NAME 'async' (1, 4) (1, 9)
r� a; NAME 'async' (1, 0) (1, 5)
NAME 'for' (1, 6) (1, 9)
NAME 'a' (1, 10) (1, 11)
NAME 'in' (1, 12) (1, 14)
NAME 'b' (1, 15) (1, 16)
COLON ':' (1, 16) (1, 17)
NAME 'pass' (1, 18) (1, 22)
r� a< NAME 'async' (1, 0) (1, 5)
NAME 'with' (1, 6) (1, 10)
NAME 'a' (1, 11) (1, 12)
NAME 'as' (1, 13) (1, 15)
NAME 'b' (1, 16) (1, 17)
COLON ':' (1, 17) (1, 18)
NAME 'pass' (1, 19) (1, 23)
r� z� NAME 'async' (1, 0) (1, 5)
DOT '.' (1, 5) (1, 6)
NAME 'foo' (1, 6) (1, 9)
r� r� r� z� NAME 'async' (1, 0) (1, 5)
NEWLINE '' (1, 5) (1, 5)
NAME 'await' (3, 0) (3, 5)
r� z� NAME 'async' (1, 0) (1, 5)
NEWLINE '' (1, 5) (1, 5)
ELLIPSIS '...' (2, 0) (2, 3)
NEWLINE '' (2, 3) (2, 3)
NAME 'await' (3, 0) (3, 5)
r� z� NAME 'async' (1, 0) (1, 5)
NEWLINE '' (1, 5) (1, 5)
NAME 'await' (2, 0) (2, 5)
r� z� NAME 'foo' (1, 0) (1, 3)
DOT '.' (1, 3) (1, 4)
NAME 'async' (1, 4) (1, 9)
PLUS '+' (1, 10) (1, 11)
NUMBER '1' (1, 12) (1, 13)
r� a; NAME 'async' (1, 0) (1, 5)
NAME 'def' (1, 6) (1, 9)
NAME 'foo' (1, 10) (1, 13)
LPAR '(' (1, 13) (1, 14)
RPAR ')' (1, 14) (1, 15)
COLON ':' (1, 15) (1, 16)
NAME 'pass' (1, 17) (1, 21)
r� a� NAME 'async' (1, 0) (1, 5)
NAME 'def' (1, 6) (1, 9)
NAME 'foo' (1, 10) (1, 13)
LPAR '(' (1, 13) (1, 14)
RPAR ')' (1, 14) (1, 15)
COLON ':' (1, 15) (1, 16)
NEWLINE '' (1, 16) (1, 16)
INDENT '' (2, -1) (2, -1)
NAME 'def' (2, 2) (2, 5)
NAME 'foo' (2, 6) (2, 9)
LPAR '(' (2, 9) (2, 10)
NAME 'await' (2, 10) (2, 15)
RPAR ')' (2, 15) (2, 16)
COLON ':' (2, 16) (2, 17)
NEWLINE '' (2, 17) (2, 17)
INDENT '' (3, -1) (3, -1)
NAME 'await' (3, 4) (3, 9)
EQUAL '=' (3, 10) (3, 11)
NUMBER '1' (3, 12) (3, 13)
NEWLINE '' (3, 13) (3, 13)
DEDENT '' (4, -1) (4, -1)
NAME 'if' (4, 2) (4, 4)
NUMBER '1' (4, 5) (4, 6)
COLON ':' (4, 6) (4, 7)
NEWLINE '' (4, 7) (4, 7)
INDENT '' (5, -1) (5, -1)
NAME 'await' (5, 4) (5, 9)
NEWLINE '' (5, 9) (5, 9)
DEDENT '' (6, -1) (6, -1)
DEDENT '' (6, -1) (6, -1)
NAME 'async' (6, 0) (6, 5)
PLUSEQUAL '+=' (6, 6) (6, 8)
NUMBER '1' (6, 9) (6, 10)
NEWLINE '' (6, 10) (6, 10)
r� a� NAME 'async' (1, 0) (1, 5)
NAME 'def' (1, 6) (1, 9)
NAME 'foo' (1, 10) (1, 13)
LPAR '(' (1, 13) (1, 14)
RPAR ')' (1, 14) (1, 15)
COLON ':' (1, 15) (1, 16)
NEWLINE '' (1, 16) (1, 16)
INDENT '' (2, -1) (2, -1)
NAME 'async' (2, 2) (2, 7)
NAME 'for' (2, 8) (2, 11)
NAME 'i' (2, 12) (2, 13)
NAME 'in' (2, 14) (2, 16)
NUMBER '1' (2, 17) (2, 18)
COLON ':' (2, 18) (2, 19)
NAME 'pass' (2, 20) (2, 24)
DEDENT '' (2, -1) (2, -1)
r� ah NAME 'async' (1, 0) (1, 5)
NAME 'def' (1, 6) (1, 9)
NAME 'foo' (1, 10) (1, 13)
LPAR '(' (1, 13) (1, 14)
NAME 'async' (1, 14) (1, 19)
RPAR ')' (1, 19) (1, 20)
COLON ':' (1, 20) (1, 21)
NAME 'await' (1, 22) (1, 27)
r� a~ NAME 'def' (1, 0) (1, 3)
NAME 'f' (1, 4) (1, 5)
LPAR '(' (1, 5) (1, 6)
RPAR ')' (1, 6) (1, 7)
COLON ':' (1, 7) (1, 8)
NEWLINE '' (1, 8) (1, 8)
INDENT '' (3, -1) (3, -1)
NAME 'def' (3, 2) (3, 5)
NAME 'baz' (3, 6) (3, 9)
LPAR '(' (3, 9) (3, 10)
RPAR ')' (3, 10) (3, 11)
COLON ':' (3, 11) (3, 12)
NAME 'pass' (3, 13) (3, 17)
NEWLINE '' (3, 17) (3, 17)
NAME 'async' (4, 2) (4, 7)
NAME 'def' (4, 8) (4, 11)
NAME 'bar' (4, 12) (4, 15)
LPAR '(' (4, 15) (4, 16)
RPAR ')' (4, 16) (4, 17)
COLON ':' (4, 17) (4, 18)
NAME 'pass' (4, 19) (4, 23)
NEWLINE '' (4, 23) (4, 23)
NAME 'await' (6, 2) (6, 7)
EQUAL '=' (6, 8) (6, 9)
NUMBER '2' (6, 10) (6, 11)
DEDENT '' (6, -1) (6, -1)
r� a� NAME 'async' (1, 0) (1, 5)
NAME 'def' (1, 6) (1, 9)
NAME 'f' (1, 10) (1, 11)
LPAR '(' (1, 11) (1, 12)
RPAR ')' (1, 12) (1, 13)
COLON ':' (1, 13) (1, 14)
NEWLINE '' (1, 14) (1, 14)
INDENT '' (3, -1) (3, -1)
NAME 'def' (3, 2) (3, 5)
NAME 'baz' (3, 6) (3, 9)
LPAR '(' (3, 9) (3, 10)
RPAR ')' (3, 10) (3, 11)
COLON ':' (3, 11) (3, 12)
NAME 'pass' (3, 13) (3, 17)
NEWLINE '' (3, 17) (3, 17)
NAME 'async' (4, 2) (4, 7)
NAME 'def' (4, 8) (4, 11)
NAME 'bar' (4, 12) (4, 15)
LPAR '(' (4, 15) (4, 16)
RPAR ')' (4, 16) (4, 17)
COLON ':' (4, 17) (4, 18)
NAME 'pass' (4, 19) (4, 23)
NEWLINE '' (4, 23) (4, 23)
NAME 'await' (6, 2) (6, 7)
EQUAL '=' (6, 8) (6, 9)
NUMBER '2' (6, 10) (6, 11)
DEDENT '' (6, -1) (6, -1)
rj rk s &r% r� �CTokenizeTest.test_async3 s� � � ���K� *� �
��� -� 0� �
���I� (� �
��� 3� 6� �
��� .� 1� �
���K� *� �
���K� *� �
��� 4� 7� �
��� 5� 8� �
���K� *� �
���G� &� �
��� 4� 7� �
��� /� 2� �
���N� -� �
���O� .� �
��� 3� 6� �
��� �#�* �X
��� H� K� �&
��� 9� <� �
��� � �
! �F
��� � �
" r' c �, � U O Q Q2 P! )r� u7 NAME 'Örter' (1, 0) (1, 5)
EQUAL '=' (1, 6) (1, 7)
STRING "u'places'" (1, 8) (1, 17)
NEWLINE '' (1, 17) (1, 17)
NAME 'grün' (2, 0) (2, 4)
EQUAL '=' (2, 5) (2, 6)
STRING "U'green'" (2, 7) (2, 15)
rj rk s &r% r� �CTokenizeTest.test_unicode= r� r' c �P � Q o, QMQMQMQMQMQMQMQMQ MQ
MQ MQ MQ
MQMQMQMQMQMQMQMQMQMQMQMQMQMQMQMQ MQMQMQMF DF oU O UQ 5 9^tt^ 2 U O [ O V2 PPP2 JH P! ) % f h J] 9 h) c �l � [ U 2 o[ [ O UO 2 2 ! r )r rC r rf r- )rg �
the_strings & r% �
get_tokens�5CTokenizeTest.test_invalid_syntax.<locals>.get_tokensJ s( � �!�&�)�J���B�B�:�CV�CV�W�X� Xr' z(1+2]z(1+2}z{1+2]�1_z1.2_�1e2_z1e+� u €�0b12�0b1_2�0b2�0b1_�0b�0o18�0o1_8�0o8�0o1_�0o�0x1_�0x�012z'sdfsdfz '''sdfsdf''�]z6 f'__{
x:d
}__'z a
)r a� ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((a)))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))r, rA r r� )r1 rn r s & r% �test_invalid_syntax�!CTokenizeTest.test_invalid_syntaxI s� � � Y�%
� �%
� �%
�
�%
�
� %
�
� %
�
�
%
�
�%
�
�%
�
�%
�
�%
�
�%
�
�%
�
� %
�
� %
�
�!%
�"
�#%
�$
�%%
�&
�'%
�(
�)%
�*
�+%
�,
�-%
�.
�/%
�0
�1%
�2
�3%
�4
�5%
�6
�7%
�8
�9%
�:
�;%
�<
"�=%
�>
�?%
�@
�A%
�H
�I%
�D�L � � �4� �(�(��!�!�(�"5�"5�z�H� )�(�M%
�L )�(�(�s �%!B �B%
� B%
c � `� ]doQ oU U]*
2 o[ U2 n[ [ O RO 2 2 oU O UQ* O [ O 2 U O UP * O [ O 2 [ UQQ2 U U2 o[ U2 nU O [ U1Q k2 U O [ [ UQQ2 P! )�d c �v � PO Q [ U 2 2 2 oUQU * Q* *
oU! )� c 3 �N � S F � D oQ U* Q* w � J P! 3h)� z if True:
r; )�.0rJ s & r% � <genexpr>�ICTokenizeTest.test_max_indent.<locals>.generate_source.<locals>.<genexpr>| s ��� � ��1�d�Q�h�,� 6� 6��s � %r� zpass
)r? rf )�indentsr� s & r% �generate_source�6CTokenizeTest.test_max_indent.<locals>.generate_source{ s2 � ��W�W� O��g�� O�O�F� �d�W�n�x�/� /�F��Mr' rP �execc �X :� [ [ O R O 2 2 ! r )rC r rf r- )� the_inputs �r% �<lambda>�/CTokenizeTest.test_max_indent.<locals>.<lambda>� s �� �D��9c�9c�dm�dv�dv�9w�4xr' rK ) r rC r rf r- r. r �DEDENTr r� rA rR )r1 � MAXINDENTr� �validrL �invalidr� s & @r% �test_max_indent� CTokenizeTest.test_max_indentw s� �� �� � �
� �A�
�.�� �U�O� ��h�@�@��AS�AS�T�U�� ������ � �(�/�/�:� ������ � �(�*<�*<�=���z�6�*�!�)�,�� �W�%� � ���*�,x�y� ��� �g�w�
�F�
r' c � � Q o[ Q2 oU O UQ2 [ Q2 oU O U U2 U U2 2 [ Q2 oU O UQ2 [ Q2 oU O U U2 U U2 2 [ Q2 oU O UQ2 [ Q 2 oU O U U2 U U2 2 P! )
c � � [ U 2 o[ O UO 2 TT Tt, tF D uq p2U 1MJ
too o! t too oh r )r r rf r- )rg rm �kindr& s & r% rn �ECTokenizeTest.test_continuation_lines_indentation.<locals>.get_tokens� sL � �!�&�)�J� �A�A�*�BU�BU�V�X�V� (:��q�6�N�V�X�
X�� Xs �Az�
def fib(n):
\
'''Print a Fibonacci series up to n.'''
\
a, b = 0, 1
ae NAME 'def' (2, 0) (2, 3)
NAME 'fib' (2, 4) (2, 7)
LPAR '(' (2, 7) (2, 8)
NAME 'n' (2, 8) (2, 9)
RPAR ')' (2, 9) (2, 10)
COLON ':' (2, 10) (2, 11)
NEWLINE '' (2, 11) (2, 11)
INDENT '' (4, -1) (4, -1)
STRING "'''Print a Fibonacci series up to n.'''" (4, 0) (4, 39)
NEWLINE '' (4, 39) (4, 39)
NAME 'a' (6, 0) (6, 1)
COMMA ',' (6, 1) (6, 2)
NAME 'b' (6, 3) (6, 4)
EQUAL '=' (6, 5) (6, 6)
NUMBER '0' (6, 7) (6, 8)
COMMA ',' (6, 8) (6, 9)
NUMBER '1' (6, 10) (6, 11)
NEWLINE '' (6, 11) (6, 11)
DEDENT '' (6, -1) (6, -1)
zu
def fib(n):
'''Print a Fibonacci series up to n.'''
a, b = 0, 1
z>
pass
\
pass
z� NAME 'pass' (2, 0) (2, 4)
NEWLINE '' (2, 4) (2, 4)
NAME 'pass' (5, 0) (5, 4)
NEWLINE '' (5, 4) (5, 4)
z+
pass
pass
z�
if x:
y = 1
\
\
\
\
foo = 1
aj NAME 'if' (2, 0) (2, 2)
NAME 'x' (2, 3) (2, 4)
COLON ':' (2, 4) (2, 5)
NEWLINE '' (2, 5) (2, 5)
INDENT '' (3, -1) (3, -1)
NAME 'y' (3, 4) (3, 5)
EQUAL '=' (3, 6) (3, 7)
NUMBER '1' (3, 8) (3, 9)
NEWLINE '' (3, 9) (3, 9)
NAME 'foo' (8, 4) (8, 7)
EQUAL '=' (8, 8) (8, 9)
NUMBER '1' (8, 10) (8, 11)
NEWLINE '' (8, 11) (8, 11)
DEDENT '' (8, -1) (8, -1)
zI
if x:
y = 1
foo = 1
)r r5 r. )r1 rn r � code_no_conts & r% �#test_continuation_lines_indentation�1CTokenizeTest.test_continuation_lines_indentation� s � � X�
� �
��
���D� # �
�, � �
� �
����D�)�:�l�+C�D�� �
��
���D� # �
� � �
� �
����D�)�:�l�+C�D�� �
��
���D� # �
�" � �
� �
����D�)�:�l�+C�Dr' r; )r
r r r
r5 rN rl ry r� r� r� r� r� r� r� r� r� r� r� r� r �skip_wasi_stack_overflowr� r� r r r s @r% rG rG � s� �� � �E�3�(? �B, �\w �r= �~$ �L �. �($ �L �, �" �"H �T
�,I�\
�%�%�'�
� (�
�,bE� bEr' rG c �( ` � \ sQ sQs n Q sQsU sP! )�CTokenizerBufferTestsi� c �� � Q QQ* QQQ* 0o[ O 2 9^tt^ 2 o[ UQU2 o[ U2 PPP2 P! ) % f h P! 9 h)z" #coding: latin-1
#r� i' z
#i' �foo)r r� r
r )r1 � test_scriptr� � file_names & r% �!test_newline_at_the_end_of_buffer�7CTokenizerBufferTests.test_newline_at_the_end_of_buffer� s] � �
�
�e�)��
�
�e�)��� � �
�
�
!�
!�X�#�H�e�[�A�I� �I� &� "�
!�
!�
!�s �A�A) �!A) r; )r
r r r
r� r r r s @r% r� r� � s �� � �
'�
'r' r� c �\ ` � \ sQ sQs n Q s\Q 2 sQ sQ sQ s Q s
Q s Q s Q
s
U sP! ) �CommandLineTesti
c � � [ O 2 U m U O [ O
U O 2 P! r )�tempfile�mktemprU r r r! rk s &r% �setUp�CommandLineTest.setUp
s( � � ���)��
� ��� �(�(�$�-�-�8r' c �P � [ O QQU 2 O 2 ! )z�Dedent *string* and strip it from its surrounding whitespaces.
This method is used by the other utility functions so that any
string to write or to match against can be freely indented.
z\s+r )r� �subr" )rg s &r% �text_normalize� CommandLineTest.text_normalize
s � � �v�v�f�c�6�*�0�0�2�2r' c � � [ U O Q 2 9^tt^ 2 oUO U2 PPP2 P! ) % f h P! 9 h)r )r3 rU r� )r1 �contentr$ s && r% �
set_source�CommandLineTest.set_source
s0 � �
�$�-�-��
%�
%�� �H�H�W� � &�
%�
%�
%�s �<�A
�A
c � � [ 2 o[ O U2 9^tt^ 2 [ O , UNU O
MQ 5 PPP2 U O
UO 2 2 ! ) % f h K/9 h)�r� )r �
contextlib�redirect_stdoutr �_mainrU r� �getvalue)r1 �flags�outputs &* r% �invoke_tokenize� CommandLineTest.invoke_tokenize
s[ � ����
�
'�
'��
/�
/� �N�N� 7�%� 7�� � � 7� 8� 0��"�"�6�?�?�#4�5�5� 0�
/�s �&A7�7B � B c �8 � U O VQ 5 9^tt^ 2 U O U2 U O U oU O U2 oU O UO 2 UO 2 2 PPP2 P! ) % f h P! 9 h))r� r� )r, r� r� r� �assertListEqualr )r1 r� �expectr� �ress &&&* r% � check_output� CommandLineTest.check_output
sn � �
�\�\��\�
5�
5� �O�O�F� #��&�&��.�C��(�(��0�F� � � �� � �!1�6�3D�3D�3F� G� 6�
5�
5�
5�s �A"B�B �B c �� � QoU O Q2 SF D7 oU O UQ5 9^tt^ 2 U O U2 oPPP2 J9 U O [ 2 9^tt^ 2 [
O [ 2 2 9^tt^ 2 U O Q2 oPPP2 PPP2 P! ) % f h J� 9 h ) % f h K.9 h ) % f h P! 9 h)�-ezS
def f():
print(x)
return None
r� z --unknown�r� z--exact)r� r, r� rA �
SystemExitr� �redirect_stderrr )r1 �
base_flags�flagr& s & r% �test_invocation� CommandLineTest.test_invocation!
s� � �&�
� ��� �
� �D�� � �4� �(�(��(�(��.�� )�(� � �
�
�z�
*�
*��+�+�H�J�7�7��(�(��5�� 8� +�
*� )�(�(��
8�7�� +�
*�
*�sA �B; �0'C"�C �)C"�;C
�C
�C
�C
�C"�"C3 �+C3 c �2 � Q oQoU O V2 P! )�a = 1a=
0,0-0,0: ENCODING 'utf-8'
1,0-1,1: NAME 'a'
1,2-1,3: OP '='
1,4-1,5: NUMBER '1'
1,5-1,6: NEWLINE ''
2,0-2,0: ENDMARKER ''
�r� )r1 r� r� s & r% �test_without_flag�!CommandLineTest.test_without_flag4
s � ��� ��
���&�)r' c �H � Q oQoQF D oU O VU2 J P! )r� a=
0,0-0,0: ENCODING 'utf-8'
1,0-1,1: NAME 'a'
1,2-1,3: EQUAL '='
1,4-1,5: NUMBER '1'
1,5-1,6: NEWLINE ''
2,0-2,0: ENDMARKER ''
r� r� )r1 r� r� r� s & r% �test_exact_flag� CommandLineTest.test_exact_flagA
s* � ��� �� &�D� � � �f�d� 3� &r' )rU )r
r r r
r� � staticmethodr� r� r� r� r� r� r� r r r s @r% r� r�
sB �� � �9� �3� �3� �6� H�6�& *� 4� 4r' r� c �8 ` � \ sQ sQs n \Q 2 sQ sQsU sP! )�StringPrefixTestiP
c �0 � [ 2 o [ O [ O [ O
2 F D1 o [
U Q 02 U O UO 2 2 J3 [ 2 o[ O 2 F D� o] oQ [ O V2 2 F D� o[ O U2 F D� o[ O STt, tF D pwUO 2 1MJ to F DF oPO U2 o U Q6X c J [
U Q 02 UO U 2 U]*
oJH J� J� U] 6X f J� Ut ! P! [ c CJ= h 9 ht toh [ c J� h 9 h)z""c 3 �P � S F � D oPO U2 w � J P! 3h)r� )r? )r� �ls & r% r� �<StringPrefixTest.determine_valid_prefixes.<locals>.<genexpr>l
s ��S�� �S�A� ����
�
�S�s � &�not)�set� itertools�chainrg �ascii_lowercase�ascii_uppercaser- �add�lowerrN �count� combinations� permutations�product�upperr? )
�single_char_valid_prefixes�letter�valid_prefixes�length�num_at_this_length�prefix�t�cr� �ps
r% �determine_valid_prefixes�)StringPrefixTest.determine_valid_prefixesQ
s^ � � &)�U�"� �o�o�f�&<�&<�f�>T�>T�U�F�
���x�r�]�#�*�.�.�v�|�|�~�>� V� ��� �o�o�'�F�!"� ��"�/�/�0J�S��� #�/�/��7�A� &� .� .��0K��A�Q�W�W�Y� ��0K� L�� �G�G�A�J�� ��:� %� !� �A�3�b��N�
+� .� .�q� 1� .�!� 3� .� M� 8� �* "�Q�&�%�%�1 (�� �
��
�� 1L�� +� !� �!�s6 �-E.�' F�((F�. E>�8E>�=E>� F�F�Fc �8 � U O [ O ] * Q2 U O [ O P * Q2 [ [ O ]P O Q2 2 oU O VO 2 2 P! )r r� r r| )r. r � StringPrefixr� r6 r� )r1 �defined_prefixess & r% �
test_prefixes� StringPrefixTest.test_prefixes�
sy � �
����.�.�q�1�3�7� ����.�.�r�2�C�8� �x� 4� 4�Q�r� :� @� @�� E�F��
���)�+H�+H�+J�Kr' r; ) r
r r r
r� r� r� r r r s @r% r� r� P
s$ �� � ��0&� �0&�fL� Lr' r� �__main__)/r� r� r= r� rg r� r! r �unittest�ior r �textwrapr r r �testr � test.supportr �test.support.script_helperr r
r �test.support.numbersr r
r& r) r r' r) r9 r\ rl rC r� r� rB r; rG r� � force_not_colorized_test_classr� r� r
�mainr; r' r% �<module>r sR �� � � � �
� � � � � � � #� � "� Y� Y��� h �8� h �T%A� � A�
?� :�x� :�<#I�H� #I�L<�H� <�.N"�� N"�b v2�8� v2�r&G�X� &G�R
.� U#�H� U#�p2�� 2�8rE�H� rE�j! '�H� -� -� '� �'�'�K4�h�'�'� K4� (�K4�\EL�x�(�(� EL�P �z�� �M�M�O� r'