Python tokenize Functions

EditRocket provides the following information on tokenize functions in the Python source code builder.

generate_tokens(readline) - The generate_tokens() generator requires one argment, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects

tokenize(readline[, tokeneater]) - The tokenize() function accepts two parameters: one representing the input stream, and one providing an output mechanism for tokenize().

untokenize(iterable) - Converts tokens back into Python source code.