Python tokenize Functions
EditRocket provides the following information on tokenize functions in the Python source code builder.
generate_tokens(readline) - The generate_tokens() generator requires one argment, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects
tokenize(readline[, tokeneater]) - The tokenize() function accepts two parameters: one representing the input stream, and one providing an output mechanism for tokenize().
untokenize(iterable) - Converts tokens back into Python source code.
