Literals are a pain to tokenize, especially when the literal can start with a valid math operator. I'm looking at you JavaScript Regex Literal and signed integer literals! You broke my code.
var x = /123/;
var y =5 /2;
z(5,+54.1, -13);
This is where you can blur the line between what the tokenizer should do and what the parser does. Going the wrong way makes things bad. Like, crashing bad. |