This fixes several bugs with regards to line breaks and comments:

- any unicode line break (line feed, vertical tab, form feed, carriage
   return, NEL, LS and PS) is considered to terminate a single-line
   comment. The line break itself is considered to be the next token
   after the comment, leading to a parser error if it is not an
   ascii character (i.e. for NEL, LS and PS).
 - unterminated multiline comments are considered illegal tokens
 - '/** /' is considered an unterminated multiline comment
   (previously, whitespace was allowed before the last '/'
This commit is contained in:
chriseth
2018-09-10 12:28:34 +02:00
parent 65a439b0fb
commit c0d9b492a2
3 changed files with 66 additions and 28 deletions
+8
View File
@@ -393,6 +393,14 @@ BOOST_AUTO_TEST_CASE(invalid_hex_literal_nonhex_string)
BOOST_CHECK_EQUAL(scanner.next(), Token::Illegal);
}
BOOST_AUTO_TEST_CASE(invalid_multiline_comment_close)
{
// This used to parse as "comment", "identifier"
Scanner scanner(CharStream("/** / x"));
BOOST_CHECK_EQUAL(scanner.currentToken(), Token::Illegal);
BOOST_CHECK_EQUAL(scanner.next(), Token::EOS);
}
BOOST_AUTO_TEST_SUITE_END()