Tokenizer class for parsing texts (also called 'scanner' or 'lexer' -> lexical analysis) More...

#include <Tokenizer.h>

Public Member Functions
PLCORE_API	Tokenizer ()
	Constructor.
	~Tokenizer ()
	Destructor.
String	GetDelimiters () const
	Get delimiters (characters that a treated as whitespace)
void	SetDelimiters (const String &sDelimiters)
	Set delimiters (characters that a treated as whitespace)
String	GetSingleChars () const
	Get single characters.
void	SetSingleChars (const String &sSingleChars)
	Set single characters.
String	GetQuotes () const
	Get characters that are used for quotes.
void	SetQuotes (const String &sQuotes)
	Set characters that are used for quotes.
String	GetCommentStartTag () const
	Get the string that starts a multi-line comment.
void	SetCommentStartTag (const String &sCommentStartTag)
	Set the string that starts a multi-line comment.
String	GetCommentEndTag () const
	Get the string that ends a multi-line comment.
void	SetCommentEndTag (const String &sCommentEndTag)
	Set the string that ends a multi-line comment.
String	GetSingleLineComment () const
	Get the string that starts a single-line comment.
void	SetSingleLineComment (const String &sSingleLineComment)
	Set the string that starts a single-line comment.
bool	IsCaseSensitive () const
	Get case sensitivity flag.
void	SetCaseSensitive (bool bCaseSensitive)
	Set case sensitivity flag.
PLCORE_API void	Start (const String &sBuffer)
	Starts the tokenizer on a string.
PLCORE_API void	Start (File &cFile)
	Starts the tokenizer on a file.
PLCORE_API void	Stop ()
	Stops the tokenizer.
PLCORE_API Array< String >	GetTokens ()
	Reads all tokens until the end of the stream.
PLCORE_API String	GetNextToken ()
	Reads the next token from the stream.
PLCORE_API bool	ExpectToken (const String &sExpected)
	Expects the next token to be equal to a given string.
PLCORE_API bool	FindToken (const String &sExpected)
	Finds a given token in the stream.
String	GetToken () const
	Returns the current token.
bool	CompareToken (const String &sExpected)
	Compares the current token with a given string.
uint32	GetPosition () const
	Returns the current position in the stream.
uint32	GetLine () const
	Returns the current line (counted by ' ' occurrences)
PLCORE_API void	PushState ()
	Saves the current state of the tokenizer on a state stack.
PLCORE_API void	PopState ()
	Restores the last saved state from the stack.
void	DropState ()
	Deletes the last saved state from the stack.
bool	ParseNumber (int &nNumber)
	Expects the next token to be a number and returns it as an integer value.
bool	ParseNumber (float &fNumber)
	Expects the next token to be a floating point number and returns it as a float value.
bool	ParseNumber (double &dNumber)
	Expects the next token to be a floating point number and returns it as a double value.
PLCORE_API bool	ParseVector (Array< String > &cVector, const String &sStart="[", const String &sEnd="]", const String &sSeparator=",")
	Expects the next tokens to be a vector and returns it as an array of strings.
PLCORE_API bool	ParseVector (Array< int > &cVector, const String &sStart="[", const String &sEnd="]", const String &sSeparator=",")
	Expects the next tokens to be a vector and returns it as an array of ints.
PLCORE_API bool	ParseVector (Array< float > &cVector, const String &sStart="[", const String &sEnd="]", const String &sSeparator=",")
	Expects the next tokens to be a vector and returns it as an array of floats.
PLCORE_API bool	ParseVector (Array< double > &cVector, const String &sStart="[", const String &sEnd="]", const String &sSeparator=",")
	Expects the next tokens to be a vector and returns it as an array of doubles.
PLCORE_API bool	ParseEquation (String &sName, String &sValue, const String &sEquation="")
	Expects the next tokens to be an equation and returns it.
bool	ParseEquation (String &sName, int &nValue, const String &sEquation="")
	Expects the next tokens to be an equation and returns it.
bool	ParseEquation (String &sName, float &fValue, const String &sEquation="")
	Expects the next tokens to be an equation and returns it.
bool	ParseEquation (String &sName, double &dValue, const String &sEquation="")
	Expects the next tokens to be an equation and returns it.

Detailed Description

Tokenizer class for parsing texts (also called 'scanner' or 'lexer' -> lexical analysis)

Remarks:: The tokenizer parses an input stream (e.g. a file or string) and produces a subsequent list of tokens by filtering out whitespace, comments and such. There are many settings that can be manipulated to serve your needs, like choosing the type of comment tags or the characters that shall be considered whitespace. In general, the tokenizer should only be used to produce a list of tokens, after that a real parser is used to check the syntax and semantics of the parsed code. In addition to this basic functionality the class also provides some advanced functions to parse typical syntax like equations (a=b) or vectors (a b c). These functions can be used if no strict syntax is needed, e.g. for config files. But they are not intended to replace a decent parser.

*    Usage example:
*    Tokenizer cTokenizer;                         // Tokenizer instance
*    String sText = "This is a test";              // Text to be parsed
*    cTokenizer.Start(sText);                      // Start tokenizer
*    String sToken;                                // String for current token
*    sToken = cTokenizer.GetNextToken();           // Will return 'This'
*    sToken = cTokenizer.GetNextToken();           // Will return 'is'
*    sToken = cTokenizer.GetNextToken();           // Will return 'a'
*    sToken = cTokenizer.GetNextToken();           // Will return 'test'
*

Constructor & Destructor Documentation

PLCORE_API PLCore::Tokenizer::Tokenizer ( )

Constructor.

PLCore::Tokenizer::~Tokenizer ( ) [inline]

Destructor.

Member Function Documentation

String PLCore::Tokenizer::GetDelimiters ( ) const [inline]

Get delimiters (characters that a treated as whitespace)

Returns:: Delimiter characters

void PLCore::Tokenizer::SetDelimiters ( const String & sDelimiters ) [inline]

Set delimiters (characters that a treated as whitespace)

Parameters:

[in] sDelimiters Delimiter characters

Remarks:: Default: " \t\r\n"

String PLCore::Tokenizer::GetSingleChars ( ) const [inline]

Get single characters.

Returns:: Single characters

void PLCore::Tokenizer::SetSingleChars ( const String & sSingleChars ) [inline]

Set single characters.

Parameters:

[in] sSingleChars Single characters

Remarks:: Default: "{}()[]<*>/=,;\""

String PLCore::Tokenizer::GetQuotes ( ) const [inline]

Get characters that are used for quotes.

Returns:: Quote characters

void PLCore::Tokenizer::SetQuotes ( const String & sQuotes ) [inline]

Set characters that are used for quotes.

Parameters:

[in] sQuotes Quote characters

Remarks:: Default: "\"\'"

String PLCore::Tokenizer::GetCommentStartTag ( ) const [inline]

Get the string that starts a multi-line comment.

Returns:: Comment start tag

void PLCore::Tokenizer::SetCommentStartTag ( const String & sCommentStartTag ) [inline]

Set the string that starts a multi-line comment.

Parameters:

[in] sCommentStartTag Comment start tag

Remarks:: Default is slash-star ("/ *" without the space in the middle, just written with a space to keep C++ compilers happy).

String PLCore::Tokenizer::GetCommentEndTag ( ) const [inline]

Get the string that ends a multi-line comment.

Returns:: Comment end tag

void PLCore::Tokenizer::SetCommentEndTag ( const String & sCommentEndTag ) [inline]

Set the string that ends a multi-line comment.

Parameters:

[in] sCommentEndTag Comment end tag

Remarks:: Default: "*\/"

String PLCore::Tokenizer::GetSingleLineComment ( ) const [inline]

Get the string that starts a single-line comment.

Returns:: Comment start tag

void PLCore::Tokenizer::SetSingleLineComment ( const String & sSingleLineComment ) [inline]

Set the string that starts a single-line comment.

Parameters:

[in] sSingleLineComment Comment start tag

Remarks:: Default: "//"

bool PLCore::Tokenizer::IsCaseSensitive ( ) const [inline]

Get case sensitivity flag.

Returns:: 'true' if the text is parsed case sensitive

void PLCore::Tokenizer::SetCaseSensitive ( bool bCaseSensitive ) [inline]

Set case sensitivity flag.

Parameters:

[in] bCaseSensitive 'true' if the text is parsed case sensitive (default is false)

Remarks:: Default: false

PLCORE_API void PLCore::Tokenizer::Start ( const String & sBuffer )

Starts the tokenizer on a string.

Parameters:

[in] sBuffer String buffer

PLCORE_API void PLCore::Tokenizer::Start ( File & cFile )

Starts the tokenizer on a file.

Parameters:

[in] cFile File to read, released automatically if Stop() is called

PLCORE_API void PLCore::Tokenizer::Stop ( )

Stops the tokenizer.

PLCORE_API Array<String> PLCore::Tokenizer::GetTokens ( )

Reads all tokens until the end of the stream.

Returns:: Array of all tokens of the stream

PLCORE_API String PLCore::Tokenizer::GetNextToken ( )

Reads the next token from the stream.

Returns:: Next token

Note:

After the token has been read this function goes to the next token in the stream
To get the read token again, use GetToken()

See also:

GetToken()

PLCORE_API bool PLCore::Tokenizer::ExpectToken ( const String & sExpected )

Expects the next token to be equal to a given string.

Returns:: 'true' if the next token is equal to the string

Note:

If the expected token has been found, the tokenizer goes to the next token in the stream, otherwise it stays at the current token
The comparison of strings is done according to the settings given in SetCaseSensitive()

See also:

PLCORE_API bool PLCore::Tokenizer::FindToken ( const String & sExpected )

Finds a given token in the stream.

Returns:: 'true' if the token has been found

Note:

Reads the next token until the expected token has been found or the end of the stream has been reached
If the function has succeed, the next call of GetNextToken() will return the desired token
The comparison of strings is done according to the settings given in SetCaseSensitive()

See also:

String PLCore::Tokenizer::GetToken ( ) const [inline]

Returns the current token.

Returns:: Current token

Note:

Does not go to the next token in the stream, so multiple calls of this function will always return the same token.

bool PLCore::Tokenizer::CompareToken ( const String & sExpected ) [inline]

Compares the current token with a given string.

Returns:: 'true' if the token is equal to the string

Note:

Does not go to the next token in the stream, so multiple calls of this function will always return the same token
The comparison of strings is done according to the settings given in SetCaseSensitive()

See also:

uint32 PLCore::Tokenizer::GetPosition ( ) const [inline]

Returns the current position in the stream.

Returns:: Position in the stream

uint32 PLCore::Tokenizer::GetLine ( ) const [inline]

Returns the current line (counted by '
' occurrences)

Returns:: Line in the file

PLCORE_API void PLCore::Tokenizer::PushState ( )

Saves the current state of the tokenizer on a state stack.

PLCORE_API void PLCore::Tokenizer::PopState ( )

Restores the last saved state from the stack.

void PLCore::Tokenizer::DropState ( ) [inline]

Deletes the last saved state from the stack.

bool PLCore::Tokenizer::ParseNumber ( int & nNumber ) [inline]

Expects the next token to be a number and returns it as an integer value.

Parameters:

[out] nNumber Receives the number

Returns:: 'true' on success, else 'false'

Note:

If a number has been found, the tokenizer goes to the next token in the stream, otherwise it stays at the current token

bool PLCore::Tokenizer::ParseNumber ( float & fNumber ) [inline]

Expects the next token to be a floating point number and returns it as a float value.

Parameters:

[out] fNumber Receives the number

Returns:: 'true' on success, else 'false'

Note:

If a number has been found, the tokenizer goes to the next token in the stream, otherwise it stays at the current token

bool PLCore::Tokenizer::ParseNumber ( double & dNumber ) [inline]

Expects the next token to be a floating point number and returns it as a double value.

Parameters:

[out] dNumber Receives the number

Returns:: 'true' on success, else 'false'

Note:

If a number has been found, the tokenizer goes to the next token in the stream, otherwise it stays at the current token

PLCORE_API bool PLCore::Tokenizer::ParseVector	(	Array< String > &	cVector,
		const String &	sStart = `"["`,
		const String &	sEnd = `"]"`,
		const String &	sSeparator = `","`
	)

Expects the next tokens to be a vector and returns it as an array of strings.

Parameters:

[out]	cVector	Receives the vector elements
[in]	sStart	Open bracket (e.g. "[")
[in]	sEnd	Closed bracket (e.g. "]")
[in]	sSeparator	Separator between the elements (e.g. ","). Can also be ""

Returns:: 'true' on success, else 'false'

Remarks:: Example: [one, two, three]

Note:

If a vector has been found, the tokenizer goes to the next token in the stream, otherwise it stays at the current token

PLCORE_API bool PLCore::Tokenizer::ParseVector	(	Array< int > &	cVector,
		const String &	sStart = `"["`,
		const String &	sEnd = `"]"`,
		const String &	sSeparator = `","`
	)

Expects the next tokens to be a vector and returns it as an array of ints.

Parameters:

[out]	cVector	Receives the vector elements
[in]	sStart	Open bracket (e.g. "[")
[in]	sEnd	Closed bracket (e.g. "]")
[in]	sSeparator	Separator between the elements (e.g. ","). Can also be ""

Returns:: 'true' on success, else 'false'

Remarks:: Example: [1, 2, 3]

Note:

If a vector has been found, the tokenizer goes to the next token in the stream, otherwise it stays at the current token

PLCORE_API bool PLCore::Tokenizer::ParseVector	(	Array< float > &	cVector,
		const String &	sStart = `"["`,
		const String &	sEnd = `"]"`,
		const String &	sSeparator = `","`
	)

Expects the next tokens to be a vector and returns it as an array of floats.

Parameters:

[out]	cVector	Receives the vector elements
[in]	sStart	Open bracket (e.g. "[")
[in]	sEnd	Closed bracket (e.g. "]")
[in]	sSeparator	Separator between the elements (e.g. ","). Can also be ""

Returns:: 'true' on success, else 'false'

Remarks:: Example: [1.0, 2.1, 3.2]

Note:

If a vector has been found, the tokenizer goes to the next token in the stream, otherwise it stays at the current token

PLCORE_API bool PLCore::Tokenizer::ParseVector	(	Array< double > &	cVector,
		const String &	sStart = `"["`,
		const String &	sEnd = `"]"`,
		const String &	sSeparator = `","`
	)

Expects the next tokens to be a vector and returns it as an array of doubles.

Parameters:

[out]	cVector	Receives the vector elements
[in]	sStart	Open bracket (e.g. "[")
[in]	sEnd	Closed bracket (e.g. "]")
[in]	sSeparator	Separator between the elements (e.g. ","). Can also be ""

Returns:: 'true' on success, else 'false'

Remarks:: Example: [1.0, 2.1, 3.2]

Note:

If a vector has been found, the tokenizer goes to the next token in the stream, otherwise it stays at the current token

PLCORE_API bool PLCore::Tokenizer::ParseEquation	(	String &	sName,
		String &	sValue,
		const String &	sEquation = `""`
	)

Expects the next tokens to be an equation and returns it.

Parameters:

[out]	sName	Name of the element
[out]	sValue	Value as a string
[in]	sEquation	Equation sign (e.g. "=")

Returns:: 'true' on success, else 'false'

Remarks:: Example: Console = On

Note:

If an equation has been found, the tokenizer goes to the next token in the stream, otherwise it stays at the current token

bool PLCore::Tokenizer::ParseEquation	(	String &	sName,
		int &	nValue,
		const String &	sEquation = `""`
	)		`[inline]`

Expects the next tokens to be an equation and returns it.

Parameters:

[out]	sName	Name of the element
[out]	nValue	Value as an int
[in]	sEquation	Equation sign (e.g. "=")

Returns:: 'true' on success, else 'false'

Remarks:: Example: Health = 100

Note:

If an equation has been found, the tokenizer goes to the next token in the stream, otherwise it stays at the current token

bool PLCore::Tokenizer::ParseEquation	(	String &	sName,
		float &	fValue,
		const String &	sEquation = `""`
	)		`[inline]`

Expects the next tokens to be an equation and returns it.

Parameters:

[out]	sName	Name of the element
[out]	fValue	Value as a float
[in]	sEquation	Equation sign (e.g. "=")

Returns:: 'true' on success, else 'false'

Remarks:: Example: Gamma = 2.1

Note:

If an equation has been found, the tokenizer goes to the next token in the stream, otherwise it stays at the current token

bool PLCore::Tokenizer::ParseEquation	(	String &	sName,
		double &	dValue,
		const String &	sEquation = `""`
	)		`[inline]`

Expects the next tokens to be an equation and returns it.

Parameters:

[out]	sName	Name of the element
[out]	dValue	Value as a double
[in]	sEquation	Equation sign (e.g. "=")

Returns:: 'true' on success, else 'false'

Remarks:: Example: Speed = 3.25

Note:

If an equation has been found, the tokenizer goes to the next token in the stream, otherwise it stays at the current token

The documentation for this class was generated from the following files:

Public Member Functions

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation