QPatternist::Tokenizer Class Reference

Base class for all tokenizers. More...

#include <qtokenizer_p.h>

Inheritance diagram for QPatternist::Tokenizer:

Public Types
typedef QExplicitlySharedDataPointer< Tokenizer >	Ptr
Public Types inherited from QPatternist::TokenSource
typedef QExplicitlySharedDataPointer< TokenSource >	Ptr
typedef QQueue< Ptr >	Queue
typedef yytokentype	TokenType

Public Functions
virtual int	commenceScanOnly ()=0
const QUrl &	queryURI () const
virtual void	resumeTokenizationFrom (const int position)=0
virtual void	setParserContext (const ParserContext::Ptr &parseInfo)=0
	Tokenizer (const QUrl &queryU)
Public Functions inherited from QPatternist::TokenSource
virtual Token	nextToken (YYLTYPE *const sourceLocator)=0
	TokenSource ()
virtual	~TokenSource ()
Public Functions inherited from QSharedData
	QSharedData ()
	Constructs a QSharedData object with a reference count of 0. More...
	QSharedData (const QSharedData &)
	Constructs a QSharedData object with reference count 0. More...

Static Protected Functions
static QString	tokenToString (const Token &token)

Properties
const QUrl	m_queryURI

Additional Inherited Members
Public Variables inherited from QSharedData
QAtomicInt	ref

Detailed Description

Base class for all tokenizers.

The main entry point is nextToken(), which ones calls to retrieve the stream of tokens this Tokenizer delivers.

See also

Building a Tokenizer for XPath or XQuery

Author

Frans Englich frans.nosp@m..eng.nosp@m.lich@.nosp@m.noki.nosp@m.a.com

Definition at line 155 of file qtokenizer_p.h.

Typedefs

Ptr

typedef QExplicitlySharedDataPointer<Tokenizer> QPatternist::Tokenizer::Ptr

Definition at line 163 of file qtokenizer_p.h.

Constructors and Destructors

Tokenizer()

QPatternist::Tokenizer::Tokenizer ( const QUrl &  queryU )

inline

Definition at line 158 of file qtokenizer_p.h.

                                             : m_queryURI(queryU)
        {
            Q_ASSERT(queryU.isValid());
        }

Functions

commenceScanOnly()

virtual int QPatternist::Tokenizer::commenceScanOnly ( )

pure virtual

Switches the Tokenizer to only do scanning, and returns complete strings for attribute value templates as opposed to the tokens for the contained expressions.

The current position in the stream is returned. It can be used to later resume regular tokenization.

Implemented in QPatternist::XSLTTokenizer, QPatternist::XQueryTokenizer, and QPatternist::TokenRevealer.

queryURI()

const QUrl& QPatternist::Tokenizer::queryURI ( ) const

inline

Returns

the URI of the resource being tokenized.

Definition at line 187 of file qtokenizer_p.h.

Referenced by QPatternist::fromYYLTYPE(), QPatternist::XSLTTokenizer::queueAVT(), QPatternist::XSLTTokenizer::queueExpression(), and QPatternist::XSLTTokenizer::queueSequenceType().

        {
            return m_queryURI;
        }

resumeTokenizationFrom()

virtual void QPatternist::Tokenizer::resumeTokenizationFrom ( const int  position )

pure virtual

Resumes regular parsing from position. The tokenizer must be in the scan-only state, which the commenceScanOnly() call transists to.

The tokenizer will return the token POSITION_SET once after this function has been called.

Implemented in QPatternist::XSLTTokenizer, QPatternist::XQueryTokenizer, and QPatternist::TokenRevealer.

setParserContext()

virtual void QPatternist::Tokenizer::setParserContext ( const ParserContext::Ptr &  parseInfo )

pure virtual

Implemented in QPatternist::XSLTTokenizer, QPatternist::XQueryTokenizer, and QPatternist::TokenRevealer.

tokenToString()

QString QPatternist::Tokenizer::tokenToString ( const Token &  token )

staticprotected

Returns a string representation of token.

This function is used for debugging purposes. The implementation of this function is in querytransformparser.ypp.

Definition at line 8003 of file qquerytransformparser.cpp.

Referenced by QPatternist::TokenRevealer::nextToken().

{
    switch(token.type)
    {
        case NCNAME:
        /* Fallthrough. */
        case QNAME:
        /* Fallthrough. */
        case NUMBER:
        /* Fallthrough. */
        case XPATH2_NUMBER:
            return token.value;
        case STRING_LITERAL:
            return QLatin1Char('"') + token.value + QLatin1Char('"');
        default:
        {
            const QString raw(QString::fromLatin1(yytname[YYTRANSLATE(token.type)]));

            /* Remove the quotes. */
            if(raw.at(0) == QLatin1Char('"') && raw.length() > 1)
                return raw.mid(1, raw.length() - 2);
            else
                return raw;
        }
    }
}

Properties

m_queryURI

const QUrl QPatternist::Tokenizer::m_queryURI

private

Definition at line 205 of file qtokenizer_p.h.

---

The documentation for this class was generated from the following files:

• /src/xmlpatterns/parser/qtokenizer_p.h
• /src/xmlpatterns/parser/qquerytransformparser.cpp