QIconvCodec Class Reference

#include <qiconvcodec_p.h>

Inheritance diagram for QIconvCodec:

Classes
class	IconvState

Public Functions
QByteArray	convertFromUnicode (const QChar *, int, ConverterState *) const
	QTextCodec subclasses must reimplement this function. More...
QString	convertToUnicode (const char *, int, ConverterState *) const
	QTextCodec subclasses must reimplement this function. More...
int	mibEnum () const
	Subclasses of QTextCodec must reimplement this function. More...
QByteArray	name () const
	QTextCodec subclasses must reimplement this function. More...
	QIconvCodec ()
	~QIconvCodec ()
Public Functions inherited from QTextCodec
virtual QList< QByteArray >	aliases () const
	Subclasses can return a number of aliases for the codec in question. More...
bool	canEncode (QChar) const
	Returns true if the Unicode character ch can be fully encoded with this codec; otherwise returns false. More...
bool	canEncode (const QString &) const
	This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.s contains the string being tested for encode-ability. More...
QByteArray	fromUnicode (const QString &uc) const
	Converts str from Unicode to the encoding of this codec, and returns the result in a QByteArray. More...
QByteArray	fromUnicode (const QChar *in, int length, ConverterState *state=0) const
	Converts the first number of characters from the input array from Unicode to the encoding of this codec, and returns the result in a QByteArray. More...
QTextDecoder *	makeDecoder () const
	Creates a QTextDecoder which stores enough state to decode chunks of char * data to create chunks of Unicode data. More...
QTextDecoder *	makeDecoder (ConversionFlags flags) const
QTextEncoder *	makeEncoder () const
	Creates a QTextEncoder which stores enough state to encode chunks of Unicode data as char * data. More...
QTextEncoder *	makeEncoder (ConversionFlags flags) const
QString	toUnicode (const QByteArray &) const
	Converts a from the encoding of this codec to Unicode, and returns the result in a QString. More...
QString	toUnicode (const char *chars) const
	This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.chars contains the source characters. More...
QString	toUnicode (const char *in, int length, ConverterState *state=0) const
	Converts the first size characters from the input from the encoding of this codec to Unicode, and returns the result in a QString. More...

Static Public Functions
static iconv_t	createIconv_t (const char *to, const char *from)
Static Public Functions inherited from QTextCodec
static QList< QByteArray >	availableCodecs ()
	Returns the list of all available codecs, by name. More...
static QList< int >	availableMibs ()
	Returns the list of MIBs for all available codecs. More...
static QTextCodec *	codecForCStrings ()
	Returns the codec used by QString to convert to and from const char * and QByteArrays. More...
static QTextCodec *	codecForHtml (const QByteArray &ba)
	Tries to detect the encoding of the provided snippet of HTML in the given byte array, ba, by checking the BOM (Byte Order Mark) and the content-type meta header and returns a QTextCodec instance that is capable of decoding the html to unicode. More...
static QTextCodec *	codecForHtml (const QByteArray &ba, QTextCodec *defaultCodec)
	Tries to detect the encoding of the provided snippet of HTML in the given byte array, ba, by checking the BOM (Byte Order Mark) and the content-type meta header and returns a QTextCodec instance that is capable of decoding the html to unicode. More...
static QTextCodec *	codecForLocale ()
	Returns a pointer to the codec most suitable for this locale. More...
static QTextCodec *	codecForMib (int mib)
	Returns the QTextCodec which matches the MIBenum mib. More...
static QTextCodec *	codecForName (const QByteArray &name)
	Searches all installed QTextCodec objects and returns the one which best matches name; the match is case-insensitive. More...
static QTextCodec *	codecForName (const char *name)
	Searches all installed QTextCodec objects and returns the one which best matches name; the match is case-insensitive. More...
static QTextCodec *	codecForTr ()
	Returns the codec used by QObject::tr() on its argument. More...
static QTextCodec *	codecForUtfText (const QByteArray &ba)
	Tries to detect the encoding of the provided snippet ba by using the BOM (Byte Order Mark) and returns a QTextCodec instance that is capable of decoding the text to unicode. More...
static QTextCodec *	codecForUtfText (const QByteArray &ba, QTextCodec *defaultCodec)
	Tries to detect the encoding of the provided snippet ba by using the BOM (Byte Order Mark) and returns a QTextCodec instance that is capable of decoding the text to unicode. More...
static void	setCodecForCStrings (QTextCodec *c)
static void	setCodecForLocale (QTextCodec *c)
	Set the codec to c; this will be returned by codecForLocale(). More...
static void	setCodecForTr (QTextCodec *c)

Properties
QTextCodec *	utf16Codec

Additional Inherited Members
Public Types inherited from QTextCodec
enum	ConversionFlag { DefaultConversion, ConvertInvalidToNull = 0x80000000, IgnoreHeader = 0x1, FreeFunction = 0x2 }
Protected Functions inherited from QTextCodec
	QTextCodec ()
	Constructs a QTextCodec, and gives it the highest precedence. More...
virtual	~QTextCodec ()
	Destroys the QTextCodec. More...

Detailed Description

Definition at line 68 of file qiconvcodec_p.h.

Constructors and Destructors

QIconvCodec()

QIconvCodec::QIconvCodec

(

)

Definition at line 93 of file qiconvcodec.cpp.

    : utf16Codec(0)
{
    utf16Codec = QTextCodec::codecForMib(1015);
    Q_ASSERT_X(utf16Codec != 0,
               "QIconvCodec::convertToUnicode",
               "internal error, UTF-16 codec not found");
    if (!utf16Codec) {
        fprintf(stderr, "QIconvCodec::convertToUnicode: internal error, UTF-16 codec not found\n");
        utf16Codec = reinterpret_cast<QTextCodec *>(~0);
    }
#if defined(Q_OS_MAC)
    if (ptr_iconv_open == 0) {
        QLibrary libiconv(QLatin1String("/usr/lib/libiconv"));
        libiconv.setLoadHints(QLibrary::ExportExternalSymbolsHint);

        ptr_iconv_open = reinterpret_cast<Ptr_iconv_open>(libiconv.resolve("libiconv_open"));
        if (!ptr_iconv_open)
            ptr_iconv_open = reinterpret_cast<Ptr_iconv_open>(libiconv.resolve("iconv_open"));
        ptr_iconv = reinterpret_cast<Ptr_iconv>(libiconv.resolve("libiconv"));
        if (!ptr_iconv)
            ptr_iconv = reinterpret_cast<Ptr_iconv>(libiconv.resolve("iconv"));
        ptr_iconv_close = reinterpret_cast<Ptr_iconv_close>(libiconv.resolve("libiconv_close"));
        if (!ptr_iconv_close)
            ptr_iconv_close = reinterpret_cast<Ptr_iconv_close>(libiconv.resolve("iconv_close"));

        Q_ASSERT_X(ptr_iconv_open && ptr_iconv && ptr_iconv_close,
        "QIconvCodec::QIconvCodec()",
        "internal error, could not resolve the iconv functions");

#       undef iconv_open
#       define iconv_open ptr_iconv_open
#       undef iconv
#       define iconv ptr_iconv
#       undef iconv_close
#       define iconv_close ptr_iconv_close
    }
#endif
}

~QIconvCodec()

QIconvCodec::~QIconvCodec

(

)

Definition at line 133 of file qiconvcodec.cpp.

{
}

Functions

convertFromUnicode()

QByteArray QIconvCodec::convertFromUnicode ( const QChar *  input, int  number, ConverterState *  state  ) const

virtual

QTextCodec subclasses must reimplement this function.

Converts the first number of characters from the input array from Unicode to the encoding of the subclass, and returns the result in a QByteArray.

state can be 0 in which case the conversion is stateless and default conversion rules should be used. If state is not 0, the codec should save the state after the conversion in state, and adjust the remainingChars and invalidChars members of the struct.

Implements QTextCodec.

Definition at line 335 of file qiconvcodec.cpp.

{
    char *inBytes;
    char *outBytes;
    size_t inBytesLeft;

#if defined(GNU_LIBICONV)
    const char **inBytesPtr = const_cast<const char **>(&inBytes);
#else
    char **inBytesPtr = &inBytes;
#endif

    IconvState *temporaryState = 0;
    QThreadStorage<QIconvCodec::IconvState *> *ts = fromUnicodeState();
    IconvState *&state = (qt_locale_initialized && ts) ? ts->localData() : temporaryState;
    if (!state) {
        iconv_t cd = QIconvCodec::createIconv_t(0, UTF16);
        if (cd != reinterpret_cast<iconv_t>(-1)) {
            if (!setByteOrder(cd)) {
                perror("QIconvCodec::convertFromUnicode: using Latin-1 for conversion, iconv failed for BOM");

                iconv_close(cd);
                cd = reinterpret_cast<iconv_t>(-1);

                return QString(uc, len).toLatin1();
            }
        }
        state = new IconvState(cd);
    }
    if (state->cd == reinterpret_cast<iconv_t>(-1)) {
        static int reported = 0;
        if (!reported++) {
            fprintf(stderr,
                    "QIconvCodec::convertFromUnicode: using Latin-1 for conversion, iconv_open failed\n");
        }
        delete temporaryState;
        return QString(uc, len).toLatin1();
    }
 
    size_t outBytesLeft = len;
    QByteArray ba(outBytesLeft, Qt::Uninitialized);
    outBytes = ba.data();

    // now feed iconv() the real data
    inBytes = const_cast<char *>(reinterpret_cast<const char *>(uc));
    inBytesLeft = len * sizeof(QChar);

    QByteArray in;
    if (convState && convState->remainingChars) {
        // we have one surrogate char to be prepended
        in.resize(sizeof(QChar) + len);
        inBytes = in.data();

        QChar remaining = convState->state_data[0];
        memcpy(in.data(), &remaining, sizeof(QChar));
        memcpy(in.data() + sizeof(QChar), uc, inBytesLeft);

        inBytesLeft += sizeof(QChar);
        convState->remainingChars = 0;
    }

    int invalidCount = 0;
    while (inBytesLeft != 0) {
        if (iconv(state->cd, inBytesPtr, &inBytesLeft, &outBytes, &outBytesLeft) == (size_t) -1) {
            if (errno == EINVAL && convState) {
                // buffer ends in a surrogate
                Q_ASSERT(inBytesLeft == 2);
                convState->remainingChars = 1;
                convState->state_data[0] = uc[len - 1].unicode();
                break;
            }

            switch (errno) {
            case EILSEQ:
                ++invalidCount;
                // fall through
            case EINVAL:
                {
                    inBytes += sizeof(QChar);
                    inBytesLeft -= sizeof(QChar);
                    break;
                }
            case E2BIG:
                {
                    int offset = ba.size() - outBytesLeft;
                    ba.resize(ba.size() * 2);
                    outBytes = ba.data() + offset;
                    outBytesLeft = ba.size() - offset;
                    break;
                }
            default:
                {
                    // note, cannot use qWarning() since we are implementing the codecForLocale :)
                    perror("QIconvCodec::convertFromUnicode: using Latin-1 for conversion, iconv failed");

                    // reset to initial state
                    iconv(state->cd, 0, &inBytesLeft, 0, &outBytesLeft);

                    delete temporaryState;
                    return QString(uc, len).toLatin1();
                }
            }
        }
    }

    // reset to initial state
    iconv(state->cd, 0, &inBytesLeft, 0, &outBytesLeft);
    setByteOrder(state->cd);

    ba.resize(ba.size() - outBytesLeft);

    if (convState)
        convState->invalidChars = invalidCount;

    delete temporaryState;
    return ba;
}

convertToUnicode()

QString QIconvCodec::convertToUnicode ( const char *  chars, int  len, ConverterState *  state  ) const

virtual

QTextCodec subclasses must reimplement this function.

Converts the first len characters of chars from the encoding of the subclass to Unicode, and returns the result in a QString.

state can be 0, in which case the conversion is stateless and default conversion rules should be used. If state is not 0, the codec should save the state after the conversion in state, and adjust the remainingChars and invalidChars members of the struct.

Implements QTextCodec.

Definition at line 168 of file qiconvcodec.cpp.

{
    if (utf16Codec == reinterpret_cast<QTextCodec *>(~0))
        return QString::fromLatin1(chars, len);

    int invalidCount = 0;
    int remainingCount = 0;
    char *remainingBuffer = 0;
    IconvState *temporaryState = 0;
    IconvState **pstate;

    if (convState) {
        // stateful conversion
        pstate = reinterpret_cast<IconvState **>(&convState->d);
        if (convState->d) {
            // restore state
            remainingCount = convState->remainingChars;
            remainingBuffer = (*pstate)->buffer;
        } else {
            // first time
            convState->flags |= FreeFunction;
            QTextCodecUnalignedPointer::encode(convState->state_data, qIconvCodecStateFree);
        }
    } else {
        QThreadStorage<QIconvCodec::IconvState *> *ts = toUnicodeState();
        if (!qt_locale_initialized || !ts) {
            // we're running after the Q_GLOBAL_STATIC has been deleted
            // or before the QCoreApplication initialization
            // bad programmer, no cookie for you
            pstate = &temporaryState;
        } else {
            // stateless conversion -- use thread-local data
            pstate = &toUnicodeState()->localData();
        }
    }

    if (!*pstate) {
        // first time, create the state
        iconv_t cd = QIconvCodec::createIconv_t(UTF16, 0);
        if (cd == reinterpret_cast<iconv_t>(-1)) {
            static int reported = 0;
            if (!reported++) {
                fprintf(stderr,
                        "QIconvCodec::convertToUnicode: using Latin-1 for conversion, iconv_open failed\n");
            }
            return QString::fromLatin1(chars, len);
        }

        *pstate = new IconvState(cd);
    }

    IconvState *state = *pstate;
    size_t inBytesLeft = len;
    // best case assumption, each byte is converted into one UTF-16 character, plus 2 bytes for the BOM
#ifdef GNU_LIBICONV
    // GNU doesn't disagree with POSIX :/
    const char *inBytes = chars;
#else
    char *inBytes = const_cast<char *>(chars);
#endif

    QByteArray in;
    if (remainingCount) {
        // we have to prepend the remaining bytes from the previous conversion
        inBytesLeft += remainingCount;
        in.resize(inBytesLeft);
        inBytes = in.data();

        memcpy(in.data(), remainingBuffer, remainingCount);
        memcpy(in.data() + remainingCount, chars, len);

        remainingCount = 0;
    }

    size_t outBytesLeft = len * 2 + 2;
    QByteArray ba(outBytesLeft, Qt::Uninitialized);
    char *outBytes = ba.data();
    do {
        size_t ret = iconv(state->cd, &inBytes, &inBytesLeft, &outBytes, &outBytesLeft);
        if (ret == (size_t) -1) {
            if (errno == E2BIG) {
                int offset = ba.size() - outBytesLeft;
                ba.resize(ba.size() * 2);
                outBytes = ba.data() + offset;
                outBytesLeft = ba.size() - offset;

                continue;
            }

            if (errno == EILSEQ) {
                // conversion stopped because of an invalid character in the sequence
                ++invalidCount;
            } else if (errno == EINVAL && convState) {
                // conversion stopped because the remaining inBytesLeft make up
                // an incomplete multi-byte sequence; save them for later
                state->saveChars(inBytes, inBytesLeft);
                remainingCount = inBytesLeft;
                break;
            }

            if (errno == EILSEQ || errno == EINVAL) {
                // skip the next character
                ++inBytes;
                --inBytesLeft;
                continue;
            }

            // some other error
            // note, cannot use qWarning() since we are implementing the codecForLocale :)
            perror("QIconvCodec::convertToUnicode: using Latin-1 for conversion, iconv failed");

            if (!convState) {
                // reset state
                iconv(state->cd, 0, &inBytesLeft, 0, &outBytesLeft);
            }

            delete temporaryState;
            return QString::fromLatin1(chars, len);
        }
    } while (inBytesLeft != 0);

    QString s;

    if (convState) {
        s = utf16Codec->toUnicode(ba.constData(), ba.size() - outBytesLeft, &state->internalState);

        convState->invalidChars = invalidCount;
        convState->remainingChars = remainingCount;
    } else {
        s = utf16Codec->toUnicode(ba.constData(), ba.size() - outBytesLeft);

        // reset state
        iconv(state->cd, 0, &inBytesLeft, 0, &outBytesLeft);
    }

    delete temporaryState;
    return s;
}

createIconv_t()

iconv_t QIconvCodec::createIconv_t ( const char *  to, const char *  from  )

static

Definition at line 463 of file qiconvcodec.cpp.

Referenced by convertFromUnicode(), and convertToUnicode().

{
    Q_ASSERT((to == 0 && from != 0) || (to != 0 && from == 0));

    iconv_t cd = (iconv_t) -1;
#if defined(__GLIBC__) || defined(GNU_LIBICONV) || defined(Q_OS_QNX)
#if defined(Q_OS_QNX)
    // on QNX the default locale is UTF-8, and an empty string will cause iconv_open to fail
    static const char empty_codeset[] = "UTF-8";
#else
    // both GLIBC and libgnuiconv will use the locale's encoding if from or to is an empty string
    static const char empty_codeset[] = "";
#endif
    const char *codeset = empty_codeset;
    cd = iconv_open(to ? to : codeset, from ? from : codeset);
#else
    char *codeset = 0;
#endif

#if defined(_XOPEN_UNIX) && !defined(Q_OS_QNX) && !defined(Q_OS_OSF)
    if (cd == (iconv_t) -1) {
        codeset = nl_langinfo(CODESET);
        if (codeset)
            cd = iconv_open(to ? to : codeset, from ? from : codeset);
    }
#endif

    if (cd == (iconv_t) -1) {
        // Very poorly defined and followed standards causes lots of
        // code to try to get all the cases... This logic is
        // duplicated in QTextCodec, so if you change it here, change
        // it there too.

        // Try to determine locale codeset from locale name assigned to
        // LC_CTYPE category.

        // First part is getting that locale name.  First try setlocale() which
        // definitely knows it, but since we cannot fully trust it, get ready
        // to fall back to environment variables.
        char * ctype = qstrdup(setlocale(LC_CTYPE, 0));

        // Get the first nonempty value from $LC_ALL, $LC_CTYPE, and $LANG
        // environment variables.
        char * lang = qstrdup(qgetenv("LC_ALL").constData());
        if (!lang || lang[0] == 0 || strcmp(lang, "C") == 0) {
            if (lang) delete [] lang;
            lang = qstrdup(qgetenv("LC_CTYPE").constData());
        }
        if (!lang || lang[0] == 0 || strcmp(lang, "C") == 0) {
            if (lang) delete [] lang;
            lang = qstrdup(qgetenv("LANG").constData());
        }

        // Now try these in order:
        // 1. CODESET from ctype if it contains a .CODESET part (e.g. en_US.ISO8859-15)
        // 2. CODESET from lang if it contains a .CODESET part
        // 3. ctype (maybe the locale is named "ISO-8859-1" or something)
        // 4. locale (ditto)
        // 5. check for "@euro"

        // 1. CODESET from ctype if it contains a .CODESET part (e.g. en_US.ISO8859-15)
        codeset = ctype ? strchr(ctype, '.') : 0;
        if (codeset && *codeset == '.') {
            ++codeset;
            cd = iconv_open(to ? to : codeset, from ? from : codeset);
        }

        // 2. CODESET from lang if it contains a .CODESET part
        codeset = lang ? strchr(lang, '.') : 0;
        if (cd == (iconv_t) -1 && codeset && *codeset == '.') {
            ++codeset;
            cd = iconv_open(to ? to : codeset, from ? from : codeset);
        }

        // 3. ctype (maybe the locale is named "ISO-8859-1" or something)
        if (cd == (iconv_t) -1 && ctype && *ctype != 0 && strcmp (ctype, "C") != 0)
            cd = iconv_open(to ? to : ctype, from ? from : ctype);


        // 4. locale (ditto)
        if (cd == (iconv_t) -1 && lang && *lang != 0)
            cd = iconv_open(to ? to : lang, from ? from : lang);

        // 5. "@euro"
        if ((cd == (iconv_t) -1 && ctype && strstr(ctype, "@euro")) || (lang && strstr(lang, "@euro")))
            cd = iconv_open(to ? to : "ISO8859-15", from ? from : "ISO8859-15");

        delete [] ctype;
        delete [] lang;
    }

    return cd;
}

mibEnum()

int QIconvCodec::mibEnum ( ) const

virtual

Subclasses of QTextCodec must reimplement this function.

It returns the MIBenum (see IANA character-sets encoding file for more information). It is important that each QTextCodec subclass returns the correct unique value for this function.

Implements QTextCodec.

Definition at line 458 of file qiconvcodec.cpp.

{
    return 0;
}

name()

QByteArray QIconvCodec::name ( ) const

virtual

QTextCodec subclasses must reimplement this function.

It returns the name of the encoding supported by the subclass.

If the codec is registered as a character set in the IANA character-sets encoding file this method should return the preferred mime name for the codec if defined, otherwise its name.

Implements QTextCodec.

Definition at line 453 of file qiconvcodec.cpp.

{
    return "System";
}

Properties

utf16Codec

QTextCodec* QIconvCodec::utf16Codec

mutableprivate

Definition at line 71 of file qiconvcodec_p.h.

Referenced by convertToUnicode(), and QIconvCodec().

---

The documentation for this class was generated from the following files:

• /src/corelib/codecs/qiconvcodec_p.h
• /src/corelib/codecs/qiconvcodec.cpp