How to write a Russian text in console?

[ChrisW67]

Back to first principles. What does this output:

#include <QtCore>
int main(int argc, char **argv)
{
    QCoreApplication app(argc, argv);
    QString test1 = QString::fromUtf8("\u0623\u062F\u062E\u0644 \u0631\u0645\u0632");
    QString test2= QString::fromUtf8("\u041F\u0440\u0438\u0432\u0435\u0442 \u041C\u0438\u0440");
    qDebug() << test1;
    qDebug() << test2;
    return 0;
}

#include <QtCore>
int main(int argc, char **argv)
{
    QCoreApplication app(argc, argv);
    QString test1 = QString::fromUtf8("\u0623\u062F\u062E\u0644 \u0631\u0645\u0632");
    QString test2= QString::fromUtf8("\u041F\u0440\u0438\u0432\u0435\u0442 \u041C\u0438\u0440");
    qDebug() << test1;
    qDebug() << test2;
    return 0;
}

To copy to clipboard, switch view to plain text mode 

For both Qt 4.8.5 and Qt 5.1.1 on Linux this puts the correct characters on the screen with a suitable font and bi-directional text rendering.

I know the input encoding is correct so "?" output is indicative of a lacking font or terminal that cannot cope with the encoded out.

[toufic.dbouk]

Trying this :

#include <QtCore>
int main()
{
    QString test1 = QString::fromUtf8("\u0623\u062F\u062E\u0644 \u0631\u0645\u0632");
    QString test2= QString::fromUtf8("\u041F\u0440\u0438\u0432\u0435\u0442 \u041C\u0438\u0440");
    qDebug() << test1;
    qDebug() << test2;
    return 0;
}

#include <QtCore>
int main()
{
    QString test1 = QString::fromUtf8("\u0623\u062F\u062E\u0644 \u0631\u0645\u0632");
    QString test2= QString::fromUtf8("\u041F\u0440\u0438\u0432\u0435\u0442 \u041C\u0438\u0440");
    qDebug() << test1;
    qDebug() << test2;
    return 0;
}

To copy to clipboard, switch view to plain text mode 

outputs : "?" with some warnings:C4566: character represented by universal-character-name '\u0623' cannot be represented in the current code page(1252)
So my system cant output such encoding ?

[ChrisW67]

Ignore the compiler warning for the time being.
Is the output from running the program all ? Characters, or just the arabic ones?
Is the result different if you set the console font to lucida and issue a "chcp 65001" command before running the program?

[toufic.dbouk]

Is the output from running the program all ? Characters, or just the arabic ones?

Just the Arabic ones, English characters outputs normal as alphabets ( if that's what you mean ).

Is the result different if you set the console font to lucida and issue a "chcp 65001" command before running the program?

How to do that ? should i compile and run the .pro or .cpp file from cmd ? can i open a Qt's cmd where it has Qt's libraries added to the cmd's environment?
Thanks for Your help.

[ChrisW67]

The first string is Arabic characters, the second is Cyrillic characters. This is what I see when I run it from a command prompt on Linux:

chrisw@newton /tmp/tt $ ./tt
"أدخل رمز"
"Привет Мир"

chrisw@newton /tmp/tt $ ./tt
"أدخل رمز"
"Привет Мир"

To copy to clipboard, switch view to plain text mode 

I just tried my example with VS 2010 on Win 7 and get a similar result to you: question marks all round. There seem to be several different problems:

• The compiler is mangling the UTF8 input by trying to interpret it in the local 8-bit encoding. The warnings are issued in the process and the UTF-8 encoded text doesn't make it into the object file intact.
• When I find a way to avoid that first mangling (by using hex directly) the result is still mangled on output.

I will try some more experiments over the weekend

[8Observer8]

ChrisW67, I want to understand that, but I don't. I read about UTF-8. How professionals solve this problem? Maybe you use "Internationalization with Qt":
- http://qt-project.org/doc/qt-5.0/qtd...alization.html
- http://qt-project.org/doc/qt-5.0/qtq...alization.html

How you do it in professional applications?

[ChrisW67]

I will try to explain what is happening. Start with this line:

QString test = QString::fromUtf8("\u041F\u0440\u0438\u0432\u0435\u0442 \u041C\u0438\u0440");

QString test = QString::fromUtf8("\u041F\u0440\u0438\u0432\u0435\u0442 \u041C\u0438\u0440");

To copy to clipboard, switch view to plain text mode 

This is what happens on Linux with GCC.

The compiler sees the \u041f and inserts the UTF8 encoded version of the U+041f character (П) into the string. That is two bytes 0xD0 and 0x9F. It does this for the whole string. The result is a C-style string of bytes (in hex) that is the UTF8 encoded string:

D0 9F D1 80 D0 B8 D0 B2 D0 B5 D1 82 20 D0 9C D0 B8 D1 80

D0 9F D1 80 D0 B8 D0 B2 D0 B5 D1 82 20 D0 9C D0 B8 D1 80

To copy to clipboard, switch view to plain text mode 

We feed that into fromUtf8() and we get a valid QString with the correct characters. When the Linux program executes, qDebug() outputs the QString correctly encoded for my UTF8 terminal and I get the expected characters on screen. The same goes for QLabel.


On Windows with MS VC++ (2010):

The compiler sees the \u041f and and tries to map the U+041f character (П) to the system's 8-bit Windows code page before putting it in the string. Unless your system code page is Windows-1251 there is not likely to be an equivalent of П and the compiler inserts ? as a placeholder for the character it could not convert. The compiler issues a warning:

warning C4566: character represented by universal-character-name '\u041F' cannot be represented in the current code page (1252)

warning C4566: character represented by universal-character-name '\u041F' cannot be represented in the current code page (1252)

To copy to clipboard, switch view to plain text mode 

It does this for the whole string. The result is a C-style string of bytes (in hex) that is not at all what you were expecting:

3F 3F 3F 3F 3F 3F 20 3F 3F 3F

3F 3F 3F 3F 3F 3F 20 3F 3F 3F

To copy to clipboard, switch view to plain text mode 

We feed that into fromUtf8() and we get a valid QString but not the correct characters. qDebug() and QLabel cannot give the expected output now. This compiler behaviour seems to be the same regardless of what encoding the input file is or whether it has a UTF8 byte-order-mark or not.

If I change the line to:

QString test = QString::fromUtf8("\xD0\x9F\xD1\x80\xD0\xB8\xD0\xB2\xD0\xB5\xD1\x82\x20\xD0\x9C\xD0\xB8\xD1\x80");

QString test = QString::fromUtf8("\xD0\x9F\xD1\x80\xD0\xB8\xD0\xB2\xD0\xB5\xD1\x82\x20\xD0\x9C\xD0\xB8\xD1\x80");

To copy to clipboard, switch view to plain text mode 

I have done the UTF8 encoding and avoid the compiler's attempt to map the characters to the Windows code page.
If I put that string on a QLabel I see the correct characters (font permitting): the data made it in.
The qDebug() output in the console is still wrong because the QString is being mapped (again) to the local 8-bit code page with the same "?" result.

I get Cyrillic output in a CMD console with:

#include <QApplication>
#include <QLabel>
#include <QDebug>
#include <QTextCodec>
 
int main(int argc, char **argv)
{
    QApplication app(argc, argv);
    QTextCodec::setCodecForLocale(QTextCodec::codecForName("utf8"));
    QString test = QString::fromUtf8("\xD0\x9F\xD1\x80\xD0\xB8\xD0\xB2\xD0\xB5\xD1\x82\x20\xD0\x9C\xD0\xB8\xD1\x80");
    qDebug() << test.toLocal8Bit();
    QLabel l(test);
    l.show();
    return app.exec();
}

#include <QApplication>
#include <QLabel>
#include <QDebug>
#include <QTextCodec>

int main(int argc, char **argv)
{
    QApplication app(argc, argv);
    QTextCodec::setCodecForLocale(QTextCodec::codecForName("utf8"));
    QString test = QString::fromUtf8("\xD0\x9F\xD1\x80\xD0\xB8\xD0\xB2\xD0\xB5\xD1\x82\x20\xD0\x9C\xD0\xB8\xD1\x80");
    qDebug() << test.toLocal8Bit();
    QLabel l(test);
    l.show();
    return app.exec();
}

To copy to clipboard, switch view to plain text mode 

If:

• I run the program from a Windows CMD shell, and
• set the shell font to "Lucida Console", and
• I execute "chcp 65001" before I run the program.

Manually doing UTF8 encoding is not a good solution, and I do not yet have a nice solution.
There is a hotfix for VC 2010 http://stackoverflow.com/questions/6...er-arrays-in-c but it seems that did not make it into 2012 and 2013.

[toufic.dbouk]

That was a good explanation. Thanks.
Just a notice here, i tried printing

std::cout << "أدخل رمز";

std::cout << "أدخل رمز";

To copy to clipboard, switch view to plain text mode 

on VS 2012, but i got ?? as output.
Saving the file with cp 1252 ( Western ) also results in ?? obviously.
Saving the file with UTF-8 cp 65001 results in some weird characters.
Saving the file with cp 1256 ( Arabic ) results in weird characters too.
That is all under Visual Studio 2012.

[8Observer8]

I was helped here: http://www.prog.org.ru/topic_26545_0.html

This is the solution:

#include <QCoreApplication>
#include <QDebug>
#include <QTextCodec>
#include <QTextStream>
#include <iostream>
 
QTextStream cin(stdin);
QTextStream cout(stdout);
 
int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);
 
    QString string = "Привет, Мир!";
    QTextCodec *codec = QTextCodec::codecForName("CP866");
 
    // std::cout
    QByteArray encodedString = codec->fromUnicode(string);
    std::cout << "std::cout = " << encodedString.data() << std::endl;
 
    // TextStream
    cout.setCodec(codec);
    cout << "TextStream = " << string << "\n";
    cout.flush();
 
    // qDebug
    QTextCodec::setCodecForLocale(codec);
    qDebug() << "qDebug() = " <<  string;
 
     // Console r/w
    cout << "Enter a text: ";
    cout.flush();
    cin.setCodec(codec);
    QString inputStr;
    // Read data from a console
    cin >> inputStr;
    cout << "From a console = " << inputStr << endl;
    cout.flush();
 
    return a.exec();
}

#include <QCoreApplication>
#include <QDebug>
#include <QTextCodec>
#include <QTextStream>
#include <iostream>
 
QTextStream cin(stdin);
QTextStream cout(stdout);
 
int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);
 
    QString string = "Привет, Мир!";
    QTextCodec *codec = QTextCodec::codecForName("CP866");
 
    // std::cout
    QByteArray encodedString = codec->fromUnicode(string);
    std::cout << "std::cout = " << encodedString.data() << std::endl;
 
    // TextStream
    cout.setCodec(codec);
    cout << "TextStream = " << string << "\n";
    cout.flush();
 
    // qDebug
    QTextCodec::setCodecForLocale(codec);
    qDebug() << "qDebug() = " <<  string;
 
     // Console r/w
    cout << "Enter a text: ";
    cout.flush();
    cin.setCodec(codec);
    QString inputStr;
    // Read data from a console
    cin >> inputStr;
    cout << "From a console = " << inputStr << endl;
    cout.flush();
    
    return a.exec();
}

To copy to clipboard, switch view to plain text mode