Results 1 to 13 of 13

Thread: Compiling and efficiency problems

  1. #1
    Join Date
    Jun 2008
    Posts
    83
    Thanks
    1
    Qt products
    Qt4
    Platforms
    Windows

    Default Compiling and efficiency problems

    Hi:

    I need to write a program that essentially needs to mane a list of a list of a list (that's right 3 times) of a certain class I created. I was having some efficiency problems so I decided to change all of the accesses to the structure form structure[i][j][k] to using the .at() function. I wrote a little code just to test this:

    This is the Class named Tile.h
    Qt Code:
    1. class Tile
    2. {
    3. public:
    4. Tile(qreal p, QList<quint16>);
    5. Tile();
    6.  
    7. //Definition of basic operations
    8. bool operator == (Tile t);
    9.  
    10. //Definition of sets and getters.
    11. QList<quint16> Indexes(){return indexes;}
    12. qreal param(){return parameter;}
    13. void setParam(qreal p){parameter = p;}
    14. void setIndexes(QList<quint16> ind){indexes = ind;}
    15.  
    16. //Function use Mainly for Debugging
    17. QString toString();
    18.  
    19. virtual ~Tile();
    20.  
    21. private:
    22. qreal parameter;
    23. QList<quint16> indexes;
    24. };
    To copy to clipboard, switch view to plain text mode 

    And the Tile.cpp
    Qt Code:
    1. Tile::Tile(qreal p, QList<quint16> ind){
    2. indexes = ind;
    3. parameter = p;
    4. elegibility = 0;
    5. }
    6.  
    7. Tile::Tile(){
    8. parameter = 0;
    9. }
    10.  
    11. /****************Operators**************************/
    12. bool Tile::operator == (Tile t){
    13. QList<quint16> comp = t.Indexes();
    14. bool res = true;
    15. for (int i = 0; i < comp.size(); i++){
    16. if (comp.at(i) != indexes.at(i)){
    17. res = false;
    18. break;
    19. }
    20. }
    21. return res;
    22. }
    23.  
    24. /***********toString******************/
    25. QString Tile::toString(){
    26. QString ws = "";
    27. for (int i = 0; i < indexes.size(); i++){
    28. ws = ws + " " + QString::number(indexes.at(i));
    29. }
    30.  
    31. if(indexes.size() == 0){
    32. ws = "Nothing to show: Value is: " + QString::number(parameter);
    33. }
    34.  
    35. return ws;
    36. }
    37.  
    38. Tile::~Tile(){
    39. }
    To copy to clipboard, switch view to plain text mode 

    Then I wrote this code in a constructor just to test using this
    Qt Code:
    1. int Im = 5;
    2. int Jm = 10;
    3. int Km = 6000;
    4. for (int i = 0; i < Im; i++){
    5. QVector < QVector<Tile> > temp2;
    6. for(int j = 0; j < Jm; j++){
    7. QVector<Tile> temp;
    8. for (int k = 0; k < Km; k++){
    9. QList<quint16> ind;
    10. ind << k;
    11. Tile t(0,ind);
    12. temp << t;
    13. }
    14. temp2 << temp;
    15. }
    16. test << temp2;
    17. }
    To copy to clipboard, switch view to plain text mode 

    Where test is defined by:
    Qt Code:
    1. QVector < QVector< QVector<Tile> > > test;
    To copy to clipboard, switch view to plain text mode 

    So then I write this line in a button action just to see if it would compile
    Qt Code:
    1. test.at(2).at(5).at(3940).setParam(test.at(2).at(5).at(3940).param() + 15.0);
    To copy to clipboard, switch view to plain text mode 

    And it doesn't and I get these two errors
    passing ‘const Tile’ as ‘this’ argument of ‘qreal Tile:aram()’ discards qualifiers
    passing ‘const Tile’ as ‘this’ argument of ‘void Tile::setParam(qreal)’ discards qualifiers

    So I change it to this in order for it to work
    Qt Code:
    1. test[2][5][3940].setParam(test[2][5][3940].param() + 15.0);
    To copy to clipboard, switch view to plain text mode 

    And this works but according to the Qt documentation [] is a lot more inefficient that at(). So I want it to change it to see if my algorithm would go a bit faster.

    So my question is are there any suggestion on how to make this a bit more efficient to see if I can shave off some minutes to the my algortihms runs or am I just stuck with the slow program. Or maybe on how could I manage a list of a list of a list of classes in a more efficient way.

    Thanks for any help

  2. #2
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: Compiling and efficiency problems

    Quote Originally Posted by aarelovich View Post
    And this works but according to the Qt documentation [] is a lot more inefficient that at(). So I want it to change it to see if my algorithm would go a bit faster.
    [] is defined as:
    Qt Code:
    1. template <typename T>
    2. inline const T &QVector<T>::operator[](int i) const
    3. { Q_ASSERT_X(i >= 0 && i < d->size, "QVector<T>::operator[]", "index out of range");
    4. return p->array[i]; }
    5. template <typename T>
    6. inline T &QVector<T>::operator[](int i)
    7. { Q_ASSERT_X(i >= 0 && i < d->size, "QVector<T>::operator[]", "index out of range");
    8. return data()[i]; }
    To copy to clipboard, switch view to plain text mode 

    and at() defined as:
    Qt Code:
    1. template <typename T>
    2. inline const T &QVector<T>::at(int i) const
    3. { Q_ASSERT_X(i >= 0 && i < d->size, "QVector<T>::at", "index out of range");
    4. return p->array[i]; }
    To copy to clipboard, switch view to plain text mode 

    You can see they are identical. Only the non-const version of [] is slower but that is obvious as data may need to be copied (which is what data() does).


    So my question is are there any suggestion on how to make this a bit more efficient to see if I can shave off some minutes to the my algortihms runs or am I just stuck with the slow program. Or maybe on how could I manage a list of a list of a list of classes in a more efficient way.
    I suggest you use some faster structure than a list. What does your list represent? Can't you use a has instead?
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  3. #3
    Join Date
    Jun 2008
    Posts
    83
    Thanks
    1
    Qt products
    Qt4
    Platforms
    Windows

    Default Re: Compiling and efficiency problems

    I'm sorry I don't know what a "has" is.

    I'll try to explain as abstractly as possible what my structure represents.
    The thing is that I need to partition the n dimensions of a user specified space each into p partitions, each value of p different for each of the n dimensions. So say that I have 3 dimensions and each dimension (representing a variable in a certain user defined range, -10 to 10, -5 to 5 and -6 to 6) is parted into say 5 6 and 7, I would have partition the entire space into 5*6*7 = 210 partitions of the space (represented by what I call a tile, so 210 tiles). These need to be copied another d times (each offseted a little differently so that the same 3 D value doesn't necessarily fall in the same tile for all of the d copies) in order to create the full structure. And I need A copies of this structure, again according to user input.

    Now the problem becomes how I use it. Say I get the 3 dimension value -3.4, 2, 5.6. This falls within one and only one of the limits of the 210 tiles for each of the d structures. I need to find it in each (that why I redifined the == method in Tile so as to use the indexOf() function of the QVector). And do some operation to the value (param) of that particular tile.

    It's complex but that is what I need to do. Can that "has" help? I'm sorry if it sounds complicated, it really is

    Thanks for any help.

  4. #4
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: Compiling and efficiency problems

    Quote Originally Posted by aarelovich View Post
    I'm sorry I don't know what a "has" is.
    Sorry I typed too fast, it should say "hash".

    In my opinion you should have a hash indexed by three (or "n" to follow your abstract example) real values which would allow you to quickly navigate through your data.

    Qt Code:
    1. struct HashKey {
    2. HashKey(double _one, double _two, double_three){ one = _one; two = _two; three = _three; }
    3. double one;
    4. double two;
    5. double three;
    6. bool operator==(const HashKey &other) const {
    7. return ((one==other.one) && (two==other.two) && (three==other.three));
    8. }
    9. };
    10.  
    11. uint qHash(const HashKey &key) { return qHash( qHash(key.one)+qHash(key.two)+qHash(key.three)); }
    12. //...
    13. QHash<HashKey, Tile> data;
    14.  
    15. data[HashKey(-3.4, 2, 5.6)] = Tile(...);
    To copy to clipboard, switch view to plain text mode 

    Note that knowing the index (key), retrieving a value from the vector is faster (O(1)) than retrieving a value from a hash (amortized O(lg(n))). But on the other hand you waste lot of time iterating over a sparse vector. For instance your comparison operator is awfully slow, there are ways to significantly speed it up (for instance calculating hashes of vectors and comparing the hashes instead of vectors themselves).
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  5. #5
    Join Date
    Jun 2008
    Posts
    83
    Thanks
    1
    Qt products
    Qt4
    Platforms
    Windows

    Default Re: Compiling and efficiency problems

    Hi. Okay. I'll read a bit about hashes and I'll try implementing what you've suggested as soon as I get a moment

    Thanks

  6. #6
    Join Date
    Jun 2008
    Posts
    83
    Thanks
    1
    Qt products
    Qt4
    Platforms
    Windows

    Default Re: Compiling and efficiency problems

    Hi:

    So I've been thinking a lot about your suggestions and there are a couple of problems.
    The first is that the state (the collection of variable values) (say (3.5, 1, -2)) has a tile with a correspoding index partition, say (1,2,3), in one of the d copies (called tilings) of the structure I mentioned and will have indexes (2,2,3) in another of the d copies because (as I tried to explain) each of the d copies has a random offset in each dimension (in this case 3) that causes the index 0,0,0 to correspond to different state values. So I have to, no matter what calculate the index of the corresponding tile of each of the tilings. But I could use the indexes like you mentioned to create the hashes out of a QList<int> indexes variable. So I slightly adapted the code you gave me to this on tiletests.h (I assumed you mistyped and my function had to have a different name that qHash)
    Qt Code:
    1. uint Hash(QList<int> ind) { return qHash( qHash(ind.at(0))+qHash(ind.at(1))+qHash(ind.at(2))); } // The real code would have for cycle that addeded up as many elements as there were on the list.
    To copy to clipboard, switch view to plain text mode 

    Then I ran a test with this code
    Qt Code:
    1. QList<int> ind;
    2. ind << 1 << 2 << 3;
    3. qDebug() << "1 >> :" << Hash(ind);
    4. ind.clear();
    5. ind << 5 << 6 << 7;
    6. qDebug() << "2 >> :" << Hash(ind);
    7. ind.clear();
    8. ind << 3 << 2 << 1;
    9. qDebug() << "3 >> :" << Hash(ind);
    To copy to clipboard, switch view to plain text mode 

    The result as I expected was this:
    1 >> : 6
    2 >> : 18
    3 >> : 6

    Which means I can't used the code exactly like this because the set of indexes 1,2,3 and 3,2,1 clearly point to a different tile.

    So I came up with this idea: I would like to use a QString as a key so that (1,2,3) becomes "1,2,3" and (3,2,1) becomes "3,2,1" which would mean each key would be different. I'm going to do the tests anyways, but I would like to hear your ideas on well, my idea.

    Thanks for all the help.

  7. #7
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: Compiling and efficiency problems

    Quote Originally Posted by aarelovich View Post
    I assumed you mistyped and my function had to have a different name that qHash
    No, I didn't. The function has to be called qHash and take a const reference to your key class.
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  8. #8
    Join Date
    May 2008
    Posts
    155
    Thanked 15 Times in 13 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default

    If you care about speed you should also look at passing parameters as const reference instead of value. saves a copy, even if it is a shallow one with shared containers...

    Here:

    Qt Code:
    1. class Tile
    2. {
    3. public:
    4. Tile(qreal p, QList<quint16>); // Here
    5. ...
    6. bool operator == (Tile t); // Here, also make it const
    7. ...
    8. QList<quint16> Indexes(){return indexes;} // make function const
    9. qreal param(){return parameter;} // same
    10. void setParam(qreal p){parameter = p;}
    11. void setIndexes(QList<quint16> ind){indexes = ind;} // const ref
    12. ...
    13. virtual ~Tile(); // will you ever derive from Tile?
    14. ...
    15. QString ws = ""; // = "" not needed.
    16.  
    17. if(indexes.size() == 0){ // .isEmpty()
    18. ...
    To copy to clipboard, switch view to plain text mode 

    Now,

    Qt Code:
    1. passing ‘const Tile’ as ‘this’ argument of ‘qreal Tile::param()’ discards qualifiers
    2. passing ‘const Tile’ as ‘this’ argument of ‘void Tile::setParam(qreal)’ discards qualifiers
    To copy to clipboard, switch view to plain text mode 

    ... that's the consequence of the missing 'const' qualification of your getters.
    Make them const, and it will work.

    Quote Originally Posted by wysota View Post
    Sorry I typed too fast, it should say "hash".

    In my opinion you should have a hash indexed by three (or "n" to follow your abstract example) real values which would allow you to quickly navigate through your data.

    Qt Code:
    1. struct HashKey {
    2. HashKey(double _one, double _two, double_three){ one = _one; two = _two; three = _three; }
    3. double one;
    4. double two;
    5. double three;
    6. bool operator==(const HashKey &other) const {
    7. return ((one==other.one) && (two==other.two) && (three==other.three));
    8. }
    9. };
    10.  
    11. uint qHash(const HashKey &key) { return qHash( qHash(key.one)+qHash(key.two)+qHash(key.three)); }
    12. //...
    13. QHash<HashKey, Tile> data;
    14.  
    15. data[HashKey(-3.4, 2, 5.6)] = Tile(...);
    To copy to clipboard, switch view to plain text mode 

    Note that knowing the index (key), retrieving a value from the vector is faster (O(1)) than retrieving a value from a hash (amortized O(lg(n))). But on the other hand you waste lot of time iterating over a sparse vector. For instance your comparison operator is awfully slow, there are ways to significantly speed it up (for instance calculating hashes of vectors and comparing the hashes instead of vectors themselves).
    A hash is not "amortized O(lg(n))", it is has worst case O(n), and all depends on the hash function...
    Last edited by wysota; 1st October 2009 at 22:54.

  9. #9
    Join Date
    Jun 2008
    Posts
    83
    Thanks
    1
    Qt products
    Qt4
    Platforms
    Windows

    Default Re: Compiling and efficiency problems

    Hi:

    Well, if you didn't mistype I'm sure I understand how I should use the hash.

    However I did try to test the difference in performance between a hash and my old approach and I've got very dissapointing results:
    This is my constructor:
    Qt Code:
    1. int Im = 5;
    2. int Jm = 10;
    3. int Km = 36864;
    4. Actions = Im;
    5. NumTiles = Km;
    6. TilesInTiling = Jm*Km;
    7. NumTilings = Jm;
    8. QVector<quint16> Mod;
    9. QVector<quint16> ind(6);
    10. ind.fill(0);
    11.  
    12. //Six dimensions with divisions 12 12 4 4 4 4
    13. Base << 1 << 12 << 144 << 576 << 2304 << 9216;
    14. Mod << 12 << 12 << 4 << 4 << 4 << 4;
    15.  
    16. start = QTime::currentTime();
    17.  
    18. for (int i = 0; i < Im; i++){
    19. for(int j = 0; j < Jm; j++){
    20. for (int k = 0; k < Km; k++){
    21. Tile t;
    22. t.setParam(k);
    23. test[HashKey(ind,i,j)] = t;
    24. ind = IncMod(ind,Mod);
    25. }
    26. }
    27. }
    28.  
    29. qDebug() << "The size is" << test.count() << "and it took" << start.msecsTo(QTime::currentTime()) << "ms";
    30.  
    31. ind.fill(0);
    32. start = QTime::currentTime();
    33. for (int i = 0; i < Im; i++){
    34. QVector <QVector <Tile> > temp1;
    35. for(int j = 0; j < Jm; j++){
    36. QVector<Tile> temp2;
    37. for (int k = 0; k < Km; k++){
    38. Tile t;
    39. t.setParam(k);
    40. t.setIndexes(ind.toList());
    41. temp2 << t;
    42. ind = IncMod(ind,Mod);
    43. }
    44. temp1 << temp2;
    45. }
    46. testvec << temp1;
    47. }
    48. qDebug() << "The size is" << testvec.size() << "and it took" << start.msecsTo(QTime::currentTime()) << "ms";
    To copy to clipboard, switch view to plain text mode 

    Then what I did is write a test code with the exact same calculations the real program would have to do except I used fantasy-on-the-fly-made-up numbers instead of real program values.

    This is the code that was supposed to test the hash:
    Qt Code:
    1. //Simulation of Eval and Deriv
    2. qreal sum = 0;
    3. QVector<uint> hashes;
    4. for (int act = 0; act < Actions; act++){
    5. sum = 0;
    6. hashes.clear();
    7. for (int tiling = 0; tiling < NumTilings; tiling++){
    8. QVector<quint16> ind;
    9. ind << 8 << 6 << 3 << 3 << 0 << 1;
    10. hashes << HashKey(ind,act,tiling);
    11. }
    12. for (int i = 0; i < NumTilings; i++){
    13. Tile t;
    14. t = test.value(hashes.at(i));
    15. sum = sum + t.getParam();
    16. }
    17. //qDebug() << "Suma: " << sum;
    18. }
    19.  
    20. //Simulation of Update.
    21. QHashIterator<uint, Tile> iter(test);
    22. while (iter.hasNext()) {
    23. iter.next();
    24. uint key = iter.key();
    25. Tile t;
    26. //qDebug() << "Pruebo con llave" << key;
    27. t = test.take(key);
    28. //qDebug() << "Anduvo";
    29. t.DecayTraces(0.7,.8);
    30. if (hashes.contains(key)){
    31. t.IncrementEtrace();
    32. t.setParam(t.getParam() + 0.8*0.6*t.Elegibility());
    33. }
    34. test[key]=t;
    35. }
    To copy to clipboard, switch view to plain text mode 

    And the code as it is right now for the vector structure:
    Qt Code:
    1. //Simulation of eval and deriv
    2. qreal sum = 0;
    3. QVector<Tile> tiles;
    4. QVector<quint16> visited_indexes;
    5. for (int act = 0; act < Actions; act++){
    6. sum = 0;
    7. tiles.clear();
    8. for (int tiling = 0; tiling < NumTilings; tiling++){
    9. QVector<quint16> ind;
    10. ind << 8 << 6 << 3 << 3 << 0 << 1; //In the real program these come from another function.
    11. Tile t(0,ind.toList());
    12. tiles << t;
    13. }
    14. int action = 3; //This would also come from somewhere else;
    15. for (int i = 0; i < NumTilings; i++){
    16. int index = testvec[action][i].indexOf(tiles.at(i));
    17. sum = sum + testvec[action][i][index].getParam();
    18. }
    19. //qDebug() << "Sum: " << sum;
    20. }
    21.  
    22.  
    23. //Simulation of Upadate
    24. bool InTakenAction = false;
    25. int action = 2;
    26. QVector<int> index_list;
    27. index_list << 1 << 2 << 3 << 4 << 5 << 6 << 7 << 8 << 9 << 10;
    28. int temp_index;
    29. for (int i = 0; i < Actions; i++){
    30. for (int j = 0; j < NumTilings; j++){
    31. if (i == action){
    32. temp_index = index_list.at(j);
    33. InTakenAction = true;
    34. }
    35. else
    36. InTakenAction = false;
    37. for (quint16 k = 0; k < NumTiles; k++){
    38. testvec[i][j][k].DecayTraces(0.7,0.8);
    39. if (InTakenAction && (k == temp_index)){
    40. testvec[i][j][k].IncrementEtrace();
    41. testvec[i][j][k] = testvec[i][j][k] + 0.8*0.6*testvec[i][j][k].Elegibility();
    42. }
    43. }
    44. }
    45. }
    To copy to clipboard, switch view to plain text mode 

    I called each test 10 times and measured the time with QTime like in the constructor. These were the results I got:
    The size is 1843200 and it took 2290 ms
    The size is 5 and it took 4425 ms
    Total Time for Test Hash 21711 ms
    Total Time for Test QVector 2889 ms

    My old method is far better, which leads me to believe that I have done something wrong. I think that the problem is the fact That I have to take and then insert each tile in the QHash in order to modify it.

    I really hope that someone can give me an idea of what I'm doing wrong.

  10. #10
    Join Date
    Jun 2008
    Posts
    83
    Thanks
    1
    Qt products
    Qt4
    Platforms
    Windows

    Default Re: Compiling and efficiency problems

    Ktk
    I saw your replies as soon as I finished my new post. I'll look into what you are saying.

    Thanks.

  11. #11
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default

    Quote Originally Posted by ktk View Post
    A hash is not "amortized O(lg(n))", it is has worst case O(n), and all depends on the hash function...
    My bad, it's amortized O(1). Old habits cause interference...

    Quote Originally Posted by aarelovich View Post
    My old method is far better, which leads me to believe that I have done something wrong.
    If you are iterating over all items at once then QHash will always be slower than QVector. It's the lookup that makes hashes fast - if you don't know the index in the container but know what item (key) you are looking for.
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  12. #12
    Join Date
    Jun 2008
    Posts
    83
    Thanks
    1
    Qt products
    Qt4
    Platforms
    Windows

    Default Re: Compiling and efficiency problems

    You are absolutely right.
    I realized that I could "fuse" both solutions. If I can calculate an absolute index I could collapse the three QVectors into one and calculate the index and access it directly saving the time to look for it. Maybe with that and the const ideas from ktk I'll be able to save some time. I'll try it and post back.

    Thanks for all the help.

  13. #13
    Join Date
    Jun 2008
    Posts
    83
    Thanks
    1
    Qt products
    Qt4
    Platforms
    Windows

    Default Re: Compiling and efficiency problems

    Hi:

    Well that did the trick. I used only One very large qvector and do a simple math calculation given a series of indexes in order to calculate (instead of search) for the position in the vector. The results after I did 20 tryouts (like the one described in the code above) was the following
    The old way: 5802 ms
    The new way: 1682 ms which is roughly 28% of the normal time. I'll make the modifications (which are mostly simplifactions) to the old code now that I'm convinced it'll make a difference.

    Thanks for all the help.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.