Results 1 to 11 of 11

Thread: Loading and saving in-memory SQLITE databases

  1. #1
    Join Date
    Aug 2008
    Posts
    70
    Thanks
    5
    Thanked 1 Time in 1 Post
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default Loading and saving in-memory SQLITE databases

    Hi,

    My application is working with a document file implemented in a sqlite database.
    So, the function "Open" is copying a file base sqlite database in memory (This is very fast).
    The function "Save" is copying a memory database into a file (This is very slow due to file access).

    For you information, here is the code to do this copy:
    Qt Code:
    1. SKGError SKGServices::copySqliteDatabase(QSqlDatabase* iFileDb, QSqlDatabase* iMemoryDb, bool iFromFileToMemory)
    2. {
    3. SKGError err;
    4. SKGTRACEINRC(10, "SKGServices::copySqliteDatabase", err);
    5. if (iFileDb && iMemoryDb) {
    6. SKGTRACEL(20) << "Input parameter [iFileDb]=[" << iFileDb->databaseName() << ']' << endl;
    7. SKGTRACEL(20) << "Input parameter [iMemoryDb]=[" << iMemoryDb->databaseName() << ']' << endl;
    8. SKGTRACEL(10) << "Input parameter [iFromFileToMemory]=[" << (iFromFileToMemory ? "FILE->MEMORY" : "MEMORY->FILE") << ']' << endl;
    9.  
    10. QString dbFileName=iFileDb->databaseName();
    11.  
    12. // Copy the schema
    13. SKGTRACEL(20) << "Coping the schema..." << endl;
    14. SKGStringListList listSqlOrder;
    15. err=SKGServices::executeSelectSqliteOrder((iFromFileToMemory ? iFileDb: iMemoryDb),
    16. "SELECT sql FROM sqlite_master WHERE sql NOT NULL and name NOT IN ('sqlite_stat1', 'sqlite_sequence')",
    17. listSqlOrder);
    18.  
    19. int nb=listSqlOrder.count();
    20. for (int i=1; err.isSucceeded() && i<nb; ++i) { //Forget header
    21. QString val=listSqlOrder.at(i).at(0);
    22. err=SKGServices::executeSqliteOrder((iFromFileToMemory ? iMemoryDb: iFileDb), val);
    23. }
    24.  
    25. // Attach the file dbFileName to the target
    26. if (err.isSucceeded()) {
    27. SKGTRACEL(20) << "Attaching the file..." << endl;
    28. err = SKGServices::executeSqliteOrder(iMemoryDb, "ATTACH DATABASE '" + dbFileName + "' as source");
    29. if (err.isSucceeded()) {
    30. // Copy the DATA from the source to the target
    31. SKGTRACEL(20) << "Coping data..." << endl;
    32. err = SKGServices::executeSqliteOrder(iMemoryDb, "BEGIN");
    33. if (err.isSucceeded()) {
    34. SKGStringListList listSqlOrder;
    35. err=SKGServices::executeSelectSqliteOrder(iMemoryDb,
    36. "SELECT name FROM source.sqlite_master WHERE type='table' and name NOT IN ('sqlite_stat1', 'sqlite_sequence')",
    37. listSqlOrder);
    38. int nb=listSqlOrder.count();
    39. for (int i=1; err.isSucceeded() && i<nb; ++i) { //Forget header
    40. QString val=listSqlOrder.at(i).at(0);
    41. if (iFromFileToMemory) err=SKGServices::executeSqliteOrder(iMemoryDb, "insert into main."+val+" select * from source."+val);
    42. else err=SKGServices::executeSqliteOrder(iMemoryDb, "insert into source."+val+" select * from main."+val);
    43. }
    44.  
    45. }
    46. if (err.isSucceeded()) err = SKGServices::executeSqliteOrder(iMemoryDb, "COMMIT");
    47.  
    48. //Detach
    49. SKGError err2=SKGServices::executeSqliteOrder(iMemoryDb, "DETACH DATABASE source");
    50. if (err.isSucceeded() && err2.isFailed()) err=err2;
    51.  
    52. //Check if created file exists
    53. if (err.isSucceeded() && !iFromFileToMemory && !QFile(dbFileName).exists()) {
    54. //Set error message
    55. err.setReturnCode(ERR_FAIL);
    56. err.setMessage(i18nc("An error message", "Creation file [%1] failed",dbFileName));
    57. }
    58. }
    59. }
    60. if (err.isFailed()) err.addError(SQLLITEERROR + ERR_FAIL, i18nc("Error message", "%1 failed", QString("copySqliteDatabase::copySqliteDatabase()")));
    61. }
    62. return err;
    63. }
    To copy to clipboard, switch view to plain text mode 

    I saw in this page http://www.sqlite.org/backup.html that new APIs are existing to do that.

    Do you know how we can use it with QT ?

  2. #2
    Join Date
    May 2009
    Location
    USA
    Posts
    300
    Thanks
    82
    Thanked 11 Times in 11 Posts
    Qt products
    Qt4
    Platforms
    Windows

    Default Re: Loading and saving in-memory SQLITE databases

    I don't know about using that with Qt. I suspect QtSql does not support it.
    I'm wondering why your write the db to the file is so slow. I am doing that with a small database and it is not slow.
    So my questions would be: how large is the database you are writing to a file?
    And, what does your select statement for getting the records look like? Are you selecting all of the records, or just some of them?
    If just some of them, what does your 'where' clause look like and do you have the proper columns indexed?
    Your problem could be a slow query due to lack of indexes.

  3. #3
    Join Date
    Aug 2008
    Posts
    70
    Thanks
    5
    Thanked 1 Time in 1 Post
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default Re: Loading and saving in-memory SQLITE databases

    Quote Originally Posted by waynew View Post
    I don't know about using that with Qt. I suspect QtSql does not support it.
    I'm wondering why your write the db to the file is so slow. I am doing that with a small database and it is not slow.
    So my questions would be: how large is the database you are writing to a file?
    The size of the file is around 4M.
    Around 20 tables.
    Most important table contains ~6000 records.

    Quote Originally Posted by waynew View Post
    And, what does your select statement for getting the records look like?
    Are you selecting all of the records, or just some of them?
    As you can see in my code, I am duplicating all schema objects (tables, indexes, triggers) except 'sqlite_stat1' and 'sqlite_sequence'.
    After that, I am duplicating all records of these tables.

    Quote Originally Posted by waynew View Post
    If just some of them, what does your 'where' clause look like and do you have the proper columns indexed?
    Your problem could be a slow query due to lack of indexes.
    Indexes have been duplicated first.

    One more remark:
    The same function is used to duplicate from MEMORY to FILE and from FILE to MEMORY.
    It's very fast from FILE to MEMORY (<500ms) but very slow (>20s) from MEMORY to FILE (My hard disc is working hard).

    Why ?

  4. #4
    Join Date
    Mar 2008
    Location
    Kraków, Poland
    Posts
    1,536
    Thanked 284 Times in 279 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Loading and saving in-memory SQLITE databases

    Quote Originally Posted by miraks View Post
    One more remark:
    The same function is used to duplicate from MEMORY to FILE and from FILE to MEMORY.
    It's very fast from FILE to MEMORY (<500ms) but very slow (>20s) from MEMORY to FILE (My hard disc is working hard).

    Why ?
    Because write operations on disk are not cached ?

  5. #5
    Join Date
    Aug 2008
    Posts
    70
    Thanks
    5
    Thanked 1 Time in 1 Post
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default Re: Loading and saving in-memory SQLITE databases

    Quote Originally Posted by Lesiok View Post
    Because write operations on disk are not cached ?
    How can I cached them ?
    All sql orders launched are done between a BEGIN and a COMMIT.

  6. #6
    Join Date
    Aug 2008
    Posts
    70
    Thanks
    5
    Thanked 1 Time in 1 Post
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default Re: Loading and saving in-memory SQLITE databases

    Please ! Do you have an answer ? I am blocked.

  7. #7
    Join Date
    Mar 2008
    Location
    Kraków, Poland
    Posts
    1,536
    Thanked 284 Times in 279 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Loading and saving in-memory SQLITE databases

    Quote Originally Posted by miraks View Post
    How can I cached them ?
    All sql orders launched are done between a BEGIN and a COMMIT.
    This is a problem for OS and/or for DB engine.

  8. #8
    Join Date
    Aug 2008
    Posts
    70
    Thanks
    5
    Thanked 1 Time in 1 Post
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default Re: Loading and saving in-memory SQLITE databases

    Quote Originally Posted by Lesiok View Post
    This is a problem for OS and/or for DB engine.
    As I said in my first topic, the solution is already existing in sqlite, this is the backup system.
    But, how to use it with QT ?

  9. #9
    Join Date
    Aug 2008
    Posts
    70
    Thanks
    5
    Thanked 1 Time in 1 Post
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default Re: Loading and saving in-memory SQLITE databases

    Do you have an answer ?

  10. #10
    Join Date
    Jul 2006
    Location
    Catalunya - Spain
    Posts
    117
    Thanks
    16
    Thanked 8 Times in 8 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Loading and saving in-memory SQLITE databases

    2 possible points of "stress" :

    As I've found some time ago, don't ask me why, it's not the same for SQLite execute some statements between a BEGIN / COMMIT block than doing :

    Qt Code:
    1. QSqlDatabase.transaction();
    2. ...
    3. ...<here ALL the stuff>
    4. ...
    5. QSqlDatabase.commit();
    To copy to clipboard, switch view to plain text mode 
    And 2 :

    seeing your code, youhave the following statement :

    Qt Code:
    1. SKGServices::executeSqliteOrder(iMemoryDb, "BEGIN");
    2. ....
    3. SKGServices::executeSqliteOrder(iMemoryDb, "COMMIT");
    To copy to clipboard, switch view to plain text mode 

    So, ONLY im-memory database has a begin / commit enclosure. The code has to be like this, or not ?

    Qt Code:
    1. SKGServices::executeSqliteOrder(iFromFileToMemory ? iMemoryD : iFileDb, "BEGIN");
    2. ....
    3. SKGServices::executeSqliteOrder(iFromFileToMemory ? iMemoryD : iFileDb, "COMMIT");
    To copy to clipboard, switch view to plain text mode 

  11. #11
    Join Date
    Aug 2008
    Posts
    70
    Thanks
    5
    Thanked 1 Time in 1 Post
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default Re: Loading and saving in-memory SQLITE databases

    Hi jpujolf,

    Thank you for you answer.
    I think that BEGIN and COMMIT must be done on memory database because all sql orders are launched on this database or on a attached database. In any case, the main database is the memory one.

    Here is my last version of code:
    Qt Code:
    1. SKGError SKGServices::copySqliteDatabase(QSqlDatabase* iFileDb, QSqlDatabase* iMemoryDb, bool iFromFileToMemory)
    2. {
    3. SKGError err;
    4. SKGTRACEINRC(10, "SKGServices::copySqliteDatabase", err);
    5. if (iFileDb && iMemoryDb) {
    6. SKGTRACEL(20) << "Input parameter [iFileDb]=[" << iFileDb->databaseName() << ']' << endl;
    7. SKGTRACEL(20) << "Input parameter [iMemoryDb]=[" << iMemoryDb->databaseName() << ']' << endl;
    8. SKGTRACEL(10) << "Input parameter [iFromFileToMemory]=[" << (iFromFileToMemory ? "FILE->MEMORY" : "MEMORY->FILE") << ']' << endl;
    9.  
    10. QString dbFileName=iFileDb->databaseName();
    11. // Copy the tables
    12. SKGStringListList listTables;
    13. int nb=0;
    14. if (err.isSucceeded())
    15. {
    16. SKGTRACEINRC(10, "SKGServices::copySqliteDatabase-COPYTABLES", err);
    17. err=SKGServices::executeSelectSqliteOrder((iFromFileToMemory ? iFileDb: iMemoryDb),
    18. "SELECT sql, tbl_name FROM sqlite_master WHERE type='table' AND sql NOT NULL and name NOT IN ('sqlite_stat1', 'sqlite_sequence')",
    19. listTables);
    20.  
    21. nb=listTables.count();
    22. for (int i=1; err.isSucceeded() && i<nb; ++i) { //Forget header
    23. QString val=listTables.at(i).at(0);
    24. err=SKGServices::executeSqliteOrder((iFromFileToMemory ? iMemoryDb: iFileDb), val);
    25. }
    26. }
    27. //Attach db
    28. if (err.isSucceeded()) {
    29. SKGTRACEINRC(10, "SKGServices::copySqliteDatabase-ATTACH", err);
    30. err = SKGServices::executeSqliteOrder(iMemoryDb, "ATTACH DATABASE '" + dbFileName + "' as source");
    31. }
    32.  
    33. //Copy records
    34. if (err.isSucceeded()) {
    35. SKGTRACEINRC(10, "SKGServices::copySqliteDatabase-COPY", err);
    36. err = SKGServices::executeSqliteOrder(iMemoryDb, "BEGIN");
    37. if (err.isSucceeded()) {
    38. for (int i=1; err.isSucceeded() && i<nb; ++i) { //Forget header
    39. QString val=listTables.at(i).at(1);
    40. if (iFromFileToMemory) err=SKGServices::executeSqliteOrder(iMemoryDb, "insert into main."+val+" select * from source."+val);
    41. else err=SKGServices::executeSqliteOrder(iMemoryDb, "insert into source."+val+" select * from main."+val);
    42. }
    43.  
    44. }
    45. SKGServices::executeSqliteOrder(iMemoryDb, "COMMIT");
    46. }
    47.  
    48. //Detach
    49. SKGError err2;
    50. {
    51. SKGTRACEINRC(10, "SKGServices::copySqliteDatabase-DETACH", err);
    52. err2=SKGServices::executeSqliteOrder(iMemoryDb, "DETACH DATABASE source");
    53. if (err.isSucceeded() && err2.isFailed()) err=err2;
    54. }
    55.  
    56. //Optimisation
    57. if (err.isSucceeded())
    58. {
    59.  
    60. QString optimisation="PRAGMA case_sensitive_like=true;;"
    61. "PRAGMA journal_mode=MEMORY;;"
    62. "PRAGMA temp_store=MEMORY;;"
    63. "PRAGMA locking_mode=EXCLUSIVE;;"
    64. "PRAGMA synchronous = OFF;;"
    65. ;
    66. err = SKGServices::executeSqliteOrder(iFromFileToMemory ? iMemoryDb: iFileDb, optimisation);
    67. }
    68.  
    69. // Copy the triggers
    70. if (err.isSucceeded())
    71. {
    72. SKGTRACEINRC(10, "SKGServices::copySqliteDatabase-COPYTRIGGER", err);
    73. SKGStringListList listSqlOrder;
    74. err=SKGServices::executeSelectSqliteOrder((iFromFileToMemory ? iFileDb: iMemoryDb),
    75. "SELECT sql FROM sqlite_master WHERE type='trigger' AND sql NOT NULL and name NOT IN ('sqlite_stat1', 'sqlite_sequence')",
    76. listSqlOrder);
    77.  
    78. int nb=listSqlOrder.count();
    79. for (int i=1; err.isSucceeded() && i<nb; ++i) { //Forget header
    80. QString val=listSqlOrder.at(i).at(0);
    81. err=SKGServices::executeSqliteOrder((iFromFileToMemory ? iMemoryDb: iFileDb), val);
    82. }
    83. }
    84.  
    85. // Copy the indexes
    86. if (err.isSucceeded())
    87. {
    88. SKGTRACEINRC(10, "SKGServices::copySqliteDatabase-COPYINDEX", err);
    89. SKGStringListList listSqlOrder;
    90. err=SKGServices::executeSelectSqliteOrder((iFromFileToMemory ? iFileDb: iMemoryDb),
    91. "SELECT sql FROM sqlite_master WHERE type='index' AND sql NOT NULL and name NOT IN ('sqlite_stat1', 'sqlite_sequence')",
    92. listSqlOrder);
    93.  
    94. int nb=listSqlOrder.count();
    95. for (int i=1; err.isSucceeded() && i<nb; ++i) { //Forget header
    96. QString val=listSqlOrder.at(i).at(0);
    97. err=SKGServices::executeSqliteOrder((iFromFileToMemory ? iMemoryDb: iFileDb), val);
    98. }
    99. }
    100.  
    101. // Copy the views
    102. if (err.isSucceeded())
    103. {
    104. SKGTRACEINRC(10, "SKGServices::copySqliteDatabase-COPYVIEW", err);
    105. SKGStringListList listSqlOrder;
    106. err=SKGServices::executeSelectSqliteOrder((iFromFileToMemory ? iFileDb: iMemoryDb),
    107. "SELECT sql FROM sqlite_master WHERE type='view' AND sql NOT NULL and name NOT IN ('sqlite_stat1', 'sqlite_sequence')",
    108. listSqlOrder);
    109.  
    110. int nb=listSqlOrder.count();
    111. for (int i=1; err.isSucceeded() && i<nb; ++i) { //Forget header
    112. QString val=listSqlOrder.at(i).at(0);
    113. err=SKGServices::executeSqliteOrder((iFromFileToMemory ? iMemoryDb: iFileDb), val);
    114. }
    115. }
    116.  
    117. //Check if created file exists
    118. if (err.isSucceeded() && !iFromFileToMemory && !QFile(dbFileName).exists()) {
    119. //Set error message
    120. err.setReturnCode(ERR_FAIL);
    121. err.setMessage(i18nc("An error message", "Creation file [%1] failed",dbFileName));
    122. }
    123. }
    124. if (err.isFailed()) err.addError(SQLLITEERROR + ERR_FAIL, i18nc("Error message", "%1 failed", QString("copySqliteDatabase::copySqliteDatabase()")));
    125. return err;
    126. }
    To copy to clipboard, switch view to plain text mode 

    This version is faster due to PRAGMA settings.
    In any case, it could be interesting to use backup sqlite API from QT.

Similar Threads

  1. Loading an image into memory
    By Luc4 in forum Qt Programming
    Replies: 6
    Last Post: 12th April 2010, 08:44
  2. Saving parts of an SQLITE database to a new file
    By zarkzervo in forum Qt Programming
    Replies: 2
    Last Post: 21st February 2010, 00:59
  3. Loading SQLite plugin. Again!
    By miwarre in forum Newbie
    Replies: 5
    Last Post: 12th November 2009, 10:23
  4. QDataStream and saving and loading QList
    By Noxxik in forum Qt Programming
    Replies: 3
    Last Post: 1st March 2009, 22:02
  5. saving a c string of variable length in a shared memory?
    By nass in forum General Programming
    Replies: 4
    Last Post: 3rd January 2007, 14:40

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.