Results 1 to 2 of 2

Thread: fast writing of large amounts of data to files

  1. #1
    Join Date
    Jan 2006
    Location
    Munich, Germany.
    Posts
    111
    Thanks
    29
    Thanked 3 Times in 2 Posts
    Qt products
    Qt3 Qt4
    Platforms
    Windows

    Default fast writing of large amounts of data to files

    Hello,
    I need to write a lot of real-time data to a file. About 3kB, several hundred times a second for minutes (or hours). I need to write a list of numbers (mostly doubles but not all) as a type of header, and an array of short ints (as data). I've got a lot of number crunching going on so I'd like the fastest way of doing this?

    Is there asynchronous file access in Qt? - so that I can just call some 'write' or '<<' and not wait for the func to return? Can I set some write buffer somewhere so that things only get flushed when really needed?

    Is packing (#pragma..) the data in memory and mem-dumping a bad idea?

    Is it better to use QDataSteam <<, or QIODevice::write ( const QByteArray & byteArray )

    I don't have too much experience with files....
    all help much appreciated
    thanks

    K

  2. #2
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,363
    Thanks
    3
    Thanked 5,012 Times in 4,791 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: fast writing of large amounts of data to files

    File access is always asynchronous - write() fills the file descriptor buffer and returns immediately. Then the system takes the responsibility of writing the data into the file (the disk needs to find the proper sector, etc.).

    What I can suggest is to increase the system buffer (I think you'll have to use native calls here), implement a kind of proxy that feeds the data into application buffer that get emptied to the disk by another thread (so that you can write large amounts of data that would normally fill up the system buffer causing your application to wait until there is free space in the buffer.

    And don't use Q*Stream classes - they are slower than using QFile directly.

Similar Threads

  1. Writing Data Aware Forms in QT4.
    By gsQT4 in forum Qt Programming
    Replies: 5
    Last Post: 21st March 2007, 11:35
  2. Replies: 6
    Last Post: 8th January 2007, 11:24

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.