Results 1 to 10 of 10

Thread: How to list large number of files & folders ?

  1. #1
    Join Date
    Sep 2011
    Location
    Bangalore
    Posts
    254
    Thanks
    92
    Thanked 16 Times in 16 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Windows

    Default How to list large number of files & folders ?

    I'm writing a utility function to list files & folders recursively/non-recursively in a given directory and returning the output as QStringList.
    Qt Code:
    1. QStringList Utility::dirListing(const QString & path, bool recursive)
    2. {
    3. QDir dir(path);
    4. if(!dir.exists())
    5. return QStringList();
    6.  
    7.  
    8. if(recursive)
    9. {
    10. QDirIterator iterator(path, QDirIterator::Subdirectories);
    11. while(iterator.hasNext())
    12. {
    13. list << iterator.next();
    14. }
    15. }
    16. else
    17. {
    18. QDirIterator iterator(path, QDirIterator::NoIteratorFlags);
    19. while(iterator.hasNext())
    20. {
    21. list << iterator.next();
    22. }
    23. }
    24. return list;
    25. }
    To copy to clipboard, switch view to plain text mode 

    But will this be fine for large data set ?

    How can I do this batch wise ?

    Kindly give me some hints to handle this case. Thank you.

  2. #2
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: How to list large number of files & folders ?

    It depends what you mean by "fine".
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  3. #3
    Join Date
    Sep 2011
    Location
    Bangalore
    Posts
    254
    Thanks
    92
    Thanked 16 Times in 16 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Windows

    Default Re: How to list large number of files & folders ?

    If my directory contains more than 10 million sub-directories and if each subdirectory consists of 10 million files, how do I do this processing without creating a overhead. Using current function will prove costly, so I'm thinking of a better logic to handle such scenarios.
    Last edited by rawfool; 23rd July 2014 at 17:49.

  4. #4
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,416
    Thanks
    37
    Thanked 1,544 Times in 1,494 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: How to list large number of files & folders ?

    What do you want to do with the file paths once you have them?

    Cheers,
    _

  5. #5
    Join Date
    Sep 2011
    Location
    Bangalore
    Posts
    254
    Thanks
    92
    Thanked 16 Times in 16 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Windows

    Default Re: How to list large number of files & folders ?

    I'm creating dirListing(const QString & path, bool recursive) function in FileUtil class as dynamic library, where user will use different APIs of this FileUtil class.
    And now the requirement is to write a function which does a feature similar to ls -lR. And this function is expected to handle large data sets and give output without any overhead or without consuming much of CPU.
    So my question is how do I design this function to give output to the caller with above conditions.
    For this I'm having one idea, which I'm not sure of implementation -
    - Can I use signals & slots mechanism to send finite amount of data in batches ? If so, then how can I design the function ?

    Please give suggestions for this scenario.
    Last edited by rawfool; 24th July 2014 at 08:33.

  6. #6
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: How to list large number of files & folders ?

    Quote Originally Posted by rawfool View Post
    If my directory contains more than 10 million sub-directories and if each subdirectory consists of 10 million files,
    Then you will run out of memory

    how do I do this processing without creating a overhead. Using current function will prove costly, so I'm thinking of a better logic to handle such scenarios.
    You might have a thread (or a couple of them) that will iterate over the directory and will signal new paths to the main thread where they will be intercepted and added to a dynamic list.
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  7. #7
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,729
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: How to list large number of files & folders ?

    Quote Originally Posted by rawfool View Post
    If my directory contains more than 10 million sub-directories and if each subdirectory consists of 10 million files,
    You are expecting to hold 1018 QStrings in-memory on a machine that likely has RAM measured in small multiples of 109 bytes. You need TARDIS RAM for that.
    how do I do this processing without creating a overhead.
    What you have asked to do is all overhead. You don't try to store such a large structure in memory. Does such a list have to exist at all given the files on disk already represent the list?
    Using current function will prove costly, so I'm thinking of a better logic to handle such scenarios.
    Yes, indeed. Process each file name as it comes off the iterator, for example.

  8. #8
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,416
    Thanks
    37
    Thanked 1,544 Times in 1,494 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: How to list large number of files & folders ?

    Quote Originally Posted by rawfool View Post
    I'm creating dirListing(const QString & path, bool recursive) function in FileUtil class as dynamic library, where user will use different APIs of this FileUtil class.
    And now the requirement is to write a function which does a feature similar to ls -lR. And this function is expected to handle large data sets and give output without any overhead or without consuming much of CPU.
    That doesn't answer my question.
    What is a use case for that function?
    What is the calling code expected to do for example?

    Cheers,
    _

  9. #9
    Join Date
    Sep 2011
    Location
    Bangalore
    Posts
    254
    Thanks
    92
    Thanked 16 Times in 16 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Windows

    Default Re: How to list large number of files & folders ?

    Actually I was told by my boss to create a function that gives list of dirs & files recursively. So I'm also not sure of use case scenario.
    Thank you.

  10. #10
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,416
    Thanks
    37
    Thanked 1,544 Times in 1,494 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: How to list large number of files & folders ?

    Ok.

    Then the best option is to just create a QDirIterator.
    Or, if the listing should first hav all files and then recurse, an object with a similar interface (hasNext/next) that internaly use QDirIterator.

    Cheers,
    _

  11. The following user says thank you to anda_skoa for this useful post:

    rawfool (25th July 2014)

Similar Threads

  1. Replies: 7
    Last Post: 15th November 2013, 18:20
  2. Replies: 13
    Last Post: 28th October 2012, 10:09
  3. How to efficiently display a large number of QPixmaps?
    By ChiliPalmer in forum Qt Programming
    Replies: 3
    Last Post: 4th April 2011, 02:25
  4. QScrollArea not displaying large number of images
    By lpkincaid in forum Qt Programming
    Replies: 1
    Last Post: 31st May 2009, 10:58
  5. import large number of images
    By sriluyarlagadda in forum Qt Programming
    Replies: 5
    Last Post: 15th May 2008, 11:26

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.