Results 1 to 4 of 4

Thread: Delete huge number of files in subdirs

  1. #1
    Join Date
    Apr 2006
    Posts
    7
    Thanked 4 Times in 1 Post
    Qt products
    Qt4
    Platforms
    Windows

    Default Delete huge number of files in subdirs

    Hi,

    I'd like to remove a large number of files (10T-1M) in a subdirectory structure efficiently. Currently I have already a recursive implementation using QFile QDir. unfortunately, this is not very efficient on Windows NTFS, maybe on other platforms too.

    Of course I'd like to be platform independent as much as possible.
    Any way to speed up this?

    Best regards,
    Richard

  2. #2
    Join Date
    Aug 2019
    Posts
    7
    Qt products
    Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Delete huge number of files in subdirs

    The problem with the recursive approach is that you are limited to quite few (depending on the platform, hardware, ...) recursive calls otherwise you will run out of memory. I don't know how deep your structure is but if you exceed the 100 recursive calls then I think the recursive approach is not a good idea.
    In this case, I would suggest you to look at some operational research methods as Depth-first search or Breadth-first search to explore your structure without recursion.

    This being said, I don't know if you can speed it up since as far as I know you still have to iterate over each file you want to delete. With such a huge amount of data, the overhead you have is probably induced by the read-write access time on the hard disk.

    But if there is a solution (without changing to another hard disk with better performances), I would be glad to hear it too

  3. #3
    Join Date
    Jan 2008
    Location
    Alameda, CA, USA
    Posts
    5,230
    Thanks
    302
    Thanked 864 Times in 851 Posts
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: Delete huge number of files in subdirs

    unfortunately, this is not very efficient on Windows NTFS
    Windoze does a lot of bookkeeping when files and directories are changed (moved, deleted, created, modified). It could be this overhead that is slow, not the actual file deletion. You might also want to make sure that Windoze isn't putting your files into the Recycle Bin. I'm not sure what actually happens when you delete a file from within Qt.

    If your files are organized such that you can simply delete the top-level directory and everything below it, then you might try firing off an external process to do that (like "rmdir /s /q dir"). You could try the experiment - make a copy of your file tree, then delete one from within Windows Explorer, and the other using a command window and the rmdir command and time the different approaches.
    <=== The Great Pumpkin says ===>
    Please use CODE tags when posting source code so it is more readable. Click "Go Advanced" and then the "#" icon to insert the tags. Paste your code between them.

  4. #4
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,416
    Thanks
    37
    Thanked 1,544 Times in 1,494 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Delete huge number of files in subdirs

    This is always going to be a bit slow due to all the directory lookups.

    A friend of mine ran into this with a large cache file directory structure on a web server, when even the "rm" tool would take minutes, sometimes even hours, to remove the tree.

    In such a setup the most efficient option is to not delete at all but putting the cache on a separate file system and reformat when needed.

    Cheers,
    _

Similar Threads

  1. Read huge files effectively
    By MrAnderson1983 in forum Qt Programming
    Replies: 3
    Last Post: 28th April 2014, 09:18
  2. Huge number of images inside a panel
    By torsak in forum Qt Programming
    Replies: 4
    Last Post: 17th October 2013, 23:09
  3. QWebView print() generates huge PDF files on Windows
    By brush.tyler in forum Qt Programming
    Replies: 2
    Last Post: 26th September 2012, 15:01
  4. Huge Mac .dmg files, can anything be done to make them smaller?
    By colin207 in forum Installation and Deployment
    Replies: 3
    Last Post: 10th February 2011, 09:06
  5. Huge error list from qt files
    By Raccoon29 in forum Qt Programming
    Replies: 4
    Last Post: 8th September 2008, 16:22

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.