QNetworkAccessManager + DiskCache => use Thread?

[Coolcat]

I use QNetworkAccessManager and related classes to stream texture and geometry data from an internet server. Using this data the application renders an interactive 3D-scene. When a button is pressed I use a QTimer at 10 ms interval to update the scene. There is only one thread, besides possible internal threads of QNetworkAccessManager. It is intended to get a smooth movement, while geometry and textures popup as soon as they are available.

To speedup the slow download from the server I'm using QNetworkDiskCache. However, when using a directory on my hard disk for caching I experience heavy lags every few seconds. This lags have a duration from 0.5 to 1 second. While I'm not sure about this, I assume this is caused by blocking disk accesses in my primary thread. To test the hypothesis I created a RAM drive and used it as cache location. Movement and animation are still not smooth, but these heavy lags are gone. (QNetworkDiskCache is the only location where I'm doing frequent disk accesses)

=> Does QNetworkDiskCache do a blocking hard disk access in the primary thread?
=> Could I solve the problem by creating my own Network-Thread which does handle everything related to QNetworkAccessManager?
=> Other suggestions?

Thanks in advance.

[wysota]

In my opinion you should implement your own subclass of QAbstractNetworkCache. The problem you are having is that the disk cache needs to update its indexes which is painfull if you have lots of data cached.

[Coolcat]

The problem you are having is that the disk cache needs to update its indexes which is painfull if you have lots of data cached.

Ok, that's possible. My cache is limited to 50 MB (default) and currently contains 7346 files / 45.3 MB. If it uses an inefficient index this is indeed the problem.

However, I checked the source code and QNetworkDiskCache seems to use just a SHA1 hash of the URL as filename. So this is a problem of the file system with too many files in a single directory, right? I will try if an updated method QNetworkDiskCachePrivate::cacheFileName(QUrl) which introduces two directory levels (first two chars from the hash) solves the problem. (Sadly the method is private...not protected....)

[wysota]

First question should be - do you need the cache to be persistent? If not then your best choice is to cache in memory.

[Coolcat]

I have implemented a non-persistent cache. Runs much faster since I'm using QHash as index. Also expire() does not scan the whole cache. I'm using a QLinkedList as index for the least recently used heuristic, where each cache item stores an iterator to its position in the list. So all operations are just O(1) in average.

Thanks