Hi there guys, for the past three months I have been developing a web crawler and everything seems to be working fine, but I feel my design of the entire application is not efficient at all. The application is in client-server architecture and the crawling functionality is performed by the server. I would like to adopt multi-threading(i.e. using the QThreadPool) in my design to increase efficiency. The application's basic functionality is to crawl 50 websites and retrieve data from the sites. At the moment the application is performing the crawling function serially (i.e. one website at a time) which is very time consuming. My server runs on a 3Mb/s internet connection, I would like the application to crawl 4 sites at a time. I don't need the code or anything like that, all I want is to pick your brain as to how you would approach such a problem. Thanking you in advance.
Bookmarks