how to manipulate a very large data using QT?

[aurora]

Hi all,
I have a very large data and i want to manipulate(displaying requested part of data) those data providing Qt User interface.
How do i start for this?
What are the requirements?
Do i need to use any database drivers like SQLite or something?

[Cruz]

How large is large and in what format is your data currently stored?
Is it in a database (which one?), or binary files, text files, punch cards...

[ChrisW67]

What are the requirements?

Requirements are the things your customers give you directly, or that you derive from customer requirements. We cannot answer this question.

"A very large data" is completely undefined. I used to work with databases that grew by 121 million records a day, they were at the small end of large then and are positively tiny by today's standards. Some people think of large as, "anything too big to fit on on Excel spreadsheet page," while others consider large as "greater than a terrabyte in a pile." Where does your requirement (there's that word again) fall on the scale?

[aurora]

How large is large and in what format is your data currently stored?
Is it in a database (which one?), or binary files, text files, punch cards...

Each table may be around 1gb....
all are just text data....

Requirements are the things your customers give you directly, or that you derive from customer requirements. We cannot answer this question.

"Requirements"...i mean to say if i'am using any database for storing text data which kind of database i should use? what are the configuration i need to do?
Does users also need to install any database to use the application i developed?
I am using "Qt Creator 2.3.0 Based on Qt 4.7.4 (32 bit)" which are the plugins required to start? How to get them?

[Talei]

QtCreator is an IDE, just a fancy text editor nothing more.
And there is no plugin for what You are looking for.

If you want to access a file, then use QFile.seek(pos); and seek to the position that You are interested in, read i.e. 1MB of the data and do something with it.
If you want to "organize" data into something more flexible, that can gave You faster access to the data, then use i.e. SQLite3 database. It's file base database and, when created properly, is fast enough.

PS> sorry for vague reply but Your question is not specific.

[ChrisW67]

Each table may be around 1gb....
all are just text data....

• Is that one 1 gb blob of text or 100 million little bits of text? (War and Peace vs. a phone book?)
• How many "tables" of 1GB?
• What do you need to be able to do with that text?
• How quickly?
• If there are some kind of "records" in here how do they relate to each other?
• Does the data need to be shared between multiple users concurrently? How many?
• Is the data read-only or updatable?
• Does the data need to be controlled/protected by some sort of permissions system?
• Do you need to be able to distribute the database data or processing load between machines or sites?

"Requirements"...i mean to say if i'am using any database for storing text data which kind of database i should use?

Any you care to choose that meets your needs; see list above for some obvious starting questions. Sqlite will handle one gigabyte database and can do free-text searching but if you need features toward the bottom of the list then a separate RDBMS system is a better choice.

what are the configuration i need to do?

To the database system: something between nothing (for Sqlite) to substantial configuration for an Oracle database with replication.

Does users also need to install any database to use the application i developed?

Maybe. For Sqlite then nothing other than Qt components is required. For other RDMS systems at least the database client libraries will be required, and of course someone has to install and configure a central server.

I am using "Qt Creator 2.3.0 Based on Qt 4.7.4 (32 bit)" which are the plugins required to start? How to get them?

In the case of Sqlite the plugin is built by default. For others supported by Qt you need to build the Qt SQL plugin matching your chosen database engine (from the Qt sources, following the general instructions). For others not supported by Qt you can write your own driver.

[aurora]

Thanks a lot Chris...

• Is that one 1 gb blob of text or 100 million little bits of text? (War and Peace vs. a phone book?)
• How many "tables" of 1GB?
• What do you need to be able to do with that text?
• How quickly?
• If there are some kind of "records" in here how do they relate to each other?
• Does the data need to be shared between multiple users concurrently? How many?
• Is the data read-only or updatable?
• Does the data need to be controlled/protected by some sort of permissions system?
• Do you need to be able to distribute the database data or processing load between machines or sites?

.

• Its 100 million lines all with same length
• Hundreds of tables around 1gb
• I should able to search in those tables(basically)
• Speed should be reliable
• No concurancy
• All tables are read only
• Not distributed system...just for single stand alone machine

[wysota]

If you have text files, why don't you use those files directly? What's the benefit of moving the data into separate storage? What kind of "search" do you have in mind? Words? Patterns? Records? Is the content in each file ordered somehow?

[aurora]

If you have text files, why don't you use those files directly? What's the benefit of moving the data into separate storage? What kind of "search" do you have in mind? Words? Patterns? Records? Is the content in each file ordered somehow?

my target was to load a 1gb text file in table widget and to perform searching operation on those files, but i was unable to perform that operation.
So i thought, if i can store it in a database i can show part of the file(or the user requested part ) in the table view..
Is the way i have choosen is wrong? is there any alternative to perform this?

[ChrisW67]

my target was to load a 1gb text file in table widget and to perform searching operation on those files, but i was unable to perform that operation.

It was always a stretch to get a 1GB collection of lines into memory for a single widget on a 32-bit machine. Even less likely if there are "Hundreds of tables around 1gb," you tried to load.

So i thought, if i can store it in a database i can show part of the file(or the user requested part ) in the table view..

Possibly, we cannot tell. You still haven't told us anything about the structure of the file content, how the user might search it, what constitutes a "part of the file", etc. It might be something like:

• Open file, seek to a byte offset, read a 4kB "page", and display that, or
• Open file, seek to a particular line number, read a specified number of lines, and display that, or
• Open file, select all paragraphs matching a pattern, and display those, or
• Open file, select all lines matching a pattern, and display 5 lines either side of each, those, or
• Open file, select all lines matching a pattern, interpret the data in the lines to compute whether to display them, and display those, or
• Open file, select all lines matching a pattern, find a set of related lines, and display those,
• Open a file, open another, find the common/different lines, and display those, or...

I don't know how you expect us to know.

Is the way i have choosen is wrong? is there any alternative to perform this?

It is neither right nor wrong. There are plenty of options but only you have the information on which to assess suitability.

[wysota]

So i thought, if i can store it in a database i can show part of the file(or the user requested part ) in the table view..

I fail to see what a database would help you with here. A text file is also a sort of a database. If you want to visualize part of the file then do just that -- load the part you need and display whatever you need.