Developing model/view for different type of objects (Long text - conpcetual question)

I am having some problems to model my data using QAbstractItemModel. I have read Model/View Programming page, and while I understand how it works I still dont know how to adapt this to my project. I also have searched for examples/open source programs but no luck.

I am working on python using PySide2 but I dont have problems reading c++ code. I will put a link at the end to my github project. Because of this question, the project is still a prototype, so dont expect high quality code.



What is this project about:
I am developing a tool to reverse engineering network packets, something like wireshark but different with scope. I need to import a file and show that data to the user. Each line on that file represents a "network event" (socket recv/send function).
A line look like this:

Code:

---

{""timestamp":"2019-Jan-27 05:34:40", "server":"192.240.44.240:27701", "source":"Server", "packets":["0500020001230300420100", "0788F276A1"]}

---

So basically a NetworkEvent consist of a timestamp, server address, source and a set of packets.
Each packet is a struct and will be parsed using autogenerated code that I dont have control. A packet or it nested structs is a:
StructItem:
- Name
- Value (can be null)
- Root (struct in the top of this hierarchy. Will always be a packet)
- Parent (parent struct)
- Members (nested structs. Emtpy if this struct represents a primitive data type)
- Start Pos (position of this structure in the packet byte array. Null in root)
- End Pos (end position of this structure in the packet byte array. Null in root)
- Byte array (only present in root. Null in childs)
- NetworkEvent (network event which this structure is part of)

Based on this data, I want to show to the user:
- A table/list (QTreeView) with all network events, with their respective timestamp, server address, source and number of packets.
- A tree (QTreeView) showing structure of packets from current selected NetworkEvent on the table.
- A hex viewer (QAbstractScrollArea/QAbstractItemView) showing binary data of current selected packet.

Features I want to implement:
1- In the hex viewer, hightlight bytes related to the selected field on the tree view. If hex viewer data doesnt correspond to clicked node root (Packet), update hex viewer data. (This is why I want to use QAbstractItemView for hex viewer)
2- In the tree view, select a node which correspond to the clicked bytes on the hex viewer.
3- TextEdit to filter Packets (and so on NetworkEvents) based on any packet field (or fields on its nested structures).
4- TextEdit to Filter NetworkEvents based on its fields.
5- Open a non-modal dialog to show a packet object tree and bytes (hex viewer), supporting features 1 & 2.

Things I would like to implement at some point:
6- Have a option to remplace the EventList with a list of Packets or some common nested struct available in all packets (as it seems, a packet its compossed of messages. Message class is autogenerated, to the program, its just another StructItem.). This is desirable but not required.

Picture of the program
This is a picture of what my program looks right now:
Attachment 13017



The question: how should I implement QAbstractItemModel to support those features?

What I have done:
I will explain then what I have done, but it is possible that I overthinked (and that is why I am asking here), and this may make no sense. Feel free to skip.

Currently I have created a BaseModel : QAbstractItemModel that serves as a Tree. Its root index is a dummy node that has as childs the list of NetworkEvents. A NetworkEvent has as child a StructItem. A StructItem has as childs others StructItems (no childs in case of leafs).

Because my business model doesnt has a "childrens" field, I have created wrapper classes (NetworkEventNode and StructNode, both based on a base class Node) that hides this behavior, so to the BaseModel it is transparent. A StructItem in NetworkEvent.packetList is a child of a NetworkEvent. A StructItem in StructItem.members is a child of a given StructItem. Should the model interact directly with business model? In that case, BaseModel should be responsible to know which field of NetworkEvent represent its childs and which field of StructItem represent its childs.

Because columns in the List/Table (QTreeView) arent common to all nodes in the BaseModel, I created a EventListModel (proxy model based on QIdentityProxyModel) that modifies BaseModel behavior to work as a table and provides access to NetworkEvent object members.
Same goes for the Object tree (QTreeView). It only shows nested StructItems, but its rootIndex is a NetworkEvent(Node), so I created a ObjectTreeModel (proxy model based on QIdentityProxyModel) that is the responsible to provide support to that Object Tree.
Currently, the hex viewer is not based on model/view but I want to change that, so it is possible to use QModelIndexes to communicate with Object Tree.

To implement the filter, I would have to insert another proxy model (QSortFilterProxyModel) between current proxy models and base model.

Now I dont feel confident continuing with this. I feel like I am overthinking and not sure if this is the way to do what I need. Should I separate the "BaseModel" in two different models? One that supports the list and another that supports object tree? What about the future HexViewer based on QAbstractItemView, should it have its own model or just have as root index the QModelIndex of selected packet in object tree?

I would like if anyone with more experience can guide me to the right path.


Link to my project: https://github.com/FYGonzalo/packet_analyzer

My suggestion would be to do separate models, each representing your data structure according to its specific needs.

Start with the table model as it is far easier to implement than a tree model.

For communication between views I would suggest to make your nodes addressable in some way that does not rely on QModelIndexes from the same base model instance.
That way you don't need to make the hex viewer an abstract item view just for addressing data.

I would even consider doing the filtering in those models that need it. Can often be implemented much more efficiently with direct access to data

Cheers,
_

Quote:

---

Originally Posted by anda_skoa

My suggestion would be to do separate models, each representing your data structure according to its specific needs.

Start with the table model as it is far easier to implement than a tree model.

For communication between views I would suggest to make your nodes addressable in some way that does not rely on QModelIndexes from the same base model instance.
That way you don't need to make the hex viewer an abstract item view just for addressing data.

I would even consider doing the filtering in those models that need it. Can often be implemented much more efficiently with direct access to data

Cheers,
_

---

I figured out that I had a lot spaghetti code. I can't get in words what was wrong.

After thinking a lot I have a better view of the problem. Basically what I do now is, for the underlying data, manually creating a tree structure based on what I want to show on the screen. Now, I have a "complex" method that is responsible to know how to create the tree structure based on my internal data, whatever that data is it. So now, in other words, the tree structure acts as a "view" to the internal data, and as a "model" for the QAbstractItemModel.

To the model (QAbstractItemModel), that tree structure is the data. Nodes hides behavior of the internal data. Simple as that. A node is a class with a name, value, parent, children. In the future I will add other attributes to the node (bytes, start, end), but that doesn't matter now. The tree structure expects that each node has a name, a value, a parent and zero or more children. Whatever else it has, doesn't matter. So NetworkEvent nodes will not have "start", "end" or "bytes" attributes, to the tree that is indifferent.

So this is an example of the tree structure behind the model.

Code:

---

Node(name='Network Events', value=object<list>)
|-- Node(name='0', value=object<NetworkEvent>)
|   |-- Node(name='timestamp', value='2019-Jan-13 01:34:27')
|   |-- Node(name='server', value='195.232.44.140:5005')
|   |-- Node(name='source', value='Server')
|   +-- Node(name='packets', value=object<list>)
|       +-- Node(name='0', value=object<Packet>)
|           |-- Node(name='header', value=object<Header>, start=0, end=5, bytes=...) ... (continues expanding)
|           +-- Node(name='content', value=object<Content>, start=6, end=., bytes=...) ... (continues expanding)
|-- Node(name='1', value=object<NetworkEvent>)
|   |-- Node(name='timestamp', value='2019-Jan-13 01:34:27')
|   |-- Node(name='server', value='195.232.44.140:5005')
|   |-- Node(name='source', value='Client')
|   +-- Node(name='packets', value=object<list>)
|       +-- Node(name='0', value=object<Packet>)
|           |-- Node(name='header', value=object<Header>, start=0, end=5, bytes=...) ... (continues expanding)
|           +-- Node(name='content', value=object<Content>, start=6, end=., bytes=...) ... (continues expanding)

---

Now I don't understand why you would recommend to use multiple models instead of a single one with different proxy models, apart of maybe being a little more complex. While having different models makes things easier, it also has its cons. In the GUI, when a user selects a network event from the network event list view, it should show its packet tree on the QTreeView. Problem is, everytime I change the QTreeView model because a user selected a different NetworkEvent, it closes every expanded node. That leads to a poor user experience. So, if I expand packet 0 from network event 0, then I go to network event 1, and again to network event 0, now packet 0 is collapsed. I want it to remember expanded/collapsed nodes in the current session.

[ 1 ] Maybe what I am doing wrong is, setting only "packets" node as data of the treeview model, instead using the entire tree structure as treeview model and then selecting the QModelIndex corresponding to "packets" node of the clicked NetworkEvent. This way the QTreeView will know which packets were collapsed/expanded.


Another way I was thinking is defining a base model with the entire tree structure, then create a proxy model to generate the network event list. When the user press a NetworkEvent on the list, pass its "packets" children node to the QTreeView model, being it responsible to recover the corresponding QModelIndex to its model (proxy or base model, doesn't matter), similar to what you said about having ways to address nodes without QModelIndex objects.
This way I avoid making use of mapFromSource and mapToSource with QModelIndex when interacting between views elements, and instead, replicate this behavior internally but with nodes. So won't be calling QTreeView.setRootIndex but instead MyCustomTreeView.setRootNode(node), and internally it will execute MyCustomTreeViewModel.getIndexFromNode(node), then setRootIndex with that index.

But this approach, the more I think, it seems like having different models with the same entire tree in the background (different models, sharing the entire background data, as you suggest).


And I guess filtering with QSortFiltProxyModels would mean invalidating all QModelIndex's of models so it doesnt matter if I filter the QAbstractItemModel using the "qt" method or I implement the filter on the tree structure then change the model, both ways would mean that the view wouldn't know which nodes were expanded/collapsed.


So my new question is, is there any visual/user experience/whatever difference, between attacking this problem, with proxy models or different models that I should take in account thinking on what I may want to implement in the future? As I see now (after realizing
[ 1 ]), I guess there is not, apart of the ease of use of not having to mess with QModelIndex mapping.
I suppose those differences may appear when trying to add up other things to the problem, like syncing changes between different QAbstractItemModels that work over the same underlying data. But this is beyond my current knowledge.


Thanks for your time, I really appreciate it. Sorry for the confusing wall of text.

Quote:

---

Originally Posted by FYgonzalo

Now I don't understand why you would recommend to use multiple models instead of a single one with different proxy models, apart of maybe being a little more complex.

---

It is mostly about complexity, both in terms of code complexity and runtime/processing complexity.

Code complexity: an approach with a single model and proxies is often much more complicated to develop and maintain
Runtime complexity: a proxy model has no "knowledge", e.g. for filtering it always has to traverse the full source model. A specialized model can use "knowledge" about the data.

Quote:

---

Originally Posted by FYgonzalo

I want it to remember expanded/collapsed nodes in the current session.

---

Then you need to react to the respective signals of QTreeView and store the expand/collapse state in your internal data structure.

Cheers,
_