QGraphicsScene/QGraphicsView performance problems

[bnilsson]

Hi!

I am trying to write a microchip CAD datafile display program using the QGraphics View framework. It is the first time i use this, and it seems to be very convenient, very little (of my own) code is needed to do what I want to do.
However, my data files may range from small to quite large, anything between a few 100kb and up to a few 100Mb. At about 20Mb, my application begins to choke, i think mainly due to memory consumption. A Linux amd64 dual core system gets completely stuck when loading a 40Mb file, I had to terminate to avoid a complete system crash.
My MacOSX ppc G5 dual core behaves just the same.
I have not tried Windows yet.

I use Qt4.3.3 and recently Qt4.4.0 snapshot, same results on each.

I have a feeling there is a lot of stuff in the QGraphics View that is not needed for my app, I just need to draw, scroll and zoom, no select, move or transform.

Is the QGraphics View framework the right thing to use for me?
If so, can I turn off some of the bells and whistles to gain performance?
Or should I use some different methods?

Any suggestions are appreciated.

[wysota]

If it chokes while loading the data, there is a chance the slowdown is not related to graphics view at all but instead to the loading routine. Use a profiler to find the bottleneck of the application. Also if you have multiple cores, try to design your application in such a way that all cores are used (especially when loading the data).

[bnilsson]

Just realized how not to use itemized data, so I am drawing in myQGraphicsView::drawBackground() instead. Now it doesn't use much memory, so I can load large files.
Now the problem is back to performance speed again, and I need to check in my own code wether the shapes are in the visible view or not, and find a way to sift through the data real fast. Also, the scrolling is directly depending on how fast I can draw.
I found that it was impossible to draw a mouse rubberband if GraphicsView was not cashed.
Is there a way to cache the background drawing?

[wysota]

Just realized how not to use itemized data, so I am drawing in myQGraphicsView::drawBackground() instead.

But now you are not using the graphics view architecture at all. You could as well use a plain QWidget instead.

[bnilsson]

Yes, I realize that. Yet the way of using methods of zooming in/out and the scrollers are the same. which is convenient for me right now. I am only a Qt beginner, so I would need to study how to do the same with more "standard" methods. Will I gain much performance if I change to QWidget? I suspect I would never get it working for a 150Mb data file if I use QGraphicsItems with QGraphicsScene/QGraphicsView.

BTW, I am using QDataStream to repeatedly read my data, even for updates. I guess this is not terribly efficient. Can you recommend another way?

[wysota]

Will I gain much performance if I change to QWidget?

You'll lose the overhead caused by everything that QGraphicsView has and QWidget has not.

I suspect I would never get it working for a 150Mb data file if I use QGraphicsItems with QGraphicsScene/QGraphicsView.

I'm not so sure. One graphics item takes about 50 bytes of memory on average. How many items would you have in a 150MB file? 1M? That's less than 100MB of RAM used for the items.

BTW, I am using QDataStream to repeatedly read my data, even for updates. I guess this is not terribly efficient. Can you recommend another way?

Hard to say without knowing what your data represents. But if you do that constantly then that is surely inefficient.

[pherthyl]

Is there a way to cache the background drawing?

view.setCacheMode(QGraphicsView::CacheBackground);

view.setCacheMode(QGraphicsView::CacheBackground);

To copy to clipboard, switch view to plain text mode 

But really you need to provide some more information. How many items, how complex are these items? Drawing them manually in drawbackground is not really the best way to go about it.

[bnilsson]

Typically, I might want to display a few thousand up to 10-20 million trapezoidal shapes, either as filled or empty, with different colours.

When I have used itemized drawing, I do

if(fill) scene->addPolygon(Trapezoid,AreaShotPen,AreaShotBrush);
	else scene->addPolygon(Trapezoid,AreaShotPen);

if(fill) scene->addPolygon(Trapezoid,AreaShotPen,AreaShotBrush);
	else scene->addPolygon(Trapezoid,AreaShotPen);

To copy to clipboard, switch view to plain text mode 

or

if(fill) scene->addRect(outRect,AreaShotPen, AreaShotBrush);
	else scene->addRect(outRect,AreaShotPen);

if(fill) scene->addRect(outRect,AreaShotPen, AreaShotBrush);
	else scene->addRect(outRect,AreaShotPen);

To copy to clipboard, switch view to plain text mode 

This should be simple, but about 1 million shapes seems to be the limit right now, both for memory consumption and the patience of the user.

I am using QRectF and QPolygonF and true microns as internal numbers, would it improve performance if I use integer QRect and QPolygon instead?

[wysota]

I wouldn't use polygons here. You have a pretty simple shape, you can create your own item class which will be much faster.

[bnilsson]

Quote:
I suspect I would never get it working for a 150Mb data file if I use QGraphicsItems with QGraphicsScene/QGraphicsView.
I'm not so sure. One graphics item takes about 50 bytes of memory on average. How many items would you have in a 150MB file? 1M? That's less than 100MB of RAM used for the items.

I have a hard time believing that. A file of 600,000 shapes costs me about 500Mb of additional RAM, which indicates almost 1kb per item. I must be doing something seriously wrong....

I wouldn't use polygons here. You have a pretty simple shape, you can create your own item class which will be much faster.

Sorry, I am too new to Qt to implement such a thing. Could you direct me to some example?
Also, polygons are mostly only about 10-20% of the total population, the rest are rectangles. Still worthwile to make a new class?

[wysota]

I have a hard time believing that. A file of 600,000 shapes costs me about 500Mb of additional RAM, which indicates almost 1kb per item. I must be doing something seriously wrong....

Because you are using QGraphicsPolygonItem. You can probably use something simpler instead.

Sorry, I am too new to Qt to implement such a thing. Could you direct me to some example?

See examples bundled with Qt.

Also, polygons are mostly only about 10-20% of the total population, the rest are rectangles. Still worthwile to make a new class?

Yes, because they are the most heavy of your items so they probably occupy about 50% of the memory used by items. A simple esitmate of the memory can be calculated as follows:
total_per_item = sizeof(QGraphicsItem) + sizeof(data_stored_within_item) where the latter is:

in case of a rect item: sizeof(QRectF) + sizeof(QPen) + sizeof(QBrush) = sizeof(QPointF)+sizeof(QSizeF) + sizeof(QPen)+sizeof(QBrush) = 2*sizeof(double) + 2*sizeof(double) + ...

in case of a polygon item: sizeof(QPolygonF) + sizeof(QPen) + sizeof(QBrush) = sizeof(QVector<QPointF>) + ... = n * sizeof(QPointF) + ... = n*2*double + ... = at least twice as much as for QRectF with n>=4.

So a rect item data occupies probably about 70-80 bytes and a polygon item probably around 100. Add all the additional stuff kept by all the classes to that and multiply by the number of items.

BTW. Also consider using fewer more complex items instead of more simpler ones. Maybe you can "connect" your trapezoids into more complex shapes?

[bnilsson]

Thanks for your engagement, but I don't seen go get anywhere.
I managed to implement a subclass of QGraphicsItem, but it does not draw, I have no idea why. Could you help me?

#ifndef SHAPEITEM_H
#define SHAPEITEM_H
 
#include <QtGui> 
#include <QGraphicsItem>
#include <QObject>
 
class ShapeItem : public QGraphicsItem
{
 
 public:
				ShapeItem(void);
				ShapeItem(const QRect *rect);
				ShapeItem(const QPolygon *poly);
 
	QRectF		boundingRect() const;
	QPainterPath shape(void) const;
 
	virtual void		paint(QPainter *painter, const QStyleOptionGraphicsItem *option,
						QWidget *widget);
private:
 
	int x0,y0,x1,y1,x2,y2,x3,y3;
	int xmin,ymin,xmax,ymax;
 };
 
#endif

#ifndef SHAPEITEM_H
#define SHAPEITEM_H

#include <QtGui> 
#include <QGraphicsItem>
#include <QObject>

class ShapeItem : public QGraphicsItem
{

 public:
				ShapeItem(void);
				ShapeItem(const QRect *rect);
				ShapeItem(const QPolygon *poly);

	QRectF		boundingRect() const;
	QPainterPath shape(void) const;

	virtual void		paint(QPainter *painter, const QStyleOptionGraphicsItem *option,
						QWidget *widget);
private:
	
	int x0,y0,x1,y1,x2,y2,x3,y3;
	int xmin,ymin,xmax,ymax;
 };

#endif

To copy to clipboard, switch view to plain text mode 

#include "shapeitem.h"
 
ShapeItem::ShapeItem(const QRect *rect)
	: QGraphicsItem()
{
	x0 = rect->x();
	y0 = rect->y();
 
	x1 = rect->x() + rect->width();
	y1 = y0;
 
	x2 = x1;
	y2 = y0 + rect->height();
 
	x3 = x0;
	y3 = y2;
 
	xmin = x0;
	xmax = x1;
	ymin = y0;
	ymax = y2;
 
}
 
ShapeItem::ShapeItem(const QPolygon *poly)
	: QGraphicsItem()
{
	poly->point(0,&x0,&y0);
	poly->point(1,&x1,&y1);
	poly->point(2,&x2,&y2);
	poly->point(3,&x3,&y3);
 
	xmin = xmax = x0;
	ymin = ymax = y0;
 
	if(x1<xmin) xmin = x1;
	if(x2<xmin) xmin = x2;
	if(x3<xmin) xmin = x3;
 
	if(y1<ymin) ymin = y1;
	if(y2<ymin) ymin = y2;
	if(y3<ymin) ymin = y3;
 
	if(x1>xmax) xmax = x1;
	if(x2>xmax) xmax = x2;
	if(x3>xmax) xmax = x3;
 
	if(y1>ymax) ymax = y1;
	if(y2>ymax) ymax = y2;
	if(y3>ymax) ymax = y3;
 
}
 
 
QRectF 
ShapeItem::boundingRect() const
{
         qreal penWidth = 1;
         return QRectF(xmin - penWidth / 2, ymin - penWidth / 2,
                       xmax + penWidth / 2, ymax + penWidth / 2);
}
 
QPainterPath
ShapeItem::shape() const
 {
     QPainterPath path;
     path.addRect(boundingRect());
     return path;
 }
 
 
void
ShapeItem::paint(QPainter *painter, const QStyleOptionGraphicsItem *option,
                QWidget *widget)
{
         painter->drawLine ( x0,y0,x1,y1 );
		 painter->drawLine ( x1,y1,x2,y2 );
		 painter->drawLine ( x2,y2,x3,y3 );
		 painter->drawLine ( x3,y3,x0,y0 );
}

#include "shapeitem.h"

ShapeItem::ShapeItem(const QRect *rect)
	: QGraphicsItem()
{
	x0 = rect->x();
	y0 = rect->y();
	
	x1 = rect->x() + rect->width();
	y1 = y0;
	
	x2 = x1;
	y2 = y0 + rect->height();
	
	x3 = x0;
	y3 = y2;
	
	xmin = x0;
	xmax = x1;
	ymin = y0;
	ymax = y2;

}

ShapeItem::ShapeItem(const QPolygon *poly)
	: QGraphicsItem()
{
	poly->point(0,&x0,&y0);
	poly->point(1,&x1,&y1);
	poly->point(2,&x2,&y2);
	poly->point(3,&x3,&y3);
	
	xmin = xmax = x0;
	ymin = ymax = y0;
	
	if(x1<xmin) xmin = x1;
	if(x2<xmin) xmin = x2;
	if(x3<xmin) xmin = x3;

	if(y1<ymin) ymin = y1;
	if(y2<ymin) ymin = y2;
	if(y3<ymin) ymin = y3;

	if(x1>xmax) xmax = x1;
	if(x2>xmax) xmax = x2;
	if(x3>xmax) xmax = x3;

	if(y1>ymax) ymax = y1;
	if(y2>ymax) ymax = y2;
	if(y3>ymax) ymax = y3;

}


QRectF 
ShapeItem::boundingRect() const
{
         qreal penWidth = 1;
         return QRectF(xmin - penWidth / 2, ymin - penWidth / 2,
                       xmax + penWidth / 2, ymax + penWidth / 2);
}

QPainterPath
ShapeItem::shape() const
 {
     QPainterPath path;
     path.addRect(boundingRect());
     return path;
 }


void
ShapeItem::paint(QPainter *painter, const QStyleOptionGraphicsItem *option,
                QWidget *widget)
{
         painter->drawLine ( x0,y0,x1,y1 );
		 painter->drawLine ( x1,y1,x2,y2 );
		 painter->drawLine ( x2,y2,x3,y3 );
		 painter->drawLine ( x3,y3,x0,y0 );
}

To copy to clipboard, switch view to plain text mode 

Not at all optimized, as soon as it draws me anything in the scene, I will test different variants. At this stage, numItems=0 in myQGraphicsItems::drawIntems, so I'm stuck.

I am using it lile this:

ShapeItem item(&outRect);
	scene->addItem(&item);
	//scene->addRect(outRect,solidBlue);

ShapeItem item(&outRect);
	scene->addItem(&item);
	//scene->addRect(outRect,solidBlue);

To copy to clipboard, switch view to plain text mode 

The commented-out line draws as it should, while the active lines does not.

[wysota]

Create the item on heap.

ShapeItem *item = new ShapeItem(...);
scene->addItem(item);

ShapeItem *item = new ShapeItem(...);
scene->addItem(item);

To copy to clipboard, switch view to plain text mode 

[bnilsson]

Thanks, now it is working.
Unfortunately, now the app started gobbling up all memory even for a smaller file, so it got worse. I will have to figure out why...

[jacek]

Unfortunately, now the app started gobbling up all memory even for a smaller file, so it got worse. I will have to figure out why...

Use valgrind. The Massif tool might be especially helpful.

[bnilsson]

Use valgrind. The Massif tool might be especially helpful.

For the moment I'm on MacOSX, so this is not applicable.

[bnilsson]

The behaviour was a bit better using Qt4.4.0 than Qt4.3.3, so maybe it's not in my code entirely. But still in the 1Gb range for a 40Mb file.
The behaviour was very strange indeed (excessive cpu and memory load when zooming IN) on my amd46 Debian using Qt4.3.3, and this particular problem was fixed by the Qt4.4.0 snapshot.

Anyway, standard usage of QGraphicsView/QGraphiscScene was not dramatically improved using Qt4.4.0.

[wysota]

Could you provide a minimal compilable example reproducing the problem so that we might try it ourselves?

[bnilsson]

Sure, the project is not very big, I will try to prepare something.
I am new to this forum, what are the provisions for uploading?

[wysota]

Use the attachments feature ("Manage attachments" button below the advanced editor).