Implimentation advice for a large data plotting application.

[Uwe]

If I understand correctly then, I need to add a type of filtering class to my data class to return effectively a lower resolution version of the data it internally stores to the QwtPlot (or QwtPlotCurve not sure which)?

Inside of your data object. I recommend to connect something to the QwtScaleWidget::scaleDivChanged() signals to activate the best resolution for the current scale ranges.

You suggested I looked at the QwtWeedingCurveFitter, however I don't seem to be able to find this, where should I be looking?

qwt_curve_fitter.[h|cpp] - didn't you write, that you are on trunk ?

I was also wondering if there is a description somewhere as to how the QwtPlotZoomer works with a QwtPlot and its associated QwtPlotCurve classes (and ultimately my data class derived from QwtSeriesData).

The zoomer manipulates the scales - your curves and your data object are completely unrelated. Note that the plot has current scale ranges, but the zoom stack is an internal thing of the zoomer. For rendering the content of the plot canvas it is unimportant if the scales are the result of a zoom operation or not.

... however I was a little surprised when the sample index that my data class was asked for always ranged over the entire data set. I had expected the zoom area to have been the bound on this.

In general the index of a point has nothing to do with its coordinates ( start the realtime example, zoom in and you will understand immediately ) !

I guess in your specific situation the values are ordered, what could be used to optimize painting but using specific characteristics of a series can only be implemented, where these are known: your data object.

Why is this not the first and last point identified from the zoomer?

Again, there are not first and last points according to coordinates. But even coordinates, that are outside of the scale ranges might be important: f.e. in case of QwtPlotCurve::Lines the connecting line might be intersecting the visible area.

Is it the QwtPlot that discards data points which fall outsize of its zoom range

QwtPlotCurve does some optimizations like polygon clipping or discarding points depending on the type of visualization. But it can't do it without knowing the points.

Uwe

[mike_the_tv]

qwt_curve_fitter.[h|cpp] - didn't you write, that you are on trunk ?

Yes I am on the trunk and I searched for QwtWeedingCurveFitter using Visual Studios search in files, but for some reason it found nothing. Now I go and look for the file you say I can find it. Not quite sure what went on there .

In general the index of a point has nothing to do with its coordinates ( start the realtime example, zoom in and you will understand immediately ) !

I guess in your specific situation the values are ordered, what could be used to optimize painting but using specific characteristics of a series can only be implemented, where these are known: your data object.

OK point taken. The realtime example plots random point whos position in the plot area bares no relationship to its position in the data set. I guess I was assuming that a data object based on a QwtSeriesData class may make some assumptions about the ordering or type of data it contained and therefore optimise access at different zoom/crops.

Again, there are not first and last points according to coordinates. But even coordinates, that are outside of the scale ranges might be important: f.e. in case of QwtPlotCurve::Lines the connecting line might be intersecting the visible area.

Again I had not considered this as you would need to know points outside the visible area in order to correctly plot the first and last visible points.

Inside of your data object. I recommend to connect something to the QwtScaleWidget::scaleDivChanged() signals to activate the best resolution for the current scale ranges.
...
The zoomer manipulates the scales - your curves and your data object are completely unrelated. Note that the plot has current scale ranges, but the zoom stack is an internal thing of the zoomer. For rendering the content of the plot canvas it is unimportant if the scales are the result of a zoom operation or not.
...
QwtPlotCurve does some optimizations like polygon clipping or discarding points depending on the type of visualization. But it can't do it without knowing the points.

I am still confused about this and I think it comes from not fully understanding the interaction between the various components within a plot. If you can bare with me on this I would really appreciate a quick summary of how things connect and interact (or please if there is a concise summary document somewhere I have missed point me at it so I don't waste your time .

What I think I understand at the moment is this. If I want to plot some data I need a QwtPlot object, or at least something derived from this. I am not sure at the moment whether deriving from QwtPlot is necessary but most of the examples do.

The to provide my data to the QwtPlot I need one or more QwtPlotCurve objects which I attach to my QwtPlot. My QwtPlotCurve is supplied with data via a QwtSeriesData object, which in my case is a derived data class based on QwtSeriesData that returns QPointF() values for samples specified by the sample index with a call to the sample() member.

From tracing with the debugger I can see that when my QwtPlot wants to plot my data, the sample() member of my derived data class is called for each data point, QwtPlot then does something with this data to cull points outside the current view, but ultimately results in my data appearing on screen.

Where my understanding now breaks down is how zooming and panning (preferably scrolling) fits in with all this. I also don't understand how the weeding out of excess data is going to work, even now I have found the weeding class. I have done similar data plotting in a previous version of my application. In this I used MFC and DirectX. Now in this set up if my plot width, in pixels, was less than the number of data points to display, i.e. we have way more detail than can be physically plotted, my code would step over each display pixel, calculate how many data samples this single pixel was representing, and then calculate a single value for all those data points to display at that pixel location. In my case I did a min/max and plotted a vertical line to indicate the min/max extent on the data.

Now the way QwtPlot works is probably completely different to what I have done previously. I have tried looking at the QwtWeedingCurveFitter() class and attached it to my QwtPlotCurve class via setCurveFitter(), however when I debug the code to work out how things connect and interact, I don;t see any of the weeding class members being called. How and where does this class get called?

I also don't really understand how the zooming works. You had said that the QwtPlotCurves and my data object are unrelated to the zoomer, this just affecting the scales on the QwtPlot I assume. You also said that I should use the QwtScaleWidgets scaleDivChanged() signal to select the best data resolution, however this signal does not seem to supply any parameters about the scale change, so I am not quite sure how I should use this notification.

The sort of effect I am after in the end is similar to your realtime example, except that the data will be available in one go and will be a linear time series plot. The ability to zoom and then pan the plot is what I need to match my original application under MFC/DirectX, however I don't really understand how to achieve this. When I tried to follow what the example was doing I quickly got lost because I think I am missing this fundemental understanding of how the various bits fit together.

Apologies for such a long reply, and many thanks so far for your assistance.

[Uwe]

I am still confused about this and I think it comes from not fully understanding the interaction between the various components within a plot.

Looks like you are trying to understand something, that is not there. Things are much simpler, than you expect:

1) The plot widget knows the transformations and scale ranges and passes them to the internal render engine. But it doesn't know much about the plot items - beside a couple of informations accessible via the QwtPlotItem base class.

2) There are 2 types of plot items: decorators ( like the grid or scales items ) or items representing some type of data. F.e. a curve represents a series of 2D points, a spectrogram represents raster data.

3) Zoomer, panner and other helper objects translate user interactions into scale changes. The plot widget doesn't know these helpers and it is of no importance if the scales were set from a zoomer, the autoscaler or are assigned from your code ( setAxisScale() ).

I have done similar data plotting in a previous version of my application. In this I used MFC and DirectX. Now in this set up if my plot width, in pixels, was less than the number of data points to display, i.e. we have way more detail than can be physically plotted, my code would step over each display pixel, calculate how many data samples this single pixel was representing, and then calculate a single value for all those data points to display at that pixel location. In my case I did a min/max and plotted a vertical line to indicate the min/max extent on the data.

You can do this as well (overloading QwtPlotCurve), but Douglas Peucker has more potential as you don't need the widget geometry. You can resample your points in advance and don't need to slow down resize ( or even worse repaint ) operations.

Now the way QwtPlot works is probably completely different to what I have done previously. I have tried looking at the QwtWeedingCurveFitter() class and attached it to my QwtPlotCurve class via setCurveFitter(),

This way ( you forgot to enable the QwtPlotCurve::Fitted attribute ) the curve resamples the points after they have been translated into widget coordinates each time your curve is painted. Because Douglas Peucker is an expensive algo this will have a negative effect and slow down your repaints.

Instead I recommend to use QwtWeedingCurveFitter to calculate a couple of arrays with less points from your series only once - when you assign your samples. You find your arrays by increasing the tolerance parameter until the number of points is below a certain limit ( f.e 1000 points ). A good step size for the tolerance depends on the bounding rect of your series, but I guess you don't need to have more than 5 arrays.

Then the only thing you need to do is to activate one of your arrays depending on the current scale ranges ( f.e. in a slot connected to scaleDivChanged() ). You can find the current scale divisions by plot->axisScaleDiv(...). Of course you can add some additional optimizations (calculating start/end indices) using the order of your points here too.

When the plot displays large scale ranges your data object will return less points from a much smaller array. If the scale ranges are small your data object will return many points, but most of them will be clipped. In both situation the number of paint operations is heavily reduced and you will see a good performance for painting your plot. ( Implementing a such a data object calculating different level of details internally is on my TODO list. )

Uwe

[darkforce]

When the plot displays large scale ranges your data object will return less points from a much smaller array. If the scale ranges are small your data object will return many points, but most of them will be clipped. In both situation the number of paint operations is heavily reduced and you will see a good performance for painting your plot. ( Implementing a such a data object calculating different level of details internally is on my TODO list. )
Uwe

The implementation of the Douglas-Puecker algorithm in QwtWeedingCurveFitter works perfectly and the painting performance gets much better with large datasets. However, when I scroll the plot (using a scrollzoomer), the algorithm is called unnecessarily even though the level of resolution did not change. This could pose a problem when really large datasets are loaded. Another feature that could be useful is to allow users to control the display of symbols either on the original data points or on the reduced data points. Maybe it is already implemented somewhere, I just did not do any research on that. Thank you for the great job!

[Uwe]

However, when I scroll the plot (using a scrollzoomer), the algorithm is called unnecessarily even though the level of resolution did not change.

It's your code, that decides, when to rebuild the internal datasets. As long as those doesn't use clipping I can't see any reason why you should rebuild them, when the resolution doesn't change significantly.

Another feature that could be useful is to allow users to control the display of symbols either on the original data points or on the reduced data points. Maybe it is already implemented somewhere

You would have to overload QwtPlotCurve::drawSymbols(), but unfortunately it's less comfortable as it sounds, because there is some substance in the implementation of the base class you would have to copy.

Uwe