Results 1 to 4 of 4

Thread: Best way to handle very-very huge amount of data

  1. #1
    Join Date
    Jan 2009
    Location
    Russia
    Posts
    309
    Thanks
    2
    Thanked 43 Times in 42 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Best way to handle very-very huge amount of data

    Hi all,

    I have a system, where I need to plot a huge-huge amount of data points. For example I have device which sends points with 8125Hz frequency.
    In this case on 24 hours it is equal to 8125 * 60 * 60 * 24 = 702 000 000 points. Also each point has 8 byte size (double), so the data size is 5 616 000 000 bytes,
    without of timestamp.. And with the timespamps data it equals to 5 616 000 000 * 2 = 11 232 000 000 bytes.

    The use case is following:

    The User can view the trends as daily log (all points for 24 hour).
    Also the user can specify a desired time range to more detail viewing (e.g. from 10:10:10 to 10:10:11).

    So, is it possible to use Qwt to draw, this amount of data for each curve (as I have more than one of this devices)?
    What is idea to do that?

    As I understand, I need to use custom QwtSeriesData implementation...
    When the called QwtSeriesData::setRectOfInterest(), I need to cache some amount of points from the points storage (e.g. from the file or database) inside of QwtSeriesData.
    And then when will be called the QwtSeriesData::sample(), then I will return the sample from this cached data. So, in this case the QwtSeriesData::size()
    should return the size of current cache.

    When the QwtSeriesData::setRectOfInterest(hugeHugeRect) specified whole daily log (points from whole 24 hours), then I need to make caching of data with some factor,
    e.g. read data from storage, every 100/1000 and so on point (do not read each point of data from storage)...

    Do I understand the idea correctly or not?

    BR,
    Denis

  2. #2
    Join Date
    Feb 2006
    Location
    Munich, Germany
    Posts
    3,311
    Thanked 879 Times in 827 Posts
    Qt products
    Qt3 Qt4 Qt/Embedded
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Best way to handle very-very huge amount of data

    Well, QwtSeriesData decouples the plot from how the samples are stored. So its idea is, that you can do it in a way what is best for your application - without bothering you with an idea how to do it.

    Uwe

  3. #3
    Join Date
    Jan 2009
    Location
    Russia
    Posts
    309
    Thanks
    2
    Thanked 43 Times in 42 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Best way to handle very-very huge amount of data

    Hi Uwe,

    so.. I have implemented it with this way (I use 10 000 000 of points for example, also it works with 50 000 000 points):

    Qt Code:
    1. // SourceSamples
    2.  
    3. class SourceSamples : public QwtSeriesData<QPointF>
    4. {
    5. public:
    6. explicit SourceSamples();
    7.  
    8. private: // virtual overrides
    9. size_t size() const final;
    10. QPointF sample(size_t i) const final;
    11. void setRectOfInterest(const QRectF &rect) final;
    12. QRectF boundingRect() const final;
    13.  
    14. private:
    15. static size_t getFactor(size_t pointsCount);
    16.  
    17. void updateCache(const QRectF &rect);
    18.  
    19. private:
    20. // Emulates the data from the external storage (e.g. file or DB).
    21. QVector<QPointF> m_storage;
    22. // Set of samples for current details level.
    23. QVector<QPointF> m_cache;
    24. };
    25.  
    26. // Calculates factor which allow to have always ~500 points to draw.
    27. size_t SourceSamples::getFactor(size_t pointsCount)
    28. {
    29. const auto factor = pointsCount / 500;
    30. return (factor > 0) ? factor : 1;
    31. }
    32.  
    33. // Refills the internal samples cache for the required rect.
    34. void SourceSamples::updateCache(const QRectF &rect)
    35. {
    36. // Binary search indexes of minimum and maximum points for
    37. // the X-axis of passed rect in "external" data storage
    38. // (which is simulates by the m_storage in our cases).
    39. const auto begin = m_storage.cbegin();
    40. const auto end = m_storage.cend();
    41.  
    42. auto compare = [](const QPointF &p1, const QPointF &p2) {
    43. return p1.x() < p2.x(); };
    44.  
    45. // Find iterator of minimal point.
    46. const QPointF p1(rect.x(), 0);
    47. auto lower = std::lower_bound(begin, end, p1, compare);
    48. // Find iterator of maximum point.
    49. const QPointF p2(rect.x() + rect.width(), 0);
    50. auto upper = std::upper_bound(begin, end, p2, compare);
    51.  
    52. // TODO: Should we to do this?
    53. if (lower == end)
    54. lower = begin;
    55. if (upper == end)
    56. upper = end - 1;
    57.  
    58. // Calculate indexed of points in "external" storage.
    59. // (in principle, it can be omitted, as we have iterators).
    60. const auto lowerIndex = std::distance(begin, lower);
    61. const auto upperIndex = std::distance(begin, upper);
    62. auto pointsCount = upperIndex - lowerIndex;
    63.  
    64. // Get required "weeding" factor.
    65. const size_t factor = getFactor(pointsCount);
    66. qDebug() << "New factor" << factor << "for" << pointsCount << "points"
    67. << "from" << lowerIndex << "," << p1.x()
    68. << "to" << upperIndex << "," << p2.x();
    69.  
    70. // Desired number of points, which will be displayed.
    71. pointsCount = qRound(pointsCount / double(factor)) + 1;
    72. qDebug() << "Reduced up to" << pointsCount << "points";
    73.  
    74. // Refill the cache again with every "factor" step.
    75. m_cache.clear();
    76. // Always add the minimum point of range.
    77. m_cache += m_storage.at(lowerIndex);
    78. for (size_t i = 1; i < pointsCount - 1; ++i) {
    79. // Add the middle points.
    80. const size_t n =(i * factor) + lowerIndex;
    81. m_cache += m_storage.at(n);
    82. }
    83. // Always add the maximum point of range.
    84. m_cache += m_storage.at(upperIndex);
    85. }
    86.  
    87. SourceSamples::SourceSamples()
    88. {
    89. // Fill the "external" storage with all available points
    90. // (we just emulate the external storage as file, or DB,
    91. // via the array of data, for simplification of a code.
    92. size_t pointsCount = 10000000;
    93. m_storage.resize(pointsCount);
    94. double x = 0.0;
    95. for (size_t i = 0; i < pointsCount; ++i) {
    96. double y = std::sin(x);
    97. m_storage[i] = { x, y };
    98. x += 0.01;
    99. }
    100.  
    101. // Fill the cache for the initial details level. We starts
    102. // it for "low" resolution (we want to view whole range).
    103. const size_t factor = getFactor(m_storage.size());
    104. pointsCount = qRound(m_storage.size() / double(factor)) + 1;
    105. qDebug() << Q_FUNC_INFO << "Initialize with" << pointsCount << "points";
    106.  
    107. // Fill the cache with every "factor" step.
    108. // Always add the minimum point of range.
    109. m_cache += m_storage.first();
    110. for (size_t i = 1; i < pointsCount - 1; ++i) {
    111. const size_t n = i * factor;
    112. // Add the middle points.
    113. m_cache += m_storage.at(n);
    114. }
    115. // Always add the maximum point of range.
    116. m_cache += m_storage.last();
    117. }
    118.  
    119. size_t SourceSamples::size() const
    120. {
    121. return m_cache.size();
    122. }
    123.  
    124. QPointF SourceSamples::sample(size_t i) const
    125. {
    126. return m_cache.at(i);
    127. }
    128.  
    129. void SourceSamples::setRectOfInterest(const QRectF &rect)
    130. {
    131. updateCache(rect);
    132.  
    133. // TODO: Should we to do it?
    134. d_boundingRect = rect;
    135. }
    136.  
    137. QRectF SourceSamples::boundingRect() const
    138. {
    139. if (d_boundingRect.width() < 0.0)
    140. d_boundingRect = qwtBoundingRect(*this);
    141. return d_boundingRect;
    142. }
    143.  
    144. // RandomPlot
    145.  
    146. RandomPlot::RandomPlot(QWidget *parent)
    147. : QwtPlot(parent)
    148. {
    149. setAutoReplot(false);
    150.  
    151. auto grid = new QwtPlotGrid;
    152. grid->enableXMin(true);
    153. grid->setMajorPen(QPen(Qt::black, 0, Qt::DotLine));
    154. grid->setMinorPen(QPen(Qt::gray, 0, Qt::DotLine));
    155. grid->attach(this);
    156.  
    157. auto zoomer = new QwtPlotZoomer(canvas());
    158. // We want to have a zoom with the X-axis only.
    159. zoomer->setAxes(QwtPlot::xBottom, QwtPlot::yRight);
    160.  
    161. auto curve = new QwtPlotCurve;
    162. curve->setItemInterest(QwtPlotCurve::ScaleInterest, true);
    163.  
    164. auto series = new SourceSamples;
    165. curve->setData(series);
    166. curve->attach(this);
    167. }
    To copy to clipboard, switch view to plain text mode 

    seems, it works, but I'm not sure that it is right idea...

    @Uwe,

    Could you please comment of this idea? Maybe it can be optimized or changed at all?

    Also I have additional questions:

    1. Should I update the d_boundingRect with "interest rect" in this? Or, I need do not touch d_boundingRect at all?

    Qt Code:
    1. void SourceSamples::setRectOfInterest(const QRectF &rect)
    2. {
    3. updateCache(rect);
    4.  
    5. // TODO: Should we to do it?
    6. d_boundingRect = rect;
    7. }
    To copy to clipboard, switch view to plain text mode 

    2. Why, when I do "comments out" of this code:

    Qt Code:
    1. ...
    2. //curve->setItemInterest(QwtPlotCurve::ScaleInterest, true);
    3. ...
    To copy to clipboard, switch view to plain text mode 

    then the method SourceSamples::setRectOfInterest() were called? The QwtPlotCurve::ScaleInterest has not effect?

  4. #4
    Join Date
    Jan 2009
    Location
    Russia
    Posts
    309
    Thanks
    2
    Thanked 43 Times in 42 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Best way to handle very-very huge amount of data

    then the method SourceSamples::setRectOfInterest() were called? The QwtPlotCurve::ScaleInterest has not effect?
    I'm sorry, I was inattentive. Seems, this ScaleInterest is enabled by default.

Similar Threads

  1. Replies: 3
    Last Post: 30th March 2016, 19:51
  2. Replies: 34
    Last Post: 22nd March 2016, 17:34
  3. How to handle amount of text blocks?
    By toyman in forum Qt Programming
    Replies: 8
    Last Post: 4th February 2014, 22:24
  4. How to handle huge data with QSqlQueryModel?
    By alizadeh91 in forum Qt Programming
    Replies: 1
    Last Post: 7th December 2013, 15:31
  5. Best way to handle huge QImage setColorTable change
    By bruceariggs in forum Qt Programming
    Replies: 2
    Last Post: 24th September 2011, 22:47

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.