[ SOLVED (kinda) ]copy structure to QByteArray

[pdoria]

Hi,

I'm trying to copy this structure to a QByteArray:

typedef struct
{
	quint32 status_change_id;
	bool result_code;		// true if sucessful, false otherwise
	quint8 reserved[3]; /* Set to 0 */
} driver_id_receipt_data_type;

typedef struct
{
	quint32 status_change_id;
	bool result_code;		// true if sucessful, false otherwise
	quint8 reserved[3]; /* Set to 0 */
} driver_id_receipt_data_type;

To copy to clipboard, switch view to plain text mode 

The structure already being declared and filled with:

driver_id_receipt_data_type receipt;
receipt.status_change_id	= payload.mid(2,4).toUInt();
receipt.result_code 		= true;
receipt.reserved[0]		= 0;
receipt.reserved[1]		= 0;
receipt.reserved[2]		= 0;

driver_id_receipt_data_type receipt;
receipt.status_change_id	= payload.mid(2,4).toUInt();
receipt.result_code 		= true;
receipt.reserved[0]		= 0;
receipt.reserved[1]		= 0;
receipt.reserved[2]		= 0;

To copy to clipboard, switch view to plain text mode 

Every approach I use (cast to (char *), ::fromRawData(), etc) I stumble upon the impossibility of reaching the desired goal...

Is there a way of doing this?

Any help appreciated
Best regards,
Pedro Doria Meunier

[caduel]

show us what you tried and elaborate on how you want to use that QByteArray - and in what way did those failures manifest themselves?

[pdoria]

Hi Caduel,

Thx for your time.

The goal here is simply put those bytes of the structure into a QByteArray which will then be passed to another function to process it.

Since I have a gadzillion of other struct types I can't possibly write a function to handle them one-by-one. I need a standard way of passing data to the receiving function. Hence the QByteArray ...

As requested here's an early example of trying to copy the data into the QByteArray:

QByteArray packet;
memcpy(packet.data(), &receipt, sizeof(receipt) );

QByteArray packet;
memcpy(packet.data(), &receipt, sizeof(receipt) );

To copy to clipboard, switch view to plain text mode 

Even this old school, deprecated method, fails miserably ...

BR,
Pedro Doria Meunier

[pdoria]

Anyone?

Pls, I'm really stuck with this ...

[jord]

Firstly, I am not familiar with QByteArray. However, clearly you need to ensure that the array within QByteArray is large enough to fit your structure. A cursory glance of the QByteArray documentation would remind you of this and how to do it. Once you have actually allocated your destination buffer, I see no reason why memcpy would not work.

Having said that, I suspect this is a solution to a problem that shouldn't exist in the first place. However, you haven't provided enough detail to be sure.

Edit: QDataStream would probably do the job too.

[pdoria]

First off many thanks to all that have replied!

Here's my (convoluted) solution:

driver_id_receipt_data_type *receipt = new driver_id_receipt_data_type;
	receipt->status_change_id	= payload.mid(2,4).toUInt();
	receipt->result_code 		= true;
	receipt->reserved[0]		= 0;
	receipt->reserved[1]		= 0;
	receipt->reserved[2]		= 0;
 
	char *p = (char*)receipt;  // cast it to char* to make a QByteArray
	QByteArray packet(p, sizeof(driver_id_receipt_data_type));

driver_id_receipt_data_type *receipt = new driver_id_receipt_data_type;
	receipt->status_change_id	= payload.mid(2,4).toUInt();
	receipt->result_code 		= true;
	receipt->reserved[0]		= 0;
	receipt->reserved[1]		= 0;
	receipt->reserved[2]		= 0;

	char *p = (char*)receipt;  // cast it to char* to make a QByteArray
	QByteArray packet(p, sizeof(driver_id_receipt_data_type));

To copy to clipboard, switch view to plain text mode 

QByteArray packet now holds the receipt data...

Knowing this a very convoluted way of doing things, I'd love to see a simpler, more direct, approach to reach the above goal...

BR,
Pedro Doria Meunier

[caduel]

two lines of code is not that convoluted, is it?

[Ginsengelf]

Hi, you could use QDataStream to write every single member of your struct to a QByteArray. That would get rid of the cast, but you would have to adjust the code each time you change your struct. It will also require more lines of code and probably be slower than your current version.

Ginsengelf

[jord]

Hi pdoria,

I would like to discourage you from serializing an object by doing a byte-by-byte copy (which is what you are doing here), explain why, and offer some advice. What you are doing may be fine for your application (only you know), but it is very easy to shoot yourself in the foot with this approach. Worse still, depending on what is in your objects, you may cause your application to crash in a completely unrelated piece of code, making it a nightmare to track down the problem.


You need to be wary of the following things with this approach:

1. The object's copy constructor is not called.

This means the copy constructor is also not called for any objects that are contained within your structs. This is disastrous for objects that allocate memory on the heap. Problems include:

• The original object and its copy interfering with one another and trashing their internal data structures.
• One of the objects getting destroyed, leaving the other with invalid pointers that it continues to use.
• Double frees of memory once both objects are destroyed, resulting in the heap getting trashed.

2. How the data is arranged in the struct and the padding between data items is compiler dependent.

If you are serializing the object with the intent of saving it to disk or sharing it with another party, be aware that different compilers and compiler settings may arrange the data in the struct differently. Hence this can fail unexpectedly.


3. Different computer architectures have different constraints on how data is represented.

Different computer architectures can have different endianness and different memory alignment requirements. As with point 2 above, this can cause issues when sharing the serialized object with another party.


A better approach is to write functions that explicitly serialize and deserialize the objects. It is more work initially, but it is robust against future changes and can avoid subtle and hard to track down errors in the future. The book 'C++ GUI Programming with Qt 4' has a section that touches on this. You can find a legit pdf of the first edition here: http://www.qtrac.eu/marksummerfield.html. The section is 'Reading and Writing Binary Data', on p.274 of this ebook.

The general approach is to add functions like the following to your structs/classes (note I haven't tested this):

QByteArray serialize()
{
    QByteArray byteArray;
 
    QDataStream stream(&byteArray, QIODevice::WriteOnly);
    stream.setVersion(QDataStream::Qt_4_5);
 
    stream << status_change_id
               << result_code
               << reserved[0]
               << reserved[1]
               << reserved[2];
 
    return byteArray;
}
 
void deserialize(const QByteArray& byteArray)
{
    QDataStream stream(byteArray);
    stream.setVersion(QDataStream::Qt_4_5);
 
    stream >> status_change_id
               >> result_code
               >> reserved[0]
               >> reserved[1]
               >> reserved[2];
}

QByteArray serialize()
{
    QByteArray byteArray;
    
    QDataStream stream(&byteArray, QIODevice::WriteOnly);
    stream.setVersion(QDataStream::Qt_4_5);
    
    stream << status_change_id
               << result_code
               << reserved[0]
               << reserved[1]
               << reserved[2];

    return byteArray;
}

void deserialize(const QByteArray& byteArray)
{
    QDataStream stream(byteArray);
    stream.setVersion(QDataStream::Qt_4_5);

    stream >> status_change_id
               >> result_code
               >> reserved[0]
               >> reserved[1]
               >> reserved[2];
}

To copy to clipboard, switch view to plain text mode 

Good luck,

Jord

[pdoria]

Jord,

I can only say this is one HECK of a good advice! THANK YOU!

It reminded me that I have to step out of pure C thinking ...

Now about the versioning ...
I believe I can't do it because that would insert the "magic numbers" in the stream's header, right?

I'm dealing with a protocol that doesn't have room for "extra info" ...

Best regards,
Pedro Doria Meunier

[pdoria]

Well ... that almost worked ... except for QByteArray ...

Given the following struct:

//! \brief SERIAL PACKET FORMAT (Physical Layer - RS232)
 
typedef struct {
	quint8 DLE;			// ASCII DLE character (16 decimal)
	quint8 packet_id;		// packet ID
					// types:
					//	6 - ACK
					//	10 - Command
					//	14 - Date/Time Data
					//	21 - NAK
					//	38 - Unit ID/ESN
					//	51 - PVT (Position, Velocity, Time) Data
					//	135 - Legacy Stop message
					//	136 - Legacy text message
					//	161 - Fleet Management packet
	quint8 size_app_payload;	// number of bytes of packet data (bytes 3 to n-4)
	QByteArray app_payload;		// 0 to 255 bytes
	quint8 checksum;		// 2's complement of the sum of all bytes from byte 1 to byte n-4 (end of the payload)
	quint8 DLE_end;			// same as DLE
	quint8 ETX;			// End of text - ASCII ETX character (3 decimal)
 
	//! \note helper functions
	QByteArray serialize()
	{
		QByteArray byteArray;
 
		QDataStream stream(&byteArray, QIODevice::WriteOnly);
 
		stream  << DLE
			<< packet_id
			<< size_app_payload
			<< app_payload
			<< checksum
			<< DLE_end
			<< ETX;
		return byteArray;
	}
 
	void deserialize(const QByteArray& byteArray)
	{
	QDataStream stream(byteArray);
 
		stream  >> DLE
			>> packet_id
			>> size_app_payload
			>> app_payload
			>> checksum
			>> DLE_end
			>> ETX;
	}
} serial_packet_format;

//! \brief SERIAL PACKET FORMAT (Physical Layer - RS232)

typedef struct {
	quint8 DLE;			// ASCII DLE character (16 decimal)
	quint8 packet_id;		// packet ID
					// types:
					//	6 - ACK
					//	10 - Command
					//	14 - Date/Time Data
					//	21 - NAK
					//	38 - Unit ID/ESN
					//	51 - PVT (Position, Velocity, Time) Data
					//	135 - Legacy Stop message
					//	136 - Legacy text message
					//	161 - Fleet Management packet
	quint8 size_app_payload;	// number of bytes of packet data (bytes 3 to n-4)
	QByteArray app_payload;		// 0 to 255 bytes
	quint8 checksum;		// 2's complement of the sum of all bytes from byte 1 to byte n-4 (end of the payload)
	quint8 DLE_end;			// same as DLE
	quint8 ETX;			// End of text - ASCII ETX character (3 decimal)

	//! \note helper functions
	QByteArray serialize()
	{
		QByteArray byteArray;
		
		QDataStream stream(&byteArray, QIODevice::WriteOnly);
		
		stream  << DLE
			<< packet_id
			<< size_app_payload
			<< app_payload
			<< checksum
			<< DLE_end
			<< ETX;
		return byteArray;
	}

	void deserialize(const QByteArray& byteArray)
	{
	QDataStream stream(byteArray);
	
		stream  >> DLE
			>> packet_id
			>> size_app_payload
			>> app_payload
			>> checksum
			>> DLE_end
			>> ETX;
	}
} serial_packet_format;

To copy to clipboard, switch view to plain text mode 

and the following code:

serial_packet_format *sPacket = new serial_packet_format;	// send fleet management packet wrapped in a serial packet format...
	sPacket->DLE=16;
	sPacket->packet_id= FLEET_MANAGEMENT;			// fleet management packet
	// fill rest
	sPacket->size_app_payload=driver_id_receipt.size();
	sPacket->app_payload = QByteArray::QByteArray ( driver_id_receipt );
	// calculate 2's complement checksum
	sPacket->checksum = CalculateChecksum(driver_id_receipt.data(), driver_id_receipt.size() );
	sPacket->DLE_end=16;
	sPacket->ETX=3;
 
	QByteArray serial_packet( sPacket->serialize() );

serial_packet_format *sPacket = new serial_packet_format;	// send fleet management packet wrapped in a serial packet format...
	sPacket->DLE=16;
	sPacket->packet_id= FLEET_MANAGEMENT;			// fleet management packet
	// fill rest
	sPacket->size_app_payload=driver_id_receipt.size();
	sPacket->app_payload = QByteArray::QByteArray ( driver_id_receipt );
	// calculate 2's complement checksum
	sPacket->checksum = CalculateChecksum(driver_id_receipt.data(), driver_id_receipt.size() );
	sPacket->DLE_end=16;
	sPacket->ETX=3;

	QByteArray serial_packet( sPacket->serialize() );

To copy to clipboard, switch view to plain text mode 

what happens is that upon

stream  << size_app_payload

stream  << size_app_payload

To copy to clipboard, switch view to plain text mode 

a DWORD containing the QByteArray size is prepended to the original QByteArray...

In other words:

Upon execution of the above code here's what's inside sPacket:

SERIAL PACKET CONTENTS (size=20):
10
FFFFFFA1
0A

00
00
00
0A --> the 4 bytes in bold shouldn't be here!

08
12
00
00
00
01
01
00
00
00
FFFFFFE4
10
03

Please advise as this is the last thing to overcome my long standing problem...

BR,
Pedro Doria Meunier

[faldzip]

QDataStream is a class for serializing objects. But it can append its own info about serializing object. So don't expect it would be byte to byte what you want. It is made in such way that when you write something to QDataStream (with operator <<) and then you can read it from it on the other end (with operator >>) then you get the same. But who cares what is in the middle :] A little example : let's say you have a QString str("Hello"); then you write it to QDataStream object. In this stream the QDataStream can do whateveer it wants it can even append " World!" to your string :] but when you read string fron QDataStream it returns what you wrote: QString("Hello"). So if you want to have a QByteArray containing only your own data then put every byte of your data separately. For example, if you have an 32-bit integer you can do:

int num = 158;
QByteArray ba;
for (int i = 0; i < 4; ++i) {
    ba += quint8((num >> i*8) & 0x000000ff);
}

int num = 158;
QByteArray ba;
for (int i = 0; i < 4; ++i) {
    ba += quint8((num >> i*8) & 0x000000ff);
}

To copy to clipboard, switch view to plain text mode 

and now you have every byte of your int written to byte array. Then you have to construct one int from 4 bytes from byte array on the other side.

[pdoria]

Regarding the last post...
(involving qdatastream << qbytearray )

From Qt Assistant, "Format of the QDataStream Operators"

QByteArray
If the byte array is null: 0xFFFFFFFF (quint32)
Otherwise: the array size (quint32) followed by the array bytes, i.e. size bytes

So, from what I see, QDataStream doesn't do whatever "comes to its mind".
It's simply doing what it's told...

What I need to do is simply eliminate the use of QByteArray from the structure or find a fast, elegant, way to chop those array size bytes ...

BR,
Pedro Doria Meunier

[jord]

Ok. By the looks of things you're sending this data to hardware for processing. I didn't realise that.

In this case you could use a struct as you were planning to. It's quite common practice. You will probably have to tell the compiler not to pad out the struct, but I see in another thread you are already aware of that. If you are only targeting a single platform/compiler and don't have endianness worries, then you could do it this way. All you need to do is reinterpret_cast the struct into an unsigned char* (or whatever type your send function takes). eg:

#pragma pack(push, 1)
struct MyStruct
{
    unsigned char c;
    unsigned int i;
};
#pragma pack(pop)
 
void send(unsigned char* buf, size_t length)
{
    // send buf
}
 
int main()
{
    MyStruct myObj;
    myObj.c = 'A';
    myObj.i = 1;
 
    send(reinterpret_cast<unsigned char*>(&myObj), sizeof(myObj));
 
    return 0;
}

#pragma pack(push, 1)
struct MyStruct
{
    unsigned char c;
    unsigned int i;
};
#pragma pack(pop)

void send(unsigned char* buf, size_t length)
{
    // send buf
}

int main()
{
    MyStruct myObj;
    myObj.c = 'A';
    myObj.i = 1;

    send(reinterpret_cast<unsigned char*>(&myObj), sizeof(myObj));

    return 0;
}

To copy to clipboard, switch view to plain text mode 

My personal preference is to write portable code. So I would only do it this way if performance absolutely demanded it. Instead I would probably detect the endianness of the host at runtime, construct an unsigned char array of the correct size, reinterpret_cast the array at the appropriate offsets to copy in the appropriate values - applying endian conversions if necessary. But to each their own.

Edit: I notice Qt provides qFromBigEndian to handle all the endian conversion stuff.

[wysota]

What's the reason for using QByteArray here? Can't you just send a pointer to the structure or something (you might also want to make the structure packed to make up for compiler data alignment mechanisms)?