Alignment (part 2): Packing

Yesterday, I wrote a bit about how the C compiler determines the alignment of structures.  I left with an example of an MS-DOS structure that didn't follow the alignment rules.

So how do you deal with this?

The first question that comes to mind is "What's the language standard say about this behavior?".  Well, this morning, I chased down what appears to be an online version of the C standard.  In section 6.5.2.1, you find:

[#11] Each non-bit-field member  of  a  structure  or  union
       object   is  aligned  in  an  implementation-defined  manner
       appropriate to its type.

Hmm. The standard says that the structure alignment is implementation defined, that's not much of a help.

It turns out that Microsoft's C compiler defines a series of #pragma's that allow the developer to specify the structure alignment, as I implied yesterday, they're called #pragma pack.  Essentially the #pragma pack allows the caller to override the compilers default rules for structure alignment.  If you say #pragma pack(4) you're saying "if the natural alignment of this structure is greater than 4, treat it as 4 instead".  Similarly, #pragma pack(1) says "Treat all the members of this structure as having a natural alignment of 1 byte".  And #pragma pack(16) says "Treat all members as having a natural alignment of 16 - essentially it's a NOP on current architectures".  You can specify the packing for an entire source file with the -Zp compiler command line switch, but that's usually hideous overkill.  Instead, most people just use #pragma pack(n) around their structure definitions.

So lets look at the examples from yesterday and see what #pragma pack does to the structure.

First, here's struct A:

struct A{   int _FieldA1;   char _FieldA2;   short _FieldA3;   char _FieldA4;   long _FieldA5;   void *_FieldA6;};

Lets consider what happens with struct A when compiled with #pragma pack(1), #pragma pack(4) and #pragma pack(16):

Field Name	Field Size	Field Offset (default)	Field Offset (#pragma pack(1)	Field Offset (#pragma pack(4)	Field Offset (#pragma pack(16)
_FieldA1	4	0	0	0	0
_FieldA2	1	4	4	4	4
_FieldA3	2	6	5	6	8
_FieldA4	1	8	7	8	12
_FieldA5	4	12	8	12	16
_FieldA6	4 (8 on 64 bit)	16	12	16	20

So if you specify #pragma pack(1), you get the tightest possible packing for data.

Now why on earth might you want to specify #pragma pack?  It turns out that for the vast majority of applications it doesn't matter.  For example, most of the Win32 API set is defined without packing (there are some exceptions like some of the messages for common controls).  And more importantly, you shouldn't ever have to specify a structure packing before including any of the Windows header files - the windows header files are supposed to be built to work regardless of any external compiler switches or flags.  This is why you see all the "#include <pshpack2.h>" etc in the header files - this ensures that the structure packing rules are locked in regardless of the -Zp switch.

But there IS one situation where this becomes important.  As I mentioned yesterday, some operating systems (like MS-DOS) have their data structures built with #pragma pack(1).  And some networking protocols (such as the CIFS protocol) are defined with packed structures.  So if you define structures to interact with those protocols, then you need to use #pragma pack(1) to ensure that your structure definition lines up with the alignment of the networked protocol.

One other thing to keep in mind. The C language contract for allocators (malloc, free, etc) is:

The  pointer  returned  if  the allocation  succeeds  is  suitably aligned so that it may be assigned to a pointer to any type of object and then used to access  such  an object  or an array of such objects in the space allocated (until the  space  is  explicitly  freed  or reallocated). 

What this means is that a C/C++ allocator cannot return an address whose address isn't at least a multiple of 8 (the largest native allocation alignment). In reality, most allocators return memory aligned on a 16 or 32 byte boundary to take advantage of cache line effects.

Edit: Corrected rules for #pragma pack(16)

Comments

• Anonymous
  April 08, 2005
  In struct A you forgot float and double...
• Anonymous
  April 08, 2005
  Actually there are a bunch of types I didn't include - floats are 4 bytes in size (and are thus the same as int's) and doubles are 8 bytes in size (and thus are the same as longlong).
  
  The intention wasn't to show all the possible data types but instead to show what happens with different types. The most interesting one is what _FieldA3 and _FieldA4 do - because they both have hidden padding associated with them.
• Anonymous
  April 08, 2005
  Actually I was under impression than modern MS C/C++ compiler packs fields on the nearest boundary that matches type alignment or current packing, therefore in example table #pragma pack(16) should be the same as #pragma pack(4)...
• Anonymous
  April 08, 2005
  The offsets for pack (16) are
  0
  4
  6
  8
  12
  16
  
  The offsets for pack (4) are
  0
  4
  6
  8
  12
  16
  
  The packing value defines the max alignment, not the min.
  
  "When you use #pragma pack(n), where n is 1, 2, 4, 8, or 16, each structure member after the first is stored on the smaller member type or n-byte boundaries."
• Anonymous
  April 08, 2005
  MSC uses a variant on the old natural alignment. For natural alignment, the offset must be a multiple of the size of the element. However, in MSC, they just cap the size to the packing size.
• Anonymous
  April 08, 2005
  My bad - I'll fix the table (teach me to write something without actually verifying it)
• Anonymous
  April 10, 2005
  I worked on a project which had lots of #pragma pack statements. We then created a .tlb file and then used the #import statement for the .tlb file. It turns out that the default implementation of #import (Visual Studio 6 and .Net) ignores the packing from the .tlb file. We contacted MS and were given a fix for our compilers.
  However, MS needed to create the fix for VS.Net. They did this after some gentle nudging on our part which was very nice of them.
  We asked 'Why does #import work this way, when it's obviously incorrect?' The response was
  'We know it's incorrect, but a lot of people rely on it being incorrect. Here's the fix, enjoy!'
  
  http://support.microsoft.com/default.aspx?scid=kb;en-us;822550
  Thanks MS!
• Anonymous
  April 11, 2005
  I mentioned in the last post that the C standard is agnostic w.r.t. alignment.
  So why on earth did Microsoft...
• Anonymous
  April 11, 2005
  James Kilner pointed us to
  http://support.microsoft.com/default.aspx?scid=kb;en-us;822550
  Thank you.
  
  There's some rather odd information here. A different article
  http://support.microsoft.com/kb/316971/
  says that it applies to VC++6 SP5. It also says Microsoft recommends that you wait for the next service pack that contains this fix. That would be VC++6 SP6, would it not?
  
  But the article that Mr. Kilner points us to says that Microsoft will never officially release a fix.
  
  Both articles say that charges for support calls for these kinds of hotfixes can be cancelled if Microsoft feels like it. Microsoft Japan told me over the phone that they don't feel like it. They wouldn't even let me tell them the Knowledge Base article numbers for which I wanted hotfixes unless I first paid for a support call.
• Anonymous
  April 11, 2005
  In response to Norman Diamonds comments. We were told that they would not release the fix due to people relying on the behaviour. To get them to even make the fix for VS.Net we had to give them a business case for why they should put the effort in to fixing the bug. We even suggested to them a means by which they could fix the bug AND keep existing functionality (hence being able to release the fix in the next service pack). They ignored our suggestion and just reimplemented the fix that they had done for VS6.
  As for the cost issue, my company had an MSDN subscription. We had to give them our details at the start, but in the end they never charged us. It may be that you would get a refund once they realise that the fix is exactly what you need.
• Anonymous
  April 15, 2005
  Well, clearly the shorter the example, the quicker people pick up the problem.
  The problem was:
  &amp;nbsp;&amp;nbsp;&amp;nbsp;...
• Anonymous
  June 15, 2005
  dear sir,
  
  It was a very good attempt to give a complete knoledge on this tricky aspect of "Pragma Pack"
  
  I work on the Unix (Sun Solaris 5.8). When I run your example provided in this text, the restults are totally different from what you have mentioned. It may be due to differences in the architectures (I am not sure).
  
  A mention that the results vary on different architectures might make your document complete.
  
  I also feel that a basic need in the change of concept also is requrired. Hope I am not wrong, if I say that the offset of any member of the structure never starts with a number that is not divisible by 2 (no matter what the pragma pack value is) !!!
  
  regards
  pavan kumar
  from India.
• Anonymous
  April 11, 2006
  看下面这个代码有什么问题：
  
  
  
  #define PROCESS_NAME _T(“C:\WindowsNotePad.ext”)
  
  
  
     if (!CreateProcessW(N
  
  ……
• Anonymous
  November 14, 2006
  Well, clearly the shorter the example, the quicker people pick up the problem. The problem was: if (!CreateProcessW(NULL,