Packed structs imply two things: No padding inside the struct AND the struct's location may be unaligned in memory. So when this u32 access fails at runtime your compiler did not generate correct code for your platform. When copying an u32 to/from a packed struct, something like memcpy needs to be used when the processor requires aligned access.
Some ARM platforms can be configured to handle unaligned access, and there is a gcc command line flag that tells the compiler it can assume this capability (sorry forgot the name of the flag, something like -munaligned-access, google it). Maybe this points you in the right direction.