Defusing a C compiler bomb

I was shown a wonderfully devious C program today.

main[-1u]={1};

This will produce a huge object file and take some time to compile. It’s just the kind of cleverness that appeals to me.

But it’s dependent on the object file format. I’ve been working with the GOFF format on mainframes of late and GOFF actually has a mechanism to handle this. I decided to try this C bomb with XLC, IBM’s C compiler on the mainframe.

To start, the original program has to be changed slightly so that XLC will accept it.

int main[0x10000000]={1};

There are two reasons for this. First, XLC complains about lacking a type declaration for main, so we put in the int. Secondly, the size of a data object in GOFF is limited. The largest object I could get to work was 1GB, so that already makes it less susceptible to this problem.

Here’s what happened when I ran XLC on the above program on one of the z/OS machines I use for work.

% cat t.c
int main[0x10000000]={1};
% xlc -q64 -c t.c
% wc -c t.o
    2400    t.o

I’ll note that the compilation only took a few seconds.

In GOFF terms, that’s a reasonably tiny object file. If we take a look at the innards of the file, we can see that it’s got the correct length of 0x40000000. (Some of the unnessary stuff has been elided for clarity. The output of the tool to view object files on z/OS is a little cryptic.)

         ESD   OWNER/    ITEM      ITEM  
  ESDID  TYPE  PARENT   OFFSET   LEN/ADA 

 >000005  ED   000004         0         0
0              FILL BYTE:  0
               NAME(C_WSA64)

 >000006  PR   000005    000000  40000000
               NAME(main)

This says that the data for main is stored with symbol number 6 in the C_WSA64 class (where "class" is roughly equivalent to a section) and has a fill byte of 0. The fill byte is the important part. It says that any part of the data not included in the text section will be filled with 0s. And sure enough, if we check the text records we only see one record, with a 1 in the first 4 bytes.

 ESDID   OFFSET    LENGTH   ENCODING  LENGTH    ------------------------------ T E X T 

>000006 00000000  00000000    0000   00000004   00000001

Compiling the program on my FreeBSD system at home with clang results in an ELF file that is over 1 GB.

So score one for GOFF! Your bank is safe from this code golf compiler bomb. However, if you need to store a large static array, well, you’re out of luck.

October 7, 2016