code hacking, zen coding

CSAW 2011 OpenGL

In this challenge, you get a Windows binary, when you start it it only display a white windows

The first surprise is that there is absolutely no packing or obfuscation of the binary, it decompiles cleanly in IDA.

What we see is that it’s a classic OpenGL application with OpenGL init and a frame loop. I used to be a demoscene coder that wrote his own 3D engine so this looked very familiar.

We will first use glIntercept to get a log file with all the OpenGL calls and their parameters:

glIntercept comes with a replacement DLL that will be loaded by the binary when initializing OpenGL.

In the log file produced we can see a few interestings things:


The background color is black.


But our draw color is black also, black on black = … black


At the end of the frame drawing, we draw a huge rectangle all over the screen in white.

So we know that we have to do, for a start, 2 things:

  • Change the default draw color to white so we can see something
  • Remove the huge rectangle from the view so we can see something also

Change the default color to white:

Address   Hex dump          Command                                  Comments
0040206B  |.  B8 0000803F   MOV EAX,3F800000
00402070  |.  894424 0C     MOV DWORD PTR SS:[LOCAL.3],EAX           ; /Arg4 => 3F800000
00402074  |.  B8 0000803F   MOV EAX,3F800000                         ; |
00402079  |.  894424 08     MOV DWORD PTR SS:[LOCAL.4],EAX           ; |Arg3 => 3F800000
0040207D  |.  B8 0000803F   MOV EAX,3F800000                         ; |
00402082  |.  894424 04     MOV DWORD PTR SS:[LOCAL.5],EAX           ; |Arg2 => 3F800000
00402086  |.  B8 0000803F   MOV EAX,3F800000                         ; |
0040208B  |.  890424        MOV DWORD PTR SS:[LOCAL.6],EAX           ; |Arg1 => 3F800000
0040208E  |.  E8 2D120000   CALL <JMP.&OPENGL32.glColor4f>           ; \OPENGL32.glColor4f

Jump over the huge rectangle:

Address   Hex dump          Command                                  Comments
0040211D  \. /E9 41010000   JMP 00402263

At this point we see now a black background, “KEY” written in white lines and many letters written in vectors randomly moving on the screen. We are on the right way.

Now we need to remove the randomness in the key letters placement and put them next to each others so we can read the flag.

Since the positionning sub is the same for all the letter, I choose to use the space previously allocated to the rand() calculation to write my new opcodes.

Also to avoid having to patch all the other sub functions drawing letter that do a glPopMatrix, we will just put a glPushMatrix at the end of the positioning function, this way the result is neutral.

New opcodes added to positioning sub:

Address   Hex dump          Command                                  Comments
00401F96  |.  B8 00000000   MOV EAX,0
00401F9B  |.  894424 08     MOV DWORD PTR SS:[LOCAL.8],EAX           ; /Arg3 => 0, don't touch Z
00401F9F  |.  B8 0000403F   MOV EAX,3F400000                         ; |
00401FA4  |.  894424 04     MOV DWORD PTR SS:[LOCAL.9],EAX           ; |Arg2 => 3F400000, move a bit to the right
00401FA8  |.  B8 0000403F   MOV EAX,3F400000                         ; |
00401FAD  |.  890424        MOV DWORD PTR SS:[LOCAL.10],EAX          ; |Arg1 => 3F400000, move a bit to the bottom
00401FB0  |.  E8 A3120000   CALL <JMP.&OPENGL32.glTranslatef>        ; \OPENGL32.glTranslatef
00401FB5  |.  83EC 0C       SUB ESP,0C
00401FB8  |.  E8 93120000   CALL <JMP.&OPENGL32.glPushMatrix>        ; Jump to OPENGL32.glPushMatrix, because the drawing sub do a glPopMatrix
00401FBD  |.  90            NOP
00401FBE  |.  90            NOP
00401FBF  |.  C9            LEAVE
00401FC0  \.  C3            RETN

And we are done. The resulting application will looks like this:

Key is “sc4#3sr1u30*0”

This challenge was really fun to do and original.