At the Defcon 16 conference in Las Vegas, Nick Harbour showed off his new Windows executable packer, PE-Scrambler. It uses some interesting, and sometimes downright devious techniques to make analysis of the binary harder.
Rather than blindly manipulating the bits and bytes of the code to compress or encrypt it as many traditional packers, PE-Scrambler disassembles the code and manipulates it at a logical level to sabotage many of the methods used by disassemblers to analyse instructions and flow. The result is a binary that is hard to get any meaningful automatic analysis of program structure for.
The techniques used to do this include altering function calls to call a single dispatcher function, destroying the call tree; chunking the code into small pieces and relocating them throughout the binary; weaving bytecode together such that they have ambiguous disassembly; and some polymorphic substitution of certain pieces of code for obfuscated alternatives.
To achieve the destruction of the call tree, PE-Scrambler creates a dispatcher function and alters every function call to call this dispatcher instead. The dispatcher needs to know what functionality to jump to for each call without giving a disassembler too many hints, and it achieves this by checking the stack for the return value of that call. This value can identify which call instruction has called the dispatcher, and using this information a lookup table is created so the dispatcher knows what functional code to jump to. In this way, all calls can reach the appropriate functionality but it is impossible for a disassembler to know what function a given call relates to without knowledge of the lookup table mechanism.
For code relocation, a variety of methods are used. The simplest method takes a relocatable chunk, moves it, and the previous chunk connects to it with a simple unconditional JMP jump.
A slightly more advanced technique looks for parts of the code where it is known that the value of the zero flag will always be a certain value. A conditional jump is then added just after this, and the chunk of code following it relocated. We know that jump will always be taken, but most disassemblers performing static analysis will not notice this and will produce two possible code paths, making comprehension of program structure more complex.
The final technique for relocatable chunks is to insert predictable conditionals. This works much like the previous method, except this time the predictable conditional is arbitrarily manufactured and inserted by PE-Scrambler. These conditions do not have to be purely statically predictable and can take advantage of things that a human knows, but a disassembler would probably never assume. The example given in the presentation involves comparing ESP with 0×80000000, and while we as humans know that ESP will always reasonably be less than this, a disassembler does not.
My favourite method used by PE-Scrambler is the production of ambiguous or conflicting disassembly. This involves weaving instructions together such that instructions jump part way into their own bytecode, altering the meaning of those bytes to other instructions. If that wasn’t cool enough, PE-Scrambler also takes advantage of the fact that according to Nick most disassemblers evaluate the false condition of a branch first, and once bytes have been disassembled the disassembler won’t try to disassemble them again. By using overlapping instructions that form a conditional like this then, incorrect or conflicting disassembly can be produced. The presentation contained several examples of ways similar to this to produce flawed disassembly.
PE-Scrambler also performs some general polymorphic alterations to the code to further obfuscate it. This involves replacing certain instructions with misleading but equivalent instructions. This can break a disassemblers ability to automatically understand what is happening at a given point in the code, adding again tot he effort required to understand the program as a reverse engineer.
Despite these pretty cool tricks, PE-Scrambler is not perfect. As it uses disassembly to perform most of these tricks, there is no margin of error for its own disassembly. If errors occur, the resulting executable may be destroyed and fail to run. Currently there is a significant proportion of files for which this can be the case, but that is expected to drop as more fixes and tweaks are added. When it does work however, it works great. And even if you don’t use PE-Scrambler itself, some of the techniques it uses should provide inspiration for things like manual obfuscation of shellcode or the like.
As well as PE-Scrambler, at the end of the presentation Nick demonstrated another tool called FindEvil. This uses the same disassembly engine as PE-Scrambler and measures the size of disassembly compared to the size of the binary. Using various ratios it makes a decision as to whether that binary is packed. Nick claimed that so far, he has had 100% success using FindEvil, though he didn’t say how many files it has been tested with. It is, however, an interesting way to detect packers.