CILFI: Automatic Function Identification in .NET Binaries
There is a specific stage of grief every .NET reverse-engineer goes through when writing the next .NET deobfuscator or config extractor. It is the realization you have to write yet another ugly pattern-matching algorithm to find the exact same string decryptor, VM opcode handler, or C2 connection initializer functions to extract obfuscator configurations or IoCs.
I got sick of it.
I wanted a CIL pattern-matching tool that is very precise, robust against adversary trickery, and yet so simple that creating a signature is a matter of seconds. But none of the solutions I found on the internet were satisfactory to me.
So, naturally, instead of continuing to suffer, I built a custom language and engine to solve it once and for all.
Meet CILFI, an intuitive function identification tool targeting the Common Intermediate Language (CIL):
CILFI: An intuitive pattern matching tool for quickly identifying methods in a .NET binary.
The problem
When writing deobfuscators or config extractors for .NET binaries, one of the first steps is to look for important functions and extracting data from them. This may include for example string decryption routines you are targeting, anti-debug protection initializers you want to remove, opcode handlers of a virtual machine you want to infer the byte instruction encoding for, or start-up routines that set up connections to a C2.
If you are a bit experienced with reading obfuscated code, you can usually eyeball pretty quickly in a decompiler which functions are responsible for this plumbing:
- String decrypters always return strings and usually contain some calls or opcodes related to cryptography (e.g., XORs, Base64 conversion calls, etc.).
Example string decryptor function.
- Virtual machine opcode handlers often operate on virtual registers or a virtual stack, and surround them with instructions that are characteristic of basic operations (e.g., addition, subtraction, multiplication, calls).
Example KoiVM virtual machine opcode handler.
You can note down their metadata tokens, and give them to your deobfuscator or extractor of choice, but when you have many samples that are all protected by the same obfuscator, doing this manually becomes annoying really quickly. Preferably, this type of grunt work should be automated as much as possible. Typically, this means you need to write some code that looks for known patterns in the CIL to distinguish it from the obfuscated user code.
And if you ask me, this is a huge pain…
Writing pattern-matching code sucks
I am pretty sure I share this annoyance with many others in the (.NET) reversing world.
I hate writing pattern-matching code.
It is not necessary difficult, it is just really tedious.
It takes a lot of ugly (often very flaky) code with many if statements, loops and accounting for the many variations code can take. It can be very time-consuming until you get it to a place where you are happy with it. Frankly, it is just not the interesting part of writing a deobfuscator. If anything, it is more of a boring, necessary annoyance that you just need to get through to get the ball rolling.
I need all this to find a single method?
None of the solutions that I have found on the web could satisfy my gripes. They always are either too verbose, hard to understand / maintain, or are too flaky to be used in a real adversary setting where binaries are actively trying to screw you over as an analyst.
Time to fix that.
Let’s write a usable pattern-matching engine
My goals for this are pretty straightforward:
- It needs to be robust: No over-reliance on string carving and matching, and no non-determinism that you get with AI. I want a fast, deterministic tool that Just Works™ and will always Just Work™, not just when it feels like it.
- It needs to be precise: If, for whatever reason, I want to find methods that make calls to a generic class with two type parameters, take in an array, and return a value typed object, I should be able to express this oddly-specific query with no problem.
- It needs to be generalizable: I don’t want to match code in one binary only. To account for possible variations code may have across samples, pattern-matching syntax similar to regular expressions and wildcards are a must.
- It needs to be dead simple: This is the most important requirement. As I said before, I hate writing code for pattern-matching. The less time I have to spend on it, the better. The process of creating a signature should thus be as frictionless as possible.
As far as I know, none of the normal programming languages can check all these boxes at the same time.
Perhaps the naive route would be to dump the entire disassembly to a file and start string grepping and/or use regular expressions, but this would hamper robustness. Furthermore, big regular expressions are notoriously hard to understand once written down and are very difficult to debug.
We can try to be clever with modern programming language constructs like C#’s or Python’s operator overloading and pattern-matching, but this only gets you so far and would still introduce a lot of friction to translate CIL code as seen in a decompiler to a different programming language.
We don’t need to be married to existing solutions or programming languages, however. So let’s just make a new one!
Designing the language
How would we go about designing such a language?
Disassembly as input
The source of all truth in a .NET binary is the CIL code that the binary contains. As such, any pattern-matching will have to be done on this level.
To make things as frictionless as possible all the way from raw code read in the disassembler to ready-to-deploy pattern-matching signature, our starting point should therefore be the disassembler’s output. And when I say that, I literally mean the raw textual representation of the CIL code that makes up the method that is produced by the tool. No fancy object models that libraries like Cecil, dnlib, or AsmResolver provide – the typical reverser does not spend time programming these libraries directly when analyzing binaries anyway (and they shouldn’t!). Instead, they are browsing decompiler code in a friendly UI and reading text from it. Therefore, ideally, a reverser should be able to copy/paste the textual code produced by the disassembler and it should be a valid signature already.
Starting a new signature should be as easy as Ctrl+C and Ctrl+V
With that in mind, this means our pattern-matching tool needs to parse the CIL grammar and make sense of it. To my knowledge (at least by the time writing this post), there exist no standalone, open-source CIL grammars that is also easily extensible/hackable. So I grabbed ANTLR, a widely used parser generator that I have used before in the past, and painstakingly redefined the CIL grammar with all its intricate details and edge-cases.
Part of a CIL Grammar, implemented in ANTLR
By the time writing of this, this grammar does not implement all features of CIL. Only a subset that would allow for basic method and code matching is included.
Adding pattern-matching syntax
As time-consuming as defining our own grammar is, it does give us a lot of benefits. In particular, when we fully own the grammar, it is incredibly easy to add extra custom syntax to any non-terminal we want.
For example, for every type of syntax element we can add an alternative token ?? to indicate a wildcard.
Adding wildcards to operands
This allows for a really intuitive workflow, where you start by copying some raw CIL code, and replace all the specific identifiers and references with ?? tokens:
Replacing specific identifers with wildcard tokens.
Sometimes we actually know concrete values for all possible operands. We can add these OR-like patterns easily by just introducing a few extra grammar rules:
Adding wildcards to operands
Now we can match on multiple, concrete operands:
Replacing operands with options.
Finally, if we are looking for specific string operands that have a certain shape to them (e.g., base64, hexadecimal, a URL, IP address), we often use regular expressions. I know I said earlier that regular expressions are notoriously hard to understand. However, when used in a very precise and local setting (such as matching individual operands), I think their compactness and flexibility vastly outweigh their flaws:
Adding regular expression support.
This extra grammar rule allows us to do exactly that:
A string matching hexadecimal syntax.
Automatic macro expansion
CIL defines a lot of opcodes that are shorthands for other opcodes. For example, the ldc.i4.0 macro pushes the integer 0 on the stack and takes only 1 byte in the code stream, while ldc.i4 0 is semantically equivalent but takes 5 bytes instead.
Different CIL bodies, same behavior.
A standard C# compiler always optimizes for CIL code size and will always optimize ldc.i4 0 to ldc.i4.0. However, our threat model assumes adversary binaries where obfuscators can decide whatever they want to thwart automatic tooling. It is therefore not guaranteed we will always encounter the most optimized version of the code.
While we added pattern-matching syntax that allows for providing alternatives, you don’t want to end up describing these possible macro expansions or shortenings every time you encounter a ldc.i4.* instruction (i.e., (ldc.i4.0 | ldc.i4.1 | ...)). Therefore, I decided that the pattern-matching engine should automatically normalize them to their fully expanded form before doing the comparison.
Code blocks and conditionals
Finally, it is not often the case that you are interested in the entire method body, but only want to look for the presence of specific instruction sub-sequences (e.g., a specific call and its arguments).
Hence, I added a new .block directive to do exactly that:
Code blocks.
This also allows for boolean circuits to be defined on top of these defined blocks, which opens up for describing code that has multiple possible implementations (similar to a condition block in a YARA rule):
Conditionals over blocks.
Putting it all together
All that is left is attaching a metadata backend to the new language, implementing the pattern-matching code, and building some infrastructure around it. I will spare you the details on that, it is pretty boring stuff.
I call the final product CILFI (CIL Function Identification). Here is a walkthrough of a typical workflow using the command-line utility from start to finish:
To make creating signatures easier, I also hacked together a Visual Studio Code extension that adds some basic highlighting and autocompletion to the editor:
Visual Studio Code extension.
Using that extension, I wrote a bunch of signatures (koivm-opcodes.cilfi) that can find all the 83 different opcode handlers defined by the KoiVM obfuscator:
Using CILFI to extract opcode handlers from KoiVM samples.
You can also instruct it to print exactly the instructions that were matched according to the signature:
Using CILFI to extract opcode handlers from KoiVM samples.
Finally, the command-line utility can be instructed to output JSON instead of flat text. This makes it suitable for assembly processing pipelines that do not use C# or AsmResolver as their main platform (e.g., for generating OldRod configurations):
JSON output
Final words
Pattern-matching code is an annoying but often necessary step when trying to build .NET deobfuscation tooling. It is time-consuming, tricky to get right, and just not really interesting compared to the actual deobfuscation logic.
CILFI was born out of pure frustration, but it has saved me countless hours of manual grunt work. If you are interested, give it a test drive yourself. CILFI can be used as a standalone command-line utility (i.e., NativeAOT compatible with no dependencies), or as a reusable library for your own projects:
Happy hacking!