Let's get things going with basics of file parsers fuzzing

So, one of my 2020 new year promises was to actually start writing to my blog. At the start, I thought writing about my current Hyper-V research, but because there has been some interest from people only entering the fuzzing scene, I decided to write about that. So let's get going and I will not start from extreme basics but close to it. I will also connect the explanations and descriptions with my own fuzzer newer version that I have already released last week ( https://github.com/FoxHex0ne/Vanapagan ).

Primary loop

So let us talk about fuzzing. I will start with a description of the primary fuzzing loop and how I have used it. So, in essence, my fuzzing loop for file parsers fuzzing has been something like that (presented in minimized python code for additional weirdness):

try:
    while True:
        clearRegistriesAndTemporaryFiles()
        originalFile = takeRandomFileFromCorrectFilesSet()
        (mutatedFile, mutationsInfo) = mutateFile(originalFile)
        crash = executeTargetedProgramWithInputFile(mutatedFile)
        if crash not null:
            logCrash(originalFile, mutatedFile, mutationsInfo, crash)
except:
    restart()

This is an overly simplified version of course, but I believe this is enough for an explanation. So let's start breaking everything apart:

• Outer try-except block and restart() 
  This is to catch all kinds of crap that might sooner or later happen while running a fuzzing loop for a longer time. Since I'm running a lot of fuzzing VM-s, then usually I just prefer that machine does the restart if something weird happens. After restart, the fuzzing simply continues. During the debugging phase, I disable restarting and just raise exception to see what happened.
• while True
  Well... it's a fuzzing loop you know :)
• clearRegistriesAndTemporaryFiles()
  This is the first part that becomes a bit more interesting. It's not needed with all the targets but with quite a lot of them still. This function would be doing the clearing up of temporary files or registry values that will otherwise influence the fuzzing process. For example, in the case of MS Office main programs (Word, Excel, Powerpoint), the safe mode can be activated when the last loop killed the main process (some registry values have to be cleared to avoid this). Or in the case of Foxit PDF reader, the temporary file is created in the temp-file directory. It is of course not deleted when a reader is killed off by the fuzzer (this would result in larger disk usage and slowdown of the reader startup after some time).
• originalFile = takeRandomFileFromCorrectFilesSet()
  I'm a big fan of mutation-based fuzzing. This is because I'm too lazy to write smart fuzzers for complex protocols and specifications (I also have a full-time job and university stuff currently so I can redirect blame for lack of time). So what I do, is I choose a set of correct inputs/files as a basis. In this line, I just select a random file from such a set.
• (mutatedFile, mutationsInfo) = mutateFile(originalFile)
  Now comes the mutation part, No matter what mutators you use (I have lately used mainly bit flipping and special values  - 0x00, 0xFF, 0xFE, 0x80, 0x7F, etc.) it's useful to also keep track of the mutations that were made. Of course, mutations can be deduced by comparing the original and mutated file, but this takes extra effort. Also, in some cases, the file is changed when opened by the targeted program (Excel & Word) and this creates additional confusion. All in all, as a result of mutation functionality, you are left with the mutated file and mutation info.
• crash = executeTargetedProgramWithInputFile(mutatedFile)
  In this part, the targeted executable is actually executed and the mutated file is provided to it as an input. This part has many moving parts in it and has to do multiple things. As a first thing, it should be functioning as a debugger (or control actual debugger) for detecting crashes inside the process. For example, if invalid memory read/write is detected, this functionality should return information about the crash (location, registry values, stack, etc.). Also, if the CPU usage from the process fells to 0%, it makes sense to stop the process. Besides, there should be an overall time limit for single process execution.
• if crash not null:
              logCrash(originalFile, mutatedFile, mutationsInfo, crash)
  This should already be quite clear from the existing context. If the crash is detected, then it should be logged somewhere along with the input that caused the crash and some initial analysis. Since the input is in the form of a mutated file, the mutation description and original file should also be stored (less work later)

And this is the main fuzzing loop that I and almost everyone else has used in one form or another.

"Vanapagan" fuzzer

So, in addition to this blog post, I have also made public the newer version of my simple Vanapagan fuzzer that is written in python (and not the nice Python 3 but now unsupported 2.7). Based on this I will describe how fuzzing loop explained before carries over to my actual fuzzer. Now what to keep in mind, is that this fuzzer is a very simple one - it just mutates the input and executes the unmodified target program. In many situations this is a rather slow way to do it - there are multiple ways to speed up execution or to test only certain parts of the main program. But I believe that this is something that comes a bit later and the goal of this post is to show the basics. So here comes overall comment and the main pieces of the Vanapagan fuzzer:



Overall usage:

• It uses WinAppDbg, pywin32 and psutil libraries. First one for debugging and rest for handling some overall process monitoring (CPU usage, etc.)
• Is configurable by JSON conf file
• I usually set it up, add autologin, automatic running, full page heap and restarting in case of exception. Then I let it run for a week or so and evaluate results. 

Pieces:

• fuzz.py
  Main loop logic and also a file to run when fuzzing
• Vanapagan\Detector\WinBasic.py
  It contains the class that does the actual debugging work and crash detection. It uses WinAppDbg library for that.
• Vanapagan\Loging\*.py
  There are two classes currently for logging any findings. One logs the results to the local filesystem (or network shared) directory and the second one can be used to log to an FTP server.
• Vanapagan\Mutator\*.py
  There is one base class and two implementations for file input mainpulation. One does the bit flipping and other places special value bytes into the input.
• Vanapagan\Utils\*.py
  Just some helper functions used by the main fuzzing loop
• Vanapagan\CrashReport.py
  Data class for the crash reports, used by the detector and logging classes.

Configuration example:

I will use fuzzing configuration for Excel as an example because it needs most of the functionality existing in my fuzzer. It should also bring out all the little things that you kind of have to think about when setting fuzzer up.

{
  "name": "excel32",

  "input": ".\\input",
  "retry": 1,
  "binaries": ["excel.exe"],
  "executable": "C:\\Program Files (x86)\\Microsoft Office\\root\\Office16\\excel.exe",
  "regsToDelete": ["HKCU\\Software\\Microsoft\\Office\\16.0\\Excel\\Resiliency"],
  "filesToDelete": [
    "C:\\Users\\jaanus\\AppData\\Local\\Temp\\*",
    "C:\\Users\\jaanus\\AppData\\Roaming\\Microsoft\\Office\\Recent\\*",
    "C:\\Users\\jaanus\\AppData\\Roaming\\Microsoft\\Windows\\Recent\\*"
  ],
  "windowToInteract": "Microsoft Excel",
  "windowToInteractKey": "y",
  "logNullCrashes": true,
  "maxWait": 90,
  "restartWhenException": false,
  "restartWhenLoop": false,

  "mutators": [
    {
      "type": "FileBitFlipping",
      "rate": 40000,
      "weight": 2
    },
    {
      "type": "FileByteValues",
      "rate": 60000
    }
  ],

  "logging": [
    {
      "type": "FtpsLoging",
      "host": "10.20.30.40",
      "username": "bot",
      "password": "s3cr3t",
      "dir": "/BOT/excel32"
    },
    {
      "type": "FilesystemLoging",
      "dir": ".\\crashes\\"
    }
  ]
}

So let's get going with different values in it:

• name - Name of the fuzzer conf, not used actually anywhere.
• input - Directory where input set locates.
• retry - How many times input is tested against target. Usually, 1 is enough, but in some cases, the target might crash for other reasons the input itself. With such targets, the retry should be higher, so in case of the crash, the input is retested to make certain that the crash was caused by it.
• binaries - List of binaries that are monitored for CPU usage. This is usually not that much needed but could be useful if the target program executes some other applications indirectly and waits for them. In such cases, the primary program CPU usage is not the only one you should monitor. In the current example, it's "excel.exe" itself, so actually, it has no meaning.
• executable - Target program location.
• regsToDelete - Registry values to delete before every execution. In the current case, it's for removing the possibility of a "safe mode" startup for excel or the recovery of old files.
• filesToDelete - Files to delete before every execution.
• windowToInteract - Window for sending keypress (windowToInteractKey value) to. In the current case, it's for the "do you want to repair this file" dialog.
• windowToInteractKey - Keypress that is sent to the window described in the previous point.
• logNullCrashes - Should we store also crashes that happen in the NULL page.
• maxWait - Maximum execution time of the target application. It is triggered when CPU usage never fells to 0.
• restartWhenException - Should fuzzer restart the computer if an exception happens. Useful during real fuzzing but should be turned off when debugging.
• restartWhenLoop - If fuzzer has looped through all input set, should the computer restart.
• mutators - What mutations are used:
• Type "FileBitFlipping" - Random bit flipping
• Type "FileByteValues" - Special byte, word and dword values
• rate - For how many bytes one change is made. So if in first mutator the rate value is 40 000 and the file size is 1 000 000 bytes, then it results 1 000 000 / 40 000 = 25 changes made.
• weight - The mutator used in every execution is chosen kind of randomly. So weight value gives mutator importance. In the current configuration, there is 2 mutator and the weight of the first one is 2. This means that 2/3 of times, the first one is used and 1/3 of times the second one.
• logging - What results logging are used:
• Type "FtpsLoging" - Logging to FTP server
• Type "FilesystemLoging" - Logging to the local filesystem
• All other values should be quite self-explaining

How I use it all:

1. Prepare crash collecting FTP server
2. Prepare VM (install & update OS, Office, Python and libraries)
3. Enable full page heap for Excel
4. Set up fuzzer (input set, conf file, etc.)
5. Test fuzzer without restarting ("restartWhenException" should be "false") until it works fine
6. Activate restarting in fuzzer conf
7. Activate autologon ( https://docs.microsoft.com/en-us/sysinternals/downloads/autologon )
8. Add fuzzer execution into "run" registry key
9. Clone & run as many VMs as possible
10. Check up on VMs after a day just in case
11. Wait a week
12. Look are there any good crashes found
      NO> Cry a bit and change fuzzing conf or pick a new target
      YES> Analyse them and leave fuzzers running for another week

Why mutation-based fuzzing

So I thought that I should also explain the reasons why & how I have chosen mutation-based fuzzing, not some other forms. The main reason is the "results for time spent" calculation. Very dumb fuzzing can't find any results in these days and while smart fuzzing is WAY better, it also takes exponentially more time do develope. This is especially important since I'm actually not doing such research for a living (at least not as a main income) and I don't want to waste a year of free time on writing smart fuzzer that might not even work that well. The mutation-based fuzzing is a good compromise - it's still interesting because you have to find a good initial set of inputs that you can start mutating (it's not that useful to choose it randomly), it does not take that long to prepare and it still gives some results. Of course, it does not always work - for example, it's harder to fuzz browser javascript engine like that or syscalls.
Overall I try to put maximum effort into collecting the best input set possible that will cover a wide range of functionality inside the targeted program. This combined with a stable fuzzing process means that I can skip other optimization options and still get results without sinking too much manual work time. The results, of course, vary heavily with such approach but I have been still quite successful by doing that.
Also, because this approach can't work that well with other types of targets that need a lot of manual work, it gives the possibility to do various types of research concurrently.

Other notes

Some additional things are useful to know before actual fuzzing (I use fuzzing in the Windows environment as an example, but similar points go for all OS-es and settings):

• Full page heap
  Activate a full page heap for a process that you are fuzzing. Look for description about it from the MS gflags page, but it's pretty much mandatory to be used when fuzzing applications that use windows heap manager. Without this, you might still catch crashes, but they will happen in quite random places. 
• Crash info gathering
  Storing crashes in the local filesystem works well when testing on a single machine. It is way less fun when running tens or hundreds of VMs. I have mostly used an FTP server or network drive to gather all crashes to a single location.
• Initial analysis
  This is not important if your target program crashes very rarely. But it becomes critical when there are a lot of crashes. In that case, you want to do at least some initial filtering and categorization of the found issues. I, for example, skip NULL page crashes and rest sort by crash location (this is also not enough if the target program is very broken and you have to start separating by stack trace, but I haven't needed it).
• Disable sleep
  I have forgotten a LOT of times with new VMs to disable sleep for OS. Because fuzzer works without user input, the OS sometime decides that there is no user activity and will go to sleep. This is not good, of course...
• GUI actions
  "File seems broken, do you want to try to fix it?" and other such dialog boxes. You usually want your fuzzer to say "hell yes" to them.

Ending comments

Everything covered in this post was as basic and introductory as possible. Fuzzing programs using such simple tools is not that effective resource-wise and not at all fancy. It still works if you have a good initial set of files, but even then, there are more effective ways to do the actual fuzzing part. But at the same time, I have been in MSRC top researchers list for 4 years now and mainly thanks to this tool and such approach. So it DOES WORK, but if I would do such research full time or my livelihood would depend on that, I would look for ways to speed up fuzzing (will write about this also in future).

In my next blog post, I will cover my way of collecting the input sets for mutation-based fuzzing. I will use my Rehepapp tool as an example and will also release some new utilities for it. I hope to get it done before the end of January because it connects strongly to the current post. But no promises...