This is part of a series of blog posts on undocumented opcode fuzzing. Previous research into undocumented opcodes such as Chris Domas’ Sandsifter project had tested them in ring 3 (user mode), but not in ring 0 (kernel mode). I was very keen to test out opcodes in ring 0 because many of them failed with GP exceptions in ring 0. GP can be an indication that an opcode exists but is privileged, although there are lots of other causes too. I was convinced I was going to discover some really interesting undocumented instructions if only I could run them in ring 0. Spoiler alert: sadly this wasn’t the case. But along the way I spent a lot of time digging around in the Linux kernel and figured out how to handle UD exceptions to test millions of instructions at a time. If you happen to have similarly misguided aims as me (‘why of course I’ll be deliberately running illegal instructions in the kernel’) then I have a solution for you! The only resources I could find suggested modifying the Interrupt Descriptor Table (IDT) or modifying the kernel code and rebuilding it; this solution requires neither and can all be done in C from within your kernel driver.

This is part of a series of blog posts on undocumented opcode fuzzing. When I started trying to test undocumented opcodes, I struggled to find a technique for integrating machine code into higher-level code. I wanted to create a C program automating the testing and analysis of millions of undocumented instructions, so it had to be machine code rather than assembly as there were no mnemonics for them. You can of course create a pure hex file and convert it into a binary, but that’s a nightmare if you want to code the equivalent of thousands of lines of C! I hope you find these techniques for executing machine code in user mode and kernel mode useful. The user mode technique is adapted from several examples on GitHub (unfortunately I can’t now find the original posts) and can also be implemented in Python.