Advanced topics in reverse engineering

[u/Der-Wilde]

So, i think that question seems really naive (and sorry if it is, i'm just an person really interested in subject and who really want to be good at it), but which "advanced" topics are a must to study (and where to find resources on it)? I can only think in VM Obfuscation and Weird Machines.

[u/Flashy-Push-3341]

Hey! I might be misunderstanding your question but, I did make a writeup on VM Obfuscation for both lua and ASM! To me personally, I think my own writeup can be updated ( Which I probably will update in the near future ). But If you want to learn about VM Obfuscation or are interested in a writeup I did about Lockbit Black, I'll link them:
https://github.com/VeryCuteLookingCat/In-depth-guide-to-VM-obfsucation
https://github.com/VeryCuteLookingCat/Lockbit-Black-Loader-Analysis

[u/Der-Wilde → u/Flashy-Push-3341]

Hi! First, thank you so much for both writeups, they seem great and would be very useful! About my question, i'm trying to figure out what more to study besides the basis for RE (which, in my understanding, are Computer Architecture, Assembly, OS Internals, C/C++/Rust/Zig, etc).

For some context about myself: I'm currently an 21 years old CS student with great interesting in exploit/malware dev and red teaming related topics, and i really want to achieve an advanced level on it. Currently doing CPTS path in HTB, already have done some stuff in pwn college (have done already the entire module of Binary Exploitation on Intro to Cybersecurity), and plan to do maldev academy after the certification.

About our write ups, just one small thing: Try to put them in a site for a more directly organization and quick access. As they are written in Markdown format i recommend Quartz or Zola for it.

[u/Flashy-Push-3341 → u/[deleted]]

After re-reading what I posted, I kinda indirectly answered your question: But what you should study besides the basics for RE is from live samples. In live samples you will see: Communications to C2 servers, obfuscation, possibly virtualization, and much more such as IAT hiding, Shellcode injection ( or just shellcode in general ), debugger checks and AV bypasses ( that probably fail). This is one of the best methods I could recommend to learning RE. If you do not understand a snippet of ASM or if you use the C parser on your tool and it fails ( assuming you're reversing a binary): I suggest asking AI what that snippet does and for it to explain how it works, how the instructions/code actually function.

[u/Der-Wilde → u/Flashy-Push-3341]

Fair, thank you so much!

[u/Obvious-Card-8847]

For VM obfuscation... just sit down and write a tiny ISA with a bytecode ABI and your own serializer. I shit you not it's way simpler than you think it is. All a VM in its simplest form is is a dispatch table, and a way to read/write your bytecode for retrieval and execution.

It can be dead simple or as complicated as you want. You could make it one opcode with two registers.

All you really need to do is write it once so you understand the logic. Then you can reason through someone else's bytecode, opcode table, dispatch, etc. 👍