Malware Anti-Debugging Techniques

#Concepts You Need to Know First

Before any of the anti-debug techniques make sense, you need a mental model of how a running Windows program is structured. This section explains the building blocks in plain language.

#What Is a Process?

A process is a running program. When you double-click malware.exe, Windows creates a process: it allocates memory, loads the executable's code into that memory, and starts executing it. Each process has:

• Its own private virtual address space — memory addresses that only this process can access. Two processes can each have data at address 0x00401000, but they are completely separate regions of RAM.
• A list of handles (see below) to resources it has opened.
• At least one thread doing the actual execution.

Processes cannot read each other's memory directly — the OS enforces isolation. A debugger can bypass this because the OS grants it special access.

#What Is a Thread?

A thread is the actual unit of execution inside a process. A process is a container (memory, handles, identity); a thread is what actually runs instructions on the CPU.

• A process always has at least one thread (the "main thread").
• Multi-threaded programs have several threads running concurrently.
• Each thread has its own call stack (where local variables live and where function return addresses are stored) and its own set of CPU registers.
• Importantly for anti-debugging: debug events are delivered per-thread. A debugger is notified when a specific thread hits a breakpoint, not when "the process" does.

#What Is a Handle?

A handle is a number (like a file descriptor in Linux) that represents an open reference to a kernel object — a file, a process, a thread, a mutex, etc. When you open a file with CreateFile(), Windows gives you back a handle. You pass that handle to ReadFile(), CloseHandle(), etc. The kernel keeps the actual object; you just hold a ticket to it.

For anti-debugging: when a debugger attaches to a process, a debug object is created in the kernel and a handle to it is stored. Certain APIs let a process ask "do I have a debug object handle?" — which is a direct signal that a debugger is present.

#What Is User Space vs Kernel Space?

The CPU has privilege levels (rings). Ring 0 is kernel mode — only the OS kernel runs here. Ring 3 is user mode — your programs, including malware and debuggers, run here.

Ring 3 (User mode):   Your process, malware, debugger, ntdll.dll, kernel32.dll
         ↕  System call (interrupt / syscall instruction)
Ring 0 (Kernel mode): Windows kernel (ntoskrnl.exe), drivers, memory management

When user-mode code needs to do something privileged (open a file, allocate memory, create a thread), it issues a system call — a controlled transition from ring 3 to ring 0. The kernel performs the action and returns the result.

Why this matters for anti-debuggingSome anti-debug techniques operate in user mode (easy for analysts to bypass by patching). Others operate in kernel mode (much harder to bypass without a kernel debugger). Many tricks exploit the boundary between them.

#What Is ntdll.dll?

ntdll.dll is the lowest-level DLL that runs in user space. It contains thin "wrapper" functions that make system calls. For example, when you call ReadFile() in kernel32.dll, it eventually calls NtReadFile() in ntdll.dll, which contains the actual syscall instruction that transitions to the kernel.

Most EDR products and anti-cheat systems work by hooking ntdll.dll — patching the first few bytes of functions like NtOpenProcess to redirect execution to their own inspection code. Many EDR evasion techniques (covered in edr-evasion.md) specifically target these hooks.

Nt functions: Functions starting with Nt or Zw in ntdll are direct wrappers around kernel system calls. NtQueryInformationProcess, NtSetInformationThread, NtCreateThreadEx — these are the raw system call wrappers. The more familiar QueryFullProcessImageName, TerminateThread, CreateThread in kernel32.dll are higher-level wrappers that eventually call the Nt versions.

#What Is the PEB (Process Environment Block)?

The PEB is a data structure the Windows kernel creates for every process in that process's own memory (user space). It is a block of information the OS maintains about the process — things like:

• The base address of the executable image
• A pointer to the list of loaded DLLs
• Environment variables
• The heap pointer
• A BeingDebugged flag (1 byte)
• A NtGlobalFlag field
• And dozens of other fields

The PEB lives at a fixed, well-known address that the process can find via a CPU segment register:

• x64: GS:[0x60] — the GS segment register points to a structure called the TEB (Thread Environment Block), and GS:[0x60] gives the PEB address
• x86: FS:[0x30]

Because the PEB is in user space, both the process itself and a userspace debugger can read and write it. The OS sets BeingDebugged = 1 when a debugger attaches, but a debugger can also set it back to 0 to hide itself — this is an arms race.

#What Is the Windows Heap?

The heap is a region of memory used for dynamic allocation — when code calls malloc() or HeapAlloc(), it gets memory from the heap. The heap manager (part of ntdll.dll) tracks which regions are in use and which are free.

The heap is not just a flat pool — it has a header structure (_HEAP) at its start containing metadata. This metadata changes depending on whether the process was started under a debugger. When a debugger is present, Windows configures the heap with extra debugging features (fill patterns to detect overflows, extra checks on free). The _HEAP.Flags and _HEAP.ForceFlags fields in the header reflect this, and malware reads them to detect the presence of a debugger.

#What Is SEH (Structured Exception Handling)?

SEH is Windows' mechanism for handling runtime errors — things like dividing by zero, accessing a null pointer, or executing invalid code. In C/C++, you use __try / __except blocks. When an exception occurs:

1. The CPU stops executing the faulting instruction
2. Windows walks up the call stack looking for a registered exception handler (__except block)
3. If found, execution jumps to that handler
4. If not found, the unhandled exception filter runs (set via SetUnhandledExceptionFilter)
5. If that also doesn't handle it, the process crashes

The critical detail for anti-debugging: a debugger intercepts exceptions before they reach the process's own handlers. This is how breakpoints work — the INT 3 instruction causes an exception, which the debugger receives. Malware exploits this by deliberately triggering exceptions and checking whether its own handler received them (no debugger) or not (debugger present).

#What Are CPU Debug Registers?

Modern x86/x64 CPUs have a set of debug registers (DR0–DR7) that exist specifically to support hardware breakpoints:

When a debugger sets a hardware breakpoint, it writes the target address into DR0–DR3 and configures DR7. There is no modification to the code at the target address (unlike software breakpoints which write 0xCC). However, these register values are readable by the process itself via GetThreadContext() — which is how malware detects hardware breakpoints.

#What Is a Symbol / Stripped Binary?

A symbol is a name attached to an address in a binary — function names, variable names, type information. When a compiler builds a debug build, it includes symbols so debuggers can show main() instead of 0x00401234. A stripped binary has had symbols removed, making it harder to analyse — you see raw addresses and have to figure out what each function does.

#How Debuggers Work — What Malware Is Detecting

Now that you have the foundations, here is exactly what happens when you attach a debugger to a process and why each action leaves a trace:

You open x64dbg and attach to malware.exe:

Step 1: Debugger calls DebugActiveProcess(malware_pid)
   → The OS kernel:
     - Creates a "debug object" (a kernel data structure for receiving debug events)
     - Sets malware's EPROCESS.DebugPort = pointer to debug object  (kernel-only)
     - Sets malware's PEB.BeingDebugged = 1  (visible in user space)
     - Sets NtGlobalFlag heap bits if the process was STARTED by the debugger
   → Malware can now detect: PEB.BeingDebugged, debug port, debug object handle

Step 2: You press F2 to set a software breakpoint at address 0x00401234
   → Debugger calls WriteProcessMemory to write 0xCC (INT 3) over the byte at 0x00401234
   → The original byte is saved in the debugger's internal breakpoint list
   → Malware can detect: 0xCC bytes in its own code via self-checksum

Step 3: Malware executes and reaches 0x00401234
   → CPU executes 0xCC → raises EXCEPTION_BREAKPOINT (0x80000003)
   → OS sees the debug port is non-NULL → sends exception to the debugger
   → Debugger restores the original byte, pauses execution, shows you the UI
   → Malware can detect: its own exception handler was NOT called (debugger consumed it)

Step 4: You press F7 (single step)
   → Debugger sets the Trap Flag (bit 8) in the EFLAGS register of the thread
   → CPU executes one instruction, then raises EXCEPTION_SINGLE_STEP (INT 1)
   → OS sends it to the debugger, debugger updates the UI
   → Malware can detect: execution is much slower than expected (timing)

Step 5: You set a hardware breakpoint (F3 in x64dbg)
   → Debugger calls SetThreadContext to write the address into DR0 and configure DR7
   → No code is modified — the breakpoint is purely in CPU registers
   → When execution hits that address, the CPU raises a debug exception
   → Malware can detect: DR registers are non-zero via GetThreadContext()

Every one of these steps leaves a detectable side effect. Anti-debugging techniques are simply ways to check for those side effects.

#Windows Anti-Debug — PEB and Heap Inspection

This group of techniques reads data structures that Windows maintains in the process's own memory. Because this data is in user space, it can be read without making any system calls — and without triggering any EDR or anti-cheat hook.

#PEB.BeingDebugged — The Simplest Check

What it isA single byte at offset +0x02 inside the PEB. The Windows kernel sets it to 1 when a debugger attaches, and back to 0 when the debugger detaches.

Why malware uses itIt's a one-instruction check — faster and stealthier than making an API call. No function call means no EDR hook fires.

How to find the PEBOn 64-bit Windows, the CPU's GS segment register always points to the current thread's TEB (Thread Environment Block). The TEB is a per-thread structure, and at offset +0x60 inside it is the pointer to the PEB. So GS:[0x60] gives you the PEB address in one instruction.

// The "normal" way — just call the API
BOOL result = IsDebuggerPresent();
// Returns TRUE if debugged, FALSE if not
// This is literally just: read PEB.BeingDebugged and return it

// What IsDebuggerPresent() actually does under the hood (x64):
PPEB pPeb = (PPEB)__readgsqword(0x60);  // read PEB pointer from GS:[0x60]
if (pPeb->BeingDebugged) {              // check byte at PEB+0x02
    exit(1);
}

// Malware often does it in raw assembly to avoid the API call showing in import table:
// mov rax, gs:[60h]   ; rax = PEB pointer
// movzx eax, [rax+2]  ; eax = PEB.BeingDebugged byte
// test eax, eax
// jnz debugger_detected

Why this alone is weakThe very first thing ScyllaHide (a debugger plugin) does when it attaches is set PEB.BeingDebugged = 0. The check is so well-known that every decent anti-anti-debug tool patches it automatically.

BypassScyllaHide or any anti-anti-debug plugin patches this to 0. A kernel-mode debugger (WinDbg in kernel debugging mode) does not set this byte at all because it operates at ring 0 and doesn't use the user-mode debug API.

#NtGlobalFlag — Detects How the Process Was Started

What it isA 32-bit field at PEB+0x68 (x86) or PEB+0xBC (x64) that holds global flags affecting system behaviour. When a process is started under a debugger (as opposed to a debugger attaching later), Windows automatically sets three bits in this field:

• Bit 4 (0x10) = FLG_HEAP_ENABLE_TAIL_CHECK — add extra bytes after heap allocations to detect overflows
• Bit 5 (0x20) = FLG_HEAP_ENABLE_FREE_CHECK — check heap blocks on free for corruption
• Bit 6 (0x40) = FLG_HEAP_VALIDATE_PARAMETERS — validate heap function parameters

Together these three bits sum to 0x70. A normally-started process has NtGlobalFlag == 0.

The key difference from BeingDebuggedBeingDebugged is set/cleared as debuggers attach and detach. NtGlobalFlag is only set when the process is launched by the debugger (via CreateProcess with debug flag). If you start the process normally and then attach the debugger later, NtGlobalFlag stays 0. So combining both checks catches both attachment scenarios.

PPEB pPeb = (PPEB)__readgsqword(0x60);

// Check if any of the three debug-heap bits are set
if (pPeb->NtGlobalFlag & 0x70) {
    // Process was started under a debugger
    exit(1);
}

BypassScyllaHide patches this field to 0 before the process code runs.

#Heap Flags — A Deeper Check of the Same Condition

What it isWhen NtGlobalFlag has the heap debug bits set, the heap manager reads them during initialisation and copies them into the heap's own header structure (_HEAP). So there are two places recording the same fact: the PEB's NtGlobalFlag, and the heap header's Flags and ForceFlags fields.

Why check both?Because ScyllaHide patches NtGlobalFlag but may forget (or be too late) to patch the heap header. Reading the heap header directly is a more robust check.

How to find the heap headerThe PEB contains a ProcessHeap pointer — the address of the process's default heap. The heap header is at that address. Its Flags field is at a different offset on 32-bit vs 64-bit Windows.

// Step 1: Get the PEB
PPEB pPeb = (PPEB)__readgsqword(0x60);

// Step 2: Get the default heap address (PEB.ProcessHeap)
PVOID pHeap = pPeb->ProcessHeap;

// Step 3: Read the Flags field from the heap header
// (These offsets are fixed by the OS version for a given architecture)
// x64 Windows: Flags at offset +0x14, ForceFlags at +0x18
DWORD heapFlags    = *(DWORD*)((BYTE*)pHeap + 0x14);
DWORD forceFlags   = *(DWORD*)((BYTE*)pHeap + 0x18);

// Normal process:         heapFlags = 0x02 (HEAP_GROWABLE only), forceFlags = 0
// Under debugger launch:  heapFlags has extra bits, forceFlags is non-zero
if ((heapFlags & ~0x02) || forceFlags != 0) {
    exit(1);
}

BypassScyllaHide patches these too. Alternatively, if the malware allocates its own heap (HeapCreate) rather than using the default one, the debug flags won't be present there — some malware avoids the default heap entirely.

#Windows Anti-Debug — API Queries

These techniques ask the Windows kernel directly whether a debugger is attached. They are more reliable than PEB tricks but more visible — they make system calls that EDR hooks can intercept.

#CheckRemoteDebuggerPresent — The Official Way

What it isA Win32 API function in kernel32.dll that asks the kernel "is this process being debugged?" It works by calling NtQueryInformationProcess (see below) with the right parameters. The "Remote" in the name is misleading — it just means it can query a different process, but you can also use it to query yourself.

What it checksIt asks the kernel for the process's debug port — the channel through which the debugger receives events. If the port exists (non-NULL handle), a debugger is attached.

BOOL bDebuggerPresent = FALSE;

// Query the current process itself
CheckRemoteDebuggerPresent(GetCurrentProcess(), &bDebuggerPresent);

if (bDebuggerPresent) {
    exit(1);
}

BypassScyllaHide hooks this function and always writes FALSE into bDebuggerPresent. A kernel debugger doesn't create a user-mode debug port, so this check returns FALSE against kernel debuggers naturally.

#NtQueryInformationProcess — Three Debug Information Classes

What it isNtQueryInformationProcess is a system call (in ntdll.dll) that retrieves all kinds of information about a process. It takes an "information class" number that selects what you want to know. Three of these classes are directly related to debugging.

Think of it as: "kernel, please tell me property #N of this process." The kernel checks the process's internal state and returns the answer.

Why use the raw Nt function instead of CheckRemoteDebuggerPresent?* Because CheckRemoteDebuggerPresent only checks one information class. By calling NtQueryInformationProcess directly, malware can check three separate classes — and debugger anti-anti-debug tools might only patch some of them.

// Information class 7 = ProcessDebugPort
// The kernel returns a handle to the debug port.
// Non-NULL = debugger attached. NULL = clean.
// (This is what CheckRemoteDebuggerPresent uses internally)
HANDLE hDebugPort = NULL;
NtQueryInformationProcess(GetCurrentProcess(), 7, &hDebugPort, sizeof(HANDLE), NULL);
if (hDebugPort != NULL) exit(1);

// Information class 30 = ProcessDebugObjectHandle
// Returns a handle to the debug *object* (a different kernel structure).
// Some debuggers (especially older ones) may not create this, so it's a cross-check.
HANDLE hDebugObj = NULL;
NTSTATUS status = NtQueryInformationProcess(GetCurrentProcess(), 30, &hDebugObj, sizeof(HANDLE), NULL);
// If status == STATUS_PORT_NOT_SET, no debug object → no debugger
if (NT_SUCCESS(status) && hDebugObj != NULL) exit(1);

// Information class 31 = ProcessDebugFlags
// Returns 0 if being debugged, 1 if not. (Inverted logic — easy to confuse)
DWORD debugFlags = 1;
NtQueryInformationProcess(GetCurrentProcess(), 31, &debugFlags, sizeof(DWORD), NULL);
if (debugFlags == 0) exit(1);

Why use all threex64dbg's ScyllaHide patches class 7 reliably. Classes 30 and 31 are patched by newer versions but not all of them. Using all three means the malware catches partially-patched analysis setups.

BypassScyllaHide hooks the ntdll NtQueryInformationProcess function and returns "safe" values for all three classes. A kernel debugger bypasses all user-mode hooks entirely because it operates below ntdll.

#Windows Anti-Debug — Exception Handling Tricks

These are the most elegant anti-debug techniques. They rely on the fundamental difference in how exceptions are handled with and without a debugger present.

Quick reminderWhen an exception occurs (access violation, division by zero, breakpoint instruction), the OS has to decide who handles it. If a debugger is attached, the OS offers the exception to the debugger first — before the process's own exception handlers. If the debugger "handles" it (consumes it and resumes), the process's __except block never runs.

#INT 3 — Breakpoint Exception Trick

What INT 3 isINT 3 (opcode 0xCC) is the CPU's software breakpoint instruction. It generates an exception with code EXCEPTION_BREAKPOINT (0x80000003). Debuggers use it constantly — every software breakpoint you set in x64dbg is a 0xCC byte written into the target address.

The anti-debug trickMalware deliberately executes INT 3 itself and sets up an SEH (__try/__except) handler to catch the resulting exception. Then it checks whether its handler ran:

__try {
    __asm { int 3 }                    // trigger a breakpoint exception
    // We reach here ONLY if the debugger consumed the exception above
    // (the __except block was skipped)
    exit(1);                           // debugger present → bail out
} __except(EXCEPTION_EXECUTE_HANDLER) {
    // We reach here ONLY if our handler caught it (no debugger)
    // Normal execution continues
}

Method 2 — Code ChecksumSince a debugger writes 0xCC bytes into code to set breakpoints, malware can scan its own code and look for 0xCC bytes where they shouldn't be:

// Scan the first 100 bytes of this function for breakpoint bytes
BYTE* code_start = (BYTE*)&my_function;
for (int i = 0; i < 100; i++) {
    if (code_start[i] == 0xCC) {
        // A software breakpoint has been placed here
        exit(1);
    }
}

Bypass for Method 1In x64dbg, press Shift+F9 ("pass exception to program") when the breakpoint fires. This tells the debugger to let the exception reach the process's own handler.

Bypass for Method 2Use hardware breakpoints (F3 in x64dbg) instead of software breakpoints. Hardware breakpoints don't write 0xCC — they use the CPU's DR registers. No 0xCC bytes appear in the code.

#INT 2D — The Skipped-Byte Trick

What INT 2D isINT 2D is a CPU interrupt that Windows uses internally for kernel-mode debug breakpoints. When a user-mode program executes INT 2D, it generates an exception that Windows handles. The trick is in how a debugger handles it differently from the OS:

; Layout in memory:
; [INT 2D instruction][NOP][start of real next instruction]
;
; Without debugger: INT 2D → exception → handler → resume at NOP → execute NOP → continue normally
; With debugger:    INT 2D → debugger skips next byte → skips NOP → lands one byte INTO the next instruction → wrong execution

int  2Dh             ; the INT 2D instruction
nop                  ; this byte is SKIPPED under a debugger
call real_function   ; without debugger: runs correctly
                     ; with debugger: executed from 2nd byte → wrong opcode → crash/wrong path

BypassScyllaHide handles INT 2D. Older debuggers may not. Patch the INT 2D byte to NOP NOP in the binary before analysis.

#CloseHandle with an Invalid Handle

What CloseHandle isCloseHandle() releases a handle — it tells the kernel "I'm done with this resource." If you pass a handle that doesn't exist or was already closed, it normally just returns FALSE with ERROR_INVALID_HANDLE.

The trickThere is one exception to this quiet failure — if a debugger is attached, calling CloseHandle with an invalid handle causes the kernel to raise an EXCEPTION_INVALID_HANDLE exception (0xC0000008). Without a debugger, the call just silently fails with no exception.

This is a feature of the Windows kernel: when a debug port is attached, certain error conditions are escalated to exceptions to help developers catch bugs. Malware weaponises this:

__try {
    CloseHandle((HANDLE)0xDEADBEEF);   // an obviously fake/invalid handle value
    // If we reach here, no exception was raised → no debugger
} __except(GetExceptionCode() == 0xC0000008   // EXCEPTION_INVALID_HANDLE
           ? EXCEPTION_EXECUTE_HANDLER
           : EXCEPTION_CONTINUE_SEARCH) {
    // We caught an EXCEPTION_INVALID_HANDLE → a debugger is attached
    exit(1);
}

Why this is hard to spotCloseHandle looks like perfectly normal cleanup code. There's no suspicious-looking API call. Nothing in the import table gives it away.

BypassIn x64dbg, go to Debug → Exceptions and add 0xC0000008 to the "pass to program" list. This tells the debugger to forward the exception to the process's own handler instead of consuming it.

#SetUnhandledExceptionFilter — Last-Resort Handler

What it isSetUnhandledExceptionFilter registers a custom "last resort" function that runs if an exception isn't caught by any __except block anywhere in the call stack. It's the very last chance to handle an error before the process crashes.

The trickMalware registers its own function as the unhandled exception filter. Then it deliberately triggers an exception that nothing in the call stack handles (like writing to address 0). Under normal conditions, the unhandled exception filter runs and the malware continues. Under a debugger, the debugger intercepts the exception first — the unhandled exception filter is never called.

LONG WINAPI MalwareUnhandledFilter(EXCEPTION_POINTERS* ep) {
    // We reach here only if no debugger is present
    // Decrypt the real payload, continue execution
    DecryptNextStage();
    return EXCEPTION_CONTINUE_EXECUTION;
}

// Register our filter
SetUnhandledExceptionFilter(MalwareUnhandledFilter);

// Deliberately crash — this exception has no __except handler anywhere
int* null_ptr = NULL;
*null_ptr = 0;  // access violation at address 0x00000000

// WITHOUT debugger: OS calls our filter → we decrypt and continue
// WITH debugger:    debugger catches the access violation first → our filter never runs
//                  → payload stays encrypted → analyst sees nothing interesting

Why this is powerful for malwareIt doubles as anti-analysis. The payload is only decrypted when running without a debugger — so even if the analyst gets a memory dump, they see an encrypted blob.

BypassIn x64dbg, when the access violation fires, press Shift+F9 to pass the exception to the process. Or use ScyllaHide which patches the exception filter mechanism.

#Windows Anti-Debug — Timing Attacks

The ideaA debugger slows execution. Single-stepping takes tens of milliseconds per instruction. Software breakpoints pause execution. Even just logging to a debugger window takes time. If malware measures how long a code block takes to run and it's much slower than expected, a debugger is probably present.

#RDTSC — CPU Cycle Counter

What RDTSC is"Read Time Stamp Counter." A single x86 CPU instruction that reads a 64-bit counter that increments every CPU clock cycle. On a 3GHz processor, this counter increases by 3 billion per second. This counter is read directly from the CPU — there's no function call, no system call, nothing to hook.

Why it's effectiveMost timing APIs (GetTickCount, Sleep) can be hooked and faked by ScyllaHide. RDTSC reads raw hardware. Hooking it requires hypervisor-level virtualisation tricks, which most analysis setups don't have. It also catches VM-based sandboxes that emulate RDTSC with fake values.

; Read counter before code
rdtsc                 ; result goes into EDX:EAX (high 32 bits : low 32 bits)
mov [timestamp1], eax

; ... the code we're timing ...

; Read counter after
rdtsc
sub eax, [timestamp1]  ; difference in CPU cycles

; At 3GHz: 2,000,000 cycles ≈ 0.67ms
; Single-stepping 10 instructions might take 500,000,000 cycles (hundreds of ms)
cmp eax, 0x200000      ; if > 2 million cycles...
ja  debugger_detected  ; ...execution was too slow → debugger present

// C equivalent using compiler intrinsic
unsigned __int64 t1 = __rdtsc();
do_something_short();
unsigned __int64 t2 = __rdtsc();

if ((t2 - t1) > 2000000ULL) {
    exit(1);  // too slow
}

BypassDon't single-step through the timed section. Use "Run to cursor" or "Run to next breakpoint" to jump over the code being timed in a single step. Or use ScyllaHide which patches RDTSC to return a constant increment each time.

#QueryPerformanceCounter / GetTickCount

Higher-level timing APIs that are easier to spoof but still catch many debugger setups. QueryPerformanceCounter uses the hardware performance counter (like RDTSC but through a system call). GetTickCount returns milliseconds since Windows started.

LARGE_INTEGER t1, t2, freq;
QueryPerformanceFrequency(&freq);   // ticks per second of the hardware counter
QueryPerformanceCounter(&t1);

do_something_short();               // should take microseconds

QueryPerformanceCounter(&t2);
double elapsed_ms = (t2.QuadPart - t1.QuadPart) * 1000.0 / freq.QuadPart;

if (elapsed_ms > 500.0) {           // > 500ms for a tiny operation → debugger stepping
    exit(1);
}

BypassScyllaHide patches these to return deterministic fake values that always show a tiny elapsed time. Or skip the timed section with "Run to cursor."

#Windows Anti-Debug — Hardware Breakpoint Detection

What hardware breakpoints are (expanding on the primer): Unlike software breakpoints which write 0xCC into code, hardware breakpoints are set by writing an address into the CPU's DR0–DR3 registers and enabling them via DR7. When the CPU executes an instruction at that address (or reads/writes to it, depending on type), it raises a debug exception without any code modification.

This makes hardware breakpoints invisible to code checksums and 0xCC scans. But they have one weakness: the registers are part of the thread's context and can be read by the process itself.

What GetThreadContext is: Each thread has a saved snapshot of all CPU registers (called the "thread context") that the OS maintains. Functions like GetThreadContext and SetThreadContext let code read and modify these registers. Debuggers use SetThreadContext to set the DR registers (hardware breakpoints). Malware uses GetThreadContext to read them and see if the analyst set any.

CONTEXT ctx = {};                            // a structure holding all CPU register values
ctx.ContextFlags = CONTEXT_DEBUG_REGISTERS;  // we only want the DR registers
GetThreadContext(GetCurrentThread(), &ctx);  // ask the OS for this thread's register values

// Check all four hardware breakpoint address registers
if (ctx.Dr0 != 0 || ctx.Dr1 != 0 || ctx.Dr2 != 0 || ctx.Dr3 != 0) {
    // At least one hardware breakpoint is set
    exit(1);
}

// DR7 is the control register — check if any breakpoints are enabled
// even if the addresses happen to be 0 somehow
if ((ctx.Dr7 & 0xFF) != 0) {
    exit(1);
}

Bonus: malware can also clear the analyst's breakpoints

Because SetThreadContext is also available to the process, malware can zero out all the DR registers, wiping the analyst's hardware breakpoints:

CONTEXT ctx = {};
ctx.ContextFlags = CONTEXT_DEBUG_REGISTERS;
GetThreadContext(GetCurrentThread(), &ctx);

// Destroy all hardware breakpoints
ctx.Dr0 = ctx.Dr1 = ctx.Dr2 = ctx.Dr3 = ctx.Dr7 = 0;
SetThreadContext(GetCurrentThread(), &ctx);

// The analyst's breakpoints are now gone without warning

BypassAfter the malware clears your breakpoints, re-set them from the debugger console. Watch for loops — some malware clears breakpoints in a loop. In that case, set a breakpoint on SetThreadContext itself (the irony being you need a breakpoint to catch the breakpoint-clearing code).

#Windows Anti-Debug — Thread Hiding

What it isA rarely-known feature of the Windows kernel allows a thread to "hide itself" from the debugger. After a specific system call, the kernel stops sending debug events (breakpoints, single steps, exceptions) from that thread to the attached debugger. The thread keeps running normally — the analyst just becomes blind to it.

The system callNtSetInformationThread is an ntdll function that modifies properties of a thread. One of its "information classes" (numbered properties) is ThreadHideFromDebugger (class number 0x11). Calling it with this class is a one-way operation — it cannot be reversed.

Why use ntdll directly?The higher-level SetThreadInformation in kernel32 doesn't expose this specific class. Malware calls NtSetInformationThread by getting its address from ntdll at runtime (via GetProcAddress), avoiding the function appearing in the import table.

// Get the function address from ntdll — it's not in kernel32's "safe" exports
typedef NTSTATUS (NTAPI* pNtSetInformationThread)(
    HANDLE ThreadHandle,
    THREADINFOCLASS ThreadInformationClass,
    PVOID ThreadInformation,
    ULONG ThreadInformationLength
);

pNtSetInformationThread NtSIT = (pNtSetInformationThread)
    GetProcAddress(GetModuleHandleA("ntdll.dll"), "NtSetInformationThread");

// Hide THIS thread from the debugger (0x11 = ThreadHideFromDebugger)
// After this line: any breakpoints in this thread silently don't fire,
// single-stepping produces no events, exceptions are handled by the OS directly.
NtSIT(GetCurrentThread(), (THREADINFOCLASS)0x11, NULL, 0);

In plain termsImagine the analyst sets a breakpoint at address X. When execution hits X, the CPU raises an exception. Normally that exception goes to the debug port and the debugger pauses. With thread hiding, the kernel simply doesn't deliver the event to the debugger — the malware keeps running right through the "breakpoint" as if it wasn't there.

BypassA kernel-mode debugger (WinDbg connected via a COM port or KDNET, debugging the whole OS rather than just one process) does not rely on the user-mode debug port — it gets its events directly from the kernel at a level below this mechanism. In user-mode debugging: break on NtSetInformationThread itself (before the call), identify when information class 0x11 is being used, and change the argument to a harmless class number.

#Windows Anti-Debug — TLS Callbacks

What TLS isThread Local Storage (TLS) is a mechanism for giving each thread its own private copy of a variable. Think of a global variable where each thread sees its own version rather than a shared one. Implemented as a section in the PE file called .tls.

What a TLS callback isPart of the TLS mechanism is a list of callback functions that the Windows loader calls automatically when a thread starts or stops. These callbacks are registered in the PE file's TLS directory. Critically, they run before main() or DllMain() is called — they run during the very early loading phase.

Why this defeats most analystsWhen you open a binary in x64dbg and press Run, x64dbg by default pauses at the entry point — the first instruction of main() or WinMain(). TLS callbacks execute before that point. An analyst who doesn't know to look for TLS callbacks will have already missed them.

How malware uses thisThe anti-debug check (or even the full payload decryption) happens in a TLS callback. By the time the analyst is at main(), the malware has already detected the debugger and set a flag. main() then checks the flag and either runs the decoy behaviour or exits.

// This function runs before main() — the analyst hasn't broken yet
void NTAPI TlsCallback(PVOID hModule, DWORD Reason, PVOID Reserved) {
    if (Reason == DLL_PROCESS_ATTACH) {
        // Run anti-debug checks here — analyst is not at a breakpoint yet
        if (IsDebuggerPresent()) {
            g_debugger_detected = TRUE;  // set a flag
        }
        // Or: decrypt and run the real payload here, put a fake decoy in main()
        DecryptRealPayload();
    }
}

// In main() — analyst IS here but it's too late:
int main() {
    if (g_debugger_detected) {
        ShowFakeErrorAndExit();  // decoy behaviour under analysis
    }
    // Real malicious code was already in TlsCallback
}

BypassIn x64dbg: Options → Preferences → Events → TLS Callbacks — check "Break on TLS Callbacks." This makes x64dbg pause at the start of each TLS callback before it runs.

#Windows Anti-Debug — Environment Fingerprinting

These checks look at the surrounding environment rather than debug-specific kernel state. They check: who launched me? What other programs are running? Does my parent look like a debugger?

#Parent Process Check

What a parent process isWhen Windows starts a process, it records which process created it — the "parent PID." Normally, when a user double-clicks malware, the parent is explorer.exe. When malware is opened from inside a debugger, the parent is the debugger itself (e.g., x64dbg.exe).

// Helper: find the parent PID of a given PID using a process snapshot
DWORD GetParentPid(DWORD pid) {
    // CreateToolhelp32Snapshot takes a snapshot of all running processes
    HANDLE hSnapshot = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
    PROCESSENTRY32 pe = { sizeof(pe) };
    Process32First(hSnapshot, &pe);
    do {
        if (pe.th32ProcessID == pid)
            return pe.th32ParentProcessID;  // found it
    } while (Process32Next(hSnapshot, &pe));
    CloseHandle(hSnapshot);
    return 0;
}

DWORD myPid    = GetCurrentProcessId();
DWORD parentPid = GetParentPid(myPid);

// Get the executable name of the parent
HANDLE hParent = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, parentPid);
char parentName[MAX_PATH] = {};
GetModuleFileNameExA(hParent, NULL, parentName, MAX_PATH);

// Is the parent a known debugger?
if (strstr(parentName, "x64dbg") ||
    strstr(parentName, "ollydbg") ||
    strstr(parentName, "windbg") ||
    strstr(parentName, "ida")) {
    exit(1);
}

BypassRename the debugger executable (e.g., rename x64dbg.exe to notepad2.exe). Or use a debugger that isn't on the hardcoded list.

#Window and Process Name Enumeration

Malware scans every window title and every running process name looking for known analysis tools. More thorough than parent check — catches tools running anywhere on the system, not just the direct parent.

// Check window class names of running windows (each GUI app registers a class name)
// x64dbg registers "Qt5QWindowIcon" (Qt framework), OllyDbg registers "OLLYDBG"
if (FindWindowA("OLLYDBG", NULL) ||
    FindWindowA("WinDbgFrameClass", NULL) ||
    FindWindowA("Qt5QWindowIcon", NULL)) {
    exit(1);
}

// Enumerate ALL running processes and check names
HANDLE hSnap = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0);
PROCESSENTRY32 pe = { sizeof(pe) };
const char* analysis_tools[] = {
    "ollydbg.exe", "x64dbg.exe", "x32dbg.exe", "windbg.exe",
    "ida.exe", "ida64.exe", "idaq.exe", "idaq64.exe",
    "procmon.exe", "procmon64.exe",    // Process Monitor (Sysinternals)
    "wireshark.exe",                   // network analysis
    "processhacker.exe",               // Process Hacker
    "pe-sieve64.exe",                  // memory scanner
    "pestudio.exe",                    // static analysis tool
    NULL
};
Process32First(hSnap, &pe);
do {
    for (int i = 0; analysis_tools[i]; i++) {
        if (_stricmp(pe.szExeFile, analysis_tools[i]) == 0) {
            exit(1);
        }
    }
} while (Process32Next(hSnap, &pe));

BypassRename tool executables. Use less-common tools not on the list (e.g., rz-debug from the rizin project). ScyllaHide can hide the debugger process from CreateToolhelp32Snapshot results.

#Windows Anti-Debug — Self-Modifying Code

What it isMalware XOR-encrypts parts of its own code and stores them encrypted in the binary. At runtime, it decrypts the code into executable memory and runs it. After execution, it re-encrypts.

Why it defeats analysis

1. Static analysis (strings, disassembly, YARA rules) sees only the encrypted bytes — the real code is never stored in plaintext on disk.
2. Software breakpoints written at addresses inside the encrypted region are overwritten by the decryption loop — the analyst's breakpoints disappear.
3. A memory dump taken while the code is in its encrypted state shows nothing useful.

extern unsigned char encrypted_function[256];  // encrypted code stored in .data section

void run_hidden_function() {
    DWORD old_protect;

    // Step 1: make the code region writable and executable
    VirtualProtect(encrypted_function, 256, PAGE_EXECUTE_READWRITE, &old_protect);

    // Step 2: decrypt in place (simple XOR — real malware uses AES or RC4)
    for (int i = 0; i < 256; i++) {
        encrypted_function[i] ^= 0x5A;
    }
    // Any 0xCC breakpoint bytes the analyst placed are now garbage — key XOR'd

    // Step 3: execute the now-plaintext code
    ((void(*)())encrypted_function)();

    // Step 4: re-encrypt — next memory scan sees only ciphertext
    for (int i = 0; i < 256; i++) {
        encrypted_function[i] ^= 0x5A;
    }

    // Restore original page permissions
    VirtualProtect(encrypted_function, 256, old_protect, &old_protect);
}

BypassSet a memory access breakpoint on the region (right-click → Breakpoint → Memory, Write access in x64dbg). This fires when the decryption loop writes to the region, before any code runs. Alternatively, break on VirtualProtect calls — the call before the XOR loop tells you where the code will be.

#Linux Anti-Debug Techniques

Linux uses different system calls and data structures, but the same principles apply: look for side effects that the OS leaves when a debugger is present.

#ptrace — The Fundamental Limit

What ptrace isOn Linux, ptrace is the system call that makes debugging possible. strace, gdb, ltrace — they all work by calling ptrace(PTRACE_ATTACH, pid) to attach to a process. Once attached, the tracer can read/write the process's memory, registers, and receive signals and system call events.

The critical constraintOnly one tracer can be attached to a process at a time. This is a hard OS limit. If process A is already tracing process B, no other process can attach to B.

The anti-debug trickA process can trace itself by calling ptrace(PTRACE_TRACEME, 0, 0, 0). This call returns 0 (success) if the process was not previously being traced, and -1 (error, errno = EPERM) if it was already being traced. By calling this at startup:

• If the call succeeds: the process has claimed the "tracer slot." Any subsequent gdb attach will fail with "Operation not permitted."
• If the call fails (returns -1): something else is already tracing us → a debugger is attached.

#include <sys/ptrace.h>
#include <errno.h>
#include <stdlib.h>

void anti_debug_check() {
    if (ptrace(PTRACE_TRACEME, 0, 0, 0) == -1) {
        // errno will be EPERM: "operation not permitted"
        // This means something else is already tracing us → debugger present
        exit(1);
    }
    // We successfully claimed the tracer slot
    // Now gdb cannot attach to us
}

BypassIn GDB, when you start a program, it calls ptrace(PTRACE_ATTACH) first before the program runs. But when using gdb ./binary (spawning rather than attaching), GDB calls ptrace(PTRACE_TRACEME) itself from the child process before exec. So the malware's call to PTRACE_TRACEME would fail because GDB already claimed it.

The bypass: intercept the ptrace syscall and fake its return value to 0:

(gdb) catch syscall ptrace
(gdb) commands
> set $rax = 0    # override the return value to "success" every time ptrace is called
> continue
> end
(gdb) run

Alternatively, patch the binary to replace the ptrace call with xor eax, eax (make it always return 0).

#/proc/self/status — Reading the TracerPid Field

What /proc isOn Linux, /proc is a special virtual filesystem maintained by the kernel. It doesn't exist on disk — the kernel generates its contents on-the-fly when you read it. Each running process has a directory at /proc/<pid>/ containing information about that process. For the current process it's at /proc/self/.

The TracerPid field/proc/self/status is a text file with dozens of fields about the current process. One of them is TracerPid. Under normal conditions it reads TracerPid: 0 — no tracer. When gdb (or strace) is attached, it shows the debugger's PID, like TracerPid: 1234.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int is_being_debugged() {
    FILE* f = fopen("/proc/self/status", "r");
    if (!f) return 0;  // can't read? assume clean

    char line[256];
    while (fgets(line, sizeof(line), f)) {
        if (strncmp(line, "TracerPid:", 10) == 0) {
            int tracer_pid = atoi(line + 10);  // parse the number after "TracerPid:"
            fclose(f);
            return tracer_pid != 0;  // non-zero = being traced
        }
    }
    fclose(f);
    return 0;
}

if (is_being_debugged()) {
    exit(1);
}

BypassUse LD_PRELOAD to intercept the fopen call and return a fake version of /proc/self/status with TracerPid: 0. Or patch the binary to remove this check. Or read the real file differently — some malware reads it via open/read syscalls directly, bypassing libc hooks.

#SIGTRAP — Signal Interception

What signals areLinux signals are asynchronous notifications sent to a process (like SIGTERM to terminate, SIGKILL to force-kill, SIGSEGV on segfault). A process can register a signal handler — a function to call when a specific signal arrives.

What SIGTRAP isSIGTRAP (signal 5) is sent to a process when a INT 3 breakpoint fires or when single-stepping. GDB's default behaviour is to consume SIGTRAP — it receives it and shows you the breakpoint in the UI, but never forwards it to the process's own signal handler.

The trickMalware registers its own SIGTRAP handler, then deliberately raises SIGTRAP. It then checks whether its handler ran:

• No debugger: OS delivers SIGTRAP to the process's own signal handler → handler runs → counter increments.
• Debugger present: GDB intercepts SIGTRAP before it reaches the process's signal handler → handler never runs → counter stays at 0.

#include <signal.h>

volatile int sigtrap_count = 0;

void our_sigtrap_handler(int sig) {
    sigtrap_count++;  // this runs only if we received the signal ourselves
}

void check_for_debugger() {
    signal(SIGTRAP, our_sigtrap_handler);  // register our handler
    raise(SIGTRAP);                         // deliberately send ourselves SIGTRAP

    // sigtrap_count should now be 1 if no debugger intercepted it
    if (sigtrap_count == 0) {
        // Our handler never ran — the debugger consumed the signal
        exit(1);
    }
}

BypassTell GDB to forward SIGTRAP to the process:

(gdb) handle SIGTRAP nostop noprint pass

nostop = don't pause execution, noprint = don't print a message, pass = forward the signal to the process's handler.

#/proc/self/maps — Debugger Library Footprint

What /proc/self/maps isA text file listing every memory region mapped in the current process, with permissions and the file it came from (if any). When GDB attaches, it loads helper libraries into the process's memory — specifically libthread_db.so (for multi-threaded debugging) and sometimes Python libraries (if GDB is built with Python scripting support).

FILE* maps = fopen("/proc/self/maps", "r");
char line[512];
while (fgets(line, sizeof(line), maps)) {
    // GDB's helper libraries leave these traces in /proc/self/maps
    if (strstr(line, "libthread_db") ||    // thread debugging library (loaded by gdb)
        strstr(line, "libpython")) {        // gdb's python scripting engine
        fclose(maps);
        exit(1);
    }
}
fclose(maps);

BypassUse a minimal GDB build without Python support. Or intercept /proc/self/maps reads with LD_PRELOAD to filter out the suspicious lines.

#macOS Anti-Debug Techniques

macOS is Unix-based but has a different security architecture. Apple has added macOS-specific features to ptrace, and the kernel maintains additional process state.

#ptrace(PT_DENY_ATTACH) — Permanent Shield

What it doesmacOS extends the ptrace system call with a non-standard flag: PT_DENY_ATTACH. A process can call this on itself. From that point forward, any attempt by a debugger to attach to the process will cause the process to be sent SIGKILL — it dies rather than being debugged.

This is one-way and permanent for the lifetime of the process. Once called, there is no way for a debugger to attach.

#include <sys/ptrace.h>

// Call early in startup — after this, debuggers cannot attach
ptrace(PT_DENY_ATTACH, 0, 0, 0);

How PT_DENY_ATTACH works under the hood: When a debugger calls ptrace(PTRACE_ATTACH, pid), the kernel checks the target process's P_LNOATTACH flag. If it's set (set by PT_DENY_ATTACH), the kernel sends SIGKILL to the target process and returns an error to the debugger.

Bypass options

• Patch the call site in the binary before running it — find the PT_DENY_ATTACH call (the constant value is 31 = 0x1F) and replace it with NOPs
• Attach with LLDB before the PT_DENY_ATTACH call executes (at the very first instruction)
• Use a kernel debugger (SIP-disabled machine with kernel debug enabled)

#sysctl — Checking the P_TRACED Flag

What sysctl isA system call that reads or writes kernel parameters. On macOS, it can also query information about running processes, including whether they are being traced.

What P_TRACED isA flag in the kernel's process structure (proc) that is set when the process is under ptrace. The sysctl call with the right parameters returns a kinfo_proc structure containing kp_proc.p_flag, and bit P_TRACED (0x00000800) is set when a debugger is attached.

#include <sys/sysctl.h>
#include <sys/types.h>

int is_being_debugged_macos() {
    struct kinfo_proc info = {};
    size_t size = sizeof(info);

    // Build the "selector" for "give me info about process with my PID"
    int mib[4] = {CTL_KERN, KERN_PROC, KERN_PROC_PID, getpid()};

    sysctl(mib, 4, &info, &size, NULL, 0);

    // P_TRACED flag = 0x00000800
    return (info.kp_proc.p_flag & P_TRACED) != 0;
}

if (is_being_debugged_macos()) {
    exit(1);
}

BypassPatch the binary to change the conditional check (flip jne to je). Or use LLDB's process handle commands to intercept the sysctl call and return clean data.

#Anti-Analysis: Code Obfuscation

The techniques above detect a live debugger. These techniques make static analysis harder — even if there's no debugger, the analyst can't read the code.

#String Obfuscation — Hiding IOCs

The problem with plain stringsWhen an analyst runs strings on a binary, every hardcoded URL, file path, registry key, and error message appears immediately. This is the fastest way to understand what malware does.

The fixXOR-encrypt every interesting string and only decrypt it in memory at the moment it's needed. The plaintext never exists in the binary on disk.

// The C2 URL stored XOR-encrypted in the binary
// The analyst running 'strings' sees: \x2b\x2e\x20\x23... (garbage)
static const unsigned char enc_url[] = {
    0x2b, 0x2e, 0x20, 0x23, 0x27, 0x71, 0x2f, 0x2f   // "https://" XOR 0x41
};
static const unsigned char KEY = 0x41;

char* get_c2_url() {
    size_t len = sizeof(enc_url);
    char* plaintext = malloc(len + 1);
    for (size_t i = 0; i < len; i++) {
        plaintext[i] = enc_url[i] ^ KEY;   // XOR each byte with the key
    }
    plaintext[len] = '\0';
    return plaintext;  // caller gets "https://" in plaintext
}

// Usage:
char* url = get_c2_url();
// ... make HTTP request to url ...
memset(url, 0, strlen(url));  // zero the plaintext after use so it's not in memory
free(url);

Analyst bypass — brute-force the key

# Try all 256 possible single-byte XOR keys
ciphertext = bytes([0x2b, 0x2e, 0x20, 0x23, 0x27, 0x71, 0x2f, 0x2f])

for key in range(256):
    plaintext = bytes(b ^ key for b in ciphertext)
    # Check if it looks like real text (URLs start with "http", paths start with "C:\", etc.)
    if b'http' in plaintext or b'cmd' in plaintext.lower() or b'C:\\' in plaintext:
        print(f"Key 0x{key:02x}: {plaintext}")

More sophisticated malware uses multi-byte keys, AES, or RC4 — these require dynamic analysis (running the binary and capturing the decrypted strings at runtime using Frida or GDB).

#API Hashing — Hiding Function Calls

The problem with normal importsWhen a Windows binary calls VirtualAllocEx or CreateRemoteThread, those function names appear in the binary's Import Address Table (IAT) — a list the OS loader uses to resolve external function addresses at startup. An analyst can see the complete list of Windows APIs the binary uses in under a second with dumpbin /imports or readelf -d.

API hashing resolves function addresses at runtime by walking the PE export table of loaded DLLs and comparing a hash of each function name against a target hash. No function names appear in the binary's import table — only the hash values (numbers).

// Step 1: compute a hash of a function name at build time
// djb2 variant:
DWORD hash_function_name(const char* name) {
    DWORD hash = 0x811C9DC5;  // initial value (FNV offset basis)
    while (*name) {
        hash ^= (unsigned char)*name++;   // XOR with the current character
        hash *= 0x01000193;               // multiply by FNV prime
    }
    return hash;
}
// hash_function_name("VirtualAllocEx") = 0xE553A458 (example)
// hash_function_name("CreateRemoteThread") = 0xC2BFCE8A (example)

// Step 2: at runtime, walk ntdll's export table looking for a matching hash
PVOID find_function(HMODULE hModule, DWORD target_hash) {
    // Parse the PE header to find the export directory
    PIMAGE_DOS_HEADER dos = (PIMAGE_DOS_HEADER)hModule;
    PIMAGE_NT_HEADERS nt  = (PIMAGE_NT_HEADERS)((BYTE*)hModule + dos->e_lfanew);
    DWORD exp_rva = nt->OptionalHeader.DataDirectory[0].VirtualAddress;
    PIMAGE_EXPORT_DIRECTORY exp = (PIMAGE_EXPORT_DIRECTORY)((BYTE*)hModule + exp_rva);

    DWORD* names    = (DWORD*)((BYTE*)hModule + exp->AddressOfNames);
    WORD*  ordinals = (WORD*) ((BYTE*)hModule + exp->AddressOfNameOrdinals);
    DWORD* funcs    = (DWORD*)((BYTE*)hModule + exp->AddressOfFunctions);

    for (DWORD i = 0; i < exp->NumberOfNames; i++) {
        char* name = (char*)((BYTE*)hModule + names[i]);
        if (hash_function_name(name) == target_hash) {
            return (PVOID)((BYTE*)hModule + funcs[ordinals[i]]);
        }
    }
    return NULL;
}

// Step 3: call functions using their resolved addresses
// The binary only stores hash values — no function names in the import table
typedef LPVOID (WINAPI* pVirtualAllocEx)(HANDLE, LPVOID, SIZE_T, DWORD, DWORD);
pVirtualAllocEx fn = (pVirtualAllocEx)find_function(GetModuleHandleA("kernel32.dll"), 0xE553A458);
fn(target_process, NULL, shellcode_size, MEM_COMMIT, PAGE_EXECUTE_READWRITE);

Analyst bypassSet a breakpoint on the find_function routine (identify it by looking for the PEB walk pattern — GS:[0x60] or the loop over export names). Every call to it reveals what function is being resolved. Or use Frida to intercept all function calls regardless of how they were resolved.

#Anti-Analysis: Packing and Encryption

What a packer isA program that takes a PE executable, compresses or encrypts it, and wraps it in a small "stub" program. When run, the stub decompresses/decrypts the real payload into memory and transfers execution to it. The binary on disk is unreadable; the real code only exists in RAM while running.

On disk (packed):
  [Packer stub code]           ← small, mostly clean-looking
  [Encrypted/compressed payload] ← high entropy, looks like random data

In RAM (after stub runs):
  [Original PE headers]
  [Original .text section]     ← real malicious code
  [Original .data section]     ← real strings, config
  [Reconstructed import table]

How to detect a packed binary before running it:

# 1. High entropy = encrypted/compressed data
# Normal code: entropy ~4-5 bits/byte (not all byte values equally likely)
# Encrypted data: entropy ~7.9 bits/byte (very close to random)
python3 -c "
import sys, math, collections
data = open(sys.argv[1], 'rb').read()
freq = collections.Counter(data)
entropy = -sum((c/len(data)) * math.log2(c/len(data)) for c in freq.values())
print(f'Entropy: {entropy:.2f} / 8.0')
" malware.exe

# 2. UPX (most common open-source packer) leaves recognisable strings
strings malware.exe | grep UPX
# UPX0, UPX1, UPX! = UPX packed

# 3. Almost no import table entries (packed binaries resolve APIs themselves)
dumpbin /imports malware.exe     # Windows
objdump -T malware.elf           # Linux
# If imports are just: ExitProcess, LoadLibraryA, GetProcAddress → probably packed

Unpacking approach (finding the Original Entry Point, or OEP):

1. Run the packed binary in a debugger
2. Set a breakpoint on VirtualAlloc (or VirtualProtect) — the stub calls this to create executable memory for the unpacked payload
3. After the call, note the memory address that was allocated
4. Set a hardware execute breakpoint at that address
5. Resume execution — the stub fills the allocation with the real code, then jumps to it
6. When your hardware breakpoint fires, you're at the OEP — the real payload
7. Dump the memory region using Scylla (x64dbg plugin) → you now have the unpacked binary

#Anti-Analysis: Anti-Disassembly

What disassembly isConverting raw machine code bytes back into human-readable assembly instructions (like mov eax, 1 or call 0x401234). Tools like IDA Pro, Ghidra, and objdump do this. Anti-disassembly tricks insert sequences of bytes that confuse the disassembler into decoding them incorrectly.

#Junk Byte Insertion

How disassemblers workMost disassemblers use "linear sweep" — they start at the first byte and decode one instruction at a time, moving forward by the instruction's length. x86 instructions are variable length (1–15 bytes). If the disassembler gets the start offset wrong, it decodes garbage.

The trickInsert a byte that will never actually be executed (because a jump skips over it), but that the disassembler interprets as the start of a long instruction, consuming the next several bytes:

; What the CPU executes:
jmp  short real_code    ; 2-byte instruction — always taken
db   0xE8               ; junk byte — never executed by CPU

real_code:
; real instructions here

; What a linear-sweep disassembler sees:
; jmp → then tries to decode 0xE8 as the start of a CALL instruction
; CALL takes a 4-byte relative offset → consumes 5 bytes total
; The disassembler now thinks instructions start at the wrong offsets
; Everything after this point is decoded as garbage

BypassIDA Pro's default disassembler uses recursive traversal — it follows code flow (analyses both branches of a conditional jump) rather than reading linearly. It won't try to decode the unreachable junk byte. If IDA gets confused, manually define the code at the correct offset: press C at the real instruction's address to force disassembly there.

#Opaque Predicates — Fake Conditional Jumps

What an opaque predicate isA conditional jump whose outcome is always the same (always taken or never taken), but which looks data-dependent to a static analyser. The disassembler tries to analyse both branches; the "never taken" branch may contain decoy code with fake IOCs to mislead analysts.

// x * x is ALWAYS >= 0 for integer arithmetic on all platforms
// A static analyser cannot always prove this algebraically
// → IDA/Ghidra may disassemble both the "true" and "false" branches
int x = some_runtime_value();   // looks like it could be anything
if ((x * x) >= 0) {             // always true
    run_real_code();
} else {
    fake_decoy_code();          // never runs, but analyst spends time on it
    // might have fake C2 IPs, fake registry keys — red herrings
}

BypassDynamic analysis — run the code and observe which branch is actually taken. A debugger will show you the real execution path. In IDA, after running with the debugger, the "never-taken" branch gets greyed out in the graph view.

#Analyst Bypass Reference

ScyllaHide is an x64dbg / OllyDbg plugin that patches the most common user-mode anti-debug checks automatically. Install it and enable all options — it handles roughly half of everything in this table without any manual work.