After working through the conceptual side of malware analysis — classification, the four analysis types, lab isolation principles — I wanted to map out what an actual end-to-end analysis of a real specimen looks like in practice, tool by tool.
This post walks through the full investigative chain for a sample called brbbot.exe: starting from static properties, moving into live behavioral observation, intercepting its network traffic, decrypting its configuration file with a debugger, and finally interacting with its command-and-control (C2) logic directly. It's structured as a practical methodology guide — the kind of thing I wish I'd had as a map before diving into the tools individually.
Problem Statement
Most malware analysis guides cover one tool in isolation — "here's how Wireshark works," "here's how x64dbg works" — without showing how those tools chain together into a single investigation. A real analysis isn't "use tool X," it's "use static analysis to form a hypothesis, use behavioral analysis to test it, use network interception to see what the hypothesis predicted, and use a debugger when you need ground truth on exactly what the code is doing."
This walkthrough is my attempt to lay out that chain clearly, end to end, against one consistent specimen.
Lab Setup (Prerequisite)
Before any of this is possible, you need two isolated VMs talking to each other on a host-only network:
REMnux VM → static IP, e.g. 192.168.56.10
Windows VM → static IP, e.g. 192.168.56.20
Both VMs need a clean snapshot taken immediately after setup — this is the rollback point you return to before and after every infection.
# On REMnux, confirm connectivity from Windows side first
ping 192.168.56.10
Step-by-Step Walkthrough
Phase 1: Static Properties Analysis
Goal: Form an initial hypothesis about the specimen without ever executing it.
On REMnux, extract embedded strings:
pestr -n 5 -o brbbot.exe
On the Windows VM, cross-check using BinText (string extraction) and PeStudio (header/import/export inspection). The combination of both tools matters — PeStudio surfaces anomalous header characteristics (suspicious imports, packer signatures, unusual section names) that raw strings won't flag on their own.
On REMnux, run peframe as a second opinion:
peframe brbbot.exe
What you're looking for at this stage: registry paths, hardcoded URLs or domains, suspicious API names in the import table (CryptDecrypt, RegSetValueEx, InternetOpen), and anything that hints at persistence or network behavior before you've run a single instruction.
Phase 2: Initial Behavioral Analysis
Goal: Observe what the specimen actually does on a live system, in real time.
Set up four monitoring tools before triggering the infection:
Process Hacker → running process tree, before/after comparison
Process Monitor → file system / registry / process activity log
Regshot → registry diff (before/after snapshots)
Wireshark → network capture (on REMnux, monitoring the link)
On REMnux:
wireshark &
# Activate capture (Ctrl+E) once the window opens
On the Windows VM: pause Process Monitor (Ctrl+E), clear its log (Ctrl+X), take the first Regshot snapshot, then activate Process Monitor capture and double-click the brbbot.exe shortcut to infect the system.
Let it run briefly, then terminate it via Process Hacker, stop all captures, and take the second Regshot snapshot to generate a diff report.
Regshot diff → reveals new files, new/modified registry keys
Process Monitor log → exportable as CSV, then visualized in ProcDOT
ProcDOT is the piece that ties Process Monitor's raw log into something readable — it builds a process/file/registry interaction diagram, with brbbot.exe flagged as the "first relevant process" so the graph centers on it.
Phase 3: Intercepting Network Traffic
Goal: Capture and decode the specimen's outbound communication, including its encoded payload.
This phase depends on controlling DNS resolution and serving fake responses so the specimen "thinks" it's talking to its real C2 infrastructure.
On REMnux, launch a fake DNS resolver:
fakedns
Verify it from the Windows side:
nslookup example.com
:: Should resolve to the REMnux VM's IP, not a real address
Reinfect the Windows VM with Wireshark capturing. At this stage, with no real web server running, you'll see the DNS resolution succeed but the subsequent HTTP request fail or hang — confirming the specimen is trying to reach out, but has nothing to talk to yet.
Then bring up a basic web server on REMnux:
httpd start
Reinfect again. This time the HTTP session should establish. In Wireshark, right-click a packet from that session and use Follow TCP Stream to read the full payload.
The exfiltrated data appears appended after a &p= parameter as hex-encoded bytes. Copy it out and save it for the next phase:
scite encoded.hex
# Paste clipboard contents, save, exit
cat encoded.hex # sanity check
Phase 4: Decrypting the Configuration File
Goal: Move from observed behavior to ground-truth understanding by debugging the binary directly.
This is where code analysis enters the picture. On the Windows VM, load the binary into x64dbg without running it yet, and set a breakpoint on the Windows API call responsible for file reads:
SetBPX ReadFile
Run the specimen inside the debugger (F9). It will pause the moment it calls ReadFile. At that point, inspecting the register holding the file handle (typically RCX on x64) and cross-referencing it against the debugger's Handles tab confirms the specimen is reading its own dropped configuration file — brbconfig.tmp.
Continue execution past the read (Alt+F9), then look for the call to CryptDecrypt further down. Set a breakpoint after that call (on the following test eax,eax instruction) and run to it (F4). At this point, the decrypted plaintext should be sitting on the stack — readable directly in the debugger's stack view.
Decrypted config (example pattern): uri=ads.php...
Separately, on REMnux, decode the network payload captured in Phase 3. The encoding here is a simple single-byte XOR — not strong cryptography, just enough to defeat naive string-based detection:
xxd -r -p encoded.hex > encoded.raw
translate.py encoded.raw decoded.txt 'byte ^ 0x5b'
cat decoded.txt
Phase 5: Experimenting with C2 Functionality
Goal: Confirm the specimen actually responds to commands from its controller, not just that it phones home.
With the web server still running on REMnux, craft a response file that issues a command:
cd /var/www
echo "cexe c:\windows\notepad.exe" > ads.php
Reinfect the Windows VM and observe in Process Hacker — within roughly 30 seconds, brbbot.exe should spawn a notepad.exe child process repeatedly, confirming the cexe (command-execute) instruction is being parsed and acted on.
To confirm the kill-switch command as well:
rm ads.php
echo tixe > ads.php
Within 30 seconds, the running brbbot.exe process should terminate on its own — validating that the malware author built in a remote self-destruct mechanism, presumably to limit forensic exposure once a campaign is detected.
httpd stop
How to Verify
- [ ] Static analysis surfaced at least one suspicious string/import before execution
- [ ] Regshot/ProcDOT confirmed a concrete artifact (registry key or dropped file)
- [ ] Wireshark captured a DNS lookup and a full HTTP session with the encoded payload
- [ ] The config file's decrypted plaintext was visible directly in the debugger
- [ ] The XOR-decoded exfiltration payload matched expectations
- [ ] The
cexecommand produced an observable child process;tixeproduced termination
What I Learned (Planning This Out)
Mapping this before running it clarified something: every phase exists to validate the previous phase's hypothesis. Static analysis says "this might talk to a C2 server." Behavioral analysis says "yes, and here's the registry key it leaves behind." Network interception says "and here's exactly what it sends." Code analysis says "and here's the exact decryption routine it uses." C2 experimentation says "and here's proof the channel is bidirectional, not just exfiltration."
It also reframed how I think about debugger breakpoints — SetBPX ReadFile isn't really about the API call itself, it's a tripwire placed at the boundary between "the malware's internal logic" and "the OS doing something observable." That's a pattern that generalizes well beyond this one specimen.
Common Mistakes
| Mistake | Why It's a Problem | Better Approach |
|---|---|---|
| Skipping the fakedns/web server setup before reinfecting | The specimen's network calls silently fail, masking real C2 behavior | Always verify fakedns and the web server respond correctly before infecting |
| Forgetting to clear Process Monitor's log before infection | Pre-existing noise drowns out the actual infection-related events | Pause and clear the log (Ctrl+X) immediately before triggering infection |
| Setting the ReadFile breakpoint without checking the handle | You might be watching the wrong file read entirely | Always cross-reference the handle value against the Handles tab |
| Treating the XOR key as a one-off curiosity | Single-byte XOR is extremely common in malware obfuscation — recognizing the pattern speeds up future analysis | Note the technique (byte-wise XOR with a static key) as a reusable indicator |
| Not reverting to a clean snapshot between exercise phases | Leftover artifacts from a previous run can produce misleading "new" findings in Regshot/ProcDOT | Revert to the clean snapshot whenever the exercise calls for a pristine state |
Conclusion
This was a full walkthrough of the analysis methodology before I've actually run a single command against the live VMs — and I think that's worth being upfront about. Mapping the entire chain first, tool by tool and phase by phase, means that when I do sit down with the actual lab, I'm executing a plan rather than improvising one.
If you've worked through a similar brbbot.exe-style exercise before, I'd be curious whether the C2 behavior matched what's outlined here, or whether there were surprises along the way.