HTB Challenge: Behind the Scenes
Challenge Description
After struggling to secure our secret strings, we finally came up with a way to make decompilation harder. Our goal was to make it impossible to figure out how the program works!
Steps to Analyze the Binary
Extract the provided zip folder.
Load the binary into Detect-It-Easy.
Identify the binary as an ELF 64-bit executable compiled with GCC (9.3.0).
Confirm that the code is written in C/C++.
Load the binary into Binary Ninja for analysis.
While the solution is relatively easy to find, we will break down the technical aspects to better understand the challenge.
Breaking Down the Challenge
This challenge involved reverse engineering, signal handling, and anti-debugging techniques. Let’s go step by step to analyze the solution.
Step 1: Understanding the Binary
Since we only have the compiled binary (behindthescenes), we used Binary Ninja to reverse engineer it.
Key Findings:
A function named segill_sigaction, acting as a signal handler.
A string in .rodata that hinted at a password check:
./challenge <password>A UD2 instruction in .rodata, which is an illegal instruction that triggers a SIGILL (Illegal Instruction Exception).
Step 2: Identifying Key Functions
By analyzing the binary, we identified two important functions:
1. SIGILL Handler (segill_sigaction)
Registered to handle SIGILL (Illegal Instruction signals).
Modifies the execution context, potentially bypassing crashes.
2. Main Function (main)
Registers segill_sigaction as the SIGILL handler.
Triggers a SIGABRT (trap 6), which would usually crash the program.
Step 3: Finding the Password Check
Within .rodata, we discovered a string that looked like a password:
Itz_0nLy_UD2
Additionally, we found:
HTB{%s}
This suggests that once the correct password is entered, the program will print a flag.
Step 4: Understanding the Role of UD2
UD2 is an x86 instruction that forces a crash by triggering SIGILL.
Normally, an illegal instruction would terminate execution, but since a custom SIGILL handler exists, we suspected:
The program intentionally executes UD2.
The handler modifies execution to prevent a crash.
This serves as an anti-debugging technique or a way to manipulate execution flow.
Step 5: Running the Program with the Password
With an understanding of the execution flow, we tested the suspected password:
./behindthescenes Itz_0nLy_UD2
It worked! The program printed the HTB flag, confirming that our password was correct.
Key Takeaways
Concept
Explanation
.rodata Section
Stores strings used in the program, including the password.
Signal Handling
The program uses sigaction(4, &handler, NULL) to catch SIGILL.
UD2 Instruction
Triggers a SIGILL exception, which the handler catches.
Execution Hijacking
The signal handler modifies program execution to prevent crashing.
Reverse Engineering
Instead of brute-forcing, we extracted the password from .rodata.
Final Thoughts
The program hides the password behind an anti-debugging trick (SIGILL).
Understanding signal handling helped us recognize the role of UD2.
Instead of guessing, we extracted the password from .rodata.
This challenge was a great mix of reverse engineering, anti-debugging, and execution manipulation.
Technical Analysis: Key Insights from the Disassembled Binary
This section provides a breakdown of the most relevant parts of the objdump -d output, focusing on the execution flow, anti-debugging techniques, password validation, and signal handling mechanisms.
1. SIGILL Handler (segill_sigaction)
The segill_sigaction function is responsible for handling SIGILL (Illegal Instruction) signals. Normally, an illegal instruction like UD2 would cause the program to crash, but here, a custom handler modifies execution instead.
0000000000001229 <segill_sigaction>:
1229: f3 0f 1e fa endbr64
122d: 55 push %rbp
122e: 48 89 e5 mov %rsp,%rbp
1231: 89 7d ec mov %edi,-0x14(%rbp)
1234: 48 89 75 e0 mov %rsi,-0x20(%rbp)
1238: 48 89 55 d8 mov %rdx,-0x28(%rbp)
124c: 48 8b 80 a8 00 00 00 mov 0xa8(%rax),%rax
124f: 48 8d 50 02 lea 0x2(%rax),%rdx
1257: 48 89 90 a8 00 00 00 mov %rdx,0xa8(%rax)
1260: c3 ret
What This Does:
Registers segill_sigaction as the SIGILL handler.
Retrieves the execution context and modifies it to bypass crashes.
Likely an anti-debugging mechanism, forcing an illegal instruction (UD2) and catching it to alter execution.
2. Registering the SIGILL Handler in main
The main function registers segill_sigaction to handle SIGILL signals.
00000000000012a5 <main>:
12a5: 48 8d 85 60 ff ff ff lea -0xa0(%rbp),%rax
12ac: 48 83 c0 08 add $0x8,%rax
12b3: e8 78 fe ff ff call 1130 <sigemptyset@plt>
12b8: 48 8d 05 6a ff ff ff lea -0x96(%rip),%rax # 1229 <segill_sigaction>
12bf: 48 89 85 60 ff ff ff mov %rax,-0xa0(%rbp)
12c6: c7 45 e8 04 00 00 00 movl $0x4,-0x18(%rbp) # Signal 4 (SIGILL)
12e1: e8 fa fd ff ff call 10e0 <sigaction@plt>
What This Does:
Calls sigaction(4, &handler, NULL), registering segill_sigaction for SIGILL signals.
Ensures that when UD2 (Illegal Instruction) is executed, the program does not crash but instead modifies execution flow.
3. Anti-Debugging via UD2
The UD2 instruction is an illegal x86 instruction that forces a SIGILL exception, commonly used as an anti-debugging measure.
12e6: 0f 0b ud2
12f1: 0f 0b ud2
130b: 0f 0b ud2
What This Does:
UD2 is executed multiple times throughout the code.
Normally, this would terminate the program.
Since segill_sigaction is registered as the SIGILL handler, execution continues instead of crashing.
4. Password Validation Mechanism
The program checks if the input matches "Itz_0nLy_UD2". This is done in multiple steps.
Checking Argument Count
12e8: 83 bd 5c ff ff ff 02 cmpl $0x2,-0xa4(%rbp) # argc == 2?
12ef: 74 1a je 130b <main+0xaa> # Jump if valid
Ensures the user provides exactly one argument.
Checking Length of Argument
131e: e8 cd fd ff ff call 10f0 <strlen@plt>
1323: 48 83 f8 0c cmp $0xc,%rax # Password length must be 12
1327: 0f 85 05 01 00 00 jne 1432 <main+0x1d1>
Ensures the password length is exactly 12 characters.
Comparing the Password String
The program checks if the input argument matches "Itz_0nLy_UD2", piece by piece.
1342: 48 8d 35 d2 0c 00 00 lea 0xcd2(%rip),%rsi # Load "Itz"
134c: e8 6f fd ff ff call 10c0 <strncmp@plt> # strncmp(argv[1], "Itz", 3)
1372: 48 8d 35 a6 0c 00 00 lea 0xca6(%rip),%rsi # Load "_0n"
13a2: 48 8d 35 7a 0c 00 00 lea 0xc7a(%rip),%rsi # Load "Ly_"
13ce: 48 8d 35 52 0c 00 00 lea 0xc52(%rip),%rsi # Load "UD2"
Compares each three-character segment separately.
5. Printing the Flag
If the password is correct, the program prints the flag in the format:
HTB{Itz_0nLy_UD2}
13f4: 48 8d 3d 30 0c 00 00 lea 0xc30(%rip),%rdi # Load format string "> HTB{%s}\n"
1400: e8 0b fd ff ff call 1110 <printf@plt> # Print the flag
What This Does:
Loads the flag format string "> HTB{%s}\n" into %rdi.
Calls printf, substituting the user’s password into the flag format.
Conclusion & Key Takeaways
Concept
Explanation
Anti-Debugging (UD2)
The UD2 instruction triggers SIGILL, but a custom handler prevents the crash.
Signal Handling
sigaction(4, &handler, NULL) catches SIGILL, allowing the program to continue execution.
Password Extraction
Instead of brute-forcing, the password "Itz_0nLy_UD2" was extracted from .rodata.
Reverse Engineering
By analyzing objdump, we reconstructed how the binary works without executing it.
Final Thoughts
The program disguises a simple password check behind an anti-debugging trick.
By modifying execution via SIGILL handling, the program prevents straightforward analysis.
However, analyzing .rodata and objdump allowed us to recover the password without brute force.
This was a great example of how signal handling, execution hijacking, and anti-debugging tricks can be used in real-world reverse engineering challenges!