blog@XeroExecute.github.io ~ $

SwampCTF: You Shall Not Passss

tags
#Re
published
reading time
6 minutes

This past weekend, I participated in SwampCTF with the Dombusters team. One of the reverse engineering challenges I solved, “You Shall Not Passss”, was interesting and, in my opinion, worth documenting for future reference.

Description

To enter in, you’ll need a key,
A secret code just meant for thee.
Type it right, don’t make a slip,
Or you’ll be locked out—oops, that’s it!

Solution

Initial static Analysis

Using a decompiler like Ghidra, the main function and a subroutine sub_17E0 were extracted. Here’s the relevant code:

  • main
main:
__int64 __fastcall main(int a1, char **a2, char **a3) {
  char *decrypted_flag_ptr = (char *)&unk_4040; // Input buffer
  char *key_stream_ptr = &byte_4060;           // Key stream
  int seed = data_Seed;                        // Seed for key generation
  char *key_stream_iter = &byte_4060;

  // First encryption loop (28 bytes)
  do {
    ++decrypted_flag_ptr;
    ++key_stream_iter;
    seed = (330 * seed + 100) % 2303;
    *(decrypted_flag_ptr - 1) ^= seed ^ *(key_stream_iter - 1);
  } while ((char *)&unk_4040 + 28 != decrypted_flag_ptr);

  // Manual unrolling for remaining bytes
  v7 = 330 * seed + 100;
  byte_4030 ^= (v7 % 2303) ^ byte_4060;
  // ... (repeats for byte_4031 to byte_4039, byte_4020 to byte_4029)
  data_Seed = (330 * v25 + 100) % 2303;

  // Second encryption loop (189 bytes, unrelated?)
  v27 = &xmmword_4180;
  do {
    v27 = (__int128 *)((char *)v27 + 1);
    ++key_stream_ptr;
    v26 = (330 * v26 + 100) % 2303;
    *((_BYTE *)v27 - 1) ^= v26 ^ *(key_stream_ptr - 1);
  } while ((__int128 *)((char *)&xmmword_4180 + 189) != v27);
  dword_4124 = v26;

  sub_17E0();
  return 0LL;
}
  • sub_17E0
void sub_17E0() {
  size_t v0 = sysconf(30); // Page size
  __m128i *v1 = (__m128i *)mmap(0LL, v0, 7, 34, -1, 0LL); // Allocate executable memory
  if (v1 == (__m128i *)-1LL) {
    perror("mmap");
  } else {
    v2 = v1;
    *v1 = _mm_load_si128((const __m128i *)&xmmword_4180); // Copy 189 bytes
    // ... (11 more 16-byte loads, partial copy to v1[11])
    ((void (__fastcall *)(void *, char *, char *, void *, void *))v1)(
      &unk_4040, &byte_4030, &byte_4020, &unk_4260, &unk_4140); // Execute as function
    munmap(v2, v0);
  }
}

The challenge involves decrypting an encrypted buffer, which is then loaded into memory and executed as shellcode. Statically analyzing encrypted shellcode is difficult because the code remains obscured until runtime. Therefore, gdb is used to dynamically analyze the shellcode once it is decrypted and loaded into memory, allowing for an easier examination of its functionality.

Debugging with GDB

Debugging poses challenges because for the analysis of the shellcode breakpoints needed to be set on memory addresses that are only available after the program starts executing.
My approach was to execute the binary in gdb, wait for the “Please Enter the Password:” prompt of this challenge, and then manually interrupt execution using ctrl+c.
At this point, breakpoints can be set in the dynamically allocated memory region since the shellcode is now already decrypted an loaded into memory. Therefore the addresses are now available and can be used to set breakpoints on.
In the next step however the addresses to set those breakpoints need to be defined.

  • Disassembly at the Prompt
    The disassembly at the “Please Enter the Password” prompt is as follows:
0x7ffff7fbf031  mov    edx, 0x24
0x7ffff7fbf036  xor    eax, eax
0x7ffff7fbf038  syscall 
→0x7ffff7fbf03a  mov    eax, 0x1
0x7ffff7fbf03f  mov    ecx, 0xc74f08c9
0x7ffff7fbf044  xor    edx, edx
0x7ffff7fbf046  mov    esi, 0x1
0x7ffff7fbf04b  imul   eax, eax, 0x14a
0x7ffff7fbf051  add    eax, 0x64

The syscall is made right before the current instruction pointer (rip). Since the eax register is set to 0, this indicates a read syscall, which is logical for reading a password. The value 0x24 in the edx register specifies the number of bytes to be read, suggesting that the password/flag is 36 bytes long.
The verification of these bytes is still an unknown. Therefore more of the shellcode needs to be disassembled and analyzed. Since there needs to be some verification logic there is likely some comparison and jump instruction intended for that. To see additional instructions after providing input, the disassembly can be single-stepped (ni) after entering a dummy password.
The next notable jump instruction is jne at 0x7ffff7fbf04b.
Looking at the jump instruction closely, it becomes clear that it is part of a verification loop for the input:

0x7ffff7fbf04b  imul   eax, eax, 0x14a      ; Generate key
0x7ffff7fbf051  add    eax, 0x64
0x7ffff7fbf076  sub    eax, edi             ; al = key
0x7ffff7fbf078  movzx  edi, BYTE PTR [r12+rdx*1]  ; Input byte
0x7ffff7fbf07d  xor    dil, al              ; Encrypt
0x7ffff7fbf084  movsx  r8d, BYTE PTR [r15+rdx*1]  ; Expected byte
0x7ffff7fbf089  cmp    dil, r8b             ; Compare
0x7ffff7fbf08c  cmovne esi, r13d            ; Set esi if mismatch
0x7ffff7fbf090  inc    rdx                  ; Next byte
0x7ffff7fbf093  cmp    rdx, 0x24            ; 36 iterations
0x7ffff7fbf097  jne    0x7ffff7fbf04b       ; Loop
0x7ffff7fbf099  test   esi, esi             ; Success if esi = 0

The loop iterates over all 36 bytes and terminates on a wrong password. This enables dumping all characters in a single run without the need for repeated execution.
The input is encrypted and checked against values loaded from the r15 register. Therefore, the next step is to dump the values at the address pointed to by r15.

$r15   : 0x0000555555558140
x/36c 0x555555558140
0xdd 0x9a 0xde 0x4e 0x69 0xe1 0xe9 0x2c 0xd2 0x4e 0xec 0xe7 0x18 0x26 0x6a 0x56
0x79 0xd8 0xa3 0x55 0x72 0xbc 0x76 0xc4 0x0c 0x0f 0x9b 0xbe 0xc6 0x81 0xe2 0x41
0x47 0xa0 0xf4 0x26

With the expected bytes after encryption, the next step is to obtain the encryption key used to encrypt the input. The expected input can be determined based on the expected output and the keystream used.
At the address 0x7ffff7fbf07d the xor operation responsible for the encryption of the input can be seen. The instruction shows that the encryption key byte is located in the al register at that point.
To dump the entire key a breakpoint can be set at that address and then the al register can be read.
Dumping all the al register values in the loop provides the entire XOR keystream needed to decrypt the flag.

0xae 0xed 0xbf 0x23 0x19 0xa2 0xbd 0x6a 0xa9 0x7b 0xdf 0xd6 0x5e 0x79 0x26 0x66
0x38 0x9c 0x92 0x1b 0x35 0xe3 0x22 0xf4 0x58 0x4e 0xd7 0xf2 0x9f 0xde 0xb0 0x14
0x0b 0x93 0xae 0x5b

Now the encrypted flag can be decrypted to reveal the expected input.
The following Python script was created to decrypt the flag:

# Expected bytes (36) from 0x555555558140
expected = [
    0xdd, 0x9a, 0xde, 0x4e, 0x69, 0xe1, 0xe9, 0x2c, 0xd2, 0x4e, 0xec, 0xe7,
    0x18, 0x26, 0x6a, 0x56, 0x79, 0xd8, 0xa3, 0x55, 0x72, 0xbc, 0x76, 0xc4,
    0x0c, 0x0f, 0x9b, 0xbe, 0xc6, 0x81, 0xe2, 0x41, 0x47, 0xa0, 0xf4, 0x26
]

# XOR keys (al) from GDB (36 bytes)
al_keys = [
    0xae, 0xed, 0xbf, 0x23, 0x19, 0xa2, 0xbd, 0x6a, 0xa9, 0x7b, 0xdf, 0xd6,
    0x5e, 0x79, 0x26, 0x66, 0x38, 0x9c, 0x92, 0x1b, 0x35, 0xe3, 0x22, 0xf4,
    0x58, 0x4e, 0xd7, 0xf2, 0x9f, 0xde, 0xb0, 0x14, 0x0b, 0x93, 0xae, 0x5b
]

flag = ''
for i in range(36):
    flag += chr(expected[i] ^ al_keys[i])

print("Flag: " + flag)