We are given a binary that reads a flag, transforms it using sub_1199, and then compares the mutated buffer against a constant target (s2) using:
if(!strncmp(s,&s2,0x25uLL))
So the goal is clear:
We must reverse the transformation performed in sub_1199 and recover the original input that produces the expected transformed bytes.
x = ((257 * a) ^ (509 * b) ^ (33 * prev_x)) & 0xFFFF
& 0xFFFF
blocks = "9c 85 b5 8d 12 a0 9b 10 e8 1f 2b b3 db 4a 87 1e 39 bd 03 32 c6 d0 82 db cd 46 82 a1 6d 09 80 e5 6c 7f 6c 82 91"
data = bytes.fromhex(blocks)
targets = []
for i in range(0, len(data), 2):
value = int.from_bytes(data[i:i+2], byteorder="little")
targets.append(value)
prev_x = 0
for candidate in targets:
found = False
for a in range(256):
for b in range(256):
x = ((257 * a) ^ (509 * b) ^ (33 * prev_x)) & 0xFFFF
if x == candidate:
print(chr(a) + chr(b), end="")
prev_x = x
found = True
break
if found:
break