Hooked on Mnemonics Worked for Me

Visualizing RC4 Key Initialization

2012-02-19T13:55:00.000-08:00

Recently, I was curious about what the key initialization for RC4 key stream would look like visually. If you are unfamiliar with RC4 I would recommend Wikipedia's article. What I'm focusing on in this post is the key scheduling algorithm.

Key scheduling algorithm:

for i from 0 to 255
    S[i] := i

endfor
j := 0
for i from 0 to 255
    j := (j + S[i] + key[i mod keylength]) mod 256
    swap values of S[i] and S[j]

endfor

What I found interesting about this algorithm is it originally starts with a 256 bytes array/list in sequential order (0,1,2,3,4...253,254,255). This range could be represented in the form of a gradient. The above image is the visual representation of each loop of the key stream. The bottom left hand corner from left to right is the initial sequential list. The loop iterates 256 times. Each time the loop iterates the key stream is modified.

Another image with the char 'B' as the key. The very top row is the complete key stream from the key-scheduling algorithm.

Another image with the char 'Z' as the key. The above image was created in Python using matplotlib. The code will iterate through chars from 'A' to 'Z'. The images will be saved with a file name of example-char.png.

import matplotlib.pyplot as plt
import numpy as np
import random
import sys

def rc4_init(key):
    # creates a list of the stream for each loop of the key stream 
    c = 0
    k = range(256)
    j = 0
    x = []
    x = x + k
    for i in range(256):
        j = (j + k[i] + ord(key[i % len(key)])) % 256
        k[i], k[j] = k[j], k[i]
        x = x + k

    return x

def createImage(key):
    # get list of RC4 key values size (256*257)
    data = np.array(rc4_init(key))
    data.shape = (257,256)
    # use heat map 
    plt.hot()
    plt.ylabel('Loop Iteration')
    plt.xlabel('Array Value 0-256')
    plt.title('RC4 Key Initialize with Value of ' + '\'' + str(key) + '\'')    
    plt.axis([0,256,0,257])
    plt.pcolormesh(data)
    plt.colorbar()
    #plt.show()
    plt.savefig('example-' + str(key) + '.png')
    plt.close()


def main(argv):
    for x in range(65,91):
        createImage(chr(x))


if __name__== '__main__':
        main(sys.argv[1:])

Visually I think this is pretty cool. I decided to take it a step further and create an animated gif. The .gif displays the image with value keys of 'A' through 'Z'. Warning the .gif is over 2MBs in size. LINK Odds are my terminology is off in some of the description of the algorithm. Please feel free to leave comments if you see one.

Shellcode Extraction, Automated Decoding and Carving using Pefile

2012-02-10T23:59:00.000-08:00

A little while back a friend of mine asked about a tool for carving out executables from file streams. PE carvers are useful for analyst who spend time analyzing files on the command line or extracting PE files from malicious documents. The initial post was going to be a simple Python script using Pefile for carving out executables. The simplicity was destroyed when I choose a random sample that used an offset specific encoding. I could have chosen another sample with a simpler encoding but that's boring. This post will cover three main topics. The first one is extracting and analyzing the shellcode from a malicious document CVE-2010-3333. The second is using patterns from encoding null bytes for automatically decoding (xor-count-up) embedded files. No static analysis or brute-forcing is used. The last topic is using Pefile for carving out executable files from file streams. This post is all Python except for the hexdump which is vim.

vim CVE-2010-3333_DOC.bad

{\rtf1{\shp{\*\shpinst{\sp{\sn pFragments}{\sv 1;1;ffffffffff050000000000000000000000..
..

:Vi can be used as a hexdump with the following command 
:%!xxd 

0000000: 7b5c 7274 6631 7b5c 7368 707b 5c2a 5c73  {\rtf1{\shp{\*\s
0000010: 6870 696e 7374 7b5c 7370 7b5c 736e 2070  hpinst{\sp{\sn p
0000020: 4672 6167 6d65 6e74 737d 7b5c 7376 2031  Fragments}{\sv 1
0000030: 3b31 3b66 6666 6666 6666 6666 6630 3530  ;1;ffffffffff050
0000040: 3030 3030 3030 3030 3030 3030 3030 3030  0000000000000000
0000050: 3030 3030 3030 3030 3030 3030 3030 3030  0000000000000000
0000060: 3030 3030 3030 3062 3735 6631 6637 6430  0000000b75f1f7d0
0000070: 3030 3038 3037 6330 3030 3038 3037 6342  000807c0000807cB
0000080: 4242 4242 4242 4243 4343 4343 4343 4344  BBBBBBBCCCCCCCCD
0000090: 4444 4444 4444 4439 3039 3039 3039 3039  DDDDDDD909090909
00000a0: 3039 3034 3134 3134 3134 3134 3134 3134  0904141414141414
00000b0: 3134 3134 3134 3134 3134 3165 3830 3130  14141414141e8010
....
0000740: 6239 3838 3730 6436 3730 6439 3839 3833  b98870d670d98983
0000750: 3038 3366 3766 3766 3766 3766 3766 3766  083f7f7f7f7f7f7f
0000760: 3765 6230 3665 6230 3430 3932 3330 3033  7eb06eb040923003
0000770: 3065 6230 3665 6230 3438 6332 3430 3033  0eb06eb048c24003
0000780: 3065 3837 3766 3466 6666 667d 7d7d 7d5c  0e877f4ffff}}}}\
0000790: 6164 6566 6c61 6e67 3130 3235 5c61 6e73  adeflang1025\ans
00007a0: 695c 616e 7369 6370 6739 3336 5c75 6332  i\ansicpg936\uc2
00007b0: 5c61 6465 6666 3331 3530 375c 6465 6666  \adeff31507\def
.....
:%!xxd -r 
:q!

Some quick notes in regards to CVE-2010-3333 via Mitre "Stack-based buffer overflow in Microsoft Office XP SP3, Office 2003 SP3, Office 2007 SP2, Office 2010, Office 2004 and 2008 for Mac, Office for Mac 2011, and Open XML File Format Converter for Mac allows remote attackers to execute arbitrary code via crafted RTF data, aka "RTF Stack Buffer Overflow Vulnerability."

The intent of this post is not to analyze CVE-2010-3333 but rather the encoding algorithm in the shellcode. For more information on the exploit see the following link. The most relevant parts are the rtf file header, the fragment objects (pFragments) and the data following the semicolon. The shellcode resides in the large ascii data block in the range 0-9A-F. If we read two ASCII chars and then treat them as hex/binary we would have valid Intel instructions. Let's see what this would look like in Python

Python 2.7 (r27:82525, Jul  4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import sys
>>> import os
>>> import pydasm
>>> # Open our specimen 
>>> f = open("CVE-2010-3333_DOC.bad","rb")
>>> f.seek(0x97)
>>> # 0x97 is the start of the ascii encoded shellcode
>>> b = f.read(2)

>>> buff = ""
>>> while b != '':
        try:
        buff = buff+chr(int(b,16))
        b = f.read(2)
        except ValueError:
        break

The variable buff contains our ascii to binary shellcode. We can use Pydasm to disassemble it.

>>> offset = 0
>>> while offset < len(buff):
    i = pydasm.get_instruction(buff[offset:],pydasm.MODE_32)
    print offset , ' ',pydasm.get_instruction_string(i,pydasm.FORMAT_INTEL, offset)
    if not i:
        break
    offset +=  i.length

    
0   inc ecx
1   inc ecx
2   inc ecx
3   inc ecx
4   inc ecx
5   inc ecx
6   inc ecx
7   inc ecx
8   inc ecx
9   inc ecx
10   inc ecx
11   inc ecx
12   call 0x12
17   add [ebx-0x3b7cdbf4],cl
23   add al,0x8d
25   dec ecx
26   adc al,[ecx-0x80]
29   xor [edi-0x80],esp
32   cmp [eax+0x600df775],edx
38   psubd mm6,[esi]
....
85   push dword 0x1300f74
90   push byte 0xffffff98
92   None

Notice after address 12 the code turns to junk instructions. This is caused by our disassembly being off by 1 byte after the call. We can modify our pydasm loop to skip this null byte.

>>> offset = 0
>>> while offset < len(buff):
    i = pydasm.get_instruction(buff[offset:],pydasm.MODE_32)
    print hex(offset) , ' ',pydasm.get_instruction_string(i,pydasm.FORMAT_INTEL, offset)
    if not i:
        break
    offset +=  i.length
    if offset == 17:
        offset += 1

        
0x0   inc ecx
0x1L   inc ecx
0x2L   inc ecx
0x3L   inc ecx
0x4L   inc ecx
0x5L   inc ecx
0x6L   inc ecx
0x7L   inc ecx
0x8L   inc ecx
0x9L   inc ecx
0xaL   inc ecx
0xbL   inc ecx
0xcL   call 0x12
0x12L   mov ecx,[esp]
0x15L   add esp,0x4
0x18L   lea ecx,[ecx+0x12]
0x1bL   inc ecx
0x1cL   xor byte [ecx],0x67
0x1fL   cmp byte [ecx],0x90
0x22L   jnz 0x1b
0x24L   or eax,0x36fa0f60
.......
0x53L   aad 0x51
0x55L   push dword 0x1300f74
0x5aL   push byte 0xffffff98
0x5cL   None

Note: address changes to hex. At the address 0x12 we can see the start of the XOR loop with a key of 0x67. The rest of the code is junk due to it being XORed. The next step to get to the second stage of the shellcode.

>>> buff2 = buff[:35]
# Read past XOR loop
>>> c = 36
>>> while c < (len(buff)-36):
    buff2 =  buff2 + chr(ord(buff[c])^0x67)
    c += 1

The above code is a simple XOR loop. Now we can dissasemble the second stage using Pydasm

>>> offset = 0
>>> while offset < len(buff2):
        i = pydasm.get_instruction(buff2[offset:],pydasm.MODE_32)
        print hex(offset) , ' ',pydasm.get_instruction_string(i,pydasm.FORMAT_INTEL, offset)
        if not i:
            break
        offset +=  i.length
        if offset == 17:
            offset += 1

        
0x0   inc ecx
0x1L   inc ecx
0x2L   inc ecx
0x3L   inc ecx
0x4L   inc ecx
0x5L   inc ecx
0x6L   inc ecx
0x7L   inc ecx
0x8L   inc ecx
0x9L   inc ecx
0xaL   inc ecx
0xbL   inc ecx
0xcL   call 0x12
0x12L   mov ecx,[esp]
0x15L   add esp,0x4
0x18L   lea ecx,[ecx+0x12]
0x1bL   inc ecx
0x1cL   xor byte [ecx],0x67
0x1fL   cmp byte [ecx],0x90
0x22L   jnz 0x8e
0x24L   pop es
0x25L   push dword 0x3519d
0x2aL   push dword 0x2d000
0x2fL   push dword 0x819d
........
0x8fL   xor edx,edx
0x91L   mov ebx,fs:[edx+0x30]    ; GET PEB
0x95L   mov ecx,[ebx+0xc]
0x98L   mov ecx,[ecx+0x1c]
....
0xbaL   mov esi,[ebx+edi*4]
0xbdL   add esi,ebp
0xbfL   cwd 
0xc0L   movsx eax,[esi]
0xc3L   cmp al,ah
0xc5L   jz 0xcf
0xc7L   ror edx,0x7        ; ROR API HASH
0xcaL   add edx,eax
0xccL   inc esi
0xcdL   jmp 0xc0
...
0x1b1L   mov [ebp+0x604],ebx
0x1b7L   xor ecx,ecx
0x1b9L   lea esi,[ebp+ecx+0x200]
0x1c0L   lodsb 
0x1c1L   xor al,cl            ; decode
0x1c3L   xchg edx,edi
0x1c5L   lea edi,[ebp+ecx+0x200]
0x1ccL   stosb 
0x1cdL   xchg edx,edi
0x1cfL   inc ecx
0x1d0L   cmp ecx,[ebp+0x604]
0x1d6L   jnz 0x1b9            ; loop
...
0x34eL   push ebp
0x34fL   mov ebp,esp
0x351L   mov eax,[edi-0x14]
0x354L   inc [eax]

Now that we have our shellcode. If we scroll up we will see a loop with xor al, cl and then an inc ecx. This is our xor-count-up loop. This type of encoding can be tricky because we have to know the exact offset of where the loop starts decoding data. Usually this means we would have to do some good old fashioned static analysis. There's only so many times we can reverse shellcode until you get bored. Let's get creative. Let's try something that hasn't been done before (probably wrong?). We know it's a xor-count-up so let's mimic the pattern the XOR would create in null data.

>>> o  = open('out.bin','wb+')
>>> d = f.read()
>>> k = ''
>>> for x in range(0,0xff):
        k = k + chr(x)

>>> k
'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r...'
# Get subset of the full XOR-Countup outpu

>>>  key = k[:15]

Note: The k value was edited due to formatting issues. For a screenshot of the output please see the following link.

This is where things get kind of interesting with the encoding. Firstly we are going to do a search for a subset (key) of the recurring pattern (k). The pattern would appear on null bytes that have been XORed with count-up. We choose a subset because the full pattern would not likely be present because the null bytes would need to have a length of 255. If we know the address of k and the algorithm we can calculate the one to one relationship of the first value 0x0-0xFF to the first byte of the file. Once we have that information we will just need to mimic the xor-count-up code from the start of the file.

>>> #  
>>> i = (0xff - (d.find(key)&0xff)) + 1
>>> b= ''
>>> for val in d:
        b = str(b) + chr(ord(val)^(i&0xff))
        i += 1
  
>>> o.write(b)
>>> o.close()
>>> f.close()
>>>

Let's see what our saved off buffer looks like in a VIM hexdump.

vim out.bin
0000000: 8d8c ae92 9090 909b 9997 9dca cccf cba3  ................
0000010: 616f 716a 5864 6874 616a 7a6c 353e 3853  aoqjXdhtajzl5>8S
0000020: 6572 204f 7571 7371 7e2a 2b2e 2c2a 427b  er Ouqsq~*+.,*B{
0000030: 4547 4413 7856 5254 404f 4e49 4f45 1d1e  EGD.xVRT@ONIOE..
0000040: 0501 076f 4741 455f 5e55 5558 540e 0f0a  ...oGAE_^UUXT...
0000050: 7077 1e30 3036 2e21 2020 2923 7f7c 7b7f  pw.006.!  )#.|{.
0000060: 660d 2127 273d 3035 316a 6b6e 6c6a 023b  f.!''=051jknlj.;
..
:/This
00255b0: 0000 0000 0408 b004 4d5a 9000 0300 0000  ........MZ......
00255c0: 0400 0000 ffff 0000 b800 0000 0000 0000  ................
00255d0: 4000 0000 0000 0000 0000 0000 0000 0000  @...............
00255e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00255f0: 0000 0000 f000 0000 0e1f ba0e 00b4 09cd  ................
0025600: 21b8 014c cd21 5468 6973 2070 726f 6772  !..L.!This progr
0025610: 616d 2063 616e 6e6f 7420 6265 2072 756e  am cannot be run
0025620: 2069 6e20 444f 5320 6d6f 6465 2e0d 0d0a   in DOS mode....
0025630: 2400 0000 0000 0000 b3bf 303a f7de 5e69  $.........0:..^i
:q!

Yeah, that's hot. Now that we have the file decoded we can carve out the executable from the data stream. The first indicator for an embedded executable is the MZ header. If we find the MZ header we would need to jump 0x3c bytes, read the value, jump to that value, check for the PE header... or we could use pefile to validate the PE file for us. All we need to do is search a stream of data for the MZ header, set the file pointer to that address, read till end of file, pass the data to pefile and then check for errors. If there is no errors we will have pefile trim the file and then write it to disk. This might not be the best method if the file is large or if we were overly concerned about file overlays.

import pefile
import re
import sys

def carve(f):
    c = 1
    for y in [tmp.start() for tmp in re.finditer('\x4d\x5a',f.read())]:
        f.seek(y)
        try:
            pe = pefile.PE(data=f.read())
        except:
            pass
            continue
        # determine file ext
        if pe.is_dll() == True:
            ext = 'dll'
        if pe.is_driver() == True:
            ext =  'sys'
        if pe.is_exe() == True:
            ext = 'exe'
        o = open(str(c)+ '.' + ext, 'wb')
        print ext , 'found at offset', hex(y) 
        o.write(pe.trim())
        o.close()
        c = c + 1
        ext = ''
        f.seek(0)
        pe.close()

def main(argv):
    if len(sys.argv) < 2:
        print "cpe.py <file-stream>"
    else:
        i = open(sys.argv[1], "rb")
        carve(i)
        i.close()
            
if __name__== '__main__':
        main(sys.argv[1:])

Output from the above script on the out.bin

python cpe.py out.bin
exe found at offset 0x7a10
dll found at offset 0x255b8

Yara + MD5

2011-12-18T08:35:00.000-08:00

There seems to be a good amount of people searching for Yara + MD5 and landing on this blog. Yara does not support MD5 hashing. There are a couple options for detecting files via MD5. One solution is to write a md5 scanner using your favorite programming language, the second is to use ClamAV and another is to use Yara and Python. Below is a quick demonstration of the later. Let's assume we already have the MD5 hash of a bad file A1EB325F994E5A1720C0E401731B5ED9 . We will now need to create a Yara rule with the MD5 hash as a string.

rule MD5_BAD_FILE
{  
strings:  
    $md5 = "A1EB325F994E5A1720C0E401731B5ED9" nocase
condition:  
    $md5
}

The Yara rule will alert on the string of the MD5 hash. Now we need some code that will open a file, hash the file and then scan the hash value using the Yara rule.

import hashlib
import sys
import imp
import yara 

def MD5(d):
# d = buffer of the read file 
# This function hashes the buffer
# source: http://stackoverflow.com/q/5853830
    if type(d) is str:
      d = StringIO(d)
    md5 = hashlib.md5()
    while True:
        data = d.read(128)
        if not data:
            break
        md5.update(data)
    return md5.hexdigest()


def yaraScan(d):
# d = buffer of the read file 
# Scans SWF using Yara
    # test if yara module is installed
    # if not Yara can be downloaded from http://code.google.com/p/yara-project/
    try:
        imp.find_module('yara')
        import yara 
    except ImportError:
        print '\t[ERROR] Yara module not installed - aborting scan'
        return
    # test for yara compile errors
    try:
        r = yara.compile(r'md5.yara')
    except:
        pass
        print '\t[ERROR] Yara compile error - aborting scan'
        return
    # get matches
    m = r.match(data=d)
    # print matches
    for X in m:
        print '\t[BAD] Yara Signature Hit:', X
    return

def main():
    try:
        f = open(sys.argv[len(sys.argv)-1],'rb+')
    except Exception:
        print '[ERROR] File can not be opended/accessed'
        return
    yaraScan(MD5(f))    

if __name__ == '__main__':
   main()

Example

C:\Documents and Settings\XOR\My Documents\Projects\yaramd5>python yaraMD5.py "6UHp0dCM12c[1].swf"
        [BAD] Yara Signature Hit: MD5_BAD_FILE

Most of the code is from xxxswf.py.

xxxswf.py

2011-12-07T21:01:00.001-08:00

xxxswf.py is a Python script for carving, scanning, compressing, decompressing and analyzing Flash SWF files. The script can be used on an individual SWF, single SWF or multiple SWFs embedded in a file stream or all files in a directory. The tool could be useful for system admistrators, incident response, exploit analyst, malware analyst or web developers.

C:\Documents and Settings\XOR\My Documents\Projects\swfxxx>python xxxswf.py -h
Usage: xxxswf.py [options] file.bad

Options:
  -h, --help            show this help message and exit
  -x, --extract         Extracts the embedded SWF(s), names it MD5HASH.swf &
                        saves it in the working dir. No addition args needed
  -y, --yara            Scans the SWF(s) with yara. If the SWF(s) is
                        compressed it will be deflated. No addition args
                        needed
  -s, --md5scan         Scans the SWF(s) for MD5 signatures. Please see func
                        checkMD5 to define hashes. No addition args needed
  -H, --header          Displays the SWFs file header. No addition args needed
  -d, --decompress      Deflates compressed SWFS(s)
  -r PATH, --recdir=PATH
                        Will recursively scan a directory for files that
                        contain SWFs. Must provide path in quotes
  -c, --compress        Compresses the SWF using Zlib</pre>

xxxswf.py with no options and a file passed. The output is extremely simple. The [SUMMARY] shows the count of embedded SWFs. The MD5 and name of the scanned file, the address of the embedded SWF and the header of the SWF. FWS is uncompressed and CWS is compressed with zlib.

C:\Documents and Settings\XOR\My Documents\Projects\swfxxx>python xxxswf.py test.swf
[SUMMARY] 1 SWF(s) in MD5:7ca4ab177f480503653702b33366111f:test.swf
        [ADDR] SWF 1 at 0xa18  - CWS Header

xxxswf.py with the -x (--extract) option. The file will be carved and saved to the working directory. The name will be the MD5 of the deflated SWF and the '.swf' extension. If there are multiple files with the same MD5 the file's name will be MD5.count.swf. The count will only go up to 50. A useful example of this will be given later.

C:\Documents and Settings\XOR\My Documents\Projects\swfxxx>python xxxswf.py -x x.bin
[SUMMARY] 2 SWF(s) in MD5:32fed596fa850057211121488f6c6b75:x.bin
        [ADDR] SWF 1 at 0x0  - FWS Header
                [FILE] Carved SWF MD5: c46299a5015c6d31ad5766cb49e4ab4b.swf
        [ADDR] SWF 2 at 0x7774  - FWS Header
                [FILE] Carved SWF MD5: c46299a5015c6d31ad5766cb49e4ab4b.2.swf

The -r or --recdir option can be used to recursively search or carve out all SWFs in a directory. This could be used on a temporary internet directory or a repository of malicious documents. It's recommend to pipe the output to a text file. The path will need to be in quotes. This can take a few minutes due to the size of the directory and the speed of your processor.

C:\Documents and Settings\XOR\My Documents\Projects\swfxxx>python xxxswf.py -x -r "C:\Documents and Settings\XOR\Desktop\samples\mal" > out.txt

vi out.txt

[SUMMARY] 1 SWF(s) in MD5:93d63b5f9167d7ab579ca9bd70d1dd3e:C:\Documents and Settings\XOR\Desktop\samples\mal\301.xls=
    [ADDR] SWF 1 at 0x13ef81 - [ERROR]: Zlib decompression error. Invalid CWS SWF

[SUMMARY] 1 SWF(s) in MD5:d2cad99c92a1a43b8ed0c217b6a501af:C:\Documents and Settings\XOR\Desktop\samples\mal\CVE-2009-3129.xls
    [ADDR] SWF 1 at 0x13ef81 - [ERROR]: Zlib decompression error. Invalid CWS SWF

[SUMMARY] 1 SWF(s) in MD5:358895e898866ef0432391b931096209:C:\Documents and Settings\XOR\Desktop\samples\mal\CWS.swf
    [ADDR] SWF 1 at 0x0  - CWS Header
        [FILE] Carved SWF MD5: f05ba07d32e9a7b47a18aa3f172ad4e5.swf

[SUMMARY] 1 SWF(s) in MD5:c46299a5015c6d31ad5766cb49e4ab4b:C:\Documents and Settings\XOR\Desktop\samples\mal\simple.swf
    [ADDR] SWF 1 at 0x0  - FWS Header
        [FILE] Carved SWF MD5: c46299a5015c6d31ad5766cb49e4ab4b.3.swf

[SUMMARY] 7 SWF(s) in MD5:7089ec4198e70f58f09547201ae4e185:C:\Documents and Settings\XOR\Desktop\samples\mal\swfxxx.py
    [ADDR] SWF 1 at 0x607  - [ERROR] Invalid SWF Version
    [ADDR] SWF 2 at 0x60b  - [ERROR] Invalid SWF Version
    [ADDR] SWF 3 at 0x958  - [ERROR] Invalid SWF Version
    [ADDR] SWF 4 at 0x981  - [ERROR] Invalid SWF Size
    [ADDR] SWF 5 at 0x18d0  - [ERROR] Invalid SWF Size
    [ADDR] SWF 6 at 0x1c45  - [ERROR] Invalid SWF Size
    [ADDR] SWF 7 at 0x1cc3  - [ERROR] Invalid SWF Size
....

The search for embedded SWFs is done simply by using a regular expression with "FWS" and "CWS" as the search criteria. This generic search will return false positives. Verifying the SWF is done by checking for a valid version, valid size and valid decompression (if compressed). Please see the function verifySWF(). This approach is time consuming but it does work. Above we can see the different errors being generated. All errors will contain the string "[ERROR]". If the sample set is large enough odds are there will be recurring MD5 file names. xxxswf.py can be used to classify or alert on commonly used MD5 SWFs. The function checkMD5 can be edited to alert on specific MD5s.

def checkMD5(md5):
# checks if MD5 has been seen in MD5 Dictionary 
# MD5Dict contains the MD5 and the CVE
# For { 'MD5':'CVE', 'MD5-1':'CVE-1', 'MD5-2':'CVE-2'}
    MD5Dict = {'c46299a5015c6d31ad5766cb49e4ab4b':'CVE-XXXX-XXXX'}
    if MD5Dict.get(md5):
        print '\t[BAD] MD5 Match on', MD5Dict.get(md5)
    return

The MD5 "c46299a5015c6d31ad5766cb49e4ab4b" was found in the x.bin example from a couple example above. MD5 scanning is done by passing the -s or --md5scan. All hashing or signature alerts contain the string [BAD].

C:\Documents and Settings\XOR\My Documents\Projects\swfxxx>python xxxswf.py -s x.bin
[SUMMARY] 2 SWF(s) in MD5:32fed596fa850057211121488f6c6b75:x.bin
        [ADDR] SWF 1 at 0x0  - FWS Header
        [BAD] MD5 Match on CVE-XXXX-XXXX
        [ADDR] SWF 2 at 0x7774  - FWS Header
        [BAD] MD5 Match on CVE-XXXX-XXXX

xxxswf can be used to decompress a single SWF by using the -d --decompress option.

C:\Documents and Settings\XOR\My Documents\Projects\swfxxx>python xxxswf.py -d test.swf
[SUMMARY] 1 SWF(s) in MD5:7ca4ab177f480503653702b33366111f:test.swf
        [ADDR] SWF 1 at 0xa18  - CWS Header
                [FILE] Carved SWF MD5: f0f40a975ef68cf6358f84515a8f103e.4.swf

It can compress SWFs using the -c or --compress options. Note: In testing I wasn't able to decompress a SWF downloaded from the internet and compress it again to get a matching MD5. A single byte is off. If someone could give me a clue on this one or recommend another technique please let me know.

C:\Documents and Settings\XOR\My Documents\Projects\swfxxx>python xxxswf.py -c f0f40a975ef68cf6358f84515a8f103e.2.swf
[SUMMARY] 1 SWF(s) in MD5:f0f40a975ef68cf6358f84515a8f103e:f0f40a975ef68cf6358f8
4515a8f103e.2.swf
        [ADDR] SWF 1 at 0x0  - FWS Header
                [FILE] Compressed SWF MD5: e9e6c13c461dc38006ff7d26c18e904e.swf

The SWF headers information can be displayed by using -H or --header

C:\Documents and Settings\XOR\My Documents\Projects\swfxxx>python xxxswf.py -H 11cc16d78597fe9999b7f6b714727ac3.10.swf
[SUMMARY] 1 SWF(s) in MD5:11cc16d78597fe9999b7f6b714727ac3:11cc16d78597fe9999b7f
6b714727ac3.10.swf
        [ADDR] SWF 1 at 0x0  - FWS Header
        [HEADER] File header: FWS
        [HEADER] File version: 7
        [HEADER] File size: 52647
        [HEADER] Rect Nbit: 15
        [HEADER] Rect Xmin: 0
        [HEADER] Rect Xmax: 11000
        [HEADER] Rect Ymin: 0
        [HEADER] Rect Ymax: 3600
        [HEADER] Frame Rate: 7936
        [HEADER] Frame Count: 1

The output for Rect is in twips. The script contains the ability to scan the deflated SWF(s) with yara. The options are -y and --yara. This makes it easy to create signatures on malicious SWF files that do not have static MD5s. Due to the scanning set only being in a SWF file the signatures can be a little more generic. Let's walk through an example using information gathered from the excellent write up by Microsoft.

http://blogs.technet.com/b/mmpc/archive/2011/03/17/a-technical-analysis-on-the-cve-2011-0609-adobe-flash-player-vulnerability.aspx

After reading the link we know some key things. We know what is triggering the exploit (bytecode verification error), we know there is some shellcode and we know there is some code for creating the heap spray. The analysis gives a nice clue about the exploit and what to target. From the analysis "The Adobe Flash file embedded inside the Excel file is another carrier for the exploit. It loads shellcode inside memory, performs heap-spraying, and loads a Flash byte stream from memory to exploit the 0-day vulnerability". If you look closely at the byte stream in the screenshot you will notice the string "43575309". What would this sring or Flash byte stream looks like if it was actually binary data and not a string?

import sys
s = "43575309"
for i in xrange(0,len(s),2): 
    sys.stdout.write(chr(int('0x'+ s[i:i+2],16)))

CWS

As mentioned earlier 'CWS' is the header for a compressed SWF. Nine is the Flash version. We have an embedded SWF that is stored into a byte array and then converted from hex to binary. Let's create a yara signature targeting this. Note: the string hexToBin is the name of a function and in a way is arbitrary. It's better to go after the code or data related to triggering the exploit. This exploit is a little more difficult because the trigger is embedded in a compressed SWF stored as ASCII hex. For more information please see my poor-grammar-non-proof-read post called An Intro to Creating Anti-Virus Signatures.

rule CVE_2011_0609
{  
strings:  
    $CWSHeader = "435753"
    $FWSHeader = "465753"
    $hex2bin = "hexToBin"
    
condition:  
    ($CWSHeader or $FWSHeader) and $hex2bin
}

Saved int the working dir as rules.yar.

C:\Documents and Settings\XOR\My Documents\Projects\swfxxx>python xxxswf.py -y "CVE-2011-0609_.xls__"
[SUMMARY] 1 SWF(s) in MD5:4bb64c1da2f73da11f331a96d55d63e2:CVE-2011-0609_.xls=__

        [ADDR] SWF 1 at 0xa18  - FWS Header
        [BAD] Yara Signature Hit: CVE_2011_0609

If you would like to import this script there is a function called bad(). This function can be used for scanning a SWF with MD5 and Yara. An open file handle will need to be passed to the function. The output will then need to be parsed for a line containing [BAD]. If interested in Yara and MD5 signatures feel free to contact me. I won't be posting my signature sets but I might be able to share depending on the organization or group.

Summary
The goal of this tool is to be able to work with embedded SWF files in an easy and quick way. This script is a work in progress. With a recent move to NYC I needed a new project. If you find any bugs or have some comments please contact me or leave a comment.

xxxswf.py - download

# xxxswf.py was created by alexander dot hanel at gmail dot com
# version 0.1 
# Date - 12-07-2011 
# To do list
#   - Tag Parser
#   - ActionScript Decompiler

import fnmatch 
import hashlib
import imp
import math
import os
import re
import struct
import sys
import time
from StringIO import StringIO
from optparse import OptionParser
import zlib

def checkMD5(md5):
# checks if MD5 has been seen in MD5 Dictionary 
# MD5Dict contains the MD5 and the CVE
# For { 'MD5':'CVE', 'MD5-1':'CVE-1', 'MD5-2':'CVE-2'}
    MD5Dict = {'c46299a5015c6d31ad5766cb49e4ab4b':'CVE-XXXX-XXXX'}
    if MD5Dict.get(md5):
        print '\t[BAD] MD5 Match on', MD5Dict.get(md5)
    return    

def bad(f):
    for idx, x in enumerate(findSWF(f)):
        tmp = verifySWF(f,x)
        if tmp != None:
            yaraScan(tmp)
            checkMD5(hashBuff(tmp))
    return 
    
def yaraScan(d):
# d = buffer of the read file 
# Scans SWF using Yara
    # test if yara module is installed
    # if not Yara can be downloaded from http://code.google.com/p/yara-project/
    try:
        imp.find_module('yara')
        import yara 
    except ImportError:
        print '\t[ERROR] Yara module not installed - aborting scan'
        return
    # test for yara compile errors
    try:
        r = yara.compile(r'rules.yar')
    except:
        pass
        print '\t[ERROR] Yara compile error - aborting scan'
        return
    # get matches
    m = r.match(data=d)
    # print matches
    for X in m:
        print '\t[BAD] Yara Signature Hit:', X
    return

def findSWF(d):
# d = buffer of the read file 
# Search for SWF Header Sigs in files
    return [tmp.start() for tmp in re.finditer('CWS|FWS', d.read())]

def hashBuff(d):
# d = buffer of the read file 
# This function hashes the buffer
# source: http://stackoverflow.com/q/5853830
    if type(d) is str:
      d = StringIO(d)
    md5 = hashlib.md5()
    while True:
        data = d.read(128)
        if not data:
            break
        md5.update(data)
    return md5.hexdigest()

def verifySWF(f,addr):
    # Start of SWF
    f.seek(addr)
    # Read Header
    header = f.read(3)
    # Read Version
    ver = struct.unpack('<b', f.read(1))[0]
    # Read SWF Size
    size = struct.unpack('<i', f.read(4))[0]
    # Start of SWF
    f.seek(addr)
    try:
        # Read SWF into buffer. If compressed read uncompressed size. 
        t = f.read(size)
    except:
        pass
        # Error check for invalid SWF
        print ' - [ERROR] Invalid SWF Size'
        return None
    if type(t) is str:
      f = StringIO(t)
    # Error check for version above 20
    if ver > 20:
        print ' - [ERROR] Invalid SWF Version'
        return None
    
    if 'CWS' in header:
        try:
            f.read(3)
            tmp = 'FWS' + f.read(5) + zlib.decompress(f.read())
            print ' - CWS Header'
            return tmp
        
        except:
            pass
            print '- [ERROR]: Zlib decompression error. Invalid CWS SWF'
            return None
        
    elif 'FWS' in header:
        try:
            tmp = f.read(size)
            print ' - FWS Header'
            return tmp
        
        except:
            pass
            print ' - [ERROR] Invalid SWF Size'
            return None
        
    else:
        print ' - [Error] Logic Error Blame Programmer'
        return None
    
def headerInfo(f):
# f is the already opended file handle 
# Yes, the format is is a rip off SWFDump. Can you blame me? Their tool is awesome.
    # SWFDump FORMAT    
    # [HEADER]        File version: 8
    # [HEADER]        File is zlib compressed. Ratio: 52%
    # [HEADER]        File size: 37536
    # [HEADER]        Frame rate: 18.000000
    # [HEADER]        Frame count: 323
    # [HEADER]        Movie width: 217.00
    # [HEADER]        Movie height: 85.00
    if type(f) is str:
      f = StringIO(f)
    sig = f.read(3)             
    print '\t[HEADER] File header:', sig
    if 'C' in sig:
        print '\t[HEADER] File is zlib compressed.'
    version = struct.unpack('<b', f.read(1))[0]
    print '\t[HEADER] File version:', version
    size = struct.unpack('<i', f.read(4))[0]
    print '\t[HEADER] File size:', size
    # deflate compressed SWF
    if 'C' in sig:
        f = verifySWF(f,0)
        if type(f) is str:
            f = StringIO(f)
        f.seek(0, 0)
        x = f.read(8)
    ta = f.tell()
    tmp = struct.unpack('<b', f.read(1))[0]
    nbit =  tmp >> 3
    print '\t[HEADER] Rect Nbit:', nbit
    # Curretely the nbit is static at 15. This could be modified in the
    # future. If larger than 9 this will break the struct unpack. Will have
    # to revist must be a more effective way to deal with bits. Tried to keep
    # the algo but damn this is ugly...
    f.seek(ta)
    rect =  struct.unpack('>Q', f.read(int(math.ceil((nbit*4)/8.0))))[0]
    tmp = struct.unpack('<b', f.read(1))[0]
    tmp = bin(tmp>>7)[2:].zfill(1)
    # bin requires Python 2.6 or higher
    # skips string '0b' and the nbit 
    rect =  bin(rect)[7:] 
    xmin = int(rect[0:nbit-1],2)
    print '\t[HEADER] Rect Xmin:', xmin
    xmax = int(rect[nbit:(nbit*2)-1],2)
    print '\t[HEADER] Rect Xmax:', xmax
    ymin = int(rect[nbit*2:(nbit*3)-1],2)
    print '\t[HEADER] Rect Ymin:', ymin
    # one bit needs to be added, my math might be off here
    ymax = int(rect[nbit*3:(nbit*4)-1] + str(tmp) ,2)
    print '\t[HEADER] Rect Ymax:', ymax
    framerate = struct.unpack('<H', f.read(2))[0]
    print '\t[HEADER] Frame Rate:', framerate
    framecount = struct.unpack('<H', f.read(2))[0] 
    print '\t[HEADER] Frame Count:', framecount
       
def walk4SWF(path):
    # returns a list of [folder-path, [addr1,addrw2]]
    # Don't ask, will come back to this code. 
    p = ['',[]]
    r = p*0
    if os.path.isdir(path) != True and path != '':
        print '\t[ERROR] walk4SWF path must be a dir.'
        return 
    for root, dirs, files in os.walk(path):
        for name in files:
            try: 
                x = open(os.path.join(root, name), 'rb')
            except:
                pass
                break
            y = findSWF(x)
            if len(y) != 0:
                # Path of file SWF
                p[0] = os.path.join(root, name)
                # contains list of the file offset of SWF header
                p[1] = y
                r.insert(len(r),p)
                p = ['',[]]
                y = ''
            x.close()
    return r

def tagsInfo(f):
    return

def fileExist(n, ext):
    # Checks the working dir to see if the file is
    # already in the dir. If exists the file will
    # be named name.count.ext (n.c.ext). No more than
    # 50 matching MD5s will be written to the dir. 
    if os.path.exists( n + '.' + ext):
                c = 2
                while os.path.exists(n + '.' + str(c) + '.' + ext):
                    c =  c + 1
                    if c == 50:
                        print '\t[ERROR] Skipped 50 Matching MD5 SWFs'
                        break
                n = n + '.' + str(c)
                
    return n + '.' + ext
    
def CWSize(f):
    # The file size in the header is of the uncompressed SWF.
    # To estimate the size of the compressed data, we can grab
    # the length, read that amount, deflate the data, then
    # compress the data again, and then call len(). This will
    # give us the length of the compressed SWF. 
    return

def compressSWF(f):
    if type(f) is str:
      f = StringIO(f)
    try:
        f.read(3)
        tmp = 'CWS' + f.read(5) + zlib.compress(f.read())
        return tmp
    except:
        pass
        print '\t[ERROR] SWF Zlib Compression Failed'
        return None

def disneyland(f,filename, options):
    # because this is where the magic happens
    # but seriously I did the recursion part last..
    retfindSWF = findSWF(f)
    f.seek(0)
    print '\n[SUMMARY] %d SWF(s) in MD5:%s:%s' % ( len(retfindSWF),hashBuff(f), filename )
    # for each SWF in file 
    for idx, x in enumerate(retfindSWF):
        print '\t[ADDR] SWF %d at %s' % (idx+1, hex(x)),
        f.seek(x)
        h = f.read(1)
        f.seek(x)
        swf = verifySWF(f,x)
        if swf == None:
            continue
        if options.extract != None:
            name = fileExist(hashBuff(swf), 'swf')
            print '\t\t[FILE] Carved SWF MD5: %s' % name 
            try:
                o = open(name, 'wb+')
            except IOError, e:
                print '\t[ERROR] Could Not Create %s ' % e
                continue 
            o.write(swf)
            o.close()
        if options.yara != None:
            yaraScan(swf)
        if options.md5scan != None:
            checkMD5(hashBuff(swf))
        if options.decompress != None:
            name = fileExist(hashBuff(swf), 'swf')
            print '\t\t[FILE] Carved SWF MD5: %s' % name 
            try:
                o = open(name, 'wb+')
            except IOError, e:
                print '\t[ERROR] Could Not Create %s ' % e
                continue
            o.write(swf)
            o.close()
        if options.header != None:
            headerInfo(swf)
        if options.compress != None:
            swf = compressSWF(swf)
            if swf == None:
                continue 
            name = fileExist(hashBuff(swf), 'swf')
            print '\t\t[FILE] Compressed SWF MD5: %s' % name
            try:
                o = open(name, 'wb+')
            except IOError, e:
                print '\t[ERROR] Could Not Create %s ' % e
                continue
            o.write(swf)
            o.close()

def main():
    # Scenarios:
    # Scan file for SWF(s)
    # Scan file for SWF(s) and extract them 
    # Scan file for SWF(s) and scan them with Yara
    # Scan file for SWF(s), extract them and scan with Yara
    # Scan directory recursively for files that contain SWF(s) 
    # Scan directory recursively for files that contain SWF(s) and extract them
    
    parser = OptionParser()
    usage = 'usage: %prog [options] <file.bad>'
    parser = OptionParser(usage=usage)
    parser.add_option('-x', '--extract', action='store_true', dest='extract', help='Extracts the embedded SWF(s), names it MD5HASH.swf & saves it in the working dir. No addition args needed')
    parser.add_option('-y', '--yara', action='store_true', dest='yara', help='Scans the SWF(s) with yara. If the SWF(s) is compressed it will be deflated. No addition args needed')
    parser.add_option('-s', '--md5scan', action='store_true', dest='md5scan', help='Scans the SWF(s) for MD5 signatures. Please see func checkMD5 to define hashes. No addition args needed')
    parser.add_option('-H', '--header', action='store_true', dest='header', help='Displays the SWFs file header. No addition args needed')
    parser.add_option('-d', '--decompress', action='store_true', dest='decompress', help='Deflates compressed SWFS(s)')
    parser.add_option('-r', '--recdir', dest='PATH', type='string', help='Will recursively scan a directory for files that contain SWFs. Must provide path in quotes')
    parser.add_option('-c', '--compress', action='store_true', dest='compress', help='Compresses the SWF using Zlib')
    
    (options, args) = parser.parse_args()

    # Print help if no argurments are passed
    if len(sys.argv) < 2:
        parser.print_help()
        return

    # Note files can't start with '-'
    if '-' in sys.argv[len(sys.argv)-1][0] and options.PATH == None:
        parser.print_help()
        return
    
    # Recusive Search
    if options.PATH != None:
        paths = walk4SWF(options.PATH)
        for y in paths:
            #if sys.argv[0] not in y[0]:
            try:
                t = open(y[0], 'rb+')
                disneyland(t, y[0],options)
            except IOError:
                pass
        return 
        
    # try to open file 
    try:
        f = open(sys.argv[len(sys.argv)-1],'rb+')
        filename = sys.argv[len(sys.argv)-1]
    except Exception:
        print '[ERROR] File can not be opended/accessed'
        return

    disneyland(f,filename,options)
    f.close()
    return 
        
if __name__ == '__main__':
   main()

Visual Studio Options for Cleaner Assembly

2011-09-05T19:25:00.000-07:00

Experimenting with Visual Studio compiler options has been an interest of mine lately. The goal is to get cleaner assembly in IDA. While writing C code I sometimes wonder "how would this C code translate to assembly?". Google gave me some decent command line options for CL but I still couldn't get the code I wanted. Luckily, I stumbled upon some extremely helpful slides by OpenSecurityTraining.com. Sadly, the slides are zipped and haven't been indexed by Google. Below are the options for Visual Studio C++ for cleaner assembly.

(right click on .c or .cpp)
Source File > Properties

Configuration Properties > C/C++ > General
> Debug Information Format - Program DataBase (/Zi)
> Warning Level - Level3 (/W3)

Configuration Properties > C/C++ > Code Generation
> Enable Minimal Rebuilt - No
> Enable C++ Exception - No
> Basic Runtime Checks - Default
> Buffer Security Check - No (/GS-)

Configuration Properties > C/C++ > Advanced
> Compile As - Compile as C Code (/TC)

Configuration Properties > Linker > General
> Enable Incremental Linking - No (/INCREMENTAL:NO)

So how does each one of these options effect a "Hello World" when we change one option, compile and then change the next?


#include 

int main(void)
{
  printf("Hello world!");
  return 0;
}

Compiled using Visual Studio 2010 C++ default settings.


.text:00411380 main            proc near               ; CODE XREF: j_main j
.text:00411380
.text:00411380 var_C0          = dword ptr -0C0h
.text:00411380
.text:00411380                 push    ebp
.text:00411381                 mov     ebp, esp
.text:00411383                 sub     esp, 0C0h
.text:00411389                 push    ebx
.text:0041138A                 push    esi
.text:0041138B                 push    edi

//////////////////////////////////////////////////////////////////////////////

.text:0041138C                 lea     edi, [ebp+var_C0]
.text:00411392                 mov     ecx, 30h
.text:00411397                 mov     eax, 0CCCCCCCCh
.text:0041139C                 rep stosd     

//////////////////////////////////////////////////////////////////////////////
// Initialization of local variables to a nonzero value. This helps identify 
// bugs that do not appear when running in debug mode. There is a greater 
// chance that stack variables will still be zero in a debug build compared 
// to a release build because of compiler optimizations of stack variables in 
// a release build. Once a program has used an area of its stack, it is never 
// reset to 0 by the compiler. Therefore, subsequent, uninitialized stack 
// variables that happen to use the same stack area can return values left 
// over from the prior use of this stack memory.
// Source http://msdn.microsoft.com/en-us/library/8wtf2dfz.aspx
//////////////////////////////////////////////////////////////////////////////

.text:0041139E                 mov     esi, esp
.text:004113A0                 push    offset aHelloWorld  ; "Hello world!"
.text:004113A5                 call    ds:__imp__printf
.text:004113AB                 add     esp, 4
.text:004113AE                 cmp     esi, esp
.text:004113B0                 call    j__RTC_CheckEsp
.text:004113B5                 xor     eax, eax
.text:004113B7                 pop     edi
.text:004113B8                 pop     esi
.text:004113B9                 pop     ebx
.text:004113BA                 add     esp, 0C0h
.text:004113C0                 cmp     ebp, esp
.text:004113C2                 call    j__RTC_CheckEsp

//////////////////////////////////////////////////////////////////////////////
// In debug versions of executables, the Microsoft Visual Studio compilers 
// insert a call to RTC_CheckEsp() after each function call. Prior to the 
// function call, the contents of the stack pointer register (ESP/RSP) are 
// saved into a general-purpose register. This function compares the contents 
// of that general-purpose register with the stack pointer to verify that the 
// stack pointer was restored correctly. This error usually occurs when a 
// function is declared using one calling-convention and actually implemented 
// using another. If there is a mismatch, the library generates an error message 
// and a breakpoint.
// Source: http://blogs.phoenix.com/phoenix_technologies_bios/2008/11/bios-undercover-porting-cc-to-phoenix-uefi.html
//////////////////////////////////////////////////////////////////////////////

.text:004113C7                 mov     esp, ebp
.text:004113C9                 pop     ebp
.text:004113CA                 retn
.text:004113CA main            endp

Size 27,136 bytes.

Configuration Properties > C/C++ > General > Debug Information Format - Program DataBase (/Zi)



//////////////////////////////////////////////////////////////////////////////
// Produces a program database (PDB) that contains type information and symbolic 
// debugging information for use with the debugger. The symbolic debugging 
// information includes the names and types of variables, as well as functions and 
// line numbers.

//  /Zi does not affect optimizations. However, /Zi does imply /debug; see /DEBUG 
// (Generate Debug Info) for more information.
// Source: http://msdn.microsoft.com/en-us/library/958x11bc%28v=vs.80%29.aspx
//////////////////////////////////////////////////////////////////////////////

.text:00401010 main            proc near               ; CODE XREF: j_main j
.text:00401010                 push    ebp
.text:00401011                 mov     ebp, esp
.text:00401013                 push    esi
.text:00401014                 mov     esi, esp
.text:00401016                 push    offset aHelloWorld ; "Hello world!"
.text:0040101B                 call    ds:__imp__printf
.text:00401021                 add     esp, 4
.text:00401024                 cmp     esi, esp
.text:00401026                 call    _RTC_CheckEsp
.text:0040102B                 xor     eax, eax
.text:0040102D                 pop     esi
.text:0040102E                 cmp     ebp, esp
.text:00401030                 call    _RTC_CheckEsp
.text:00401035                 pop     ebp
.text:00401036                 retn
.text:00401036 main            endp

Size 26,624 bytes.

Configuration Properties > C/C++ > Code Generation > Enable Minimal Rebuilt - No (/GM-)
* No changes in code or size.

Configuration Properties > C/C++ > Code Generation > Enable C++ Exception - No
* No changes in code or size.

Configuration Properties > C/C++ > Code Generation > Basic Runtime Checks - Default



//////////////////////////////////////////////////////////////////////////////
// Used to enable and disable the run-time error checks feature, in conjunction 
// with the runtime_checks pragma.
// Source:http://msdn.microsoft.com/en-us/library/8wtf2dfz.aspx
//////////////////////////////////////////////////////////////////////////////

.text:00401010 main            proc near               ; CODE XREF: j_main j
.text:00401010                 push    ebp
.text:00401011                 mov     ebp, esp
.text:00401013                 push    offset aHelloWorld ; "Hello world!"
.text:00401018                 call    ds:__imp__printf
.text:0040101E                 add     esp, 4
.text:00401021                 xor     eax, eax
.text:00401023                 pop     ebp
.text:00401024                 retn
.text:00401024 main            endp

Size 18,432 bytes

Configuration Properties > C/C++ > Code Generation > Buffer Security Check - No (/GS-
* No changes in code or size.

Project Name > Properties > Configuration Properties > Linker > General
> Enable Incremental Linking - No (/INCREMENTAL:NO)



//////////////////////////////////////////////////////////////////////////////
// The /INCREMENTAL option controls how the linker handles incremental linking.
// By default, the linker runs in incremental mode. To override a default 
// incremental link, specify /INCREMENTAL:NO. An incrementally linked program is 
// functionally equivalent to a program that is nonincrementally linked. However, 
// because it is prepared for subsequent incremental links, an incrementally 
// linked executable (.exe) file or dynamic-link library (DLL). Is larger than a 
// nonincrementally linked program because of padding of code and data. (Padding 
// allows the linker to increase the size of functions and data without recreating 
// the .exe file.) May contain jump thunks to handle relocation of functions to 
// new addresses. 
// Source: http://msdn.microsoft.com/en-us/library/4khtbfyf%28v=vs.80%29.aspx 
//////////////////////////////////////////////////////////////////////////////

.text:00401000 main            proc near               ; CODE XREF: __tmainCRTStartup+1BA p
.text:00401000                 push    ebp
.text:00401001                 mov     ebp, esp
.text:00401003                 push    offset aHelloWorld ; "Hello world!"
.text:00401008                 call    ds:__imp__printf
.text:0040100E                 add     esp, 4
.text:00401011                 xor     eax, eax
.text:00401013                 pop     ebp
.text:00401014                 retn
.text:00401014 main            endp

Size 8,192 bytes

Configuration Properties > C/C++ > Optimization - Minimum Size (/01)


//////////////////////////////////////////////////////////////////////////////
// The /O options control various optimizations that help you create code for
// maximum speed or minimum size. /O1 optimizes code for minimum size.
// Source: http://msdn.microsoft.com/en-us/library/k1ack8f1.aspx
//////////////////////////////////////////////////////////////////////////////

.text:00401000 main            proc near               ; CODE XREF: __tmainCRTStartup+1BA p
.text:00401000                 push    offset aHelloWorld ; "Hello world!"
.text:00401005                 call    ds:__imp__printf
.text:0040100B                 pop     ecx
.text:0040100C                 xor     eax, eax
.text:0040100E                 retn
.text:0040100E main            endp
.text:0040100E

Size 8,192 bytes

The Massive 2 mb Shellcode API Hash List

2011-06-29T20:46:00.000-07:00

I have been trying to get the Malware Analysis Search to include a page for searching Windows API Hashes. The ror hashing technique is quite popular in shellcode. I'm not going to write about the technique but there are plenty of great posts on it. I'd recommend Rolf's post on OpenRCE. I hashed (key values 0x0 through 0x1f) over 5k exported functions in advapi32.dll, gdi32.dll, kernel32.dll, netapi32.dll, ntdll.dll, shlwapi.dll, shlwapi.dll, user32.dll, wininet.dll, ws2_32.dll and ws2help.dll. Currently, I have had no luck with getting Google to index the page. So here is a Google spreadsheet link for anyone who would like a local copy. Just a warning the spreadsheet is 2 MB.

IDA MSDN Local Lookup

2011-03-02T19:53:00.000-08:00

Most of my reverse engineering is done offline. Having the Windows SDK Document Explorer (aka dexplorer.exe) installed is extremely helpful. Even when I'm online I prefer it over the MSDN library. Below is a python script I created so I can see the reference to an API in two clicks. The QT version of IDA has a tendency to add junk data to the copied string. I use the old version rather than the QT. Use at your own risk.


import subprocess
import sys
import idaapi
from win32clipboard import *
import win32api
# http://sourceforge.net/projects/pywin32/files/pywin32/

# NOTE Older version of IDA will crash if the child process of dexplorer.exe is not closed before closing out IDA. 
# The newer version of IDA will crash if subprocess.call(cmd) is used. 

# Usage load the python script, highlight the API, CTRL-C, then ALT-Z, Offline MSDN
# The API name has to be exact: GetAtomName corrent, GetAtomNameA Error 

def lz_msdn():
        #Right click on Windows SDK Documentaion > Properties > Open File Location. 
        dexplore_path = r'"C:\Program Files (x86)\Common Files\microsoft shared\Help 9\dexplore.exe"'
        #Open up dexplore.exe search for a random API, the url above the function title will have the namespace
        #Example ms-help://MS.W7SDK.1033/MS.W7SDKCOM.1033/dllproc/base/getprocaddress.htm, copy from "://" to '/' 
        namespace = "MS.W7SDK.1033/"

        #Copy the contents of the clipboard
        OpenClipboard() 
        api = GetClipboardData(win32con.CF_TEXT) # get clipboard data
        CloseClipboard()
 
 #Create dexplore cmd line
        cmd = dexplore_path  + " " + "/helpcol ms-help://" + namespace + " " + "/LaunchFKeywordTopic" + " " + '"' + api + '"'
        subprocess.Popen(cmd)
        print cmd
        
 
if __name__ == '__main__':
        if sys.platform == 'win32':
                from win32clipboard import *
                import win32gui, win32con
                idaapi.CompileLine('static altz() { RunPythonStatement("lz_msdn()"); }')
                AddHotkey("alt+z", 'altz')

An Intro to Creating Anti-Virus Signatures:

2011-01-17T19:44:00.001-08:00

This is an introductory post on creating anti-virus signatures. This post will cover the three main types of signature detections. The most common signatures are hashes, byte-signature and heuristics. The intended audience is for malware analyst and reverse engineers. Experience in binary code analysis is expected. This article is going to focus primarily on creating signatures for Microsoft Portable Executables. The ideas expressed in this post could also be used for creating signatures on other file types that contain exploits. The reason for this post is because there is very little information in regards to creating anti-virus signatures on the internet. A few articles exist on how to create signatures with ClamAV or using diffing tools. I'm hoping the information in this post will be helpful for others in aiding in creating signatures for classifying malware or detecting malicious documents. If you have any questions or would like more clarity please leave a comment.

To error on the side of caution.
There are a couple best practices for creating signatures. The first one is do not target packer or cryptography library code. This type of code is often reused and has a high potential for false positives. The code is usually complicated and is hard to understand the assembly representation. In situations where the packer is unique the code can be used to identify the family of malware. This type of detection has it's flaws. Detecting malware based off of packers gives very little insight into classifying multiple variants. The code could be reused across multiple variants or authors. Packed code creates a dilemma for signature detection. If the files have been packed or compressed, the file will need to uncompressed or dumped before scanned. Anti-virus engines use emulators and unpackers to get the files to an uncompressed or dumped state before scanning the file. If the files are compressed or packed tools such as Python, TitanEngine or the Immunity Debugger could be used for creating dumps or uncompressed files. A side effect of these tools is they will need to be run in a isolated environment such as VMware or Wine. ClamAV can also unpack some packers such as UPX, FSG, Petite and a couple others.

Data that has been obfuscated or compressed should never be used as a candidate for a signature. As in the case of file hashes such as MD5; changing one byte of data can change the obfuscated or compressed code. Since the bytes can be easily changed by different data or key, there is a chance that the data will not be present in other variants. An example would be if the code obfuscated a URL and then the author changes the URL. All of the code would be static but the bytes of the obfuscated URL would be different. Hence breaking the signature. This scenario is not only applicable for PE files but also compressed steams in PDFs, compressed SWFs and other compressed or obfuscated data. If there is data in a section of a Portable Executable (PE) that is encrypted and is static throughout multiple files, this would be a good candidate for a sectional MD5.

The second best practice is do not create signatures on code that can not be understood. Experience in reverse engineering and analyzing a wide range of code will help with choosing sections of code that will not give a false positive. If the code's function can not be understood along with what is calling code, it should not be used as a signature. This will help eliminate the possibility of false positives on library code and other commonly used code. IDA's Flirt signature is extremely useful for helping with identifying common compiler code. If developers of open source and closed source libraries created Flirt signatures and sent them to Hex-Rays they would never have an anti-virus software detect their code again. If the code can be understood then more analysis will be needed or another section of code should be targeted.

The third is look for the author's hand or internet. A couple of questions to ask in regards to the authors hand. Is the section of code being targeted applicable only towards this piece of malware or is it a common routine such as reading a registry setting? Would the author have written this piece of code for this piece of malware or have hundred authors written the same routine for different programs? This can be difficult to access. Google can be helpful in this situation. Sometimes googling a combination of the APIs can return examples of the code or the malware source code.

The fourth best practice is to make sure to understand the scope of the scanned files and implications of detection. If the signatures are scanning a repository of malware then the signatures can be vague. If the signature will be scanning desktops across an enterprise network and deleting the detected files, the signature must be accurate. It's always best to error on the side of caution when their is a chance that removing a file can have negative effect. Creating multiple signature for multiple variants is usually safer than creating a generic signature that has potential for false positives.

The final best practice, strings should be used as a last resort. Strings rarely give any details about the actual code. Strings and data are cheap, code is more expensive and time consuming. Code is more likely to stay static. In some situations such as executables written in Visual Basic strings are advantageous but overall code is best to target. Code and strings can always be combined to make a stronger signatures.

Tools
There are a couple of open source tools that can be used for scanning files and will be used for examples in this article. Once the initial learning curve is over, these tools are easy to use and can be adapted quickly to most environments. These tools can be run in a Linux or Windows environments. Below is the name of tool and some of it's scanning capabilities. All commands are included in the examples so the reader can try out the examples.

* ClamAV - Hex Byte Scanning, regex, md5 file scanning, md5 sectional scanning, sigtool ( tool for creating signatures and hashes )
* Yara - A powerful rule based scanner that supports many conditions and data types, does not support hashing
* ssdeep - A tool for creating and comparing context triggered piecewise hash.

Hash Signatures
The most basic and easiest type of signature is a hash value. A hash value is created by a hash function that is a procedure or mathematical function which converts a large amount of data into a single value. The most commonly used hash function is MD5 and SHA-1. These hash functions are extremely accurate. For example if there is block of data that is hashed and then the same block of data has a byte changed then rehashed the hash values will be different.


md5 of the string "WeBuiltThisCity!"  = "07363ec9aa7a39c28da675ee2291946b"
md5 of the string "WeBuiltThisCity" = "80b135388f2c979d53c229546c129a61"

Md5 based signatures can be created using ClamAV. Yara does not support file hashing. ClamAV requires two attributes in order to create a MD5 hash signature. The first is the file size in bytes and the second is the MD5 hash. ClamAV comes with a tool called sigtool that can be used to generate MD5 signatures. Sigtool can be found in the "bin" directory in the installation folder of ClamAV.


C:\XOR\clamwin\bin>sigtool.exe --md5 1.txt
07363ec9aa7a39c28da675ee2291946b:16:C:\XOR\1.txt

ClamAV signatures are separated with by a colon ':'. The first part is the MD5 (07363ec9aa7a39c28da675ee2291946b), the second is the size (16) and the last part is the file location or output. The output is usually the malware name or something specific to the file. Colons can not be used in the output because these are treated as special characters in ClamAV signatures database. MD5 signatures need to be saved in a file with a .hdb extension. The above output from sigtool could be piped ( > ) to a file called shredder.hdb. This makes it very easy to create MD5 signatures with all files in a directory using a simple for loop in bash or the window command line. See reference 1 for more information. This also makes it possible to create ClamAV signatures from Virustotal results without having the file because the MD5 and file size are provided in the Virustotal Analysis.

Hash values are useful if the malware sample is static but if one byte in the executable is changed; the hash signature is broken. In some instances the code of the executable never changes but the data that the executable uses does. An example of code that might not change would be the executable produced by a GUI Remote Access Trojan kit. The codes sections would be the same but if one user used 192.168.0.1 and another used 192.160.2 the data sections would be different. In this example sectional hashing could be used to target the codes section data block and create a hash based signature.


Example of a sectional hash using ClamAV:
PESectionSize:MD5:MalwareName
.code:80b135388f2c979d53c229546c129a61:Rocksteady

The signature will need to be saved in a file with a .mdb file extension in order to be in the proper ClamAV format. As stated before colons are treated as special characters for separating data in the ClamAV signature database. The first part is the PE section name (.code), the second is the MD5 hash of the section (80b135388f2c979d53c229546c129a61) and the last is the output or name of the malware (Rocksteady). See reference 1 for more information.

Hashing can also be used to identify files that have been modified. Fuzzy hashing is a combination of recursive computing and context triggered hashing. It can be used to identify files that have had data deleted, modified or new data inserted. Jesse Kornblum is the developer of a tool called ssdeep. Below is an example of comparing two executable files with a single byte difference.


C:\XOR>md5sum *
623eea5c4c6209e1ebe44b2e6ca16428 *simple - edit.exe
e300ef28554d39ee3668dca05d5d5415 *simple.exe

C:\XOR>ssdeep *
ssdeep,1.1--blocksize:hash:hash,filename
384:hJyMBCI1Ex85hKuwoK/ZiBqv5ICmOKk7iw:hJlC38wBY0,"C:\XOR\simple - edit.exe"
384:BJyMBCI1Ex85hKuwoK/ZiBqv5ICmOKk7iw:BJlC38wBY0,"C:\XOR\simple.exe"

C:\XOR\simple.exe matches C:\XOR\simple - edit.exe (99)

Fuzzy hashing requires all the hashes to be stored in an ascii text file and then each hashed file will need to be matched against the ascii hash text file. This type of scanning is more computational intensive than other forms of scanning due to the hashing and comparing of all the hash values. An issue with fuzzy hashing ( or all hashes for that matter ) is that what might seem like slight modifications by the author/programmer can noticeably change the calculated match value. Our simple.exe was modified by changing some of the output strings and recompiled as simple2.exe; no code was changed.


C:\XOR\ssdeep *
ssdeep,1.1--blocksize:hash:hash,filename
384:BJyMBCI1Ex85hKuwoK/ZiBqv5ICmOKk7iw:BJlC38wBY0,"C:\XOR\simple.exe"
384:5JTMJCv1w05hDuwoKoRiBqP5IqmuKk7iw:5JOCXrQQ0,"C:XOR\simple2.exe"

C:\XOR\diff-strings\New Folder>ssdeep
ssdeep: No input files

C:\XOR\diff-strings\New Folder>ssdeep -d *

C:\XOR\diff-strings\New Folder>ssdeep *
ssdeep,1.1--blocksize:hash:hash,filename
384:BJyMBCI1Ex85hKuwoK/ZiBqv5ICmOKk7iw:BJlC38wBY0,"C:\XOR\diff-strings\New Folde
r\simple.exe"
384:5JTMJCv1w05hDuwoKoRiBqP5IqmuKk7iw:5JOCXrQQ0,"C:\XOR\diff-strings\New Folder\
simple2.exe"

C:\XOR\>ssdeep simple.exe > out.txt
C:\XOR\>ssdeep simple2.exe > out2.txt

C:\XOR\>ssdeep -d out*
C:\XOR\out2.txt matches C:\XOR\out.txt (50)

Since all hashes need to be saved to a text file and then compared ssdeep can be computational expensive. See reference 2 for more information.

Byte-Signatures
Byte-signature or byte detections are a signature based off a sequence of file bytes that are present in a file or data stream. Byte signatures are a very common form of detection and have been used since the first anti-virus scanner. Their usefulness is due to the accuracy they provide for detecting a sequence of bytes. The sequence of bytes is chosen because it exist in multiple variants of malware from the same family. Byte-signatures can be any type of data such as code or data contained inside of a data stream of an executable, a XORed .pdf or a Word document.


Example of a byte signature in the ClamAV format.
Simple dot exe:1:90FF1683EE0483EB0175F6

"Simple dot exe" is the output displayed by the ClamAV scanner; this is usually the malware family name. The second section is for the ClavAV engine to know the file type of the scanned file. The value '1' is for the engine to scan portable executable files. To scan any file type the value '0' needs to be used. The "90FF1683EE0483EB0175F6" is the hexadecimal representation of the opcodes. It represents the below assembly.


start: 0x401A2E  length: 0xC
90     nop
FF 16     call    dword ptr [esi]
83 EE 04    sub     esi, 4
83 EB 01    sub     ebx, 1
75 F6     jnz     short loc_401A30

Example of a byte signature in the Yara format.
rule example
{
strings:
signature = { 66 90 FF 16 83 EE 04 83 EB 01 75 F6 }
condition:
signature
}

Yara's signature format is much different than ClamAV. The syntax style is similar to a C struct. The first string defines that we are creating a rule of "example". The "strings" and "condition" are keywords used by the Yara engine. The strings defines the signatures. Yara's signatures can be any many formats from strings, hex-bytes, regex and a number of other formats. The keyword "condition" defines under what circumstances Yara should alert on the signature. There are many conditions that can be constructed for Yara to detect a file. See reference 6 for more information.

Binary Diffing
Manual and automated analysis techniques can be used for finding code blocks that are present throughout variants. The process of comparing multiple executables for similarities is called binary diffing. Manual binary diffing consists of reviewing the disassembly of multiple files and noting sections of code that are present in multiple files. Once blocks of code are identified a side-by-side comparison can be done checking for similarities. If the code has a high similarity it might be a good candidate for a byte-signature. A semi-automated approach of binary diffing would consist of grouping files into sets of files that are similar, use a disassembler to to get the assembly in a text output, diff the outputs, create more specific sets for diffs that match and then manually analyze the sections of codes that were found in the diffs that matched. For example, we could use IDA to create the assembly output and then use Kdiff to diff the assembly outputs.


C:\XOR\>dir
     23,503 simple.exe
     23,503 simple2.exe

C:\XOR\>"C:\Program Files\IDA Free\idag.exe" -B "simple.exe"

C:\XOR\>"C:\Program Files\IDA Free\idag.exe" -B "simple2.exe"

C:\XOR\>dir
     51,262 simple.asm
     23,503 simple.exe
    204,956 simple.idb
     51,274 simple2.asm
     23,503 simple2.exe
    204,956 simple2.idb

The "-B" flag is for creating an IDA database (.idb) and a text output of the assembly (.asm). The assembly output can then be diffed. Kdiff is a good tool to diff files because it does not do a straight diff of the two files. It will try to align sections that our separated by large amounts of data or code. These sections could be different because of a function that wasn't included in one executable or the presence of junk code or data. KDiff can also diff up to three files at a time. Automated tools such as Bindiff, PatchDiff2 and DarunGrim can be used to find differences between IDA database files. These tools are more specifically designed towards binary diffing paths. VxClass is another tool that can be used for creating sets of variants, binary diffing and identifying sections of code that could be used for potential signatures **.

Diffing is also very useful for displaying sections of code where wildcards will need to be used for byte-signatures. The wildcards will allow the scanner to ignore bytes of code that are not static throughout multiple variants. The non-static bytes could be caused by the insertion of junk code, addition or removal of code or the addition or removal of data. A slight change in file alignment can break byte-signatures that targets instructions. Common instructions that use offsets based off of file alignment are long jmps, call sub-routine or call an api or others that cmp, mov, sub, add, etc that reference offsets. It's best when creating signatures to always add the wildcards for the offsets even if they are static throughout multiple variants. An example below can be seen how file alignment can break byte-signatures. The two code blocks are from simple.exe and simple2.exe. The starting codes address, functionality and length are exactly the same. The only difference between the two blocks of code are the address offsets of the dword that is compared against ebx. The difference in file alignment was caused by the output strings being a different length.


simple.exe
start: 0x401902  length: 0x14
81 FB 8C 31 40 00   cmp     ebx, offset dword_40318C
0F 83 3A FF FF FF   jnb     loc_401848
BE 00 00 40 00    mov     esi, 400000h
8D 7D E0    lea     edi, [ebp+var_20]

81 FB 8C 31 40 00 0F 83 3A FF FF FF BE 00 00 40 00 8D 7D E0

simple2.exe
start: 0x401902  length: 0x14
81 FB 94 31 40 00   cmp     ebx, offset dword_403194
0F 83 3A FF FF FF   jnb     loc_401848
BE 00 00 40 00    mov     esi, 400000h
8D 7D E0    lea     edi, [ebp+var_20]

81 FB 94 31 40 00 0F 83 3A FF FF FF BE 00 00 40 00 8D 7D E0

ClamAV Format
Both Simple dot exe:1:81FB********0F833AFFFFFFBE000040008D7DE0

Yara Format
rule both_simple_dot_exe
{
strings:
signature = { 81 FB ?? ?? ?? ?? 0F 83 3A FF FF FF BE 00 00 40 00 8D 7D E0 }
condition:
signature
}

Heuristics
The last type of signature detections is heuristics. Heuristics is used when the malware is too complex for hash and byte-signatures. Heuristics is a general term for the different techniques used to detect malware by their behavior. It is one of the most complex forms of detections. An anti-virus engine might use emulation, API hooking, sand-boxing, file anomalies and other analysis techniques. Each anti-virus engine uses different algorithms and different proprietary techniques. A simple example of creating a heuristics signature would include an API logger and rules based off the APIs. Let's start with the "Hello World" of malware, Poison Ivy. We can use a server as an example for creating an API rule based signature. When Poison Ivy is executed it will create a mutex, write a registry key and copy itself over to the System32 directory. The default mutex for Poison Ivy is ")!VoqA.I4". Using Kerberos API Monitor we can simulate an API hook (that an anti-virus engine might use). A heuristics API rule for Poison Ivy can be seen below.


Rule A
An API call to RtlMoveMemory with a string of "SOFTWARE\Classes\http\shell\open\commandV"

Rule B
An API call to CreateMutexA with a string of ")!VoqA.I4"

Rule C
An API call to GetSystemDirectory

if ( Rule A then Rule B then Rule C )
then
Process  = PoisonIvy

Keribos Output
Rule A
simple.exe   | 00401447 | RtlMoveMemory(0012F458, 0040162F: "SOFTWARE\Classes\http\shell\open\commandV", 00000028) returns: 0012F458
...........
Rule B
simple.exe   | 0040155D | CreateMutexA(00000000, 00000000, 0012F43B: ")!VoqA.I4") returns: 0000003C
...........
Rule C
simple.exe   | 004018BF | GetSystemDirectoryA(0012F6F1, 000000FF) returns: 00000013

Parsing API logs from tools such as Norman and CWS Sandbox could be used to create similar rules for malware that is packed, encrypted or hash or byte-signatures fail to detect.

Conclusion
This post is just an introductory to creating anti-virus signatures. I am by no means an expert in the different technologies and algorithms used for scanning files. My experience is limited to hashing, byte-signatures, memory offset byte-signatures and creating detection mechanisms for malicious documents. I hope this article was useful and might give some ideas for other reverse engineers and malware analyst. If you know of any good articles on anti-virus scanning and the algorithms please leave a comment. I'd like to thank b0ne for all his help over the years.

References:
1. www.clamav.net/doc/latest/signatures.pdf
2. http://dfrws.org/2006/proceedings/12-Kornblum.pdf
3. http://www.virustotal.com/file-scan/report.html?id=fab272012d934f75915cd888f213e8857c390086363351eab3bf69f19ce67b65-1292244815
4. http://www.zynamics.com/bindiff.html, http://code.google.com/p/patchdiff2/, http://www.darungrim.org/
5. http://kdiff3.sourceforge.net/
6. http://code.google.com/p/yara-project/downloads/list
** The author has never personally used VxClass. This information is second hand from Zynamics blog and site.

Malware Analysis Search

2010-11-09T18:19:00.001-08:00

This is a custom Google that searches anti-virus analysis pages, malware analysis blogs and other related malware/RCE websites. Currently about 75 different sites are used. This helps with removing all the clutter of forums and other useless search results. Sometimes when looking at malware I want to know if someone else has already analyzed it. Hopefully this will be helpful.

Easily Memorable Google Hosted Link (Thanks Google!)
http://www.google.com/cse/home?cx=011750002002865445766:pc60zx1rliu

Currently the following sites are being searched.
http://xml.ssdsandbox.net/archive/
http://www.threatexpert.com/report.aspx?
http://www.virustotal.com/file-scan/
http://blog.fireeye.com/
http://blogs.technet.com/b/mmpc/
http://www.microsoft.com/security/portal/Threat/Encyclopedia/
http://vrt-sourcefire.blogspot.com/
http://community.websense.com/blogs/securitylabs/
http://blog.scansafe.com/
http://www.f-secure.com/weblog/
http://www.f-secure.com/v-descs/
http://blog.fortinet.com/
http://www.fortiguard.com/encyclopedia/virus/
http://www.securelist.com/en/
http://www.prevx.com/blog.asp
http://research.pandasecurity.com/
http://www.pandasecurity.com/homeusers/security-info/about-malware/encyclopedia/
‪http://www.avira.com/en/support-threats-summary/‬
‪http://techblog.avira.com/en/‬
‪http://eureka.cyber-ta.org/‬
‪http://twitter.com/taviso/‬
‪http://twitter.com/sans_isc/‬
‪http://twitter.com/RolfRolles/‬
‪http://twitter.com/rcecoder/‬
‪http://twitter.com/pedramamini/‬
‪http://twitter.com/OComputing/‬
‪http://twitter.com/ochsff/‬
‪http://twitter.com/nicolasbrulez/‬
‪http://twitter.com/nickharbour/‬
‪http://twitter.com/msuiche/‬
‪http://twitter.com/mlsau/‬
‪http://twitter.com/mikkohypponen/‬
‪http://twitter.com/mdowd/‬
‪http://twitter.com/jvanegue/‬
‪http://twitter.com/j00ru/‬
‪http://twitter.com/Ivanlef0u/‬
‪http://twitter.com/hdmoore/‬
‪http://twitter.com/halvarflake/‬
‪http://twitter.com/erocarrera/‬
‪http://twitter.com/DidierStevens/‬
‪http://twitter.com/egyp7/‬
‪http://twitter.com/dinodaizovi/‬
‪http://twitter.com/codypierce/‬
‪http://twitter.com/attractr/‬
‪http://twitter.com/alexsotirov/‬
‪http://bugix-security.blogspot.com/‬
‪http://seclists.org/#fulldisclosure‬
‪http://blog.trendmicro.com/‬
‪http://www.exploit-db.com/‬
‪http://xanalysis.blogspot.com/‬
‪http://research.zscaler.com/‬
‪http://jsunpack.blogspot.com/‬
‪http://www.sophos.com/security/analyses/‬
‪http://www.symantec.com/security_response/‬
‪http://www.symantec.com/connect/blogs/‬
‪http://vil.nai.com/vil/content/‬
‪http://blogs.mcafee.com/mcafee-labs/‬

Reviewing the links you might have noticed a lot of twitter feeds. These are useful for finding information on exploits or 0days. I just started ripping through my RSS feed and will be adding more sites as I come across them. If I'm missing something please leave a comment.

Give it a shot.

Malware Analysis Search

Creating Your Own Virustotal..Well Kind Of..Ok, Not Really

2010-04-28T19:59:00.000-07:00

Virustotal is free website that allows individual to upload files to be scanned by 41 anti-virus engines and file identification tools. Virustotal was created in 2004 by Hispasec Sistemas. Virustotal is a very valuable tool for many system admins, technical support engineers or anyone else who is curious if a file malicious. The main downside to Virustotal is that you have to submit your samples through a web interface or email. Sometimes it's nicer to have a quick report from the command line but in certain situations you can't submit the samples.

This post focuses on analyzing Microsoft Portable Executable (PE) files using Python, PeFile, PEID and a command line anti-virus scanner. This post will display the Python code for creating the PE information displayed under the anti-Virus scanner results on Virustotal results page. The post also will contain the code to call and display the results for an anti-virus command line scanner.

Python still feels new to me so please feel free to email me or leave a comment if you have any recommendations or advice on the code. The full source code can be found here.

Let's start at the top of the code with the imports.


import sys
import os
import pefile
import peutils
import math
import time
import datetime
import subprocess

The only non-standard modules listed above are Pefile and Peutils. These were created by Ero Carrea. Both of these modules can be found here. For anyone new to Python, modules are files that contain statements and definitions that can be used to import functionality. Please see this article for more details.

Below is the first function that will be called. The function "attributes" extracts basic information from the Portable Executable. The Portable Executable is a file format that the Windows operating system uses to encapsulates data and code. The Windows OS Loader uses the data structure to manage the wrapped executable code.


## Print PE file attributes
def attributes():  
        print "Image Base:", hex(pe.OPTIONAL_HEADER.ImageBase)
        print "Address Of Entry Point:", hex(pe.OPTIONAL_HEADER.AddressOfEntryPoint)
        machine = 0
        machine = pe.FILE_HEADER.Machine
        print "Required CPU type:", pefile.MACHINE_TYPE[machine]
        dll = pe.FILE_HEADER.IMAGE_FILE_DLL
        print "DLL:", dll
        print "Subsystem:", pefile.SUBSYSTEM_TYPE[pe.OPTIONAL_HEADER.Subsystem]
        print "Compile Time:",   datetime.datetime.fromtimestamp(pe.FILE_HEADER.TimeDateStamp)
        print "Number of RVA and Sizes:", pe.OPTIONAL_HEADER.NumberOfRvaAndSizes

Above we are using PeFile to read and display data from the PE format. Before using Pefile the executable will need to be loaded into memory. This will be covered when Main() is discussed.

Output:


Portable Executable Information
Optional Header: 0x400000
Address Of Entry Point: 0x78020
Required CPU type: IMAGE_FILE_MACHINE_I386
DLL: False
Subsystem: IMAGE_SUBSYSTEM_WINDOWS_GUI
Compile Time: 2007-04-29 05:43:12
Number of RVA and Sizes: 16

Attributes from the PE can give information about the executable that can be valuable when analyzing an unknown executable. The first output line is the Image Base of the executable. The default value for an application is 0x00400000 and 0x10000000 for a Dll. A value that is non-standard could be a possible flag and should be noted.

The second output is the address of the entry point. The entry point is the starting address for the executable, address of the initialization function for device drivers and the entry point is optional for a Dll. The standard value for the entry point is 0x01000. If the value is non-standard it should be noted because the entry point could have possibly been changed by a packer or obfuscation tool.

The third output is the Machine Type. This can be used to identify if the file is 32-bit or 64-bit. If the Machine Type value is IMAGE_FILE_MACHINE_I386 then executable is 32-bit. To identify a 64-bit executable the Machine Type value would be IMAGE_FILE_MACHINE_AMD64 or IMAGE_FILE_MACHINE_IA64.

The fourth output simply identifies if the executable is DLL or not.

The fifth output identifies the subsystem type. In the example above IMAGE_SUBSYSTEM_WINDOWS_GUI identifies that the executable has a Windows GUI. If the subsystem was IMAGE_SUBSYSTEM_WINDOWS_CUI it would mean the executable is a console application. The third common subsystem type is IMAGE_SUBSYSTEM_NATIVE. This is reserved for drivers and native system processes. Just a side not to idenitfy a Xbox executable the subsystem type would be IMAGE_SUBSYSTEM_XBOX.

The sixth line of output is the time the executable was compiled.

The last line of output displays the NumberOfRvaAndSizes. The default value for this attribute is 0x10 or 16 in decimal. Modification of this value can be used to crash Ollydbg. If the value is non-standard and Ollydbg gives an error patching this field might be needed. These values are helpful because they give clues if there is anything non-standard about the executable. Non-standard values could be from a packer, some code obfuscation tool or by the programmer passing the compiler flags.


def sections_analysis():
        print "Number of Sections:", pe.FILE_HEADER.NumberOfSections
        print
        print "Section  VirtualAddress VirtualSize SizeofRawData Entropy"
        for section in pe.sections:
                print "%-8s"  % section.Name, "%-14s" % hex(section.VirtualAddress), "%-11s" % hex(section.Misc_VirtualSize),\
                      "%-13s" % section.SizeOfRawData, "%.2f" % E(section.data)
        print

The next function called is "sections_analysis". This sections outputs the name of the sections, it's virtual address, it's virtual size, size of raw data and it's entropy. This information is valuable because certain packers will modify these attributes. Packers will sometime rename sections, create new sections, or modify other attributes. For a detailed analysis of how packer work please see Websense's "The History of Packing Technology".

Output:


Number of Sections: 3

Section  VirtualAddress VirtualSize SizeofRawData Entropy
UPX0     0x1000               0x42000     0                          0.00
UPX1     0x43000            0x36000     217600               7.93
.rsrc        0x79000            0x2000       7680                     4.00

The above output gives hints to what type of packer was used just be reviewing the section names. In the case of the test executable, Putty.exe was compressed using UPX. The string UPX can be observed in the section names. Two other attributes that should be mentioned is the zero size of the raw data of UPX0 and the high entropy of the section UPX1. During execution of a packed executable UPX will need to write the uncompressed data to a section. The section UPX0 with a size of zero will be used for storing the uncompressed data. Not all packers will use this technique but this is another characteristics that should be noted.


## Entropy calculation from Ero Carrera's blog ###############
def E(data):
        entropy = 0  
        if not data:
                return 0
        ent = 0
        for x in range(256):
                p_x = float(data.count(chr(x)))/len(data)
                if p_x > 0:
                        entropy += - p_x*math.log(p_x, 2)
        return entropy

Entropy is a measurement of how organized or disorganized data is. The more random the data is the higher the entropy will be. Packers will apply an algorithm to either compress the data or obfuscate it. The output of the packed data has a higher entropy and is more random than the original compiled executable. The entropy range for the "E" function ranges between 0.0 and 8.0. The closer the entropy is to 8.0 the higher the chances that the section is packed or obfuscated. Please see here for more details.


## Load PEID userdb.txt database and scan file
def PEID():
        signatures = peutils.SignatureDatabase('userdb.txt')
        matches = signatures.match_all(pe,ep_only = True)
        print "PEID Signature Match(es): ", matches
        print

PEID is tool for detecting common packers, cryptors and compilers for PE files. By identifying the algorithm the PE file is compressed or obfuscated it can help speed up analysis. Google, the name of the packer and the string "tutorial" will usually return an analysis on how to unpack the file. Pefile has functionality to use PEID's user database (userdb.txt) to scan a PE file. The main PEID user database was created by BobSoft and can be downloaded here. Many individuals use Panda Anti-Virus userdb.txt. A warning to users of Panda's userdb.txt. Pefile will throw exceptions and will not work due to some non-standard characters. It's recommended to use BobSoft's userdb.txt.

Output:


PEID Signature Matche(s):  [['UPX 2.90 [LZMA] -> Markus Oberhumer, Laszlo Molnar & John Reiser']]

The last function is called IAT(). This function will display all imported Dlls and the imported API name.


## Dump Imports
def IAT():
        print "Imported DLLS:"
        i = 1
        for entry in pe.DIRECTORY_ENTRY_IMPORT:
                bool = 1 ## For Formattting 
                print "%2s" % [i], "%-17s" % entry.dll
                print "\t",
                for imp in entry.imports:
                        if bool:
                                print "%-1s" % imp.name,
                                bool = 0
                        else:
                                sys.stdout.write("%s%s" % (", ",imp.name)) # Python Print adds a blank space 
                print
                i += 1

Reviewing all API calls can help with giving a high level view of the behavior of the executable. If the APIs list is sparse this might be a sign that the executable has had it's Import table removed. The standard "Hello, World" complied in C using LCC would contain 19 API names. If there are only three APIs listed such as LoadLibrary, GetProcAddress and ExitProcess odds are the file is packed or obfuscated in some manner.

Output:


Imported DLLS:
[1] KERNEL32.DLL     
 LoadLibraryA, GetProcAddress, ExitProcess
[2] ADVAPI32.dll     
 RegCloseKey
[3] COMCTL32.dll     
 None
[4] comdlg32.dll     
 ChooseFontA
[5] GDI32.dll        
 LineTo
[6] IMM32.dll        
 ImmGetContext
[7] SHELL32.dll      
 ShellExecuteA
[8] USER32.dll       
 GetDC
[9] WINMM.dll        
 PlaySoundA
[10] WINSPOOL.DRV     
 WritePrinter

The final function calls a local command line anti-virus scanner.


## Print Sophos
def sophos(filetmp):
        print "Sophos Scan in progress.."
        output = "None"
        path = os.path.abspath(filetmp)
        pwd = os.getcwd()
        output = subprocess.call([os.path.join(pwd, 'cmd_scan', 'Sophos', 'SAV32CLI.EXE'), path])

Output:


Sophos Scan in progress..
Sophos Anti-Virus
Version 4.52.0 [Win32/Intel]
Virus data version 4.52E, April 2010
Includes detection for 1542982 viruses, trojans and worms
Copyright (c) 1989-2010 Sophos Plc. All rights reserved.

System time 21:36:32, System date 03 May 2010

Quick Scanning

1 file swept in 3 seconds.
No viruses were discovered.
Ending Sophos Anti-Virus.

This function will need to be copied and modified for each scanner. There are a few free anti-virus scanners. The function above used Sophos's free command line scanner. The scanner can be downloaded here. Other free scanner are ClamAV, Panda and a couple of others. ClamAV can be used as a portable application and does not need to be fully installed. The portable ClamAV can be downloaded here. Most purchased anti-virus applications do have command line interface.


if len(sys.argv) < 2:
        print "Pyton Script "
        sys.exit(3)
exename = sys.argv[1]
pe = pefile.PE(exename)
print "\nPortable Executable Information"
attributes()
sections_analysis()
PEID()
IAT()
sophos(exename)

##   <- Format bug with SyntaxHighlighter (remove line)

This script will need to have a Portable executable file passed to it. Once a file is passed Pefile will load the executable and then call all the functions.

In closing, this script is a quick example of using Python to analyze an unknown executable file. The current version is 0.01. Overtime I'll keep updating the script. As previously stated if there are any question or ideas please leave a comment.

01010101

2010-04-10T15:04:00.000-07:00

Ones and zeros is a very popular term for computers. Unfortunately most individuals do not know how those numbers equate to actual programming instructions or data. Below is a quick walk through of how those ones and zeros are interpreted.

Binary
.code      
01010101 10001001 11100101 10000011 11101100 00001000 10000011 11100100 11110000 10111000 00000000 00000000 00000000 00000000 10000011 11000000 00001111 10000011 11000000 00001111 11000001 11101000 00000100 11000001 11100000 00000100 10001001 01000101 11111100 10001001 01000101 11111100 11101000 00011011 00000101 00000000 00000000 11101000 11010110 00000001 00000000 00000000 11000111 00000100 00100100 00000000 00110000 01000000 00000000 11101000 01110010 00000101 00000000 00000000 11001001 11000011 
.data 
01001001 00100000 01001100 01101111 01110110 01100101 00100000 01011001 01101111 01110101

Binary is a base-2 numeral counting system. The base-2 means that the binary number can be represented by two mutually exclusive states. These states can be for example either On or Off, as in the case of a light bulb. Each one and zero above represents one of those states. A simple way to think about the data above is to visualize 528 light bulbs in a row either off or on depending on their state.

Binary numbers can be used to represent integers (4,234,-6) rather than just being for a single one or zero state. For example, the first eight binary digits above 01010101 is equal to 85 in decimal (base-10 or using your fingers). Here is the classical arthritic for converting binary to decimal: [(0) × 2^7] + [(1) × 2^6] +[(0) × 2^5] + [(1) × 2^4] + [(0) × 2^3] + [(1) × 2^2] + [(0) × 2^1] + [(1) × 2^0] = 85. Basically what this means is that we can use ones and zeros to represent an infinite set of integers.

In the early years of computing, binary numbers were used to program in machine code. This is no longer the case.

Hexadecimal


.code
85 89 E5 83 EC 08 83 E4 F0 B8 00 00 00 00 83 C0 0F 83 C0 0F C1 E8 04 C1 E0 04 89 45 FC 89 45 FC E8 1B 05 00 00 E8 D6 01 00 00 C7 04 24 00 30 40 
00 E8 72 05 00 00 C9 C3
.data
49 20 4C 6F 76 65 20 59 6F 75

Programming in binary (also called machine code) is an extremely tedious process. In order to simplify this process mnemonic codes were used rather than binary digits. Mneomic codes are helpful because they are closer to human language. Mneomic codes are represented in the numerical format of hexadecimal.

Hexadecimal is a base-16 numeral format. Rather than having two states as in the case of binary hexadecimal can have 16 states. The different states are represented by the digits 1,2,3,4...9,A,B,C,D,E,F. The digit 'E' equals 15 in decimal. Hexadecimal is easy for computers because 2 (on or off) to the power of 4 equals 16. Hexadecimal is visually easier to read than binary. 01010101 binary equals 55 hexadecimal.

Assembly Language


.code
PUSH EBP
MOV EBP,ESP
SUB ESP,8
AND ESP,FFFFFFF0
MOV EAX,0
ADD EAX,0F
ADD EAX,0F
SHR EAX,4
SHL EAX,4
MOV [LOCAL.1],EAX
MOV EAX,[LOCAL.1]
CALL love.00401820
CALL love.004014E0
MOV DWORD PTR SS:[ESP],love.00403000     ; |ASCII "I Love You."
CALL                 ; \printf
LEAVE
RETN
.data
I Love You.

Converting hexadecimal to mnemonics code is done by a disassembler. When a disassembler converts hexadecimal to mnemonics code the output is assembly language. The hexadecimal has to be valid mnemonics code in order to be converted to assembly language. The hexadecimal to mnemonic code conversion is defined by what processor the computer using. In the case about the assembly language was been defined by Intel (x86).

The ".code" indicates that the hexadecimal is executable and not data. Example of executable code would be "MOV EBP,ESP". This assembly language means move ESP into EBP. The CPU would interrupt this assembly language as a doable action. The ".data" indicates that the hexadecimal is data. An example of data would be the phrase "I Love You.".

C Programming Language


#include

main()
{
    printf("I Love You"); 
}

Assembly Language is considered a low level programming language. This is due to it's close interaction with the CPU. Interaction at this level is extremely complicated. To ease the complexity higher level programming languages were created. An example of this can be seen in the code above written in the C programming language. Three lines of code in C is equal to 17 lines of assembly.

In order to get from C to the binary digits the code would need to be compiled. A compiler converts source code to a binary format. The final output that the user would see if they clicked on the compiled C code.

I Love You. _

To recap the computer at it's lowest level stores everything as ones and zeros, the source code is compiled to create the binary, those ones and zeros can be read as hexadecimal, the hexadecimal can then be converted to either assembly language or data.

2010-04-10T14:59:00.000-07:00


 int main() 
 {
        printf("hello, world");
        return 0;
 }