Experiment Tutorial

This experiment tutorial help you to finish the evaluation described by this paper.

It’s similar with the evaluation section Chucky: Exposing Missing Checks in Source Code for Vulnerability Discovery, but the ROC curves are generated by the middle result(The rank lists of similar functions).

To do the experiment, you should do the following steps:

  1. Generate the code database.
  2. Modify the code.
  3. Run the automatic script.

Generate the Database

The database can be generated by joern(2.0-3.0) according to the method Fabian described in Chucky paper. That is, patch the vulnerability as the original version, then remove one check in one function from the original versions in a round robin fashion to generate such many code versions and then use joern to generate the code graph database for each vulnerable version. The version and the respective vulnerability number are listed below.

Project Vulnerability Declaration Type Symbol TYPE #With Check #Symbol Users #F LOC
firefox-4.0(/js) CVE-2010-3183 uintN argc parameter 10 557 5649 372450
linux-2.6.34.13(/fs) CVE-2010-2071 struct dentry* dentry parameter 8 1104 19178 955943
libpng-1.2.44 CVE-2011-2692 png_uint_32 length parameter 19 29 473 40255
libtiff-3.9.4 CVE-2010-2067 TIFFDirEntry* dir parameter 9 75 609 332762
pidgin-2.7.3(/libpurple) CVE-2010-3711
purple_base64_decode callee 18 30 7390 332762

Modify the Code

  1. Remove the # symbol at the head of the two lines in the try block of function analyze():

    #for n in nearestNeighbors:
#    print str(n)+"\t"+n.location()

  2. Comment out all the following code in try block(that means we just print the neighborhood selection result).

  3. Define the environment variable $NEO4J_HOME to point it to your neo4j program directory.

  4. Change the variable cfgfile in the script file neighbor to the absolute location of the configuration file neo4j-server.properties.

  5. change the variable line in neighbor to the line of variable org.neo4j.server.database.location in the configuration file conf/neo4j-server.properties of your Neo4j database.

    line=11
  1. Change the value of the dbpath to the location of all of your database.Note that the directory must be organized as $dbpath/$projname/$funcname/.joernIndex. The projenames and funcnames must be equal to the names listed in the script file neighbor.

Run the Auto-Script

$ cd chucky-ng/chucky
$ neighbor
$ python ROC.py

The shell script neighbor dump the result of KNN algorithm to the current file directory, then the ROC.py read the directory and generate the points in the directory named ROC.

Output Hierarchy

  • The directory neighbors output by script neighbor will hold the hierarchy $neighbors/$projname/$function_name, for example, neighbors/libpng/png_handle_cHRM.
  • The final ROC points will be generated in file ROC/$projname-neighbors_ROC, for example, ROC/libpng-neighbors_ROC).

At last, you can import these files of ROC point lists into drawing program to plot the diagram.

Details About the 64 Function

Here is the detail information about the 64 function for evaluation.

Firefox-4.0

Order Function Location
1 array_concat js/src/jsarray.cpp
2 array_extra js/src/jsarray.cpp
3 array_indexOfHelper js/src/jsarray.cpp
4 array_slice js/src/jsarray.cpp
5 array_splice js/src/jsarray.cpp
6 array_unshift js/src/jsarray.cpp
7 js::array_sort js/src/jsarray.cpp
8 LookupGetterOrSetter js/src/xpconnect/src/xpcquickstubs.cpp
9 DefineGetterOrSetter js/src/xpconnect/src/xpcquickstubs.cpp
10 PropertyOpForwarder js/src/xpconnect/src/xpcquickstubs.cpp

linux-2.6.34.13

Order Function Location
1 btrfs_xattr_acl_set fs/btrfs/acl.c
2 jffs2_acl_setxattr fs/jffs2/acl.c
3 ext2_xattr_set_acl fs/ext2/acl.c
4 ext3_xattr_set_acl fs/ext3/acl.c
5 ext4_xattr_set_acl fs/ext4/acl.c
6 ocfs2_xattr_acl_set fs/ocfs2/acl.c
7 generic_acl_set fs/generic_acl.c
8 posix_acl_set fs/reiserfs/xattr_acl.c

libpng-1.2.44

Order Function Location
1 png_handle_Bkgd pngrutil.c
2 png_handle_cHRM pngrutil.c
3 png_handle_gAMA pngrutil.c
4 png_handle_iCCP pngrutil.c
5 png_handle_IEND pngrutil.c
6 png_handle_IHDR pngrutil.c
7 png_handle_iTXt pngrutil.c
8 png_handle_oFFs pngrutil.c
9 png_handle_pHYs pngrutil.c
10 png_handle_PLTE pngrutil.c
11 png_handle_sBIT pngrutil.c
12 png_handle_sCAL pngrutil.c
13 png_handle_sPLT pngrutil.c
14 png_handle_sRGB pngrutil.c
15 png_handle_tEXt pngrutil.c
16 png_handle_tIME pngrutil.c
17 png_handle_tRNS pngrutil.c
18 png_handle_unknown pngrutil.c
19 png_handle_zTXt pngrutil.c

tiff-3.9.4

Order Function Location
1 TIFFFetchByteArray libtiff/tif_dirread.c
2 TIFFFetchLongArray libtiff/tif_dirread.c
3 TIFFFetchPerSampleAnys libtiff/tif_dirread.c
4 TIFFFetchPerSampleLongs libtiff/tif_dirread.c
5 TIFFFetchPerSampleShorts libtiff/tif_dirread.c
6 TIFFFetchShortArray libtiff/tif_dirread.c
7 TIFFFetchShortPair libtiff/tif_dirread.c
8 TIFFFetchString libtiff/tif_dirread.c
9 TIFFFetchSubjectDistance libtiff/tif_dirread.c

Pidgin-2.7.3

Order Function Location
1 digest_md5_handle_chanllenge lipurple/protocols/jabber/auth_digest_md5.c
2 do_buddy_avatar_update_data lipurple/protocols/jabber/useravatar.c
3 got_sessionreq lipurple/protocols/msn/slp.c
4 jabber_data_create_from_xml lipurple/protocols/jabber/data.c
5 jabber_ibb_parse lipurple/protocols/jabber/ibb.c
6 jabber_scram_feed_parser lipurple/protocols/jabber/auth_scram.c
7 jabber_vcard_parse lipurple/protocols/jabber/buddy.c
8 jabber_vcard_parse_avatar lipurple/protocols/jabber/presence.c
9 jabber_vacard_save_mine lipurple/protocols/jabber/buddy.c
10 msim_msg_get_binary_from_element lipurple/protocols/myspace/message.c
11 msn_oim_report_to_user lipurple/protocols/msn/oim.c
12 msn_switchboard_shoe_ink lipurple/protocols/msn/switchboard.c
13 purple_mime_decode_field lipurple/util.c
14 purple_ntlm_parse_type2 lipurple/ntlm.c
15 scram_handle_challenge lipurple/protocols/jabber/auth_scram.c
16 scram_handle_success lipurple/protocols/jabber/auth_scram.c
17 yahoo_process_p2p lipurple/protocols/yahoo/libymsg.c
18 yahoo_process_status lipurple/protocols/yahoo/libymsg.c