Skip to content

CGC Vulnerability Identification Progress

Jump to bottom Edit New page
kingloko edited this page Feb 4, 2018 · 1 revision

Introduction

This page tracks our progress in working towards full identification of the vulnerabilities found in DARPA's Cyber Grand Challenge dataset. A huge thank you to Trail of Bits for porting these challenge binaries from DECREE to Linux, macOS, and Windows!

Testing Queries

If you want to test these queries, you have two options.

  • Compile your own cb-multios binaries and analyze them with PM
  • Use our compiled cb-multios binaries and our pre-analyzed data set

If you want to use our data. You can download the cb-multios binaries from this link (239 MB), and download analysis files from our open S3 directory.

If you use our data and your own binaries to verify queries, you'll likely encounter discrepancies if you used a different version of Clang to compile. So we suggest using your own analysis and binaries, or using both our analysis and our binaries.

Common Weakness Enumeration

The DARPA Cyber Grand Challenge (CGC) performers adhered to a very convenient format when documenting their challenge binaries (CB). Not only do they make it clear what the vulnerabilities are in the CBs, they tell you where they are, the challenges involved in automated discovery of these vulnerabilities, and any applicable CWEs.

CWEs are MITRE's Common Weakness Enumeration Check out of a Use After Free (UAF) vulnerability class at CWE-416. The GCG CB "FileSys" is an example of a CB with this vulnerability.

DARPA Cyber Grand Challenge Dataset

The list below is broken down into CWEs and any relevant DARPA Cyber Grand Challenge CBs. As we test new queries, we'll run them against all related CBs in the dataset and see which ones it finds, and which ones it doesn't. This will give us a sense of how well Paper Machete is doing against these binaries.

Performing false positive testing on the entire dataset is extremely important to us, but for the time being, we're focusing on writing the first pass of queries and tracking how well they do against problems they should work on.

The following sections list the CWEs, related CGC challenge binaries, the query that finds the vulnerability, and a link to the cb-multios ported challenge problem.

CWE-120: Buffer Copy without Checking Size of Input

"The program copies an input buffer to an output buffer without verifying that the size of the input buffer is less than the size of the output buffer, leading to a buffer overflow."

Challenge Binary Found? Query / Failure reason
AIS-Lite Untested n/a
basic_messaging Untested n/a
CGC_Planet_Markup_Language_Parser Untested n/a
CGC_Video_Format_Parser_and_Viewer Untested n/a
Charter Untested n/a
Eddy Untested n/a
FSK_Messaging_Service Untested n/a
GPS_Tracker Untested n/a
hawaii_sets Untested n/a
HeartThrob Untested n/a
Mixology Untested n/a
Monster_Game Untested n/a
Music_Store_Client Untested n/a
Network_Queuing_Simulator Untested n/a
No_Paper._Not_Ever._NOPE Untested n/a
One_Vote Untested n/a
online_job_application Untested n/a
online_job_application2 Untested n/a
On_Sale Untested n/a
PCM_Message_decoder Untested n/a
REMATCH_1--Hat_Trick--Morris_Worm Untested n/a
REMATCH_2--Mail_Server--Crackaddr Untested n/a
SFTSCBSISS Untested n/a
String_Storage_and_Retrieval Untested n/a
Tick-A-Tack Untested n/a
ValveChecks Untested n/a

CWE-121: Stack-based Buffer Overflow

"A stack-based buffer overflow condition is a condition where the buffer being overwritten is allocated on the stack (i.e., is a local variable or, rarely, a parameter to a function)."

Challenge Binary Found? Query / Failure reason
BIRC Untested n/a
Barcoder Untested n/a
Bloomy_Sunday Untested n/a
CGC_Hangman_Game Untested n/a
CTTP Untested n/a
CableGrind Untested n/a
Casino_Games Untested n/a
Childs_Game Untested n/a
commerce_website Untested n/a
Corinth Untested n/a
Cromulence_All_Service Untested n/a
Diary_Parser Untested n/a
Differ Untested n/a
Document_Rendering_Engine Untested n/a
Dungeon_Master Untested n/a
Email_System_2 Untested n/a
Estadio Untested n/a
FSK_BBS Untested n/a
FaceMag Untested n/a
GPS_Tracker Untested n/a
Ghost_In_The_CGC Untested n/a
greeter Untested n/a
Gridder Untested n/a
KTY_Pretty_Printer Untested n/a
LazyCalc Untested n/a
Matrix_Math_Calculator Untested n/a
middleout Untested n/a
Movie_Rental_Service Untested n/a
Multi_User_Calendar Untested n/a
Neural_House Untested n/a
No_Paper._Not_Ever._NOPE Untested n/a
On_Sale Untested n/a
One_Amp Untested n/a
One_Vote Untested n/a
PKK_Steganography Untested n/a
Packet_Reciver Untested n/a
Palindrome Untested n/a
Palindrome2 Untested n/a
Parking_Permit_Management_System_PPMS Untested n/a
Pattern_Finder Untested n/a
payroll Untested n/a
Personal_Fitness_Manager Untested n/a
Pipelined Untested n/a
REMATCH_3--Address_Resolution_Service--SQL_Slammer Untested n/a
root64_and_parcour Untested n/a
SAuth Untested n/a
Sample_Shipgame Untested n/a
Secure_Compression Untested n/a
ShoutCTF Untested n/a
Space_Attackers Untested n/a
The_Longest_Road Untested n/a
Thermal_Controller_v2 Untested n/a
Thermal_Controller_v3 Untested n/a
ValveChecks Untested n/a
WhackJack Untested n/a
XStore Untested n/a

CWE-129: Improper Validation of Array Index

"The product uses untrusted input when calculating or using an array index, but the product does not validate or incorrectly validates the index to ensure the index references a valid position within the array."

Challenge Binary Found? Query / Failure reason
AIS-Lite Untested n/a
Blubber Untested n/a
CAT Untested n/a
CGC_Image_Parser Untested n/a
Carbonate Untested n/a
electronictrading Untested n/a
FISHYXML Untested n/a
Glue Untested n/a
Griswold Untested n/a
H20FlowInc Untested n/a
HIGHCOO Untested n/a
LAN_Simulator Untested n/a
Matchmaker Untested n/a
Material_Tempurature_Simulation Untested n/a
matrices_for_sale Untested n/a
Modern_Family_Tree Untested n/a
Query_Calculator Untested n/a
Recipe_and_Pantry_Manager Untested n/a
router_simulator Untested n/a
SCUBA_Dive_Logging Untested n/a
Secure_Compression Untested n/a
SOLFEDGE Untested n/a
stream_vm2 Untested n/a
Venture_Calculator Untested n/a

CWE-134: Use of Externally-Controlled Format String

"The software uses a function that accepts a format string as an argument, but the format string originates from an external source."

Challenge Binary Found? Query / Failure reason
Barcoder Yes cwe_134_v1.py
Checkmate No Param is a MLIL_CONST_PTR not a MLIL_VAR_SSA, phi functions as param need to be resolved to determine possible origins
CNMP No Issue originates from cgc_syslog to a format string
CGC_Hangman_Game Untested n/a
Kaprica_Go No Add checking for dprintf
Kaprica_Script_Interpreter Untested n/a
middleware_handshake Untested n/a
Movie_Rental_Service_Redux No Query fooled by use of cgc_fdprintf, not apparent in source
One_Vote Untested n/a
Packet_Analyzer Untested n/a
Shipgame_1 No Need a snprintf signature
Shipgame_2 No Need a snprintf signature
Shipgame_3 No Need a snprintf signature
Terrible_Ticket_Tracker Yes cwe_134_v1.py

CWE-788: Access of Memory Location After End of Buffer

"The software reads or writes to a buffer using an index or pointer that references a memory location after the end of the buffer."

Challenge Binary Found? Query / Failure reason
AIS-Lite Untested n/a
Document_Rendering_Engine Untested n/a
Dungeon_Master Untested n/a
FablesReport Untested n/a
FUN Untested n/a
H20FlowInc Untested n/a
KKVS Untested n/a
middleout Untested n/a
Modern_Family_Tree Untested n/a
One_Vote Untested n/a
On_Sale Untested n/a
Order_Up Untested n/a
Rejistar Untested n/a
REMATCH_6--Secure_Server--Heartbleed Untested n/a
Single-Sign-On Untested n/a
Add a custom footer

Getting Started

Results & Performance

Clone this wiki locally