🛀🏻 🥅 💆🏽 DevSecOps: organizing source code fuzzing 🏕️ 😩 🦂

Having learned the results of a vote held in one of our previous articles , we decided to discuss in more detail the issue of organizing fuzzing. In addition, as part of the online meeting on information security "Digital Security ON AIR", we presented a report based on our experience in DevSecOps , where we also talked about this interesting topic.

Youtube-. , !

- ?
- ?
- ?
- - Google
- Google ?
- ?

Building Secure and Reliable Systems Google. , . IT- , , .

?

, . , , , . . , . IT : . - . , , , . , (CI), , . , CI, .

CI/CD , , . , , , . . , . . .

, , , , , . , , . . — — . . :)

https://resources.securitycompass.com/blog/how-to-sell-training-costs-internally-2

, : , . , . . . , - , , , , - , . . , — . .

. . , .

?

. "" ("fuzzing") 30 An Empirical Study of the Reliability of UNIX Utilities. . 1988 , . fuzzer. . : “ . , ”. :

— , , .

, ++, :

;
, libFuzzer AFL, , , C C++;
C C++.

Rust Go.

, . .

The Art, Science, and Engineering of Fuzzing:A Survey

. :

:
- Black-box — .
- Gray-box — - ,
- White-box — .
, :
- source-based —
- binary-based —
:
- feedback driven —
- not feedback driven —
, :

, binary-based black-box, source-based — gray- white-box. source-based , DevSecOps opensource-, , .

?

- : " 30 ( , 70), ?".

Microsoft Secure Development Lifecycle

, , black-box-. 2013 , feedback-driven fuzzing, . , .

Feedback-driven fuzzing — , , . , (feedback) . . , , . — , .

feedback-driven American Fuzzy Lop Michael Zalewski. AFL 2013 .

AFL , , :

. shared bitmap;
AFL , . fork;
" — — ". , .

, , . .

AFL . . Google Summer of Code (GSoC), . AFL, (, ). , AFL . , AFL , - — AFL++.

AFL , , . , , .

. 2015 libFuzzer — LLVM. LibFuzzer feedback-driven in-memory , :

;
;
toolchain.

Continuous Fuzzing, . , , , .

	Code analysis	Fuzzing
CI	*

	~100%



	*

. , - , , , , .

. . — , .

— . , .

“How to Prevent the next Heartbleed”, openssl heartbleed. , , heartbleed. , 100% , .

Continuous Fuzzing? Continuous fuzzing — . , . — .

, QA- . , . . continuous fuzzing : , , — . . , — . , .

. . Google . 2019- , ClusterFuzz 16000 Chrome 11000 160 .

. , , . ClusterFuzz Google LibFuzzer, AFL HonggFuzz ( LibFuzzer’a). Microsoft Springfield, SaaS. Mozilla, , (Grizzly) . Github . , ossfuzz .

, , , , "" " ". ? ClusterFuzz’a, Google , . , . - , " " .

, Continuous Fuzzing , , Google. — Continuous Fuzzing . , .

, . Google, .

, , 3 :

(Attack Surface)
- , (: Sourcetrail)
- MindMap- (: Xmind)
- AFL ( — AFL++)
- LibFuzzer (HonggFuzz)
- KLEE*
(Sanitizers)

. , sourcetrail, understand, - IDE. , "" -. - , . , . - , . . . "" (code coverage), , , .

AFL libFuzzer , , . . KLEE , . KLEE ( , KLEE ). - SMT, , . , KLEE , . KLEE , . , .

— . , . . , , , . , . , , . .

GCC Clang. , .

, Attack Surface. FuzzCon, 25- , :

, . , , , . .

Attack Surface

"attack surface" (" ")? , , , Wired. : , " ".

, , . — , , /API ..

. ( , ):

— (XML, JSON), (, , );
— (HTTP, SMTP), (SSL/TLS), (Firefox, Chrome) (ZIP, TAR).

, gui.

, .

, :

. , , .

(fuzz target) — , API. , , .

attack surface. :

, . , .
. , html, png, zip-. , .
(, , ), .
, API, , , . , — .

opensource- sourcetrail

, : ! " ", , .

, , , — (seeds). — , , . .

, , . . , - .

seeds , :

. , .
, . , .
. , , . .

. (corpus) — -, . , , . , — , — .. , (feedback-driven) — , , .

. . , . , AFL, . AFL Michal Zalewski . .

. libfuzzer: , , . .

, , . llvm, clang . :

#include "MyAPI.h"

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
  MyAPI_ProcessInput(Data, Size);
  return 0;
}

, MyAPI_ProcessInput Data Size. , . , — .

libfuzzer

<!--
INFO: Seed: 2240819152
INFO: Loaded 1 modules   (6 inline 8-bit counters): 6 [0x565e90, 0x565e96), 
INFO: Loaded 1 PC tables (6 PCs): 6 [0x541908,0x541968), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2  INITED cov: 3 ft: 4 corp: 1/1b lim: 4 exec/s: 0 rss: 35Mb
=================================================================
==29562==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff2fba1ba0 at pc 0x0000004dda61 bp 0x7fff2fba1b30 sp 0x7fff2fba12d0
WRITE of size 65 at 0x7fff2fba1ba0 thread T0
    #0 0x4dda60 in strcpy (/home/user/Documents/tmp/main+0x4dda60)
    #1 0x52540f in MyAPI_ProcessInput(char const*) (/home/user/Documents/tmp/main+0x52540f)
    #2 0x52561e in LLVMFuzzerTestOneInput (/home/user/Documents/tmp/main+0x52561e)
    #3 0x42fe0a in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/user/Documents/tmp/main+0x42fe0a)
    #4 0x42f3a5 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) (/home/user/Documents/tmp/main+0x42f3a5)
    #5 0x4310ee in fuzzer::Fuzzer::MutateAndTestOne() (/home/user/Documents/tmp/main+0x4310ee)
    #6 0x431dc5 in fuzzer::Fuzzer::Loop(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, fuzzer::fuzzer_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (/home/user/Documents/tmp/main+0x431dc5)
    #7 0x427df0 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/user/Documents/tmp/main+0x427df0)
    #8 0x44b402 in main (/home/user/Documents/tmp/main+0x44b402)
    #9 0x7f7ee294409a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
    #10 0x421909 in _start (/home/user/Documents/tmp/main+0x421909)

Address 0x7fff2fba1ba0 is located in stack of thread T0 at offset 96 in frame
    #0 0x5252ef in MyAPI_ProcessInput(char const*) (/home/user/Documents/tmp/main+0x5252ef)

  This frame has 1 object(s):
    [32, 96) 'buf' <== Memory access at offset 96 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow (/home/user/Documents/tmp/main+0x4dda60) in strcpy
Shadow bytes around the buggy address:
  0x100065f6c320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100065f6c330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100065f6c340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100065f6c350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100065f6c360: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 00 00 00
=>0x100065f6c370: 00 00 00 00[f3]f3 f3 f3 00 00 00 00 00 00 00 00
  0x100065f6c380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100065f6c390: f1 f1 f1 f1 00 00 00 00 f2 f2 f2 f2 f8 f3 f3 f3
  0x100065f6c3a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100065f6c3b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100065f6c3c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==29562==ABORTING
MS: 1 InsertRepeatedBytes-; base unit: adc83b19e793491b1c6ea0fd8b46cd9f32e592fc
0xa,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,
\x0a\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff
artifact_prefix='./'; Test unit written to ./crash-76387a0aaeb6a1d2b6b6f095ab49c927c00243e5
-->

, . , : , , .

. whitebox- unix-.

*	LibFuzzer	AFL	ASAN/UBSAN/TSAN
Windows	±	-	±
Linux	+	+	+
OSX	+	+	+

, Windows, . -, . -, , , . unix– . .

, , , libfuzzer – LLVM, AFL llvm_mode, readme :

, .

LLVM — clang. , , , .

, , , , , , . , cmake :

# Step 1
add_custom_target(${FUZZ_TARGET_NAME})

# Step 2
function(add_fuzzer name path)
    add_executable(${name} ${SRC} ${path})
    target_compile_options(...)
    set_target_properties(...)
    add_dependencies(${FUZZ_TARGET_NAME} ${name})

# Step 3
set(fuzzers my_ideal_fuzzer1 my_ideal_fuzzer2 ...)
foreach(fuzzer ${fuzzers})
    add_fuzzer(${fuzzer} ${FUZZERS_DIR})
endforeach()