Fuzzing is an efficient method for locating software program vulnerabilities. Over the previous few years Android has been centered on bettering the effectiveness, scope, and comfort of fuzzing throughout the group. This effort has instantly resulted in improved check protection, fewer safety/stability bugs, and better code high quality. Our implementation of steady fuzzing permits software program groups to search out new bugs/vulnerabilities, and stop regressions robotically with out having to manually provoke fuzzing runs themselves. This put up recounts a quick historical past of fuzzing on Android, shares how Google performs fuzzing at scale, and paperwork our expertise, challenges, and success in constructing an infrastructure for automating fuzzing throughout Android. When you’re considering contributing to fuzzing on Android, we’ve included directions on tips on how to get began, and knowledge on how Android’s VRP rewards fuzzing contributions that discover vulnerabilities.
A Transient Historical past of Android Fuzzing
Fuzzing has been round for a few years, and Android was among the many early massive software program tasks to automate fuzzing and prioritize it equally to unit testing as a part of the broader purpose to make Android essentially the most safe and steady working system. In 2019 Android kicked off the fuzzing undertaking, with the purpose to assist institutionalize fuzzing by making it seamless and a part of code submission. The Android fuzzing undertaking resulted in an infrastructure consisting of Pixel telephones and Google cloud based mostly digital gadgets that enabled scalable fuzzing capabilities throughout your complete Android ecosystem. This undertaking has since grown to turn into the official inside fuzzing infrastructure for Android and performs 1000’s of fuzzing hours per day throughout lots of of fuzzers.
Beneath the Hood: How Is Android Fuzzed
Step 1: Outline and discover all of the fuzzers in Android repo
Step one is to combine fuzzing into the Android construct system (Soong) to allow construct fuzzer binaries. Whereas builders are busy including options to their codebase, they will embrace a fuzzer to fuzz their code and submit the fuzzer alongside the code they’ve developed. Android Fuzzing makes use of a construct rule referred to as cc_fuzz (see instance beneath). cc_fuzz (we additionally assist rust_fuzz and java_fuzz) defines a Soong module with supply file(s) and dependencies that may be constructed right into a binary.
cc_fuzz {
identify: "fuzzer_foo",
srcs: [
"fuzzer_foo.cpp",
],
static_libs: [
"libfoo",
],
host_supported: true,
}
A packaging rule in Soong finds all of those cc_fuzz definitions and builds them robotically. The precise fuzzer construction itself may be very easy and consists of 1 primary methodology (LLVMTestOneInput):
#embrace <stddef.h>
#embrace <stdint.h>
extern "C" int LLVMFuzzerTestOneInput(
const uint8_t *knowledge,
size_t dimension) {
// Right here you invoke the code to be fuzzed.
return 0;
}
This fuzzer will get robotically constructed right into a binary and together with its static/dynamic dependencies (as specified within the Android construct file) are packaged into a zipper file which will get added to the principle zip containing all fuzzers as proven within the instance beneath.
Step 2: Ingest all fuzzers into Android builds
As soon as the fuzzers are discovered within the Android repository and they’re constructed into binaries, the following step is to add them to the cloud storage in preparation to run them on our backend. This course of is run a number of instances day by day. The Android fuzzing infrastructure makes use of an open supply steady fuzzing framework (Clusterfuzz) to run fuzzers constantly on Android gadgets and emulators. So as to run the fuzzers on clusterfuzz, the fuzzers zip information are renamed after the construct and the most recent construct will get to run (see diagram beneath):
The fuzzer zip file accommodates the fuzzer binary, corresponding dictionary in addition to a subfolder containing its dependencies and the git revision numbers (sourcemap) comparable to the construct. Sourcemaps are used to reinforce stack traces and produce crash studies.
Step 3: Run fuzzers constantly and discover bugs
Working fuzzers constantly is completed via scheduled jobs the place every job is related to a set of bodily gadgets or emulators. A job can be backed by a queue that represents the fuzzing duties that have to be run. These duties are a mixture of operating a fuzzer, reproducing a crash present in an earlier fuzzing run, or minimizing the corpus, amongst different duties.
Every fuzzer is run for a number of hours, or till they discover a crash. After the run, Android fuzzing takes the entire fascinating enter found throughout the run and provides it to the fuzzer corpus. This corpus is then shared throughout fuzzer runs and grows over time. The fuzzer is then prioritized in subsequent runs in response to the expansion of recent protection and crashes discovered (if any). This ensures we offer the simplest fuzzers extra time to run and discover fascinating crashes.
Step 4: Generate fuzzers line protection
What good is a fuzzer if it’s not fuzzing the code you care about? To enhance the standard of the fuzzer and to watch the general progress of Android fuzzing, two sorts of protection metrics are calculated and obtainable to Android builders. The primary metric is for edge protection which refers to edges within the Management Move Graph (CFG). By instrumenting the fuzzer and the code being fuzzed, the fuzzing engine can observe small snippets of code that get triggered each time execution circulate reaches them. That means, fuzzing engines know precisely what number of (and what number of instances) every of those instrumentation factors obtained hit on each run to allow them to mixture them and calculate the protection.
INFO: Seed: 2859304549 INFO: Loaded 1 modules (773 inline 8-bit counters): 773 [0x5610921000, 0x5610921305), INFO: Loaded 1 PC tables (773 PCs): 773 [0x5610921308,0x5610924358), INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes INFO: A corpus is not provided, starting from an empty corpus #2 INITED cov: 2 ft: 2 corp: 1/1b lim: 4 exec/s: 0 rss: 24Mb #413 NEW cov: 3 ft: 3 corp: 2/9b lim: 8 exec/s: 0 rss: 24Mb L: 8/8 MS: 1 InsertRepeatedBytes- #3829 NEW cov: 4 ft: 4 corp: 3/17b lim: 38 exec/s: 0 rss: 24Mb L: 8/8 MS: 1 ChangeBinInt- ...
Line coverage inserts instrumentation points specifying lines in the source code. Line coverage is very useful for developers as they can pinpoint areas in the code that are not covered and update their fuzzers accordingly to hit those areas in future fuzzing runs.
Drilling into any of the folders can show the stats per file:
Further clicking on one of the files shows the lines that were touched and lines that never got coverage. In the example below, the first line has been fuzzed ~5 million times, but the fuzzer never makes it into lines 3 and 4, indicating a gap in the coverage for this fuzzer.
We have dashboards internally that measure our fuzzing coverage across our entire codebase. In order to generate these coverage dashboards yourself, you follow these steps.
Another measurement of the quality of the fuzzers is how many fuzzing iterations can be done in one second. It has a direct relationship with the computation power and the complexity of the fuzz target. However, this parameter alone can not measure how good or effective the fuzzing is.
How we handle fuzzer bugs
Android fuzzing utilizes the Clusterfuzz fuzzing infrastructure to handle any found crashes and file a ticket to the Android security team. Android security makes an assessment of the crash based on the Android Severity Guidelines and then routes the vulnerability to the proper team for remediation. This entire process of finding the reproducible crash, routing to Android Security, and then assigning the issue to a team responsible can take as little as two hours, and up to a week depending on the type of crash and the severity of the vulnerability.
One example of a recent fuzzer success is (CVE 2022-20473), where an internal team wrote a 20-line fuzzer and submitted it to run on Android fuzzing infra. Within a day, the fuzzer was ingested and pushed to our fuzzing infrastructure to begin fuzzing, and shortly found a critical severity vulnerability! A patch for this CVE has been applied by the service team.
Why Android Continues to Invest in Fuzzing
Protection Against Code Regressions
The Android Open Source Project (AOSP) is a large and complex project with many contributors. As a result, there are thousands of changes made to the project every day. These changes can be anything from small bug fixes to large feature additions, and fuzzing helps to find vulnerabilities that may be inadvertently introduced and not caught during code review.
Continuous fuzzing has helped to find these vulnerabilities before they are introduced in production and exploited by attackers. One real-life example is (CVE-2023-21041), a vulnerability discovered by a fuzzer written three years ago. This vulnerability affected Android firmware and could have led to local escalation of privilege with no additional execution privileges needed. This fuzzer was running for many years with limited findings until a code regression led to the introduction of this vulnerability. This CVE has since been patched.
Protection against unsafe memory language pitfalls
Android has been a huge proponent of Rust, with Android 13 being the first Android release with the majority of new code in a memory safe language. The amount of new memory-unsafe code entering Android has decreased, but there are still millions of lines of code that remain, hence the need for fuzzing persists.
No One Code is Safe: Fuzzing code in memory-safe languages
Our work does not stop with non-memory unsafe languages, and we encourage fuzzer development in languages like Rust as well. While fuzzing won’t find common vulnerabilities that you would expect to see memory unsafe languages like C/C++, there have been numerous non-security issues discovered and remediated which contribute to the overall stability of Android.
Fuzzing Challenges
In addition to generic C/C++ binaries issues such as missing dependencies, fuzzers can have their own classes of problems:
Low executions per second: in order to fuzz efficiently, the number of mutations has to be in the order of hundreds per second otherwise the fuzzing will take a very long time to cover the code. We addressed this issue by adding a set of alerts that continuously monitor the health of the fuzzers as well as any sudden drop in coverage. Once a fuzzer is identified as underperforming, an automated email is sent to the fuzzer author with details to help them improve the fuzzer.
Fuzzing the wrong code: Like all resources, fuzzing resources are limited. We want to ensure that those resources give us the highest return, and that generally means devoting them towards fuzzing code that processes untrusted (i.e. potentially attacker controlled) inputs. This can cover any way that the phone can receive input including Bluetooth, NFC, USB, web, etc. Parsing structured input is particularly interesting since there is room for programming errors due to specs complexity. Code that generates output is not particularly interesting to fuzz. Similarly internal code that is not exposed publicly is also less of a security concern. We addressed this issue by identifying the most vulnerable code (see the following section).
What to fuzz
In order to fuzz the most important components of the Android source code, we focus on libraries that have:
- A history of vulnerabilities: the history should not be the distant history since context change but more focus on the last 12 months.
- Recent code changes: research indicates that more vulnerabilities are found in recently changed code than code that is more stable.
- Remote access: vulnerabilities in code that are reachable remotely can be critical.
- Privileged: Similarly to #3, vulnerabilities in code that runs in privileged processes can be critical.
How to submit a fuzzer to AOSP
We’re constantly writing and improving fuzzers internally to cover some of the most sensitive areas of Android, but there is always room for improvement. If you’d like to get started writing your own fuzzer for an area of AOSP, you’re welcome to do so to make Android more secure (example CL):
- Get Android source code
- Have a testing phone?
- Write a fuzz target (follow guidelines in ‘What to fuzz’ section)
- Upload your fuzzer to AOSP.
Get started by reading our documentation on Fuzzing with libFuzzer and check your fuzzer into the Android Open Source project. If your fuzzer finds a bug, you can submit it to the Android Bug Bounty Program and could be eligible for a reward!
