Skip links

Smart Contract Auditing: Human vs. Machine

In this article we are benchmarking several auditing tools. The smart contract security audit is a critical phase in the development of smart contracts. The DAO hack was just one trip in the odyssey to secure Ethereum smart contracts and compatible blockchains like RSK and Cardano. It is important to highlight that back in 2016 A Call for a Temporary Moratorium on The DAO was published but it fell on deaf ears. Lesson learned: never deploy a smart contract without one or more security audits. It is tempting to use automated security tools to save time and money but they cannot replace human brains right now.

Linters and analysis tools are used to quickly point out errors in the syntax and in the smart contract logic. More complex tools can be used to better analyze in depth the behavior of smart contracts.

In this article we will examine how some of these tools fare against well known vulnerabilities extracted from the Smart Contract Weakness Registry.

Tools

We selected a set of tools that have different ways to analyse and report errors and are well known in the community. We have linters like Solhint and Solium that will perform an analysis at the source code level, and other more advanced like Oyente that acts at the opcode level to detect erroneous behavior.

Manticore

A symbolic execution tool that provides an API for writing your own tests from within Python. It comes with a set of heuristics for detecting some common mistakes, and we checked those.

Mythril

Mythril is a tool defined specifically for security and does seem to fare well against common bugs, yet still wasn’t perfect. It can also generate a control flow graph and can install a certain version of solc.

Oyente

An Analysis Tool for Smart Contracts. It performs the evaluation a the opcode level so problems at the Solidity syntax will not be detected.

Our tests were done with the latest release available as a docker container which includes Oyente v0.2.7.3.

SmartCheck

SmartCheck is an online tool that automatically checks Smart Contracts for vulnerabilities and bad practices. We testested against v0.0.9 released on October 25.

Solhint

This tool examines syntax and style errors, and a few security problems. We ran our tests against Solhint v1.4.0.

Solium

This is a linter designed to highlight style issues with the code. However, it also accepts some plugins that help detect security issues too. We tested against version 1.1.8. This tool has a plugin system, which provides the ability to extend it.

SonarSolidity

This is a tool provided as a plugin to the SonarQube console to analyse the code. It classifies what it finds in “code smells”, vulnerabilities and bugs. It can also import Solium reports, which can be further enhanced with Solium’s own plugins. We tested version 3.2.0.1227 by itself to see what we found.

Tests

We selected a representative subset from the SWC registry to run our test against. We specially chose some from the ranking in the Decentralized Application Security Project of vulnerabilities with a largest impact.

    • SWC-100: Function Default Visibility
      In solidity functions without visibility are marked as public and it will expose them, .
    • SWC-101: Integer Overflow and Underflow
      The result of arithmetic operations might exceed the maximum type value, or fall below the minimum if not properly addresses these will trigger unintended behavior.
    • SWC-106: Unprotected SELFDESTRUCT Instruction
      The SELFDESTRUCT operation causes the smart contract calling it to be deleted from the state and it cannot be called afterward. The attack to the Parity Multi-Sig wallet was caused by a contract that wasn’t properly initialized and left it unprotected.
    • SWC-107: Reentrancy
      When a contracts makes a calls to external contract it giving up control and it is possible for another function on the same contract to be called.
    • SWC-108: State Variable Default Visibility
      Explicitly defining storage variable visibility prevent the misuse of them from inheriting contracts.
    • SWC-109: Uninitialized Storage Pointer
      Uninitialized local storage variables can point to existing storage variables in the contract incorrect use will cause inadvertently overwrites.
    • SWC-112: Delegatecall to Untrusted Callee
      To call an untrusted contracts with delegatecall enables the target to modify the storage of the caller.
    • SWC-113: DoS with Failed Call
      Making calls to external contracts might fail. If several calls are executed inside a loop a single failure will cause the whole transaction to revert.
    • SWC-114: Transaction Order Dependence
      Miners are allowed to reorder transactions. A contract that depends on certain transaction order might be susceptible to attacks, for example Front Running.
    • SWC-116: Timestamp Dependence
      The timestamp in the block is controlled by the miner and it can manipulate it at his convenience. It should not be used for critical calculations.
    • SWC-119: Shadowing State Variables
      Solidity allow using the same names for variables in different contexts. Using the same name is confusing and might cause subtle bugs.
  • SWC-120: Weak Sources of Randomness from Chain Attributes
    The possible sources of randomness available to smart contracts is limited and an incorrect calculation can cause the results not to be random. The SmartBillions lottery was attacked through the incorrect assumption that all block hashes were accessible to smart contracts, only the last 256 hashes are available.

Results

The following table resumes the execution of the differents tools against our tests.

SWCDescriptionManticoreMythrilOyenteSmartecSolhintSoliumSonarSolidity
100Function Default Visibilityn/a (1)✔ (2.a)
101Integer Overflow and Underflow✔ (2.b)
106Unprotected SELFDESTRUCT Instruction
107Reentrancy
108State Variable Default Visibilityn/a (1)✔ (2.c)
109Uninitialized Storage Pointer
112Delegatecall to Untrusted Callee✘ (2.d)
113DoS with Failed Call
114Transaction Order Dependence
116Timestamp Dependence
119Shadowing State Variablesn/a (1)✔ (2.e)
120Weak Sources of Randomness from Chain Attributes

Notes:

    1. Oyente analysis is at the EVM level so it will not be able to detect issues with the Solidity language.
  1. Solium uses a plugin architecture, to detect some of the attacks a separate plugin has to be enabled.
      1. security/enforce-explicit-visibility
      1. zeppelin/no-arithmetic-operations
      1. zeppelin/constant-candidates
      1. security/no-low-level-calls
    1. zeppelin/no-state-variable-shadowing

Conclusion

Most of the tools analyzed find the more common errors like function or variable visibility or arithmetic overflows/underflows. Some event will identify and recommend better practices.

Most of them will fail against more complex attacks like transaction reordering, or denial of service attacks. These attacks are complex to detect for a competent auditor and often requires interaction between several contracts.

In general the outcome was far from perfect. They find the more simple errors and failed against more complex attacks. Humans win.

If you liked this article, you might also like: