Static analysis and penetration testing are common techniques used to discover security bugs in implementation code. Penetration testing is often performed in black-box way by probing the attack surface of a running system and discovering its security holes. Static analysis techniques operate in a white-box way by analyzing the source code of a system and identifying security weaknesses. Because of their different nature, the two techniques report their findings in two different ways. This paper presents an exploratory study meant to determine whether a vulnerability report generated by a security tool based on static analysis is more or less useful than a report generated by a security tool based on penetration testing. The usefulness is judged from the perspective of the developers that have to devise a vulnerability-fixing patch. The initial results show an advantage when using penetration testing in one of the two cases we investigated.
PDF version of the paper.