Fun_People Archive
13 Jun
Making Programs Explode

Date: Tue, 13 Jun 95 12:19:54 PDT
From: Peter Langston <psl>
To: Fun_People
Subject: Making Programs Explode

From: Dave@Yost.com

A recent notice for a talk:

Speaker:	Barton P. Miller
		Computer Sciences Department
		University of Wisconsin

Title:          Making Programs Explode: Using Simple Random
		Testing on Real Programs

			Abstract

In 1990, we published the results of a study of the reliability of standard
UNIX utility programs. This study showed that by using simple (almost
simplistic) random testing techniques, we could crash or hang 25-33% of the
these utility programs. Five years later, we have repeated and significantly
extended this study using the same basic techniques: subjecting programs to
random input streams. A distressingly large number of UNIX utilities still
crash with our tests.

We tested a wide variety of utility programs on nine UNIX platforms.  The
programs were sent random input streams.  We used a conservative and crude
measure of reliability: a program is considered unreliable if it crashes
with a core dump or hangs (infinite loop). We used the random testing to
also test X-Window applications and servers, network servers, and system
library interfaces.

The major results of this study are: (1) In the last five years, all
previously-tested versions of UNIX made noticeable improvements in the
reliability of their utilities.  But ... the failure rate of these systems
is still distressingly high (from 18-23% in the 1995 study). (3) Even worse
is that many of the same bugs that we reported in 1990 are still present in
the code releases of 1995. (4) The failure rate of utilities on the
commercial versions of UNIX that we tested (from Sun, IBM, SGI, DEC, and
NEXT) ranged from 15 to 43%. (5) The failure rate of the utilities on the
freely-distributed Linux version of UNIX was second-lowest, at 11%. (6) The
failure rate of the public GNU utilities was the lowest in our study, at
only 7%. (7) We could not crash network services on any of the versions of
UNIX that we tested. (8) Almost a quarter of the X- Window applications that
we tested crash on purely random input data streams (random binary data).
More significant is that more than 40% of the applications crash given
random, but legal X-event streams. (9) We could not crash X server on the
versions of UNIX that we tested (i.e., sending random data streams to the
server).

© 1995 Peter Langston