criterion-1.6.3.0: Robust, reliable performance measurement and analysis
Copyright(c) 2009-2014 Bryan O'Sullivan
LicenseBSD-style
Maintainerbos@serpentine.com
Stabilityexperimental
PortabilityGHC
Safe HaskellTrustworthy
LanguageHaskell2010

Criterion.Main

Description

Wrappers for compiling and running benchmarks quickly and easily. See defaultMain below for an example.

All of the IO-returning functions in this module initialize the timer before measuring time (refer to the documentation for initializeTime for more details).

Synopsis

How to write benchmarks

The Benchmarkable type is a container for code that can be benchmarked. The value inside must run a benchmark the given number of times. We are most interested in benchmarking two things:

  • IO actions. Most IO actions can be benchmarked directly.
  • Pure functions. GHC optimises aggressively when compiling with -O, so it is easy to write innocent-looking benchmark code that doesn't measure the performance of a pure function at all. We work around this by benchmarking both a function and its final argument together.

Benchmarking IO actions

Most IO actions can be benchmarked easily using one of the following two functions:

nfIO   :: NFData a => IO a -> Benchmarkable
whnfIO ::               IO a -> Benchmarkable

In certain corner cases, you may find it useful to use the following variants, which take the input as a separate argument:

nfAppIO   :: NFData b => (a -> IO b) -> a -> Benchmarkable
whnfAppIO ::               (a -> IO b) -> a -> Benchmarkable

This is useful when the bulk of the work performed by the function is not bound by IO, but rather by pure computations that may optimize away if the argument is known statically, as in nfIO/whnfIO.

Benchmarking pure code

Because GHC optimises aggressively when compiling with -O, it is potentially easy to write innocent-looking benchmark code that will only be evaluated once, for which all but the first iteration of the timing loop will be timing the cost of doing nothing.

To work around this, we provide two functions for benchmarking pure code.

The first will cause results to be fully evaluated to normal form (NF):

nf :: NFData b => (a -> b) -> a -> Benchmarkable

The second will cause results to be evaluated to weak head normal form (the Haskell default):

whnf :: (a -> b) -> a -> Benchmarkable

As both of these types suggest, when you want to benchmark a function, you must supply two values:

  • The first element is the function, saturated with all but its last argument.
  • The second element is the last argument to the function.

Here is an example that makes the use of these functions clearer. Suppose we want to benchmark the following function:

firstN :: Int -> [Int]
firstN k = take k [(0::Int)..]

So in the easy case, we construct a benchmark as follows:

nf firstN 1000

Fully evaluating a result

The whnf harness for evaluating a pure function only evaluates the result to weak head normal form (WHNF). If you need the result evaluated all the way to normal form, use the nf function to force its complete evaluation.

Using the firstN example from earlier, to naive eyes it might appear that the following code ought to benchmark the production of the first 1000 list elements:

whnf firstN 1000

Since we are using whnf, in this case the result will only be forced until it reaches WHNF, so what this would actually benchmark is merely how long it takes to produce the first list element!

Types

data Benchmark #

Instances

Instances details
Show Benchmark 
Instance details

Defined in Criterion.Measurement.Types

Methods

showsPrec :: Int -> Benchmark -> ShowS

show :: Benchmark -> String

showList :: [Benchmark] -> ShowS

Creating a benchmark suite

env :: NFData env => IO env -> (env -> Benchmark) -> Benchmark #

envWithCleanup :: NFData env => IO env -> (env -> IO a) -> (env -> Benchmark) -> Benchmark #

perBatchEnv :: (NFData env, NFData b) => (Int64 -> IO env) -> (env -> IO b) -> Benchmarkable #

perBatchEnvWithCleanup :: (NFData env, NFData b) => (Int64 -> IO env) -> (Int64 -> env -> IO ()) -> (env -> IO b) -> Benchmarkable #

perRunEnv :: (NFData env, NFData b) => IO env -> (env -> IO b) -> Benchmarkable #

perRunEnvWithCleanup :: (NFData env, NFData b) => IO env -> (env -> IO ()) -> (env -> IO b) -> Benchmarkable #

toBenchmarkable :: (Int64 -> IO ()) -> Benchmarkable #

bench :: String -> Benchmarkable -> Benchmark #

bgroup :: String -> [Benchmark] -> Benchmark #

Running a benchmark

nf :: NFData b => (a -> b) -> a -> Benchmarkable #

whnf :: (a -> b) -> a -> Benchmarkable #

nfIO :: NFData a => IO a -> Benchmarkable #

nfAppIO :: NFData b => (a -> IO b) -> a -> Benchmarkable #

whnfAppIO :: (a -> IO b) -> a -> Benchmarkable #

Turning a suite of benchmarks into a program

defaultMain :: [Benchmark] -> IO () Source #

An entry point that can be used as a main function.

import Criterion.Main

fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n-1) + fib (n-2)

main = defaultMain [
       bgroup "fib" [ bench "10" $ whnf fib 10
                    , bench "35" $ whnf fib 35
                    , bench "37" $ whnf fib 37
                    ]
                   ]

defaultMainWith :: Config -> [Benchmark] -> IO () Source #

An entry point that can be used as a main function, with configurable defaults.

Example:

import Criterion.Main.Options
import Criterion.Main

myConfig = defaultConfig {
              -- Resample 10 times for bootstrapping
              resamples = 10
           }

main = defaultMainWith myConfig [
         bench "fib 30" $ whnf fib 30
       ]

If you save the above example as "Fib.hs", you should be able to compile it as follows:

ghc -O --make Fib

Run "Fib --help" on the command line to get a list of command line options.

defaultConfig :: Config Source #

Default benchmarking configuration.

Other useful code

makeMatcher Source #

Arguments

:: MatchType 
-> [String]

Command line arguments.

-> Either String (String -> Bool) 

Create a function that can tell if a name given on the command line matches a benchmark.

runMode :: Mode -> [Benchmark] -> IO () Source #

Run a set of Benchmarks with the given Mode.

This can be useful if you have a Mode from some other source (e.g. from a one in your benchmark driver's command-line parser).