criterion performance measurements
overview
want to understand this report?
sum/Data.List.foldl'
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.033215144745678e-5 | 8.034831556854684e-5 | 8.042360293997327e-5 |
| Standard deviation | 2.1489161408174037e-8 | 9.274221760105146e-8 | 2.054350807787325e-7 |
Outlying measurements have no (8.196161464380783e-3%) effect on estimated standard deviation.
sum/Control.Monad.foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.893094470139216e-4 | 5.893446356729368e-4 | 5.893754358000811e-4 |
| Standard deviation | 9.264251947650342e-8 | 1.1301214782298991e-7 | 1.3834803574739785e-7 |
Outlying measurements have slight (1.2193263222069792e-2%) effect on estimated standard deviation.
sum/low level
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.75494638297788e-6 | 5.755071343980865e-6 | 5.755257389199044e-6 |
| Standard deviation | 3.8674883335235203e-10 | 5.273355820450178e-10 | 7.494206786073123e-10 |
Outlying measurements have no (5.681632653061225e-3%) effect on estimated standard deviation.
sum/boxed vectors
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.763064279419362e-6 | 5.763528265144594e-6 | 5.764447684665212e-6 |
| Standard deviation | 1.0255648544522794e-9 | 2.041725058859452e-9 | 4.014662386186746e-9 |
Outlying measurements have no (5.68163265306107e-3%) effect on estimated standard deviation.
sum/unboxed vectors
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.763014577011684e-6 | 5.763280678098752e-6 | 5.763567271439581e-6 |
| Standard deviation | 7.92330947441478e-10 | 9.496775994187801e-10 | 1.1942499826072816e-9 |
Outlying measurements have no (5.681632653061224e-3%) effect on estimated standard deviation.
sum/conduit, pure, fold
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8527384426530305e-4 | 1.8528562883799775e-4 | 1.852990039060086e-4 |
| Standard deviation | 3.552313730721171e-8 | 4.3716257725035075e-8 | 5.4794658608699845e-8 |
Outlying measurements have no (9.522928994082842e-3%) effect on estimated standard deviation.
sum/conduit, pure, foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.519044416990382e-4 | 6.519587755424521e-4 | 6.52023846948967e-4 |
| Standard deviation | 1.780542747784484e-7 | 2.0865198471971387e-7 | 2.6103930933105816e-7 |
Outlying measurements have slight (1.2497997115846685e-2%) effect on estimated standard deviation.
sum/conduit, IO, fold
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8160556750120154e-4 | 1.8161841898503824e-4 | 1.816390147490908e-4 |
| Standard deviation | 4.0363888362977336e-8 | 5.5699862212050825e-8 | 8.569937310360774e-8 |
Outlying measurements have no (9.433106575963617e-3%) effect on estimated standard deviation.
sum/conduit, IO, foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0414163579356862e-3 | 1.0415719830232397e-3 | 1.0417403314087678e-3 |
| Standard deviation | 4.699858152942583e-7 | 5.627672536384494e-7 | 6.799521598362818e-7 |
Outlying measurements have slight (1.4081632653061097e-2%) effect on estimated standard deviation.
monte carlo/low level
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.431719735607036e-3 | 3.4330595895722483e-3 | 3.4343207703042143e-3 |
| Standard deviation | 3.2538746618349146e-6 | 3.951941254809009e-6 | 5.015220484912488e-6 |
Outlying measurements have slight (2.082390221819791e-2%) effect on estimated standard deviation.
monte carlo/conduit
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3342102033429564e-2 | 1.3345094344236209e-2 | 1.3348103017699425e-2 |
| Standard deviation | 5.9987052702949865e-6 | 7.885306057091114e-6 | 1.1098175126678947e-5 |
Outlying measurements have slight (3.698224852070998e-2%) effect on estimated standard deviation.
sliding window/10/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5121463990432327e-3 | 1.5124610641006704e-3 | 1.5127383247081961e-3 |
| Standard deviation | 8.293010402216655e-7 | 1.0171638161084653e-6 | 1.4014222820957787e-6 |
Outlying measurements have slight (1.5621063240110861e-2%) effect on estimated standard deviation.
sliding window/10/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.664269094192341e-3 | 1.6644938623150208e-3 | 1.6647332772209151e-3 |
| Standard deviation | 6.448873822578524e-7 | 8.044664284956318e-7 | 9.988888896329453e-7 |
Outlying measurements have slight (1.6124697661918687e-2%) effect on estimated standard deviation.
sliding window/10/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.30110197398555e-3 | 2.3016037241758444e-3 | 2.302504020535686e-3 |
| Standard deviation | 1.3802311659640296e-6 | 2.2589086343959683e-6 | 3.736603773703366e-6 |
Outlying measurements have slight (1.7851239669421298e-2%) effect on estimated standard deviation.
sliding window/10/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3081524492889e-3 | 2.3084829304719683e-3 | 2.3088741228707026e-3 |
| Standard deviation | 9.907349367854003e-7 | 1.224293389727956e-6 | 1.612018000108829e-6 |
Outlying measurements have slight (1.785123966942134e-2%) effect on estimated standard deviation.
sliding window/10/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.920406317601597e-3 | 8.923635929750448e-3 | 8.931110143842859e-3 |
| Standard deviation | 6.940930897089675e-6 | 1.3226464464423624e-5 | 2.4144553404562662e-5 |
Outlying measurements have slight (3.0273437499999948e-2%) effect on estimated standard deviation.
sliding window/10/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.62189931495284e-3 | 9.624173179040381e-3 | 9.627410837876303e-3 |
| Standard deviation | 4.9086661296791175e-6 | 7.482098489619955e-6 | 1.0612225646254854e-5 |
Outlying measurements have slight (3.121748178980213e-2%) effect on estimated standard deviation.
sliding window/30/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.410905895908723e-3 | 3.4120902830370144e-3 | 3.413930393504338e-3 |
| Standard deviation | 3.3696685596587884e-6 | 4.5934133395930765e-6 | 7.249340637863483e-6 |
Outlying measurements have slight (2.039930555555541e-2%) effect on estimated standard deviation.
sliding window/30/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.9778998200768016e-3 | 1.979106359854091e-3 | 1.9823000597584788e-3 |
| Standard deviation | 2.9178522675574305e-6 | 6.242964702195969e-6 | 1.179638083825667e-5 |
Outlying measurements have slight (1.6944114149821585e-2%) effect on estimated standard deviation.
sliding window/30/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.2348945282162553e-3 | 2.2356248901398e-3 | 2.236665512095786e-3 |
| Standard deviation | 2.216450632998817e-6 | 2.9575828766017627e-6 | 4.332828966295815e-6 |
Outlying measurements have slight (1.785123966942149e-2%) effect on estimated standard deviation.
sliding window/30/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.3197094537074255e-3 | 4.326742351604152e-3 | 4.340300153702711e-3 |
| Standard deviation | 1.2703045078104195e-5 | 2.952731885415408e-5 | 4.7431595128878856e-5 |
Outlying measurements have slight (2.271498107084897e-2%) effect on estimated standard deviation.
sliding window/30/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.005120778213919e-3 | 9.016016444117973e-3 | 9.052080672038847e-3 |
| Standard deviation | 1.0114750400221084e-5 | 5.1288065647645144e-5 | 1.0480072096212416e-4 |
Outlying measurements have slight (3.02734375e-2%) effect on estimated standard deviation.
sliding window/30/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.156705331664747e-3 | 9.161889902371882e-3 | 9.178407171242131e-3 |
| Standard deviation | 8.996614056003973e-6 | 2.3436167445973076e-5 | 4.524110307646063e-5 |
Outlying measurements have slight (3.1217481789802035e-2%) effect on estimated standard deviation.
sliding window/100/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.58894987486149e-3 | 9.60188056114661e-3 | 9.65019223138805e-3 |
| Standard deviation | 5.322718813972647e-6 | 6.587233703316086e-5 | 1.360222127418246e-4 |
Outlying measurements have slight (3.121748178980223e-2%) effect on estimated standard deviation.
sliding window/100/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.068415838213535e-3 | 3.0718637004761283e-3 | 3.078299872354803e-3 |
| Standard deviation | 1.0457825956475865e-5 | 1.4220792106493137e-5 | 2.201785995492172e-5 |
Outlying measurements have slight (1.9991670137442505e-2%) effect on estimated standard deviation.
sliding window/100/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1247138982424256e-3 | 2.1265712833072616e-3 | 2.1321980619154316e-3 |
| Standard deviation | 4.568357107618319e-6 | 9.609144097562137e-6 | 2.118487701368696e-5 |
Outlying measurements have slight (1.7538265306122347e-2%) effect on estimated standard deviation.
sliding window/100/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0591848600051624e-2 | 1.0598037849423834e-2 | 1.0604878473370358e-2 |
| Standard deviation | 1.3272513969825497e-5 | 1.7192781503532566e-5 | 2.34960352671549e-5 |
Outlying measurements have slight (3.3293697978596756e-2%) effect on estimated standard deviation.
sliding window/100/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.941056463180237e-3 | 9.951146940081446e-3 | 9.975642177625671e-3 |
| Standard deviation | 1.826633073396144e-5 | 4.079095311454e-5 | 8.009561942036741e-5 |
Outlying measurements have slight (3.222222222222222e-2%) effect on estimated standard deviation.
sliding window/100/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.902904071285762e-3 | 8.910083643299656e-3 | 8.916947784902342e-3 |
| Standard deviation | 1.6365889731320598e-5 | 2.005669081499954e-5 | 2.452411465464508e-5 |
Outlying measurements have slight (3.02734375e-2%) effect on estimated standard deviation.
sliding window/1000/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.560325440858234e-2 | 7.565166190421009e-2 | 7.569043554164255e-2 |
| Standard deviation | 4.524297756868892e-5 | 7.212727185778656e-5 | 1.1390209690250106e-4 |
Outlying measurements have slight (8.264462809917357e-2%) effect on estimated standard deviation.
sliding window/1000/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5837691442957436e-2 | 1.5857220380524326e-2 | 1.5876813159322378e-2 |
| Standard deviation | 4.00235343327669e-5 | 4.9329588104041284e-5 | 6.060645782116215e-5 |
Outlying measurements have slight (3.993055555555522e-2%) effect on estimated standard deviation.
sliding window/1000/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8902010564375157e-3 | 1.929666723096848e-3 | 2.0827646790144917e-3 |
| Standard deviation | 9.01911685821813e-6 | 2.5089858563317156e-4 | 5.340067622438734e-4 |
Outlying measurements have severe (0.7998807741188889%) effect on estimated standard deviation.
sliding window/1000/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.675560602349576e-2 | 7.689161844875034e-2 | 7.697809272700434e-2 |
| Standard deviation | 9.962167516221392e-5 | 1.7143225079380256e-4 | 2.51493720574029e-4 |
Outlying measurements have slight (9.0e-2%) effect on estimated standard deviation.
sliding window/1000/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1787315776038874e-2 | 2.1818103048044123e-2 | 2.184140613003472e-2 |
| Standard deviation | 4.1205098967301716e-5 | 5.9750319224589364e-5 | 8.751135254976553e-5 |
Outlying measurements have slight (4.75e-2%) effect on estimated standard deviation.
sliding window/1000/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.945312644122484e-3 | 7.959994978019648e-3 | 7.976564376473996e-3 |
| Standard deviation | 3.3326724912734826e-5 | 4.2462388987713765e-5 | 5.684597145209705e-5 |
Outlying measurements have slight (2.8546712802768166e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.