criterion performance measurements
overview
want to understand this report?
sum/Data.List.foldl'
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.076625071921971e-5 | 8.07735276067103e-5 | 8.078190712255119e-5 |
| Standard deviation | 2.2319285480238645e-8 | 2.6598291055114367e-8 | 3.5057953315985684e-8 |
Outlying measurements have no (8.196161464380642e-3%) effect on estimated standard deviation.
sum/Control.Monad.foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.96307493222459e-4 | 5.963515205726343e-4 | 5.964017958334409e-4 |
| Standard deviation | 1.2955126986251228e-7 | 1.6283731166833463e-7 | 2.4753998739041407e-7 |
Outlying measurements have slight (1.2193263222069805e-2%) effect on estimated standard deviation.
sum/low level
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.7543410315100955e-6 | 5.754453620738119e-6 | 5.754572129041224e-6 |
| Standard deviation | 3.030039514718732e-10 | 3.6794184636499837e-10 | 4.494446348773569e-10 |
Outlying measurements have no (5.681632653060895e-3%) effect on estimated standard deviation.
sum/boxed vectors
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.762209002143261e-6 | 5.762455871605136e-6 | 5.762746398693253e-6 |
| Standard deviation | 7.072097454620155e-10 | 8.617953865852868e-10 | 1.0829667761090853e-9 |
Outlying measurements have no (5.6816326530612065e-3%) effect on estimated standard deviation.
sum/unboxed vectors
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.761926293457681e-6 | 5.762200981699823e-6 | 5.762549214643014e-6 |
| Standard deviation | 7.411016130622529e-10 | 1.0212932391870627e-9 | 1.488771995327922e-9 |
Outlying measurements have no (5.681632653061203e-3%) effect on estimated standard deviation.
sum/conduit, pure, fold
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8543616796795728e-4 | 1.8545189747245103e-4 | 1.8546706711665547e-4 |
| Standard deviation | 4.500373013929295e-8 | 5.3052376310271026e-8 | 6.306961355727904e-8 |
Outlying measurements have no (9.522928994082842e-3%) effect on estimated standard deviation.
sum/conduit, pure, foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.56245178285084e-4 | 6.56297140820494e-4 | 6.563445564418866e-4 |
| Standard deviation | 1.434853327981052e-7 | 1.7625135321985597e-7 | 2.247625650036078e-7 |
Outlying measurements have slight (1.2497997115846819e-2%) effect on estimated standard deviation.
sum/conduit, IO, fold
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8112654483517154e-4 | 1.811377614755824e-4 | 1.8115071627119932e-4 |
| Standard deviation | 3.2569087723018645e-8 | 4.049611338535633e-8 | 5.6269029365643614e-8 |
Outlying measurements have no (9.433106575963702e-3%) effect on estimated standard deviation.
sum/conduit, IO, foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.180219297888376e-3 | 1.1811758201898004e-3 | 1.182023229809276e-3 |
| Standard deviation | 2.399053657311654e-6 | 2.961573335971435e-6 | 3.515611916946113e-6 |
Outlying measurements have slight (1.4702606371129428e-2%) effect on estimated standard deviation.
monte carlo/low level
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.4274717669539683e-3 | 3.4292498974775156e-3 | 3.431689746083928e-3 |
| Standard deviation | 4.908269002337966e-6 | 6.743784210920239e-6 | 9.525074927614487e-6 |
Outlying measurements have slight (2.0399305555555546e-2%) effect on estimated standard deviation.
monte carlo/conduit
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.345992398479883e-2 | 1.3463418316743732e-2 | 1.346650472446629e-2 |
| Standard deviation | 6.145845876924239e-6 | 8.136416223368085e-6 | 1.1526928278230706e-5 |
Outlying measurements have slight (3.698224852070986e-2%) effect on estimated standard deviation.
sliding window/10/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6263447235447326e-3 | 1.626645041708454e-3 | 1.6273441556666735e-3 |
| Standard deviation | 6.767390956517569e-7 | 1.4998791881261344e-6 | 2.7799252208028605e-6 |
Outlying measurements have slight (1.6124697661918656e-2%) effect on estimated standard deviation.
sliding window/10/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6738785786272728e-3 | 1.6741446828618313e-3 | 1.6744343997189914e-3 |
| Standard deviation | 6.337661574807985e-7 | 9.57914753696589e-7 | 1.4375572011064374e-6 |
Outlying measurements have slight (1.612469766191867e-2%) effect on estimated standard deviation.
sliding window/10/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.183041137200511e-3 | 2.1835075996438627e-3 | 2.184178942810925e-3 |
| Standard deviation | 1.3494292053777067e-6 | 1.905171973854273e-6 | 2.882305422603209e-6 |
Outlying measurements have slight (1.7538265306122295e-2%) effect on estimated standard deviation.
sliding window/10/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.4072982480577557e-3 | 2.407711408953955e-3 | 2.4082319520334597e-3 |
| Standard deviation | 1.1299652528381399e-6 | 1.4737237472919676e-6 | 2.2714616975018044e-6 |
Outlying measurements have slight (1.8175582990397753e-2%) effect on estimated standard deviation.
sliding window/10/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.833145856415537e-3 | 7.834836410644948e-3 | 7.83753764368557e-3 |
| Standard deviation | 4.162845406784194e-6 | 6.475020235199359e-6 | 1.0369345058512056e-5 |
Outlying measurements have slight (2.8546712802768055e-2%) effect on estimated standard deviation.
sliding window/10/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.715663619936172e-3 | 8.716943263996729e-3 | 8.718462160342048e-3 |
| Standard deviation | 3.1965389831137047e-6 | 4.053034127698587e-6 | 5.454948622038106e-6 |
Outlying measurements have slight (3.02734375e-2%) effect on estimated standard deviation.
sliding window/30/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.5727840205980266e-3 | 3.57370170938355e-3 | 3.574703198598677e-3 |
| Standard deviation | 2.323114539904125e-6 | 2.9855264839360846e-6 | 4.257701224321091e-6 |
Outlying measurements have slight (2.082390221819813e-2%) effect on estimated standard deviation.
sliding window/30/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.9735229063932267e-3 | 1.974398965218041e-3 | 1.9769623658940053e-3 |
| Standard deviation | 1.8241465060781432e-6 | 4.682797093650898e-6 | 9.06122397969522e-6 |
Outlying measurements have slight (1.694411414982164e-2%) effect on estimated standard deviation.
sliding window/30/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1072799330538027e-3 | 2.107512060330707e-3 | 2.107740273174009e-3 |
| Standard deviation | 6.013831493455535e-7 | 7.602571850678668e-7 | 9.71447868433487e-7 |
Outlying measurements have slight (1.7538265306122448e-2%) effect on estimated standard deviation.
sliding window/30/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.4703347673155916e-3 | 4.471458589679281e-3 | 4.472581657016265e-3 |
| Standard deviation | 2.8548287234460216e-6 | 3.5584389960381064e-6 | 4.7520948024812996e-6 |
Outlying measurements have slight (2.271498107084911e-2%) effect on estimated standard deviation.
sliding window/30/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.07972241890516e-3 | 8.087312801160344e-3 | 8.097501139332677e-3 |
| Standard deviation | 1.8852823360199703e-5 | 2.5163927507080477e-5 | 3.3426165795874874e-5 |
Outlying measurements have slight (2.938475665748393e-2%) effect on estimated standard deviation.
sliding window/30/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.407689643070726e-3 | 8.40868808615223e-3 | 8.40980789284093e-3 |
| Standard deviation | 2.4968105333668423e-6 | 3.0207778755796944e-6 | 4.158030032443996e-6 |
Outlying measurements have slight (2.938475665748387e-2%) effect on estimated standard deviation.
sliding window/100/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.684477831990638e-3 | 9.68565790406188e-3 | 9.687801553428224e-3 |
| Standard deviation | 2.686690833137668e-6 | 3.979919172945676e-6 | 6.267001602502878e-6 |
Outlying measurements have slight (3.1217481789802066e-2%) effect on estimated standard deviation.
sliding window/100/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.053628608192978e-3 | 3.0537886384262654e-3 | 3.0539868124331133e-3 |
| Standard deviation | 4.639818330885839e-7 | 6.12186567860757e-7 | 8.032629504280269e-7 |
Outlying measurements have slight (1.9599999999999885e-2%) effect on estimated standard deviation.
sliding window/100/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.077074270049753e-3 | 2.07780896517155e-3 | 2.0783592312242096e-3 |
| Standard deviation | 1.7492015268540709e-6 | 2.243942336778958e-6 | 3.0735059816964964e-6 |
Outlying measurements have slight (1.7236072637734686e-2%) effect on estimated standard deviation.
sliding window/100/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0662185623535204e-2 | 1.0663944410453534e-2 | 1.0665662121510945e-2 |
| Standard deviation | 3.618261629000133e-6 | 4.5440534637129575e-6 | 6.1590133555100105e-6 |
Outlying measurements have slight (3.329369797859691e-2%) effect on estimated standard deviation.
sliding window/100/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.167151109252389e-3 | 9.172107203258126e-3 | 9.178715075191957e-3 |
| Standard deviation | 1.1965084013062493e-5 | 1.5385634120618966e-5 | 2.172482561829554e-5 |
Outlying measurements have slight (3.1217481789802017e-2%) effect on estimated standard deviation.
sliding window/100/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.30892777945231e-3 | 8.31012496276381e-3 | 8.311076511085433e-3 |
| Standard deviation | 2.348718591453453e-6 | 3.0971270751338295e-6 | 4.157974660609815e-6 |
Outlying measurements have slight (2.938475665748393e-2%) effect on estimated standard deviation.
sliding window/1000/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.553959456596546e-2 | 7.554151402465234e-2 | 7.554255016656282e-2 |
| Standard deviation | 1.1487686995627327e-6 | 2.430921004592891e-6 | 3.824837860889446e-6 |
Outlying measurements have slight (8.264462809917356e-2%) effect on estimated standard deviation.
sliding window/1000/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5768389859687072e-2 | 1.5773725895057912e-2 | 1.5777333314063616e-2 |
| Standard deviation | 6.768839439854426e-6 | 1.0941519994623677e-5 | 1.8720031587695113e-5 |
Outlying measurements have slight (3.993055555555529e-2%) effect on estimated standard deviation.
sliding window/1000/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8927674276428236e-3 | 1.930939937881697e-3 | 2.1199686149731565e-3 |
| Standard deviation | 2.1366589360845364e-6 | 2.4846110441386124e-4 | 5.713494681362112e-4 |
Outlying measurements have severe (0.7828688119209188%) effect on estimated standard deviation.
sliding window/1000/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.664374988128649e-2 | 7.667583474340897e-2 | 7.672582492570584e-2 |
| Standard deviation | 4.641719302352641e-5 | 6.373653591338158e-5 | 8.198058133691391e-5 |
Outlying measurements have slight (9.000000000000001e-2%) effect on estimated standard deviation.
sliding window/1000/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1661521343311254e-2 | 2.168172667486782e-2 | 2.1702026094503175e-2 |
| Standard deviation | 3.873307201380898e-5 | 4.664165415880615e-5 | 5.599863179705253e-5 |
Outlying measurements have slight (4.75e-2%) effect on estimated standard deviation.
sliding window/1000/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.778304639894896e-3 | 7.788923688839331e-3 | 7.8062040252015975e-3 |
| Standard deviation | 2.4884879990756348e-5 | 3.703941534927384e-5 | 6.343481053787911e-5 |
Outlying measurements have slight (2.8546712802768066e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.