criterion performance measurements
overview
want to understand this report?
sum/Data.List.foldl'
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.067303208425206e-5 | 8.07198463185624e-5 | 8.08050211294153e-5 |
| Standard deviation | 1.3741523335500288e-7 | 1.993119473968741e-7 | 2.8839446754337276e-7 |
Outlying measurements have no (8.19616146438081e-3%) effect on estimated standard deviation.
sum/Control.Monad.foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.011624034055862e-4 | 6.031708996852516e-4 | 6.049039686555583e-4 |
| Standard deviation | 6.252635968113406e-6 | 6.586160131626698e-6 | 7.105309310976541e-6 |
Outlying measurements have slight (1.2193263222069874e-2%) effect on estimated standard deviation.
sum/low level
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.757158740711335e-6 | 5.758173538378857e-6 | 5.759522634893695e-6 |
| Standard deviation | 3.0238294718937004e-9 | 3.929781740737584e-9 | 4.72111496397018e-9 |
Outlying measurements have no (5.681632653061224e-3%) effect on estimated standard deviation.
sum/boxed vectors
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.771454572836373e-6 | 5.775174134617033e-6 | 5.781837127867366e-6 |
| Standard deviation | 7.753079241314347e-9 | 1.6378391127046583e-8 | 2.6329095276813736e-8 |
Outlying measurements have no (5.681632653061076e-3%) effect on estimated standard deviation.
sum/unboxed vectors
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.766356230513231e-6 | 5.766778256344031e-6 | 5.76734072722361e-6 |
| Standard deviation | 1.2997998422312048e-9 | 1.6143520094956084e-9 | 2.2619525392647665e-9 |
Outlying measurements have no (5.681632653061177e-3%) effect on estimated standard deviation.
sum/conduit, pure, fold
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.62459851076306e-6 | 8.625796494570802e-6 | 8.627537790435071e-6 |
| Standard deviation | 3.564970051601737e-9 | 4.9405916952920725e-9 | 6.230131748053822e-9 |
Outlying measurements have no (5.952167521244936e-3%) effect on estimated standard deviation.
sum/conduit, pure, foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.821371144386792e-5 | 8.822673235381544e-5 | 8.824615506803203e-5 |
| Standard deviation | 3.8570299084823315e-8 | 5.1818351959837755e-8 | 7.415505566028389e-8 |
Outlying measurements have no (8.26388888888865e-3%) effect on estimated standard deviation.
sum/conduit, IO, fold
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1477972328638122e-5 | 1.1478934435329431e-5 | 1.1481367623826323e-5 |
| Standard deviation | 1.753773708530283e-9 | 4.798642971445351e-9 | 8.43961750438064e-9 |
Outlying measurements have no (6.172601365687805e-3%) effect on estimated standard deviation.
sum/conduit, IO, foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.111347742752568e-5 | 8.113185444448489e-5 | 8.11538149541583e-5 |
| Standard deviation | 6.082237398947374e-8 | 6.974800400973953e-8 | 7.905170082731158e-8 |
Outlying measurements have no (8.196161464380848e-3%) effect on estimated standard deviation.
monte carlo/low level
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.423556355966472e-3 | 3.42474788588011e-3 | 3.4260789705983013e-3 |
| Standard deviation | 3.3855170696501694e-6 | 4.278017349977473e-6 | 6.098110547192784e-6 |
Outlying measurements have slight (2.0399305555555254e-2%) effect on estimated standard deviation.
monte carlo/conduit
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0633766583864291e-2 | 1.0636014074026142e-2 | 1.0638635787666472e-2 |
| Standard deviation | 5.295440342475829e-6 | 6.740976289896081e-6 | 8.337337210665644e-6 |
Outlying measurements have slight (3.329369797859686e-2%) effect on estimated standard deviation.
sliding window/10/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.433755546658281e-3 | 1.4339300840904015e-3 | 1.4342158138119234e-3 |
| Standard deviation | 5.404100456766932e-7 | 7.470791046397037e-7 | 1.2639584188300813e-6 |
Outlying measurements have slight (1.5380859374999998e-2%) effect on estimated standard deviation.
sliding window/10/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7647708959538603e-3 | 1.7649582202758436e-3 | 1.7652004910999463e-3 |
| Standard deviation | 5.307439889567579e-7 | 7.264711147800921e-7 | 1.0122487370753399e-6 |
Outlying measurements have slight (1.638888888888889e-2%) effect on estimated standard deviation.
sliding window/10/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.307697060735648e-3 | 2.308006579834558e-3 | 2.3083765254495233e-3 |
| Standard deviation | 9.130993974824338e-7 | 1.1013327346373966e-6 | 1.3573300895270382e-6 |
Outlying measurements have slight (1.7851239669421485e-2%) effect on estimated standard deviation.
sliding window/10/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7909291128899653e-3 | 1.791186780587607e-3 | 1.7915446670484303e-3 |
| Standard deviation | 8.052321296656053e-7 | 1.0199337342563257e-6 | 1.3248693509748036e-6 |
Outlying measurements have slight (1.666187877046796e-2%) effect on estimated standard deviation.
sliding window/10/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.492638633722915e-3 | 7.497309170257165e-3 | 7.5018286917592596e-3 |
| Standard deviation | 8.958303247166203e-6 | 1.2978583831175979e-5 | 1.9269404459665193e-5 |
Outlying measurements have slight (2.8546712802768163e-2%) effect on estimated standard deviation.
sliding window/10/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.725029114238423e-3 | 7.725981873724272e-3 | 7.727161865733911e-3 |
| Standard deviation | 2.328863803851482e-6 | 2.9470742696865243e-6 | 3.843473786994938e-6 |
Outlying measurements have slight (2.8546712802768163e-2%) effect on estimated standard deviation.
sliding window/30/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.357581256788434e-3 | 3.3582667053036848e-3 | 3.358927718743588e-3 |
| Standard deviation | 1.892226914464939e-6 | 2.2842022373262424e-6 | 2.721080456445116e-6 |
Outlying measurements have slight (2.039930555555551e-2%) effect on estimated standard deviation.
sliding window/30/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0643995919089196e-3 | 2.064562874339967e-3 | 2.064741202968844e-3 |
| Standard deviation | 4.3560881646484476e-7 | 5.547882633920739e-7 | 7.177021379008462e-7 |
Outlying measurements have slight (1.7236072637734686e-2%) effect on estimated standard deviation.
sliding window/30/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.229676347674519e-3 | 2.229800648755914e-3 | 2.229923388731439e-3 |
| Standard deviation | 3.3282526063210793e-7 | 4.0800343337655994e-7 | 5.046786180861538e-7 |
Outlying measurements have slight (1.7851239669421485e-2%) effect on estimated standard deviation.
sliding window/30/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.957798654771626e-3 | 3.958506472645054e-3 | 3.959462085920789e-3 |
| Standard deviation | 2.1117310309652683e-6 | 2.6409404694030396e-6 | 3.3904349321111623e-6 |
Outlying measurements have slight (2.1728395061728398e-2%) effect on estimated standard deviation.
sliding window/30/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.5065420331595235e-3 | 7.528469387942153e-3 | 7.566212917795683e-3 |
| Standard deviation | 5.159849193849123e-5 | 8.00279617108593e-5 | 1.0804689725973929e-4 |
Outlying measurements have slight (2.8546712802768163e-2%) effect on estimated standard deviation.
sliding window/30/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.4405893728543655e-3 | 7.441502143418648e-3 | 7.442474401818854e-3 |
| Standard deviation | 2.1144367911220585e-6 | 2.770153660832705e-6 | 3.696140396227849e-6 |
Outlying measurements have slight (2.775510204081624e-2%) effect on estimated standard deviation.
sliding window/100/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.593669889036474e-3 | 9.594700869461029e-3 | 9.595763102057226e-3 |
| Standard deviation | 2.4618938219140506e-6 | 2.9134866260795166e-6 | 3.425138397203845e-6 |
Outlying measurements have slight (3.121748178980215e-2%) effect on estimated standard deviation.
sliding window/100/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.141508681383835e-3 | 3.1416328320063103e-3 | 3.141793088232728e-3 |
| Standard deviation | 3.947490112035477e-7 | 4.7484400957515567e-7 | 5.99146987060793e-7 |
Outlying measurements have slight (1.999167013744262e-2%) effect on estimated standard deviation.
sliding window/100/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.17952784524866e-3 | 2.1800626044716116e-3 | 2.180606331858677e-3 |
| Standard deviation | 1.4633739568878131e-6 | 1.8421181070354314e-6 | 2.4506437190570987e-6 |
Outlying measurements have slight (1.753826530612243e-2%) effect on estimated standard deviation.
sliding window/100/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0187372886290342e-2 | 1.0190313993952033e-2 | 1.019334422283646e-2 |
| Standard deviation | 6.827903764299878e-6 | 7.872081427667579e-6 | 9.109704225905723e-6 |
Outlying measurements have slight (3.222222222222223e-2%) effect on estimated standard deviation.
sliding window/100/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.51763968662028e-3 | 8.524843434786959e-3 | 8.531044038348415e-3 |
| Standard deviation | 1.5288887251111184e-5 | 1.9548750025840968e-5 | 2.6329269504838102e-5 |
Outlying measurements have slight (2.9384756657483767e-2%) effect on estimated standard deviation.
sliding window/100/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.350768714445918e-3 | 7.352400156351931e-3 | 7.353998280948293e-3 |
| Standard deviation | 3.866234922000348e-6 | 4.795074542533123e-6 | 6.439970525842065e-6 |
Outlying measurements have slight (2.7755102040816212e-2%) effect on estimated standard deviation.
sliding window/1000/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.552938935706177e-2 | 7.553355166585181e-2 | 7.554137867065626e-2 |
| Standard deviation | 3.9721385783036414e-6 | 9.390559833373168e-6 | 1.5396647604988064e-5 |
Outlying measurements have slight (8.264462809917332e-2%) effect on estimated standard deviation.
sliding window/1000/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5866343710220197e-2 | 1.5872063299647e-2 | 1.5874626687573914e-2 |
| Standard deviation | 4.714874326436047e-6 | 8.768389303695783e-6 | 1.658745027992085e-5 |
Outlying measurements have slight (3.9930555555555476e-2%) effect on estimated standard deviation.
sliding window/1000/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8934266901516161e-3 | 1.934962532754441e-3 | 2.0997918294427074e-3 |
| Standard deviation | 3.364332245320802e-6 | 2.7001672239327837e-4 | 5.742529961738408e-4 |
Outlying measurements have severe (0.8181111392845363%) effect on estimated standard deviation.
sliding window/1000/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.627237162916116e-2 | 7.627623603909162e-2 | 7.627925627369173e-2 |
| Standard deviation | 4.182556273872257e-6 | 5.751377002891961e-6 | 7.256836411057105e-6 |
Outlying measurements have slight (8.999999999999994e-2%) effect on estimated standard deviation.
sliding window/1000/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1005194166719608e-2 | 2.1028253514759104e-2 | 2.1050742105979972e-2 |
| Standard deviation | 4.4705933127223344e-5 | 5.5149306958155336e-5 | 7.005159116694605e-5 |
Outlying measurements have slight (4.5351473922902376e-2%) effect on estimated standard deviation.
sliding window/1000/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.806760532508497e-3 | 6.810780296934259e-3 | 6.815663726705223e-3 |
| Standard deviation | 9.153676997484933e-6 | 1.2779837626246097e-5 | 1.7865725200316664e-5 |
Outlying measurements have slight (2.7006172839506133e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.