criterion performance measurements
overview
want to understand this report?
sum/Data.List.foldl'
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.065956996647648e-5 | 8.066851301303836e-5 | 8.068619896731302e-5 |
| Standard deviation | 2.5407974813797872e-8 | 3.992244904950421e-8 | 7.016597168503847e-8 |
Outlying measurements have no (8.196161464380848e-3%) effect on estimated standard deviation.
sum/Control.Monad.foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.842372592887739e-4 | 5.842749061738654e-4 | 5.843173848219795e-4 |
| Standard deviation | 1.2085413784013693e-7 | 1.4094098389233178e-7 | 1.7222275017001099e-7 |
Outlying measurements have slight (1.2193263222069765e-2%) effect on estimated standard deviation.
sum/low level
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.754412621539532e-6 | 5.754632705563855e-6 | 5.755010938189453e-6 |
| Standard deviation | 5.47747471539394e-10 | 1.0040736429956428e-9 | 1.7581982860606895e-9 |
Outlying measurements have no (5.681632653061225e-3%) effect on estimated standard deviation.
sum/boxed vectors
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.764025791163836e-6 | 5.764466679942933e-6 | 5.765059895718508e-6 |
| Standard deviation | 1.2308615719524405e-9 | 1.5949801852768564e-9 | 2.0803121436023114e-9 |
Outlying measurements have no (5.6816326530611215e-3%) effect on estimated standard deviation.
sum/unboxed vectors
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.764494243171081e-6 | 5.764974949877927e-6 | 5.765459679948825e-6 |
| Standard deviation | 1.2882712650142253e-9 | 1.547526192651043e-9 | 1.9141609548929535e-9 |
Outlying measurements have no (5.681632653061225e-3%) effect on estimated standard deviation.
sum/conduit, pure, fold
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.614887296052375e-6 | 8.615085207403243e-6 | 8.615323179221518e-6 |
| Standard deviation | 6.085672730335036e-10 | 7.532895394042403e-10 | 9.894627759196977e-10 |
Outlying measurements have no (5.9521675212449165e-3%) effect on estimated standard deviation.
sum/conduit, pure, foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.355516630429168e-5 | 8.356186224010829e-5 | 8.356848006844966e-5 |
| Standard deviation | 1.8455259235061703e-8 | 2.2316897523092854e-8 | 2.8057865665749975e-8 |
Outlying measurements have no (8.196161464380788e-3%) effect on estimated standard deviation.
sum/conduit, IO, fold
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1480558754269431e-5 | 1.1480941658506174e-5 | 1.1481437410450051e-5 |
| Standard deviation | 1.0353700679403412e-9 | 1.4325234731143843e-9 | 2.113261641926567e-9 |
Outlying measurements have no (6.1726013656880525e-3%) effect on estimated standard deviation.
sum/conduit, IO, foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.930164619411127e-5 | 7.931796017897935e-5 | 7.933540854746231e-5 |
| Standard deviation | 4.588026657164087e-8 | 5.608861481065683e-8 | 6.807596589133285e-8 |
Outlying measurements have no (8.129535071217353e-3%) effect on estimated standard deviation.
monte carlo/low level
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.468621603600873e-3 | 3.4702818571291758e-3 | 3.471678181398824e-3 |
| Standard deviation | 4.128110783919427e-6 | 5.122190451742985e-6 | 7.24818757703174e-6 |
Outlying measurements have slight (2.0823902218198184e-2%) effect on estimated standard deviation.
monte carlo/conduit, ConduitM primitives
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5761479391294346e-2 | 1.5784378490364525e-2 | 1.5802088419419755e-2 |
| Standard deviation | 3.476611060925457e-5 | 4.911114248078068e-5 | 6.997118398669895e-5 |
Outlying measurements have slight (3.993055555555551e-2%) effect on estimated standard deviation.
monte carlo/conduit, ConduitM primitives, Codensity
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.201116136559248e-2 | 2.2025541652676645e-2 | 2.2051873339856847e-2 |
| Standard deviation | 2.7627936315397216e-5 | 4.246870603651231e-5 | 6.671778234140492e-5 |
Outlying measurements have slight (4.750000000000001e-2%) effect on estimated standard deviation.
monte carlo/conduit, ConduitM primitives, explicit binding order
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.1898400560597075e-3 | 5.1913310144913235e-3 | 5.193175718369111e-3 |
| Standard deviation | 4.163248472560582e-6 | 5.2708228667269145e-6 | 6.730119285616033e-6 |
Outlying measurements have slight (2.4375e-2%) effect on estimated standard deviation.
monte carlo/conduit, Pipe primitives
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5471177838481424e-2 | 1.5476630052212212e-2 | 1.5483700611579635e-2 |
| Standard deviation | 1.1127094419313163e-5 | 1.6238946583610072e-5 | 2.1673924738439524e-5 |
Outlying measurements have slight (3.9930555555555414e-2%) effect on estimated standard deviation.
monte carlo/conduit, Pipe constructos
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.843396575441765e-3 | 4.84489279452913e-3 | 4.846964816251458e-3 |
| Standard deviation | 3.9763328214235104e-6 | 5.475680511907445e-6 | 8.140812372889953e-6 |
Outlying measurements have slight (2.324263038548744e-2%) effect on estimated standard deviation.
sliding window/10/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3526646782027522e-3 | 1.3527744204972923e-3 | 1.3530110911294906e-3 |
| Standard deviation | 2.955116223106822e-7 | 4.923769164000065e-7 | 9.156764929057416e-7 |
Outlying measurements have slight (1.514792899408277e-2%) effect on estimated standard deviation.
sliding window/10/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7391117923924691e-3 | 1.739234968656599e-3 | 1.739370792696255e-3 |
| Standard deviation | 3.724059867868702e-7 | 4.4971597482458963e-7 | 5.582546330272438e-7 |
Outlying measurements have slight (1.6388888888888887e-2%) effect on estimated standard deviation.
sliding window/10/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.172913667485704e-3 | 2.1731267104408755e-3 | 2.1733628466452564e-3 |
| Standard deviation | 6.19946584788987e-7 | 7.435238469265922e-7 | 9.59480865088757e-7 |
Outlying measurements have slight (1.75382653061223e-2%) effect on estimated standard deviation.
sliding window/10/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7915872086630353e-3 | 1.7919233339981039e-3 | 1.7923226085130443e-3 |
| Standard deviation | 1.0090555216299087e-6 | 1.3017997467562972e-6 | 1.679277054803635e-6 |
Outlying measurements have slight (1.666187877046811e-2%) effect on estimated standard deviation.
sliding window/10/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.45759740319892e-3 | 7.4598164581618e-3 | 7.462183437193626e-3 |
| Standard deviation | 5.5962068439259386e-6 | 6.883815071047344e-6 | 8.597655809479361e-6 |
Outlying measurements have slight (2.7755102040816156e-2%) effect on estimated standard deviation.
sliding window/10/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.81414950730993e-3 | 7.814932461556926e-3 | 7.815830839631022e-3 |
| Standard deviation | 1.7551130329018508e-6 | 2.3649953159083152e-6 | 3.1524156060397802e-6 |
Outlying measurements have slight (2.8546712802768166e-2%) effect on estimated standard deviation.
sliding window/30/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.4405274229914753e-3 | 3.441171289421724e-3 | 3.4419414614259257e-3 |
| Standard deviation | 1.953306393646736e-6 | 2.3408970131815534e-6 | 2.8610807748247496e-6 |
Outlying measurements have slight (2.082390221819824e-2%) effect on estimated standard deviation.
sliding window/30/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.046244686873941e-3 | 2.046380204237562e-3 | 2.0465732948174134e-3 |
| Standard deviation | 4.0082898659279606e-7 | 5.388829801286559e-7 | 8.501956395930682e-7 |
Outlying measurements have slight (1.723607263773469e-2%) effect on estimated standard deviation.
sliding window/30/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.096093102977742e-3 | 2.096323997175508e-3 | 2.096703507874978e-3 |
| Standard deviation | 6.920533289166901e-7 | 1.0065051061323785e-6 | 1.4638456080296715e-6 |
Outlying measurements have slight (1.7538265306122448e-2%) effect on estimated standard deviation.
sliding window/30/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.913969908763736e-3 | 3.91508245156891e-3 | 3.916534769363819e-3 |
| Standard deviation | 2.8729017346099914e-6 | 4.121290939337299e-6 | 6.624060782700059e-6 |
Outlying measurements have slight (2.1728395061728398e-2%) effect on estimated standard deviation.
sliding window/30/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.489493127027179e-3 | 7.490832107944578e-3 | 7.4926498271106045e-3 |
| Standard deviation | 3.190156869053635e-6 | 4.293593710447965e-6 | 6.203008045855209e-6 |
Outlying measurements have slight (2.775510204081632e-2%) effect on estimated standard deviation.
sliding window/30/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.481192546048042e-3 | 7.482533644135223e-3 | 7.4843317077191345e-3 |
| Standard deviation | 3.3479624234482957e-6 | 4.5539397162950845e-6 | 6.4418015245584885e-6 |
Outlying measurements have slight (2.7755102040816326e-2%) effect on estimated standard deviation.
sliding window/100/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0478306520018505e-2 | 1.0483906307161684e-2 | 1.0503725060803581e-2 |
| Standard deviation | 4.813960757882597e-6 | 2.622430034130168e-5 | 5.352473368020901e-5 |
Outlying measurements have slight (3.2222222222222124e-2%) effect on estimated standard deviation.
sliding window/100/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.1369570330653253e-3 | 3.138080515392989e-3 | 3.1396499610659157e-3 |
| Standard deviation | 3.166483720689194e-6 | 4.25627352565169e-6 | 6.111857096747686e-6 |
Outlying measurements have slight (1.9991670137442612e-2%) effect on estimated standard deviation.
sliding window/100/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.067595994587502e-3 | 2.069617322345205e-3 | 2.0720676423318647e-3 |
| Standard deviation | 6.502625748664143e-6 | 7.171425585100934e-6 | 7.862403558722112e-6 |
Outlying measurements have slight (1.7236072637734686e-2%) effect on estimated standard deviation.
sliding window/100/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1065321176221086e-2 | 1.106848685807341e-2 | 1.1074477423462759e-2 |
| Standard deviation | 6.9610497217988226e-6 | 1.076503532349315e-5 | 1.5911454262575558e-5 |
Outlying measurements have slight (3.329369797859683e-2%) effect on estimated standard deviation.
sliding window/100/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.558120379218341e-3 | 8.561677027165104e-3 | 8.566627975010328e-3 |
| Standard deviation | 8.299667112775489e-6 | 1.1430816741259301e-5 | 1.7500341899429326e-5 |
Outlying measurements have slight (3.0273437499999997e-2%) effect on estimated standard deviation.
sliding window/100/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.252947219999909e-3 | 7.2710024774443665e-3 | 7.342785992038761e-3 |
| Standard deviation | 5.593476853765493e-6 | 9.895345340489151e-5 | 2.0616188665884558e-4 |
Outlying measurements have slight (2.7755102040816312e-2%) effect on estimated standard deviation.
sliding window/1000/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.963374602889505e-2 | 8.96450087307657e-2 | 8.966334081171347e-2 |
| Standard deviation | 5.649205118194016e-6 | 2.3740108640893234e-5 | 2.971847964501684e-5 |
Outlying measurements have slight (9.0e-2%) effect on estimated standard deviation.
sliding window/1000/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5871278593983726e-2 | 1.5875131819062714e-2 | 1.587987282643851e-2 |
| Standard deviation | 7.701224599638815e-6 | 1.081501717441649e-5 | 1.483094304996885e-5 |
Outlying measurements have slight (3.993055555555543e-2%) effect on estimated standard deviation.
sliding window/1000/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.869313377274502e-3 | 1.9092890056287243e-3 | 2.0639528971656876e-3 |
| Standard deviation | 5.871802030097757e-6 | 2.5250205015426295e-4 | 5.362143068626006e-4 |
Outlying measurements have severe (0.8002303073563125%) effect on estimated standard deviation.
sliding window/1000/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.034155810663917e-2 | 9.035802410967525e-2 | 9.037295629949921e-2 |
| Standard deviation | 1.7977765625972805e-5 | 2.4718588080055677e-5 | 3.432225493671321e-5 |
Outlying measurements have slight (9.000000000000001e-2%) effect on estimated standard deviation.
sliding window/1000/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0996476066078275e-2 | 2.1007513503568144e-2 | 2.1020887649093954e-2 |
| Standard deviation | 2.0692045915082105e-5 | 2.7772798747798234e-5 | 3.584090473511911e-5 |
Outlying measurements have slight (4.5351473922902494e-2%) effect on estimated standard deviation.
sliding window/1000/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.727542535482776e-3 | 6.733433662674848e-3 | 6.739396751063569e-3 |
| Standard deviation | 1.2337249169071239e-5 | 1.6293028607252863e-5 | 2.1809990325873184e-5 |
Outlying measurements have slight (2.7006172839506046e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.