criterion performance measurements
overview
want to understand this report?
sum/Data.List.foldl'
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.091594041034686e-5 | 8.092091053864229e-5 | 8.092706505040016e-5 |
| Standard deviation | 1.467933940674516e-8 | 1.8471114932548828e-8 | 2.455026343862431e-8 |
Outlying measurements have no (8.196161464380848e-3%) effect on estimated standard deviation.
sum/Control.Monad.foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.882807141640935e-4 | 5.883726264555551e-4 | 5.885267249498629e-4 |
| Standard deviation | 2.453528204719336e-7 | 3.928724502321958e-7 | 6.153549337114866e-7 |
Outlying measurements have slight (1.2193263222069806e-2%) effect on estimated standard deviation.
sum/low level
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.754451874247458e-6 | 5.7548241459919545e-6 | 5.756129979781408e-6 |
| Standard deviation | 5.381498517651273e-10 | 2.0995217550281856e-9 | 4.260172994766873e-9 |
Outlying measurements have no (5.681632653061087e-3%) effect on estimated standard deviation.
sum/boxed vectors
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.7616626711131035e-6 | 5.761889990742393e-6 | 5.762145529329323e-6 |
| Standard deviation | 6.970184495831387e-10 | 8.253880035320092e-10 | 1.0106292367378116e-9 |
Outlying measurements have no (5.681632653061224e-3%) effect on estimated standard deviation.
sum/unboxed vectors
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.762589956659088e-6 | 5.76285575636608e-6 | 5.76319281443714e-6 |
| Standard deviation | 7.593783798715292e-10 | 9.75816667531193e-10 | 1.3335662403744115e-9 |
Outlying measurements have no (5.681632653061225e-3%) effect on estimated standard deviation.
sum/conduit, pure, fold
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.620177199081375e-6 | 8.620472275756653e-6 | 8.620917254257966e-6 |
| Standard deviation | 8.906471854011784e-10 | 1.1589824354371925e-9 | 1.6934808173822908e-9 |
Outlying measurements have no (5.952167521244936e-3%) effect on estimated standard deviation.
sum/conduit, pure, foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.021030264002661e-5 | 9.021490790034398e-5 | 9.022112509582912e-5 |
| Standard deviation | 1.4394629454281839e-8 | 1.770527916486982e-8 | 2.2388961084140825e-8 |
Outlying measurements have no (8.332744862650761e-3%) effect on estimated standard deviation.
sum/conduit, IO, fold
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1476337083925503e-5 | 1.1476634584963182e-5 | 1.1477233094081153e-5 |
| Standard deviation | 9.956645044557004e-10 | 1.349320593873456e-9 | 2.1120185823227134e-9 |
Outlying measurements have no (6.172601365688052e-3%) effect on estimated standard deviation.
sum/conduit, IO, foldM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.460116122695241e-5 | 8.465304689114422e-5 | 8.474560968110133e-5 |
| Standard deviation | 1.6277341065556115e-7 | 2.2660871991410745e-7 | 3.43875603427452e-7 |
Outlying measurements have no (8.26388888888889e-3%) effect on estimated standard deviation.
monte carlo/conduit, stream fusion
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.253131673242632e-3 | 8.258963177408768e-3 | 8.271730043859865e-3 |
| Standard deviation | 1.5186713255778434e-5 | 2.4870772148564208e-5 | 4.353505877505575e-5 |
Outlying measurements have slight (2.9384756657483774e-2%) effect on estimated standard deviation.
monte carlo/low level
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.4326715574835697e-3 | 3.4360382644052613e-3 | 3.4401614024322812e-3 |
| Standard deviation | 9.770973151965788e-6 | 1.189664304664663e-5 | 1.4718211246790648e-5 |
Outlying measurements have slight (2.0823902218198277e-2%) effect on estimated standard deviation.
monte carlo/conduit, stream
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.773133562804749e-3 | 4.776566171183987e-3 | 4.781786256970655e-3 |
| Standard deviation | 9.659095669042616e-6 | 1.251577503451541e-5 | 1.6588050242117556e-5 |
Outlying measurements have slight (2.3242630385487528e-2%) effect on estimated standard deviation.
monte carlo/conduit, ConduitM primitives
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.271996282088649e-3 | 5.276192006542395e-3 | 5.280354508561703e-3 |
| Standard deviation | 1.0246037551717183e-5 | 1.2504696240291832e-5 | 1.5017443369794541e-5 |
Outlying measurements have slight (2.4375e-2%) effect on estimated standard deviation.
monte carlo/conduit, ConduitM primitives, Codensity
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.514092514398126e-3 | 6.518285240206057e-3 | 6.523643717865826e-3 |
| Standard deviation | 1.092032674096062e-5 | 1.3770368097696992e-5 | 1.711226624009795e-5 |
Outlying measurements have slight (2.629656683710726e-2%) effect on estimated standard deviation.
monte carlo/conduit, ConduitM primitives, explicit binding order
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.270688682633458e-3 | 5.275350945494108e-3 | 5.281676481234446e-3 |
| Standard deviation | 1.1905966164587227e-5 | 1.6275432998620384e-5 | 2.397554880146709e-5 |
Outlying measurements have slight (2.4375000000000004e-2%) effect on estimated standard deviation.
monte carlo/conduit, Pipe primitives
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6172263308478655e-2 | 1.618851681197337e-2 | 1.622009326794064e-2 |
| Standard deviation | 3.122623255850061e-5 | 5.4330843341624857e-5 | 1.0006949620985029e-4 |
Outlying measurements have slight (3.993055555555545e-2%) effect on estimated standard deviation.
monte carlo/conduit, Pipe constructos
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.876059306318267e-3 | 4.881512085857501e-3 | 4.888425190470118e-3 |
| Standard deviation | 1.5619571586911622e-5 | 1.9092012359922357e-5 | 2.43813547469023e-5 |
Outlying measurements have slight (2.3795359904818368e-2%) effect on estimated standard deviation.
sliding window/10/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6472466155062454e-3 | 1.6488329634360597e-3 | 1.651442325180042e-3 |
| Standard deviation | 4.1392184366538905e-6 | 6.925345097909881e-6 | 1.0854306634670237e-5 |
Outlying measurements have slight (1.6124697661918756e-2%) effect on estimated standard deviation.
sliding window/10/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.7238692659812308e-3 | 1.72496672961846e-3 | 1.7268567392209769e-3 |
| Standard deviation | 3.598844949525555e-6 | 4.680746553573781e-6 | 6.5856427457470115e-6 |
Outlying measurements have slight (1.638888888888889e-2%) effect on estimated standard deviation.
sliding window/10/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.134348099438947e-3 | 2.1358658523623945e-3 | 2.138424418764505e-3 |
| Standard deviation | 4.008225873152173e-6 | 6.607722930046053e-6 | 1.0798875298825376e-5 |
Outlying measurements have slight (1.7538265306122448e-2%) effect on estimated standard deviation.
sliding window/10/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.013972934849674e-3 | 2.0158282355402894e-3 | 2.018834914203063e-3 |
| Standard deviation | 4.781289874382635e-6 | 7.9132819557204e-6 | 1.1960434120144488e-5 |
Outlying measurements have slight (1.7236072637734686e-2%) effect on estimated standard deviation.
sliding window/10/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.173020453811222e-3 | 4.1760583176076085e-3 | 4.180595428157465e-3 |
| Standard deviation | 8.587043523405868e-6 | 1.1130551302547828e-5 | 1.4588360661710835e-5 |
Outlying measurements have slight (2.2210743801652756e-2%) effect on estimated standard deviation.
sliding window/10/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.256983024257013e-3 | 4.2605806506288675e-3 | 4.266666916835026e-3 |
| Standard deviation | 9.46820068374062e-6 | 1.4357434797362394e-5 | 2.2855067640173208e-5 |
Outlying measurements have slight (2.221074380165263e-2%) effect on estimated standard deviation.
sliding window/30/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.5137412948345317e-3 | 3.5157234276002314e-3 | 3.519069714181917e-3 |
| Standard deviation | 5.567827254140382e-6 | 8.205647116724825e-6 | 1.1332444517297673e-5 |
Outlying measurements have slight (2.0823902218198277e-2%) effect on estimated standard deviation.
sliding window/30/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.022581938940886e-3 | 2.0242389679873845e-3 | 2.0264197189779993e-3 |
| Standard deviation | 4.751064059886647e-6 | 6.343857323377534e-6 | 9.303477207141359e-6 |
Outlying measurements have slight (1.7236072637734686e-2%) effect on estimated standard deviation.
sliding window/30/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0585655457193465e-3 | 2.0600730466896087e-3 | 2.0626747607942167e-3 |
| Standard deviation | 4.9353685118565595e-6 | 6.697327254039141e-6 | 9.765613152073928e-6 |
Outlying measurements have slight (1.7236072637734686e-2%) effect on estimated standard deviation.
sliding window/30/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.038248649331312e-3 | 4.0408734303492575e-3 | 4.0444781158894385e-3 |
| Standard deviation | 6.8824313745636505e-6 | 9.702839416241223e-6 | 1.2312820932448278e-5 |
Outlying measurements have slight (2.1728395061728308e-2%) effect on estimated standard deviation.
sliding window/30/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.425089055953215e-3 | 4.428718966604835e-3 | 4.435648508397139e-3 |
| Standard deviation | 9.275900646519568e-6 | 1.575208119097481e-5 | 2.5799454591808568e-5 |
Outlying measurements have slight (2.271498107084876e-2%) effect on estimated standard deviation.
sliding window/30/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.185057921864447e-3 | 4.1875439782907056e-3 | 4.1920821005788965e-3 |
| Standard deviation | 6.7895805377417215e-6 | 1.01565203463177e-5 | 1.7134990362178224e-5 |
Outlying measurements have slight (2.2210743801652895e-2%) effect on estimated standard deviation.
sliding window/100/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.666569026488468e-3 | 9.67186678133863e-3 | 9.678945188845987e-3 |
| Standard deviation | 1.4179855573171124e-5 | 1.746119421534582e-5 | 2.195613840001095e-5 |
Outlying measurements have slight (3.1217481789801948e-2%) effect on estimated standard deviation.
sliding window/100/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.1014579748012056e-3 | 3.1034435570137416e-3 | 3.107857267922585e-3 |
| Standard deviation | 4.804819242657887e-6 | 9.153556967839823e-6 | 1.6710713707851066e-5 |
Outlying measurements have slight (1.999167013744273e-2%) effect on estimated standard deviation.
sliding window/100/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0304989432279553e-3 | 2.0322560420480327e-3 | 2.0345497323940856e-3 |
| Standard deviation | 4.773047824183959e-6 | 6.276983697458527e-6 | 8.360152180537352e-6 |
Outlying measurements have slight (1.7236072637734686e-2%) effect on estimated standard deviation.
sliding window/100/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.027123674601107e-2 | 1.0280761436659503e-2 | 1.0294514100885901e-2 |
| Standard deviation | 1.9821299233230366e-5 | 3.2297514802670876e-5 | 4.71850138714446e-5 |
Outlying measurements have slight (3.2222222222222215e-2%) effect on estimated standard deviation.
sliding window/100/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.4573894162667745e-3 | 5.460872564143798e-3 | 5.4666960094290305e-3 |
| Standard deviation | 8.698494533026172e-6 | 1.336161422312811e-5 | 2.0078336476992952e-5 |
Outlying measurements have slight (2.4375e-2%) effect on estimated standard deviation.
sliding window/100/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.1354323306087605e-3 | 4.138474197330554e-3 | 4.143908282995059e-3 |
| Standard deviation | 7.873948312858421e-6 | 1.2501873003743917e-5 | 2.0565676397992192e-5 |
Outlying measurements have slight (2.2210743801652895e-2%) effect on estimated standard deviation.
sliding window/1000/low level, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.549517627459404e-2 | 7.553873940648279e-2 | 7.5575813760112e-2 |
| Standard deviation | 5.0658841147182914e-5 | 6.822366967348134e-5 | 8.482328442133266e-5 |
Outlying measurements have slight (8.264462809917356e-2%) effect on estimated standard deviation.
sliding window/1000/low level, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5818468973285444e-2 | 1.583254513077027e-2 | 1.5863700975511683e-2 |
| Standard deviation | 2.382920719541112e-5 | 5.2831917706781453e-5 | 9.156277684755489e-5 |
Outlying measurements have slight (3.99305555555555e-2%) effect on estimated standard deviation.
sliding window/1000/low level, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.834554917745861e-3 | 1.8741992999316674e-3 | 2.029652802126416e-3 |
| Standard deviation | 5.818656394509239e-6 | 2.52274220573295e-4 | 5.362656527467201e-4 |
Outlying measurements have severe (0.8005810571968061%) effect on estimated standard deviation.
sliding window/1000/conduit, Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.620182935490194e-2 | 7.627375197048512e-2 | 7.635256317248823e-2 |
| Standard deviation | 9.534141245131711e-5 | 1.2052232108558411e-4 | 1.495646031843688e-4 |
Outlying measurements have slight (9.000000000000001e-2%) effect on estimated standard deviation.
sliding window/1000/conduit, boxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.813423821009181e-2 | 1.8152172845586262e-2 | 1.817317383146529e-2 |
| Standard deviation | 3.767581318787379e-5 | 4.579297736529254e-5 | 5.6355819079756716e-5 |
Outlying measurements have slight (4.338842975206612e-2%) effect on estimated standard deviation.
sliding window/1000/conduit, unboxed Vector
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.034788300917793e-3 | 4.045083064460538e-3 | 4.057004414324823e-3 |
| Standard deviation | 2.841073486330649e-5 | 3.588873357342879e-5 | 4.545595149447844e-5 |
Outlying measurements have slight (2.1728395061728394e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.