freetype-devel
[Top][All Lists]
Advanced
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: -warmup

From: Ahmet Göksu
Subject: Re: -warmup
Date: Wed, 16 Aug 2023 23:32:29 +0300
Hi,
I have added the total table that you suggested.

I think Get_Char_Index is not the problem, the results varies all the time. Here are the three results that i had in the same minute (one has different flags).

Should I proceed to detect outliers? 

Since we do not get the same error rate consistently,  I think we will not find the target we expected by outliers.

Best,
Goksu
goksu.in
On 7 Aug 2023 15:57 +0300, Werner LEMBERG <wl@gnu.org>, wrote:

What exactly means 'Baseline (ms)'? Is the shown number the time
 for one loop? For all loops together? Please clarify and mention
 this on the HTML page.

Clarified that the times are milliseconds for the cumulative time
for all iterations.

Thanks. The sentence is not easily comprehensible. Perhaps change it
to something like

```
Cumulative time for all iterations. Smaller values means better.
```

BTW, in column 'N' I see stuff like '68160 | 65880'. What does this
mean? Please add an explanatory comment to the HTML page.

Another thing: Please mention on the HTML page the completion time for
each test, and the total execution time of all tests together.

Looking at the 'Load_Advances (Unscaled)' row, I think that 100%
 difference between 0.001 and 0.002 doesn't make any sense. How do
 you compute the percentage? Is this based on the cumulative time
of  all loops? If so, and you really get such small numbers, there
must  be some fine-tuning for high-speed tests (for example,
increasing N  for this particular test by a factor of 10, say) to
get meaningful  timing values.

it was cumulative time in milliseconds but converted it microseconds
as how it was and it seem got better.

We are getting nearer, again :-)

What worries me, though, is that we still have such enormous
differences. For `Get_Char_Index` I think it's lack of precision.
Please try to fix this – if the ratio

cumulative_time / N

is smaller than a given threshold, N must be increased a lot. In
other words, for `Roboto_subset.ttf`, N should be set to, say, 10*N.

For the other large differences I think we need some statistical
analysis to get better results – simple cumulation is not good enough.
In particular, outliers should be removed (at least this is my
hypothesis). Maybe you can look up the internet to find some simple
code to handle them.

An idea to identify outliers could be to split the cumulation time
into, say, 100 smaller intervals. You can the discard the too-large
values and compute the mean of the remaining data. My reasoning is
that other CPU activity happens in parallel, but only for short
amounts of time.

Have you actually done a statistical analysis of, say, 'Load_Advances
(Normal)' for `Arial_subset.ttf`? For example, printing all timings
of the datapoints as histograms for runs A and B? *Are* there
outliers? Maybe there is another statistical mean value that gives
more meaningful results.


Werner

Freetype Benchmark Results

Warning: Baseline and Benchmark have the same commit ID!

Info

InfoBaselineBenchmark
Parameters-c 1000 -w 50-c 1000 -w 50
Commit ID4bcd97114bcd9711
Commit Date2023-08-07 15:11:28 +03002023-08-07 15:11:28 +0300
BranchGSoC-2023-AhmetGSoC-2023-Ahmet

* Cumulative time for all iterations. Smaller values are better.

** N count in (x | y) format is for showing baseline and benchmark N counts seperately when they differs.

Total Results

TestNBaseline (µs) Benchmark (µs)Difference (%)
Load500000 3482359 3519568-1.1
Load_Advances (Normal)500000 2922257 3129046-7.1
Load_Advances (Fast)500000 13671 15627-14.3
Load_Advances (Unscaled)500000 12459 15309-22.9
Render500000 1827123 1878593-2.8
Get_Glyph500000 768199 7543151.8
Get_Char_Index470000 13396 127994.5
Iterate CMap5000 9860 93784.9
New_Face5000 248768 2405913.3
Embolden500000 1107786 1109611-0.2
Stroke294205 | 289365 7804180 77762470.4
Get_BBox500000 491174 496942-1.2
Get_CBox500000 355009 355822-0.2
New_Face & load glyph(s)500000 571172 5470084.2
TOTAL5774205 | 5769365 1962741319860856 -1.2

Results for Roboto_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load120000 6119135767105.8
Load_Advances (Normal)120000 498485538696-8.1
Load_Advances (Fast)120000 29644976-67.9
Load_Advances (Unscaled)120000 27905476-96.3
Render120000 419155437405-4.4
Get_Glyph120000 1738201661314.4
Get_Char_Index94000 270126412.2
Iterate CMap1000 2084186110.7
New_Face1000 4206642190-0.3
Embolden120000 220170222545-1.1
Stroke62880 | 61080 163185216240320.5
Get_BBox120000 1081191078920.2
Get_CBox120000 8159783621-2.5
New_Face & load glyph(s)120000 104217107909-3.5
TOTAL2715960 390193339220850.5

Results for Arial_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 8533098365162.0
Load_Advances (Normal)95000 680709732802-7.7
Load_Advances (Fast)95000 3067244820.2
Load_Advances (Unscaled)95000 2527225610.7
Render95000 3577913444293.7
Get_Glyph95000 143432147740-3.0
Get_Char_Index94000 27132881-6.2
Iterate CMap1000 18261829-0.2
New_Face1000 4777650312-5.3
Embolden95000 199248213090-6.9
Stroke57665 | 56430 148541114812960.3
Get_BBox95000 95626946631.0
Get_CBox95000 6872069159-0.6
New_Face & load glyph(s)95000 103873105626-1.7
TOTAL2206095 404602840850471.0

Results for TimesNewRoman_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 916144926065-1.1
Load_Advances (Normal)95000 813784875632-7.6
Load_Advances (Fast)95000 24172424-0.3
Load_Advances (Unscaled)95000 226922580.5
Render95000 378067383493-1.4
Get_Glyph95000 1586901478196.9
Get_Char_Index94000 3060241821.0
Iterate CMap1000 2331190718.2
New_Face1000 646015439115.8
Embolden95000 2847732650596.9
Stroke44365 | 44460 15442481545269-0.1
Get_BBox95000 109597115606-5.5
Get_CBox95000 7054071610-1.5
New_Face & load glyph(s)95000 130336134454-3.2
TOTAL2180825 448085745284051.1

Results for Tahoma_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 568933628873-10.5
Load_Advances (Normal)95000 481014513665-6.8
Load_Advances (Fast)95000 24772870-15.9
Load_Advances (Unscaled)95000 22732423-6.6
Render95000 330628358159-8.3
Get_Glyph95000 143448146595-2.2
Get_Char_Index94000 245824261.3
Iterate CMap1000 18181909-5.0
New_Face1000 49076464685.3
Embolden95000 206672209651-1.4
Stroke62700 | 61845 156047915474800.8
Get_BBox95000 8971989857-0.2
Get_CBox95000 6536465460-0.1
New_Face & load glyph(s)95000 102598987873.7
TOTAL2216545 360695737146233.0

Results for Verdana_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 532060551404-3.6
Load_Advances (Normal)95000 448265468251-4.5
Load_Advances (Fast)95000 27462909-5.9
Load_Advances (Unscaled)95000 26002896-11.4
Render95000 341482355107-4.0
Get_Glyph95000 1488091460301.9
Get_Char_Index94000 246424331.3
Iterate CMap1000 18011872-3.9
New_Face1000 4524947230-4.4
Embolden95000 196923199266-1.2
Stroke66595 | 65550 158219015781700.3
Get_BBox95000 8811388924-0.9
Get_CBox95000 68788659724.1
New_Face & load glyph(s)95000 13014810023223.0
TOTAL2224145 359163836106960.5

Freetype Benchmark

Freetype Benchmark Results

Warning: Baseline and Benchmark have the same commit ID!

Info

InfoBaselineBenchmark
Parameters-c 550 -w 50-c 550 -w 50
Commit ID4bcd97114bcd9711
Commit Date2023-08-07 15:11:28 +03002023-08-07 15:11:28 +0300
BranchGSoC-2023-AhmetGSoC-2023-Ahmet

* Cumulative time for all iterations. Smaller values are better.

** N count in (x | y) format is for showing baseline and benchmark N counts seperately when they differs.

Total Results

TestNBaseline (µs) Benchmark (µs)Difference (%)
Load275000 1940945 1960259-1.0
Load_Advances (Normal)275000 1693643 1757029-3.7
Load_Advances (Fast)275000 6966 68471.7
Load_Advances (Unscaled)275000 6382 63220.9
Render275000 968631 1047055-8.1
Get_Glyph275000 410428 445338-8.5
Get_Char_Index258500 6847 7183-4.9
Iterate CMap2750 5289 51642.4
New_Face2750 137845 1372680.4
Embolden275000 584074 5811100.5
Stroke262730 | 262890 7008914 7043463-0.5
Get_BBox275000 260539 272939-4.8
Get_CBox275000 188954 210858-11.6
New_Face & load glyph(s)275000 297956 304196-2.1
TOTAL3276730 | 3276890 1351741313785031 -2.0

Results for Roboto_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load66000 3138892994864.6
Load_Advances (Normal)66000 287800296824-3.1
Load_Advances (Fast)66000 15561615-3.8
Load_Advances (Unscaled)66000 14631486-1.6
Render66000 223554243002-8.7
Get_Glyph66000 92555118479-28.0
Get_Char_Index51700 13681536-12.3
Iterate CMap550 9791078-10.1
New_Face550 2278125196-10.6
Embolden66000 1177911164941.1
Stroke62280 | 61680 162831316204860.5
Get_BBox66000 5578760158-7.8
Get_CBox66000 4288156697-32.2
New_Face & load glyph(s)66000 6467667592-4.5
TOTAL1549560 285539329101291.9

Results for Arial_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load52250 504744534834-6.0
Load_Advances (Normal)52250 351531416852-18.6
Load_Advances (Fast)52250 12671300-2.6
Load_Advances (Unscaled)52250 11971203-0.5
Render52250 183606192968-5.1
Get_Glyph52250 79899790171.1
Get_Char_Index51700 16531664-0.7
Iterate CMap550 104110410.0
New_Face550 376372978520.9
Embolden52250 1086371040534.2
Stroke52250 13405441359642-1.4
Get_BBox52250 49978495880.8
Get_CBox52250 3629136601-0.9
New_Face & load glyph(s)52250 56463551802.3
TOTAL1255100 275448828637284.0

Results for TimesNewRoman_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load52250 528080557600-5.6
Load_Advances (Normal)52250 4962444591307.5
Load_Advances (Fast)52250 1523135810.8
Load_Advances (Unscaled)52250 131312306.3
Render52250 208907217202-4.0
Get_Glyph52250 79182776821.9
Get_Char_Index51700 127312720.1
Iterate CMap550 9819641.7
New_Face550 29084267618.0
Embolden52250 139582139743-0.1
Stroke43700 | 44460 15395451540181-0.0
Get_BBox52250 5772967301-16.6
Get_CBox52250 3704346742-26.2
New_Face & load glyph(s)52250 6764671096-5.1
TOTAL1238760 318813232082620.6

Results for Tahoma_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load52250 3184273076733.4
Load_Advances (Normal)52250 270798277004-2.3
Load_Advances (Fast)52250 12741277-0.2
Load_Advances (Unscaled)52250 121611872.4
Render52250 175500197229-12.4
Get_Glyph52250 8060184321-4.6
Get_Char_Index51700 12691435-13.1
Iterate CMap550 1303104020.2
New_Face550 2435627740-13.9
Embolden52250 111778113705-1.7
Stroke52250 12508291295714-3.6
Get_BBox52250 4772849956-4.7
Get_CBox52250 3545536577-3.2
New_Face & load glyph(s)52250 5241957650-10.0
TOTAL1255100 237295324525083.4

Results for Verdana_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load52250 2758052606665.5
Load_Advances (Normal)52250 287270307219-6.9
Load_Advances (Fast)52250 134612973.6
Load_Advances (Unscaled)52250 11931216-1.9
Render52250 177064196654-11.1
Get_Glyph52250 7819185839-9.8
Get_Char_Index51700 128412760.6
Iterate CMap550 9851041-5.7
New_Face550 2398727786-15.8
Embolden52250 106286107115-0.8
Stroke52250 124968312274401.8
Get_BBox52250 49317459366.9
Get_CBox52250 37284342418.2
New_Face & load glyph(s)52250 56752526787.2
TOTAL1255100 234644723504040.2

Freetype Benchmark

Freetype Benchmark Results

Warning: Baseline and Benchmark have the same commit ID!

Info

InfoBaselineBenchmark
Parameters-c 1000 -w 50-c 1000 -w 50
Commit ID4bcd97114bcd9711
Commit Date2023-08-07 15:11:28 +03002023-08-07 15:11:28 +0300
BranchGSoC-2023-AhmetGSoC-2023-Ahmet

* Cumulative time for all iterations. Smaller values are better.

** N count in (x | y) format is for showing baseline and benchmark N counts seperately when they differs.

Total Results

TestNBaseline (µs) Benchmark (µs)Difference (%)
Load500000 3574511 33847525.3
Load_Advances (Normal)500000 3001429 3115941-3.8
Load_Advances (Fast)500000 12871 13708-6.5
Load_Advances (Unscaled)500000 11982 12662-5.7
Render500000 1829127 1895816-3.6
Get_Glyph500000 755948 7476541.1
Get_Char_Index470000 12153 12350-1.6
Iterate CMap5000 9339 91452.1
New_Face5000 233723 235328-0.7
Embolden500000 1094561 1096360-0.2
Stroke292330 | 294035 7764649 7786202-0.3
Get_BBox500000 493151 493549-0.1
Get_CBox500000 365376 3565502.4
New_Face & load glyph(s)500000 566729 5421194.3
TOTAL5772330 | 5774035 1972554919702136 0.1

Results for Roboto_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load120000 5873285730382.4
Load_Advances (Normal)120000 494478515420-4.2
Load_Advances (Fast)120000 31133278-5.3
Load_Advances (Unscaled)120000 27933210-14.9
Render120000 439328452423-3.0
Get_Glyph120000 168057168854-0.5
Get_Char_Index94000 24122629-9.0
Iterate CMap1000 18211862-2.3
New_Face1000 4123141691-1.1
Embolden120000 221434221765-0.1
Stroke63000 | 62520 162983316253800.3
Get_BBox120000 106660107050-0.4
Get_CBox120000 89839824108.3
New_Face & load glyph(s)120000 12774010432318.3
TOTAL2717520 39160673903333-0.3

Results for Arial_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 8636608120126.0
Load_Advances (Normal)95000 703405741526-5.4
Load_Advances (Fast)95000 24492520-2.9
Load_Advances (Unscaled)95000 22922388-4.2
Render95000 3493303468350.7
Get_Glyph95000 1511981450184.1
Get_Char_Index94000 24212424-0.1
Iterate CMap1000 188418133.8
New_Face1000 51398490084.6
Embolden95000 2007961972971.7
Stroke56620 | 57665 14809131485980-0.3
Get_BBox95000 95990940582.0
Get_CBox95000 6981069850-0.1
New_Face & load glyph(s)95000 1107181051755.0
TOTAL2206285 40862644055904-0.7

Results for TimesNewRoman_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 9520519171503.7
Load_Advances (Normal)95000 827119873521-5.6
Load_Advances (Fast)95000 245424251.2
Load_Advances (Unscaled)95000 228122740.3
Render95000 374973380720-1.5
Get_Glyph95000 145054146668-1.1
Get_Char_Index94000 24222440-0.7
Iterate CMap1000 18511881-1.6
New_Face1000 5096151140-0.4
Embolden95000 264007272286-3.1
Stroke43985 | 44175 15380281543158-0.3
Get_BBox95000 110972111756-0.7
Get_CBox95000 72576706142.7
New_Face & load glyph(s)95000 1319441303981.2
TOTAL2180160 447669345064310.7

Results for Tahoma_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 6022435447909.5
Load_Advances (Normal)95000 493275498016-1.0
Load_Advances (Fast)95000 24192456-1.5
Load_Advances (Unscaled)95000 23452460-4.9
Render95000 320830361343-12.6
Get_Glyph95000 1440441433260.5
Get_Char_Index94000 24132414-0.0
Iterate CMap1000 195518356.1
New_Face1000 45833457640.2
Embolden95000 206447206567-0.1
Stroke63270 | 64505 15577711567078-0.6
Get_BBox95000 8987090831-1.1
Get_CBox95000 6522267532-3.5
New_Face & load glyph(s)95000 99549101768-2.2
TOTAL2219775 363421636361800.1

Results for Verdana_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 5692295377625.5
Load_Advances (Normal)95000 483152487458-0.9
Load_Advances (Fast)95000 24363029-24.3
Load_Advances (Unscaled)95000 22712330-2.6
Render95000 344666354495-2.9
Get_Glyph95000 1475951437882.6
Get_Char_Index94000 248524431.7
Iterate CMap1000 182817544.0
New_Face1000 4430047725-7.7
Embolden95000 2018771984451.7
Stroke65455 | 65170 15581041564606-0.4
Get_BBox95000 8965989854-0.2
Get_CBox95000 67929661442.6
New_Face & load glyph(s)95000 96778100455-3.8
TOTAL2222625 36123093600288-0.3

Freetype Benchmark
reply via email to
[Prev in Thread] Current Thread [Next in Thread]