freetype-devel
[Top][All Lists]
Advanced
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: -warmup

From: Ahmet Göksu
Subject: Re: -warmup
Date: Fri, 18 Aug 2023 17:49:11 +0300
Hİ,
I have edited the code aligning with the Hin-Tak’s suggestion. Here is the two results pages, also pushed on gitlab.

Best,
Goksu
goksu.in
On 18 Aug 2023 14:02 +0300, Werner LEMBERG <wl@gnu.org>, wrote:
What happens if you use, say, `-c 100000', just running the
`Get_Char_Index` test? Are the percental timing differences then
still that large?
Actually Get_Char_Index, on the three pages I have sent in the
prev. mail, is higher than 6% only 4 times out of 15 total. (which is
seem on other tests as well).

Well, the thing is that IMHO the difference should be *much* smaller –
your HTML pages show the execution of identical code on an identical
machine, right?

about outliers, i splitted every tests into chuncks that is sized
100. Made IQR calculations and calculated average time on valid
chunks. you can find the result in the attachment also pushed to
gitlab.

Thanks. Hin-Tak gave additional suggestions how to possibly improve
the removal of outliers.

also, since statistics and benchmarking are a sciences their self, i
am a bit struggling while approaching the problem as well as feels
like out of the gsoc project scope.

Indeed, the focus lately shifted from a representational aspect to a
more thorough approach how to handle benchmarks in general. You are
done with the first part, more or less, and it looks fine. The
latter, however, is definitely part of the GSoC project, too, and I'm
surprised that you think this might not be so: What are benchmark
timings good for if the returned values are completely meaningless?

In most cases, a small performance optimization in FreeType might
yield, say, an improvement of 1%. Right now, such a change would not
be detectable at all if using the framework you are working on – it
would be completely hidden by noise.

To summarize: Benchmark comparisons only work if there is a sound
mathematical foundation to reduce the noise. I don't ask you to
reinvent the wheel, but please do some more internet research and
check existing code how to tackle such problems. I'm 100% sure that
such code already exists (for example, the Google benchmark stuff
mentioned in a previous e-mail, scientific papers on arXiv, etc.,
etc.) and can be easily used, adapted, and simplified for our
purposes.


Werner

Freetype Benchmark Results

Warning: Baseline and Benchmark have the same commit ID!

Info

InfoBaselineBenchmark
Parameters-c 550 -w 50-c 550 -w 50
Commit ID3553148135531481
Commit Date2023-08-18 02:04:38 +03002023-08-18 02:04:38 +0300
BranchGSoC-2023-AhmetGSoC-2023-Ahmet

* Average time for all iterations. Smaller values are better.

** N count in (x | y) format is for showing baseline and benchmark N counts seperately when they differs.

Total Results

TestNBaseline (µs) Benchmark (µs)Difference (%)
Load250000 178180 190899-7.1
Load_Advances (Normal)250000 161654 1594301.4
Load_Advances (Fast)250000 1047 1130-7.9
Load_Advances (Unscaled)250000 996 1006-1.0
Render250000 278410 2766440.6
Get_Glyph250000 213564 2093142.0
Get_Char_Index235000 1012 9921.9
Iterate CMap2500 873 908-4.0
New_Face2500 12862 13004-1.1
Embolden250000 229042 2261561.3
Stroke250000 956152 9556160.1
Get_BBox250000 180922 1771082.1
Get_CBox250000 211855 2114930.2
New_Face & load glyph(s)250000 29090 287781.1
TOTAL2990000 24556582452479 0.1

Results for Roboto_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load60000 2996431568-5.3
Load_Advances (Normal)60000 29942273428.7
Load_Advances (Fast)60000 2332320.6
Load_Advances (Unscaled)60000 2202200.0
Render60000 5317953842-1.2
Get_Glyph60000 3882039532-1.8
Get_Char_Index47000 197198-0.3
Iterate CMap500 185194-4.6
New_Face500 230822671.8
Embolden60000 4110941912-2.0
Stroke60000 213674213932-0.1
Get_BBox60000 17216160356.9
Get_CBox60000 3897640102-2.9
New_Face & load glyph(s)60000 556455540.2
TOTAL1416000 4715874729300.3

Results for Arial_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 3783443165-14.1
Load_Advances (Normal)47500 37650352156.5
Load_Advances (Fast)47500 202284-40.6
Load_Advances (Unscaled)47500 192198-3.1
Render47500 6076163992-5.3
Get_Glyph47500 47014468730.3
Get_Char_Index47000 1911881.6
Iterate CMap500 1641620.6
New_Face500 271226870.9
Embolden47500 49745493860.7
Stroke47500 182028185064-1.7
Get_BBox47500 4436044575-0.5
Get_CBox47500 4663646844-0.4
New_Face & load glyph(s)47500 580956961.9
TOTAL1141000 5152985243291.8

Results for TimesNewRoman_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 5399859007-9.3
Load_Advances (Normal)47500 4127043176-4.6
Load_Advances (Fast)47500 2052050.0
Load_Advances (Unscaled)47500 194195-0.5
Render47500 70520690722.1
Get_Glyph47500 54836538261.8
Get_Char_Index47000 1981951.5
Iterate CMap500 1801668.1
New_Face500 28382849-0.4
Embolden47500 59834593460.8
Stroke47500 2393062392880.0
Get_BBox47500 5187952767-1.7
Get_CBox47500 5357455530-3.7
New_Face & load glyph(s)47500 70127013-0.0
TOTAL1141000 6358446426341.1

Results for Tahoma_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 2729033068-21.2
Load_Advances (Normal)47500 2619028596-9.2
Load_Advances (Fast)47500 202205-1.5
Load_Advances (Unscaled)47500 193197-2.1
Render47500 4617446322-0.3
Get_Glyph47500 36092359830.3
Get_Char_Index47000 2302156.3
Iterate CMap500 178203-13.7
New_Face500 25522762-8.2
Embolden47500 3899039810-2.1
Stroke47500 161374162131-0.5
Get_BBox47500 3328533466-0.5
Get_CBox47500 3579236093-0.8
New_Face & load glyph(s)47500 534853160.6
TOTAL1141000 4138904243642.5

Results for Verdana_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 290932409217.2
Load_Advances (Normal)47500 26602251015.6
Load_Advances (Fast)47500 2052040.5
Load_Advances (Unscaled)47500 1971960.5
Render47500 47776434169.1
Get_Glyph47500 368023310110.1
Get_Char_Index47000 196197-0.5
Iterate CMap500 166183-10.2
New_Face500 245224400.5
Embolden47500 39363357029.3
Stroke47500 1597701552002.9
Get_BBox47500 341843026611.5
Get_CBox47500 368783292410.7
New_Face & load glyph(s)47500 535651992.9
TOTAL1141000 419040388223-7.4

Freetype Benchmark

Freetype Benchmark Results

Warning: Baseline and Benchmark have the same commit ID!

Info

InfoBaselineBenchmark
Parameters-c 550 -w 50-c 550 -w 50
Commit IDf3dfede6f3dfede6
Commit Date2023-08-18 17:42:53 +03002023-08-18 17:42:53 +0300
BranchGSoC-2023-AhmetGSoC-2023-Ahmet

* Average time for all iterations. Smaller values are better.

** N count in (x | y) format is for showing baseline and benchmark N counts seperately when they differs.

Total Results

TestNBaseline (µs) Benchmark (µs)Difference (%)
Load250000 195162 1924241.4
Load_Advances (Normal)250000 176304 1653636.2
Load_Advances (Fast)250000 1108 1206-8.8
Load_Advances (Unscaled)250000 1087 10027.8
Render250000 287896 2817382.1
Get_Glyph250000 229158 2094608.6
Get_Char_Index235000 992 9741.8
Iterate CMap2500 869 8620.7
New_Face2500 14858 1294412.9
Embolden250000 251134 22580310.1
Stroke250000 1029943 9529077.5
Get_BBox250000 194832 1756329.9
Get_CBox250000 240592 20872913.2
New_Face & load glyph(s)250000 30726 289565.8
TOTAL2990000 26546622458002 7.4

Results for Roboto_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load60000 32608299868.0
Load_Advances (Normal)60000 2551427810-9.0
Load_Advances (Fast)60000 234236-0.4
Load_Advances (Unscaled)60000 222222-0.2
Render60000 592805337410.0
Get_Glyph60000 465643866017.0
Get_Char_Index47000 189192-1.9
Iterate CMap500 162199-22.5
New_Face500 23652473-4.6
Embolden60000 42693407974.4
Stroke60000 2319272133168.0
Get_BBox60000 172141545810.2
Get_CBox60000 590793880034.3
New_Face & load glyph(s)60000 588156733.5
TOTAL1416000 523930467197-10.8

Results for Arial_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 4671850918-9.0
Load_Advances (Normal)47500 489634110216.1
Load_Advances (Fast)47500 2202037.9
Load_Advances (Unscaled)47500 2121948.5
Render47500 68496633497.5
Get_Glyph47500 566824748016.2
Get_Char_Index47000 2051916.8
Iterate CMap500 1761665.7
New_Face500 4460264840.6
Embolden47500 653324919224.7
Stroke47500 20966818130813.5
Get_BBox47500 45346437723.5
Get_CBox47500 48516460815.0
New_Face & load glyph(s)47500 625457128.7
TOTAL1141000 601248532316-11.5

Results for TimesNewRoman_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 4952253178-7.4
Load_Advances (Normal)47500 42683417592.2
Load_Advances (Fast)47500 2042030.5
Load_Advances (Unscaled)47500 1951940.5
Render47500 6927471641-3.4
Get_Glyph47500 55683541392.8
Get_Char_Index47000 195196-0.8
Iterate CMap500 1661660.6
New_Face500 281627721.6
Embolden47500 61636608351.3
Stroke47500 2434762407781.1
Get_BBox47500 54669525333.9
Get_CBox47500 56070548822.1
New_Face & load glyph(s)47500 69316942-0.2
TOTAL1141000 643522640218-0.5

Results for Tahoma_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 381732983021.9
Load_Advances (Normal)47500 343922779619.2
Load_Advances (Fast)47500 2252250.0
Load_Advances (Unscaled)47500 2141987.5
Render47500 46312458850.9
Get_Glyph47500 36125354521.9
Get_Char_Index47000 2051983.4
Iterate CMap500 19616715.0
New_Face500 277525617.7
Embolden47500 446503854213.7
Stroke47500 18529616230812.4
Get_BBox47500 35112327526.7
Get_CBox47500 424823513017.3
New_Face & load glyph(s)47500 6473538216.8
TOTAL1141000 472629416426-11.9

Results for Verdana_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 2814128512-1.3
Load_Advances (Normal)47500 2475226896-8.7
Load_Advances (Fast)47500 224339-51.0
Load_Advances (Unscaled)47500 24419420.5
Render47500 4453447489-6.6
Get_Glyph47500 34104337301.1
Get_Char_Index47000 1981961.0
Iterate CMap500 1681661.5
New_Face500 24422490-2.0
Embolden47500 36822364361.0
Stroke47500 1595771551982.7
Get_BBox47500 424933111626.8
Get_CBox47500 34445338361.8
New_Face & load glyph(s)47500 51865246-1.2
TOTAL1141000 413332401844-2.8

Freetype Benchmark
reply via email to
[Prev in Thread] Current Thread [Next in Thread]