README.md 6.57 KB
Newer Older
1
# qmlbench
Gunnar Sletta's avatar
Gunnar Sletta committed
2

3
4
5
6
qmlbench is a tool for benchmarking Qt, QML and QtQuick as a whole stack rather
than in isolation. The benchmarks it provides exercise a very large part of
Quick, QML, Gui, Core, and as a result can be considered a decent metric for
overall Qt performance.
Gunnar Sletta's avatar
Gunnar Sletta committed
7

8
9
10
11
12
13
As with any benchmarking, if you want reliable results, the first thing you must
attend to is ensuring that the system under test is doing the absolute minimum
possible. "Typical" operating systems (OS X, Ubuntu, ...) all have a lot of
"background noise" in the form of file indexing, automatic updates. All of this
should be disabled if possible. You may want to consider disabling networking
for the duration of the benchmarking run to help keep things "quiet".
Gunnar Sletta's avatar
Gunnar Sletta committed
14

15
16
17
18
19
In terms of running applications, you should have the bare minimum if at all
possible. No email clients, no IRC, no media players. Only your terminal, and
qmlbench should be running. If possible (e.g. on Linux), you may want to even
look into setting up a custom desktop environment session that doesn't launch
anything else (so no gnome-session etc).
Gunnar Sletta's avatar
Gunnar Sletta committed
20

21
22
Remember: Any background process that might pop up will cause instability in
the results!
Gunnar Sletta's avatar
Gunnar Sletta committed
23

24
25
# shells

26
qmlbench provides a number of "shells", which are the containers for the
27
28
29
benchmarking logic. Different shells work in different ways, and might measure
different things.

Robin Burchell's avatar
Robin Burchell committed
30
31
32
33
34
35
36
37
38
39
## frame-count shell

The frame-count shell constantly performs a set number of operations each frame,
for a given amount of (real world) time, and counts the number of frames the
window is able to generate. If the GL drivers & OS support it, vsync will be
disabled.

The frame-count shell is used for automated benchmark runs, as it is less
susceptible to variance than the sustained-fps shell.

40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
## sustained-fps shell

The sustained-fps shell provides a measure of how many times a certain operation
can be performed per frame while sustaining a velvet smooth framerate matching the
host's screen refresh rate.

Summarized, it can be looked at as a test of:
*How much stuff can I do before things start to fall apart.*.

Note that the sustained-fps shell _requires_ a stable frame rate (see the
prerequisites section), which is often hard for systems to provide, so this
shell may not be a good choice for you, and may not be appropriate for automated
regression testing.

### prerequisites

As already mentioned, the goal of the sustained-fps shell is to find how much is
possible while hitting a perfect framerate. So the first goal is to verify
that your system is capable of hitting a perfect framerate at all. This is
called the swap test. Test it by running the following, and observing the screen:
Gunnar Sletta's avatar
Gunnar Sletta committed
60

61
> ./tools/decidefps/decidefps
Gunnar Sletta's avatar
Gunnar Sletta committed
62

63
64
65
If you see a pulsating purple rectangle you are in good shape. If you see
flashes of red and/or blue or if the purple rectangle has horizontal regions
of red and blue, your system is not able to hit a perfect framerate.
Gunnar Sletta's avatar
Gunnar Sletta committed
66

67
68
Fix that first, or you will not get stable/sensible results from the
sustained-fps shell.
Gunnar Sletta's avatar
Gunnar Sletta committed
69

70
71
72
73
74
75
Lastly, check if your system is running with the 'basic' render loop
(QSG_INFO=1 in the environment will tell you). If you are, you will most likely
not be able to get velvet FPS because the animations are timer driven and will
skip once or twice per frame by default. If threaded is not an option for
whatever reason, try with QSG_RENDER_LOOP=windows (which uses vsync driven
animations).
Gunnar Sletta's avatar
Gunnar Sletta committed
76

77
78
79
80
81
82
83
84
## static-count shell

The static-count shell constantly performs a set number of operations (the
staticCount property in benchmarks) each frame. It is useful for profiling.

You can override the staticCount value by passing a `--count N` command line
argument, where N is the staticCount you want to use instead.

Robin Burchell's avatar
Robin Burchell committed
85
86
The static-count shell is most useful for profiling, as it will repeat the same
operation continuously.
Gunnar Sletta's avatar
Gunnar Sletta committed
87

Robin Burchell's avatar
Robin Burchell committed
88
## the qmlbench tool
Gunnar Sletta's avatar
Gunnar Sletta committed
89
90
91

It comes with a number of default settings which aim to help give stable numbers:

Robin Burchell's avatar
Robin Burchell committed
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
* ``--fps-override``: This one may be very important when using the
  sustained-fps shell. When you ran decidefps,
  it told you a fairly accurate FPS. If QScreen reports something different
  through refreshRate() the calibration won't work and the resulting numbers
  won't mean anything. On OSX, we should be mostly good. On Linux, it is not
  uncommon that the QScreen value is wrong so this option is needed as an
  override to make the tests stable. The tool has not been tested on Windows at
  the time when this is written.
* ``--delay [ms]``: defaults to 2 seconds. An idle time after showing the window
  before starting the benchmark. Especially useful on OSX where the system
  specific to-fullscreen animation takes a while. And that to-fullscreen will
  severly damage the benchmarks.
* ``--help``: Tells you the other options.

## the actual benchmarks

Benchmarks are divided into two types, automated, and manual. The automated
tests are useful for regression testing, and the manual tests are useful for
determining the capabilities of a new piece of hardware.

To run all the automated tests, simply run this, and go have lunch (it is not a
fast process):

> ./src/qmlbench benchmarks/auto/

You can also run a subset of them by providing that directory instead:

> ./src/qmlbench benchmarks/auto/creation/quick.item/

Or even individual tests, by passing the filenames directly.

Mitch Curtis's avatar
Mitch Curtis committed
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
## the results

Here's an example of running `src/qmlbench --shell frame-count /Users/mitch/dev/qmlbench/benchmarks/auto/creation/quick.controls2/delegates_tumbler.qml` on macOS:

    ID:
    OS:          macOS High Sierra (10.13)
    QPA:         cocoa
    GL_VENDOR:   ATI Technologies Inc.
    GL_RENDERER: AMD Radeon Pro 460 OpenGL Engine
    GL_VERSION:  2.1 ATI-1.60.24
    running: /Users/mitch/dev/qmlbench/benchmarks/auto/creation/quick.controls2/delegates_tumbler.qml
        153 frames
        152 frames
        154 frames
        154 frames
        153 frames
        Average: 153.2  frames; MedianAll=153; StdDev=0.83666, CoV=0.00546123
    All done...

The test was repeated 5 times, and for each run, the frames are printed. For the frame-count shell, the higher the number of frames is, the better.

- "Average: 153.2 frames" - this is the average of each run.
- "MedianAll=153" - the [median](https://en.wikipedia.org/wiki/Median) of all of the runs:
   (152, 153, 153, 154, 154)
               ^
- "StdDev=0.83666" - the [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation). The lower the better.
- "CoV=0.00546123" - the [coefficient of variation](https://en.wikipedia.org/wiki/Coefficient_of_variation). The lower the better; a general guide is to aim for less than 5% CoV.
150
151
152
153

## Creating benchmarks

See [Benchmarks/benchmarks.md](./benchmarks/benchmarks.md)