Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
1 result
Name Last commit Last update
JOM
flex_bison
thrift
web_index
README.md
REST_API.md
config.json
jobNotificationHandler.js
package-lock.json
package.json
regressionFinder.py
server.js
testHandler.js
toolbox.js
README.md

QMLBench Regression Finder

QMLBench Regression Finder is a tool to assist in benchmarking changes and identifying bad commits in exiting branches.

What can I do with it?

This framework provides a web interface for three types of benchmark runs:

  1. Automatic bisect
  2. Single commit
  3. Two commit comparison

How does it work?

The QMLBench Regression Finder framework consists of a two parts:

  1. A nodejs server that operates a lightweight web interface, a job scheduler, and email notification subsystem
  2. A python script that builds, tests, and collects results from QMLBench

The framework is platform independent and can be deployed to as many clients as desired. A standalone mode is also available for local benchmarking.

Installation

Pre-requisites

  1. All the usual stuff to build Qt from source
  2. NodeJS & npm - Built on NodeJS 10.15.3, but any recent version will likely work
  3. Python 3
    • Python3 submodules (installable with pip / pip3):
      • argparse
      • requests
      • thriftpy
      • packaging
  4. (Windows) MSVC Build Tools 2019 - Other versions may work, but VS2019 is recommended for building Qt. Build tools are already included in full installations of Microsoft Visual Studio 2019.

How to install

  1. Download or clone this repository
  2. In the repository directory, run the following:
    1. npm install

Configure your installation

  • Modify config.json as necessary, or set environment variables to override the config file.
    1. BASE_URL - This url refers to the URL or IP where the scheduler can be reached. Default: localhost
    2. HTTP_PORT - The port on which to run the server. All instances of this system on a network must be configured to use the same port. Default: 8080
    3. SMTP_SERVER - [Optional] A SMTP server that can send emails. Authentication not supported - The mail server must accept anonymous connections. Default: smtp.intra.qt.io
    4. SMTP_PORT [Optional] - Specify the port to connect to the mail server. Default: 25
    5. VS_BUILD_TOOLS [Windows build hosts only] - The full path to VsDevCmd.bat provided by Microsoft Visual Studio Default: "C:/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/Common7/Tools/VsDevCmd.bat"
    6. agents [Optional: see "How to Run"] - A list of IP strings which have the agent deployed as a testrunner. It is unnecessary to add localhost to this list.

Running

  1. In the repository directory, run one of the following commands:
    1. npm start standalone
      • Start the scheduler in standalone mode. This mode allows for jobs to be run on the local machine, and does not require additional servers to act as hosts.
    2. npm start 10.9.70.10 10.9.70.11 10.9.70.12
      • Start the server in scheduler mode with local jobs enabled. This will allow scheduling jobs on both the local machine and remote hosts. A space separated list of agents can be accepted in addition to agents specified in the config.json file.
    3. npm start nolocal 10.9.70.10 10.9.70.11 10.9.70.12
      • (Recommended configuration) Start the server in scheduler only mode. This will prevent the local host from being available as a benchmarking host. A space separated list of agents can be accepted in addition to agents specified in the config.json file.
    4. npm start agent
      • Start the server in remote agent mode. This disables the scheduler interface and configures the server to listen for connections from a scheduling server. This is the mode that must be used on remote agents.

Using the interface

General guidelines

  1. If using the bisect or two commit comparison, the commits must be on the same major branch.
  2. If bisecting, you need to have a rough idea of the expected performance regression. Testing regressions smaller than ~5% is not recommended due to natural test noise.
  3. It is strongly recommended that a QMLBench Regression Finder running as a agent is hosted on a machine that is performing no other tasks and has been configured for performance stability.
  4. You need to know a specific QMLBench test that is affected by a given commit's changes.
    • You may also write and upload custom QMLBench benchmarks.
    • See QMLBench for an overview of the QMLBench test framework.
    • The benchmark creation/quick.item/delegates_item.qml may provide a good starting point for writing your own QMLBench benchmark.
  5. [Important] This tool works by cherry-picking a given change to test onto the closest-related qt5.git sha. If your change to test requires additional patches to build, enter them when scheduling the test. Note that this also applies to already merged changes. If the tool is having trouble building a change, check if there are any patches that might apply.

Automatic Bisect

This mode takes two commits from the same major branch (i.e. 5.15) and emulates a git-bisect like behavior, testing various commits in-between the known good and known bad commits to find a performance regression.

Notes:

  1. The "known good" commit must be an ancestor (is older and committed to the same branch) of the "known bad" commit.
  2. The commits must have been merged into Qt successfully.

Single commit

This mode tests a single commit and provides a single benchmark result. In this mode, WIP (work-in-progress) commits and commits not yet merged can be tested.

Two Commit Comparison

This mode tests two commits and provides results for each, as well as a percentage difference between the two. In this mode, WIP (work-in-progress) commits and commits not yet merged can be tested.

Notes:

  1. Though the commits can be WIP or unmerged, they must still be in the same module and of the same branch. Use multiple runs of the single-commit test mode to test commits in different branches or modules.

Troubleshooting

Common problems

  1. Build failures
    1. Bad checkout of some qt repo. If a test fails to build when you know it should, try simply deleting the QtBuild directory on the agent and re-stage the rest run.
    2. In some rare cases, a commit may not be able to be built with this tool if it had unmet dependencies when merged in COIN that weren't caught.
      • If necessary, you can add additional patchrefs or shas to cherry-pick when testing a change.
  2. Failure to produce a test result
    1. There is no validation of custom QML benchmark file. If everything looks good in the build but QMLBench fails to produce a result or crashes, make sure your custom QMLBench benchmark .qml file is formatted correctly.
    2. OpenGL is currently required on all platforms. If no OpenGL is installed, or if Angle is selected on windows but unsupported on the host, QMLBench will likely crash.
  3. Test result instability
    1. If you're using icecc in your linux environments and encounter test performance instability, ensure icecc's config is set to disable remote jobs.
    2. Be sure the system was not attempting to perform automatic updates.
    3. Increase the count of test repeats. Default is 10. Each repeat runs for approximately 20 seconds.
      • Check regressionFinder.py::runBenchmark() to adjust the number of test repeats.