Commit 4290424f authored by Daniel Smith's avatar Daniel Smith

Added environment variables for SMTP Server config. Updated Readme. Switched to LF line endings

parent 6829ebb0
# QMLBench Regression Finder
QMLBench Regression Finder is a tool to assist in benchmarking changes and identifying bad commits in exiting branches.
### What can I do with it?
This framework provides a web interface for three types of benchmark runs:
1. [Automatic bisect](./README.md#Automatic_Bisect)
2. [Single commit](./README.md#Single-Commit)
3. [Two commit comparison](./README.md#Two-Commit-Comparison)
### How does it work?
The QMLBench Regression Finder framework consists of a two parts:
1. A nodejs server that operates a lightweight web interface, a job scheduler, and email notification subsystyem
2. A python script that builds, tests, and collects results from QMLBench
The framework is platform independent and can be deployed to as many clients as desired. A standalone mode is also available for local benchmarking.
### Installation pre-requisites
1. [All the usual stuff to build Qt from source](https://wiki.qt.io/Building_Qt_5_from_Git)
2. [NodeJS](https://nodejs.org/en/) - Built on 10.15.3, but any recent version will likely work
3. [Python 3](https://www.python.org/downloads/)
- Python3 submodules (installable with pip / pip3):
- argparse
- requests
- thriftpy
- packaging
4. (Windows) [MSVC Build Tools 2017](https://aka.ms/AA363al) - Other versions may work, but VS2017 is recommended for building Qt. Build tools are already included in full installations of Microsoft Visual Studio 2017.
### How to install
1. Download or clone this repository
2. In the repository directory, run the following:
1. npm install
- __Note__: Python2 may be required for the "diskspace" module. If you encouter errors regarding this module during the "npm install" step, ensure that your Python2 installation directory is added to the __beginning__ of your "PATH" environment variable for this step.
3. Celebrate! Installation is complete
### How to run
1. In the repository directory, run one of the following commands:
1. ```npm start standalone```
- Start the scheduler in standalone mode. This mode allows for jobs to be run on the local machine, and does not require additional servers to act as hosts.
2. ```npm start 10.9.70.10 10.9.70.11 10.9.70.12```
- Start the server in scheduler mode with local jobs enabled. This will allow scheduling jobs on both the local machine and remote hosts. A space separated list of remote hosts is required.
3. ```npm start nolocal 10.9.70.10 10.9.70.11 10.9.70.12```
- *__(Recommended configuration)__* Start the server in scheduler only mode. This will prevent the local host from being available as a benchmarking host. A space separated list of remote hosts is required.
4. ```npm start```
- Start the server in remote host mode. This disables the scheduler interface and configures the server to listen for connections from a scheduling server. This is the mode that must be used on remote hosts.
### Optional parameters
Optional parameters are set via environment variables. The following options are available.
1. ```HTTP_PORT``` - The port on which to run the server. All instances of this system on a network must be configured to use the same port. ```Default: 8080```
2. ```BUILD_CORES``` - The number of CPU threads to use when building Qt ```Default: 8```
3. ```VS_DEV_ENV``` - The full path to VsDevCmd.bat provided by Microsoft Visual Studio ```Default: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/Common7/Tools/VsDevCmd.bat```
## Using the interface
### General guidelines
1. If using the bisect or two commit comparison, the commits must be on the same major branch.
2. You need to know which module the commit(s) were made in (i.e. qtdeclarative or qtbase for example)
3. You need to have a rough idea of the expected performance regression. Testing regressions smaller than ~5% is not recommended due to natural test noise.
4. You need to know a specific QMLBench test that is affected by a given commit's changes.
### Automatic Bisect
This mode takes two commits from the same major branch (i.e. 5.12) and emulates a git-bisect like behavior, testing various commits in-between the known good and known bad commits to find a performance regression.
Notes:
1. The "known good" commit must be an ancestor (is older and commited to the same branch) of the "known bad" commit.
2. The commits must have been merged into Qt successfully.
### Single commit
This mode tests a single commit and provides a single benchmark result. In this mode, WIP (work-in-progress) commits and commits not yet merged can be tested.
### Two Commit Comparison
This mode tests two commits and provides results for each, as well as a percentage difference between the two. In this mode, WIP (work-in-progress) commits and commits not yet merged can be tested.
Notes:
1. Though the commits can be WIP or unmerged, they must still be in the same module and of the same branch. Use multiple runs of the single-commit test mode to test commits in different branches or modules.
\ No newline at end of file
# QMLBench Regression Finder
QMLBench Regression Finder is a tool to assist in benchmarking changes and identifying bad commits in exiting branches.
### What can I do with it?
This framework provides a web interface for three types of benchmark runs:
1. [Automatic bisect](./README.md#Automatic_Bisect)
2. [Single commit](./README.md#Single-Commit)
3. [Two commit comparison](./README.md#Two-Commit-Comparison)
### How does it work?
The QMLBench Regression Finder framework consists of a two parts:
1. A nodejs server that operates a lightweight web interface, a job scheduler, and email notification subsystyem
2. A python script that builds, tests, and collects results from QMLBench
The framework is platform independent and can be deployed to as many clients as desired. A standalone mode is also available for local benchmarking.
### Installation pre-requisites
1. [All the usual stuff to build Qt from source](https://wiki.qt.io/Building_Qt_5_from_Git)
2. [NodeJS](https://nodejs.org/en/) - Built on 10.15.3, but any recent version will likely work
3. [Python 3](https://www.python.org/downloads/)
- Python3 submodules (installable with pip / pip3):
- argparse
- requests
- thriftpy
- packaging
4. (Windows) [MSVC Build Tools 2017](https://aka.ms/AA363al) - Other versions may work, but VS2017 is recommended for building Qt. Build tools are already included in full installations of Microsoft Visual Studio 2017.
### How to install
1. Download or clone this repository
2. In the repository directory, run the following:
1. npm install
- __Note__: Python2 may be required for the "diskspace" module. If you encouter errors regarding this module during the "npm install" step, ensure that your Python2 installation directory is added to the __beginning__ of your "PATH" environment variable for this step.
3. Celebrate! Installation is complete
### How to run
1. In the repository directory, run one of the following commands:
1. ```npm start standalone```
- Start the scheduler in standalone mode. This mode allows for jobs to be run on the local machine, and does not require additional servers to act as hosts.
2. ```npm start 10.9.70.10 10.9.70.11 10.9.70.12```
- Start the server in scheduler mode with local jobs enabled. This will allow scheduling jobs on both the local machine and remote hosts. A space separated list of remote hosts is required.
3. ```npm start nolocal 10.9.70.10 10.9.70.11 10.9.70.12```
- *__(Recommended configuration)__* Start the server in scheduler only mode. This will prevent the local host from being available as a benchmarking host. A space separated list of remote hosts is required.
4. ```npm start```
- Start the server in remote host mode. This disables the scheduler interface and configures the server to listen for connections from a scheduling server. This is the mode that must be used on remote hosts.
### Optional parameters
Optional parameters are set via environment variables. The following options are available.
1. ```HTTP_PORT``` - The port on which to run the server. All instances of this system on a network must be configured to use the same port. ```Default: 8080```
2. ```BUILD_CORES``` - The number of CPU threads to use when building Qt ```Default: [All available cores]```
3. ```VS_DEV_ENV``` - The full path to VsDevCmd.bat provided by Microsoft Visual Studio ```Default: "C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/Common7/Tools/VsDevCmd.bat"```
4. ```SMTP_SERVER``` - A SMTP server that can send emails. Authentication not supported - The mail server must accept anonymous connections. ```Default: smtp.intra.qt.io```
5. ```SMTP_PORT``` - Specify the port to connect to the mail server. ```Default: 25```
## Using the interface
### General guidelines
1. If using the bisect or two commit comparison, the commits must be on the same major branch.
2. You need to know which module the commit(s) were made in (i.e. qtdeclarative or qtbase for example)
3. You need to have a rough idea of the expected performance regression. Testing regressions smaller than ~5% is not recommended due to natural test noise.
4. It is **strongly recommended** that a QMLBench Regression Finder running as a slave is hosted on a machine that is performing no other tasks and has been configured for performance stability.
4. You need to know a specific QMLBench test that is affected by a given commit's changes.
- You may also write and upload custom QMLBench benchmarks.
- See [QMLBench](https://github.com/qt-labs/qmlbench) for an overview of the QMLBench test framework.
- The benchmark [creation/quick.item/delegates_item.qml](https://github.com/qt-labs/qmlbench/blob/master/benchmarks/auto/creation/quick.item/delegates_item.qml) may provide a good starting point for writing your own QMLBench benchmark.
### Automatic Bisect
This mode takes two commits from the same major branch (i.e. 5.12) and emulates a git-bisect like behavior, testing various commits in-between the known good and known bad commits to find a performance regression.
Notes:
1. The "known good" commit must be an ancestor (is older and commited to the same branch) of the "known bad" commit.
2. The commits must have been merged into Qt successfully.
### Single commit
This mode tests a single commit and provides a single benchmark result. In this mode, WIP (work-in-progress) commits and commits not yet merged can be tested.
### Two Commit Comparison
This mode tests two commits and provides results for each, as well as a percentage difference between the two. In this mode, WIP (work-in-progress) commits and commits not yet merged can be tested.
Notes:
1. Though the commits can be WIP or unmerged, they must still be in the same module and of the same branch. Use multiple runs of the single-commit test mode to test commits in different branches or modules.
## Troubleshooting
### Common problems
1. **Build failures**
1. Bad checkout of some qt repo. If a test fails to build when you know it should, try simply deleting the QtBuild directory on the slave and re-stage the rest run.
2. **Failure to produce a test result**
1. There is no validation of custom QML benchmark file. If everything looks good in the build but QMLBench fails to produce a result or crashes, make sure your custom QMLBench benchmark .qml file is formatted correctly.
2. OpenGL is currently required on all platforms. If no OpenGL is installed, or if Angle is selected on windows but unsupported on the host, QMLBench will likely crash.
3. **Test result instability**
1. If you're using icecc in your linux environments and encounter test performance instability, ensure icecc's config is set to disable remote jobs.
2. Be sure the system was not attempting to perform automatic updates.
3. Increase the count of test repeats. Default is 10. Each repeat runs for approximately 20 seconds.
- Check regressionFinder.py::runBenchmark() to adjust the number of test repeats.
import os
import subprocess
import argparse
import platform
import stat
import shutil
import json
import requests
from thrift import storagestructs
import thriftpy.utils
import thriftpy
import re
from time import sleep
import atexit
import packaging
args = []
basedir = os.getcwd()
builddir = os.path.join(basedir, "QtBuild")
installdir = os.path.join(builddir, "Install")
isWindowsOS = (platform.system() == 'Windows')
compiler = basedir + "\\JOM\\jom.exe" if isWindowsOS else "make"
exeExt = '.exe' if isWindowsOS else ''
goodBaselineScore = 0
badBaselineScore = 0
regressionTargetScore = 0
observedRegressionPercent = 0
refQtbaseRev = ""
testType = ""
noRebuild = False
finalResults = []
# Bisect tracking items
revisionList = []
bisectResults = [[], ]
bisectPosition = 0
bisectGoodPos = 0
bisectBadPos = 0
bisectLastStatus = ""
bisectLastDirection = ""
def on_rm_error(func, path, exc_info):
# Handler for files that fail to be removed.
# path contains the path of the file that couldn't be removed
# let's just assume that it's read-only and unlink it.
try:
os.chmod(path, stat.S_IWRITE)
os.unlink(path)
except Exception as e:
print("There was an error removing a file from disk. Exception: {0}".format(e))
def parseArgs():
parser = argparse.ArgumentParser()
parser.add_argument("--setupEnv", dest="setupEnv", action="store_true", help="Run with --setupEnv to initialize the build environment by cloning needed repos or pulling updates as needed. Exits upon completion.")
parser.add_argument("--branch", dest="branch", type=str, help="Branch of Qt such as \'5.12\' or \'dev\'")
parser.add_argument("--moduleToTest", dest="module", type=str, help="Module where regression is suspected such as \'qtbase\'.")
parser.add_argument("--knownBadRev", dest="knownBadRev", type=str, help="Known bad revision in the module where a regression is expected.")
parser.add_argument("--knownGoodRev", dest="knownGoodRev", type=str, help="Known good revision in the module where a regression is expected")
parser.add_argument("--regressionTarget", dest="regressionTarget", type=int, help="Expected regression, in percent, to assist in finding the correct regressive commit. Be conservative if unsure.")
parser.add_argument("--fuzzyRange", dest="fuzzyRange", type=int, default=2, help="Provide a value to use as a fuzzy range to check the regression against. Using \'--regressionTarget 20 --fuzzyRange 5\' will prefer regressions between 15-25%% as the bad commit")
parser.add_argument("--buildCores", dest="buildCores", type=int, default=8, help="Number of build cores to use when building")
parser.add_argument("--wipeWorkspace", dest="wipeWorkspace", action="store_true", help="Clear the entire workspace and force re-cloning of all modules")
parser.add_argument("--VSDevEnv", dest="VSDevEnv", help="Full path to Visual studio VsDevCmd.bat file for windows build environments. Defaults to default installation for VS 2017 Build Tools", default="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/Common7/Tools/VsDevCmd.bat")
parser.add_argument("--benchmark", dest="benchmark", type=str, help="path to single benchmark to run. Only run one benchmark at a time!")
parser.add_argument("--testSingleCommit", dest="testSingleCommit", type=str, help="Set this parameter to a commit ID to benchmark. Setting this parameter overrides bisecting behavior.")
parser.add_argument("--testTwoCommit", dest="testTwoCommit", type=str, help="Set this parameter to a comma-separated list of two commit IDs to benchmark and compare. Setting this parameter overrides bisecting behavior.")
parser.add_argument("--FirstBuildOnHead", dest="firstBuildOnHead", action="store_true", help="Enable this parameter to build the first commit against branch HEAD instead of searching for a COIN integration.")
parser.add_argument("--SecondBuildOnHead", dest="secondBuildOnHead", action="store_true", help="Enable this parameter to build the second commit against branch HEAD instead of searching for a COIN integration.")
parser.add_argument("--OpenGLBackend", dest="openGLBackend", type=str, default="desktop", help="Render backend options. Valid options are \'dekstop\', \'angle\', \'software\'")
parser.add_argument("--jobName", dest="jobName", type=str, help="unique job name used for writing results file to logs directory. Typically a hash of the job to be run.")
parser.add_argument("--environment", dest="environment", type=str, help="Comma separated list of environment variables and values to use for the build and test environment.")
return parser.parse_args()
def initRepository(repo):
# Clone any needed repositories. If they already exist, sweep them and pull changes.
module = repo[repo.index('/') + 1:]
branch = args.branch if module != "qmlbench" else "master"
if not branch:
branch = "dev"
print(repo, branch)
def cloneRepo():
subprocess.run(["git", "clone", "-b", branch, f'https://code.qt.io/{repo}.git'], stderr=subprocess.PIPE, universal_newlines=True, cwd=builddir)
subprocess.run(["git", "submodule", "update", "--init"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
# clone modules if necessary, otherwise just pull latest.
if not os.path.exists(os.path.join(builddir, module)):
cloneRepo()
else:
subprocess.run(["git", "clean", "-dqfx"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
subprocess.run(["git", "reset", "--hard", f"origin/{args.branch}", "--quiet"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
subprocess.run(["git", "pull", "origin", args.branch, "--quiet"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
subprocess.run(["git", "checkout", branch], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
if repo == "qt/qt5":
# Run the reset and pull again. Sometimes the previous pull leads to merge conflicts.
# We don't care, just needed the first pull to get the lastest submodule list
# and then reset and pull again to get the latest changes.
subprocess.run(["git", "reset", "--hard", f"origin/{args.branch}", "--quiet"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
subprocess.run(["git", "pull", "origin", args.branch, "--quiet"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
subprocess.run(["git", "submodule", "update", "--init"], stderr=subprocess.PIPE, universal_newlines=True, cwd=builddir)
def validateTest():
# Verify the selected test exists
global args
if not os.path.exists(os.path.join(builddir, 'qmlbench', args.benchmark)):
args.benchmark = os.path.join("benchmarks", args.benchmark)
if not os.path.exists(os.path.join(builddir, 'qmlbench', args.benchmark)):
print(f"Specified benchmark at {os.path.join(builddir, 'qmlbench', args.benchmark)} does not exist. Please check the path and try again.")
return False
return True
def validateCommits():
# Verify that the provided commit(s) exist in the Qt repository.
def revparse(rev):
# Rev-parse our given SHA1 and see if Git returns anything. If it pushes anything to stdout, a match was found.
proc = subprocess.run(["git", "rev-parse", rev], cwd=os.path.join(builddir, args.module),
stderr=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True, shell=isWindowsOS)
if proc.stderr:
if lookupUnmergedCommit(rev, args.module):
return revparse(rev)
else:
print(f"Unable to verify commit {rev}. Please check to make sure you have the correct repository and there are no typos.")
return False
else:
print(proc.stdout.strip().split("\n"))
if len(proc.stdout.strip().split("\n")) > 1:
# Rev-parse will print one matching SHA per line.
print(f"Provided commit {rev} matches more than one commit. Try using the full SHA1.")
return False
else:
return proc.stdout.strip()
global args
global noRebuild
# Do some logic to determine which type of test we're running.
if args.knownGoodRev:
args.knownGoodRev = args.knownGoodRev.strip() # Cleanup inputs
args.knownBadRev = args.knownBadRev.strip()
# Try to see if we have at least one commit between the known good and bad commits. This will fail if the known bad is not an ancestor of the known good.
proc = subprocess.run(["git", "rev-list", f"{args.knownGoodRev}..{args.knownBadRev}", f"^{args.knownGoodRev}"], cwd=os.path.join(builddir, args.module),
stderr=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True, shell=isWindowsOS)
if proc.returncode != 0:
print(f"ERROR: Unable to verify selected commits. Please double check your input.\nGit error: {proc.stderr}")
return False
try:
revisionList = proc.stdout.split('\n')
except Exception as e:
print(f"ERROR: No commits between good and bad commits. Nothing to test. Hint: Known bad commit {args.knownBadRev} might be the bad commit.")
return False
if revisionList:
revisionList.pop() # rev-list will return the list with the known bad commit as the head element. Get rid of it permenantly.
if len(revisionList) < 1:
print(f"ERROR: No commits between good and bad commits. Nothing to test. Hint: Known bad commit {args.knownBadRev} might be the bad commit.")
return False
if args.module == "qtbase":
print("Module to test is Qtbase. Verifying that COIN data is available for other modules before proceeding...")
if not findRevToBuild(args.knownGoodRev, "qtdeclarative", args.branch) or not findRevToBuild(args.knownBadRev, "qtdeclarative", args.branch):
return False
else:
print("COIN data verified. Continuing.")
return True
elif args.testSingleCommit:
args.testSingleCommit = revparse(args.testSingleCommit.strip())
if not args.testSingleCommit:
return False
else:
if args.module == "qtbase":
print("Module to test is Qtbase. Verifying that COIN data is available for other modules before proceeding...")
if not findRevToBuild(args.testSingleCommit, "qtdeclarative", args.branch):
return False
else:
print("COIN data verified. Continuing.")
return True
elif args.testTwoCommit:
firstCommit = revparse(args.testTwoCommit.split(',')[0].strip())
secondCommit = args.testTwoCommit.split(',')[1].strip()
if secondCommit == "PARENT":
noRebuild = True
secondCommit = findRevToBuild(firstCommit, args.module, "", True)
print(f"Second Commit found: {secondCommit}")
args.testTwoCommit = f"{secondCommit},{firstCommit}" # Reverse the order of testing when the user selected the easy regression test option. Expect regression from parent to child, not improvement.
else:
secondCommit = revparse(secondCommit)
args.testTwoCommit = f"{firstCommit},{secondCommit}"
if not firstCommit and secondCommit:
return False
else:
if args.module == "qtbase":
print("Module to test is Qtbase. Verifying that COIN data is available for other modules before proceeding...")
if not findRevToBuild(firstCommit, "qtdeclarative", args.branch) or not findRevToBuild(secondCommit, "qtdeclarative", args.branch):
return False
else:
print("COIN data verified. Continuing.")
return True
def setWindowsEnv():
# runs the vsDevCmd file from the visual studio installation
vars = subprocess.check_output([args.VSDevEnv, '&&', 'set'])
# splits the output of the batch file and saves PATH variables from the batch to the local os.environ
for var in vars.splitlines():
var = var.decode('cp1252')
k, _, v = map(str.strip, var.strip().partition('='))
if k.startswith('?'):
continue
os.environ[k] = v
os.environ["PATH"] += (";" + builddir).replace("/", "\\")
os.environ["PATH"] += (";" + basedir + "/flex_bison/").replace("/", "\\")
os.environ["QTDIR"] = (builddir).replace("/", "\\")
def prepareEnv():
# A place to do OS specific actions on startup.
if args.environment:
for variable in args.environment.split(','):
varName, varValue = variable.split('=')
os.environ[varName] = varValue
if isWindowsOS:
# set up windows build env
print('Setting up windows build environment.')
setWindowsEnv()
def buildModule(module):
# Build the specified Qt module
print(f"Preparing to build {module}")
os.chdir(os.path.join(builddir, module))
if (module == "qtbase"):
if isWindowsOS:
configurecmd = ["configure.bat", "-prefix", installdir, "-no-pch", "-nomake", "tests", "-nomake", "examples", "-release", "-opensource", "-confirm-license", "-no-warnings-are-errors", "-opengl", "dynamic"]
else:
configurecmd = ["./configure", "-prefix", installdir, "-no-pch", "-developer-build", "-nomake", "tests", "-nomake", "examples", "-release", "-opensource", "-confirm-license", "-no-warnings-are-errors"]
print(f"Running Configure for Qtbase")
subprocess.run(configurecmd, check=False, shell=isWindowsOS)
else:
# qmake
print(f"Running QMake for {module}")
subprocess.run([os.path.join(installdir, "bin", f"qmake{exeExt}", ), f"{module}.pro"],
universal_newlines=True, shell=isWindowsOS)
# build it!
print(f"Building {module}...")
subprocess.run([compiler, "-j", f"{args.buildCores}"], universal_newlines=True, shell=isWindowsOS)
if not module == "qmlbench":
installModule(module)
if isWindowsOS and module == "qmlbench":
print(f"Building WinDeployQt")
# Also build and run winDeployQt on qmlbench
subprocess.run([os.path.join(installdir, "bin", f"qmake{exeExt}", ), "windeployqt.pro"],
universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, "qttools", "src", "windeployqt"))
subprocess.run([compiler, "-j", f"{args.buildCores}"],
universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, "qttools", "src", "windeployqt"))
subprocess.run([compiler, "install", "-j", f"{args.buildCores}"],
universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, "qttools", "src", "windeployqt"))
print(f"Running WinDeployQt against QMLBench")
proc = subprocess.run(["windeployqt.exe", "--qmldir", os.path.join(builddir, "qmlbench", "benchmarks"),
os.path.join(builddir, "qmlbench", "src", "release", "qmlbench.exe")],
universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(installdir, "bin"), check=True)
print(proc)
os.chdir(builddir)
def installModule(module):
# Run make install on the module.
print(f"Installing {module}")
subprocess.run([compiler, "install", "-j", f"{int(args.buildCores) * 3 }"], universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, module))
def runBenchmark():
# Actually execute qmlbench
benchmarkEnv = os.environ.copy()
if isWindowsOS:
benchmarkEnv["QT_OPENGL"] = args.openGLBackend
print(f"OpenGL Backend set to {benchmarkEnv['QT_OPENGL']}")
print("Cooling off for 20 seconds before benchmarking...")
sleep(20)
print(f"Starting Benchmark")
with open(os.path.join(builddir, "qmlbench", "results.json"), mode='wb') as results: # Write output as JSON to results file for later collection.
if isWindowsOS:
subprocess.run([os.path.join(builddir, "qmlbench", "src", "release", f"qmlbench{exeExt}"), "--json", "--shell", "frame-count", "--repeat", "10", os.path.join(builddir, "qmlbench", args.benchmark)], env=benchmarkEnv, stdout=results, shell=isWindowsOS)
else:
subprocess.run([os.path.join(builddir, "qmlbench", "src", f"qmlbench{exeExt}"), "--json", "--shell", "frame-count", "--repeat", "10", os.path.join(builddir, "qmlbench", args.benchmark)], stdout=results, shell=isWindowsOS)
print(f"Completed Benchmark")
def parseResults():
# Open the results file and find the "average" result key.
print(f"Parsing Benchmark results")
resultsJSON = {} # Typing.Dict()
benchmark = args.benchmark.replace('\\', '/')
with open(os.path.join(builddir, "qmlbench", "results.json")) as results:
resultsJSON = json.loads(results.read())
for key in resultsJSON:
if benchmark in key:
return int(resultsJSON[key]['average'])
return False
def findRevToBuild(refRevision, module, branch="", returnParent=False):
# Do a whole bunch of stuff to associate the revision we should build of other qt modules.
def getParentRev(commitJSON):
for commit in commitJSON:
parentRev = commit["revisions"][refRevision]["commit"]["parents"][0]["commit"]
if parentRev:
return parentRev
# Search codereview for the change and return some specific details. Start the saved response from codereview at position 4 to ignore some garbage in the response.
commitRaw = requests.get(f"https://codereview.qt-project.org/changes/?q=commit:{refRevision}&o=CURRENT_REVISION&o=CURRENT_COMMIT").text[4:]
if not commitRaw:
return False
try:
commitJSON = json.loads(commitRaw) # Try to parse the JSON
except IndexError:
print("Unable to find revision on codereview.")
saveResult(refRevision, "RevNotFound")
return False
if returnParent: # Just return the direct parent of refRevision. This would usually be used by the TwoCommit mode for easy regression testing against a new commit's parent.
return getParentRev(commitJSON)
changeID = ""
commitDetailsRaw = ""
# At least we parsed the response into valid JSON. Now, Look through the responses, validate that things are correct, and retreive the comments list.
for index, commit in enumerate(commitJSON):
if branch not in commit["branch"]:
print(f"ERROR: Selected revision of {module} was not commited on the {branch} branch. It was commited to {commit['branch']}. Please correct and resubmit the job.")
if index >= (len(commitJSON) - 1):
return False
else:
continue
elif args.module not in commit["project"]:
print(f"ERROR: Selected revision is not in {args.module}. It is part of {commit['project'].split('/')[1]}. Please correct and resubmit the job.")
if index >= (len(commitJSON) - 1):
return False
else:
continue
elif branch in commit["branch"] and module == args.module:
# This logic branch will return immediately because it's the module we're testing a specific commit.
# We'll check it out directly and just wanted to verify it exists.
# But if the module to test is qtquick3d, verify that the commit date is after the first
# known integration and the branch to test is new enough.
if module == "qtquick3d":
if subprocess.run(["git", "show", "-s", "--format=%ct", refRevision], cwd=os.path.join(builddir, "qt5", args.module), stdout=subprocess.PIPE).stdout < 1566565560:
return False
if (packaging.version.parse(args.branch) if not args.branch == "dev" else False) < packaging.version.parse(5.14):
return False
else:
return True
changeID = commit["change_id"]
print(f"Found Change ID: {changeID} based on commit {refRevision}")
# Pull the comments history on the change so we can look for a COIN integration ID.
request = requests.get(f"https://codereview.qt-project.org/changes/{changeID}/detail")
if request.status_code != 200:
continue # Something's fishy about the commit and codereview doesn't recognize it. Try the next commit in the list of responses from our original query.
else:
commitDetailsRaw = request.text[4:] # Gerrit responded favorably. Save the response, but still look at the others in the list from the original query.
if commitDetailsRaw:
break
# Try to load the commit's JSON
try:
commitDetails = json.loads(commitDetailsRaw)
except json.decoder.JSONDecodeError:
saveResult(refRevision, "RevFoundButBadAPIDetailResponse")
# Try again with the reference commit's parent. It should have been integrated.
parentRev = getParentRev(commitJSON)
sha = findRevToBuild(parentRev, module, branch)
if sha:
print("WARN: Using the reference commit's parent to build against. This should only happen if you're testing a change that hasn't been merged yet.")
return sha
else:
print("ERROR: Failed to load commit details from gerrit, and failed to use commit parent as backup. Please report this error.")
return False
integrationId = ""
# Scan the comments section for COIN's telltale message.
for message in commitDetails["messages"]:
if "Continuous Integration: Passed" in message["message"]:
integrationId = re.search(r"[0-9]{10}", message["message"]).group(0)
print(f"Found COIN integration {integrationId}")
break # Found it. No need to look at other comments.
# Some changes get pushed directly for some reason. They're untestible because we can't ask COIN what to build against unless it has a parent commit that integrated normally.
if not(integrationId):
print("WARN: The change ID selected did not have a COIN integration.")
for message in commitDetails["messages"]:
if "Change has been successfully cherry-picked" in message["message"]:
print(f"WARN: Change was Cherry-picked. Manual Review suggested.")
saveResult(refRevision, "NoIntegrationCherryPicked")
break
# Try again with the reference commit's parent. It should have been integrated.
parentRev = getParentRev(commitJSON)
print(f"INFO: Found Parent {parentRev}, trying it instead")
sha = findRevToBuild(parentRev, module, branch)
if sha:
print("WARN: Using the reference commit's parent to build against. This should only happen if you're testing a change that hasn't been merged yet.")
return sha
else:
print("ERROR: Failed to load commit details from gerrit, and failed to use commit parent as backup. Please report this error.")
return False
# Try to pull the saved integration data from coin for the integration we found. It contains the shas of all modules it built with at the time.
data = requests.get(f"https://testresults.qt.io/logs/qt/{args.module}/tasks/{integrationId}.thrift_bin").content
if b"404 Not Found" in data:
print(f"ERROR: Failed to find COIN data for integration ID {integrationId}. Maybe the commit we're trying isn't in the selected module of {args.module}. Trying the parent commit's integration data just in case.")
# Try again with the reference commit's parent. It should have been integrated.
parentRev = getParentRev(commitJSON)
print(f"INFO: Found Parent {parentRev}, trying it instead")
sha = findRevToBuild(parentRev, module, branch)