diff --git a/README.md b/README.md
index 35da454c9648c96c3b496413b8793d04fe961707..4e99b07a5da03a09ed0d22b7cd8b4f3f54f5e473 100644
--- a/README.md
+++ b/README.md
@@ -1,76 +1,96 @@
-# QMLBench Regression Finder
-QMLBench Regression Finder is a tool to assist in benchmarking changes and identifying bad commits in exiting branches.
-
-### What can I do with it?
-This framework provides a web interface for three types of benchmark runs:
-1. [Automatic bisect](./README.md#Automatic_Bisect)
-2. [Single commit](./README.md#Single-Commit)
-3. [Two commit comparison](./README.md#Two-Commit-Comparison)
-
-### How does it work?
-The QMLBench Regression Finder framework consists of a two parts:
-1. A nodejs server that operates a lightweight web interface, a job scheduler, and email notification subsystyem
-2. A python script that builds, tests, and collects results from QMLBench
-
-The framework is platform independent and can be deployed to as many clients as desired. A standalone mode is also available for local benchmarking.
-
-### Installation pre-requisites
-1. [All the usual stuff to build Qt from source](https://wiki.qt.io/Building_Qt_5_from_Git)
-2. [NodeJS](https://nodejs.org/en/) - Built on 10.15.3, but any recent version will likely work
-3. [Python 3](https://www.python.org/downloads/)
-   - Python3 submodules (installable with pip / pip3): 
-      - argparse
-      - requests
-      - thriftpy
-      - packaging
-4. (Windows) [MSVC Build Tools 2017](https://aka.ms/AA363al) - Other versions may work, but VS2017 is recommended for building Qt. Build tools are already included in full installations of Microsoft Visual Studio 2017.
-
-### How to install
-1. Download or clone this repository
-2. In the repository directory, run the following:
-   1. npm install
-      - __Note__: Python2 may be required for the "diskspace" module. If you encouter errors regarding this module during the "npm install" step, ensure that your Python2 installation directory is added to the __beginning__ of your "PATH" environment variable for this step.
-3. Celebrate! Installation is complete
-
-### How to run
-1. In the repository directory, run one of the following commands:
-   1. ```npm start standalone```
-      - Start the scheduler in standalone mode. This mode allows for jobs to be run on the local machine, and does not require additional servers to act as hosts.
-    2. ```npm start 10.9.70.10 10.9.70.11 10.9.70.12```
-       - Start the server in scheduler mode with local jobs enabled. This will allow scheduling jobs on both the local machine and remote hosts. A space separated list of remote hosts is required.
-    3. ```npm start nolocal 10.9.70.10 10.9.70.11 10.9.70.12```
-       - *__(Recommended configuration)__* Start the server in scheduler only mode. This will prevent the local host from being available as a benchmarking host. A space separated list of remote hosts is required.
-    4. ```npm start```
-       - Start the server in remote host mode. This disables the scheduler interface and configures the server to listen for connections from a scheduling server. This is the mode that must be used on remote hosts.
-
-   ### Optional parameters
-   Optional parameters are set via environment variables. The following options are available.
-
-   1. ```HTTP_PORT``` - The port on which to run the server. All instances of this system on a network must be configured to use the same port. ```Default: 8080```
-   2. ```BUILD_CORES``` - The number of CPU threads to use when building Qt ```Default: 8```
-   3. ```VS_DEV_ENV``` - The full path to VsDevCmd.bat provided by Microsoft Visual Studio ```Default: C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/Common7/Tools/VsDevCmd.bat```
-
-## Using the interface
-
-### General guidelines
-
-   1. If using the bisect or two commit comparison, the commits must be on the same major branch.
-   2. You need to know which module the commit(s) were made in (i.e. qtdeclarative or qtbase for example)
-   3. You need to have a rough idea of the expected performance regression. Testing regressions smaller than ~5% is not recommended due to natural test noise.
-   4. You need to know a specific QMLBench test that is affected by a given commit's changes. 
-
-### Automatic Bisect
-   This mode takes two commits from the same major branch (i.e. 5.12) and emulates a git-bisect like behavior, testing various commits in-between the known good and known bad commits to find a performance regression.
-   
-   Notes:
-   1. The "known good" commit must be an ancestor (is older and commited to the same branch) of the "known bad" commit.
-   2. The commits must have been merged into Qt successfully.
-
-### Single commit
-   This mode tests a single commit and provides a single benchmark result. In this mode, WIP (work-in-progress) commits and commits not yet merged can be tested.
-
-### Two Commit Comparison
-   This mode tests two commits and provides results for each, as well as a percentage difference between the two. In this mode, WIP (work-in-progress) commits and commits not yet merged can be tested.
-   
-   Notes:
-   1. Though the commits can be WIP or unmerged, they must still be in the same module and of the same branch. Use multiple runs of the single-commit test mode to test commits in different branches or modules.
\ No newline at end of file
+# QMLBench Regression Finder
+QMLBench Regression Finder is a tool to assist in benchmarking changes and identifying bad commits in exiting branches.
+
+### What can I do with it?
+This framework provides a web interface for three types of benchmark runs:
+1. [Automatic bisect](./README.md#Automatic_Bisect)
+2. [Single commit](./README.md#Single-Commit)
+3. [Two commit comparison](./README.md#Two-Commit-Comparison)
+
+### How does it work?
+The QMLBench Regression Finder framework consists of a two parts:
+1. A nodejs server that operates a lightweight web interface, a job scheduler, and email notification subsystyem
+2. A python script that builds, tests, and collects results from QMLBench
+
+The framework is platform independent and can be deployed to as many clients as desired. A standalone mode is also available for local benchmarking.
+
+### Installation pre-requisites
+1. [All the usual stuff to build Qt from source](https://wiki.qt.io/Building_Qt_5_from_Git)
+2. [NodeJS](https://nodejs.org/en/) - Built on 10.15.3, but any recent version will likely work
+3. [Python 3](https://www.python.org/downloads/)
+   - Python3 submodules (installable with pip / pip3): 
+      - argparse
+      - requests
+      - thriftpy
+      - packaging
+4. (Windows) [MSVC Build Tools 2017](https://aka.ms/AA363al) - Other versions may work, but VS2017 is recommended for building Qt. Build tools are already included in full installations of Microsoft Visual Studio 2017.
+
+### How to install
+1. Download or clone this repository
+2. In the repository directory, run the following:
+   1. npm install
+      - __Note__: Python2 may be required for the "diskspace" module. If you encouter errors regarding this module during the "npm install" step, ensure that your Python2 installation directory is added to the __beginning__ of your "PATH" environment variable for this step.
+3. Celebrate! Installation is complete
+
+### How to run
+1. In the repository directory, run one of the following commands:
+   1. ```npm start standalone```
+      - Start the scheduler in standalone mode. This mode allows for jobs to be run on the local machine, and does not require additional servers to act as hosts.
+    2. ```npm start 10.9.70.10 10.9.70.11 10.9.70.12```
+       - Start the server in scheduler mode with local jobs enabled. This will allow scheduling jobs on both the local machine and remote hosts. A space separated list of remote hosts is required.
+    3. ```npm start nolocal 10.9.70.10 10.9.70.11 10.9.70.12```
+       - *__(Recommended configuration)__* Start the server in scheduler only mode. This will prevent the local host from being available as a benchmarking host. A space separated list of remote hosts is required.
+    4. ```npm start```
+       - Start the server in remote host mode. This disables the scheduler interface and configures the server to listen for connections from a scheduling server. This is the mode that must be used on remote hosts.
+
+   ### Optional parameters
+   Optional parameters are set via environment variables. The following options are available.
+
+   1. ```HTTP_PORT``` - The port on which to run the server. All instances of this system on a network must be configured to use the same port. ```Default: 8080```
+   2. ```BUILD_CORES``` - The number of CPU threads to use when building Qt ```Default: [All available cores]```
+   3. ```VS_DEV_ENV``` - The full path to VsDevCmd.bat provided by Microsoft Visual Studio ```Default: "C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/Common7/Tools/VsDevCmd.bat"```
+   4. ```SMTP_SERVER``` - A SMTP server that can send emails. Authentication not supported - The mail server must accept anonymous connections. ```Default: smtp.intra.qt.io```
+   5. ```SMTP_PORT``` - Specify the port to connect to the mail server. ```Default: 25```
+
+## Using the interface
+
+### General guidelines
+
+   1. If using the bisect or two commit comparison, the commits must be on the same major branch.
+   2. You need to know which module the commit(s) were made in (i.e. qtdeclarative or qtbase for example)
+   3. You need to have a rough idea of the expected performance regression. Testing regressions smaller than ~5% is not recommended due to natural test noise.
+   4. It is **strongly recommended** that a QMLBench Regression Finder running as a slave is hosted on a machine that is performing no other tasks and has been configured for performance stability.
+   4. You need to know a specific QMLBench test that is affected by a given commit's changes.
+      - You may also write and upload custom QMLBench benchmarks.
+      - See [QMLBench](https://github.com/qt-labs/qmlbench) for an overview of the QMLBench test framework.
+      - The benchmark [creation/quick.item/delegates_item.qml](https://github.com/qt-labs/qmlbench/blob/master/benchmarks/auto/creation/quick.item/delegates_item.qml) may provide a good starting point for writing your own QMLBench benchmark.
+
+### Automatic Bisect
+   This mode takes two commits from the same major branch (i.e. 5.12) and emulates a git-bisect like behavior, testing various commits in-between the known good and known bad commits to find a performance regression.
+   
+   Notes:
+   1. The "known good" commit must be an ancestor (is older and commited to the same branch) of the "known bad" commit.
+   2. The commits must have been merged into Qt successfully.
+
+### Single commit
+   This mode tests a single commit and provides a single benchmark result. In this mode, WIP (work-in-progress) commits and commits not yet merged can be tested.
+
+### Two Commit Comparison
+   This mode tests two commits and provides results for each, as well as a percentage difference between the two. In this mode, WIP (work-in-progress) commits and commits not yet merged can be tested.
+   
+   Notes:
+   1. Though the commits can be WIP or unmerged, they must still be in the same module and of the same branch. Use multiple runs of the single-commit test mode to test commits in different branches or modules.
+
+## Troubleshooting
+
+### Common problems
+   1. **Build failures**
+      1. Bad checkout of some qt repo. If a test fails to build when you know it should, try simply deleting the QtBuild directory on the slave and re-stage the rest run.
+   2. **Failure to produce a test result**
+      1. There is no validation of custom QML benchmark file. If everything looks good in the build but QMLBench fails to produce a result or crashes, make sure your custom QMLBench benchmark .qml file is formatted correctly.
+      2. OpenGL is currently required on all platforms. If no OpenGL is installed, or if Angle is selected on windows but unsupported on the host, QMLBench will likely crash.
+   3. **Test result instability**
+      1. If you're using icecc in your linux environments and encounter test performance instability, ensure icecc's config is set to disable remote jobs.
+      2. Be sure the system was not attempting to perform automatic updates.
+      3. Increase the count of test repeats. Default is 10. Each repeat runs for approximately 20 seconds.
+         - Check regressionFinder.py::runBenchmark() to adjust the number of test repeats.
diff --git a/regressionFinder.py b/regressionFinder.py
index 1b536a65e6d633686347c904d905fa7d4bea9065..a86a1f3464c222adb2965a58362100ca323e9599 100644
--- a/regressionFinder.py
+++ b/regressionFinder.py
@@ -1,940 +1,940 @@
-import os
-import subprocess
-import argparse
-import platform
-import stat
-import shutil
-import json
-import requests
-from thrift import storagestructs
-import thriftpy.utils
-import thriftpy
-import re
-from time import sleep
-import atexit
-import packaging
-
-args = []
-basedir = os.getcwd()
-builddir = os.path.join(basedir, "QtBuild")
-installdir = os.path.join(builddir, "Install")
-isWindowsOS = (platform.system() == 'Windows')
-compiler = basedir + "\\JOM\\jom.exe" if isWindowsOS else "make"
-exeExt = '.exe' if isWindowsOS else ''
-goodBaselineScore = 0
-badBaselineScore = 0
-regressionTargetScore = 0
-observedRegressionPercent = 0
-refQtbaseRev = ""
-testType = ""
-noRebuild = False
-finalResults = []
-
-# Bisect tracking items
-revisionList = []
-bisectResults = [[], ]
-bisectPosition = 0
-bisectGoodPos = 0
-bisectBadPos = 0
-bisectLastStatus = ""
-bisectLastDirection = ""
-
-
-def on_rm_error(func, path, exc_info):
-    # Handler for files that fail to be removed.
-    # path contains the path of the file that couldn't be removed
-    # let's just assume that it's read-only and unlink it.
-    try:
-        os.chmod(path, stat.S_IWRITE)
-        os.unlink(path)
-    except Exception as e:
-        print("There was an error removing a file from disk. Exception: {0}".format(e))
-
-
-def parseArgs():
-    parser = argparse.ArgumentParser()
-    parser.add_argument("--setupEnv", dest="setupEnv", action="store_true", help="Run with --setupEnv to initialize the build environment by cloning needed repos or pulling updates as needed. Exits upon completion.")
-    parser.add_argument("--branch", dest="branch", type=str, help="Branch of Qt such as \'5.12\' or \'dev\'")
-    parser.add_argument("--moduleToTest", dest="module", type=str, help="Module where regression is suspected such as \'qtbase\'.")
-    parser.add_argument("--knownBadRev", dest="knownBadRev", type=str, help="Known bad revision in the module where a regression is expected.")
-    parser.add_argument("--knownGoodRev", dest="knownGoodRev", type=str, help="Known good revision in the module where a regression is expected")
-    parser.add_argument("--regressionTarget", dest="regressionTarget", type=int, help="Expected regression, in percent, to assist in finding the correct regressive commit. Be conservative if unsure.")
-    parser.add_argument("--fuzzyRange", dest="fuzzyRange", type=int, default=2, help="Provide a value to use as a fuzzy range to check the regression against. Using \'--regressionTarget 20 --fuzzyRange 5\' will prefer regressions between 15-25%% as the bad commit")
-    parser.add_argument("--buildCores", dest="buildCores", type=int, default=8, help="Number of build cores to use when building")
-    parser.add_argument("--wipeWorkspace", dest="wipeWorkspace", action="store_true", help="Clear the entire workspace and force re-cloning of all modules")
-    parser.add_argument("--VSDevEnv", dest="VSDevEnv", help="Full path to Visual studio VsDevCmd.bat file for windows build environments. Defaults to default installation for VS 2017 Build Tools", default="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/Common7/Tools/VsDevCmd.bat")
-    parser.add_argument("--benchmark", dest="benchmark", type=str, help="path to single benchmark to run. Only run one benchmark at a time!")
-    parser.add_argument("--testSingleCommit", dest="testSingleCommit", type=str, help="Set this parameter to a commit ID to benchmark. Setting this parameter overrides bisecting behavior.")
-    parser.add_argument("--testTwoCommit", dest="testTwoCommit", type=str, help="Set this parameter to a comma-separated list of two commit IDs to benchmark and compare. Setting this parameter overrides bisecting behavior.")
-    parser.add_argument("--FirstBuildOnHead", dest="firstBuildOnHead", action="store_true", help="Enable this parameter to build the first commit against branch HEAD instead of searching for a COIN integration.")
-    parser.add_argument("--SecondBuildOnHead", dest="secondBuildOnHead", action="store_true", help="Enable this parameter to build the second commit against branch HEAD instead of searching for a COIN integration.")
-    parser.add_argument("--OpenGLBackend", dest="openGLBackend", type=str, default="desktop", help="Render backend options. Valid options are \'dekstop\', \'angle\', \'software\'")
-    parser.add_argument("--jobName", dest="jobName", type=str, help="unique job name used for writing results file to logs directory. Typically a hash of the job to be run.")
-    parser.add_argument("--environment", dest="environment", type=str, help="Comma separated list of environment variables and values to use for the build and test environment.")
-
-    return parser.parse_args()
-
-
-def initRepository(repo):
-    # Clone any needed repositories. If they already exist, sweep them and pull changes.
-    module = repo[repo.index('/') + 1:]
-    branch = args.branch if module != "qmlbench" else "master"
-    if not branch:
-        branch = "dev"
-
-    print(repo, branch)
-
-    def cloneRepo():
-        subprocess.run(["git", "clone", "-b", branch, f'https://code.qt.io/{repo}.git'], stderr=subprocess.PIPE, universal_newlines=True, cwd=builddir)
-        subprocess.run(["git", "submodule", "update", "--init"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
-        
-
-    # clone modules if necessary, otherwise just pull latest.
-    if not os.path.exists(os.path.join(builddir, module)):
-        cloneRepo()
-    else:
-        subprocess.run(["git", "clean", "-dqfx"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
-        subprocess.run(["git", "reset", "--hard", f"origin/{args.branch}", "--quiet"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
-        subprocess.run(["git", "pull", "origin", args.branch, "--quiet"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
-        subprocess.run(["git", "checkout", branch], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
-
-    if repo == "qt/qt5":
-        # Run the reset and pull again. Sometimes the previous pull leads to merge conflicts.
-        # We don't care, just needed the first pull to get the lastest submodule list
-        # and then reset and pull again to get the latest changes.
-        subprocess.run(["git", "reset", "--hard", f"origin/{args.branch}", "--quiet"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
-        subprocess.run(["git", "pull", "origin", args.branch, "--quiet"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
-        subprocess.run(["git", "submodule", "update", "--init"], stderr=subprocess.PIPE, universal_newlines=True, cwd=builddir)
-
-def validateTest():
-    # Verify the selected test exists
-    global args
-    if not os.path.exists(os.path.join(builddir, 'qmlbench', args.benchmark)):
-        args.benchmark = os.path.join("benchmarks", args.benchmark)
-        if not os.path.exists(os.path.join(builddir, 'qmlbench', args.benchmark)):
-            print(f"Specified benchmark at {os.path.join(builddir, 'qmlbench', args.benchmark)} does not exist. Please check the path and try again.")
-            return False
-    return True
-
-
-def validateCommits():
-    # Verify that the provided commit(s) exist in the Qt repository.
-
-    def revparse(rev):
-        # Rev-parse our given SHA1 and see if Git returns anything. If it pushes anything to stdout, a match was found.
-        proc = subprocess.run(["git", "rev-parse", rev], cwd=os.path.join(builddir, args.module),
-                              stderr=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True, shell=isWindowsOS)
-        if proc.stderr:
-            if lookupUnmergedCommit(rev, args.module):
-                return revparse(rev)
-            else:
-                print(f"Unable to verify commit {rev}. Please check to make sure you have the correct repository and there are no typos.")
-                return False
-        else:
-            print(proc.stdout.strip().split("\n"))
-            if len(proc.stdout.strip().split("\n")) > 1:
-                # Rev-parse will print one matching SHA per line.
-                print(f"Provided commit {rev} matches more than one commit. Try using the full SHA1.")
-                return False
-            else:
-                return proc.stdout.strip()
-
-    global args
-    global noRebuild
-
-    # Do some logic to determine which type of test we're running.
-    if args.knownGoodRev:
-        args.knownGoodRev = args.knownGoodRev.strip()  # Cleanup inputs
-        args.knownBadRev = args.knownBadRev.strip()
-        # Try to see if we have at least one commit between the known good and bad commits. This will fail if the known bad is not an ancestor of the known good.
-        proc = subprocess.run(["git", "rev-list", f"{args.knownGoodRev}..{args.knownBadRev}", f"^{args.knownGoodRev}"], cwd=os.path.join(builddir, args.module),
-                              stderr=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True, shell=isWindowsOS)
-
-        if proc.returncode != 0:
-            print(f"ERROR: Unable to verify selected commits. Please double check your input.\nGit error: {proc.stderr}")
-            return False
-
-        try:
-            revisionList = proc.stdout.split('\n')
-        except Exception as e:
-            print(f"ERROR: No commits between good and bad commits. Nothing to test. Hint: Known bad commit {args.knownBadRev} might be the bad commit.")
-            return False
-
-        if revisionList:
-            revisionList.pop()  # rev-list will return the list with the known bad commit as the head element. Get rid of it permenantly.
-
-        if len(revisionList) < 1:
-            print(f"ERROR: No commits between good and bad commits. Nothing to test. Hint: Known bad commit {args.knownBadRev} might be the bad commit.")
-            return False
-        if args.module == "qtbase":
-            print("Module to test is Qtbase. Verifying that COIN data is available for other modules before proceeding...")
-            if not findRevToBuild(args.knownGoodRev, "qtdeclarative", args.branch) or not findRevToBuild(args.knownBadRev, "qtdeclarative", args.branch):
-                return False
-            else:
-                print("COIN data verified. Continuing.")
-        return True
-
-    elif args.testSingleCommit:
-        args.testSingleCommit = revparse(args.testSingleCommit.strip())
-        if not args.testSingleCommit:
-            return False
-        else:
-            if args.module == "qtbase":
-                print("Module to test is Qtbase. Verifying that COIN data is available for other modules before proceeding...")
-                if not findRevToBuild(args.testSingleCommit, "qtdeclarative", args.branch):
-                    return False
-                else:
-                    print("COIN data verified. Continuing.")
-            return True
-    elif args.testTwoCommit:
-        firstCommit = revparse(args.testTwoCommit.split(',')[0].strip())
-        secondCommit = args.testTwoCommit.split(',')[1].strip()
-        if secondCommit == "PARENT":
-            noRebuild = True
-            secondCommit = findRevToBuild(firstCommit, args.module, "", True)
-            print(f"Second Commit found: {secondCommit}")
-            args.testTwoCommit = f"{secondCommit},{firstCommit}"  # Reverse the order of testing when the user selected the easy regression test option. Expect regression from parent to child, not improvement.
-        else:
-            secondCommit = revparse(secondCommit)
-            args.testTwoCommit = f"{firstCommit},{secondCommit}"
-        if not firstCommit and secondCommit:
-            return False
-        else:
-            if args.module == "qtbase":
-                print("Module to test is Qtbase. Verifying that COIN data is available for other modules before proceeding...")
-                if not findRevToBuild(firstCommit, "qtdeclarative", args.branch) or not findRevToBuild(secondCommit, "qtdeclarative", args.branch):
-                    return False
-                else:
-                    print("COIN data verified. Continuing.")
-            return True
-
-
-def setWindowsEnv():
-    # runs the vsDevCmd file from the visual studio installation
-    vars = subprocess.check_output([args.VSDevEnv, '&&', 'set'])
-
-    # splits the output of the batch file and saves PATH variables from the batch to the local os.environ
-    for var in vars.splitlines():
-        var = var.decode('cp1252')
-        k, _, v = map(str.strip, var.strip().partition('='))
-        if k.startswith('?'):
-            continue
-        os.environ[k] = v
-
-    os.environ["PATH"] += (";" + builddir).replace("/", "\\")
-    os.environ["PATH"] += (";" + basedir + "/flex_bison/").replace("/", "\\")
-    os.environ["QTDIR"] = (builddir).replace("/", "\\")
-
-
-def prepareEnv():
-    # A place to do OS specific actions on startup.
-    if args.environment:
-        for variable in args.environment.split(','):
-            varName, varValue = variable.split('=')
-            os.environ[varName] = varValue
-
-    if isWindowsOS:
-        # set up windows build env
-        print('Setting up windows build environment.')
-        setWindowsEnv()
-
-
-def buildModule(module):
-    # Build the specified Qt module
-    print(f"Preparing to build {module}")
-    os.chdir(os.path.join(builddir, module))
-    if (module == "qtbase"):
-        if isWindowsOS:
-            configurecmd = ["configure.bat", "-prefix", installdir, "-no-pch", "-nomake", "tests", "-nomake", "examples", "-release", "-opensource", "-confirm-license", "-no-warnings-are-errors", "-opengl", "dynamic"]
-
-        else:
-            configurecmd = ["./configure", "-prefix", installdir, "-no-pch", "-developer-build", "-nomake", "tests", "-nomake", "examples", "-release", "-opensource", "-confirm-license", "-no-warnings-are-errors"]
-        print(f"Running Configure for Qtbase")
-        subprocess.run(configurecmd, check=False, shell=isWindowsOS)
-
-    else:
-        # qmake
-        print(f"Running QMake for {module}")
-        subprocess.run([os.path.join(installdir, "bin", f"qmake{exeExt}", ), f"{module}.pro"],
-                       universal_newlines=True, shell=isWindowsOS)
-
-    # build it!
-    print(f"Building {module}...")
-    subprocess.run([compiler, "-j", f"{args.buildCores}"], universal_newlines=True, shell=isWindowsOS)
-    if not module == "qmlbench":
-        installModule(module)
-
-    if isWindowsOS and module == "qmlbench":
-        print(f"Building WinDeployQt")
-        # Also build and run winDeployQt on qmlbench
-        subprocess.run([os.path.join(installdir, "bin", f"qmake{exeExt}", ), "windeployqt.pro"],
-                       universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, "qttools", "src", "windeployqt"))
-        subprocess.run([compiler, "-j", f"{args.buildCores}"],
-                       universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, "qttools", "src", "windeployqt"))
-        subprocess.run([compiler, "install", "-j", f"{args.buildCores}"],
-                       universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, "qttools", "src", "windeployqt"))
-        print(f"Running WinDeployQt against QMLBench")
-        proc = subprocess.run(["windeployqt.exe", "--qmldir", os.path.join(builddir, "qmlbench", "benchmarks"),
-                               os.path.join(builddir, "qmlbench", "src", "release", "qmlbench.exe")],
-                              universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(installdir, "bin"), check=True)
-        print(proc)
-
-    os.chdir(builddir)
-
-
-def installModule(module):
-    # Run make install on the module.
-    print(f"Installing {module}")
-    subprocess.run([compiler, "install", "-j", f"{int(args.buildCores) * 3 }"], universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, module))
-
-
-def runBenchmark():
-    # Actually execute qmlbench
-    benchmarkEnv = os.environ.copy()
-    if isWindowsOS:
-        benchmarkEnv["QT_OPENGL"] = args.openGLBackend
-        print(f"OpenGL Backend set to {benchmarkEnv['QT_OPENGL']}")
-
-    print("Cooling off for 20 seconds before benchmarking...")
-    sleep(20)
-    print(f"Starting Benchmark")
-    with open(os.path.join(builddir, "qmlbench", "results.json"), mode='wb') as results:  # Write output as JSON to results file for later collection.
-        if isWindowsOS:
-            subprocess.run([os.path.join(builddir, "qmlbench", "src", "release", f"qmlbench{exeExt}"), "--json", "--shell", "frame-count", "--repeat", "10", os.path.join(builddir, "qmlbench", args.benchmark)], env=benchmarkEnv, stdout=results, shell=isWindowsOS)
-        else:
-            subprocess.run([os.path.join(builddir, "qmlbench", "src", f"qmlbench{exeExt}"), "--json", "--shell", "frame-count", "--repeat", "10", os.path.join(builddir, "qmlbench", args.benchmark)], stdout=results, shell=isWindowsOS)
-    print(f"Completed Benchmark")
-
-
-def parseResults():
-    # Open the results file and find the "average" result key.
-    print(f"Parsing Benchmark results")
-    resultsJSON = {}  # Typing.Dict()
-    benchmark = args.benchmark.replace('\\', '/')
-    with open(os.path.join(builddir, "qmlbench", "results.json")) as results:
-        resultsJSON = json.loads(results.read())
-
-    for key in resultsJSON:
-        if benchmark in key:
-            return int(resultsJSON[key]['average'])
-
-    return False
-
-
-def findRevToBuild(refRevision, module, branch="", returnParent=False):
-    # Do a whole bunch of stuff to associate the revision we should build of other qt modules.
-
-    def getParentRev(commitJSON):
-        for commit in commitJSON:
-            parentRev = commit["revisions"][refRevision]["commit"]["parents"][0]["commit"]
-            if parentRev:
-                return parentRev
-
-    # Search codereview for the change and return some specific details. Start the saved response from codereview at position 4 to ignore some garbage in the response.
-    commitRaw = requests.get(f"https://codereview.qt-project.org/changes/?q=commit:{refRevision}&o=CURRENT_REVISION&o=CURRENT_COMMIT").text[4:]
-    if not commitRaw:
-        return False
-    try:
-        commitJSON = json.loads(commitRaw)  # Try to parse the JSON
-    except IndexError:
-        print("Unable to find revision on codereview.")
-        saveResult(refRevision, "RevNotFound")
-        return False
-
-    if returnParent:  # Just return the direct parent of refRevision. This would usually be used by the TwoCommit mode for easy regression testing against a new commit's parent.
-        return getParentRev(commitJSON)
-
-    changeID = ""
-    commitDetailsRaw = ""
-
-    # At least we parsed the response into valid JSON. Now, Look through the responses, validate that things are correct, and retreive the comments list.
-    for index, commit in enumerate(commitJSON):
-        if branch not in commit["branch"]:
-            print(f"ERROR: Selected revision of {module} was not commited on the {branch} branch. It was commited to {commit['branch']}. Please correct and resubmit the job.")
-            if index >= (len(commitJSON) - 1):
-                return False
-            else:
-                continue
-        elif args.module not in commit["project"]:
-            print(f"ERROR: Selected revision is not in {args.module}. It is part of {commit['project'].split('/')[1]}. Please correct and resubmit the job.")
-            if index >= (len(commitJSON) - 1):
-                return False
-            else:
-                continue
-        elif branch in commit["branch"] and module == args.module:
-            # This logic branch will return immediately because it's the module we're testing a specific commit.
-            # We'll check it out directly and just wanted to verify it exists.
-            # But if the module to test is qtquick3d, verify that the commit date is after the first
-            # known integration and the branch to test is new enough.
-            if module == "qtquick3d":
-                if subprocess.run(["git", "show", "-s", "--format=%ct", refRevision], cwd=os.path.join(builddir, "qt5", args.module), stdout=subprocess.PIPE).stdout < 1566565560:
-                    return False
-                if (packaging.version.parse(args.branch) if not args.branch == "dev" else False) < packaging.version.parse(5.14):
-                    return False
-            else:
-                return True
-
-        changeID = commit["change_id"]
-        print(f"Found Change ID: {changeID} based on commit {refRevision}")
-
-        # Pull the comments history on the change so we can look for a COIN integration ID.
-        request = requests.get(f"https://codereview.qt-project.org/changes/{changeID}/detail")
-        if request.status_code != 200:
-            continue  # Something's fishy about the commit and codereview doesn't recognize it. Try the next commit in the list of responses from our original query.
-        else:
-            commitDetailsRaw = request.text[4:]  # Gerrit responded favorably. Save the response, but still look at the others in the list from the original query.
-            if commitDetailsRaw:
-                break
-
-    # Try to load the commit's JSON
-    try:
-        commitDetails = json.loads(commitDetailsRaw)
-    except json.decoder.JSONDecodeError:
-        saveResult(refRevision, "RevFoundButBadAPIDetailResponse")
-
-        # Try again with the reference commit's parent. It should have been integrated.
-        parentRev = getParentRev(commitJSON)
-
-        sha = findRevToBuild(parentRev, module, branch)
-        if sha:
-            print("WARN: Using the reference commit's parent to build against. This should only happen if you're testing a change that hasn't been merged yet.")
-            return sha
-        else:
-            print("ERROR: Failed to load commit details from gerrit, and failed to use commit parent as backup. Please report this error.")
-            return False
-
-    integrationId = ""
-
-    # Scan the comments section for COIN's telltale message.
-    for message in commitDetails["messages"]:
-        if "Continuous Integration: Passed" in message["message"]:
-            integrationId = re.search(r"[0-9]{10}", message["message"]).group(0)
-            print(f"Found COIN integration {integrationId}")
-            break  # Found it. No need to look at other comments.
-
-    # Some changes get pushed directly for some reason. They're untestible because we can't ask COIN what to build against unless it has a parent commit that integrated normally.
-    if not(integrationId):
-        print("WARN: The change ID selected did not have a COIN integration.")
-        for message in commitDetails["messages"]:
-            if "Change has been successfully cherry-picked" in message["message"]:
-                print(f"WARN: Change was Cherry-picked. Manual Review suggested.")
-                saveResult(refRevision, "NoIntegrationCherryPicked")
-                break
-        # Try again with the reference commit's parent. It should have been integrated.
-        parentRev = getParentRev(commitJSON)
-        print(f"INFO: Found Parent {parentRev}, trying it instead")
-        sha = findRevToBuild(parentRev, module, branch)
-        if sha:
-            print("WARN: Using the reference commit's parent to build against. This should only happen if you're testing a change that hasn't been merged yet.")
-            return sha
-        else:
-            print("ERROR: Failed to load commit details from gerrit, and failed to use commit parent as backup. Please report this error.")
-            return False
-
-    # Try to pull the saved integration data from coin for the integration we found. It contains the shas of all modules it built with at the time.
-    data = requests.get(f"https://testresults.qt.io/logs/qt/{args.module}/tasks/{integrationId}.thrift_bin").content
-    if b"404 Not Found" in data:
-        print(f"ERROR: Failed to find COIN data for integration ID {integrationId}. Maybe the commit we're trying isn't in the selected module of {args.module}. Trying the parent commit's integration data just in case.")
-
-        # Try again with the reference commit's parent. It should have been integrated.
-        parentRev = getParentRev(commitJSON)
-        print(f"INFO: Found Parent {parentRev}, trying it instead")
-        sha = findRevToBuild(parentRev, module, branch)
-        if sha:
-            print("WARN: Using the reference commit's parent to build against because the target commit had no COIN data.")
-            return sha
-        else:
-            print(f"ERROR: Failed to load parent COIN integration data. This is probably permanently fatal. Commit {refRevision} can't be reliably tested.")
-            return False
-        saveResult(refRevision, "BadCOINData")
-        return False
-    else:
-        try:
-            # Convert the integration's binary thrift_bin file back into a workitem object.
-            workitem = thriftpy.utils.deserialize(storagestructs.Task(), data)
-        except Exception as e:
-            print(f"Failed to extract COIN data for integration ID {integrationId}.")
-            # Try again with the reference commit's parent. It should have been integrated.
-            parentRev = getParentRev(commitJSON)
-            print(f"INFO: Found Parent {parentRev}, trying it instead")
-            sha = findRevToBuild(parentRev, module, branch)
-            if sha:
-                print("WARN: Using the reference commit's parent to build against because the target commit had bad COIN data.")
-                return sha
-            else:
-                print(f"ERROR: Failed to load parent COIN integration data. This is probably permanently fatal. Commit {refRevision} can't be reliably tested.")
-                return False
-
-        # Ask git to identify the SHA1 of our target module to build based on the sha of the change we're testing. This uses the Qt5 supermodule.
-
-        try:
-            proc = subprocess.run(["git", "ls-tree", workitem.product.sha1, module, "-z"], cwd=os.path.join(builddir, "qt5"), stdout=subprocess.PIPE)
-            sha = str(proc.stdout).split(" ")[2].split("\\t")[0]
-            print(f'Found {module} sha1: {sha}')
-            return sha
-        except Exception:
-            # Couldn't find the qt5 product sha from the module to build.
-            print(f"ERROR: Couldn't find COIN product sha {workitem.product.sha1} for module {module} in qt5.")
-            saveResult(refRevision, "BadCOINData")
-            return False
-        #####################################################################################################################
-        ### Try to use the internal COIN as a backup. This code will not be functional outside of COIN's local network!!! ###
-        #####################################################################################################################
-
-        # data = requests.get(f"https://testresults.qt.io/coin/api/item/qt/{args.module}/tasks/{integrationId}.thrift_bin").content
-        # print(data)
-        # try:
-        #     workitem = thriftpy.utils.deserialize(storagestructs.Task(), data)
-        #     print(workitem.product)
-        #     proc = subprocess.run(["git", "ls-tree", workitem.product.sha1, module, "-z"], cwd=os.path.join(builddir, "qt5"), stdout=subprocess.PIPE)
-        #     sha = str(proc.stdout).split(" ")[2].split("\\t")[0]
-        #     print(f'Found {module} sha1: {sha} from internal coin.')
-        #     return sha
-        # except:
-        #     print(f"Also Failed to import COIN data from internal COIN for integration ID {integrationId}. Maybe the commit we're trying isn't in the selected module of {args.module}")
-
-        ############################
-        ### End of internal code ###
-        ############################
-
-
-def determineBuildEligibility(revision, module, branch):
-    # Verify that the SHA provided for our module to test is valid and can be built.
-    return findRevToBuild(revision, module, branch)
-
-
-def lookupUnmergedCommit(revision, module):
-    commitRaw = requests.get(f"https://codereview.qt-project.org/changes/?q=commit:{revision}&o=ALL_REVISIONS&o=CURRENT_COMMIT").text[4:]
-    if not commitRaw:
-        return False
-    try:
-        commitJSON = json.loads(commitRaw)[0]
-    except IndexError:
-        print(f"ERROR: Unable to find revision on codereview: {revision}")
-        return False
-    try:
-        # Get the fetch addresses so we can pull and check out the revision.
-        for item in commitJSON["revisions"]:
-            if revision in item:
-                revision = item
-        url = commitJSON["revisions"][revision]["fetch"]["anonymous http"]["url"]
-        ref = commitJSON["revisions"][revision]["fetch"]["anonymous http"]["ref"]
-    except Exception:
-        print(f"ERROR: Unable to get ref fetch data for commit to test {revision}")
-        return False
-    print(f"Fetching {module} patch ref {ref}")
-    subprocess.run(["git", "fetch", url, ref], universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, module))
-    return True
-
-
-def checkout(revision, module, bisecting=False, branch="", buildOnHead=False):
-    # Run git checkouts
-    print(f"Running Checkout for {module}, using revision {revision} as the reference")
-    print(f"This module should {'' if buildOnHead  else 'not '}be built on head ")
-    if module == args.module:
-        print(f"This is the module to test. Cleaning {module}.")
-        subprocess.run(["git", "clean", "-dqfx"], universal_newlines=True, cwd=os.path.join(builddir, module))
-        print(f"Trying checkout of module to test with revision {revision}")
-        proc = subprocess.run(["git", "checkout", revision], cwd=os.path.join(builddir, module), stderr=subprocess.PIPE,
-                              universal_newlines=True, shell=isWindowsOS)
-        if "reference is not a tree" in proc.stderr:
-            # We must be trying to build a commit that's not yet merged. Let's look it up.
-            lookupUnmergedCommit(revision, module)
-            print(f"Checking out unmerged commit {revision}")
-            proc = subprocess.run(["git", "checkout", "FETCH_HEAD"], cwd=os.path.join(builddir, module), stderr=subprocess.PIPE,
-                                  universal_newlines=True, shell=isWindowsOS)
-            return True
-
-        ########### NOT YET PROPERLY IMPLEMENTED ###########
-        # needPatch = False # TESTING
-        # if needPatch:
-        #     print("Applying patches...")
-        #     print("Applying patch refs/changes/77/249377/2: QtQuickControls2: QQuickPopupPositioner: fix crash on application exit")
-        #     subprocess.run(["git", "fetch", "https://codereview.qt-project.org/qt/qtquickcontrols2", "refs/changes/77/249377/2"], universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, module))
-        #     subprocess.run(["git", "cherry-pick", "FETCH_HEAD", "-X", "theirs"], universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, module))
-
-            # Example of implementing hard-coded patch:
-            # try: # merge-base gives no output if not ancestor.
-            #     subprocess.run(["git", "merge-base", "--is-ancestor", "fa4b1aa6024f5f3b2d0e0502561b1eaedddd0c78", "HEAD"], stdout = subprocess.PIPE, universal_newlines=True, shell=isWindowsOS).stdout.splitlines()[0]
-            # except IndexError:
-            #     # QtQuickControls2: QQuickPopupPositioner: fix crash on application exit
-            #     subprocess.run(["git", "cherry-pick", "fa4b1aa6024f5f3b2d0e0502561b1eaedddd0c78"], universal_newlines=True, shell=isWindowsOS)
-
-        return True
-
-    else:
-        # This is one of the dependent modules
-        if buildOnHead:
-            revToBuild = "HEAD"
-        else:
-            revToBuild = findRevToBuild(revision, module, branch)
-
-        if not revToBuild:
-            return False
-        else:
-            proc = subprocess.run(["git", "checkout", revToBuild], cwd=os.path.join(builddir, module),
-                                  universal_newlines=True, shell=isWindowsOS)
-
-            actualRevision = subprocess.run(["git", "show", "-s", "--date=short", "--format=%cd"], cwd=os.path.join(builddir, module),
-                                            stdout=subprocess.PIPE, universal_newlines=True, shell=isWindowsOS).stdout.strip()
-            print(f"Checking out {module} from date {actualRevision} with SHA/tag {revToBuild}")
-            return True
-
-
-def cleanup(prepForBuild=False, noRebuild=False):
-    # Clean up our files and start fresh each time, or just uninstall and clean the module to test if we already specified to build against parent.
-
-    def clean(module):
-        subprocess.run(["git", "clean", "-dqfx"], universal_newlines=True, cwd=os.path.join(builddir, module))
-        if os.path.exists(os.path.join(builddir, module, '.git', 'index.lock')):
-            # Force git to accept a new process. If this file exists, it probably means an earlier job was cancelled or crashed while doing git stuff.
-            os.remove(os.path.join(builddir, module, '.git', 'index.lock'))
-            proc = subprocess.run(["git", "reset", "--hard", "origin/head"], universal_newlines=True, cwd=os.path.join(builddir, module))
-            if proc.returncode != 0:
-                print(f"ERROR: The git directory for {module} is corrupted. We'll try to remove it and reclone.")
-                shutil.rmtree(os.path.join(builddir, module), on_rm_error)
-                initRepository(module)
-
-    if noRebuild:
-        subprocess.run([compiler, "uninstall", "-j", f"{int(args.buildCores) * 3 }"], universal_newlines=True, cwd=os.path.join(builddir, args.module))
-        clean(args.module)
-    else:
-        shutil.rmtree(os.path.join(builddir, "Install"), on_rm_error)
-        if prepForBuild:
-            for module in next(os.walk(builddir, followlinks=False))[1]:
-                print(f"cleaning {module}")
-                clean(module)
-
-
-def gatherBaselines():
-    # In a bisect operation, test known good and known bad and set our regression threshold.
-    global goodBaselineScore
-    global badBaselineScore
-    global regressionTargetScore
-    global observedRegressionPercent
-
-    cleanup(True)
-    if not buildChanges(args.knownGoodRev):
-        print("Error!!! Good baseline failed to build! Maybe you had the wrong module to test selected.")
-        exit(1)
-    runBenchmark()
-    goodBaselineScore = parseResults()
-    print(f'Good baseline scored {goodBaselineScore}')
-
-    cleanup(True)
-    if not buildChanges(args.knownBadRev):
-        print("Error!!! Good baseline failed to build! Maybe you had the wrong module to test selected.")
-        exit(1)
-    runBenchmark()
-    badBaselineScore = parseResults()
-    print(f'Bad baseline scored {badBaselineScore}\n')
-
-    print(f'Expected regression is {args.regressionTarget}%.')
-    regressionTargetScore = int(goodBaselineScore - (goodBaselineScore * (args.regressionTarget / 100)))
-    observedRegressionPercent = int((((goodBaselineScore - badBaselineScore) / goodBaselineScore) * 100))
-
-    if badBaselineScore > regressionTargetScore:
-        print("The \'bad\' commit provided did not meet the expected target regression percentage. Are you sure this is correct?")
-        print(f"The provided \'bad\' commit was only {observedRegressionPercent}% lower than the \'good\' commit.")
-        print("Aborting the test. Please adjust the expected regression target or provide a new \'bad\' commit ID.\n")
-        exit(2)  # Reserve exit status of "2" for this use.
-
-    else:
-        severity = abs(args.regressionTarget - observedRegressionPercent)
-        print(f"Observed regression of {observedRegressionPercent} is {severity}% greater than the expected target regression.\nYou may need to run this tool again with a different \'good\' later.\n")
-        print("Continuing... We'll try to find the single commit that caused a regression close to the target.")
-
-
-def gatherRevisionsToTest():
-    # In a bisect, find the full list of commits between known good and known bad.
-    global revisionList
-
-    revisionList = subprocess.run(["git", "rev-list", f"{args.knownGoodRev}..{args.knownBadRev}", f"^{args.knownGoodRev}"], cwd=os.path.join(builddir, args.module),
-                                  stdout=subprocess.PIPE, universal_newlines=True, shell=isWindowsOS).stdout.split('\n')
-
-    if not revisionList:
-        return False
-
-    revisionList.pop()  # Git will return our known bad revision at the top of the list. Remove it since we already test it as part of the baseline process.
-    revisionList.reverse()  # Reverse the list so that the known bad is at the end and known good at the start.
-
-
-def bisectOperator(action):
-    # Bisect logic that acts like Git's bisect command.
-    # Return commit info if we're done bisecting, otherwise, false.
-    global bisectPosition
-    global bisectBadPos
-    global bisectGoodPos
-    global bisectLastStatus
-    global bisectLastDirection
-
-    if action == "start":
-        gatherRevisionsToTest()
-        bisectPosition = int(len(revisionList) / 2)
-        bisectGoodPos = 0
-        bisectBadPos = len(revisionList) - 1
-        return False
-
-    elif action == "good":
-        bisectGoodPos = bisectPosition
-        bisectLastStatus = "good"
-        bisectLastDirection = "plus"
-        if (bisectBadPos - bisectGoodPos) < 2:
-            return revisionList[bisectPosition + 1]
-        bisectPosition = bisectPosition + int((bisectBadPos - bisectGoodPos) / 2)
-        return False
-
-    elif action == "bad":
-        bisectBadPos = bisectPosition
-        bisectLastStatus = "bad"
-        bisectLastDirection = "minus"
-        if (bisectBadPos - bisectGoodPos) < 2:
-            return revisionList[bisectPosition]
-        bisectPosition = bisectPosition - int((bisectBadPos - bisectGoodPos) / 2)
-        return False
-
-    elif action == "skip":
-        if bisectLastStatus == "good":
-            bisectPosition += 1
-            bisectLastDirection = "plus"
-        elif bisectLastStatus == "bad":
-            bisectPosition -= 1
-            bisectLastDirection = "minus"
-        elif bisectLastStatus == "skip":
-            print("received skip...")
-            if bisectLastDirection == "minus":
-                print("is Minus")
-                bisectPosition -= 1
-            elif bisectLastDirection == "plus":
-                print("is Plus")
-                bisectPosition += 1
-            else:
-                print(f"is Other? ...{bisectLastDirection}...")
-                bisectPosition += 1
-        else:
-            print(f"Bisect Last Direction is something else ...{bisectLastDirection}...")
-            bisectPosition += 1
-
-        bisectLastStatus = "skip"
-        if (bisectPosition <= bisectGoodPos and bisectLastDirection == "minus") or (bisectPosition >= bisectBadPos and bisectLastDirection == "plus"):
-            revisions = []
-            for revision in revisionList[bisectGoodPos:bisectBadPos]:
-                revisions.append(revision)
-            return revisions
-
-        return False
-
-    print(f"ERROR: Action was invalid. This should not happen. Ever. Action: {action}")
-
-
-def saveResult(revision, status, score=None):
-    # Add the bisect result to our list of results.
-    global bisectResults
-    bisectResults[0].append({"revision": revision, "status": status, "score": score})
-
-
-def manualBisect():
-    # Do the bisect action. Good and Bad baselines have already been taken.
-    global finalResults
-    print("Beginning bisect process")
-
-    def iterate(branch):
-        # The logic
-        global goodBaselineScore
-        global badBaselineScore
-        global regressionTargetScore
-        global observedRegressionPercent
-
-        print(f"++++++ Testing new index {bisectPosition} with revision {revisionList[bisectPosition]}\nCurrent Known Good index: {bisectGoodPos} with revision {revisionList[bisectGoodPos]}\n\
-Current Known Bad index: {bisectBadPos} with revision {revisionList[bisectBadPos]}\nLast revision was {bisectLastStatus} and we adjusted the index to the {bisectLastDirection}")
-
-        currentRevision = revisionList[bisectPosition]
-        cleanup(True)
-        if not buildChanges(currentRevision, True, branch):
-            # The build either failed or the module revision doesn't point to a COIN integration.
-            print("Skipping bisected checkout.")
-            foundRegression = bisectOperator("skip")
-            if foundRegression:
-                print(f"Skipping this iteration completed the bisect, but returned more than one commit. Please investigate the following commits\n{foundRegression}")
-                bisectResults.append(foundRegression)
-                return foundRegression
-            else:
-                iterate(branch)
-        else:
-            runBenchmark()
-            resultScore = parseResults()
-            if not resultScore:
-                saveResult(currentRevision, "fail")
-                print("Unable to parse a result. Maybe the test crashed with this revision. Skipping revision.")
-                foundRegression = bisectOperator("skip")
-                if foundRegression:
-                    print(f"Skipping this iteration completed the bisect, but returned more than one commit. Please investigate the following commits\n{foundRegression}")
-                    bisectResults.append(foundRegression)
-                    return foundRegression
-                else:
-                    iterate(branch)
-            else:
-                print(f"Iteration scored {resultScore}, {int((((goodBaselineScore - resultScore)/goodBaselineScore)*100))}% off from good baseline score of {goodBaselineScore}")
-                if resultScore > regressionTargetScore:
-                    saveResult(currentRevision, "good", resultScore)
-                    foundRegression = bisectOperator("good")
-                    if foundRegression:
-                        print(f"We found a single commit that caused the regression. Please investigate {foundRegression}")
-                        bisectResults.append([foundRegression])
-                        return [foundRegression]
-                    else:
-                        iterate(branch)
-                else:
-                    saveResult(currentRevision, "bad", resultScore)
-                    foundRegression = bisectOperator("bad")
-                    if foundRegression:
-                        print(f"We found a single commit that caused the regression. Please investigate {foundRegression}")
-                        bisectResults.append([foundRegression])
-                        return [foundRegression]
-                    else:
-                        iterate(branch)
-
-    bisectOperator("start")
-    commit = iterate(args.branch)
-    return commit
-
-
-def testSingleCommit(commit, buildOnHead=False, noRebuild=False):
-    # We're testing one commit by itself
-    cleanup(True, noRebuild)
-    if buildChanges(commit, buildOnHead=buildOnHead, noRebuild=noRebuild):
-        runBenchmark()
-    else:
-        exit(1)
-    resultScore = parseResults()
-    if not resultScore:
-        print("Unable to parse a result. Maybe the test crashed with this revision.")
-        return False
-    print(f"Commit {commit} scored {resultScore}")
-    return resultScore
-
-
-def initRepositories():
-    # Prepare the system for building. Clone required repos
-    if args.wipeWorkspace:
-        shutil.rmtree(os.path.join(builddir, "."), on_rm_error)
-        for dir in os.walk(builddir):
-            print(dir)
-
-    if not os.path.exists(builddir):
-        os.makedirs(builddir)
-
-    initRepository("qt/qt5")
-    initRepository("qt/qtbase")
-    initRepository("qt/qtdeclarative")
-    initRepository("qt/qtquickcontrols")
-    initRepository("qt/qtquickcontrols2")
-    initRepository("qt/qtgraphicaleffects")
-    initRepository("qt/qtsvg")
-    initRepository("qt/qtquick3d")
-    initRepository("qt-labs/qmlbench")
-
-    if (isWindowsOS):
-        initRepository("qt/qttools")
-
-    if args.setupEnv:
-        # RegressionFinder.py was called with --setupEnv and will exit after cloning repos
-        exit(0)
-
-
-def resultWriter():
-    # Dump our test results to a log file
-    if not os.path.exists(os.path.join(basedir, "logs", platform.node(), "results")):
-        os.mkdir(os.path.join(basedir, "logs", platform.node(), "results"))
-
-    data = {}
-    data['jobName'] = args.jobName
-    data['testType'] = testType
-    data['testHost'] = platform.node()
-    data['commits'] = args.testSingleCommit if testType == "singleCommit" else args.testTwoCommit if testType == "twoCommit" else f"{args.knownGoodRev},{args.knownBadRev}"
-    data['branch'] = args.branch
-    data['module'] = args.module
-    if testType == 'bisect':
-        data["bisectResults"] = bisectResults
-        data["baselineResults"] = [goodBaselineScore, badBaselineScore, observedRegressionPercent]
-    data['finalResults'] = finalResults
-
-    with open(os.path.join(basedir, "logs", platform.node(), "results", f"{args.jobName}.json"), mode="w") as outfile:
-        json.dump(data, outfile)
-
-
-def on_exit():
-    # At exit time, whether successful or not, print results to screen and try to write the results file.
-    if not args.setupEnv:
-        if finalResults or bisectResults:
-            resultWriter()
-        print("Complete! Results:")
-        for tested in bisectResults[0]:
-            print(tested)
-        if finalResults:
-            print(finalResults)
-
-
-def buildChanges(revision, bisecting=False, branch="", buildOnHead=False, noRebuild=False):
-    if noRebuild:
-        # Just build the module under test. This is used when regression testing against parent, since all other modules will be the same SHA anyway.
-        if not checkout(revision, args.module, branch, buildOnHead=buildOnHead):
-            return False
-        buildModule(args.module)
-    else:
-        # Build all the modules.
-        if bisecting:
-            if not determineBuildEligibility(revision, args.module, branch):  # Make sure that the revision we're going to test is valid and can be built, so we don't waste time.
-                return False
-        modulesToBuild = ["qtbase", "qtdeclarative", "qtquickcontrols", "qtquickcontrols2", "qtgraphicaleffects", "qtsvg"]
-        if args.module == "qtquick3d":
-            modulesToBuild.append("qtquick3d")
-        for module in modulesToBuild:  # Qtbase must be built first.
-            if not checkout(revision, module, branch, buildOnHead=buildOnHead):
-                return False
-            buildModule(module)
-
-    buildModule("qmlbench")
-    return True
-
-
-if __name__ == "__main__":
-    atexit.register(on_exit)
-    args = parseArgs()
-    prepareEnv()
-    initRepositories()
-    if not validateCommits() or not validateTest():  # Validate inputs!
-        exit(1)
-    if args.testSingleCommit:
-        print("Entering single commit benchmark mode...")
-        testType = "singleCommit"
-        print(f"Building on head? {args.firstBuildOnHead}")
-        result = testSingleCommit(args.testSingleCommit, args.firstBuildOnHead)
-        if not result:
-            exit(1)
-        finalResults = [result]
-    elif args.testTwoCommit:
-        print("Entering two commit compare mode...")
-        testType = "twoCommit"
-        firstCommit = args.testTwoCommit.split(',')[0].strip()
-        secondCommit = args.testTwoCommit.split(',')[1].strip()
-        result1 = testSingleCommit(firstCommit, args.firstBuildOnHead)
-        if not result1:
-            exit(1)
-        result2 = testSingleCommit(secondCommit, args.secondBuildOnHead, noRebuild)
-        if not result2:
-            exit(1)
-        observedRegressionPercent = 0
-        if secondCommit == "PARENT":
-            observedRegressionPercent = int((((result2 - result1) / result2) * 100))
-        else:
-            observedRegressionPercent = int((((result1 - result2) / result1) * 100))
-        print(f"First commit scored {result1}. Second commit scored {result2}, a {observedRegressionPercent}% difference.")
-        finalResults = [result1, result2, observedRegressionPercent]
-    else:
-        testType = "bisect"
-        gatherBaselines()
-        manualBisect()
-
-# sample string
-# python .\regressionFinder.py --branch dev --buildCores 8 --benchmark benchmarks/auto/js/sum10k.qml --regressionTarget 20 --knownGoodRev 85fc49612816dcfc81c9dc265b146b0b90b0f184 --knownBadRev eace041161a03a849d3896af65493b7885cecc04 --moduleToTest qtdeclarative
+import os
+import subprocess
+import argparse
+import platform
+import stat
+import shutil
+import json
+import requests
+from thrift import storagestructs
+import thriftpy.utils
+import thriftpy
+import re
+from time import sleep
+import atexit
+import packaging
+
+args = []
+basedir = os.getcwd()
+builddir = os.path.join(basedir, "QtBuild")
+installdir = os.path.join(builddir, "Install")
+isWindowsOS = (platform.system() == 'Windows')
+compiler = basedir + "\\JOM\\jom.exe" if isWindowsOS else "make"
+exeExt = '.exe' if isWindowsOS else ''
+goodBaselineScore = 0
+badBaselineScore = 0
+regressionTargetScore = 0
+observedRegressionPercent = 0
+refQtbaseRev = ""
+testType = ""
+noRebuild = False
+finalResults = []
+
+# Bisect tracking items
+revisionList = []
+bisectResults = [[], ]
+bisectPosition = 0
+bisectGoodPos = 0
+bisectBadPos = 0
+bisectLastStatus = ""
+bisectLastDirection = ""
+
+
+def on_rm_error(func, path, exc_info):
+    # Handler for files that fail to be removed.
+    # path contains the path of the file that couldn't be removed
+    # let's just assume that it's read-only and unlink it.
+    try:
+        os.chmod(path, stat.S_IWRITE)
+        os.unlink(path)
+    except Exception as e:
+        print("There was an error removing a file from disk. Exception: {0}".format(e))
+
+
+def parseArgs():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--setupEnv", dest="setupEnv", action="store_true", help="Run with --setupEnv to initialize the build environment by cloning needed repos or pulling updates as needed. Exits upon completion.")
+    parser.add_argument("--branch", dest="branch", type=str, help="Branch of Qt such as \'5.12\' or \'dev\'")
+    parser.add_argument("--moduleToTest", dest="module", type=str, help="Module where regression is suspected such as \'qtbase\'.")
+    parser.add_argument("--knownBadRev", dest="knownBadRev", type=str, help="Known bad revision in the module where a regression is expected.")
+    parser.add_argument("--knownGoodRev", dest="knownGoodRev", type=str, help="Known good revision in the module where a regression is expected")
+    parser.add_argument("--regressionTarget", dest="regressionTarget", type=int, help="Expected regression, in percent, to assist in finding the correct regressive commit. Be conservative if unsure.")
+    parser.add_argument("--fuzzyRange", dest="fuzzyRange", type=int, default=2, help="Provide a value to use as a fuzzy range to check the regression against. Using \'--regressionTarget 20 --fuzzyRange 5\' will prefer regressions between 15-25%% as the bad commit")
+    parser.add_argument("--buildCores", dest="buildCores", type=int, default=8, help="Number of build cores to use when building")
+    parser.add_argument("--wipeWorkspace", dest="wipeWorkspace", action="store_true", help="Clear the entire workspace and force re-cloning of all modules")
+    parser.add_argument("--VSDevEnv", dest="VSDevEnv", help="Full path to Visual studio VsDevCmd.bat file for windows build environments. Defaults to default installation for VS 2017 Build Tools", default="C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/Common7/Tools/VsDevCmd.bat")
+    parser.add_argument("--benchmark", dest="benchmark", type=str, help="path to single benchmark to run. Only run one benchmark at a time!")
+    parser.add_argument("--testSingleCommit", dest="testSingleCommit", type=str, help="Set this parameter to a commit ID to benchmark. Setting this parameter overrides bisecting behavior.")
+    parser.add_argument("--testTwoCommit", dest="testTwoCommit", type=str, help="Set this parameter to a comma-separated list of two commit IDs to benchmark and compare. Setting this parameter overrides bisecting behavior.")
+    parser.add_argument("--FirstBuildOnHead", dest="firstBuildOnHead", action="store_true", help="Enable this parameter to build the first commit against branch HEAD instead of searching for a COIN integration.")
+    parser.add_argument("--SecondBuildOnHead", dest="secondBuildOnHead", action="store_true", help="Enable this parameter to build the second commit against branch HEAD instead of searching for a COIN integration.")
+    parser.add_argument("--OpenGLBackend", dest="openGLBackend", type=str, default="desktop", help="Render backend options. Valid options are \'dekstop\', \'angle\', \'software\'")
+    parser.add_argument("--jobName", dest="jobName", type=str, help="unique job name used for writing results file to logs directory. Typically a hash of the job to be run.")
+    parser.add_argument("--environment", dest="environment", type=str, help="Comma separated list of environment variables and values to use for the build and test environment.")
+
+    return parser.parse_args()
+
+
+def initRepository(repo):
+    # Clone any needed repositories. If they already exist, sweep them and pull changes.
+    module = repo[repo.index('/') + 1:]
+    branch = args.branch if module != "qmlbench" else "master"
+    if not branch:
+        branch = "dev"
+
+    print(repo, branch)
+
+    def cloneRepo():
+        subprocess.run(["git", "clone", "-b", branch, f'https://code.qt.io/{repo}.git'], stderr=subprocess.PIPE, universal_newlines=True, cwd=builddir)
+        subprocess.run(["git", "submodule", "update", "--init"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
+        
+
+    # clone modules if necessary, otherwise just pull latest.
+    if not os.path.exists(os.path.join(builddir, module)):
+        cloneRepo()
+    else:
+        subprocess.run(["git", "clean", "-dqfx"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
+        subprocess.run(["git", "reset", "--hard", f"origin/{args.branch}", "--quiet"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
+        subprocess.run(["git", "pull", "origin", args.branch, "--quiet"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
+        subprocess.run(["git", "checkout", branch], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
+
+    if repo == "qt/qt5":
+        # Run the reset and pull again. Sometimes the previous pull leads to merge conflicts.
+        # We don't care, just needed the first pull to get the lastest submodule list
+        # and then reset and pull again to get the latest changes.
+        subprocess.run(["git", "reset", "--hard", f"origin/{args.branch}", "--quiet"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
+        subprocess.run(["git", "pull", "origin", args.branch, "--quiet"], stderr=subprocess.PIPE, universal_newlines=True, cwd=os.path.join(builddir, module))
+        subprocess.run(["git", "submodule", "update", "--init"], stderr=subprocess.PIPE, universal_newlines=True, cwd=builddir)
+
+def validateTest():
+    # Verify the selected test exists
+    global args
+    if not os.path.exists(os.path.join(builddir, 'qmlbench', args.benchmark)):
+        args.benchmark = os.path.join("benchmarks", args.benchmark)
+        if not os.path.exists(os.path.join(builddir, 'qmlbench', args.benchmark)):
+            print(f"Specified benchmark at {os.path.join(builddir, 'qmlbench', args.benchmark)} does not exist. Please check the path and try again.")
+            return False
+    return True
+
+
+def validateCommits():
+    # Verify that the provided commit(s) exist in the Qt repository.
+
+    def revparse(rev):
+        # Rev-parse our given SHA1 and see if Git returns anything. If it pushes anything to stdout, a match was found.
+        proc = subprocess.run(["git", "rev-parse", rev], cwd=os.path.join(builddir, args.module),
+                              stderr=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True, shell=isWindowsOS)
+        if proc.stderr:
+            if lookupUnmergedCommit(rev, args.module):
+                return revparse(rev)
+            else:
+                print(f"Unable to verify commit {rev}. Please check to make sure you have the correct repository and there are no typos.")
+                return False
+        else:
+            print(proc.stdout.strip().split("\n"))
+            if len(proc.stdout.strip().split("\n")) > 1:
+                # Rev-parse will print one matching SHA per line.
+                print(f"Provided commit {rev} matches more than one commit. Try using the full SHA1.")
+                return False
+            else:
+                return proc.stdout.strip()
+
+    global args
+    global noRebuild
+
+    # Do some logic to determine which type of test we're running.
+    if args.knownGoodRev:
+        args.knownGoodRev = args.knownGoodRev.strip()  # Cleanup inputs
+        args.knownBadRev = args.knownBadRev.strip()
+        # Try to see if we have at least one commit between the known good and bad commits. This will fail if the known bad is not an ancestor of the known good.
+        proc = subprocess.run(["git", "rev-list", f"{args.knownGoodRev}..{args.knownBadRev}", f"^{args.knownGoodRev}"], cwd=os.path.join(builddir, args.module),
+                              stderr=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True, shell=isWindowsOS)
+
+        if proc.returncode != 0:
+            print(f"ERROR: Unable to verify selected commits. Please double check your input.\nGit error: {proc.stderr}")
+            return False
+
+        try:
+            revisionList = proc.stdout.split('\n')
+        except Exception as e:
+            print(f"ERROR: No commits between good and bad commits. Nothing to test. Hint: Known bad commit {args.knownBadRev} might be the bad commit.")
+            return False
+
+        if revisionList:
+            revisionList.pop()  # rev-list will return the list with the known bad commit as the head element. Get rid of it permenantly.
+
+        if len(revisionList) < 1:
+            print(f"ERROR: No commits between good and bad commits. Nothing to test. Hint: Known bad commit {args.knownBadRev} might be the bad commit.")
+            return False
+        if args.module == "qtbase":
+            print("Module to test is Qtbase. Verifying that COIN data is available for other modules before proceeding...")
+            if not findRevToBuild(args.knownGoodRev, "qtdeclarative", args.branch) or not findRevToBuild(args.knownBadRev, "qtdeclarative", args.branch):
+                return False
+            else:
+                print("COIN data verified. Continuing.")
+        return True
+
+    elif args.testSingleCommit:
+        args.testSingleCommit = revparse(args.testSingleCommit.strip())
+        if not args.testSingleCommit:
+            return False
+        else:
+            if args.module == "qtbase":
+                print("Module to test is Qtbase. Verifying that COIN data is available for other modules before proceeding...")
+                if not findRevToBuild(args.testSingleCommit, "qtdeclarative", args.branch):
+                    return False
+                else:
+                    print("COIN data verified. Continuing.")
+            return True
+    elif args.testTwoCommit:
+        firstCommit = revparse(args.testTwoCommit.split(',')[0].strip())
+        secondCommit = args.testTwoCommit.split(',')[1].strip()
+        if secondCommit == "PARENT":
+            noRebuild = True
+            secondCommit = findRevToBuild(firstCommit, args.module, "", True)
+            print(f"Second Commit found: {secondCommit}")
+            args.testTwoCommit = f"{secondCommit},{firstCommit}"  # Reverse the order of testing when the user selected the easy regression test option. Expect regression from parent to child, not improvement.
+        else:
+            secondCommit = revparse(secondCommit)
+            args.testTwoCommit = f"{firstCommit},{secondCommit}"
+        if not firstCommit and secondCommit:
+            return False
+        else:
+            if args.module == "qtbase":
+                print("Module to test is Qtbase. Verifying that COIN data is available for other modules before proceeding...")
+                if not findRevToBuild(firstCommit, "qtdeclarative", args.branch) or not findRevToBuild(secondCommit, "qtdeclarative", args.branch):
+                    return False
+                else:
+                    print("COIN data verified. Continuing.")
+            return True
+
+
+def setWindowsEnv():
+    # runs the vsDevCmd file from the visual studio installation
+    vars = subprocess.check_output([args.VSDevEnv, '&&', 'set'])
+
+    # splits the output of the batch file and saves PATH variables from the batch to the local os.environ
+    for var in vars.splitlines():
+        var = var.decode('cp1252')
+        k, _, v = map(str.strip, var.strip().partition('='))
+        if k.startswith('?'):
+            continue
+        os.environ[k] = v
+
+    os.environ["PATH"] += (";" + builddir).replace("/", "\\")
+    os.environ["PATH"] += (";" + basedir + "/flex_bison/").replace("/", "\\")
+    os.environ["QTDIR"] = (builddir).replace("/", "\\")
+
+
+def prepareEnv():
+    # A place to do OS specific actions on startup.
+    if args.environment:
+        for variable in args.environment.split(','):
+            varName, varValue = variable.split('=')
+            os.environ[varName] = varValue
+
+    if isWindowsOS:
+        # set up windows build env
+        print('Setting up windows build environment.')
+        setWindowsEnv()
+
+
+def buildModule(module):
+    # Build the specified Qt module
+    print(f"Preparing to build {module}")
+    os.chdir(os.path.join(builddir, module))
+    if (module == "qtbase"):
+        if isWindowsOS:
+            configurecmd = ["configure.bat", "-prefix", installdir, "-no-pch", "-nomake", "tests", "-nomake", "examples", "-release", "-opensource", "-confirm-license", "-no-warnings-are-errors", "-opengl", "dynamic"]
+
+        else:
+            configurecmd = ["./configure", "-prefix", installdir, "-no-pch", "-developer-build", "-nomake", "tests", "-nomake", "examples", "-release", "-opensource", "-confirm-license", "-no-warnings-are-errors"]
+        print(f"Running Configure for Qtbase")
+        subprocess.run(configurecmd, check=False, shell=isWindowsOS)
+
+    else:
+        # qmake
+        print(f"Running QMake for {module}")
+        subprocess.run([os.path.join(installdir, "bin", f"qmake{exeExt}", ), f"{module}.pro"],
+                       universal_newlines=True, shell=isWindowsOS)
+
+    # build it!
+    print(f"Building {module}...")
+    subprocess.run([compiler, "-j", f"{args.buildCores}"], universal_newlines=True, shell=isWindowsOS)
+    if not module == "qmlbench":
+        installModule(module)
+
+    if isWindowsOS and module == "qmlbench":
+        print(f"Building WinDeployQt")
+        # Also build and run winDeployQt on qmlbench
+        subprocess.run([os.path.join(installdir, "bin", f"qmake{exeExt}", ), "windeployqt.pro"],
+                       universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, "qttools", "src", "windeployqt"))
+        subprocess.run([compiler, "-j", f"{args.buildCores}"],
+                       universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, "qttools", "src", "windeployqt"))
+        subprocess.run([compiler, "install", "-j", f"{args.buildCores}"],
+                       universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, "qttools", "src", "windeployqt"))
+        print(f"Running WinDeployQt against QMLBench")
+        proc = subprocess.run(["windeployqt.exe", "--qmldir", os.path.join(builddir, "qmlbench", "benchmarks"),
+                               os.path.join(builddir, "qmlbench", "src", "release", "qmlbench.exe")],
+                              universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(installdir, "bin"), check=True)
+        print(proc)
+
+    os.chdir(builddir)
+
+
+def installModule(module):
+    # Run make install on the module.
+    print(f"Installing {module}")
+    subprocess.run([compiler, "install", "-j", f"{int(args.buildCores) * 3 }"], universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, module))
+
+
+def runBenchmark():
+    # Actually execute qmlbench
+    benchmarkEnv = os.environ.copy()
+    if isWindowsOS:
+        benchmarkEnv["QT_OPENGL"] = args.openGLBackend
+        print(f"OpenGL Backend set to {benchmarkEnv['QT_OPENGL']}")
+
+    print("Cooling off for 20 seconds before benchmarking...")
+    sleep(20)
+    print(f"Starting Benchmark")
+    with open(os.path.join(builddir, "qmlbench", "results.json"), mode='wb') as results:  # Write output as JSON to results file for later collection.
+        if isWindowsOS:
+            subprocess.run([os.path.join(builddir, "qmlbench", "src", "release", f"qmlbench{exeExt}"), "--json", "--shell", "frame-count", "--repeat", "10", os.path.join(builddir, "qmlbench", args.benchmark)], env=benchmarkEnv, stdout=results, shell=isWindowsOS)
+        else:
+            subprocess.run([os.path.join(builddir, "qmlbench", "src", f"qmlbench{exeExt}"), "--json", "--shell", "frame-count", "--repeat", "10", os.path.join(builddir, "qmlbench", args.benchmark)], stdout=results, shell=isWindowsOS)
+    print(f"Completed Benchmark")
+
+
+def parseResults():
+    # Open the results file and find the "average" result key.
+    print(f"Parsing Benchmark results")
+    resultsJSON = {}  # Typing.Dict()
+    benchmark = args.benchmark.replace('\\', '/')
+    with open(os.path.join(builddir, "qmlbench", "results.json")) as results:
+        resultsJSON = json.loads(results.read())
+
+    for key in resultsJSON:
+        if benchmark in key:
+            return int(resultsJSON[key]['average'])
+
+    return False
+
+
+def findRevToBuild(refRevision, module, branch="", returnParent=False):
+    # Do a whole bunch of stuff to associate the revision we should build of other qt modules.
+
+    def getParentRev(commitJSON):
+        for commit in commitJSON:
+            parentRev = commit["revisions"][refRevision]["commit"]["parents"][0]["commit"]
+            if parentRev:
+                return parentRev
+
+    # Search codereview for the change and return some specific details. Start the saved response from codereview at position 4 to ignore some garbage in the response.
+    commitRaw = requests.get(f"https://codereview.qt-project.org/changes/?q=commit:{refRevision}&o=CURRENT_REVISION&o=CURRENT_COMMIT").text[4:]
+    if not commitRaw:
+        return False
+    try:
+        commitJSON = json.loads(commitRaw)  # Try to parse the JSON
+    except IndexError:
+        print("Unable to find revision on codereview.")
+        saveResult(refRevision, "RevNotFound")
+        return False
+
+    if returnParent:  # Just return the direct parent of refRevision. This would usually be used by the TwoCommit mode for easy regression testing against a new commit's parent.
+        return getParentRev(commitJSON)
+
+    changeID = ""
+    commitDetailsRaw = ""
+
+    # At least we parsed the response into valid JSON. Now, Look through the responses, validate that things are correct, and retreive the comments list.
+    for index, commit in enumerate(commitJSON):
+        if branch not in commit["branch"]:
+            print(f"ERROR: Selected revision of {module} was not commited on the {branch} branch. It was commited to {commit['branch']}. Please correct and resubmit the job.")
+            if index >= (len(commitJSON) - 1):
+                return False
+            else:
+                continue
+        elif args.module not in commit["project"]:
+            print(f"ERROR: Selected revision is not in {args.module}. It is part of {commit['project'].split('/')[1]}. Please correct and resubmit the job.")
+            if index >= (len(commitJSON) - 1):
+                return False
+            else:
+                continue
+        elif branch in commit["branch"] and module == args.module:
+            # This logic branch will return immediately because it's the module we're testing a specific commit.
+            # We'll check it out directly and just wanted to verify it exists.
+            # But if the module to test is qtquick3d, verify that the commit date is after the first
+            # known integration and the branch to test is new enough.
+            if module == "qtquick3d":
+                if subprocess.run(["git", "show", "-s", "--format=%ct", refRevision], cwd=os.path.join(builddir, "qt5", args.module), stdout=subprocess.PIPE).stdout < 1566565560:
+                    return False
+                if (packaging.version.parse(args.branch) if not args.branch == "dev" else False) < packaging.version.parse(5.14):
+                    return False
+            else:
+                return True
+
+        changeID = commit["change_id"]
+        print(f"Found Change ID: {changeID} based on commit {refRevision}")
+
+        # Pull the comments history on the change so we can look for a COIN integration ID.
+        request = requests.get(f"https://codereview.qt-project.org/changes/{changeID}/detail")
+        if request.status_code != 200:
+            continue  # Something's fishy about the commit and codereview doesn't recognize it. Try the next commit in the list of responses from our original query.
+        else:
+            commitDetailsRaw = request.text[4:]  # Gerrit responded favorably. Save the response, but still look at the others in the list from the original query.
+            if commitDetailsRaw:
+                break
+
+    # Try to load the commit's JSON
+    try:
+        commitDetails = json.loads(commitDetailsRaw)
+    except json.decoder.JSONDecodeError:
+        saveResult(refRevision, "RevFoundButBadAPIDetailResponse")
+
+        # Try again with the reference commit's parent. It should have been integrated.
+        parentRev = getParentRev(commitJSON)
+
+        sha = findRevToBuild(parentRev, module, branch)
+        if sha:
+            print("WARN: Using the reference commit's parent to build against. This should only happen if you're testing a change that hasn't been merged yet.")
+            return sha
+        else:
+            print("ERROR: Failed to load commit details from gerrit, and failed to use commit parent as backup. Please report this error.")
+            return False
+
+    integrationId = ""
+
+    # Scan the comments section for COIN's telltale message.
+    for message in commitDetails["messages"]:
+        if "Continuous Integration: Passed" in message["message"]:
+            integrationId = re.search(r"[0-9]{10}", message["message"]).group(0)
+            print(f"Found COIN integration {integrationId}")
+            break  # Found it. No need to look at other comments.
+
+    # Some changes get pushed directly for some reason. They're untestible because we can't ask COIN what to build against unless it has a parent commit that integrated normally.
+    if not(integrationId):
+        print("WARN: The change ID selected did not have a COIN integration.")
+        for message in commitDetails["messages"]:
+            if "Change has been successfully cherry-picked" in message["message"]:
+                print(f"WARN: Change was Cherry-picked. Manual Review suggested.")
+                saveResult(refRevision, "NoIntegrationCherryPicked")
+                break
+        # Try again with the reference commit's parent. It should have been integrated.
+        parentRev = getParentRev(commitJSON)
+        print(f"INFO: Found Parent {parentRev}, trying it instead")
+        sha = findRevToBuild(parentRev, module, branch)
+        if sha:
+            print("WARN: Using the reference commit's parent to build against. This should only happen if you're testing a change that hasn't been merged yet.")
+            return sha
+        else:
+            print("ERROR: Failed to load commit details from gerrit, and failed to use commit parent as backup. Please report this error.")
+            return False
+
+    # Try to pull the saved integration data from coin for the integration we found. It contains the shas of all modules it built with at the time.
+    data = requests.get(f"https://testresults.qt.io/logs/qt/{args.module}/tasks/{integrationId}.thrift_bin").content
+    if b"404 Not Found" in data:
+        print(f"ERROR: Failed to find COIN data for integration ID {integrationId}. Maybe the commit we're trying isn't in the selected module of {args.module}. Trying the parent commit's integration data just in case.")
+
+        # Try again with the reference commit's parent. It should have been integrated.
+        parentRev = getParentRev(commitJSON)
+        print(f"INFO: Found Parent {parentRev}, trying it instead")
+        sha = findRevToBuild(parentRev, module, branch)
+        if sha:
+            print("WARN: Using the reference commit's parent to build against because the target commit had no COIN data.")
+            return sha
+        else:
+            print(f"ERROR: Failed to load parent COIN integration data. This is probably permanently fatal. Commit {refRevision} can't be reliably tested.")
+            return False
+        saveResult(refRevision, "BadCOINData")
+        return False
+    else:
+        try:
+            # Convert the integration's binary thrift_bin file back into a workitem object.
+            workitem = thriftpy.utils.deserialize(storagestructs.Task(), data)
+        except Exception as e:
+            print(f"Failed to extract COIN data for integration ID {integrationId}.")
+            # Try again with the reference commit's parent. It should have been integrated.
+            parentRev = getParentRev(commitJSON)
+            print(f"INFO: Found Parent {parentRev}, trying it instead")
+            sha = findRevToBuild(parentRev, module, branch)
+            if sha:
+                print("WARN: Using the reference commit's parent to build against because the target commit had bad COIN data.")
+                return sha
+            else:
+                print(f"ERROR: Failed to load parent COIN integration data. This is probably permanently fatal. Commit {refRevision} can't be reliably tested.")
+                return False
+
+        # Ask git to identify the SHA1 of our target module to build based on the sha of the change we're testing. This uses the Qt5 supermodule.
+
+        try:
+            proc = subprocess.run(["git", "ls-tree", workitem.product.sha1, module, "-z"], cwd=os.path.join(builddir, "qt5"), stdout=subprocess.PIPE)
+            sha = str(proc.stdout).split(" ")[2].split("\\t")[0]
+            print(f'Found {module} sha1: {sha}')
+            return sha
+        except Exception:
+            # Couldn't find the qt5 product sha from the module to build.
+            print(f"ERROR: Couldn't find COIN product sha {workitem.product.sha1} for module {module} in qt5.")
+            saveResult(refRevision, "BadCOINData")
+            return False
+        #####################################################################################################################
+        ### Try to use the internal COIN as a backup. This code will not be functional outside of COIN's local network!!! ###
+        #####################################################################################################################
+
+        # data = requests.get(f"https://testresults.qt.io/coin/api/item/qt/{args.module}/tasks/{integrationId}.thrift_bin").content
+        # print(data)
+        # try:
+        #     workitem = thriftpy.utils.deserialize(storagestructs.Task(), data)
+        #     print(workitem.product)
+        #     proc = subprocess.run(["git", "ls-tree", workitem.product.sha1, module, "-z"], cwd=os.path.join(builddir, "qt5"), stdout=subprocess.PIPE)
+        #     sha = str(proc.stdout).split(" ")[2].split("\\t")[0]
+        #     print(f'Found {module} sha1: {sha} from internal coin.')
+        #     return sha
+        # except:
+        #     print(f"Also Failed to import COIN data from internal COIN for integration ID {integrationId}. Maybe the commit we're trying isn't in the selected module of {args.module}")
+
+        ############################
+        ### End of internal code ###
+        ############################
+
+
+def determineBuildEligibility(revision, module, branch):
+    # Verify that the SHA provided for our module to test is valid and can be built.
+    return findRevToBuild(revision, module, branch)
+
+
+def lookupUnmergedCommit(revision, module):
+    commitRaw = requests.get(f"https://codereview.qt-project.org/changes/?q=commit:{revision}&o=ALL_REVISIONS&o=CURRENT_COMMIT").text[4:]
+    if not commitRaw:
+        return False
+    try:
+        commitJSON = json.loads(commitRaw)[0]
+    except IndexError:
+        print(f"ERROR: Unable to find revision on codereview: {revision}")
+        return False
+    try:
+        # Get the fetch addresses so we can pull and check out the revision.
+        for item in commitJSON["revisions"]:
+            if revision in item:
+                revision = item
+        url = commitJSON["revisions"][revision]["fetch"]["anonymous http"]["url"]
+        ref = commitJSON["revisions"][revision]["fetch"]["anonymous http"]["ref"]
+    except Exception:
+        print(f"ERROR: Unable to get ref fetch data for commit to test {revision}")
+        return False
+    print(f"Fetching {module} patch ref {ref}")
+    subprocess.run(["git", "fetch", url, ref], universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, module))
+    return True
+
+
+def checkout(revision, module, bisecting=False, branch="", buildOnHead=False):
+    # Run git checkouts
+    print(f"Running Checkout for {module}, using revision {revision} as the reference")
+    print(f"This module should {'' if buildOnHead  else 'not '}be built on head ")
+    if module == args.module:
+        print(f"This is the module to test. Cleaning {module}.")
+        subprocess.run(["git", "clean", "-dqfx"], universal_newlines=True, cwd=os.path.join(builddir, module))
+        print(f"Trying checkout of module to test with revision {revision}")
+        proc = subprocess.run(["git", "checkout", revision], cwd=os.path.join(builddir, module), stderr=subprocess.PIPE,
+                              universal_newlines=True, shell=isWindowsOS)
+        if "reference is not a tree" in proc.stderr:
+            # We must be trying to build a commit that's not yet merged. Let's look it up.
+            lookupUnmergedCommit(revision, module)
+            print(f"Checking out unmerged commit {revision}")
+            proc = subprocess.run(["git", "checkout", "FETCH_HEAD"], cwd=os.path.join(builddir, module), stderr=subprocess.PIPE,
+                                  universal_newlines=True, shell=isWindowsOS)
+            return True
+
+        ########### NOT YET PROPERLY IMPLEMENTED ###########
+        # needPatch = False # TESTING
+        # if needPatch:
+        #     print("Applying patches...")
+        #     print("Applying patch refs/changes/77/249377/2: QtQuickControls2: QQuickPopupPositioner: fix crash on application exit")
+        #     subprocess.run(["git", "fetch", "https://codereview.qt-project.org/qt/qtquickcontrols2", "refs/changes/77/249377/2"], universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, module))
+        #     subprocess.run(["git", "cherry-pick", "FETCH_HEAD", "-X", "theirs"], universal_newlines=True, shell=isWindowsOS, cwd=os.path.join(builddir, module))
+
+            # Example of implementing hard-coded patch:
+            # try: # merge-base gives no output if not ancestor.
+            #     subprocess.run(["git", "merge-base", "--is-ancestor", "fa4b1aa6024f5f3b2d0e0502561b1eaedddd0c78", "HEAD"], stdout = subprocess.PIPE, universal_newlines=True, shell=isWindowsOS).stdout.splitlines()[0]
+            # except IndexError:
+            #     # QtQuickControls2: QQuickPopupPositioner: fix crash on application exit
+            #     subprocess.run(["git", "cherry-pick", "fa4b1aa6024f5f3b2d0e0502561b1eaedddd0c78"], universal_newlines=True, shell=isWindowsOS)
+
+        return True
+
+    else:
+        # This is one of the dependent modules
+        if buildOnHead:
+            revToBuild = "HEAD"
+        else:
+            revToBuild = findRevToBuild(revision, module, branch)
+
+        if not revToBuild:
+            return False
+        else:
+            proc = subprocess.run(["git", "checkout", revToBuild], cwd=os.path.join(builddir, module),
+                                  universal_newlines=True, shell=isWindowsOS)
+
+            actualRevision = subprocess.run(["git", "show", "-s", "--date=short", "--format=%cd"], cwd=os.path.join(builddir, module),
+                                            stdout=subprocess.PIPE, universal_newlines=True, shell=isWindowsOS).stdout.strip()
+            print(f"Checking out {module} from date {actualRevision} with SHA/tag {revToBuild}")
+            return True
+
+
+def cleanup(prepForBuild=False, noRebuild=False):
+    # Clean up our files and start fresh each time, or just uninstall and clean the module to test if we already specified to build against parent.
+
+    def clean(module):
+        subprocess.run(["git", "clean", "-dqfx"], universal_newlines=True, cwd=os.path.join(builddir, module))
+        if os.path.exists(os.path.join(builddir, module, '.git', 'index.lock')):
+            # Force git to accept a new process. If this file exists, it probably means an earlier job was cancelled or crashed while doing git stuff.
+            os.remove(os.path.join(builddir, module, '.git', 'index.lock'))
+            proc = subprocess.run(["git", "reset", "--hard", "origin/head"], universal_newlines=True, cwd=os.path.join(builddir, module))
+            if proc.returncode != 0:
+                print(f"ERROR: The git directory for {module} is corrupted. We'll try to remove it and reclone.")
+                shutil.rmtree(os.path.join(builddir, module), on_rm_error)
+                initRepository(module)
+
+    if noRebuild:
+        subprocess.run([compiler, "uninstall", "-j", f"{int(args.buildCores) * 3 }"], universal_newlines=True, cwd=os.path.join(builddir, args.module))
+        clean(args.module)
+    else:
+        shutil.rmtree(os.path.join(builddir, "Install"), on_rm_error)
+        if prepForBuild:
+            for module in next(os.walk(builddir, followlinks=False))[1]:
+                print(f"cleaning {module}")
+                clean(module)
+
+
+def gatherBaselines():
+    # In a bisect operation, test known good and known bad and set our regression threshold.
+    global goodBaselineScore
+    global badBaselineScore
+    global regressionTargetScore
+    global observedRegressionPercent
+
+    cleanup(True)
+    if not buildChanges(args.knownGoodRev):
+        print("Error!!! Good baseline failed to build! Maybe you had the wrong module to test selected.")
+        exit(1)
+    runBenchmark()
+    goodBaselineScore = parseResults()
+    print(f'Good baseline scored {goodBaselineScore}')
+
+    cleanup(True)
+    if not buildChanges(args.knownBadRev):
+        print("Error!!! Good baseline failed to build! Maybe you had the wrong module to test selected.")
+        exit(1)
+    runBenchmark()
+    badBaselineScore = parseResults()
+    print(f'Bad baseline scored {badBaselineScore}\n')
+
+    print(f'Expected regression is {args.regressionTarget}%.')
+    regressionTargetScore = int(goodBaselineScore - (goodBaselineScore * (args.regressionTarget / 100)))
+    observedRegressionPercent = int((((goodBaselineScore - badBaselineScore) / goodBaselineScore) * 100))
+
+    if badBaselineScore > regressionTargetScore:
+        print("The \'bad\' commit provided did not meet the expected target regression percentage. Are you sure this is correct?")
+        print(f"The provided \'bad\' commit was only {observedRegressionPercent}% lower than the \'good\' commit.")
+        print("Aborting the test. Please adjust the expected regression target or provide a new \'bad\' commit ID.\n")
+        exit(2)  # Reserve exit status of "2" for this use.
+
+    else:
+        severity = abs(args.regressionTarget - observedRegressionPercent)
+        print(f"Observed regression of {observedRegressionPercent} is {severity}% greater than the expected target regression.\nYou may need to run this tool again with a different \'good\' later.\n")
+        print("Continuing... We'll try to find the single commit that caused a regression close to the target.")
+
+
+def gatherRevisionsToTest():
+    # In a bisect, find the full list of commits between known good and known bad.
+    global revisionList
+
+    revisionList = subprocess.run(["git", "rev-list", f"{args.knownGoodRev}..{args.knownBadRev}", f"^{args.knownGoodRev}"], cwd=os.path.join(builddir, args.module),
+                                  stdout=subprocess.PIPE, universal_newlines=True, shell=isWindowsOS).stdout.split('\n')
+
+    if not revisionList:
+        return False
+
+    revisionList.pop()  # Git will return our known bad revision at the top of the list. Remove it since we already test it as part of the baseline process.
+    revisionList.reverse()  # Reverse the list so that the known bad is at the end and known good at the start.
+
+
+def bisectOperator(action):
+    # Bisect logic that acts like Git's bisect command.
+    # Return commit info if we're done bisecting, otherwise, false.
+    global bisectPosition
+    global bisectBadPos
+    global bisectGoodPos
+    global bisectLastStatus
+    global bisectLastDirection
+
+    if action == "start":
+        gatherRevisionsToTest()
+        bisectPosition = int(len(revisionList) / 2)
+        bisectGoodPos = 0
+        bisectBadPos = len(revisionList) - 1
+        return False
+
+    elif action == "good":
+        bisectGoodPos = bisectPosition
+        bisectLastStatus = "good"
+        bisectLastDirection = "plus"
+        if (bisectBadPos - bisectGoodPos) < 2:
+            return revisionList[bisectPosition + 1]
+        bisectPosition = bisectPosition + int((bisectBadPos - bisectGoodPos) / 2)
+        return False
+
+    elif action == "bad":
+        bisectBadPos = bisectPosition
+        bisectLastStatus = "bad"
+        bisectLastDirection = "minus"
+        if (bisectBadPos - bisectGoodPos) < 2:
+            return revisionList[bisectPosition]
+        bisectPosition = bisectPosition - int((bisectBadPos - bisectGoodPos) / 2)
+        return False
+
+    elif action == "skip":
+        if bisectLastStatus == "good":
+            bisectPosition += 1
+            bisectLastDirection = "plus"
+        elif bisectLastStatus == "bad":
+            bisectPosition -= 1
+            bisectLastDirection = "minus"
+        elif bisectLastStatus == "skip":
+            print("received skip...")
+            if bisectLastDirection == "minus":
+                print("is Minus")
+                bisectPosition -= 1
+            elif bisectLastDirection == "plus":
+                print("is Plus")
+                bisectPosition += 1
+            else:
+                print(f"is Other? ...{bisectLastDirection}...")
+                bisectPosition += 1
+        else:
+            print(f"Bisect Last Direction is something else ...{bisectLastDirection}...")
+            bisectPosition += 1
+
+        bisectLastStatus = "skip"
+        if (bisectPosition <= bisectGoodPos and bisectLastDirection == "minus") or (bisectPosition >= bisectBadPos and bisectLastDirection == "plus"):
+            revisions = []
+            for revision in revisionList[bisectGoodPos:bisectBadPos]:
+                revisions.append(revision)
+            return revisions
+
+        return False
+
+    print(f"ERROR: Action was invalid. This should not happen. Ever. Action: {action}")
+
+
+def saveResult(revision, status, score=None):
+    # Add the bisect result to our list of results.
+    global bisectResults
+    bisectResults[0].append({"revision": revision, "status": status, "score": score})
+
+
+def manualBisect():
+    # Do the bisect action. Good and Bad baselines have already been taken.
+    global finalResults
+    print("Beginning bisect process")
+
+    def iterate(branch):
+        # The logic
+        global goodBaselineScore
+        global badBaselineScore
+        global regressionTargetScore
+        global observedRegressionPercent
+
+        print(f"++++++ Testing new index {bisectPosition} with revision {revisionList[bisectPosition]}\nCurrent Known Good index: {bisectGoodPos} with revision {revisionList[bisectGoodPos]}\n\
+Current Known Bad index: {bisectBadPos} with revision {revisionList[bisectBadPos]}\nLast revision was {bisectLastStatus} and we adjusted the index to the {bisectLastDirection}")
+
+        currentRevision = revisionList[bisectPosition]
+        cleanup(True)
+        if not buildChanges(currentRevision, True, branch):
+            # The build either failed or the module revision doesn't point to a COIN integration.
+            print("Skipping bisected checkout.")
+            foundRegression = bisectOperator("skip")
+            if foundRegression:
+                print(f"Skipping this iteration completed the bisect, but returned more than one commit. Please investigate the following commits\n{foundRegression}")
+                bisectResults.append(foundRegression)
+                return foundRegression
+            else:
+                iterate(branch)
+        else:
+            runBenchmark()
+            resultScore = parseResults()
+            if not resultScore:
+                saveResult(currentRevision, "fail")
+                print("Unable to parse a result. Maybe the test crashed with this revision. Skipping revision.")
+                foundRegression = bisectOperator("skip")
+                if foundRegression:
+                    print(f"Skipping this iteration completed the bisect, but returned more than one commit. Please investigate the following commits\n{foundRegression}")
+                    bisectResults.append(foundRegression)
+                    return foundRegression
+                else:
+                    iterate(branch)
+            else:
+                print(f"Iteration scored {resultScore}, {int((((goodBaselineScore - resultScore)/goodBaselineScore)*100))}% off from good baseline score of {goodBaselineScore}")
+                if resultScore > regressionTargetScore:
+                    saveResult(currentRevision, "good", resultScore)
+                    foundRegression = bisectOperator("good")
+                    if foundRegression:
+                        print(f"We found a single commit that caused the regression. Please investigate {foundRegression}")
+                        bisectResults.append([foundRegression])
+                        return [foundRegression]
+                    else:
+                        iterate(branch)
+                else:
+                    saveResult(currentRevision, "bad", resultScore)
+                    foundRegression = bisectOperator("bad")
+                    if foundRegression:
+                        print(f"We found a single commit that caused the regression. Please investigate {foundRegression}")
+                        bisectResults.append([foundRegression])
+                        return [foundRegression]
+                    else:
+                        iterate(branch)
+
+    bisectOperator("start")
+    commit = iterate(args.branch)
+    return commit
+
+
+def testSingleCommit(commit, buildOnHead=False, noRebuild=False):
+    # We're testing one commit by itself
+    cleanup(True, noRebuild)
+    if buildChanges(commit, buildOnHead=buildOnHead, noRebuild=noRebuild):
+        runBenchmark()
+    else:
+        exit(1)
+    resultScore = parseResults()
+    if not resultScore:
+        print("Unable to parse a result. Maybe the test crashed with this revision.")
+        return False
+    print(f"Commit {commit} scored {resultScore}")
+    return resultScore
+
+
+def initRepositories():
+    # Prepare the system for building. Clone required repos
+    if args.wipeWorkspace:
+        shutil.rmtree(os.path.join(builddir, "."), on_rm_error)
+        for dir in os.walk(builddir):
+            print(dir)
+
+    if not os.path.exists(builddir):
+        os.makedirs(builddir)
+
+    initRepository("qt/qt5")
+    initRepository("qt/qtbase")
+    initRepository("qt/qtdeclarative")
+    initRepository("qt/qtquickcontrols")
+    initRepository("qt/qtquickcontrols2")
+    initRepository("qt/qtgraphicaleffects")
+    initRepository("qt/qtsvg")
+    initRepository("qt/qtquick3d")
+    initRepository("qt-labs/qmlbench")
+
+    if (isWindowsOS):
+        initRepository("qt/qttools")
+
+    if args.setupEnv:
+        # RegressionFinder.py was called with --setupEnv and will exit after cloning repos
+        exit(0)
+
+
+def resultWriter():
+    # Dump our test results to a log file
+    if not os.path.exists(os.path.join(basedir, "logs", platform.node(), "results")):
+        os.mkdir(os.path.join(basedir, "logs", platform.node(), "results"))
+
+    data = {}
+    data['jobName'] = args.jobName
+    data['testType'] = testType
+    data['testHost'] = platform.node()
+    data['commits'] = args.testSingleCommit if testType == "singleCommit" else args.testTwoCommit if testType == "twoCommit" else f"{args.knownGoodRev},{args.knownBadRev}"
+    data['branch'] = args.branch
+    data['module'] = args.module
+    if testType == 'bisect':
+        data["bisectResults"] = bisectResults
+        data["baselineResults"] = [goodBaselineScore, badBaselineScore, observedRegressionPercent]
+    data['finalResults'] = finalResults
+
+    with open(os.path.join(basedir, "logs", platform.node(), "results", f"{args.jobName}.json"), mode="w") as outfile:
+        json.dump(data, outfile)
+
+
+def on_exit():
+    # At exit time, whether successful or not, print results to screen and try to write the results file.
+    if not args.setupEnv:
+        if finalResults or bisectResults:
+            resultWriter()
+        print("Complete! Results:")
+        for tested in bisectResults[0]:
+            print(tested)
+        if finalResults:
+            print(finalResults)
+
+
+def buildChanges(revision, bisecting=False, branch="", buildOnHead=False, noRebuild=False):
+    if noRebuild:
+        # Just build the module under test. This is used when regression testing against parent, since all other modules will be the same SHA anyway.
+        if not checkout(revision, args.module, branch, buildOnHead=buildOnHead):
+            return False
+        buildModule(args.module)
+    else:
+        # Build all the modules.
+        if bisecting:
+            if not determineBuildEligibility(revision, args.module, branch):  # Make sure that the revision we're going to test is valid and can be built, so we don't waste time.
+                return False
+        modulesToBuild = ["qtbase", "qtdeclarative", "qtquickcontrols", "qtquickcontrols2", "qtgraphicaleffects", "qtsvg"]
+        if args.module == "qtquick3d":
+            modulesToBuild.append("qtquick3d")
+        for module in modulesToBuild:  # Qtbase must be built first.
+            if not checkout(revision, module, branch, buildOnHead=buildOnHead):
+                return False
+            buildModule(module)
+
+    buildModule("qmlbench")
+    return True
+
+
+if __name__ == "__main__":
+    atexit.register(on_exit)
+    args = parseArgs()
+    prepareEnv()
+    initRepositories()
+    if not validateCommits() or not validateTest():  # Validate inputs!
+        exit(1)
+    if args.testSingleCommit:
+        print("Entering single commit benchmark mode...")
+        testType = "singleCommit"
+        print(f"Building on head? {args.firstBuildOnHead}")
+        result = testSingleCommit(args.testSingleCommit, args.firstBuildOnHead)
+        if not result:
+            exit(1)
+        finalResults = [result]
+    elif args.testTwoCommit:
+        print("Entering two commit compare mode...")
+        testType = "twoCommit"
+        firstCommit = args.testTwoCommit.split(',')[0].strip()
+        secondCommit = args.testTwoCommit.split(',')[1].strip()
+        result1 = testSingleCommit(firstCommit, args.firstBuildOnHead)
+        if not result1:
+            exit(1)
+        result2 = testSingleCommit(secondCommit, args.secondBuildOnHead, noRebuild)
+        if not result2:
+            exit(1)
+        observedRegressionPercent = 0
+        if secondCommit == "PARENT":
+            observedRegressionPercent = int((((result2 - result1) / result2) * 100))
+        else:
+            observedRegressionPercent = int((((result1 - result2) / result1) * 100))
+        print(f"First commit scored {result1}. Second commit scored {result2}, a {observedRegressionPercent}% difference.")
+        finalResults = [result1, result2, observedRegressionPercent]
+    else:
+        testType = "bisect"
+        gatherBaselines()
+        manualBisect()
+
+# sample string
+# python .\regressionFinder.py --branch dev --buildCores 8 --benchmark benchmarks/auto/js/sum10k.qml --regressionTarget 20 --knownGoodRev 85fc49612816dcfc81c9dc265b146b0b90b0f184 --knownBadRev eace041161a03a849d3896af65493b7885cecc04 --moduleToTest qtdeclarative
diff --git a/server.js b/server.js
index cb4cd643e843b381f7dfc4572cc596ce47a5f091..057e451ac29867240ce30594c86ea432a9d0a7a6 100644
--- a/server.js
+++ b/server.js
@@ -1,2028 +1,2035 @@
-'use strict';
-const { spawn } = require('child_process');
-const express = require('express');
-const fileUpload = require('express-fileupload');
-var events = require('events');
-const moment = require('moment');
-var momentDurationFormatSetup = require('moment-duration-format');
-momentDurationFormatSetup(moment);
-const os = require('os');
-const myname = os.hostname();
-var isPrimaryHost;
-let path = os.platform() === 'win32' ? 'c:' : '/';
-var pathmodule = require('path');
-var slaves = [];
-var bodyParser = require('body-parser');
-var ip = require('ip');
-const disk = require('diskusage');
-const axios = require('axios');
-const prettyBytes = require('pretty-bytes');
-const JSum = require('jsum');
-var fs = require('fs');
-const nodemailer = require('nodemailer');
-const readLastLines = require('read-last-lines');
-var validator = require('email-validator');
-const serveIndex = require('serve-index');
-var updateLockout = false;
-const tempWrite = require('temp-write');
-const Buffer = require('buffer/').Buffer
-const del = require('del');
-
-// Declarations
-var hosts = [];
-var jobQueue = [];
-var runningJob = {};
-var remoteRunningJobs = [];
-var remoteQueuedJobs = [];
-var completedJobs = [];
-var updateQueue = [];
-var recentErrors = [];
-const notifyJob = new events.EventEmitter();
-var locked = false;
-var nextErrorClearing;
-var standalone = false;
-var nolocal = false;
-var expectedDurations = {bisect: 14400000, singleCommit: 1350000, twoCommit: 2750000};
-var webPort = 8080;
-var buildCores;
-var vsDevEnv;
-var testNames = [];
-
-
-// Initialize some things and check our startup parameters
-
-String.prototype.insert = function(index, string) { // Custom prototype to inject a string into another at index
-  if (index > 0)
-    return this.substring(0, index) + string + this.substring(index, this.length);
-  else
-    return string + this;
-};
-
-if (process.argv[2] == 'standalone'){ // Startup in standalone mode
-  if (process.argv[3] == 'nolocal'){
-    console.log("ERROR: \'standalone\' parameter cannot be combined with nolocal! There will be nowhere to run jobs!");
-  }
-  console.log('Starting in standalone mode. Ignoring all slave hosts passed in parameters.');
-  isPrimaryHost = true;
-  standalone = true;
-} else if (process.argv[2] == 'nolocal'){ // Startup with a local worker
-  if (process.argv[3] == undefined || process.argv[3] == 'standalone'){ // verify we have at least one slave worker
-    console.log("ERROR: Use of \'nolocal\' parameter requires addition of at least one slave IP in parameter list, and CANNOT be combined with \'standalone\'");
-    process.exit(0);
-  }
-  console.log('Starting in nolocal mode. Will not offer self as a slave host.');
-
-  isPrimaryHost = true;
-  nolocal = true;
-
-  process.argv.forEach((val, index) => {
-    if (index > 2) {
-      slaves.push(val);
-      console.log(`Got slave ${index - 2}: ${val}`);
-    }
-  });
-} else if (process.argv[2] != undefined){ // Assume all our arguments are slave worker IPs.
-  isPrimaryHost = true;
-
-  process.argv.forEach((val, index) => {
-    if (index > 1) {
-      slaves.push(val);
-      console.log(`Got slave ${index - 1}: ${val}`);
-    }
-  });
-}
-
-console.log(`I am ${isPrimaryHost ? '' : 'not '}a primary host, and I ${nolocal ? 'will not' : 'will'} run local jobs.`);
-
-if (!nolocal){
-  slaves.push(ip.address()); // List self as a slave even if we're running as master
-}
-
-// Pull optional environment variables
-
-if (process.env.HTTP_PORT){
-  webPort = process.env.HTTP_PORT;
-}
-if (process.env.BUILD_CORES){
-  buildCores = process.env.BUILD_CORES;
-}
-if (process.env.VS_DEV_ENV){
-  vsDevEnv = process.env.VS_DEV_ENV;
-}
-
-notifyJob.on('updateRemote', function(recursive){
-  // This signal should be emitted when an update from a remote worker is received.
-  if (!updateLockout){
-    updateLockout = true; // lock the work arrays from being modified by anyone else
-    updateJob(updateQueue.pop()); // process next in line
-  } else if (recursive){
-    // If recursive was passed, we got here directly from processing a previous update. In this case, the lock is not removed in order to prevent a race condition in updating.
-    updateJob(updateQueue.pop());
-  }
-});
-
-function addUpdateToQueue(req, res){
-  // Received an HTTP request to update a job. Queue the request for processing.
-  console.log(`Received Job update "${req.body.type}" for job ${req.body.job.jobHash}`);
-  updateQueue.push({job: req.body.job, type: req.body.type});
-  notifyJob.emit('updateRemote'); // try to process the request immediately.
-  if (res){
-    res.sendStatus(200); // Respond to the caller that we received the request.
-  }
-}
-
-function updateJob(job){
-  // Search our arrays based on the type of request received. When the job is found, splice it out of it's array and place the updated job we got where it needs to go.
-
-  var tempjob;
-
-  if (job.type == 'running'){
-    var index = remoteQueuedJobs.findIndex(x => x.jobHash == job.job.jobHash);
-    if (index >= 0){
-      tempjob = remoteQueuedJobs.splice(index, 1);
-      console.log(`Moved job ${tempjob[0].jobHash} from remote queued to remote running.`)
-    }
-    remoteRunningJobs.push(job.job);
-
-  } else if (job.type == 'finished' || job.type == 'cancelled'){
-    var index = remoteRunningJobs.findIndex(x => x.jobHash == job.job.jobHash);
-    if (index >= 0){
-      tempjob = remoteRunningJobs.splice(index, 1);
-      console.log(`Moved job ${tempjob[0].jobHash} from remote running to Finished.`);
-    } else {
-      index = remoteQueuedJobs.findIndex(x => x.jobHash == job.job.jobHash);
-      if (index >= 0){
-        tempjob = remoteQueuedJobs.splice(index, 1);
-      } else {
-        console.log(`Received notification that job ${job.job.jobHash} was cancelled or finished, but we can't find it. This probably means that we missed an earlier update...`);
-      }
-      console.log(`Moved job ${tempjob[0].jobHash} from remote queued to Finished.`);
-    }
-    
-    completedJobs.push(job.job);
-
-  } else if (job.type == 'cancelling'){
-    var index = remoteRunningJobs.findIndex(x => x.jobHash == job.job.jobHash);
-    if (index >= 0){
-      remoteRunningJobs[index].status = 'cancelling';
-      console.log(`Set remote running job ${remoteRunningJobs[index].jobHash} to cancelling.`);
-    } else {
-      index = remoteQueuedJobs.findIndex(x => x.jobHash == job.job.jobHash);
-      if (index >= 0){
-        remoteQueuedJobs[index].status = 'cancelling';
-        console.log(`Set remote queued job ${remoteQueuedJobs[index].jobHash} to cancelling.`);
-      } else {
-        console.log(`Received notification that job ${job.job.jobHash} was cancelled, but we can't find it. This probably means that we missed an earlier update...`);
-      }
-    }
-
-  } else {
-    console.log(`WARN: Received unknown update type ${job.type} for job ${job.job.jobHash}\nIf this job exists, it will probably be stuck now.`);
-    recentErrors.push({
-      error: `WARN: Received unknown update type ${job.type} for job ${job.job.jobHash}\nIf this job exists, it will probably be stuck now.`,
-      job: job.job,
-    });
-  }
-
-  if (updateQueue.length > 0){
-    notifyJob.emit('updateRemote', true); // Process the next update in queue if there is one, and keep the updater locked out in the meantime.
-  } else {
-    updateLockout = false; // End the lockout
-  }
-}
-
-notifyJob.on('updateRemoteJob', function(job, action){
-  // A remote host will emit this signal when a job's state changes. Send the update type and a copy of the job to the remote master.
-  console.log(`Updating master at ${job.master_host} with job status ${job.status} for ${job.jobHash}`);
-  axios.post(`http://${job.master_host}:${webPort}/pushJobUpdate`, {job: job, type: action}).then({
-  }).catch(error => {
-    // This is only a WARN because the job will continue processing locally, even if the master host isn't aware of it.
-    console.log('WARN: Unable to post job update to master host. It may be offline.');
-  });
-});
-
-notifyJob.on('JobDone', function(){
-  // A host will emit this signal when an active process completes, regardless of success
-
-  delete runningJob['processHandle']; // Clean up and delete the child_process handle from the job. Attempting to convert the process handle to a string when updating the master host would result in an error, so it has to go anyway.
-  const job = runningJob; // make a local shallow copy
-  console.log(`finished job ${job.jobHash}`);
-  runningJob = {}; // empty the currently running local job slot.
-  completedJobs.push(job); // Save the job to the finished list
-  locked = false; // unlock the job queue
-
-  if (job.remote_job){
-    notifyJob.emit('updateRemoteJob', job, 'finished'); // Notify scheduler
-    if (jobQueue.length != 0){
-      setTimeout(function() { notifyJob.emit('newLocalJob'); }, 3000); // Delay starting the next job for a few seconds to prevent race conditions and make a better UI experience.
-    }
-  } else {
-    if (jobQueue.length != 0){
-      notifyJob.emit('newLocalJob'); // If we're not acting as a remote host, we can just execute immediately.
-    }
-  }
-  if (validator.validate(job.owner)){
-    // Send an email to the job owner if they provided a valid email address
-
-    const readyEmitter = new events.EventEmitter();
-    var errorCaseLogTail = '';
-
-    function jobFormatter(job, format){
-      // Do some formatting magic
-
-      function resultText(result){
-        if(result < 0 ){
-          return `Observed improvement: ${Math.abs(result)}%`;
-        } else if (result > 0){
-          return `Observed regression: ${result}%`;
-        } else {
-          return "-- No change in score --";
-        }
-      }
-
-      const commitString = job.commit ? job.commit : job.firstCommit ? job.firstCommit.slice(0, 10) + '..' + job.secondCommit.slice(0, 10) : job.good_commit.slice(0, 10) + '..' + job.bad_commit.slice(0, 10);
-      const backendString = job.platform == 'win32' && job.openGLBackend ? ` - Open GL Backend: ${job.openGLBackend}` : '';
-      var resultString = '';
-      if (job.resultJSON && job.status != 'failed' && job.status != 'cancelled'){
-        if (job.testType == 'singleCommit') {
-
-          resultString = `Benchmark result: ${job.resultJSON.finalResults[0]}`;
-        } else if (job.testType == 'twoCommit'){
-          resultString = `Results<br>
-                                    First commit result (${job.firstCommit.slice(0, 10)}): ${job.resultJSON.finalResults[0]}<br>
-                                    Second commit result (${job.secondCommit.slice(0, 10)}): ${job.resultJSON.finalResults[1]}<br>
-                                    ${resultText(job.resultJSON.finalResults[2])}%`;
-        } else if (job.testType == 'bisect'){
-          const bisectResult = job.resultJSON.bisectResults[1].length > 1 ?
-            `Bisect returned multiple commits. This is probably due to skipped commits.<br>
-                                        Please investigate the following commits:<br>
-                                        ${job.resultJSON.bisectResults[1].slice(0,).join('<br>')}<br><br>
-                                        `
-            : `Bisect returned a single commit: ${job.resultJSON.bisectResults[1]}<br><br>`;
-
-          resultString = `Results<br>
-                                    Good commit result (${job.good_commit.slice(0, 10)}): ${job.resultJSON.baselineResults[0]}<br>
-                                    Bad commit result (${job.bad_commit.slice(0, 10)}): ${job.resultJSON.baselineResults[1]}<br>
-                                    ${resultText(job.resultJSON.baselineResults[2])}%<br>
-                                    ${bisectResult}
-                                    Tested Commits raw results:<br>
-                                    ${JSON.stringify(job.resultJSON.bisectResults[0], null, '&nbsp;').replace(/\n/g, '<br>')}<br><br>
-                                    `;
-        }
-      }
-
-      var testString = '';
-      if (job.test_name){
-        testString = `Benchmark: ${job.test_name}<br>`;
-      } else if (job.custom_benchmark_file){
-          testString = `Custom benchmark: ${job.custom_benchmark_file.name}<br>`;
-      }
-
-      if (format == 'plaintext'){
-        // Provide a plaintext email
-        return `Your QMLBench ${job.testType} job for '${job.title}' with jobHash ${job.jobHash.slice(0, 10)} on host ${job.test_hosts[0]}:\n Date: ${moment(job.timestamp).format('ddd DD MMM YYYY - HH:MM')}\nStatus: ${job.status}\n
-                Tested commit(s): ${commitString}\n\n${testString.replace(/<br>/g, '\n')}\nResults:\n${resultString.replace(/<br>/g, '\n')}${errorCaseLogTail}`;
-
-      } else {
-        // Provide a richtext/html email
-        return `
-                <style>
-                .failed { color: darkred; }
-                .finished { color: darkgreen; }
-                .cancelled { color: darkorange; }
-                </style>
-
-                <body>
-                <div class="${job.status}">
-                Your QMLBench <b>${job.testType}</b> job for ${job.title} with jobHash <font color="#454545"><b>${job.jobHash.slice(0, 10)}</b></font> on host ${job.test_hosts[0]} from ${moment(job.timestamp).format('ddd DD MMM YYYY - HH:MM')} finished with status: ${job.status}<br>
-                Tested commit(s) ${commitString}<br>
-                ${testString}<br>
-                <font color="#454545">
-                ${resultString}${backendString}
-                ${errorCaseLogTail.replace(/\n/g, '<br>')}
-                </font>
-                </div>
-                </body>
-                `;
-      }
-    }
-
-
-    readyEmitter.on('ready', function(){
-      // Only send the email once we have everything all put together.
-
-      // create reusable transporter object using the default SMTP transport
-      let transporter = nodemailer.createTransport({
-        host: 'smtp.intra.qt.io',
-        port: 25,
-        secure: false, // true for 465, false for other ports
-        tls: {
-          rejectUnauthorized: false,
-        },
-      });
-
-      // setup email data with unicode symbols
-      let mailOptions = {
-        from: '"QMLBench Bot" <qmlbenchbot@intra.qt.io>', // sender address
-        to: job.owner, // list of receivers
-        subject: `${job.status}: QMLBench ${job.testType} Job ${job.jobHash.slice(0, 10)}`, // Subject line
-        text: jobFormatter(job, 'plaintext'), // plain text body
-        html: jobFormatter(job, 'html'), // html body
-      };
-
-      // send mail with defined transport object
-      transporter.sendMail(mailOptions).catch(console.error);
-    });
-
-    if (job.status == 'failed'){
-      // Try to read the logfile and pass the last bit of that along to the job owner.
-      readLastLines.read(`logs/${job.test_hosts[0]}/${job.jobHash}.txt`, 20)
-        .then((lines) => {
-          errorCaseLogTail = 'Last 20 lines of the build/test log:\n';
-          errorCaseLogTail += lines;
-          readyEmitter.emit('ready'); // Okay, that's it! Send the email.
-        });
-    } else {
-      readyEmitter.emit('ready'); // Nothing special to do. Send the email.
-    }
-
-  }
-});
-
-function runTest(test) {
-  // Run the given test on this host
-
-  console.log('Now executing test with params:');
-  console.log(test);
-  console.log(`Job ${test.jobHash.slice(0, 10)} is expected to take ${moment.duration(test.expectedDuration).humanize()}`);
-
-  // Set arguments for regressionfinder.py to really do the build and run the test
-  var args = [];
-  var re = /\\/gi; //pass forward slashes, even on windows to avoid broken escaping.
-  if (test.testType == 'bisect'){
-    args = [].concat(['./regressionFinder.py', '--branch', test.branch, '--moduleToTest', test.module, '--knownBadRev', test.bad_commit,
-      '--knownGoodRev', test.good_commit, '--benchmark', test.test_name ? test.test_name : test.custom_benchmark_file.tempFilePath.replace(re, "/"), '--regressionTarget', test.expected_regression, '--buildCores', buildCores || os.cpus().length]);
-  } else if (test.testType == 'singleCommit'){
-    args = [].concat(['./regressionFinder.py', '--branch', test.branch, '--moduleToTest', test.module, '--testSingleCommit', test.commit,
-      '--benchmark', test.test_name ? test.test_name : test.custom_benchmark_file.tempFilePath.replace(re, "/"), '--buildCores', buildCores || os.cpus().length]);
-    if (test.firstCommitBuildOnHead){
-      args.push('--FirstBuildOnHead');
-    }
-  } else if (test.testType == 'twoCommit'){
-    args = [].concat(['./regressionFinder.py', '--branch', test.branch, '--moduleToTest', test.module,
-      '--testTwoCommit', test.firstCommit + ',' + test.secondCommit,
-      '--benchmark', test.test_name ? test.test_name : test.custom_benchmark_file.tempFilePath.replace(re, "/"), '--buildCores', buildCores || os.cpus().length]);
-    if (test.firstCommitBuildOnHead){
-      args.push('--FirstBuildOnHead');
-    }
-    if (test.secondCommitBuildOnHead){
-      args.push('--SecondBuildOnHead');
-    }
-  } else {
-    console.log(`ERROR: Unrecognized test type of ${test.testType}! This should not happen.`);
-  }
-
-  args.push('--jobName', test.jobHash); // Give our job a name so it can write logs to disk with a unique ID.
-
-  if (vsDevEnv){
-    args.push('--VSDevEnv', vsDevEnv); // Pass along a custom VS Developer Environment if the user set it when starting the server.
-  }
-  if (test.openGLBackend && test.platform == 'win32'){
-    args.push('--OpenGLBackend', test.openGLBackend); // Set the OpenGL backend override if selected in the web interface and this system is Windows.
-  }
-
-  if (test.environment){
-    if (Array.isArray(test.environment)){
-      args.push('--environment', test.environment.join(','));
-    } else {
-      args.push('--environment', test.environment);
-    }
-
-  }
-
-  var pythonexe = os.platform() == 'win32' ? 'python' : 'python3'; // Python3 for windows will simply be 'python'.
-  console.log('Launching ' + pythonexe + ' ' + args.join(' '));
-  test.status = 'running';
-  const testProcess = spawn(pythonexe, args, os.platform() == 'win32' ? {detached: false} : {detached: true});
-
-  const logFile = fs.createWriteStream(`logs/${myname}/${test.jobHash}.txt`); // dump process console output to file.
-
-  // Unfortunately I can't find a way to dump both stdout and stderr at the same time to the same file, so print statements in the python script will always appear at the end of the log in their own section.
-  // If you read this and know how to fix it, let me know @ daniel.smith[.at.]qt.io
-  testProcess.stdout.on('data', (data) => {
-    logFile.write(data); // log to file
-  });
-  testProcess.stderr.on('data', (data) => {
-    logFile.write(data); // log to file
-  });
-
-  testProcess.on('close', code => {
-    // The process finished-- Do some tasks
-
-    console.log(`Process for job ${test.jobHash.slice(0, 10)} exited with code ${code}`);
-    test.logURL = `http://QMLBench.intra.qt.io/logs/${myname}/${test.jobHash}.txt`;
-    test.finishTime = Date.now();
-    test.actualDuration = test.finishTime - test.startTime;
-    if (!isPrimaryHost){ // All logs should get copied to the scheduler host.
-      const { exec } = require('child_process');
-      exec(`${os.platform() == 'win32' ? 'copy' : 'cp'} ${pathmodule.join(__dirname, 'logs', myname, test.jobHash + '.txt')} ${pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs', myname)}`, (error, stdout, stderr) => {
-        if (error) {
-          console.error(`Failed to copy the log file: ${error}`); // log failed file copies but don't error out.
-        }
-      });
-      exec(`${os.platform() == 'win32' ? 'copy' : 'cp'} ${pathmodule.join(__dirname, 'logs', myname, 'results', test.jobHash + '.json')} ${pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs', myname, 'results')}`, (error, stdout, stderr) => {
-        if (error) {
-          console.error(`Failed to copy the log file: ${error}`); // log failed file copies but don't error out.
-        }
-      });
-    }
-    if (test.status != 'cancelling'){
-      // If the test is in cancelling status, it means that someone pressed the cancel button. So provided it exited naturally, try to get test results from the run, even if the job failed and hard-exited.
-      try {
-        const f = fs.readFileSync(`logs/${test.test_hosts[0]}/results/${test.jobHash}.json`);
-        test.resultJSON = JSON.parse(f);
-      } catch (err){
-        console.log('ERROR: Failed to read the results file. Maybe the test failed to finish...');
-        // console.log(err);
-      }
-
-      if (code != 0 || code == 'null') {
-        console.log(test.title, 'Failed: with code', code);
-        test.status = 'failed';
-      } else {
-        console.log(test.title, 'finished with code ', code);
-        test.status = 'finished';
-        // Update the new expected durations for this server/job type
-        console.log(`Stored expected:${expectedDurations[test.testType]}\nActual Duration: ${moment.duration(test.actualDuration, 'ms').humanize()}\nNew Duration for ${test.testType}: ${Math.ceil((expectedDurations[test.testType] + test.actualDuration) / 2)}`);
-        expectedDurations[test.testType] = Math.ceil((expectedDurations[test.testType] + test.actualDuration) / 2);
-      }
-    } else {
-      // The job was in cancelling status when it exited. Set it to cancelled now.
-      test.status = 'cancelled';
-    }
-
-    if(test.custom_benchmark_file){
-      if(test.custom_benchmark_file.tempFilePath){
-        (async () => {
-          var re = /\\/gi;
-          // can't use path.normalize here. del@5.0.0 module doesn't seem to work with windows style paths at the moment.
-          const deletedPaths = await del([pathmodule.dirname(test.custom_benchmark_file.tempFilePath).replace(re, "/")], {force: true});
-          console.log('Deleted files and directories:\n', deletedPaths.join('\n'));
-      })(error => {console.log(error)});
-      }
-    }
-
-    notifyJob.emit('JobDone'); // Emit the JobDone signal to do some more tasks and prep for the next job in queue.
-
-    // saveTests(); //Backup the JSON file on disk for the status web page. // Not implemented!
-  });
-  return testProcess;
-}
-
-
-notifyJob.on('newLocalJob', function(){
-  // We received a job for this server to run. If we're not locked out due to another current process, execute it immediately.
-  if (!locked){
-    locked = true; // Lock out other jobs from trying to run.
-    runningJob = jobQueue.pop(); // Grab a job from the queue
-    runningJob.status = 'running';
-    runningJob.startTime = Date.now(); // Start the timer so we can track how long these things take...
-    if (runningJob.remote_job){
-      notifyJob.emit('updateRemoteJob', runningJob, 'running'); // Notify the scheduler that we've moved this job into running status.
-    }
-    setTimeout(function() { runningJob.processHandle = runTest(runningJob); }, 1000); // set the process handle and hand the job over to the test runner
-  }
-});
-
-function clearErrors(req, res){
-  // Clear system errors on the web interface when running in scheduler mode.
-  recentErrors.length = 0;
-  clearTimeout(nextErrorClearing);
-
-  if (res){
-    res.redirect('/');
-  }
-}
-
-function clearFinishedJobs(req, res){
-  // Clear the list of completed jobs
-  completedJobs = [];
-
-  if (res){
-    res.redirect('/');
-  }
-}
-
-function errorsPage(req, res){
-  // Display stored system errors.
-
-  var pageHTML = `
-    <head>
-    <title>Benchmark System Errors</title>
-    <link rel="shortcut icon" href="assets/favicon.ico"/> 
-    </head>
-    <style>
-    body{
-      font: 13px "Helvetica Neue", "Lucida Grande", "Arial";
-      background: #ECE9E9 -webkit-gradient(linear, 0% 0%, 0% 100%, from(#fff), to(#ECE9E9));
-      background-repeat: no-repeat;
-      color: #555;
-      -webkit-font-smoothing: antialiased;
-    }
-    </style>
-    <button onclick="window.location.href='/'">Back</button>
-    <button onclick="window.location.href='/clearRecentErrors'">Clear all errors</button>
-    <br>
-    `;
-  if (recentErrors){
-    for (var i in recentErrors){
-      pageHTML += `
-            <h3>Error ${recentErrors[i].error} on ${recentErrors[i].job.test_hosts[0]}</h3>
-            <pre>${JSON.stringify(recentErrors[i].job, null, 4)}</pre>
-            <br>
-            `;
-    }
-  } else {
-    pageHTML += `
-        <h2>💤 Things are looking pretty quiet around here... 💤</h2>
-        `;
-  }
-
-  res.send(pageHTML);
-}
-
-notifyJob.on('finishedScheduling', function(req, res){
-  // Redirect the scheduler back to the homepage once we've sent out and received a response from all job requests.
-  setTimeout(function() { res.redirect('/'); }, 800, res);
-});
-
-function scheduleJobRequest(req, res){
-  // This is triggered in two ways [1. When a user presses the 'schedule' button in the web interface] of [2. When a remote server receives a request to schedule a new job.]
-  const readyEmitter = new events.EventEmitter();
-
-  console.log('Received request to schedule new job:');
-  console.log(req.body); // Dump the job request to the console for logging and diagnostic purposes.
-  console.log(req.files);
-
-  if (!Array.isArray(req.body.test_hosts)){ // Magic. This allows us to use the same endpoint for scheduling a job on both one and multiple hosts.
-    req.body.test_hosts = [req.body.test_hosts];
-  }
-
-  var numJobsToSchedule = req.body.test_hosts.length; // Count how many times we need to schedule. Used for knowing when to redirect the scheduler browser
-  var jobsScheduled = 0;
-
-  notifyJob.on('jobScheduled', function(req, res){
-    jobsScheduled += 1;
-    if (jobsScheduled == numJobsToSchedule){ // Received responses from all job schedule requests.
-      notifyJob.emit('finishedScheduling', req, res);
-    }
-  });
-
-  readyEmitter.on('ready', function(title){
-    // The ready signal is fired when we've finished polling codereview for a job title from one of the commits.
-
-    for (var i in req.body.test_hosts){ // Create a job for each test host
-      const job = {
-        timestamp: Date.now(),
-        testType: req.body.testType,
-        platform: req.body.platform,
-        owner: req.body.owner,
-        title: title,
-        status: req.body.status,
-        branch: req.body.branch,
-        module: req.body.module,
-        commit: req.body.commit,
-        firstCommitBuildOnHead: req.body.firstCommitBuildOnHead,
-        secondCommitBuildOnHead: req.body.secondCommitBuildOnHead,
-        firstCommit: req.body.firstCommit,
-        secondCommit: req.body.secondCommit,
-        bad_commit: req.body.bad_commit,
-        good_date: req.body.good_date,
-        good_commit: req.body.good_commit,
-        openGLBackend: req.body.openGLBackend,
-        test_name: req.body.test_name,
-        custom_benchmark_file: req.files ? req.files.custom_benchmark_file : req.body.custom_benchmark_file,
-        expected_regression: req.body.expected_regression,
-        test_all: req.body.test_all,
-        test_hosts: Array.isArray(req.body.test_hosts) ? [req.body.test_hosts[i]] : [req.body.test_hosts],
-        environment: req.body.environment,
-        remote_job: req.body.remote_job,
-        master_host: req.body.master_host,
-      };
-
-      //Dump the data from the custom file to disk as a temp file.
-      if (job.custom_benchmark_file){
-        // Check to see if we have a properly formed data buffer. If not, re-create it from the object.
-        if (! (job.custom_benchmark_file.data.constructor && job.custom_benchmark_file.data.constructor.isBuffer && job.custom_benchmark_file.data.constructor(job.custom_benchmark_file.data))){
-          job.custom_benchmark_file.data = Buffer.from(job.custom_benchmark_file.data.data)
-        }
-      }
-
-      if (job.owner == ''){
-        job.owner = 'default'; // Set a display name if the email address field wasn't filled.
-      }
-
-      if (req.body.test_hosts[i] == myname){
-        // Either we're a remote host and received a job, or we're the scheduler and have a job thread available.
-        console.log('executing test locally.');
-        if (job.custom_benchmark_file){
-          job.custom_benchmark_file['tempFilePath'] = tempWrite.sync(job.custom_benchmark_file.data, `${job.custom_benchmark_file.name}`);
-        }
-        const jobHash = JSum.digest(job, 'SHA256', 'hex'); // Create a unique hash for this job
-        console.log(`Hash for this job: ${jobHash}`);
-        job['jobHash'] = jobHash;
-
-        if (runningJob.jobHash === jobHash || jobQueue.findIndex(x => x.jobHash == jobHash) >= 0){
-          // This should never happen. It could only really happen if the scheduler sends the same exact job twice (including the timestamp on it)
-          console.log('Returning error. Job already exists.');
-          if (!isPrimaryHost){
-            res.status(500).send({ error: 'jobExists', job: job });
-            continue;
-          } else {
-            recentErrors.push({ error: 'jobExists', job: job });
-            nextErrorClearing = setTimeout(clearErrors, 900000);
-            notifyJob.emit('jobScheduled', req, res);
-            continue;
-          }
-        }
-        job.platform = os.platform();
-        job.status = 'staged'; // All jobs go to the queue first. No cutting line, even if there isn't one.
-        job.expectedDuration = expectedDurations[job.testType]; // Each server keeps an expected duration for each type of job so we can give more accurate estimates.
-        jobQueue.push(job);
-        if (job.remote_job){
-          res.send(job); // Send the completed job object to the scheduler.
-        } else {
-          notifyJob.emit('jobScheduled', req, res); // We're running this on the scheduler itself, need to send a different kind of response.
-        }
-        notifyJob.emit('newLocalJob'); // Notify this server that there's a new job and execute if the queue's empty.
-      } else {
-        // We're processing the meta job request and there's a host that isn't the scheduler. Mark it as remote and forward it.
-        job['remote_job'] = true;
-        job['master_host'] = ip.address(); // Provide some info about where to send future job updates.
-        // delete job['custom_benchmark_file'];
-        // data = new FormData();
-        // data.extend(job);
-        // data.append('custom_benchmark_file', )
-        
-        
-        console.log(`Posting job to http://${hosts[hosts.findIndex(x => x.hostname == req.body.test_hosts[i])].ip}:${webPort}/remoteSchedule`);
-        axios.post(`http://${hosts[hosts.findIndex(x => x.hostname == req.body.test_hosts[i])].ip}:${webPort}/remoteSchedule`, job).then(
-          response => {
-            console.log('Sent the job to remote host and got a response. Looks good.');
-            remoteQueuedJobs.push(response.data);
-            notifyJob.emit('jobScheduled', req, res); // +1 to scheduled jobs
-          }).catch(error => {
-          console.log(error.response.data);
-          if (error.response.data.error == 'jobExists'){
-            recentErrors.push(error.response.data);
-            nextErrorClearing = setTimeout(clearErrors, 900000);
-          } else {
-            console.log(error);
-          }
-          notifyJob.emit('jobScheduled', req, res); // +1 to scheduled jobs
-        });
-      }
-    }
-  });
-
-
-  if (!req.body.title){
-    // Initialize the job with a title pulled from one of the commits provided.
-    const sha = req.body.commit ? req.body.commit : req.body.firstCommit ? req.body.firstCommit : req.body.bad_commit;
-
-    axios.get(`https://codereview.qt-project.org/changes/?q=commit:${sha}`, {timeout: 2000}).then(
-      response => {
-        const title = JSON.parse(response.data.slice(4,))[0].subject;
-        // console.log(`Found sha ${sha} on codereview with title '${title}'.`)
-        readyEmitter.emit('ready', title);
-      }).catch(error => {
-      // console.log(error);
-      const title = 'Unknown Job';
-      readyEmitter.emit('ready', title); // Go ahead and schedule jobs without a title. Hopefully it works, but good luck.
-    });
-  } else {
-    readyEmitter.emit('ready', req.body.title); // We got a title, schedule the jobs using this.
-  }
-
-}
-
-function queryJobs(req, res){
-  // We're a remote host and the scheduler asked us for our jobs. Give it to 'em!
-  console.log('Sending current running and queued job list...');
-  console.log(`${JSON.stringify({runningJob: runningJob.jobHash, queuedJobs: jobQueue})}`);
-
-  // Create a temporary job object that we'll send because stringifying a process handle of the running process will fail in a very nasty way, so just don't include it. It's only useful on the host running the process anyhow.
-  const tempRunningJob = {
-    jobHash: runningJob.jobHash,
-    platform: runningJob.platform,
-    startTime: runningJob.startTime,
-    expectedDuration: runningJob.expectedDuration,
-    timestamp: runningJob.timestamp,
-    testType: runningJob.testType,
-    owner: runningJob.owner,
-    title: runningJob.title,
-    status: runningJob.status,
-    branch: runningJob.branch,
-    module: runningJob.module,
-    commit: runningJob.commit,
-    firstCommitBuildOnHead: runningJob.firstCommitBuildOnHead,
-    secondCommitBuildOnHead: runningJob.secondCommitBuildOnHead,
-    firstCommit: runningJob.firstCommit,
-    secondCommit: runningJob.secondCommit,
-    bad_commit: runningJob.bad_commit,
-    good_date: runningJob.good_date,
-    good_commit: runningJob.good_commit,
-    openGLBackend: req.body.openGLBackend,
-    test_name: runningJob.test_name,
-    custom_benchmark_file: runningJob.custom_benchmark_file,
-    expected_regression: runningJob.expected_regression,
-    test_all: runningJob.test_all,
-    test_hosts: runningJob.test_hosts,
-    environment: runningJob.environment,
-    remote_job: runningJob.remote_job,
-    master_host: runningJob.master_host,
-  };
-
-  res.send({runningJob: tempRunningJob, queuedJobs: jobQueue});
-}
-
-function querySlaveJobs(req, res){
-  // Trigger this to dump the scheduler's local cache of it's remote hosts's jobs and ask them what they have instead. This helps deal with any network hiccups that might have caused a missed update.
-
-  const readyEmitter = new events.EventEmitter();
-  var countSlaveQueries = 0; // How many queries do we need to do before we refresh the scheduler's page?
-
-  readyEmitter.on('finished', function(){
-    countSlaveQueries += 1;
-    if (countSlaveQueries >= slaves.length){ // have we gotten all our queries back?
-      readyEmitter.emit('ready');
-    }
-  });
-
-  readyEmitter.on('ready', function(){
-    if (res){
-      res.redirect('/'); // Redirect the scheduler page back home and force a refresh.
-    }
-  });
-
-  console.log('Clearing remote job listing cache.');
-  remoteRunningJobs = [];
-  remoteQueuedJobs = [];
-  console.log('Gathering remote host jobs.');
-  for (var i in slaves){
-    if (slaves[i] != ip.address()){
-      axios.get(`http://${slaves[i]}:${webPort}/queryJobs`, {timeout: 2000}).then(
-        response => {
-          console.log('Received remote jobs. Updating local lists...');
-          if (Object.entries(response.data.runningJob).length > 2){ // Check to make sure the response isn't a literal empty object such as '{}'
-            if (remoteRunningJobs.findIndex(x => x.jobHash == response.data.runningJob.jobHash) < 0){ // make sure we didn't already receive this same job. It'd be odd, but may as well protect against it.
-              remoteRunningJobs.push(response.data.runningJob);
-            }
-          }
-
-          if (response.data.queuedJobs.length > 0){ // Should be a list of jobs
-            for (var j in response.data.queuedJobs){
-              if (remoteQueuedJobs.findIndex(x => x.jobHash == response.data.queuedJobs[j].jobHash) < 0){ // make sure we didn't already receive this same job. It'd be odd, but may as well protect against it.
-                remoteQueuedJobs.push(response.data.queuedJobs[j]);
-              }
-            }
-          }
-          readyEmitter.emit('finished'); // +1 to the number of remote hosts we've queried.
-        }
-      ).catch(error => {
-        console.log(`WARN: Failed to query ${slaves[i]} for jobs`); // Just a WARN because it's not fatal that one of our remote hosts is offline for a minute.
-        readyEmitter.emit('finished');
-      });
-    }
-  }
-}
-
-notifyJob.on('cancel', function(job, req, res){
-  // Cancel a locally running job
-  if (job.status == 'running'){ // If it's a running job we need to try and kill the process.
-    job.status = 'cancelling';
-
-    if (job.remote_job){
-      console.log(`Preparing to send update to master host about cancelling jobHash ${job.jobHash}`);
-      notifyJob.emit('updateRemoteJob', job, 'cancelling'); // update the scheduler to let it know a job is in cancelling status.
-    }
-
-    // Use platform specific method to kill the entire process tree. Unfortunately, we just have to trust here.
-    if (os.platform() === 'win32'){
-      console.log(`Trying to kill windows job with jobHash ${job.jobHash}`);
-      var spawn = require('child_process').spawn;
-      spawn('taskkill', ['/pid', job.processHandle.pid, '/F', '/T']);
-    } else {
-      console.log(`Trying to kill linux job with jobHash ${job.jobHash}`);
-      process.kill(-job.processHandle.pid);
-    }
-    delete job.processHandle; // Clean up the process handle since it's dead anyway. This makes it easier to stringify the job when sending updates to the scheduler, too.
-
-  } else {
-    // Not a running job, must be queued.
-    console.log(`Moving queued job to completed-cancelled with jobHash ${job.jobHash}`);
-    job.status = 'cancelled'; // No need to go through the work of sending an update for "cancelling", just go straight to "cancelled"
-    completedJobs.push(job);
-    jobQueue.splice(jobQueue.findIndex(x => x.jobHash == job.jobHash)); // pop the queued job out of the queue.
-
-    if (job.remote_job){
-      console.log(`Preparing to send update to master host about cancelled jobHash ${job.jobHash}`);
-      notifyJob.emit('updateRemoteJob', job, 'cancelled');
-    }
-  }
-
-  setTimeout(function() { res.redirect('/'); }, 500, res);
-});
-
-function cancelJob(req, res){
-  // A user pressed the "cancel" button in the web interface on a job
-
-  console.log(`Received cancel request for job ${req.query.jobHash}`);
-
-  // Do some logic to find out which array the job lives in. Forward the cancel request to a handler based on that.
-  if (req.query.jobHash === runningJob.jobHash){
-    console.log('Cancelling local running job');
-    notifyJob.emit('cancel', runningJob, req, res);
-
-  } else if (jobQueue[jobQueue.findIndex(x => x.jobHash == req.query.jobHash)]){
-    console.log('Cancelling local queued job');
-    notifyJob.emit('cancel', jobQueue[jobQueue.findIndex(x => x.jobHash == req.query.jobHash)], req, res);
-
-  } else if (remoteRunningJobs[remoteRunningJobs.findIndex(x => x.jobHash == req.query.jobHash)]){
-    console.log('Cancelling remote running job');
-    var remoteIP = hosts[hosts.findIndex(x => x.hostname == remoteRunningJobs[remoteRunningJobs.findIndex(x => x.jobHash == req.query.jobHash)].test_hosts[0])].ip;
-    axios.get(`http://${remoteIP}:${webPort}/cancelJob?jobHash=${req.query.jobHash}`, {timeout: 2000})
-      .then(response => {
-        setTimeout(function() { res.redirect('/'); }, 500, res); // Delay the redirect to the remote host might have time to update the job before we refresh the page for the user.
-      }).catch(error => {
-        console.log(`ERROR: Failed to post cancel request to ${remoteIP}`);
-        recentErrors.push({
-          error: `ERROR: Failed to post cancel request for job hash ${req.query.jobHash.slice(0, 10)}`,
-          job: remoteRunningJobs[remoteRunningJobs.findIndex(x => x.jobHash == req.query.jobHash)],
-        });
-        res.redirect('/');
-      });
-
-  } else if (remoteQueuedJobs[remoteQueuedJobs.findIndex(x => x.jobHash == req.query.jobHash)]){
-    console.log('Cancelling remote queued job');
-    var remoteIP = hosts[hosts.findIndex(x => x.hostname == remoteQueuedJobs[remoteQueuedJobs.findIndex(x => x.jobHash == req.query.jobHash)].test_hosts[0])].ip;
-    axios.get(`http://${remoteIP}:${webPort}/cancelJob?jobHash=${req.query.jobHash}`, {timeout: 2000})
-      .then(response => {
-        setTimeout(function() { res.redirect('/'); }, 500, res); // Delay the redirect to the remote host might have time to update the job before we refresh the page for the user.
-      }).catch(error => {
-        console.log(`ERROR: Failed to post cancel request to ${remoteIP}`);
-        recentErrors.push({
-          error: `ERROR: Failed to post cancel request for job hash ${req.query.jobHash.slice(0, 10)}`,
-          job: remoteRunningJobs[remoteRunningJobs.findIndex(x => x.jobHash == req.query.jobHash)],
-        });
-        res.redirect('/');
-      });
-
-  } else {
-    console.log("not cancelling anything. Couldn't find the job hash?"); // This shouldn't happen.
-    res.redirect('/');
-  }
-}
-
-function schedulerPage(req, res) {
-  // Display the scheduler page to the web user
-
-  if (hosts.length === 0){
-    axios.get(`http://${ip.address()}:${webPort}`), {timeout: 2000}; // Load the homepage of myself in case someone loaded the scheduler immediately after startup. This forces a refresh of remote slaves.
-  }
-
-  // Populate the list of test hosts
-  var hostSelectionComboBox = `<select id="test_hosts" name="test_hosts" multiple required size=${hosts.length}>`;
-  if (hosts.length === 0){
-    hostSelectionComboBox = '<option value="" disabled>No hosts - Please refresh</option>';
-  } else {
-    for (var i in hosts){
-      hostSelectionComboBox += `<option value="${hosts[i].hostname}">${hosts[i].hostname}</option>`;
-    }
-    hostSelectionComboBox += '</select>';
-  }
-
-  if (!testNames){
-    gatherTestNames();
-  }
-
-  // Send the HTML page.
-  res.send(`
-    <head>
-    <title>Schedule a new job</title>
-    <link rel="stylesheet" href="awesomeplete/awesomplete.css" />
-    <link rel="shortcut icon" href="assets/favicon.ico"/> 
-    </head>
-
-    <style>
-    body{
-      font: 13px "Helvetica Neue", "Lucida Grande", "Arial";
-      background: #ECE9E9 -webkit-gradient(linear, 0% 0%, 0% 100%, from(#fff), to(#ECE9E9));
-      background-repeat: no-repeat;
-      color: #555;
-      -webkit-font-smoothing: antialiased;
-    }
-    .tooltip{display:inline;position:relative}
-    .tooltip:hover{text-decoration:none}
-
-    .tooltip:hover:after{
-    background:#111;
-    background:rgba(0,0,0,.8);
-    border-radius:5px;
-    bottom:18px;
-    width: 300px;
-    color:#fff;
-    content:attr(help);
-    display:block;
-    left:50%;
-    padding:5px 15px;
-    position:absolute;
-    white-space:pre-wrap;
-    z-index:98
-    }
-    .tooltip:hover:before{
-        border:solid;
-        border-color:#111 transparent;
-        border-width:6px 6px 0 6px;
-        bottom:12px;
-        content:"";
-        display:block;
-        left:75%;
-        position:absolute;
-        z-index:99
-    }
-
-    .BasicButton {
-        -moz-box-shadow:inset 0px 1px 3px 0px #91b8b3;
-        -webkit-box-shadow:inset 0px 1px 3px 0px #91b8b3;
-        box-shadow:inset 0px 1px 3px 0px #91b8b3;
-        background:-webkit-gradient(linear, left top, left bottom, color-stop(0.05, #768d87), color-stop(1, #6c7c7c));
-        background:-moz-linear-gradient(top, #768d87 5%, #6c7c7c 100%);
-        background:-webkit-linear-gradient(top, #768d87 5%, #6c7c7c 100%);
-        background:-o-linear-gradient(top, #768d87 5%, #6c7c7c 100%);
-        background:-ms-linear-gradient(top, #768d87 5%, #6c7c7c 100%);
-        background:linear-gradient(to bottom, #768d87 5%, #6c7c7c 100%);
-        filter:progid:DXImageTransform.Microsoft.gradient(startColorstr='#768d87', endColorstr='#6c7c7c',GradientType=0);
-        background-color:#768d87;
-        -moz-border-radius:5px;
-        -webkit-border-radius:5px;
-        border-radius:5px;
-        border:1px solid #566963;
-        display:inline-block;
-        cursor:pointer;
-        color:#ffffff;
-        font-family:Arial;
-        font-size:12px;
-        font-weight:bold;
-        padding:3px 8px;
-        text-decoration:none;
-        text-shadow:0px -1px 0px #2b665e;
-    }
-    .BasicButton:hover {
-        background:-webkit-gradient(linear, left top, left bottom, color-stop(0.05, #6c7c7c), color-stop(1, #768d87));
-        background:-moz-linear-gradient(top, #6c7c7c 5%, #768d87 100%);
-        background:-webkit-linear-gradient(top, #6c7c7c 5%, #768d87 100%);
-        background:-o-linear-gradient(top, #6c7c7c 5%, #768d87 100%);
-        background:-ms-linear-gradient(top, #6c7c7c 5%, #768d87 100%);
-        background:linear-gradient(to bottom, #6c7c7c 5%, #768d87 100%);
-        filter:progid:DXImageTransform.Microsoft.gradient(startColorstr='#6c7c7c', endColorstr='#768d87',GradientType=0);
-        background-color:#6c7c7c;
-    }
-    .BasicButton:active {
-        position:relative;
-        top:1px;
-    }    
-
-    </style>
-
-
-    <body>
-    <button onclick="window.location.href='/'">Back</button>
-    <button onclick="location.href='mailto:daniel.smith@qt.io';" style="position:absolute;right:10px;top:10px;">report bug / request feature</button>
-    <br>
-    <table align="center">
-    <tr>
-        <td align="center" colspan="3"><h2>What would you like to schedule?</h2></td>
-    </tr>
-    <tr>
-        <td><button onclick="BasicScheduler()">Automated bisect on merged commits</button></td>
-        <td><button onclick="SingleCommitScheduler()">Single Commit/patch</button></td>
-        <td><button onclick="TwoCommitScheduler()">Two Commit/patch Compare</button></td>
-    </tr>
-    </table>
-    <br>
-
-    <div id="basicScheduler" style="display:none" align="center">
-    <form id="basicSchedulerForm" action="/remoteSchedule/" method="post" encType="multipart/form-data">
-    <input type="hidden" name="testType" value="bisect"/>
-    <label for="branch">Qt major branch:</label>
-    <input id="branch" type="text" name="branch" style="width:50;" required placeholder="5.12"><br>
-    <label for="module">Qt module to bisect: </label>
-    <select id="module" name="module" style="width:200;" required placeholder="qtbase">
-    <option value="qtbase">QtBase</option>
-    <option value="qtdeclarative">QtDeclarative</option>
-    <option value="qtquickcontrols">QtQuickControls</option>
-    <option value="qtquickcontrols2">QtQuickControls2</option>
-    <option value="qtsvg">QtSvg</option>
-    <option value="qtquick3d">QtQuick3d</option>
-    </select>
-    <br>
-    <label for="openGLBackend">OpenGL Backend: </label>
-    <select id="openGLBackend" name="openGLBackend" style="width:100;" placeholder="desktop">
-    <option value="desktop">desktop</option>
-    <option value="angle">Angle</option>
-    <option value="software">software</option>
-    </select>
-    <b help="Only affects Windows. Override the OpenGL rendering backend to use for these tests. Leave default of \'desktop\' if unsure." class="tooltip">(What's this?)</b>
-    <br>
-    <hr style="width:20%">
-    <label for="good_commit">Known good commit: </label><br>
-    <input id="good_commit" type="text" name="good_commit" style="width:330; "value="" required placeholder="becac559b713dfadd27198d683fde70551a543c8"><br>
-    <p><b>Important:</b> Good (older) commit must be an ancestor to the bad commit</p>
-    <p><b>Important:</b> Commit must have been integrated successfully in COIN!</p>
-    <hr style="width:20%">
-    <label for="bad_commit">Known bad commit: </label><br>
-    <input id="bad_commit" type="text" name="bad_commit" style="width:330; "value="" placeholder="Leave blank to use HEAD"><br>
-    <p><b>Important:</b> Commit must have been integrated successfully in COIN!</p>
-    <hr style="width:20%">
-    <label for="test_name"><b>1)</b> Select an existing QMLBench test: </label><b help="Start typing a test name to benchmark from the QMLBench test suite to provide suggestions." class="tooltip">(Help)</b>  <a href='/benchmarks' class="BasicButton">Browse benchmarks</a><br>
-    <input class="awesomplete" id="test_name" type="text" data-list="${testNames}" style="width:600;" name="test_name" required placeholder="f.ex: auto/animations/comparison/moving-images-animations.qml"><br>
-    <p><b>2)</b> Or upload custom <b>.qml</b> benchmark file:</p>
-    <div id="custom_benchmark_file_label" style="width:15%;padding:2px">No custom file set</div>
-    <input id="custom_benchmark_file" name="custom_benchmark_file" type="file" accept=".qml" hidden onchange="updateFileLabel(this.files)" /><br>
-    <input id='benchmarkButtonBisect' type='button' value='Upload custom benchmark' /><br>
-    <hr style="width:20%">
-    <label for="expected_regression">Expected regression, in percentage: </label>
-    <input id="expected_regression" type="number" min="2" max="100" style="width:40;" name="expected_regression" value="" placeholder="20">%<br>
-    <p><b>Note:</b> Be conservative with the regression target.<br>It is not recommended to search for regressions below 5% due to test noise.</p>
-    <hr style="width:20%">
-    <label for="test_hosts">Run job on host:</label><br>
-    ${hostSelectionComboBox}
-    <hr style="width:20%">
-    <label for="environment">Set custom environment variables on hosts:</label><br>
-    <table>
-        <th>Variables</th><th>Variable name</th><th>Variable Value</th>
-        <tr valign="top" height=0%>
-            <td rowspan=2>
-                <select id="environment" name="environment" size=2 style="min-width:100px;overflow:hidden" multiple></select>
-            </td>
-            <td><input id="environmentName" name="environmentName" type="text" style="width:200"></input></td>
-            <td><input id="environmentValue" name="environmentValue" type="text" style="width:200"></input></td>
-        </tr>
-        <tr valign="top">
-        <td align="center" colspan=2 style="padding-top:5px"><button type="button" onclick="AddEnvironmentVariable()">Add Environment Variable</button></td>
-    </table>
-    <hr style="width:20%">
-    <label for="owner">Enter email to receive notices about this job: </label><br>
-    <input id="owner" type="email" style="width:400;" name="owner" placeholder="someone@qt.io"><br>
-    <br>
-    <input type="submit" id="submit" disabled value="Schedule Bisect">
-    </form>
-    </div>
-    </div>
-
-    <div id="singleCommitScheduler" style="display:none" align="center">
-    <form id="singleCommitSchedulerForm" action="/remoteSchedule/" method="post" encType="multipart/form-data">
-    <input type="hidden" name="testType" value="singleCommit"/>
-    <label for="branch">Qt major branch:</label>
-    <input id="branch" type="text" name="branch" style="width:50;" required placeholder="5.12"><br>
-    <label for="module">Qt module to test: </label>
-    <select id="module" name="module" style="width:200;" required placeholder="qtbase">
-    <option value="qtbase">QtBase</option>
-    <option value="qtdeclarative">QtDeclarative</option>
-    <option value="qtquickcontrols">QtQuickControls</option>
-    <option value="qtquickcontrols2">QtQuickControls2</option>
-    <option value="qtsvg">QtSvg</option>
-    <option value="qtquick3d">QtQuick3d</option>
-    </select>
-    <br>
-    <label for="openGLBackend">OpenGL Backend: </label>
-    <select id="openGLBackend" name="openGLBackend" style="width:100;" placeholder="desktop">
-    <option value="desktop">desktop</option>
-    <option value="angle">Angle</option>
-    <option value="software">software</option>
-    </select>
-    <b help="Only affects Windows. Override the OpenGL rendering backend to use for these tests. Leave default of \'desktop\' if unsure." class="tooltip">(What's this?)</b>
-    <br>
-    <hr style="width:20%">
-    <label for="commit">Commit to benchmark: </label><br>
-    <input id="commit" type="text" name="commit" style="width:330; "value="" required placeholder="becac559b713dfadd27198d683fde70551a543c8"><br>
-    Build this Commit against branch HEAD: <input type="checkbox" name="firstCommitBuildOnHead" value="True"> <b help="If not checked: The system will search for a COIN integration relating to the commit. If one is not found, the COIN integration of the commit's direct parent will be used." class="tooltip">(Help)</b><br>
-    <hr style="width:20%">
-    <label for="test_name"><b>1)</b> Select an existing QMLBench test: </label><b help="Start typing a test name to benchmark from the QMLBench test suite to provide suggestions." class="tooltip">(Help)</b>  <a href='/benchmarks' class="BasicButton">Browse benchmarks</a><br>
-    <input class="awesomplete" id="test_name" type="text" data-list="${testNames}" style="width:600;" name="test_name" required placeholder="f.ex: auto/animations/comparison/moving-images-animations.qml"><br>
-    <p><b>2)</b> Or upload custom <b>.qml</b> benchmark file:</p>
-    <div id="custom_benchmark_file_label" style="width:15%;padding:2px">No custom file set</div>
-    <input id="custom_benchmark_file" name="custom_benchmark_file" type="file" accept=".qml" onchange="updateFileLabel(this.files)" hidden /><br>
-    <input id='benchmarkButtonSingleCommit' type='button' value='Upload custom benchmark' /><br>
-    <hr style="width:20%">
-    <label for="test_hosts">Run job on host:</label><br>
-    ${hostSelectionComboBox}
-    <hr style="width:20%">
-    <label for="environment">Set custom environment variables on hosts:</label><br>
-    <table>
-        <th>Variables</th><th>Variable name</th><th>Variable Value</th>
-        <tr valign="top" height=0%>
-            <td rowspan=2>
-                <select id="environment" name="environment" size=2 style="min-width:100px;overflow:hidden" multiple></select>
-            </td>
-            <td><input id="environmentName" name="environmentName" type="text" style="width:200"></input></td>
-            <td><input id="environmentValue" name="environmentValue" type="text" style="width:200"></input></td>
-        </tr>
-        <tr valign="top">
-        <td align="center" colspan=2 style="padding-top:5px"><button type="button" onclick="AddEnvironmentVariable()">Add Environment Variable</button></td>
-    </table>
-    <hr style="width:20%">
-    <label for="owner">Enter email to receive notices about this job: </label><br>
-    <input id="owner" type="email" style="width:400;" name="owner" placeholder="someone@qt.io"><br>
-    <br>
-    <input type="submit" id="submit" disabled value="Schedule Single Commit">
-    </form>
-    </div>
-
-    <div id="twoCommitScheduler" style="display:none" align="center">
-    <form id="twoCommitSchedulerForm" action="/remoteSchedule/" method="post" encType="multipart/form-data">
-    <input type="hidden" name="testType" value="twoCommit"/>
-    <label for="branch">Qt major branch:</label>
-    <input id="branch" type="text" name="branch" style="width:50;" required placeholder="5.12"><br>
-    <label for="module">Qt module to test: </label>
-    <select id="module" name="module" style="width:200;" required placeholder="qtbase">
-    <option value="qtbase">QtBase</option>
-    <option value="qtdeclarative">QtDeclarative</option>
-    <option value="qtquickcontrols">QtQuickControls</option>
-    <option value="qtquickcontrols2">QtQuickControls2</option>
-    <option value="qtsvg">QtSvg</option>
-    <option value="qtquick3d">QtQuick3d</option>
-    </select>
-    <br>
-    <label for="openGLBackend">OpenGL Backend: </label>
-    <select id="openGLBackend" name="openGLBackend" style="width:100;" placeholder="desktop">
-    <option value="desktop">desktop</option>
-    <option value="angle">Angle</option>
-    <option value="software">software</option>
-    </select>
-    <b help="Only affects Windows. Override the OpenGL rendering backend to use for these tests. Leave default of \'desktop\' if unsure." class="tooltip">(What's this?)</b>
-    <br>
-    <hr style="width:20%">
-    <label for="firstCommit">First commit to benchmark: </label><br>
-    <input id="firstCommit" type="text" name="firstCommit" style="width:330; "value="" required placeholder="becac559b713dfadd27198d683fde70551a543c8"><br>
-    Build First Commit against branch HEAD: <input type="checkbox" name="firstCommitBuildOnHead" value="True"><b help="If not checked: The system will search for a COIN integration relating to the commit. If one is not found, the COIN integration of the commit's direct parent will be used." class="tooltip">(Help)</b><br>
-    Regression test this commit against its parent <input type="checkbox" name="simpleRegressionTest" id="simpleRegressionTest" value="True" onchange="disableSecondCommit()"> <b help="Check this box to perform a simple regression test of this commit against its parent to determine if the new commit causes a regression." class="tooltip">(Help)</b><br>
-    <hr style="width:20%">
-    <label for="secondCommit">Second commit to benchmark: </label><br>
-    <input id="secondCommit" type="text" name="secondCommit" id="secondCommit" style="width:330; "value="" required placeholder="becac559b713dfadd27198d683fde70551a543c8"><br>
-    Build Second Commit against branch HEAD: <input type="checkbox" name="secondCommitBuildOnHead" id="secondCommitBuildOnHead" value="True"><b help="If not checked: The system will search for a COIN integration relating to the commit. If one is not found, the COIN integration of the commit's direct parent will be used." class="tooltip">(Help)</b><br>
-    <hr style="width:20%">
-    <label for="test_name"><b>1)</b> Select an existing QMLBench test: </label><b help="Start typing a test name to benchmark from the QMLBench test suite to provide suggestions." class="tooltip">(Help)</b>  <a href='/benchmarks' class="BasicButton">Browse benchmarks</a><br>
-    <input class="awesomplete" id="test_name" type="text" data-list="${testNames}" style="width:600;" name="test_name" required placeholder="f.ex: auto/animations/comparison/moving-images-animations.qml"><br>
-    <p><b>2)</b> Or upload custom <b>.qml</b> benchmark file:</p>
-    <div id="custom_benchmark_file_label" style="width:15%;padding:2px">No custom file set</div>
-    <input id="custom_benchmark_file" name="custom_benchmark_file" type="file" accept=".qml" onchange="updateFileLabel(this.files)" hidden /><br>
-    <input id='benchmarkButtonTwoCommit' type='button' value='Upload custom benchmark' /><br>
-    <hr style="width:20%">
-    <label for="test_hosts">Run job on host:</label><br>
-    ${hostSelectionComboBox}
-    <hr style="width:20%">
-    <label for="environment">Set custom environment variables on hosts:</label><br>
-    <table>
-        <th>Variables</th><th>Variable name</th><th>Variable Value</th>
-        <tr valign="top" height=0%>
-            <td rowspan=2>
-                <select id="environment" name="environment" size=2 style="min-width:100px;overflow:hidden" multiple></select>
-            </td>
-            <td><input id="environmentName" name="environmentName" type="text" style="width:200"></input></td>
-            <td><input id="environmentValue" name="environmentValue" type="text" style="width:200"></input></td>
-        </tr>
-        <tr valign="top">
-        <td align="center" colspan=2 style="padding-top:5px"><button type="button" onclick="AddEnvironmentVariable()">Add Environment Variable</button></td>
-    </table>
-    <hr style="width:20%">
-    <label for="owner">Enter email to receive notices about this job: </label><br>
-    <input id="owner" type="email" style="width:400;" name="owner" placeholder="someone@qt.io"><br>
-    <br>
-    <input type="submit" id="submit" disabled value="Schedule Comparison">
-    </form>
-    </div>
-
-    </body>
-    
-    <script>
-    const basicScheduler = document.getElementById("basicScheduler");
-    const singleCommitScheduler = document.getElementById("singleCommitScheduler");
-    const twoCommitScheduler = document.getElementById("twoCommitScheduler");
-    var activeElement;
-
-    // Tool to help find the right child node in the different forms available
-    var _Utils = function ()
-    {
-        this.findChildById = function (element, childID, isSearchInnerDescendant) // isSearchInnerDescendant <= true for search in inner childern 
-        {
-            var retElement = null;
-            var lstChildren = isSearchInnerDescendant ? Utils.getAllDescendant(element) : element.childNodes;
-    
-            for (var i = 0; i < lstChildren.length; i++)
-            {
-                if (lstChildren[i].id == childID)
-                {
-                    retElement = lstChildren[i];
-                    break;
-                }
-            }
-    
-            return retElement;
-        }
-    
-        this.getAllDescendant = function (element, lstChildrenNodes)
-        {
-            lstChildrenNodes = lstChildrenNodes ? lstChildrenNodes : [];
-    
-            var lstChildren = element.childNodes;
-    
-            for (var i = 0; i < lstChildren.length; i++) 
-            {
-                if (lstChildren[i].nodeType == 1) // 1 is 'ELEMENT_NODE'
-                {
-                    lstChildrenNodes.push(lstChildren[i]);
-                    lstChildrenNodes = Utils.getAllDescendant(lstChildren[i], lstChildrenNodes);
-                }
-            }
-    
-            return lstChildrenNodes;
-        }        
-    }
-    var Utils = new _Utils;
-
-    //Register listeners for file upload buttons.
-    document.getElementById('benchmarkButtonBisect').addEventListener('click', openDialog);
-    document.getElementById('benchmarkButtonTwoCommit').addEventListener('click', openDialog);
-    document.getElementById('benchmarkButtonSingleCommit').addEventListener('click', openDialog);
-    function openDialog() {
-        var e = Utils.findChildById(activeElement, "custom_benchmark_file", true);
-        e.click();
-        var benchmarktextfield = Utils.findChildById(activeElement, "test_name", true);
-        benchmarktextfield.disabled = true;
-    }
-
-    function updateFileLabel(files) {
-      var e = Utils.findChildById(activeElement, "custom_benchmark_file_label", true);
-      e.textContent = \`Uploaded \${files[0].name}\`;
-      e.style.backgroundColor = "lightgreen"
-    }
-
-    // Methods to change which scheduler is visible
-    function BasicScheduler() {
-        if (basicScheduler.style.display === "none") {
-            basicScheduler.style.display = "block";
-            activeElement = basicScheduler;
-            singleCommitScheduler.style.display = "none";
-            twoCommitScheduler.style.display = "none"
-          } else {
-            basicScheduler.style.display = "none";
-          }
-        enableButton();
-    }
-
-    function SingleCommitScheduler(){
-        if (singleCommitScheduler.style.display === "none") {
-            basicScheduler.style.display = "none";
-            singleCommitScheduler.style.display = "block";
-            activeElement = singleCommitScheduler;
-            twoCommitScheduler.style.display = "none"
-          } else {
-            singleCommitScheduler.style.display = "none";
-          }
-        enableButton();
-    }
-
-    function TwoCommitScheduler(){
-        if (twoCommitScheduler.style.display === "none") {
-            basicScheduler.style.display = "none";
-            singleCommitScheduler.style.display = "none";
-            twoCommitScheduler.style.display = "block"
-            activeElement = twoCommitScheduler;
-          } else {
-            twoCommitScheduler.style.display = "none";
-          }
-        enableButton();
-    }
-
-    // Keep the submit buttons disabled until a valid test host is selected 
-    function enableButton(){
-        var e = Utils.findChildById(activeElement, "test_hosts", true)
-        if (e.options){
-            Utils.findChildById(activeElement, "submit", true).disabled = false;
-        } else {
-            Utils.findChildById(activeElement, "submit", true).disabled = false;
-        }
-    }
-    
-    function disableSecondCommit(){
-        var e = Utils.findChildById(activeElement, "simpleRegressionTest", true)
-        if (e.checked){
-            var secondCommit = Utils.findChildById(activeElement, "secondCommit", true);
-            secondCommit.readOnly = true;
-            secondCommit.value = "PARENT";
-            secondCommit.style.backgroundColor = "#C0C0C0";
-            Utils.findChildById(activeElement, "secondCommitBuildOnHead", true).disabled = true;
-        } else{
-            var secondCommit = Utils.findChildById(activeElement, "secondCommit", true);
-            secondCommit.readOnly = false;
-            secondCommit.value = "";
-            secondCommit.style.backgroundColor = "";
-            Utils.findChildById(activeElement, "secondCommitBuildOnHead", true).disabled = false;
-        }
-        console.log("Finished")
-    }
-
-    function AddEnvironmentVariable(){
-        var e = Utils.findChildById(activeElement, "environment", true)
-        var n = Utils.findChildById(activeElement, "environmentName", true)
-        var v = Utils.findChildById(activeElement, "environmentValue", true)
-        if(n.value != "" && v.value != "") {
-            var option = document.createElement("option");
-            option.text = \`\${n.value}=\${v.value}\`;
-            option.selected = true;
-            e.add(option);
-            e.size = e.length
-            n.value = ""
-            v.value = ""
-        }
-    }
-
-    </script>
-    <script src="awesomeplete/awesomplete.js" async>
-        // var input = document.getElementById("test_name");
-        // new Awesomplete(input, {
-        //     list: [${testNames}]
-        // });
-    </script>
-    `);
-}
-
-function homePage(req, res){
-  // Serve the scheduler's home page with status about remote hosts, current, queued, and finished jobs.
-
-  function sendSlaveData(){ // This is called when the host is not the scheduler. It responds with status data.
-    const readyEmitter = new events.EventEmitter();
-    readyEmitter.on('ready', function(){
-      res.send({
-        freediskspace: freediskspace,
-        uptime: process.uptime(),
-        hostname: myname,
-        ip: ip.address(),
-      });
-    });
-
-
-    console.log('Gathering disk space data.');
-    var freediskspace = -1;
-    function setDiskSpace(bytesfree){
-      freediskspace = bytesfree;
-      readyEmitter.emit('ready');
-    }
-
-    disk.check(path)
-      .then(info => {
-        setDiskSpace(info.available);
-      })
-      .catch(err => {
-        console.error(err);
-        readyEmitter.emit('ready');
-      });
-  }
-
-  function jobFormatter(job){
-    // Dynamically format a job for display on the page
-
-    function resultText(result){
-      if(result < 0 ){
-        return `Observed improvement: ${Math.abs(result)}%`;
-      } else if (result > 0){
-        return `Observed regression: ${result}%`;
-      } else {
-        return "-- No change in score --";
-      }
-    }
-
-    const commitString = job.commit ? job.commit : job.firstCommit ? job.firstCommit.slice(0, 10) + '..' + job.secondCommit.slice(0, 10) : job.good_commit ? job.good_commit.slice(0, 10) + '..' + job.bad_commit.slice(0, 10) : '';
-    const backendString = job.platform == 'win32' && job.openGLBackend ? ` - Open GL Backend: ${job.openGLBackend}` : '';
-    var resultString = '';
-    if (job.resultJSON && job.status != 'failed' && job.status != 'cancelled'){
-      if (job.testType == 'singleCommit') {
-        resultString = `Benchmark result: ${job.resultJSON.finalResults[0]}`;
-      } else if (job.testType == 'twoCommit'){
-        resultString = `<b>Results: </b>First commit result (${job.firstCommit.slice(0, 10)}): ${job.resultJSON.finalResults[0]}, 
-                                Second commit result (${job.secondCommit.slice(0, 10)}): ${job.resultJSON.finalResults[1]}<br>
-                                ${resultText(job.resultJSON.finalResults[2])}`;
-      }
-      if (job.testType == 'bisect'){
-        const bisectResult = job.resultJSON.bisectResults[1].length > 1 ?
-          `Bisect returned multiple commits:
-                                    ${job.resultJSON.bisectResults[1].slice(0,).join(' ')}
-                                    `
-          : `Bisect returned a single commit: ${job.resultJSON.bisectResults[1]}`;
-
-        resultString = `<b>Results: </b>Good commit result (${job.good_commit.slice(0, 10)}): ${job.resultJSON.baselineResults[0]}, 
-                                Bad commit result (${job.bad_commit.slice(0, 10)}): ${job.resultJSON.baselineResults[1]}<br>
-                                ${resultText(job.resultJSON.baselineResults[2])}<br>
-                                ${bisectResult}
-                                `;
-      }
-    }
-
-    var testString = '';
-    if (job.test_name){
-      testString = `Benchmark: ${job.test_name}`;
-    } else if (job.custom_benchmark_file){
-        testString = `Custom benchmark: ${job.custom_benchmark_file.name}`;
-    }
-
-    var timeEstimate = '';
-    if (job.startTime && job.status === 'running'){
-      if (job.expectedDuration - (Date.now() - job.startTime) < 1){
-        timeEstimate = `Job is about ${moment.duration(Math.abs(moment.duration(moment.duration(job.expectedDuration, 'ms').subtract(moment.duration(moment(Date.now()).diff(moment(job.startTime)))))), 'ms').format('h [hrs], m [min]')} overdue`;
-      } else {
-        timeEstimate = `About ${moment.duration(moment.duration(job.expectedDuration, 'ms').subtract(moment.duration(moment(Date.now()).diff(moment(job.startTime))))).format('h [hrs], m [min]')} remaining`;
-      }
-    } else if (job.status == 'staged'){
-      timeEstimate = `Expected to run for about ${moment.duration(job.expectedDuration, 'ms').format('h [hrs], m [min]')}`;
-    }
-    var logButton = '';
-    if (job.status != 'staged'){
-      logButton = `<br><button onclick="window.location.href='${job.status == 'running' ? 'http://' + (hosts[hosts.findIndex(x => x.hostname == job.test_hosts[0])] ? hosts[hosts.findIndex(x => x.hostname == job.test_hosts[0])] : {ip: '127.0.0.1'}).ip + ':' + webPort : ''}/logs/${job.test_hosts[0]}/${job.jobHash}.txt'">View Log</button>`;
-    }
-
-    // Return a formatted HTML string of the job to insert into the page.
-    return `
-        <b>${job.testType}
-        </b> job <font color="#454545"><b>${job.jobHash.slice(0, 10)}: ${job.title}<br>User: ${job.owner}</b></font> - Host: ${job.test_hosts[0]} - Timestamp: ${moment(job.timestamp).format('ddd DD MMM YYYY - HH:mm')} is ${job.status}<br>
-        Tested commit(s) ${commitString}${backendString}<br>
-        ${testString}
-        ${timeEstimate ? '<br>' + timeEstimate : ''}
-        ${resultString ? '<br>' + resultString : ''}
-        ${logButton}
-        `;
-  }
-
-  if (isPrimaryHost && !req.query.status){ // Serve the web page if we're the scheduler and status wasn't explicitly requested.
-    const readyEmitter = new events.EventEmitter();
-    // build the basic table element for the remote hosts
-    var slavesStatusHTML = '<table align="center"><tr><th>Slave Hostname</th><th>Online</th><th>Free Disk Space</th><th>Uptime</th><th>IP address</th></tr>';
-
-    var currentJobsHTML = nolocal ? '' : '<h3>Current Local Jobs</h3>';
-
-    // Do a whole bunch of stuff to build the various job type lists.
-    if (Object.entries(runningJob).length != 0 && runningJob.constructor === Object){ // Only display local jobs if there are any.
-
-      currentJobsHTML += `
-            <h4>Running:</h4>
-            <ul>
-            <li class="${runningJob.status}" style="position:relative">
-            <div class="lds-css ng-scope">
-            <div class="lds-spin" style="width:100%;height:100%;position:absolute;left:-35px;top:30%;"><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div></div>
-        ${jobFormatter(runningJob)} <button onclick="doCancel('${runningJob.jobHash}')" ${runningJob.status == 'cancelling' ? 'disabled' : ''}>Cancel</button></li>
-            </ul>
-            `;
-    }
-    if (jobQueue.length != 0){
-      currentJobsHTML += '<h4>Queued:</h4><ul>';
-      for (var i = jobQueue.length - 1; i >= 0; i--){
-        currentJobsHTML += `<li class="${jobQueue[i].status}">
-                ${jobFormatter(jobQueue[i])} <button onclick="doCancel('${jobQueue[i].jobHash}')" ${jobQueue[i].status == 'cancelling' ? 'disabled' : ''}>Cancel</button></li>
-                `;
-      }
-      currentJobsHTML += '</ul>';
-    }
-
-    var remoteJobsHTML = '';
-
-    if (remoteRunningJobs.length != 0){ // Only display remote running jobs if there are any
-      remoteJobsHTML += '<h4>Running:</h4><ul>';
-      for (var i = remoteRunningJobs.length - 1; i >= 0; i--){
-        remoteJobsHTML += `
-                <li class="${remoteRunningJobs[i].status}" style="position:relative;">
-                <div class="lds-css ng-scope">
-                <div class="lds-spin" style="width:100%;height:100%;position:absolute;left:-35px;top:30%;"><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div></div>
-                ${jobFormatter(remoteRunningJobs[i])} <button onclick="doCancel('${remoteRunningJobs[i].jobHash}')" ${remoteRunningJobs[i].status == 'cancelling' ? 'disabled' : ''}>Cancel</button></li>
-                `;
-      }
-      remoteJobsHTML += '</ul>';
-    }
-
-    if (remoteQueuedJobs.length != 0){ // Only display remote queued jobs if there are any
-      remoteJobsHTML += '<h4>Queued:</h4><ul>';
-      for (var i = remoteQueuedJobs.length - 1; i >= 0; i--){
-        remoteJobsHTML += `<li class="${remoteQueuedJobs[i].status}">
-                ${jobFormatter(remoteQueuedJobs[i])}<button onclick="doCancel('${remoteQueuedJobs[i].jobHash}')" ${remoteQueuedJobs[i].status == 'cancelling' ? 'disabled' : ''}>Cancel</button></li>
-                `;
-      }
-      remoteJobsHTML += '</ul>';
-    }
-
-    var completedJobsHTML = '';
-
-    if (completedJobs.length != 0){ // Add completed jobs to the completed list
-      if (completedJobs.length > 15){
-        completedJobs = completedJobs.slice(-15,);
-      }
-      completedJobsHTML += '<ul>';
-      for (var i = completedJobs.length - 1; i >= 0; i--){
-        completedJobsHTML += `
-                <li class= "${completedJobs[i].status}">${jobFormatter(completedJobs[i])}</li>
-                `;
-      }
-      completedJobsHTML += '</ul>';
-      completedJobsHTML = `<div style="padding-left:25px"><button onclick="window.location.href='/clearFinishedJobs'">Clean up old jobs</button></div>` + completedJobsHTML;
-    }
-
-    var basePageHTML = `
-        <head>
-        <title>QMLBench Regression finder: ${myname}</title>
-        <meta http-equiv="Refresh" content="90">
-        <link rel="shortcut icon" href="assets/favicon.ico"/> 
-        </head>
-        <style>
-        body{
-          font: 13px "Helvetica Neue", "Lucida Grande", "Arial";
-          background: #ECE9E9 -webkit-gradient(linear, 0% 0%, 0% 100%, from(#fff), to(#ECE9E9));
-          background-repeat: no-repeat;
-          color: #555;
-          -webkit-font-smoothing: antialiased;
-        }
-        table {
-            border: 1px solid black;
-        }
-        th { 
-            text-align: center;
-            border: 1px solid black;
-            padding-left: 15px;
-            padding-right: 15px;
-        }
-        td { 
-            padding-left: 15px;
-            padding-right: 15px;
-            text-align: center;
-            border: 1px solid black;
-        }
-        .failed { color: darkred; }
-        .finished { color: darkgreen; }
-        .running { color: ForestGreen; animation: RunningBlinker 3s linear infinite;}
-        .cancelling { color: gold; animation: CancellingBlinker 2s linear infinite;}
-        .cancelled { color: gray; }
-        .staged { color: royalblue; }
-
-        @keyframes CancellingBlinker {
-            50% {
-              color: chocolate;
-            }
-        }
-
-        @keyframes RunningBlinker {
-            50% {
-              color: MediumAquamarine;
-            }
-        }
-        @keyframes lds-spin {
-            0% {
-              opacity: 1;
-              -webkit-transform: scale(1.1, 1.1);
-              transform: scale(1.1, 1.1);
-            }
-            100% {
-              opacity: 0;
-              -webkit-transform: scale(1, 1);
-              transform: scale(1, 1);
-            }
-          }
-          @-webkit-keyframes lds-spin {
-            0% {
-              opacity: 1;
-              -webkit-transform: scale(1.1, 1.1);
-              transform: scale(1.1, 1.1);
-            }
-            100% {
-              opacity: 0;
-              -webkit-transform: scale(1, 1);
-              transform: scale(1, 1);
-            }
-          }
-          .lds-spin {
-            position: relative;
-          }
-          .lds-spin div > div {
-            position: absolute;
-            width: 40px;
-            height: 40px;
-            border-radius: 50%;
-            background: #5079c4;
-            -webkit-animation: lds-spin 1s linear infinite;
-            animation: lds-spin 1s linear infinite;
-          }
-          .lds-spin div:nth-child(1) > div {
-            left: 140px;
-            top: 80px;
-            -webkit-animation-delay: -0.875s;
-            animation-delay: -0.875s;
-          }
-          .lds-spin > div:nth-child(1) {
-            -webkit-transform: rotate(0deg);
-            transform: rotate(0deg);
-            -webkit-transform-origin: 160px 100px;
-            transform-origin: 160px 100px;
-          }
-          .lds-spin div:nth-child(2) > div {
-            left: 122.42640685999999px;
-            top: 122.42640685999999px;
-            -webkit-animation-delay: -0.75s;
-            animation-delay: -0.75s;
-          }
-          .lds-spin > div:nth-child(2) {
-            -webkit-transform: rotate(45deg);
-            transform: rotate(45deg);
-            -webkit-transform-origin: 142.42640686px 142.42640686px;
-            transform-origin: 142.42640686px 142.42640686px;
-          }
-          .lds-spin div:nth-child(3) > div {
-            left: 80px;
-            top: 140px;
-            -webkit-animation-delay: -0.625s;
-            animation-delay: -0.625s;
-          }
-          .lds-spin > div:nth-child(3) {
-            -webkit-transform: rotate(90deg);
-            transform: rotate(90deg);
-            -webkit-transform-origin: 100px 160px;
-            transform-origin: 100px 160px;
-          }
-          .lds-spin div:nth-child(4) > div {
-            left: 37.57359314px;
-            top: 122.42640685999999px;
-            -webkit-animation-delay: -0.5s;
-            animation-delay: -0.5s;
-          }
-          .lds-spin > div:nth-child(4) {
-            -webkit-transform: rotate(135deg);
-            transform: rotate(135deg);
-            -webkit-transform-origin: 57.57359314px 142.42640686px;
-            transform-origin: 57.57359314px 142.42640686px;
-          }
-          .lds-spin div:nth-child(5) > div {
-            left: 20px;
-            top: 80px;
-            -webkit-animation-delay: -0.375s;
-            animation-delay: -0.375s;
-          }
-          .lds-spin > div:nth-child(5) {
-            -webkit-transform: rotate(180deg);
-            transform: rotate(180deg);
-            -webkit-transform-origin: 40px 100px;
-            transform-origin: 40px 100px;
-          }
-          .lds-spin div:nth-child(6) > div {
-            left: 37.57359314px;
-            top: 37.57359314px;
-            -webkit-animation-delay: -0.25s;
-            animation-delay: -0.25s;
-          }
-          .lds-spin > div:nth-child(6) {
-            -webkit-transform: rotate(225deg);
-            transform: rotate(225deg);
-            -webkit-transform-origin: 57.57359314px 57.57359314px;
-            transform-origin: 57.57359314px 57.57359314px;
-          }
-          .lds-spin div:nth-child(7) > div {
-            left: 80px;
-            top: 20px;
-            -webkit-animation-delay: -0.125s;
-            animation-delay: -0.125s;
-          }
-          .lds-spin > div:nth-child(7) {
-            -webkit-transform: rotate(270deg);
-            transform: rotate(270deg);
-            -webkit-transform-origin: 100px 40px;
-            transform-origin: 100px 40px;
-          }
-          .lds-spin div:nth-child(8) > div {
-            left: 122.42640685999999px;
-            top: 37.57359314px;
-            -webkit-animation-delay: 0s;
-            animation-delay: 0s;
-          }
-          .lds-spin > div:nth-child(8) {
-            -webkit-transform: rotate(315deg);
-            transform: rotate(315deg);
-            -webkit-transform-origin: 142.42640686px 57.57359314px;
-            transform-origin: 142.42640686px 57.57359314px;
-          }
-          .lds-spin div:nth-child(9) > div {
-            left: 140px;
-            top: 80px;
-            -webkit-animation-delay: 0.125s;
-            animation-delay: 0.125s;
-          }
-          .lds-spin > div:nth-child(9) {
-            -webkit-transform: rotate(360deg);
-            transform: rotate(360deg);
-            -webkit-transform-origin: 160px 100px;
-            transform-origin: 160px 100px;
-          }
-          .lds-spin {
-            width: 30px !important;
-            height: 30px !important;
-            -webkit-transform: translate(-15px, -15px) scale(0.15) translate(15px, 15px);
-            transform: translate(-15px, -15px) scale(0.15) translate(15px, 15px);
-          }
-        </style>
-
-        <body onload="checkForCancellingAndQueuedJobs()">
-        <h3>QMLBench Regression finder: ${myname}</h3>
-        <button onclick="location.href='mailto:daniel.smith@qt.io';" style="position:absolute;right:10px;top:10px;">report bug / request feature</button>
-        <br>
-        <div align="center">
-        <button onclick="window.location.href='/scheduleJob'">Schedule new job</button>
-        <button onclick="window.location.href='/refreshRemoteJobs'" style="display:${standalone ? 'none' : 'block'}">Refresh Remote Jobs</button>
-        <button onclick="window.location.href='/logs'">Browse log directory</button>
-        <button onclick="window.location.href='/recentErrors'" style="background-color:${recentErrors.length > 0 ? 'red' : ''}">Recent system errors (${recentErrors.length})</button>
-        </div>
-
-        <br>
-        <div id="jobs">
-        ${currentJobsHTML}
-        <br>
-        <h3 style="display:${standalone ? 'none' : 'block'}">Current Remote Jobs</h3>
-        ${remoteJobsHTML}
-        <br style="display:${standalone ? 'none' : 'block'}">
-        <h3>Completed Jobs</h3>
-        ${completedJobsHTML}
-        <br>
-        </div>
-        </body>
-
-        <script>
-
-        function doCancel(jobHash){
-            window.location.href='/cancelJob/?jobHash=' + jobHash;
-        }
-
-        function checkForCancellingAndQueuedJobs(){ // Refresh the page often if there are cancelling jobs or queued jobs while none are running 
-            //console.log(document.getElementById('jobs').innerHTML.search("is cancelling"));
-            if (document.getElementById('jobs').innerHTML.search("is cancelling") > -1){
-                setTimeout(function() {location.reload(true)}, 5000);
-            } else if (document.getElementById('jobs').innerHTML.search("is running") < 0 && document.getElementById('jobs').innerHTML.search("is staged") > -1){
-                setTimeout(function() {location.reload(true)}, 5000);
-            }
-        }
-        </script>
-        `;
-
-    const injectIndex = basePageHTML.indexOf('</h3>') + 5; // Find the location of the the page title. This is where we'll inject our remote host status HTML
-    var finishedRequestCount = 0;
-
-    readyEmitter.on('ready', function(){ // All done constructing the page. Send it to the requester.
-      setTimeout(function(){
-        slavesStatusHTML += '</table>';
-        res.send(basePageHTML.insert(injectIndex, slavesStatusHTML));
-      }, 200);
-    });
-
-    readyEmitter.on('finished', function(){ // This signal is emitted when a remote host responds with it's status while we build the page HTML.
-      finishedRequestCount += 1;
-      if (finishedRequestCount == slaves.length){
-        readyEmitter.emit('ready');
-      }
-    });
-
-    function addToSlaveHTML(html){
-      slavesStatusHTML += html;
-    }
-    if (slaves){
-      for (var slave in slaves){
-        axios.get(`http://${slaves[slave]}:${webPort}/?status=true`, {timeout: 2000}) // Query the remote host (or self) for status information
-          .then(response => {
-            if (hosts.findIndex(x => x.ip == response.data.ip) < 0){ // Add the friendly hostname to the list of hosts if it's not already there.
-              hosts.push({
-                hostname: response.data.hostname,
-                ip: response.data.ip,
-              });
-            }
-            addToSlaveHTML(`
-                    <tr>
-                        <td>${response.data.hostname}</td>
-                        <td style="color:green">Online</td>
-                        <td>${prettyBytes(response.data.freediskspace)}</td>
-                        <td>${moment.duration(response.data.uptime, 'seconds').format()}</td>
-                        <td>${response.data.ip}</td>
-                    </tr>
-                    `);
-            readyEmitter.emit('finished');
-          })
-          .catch(error => { // Still display the host in the table, but with a connection error.
-            var offlineMsg;
-            if (error.response){
-              offlineMsg = `${error.response.status} ${error.response.statusText}`;
-            } else {
-              offlineMsg = `${error.message}`;
-            }
-
-            addToSlaveHTML(`
-                    <tr>
-                        <td>-</td>
-                        <td style="color:red">Offline - ${offlineMsg}</td>
-                        <td>-</td>
-                        <td>-</td>
-                        <td>${error.request._options.hostname}</td>
-                    </tr>
-                    `);
-            readyEmitter.emit('finished');
-          });
-      }
-    } else {
-      addToSlaveHTML(`
-                    <tr>
-                        <td>-</td>
-                        <td>-</td>
-                        <td>-</td>
-                        <td>-</td>
-                        <td>-</td>
-                    </tr>
-                    `);
-      readyEmitter.emit('ready');
-    }
-  } else {
-    sendSlaveData();
-  }
-}
-
-
-function runServer(){
-  // magic sauce. Start the server.
-  var server = express();
-  server.use(bodyParser.json()); // to support JSON-encoded bodies
-  server.use(bodyParser.urlencoded({ // to support URL-encoded bodies
-    extended: true,
-  }));
-  server.use(fileUpload({
-    // useTempFiles : true,
-    // tempFileDir : os.tmpdir(),
-    // preserveExtension : true
-  }));
-  server.use(express.json()); // to support JSON-encoded bodies
-  server.use(express.urlencoded()); // to support URL-encoded bodies
-  server.use('/logs/', express.static(`${__dirname}/logs/`));
-  server.use('/logs/', serveIndex(`${__dirname}/logs/`, {view: 'details'}));
-  server.use('/awesomeplete', express.static('node_modules/awesomplete'));
-  server.use('/benchmarks/', express.static(`${__dirname}/QtBuild/qmlbench/benchmarks/`));
-  server.use('/assets/', express.static(`${__dirname}/assets/`));
-  server.use('/benchmarks/', serveIndex(`${__dirname}/QtBuild/qmlbench/benchmarks/`, {view: 'details'}));
-
-
-  server.all('/remoteSchedule', (req, res) => scheduleJobRequest(req, res));
-  server.all('/scheduleJob', (req, res) => schedulerPage(req, res));
-  server.all('/queryJobs', (req, res) => queryJobs(req, res));
-  server.all('/refreshRemoteJobs', (req, res) => querySlaveJobs(req, res));
-  server.all('/pushJobUpdate', (req, res) => addUpdateToQueue(req, res));
-  server.all('/cancelJob', (req, res) => cancelJob(req, res));
-  server.all('/recentErrors', (req, res) => errorsPage(req, res));
-  server.all('/clearRecentErrors', (req, res) => clearErrors(req, res));
-  server.all('/clearFinishedJobs', (req, res) => clearFinishedJobs(req, res));
-  server.all('/', (req, res) => homePage(req, res));
-
-  console.log(`Starting server... Listening on port ${webPort}`);
-
-  server.listen(webPort); // Actually spin up the server with the configurations above.
-}
-
-// Create all our logging directories
-if (!fs.existsSync(pathmodule.join(__dirname, 'logs'))){
-  fs.mkdirSync(pathmodule.join(__dirname, 'logs'));
-}
-if (!fs.existsSync(pathmodule.join(__dirname, 'logs', myname))){
-  fs.mkdirSync(pathmodule.join(__dirname, 'logs', myname));
-}
-if (!fs.existsSync(pathmodule.join(__dirname, 'logs', myname, 'results'))){
-  fs.mkdirSync(pathmodule.join(__dirname, 'logs', myname, 'results'));
-}
-
-if (!fs.existsSync(pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs'))){
-  console.log('creating log directory');
-  fs.mkdirSync(pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs'));
-}
-if (!fs.existsSync(pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs', myname))){
-  console.log('creating my own folder');
-  fs.mkdirSync(pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs', myname));
-}
-if (!fs.existsSync(pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs', myname, 'results'))){
-  console.log('creating results dir');
-  fs.mkdirSync(pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs', myname, 'results'));
-}
-
-function gatherTestNames(){
-  // Gather test names for partial match suggestions on the scheduler page. If QMLBench isn't already initialized, do it now and block the scheduler.
-
-  function loadTests(){
-    var walk = require('walk');
-
-    // Walker options
-    var walker = walk.walk('./QtBuild/qmlbench/benchmarks/', { followLinks: false });
-
-    walker.on('file', function(root, stat, next) {
-      // Add this file to the list of files
-      if (!(stat.name.toLowerCase() == 'readme.md')){
-        const file = (root + '/' + stat.name).substring(31,);
-        testNames.push(`${file.replace(/\\/g, '/')}`);
-      }
-      next();
-    });
-
-    walker.on('end', function() {
-      console.log(`Loaded ${testNames.length} tests from QMLBench.`);
-    });
-  }
-
-  if (!fs.existsSync(pathmodule.join(__dirname, 'QtBuild'))){
-    fs.mkdirSync(pathmodule.join(__dirname, 'QtBuild'));
-  }
-  if (!fs.existsSync(pathmodule.join(__dirname, 'QtBuild', 'qmlbench'))){
-    locked = true;
-    console.log("QMLBench isn't already available locally. Setting up the repositories. This may take a minute and the scheduler will not be available for jobs until this completes.");
-    console.log('WARN: Test name suggestions will be unavailable until this process completes...');
-    runningJob = {
-      testType: 'STARTUP',
-      test_hosts: [myname],
-      status: 'running',
-      jobHash: 'NO_JOBHASH',
-      title: 'Initializing Qt repositories. This may take a minute.',
-      expectedDuration: '300000',
-    };
-    var pythonexe = os.platform() == 'win32' ? 'python' : 'python3'; // Python3 for windows will simply be 'python'.
-    const testProcess = spawn(pythonexe, ['regressionFinder.py', '--setupEnv'], os.platform() == 'win32' ? {detached: false} : {detached: true});
-    runningJob.processHandle = testProcess;
-    testProcess.on('close', code => {
-      console.log('Finished setting up the repositories. Ready to accept new jobs.');
-      runningJob = {};
-      locked = false;
-      if (jobQueue.length != 0){
-        notifyJob.emit('newLocalJob'); // If we're not acting as a remote host, we can just execute immediately.
-      }
-      loadTests();
-    });
-  } else {
-    loadTests();
-  }
-}
-
-function startup(){
-// Do some pre-flight checks. Try to query the remote hosts before we even start the server listening so the data is already there, if the remote hosts are online first.
-
-  const readyEmitter = new events.EventEmitter();
-  var finishedRequestCount = 0;
-  readyEmitter.on('ready', function(){ // We're finished querying remote hosts. Start the local webserver.
-
-    if (isPrimaryHost){
-      querySlaveJobs(); // Query all remote hosts for their current work before starting the server.
-    }
-    runServer();
-  });
-
-  readyEmitter.on('finished', function(){
-    finishedRequestCount += 1;
-    if (finishedRequestCount >= slaves.length){ // Have we gotten responses from everyone?
-      readyEmitter.emit('ready');
-    }
-  });
-
-  if (slaves){
-    console.log('Querying slave IPs for hostname info...'); // Gather remote host data
-    for (var slave in slaves){
-      if (slaves[slave] == ip.address() && !nolocal){ // Add self to the list of hosts if the 'nolocal' parameter wasn't set.
-        hosts.push(
-          {
-            hostname: myname,
-            ip: ip.address(),
-          });
-        readyEmitter.emit('finished');
-      } else {
-        axios.get(`http://${slaves[slave]}:${webPort}/?status=true`, {timeout: 2000}) // Qeuery everyone else
-          .then(response => {
-            console.log(`Adding ${response.data.hostname} to the slave list`);
-            hosts.push(
-              {
-                hostname: response.data.hostname,
-                ip: response.data.ip,
-              });
-            readyEmitter.emit('finished');
-          })
-          .catch(error => {
-            console.log(`INFO: Host at ${error.request._options.hostname} appears to be offline.`);
-            readyEmitter.emit('finished');
-          });
-      }
-    }
-  } else {
-    readyEmitter.emit('finished'); // No hosts? Odd, but okay. Continue.
-  }
-
-  gatherTestNames();
-}
-
-startup(); // Call pre-flight checks, which calls the server-startup, which runs the scheduler and web interface
+'use strict';
+const { spawn } = require('child_process');
+const express = require('express');
+const fileUpload = require('express-fileupload');
+var events = require('events');
+const moment = require('moment');
+var momentDurationFormatSetup = require('moment-duration-format');
+momentDurationFormatSetup(moment);
+const os = require('os');
+const myname = os.hostname();
+var isPrimaryHost;
+let path = os.platform() === 'win32' ? 'c:' : '/';
+var pathmodule = require('path');
+var slaves = [];
+var bodyParser = require('body-parser');
+var ip = require('ip');
+const disk = require('diskusage');
+const axios = require('axios');
+const prettyBytes = require('pretty-bytes');
+const JSum = require('jsum');
+var fs = require('fs');
+const nodemailer = require('nodemailer');
+const readLastLines = require('read-last-lines');
+var validator = require('email-validator');
+const serveIndex = require('serve-index');
+var updateLockout = false;
+const tempWrite = require('temp-write');
+const Buffer = require('buffer/').Buffer
+const del = require('del');
+
+// Declarations
+var hosts = [];
+var jobQueue = [];
+var runningJob = {};
+var remoteRunningJobs = [];
+var remoteQueuedJobs = [];
+var completedJobs = [];
+var updateQueue = [];
+var recentErrors = [];
+const notifyJob = new events.EventEmitter();
+var locked = false;
+var nextErrorClearing;
+var standalone = false;
+var nolocal = false;
+var expectedDurations = {bisect: 14400000, singleCommit: 1350000, twoCommit: 2750000};
+var webPort = 8080;
+var buildCores;
+var vsDevEnv;
+var testNames = [];
+var smtpServer = 'smtp.intra.qt.io';
+var smtpPort = 25;
+
+
+// Initialize some things and check our startup parameters
+
+String.prototype.insert = function(index, string) { // Custom prototype to inject a string into another at index
+  if (index > 0)
+    return this.substring(0, index) + string + this.substring(index, this.length);
+  else
+    return string + this;
+};
+
+if (process.argv[2] == 'standalone'){ // Startup in standalone mode
+  if (process.argv[3] == 'nolocal'){
+    console.log("ERROR: \'standalone\' parameter cannot be combined with nolocal! There will be nowhere to run jobs!");
+  }
+  console.log('Starting in standalone mode. Ignoring all slave hosts passed in parameters.');
+  isPrimaryHost = true;
+  standalone = true;
+} else if (process.argv[2] == 'nolocal'){ // Startup with a local worker
+  if (process.argv[3] == undefined || process.argv[3] == 'standalone'){ // verify we have at least one slave worker
+    console.log("ERROR: Use of \'nolocal\' parameter requires addition of at least one slave IP in parameter list, and CANNOT be combined with \'standalone\'");
+    process.exit(0);
+  }
+  console.log('Starting in nolocal mode. Will not offer self as a slave host.');
+
+  isPrimaryHost = true;
+  nolocal = true;
+
+  process.argv.forEach((val, index) => {
+    if (index > 2) {
+      slaves.push(val);
+      console.log(`Got slave ${index - 2}: ${val}`);
+    }
+  });
+} else if (process.argv[2] != undefined){ // Assume all our arguments are slave worker IPs.
+  isPrimaryHost = true;
+
+  process.argv.forEach((val, index) => {
+    if (index > 1) {
+      slaves.push(val);
+      console.log(`Got slave ${index - 1}: ${val}`);
+    }
+  });
+}
+
+console.log(`I am ${isPrimaryHost ? '' : 'not '}a primary host, and I ${nolocal ? 'will not' : 'will'} run local jobs.`);
+
+if (!nolocal){
+  slaves.push(ip.address()); // List self as a slave even if we're running as master
+}
+
+// Pull optional environment variables
+
+if (process.env.HTTP_PORT){
+  webPort = process.env.HTTP_PORT;
+}
+if (process.env.BUILD_CORES){
+  buildCores = process.env.BUILD_CORES;
+}
+if (process.env.VS_DEV_ENV){
+  vsDevEnv = process.env.VS_DEV_ENV;
+}
+if (process.env.SMTP_SERVER){
+  smtpServer = process.env.SMTP_SERVER;
+}
+if (process.env.SMTP_SERVER){
+  smtpPort = Number(process.env.SMTP_PORT);
+}
+notifyJob.on('updateRemote', function(recursive){
+  // This signal should be emitted when an update from a remote worker is received.
+  if (!updateLockout){
+    updateLockout = true; // lock the work arrays from being modified by anyone else
+    updateJob(updateQueue.pop()); // process next in line
+  } else if (recursive){
+    // If recursive was passed, we got here directly from processing a previous update. In this case, the lock is not removed in order to prevent a race condition in updating.
+    updateJob(updateQueue.pop());
+  }
+});
+
+function addUpdateToQueue(req, res){
+  // Received an HTTP request to update a job. Queue the request for processing.
+  console.log(`Received Job update "${req.body.type}" for job ${req.body.job.jobHash}`);
+  updateQueue.push({job: req.body.job, type: req.body.type});
+  notifyJob.emit('updateRemote'); // try to process the request immediately.
+  if (res){
+    res.sendStatus(200); // Respond to the caller that we received the request.
+  }
+}
+
+function updateJob(job){
+  // Search our arrays based on the type of request received. When the job is found, splice it out of it's array and place the updated job we got where it needs to go.
+
+  var tempjob;
+
+  if (job.type == 'running'){
+    var index = remoteQueuedJobs.findIndex(x => x.jobHash == job.job.jobHash);
+    if (index >= 0){
+      tempjob = remoteQueuedJobs.splice(index, 1);
+      console.log(`Moved job ${tempjob[0].jobHash} from remote queued to remote running.`)
+    }
+    remoteRunningJobs.push(job.job);
+
+  } else if (job.type == 'finished' || job.type == 'cancelled'){
+    var index = remoteRunningJobs.findIndex(x => x.jobHash == job.job.jobHash);
+    if (index >= 0){
+      tempjob = remoteRunningJobs.splice(index, 1);
+      console.log(`Moved job ${tempjob[0].jobHash} from remote running to Finished.`);
+    } else {
+      index = remoteQueuedJobs.findIndex(x => x.jobHash == job.job.jobHash);
+      if (index >= 0){
+        tempjob = remoteQueuedJobs.splice(index, 1);
+      } else {
+        console.log(`Received notification that job ${job.job.jobHash} was cancelled or finished, but we can't find it. This probably means that we missed an earlier update...`);
+      }
+      console.log(`Moved job ${tempjob[0].jobHash} from remote queued to Finished.`);
+    }
+    
+    completedJobs.push(job.job);
+
+  } else if (job.type == 'cancelling'){
+    var index = remoteRunningJobs.findIndex(x => x.jobHash == job.job.jobHash);
+    if (index >= 0){
+      remoteRunningJobs[index].status = 'cancelling';
+      console.log(`Set remote running job ${remoteRunningJobs[index].jobHash} to cancelling.`);
+    } else {
+      index = remoteQueuedJobs.findIndex(x => x.jobHash == job.job.jobHash);
+      if (index >= 0){
+        remoteQueuedJobs[index].status = 'cancelling';
+        console.log(`Set remote queued job ${remoteQueuedJobs[index].jobHash} to cancelling.`);
+      } else {
+        console.log(`Received notification that job ${job.job.jobHash} was cancelled, but we can't find it. This probably means that we missed an earlier update...`);
+      }
+    }
+
+  } else {
+    console.log(`WARN: Received unknown update type ${job.type} for job ${job.job.jobHash}\nIf this job exists, it will probably be stuck now.`);
+    recentErrors.push({
+      error: `WARN: Received unknown update type ${job.type} for job ${job.job.jobHash}\nIf this job exists, it will probably be stuck now.`,
+      job: job.job,
+    });
+  }
+
+  if (updateQueue.length > 0){
+    notifyJob.emit('updateRemote', true); // Process the next update in queue if there is one, and keep the updater locked out in the meantime.
+  } else {
+    updateLockout = false; // End the lockout
+  }
+}
+
+notifyJob.on('updateRemoteJob', function(job, action){
+  // A remote host will emit this signal when a job's state changes. Send the update type and a copy of the job to the remote master.
+  console.log(`Updating master at ${job.master_host} with job status ${job.status} for ${job.jobHash}`);
+  axios.post(`http://${job.master_host}:${webPort}/pushJobUpdate`, {job: job, type: action}).then({
+  }).catch(error => {
+    // This is only a WARN because the job will continue processing locally, even if the master host isn't aware of it.
+    console.log('WARN: Unable to post job update to master host. It may be offline.');
+  });
+});
+
+notifyJob.on('JobDone', function(){
+  // A host will emit this signal when an active process completes, regardless of success
+
+  delete runningJob['processHandle']; // Clean up and delete the child_process handle from the job. Attempting to convert the process handle to a string when updating the master host would result in an error, so it has to go anyway.
+  const job = runningJob; // make a local shallow copy
+  console.log(`finished job ${job.jobHash}`);
+  runningJob = {}; // empty the currently running local job slot.
+  completedJobs.push(job); // Save the job to the finished list
+  locked = false; // unlock the job queue
+
+  if (job.remote_job){
+    notifyJob.emit('updateRemoteJob', job, 'finished'); // Notify scheduler
+    if (jobQueue.length != 0){
+      setTimeout(function() { notifyJob.emit('newLocalJob'); }, 3000); // Delay starting the next job for a few seconds to prevent race conditions and make a better UI experience.
+    }
+  } else {
+    if (jobQueue.length != 0){
+      notifyJob.emit('newLocalJob'); // If we're not acting as a remote host, we can just execute immediately.
+    }
+  }
+  if (validator.validate(job.owner)){
+    // Send an email to the job owner if they provided a valid email address
+
+    const readyEmitter = new events.EventEmitter();
+    var errorCaseLogTail = '';
+
+    function jobFormatter(job, format){
+      // Do some formatting magic
+
+      function resultText(result){
+        if(result < 0 ){
+          return `Observed improvement: ${Math.abs(result)}%`;
+        } else if (result > 0){
+          return `Observed regression: ${result}%`;
+        } else {
+          return "-- No change in score --";
+        }
+      }
+
+      const commitString = job.commit ? job.commit : job.firstCommit ? job.firstCommit.slice(0, 10) + '..' + job.secondCommit.slice(0, 10) : job.good_commit.slice(0, 10) + '..' + job.bad_commit.slice(0, 10);
+      const backendString = job.platform == 'win32' && job.openGLBackend ? ` - Open GL Backend: ${job.openGLBackend}` : '';
+      var resultString = '';
+      if (job.resultJSON && job.status != 'failed' && job.status != 'cancelled'){
+        if (job.testType == 'singleCommit') {
+
+          resultString = `Benchmark result: ${job.resultJSON.finalResults[0]}`;
+        } else if (job.testType == 'twoCommit'){
+          resultString = `Results<br>
+                                    First commit result (${job.firstCommit.slice(0, 10)}): ${job.resultJSON.finalResults[0]}<br>
+                                    Second commit result (${job.secondCommit.slice(0, 10)}): ${job.resultJSON.finalResults[1]}<br>
+                                    ${resultText(job.resultJSON.finalResults[2])}%`;
+        } else if (job.testType == 'bisect'){
+          const bisectResult = job.resultJSON.bisectResults[1].length > 1 ?
+            `Bisect returned multiple commits. This is probably due to skipped commits.<br>
+                                        Please investigate the following commits:<br>
+                                        ${job.resultJSON.bisectResults[1].slice(0,).join('<br>')}<br><br>
+                                        `
+            : `Bisect returned a single commit: ${job.resultJSON.bisectResults[1]}<br><br>`;
+
+          resultString = `Results<br>
+                                    Good commit result (${job.good_commit.slice(0, 10)}): ${job.resultJSON.baselineResults[0]}<br>
+                                    Bad commit result (${job.bad_commit.slice(0, 10)}): ${job.resultJSON.baselineResults[1]}<br>
+                                    ${resultText(job.resultJSON.baselineResults[2])}%<br>
+                                    ${bisectResult}
+                                    Tested Commits raw results:<br>
+                                    ${JSON.stringify(job.resultJSON.bisectResults[0], null, '&nbsp;').replace(/\n/g, '<br>')}<br><br>
+                                    `;
+        }
+      }
+
+      var testString = '';
+      if (job.test_name){
+        testString = `Benchmark: ${job.test_name}<br>`;
+      } else if (job.custom_benchmark_file){
+          testString = `Custom benchmark: ${job.custom_benchmark_file.name}<br>`;
+      }
+
+      if (format == 'plaintext'){
+        // Provide a plaintext email
+        return `Your QMLBench ${job.testType} job for '${job.title}' with jobHash ${job.jobHash.slice(0, 10)} on host ${job.test_hosts[0]}:\n Date: ${moment(job.timestamp).format('ddd DD MMM YYYY - HH:MM')}\nStatus: ${job.status}\n
+                Tested commit(s): ${commitString}\n\n${testString.replace(/<br>/g, '\n')}\nResults:\n${resultString.replace(/<br>/g, '\n')}${errorCaseLogTail}`;
+
+      } else {
+        // Provide a richtext/html email
+        return `
+                <style>
+                .failed { color: darkred; }
+                .finished { color: darkgreen; }
+                .cancelled { color: darkorange; }
+                </style>
+
+                <body>
+                <div class="${job.status}">
+                Your QMLBench <b>${job.testType}</b> job for ${job.title} with jobHash <font color="#454545"><b>${job.jobHash.slice(0, 10)}</b></font> on host ${job.test_hosts[0]} from ${moment(job.timestamp).format('ddd DD MMM YYYY - HH:MM')} finished with status: ${job.status}<br>
+                Tested commit(s) ${commitString}<br>
+                ${testString}<br>
+                <font color="#454545">
+                ${resultString}${backendString}
+                ${errorCaseLogTail.replace(/\n/g, '<br>')}
+                </font>
+                </div>
+                </body>
+                `;
+      }
+    }
+
+
+    readyEmitter.on('ready', function(){
+      // Only send the email once we have everything all put together.
+
+      // create reusable transporter object using the default SMTP transport
+      let transporter = nodemailer.createTransport({
+        host: smtpServer,
+        port: smtpPort,
+        secure: smtpPort === 465 ? true : false, // true for 465, false for other ports
+        tls: {
+          rejectUnauthorized: false,
+        },
+      });
+
+      // setup email data with unicode symbols
+      let mailOptions = {
+        from: '"QMLBench Bot" <qmlbenchbot@intra.qt.io>', // sender address
+        to: job.owner, // list of receivers
+        subject: `${job.status}: QMLBench ${job.testType} Job ${job.jobHash.slice(0, 10)}`, // Subject line
+        text: jobFormatter(job, 'plaintext'), // plain text body
+        html: jobFormatter(job, 'html'), // html body
+      };
+
+      // send mail with defined transport object
+      transporter.sendMail(mailOptions).catch(console.error);
+    });
+
+    if (job.status == 'failed'){
+      // Try to read the logfile and pass the last bit of that along to the job owner.
+      readLastLines.read(`logs/${job.test_hosts[0]}/${job.jobHash}.txt`, 20)
+        .then((lines) => {
+          errorCaseLogTail = 'Last 20 lines of the build/test log:\n';
+          errorCaseLogTail += lines;
+          readyEmitter.emit('ready'); // Okay, that's it! Send the email.
+        });
+    } else {
+      readyEmitter.emit('ready'); // Nothing special to do. Send the email.
+    }
+
+  }
+});
+
+function runTest(test) {
+  // Run the given test on this host
+
+  console.log('Now executing test with params:');
+  console.log(test);
+  console.log(`Job ${test.jobHash.slice(0, 10)} is expected to take ${moment.duration(test.expectedDuration).humanize()}`);
+
+  // Set arguments for regressionfinder.py to really do the build and run the test
+  var args = [];
+  var re = /\\/gi; //pass forward slashes, even on windows to avoid broken escaping.
+  if (test.testType == 'bisect'){
+    args = [].concat(['./regressionFinder.py', '--branch', test.branch, '--moduleToTest', test.module, '--knownBadRev', test.bad_commit,
+      '--knownGoodRev', test.good_commit, '--benchmark', test.test_name ? test.test_name : test.custom_benchmark_file.tempFilePath.replace(re, "/"), '--regressionTarget', test.expected_regression, '--buildCores', buildCores || os.cpus().length]);
+  } else if (test.testType == 'singleCommit'){
+    args = [].concat(['./regressionFinder.py', '--branch', test.branch, '--moduleToTest', test.module, '--testSingleCommit', test.commit,
+      '--benchmark', test.test_name ? test.test_name : test.custom_benchmark_file.tempFilePath.replace(re, "/"), '--buildCores', buildCores || os.cpus().length]);
+    if (test.firstCommitBuildOnHead){
+      args.push('--FirstBuildOnHead');
+    }
+  } else if (test.testType == 'twoCommit'){
+    args = [].concat(['./regressionFinder.py', '--branch', test.branch, '--moduleToTest', test.module,
+      '--testTwoCommit', test.firstCommit + ',' + test.secondCommit,
+      '--benchmark', test.test_name ? test.test_name : test.custom_benchmark_file.tempFilePath.replace(re, "/"), '--buildCores', buildCores || os.cpus().length]);
+    if (test.firstCommitBuildOnHead){
+      args.push('--FirstBuildOnHead');
+    }
+    if (test.secondCommitBuildOnHead){
+      args.push('--SecondBuildOnHead');
+    }
+  } else {
+    console.log(`ERROR: Unrecognized test type of ${test.testType}! This should not happen.`);
+  }
+
+  args.push('--jobName', test.jobHash); // Give our job a name so it can write logs to disk with a unique ID.
+
+  if (vsDevEnv){
+    args.push('--VSDevEnv', vsDevEnv); // Pass along a custom VS Developer Environment if the user set it when starting the server.
+  }
+  if (test.openGLBackend && test.platform == 'win32'){
+    args.push('--OpenGLBackend', test.openGLBackend); // Set the OpenGL backend override if selected in the web interface and this system is Windows.
+  }
+
+  if (test.environment){
+    if (Array.isArray(test.environment)){
+      args.push('--environment', test.environment.join(','));
+    } else {
+      args.push('--environment', test.environment);
+    }
+
+  }
+
+  var pythonexe = os.platform() == 'win32' ? 'python' : 'python3'; // Python3 for windows will simply be 'python'.
+  console.log('Launching ' + pythonexe + ' ' + args.join(' '));
+  test.status = 'running';
+  const testProcess = spawn(pythonexe, args, os.platform() == 'win32' ? {detached: false} : {detached: true});
+
+  const logFile = fs.createWriteStream(`logs/${myname}/${test.jobHash}.txt`); // dump process console output to file.
+
+  // Unfortunately I can't find a way to dump both stdout and stderr at the same time to the same file, so print statements in the python script will always appear at the end of the log in their own section.
+  // If you read this and know how to fix it, let me know @ daniel.smith[.at.]qt.io
+  testProcess.stdout.on('data', (data) => {
+    logFile.write(data); // log to file
+  });
+  testProcess.stderr.on('data', (data) => {
+    logFile.write(data); // log to file
+  });
+
+  testProcess.on('close', code => {
+    // The process finished-- Do some tasks
+
+    console.log(`Process for job ${test.jobHash.slice(0, 10)} exited with code ${code}`);
+    test.logURL = `http://QMLBench.intra.qt.io/logs/${myname}/${test.jobHash}.txt`;
+    test.finishTime = Date.now();
+    test.actualDuration = test.finishTime - test.startTime;
+    if (!isPrimaryHost){ // All logs should get copied to the scheduler host.
+      const { exec } = require('child_process');
+      exec(`${os.platform() == 'win32' ? 'copy' : 'cp'} ${pathmodule.join(__dirname, 'logs', myname, test.jobHash + '.txt')} ${pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs', myname)}`, (error, stdout, stderr) => {
+        if (error) {
+          console.error(`Failed to copy the log file: ${error}`); // log failed file copies but don't error out.
+        }
+      });
+      exec(`${os.platform() == 'win32' ? 'copy' : 'cp'} ${pathmodule.join(__dirname, 'logs', myname, 'results', test.jobHash + '.json')} ${pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs', myname, 'results')}`, (error, stdout, stderr) => {
+        if (error) {
+          console.error(`Failed to copy the log file: ${error}`); // log failed file copies but don't error out.
+        }
+      });
+    }
+    if (test.status != 'cancelling'){
+      // If the test is in cancelling status, it means that someone pressed the cancel button. So provided it exited naturally, try to get test results from the run, even if the job failed and hard-exited.
+      try {
+        const f = fs.readFileSync(`logs/${test.test_hosts[0]}/results/${test.jobHash}.json`);
+        test.resultJSON = JSON.parse(f);
+      } catch (err){
+        console.log('ERROR: Failed to read the results file. Maybe the test failed to finish...');
+        // console.log(err);
+      }
+
+      if (code != 0 || code == 'null') {
+        console.log(test.title, 'Failed: with code', code);
+        test.status = 'failed';
+      } else {
+        console.log(test.title, 'finished with code ', code);
+        test.status = 'finished';
+        // Update the new expected durations for this server/job type
+        console.log(`Stored expected:${expectedDurations[test.testType]}\nActual Duration: ${moment.duration(test.actualDuration, 'ms').humanize()}\nNew Duration for ${test.testType}: ${Math.ceil((expectedDurations[test.testType] + test.actualDuration) / 2)}`);
+        expectedDurations[test.testType] = Math.ceil((expectedDurations[test.testType] + test.actualDuration) / 2);
+      }
+    } else {
+      // The job was in cancelling status when it exited. Set it to cancelled now.
+      test.status = 'cancelled';
+    }
+
+    if(test.custom_benchmark_file){
+      if(test.custom_benchmark_file.tempFilePath){
+        (async () => {
+          var re = /\\/gi;
+          // can't use path.normalize here. del@5.0.0 module doesn't seem to work with windows style paths at the moment.
+          const deletedPaths = await del([pathmodule.dirname(test.custom_benchmark_file.tempFilePath).replace(re, "/")], {force: true});
+          console.log('Deleted files and directories:\n', deletedPaths.join('\n'));
+      })(error => {console.log(error)});
+      }
+    }
+
+    notifyJob.emit('JobDone'); // Emit the JobDone signal to do some more tasks and prep for the next job in queue.
+
+    // saveTests(); //Backup the JSON file on disk for the status web page. // Not implemented!
+  });
+  return testProcess;
+}
+
+
+notifyJob.on('newLocalJob', function(){
+  // We received a job for this server to run. If we're not locked out due to another current process, execute it immediately.
+  if (!locked){
+    locked = true; // Lock out other jobs from trying to run.
+    runningJob = jobQueue.pop(); // Grab a job from the queue
+    runningJob.status = 'running';
+    runningJob.startTime = Date.now(); // Start the timer so we can track how long these things take...
+    if (runningJob.remote_job){
+      notifyJob.emit('updateRemoteJob', runningJob, 'running'); // Notify the scheduler that we've moved this job into running status.
+    }
+    setTimeout(function() { runningJob.processHandle = runTest(runningJob); }, 1000); // set the process handle and hand the job over to the test runner
+  }
+});
+
+function clearErrors(req, res){
+  // Clear system errors on the web interface when running in scheduler mode.
+  recentErrors.length = 0;
+  clearTimeout(nextErrorClearing);
+
+  if (res){
+    res.redirect('/');
+  }
+}
+
+function clearFinishedJobs(req, res){
+  // Clear the list of completed jobs
+  completedJobs = [];
+
+  if (res){
+    res.redirect('/');
+  }
+}
+
+function errorsPage(req, res){
+  // Display stored system errors.
+
+  var pageHTML = `
+    <head>
+    <title>Benchmark System Errors</title>
+    <link rel="shortcut icon" href="assets/favicon.ico"/> 
+    </head>
+    <style>
+    body{
+      font: 13px "Helvetica Neue", "Lucida Grande", "Arial";
+      background: #ECE9E9 -webkit-gradient(linear, 0% 0%, 0% 100%, from(#fff), to(#ECE9E9));
+      background-repeat: no-repeat;
+      color: #555;
+      -webkit-font-smoothing: antialiased;
+    }
+    </style>
+    <button onclick="window.location.href='/'">Back</button>
+    <button onclick="window.location.href='/clearRecentErrors'">Clear all errors</button>
+    <br>
+    `;
+  if (recentErrors){
+    for (var i in recentErrors){
+      pageHTML += `
+            <h3>Error ${recentErrors[i].error} on ${recentErrors[i].job.test_hosts[0]}</h3>
+            <pre>${JSON.stringify(recentErrors[i].job, null, 4)}</pre>
+            <br>
+            `;
+    }
+  } else {
+    pageHTML = pageHTML + `
+        <h2>💤 Things are looking pretty quiet around here... 💤</h2>
+        `;
+  }
+
+  res.send(pageHTML);
+}
+
+notifyJob.on('finishedScheduling', function(req, res){
+  // Redirect the scheduler back to the homepage once we've sent out and received a response from all job requests.
+  setTimeout(function() { res.redirect('/'); }, 800, res);
+});
+
+function scheduleJobRequest(req, res){
+  // This is triggered in two ways [1. When a user presses the 'schedule' button in the web interface] of [2. When a remote server receives a request to schedule a new job.]
+  const readyEmitter = new events.EventEmitter();
+
+  console.log('Received request to schedule new job:');
+  console.log(req.body); // Dump the job request to the console for logging and diagnostic purposes.
+  console.log(req.files);
+
+  if (!Array.isArray(req.body.test_hosts)){ // Magic. This allows us to use the same endpoint for scheduling a job on both one and multiple hosts.
+    req.body.test_hosts = [req.body.test_hosts];
+  }
+
+  var numJobsToSchedule = req.body.test_hosts.length; // Count how many times we need to schedule. Used for knowing when to redirect the scheduler browser
+  var jobsScheduled = 0;
+
+  notifyJob.on('jobScheduled', function(req, res){
+    jobsScheduled += 1;
+    if (jobsScheduled == numJobsToSchedule){ // Received responses from all job schedule requests.
+      notifyJob.emit('finishedScheduling', req, res);
+    }
+  });
+
+  readyEmitter.on('ready', function(title){
+    // The ready signal is fired when we've finished polling codereview for a job title from one of the commits.
+
+    for (var i in req.body.test_hosts){ // Create a job for each test host
+      const job = {
+        timestamp: Date.now(),
+        testType: req.body.testType,
+        platform: req.body.platform,
+        owner: req.body.owner,
+        title: title,
+        status: req.body.status,
+        branch: req.body.branch,
+        module: req.body.module,
+        commit: req.body.commit,
+        firstCommitBuildOnHead: req.body.firstCommitBuildOnHead,
+        secondCommitBuildOnHead: req.body.secondCommitBuildOnHead,
+        firstCommit: req.body.firstCommit,
+        secondCommit: req.body.secondCommit,
+        bad_commit: req.body.bad_commit,
+        good_date: req.body.good_date,
+        good_commit: req.body.good_commit,
+        openGLBackend: req.body.openGLBackend,
+        test_name: req.body.test_name,
+        custom_benchmark_file: req.files ? req.files.custom_benchmark_file : req.body.custom_benchmark_file,
+        expected_regression: req.body.expected_regression,
+        test_all: req.body.test_all,
+        test_hosts: Array.isArray(req.body.test_hosts) ? [req.body.test_hosts[i]] : [req.body.test_hosts],
+        environment: req.body.environment,
+        remote_job: req.body.remote_job,
+        master_host: req.body.master_host,
+      };
+
+      //Dump the data from the custom file to disk as a temp file.
+      if (job.custom_benchmark_file){
+        // Check to see if we have a properly formed data buffer. If not, re-create it from the object.
+        if (! (job.custom_benchmark_file.data.constructor && job.custom_benchmark_file.data.constructor.isBuffer && job.custom_benchmark_file.data.constructor(job.custom_benchmark_file.data))){
+          job.custom_benchmark_file.data = Buffer.from(job.custom_benchmark_file.data.data)
+        }
+      }
+
+      if (job.owner == ''){
+        job.owner = 'default'; // Set a display name if the email address field wasn't filled.
+      }
+
+      if (req.body.test_hosts[i] == myname){
+        // Either we're a remote host and received a job, or we're the scheduler and have a job thread available.
+        console.log('executing test locally.');
+        if (job.custom_benchmark_file){
+          job.custom_benchmark_file['tempFilePath'] = tempWrite.sync(job.custom_benchmark_file.data, `${job.custom_benchmark_file.name}`);
+        }
+        const jobHash = JSum.digest(job, 'SHA256', 'hex'); // Create a unique hash for this job
+        console.log(`Hash for this job: ${jobHash}`);
+        job['jobHash'] = jobHash;
+
+        if (runningJob.jobHash === jobHash || jobQueue.findIndex(x => x.jobHash == jobHash) >= 0){
+          // This should never happen. It could only really happen if the scheduler sends the same exact job twice (including the timestamp on it)
+          console.log('Returning error. Job already exists.');
+          if (!isPrimaryHost){
+            res.status(500).send({ error: 'jobExists', job: job });
+            continue;
+          } else {
+            recentErrors.push({ error: 'jobExists', job: job });
+            nextErrorClearing = setTimeout(clearErrors, 900000);
+            notifyJob.emit('jobScheduled', req, res);
+            continue;
+          }
+        }
+        job.platform = os.platform();
+        job.status = 'staged'; // All jobs go to the queue first. No cutting line, even if there isn't one.
+        job.expectedDuration = expectedDurations[job.testType]; // Each server keeps an expected duration for each type of job so we can give more accurate estimates.
+        jobQueue.push(job);
+        if (job.remote_job){
+          res.send(job); // Send the completed job object to the scheduler.
+        } else {
+          notifyJob.emit('jobScheduled', req, res); // We're running this on the scheduler itself, need to send a different kind of response.
+        }
+        notifyJob.emit('newLocalJob'); // Notify this server that there's a new job and execute if the queue's empty.
+      } else {
+        // We're processing the meta job request and there's a host that isn't the scheduler. Mark it as remote and forward it.
+        job['remote_job'] = true;
+        job['master_host'] = ip.address(); // Provide some info about where to send future job updates.
+        // delete job['custom_benchmark_file'];
+        // data = new FormData();
+        // data.extend(job);
+        // data.append('custom_benchmark_file', )
+        
+        
+        console.log(`Posting job to http://${hosts[hosts.findIndex(x => x.hostname == req.body.test_hosts[i])].ip}:${webPort}/remoteSchedule`);
+        axios.post(`http://${hosts[hosts.findIndex(x => x.hostname == req.body.test_hosts[i])].ip}:${webPort}/remoteSchedule`, job).then(
+          response => {
+            console.log('Sent the job to remote host and got a response. Looks good.');
+            remoteQueuedJobs.push(response.data);
+            notifyJob.emit('jobScheduled', req, res); // +1 to scheduled jobs
+          }).catch(error => {
+          console.log(error.response.data);
+          if (error.response.data.error == 'jobExists'){
+            recentErrors.push(error.response.data);
+            nextErrorClearing = setTimeout(clearErrors, 900000);
+          } else {
+            console.log(error);
+          }
+          notifyJob.emit('jobScheduled', req, res); // +1 to scheduled jobs
+        });
+      }
+    }
+  });
+
+
+  if (!req.body.title){
+    // Initialize the job with a title pulled from one of the commits provided.
+    const sha = req.body.commit ? req.body.commit : req.body.firstCommit ? req.body.firstCommit : req.body.bad_commit;
+
+    axios.get(`https://codereview.qt-project.org/changes/?q=commit:${sha}`, {timeout: 2000}).then(
+      response => {
+        const title = JSON.parse(response.data.slice(4,))[0].subject;
+        // console.log(`Found sha ${sha} on codereview with title '${title}'.`)
+        readyEmitter.emit('ready', title);
+      }).catch(error => {
+      // console.log(error);
+      const title = 'Unknown Job';
+      readyEmitter.emit('ready', title); // Go ahead and schedule jobs without a title. Hopefully it works, but good luck.
+    });
+  } else {
+    readyEmitter.emit('ready', req.body.title); // We got a title, schedule the jobs using this.
+  }
+
+}
+
+function queryJobs(req, res){
+  // We're a remote host and the scheduler asked us for our jobs. Give it to 'em!
+  console.log('Sending current running and queued job list...');
+  console.log(`${JSON.stringify({runningJob: runningJob.jobHash, queuedJobs: jobQueue})}`);
+
+  // Create a temporary job object that we'll send because stringifying a process handle of the running process will fail in a very nasty way, so just don't include it. It's only useful on the host running the process anyhow.
+  const tempRunningJob = {
+    jobHash: runningJob.jobHash,
+    platform: runningJob.platform,
+    startTime: runningJob.startTime,
+    expectedDuration: runningJob.expectedDuration,
+    timestamp: runningJob.timestamp,
+    testType: runningJob.testType,
+    owner: runningJob.owner,
+    title: runningJob.title,
+    status: runningJob.status,
+    branch: runningJob.branch,
+    module: runningJob.module,
+    commit: runningJob.commit,
+    firstCommitBuildOnHead: runningJob.firstCommitBuildOnHead,
+    secondCommitBuildOnHead: runningJob.secondCommitBuildOnHead,
+    firstCommit: runningJob.firstCommit,
+    secondCommit: runningJob.secondCommit,
+    bad_commit: runningJob.bad_commit,
+    good_date: runningJob.good_date,
+    good_commit: runningJob.good_commit,
+    openGLBackend: req.body.openGLBackend,
+    test_name: runningJob.test_name,
+    custom_benchmark_file: runningJob.custom_benchmark_file,
+    expected_regression: runningJob.expected_regression,
+    test_all: runningJob.test_all,
+    test_hosts: runningJob.test_hosts,
+    environment: runningJob.environment,
+    remote_job: runningJob.remote_job,
+    master_host: runningJob.master_host,
+  };
+
+  res.send({runningJob: tempRunningJob, queuedJobs: jobQueue});
+}
+
+function querySlaveJobs(req, res){
+  // Trigger this to dump the scheduler's local cache of it's remote hosts's jobs and ask them what they have instead. This helps deal with any network hiccups that might have caused a missed update.
+
+  const readyEmitter = new events.EventEmitter();
+  var countSlaveQueries = 0; // How many queries do we need to do before we refresh the scheduler's page?
+
+  readyEmitter.on('finished', function(){
+    countSlaveQueries += 1;
+    if (countSlaveQueries >= slaves.length){ // have we gotten all our queries back?
+      readyEmitter.emit('ready');
+    }
+  });
+
+  readyEmitter.on('ready', function(){
+    if (res){
+      res.redirect('/'); // Redirect the scheduler page back home and force a refresh.
+    }
+  });
+
+  console.log('Clearing remote job listing cache.');
+  remoteRunningJobs = [];
+  remoteQueuedJobs = [];
+  console.log('Gathering remote host jobs.');
+  for (var i in slaves){
+    if (slaves[i] != ip.address()){
+      axios.get(`http://${slaves[i]}:${webPort}/queryJobs`, {timeout: 2000}).then(
+        response => {
+          console.log('Received remote jobs. Updating local lists...');
+          if (Object.entries(response.data.runningJob).length > 2){ // Check to make sure the response isn't a literal empty object such as '{}'
+            if (remoteRunningJobs.findIndex(x => x.jobHash == response.data.runningJob.jobHash) < 0){ // make sure we didn't already receive this same job. It'd be odd, but may as well protect against it.
+              remoteRunningJobs.push(response.data.runningJob);
+            }
+          }
+
+          if (response.data.queuedJobs.length > 0){ // Should be a list of jobs
+            for (var j in response.data.queuedJobs){
+              if (remoteQueuedJobs.findIndex(x => x.jobHash == response.data.queuedJobs[j].jobHash) < 0){ // make sure we didn't already receive this same job. It'd be odd, but may as well protect against it.
+                remoteQueuedJobs.push(response.data.queuedJobs[j]);
+              }
+            }
+          }
+          readyEmitter.emit('finished'); // +1 to the number of remote hosts we've queried.
+        }
+      ).catch(error => {
+        console.log(`WARN: Failed to query ${slaves[i]} for jobs`); // Just a WARN because it's not fatal that one of our remote hosts is offline for a minute.
+        readyEmitter.emit('finished');
+      });
+    }
+  }
+}
+
+notifyJob.on('cancel', function(job, req, res){
+  // Cancel a locally running job
+  if (job.status == 'running'){ // If it's a running job we need to try and kill the process.
+    job.status = 'cancelling';
+
+    if (job.remote_job){
+      console.log(`Preparing to send update to master host about cancelling jobHash ${job.jobHash}`);
+      notifyJob.emit('updateRemoteJob', job, 'cancelling'); // update the scheduler to let it know a job is in cancelling status.
+    }
+
+    // Use platform specific method to kill the entire process tree. Unfortunately, we just have to trust here.
+    if (os.platform() === 'win32'){
+      console.log(`Trying to kill windows job with jobHash ${job.jobHash}`);
+      var spawn = require('child_process').spawn;
+      spawn('taskkill', ['/pid', job.processHandle.pid, '/F', '/T']);
+    } else {
+      console.log(`Trying to kill linux job with jobHash ${job.jobHash}`);
+      process.kill(-job.processHandle.pid);
+    }
+    delete job.processHandle; // Clean up the process handle since it's dead anyway. This makes it easier to stringify the job when sending updates to the scheduler, too.
+
+  } else {
+    // Not a running job, must be queued.
+    console.log(`Moving queued job to completed-cancelled with jobHash ${job.jobHash}`);
+    job.status = 'cancelled'; // No need to go through the work of sending an update for "cancelling", just go straight to "cancelled"
+    completedJobs.push(job);
+    jobQueue.splice(jobQueue.findIndex(x => x.jobHash == job.jobHash)); // pop the queued job out of the queue.
+
+    if (job.remote_job){
+      console.log(`Preparing to send update to master host about cancelled jobHash ${job.jobHash}`);
+      notifyJob.emit('updateRemoteJob', job, 'cancelled');
+    }
+  }
+
+  setTimeout(function() { res.redirect('/'); }, 500, res);
+});
+
+function cancelJob(req, res){
+  // A user pressed the "cancel" button in the web interface on a job
+
+  console.log(`Received cancel request for job ${req.query.jobHash}`);
+
+  // Do some logic to find out which array the job lives in. Forward the cancel request to a handler based on that.
+  if (req.query.jobHash === runningJob.jobHash){
+    console.log('Cancelling local running job');
+    notifyJob.emit('cancel', runningJob, req, res);
+
+  } else if (jobQueue[jobQueue.findIndex(x => x.jobHash == req.query.jobHash)]){
+    console.log('Cancelling local queued job');
+    notifyJob.emit('cancel', jobQueue[jobQueue.findIndex(x => x.jobHash == req.query.jobHash)], req, res);
+
+  } else if (remoteRunningJobs[remoteRunningJobs.findIndex(x => x.jobHash == req.query.jobHash)]){
+    console.log('Cancelling remote running job');
+    var remoteIP = hosts[hosts.findIndex(x => x.hostname == remoteRunningJobs[remoteRunningJobs.findIndex(x => x.jobHash == req.query.jobHash)].test_hosts[0])].ip;
+    axios.get(`http://${remoteIP}:${webPort}/cancelJob?jobHash=${req.query.jobHash}`, {timeout: 2000})
+      .then(response => {
+        setTimeout(function() { res.redirect('/'); }, 500, res); // Delay the redirect to the remote host might have time to update the job before we refresh the page for the user.
+      }).catch(error => {
+        console.log(`ERROR: Failed to post cancel request to ${remoteIP}`);
+        recentErrors.push({
+          error: `ERROR: Failed to post cancel request for job hash ${req.query.jobHash.slice(0, 10)}`,
+          job: remoteRunningJobs[remoteRunningJobs.findIndex(x => x.jobHash == req.query.jobHash)],
+        });
+        res.redirect('/');
+      });
+
+  } else if (remoteQueuedJobs[remoteQueuedJobs.findIndex(x => x.jobHash == req.query.jobHash)]){
+    console.log('Cancelling remote queued job');
+    var remoteIP = hosts[hosts.findIndex(x => x.hostname == remoteQueuedJobs[remoteQueuedJobs.findIndex(x => x.jobHash == req.query.jobHash)].test_hosts[0])].ip;
+    axios.get(`http://${remoteIP}:${webPort}/cancelJob?jobHash=${req.query.jobHash}`, {timeout: 2000})
+      .then(response => {
+        setTimeout(function() { res.redirect('/'); }, 500, res); // Delay the redirect to the remote host might have time to update the job before we refresh the page for the user.
+      }).catch(error => {
+        console.log(`ERROR: Failed to post cancel request to ${remoteIP}`);
+        recentErrors.push({
+          error: `ERROR: Failed to post cancel request for job hash ${req.query.jobHash.slice(0, 10)}`,
+          job: remoteRunningJobs[remoteRunningJobs.findIndex(x => x.jobHash == req.query.jobHash)],
+        });
+        res.redirect('/');
+      });
+
+  } else {
+    console.log("not cancelling anything. Couldn't find the job hash?"); // This shouldn't happen.
+    res.redirect('/');
+  }
+}
+
+function schedulerPage(req, res) {
+  // Display the scheduler page to the web user
+
+  if (hosts.length === 0){
+    axios.get(`http://${ip.address()}:${webPort}`), {timeout: 2000}; // Load the homepage of myself in case someone loaded the scheduler immediately after startup. This forces a refresh of remote slaves.
+  }
+
+  // Populate the list of test hosts
+  var hostSelectionComboBox = `<select id="test_hosts" name="test_hosts" multiple required size=${hosts.length}>`;
+  if (hosts.length === 0){
+    hostSelectionComboBox = '<option value="" disabled>No hosts - Please refresh</option>';
+  } else {
+    for (var i in hosts){
+      hostSelectionComboBox += `<option value="${hosts[i].hostname}">${hosts[i].hostname}</option>`;
+    }
+    hostSelectionComboBox += '</select>';
+  }
+
+  if (!testNames){
+    gatherTestNames();
+  }
+
+  // Send the HTML page.
+  res.send(`
+    <head>
+    <title>Schedule a new job</title>
+    <link rel="stylesheet" href="awesomeplete/awesomplete.css" />
+    <link rel="shortcut icon" href="assets/favicon.ico"/> 
+    </head>
+
+    <style>
+    body{
+      font: 13px "Helvetica Neue", "Lucida Grande", "Arial";
+      background: #ECE9E9 -webkit-gradient(linear, 0% 0%, 0% 100%, from(#fff), to(#ECE9E9));
+      background-repeat: no-repeat;
+      color: #555;
+      -webkit-font-smoothing: antialiased;
+    }
+    .tooltip{display:inline;position:relative}
+    .tooltip:hover{text-decoration:none}
+
+    .tooltip:hover:after{
+    background:#111;
+    background:rgba(0,0,0,.8);
+    border-radius:5px;
+    bottom:18px;
+    width: 300px;
+    color:#fff;
+    content:attr(help);
+    display:block;
+    left:50%;
+    padding:5px 15px;
+    position:absolute;
+    white-space:pre-wrap;
+    z-index:98
+    }
+    .tooltip:hover:before{
+        border:solid;
+        border-color:#111 transparent;
+        border-width:6px 6px 0 6px;
+        bottom:12px;
+        content:"";
+        display:block;
+        left:75%;
+        position:absolute;
+        z-index:99
+    }
+
+    .BasicButton {
+        -moz-box-shadow:inset 0px 1px 3px 0px #91b8b3;
+        -webkit-box-shadow:inset 0px 1px 3px 0px #91b8b3;
+        box-shadow:inset 0px 1px 3px 0px #91b8b3;
+        background:-webkit-gradient(linear, left top, left bottom, color-stop(0.05, #768d87), color-stop(1, #6c7c7c));
+        background:-moz-linear-gradient(top, #768d87 5%, #6c7c7c 100%);
+        background:-webkit-linear-gradient(top, #768d87 5%, #6c7c7c 100%);
+        background:-o-linear-gradient(top, #768d87 5%, #6c7c7c 100%);
+        background:-ms-linear-gradient(top, #768d87 5%, #6c7c7c 100%);
+        background:linear-gradient(to bottom, #768d87 5%, #6c7c7c 100%);
+        filter:progid:DXImageTransform.Microsoft.gradient(startColorstr='#768d87', endColorstr='#6c7c7c',GradientType=0);
+        background-color:#768d87;
+        -moz-border-radius:5px;
+        -webkit-border-radius:5px;
+        border-radius:5px;
+        border:1px solid #566963;
+        display:inline-block;
+        cursor:pointer;
+        color:#ffffff;
+        font-family:Arial;
+        font-size:12px;
+        font-weight:bold;
+        padding:3px 8px;
+        text-decoration:none;
+        text-shadow:0px -1px 0px #2b665e;
+    }
+    .BasicButton:hover {
+        background:-webkit-gradient(linear, left top, left bottom, color-stop(0.05, #6c7c7c), color-stop(1, #768d87));
+        background:-moz-linear-gradient(top, #6c7c7c 5%, #768d87 100%);
+        background:-webkit-linear-gradient(top, #6c7c7c 5%, #768d87 100%);
+        background:-o-linear-gradient(top, #6c7c7c 5%, #768d87 100%);
+        background:-ms-linear-gradient(top, #6c7c7c 5%, #768d87 100%);
+        background:linear-gradient(to bottom, #6c7c7c 5%, #768d87 100%);
+        filter:progid:DXImageTransform.Microsoft.gradient(startColorstr='#6c7c7c', endColorstr='#768d87',GradientType=0);
+        background-color:#6c7c7c;
+    }
+    .BasicButton:active {
+        position:relative;
+        top:1px;
+    }    
+
+    </style>
+
+
+    <body>
+    <button onclick="window.location.href='/'">Back</button>
+    <button onclick="location.href='mailto:daniel.smith@qt.io';" style="position:absolute;right:10px;top:10px;">report bug / request feature</button>
+    <br>
+    <table align="center">
+    <tr>
+        <td align="center" colspan="3"><h2>What would you like to schedule?</h2></td>
+    </tr>
+    <tr>
+        <td><button onclick="BasicScheduler()">Automated bisect on merged commits</button></td>
+        <td><button onclick="SingleCommitScheduler()">Single Commit/patch</button></td>
+        <td><button onclick="TwoCommitScheduler()">Two Commit/patch Compare</button></td>
+    </tr>
+    </table>
+    <br>
+
+    <div id="basicScheduler" style="display:none" align="center">
+    <form id="basicSchedulerForm" action="/remoteSchedule/" method="post" encType="multipart/form-data">
+    <input type="hidden" name="testType" value="bisect"/>
+    <label for="branch">Qt major branch:</label>
+    <input id="branch" type="text" name="branch" style="width:50;" required placeholder="5.12"><br>
+    <label for="module">Qt module to bisect: </label>
+    <select id="module" name="module" style="width:200;" required placeholder="qtbase">
+    <option value="qtbase">QtBase</option>
+    <option value="qtdeclarative">QtDeclarative</option>
+    <option value="qtquickcontrols">QtQuickControls</option>
+    <option value="qtquickcontrols2">QtQuickControls2</option>
+    <option value="qtsvg">QtSvg</option>
+    <option value="qtquick3d">QtQuick3d</option>
+    </select>
+    <br>
+    <label for="openGLBackend">OpenGL Backend: </label>
+    <select id="openGLBackend" name="openGLBackend" style="width:100;" placeholder="desktop">
+    <option value="desktop">desktop</option>
+    <option value="angle">Angle</option>
+    <option value="software">software</option>
+    </select>
+    <b help="Only affects Windows. Override the OpenGL rendering backend to use for these tests. Leave default of \'desktop\' if unsure." class="tooltip">(What's this?)</b>
+    <br>
+    <hr style="width:20%">
+    <label for="good_commit">Known good commit: </label><br>
+    <input id="good_commit" type="text" name="good_commit" style="width:330; "value="" required placeholder="becac559b713dfadd27198d683fde70551a543c8"><br>
+    <p><b>Important:</b> Good (older) commit must be an ancestor to the bad commit</p>
+    <p><b>Important:</b> Commit must have been integrated successfully in COIN!</p>
+    <hr style="width:20%">
+    <label for="bad_commit">Known bad commit: </label><br>
+    <input id="bad_commit" type="text" name="bad_commit" style="width:330; "value="" placeholder="Leave blank to use HEAD"><br>
+    <p><b>Important:</b> Commit must have been integrated successfully in COIN!</p>
+    <hr style="width:20%">
+    <label for="test_name"><b>1)</b> Select an existing QMLBench test: </label><b help="Start typing a test name to benchmark from the QMLBench test suite to provide suggestions." class="tooltip">(Help)</b>  <a href='/benchmarks' class="BasicButton">Browse benchmarks</a><br>
+    <input class="awesomplete" id="test_name" type="text" data-list="${testNames}" style="width:600;" name="test_name" required placeholder="f.ex: auto/animations/comparison/moving-images-animations.qml"><br>
+    <p><b>2)</b> Or upload custom <b>.qml</b> benchmark file:</p>
+    <div id="custom_benchmark_file_label" style="width:15%;padding:2px">No custom file set</div>
+    <input id="custom_benchmark_file" name="custom_benchmark_file" type="file" accept=".qml" hidden onchange="updateFileLabel(this.files)" /><br>
+    <input id='benchmarkButtonBisect' type='button' value='Upload custom benchmark' /><br>
+    <hr style="width:20%">
+    <label for="expected_regression">Expected regression, in percentage: </label>
+    <input id="expected_regression" type="number" min="2" max="100" style="width:40;" name="expected_regression" value="" placeholder="20">%<br>
+    <p><b>Note:</b> Be conservative with the regression target.<br>It is not recommended to search for regressions below 5% due to test noise.</p>
+    <hr style="width:20%">
+    <label for="test_hosts">Run job on host:</label><br>
+    ${hostSelectionComboBox}
+    <hr style="width:20%">
+    <label for="environment">Set custom environment variables on hosts:</label><br>
+    <table>
+        <th>Variables</th><th>Variable name</th><th>Variable Value</th>
+        <tr valign="top" height=0%>
+            <td rowspan=2>
+                <select id="environment" name="environment" size=2 style="min-width:100px;overflow:hidden" multiple></select>
+            </td>
+            <td><input id="environmentName" name="environmentName" type="text" style="width:200"></input></td>
+            <td><input id="environmentValue" name="environmentValue" type="text" style="width:200"></input></td>
+        </tr>
+        <tr valign="top">
+        <td align="center" colspan=2 style="padding-top:5px"><button type="button" onclick="AddEnvironmentVariable()">Add Environment Variable</button></td>
+    </table>
+    <hr style="width:20%">
+    <label for="owner">Enter email to receive notices about this job: </label><br>
+    <input id="owner" type="email" style="width:400;" name="owner" placeholder="someone@qt.io"><br>
+    <br>
+    <input type="submit" id="submit" disabled value="Schedule Bisect">
+    </form>
+    </div>
+    </div>
+
+    <div id="singleCommitScheduler" style="display:none" align="center">
+    <form id="singleCommitSchedulerForm" action="/remoteSchedule/" method="post" encType="multipart/form-data">
+    <input type="hidden" name="testType" value="singleCommit"/>
+    <label for="branch">Qt major branch:</label>
+    <input id="branch" type="text" name="branch" style="width:50;" required placeholder="5.12"><br>
+    <label for="module">Qt module to test: </label>
+    <select id="module" name="module" style="width:200;" required placeholder="qtbase">
+    <option value="qtbase">QtBase</option>
+    <option value="qtdeclarative">QtDeclarative</option>
+    <option value="qtquickcontrols">QtQuickControls</option>
+    <option value="qtquickcontrols2">QtQuickControls2</option>
+    <option value="qtsvg">QtSvg</option>
+    <option value="qtquick3d">QtQuick3d</option>
+    </select>
+    <br>
+    <label for="openGLBackend">OpenGL Backend: </label>
+    <select id="openGLBackend" name="openGLBackend" style="width:100;" placeholder="desktop">
+    <option value="desktop">desktop</option>
+    <option value="angle">Angle</option>
+    <option value="software">software</option>
+    </select>
+    <b help="Only affects Windows. Override the OpenGL rendering backend to use for these tests. Leave default of \'desktop\' if unsure." class="tooltip">(What's this?)</b>
+    <br>
+    <hr style="width:20%">
+    <label for="commit">Commit to benchmark: </label><br>
+    <input id="commit" type="text" name="commit" style="width:330; "value="" required placeholder="becac559b713dfadd27198d683fde70551a543c8"><br>
+    Build this Commit against branch HEAD: <input type="checkbox" name="firstCommitBuildOnHead" value="True"> <b help="If not checked: The system will search for a COIN integration relating to the commit. If one is not found, the COIN integration of the commit's direct parent will be used." class="tooltip">(Help)</b><br>
+    <hr style="width:20%">
+    <label for="test_name"><b>1)</b> Select an existing QMLBench test: </label><b help="Start typing a test name to benchmark from the QMLBench test suite to provide suggestions." class="tooltip">(Help)</b>  <a href='/benchmarks' class="BasicButton">Browse benchmarks</a><br>
+    <input class="awesomplete" id="test_name" type="text" data-list="${testNames}" style="width:600;" name="test_name" required placeholder="f.ex: auto/animations/comparison/moving-images-animations.qml"><br>
+    <p><b>2)</b> Or upload custom <b>.qml</b> benchmark file:</p>
+    <div id="custom_benchmark_file_label" style="width:15%;padding:2px">No custom file set</div>
+    <input id="custom_benchmark_file" name="custom_benchmark_file" type="file" accept=".qml" onchange="updateFileLabel(this.files)" hidden /><br>
+    <input id='benchmarkButtonSingleCommit' type='button' value='Upload custom benchmark' /><br>
+    <hr style="width:20%">
+    <label for="test_hosts">Run job on host:</label><br>
+    ${hostSelectionComboBox}
+    <hr style="width:20%">
+    <label for="environment">Set custom environment variables on hosts:</label><br>
+    <table>
+        <th>Variables</th><th>Variable name</th><th>Variable Value</th>
+        <tr valign="top" height=0%>
+            <td rowspan=2>
+                <select id="environment" name="environment" size=2 style="min-width:100px;overflow:hidden" multiple></select>
+            </td>
+            <td><input id="environmentName" name="environmentName" type="text" style="width:200"></input></td>
+            <td><input id="environmentValue" name="environmentValue" type="text" style="width:200"></input></td>
+        </tr>
+        <tr valign="top">
+        <td align="center" colspan=2 style="padding-top:5px"><button type="button" onclick="AddEnvironmentVariable()">Add Environment Variable</button></td>
+    </table>
+    <hr style="width:20%">
+    <label for="owner">Enter email to receive notices about this job: </label><br>
+    <input id="owner" type="email" style="width:400;" name="owner" placeholder="someone@qt.io"><br>
+    <br>
+    <input type="submit" id="submit" disabled value="Schedule Single Commit">
+    </form>
+    </div>
+
+    <div id="twoCommitScheduler" style="display:none" align="center">
+    <form id="twoCommitSchedulerForm" action="/remoteSchedule/" method="post" encType="multipart/form-data">
+    <input type="hidden" name="testType" value="twoCommit"/>
+    <label for="branch">Qt major branch:</label>
+    <input id="branch" type="text" name="branch" style="width:50;" required placeholder="5.12"><br>
+    <label for="module">Qt module to test: </label>
+    <select id="module" name="module" style="width:200;" required placeholder="qtbase">
+    <option value="qtbase">QtBase</option>
+    <option value="qtdeclarative">QtDeclarative</option>
+    <option value="qtquickcontrols">QtQuickControls</option>
+    <option value="qtquickcontrols2">QtQuickControls2</option>
+    <option value="qtsvg">QtSvg</option>
+    <option value="qtquick3d">QtQuick3d</option>
+    </select>
+    <br>
+    <label for="openGLBackend">OpenGL Backend: </label>
+    <select id="openGLBackend" name="openGLBackend" style="width:100;" placeholder="desktop">
+    <option value="desktop">desktop</option>
+    <option value="angle">Angle</option>
+    <option value="software">software</option>
+    </select>
+    <b help="Only affects Windows. Override the OpenGL rendering backend to use for these tests. Leave default of \'desktop\' if unsure." class="tooltip">(What's this?)</b>
+    <br>
+    <hr style="width:20%">
+    <label for="firstCommit">First commit to benchmark: </label><br>
+    <input id="firstCommit" type="text" name="firstCommit" style="width:330; "value="" required placeholder="becac559b713dfadd27198d683fde70551a543c8"><br>
+    Build First Commit against branch HEAD: <input type="checkbox" name="firstCommitBuildOnHead" value="True"><b help="If not checked: The system will search for a COIN integration relating to the commit. If one is not found, the COIN integration of the commit's direct parent will be used." class="tooltip">(Help)</b><br>
+    Regression test this commit against its parent <input type="checkbox" name="simpleRegressionTest" id="simpleRegressionTest" value="True" onchange="disableSecondCommit()"> <b help="Check this box to perform a simple regression test of this commit against its parent to determine if the new commit causes a regression." class="tooltip">(Help)</b><br>
+    <hr style="width:20%">
+    <label for="secondCommit">Second commit to benchmark: </label><br>
+    <input id="secondCommit" type="text" name="secondCommit" id="secondCommit" style="width:330; "value="" required placeholder="becac559b713dfadd27198d683fde70551a543c8"><br>
+    Build Second Commit against branch HEAD: <input type="checkbox" name="secondCommitBuildOnHead" id="secondCommitBuildOnHead" value="True"><b help="If not checked: The system will search for a COIN integration relating to the commit. If one is not found, the COIN integration of the commit's direct parent will be used." class="tooltip">(Help)</b><br>
+    <hr style="width:20%">
+    <label for="test_name"><b>1)</b> Select an existing QMLBench test: </label><b help="Start typing a test name to benchmark from the QMLBench test suite to provide suggestions." class="tooltip">(Help)</b>  <a href='/benchmarks' class="BasicButton">Browse benchmarks</a><br>
+    <input class="awesomplete" id="test_name" type="text" data-list="${testNames}" style="width:600;" name="test_name" required placeholder="f.ex: auto/animations/comparison/moving-images-animations.qml"><br>
+    <p><b>2)</b> Or upload custom <b>.qml</b> benchmark file:</p>
+    <div id="custom_benchmark_file_label" style="width:15%;padding:2px">No custom file set</div>
+    <input id="custom_benchmark_file" name="custom_benchmark_file" type="file" accept=".qml" onchange="updateFileLabel(this.files)" hidden /><br>
+    <input id='benchmarkButtonTwoCommit' type='button' value='Upload custom benchmark' /><br>
+    <hr style="width:20%">
+    <label for="test_hosts">Run job on host:</label><br>
+    ${hostSelectionComboBox}
+    <hr style="width:20%">
+    <label for="environment">Set custom environment variables on hosts:</label><br>
+    <table>
+        <th>Variables</th><th>Variable name</th><th>Variable Value</th>
+        <tr valign="top" height=0%>
+            <td rowspan=2>
+                <select id="environment" name="environment" size=2 style="min-width:100px;overflow:hidden" multiple></select>
+            </td>
+            <td><input id="environmentName" name="environmentName" type="text" style="width:200"></input></td>
+            <td><input id="environmentValue" name="environmentValue" type="text" style="width:200"></input></td>
+        </tr>
+        <tr valign="top">
+        <td align="center" colspan=2 style="padding-top:5px"><button type="button" onclick="AddEnvironmentVariable()">Add Environment Variable</button></td>
+    </table>
+    <hr style="width:20%">
+    <label for="owner">Enter email to receive notices about this job: </label><br>
+    <input id="owner" type="email" style="width:400;" name="owner" placeholder="someone@qt.io"><br>
+    <br>
+    <input type="submit" id="submit" disabled value="Schedule Comparison">
+    </form>
+    </div>
+
+    </body>
+    
+    <script>
+    const basicScheduler = document.getElementById("basicScheduler");
+    const singleCommitScheduler = document.getElementById("singleCommitScheduler");
+    const twoCommitScheduler = document.getElementById("twoCommitScheduler");
+    var activeElement;
+
+    // Tool to help find the right child node in the different forms available
+    var _Utils = function ()
+    {
+        this.findChildById = function (element, childID, isSearchInnerDescendant) // isSearchInnerDescendant <= true for search in inner childern 
+        {
+            var retElement = null;
+            var lstChildren = isSearchInnerDescendant ? Utils.getAllDescendant(element) : element.childNodes;
+    
+            for (var i = 0; i < lstChildren.length; i++)
+            {
+                if (lstChildren[i].id == childID)
+                {
+                    retElement = lstChildren[i];
+                    break;
+                }
+            }
+    
+            return retElement;
+        }
+    
+        this.getAllDescendant = function (element, lstChildrenNodes)
+        {
+            lstChildrenNodes = lstChildrenNodes ? lstChildrenNodes : [];
+    
+            var lstChildren = element.childNodes;
+    
+            for (var i = 0; i < lstChildren.length; i++) 
+            {
+                if (lstChildren[i].nodeType == 1) // 1 is 'ELEMENT_NODE'
+                {
+                    lstChildrenNodes.push(lstChildren[i]);
+                    lstChildrenNodes = Utils.getAllDescendant(lstChildren[i], lstChildrenNodes);
+                }
+            }
+    
+            return lstChildrenNodes;
+        }        
+    }
+    var Utils = new _Utils;
+
+    //Register listeners for file upload buttons.
+    document.getElementById('benchmarkButtonBisect').addEventListener('click', openDialog);
+    document.getElementById('benchmarkButtonTwoCommit').addEventListener('click', openDialog);
+    document.getElementById('benchmarkButtonSingleCommit').addEventListener('click', openDialog);
+    function openDialog() {
+        var e = Utils.findChildById(activeElement, "custom_benchmark_file", true);
+        e.click();
+        var benchmarktextfield = Utils.findChildById(activeElement, "test_name", true);
+        benchmarktextfield.disabled = true;
+    }
+
+    function updateFileLabel(files) {
+      var e = Utils.findChildById(activeElement, "custom_benchmark_file_label", true);
+      e.textContent = \`Uploaded \${files[0].name}\`;
+      e.style.backgroundColor = "lightgreen"
+    }
+
+    // Methods to change which scheduler is visible
+    function BasicScheduler() {
+        if (basicScheduler.style.display === "none") {
+            basicScheduler.style.display = "block";
+            activeElement = basicScheduler;
+            singleCommitScheduler.style.display = "none";
+            twoCommitScheduler.style.display = "none"
+          } else {
+            basicScheduler.style.display = "none";
+          }
+        enableButton();
+    }
+
+    function SingleCommitScheduler(){
+        if (singleCommitScheduler.style.display === "none") {
+            basicScheduler.style.display = "none";
+            singleCommitScheduler.style.display = "block";
+            activeElement = singleCommitScheduler;
+            twoCommitScheduler.style.display = "none"
+          } else {
+            singleCommitScheduler.style.display = "none";
+          }
+        enableButton();
+    }
+
+    function TwoCommitScheduler(){
+        if (twoCommitScheduler.style.display === "none") {
+            basicScheduler.style.display = "none";
+            singleCommitScheduler.style.display = "none";
+            twoCommitScheduler.style.display = "block"
+            activeElement = twoCommitScheduler;
+          } else {
+            twoCommitScheduler.style.display = "none";
+          }
+        enableButton();
+    }
+
+    // Keep the submit buttons disabled until a valid test host is selected 
+    function enableButton(){
+        var e = Utils.findChildById(activeElement, "test_hosts", true)
+        if (e.options){
+            Utils.findChildById(activeElement, "submit", true).disabled = false;
+        } else {
+            Utils.findChildById(activeElement, "submit", true).disabled = false;
+        }
+    }
+    
+    function disableSecondCommit(){
+        var e = Utils.findChildById(activeElement, "simpleRegressionTest", true)
+        if (e.checked){
+            var secondCommit = Utils.findChildById(activeElement, "secondCommit", true);
+            secondCommit.readOnly = true;
+            secondCommit.value = "PARENT";
+            secondCommit.style.backgroundColor = "#C0C0C0";
+            Utils.findChildById(activeElement, "secondCommitBuildOnHead", true).disabled = true;
+        } else{
+            var secondCommit = Utils.findChildById(activeElement, "secondCommit", true);
+            secondCommit.readOnly = false;
+            secondCommit.value = "";
+            secondCommit.style.backgroundColor = "";
+            Utils.findChildById(activeElement, "secondCommitBuildOnHead", true).disabled = false;
+        }
+        console.log("Finished")
+    }
+
+    function AddEnvironmentVariable(){
+        var e = Utils.findChildById(activeElement, "environment", true)
+        var n = Utils.findChildById(activeElement, "environmentName", true)
+        var v = Utils.findChildById(activeElement, "environmentValue", true)
+        if(n.value != "" && v.value != "") {
+            var option = document.createElement("option");
+            option.text = \`\${n.value}=\${v.value}\`;
+            option.selected = true;
+            e.add(option);
+            e.size = e.length
+            n.value = ""
+            v.value = ""
+        }
+    }
+
+    </script>
+    <script src="awesomeplete/awesomplete.js" async>
+        // var input = document.getElementById("test_name");
+        // new Awesomplete(input, {
+        //     list: [${testNames}]
+        // });
+    </script>
+    `);
+}
+
+function homePage(req, res){
+  // Serve the scheduler's home page with status about remote hosts, current, queued, and finished jobs.
+
+  function sendSlaveData(){ // This is called when the host is not the scheduler. It responds with status data.
+    const readyEmitter = new events.EventEmitter();
+    readyEmitter.on('ready', function(){
+      res.send({
+        freediskspace: freediskspace,
+        uptime: process.uptime(),
+        hostname: myname,
+        ip: ip.address(),
+      });
+    });
+
+
+    console.log('Gathering disk space data.');
+    var freediskspace = -1;
+    function setDiskSpace(bytesfree){
+      freediskspace = bytesfree;
+      readyEmitter.emit('ready');
+    }
+
+    disk.check(path)
+      .then(info => {
+        setDiskSpace(info.available);
+      })
+      .catch(err => {
+        console.error(err);
+        readyEmitter.emit('ready');
+      });
+  }
+
+  function jobFormatter(job){
+    // Dynamically format a job for display on the page
+
+    function resultText(result){
+      if(result < 0 ){
+        return `Observed improvement: ${Math.abs(result)}%`;
+      } else if (result > 0){
+        return `Observed regression: ${result}%`;
+      } else {
+        return "-- No change in score --";
+      }
+    }
+
+    const commitString = job.commit ? job.commit : job.firstCommit ? job.firstCommit.slice(0, 10) + '..' + job.secondCommit.slice(0, 10) : job.good_commit ? job.good_commit.slice(0, 10) + '..' + job.bad_commit.slice(0, 10) : '';
+    const backendString = job.platform == 'win32' && job.openGLBackend ? ` - Open GL Backend: ${job.openGLBackend}` : '';
+    var resultString = '';
+    if (job.resultJSON && job.status != 'failed' && job.status != 'cancelled'){
+      if (job.testType == 'singleCommit') {
+        resultString = `Benchmark result: ${job.resultJSON.finalResults[0]}`;
+      } else if (job.testType == 'twoCommit'){
+        resultString = `<b>Results: </b>First commit result (${job.firstCommit.slice(0, 10)}): ${job.resultJSON.finalResults[0]}, 
+                                Second commit result (${job.secondCommit.slice(0, 10)}): ${job.resultJSON.finalResults[1]}<br>
+                                ${resultText(job.resultJSON.finalResults[2])}`;
+      }
+      if (job.testType == 'bisect'){
+        const bisectResult = job.resultJSON.bisectResults[1].length > 1 ?
+          `Bisect returned multiple commits:
+                                    ${job.resultJSON.bisectResults[1].slice(0,).join(' ')}
+                                    `
+          : `Bisect returned a single commit: ${job.resultJSON.bisectResults[1]}`;
+
+        resultString = `<b>Results: </b>Good commit result (${job.good_commit.slice(0, 10)}): ${job.resultJSON.baselineResults[0]}, 
+                                Bad commit result (${job.bad_commit.slice(0, 10)}): ${job.resultJSON.baselineResults[1]}<br>
+                                ${resultText(job.resultJSON.baselineResults[2])}<br>
+                                ${bisectResult}
+                                `;
+      }
+    }
+
+    var testString = '';
+    if (job.test_name){
+      testString = `Benchmark: ${job.test_name}`;
+    } else if (job.custom_benchmark_file){
+        testString = `Custom benchmark: ${job.custom_benchmark_file.name}`;
+    }
+
+    var timeEstimate = '';
+    if (job.startTime && job.status === 'running'){
+      if (job.expectedDuration - (Date.now() - job.startTime) < 1){
+        timeEstimate = `Job is about ${moment.duration(Math.abs(moment.duration(moment.duration(job.expectedDuration, 'ms').subtract(moment.duration(moment(Date.now()).diff(moment(job.startTime)))))), 'ms').format('h [hrs], m [min]')} overdue`;
+      } else {
+        timeEstimate = `About ${moment.duration(moment.duration(job.expectedDuration, 'ms').subtract(moment.duration(moment(Date.now()).diff(moment(job.startTime))))).format('h [hrs], m [min]')} remaining`;
+      }
+    } else if (job.status == 'staged'){
+      timeEstimate = `Expected to run for about ${moment.duration(job.expectedDuration, 'ms').format('h [hrs], m [min]')}`;
+    }
+    var logButton = '';
+    if (job.status != 'staged'){
+      logButton = `<br><button onclick="window.location.href='${job.status == 'running' ? 'http://' + (hosts[hosts.findIndex(x => x.hostname == job.test_hosts[0])] ? hosts[hosts.findIndex(x => x.hostname == job.test_hosts[0])] : {ip: '127.0.0.1'}).ip + ':' + webPort : ''}/logs/${job.test_hosts[0]}/${job.jobHash}.txt'">View Log</button>`;
+    }
+
+    // Return a formatted HTML string of the job to insert into the page.
+    return `
+        <b>${job.testType}
+        </b> job <font color="#454545"><b>${job.jobHash.slice(0, 10)}: ${job.title}<br>User: ${job.owner}</b></font> - Host: ${job.test_hosts[0]} - Timestamp: ${moment(job.timestamp).format('ddd DD MMM YYYY - HH:mm')} is ${job.status}<br>
+        Tested commit(s) ${commitString}${backendString}<br>
+        ${testString}
+        ${timeEstimate ? '<br>' + timeEstimate : ''}
+        ${resultString ? '<br>' + resultString : ''}
+        ${logButton}
+        `;
+  }
+
+  if (isPrimaryHost && !req.query.status){ // Serve the web page if we're the scheduler and status wasn't explicitly requested.
+    const readyEmitter = new events.EventEmitter();
+    // build the basic table element for the remote hosts
+    var slavesStatusHTML = '<table align="center"><tr><th>Slave Hostname</th><th>Online</th><th>Free Disk Space</th><th>Uptime</th><th>IP address</th></tr>';
+
+    var currentJobsHTML = nolocal ? '' : '<h3>Current Local Jobs</h3>';
+
+    // Do a whole bunch of stuff to build the various job type lists.
+    if (Object.entries(runningJob).length != 0 && runningJob.constructor === Object){ // Only display local jobs if there are any.
+
+      currentJobsHTML += `
+            <h4>Running:</h4>
+            <ul>
+            <li class="${runningJob.status}" style="position:relative">
+            <div class="lds-css ng-scope">
+            <div class="lds-spin" style="width:100%;height:100%;position:absolute;left:-35px;top:30%;"><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div></div>
+        ${jobFormatter(runningJob)} <button onclick="doCancel('${runningJob.jobHash}')" ${runningJob.status == 'cancelling' ? 'disabled' : ''}>Cancel</button></li>
+            </ul>
+            `;
+    }
+    if (jobQueue.length != 0){
+      currentJobsHTML += '<h4>Queued:</h4><ul>';
+      for (var i = jobQueue.length - 1; i >= 0; i--){
+        currentJobsHTML += `<li class="${jobQueue[i].status}">
+                ${jobFormatter(jobQueue[i])} <button onclick="doCancel('${jobQueue[i].jobHash}')" ${jobQueue[i].status == 'cancelling' ? 'disabled' : ''}>Cancel</button></li>
+                `;
+      }
+      currentJobsHTML += '</ul>';
+    }
+
+    var remoteJobsHTML = '';
+
+    if (remoteRunningJobs.length != 0){ // Only display remote running jobs if there are any
+      remoteJobsHTML += '<h4>Running:</h4><ul>';
+      for (var i = remoteRunningJobs.length - 1; i >= 0; i--){
+        remoteJobsHTML += `
+                <li class="${remoteRunningJobs[i].status}" style="position:relative;">
+                <div class="lds-css ng-scope">
+                <div class="lds-spin" style="width:100%;height:100%;position:absolute;left:-35px;top:30%;"><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div><div><div></div></div></div>
+                ${jobFormatter(remoteRunningJobs[i])} <button onclick="doCancel('${remoteRunningJobs[i].jobHash}')" ${remoteRunningJobs[i].status == 'cancelling' ? 'disabled' : ''}>Cancel</button></li>
+                `;
+      }
+      remoteJobsHTML += '</ul>';
+    }
+
+    if (remoteQueuedJobs.length != 0){ // Only display remote queued jobs if there are any
+      remoteJobsHTML += '<h4>Queued:</h4><ul>';
+      for (var i = remoteQueuedJobs.length - 1; i >= 0; i--){
+        remoteJobsHTML += `<li class="${remoteQueuedJobs[i].status}">
+                ${jobFormatter(remoteQueuedJobs[i])}<button onclick="doCancel('${remoteQueuedJobs[i].jobHash}')" ${remoteQueuedJobs[i].status == 'cancelling' ? 'disabled' : ''}>Cancel</button></li>
+                `;
+      }
+      remoteJobsHTML += '</ul>';
+    }
+
+    var completedJobsHTML = '';
+
+    if (completedJobs.length != 0){ // Add completed jobs to the completed list
+      if (completedJobs.length > 15){
+        completedJobs = completedJobs.slice(-15,);
+      }
+      completedJobsHTML += '<ul>';
+      for (var i = completedJobs.length - 1; i >= 0; i--){
+        completedJobsHTML += `
+                <li class= "${completedJobs[i].status}">${jobFormatter(completedJobs[i])}</li>
+                `;
+      }
+      completedJobsHTML += '</ul>';
+      completedJobsHTML = `<div style="padding-left:25px"><button onclick="window.location.href='/clearFinishedJobs'">Clean up old jobs</button></div>` + completedJobsHTML;
+    }
+
+    var basePageHTML = `
+        <head>
+        <title>QMLBench Regression finder: ${myname}</title>
+        <meta http-equiv="Refresh" content="90">
+        <link rel="shortcut icon" href="assets/favicon.ico"/> 
+        </head>
+        <style>
+        body{
+          font: 13px "Helvetica Neue", "Lucida Grande", "Arial";
+          background: #ECE9E9 -webkit-gradient(linear, 0% 0%, 0% 100%, from(#fff), to(#ECE9E9));
+          background-repeat: no-repeat;
+          color: #555;
+          -webkit-font-smoothing: antialiased;
+        }
+        table {
+            border: 1px solid black;
+        }
+        th { 
+            text-align: center;
+            border: 1px solid black;
+            padding-left: 15px;
+            padding-right: 15px;
+        }
+        td { 
+            padding-left: 15px;
+            padding-right: 15px;
+            text-align: center;
+            border: 1px solid black;
+        }
+        .failed { color: darkred; }
+        .finished { color: darkgreen; }
+        .running { color: ForestGreen; animation: RunningBlinker 3s linear infinite;}
+        .cancelling { color: gold; animation: CancellingBlinker 2s linear infinite;}
+        .cancelled { color: gray; }
+        .staged { color: royalblue; }
+
+        @keyframes CancellingBlinker {
+            50% {
+              color: chocolate;
+            }
+        }
+
+        @keyframes RunningBlinker {
+            50% {
+              color: MediumAquamarine;
+            }
+        }
+        @keyframes lds-spin {
+            0% {
+              opacity: 1;
+              -webkit-transform: scale(1.1, 1.1);
+              transform: scale(1.1, 1.1);
+            }
+            100% {
+              opacity: 0;
+              -webkit-transform: scale(1, 1);
+              transform: scale(1, 1);
+            }
+          }
+          @-webkit-keyframes lds-spin {
+            0% {
+              opacity: 1;
+              -webkit-transform: scale(1.1, 1.1);
+              transform: scale(1.1, 1.1);
+            }
+            100% {
+              opacity: 0;
+              -webkit-transform: scale(1, 1);
+              transform: scale(1, 1);
+            }
+          }
+          .lds-spin {
+            position: relative;
+          }
+          .lds-spin div > div {
+            position: absolute;
+            width: 40px;
+            height: 40px;
+            border-radius: 50%;
+            background: #5079c4;
+            -webkit-animation: lds-spin 1s linear infinite;
+            animation: lds-spin 1s linear infinite;
+          }
+          .lds-spin div:nth-child(1) > div {
+            left: 140px;
+            top: 80px;
+            -webkit-animation-delay: -0.875s;
+            animation-delay: -0.875s;
+          }
+          .lds-spin > div:nth-child(1) {
+            -webkit-transform: rotate(0deg);
+            transform: rotate(0deg);
+            -webkit-transform-origin: 160px 100px;
+            transform-origin: 160px 100px;
+          }
+          .lds-spin div:nth-child(2) > div {
+            left: 122.42640685999999px;
+            top: 122.42640685999999px;
+            -webkit-animation-delay: -0.75s;
+            animation-delay: -0.75s;
+          }
+          .lds-spin > div:nth-child(2) {
+            -webkit-transform: rotate(45deg);
+            transform: rotate(45deg);
+            -webkit-transform-origin: 142.42640686px 142.42640686px;
+            transform-origin: 142.42640686px 142.42640686px;
+          }
+          .lds-spin div:nth-child(3) > div {
+            left: 80px;
+            top: 140px;
+            -webkit-animation-delay: -0.625s;
+            animation-delay: -0.625s;
+          }
+          .lds-spin > div:nth-child(3) {
+            -webkit-transform: rotate(90deg);
+            transform: rotate(90deg);
+            -webkit-transform-origin: 100px 160px;
+            transform-origin: 100px 160px;
+          }
+          .lds-spin div:nth-child(4) > div {
+            left: 37.57359314px;
+            top: 122.42640685999999px;
+            -webkit-animation-delay: -0.5s;
+            animation-delay: -0.5s;
+          }
+          .lds-spin > div:nth-child(4) {
+            -webkit-transform: rotate(135deg);
+            transform: rotate(135deg);
+            -webkit-transform-origin: 57.57359314px 142.42640686px;
+            transform-origin: 57.57359314px 142.42640686px;
+          }
+          .lds-spin div:nth-child(5) > div {
+            left: 20px;
+            top: 80px;
+            -webkit-animation-delay: -0.375s;
+            animation-delay: -0.375s;
+          }
+          .lds-spin > div:nth-child(5) {
+            -webkit-transform: rotate(180deg);
+            transform: rotate(180deg);
+            -webkit-transform-origin: 40px 100px;
+            transform-origin: 40px 100px;
+          }
+          .lds-spin div:nth-child(6) > div {
+            left: 37.57359314px;
+            top: 37.57359314px;
+            -webkit-animation-delay: -0.25s;
+            animation-delay: -0.25s;
+          }
+          .lds-spin > div:nth-child(6) {
+            -webkit-transform: rotate(225deg);
+            transform: rotate(225deg);
+            -webkit-transform-origin: 57.57359314px 57.57359314px;
+            transform-origin: 57.57359314px 57.57359314px;
+          }
+          .lds-spin div:nth-child(7) > div {
+            left: 80px;
+            top: 20px;
+            -webkit-animation-delay: -0.125s;
+            animation-delay: -0.125s;
+          }
+          .lds-spin > div:nth-child(7) {
+            -webkit-transform: rotate(270deg);
+            transform: rotate(270deg);
+            -webkit-transform-origin: 100px 40px;
+            transform-origin: 100px 40px;
+          }
+          .lds-spin div:nth-child(8) > div {
+            left: 122.42640685999999px;
+            top: 37.57359314px;
+            -webkit-animation-delay: 0s;
+            animation-delay: 0s;
+          }
+          .lds-spin > div:nth-child(8) {
+            -webkit-transform: rotate(315deg);
+            transform: rotate(315deg);
+            -webkit-transform-origin: 142.42640686px 57.57359314px;
+            transform-origin: 142.42640686px 57.57359314px;
+          }
+          .lds-spin div:nth-child(9) > div {
+            left: 140px;
+            top: 80px;
+            -webkit-animation-delay: 0.125s;
+            animation-delay: 0.125s;
+          }
+          .lds-spin > div:nth-child(9) {
+            -webkit-transform: rotate(360deg);
+            transform: rotate(360deg);
+            -webkit-transform-origin: 160px 100px;
+            transform-origin: 160px 100px;
+          }
+          .lds-spin {
+            width: 30px !important;
+            height: 30px !important;
+            -webkit-transform: translate(-15px, -15px) scale(0.15) translate(15px, 15px);
+            transform: translate(-15px, -15px) scale(0.15) translate(15px, 15px);
+          }
+        </style>
+
+        <body onload="checkForCancellingAndQueuedJobs()">
+        <h3>QMLBench Regression finder: ${myname}</h3>
+        <button onclick="location.href='mailto:daniel.smith@qt.io';" style="position:absolute;right:10px;top:10px;">report bug / request feature</button>
+        <br>
+        <div align="center">
+        <button onclick="window.location.href='/scheduleJob'">Schedule new job</button>
+        <button onclick="window.location.href='/refreshRemoteJobs'" style="display:${standalone ? 'none' : 'block'}">Refresh Remote Jobs</button>
+        <button onclick="window.location.href='/logs'">Browse log directory</button>
+        <button onclick="window.location.href='/recentErrors'" style="background-color:${recentErrors.length > 0 ? 'red' : ''}">Recent system errors (${recentErrors.length})</button>
+        </div>
+
+        <br>
+        <div id="jobs">
+        ${currentJobsHTML}
+        <br>
+        <h3 style="display:${standalone ? 'none' : 'block'}">Current Remote Jobs</h3>
+        ${remoteJobsHTML}
+        <br style="display:${standalone ? 'none' : 'block'}">
+        <h3>Completed Jobs</h3>
+        ${completedJobsHTML}
+        <br>
+        </div>
+        </body>
+
+        <script>
+
+        function doCancel(jobHash){
+            window.location.href='/cancelJob/?jobHash=' + jobHash;
+        }
+
+        function checkForCancellingAndQueuedJobs(){ // Refresh the page often if there are cancelling jobs or queued jobs while none are running 
+            //console.log(document.getElementById('jobs').innerHTML.search("is cancelling"));
+            if (document.getElementById('jobs').innerHTML.search("is cancelling") > -1){
+                setTimeout(function() {location.reload(true)}, 5000);
+            } else if (document.getElementById('jobs').innerHTML.search("is running") < 0 && document.getElementById('jobs').innerHTML.search("is staged") > -1){
+                setTimeout(function() {location.reload(true)}, 5000);
+            }
+        }
+        </script>
+        `;
+
+    const injectIndex = basePageHTML.indexOf('</h3>') + 5; // Find the location of the the page title. This is where we'll inject our remote host status HTML
+    var finishedRequestCount = 0;
+
+    readyEmitter.on('ready', function(){ // All done constructing the page. Send it to the requester.
+      setTimeout(function(){
+        slavesStatusHTML += '</table>';
+        res.send(basePageHTML.insert(injectIndex, slavesStatusHTML));
+      }, 200);
+    });
+
+    readyEmitter.on('finished', function(){ // This signal is emitted when a remote host responds with it's status while we build the page HTML.
+      finishedRequestCount += 1;
+      if (finishedRequestCount == slaves.length){
+        readyEmitter.emit('ready');
+      }
+    });
+
+    function addToSlaveHTML(html){
+      slavesStatusHTML += html;
+    }
+    if (slaves){
+      for (var slave in slaves){
+        axios.get(`http://${slaves[slave]}:${webPort}/?status=true`, {timeout: 2000}) // Query the remote host (or self) for status information
+          .then(response => {
+            if (hosts.findIndex(x => x.ip == response.data.ip) < 0){ // Add the friendly hostname to the list of hosts if it's not already there.
+              hosts.push({
+                hostname: response.data.hostname,
+                ip: response.data.ip,
+              });
+            }
+            addToSlaveHTML(`
+                    <tr>
+                        <td>${response.data.hostname}</td>
+                        <td style="color:green">Online</td>
+                        <td>${prettyBytes(response.data.freediskspace)}</td>
+                        <td>${moment.duration(response.data.uptime, 'seconds').format()}</td>
+                        <td>${response.data.ip}</td>
+                    </tr>
+                    `);
+            readyEmitter.emit('finished');
+          })
+          .catch(error => { // Still display the host in the table, but with a connection error.
+            var offlineMsg;
+            if (error.response){
+              offlineMsg = `${error.response.status} ${error.response.statusText}`;
+            } else {
+              offlineMsg = `${error.message}`;
+            }
+
+            addToSlaveHTML(`
+                    <tr>
+                        <td>-</td>
+                        <td style="color:red">Offline - ${offlineMsg}</td>
+                        <td>-</td>
+                        <td>-</td>
+                        <td>${error.request._options.hostname}</td>
+                    </tr>
+                    `);
+            readyEmitter.emit('finished');
+          });
+      }
+    } else {
+      addToSlaveHTML(`
+                    <tr>
+                        <td>-</td>
+                        <td>-</td>
+                        <td>-</td>
+                        <td>-</td>
+                        <td>-</td>
+                    </tr>
+                    `);
+      readyEmitter.emit('ready');
+    }
+  } else {
+    sendSlaveData();
+  }
+}
+
+
+function runServer(){
+  // magic sauce. Start the server.
+  var server = express();
+  server.use(bodyParser.json()); // to support JSON-encoded bodies
+  server.use(bodyParser.urlencoded({ // to support URL-encoded bodies
+    extended: true,
+  }));
+  server.use(fileUpload({
+    // useTempFiles : true,
+    // tempFileDir : os.tmpdir(),
+    // preserveExtension : true
+  }));
+  server.use(express.json()); // to support JSON-encoded bodies
+  server.use(express.urlencoded()); // to support URL-encoded bodies
+  server.use('/logs/', express.static(`${__dirname}/logs/`));
+  server.use('/logs/', serveIndex(`${__dirname}/logs/`, {view: 'details'}));
+  server.use('/awesomeplete', express.static('node_modules/awesomplete'));
+  server.use('/benchmarks/', express.static(`${__dirname}/QtBuild/qmlbench/benchmarks/`));
+  server.use('/assets/', express.static(`${__dirname}/assets/`));
+  server.use('/benchmarks/', serveIndex(`${__dirname}/QtBuild/qmlbench/benchmarks/`, {view: 'details'}));
+
+
+  server.all('/remoteSchedule', (req, res) => scheduleJobRequest(req, res));
+  server.all('/scheduleJob', (req, res) => schedulerPage(req, res));
+  server.all('/queryJobs', (req, res) => queryJobs(req, res));
+  server.all('/refreshRemoteJobs', (req, res) => querySlaveJobs(req, res));
+  server.all('/pushJobUpdate', (req, res) => addUpdateToQueue(req, res));
+  server.all('/cancelJob', (req, res) => cancelJob(req, res));
+  server.all('/recentErrors', (req, res) => errorsPage(req, res));
+  server.all('/clearRecentErrors', (req, res) => clearErrors(req, res));
+  server.all('/clearFinishedJobs', (req, res) => clearFinishedJobs(req, res));
+  server.all('/', (req, res) => homePage(req, res));
+
+  console.log(`Starting server... Listening on port ${webPort}`);
+
+  server.listen(webPort); // Actually spin up the server with the configurations above.
+}
+
+// Create all our logging directories
+if (!fs.existsSync(pathmodule.join(__dirname, 'logs'))){
+  fs.mkdirSync(pathmodule.join(__dirname, 'logs'));
+}
+if (!fs.existsSync(pathmodule.join(__dirname, 'logs', myname))){
+  fs.mkdirSync(pathmodule.join(__dirname, 'logs', myname));
+}
+if (!fs.existsSync(pathmodule.join(__dirname, 'logs', myname, 'results'))){
+  fs.mkdirSync(pathmodule.join(__dirname, 'logs', myname, 'results'));
+}
+
+if (!fs.existsSync(pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs'))){
+  console.log('creating log directory');
+  fs.mkdirSync(pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs'));
+}
+if (!fs.existsSync(pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs', myname))){
+  console.log('creating my own folder');
+  fs.mkdirSync(pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs', myname));
+}
+if (!fs.existsSync(pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs', myname, 'results'))){
+  console.log('creating results dir');
+  fs.mkdirSync(pathmodule.join(os.homedir(), 'QMLBenchRegressionLogs', myname, 'results'));
+}
+
+function gatherTestNames(){
+  // Gather test names for partial match suggestions on the scheduler page. If QMLBench isn't already initialized, do it now and block the scheduler.
+
+  function loadTests(){
+    var walk = require('walk');
+
+    // Walker options
+    var walker = walk.walk('./QtBuild/qmlbench/benchmarks/', { followLinks: false });
+
+    walker.on('file', function(root, stat, next) {
+      // Add this file to the list of files
+      if (!(stat.name.toLowerCase() == 'readme.md')){
+        const file = (root + '/' + stat.name).substring(31,);
+        testNames.push(`${file.replace(/\\/g, '/')}`);
+      }
+      next();
+    });
+
+    walker.on('end', function() {
+      console.log(`Loaded ${testNames.length} tests from QMLBench.`);
+    });
+  }
+
+  if (!fs.existsSync(pathmodule.join(__dirname, 'QtBuild'))){
+    fs.mkdirSync(pathmodule.join(__dirname, 'QtBuild'));
+  }
+  if (!fs.existsSync(pathmodule.join(__dirname, 'QtBuild', 'qmlbench'))){
+    locked = true;
+    console.log("QMLBench isn't already available locally. Setting up the repositories. This may take a minute and the scheduler will not be available for jobs until this completes.");
+    console.log('WARN: Test name suggestions will be unavailable until this process completes...');
+    runningJob = {
+      testType: 'STARTUP',
+      test_hosts: [myname],
+      status: 'running',
+      jobHash: 'NO_JOBHASH',
+      title: 'Initializing Qt repositories. This may take a minute.',
+      expectedDuration: '300000',
+    };
+    var pythonexe = os.platform() == 'win32' ? 'python' : 'python3'; // Python3 for windows will simply be 'python'.
+    const testProcess = spawn(pythonexe, ['regressionFinder.py', '--setupEnv'], os.platform() == 'win32' ? {detached: false} : {detached: true});
+    runningJob.processHandle = testProcess;
+    testProcess.on('close', code => {
+      console.log('Finished setting up the repositories. Ready to accept new jobs.');
+      runningJob = {};
+      locked = false;
+      if (jobQueue.length != 0){
+        notifyJob.emit('newLocalJob'); // If we're not acting as a remote host, we can just execute immediately.
+      }
+      loadTests();
+    });
+  } else {
+    loadTests();
+  }
+}
+
+function startup(){
+// Do some pre-flight checks. Try to query the remote hosts before we even start the server listening so the data is already there, if the remote hosts are online first.
+
+  const readyEmitter = new events.EventEmitter();
+  var finishedRequestCount = 0;
+  readyEmitter.on('ready', function(){ // We're finished querying remote hosts. Start the local webserver.
+
+    if (isPrimaryHost){
+      querySlaveJobs(); // Query all remote hosts for their current work before starting the server.
+    }
+    runServer();
+  });
+
+  readyEmitter.on('finished', function(){
+    finishedRequestCount += 1;
+    if (finishedRequestCount >= slaves.length){ // Have we gotten responses from everyone?
+      readyEmitter.emit('ready');
+    }
+  });
+
+  if (slaves){
+    console.log('Querying slave IPs for hostname info...'); // Gather remote host data
+    for (var slave in slaves){
+      if (slaves[slave] == ip.address() && !nolocal){ // Add self to the list of hosts if the 'nolocal' parameter wasn't set.
+        hosts.push(
+          {
+            hostname: myname,
+            ip: ip.address(),
+          });
+        readyEmitter.emit('finished');
+      } else {
+        axios.get(`http://${slaves[slave]}:${webPort}/?status=true`, {timeout: 2000}) // Qeuery everyone else
+          .then(response => {
+            console.log(`Adding ${response.data.hostname} to the slave list`);
+            hosts.push(
+              {
+                hostname: response.data.hostname,
+                ip: response.data.ip,
+              });
+            readyEmitter.emit('finished');
+          })
+          .catch(error => {
+            console.log(`INFO: Host at ${error.request._options.hostname} appears to be offline.`);
+            readyEmitter.emit('finished');
+          });
+      }
+    }
+  } else {
+    readyEmitter.emit('finished'); // No hosts? Odd, but okay. Continue.
+  }
+
+  gatherTestNames();
+}
+
+startup(); // Call pre-flight checks, which calls the server-startup, which runs the scheduler and web interface