creator-autotest.qdoc 14.2 KB
Newer Older
1 2
/****************************************************************************
**
3
** Copyright (C) 2017 The Qt Company Ltd.
4
** Contact: https://www.qt.io/licensing/
5
**
6
** This file is part of the Qt Creator documentation.
7 8 9 10 11
**
** Commercial License Usage
** Licensees holding valid commercial Qt licenses may use this file in
** accordance with the commercial license agreement provided with the
** Software or, alternatively, in accordance with the terms contained in
12 13 14
** a written agreement between you and The Qt Company. For licensing terms
** and conditions see https://www.qt.io/terms-conditions. For further
** information use the contact form at https://www.qt.io/contact-us.
15
**
16 17 18 19 20 21 22
** GNU Free Documentation License Usage
** Alternatively, this file may be used under the terms of the GNU Free
** Documentation License version 1.3 as published by the Free Software
** Foundation and appearing in the file included in the packaging of
** this file. Please review the following information to ensure
** the GNU Free Documentation License version 1.3 requirements
** will be met: https://www.gnu.org/licenses/fdl-1.3.html.
23 24 25 26 27 28 29 30 31 32 33
**
****************************************************************************/

/*!
    \contentspage {Qt Creator Manual}
    \previouspage creator-cpu-usage-analyzer.html
    \page creator-autotest.html
    \nextpage creator-advanced.html

    \title Running Autotests

34 35 36 37 38
    \QC integrates the \l{Qt Test} framework and
    \l{https://github.com/google/googletest}{Google C++ Testing Framework} for
    unit testing applications and libraries. You can use \QC to build and run
    Qt tests, Qt Quick tests (QML-based Qt tests), and Google tests for your
    projects.
39 40 41

    \image qtcreator-autotests.png

42
    \section1 Creating Tests
43

44 45 46 47 48
    You can use a wizard to create projects that contain Qt or Google tests.

    \section2 Creating Qt Tests

    To create a Qt test:
49 50 51

    \list 1
        \li Select \uicontrol File > \uicontrol {New File or Project} >
52
            \uicontrol {Other Project} > \uicontrol {Auto Test Project} >
53 54
            \uicontrol Choose to create a project with boilerplate code for a
            Qt test.
55 56

        \li In the \uicontrol {Project and Test Information} dialog, specify
57
            settings for the project and test:
58 59 60

            \list 1

61 62 63
                \li In the \uicontrol {Test framework} field, select
                    \uicontrol {Qt Test}.

64 65 66 67 68 69 70 71 72 73 74 75 76 77 78
                \li Select the \uicontrol {GUI Application} check box to create
                    a Qt application.

                \li In the \uicontrol {Test case name} field, enter a name for
                    the test case.

                \li Select the \uicontrol {Requires QApplication} check box to
                    add the include statement for QApplication to the main.cpp
                    file of the project.

                \li Select the \uicontrol {Generate initialization and cleanup
                    code} checkbox to add functions to your test that are
                    executed by the testing framework to initialize and clean
                    up the test.

79 80 81
                \li In the \uicontrol {Build system} field, select the build
                    system to use for building the project: qmake, CMake, or
                    Qbs.
82 83 84 85 86

            \endlist

    \endlist

87 88 89
    \QC creates the test in the specified project directory. Edit the .cpp file
    to add private slots for each test function in your test. For more information
    about creating Qt tests, see \l{Creating a Test}.
90

91 92 93 94 95 96
    \section2 Creating Google Tests

    To create a Google test:

    \list 1
        \li Select \uicontrol File > \uicontrol {New File or Project} >
97
            \uicontrol {Other Project} > \uicontrol {Auto Test Project} >
98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124
            \uicontrol Choose to create a project with boilerplate code for a
            Google test.

        \li In the \uicontrol {Project and Test Information} dialog, specify
            settings for the project and test:

            \list 1

                \li In the \uicontrol {Test framework} field, select
                    \uicontrol {Google Test}.

                \li In the \uicontrol {Test case name} field, enter a name for
                    the test case.

                \li In the \uicontrol {Test set name} field, enter a name for
                    the test set.

                \li Select the \uicontrol {Enable C++ 11} check box to
                    support C++ 11 features in the test.

                \li In the \uicontrol {Google test repository} field, select
                    a directory that contains a clone of the googletest
                    repository.

                    To use an installed Google C++ Testing framework instead,
                    see \l{Setting Up the Google C++ Testing Framework}.

125 126 127 128
                \li In the \uicontrol {Build system} field, select the build
                    system to use for building the project: qmake, CMake, or
                    Qbs.

129 130 131 132
            \endlist

    \endlist

133
    \QC creates the test in the specified project directory.
134 135 136
    For more information about creating Google tests, see the
    \l{https://github.com/google/googletest/blob/master/googletest/docs/Primer.md}
    {Google Test Primer}.
137

138
    \section1 Setting Up the Google C++ Testing Framework
139

140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183
    To build and run Google tests, you must have the Google C++ Testing
    framework installed and configured on the development host. You can either
    clone it from Git Hub or install it from an installation package.

    To configure a project to use a cloned Google testing framework, edit the
    \c INCLUDEPATH variable in the project file (.pro) to include the source
    and \c include folders of Google Test's \c googletest and \c googlemock.
    Usually, you need to add the following subfolders:

    \list
        \li \c googletest
        \li \c googlemock
        \li \c googletest/include
        \li \c googlemock/include
    \endlist

    You also need to add the necessary files to the \c SOURCES variable. For
    example:

    \list
        \li \c googletest/src/gtest-all.cc
        \li \c googlemock/src/gmock-all.cc
    \endlist

    To configure a project to use an installed Google testing framework package,
    add the following include paths to the .pro file:

    \list
        \li \c <googletest_install_path>/include/gtest
        \li \c <googletest_install_path>/include/gmock
    \endlist

    Then add linker options to be able to find the libraries and to link against
    them. For example, for qmake based projects, you typically need to add the
    following values to the .pro file:

    \list
        \li \c {LIBS += -lgtest -L<path_to_gtest_lib>}
        \li \c {LIBS += -lgmock -L<path_to_gmock_lib>}
    \endlist

    \section1 Building and Running Tests

    To build and run tests:
184 185 186

    \list 1

187
        \li Open a project that contains tests.
188

189 190
        \li In the \uicontrol Tests view, select the tests to run.

191
        \li In the \uicontrol {Test Results} output pane, select
192
            \inlineimage run_small.png
193 194 195 196 197 198 199 200 201 202
            (\uicontrol {Run All Tests}) to run all test or
            \inlineimage qtcreator-run-selected-tests.png
            (\uicontrol {Run Selected Tests}) to run the selected tests.

            \note By default, \QC builds a project before deploying and running
            it.

    \endlist

    If a test takes more than a minute to execute, the default timeout might
203
    stop the test execution. To increase the timeout, select \uicontrol Tools >
204
    \uicontrol Options > \uicontrol {Testing} > \uicontrol General.
205

206 207 208 209 210 211 212 213 214 215
    \section2 Selecting Tests to Run

    The \uicontrol Tests view shows all the tests found for the currently active
    test frameworks in the current project. Select the test cases to run.

    \image qtcreator-tests-view.png

    If a Qt Quick test case does not have a name, it is marked
    \uicontrol Unnamed in the list. Unnamed test cases are executed when you
    select \uicontrol {Run All Tests}. You cannot select or deselect them.
216

217
    \QC scans the project for tests when you open the project and updates the
218
    test list for the currently active test frameworks when you edit tests.
219 220 221 222
    To refresh the view, select \uicontrol {Rescan Tests} in the context menu.

    You can add filters to specify the directories within the current project
    to scan for tests. Select \uicontrol Tools > \uicontrol Options >
223
    \uicontrol {Testing} > \uicontrol General > \uicontrol Add, and
224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239
    specify paths to the directories to scan for tests. Wildcards are not
    supported in the filter expressions.

    \image qtcreator-autotests-options.png

    To show or hide init and cleanup or data functions in the \uicontrol Tests
    view, select \inlineimage filtericon.png
    (\uicontrol {Filter Test Tree}), and then select \uicontrol {Show Init and
    Cleanup Functions} or \uicontrol {Show Data Functions}. Double-click a
    function in the list to open its source code in the code editor.

    The test cases are listed in alphabetic order. To list them in the order in
    which they are defined in the source code, select \inlineimage leafsort.png
    (\uicontrol {Sort Naturally}).

    \section2 Specifying Settings for Running Qt Tests
240

241 242
    The code inside a benchmark test is measured, and possibly also repeated
    several times in order to get an accurate measurement. This depends on the
243 244
    measurement back-end that you can select in the
    \uicontrol {Benchmark Metrics} group in \uicontrol Tools >
245
    \uicontrol Options > \uicontrol {Testing} > \uicontrol {Qt Test}:
246
    walltime, CPU tick counter, event counter, Valgrind Callgrind, and Linux
247 248
    Perf. For more information, see \l{Creating a Benchmark}.

249 250
    \image qtcreator-autotests-options-qt.png

251 252 253
    To receive verbose output when running benchmarks, select the
    \uicontrol {Verbose benchmarks} check box.

254 255 256
    To allow the debugger to interrupt Qt tests on assertions, select the
    \uicontrol {Disable crash handler while debugging} check box.

257 258 259 260 261 262
    To record information about signals and slots in the test log, select the
    \uicontrol {Log signals and slots} check box.

    \section2 Specifying Settings for Running Google Tests

    To specify settings for running Google tests, select \uicontrol Tools >
263
    \uicontrol Options > \uicontrol {Testing} > \uicontrol {Google Test}.
264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279

    \image qtcreator-autotests-options-google.png

    To run disabled tests, select the \uicontrol {Run disabled tests} check box.

    To run several iterations of the tests, select the \uicontrol {Repeat tests}
    check box and enter the number of times the tests should be run in the
    \uicontrol Iterations field. To make sure that the tests are independent and
    repeatable, you can run them in a different order each time by selecting the
    \uicontrol {Shuffle tests} check box.

    To turn failures into debugger breakpoints, select the
    \uicontrol {Break on failure while debugging} check box. To turn assertion
    failures into C++ exceptions, select the \uicontrol {Throw on failure} check
    box.

280 281
    \section1 Viewing Test Output

282 283 284 285 286 287 288
    The test results are displayed in the \uicontrol {Test Results} output pane
    in XML format. XML can be parsed more easily and reliably than plain text.

    However, if a Qt test crashes, it might not produce complete XML code that
    can be parsed, which might lead to information loss. The lost information
    might be retrievable when viewing the results as plain text.
    To view the results of Qt tests as plain text, select \uicontrol Tools >
289
    \uicontrol Options > \uicontrol {Testing} > \uicontrol {Qt Test}, and
290 291 292 293 294 295 296
    then deselect the \uicontrol {Use XML output} check box. Then select the
    \inlineimage text.png
    (\uicontrol {Switch Between Visual and Text Display}) button in the
    \uicontrol {Test Results} output pane to switch to the text display.

    The following table lists the messages that the \uicontrol {Test Results}
    output pane displays:
297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323

    \table
        \header
            \li Result
            \li Description
        \row
            \li BENCH
            \li Benchmark test.
        \row
            \li BFAIL
            \li Blacklisted test case failed. Since Qt 5.4, you can
                provide a BLACKLIST file for tests. It is mainly used internally
                by the Qt CI system.
        \row
            \li BPASS
            \li Blacklisted test case passed.
        \row
            \li DEBUG
            \li Debug message.
        \row
            \li XFAIL
            \li Test case is expected to fail, so it is marked by using the
                QEXPECT_FAIL macro. If the test case passes instead, an
                unexpected pass (XPASS) is written to the test log.
        \row
            \li FAIL
            \li Test case failed. Double-click the line for more information.
324 325 326 327 328 329 330
        \row
            \li FATAL
            \li A fatal error occurred that stops the test case from being run,
                for example.
        \row
            \li INFO
            \li Informative message.
331 332 333 334 335 336 337 338 339
        \row
            \li INTERNAL
            \li Internal message.
        \row
            \li PASS
            \li Test case passed.
        \row
            \li SKIP
            \li Test case was skipped.
340 341 342
        \row
            \li SYSTEM
            \li An error message received from or influenced by the OS.
343 344 345 346 347 348 349 350 351
        \row
            \li XPASS
            \li Test case passed even though it was expected to fail.
        \row
            \li WARN
            \li Warning message.
    \endtable

    To view only messages of a particular type, select
352
    \inlineimage filtericon.png
353 354 355
    (\uicontrol {Filter Test Results}), and then select the types of messages to
    show.

356 357 358
    By default, test result output is limited to 100,000 characters. The output
    pane is automatically scrolled down when new results are added. To display
    full results, select \uicontrol Tools > \uicontrol Options >
359
    \uicontrol {Testing} > \uicontrol General, and then deselect the
360 361 362 363 364 365
    \uicontrol {Limit result output} check box. To disable automatic scrolling,
    deselect the \uicontrol {Automatically scroll results} check box.

    Internal messages and run configuration warnings for guessed configurations
    are omitted by default. To view them, deselect the \uicontrol {Omit internal
    messages} and \uicontrol {Omit run configuration warnings} check boxes.
366
*/