See [global.names] (17.6.4.3.2 in the C++11 spec.)

Change-Id: I8434496dbe392b52d339d5f17cfaeee8dbd88995
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@digia.com>

Change-Id: I9370da5f3159f6d8ad9ee9f8fe949f66eb235592
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@digia.com>
Reviewed-by: Orgad Shaneh <orgads@gmail.com>

This caused displaced highlighting of macro uses after #if constructs.

MacroUse::utf16charBegin() was based on the last "continuation token",
which was wrong.

Change-Id: I89983d82fcf804ba853c04a59a7533c489785d05
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

In most cases we need to work with the utf16 indices. Only in
cppfindreferences the byte interface is still needed since there we read
in files and work on a QByteArray to save memory.

Change-Id: I6ef6a93fc1875a8c9a305c075d51a9ca034c41bb
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

The Lexer can handle it now.

Task-number: QTCREATORBUG-7356
Change-Id: I8c4b03a247656e013d44c3cedca4835e133d4036
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

This will avoid confusion when later more length and indices methods are
added.

In Token:
    length() --> bytes()
    begin() --> bytesBegin()
    end() --> bytesEnd()

Change-Id: I244c69b022e239ee762b4114559e707f93ff344f
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

They are already inlined. Now it's easier to find read-only accesses.

Change-Id: I9aaeca3bc5860e3a20a536a2484925e4334c005f
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

Change-Id: I3be73c2a4d256a0d28e863fd2517a4d7442b34d8
Reviewed-by: Mitch Curtis <mitch.curtis@digia.com>
Reviewed-by: Leena Miettinen <riitta-leena.miettinen@digia.com>

Change-Id: I4d99053f540073483c16ce842426bf8cd3def421
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@digia.com>

baseLine is used for generating new tokens, which are
later added m_state.m_lineRef - 1 again

Change-Id: I33928a90988e8d4c317ae460647f16f87da5b155
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@digia.com>

* If the macro is defined before, track its reference
* Synchronize environment line before calling remove, which
  currently sets incorrect line
* Set macro offset

Task-number: QTCREATORBUG-10454
Change-Id: I480d16423a976a025bb8c71046610a46f9d7b0fd
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@digia.com>

The following snippet demonstrates the problem:

--- snip ---
// comment \

#include <something.h>
...

class Foo
{
    ...
};
--- snap ---

If there are >=9 empty/preprocessor lines, the preprocessed source
becomes
// comment \
# 12 "file.cpp"
...

The lexer considers the line marker as a continued C++ comment, and
highlighting is broken

Change-Id: I30a2fc7d19b279316e9273697179c90d81099573
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

Task-number: QTCREATORBUG-11216
Change-Id: Iac10e75f0f5c504b79e8466607dc1f478e578f99
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

Change-Id: I87d6e76be3030e617603209c3a1b8b3c476cf1f6
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@digia.com>

Change-Id: Ib5423fdd064e4546f848c0b640b0ed0514c26d3a
Reviewed-by: Leena Miettinen <riitta-leena.miettinen@digia.com>
Reviewed-by: Kai Koehne <kai.koehne@digia.com>

This takes too much memory. For qtcreator.pro the numbers are as
follows:

    Patch applied:  ~ 1600MB (RES)
    Patch reverted: ~  510MB (RES)

This reverts commit 4c2daa90.

Task-number: QTCREATORBUG-10973
Change-Id: I843bd7c1ea4a26a1ec55ddc14c2a34a98d040922
Reviewed-by: hjk <hjk121@nokiamail.com>
Reviewed-by: Robert Loehning <robert.loehning@digia.com>
Reviewed-by: Eike Ziller <eike.ziller@digia.com>
Reviewed-by: Orgad Shaneh <orgads@gmail.com>
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

This adds definitions for the macros __FILE__, __LINE__, __DATE__ and
__TIME__ on demand.

As a side effect, this also introduces highlighting for the uses of
these macros.

Task-number: QTCREATORBUG-8036
Change-Id: Ib7546c7d45d2eecbc50c7883fc684e3497154405
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>
Reviewed-by: Eike Ziller <eike.ziller@digia.com>
Reviewed-by: hjk <hjk121@nokiamail.com>

Breaks highlighting for macros using the predefined macros.

This reverts commit 1d834c11.

Change-Id: Ic13c407e293a806a63ff30153864530df6a32e47
Reviewed-by: hjk <hjk121@nokiamail.com>
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

Preprocessor variables __LINE__,__FILE__,__TIME__,__DATE__ where destroying
the following systems when affected variables were standing  within the
same line with those variables:
* highlighting
* refactoring
* local renaming

Task-number: QTCREATORBUG-8036
Change-Id: I1a4b919d15812872ca5a8e63b1031ec1ab144c22
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@digia.com>

Only methods as programming functions are affected. Besides renaming
some actions like "Switch Between Function Declaration/Definition" this
mostly touches (api) code comments.

This is a follow-up patch to commit 872bfb70.

Change-Id: Icb65e8d73b59a022f8885b14df497169543a3b92
Reviewed-by: hjk <hjk121@nokiamail.com>

Change-Id: Ibe6f41ac15df1ec685b0d0766ff568abf6f3ae7e
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@digia.com>

Change-Id: I1309db70e98d678e150388c76ce665e988fdf081
Reviewed-by: Eike Ziller <eike.ziller@digia.com>

By lexing the first token after a macro call (meaning: the token after
the closing parenthesis (which was passed to handleFunctionLikeMacro
which in turn pushed it back into the token buffer)), a token buffer
might be popped, which unblocks the macro that generated the actual
param pack. The effect was that if this happens in the expansion of a
recursive macro (with parameters!), the preprocessor ended up in an
infinite loop.

Task-number: QTCREATORBUG-9015
Task-number: QTCREATORBUG-9447

Change-Id: I0d83c59188ec15c4a948970e9fa944a17d765475
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@digia.com>

Moved it from the handleIfDefDirective to the pre-defined macros, so
that #if defined() can also see it.

Task-number: QTCREATORBUG-9322
Change-Id: Icbecad5c885dd2374b559969c99631c3ddc73844
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@digia.com>

Doing so resulted in an incorrect position for the EOF token when the
preprocessed output would be parsed. That in turn leads to incorrect
insertion positions for refactoring actions.

This is especially true when a file contains only preprocessor
directives: the EOF token would point to line 1 column 1, which is
usually not the place where code should be inserted.

Change-Id: I7d359aa7a6c04bc52c8b873fd49ad6afc3a77319
Reviewed-by: hjk <hjk121@nokiamail.com>

Preprocessor did not correctly handle when variadic macro arguments were not
provided at all, if there were other arguments: macro was not expanded
in case only the non variadic arguments were given.

 #define MACRO(...)       used to work fine for 0 or more arguments.
 #define MACRO(ARG0, ...) used to work only for 2 or more arguments, now fixed.

Change-Id: I64e9199ceccae05618a49931c2adad8e4f9471ba
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@digia.com>

Change-Id: I45399a196500cbc6aecedfaa94a50890914906f1
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

Change-Id: If86c73945808e871f60fdf231b91e02f66bd1b32
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

Change-Id: I896423172aed0e19feaaf9144ed5f18ba99c720b
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

This prevents a whole lot of re-allocations when the output byte array
needs to grow. It also prevents some heap fragmentation for big files.

Because the preprocessed output is short lived (it will be parsed
immediately after, and then discarded), it is not squeezed to the
minimal size. This would result in another allocation.

Change-Id: I4974be5144f88cdfc4ddc9d8330200725aa90803
Reviewed-by: hjk <hjk121@nokiamail.com>
Reviewed-by: Eike Ziller <eike.ziller@digia.com>

... and adjust INCLUDEPATH accordingly.

while i'm at messing with include statements, also re-order the include
blocks according to policy and sort them within bigger blocks.

Change-Id: I7762abfd7c4ecf59432b99db2f424e4fa25733a5
Reviewed-by: Tobias Hunger <tobias.hunger@digia.com>
Reviewed-by: Eike Ziller <eike.ziller@digia.com>

Change-Id: I37bc8f9bc874de52281b8597838ef8addae1a1a3
Reviewed-by: Tobias Hunger <tobias.hunger@digia.com>

Change-Id: Ic6a9ff0359625021ebc061d22db6811814534205
Reviewed-by: Kai Koehne <kai.koehne@digia.com>

Track the typical #ifndef/#define/#endif usage in header files to see if
the macro is an include guard. If so, store it in the Document. No
behavioural change, just recording the name.

This can be used in the future to track if a file needs to be re-parsed
when a macro changes: if it was used in the file, and not defined in it
nor being the include-guard, a file should be re-preprocessed and
re-parsed.

It can also be used to check if two files have the same include guard.

Change-Id: I2715f529997a7b24a11bdbc6150652e2669f1a46
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@digia.com>

When in 'keep comments' mode, the preprocessor does not properly handle macro calls with
comments between the macro name and the opening parenthesis: "FOO /*something to say*/
(45)".

Change-Id: I6fe733242e4d2ccff2985d17399d0a084917415a
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

- Fixed blocking macro check
- Enabled buffer "compression": when tokens are generated and no new
macro is being blocked, then prepend the tokens to the previous buffer.
This happens a lot when undo-ing look-ahead.
- Added documentation

Change-Id: I6fa816d94ce4696e473bdbc4f3bf477d77e4dd51
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@digia.com>

This does not yet resolve the file using the proper mechanism.

Change-Id: I04913e8b01ae0c3411961f0c1cffe07202f06a0a
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

#!/usr/bin/env ruby

Dir.glob('**/*.cpp') { |file|
  # skip ast (excluding paste, astpath, and canv'ast'imer)
  next if file =~ /ast[^eip]|keywords\.|qualifiers|preprocessor|names.cpp/i
  s = File.read(file)
  next if s.include?('qlalr')
  orig = s.dup
  s.gsub!(/\n *if [^\n]*{\n[^\n]*\n\s+}(\s+else if [^\n]* {\n[^\n]*\n\s+})*(\s+else {\n[^\n]*\n\s+})?\n/m) { |m|
    res = $&
    if res =~ /^\s*(\/\/|[A-Z_]{3,})/ # C++ comment or macro (Q_UNUSED, SDEBUG), do not touch braces
      res
    else
      res.gsub!('} else', 'else')
      res.gsub!(/\n +} *\n/m, "\n")
      res.gsub(/ *{$/, '')
    end
  }
  s.gsub!(/ *$/, '')
  File.open(file, 'wb').write(s) if s != orig
}

Change-Id: I3b30ee60df0986f66c02132c65fc38a3fbb6bbdc
Reviewed-by: hjk <qthjk@ovi.com>

Turns out QByteArray::setNum() is unnecessarily slow (will be fixed
independently), but even then, we can be faster.

Change-Id: I663bd2b8cc844bbe800879bccfa57999d020ba3b
Reviewed-by: Erik Verbruggen <erik.verbruggen@digia.com>

Change-Id: Ibe4cc69eafd14dab7707862b1068ce1e21b1d8e0
Reviewed-by: Nikolai Kosjar <nikolai.kosjar@digia.com>