Merge branch 'master' into release

Author: jhasse

.gitignore

@@ -32,3 +32,6 @@ TAGS
 # Ninja output
 .ninja_deps
 .ninja_log
+
+# Visual Studio Code project files
+/.vscode/

.travis.yml

@@ -1,6 +1,15 @@
+matrix:
+  include:
+    - os: linux
+      compiler: gcc
+    - os: linux
+      compiler: clang
+    - os: osx
 sudo: false
 language: cpp
-compiler:
-  - gcc
-  - clang
-script: ./configure.py --bootstrap && ./ninja all && ./ninja_test --gtest_filter=-SubprocessTest.SetWithLots && ./misc/ninja_syntax_test.py
+script:
+  - ./configure.py --bootstrap
+  - ./ninja all
+  - ./ninja_test --gtest_filter=-SubprocessTest.SetWithLots
+  - ./misc/ninja_syntax_test.py
+  - ./misc/output_test.py

HACKING.md

@@ -13,14 +13,50 @@ run `ninja_test` when developing.
 Ninja is built using itself.  To bootstrap the first binary, run the
 configure script as `./configure.py --bootstrap`.  This first compiles
 all non-test source files together, then re-builds Ninja using itself.
-You should end up with a `ninja` binary (or `ninja.exe`) in the source root.
+You should end up with a `ninja` binary (or `ninja.exe`) in the project root.
 
 #### Windows
 
 On Windows, you'll need to install Python to run `configure.py`, and
 run everything under a Visual Studio Tools Command Prompt (or after
-running `vcvarsall` in a normal command prompt).  See below if you
-want to use mingw or some other compiler instead of Visual Studio.
+running `vcvarsall` in a normal command prompt).
+
+For other combinations such as gcc/clang you will need the compiler
+(gcc/cl) in your PATH and you will have to set the appropriate
+platform configuration script.
+
+See below if you want to use mingw or some other compiler instead of
+Visual Studio.
+
+##### Using Visual Studio
+Assuming that you now have Python installed, then the steps for building under
+Windows using Visual Studio are:
+
+Clone and checkout the latest release (or whatever branch you want). You
+can do this in either a command prompt or by opening a git bash prompt:
+
+```
+    $ git clone git://github.com/ninja-build/ninja.git && cd ninja
+    $ git checkout release
+```
+
+Then:
+
+1. Open a Windows command prompt in the folder where you checked out ninja.
+2. Select the Microsoft build environment by running
+`vcvarsall.bat` with the appropriate environment.
+3. Build ninja and test it.
+
+The steps for a Visual Studio 2015 64-bit build are outlined here:
+
+```
+    > "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x64
+    > python configure.py --bootstrap
+    > ninja --help
+```
+Copy the ninja executable to another location, if desired, e.g. C:\local\Ninja.
+
+Finally add the path where ninja.exe is to the PATH variable.
 
 ### Adjusting build flags
 
@@ -73,17 +109,9 @@ build "all" before committing to verify the other source still works!
 
 ## Testing performance impact of changes
 
-If you have a Chrome build handy, it's a good test case.  Otherwise,
-[the github downoads page](https://github.com/ninja-build/ninja/releases)
-has a copy of the Chrome build files (and depfiles). You can untar
-that, then run
-
-    path/to/my/ninja chrome
-
-and compare that against a baseline Ninja.
-
-There's a script at `misc/measure.py` that repeatedly runs a command like
-the above (to address variance) and summarizes its runtime.  E.g.
+If you have a Chrome build handy, it's a good test case.  There's a
+script at `misc/measure.py` that repeatedly runs a command (to address
+variance) and summarizes its runtime.  E.g.
 
     path/to/misc/measure.py path/to/my/ninja chrome
 
@@ -95,7 +123,7 @@ and run that directly on some representative input files.
 Generally it's the [Google C++ coding style][], but in brief:
 
 * Function name are camelcase.
-* Member methods are camelcase, expect for trivial getters which are
+* Member methods are camelcase, except for trivial getters which are
   underscore separated.
 * Local variables are underscore separated.
 * Member variables are underscore separated and suffixed by an extra

RELEASING

@@ -1,19 +1,20 @@
 Notes to myself on all the steps to make for a Ninja release.
 
 Push new release branch:
-1. Consider sending a heads-up to the ninja-build mailing list first
-2. Make sure branches 'master' and 'release' are synced up locally
-3. update src/version.cc with new version (with ".git"), then
+1. Run afl-fuzz for a day or so (see HACKING.md) and run ninja_test
+2. Consider sending a heads-up to the ninja-build mailing list first
+3. Make sure branches 'master' and 'release' are synced up locally
+4. Update src/version.cc with new version (with ".git"), then
        git commit -am 'mark this 1.5.0.git'
-4. git checkout release; git merge master
-5. fix version number in src/version.cc (it will likely conflict in the above)
-6. fix version in doc/manual.asciidoc (exists only on release branch)
-7. commit, tag, push (don't forget to push --tags)
+5. git checkout release; git merge master
+6. Fix version number in src/version.cc (it will likely conflict in the above)
+7. Fix version in doc/manual.asciidoc (exists only on release branch)
+8. commit, tag, push (don't forget to push --tags)
        git commit -am v1.5.0; git push origin release
        git tag v1.5.0; git push --tags
        # Push the 1.5.0.git change on master too:
        git checkout master; git push origin master
-8. construct release notes from prior notes
+9. Construct release notes from prior notes
    credits: git shortlog -s --no-merges REV..
 
 Release on github:

appveyor.yml

@@ -0,0 +1,40 @@
+version: 1.0.{build}
+image: Visual Studio 2017
+
+environment:
+  CLICOLOR_FORCE: 1
+  CHERE_INVOKING: 1 # Tell Bash to inherit the current working directory
+  matrix:
+    - MSYSTEM: MINGW64
+    - MSYSTEM: MSVC
+
+for:
+  -
+    matrix:
+      only:
+        - MSYSTEM: MINGW64
+    build_script:
+      ps: "C:\\msys64\\usr\\bin\\bash -lc @\"\n
+      pacman -S --quiet --noconfirm --needed re2c 2>&1\n
+      sed -i 's|cmd /c $ar cqs $out.tmp $in && move /Y $out.tmp $out|$ar crs $out $in|g' configure.py\n
+      ./configure.py --bootstrap --platform mingw 2>&1\n
+      ./ninja all\n
+      ./ninja_test 2>&1\n
+      ./misc/ninja_syntax_test.py 2>&1\n\"@"
+  -
+    matrix:
+      only:
+        - MSYSTEM: MSVC
+    build_script:
+    - cmd: >-
+        call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"
+
+        python configure.py --bootstrap
+
+        ninja.bootstrap.exe all
+
+        ninja_test
+
+        python misc/ninja_syntax_test.py
+
+test: off

configure.py

@@ -98,7 +98,7 @@ def is_aix(self):
         return self._platform == 'aix'
 
     def uses_usr_local(self):
-        return self._platform in ('freebsd', 'openbsd', 'bitrig', 'dragonfly')
+        return self._platform in ('freebsd', 'openbsd', 'bitrig', 'dragonfly', 'netbsd')
 
     def supports_ppoll(self):
         return self._platform in ('freebsd', 'linux', 'openbsd', 'bitrig',
@@ -256,7 +256,7 @@ def _run_command(self, cmdline):
 if '--bootstrap' in configure_args:
     configure_args.remove('--bootstrap')
 n.variable('configure_args', ' '.join(configure_args))
-env_keys = set(['CXX', 'AR', 'CFLAGS', 'LDFLAGS'])
+env_keys = set(['CXX', 'AR', 'CFLAGS', 'CXXFLAGS', 'LDFLAGS'])
 configure_env = dict((k, os.environ[k]) for k in os.environ if k in env_keys)
 if configure_env:
     config_str = ' '.join([k + '=' + pipes.quote(configure_env[k])
@@ -356,6 +356,11 @@ def binary(name):
     if platform.uses_usr_local():
         cflags.append('-I/usr/local/include')
         ldflags.append('-L/usr/local/lib')
+    if platform.is_aix():
+        # printf formats for int64_t, uint64_t; large file support
+        cflags.append('-D__STDC_FORMAT_MACROS')
+        cflags.append('-D_LARGE_FILES')
+
 
 libs = []
 
@@ -397,6 +402,10 @@ def shell_escape(str):
 
 if 'CFLAGS' in configure_env:
     cflags.append(configure_env['CFLAGS'])
+    ldflags.append(configure_env['CFLAGS'])
+if 'CXXFLAGS' in configure_env:
+    cflags.append(configure_env['CXXFLAGS'])
+    ldflags.append(configure_env['CXXFLAGS'])
 n.variable('cflags', ' '.join(shell_escape(flag) for flag in cflags))
 if 'LDFLAGS' in configure_env:
     ldflags.append(configure_env['LDFLAGS'])
@@ -405,7 +414,7 @@ def shell_escape(str):
 
 if platform.is_msvc():
     n.rule('cxx',
-        command='$cxx $cflags -c $in /Fo$out',
+        command='$cxx $cflags -c $in /Fo$out /Fd' + built('$pdb'),
         description='CXX $out',
         deps='msvc'  # /showIncludes is included in $cflags.
     )
@@ -476,6 +485,9 @@ def has_re2c():
 n.newline()
 
 n.comment('Core source files all build into ninja library.')
+cxxvariables = []
+if platform.is_msvc():
+    cxxvariables = [('pdb', 'ninja.pdb')]
 for name in ['build',
              'build_log',
              'clean',
@@ -496,15 +508,15 @@ def has_re2c():
              'string_piece_util',
              'util',
              'version']:
-    objs += cxx(name)
+    objs += cxx(name, variables=cxxvariables) 
 if platform.is_windows():
     for name in ['subprocess-win32',
                  'includes_normalize-win32',
                  'msvc_helper-win32',
                  'msvc_helper_main-win32']:
-        objs += cxx(name)
+        objs += cxx(name, variables=cxxvariables)
     if platform.is_msvc():
-        objs += cxx('minidump-win32')
+        objs += cxx('minidump-win32', variables=cxxvariables)
     objs += cc('getopt')
 else:
     objs += cxx('subprocess-posix')
@@ -527,7 +539,7 @@ def has_re2c():
 all_targets = []
 
 n.comment('Main executable is library plus main() function.')
-objs = cxx('ninja')
+objs = cxx('ninja', variables=cxxvariables)
 ninja = n.build(binary('ninja'), 'link', objs, implicit=ninja_lib,
                 variables=[('libs', libs)])
 n.newline()
@@ -542,6 +554,8 @@ def has_re2c():
 n.comment('Tests all build into ninja_test executable.')
 
 objs = []
+if platform.is_msvc():
+    cxxvariables = [('pdb', 'ninja_test.pdb')]
 
 for name in ['build_log_test',
              'build_test',
@@ -560,10 +574,10 @@ def has_re2c():
              'subprocess_test',
              'test',
              'util_test']:
-    objs += cxx(name)
+    objs += cxx(name, variables=cxxvariables)
 if platform.is_windows():
     for name in ['includes_normalize_test', 'msvc_helper_test']:
-        objs += cxx(name)
+        objs += cxx(name, variables=cxxvariables)
 
 ninja_test = n.build(binary('ninja_test'), 'link', objs, implicit=ninja_lib,
                      variables=[('libs', libs)])
@@ -579,7 +593,9 @@ def has_re2c():
              'hash_collision_bench',
              'manifest_parser_perftest',
              'clparser_perftest']:
-  objs = cxx(name)
+  if platform.is_msvc():
+    cxxvariables = [('pdb', name + '.pdb')]
+  objs = cxx(name, variables=cxxvariables)
   all_targets += n.build(binary(name), 'link', objs,
                          implicit=ninja_lib, variables=[('libs', libs)])
 

doc/manual.asciidoc

@@ -155,11 +155,10 @@ design is quite clever.
 Ninja's benefit comes from using it in conjunction with a smarter
 meta-build system.
 
-http://code.google.com/p/gyp/[gyp]:: The meta-build system used to
+https://gn.googlesource.com/gn/[gn]:: The meta-build system used to
 generate build files for Google Chrome and related projects (v8,
-node.js).  gyp can generate Ninja files for all platforms supported by
-Chrome. See the
-https://chromium.googlesource.com/chromium/src/+/master/docs/ninja_build.md[Chromium Ninja documentation for more details].
+node.js), as well as Google Fuschia.  gn can generate Ninja files for
+all platforms supported by Chrome.
 
 https://cmake.org/[CMake]:: A widely used meta-build system that
 can generate Ninja files on Linux as of CMake version 2.8.8.  Newer versions
@@ -594,7 +593,7 @@ Ninja supports this processing in two forms.
    to its stdout.  Ninja then filters these lines from the displayed
    output.  No `depfile` attribute is necessary, but the localized string
    in front of the the header file path. For instance
-   `msvc_deps_prefix = Note: including file: `
+   `msvc_deps_prefix = Note: including file:`
    for a English Visual Studio (the default). Should be globally defined.
 +
 ----
@@ -881,7 +880,8 @@ quoting rules are deterimined by the called program, which on Windows
 are usually provided by the C library.  If you need shell
 interpretation of the command (such as the use of `&&` to chain
 multiple commands), make the command execute the Windows shell by
-prefixing the command with `cmd /c`.
+prefixing the command with `cmd /c`. Ninja may error with "invalid parameter"
+which usually indicates that the command line length has been exceeded.
 
 [[ref_outputs]]
 Build outputs

misc/ninja-mode.el

@@ -56,7 +56,7 @@
                (save-excursion
                  (goto-char (line-end-position 0))
                  (or
-                  ;; If we're continuting the previous line, it's not a
+                  ;; If we're continuing the previous line, it's not a
                   ;; comment.
                   (not (eq ?$ (char-before)))
                   ;; Except if the previous line is a comment as well, as the

misc/ninja.vim

@@ -1,8 +1,8 @@
 " ninja build file syntax.
 " Language: ninja build file as described at
 "           http://ninja-build.org/manual.html
-" Version: 1.4
-" Last Change: 2014/05/13
+" Version: 1.5
+" Last Change: 2018/04/05
 " Maintainer: Nicolas Weber <nicolasweber@gmx.de>
 " Version 1.4 of this script is in the upstream vim repository and will be
 " included in the next vim release. If you change this, please send your change
@@ -21,7 +21,10 @@ set cpo&vim
 
 syn case match
 
-syn match ninjaComment /#.*/  contains=@Spell
+" Comments are only matched when the # is at the beginning of the line (with
+" optional whitespace), as long as the prior line didn't end with a $
+" continuation.
+syn match ninjaComment /\(\$\n\)\@<!\_^\s*#.*$/  contains=@Spell
 
 " Toplevel statements are the ones listed here and
 " toplevel variable assignments (ident '=' value).
@@ -38,12 +41,13 @@ syn match ninjaKeyword "^subninja\>"
 " limited set of magic variables, 'build' allows general
 " let assignments.
 " manifest_parser.cc, ParseRule()
-syn region ninjaRule start="^rule" end="^\ze\S" contains=ALL transparent
-syn keyword ninjaRuleCommand contained command deps depfile description generator
+syn region ninjaRule start="^rule" end="^\ze\S" contains=TOP transparent
+syn keyword ninjaRuleCommand contained containedin=ninjaRule command
+                                     \ deps depfile description generator
                                      \ pool restat rspfile rspfile_content
 
-syn region ninjaPool start="^pool" end="^\ze\S" contains=ALL transparent
-syn keyword ninjaPoolCommand contained depth
+syn region ninjaPool start="^pool" end="^\ze\S" contains=TOP transparent
+syn keyword ninjaPoolCommand contained containedin=ninjaPool  depth
 
 " Strings are parsed as follows:
 " lexer.in.cc, ReadEvalString()