Title: Flymake -- an on-the-fly syntax checker for Emacs
Author: Pavel Kobiakov
Email: pk_at_work@yahoo.com
Environment: GNU emacs, GNU make
Keywords: on-the-fly syntax check, emacs, make
Description: Performs on-the-fly syntax checks on the files being edited
using the external syntax check tool (usually the compiler).
Highlights erroneous lines and displays associated
error messages.
Switching to the GNU Emacs/GNU Make environment from the Microsoft Visual Studio 6.0/Visual Assist 6.0 pair had both positive and negative consequences. On one hand, memory consumption was dramatically reduced, making many things work much faster then they used to. On the other hand, such nice IDE features as context aware auto-completion and auto parameter info were no longer available. This resulted in an increased amount of typos and errors, not to be revealed until compilation time.
Furthermore, even Visual Assist was not able to correctly parse all of the C++ constructs I've been using. This is especially true for non-trivial templates as well as macros. In the end, the ultimate tool for checking syntax is the compiler, as it can easily handle any valid constructs and report errors if any. So, having compiler make on-the-fly syntax checks of the source code being edited seems like a good idea.
Users of some modern Java IDEs, like IntelliJ IDEA, enjoy this feature for a relatively long time. I don't know of any C++ IDE with this feature, and Visual C++ 6.0/.NET lack it as well. So came the idea of developing of an on-the-fly syntax checker for Emacs -- flymake.
Flymake is implemented as an Emacs minor mode. It runs the syntax check tool (the compiler for C++) in the background, passing it a temporarily copy of the current buffer and parses the output for known error/warning message patterns. Flymake then highlights erroneous lines (that is, lines for which at least one error or warning has been reported), and displays an overall buffer status in the mode line, as shown on the figure below.
File File.cpp here has 1 error and 1 warning. Using flymake-goto-next-error, it is posible to
jump to erroneous line 5, highlighted in red, and calling flymake-display-err-menu-for-current-line will
popup a menu containing error text provided by the compiler. Line 1 is highlighted in blue, as it only has
a warning attached to it. As calling flymake-display-err-menu-for-current-line
shows, this warning really belongs to line 5 of File.h.
Selecting the menu item will jump to line 5 of File.h, which has the same error,
as shown on the next figure.
Notice that line 1 of File.h is highlighted in red, as File.cpp,
which was actually used to syntax check the .h file, still contains 1 error and 1 warning.
Syntax check is done 'on-the-fly'. It is started whenever a) a newline character is added to the buffer, b) buffer is saved, and c) some changes were made to the buffer more than 1 second ago.
When flymake mode is active, any of the 3 conditions stated above will cause flymake to try to syntax check the current buffer. Flymake first determines whether it is able to do syntax check. It then saves a copy of the buffer in a temporary file in the buffer's directory, creates a syntax check command and launches a process with this command. The output is parsed using known error message patterns, and error information (file name, line number, type and text) is saved. After the process has finished, flymake highlights erroneous lines in the buffer using the accumulated error information.
A simple algorithm based of buffer filename is used to determine whether a buffer can be syntax checked. There are 3 possible options:
A mapping from filename regexp to syntax check mode is stored in a customizable flymake-allowed-file-name-masks
variable.
After the possibility of the syntax check has been determined, a temporary copy of the current buffer is made so
that the most recent unsaved changes could be seen by the syntax check tool. Making a copy is quite straightforward
in a standalone case (mode 1), as it's just saving buffer contents in a file. Things get trickier, however,
when master file is involved, as it requires to a) locate master file and b) patch it to include the current file using
its new name. Locating master file is discussed in the following section.
Patching just changes all appropriate lines of the master file so that they use the new (temporary) name
of the current file. For example, suppose current file name is File.h, the master file is
File.cpp, and it includes current file via #include "File.h". Current file's copy is
saved to file File_flymake.h, so the include line must be changed to #include "File_flymake.h".
Finally, patched master file is saved to File_flymk_master.cpp, and the last one is passed to the
syntax check tool.
Master file is located in two steps.
First, a list of possible master files is built. Again, a simple name matching is used to find the files.
For a C++ header File.h, flymake searches for all .cpp files in the directories whose relative
paths are stored in a customizable variable flymake-master-file-dirs, which usually contains
something like ("." "./src"). No more than flymake-master-file-count-limit entries is added
to the master file list. The list is then sorted to move files with names File.cpp to the top.
Next, each master file in a list is checked to contain the appropriate include directives. No more than
flymake-check-file-limit of each file are parsed.
For File.h, the include directives to look for are #include "File.h",
#include "../File.h", etc. Each include is checked via the include directories (next section)
to be sure it points to the correct File.h.
First matching master file found stops the search. The master file is then patched and saved to disk. In case no master file was found, syntax check is aborted, and corresponding status is reported in the mode line.
Two sets of include directories are distinguished: system include directories and project include directories.
The former is just the contents of the INCLUDE environment variable. The latter is not so easy to obtain,
and the way it can be obtained can vary greatly for different project. Therefore, a customizable variable
flymake-get-project-include-dirs-function is used to provide the way to implement the desired behaviour.
The default implementation, flymake-get-project-include-dirs-imp utilizes the particular feature of the
build system I'm using: project include directories are stored in a makefile variable, whose value can
be obtained via a call to make. This requires a correct
base directory, that is a directory containing a correct Makefile, be determined.
As obtaining the project include directories might be a costly operation, its return value is cached in the hash table. The cache is cleared in the beginning of every syntax check attempt.
Currently, flymake uses make to perform all syntax checks, and all make configuration data
is usually stored in a file called Makefile. To allow for future extensions, flymake uses
a notion of buildfile to reference the 'project configuration' file.
Searching for a buildfile is done in a manner similar to that of searching for possible master files.
A customizable variable flymake-buildfile-dirs holds a list of relative paths to the buildfile. They are
checked sequentially until a buildfile is found. In case there's no build file, syntax check is aborted.
Buildfile values are also cached.
After all required files have been located and temporary copies made, a syntax check command is constructed. It
consists of two parts: program name, read from the flymake-program-name variable, and program arguments,
obtained via a call to customizable function flymake-get-program-args-function. The function has
2 parameters, a file name of the file to pass to the syntax check tool and a base directory.
Default implementation returns the following arguments:
'-s -C "base directory" SOURCES="relative path/file name" check-syntax'.
The correct relative path from the base directory to the source directory is build with the help of the
flymake-build-relative-path function.
The output generated by the syntax check tool is parsed in the process filter
using the error message patterns stored in the
flymake-err-line-patterns variable. This variable contains a list of items of the form
(regexp file-idx line-idx err-text-idx), used to determine whether a particular line
is an error message and extract
file name, line number and error text, respectively. Error type (error/warning) is also guessed by matching
error text with the '^warning' pattern. Anything that was not classified as a warning is considered an error.
Type is then used to sort error menu items, which shows error messages first.
The error information obtained is saved in a buffer local variable. The buffer for which the process output belongs is determined from the pid->buffer mapping updated after every process launch/exit.
Highlighting is implemented with overlays and happens in the process sentinel, after all temporary files have
been deleted.
2 customizable faces are used: flymake-errline-face and
flymake-warnline-face. Errors belonging outside the current buffer are considered to belong to line 1 of
the current buffer.
After syntax check is finished, its results are displayed in the mode line. The following statuses are defined.
| RUNNING | Flymake is currently running. |
| PASSED | Syntax check was successfully passed (no errors, no warnings). |
| E/W | Number of errors/warnings found by the syntax check process. |
| STOPPED | Syntax check was killed as a result of starting project compilation. |
| CFGERR | Syntax check process returned nonzero exit code, but no errors/warnings were reported. This indicates a possible configuration error. |
| COMP | Syntax check cannot be started as compilation is currently in progress. |
| NOMASTER | Flymake was unable to find master file for the current buffer. |
| NOMK | Flymake was unable to find buildfile for the current buffer. |
| PROCERR | Flymake was unable to launch syntax check process. |
For each erroneous line, a popup menu with all errors reported for that line can be displayed. It is build from
the error information stored in the buffer. Selecting the item whose error belongs to another file brings forward
that file with the help of the flymake-goto-file-and-line function.
The only mode flymake currently knows about is compile.
Flymake won't start syntax check if it thinks the compilation is in progress. The check is made by the
flymake-compilation-is-running, which tests the compilation-in-progress variable.
Flymake also provides an alternative command for starting compilation, flymake-compile:
(defun flymake-compile()
"kill all flymake syntax checks, start compilation"
(interactive)
(flymake-stop-all-syntax-checks)
(call-interactively 'compile)
)
It just kills all the active syntax check processes before calling compile.
The whole idea of the on-the-fly syntax check will be compromised in case any single syntax check will require a significant amount of time to complete. So flymake is certainly meant to be used on a fast machine. Also, any possible optimizations, like switching on precompiled headers generation, will be of great help.
To give you some numbers, here's the check times for several C++ source files compiled on a Pentium-IV 1.7 GHz machine.
| Number of lines in a file | Precompiled headers | Syntax check time, in seconds |
| 1000 | ON | 1.0 |
| 1000 | OFF | 3.6 |
| 500 | ON | 0.7 |
| 500 | OFF | 3.1 |
| 10 | ON | 0.5 |
| 10 | OFF | 0.1 |
1 second delay is barely noticable, and even with a 3-4 seconds delay on-the-fly syntax check might still be very useful.
Flymake makes syntax check attempts quite often. It might be unable to launch syntax check process for some reasons (no buildfile, etc.). Nevertheless, the failed attempts themselves might take some time (getting include directories, looking for a master file), interfering with the editing process, which is sometimes annoying. The possible solution is to switch off flymake-mode for such buffers.
Algorithms used for locating buildfiles and master files are quite simple, and won't cover all the use cases.
Flymake has only been used in Windows (W2K SP2) environment, with make and the
Visual C++ 6.0 compiler as the syntax check tool.
So, switching to other operating system/build tool/compiler might require a considerable amount of customizations/fixes.
For example, the only error message pattern flymake understands as of now is that for the Visual C++ compiler.
Flymake might be of a great help both when writing new code and changing the old code. When seen immediately, errors and warnings are much easier to fix. Warnings issued during compilation are sometimes left alone, as the project still 'builds OK' (unless 'treat warnings as errors' option is used). With flymake, those warnings will be displayed immediately, and so more likely be fixed.
Flymake requires quite a fast machine to run, but fast machines are not so rare nowadays.