Re: [PATCH] Add 2-pass mode to gtags

[Hideki IWAMOTO]

Hi.
This is an experimental patch that adds builtin parser. 
The performance of making ttag files is greatly improved by building parser 
into gtags. 
When btree chache is big enough, speed may improve than double.
Even when btree cache is small, it is faster than the default mode. 

Some points are different from the previous patch.
 * Source files are parsed only once. 
   The candidate of reference and other symbols are written to the temporary 
file in Pass1,
   and it is read in Pass2.
 * I/O decreases by changing the reading and writing of the pipe from ctags-x 
into tag record.

GTAGSLABEL=default
 Pass 1:
                   +--------------+  ctags-x   +-------+
  source files ===>| gtags-parser |===========>| gtags |===> GTAGS
                   |              |            |       |===> GPATH
                   +--------------+            +-------+
 Pass 2:
                   +--------------+  ctags-x   +-------+
  source files ===>| gtags-parser |===========>| gtags |===> GRTAGS
         GTAGS ===>|              |  GPATH ===>|       |
                   +--------------+            +-------+
 Pass 3:
                   +--------------+  ctags-x   +-------+
  source files ===>| gtags-parser |===========>| gtags |===> GSYMS
         GTAGS ===>|              |  GPATH ===>|       |
                   +--------------+            +-------+

GTAGSLABEL=builtin
 Pass 1:
                       +-------+ tag record  +-------+
  source files =======>| gtags |============>| gtags |===> GTAGS
                       |       |             +-------+
                       |       |=========================> GPATH
                       |       |
                       |       | tag record
                       |       |=========================> tempfile
                       +-------+
 Pass 2:
           tag record  +-------+ tag record  +-------+
 tempfile ============>| gtags |============>| gtags |===> GRTAGS
                       |       |             +-------+
                       |       | tag record  +-------+
    GTAGS ============>|       |============>| gtags |===> GSYMS
                       +-------+             +-------+

$ foreach cpumask ( 1 3 )
foreach? foreach label ( default builtin )
foreach? echo ==== cpumask:$cpumask label:$label ====
foreach? foreach cachesize ( 8388608 16777216 33554432 67108864 134217728 )
foreach? rm -fr linux-2.6.32; tar xfj ~/download/linux/linux-2.6.32.tar.bz2; 
sync
foreach? (cd linux-2.6.32; time taskset $cpumask env GTAGSLABEL=$label 
GTAGSCACHE=$cachesize gtags)
foreach? end
foreach? end
foreach? end
==== cpumask:1 label:default ====
124.195u 25.721s 3:54.49 63.9%  0+0k 0+0io 0pf+0w
125.515u 20.253s 3:12.91 75.5%  0+0k 0+0io 1pf+0w
125.891u 18.617s 2:41.68 89.3%  0+0k 0+0io 0pf+0w
125.991u 17.161s 2:35.00 92.3%  0+0k 0+0io 0pf+0w
125.943u 16.693s 2:39.12 89.6%  0+0k 0+0io 1pf+0w
==== cpumask:1 label:builtin ====
43.238u 15.580s 3:18.02 29.6%   0+0k 0+0io 4pf+0w
42.906u 10.388s 2:43.16 32.6%   0+0k 0+0io 1pf+0w
43.862u 7.188s 1:40.80 50.6%    0+0k 0+0io 1pf+0w
44.598u 5.008s 1:09.73 71.1%    0+0k 0+0io 0pf+0w
44.502u 4.820s 1:08.34 72.1%    0+0k 0+0io 2pf+0w
==== cpumask:3 label:default ====
121.243u 24.605s 3:08.54 77.3%  0+0k 0+0io 3pf+0w
121.339u 20.029s 2:44.64 85.8%  0+0k 0+0io 0pf+0w
122.543u 18.173s 2:24.77 97.1%  0+0k 0+0io 0pf+0w
123.579u 16.721s 2:20.03 100.1% 0+0k 0+0io 0pf+0w
122.915u 16.689s 2:21.39 98.7%  0+0k 0+0io 0pf+0w
==== cpumask:3 label:builtin ====
41.426u 15.696s 2:49.34 33.7%   0+0k 0+0io 4pf+0w
41.762u 10.520s 2:24.95 36.0%   0+0k 0+0io 0pf+0w
42.790u 6.400s 1:29.03 55.2%    0+0k 0+0io 2pf+0w
42.942u 4.904s 1:03.08 75.8%    0+0k 0+0io 1pf+0w
42.966u 4.612s 0:58.23 81.6%    0+0k 0+0io 1pf+0w


Diffstat:
 configure.ac                   |   13 +
 global/global.c                |  146 +++++++++++++++++++++
 gtags-parser/C.c               |  249 +++++++++++++-----------------------
 gtags-parser/Cpp.c             |  249 +++++++++++++-----------------------
 gtags-parser/Makefile.am       |    9 -
 gtags-parser/asm_parse.y       |  106 ++++++---------
 gtags-parser/asm_scan.l        |   52 +++----
 gtags-parser/gctags.c          |  220 ++++++++------------------------
 gtags-parser/gctags.h          |   62 ---------
 gtags-parser/java.c            |   46 ++----
 gtags-parser/parser.c          |  276 +++++++++++++++++++++++++++++++++++++++++
 gtags-parser/parser.h          |   52 +++++++
 gtags-parser/parser_internal.h |   53 +++++++
 gtags-parser/php.l             |  143 +++++++++------------
 gtags.conf.in                  |    8 +
 gtags/gtags.c                  |  257 +++++++++++++++++++++++++++++++++-----
 libutil/gtagsop.c              |  276 +++++++++++++++++++++++++++++++++--------
 libutil/gtagsop.h              |   12 +
 18 files changed, 1383 insertions(+), 846 deletions(-)

On Tue, 08 Dec 2009 01:59:55 +0900, Hideki IWAMOTO wrote...
> Hi.
> On Mon, 23 Nov 2009 22:49:05 +0900, Hideki IWAMOTO wrote...
> > When bree cache is small, current implementation of 2-pass mode is slower 
> > than default mode.
> 
> Because I cannot solve this problem, I will remove 2-pass mode from the 
> repository. 
> 
> ----
> Hideki IWAMOTO  address@hidden
> 
> 
> _______________________________________________
> Bug-global mailing list
> address@hidden
> http://lists.gnu.org/mailman/listinfo/bug-global

----
Hideki IWAMOTO  address@hidden

20100104-builtin-parser-2.patch
Description: Binary data