Re: [PATCH] Add 2-pass mode to gtags

[Hideki IWAMOTO]

Hi.

> It seems that moving use of defined() from gtags-parser to gtags is bad idea.
> I will reimplement 2-pass mode in another way.

Rewritten version is attached. 
If an appropriate configuration variable is set, source files are parsed
only once in the anchor loading of htags 

Diffstat:
 ChangeLog                |   74 ++++++++---------
 global/global.c          |  136 ++++++++++++++++++++-----------
 gtags-parser/C.c         |  188 +++++++++++--------------------------------
 gtags-parser/Cpp.c       |  203 ++++++++++++-----------------------------------
 gtags-parser/asm_parse.y |   71 +++++-----------
 gtags-parser/asm_scan.l  |   22 ++---
 gtags-parser/gctags.c    |   41 +++++++--
 gtags-parser/gctags.h    |   49 +++++++++--
 gtags-parser/java.c      |   26 ++----
 gtags-parser/manual.in   |    8 +
 gtags-parser/php.l       |   67 +++++----------
 gtags.conf.in            |   11 +-
 gtags/gtags.c            |  120 +++++++++++++++++----------
 htags/anchor.c           |   96 ++++++++++++++--------
 htags/anchor.h           |    2
 libutil/Makefile.am      |    4
 libutil/format.h         |   29 ++++--
 libutil/global.h         |    1
 libutil/parser.c         |   84 +++++++++++++++++++
 libutil/parser.h         |   38 ++++++++
 libutil/pathconvert.c    |   18 +++-
 libutil/pathconvert.h    |    7 -
 22 files changed, 680 insertions(+), 615 deletions(-)

On Mon, 23 Nov 2009 22:49:05 +0900, Hideki IWAMOTO wrote...
> Hi.
> 
> > Since there are a lot of changes in the parser, the output should be seen by
> > a lot of people before the formal release, I think.
> 
> When bree cache is small, current implementation of 2-pass mode is slower 
> than default mode.
> 
> $ foreach cpumask ( 1 3 )
> foreach? foreach label ( default 2pass )
> foreach? echo ==== cpumask:$cpumask label:$label ====
> foreach? foreach cachesize ( 8388608 16777216 33554432 67108864 )
> foreach? rm -fr linux-2.6.31; tar xfj ~/download/linux/linux-2.6.31.tar.bz2; 
> sync
> foreach? (cd linux-2.6.31; time taskset $cpumask env GTAGSLABEL=$label 
> GTAGSCACHE=$cachesize gtags)
> foreach? end
> foreach? end
> foreach? end
> ==== cpumask:1 label:default ====
> 114.267u 24.085s 3:09.48 73.0%  0+0k 0+0io 1pf+0w
> 115.367u 20.521s 2:55.73 77.3%  0+0k 0+0io 0pf+0w
> 116.227u 18.081s 2:32.98 87.7%  0+0k 0+0io 0pf+0w
> 116.671u 16.093s 2:24.36 91.9%  0+0k 0+0io 0pf+0w
> ==== cpumask:1 label:2pass ====
> 91.353u 19.205s 3:48.79 48.3%   0+0k 0+0io 0pf+0w
> 92.317u 16.501s 3:05.40 58.6%   0+0k 0+0io 0pf+0w
> 92.797u 13.580s 2:08.52 82.7%   0+0k 0+0io 0pf+0w
> 93.929u 11.080s 1:56.03 90.4%   0+0k 0+0io 0pf+0w
> ==== cpumask:3 label:default ====
> 112.507u 23.593s 2:48.80 80.6%  0+0k 0+0io 1pf+0w
> 112.991u 18.965s 2:30.06 87.9%  0+0k 0+0io 0pf+0w
> 113.591u 17.305s 2:13.30 98.1%  0+0k 0+0io 0pf+0w
> 114.831u 16.113s 2:11.92 99.2%  0+0k 0+0io 0pf+0w
> ==== cpumask:3 label:2pass ====
> 88.773u 18.929s 3:17.01 54.6%   0+0k 0+0io 0pf+0w
> 90.085u 15.108s 2:35.38 67.6%   0+0k 0+0io 0pf+0w
> 90.877u 12.204s 1:44.62 98.5%   0+0k 0+0io 0pf+0w
> 91.217u 10.540s 1:29.74 113.3%  0+0k 0+0io 0pf+0w
> 
> It seems that moving use of defined() from gtags-parser to gtags is bad idea.
> I will reimplement 2-pass mode in another way.
> Since new 2-pass mode is implemented by adding tag type to output of 
> gtags-parser,
> the specifications of parser will be compatible with default mode.
> 
> 
> On Thu, 22 Oct 2009 15:44:28 +0900, Shigio YAMAGUCHI wrote...
> > Hi Hideki,
> > 
> > > To improve the performance of gtags, this patch adds 2-pass mode. 
> > > 
> > > The following three points are improved by moving use of function 
> > > defined()
> > > from gtags-parser to gtags. 
> > >  1. Decreases the frequency of parsing source file from three times to 
> > > two times.
> > >  2. Decreases reading of GTAGS by keeping cache of GTAGS
> > >     beyond the duration of parser process. 
> > >  3. Increases parallelism of gtags-parser and gtags on multiprocessor 
> > > system. 
> > 
> > Great!!! I named your code 'Giant steps'.
> > 
> > Since there are a lot of changes in the parser, the output should be seen by
> > a lot of people before the formal release, I think.
> > How about releasing twice like follows?
> > 
> >                         include Giant steps?    enable Giant steps by 
> > default?
> > ------------------------------------------------------------------------------
> > GLOBAL-5.7.7(beta)      Yes                     No
> > GLOBAL-5.8(formal)      Yes                     Yes
> > 
> > Thank you for your wonderful 'Giant steps'!
> > --
> > Shigio YAMAGUCHI <address@hidden>
> > PGP fingerprint: D1CB 0B89 B346 4AB6 5663  C4B6 3CA5 BBB3 57BE DDA3
> 
> ----
> Hideki IWAMOTO  address@hidden
> 
> 
> _______________________________________________
> Bug-global mailing list
> address@hidden
> http://lists.gnu.org/mailman/listinfo/bug-global

----
Hideki IWAMOTO  address@hidden

20091202-new-2pass.patch
Description: Binary data