Re: [PATCH] Add 2-pass mode to gtags

[Hideki IWAMOTO]

Hi.
> By the way, it seems that you have done the following modifications at a time.
> 
> 1. built-in parser
> 2. giant steps
> 3. 1pass parsing using temporary file
> 4. external writer
> 
> Though these might be supplemented each other, they are basically separate.
> Is it difficult to implement these optimizations one by one?

Four incremental patch files are attached. They are used as follows. 


 [ Current CVS version ]
Pass 1:
                   +--------------+  ctags-x   +-------+
  source files ===>| gtags-parser |===========>| gtags |===> GTAGS
                   |              |            |       |===> GPATH
                   +--------------+            +-------+
Pass 2:
                   +--------------+  ctags-x   +-------+
  source files ===>| gtags-parser |===========>| gtags |===> GRTAGS
         GTAGS ===>|              |  GPATH ===>|       |
                   +--------------+            +-------+
Pass 3:
                   +--------------+  ctags-x   +-------+
  source files ===>| gtags-parser |===========>| gtags |===> GSYMS
         GTAGS ===>|              |  GPATH ===>|       |
                   +--------------+            +-------+

                |
                | Apply 20100109-builtin-parser.patch
                |
                V

Pass 1:
                       +-------+
  source files =======>| gtags |============> GTAGS
                       |       |============> GPATH
                       +-------+
Pass 2:
                       +-------+
  source files =======>| gtags |============> GRTAGS
         GTAGS =======>|       |
         GPATH =======>|       |
                       +-------+
Pass 3:
                       +-------+
  source files =======>| gtags |============> GSYMS
         GTAGS =======>|       |
         GPATH =======>|       |
                       +-------+

                |
                | Apply 20100109-external-writer.patch
                |
                V

Pass 1:
                       +-------+ tag record  +-------+
  source files =======>| gtags |============>| gtags |===> GTAGS
                       |       |             +-------+
                       |       |=========================> GPATH
                       +-------+
Pass 2:
                       +-------+ tag record  +-------+
  source files =======>| gtags |============>| gtags |===> GRTAGS
         GTAGS =======>|       |             +-------+
         GPATH =======>|       |
                       +-------+
Pass 3:
                       +-------+ tag record  +-------+
  source files =======>| gtags |============>| gtags |===> GSYMS
         GTAGS =======>|       |             +-------+
         GPATH =======>|       |
                       +-------+

                |
                | Apply 20100108-2pass-parsing.patch
                |
                V
Pass 1:
                       +-------+ tag record  +-------+
  source files =======>| gtags |============>| gtags |===> GTAGS
                       |       |             +-------+
                       |       |=========================> GPATH
                       +-------+
Pass 2:
                       +-------+ tag record  +-------+
  source files =======>| gtags |============>| gtags |===> GRTAGS
                       |       |             +-------+
         GTAGS =======>|       | tag record  +-------+
         GPATH =======>|       |============>| gtags |===> GSYMS
                       +-------+             +-------+

                |
                | Apply 20100108-1pass-parsing.patch
                |
                V

Pass 1:
                       +-------+ tag record  +-------+
  source files =======>| gtags |============>| gtags |===> GTAGS
                       |       |             +-------+
                       |       |=========================> GPATH
                       |       | tag record
                       |       |=========================> tempfile
                       +-------+
Pass 2:
           tag record  +-------+ tag record  +-------+
 tempfile ============>| gtags |============>| gtags |===> GRTAGS
                       |       |             +-------+
                       |       | tag record  +-------+
    GTAGS ============>|       |============>| gtags |===> GSYMS
                       +-------+             +-------+


Diffstat:
$ diffstat 20100109-builtin-parser.patch
 configure.ac              |   11
 global/global.c           |  145 +++++++++
 gtags.conf.in             |    8
 gtags/gtags.c             |  125 +++++++-
 gtags/manual.in           |    2
 libutil/.cvsignore        |    4
 libutil/C.c               |  664 +++++++++++++++++++++++++++++++++++++++++++
 libutil/Cpp.c             |  706 ++++++++++++++++++++++++++++++++++++++++++++++
 libutil/Makefile.am       |    7
 libutil/asm_parse.y       |  166 ++++++++++
 libutil/asm_scan.l        |  210 +++++++++++++
 libutil/global.h          |    1
 libutil/gtagsop.c         |   87 +++++
 libutil/gtagsop.h         |    2
 libutil/java.c            |  163 ++++++++++
 libutil/parser.c          |  277 ++++++++++++++++++
 libutil/parser.h          |   53 +++
 libutil/parser_internal.h |   52 +++
 libutil/php.l             |  357 +++++++++++++++++++++++
 reconf.sh                 |   13
 20 files changed, 3030 insertions(+), 23 deletions(-)
$ diffstat 20100109-external-writer.patch
 gtags/gtags.c     |   36 ++++++++++++++++++-
 libutil/gtagsop.c |  100 ++++++++++++++++++++++++++++++++++++++++++++++--------
 libutil/gtagsop.h |    7 +++
 3 files changed, 126 insertions(+), 17 deletions(-)
$ diffstat 20100108-2pass-parsing.patch
 gtags.c |  135 +++++++++++++++++++++++++++++++++++++++++-----------------------
 1 files changed, 87 insertions(+), 48 deletions(-)
$ diffstat 20100108-1pass-parsing.patch
 gtags/gtags.c     |  103 +++++++++++++++++++-----------------------------------
 libutil/gtagsop.c |   35 ++++++++++++++++++
 libutil/gtagsop.h |    1
 3 files changed, 74 insertions(+), 65 deletions(-)

On Wed, 06 Jan 2010 20:56:54 +0900, Shigio YAMAGUCHI wrote...
> Hi
> > Hi.
> > This is an experimental patch that adds builtin parser. 
> > The performance of making ttag files is greatly improved by building parser 
> > into gtags. 
> > When btree chache is big enough, speed may improve than double.
> > Even when btree cache is small, it is faster than the default mode. 
> 
> It's very fast! It's great optimization.
> 
> By the way, it seems that you have done the following modifications at a time.
> 
> 1. built-in parser
> 2. giant steps
> 3. 1pass parsing using temporary file
> 4. external writer
> 
> Though these might be supplemented each other, they are basically separate.
> Is it difficult to implement these optimizations one by one?
> 
> --
> Shigio YAMAGUCHI <address@hidden>
> PGP fingerprint: D1CB 0B89 B346 4AB6 5663  C4B6 3CA5 BBB3 57BE DDA3

----
Hideki IWAMOTO  address@hidden

20100109-builtin-parser.patch
Description: Binary data

20100109-external-writer.patch
Description: Binary data

20100108-2pass-parsing.patch
Description: Binary data

20100108-1pass-parsing.patch
Description: Binary data