patch-gnuradio
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Patch-gnuradio] Enhancing Gnuradio Doxygen Generated Official Documenta


From: Firas Abbas
Subject: [Patch-gnuradio] Enhancing Gnuradio Doxygen Generated Official Documentation
Date: Sat, 15 Dec 2007 22:49:07 -0800 (PST)

Dear Friends,


During the past period, I did a study on how to enhance our automatically doxygen generated gnuradio  documentation. I came up with :

1) The gnuradio code is mainly consists of  C++ code ( *.cc, *.h) and python code ( *.py ) (this is already known fact).

2) The C++ code classes and functions documentation are formated  to confirm with doxygen  special  commands  [which starts with a backslash (\) or an at-sign (@)]. Thus, no problems in generating c++ documentation using doxygen.

3) The problem is in python code. The gnuradio python code documentation  are formated  to  confirm with epydoc markup language and not doxygen.
This formating looks ugly when interpreted by doxygen, so we cannot generate clear and nicely look python documentation using doxygen.

4) These facts leads to :

a) We have to use doxygen for C++ documentation, and epydoc for python code. This is not desirable, because we will end up with two separated documents and moving from one to the other will make us sick!!.

b) Or, rewrite all python documentation to confirm with doxygen format. In this case we will loose the functionality of __doc__ and will may not be able to use epydoc any more.

c) Or, to document everything twice (this will kill us).

d) Or (preferred), to use a filter program that will modify the python code input to the doxygen in runtime. 
This will makes it possible to use the Doxygen syntax inside python docstrings and automatically generate API documentation out of it, instead of being forced to use the non-native documentation blocks over the standard docstrings or to document everything twice.

5) Fortunately, doxygen has the ability (a really great feature) to automatically run a filter program during runtime on all its input files before parsing.

6) Doing a Google search for such python filter program, I found two programs:

a) The first one was old and not updated (written in 2003 and it is not working well with the currently updated doxygen versions).

b) The second one (doxypy.py) looks good.  It is a python code written under GPL V2.
The program description says : "The doxypy is an input filter for Doxygen, it preprocesses python files so that docstrings of classes and functions are reformatted into Doxygen-conform documentation blocks"

7) The done work was :

a) Modifying the doxygen-in file (Doxyfile.in) to use the new filter.

b) Redefining the paths in this file to exclude unwanted swig automatically generated python codes (which  does not  contain  any documentation).

c) Modifying the  doxygen  group definitions  file  (group_defs.dox) to classify  gnuradio  blocks into a new group categories for ease of exploring.

If Eric and Johnathan, approves the new documentation categories and/or suggested a modifications,  then I will slightly modify (and submit to patch-gnuradio) most of gnuradio written source code files (about 80% - 90%) to be confirmed with the new suggested categories (putting the @ingroup documentation line).

d) In this case (since I will edit all source files anyway ), I will try also to add a new documentations to these files (I will need more time to do it).

e) The new generated doxygen html files can be found in :

http://rapidshare.com/files/76826593/gnuradio-html-documentation.tar.gz


8) To generate the new doxygen documentation files in your PC (if gnuradio is already installed):

a) Copy the attached files [doxypy.py ,  Doxyfile , group_defs.dox and main_page.dox]  into  your  xxxxx/gnuradio-core/doc  folder.

b)  Change directory to xxxxx/gnuradio-core/doc.

c) Run  doxygen program without any arguments.

9) For new gnuradio installation, the attached file [Doxyfile.in ] should be installed into  your  xxxxx/gnuradio-core/doc folder along with the [doxypy.py , group_defs.dox and main_page.dox] files .

Note:
I generated the [Doxyfile] file using doxygen version 1.5.3.

10) Waiting for suggestions/modifications before I start modifying the gnuradio source files.


Regards,

Firas Abbas


#!/usr/bin/python

__applicationName__ = "doxypy"
__blurb__ = """
doxypy is an input filter for Doxygen. It preprocesses python
files so that docstrings of classes and functions are reformatted
into Doxygen-conform documentation blocks.
"""

__doc__ = __blurb__ + \
"""
In order to make Doxygen preprocess files through doxypy, simply
add the following lines to your Doxyfile:
        FILTER_SOURCE_FILES = YES
        INPUT_FILTER = "python /path/to/doxypy.py"
"""

__version__ = "0.3rc1"
__date__ = "1st December 2007"
__website__ = "http://code.foosel.org/doxypy";

__author__ = (
        "Philippe 'demod' Neumann (doxypy at demod dot org)",
        "Gina 'foosel' Haeussge (gina at foosel dot net)" 
)

__licenseName__ = "GPL v2"
__license__ = """This program is free software: you can redistribute it and/or 
modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.
"""

import sys
import re

from optparse import OptionParser, OptionGroup

class FSM(object):
        """     FSM implements a finite state machine. Transitions are given as
                4-tuples, consisting of an origin state, a target state, a 
condition
                for the transition (given as a reference to a function which 
gets called
                with a given piece of input) and a pointer to a function to be 
called
                upon the execution of the given transition. 
        """
        
        def __init__(self, start_state=None, transitions=[]):
                self.transitions = transitions
                self.current_state = start_state
                self.current_input = None
                self.current_transition = None
                
        def setStartState(self, state):
                self.current_state = state

        def addTransition(self, from_state, to_state, condition, callback):
                self.transitions.append([from_state, to_state, condition, 
callback])
                
        def makeTransition(self, input):
                """ Makes a transition based on the given input.
                        @param  input   input to parse by the FSM
                """
                for transition in self.transitions:
                        [from_state, to_state, condition, callback] = transition
                        if from_state == self.current_state:
                                match = condition(input)
                                if match:
                                        self.current_state = to_state
                                        self.current_input = input
                                        self.current_transition = transition
                                        callback(match)
                                        return


class Doxypy(object):
        def __init__(self):
                self.start_single_comment_re = re.compile("^\s*(''')")
                self.end_single_comment_re = re.compile("(''')\s*$")
                
                self.start_double_comment_re = re.compile("^\s*(\"\"\")")
                self.end_double_comment_re = re.compile("(\"\"\")\s*$")
                
                self.single_comment_re = re.compile("^\s*(''').*(''')\s*$")
                self.double_comment_re = 
re.compile("^\s*(\"\"\").*(\"\"\")\s*$")
                
                self.defclass_re = re.compile("^(\s*)(def .+:|class .+:)")
                self.empty_re = re.compile("^\s*$")
                self.hashline_re = re.compile("^\s*#.*$")
                self.importline_re = re.compile("^\s*(import |from .+ import)")

                ## Transition list format
                #  ["FROM", "TO", condition, action]
                transitions = [
                        ### FILEHEAD
                        
                        # single line comments
                        ["FILEHEAD", "FILEHEAD", self.single_comment_re.search, 
self.appendCommentLine],
                        ["FILEHEAD", "FILEHEAD", self.double_comment_re.search, 
self.appendCommentLine],
                        
                        # multiline comments
                        ["FILEHEAD", "FILEHEAD_COMMENT_SINGLE", 
self.start_single_comment_re.search, self.appendCommentLine],
                        ["FILEHEAD_COMMENT_SINGLE", "FILEHEAD", 
self.end_single_comment_re.search, self.appendCommentLine],
                        ["FILEHEAD_COMMENT_SINGLE", "FILEHEAD_COMMENT_SINGLE", 
self.catchall, self.appendCommentLine],
                        ["FILEHEAD", "FILEHEAD_COMMENT_DOUBLE", 
self.start_double_comment_re.search, self.appendCommentLine],
                        ["FILEHEAD_COMMENT_DOUBLE", "FILEHEAD", 
self.end_double_comment_re.search, self.appendCommentLine],
                        ["FILEHEAD_COMMENT_DOUBLE", "FILEHEAD_COMMENT_DOUBLE", 
self.catchall, self.appendCommentLine],
                        
                        # other lines
                        ["FILEHEAD", "FILEHEAD", self.empty_re.search, 
self.appendFileheadLine],
                        ["FILEHEAD", "FILEHEAD", self.hashline_re.search, 
self.appendFileheadLine],
                        ["FILEHEAD", "FILEHEAD", self.importline_re.search, 
self.appendFileheadLine],
                        ["FILEHEAD", "DEFCLASS", self.defclass_re.search, 
self.resetCommentSearch],
                        ["FILEHEAD", "DEFCLASS_BODY", self.catchall, 
self.appendFileheadLine],

                        ### DEFCLASS
                        
                        # single line comments
                        ["DEFCLASS", "DEFCLASS_BODY", 
self.single_comment_re.search, self.appendCommentLine],
                        ["DEFCLASS", "DEFCLASS_BODY", 
self.double_comment_re.search, self.appendCommentLine],
                        
                        # multiline comments
                        ["DEFCLASS", "COMMENT_SINGLE", 
self.start_single_comment_re.search, self.appendCommentLine],
                        ["COMMENT_SINGLE", "DEFCLASS_BODY", 
self.end_single_comment_re.search, self.appendCommentLine],
                        ["COMMENT_SINGLE", "COMMENT_SINGLE", self.catchall, 
self.appendCommentLine],
                        ["DEFCLASS", "COMMENT_DOUBLE", 
self.start_double_comment_re.search, self.appendCommentLine],
                        ["COMMENT_DOUBLE", "DEFCLASS_BODY", 
self.end_double_comment_re.search, self.appendCommentLine],
                        ["COMMENT_DOUBLE", "COMMENT_DOUBLE", self.catchall, 
self.appendCommentLine],

                        # other lines
                        ["DEFCLASS", "DEFCLASS", self.empty_re.search, 
self.appendDefclassLine],
                        ["DEFCLASS", "DEFCLASS", self.defclass_re.search, 
self.resetCommentSearch],
                        ["DEFCLASS", "DEFCLASS_BODY", self.catchall, 
self.stopCommentSearch],
                        
                        ### DEFCLASS_BODY
                        
                        ["DEFCLASS_BODY", "DEFCLASS", self.defclass_re.search, 
self.startCommentSearch],
                        ["DEFCLASS_BODY", "DEFCLASS_BODY", self.catchall, 
self.appendNormalLine],
                ]
                
                self.fsm = FSM("FILEHEAD", transitions)
                
                self.output = []
                
                self.comment = []
                self.filehead = []
                self.defclass = []
                self.indent = ""
        
        def catchall(self, input):
                """     The catchall-condition, always returns true. """
                return True
        
        def resetCommentSearch(self, match):
                """ Restarts a new comment search for a different triggering 
line.
                        Closes the current commentblock and starts a new 
comment search.
                """
                self.__closeComment()
                self.startCommentSearch(match)
        
        def startCommentSearch(self, match):
                """ Starts a new comment search.
                        Saves the triggering line, resets the current comment 
and saves
                        the current indentation.
                """
                self.defclass = [self.fsm.current_input]
                self.comment = []
                self.indent = match.group(1)
        
        def stopCommentSearch(self, match):
                """ Stops a comment search.
                        Closes the current commentblock, resets the triggering 
line and
                        appends the current line to the output.
                """
                self.__closeComment()
                
                self.defclass = []
                self.output.append(self.fsm.current_input)
        
        def appendFileheadLine(self, match):
                """     Appends a line in the FILEHEAD state.
                        Closes the open comment block, resets it and appends 
the current line.
                """ 
                self.__closeComment()
                self.comment = []
                self.output.append(self.fsm.current_input)

        def appendCommentLine(self, match):
                """     Appends a comment line.
                        The comment delimiter is removed from multiline start 
and ends as
                        well as singleline comments.
                """
                (from_state, to_state, condition, callback) = 
self.fsm.current_transition
                
                # single line comment
                if (from_state == "DEFCLASS" and to_state == "DEFCLASS_BODY") \
                or (from_state == "FILEHEAD" and to_state == "FILEHEAD"):
                        # remove comment delimiter from begin and end of the 
line
                        activeCommentDelim = match.group(1)
                        line = self.fsm.current_input
                        
self.comment.append(line[line.find(activeCommentDelim)+len(activeCommentDelim):line.rfind(activeCommentDelim)])
                        if (to_state == "DEFCLASS_BODY"):
                                self.__closeComment()
                                self.defclass = []
                # multiline start
                elif from_state == "DEFCLASS" or from_state == "FILEHEAD":
                        # remove comment delimiter from begin of the line
                        activeCommentDelim = match.group(1)
                        line = self.fsm.current_input
                        
self.comment.append(line[line.find(activeCommentDelim)+len(activeCommentDelim):])
                # multiline end
                elif to_state == "DEFCLASS_BODY" or to_state == "FILEHEAD":
                        # remove comment delimiter from end of the line
                        activeCommentDelim = match.group(1)
                        line = self.fsm.current_input
                        
self.comment.append(line[0:line.rfind(activeCommentDelim)])
                        if (to_state == "DEFCLASS_BODY"):
                                self.__closeComment()
                                self.defclass = []
                # in multiline comment
                else:
                        # just append the comment line
                        self.comment.append(self.fsm.current_input)
        
        def appendNormalLine(self, match):
                """ Appends a line to the output. """
                self.output.append(self.fsm.current_input)
                
        def appendDefclassLine(self, match):
                """ Appends a line to the triggering block. """
                self.defclass.append(self.fsm.current_input)
        
        def __closeComment(self):
                """ Appends any open comment block and triggering block to the 
output. """
                if self.comment:
                        block = self.makeCommentBlock()
                        self.output.extend(block)
                if self.defclass:
                        self.output.extend(self.defclass)

        def makeCommentBlock(self):
                """ Indents the current comment block with respect to the 
current
                        indentation level.
                        @returns a list of indented comment lines
                """
                doxyStart = "##"
                commentLines = self.comment
                
                if options.strip:
                        commentLines = map(str.strip, commentLines)
                
                commentLines = map(lambda x: "%s# %s" % (self.indent, x), 
commentLines)
                l = [self.indent + doxyStart]
                l.extend(commentLines)
                         
                return l
        
        def parse(self, input):
                """ Parses a python file given as input string and returns the 
doxygen-
                        compatible representation.
                        @param  input   the python code to parse
                        @returns the modified python code
                """ 
                lines = input.split("\n")
                
                for line in lines:
                        self.fsm.makeTransition(line)
                
                if self.fsm.current_state == "DEFCLASS":
                        self.__closeComment()
                
                return "\n".join(self.output)
        
def loadFile(filename):
        """ Loads file "filename" and returns the content.
                @param   filename       The name of the file to load
                @returns the content of the file.
        """
        f = open(filename, 'r')
        
        try:
                content = f.read()
                return content
        finally:
                f.close()

def optParse():
        """ Parses commandline options. """
        parser = OptionParser(prog=__applicationName__, version="%prog " + 
__version__)
        
        parser.set_usage("%prog [options] filename")
        parser.add_option("--trim", "--strip",
                action="store_true", dest="strip",
                help="enables trimming of docstrings, might be useful if you 
get oddly spaced output"
        )
        
        ## parse options
        global options
        (options, filename) = parser.parse_args()
        
        if not filename:
                print >>sys.stderr, "No filename given."
                sys.exit(-1)
        
        return filename[0]

def main():
        """ Opens the file given as first commandline argument and processes it,
                then prints out the processed file.
        """
        filename = optParse()
        
        try:
                input = loadFile(filename)
        except IOError, (errno, msg):
                print >>sys.stderr, msg
                sys.exit(-1)
        
        fsm = Doxypy()
        output = fsm.parse(input)
        print output

if __name__ == "__main__":
        main()

Attachment: Doxyfile.in
Description: pat1662019405

Attachment: Doxyfile
Description: pat235785580

Attachment: group_defs.dox
Description: pat1945927508

Attachment: main_page.dox
Description: pat720811282


reply via email to

[Prev in Thread] Current Thread [Next in Thread]