Re: [Logs-devel] LoGS userfriendliness?

[Jim Prewett]

> On 5/2/06, Jim Prewett <address@hidden> wrote:
> > 
> > Hi Vijay,
> > 
> > This all looks very nice!  Good work!  Now, let me rip it apart (I hope
> > you will return the favor soon :)
> > 
> > > Also, since speed is the key, it's better we directly implement the
> > > macro rather than have it generated.
> > 
> > I don't understand what you mean; I'm having problems parsing that
> > sentence.
> 
> Note to self: do not write when falling asleep.  What I meant was that
> we directly write the parser for this macro rather than have the
> parser generator do everything for us based on the grammar.  But first
> I must formally define the grammar.

Ok, are you worried here about the speed of, essentially, compiling the 
ruleset file?

by the time we're actually processing log messages, the work should be 
done, I think.  Am I misunderstanding?

In other words, I think this macro might be acceptable to create the 
rules as all the "extra work" is done at compilation time.

(defmacro bad-rule-generator (stuff)
  (let ((nonsense (loop for i from 1 to 1000000 do (waste-time))))
    `(make-instance 'rule ...))))

This reminds me, I need to understand how to compile a ruleset file.  I 
think wrapping most of it with eval-when will do the trick.  IIRC, right 
now the ruleset is always compiled when loaded. (err, maybe not the best 
idea)

> As soon as I woke up this morning, the first thing on my mind was
> File_Follower class :-)  I'll look at that more closely.

yeah, File_Followers are nice :)  I'm glad Dave thought them up :)  The 
basic idea is GNU tail with the -f option on steroids :)

What I like is that addding PBS-File-Follower was super-easy (basically, 
they have a different "roll-over" scheme than syslog files; a new file is 
created for each day and named with a timestamp).

> You know, I don't like it either.  It isn't as if the only possible
> actions are binary operations.  We should look at it later or write a
> separate macro that accepts infix syntax.

agreed.

> > The hard part, IIRC, is when you're trying to generate new rules. :)
> > 
> > Lets say I'm doing some security analysis work.  I set up some (likely
> > complex) scheme to detect port scans from IP addresses.  When I see such a
> > port scan, I want to watch to make sure that I don't get any successful
> > user logins from that host.  When I do see a login from one of those hosts
> > for username fred, I want to gather every message I can find that mentions
> > fred, or the IP.  (oooh!  I think thats a *nice* example!  I'm going to
> > find a paper or presentation to stick that in! :)
> > 
> > That can probably be accomplished in some way with all of LoGS,
> > Logsurfer, and SEC.
> 
> (LoGS with rule r1 matching regexp (format nil "fred|~A" +IP-regexp+) ; oops
>     and rule r1-1 matching regexp ...) ;; ok you're right.  I'll
> think about this too.

LOL!  I got to almoast exactly the same point and said OOPS! :)

I was using the keyword 'creating' instead of 'and' and I don't think your 
'with' is needed.

I think we can ditch the 'LoGS' in the above rule; something like: 
(rule named r1 ;; I think names should be optional
               ;; is r1 the symbol 'r1 or the string "r1" ?
      matching regexp (format NIL "portscan from ~A" +IP-REGEXP+)
      creating
      ;; i think the parens around rule2 are needed
      (rule named r2 ... ))

> > > we need to include words for contexts and ways to delete rules.
> > 
> > I think you're forgetting about the fact that rules have (gobally unique
> > and stored in a hash table by them) names.  If a name is not specified, a
> > symbol is generated (gensym) to be used for the name.
> 
> You're right.  Again, this morning I got the idea that we include rule
> names in the syntax itself as "with rule rule-name match ..."

yep! :)

> 
> > My idea there was that rules could be easily retrieved for deletion by
> > name, even if one was not supplied by the user.
> 
> Anonymous rules all right too :-)  What I'd meant was that the syntax
> take rule deletions.
> 
> (LoGS with rule r-n matching "something" action delete r-n) or
> something, I guess.

I think we're sort-of talking in circles here :)  

A rule could certainly look like this:

(let ((subordinate-rules NIL))
   (make-instance 'rule
                   :match #'some-match-func
                   :actions 
                    (list 
                      (lambda (message)
                        (if (should-delete-subordinates message)
                          (progn 
                            (mapcar #'delete subordinate-rules)
                            (setf subordinate-rules NIL))
                          (push (make-instance 'rule ...) 
                             subordinate-rules)))))))


Jim