Re: [gpsd-users] using the python gps module without scons install

[Gary E. Miller]

Yo shouldbe!

On Fri, 8 Mar 2019 02:51:21 +0000
shouldbe q931 <address@hidden> wrote:

> I did not have any issue building gpsd, which obviously needs the
> scons commands (commands that are either "scons" on its own, or
> "scons" followed by a space and then further words such as "scons
> timeservice=yes", or "scons check" or "scons install") to be run in
> the directory that gpsd has been cloned to or untarred to.

Good.

> > > Because I had misunderstood from the python documentation what
> > > path I needed to provide in $PYTHONPATH for ntpsec to "build"
> > > ntploggps,  
> >
> > Uh, no.  PYTHONPATH is not required to "build" ntploggps.  You just
> > need the prerequisites in your sys.path.  Which is easy if you
> > installed your distro packages for the prerequisites.  
> 
> The distro package for the python gps module also installs gpsd (its a
> dependancy), which would run counter to what I am attempting to
> acheive, yes I am aware that I am able to force the install without
> the dependancy, but I would prefer to build from source.

Ugh, lost me.  Why would you install the python gps module?  The
clockmaker script, or the normal soruce build, install it.

In fact, having the gps module from different sources WILL cause
problems.  The gpsd clients, and the gpsd python module is a matched
set.  DO NOT MIX AND MATCH.

> > > was under the misunderstanding that "something else" needed to be
> > > done after building gpsd, rather than as Rob pointed out, simply
> > > using the gpsd directory in $PYTHONPATH instead of gpsd/gps  
> >
> > Correct.  That is the clockmaker way.  Not the normal way.  
> 
> The clockmaker script does not build ntploggps, from your previous,
> and my reponse above, you would appear to be suggesting that to build
> ntploggps, one should install a distro gpsd rather than build gpsd
> from source, have I misunderstood something here ?

Yup, you misunderstand.

The 3 options, in increasing order of goodness:

1. Install your distro pgpsd package.

2. Use clockmaker to install gpsd.

3. Use the recommended procedure in build.txt to install gpsd from source.

Pick ONE.  Not TWO, Not THREE.  There is no mix and match.

> > > > Try not to confuse how you install and use gpsd with clockmaker,
> > > > versus the normal install and useage.
> > > >  
> > > I don't see a difference between the two apart from just calling
> > > scons on its own and calling scons with several "configure alike"
> > > statements  
> >
> > Yes, you do not see the differences.  scons is NOT one of them.  
> 
> One of what ?

One of the differences that you mentioned.

> > Try a normal gpsd install, using the build.txt file, and youll see
> > the differences.  
> 
> Well, yes, as per my original email on the subject, I'm very aware
> that if I run scons install it will copy the files for the gps module
> into /usr/local/lib/python2.7/dist-packages/gps/ .

Not my point.  That would be mixing options 2 and 3.

Do not do that!

> And as per my
> original email, I was attempting to avoid adding another step that
> required root

And yet, you persist...  clockmaker is for people that don't want root.
Installing it in /usr/local needs root.  No way around the duality.

> > > > > is waf/python also searching for a matching
> > > > > directory name, or is there something else in the gpsd
> > > > > directory that it is using ?  
> > > >
> > > > Lost me.  There is no waf/python.
> > > >  
> > > Apologies, I used "waf/python"  instead of "waf and/or python"  
> >
> > Still lost me.  waf is a python program,  there is no "or".  
> 
> I used and/or as I was unsure as to the correct terminology, waf is
> searching for the module, but waf is a python program, and using
> python to search, so in an attempt to reduce any frustration by my
> using the incorrect term, I provided both.

Which is just confusing to everyone.  Terminology is important.

> The text that I was asking the question about is "When a module named
> spam is imported, the interpreter first searches for a built-in module
> with that name. If not found, it then searches for a file named
> spam.py in a list of directories given by the variable sys.path"

Yes, I am very familiar with that.  Except that is not quite right.  I'll
avoid that digression.

> To highlight
> 1. searches for a .py file
> 2. in list of directories given by the variable sys.path

Yes, as we already discussed.  But you miss the third case, which gpsd uses.
The module is actually a directory.

> The python gps module appears (from the presence of the _init__.py) to
> be in the gpsd/gps directory, not in the gpsd directory.

Actually, the gpsd/gps directory IS the module.  All of it is the
module.  That is correct per the Python doc and over a decade of
experience with gpsd.

>  is python also searching for a matching directory name, or is there
> something else in the gpsd directory that it is using ?

As previously stated in several emails, sys.path is searched.  There is
no "something else".  The module is the directory gpsd/gps.

> > > > When you run a python program in the location that clockmaker
> > > > tells you to, then the gps is in that directory (.), which is
> > > > in sys.path. 
> > > In the clockmaker script, ./waf configure (with some additional
> > > options) is run in ntpsec/, not in gpsd/  
> >
> > Uh, no.  Look again.  A lot of "cd"ing going on.  
> 
> I disagree.

The script is clear.  I'm not sure how to make it clearer.

I suggest you study up on python modules.  Many things that you
are missing.

> There are three instance of waf in clockmaker
> Line 338  and 339 are clearly run in ntpsec/
> 
>     os.chdir("ntpsec")
>     os.system("./waf configure --refclock=nmea,pps,shm,gpsd")
>     os.system("./waf build")
>     os.chdir("..")

Of course.  ntpsec uses waf.  gpsd uses scons.  No point to scons
with ntpsec or waf with gpsd.

> Line 371 is also cleary run in ntpsec/
>     os.system("(cd ntpsec && ./waf install) && cp ntp.conf /etc")

Of course.  The point?

> Just in case you meant scons instead of waf, there are also three
> instances of scons in clockmaker

I'm very familiar with that.  Since gpsd needs scons to build the
clockmaker script clearly uses scons.

Your point?

> > > I'm not trying to make a "better" clockmaker script, I'm using it
> > > as an initial base to learn how to be able to a scripted install
> > > of gpsd and ntpsec running as separate services, and to have the
> > > logging for ntpviz installed as well  
> >
> > Which is to misunderstand the purpose of clockmaker.  That has led,
> > and will lead, to much confusion.  
> 
> Purpose of clockmaker "Configure an SBC as a single-purpose
> timeserver."

Yes.  With the left unsaid: for people too scared to use root and
install the standard source code way.

> My purposes, add logging of TDOP and number of visible satellites to
> clockmaker, learn what makes the clockmaker script "tick" (yes I know
> it's an appaling pun)

Oh, boy, are you lost...  Those are already there, just run cgps and
see.

BTW, TDOP is pretty useless...

> > Do it the "clockmaker" way, or do it the normal way.  Mixing them up
> > is confusing you.  
> 
> What was confusing me was a misunderstanding on the path that was
> required for $PYTHONPATH

So, all clear now?  Your above statements do not make me think so.

> > > I have to disagree, when "scons install" is run as root it puts
> > > files in several paths that are not in the user's home directory
> > > /usr/local/, /usr/local/lib/
> > > /usr/local/lib/python2.7/dist-packages/gps, and /usr/local/include
> > > I have seen this bahaviour under rasbian on a pi and in the
> > > Ubuntu VM  
> >
> > Which is why the clockmaker script does not tell you to do that!  
> 
> from the howto
> This step can be automated with "./clockmaker --install" done as root.
> # cd gpsd; scons install

As it says: "can be" but the silent recommendation is not to.  I recommend
that you do not.

> if the instructions from build.txt are followed, the same paths have
> files copied to them, build.txt includes "The default value is
> "/usr/local"",

Yup.  Directly from the FHS.

> could it be that you are using a different distro or
> have other configuration options provide different results ? such as
> "scons prefix=/home/pi"

Lost me.  Install path has nothing to do with your distro.

By default, observing the FHS:

1. packages install in /usr

2. source code packages install in /usr

3. people scared to use root, or lacking root, install in their
   home directoy (/home/pi).

We are talking about #2 and #3, not #1.  Also, you seem to be conflating @3 and 
#3.

> > Don't mix up recipes until you are a good chef.  The clockmaker
> > purpose and install and not the same as the normal purpose and
> > install.  
> 
> 1. I would say that taking something that works one way, and learning
> how to modify it to work a different way, is a key part of becoming a
> chef, another key part is experimention.

Yes.  But only when you can keep the different methods separate in your
head.  An advanced skill you have not shown yet.

> 2. I presume you meant "and install are not the same", to which I
> would say that the original purpose has been subverted.

Uh, lost me.  Best to never think I did not say what I meant.  This
has led you astray many times here already.

> > Or just use "scons uninstall".  Can't get easier than that.  
> 
> I had not seen "scons uninstall" in build.txt

And truly it is not there.  There are also no elephants there.  Your point?

This is UNIX, until you have read ALL the doc, you can't complain that
one part of the doc does not say what you want it to say.

OTOH, feel free to submit a patch if you find a way to improve the doc.

> > > > I see an ntpsec configure, but no gpsd configure?
> > > >  
> > > Sorry, you've lost me, I was under the obviously mistaken
> > > understanding that scons without install was a combined configure
> > > and build, with the configure "options" being passed on the
> > > command line.  
> >
> > Yes, mistaken.  Not how you think. You sent me your ntpsec config,
> > not your gpsd config.  We are talking about gpsd here.  
> 
> Ok, so the "timeservice=yes" are build options, not configure options,

Uh, exactly backwards.  timeservice=yes is a configure option, not
a build option.  You are confused since if no configure is done
explicitly it is done implicitly.

> there is no equivalent to ./waf configure in either build.txt or
> clockmaker,

Once again.  waf is for ntpsec.  scons is for gpsd.

Try to keep the differences in mind.  UNtil you keep the separate in your
your head you can't make any inferences.

> I do not understand what you asked to see considering that

Clearly.

> I had also posted the bash script I had used, which I would have hoped
> made it obvious exactly what commands I was running to duplicate what
> I was seeing.

Sorry, too many things for me to do rather than helping you debug
your bash.  Get it working, then I'll look at it.  Until then, only
questions.

> Just to reiterate, I was not suggesting that either gpsd or ntpsec had
> a fault, I was trying to understand why what I had thought would work
> did not work, and from the reply from Rob, I now have something that
> works.

Well, then, all done then?

> Now I have found
> https://chrisyeh96.github.io/2017/08/08/definitive-guide-python-imports.html#basics-of-the-python-import-and-syspath
> which has in it
> 
> "Technically, Python’s documentation is incomplete. The interpreter
> will not only look for a file (i.e. module) named spam.py, it will
> also look for a folder (i.e. package) named spam"

Not exactly true, bordering onn misleading.  I sugest you read the
Python doc again to see how that is not exactly correct.

That stuff is indeed in the python doc.  Not easy to grok, but it is
there.

> So I made three mistakes, a newbie mistake to expect the python.org
> documentation to be complete,

I have to disagree.  It is in the doc, you just missed it.  A very easy
thing to do given the size and lack of organization of the doc.

And your understanding is still not correct as it applies to gpsd.

> a typo in just the wrong place in the
> first post, and not to show how I could reproduce what I was seeing on
> a reduced system (such as the bash script) in my first post. I have
> learnt from these.

Good.

> I can now go back to hacking around on clockmaker (-:

If you do get something that seems to work, post it here, maybe it
will be useful for someone else.  The extensive build doc for gpsd and
ntpsd are there because there are many different ways to configure and
build.  Not as many ways as there need to be.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
        address@hidden  Tel:+1 541 382 8588

            Veritas liberabit vos. -- Quid est veritas?
    "If you can’t measure it, you can’t improve it." - Lord Kelvin

pgphrBl0ISOEp.pgp
Description: OpenPGP digital signature