Re: [gpsd-users] using the python gps module without scons install

[shouldbe q931]

Hi Rob and Gary,

Many thanks for your help, it is greatly appreciated.

Cheers

Arne

On Fri, Mar 8, 2019 at 3:40 AM Gary E. Miller <address@hidden> wrote:
>
> Yo shouldbe!
>
> On Fri, 8 Mar 2019 02:51:21 +0000
> shouldbe q931 <address@hidden> wrote:
>
> > I did not have any issue building gpsd, which obviously needs the
> > scons commands (commands that are either "scons" on its own, or
> > "scons" followed by a space and then further words such as "scons
> > timeservice=yes", or "scons check" or "scons install") to be run in
> > the directory that gpsd has been cloned to or untarred to.
>
> Good.
>
> > > > Because I had misunderstood from the python documentation what
> > > > path I needed to provide in $PYTHONPATH for ntpsec to "build"
> > > > ntploggps,
> > >
> > > Uh, no.  PYTHONPATH is not required to "build" ntploggps.  You just
> > > need the prerequisites in your sys.path.  Which is easy if you
> > > installed your distro packages for the prerequisites.
> >
> > The distro package for the python gps module also installs gpsd (its a
> > dependancy), which would run counter to what I am attempting to
> > acheive, yes I am aware that I am able to force the install without
> > the dependancy, but I would prefer to build from source.
>
> Ugh, lost me.  Why would you install the python gps module?  The
> clockmaker script, or the normal soruce build, install it.
>
> In fact, having the gps module from different sources WILL cause
> problems.  The gpsd clients, and the gpsd python module is a matched
> set.  DO NOT MIX AND MATCH.
>
> > > > was under the misunderstanding that "something else" needed to be
> > > > done after building gpsd, rather than as Rob pointed out, simply
> > > > using the gpsd directory in $PYTHONPATH instead of gpsd/gps
> > >
> > > Correct.  That is the clockmaker way.  Not the normal way.
> >
> > The clockmaker script does not build ntploggps, from your previous,
> > and my reponse above, you would appear to be suggesting that to build
> > ntploggps, one should install a distro gpsd rather than build gpsd
> > from source, have I misunderstood something here ?
>
> Yup, you misunderstand.
>
> The 3 options, in increasing order of goodness:
>
> 1. Install your distro pgpsd package.
>
> 2. Use clockmaker to install gpsd.
>
> 3. Use the recommended procedure in build.txt to install gpsd from source.
>
> Pick ONE.  Not TWO, Not THREE.  There is no mix and match.
>
> > > > > Try not to confuse how you install and use gpsd with clockmaker,
> > > > > versus the normal install and useage.
> > > > >
> > > > I don't see a difference between the two apart from just calling
> > > > scons on its own and calling scons with several "configure alike"
> > > > statements
> > >
> > > Yes, you do not see the differences.  scons is NOT one of them.
> >
> > One of what ?
>
> One of the differences that you mentioned.
>
> > > Try a normal gpsd install, using the build.txt file, and youll see
> > > the differences.
> >
> > Well, yes, as per my original email on the subject, I'm very aware
> > that if I run scons install it will copy the files for the gps module
> > into /usr/local/lib/python2.7/dist-packages/gps/ .
>
> Not my point.  That would be mixing options 2 and 3.
>
> Do not do that!
>
> > And as per my
> > original email, I was attempting to avoid adding another step that
> > required root
>
> And yet, you persist...  clockmaker is for people that don't want root.
> Installing it in /usr/local needs root.  No way around the duality.
>
> > > > > > is waf/python also searching for a matching
> > > > > > directory name, or is there something else in the gpsd
> > > > > > directory that it is using ?
> > > > >
> > > > > Lost me.  There is no waf/python.
> > > > >
> > > > Apologies, I used "waf/python"  instead of "waf and/or python"
> > >
> > > Still lost me.  waf is a python program,  there is no "or".
> >
> > I used and/or as I was unsure as to the correct terminology, waf is
> > searching for the module, but waf is a python program, and using
> > python to search, so in an attempt to reduce any frustration by my
> > using the incorrect term, I provided both.
>
> Which is just confusing to everyone.  Terminology is important.
>
> > The text that I was asking the question about is "When a module named
> > spam is imported, the interpreter first searches for a built-in module
> > with that name. If not found, it then searches for a file named
> > spam.py in a list of directories given by the variable sys.path"
>
> Yes, I am very familiar with that.  Except that is not quite right.  I'll
> avoid that digression.
>
> > To highlight
> > 1. searches for a .py file
> > 2. in list of directories given by the variable sys.path
>
> Yes, as we already discussed.  But you miss the third case, which gpsd uses.
> The module is actually a directory.
>
> > The python gps module appears (from the presence of the _init__.py) to
> > be in the gpsd/gps directory, not in the gpsd directory.
>
> Actually, the gpsd/gps directory IS the module.  All of it is the
> module.  That is correct per the Python doc and over a decade of
> experience with gpsd.
>
> >  is python also searching for a matching directory name, or is there
> > something else in the gpsd directory that it is using ?
>
> As previously stated in several emails, sys.path is searched.  There is
> no "something else".  The module is the directory gpsd/gps.
>
> > > > > When you run a python program in the location that clockmaker
> > > > > tells you to, then the gps is in that directory (.), which is
> > > > > in sys.path.
> > > > In the clockmaker script, ./waf configure (with some additional
> > > > options) is run in ntpsec/, not in gpsd/
> > >
> > > Uh, no.  Look again.  A lot of "cd"ing going on.
> >
> > I disagree.
>
> The script is clear.  I'm not sure how to make it clearer.
>
> I suggest you study up on python modules.  Many things that you
> are missing.
>
> > There are three instance of waf in clockmaker
> > Line 338  and 339 are clearly run in ntpsec/
> >
> >     os.chdir("ntpsec")
> >     os.system("./waf configure --refclock=nmea,pps,shm,gpsd")
> >     os.system("./waf build")
> >     os.chdir("..")
>
> Of course.  ntpsec uses waf.  gpsd uses scons.  No point to scons
> with ntpsec or waf with gpsd.
>
> > Line 371 is also cleary run in ntpsec/
> >     os.system("(cd ntpsec && ./waf install) && cp ntp.conf /etc")
>
> Of course.  The point?
>
> > Just in case you meant scons instead of waf, there are also three
> > instances of scons in clockmaker
>
> I'm very familiar with that.  Since gpsd needs scons to build the
> clockmaker script clearly uses scons.
>
> Your point?
>
> > > > I'm not trying to make a "better" clockmaker script, I'm using it
> > > > as an initial base to learn how to be able to a scripted install
> > > > of gpsd and ntpsec running as separate services, and to have the
> > > > logging for ntpviz installed as well
> > >
> > > Which is to misunderstand the purpose of clockmaker.  That has led,
> > > and will lead, to much confusion.
> >
> > Purpose of clockmaker "Configure an SBC as a single-purpose
> > timeserver."
>
> Yes.  With the left unsaid: for people too scared to use root and
> install the standard source code way.
>
> > My purposes, add logging of TDOP and number of visible satellites to
> > clockmaker, learn what makes the clockmaker script "tick" (yes I know
> > it's an appaling pun)
>
> Oh, boy, are you lost...  Those are already there, just run cgps and
> see.
>
> BTW, TDOP is pretty useless...
>
> > > Do it the "clockmaker" way, or do it the normal way.  Mixing them up
> > > is confusing you.
> >
> > What was confusing me was a misunderstanding on the path that was
> > required for $PYTHONPATH
>
> So, all clear now?  Your above statements do not make me think so.
>
> > > > I have to disagree, when "scons install" is run as root it puts
> > > > files in several paths that are not in the user's home directory
> > > > /usr/local/, /usr/local/lib/
> > > > /usr/local/lib/python2.7/dist-packages/gps, and /usr/local/include
> > > > I have seen this bahaviour under rasbian on a pi and in the
> > > > Ubuntu VM
> > >
> > > Which is why the clockmaker script does not tell you to do that!
> >
> > from the howto
> > This step can be automated with "./clockmaker --install" done as root.
> > # cd gpsd; scons install
>
> As it says: "can be" but the silent recommendation is not to.  I recommend
> that you do not.
>
> > if the instructions from build.txt are followed, the same paths have
> > files copied to them, build.txt includes "The default value is
> > "/usr/local"",
>
> Yup.  Directly from the FHS.
>
> > could it be that you are using a different distro or
> > have other configuration options provide different results ? such as
> > "scons prefix=/home/pi"
>
> Lost me.  Install path has nothing to do with your distro.
>
> By default, observing the FHS:
>
> 1. packages install in /usr
>
> 2. source code packages install in /usr
>
> 3. people scared to use root, or lacking root, install in their
>    home directoy (/home/pi).
>
> We are talking about #2 and #3, not #1.  Also, you seem to be conflating @3 
> and #3.
>
> > > Don't mix up recipes until you are a good chef.  The clockmaker
> > > purpose and install and not the same as the normal purpose and
> > > install.
> >
> > 1. I would say that taking something that works one way, and learning
> > how to modify it to work a different way, is a key part of becoming a
> > chef, another key part is experimention.
>
> Yes.  But only when you can keep the different methods separate in your
> head.  An advanced skill you have not shown yet.
>
> > 2. I presume you meant "and install are not the same", to which I
> > would say that the original purpose has been subverted.
>
> Uh, lost me.  Best to never think I did not say what I meant.  This
> has led you astray many times here already.
>
> > > Or just use "scons uninstall".  Can't get easier than that.
> >
> > I had not seen "scons uninstall" in build.txt
>
> And truly it is not there.  There are also no elephants there.  Your point?
>
> This is UNIX, until you have read ALL the doc, you can't complain that
> one part of the doc does not say what you want it to say.
>
> OTOH, feel free to submit a patch if you find a way to improve the doc.
>
> > > > > I see an ntpsec configure, but no gpsd configure?
> > > > >
> > > > Sorry, you've lost me, I was under the obviously mistaken
> > > > understanding that scons without install was a combined configure
> > > > and build, with the configure "options" being passed on the
> > > > command line.
> > >
> > > Yes, mistaken.  Not how you think. You sent me your ntpsec config,
> > > not your gpsd config.  We are talking about gpsd here.
> >
> > Ok, so the "timeservice=yes" are build options, not configure options,
>
> Uh, exactly backwards.  timeservice=yes is a configure option, not
> a build option.  You are confused since if no configure is done
> explicitly it is done implicitly.
>
> > there is no equivalent to ./waf configure in either build.txt or
> > clockmaker,
>
> Once again.  waf is for ntpsec.  scons is for gpsd.
>
> Try to keep the differences in mind.  UNtil you keep the separate in your
> your head you can't make any inferences.
>
> > I do not understand what you asked to see considering that
>
> Clearly.
>
> > I had also posted the bash script I had used, which I would have hoped
> > made it obvious exactly what commands I was running to duplicate what
> > I was seeing.
>
> Sorry, too many things for me to do rather than helping you debug
> your bash.  Get it working, then I'll look at it.  Until then, only
> questions.
>
> > Just to reiterate, I was not suggesting that either gpsd or ntpsec had
> > a fault, I was trying to understand why what I had thought would work
> > did not work, and from the reply from Rob, I now have something that
> > works.
>
> Well, then, all done then?
>
> > Now I have found
> > https://chrisyeh96.github.io/2017/08/08/definitive-guide-python-imports.html#basics-of-the-python-import-and-syspath
> > which has in it
> >
> > "Technically, Python’s documentation is incomplete. The interpreter
> > will not only look for a file (i.e. module) named spam.py, it will
> > also look for a folder (i.e. package) named spam"
>
> Not exactly true, bordering onn misleading.  I sugest you read the
> Python doc again to see how that is not exactly correct.
>
> That stuff is indeed in the python doc.  Not easy to grok, but it is
> there.
>
> > So I made three mistakes, a newbie mistake to expect the python.org
> > documentation to be complete,
>
> I have to disagree.  It is in the doc, you just missed it.  A very easy
> thing to do given the size and lack of organization of the doc.
>
> And your understanding is still not correct as it applies to gpsd.
>
> > a typo in just the wrong place in the
> > first post, and not to show how I could reproduce what I was seeing on
> > a reduced system (such as the bash script) in my first post. I have
> > learnt from these.
>
> Good.
>
> > I can now go back to hacking around on clockmaker (-:
>
> If you do get something that seems to work, post it here, maybe it
> will be useful for someone else.  The extensive build doc for gpsd and
> ntpsd are there because there are many different ways to configure and
> build.  Not as many ways as there need to be.
>
> RGDS
> GARY
> ---------------------------------------------------------------------------
> Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
>         address@hidden  Tel:+1 541 382 8588
>
>             Veritas liberabit vos. -- Quid est veritas?
>     "If you can’t measure it, you can’t improve it." - Lord Kelvin