On Thu, Aug 15, 2019 at 12:50 AM Gary E. Miller <
address@hidden> wrote:
Yo pisymbol!
On Wed, 14 Aug 2019 23:51:05 -0400
"pisymbol ." <address@hidden> wrote:
> I have a Python program that does the following:
Care to share the code? Hard to debug something we can not see...
Sure, no problem.
> - Spawns a process via multiprocessing that runs in a tight loop
> reading TPV reports and saves the latest reading to a buffer. I'm
> using the stream() API.
>
> - I have two other threads that pick up this buffer periodically (it's
> actually threads that are timestamping two video streams via
> GStreamer at ~30fps).
It took years for gpsd to get the buffering right. Many ways to fail...
Yes, totally understandable.
> - The GPS NMEA strings are emitted at 10Hz
Which you will eventually find is pointless...
I think understand why, but can I hear it from you Gary?
> What I'm finding is that my GPS reader process sometimes lags behind
> the frames. Basically, I will see at times a second go by before my
> process drains all the reports to the current one.
See, harder than it looks. What sort of CPU?
Jetson TX2 (four ARM cores). But it's an embedded platform and recording 4k video from two streams to boot!
> This means I see
> several frames with the same latitude/longitude before the next frame
> gets updated. Obviously if the frames are emitted at 30Hz and the GPS
> strings are at 10Hz I would expect three or four duplicate
> coordinates every second. But it varies wildly.
I'm not sure why you would expect duplicates. I would call that a bug in
your program.
No, it's simple math:
10Hz -> Trimble
30Hz -> Frames
I am emitting frames faster than the Trimble can emit a NMEA, hence I will always have duplicates (3-4 per second ideally).
Of course, that's not what I'm seeing exactly. Sometimes I see a lot of duplicates.
> I suspect this is a scheduling issue. My process just simply isn't
> real-time and the scheduler doesn't allow my process to drain the
> reports fast enough (or recv is too slow?).
Yup, you got issues.
I know, but I talk to my wife a lot. She's a big help.
> However, one "fix" for this would be if I could get the last valid TPV
> report from gpsd without having to manually drain the stream myself.
Those are one and the same...
What I mean is this:
In [20]: g.next()
Out[20]: <dictwrapper: {'mode': 3, 'ept': 0.005, 'class': 'TPV', 'lat': 41.671886859, 'speed': 0.007, 'lon': -93.718452103, 'alt': 254.479, 'track': 124.635, 'eph': 19.0, 'time': '2019-08-15T11:05:22.000Z', 'status': 2, 'climb': -0.05, 'device': 'tcp://192.168.142.1:5017'}>
In [21]: g.next()
Out[21]: <dictwrapper: {'mode': 3, 'ept': 0.005, 'class': 'TPV', 'lat': 41.671886841, 'speed': 0.002, 'lon': -93.718452149, 'alt': 254.485, 'track': 0.0, 'eph': 19.0, 'time': '2019-08-15T11:05:22.100Z', 'status': 2, 'climb': 0.06, 'device': 'tcp://192.168.142.1:5017'}>
In [22]: g.next()
Out[22]: <dictwrapper: {'mode': 3, 'ept': 0.005, 'class': 'TPV', 'lat': 41.671886828, 'speed': 0.003, 'lon': -93.718452157, 'alt': 254.483, 'track': 0.0, 'eph': 19.0, 'time': '2019-08-15T11:05:22.200Z', 'status': 2, 'climb': -0.02, 'device': 'tcp://192.168.142.1:5017'}>
In [23]: g.next()
Out[23]: <dictwrapper: {'mode': 3, 'ept': 0.005, 'class': 'TPV', 'lat': 41.671886845, 'speed': 0.007, 'lon': -93.71845212, 'alt': 254.482, 'track': 221.67, 'eph': 19.0, 'time': '2019-08-15T11:05:22.300Z', 'status': 2, 'climb': -0.01, 'device': 'tcp://192.168.142.1:5017'}>
In [24]: g.next()
Out[24]: <dictwrapper: {'mode': 3, 'ept': 0.005, 'class': 'TPV', 'lat': 41.671886835, 'speed': 0.006, 'lon': -93.718452139, 'alt': 254.485, 'track': 241.085, 'eph': 19.0, 'time': '2019-08-15T11:05:22.400Z', 'status': 2, 'climb': 0.03, 'device': 'tcp://192.168.142.1:5017'}>
In [25]: g.next()
Out[25]: <dictwrapper: {'mode': 3, 'ept': 0.005, 'class': 'TPV', 'lat': 41.67188685, 'speed': 0.01, 'lon': -93.718452118, 'alt': 254.48, 'track': 45.457, 'eph': 19.0, 'time': '2019-08-15T11:05:22.500Z', 'status': 2, 'climb': -0.05, 'device': 'tcp://192.168.142.1:5017'}>
In [26]:
I am waiting seconds before each manually g.next() call but I still see the buffered NMEA string - not the one that just arrived. Please see the 'time' field. It's clear that gpsd buffers these samples. Can I turn that off in stream mode? Or is there another mode that always just fetch the latest NMEA sentence so I don't have to manually drain them?
> Is this possible outside of opening up a raw socket to my device and
> parsing the NMEA strings myself (something I really don't want to do).
I don't see how that could work either.
Interpolation.
My guess is you are doing something obviously wrong. Your latency is
self inflicted, but with no source code no way we can help you.
I sure hope so. But here's hopefully a readable sample:
class Trimble(object):
...
127 def _reader(self, control, nmea, nmea_lock):
128 """ Main reader process to copy in NMEA strings from gpsd """
129
130 trimble = None
131
132 os.nice(-10) # Match gpsd
133 while control[Trimble.CTRL_RUNNING]:
134 try:
135 if control[Trimble.CTRL_PAUSED]:
136 time.sleep(1)
137 continue
138
139 if control[Trimble.CTRL_CONNECTED] and not trimble:
140 trimble = gps.gps()
141 trimble.stream(flags = gps.WATCH_ENABLE)
142
143 if trimble:
144 for report in trimble:
145 if not control[Trimble.CTRL_RUNNING] or not control[Trimble.CTRL_CONNECTED] or control[Trimble.CTRL_PAUSED]:
146 break
147 if report['class'] == 'TPV':
148 with nmea_lock:
149 for key in ['lat', 'lon', 'time']:
150 if key == 'time':
151 ts = report.get(key)
152 if ts:
153 nmea[key] = datetime.strptime(ts, '%Y-%m-%dT%H:%M:%S.%fZ').replace(tzinfo=timezone.utc).timestamp()
154 else:
155 nmea[key] = report.get(key)
156 except Exception as e:
157 print(str(e))
158 finally:
159 if trimble:
160 trimble.close()
161 trimble = None
162
163 def sample(self):
164 """ Read current Trimble reading """
165
166 with self.nmea_lock:
167 return self.nmea.copy()
So I have another thread that get calls per video frame. At the time of the frame, I call sample() above to copy out the latest NMEA string my reader has processed.
The nmea, nmea-lock, and control structures are managed by multiprocessing.Mananger so tha video thread can copy in the nmea by the _reader process. You can ignore L135-L147 since I can confirm the process is happily in the for loop reading strings all the time (L147 on). Note that the Trimble only emits TPV objects as per its configuration right now.
Note that this "works" in that I get plenty of samples per second. The problem is I don't get them fast enough. And sometimes it seems like the _reader() doesn't update the "nmea" dictionary in over a second, i.e. I see update lag on the "nmea" dict shared between the reader process and my Python program under the interpreter.
1) The nmea_lock is contentious (I tried removing it just for S&G's and it did nothing to.help, pretty sure self.nmea.copy() is not thread safe anyway and I rarely rely on CPython implementation nuances).
3) The system is overloaded so the scheduler just can't give this process enough time slice to keep up (hence why I tried the os.nice(-10))
5) I have a bug in my crappy code above. I sure hope so! Would make my life a lot less stressful.