Parsing GPS receiver output via regex in Python

I have a friend who is finishing up his masters degree in aerospace engineering. For his final project, he is on a small team tasked with writing a program for tracking weather balloons, rockets and satellites. The program receives input from a GPS device, does calculations with the data, and uses the results of those calculations to control a series of motors designed to orientate a directional communication antenna, so the balloon, rocket or satellite always stays in focus.

Though somewhat of a (eternal) beginner myself, I have more programming experience than my friend. So when he asked me for advice, I convinced him to write the program in Python, my language of choice.

At this point in the project, we are working on the code that parses the input from the GPS device. Here is some example input, with the data we need to extract in bold:

$GPRMC,092204.999,4250.5589,S,14718.5084,E,1,12,24.4,89.6,M,,,0000*1F $GPRMC,093345.679,4234.7899,N,11344.2567,W,3,02,24.5,1000.23,M,,,0000*1F $GPRMC,044584.936,1276.5539,N,88734.1543,E,2,04,33.5,600.323,M,,,*00 $GPRMC,199304.973,3248.7780,N,11355.7832,W,1,06,02.2,25722.5,M,,,*00 $GPRMC,066487.954,4572.0089,S,45572.3345,W,3,09,15.0,35000.00,M,,,*1F

Here is some further explanation of the data:

"I looks like I'll need five things out of every line. And bear in mind that any one of these area's may be empty. Meaning there will be just two commas right next to each other. Such as ',,,' There are two fields that may be full at any time. Some of them only have two or three options that they may be but I don't think I should be counting on that."

Two days ago my friend was able to acquire the full log from the GPS receiver used to track a recent weather balloon launch. The data is quite long, so I put it all in this pastebin.

I am still rather new with regular expressions myself, so I am looking for some assistance.


Asked by: Lydia720 | Posted: 01-10-2021






Answer 1

splitting should do the trick. Here's a good way to extract the data, as well:

>>> line = "$GPRMC,199304.973,3248.7780,N,11355.7832,W,1,06,02.2,25722.5,M,,,*00"
>>> line = line.split(",")
>>> neededData = (float(line[2]), line[3], float(line[4]), line[5], float(line[9]))
>>> print neededData
(3248.7779999999998, 'N', 11355.7832, 'W', 25722.5)

Answered by: Blake270 | Posted: 02-11-2021



Answer 2

You could use a library like pynmea2 for parsing the NMEA log.

>>> import pynmea2
>>> msg = pynmea2.parse('$GPGGA,142927.829,2831.4705,N,08041.0067,W,1,07,1.0,7.9,M,-31.2,M,0.0,0000*4F')
>>> msg.timestamp, msg.latitude, msg.longitude, msg.altitude
(datetime.time(14, 29, 27), 28.524508333333333, -80.683445, 7.9)

Disclaimer: I am the author of pynmea2

Answered by: Agata792 | Posted: 02-11-2021



Answer 3

It's simpler to use split than a regex.

>>> line="$GPRMC,092204.999,4250.5589,S,14718.5084,E,1,12,24.4,89.6,M,,,0000*1F "
>>> line.split(',')
['$GPRMC', '092204.999', '4250.5589', 'S', '14718.5084', 'E', '1', '12', '24.4', '89.6', 'M', '', '', '0000*1F ']
>>> 

Answered by: First Name487 | Posted: 02-11-2021



Answer 4

Those are comma separated values, so using a csv library is the easiest solution.

I threw that sample data you have into /var/tmp/sampledata, then I did this:

>>> import csv
>>> for line in csv.reader(open('/var/tmp/sampledata')):
...   print line
['$GPRMC', '092204.999', '**4250.5589', 'S', '14718.5084', 'E**', '1', '12', '24.4', '**89.6**', 'M', '', '', '0000\\*1F']
['$GPRMC', '093345.679', '**4234.7899', 'N', '11344.2567', 'W**', '3', '02', '24.5', '**1000.23**', 'M', '', '', '0000\\*1F']
['$GPRMC', '044584.936', '**1276.5539', 'N', '88734.1543', 'E**', '2', '04', '33.5', '**600.323**', 'M', '', '', '\\*00']
['$GPRMC', '199304.973', '**3248.7780', 'N', '11355.7832', 'W**', '1', '06', '02.2', '**25722.5**', 'M', '', '', '\\*00']
['$GPRMC', '066487.954', '**4572.0089', 'S', '45572.3345', 'W**', '3', '09', '15.0', '**35000.00**', 'M', '', '', '\\*1F']

You can then process the data however you wish. It looks a little odd with the '**' at the start and end of some of the values, you might want to strip that stuff off, you can do:

>> eastwest = 'E**'
>> eastwest = eastwest.strip('*')
>> print eastwest
E

You will have to cast some values as floats. So for example, the 3rd value on the first line of sample data is:

>> data = '**4250.5589'
>> print float(data.strip('*'))
4250.5589

Answered by: Adelaide645 | Posted: 02-11-2021



Answer 5

You should also first check the checksum of the data. It is calculated by XORing the characters between the $ and the * (not including them) and comparing it to the hex value at the end.

Your pastebin looks like it has some corrupt lines in it. Here is a simple check, it assumes that the line starts with $ and has no CR/LF at the end. To build a more robust parser you need to search for the '$' and work through the string until hitting the '*'.

def check_nmea0183(s):
    """
    Check a string to see if it is a valid NMEA 0183 sentence
    """
    if s[0] != '$':
        return False
    if s[-3] != '*':
        return False

    checksum = 0
    for c in s[1:-3]:
        checksum ^= ord(c)

    if int(s[-2:],16) != checksum:
        return False

    return True

Answered by: Jared938 | Posted: 02-11-2021



Answer 6

If you need to do some more extensive analysis of your GPS data streams, here is a pyparsing solution that breaks up your data into named data fields. I extracted your pastebin'ned data to a file gpsstream.txt, and parsed it with the following:

"""
 Parse NMEA 0183 codes for GPS data
 http://en.wikipedia.org/wiki/NMEA_0183

 (data formats from http://www.gpsinformation.org/dale/nmea.htm)
"""
from pyparsing import *

lead = "$"
code = Word(alphas.upper(),exact=5)
end = "*"
COMMA = Suppress(',')
cksum = Word(hexnums,exact=2).setParseAction(lambda t:int(t[0],16))

# define basic data value forms, and attach conversion actions
word = Word(alphanums)
N,S,E,W = map(Keyword,"NSEW")
integer = Regex(r"-?\d+").setParseAction(lambda t:int(t[0]))
real = Regex(r"-?\d+\.\d*").setParseAction(lambda t:float(t[0]))
timestamp = Regex(r"\d{2}\d{2}\d{2}\.\d+")
timestamp.setParseAction(lambda t: t[0][:2]+':'+t[0][2:4]+':'+t[0][4:])
def lonlatConversion(t):
    t["deg"] = int(t.deg)
    t["min"] = float(t.min)
    t["value"] = ((t.deg + t.min/60.0) 
                    * {'N':1,'S':-1,'':1}[t.ns] 
                    * {'E':1,'W':-1,'':1}[t.ew])
lat = Regex(r"(?P<deg>\d{2})(?P<min>\d{2}\.\d+),(?P<ns>[NS])").setParseAction(lonlatConversion)
lon = Regex(r"(?P<deg>\d{3})(?P<min>\d{2}\.\d+),(?P<ew>[EW])").setParseAction(lonlatConversion)

# define expression for a complete data record
value = timestamp | Group(lon) | Group(lat) | real | integer | N | S | E | W | word
item = lead + code("code") + COMMA + delimitedList(Optional(value,None))("datafields") + end + cksum("cksum")


def parseGGA(tokens):
    keys = "time lat lon qual numsats horiz_dilut alt _ geoid_ht _ last_update_secs stnid".split()
    for k,v in zip(keys, tokens.datafields):
        if k != '_':
            tokens[k] = v
    #~ print tokens.dump()

def parseGSA(tokens):
    keys = "auto_manual _3dfix prn prn prn prn prn prn prn prn prn prn prn prn pdop hdop vdop".split()
    tokens["prn"] = []
    for k,v in zip(keys, tokens.datafields):
        if k != 'prn':
            tokens[k] = v
        else:
            if v is not None:
                tokens[k].append(v)
    #~ print tokens.dump()

def parseRMC(tokens):
    keys = "time active_void lat lon speed track_angle date mag_var _ signal_integrity".split()
    for k,v in zip(keys, tokens.datafields):
        if k != '_':
            if k == 'date' and v is not None:
                v = "%06d" % v
                tokens[k] = '20%s/%s/%s' % (v[4:],v[2:4],v[:2])
            else:
                tokens[k] = v
    #~ print tokens.dump()


# process sample data
data = open("gpsstream.txt").read().expandtabs()

count = 0
for i,s,e in item.scanString(data):
    # use checksum to validate input 
    linebody = data[s+1:e-3]
    checksum = reduce(lambda a,b:a^b, map(ord, linebody))
    if i.cksum != checksum:
        continue
    count += 1

    # parse out specific data fields, depending on code field
    fn = {'GPGGA' : parseGGA, 
          'GPGSA' : parseGSA,
          'GPRMC' : parseRMC,}[i.code]
    fn(i)

    # print out time/position/speed values
    if i.code == 'GPRMC':
        print "%s %8.3f %8.3f %4d" % (i.time, i.lat.value, i.lon.value, i.speed or 0) 


print count

The $GPRMC records in your pastebin don't seem to quite match with the ones you included in your post, but you should be able to adjust this example as necessary.

Answered by: Ted508 | Posted: 02-11-2021



Answer 7

I suggest a small fix in your code because if used to parse data from the previous century the date looks like sometime in the future (for instance 2094 instead of 1994)

My fix is not fully accurate, but I take the stand that prior to the 70's no GPS data existed.

In the def parse function for RMC sentences just replace the format line by:

p = int(v[4:])
print "p = ", p
if p > 70:
    tokens[k] = '19%s/%s/%s' % (v[4:],v[2:4],v[:2])
else:
    tokens[k] = '20%s/%s/%s' % (v[4:],v[2:4],v[:2])

This will look at the two yy digits of the year and assume that past year 70 we are dealing with sentences from the previous century. It could be better done by comparing to today's date and assuming that every time you deal with some data in the future, they are in fact from the past century

Thanks for all the pieces of code your provided above... I had some fun with this.

Answered by: Carina593 | Posted: 02-11-2021



Answer 8

This is a GPRMC string. After splitting the string, you need to parse latitude and longitude values.

line = "$GPRMC,199304.973,3248.7780,N,11355.7832,W,1,06,02.2,25722.5,M,,,*00"
line = line.split(",")

In latitude and longitude part ([..., '3248.7780', 'N', '11355.7832, 'W', ...]):

  • The first number is not a pure number, it is a number which is concatenated like a string. I mean, 3248.7780 refers 32 degree, 48.7780 minutes (latitude)
  • The second number (11355.7832) refers 113 degree, 55.7832 minutes (longitude)

They cannot be used in a formula as it is. They have to be converted to decimal degree.

def toDD(s):
    d = float(s[:-7])
    m = float(s[-7:]) / 60
    return d + m

lat_lon = (toDD(line[2]), line[3], toDD(line[4]), line[5])
print(lat_lon)

# (32.81296666666667, 'N', 113.92972, 'W')

Answered by: Alfred293 | Posted: 02-11-2021



Similar questions

python - Receiver doesn't notice signal launch

I know this has been asked and answered before, but none of the proposed solutions has worked for me. This is my signals.py: from django.dispatch import Signal my_signal = Signal(providing_args=['arg1', 'arg2', 'arg3']) In the other hand, I have a file called handlers.py which looks like this: from django.dispatch import receiver from models impor...


sockets - Does python UDP receiver always receives only one message?

About UDP receivers. Consider the sample code below, do I have to consider that I might receive multiple UDP messaged in the recfrom method ? import socket s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.bind(("", 99)) while 1: data, addr = s.recvfrom(1024) someFunction(data)


python - Django receiver check if first create

The idea of the code below should be that it only fires if the field verification_pin is empty i.e. on a new record. However, it seems that every time I save the model it generates a new pin ignoring if instance.verification_pin is None statement, why, what have I missed? @receiver(pre_save, sender=CompanyUser) def my_callback(sender, instance, *args, **kwargs): if instance.ve...


python - Can I use a Twisted GTK Reactor with a UDP Receiver?

I want to add a UDP receiver: from twisted.internet.protocol import DatagramProtocol from twisted.internet import reactor class UDP(DatagramProtocol): def datagramReceived(self, data, (host, port)): #Receive a string "X Y" and add to plot. print "Received %r from %s:%d" % (data, host, port) reactor.listenUDP(9999, UDP()) reactor.run() To my existing matplotlib code that ...


Receiver loses the same last bytes of file lost sent via TCP socket in python

I send a .txt file(about 87 kbyte size) from a client to a server over TCP with the following code(python): Client: f = open(filename, 'r') while 1: data = f.read(1024) if not data: data='*Endoffile*!' con.send('%1024...


Is there a way to use Python to access Citrix Receiver, containing SQL database?

How to query sql server database programatically, which is currently accessible using citrix receiver Similar to this question...but I'm trying to use Python to write SQL queries, but I'm having trouble because Citrix Receiver is in the way. Do I have to ask for direct d...


python - Why we should declare name of queue in receiver?

I am learning RabbitMQ. I can't understand why in example of receive_logs.py we setting type of the exchange: channel.exchange_declare(exchange='logs', exchange_type='fanout') and binding name of queue with this exchange: channel.queue...


Python SNMP Trap Receiver

I received an SNMP trap message in Python3, and I got a hexadecimal number. How do I convert it to a String so I can see it? Received Data(Hex) b'0E\x02\x01\x01\x04\x06404040\xa78\x02\x04\x00\xf6\x17~\x02\x01\x00\x02\x01\x000*0\x0f\x06\x08+\x06\x01\x02\x01\x01\x03\x00C\x03\x01k+0\x17\x06\n+\x06\x01\x06\x03\x01\x01\x04\x01\x00\x06\t+\x06\x01\x06\x03\x01\x01\x05\x01' This is my SNMP trap recei...


get email receiver language setting python

I want to know if it's possible to get the email language setting of a user using python. To be more specific, I'm trying to set up an email notification system that notifies users by email when they receive a message, I don't want to really use their browser language setting because the sender and receiver could have different language setting on their browser. I want to know if it's possible, knowing the receiver's email...


python - A receiver that can receive both TCP images and UDP texts

I'm new to socket programming . I've implemented 2 separated codes on the same host. One of them is supposed to receive images using TCP protocol and the second one is supposed to receive text messages through UDP protocol. Both of them are working fine separately. Here are the codes: Image receiver (TCP): from __future__ import print_function import socket from struct import unpack...






Still can't find your answer? Check out these communities...



PySlackers | Full Stack Python | NHS Python | Pythonist Cafe | Hacker Earth | Discord Python



top