How do refine a huge KML file & strip out certain data?

I have been recording gps tracks of where i've been for 10 years now. They have been recorded using various different mobile apps and currently I organise them in google earth in folders for each month.

The saved KML file is about 170MB and contains lots of information that I do not want, such as email addresses, url's, addresses and lots of other stuff that is put there by what ever app I was using at the time.

Is there a way to refine this information, or a way to strip out the unnecessary data so that I just have the coordinates? 

Nick

Hi Nick,



I'm no expert in KML, but I have had some experience in editing KML files created by the Google Earth tools, including Paths. If I had your problem, this is what I would try: Using a text editor (I like to use Edit Pad Lite, a free download, but you must always remember to set Convert > Text Encoding to Unicode-UFT8 before proceeding), copy your file in GE and paste it into the text editor. Since it was likely created by your GPS device and not by GE, it will contain a lot of stuff that a GE-created path will not have, and that's the stuff you want to get rid of, so I would also create a simple GE Path, so you can tell what is necessary. Here's the code for a simple path I created with the GE Path tool:



<?xml version="1.0" encoding="UTF-8"?>

<kml xmlns="http://www.opengis.net/kml/2.2" xmlns:gx="http://www.google.com/kml/ext/2.2" xmlns:kml="http://www.opengis.net/kml/2.2" xmlns:atom="http://www.w3.org/2005/Atom">

<Document>

<name>KmlFile</name>

<StyleMap id="m_ylw-pushpin">

<Pair>

<key>normal</key>

<styleUrl>#s_ylw-pushpin</styleUrl>

</Pair>

<Pair>

<key>highlight</key>

<styleUrl>#s_ylw-pushpin_hl</styleUrl>

</Pair>

</StyleMap>

<Style id="s_ylw-pushpin_hl">

<IconStyle>

<scale>1.3</scale>

<Icon>

<href>http://maps.google.com/mapfiles/kml/pushpin/ylw-pushpin.png</href>

</Icon>

<hotSpot x="20" y="2" xunits="pixels" yunits="pixels"/>

</IconStyle>

</Style>

<Style id="s_ylw-pushpin">

<IconStyle>

<scale>1.1</scale>

<Icon>

<href>http://maps.google.com/mapfiles/kml/pushpin/ylw-pushpin.png</href>

</Icon>

<hotSpot x="20" y="2" xunits="pixels" yunits="pixels"/>

</IconStyle>

</Style>

<Placemark>

<name>Sample GE Path</name>

<styleUrl>#m_ylw-pushpin</styleUrl>

<LineString>

<tessellate>1</tessellate>

<coordinates>

135.5858733533433,34.64288634547518,0 135.5855527154511,34.64287915203258,0 135.585584063897,34.6421135367449,0 135.5857434960758,34.64206450471972,0 135.5857644321109,34.64139022734445,0 135.5823458252233,34.63972670157465,0 135.5815906527926,34.63967577429509,0 135.5816352013729,34.63934688689761,0 135.5811687385088,34.63935710793824,0 135.5812209158523,34.6387809363732,0 135.5813388223314,34.63712729183641,0 135.5815072348136,34.63425548579849,0 135.5815583177224,34.63301381050191,0 135.5817047027257,34.63298263052323,0 135.5816838094989,34.63245874486267,0 135.5813371508274,34.63203957659776,0

</coordinates>

</LineString>

</Placemark>

</Document>

</kml>


When you compare to two files, you should be able to pretty much understand what you do and don't need for the simpler file you want to convert to. It may be as simple as copying the coordinates data from your original file, and pasting into the simple file in place of the coordinates that are there.



I can't guarantee that this will work for you, but as long as you keep your original file intact, your efforts shouldn't create any harm. At least my efforts haven't for me. If you would like someone to discuss your efforts with, as you venture into the jungle, I'm available to look at what you're doing, and maybe make suggestions.

Attachments:

Sample GE Path.kmz (830 B)

Hi Washi

Thanks for your reply, this is very helpful. I'd also arrived at the conclusion that there was not going to be a magic tool that would do all the hard work for me, and that the only way was to do it manually. I've been reading about kml etc and think i'll be able to strip out personal data using a text editor with regex support. 

In learning more I found out that my next challenge will be the fact that a lot of the tracks are recorded using <gx:track> and not <LineString>. The trouble I have with <gx:track> is that it also stores time data. So when I look at all my tracks in google earth, I have to make sure that the time & date sliders are set right otherwise it won't show all my tracks. 

Anyway, i'll keep digging away at it and hopefully I'll crack it soon, or if anyone knows an easy way to convert a <gx:tour> to a line/path?

Nick 

What does the data look like in your more complex files? In a KML Path file, the data is Longitude, Latitude, 0, with the zero (I presume) being nothing more than an end-of-data marker. If you can use search and replace tools (I often copy kml files into my word processing program because it's easier for me to use the search and replace tools there) to isolate longitude and latitude data and get a zero marker, then I'm pretty confident that you could paste it into a simple path file created by GE.

I succeeded by opening up the kml in a text editor and using regex to remove the data I didn't want. 

This brought the file size down quite a bit; from 169MB to 116MB. I was also able to remove all the time data so now when I view it on GE, I don't need to worry about the time-range being correct to view the tracks.

The hardest part was finding the right regex string but I got there in the end. If anyone's interested, here a few of the strings I used in SublimeText:

To remove the time data:
<when>.*</when>
<TimeSpan>(.|\n)*?<\/TimeSpan>

To remove the personal data:
<description>(.|\n)*?<\/description>
<ExtendedData>(.|\n)*?<\/ExtendedData>
<Address>(.|\n)*?<\/Address>


To remove the newly created blank lines:
^[\s]*?[\n\r]+

I have attached a gif showing a before / after, you should be able to see on the old kml the time slider going all over the place and no time slider on the cleaned kml.




I've attached a couple of pics of the new tracks.




Washi, thanks for your help. We got there in the end!

Nick

Glad to see you got the job done!  Your task was much more massive than I imagined.  Incidentally, if you ever wish to insert code into a post on this forum, this interface has a convenient tool (C in a box icon) that lets you preserve the formatting of the code you're sharing.


insert code here
You just paste the code into the box over the words "insert code here".