Banner

Friday, 20 March 2015

Transferring a route from QGIS to your GPS

In my last blog post, we created a created a 500 km continuous line representing a ski route across Nordryggen in Norway. I need to transfer this route to my Garmin GPS so I can use it for navigation while skiing. How can it be done?

Cross-country skiing in Skarvheimen, Norway. Photo: Bjørn Sandvik

Open the route in QGIS, right click the layer and select "Save As...". Select "GPS eXchange Format [GPX]" as the format, and "WGS 84" as the coordinate reference system (CRS). I'm also skipping attribute creation as my line only contains coordinates.



QGIS saves the line as a GPX route. You can import this route in Garmin BaseCamp (File -> Import). Give the track a meaningful name.

My 500 km route consists of 3867 points, but most Garmin GPS units are only capable of showing 250 points per route. You can get around this limitation by converting the route into a track. Right-click the route in Basecamp and select "Create Track from Route".


Transfer the track to your GPS unit:


You should now find the track in the "Track Manager" on your GPS. Select "View Map" to see it and then "Go" if you want to navigate along it. Depending on the maps you have on your device, you can also display an elevation plot. Since the original track don't contain elevation data, the GPS will try to fetch it from your map.

My GPS is now loaded and I'm ready to go!

All data are available on GitHub.

Nordryggen on skis for 25 days - creating a route map

I’m currently doing my last preparations for a 25 days skiing trip across Nordryggen in Norway. It will of course depend on weather, snow conditions and blisters, but hopefully the conditions will be bearable. Norway has a great network of 500 cabins maintained by the Norwegian Trekking Association. The longest connected cross country skiing track I’ve found is around 500 km, - how to map it?

The map we're going to create with QGIS and CartoDB. Data from the Norwegian Mapping Authority.

Nordryggen (“the north ridge”) is a fairly new name of the 1,700 km mountain range that runs through the Scandinavian Peninsula. My plan is to ski around 500 km in the southern part of Norway, most of it above the tree line.

Cross-country skiing in Jotunheimen. Photo: Bjørn Sandvik

You can study the waymarked ski routes on UT.no, or download the data from the Norwegian Mapping Authority if you want to map it yourself.  The dataset is available as a PostGIS dump or in SOSI. SOSI is a common vector data format used to exchange geographical information in Norway, and not very well supported by various mapping applications. Luckily we have Sosicon, a great open source converter by Espen Andersen. I ran this command on the SOSI file:

./sosicon -2shp Tur\ og\ friluftsruter.sos 

In return I got 4 shapefiles for ski routes, hiking trails, bike trails and POIs.

If you open the ski routes shapefile in QGIS it won’t tell you much without a basemap. Let's use the map tile service from the Norwegian Mapping Authority. In QGIS, click on the “Add WMS/WMTS Layer” button in the left toolbar. Click the “New” button to create a connection. Add a name and copy this URL:

http://opencache.statkart.no/gatekeeper/gk/gk.open_wmts?Version=1.0.0&service=wmts&request=getcapabilities



Click “OK” and “Connect”. You will now get a long list of layers in different map projections. I recommend using “norges_grunnkart” or “topo2” (detailed) in ESPG:32633 (UTM 33).



Click on “Add” and you should see a topographic map of Norway. Now you can add the shapefile, and the ski tracks will show on top.

Waymarked ski routes shown in QGIS with my route selected between Sota Sæter and Ljosland.

I selected the track segments I plan to follow, and saved the selection in a new route shapefile. I want to have the route as a continuous line with the coordinates in order, as this will allow me to use it for navigation with my GPS unit (see my next blog post). I'm using the "Join multiple lines" plugin to get the desired result.

Installing plugins is very easy with the QGIS Plugin Manager. After installation, you'll find the "Join multiple lines" plugin in the Vector menu. 

You can use the Field calculator to calculate the length of the continuous track:


This outputs a length of 505 km.

Let's move on to CartoDB to create the route map. First, I'm uploading the zipped route shapefile to CartoDB. I've also created a table with the cabin name and positions along the route.

Create a new visualzation and add the route and cabin tables. You can add the same basemap as we used in QGIS with this URL:

http://opencache.statkart.no/gatekeeper/gk/gk.open_gmaps?layers=norges_grunnkart&zoom={z}&x={x}&y={y}



The cabins are styled differently depending on the zoom level. This is the final map:


All data is available on GitHub. Map data from the Norwegian Mapping Authority.

Sunday, 12 October 2014

Creating 3D terrains with Cesium

Previously, I’ve used three.js to create 3D terrain maps in the browser (1, 2, 3, 4, 5, 6). It worked great for smaller areas, but three.js doesn’t have built-in support for tiling and advanced LOD algorithms needed to render large terrains. So I decided to take Cesium for a spin.


Cesium is a JavaScript library for creating 3D globes and 2D maps in the browser without a plugin. Like three.js, it uses WebGL for hardware-accelerated graphics. Cesium allows you to add your own terrain data, and this blog post will show you how.


Compared to the dying Google Earth plugin, it's quite complicated to get started with Cesium. The source code is well documented and the live coding Sandcastle is great, but there is a lack of tutorials and my development slows down when I have to deal with a lot of math.

That said, I was able to create an app streaming my own terrain and imagery with a few lines of code. There is also WebGL Earth, a wrapper around Cesium giving you an API similar to well-known Leaflet. I expect to see more functions or wrappers to make stuff like camera positioning easier in the future.

How can you add your own terrain data to Cesium? 

First, you need to check if you really need it. You have the option to stream high-resolution terrain data directly from the servers at AGI. It's free to use on public sites under the terms of use. If you want to host the terrain data on your own servers, AGI provides a commercial product - the STK Terrain Server. Give it a try, if you have a budget!

I was looking for an open source solution, and found out that Cesium supports two terrain formats:
  1. heightmap
  2. quantized-mesh
The tiled heightmap format is similar to the one I used for three.js. Each tile contains 65 x 65 height values, which overlap their neighbors at their edges to create a seamless terrain. Cesium translates the heightmap tiles into a uniform triangle mesh, as I did in three.js. The downside of this format is the uniform grid, you use the same amount of data to represent both flat and hilly terrain.

The regular terrain mesh made from heightmap tiles. 

The quantized-mesh format follows the same tile structure as heightmap tiles, but each tile is better optimised for large-scale terrain rendering. Instead of creating a dense, uniform triangle mesh in the browser, an irregular triangle mesh is pre-rendered for each tile. It's a better representation of the landscape, having less detail in flat areas while increasing the density in steep terrain. The mesh terrain is also more memory efficient and renders faster.

The irregular terrain mesh from quantized-mesh tiles. Larger triangles have less height variation. 

Unfortunately, I haven't found any open source tools to create tiles in the quantized-mesh format - please notify me if you know how to do it!

You can generate heightmap tiles with Cesium Terrain Builder, a great command-line utility by Homme Zwaagstra at the GeoData Institute, University of Southampton.

I'm using the same elevation data as I did for my three.js maps, but this time in full 10 meter resolution. I'm just clipping the data to my focus area (Jotunheimen) using EPSG:4326, the World Geodetic System (WGS 84).

gdalwarp -t_srs EPSG:4326 -te 7.2 60.9 9.0 61.7 -co compress=lzw -r bilinear jotunheimen.vrt jotunheimen.tif

I went for the easy option, and installed Cesium Terrain Builder using the Docker image. First I installed Docker via Homebrew.  I was not able to mount my hard drive with this method, so I downloaded the elevation data from my public Dropbox folder:

wget https://dl.dropboxusercontent.com/u/1234567/jotunheimen.tif

I used the ctb-tile command to generate the tileset:

ctb-tile --output-dir ./tiles jotunheimen.tif

The command returned 65 000 tiles down to zoom level 15. I compressed the tiles into one file:

tar cvzf tiles.tar.gz tiles

and used the Dropbox uploader to get the tiles back to my hard drive:

./dropbox_uploader.sh upload tiles.tar.gz tiles.tar.gz

So I got 65 000 terrain tiles on my server, how can I see the beauty in Cesium? It required some extra work:
  1. First I had to add a missing top level tile that Cesium was expecting. 
  2. Cesium was also looking for a layer.json file which I had to create:

    {
      "tilejson": "2.1.0",
      "format": "heightmap-1.0",
      "version": "1.0.0",
      "scheme": "tms",
      "tiles": ["{z}/{x}/{y}.terrain?v={version}"]
    }

  3. Lastly, I added a .htaccess file to support CORS and gzipped terrain tiles: 

Then I was ready to go!

Beautiful terrain rendered with 10 m elevation data from the Norwegian Mapping Authority. Those who know Jotunheimen, will notice Skogadalsbøen by the river and Stølsnostind and Falketind surrounded by glaciers in the background.

The terrain is a bit blocky (see the mount Falketind to the left), but I'm not sure if this is happening in Cesium Terrain Builder or Cesium itself. The quantized-mesh tiles from AGI gives a better result. 

I'm not able so show an interactive version, as I'm using detailed aerial imagery from "Norge i bilder", which are not publicly available.

Friday, 29 August 2014

Geotagging and Picasa Web Albums API, or was it Google+ Photos?

In my last blog post, I presented a new plugin, Leaflet.Photo, that allows you to display geotagged photos from any source. Among them was Google+ Photos and Picasa Web Albums API. My plan is to use this API for my travel map, and this is why.

Does Picasa Web Albums still exist? 
It's a bit messy these days. Google is trying to transition from Picasa Web Albums to Google+ Photos, as photos are the number one things that people want to share on social networks. When you use Picasa to share your albums (Sync to Web), the album URL is now on your Google+ profile, and not on Picasa Web Albums (which is just redirecting me to Google+). This is the URL to the public album from my trip to the Trollfjord:

https://plus.google.com/photos/+BjørnSandvik/albums/6052628080819524545

It also works with your Google+ user id:

https://plus.google.com/photos/118196887774002693676/albums/6052628080819524545

My public Google+ web album. The album contains both photos and videos. 

The thing is, there is no Google+ API for photos and videos yet (apparently they were working on it back in 2011). But the Google Web Albums API still works on your Google+ albums.

Google Web Albums API is not the easiest API I've worked with, but it's flexible and quite fast. This is an XML feed of my public album from Trollfjord:

https://picasaweb.google.com/data/feed/api/user/118196887774002693676/albumid/6052628080819524545

The user number and album id is the same as above. Or better for your JavaScript apps, a JSON feed:

https://picasaweb.google.com/data/feed/api/user/118196887774002693676/albumid/6052628080819524545?alt=json

And if you're still using JSONP:

https://picasaweb.google.com/data/feed/api/user/118196887774002693676/albumid/6052628080819524545?alt=json-in-script

If you click on any of these links, you'll see that it's not a very compact format. There is a lot of data that you don't need. Although complicated, you can select the fields you want to include in the feed. This is how I selected the following elements:
  • Photo URL: entry/media:group/media:content
  • Photo caption: entry/media:group/media:description
  • Photo thumbnail URL: entry/media:group/media:thumbnail
  • Photo timestamp: entry/gphoto:timestamp
  • Photo location: entry/georss:where

This is the new URL:

https://picasaweb.google.com/data/feed/api/user/118196887774002693676/albumid/6052628080819524545?alt=json&fields=entry/media:group/media:content,entry/media:group/media:description,entry/media:group/media:thumbnail,entry/gphoto:timestamp,entry/georss:where

While researching, I also learnt that I could use the imgmax attribute to specify the size of the photos referenced in the photo URL. Neat!

So why should I use this (relatively) old API?
Compared to other popular social media sites, Google don't strip off the meta information of your photos. Instead it uses the build in support for image metadata extensively. Hopefully Google will continue to do this, although social media sites have reasons not doing so.

This means that Google don't lock you in. I can change the location of my photos using my GPS tracks, and it's reflected where I embed my photos. I can edit the image captions in Picasa and it's stored within the image file, allowing me to write the caption once and use it everywhere.

So what is my album workflow for my travel map. Before starting my journey, I'm creating a new Google+ album. The feed from this album is attached to my map, by simply passing on the album id. While on journey, I use the Google Photos app to add photos to the album, that will automagically show up on the map as well. Back from trip, I can add and edit photos from my digital camera in Picasa and sync them to the web album.

Update 12 September 2015: I'm having trouble uploading images with the Google photos app. The images are geotagged and the location data shows in the Google+ album, but unfortunately the location data is not included in the API feed. Please notify me if you're able to get this to work. 

Photos from Google+ shown on my travel map. 

PS! This blog post is not sponsored by Google :-) 

Showing geotagged photos on a Leaflet map

Using Instagram for my real-time travel map had too many limitations, so I decided to use Google+ photos or Picasa Web Albums instead. I've create a new plugin, Leaflet.Photo, that allows you to add geotagged photos to your map, from any source.

The plugin plays well with the great Leaflet.markercluster plugin, showing your photos in clusters. To make the plugin more versatile, it doesn't deal with AJAX loading or image presentation except the thumbnails on the map. Use your AJAX loader of choice, and simply pass on an array of photo objects to the plugin. 

The photo objects can include the properties you like, but the following are required:
  • lat: latitude of photo
  • lng: longitude og photo
  • thumbnail: url to thumbnail photo
I've kept the squared thumbnails of Instagram, as I think it look nicer than variable size thumbnails. Since the photos can have any dimensions, I'm using a CSS technique to crop and center the thumbnails. 

I've created three examples using Picasa Web Albums Data API, CartoDB (synced with Instagram) and Norvegiana API. With CartoDB you can easily get the required photo properties by manipulating the SQL query. Other APIs will require some data parsing before you pass on the photo objects to the plugin. All examples show the photos in a popup when you click/tap on them, but do whatever you like! On my travel map (click on "Bilder"), I'm using my own lightbox showing all photos in a cluster.

Google+ photos


Photos and videos from Google+. See the small animated GIF video thumbnails. 


Instagram / CartoDB


Instagram photos synced with CartoDB. 


Norvegiana 


Historic photos of Oslo from Norvegiana API. 



Enjoy! :-)

Thursday, 28 August 2014

Geotagging photos using GPS tracks, ExifTool and Picasa

I take a lot of photos while trekking, and most of the time I'm also carrying a GPS with me. As my camera don't have a built-in GPS, my photos are not geotagged while shooting. Luckily, this is an easy task if you've kept your GPS logs from the trip. 

I'm still very happy with my Garmin GPSmap 60CSx that I bought 7 years ago. By changing the setup, the GPS allows me to automatically save the tracks to the memory card. I get one GPX file for each day trekking named with the date. I can easily transfer these tracks to my computer or smartphone with a cable or a card reader. 

Before I converted to Mac, I used GeoSetter to geotag my photos on Windows. Now, I want to do it on the command line using the great ExifTool by Phil Harvey. I installed it on my MacBook using Homebrew:

brew install exiftool

After copying my GPX file to the image folder, I'm simply running:

exiftool -geotag=my.gpx ./

If you forgot to sync the camera and GPS time before your trip, you can use the geosync-option to fix it: (60:00 = 60 minutes):

exiftool -geotag=20140329.gpx -geosync=-60:00 ./

You have a lot of options, so make sure to read the "Geotagging with ExifTool" documentation. ExifTool is modifying the Exif headers of your image files, storing the location data in the same file. 

To see the result on a map, I'm using Picasa.  


Click the map pin button (bottom right) to see the map. If the positions are not shown on the map, try to right-click the image folder and select "Refresh Thumbnails". 

If you don't have a GPS track you can always use Picasa to manually geotag your photos. 

Be aware! I just learnt that social media sites like Facebook, Twitter and Instagram removes the Exif data from your images. Google+ don't. 

Now, how can you display the photos on your own map? It will be the topic of my next blog post. 

Wednesday, 27 August 2014

Live tracking in Lofoten and Vesterålen

Last weekend, I had a great trip to scenic Lofoten and Vesterålen in Northern Norway. I brought my tracking gear to test my new real time travel map. How did it go?

Our first trip was to Trollfjord, a 2 km long fjord with a narrow entrance and steep-sided mountains. It's a famous tourist spot in the Lofoten archipelago, but not many leave the boat at the fjord's end to hike up to the Trollfjord hut.

The small Trollfjord hut.

Trollfjord goes in an east-west direction, and I expected to be in the "satellite shadow" being far north and having steep mountains blocking the sky towards the south. My good old Garmin GPSmap 60CSx did well in the rugged landscape, while my satellite SPOT messenger had some difficulties finding and sending positions. 

Live track from my SPOT messenger (interactive map).

GPS track from my Garmin GPS.

The great thing of using CartoDB to sync my SPOT-data, is that you can edit your positions with ease after the trip. 

Tip! The default basemaps in CartoDB are not very detailed for Norwegian mountains, but you can easily add a basemap (Topo2) from the Norwegian Mapping Authority ("Kartverket") with this URL: 

http://opencache.statkart.no/gatekeeper/gk/gk.open_gmaps?layers=topo2&zoom={z}&x={x}&y={y}

Changing the basemap of CartoDB.  


It's then easy to edit or delete the wrong positions:


Be aware! I was struggling editing my CartoDB-tables from my smartphone, but it was not possible to edit the content of table cells. Hopefully, the CartoDB team will make their editor more mobile friendly in the future. 


Another issue was to get the correct time and position of Instagram photos on the map. Trollfjord is an area with poor mobile coverage. When I took photos with the Instagram app it was struggling placing the photos on the map. It worked better if I took the photos the the built-in camera app of my phone (with geotagging activated) and then posting the photo with the Instagram app. 


If I didn't have mobile coverage, I would just retry posting the photo when back to civilisation. The time associated with the image is when it was sent and not taken. I'm going to check if I can extract the shooting time from the Exif headers of the image.

Our second trip went through Møysalen national park, one of very few national parks in Norway that goes all the way down to sea level. 

Møysalen national park

Here we went in a south-north direction, and my SPOT messenger did better as there was less mountains blocking the satellites. 

Map and elevation profile of a 2-day hike through Møysalen national park (interactive map).  

The web service from the Norwegian Mapping Authority ("Kartverket") seemed to have some technical troubles this weekend, so the altitude values and place names was not updated instantly. When the web service was failing my script stopped and the weather report from yr.no was not fetched either. I'm going to improve the error handling before my next trip. 

I also took a lot of photos with my compact camera while trekking, and I would like to show these on the map as well. My camera don't have a GPS receiver, but I should be able to geotag my photos by using my GPS track. It will be the topic of my next blog post. In the meanwhile, here are some of the photos: 

Hurtigruta in Trollfjord

Trollfjord by night

Cloudberries

Trollfjordtindan.

Seagull

Seagulls

Sea eagle in Raftsundet.

Durmålstindan

Tverrelvtindan

Cold and fresh bath at Snytindhytta.

Monday, 18 August 2014

Making a real time travel map

I had to quit my trip form Oslo to Bergen already on day three - and I have to wait until August 2015 for a second try. I still got time to gain some experience in real time tracking - and mapping. Based on this experience I've made a new version of my live travel map: turban.no



This is a private project to learn new skills - where I care more about new standards and less about old browsers. I'm using CSS3 and HTML5 extensively, the the map will probably not show in Internet Explorer < 10, but it should work well on your tablet or smartphone.


My previous map was about 1 MB to load on my mobile, as I really took off mixing Leaflet, Highcharts, Ext JS, jQuery and Fancybox. I'm now left with only Leaflet and D3.js and only 72 kB gzipped JavaScript. It was a bit more work to create an elevation chart width D3.js, but it's very flexible when you get the grasp of it. I also used D3.js to create a lightbox gallery to show my Intagram photos, as it can easily replace jQuery for selectors and animations.

This is a single page application running in your browser with a CartoDB-backend. The only thing I've changed on my server is the .htaccess file to point all request to the same index.html file. Then I'm using the HTML5 History API to create nice looking URLs to different trips. I've also extended the application to support different users, but I have no plans to create a public web service.

The full application code is not available, but the different bits and pieces are and will. It's still work in progress. The next steps will be to improve the experience on touch screens, add a 3D display and maybe create a mobile app with PhoneGap.

I want to share some experiences I had when creating this map - and I would very much like your feedback!

When you visit the site, you can select between different trips. I'm creating new trips by simply adding new rows to a CartoDB-table. The track and images for each trip are fetched based on time attributes.


You can also link directly to a trip, like: http://turban.no/bjorn/oslo-gaastjern

You can mouseover or click the track to se place names and altitudes. To improve the performance, I'm only drawing the line and not the individual points. To find the nearest point to a mouse/tap position, I'm doing a nearest neighbour search.


Actually I'm drawing two track lines; the stippled line, and a thicker invisible line to make it easier to catch mouseover and click events, especially on touch devices. This is the line shown with less transparency:


The image above show the popup, with terrain type and a weather forecast for this specific location at the time I was there. The track interactions are also linked to the elevation chart:


If you mouseover the track, the same position will show on the elevation chart, and vice versa. Both the track and the chart show the live position with a pulsing marker. I'm also marking the overnight stays, as my SPOT device allows me to send custom messages. The elevation chart reads right to left, as this was the direction of my trip. The direction can be changed for each trip.

If you navigate around in the map, you'll see that the elevation chart is changing to reflect the view:


This is done using Crossfilter to quickly select the points within the map view, although my iPad gets a bit sluggish with instant updates while dragging.

Instagram photos are displayed on the map using the great Leaflet.markercluster plugin:



The photos are shown in a lightbox where you can click/tap through the photos in a cluster (no swipe support yet):



All elements are responsive and should adapt to different screen sizes. I've also made a build process with Grunt to concatenate and compress all the CSS and JavaScript into single files. LESS are used to get rid of all the browser prefixes in CSS. I also made a custom build of D3.js to only include the bits I used, reducing the size to one third.

Continuing the work when there are new trips coming up!