Banner

Thursday, 4 June 2015

Master maps

I’m going freelance over the summer, after 5 great years at the Norwegian Broadcasting Corporation (NRK). It was not an easy decision, but I have to try. I’ll tell more about my plans later. Please sign up to get notified about my services.



Some of the projects I've been working on at NRK:

The flexible mapping stack of NRK.no, allowing journalists and digital storytellers to create advanced maps in minutes. 

"Kartoteket" - our in-house mapping tool built on top of our mapping stack.

Digital storytelling using NRKs mapping stack and Mapbox. 

Digital storytelling using NRKs mapping stack and Mapbox. 

Flood maps using NRKs mapping stack and CartoDB.

Radon affected areas in Norway using NRKs mapping stack.

Our popular photo maps

Video map of the long running TV show Norge Rundt.
Tracking of "Sommerbåten" along the coast of Norway.

Other work.

Thursday, 23 April 2015

Real time satellite tracking of your journeys - how does it work?

I'm back in Oslo after my 25 days ski trip across Nordryggen in Norway. It was a great journey, and I would highly recommend doing all or parts of it if you enjoy cross-country skiing. Just be prepared for shifting weather conditions.
The goal of the trip was also to test my solution for real time satellite tracking, explained in several of my previous blog posts. It worked out really well, and people were able to follow along in the comfort of their sofa.


I fastened a Spot Satellite Messenger to the top of my backpack, and left the device in tracking mode while skiing. The device sent my current position every 5 minutes, allowing me to update the map without any mobile coverage. When we arrived at a mountain hut, I pressed the OK button to set up a bed. I also programmed a button to show a snow cave, in case we wouldn't reach a hut. Luckily we didn't have to use it :-)

My map and elevation plot of the 25 days ski trip across Nordryggen. Most of the trip is above tree line, and there are only 5 road crossings in total. 

The SPOT messenger only sends my time and position, so I had to create a web service to retrieve extra information about each location. I'm using a service from the Norwegian Mapping Authority
 to retrieve the altitude, nearest place name and the terrain type. Earlier this winter, I experienced that the service did't return any altitude if I was skiing on lakes, so I'm using the Google Elevation API to avoid gaps in the elevation profile.

By knowing the time and location, I could create an automatic service to obtain more information to enrich the map. In addition to elevation and place name, I've added a weather report.  The image show Bjordalsbu, the highest lying hut on the route 1586 m, which we visited in a strong breeze. 

While skiing, I used Instagram to post photos that would instantly show on the map as well. This required mobile coverage, which is sparse in the mountains. After the trip, I synced my camera photos with my GPS track to be able to show them along the route.

Click "Bilder" in the top menu to see the photos along the route. 

A few of my photos:

Eidsbugarden in Jotunheimen. 

Iungsdalshytta in Skarveimen. 

Taumevatn in Ryfylkeheiane.

Gaukhei in Setesdalsheiane. 

End of trip - and the snow - in Ljosland. 
More photos in my Google+ album.

Friday, 20 March 2015

Transferring a route from QGIS to your GPS

In my last blog post, we created a created a 500 km continuous line representing a ski route across Nordryggen in Norway. I need to transfer this route to my Garmin GPS so I can use it for navigation while skiing. How can it be done?

Cross-country skiing in Skarvheimen, Norway. Photo: Bjørn Sandvik

Open the route in QGIS, right click the layer and select "Save As...". Select "GPS eXchange Format [GPX]" as the format, and "WGS 84" as the coordinate reference system (CRS). I'm also skipping attribute creation as my line only contains coordinates.



QGIS saves the line as a GPX route. You can import this route in Garmin BaseCamp (File -> Import). Give the track a meaningful name.

My 500 km route consists of 3867 points, but most Garmin GPS units are only capable of showing 250 points per route. You can get around this limitation by converting the route into a track. Right-click the route in Basecamp and select "Create Track from Route".


Transfer the track to your GPS unit:


You should now find the track in the "Track Manager" on your GPS. Select "View Map" to see it and then "Go" if you want to navigate along it. Depending on the maps you have on your device, you can also display an elevation plot. Since the original track don't contain elevation data, the GPS will try to fetch it from your map.

My GPS is now loaded and I'm ready to go!

All data are available on GitHub.

Nordryggen on skis for 25 days - creating a route map

I’m currently doing my last preparations for a 25 days skiing trip across Nordryggen in Norway. It will of course depend on weather, snow conditions and blisters, but hopefully the conditions will be bearable. Norway has a great network of 500 cabins maintained by the Norwegian Trekking Association. The longest connected cross country skiing track I’ve found is around 500 km, - how to map it?

The map we're going to create with QGIS and CartoDB. Data from the Norwegian Mapping Authority.

Nordryggen (“the north ridge”) is a fairly new name of the 1,700 km mountain range that runs through the Scandinavian Peninsula. My plan is to ski around 500 km in the southern part of Norway, most of it above the tree line.

Cross-country skiing in Jotunheimen. Photo: Bjørn Sandvik

You can study the waymarked ski routes on UT.no, or download the data from the Norwegian Mapping Authority if you want to map it yourself.  The dataset is available as a PostGIS dump or in SOSI. SOSI is a common vector data format used to exchange geographical information in Norway, and not very well supported by various mapping applications. Luckily we have Sosicon, a great open source converter by Espen Andersen. I ran this command on the SOSI file:

./sosicon -2shp Tur\ og\ friluftsruter.sos 

In return I got 4 shapefiles for ski routes, hiking trails, bike trails and POIs.

If you open the ski routes shapefile in QGIS it won’t tell you much without a basemap. Let's use the map tile service from the Norwegian Mapping Authority. In QGIS, click on the “Add WMS/WMTS Layer” button in the left toolbar. Click the “New” button to create a connection. Add a name and copy this URL:

http://opencache.statkart.no/gatekeeper/gk/gk.open_wmts?Version=1.0.0&service=wmts&request=getcapabilities



Click “OK” and “Connect”. You will now get a long list of layers in different map projections. I recommend using “norges_grunnkart” or “topo2” (detailed) in ESPG:32633 (UTM 33).



Click on “Add” and you should see a topographic map of Norway. Now you can add the shapefile, and the ski tracks will show on top.

Waymarked ski routes shown in QGIS with my route selected between Sota Sæter and Ljosland.

I selected the track segments I plan to follow, and saved the selection in a new route shapefile. I want to have the route as a continuous line with the coordinates in order, as this will allow me to use it for navigation with my GPS unit (see my next blog post). I'm using the "Join multiple lines" plugin to get the desired result.

Installing plugins is very easy with the QGIS Plugin Manager. After installation, you'll find the "Join multiple lines" plugin in the Vector menu. 

You can use the Field calculator to calculate the length of the continuous track:


This outputs a length of 505 km.

Let's move on to CartoDB to create the route map. First, I'm uploading the zipped route shapefile to CartoDB. I've also created a table with the cabin name and positions along the route.

Create a new visualzation and add the route and cabin tables. You can add the same basemap as we used in QGIS with this URL:

http://opencache.statkart.no/gatekeeper/gk/gk.open_gmaps?layers=norges_grunnkart&zoom={z}&x={x}&y={y}



The cabins are styled differently depending on the zoom level. This is the final map:


All data is available on GitHub. Map data from the Norwegian Mapping Authority.

Sunday, 12 October 2014

Creating 3D terrains with Cesium

Previously, I’ve used three.js to create 3D terrain maps in the browser (1, 2, 3, 4, 5, 6). It worked great for smaller areas, but three.js doesn’t have built-in support for tiling and advanced LOD algorithms needed to render large terrains. So I decided to take Cesium for a spin.


Cesium is a JavaScript library for creating 3D globes and 2D maps in the browser without a plugin. Like three.js, it uses WebGL for hardware-accelerated graphics. Cesium allows you to add your own terrain data, and this blog post will show you how.


Compared to the dying Google Earth plugin, it's quite complicated to get started with Cesium. The source code is well documented and the live coding Sandcastle is great, but there is a lack of tutorials and my development slows down when I have to deal with a lot of math.

That said, I was able to create an app streaming my own terrain and imagery with a few lines of code. There is also WebGL Earth, a wrapper around Cesium giving you an API similar to well-known Leaflet. I expect to see more functions or wrappers to make stuff like camera positioning easier in the future.

How can you add your own terrain data to Cesium? 

First, you need to check if you really need it. You have the option to stream high-resolution terrain data directly from the servers at AGI. It's free to use on public sites under the terms of use. If you want to host the terrain data on your own servers, AGI provides a commercial product - the STK Terrain Server. Give it a try, if you have a budget!

I was looking for an open source solution, and found out that Cesium supports two terrain formats:
  1. heightmap
  2. quantized-mesh
The tiled heightmap format is similar to the one I used for three.js. Each tile contains 65 x 65 height values, which overlap their neighbors at their edges to create a seamless terrain. Cesium translates the heightmap tiles into a uniform triangle mesh, as I did in three.js. The downside of this format is the uniform grid, you use the same amount of data to represent both flat and hilly terrain.

The regular terrain mesh made from heightmap tiles. 

The quantized-mesh format follows the same tile structure as heightmap tiles, but each tile is better optimised for large-scale terrain rendering. Instead of creating a dense, uniform triangle mesh in the browser, an irregular triangle mesh is pre-rendered for each tile. It's a better representation of the landscape, having less detail in flat areas while increasing the density in steep terrain. The mesh terrain is also more memory efficient and renders faster.

The irregular terrain mesh from quantized-mesh tiles. Larger triangles have less height variation. 

Unfortunately, I haven't found any open source tools to create tiles in the quantized-mesh format - please notify me if you know how to do it!

You can generate heightmap tiles with Cesium Terrain Builder, a great command-line utility by Homme Zwaagstra at the GeoData Institute, University of Southampton.

I'm using the same elevation data as I did for my three.js maps, but this time in full 10 meter resolution. I'm just clipping the data to my focus area (Jotunheimen) using EPSG:4326, the World Geodetic System (WGS 84).

gdalwarp -t_srs EPSG:4326 -te 7.2 60.9 9.0 61.7 -co compress=lzw -r bilinear jotunheimen.vrt jotunheimen.tif

I went for the easy option, and installed Cesium Terrain Builder using the Docker image. First I installed Docker via Homebrew.  I was not able to mount my hard drive with this method, so I downloaded the elevation data from my public Dropbox folder:

wget https://dl.dropboxusercontent.com/u/1234567/jotunheimen.tif

I used the ctb-tile command to generate the tileset:

ctb-tile --output-dir ./tiles jotunheimen.tif

The command returned 65 000 tiles down to zoom level 15. I compressed the tiles into one file:

tar cvzf tiles.tar.gz tiles

and used the Dropbox uploader to get the tiles back to my hard drive:

./dropbox_uploader.sh upload tiles.tar.gz tiles.tar.gz

So I got 65 000 terrain tiles on my server, how can I see the beauty in Cesium? It required some extra work:
  1. First I had to add a missing top level tile that Cesium was expecting. 
  2. Cesium was also looking for a layer.json file which I had to create:

    {
      "tilejson": "2.1.0",
      "format": "heightmap-1.0",
      "version": "1.0.0",
      "scheme": "tms",
      "tiles": ["{z}/{x}/{y}.terrain?v={version}"]
    }
  3. Lastly, I added a .htaccess file to support CORS and gzipped terrain tiles: 

Then I was ready to go!

Beautiful terrain rendered with 10 m elevation data from the Norwegian Mapping Authority. Those who know Jotunheimen, will notice Skogadalsbøen by the river and Stølsnostind and Falketind surrounded by glaciers in the background.

The terrain is a bit blocky (see the mount Falketind to the left), but I'm not sure if this is happening in Cesium Terrain Builder or Cesium itself. The quantized-mesh tiles from AGI gives a better result. 

I'm not able so show an interactive version, as I'm using detailed aerial imagery from "Norge i bilder", which are not publicly available.

UPDATE: 

Friday, 29 August 2014

Geotagging and Picasa Web Albums API, or was it Google+ Photos?

In my last blog post, I presented a new plugin, Leaflet.Photo, that allows you to display geotagged photos from any source. Among them was Google+ Photos and Picasa Web Albums API. My plan is to use this API for my travel map, and this is why.

Does Picasa Web Albums still exist? 
It's a bit messy these days. Google is trying to transition from Picasa Web Albums to Google+ Photos, as photos are the number one things that people want to share on social networks. When you use Picasa to share your albums (Sync to Web), the album URL is now on your Google+ profile, and not on Picasa Web Albums (which is just redirecting me to Google+). This is the URL to the public album from my trip to the Trollfjord:

https://plus.google.com/photos/+BjørnSandvik/albums/6052628080819524545

It also works with your Google+ user id:

https://plus.google.com/photos/118196887774002693676/albums/6052628080819524545

My public Google+ web album. The album contains both photos and videos. 

The thing is, there is no Google+ API for photos and videos yet (apparently they were working on it back in 2011). But the Google Web Albums API still works on your Google+ albums.

Google Web Albums API is not the easiest API I've worked with, but it's flexible and quite fast. This is an XML feed of my public album from Trollfjord:

https://picasaweb.google.com/data/feed/api/user/118196887774002693676/albumid/6052628080819524545

The user number and album id is the same as above. Or better for your JavaScript apps, a JSON feed:

https://picasaweb.google.com/data/feed/api/user/118196887774002693676/albumid/6052628080819524545?alt=json

And if you're still using JSONP:

https://picasaweb.google.com/data/feed/api/user/118196887774002693676/albumid/6052628080819524545?alt=json-in-script

If you click on any of these links, you'll see that it's not a very compact format. There is a lot of data that you don't need. Although complicated, you can select the fields you want to include in the feed. This is how I selected the following elements:
  • Photo URL: entry/media:group/media:content
  • Photo caption: entry/media:group/media:description
  • Photo thumbnail URL: entry/media:group/media:thumbnail
  • Photo timestamp: entry/gphoto:timestamp
  • Photo location: entry/georss:where

This is the new URL:

https://picasaweb.google.com/data/feed/api/user/118196887774002693676/albumid/6052628080819524545?alt=json&fields=entry/media:group/media:content,entry/media:group/media:description,entry/media:group/media:thumbnail,entry/gphoto:timestamp,entry/georss:where

While researching, I also learnt that I could use the imgmax attribute to specify the size of the photos referenced in the photo URL. Neat!

So why should I use this (relatively) old API?
Compared to other popular social media sites, Google don't strip off the meta information of your photos. Instead it uses the build in support for image metadata extensively. Hopefully Google will continue to do this, although social media sites have reasons not doing so.

This means that Google don't lock you in. I can change the location of my photos using my GPS tracks, and it's reflected where I embed my photos. I can edit the image captions in Picasa and it's stored within the image file, allowing me to write the caption once and use it everywhere.

So what is my album workflow for my travel map. Before starting my journey, I'm creating a new Google+ album. The feed from this album is attached to my map, by simply passing on the album id. While on journey, I use the Google Photos app to add photos to the album, that will automagically show up on the map as well. Back from trip, I can add and edit photos from my digital camera in Picasa and sync them to the web album.

Update 12 September 2015: I'm having trouble uploading images with the Google photos app. The images are geotagged and the location data shows in the Google+ album, but unfortunately the location data is not included in the API feed. Please notify me if you're able to get this to work. 

Photos from Google+ shown on my travel map. 

PS! This blog post is not sponsored by Google :-) 

Showing geotagged photos on a Leaflet map

Using Instagram for my real-time travel map had too many limitations, so I decided to use Google+ photos or Picasa Web Albums instead. I've create a new plugin, Leaflet.Photo, that allows you to add geotagged photos to your map, from any source.

The plugin plays well with the great Leaflet.markercluster plugin, showing your photos in clusters. To make the plugin more versatile, it doesn't deal with AJAX loading or image presentation except the thumbnails on the map. Use your AJAX loader of choice, and simply pass on an array of photo objects to the plugin. 

The photo objects can include the properties you like, but the following are required:
  • lat: latitude of photo
  • lng: longitude og photo
  • thumbnail: url to thumbnail photo
I've kept the squared thumbnails of Instagram, as I think it look nicer than variable size thumbnails. Since the photos can have any dimensions, I'm using a CSS technique to crop and center the thumbnails. 

I've created three examples using Picasa Web Albums Data API, CartoDB (synced with Instagram) and Norvegiana API. With CartoDB you can easily get the required photo properties by manipulating the SQL query. Other APIs will require some data parsing before you pass on the photo objects to the plugin. All examples show the photos in a popup when you click/tap on them, but do whatever you like! On my travel map (click on "Bilder"), I'm using my own lightbox showing all photos in a cluster.

Google+ photos


Photos and videos from Google+. See the small animated GIF video thumbnails. 


Instagram / CartoDB


Instagram photos synced with CartoDB. 


Norvegiana 


Historic photos of Oslo from Norvegiana API. 



Enjoy! :-)

Thursday, 28 August 2014

Geotagging photos using GPS tracks, ExifTool and Picasa

I take a lot of photos while trekking, and most of the time I'm also carrying a GPS with me. As my camera don't have a built-in GPS, my photos are not geotagged while shooting. Luckily, this is an easy task if you've kept your GPS logs from the trip. 

I'm still very happy with my Garmin GPSmap 60CSx that I bought 7 years ago. By changing the setup, the GPS allows me to automatically save the tracks to the memory card. I get one GPX file for each day trekking named with the date. I can easily transfer these tracks to my computer or smartphone with a cable or a card reader. 

Before I converted to Mac, I used GeoSetter to geotag my photos on Windows. Now, I want to do it on the command line using the great ExifTool by Phil Harvey. I installed it on my MacBook using Homebrew:

brew install exiftool

After copying my GPX file to the image folder, I'm simply running:

exiftool -geotag=my.gpx ./

If you forgot to sync the camera and GPS time before your trip, you can use the geosync-option to fix it: (60:00 = 60 minutes):

exiftool -geotag=20140329.gpx -geosync=-60:00 ./

You have a lot of options, so make sure to read the "Geotagging with ExifTool" documentation. ExifTool is modifying the Exif headers of your image files, storing the location data in the same file. 

To see the result on a map, I'm using Picasa.  


Click the map pin button (bottom right) to see the map. If the positions are not shown on the map, try to right-click the image folder and select "Refresh Thumbnails". 

If you don't have a GPS track you can always use Picasa to manually geotag your photos. 

Be aware! I just learnt that social media sites like Facebook, Twitter and Instagram removes the Exif data from your images. Google+ don't. 

Now, how can you display the photos on your own map? It will be the topic of my next blog post. 

Wednesday, 27 August 2014

Live tracking in Lofoten and Vesterålen

Last weekend, I had a great trip to scenic Lofoten and Vesterålen in Northern Norway. I brought my tracking gear to test my new real time travel map. How did it go?

Our first trip was to Trollfjord, a 2 km long fjord with a narrow entrance and steep-sided mountains. It's a famous tourist spot in the Lofoten archipelago, but not many leave the boat at the fjord's end to hike up to the Trollfjord hut.

The small Trollfjord hut.

Trollfjord goes in an east-west direction, and I expected to be in the "satellite shadow" being far north and having steep mountains blocking the sky towards the south. My good old Garmin GPSmap 60CSx did well in the rugged landscape, while my satellite SPOT messenger had some difficulties finding and sending positions. 

Live track from my SPOT messenger (interactive map).

GPS track from my Garmin GPS.

The great thing of using CartoDB to sync my SPOT-data, is that you can edit your positions with ease after the trip. 

Tip! The default basemaps in CartoDB are not very detailed for Norwegian mountains, but you can easily add a basemap (Topo2) from the Norwegian Mapping Authority ("Kartverket") with this URL: 

http://opencache.statkart.no/gatekeeper/gk/gk.open_gmaps?layers=topo2&zoom={z}&x={x}&y={y}

Changing the basemap of CartoDB.  


It's then easy to edit or delete the wrong positions:


Be aware! I was struggling editing my CartoDB-tables from my smartphone, but it was not possible to edit the content of table cells. Hopefully, the CartoDB team will make their editor more mobile friendly in the future. 


Another issue was to get the correct time and position of Instagram photos on the map. Trollfjord is an area with poor mobile coverage. When I took photos with the Instagram app it was struggling placing the photos on the map. It worked better if I took the photos the the built-in camera app of my phone (with geotagging activated) and then posting the photo with the Instagram app. 


If I didn't have mobile coverage, I would just retry posting the photo when back to civilisation. The time associated with the image is when it was sent and not taken. I'm going to check if I can extract the shooting time from the Exif headers of the image.

Our second trip went through Møysalen national park, one of very few national parks in Norway that goes all the way down to sea level. 

Møysalen national park

Here we went in a south-north direction, and my SPOT messenger did better as there was less mountains blocking the satellites. 

Map and elevation profile of a 2-day hike through Møysalen national park (interactive map).  

The web service from the Norwegian Mapping Authority ("Kartverket") seemed to have some technical troubles this weekend, so the altitude values and place names was not updated instantly. When the web service was failing my script stopped and the weather report from yr.no was not fetched either. I'm going to improve the error handling before my next trip. 

I also took a lot of photos with my compact camera while trekking, and I would like to show these on the map as well. My camera don't have a GPS receiver, but I should be able to geotag my photos by using my GPS track. It will be the topic of my next blog post. In the meanwhile, here are some of the photos: 

Hurtigruta in Trollfjord

Trollfjord by night

Cloudberries

Trollfjordtindan.

Seagull

Seagulls

Sea eagle in Raftsundet.

Durmålstindan

Tverrelvtindan

Cold and fresh bath at Snytindhytta.