The Most Interesting Mapper

The Most Interesting Mapper in the World

In case you live under a rock, this particular meme has its origins in a set of beer commercials that I find kind of amusing. Here is a decent compilation of them.

<sorry to interrupt this completely awesome post but quick note: This plugin has been turned into a core JOSM feature. See "Update 2" at the bottom of this post>

But seriously, the ImproveWayAccuracy plugin is a wonderful tool. Perhaps the prime use case to show it off is a smoothly curving road that was originally either mapped from low resolution imagery or (around here) imported from TIGER data that may have used far too few nodes to make a smooth curve. To fix this by hand is highly annoying. You have to drag nodes around and then grab the + between two existing nodes to create a new one and drag it to where you want it, repeating that as often as needed to achieve a smooth enough curve. Each node requires several precise mouse movements. 

It is much easier to draw a way from scratch where you just move your mouse along the curve and click as needed to form a smooth looking curve. Well, the ImproveWayAccuracy plugin makes this possible to do with existing nodes/ways!

Random computer upgrade notes

I recently did some reshuffling of the computers in my home. I built a brand new gaming system from parts I bought on newegg. Intel i5 Sandybridge, 16 GB of memory and an nvidia video card that not only has a 120mm fan, but has two of them. The insanity. Then I used the parts from the old gaming system (first generation i5, 12 GB, ATI graphics) to give my ancient linux box a major upgrade.

Of Linux and Distros
The Linux box was so old (AMD Athlon x2 4400+ with 3 GB) that it was becoming a pain to use for any kind of desktop applications. I ended up neglecting it and spending most of my time on my windows box (I know! The horror!) with multiple putty sessions open. Also, it was running Gentoo Linux. In my local Linux user group, we have determined that people who are over 30 don't use Gentoo. It is a worthy distro and has many strengths. I have been using it for over 10 years on some system or another and because of its minimal nature, I learned a lot about Linux that I would otherwise be oblivious to. But it can be a real pain, especially to maintain for desktop use. I was getting tired of every other emerge --sync generating package conflicts, lengthy config file merges and the occasional system toolchain breakage that you had to resolve before rebooting (or loosing power!)  otherwise the system would completely eat itself. So, being a distinguished 31 years of age now, I decided it was time to drop Gentoo. So long and thanks for all the fish.

Simple OpenStreetMap tile rendering with Maperitive

Lately there have been a lot of people popping into the OSM IRC channel (#osm on oftc.net) because they have been getting the "you've been bad" tile from the OSM tile server:

The reason is that they are using tile scraping applications that violate the OSM tile usage policy. Typically people just want to download tiles for their city or a vacation destination or travel route in order to pre-load them on a mobile device for offline use.

This is of course a perfect use case for OSM. The problem is that OSM is a project run by volunteers and with limited server capacity. Our primary product is geographic data, not map tiles. So when these tile scraping programs start hitting high zoom levels, it can cause serious problems for the tile server. The solution is to download OSM data instead of rendered tiles. Then you can render your own tiles or even use offline vector rendering.

One tool to make your own tiles is Maperitive. It is pretty easy to use and by default generates tiles that are fairly similar in style to the default mapnik style you see on osm.org although it is very customizable. It is a .NET application although it claims to work in Linux/OSX with mono. Also, while it is free to download, it is not open source which is kind of rare for the OSM world. These two things may cause some people to have an allergic reaction (I'm looking at you, Serge) but if you're still reading, here is a quick lesson on how to use it to render your own tiles. If you keep reading, you should be able to start rendering your own tiles in about 10 minutes. There is a tutorial video and some detailed documentation on the Maperitive website but I don't see a basic text based tutorial that walks you through the common "render some tiles for this area" use case.

Getting Started
Installation is pretty straightforward. Just head over to http://maperitive.net/ and find the "Download" link. Grab the latest .zip file (Maperitive-1228.zip right now) and download it. Once the download finishes, just unzip the Maperitive folder inside of the zip file somewhere and you're done. Open the Maperitive.exe file to fire it up.

Cleanup on the Map Aisle

Spills happen. A sticky honey jar in the grocery store. A glass of orange juice at breakfast. Oil spills in the Gulf of Mexico. Node spills in OpenStreetMap. Some can be cleaned up with a mop and some water. Others require a more technical solution. What is a "node spill" you ask? It's when nodes get uploaded to OSM that don't have any tags on them and aren't part of a way. These tagless, unconnected nodes add no useful information to the map and are just dead weight in the database. Where do they come from? There are at least a couple of common sources of node spills. One is editor bugs and simple user error. These are generally pretty small spills of 10s or maybe 100s of nodes. The bigger problem is imports.

The topic of imports deserves its own post. For now I'll just say that badly performed imports and insufficient checking after imports can lead to 10s of thousands of empty nodes. The basic problem is that nodes get uploaded first and don't become part of a way until the way is uploaded later. So if something goes wrong with the way upload, the nodes end up in an orphaned state.

I have come across a few of these nodes before but what really caught my attention was a failed NHD (National Hydrography Dataset) import in Oklahoma. It happened to poke up into Kansas a little bit where I noticed it next to a state highway I was editing. After a lengthy thread on the talk-us mailing list I eventually found a good way to detect these nodes. Since then I have made my way across the US and some of Canada deleting these useless nodes from the database. As detailed in this message it goes something like this:

• Use the XAPI to perform a query of the form /node[not(way)][bbox=a,b,c,d]
• Open the result of this query in JOSM and apply a filter to hide all nodes with tags
• Perform some checks to make sure there really is no useful data
• Delete nodes and upload

The Main Attraction

I have mentioned the TIGER import several times. The initial import tagged roads with abbreviations in their name. For example "Ave" instead of "Avenue" or "St" instead of "Street." However the general rule in OSM is to not abbreviate things. So a script was devised to automatically expand abbreviations. This script was run by the user balrog-kun. Until tonight I assumed that it had been run over all of the TIGER data. But then I saw a few abbreviations while working on some highways in Kansas and decided to dig a little deeper. I did some XAPI queries (using the database I talked about in a previous post!)  to see if anything interesting stood out. I did one query for all streets named "Main St" and another for all roads named "Main Street" and here is what I got back.

Green is "Main St" and the rest is "Main Street"

One Mapper, One Year

Today is the one year anniversary of my first OpenStreetMap edit! I thought I would take the opportunity to reflect on how OSM and myself have had an effect on each other. My intention is not to brag about all the mapping I have done but rather to point out some different ways in which contributions can be made and some of the tools I use when mapping.

My very first edit was to add a foot/bike path near my apartment so that it would show up on OpenCycleMap. Unfortunately the imported TIGER roads were so far off that I had to realign them before I could add the path in the correct location. And of course correcting the ones right around the path led me to correct the ones connecting to those and the ones connecting to those and so on... At the time I didn't even know what TIGER was or what all those tiger:* tags were on all the roads.

Some highlights since then:

• GPS trackpoints uploaded: 283,271 in 75 traces (most from my Garmin Edge 305)
• 410 changesets created using Potlatch, JOSM, OsmAnd and Vespucci
• Pictures taken: over 2,000 (4.5 GB saved)  Most were geotagged to a GPS trace to aid in mapping features (uploaded some of them to openstreetview.org)
• Some of the locations I have edited: 
• My university campus
• My city
• My state
• My country
• My high school
• My childhood home
• Some random place where there was an earthquake
• Map objects affected (according to hdyc.neis-one.org - last updated in January)
• Nodes: about 165,000
• Ways: about 24,500
• Relations: about 500 (I created over 200 county relations in 3 states since the last update)
• Contacted multiple government agencies in Kansas to get explicit permission to use their data to assist in mapping. Documented these data sources on the Kansas page in the wiki
• Started a blog (self referring link!)

Now for a pretty picture. Here are all the GPS traces in my city. I think there may be one or two from other people but most of it is me.

Keeping a database up to date with osmosis

This is a follow-up to my previous post about importing a planet file into a pgsnapshot database schema. However the same basic process applies to other schemas that osmosis can write to.

To briefly set the stage: After downloading the weekly planet file and processing/importing it into a database, the data is now probably 3 or 4 days old. OpenStreetMap provides both minutely and hourly change files that can be applied to the database to bring it up to date with what is live right now in the master OSM database. If you think about it, that is actually a pretty sweet deal. Someone on the other side of the planet can spot a new restaurant while on their lunch break, use an application on their phone to upload it to OSM and you will get the change applied to your local database in less than  120 seconds. Does this blow anyone else's mind just a little? Open data is awesome like that.

Parse the Planet! (into a pgsnapshot database)

What makes OpenStreetMap... well... Open? In a link: http://planet.openstreetmap.org/

Yes, of course there are the legal issues of having an open license, the transparent community processes to govern the project and a whole host of open source software to create, manipulate, consume and display map data. But at the end of the day, none of that would matter if the raw data wasn't available.

As is obvious from my previous posts, I have dabbled in manipulating this raw data in the past, with varying degrees of success. After learning from my previous attempts and hanging around on the mailing lists and IRC channel, picking up information here and there I decided to take another crack at processing a planet file. I ended up finding a partner in crime: Paul Norman. He was wanting to import a planet file for use in a private XAPI server using the new java based XAPI implementation written by Ian Dees. This sounded like a good idea so I decided to follow along. We bounced commands and results off of each other for a while. I ended up going a slightly different route in some of the processing because I had some different hardware available to me.

The first thing to know about processing planet data is that there are several different database schemas that can be used to hold the data. This topic probably deserves its own post at some point. Here is a brief list of the ones I am at least minimally familiar with. I know there are some others (like the gazetteer database used by nominatim):

• API database: Creates a duplicate of the master OSM database. Does not use any GIS features. Nodes are stored with lat/lon values as simple numbers. Contains tables for: ways, tags, users, changesets. Can track history of objects once you start applying updates.
• "pgsnapshot" schema: creates a schema that utilizes the GIS features of postgis that allows you to run spatial queries against OSM data. Still maintains all tag information but they are in a single "hstore" column, not in their own relational tables. Changesets are not stored nor is any information about users except the ID and username. History is not maintained as updates are applied. There is also a related "pgsimple" schema that seems to be an older version of the pgsnapshot schema that may have some advantages. Differences are discussed in the "pgsnapshot_and_pgsimple.txt" file in the osmosis script directory.
• osm2pgsql rendering schema: This schema is created by the osm2pgsql tool and used by mapnik to render map tiles. It uses GIS features as well but it is lossy and very de-normalized. Only objects and tags that you care about rendering are imported. This means it is also smaller and faster to get set up.

The XAPI uses the pgsnapshot schema so this is what I set up. The instructions on how to get going are provided in the xapi readme file but I will go ahead and copy it here with some additional notes.

Dual Carriageways

When I first started editing OSM, I was mostly doing residential roads and bike trails. Eventually I worked my way to major roads. Some of them had dual carriageways. In my case these were also US/State highways. Somehow I got the two mixed up in my head and thought that mapping any dual carriageway road required relations. Since I didn't know much about relations at the time I was kind of scared to touch them. Turns out relations ARE used to describe highways however they are not required for dual carriageway roads.

Since I've recently been working on "dualizing" roads in the Kansas City area, I thought I would throw up a quick post about the process. It has probably already been done (I seem to recall a "TIGER fixup" tutorial that described the process) but here is my version.

To Start off with, let's define a dual carriageway road. There may be some difference of opinion (there usually is in OSM) but in general it is any road where the opposing streams of traffic are separated by a physical barrier. A grass median (think interstates) or a strip of trees in the middle of a boulevard are prime examples. Sometimes the barrier is just a small strip of raised pavement or bricks. Regardless, it is something that would make your car go "bump" or even cause damage if you were to try and drive through/over it.

Airports, Part 2

In my first installment of the airport series I presented some basic mapping techniques I used while mapping rural airports in Kansas. One comment I got on IRC suggested using a slightly different tagging scheme for displaced thresholds. The suggestion (I believe it was from pnorman) was to tag it with two tags. One being aeroway=runway and the other being runway=displaced_threshold. While I haven't gone back and changed my previous work, this may be a better way of doing it. First of all, they would get rendered on the default map. I know, I know... "don't tag for the renderer!" But here's the thing: This is technically not incorrect. Displaced thresholds are considered part of the runway and are included when the runway length is specified. So tagging it as a runway but then specifying that it is a specific type of runway is not incorrect tagging. On standard maps it would be rendered as part of the runway but someone making an aeronautical themed map could use the runway=displaced_threshold tag to differentiate it and produce a customized rendering.

Before I go on, I am just going to repost a link to the OSM wiki page for Airports as a reference.

Now on to some new material. I was moving west-to-east through Kansas so the first "big deal" airport I came to was the Salina Municipal Airport. Its 17/35 runway is a massive 12,300 feet (3.8 kilometers) long. Any aircraft currently in service could land here with room to spare. Why does a municipal airport in a city of under 50,000 people in the middle of Kansas have such a massive runway? The military, of course! This facility was built at the beginning of World War II and was used to train heavy bomber crews. Then during the cold war it became part of Strategic Air Command operations until it was closed and turned over to the city of Salina in 1965. (source: wikipedia) Now it serves mostly general aviation but is also used by Kansas State University's aviation program which is based at the Salina campus. A few years ago it was used by Steve Fossett as the start and end point of his solo, nonstop flight around the globe. And I got to bill them for the telecommunications services they used while doing it! (My current day job deals a lot with the telecom billing system at KSU)

I promise, I'll get to the mapping in this post, eventually. But I should point out that this is one thing I love about contributing to OSM. Seeing something unusual in aerial imagery can often lead to an hour spent on wikipedia or a fun discussion on IRC involving people from around the world or, occasionally, a field trip to go see what the heck is happening "on the ground" to cause it to show up a certain way in aerial imagery.

Airports, Part 1

For the past few days I've gone on an airport binge in OpenStreetMap. I ended up learning a lot about airports. It was fun to start mapping, then come across something different or notice a new detail and realizing that this too can be mapped! Here are my musings on the subject. Feel free to tell me I did it all wrong in the comments :)

The recent import of airport nodes from www.ourairports.com and the subsequent discussion sparked the idea to map easily identifiable runways from aerial imagery. So I downloaded all "aeroway=aerodrome" nodes within Kansas from the XAPI. In addition to the above mentioned import, a lot of airports were imported from USGS GNIS (Geographic Names Information System) back in early 2009. In total there were just under 600.

This being Kansas, a vast majority of those 600 "airports" are nothing more than a flat spot in a field where farmer Joe lands his crop duster. They have such names as "Lundgren Hereford Ranch Airport" or "Flying N Ranch Airport." They should probably be downgraded from "aerodrome" to something else and not rendered at zoom level 10 like they are now. Right now looking at western Kansas at z10 reveals a ridiculous number of airports being rendered. I have yet to see any consensus on a tagging scheme for smaller airports so I mostly just ignored them. Additionally, the aerial imagery I was using (USGS NAIP) wasn't really good enough to reliably pinpoint these grass strips in the middle green fields. It also didn't help that the GNIS information is often off by quite a bit so I wasn't sure if the runway was just blending in or if there really wasn't one in the field I was looking at but rather 2 fields over.

The more interesting airports are the municipal air fields. Most of these have a paved runway, sometimes in addition to a grass one. Even small airports stick out when they have a paved runway: