County Borders in OpenStreetMap

I have had a fairly steady relationship with county borders in OSM. They were originally imported before I joined the project in 2010. However they were imported as overlapping closed ways which is not ideal. In some places people had started working on de-duplicating the ways and turning them into relations. However this is tedious work and some of the people who attempted it didn't quite understand how relations worked so it was pretty much a big mess.

Throughout 2011 and 2012 I periodically came back and worked on county borders and eventually got to a place where there was a relation for every county in the nation. There are still some minor differences in tagging but I wasn't as concerned with that and tried not to change the work of other mappers as long as the relation existed and had valid geometry. At the same time I also reduced complexity at state borders. There used to be a mess of admin boundary ways. One for the state, one for each of the counties on either side of the state border and possibly also some city boundaries. They usually didn't match up very well either so it looked terrible. I didn't mess with cities but at least I made the state and county borders share a single way to reduce clutter.

At some point I also noticed (thanks to the MapQuest Open rendering which renders county names) that the place=county nodes were all placed at the extreme eastern border of all the counties. This caused the county names to be rendered in unexpected locations. So I ended up fixing those as well. Here is a screenshot of the last changeset where I moved over 1,100 nodes to the middle(ish) of their respective counties:

The Effects

So that was nice. But there was still some weirdness with counties, particularly when geocoding in Nominatim. Especially if there was a city with the same name as a county. For example, I searched for "Kearney, NE" and the first result was the place=county node for Kearney County in Nebraska. What I was looking for and expected to find first was the city of Kearney which is in Buffalo county. Then there is the issue of Nominatim finding both the county node and the boundary relation when you search for a county. The same thing often happens with cities. Really, they are the same thing and should show up as one result. And sometimes it might. Nominatim tries to be smart about this and does remarkably well but sometimes it needs help.

You might say we should just delete the node since we already have the boundary information. But very few renderers use boundary relations to render place names from. They all rely on these nodes. The same goes for cities, states and even countries I believe. There is also the oddness of what to use for county name. The relations all have a "name=Kearney County" tag while the nodes just have "name=Kearney".

After poking around a little and bugging lonvia (maintainer of nominatim.osm.org) about it a couple of times, I learned a few things about Nominatim and was able to make some fixes. It was my intent to come back to this some day and figure it all out and fix up county relations once and for all. I still hadn't gotten around to that but just in the last few days, new user revent was getting annoyed at the geocoding problems in Texas and really dove into Nominatim to figure out the best practices. He wrote up a diary entry on the subject and we have since then talked together some more on IRC and since I had work cancelled today because of snow I thought I would write up this post about my current understanding of things:

• Nominatim is able to "link" multiple elements into a single feature. It tries to do this automatically but doesn't always get it right.
• Nominatim uses Wikipedia to help determine the importance of a place.
• Nominatim makes use of the alt_name tag.
• After editing the map, Nominatim does not always fully reindex an area. It might depend on current workload of the server so things can seem inconsistent. But a manual reindex can be forced by an admin after a large area has undergone a lot of edits.

The linking is important because this allows the county border and the node to be linked into a single feature so that only one result is returned when you search for a particular county. It also improves addressing since Nominatim no longer has to choose between two different things to associate addresses with. This linking can be accomplished by adding the node to the county border relation with a role of "label".

The use of Wikipedia is interesting and (I think) the source of my problems with Kearney, NE that I mentioned above. Nominatim was linking the county node to the Kearney, Nebraska Wikipedia page based on name tag. This helped bump its importance up to where it was returned before the city. Now that I have added the Kearney county node to the Kearney county relation and specified a "wikipedia=en:Kearney County, Nebraska" tag, it links things to the correct Wikipedia page and the search results return the city first, as expected.

The Fix

So to summarize, I believe revent and I have settled on the ideal way to map a county. Using Kearney county as an example:
One relation with the following tags:

type=boundary
boundary=administrative
admin_level=6
name=Kearney County
alt_name=Kearney
wikipedia=en:Kearney County, Nebraska

In addition, the following tags are in widespread use:

border_type=county
nist:state_fips=31
nist:fips_code=31099

I'm not sure the border_type tag is actually used for much but it has a fair amount of existing use and seems to further differentiate borders that are the same admin_level based on local conventions - county vs borough and such. The two nist: tags were part of the import and I have actually used the state_fips as an easy way to download all county relations in a given state to work on. These FIPS codes have been withdrawn by NIST recently but are still commonly used. We may need to look at replacing or augmenting these tags with an ISO standard at some point but for now they work.

In addition to the relation tags the following relation memberships are recommended:

• Obviously the ways that make up the border with a role of "outer" in the correct order so that they form a closed ring
• The node with the following two tags: place=county and name=Kearney with a role of "label"
• The place=city/town node of the county seat with a role of "admin_centre"

The county seat city node as an admin centre (yes, use the British spelling) is kind of a new thing to me. I'm not 100% sold on it but it does seem like useful information to have.

If you have looked at very many county nodes you might notice that the "is_in" tag is conspicuously absent. It turns out, it causes more problems than it solves and should probably be left off or removed. I have always found it a bit silly anyway. OSM is a geodatabase - there is no need to indicate location in a tag.

I have also been promoting any county seats that were place=village to place=town. I know I saw this as a thing on the OSM wiki at one point although I can't find it right now. While they may not meet the population guidelines of a "town" according to the wiki, I think county seats carry extra importance due to their governmental functions. It also means that at least some city names are rendered on maps at low zoom levels in low population areas. Without this promotion you could pan around in the Kansas/Nebraska/Dakotas area for several screens at zoom level 9 or 10 and not see a single city name on the map on osm.org. In some ways this is a rendering issue but I don't feel like this is an egregious case of tagging for the renderer.

Your Turn!

So if you are looking for something to do, jump in and make sure your county is correctly linked and tagged. I have already done Kansas, Nebraska and South Dakota but I probably won't have time to do everything in the near future and I will concentrate on lower population areas first so please feel free to jump in! If there is interest, I may do another post about my workflow to do a state-wide edit. Otherwise you can always find me on the #osm or #osm-us IRC channels if you have questions.

UPDATE! I have since written another post detailing my workflow: http://ksmapper.blogspot.com/2014/02/workflow-for-fixing-county-borders.html

Incidentally, a lot of the same concepts apply to cities as well. If you do a Nominatim search for "Los Angeles, CA" right now it will return two results. One for the node and one for the boundary relation. Making the node a member of the relation should fix that.
(sigh. In doing that search I just saw that the relation for LA is completely broken anyway. Admin boundaries suck.)

In closing, here is a screen shot of the completed county relations in South Dakota:

Too Much of a Good Thing

What makes OpenStreetMap great? Well there may be multiple answers from different people. But one thing is certain: Our geodata is awesome. I mean it is kind of the reason that we exist. So more is better, right? We are, after all, trying to map the entire planet.

But it turns out, there can actually be too much of a good thing. One of my first encounters with such a situation was while doing some fixing up of imported TIGER data somewhere out west. I came across this:

Notice the scale. We are looking at a little over 100 meters of country highway and there are over 300 nodes in this image. This is obviously beyond ridiculous and I can assure you that this particular instance was cleaned up long ago and those nodes met their appropriate end in a special level of /dev/null.

But this is not an isolated incident. I keep coming across such over-noding. So I decided to use my local planet database (which I previously blogged about setting up) to try and find out how much this really happens and where such things come from.

Preparing for Operation Cowboy

If you haven't heard, there's a major mapping event about to happen. Dubbed Operation Cowboy, this will be the second event of its kind, the first being Night of the Living Maps which happened back in February. As described on the wiki, these events are kind of like a global LAN party for maps. For one weekend people who choose to participate will focus their mapping efforts on a common goal. This time the organizers decided to focus on the USA.

We of course have plenty of mapping to do around here. Cleaning up and updating the TIGER road import is kind of a never-ending task. In some of the sparsely populated areas of the country there aren't really any local mappers to speak of so there is ample work for "armchair mappers" to come in and do some map maintenance.

Considering the audience and the event, I decided to try and get a few things tweaked before Operation Cowboy goes down.

TIGER 2012 road name tiles

First up was improving the TIGER 2012 road name tiles being served up by the OSM-US tile server. They were originally set up by Ian Dees and are a great resource for tracing roads that were built since the original TIGER data was imported. The only issue I had with them is that they show the street name in the Census Bureau's abbreviated format. In OSM we prefer to use unabbreviated names because it reduces ambiguity. Does "St" mean "Street" or "Saint"? While that one might not be too hard to figure out and expanding "Ave" to "Avenue" might be trivial, there are a lot of abbreviations in the data and some of them are not easy to guess if you aren't familiar with the area.

Any guesses as to what the following mean? Brg. Cll. Natl Lkshr. Ofc. Pso. RM. BIA Rte? (see answer key at end of post) Most people in the US can probably guess some of them but there are definitely some regional ones in there that many people may not be familiar with. Then add in abbreviated directional prefixes. And now suppose you are a European trying to figure all of this out during Operation Cowboy. It's going to look like a secret code.

I won't go into the technical details in this post (might save that for another one after I clean up the process doumentation) but it did involve 27 GB of data in 3,200 .dbf files, about 10 different command line utilties and several large SQL queries. The end result was a mapping of 10.8 million line IDs (linearid field in TIGER speak) to fully expanded road names which I applied to the rendering database. The result can be seen on this tile preview page.

Imagery preset updates

Related to the road name tiles, I noticed that Potlatch2 had a TIGER road name tiles preset in the "Background" menu but it was still pointing at tiles with 2011 data in them. Thanks to the magic of github, I was able to get that fixed easily.

Then I noticed that the JOSM imagery sources page didn't have this tile set listed at all. That one was a little trickier to fix. It involved coming up with a simplified polygon of the US border to define the available extent of the tile layer as well as putting a base64 encoded image into an XML tag on the JOSM wiki. The image+base64+XML+wiki thing seems a little nasty but the JOSM functionality that this enables is actually pretty neat. If you download OSM data inside of the US, it will pop up a notice that informs you that that there is additional background imagery available and lets you add it to the "Imagery" menu with a single click. The base64 encoded image is displayed next to the menu option. In this case it is the TIGER logo:

RAWR!

TIGER tag removal

The TIGER road import put some tags on imported objects that weren't really necessary. As people edit roads, some of the tags become even less useful as the information from the tiger tags diverges from the "normal" OSM tags. Some of the tags were useful at the time for the upload process itself but are now just annoying bloat that confuses new users.

Back in July I started a mailing list thread about having editors automatically drop some of these tags like they already drop the created_by tag. Out of that discussion came a JOSM trac ticket and patch that does exactly this. Right before uploading, JOSM checks to see if any of the modified objects it is about to upload have "discardable" TIGER tags on them and removes them.

In September, I opened a ticket against Potlatch2 with the same feature request. Since nothing had happened with that ticket and I already had the Potlatch2 repository cloned from the tile URL change pull request, I decided to take matters into my own hands. I had never so much as glanced at ActionScript but fortunately P2 was a breeze to get running in a development environment. The developer documentation page on the wiki got me up and running in a matter of minutes. Although I did have to give ant more PermGen space to get through compilation. I should probably update the wiki with that tidbit...

After a little over an hour I had another pull request fashioned which Richard was kind enough to merge the next day. I believe this went live on osm.org within the last few hours. So now every way that is touched during Operation Cowboy will help to clean this clutter out of the database without the user having to do anything.

YEEEEHAW!

So that's how I prepared for Operation Cowboy. Unfortunately the actual event will take place over the Thanksgiving weekend which means I will be in Nebraska with very limited internet access so I won't be able to participate much with actual mapping. I will probably manage to hang out in the IRC channel a bit to try and field some questions if they come up. Please join me! (#opc2012 on irc.oftc.net)

Answer key for TIGER abbreviations:

Brg = Bridge

Cll = Calle (Spanish)

Natl Lkshr = National Lakeshore

Ofc = Office

Pso = Paseo (Spanish)

RM = Ranch to Market Road

BIA Rte = Bureau of Indian Affairs Route

Cleanup on the Map Aisle

Spills happen. A sticky honey jar in the grocery store. A glass of orange juice at breakfast. Oil spills in the Gulf of Mexico. Node spills in OpenStreetMap. Some can be cleaned up with a mop and some water. Others require a more technical solution. What is a "node spill" you ask? It's when nodes get uploaded to OSM that don't have any tags on them and aren't part of a way. These tagless, unconnected nodes add no useful information to the map and are just dead weight in the database. Where do they come from? There are at least a couple of common sources of node spills. One is editor bugs and simple user error. These are generally pretty small spills of 10s or maybe 100s of nodes. The bigger problem is imports.

The topic of imports deserves its own post. For now I'll just say that badly performed imports and insufficient checking after imports can lead to 10s of thousands of empty nodes. The basic problem is that nodes get uploaded first and don't become part of a way until the way is uploaded later. So if something goes wrong with the way upload, the nodes end up in an orphaned state.

I have come across a few of these nodes before but what really caught my attention was a failed NHD (National Hydrography Dataset) import in Oklahoma. It happened to poke up into Kansas a little bit where I noticed it next to a state highway I was editing. After a lengthy thread on the talk-us mailing list I eventually found a good way to detect these nodes. Since then I have made my way across the US and some of Canada deleting these useless nodes from the database. As detailed in this message it goes something like this:

• Use the XAPI to perform a query of the form /node[not(way)][bbox=a,b,c,d]
• Open the result of this query in JOSM and apply a filter to hide all nodes with tags
• Perform some checks to make sure there really is no useful data
• Delete nodes and upload