North River Geographic Systems Inc

Geospatial Problem Solving

Open Street Map

Finally pulling data out of OSM

……as opposed to pushing data into OSM.

So I’ve been working on a OpenStreetMap workshop for URISA. I had originally written it 6 or 7 years ago and it’s been touched in spots but not “re-written”. I am doing a top to bottom shakedown of the workshop and fixing slides and adding new information. It’s been like opening a time capsule of sorts of what has changed and what hasn’t changed in OSM/My Life over the last 7 years.

What does this have to do with anything?

When I had last touched the workshop I’m not 100% sure that Overpass was a thing. I know there was a tool to extract OSM data into QGIS and there was a tool in ArcMap to pull down data. You could always download a planet file or use JOSM to download data.

Over the last few years I became aware of overpass but hadn’t used it. I decided to dig into it just a bit for the benefit of the workshop. Overpass lets you specify tags (or attributes) and run that query against the OSM data. You can then download it. Of course you need to be gentle – running a large area with a lot of data will timeout – you’re better off grabbing a OSM file from Geofabrik and processing that.

As of late I’ve gotten fascinated by fire hydrants in OSM – so I ran a query of just TN for Fire hydrants. For the record there are 837 fire hydrants in and around TN. I posted the Overpass query below: 

[out:xml][timeout:25];
// gather results
(
  // query part for: “amenity=fire_hydrant”
  node["emergency"="fire_hydrant"](35.00208, -90.34742, 36.61557, -81.64104);
);
// print results
out body;
>;
out skel qt;

So what? Well – now that I have this query written I can actually send it to overpass by command line: 

wget -O output.osm --post-file=firehydrant.osm "https://overpass-api.de/api/interpreter"

I can now grab OSM data when I want it (within reason). I could set up a script to run once a week or month and see what has changed.

How does this fit into my favorite piece of software? QGIS has a plugin called QuickOSM .

QuickOSM is leveraging overpass to grab data. I can use place names like Chattanooga to constrain my download. I can specify my overpass file from above and use that. It will even let you parse a OSM file and extract data from that.

Anyway – it was a fun hour of killing time and pulling data out of OSM vs putting data into OSM. Plus if OSM is all you have – the Quick OSM plugin makes it easy to leverage that data to create a map.

Parking Lots and Lines

New Year – Different posts.

I love Listservs – I’ve resigned from all the OSM listservs though. It takes the stress level way down. Of course I still edit. Chattanooga from an OSM standpoint is pretty decent. Three things I usually tackle are buildings, parking lots, and Fire Hydrants. Why parking lots? If you ever wondered how much space we devote to parking cars….try outlining them. I want to fully map parking spots this year with number of spaces, handicap access, etc.

Me and JOSM

Two things are problematic in my OSM mapping around here:

  • I don’t have a drone – hence I don’t have updated imagery. I have several tricks which gives me nearly current imagery. That and some field work and it’s mostly all good.
  • Now vs 10 years ago has brought more editors. I’m always curious who – is it local or someone armchair mapping. I really don’t care as long as the map is updated correctly.

My mapping isn’t continuous and considering how much free time I had in 2020, I decided not to map and instead go canoeing. When I finally did jump back in someone had been adding a lot of service roads and parking isles. I check and it’s Amazon Logistics.

The screenshot below shows nothing more than lines – BUT you will notice some small gate-like symbology. Apparently the drivers are relaying info back to the editors. If you look at the page for the editors and reviewers there is a LOT of them also. I welcome my new Amazon Overlords. I’m the guy putting in the parking lots.

Anyway – it did strike me that you have yet another giant corporation harvesting out of OSM. I guess that speaks to the overall quality of what they are getting. I do wonder how much the map changes from these editors. It’s a complicated mapping life out here at the moment. You may have an entire delivery infrastructure relying on this data. That I am contributing to…..

With that I decided to join OSM US for the low low price of whatever it was. That breaks several years long standing of “Join nothing” but it is a New Year.

OSM in Chattanooga

So my back and forth with OSM continues these days. Overall I enjoy it If I stay out of the way and map. I’m also tired of mapping mostly by myself – there are a few here that help and the High School chips in when they have time.

Probably 4 or 5 years ago I started a facebook page for OSM (because that was as organized as I had planned on making this). We are kickstarting the group again. So there will be a small meetup at the Tap House tomorrow at 6:30 EST

I’d love to get more of the “new” part of the River Walk finalized in OSM. Someone GPS’d it – now I’d like to fill out the amenities such as bathrooms and bike “stuff”.

If you are in town Swing by – if you’re interested in more meetups add yourself to the group. See you tomorrow.

Fidget Spinning and OpenStreetMap

Yeah – It’s been a while since I’ve talked about OSM. I’ve been busier than normal – and it’s a good thing. I don’t have a lot of time to “learn” or “play” with much of anything. I’ve helped a bit with the MapLesotho  and I’ve done a few other special projects but I’ve more or less just been hands off with OpenStreetMap.

So with what little bit of free time I’ve had I’ve popped open JOSM and started moving roads to their “close enough to right” location using the BING imagery in East Tennessee. It’s fun work if I get a spare 30 minutes and can make some quick edits. I call it my fidget spinning for a mapping guy.

It’s amazing how bad the roads are…and the major water bodies. There are some major roads which have been corrected. About all the residential roads haven’t. I’m not entirely sure how I’m going to name a lot of the roads. It’s slowly getting fixed. Overall East TN is much improved but it needs more improvement.

In all of this I’ve had a couple of people want to get involved – so I’m slowly dragging them into the OSM world. I’m a bit torn as I’m hesitant to say “join the mailing lists”. As I haven’t and won’t.

The big thing I did notice – Digital Globe Imagery. In JOSM/ID you have access to it. It’s newer in many cases than the bing imagery – although the resolution isn’t as good. Except – I can live with that because I’m finding new building construction. I’m finding new road construction. So my edits will be more current.

Bing

Digital Globe

Of course I need to get back into some form of community involvement. At one time I had a blurb on my business website on OSM and how to get involved locally. At some point we started a Facebook group that has fallen into 3 years of no posts. Soooo – it’s starting back up as a place to post information. Maybe at some point I’ll also find that wiki page I started and add to it.

Updating Roads in Chattanooga TN in OSM

I’ve been attempting to get better with PostGIS. I’ve also been trying to take baby steps back into the world of OpenStreetMap. I’ve also been trying to take baby steps back into my City’s Open Data Initiative.

Why? I’ve been away for long enough – plus – never quit learning.

I’ve only so many hours in a day. To be honest with my attempts at getting back outdoors and doing things that interest me, I’ve more or less stopped a lot of side projects. With the weather currently being disagreeable to most things outdoors I’ve decided to start diving back into a few things I’ve left hanging.

The OpenStreetMap group here in town is a bit hit or miss. We have a high school that maps. We have a few people diving in here and there. There’s no coordination to the madness and I tried, as did a few of us, to coordinate some of it. I finally gave up. So I do a *few* things in OpenStreetMap here locally but not much.

I made a decision to:

  • Map as many Churches as I can (more on that later).
  • Fix the Roads

Fixing roads is a bit of a chore. I hate touching interstates because I don’t understand the relations involved. There’s probably some “OSM Roads” email list but I just don’t want to get that committed. I’m happier working by myself for a bit or in smaller groups. I’ve also never took a look at the road names in OSM – I’ve worried with alignment but never “names”.  About the time I was moving back into Chattanooga mom called, “They changed two road names in the neighborhood!”. I actually think road names had been changing for a while in the county as the County GIS department did their best to clean up confusing road names for 911 purposes. I *think* this data is then fed to the city or maybe the city is changing road names at the same time. Whatever the case – names are changing and it’s already fixed in Google Maps.

Chattanooga has an open data portal.  I’ve been randomly mucking around looking at data and found the Chattanooga city addresses. I also found that they’ve already been donated to the openaddresses project. I downloaded them and loaded them into PostGIS. There are 102761 points. Each has a street address. I grabbed the roads from OSM and the city boundary from the open data portal. Through  series of maneuvers in SQL that would horrify the more well educated – I actually compared the two and built some foreign keys and other things to make joins easier. Through that my favorite color scheme in the world – Red Green. Green is right as far as I can tell and Red isn’t.

What I did find:

  • my join was a bit sloppier than I wanted but overall I can pick out the problems between the Chattanooga’s address data and the OSM data.
  • I spent more time than I should have reading the FGDC Address standard – BUT – it helped (plus SQL examples).
  • There aren’t a lot of errors but enough. In some cases there was a completely new name applied by the city to a street. In many cases the road in misspelled in OSM (which is probably a holdover from the days of TIGER Data).  Some are from the high school. They are great at editing but not so much at cleaning up. I need to work on that.
  • The cities address data extends into the county by about 1000 feet. Which means when I clipped my roads I lost some data I need and will need to fill back in. Thanks to the power of scripts and me being anal retentive – not that big of a deal and I’ll deal with it as it becomes an issue.

Of course the big question is “Is the cities address data correct?”. As a GIS person I say Yes. As a guy that has worked in GIS for 25 years now I’ll say “probably because I’ve talked to no one over addressing”. So I’ve making a list and print it out and over the coarse of the next few months I’m going to inspect many of the things I’m seeing as wrong. There are three on the way to my parent’s house. I’ll probably even hassle the city/county a bit for information. I’d like to get them more involved if at all possible.

Why fix it? Why not. OSM is enough of a force that it should get fixed – BUT – as GIS people our attention is diverted by what the clients use. You never hear someone say “Well I was using OSM to find my way to the mall…..” it’s usually “Google It” or “what does Google/Waze/Something else say”. It’s not that hard to fix problems once you have a list.

Anyway – more on this as the months progress. This will most likely occur “as time permits”.