Topology

Topology: QGIS Geometry Checker and other tools

So lets dive more into “I need to fix my data”. In this case I have two layers – one is polygon and one is linestring.The polygon layer has overlaps and gaps. The linestring layer has dangles and is not intersected correctly causing more dangles. We looked at the polygon in the last post….and that’s what we will start with….and we have 208 errors which are gaps. The errors are the red lines you see along the borders.

So how am I going to fix this? Well I can always zoom to each error and manually fix it. It’s a slow go but At least I know what’s happening. If I think back to the ESRI world it seems their topology tool automatically fixes errors. Well – automatically with some intervention but it’s faster than manual.

Which brings us to this tool called Geometry Checker. It’s a plugin but it’s part of QGIS Core so you have it when you install QGIS. The goal is to clean up errors and it also has the added bonus to run some checks on your data.

Geometry Checker lets me look at some additional things like duplicate nodes. A Check for gaps. A Check for self intersections. A Check for smaller sliver polygons. It will also check to see that the layer you are working with follows the boundaries of a second layer. So do I need to check that this layer fits with the county? I probably should. So I’m going to check for polygons smaller than 1 square mile. Holes. Overlaps smaller than 1 square mile. Gaps bigger than 10 square miles. Which at this point there is no method to my madness – more just testing things to see what comes up.

You have 2 options at the end of this. One is to edit the layers “in place” which means the originals get changed. The other option is geopackage. As all my data is currently in postgis and I hate the idea of “edit in place” – I’m going to a geopackage and fix my data and then shove it back into the database.

When geometry checker is run a new geopackage is created. I am prompted with a list of errors I can work my way through and click a button to automatically fix the problem. I also have a setting for “error resolution settings” which will let me set some defaults like “merge the sliver polygons into the largest shared edge”. Overall it’s pretty awesome. It appears I can even fix more than one error at a time. I didn’t have much luck doing that BUT – I’m happy with one at a time in most cases.

What about the roads?

I have a road layer with roads that aren’t properly intersected. The Topology Checker works great and I can find them all…..so the Geometry Checker……

If you look at the Geometry checker I can check line strings for length, dangles, self intersection and a few other things. What I don’t get is a way to fix dangles like I have pictured above. Which sorta brings me to “I need a way to snap and break lines at intersections easily”. I can manually do it – but if I have a bad road layer (water lines, linear anything) that can be a pain to walk through.

but……there’s always a but…..

There are 2 tools in 3.14 (maybe earlier but that’s what I’m using) called Snap and Split Lines with Lines. So I’m going to run a Snap Geometries to Layer and then a Split Lines with Lines against the snapped layer and my Dangle issue almost goes away. Roads are always going to have dangling nodes – lets just not have them in the middle of the road. No i go back to Geometry Checker and look for duplicate nodes and other things.

So I have some tools to fix my data. Here would be my pros and cons:

Pros:

You can fix your data. Be it a shapefile, geopackage, postgis table, etc – the tools work. It’s a lot of tools also – if you’re coming from the ArcGIS world where you have to have a licensing level to run topology – you don’t have that problem anymore. You don’t have a table/data layer called topology – but you have a lot of tools to clean up your data and make it topologically correct.
New tools are coming constantly. I had a game plan for writing this article and discovered Snap and Break Lines with Lines. So if it seems like I shoved that in at the last minute (I did).

Cons:

You’ll notice back in the article you’ll notice I didn’t really give numbers on problems found. The Topology Checker would notice 208 gaps in my data and Geometry Checker would find 384. They are two different tools that (to me) serve two different purposes. So one might report more. One might report less. I would use both as a running check on my data and I might use topology checker throughout the day and geometry checker at the end of the day.
It’s not really a con – but when you’re fixing your data you may pull that data out of the original table (as I did) to fix it and then push your fixed data back into the database. Maybe I make a new table. I’m not sure on workflow yet so maybe this comes toward the end of the project.

One final word on Geopackage…..

One thing I have sort of taken for granted is geopackage. I say geopackage a lot. When I teach a class I yell geopackage. It wasn’t until I wrote this that I noticed everything is dumping back to geopackage. Some GRASS tools sometimes went back to shapefile in previous versions. It seems like there were a few tools that would in QGIS that wanted to go back to shape – it’s all Geopackage now. It’s nice.

—BUT—

Maybe I don’t want this much work fixing my data. I have geometry checker….and I have some tools from the processing toolbox……what if I wanted one tool to run my topology checks BECAUSE I’m really wanting to network these roads together…..what could that tool be………..those tools be…….

QGIS: Revisiting Topology

When you’re writing a blog two things tend to happen:

You rehash old topics that you covered 4 or 5 years ago. Usually that’s fine because something has changed enough you can cover it again and not sound repetitive.
You get to the point where “I really don’t want to talk about anything”.

I’ve been stuck in between the 2 for the last bit. After a small break from the TN 911 address project that is getting some more attention. The biggest bit of activity is on “Well how do I make sure the data isn’t screwed up….”

We’ve had to fix some data as of late – nothing terrible but it’s brought up a lot of questions on my end. One question has been “How do I make sure the data stays in good shape vs mistakes slowly propagating out”. Which then led to questions on topology and workflow. Some problems are topological and some are going to be attribution….some cover both…..

So I decided to rehash an old subject of topology that I’ve touched on a few times. I’m starting back at square one because maybe some of you know and maybe some don’t – but I’m starting on a workflow for the TN 911 Server so it gives me a chance to talk a little. Did you know you can check the topology of your data in QGIS?

Back when I first wrote about this….7 years ago…..this was a new feature for QGIS. Kind of a big deal for me because I come from a data background and I always like tools to help fix data. The topology checker isn’t there to fix bad data – it is there to point out bad data. It’s now a core plugin so you have to activate it but once you do you’ll get one added icon and one new panel to your QGIS Display.

For fun I’m going to build a rule that says the ESN layer (the black polygons) can’t have gaps. So when I run the tool it will highlight the problems:

What happened? More than likely a boundary was moved and the adjoining boundary didn’t move. Gaps were created. In some cases overlaps. Sometimes duplicate polygons.

It will walk me to each error and give me the chance to fix it. What? You mean it won’t fix it for me? Yes and No but that’s coming next. So you can sort of quickly get a visual on what is and isn’t wrong with your data. You can check lines and points also. You can also check to see if points are covered by polygons or points are at the end of lines. If you look at the help you’ll see an entire list of things it can check.

So welcome to “one way to check your data”. Next up – One of several ways to fix your data which I hope eventually leads me into this PGRoutingÂ bit of excitement I’ve gotten myself into. Which if you start looking at the next few blog posts you’re going to say “Well it looks like you’re trying to do routing the in the database but you need to clean the data up first”.

QGIS 2.12: Check Geometries Plugin

QGIS 2.12 is out and I’ve been a bit quiet as I’ve been exploring it as well as working with it.

As my proficiency with these open source mapping tools grow, I’m starting to learn some QA/QC (quality assurance/quality control) methods for checking data. If data resides in postgis I’m slowly building out tools for attribution and geometry and I’m doing the same with GRASS and QGIS.

Somewhere it seems around 2.2 (I may be wrong) a topology plugin appeared. For those of you familiar with the way ArcGIS Topology works you can go in and flag and fix problems with your data. There’s a large PDF suitable for framing on the ArcGIS Site to tell you everything it can do. I like the Topology checks in ArcGIS – you find errors and you fix them. The only problem I had with the tool it was a license level tool – those of you who bought arcview couldn’t use it. The QGIS Topology tool came along and while it wasn’t as robust as ArcGIS – you had a tool for your data to at least flag (there is no automatic fix built in yet) errors.

So this very innocent looking data layer has at least two problems (four total but I only checked for invalid topology and overlapping geometry)

Running QGIS Topology Checks:

There’s another tool with 2.12 that has been introduced – the Check Geometries Plugin. Once again – if you are familiar with the ArcGIS topology tools this won’t feel foreign at all.

So this tool (you need to activate it under the Plugins Menu) gives you a chance to check your data and FIX IT AUTOMATICALLY. It can check for sliver polygons, data that self intersects, data that may not match a minimal mapping unit, and duplicated data (and more). It checks for errors in a different manner than the topology tool. If I run this against my dataset I can find at least three problems. I attempted to set up the exact same checks I used with the Topology plugin plus one more check for holes or gaps:

Once you find a problem you can move around automatically and fix either manually or automatically the problems. I’ve been playing with it and I haven’t decided on a methodology yet for using the two in a complimentary fashion – there may not be one. Between these two tools you almost have the ArcGIS Topology tool in QGIS though.Â I would strongly suggest playing around with it for a bit before you throw it into a production environment.

The bad – and I hate to say that this is even bad – It’s just different. These two tools really work independently of each other. So you’re going to be tempted (speaking to the former ArcGIS users) to use them both like they are a connected piece of software (notice I said above I’m attempting to work out a methodology to use them together because it’s too tempting). If I use both tools to look for all the errors in this layer the topology tool wins with 4 total errors while the geometry tool finds 3 errors (I built the dataset with 4 errors). The geometry tool does check for minimum mapping sizes and that’s something the topology tool doesn’t do. With another dataset it incorrectly flagged some geometry as errors that weren’t. Don’t worry – Bug report was submitted. It may not even be a bug but it doesn’t hurt.

Overall – check geometries is a welcome addition. You just need to be aware it’s different from the Topology tool. With the Check Geometries tool being “brand new” (so to speak) I would expect changes as feed back starts coming back into the QGIS project.

Back when I was 17 I bought a jeep. At 17 I thought it was the sexiest thing ever – most everyone else had determined I was driving my future coffin. I ended up at my Uncle’s house and I had been working on every part of the car that seemed to have problems. With a stethoscope, a hammer, a few electrical checking gadgets he found about three more problems I had completely missed. I was bummed. “I wished you weren’t finding problems with my jeep”. To which he pointed out “I’m not making more problems – just finding the ones that exist.” You now have more tools in you Desktop GIS to find errors and make clean data.

A Tale of Two Topologies

It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of FOSS4G, it was the season of Commercial Software, it was the spring of hope, it was the winter of despair, we had everything before us for the right price, we had nothing before us because we didn’t RTFM.

My apologies to Charles Dickens. My compliments to the guillotine.

To say that this dataset had some miles on it isÂ an understatement. Â It had been a shapefile. It had become a geodatabase. It went back to shapefiles. It became 4, 16, 80+ shapefiles, and has now been moved to PostGIS and I’m down to 20+ tables plus a lot of Foreign Keys. Why? Well – I upgraded a client from ArcView (I know it’s not called that) to QGIS/PostGIS. Why is it an upgrade to Free Software? Because we have more functionality and much much better data. I’ve talked about these guys before – and I have some posts I’ve left hanging but now we talk about topology.

This data in all of it’s changing and moving had developed some problems. I will accept full blame for the problems developing because I’m better than this data looked. Last year I made the choice to abandon shapefiles and learn postgis so the last jump was made and it has made everyone’s life much better. Don’t tell my client but I tell them it takes a day to prep the data for delivery to their client – it really only takes an hour now (it used to take 24 hours) – and in those extra hours I clean up the data. No one has noticed, but me, of the problems with gaps and overlaps and zero area polygons. I like clean data. I guess I am a dying breed that still thinks the data is the most important thing and not having all your data sitting exposed in all of it’s terribleness out in some online story map thing.

For kicks last week I started on the topological errors and the first thing I did was read up on PostGIS topology. I started. I failed miserably due to some error that made no sense and I gave up. It’s on my list. I will “get it” eventually – and I was close this time. Really Close.

So for my next trick I pulled up QGIS and looked:

For those of you that don’t know – shapefiles aren’t a topological dataset. It’s easy to get things in a mess if you aren’t careful. QGIS can check shapefiles for topological rules (Arcview can’t last time I checked – license issue). In this case I was checking for gaps. What I see is a lot of red that areÂ problems from GPS data and problems that appeared when I pulled this dataset apart two years ago and recombined it to fix another problem.

So what do I do to fix it?

Well – for fun I pulled it into ArcGIS. I dumped my data out back into a shapefile (shudder) and imported it into a file based geodatabase. I built topology (I’ve got ArcINFO – and I know it’s not called that anymore) and the topology build removed all the errors in about 15 to 20 seconds. I know – I’m mentioning ArcGIS as a solution. Hey – it worked.

BUT – I’ve switched my client to QGIS and PostGIS – do I really want to fall back to ESRI as a fix? No – I’m going to make this work with the tools my client has available.

So I started reading up on GRASS. You get GRASS when you install QGIS. Right now I think it’s GRASS 7 (or at least on my machine it’s Grass 7) and GRASS has this wonderful 30 year run. Yes – the software has been around for 30+ years. GRASS is one of those things that if you ask someone you first get 1. A Drug reference and 2. I used it in College once with another Drug Reference. Of course it’s 2015 and with the number of colleges flailing with GIS you’re not even getting a 2 anymore. Just glassy eyed stares of “Well I made a Story map”.

Grass 7 is really – for lack of a better word – pretty now…or nicer. I’m not sure how to describe it. Grass 6.x always to me was a bit confusing – GRASS 7 isn’t. Maybe it’s me. Maybe not. The user interface is more intuitive – the Help button does just that. It’s not as “mysterious” as it has been in the past. I used it and with a little effort (I need to RTFM more) I was working. It’s powerful. To get QGIS/GRASS/GDAL and all these things in one QGIS install – that’s huge. There’s no excuse for you as a GIS person to not have these packages installed on your computer.

GRASS makes me nostalgic for the old days of Arc Workstation. GRASS datasets are topological. Problem being if you’re using the GUI for import there is no place to set snapping (that I could find). So I imported data and I kept the problems. After reading the manual I did this little command:Â v.in.ogr /export/data/topo/stands.shp out=stands_snap snap=0.001 and re-imported my data. IÂ hadÂ Clean data in about 15 seconds (probably less). What happened is GRASS pulled it in and snapped the data as it was being converted (same as ArcGIS). If you watch the command line you get a lot of information as to what it’s doing.

I checked my ArcGIS software. I checked QGIS. After all this was done I had 10 problems using the topology tools in both softwares with this new clean dataset. 10 problems that took me 10 minutes to fix. Please note I had the exact same problems in ArcGIS and QGIS – no difference. Â I took the GRASS snapped dataÂ and put it back into PostGIS and I’m done. Clean data now. Happy Clean Data as I invoke my inner Bob Ross.

So what did we learn from all of this:

1. If you aren’t worried about topology as a GIS Person – turn in your badge at the door. If you aren’t worried about your data you’re a terrible person.

2. If you believe you can’t create production professional data using FOSS4G. Turn in your mouse and your half completed GISP application. IÂ took QGIS/PostGIS/GRASS and created a topologically clean production dataset in a little under an hour. My next dataset will probably take about 15 minutes.

3. I failed slightly – I could connect GRASS to PostGIS and I didn’t. I would have avoided the whole “going back to shapefiles” thing again. That won’t happen next time. It’s going to be connected from here on out. It will be pulled into Grass and pushed back into PostGIS.

4. My hope is that I get my topology problems figured out in PostGIS. I’m close.

Anyway – excuse my slight irreverence in this post as I poked a bit. I walked out of a professional meeting the other day where I got the “you only use QGIS because you’re too cheap to buy ArcGIS”. Well……no. So expect the next few posts to dive into a lot of technical as to why this is a good way to work.Â Plus you get commercial support from a lot of companies while doing it. It’s a good choice. Granted – there are a lot of tools out in GIS land – but picking these tools doesn’t indicate anything about the user or your organization.

QGIS Part II: Editing and Data

It’s truly an odd thing announcing this because I’ve probably said it like 3 or 4 times up here…well – hinted it was coming. Now it’s here. I announce QGIS Part II: Editing and Data.

Now for some explanation. This class picks up where part I leaves off….or explores the software even further.

Editing
Topology
Widgets
Symbology
Data Conversion
Databases (spatialite).

I spent way more time than planned on Spatialite. I also spent more time comparing and contrasting with ArcGIS. It’s important that these classes take the stance of best tool for the job. Which is the back bone of the spatial connect process. Process over software. Make your software less of a center point. Worry about your data.

Price – $325 and it runs one day. I say that and if I can get Intro I and II backed up against each other I’m probably going to do two days for a combined price of $575. It’s not much of a discount but it’s still pretty affordable and I give you a lot of information for the price. Plus you have the option of calling me up whenever.

Next thing – Online classes. I’ve been asked. I’ve had offers made. I’m still going to go no (for now). It’s important (to me) you get the best use of your time and I still argue an online class isn’t going to do that. Yes it opens up the training to more people and I might change my mind at some point. For now I’m still going with “in person” for training. I guess that’s still the old ESRI Authorized trainer coming out in me. Yes I was one of those also.

Scheduling – the last two classes have taken a more organic approach. My best class was in Kingsport TN and was brought about by the SE Tennessee Development District and the local NETGIS users group. As the rest of the year moves on I’m probably going to try to take that approach – As I continue folding this more into the spatial connect process I will push to do training for groups (clients) with a vested interest in expanding their GIS setup…and the occasional conferences. So contact me if you are interested in bringing the class to your organization – the only thing that might change the price is if I have to drive longer than 3 or 4 hours (or fly).