North River Geographic Systems Inc

Spatial Problem Solving

911

Topology: QGIS Geometry Checker and other tools

· ·

So lets dive more into “I need to fix my data”. In this case I have two layers – one is polygon and one is linestring.The polygon layer has overlaps and gaps. The linestring layer has dangles and is not intersected correctly causing more dangles. We looked at the polygon in the last post….and that’s what we will start with….and we have 208 errors which are gaps. The errors are the red lines you see along the borders.

So how am I going to fix this? Well I can always zoom to each error and manually fix it. It’s a slow go but At least I know what’s happening. If I think back to the ESRI world it seems their topology tool automatically fixes errors. Well – automatically with some intervention but it’s faster than manual.

Which brings us to this tool called Geometry Checker. It’s a plugin but it’s part of QGIS Core so you have it when you install QGIS. The goal is to clean up errors and it also has the added bonus to run some checks on your data.

Geometry Checker lets me look at some additional things like duplicate nodes. A Check for gaps. A Check for self intersections. A Check for smaller sliver polygons. It will also check to see that the layer you are working with follows the boundaries of a second layer. So do I need to check that this layer fits with the county? I probably should. So I’m going to check for polygons smaller than 1 square mile. Holes. Overlaps smaller than 1 square mile. Gaps bigger than 10 square miles. Which at this point there is no method to my madness – more just testing things to see what comes up.

You have 2 options at the end of this. One is to edit the layers “in place” which means the originals get changed. The other option is geopackage. As all my data is currently in postgis and I hate the idea of “edit in place” – I’m going to a geopackage and fix my data and then shove it back into the database.

When geometry checker is run a new geopackage is created. I am prompted with a list of errors I can work my way through and click a button to automatically fix the problem. I also have a setting for “error resolution settings” which will let me set some defaults like “merge the sliver polygons into the largest shared edge”. Overall it’s pretty awesome. It appears I can even fix more than one error at a time. I didn’t have much luck doing that BUT – I’m happy with one at a time in most cases.

What about the roads?

I have a road layer with roads that aren’t properly intersected. The Topology Checker works great and I can find them all…..so the Geometry Checker……

If you look at the Geometry checker I can check line strings for length, dangles, self intersection and a few other things. What I don’t get is a way to fix dangles like I have pictured above. Which sorta brings me to “I need a way to snap and break lines at intersections easily”. I can manually do it – but if I have a bad road layer (water lines, linear anything) that can be a pain to walk through.

but……there’s always a but…..

There are 2 tools in 3.14 (maybe earlier but that’s what I’m using) called Snap and Split Lines with Lines. So I’m going to run a Snap Geometries to Layer and then a Split Lines with Lines against the snapped layer and my Dangle issue almost goes away. Roads are always going to have dangling nodes – lets just not have them in the middle of the road. No i go back to Geometry Checker and look for duplicate nodes and other things.

So I have some tools to fix my data. Here would be my pros and cons:

Pros:

  • You can fix your data. Be it a shapefile, geopackage, postgis table, etc – the tools work. It’s a lot of tools also – if you’re coming from the ArcGIS world where you have to have a licensing level to run topology – you don’t have that problem anymore. You don’t have a table/data layer called topology – but you have a lot of tools to clean up your data and make it topologically correct.
  • New tools are coming constantly. I had a game plan for writing this article and discovered Snap and Break Lines with Lines. So if it seems like I shoved that in at the last minute (I did).

Cons:

  • You’ll notice back in the article you’ll notice I didn’t really give numbers on problems found. The Topology Checker would notice 208 gaps in my data and Geometry Checker would find 384. They are two different tools that (to me) serve two different purposes. So one might report more. One might report less. I would use both as a running check on my data and I might use topology checker throughout the day and geometry checker at the end of the day.
  • It’s not really a con – but when you’re fixing your data you may pull that data out of the original table (as I did) to fix it and then push your fixed data back into the database. Maybe I make a new table. I’m not sure on workflow yet so maybe this comes toward the end of the project.

One final word on Geopackage…..

One thing I have sort of taken for granted is geopackage. I say geopackage a lot. When I teach a class I yell geopackage. It wasn’t until I wrote this that I noticed everything is dumping back to geopackage. Some GRASS tools sometimes went back to shapefile in previous versions. It seems like there were a few tools that would in QGIS that wanted to go back to shape – it’s all Geopackage now. It’s nice.

—BUT—

Maybe I don’t want this much work fixing my data. I have geometry checker….and I have some tools from the processing toolbox……what if I wanted one tool to run my topology checks BECAUSE I’m really wanting to network these roads together…..what could that tool be………..those tools be…….

 

 

 

 

 

 

 

 

 

QGIS: Revisiting Topology

· ·

When you’re writing a blog two things tend to happen:

  • You rehash old topics that you covered 4 or 5 years ago. Usually that’s fine because something has changed enough you can cover it again and not sound repetitive.
  • You get to the point where “I really don’t want to talk about anything”.

I’ve been stuck in between the 2 for the last bit. After a small break from the TN 911 address project that is getting some more attention. The biggest bit of activity is on “Well how do I make sure the data isn’t screwed up….”

We’ve had to fix some data as of late – nothing terrible but it’s brought up a lot of questions on my end. One question has been “How do I make sure the data stays in good shape vs mistakes slowly propagating out”. Which then led to questions on topology and workflow. Some problems are topological and some are going to be attribution….some cover both…..

So I decided to rehash an old subject of topology that I’ve touched on a few times. I’m starting back at square one because maybe some of you know and maybe some don’t – but I’m starting on a workflow for the TN 911 Server so it gives me a chance to talk a little. Did you know you can check the topology of your data in QGIS?

Back when I first wrote about this….7 years ago…..this was a new feature for QGIS. Kind of a big deal for me because I come from a data background and I always like tools to help fix data. The topology checker isn’t there to fix bad data – it is there to point out bad data. It’s now a core plugin so you have to activate it but once you do you’ll get one added icon and one new panel to your QGIS Display.

For fun I’m going to build a rule that says the ESN layer (the black polygons) can’t have gaps. So when I run the tool it will highlight the problems:

What happened? More than likely a boundary was moved and the adjoining boundary didn’t move. Gaps were created. In some cases overlaps. Sometimes duplicate polygons.

It will walk me to each error and give me the chance to fix it. What? You mean it won’t fix it for me? Yes and No but that’s coming next. So you can sort of quickly get a visual on what is and isn’t wrong with your data. You can check lines and points also. You can also check to see if points are covered by polygons or points are at the end of lines. If you look at the help you’ll see an entire list of things it can check.

So welcome to “one way to check your data”. Next up – One of several ways to fix your data which I hope eventually leads me into this PGRouting  bit of excitement I’ve gotten myself into. Which if you start looking at the next few blog posts you’re going to say “Well it looks like you’re trying to do routing the in the database but you need to clean the data up first”.

 

Fishbones for the TN NG911 Address Server

· ·

It’s been a bit since I’ve talked about the TN address server. I’ve been slowly adding more functionality to it and some of that functionality has been more difficult than others.

If you’ve played in the address space any length of time you’ve heard of fishbones – and that is simply drawing a line from the address point to it’s “spot on the street”. Granted if you’ve ever done an addressing project from scratch (I had the joy to do that a few years ago in the Caribbean) it makes a lot of sense. You can quickly see problems where lines cross over other lines or where addresses have no lines at all.  Please read up here if you are so inclined.

There are probably multiple ways to do this  but I’ve gotten fishboning (if that’s a word) running in the TN NG911 Address Server. In PostGIS with less than 25 lines of SQL (and I’m quite proud of myself – 5 years ago I sucked at SQL. I still suck at it – but way less).  While it’s not quite ready for prime time it’s close – I’m just trying to figure out what I can pull out of this that makes the County’s job easier in the addressing realm. I’d love to move it to python at some point – and it’s probably doable (well I know it is – it’s just doing it).

Three things happen:

  1. I join the address point to the closest road using ST_LineLocatePoint
  2. Does the closest point fit the address range on the street (double bonus I also check left and right).
  3. Finish it up with ST_Makeline to join the spot closest to the street to the address point.

Which quickly shows something is wrong here:

Bonus for this one as is it’s not incredibly obvious what the problem is at first glance….UNTIL you see you have an addressing issue where you have 2, 3, and 4 on the same side of the street.

Hurrah for misspelled street names by leaving a W off

Anyway – there will be more announcements coming like this “running in the cloud” and other things. I’d like to get the fishbones running once a night or realistically run the one for the new address. Why not both probably. Anyway – pretty nice for an ArcView/FGDB Replacement.

Chattanooga QGIS Conference Update

· ·

So things are moving. Things are always a bit terrifying as this starts pulling together. I’ve been tossing updates everywhere – so here’s an update for the Blog.

Official Stuff

Website: https://wiki.osgeo.org/wiki/20191000-QGIS-US

Date: October 25th 2019

So the first big thing: A Social. I’m headed down to the Tap House this week to make sure everything is arranged for a social On October 24th.

Talks

Talks are rolling in for Friday. There is still room to submit a paper if you wish. The big one is I’m hopefully (and I’m at the mercy of the internet) is I’ve got Nyall Dawson on Video from Australia speaking first.

Here are some of the abstracts that have been submitted:

  • QGIS: It’s all about Form(s) – QGIS comes with the ability to create forms to help you input data or capture data. You can build simple forms to help users avoid misspellings and help speed up data capture. You can also build forms where data entry is dependent on previous data entered as well as forms where data entry depends on other GIS data. Building Forms in QGIS is easy, powerful, and easily done if you have no programming experience. I’m going to attempt to talk you through the easiest to make forms plus one difficult form to show you how easy this can be (or hard depending on how the demos work).
  • Using QGIS with TN’s 911 Database Standard – Henry County 911 recently moved away form their COTS system and started using QGIS, PostGIS, and Fulcrum to collect and maintain the county’s 911 data. This talk covers some of the in’s and out’s of switching software, tools, and workflows.
  • EcoValuator:  EcoValuator is built on python and provides a simple means of estimating the dollar value of a study area, based on the land cover types in that study area, and on your ecosystem service of choice. This presentation will cover background information about the project, what are ecosystem services, how the EcoValuator works, challenges of using the QGIS python API, and demonstration of the plugin in action.
  • Building a cloud-based mapping and analysis solution based on Openlayers and CesiumJS. 18 month ago our team felt the need for having a next Gen cloud mapping solution. We did not want to leave it all to ArcGIS Online, so we decided to build our own. It is designed to make maps, analyze data, add custom database solutions, and to share maps and data with fellow users.Take a look behind the scene of the journey of putting it all together and learn how far we have come and what are our ideas in terms of connecting Open Source solutions like QGIS and GeoServer.

…..and more are coming.

This is the third meeting I’ve put on since 2017. All meetings are centered around FOSS4G with this being the  first QGIS focused one. I’ve had emails and conversations from all over the US on this meeting. One thing that keeps coming up is people want more open source GIS meetings. The tough part is it has to be a community run affair or someone (yours truly) jumps up and puts this together. I really need to start earlier and shoot for bigger – BUT – there’s only so much I can do. We have the FOSS4G NA meetings that are well run and much bigger – but this seems to scratch an itch. How does this get bigger? I have no idea and I’m open to suggestions.

Anyway – the clock is ticking. The more the merrier at this thing so please make plans to attend.

 

 

TN 911 Server Updates

· ·

The TN NG911 server got an upgrade this month with some quality assurance checks. After working with Henry County for the last bit we’re into more “what if” scenarios.

  • What if we ran checks for misspelled addresses or improperly labeled addresses?
  • What if we started filling out the fields we weren’t filling with out with our ArcGIS/Geodatabase setup?

Working with Chad Howard (911 GIS Guru for Henry County TN) we’ve started playing with data problems. One thing I learned is that the amount of quality assurance checks at the county level are lacking – not because anyone didn’t want to do it it’s just hard working with a basic arcgis license to do anything significant. That’s where postgis comes into play. I’ve got a total of three checks on the addresses planned with two implemented as of right now.

What if you accidentally add two address points with the same geometry? A red circle appears in one of the layers in your database (and in your QGIS session) alerting you to the mistake. What if you accidentally misspell an address street name and that doesn’t match a street? A blue circle appears. If you look at the example it looks like someone can’t spell. Actually the old setup was adding spaces to names. So instead of “Main” you had “Main “. That’s an easy enough fix we can do in a bulk edit.

I expect these next set of checks to slowly become irrelevant over the next bit as the client takes full advantage of QGIS’s forms for data entry. Once again we’re hitting spaces as opposed to nulls in the data in 80% of the cases. We should only have 4 pre-directionals – well at most 8 –  but for this county they only have 4.

We can bulk edit the spaces away in the database. It’s nice to have this table sitting in the qaqc schema just to check occasionally but I do expect that to drift off in significance the more we use Fulcrum and QGIS’s Forms for data input. These checks check everything from street types, names, to directionals. Like I said – over time these start drifting off in significance but it’s nice for now.

Anyway – more quality checks are planned but we’re over he first hurdle to making this even more useful and easy.

Contact