911

Loading a DEM into PostGIS

November 5, 2021 by rjhale

I haven’t mentioned the TN911 project in a while. Work happens on that when I have time or are motivated – I took a bit of a break to pursue some other tech rabbit holes. I ended up back on it due to some work with pyqgis/addressing questions which led me back to the TN 911 Standard.

My thought on the database is to let it do most of the work. There is some work that occurs in qgis but:

date attributes are assigned at the database
street concatenation happens there
a lot of quality control happens there

In the TN NG911 standard every point needs an elevation. TN also has LIDAR data to provide that elevation. Deriving the elevation with QGIS is easy enough – how hard would it be to store it all in PostGIS?

Not that hard. You can run this if you have postgis installed:

raster2pgsql -C -I -M -F -Y -l 5 -s 2274 -t 100x100 dem.tif tn911.dem > ng911_dem.sql

I leave it up to you to dig through all the commands in the raster2pgsql command.

Once loaded into the database with psql the DEM appears much like you’d expect in QGIS. I connect to the database and add it.

The nice thing is I can now update all the addresses and set up triggers to handle this with new addresses:

update tn911.address_points set z_val = round(st_value(rast, a.geom)/3.28081) from tn911.dem,tn911.address_points as a WHERE ST_Intersects(rast, a.geom)=true;

My biggest issue in doing this was actually the projection. The vector data is in EPSG:2274. The DEM data resides in EPSG:6576. I generally try to retain data projections – but I was having to transform (project on the fly) my calculations. Which made everything slower. So I moved the DEM to EPSG:2274.

Anyway – one more thing done for the standard. Next up some pyqgis and some data quality checks.

August 2005

August 29, 2021 by rjhale

OK – lets drag up some old memories that are a bit cloudy. Lets flash back to 2005.

Back in 2005 I was working for the Feds. I think at that point I was toying with leaving and doing something else: consulting, working for another company, etc. Really I was in wanderlust mode at that point and the least interesting thing I was doing was my job.

I was watching Hurricane Katrina unfold on television. When at work I was watching all the news websites and when I got home it was news until I went to bed.

New Orleans, UNITED STATES: A plea for help appears on the roof of a home flooded in the aftermath of Hurricane Katrina in New Orleans, Louisiana, 04 September 2005. New Orleans began counting its dead 04 September as US troops turned to the gruesome task of harvesting bloated corpses from the hurricane-torn city’s flooded streets and homes. AFP PHOTO/POOL/Robert GALBRAITH (Photo credit should read ROBERT GALBRAITH/AFP/Getty Images)

One question I had when this was happening was “how are they handling this from a GIS perspective?” Web maps were sort of a thing. We had mobile data collectors. You can’t do squat without electricity. Like I said, I was Federal and if I was going out in the field I’m printing maps off and dragging them with me. I didn’t have a smart phone. Were there smart phones? I don’t remember actually. Seems like my first smart phone was 2011 and that one wasn’t that smart and I’m slow to adopt new tech.

Flash forward months, maybe even a year, and two things happened. I was sitting at a conference and a local community college from Mississippi was discussing their Katrina experience: Living in an RV, passing around USB sticks, making maps for rescue, stealing gas for the generator, and taking turns sleeping. It was pretty amazing – amazing enough I remember that talk 16 years later.

Seems like a few months after that we had an ESRI tech on site and we were chatting and he talked about how this new thing called Google Maps had caught them off guard. Google Earth was the one piece of software I remember from Katrina. The news was using it. Search and Rescue was using and it seems like there were some complaints the imagery had some offset – and really the answer was yes there are problems but this wasn’t intended for search and rescue. Imagery was updated while the disaster unfolded.

This morning I’m looking at Google Earth 16 years later.

My group hated Google Earth. Well – not so much hate it just didn’t understand it. We tried to force departments not to use it because the imagery didn’t come from us – and ultimately I think we hated it because it was so easy. We felt we weren’t needed if you could look at data/imagery easily. No one cared about accuracy or data lineage – double click and boom you were looking at something.

I spent this week updating Waverly TN in OpenStreetMap. How does OSM look for New Orleans? It’s pretty good. Waverly was flooded out last week with 17 or 18 inches of rain in a small amount of time. 20 something dead. It was messy.

Anyway – no profound vision or words of wisdom. Get out if you can. Keep an axe in the attic. Lets hope this plays out for the best and plan for the worst. Will it be different or will this be our keynote address at every GIS conference for the next year?

Drop Down Menus for the TN NG911 Server

December 4, 2020 by rjhale

The problem is I have too many options. Options are awesome except when you’re dealing with people inputting data and you only want them to do it one way.

QGIS has had the ability to build drop down lists for a good while and it’s a very flexible way to work. For those of you in ESRI land imagine Domains and Subtypes on Steroids. In earlier versions you had widgets and currently you have Attribute Forms.

If you have a data layer in QGIS (all my layers are coming out of PostgreSQL/PostGIS) just right click the layer and go to Properties. You’ll notice all your fields have a chance to be a “widget”.

Addressing data is probably 90% picking existing attributes. You don’t want to get too fancy in making up new stuff. For instance – Street Type.

I ran out of fingers and toes on counting the number of street types. For the TN guys they all get a ESRI File Based Geodatabase with this information spelled out as Domain tables. How do you do that here? QGIS gives you three ways to make this happen:

Unique Values sums up an existing data set and you can get a pick list of existing data. The good thing is you don’t have to do anything really to make it work and the bad is you’ll get Avenue and Avenew. Misspellings aren’t your friend.
Value Maps let you fill things out or import a CSV. So if you save this information in a spreadsheet you can import it in.
Value relations pulls from an existing table. So if you build a table with this information you can pull that in by first adding the information to your QGIS session and then pulling it into the widget.

In order to cut out down the options, I pre-built all the tables needed into the database. I add them into qgis, group them, and collapse the group because No one needs to see them. Yet. You can’t use a relation widget with the table being added to QGIS.

Back to the Attribute Forms and this is the boring part. I want to select the type table and apply it to the type field using a Value Relation. The important part is the key and the value is the description of that attribute. Both things could be “type” but I chose to have a type and a description:

Go through and start setting up your Forms. What if you have a second person working on this?

Save your style to the database source:

QGIS saves a table in the public schema and everyone with the proper permissions and access is good to go. Plus you don’t have to touch everyone’s QGIS session…well much. Users still have to load the tables to get started but good clean directions should work. Of course this does make me still debate a plugin that does part of this for you.

What? Yeah – it’s that easy. Everyone working with drop down menus and making good clean data.

Topology: QGIS Geometry Checker and other tools

August 25, 2020 by rjhale

So lets dive more into “I need to fix my data”. In this case I have two layers – one is polygon and one is linestring.The polygon layer has overlaps and gaps. The linestring layer has dangles and is not intersected correctly causing more dangles. We looked at the polygon in the last post….and that’s what we will start with….and we have 208 errors which are gaps. The errors are the red lines you see along the borders.

So how am I going to fix this? Well I can always zoom to each error and manually fix it. It’s a slow go but At least I know what’s happening. If I think back to the ESRI world it seems their topology tool automatically fixes errors. Well – automatically with some intervention but it’s faster than manual.

Which brings us to this tool called Geometry Checker. It’s a plugin but it’s part of QGIS Core so you have it when you install QGIS. The goal is to clean up errors and it also has the added bonus to run some checks on your data.

Geometry Checker lets me look at some additional things like duplicate nodes. A Check for gaps. A Check for self intersections. A Check for smaller sliver polygons. It will also check to see that the layer you are working with follows the boundaries of a second layer. So do I need to check that this layer fits with the county? I probably should. So I’m going to check for polygons smaller than 1 square mile. Holes. Overlaps smaller than 1 square mile. Gaps bigger than 10 square miles. Which at this point there is no method to my madness – more just testing things to see what comes up.

You have 2 options at the end of this. One is to edit the layers “in place” which means the originals get changed. The other option is geopackage. As all my data is currently in postgis and I hate the idea of “edit in place” – I’m going to a geopackage and fix my data and then shove it back into the database.

When geometry checker is run a new geopackage is created. I am prompted with a list of errors I can work my way through and click a button to automatically fix the problem. I also have a setting for “error resolution settings” which will let me set some defaults like “merge the sliver polygons into the largest shared edge”. Overall it’s pretty awesome. It appears I can even fix more than one error at a time. I didn’t have much luck doing that BUT – I’m happy with one at a time in most cases.

What about the roads?

I have a road layer with roads that aren’t properly intersected. The Topology Checker works great and I can find them all…..so the Geometry Checker……

If you look at the Geometry checker I can check line strings for length, dangles, self intersection and a few other things. What I don’t get is a way to fix dangles like I have pictured above. Which sorta brings me to “I need a way to snap and break lines at intersections easily”. I can manually do it – but if I have a bad road layer (water lines, linear anything) that can be a pain to walk through.

but……there’s always a but…..

There are 2 tools in 3.14 (maybe earlier but that’s what I’m using) called Snap and Split Lines with Lines. So I’m going to run a Snap Geometries to Layer and then a Split Lines with Lines against the snapped layer and my Dangle issue almost goes away. Roads are always going to have dangling nodes – lets just not have them in the middle of the road. No i go back to Geometry Checker and look for duplicate nodes and other things.

So I have some tools to fix my data. Here would be my pros and cons:

Pros:

You can fix your data. Be it a shapefile, geopackage, postgis table, etc – the tools work. It’s a lot of tools also – if you’re coming from the ArcGIS world where you have to have a licensing level to run topology – you don’t have that problem anymore. You don’t have a table/data layer called topology – but you have a lot of tools to clean up your data and make it topologically correct.
New tools are coming constantly. I had a game plan for writing this article and discovered Snap and Break Lines with Lines. So if it seems like I shoved that in at the last minute (I did).

Cons:

You’ll notice back in the article you’ll notice I didn’t really give numbers on problems found. The Topology Checker would notice 208 gaps in my data and Geometry Checker would find 384. They are two different tools that (to me) serve two different purposes. So one might report more. One might report less. I would use both as a running check on my data and I might use topology checker throughout the day and geometry checker at the end of the day.
It’s not really a con – but when you’re fixing your data you may pull that data out of the original table (as I did) to fix it and then push your fixed data back into the database. Maybe I make a new table. I’m not sure on workflow yet so maybe this comes toward the end of the project.

One final word on Geopackage…..

One thing I have sort of taken for granted is geopackage. I say geopackage a lot. When I teach a class I yell geopackage. It wasn’t until I wrote this that I noticed everything is dumping back to geopackage. Some GRASS tools sometimes went back to shapefile in previous versions. It seems like there were a few tools that would in QGIS that wanted to go back to shape – it’s all Geopackage now. It’s nice.

—BUT—

Maybe I don’t want this much work fixing my data. I have geometry checker….and I have some tools from the processing toolbox……what if I wanted one tool to run my topology checks BECAUSE I’m really wanting to network these roads together…..what could that tool be………..those tools be…….

QGIS: Revisiting Topology

August 20, 2020 by rjhale

When you’re writing a blog two things tend to happen:

You rehash old topics that you covered 4 or 5 years ago. Usually that’s fine because something has changed enough you can cover it again and not sound repetitive.
You get to the point where “I really don’t want to talk about anything”.

I’ve been stuck in between the 2 for the last bit. After a small break from the TN 911 address project that is getting some more attention. The biggest bit of activity is on “Well how do I make sure the data isn’t screwed up….”

We’ve had to fix some data as of late – nothing terrible but it’s brought up a lot of questions on my end. One question has been “How do I make sure the data stays in good shape vs mistakes slowly propagating out”. Which then led to questions on topology and workflow. Some problems are topological and some are going to be attribution….some cover both…..

So I decided to rehash an old subject of topology that I’ve touched on a few times. I’m starting back at square one because maybe some of you know and maybe some don’t – but I’m starting on a workflow for the TN 911 Server so it gives me a chance to talk a little. Did you know you can check the topology of your data in QGIS?

Back when I first wrote about this….7 years ago…..this was a new feature for QGIS. Kind of a big deal for me because I come from a data background and I always like tools to help fix data. The topology checker isn’t there to fix bad data – it is there to point out bad data. It’s now a core plugin so you have to activate it but once you do you’ll get one added icon and one new panel to your QGIS Display.

For fun I’m going to build a rule that says the ESN layer (the black polygons) can’t have gaps. So when I run the tool it will highlight the problems:

What happened? More than likely a boundary was moved and the adjoining boundary didn’t move. Gaps were created. In some cases overlaps. Sometimes duplicate polygons.

It will walk me to each error and give me the chance to fix it. What? You mean it won’t fix it for me? Yes and No but that’s coming next. So you can sort of quickly get a visual on what is and isn’t wrong with your data. You can check lines and points also. You can also check to see if points are covered by polygons or points are at the end of lines. If you look at the help you’ll see an entire list of things it can check.

So welcome to “one way to check your data”. Next up – One of several ways to fix your data which I hope eventually leads me into this PGRouting bit of excitement I’ve gotten myself into. Which if you start looking at the next few blog posts you’re going to say “Well it looks like you’re trying to do routing the in the database but you need to clean the data up first”.