North River Geographic Systems Inc

Spatial Problem Solving

You are here: Home / Blog

Blog

Drop Down Menus for the TN NG911 Server

· · Leave a Comment

The problem is I have too many options. Options are awesome except when you’re dealing with people inputting data and you only want them to do it one way.

QGIS has had the ability to build drop down lists for a good while and it’s a very flexible way to work. For those of you in ESRI land imagine Domains and Subtypes on Steroids. In earlier versions you had widgets and currently you have Attribute Forms.

If you have a data layer in QGIS (all my layers are coming out of PostgreSQL/PostGIS) just right click the layer and go to Properties. You’ll notice all your fields have a chance to be a “widget”.

QGIS Layer Properties

Addressing data is probably 90% picking existing attributes. You don’t want to get too fancy in making up new stuff. For instance – Street Type.

USPS Sample Street Types

I ran out of fingers and toes on counting the number of street types. For the TN guys they all get a ESRI File Based Geodatabase with this information spelled out as Domain tables. How do you do that here? QGIS gives you three ways to make this happen:

  • Unique Values sums up an existing data set and you can get a pick list of existing data. The good thing is you don’t have to do anything really to make it work and the bad is you’ll get Avenue and Avenew. Misspellings aren’t your friend.
  • Value Maps let you fill things out or import a CSV. So if you save this information in a spreadsheet you can import it in.
  • Value relations pulls from an existing table. So if you build a table with this information you can pull that in by first adding the information to your QGIS session and then pulling it into the widget.

In order to cut out down the options, I pre-built all the tables needed into the database. I add them into qgis, group them, and collapse the group because No one needs to see them. Yet. You can’t use a relation widget with the table being added to QGIS.

QGIS grouped items

Back to the Attribute Forms and this is the boring part. I want to select the type table and apply it to the type field using a Value Relation. The important part is the key and the value is the description of that attribute. Both things could be “type” but I chose to have a type and a description:

QGIS setting up Type

Go through and start setting up your Forms. What if you have a second person working on this?

Save your style to the database source:

QGIS saves a table in the public schema and everyone with the proper permissions and access is good to go. Plus you don’t have to touch everyone’s QGIS session…well much. Users still have to load the tables to get started but good clean directions should work. Of course this does make me still debate a plugin that does part of this for you.

What? Yeah – it’s that easy. Everyone working with drop down menus and making good clean data.

 

 

 

 

Dockering with Docker

· · Leave a Comment

It’s been a while since I posted anything mostly because I have no clue what to talk about these days. I’ve been working away on the Tn Address Server and one thing I wanted to do is to get it working in docker. Which – many of you are probably going IT’S NOT THAT HARD. Generally you are correct but it’s still a weird area for me to jump into. I’ve toyed with Docker for years – in general it works but I hardly ever have a reason to do much with it. Except now with the way I’m working on the TN address server I needed a way to “spin up” a postgis server and test everything and then take it down.

So I jumped back into Docker. Again.

You can build your own docker instance. You create a dockerfile and start adding whatever you need but I quickly decided I didn’t want to build my own dockerfile…yet.  After doing a bit of research Kartoza already has one built. Also – they have one for Geoserver . Geoserver has been one of those things that I don’t have quite integrated into the flow of the address sever yet – but I’m working on it.

Sitting down with Docker I had four things I wanted to do:

  • Functional database I can start and stop
  • Local Storage
  • I can sit in another room and work and still get to it on this machine.
  • I don’t want to pile of things installed on my computer.

Simple? Yes…mostly. It seems like for most of my career I’ve been dancing in between developer, system administrator, and geo person. I never seem to have complete control over any of the things except being a “geo person” scratches the itch more than the other two. So that started a day long “How do I…..?” which resulted in again reaffirming that stack exchange is the devil, documentation can be boring, and a lot of people can’t write anything to save their lives (myself included about half the time).

The biggest headache was keeping my data permanent on my hard drive. I can mount the geoserver docker to a directory  – why can’t I mount the postgis docker….and surprisingly there is a wealth of misinformation out there. In short – you have to make a volume for postgis to work – which is basically doing the following:

  • Create a /directory/somewhere
  • Create a volume: docker volume create –driver local –name pg_data –opt type=none –opt device=/directory/somewhere –opt o=bind

Finally I did the following:

docker run –name=postgis -d -e POSTGRES_USER=user -e POSTGRES_PASS=pass -e POSTGRES_DBNAME=tndemo -e ALLOW_IP_RANGE=0.0.0.0/0 -p 5432:5432 -v pg_data:/var/lib/postgresql/ kartoza/postgis:latest

Then

docker run -d -p 8580:8080 –name “geoserver” –link postgis:postgis -p 8080:8080 -v /media/data/demo/geoserver_data:/opt/geoserver/data_dir kartoza/geoserver:latest

PostGIS and Geoserver up and running in harmony…in docker. I was pleased. I can connect geoserver to postgis. I can connect to both things from my couch while I listen to the Baby Yoda Chronicles.

I was also a bit dismayed. Mostly at me.

To burn the amount of time I did getting to a functional setup was way too long. Not anyone fault as I have directions, functioning internet, and the ability to ask questions in the community. Which has been a problem for me as of late – I remember back to past failures and start fresh on a project and get haunted by the past failures. So much so I’ve been “tuning out” more. The pandemic has made things frustrating enough without my brain getting in the way. So I’m back to the art of learning this month and tossing out the old.

Now that this is running I can build and break the database at will now and hopefully get a running version of this that does everything I want it to do.

Maybe I’ll build a dockerfile just to say I did it.

Maybe I’ll do a few other fun things.

 

 

Modeling Part 2

· ·

Not being content with my last model….

.

I dove back in because I really need to name the points something because the forestry guys almost always use GARMIN GPS equipment. After sitting on this for a week the question became “How do I name everything” and get it to the point where the model handles 90% of it:

  1. Name the point and since I’m exporting GPX the field has to be “name”
  2. Perform some sort of field calculation to get a “pt1” for a name
  3. Label it in QGIS because I really want to be lazy.

I found the rest of the tools to let me do this:

  1. Add autoincrement field which gave me a 1 through X number
  2. Field calculator let me name this ‘pt’ || “AUTO”
  3. Set layer style

My Model now looks like:

Not to go over the additions in excruciating detail but three tools later and I have a functional point layer. My next question is can I slap this in github and share it – which I’m going to do shortly. Will this break the layer style? I’m not sure – but I can save this with a ton of notes for more development.

I’ve really ignored the functionality of the Processing Toolbox. This small dive into it has highlighted some sore spots I’ve had with the TN 911 address project. I get frustrated because I’m not the best in Python. The big problem I’ve had has been working through a few problems that are going to require some sort of interaction between the 911 address person and the database and I can’t do that part in SQL (I could – but it would create too much black magic). I can fix it with a model though.

Then all my python fears get a little bit better because all of this can be moved to python and I can pick at it.

My 10 to 15 minutes of point building are now down to 10 seconds.

Now to get them using something other than a Garmin GPS Unit – like their phones.

 

Modeling in QGIS

· ·

I should model more….but I don’t. You’re welcome.

Anyway – the last few weeks I’ve been boring you with tales of topology and now I’m going to bore you with tales of building a model. One of my longest running jobs has been in Forestry and that’s pretty much a set of repeatable processes. With the current lockdown I’ve been experimenting with client data and how to speed up some of what I’ve been doing.

QGIS has a modeler. In ArcGIS, Modelbuilder was probably a favorite because I’m always doing repeatable processes. Granted I’ve offloaded a ton of things to PostGIS BUT – wouldn’t if be cool if…..

The Forestry guys get a new property to look at and I usually do something clumsy to get it into QGIS. If it’s an inventory they go and walk it. They usually walk it at a certain interval so they call me and I:

  • Create a Grid using the Regular Points processing tool.
  • Manually Delete the points that fall too far outside the boundary
  • Name the points

This takes anywhere from 5 minutes to 15 minutes depending on the property and the how well we feel the boundary fits the ground. A simple model would look like this:

With the QGIS Processing Modeler you have to set up an Input before you can do anything. So my input will be a vector layer. I will drag the Regular Points Algorithm in and give it an output. If I double click Regular points I need an extent to draw the points. I’m going to set that to a model input. I give it a name and you can see how everything is starting to connect. I also give it some spacing which is 264 feet (which is 4 chains):

Not bad but not great:

but I need to clip it to the boundary……and assuming the boundary is wrong I probably need to clip include some points outside the boundary. I also don’t want to open the model every time and change the spacing. Soooooooo…..Lets get the the “Poof” Magic part:

What did I do?

  • Buffer the input property
  • Create a grid based on this buffered polygon
  • Extract the points from the buffered area
  • Create a new layer

That’s almost perfect. So 5 minutes down to this. Of course there are things I need to change like “what if they want an irregular cruise” that might be a 4×3 or a 5×3. I haven’t quite figured out numbering the points but I”m assuming I’ll stumble into it it shortly.

I haven’t used the modeler in quite a while but with some extra time I’m going to shrink some of my processes to as “quick as possible”.  Probably play around with Python some but I don’t really have to – this works for what I need to do with some small adjustments.

 

 

 

 

 

Topology: QGIS Geometry Checker and other tools

· ·

So lets dive more into “I need to fix my data”. In this case I have two layers – one is polygon and one is linestring.The polygon layer has overlaps and gaps. The linestring layer has dangles and is not intersected correctly causing more dangles. We looked at the polygon in the last post….and that’s what we will start with….and we have 208 errors which are gaps. The errors are the red lines you see along the borders.

So how am I going to fix this? Well I can always zoom to each error and manually fix it. It’s a slow go but At least I know what’s happening. If I think back to the ESRI world it seems their topology tool automatically fixes errors. Well – automatically with some intervention but it’s faster than manual.

Which brings us to this tool called Geometry Checker. It’s a plugin but it’s part of QGIS Core so you have it when you install QGIS. The goal is to clean up errors and it also has the added bonus to run some checks on your data.

Geometry Checker lets me look at some additional things like duplicate nodes. A Check for gaps. A Check for self intersections. A Check for smaller sliver polygons. It will also check to see that the layer you are working with follows the boundaries of a second layer. So do I need to check that this layer fits with the county? I probably should. So I’m going to check for polygons smaller than 1 square mile. Holes. Overlaps smaller than 1 square mile. Gaps bigger than 10 square miles. Which at this point there is no method to my madness – more just testing things to see what comes up.

You have 2 options at the end of this. One is to edit the layers “in place” which means the originals get changed. The other option is geopackage. As all my data is currently in postgis and I hate the idea of “edit in place” – I’m going to a geopackage and fix my data and then shove it back into the database.

When geometry checker is run a new geopackage is created. I am prompted with a list of errors I can work my way through and click a button to automatically fix the problem. I also have a setting for “error resolution settings” which will let me set some defaults like “merge the sliver polygons into the largest shared edge”. Overall it’s pretty awesome. It appears I can even fix more than one error at a time. I didn’t have much luck doing that BUT – I’m happy with one at a time in most cases.

What about the roads?

I have a road layer with roads that aren’t properly intersected. The Topology Checker works great and I can find them all…..so the Geometry Checker……

If you look at the Geometry checker I can check line strings for length, dangles, self intersection and a few other things. What I don’t get is a way to fix dangles like I have pictured above. Which sorta brings me to “I need a way to snap and break lines at intersections easily”. I can manually do it – but if I have a bad road layer (water lines, linear anything) that can be a pain to walk through.

but……there’s always a but…..

There are 2 tools in 3.14 (maybe earlier but that’s what I’m using) called Snap and Split Lines with Lines. So I’m going to run a Snap Geometries to Layer and then a Split Lines with Lines against the snapped layer and my Dangle issue almost goes away. Roads are always going to have dangling nodes – lets just not have them in the middle of the road. No i go back to Geometry Checker and look for duplicate nodes and other things.

So I have some tools to fix my data. Here would be my pros and cons:

Pros:

  • You can fix your data. Be it a shapefile, geopackage, postgis table, etc – the tools work. It’s a lot of tools also – if you’re coming from the ArcGIS world where you have to have a licensing level to run topology – you don’t have that problem anymore. You don’t have a table/data layer called topology – but you have a lot of tools to clean up your data and make it topologically correct.
  • New tools are coming constantly. I had a game plan for writing this article and discovered Snap and Break Lines with Lines. So if it seems like I shoved that in at the last minute (I did).

Cons:

  • You’ll notice back in the article you’ll notice I didn’t really give numbers on problems found. The Topology Checker would notice 208 gaps in my data and Geometry Checker would find 384. They are two different tools that (to me) serve two different purposes. So one might report more. One might report less. I would use both as a running check on my data and I might use topology checker throughout the day and geometry checker at the end of the day.
  • It’s not really a con – but when you’re fixing your data you may pull that data out of the original table (as I did) to fix it and then push your fixed data back into the database. Maybe I make a new table. I’m not sure on workflow yet so maybe this comes toward the end of the project.

One final word on Geopackage…..

One thing I have sort of taken for granted is geopackage. I say geopackage a lot. When I teach a class I yell geopackage. It wasn’t until I wrote this that I noticed everything is dumping back to geopackage. Some GRASS tools sometimes went back to shapefile in previous versions. It seems like there were a few tools that would in QGIS that wanted to go back to shape – it’s all Geopackage now. It’s nice.

—BUT—

Maybe I don’t want this much work fixing my data. I have geometry checker….and I have some tools from the processing toolbox……what if I wanted one tool to run my topology checks BECAUSE I’m really wanting to network these roads together…..what could that tool be………..those tools be…….

 

 

 

 

 

 

 

 

 

Contact