North River Geographic Systems Inc

Geospatial Problem Solving

processing

QGIS Advanced Training Aug 31-Sep 2 2021

The ‘QGIS Advanced Class” makes it’s first outing on August 31st through September 2nd 2021 virtually. We’ll be doing this over zoom.

So what are we doing in this class? I’m taking a deeper dive into:

  • Field Calculator
  • Processing Tools
  • Processing Models
  • Editing
  • Forms
  • SQL

We do this 4 hours a day for 3 days. Why? Because I’d prefer to break this up into digestible chunks. Since this is the first class I’m going to run this cheaper because I expect I’m going to be making a lot of notes.

Anyway – once more final class is coming on this – and I’ll have more or less a week full of QGIS goodness for the masses.

This class provides a certificate which means your participation benefits QGIS financially. Learn and make the software better!

Modeling Part 2

Not being content with my last model….

.

I dove back in because I really need to name the points something because the forestry guys almost always use GARMIN GPS equipment. After sitting on this for a week the question became “How do I name everything” and get it to the point where the model handles 90% of it:

  1. Name the point and since I’m exporting GPX the field has to be “name”
  2. Perform some sort of field calculation to get a “pt1” for a name
  3. Label it in QGIS because I really want to be lazy.

I found the rest of the tools to let me do this:

  1. Add autoincrement field which gave me a 1 through X number
  2. Field calculator let me name this ‘pt’ || “AUTO”
  3. Set layer style

My Model now looks like:

Not to go over the additions in excruciating detail but three tools later and I have a functional point layer. My next question is can I slap this in github and share it – which I’m going to do shortly. Will this break the layer style? I’m not sure – but I can save this with a ton of notes for more development.

I’ve really ignored the functionality of the Processing Toolbox. This small dive into it has highlighted some sore spots I’ve had with the TN 911 address project. I get frustrated because I’m not the best in Python. The big problem I’ve had has been working through a few problems that are going to require some sort of interaction between the 911 address person and the database and I can’t do that part in SQL (I could – but it would create too much black magic). I can fix it with a model though.

Then all my python fears get a little bit better because all of this can be moved to python and I can pick at it.

My 10 to 15 minutes of point building are now down to 10 seconds.

Now to get them using something other than a Garmin GPS Unit – like their phones.

 

QGIS Processing: History

If you’re anything like me (hopefully you aren’t) you’ll do something and immediately wished you hadn’t closed the processing window. Clip, simplify, select by location……

So in this case I had a call about contouring a Digital Elevation Model. Contours are easy. I ran them at 10…or maybe 20.

QGIS and DEM

I decided to clip them with the county boundary since the DEM extends outside of the county into the neighboring counties and states.

QGIS and Clipped DEM

So the question is now “WHAT DID YOU DO”. You get up. You come back. Cat jumps in your lap. Your Mom calls. QGIS keeps a history of your processing commands. If you look at the processing toolbox you’ll see a small clock.

Click it and another windows opens

Here is the cool thing. You click any of the commands and you’ll be taken back to your original QGIS Processing menu. So if I can contours and don’t like them at 20 foot intervals? Go back and check the history, double click,  and change it to 10 and run it again. You like python? You have what you need to know and you can run that from the python console. I made one small change and dumped it to a shape this time vs a memory layer:

So if you do forget a tool setting in the processing tools after you closed it – just run it again by looking at your history for the processing toolbox.

 

Geoprocessing in QGIS: 2.14 vs 2.16

So at least once a week I’m getting a phone call or an email on QGIS. This one was pretty awesome and it pointed out I need to cover geoprocessing in Class in a little more detail.

The question was “In our transition out of ESRI products we hit a snag. We want to merge or dissolve 925,000 polygons on an attribute. QGIS Hangs. Arcmap Completes it. Why doesn’t QGIS complete the task?” They took an abbreviated version of the QGIS class at FOSS4GNA back in the spring.

selection_429

I thought about it and started running some tests. You can dissolve/merge the data two ways in QGIS and you can do it in PostGIS. ArcGIS people tend to get hung up on this as you have one way in ArcMap. So I listed out the three ways from Simplest to hardest (Finals: GRASS won for speediest answer, PostGIS won for cleanest answer, QGIS won for simplest answer).

The Question brought up a good point. When you think about “geoprocessing” if you’re an ESRI user you drop back to ArcToolbox. selection_430

If you think of Geoprocessing in QGIS you actually can get two “correct” answers. One deals with FWTools and one deals with the Toolbox. Yes – both QGIS and ESRI have Toolboxes. Before QGIS 2.14 you had FWtools in the vector menu. That software was there from the beginning or close and is derived from http://fwtools.maptools.org/ (FW is Frank Warmerdam). Sextante started making an appearance sometime in 2.0 as a plugin. Sextante combined a lot of tools into one menu system. Eventually it made it was as a part of the core release and became the Processing Toolbox. FWTools started getting out of date. Somewhere in 2.12 or earlier it was decided to replace FWTOOls with the Processing Toolbox

The fun is at the 2.14 release. It’s a Long term release. If you’re running an earlier version you will still see FWTools. As I’m sitting here looking at 2.14.6 It appears FWTools is gone. In 2.16 the vector menu at the top has regained all the functionality you remember from 2.1 or early versions of 2.14 – it just looks different. You now have unified processing tools in 2.16. No FWTools.

selection_431Back to the problem. The student from class was running 2.14.2. They still had FWTools and “googling” pointed them to the answer being in the FWTools toolset under vector. The merge didn’t work. The Dissolve in the processing toolbox finished in about 20 minutes. So my followup email in a few days will be – Did it work and do you feel better? Are you really dropping the other stuff? I’ve yet to suggest people ditch their commercial GIS system. I do have more and more people starting from scratch and going open source. It’s interesting.

The moral to the story – QGIS is evolving. If you decide “I’m going to sit at this one version for a year” you are missing out. You should at least think of upgrading like you would an ESRI Service pack. If you are on 2.14 – upgrade to the current LTR (Long Term Release) release (which is 2.14.6 as of this date of posting). If you want to live on the edge – go with 2.16 (I did but I’m weird that way).

selection_432

 

Adventures in LIDAR and GRASS in Tennessee: Part 1

There are days when work slows – today was one of those days. So I can either go goof off or learn something. I decided to learn something.

Back when I worked for the Federal Gov’t I was somewhat in charge of creating ortho imagery. At the time (i don’t even want to say when this was) we either grabbed whatever elevation data was available or we built the data in the photogrammetry department. About the time I had decided to leave LIDAR was coming down in price and we had just started the jump into that. Flash forward 8 years later and LIDAR is pretty common place. They have hardware small enough for drones and there are myriad ways to get data. You can even get data freely available from the Federal Government.

I know – it’s not LIDAR but it’s what I think of every time I say Drone. Stop making fun of Drones! No.

I’ve numerous and myriad clients and sometimes I wonder what I can do to make their lives better and expand available services. I also watch the twitters and I saw a workshop occurring just south of my location using ArcGIS and an extension to process LIDAR. So once again I was wondering about LIDAR. Can I process it? I know I have enough tools at my disposal – like  GRASS.

I was recommended the NOAA Website to look for data. I did some digging. Guess what? LIDAR in my area is freely available now.

Selection_310

Tennessee has a very advanced GIS program. It started sometime in the mid-90’s and has taken a very encompassing view of GIS. They work at a state level for planning and coordination. The killer for me is most of the data is locked up at the counties or is for sale. It’s rare for me to convince a client to purchase any data – even data where we could really do something good with it. Counties or Gov’t agencies within the state used to take the view of “we paid for it and so do you”. Some still do. Tennessee released two counties worth or LIDAR and it appears a lot of the state is going into the National Map. So maybe things are changing for the better.

The TN GIS Server has the data. It’s small sets so I’m going to have to do a few things to grab it all. For testing purposes I only need a tile. I need a tile with Hardwood and Pine Plantations (yes – Forestry – I know what I want to test now).

TN has a very advanced program – but it’s all ESRI. So when I downloaded the LIDAR data it’s in zlas format. In case you haven’t kept up there’s been a little bit of a format war occurring. ESRI pushes one format called zlas which is (my opinion) ripped off of laszip. A lot of other people are using laszip. The short story is laszip is open – zlas isn’t. If you have ESRI software and you’ve bought the right extensions someone will tell you zlas. If you’re operating outside of that realm or are considering lidar - laszip. It should be laszip anyway – life is to short for locked up data formats. It’s not entirely locked up – I can build a tool to unlock it. I’ll get to that after I build a toaster to make toast.

So I want to do this one thing here – Tennessee OIR (or whoever) – Thank you for releasing this data. This is HUGE. The state sits on a treasure trove of data that it should open and this is a beautiful first step. Open up the format. Use laszip. If people want to go zlas they will – please don’t make your users start off having to deal with a proprietary format. I know you have a contract with ESRI. That’s fine – just realize there are more options than that one software vendor out and about.

So what do you do with presented with zlas? You open it up. For this I ripped zlas to las. RapidLasso made a toaster (of sorts) and are giving it away.  In other words I took a 235mb file and turned it into a 1.3 GB uncompressed open format.

Selection_311

….and what good does that do? Well – part 2 is coming. Right now I’m watching GRASS  gleefully process this data and I’m not sure what is going to happen. GRASS You say again? Not QGIS? You mean that really confusing horrible interfaced piece of software I once used in College. No. It’s the revamped new GUI interface that has 30 years of development behind it and it has all these really great tools for LIDAR. Yes – ye who have a limited budget and possibly can’t afford the flashy stuff have options. You’ve got a whole community of options. One being GRASS that you get when you download QGIS.

Selection_312
Point Cloud in GRASS – total point count is “A LOT”

I know it’s orange. I actually hate orange but GRASS defaulted and I don’t want to spend that much time making it not orange – this is Tennessee after all.