North River Geographic Systems Inc

Spatial Problem Solving

QGIS

Oh no he’s talking about training again

· ·

I’ve been debating shaking up the blog portion of the website for a bit. I start writing and then go “What’s the point” and I’m sitting on a half dozen technical/non-technical things I’ve been working on. So anyway – where to start for this one.

So the Virtual training seems to be working. I teach the first one his month (actually just finished), the next one in April and after that I’ll know what is working and what doesn’t. There have been a never ending stream of emails from people on ranging from “I want to take an advanced class” to “Introduction is too simple” and “what are your plans for something harder”. To which I have asked “What do you consider harder?” and the answer is always “Not Introduction”. So I got that going for me which is nice.

There is another class coming. Actually when I had made all the arrangements to teach in a classroom setting – the class had turned into a 2 day event with the possibility of a third happening. Way back when I was an ESRI instructor it seemed like Intro to Desktop was 2 or 3 days, Intermediate was 2 days, and Advanced was two days. Which I have no plans on doing for 7 or 8 days – BUT – the second class is proving problematic as I’m sitting on about 19 hours of information which is about over 2 full days and almost Five 4 hour days virtually. What to do……..

The thought is currently to call this QGIS: Tools and Processing. Which means I get to cover Processing Tools, Modeler, Editing, and Forms. The problem with that is currently that looks like 10 to 12 hours of “stuff” and I’d like to keep this at 8. Maybe I can’t. Maybe it turns into 3 days which are 4 hours long.

Which in my fever driven fantasies about training that gives me an:

  • Introduction covering the basics
  • The Intermediate part where we are building models and exposing editing tools like forms.
  • Advanced…………

Advanced would end up being PostGIS being brought into the mix with QGIS. So end it on an Enterprise worthy note. Your organization needs multiple editors and a data repository and the normal directory structure isn’t cutting it so you would want this class. Of course this doesn’t cover:

  • Reports/Atlas
  • Time manager
  • LIDAR
  • Stupid amounts of Symbology like the Geometry Generator.

Those 4 things don’t suck up much of my life. They are important though….so Decisions. I don’t want to turn this into a training business which is a problem I worry about. I like working with data and clients. I also don’t want a repeat of 2020 where I realize how fragile my business is against a very small germ.

Anyway – more on training later….

Intro to QGIS Virtual Training – April 27-28 2021

· ·

With the first class filling up I’ve decided to do another Intro to QGIS Class. Which I know – 2 intro classes but it’s a popular subject for those that just want to know something about QGIS. Again it will be 4 hours a day for 2 days from 12 EST to 4 EST. Plus virtual – which means I’m live.

I’ve revamped the materials to take advantage of the current releases and cleaned up the exercises. So those are now shorter and more plentiful and the hope is you can just do them without me slowing down much. We have a lot of ground to cover.

Again we’ll be covering:

  • Interface
  • Data and Symbology
  • Processing tools
  • Editing
  • Plugins
  • Layouts

Everyone gets a certificate from the QGIS organization (which makes it easy for me to push some of the money back to QGIS).

Anyway – show up. Limit of 10. Cost is $150 US Dollars.

Register here!

Here’s the big change from the March Class:

  • I’m working on a refund policy for those that sign up but can’t attend. Since I’m using EventBrite I’m losing a small part of the cost for less headache. Anyway – I’ll have that posted shortly. Maybe I can put you in another class OR I’ll refund the money or something. We’ll figure it out.
  • If you’re a conservation group and you can’t swing the $150 give me a shout and we’ll figure out a way to get you in the class.

LIDAR and QGIS 3.17

· ·

I usually try to not write about the “pre” release (technically it’s master) of QGIS. Of course I went and said “HEY I want to talk about QGIS at a conference” and a large part of what I wanted to talk about was the new release. Which also included Point Clouds……and 3.18 is coming tomorrow. Which led to a frustrating demo which is why I’m writing this out.

TN has all their LIDAR released as open data and I’ve played with it in GRASS and PDAL and in general it’s a lot. It’s a lot of thinking. It’s just a lot of points and frustration. The data is all stored in zlas and it’s a pain to do anything with since I’m not working in ESRI Software. Generally I uncompress it and keep the uncompressed copy so I’m now hording 4 counties of LIDAR. I tend to stay excited about it for a day and then drift back to PostGIS and QGIS until the need to hate myself happens again.

AS of later I’ve been dealing with a side project where elevation is important. I’ve had questions and wondered if my approach on that has been right. I’ve also been running QGIS master as I’m curious on how I can leverage some of the new functionality with clients and projects.

So my first whack at points looking like this in QGIS:

Floating point cloud which is like 500ish feet off the ground. At first I thought this was a feature and then I had decided it was a bug and luckily I can shoot the developers a quick message and Nyall Popped up and said “projection problem”.

So Tn’s lidar is all hosted here. I had to go back and read the metadata. What I learned is I had a horizonal projection problem as I had assumed X and I had a vertical projection problem because I didn’t think that was an issue. In pulling the LIDAR data out of the zlas format I’m pretty sure I wrecked the projection. So in going back I needed to fix it. How do I fix it? PDAL. Going back I determined that my new LAZ file was EPSG: 6576 in the horizontal and EPSG: 6360 in the vertical. So I fixed it by this bit of trickery with PDAL:

I don’t want to get into all the technical on why this worked and how I did it – except that PDAL has some amazing command line abilities for fixing LAZ/LAS files….especially the projection.

Here is the cool thing. Upon adding the new data to QGIS I was prompted that I was missing a needed transformation. QGIS is running off Proj 7. I was prompted to click a link and fix it. I blindly clicked. I blindly imported.

It worked. LIDAR matches my DEM. Proj knows the problem and QGIS fixes it. Clickity Click. Probably doesn’t happen in 100% of the cases but it worked for me.

“I HAVE TO DO ALL THIS CLICKING OMG I HAVE LIDAR I DON’T KNOW THAT MUCH ABOUT IT” you will say. I think in most cases if you have a LAS/LAZ file you’re fine. This was a special case of me having data with a screwed up Projection/No Projection in preparation for a demo. So you don’t need to know things about PDAL and things about your LIDAR. You add it and go. It’s not a bad thing to know – but overall you should be fine.

This is all pretty amazing actually. You have this small open source desktop that can render a point cloud. I think this opens the door to analysis and other things that’s usually been outside of QGIS’s normal operations. GRASS can already work with LIDAR. Now you can look at it. Oh the things you’ll do.

Offline Editing Plugin: TN911 Project

· ·

One of the weirder problems we had on moving the ESRI setup was editing the data. It wasn’t the normal “hey lets add an address” but it was tackling the idea of “We need to edit the data in the field”.

As it was explained “the way we did things”  was to drag the data out in the field with the editor because they were running a simple Arcmap/FBGDB. Now that we had moved to a Postgresql database I wasn’t keen on people “carrying the data out”. I preferred they connect to the database and edit it like civilized folk.

That is a problem because the county is rural and connections are sketchy and all of that led back to the idea of “Take a copy of the data with you”. We had a mobile solution but nothing making my life easier.

There is a plugin that originally caused much angst and it is called the OfflineEditing Plugin. It’s seriously the most no-thrills thing you will find in QGIS.

At one point in the early stages we had a lot of errors and I couldn’t figure out why. We have an ID attached to each feature that never changes. The client had discovered the plugin and it would let you check data out and edit it and check it back in. Which was what we wanted to happen…..except back when 3.0 was brand new we had a lot of things change unexpectedly. I had a trigger setup that added the primary key to the features ID. So data would come out. Data would go in and between the triggers and functions the ID would change. To the client this looked like a great solution. The intern thought it was awesome. The guy watching things change in the database didn’t have much joy.

I’ve not touched it since 3.2 or 3.4 in QGIS was out.

I’m running a “beta” (3.17) copy of QGIS at the moment. So I built a quick database and decided to play. It checks your data out and you then have a seamless offline dataset stored in a geopackage (the original data is in a postgresql database). I check it out and edit and check it back….and……

It worked. ID’s were respected. Sequences were maintained. I added data and checked data in and Edit Dates and Edit Times were also saved (at the time of check-in). Since I checked the entire database out and checked it in things I didn’t work on received a new edit date. Which – I think there is a way around that but that goes beyond the 1 hour test today. Which – given the 2+ years since I’ve touched it this plugin may have been working fine for a bit.

So what does this mean? Well – you can take your laptop and check your data out and go into the field. AWESOME IT’S VERSIONED….well not in the way ESRI versions anything so you will overwrite some data if you aren’t careful. It does make this setup a little more flexible although I’d rather toss this into a mobile app and only worry about the things we need to worry about – in other words if you’re working on addressing in a neighborhood I only care about that neighborhood. Which means I’d probably split that data out – check it out – edit it – Check it in and then merge it into your original data. More work but safer. I like safe.

Overall – a much better experience with the Offline Editing Plugin this go around.

I’m making a Map: Location

· ·

I should write something. My last few “2020 in Review” posts aren’t working like I’d wish – so I sat down last night to make a map.

A map of what? Good question. I think in my every day attempts of databases and technical wizardry I lose sight of “I actually like maps” and I’m not the best at it. Given my background I should be better. I decided to make a series of maps on “things I like”: Canoeing, Hiking, Bike riding, etc. So my first shot is a canoeing map. A very simple map of something I like to do.

The location is easy – The start of the TN River Gorge. It’s a great 8ish mile paddle. Simple. Quiet.

canoe
Looking into the Tn River Gorge

My first thought was to do what I normally do and just drape things over a aerial photo. I’ve decided to make it a bit more fun and not go that route – This may end up being Black and White. I’m also going to stick to two sizes of paper in this series of maps. Either 8×11 or 11×17.

When you canoe this loop – you start north and head south and loop around an island and then head back north again. The first half is all upstream (heading south) and the last half is all downstream (heading north). Bonus it’s all NW-SE trending so it will need to be rotated.

I start building it in QGIS with some aerial imagery and Hillshades for fun:

QGIS Canoe Map setup

If I make a 11×17 layout it looks more like this because I’ve added 120 degrees of rotation:

QGIS canoe map setup

As I sat here I had a question. I scale it. I generally start to get an idea of “what” I want. I know the extent of the map but can I get the extent back into QGIS so I can start clipping things and not having to include 500 square miles of stuff for a map. I could just set up the boundary back into QGIS as delimited text (make points -> draw polygon). I could just “eye ball it” which is how I do about everything from Jeep repair to Home Repair.

For fun I searched the tool box and….Print Layout Map to Layer

I run that and……I know where my map falls in the main window. So I can buffer it and start clipping data.

Why even post….I had no clue this tool was even here. My next map (another canoeing map) will be a bit larger than 11×17 so I’ll nee to use Atlas to setup 2 to 3 pages. I now have the general extent for 11×17 at X scale, copy and past this boundary a few times, and rotate it to get the general layout locations I will need

Map 1 is well on it’s way to happening. This is a bit therapeutic and that’s something I need at the moment.

 

 

Contact