North River Geographic Systems Inc

Spatial Problem Solving

You are here: Home / Blog

Blog

LIDAR and QGIS 3.17

· · Leave a Comment

I usually try to not write about the “pre” release (technically it’s master) of QGIS. Of course I went and said “HEY I want to talk about QGIS at a conference” and a large part of what I wanted to talk about was the new release. Which also included Point Clouds……and 3.18 is coming tomorrow. Which led to a frustrating demo which is why I’m writing this out.

TN has all their LIDAR released as open data and I’ve played with it in GRASS and PDAL and in general it’s a lot. It’s a lot of thinking. It’s just a lot of points and frustration. The data is all stored in zlas and it’s a pain to do anything with since I’m not working in ESRI Software. Generally I uncompress it and keep the uncompressed copy so I’m now hording 4 counties of LIDAR. I tend to stay excited about it for a day and then drift back to PostGIS and QGIS until the need to hate myself happens again.

AS of later I’ve been dealing with a side project where elevation is important. I’ve had questions and wondered if my approach on that has been right. I’ve also been running QGIS master as I’m curious on how I can leverage some of the new functionality with clients and projects.

So my first whack at points looking like this in QGIS:

Floating point cloud which is like 500ish feet off the ground. At first I thought this was a feature and then I had decided it was a bug and luckily I can shoot the developers a quick message and Nyall Popped up and said “projection problem”.

So Tn’s lidar is all hosted here. I had to go back and read the metadata. What I learned is I had a horizonal projection problem as I had assumed X and I had a vertical projection problem because I didn’t think that was an issue. In pulling the LIDAR data out of the zlas format I’m pretty sure I wrecked the projection. So in going back I needed to fix it. How do I fix it? PDAL. Going back I determined that my new LAZ file was EPSG: 6576 in the horizontal and EPSG: 6360 in the vertical. So I fixed it by this bit of trickery with PDAL:

I don’t want to get into all the technical on why this worked and how I did it – except that PDAL has some amazing command line abilities for fixing LAZ/LAS files….especially the projection.

Here is the cool thing. Upon adding the new data to QGIS I was prompted that I was missing a needed transformation. QGIS is running off Proj 7. I was prompted to click a link and fix it. I blindly clicked. I blindly imported.

It worked. LIDAR matches my DEM. Proj knows the problem and QGIS fixes it. Clickity Click. Probably doesn’t happen in 100% of the cases but it worked for me.

“I HAVE TO DO ALL THIS CLICKING OMG I HAVE LIDAR I DON’T KNOW THAT MUCH ABOUT IT” you will say. I think in most cases if you have a LAS/LAZ file you’re fine. This was a special case of me having data with a screwed up Projection/No Projection in preparation for a demo. So you don’t need to know things about PDAL and things about your LIDAR. You add it and go. It’s not a bad thing to know – but overall you should be fine.

This is all pretty amazing actually. You have this small open source desktop that can render a point cloud. I think this opens the door to analysis and other things that’s usually been outside of QGIS’s normal operations. GRASS can already work with LIDAR. Now you can look at it. Oh the things you’ll do.

Selecting by Location

· ·

You would think after doing this for as long as I have I would “have an answer” for about everything. Last week found me going “well select by location and check…well…check….what do you want? Well pick intersects… no…within…no… cross…well……&*^%”

So I started a small journey of self discovery on “what does every check box do”. Surprisingly – I didn’t know what every checkbox did…and again – I’ve been doing this too long not to know. Yes I can read the help but…..

I pulled out all the counties that surround me and randomly generated 5000 points and buffered them. Then I just started clicking and walking through “why”. To be honest I never just create test data – and that was amazingly fast and made my life pretty simple.

So – here we go. I will select my county and crank up “Select by Location” in QGIS:

  • “Intersects” does exactly what you think it does. If the polygon “intersects” my county it is selected. I have 851 polygons touching my county.
  • “Are Within” does what I expected as I have 651 polygons within my county.
  • “Overlap” does what I want – I have 166 polygons that touch the county line. They don’t reside in any one county
  • “Disjoint” gives me everything outside. I have 4149 polygons that don’t touch my county. Which makes sense as I have 851 that intersect
  • “Contain” does nothing. Which doesn’t make sense as it says “contain”. If I select my county and and run against the 5000 polygons nothing happens. If I select a polygon within the county and flip my select: Give me all the counties that contain this one polygon – that works.
  • “Touch” does nothing because I expect it to be similar so overlap. Select my county and run against the polygons and nothing. Select my county and ask it to identify every county that touches my selected one and that works. My county touches 10 other counties.
  • “Equal” does exactly what I expect as none of my 5000 polygons “Equals” the geometry of my county.
  • The weirdest is “cross”. I had to do some digging. So with cross a polygon can cross another polygon but there is nothing but more polygons. Lets say a line crosses into the county – that intersection is a point. After doing some researching crosses mainly is for lines crossing other lines or polygons. Which if I do a little bit of trickery with QGIS by converting the polygons to lines and running “Select By Location”. We have 166 selected lines which matches what I have in overlap.

So anyway – Select by Location. It does everything you want and some stuff you’re not expecting. Which – this was fun and I might do it with another processing tool I should 100% understand and don’t.

QGIS Virtual Training – March 23-24th 2021

· ·

  • March 23rd 2021 – 12pm to 4pm EST
  • March 24th 2021 – 12pm to 4pm EST
  • Cost: $150
  • Tickets: EventBrite
  • Participants: 10

So it’s been a year since I’ve taught this class. What a year.

Surprise – Covid is still causing issues. So I’ve decided to go virtual with the class. I’m doing two four hour sessions. I’ve reduced the cost of the class because it’s Virtual AND this won’t be as fun as sitting in a room with everyone. We will make it slightly fun though. Actually a lot of fun.

Anyway – the class is for beginners with QGIS. Maybe you’re curious about using it. Maybe you’ve tried and gotten frustrated. We will cover the interface, data types, layouts, and as many tools as we can. By the time you leave you should be able to do some editing, add data, make a map, and break something. I’ll also be hanging around after class to answer questions or just talk.

We all get a QGIS certificate which helps that project. I’m also holding this to 10 participants. The biggest class I’ve taught had 100 people in it and the smallest has had 3. Ten is a good number. Ten works.

THE BIGGER Conversation

I have 24 (almost 32) hours of material covering Editing, Forms, SQL, Mobile, Field Calculator, etc. No one wants to sit with me for 3 or 4 straight days for 8 hours a day in this class. So I’m splitting it up into probably 3 classes which is how it was originally written before I morphed it into one class. I’m going to teach the intro class for a bit and then offer the rest. I will probably offer a discount for repeat participants.

I’ve been thinking about this for about 2 months and trying to decide what to do and how to do it. Navigating the pandemic as a small business has been maddening. The thought of doing an online class is somewhat exhausting……..except…….

I’ve been teaching this class for about 7 years. The class has changed as QGIS has changed and it went from “talking” to more “doing”. So the problem will be now how to get the next two classes to make sense with regards to the first one…in a virtual environment.

Anyway – see you virtually. Well 10 of you.

Offline Editing Plugin: TN911 Project

· ·

One of the weirder problems we had on moving the ESRI setup was editing the data. It wasn’t the normal “hey lets add an address” but it was tackling the idea of “We need to edit the data in the field”.

As it was explained “the way we did things”  was to drag the data out in the field with the editor because they were running a simple Arcmap/FBGDB. Now that we had moved to a Postgresql database I wasn’t keen on people “carrying the data out”. I preferred they connect to the database and edit it like civilized folk.

That is a problem because the county is rural and connections are sketchy and all of that led back to the idea of “Take a copy of the data with you”. We had a mobile solution but nothing making my life easier.

There is a plugin that originally caused much angst and it is called the OfflineEditing Plugin. It’s seriously the most no-thrills thing you will find in QGIS.

At one point in the early stages we had a lot of errors and I couldn’t figure out why. We have an ID attached to each feature that never changes. The client had discovered the plugin and it would let you check data out and edit it and check it back in. Which was what we wanted to happen…..except back when 3.0 was brand new we had a lot of things change unexpectedly. I had a trigger setup that added the primary key to the features ID. So data would come out. Data would go in and between the triggers and functions the ID would change. To the client this looked like a great solution. The intern thought it was awesome. The guy watching things change in the database didn’t have much joy.

I’ve not touched it since 3.2 or 3.4 in QGIS was out.

I’m running a “beta” (3.17) copy of QGIS at the moment. So I built a quick database and decided to play. It checks your data out and you then have a seamless offline dataset stored in a geopackage (the original data is in a postgresql database). I check it out and edit and check it back….and……

It worked. ID’s were respected. Sequences were maintained. I added data and checked data in and Edit Dates and Edit Times were also saved (at the time of check-in). Since I checked the entire database out and checked it in things I didn’t work on received a new edit date. Which – I think there is a way around that but that goes beyond the 1 hour test today. Which – given the 2+ years since I’ve touched it this plugin may have been working fine for a bit.

So what does this mean? Well – you can take your laptop and check your data out and go into the field. AWESOME IT’S VERSIONED….well not in the way ESRI versions anything so you will overwrite some data if you aren’t careful. It does make this setup a little more flexible although I’d rather toss this into a mobile app and only worry about the things we need to worry about – in other words if you’re working on addressing in a neighborhood I only care about that neighborhood. Which means I’d probably split that data out – check it out – edit it – Check it in and then merge it into your original data. More work but safer. I like safe.

Overall – a much better experience with the Offline Editing Plugin this go around.

Parking Lots and Lines

· ·

New Year – Different posts.

I love Listservs – I’ve resigned from all the OSM listservs though. It takes the stress level way down. Of course I still edit. Chattanooga from an OSM standpoint is pretty decent. Three things I usually tackle are buildings, parking lots, and Fire Hydrants. Why parking lots? If you ever wondered how much space we devote to parking cars….try outlining them. I want to fully map parking spots this year with number of spaces, handicap access, etc.

Me and JOSM

Two things are problematic in my OSM mapping around here:

  • I don’t have a drone – hence I don’t have updated imagery. I have several tricks which gives me nearly current imagery. That and some field work and it’s mostly all good.
  • Now vs 10 years ago has brought more editors. I’m always curious who – is it local or someone armchair mapping. I really don’t care as long as the map is updated correctly.

My mapping isn’t continuous and considering how much free time I had in 2020, I decided not to map and instead go canoeing. When I finally did jump back in someone had been adding a lot of service roads and parking isles. I check and it’s Amazon Logistics.

The screenshot below shows nothing more than lines – BUT you will notice some small gate-like symbology. Apparently the drivers are relaying info back to the editors. If you look at the page for the editors and reviewers there is a LOT of them also. I welcome my new Amazon Overlords. I’m the guy putting in the parking lots.

Anyway – it did strike me that you have yet another giant corporation harvesting out of OSM. I guess that speaks to the overall quality of what they are getting. I do wonder how much the map changes from these editors. It’s a complicated mapping life out here at the moment. You may have an entire delivery infrastructure relying on this data. That I am contributing to…..

With that I decided to join OSM US for the low low price of whatever it was. That breaks several years long standing of “Join nothing” but it is a New Year.

Contact