QGIS

LIDAR and QGIS 3.17

February 18, 2021 by rjhale

I usually try to not write about the “pre” release (technically it’s master) of QGIS. Of course I went and said “HEY I want to talk about QGIS at a conference” and a large part of what I wanted to talk about was the new release. Which also included Point Clouds……and 3.18 is coming tomorrow. Which led to a frustrating demo which is why I’m writing this out.

TN has all their LIDAR released as open data and I’ve played with it in GRASS and PDAL and in general it’s a lot. It’s a lot of thinking. It’s just a lot of points and frustration. The data is all stored in zlas and it’s a pain to do anything with since I’m not working in ESRI Software. Generally I uncompress it and keep the uncompressed copy so I’m now hording 4 counties of LIDAR. I tend to stay excited about it for a day and then drift back to PostGIS and QGIS until the need to hate myself happens again.

AS of later I’ve been dealing with a side project where elevation is important. I’ve had questions and wondered if my approach on that has been right. I’ve also been running QGIS master as I’m curious on how I can leverage some of the new functionality with clients and projects.

So my first whack at points looking like this in QGIS:

Floating point cloud which is like 500ish feet off the ground. At first I thought this was a feature and then I had decided it was a bug and luckily I can shoot the developers a quick message and Nyall Popped up and said “projection problem”.

So Tn’s lidar is all hosted here. I had to go back and read the metadata. What I learned is I had a horizonal projection problem as I had assumed X and I had a vertical projection problem because I didn’t think that was an issue. In pulling the LIDAR data out of the zlas format I’m pretty sure I wrecked the projection. So in going back I needed to fix it. How do I fix it? PDAL. Going back I determined that my new LAZ file was EPSG: 6576 in the horizontal and EPSG: 6360 in the vertical. So I fixed it by this bit of trickery with PDAL:

I don’t want to get into all the technical on why this worked and how I did it – except that PDAL has some amazing command line abilities for fixing LAZ/LAS files….especially the projection.

Here is the cool thing. Upon adding the new data to QGIS I was prompted that I was missing a needed transformation. QGIS is running off Proj 7. I was prompted to click a link and fix it. I blindly clicked. I blindly imported.

It worked. LIDAR matches my DEM. Proj knows the problem and QGIS fixes it. Clickity Click. Probably doesn’t happen in 100% of the cases but it worked for me.

“I HAVE TO DO ALL THIS CLICKING OMG I HAVE LIDAR I DON’T KNOW THAT MUCH ABOUT IT” you will say. I think in most cases if you have a LAS/LAZ file you’re fine. This was a special case of me having data with a screwed up Projection/No Projection in preparation for a demo. So you don’t need to know things about PDAL and things about your LIDAR. You add it and go. It’s not a bad thing to know – but overall you should be fine.

This is all pretty amazing actually. You have this small open source desktop that can render a point cloud. I think this opens the door to analysis and other things that’s usually been outside of QGIS’s normal operations. GRASS can already work with LIDAR. Now you can look at it. Oh the things you’ll do.

Selecting by Location

February 15, 2021 by rjhale

You would think after doing this for as long as I have I would “have an answer” for about everything. Last week found me going “well select by location and check…well…check….what do you want? Well pick intersects… no…within…no… cross…well……&*^%”

So I started a small journey of self discovery on “what does every check box do”. Surprisingly – I didn’t know what every checkbox did…and again – I’ve been doing this too long not to know. Yes I can read the help but…..

I pulled out all the counties that surround me and randomly generated 5000 points and buffered them. Then I just started clicking and walking through “why”. To be honest I never just create test data – and that was amazingly fast and made my life pretty simple.

So – here we go. I will select my county and crank up “Select by Location” in QGIS:

“Intersects” does exactly what you think it does. If the polygon “intersects” my county it is selected. I have 851 polygons touching my county.
“Are Within” does what I expected as I have 651 polygons within my county.
“Overlap” does what I want – I have 166 polygons that touch the county line. They don’t reside in any one county

“Disjoint” gives me everything outside. I have 4149 polygons that don’t touch my county. Which makes sense as I have 851 that intersect
“Contain” does nothing. Which doesn’t make sense as it says “contain”. If I select my county and and run against the 5000 polygons nothing happens. If I select a polygon within the county and flip my select: Give me all the counties that contain this one polygon – that works.
“Touch” does nothing because I expect it to be similar so overlap. Select my county and run against the polygons and nothing. Select my county and ask it to identify every county that touches my selected one and that works. My county touches 10 other counties.
“Equal” does exactly what I expect as none of my 5000 polygons “Equals” the geometry of my county.
The weirdest is “cross”. I had to do some digging. So with cross a polygon can cross another polygon but there is nothing but more polygons. Lets say a line crosses into the county – that intersection is a point. After doing some researching crosses mainly is for lines crossing other lines or polygons. Which if I do a little bit of trickery with QGIS by converting the polygons to lines and running “Select By Location”. We have 166 selected lines which matches what I have in overlap.

So anyway – Select by Location. It does everything you want and some stuff you’re not expecting. Which – this was fun and I might do it with another processing tool I should 100% understand and don’t.

QGIS Virtual Training – March 23-24th 2021

February 9, 2021 by rjhale

March 23rd 2021 – 12pm to 4pm EST
March 24th 2021 – 12pm to 4pm EST
Cost: $150
Tickets: EventBrite
Participants: 10

So it’s been a year since I’ve taught this class. What a year.

Surprise – Covid is still causing issues. So I’ve decided to go virtual with the class. I’m doing two four hour sessions. I’ve reduced the cost of the class because it’s Virtual AND this won’t be as fun as sitting in a room with everyone. We will make it slightly fun though. Actually a lot of fun.

Anyway – the class is for beginners with QGIS. Maybe you’re curious about using it. Maybe you’ve tried and gotten frustrated. We will cover the interface, data types, layouts, and as many tools as we can. By the time you leave you should be able to do some editing, add data, make a map, and break something. I’ll also be hanging around after class to answer questions or just talk.

We all get a QGIS certificate which helps that project. I’m also holding this to 10 participants. The biggest class I’ve taught had 100 people in it and the smallest has had 3. Ten is a good number. Ten works.

THE BIGGER Conversation

I have 24 (almost 32) hours of material covering Editing, Forms, SQL, Mobile, Field Calculator, etc. No one wants to sit with me for 3 or 4 straight days for 8 hours a day in this class. So I’m splitting it up into probably 3 classes which is how it was originally written before I morphed it into one class. I’m going to teach the intro class for a bit and then offer the rest. I will probably offer a discount for repeat participants.

I’ve been thinking about this for about 2 months and trying to decide what to do and how to do it. Navigating the pandemic as a small business has been maddening. The thought of doing an online class is somewhat exhausting……..except…….

I’ve been teaching this class for about 7 years. The class has changed as QGIS has changed and it went from “talking” to more “doing”. So the problem will be now how to get the next two classes to make sense with regards to the first one…in a virtual environment.

Anyway – see you virtually. Well 10 of you.

I’m making a Map: Location

December 23, 2020 by rjhale

I should write something. My last few “2020 in Review” posts aren’t working like I’d wish – so I sat down last night to make a map.

A map of what? Good question. I think in my every day attempts of databases and technical wizardry I lose sight of “I actually like maps” and I’m not the best at it. Given my background I should be better. I decided to make a series of maps on “things I like”: Canoeing, Hiking, Bike riding, etc. So my first shot is a canoeing map. A very simple map of something I like to do.

The location is easy – The start of the TN River Gorge. It’s a great 8ish mile paddle. Simple. Quiet.

My first thought was to do what I normally do and just drape things over a aerial photo. I’ve decided to make it a bit more fun and not go that route – This may end up being Black and White. I’m also going to stick to two sizes of paper in this series of maps. Either 8×11 or 11×17.

When you canoe this loop – you start north and head south and loop around an island and then head back north again. The first half is all upstream (heading south) and the last half is all downstream (heading north). Bonus it’s all NW-SE trending so it will need to be rotated.

I start building it in QGIS with some aerial imagery and Hillshades for fun:

If I make a 11×17 layout it looks more like this because I’ve added 120 degrees of rotation:

As I sat here I had a question. I scale it. I generally start to get an idea of “what” I want. I know the extent of the map but can I get the extent back into QGIS so I can start clipping things and not having to include 500 square miles of stuff for a map. I could just set up the boundary back into QGIS as delimited text (make points -> draw polygon). I could just “eye ball it” which is how I do about everything from Jeep repair to Home Repair.

For fun I searched the tool box and….Print Layout Map to Layer

I run that and……I know where my map falls in the main window. So I can buffer it and start clipping data.

Why even post….I had no clue this tool was even here. My next map (another canoeing map) will be a bit larger than 11×17 so I’ll nee to use Atlas to setup 2 to 3 pages. I now have the general extent for 11×17 at X scale, copy and past this boundary a few times, and rotate it to get the general layout locations I will need

Map 1 is well on it’s way to happening. This is a bit therapeutic and that’s something I need at the moment.

Drop Down Menus for the TN NG911 Server

December 4, 2020 by rjhale

The problem is I have too many options. Options are awesome except when you’re dealing with people inputting data and you only want them to do it one way.

QGIS has had the ability to build drop down lists for a good while and it’s a very flexible way to work. For those of you in ESRI land imagine Domains and Subtypes on Steroids. In earlier versions you had widgets and currently you have Attribute Forms.

If you have a data layer in QGIS (all my layers are coming out of PostgreSQL/PostGIS) just right click the layer and go to Properties. You’ll notice all your fields have a chance to be a “widget”.

Addressing data is probably 90% picking existing attributes. You don’t want to get too fancy in making up new stuff. For instance – Street Type.

I ran out of fingers and toes on counting the number of street types. For the TN guys they all get a ESRI File Based Geodatabase with this information spelled out as Domain tables. How do you do that here? QGIS gives you three ways to make this happen:

Unique Values sums up an existing data set and you can get a pick list of existing data. The good thing is you don’t have to do anything really to make it work and the bad is you’ll get Avenue and Avenew. Misspellings aren’t your friend.
Value Maps let you fill things out or import a CSV. So if you save this information in a spreadsheet you can import it in.
Value relations pulls from an existing table. So if you build a table with this information you can pull that in by first adding the information to your QGIS session and then pulling it into the widget.

In order to cut out down the options, I pre-built all the tables needed into the database. I add them into qgis, group them, and collapse the group because No one needs to see them. Yet. You can’t use a relation widget with the table being added to QGIS.

Back to the Attribute Forms and this is the boring part. I want to select the type table and apply it to the type field using a Value Relation. The important part is the key and the value is the description of that attribute. Both things could be “type” but I chose to have a type and a description:

Go through and start setting up your Forms. What if you have a second person working on this?

Save your style to the database source:

QGIS saves a table in the public schema and everyone with the proper permissions and access is good to go. Plus you don’t have to touch everyone’s QGIS session…well much. Users still have to load the tables to get started but good clean directions should work. Of course this does make me still debate a plugin that does part of this for you.

What? Yeah – it’s that easy. Everyone working with drop down menus and making good clean data.