Lidar

QGIS, 3DEP, and You

I did a demo the other day for some foresters and I almost wanted to walk into traffic by the end of it. Not because of anything with QGIS and LiDAR – but I think I finally got “Virtual Fatigue”.

Part of the demo worked amazingly well. If you’re not familiar with TN’s LiDAR Program most (I think all) of the state is covered in free LiDAR. Yes Free. You can grab a county’s worth of LiDAR data and get to work. Here’s the other thing you need to know – it’s in zLas format. Yes – zLAS – the most annoying of the formats. I have to uncompress it using LAS Liberator. Then there is about 20 minutes of hoping I’m not at the corner of 4 tiles. It’s painful.

PDAL has been the engine that has been making LiDAR viewable in QGIS for a couple of versions now. I don’t tend to use it a lot (because I don’t do a ton of work with LiDAR) but when needed it it’s invaluable. For me the entire crunch of doing anything with lidar has been:

up to a 50 GB Download for an entire county
Conversion from zlas to laz
Define the vertical and horizontal projection of a tile using PDAL.

For the demo I wanted to do something “lidar” for a group of foresters to show them that FOSS4G works and we’re not just some weird hobby. I knew PDAL was being used on the backend of the USGS LiDAR Explorer but had not used it.

You can select an area (this is only for the US) and ONLY GET THAT AREA. I picked a pine plantation that had been partially thinned. I select the area using the interface. Choose my download format and PROJECTION PARAMETERS and then something cool happens. I generate a PDAL Pipeline JSON file and from the command line:

pdal pipeline process.json

I drag and drop the output of the above process into QGIS:

My life just got a lot easier and my blood pressure dropped like 10 points. LiDAR in TN is no longer a multi-hours scripted affair full of yelling. It’s literally pick what you want and then do the thing you want to do. QGIS gets easier. GRASS gets easier. Probably easier for the ESRI users also.

Anyway – check out LiDAR explorer from the USGS (especially if you’re in TN). It’s worth a few minutes of digging to learn about this tool.

LIDAR and QGIS 3.17

I usually try to not write about the “pre” release (technically it’s master) of QGIS. Of course I went and said “HEY I want to talk about QGIS at a conference” and a large part of what I wanted to talk about was the new release. Which also included Point Clouds……and 3.18 is coming tomorrow. Which led to a frustrating demo which is why I’m writing this out.

TN has all their LIDAR released as open data and I’ve played with it in GRASS and PDAL and in general it’s a lot. It’s a lot of thinking. It’s just a lot of points and frustration. The data is all stored in zlas and it’s a pain to do anything with since I’m not working in ESRI Software. Generally I uncompress it and keep the uncompressed copy so I’m now hording 4 counties of LIDAR. I tend to stay excited about it for a day and then drift back to PostGIS and QGIS until the need to hate myself happens again.

AS of later I’ve been dealing with a side project where elevation is important. I’ve had questions and wondered if my approach on that has been right. I’ve also been running QGIS master as I’m curious on how I can leverage some of the new functionality with clients and projects.

So my first whack at points looking like this in QGIS:

Floating point cloud which is like 500ish feet off the ground. At first I thought this was a feature and then I had decided it was a bug and luckily I can shoot the developers a quick message and Nyall Popped up and said “projection problem”.

So Tn’s lidar is all hosted here. I had to go back and read the metadata. What I learned is I had a horizonal projection problem as I had assumed X and I had a vertical projection problem because I didn’t think that was an issue. In pulling the LIDAR data out of the zlas format I’m pretty sure I wrecked the projection. So in going back I needed to fix it. How do I fix it? PDAL. Going back I determined that my new LAZ file was EPSG: 6576 in the horizontal and EPSG: 6360 in the vertical. So I fixed it by this bit of trickery with PDAL:

I don’t want to get into all the technical on why this worked and how I did it – except that PDAL has some amazing command line abilities for fixing LAZ/LAS files….especially the projection.

Here is the cool thing. Upon adding the new data to QGIS I was prompted that I was missing a needed transformation. QGIS is running off Proj 7. I was prompted to click a link and fix it. I blindly clicked. I blindly imported.

It worked. LIDAR matches my DEM. Proj knows the problem and QGIS fixes it. Clickity Click. Probably doesn’t happen in 100% of the cases but it worked for me.

“I HAVE TO DO ALL THIS CLICKING OMG I HAVE LIDAR I DON’T KNOW THAT MUCH ABOUT IT” you will say. I think in most cases if you have a LAS/LAZ file you’re fine. This was a special case of me having data with a screwed up Projection/No Projection in preparation for a demo. So you don’t need to know things about PDAL and things about your LIDAR. You add it and go. It’s not a bad thing to know – but overall you should be fine.

This is all pretty amazing actually. You have this small open source desktop that can render a point cloud. I think this opens the door to analysis and other things that’s usually been outside of QGIS’s normal operations. GRASS can already work with LIDAR. Now you can look at it. Oh the things you’ll do.

The Third Day of XYMas – GRASS

I know – I’m running behind. So look at these 12 days as either “hints” or “tools”….or both. Being a consultant I walk into some weird situations with data and clients – so it helps to take a full complement of tools into a situation….and a good attitude but that’s on Day 12.

So what do I give you on day 3? GRASS. Yeah and it’s the running joke EVERY.SINGLE.TIME. “I tried GRASS back in College”.

If GDAL is a swiss army knife and QGIS is my hatchet – this is my sledgehammer. I may not use it as much as I need to – but when I need it it works oh so well.

GRASS has been under some form of development since 1982. It was opened up in the mid 90’s and it’s still actively developed today. Python friendly. Command line friendly. It has it all – even a friendly gui so you can push buttons.

So what do I use it for? If you’re an active QGIS user you’ll see a lot of the tools exposed in the processing toolbox. You can gleefully play along with it there if you wish. I would encourage you to look at the tutorials the community has provided. If you’re dealing with a lot of raster data it’s pretty awesome. I’ve used it for drive time analysis with vector data. My one secret weapon with vector data has been the topology tools. If I receive some really bad vector data I’ll import it into GRASS and clean it up. Yes – for those of you that miss clean and build from your workstation days you can relive some of that excitement here.

A couple of years ago I had a call from a drone company. They wanted to leverage their raster data beyond the commercial software they were using. They wanted multiple products with multiple use cases and a lot of clients all from a server out in the “cloud”. My answer? Something really complicated involving some weird process? No – the Second worst programmer in existence (me) hammered out a process leveraging GRASS and python. We were adding and subtracting elevation models after some hefty discussion using the processes found in GRASS. which to me was somewhat poetic – this, probably the oldest piece of software I use, was working with Drones, probably the most disruptive thing to hit aerial imagery collection in forever.

Right now I’ve been using it with LIDAR data. TN freed up a lot of LIDAR data (points and a surface model) for people to work with. I have been known to download the data and mosaic it with GRASS. I’ve played with point clouds in here also. One of the best thing about GRASS is it’s rock solid. I’ve not crashed it often and I really have to do something super stupid to make it go down. If I want to toss a few hundred GB of raster data at it that’s fine as it will churn through that data without blinking.

On Day 3 – Try the Geographic Resources Analysis Support System software. The Ultimate why?

Captain Kirk said so….

Class, GRASS, and LAS – LIDAR in TN Part 3

And now for Part 3.

So something exciting happened. If you’ve been keeping up with the TN GIS Data site you’ll notice a lot more data has been released in uncomfortably HUGE zip files. I downloaded two counties worth and have been exploring the data. In the meantime I started getting a few emails of “hey we’re getting data on a hard drive”. The cool thing with that is I received a tile from a county and it’s in LAS. If you read through some of my other rants I’ve been annoyed by the fact this is in zlas format. If you do some digging on zlas it’s more or less ESRI going “Hey we’ve made a format that only we use” and you have to unlock it to make it useful by anything else. We’ll – the counties are getting unlocked data. Victory.

BUT – there’s one more Problem. In all my examples I’ve been using GRASS. So I tried to pull the data in and nothing good happened. Actually – nothing happened. It was then I started learning about liblas and the LAS Library Versions. The short story is GRASS will read the 1.2 LAS Version. The data provided by the state is in 1.4. So you can’t automatically use GRASS to do anything with the data.

EXCEPT PDAL TO THE RESCUE! Probably the one thing I’m learning is I’m not short on tools to use with LIDAR. I’m up to: GRASS, PDAL, SAGA, and LASTOOLS. I will spare you all the information on how to get PDAL – read about it on the website. For those of you on Windows: if you have QGIS from the OSGEO Network Installer, you can download PDAL painlessly and be quite happy. As I’m running Linux on everything I grabbed a copy and compiled it and have it locally. I’m also quite happy with that.

So the one thing you need to know is this: pdal translate inputlasfile outputlasfile . Issue that command from the OSGEO window. Something wonderful happens – you get a LAS file v 1.2 you can run through GRASS.

You can do things like making a map of the intensity values:

Or a hillshade for posterity:

To get from an LAS file to anything above is about 10 minutes. Maybe 12 minutes. I’m running this on a laptop with a lot of RAM – but I could do this on a much lower end machine.

I’ve covered GRASS and LAS….Oh Class! Read your metadata. If you read Part 2 above you’ll notice there is a change in Classification with the new data:

Classification	Value
Unclassified	1
Bare Earth	2
Low Noise	6
Water	9
Ignored ground	10
Bridges	17
High Noise	18

…and there – I finished my horrific joke of LAS, GRASS, and Class. I’m not explaining it.

It’s happening. LAS files for TN are out and about. My first thought is what can I do with this….my second is what are you TN Citizens going to do with it. My first two requests have been:

How do we get Tree Height?
Can we see Archaeological sites?

So – my next goal is to process A LOT of LIDAR at night. I’m going to set up a process to fix the format (unfortunately the data on the States website is still zlas), fix the library version, and then process a lot of data. Overnight? During the day? Who knows.

Georgia URISA Luncheon – January 10th 2016

LIDAR is a hot topic in Georgia these days. So – for the January 2016 Luncheon more LIDAR.

“LIDAR Updates – Changes in the Technology and Coverage”
Speaker Name: Ernie Smith

Presentation Summary: Like all Technology LiDAR has become very dynamic with new sensors and available data. We will talk about the advancements and availability of Sensors, Standards, and Data sets. There are many changes in the technology and unless you are working with it everyday, it’s difficult to keep up with all of the advancements.

Standards are also changing due to the sensor technology advancement which means practices and applications for LiDAR are growing. Also drones are being used for data collection and there are some things to be aware of for that use of Technology. We will present a map that gives our best understanding of where LiDAR data exists in Georgia as well.

Location:
Henry County Water Authority (Engineering Building)
100 Westridge Industrial Boulevard
McDonough, GA 30253