processing Archives • North River Geographic Systems Inc

So at least once a week I’m getting a phone call or an email on QGIS. This one was pretty awesome and it pointed out I need to cover geoprocessing in Class in a little more detail.

The question was “In our transition out of ESRI products we hit a snag. We want to merge or dissolve 925,000 polygons on an attribute. QGIS Hangs. Arcmap Completes it. Why doesn’t QGIS complete the task?” They took an abbreviated version of the QGIS class at FOSS4GNA back in the spring.

I thought about it and started running some tests. You can dissolve/merge the data two ways in QGIS and you can do it in PostGIS. ArcGIS people tend to get hung up on this as you have one way in ArcMap. So I listed out the three ways from Simplest to hardest (Finals: GRASS won for speediest answer, PostGIS won for cleanest answer, QGIS won for simplest answer).

The Question brought up a good point. When you think about “geoprocessing” if you’re an ESRI user you drop back to ArcToolbox.

If you think of Geoprocessing in QGIS you actually can get two “correct” answers. One deals with FWTools and one deals with the Toolbox. Yes – both QGIS and ESRI have Toolboxes. Before QGIS 2.14 you had FWtools in the vector menu. That software was there from the beginning or close and is derived from http://fwtools.maptools.org/ (FW is Frank Warmerdam). Sextante started making an appearance sometime in 2.0 as a plugin. Sextante combined a lot of tools into one menu system. Eventually it made it was as a part of the core release and became the Processing Toolbox. FWTools started getting out of date. Somewhere in 2.12 or earlier it was decided to replace FWTOOls with the Processing Toolbox

The fun is at the 2.14 release. It’s a Long term release. If you’re running an earlier version you will still see FWTools. As I’m sitting here looking at 2.14.6 It appears FWTools is gone. In 2.16 the vector menu at the top has regained all the functionality you remember from 2.1 or early versions of 2.14 – it just looks different. You now have unified processing tools in 2.16. No FWTools.

Back to the problem. The student from class was running 2.14.2. They still had FWTools and “googling” pointed them to the answer being in the FWTools toolset under vector. The merge didn’t work. The Dissolve in the processing toolbox finished in about 20 minutes. So my followup email in a few days will be – Did it work and do you feel better? Are you really dropping the other stuff? I’ve yet to suggest people ditch their commercial GIS system. I do have more and more people starting from scratch and going open source. It’s interesting.

The moral to the story – QGIS is evolving. If you decide “I’m going to sit at this one version for a year” you are missing out. You should at least think of upgrading like you would an ESRI Service pack. If you are on 2.14 – upgrade to the current LTR (Long Term Release) release (which is 2.14.6 as of this date of posting). If you want to live on the edge – go with 2.16 (I did but I’m weird that way).

There are days when work slows – today was one of those days. So I can either go goof off or learn something. I decided to learn something.

Back when I worked for the Federal Gov’t I was somewhat in charge of creating ortho imagery. At the time (i don’t even want to say when this was) we either grabbed whatever elevation data was available or we built the data in the photogrammetry department. About the time I had decided to leave LIDAR was coming down in price and we had just started the jump into that. Flash forward 8 years later and LIDAR is pretty common place. They have hardware small enough for drones and there are myriad ways to get data. You can even get data freely available from the Federal Government.

I know – it’s not LIDAR but it’s what I think of every time I say Drone. Stop making fun of Drones! No.

I’ve numerous and myriad clients and sometimes I wonder what I can do to make their lives better and expand available services. I also watch the twitters and I saw a workshop occurring just south of my location using ArcGIS and an extension to process LIDAR. So once again I was wondering about LIDAR. Can I process it? I know I have enough tools at my disposal – like GRASS.

I was recommended the NOAA Website to look for data. I did some digging. Guess what? LIDAR in my area is freely available now.

Tennessee has a very advanced GIS program. It started sometime in the mid-90’s and has taken a very encompassing view of GIS. They work at a state level for planning and coordination. The killer for me is most of the data is locked up at the counties or is for sale. It’s rare for me to convince a client to purchase any data – even data where we could really do something good with it. Counties or Gov’t agencies within the state used to take the view of “we paid for it and so do you”. Some still do. Tennessee released two counties worth or LIDAR and it appears a lot of the state is going into the National Map. So maybe things are changing for the better.

The TN GIS Server has the data. It’s small sets so I’m going to have to do a few things to grab it all. For testing purposes I only need a tile. I need a tile with Hardwood and Pine Plantations (yes – Forestry – I know what I want to test now).

TN has a very advanced program – but it’s all ESRI. So when I downloaded the LIDAR data it’s in zlas format. In case you haven’t kept up there’s been a little bit of a format war occurring. ESRI pushes one format called zlas which is (my opinion) ripped off of laszip. A lot of other people are using laszip. The short story is laszip is open – zlas isn’t. If you have ESRI software and you’ve bought the right extensions someone will tell you zlas. If you’re operating outside of that realm or are considering lidar – laszip. It should be laszip anyway – life is to short for locked up data formats. It’s not entirely locked up – I can build a tool to unlock it. I’ll get to that after I build a toaster to make toast.

So I want to do this one thing here – Tennessee OIR (or whoever) – Thank you for releasing this data. This is HUGE. The state sits on a treasure trove of data that it should open and this is a beautiful first step. Open up the format. Use laszip. If people want to go zlas they will – please don’t make your users start off having to deal with a proprietary format. I know you have a contract with ESRI. That’s fine – just realize there are more options than that one software vendor out and about.

So what do you do with presented with zlas? You open it up. For this I ripped zlas to las. RapidLasso made a toaster (of sorts) and are giving it away. In other words I took a 235mb file and turned it into a 1.3 GB uncompressed open format.

….and what good does that do? Well – part 2 is coming. Right now I’m watching GRASS gleefully process this data and I’m not sure what is going to happen. GRASS You say again? Not QGIS? You mean that really confusing horrible interfaced piece of software I once used in College. No. It’s the revamped new GUI interface that has 30 years of development behind it and it has all these really great tools for LIDAR. Yes – ye who have a limited budget and possibly can’t afford the flashy stuff have options. You’ve got a whole community of options. One being GRASS that you get when you download QGIS.

Selection_312 — Point Cloud in GRASS – total point count is “A LOT”

I know it’s orange. I actually hate orange but GRASS defaulted and I don’t want to spend that much time making it not orange – this is Tennessee after all.

processing

Geoprocessing in QGIS: 2.14 vs 2.16

Adventures in LIDAR and GRASS in Tennessee: Part 1

Contact