Python

GIS IS NOT PYTHON

March 6, 2017 by rjhale

Got that out of the way – There – let me backtrack a little……

I started in GIS back in 1992. Maybe it was 1993. Several things started happening about the time I sat down in front of the GIS workstation to start a career:

I figured out I should have paid attention in typing class back in 1987.
I sort of liked this whole maps and polygons and data thing
I also figured out that I could write very small programs to handle repetitive tasks both in the Operating System and ArcINFO.
I also should have taken more than 1 computer class.

Now – flash forward to 2017 (I’ll use the year for posterity) and I’ve had the following conversation with three people (probably more). I’ll paraphrase:

Person1: “Yeah I’m new to GIS – I’m going to learn python so I can do GIS”
Person2: “I’m using ArcGIS – my new goal to to learn python to do more GIS”.
Person3: “I’m learning python as fast as I can so I can understand GIS”.

Probably the longest conversation I had out of the 3 conversations paraphrased:

Me: Do you know how much I’ve used Python this month?
Person: No – a lot?
Me: None.
Person: OMG WHAT?!?!

Python is everyone’s favorite. I like it. I know enough python to tear something up. I’m slightly javascript literate. I can write a shell script with some help from the google. I want to write a QGIS plugin which will be in python (at some point). I thought this morning about trying to use python to execute sql code against postgis because why not.

None of that fixes the current issue where a client has done something to a lot of data. I need to fix it. I’m not sure what I’m going to do to fix it. I have Geometry and Attribute issues. I can almost guarantee whatever I do probably won’t be in python. I’m probably going to spend a few hours talking to the client and asking “Why/What/How did we get to this point?”

Why am I having these conversations with people on programming? I remember back when ESRI started including python (ArcPy) in ArcGIS. I was pumped. I imagined this was going to be a lot like AML and things were going to be productive again. Back in the time of Avenue and AML, writing a small menu or anything to save clicks was like a caveman discovering fire. Python in ArcGIS didn’t lead to me writing a ton of scripts. I actually started leveraging more tools in ArcGIS to fix data (like model builder). Python provided that last little scratch for an itch though and it was useful. Most of my GIS Programming friends moved into different jobs and probably better pay. I remember day dreaming maybe that was my way out “Get better at programming and ascend to a better life in GIS”. Never happened for me – I’m not a programmer. I guess it’s still a thing in the GIS realm – “Programming gets you a better job in GIS”. At some point it’s turned into “Python Programming is GIS”.

If I flash back through my GIS career – oh the languages I have seen and been used by: AML, Avenue, VB, VBA, Perl (why that? long story), VB.Net, Python, and I’m waiting for the next thing. I’ve learned a lot of stuff and the only thing that has stuck longer than the rest is python. Two things have been a constant in this career since day 1: data and people. Both can be problems and both make the GIS tick along. It never changes by much. Some of the same conversations I had 20 years ago I still have on data with people: “What do you need to do and what do we need to collect to make that happen?”

So for those of you starting out in GIS:

Learn about projections
Learn about databases
Learn about data
Learn about the web
Learn about your software of choice be it ArcGIS or QGIS
Learn about LIDAR/GPS/Imagery
Somewhere in that mix of things – experience a programming language. Play with 2 of them. Maybe 3. Find one you enjoy and stick with it. You may end up making a career out of it.

Just don’t learn python thinking that is how you learn GIS. It helps – but not much.

GIS is a big world. Programming is one part of the GIS machine.

GRASS GIS Meetup – Raleigh NC November 5th 2016

October 27, 2016 by rjhale

Interested in using GRASS GIS as a geospatial processing backend? Or for reproducible research with Python? Or as a surprisingly powerful desktop GIS? Then come to the November GRASS GIS Raleigh meetup which will be specifically focused on getting newcomers started with anything ranging from using GRASS GIS to programing and contributing. The meetup is planned for Saturday, November 5. Meet in 4502 Fishbowl at the Hunt library at NC State Centennial Campus at 2 PM. Map here. Join any time during the afternoon; we will be there at least till 6 PM, but there is an option to stay longer.

Contact Vashek if interested in attending, or if you have any questions:
wenzeslaus@gmail.com

More information: https://grasswiki.osgeo.org/wiki/GRASS_GIS_Raleigh_meetups_2016

QGIS and Hyperlinks

August 21, 2016 by rjhale

It’s funny how things “come up” during the week. I had two people at random go “I wished QGIS did the hyperlink thing that Arcview does”. I hadn’t really thought about the word hyperlinks in a while.

If you ever had the joy of using ArcView 3.x you discovered hyperlinking. You had a field in your shapefile and in that field you had “C:\data\photo.jpeg” or something else. You enabled hyperlinking and then using the “lightening bolt” icon to display the photo. ArcGIS hasn’t changed much – you still have the functionality and you still have the lightening bolt icon to display documents.

QGIS handles documents a bit differently. The end result is the same though. QGIS has actions.

Actions do things. They can open a document, open a web browser, or run scripts or external programs. If you look at the main interface you’re going to see this gear icon:

If you right click a layer and go to properties, You’ll see the actions tab. There’s a button on the bottom right to create a set of default actions.

If you double click the “Open File” line you can set something up similar to a hyperlink.

If you are on windows if you just put in “open” and insert your field you are good to go. If you had a field named photo your line would look like: open [% “photo” %] . Windows will open that link with the default program. On linux I’m going to change the type of action to Unix and then actually list out the program used to open a file. For PDF’s I used atril so I put in /usr/bin/atril as opposed to open.

If you click on actions you can then open that document specified in the field…or a URL…or say you have a small custom script that you need to run on one feature when you click on it.

So you do have hyperlinks functionality but it’s a bit different. There are several videos on youtube as this feature has been around for a while. Good luck – hopefully if you’re googling hyperlinks and qgis you might run into this.

Let’s hope this week someone asks about opening a .e00 file. I’m old and this is where I shine.

Upcoming fun in QGIS 2.8: Define your own Python Functions

February 10, 2015 by rjhale

I wrestled with this blog post quite a bit. One of the spots I’ve fallen short in my GIS life has been programming. I’ve done the whole path of AML, Avenue, some dive into VB, mix in some shell scripting….then python. Toss all of that out now as history except for python. It takes me about a day to get back into python mode because it’s just a tool I use occasionally. I need to use it more – but another story for another time.

Anyone else ever type pythong?

Anyway – When ArcGIS 10 came out I started working my way through ArcPy. When QGIS 2.6 came out I decided to start working my way through plugins and the python console. My goal is to write a plugin. I’ve got an itch to scratch on a project and I think this is going to be my outlet.

I started playing with QGIS 2.8 and found something nifty in the field calculator and the expression editor: Functions. You can write your own in python. Of course no matter what I say at this point at best I’m only badly repeating what The Great Nathan has already written:

A new feature for QGIS 2.8 is a function editor for expressions. Being able to define your own custom functions has be possible for a while now, over a year, I even have a plugin (Expressions+) that adds some extra ones, however it wasn’t easy for new users and required you to create files that lived on python path while also editing the startup.py file for QGIS so that functions got registed at start up. Way too much effort for my liking.

Complete with Video From Nathan:

So as I always say – learn some python and that’s actually not a wasted thing to do. Of course as I look back I do wonder what is next “thing” I might need to learn.

…and as I always say – It’s nice to use qgis. If for nothing else because I can email the developers. I can say “HEY WHAT ABOUT…..” and they will respond back with “You’re insane” (no one has ever done that) or “Hey that’s a half decent idea that won’t wreck the project”.

Intro to QGIS class is now online!

Pyramid Layers for QGIS and ArcGIS

February 17, 2014 by rjhale

I guess I’ve been doing this way too long. I remember when pyramid layers on imagery was a bit “new” to those of us who had been plunking along with ArcINFO.

I’ve seen a few mentions in various places I’ve been hanging out about “My imagery is slow”. Invariably someone posts back “Build pyramids layers”. Of course I sorta bang my head on my desk and think “everyone should know…” BUT – not everyone does.

To thieve a bit from some ArcGIS Online help: Pyramids are reduced resolution representations of your dataset used to improve performance. Pyramids can speed up the display of raster data by retrieving only the data at a specified resolution that is required for the display. With pyramids, a lower-resolution copy of the data displays quickly when drawing the entire dataset. As you zoom in, levels with finer resolutions are drawn; performance is maintained because you’re drawing successively smaller areas.

So with ArcGIS pyramids are pretty much almost a default action. You can set to automatically build pyramids when you add an image. For instance today I was dealing with a 11 GB tif. Of course as I normally do I got a bit adventurous and did a few different things to build pyramid layers. Why? Because anyone can. Most of NRGS’s clients don’t deal with server environment. So if you deal with a lot of imagery it can get to be a hassle. You need to know about pyramid layers.

In this example we deal with ArcGIS and QGIS. Technically we’re dealing with ArcGIS and gdal. If you ever wondered if the pyramid file (.ovr) was viewable in both ArcGIS and QGIS – it is. Build it once and either can read it. There are three ways to go about building pyramid layers.

in ArcGIS: Proceed to ArcToolbox and go to Data management -> Raster -> Raster Properties. You can build pyramids there. ArcGIS has several defaults set. There’s several tools – one of which while build an entire directory of imagery.

in QGIS: Go to Raster -> Miscellaneous -> Build Pyramids. You’ve got a few more options “exposed” with this menu. Once again you can fill out all the particulars and let it build away. You even have an option to “clean” the pyramids which deletes them. You also have a nice little check box for processing an entire directory.

So the interesting thing with the QGIS pyramid layer is the text box at the bottom. The pyramid building in QGIS is gdal. It’s the gdaladdo command specifically. If you notice at the bottom you have a command starting to take shape. So if you get fairly adventurous the entire command starts looking like:

gdaladdo -r average -ro –config COMPRESS_OVERVIEW JPEG –config USE_RRD NO –config JPEG_QUALITY 80 –config TILED YES –config PHOTOMETRIC_OVERVIEW YCBCR –config GDAL_CACHEMAX 10240 ortho_1-1_1n_s_al029_2013_1.tif 2 4 8 16

My only addition to that command is the –config GDAL_CACHEMAX 10240 portion. I’m telling gdal to use 10Gb of RAM to speed up processing. I run that from either a CMD tool or a shell.

Three ways to build pyramids. Why? Well – if you have a copy of ArcGIS do you want to tie it up building pyramids? Yes – you can create a separate process but why not leverage a another computer if you have one. If you have two machines and one license of ArcGIS….load QGIS. Load gdal on the server. With just a small amount of mix and match you’ve got more options and *hopefully* get slightly more flexible. Perhaps you pursue that elusive batch command that will run when you are at home. Like I said in the beginning – the pyramid file is usable by both ArcGIS and QGIS. So build it once and use it multiple places. Of course all of this is negated a bit if you have a image server. That’s another story for another time.

And with that – yes – I just babbled for one page about building pyramids layers for a raster file.