North River Geographic Systems Inc

Geospatial Problem Solving

Imagery

The Third Day of XYMas – GRASS

I know – I’m running behind. So look at these 12 days as either “hints” or “tools”….or both.  Being a consultant I walk into some weird situations with data and clients – so it helps to take a full complement of tools into a situation….and a good attitude but that’s on Day 12.

So what do I give you on day 3? GRASS.  Yeah and it’s the running joke EVERY.SINGLE.TIME. “I tried GRASS back in College”.

If GDAL is a swiss army knife and QGIS is my hatchet – this is my sledgehammer. I may not use it as much as I need to – but when I need it it works oh so well.

grass
Don’t worry it’s got a GUI

GRASS has been under some form of development since 1982. It was opened up in the mid 90’s and it’s still actively developed today. Python friendly. Command line friendly. It has it all – even a friendly gui so you can push buttons.

 

So what do I use it for? If you’re an active QGIS user you’ll see a lot of the tools exposed in the processing toolbox. You can gleefully play along with it there if you wish. I would encourage you to look at the tutorials the community has provided. If you’re dealing with a lot of raster data it’s pretty awesome. I’ve used it for drive time analysis with vector data. My one secret weapon with vector data has been the topology tools. If I receive some really bad vector data I’ll import it into GRASS and clean it up. Yes – for those of you that miss clean and build from your workstation days you can relive some of that excitement here.

A couple of years ago I had a call from a drone company. They wanted to leverage their raster data beyond the commercial software they were using. They wanted multiple products with multiple use cases and a lot of clients all from a server out in the “cloud”. My answer? Something really complicated involving some weird process? No – the Second worst programmer in existence (me) hammered out a process leveraging GRASS and python. We were adding and subtracting elevation models after some hefty discussion using the processes found in GRASS. which to me was somewhat poetic – this, probably the oldest piece of software I use, was working with Drones, probably the most disruptive thing to hit aerial imagery collection in forever.

Right now I’ve been using it with LIDAR data. TN freed up a lot of LIDAR data (points and a surface model) for people to work with. I have been known to download the data and mosaic it with GRASS. I’ve played with point clouds in here also. One of the best thing about GRASS is it’s rock solid. I’ve not crashed it often and I really have to do something super stupid to make it go down. If I want to toss a few hundred GB of raster data at it that’s fine as it will churn through that data without blinking.

On Day 3 – Try the Geographic Resources Analysis Support System software. The Ultimate why?

Captain Kirk said so….

 

 

 

Hard Work

Sunset
Sunset Above the Clouds

“An idea I live by comes from a John Wayne movie called Rio Lobo.  John Wayne and Jack Elam are on one side of a crick shooting their guns at the bad guys on the other side.  John Wayne looks down at Jack Elam and says “Scatter gun’s useless.”  Jack Elam says “Don’t mind if I shoot, do you? It just makes me feel better.”

“I love that attitude.  If you don’t shoot, you have no chance of killing the bad guys on the other side of the river.  If you don’t shoot, you have no chance of making a basket.  You have to be willing to try.

“I balance that with an idea I take from Clint Eastwood’s movie Magnum Force.  Toward the end of the movie, Clint Eastwood squares off with a dirty cop who’d tried to kill him.  He secretly activates a bomb in the cop’s car and as the cop drives off his car blows up.  Clint Eastwood says “Man’s gotta know his limitations.”

“That’s the other thing I think about when I’m coaching.  Everybody should understand what they can and cannot do.”

Roy Williams, Hard Work: A life on and off the court

For those of you who do not know, I was born and raised a Tarheel fan and that definitely gets me in trouble this time of year living in Kentucky.  I may bleed the wrong color blue (powder instead of big), but at least once a year and often twice one of my favorite teams has to lose.  But these words of wisdom comes from a coach who just earned the title of fastest to 300 wins in the ACC and they seem to be particularly applicable to me the past few days.  Plus, Coach Williams is fiercely competitive which is a trait I can relate to entirely too well.  Tonight, I am hoping my fellow Kentuckians will forgive my “incorrect” powder blue preferences and unite with me to beat the true enemy.  Go Heels.  Beat Dook!!!!

SnowLine
Snow Line
SnowyFarms
Snowy Horse Farms at Dusk

I first heard this quote (thanks to Audible audiobooks) as I drove on snow-covered roads to the airport to attempt to do some remote sensing.  I wasn’t particularly happy about this outing because if you think about the percentage of land that had good remote sensing conditions last weekend is incredibly low.  And Kentucky had just received some more snow.  So much snow that our first attempt of flying Saturday had been thwarted completely.  I was totally skeptical and ready to turn around and go back to bed.  But, after about an hour of scheming, we took off.  We almost had to go back, though, because I didn’t have video so I couldn’t tell if the camera door was frozen shut or if the vide was just being finicky.  Scary and embarrassing at the same time.  But, I managed to get things working and it turned out to be a pretty successful day of 20 lines spread across four states (AR, TN, MS, and KY) and 395 images.  (It has been QC-ed and everything was accepted!!!)

SunsetProp
Sunset on the propeller

If I had gone with my original gut feeling and not shot the imagery, I would have been really frustrated.  Not to mention the fact that I probably would have been driving back on roads that were still nasty from the snow we received overnight.  Instead, I worked.  When I drove home that night, the roads were nearly clear and we had several hundred images “in the can” so to speak.  Productivity! And, along with that, we were smart and knew our limitations on Saturday.  Yes, we probably missed some photography but we were safe.  And, yes, I love my job.  Writing very detailed FGDC Remote Sensing Extension metadata one day and flying the data the next!

But this quote is one I am going to try to take to heart in more ways than just remote sensing because I truly believe that hard work has the potential to pay off in all aspects of our lives.

Pyramid Layers for QGIS and ArcGIS

I guess I’ve been doing this way too long. I remember when pyramid layers on imagery was a bit “new” to those of us who had been plunking along with ArcINFO.

I’ve seen a few mentions in various places I’ve been hanging out about “My imagery is slow”. Invariably someone posts back “Build pyramids layers”. Of course I sorta bang my head on my desk and think “everyone should know…” BUT – not everyone does.

To thieve a bit from some ArcGIS Online help: Pyramids are reduced resolution representations of your dataset used to improve performance. Pyramids can speed up the display of raster data by retrieving only the data at a specified resolution that is required for the display. With pyramids, a lower-resolution copy of the data displays quickly when drawing the entire dataset. As you zoom in, levels with finer resolutions are drawn; performance is maintained because you’re drawing successively smaller areas.

So with ArcGIS pyramids are pretty much almost a default action. You can set to automatically build pyramids when you add an image. For instance today I was dealing with a 11 GB tif. Of course as I normally do I got a bit adventurous and did a few different things to build pyramid layers. Why? Because anyone can. Most of NRGS’s clients don’t deal with server environment. So if you deal with a lot of imagery it can get to be a hassle. You need to know about pyramid layers.

In this example we deal with ArcGIS and QGIS. Technically we’re dealing with ArcGIS and gdal. If you ever wondered if the pyramid file (.ovr) was viewable in both ArcGIS and QGIS – it is. Build it once and either can read it. There are three ways to go about building pyramid layers.

  • in ArcGIS: Proceed to ArcToolbox and go to Data management -> Raster -> Raster Properties. You can build pyramids there. ArcGIS has several defaults set. There’s several tools – one of which while build an entire directory of imagery.

arcgis_pyramids

  • in QGIS: Go to Raster -> Miscellaneous -> Build Pyramids. You’ve got a few more options “exposed” with this menu. Once again you can fill out all the particulars and let it build away. You even have an option to “clean” the pyramids which deletes them. You also have a nice little check box for processing an entire directory.

qgis_pyramids

So the interesting thing with the QGIS pyramid layer is the text box at the bottom. The pyramid building in QGIS is gdal. It’s the gdaladdo command specifically. If you notice at the bottom you have a command starting to take shape. So if you get fairly adventurous the entire command starts looking like:

gdaladdo -r average -ro –config COMPRESS_OVERVIEW JPEG –config USE_RRD NO –config JPEG_QUALITY 80 –config TILED YES –config PHOTOMETRIC_OVERVIEW YCBCR –config GDAL_CACHEMAX 10240 ortho_1-1_1n_s_al029_2013_1.tif 2 4 8 16

My only addition to that command is the –config GDAL_CACHEMAX 10240 portion. I’m telling gdal to use 10Gb of RAM to speed up processing. I run that from either a CMD tool or a shell.

Three ways to build pyramids. Why? Well – if you have a copy of ArcGIS do you want to tie it up building pyramids? Yes – you can create a separate process but why not leverage a another computer if you have one. If you have two machines and one license of ArcGIS….load QGIS. Load gdal on the server. With just a small amount of mix and match you’ve got more options and *hopefully* get slightly more flexible. Perhaps you pursue that elusive batch command that will run when you are at home. Like I said in the beginning – the pyramid file is usable by both ArcGIS and QGIS. So build it once and use it multiple places. Of course all of this is negated a bit if you have a image server. That’s another story for another time.

And with that – yes – I just babbled for one page about building pyramids layers for a raster file.

OpenGeo Suite 4.0.2

So I’m not even sure how I’m going to wrap this post up quickly and efficiently. Here goes…..

We’ve a client that needs a “web map” – so it’s led us to a ton of digging and learning. Tonights fun was the OpenGeo suite. The best way to describe the OpenGeo suite from Boundless Geo is to describe it as ArcGIS Server with an open source accent. It runs on windows, linux, mac, and probably something else if you can find it.

With OpenGeo you get four pieces software:

  • GeoServer – serves the data
  • Postgresql/PostGIS – stores the data
  • Geowebcache – tile your raster data
  • GeoExplorer – display your data

It took me about an hour to create my first map published out of the suite – most of that time was spent reading the documentation. I uploaded a shapefile and here is what I ended up with.

Screenshot - 01262014 - 09:48:35 PM

Earlier in 2013 Boundless had announced support for QGIS. From all of the blog posts and classes you can guess we’re fans. There’s also a plugin for the OpenGeo Suite called OpenGeo explorer

Screenshot - 01262014 - 09:20:12 PM

With this plugin you you can publish data directly from QGIS into the OpenGeo suite. Actually – if I read the docs correctly you can publish into any Geoserver install (so you have some flexibility).

Just to wrap all this up:

  • Create data in QGIS
  • Publish data with symbology into Postgis/GeoServer
  • Data can then be viewed using GeoExplorer and shared with others.
  • Data can be pushed out into other application from GeoServer.
  • Any software supporting Postgresql/PostGIS can access any stored data

There are a lot of GIS solutions currently and there are more appearing everyday. OpenGeo wraps 4 packages up into an interconnected server product that give you some powerful options for a GIS program. Plus you’ve got a desktop hook

Overall – it’s worth your time to read up on their offerings.

Hollywood can do it why can’t you….

“Hollywood can do it why can’t you?” was the sentence that about put me over the edge back in 2000. I had inherited a “position” where I was going to be the “ortho guy”. Essentially the guy that took 9×9 scanned aerial prints and made orthoimagery out of them. Correct scaling issues and incorporate an elevation model. We had a scanner, we had software, and I had a slight idea of how it worked. Not a great idea so that’s why you had support, manuals, and a list of phone numbers of people I could call and go “I don’t know why but the RC30 Camera report hates  Fiducial 4”. I had one job where I was just trying to piece together some imagery and it wouldn’t pull together. I had a bad survey point and one of the bosses said “”Hollywood can do it why can’t you? Just put it all together and “fly” over it. Like Hollywood…”

Anyway – yes – People used to strap cameras with film in them to planes to get aerial photography.

I saw two things that made me remember that stupid story over the last month or so. This article kicked off the thought train.:

Planet Labs: They are tossing out a “flock” of satellites from the International Space Station to image the earth. What they lack in pixel resolution they make up for in timeliness. Almost constant earth monitoring from these small satellites. Brand new imagery. Constantly. I can only imagine the headache of storage.

Dove_nest_2

 

 

 

 

Sky Box Imaging: Video from a satellite. That’s all I can say on that one. Video. Imagine overlaying your data on video.

So yeah – Hollywood.