North River Geographic Systems Inc

Spatial Problem Solving

rjhale

The Sixth Day of XYMas…….North

· ·

Well – so much for wrapping this up before Christmas. I’ve got one more “general” post and then some specifics on QGIS and other things. Well – maybe not. I really don’t have much of a plan except write. in fact – writing with little to no editing.

So a few years ago I had a certification and I was looking at renewing it and people were talking about a test. There was a lot of discussion. In the middle of the  discussion someone said “Well what if we make it too centered on one piece of software” and then someone else was worried “what if we make it practical” and anyway someone had the bad idea to turn to me and go “Hey what do you think?”

I said “I think you should take every GIS Person out with a map and a compass and turn them loose and if they don’t show up where they need to be they fail”.

<crickets>

…and that didn’t happen. As  geo people we’ve turned maps into this art form to convey information while others of us have turned the information into bits and bytes and don’t care what it looks like on a piece of paper. There’s absolutely nothing wrong with that. I wished I was better doing both. I often sit and look at maps I make and wonder “Is this is usable for Navigation”. I’ll go out on a limb and say we rarely navigate with the maps we make. To be honest – why would you. We’ve got enough information on our maps it’s hard to get lost in an urban setting. What about out in the wilderness?

Way back in 1994 I won a compass for being smart or something during my BS Degree in Geology. More than likely “something” because back in the day I was a hair covered geologist. Back then you might have to make measurements and make notes on a map and possibly have to connect Point A to Point B with a pencil. I’ve rarely used the compass for much because not much long after getting this I dove into the world of computers and lost my hair and my rigged chiseled six pack abs.

For 2020 my plan is to make a map and go out with a compass and navigate around. Worry about Magnetic declination. Measure my pace. Make notes. No mobile data collection. No Maps on the web. Maybe I’ll stick a compass on my canoe and just worry about direction. Will it make me better at my job? Maybe. Will it be nice to get back to basics with a map and navigation. Oh yeah. How many of you can read contours on a map? Worry about contours! Do you put magnetic declination on a map?

So – get outside if possible and navigate around. Make some notes. Hey – you might try to make a map from scratch. Buy a compass (no clue what kind or type). Get out.

I had to look hard for someone that wasn’t a bearded-male-end-of-the-world-prepper to describe maps and compasses. Enjoy.

 

 

 

The Fifth Day of XYMas…..My Workstation

· ·

What? Hardware? You’ve only been complaining about Python for 3 out of the 4 articles….

So yeah I decided last night to talk about my computer…or at least my history of “computing” and how I ended up with this workstation. When young Randy walked out of college I’m not entirely sure that he had a computer.  It wasn’t until something like 1998 where I made my first purchase of a large unwieldy desktop (Gateway). Most of my computing needs were handled at work (Federal Gov’t) where we had this mix of Sun Sparc stations, Intergraph Hardware, and some HP workstations.

When we made the move from ESRI Software on Unix machines to Windows machines I suddenly had to care about hardware and all the parts inside of a computer because the IT Department didn’t care. So we’d sit down and go “well we think we need this so lets double it” and we had some very very expensive computers back in the late 90s to early 2000s. How expensive? I can’t put my fingers on any proof but at one point we purchased a Dell Workstation that cost about $14,000 dollars. I think that was right about 2001 or 2002. We had a Silicon Graphics Workstation that I’m 100% sure cost something like $35000 in 1999 dollars.

hp workstation
I didn’t get rid of it. Vintage Dell running Windows XP from 2006

Flash forward to “Randy owning a small business” and what did I do? My first purchase was a Dell mobile workstation ($3000 pictured above). Next was a Dell lower end workstation that was about $1500. One day for fun I bought a cheap HP Laptop in late 2012. I think it was 14inch and it had a 100 GB hard drive and a whole 4GB of RAM for $350. I loaded Linux on it and started playing around in my spare time with it and trying to see what could I do GIS wise. Over time the answer was “I can actually do quite a bit in 2013 with this laptop”.  Sometime that same year I started my move into a mixed environment of ESRI/FOSS4G. Sometime late in 2013 I decided to move my main workstation (Desktop) over to linux and run windows 7 in a virtual machine. That led to more experimentation.

What did I learn from all of this? FOSS4G runs light on the hardware end. I can work with cheaper equipment – not that I like or advise doing it but I can if I have to. I decided this year to go more high end than I have in the past so I bought a System76 Laptop.

What does this get me? Well – I’m looking at at least 4 to 6 years of service. I carry it out with me to clients. I might carry it into the field. I went canoeing with it once (no it’s not water proof but I didn’t feel like leaving it in the car). 32 GB of ram is a bit much – but I wanted as much as I could afford for longevity purposes. I went slim on hard drive space because more of my life is going “in the cloud” so to speak.

I’ve got another laptop with 16gb of Ram and it’s very capable – I just like the system76 one better. It’s a lenovo business class machine and it’s fine. I’ve done one thing with it and that was stick a SSd drive in it to speed it up – other than that it’s stock and boring.

What do I do when I get my laptop?

I also run whatever is current from Ubuntu. Yes – Current. I don’t do Long Term Releases except on servers and one desktop machine. Although I’m currently using Pop-OS and enjoying it (it’s based off Ubuntu).

…and I do more things – but this is a good first whack at a workstation. My assumption is when I go to a clients I have no internet. I can’t tether to my phone is my next assumption because cell service won’t be available. It’s pretty much me and what I take with me (so I take all the software). No joke – I went to a client’s location in NW Tennessee and had no phone service for 4 days (Thanks Sprint) and at least one day of 0 internet. I was at the mercy of the Hotel and the Client for internet most days and that wasn’t a given. I was left with my brain to get work done – no “hive mind” for me that day.

laptop
Stickers are the Prison Tattoo of the 21st Century

What do I tell my clients? Get what you can afford. Maybe go just a bit better than you think but this isn’t a long conversation of gaming computers vs workstation vs nvidia vs radeon vs etc etc. It’s actually nice to look at a laptop and go “If you don’t blow up you’re good for 4-6 years”.  I’m all for equipment with warranties and not running hardware to death – I just don’t care a lot about hardware anymore. My current fantasy is linux on a chromebook to see what I can do with some lower end computing power where my battery runs a good part of the day. Can I have a decent workstation with the cheapest of cheapest hardware? I think so.

“Yeah but what if I run Windows?” That’s 200% fine and there is nothing wrong with it (and some days I wonder if it would make my life easier). You may need something a bit better than a business class machine but you don’t need a gaming computer to run your FOSS4G. If you’re running ESRI Software there is enough info out there on what you need – but whatever you get your FOSS4G will work just fine on it. I say that and don’t be completely ignorant about your hardware – but you don’t need to worry as much as you think you need to worry. I’m big on not worrying these days.

So anyway – Hardware. Something I don’t worry a ton about BUT it is important BUT it’s less important than having good clean data.

 

The Fourth Day of XYMas….SQL

· ·

First – I’m glad I’m doing this as it’s making me get back into the habit of posting a blog.

So I’m going to cheat a little and break up SQL and Spatial SQL. In my head it’s two things and technically maybe it’s only one. I am by no means a SQL Expert – but I wish to tell you a small story on why it is important and why I’ve been pouring more time into it.

About 6 years ago I had a turning point on data and work and the tools. I was moving out of a proprietary software life into the one I know now. I was sitting on a job south of here (in the Caribbean) and was wrestling through the world of databases. The job was centered in PostgreSQL and Postgis. I had spent until that moment letting my data be a series of “Flat” files or “Flat Containers that held data”. Pretty much all I did up until that point was Simple SQL clause with a where from a gui. Except I didn’t know it was SQL. All I knew was “push a button”.

So here I sit on a job and truly unable to function. So I started learning SQL on the job. Literally “What can I learn tonight that I will have to use tomorrow”.

Something like 6 years later I use it about every day. I’m not great at it but I’m getting better. What am I doing with it? Well – nothing grand but it’s helping me at least do some sort of quality check on my data and that’s like having a super power. I do one of two things when I get data now – i either load it into PostgreSQL/Postgis and pick at the data OR I use the virtual layers functionality in QGIS to look at it before I go as far as to stick it into QGIS.

For the purposes of this blog post – lets look at DB Manager in QGIS. If you add your shapefiles/scatch layers/whatever here – you can start to poke at it just a little.

You can throw SQL against your data layers. It’s not as powerful as having your data in a database (because I’m sure 100% of the SQL commands aren’t there) BUT – it’s a good first start at going “Hey this data isn’t as bad as I thought” or “Take this out back and burn it”.

How do you learn SQL? A book. A website. Load postgresql and start playing. My big learning experience came a few years later when I had reached the end of my rope with a client. I spent a week cleaning up their data in Postgresql. Granted I spent a ton of time searching the internet for an answer but it’s not time I’ve wasted. I learned a lot of good things. Saved a client relationship. Ended up having the best data I think I’ve ever had for a client at the end of that week.

Much like in the first day of XYMas – I think this is a skill I’ll keep using until I’m done. By done I mean the moment I run away from Geo with what’s left of my hair on fire.

The Third Day of XYMas – GRASS

· ·

I know – I’m running behind. So look at these 12 days as either “hints” or “tools”….or both.  Being a consultant I walk into some weird situations with data and clients – so it helps to take a full complement of tools into a situation….and a good attitude but that’s on Day 12.

So what do I give you on day 3? GRASS.  Yeah and it’s the running joke EVERY.SINGLE.TIME. “I tried GRASS back in College”.

If GDAL is a swiss army knife and QGIS is my hatchet – this is my sledgehammer. I may not use it as much as I need to – but when I need it it works oh so well.

grass
Don’t worry it’s got a GUI

GRASS has been under some form of development since 1982. It was opened up in the mid 90’s and it’s still actively developed today. Python friendly. Command line friendly. It has it all – even a friendly gui so you can push buttons.

 

So what do I use it for? If you’re an active QGIS user you’ll see a lot of the tools exposed in the processing toolbox. You can gleefully play along with it there if you wish. I would encourage you to look at the tutorials the community has provided. If you’re dealing with a lot of raster data it’s pretty awesome. I’ve used it for drive time analysis with vector data. My one secret weapon with vector data has been the topology tools. If I receive some really bad vector data I’ll import it into GRASS and clean it up. Yes – for those of you that miss clean and build from your workstation days you can relive some of that excitement here.

A couple of years ago I had a call from a drone company. They wanted to leverage their raster data beyond the commercial software they were using. They wanted multiple products with multiple use cases and a lot of clients all from a server out in the “cloud”. My answer? Something really complicated involving some weird process? No – the Second worst programmer in existence (me) hammered out a process leveraging GRASS and python. We were adding and subtracting elevation models after some hefty discussion using the processes found in GRASS. which to me was somewhat poetic – this, probably the oldest piece of software I use, was working with Drones, probably the most disruptive thing to hit aerial imagery collection in forever.

Right now I’ve been using it with LIDAR data. TN freed up a lot of LIDAR data (points and a surface model) for people to work with. I have been known to download the data and mosaic it with GRASS. I’ve played with point clouds in here also. One of the best thing about GRASS is it’s rock solid. I’ve not crashed it often and I really have to do something super stupid to make it go down. If I want to toss a few hundred GB of raster data at it that’s fine as it will churn through that data without blinking.

On Day 3 – Try the Geographic Resources Analysis Support System software. The Ultimate why?

Captain Kirk said so….

 

 

 

The Second Day of XYMas….GDAL

· ·

So going back to the college kids and the “OMG I gotta learn python”. It actually it helps if you do learn python. It will help your career and it’s always nice to know how to program in something….

BUT

I will submit on this the second day of XYMas that you should learn GDAL before you dive into python if you’re interests are centered around location. GDAL is the underpinning of a lot of GIS Software (ArcGIS and QGIS just to list 2). It’s command line based (See the previous post) and it’s all powerful as you can manipulate vector data and raster data and there are a ton of resources for using it with python out on the internet. No license required. All you need is a strong heart and nimble fingers. Remember – GDAL tools work on raster and OGR tools work on vector. That’s lesson 1.

As my small working example or why this knowledge will serve you well:

  1. Lets download Significant Earthquakes from the USGS as a csv file: https://earthquake.usgs.gov/earthquakes/feed/v1.0/csv.php
  2. I’m going to use ogr2ogr to move it into something other than csv:  ogr2ogr -f GPKG equakes.gpkg significant_month.csv -oo X_POSSIBLE_NAMES=longitude -oo Y_POSSIBLE_NAMES=latitude -a_srs ‘EPSG:4326’
  3. I’m now going to query all the earthquakes > than a 5.0 from GDAL: ogrinfo equakes.gpkg -dialect sqlite -sql “select mag from significant_month where mag > 5.0 order by mag desc”

For fun I’m going to drop it into QGIS just to see everything (this was pre-New Zealand Volcanic Eruption):

QGIS 3.10

There are a few things I could do to make this better – like making sure the magnitude is a number field and not a string but for the example it works.

Last year I had a client suddenly surprise me with a “Hey – we gotta deliver some data to the state and it has to be in an ESRI acceptable format”. Which – no big deal except I had spent a good deal of time moving them to postgis to get away from ESRI. A year ago we set up a script and it’s been running for 15 months or so. We’ve upgraded software – we’ve changed servers. With a few modifications it’s still running.

ogr2ogr -f GPKG TCStransport.gpkg -nlt point PG:”host=host user=user dbname=dbname password=pass” -nln “addresspoints” -sql “select id, oirid, geom, r_segid, a_segid, seg_side, gislink, cast(structype as SMALLINT), strucdesc, stnum_h, stnum_l, stnum, stnumsuf, building, floor, unit_type, unit_num, secuntnum, predir, pretype, name, type, sufdir, postmod, address, addr_esn, label, subname, vanity, zip, zip4, esn, city, county, state, cast(source as SMALLINT), lon, lat, x_sp, y_sp, z_val, gpsdate, addrauth, editor, geomod, geosrce, geodate, attmod, attsrce, attdate, cast(status as SMALLINT), delnotes, right_left, comment, readdress, old_address, color, point_type, street, cad_name, loc_note, cad_post from addresspoints”

Think about that – a small program that I (2nd worst programmer in the known universe) wrote that has now run through 1 server upgrade and 3 software upgrades without breaking. Every night for over a year. I should have set some sort of $3.50 a day payment plan based on success.

….and I’ve not even discussed the work you can do with Raster data. Actually – lets discuss the GDAL part. I uncompressed a SID image and needed it for a client – so I ran this to compress it back to something normal sized:

gdal_translate -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -tr 1 1 -mask 3 ortho_1-1_hc_s_tn065_2018_1.tif NAIP_2018.tif

Could I have added more “things” to make it better/smaller? Probably – but it worked. So I can take any sort of image file and un-compress, compress, change the format, reproject, etc……All from GDAL. It works with everything including ArcMap/Pro/Whatever it’s called.

NO LICENSE REQUIRED. IT WORKS WITH PYTHON FOR YOU PEOPLE WHO WANT TO LOVE PYTHON.

For this second day of XYMAS I point you to:

So I do ask you to learn GDAL and then move into python. It’s great software. It’s an active community. You will not waste your time learning this skill – if you’re going to stick around the GEO world you’re going to touch it at some point (or you should – if you don’t I have questions on why).

 

Contact