rjhale

The Seventh day of XYMas…PostGIS

rjhale · Dec 30, 2019 ·

You knew this was coming….which I should have put this in the SQL post but I really felt like it needed it’s own short article.

I was introduced to PostGIS about 6 years ago. It took me about a year to wrap my head around it. When I started using it my first inclination was “Well this isn’t much different than SDE” except I had never used SDE for anything more than data storage. Suddenly I could use SQL and muck around in the data and do things.

I enjoy postgis – I still suck at it. I sit and struggle with the simpler things (In my opinion) but it’s the most satisfying tool I use daily.

If you went back to my second day of XYMas where I said ‘Learn GDAL’ and I’m going to somewhat contradict myself and go ‘Learn Postgis’ and at the very least learn Spatial SQL. Earlier I was talking to someone who was just starting out her career in conservation and her question was “I want to learn more about GIS so what do I need to do?”. She has a wide array of tools at her command from the current offerings from ESRI to QGIS. My response was “Let’s load all your data into PostGIS/PostgreSQL and you start working with it” (that’s the one tool we don’t have loaded). Which then lead to more of a discussion on why I thought this was important. I know what will happen:

She will be slow to get started
There will be one day where she’s alone in the office and the tech bug will hit and she will start working her way into PostGIS
Suddenly you realize you’ve got a whole new world of ways to look at your data.

Anyway – yeah I know I’ve not dove into anything practical – but I actually have no idea how to do anything practical in the short space these articles take up. All I can say is learn:

The Sixth Day of XYMas…….North

rjhale · Dec 24, 2019 ·

Well – so much for wrapping this up before Christmas. I’ve got one more “general” post and then some specifics on QGIS and other things. Well – maybe not. I really don’t have much of a plan except write. in fact – writing with little to no editing.

So a few years ago I had a certification and I was looking at renewing it and people were talking about a test. There was a lot of discussion. In the middle of the discussion someone said “Well what if we make it too centered on one piece of software” and then someone else was worried “what if we make it practical” and anyway someone had the bad idea to turn to me and go “Hey what do you think?”

I said “I think you should take every GIS Person out with a map and a compass and turn them loose and if they don’t show up where they need to be they fail”.

…and that didn’t happen. As geo people we’ve turned maps into this art form to convey information while others of us have turned the information into bits and bytes and don’t care what it looks like on a piece of paper. There’s absolutely nothing wrong with that. I wished I was better doing both. I often sit and look at maps I make and wonder “Is this is usable for Navigation”. I’ll go out on a limb and say we rarely navigate with the maps we make. To be honest – why would you. We’ve got enough information on our maps it’s hard to get lost in an urban setting. What about out in the wilderness?

Way back in 1994 I won a compass for being smart or something during my BS Degree in Geology. More than likely “something” because back in the day I was a hair covered geologist. Back then you might have to make measurements and make notes on a map and possibly have to connect Point A to Point B with a pencil. I’ve rarely used the compass for much because not much long after getting this I dove into the world of computers and lost my hair and my rigged chiseled six pack abs.

For 2020 my plan is to make a map and go out with a compass and navigate around. Worry about Magnetic declination. Measure my pace. Make notes. No mobile data collection. No Maps on the web. Maybe I’ll stick a compass on my canoe and just worry about direction. Will it make me better at my job? Maybe. Will it be nice to get back to basics with a map and navigation. Oh yeah. How many of you can read contours on a map? Worry about contours! Do you put magnetic declination on a map?

So – get outside if possible and navigate around. Make some notes. Hey – you might try to make a map from scratch. Buy a compass (no clue what kind or type). Get out.

I had to look hard for someone that wasn’t a bearded-male-end-of-the-world-prepper to describe maps and compasses. Enjoy.

The Fifth Day of XYMas…..My Workstation

rjhale · Dec 20, 2019 ·

What? Hardware? You’ve only been complaining about Python for 3 out of the 4 articles….

So yeah I decided last night to talk about my computer…or at least my history of “computing” and how I ended up with this workstation. When young Randy walked out of college I’m not entirely sure that he had a computer. It wasn’t until something like 1998 where I made my first purchase of a large unwieldy desktop (Gateway). Most of my computing needs were handled at work (Federal Gov’t) where we had this mix of Sun Sparc stations, Intergraph Hardware, and some HP workstations.

When we made the move from ESRI Software on Unix machines to Windows machines I suddenly had to care about hardware and all the parts inside of a computer because the IT Department didn’t care. So we’d sit down and go “well we think we need this so lets double it” and we had some very very expensive computers back in the late 90s to early 2000s. How expensive? I can’t put my fingers on any proof but at one point we purchased a Dell Workstation that cost about $14,000 dollars. I think that was right about 2001 or 2002. We had a Silicon Graphics Workstation that I’m 100% sure cost something like $35000 in 1999 dollars.

hp workstation — I didn’t get rid of it. Vintage Dell running Windows XP from 2006

Flash forward to “Randy owning a small business” and what did I do? My first purchase was a Dell mobile workstation ($3000 pictured above). Next was a Dell lower end workstation that was about $1500. One day for fun I bought a cheap HP Laptop in late 2012. I think it was 14inch and it had a 100 GB hard drive and a whole 4GB of RAM for $350. I loaded Linux on it and started playing around in my spare time with it and trying to see what could I do GIS wise. Over time the answer was “I can actually do quite a bit in 2013 with this laptop”. Sometime that same year I started my move into a mixed environment of ESRI/FOSS4G. Sometime late in 2013 I decided to move my main workstation (Desktop) over to linux and run windows 7 in a virtual machine. That led to more experimentation.

What did I learn from all of this? FOSS4G runs light on the hardware end. I can work with cheaper equipment – not that I like or advise doing it but I can if I have to. I decided this year to go more high end than I have in the past so I bought a System76 Laptop.

What does this get me? Well – I’m looking at at least 4 to 6 years of service. I carry it out with me to clients. I might carry it into the field. I went canoeing with it once (no it’s not water proof but I didn’t feel like leaving it in the car). 32 GB of ram is a bit much – but I wanted as much as I could afford for longevity purposes. I went slim on hard drive space because more of my life is going “in the cloud” so to speak.

I’ve got another laptop with 16gb of Ram and it’s very capable – I just like the system76 one better. It’s a lenovo business class machine and it’s fine. I’ve done one thing with it and that was stick a SSd drive in it to speed it up – other than that it’s stock and boring.

What do I do when I get my laptop?

I also run whatever is current from Ubuntu. Yes – Current. I don’t do Long Term Releases except on servers and one desktop machine. Although I’m currently using Pop-OS and enjoying it (it’s based off Ubuntu).

…and I do more things – but this is a good first whack at a workstation. My assumption is when I go to a clients I have no internet. I can’t tether to my phone is my next assumption because cell service won’t be available. It’s pretty much me and what I take with me (so I take all the software). No joke – I went to a client’s location in NW Tennessee and had no phone service for 4 days (Thanks Sprint) and at least one day of 0 internet. I was at the mercy of the Hotel and the Client for internet most days and that wasn’t a given. I was left with my brain to get work done – no “hive mind” for me that day.

laptop — Stickers are the Prison Tattoo of the 21st Century

What do I tell my clients? Get what you can afford. Maybe go just a bit better than you think but this isn’t a long conversation of gaming computers vs workstation vs nvidia vs radeon vs etc etc. It’s actually nice to look at a laptop and go “If you don’t blow up you’re good for 4-6 years”. I’m all for equipment with warranties and not running hardware to death – I just don’t care a lot about hardware anymore. My current fantasy is linux on a chromebook to see what I can do with some lower end computing power where my battery runs a good part of the day. Can I have a decent workstation with the cheapest of cheapest hardware? I think so.

“Yeah but what if I run Windows?” That’s 200% fine and there is nothing wrong with it (and some days I wonder if it would make my life easier). You may need something a bit better than a business class machine but you don’t need a gaming computer to run your FOSS4G. If you’re running ESRI Software there is enough info out there on what you need – but whatever you get your FOSS4G will work just fine on it. I say that and don’t be completely ignorant about your hardware – but you don’t need to worry as much as you think you need to worry. I’m big on not worrying these days.

So anyway – Hardware. Something I don’t worry a ton about BUT it is important BUT it’s less important than having good clean data.

The Fourth Day of XYMas….SQL

rjhale · Dec 19, 2019 ·

First – I’m glad I’m doing this as it’s making me get back into the habit of posting a blog.

So I’m going to cheat a little and break up SQL and Spatial SQL. In my head it’s two things and technically maybe it’s only one. I am by no means a SQL Expert – but I wish to tell you a small story on why it is important and why I’ve been pouring more time into it.

About 6 years ago I had a turning point on data and work and the tools. I was moving out of a proprietary software life into the one I know now. I was sitting on a job south of here (in the Caribbean) and was wrestling through the world of databases. The job was centered in PostgreSQL and Postgis. I had spent until that moment letting my data be a series of “Flat” files or “Flat Containers that held data”. Pretty much all I did up until that point was Simple SQL clause with a where from a gui. Except I didn’t know it was SQL. All I knew was “push a button”.

So here I sit on a job and truly unable to function. So I started learning SQL on the job. Literally “What can I learn tonight that I will have to use tomorrow”.

Something like 6 years later I use it about every day. I’m not great at it but I’m getting better. What am I doing with it? Well – nothing grand but it’s helping me at least do some sort of quality check on my data and that’s like having a super power. I do one of two things when I get data now – i either load it into PostgreSQL/Postgis and pick at the data OR I use the virtual layers functionality in QGIS to look at it before I go as far as to stick it into QGIS.

For the purposes of this blog post – lets look at DB Manager in QGIS. If you add your shapefiles/scatch layers/whatever here – you can start to poke at it just a little.

You can throw SQL against your data layers. It’s not as powerful as having your data in a database (because I’m sure 100% of the SQL commands aren’t there) BUT – it’s a good first start at going “Hey this data isn’t as bad as I thought” or “Take this out back and burn it”.

How do you learn SQL? A book. A website. Load postgresql and start playing. My big learning experience came a few years later when I had reached the end of my rope with a client. I spent a week cleaning up their data in Postgresql. Granted I spent a ton of time searching the internet for an answer but it’s not time I’ve wasted. I learned a lot of good things. Saved a client relationship. Ended up having the best data I think I’ve ever had for a client at the end of that week.

Much like in the first day of XYMas – I think this is a skill I’ll keep using until I’m done. By done I mean the moment I run away from Geo with what’s left of my hair on fire.

The Third Day of XYMas – GRASS

rjhale · Dec 18, 2019 ·

I know – I’m running behind. So look at these 12 days as either “hints” or “tools”….or both. Being a consultant I walk into some weird situations with data and clients – so it helps to take a full complement of tools into a situation….and a good attitude but that’s on Day 12.

So what do I give you on day 3? GRASS. Yeah and it’s the running joke EVERY.SINGLE.TIME. “I tried GRASS back in College”.

If GDAL is a swiss army knife and QGIS is my hatchet – this is my sledgehammer. I may not use it as much as I need to – but when I need it it works oh so well.

GRASS has been under some form of development since 1982. It was opened up in the mid 90’s and it’s still actively developed today. Python friendly. Command line friendly. It has it all – even a friendly gui so you can push buttons.

So what do I use it for? If you’re an active QGIS user you’ll see a lot of the tools exposed in the processing toolbox. You can gleefully play along with it there if you wish. I would encourage you to look at the tutorials the community has provided. If you’re dealing with a lot of raster data it’s pretty awesome. I’ve used it for drive time analysis with vector data. My one secret weapon with vector data has been the topology tools. If I receive some really bad vector data I’ll import it into GRASS and clean it up. Yes – for those of you that miss clean and build from your workstation days you can relive some of that excitement here.

A couple of years ago I had a call from a drone company. They wanted to leverage their raster data beyond the commercial software they were using. They wanted multiple products with multiple use cases and a lot of clients all from a server out in the “cloud”. My answer? Something really complicated involving some weird process? No – the Second worst programmer in existence (me) hammered out a process leveraging GRASS and python. We were adding and subtracting elevation models after some hefty discussion using the processes found in GRASS. which to me was somewhat poetic – this, probably the oldest piece of software I use, was working with Drones, probably the most disruptive thing to hit aerial imagery collection in forever.

Right now I’ve been using it with LIDAR data. TN freed up a lot of LIDAR data (points and a surface model) for people to work with. I have been known to download the data and mosaic it with GRASS. I’ve played with point clouds in here also. One of the best thing about GRASS is it’s rock solid. I’ve not crashed it often and I really have to do something super stupid to make it go down. If I want to toss a few hundred GB of raster data at it that’s fine as it will churn through that data without blinking.

On Day 3 – Try the Geographic Resources Analysis Support System software. The Ultimate why?

Captain Kirk said so….

rjhale

Contact