North River Geographic Systems Inc

Spatial Problem Solving

QGIS

Prayers and Hay

· ·

I know – it’s been a while. I’ve been busy.

I think this first quarter of small business has been the best I’ve had in quite a while. I could point to a lot of things on why I think this has happened but I often find it’s just guessing. When you’re a one man show running off services and support there’s nothing tangible like product sales or earning reports that make sense. Some months you’re busier than others. Some months you aren’t. You form business partnerships and sometimes those fade. Sometimes they get better.

It’s a dance on what you can/can’t do and learning what jobs you enjoy and what jobs you don’t. One thing I have noticed is QGIS and “being different” has taken me more and more into the backwaters of what I consider my service area. When a small municipality/company is faced with the juggernaut of changing software and ever climbing support agreements the whole idea of different sells itself. I have entered several “one stop light towns” to discuss open source software. It’s towns the big guys can’t touch and won’t. I’m OK with this. Sometimes good things happen and sometimes they don’t.

Consulting is sometimes more stressful than it needs to be. I’ve been known for getting agitated so I do my best to not let things bother me these days. Church is also stressful for me. When you grow up in the south you get dragged to church more often than not. I spent my fair amount of time in a church setting singing hymns and worrying over your position in the Universe. I tend to find a lot of humor in religion. I may poke at things like Christmas and Easter in jest – but there is a cut off line and I try to not cross it. When I was 2000 miles south working at the equator that turned into a thing you had to constantly be aware of. Sacred cows make the best hamburger – BUT – there’s a time and place for joking.

I noticed the other day  was getting prayed over a lot. I go to lunch  at a clients location and someone blesses the meal. Sometimes I get thrown in for good measure. I say afterwards “thanks I can use all the help I can get”. It’s interesting. Usually I wouldn’t think twice about it but I was at a clients location and they had easter eggs out. I grabbed one. It was an invite to the biggest church in town for Sunday morning service. Sunrise service followed with regular service and dinner. I will say fried chicken in a church setting is probably some of the best fried chicken you can get.

Not even thinking I opened the egg and there was not chocolate.

Me: Oh man.
Employee: What’s wrong?
Me: I was hoping for chocolate.
Employee: Better than chocolate it’s an invite to church – you shouldn’t miss church!
Me: <thinking Crap> Hey – I might show up. Thanks. I was just overly hungry this morning and looking for a snack.

I think in February every meal I ate with clients was prayed over. The first half of the month I didn’t notice. The last half I was wondering if the streak would continue. Which shouldn’t be a worry – but I like averages. Last week I was driving back from a client meeting and thought “You know I wouldn’t even be here except for the community effort of open source. The only reason I’m here is ripping out ArcGIS and replacing it with QGIS/PostGIS.” Business is dragging me back into a setting from which I’ve disappeared.

Not that I’m looking to dive back into an organized religious setting. For those of you outside the US – if you haven’t noticed it’s stressful times for some of us over here. Maybe not for all – but it is for me. So – I’ve decided to turn these meetings into my own “Peace Love and GIS” type of outreach. Don’t stress and don’t worry about politics. “Hey maaaaan do your thing and I’m doing mine over here. Hey lets talk about your data.” Last week I learned what “tractor and hay bale making machine worked better than another tractor hay bale making machine”.  Which had squat to do with Geo – but it was a conversation that started out of “Can you make a layout in QGIS for me?”. The client enjoyed telling me and I enjoyed hearing about it. We talked about church. We talked about sewer lines. We talked about dump trucks. We talked about his kids and vacations.

Anyway – I know this is just a rambling post BUT – it’s more stories I seem to build up from this small business gig these days. A few years ago I found myself standing at a small town corner thinking “I don’t have time for this”. These days I don’t spend enough time standing still so I’ll spend 5 extra minutes taking it all in today. Maybe 10 and I might find some random person to talk to for a bit. Maybe I’ll eat lunch there but I’m not spending all day standing around talking. I have things to do and people to worry over.

I will most likely sleep in on Sunday. For those of you not and heading out with your family – Enjoy the day!

NRGS and QGIS Certification

· ·

Certification? Yeah. Just calm down and let me explain.

I’m a fan and not a fan of certification. At some point I was a GISP. At some point I was ESRI Desktop Certified. There’s been talk of how to have QGIS Certification for a while. I’ve not been part of the conversation. The other day I found out it was a thing now. Usually I would go “Pfffbbbbt” and ignore it….BUT….here’s what hooked me.

In short (read up here)  – all the talk centered around QGIS certification boiled down to two camps: You had your GISP Approach where your portfolio speaks for it’s self. You have a test (IT Certifications) that certify you on an exam. QGIS isn’t an org that can really do either – so they went with option 3 – the community decides who can and can’t be certified. I submitted my class for review and it passed.

So what does that mean? I have an 8 hour class that you take. At the end I submit your name and 20 Euros back to QGIS and you get an ID that can be checked to see if you took the class. Will you win jobs and make your fortune based off this certification? No. You’re supporting the community by taking it though. You can still buy a book and read all your want and do awesome things with QGIS. You can download it for your platform of choice and use it with no certification. It’s made me think about giving the students a test at the end of the class.  Haven’t decided on that one. So we’ll see what happens. I want to “do this right” in my head for the community. I did decide that the classes I teach at Conferences that are compressed down to 4 hours aren’t going to qualify. You’ll have to sit with me with 8 hours.

To be mentioned in this group –  that’s pretty cool at least for me.

This gives me a way to donate back and not feel like I’m just leeching off the good work of other people. I still have other plans for financial support – but I’m getting to that one.

We will see what happens. I’ve got to start scheduling some classes. Want a class? Looking at breaking out of your commercial loop of software changes and portals and what not? Give QGIS a try.

 

 

I didn’t know it was called Refactor Fields in QGIS

· ·

The title may be the worst part of this post. Hopefully. Anyway – here’s what happened. I will explain it in all of it’s goofy detail with some hope that by the end you’ll do the same thing I did.

I’ve a client that is currently  making me push what I can do with QGIS/PostGIS. We hit a technical bump with data collection and to fix it they walked in with a new app they had purchased from some online app store that records data from their super accurate GPS Unit. The manufacturer of the GPS unit had one – but they found this scratched the itch more than the one they could have used….and the one they were using. My eye didn’t twitch much. I just cautioned against randomly doing things with your setup. I mean what if it makes a file you can’t import into your database. What if………

They have in postgresql a table that was more or less this: column1, column2, column3 . This app could create several data formats that ultimately me with: Foo1, column2, Bar3. You can’t just “copy and paste” the data in because the columns need to match. So I need to force it to match. No matter what trick I tried it wasn’t easy BUT some of them did work.

An Example of my thought process.  My python skills aren’t great – BUT – using the psycopg2 library, some ogr, and “the google” I was able to connect and change my data and push into Postgresql. More or less you parse out the columns and match them to the PostgreSQL table:

import psycopg2
import osgeo.ogr 
#import shapely

shapefile = osgeo.ogr.Open("/home/rjhale/gis/shapefiles/monument.shp")
layer = shapefile.GetLayer(0)

conn = psycopg2.connect("dbname='database' user='me' \ 
       host='localhost' password='itsasecret'") 
cur = conn.cursor()

for i in range(layer.GetFeatureCount()):
      feature = layer.GetFeature(i)
      descriptio = feature.GetField("descriptio")
      pos_x_sf = feature.GetField("Pos_x_sf")
      pos_y_sf = feature.GetField("Pos_y_sf")
      elev_z_sf = feature.GetField("Elev_z_sf")
      photos = feature.GetField("photos")
      local_date = feature.GetField("local_date")
      pdop = feature.GetField("pdop") 
      #Get feature geometry
      geom = feature.GetGeometryRef()
      #Convert geometry to WKT format
      wkt = geom.ExportToWkt()
      #Insert data into database, converting WKT geometry to PostGIS geometry
      cur.execute("INSERT INTO test.monument(descriptio, \ 
                   pos_x_sf, pos_y_sf, elev_z_sf, \
                   photos, local_date, pdop, geom) \
                   VALUES ('{}', '{}', '{}', '{}', '{}', '{}', '{}', \ 
                   ST_GeomfromText('{}',6447))"\
                  .format(descriptio, pos_x_sf, pos_y_sf, elev_z_sf, photos, \
                  local_date, pdop, wkt))

 

Holy crap. It worked. I really need to up my python game BUT – no one wanted to run something from cmd line. MAKE A PLUGIN! Which I considered. In my digging I found the refactor fields tool  in QGIS. It’s very similar to loading data into a database with ArcGIS for those of you stuck on that side of the fence.

Refactor Fields

So how does it work? It’s easy. Load your data that doesn’t quite match anything as the input layer. Load the table you want it to match in the “load fields from layer” part at the bottom. Click Load Fields.

loaded refactor fields

So what just happened? Refactor fields took my shapefile and matched it to the columns in PostgreSQL. If it can’t “guess” you have the option to make it match by typing in the fields. It generates a new layer that I can “copy and paste” into the database or import into the database. For some of the data I’m having to clean out the fields – so things that don’t match in the shapfile I’m removing from the new layer using this processing tool. Which – maybe I need to add those fields to PostgreSQL. I really want this table stabilized before we start. Although I’ve not tried it I’m guessing this works with any layer you want – I’m just using it against postgresql. I also will say this probably won’t match projections – so get your projection worries figured out before hand – this just handles the attributes.

Right now I’m  looking at a model so they can ultimately just “ClICK SOMETHING”. We all like menus. Mostly.

Anyway – load your data. QGIS has many tricks to make your life easy and I had no clue. I did learn something on the Python side of life though and for that I’m happy even though it turned into a dead-end.

 

 

 

Password protect your password in QGIS

· ·

There are a number of things I just “do” in QGIS these days and don’t think twice about them. Work is picking up on the FOSS4G side of life and I find myself more going “Well yeah you just do X and then you have Y – I thought people knew that.”

Today turned into a discussion on the Authentication manager. So what is it and why should you care? This will be a 2 parter.

If you’ve used postgis with QGIS  – you’ve seen this part of a menu pop up at some point.

You can supply a username and a password and connect to things. My “thing” is currently PostGIS/PostgreSQL and I use this a lot. The bad part is my credentials are stored in plain text. All I need to do is pop open the QGIS file or a qlr file with a text editor and my credentials are sitting there for the world. If you were to click on the red box labeled configurations (as pictured above) you get a different menu.

Click add and set a password you will remember:

and here we are:

 

 

 

 

 

 

 

You provide a Name and in my case I name it for what I’m going to be doing: “PostGIS” and then my postgres/postgis credentials. You can let it generate an ID or set one yourself. When you save it it creates a spatialite database in your .qgis2 directory. From here going forward you’re going to be asked for  the master password that you set in the very beginning. You can have different Authentication credentials for each part of your QGIS life. I might have one for PostgreSQL, SQL Server, Geoserver, etc.

So what happens after it’s set – let me connect to a table in postgis after setting up my connection:

and I connect:

Notice my password and login are now hidden by the Authcfg ID – which I let default to some alphanumeric combination. Maybe you don’t want it to default (which will be part 2’s discussion).

Why worry about it? Maybe you are in a corporate environment and your password into things is a bit important. Maybe you are hitting web services and have a PKI Certificate.

So – if you want all the information go here  because I have no doubt missed something or confused some aspect of the explanation. Like I said – all of this started with me going “Hey you should….” and the client going “WHAT”.

Security – it’s not overrated.

So you’re going to write a year end review……..

· ·

As the title would suggest – a year end review. I have started this piece a few times and trashed each one. One was too technical. One was too emotional. One just wasn’t clicking. All that tells me is this needs to remain unedited. So you get what you get…….in no particular order.

2017 was weird on a lot of fronts. On a personal level I found myself making peace with a lot of “things that go bump in the night” – those things that, when you should be sleeping, tend to come and cause more questions and angst. From a poorly paraphrased conversation with Atanas Entchev I once said “It’s not personal it’s just business” and he remarked back “It’s all personal – it’s your business”. So there was much reckoning on the business front and I started acting more like an entity that makes money. 2016 was the year I learned to say no and 2017 was the year I looked at a job for two things: income potential and how much difference would it make in the world. Some prospective clients were told no. It was just that easy. No. Some jobs where there was only going to be “break even” potential did get work done because their mission was too great to be ignored. I also separated myself a bit more from the business. Randy made an appearance. Randal stayed at work.  I did completely miss another work anniversary where I should have went “Woooooooo 11 years in business”…..

On the personal side of life it was a bit brutal.  A good friend of mine just up and died. So I found myself in the pouring rain one day in April standing outside his house as he was wheeled out in a body bag. No final joke. No:

“Hey we need to get you to a hospital”
A Hospital…. what is it?
“It’s a big building with patients but that’s not important right now.”

Anyway – I’ve been working with his girlfriend attempting to get things wrapped up. Death is a messy business if you don’t have a plan for it. I knew people that suddenly passed – but never someone that didn’t have at least a clue something wasn’t right. He just went to sleep. I also found myself questioning every bodily function after he died. I burp and “Oh man – I’ve never burped at 8:36 PM – I wonder if that’s a sign I’m about to die”.

One family member had multiple surgeries. So many in fact we had to have “the talk”. The talk where you are the executor because it can get messy if the surgery doesn’t go as planned. The executor – because you are the one guy that doesn’t let emotions get in the way to make a decision. Which isn’t entirely true – BUT…….hey no one reads this blog in my family.

All of that really left me (if you bounce up a few paragraphs to the personal part) going “What the hell am I doing?”.  Some self examination is good for the soul. There was a lot of that. A lot. A lot of changes. Some say I got less grouchy. Some say I got more grouchy. Eh – maybe a little bit of both. Overall I am trying more. Trying what? To be better. Mom tossed her #metoo moment out one day. So for those who go “man everyone is a bit touchy these days”. I will agree – but – sit with your mom and have that talk. So changes. Lots of changes.

I did wrap up in July the “year of Canoeing”. So I had to be in my canoe for at least an hour a month for a year. If I got out once I knew I’d stay out for 3 or 4 or more hours. I did it. It wasn’t easy but I did it. This year? Ride my very old mountain bike in one new place a month. So I expect a road trip or a few as a peddle around the area. Maybe I get a new bike.

On the work side of life – I made the decision to remain off any boards of any Geo related things and instead throw my energy into QGIS. Why? I like the community. I enjoy the software. So I’m going to do my best to integrate more and support the project and spread the word. To that end I started working on writing a plugin a few days ago. I know what I want to do. I’ve got a decent enough understanding of Python to make it happen…so Why Not? I also found myself neck deep in postgis. I finally “get it” – and so I’ve been just enjoying that piece of software. Wrote my first nested sql statement and had it run “first time”. I was so excited I accidentally deleted the entire directory where the script was sitting and had to re-write it. Nothing like a plan. On another side note – I had to use ArcGIS multiple times this year at a client’s location. What did I learn from that: 20 something years of using the software and I still can sit down an just start using it when I haven’t touched it in half a year.

With the help of several we have QGS.US back up and running. I need to work on that more in the next few weeks. I’m inspired and want to have an event of some sort centered around QGIS.

I did hold a “foss4g” mini event in Knoxville. It went better than I’d hoped for – except for the financial end. I gotta work on “Make sure things are paid for”….but – worth it. I held it. I started a small thing. We’re doing it again in 2018 – I just have to figure out “when”.  Show up if you can.

I ended up on a massive road trip to Boston to the FOSS4G event there. I can’t say enough good things about that. I met people. I talked. I got to meet some long time “heroes” in the FOSS4G world. It was a week of wearing myself out. Boston FOSS4G committee – thank you for the event. It was massive. It was fun.

I also started “rebuilding” the QGIS class after a year off. I’m three chapters in to having a QGIS 3.0 class. New exercises. New Data. New Pricing Scheme. What would that be? I have no clue as of yet but I’m aiming for organizations because that’s the only time I have success in training. Read above where I covered “income”. But do remember “make a difference”. I’ve toyed for doing the class for a small donation also just to upset the apple cart.

So what did we learn this year kids?

  • Life is short – make a difference
  • Love your neighbor even if don’t subscribe to any major religious institution that yells that concept.
  • The past is the past – it’s isn’t going anywhere but you are.
  • Get involved in the Open Source Side of life kids – if you aren’t you are missing out. You may not get the chance to use it at work. Use it at home. Load Linux. Load QGIS. Contribute. Write documentation.
  • Do one kind thing a day.
  • Read Geohipster!

Get busy – 2018 is coming.


 

Contact