North River Geographic Systems Inc

Spatial Problem Solving

QGIS

Intro to QGIS Training – September 25th 2018 – Montgomery Alabama

· · Leave a Comment

The Intro to QGIS class is going to Alabama! Actually – Montgomery Alabama September 25th 2018.

It was picked to start off the Alabama Damage Summit. Also – with much luck this will be certified! Actually – there is no luck about it. It’s a certified class. Which means approximately 25 dollars per student goes back to the QGIS Organization.

Anyway – if you’re coming to the conference sign up! If you’re not ding the conference organizers and maybe you can attend. The more the merrier at this event.

What will we cover?

    1. QGIS Community
    2. Navigating the Interface
    3. Data and Symbology
    4. Geoprocessing and Plugins
    5. Editing
    6. Layout Manager
    7. Databases

The Database section might change a bit depending on time constraints. It should be a blast – so come down/up/over/back and learn some new software.

QGIS 3.2: Metadata

· · 2 Comments

On the heels of the “document your data” post  I’m about to turn my writing towards QGIS 3.2.

One of the things you should worry about with your data is documenting. Why did you do what you did with attributes and fields? Amazingly enough – most of my clients don’t know or just go “OK” and add whatever they think they need. They are busy. Busy people sometimes don’t slow down enough. In the case of your data – slow is better.

Which brings us to – Metadata. I don’t know the current state of metadata in the world. I searched the internet. My eyes went crossed as I read things and I restarted this post.

If you look at 2.18 (Current Long Term Release), metadata in QGIS looks like this:

 

qgis 2.18 metadata

Not bad considering the this is a newer development in the life of this Open Source Desktop.

If you take a look at QGIS 3.2:

qgis 3.2 metadata

You can write metadata. You can save it. You can reload it for later editing. This was a funded feature for QGIS by Arpa Piemonte ( The Italian EPA – which I would assume is comparable to the US EPA) and implemented by North Road. How excellent is that? Actually – very excellent. Imagine if my fellow users here in the US pitched in for development? Good things will happen.

So my next go around I’ll most likely be using this feature for documentation. For fun I might document parts of the current project and see how it blends with the FGDC Metadata standards.

Any documentation is better than none – so write!

Rolling out QGIS/PostGIS Part 1: Document

· · Leave a Comment

I hated putting Part 1 up there. I start a lot of Part 1 posts and never get to Part 2…..I’m planning on this being a multipart post. After a while of screaming ‘hey why don’t we put QGIS everywhere’ I had a client decide to replace ArcGIS as their primary GIS. We were working along with QGIS/PostGIS/Fulcrum/ArcGIS and it finally got to a point to just switch. I’ve done a lot of QGIS installs and a lot of ArcGIS installs and enough PostGIS/PostgreSQL installs It doesn’t worry me – what does worry me? Data.

Way back in 1995 I was working for the Federal Government doing commercial mapping and the one thing I took for granted at the end was “the data dictionary”. We would work a project and at the end one of the guys would sit down and crack open a spreadsheet program and document every layer we had made and why we had made it. I guess sometime in 1996 or close to that we had FGDC metadata roll out and to be honest I don’t remember if we continued on with the Data Dictionary or dove into the FGDC Standard.

The client had a file based Geodatabase they had been working out of for a good while. It was functional. It had domains and subtypes and all the things you would hope a properly built ESRI file based database would have. The one thing it didn’t have were “reasons”. As in “why is this field here and not here and why is this here?” There were a lot of domains. Not so many subtypes though.  Apparently there had been somewhat regular revolving door of consultants. Everyone added their spin on things and here I sat….adding my spin…or should I say changing it all. So that started a whole list of questions which ends with me writing everything down in Google Docs. Yes – 22 something years later I built a data dictionary to describe all the attributes and columns.

Domains
Domains in ArcGIS

If I look back at my time for data conversion and banging my head on the wall – 60% was asking “Why?”. Then documenting the responses. In some cases we just had to accept “that’s just the way it is” and in others I was able to trim some bloat out of the database. I still think there needs to be a second round of trimming on this database.

So I’m going to spend some time talking about this move. Then talk about QGIS 3.2 because that came out – and there are things in that release that make me want to upgrade them off the LTR right now. For those of you not familiar with QGIS once a year you get a Long Term Release. It’s awesome because “things don’t change” – it’s stable. So we have it in enough places it’s going to sit until October at least.

Part 1 for all of it’s rambling ends…or starts right here: Document the data. Sit with the GIS person and ask why to the point you’ll get on his or her nerves. Your default answer to everything should be “delete”. Except you don’t. Use  ArcGIS because there are going to be a few things you aren’t going to find talking to anyone.  Once you’ve written everything down you can think of then you can start. Take the goofiest layer you have and move it into PostGIS. Check it. Double check it. Check it again. Check Field Lengths. Check Date Fields. Check to make sure someone hasn’t “shapefiled” your data and you’re left with “descriptio” vs descriptions.

Example – this database had two fields that completely baffled me: Location and Place. Location was for “where” the data resided as in an address. Place was a placeholder for notes. I had no clue. Luckily the client was patient as I asked more and more questions. Note the screen shot.

QGIS Screenshot

The other reason I’m writing this all up here – I think this is about to be a service: Migrate over to a free and open source server that is commercially supported for clients that can’t afford all the licensing that comes with an ESRI option. Of course I don’t want to call it an ‘open geo suite’ since that name has been used. I don’t plan on making downloads. My one big heartburn was I didn’t load geoserver for them – I couldn’t justify the overhead and banging my head on Windows Server more plus they had a bare bones server. Being able to edit the data was more important than pushing services out. Maybe later – but for now – They are editing and making maps and making decisions based off where we are.

 

 

Updating the Tutorial Page

· ·

When you’re the one man shop you’re constantly trying to make life easier as far as maintaining a web presence. I started a few years ago a Tutorials page which was super fun at first. Until it started growing and then I started falling behind updating them. Plus I’d get the occasional email of “hey this didn’t work” or “hey you’re awesome”.  The “hey this didn’t work” usually started me down a rabbit hole of updating the page and/or testing the procedure.

So  – I’m about to party like it’s 2014 and jump on the github bandwagon. I know – it’s all the rage these days. To be honest – I don’t use github much. As of late I’ve been working on using it more than I normally do.

Every Tutorial is going to get it’s own repository and that way if there are problems – I keep that problem with that repository. I make notes and keep things more regulated. Update as needed.

The next issue is “how quick” does this happen? As quick as I can. I’m probably going to retire the ArcGIS tutorials and update everything to QGIS to 3.x. Anyway – bear with me. It’s slowly moving in a better direction. With this week being a holiday week I should have more time than normal to push this along.

What was the first one? Putting an image into a geopackage.

Last Year’s Hurricane Season in the USVI

· ·

So I’ll talk about this finally. I usually don’t talk about “work” unless it’s in generalities about software and processes. I probably should talk about work more.

I’ve never been in a hurricane. I think so far I’ve been in about a handful of earthquakes, stood on one volcano, and I think I’ve seen one tornado. Not that I’m looking to be in a natural disaster – I just don’t have a frame of reference for what happens during a hurricane or a tornado.

If you flash back to early September 2017 the Caribbean was gearing up for a hurricane and. Turns out two hit close together. I had the chance to work in the US Virgin Islands on an addressing project with AppGeo and Spatial Focus several years ago. It’s been a “wait and see” type of deal. We did a little and watch as red tape and things slow it down. I called a few friends down there before the hurricanes hit and wished them well. Overall – there wasn’t a ton of fear but a lot of concern. I was concerned because the 911 system isn’t great as the addressing isn’t wonderful in the USVI.

My memory is getting fuzzy – but it seems like both hurricanes hit and it was a while before anyone could call/email me. Maybe it was late September or October before the phone rang. They needed help to figure out where the tarps needed to go.  FEMA/USACE was headed down to bring tarps and supplies – maybe they were already there. A sneaker net of GIS data had cranked itself up on the island as thumb drives were passed around. I started trying to figure out “what made sense with the base data I had available”. I started digging into what USACE and What FEMA would do if your roof was damaged. More or less if you had 50% of the structure of the roof left they could put a tarp on it. I came up with a plan and contacted AppGeo and they tossed a person my way.

Structures were categorized:

  • 1 – 25% Roof Damage
  • 26 to 50% Roof Damage
  • 51 to 75% Roof Damage
  • 76 to 100% Roof Damage
  • Structure Destroyed
  • No Damage
  • No Imagery

NOAA flew imagery of the area. The imagery was a bit rough around the edges but it worked. Actually – let me back up – for what it was it was awesome. Granted it didn’t lend itself to any sort of automatic classification as there was a lot of building lean and some missing imagery. It worked though. That was a great free resource by NOAA.

Hey that’s a lot of points. Yes – 48380 points covering three islands. Granted I didn’t get every house as I had nothing much to start out with except OSM and some USDA NAIP imagery. I would guess I’m within 10% of an accurate building count. Some people have offices/live in shipping containers down there. WHICH – when I was there made me go “awesome” but when looking at the imagery at a shipping container blowing a half mile across a field – not so awesome.

It did give me flash backs to my old remote sensing days with the government. Although this was less refined – it was a game in the afternoons to see how many points I could collect in an hour. Some neighborhoods were easy. Some not so much as there wasn’t much left. The other nice thing that happened was sometime around December I switched over to QGIS 3.0 and started seeing what I could/couldn’t do with this almost released software. Several bug reports were filed so more good came out of this.

The data started making it’s way out to one agency after another. My big fear then was “what if someone uses this beyond ‘roofs'”. I wasn’t doing a full blow damage assessment because I was seeing some evidence that maybe the roof survived but the bottom floors didn’t. Plus I wasn’t on location.

It took me a bit longer to finish than I wanted – Volunteer projects tend to be the last thing I work on most days.

I did start pulling numbers out of some of the work – this is for St John Island:

Count Classification Percent
1487 1 – 25% Roof Damage 41.42
326 26 to 50% Roof Damage 9.08
131 51 to 75% Roof Damage 3.65
126 76 to 100% Roof Damage 3.51
1007 No Damage 28.05
17 No Imagery 0.47
496 Structure Destroyed 13.82
3590 Total 100

So 25% of the houses had no roof damage. 60% had something happen. More or less 15% were leveled. I’ve heard ancillary stories where almost all houses had water coming in them and in a warm tropical climate you get mold. Mold would make the house unusable for me.

Anyway – I worry about the 2018 Hurricane season that is upon us. Maybe it will be quiet – maybe it won’t be. AppGeo has Mapgeo for the islands. It worked out well. I’m going to try to “build” a database here so in case something happens and I’m not starting from scratch in case I get to help out again. I hope I don’t help as nothing happens.

 

Contact