North River Geographic Systems Inc

Spatial Problem Solving

You are here: Home / Blog

Blog

Offline Editing Plugin: TN911 Project

· ·

One of the weirder problems we had on moving the ESRI setup was editing the data. It wasn’t the normal “hey lets add an address” but it was tackling the idea of “We need to edit the data in the field”.

As it was explained “the way we did things”  was to drag the data out in the field with the editor because they were running a simple Arcmap/FBGDB. Now that we had moved to a Postgresql database I wasn’t keen on people “carrying the data out”. I preferred they connect to the database and edit it like civilized folk.

That is a problem because the county is rural and connections are sketchy and all of that led back to the idea of “Take a copy of the data with you”. We had a mobile solution but nothing making my life easier.

There is a plugin that originally caused much angst and it is called the OfflineEditing Plugin. It’s seriously the most no-thrills thing you will find in QGIS.

At one point in the early stages we had a lot of errors and I couldn’t figure out why. We have an ID attached to each feature that never changes. The client had discovered the plugin and it would let you check data out and edit it and check it back in. Which was what we wanted to happen…..except back when 3.0 was brand new we had a lot of things change unexpectedly. I had a trigger setup that added the primary key to the features ID. So data would come out. Data would go in and between the triggers and functions the ID would change. To the client this looked like a great solution. The intern thought it was awesome. The guy watching things change in the database didn’t have much joy.

I’ve not touched it since 3.2 or 3.4 in QGIS was out.

I’m running a “beta” (3.17) copy of QGIS at the moment. So I built a quick database and decided to play. It checks your data out and you then have a seamless offline dataset stored in a geopackage (the original data is in a postgresql database). I check it out and edit and check it back….and……

It worked. ID’s were respected. Sequences were maintained. I added data and checked data in and Edit Dates and Edit Times were also saved (at the time of check-in). Since I checked the entire database out and checked it in things I didn’t work on received a new edit date. Which – I think there is a way around that but that goes beyond the 1 hour test today. Which – given the 2+ years since I’ve touched it this plugin may have been working fine for a bit.

So what does this mean? Well – you can take your laptop and check your data out and go into the field. AWESOME IT’S VERSIONED….well not in the way ESRI versions anything so you will overwrite some data if you aren’t careful. It does make this setup a little more flexible although I’d rather toss this into a mobile app and only worry about the things we need to worry about – in other words if you’re working on addressing in a neighborhood I only care about that neighborhood. Which means I’d probably split that data out – check it out – edit it – Check it in and then merge it into your original data. More work but safer. I like safe.

Overall – a much better experience with the Offline Editing Plugin this go around.

Parking Lots and Lines

· ·

New Year – Different posts.

I love Listservs – I’ve resigned from all the OSM listservs though. It takes the stress level way down. Of course I still edit. Chattanooga from an OSM standpoint is pretty decent. Three things I usually tackle are buildings, parking lots, and Fire Hydrants. Why parking lots? If you ever wondered how much space we devote to parking cars….try outlining them. I want to fully map parking spots this year with number of spaces, handicap access, etc.

Me and JOSM

Two things are problematic in my OSM mapping around here:

  • I don’t have a drone – hence I don’t have updated imagery. I have several tricks which gives me nearly current imagery. That and some field work and it’s mostly all good.
  • Now vs 10 years ago has brought more editors. I’m always curious who – is it local or someone armchair mapping. I really don’t care as long as the map is updated correctly.

My mapping isn’t continuous and considering how much free time I had in 2020, I decided not to map and instead go canoeing. When I finally did jump back in someone had been adding a lot of service roads and parking isles. I check and it’s Amazon Logistics.

The screenshot below shows nothing more than lines – BUT you will notice some small gate-like symbology. Apparently the drivers are relaying info back to the editors. If you look at the page for the editors and reviewers there is a LOT of them also. I welcome my new Amazon Overlords. I’m the guy putting in the parking lots.

Anyway – it did strike me that you have yet another giant corporation harvesting out of OSM. I guess that speaks to the overall quality of what they are getting. I do wonder how much the map changes from these editors. It’s a complicated mapping life out here at the moment. You may have an entire delivery infrastructure relying on this data. That I am contributing to…..

With that I decided to join OSM US for the low low price of whatever it was. That breaks several years long standing of “Join nothing” but it is a New Year.

The Year in Review – 2020 Edition

· ·

For my own sanity I usually do a year end Health Check on the business and see what went well and what didn’t.  This blog post has went so well I’m on the 7th re-write. The first post was exceptionally dark and put me in a funk for a week….the next six became increasingly easier to deal with by acknowledging the current situation…..and that situation is covid-19. Covid and Me. Life in general.

I decided to do this one last time – and this will be the final attempt whether it’s rambling or to the point….who knows. I did some editing because I’m me.

I feel like I’ve lost sight of what makes me tick and why I started this business. The last 6 months of 2020 was like a giant spotlight on a lot of things that were making me uneasy.

  • There’s a ton of things I enjoy doing I’ve stopped doing. There are interests I’ve stopped pursuing that bring me a lot of joy. Reading. Camping. Intellectual pursuits grow dusty. Tech for the sake of tech.
  • The business was a money making endeavor that was going to occasionally allow me to do some fun things and “help”.  It turned into a struggle this year – and 95% of that wasn’t caused by me  – but it was and continues to be tough.

If you came here for glorious tales of awesome business stuff…….stop here. If you haven’t noticed this year has been…well….A giant flaming pile. I’ve had worse years – but not where the entire Flat Earth was having them too. I’ve been lucky as I’ve not gotten sick nor had any friends pass from Covid. It did wreck business a bit. It has caused a lot of stress on my end for multiple reasons. From arguing with friends and family to watching job after job stop and go on hold. I’ve watched several on social media lose friends and family and had friends lose people they care about. That made this post a bit hard to write as the first post was filled with complaining and that didn’t feel right. Each subsequent iteration got lengthier as I tried to convince myself and you – that I had a handle on things.

I don’t. That’s fine. I’m not sure many do.

I’m still here though.

By the time summer had arrived we had been in lock down 2 months. I think. Honestly I can’t remember.  I have a general policy that if a job makes no movement in 90 days it’s dead. It’s not always the case – but in this case they ended. When you do consulting and the phone just stops ringing it’s a bit unnerving. So there wasn’t much to do but:

  • Do some Hiking
  • Do some Canoeing
  • Do some Bike riding
  • Do some Thinking

After leaving a job this summer for non-payment I had to take a step back. I usually don’t stop for non-payment. Which is stupid on my part. I try to make things work and finally I got to the point of “I can sit here getting yelled at and not getting paid or I can go for a hike and not get paid”.  At the end of the hike I sat down and made a list of things I love and hate. Things I hate and things I like and Why am I doing or not doing those things. Why am I letting this startup make me miserable. Why Why Why………

I found myself working on the weekends part time for the non-profit for which I volunteer. They run an outdoor store full of camping and hiking gear. So during the week I would scream at NRGS and covid and on the weekends I lived a life I wanted 30 years ago…back when I was more familiar with Geology and the best spots to camp in the southeast. During that time I had a lot of invites for camping, canoeing, and hiking. It was a nice change.

There were come positives this year – so all wasn’t lost:

  • Work continued on the 911 project – I added more functionality and branched into the NENA standard.
  • Renewed one contract that will help fund the 911 project
  • Conquered my fear of github
  • Docker is less mysterious as I’ve got it working with minimal screaming.
  • I fell more in love with PostGIS and did more things with it. Really if my life just boiled down to QGIS and PostGIS I’d be happy until I was dead.

There were enough loss to fill out several pages. It is frustrating when things spin out of control. Especially when you can’t do anything about it. If you were sick there was nothing I could do. In general there wasn’t much of anything you could do. Out of all of this and the sitting and the fidgeting I did find things that absolutely fascinated me. Like:

……..as well as a whole host of youtube videos and other small bit of information that made this all a little more bearable. I rebuilt an 50 year old coleman stove. I fixed several things on my jeep that were broken.  I have so curated my twitter feed that what was termed “doom scrolling” really wasn’t for me. I had all this information from blogs, github, and videos that appeared on twitter and I could watch it night and feel…well….better. Twitter is still a dumpster fire – but a fire I’ve seemed to curate into something enjoyable.

Many of you hate virtual conferences. I completely get it – BUT the things I had the chance to attend this year were awesome. From England to the US to Canada. I didn’t even have to put on pants. I did though. I know virtual conferences are different and not what you want. You want small Danishes and coffee and the chance to hassle people in person…..BUT – I would have never gotten to see different talks live (and yeah I could go rewatch but I like ‘live’).

NRGS is getting a reboot of sorts. It’s not the end. Covid has changed things. I’ve changed. We scream “Get things back to Normal” but I don’t think things will…or really they should. We say the new normal with some hope it’s not. I think in general it is – and that new normal means the way I was working won’t be sustainable anymore…or if it is – It’s still half a year from being back to normal or  maybe longer. Plus I have this great list of things I need for personal and professional peace of mind. Jeep – you’re gonna do a road trip shortly. Yosemite National Park – I’m coming for you. Probably not both at the same time. Maybe. Although………

Good People we all must change some. Here’s to a better 2021. To all of you – I thank you immensely.

 

I’m making a Map: Location

· ·

I should write something. My last few “2020 in Review” posts aren’t working like I’d wish – so I sat down last night to make a map.

A map of what? Good question. I think in my every day attempts of databases and technical wizardry I lose sight of “I actually like maps” and I’m not the best at it. Given my background I should be better. I decided to make a series of maps on “things I like”: Canoeing, Hiking, Bike riding, etc. So my first shot is a canoeing map. A very simple map of something I like to do.

The location is easy – The start of the TN River Gorge. It’s a great 8ish mile paddle. Simple. Quiet.

canoe
Looking into the Tn River Gorge

My first thought was to do what I normally do and just drape things over a aerial photo. I’ve decided to make it a bit more fun and not go that route – This may end up being Black and White. I’m also going to stick to two sizes of paper in this series of maps. Either 8×11 or 11×17.

When you canoe this loop – you start north and head south and loop around an island and then head back north again. The first half is all upstream (heading south) and the last half is all downstream (heading north). Bonus it’s all NW-SE trending so it will need to be rotated.

I start building it in QGIS with some aerial imagery and Hillshades for fun:

QGIS Canoe Map setup

If I make a 11×17 layout it looks more like this because I’ve added 120 degrees of rotation:

QGIS canoe map setup

As I sat here I had a question. I scale it. I generally start to get an idea of “what” I want. I know the extent of the map but can I get the extent back into QGIS so I can start clipping things and not having to include 500 square miles of stuff for a map. I could just set up the boundary back into QGIS as delimited text (make points -> draw polygon). I could just “eye ball it” which is how I do about everything from Jeep repair to Home Repair.

For fun I searched the tool box and….Print Layout Map to Layer

I run that and……I know where my map falls in the main window. So I can buffer it and start clipping data.

Why even post….I had no clue this tool was even here. My next map (another canoeing map) will be a bit larger than 11×17 so I’ll nee to use Atlas to setup 2 to 3 pages. I now have the general extent for 11×17 at X scale, copy and past this boundary a few times, and rotate it to get the general layout locations I will need

Map 1 is well on it’s way to happening. This is a bit therapeutic and that’s something I need at the moment.

 

 

Drop Down Menus for the TN NG911 Server

· ·

The problem is I have too many options. Options are awesome except when you’re dealing with people inputting data and you only want them to do it one way.

QGIS has had the ability to build drop down lists for a good while and it’s a very flexible way to work. For those of you in ESRI land imagine Domains and Subtypes on Steroids. In earlier versions you had widgets and currently you have Attribute Forms.

If you have a data layer in QGIS (all my layers are coming out of PostgreSQL/PostGIS) just right click the layer and go to Properties. You’ll notice all your fields have a chance to be a “widget”.

QGIS Layer Properties

Addressing data is probably 90% picking existing attributes. You don’t want to get too fancy in making up new stuff. For instance – Street Type.

USPS Sample Street Types

I ran out of fingers and toes on counting the number of street types. For the TN guys they all get a ESRI File Based Geodatabase with this information spelled out as Domain tables. How do you do that here? QGIS gives you three ways to make this happen:

  • Unique Values sums up an existing data set and you can get a pick list of existing data. The good thing is you don’t have to do anything really to make it work and the bad is you’ll get Avenue and Avenew. Misspellings aren’t your friend.
  • Value Maps let you fill things out or import a CSV. So if you save this information in a spreadsheet you can import it in.
  • Value relations pulls from an existing table. So if you build a table with this information you can pull that in by first adding the information to your QGIS session and then pulling it into the widget.

In order to cut out down the options, I pre-built all the tables needed into the database. I add them into qgis, group them, and collapse the group because No one needs to see them. Yet. You can’t use a relation widget with the table being added to QGIS.

QGIS grouped items

Back to the Attribute Forms and this is the boring part. I want to select the type table and apply it to the type field using a Value Relation. The important part is the key and the value is the description of that attribute. Both things could be “type” but I chose to have a type and a description:

QGIS setting up Type

Go through and start setting up your Forms. What if you have a second person working on this?

Save your style to the database source:

QGIS saves a table in the public schema and everyone with the proper permissions and access is good to go. Plus you don’t have to touch everyone’s QGIS session…well much. Users still have to load the tables to get started but good clean directions should work. Of course this does make me still debate a plugin that does part of this for you.

What? Yeah – it’s that easy. Everyone working with drop down menus and making good clean data.

 

 

 

 

Contact