911

Drop Down Menus for the TN NG911 Server

December 4, 2020 by rjhale

The problem is I have too many options. Options are awesome except when you’re dealing with people inputting data and you only want them to do it one way.

QGIS has had the ability to build drop down lists for a good while and it’s a very flexible way to work. For those of you in ESRI land imagine Domains and Subtypes on Steroids. In earlier versions you had widgets and currently you have Attribute Forms.

If you have a data layer in QGIS (all my layers are coming out of PostgreSQL/PostGIS) just right click the layer and go to Properties. You’ll notice all your fields have a chance to be a “widget”.

Addressing data is probably 90% picking existing attributes. You don’t want to get too fancy in making up new stuff. For instance – Street Type.

I ran out of fingers and toes on counting the number of street types. For the TN guys they all get a ESRI File Based Geodatabase with this information spelled out as Domain tables. How do you do that here? QGIS gives you three ways to make this happen:

Unique Values sums up an existing data set and you can get a pick list of existing data. The good thing is you don’t have to do anything really to make it work and the bad is you’ll get Avenue and Avenew. Misspellings aren’t your friend.
Value Maps let you fill things out or import a CSV. So if you save this information in a spreadsheet you can import it in.
Value relations pulls from an existing table. So if you build a table with this information you can pull that in by first adding the information to your QGIS session and then pulling it into the widget.

In order to cut out down the options, I pre-built all the tables needed into the database. I add them into qgis, group them, and collapse the group because No one needs to see them. Yet. You can’t use a relation widget with the table being added to QGIS.

Back to the Attribute Forms and this is the boring part. I want to select the type table and apply it to the type field using a Value Relation. The important part is the key and the value is the description of that attribute. Both things could be “type” but I chose to have a type and a description:

Go through and start setting up your Forms. What if you have a second person working on this?

Save your style to the database source:

QGIS saves a table in the public schema and everyone with the proper permissions and access is good to go. Plus you don’t have to touch everyone’s QGIS session…well much. Users still have to load the tables to get started but good clean directions should work. Of course this does make me still debate a plugin that does part of this for you.

What? Yeah – it’s that easy. Everyone working with drop down menus and making good clean data.

TN NG-911 Server

July 24, 2019 by rjhale

Last year I had the chance to work with Henry County TN’s 911 group on a software migration project. I moved the organization off a basic commercial GIS product to an open source enterprise server. Overall it was a great project with a defined end goal and a lot of flexibility in making it happen.

Once the project ended I never stopped working on the database and the tools. Over the last few months I’ve had the chance to turn the database into something that is 100% compliant with the State of TN’s NG911 System. I present the TN NG911 Server.

What does it look like?

Fulcrum for Mobile
QGIS for a GIS Desktop
PostGIS/Postgresql for the database
Geoserver to provide OGC compliant services

The workflow is simple. Field Personnel go out and collect data using menus set up in Fulcrum.

Data is stored in PostGIS/PostgreSQL and edited with custom menus in QGIS. The database is served out through Geoserver as an OGC service.

Data is exported from the server at night to the the states ESRI database.

The bonuses to this approach:

100% TN NG911 Compliant
Cloud Deployment Or Onsite Deployment
Local TN Support for your organization
Can Interface with Computer Aided Dispatch Systems through managed Exports.
Quality Checks on your attributes
Topology checks for your data
Unlimited Desktop Deployments for everyone
Can work with TN’s LIDAR Data and TN’s Imagery Services.
Handles more than just the 3 data layers the state needs for NG911

Granted it’s not 100% open source as Fulcrum is a commercial product. Fulcrum will need to be purchased but it’s only a fraction of what you could be spending on software and it has more uses than just address collection.

The TN NG911 Server has a solid deployment that’s been in production a year. If you’re interested in integrating Open Source GIS into your 911 system you should at least look at this approach. Everyone needs good clean data.

Future plans have a plugin and more quality checks on the data. For Version 1 though this is very solid and very functional.

What about the NENA Standard? The TN Standard is not NENA Compliant! Well – stay tuned for more news.

PostgreSQL Hack

February 21, 2019 by rjhale

Many of you will look and go “OMG – Randy has a new hack for a database”. No – Randy is a hack at databases.

So about 4 or 5 years ago I was dropped into the idea of databases. I was working a job several thousands miles to the south and life was easier with a database as opposed to flat files. Up until that point I had flirted with databases and taken some online courses. I would make small databases but it wasn’t something I had to deal with day to day. I did deal with it indirectly – because at the time my employer was running ArcSDE on top of Oracle – BUT – as a lowly employee I wasn’t allowed to do much with it. That was the job of the admins. I made ESRI File based Geodatabases and was quite happy. Every now and then I’d even make Microsoft Access based ones.

So flash forward to this last year and I’ve been up to my neck in databases. A lot of you will shrug. You do amazing things with PostgreSQL and SQL server and it’s not that big of a deal. Last year I did my third big migration of flat Geo files into PostgreSQL/PostGIS. It was a fairly simple thing to do: QGIS, PostGIS, and I loaded Geoserver (but we still haven’t used it – I’m confident we will).

The client moved from a “ArcView/File based Geodatabse” environment to a multiuser database environment. They are a 911 and it’s a small county. They went from 1 person being able to work to 3 people working and viewing the data. I think currently they have 4 people adding data and updating data now. That’s a win.

What wasn’t a win was my moving of data into the database. File based Geodatabases don’t have a primary key. Maybe there is some way to deal with that these days – I don’t know. So they were using an extension called Attribute Assistant to deal with unique columns and sequences in ArcMap. I loaded the data and didn’t pay enough attention to what was happening.

To make a short story longer – I mucked up the unique column that the data had. I didn’t know it had to be assigned then never change – as explained “it just needs to be unique”. When I loaded it the Primary key and the Unique ID were off. Primary key was 4 and ID was 5. Look there is a primary key of 10 and a ID of 247. The unique ID had nothing to do with the county data – but everything with the state 911 database. So I calc’d the ID off the Primary key and that’s when things went stupid. So what magical GIS thing did I do? I didn’t.

I made a new index column.

update hc911.addresspoints set index = right(oirid, -6)::int;

Which ripped the number off the unique column and assigned it to a new field. I built a new sequence with a new incrementing value. I messed that up. I ended up redefining a new primary key. Fixed. No GIS knowledge applied – just databases.

How do you know you are tired? This is how!

Which – not a huge deal but for me a learning experience. I need to spend one day talking to the client and making out a path way before we migrate. Migrating data took 2 hours. Institutional knowledge probably would have taken 8 hours. The next one won’t have this problem. You’re going to have to talk to me a lot about your data and process. Way more than you want and that’s fine – we just need to talk.

Data is everything. To make everyone happy I set up a few cron jobs to dump data into Geopackage and Shapefiles using OGR. They’ve been quietly chugging along for 6 months now doing their work with no applause or fanfare. More people in the office are slowly going “hey you mean I can use QGIS to connect to the database? I can see the data?”.

I can say this whole move minus the ID hiccup has been a huge win for the organization. I was just lucky enough to be the guy to do it.

So if you’re an organization struggling with your commercial system – why not take a look at this side of the fence. We’re a lot of fun. If you’re wondering “Hey what should I learn these days” – I invite you to dive into PostgreSQL/PostGIS.

Adding Triggers to Geopackage

August 30, 2018 by rjhale

In between work I get a chance to play around with software. I’ve been digging around postgis more and more these days trying to get a better understanding of what it will do or won’t do. It’s a never ending learning experience for me. Sometimes it pays off – the never ending tutorial left me suggesting we dump a database to a geopackage.

I’m great at using geopackage in sentences like “USE GEOPACKAGE” and “Hey – what do you think about geopackage?”. Since I’m actually officially using it I decided to do some digging. It’s a SQLite container. Since it’s a SQLite container I should be able to do some things with it – like…..

I can use the spatialite gui and throw SQL against it and change the tables.

While talking to another person the question started with ‘Well – why can’t we just do the data collection in geopackage?’. Which we did some back and forth and in some cases this would be acceptable vs doing a full postgis install.

So my first question is: Can I make a trigger in a geopackage? The answer is yes. For those of you who don’t know what a trigger does – it responds to some action in a database. In postgresql, I put multiple triggers in a database for a client. Some triggers calculate lengths and some areas and some make a note of who edited a feature.

After some digging I found the syntax and wrote a trigger for SQLite/Geopackage in the form of:

CREATE TRIGGER update_esn AFTER INSERT
ON addresses
BEGIN
update addresspoints set “esn” = (select esn from polygon where st_within(new.geom, geom));
END;

I used the spatialte gui to add the trigger. So if I pass this off to a client they can edit an address and have it automatically pull in the emergency service number. Magic? Eh – close enough for me.

While it’s not going to work for a multi-user situation (or will it – I’ve not investigated) – it works. So now I’m adding widgets to QGIS to aid in data collection.