ArcGIS Desktop

QGIS Topology and Emergency

I’m 44. When I was a kid probably one of my favorite shows was Emergency. Never seen it? You are missing out. Not terribly long ago I bought a digital antennae for a television and I discovered it’s on everyday. I had toy firetrucks and lego skyscrapers and everything but the best prop a TV firefighter/paramedic could have – fire. Sweet Sweet fire……

Anyway – I had a former QGIS Student give me a shout last week. They were asked to display in their GIS 5 mile road distances from Fire Stations. So every fire station needed a 5 mile distance band (not a buffer) to help with their ISO rating for insurance. I’ve never worked for a county but I’ve met people who do this and have had people describe the process. They are an ESRI shop. They don’t have any of the extensions. They have QGIS. QGIS fills in where they need extended functionality.

So the first thing I did was recreate this blog post. I pulled the Data into GRASS 7.0.4 and built a network and produced a 5 mile distance band. Except there was this one problem. The road shapefile (I hate shapefiles) wasn’t snapped everywhere so the distance band was wrong. I pulled the original shapefile into QGIS and used the Topology tool to take a quick look:

Every road should have anywhere from one to no red dots. As you can see several have two – one at the beginning and one at the end. In some cases you had roads that didn’t intersect and some where the roads just didn’t meet up. You also had some duplicated data.

So how do you fix it? Do you go road by road and check it? I actually like that idea and whole-heartily support it BUT…..there needs to be a quicker way to fix this. This is where ESRI is awesome if you have ArcEditor or ArcINFO or ArcGIS for Desktop Extended Enterprise Universe something. You can import your data into a Geodatabase and build some topology. Unless you’re stuck with Arcview………..

I sat down and started reading about v.clean in GRASS. V.clean gives you several tools from intersecting lines, to snapping, to removing dangles in data. GRASS is a topological GIS. Shapefiles aren’t topological. So I started writing directions on this small county shop to pull their data into GRASS. GRASS doesn’t scare me – it scares most GIS people.

Over the last year the processing tools have kept getting better and better in QGIS. If you’ve downloaded QGIS go to the processing menu and open the toolbox. It looks a lot like ArcToolbox. The Processing Toolbox gives you functionality from other software like GRASS and GDAL. You can search for v.clean:

So this is really turning into something a lot easier than I’d hoped.

I decided I needed three tools from v.clean: snap, break, remove dangle. So I ran it three times – once for each tool. Snap made sure lines met. Break intersected the roads. Remove dangle removed very small overshoots.

When I finished my cleaning of road data:

I took a look before I ran clean and I had 3200+ errors in QGIS. After I had 2600+ errors. I fixed a lot of things. I most likely messed a few things up. That’s why you run things with a small threshold or tolerance. I cleaned a shapefile. One command later (v.net.iso) and:

I networked my now clean shapefiles. They have their distance band for the fire station.

The plus on this is v.clean didn’t fundamentally change their data. Uppercase field names remained uppercase. The isoline portion did change the data as I lost some of my attributes and added one field (but that’s fixable).

Pro: Free Software It works. It’s now a documented process (that will get posted up here shortly). You don’t have to leave QGIS to run your tools.

Con: It does produce more than a few data files. The process is a bit convoluted. Shapefiles. It doesn’t work in GRASS 7.0.3 so you need 7.0.4.

The most important part: the county can process data to help their citizens.

So I get to the end of this and I know what you’re thinking. OMG who the freak wants to run data through GRASS? It works. You know what I’m thinking? OMG I hate shapefiles so how do I do this in PostGIS? The GRASS Tools are still exporting to shapefile. If I jump into GRASS I can push data back and forth to PostGIS.

Of course there are topology tools in PostGIS…………

The Cost of QGIS

The too long didn’t read version is “It’s mostly Free”. This post is more targeted to the new user or the curious user because I’m getting a lot of questions these days from those good people.

To be sorta honest I don’t care what software you use. I’ve got ESRI clients and FOSS4G clients. I have some that are a mix. I really enjoy a mix of FOSS4G and Commercial software just for the flexibility.

To be completely and entirely honest (which you may not be used to with consultants) – I care. If you walk into the door with nothing I’m going to push you in a FOSS4G direction. We will start with QGIS and work PostGIS into the Mix. If you walk in the door a full ESRI shop I’m going to suggest we mix it up a bit with some FOSS4G. Why – because you need to break out of your shell.

A few of my clients (who appeared on my doorstep over the last bit) went the route of “We are picking QGIS because it’s free”. I always groan a bit because while it’s free – it’s not free. Just because you have software doesn’t mean you have a Geographic Information System. Software is one component in a long process of spending money on a functional GIS. As I tell people all the time – “It’s the process”. Process trumps software and if you don’t have a plan to get from Point G to Point S it doesn’t much matter what you are using. So yes – Free – but – that’s only one part of the big picture.

I was going to do a top 10 reasons to used QGIS. Everyone loves a list but the more I thought about it the more I drifted into a top 15 and then it dwindled to 12. Then shot up….etc. Finally I just went with 5…..

It I use the QGIS Class I teach as a blueprint (and I’ve been re-writing it for 2.14.1 and adding extra chapters) for why I tell people to USE QGIS it’s the following:

Community. Join the QGIS Users list. I encourage them to post bugs on the hub.qgis.org website. Once you start participating in the community the value of QGIS goes way up. I will argue it is the most valuable component to QGIS. Paid support is nice because you can call up someone and yell at them – but communal support for your software….that is huge.
It’s Professional Grade Software. You can edit data. You can view data. You can manipulate imagery. You can view Imagery. You an make a map. You can consume standard OGC Services. It has small foot print so you might be able to leverage some lower end hardware. It works and it’s not that hard to use.
Going back to 1 – you have a say so in it’s development. You can vote by filing bugs or you can vote with your currency and donate/hire a developer or you can talk to the community. YOU HAVE A VOICE. That’s a powerful thing anywhere.
It’s Open Source. You can see the code. Which if you are like me looking at pages of software code is about as exciting as watching paint dry on most days….but you can if you wish. You can help document. You can write small articles on a blog for people to follow along.

If you decide at this point you want to use QGIS sit down and get a plan. Use a lot of words like shareholder and stakeholder but look for people who want this to happen and who want their work life to get better and more informative. Don’t start off going “Well the software is free”. You get this foggy discussion of “This is going to be really cheap because of free software”. It’s not going to be as cheap as you think. It helps you aren’t spending a ton of money on desktop software – but that frees you up to leverage that money in another direction – like better data…or GPS Data collection…or a server…

…….and then finally: 5. It’s FREE. Yeah it’s free. It’s an investment…and it just so happens it’s financially free to get this started. When I was an ESRI Certified Trainer I always felt a bit guilty (but it was good business) in seeing someone buy tens of thousands of dollars of software only to call me in for training classes that ran upwards of 500 dollars a day per person. Most times they had no plan or barely a plan so the training “helped” some but not really. The QGIS class is a bit different as I’m running it cheaper and different goals these days. If your budget is tight there’s nothing wrong with going “we’ve got to start out cheap” – just realize there are at least 4 other reasons to pick it that are better (in my opinion) than “It’s FREE”. Just so you know – I went from ArcGIS Enterprise….Desktop…Professional…..whatever it is called to QGIS/PostGIS. I went from Commercial to “Free” for my GIS life and Services. Why? 1 through 4 above.

There are 50 other reasons to use QGIS. I listed 5 and really 2 are repeats and for me because it’s all community oriented. To me it’s also fuI have slightly loftier goals than moneyn…each release brings more functionality (fun is not a good reason to pick software) and more stability. Some might go “It Crashes”. Yes it does. Find me software that doesn’t.

Anyway – if you stumble upon this article while searching for more information. Join the QGIS listserve. Call or email if you want – that seems to happen a lot these days and I like answering questions.

Happy QGIS’ing! Download it and give it a shot. You can Dooooooo it.

It’s not that exciting

I don’t even know how to write this up – but given the last few days it warrants some mention – although not a lot. It’s almost entirely self serving except I will stop short of any “I am so SMRT”. I’m not.

Many of you probably go “What do you do everyday?” and as I just put my cat down from an exciting back rub – I would answer “Geospatial stuff with data”. Most of my clients are desktop oriented and through the years they’ve been getting more complicated. When I started life as a business I remortgaged my house to buy ArcGIS….errr….ArcINFO. I worked. I had some spectacular successes and failures that could only be viewed with the help of popcorn and koolaid.

Over the last few years I’ve switched to almost all open source. I say that and I had to break out my increasingly outdated ArcGIS license the other day for some data issues. I’m on the ‘Commercial support for QGIS’ list – I became active with the community and pleaded my case and was placed on it. There are a few of us in North America on it. I do QGIS classes – probably my noisiest spot for the last bit – but I much prefer working. When I was a “commercial gis software” Business Partner I had three clients because of that designation. I’ve gotten two because of the commercial support for QGIS list. I should hit three this year. Maybe four.

I had a phone call from a prospective client last week that was the one I’ve been wanting. They had decided to forego the usual GIS setup and dive into QGIS and PostGIS. They had built a database and I’ve spent the last 24 hours realizing I don’t know enough on databases….again…and I know like 500% more than I knew last year. QGIS isn’t a problem – I’m learning more and hope 2016 is my year of writing a plugin. The kicker – this wasn’t a decision where they had purchased software and went “This won’t work”. This was their first choice.

It’s not that exciting – once we get the database worked out and the QGIS installs finalized there may be 20 or so people working off and on building geographic data. The description of the problem from the client was a commercial setup would be the equivalent of hiring an employee. In other words – they’d rather hire somone vs buying software. In my world the desktop reigns supreme still and it might be dying – but very slowly. So how did the person (and I’m not mentioning clients names) find out about QGIS? – the local gov’t office was using it as opposed to buying more commercial software.

If you’re on the QGIS list and you watch there are pretty big installs occurring outside of North America with QGIS and PostGIS. Here in the US the QGIS list isn’t that noisy – Except it’s changing. Slowly. When I work for someone I’m generally offering services. I’ve only had one person demand commercial software. So I keep moving forward with my goofy open source hippie software. This install we are working out will be not massive – but of good size. After a few weeks they should start seeing a return on investment that exceeds what I’m getting paid to do. The nice part once we get over the technical hurdles I can help them answer questions and make plans with the help of this open source software. I can be a Geo Person. I can worry about Data (with a capital D) because this is going to be a stable setup.

So it can be done. With this client under my belt it can be done even easier. You can roll out a GIS setup and help your company/group out…and not lose anything in the process. I used to go into these situations going “If you feel you need to switch you can – this is all commercial software friendly”. This time I didn’t mention it. They are here to stay.

So like I said – not that exciting…..BUT – exciting. QGIS….PostGIS…..Maybe Geoserver/Mapserver? I’m sitting at the ground floor of a clients discovery of GIS through open source software.

It sort of is exciting.

And there – I went the entire blog article without saying ESRI.

Georgia URISA January 2016 – Marietta Cemetery Project: A Mobile Editing Experience

In an effort to retell the stories of old by way of new technology, Marietta’s GIS group recently embarked on a journey to enhance its citizens’ cemetery viewing experience using ArcGIS Online, Collector for ArcGIS, Geocortex Essentials, and Esri Story Maps. This undertaking was Marietta’s introduction to mobile editing, and they’d like to share their experience with those interested.

Presenter: Ross Brewer – City of Marietta

As a KSU graduate, Ross began his career working within the GIS Core Group of Cobb County Government. He later followed an opportunity to remain in local-government GIS by joining the city of Marietta’s GIS group in 2009. He has since served two years on the GA URISA Board as Membership Chair, and now has a greater appreciation for the hard work of Board members and for the value of the organization as a whole.

Luncheon Overview: Luncheons are held on the 2nd Tuesday of every month unless otherwise noted. They are intended to provide a social and friendly atmosphere for learning about the successes of our members and to provide opportunities for networking. All are invited to join and students are especially encouraged! You’ll earn 0.1 conference attendance points towards your GISP Certification by attending each luncheon.

Agenda: Meetings are orchestrated according to the following agenda:

11:30 a.m. – 12:00 p.m. – Lunch and Networking
12:00 p.m. – 12:10 p.m. – Announcement
12:10 p.m. – 1:10 p.m. – Presentation
1:10 p.m. – 1:30 p.m. – Discussion

https://www.gaurisa.org

QGIS and Spatialite Part 1: Scene of the Crime

I had mentioned in the last post about the switch to spatialite for the Intro to QGIS class. I had a lot of driving to do a few nights ago so I started sketching more of it out. The second thought was I’m going to use the same data I’m using for the QGIS Class. Which actually leaves me with a pile of data I’ve been playing with as of late….which coincidentally was my first though: I’VE GOT SOME DATA ALREADY DOWNLOADED THAT I’M NOT GOING TO USE IN THE CLASS.

So – I’m going to do a series of blog posts on Spatialite, QGIS, things in general while I muddle through this idea of a class/tutorial.

So just to start off the playing field as level as we can:

Shapefiles: You are at least going to have three files that will be .shp .shx and .dbf. Since this format floats back to the early 90’s (as far as I could tell – quick wikipedia search lead me down that road) it’s using dbf to store data. So you are limited to 10 characters for fields and you also have a bout a 2 GB limit. You find them everywhere and almost everything reads them. If you were to say “I need one file format everything can read/write” – well – you have it.
Spatialite: SQLite is a file database. Spatialite extends the core of SQLite to store geometry. You have one file. Usually that one file has a .sqlite or sometimes a .db extension. It’s new(ish) with a 2008 appearance. It’s been one of those things that has long been said “this will kill off the shapefile” and it hasn’t. Mostly because ESRI will read them and editing them is a whole different story. The big thing you need to be aware of is the latest new kid on the block is called Geopackage and it is built around spatialite. With Geopackage everything should be able to read and write to one common format. There isn’t much of a file limit with spatialite – so you can go way over the 2GB mark and not worry about it.

Chattanooga has an open data site. In the open data site we’ve got some released crime data that shows that people really need to be nicer to each other. I grabbed it. Exported it out in CSV Format and from there took it to QGIS.

I opened it in QGIS (using the add delimited text layer tool) to take a look at it and 155240 features later:

Nice long column names and it has a lot of information. I’m going to dump it to shapefile:

….and column truncation but the file size dropped to about 4.5Mb and display time is good. It takes about 2 to three seconds to refresh. I really want my data to remain intact though – I don’t like that my CodeDescription column renamed itself to CodeDescri.

So with QGIS I’m going to save my CSV out as Spatialite. In order to convert/project data in QGIS you right click on it and go to “Save as…”:

So the conversion itself isn’t blazing fast. I didn’t time it but it was close to 40 seconds and that is just long enough to worry you. The spatialite database checks in at 53Mb. A little more than double the original csv but not horrible. I’ve kept column names intact.

OK Randy – you’ve taken a CSV and thrown it into QGIS and then thrown it into something and I can’t read it in ArcGIS.

The only Screen Shot of data you will get will be from ArcMap (Arcview 10.3.1). Note – you can’t edit it but you can read it (a lot better than my last check with 10.2.2)

So what’s next? What’s so great about pushing your data out of shapefiles into Spatialite? WHY?

Those answers and more are coming up in Part II.