North River Geographic Systems Inc

Spatial Problem Solving

ArcGIS

QGIS and Spatialite Part 1: Scene of the Crime

· ·

I had mentioned in the last post about the switch to spatialite for the Intro to QGIS class. I had a lot of driving to do a few nights ago so I started sketching more of it out. The second thought was I’m going to use the same data I’m using for the QGIS Class. Which actually leaves me with a pile of data I’ve been playing with as of late….which coincidentally was my first though: I’VE GOT SOME DATA ALREADY DOWNLOADED THAT I’M NOT GOING TO USE IN THE CLASS.

So – I’m going to do a series of blog posts on Spatialite, QGIS, things in general while I muddle through this idea of a class/tutorial.

So just to start off the playing field as level as we can:

  • Shapefiles:  You are at least going to have three files that will be .shp .shx and .dbf. Since this format floats back to the early 90’s (as far as I could tell – quick wikipedia search lead me down that road) it’s using dbf to store data. So you are limited to 10 characters for fields and you also have a bout a 2 GB limit. You find them everywhere and almost everything reads them.  If you were to say “I need one file format everything can read/write” – well – you have it.
  • Spatialite: SQLite is a file database. Spatialite extends the core of SQLite to store geometry. You have one file. Usually that one file has a .sqlite or sometimes a .db extension. It’s new(ish) with a 2008 appearance. It’s been one of those things that has long been said “this will kill off the shapefile” and it hasn’t. Mostly because ESRI will read them and editing them is a whole different story. The big thing you need to be aware of is the latest new kid on the block is called Geopackage and it is built around spatialite. With Geopackage everything should be able to read and write to one common format. There isn’t much of a file limit with spatialite – so you can go way over the 2GB mark and not worry about it.

Chattanooga has an open data site. In the open data site we’ve got some released crime data that shows that people really need to be nicer to each other. I grabbed it. Exported it out in CSV Format and from there took it to QGIS.

I opened it in QGIS (using the add delimited text layer tool) to take a look at it and 155240 features later:

crime_csv

Nice long column names and it has a lot of information. I’m going to dump it to shapefile:

crime_shape

….and column truncation but the file size dropped to about 4.5Mb and display time is good. It takes about 2 to three seconds to refresh. I really want my data to remain intact though – I don’t like that my CodeDescription column renamed itself to CodeDescri.

So with QGIS I’m going to save my CSV out as Spatialite. In order to convert/project data in QGIS you right click on it and go to “Save as…”:

spatialite

So the conversion itself isn’t blazing fast. I didn’t time it but it was close to 40 seconds and that is just long enough to worry you. The spatialite database checks in at 53Mb. A little more than double the original csv but not horrible. I’ve kept column names intact.

OK Randy – you’ve taken a CSV and thrown it into QGIS and then thrown it into something and I can’t read it in ArcGIS.

add

The only Screen Shot of data you will get will be from ArcMap (Arcview 10.3.1). Note – you can’t edit it but you can read it (a lot better than my last check with 10.2.2)

arcgis

So what’s next? What’s so great about pushing your data out of shapefiles into Spatialite? WHY?

Those answers and more are coming up in Part II.

QGIS 2.12: Check Geometries Plugin

· ·

QGIS 2.12 is out and I’ve been a bit quiet as I’ve been exploring it as well as working with it.

As my proficiency with these open source mapping tools grow, I’m starting to learn some QA/QC (quality assurance/quality control) methods for checking data. If data resides in postgis I’m slowly building out tools for attribution and geometry and I’m doing the same with GRASS and QGIS.

Somewhere it seems around 2.2 (I may be wrong) a topology plugin appeared. For those of you familiar with the way ArcGIS Topology works you can go in and flag and fix problems with your data. There’s a large PDF suitable for framing on the ArcGIS Site to tell you everything it can do. I like the Topology checks in ArcGIS – you find errors and you fix them. The only problem I had with the tool it was a license level tool – those of you who bought arcview couldn’t use it. The QGIS Topology tool came along and while it wasn’t as robust as ArcGIS – you had a tool for your data to at least flag (there is no automatic fix built in yet) errors.

So this very innocent looking data layer has at least two problems (four total but I only checked for invalid topology and overlapping geometry)

Very Boring Data
Very Boring Data

Running QGIS Topology Checks:

topology

There’s another tool with 2.12 that has been introduced – the Check Geometries Plugin. Once again – if you are familiar with the ArcGIS topology tools this won’t feel foreign at all.

check_geometries

So this tool (you need to activate it under the Plugins Menu) gives you a chance to check your data and FIX IT AUTOMATICALLY. It can check for sliver polygons, data that self intersects, data that may not match a minimal mapping unit, and duplicated data (and more). It checks for errors in a different manner than the topology tool. If I run this against my dataset I can find at least three problems. I attempted to set up the exact same checks I used with the Topology plugin plus one more check for holes or gaps:

results

Once you find a problem you can move around automatically and fix either manually or automatically the problems. I’ve been playing with it and I haven’t decided on a methodology yet for using the two in a complimentary fashion – there may not be one. Between these two tools you almost have the ArcGIS Topology tool in QGIS though.  I would strongly suggest playing around with it for a bit before you throw it into a production environment.

The bad – and I hate to say that this is even bad – It’s just different. These two tools really work independently of each other. So you’re going to be tempted (speaking to the former ArcGIS users) to use them both like they are a connected piece of software (notice I said above I’m attempting to work out a methodology to use them together because it’s too tempting). If I use both tools to look for all the errors in this layer the topology tool wins with 4 total errors while the geometry tool finds 3 errors (I built the dataset with 4 errors). The geometry tool does check for minimum mapping sizes and that’s something the topology tool doesn’t do. With another dataset it incorrectly flagged some geometry as errors that weren’t. Don’t worry – Bug report was submitted. It may not even be a bug but it doesn’t hurt.

Overall – check geometries is a welcome addition. You just need to be aware it’s different from the Topology tool. With the Check Geometries tool being “brand new” (so to speak) I would expect changes as feed back starts coming back into the QGIS project.

Back when I was 17 I bought a jeep. At 17 I thought it was the sexiest thing ever – most everyone else had determined I was driving my future coffin. I ended up at my Uncle’s house and I had been working on every part of the car that seemed to have problems. With a stethoscope, a hammer, a few electrical checking gadgets he found about three more problems I had completely missed. I was bummed. “I wished you weren’t finding problems with my jeep”. To which he pointed out “I’m not making more problems – just finding the ones that exist.” You now have more tools in you Desktop GIS to find errors and make clean data.

A Tale of Two Topologies

· ·

It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of FOSS4G, it was the season of Commercial Software, it was the spring of hope, it was the winter of despair, we had everything before us for the right price, we had nothing before us because we didn’t RTFM.

My apologies to Charles Dickens. My compliments to the guillotine.

To say that this dataset had some miles on it is an understatement.  It had been a shapefile. It had become a geodatabase. It went back to shapefiles. It became 4, 16, 80+ shapefiles, and has now been moved to PostGIS and I’m down to 20+ tables plus a lot of Foreign Keys. Why? Well – I upgraded a client from ArcView (I know it’s not called that) to QGIS/PostGIS. Why is it an upgrade to Free Software? Because we have more functionality and much much better data. I’ve talked about these guys before – and I have some posts I’ve left hanging but now we talk about topology.

This data in all of it’s changing and moving had developed some problems. I will accept full blame for the problems developing because I’m better than this data looked. Last year I made the choice to abandon shapefiles and learn postgis so the last jump was made and it has made everyone’s life much better. Don’t tell my client but I tell them it takes a day to prep the data for delivery to their client – it really only takes an hour now (it used to take 24 hours) – and in those extra hours I clean up the data. No one has noticed, but me, of the problems with gaps and overlaps and zero area polygons. I like clean data. I guess I am a dying breed that still thinks the data is the most important thing and not having all your data sitting exposed in all of it’s terribleness out in some online story map thing.

For kicks last week I started on the topological errors and the first thing I did was read up on PostGIS topology. I started. I failed miserably due to some error that made no sense and I gave up. It’s on my list. I will “get it” eventually – and I was close this time. Really Close.

So for my next trick I pulled up QGIS and looked:

qgis_topo

For those of you that don’t know – shapefiles aren’t a topological dataset. It’s easy to get things in a mess if you aren’t careful. QGIS can check shapefiles for topological rules (Arcview can’t last time I checked – license issue). In this case I was checking for gaps. What I see is a lot of red that are problems from GPS data and problems that appeared when I pulled this dataset apart two years ago and recombined it to fix another problem.

So what do I do to fix it?

Well – for fun I pulled it into ArcGIS. I dumped my data out back into a shapefile (shudder) and imported it into a file based geodatabase. I built topology (I’ve got ArcINFO – and I know it’s not called that anymore) and the topology build removed all the errors in about 15 to 20 seconds. I know – I’m mentioning ArcGIS as a solution. Hey – it worked.

arcgis_topo

BUT – I’ve switched my client to QGIS and PostGIS – do I really want to fall back to ESRI as a fix? No – I’m going to make this work with the tools my client has available.

So I started reading up on GRASS. You get GRASS when you install QGIS. Right now I think it’s GRASS 7 (or at least on my machine it’s Grass 7) and GRASS has this wonderful 30 year run. Yes – the software has been around for 30+ years. GRASS is one of those things that if you ask someone you first get 1. A Drug reference and 2. I used it in College once with another Drug Reference. Of course it’s 2015 and with the number of colleges flailing with GIS you’re not even getting a 2 anymore. Just glassy eyed stares of “Well I made a Story map”.

Grass 7 is really – for lack of a better word – pretty now…or nicer. I’m not sure how to describe it. Grass 6.x always to me was a bit confusing – GRASS 7 isn’t. Maybe it’s me. Maybe not. The user interface is more intuitive – the Help button does just that. It’s not as “mysterious” as it has been in the past. I used it and with a little effort (I need to RTFM more) I was working. It’s powerful. To get QGIS/GRASS/GDAL and all these things in one QGIS install – that’s huge. There’s no excuse for you as a GIS person to not have these packages installed on your computer.

grass

GRASS makes me nostalgic for the old days of Arc Workstation. GRASS datasets are topological. Problem being if you’re using the GUI for import there is no place to set snapping (that I could find). So I imported data and I kept the problems. After reading the manual I did this little command: v.in.ogr /export/data/topo/stands.shp out=stands_snap snap=0.001 and re-imported my data. I had Clean data in about 15 seconds (probably less). What happened is GRASS pulled it in and snapped the data as it was being converted (same as ArcGIS). If you watch the command line you get a lot of information as to what it’s doing.

I checked my ArcGIS software. I checked QGIS. After all this was done I had 10 problems using the topology tools in both softwares with this new clean dataset. 10 problems that took me 10 minutes to fix. Please note I had the exact same problems in ArcGIS and QGIS – no difference.  I took the GRASS snapped data and put it back into PostGIS and I’m done. Clean data now. Happy Clean Data as I invoke my inner Bob Ross.

So what did we learn from all of this:

1. If you aren’t worried about topology as a GIS Person – turn in your badge at the door. If you aren’t worried about your data you’re a terrible person.

2. If you believe you can’t create production professional data using FOSS4G. Turn in your mouse and your half completed GISP application. I took QGIS/PostGIS/GRASS and created a topologically clean production dataset in a little under an hour. My next dataset will probably take about 15 minutes.

3. I failed slightly – I could connect GRASS to PostGIS and I didn’t. I would have avoided the whole “going back to shapefiles” thing again. That won’t happen next time. It’s going to be connected from here on out. It will be pulled into Grass and pushed back into PostGIS.

4. My hope is that I get my topology problems figured out in PostGIS. I’m close.

Anyway – excuse my slight irreverence in this post as I poked a bit. I walked out of a professional meeting the other day where I got the “you only use QGIS because you’re too cheap to buy ArcGIS”. Well……no. So expect the next few posts to dive into a lot of technical as to why this is a good way to work. Plus you get commercial support from a lot of companies while doing it. It’s a good choice. Granted – there are a lot of tools out in GIS land – but picking these tools doesn’t indicate anything about the user or your organization.

Clipping in ArcGIS and Clipping in QGIS

· ·

How did I end with this blog post. I’m using QGIS for about 90% of my GIS work these days and I spent about 6 hours digitizing some fairly complicated land cover. Out of that – you get the following…..

It was always one of the things that was hard to explain to someone brand new to GIS: ArcGIS has two clip tools.

If I run into one problem when cleaning up data is polygon overlap. Someone will draw a polygon and then draw another either over it of overlapping it. In classes and with clients I always tell them “Imagine the world is flat like a pancake. In order to get polygon 2 “into” polygon 1 you have to clip/remove that overlapping portion with the editor tool”.

Example:

one

You have a polygon

 

two

 

You add a second polygon partially occluding the first

three

 

You initiate the clip tool on your editor toolbar

 

 

four

 

If I remove the second polygon you can see how the first polygon has been clipped

So where is the clipping tool in QGIS? Well…..there is a duplicate geoprocessing clipping tool. There isn’t a “editing” clip tool built into the core of qgis. Except there are ways of “clipping” that will accomplish the same feat in slightly different ways.

  • If you are digitizing polygons and you want to digitize an interior polygon (lets say put a body of water in a field) you can use the “fill ring” tool to digitize and it instantly clips the big polygon to fit the small polygon. It is found on the Digitizing Toolbar.
  • Clipper plugin. Select the smaller polygon and click the clipper button and it clips.
  • There is another plugin called Digitizing Tools. If you do any amount of digitizing in QGIS I put this one up there with “Needed”.

DigitizingTools

 

  It splits multipolygons. It does all sorts of Trickery with your digitized lines…..and it also clips. It clips against a different layer. A while back I talked about Memory Layers and Scratch Layers.  Well – before I found the clipper plugin I was copying data into memory and running “Cut Polygon with another layer”.   Lets say you have multiple polygons that you want to clip against the big one.

  • Select and Copy them into a Memory Layermemorylayer
  • Delete them out of the original layer
  • Run “Cut polygon with another layer” and that other layer will be the memory layer (defauly layer name is Pasted).
  • Copy and Paste them back in if that’s the end goal.

Example:

digitizingtools
BEFORE
digitizingtoolsafter
After Digitizing Tools

 

Would I want to do this on lots and lots of features? No. This is for digitizing and not geoprocessing and clipping thousands of things against thousands of other things.

Clipper might be able to do this though. The few times I tried I created topology issues so clipper has been to just clip one thing against one other thing. I have always like the clip tool with ArcGIS but it can only run one polygon at once. Is the QGIS way better? Not necessarily. Is the ArcGIS way better? No – it’s just more familiar. Which ever software is your weapon of choice you can do the same feats of GIS trickery.

Is Desktop GIS dead? I still say No.

ArcGIS lyr files and QGIS qlr files

· ·

I spend a ton of time talking about QGIS up here. I like QGIS. I like GIS in general and there are some days I wonder if I’m unintentionally ignoring the ESRI side of life. Actually I don’t for clients – 20+ years of Using ESRI software leaves me in a good spot to answer all sorts of questions.

So I went to visit a potential client this last week. It’s funny how a 20 minute visit can turn into a 3 hour discussion of GIS. Short story – we covered two pieces of software: ArcGIS, and QGIS. Shorter story – they are moving from ArcGIS to QGIS.  It nice getting the view of someone casually using GIS (both packages) software who doesn’t do this day in and day out.

The conversation covered a lot of ground. One of things we talked about was lyr files. As a former instructor in the dark arts of ArcGIS, lyr files were one of those things everyone seemed to know something about and no one was entirely sure what to do with them. You see a lot of “Can I use LYR files in QGIS?” Well – no but here is a story on lyr and qlr files.

The short story on lyr files:

1. Symbolize your data layer in ArcGIS.

arcgis_lyr

2. You can even set up a definition query to single out a particular feature in your layer. In this case I’ve singled out a tract of land a client works.

3. Right click your layer in the table of contents and save a lyr file.

Lyr files are super handy. You can symbolize data. You can label. You can set definition queries and save it all to a lyr file. ArcGIS treats an lyr like data (BUT IT’S NOT). You can add them by hitting the add data button so a lot of times people can and will get confused and think an lyr is data. So you will get emails from people going “Here I sent you some data” and it’s a bunch of lyr files. I generally make it a point to tell people to name them and make a different folder and keep them all there. Treat them as part of your project. Treat them as important. Don’t think they are data though.

BUT – What about QGIS?

1. Symbolize your data in QGIS

qgis_lyr

 

2. You can even set up a feature subset (definition query) to single out a particular feature in your layer. In this case I’ve singled out a tract of land a client works.

3. Right click your layer Layer Panel and save a layer definition file (a qlr).

Notice I tried to give this an identical three step description. You can’t add qlr files with an “add data” button in QGIS. You can drag and drop it onto your map canvas or use Layer -> Add from Layer Definition File to use it. No data is saved. This is just a pointer back to your data with symbology and other info saved.

Quickly…….

  • So can you use a lyr with QGIS? Nope.
  • Can you use a qlr with ArcGIS? Nope.
  • Can you define symbology in ArcGIS from a lyr file? Yes! Just import your symbology and choose your lyr and if the attribution matches you are good.
  • Can you define symbology in QGIS from a qlr file? No – but that is what qml and sld files are for. Save those from your property menu and you can use those to symbolize other data.
  • Can you open a lyr in a text editor and see how it works? NO…I think the greatest shortcoming of lyr is this.
  • Can you open a qlr in a text editor and see how it works? YES…it’s an XML based file.

So as always make your life easier. Save lyr and qlr files with your data because you don’t want to spent tons of time symbolizing it over and over. Remember – neither qlr or lyr hold any data – just symbology and data queries. So don’t email your friends or consultants lyr or glr files.

Explore more here and here!

**Update

 

Contact