open source

Year End Review: The Hook Up

OK – so this will be less dramatic than it sounds – but….this year I stepped back into partnerships. Except Partnerships sounds so formal – I always looked at it as “businesses that complement what I am doing – and (hopefully) I complement what they are doing”. To keep it shorter than that – Partnerships.

I always enter into it the same way: “If this isn’t working fire me”. Back when I started business I used to partner with a lot of different firms. I slowly dropped them all when I figured out it wasn’t working for me. Plus I don’t want to be a “one stop shop”. I can’t do everything. The things I do I want to do well though. So I’ve got this whole opensource thing going on, except I’m also using proprietary tools….like ArcGIS AND

Fulcrum: From that fateful day back in 2013 when I had a project rescued by fulcrum – I’ve been a fan. I’ve been in GIS a long time and over the last bit I’ve submerged myself in databases and data. Fulcrum scratches that Mobile itch. It’s awesome you’ve got a giant yellow GPS that gets readings down to a 1/5 of a cubit – but what do you do after it’s collected? How do you maintain your data?

So this year I kept getting the “Hey we want to put our data on a map” crowd. Yes – ArcGISOnline is happening – but I’m not a fan. I one time set up a online map and blew through 100 credits. No one could tell me what happened to them – so I just sorta stopped. Yeah – you can go cold turkey off ArcGISonline. So what am I doing? MapGeo from AppGeo.

MapGeo: A little bit of open source and a little bit of Google and a little bit of Carto – it’s the perfect hybrid solution. You’re not wrapped up in Credits and tied to one vendor for your GIS Life. It’s mobile friendly. It’s fulcrum friendly. IT’S QGIS FRIENDLY. It’s PostGIS Friendly. So what do I bring to the equation? Help. Technical Hand Holding. If you go with the full package you can load your own data. They will load it. I’ll even help.

…and that’s it. How undramatic was that? For me it was a bit step. As a one man, a dream, and a laptop (and 2 desktop machines) essentially goes “I trust these two companies”. You know what? I even turned down a partnership this year. I was approached by one exceedingly large company and it didn’t make sense. So I said “Love you mean it but I can’t”. I quickly walked away from that one. Not that it was a bad deal at all – it just wasn’t what I wanted to do.

Plans on partnering for 2017? I have none. Something may pop up but right now I’m happy with the 2 companies that I’m working with…or with which I am working to keep my english teacher from spinning in her grave….assuming she’s dead. She may not be. If you’re reading this I’m sorry. Anyway – I can help you collect data and put your “place” on the web. I can help you move your GIS life back into a good mix of open and proprietary – Or even better – Open source on the server and desktop, fulcrum for mobile, and MapGeo for your online presence.

Next up – another Year in Review and I’ll probably review training.

Adventures in LIDAR and GRASS in Tennessee: Part 1

There are days when work slows – today was one of those days. So I can either go goof off or learn something. I decided to learn something.

Back when I worked for the Federal Gov’t I was somewhat in charge of creating ortho imagery. At the time (i don’t even want to say when this was) we either grabbed whatever elevation data was available or we built the data in the photogrammetry department. About the time I had decided to leave LIDAR was coming down in price and we had just started the jump into that. Flash forward 8 years later and LIDAR is pretty common place. They have hardware small enough for drones and there are myriad ways to get data. You can even get data freely available from the Federal Government.

I know – it’s not LIDAR but it’s what I think of every time I say Drone. Stop making fun of Drones! No.

I’ve numerous and myriad clients and sometimes I wonder what I can do to make their lives better and expand available services. I also watch the twitters and I saw a workshop occurring just south of my location using ArcGIS and an extension to process LIDAR. So once again I was wondering about LIDAR. Can I process it? I know I have enough tools at my disposal – like GRASS.

I was recommended the NOAA Website to look for data. I did some digging. Guess what? LIDAR in my area is freely available now.

Tennessee has a very advanced GIS program. It started sometime in the mid-90’s and has taken a very encompassing view of GIS. They work at a state level for planning and coordination. The killer for me is most of the data is locked up at the counties or is for sale. It’s rare for me to convince a client to purchase any data – even data where we could really do something good with it. Counties or Gov’t agencies within the state used to take the view of “we paid for it and so do you”. Some still do. Tennessee released two counties worth or LIDAR and it appears a lot of the state is going into the National Map. So maybe things are changing for the better.

The TN GIS Server has the data. It’s small sets so I’m going to have to do a few things to grab it all. For testing purposes I only need a tile. I need a tile with Hardwood and Pine Plantations (yes – Forestry – I know what I want to test now).

TN has a very advanced program – but it’s all ESRI. So when I downloaded the LIDAR data it’s in zlas format. In case you haven’t kept up there’s been a little bit of a format war occurring. ESRI pushes one format called zlas which is (my opinion) ripped off of laszip. A lot of other people are using laszip. The short story is laszip is open – zlas isn’t. If you have ESRI software and you’ve bought the right extensions someone will tell you zlas. If you’re operating outside of that realm or are considering lidar – laszip. It should be laszip anyway – life is to short for locked up data formats. It’s not entirely locked up – I can build a tool to unlock it. I’ll get to that after I build a toaster to make toast.

So I want to do this one thing here – Tennessee OIR (or whoever) – Thank you for releasing this data. This is HUGE. The state sits on a treasure trove of data that it should open and this is a beautiful first step. Open up the format. Use laszip. If people want to go zlas they will – please don’t make your users start off having to deal with a proprietary format. I know you have a contract with ESRI. That’s fine – just realize there are more options than that one software vendor out and about.

So what do you do with presented with zlas? You open it up. For this I ripped zlas to las. RapidLasso made a toaster (of sorts) and are giving it away. In other words I took a 235mb file and turned it into a 1.3 GB uncompressed open format.

….and what good does that do? Well – part 2 is coming. Right now I’m watching GRASS gleefully process this data and I’m not sure what is going to happen. GRASS You say again? Not QGIS? You mean that really confusing horrible interfaced piece of software I once used in College. No. It’s the revamped new GUI interface that has 30 years of development behind it and it has all these really great tools for LIDAR. Yes – ye who have a limited budget and possibly can’t afford the flashy stuff have options. You’ve got a whole community of options. One being GRASS that you get when you download QGIS.

Selection_312 — Point Cloud in GRASS – total point count is “A LOT”

I know it’s orange. I actually hate orange but GRASS defaulted and I don’t want to spend that much time making it not orange – this is Tennessee after all.

FOSS4GNA 2016 – First Post

It’s over.

As one friend pointed out there are something like 4 stages of conference attendance:

Go for the Tech
Go for the Friends
Go for Free Beer
You Stop Going

On most everything I’m at 4. It takes a lot to drag me to a conference these days. I think my most attended conference is the Georgia URISA Conference. I’ve made three FOSS4G Conferences. For FOSS4G events I’m still at a 1 and 2.

There’s going to be several blog posts over the next few days on the conference. I can’t cram all my thoughts into one post. Plus my posts have been getting longer and longer and I really want to shorten them up a bit.

I’ve made the conferences in 2013, 2015, and 2016. Each one gets a little bit bigger…..BUT – each one has the same community feel I’ve missed elsewhere and enjoy. It really doesn’t matter who is the super star rolling into this conference – you’re going to have the same good group of people finely arrayed from business casual to “Well Day 3 for these Cargo pants”. You have people who really get don’t enjoy crowds and then you have the people who can command a room full of people. This conference accepts all kinds and it’s nice. It’s refreshing. It’s the way GIS should be.

We were somewhere around 550 strong for FOSS4GNA – which if you noticed there was also the ESRI SERUG or whatever conference running at the same time during this event. Overall – didn’t hurt us a bit. In fact – ESRI – please plan competing ones from here on out. It actually may have helped.

If you haven’t been to a FOSS4G Event you should go. This is coming from a guy who is at a 4 for everything. You will meet friends, you’ll learn something, you will walk away refreshed (and a little tired).

Like I said – more posts coming shortly. This was my get up and “stretch and warm up” post for the rest of this week.

Just in case you’re debating going to something:

Why show up at FOSS4GNA 2016?

I was debating this post most of the weekend – so I decided to set the egg timer at 20 minutes and run with it. Why show up at FOSS4GNA 2016? I’m saying “SHOW UP” and I know that’s not a good enough reason. Here are some statistics from the 2015 conference.

So you don’t want to show up because you’ve never been to one?

Almost 60% of the people there haven’t been to a FOSS4GNA event. 2015 was my second event.

I know what you’re thinking. I bet this thing is going to be full of those people – developers. I’m not going to understand half of what they are saying as they gulp down coffee and point excitedly at a half put together presentation they started 15 minutes before the presentation.

OK – so there are a lot of those people there – developers. Except – these good people are developing for you also. You can actually walk up to them and go “What were you thinking” and they will tell you. The one thing I was struck by in 2015 were the number of Government folk there – Open Source GIS just isn’t for Universities and Developers and people with weird itches to scratch. There were a lot of local government people from management down to technicians with GIS problems to fix and data to maintain. Probably the most interesting guy I met was a guy from LA County (county employee) who was going to use QGIS on a tablet for data display/collection. Another guy worked for San Francisco and he was throwing Geoserver on anything he could to decrease their reliance on ArcServer.

Are you alone at this thing? I consult and teach so I fell in the 13%. There are a ton of people there building commercial products. You’ve got the University people filling you about 16% of the group. I’m pretty sure a lot of Gov’t fell in the 17%. It’s a good mix of people. You are going to have a lot in common with most everyone there.

Finally as my egg timer is about to beep….

So I use FOSS4G in my consulting life. I’m using it almost 90% of the time these days if I have a geo problem to solve. If you skip to the last line – close to 10% of the the attendees don’t use it. They were there out of curiosity. I have several clients gleefully using it to maintain all their GIS data.

So show up. Come to Raleigh and look at the 10 workshops and close to 100 presentations on open source GIS.

oh and Thank you Eclipse Foundation for collecting this information….

Beep

What plugins do I use from QGIS?

One of the things that comes up in the intro to QGIS class are plugins. Plugins are an excellent way to extend functionality of QGIS. I used to make you pick and tell me why the plugin you installed was awesome. Generally I found that everyone was over whelmed and couldn’t decide in 20 minutes. I was having a conversation with a client and the question of plugins came up. So what plugins do I tell people use? So what do I use and how does that relate to ArcGIS Desktop?

This isn’t all the ones you could use – just the ones I use weekly (sometimes daily). Some of these are in core (i.e. you get them when you download QGIS) and some you have to download through the plugin manager.

Azimuth and Distance Plugin – What happens when someone hands you a pile of bearings and distances and wants you to build a property boundary? This plugin makes it pretty easy to put all that surveyed information into QGIS only to see it not close at the end (ha).

DB Manager – This plugin comes in core. It works with PostGIS and Spatialite. Since that’s what I use (and shapefiles occasionally) it works. For those of you in MSSQL land or Oracle land – there’s probably other ways to pass SQL commands to a database. You can import and export data in and out of those two databases. You can drag and drop data into QGIS. It’s a handy plugin if you’re using databases (postgis and spatialite) to store your data and I’m doing that very thing more and more these days.

Digitizing Tools – There were certain things I missed when making the transition from ArcGIS. I missed explode – this plugin provides provides that functionality. I missed clip (the one under the editor toolbar) and that is back. Cut layers, flip lines, and fill gaps between polygons. I don’t use every tool – but it adds enough tools I like to make it work.

Grass 7 – At the time this is written 2.12 is out. I made the jump from 2.8.X to 2.12 in the last few months. For those of you in ArcGIS land you have this extension called Spatial Analyst and several other “analyst” extensions. This plugin connects you to the underlying GRASS Software install and lets you view and run GRASS Commands from within QGIS. The big deal is – GRASS 7. For those of you who have opened GRASS and have went “Oh.My.GOD” – you are still going to do this….but it’s a lot easier to use. This plugin makes it easier. You get GRASS Software’s 30 year goodness into your GIS. Process Lidar….Pan Sharpen Landsat…and build networked vector datasets – and more.

Spatial Query Plugin – This plugin offers about every select by location choice you can come up with at any given time. Add two layers to the map create a new selection based on your parameters…and there are a lot of parameters. This one comes in core and it’s one I never turn off.

Topology Checker, Geometry Snapper, Geometry Checker – So I put three plugins in one description…why? Well – they are all three doing one thing – making better data. They are just all going about it differently. The one I use the most and will continue to use is topology checker. You can build topology rules and look at your data and look for errors. I haven’t used the other two enough yet but I’m working on it. More choices are good – and between these three you shouldn’t ‘bad’ data – and by bad I’m talking about Geometry.

Group Stats – Take your data and summarize it. Make Pivot Tables. Look at your shapefiles and Databases in different ways. It’s not the most intuitive plugin – but it can allow you to look at your data as if you had arceditor or ArcINFO and the much loved Frequency command. Granted if you have your data residing in a database you can throw SQL statements at it but it you don’t (or you want to do this graphically) this is your plugin.

AutoTrace – I love the trace tool in ArcGIS. Lutra Consulting developed something very similar to trace called AutoTrace. You add a new layer and you can trace an existing layer by pressing a combination of shift and ctrl to replicate that existing feature. As I sit here typing this I’ve only used it with polygons and not lines – but it works. It just takes a little gritting your teeth if you’re an avid ArcGIS user.

Georeferencer – When you get an unreferenced map/image and you want to georeference it? Georerferencer. It comes with core. Once you are done – you can add it to your QGIS Project. Polynomial 1, 2, or 3? Helmert? Projective? It’s here. I actually like having two windows for Georeferencing. Having them in one window is novel with ArcGIS – but I’m weird. I like them in two windows.

QuickMapServices – yes and for all of my “I USE QGIS” talking I do I just found this plugin. What does it do? Well – consider it a drop in replacement for openlayers. I love the openlayers plugin (and the web software) but I finally made it to a place where I dropped it out of the class. Why? It pulls in a lot of imagery sources (like Google) except you can’t make a map with it. With QuickMapServices you can make a layout. NOW – I saw all of that to say this plugin pulls in a lot of services and I’m not sure on how clear the licensing is – like with the openlayers plugin. I love pulling in Google imagery but you are forbidden from doing data collection with that layer – and to me the temptation is to great to have it sitting there….but I do have it sitting there.

What else do I use? There are a lot of things and not enough time. All of these are the weekly plugins I grab and use. There are others I use more sparingly like some of the CAD Tools or the MetaSearch Client. There’s a plugin that lets you draw squares and rectangles. There is one for surveying. There are probably a ton of others that I would find useful I haven’t stumbled onto them yet.

Does everyone need the ones mentioned above? I have no idea. I find them useful and I’m up to about 90% use of QGIS in my every day world. I would suggest giving them a whirl and if you have one I left out leave a comment. I’m all for spreading the knowledge.