49 Responses on the QGIS Support Poll.

Before I post the results a bit of “thinking”…..

It was an interesting poll. Scientific? No. Slightly revealing? Yes. I’m going to redo the poll and make it permanent on the website – I also want to make the results on permanent display. Maybe it can be a running indication of what people are thinking with regards to open source and support.

So I teach QGIS classes and it’s interesting. I keep a log of who comes to the classes and mental notes of things said before and during class. About half the people are Government. Another 25% are from a university, and about 25% are private business. Throw in the occasional non profit and student or person I let in for free just to mess with the percentages a bit.

When people come and take the QGIS classes the general attitude is:

I want to see what that “Free stuff” does. It can’t be all that good – because it’s free. By the end of class most feel “I can use this to supplement my GIS Program”. That’s partially what I preach these days is to care more about your data than your unforgettable brand of software. That’s not entirely fair because software does matter. If you’ve sunk $250,000 into an ESRI Solution I’m not going to tell you to junk it and walk away.

Since I didn’t ask people if I could post their comments I wont’ repeat them verbatim:

  • There is a market in <not the US> for commercial support of QGIS
  • This poll is misleading because QGIS isn’t ESRI Software (I did a terrible job paraphrasing that one – but I’m pretty sure I know where this one originated from…)
  • I need programming support more than technical support. I need plugins written.
  • Some users require (need) paid support to get their companies to support it.
  • QGIS has commercial support in the UK
  • I only use QGIS when I can’t do things in ArcGIS so I won’t buy support.
  • QGIS is making inroads into the university environment.

Is NRGS going to provide commercial support? Eh – I have no clue. I argue it in my head that it’s a good thing and a bad thing. I use QGIS for probably 70% of my project based work. I’m still using ESRI for most cartography and for random things I can’t get FOSS4G to do easily. I picked up a client that I’m going to be discussing quite a bit and I’ve two directions for a end “product”. Based off conversation this week the client will be a mixed software environment with an open source back end and a commercial front end (maybe). This is all still in the design phase.

With all of that…..

Screenshot from 2014-08-24 11:00:55

 

 

 

Finding Nemo….errrrrr Data.

This all started off with a small project I am doing for a non profit. The project will possibly get a bit larger as time goes on but we are still in the beginning stages of it. My client has no knowledge of GIS. They’ve been haphazardly locating hospitals with which to work and placing them on a map. Which to us seems crazy, but to them it gets the job done. I’ve been trying to put together an estimate for work and there are way too many unknowns. One estimate had me way to high….one had me way too low.

So yesterday I started digging. I won’t name states – but I will give you an overview of the process. I start this with “They’ve been hand digitizing hospitals”.

  • My first thought was the National Map…except something like Hospital locations are way outside the scope of that project.
  • My next guess was census which seems to have some hospital locations but they are polygons. I would like better than centroid and I would like something more “authoritative” coming from the states.

I decided to visit each states “GIS” setup…department….portal….website…..library. This takes place over 5 hours….I think…maybe more.

  • With two of the states I was able to google “Statename Hospital shapefile” and find exactly what I was looking for in less than two clicks.
  • Every major university has a half-assed page devoted to GIS links. Yes I said half-assed. Most of these aren’t curated or maintained.
  • In every search mapcruzin was ranked at the top or close to the top for data searches. I have 0 idea where this data came from – so much so I didn’t care to even investigate it.
  • One state’s search returned metadata (Yay) which pointed me to an online link that was dead (Boo). I vented over twitter and ended up using the Internet Wayback Machine thanks to @jalbertbowdenii.   He then pointed me to the new states data “portal” and I found that data. I had been searching for “Statename GIS Data” and was really coming up empty. All that came back was a website with no clear link to data. Lots of pics and mentions of awards and presentations…..but data.
  • One state search lead me exactly to that states data portal. A search on that portal for “shapefile” returned 0 results. After some exploring I discovered a second link to another portal which was down for maintenance. I ended up emailing the admin who has pointed me in another direction. There was another link that took me to a google map of all the states hospitals with no way to look at the data – just the map.
  • One state had all of their data as either “statewide” or “county”. If you didn’t find data in one you had to look in the other one. Of course hospitals were called “acute care health facilities” or something that to me said “here is where I will be at 70″ and not in a hospital. Data was in a executable zip file.
  • One states search had me register with their data portal. I downloaded the data only to find it in an ESRI personal geodatabase format. I have ArcGIS Desktop so I opened it and exported it out. I could have done this in QGIS – but time. This website also had a ton of .e00 files. What are those you ask? Look it up kids.
  • One state’s data portal was being replaced…or had been. So all of their links were dead. One link did work – the states Health department where a search for hospitals returned a 50 some odd page PDF with hospital information and addresses scattered about.
  • Right now I’m down to 2 emails and I’ve got the region covered in hospitals. except now I’m up to 8 hours of time spent on something that shouldn’t be this hard. I know 8 hours compared to hand digitizing all this data is nothing…but….8 hours to find location data for hospitals? I’m not searching for the nearest fast food restaurant. I’m searching for a building to provide health care.

So I’ve decided that the idea of accessible GIS Data must really be a joke on some level. We hear about it. We talk about it. Vendors tell us how open their product is and how data is being shared. I’m up to 8 hours in my search. Crazy that they were digitizing this data by hand? Not so much when you look at what you have to go through to find data. How hard is it to find data relating to hospitals for your state. Look. In some cases it will be easy…in some……it’s really a joke. Open data doesn’t mean much if you can’t find it.

No two states had the same setup. Almost all on some level were discussing their “data portal” or “GIS Portal”. Only in one instance did metadata come into play….and it had the wrong online linkage. Why not put your metadata up as a page with the right online linkage? 10 states, 10 pages, 10 links, and 10 downloads. Silly – because you don’t get to spend half a year building a portal with pics of people and awards and presentations.

For those of you with data portals that are jut that – spots to share and download data. THANK YOU.

Except – I can download Mars Geology Shapefiles. Use Google for that and it’s the first return….

 

 

ArcMap won’t kill off ArcInfo Workstation

If I remember correctly I was in Knoxville and we had gotten the first look at ArcMap 8.0. It’s been too many years ago to care or to count – I just remember sitting there going “Dammit – I spent all this time learning AML and Avenue”. The word being passed around as I remember it was ArcMap was the beginning stages of killing off ArcView 3.x. ArcView had made tremendous in roads and had pretty much became the go to standard for all my clients when I worked for TVA. Workstation was safe. I loved workstation. For all it’s command line craziness I knew one thing – if someone said they were using workstation they had a clue as to what they were doing. One of the managers with strong ties back to Redlands said during the class “This will kill off workstation”.

As time went on – Workstation on Unix slowly moved to Workstation for NT. ArcMap turned up in the office slowly around 2001 when XP finally appeared. Eventually I opened workstation less and less as coverages were replaced with shapefiles and shapefiles were pushed into a geodatabase. Eventually we moved into SDE which killed off a beautiful workflow we had set up with ArcINFO workstation. We started hearing about pushing maps into 3-D and onto the web. I still hear about pushing maps into 3-D and the web.

For the last year I’ve watched ArcGISOnline creep into everything. It’s not a bad thing – but I don’t really consider it a great thing. Call me your crazy uncle who turns up to thanksgiving late in a rusted out jeep, but I’m still not convinced the data is secure. I will say the same for any “cloud” environment in which you don’t hold the kill switch.

So anyway – I spent the morning reading up on ArcGISPro. Watching videos. I started having flashbacks. ArcWorkstation won’t be replaced. Arcview is safe. Deprecation. Will Arcmap run with workstation….Will ArcGISPro run with Arcmap….

I won’t even get into 3D.

For the record – it may take a few years but I consider this the preliminary announcement for the end ArcGIS for Desktop. The desktop platform has gotten stale. This is it. The next “revolution” will be ArcGISPro. Still with the tiered licensing from what I can figure out. “Hi you’ve just purchased ArcGIS Profesional Basic for Desktop”.

Innovation is great. Getting ESRi into 64bit for the desktop has long been overdue mainly because I still consider Desktop to have too much 80/90’s workstation core at it’s heart. You have to rewrite it to get rid of it. If you are going to rewrite it you might as well start killing off the old product line. Software changes and it’s not a bad thing….but – there’s a lot of “transformational” talk during this video.  Which usually means transforming your budget also.

It’s worth the watch – if only to remember the palm springs demo that was rolled out for years. I do miss the palm springs demo.

 

Charlie

So you know you’re getting old when you get random emails telling you a former co-worker has passed on. I’ll add him to the list. We all get old – life etc. Of course I’m telling everyone the obvious.

So my former life at TVA was a mixed bag. I hate politics. I hate red tape. Those two single things made me a complete pain when I worked at TVA. There were two GIS shops in TVA – one in Chattanooga and one in Norris TN. Of course the one in Chattanooga was better – since that was the one I worked in…or at least that’s what I believed. The other shop in Norris was headed up by Charlie Smart. Charlie never was one to get excited about red tape or politics – I would sit through a meeting and would almost immediately go “this is bullsh*t” because – well – see the previous few sentences. The only time both shops were together was news of a re-organization. Charlie would almost always sit there smiling and go “well – that was a good meeting”. Charlie had been doing this a lot longer than I had – and he was used to the ebb and the flow. It was something I could never get used to when I worked there. I guess you could say I’m better at it now. Somewhat.

Charlie was doing GIS at TVA back in the 70’s. Who knows if it was “GIS” at that point. Science so new people are most likely going “This will never work”. When I came along he was probably 30 years into his career. He also was into shooting high powered rifles at a gun club. People would tell me to speak up when talking to him.

Anyway – I never had a chance to work with the man enough. He retired shortly after I left.

So Joyce, Bruce, Scott,..now Charlie….The list goes on.

2692641_220w

There are weeks like the last two were I think I’ve been doing this way too long.

Hard Work

Sunset
Sunset Above the Clouds

“An idea I live by comes from a John Wayne movie called Rio Lobo.  John Wayne and Jack Elam are on one side of a crick shooting their guns at the bad guys on the other side.  John Wayne looks down at Jack Elam and says “Scatter gun’s useless.”  Jack Elam says “Don’t mind if I shoot, do you? It just makes me feel better.”

“I love that attitude.  If you don’t shoot, you have no chance of killing the bad guys on the other side of the river.  If you don’t shoot, you have no chance of making a basket.  You have to be willing to try.

“I balance that with an idea I take from Clint Eastwood’s movie Magnum Force.  Toward the end of the movie, Clint Eastwood squares off with a dirty cop who’d tried to kill him.  He secretly activates a bomb in the cop’s car and as the cop drives off his car blows up.  Clint Eastwood says “Man’s gotta know his limitations.”

“That’s the other thing I think about when I’m coaching.  Everybody should understand what they can and cannot do.”

Roy Williams, Hard Work: A life on and off the court

For those of you who do not know, I was born and raised a Tarheel fan and that definitely gets me in trouble this time of year living in Kentucky.  I may bleed the wrong color blue (powder instead of big), but at least once a year and often twice one of my favorite teams has to lose.  But these words of wisdom comes from a coach who just earned the title of fastest to 300 wins in the ACC and they seem to be particularly applicable to me the past few days.  Plus, Coach Williams is fiercely competitive which is a trait I can relate to entirely too well.  Tonight, I am hoping my fellow Kentuckians will forgive my “incorrect” powder blue preferences and unite with me to beat the true enemy.  Go Heels.  Beat Dook!!!!

SnowLine
Snow Line
SnowyFarms
Snowy Horse Farms at Dusk

I first heard this quote (thanks to Audible audiobooks) as I drove on snow-covered roads to the airport to attempt to do some remote sensing.  I wasn’t particularly happy about this outing because if you think about the percentage of land that had good remote sensing conditions last weekend is incredibly low.  And Kentucky had just received some more snow.  So much snow that our first attempt of flying Saturday had been thwarted completely.  I was totally skeptical and ready to turn around and go back to bed.  But, after about an hour of scheming, we took off.  We almost had to go back, though, because I didn’t have video so I couldn’t tell if the camera door was frozen shut or if the vide was just being finicky.  Scary and embarrassing at the same time.  But, I managed to get things working and it turned out to be a pretty successful day of 20 lines spread across four states (AR, TN, MS, and KY) and 395 images.  (It has been QC-ed and everything was accepted!!!)

SunsetProp
Sunset on the propeller

If I had gone with my original gut feeling and not shot the imagery, I would have been really frustrated.  Not to mention the fact that I probably would have been driving back on roads that were still nasty from the snow we received overnight.  Instead, I worked.  When I drove home that night, the roads were nearly clear and we had several hundred images “in the can” so to speak.  Productivity! And, along with that, we were smart and knew our limitations on Saturday.  Yes, we probably missed some photography but we were safe.  And, yes, I love my job.  Writing very detailed FGDC Remote Sensing Extension metadata one day and flying the data the next!

But this quote is one I am going to try to take to heart in more ways than just remote sensing because I truly believe that hard work has the potential to pay off in all aspects of our lives.

Code Like Crazy

A mentee came across my “mentoring coordinator” desk.  The mentee was looking for a mentor who would help with programming skills and career paths.  While I am in no way qualified to help with either of these questions, it has me thinking.

My first thought is that the programming language choice doesn’t matter.  A solid base in object oriented fundamentals would be more important than choosing a language.  They’re ever-changing and the next best thing is around the corner, and the older stuff is  constantly undergoing revision or threatening to become extinct.  (Ahem, VBA, I love you…but…I wish I’d never met you.)  That leads me to training sites such as Lynda.com and GeospatialTraining.com.  Lynda doesn’t have anything specifically GIS, but their fundamental courses are excellent and teach JavaScript, perfect for the next step into GIS related training through GeospatialTraining.  There are always other courses through universities or your favorite web training system.  Some searching may be required, but there are tons of virtual courses out there on programming.  Better yet, there are in person training places such as, you know, the hosts of this blog.  (And, by the way, I found this link while writing and it is a very good read.)

My next bought is that different places have different language specifications.  Often because they do different things (duh…web, server, desktop applications).  Since I really can’t talk about career planning, I can’t imagine how to find out what programming language your next job would desire.  Based on my own experience, though, I would recommend picking an open source language because those have seemed more stable over my years of experience.  The rest of my story is strictly a narrative, but maybe someone else will see the crazy path I took and find it useful.

So, personally, my paths of coding language choices has been forked and has several roundabouts and even a few dead ends.  My first exposure to programming did not go very well as it was a hurried introduction to C++.  I was running away screaming.  Then, in order to be accepted into a program, I had to take a programming course. Luckily, the second time it took.  I had a much more relaxed introduction with C++ that I loved.  It was so awesome that I almost changed career paths and went to programming full time.  Although I look back on the choice to stick with the spatial route now and then, I’m glad I’m spatial.  (Yes, you are supposed to read that special.)  But that well-taught introductory course paved the way for me to dabble in Perl, Avenue, VBA, Python, JavaScript, and, yes, I’m old enough to have even played around in AML.  Best of all, when I received my first job I was able to say I knew how to program.  I was even lucky enough to be able to jump from C++ to C#.

Every once in a while I get frustrated because, let me tell you, for ArcObjects, Visual Studio Express doesn’t work so well! If you can’t tell, I learned that the hard way.  I had a piece of code functioning in Visual Studio, switched it to express and nothing.  I rewrote the whole thing, checked all the logic and then finally realized it was just that I was using Express.  That put me off coding for a while.  Especially because the help for Express couldn’t tell me why I was having such a hard time.  Everything should have worked, but it sure didn’t.  So, hiatus.

Speaking of hiatus, I think it is incredibly important in learning to program to set attainable goals.  It should be logical to anyone who has had any programming experience that they need to make a change and then debug.  But…well…every once in a while I forget this and bite of way more than I can chew.  Thus, I’ll code like crazy for a spell and then nothing for a long while.  Goals.  Small ones. They’re good.

Back to the story.  There’s the fun fact that I can only use Avenue on an XP machine since ArcView 3.x doesn’t like Windows 7.  That’s pretty useless.  And, just when I thought Esri had a great training solution to switch from ArcView 3.2 to the impressive ArcGIS Desktop, which I partially did, I am now running into ArcGIS’s threat to do away with VBA for years now.  Great.  Now I have two piece of useless code.

The moral of these stories is that finding a language you can write without any expensive IDE and software ‘s flakey support and keep small goals.  You can always branch out later.

Shout out: This weekend (February 21-23, 2014) Code For America is doing “Code Across” with events many, many places.  There are some GeoGeeks involved (I’m excited Lyzi Diamond is in Lexington, Kentucky) and it is just awesome.  So check it out.

Side note: if you are interested in signing up for a mentoring, I recommend two options: URISA’s Mentoring program through the Vanguard Cabinet and ASPRS’s Mentoring program through the Young Professionals Council.  Both are seeking mentors, mentees, and mentoring coordinators.  I am actively involved in both so please feel free to complain directly to me if you don’t hear from us in a timely manner.  So far, though, I have heard great stories and also had excellent experiences.  It is always a tremendous amount of fun to have a conversation that doesn’t start with an explanation of what you do!

Scholarship to State of the Map US 2014

I stumbled on this article and it made me chuckle. It shouldn’t at all. I guess given what all we’ve been experiencing with the OSM Crowd I’ve gotten a bit jaded (way more than Carol) about the whole ecosystem of OSM. I’ve got a blog post coming about the entirety of OSM….and I’m not sure exactly how to put it all together without coming off as a goob.

The article for GIS Lounge starts off with “scholarships for women to attend state of the map” and ends with “scholarships to State of the Map US isn’t restricted solely to women”. I get it. If there was ever a spot where more women are needed – it’s OpenStreetMap. Like with most mapping things – group is male heavy. I’ll even go with white male heavy. We need more women in the field. OSM needs more women contributing. We do here in the US. We do in the whole GIS industry.

My current theory is the OSM community is actually quite small. There are 1,000,000 accounts. I would put active users probably less than 50,000 after doing some reading.  I could probably even narrow that down some more (half). It’s a small group. It’s a loud group with no central ruling body and it’s all truly crowd sourced. Self forming groups. Self forming committees. To me a group that fears the new user – they need the new user though. They really need the disruption of the new voice.

I say they don’t and I can point to dozens of instances where this isn’t true. I can point to a lot of people who happily want new people, innovation, and change. I spent last weekend working in the US Virgin Islands fixing roads and mapping cities. I may spend some time this week working in Athens Ga. AthOSM is coming shortly. I plan on continuing my disruption.

Scholarship ends today – apply and go. Go if you are white black male female or other. Learn about OSM. Insert yourself. Be disruptive. Cause some change. The project needs that more than ever.

Hollywood can do it why can’t you….

“Hollywood can do it why can’t you?” was the sentence that about put me over the edge back in 2000. I had inherited a “position” where I was going to be the “ortho guy”. Essentially the guy that took 9×9 scanned aerial prints and made orthoimagery out of them. Correct scaling issues and incorporate an elevation model. We had a scanner, we had software, and I had a slight idea of how it worked. Not a great idea so that’s why you had support, manuals, and a list of phone numbers of people I could call and go “I don’t know why but the RC30 Camera report hates  Fiducial 4″. I had one job where I was just trying to piece together some imagery and it wouldn’t pull together. I had a bad survey point and one of the bosses said “”Hollywood can do it why can’t you? Just put it all together and “fly” over it. Like Hollywood…”

Anyway – yes – People used to strap cameras with film in them to planes to get aerial photography.

I saw two things that made me remember that stupid story over the last month or so. This article kicked off the thought train.:

Planet Labs: They are tossing out a “flock” of satellites from the International Space Station to image the earth. What they lack in pixel resolution they make up for in timeliness. Almost constant earth monitoring from these small satellites. Brand new imagery. Constantly. I can only imagine the headache of storage.

Dove_nest_2

 

 

 

 

Sky Box Imaging: Video from a satellite. That’s all I can say on that one. Video. Imagine overlaying your data on video.

So yeah – Hollywood.

Spatialite and ArcGIS and QGIS

I know this is old news as I just upgraded to ArcGIS 10.2. Whaaaaa?? BUT ALL YOU EVER TALK ABOUT IS QGIS?!?!!?? It’s true – I do use ArcGIS – It helps me more annoying. I know – I didn’t think it was possibly either.

Over the last year we (NRGS) have been taking a different approach to life and business. Granted I’ve been pushing in that direction but this year it started to actually gain some traction. You really need a mix of tools in your GIS Bag of tricks – most of the mix is Free and Open Source (and you might find yourself so compelled you might just work with the other stuff one day). After all your data is the most important piece to your GIS puzzle right?

So Spatialite – that little used (or widely used) database format (technically sqlite with spatial added in) that keeps threatening to do something at some point…. Well – It did something. ArcGIS 10.2 (upgrade ran smooth by the way – probably the smoothest ever) came with this little hidden gem that gives you support for spatialite. You can’t edit – but you can read and import data into it – which is a good thing (“WELL HELL IF I CAN’T EDIT WHAT GOOD IS IT????!!1!” – Just hold on).

I’ve got a project I’ve been working on and I dumped it from a coverage File Based Geodatabase back to shapefiles. I committed myself to doing all of my editing and data prep in QGIS. So far ArcGIS has come into play twice – topology and printing. Had I not been frazzled I would have finished up topology in QGIS. I’ve not been happy about the move to shapefiles because stupid things can happen upon occasion. So in my post Christmas humbug I started playing around.

I did this in the ArcGIS 10.2 python window:

#declare the variable
sqlite_database = 'C:/data/rjhale_arcgistest.sqlite'
#create the spatialite database
arcpy.gp.CreateSQLiteDatabase(sqlite_database, “SPATIALITE”)

I did try this with “ST_GEOMETRY” and couldn’t get it to work. I most likely was doing something wrong – but – for the purposes of this exercise “SPATIALITE” worked.

I then imported a feature class. I jumped over to QGIS and began editing away. So this jump did three things of importance for you the casual reader of this blog.

  1. I just created an open format and transferred data out of my geodatabase into spatialite.
  2. I just avoided the whole shapefile issue and getting truncated field names (i.e. 10 characters for a field name)

For fun I did the exact opposite and was able to view the data (QGIS to ArcGIS). For ArcGIS editing purposes I exported it out back into a File Based Geodatabase (or SDE).

spatialite

Just to really mix it up I downloaded the spatialite gui and created a database and imported a shapefile and passed it around. For the record I passed data between two different software programs and two operating systems. Given an upcoming project we will be passing data between macs, linux, and windows (7&8) and this might be the best way to go about it.

…AND I know I’m oversimplifying this a bit. I have no doubt there’s going to be some “hiccup” along the way of moving data and there’s probably something I’ve missed. QGIS still flails a bit with spatialite (IMO) and I’m pretty sure creating spatialite databases in arcgis has some problem somewhere. On Ubuntu (xubuntu) 12.04 LTS, creating a spatialite database from QGIS/Spatialite GUI is a 10 minute exercise in pain because of libraries or something. From the command line it takes 2 seconds to make one. So it’s not ever going to be super easy – but if it was everyone would be in GIS.

The challenge – Think differently. If you’re a small organization and you are a bit strapped for resources you now have (assuming you have ArcGIS 10.2 – if not there are other things that can be done) that would allow you to expand your data into an open format beyond shapefiles – example:

  • Your water utility needs to edit some data. Dump what you have into a spatiaite database and load QGIS onto their machines and let them edit. It keeps them out of your database and gives them the power to fix things. They bring the data back to you.
  • You have some field crews going out for something and you can’t afford another copy of ArcGIS – dump your data into spatialite and send them out with QGIS and let them bring the edits back to you.
  • Are you getting an intern to help you convert some old maps into digital and all they have is a mac…well – Spatialite and QGIS. At worst shapefiles but if you have existing data in SDE or some database…….

The bad – you have to write some documentation and maybe have a brief tutorial on what buttons to push. If you need to do more complex editing you always have “home base” covered with ArcGIS (I could argue you can do complicated editing in QGIS also). Let your inner control freak out for a bit and write some documentation.

Open Source software always have a problem with being mystical and magical. Documentation isn’t sexy as we’ve found with so many things this year – it’s riddled with acronyms and things you aren’t used to seeing. We’ve (NRGS) are trying to demystify that a bit. Open Source GIS can be your ally when you need it. ArcGIS is always there if you have it and it’s comfortable because it hasn’t changed greatly in 10 years.

…and Of course – shapefiles always work, but this opens your world up just a bit more if your gis is a monoculture.

Think Differently.