ArcGIS Desktop

Moving a Project from ArcGIS to QGIS: Map Composer

rjhale · Dec 8, 2014 ·

In my washout geology career…..computers were paperweights. We had to make maps – so we would make them. HAND DRAWN MAPS…or we drafted them on 1:24000 maps so we didn’t have to worry about scale and north arrows.

So I’m up to Part 2 of these un-numbered blog posts on how this is going to work. I’m moving a project from ArcGIS to QGIS. Project was in Data Driven Pages in ArcGIS and I’m trying to figure out if I can move this to Atlas in QGIS. Luckily I don’t have to re-develop the map layout – which was done by someone much more talented than myself. I must fill you in that it took multiple iterations of going: THIS? No….(changes) THIS! No…..(changes) THIS? No…..(dammit changes) &*(%$ THIS?!?! OK GOOD I HATE EVERYONE.

It’s never going to be a one to one crossover. ArcGIS has great map making tools. It makes a nice cartographic map. The question is can I transfer it over to QGIS. Why? Because my data resides in PostGIS/Postgresql

The answer is 90% yes. So I’m shooting for this:

It’s a nice legend. Everything is explained. We also pulled some dynamic text into it. So WINDING OAKS and TOM WADE were automatically generated. The tract ID in the lower right was autogenerated. The acres in red weren’t. As this evolved we went from one layer to about three and we were pulling information from all three to put in the map. We ended up running a frequency of tree cover and having to hand code the result. Which now that I think about it we could have been smarter – but our fluff time was eaten up with changes to the map. If I can offer some advice – beat your client over the head during the design phase. Spend lots of time on the design – it will save you time during production.

What I now have:

So – things that were completely different in this transition.

Both QGIS and ArcGIS offer rulers. I will argue the rulers are a bit more stable in ArcGIS than QGIS – but the functionality is the same. At times the rulers in QGIS would move things I didn’t want moved. It happens and wasn’t a heartbreak by any stretch. So lay out your rulers and line up your map annotation/legends/things.

If you notice above in the QGIS legend (one image up – the big image) I’ve got this really nice line that extends out from Stands/U.S. Acres. It creates a separation from the important part of the map – the stand information. QGIS can draw rectangles, triangles, and ellipses – no line. It can draw an arrow. So I decided to draw arrows and remove the arrow head. I could have use rectangles with a white and black border and gotten the same look…but I went the line route because I was bumping things around to match my existing PDF.
The legend. Here was a problem. In arcgis you have this wonderful thing you can do with a legend. You can ungroup it. Once it’s ungrouped you can move it around and do all sorts of things to it – except put it back together and have it act as a legend. I’ve 9 things I’m showing. In Arcgis we add a legend and ungroup it and spread those things out. In QGIS – you can’t ungroup your legend. After a bit of thinking…..I ended up adding the legend three times and removing the bits I didn’t want to see. So instead of a legend with 9 items – I had a legend with 4 items or 3 items or 2 items. IT’S NOT WRONG….I’m so used to ArcGIS (12 something years with the same interface) that I want to go “Why…” – but it’s not wrong. It’s just different. One nice thing about QGIS is you never break the link back to the map. It’s always a legend. So If I add another layer – I just have to pick which of the three legends the new item goes in.

North Arrow – so. I’m running linux. I have no doubt I could thieve the north arrow out of my ArcGIS install. QGIS is that flexible and you can grab a font, an SVG, or about anything and make a North Arrow. I’m not going to read the ArcGIS terms of license something agreement to figure out if I could use the north arrow. So I used the one pictured.
Scale – If I had one complaint…it’s I have to stick with a ratio and not a 1 inch = xxx feet. I like ratios..client not so much. BUT – I think I have a fix for this shortly after I typed the word complaint. Stay tuned.

So the question is now – how do I fix the parts of this map that need to be dynamic? I have to pull acres in because this is a land cover map. I need to pull tract names, property names, and more from different layers. How Do I do that automatically and not hand type it all in? I think it’s possible and I have a plan. Will the plan work? Don’t know. You will know after I do….

Moving a Project From ArcGIS to QGIS: Symbology

rjhale · Dec 6, 2014 ·

As I sit here starting to lay out this transition I decided I’d better start scribbling a few things down. This isn’t done to make much of a statement. This is mostly trying to fix a problem. As I’ve been said in a recent presentation I made – “GIS is flexible if you let it”. Will this work for everyone – I doubt it. For this client It will.

I’ve a forestry client. Things started getting unholy complicated this year. I wrote a post on moving their data from shapefiles to postgis. I’ve still been using ArcGIS for maps and various things but the data resides in PostGIS. I dump data to shapefiles so they can look at it. Except it’s not a good fix. I don’t wish or want to dump data to shapefiles. Things get changed and I’m left trying to move the changes back into postgis. So I really want to get them editing the data as it resides on their system.

I started this morning laying out a plan. It’s gotten complicated in the last 20 minutes. You can’t eat an elephant whole…and I can’t fix this all at once. So we’re going to piecemeal it all together.

Lets kick this off where the problems started. Maps. This time last year NRGS got a fairly complicated request for maps. The forestry client manages about 30,000 acres of land. There are multiple disconnected polygons scattered over multiple counties and two states (but only one projection). The mapbook went from something that was two button clicks in Data Driven Pages to a 10 hour slog of exporting the changes for a few weeks. It wasn’t automated. It couldn’t be with all the individual changes we had to make per map. When you work with a client enough you get a feeling sometimes. I’ve got a feeling these maps are about to make another appearance. I want this to work easier with Postgis.

The first thing I’m doing is migrating my symbology. There is some not overly complicated symbology – but it’s nice symbology for these datasets. It’s not harsh…a bit green. Green is good though. The worst one is the stands layer. I opened ArcGIS and got started.

I was able to do this in QGIS 2.6 in about 15 minutes.

Color wise it’s a duplicate (except for one attribute – I’m getting to that one). Colors palettes aren’t 1:1. ESRI has there way of doing it and QGIS has it’s way. I typed the first few RGB values in by hand. I then found in QGIS 2.6:

QGIS has a color picker. So I just worked my way down the line and changed the fill and the border colors to be identical. That only left me with one “diagonal issue”. One of my symbols in arcgis was a diagonal fill. There is enough flexibility in QGIS 2.6 you can recreate that also. You add a line pattern fill and add a simple fill and thanks to the color picker:

Granted it’s not quite perfect – BUT – it’s close. Very Very close. Line weights don’t quite equal. A lines weight of 5 in ArcGIS appears to be about a 2 or 3 in QGIS. The way the map renders looks a bit different but I’m also jumping across platforms and Virtual Machines and things.

So symbology has transitioned. Next question is Atlas vs Data driven Pages. Can I take something that we had to individually change in Data Driven Pages and automate it in Atlas? I dunno. Half of me says yes…other half says no. The important thing to remember is for this client – this is all being done with 100% free and open source software. Could I have done this 5 years ago. Nope. Partially maturity of software and partially I was hung up on one solution. Now…It’s happening.

I’m saving everything as a qml and sld in qgis. I’m hedging my bets and saving all my ArcGIS symbology as a lyr.

Stay tuned as we tackle more issues such as connecting three other workstations to this database, “stop turning off your computer”, and my favorite – Can we get some static IP addresses on the network please.

Missing data and a KMZ file

rjhale · Jun 16, 2014 ·

Data is quite fascinating. Mainly because there is so much of it….and it never seems to do what you want.

The email came from Atanas Entchev. He had recorded some bike routes and exported out a kmz file from a phone app. The app recorded tracks and points. By the time he had exported the data out all that appeared were two points – the beginning and the end of his route. No tracks. Except when loaded into Google maps there were tracks.

So in all of my grouchiness and being old – I like data. I remember when I first started out in GIS all I was familiar with were coverages. Then shapefiles appeared. I remember at one point getting a CSV file of coordinates….then as my career kept diverting there were cad files and dwg, dgn, and a handful of image formats. I remember when KML files appeared I couldn’t stand them because all you could do with ArcGIS was to convert them to shapefile or some other format. They were very limited. It may have been all you could is convert data to KML. I honestly can’t remember. I do remember opening up my first one in a text editor and discovering XML. I still don’t entirely understand XML – except for the fact it’s just another file that contains data. If my software won’t convert it I always have a text editor.

One thing we always have for better or worse are standards. KML is an OGC standard….well…mostly. In my searching I found other people complaining about getting KMZ (please note I didn’t say KML) and not finding all the data that was supposed to be there. A KMZ is a zipped up KML. So if you want – change the name of your KMZ to zip – or if you have a bike.kmz file change it to bike.zip. Then unzip it. There is your KML.

Where was I – oh yeah standards. So KML is an OGC standard. Except the OGC also allowed for it to be extendable. In that extendability I would argue the standard has been broken. In Atana’s KML there was a tag called gx:Track that nothing recognized except for Google.

Example:

</ExtendedData>
</gx:Track>
<gx:Track>
<when>2014-05-26T17:30:27.580Z</when>
<gx:coord>-74.451391 40.508723 -46.29999923706055</gx:coord>
<when>2014-05-26T17:31:19.236Z</when>

For those who have dealt with GPX files the above should look familiar. KML now has a gps component (which makes complete sense considering I mentioned this data was collected with a phone app). The gx:Track tag describes a line (hence track).

What to do? In my case I pulled out the section defined by gx:Track and converted it into a CSV file. From there I built a polyline using QGIS. Granted this wasn’t a button click solution but I did pull the data out and make it useful. For the record I ended up using QGIS, ArcGIS, and GDAL in my investigations. I kept hoping that the actual problem was the software I was using couldn’t read it and something eventually would. Well – VIM to the rescue.

So read Atana’s post and he talks about one more way to get the data converted from TPStigers.

So what did I learn?

Ascii makes the world go round
Nothing is standard
I need to ride my bike more.

Starting a GIS

rjhale · May 13, 2014 ·

I had a call the other day from a potential client. I actually hate that term – “potential client”. That’s an apt description for everyone these days. I went in and spoke with them and they were suffering from a GIS problem I’m seeing everywhere. They had a need and spoke to a sales person on fixing it. The remedy was actually a pile of software and hardware and I’m pretty sure the salesperson walked off with a nice bonus. I’m not sure of the final cost to potential client. They weren’t happy. I’m guessing they were invested to the tune of 50k or better. Software and some hardware and that didn’t count training and the consultant. In fact they are now so strapped budget wise they will remain a potential client.

Because of the client space I inhabit I struggle a bit daily with the idea of a GIS. A lot of times people mistaken me for the open source zealot. I’ve got a copy of ArcINFO (it’s that name till I die) and I regularly use all the other FOSS4G tools like gdal, qgis, postgis, and geoserver. I use fulcrum. I use a lot of software. What I’ve been telling people that it’s not the name brand of the software you are using. It’s more process than the software. There are a lot of great tools out there that are free or low cost.They aren’t as pretty as the commercial variety – but they work just as well. As I tell people – a clip is a clip and a buffer is a buffer. All GIS software has the same goal – some of the software speaks with an accent.

If you are debating starting a GIS – Find someone who isn’t a salesperson and sit down and ask them what they would do. If they only mention designer brand E please remember it’s 2014. Educate yourself. You might need to talk to several people. Ask a lot of questions. Offend. Ask why every chance you get. Ask who your consultant is affiliated with – are they a “business partner” and what does that get you.

In many of my talks lately I’ve been asking people how their budgets are holding up. That’s an important topic for me since I’m a consultant. Many times I’m hearing pretty much the same thing:

Don’t sell me more software
I bought some software and it’s not helping
Maintenance is killing us

If you’re starting a GIS (and yes there are several of you out there) – explore your options. If you run to ESRI or any large commercial firm expect a sales job. I can almost quote verbatim what you are going to here if you are a manager speaking to sales people. Just remember sales – this is going to make buying a used car seem pleasurable depending on who you are talking too. Please remember at the end of the day the only thing that matters is your data.

If you start exploring the Open Source GIS space you are going to get a lot of kick back. To quote two great examples of what you are going to hear:

People who use free software should work for free.
You will never get support from “those people”. They think everything is free and they are here like we are…so if you use the free stuff it’s going to break and you are going to look like an idiot.

I’ve actually heard the above – it’s quite wonderful and extremely humorous considering the source of both of those quotes.

If you are starting a GIS please consider the following:

Decide what data you need. Does it exist already? Do you need to go out and collect it? Can you collect it or do you need to hire someone to do it.
Look at your software needs. How much do you want to spend. Look at open source. Look at the commercial stuff. Can you mix and stretch those dollars just a bit more than you think? BTW – you can mix and match. All you need is a process.
Budget – Budget for a year. Budget for two. Then Budget for five. Budget like this might be successful. Look at your software also. Are you going to spend enough on software you could hire someone else? It happens. That’s a huge argument (for me) for mixing in open source. Jobs. Hiring someone does more for your community than paying more money for software.

It’s 2014. Software doesn’t have to be from one vendor. Your entire GIS setup doesn’t have to be a mono-culture. Servers (think cloud (another word I hate)) are cheaper than they ever have been. Pick the right software and you don’t have to spend a fortune on a workstation. Worry about your data. If you spend more time worry about your software than letting your data work for you you are doing it wrong.

ArcMap won’t kill off ArcInfo Workstation

rjhale · Apr 19, 2014 ·

If I remember correctly I was in Knoxville and we had gotten the first look at ArcMap 8.0. It’s been too many years ago to care or to count – I just remember sitting there going “Dammit – I spent all this time learning AML and Avenue”. The word being passed around as I remember it was ArcMap was the beginning stages of killing off ArcView 3.x. ArcView had made tremendous in roads and had pretty much became the go to standard for all my clients when I worked for TVA. Workstation was safe. I loved workstation. For all it’s command line craziness I knew one thing – if someone said they were using workstation they had a clue as to what they were doing. One of the managers with strong ties back to Redlands said during the class “This will kill off workstation”.

As time went on – Workstation on Unix slowly moved to Workstation for NT. ArcMap turned up in the office slowly around 2001 when XP finally appeared. Eventually I opened workstation less and less as coverages were replaced with shapefiles and shapefiles were pushed into a geodatabase. Eventually we moved into SDE which killed off a beautiful workflow we had set up with ArcINFO workstation. We started hearing about pushing maps into 3-D and onto the web. I still hear about pushing maps into 3-D and the web.

For the last year I’ve watched ArcGISOnline creep into everything. It’s not a bad thing – but I don’t really consider it a great thing. Call me your crazy uncle who turns up to thanksgiving late in a rusted out jeep, but I’m still not convinced the data is secure. I will say the same for any “cloud” environment in which you don’t hold the kill switch.

So anyway – I spent the morning reading up on ArcGISPro. Watching videos. I started having flashbacks. ArcWorkstation won’t be replaced. Arcview is safe. Deprecation. Will Arcmap run with workstation….Will ArcGISPro run with Arcmap….

I won’t even get into 3D.

For the record – it may take a few years but I consider this the preliminary announcement for the end ArcGIS for Desktop. The desktop platform has gotten stale. This is it. The next “revolution” will be ArcGISPro. Still with the tiered licensing from what I can figure out. “Hi you’ve just purchased ArcGIS Profesional Basic for Desktop”.

Innovation is great. Getting ESRi into 64bit for the desktop has long been overdue mainly because I still consider Desktop to have too much 80/90’s workstation core at it’s heart. You have to rewrite it to get rid of it. If you are going to rewrite it you might as well start killing off the old product line. Software changes and it’s not a bad thing….but – there’s a lot of “transformational” talk during this video. Which usually means transforming your budget also.

It’s worth the watch – if only to remember the palm springs demo that was rolled out for years. I do miss the palm springs demo.

ArcGIS Desktop

Contact