North River Geographic Systems Inc

Geospatial Problem Solving

You are here: Home / Blog

Blog

Add Geoprocessing Tools to the QGIS Toolbar

by Leave a Comment

I am going to attempt a post a week on something and I’m already behind.

All of my work is falling into one of two categories at the moment: PostGIS and QGIS. At the moment I’m loading data into PostGIS using QGIS. So I’m:

  • I’m looking at the data going “What the fork is this”
  • When I figure it out it’s then “I’m going to push this in and toss some permissions on it”.

It’s not like pushing data into PostGIS is a complicated process these days: OGR2OGR, QGIS, and I’m pretty sure GRASS can import – so it’s whatever makes my life easier which is QGIS at the moment. So generally I like the “Export to PostgreSQL” tool in the Processing Toolbox. Once you use it it floats to the top under the Recently Used Geoprocessing tools.

At that point in my existence it’s literally two clicks away. One could really “Type it out” using the locate toolbar so no clicking is involved – BUT – what if I wanted that geoprocessing tool on my toolbar.

While updating one of the two QGIS Classes – I stumbled into this “one really cool trick”. It all is hidden under the options menu. Go to Settings -> Options and navigate to the processing tab.

Scary Right? It’s not that bad. Here is the hardest part of this whole thing – Click on Menus and you’ll see a host of options. Click on QGIS (native c++) and look for Buffer. Click on Buffer and click on the checkbox to “Add Button in Toolbar”. You may need to dig around to find other tools. QGIS keeps them divided into 3 categories.

Close QGIS

Re-Open It.

Hey you have a brand new toolbar called Processing Algorithms and you’ve got a brand new button with a pic on it that looks like an Boomerang (or upside down ‘L’). Press that and you have the buffer geoprocessing tool ready for use.

So – entirely necessary? In my case not really – it is nice to have access to it in one click. It is nice from the standpoint that I can take an inexperienced user and customize the interface to where they don’t need to know where this tool is or how to get to it – just push a button. You can keep adding tools that you want up there – in this case I really only wanted two. When I’m done I’ll “turn them off” but for now I like my extra toolbar.

The Year in Review – 2021 Edition

by

Just to prepare for the 2021 edition I went back and re-read the 2020 edition. Yikes. So I started writing and I’ve changed this post up several times. I’ve decided much like I did last year “this is the final shot with a few edits”.

2020 was messy for everyone especially on my end of things. So in January of this year I started making some changes. It’s becoming overhaul of North River Geographic Systems, Inc and Randy, Inc. Through the last few years I have been making small changes business wise like “No I don’t want this project” and “PAY ME” but nothing like “What do I want to accomplish?”. I had found myself saying negative things about the business. I haven’t said anything really great about NRGS for a bit actually. Business can be frustrating especially for me. The bad parts were way outweighing the good parts and there are a lot of good parts that I never acknowledge.

Summer trip to the Conasauga River

So I started with “A Plan to get me back on my feet”. I don’t think I’ve had a coherent plan in the last few years so this was step 1. New attitude. A better direction. The “Shiny Bigger Plan” is being written now.

If I were to sum up this year it was “Challenging but Better than I Hoped”. Which is weird to say. If this hadn’t been good I think I’d most likely be doing something else at this point. Yeah – my January 2021 decision was an A. Fix This or B. Do Something Else. I was that frustrated with everything.

Piloting for Swim the Suck in September

I ended up picking up more clients in 2021. The clients I picked up centered around two pieces of software: QGIS and PostGIS. I did have multiple calls about Geoserver WHICH is still something I’m getting better at and I still don’t feel like an expert by any stretch. The best paying jobs were PostGIS centric. The fun weird ones were all QGIS. I solidified my docker game somewhat. I started diving back into Python.

Strangest call: One day I received a call out of nowhere from someone who noticed I did QGIS support. Their job was laying irrigation pipe in farm land in Northern Ohio and they were stringing together a process to patch up some old Trimble Software that ran in a tractor/ditch digging thing. One zoom call later and I figured out something of a process with QGIS (not my greatest work by any stretch). It’s enough to get him working for the twice a year he needs data to run the tractor to do irrigation. He occasionally calls and I enjoy hearing about his work. So far I’ve seen his pics of the Ditch Digging Tractor Thing, farm as he shoved the laptop out his kitchen window, and we’ve traded jokes over the phone.

Somehow I ended up with Kittens.

Training: Training went virtual with two classes with a third class in the works. I think 2021 was affirmation I can do virtual training. So much so I’m unconvinced I will ever train “for me” in a classroom setting again. In the next few months I’m going to have three classes: “Intro”, “Intermediate”, and “Final” (which could be advanced or something – I’m still drafting it out). When I get the third class completed I think I’m done with classes. I’ll be happy to do those three.

Life in General: Good and Challenging. I had some personal goals I didn’t meet. I had some I did. Parents got older. I got older. I didn’t get COVID. So far I’ve had no one close to me die from it. I count myself exceptionally lucky. One friend passed from a brain tumor which was a bit surreal. Get through almost 2 years of a pandemic and then die from something that had nothing to do with COVID. It was a very hard reminder that life is short.

Phillip made the next big adventure this year. F*ck Cancer.

I decided part of the way through this year to start learning about software I don’t normally touch just for kicks plus I needed the mental stimulation. I miss learning new software just for fun. So I did things randomly when time permitted- What did I do?

  • Mapserver – I feel like I owe it to myself to learn about a piece of software that is one of the foundations of Free and Open Source Software. I installed it. Created a mapfile. I was able to look at the data I created.
  • Leaflet – Believe it or not – I hate “web maps” mainly because I never have time to learn anything about them. So I sat down and made a map in leaflet. Easy? Mostly. I’ll probably make some more but nothing too fancy. There are people better at this than I ever will be. All 100% hand crafted in VIM with some “Copy and Paste” for taste.
  • OpenStreetMap – I know plenty about OSM but what about downloading the data and doing something like displaying it in Leaflet.
  • PyQGIS – I sat down and made a few small scripts to run in QGIS. Why? Because I love QGIS but the whole programming part is still a bit cloudy for me. So I started doing small things and eventually the smaller things got bigger and MAYBE I can make a plugin. Maybe.
  • AWS – I had a complete implosion with my first dance into Amazon Web Services. I signed up for the free tier and 1 months and $400 dollars later I was attempting to explain HOW I racked up that large of a charge to Amazon Support. Thankfully Amazon refunded my money. I cancelled my account. We all lived happily ever after.

In 2022 I’m also focusing on “personal things I want to do”. That’s going to leave less time for technical but that’s OK. I need self improvement as well as “tech skillz”.

Found a cemetery near my house that I had no clue existed!

I made several conferences as “The weird guy that talks about QGIS”. I made my second FOSS4G International Meeting. WHICH I KNOW THIS WILL BE UNPOPULAR……BUT….. : I miss in person conferences as much as anyone but the amount of knowledge getting out into the world through these virtual conferences is astounding. It’s a low bar of entry for everyone and it’s become something of great value for me. SO – for you who are planning conferences – Consider some virtual offerings even if you’re doing in person.

Anyway – that’s about it. Social Media became less a part of my existence in some ways. It became a better outlet in others. Twitter did get fine tuned to be 99% Geo. Linkedin was Linkedin. I am debating a “virtual” get-together” for this year. I’ve missed 2020 and 2021 with the local QGIS Meetups. I don’t think Covid is going any where for a while still SO – maybe we do something virtual. Don’t know. I’m still figuring that one out.

So with that – Enjoy some noise from the southern part of the United States.

Exploring QGIS – Jan 25th-26th 2022

by

First Class of 2022 – with a full schedule preparing to rollout in the next couple of weeks for the first half of 2022.

What will we do? Explore QGIS (which is the revamped Intro class) will walk you through an introduction to the software and it’s capabilities. With any luck you’re making a map at the end of the class.

Things we will cover:

  • Interface
  • Community
  • Data
  • Symbology
  • Processing
  • Layouts.

Part of the proceeds from the class funnel into the QGIS Software.

  • Dates: Jan 25th-26th 2022 from 12 to 4 EST. Just 4 hours a day for 2 days.
  • Cost: $175 US
  • Link: Eventbrite
  • Location: Zoom – so it’s everywhere and no where.
  • I’ll be sending out data/PDFS the closer we get to the class.
  • Seats: 10

Loading a DEM into PostGIS

by

I haven’t mentioned the TN911 project in a while. Work happens on that when I have time or are motivated – I took a bit of a break to pursue some other tech rabbit holes. I ended up back on it due to some work with pyqgis/addressing questions which led me back to the TN 911 Standard.

My thought on the database is to let it do most of the work. There is some work that occurs in qgis but:

  • date attributes are assigned at the database
  • street concatenation happens there
  • a lot of quality control happens there

In the TN NG911 standard every point needs an elevation. TN also has LIDAR data to provide that elevation. Deriving the elevation with QGIS is easy enough – how hard would it be to store it all in PostGIS?

Not that hard. You can run this if you have postgis installed: 

raster2pgsql -C -I -M -F -Y -l 5 -s 2274 -t 100x100 dem.tif tn911.dem > ng911_dem.sql

I leave it up to you to dig through all the commands in the raster2pgsql command.

Once loaded into the database with psql the DEM appears much like you’d expect in QGIS. I connect to the database and add it.

The nice thing is I can now update all the addresses and set up triggers to handle this with new addresses: 

update tn911.address_points set z_val = round(st_value(rast, a.geom)/3.28081) from tn911.dem,tn911.address_points as a WHERE ST_Intersects(rast, a.geom)=true;

My biggest issue in doing this was actually the projection. The vector data is in EPSG:2274. The DEM data resides in EPSG:6576. I generally try to retain data projections – but I was having to transform (project on the fly) my calculations. Which made everything slower. So I moved the DEM to EPSG:2274.

Anyway – one more thing done for the standard. Next up some pyqgis and some data quality checks.

Finally pulling data out of OSM

by

……as opposed to pushing data into OSM.

So I’ve been working on a OpenStreetMap workshop for URISA. I had originally written it 6 or 7 years ago and it’s been touched in spots but not “re-written”. I am doing a top to bottom shakedown of the workshop and fixing slides and adding new information. It’s been like opening a time capsule of sorts of what has changed and what hasn’t changed in OSM/My Life over the last 7 years.

What does this have to do with anything?

When I had last touched the workshop I’m not 100% sure that Overpass was a thing. I know there was a tool to extract OSM data into QGIS and there was a tool in ArcMap to pull down data. You could always download a planet file or use JOSM to download data.

Over the last few years I became aware of overpass but hadn’t used it. I decided to dig into it just a bit for the benefit of the workshop. Overpass lets you specify tags (or attributes) and run that query against the OSM data. You can then download it. Of course you need to be gentle – running a large area with a lot of data will timeout – you’re better off grabbing a OSM file from Geofabrik and processing that.

As of late I’ve gotten fascinated by fire hydrants in OSM – so I ran a query of just TN for Fire hydrants. For the record there are 837 fire hydrants in and around TN. I posted the Overpass query below: 

[out:xml][timeout:25];
// gather results
(
  // query part for: “amenity=fire_hydrant”
  node["emergency"="fire_hydrant"](35.00208, -90.34742, 36.61557, -81.64104);
);
// print results
out body;
>;
out skel qt;

So what? Well – now that I have this query written I can actually send it to overpass by command line: 

wget -O output.osm --post-file=firehydrant.osm "https://overpass-api.de/api/interpreter"

I can now grab OSM data when I want it (within reason). I could set up a script to run once a week or month and see what has changed.

How does this fit into my favorite piece of software? QGIS has a plugin called QuickOSM .

QuickOSM is leveraging overpass to grab data. I can use place names like Chattanooga to constrain my download. I can specify my overpass file from above and use that. It will even let you parse a OSM file and extract data from that.

Anyway – it was a fun hour of killing time and pulling data out of OSM vs putting data into OSM. Plus if OSM is all you have – the Quick OSM plugin makes it easy to leverage that data to create a map.