Open Source GIS

Mobile Data Collection: Input

rjhale · Feb 25, 2020 · Leave a Comment

If you flash back to 2008 you would have seen me sitting on my couch unwrapping a Trimble SB (?) running Arcpad. At the time I thought “collecting data” would be a beneficial add-on to me being a consultant. Spoiler alert – It wasn’t.

If you flash back to 2013 I was sitting in the Caribbean needed to collect data and I was holding a new giant yellow GPS unit that wouldn’t quit rebooting when it got hot. Spoiler Alert – It’s always hot (generally) in the Caribbean. What I did find is Fulcrum – and that allowed me and about 5 or so folk to disperse and work all over the place and collect data – WITH OUR PHONE. Which for 2013 was pretty awesome for me.Since then I’ve always been interested in how people are collecting data. I will make fun of you for doing it “Pencil and Paper” but I will admit to having a field journal and pencils in my laptop bag just in case.

A client asked a few weeks back “We need to take a copy of the data into the field” and that trip into the field was going to be with no to limited connectivity, no data collection, but they needed to know where they were. They are a IOS/Mac heavy group. Print a map seemed a bit harsh and working through some other hoops seemed unnecessary.

So my suggestion was Input. So what is an Input? Lutra Consulting developed a mobile app built off the QGIS Source Code. It runs on Android and IOS. There are tutorials out there on using Input. I can’t do better than the ones that are out so I won’t even try. I can throw in some screen shots and just general discussion on why Input is pretty cool from an avid QGIS user. When you get it installed on your mobile device of choice you get this:

Simple. Home being the map screen, my projects being things you can work on, shared being projects that are shared with you from others, and finally explore just in case you can’t find your project.

So we will tackle this one of two ways. One way with Mergin and one without. What’s Mergin? Mergin is the commercial end of Input. Commercial? What? Yes – there is a commercial end of Input.

1. Install Mergin and Get an account. Don’t worry – it’s free for small projects. Download Mergin from the Plugin Manager in QGIS. Mergin will appear in your browser panel.

2. Build a project in QGIS. Import your data into Geopackage. Build the project the way you’d want it to be displayed on your phone. When I say “build your project” you’re going to build widgets for data input, set up labels, and set up your colors. I prefer boring as seen by this screenshot.

3. Import your project into Mergin (You’ll have to connect the plugin to your Mergin Account).

Give it a few minutes and you can now download your project into Input and start working. Edit your points, lines, and polygons….and attributes….

4. Sync your project back to Mergin. Download it back to your computer.

One word of caution :

You’re going to build a project .
You’re going to COPY it to the server using Mergin.
Move it to your mobile device.
Edit it locally and sync it back to the server
Eventually you’re going to download it back to your compute using Mergin
So at the end you’ll have one old project and one new one sitting on your machine. Just pay attention. I did get confused at first but I’m slow.

You don’t want to use Mergin?

Then copy your data to your mobile device.
Create a directory under the projects folder under Input and move your data there.
Start working. At some point you’ll have to manually move things back to your laptop.

….and there you go.

So where does that leave me with all of my current love of data and databases? So as an experiment I connected to a test server and edited data against PostGIS (live). I wouldn’t suggest that for day to day because “you always need a network connection of some sort”. So I’d always build a local project to share to your mobile device.

I need more than “me” collecting data – then read up. Several people can edit and Lutra has used Geodiff to handle multiple users. All your edits can go back into the same geopackage that you download back to your Workstation.

So should you pay for Mergin? If I stuff a street centerline file, building footprints, streams, and a few other things for a medium sized city I’m getting close to 100mb of disk space. So I’d say pay. If you just can’t pay – then keep it under 100Mb or manually move your data over.

Overall I like it. For years I’ve said “QGIS on Mobile” and now we have Qfield and Input.

The First Day of XYMAS…..The Command Line

rjhale · Dec 13, 2019 ·

I used to go to a lot of schools to talk about maps – be it High School or College several times a year. Some have learned “Hey we don’t want Randy to talk to the impressionable kids”. I still do it if requested but I don’t make it well known that I will just show up and talk about maps and data. This year for GIS Day I went to the University of Tennessee at Knoxville to hassle the kids about all things open source. There were a large percentage that went “I’m learning python so I can do GIS – do you know python?” and I would go “I know some python but you don’t have to know python to do this”. Which started me thinking about young Randy back 27 <gasp> years ago and what skills I had learned then (1992) that were still relevant now.

When I started my geo career we were running Unix workstations. I had a system administrator that worked with my department that started teaching me small tricks to make my life (and his life) a little easier. It all started out with a terminal. From there I started learning how to run commands or build small scripts to rename files and move data from point A to point B on the network. Eventually the Unix machines gave way to Windows Machines and the problems remained: small repeatable problems I can fix from a command line. Today, the Windows machines have given way to MAC/Linux machines which is even better (but Windows 10 isn’t all that bad (yeah I said it)).

What does this have to do with GIS? Not much. What does this have to do with a life long skill set? Everything. Geo software, operating systems, computing power have changed immensely but the small things I learned on the command line grew a better understanding of programming, networking, and “system administration” in general.

I need to move some files on my machine – the files are spread over multiple directories. From my linux box I do: find . -type f -print0 | xargs -0 mv -t /home/rjhale/biketrails . I think that command moved about 30 or 40 shapefiles into one common area for me to eventually merge into one file.

What has been the longest running single program I’ve used over 27 years that’s still a thing? It’s VI (currently VIM). It’s all command line.

VI is a text editor. It’s simple and it’s standard with a Unix operating system and was released in 1978. VIM is a clone of VI and was released in 1991. It’s simple and powerful and you don’t have to leave the keyboard to use it. There is a version for every operating system so you aren’t left out. What do I use it for?

Make notes while working on a project.
Write scripts.
Automate changes to text files.

I’ve been using that since 1992.

Like I mentioned above – the scripting led to an exploration of networking. I’m not a sysadmin by any stretch but if something goes wrong on a server I can at least craft a half competent explanation for someone who knows what their doing as a start. The simpler the better as in “I can’t ping this box” OR “Can you give me access to the network through <fill in the blank>”.

So where does that leave us for the firs it day of XYMas? Learn some command line tricks and tools. It doesn’t have to be something complicated and it doesn’t have to be something grand. It has served me well over 27 years. Think Small and Simple. Do something fun for your first outing.

Take a look at some tutorials out in the internet (as always these may or may not be active at some point in the future)

FOSS4G-NA 2019

rjhale · Oct 10, 2018 ·

FOSS4G-NA popped up on my radar this week.

So from just the website and a few twitter noise makers it looks like It’s headed to Mission Beach California (San Diego) April 16-18 2019. I’ve made the last 3 – so It looks like I’m going to do my best to make this one. Road Trip? I encourage you to attend and support the FOSS4G North America Community.

20 minutes at the East TN TNGIC Meeting

rjhale · Sep 6, 2018 ·

I went through a year or so of not talking at conferences. For local ones I’d show up and do something. I taught a workshop at FOSS4GNA 2018. I did a workshop at FOSS4G 2017 in Boston (and it sorta sucked). I need to get out of that mode and talk more – but I’m happier being quiet these days and working with clients.

Anyway – at FOSS4GNA 2018 I discovered something – the OSGEO Suite had disappeared from Boundless’s Website and was replaced with a github repository. It wasn’t long after Don Meltz put up an excellent explanation of the OSGEO Suite. The OSGEO Suite being gone isn’t a problem. It does increase the amount of talking I have to do spend explaining it’s not a problem. I could even compile the suite – but my clients can’t. They are smart people – but the time to compile the boundless suite won’t be happening.

One of the things I keep running into is the idea of an Open Source Server. The OSGEO Suite was well known and I get people asking “Hey – can you set us up a geo server”. They mean the opengeo suite. I then launch into an explanation of what makes up a “geo server” and they go a bit blank and answer me “Yeah – one of those”.

So for the next bit I’m gonna be doing something like this:

I want to explain how this works to people. We’ve gotten a bit deaf by hearing “ArcServer” at every conference in TN as this one monolithic thing you have to install to make a web map…which really don’t need since AGOL appeared but…you want SDE and it’s not called that and you have two people editing but you need SQl Server but you don’t….

Anyway, QGIS/PostgreSQL/PostGIS I’ve installed multiple times and as of late I’m experimenting with geoserver and what it can/can’t do. I have no grand delusions of creating ‘Randy’s Open Geo Spatial Super Duper Server’. I want to get to the point where maybe mapserver is an option in this. So much to learn…so little time.

I suck at github – but I’ve been more active as I try to develop a skill set and get over the angst of pushing and pulling things. I’ve built a vagrant box to install PostGIS/PostgreSQL/Geoserver so I can have something to point at plus it gives me a starting point for an upcoming class I’m in the middle of building. The vagrant box isn’t pretty – I expect to change it over the next bit as it’s mostly pieced together from a lot of other people’s work (Coleman McCormick, Dave Smith, etc). It works though and I have everything but pgadmin 4 running in one spot. Which – I don’t think this is anywhere near “install this for a server” because this is mostly just for talking, training, and testing.

So why vagrant? Why not docker? I can gloss over vagrant faster at theses talks than I can docker. I have 16 minutes to draw diagrams and about 4 minutes to publish one piece of data: QGIS -> PostGIS -> Geoserver -> Leaflet.

Anyway – if you’re up for the possibility of a train wreck of a presentation come by the East Tennessee Meeting in Kingsport next week as I try to explain how this works and hopefully you get excited enough to install some or all of these components and do something cool with your data.

Adding Triggers to Geopackage

rjhale · Aug 30, 2018 ·

In between work I get a chance to play around with software. I’ve been digging around postgis more and more these days trying to get a better understanding of what it will do or won’t do. It’s a never ending learning experience for me. Sometimes it pays off – the never ending tutorial left me suggesting we dump a database to a geopackage.

I’m great at using geopackage in sentences like “USE GEOPACKAGE” and “Hey – what do you think about geopackage?”. Since I’m actually officially using it I decided to do some digging. It’s a SQLite container. Since it’s a SQLite container I should be able to do some things with it – like…..

I can use the spatialite gui and throw SQL against it and change the tables.

While talking to another person the question started with ‘Well – why can’t we just do the data collection in geopackage?’. Which we did some back and forth and in some cases this would be acceptable vs doing a full postgis install.

So my first question is: Can I make a trigger in a geopackage? The answer is yes. For those of you who don’t know what a trigger does – it responds to some action in a database. In postgresql, I put multiple triggers in a database for a client. Some triggers calculate lengths and some areas and some make a note of who edited a feature.

After some digging I found the syntax and wrote a trigger for SQLite/Geopackage in the form of:

CREATE TRIGGER update_esn AFTER INSERT
ON addresses
BEGIN
update addresspoints set “esn” = (select esn from polygon where st_within(new.geom, geom));
END;

I used the spatialte gui to add the trigger. So if I pass this off to a client they can edit an address and have it automatically pull in the emergency service number. Magic? Eh – close enough for me.

While it’s not going to work for a multi-user situation (or will it – I’ve not investigated) – it works. So now I’m adding widgets to QGIS to aid in data collection.

Open Source GIS

Contact