OSM Community…pause…NOT!

**Updates are at the bottom of this post**

Let me be clear, I like the concept behind Openstreetmap (OSM), as I understand it – a map made by volunteers that supports communities. I’m sold. I’m ready to support. I want to contribute! I live in an awesome community and want to volunteer to help them through mapping. Others here in Athens seem interested in volunteering and contributing.

But how on earth am I, are we, going to create a mapping community in Athens if the OSM “ambassadors” don’t try to connect with their contributors in supportive and constructive ways? …Welcome to the Fall 2013 US Openstreetmap Editathon and the OSM Community.

OSM promotes getting outdoors and mapping. You can use a GPS device or apps for your smartphone (Android or iPhone), like OSM Tracker, or you can even create Field Papers for drawing in your edits and uploading them later. But what if you have a community member, like a local government agency, that has data that could speed up map development and stimulate other parts of the community to map what interests them most (i.e. community gardens, bike routes, bars and restaurants)? Datasets, like building footprints, that cover larger areas need to be imported into OSM a bit differently. The rules for standard, in the field collection, is easy and pretty straight forward – go forth and collect. The rules for importing data, not so much. In fact, there are disputes about whether imports should be allowed at all. You can read about a few of them here, here, and here. (All found on the OSM Import Wiki page – more on that later.)

Several years ago, while at Gainesville State College (R.I.P.), I offered to help with imports then when OSM was still gaining footing here in the States. I was even helping with tool development that I hoped would make the process easier. You know, to help the OSM community. I couldn’t find anyone on the OSM side willing to enter into a constructive discussion about imports. After many failed attempts to connect with anyone, I threw my hands up and walked away from OSM. The reason then was; why would I want to be in, support, and contribute to a community that didn’t want me, especially if I’m volunteering my free time to do so?

Since then, Randy has been heavily involved with OSM, even having served on their board. I’ve been edging closer to it again and actually feeling that desire to get more involved. It’s a good thing. I see the potential of it and the impact it could have. I felt ready to give OSM another try.

This helped. The BBQ was tasty as well.
This helped. The BBQ was tasty as well.

Friday, I met with Athens-Clarke County Planning GIS. I explained what I wanted to do, got the data, and was giddy. Giddy, I tell you! I couldn’t wait to start adding building footprints to the County. I even invited my new GIS contact to come to our little unofficial Athens OSM Editathon that we had yesterday for a few hours at Pulaski Heights BBQ. FYI, for anyone wondering, local government and their employees are very often part of your community. So, yes, I want to invite local government and anyone else living, working, studying, and enjoying in Athens to contribute since it is their community too. That means their OSM contribution can be their data for importing.

In my excitement, I started processing building footprints. I came up with a systematic approach in ArcMap that involved separating out areas by Census Block Groups with a little model I built. I even joined it with Zoning data to help with some of the general attribution, like “residential” and “commercial”. I also created new fields that would be their keys and values for tagging, including the previously mentioned attribution. It’s base data, that’s all. Part of a framework. I did make one fatal error even with all my planning; tags are case sensitive. The first two grids I uploaded didn’t display properly because of this oversight and Randy bailed me out by reverting to a previous version to remove my bad data. Mistakes happen. Nothing serious. Moving on. I started my uploads again after corrections were made and I was getting in a groove. It’s here that I received a message through OSM from one of their Data Work Group members. This is when I first learned about the OSM Import Wiki and at the same time I was being informed that the work I was doing, this time correctly, was going to be removed. There was no effort to connect or discuss, just to inform me of what they were going to do. I replied letting them know I was being instructed by someone knowledgeable in OSM imports and figured that would be enough and continued, unsure if I would get a response. Several hours later and quite a number of grid imports later, I had almost gotten all buildings for downtown Athens up when I got this in mid-import:

Error while using JOSM for OSM imports.
Error while using JOSM for OSM imports.

Yep, I was blocked. Stopped dead in my tracks. I messaged them again to ask why I had been blocked and received “You continued importing.” with another message finally responding to my previous reply. And then I got to watch as my contributions began to disappear from the map. The block on my account was eventually lifted, but what incentive did I have at that point to continue? This was the second time, after all, that I tried to contribute in my own community and was failing again.

What I learned from this is if I want to pursue imports again, I will have quite a wait and there is no guarantee that my request will be approved. I understand there is a protocol in place now, which I just became aware of, but something that confuses me a bit is that you can download the OSM planet files with all the data to create an alternate branch of the OSM map. The source code for everything OSM-related is out there seemingly to encourage people to do just that. You can add this dataset to your server and edit your alternate version there using JOSM, I think, and maybe others. You can add all kinds of stuff to that map either by field observations, GPS, or importing. When you are done, the OSM licensing agreement says you must upload any changes back to the main OSM map. So at that point, how will OSM know where the data really came from, if it was digitized in or imported? How will OSM know if you followed the guidelines in the OSM Import Wiki? How will they manage such an import back into the main map? This isn’t even my revelation as I have read a bit about others branching the OSM map because they knew their data would not be appropriate or acceptable for the main OSM map. Of course this is also not something OSM really wants because, well, it’s one map for the OSM “community”, right? I am hoping someone from the OSM community will eventually like to discuss these things with me or perhaps they will choose to just further alienate me, and others like me, instead. I am seeing a few rising up slowly to the surface that may just be ready. So, yes, I remain hopeful.

In the end, I think OSM has a very different interpretation of what community and open is versus what others of us might consider it to be. That’s perhaps my biggest misunderstanding in all of this. Anyway, here’s hoping we can resolve this. Cheers!

Click here to visit OSM and see what's left of the building footprints in Athens.
Click here to visit OSM and see what’s left of the building footprints in Athens.

Update – 10.22.13: Some of our concerns were brought up in last night’s OSM-US Board meeting and we are encouraged that we might move forward soon with continuing to map Athens. If all continues to go well, the next step will be completing the documentation in the OSM Wiki for our project and begin the formal/informal (unsure which) training process or some kind of overview for the importing process. I’m very interested to learn what I was doing wrong and what the right process is.

Update – 10.23.13: Interesting timing. MIT Technology Review posted an article called The Decline of Wikipedia which seems to also explain some of what we have been observing with OpenStreetMap. Fascinating.

Update – 10.30.13: Giving a shout-out to all the folks from OSM Blog.DE! Ich gruesse Euch alle! Also, I’d like to make a tiny correction (nur ein ganz kliene Korrektur) to what is posted there. This blog was about my (Carol’s) experience attempting to contribute data, but more importantly, it discusses my experience as a new contributor in the OSM Community.

Update – 11.05.13: This is ongoing but appears to be making some progress, at least I think so. If you would like to follow the discussions going on currently within the Import-US mailing list, you can view  the archive here: https://lists.openstreetmap.org/pipermail/imports-us/. Interestingly enough, there is another mailing list I found out about through our discussions called Diversity-Talk that “was created as an explicit acknowledgement that the OSM Community is not an inclusive as it should be, claims to be, or aspires to be.” You can view or join that group here: https://lists.openstreetmap.org/listinfo/diversity-talk. Randy has mentioned that there might need to be a mailing list group for GIS working in OSM to discuss GIS data submissions before they are sent to the Import groups for further evaluation but we aren’t sure how many GIS folks are in the OSM world who would be interested in that. Are you? Let us know!

Update – 11.24.13: I seem to have hit a wall communicating with OSM. I’ll be approaching this from a different angle. The thought is now that we follow all the steps to do an import and document that process to share its success or failure. Once we begin, I’ll start a new blog post and keep updating it.

Comments

  1. says

    I’m completely sympathetic to your situation (I ran into it early on in the Chicago buildings import), but “community” doesn’t mean “freely load data into OSM”, it means consulting with the people affected by change you’re making, to check that the license of the original data is compatible with OSM’s, the tagging conversion is sensible, and that any existing data is accounted for.

    I completely agree that in this situation the response was less than helpful and that there are some heavy-handed people pushing weight around that isn’t really representative of the overall community, but they’re the ones that have stepped up to solve a problem. If you don’t like it, join them (the Data Working Group) and help come up with (a) saner rules and (b) more friendly and inviting interactions.

    • Carol says

      Ian, I understand the need for rules, but contributors can’t be expected to follow what they don’t know about. I just sent a horribly long email with deeper explanations to the Data Working Group. I hope you are in that group so that you can also see why it is perceived that I did not follow the rules. I provided my insight into several things including the rules and interactions. We are on the same page and I’m trying to make this experience into a constructive exercise. After the breakdown in communication yesterday, I felt that by writing this post I could bring more attention to this in the hopes it will help the OSM community and contributors.

  2. says

    “When you are done, the OSM licensing agreement says you must upload any changes back to the main OSM map.”

    I think this might be at the heart of the issue here. Honestly, it doesn’t say that at all.

    The agreement says that you must be prepared to make your changes available under the same terms. In other words, OSM has the right to take your changes if it wants to. But there’s an “if” there.

    • Carol says

      Richard, I did a poor job making that point. Thank you for clarifying that. So would you be in favor then of branching the OSM map instead of the main map supporting imports? Perhaps that’s a discussion to have.

      However, it’s not the heart of the issue from my perspective, though that may be the heart of it for OSM. My issue has more to do with communicating the rules of import edits by displaying them on the http://www.openstreetmap.org website and in any OSM editing tools that currently allow bulk import edits. You can’t expect someone to follow the rules if they are not clearly displayed and defined. The approach is also counter-productive for building a community.

      • says

        Oh, absolutely. OSM’s communications in general, and documentation in particular, have been sorely lacking since the start of the project and desperately need volunteers to help improve them. Sadly I found last year that, when trying to tackle the communications side of things, the pushback from (shall we say) entrenched interests was such as to make it pretty much impossible. :(

        • Carol says

          I’m hoping this experience opens a dialog that will support improvements we can all benefit from. I’ll post a portion of the email that I sent to the OSM Data Working Group (who have reached out to me, by the way) that explains better what I experienced along with my observations and possible suggestions.

          **********************************************************************************************************************************************

          Thank you to anyone who reads this. I know it’s long but I felt it best to go ahead and just put it all out on the table.

          First, I tried to sign up last night for the @import and @import-us mailing lists but I’m not sure if I am part of the list yet.

          Second, confession time. My sins against OSM are as follows:

          Uploading “bad imports” – I wasn’t uploading the building footprints in one go around because we, Randy and myself, had decided that we could better identify issues and address them immediately if I imported them in sections. I used Census Block Groups as my grid which seemed like manageable sizes. Unfortunately, I did make a mistake with my first two imports where I neglected to notice that tags are case sensitive. Fortunately, because of the way we were doing the uploads, it was easy to identify the issue and resolve it with a revert. Side note: I noticed that in the History, all of my uploads were labeled “bad import” with no correspondence explaining why that would be true. They came in fine and were tagged appropriately after my first two snafus.

          Not following the “guidelines” – Until my first email from [DWG member], I was unaware of any guidelines for importing data. There is no mention of this on http://www.openstreetmap.org. There is no indication on OSM.org that imports are not only “tricky” but that they are actually frowned upon by some members of the OSM community, some of which are on the OSM import team. There is nothing on OSM.org that suggests that there are guidelines, or, as someone else suggested, non-negotiable rules set in place that result in work being erased. Without this knowledge, I was pretty much set on a path for failure before I even started.

          No formal licensing statement for the data – I was including “ACCGIS” in the source of the imports but because I didn’t know about the OSM Import Wiki, I didn’t know there was a place for this information. I will work tomorrow to get a formal statement as to the licensing agreement Athens-Clarke County GIS has pertaining to the data I acquired on Friday. However, after viewing the Import Catalogue page, I have to say that it is still unclear as to how simply linking to Creative Commons or stating “Public Domain” is considered an official statement from the source data provider. There seems to be little to no standard here and I could essentially write in whatever I wanted if I were less than honest….or is there someone in the OSM community that is verifying this information with the source data provider?

          As for my previous bad experience, it was very similar to this experience and I’d rather move forward and focus on the present rather than rehash something that happened several years ago. The difference is, I threw my hands up and walked away from OSM before I got to actually import anything so I never became an “issue” from the OSM perspective.

          The key to all of this is that I was essentially being punished for rules I was expected to know but were not clearly displayed or defined. I can’t be the only one that has mentioned this or simply walked away entirely from OSM (as I did my first time). I believe that having guidelines for import data is as equally important as having guidelines for conduct for OSM representatives or “ambassadors”. Perhaps there needs to be an established support/welcoming committee that makes first contact to understand the “flagged” contributor’(s’) motivations and intentions to offer assistance with the import process keeping in mind that perhaps the data imports are driven by the local communities themselves. If they are repeat offenders, by all means send in the hounds.

          It feels a bit like OSM is only there for communities that it wants to be there for, which at the moment seems to be the manual editors, not the import editors. That’s ok, but then say that. State it somewhere clearly, not only on the website but within any OSM editing tools that “allow” bulk imports. However, if you look at local communities in the way I do, you see the connections and variety within them make each one unique, consisting of any number of families, schools, churches, businesses, and local government, understanding that all these entities are equally invested in their communities. Each one a potential OSM contributor. This means that a participating community might have manual and import OSM editors. I had to wonder last night what would have happened had I painstakingly digitized all those building footprints into a shapefile using ArcMap and some aerial photography and then attempted to go about the import. Just because I chose to do my work somewhere other than the OSM tools provided, I would have been flagged as soon as I began my imports. However, if I had simply worked away in iD or Potlatch and made the same tagging error I did yesterday, I’d work away relatively unnoticed. That seems a bit odd, doesn’t it?

          Lastly, understand that I TOTALLY understand the issues you must have dealt with concerning imports. This isn’t about having rules, it’s about communicating them to your contributors in a way that doesn’t make them want to just give up.

          I do see the value in OSM and I do want to contribute, not just for OSM but for my community. Yes, I am passionate about this, which is why I have spoken up. I’d like to see the guidelines clarified and improved upon. I see this as an opportunity for both of us.

          Sincerely
          -Carol

          • says

            “I’d like to see the guidelines clarified and improved upon”

            works better in OSM as:

            “I’d like to help clarify and improve the guidelines. How can I be of use?”

            :)

  3. ksutoebee says

    Your situation is definitely unfortunate but the OSM community has developed these import guidelines for a very good reason. We don’t just want all the geodata we can get our hands on. We want good – and maintainable – geodata. OSM’s technology has developed from the ground up based on the idea of individuals uploading data that they have carefully collected and digitized and integrated into existing data. Imports are huge dumps of data, often comparable to several man-years of “traditional” OSM mapper time. Obviously this means that it can be a great help to completing the map. But it also means that if anything is off with the import, even slightly, it can cause YEARS worth of work for people to correct. The biggest import was the TIGER road data in 2007. Just last year we finally fixed most of the routing errors in that data set. The tags are still a mess and we are still working on a way to update from newer versions of TIGER. Similar story for administrative boundary and GNIS data which is still terrible in some places. And that’s just here in the US. Other communities have faced similar problems with other imports, big and small. We’ve been burned by well meaning people enough times that it was decided to bring some more oversight to the process.

    Once an import is uploaded it becomes part of the global OSM data set and is not nearly as easy to update or correct as you might think. What if someone has modified an imported object? Your update/correction may wipe out their data. Now you have to manually review things. The famous quote “measure twice, cut once” applies very much here. It is a lot easier to fix glitches before the initial upload. This really is, in my opinion, the primary reason for the import guidelines and why we have started enforcing them more.

    Specific problems that I have seen with past imports:
    - no simplification. Perfectly square buildings that could be mapped with 4 nodes getting imported with 100 nodes in them can cause severe problems for others trying to map in the area.
    - bad tagging. Your capitalization error is pretty mild. Including all attributes from shapefiles is a common mistake. 90% of them are useless to OSM and will be useless to the importer as soon as people start editing the imported data and modifying them.
    - Ignoring existing data. This is a *big* no-no. Stomping on another users data is the opposite of community building. Just because data comes from an “authoritative” source doesn’t make it better or more accurate than what is already in OSM.

    A lot of these errors can be spotted very quickly by experienced mappers. This is why community consultation is an essential part of the import process.

    No one ever intends to perform a bad import. It’s just that doing a good import is somewhat challenging because of how OSM is built. Spoken as someone who has done some (small) imports myself.

    The good news is that building footprints are definitely valuable to OSM. Such imports have already been performed in several places and have yielded good results. So unless your source data is terribly out of date or has some other major flaw in it, I am certain the community will not reject it.

    • says

      We agree on the process for imports 100%. Now that I know that the guidelines are now hard and fast laws. Guidelines suggest something less formal (in our world) – and as I explained the last time I did an import there weren’t any guidelines. It was prepare and test and pick and test and then pray when you uploaded.

      That’s been somewhat the entire thought during this – if you goof up – and we did – what is the correct response from the community. I don’t think it was the one we got – which has led to a ton of emails and discussion both good and bad. I think overall good – and quite fascinating.

    • Carol says

      Your comment is appreciated. Again, there is no dispute over having guidelines or rules. It’s unfortunate that our message is getting lost behind that. This is about the community that OSM claims to represent versus the community it presents, two very different things from my experiences. I have suspended all edits/imports since my account was blocked and will continue to wait until I have more answers to my questions. They are being discussed which is all I can ask for at this time. As I told the Data Working Group in a later email, “…I am making this effort because I see the value in OSM and I want to help my local community. I am offering to contribute to both communities but I’m evaluating if the OSM culture is something I want to introduce to my local community members who are interested in supporting this effort. So the vetting process goes in both directions.”

      We are not even just interested in imports. We want to get out and map just like you all do. We want to organize events in our local community and help with mapping what is important to them. Our city has very little data in OSM right now.

      I look forward to the challenges you have described. I heart geodata, just like you do.

  4. malenki says

    Thanks for not walking away again but still hanging on and discussing annoying matters.

    For the “point out to import rules in tools”: would it be a good idea to let JOSM pop up a warning with a link to the Import guidelines when someone wants to use it to upload a bunch of data without creating this by hand? I assume it is possible that the programmers of JOSM can have a look how long this instance has been running and maybe check what distances the mouse pointer has been running and how much clicks were made – and compare that to the amount of new data to upload.

    • Carol says

      Malenki, thank you for your response and suggestion! Having a message in OSM editing tools which allow imports that warns contributors of the consequences and guides them to the protocol page for imports is something that we have suggested. The good news is that the OSM-US Board is currently evaluating this! We didn’t take the idea as far as you have, though of course that could be a possibility depending on the decisions made. I think that simply warning people of the consequences would be enough in most cases and create the desired effect of having an interested contributor contact one of the work groups in charge for more information. Placing too many upfront restrictions or gatekeepers in place might discourage some contributors, since that kind of stance usually takes place in more closed communities than open ones.

      Keep in mind that the current landing page for the OSM wiki for Imports states nothing about manual imports, only “scripted imports and automated edits”. I was checking and fixing everything that I wanted to import before adding it to OSM. Therefore, once I knew about the guidelines, the information found on that page still did not make me think I was doing anything wrong because I didn’t interpret what I was doing as either scripted or automated.

      If the OSM community does not want anyone except the few individual within a special work group to do imports, then they could just cut off access to those tools that allow imports. However, I think this would hinder their progress since they obviously want some of the GIS data (i.e. the NYC building footprints) that already exist and seem to need the additional support to accomplish their goals, whatever they may be at this time. Not to mention, it kind of takes the “open” out of OSM.

Trackbacks