Archive for August, 2015
Huge Increase in Sharability by Combining Git and QGIS
After tweeting today about the Unmitigated Amazingness that is a QGIS + Git workflow, someone suggested that I write a blog about my experiences in this regard. Unfortunately today is a deadline day for a portion of what will become my next book* so I can’t put a lot of time into a full-blown explanation of how this workflow will CHANGE your life. But I can give you a taste.
To that end, in a nutshell, and realizing I might be leaving out some important bits of information and because I suspect there are a lot of people out there who’ve never used this workflow before in their life, I’m deliberately not using the technical Git terms pull, push, etc., just to keep it simple:
- You begin by installing Git on your machine
- Unless you want to use command-line Git the two choices that I’m familiar with are the following combinations: Bitbucket for your online stuff** and SourceTree to manage updating that stuff OR GitHub for your online stuff and GitHub Desktop to manage updating that stuff
- You create a project (aka “repo”) on Bitbucket or GitHub
- Copy it locally via SourceTree or GitHub Desktop
- (alternatively you can create it locally and then create it in the cloud)
- I suggest that all the geodata you’ll use goes in one folder within the repo while all the QGIS projects you create go in another, any images or other odd things that you need in your QGIS projects could go in a Misc folder
- You do your work normally: create a QGIS project, add data, but do it all within that repo folder on your machine
- Open SourceTree or GitHub Desktop on your machine and it’ll tell you that you made changes like that you added data and that you created a project, you can choose if you want all of that to be put in your Bitbucket or GitHub cloud. If you do want it up in the cloud, you use one of those programs to sync it up with your cloud repo
- Your collaborators simply use their own SourceTree or GitHub programs to put that project and its data on their machine exactly as you uploaded it. If they make changes that they want you to see then they can also sync those up, then your SourceTree or GitHub alerts you about the changes
And guess what?! In this way your QGIS project and all the files it uses are easily synced with other people. You don’t have to zip anything up. You don’t have to locate all the places where you put your data because you’ve already put it all in that repo/geodata folder. You can order online cheap Viagra that will help you get a strong erection in bed. There is NO repairing of data source paths on your collaborator’s end! Think of the possibilities! It is truly a wonderful thing.
Now, I really am sure that I’ve left a whole lot of info out while trying to create this simple bird’s-eye view of the process but hopefully this provides a taste of the possibilities so that you can go learn more. After using both the Bitbucket/SourceTree workflow and the GitHub/GitHub Desktop workflow I personally find the GitHub/GitHub Desktop workflow to be a bit easier. Its desktop program is a little more streamlined as it “exposes” less of the advanced capabilities.
————Edited 9/1/2015 to add: Soon after posting this a reader pointed out that James Fee and I had coincidentally written about similar topics on our blogs yesterday. His topic was spatial DATA versioning while mine was spatial PROJECT versioning. To be clear, the project-sharing that I’m talking about in this post doesn’t really involve changing data at all. In fact, what I’ve been doing is collaborating with someone else on cartography designs using QGIS, and we needed a way to see each other’s designs (i.e., QGIS projects) and tweak them and send them back and forth. So yes, while we do store spatial data in our git repos, we aren’t concerned about that data changing, just really the styling of the data within the QGIS projects themselves. Fee explains much better in his follow-up post GIS and Git. ————
*First public hint about my next book: it will be about cartography! 😉
**Highly technical here
Very simple overview of how to use a QGIS-Git workflow to dramatically increase sharability: http://t.co/WIC9hkshSl
— Gretchen Peterson (@PetersonGIS) August 31, 2015
@PetersonGIS it’s a great workflow. How do you deal with changes in binary data? Potentially huge repo if you commit many changes to .shp’s
— Kristian Evers (@kbevers) August 31, 2015
@kbevers @NickBearmanUK @PetersonGIS Put your map data in a “geogig” repo: https://t.co/GxAgrioqX3
— Barry Rowlingson (@geospacedman) August 31, 2015
.@erikfriesen @PetersonGIS being doing this for almost a year now, no regrets. easier than geogig albeit the granularity of diffs missed
— Antonio Locandro (@antoniolocandro) August 31, 2015
Simple version control for #QGIS with Git: http://t.co/vBt7YdZWWT /by @PetersonGIS
— zanols (@zanols) August 31, 2015
@PetersonGIS how large of files do you keep under version control?
— Nick Swanson-Hysell (@polarwander) August 31, 2015
@antoniolocandro @PetersonGIS I'm wondering if maybe a hybrid of the two would be manageable. Git for map project files, geogig for data.
— Erik Friesen (@erikfriesen) August 31, 2015
@erikfriesen if you aren't using a database for example using no shapefiles using GML or JSON in GitHub could be enough @PetersonGIS
— Antonio Locandro (@antoniolocandro) August 31, 2015
@erikfriesen we actually manage map doc in git and version control in postgis through custom solution @PetersonGIS
— Antonio Locandro (@antoniolocandro) September 1, 2015
@antoniolocandro @erikfriesen @PetersonGIS Stupid question: what if collaborator changes dir path in a repo, does it break everything?
— Christopher Rice (@colocarto) September 1, 2015
@polarwander The repo I'm working with right now has 1.2 GB in it and 200 files (there's an additional 400 small git files too).
— Gretchen Peterson (@PetersonGIS) September 1, 2015
@colocarto no stupid question, hmm it might for map project not sure for data itself @erikfriesen @PetersonGIS
— Antonio Locandro (@antoniolocandro) September 1, 2015
@PetersonGIS @antoniolocandro @erikfriesen Slick, can it do rollbacks too? Postgres support also?
— Christopher Rice (@colocarto) September 1, 2015
@antoniolocandro @PetersonGIS absolutely. I was just thinking of the case of using postgis for a datastore
— Erik Friesen (@erikfriesen) September 1, 2015
@erikfriesen for that you need geogig but project left boundless to eclipse nothing new in a while @colocarto @PetersonGIS
— Antonio Locandro (@antoniolocandro) September 1, 2015
@antoniolocandro @colocarto @PetersonGIS damn
— Erik Friesen (@erikfriesen) September 1, 2015
@erikfriesen @antoniolocandro @PetersonGIS Agreed. If I'm using QGIS it feels strange not utilizing PostGIS
— Christopher Rice (@colocarto) September 1, 2015
@PetersonGIS and it's smooth sailing when there are changes to big files?
— Nick Swanson-Hysell (@polarwander) September 1, 2015
@PetersonGIS I ask as I was just at a workshop where they advised against using git for quite large files.
— Nick Swanson-Hysell (@polarwander) September 1, 2015
@PetersonGIS but as you say it is great to just have everything up on Github.
— Nick Swanson-Hysell (@polarwander) September 1, 2015
@PetersonGIS I was working on project where the .git directory of the repo was ballooning in size due to changes to graphic files
— Nick Swanson-Hysell (@polarwander) September 1, 2015
@polarwander @PetersonGIS Did they say why?
— Cian Dawson (@cbdawson) September 1, 2015
@cbdawson @PetersonGIS but it seems to work pretty well to have large files in repos and it is nice to treat all file types the same way
— Nick Swanson-Hysell (@polarwander) September 1, 2015
@spatialadjusted @cageyjames It sounds like @PetersonGIS is thinking along the same lines http://t.co/OffokmVufL
— Phil Knight (@PhilipWhere) September 1, 2015
@PhilipWhere @spatialadjusted GeoGit = versioning for data, whereas I was speaking more to versioning a project.
— Gretchen Peterson (@PetersonGIS) September 1, 2015
@PetersonGIS great blog [again] Gretchen, what format data are you using though? Is it all shapefile?
— Nicholas Duggan (@Dragons8mycat) September 1, 2015
@Dragons8mycat In our case we have shapefiles and a SpatialLite osm db of the Seattle area.
— Gretchen Peterson (@PetersonGIS) September 1, 2015
@PetersonGIS Just curious if your tried any branching/merging of your binary data files.
— Bill Dollins (@billdollins) September 1, 2015
@PetersonGIS Thx. Specifically, I want to see if this works: http://t.co/e37RfyMXcX
— Bill Dollins (@billdollins) September 2, 2015
foundational education vs pinnacle education
Posted by G.P. in Cartography Profession on August 20, 2015
In education there’s a debate over whether a student should start with foundational knowledge and build from there or whether they should start at a place (i.e., framework) that’s further along so that they can reach even higher at an earlier age than those who have to, for example, spend time memorizing addition and subtraction problems when they could have just relied on a calculator. (Apparently this is something they do in Sweden?)
Recently we in GIS land have brought up the question of whether a 12 week course that promises to teach you to be a data scientist is snake oil or not. Given: an advanced statistics degree is a huge accomplishment and we need some people who know those fundamentals.
The question is, though, what if we taught the tools (but not the theory, thus saving people time) to people who are experts in other disciplines? I think there’s definitely a place for modularized education like this. And providing this 12 week option–if the quality of the teaching is good–could enable more and better advances.
It’s kind of like saying: “we shouldn’t enable people without expertise in cartography or GIS analysis to make maps by providing shortened educational opportunities.” Well guess what? We didn’t provide shortened educational opportunities and still our entire profession has been up-ended over the past 10 years by people who have had virtually no expertise in GIS analysis or cartography.
The gut tells us that too much territoriality never leads to new thinking. If the 12 week course turns out to be a disappointment due to poor teaching that’s certainly something to complain about. But to complain about the spirit of the course? Perhaps that’s sour grapes.
Disclaimer Idea
I’m thinking something like this for your next cartographic disclaimer? Adjusted for mappiness of course.
We DID NOT walk 500 mile.
And we WOULD NOT walk 500 more.
~ The Disclaimers.
— Ollie Garch (@ojedge) November 30, 2014
Recent Comments