Data Wrangling: The Truii Journey
Truii is the result of a long standing itch that I couldn’t find someone else to scratch. My background is in environmental research and quantitative modelling of ecological systems: particularly the field of hydrology and environmental water management www.yorb.com.au. I faced the same problem whilst I worked in a variety of roles from Chief Scientist for Water Quality Monitoring and Assessment with the Queensland Government, running collaborative research projects across government and research institutions from CSIRO and more recently conducting collaborative research and software development as a contract research company (Yorb). The problem I faced for every project was how to keep on top of the project data. We would routinely have a project team of 5-10 people from 3-5 different organisations all interacting with the project data.
The result of many hands in the data-pot is a lot of confusing naming conventions, multiple versions of almost identical files and ultimately a lot of data wrangling and re-wrangling every time the data is updated. We didn’t want a complete statistical package – we already have plenty of those, or a complete GIS package. We simply wanted the data to be in a place where everyone could access it, version control would be managed and all the basic data wrangling functionality would be in one place. We also wanted to be able to instantly visualise the data – simple graphs and maps that are always linked to the data so as it gets wrangled you get instant feedback on how that looks in a plot.
What we didn’t want was a ‘database’, you know the type, a complicated series of tables and mandatory fields that only one person knows how to use and which are a pain to write queries for unless you are database savvy. We wanted to make the data easy to interact with for the project team- some of whom are not specialist data analysts or statisticians but practitioners or data collectors. We didn’t want a full stats package or full GIS package because they are difficult to learn, but some basic functionality that will allow you to see the key insights from your data that you can always engage a data analyst to further explore. We needed something simple yet powerful. Once we got to this point of a simple flexible online data sharing and analysis platform it was a small step to embrace data democratisation.
Data democratisation
The concept of data democratisation is one close to our hearts. After working in a number of roles across government and research organisations we have had a glimpse of the tremendous amount of time, intellect and money invested in creating some really useful public data assets. Many government and research organisations are only too willing to share their data, but the sticking point then is how to empower individuals and groups to make the most it. We reckon that what was needed was a platform that made it easy to find and interact with public data as well as empower you to take control of your own data. In our view everyone has a data analyst inside – we all get how revealing a good graphic can be (see the life expectancy chart – I’m pretty chuffed about living in Australia) but we don’t necessarily have the experience of doing the number crunching ourselves. With our global data records doubling about every 40 months (yes there is twice as much data in the world today than in late 2010!) we wanted to contribute to empowering the non-data specialist the release the analyst within. Data democratisation isn’t exactly big data in the corporate sense, but the same concepts apply – of the vast collection of data how do I sift through it to find the bits relevant to me and combine it with my understanding of the world to make some interpretation that I can use to help me better understand the world.
Life expectancy by country: Source: http://www.disabled-world.com
With these combined passions of data democracy and trying to make project level data management a little easier we built Truii. Over time and with feedback from our users we hope that Truii provides value to other project teams and individuals who collect, store, analyse and report on data.
If you enjoyed this post, don’t forget to to sign up to the blog – form on the right.
