linux.conf.au 2017 Hobart – The future of open source

This year, linux.conf.au 2017 headed to the picturesque state of Tasmania, to Hobart’s Wrest Point convention centre, and the theme of the conference was ‘the future of open source’. My key takeaway from the conference was that:

The future will be built on trust, and trust takes many forms –
  1. Trusting that data and systems have confidentiality, integrity and availability – traditional security
  2. Trusting that digital experiences will be pleasant, safe and as frictionless as possible – user experience and community experience
  3. Trusting that people will build the future that they want – agency and empowerment

This blog post is going to explore some of my picks from the conference through these lenses.

Security, privacy and integrity

Security, privacy and integrity was a recurring theme of the conference.

Michael Cordover – The Future of Privacy

Michael Cordover‘s talk, ‘The Future of Privacy‘, was perhaps the most thought-provoking talk around privacy. Michael provided a history of privacy, underscoring how technology has shaped notions of what it means to be left alone, and what it means to have personal data remain private. In our ubiquitously-connected, always-on world, it’s becoming harder to delineate what informed consent means – given that data can be inferred by association (which is exactly how Tapad‘s technology is designed). It’s also harder for people to be aware of how apps and platforms are using data – terms and conditions are hard to read, and detract from usability. Practically, it’s hard to own your own data – you essentially have to run your own services. Open systems, decentralisation, federation and non-permissive by default are Cordover’s answers to these problems – but these all pay a usability price. In Cordover’s words,

There’s no easy path forward that ordinary people can take.

David Bell – In Case of Emergency: Break Glass – BCP, DRP, & Digital Legacy

As a first time linux.conf.au Speaker, David delivered a solid presentation covering business continuity planning, disaster recovery planning and digital legacy. His focus was on ensuring that appropriate planning was done before business interruption events. He also covered personal digital legacy – an almost-unexplored topic – for example – would the people you leave behind when you die know how to access your passwords?

George Fong – The Security and Integrity of the Internet

George Fong (previous Chair of Internet Australia, current Deputy Chancellor at Federation University) delivered a very strong presentation which advocated for the defence of security and integrity of the internet, largely because governments and other non-technical actors in the ecosystem don’t trust the internet – the “cybers”.

The key takeaway from George’s talk that continued to resonate for days afterwards was:

Trust is the byproduct of integrity

Using examples such as Dirty COW and Heartbleed, Fong opined that we as an opensource community need to make sure that Linux – which the foundation of the internet rests upon – is trustworthy. Bugs are only shallow if many eyeballs are on them, and all too often there aren’t enough eyeballs. Using the analogy of seatbelts, and how few of us would ever feel safe and secure driving without one, he articulated how the internet in many ways is still a frontier, devoid of strong security measures and protocols that ensure safety and integrity – and therein, trust.

Touching on another key theme of the conference – agency and empowerment – he urged the audience to grasp that they, we, the open source community are the voices of the internet. Fong encouraged us to use those voices to better educate the public on what we do – we need to promote our activities to strengthen integrity. Things are broken – and we’re not helping. It’s up to us to fix the problem.

On a side note, as the recently-elected President of Linux Australia, I’m looking forward to working with George, and recently-appointed Chair of Internet Australia, Anne Hurley, to identify how we can work collaboratively together on some of these aims – as Internet Australia and Linux Australia have some overlap in mission, values and remit.

Jon Oxer – Network Protocol Analysis for IoT Devices

Nowhere is security, privacy and integrity more pressing that in the field of Internet of Things. There were several IoT related talks this year, but two that stood out. Firstly, Jon Oxer‘s talk on Network Protocol Analysis for IoT Devices was an eye-opener into the history of the radio frequency spectrum, how some of it is unregulated, but moreover how device protocols can be reverse engineered with simple equipment and a penchant for code-breaking. Oxer showed how simple it is to launch a man-in-the-middle attack on IoT devices on the RF 422 MHz band by intercepting their transmissions, decoding their protocols and then using a playback attack. We definitely need better encryption in IoT.

Christopher Biggs – How to Defend Yourself from your Toaster

Christopher Biggs also gave an excellent security talk around IoT – ‘How to defend yourself from your toaster‘, however he tackled it from the perspective of an IoT device manufacturer or developer – clearly articulating what features and functions should be included in new IoT devices. Although he didn’t frame it as such, his talk was basically outlining a maturity model for IoT devices. For example, devices with low maturity have poor user interfaces, no provision for maintenance, and employ poor security practices – such as having insecure protocols (such as telnet) available. He provided useful advice for improving maturity, for instance port-scanning devices to see which ports are open, and what data is being transmitted. One of the key takeaways here was that if you are designing an IoT device, or managing a fleet of IoT devices, that you need to get someone else to do the hard parts. Apple, Amazon and Google all now have SDKs available for IoT, but the drawback is that most of them are not open sourced.

Biggs spoke of a metric that I hadn’t heard before in this space – MTT1C – mean time to first compromise – or the length of time it takes an IoT device to be compromised once it’s placed on the public internet. This got me thinking that I haven’t seen anywhere a capability maturity model for enterprise IoT – for instance the practices, support, metrics and continuous improvement that would be used in a large organisational deployment of IoT. Perhaps this is something that the standards bodies in this space – Open Connectivity Foundation, BITAG and Resin.io – will develop in time.

Dr Vanessa Teague – Election Software

Dr Vanessa Teague gave one of my favourite talks of the conference on e-voting systems, and the general problem of end to end verification. Using a number of examples of how companies have (or have not) implemented verification, she articulated a number of anomalies with current e-voting systems in NSW, which are soon to be used in both WA and Victoria. Given the recent controversy around United States elections, this talk was particularly timely, and gave rise to a number of uncomfortable questions – such as just how many votes does it take to change an election result, and possibly the course of history?

One of the most resonating points within Dr Teague’s talk was the rejection of an e-voting system – V-Vote – which had superior verification capabilities, but poor user experience and usability qualities. This touches on the second theme which emerged from #lca2017 – it is not sufficient for a product, tool or platform to be functional – it must also have form. People are persuaded by the shiny – and rather than scoff at this – default behaviour for a lot of our community – we need to recognise and respond to this.

Dr Teague was an engaging, humourous and articulate speaker, and I’d really like to hear more from her in future conf lineups.

User experience and community experience

It may be unusual to relate user experience and customer / community experience to trust, but I see it as fitting. Our experience with a task, a process, or an interaction either enhances or erodes our trust in the organisation, platform or person with whom we’re interacting.

Donna Benjamin – I am your User, why do you Hate me?

Donna Benjamin‘s excellent talk aimed to bring a user experience / human-centred design element to open source developers by questioning some of the fundamental ‘defaults’ we tend to hold. Using project onboard experiences as a lens to explore how we treat newcomers, she demonstrated that our actions are turning people away from opensource – exactly the opposite effect that we’re aiming for. She outlined how contributions in triage, review and testing are not valued as highly as code contributions, again presenting a barrier to increasing participation and diversity. Benjamin argued for the open source community to see users not in terms of what they can’t do – develop software – but as people – with needs and emotions.

This talk highlighted for me the lack of design thinking, human-centred design and user experience practices that are adopted not just on open source products, but to communities in general. Lowering ‘friction’ – the antithesis of good user experience – is something that both open source products and open source communities need to get better at.

Rikki Endsley – The proper care and feeding of communities and carnivorous plants

Rikki Endsley‘s talk likewise touched on how managing communities is a complex task, often fraught with pitfalls. The key takeaway was that you can’t change everything at once – you need to change elements of the community carefully, then have the metrics available to measure the impact of the change.

VM Brasseur – The Business of Community

VM Brasseur‘s talk was a practical guide for people working inside companies to ‘sell’ support of open source projects to management. This talk was framed along three key topics – benefits, costs and implementation. Benefits such as word of mouth marketing, stronger brand recognition, and more effective upstream support are all selling points. One of the strong points of this talk was the recognition of in-kind / non-monetary support to open source communities by business, such as the provision meeting space, marketing, guidance, leadership and mentoring. In particular, Brasseur cautioned that businesses should ask the community what it needed – rather than making assumptions – and providing, for instance, unwanted promotional goodies. Although implementation plans will vary across companies, Brasseur provided some generic advice, such as having clear goals and objectives for community support, setting expectations and being transparent about the company’s intentions.

Nadia Eghbal – Consider the Maintainer (keynote)

Nadia’s keynote brought to the fore many simmering tensions within the open source community. Essentially, the burden of maintaining open source software falls to a few dedicated maintainers, who in some cases may be supporting a product with a user base of tens or thousands of uses.

Eghbal set out four freedoms for open source producers / maintainers, being:

  • The freedom to decide who participates in your community
  • The freedom to say no to contributions or requests
  • The freedom to define the priorities and policies of the project
  • The freedom to step down or move on from a project, temporarily or permanently

Whether these freedoms are embraced and used to support open source maintainers remains to be seen.

Nadia Eghbal keynoting linux.conf.au 2017
Nadia Eghbal keynoting linux.conf.au 2017

Agency and empowerment

The third key theme that was reflected in the conference programme was that of agency and empowerment – being the changes that we want to see in the open source world.

Pia Waugh – Choose your own adventure

Pia Waugh kicked off this theme, delivering the first conference keynote, where she gave a retrospective on human evolution, and then extrapolated this to the future of open source, articulating how we’re likely to see a decentralisation of power in order to strengthen democracy. She went on to challenge a number of existing paradigms, calling them out as anachronisms as the world has evolved.

This talk was full of Waugh’s trademark energy and vibrancy, and was an excellent choice to open the conference.

Dr Audrey Lobo-Pulo – Publicly Releasing Government Models

Dr Audrey Lobo-Pulo’s talk extended the open data movement by advocating for the public release of government open source models – financial and economic models used to assess public policy decisions – in essence, virtual worlds to explore the implications of policy.

The key takeaway from her talk was that industry and business also stand to benefit greatly from the release of these models, as they could then be combined with private data – in a unique public private partnership. Lobo-Pulo put forward the four components of government policy models (shown below) – and how each contributes the accuracy and validity of the model.

Karen M. Sandler – Surviving the Next 30 Years of Free Software

Karen‘s sensitive and tactful talk recognised the fact that as a community, many of our pillars and key contributors are aging, and that over the next few years we are likely to bid goodbye to many in our community. Her talk explored the different ways in which copyrights can be assigned after death, and the key issues to consider – empowering us to make informed and well founded decisions while we are in a position to do so. Few presenters could have handled this difficult topic with such aplomb, and as usual Karen’s grace, wit and wisdom shone through.

Closing thoughts

Again, linux.conf.au delivered engaging, thought-provoking and future-looking talks from a range of experienced, vibrant and wise Speakers – and again it was an excellent investment of time. The diversity of Speakers this year was excellent, if perhaps erring on the non-technical side.

Open source still faces a number of challenges – the ecosystem is often underfunded, maintainers are prone to burnout and we still haven’t realised that UX needs to be a key part of what we’re all about. But that’s part of the fun – we have the power to evolve just like the rest of the world.

And I can’t wait for a bit of history repeating at Sydney 2018!

Save

Save

Save

Save

Save

Save

My talk picks for #lca2017 – linux.conf.au

linux.conf.au 2017 heads to Hobart, where it was last held in 2009. I absolutely love Tasmania – especially its food and scenery – and am looking forward to heading over.

So, here’s my talk picks  – keeping in mind that I’m more devops than kernel hacker – so YMMV.

Executive Summary

  • Monday 16th – Networking breakfast, possibly some WootConf sessions and / or Open Knowledge Miniconf sessions.
  • Tuesday 17th – Law and policy Miniconf, Community Leadership Summit
  • Wednesday 18th – Future Privacy by Michael Cordover, In Case of Emergency – Break Glass by David Bell, Handle Conflict Like a Boss by Deb Nicholson, Internet of Terrible Things by Matthew Garrett.
  • Thursday 19th – Network Protocols for IoT Devices by Jon Oxer, Compliance with the GPL by Karen Sandler and Bradley M. Kuhn, Open source and innovation by Allison Randall and Surviving the next 30 years of open source by Karen Sandler.
  • Friday 20th – Publicly releasing government models by Audrey Lobo-Pulo

Monday 16th January

I’m keeping Monday open as much as possible, in case there are last minute things we need to do for the Linux Australia AGM, but will definitely start the day with the Opening Reception and Networking Breakfast. A networking breakfast is an unusual choice of format for the Professional Delegates Networking Session (PDNS), but I can see some benefits to it such as being able to initiate key relationships and talking points early in the conference. The test of course will be attendance, and availability of tasty coffee 😀

If I get a chance I’ll see some of the WootConf sessions and/or Open Knowledge Miniconf sessions (the Open Knowledge Miniconf schedule hadn’t been posted at the time of writing).

Tuesday 17th January

The highlight for me in Tuesday’s schedule is the excellent Pia Waugh talking ‘Choose your own Adventure‘. This talk is based on Waugh’s upcoming book, and the philosophical foundations, macroeconomic implications and strategic global trends cover a lot of ground – ground that needs to be covered.

As of the time of writing, the schedule for the Law and Policy Miniconf hadn’t been released, but this area is of interest to me – as is the Community Leadership Summit. I’m interested to see how the Community Leadership Summit is structured this year; in 2015 it had a very unconference feel. This was appropriate for the session at the time, but IMHO what the Community Leadership Summit needs to move towards are concrete deliverables – such as say a whitepaper advising Linux Australia Council on where efforts should be targeted in the year ahead. In this way, the Summit would be able to have a tangible, clear impact.

Wednesday 18th January

I’ll probably head to Dan Callahan’s keynote on ‘Designing for failure’. It’s great to see Jonathan Corbet’s Kernel Report get top billing, but my choice here is between the ever-excellent Michael Codover’s ‘Future Privacy‘ and Cedric Bail’s coverage of ‘Enlightenment Foundation Libraries for Wearables‘. Next up, I’ll be catching David Bell (Director, LCA2016) talking ‘In case of emergency – break glass – BCP, DRP and Digital Legacy‘. There’s nothing compelling for me in the after lunch session, except perhaps Josh Simmon’s ‘Building communities beyond the black stump‘, but this one’s probably too entry-level for me, so it might be a case of long lunch / hallway track.

After afternoon tea, I’ll likely head to Deb Nicholson’s ‘Handle conflict like a boss‘, and then Matthew Garett‘s ‘Internet of terrible things‘ – because Matthew Garrett 😀

Then, it will be time for the Penguin Dinner!

Thursday 19th January

First up, I’m really looking forward to Nadia Eghbal’s ‘People before code‘ keynote about the sustainability of open source projects.

Jon Oxer’s ‘Network Protocol Analysis for IoT Devices‘ is really appealing, particularly given the rise and rise of IoT equipment, and the lack of standards in this space.

It might seem like a dry topic for some, but Bradley M. Kuhn and Karen Sandler from the Software Freedom Conservancy will be able to breathe life into ‘Compliance with the GPL‘ if anyone can; they also bring with them considerable credibility on the topic.

After lunch, I’ll be catching Allison Randall talking on ‘Open source and innovation‘ and then Karen Sandler on ‘Surviving the next 30 years of open source‘. These talks are related, and speak to the narrative of how open source is evolving into different facets of our lives – how does open source live on when we do not?

Friday 20th January

After the keynote, I’ll be catching Audrey Lobo-Pulo on ‘Publicly releasing government models‘ – this ties in with a lot of the work I’ve been doing in open data, and government open data in particular. After lunch, I’m looking forward to James Scheibner’s ‘Guide to FOSS licenses‘, and to finish off the conference on a high note, the ever-erudite and visionary George Fong on ‘Defending the security and integrity of the ‘Net’. Internet Australia, of which Fong is the chair, has many values in common with Linux Australia, and I foresee the two organisations working more closely together in the future.

What are your picks for #lca2017?

Geelong Libraries by branch – a data visualisation

At a glance

Introduction

Geelong Regional Libraries Corporation (GRLC) came on board GovHack this year, and as well as being a sponsor, opened a number of datasets for the hackathon. Being the lead organiser for GovHack, I didn’t actually get a chance to explore the open data during the competition. However, while I was going for a walk one day – as it always does – I had an idea around how the GRLC datasets could be visualised. I’d previously done some work visualising data using a d3.chord layout, and while this data wasn’t suitable for that type of layout, the concept of using annulars – donut charts – to represent and compare the datasets seemed appealing. There was only one problem – I’d never tackled anything like this before.

Challenge: accepted

Understanding what problem I was trying to solve

Of course the first question here was what problem I was trying to solve (thanks Radia Perlman for teaching me to always solve the right problem – I’ll never forget your LCA2013 keynote). Was this an exploratory data visualisation or an explanatory one? This led to formulating a problem statement:

How do the different Libraries in the Geelong region compare to each other in terms of holdings, membership, visits and other attributes?

This clearly established some parameters for the visualisation: it was going to be exploratory, and comparative. It would need to have a way to identify each Library – likely via colour code, and have appropriate use of shapes and axes to allow for comparison. While I was tempted to use a stacked bar chart, I really wanted to dig deeper into d3.js and extend my skills in this Javascript library – so resolved to visualise the data using circular rings.

Colour selection

The first challenge was to ensure that the colours of the visualisation were both appealling and appropriate. While this seems an unlikely starting place for a visualisation – with most practitioners opting to get the basic shape right first, for this project getting the colours right felt like the best starting point. For inspiration, I turned to the Geelong Regional Library Corporation’s Annual Report, and used the ColorZilla extension to eyedropper the key brand colours used in the report. However, this only provided about 7 colours, and I needed 17 in order to map each of the different libraries. In order to identify ‘in between’ colours, I used this nifty tool from Meyerweb, which is super-handy for calculating gradients. The colours were then used as an array for a d3.scaleOrdinal object, and mapped to each library.

var color = d3.scaleOrdinal()
    .range([
        "#59d134",
        "#4CCA62",
        "#40C28F",
        "#33bbbd",
        "#36AFC7",
        "#3b98da",
        "#427DC9",
        "#5148a6",
        "#8647A8",
        "#BC47A9",
        "#f146ab",
        "#F03E85",
        "#F0355E",
        "#f0431e",
        "#F8880F",
        "#FFCC00",
        "#ACCF1A"
      ])
    .domain([
        "Geelong",
        "Belmont",
        "Corio",
        "Geelong West",
        "Waurn Ponds",
        "Ocean Grove",
        "Newcomb",
        "Torquay",
        "Drysdale",
        "Lara",
        "Bannockburn",
        "Queenscliff",
        "Chilwell",
        "Highton",
        "Mobile Libraries",
        "Barwon Heads",
        "Western Heights College"
    ]);

Annular representation of data using d3.pie

Annular representation of data - step 1
First step in annular representation

The first attempt at representing the data was … a first attempt. While I was able to create an annular representation (donut chart) from the data using d3.pie and d3.arc, the labels of the Libraries themselves weren’t positioned well. The best tutorial I’ve read on this topic by far is from data visualisation superstar, Nadieh Bremer, over on her blog, Visual Cinnamon. I decided to leave labels on the arcs as a challenge for later in the process, and instead focus on the next part of visualisation – multiple annulars in one visualisation.

Multiple annulars in one visualisation

Annular representation of data - step 2
Uh-oh!

The second challenge was to place multiple annulars – one for each dataset – within the same svg. Normally with d3.js, you create an svg object which is appended to the body element of the html document. So what happens when you place two d3.pie objects on the svg object? You guessed it! Fail! The two annulars were positioned one under the other, rather than over the top of each other. I was stuck on this problem for a while, until I realised that the solution was to place different annulars on different layers within the svg object. This also gave more control over the visualisation. However, SVG doesn’t have layers as part of its definition – objects in SVG are drawn one on top of the other, with the last drawn object ‘on top’ – sometimes called stacking . But by creating groups within the BaseSvg like the below, for shapes to be drawn within, I was able to approximate layering.

var BaseSvg = d3.select("body").append("svg")
    .attr("width", width)
    .attr("height", height)
    .append("g")
    .attr("transform", "translate(" + (width / 2 - annularXOffset) + "," + (height / 2 - annularYOffset) + ")");

/*
  Layers for each annular
*/

var CollectionLayer = BaseSvg.append('g');
var LoansLayer      = BaseSvg.append('g');
var MembersLayer    = BaseSvg.append('g');
var EventsLayer     = BaseSvg.append('g');
var VisitsLayer     = BaseSvg.append('g');
var WirelessLayer   = BaseSvg.append('g');
var InternetLayer   = BaseSvg.append('g');
var InternetLayer   = BaseSvg.append('g');
var TitleLayer      = BaseSvg.append('g');
var LegendLayer     = BaseSvg.append('g');

At this point I found Scott Murray’s SVG Primer very good reading.

Annular representation of data - step 3
The annulars are now positioned concentrically

I was a step closer!

Adding in parameters for spacing and width of the annulars

Once I’d figured out how to get annulars rendering on top of each other, it was time to experiment with the size and shape of the rings. In order to do this, I tried to define a general approach to the shapes that were being built. That general approach looked a little like this (well, it was a lot more scribble).

General approach to calculating size and proportion of multiple annulars
General approach to calculating size and proportion of multiple annulars

By being able to define a general approach, I was able to declare variables for elements such as the annular width and annular spacing, which became incredibly useful later as more annulars were added – the positioning and shape of the arcs for each annular could be calculated mathematically using these variables (see the source code for how this was done).

var annularXOffset  = 100; // how much to shift the annulars horizontally from centre
var annularYOffset  = 0; // how much to shift the annulars vertically from centre
var annularSpacing  = 26; // space between different annulars
var annularWidth    = 22; // width of each annular
var annularMargin   = 70; // margin between annulars and canvas
var padAngle        = 0.027; // amount that each segment of an annular is padded
var cornerRadius    = 4; // amount that the sectors are rounded

This allowed me to ‘play around’ with the size and shape of the annulars until I got something that was ‘about right’.

Annular representation of data - step 4
Annular spacing overlapped

 

Annular representation of data - step 3
Annular widths and spacing looking better

At this stage I also experimented with the padAngle of the annular arcs (also defined as a variable for easy tweaking), and with the stroke weight and colour, which was defined in CSS. Again, I took inspiration from GRLC’s corporate branding.

Placing dataset labels on the arcs

Now that I had the basic shape of the visualisation, the next challenge was to add dataset labels. This was again a major blocking point, and it took me a lot of tinkering to finally realise that the dataset labels would need to be svg text, sitting on paths created from separate arcs than that rendered by the d3.pie function. Without separate paths, the text wrapped around each arc segment in the annular – shown below. So, for each dataset, I created a new arc and path for the dataset label to be rendered on, and then appended a text element to the path. I’d never used this technique in svg before and it was an interesting learning experience.

Annular representation of data - step 6
Text on arcs is a dark art

Having sketched out a general approach again helped here, as with the addition of a few extra variables I was able to easily create new arcs for the dataset text to sit on. A few more variables to control the positioning of the dataset labels, and voila!

Annular representation of data - step 7
Dataset labels looking good

Adding a legend

The next challenge was to add a legend to the diagram, mostly because I’d decided that the infographic would be too busy with Library labels on each data point. This again took a bit of working through, because while d3.js has a d3.legend function for constructing legends, it’s only intended for data plotted horizontally or vertically, not 7 data sets plotted on consecutive annulars. This tutorial from Zero Viscosity and this one from Competa helped me understand that a legend is really just a group of related rectangles.

var legend = LegendLayer.selectAll("g")
    .data(color.domain())
    .enter()
    .append('g')
    .attr('x', legendPlacementX)
    .attr('y', legendPlacementY)
    .attr('class', 'legend')
    .attr('transform', function(d, i) {
        return 'translate(' + (legendPlacementX + legendWidth) + ',' + (legendPlacementY + (i * legendHeight)) + ')';
});

legend.append('rect')
    .attr('width', legendWidth)
    .attr('height', legendHeight)
    .attr('class', 'legendRect')
    .style('fill', color)
    .style('stroke', legendStrokeColor);

legend.append('text')
    .attr('x', legendWidth + legendXSpacing)
    .attr('y', legendHeight - legendYSpacing)
    .attr('class', 'legendText')
    .text(function(d) { return d; });
Annular representation of data - step 8
The legend isn’t positioned correctly

Again, the positioning took a little work, but eventually I got the legend positioned well.

Annular representation of data - step 9
The legend is finally positioned well

Responsive design and data visualisation with d3.js

One of the other key challenges with this project was attempting to have a reasonably responsive design. This appears to be incredibly hard to do with d3.js. I experimented with a number of settings to aim for a more responsive layout. Originally, the narrative text was positioned in a sidebar to the right of the image, but at different screen resolutions the CSS float rendered awkwardly, so I decided to use a one column layout instead, and this worked much better at different resolutions.

Next, I experimented with using the Javascript values innerWidth and innerHeight to help set the width and height of the svg element, and also dynamically positioned the legend. This gave a much better, while not perfect, rendering at different resolutions. It’s still a little hinkey, particularly at smaller resolutions, but is still an incremental improvement.

Thinking this through more deeply, although SVG and d3.js in general are vector-based, and therefore lend themselves well to responsive design to begin with, there are a number of elements which don’t scale well at different resolutions – such as text sizes. Unless all these elements were to be made dynamic, and likely conditional on viewport and orientation, then it’s going to be challenging indeed to produce a visualisation that’s fully responsive.

Adding tooltips

While I was reasonably pleased with the progress on the project, I felt that the visualisation needed an interactive element. I considered using some sort of arc tween to show movement between data sets, but given that historical data (say for previous years) wasn’t available, this didn’t seem to be an appropriate choice.

After getting very frustrated with the lack of built in tooltips in d3.js itself, I happened upon the d3.tip library. This was a beautifully written addition to d3.js, and although its original intent was for horizontal and vertical chart elements, it worked passably on annular segments.

Annular representation of data - step 10
Adding tooltips

Drawbacks in using d3.tip for circular imagery

One downside I found in using this library was the way in which it considers the positioning of the tooltip – this has some unpredictable, and visually unpleasant results when data is being represented in circular format. In particular, the way that d3.tip calculates the ‘centre’ of the object that it is applied to does not translate well to arc and circular shapes. For instance, look at how the d3.tip is applied to arc segments that are large and have only small amounts of curvature – such as the Geelong arc segment for ‘Members’. I’ve had a bit of a think about how to solve this problem, and the solution involves a more optimal approach to calculating the the ‘centre’ point of an arc segment.

This is beyond what I’m capable of with d3.js, but wanted to call this out as a future enhancement and exploration.

Adding percentage values to the tooltip with d3.nest and d3.sum

The next key challenge was to include the percentage figure, as well as Library and data value in the d3.tip. This was significantly more challenging than I had anticipated, and meant learning up on d3.nest and d3.sum functions. These tutorials from Phoebe Bright, and LearnJS were helpful, and Zan Armstrong’s tutorial on d3.format helped me get the precision formatting correct. After much experimentation, it turned out that summing the values of each dataset (in order to calculate percentage) was but a mere three lines of Javascript:

var CollectionItemCount = d3.nest()
    .rollup(function (v) { return d3.sum(v, function(d) { return d.Items})})
    .entries(CollectionData);

Concluding remarks

Data visualisation is much more challenging than I thought it would be, and the learning curve for d3.js is steep – but it’s worth it. This exercise drew on a range of technical skills, including circular trigonometry, HTML and knowledge of the DOM, CSS and Javascript, and above all the ability to ‘break a problem down’ and look at it from multiple angles (no pun intended).

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save