We came, we saw, we badged: an update on the Insignia project

Now that the official part of the Insignia project has wrapped up, we thought it would be a good time for an update on how our badges pilot project went. Looking back on the past year, it’s been an interesting ride for our project team. Here are some of our results and lessons learned:

The process for implementing badges was more complex than we could have anticipated.

As we mentioned in the paper we presented at the ascilite conference, our greatest challenge was that our institution’s policy for trialling new technologies was too onerous and time-consuming for small, short-term pilot programs. In most cases, small pilots of new technologies are forced to go “underground” and trial things without official support (which doesn’t benefit the larger university community). The need for a robust and stable learning management system is not under dispute, but there needs to be some mechanism to help foster limited trials without having to change the entire enterprise system.

We found significant interest in badges around the ACT, Australia, and the higher education sector more broadly.

Wow! So many institutions and colleagues are genuinely excited about the potential for digital badges in their areas. Most of the fellow badgers we met were investigating badges for co-curricular or supplementary purposes (rather than including badges as part of the core curriculum) or as part of a “gamified” experience of a degree program. We were also contact by students who were keen to set up their own badging system to credentialise participation in student-run programs. And all this within only the limited scope of this project! It would be fantastic to see ongoing discussions of inter-operable and widely sharable badges moving between these institutions in the future.

As a new and still immature technology, the students who received the badges were not quite sure what they represented or how to utilise them.

After receiving badges through the Insignia project, several students commented that they were unsure what the value of the badge was, or how to use it. This was likely amplified by the use of a third-party service. In our survey one student commented: ” I probably won’t sign up for this badge because I’ve no idea what Credly is nor of its formal relationship with the ANU.” This is a fair point! Our analysis of the badge take-up confirms this: only 17% of the students eligible for the badges chose to receive one.

Without the reputation of the university behind it, badges lose a lot of their value (see more on this below). The network economy of badges, as it were, is still too immature. Comprehensive education is needed around the concept of badges in order to ensure that students understand what the badges are for, and also how to display them appropriately for employers and others to view them.

To have a truly effective badging system, you must have an associated portfolio or backpack system to display them.

Being able to display your badges in a coherent, curated way is also key to their usefulness. Without an official portfolio or backpack space, endorsed and supported by the university, the badges were shared on places like Facebook and Twitter. While these social media spaces might be partially used for academic purposes, they are not primarily a portfolio or professional space. An ANU-branded portfolio would legitimise any badges shared within it.

Institutional reputation and credibility are key to badges holding value to students and employers.

Badges are semiotically dense: they need to communicate an extraordinary amount of information in a small, pixelated image. We discused this in the previous post on badge design: it matters a great deal who gave a credential, and this needed to be represented in how it looks as a future employer needs to be able to tell who granted it from a glance. Many of our focus group participants had concerns about the perceived lack of trust in digital credentials, and an “official” looking badge might be one way to address this. For intellectual property reasons the ANU crest could not be included in the badge design, but future badges should likely follow the ANU brand visuals more closely in order to carry the reputational credibility.

Stay posted to the blog for future updates on how badges can be used for libraries, and for our final white paper on implementing digital badges in higher education.

Safely home from #ascilite2014

Emily and I had quite the eventful trip to Ascilite this year, with many delays, cancelled flights, and bus rides to get to Dunedin on time. But we made it safe and sound, and were very pleased to present on our digital badges project.

I’ve collected many of the tweets and photos from our presentation into a Storify, which you can access here:

https://storify.com/katiedigc/insignia-project-at-ascilite2014 

A common theme of our discussions with another conference participants was the shared difficulties in testing new technologies or innovative approaches to teaching in the context of a complex institution like a university. The many committees, funding requirements, staffing concerns, legal issues, and so on are a key part of quality assurance and oversight, but are cumbersome to deal with for short projects like ours. In our case, we found the process of implementing badges more in need of investigation than the effectiveness of the badges themselves.

We were all very honoured to receive the conference award for Best Concise Paper.

Congrats @katiedigc @tinselturtle @thesiswhisperer on best concise paper #ascilite2014 #anu #badges http://t.co/Jxertvo9Id—
Ricky Vuckovic (@RickyVuckovic) November 25, 2014

You can access our paper in full as part of the ascilite proceedings, available here.

Other Badge Projects at Ascilite

It was interesting to see other badges projects presented as well, such as those used in the Carpe Diem MOOC. In this context the badges were attached to each stage of the MOOC, with the final badge as the completion certificate. Kulari Lokuge Dona, who was presenting as part of the Carpe Diem team, discussed how the badges did function to provoke competitiveness in the MOOC participants: while the number of participants in the MOOC did decline throughout its delivery, the completion rate was still much higher that most MOOCs, with around 17% of participants collecting the final badge.

https://twitter.com/tinselturtle/status/536651679219728387

Whether or not this can be attributed directly to badges is, of course, not certain but it is certainly an interesting point. More information can be found in their paper, available here.

Mark Northover and William Liu from AUT also presented on using badges to mark sustainable  and environmentally friendly teaching practices.

#openbadges at AUT w @kiwimarc <- come share at @OB_ANZ pls @kiwimarc! #ascilite2014 #obanz pic.twitter.com/x5u4ioJf2t

— 👩🏻‍💻Joyce Seitzinger Threads: @catspyjamasnz (@catspyjamasnz) November 25, 2014

In this case, the academics completed a self-assessment to rate how sustainble their courses were according to a series of criteria, and then were able to display their badge to students and colleagues. Their paper is available here.

And finally, Elliott, Clayton, and Iwata explored the use of badges for motivating students. Unfortunately I was in another session while this paper was presented, but it investigates how badges were used for English language development among Japanese medical professionals. For more details, you can find the full paper here.

We’re nearing the final stages of our project now, but will hopefully have a few more updates for you and some reflections in the upcoming weeks.

Ascilite2014 Presentation

Emily and I are presenting on our badge trouble at Ascilite in Dunedin today, at 3.10pm (NZ time) in the lecture theatre. Here’s a quick link to our Prezi if you’d like to follow along.

http://www.prezi.com/ow9w9hluvxrl/

You can also follow the conversation on Twitter on the hashtag #ascilite2014!

State of the union with ANU badges

As our badge team heads towards the end of the year and to the ascilite conference in Dunedin, New Zealand, we thought it was due time to give an update on our badges project thus far. At the time of writing, we have successfully delivered 37 badges to ANU research students for completing modules on research integrity and library research skills. There’s a new page describing our badges available here. As it turns out, the design and delivery of the badges were the  most straightforward part of the process. But as badges are intended to be moveable and visible externally, they cause a lot of problems for existing university systems and processes. Badges are still in their infancy, and a full institutional implementation is a time-consuming and expensive proposition involving multiple governance groups, committees, legal teams, and so on. Communication with students and stakeholders about the function and role of badges is another key area which will need to be addressed if badges are to be used in larger projects. After all, if you don’t know what it means or what to do with it, the badge loses much of its value.

I’m sure we will have many productive discussions about badges at ascilite next week, but for any readers out there who are working on their own badges projects, we’d love to hear how the implementation side of your project is going. Leave us a note in the comments!

 

 

 

Do looks matter? Discussing badge design with ANU students

Hi! I’m Katie Freund, and I’m a new staff member at ANU also working on the badges project with Inger. I currently work as a digital learning developer in the College of Asia and the Pacific and am responsible for the learning design and implementation of badges for our project. I’d like to share some of the findings from our initial focus groups with ANU students about badge design. But first, a quick update about where we are up to with our badges project.

After several months of investigating the various options for delivering badges, our team has finally set up and issued our first badges to students! As Inger discussed in a previous post, using badges in ANU’s existing learning management system (Moodle) was not as easy as it appeared. As we were faced with several months of committee meetings and user testing in order to activate the badges plugin for Moodle, our team decided we’d need to find another avenue to deliver the badges. Extensive consultation with legal and marketing teams resulted, and in the end we have signed up for Credly, a commercial service for delivering badges. After students complete the relevant activities inside Moodle, they can receive a badge via Credly for their records. As of today, we’ve given out nine badges!

As you might image, there were many discussions as to what the badge could (or should) look like. What shapes, colours, and icons could we use to represent the achievements of ANU students? How could we communicate information about different types of courses in the design of the badge? And how important is how the badge looks to the students who will be earning them?

We posed this question to a focus group of HDR and postgraduate students here at ANU, and were very surprised by some of their responses. We showed several existing badges, such as those possible using Credly’s badge builder…

 

…and those offered by the Chicago Summer of Learning.

 

These were roundly dismissed by the ANU students:

“The graphic stuff is gimmicky.”

“It looks like the stamps by little boy comes from childcare with!”

and my favourite, “We don’t want it to look like it was fun being at ANU!”

The group was adamant that it needed to include the university logo, and that the icons in the centre were too childish for their purposes. They spoke about ANU’s reputation and brand in ways that would make the marketing teams very proud: they wanted badges that would communicate the seriousness and weight of their university degrees. In fact, rather than showcasing the particular skill they had developed, the focus group participants indicated it was the ANU brand that they wanted to communicate instead.

But as most unviersity staff know, the brand and logo of the institution is something closely held, and even more closely guarded. ANU’s legal and marketing teams brought up concerns about the appropriate use of the logo when using a third-party service like Credly. In order to proceed with using the third-party service, it became apparent we would not be able to use the university crest or logo in any of our badges. So the challenge went to the graphic design team at ANU Marketing: how do you evoke the ANU brand without actually using any of the brand elements in your design, in a way that is scalable and sustainable for any future badges that might be needed?

Luckily, the team had some lovely solutions for us.

 

We have used the stars to indicate the level of the course (1-5 stars), and the third badge resembles the seal on a testamur for high-level badges of achievement.  The scroll in the middle badge evokes the ANU logo without actually including it. The icons can be customised to suit the type of course, and the different colleges can use their official colours. While students were not as interested in icons for communicating the type of course, we believed it was essential to have them in order to quickly and clearly communicate what the badge is for possible employers – particularly if a student receives several badges from the ANU.

Our next step is to show these badges to the students in our pilot project for further feedback and discussion, but we’d love to hear your thoughts as well. If you are using badges at your institution, are your students as brand-conscious as those at ANU? How are your institutions dealing with issues of brand identity and use of logos on badges?

 

You can follow Katie on Twitter @katiedigc for more on badges and everything ed tech!

Are open badges just empty motivators?

I’ve been worried, since I started the INSIGNIA project, that ‘the people’ wouldn’t really get the point of open badges for researchers.

By ‘the people’ I mean PhD students and their supervisors. No one has ever tried open badges for researchers before – at least as far as I am aware. I have no idea whether they will be interested in doing the web based tasks we develop, or in earning the open badges we make.

One of my chief worries was about the ‘gamification’ model that seems to underlie the open badges idea – badges are a small reward for a certain achievement. Would this audience of researchers see this idea as, well… naff? It’s not the kind of thing that one puts in a grant application of course, but it is a legitimate worry.

Carla Casilli states that “all badges have value” and that, understood as a whole, multiple badges work can act as signifiers for identity. I’ve sympathy with this argument, but I wonder, in this particular case that there are other, strong forces of identity formation at work. Researchers are understood as researchers through their outputs (in most cases journal papers), which gain meaning within a disciplinary community.

A journal paper is the ultimate “badge” for a researcher – it’s a concrete demonstration of skills and abilities.

My other worry is the age of the intended audience for these badges. At the moment, in Australia, the average age on commencement of the PhD is 34 years old. The supervisors are older still: there are more academics over the age of 50 than under. I wonder how many PhD students and academics play computer games and have experienced this kind of reward system?

It may well be that no one in our intended audience really gets it, but that’s the point of research isn’t it? To ferret out and explore the unknowns.

I’m happy to report I am losing less sleep over badge acceptance since we ran a Thesis Bootcamp here at ANU. For those of you not familiar with this concept, the thesis bootcamp concept was developed by Peta Freestone and Liam O’Connell of the Melbourne university graduate school. It’s a simple concept really – gather together people who are all at or very near completion of their PhD. Feed them, console them and give them space and time to just write the thesis. Liam kindly agreed to come to ANU and help me run my first one and learn from him how the University of Melbourne system works.

Part of this system – and the fiendishly clever bit really – is the rewards: a series of lego shaped squeezy blocks which are awarded for every 5000 words written. The blocks come in four colours: green for 5000 words, blue for 10,000 words, red for 15,000 words and gold for 20,000 words. When Liam introduced the idea to our bootcampers there was a palpable air of scepticism in the room. I’m sure the PhD students were thinking to themselves: “this is kind of naff” – since I was already thinking of the blocks as ‘analogue badges’ my heart sank.

Liam reassured me that “this always happens” and we carried on. It took until mid morning on the Saturday for one of our bootcampers, Natalie, to declare she had reached the 5000 word target. The bootcampers looked on with amused bewilderment as  we made a big fuss of the achievement and wrote Natalie’s name on the white board under the “5000 club”.

There was no visible sign that the blocks were working as a motivator until after the third one ‘dropped’. The bootcampers no longer looked on with amusement, they clapped and cheered their colleagues’ achievement. I heard more than one of them mutter “I must get to the first block!”.

Green blocks dropped faster and faster as people redoubled their efforts. I guess some of them felt left out and some of them were getting competitive. Whatever the psychology, before too long all the green blocks were awarded and a lot of our bootcampers had progressed much further than that. Two bootcampers got gold blocks and four of them declared they had finished all the writing they needed to do to complete the thesis. After that comes editing of course, but having the words to edit is half the battle.

In the kitchen one of the bootcampers who had just got his gold block remarked: “If only I’d known all it took was an empty motivator Inger, I would have been finished long ago!”. It was funny because it was true – the squeezy lego block had absolutely no intrinsic value – it only cost me 50 cents. But deployed as a motivator in a social setting it had amazing power.

I’ve been thinking about the lessons learned from Bootcamp for about a month now and here are two reasons why I think the blocks worked:

1) The aesthetics of the blocks – and the way they related to each other – was important. Liam pointed out that the blocks clip together pleasingly to give a visible sense of progress (we called it “the word wall”). The branding on them worked too – many bootcampers reported displaying it proudly in their office. The lego blocks became a tangible demonstration of membership, much like a branded hoodie or tshirt with your university logo on it. Most people crave some form of membership.

2) Seeing others get the blocks was important. The social dynamics we worked to create were crucial in this instance.  The block – or an open badge for that matter – is a token whose meaning and significance is socially made. I think the bootcampers initially felt a bit embarrassed cheering when someone got a block. Like me, no doubt, they had learned to be ‘cool’ about displays of enthusiasm by teachers in high school. But the initial scepticism died away after a couple of their colleagues got their prize and were so obviously pleased. The competitive spirit, or FOMO, started to kick in as they watched. Once they started joining in the cheering their desire to get a block themselves seemed to increase too.

I’m interested in what others think. What other reasons might there be for the blocks working in this way? What tactics and strategies can we use to mobilise the desire for open badges in our audience? What are the barriers?

A null result is still a result. Right?

In the Unjaded Reflections blog post “Why should you create an open badge system”, Jade Forester has three warnings for people designing open badge systems:

1. Don’t try to make your badge system grow too much, too fast;
2. Ensure you have the technical and human resources you need to fully support your badge system;
3. Make sure there’s a need for badges

Starting INSIGNIA as a small scale system, with available resources, certainly sounded like a sensible approach, especially given this is only a seed project. If the need for the badge is clear, it’s likely to help with acceptance of open badges in the research community. Given these constraints, I seemed to have a perfect starting point: ANU’s compulsory, yet non credit bearing, research integrity course.

This course is a problem I had inherited from my predecessor, who built it in collaboration with the community, but without a good assessment regime or ongoing maintenance strategy. The course attempts to explain, mostly through case studies, how complex research integrity issues, such as authorship, plagiarism, informed consent etc, are in practice. The material was great, but I’ve been working for nearly a year to move the course from his bespoke online platform into our ANU learning management system (on Moodle) and to introduce a quiz as the final assessment activity (yes – this seemingly straightforward task took nearly a year. Committees eh?).

Taking Jade Forester’s advice on board, I thought issuing a badge for the successful completion of the final quiz would be a good first step towards an INSIGNIA series of badges for research education. I needed a way to provide students with evidence that they had completed the course and I had available resources. The research integrity course content was already there in Moodle 2.5, which I know has the capacity to create and issue badges. Red Dog Learning, who had been helping me to build the course, even offered to make the badge for me as the last part of their contract.

The problem, I quickly discovered, was that people and technical resources are not always enough. The barriers are more likely to be hidden and, just like a coral reef just under your bow, you sometimes don’t know they are there until you run aground.

Last week I happened to wander into the lunch room at the same time as the ANU online team, who are the people responsible for looking after ANU’s implementation of Moodle. Over my burrito I asked the team if our Moodle had the badges option turned on because I couldn’t see it anywhere when I was editing my course. The team told me they had not turned it on because no one had asked for it.

I told them I needed it and politely asked if they could turn it on.

They hesitated. “Most people don’t have a good reason for using open badges” said one, cautiously. I suspected this was a gentle way of saying “no” so I pounced on it.

“Well I have a whole pedagogical model” I replied. Then I explained the idea of open badges providing a ‘respectful pedagogy’ for research students, who may be well versed in the material within a course and just needed to demonstrate competency – my research integrity course being a good case in point.

The ANU online team nodded and looked thoughtful.

“We’ve been reluctant to turn it on because anyone could offer a badge for anything” another member of the team said.

I replied that was rather the point of open badges, but I could see the problem with it in a university context. Anything with the ANU brand on it, and no quality control mechanism behind it, was challenging to the credentialing authority of our institution. Poorly designed badges, like poorly designed courses, have the potential to reflect badly on us and might even undermine our credibility as an education provider.

I explained I was happy to have this conversation because my research was designed to elicit just these issues and write about them. But I couldn’t get very far in my project if I couldn’t actually make any badges available. Perhaps they turn on the badge making capacity, but just for me. I cheekily promised to be good.

They told me that turning on the capacity for making badges would have to pass through their stakeholder consultation mechanisms. I’ve been in the university long enough to spot the possibility of slow death by committee when I see it, so I offered to write a memo for the committee and the DVC A – to explain the project and the need for badges.

Still they hesitated. I couldn’t understand why.

As it turned out, the ANU online were worried about the possible privacy implications of open badges. The idea of a badge is that it can be easily moved ‘out’ of the Moodle platform and displayed in other places, like the Mozilla backpack. If the badge ‘carried’ data about the student with it, like an email address, this would be a breach of privacy legislation and the insitution would be subject to up to $300,000 in fines. Apparently this was complicated by the fact that ANU is a federal university (actually the only federal university, since all the others are run by state governments).

I started to get a head ache. “Would an open badge project off the ground actually require an act of Parliament?” I wondered aloud. Then I laughed, you know – in horror. My project was melting around my ears.

ANU online team looked at me sympathetically. They didn’t want to stifle innovation they explained, but they had Responsibilities. I believed them, but had to ask the next obvious question: “can we put in a check box that students can tick so they are giving consent for their private information to be shared in the badge?” Well, that probably contravened the legislation as well they explained. I must have looked despondant at that point because the team did, bless their hearts, tell me they would explore whether Moodle could selectively turn off student information so that the badge could be ‘clean’.

This is, I suspect, where the road gets bumpy… A null result is still a result. Right?

Oh well. At least I managed to use Makebadges to do the graphics for the badge. If you are interested in designing your own badges, I’ve found some good sites which I have started to compile into a ‘make your own badges!” page. I’ll be interested in your comments – has anyone else had a similar experience?

This project is alive… at least I think so.

The INSIGNIA project got funded! I’ll be using this blog to record progress on the project.

Thanks to the Office for Learning and Teaching for providing ANU with seed funding to explore the use of open badges in research education. The project will be a collaboration between the research training team, the College of Asia and the Pacific and the ANU library, led by myself.

You’re probably wondering what the research project is all about, so here’s a brief outline:

In Australia research students need only write a long thesis or dissertation, most do not do any formal course work with their degree. People now enter research degree study at all ages and stages, and from many walks of life. No longer can we assume everyone starts with the same knowledge.

There is a growing need to provide research students with more formal training in areas like research integrity, research methods and digital literacy. But rather than sit through long courses which might not suit their needs, research students need ways to demonstrate they have competence, or find out what they still need to know. This project aims to explore alternative ways of providing this through open badges.

As you will know, in a conventional course you have regular delivery of content (in the form of lectures, tutorials or modules), activities and then assessment. Courses are linear through time, with clear roles for teachers and students. By contrast, open badges suggest an alternative framework for learning, one that is dynamic and complex.

An open badge is a piece of digital information that is issued for recognition of a skill or ability. One, relatively conventional, way to use open badges is to issue them for completion of an assessment task or activity. The key player in the field of open badging is Mozilla who are authoring a set of standards and tools to facilitate the design and delivery. Their forthcoming Badgekit will provide crucial tools for this project. Once it is issued the badge may be displayed by the learner on digital platforms like blogs or in Mozilla’s Open Badges Backpack.

There are many ways to think about open badges, but the first type we plan to explore is a ‘recognition of learning’ approach, which treats open badges as similar to achievements in computer gaming. To earn a badge learners must complete a challenge which demonstrates their knowledge. The course content to support a badge activity might be provided, or learners may customise their own learning from a range of course materials which have been curated by a teacher or a community. Or a learner might just already know how to do the task.

There are three main research questions which we will try to answer with this work:

• Are open badges useful for research education?
• Will students (and supervisors) want to use them?
• What are some of the barriers to implementing open badges within conventional university settings?

That’s probably enough for a start… but I’d like to take the opportunity here to acknowledge the work of Dr Lyndsay Agans who worked with me on the initial project proposal. Lyndsay has now left ANU and can no longer be involved with the project, but it’s fair to say this grant would not have been successful without her input. I’ll do my best to carry on alone!

if you are interested in reading more, here are three references on open badges which we appended to the original application:

• Glover, Ian. “Open Badges: A Visual Method of Recognising Achievement and Increasing Learner Motivation.” Student Engagement and Experience Journal 2, no. 1 (2013).
• Goligoski, Emily. “Motivating the Learner: Mozilla’s Open Badges Program.”Access to Knowledge: A Course Journal 4, no. 1 (2012).
• Halavais, Alexander M. C. “A Genealogy of Badges: Inherited Meaning and Monstrous Moral Hybrids.” Information, Communication, and Society 15, no. 3 (2012): 354-373.