J4992 Convergence Capstone

1

Monday, May 7, 2012

J4992 Convergence Capstone:

Massive Scale

Online Conversations

Mentor: Ryanne Dolan

Leanne Butkovic

Brent Davidson

Nicole Thompson

Executive Summary

For our capstone project, we explored the possibilities massive-scale online conversations have to offer the world of journalism. We worked with MU computer scientist Ryanne Dolan, who developed a back end system that could allow thousands of Internet users to be commenting simultaneously on a website in real-time without the system crashing. It also allows for the automatic filtering of comments to scale down the potentially massive size of the conversation, and detailed analytics based on these subconversations. Our team’s task was to research ways in which this technology could be useful in journalism and design a user interface for a website using this technology.

We conducted background research and surveys to determine the current state of online commenting and its potential in the future. We found that, while the majority of those surveyed read many news articles each week, the vast majority never comment. We found that many news editors believe a primary reason readers are discouraged from commenting is that they can’t figure out how, and readers say they do not comment because the process of signing in and creating a profile takes too long.

Based on these findings, and additional research and collaboration, we developed a prototype news website that incorporated Ryanne’s system. Features of our system included comment placement inline with the article in a fixed position along the article’s right side to encourage reader interaction with the article: The ability to “pin” paragraphs of text in the article, marking them as interesting or as something you’d like to comment on; easy, one-step registration by entering a name or pseudonym when you submit your first comment; separate tabs for all the comments, “my pins” (comments and parts of the article the reader finds interesting) and “hot topics” (most-pinned parts of the article); ability to look back at what a user has pinned by clicking on their profile picture, in order to see what they are interested in and have a more informed discussion with them; and the option to see comment threads, by clicking “replies” at the bottom of each comment.

Through user testing, we found many of these features would encourage readers to comment more often. However, there is still work to be done by future convergence students to get this system to a place where it can be implemented on actual news sites. One feature of this system that has the most potential benefit for journalism is the ability to mine detailed data from the system. A massive-scale test, using thousands of users, would allow the full exploration of this system’s data-mining potential. Through research and interviews, the need to connect such a system to a user profile appeared necessary. We have designed a static prototype for a user dashboard, or home base, but making it functional has yet to be accomplished. Also, the world of online journalism stretches well beyond text articles. This system could be implemented into video stories or live-streamed events as well.

Ultimately, this system has the potential to develop online “communities of practice” – new networks of people with similar thoughts or interests – and this has huge potential benefits for the world of journalism. All editors need to do is listen. Listen to the ideas being evoked by readers throughcomments on their organization’s website. This can build sources and inspiration for stories. It can aid in giving your readership the news it needs and craves.

Background

In order to create an innovative and useful user interface for our commenting system, we had to first gain a solid understanding of the current state of commenting.

The system we are developing would be a separate plug-in news sites would install into their sites. We researched what other commenting plug-ins existed. One popular option was Disqus, which is used by large news organizations including CNN, Time and Fox News. It integrates social media, such as Facebook and Twitter, one of its main draws. Pluck, Echo and LiveFyre are similar existing systems. Some flaws we identified in these systems is that they are limited to a linear comment stream and do not allow for very large-scale conversations.

We looked at examples of how other sites use their own homegrown commenting systems. One positive example we found was The New York Times. The site filters comments based on “reader picks” and “NYT picks.” This allows readers to read the most relevant, insightful or popular comments without having to sift through everything else, but they have the option to sift through everything if they want to. The reader is given more control to have the commenting experience they want.

One example we didn’t like was CBS. At the bottom of an article, there is simply a bolded line of text telling how many comments there are on the article. To see the comments, the reader must click on the line of text. This isn’t necessarily intuitive, and is a barrier to commenting. It also disassociates the comments from the article.

Another example of a commenting system that isn’t intuitive is the Wisconsin State Journal. A line of text at the bottom of an article prompts the reader to sign in or create a profile to comment. There are links that allow you to do this in the line of text, but it is not apparent that they are links until you roll over them. Also, requiring a reader to register and login to comment discourages them from commenting.

Through examining specific cases, we developed a list of different benefits derived from a comment. A comment is:

• Useful content – Just like the article itself, the comments are useful content for the reader and the editor.
• Additive – Comments add content, value and insight to the website that was not there before at no cost to the news organization.
• Measurable – Editors can glean analytics from comments and commenters, learning more about their readership and what news they are the most interested in. This is also beneficial to show advertisers.
• A relationship – Commenting establishes a relationship between both the users and other users, and the news organization and the users. It’s a way to interact and discuss. The Missourian likes to ask its editors “How can we make the Missourian Columbia’s coffee shop?”. Establishing a these relationships through commenting can do that.

Through examining specific cases, we also developed a list of the best ways to capture a comment. A commenting system should be:

• Easy – The actions necessary for a reader to enter a comment or complete any further interactions in a commenting system must be intuitive and easy to do. The reader should be allowed to invest most of their energy into coming up with an insightful comment, rather than figuring out how to input that comment. If the process of entering a comment is not challenging, the reader may be provoked to comment more.
• Fast – Today’s readers have short attention spans. If you put too many barriers in the way of commenting, the reader will get bored and not take the time to full interact with the news. One example of this is a lengthy registration process.
• Compelling – Readers should be given the means to get the most interesting stuff out of the article, and be able to find the comments they find the most interesting. The interface should allow them to interact with the article and other commenters in various ways, branch out to social media, and be more than a standard, linear, unfiltered system.

We also looked at research that has been done on commenting, community engagement and other fields related to our project. We looked into Joy Mayer’s research published on the Reynolds Journalism Institute’s blog on January 12, 2012 on audience engagement. Part of her research involved analytics reporting. One key element of our system is the potential to mine detailed analytics. Mayer’s report stated that their enhanced focus on analytics led her team to know more about the Columbia Missourian’s (the newspaper for which the team was working) audience than they had previously, specifically their local audience. The team tracked the effect of social media on traffic and began projecting real-time data on the wall of the newsroom, which raised the general awareness of the room. The newfound knowledge has led this team to work more closely with the business side of the paper to help them understand who they serve and at times use analytics to make decisions about what information users are looking for and adjust coverage based on it.

After reading her research, we conducted an interview with Mayer. She told us that, as far as analytics go, she would really like to be able to filter geographically and semantically (see trending topics, words, etc.). She’d like to be able to see on which topics people are having in depth, intelligent conversations, and on which topics people just banter, argue, and have low-level conversations. She was excited about the potential for this because she sees it as a way to tell what people are really passionate about.

When asked how she felt about chat room-like commenting on a news site, Mayer said the idea had been thrown around the Missourian before, and she likes it. She suggested, for example, that there could be a chat room that focuses on what the city council is doing that uses our technology. She thinks that there could be tons of uses for this kind of technology. However, she thinks each person would have to be associated with a profile. It could not be anonymous commenting. Keeping it anonymous would also hinder analytics. Also, it’s just the Missourian’s policy to not have anonymous comments. She said the Columbia Daily Tribune does allow anonymous commenting, and the comments on their articles are often degraded to low-level arguments. As a potential user of our system, Mayer wants it to be attached to a profile to allow for better analytics and the ability for the Missourian to contact the person and interact with them if need be.

In a study conducted at Pennsylvania State University titled “Finding Meaning in Very-Large Scale Conversations,” researchers discuss the potential benefits of analyzing very-large scale conversations, the same type of conversations we may be able to capture with our real-time technology. The Penn State researchers state very-large scale conversations are interesting artifacts for research because “because these networked conversations typically revolve around specific topics and interests, their content can be analyzed to understand how people use, express and learn knowledge over time. Their public nature makes them accessible to researchers. And their size provides opportunities for large conversational studies that could be difficult to collect and analyze in other media (e.g. face-to-face conversations).”

One way our team envisions the analytics side of our system functioning is similar to an information graphic called “Most-tweeted moments of the State of the Union.” This graphic was shown during a news-of-the-week discussion in capstone. The graphic provides this data visualization:

With the pinning function in our system, which will be explained in further detail later, parts of an article or video could be marked as interesting or commented on. Like in this twitter diagram, editors could see which parts of an article or video were provoking the most discussion and learn more about what is important to their readers. These analytics for twitter data are typically restricted be Twitter, and hard to get your hands on. With our system, editors would have full control. In a research paper titled “A Visual Backchannel for Large-Scale Events,” researchers explore the power of visualizing Twitter data and attempt to develop a program that can do just that. They state that “microblogging communities, such as Twitter, are increasingly used as digital backchannels for timely exchange of brief comments and impressions during political speeches, sport competitions, natural disasters, and other large events. Currently, shared updates are typically displayed in the form of a simple list, making it difficult to get an overview of the fast-paced discussions asit happens in the moment and how it evolves over time.” Like this team, we would like to provide detailed visualizations of our data to give it a higher impact.

Surveys

In order to better understand the current state of commenting and what is needed for commenting’s future, out team conducted two online surveys. For the first, we surveyed web editors and community engagement editors from the RJI listserv, asking them questions about their organization’s current commenting system and what new features might be useful. 28 out of 620 of those surveyed responded, or 4.5 percent. Here are some of our most relevant findings:

For our second survey, we surveyed students in the J2150 – Multimedia Journalism class at the University of Missouri to learn more about Internet users’ current commenting habits and what would cause them to change those habits. Out of 372 students who received the survey, 10.2 percent responded. Here are some of our most relevant findings:

Development Process

This software had a three step development process that started with several rounds of individual work, which was then combed for the best ideas to combine into one concept that we would work into a prototype, and finally used user testing to make a round of refinements.

Ideas for individual work were sourced from several brainstorming sessions documenting the needs of news site users. We based this knowledge on our first-hand experiences of news sites and Ryanne’s knowledge of network structures. Ryanne noted that networks follow Metcalfe’s law, stating that the value of the network is proportional to the square of the number of nodes. From our research on online news in previous classes and Ryanne’s knowledge in the server side of this technology, we decided our goal was to make a UI that increased the number of users commenting, their frequency of commenting and provided news organizations with valuable data about interactions between users.

With the problems addressed and proposed solutions defined, each member developed their own brief concept of a new commenting system. We took these ideas through several versions for the course of two to three weeks, collaborating on universal ideas, but diligently keeping our concepts different from the others. At the end of this process we had three very different designs addressing more problems than we had originally documented. The best ideas from these concepts were combined into one, higher fidelity concept ready for prototype development.

The prototype development was the longest portion of this process. We looked at the established concept, then broke the product into chunks, divvying specific parts of the prototype to each team member. This work included designing all the graphic assets, translating our ideas into feasible work for Ryanne and evaluating our ideas as they became functional and making any changes immediately needed. The last step in prototype development was creating a home for our commenting system interface. A mock news site, The Pacific was created and our system integrated.

With a live URL to share, we began user testing to see if users could understand the new system that combines several generally accepted internet behaviors into a new environment echoing the UX of a social network but without the tenets of friends or followers. This user testing resulted in a few key insights that continue to drive changes in the software. The two strongest feedbacks were the UI did not make pinning immediately obvious, and users want to use the @ symbol to make at-replies. Since pinning is the system’s differentiating function we took this feedback seriously, making changes that increased pinning’s prominence. We added small pin icons next to ‘pinnable’ content and small counters to show the number of pins a comment or paragraph has received. We also implemented at-replies, though more testing is needed to prove the most effective way to show a user has been replied to.

Acting upon the user testing may have been the most difficult part of the process, because it is a significant research task that needs to yield very specific, actionable changes to the UI. We advise any future capstone team to base a large portion of their work on the results of user testing, while leaving some room for gut feeling since this is a rather different way to comment.

Several rounds of user testing will be necessary to create a smooth, intuitive way to use pinning as a tool to relate users to each other and increase engagement. As mentioned above, pinning is this system’s central feature. It is also used on several sites like Pintrist. However, our system does not share pinboard functionality with pintrist, so let’s define pinning with respect to this system.

What is Pinning and How Does it Work?

Pin*ningverb -Marking a comment, paragraph, portion of a video, or other consumable content as relevant to your comments and/or interests, and of enough significance to be shared with the commenting community.