Design Guidelines for Effective Recommender System Interfaces Based on a
Usability Criteria Conceptual Model: Results from a College Student Population
A. Ant Ozok, Quyin Fan, Anthony F. Norcio
Response Letter to the Referee Comments
The authors very much appreciate the Referee’s invaluable comments throughout the paper. We made sure to integrate all of these comments into our paper and believe the paper now looks significantly different than the previous version. We also believe that the overall impact of our paper has significantly increased after our revision process thanks to our Reviewer’s insightful comments and points. Below are our responses to all of the comments.
Comment:
The paper suffers from considerable problems. First, its findings are not representative since they are based on a purely student population which is not representative for users of e-commerce. The arguments for dealing with this problem are less than convincing. So the whole results need to be interpreted with regard to what student users of e-commerce sites may want.
Response:
The authors believe that this is a valid point. In response, we mostly re-wrote massive sections of our paper to address the issue that the results are entirely based on a student population, but they may be applicable to the general population due to previous literature indicating the representativeness of this particular population. We changed our title to reflect this particular boundary to the title above, and this issue is addressed throughout the paper in no less than thirty different places. No less than one full page has been added addressing this issue with four references, particularly on Page 6. The issue is again addressed on Page47, in Limitations and Future Directions section. Overall, the authors acknowledge that having students as the participant group is a limitation, but the group is representative enough for the generalization of the findings to the majority of e-commerce shopper population.
Comment:
Second, the paper postulates general finding for RS. However, it looks rather obvious that different products will need different presentation/information in recommender systems.
Therefore, I think the results need to be discussed more specifically. The list of bullet points in chapter 4 needs much more contextualization with regard to the products, user population etc.
Response:
Once again, the authors very much appreciate this thoughtful comment. This issue is addressed on Pages 26, 27 and 35, and a new paper (Im and Hars, 2007) is references which looked at this “one size does not fit all” issue. The issue is also addressed in the Limitations and Future Directions section, on Page 47. Most specifically, the bullet points summarizing the main findings in Chapter 4 have been revised accordingly, honoring the request of the Referee regarding the discussion of these findings more in context (Pages 36, 38, 39, 40, and 42). We believe that with these changes, the product specifity issue is adequately addressed and accounted for in the new version of the paper.
Comment:
Third, the state of the art in RS is not well enough represented. There is much more work on the design of RS than it is represented in the references. Please have a look at the CHI or interact conferences and the related journals of the HCI community.
Response:
We did exactly what the Referee advised us to do in order to ensure the state-of-the-art aspect of our paper concerning the latest happenings in HCI issues involving Recommender Systems. As a result we added a total of fourteen new references to our paper, the majority of which is presented in the re-titled Chapter 2.1.3 (Pages 9-10). New references are added in other places of the paper as well, as well as a mention of a brand-new conference (started in 2007) on Recommender Systems. The very up-to-date user issues of recommender systems are believed to be addressed adequately this way.
We believe that with addressing the above issues thoroughly, the major concerns the Reviewer had are greatly taken care of. But if there are any more concerns, the authors will be happy to promptly and positively respond to them as well.
Comment:
Some minor problems:
Given that RS seem to work perfectly in practise (see Amazon), what is the actual motivation for the authors work? Numerous design guidelines for web applications exist. Why are RS worth deeper consideration? The authors state that “On the other hand, recommender systems cannot be viewed as essential Web applications as e-commerce pages would still run without the presence of such systems” (page 9). The authors should give better arguments at this point.
Response:
The authors very much agree with this comment. On Pages 3 and 4, further explanation is presented on the motivation of the current study, indicating that recommender systems are concluded to be independent enough entities to be studied separately. The statement on Page 9 indicated by the Referee is removed. The issue of the literature being lacking in usability studies concerning recommender systems is again addressed on Page 12.
Comment:
It does not initially become clear, which perspective the authors take up: The vendors or the consumers’ perspective. So when are RS successful? If the sales increase on the vendors’ side or if the felt satisfaction increases on the consumers’ side.
Response:
This issue is addressed on Page 6, indicating that the results can be useful to both consumers and vendors. Detailed explanation is given on the potential benefit of this study. Additionally, the metrics for success in recommender systems are discussed on Page 4.
Comment:
Why do the authors distinguish between RS and collaborative filtering (page 2)? It seems from the description that your results can be applied to CF as well. If not, this should be pointed out more clearly.
Response:
The point is well taken and almost all of the references to collaborative filtering are removed. An explanation is given on the possibility of the current study results being applicable to collaborative filtering. The changes are on Pages 2 (Objective, Motivation and Significance) and 44 (as part of Limitations).
Comment:
Section 2 contains paragraphs that are difficult to understand (see end of 2.1.1). In section
2.1.3 the authors state that “Interestingly, they indicated that users like feedback when they are presented on an interactive interface” (page 8). Why is this interesting? Section 2.2 needs some restructuring in order to become more comprehensible. The authors seem to introduce different models and levels of usability criteria. How are these interrelated? How do they affect your study? The final paragraph in section 2.2 is difficult to understand.
Response:
In response to this comment, the entire Chapter 2, especially Section 2.2, is re-structured and massively re-written. This can be observed in the changes on Pages 7-16. We made sure to eliminate all redundant and difficult to understand sentences, and the sentence in question on Page 8 has been modified to address the Referee’s concern. We believe the paper became much easier to understand with these changes we conducted.
Comment:
Considerable parts of the results don’t seem to be surprising (e.g. recommendations should not require too much screen space or should be displayed only if space permitted or recommendations should be relevant for the user). Those parts appear somewhat lengthy while other results are more surprising/interesting (especially those of correlation analysis and the guidelines). The authors should give more attention to the ‘interesting and surprising’ aspects and contextualize them.
Response:
The authors very much appreciate this comment. To address it, we gave the entire paper several reads to make it very much to the point, getting rid of all of the lengthy sections. Additionally, the more interesting and surprising findings are discussed much more thoroughly in the new version of our paper. Particularly Pages 39, 40 and 41 discuss the correlations and considerably more interesting results much more in detail now. Some parts of final recommendations are also adjusted accordingly (on Page 42).
Overall, the authors believe that 100% of the comments by our Referee have been addressed. We would like to express our sincere gratitude to our Referee and our Editor for all their interest and efforts in this regard and hope that this new version of our paper can be published in the prestigious BIT.
1