Evaluating TOTEM

External testing of TOTEM was launched in September 2011. It aimed to identify how users from the wider digital preservation professional community might integrate the tool in their work, and to analyse what further refinements of TOTEM are the most important for its future successful adoption. Two methods were employed to gather identical sets of data from different user groups: an online survey prepared and distributed in September 2011 through the SurveyMonkey platform, and a paper questionnaire used during the first KEEP dissemination workshop in Paris on 22-23 September 2011. Both groups of respondents had the same number of participants (seven in each group which is a good number for feedback in such a specialised domain with compact professional community) but with different backgrounds. All the participants in the Paris workshop were librarians or archivists, while those who filled in the online survey were predominantly researchers and data managers. Both groups confirmed that TOTEM is a useful tool, which they anticipated would be used within their organisations on a monthly basis. The participants foresaw a wide range of possible uses of TOTEM ranging from strategic tasks such as preservation planning, evaluating risks or documenting preservation environments, to very pragmatic tasks such as finding tools for working with files. The feedback from the group having had hands-on experience (the one including respondents from the workshop held in Paris) was more positive when evaluating the features of TOTEM, its ease of use, and the quality of the search results. The external testing gathered a rich set of comments which will be used to guide future development of this and other KEEP tools. This chapter provides the methodological framework for the user study and findings the initial studies.