An Interactive IR study of Evaluation Methods Comparing search behavior of older and younger searchers APPENDIX

Transkript

1 An Interactive IR study of Evaluation Methods Comparing search behavior of older and younger searchers APPENDIX Barbara Hammer-Aebi Kirstine Wilfred Christensen Master s thesis 2008 Department of Information Studies Royal School of Library and Information Science Supervisor: Birger Larsen

2 1 Questionnaires in English Start questionnaire Pre-task questionnaire Post-task Questionnaire End questionnaire Questionnaires in Danish Start spørgeskema Spørgeskema til før hver opgave Spørgeskema efter en søgeopgave Afsluttende spørgeskema Consent form Simulated work task descriptions Permutation scheme for tasks Additional tables and diagrams from results Task differences Query length Rank Task time Familiarity Search experience Queries per task Viewed results per task Letter to potential participants Church newsletter

3 1 Questionnaires in English 1.1 Start questionnaire User number: Age Education level First spoken language Occupation (current or former) Overall, how many years have you been searching for information using the Internet? How often do you search using a search engine? How often do you use Wikipedia? I generally find what I am looking for when I search on-line resources Never Once or twice a year Once or twice a month Once or twice a week One or more times a day Never Once or twice a year Once or twice a month Once or twice a week One or more times a day Strongly agree Agree Not sure Disagree Strongly disagree 3

4 How many search terms du you usually enter in the search box? How many results do you usually look at/click on, when you are shown a page of results from e.g. Google or the like? How much time do you normally use on searching for information, when you have something you want to find? 1-2 search terms 3-4 search terms 5-10 search terms More than 10 search terms Only the first result 2-4 results 5-10 results More than 10 results Less than 5 minutes 5-10 minutes minutes minutes More than 30 minutes Can you give examples of typical search engines or web pages that you use to search for information: Can you mention topics that you recently have searched for or purposes where you used a search engine to look for information? You may continue on the back of the paper. As a continuation of the previous question: Is this typical for what you generally search for? Yes Somewhat No 4

5 1.2 Pre-task questionnaire User number How familiar are you with the topic of the search task? How interesting do you find the topic of the search task? Do you think it will be difficult to find information for this task? Topic number not familiar at all not familiar somewhat familiar very familiar very uninteresting uninteresting somewhat interesting very interesting not at all not very somewhat difficult very difficult Comments 5

6 1.3 Post-task Questionnaire User number How would you rate this experience How would you rate the amount of time available to do this task? How certain are you that you completed the task correctly? How easy was it to do the task? How satisfied are you with the information you found? Topic number frustrating slightly frustrating neutral pleasing very pleasing much more needed more needed just right more than necessary a lot more than necessary very uncertain uncertain somewhat certain very certain very difficult difficult somewhat easy very easy Not at all satisfied Not satisfied somewhat satisfied very satisfied Comments 6

7 1.4 End questionnaire User number: How easy was it to use the system? How easy was it to learn to use the system? Do you think the system was good at finding information? Do you think that the list of results was well-structured? Do you think there was enough information on the list of results to be able to choose which page to visit? Was it easy to figure out why a certain page had been found? very easy easy somewhat difficult very difficult very easy easy somewhat difficult very difficult very good good somewhat bad very bad very clear clear somewhat confusing very confusing not at all enough not enough just right too much much too much very easy easy somewhat difficult very difficult 7

8 What did you like about the search system? What did you dislike about the search system? 8

9 Would you have wanted any other kind of functionalities? Do you have any general comments? 9

10 2 Questionnaires in Danish 2.1 Start spørgeskema I dette spørgeskema bedes du først om nogle almindelige oplysninger vedrørende din person. I anden afdeling følger der spørgsmål om dine tidligere erfaringer med at søge informationer på internettet. Bruger nummer Alder Uddannelse Modersmål Beskæftigelse (nuværende eller tidligere) I hvor lang tid har du brugt internettet til søgning? Angiv antal måneder eller år. Hvor ofte søger du med en søgemaskine? Hvor ofte bruger du Wikipedia? Plejer du at finde, hvad du søger efter? aldrig en eller to gange om året en eller to gange om måneden en eller to gange per uge dagligt aldrig en eller to gange om året en eller to gange om måneden en eller to gange per uge dagligt Næsten hver gang Som oftest Nogen gange Sjældent Næsten aldrig 10

11 Hvor mange søgeord indtaster du typisk i en søgeboks? Hvor mange resultater kigger/klikker du normalt på, hvis du får vist en typisk Google søgning eller lignende? Hvor lang tid bruger du normalt på at søge efter information, når du har noget du gerne vil lede efter? 1-2 søgeord 3-4 søgeord 5-10 søgeord Mere end 10 søgeord Kun det første resultat 2-4 resultater 5-10 resultater Flere end 10 resultater Under 5 minutter 5-10 minutter minutter minutter Mere end 30 minutter Kan du give eksempler på typiske søgemaskiner eller web-steder du bruger til at søge efter information Kan du nævne nogle emner, du for nyligt har søgt efter, eller nogle formål hvor du skulle søge efter information i en søgemaskine? Skriv eventuelt på bagsiden af skemaet. Som fortsættelse af forrige spørgsmål: Er det typisk for hvad du generelt søger efter? Ja Nogenlunde Nej 11

12 2.2 Spørgeskema til før hver opgave Bruger nummer Kender du noget til emnet i opgaven i forvejen? Hvor interessant synes du, at emnet i opgave er? Tror du det bliver svært at finde information om emnet? slet ikke lidt hverken eller meget rigtig meget slet ikke lidt hverken eller interessant meget interessant slet ikke lidt hverken eller svært meget svært Kommentarer 12

13 2.3 Spørgeskema efter en søgeopgave Bruger nummer Hvordan var det at søge information om dette emne? Var der tid nok? Hvor sikker er du på at have fundet det som blev efterspurgt? Hvordan var sværhedsgraden? Hvor tilfreds er du med det du fandt? meget frustrerende frustrerende hverken eller tilfredsstilende Meget tilfredsstillende meget uenig uenig ved ikke enig meget enig meget usikker usikker hverken eller sikker helt sikker meget svær svær hverken eller let meget let meget utilfreds utilfreds hverken eller tilfreds meget tilfreds Kommentarer 13

14 2.4 Afsluttende spørgeskema Bruger nummer Hvor nemt var det at søge i søgemaskinen? Hvor nemt var det at lære at bruge søgemaskinen? Synes du systemet var god til at finde oplysninger? Synes du listen med resultater var overskuelig? Synes du der var information nok i resultatlisten til at vælge hvilken side du skulle ind på? Var det nemt at finde ud af, hvorfor en bestemt side var blevet fremfundet? meget nemt nemt hverken eller svært meget svært meget nemt nemt hverken eller svært meget svært meget god god hverken eller ikke god dårlig meget overskuelig overskuelig hverken eller lidt forvirrende meget forvirrende alt for lidt for lidt lige tilpas lidt for meget alt for meget meget nemt nemt hverken eller svært meget svært 14

15 Hvad ved søgemaskinen kunne du særlig godt lide? Hvad ved søgemaskinen kunne du ikke lide? 15

16 Hvad ville du have ønsket at have mulighed for? Har du andre kommentarer? 16

17 3 Consent form This study is conducted by Kirstine Wilfred Christensen Barbara Hammer-Aebi Supervisor: Birger Larsen Institut for informationsstudier Danmarks Biblioteksskole Birketinget 6, 2300 København S. Thank you for your interest in this research study. Participation is voluntary. You may withdraw from participation at any time, and you may decline to answer any question. We will track the pages that you visit while responding to the search tasks, and the search terms you use, using computer software. It is anticipated that the online searching will take approximately two hours. We will also ask you to fill in a number of questionnaires at various stages during the study. The results of your interaction with the search system will be reported without any reference to you specifically. Your name will appear only on this sheet, and the participant schedule. No names will be attached to the logs of your interactions with the search systems. While not attaching your name to the search logs will not render you unidentifiable, only the researchers will have access to this data and for the purposes of data analysis only. All information that you provide will be treated confidentially and your identity will not be revealed in reporting the study results. I have read the explanation about this study. I have been given the opportunity to discuss it and my questions have been answered to my satisfaction. I hereby consent to take part in this study and have my search interactions tracked by computer software. However, I realize that my participation is voluntary and that I am free to withdraw from the study at any time. Name Signature 17

18 4 Simulated work task descriptions Task no Task description Type Structure 1 Last year you met some Brazilian students. Speaking with them about their country, you became interested in visiting Brazil. However, as Brazil is a very large country, beyond all the information that they have already given you, you would like to search for further information about the cities and their tourist spots and thus, decide which cities you should or should not visit. Decision making Hierarchical Dansk version Sidste år mødte du nogle brasilianske studerende. I dine samtaler med dem om deres land, blev du interesseret i at besøge Brasilien. Eftersom Brasilien er et meget stort land, ønsker du at søge yderligere information om brasilianske byer og deres turistattraktioner for at kunne vælge ud hvilke du skal besøge. 2 Your friends who have an interest in art have been debating the French Impressionism exhibit at a local art gallery. One claims that Renoir is the best impressionist ever, while the other argues for another. You decide to do some research first so you can enter the debate. You consider Degas, Monet and Renoir to construct an argument for the one that best represents the spirit of the impressionist movement. Find some articles to help you choose between the painters. Decision making Hierarchical Dansk version Dine kunstinteresserede venner har diskuteret impressionisme udstillingen i det lokale galleri. En af dem mener, at Renoir er den bedste impressionist nogensinde, mens den anden synes, at der er en anden impressionist er bedre. Du beslutter dig for at søge oplysninger, så du kan bidrage til diskussionen. Du tænker på at finde ud af om Degas, Monet eller Renoir er den bedste repræsentant for den impressionistiske bevægelse. Find nogle artikler, som kan hjælpe 18

19 dig til at vælge mellem de forskellige malere. 3 As a tourist in Paris, you have time to make a single day-trip outside the city to see one of the attractions in the region. Your friend would prefer to stay in Paris, but you are trying to decide between visiting the cathedral in Chartres or the palace in Versailles, since you have heard that both are spectacular. What information will you use to make an informed decision and convince your friend to join you? You should consider the history and architecture, the distance and different options for travelling there. Decision making Parallel Dansk version På en tur til Paris afsætter du en dag til at besøge en seværdighed udenfor byen. Din rejsekammerat synes I burde blive i byen, men du vil gerne vælge mellem at besøge katedralen i Chartres eller slottet i Versailles, da du har hørt godt om begge steder. Hvilken information vil du bruge for at beslutte dig og få din rejsekammerat til at indvillige i dit forslag? Du bør tage hensyn til både historien, arkitekturen, afstanden og forskellige muligheder for at komme derhen. 4 As a member of a local environmental group who is starting a campaign to save a large local nature reserve, you want to find some information about the impact of removing the trees (logging) for the local pulp and paper industry and mining the coal that lies beneath it. Your group has had a major discussion about whether logging or mining is more ecologically devastating. To add to the debate, you do your own research to determine which side you will support. Decision making Parallel Dansk version Du er medlem af en forening som kæmper for miljøforhold i din region. Gruppen vil gerne bevare et stort naturområde, frem for at det skal ryddes så træet kan bruges til papirsindustrien i området og den kul der i undergrunden kan 19

20 udvindes. Foreningen har haft en større diskussion om skovhugst eller minedrift har værst indflydelse på miljøet. Du vil gerne finde informationer om dette for at kunne tage stilling i debatten. 5 A friend has just sent an from an Internet café in the southern USA where she is on a hiking trip. She tells you that she has just stepped into an anthill of small red ants and has a large number of painful bites on her leg. She wants to know what species of ants they are likely to be, how dangerous they are and what she can do about the bites. What will you tell her? Fact finding Hierarchical Dansk version Du har modtaget en fra en ven, sendt fra en internetcafé i det sydlige USA hvor hun er på friluftstur. Hun skriver, at hun er trådt i en myretue med små røde myrer og har nu en masse små røde bid på benet, som gør meget ondt. Hun vil gerne vide hvilken art myre, der kunne være tale om, hvor farlige de er, og om man kan gøre noget ved biddene. Hvad vil du svare hende? 6 You enjoy eating mushrooms, especially chanterelles, and a friend who is an amateur mushroom picker indicates that he has found a good source, and invites you along. He warns you that chanterelles can be confused with a deadly species for which there is no known antidote. You decide that you must know what you are looking for before you go mushroom picking. What species was he referring to? How can you tell the difference? Fact finding Hierarchical Dansk version Du kan godt lide at spise svampe, især kantareller, og en ven som samler svampe som hobby fortæller dig, han har fundet et godt sted med mange kantareller. Han inviterer dig med derhen en dag. Han advarer om, at kantareller nemt kan forveksles med en giftig svamp som er 20

21 dødelig, og for hvilken der ikke findes nogen modgift. Du beslutter dig for at tage med, men at læse godt op om emnet inden i drager af sted. Hvilke svampearter taler han om? Hvordan kan man kende forskel på de to svampesorter? 7 As a history buff, you have heard of the Quiet Revolution, the Peaceful Revolution and the Velvet Revolution. For a skill-testing question to win a journey for two, you have been asked how they differ from the April 19th revolution Fact finding Parallel Dansk version Da du er meget interesseret i historie, kender du til The Quiet Revolution, The Peaceful Revolution og fløjlsrevolutionen. Som del af en quiz, ved hvilken man kan vinde en rejse for to, skal du finde ud af, hvordan disse adskiller sig fra April revolutionen. 8 In one of your previous Web experiences, you came across a long list of castles that covered the globe. At the time, you noted that some are called castles, while others are called fortresses, and Canada unexpectedly has castles while Denmark also has fortresses! So now you wonder: what is the difference between a fortress and a castle? So you check the Web for a clarification, and to find a good example of a castle and fortress in Canada and Denmark. Fact finding Parallel Dansk version En dag hvor du surfede på nettet blev du fascineret af en lang liste over slotte hele kloden rundt. Listen var på engelsk og du opdagede at nogen af slottene blev betegnet som castles, mens andre var fortresses. Du undrede dig over, at der i Canada kun er Castles mens der i Danmark også er fortresses. Derfor vil du nu gerne finde ud af hvad præcis forskellen er mellem de to engelske ord. Du vil gerne finde en forklaring på hvad de to ord betyder, og et eksempel af hver i henholdsvis Canada og Danmark. 9 A close friend is planning to buy a car for the first time, but is worried about fuel costs and the Information gathering Hierarchical 21

22 impact on the environment. The friend has asked for help in learning about options for vehicles that are more fuel efficient and environmentally friendly. What types of different types of engines, manufacturers and models of cars might be of interest to your friend? What would be the benefits of using such vehicles? Dansk version En god ven er ved at købe sin første bil, men er bekymret over omkostningerne til benzin og indvirkningen på miljøet. Han har bedt dig om hjælp med at finde biler som kører langt på literen og er miljøvenlige. Hvilke typer motorer og hvilke modeller biler kunne komme på tale? Hvilke fordele er der ved at bruge disse køretøjer? 10 You recently heard about the book "Fast Food Nation," and it has really influenced the way you think about your diet. You note in particular the amount and types of food additives contained in the things that you eat every day. Now you want to understand which food additives pose a risk to your physical health, and are likely to be listed on grocery store labels. Information gathering Hierarchical Dansk version Du har for nyligt hørt om bøgen Fast Food Nation, og den har sat mange tanker I gang om din kost. Du har især lagt mærke til alle de tilsætningsstoffer, der er i nogle af de ting du spiser dagligt. Nu vil du undersøge hvilke tilsætningsstoffer, der er farlige for dit helbred og som kan findes på indholdsdeklarationerne på maden. 11 Friends are planning to build a new house and have heard that using solar energy panels for heating can save a lot of money. Since they do not know anything about home heating and the issues involved, they have asked for your help. You are uncertain as well, and do some research to identify some issues that need to be considered in deciding between more conventional methods of home heating and solar Information gathering Parallel 22

23 panels. Dansk version Et vennepar skal til at bygge nyt hus og har hørt at man kan spare penge I det lange løb hvis man opsætter solpaneler. De ved ikke noget om opvarmning af huse i forvejen, så de beder dig om hjælp. Du er selv usikker på, hvilke råd du kan give dem og vil forsøge at finde ud af de vigtigste problematikker som skal tages højde for når man har valget mellem solpaneler og traditionel opvarmning af huset. 12 You just joined the citizen s advisory committee for the city of St. John s, Newfoundland. With the increase in fuel costs, the city council is contemplating supplementing its power with alternative energy. Tidal power and wind power are being discussed among your fellow committee members. As you want to be fully informed when you attend the next meeting, you research the pros and cons of each type. Information gathering Parallel Dansk version Du er lige blevet medlem af en borgergruppe i St. John s på Newfoundland. Da oliepriserne stiger overvejer byrådet at supplere sin nuværende energiforsyning med alternative energikilder. Der er tale om bølgeenergi og vindenergi. Du vil gerne være velinformeret til det næste møde, derfor undersøger du fordele og ulemper ved de to energikilder. Training task You have decided that you want to eat healthier, and you want to know a variety of grains and their nutritional benefits, and how you can cook them to improve your quality of eating. Therefore, you are interested in finding information about grains that describes their nutritional benefits, or how they are cooked, or about a dish that uses grains as a main ingredient. Find articles about grains that describe their nutritional benefits and how they are cooked, or an article about a dish that uses grains as a main 23

24 ingredient. Dansk version Efter en stribe af små virussygdomme har du besluttet dig for at spise sundere. Du ved en hel del om grøntsager, men en ven har nævnt at forskellige kornsorter også kan bidrage meget til en varieret og sund kost. Derfor vil du gerne vide mere om kornsorter, ernæringsmæssige fordele de hver især har og hvordan man tilbereder dem. Task no Task description Type Structure The original task from INEX 2006/2007 that was deselected for this study. Your community is contemplating building a bridge across a span of water measuring 1000 M in order to ease traffic congestion. There will be a presentation this evening about the type of bridge proposed for the project. To date, many types of bridges have been discussed: folding bridge, suspension bridge, retractable bridge, and bascule bridge. In order to be well informed when you attend the meeting, you need information on what type of bridge would best suit the community s needs, bearing in mind that the solution must accommodate vehicles and be sturdy enough to withstand a body of water that can be rough and frozen over with ice in winter. Decision making Hierarchical 24

25 5 Permutation scheme for tasks User number Task 1 or 2 Task 3 or 4 Task 5 or 6 Task 7 or 8 Task 9 or 10 Task 11 or 12 DM H DM P FF H FF P IG H IG P 1 - Pilot

26 6 Additional tables and diagrams from results 6.1 Task differences task type task structure task nr Avg. results pr task Avg. no of results pr search no of searches pr task users pr task search terms pr search Avg. rank pr task Dm h 1 13,4 2, ,8 9,4 Dm h 2 12,7 2, ,4 7,7 Dm p 3 7,0 1, ,7 3,8 Dm p 4 7,0 2, ,9 12,8 Ff h 5 10,1 2, ,2 Ff h 6 3,3 1, ,1 Ff p 7 12,3 2, ,2 Ff p 8 9,3 2, ,3 Ig h 9 7,8 1, ,9 Ig h 10 12,1 2, ,9 Ig p 11 13,4 2, ,0 Ig p 12 8,9 3, ,8 Table 1. Task differences 6.2 Query length Age Search experience No search terms old young medium high very high 1 20% 9% 19% 11% 12% 2 28% 38% 31% 24% 46% 3 29% 30% 26% 32% 29% 4 8% 12% 10% 16% 7% 5 6% 7% 6% 10% 3% 6 5% 2% 5% 4% 2% 7 1% 1% 2% 0% 1% 8 1% 1% 0% 1% 1% 9 2% 0% 1% 1% 0% 10 0% 0% 0% 0% 0% 11 0% 0% 0% 0% 0% 12 1% 0% 0% 0% 0% Table 2. Query length by age and search experience 11,3 3,3 12,6 9,7 7,4 9,1 16,2 10,8 26

27 No search terms hierarchical parallel old young total old young total 1 27% 11% 16% 14% 7% 9% 2 29% 25% 26% 27% 49% 42% 3 22% 32% 29% 35% 29% 31% 4 10% 16% 14% 6% 9% 8% 5 3% 9% 7% 9% 5% 6% 6 6% 3% 4% 4% 1% 2% 7 1% 2% 1% 1% 0% 0% 8 1% 2% 1% 1% 0% 1% 9 1% 0% 0% 2% 0% 1% 10 0% 0% 0% 0% 0% 0% 11 0% 0% 0% 0% 0% 0% 12 0% 0% 0% 1% 0% 0% total 100% 100% 100% 100% 100% 100% Table 3. Query length by task structure and age Searcher group Standard deviation Average old hierarchical 1,7 2,7 old parallel 1,9 3,0 old total 1,8 3,0 young hierarchical 1,5 3,2 young parallel 1,0 2,6 young total 1,3 2,8 Table 4. Average and standard deviation of query length by task structure and age Decision making Fact finding Information gathering No search terms old young total old young total old young total 1 20% 10% 13% 25% 13% 16% 17% 2% 8% 2 27% 43% 38% 26% 50% 43% 30% 16% 22% 3 30% 29% 29% 30% 22% 24% 27% 44% 37% 4 11% 11% 11% 6% 9% 8% 7% 18% 13% 5 3% 7% 6% 2% 5% 4% 11% 10% 11% 6 3% 1% 1% 8% 2% 3% 6% 5% 5% 7 2% 1% 1% 2% 0% 1% 0% 2% 1% 8 0% 0% 0% 2% 1% 1% 1% 3% 2% 9 3% 0% 1% 0% 0% 0% 1% 0% 1% 10 0% 0% 0% 0% 0% 0% 0% 0% 0% 11 0% 0% 0% 0% 0% 0% 0% 0% 0% 12 2% 0% 0% 0% 0% 0% 0% 0% 0% 100% 100% 100% 100% 100% 100% 100% 100% 100% Table 5. Query length by task type and age 27

28 6.3 Rank Age search experience rank old young medium high very high 1 13% 19% 22,5% 15,7% 16,4% 2 9% 12% 12,5% 10,7% 10,9% 3 8% 13% 12,5% 13,3% 8,8% 4 8% 7% 7,5% 7,7% 6,7% 5 6% 6% 6,3% 6,5% 4,9% 6 4% 5% 3,8% 4,1% 5,5% 7 4% 3% 7,5% 3,3% 3,3% 8 3% 5% 2,5% 3,0% 5,5% 9 3% 3% 2,5% 2,4% 3,6% 10 4% 4% 2,5% 4,7% 4,0% Table 6. Search experience and age distribution of results viewed at rank

29 hierarchical parallel rank old young Total old young Total 1 12% 20% 17% 15% 17% 16% 2 8% 12% 10% 10% 13% 12% 3 11% 12% 12% 6% 14% 11% 4 8% 5% 6% 8% 9% 8% 5 8% 4% 6% 3% 8% 6% 6 6% 7% 6% 2% 4% 3% 7 6% 4% 5% 3% 2% 3% 8 3% 4% 4% 3% 6% 4% 9 3% 3% 3% 3% 3% 3% 10 5% 4% 4% 3% 5% 4% 11 1% 2% 2% 2% 0% 1% 12 1% 2% 2% 3% 1% 2% 13 3% 1% 2% 2% 2% 2% 14 3% 2% 2% 1% 0% 1% 15 2% 2% 2% 0% 2% 1% 16 2% 1% 2% 1% 2% 2% 17 1% 2% 2% 3% 2% 2% 18 0% 2% 2% 1% 1% 1% 19 1% 0% 1% 1% 1% 1% 20 0% 1% 1% 1% 0% 0% 21 1% 1% 1% 2% 0% 1% 22 0% 0% 0% 1% 0% 1% 23 0% 1% 1% 0% 1% 1% 24 1% 0% 1% 1% 0% 1% 25 1% 0% 1% 3% 0% 1% 26 1% 0% 1% 1% 0% 0% 27 1% 0% 1% 3% 0% 1% 28 0% 0% 0% 2% 0% 1% 29 1% 1% 1% 1% 0% 1% 30 1% 0% 0% 2% 0% 1% 31 0% 0% 0% 1% 0% 1% 32 1% 0% 1% 1% 0% 0% 34 1% 1% 1% 1% 0% 1% 35 1% 0% 0% 1% 0% 0% 36 0% 0% 0% 1% 0% 1% 37 1% 0% 1% 0% 0% 0% 38 2% 0% 1% 0% 0% 0% 39 0% 0% 0% 1% 0% 1% 40 1% 0% 0% 1% 0% 1% 41 1% 0% 0% 1% 0% 0% 42 0% 0% 0% 0% 0% 0% 43 1% 0% 1% 0% 0% 0% 44 0% 0% 0% 1% 0% 1% 45 0% 0% 0% 1% 1% 1% 46 1% 0% 0% 1% 0% 0% 47 0% 0% 0% 1% 0% 0% 48 1% 0% 0% 1% 0% 1% 49 0% 0% 0% 1% 0% 0% 50 0% 0% 0% 1% 0% 0% 29

30 decision making fact finding information gathering rank old young total old young Total old young total 1 16% 27% 23% 14% 14% 14% 11% 14% 13% 2 14% 14% 14% 6% 11% 9% 7% 12% 10% 3 7% 18% 14% 14% 13% 14% 6% 8% 7% 4 8% 5% 6% 9% 9% 9% 6% 7% 7% 5 2% 3% 3% 7% 5% 6% 7% 10% 9% 6 6% 5% 5% 1% 5% 4% 3% 6% 5% 7 2% 2% 2% 6% 3% 4% 6% 5% 5% 8 5% 4% 5% 0% 5% 3% 2% 5% 4% 9 5% 2% 3% 3% 1% 1% 2% 6% 4% 10 4% 4% 4% 6% 4% 5% 2% 5% 4% 11 2% 2% 2% 0% 2% 1% 2% 1% 1% 12 2% 2% 2% 3% 2% 2% 2% 1% 1% 13 2% 1% 1% 3% 3% 3% 2% 1% 2% 14 0% 1% 1% 3% 0% 1% 2% 2% 2% 15 0% 1% 0% 0% 2% 1% 2% 4% 3% 16 0% 2% 1% 4% 2% 3% 1% 1% 1% 17 2% 1% 1% 0% 3% 2% 3% 2% 3% 18 0% 1% 1% 1% 3% 2% 0% 2% 1% 19 0% 1% 0% 1% 1% 1% 2% 1% 1% 20 1% 1% 1% 0% 1% 0% 0% 0% 0% 21 3% 1% 2% 3% 1% 1% 0% 1% 0% 22 0% 0% 0% 0% 1% 0% 2% 0% 1% 23 0% 1% 1% 0% 0% 0% 0% 2% 1% 24 2% 0% 1% 0% 1% 0% 0% 1% 0% 25 2% 0% 1% 1% 0% 0% 2% 1% 1% 26 1% 0% 0% 0% 0% 0% 2% 1% 1% 27 0% 1% 0% 3% 0% 1% 2% 0% 1% 28 1% 0% 0% 0% 0% 0% 2% 1% 1% 29 0% 0% 0% 1% 2% 2% 1% 0% 0% 30 0% 0% 0% 1% 1% 1% 2% 0% 1% 31 0% 0% 0% 3% 1% 1% 0% 1% 0% 32 3% 0% 1% 0% 0% 0% 0% 1% 0% 34 1% 0% 0% 0% 1% 1% 2% 0% 1% 35 0% 0% 0% 0% 0% 0% 2% 0% 1% 36 0% 1% 0% 0% 0% 0% 2% 0% 1% 37 1% 1% 1% 0% 0% 0% 0% 1% 0% 38 1% 0% 0% 0% 0% 0% 2% 0% 1% 39 0% 0% 0% 1% 1% 1% 1% 0% 0% 40 1% 0% 0% 0% 1% 0% 2% 0% 1% 41 1% 0% 0% 0% 0% 0% 1% 0% 0% 42 0% 1% 0% 0% 0% 0% 0% 0% 0% 43 1% 0% 0% 0% 0% 0% 1% 0% 0% 44 0% 0% 0% 0% 1% 0% 1% 0% 0% 45 0% 0% 0% 1% 1% 1% 1% 1% 1% 46 0% 0% 0% 0% 0% 0% 2% 0% 1% 47 0% 0% 0% 1% 1% 1% 0% 0% 0% 48 1% 0% 0% 1% 0% 0% 1% 0% 0% 49 0% 0% 0% 0% 1% 0% 1% 0% 0% 50 0% 0% 0% 0% 0% 0% 1% 0% 0% 30

31 An Interactive IR study of Evaluation Methods - APPENDIX 25% 20% 15% 10% hierarchical old hierarchical young parallel old parallel young 5% 0% rank Figure 1. Rank of viewed results by age and task structure 25% 20% 15% 10% decision making Fact finding Information Gathering 5% 0% Rank of viewed result Figure 2. Rank of viewed results by task type 31

32 18% 16% 14% 12% 10% 8% decision making old fact finding old information gathering old 6% 4% 2% 0% rank Figure 3. Rank of viewed result by age and task type 6.4 Task time User no Age group Task time Task type task structure 2 Old 00:14:01 decision making parallel 2 Old 00:03:10 fact finding hierarchical 2 Old 00:14:26 information gathering parallel 2 Old 00:11:35 decision making hierarchical 3 Old 00:13:44 fact finding parallel 3 Old 00:19:38 information gathering hierarchical 3 Old 00:15:10 decision making parallel 3 Old 00:21:13 fact finding hierarchical 4 Old 00:09:10 information gathering parallel 4 Old 00:25:10 decision making hierarchical 4 Old 00:10:35 fact finding parallel 4 Old 00:07:02 information gathering hierarchical 5 Old 00:19:08 decision making hierarchical 5 Old 00:23:49 fact finding parallel 5 Old 00:06:41 information gathering hierarchical 5 Old 00:02:25 decision making parallel 6 Old 00:10:24 fact finding hierarchical 6 Old 00:07:38 information gathering parallel 6 Old 00:02:56 decision making hierarchical 32

33 6 Old 00:02:24 fact finding parallel 7 Old 00:14:54 information gathering hierarchical 7 Old 00:07:11 decision making parallel 7 Old 00:08:48 fact finding hierarchical 7 Old 00:11:13 information gathering parallel 8 Young 00:09:21 decision making parallel 8 Young 00:03:21 fact finding hierarchical 8 Young 00:08:38 information gathering parallel 8 Young 00:08:35 decision making hierarchical 9 Young 00:13:30 fact finding parallel 9 Young 00:09:35 information gathering hierarchical 9 Young 00:13:33 decision making parallel 9 Young 00:07:58 fact finding hierarchical 10 Young 00:27:29 information gathering parallel 10 Young 00:24:41 decision making hierarchical 10 Young 00:23:12 fact finding parallel 10 Young 00:17:22 information gathering hierarchical 11 Young 00:07:42 decision making hierarchical 11 Young 00:12:01 fact finding parallel 11 Young 00:09:19 information gathering hierarchical 11 Young 00:10:24 decision making parallel 12 Young 00:16:14 fact finding hierarchical 12 Young 00:11:18 information gathering parallel 12 Young 00:14:58 decision making hierarchical 12 Young 00:18:32 fact finding parallel 13 Old 00:16:43 information gathering hierarchical 13 Old 00:16:09 decision making parallel 13 Old 00:10:27 fact finding hierarchical 13 Old 00:18:35 information gathering parallel 14 Young 00:27:16 decision making parallel 14 Young 00:11:41 fact finding hierarchical 14 Young 00:16:38 information gathering parallel 14 Young 00:27:04 decision making hierarchical 15 Young 00:17:51 fact finding parallel 15 Young 00:13:54 information gathering hierarchical 15 Young 00:12:52 decision making parallel 15 Young 00:14:59 fact finding hierarchical 16 Young 00:16:33 information gathering parallel 16 Young 00:16:50 decision making hierarchical 16 Young 00:05:03 fact finding parallel 16 Young 00:10:04 information gathering hierarchical 17 Old 00:29:47 decision making hierarchical 17 Old 00:20:37 fact finding parallel 17 Old 00:24:12 information gathering hierarchical 17 Old 00:15:40 decision making parallel 18 Young 00:14:31 fact finding hierarchical 18 Young 00:13:22 information gathering parallel 18 Young 00:09:11 decision making hierarchical 18 Young 00:10:36 fact finding parallel 19 Old 00:22:17 information gathering hierarchical 33

34 19 Old 00:14:06 decision making parallel 19 Old 00:07:00 fact finding hierarchical 19 Old 00:24:17 information gathering parallel 20 Old 00:12:54 fact finding parallel 20 Old 00:08:52 information gathering hierarchical 20 Old 00:02:37 decision making parallel 20 Old 00:06:25 fact finding hierarchical 21 Old 00:12:41 information gathering hierarchical 21 Old 00:11:36 decision making parallel 21 Old 00:04:23 fact finding hierarchical 21 Old 00:06:00 information gathering parallel Table 7 All task times task time old average old standard deviation young average young standard deviation total average parallel 00:12:28 00:06:16 00:14:54 00:06:10 00:13:34 hierarchical 00:13:20 00:07:49 00:13:13 00:05:58 00:13:17 Table 8 Task structure and task time average task time old average old standard deviation young average young standard deviation fact finding 00:11:08 00:06:45 00:13:02 00:05:31 00:12:03 decision making 00:13:24 00:08:05 00:15:12 00:07:16 00:14:14 information gathering 00:14:01 00:06:21 00:14:01 00:05:29 00:14:01 Table 9 Task time and task type all 6.5 Familiarity How familiar are you with the topic of the search task? task time not at all not very somewhat Familiar familiar familiar total Table 10. Familiarity with task topic and number of minutes used to complete task 34

35 6.6 Search experience Search experience medium high very high old young Table 11. Number of participants in the two age groups and their level of search experience 6.7 Queries per task Old hierarchical Old parallel Young hierarchical Young parallel average searches per task 4,2 4,3 10,0 12,0 standard deviations 3,5 2,4 5,5 6,4 Table 12 Task structure average standard deviation old decision making 4,6 2,5 old fact finding 3,6 2,8 old information gathering 4,4 3,5 young decision making 14 6,6 young fact finding 9,8 5 young information gathering 9,1 5,6 Table 13 Task type 6.8 Viewed results per task task structure average standard deviation hierarchical old 7,3 5,5 hierarchical young 13,6 7,9 hierarchical total 10,3 7,4 parallel old 6,9 5,5 parallel young 11,8 6,3 parallel total 9,2 6,3 Table 14 Viewed results per task structure and age 35

36 task type average standard deviation decision making old 6,8 4,9 decision making young 13,5 6,3 decision making total 9,9 6,4 fact finding old 5,8 4,7 fact finding young 10,9 7,1 fact finding total 8,5 6,5 information gathering old 8,3 6,5 information gathering young 14 8,1 information gathering total 10,7 7,6 Grand Total 9,71 6,9 Table 15. Viewed results per task type and age 36

37 7 Letter to potential participants København den 14. februar 2008 Undersøgelse af søgemaskiner Vi er to specialestuderende ved Danmarks Biblioteksskole som i de kommende måneder vil undersøge ældres brug af søgemaskiner. Vi vil spørge, om du vil hjælpe os ved at deltage i vores brugerundersøgelse. Vi har lagt mærke til, at undersøgelser indenfor dette område hovedsageligt tager udgangspunkt i brugere i aldersgruppen Derfor vil vi sætte fokus på, om søgemaskiner også imødekommer andre brugergruppers behov. Som grundlag for vores speciale vil vi teste om søgemaskinen fremfinder det som brugeren søger, om den hjælper med at skelne mellem interessante og uinteressante resultater og om ældres måde at søge informationer på internettet adskiller sig fra andre aldersgrupper. Da søgesystemet er på engelsk skal deltagerne have gode engelskkundskaber. Det ville også være en god ide at deltagerne tidligere har brugt internettet og søgemaskiner som google. Der vil blive tale om At blive fortrolig med søgemaskinen ved en prøveopgave Søge efter oplysninger om to emner der er beskrevet på engelsk og dansk Pause med kaffe og kage Søge efter oplysninger om to yderligere emner Udfylde et spørgeskema Vi regner med at det foregår i marts måned, enten på Danmarks Biblioteksskole, eller et sted i København efter dit ønske. Har du en ægtefælle eller bekendt som gerne vil deltage er de meget velkomne. Hvis du vil deltage kan du kontakte os på k03sich@dbstud.dk eller Telefonnummer (Kirstine) eller (Barbara) Med venlig Hilsen Kirstine Wilfred Christensen og Barbara Hammer-Aebi Institut for Informationsstudier Danmarks Biblioteksskole 37

38 8 Church newsletter To whom it may concern We are two students from the Royal School for Library and Information Science in Copenhagen who are writing our Master's thesis on how older people use search engines. During our education we encountered a large number of studies on how people use search engines. The persons participating in these studies are most often university students, and we find this to be an insufficient representation of the broader public. In our research for our Master's thesis we concentrate on how older people use the Internet and search engines. In order to make the results of our study comparable to earlier studies, the search engine, the pages shown and the tasks need to be in English. However, since most older people in Denmark often do not have sufficient English skills for participating in such a study, we hope that some of the active parishioners at your church could help us.. We would like to ask you, if we may tell about our project at one of the social activities at St. Albans church, of if we might put up a flyer/poster? Best regards Barbara Hammer-Aebi and Kirstine Wilfred Christensen 38