What is Usability tests Birgitte Berg 1
Agenda 1 2 3 4 5 6 7 Introduction Definition of an usability test Usability test formative/summative How is an usability test done Standards FDA and IEC Conclusion/Summary Q&A 2
Definition of a usability test User testing (face to face interview) conducted with representative end users to obtain direct information and observations about: How people will use the system, Whether they encounter problems, And how they deal with any problems Procedure to assess usability and to determine whether usability goals have been achieved Use of risk analysis to identify critical and essential tasks for testing (Definition from HE75) 14 September 2012
Formative test Formative Usability Testing Definition (HE75): Usability testing that is performed early with simulations and the earliest working prototypes and that explores whether usability objectives are attainable, but without strict acceptance criteria How used: Early and often during design to improve the interface and remove usability and use error risk problems when the design still can be modified 14 September 2012
Formative test One way mirror Interviewer Subject Moderator (data locker) Video
Usability test how is it done? Participant is welcomed and trained in the device asked to perform some handling and differentation tasks ( in use scenerios ) observed while performing tasks and use errors are carefully registered Use error Pattern of use failure that indicates a failure mode that is likely to occur with use and this has a reasonable possibility of predictability of occurence. Note: Use errors can be addressed and minimized by the device designer and proactively identified through the use of techniques such as usability testing and hazard analysis. (Definition from HE75) 14 September 2012
Overview of types of usability testing and considerations from the HE 75 guidance document Type Description When in design cycle Formative usability testing Minimum sample size (per group) Considerations Exploratory High-level test of users performing tasks Conceptual design 5 to 8 more Based on simulations of early concepts that could be very lowfidelity paper prototypes or foamcore models. Useful to employ the think aloud protocol. Assessment Representative users performing real tasks (1:1 or two working together) Iteratively throughout the design cycle 5 to 8 Used to gauge whether usability objectives are obtainable. Early designs or computer simulations. Comparison (contrast) Comparison (competitive) Comparison of two or more design alternatives Tests against competitors Uls During design 5 to 8 Used to decide if one Ul concept or prototype is better than another During design 5 to 8 Used to learn about best Ul features of competitors. May be done early during conceptual design or at end of design cycle. Summative usability testing Validation Real users and real tasks are tested with final design End of design cycle At least 15 to 20 Validates the design with usability objectives as acceptance criteria. Should include training, documentation and labelling, and riskmitigation controls related to use-related hazards. 14 September 2012
Formative Test - Real users eg. Diabetes care 5 children 5 adults/elderly 5 caregivers 5 physicians/nurses 5 inpatient nurses 5 pharmacists Total of 20-30 users Each interview takes about 1hour 14 September 2012
Impairments If participants with real impairment can t be recruited, impairments can be simulated in terms of Colour blindness (Variantor glasses) Blurred vision Hearing problems Manual dexterities 14 September 2012
Worst case conditions: Distractions Distractions will be presented for some participants during the test in terms of: Telephone call (conversation with participant regarding selling a news paper) Participant requested to leave the room Informal conversation during handling Dimmed light 14 September 2012
Formative test overall risk management process Formative Tests are used to identify early use related hazards in initial hazards in initial designs of the user interface May test only certain aspects of the interaction that have been designed and prototyped Are constantly looking for potential ways to eliminate use error hazards and improve overall design 14 September 2012
What formative Usability Test is Focus group NOT These are not tests of device use or risks in device interaction, they provide a different kind of information This approach is useful measuring opinions, attitudes and speculation about a product including a device Customer Preference surveys Clinical device Studies Anecdotal information from Early equipment trials Training session 14 September 2012
Design Tools: standards (Best Practice) Reviews Use HE-75 or other user interface design standards as a guideline for considerations of best practices for Display design, use of colour, brightness, contrast etc. Software layout, menu structure etc. Control design: buttons, dials etc. Labelling: quick reference, user manuals Auditory alarms: frequency, sequence, loudness etc. Other aspects of UI 14 September 2012
Interface Design Tools: Rapid Prototyping Used early in design stage (can be low tech paper based) or can be simulated on a screen Quick turnaround for formative usability testing of prototypes and discovering use errors potential induced by deign features Rapid iterative design Requires minimum development efforts 14 September 2012
Common Formative Testing NOT pre-testing protocol Leading or biasing the user Mistakes Helping to get the tasks completed Talking too much Rushing the subject Making the subject feel inferior Not making sessions friendly and interactive Not keeping track of timing and having to rush to finish all tasks in the protocol 14 September 2012
Formative Evaluation - Summary Formative evaluations and testing should be an iterative process in the design of medical device interfaces A variety of techniques may be used to modify the design to maximise user efficiency and reduce use errors Prototypes and working models can be valuable ways of evaluating interface options in the hands of small number of users Document your formative evaluations! This is vital to the overall pre-market human factors submission to FDA 14 September 2012
Definition of a usability test Summative Usability Testing Usability testing performed in the late stages of design Note : Summative usability tests include verification and validation, and it is a recommended best practice to have a formal acceptance criteria (e.g. usability objectives for human performance and satisfaction ratings) How used? Testing interaction with the device in a simulated or actual environment. The proof of safe use for FDA submissions (Definition from HE75) 14 September 2012
Summative test (Validation test) - purpose This is the culmination of Human Factor Usability efforts It is assumed that modifications of the User Interface resulting from formative evaluation are implemented including modifications to labelling and training If preliminary analyses, formative evaluations and design modifications and development was done well, the validation testing should find good results 14 September 2012
Summative test (Validation test) - purpose The test is the primary evidence to the Authorities To support the conclusion that the final user interface design is reasonably safe for intended users, uses and use environments 14 September 2012
HF/Usability Validation Should be generalizable to expected actual use Validation testing is a mini representation of actual use Representative users (e.g. diabetes patients) Use environments (home settings, hospitals etc) Systematic observation of task performance and Subjective impressions of test participants 14 September 2012
Overview of types of usability testing and considerations from the HE 75 guidance document Type Description When in design cycle Formative usability testing Minimum sample size (per group) Considerations Exploratory High-level test of users performing tasks Conceptual design 5 to 8 more Based on simulations of early concepts that could be very lowfidelity paper prototypes or foamcore models. Useful to employ the think aloud protocol. Assessment Representative users performing real tasks (1:1 or two working together) Iteratively throughout the design cycle 5 to 8 Used to gauge whether usability objectives are obtainable. Early designs or computer simulations. Comparison (contrast) Comparison (competitive) Comparison of two or more design alternatives Tests against competitors Uls During design 5 to 8 Used to decide if one Ul concept or prototype is better than another During design 5 to 8 Used to learn about best Ul features of competitors. May be done early during conceptual design or at end of design cycle. Summative usability testing Validation Real users and real tasks are tested with final design End of design cycle At least 15 to 20 Validates the design with usability objectives as acceptance criteria. Should include training, documentation and labelling, and riskmitigation controls related to use-related hazards. 14 September 2012
Summative Test - Real users eg. Diabetes care 15 children 15 adults/elderly 15 caregivers 15 physicians/nurses 15 inpatient nurses 15 pharmacists Total of 100 120 users Each interview takes about 1½ hour 14 September 2012
Conclusion and decisions after usability test Analysis 1 2 Usability test Final conclusive report -> CRO report Device failed: Repeat circle Device passed: End 14 September 2012 3
Regulatory framework & guidance FDA draft guidance: Applying Human Factors Usability and Engineering to optimize Medical device Design (FDA 2011) ANSI/AAMI HE75:2009, Human factors engineering Design of medical devices, developed by Association for the Advancement of Medical Instrumentation IEC 62366:2007, Medical devices -- Application of usability engineering to medical devices, published by the International Electro-technical Commission (IEC). Medical Device Safety Incorporating Human Factors Engineering into Risk Management, July 18, 2000, available on the U.S. Food and Drug Administration s website 14 September 2012
Courses AAMI (Association for the advancement of Medical Instrumentation) - http://www.aami.org/meetings/hf/index.html - course in March 2013 in Baltimore DS - Brugervenlighed for medicinsk udstyr - Introduktion til standarden DS/EN 62366:2008, 11. oktober 2012. 14 September 2012
Klinisk afprøvning /usability Lovgivning Rådet Direktiv 93/42/EEC om medicinsk udstyr med ændringer (Rådets Direktiv 2007/47/EC) Ministeriet for Sundhed og Forebyggelses bekendtgørelse nr. 1263 af 15. december 2008 om medicinsk udstyr DS/EN ISO 14155:2011: Klinisk undersøgelse af medicinsk udstyr til brug på mennesker DS/EN 62366:2008: Medicinsk udstyr Indbyggelse af anvendelighed i medicinsk udstyr ANSI/AAMI HE75: 2009, Human Factors Engineering Design of Medical Devices. www.medicinskudstyr.dk / www.ds.dk / www.fda.gov
MTIC Klinisk afprøvning af medicinsk udstyr 28. August 2012 Rikke Arendt Christiansen Helene Quie Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Klinisk afprøvning /usability Lovgivning Rådet Direktiv 93/42/EEC om medicinsk udstyr med ændringer (Rådets Direktiv 2007/47/EC) Ministeriet for Sundhed og Forebyggelses bekendtgørelse nr. 1263 af 15. december 2008 om medicinsk udstyr DS/EN ISO 14155:2011: Klinisk undersøgelse af medicinsk udstyr til brug på mennesker DS/EN 62366:2008: Medicinsk udstyr Indbyggelse af anvendelighed i medicinsk udstyr ANSI/AAMI HE75: 2009, Human Factors Engineering Design of Medical Devices. www.medicinskudstyr.dk / www.ds.dk / www.fda.gov Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
MDD definition Annex I Definitioner Udstyret skal konstrueres og fremstilles på en sådan måde, at det ikke forværrer patientens kliniske tilstand eller bringer vedkommendes sikkerhed i fare og heller ikke er til fare for brugerens og en eventuel tredjemands sikkerhed Det indebærer: at risici i forbindelse med forkert brug som følge af udstyrets ergonomiske karakteristika og de omgivelser, hvori det skal anvendes (design for patientsikkerhed), begrænses i videst mulig omfang, og at der tages hensyn til brugernes teknologiske viden, erfaring og uddannelse, og hvor det er hensigtsmæssigt, de lægelige og fysiske omstændigheder (design for lægfolk, erhvervsfolk, handicappede eller andre bruger). Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
MDD definition Annex I Definitioner Risikoen for enhver bivirkning og uønsket følgevirkning skal stå i et acceptabelt forhold til den angivne ydeevne. Påvisningen af overensstemmelse med de væsentlige krav skal omfatte en klinisk evaluering i overensstemmelse med bilag X. Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
MDD definition - klinisk data Definitioner Sikkerhedsdata eller data om ydeevne, der stammer fra brugen af medicinsk udstyr. Kliniske data indsamles ved: a) klinisk(e) afprøvning(er) af det medicinske udstyr, b) klinisk(e) afprøvning(er) eller andre undersøgelser, der er offentliggjort i den videnskabelige litteratur, af lignende medicinsk udstyr, hvis ækvivalens med det pågældende medicinske udstyr kan godtgøres, eller c) offentliggjorte eller ikke-offentliggjorte rapporter om anden klinisk erfaring med enten det pågældende medicinske udstyr eller et lignende medicinsk udstyr, hvis ækvivalens med det pågældende medicinske udstyr kan godtgøres. Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Definitioner MDD definition - klinisk evaluering Bekræftelsen af, at et medicinsk udstyr under normale anvendelsesforhold opfylder kravene med hensyn til karakteristika og ydeevne, samt vurderingen af bivirkninger og acceptabiliteten af forholdet mellem fordele og risici En klinisk evaluering skal følge en defineret og metodologisk forsvarlig fremgangsmåde baseret på kritisk evaluering af: 1. Relevant, tilgængelig videnskabelig litteratur, eller 2. Resultaterne af alle gennemførte kliniske afprøvninger, eller 3. De kombinerede kliniske data i punkt 1 og 2. Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Definitioner ISO 14155 definition - klinisk afprøvning Ethvert forsøg på mennesker, der har til formål at afdække eller efterprøve sikkerheden og/eller ydeevnen af medicinsk udstyr. DS/EN 62366 definition brugervenlighed Karakteristika ved brugerfladen, som dokumenterer at denne er nem at bruge, er designet rigtigt, og fungerer som tiltænkt/beskrevet til brugeres tilfredshed Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Usability vs klinisk afprøvning Eksempler Plejeseng med ny robotteknologi til styring af seng, der muliggør selvbetjening af seng for personer med stærk nedsat bevægelsesevne Sensorer til måling af muskelaktivitet i forbindelse med genoptræning Software til beregning af patienters sundhedstilstand efter indtastning af vitale parametre Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Klinisk afprøvning af medicinsk udstyr - processen Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Klinisk afprøvning Planlægning Deltagere i planlægning og gennemførsel af en klinisk afprøvning skal have dokumenteret træning/ekspertise inden for området Begrundelse for gennemførsel og design af en klinisk afprøvning bør have baggrund i en gennemgang af relevant litteratur, pre-klinisk dokumentation og kliniske risici identificeret Er produktet anvendelig til det formål og de brugere/patienter, som det er tiltænkt til. Resultaterne skal være klinisk relevante, opnået vha. videnskabelige principper og understøtte formålet med afprøvningen Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Klinisk afprøvning Dokumentation Klinisk afprøvningsplan (CIP) Investigator s Brochure (IB) Case Report Forms (CRF) Patientinformation (PIC) og informeret samtykke (ICF) Tilladelser (etisk komite, sundhedsstyrelsen) Kontrakter Monitoreringsplan Statistisk analyse plan (SAP) Kvalitetssystem til kliniske afprøvninger Budget Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Klinisk afprøvning CIP ID af dokument (fx version), involverede parter (Sponsor, Investigatorer) Synopsis Beskrivelse af udstyret Design af afprøvning Formål og hypoteser Beskrivelse af patientpopulation (fx in- og eksklusionskriterier) Procedurer Monitorering Statistik Håndtering af hændelser/bivirkninger Kvalitetssikring (fx datahåndtering, tidlig afslutning, afvigelser) Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Klinisk afprøvning IB Resume af litteratur Generel beskrivelse af udstyret Udstyrets virkemåde, herunder brugsanvisning Forventet klinisk ydeevne Materialer Præ-kliniske test resume Resume af tidligere klinisk erfaring Liste over anvendte standarder Resultat af risikoanalyse Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Klinisk afprøvning Andre dokumenter CV - investigatorer, projektsygeplejersker Identifikation af institution, der skal deltage i afprøvningen fx Aarhus Universitetshospital, Klinisk Gastroenterologisk Afdeling Etisk komite afgørelse, korrespondance Kompetente myndigheder afgørelse, korrespondance Aftaler Sponsor/investigator og andre fx laboratorier, CRO Forsikring (hvis relevant) Patientinformation/Informeret samtykke Case report forms Formularer til indrapportering af hændelser Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Klinisk afprøvning Før afprøvningen går i gang Holdsammensætning Færdiggørelse af al dokumentation Fremstilling af produkter til afprøvning Kontrakter Monitoreringsplan Træning/uddannelse Tilladelser etisk komite, kompetente myndigheder, datatilsynet Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Klinisk afprøvning Under afprøvningen Produkthåndtering lagerstyring Monitorering kildedataverificering, device accountability, hændelser, CRF Indrapportering af hændelser, afvigelser Overvågning af afprøvningen fx sikkerhedskomite Årlige sikkerhedsrapporter Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Klinisk afprøvning Efter afprøvningen Returnering af produkter Sikre at dokumentation på site er i orden Statistisk analyse Endelig rapport (indsendes til myndigheder og etisk komite) Opbevaring/arkivering af data Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Klinisk afprøvning Hvad skal kliniske data så bruges til? Dokumentation til endelig CE - mærkning Input til nye mulige/nødvendige afprøvninger Forlængelse af eksisterende studie Publikationsplan/strategi Feedback til Marketing/salg Planlægning af post-market surveillance, PMS (markedsovervågning) Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com
Spørgsmål? Qmed Consulting ApS Niels Juelsgade 11 4600 Køge Denmark TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: info@qmed-consulting.com