Digital physiotherapy rehabilitation!

Transkript

1 ! Digital physiotherapy rehabilitation! DAT2 A400

2 Copyright Aalborg University 2014 This report is written in LATEX using a 12pt Baskerville typefont variant.

3 Department of Computer Science Selma Lagerlöfs Vej 300 DK-9220 Aalborg Ø Title: Digital Physiotherapy Rehabilitation Project Period: Fall Semester 2014 Project Group: A400, Second Semester (DAT2) Participant(s): Andreas Hairing Klostergaard Arash Michael Sami Kjær Kasper Kiesbye Bendixen Michael Kusk Christensen Mikko Lukas Räsänen Sebastian Rom Kristensen Søren Nørgreen Gustafsson Supervisor(s): Rikke Hagensby Jensen Copies: 9 Abstract: This report will examine physiotherapy and relevant subjects, in order to determine, whether it is possible to use technology to assist people with physiotherapy. The report will examine the groups involved with physiotherapy, and how their interests would affect a software solution. It will also examine various types of technology in order to determine, which is most useful for physiotherapy. Based on these factors, a software solution for assisting people with physiotherapy will be developed. This solution will use various types of theory to model the human body and its movement. The solution will also be tested, in order to ascertain, that it fulfills the demands, that are specified earlier. Pages in Report: 100 Pages in Total: 136 Date of Completion: May The content of this report is freely available, but publication (with reference) may only be pursued due to agreement with the author.

4 Andreas Hairing Klostergaard Arash Michael Sami Kjær Signature Signature Kasper Kiesbye Bendixen Michael Kusk Christensen Signature Signature Mikko Lukas Räsänen Sebastian Rom Kristensen Signature Signature Søren Nørgreen Gustafsson Signature

5 Contents Preface IX Introduction 1 I Problem Analysis 3 1 Methods used 5 2 People and stakeholders Physiotherapy Patients The physiotherapist Developers The healthcare sector Summary Motivation What is motivation Intrinsic and extrinsic motivation How to increase motivation? Importance of motivation in physiotherapy Motivation using VR Technology Analysis Motion Capture Motion-based Technologies The Intelligent Elastic Kinovea Photogrammetry Nintendo Wii Playstation Move Microsoft Kinect Summary Scope of project Kinect People and Stakeholders Problem Statement Solution Specifications 31 V

6 II Development 33 7 Theory Modelling Linear algebra Solution Program design Model, View, Controller Data handling Interface Implementation Motivation Testing What is testing and why it is important Black Box White box Unit test Our testing III Conclusion Reflections The Physiotherapist Current State of the Solution Interface Comparison Model, View, Controller Conclusion Future works Motivation Game Elements Interface Relative space Bibliography 97

7 IV Appendices 101 A Transcript: The exercise 103 B Transcript: Patients and motivation 107 C Transcript: Physiotherapist 115 D Transcript: Organisation 121 E Class Diagram 123

8

9 Preface This report is written by the second semester (DAT2) Computer Science project group A400 at Aalborg University, spring The report focuses on the subject of Digital Physiotherapy Rehabilitation, for which a software solution has also been developed.. We want to thank Amanda Hill, Erik Stubkjær, Rikke Hagensby Jensen and Thomas Gabriel Schreyer for their help in relation to the work behind this report. The front page shows the Vitruvian Man, along with an overlapping, connected set of dots, that represents the skeleton used with the Microsoft Kinect, which is also pictured. Source placement Whenever a source is placed in front of the punctuation like this [1]. It indicates that the specific sentence refers to that source. If a source is placed after the punctuation, which is usually at the end of a paragraph, it indicates, a portion of the paragraph is referring to the source, like this. [2] Acronyms API Application Programming Interface. 27 GUI Graphical User Interface. 51, 55, 56, 88, 94 MET Motivational Enhancement Therapy. 16 MVC Model, View, Controller. 50 SDK Software Development Kit , 51 UI User Interface. 55 VR Virtual Reality. 16, 17 IX

10

11 Introduction Many people will become injured in the course of their lifetime. It could be sport injuries, injuries acquired at work or any disabilities such as missing motor skills. Sometimes these kind of injuries mean that parts of the body needs retraining to return to normal functionality. Some of them require some kind of treatment that often involves a physiotherapist. Most physiotherapeutic treatments includes home exercises such as stretches. These represent a responsibility for the patient, that is not always upheld. But can technology assist people with remembering to, and wanting to, perform these often trivial, boring or hard exercises at home, without outside help or supervision? We believe that technology can help us assist people and help them perform their exercises properly. We believe that motion sensor technologies could be a huge help in achieving this increased efficiency in many different scenarios. Physiotherapy is a type of healthcare that uses examination, evaluation, diagnosis and physical intervention. Motion sensor technologies have the potential to improve several aspects of physiotherapy when the physiotherapist is not available. Examination can be done by a physiotherapist but the evaluation could be performed by a piece of technology. A motion sensor could help a person keep track of their progress and their degree of correctness when doing exercises. Different sensor technologies have varying degrees of accuracy and features. Comparing these could reveal, which ones might be most useful, in assisting people in need of physiotherapy. This brings us to our initial problem: How can we digitally assist people in need of physiotherapy? 1

12

13 Part I Problem Analysis 3

14

15 1 Methods used We need to get a better understanding of our problem and its context. For this analysis, we will primarily be using the Laudon & Laudon Model, shown in fig. 1.1, which addresses some key points to get an understanding of the problem and the context. This model builds on the analysis of people and stakeholders, which we define as groups of people affected by the project, and groups and organisations that can influence the project directly. It also examines technologies and the problem of obtaining enough information to develop a solution. The product we are looking to develop is a software solution to the problem of, how we can digitally assist people with physiotherapy. To obtain data for the analysis of people and stakeholders we have conducted an interview with a physiotherapist. This provides an inside view from a physiotherapist and provides a large amount of data for the analysis of both the people and stakeholders. Figure 1.1: Laudon & Laudon 5

16 Part I: Problem Analysis Figure 1.2: Interview Figure The data obtained in the interview is qualitative data, but is backed up by other sources. The way the interview is analysed can be described using fig. 1.2 where it is shown how the different parts of the interview is collected under different tags, that indicate the subject that the part of the interview is relevant for. So as shown, there is an interest group called patients which is where parts of the interview concerning the patient analysis is. [3] The Laudon & Laudon model is also used to analyse the various technologies available that could help us with assisting people in physiotherapy a so called technology assessment. This will lead to the conclusion on what technology will be used in the further development of this project. 1 1 The content of this chapter is based on the lectures during our first semester course of Problem-based Learning in Science, Technology and Society. 6

17 2 People and stakeholders This section will introduce the people and stakeholder part of the Laudon & Laudon model, describing the people and organisations, that could have an interest in the project. We will take a closer look at which people are involved and/or could have an interest in a digital solution to help physiotherapy and what organizational involvement there is in physiotherapy. The people and stakeholder analysis is performed on the background of qualitative data obtained through an interview with a physiotherapist, which provides an insight into some of the problems related to physiotherapy. The interview with the physiotherapist is considered generally indicative of the current state of physiotherapy, and his statements are supported by other sources. Additionally, this section is also based upon data showing that technology is increasingly being used to assist patients with their rehabilitation and exercises. The interview was conducted through a conversation with physiotherapist Thomas Gabriel Schreyer who we asked a line of questions in order to explain some general things concerning physiotherapy and technology in relation to physiotherapy in appendices A to D. 2.1 Physiotherapy Before looking at physiotherapists as a stakeholder, we need a better understanding of, what physiotherapy is. This introduces the assessment of the problem which is shown in the Laudon & Laudon model. Physiotherapy is a form of healthcare, that is used to help people restore movement and function in their muscles and joints. Physiotherapy is typically prescribed for people affected by injuries, illness or disabilities. The practice of physiotherapy is done through movement and exercise, manual therapy, education and advice. The purpose of physiotherapy is to maintain health for patients, to manage and prevent pain and further injuries. Part of physiotherapy is also encouraging and motivating the patient to keep doing movement and exercise to maintain a healthy body. Physiotherapy is normally done through examination and diagnosis by a physiotherapist, followed by a physical intervention and an evaluation of the patients progress throughout the process. Physiotherapy can be done both for individuals 7

18 Part I: Problem Analysis and for groups. A physiotherapist can be associated with different types of centres and institutions such as hospitals, sport centres, rehabilitation centres or similar. Physiotherapy is based upon professional research and education. The process involves a consultation between the patient and the physiotherapist, where the injury is diagnosed, the physiotherapist performs an intervention and constructs a set of exercises for the patient to perform at the clinic or at home. Followed up by meetings to ensure progress and that the right measures are being taken. Beyond the interaction between the physiotherapist and the patient, the process of physiotherapy can also be affected by external factors, such as the patients family, caretakers, and their local environment. Physiotherapy as a profession is regulated by laws. This introduces the healthcare sector as an organization that supports and regulates physiotherapy clinics. As such physiotherapy is a big part of the healthcare system and of the public welfare. [42, 9] 2.2 Patients When examining physiotherapy and the involved parties, one of the most important of these parties are the patients receiving physiotherapeutic treatment. These patients have an interest in improving the treatment they receive. This interest could be in faster and/or improved results, an easier course of treatment or to improve their own motivation going into the treatment. An existing problem is that the physiotherapist cannot necessarily be present every time the patient or patients have to do exercises. There are not enough resources for the physiotherapist to be constantly present. Therefore, part of the course of treatment becomes home exercises that the patient is responsible for. This means that the physiotherapist delegates some of the responsibility for the treatment to the patient and their ability to take action and administrate exercises on their own. This presents a problem to some patients, who might see physiotherapy as a nine to five job, where they work at the clinic, and when they get home they are off duty, like they would be if they were working. Other patients may face problems with motivation, or the fact that they are in pain and therefore feel discouraged [appendix B]. The problem represented by home exercises is a very diverse problem and can be hard to pinpoint exactly, as each person has their own reason as to why setting apart time and effort to do exercises at home presents a problem. Through this we can derive that motivation has a large effect on physiotherapy. 8

19 Chapter 2: People and stakeholders Improving a patient s motivation could help the patient feeling less obligated or stressed during the course of treatment, and improve their ability to perform their exercises. This could not only improve the efficiency of the patients treatment, but also improve their experience, as being motivated would mean, that they actually wanted to perform their exercises. [4] Physiotherapy patients are of all ages and these people might have different preferences among the various ways to motivate them, but ideally a solution could have tailored systems for different groups of patients. While patients are grouped mostly by the type of injury they have, instead of their age group, it is possible that a solution could hold packages of different exercise types, so that patients would not be overburdened with unnecessary exercises. Children, teenagers, adults and elderly people are all different and would most likely require different kinds of motivation. Active adults, who are used to exercise and sports, might not find a mini game as appealing as children or elderly people would. For a person who is already involved in sports, it might be a question of keeping them from overdoing the exercises, while others might need motivation to get started or keep going. Fine tuning the number and type of exercises must be part of the physiotherapist s job, as the consultations between the physiotherapist and the patient is there to yield information to the physiotherapist, so that he can make educated decisions on, what the patient s needs are. This is where he is supposed to be the judge of, whether this patient could use a digital solution like a motion sensor, or not. This would allow the patient to perform their exercises at home, with the assistance of the digital solution, which would ideally improve the efficiency of the exercises. A downside for the patient could be that they would need to have some knowledge of how to use the system that accommodates the digital solution. Ideally the installation should be easy or plug and play, but it would still need adequate hardware, which the patient would either have to own or borrow from somewhere. Patients might not find this appealing and would then discard using such a solution. [appendix B] 2.3 The physiotherapist The physiotherapists themselves are of course also a stakeholder. They could have great interest in increasing the effectiveness of home exercises to increase efficiency and potentially shorten or improve the course of treatment for their patients. 9

20 Part I: Problem Analysis The fact is that most physiotherapists are short on time, as they have many patients and consultations, which are often short and are used by the physiotherapist to keep track of progress and give instructions on exercises. Improving home exercises and making it easier for the patients by allowing them to use a digital solution that could provide guidance on how to perform exercises might help the effectiveness of the exercises, and improve the outcome of the consultations. Also with the increase in population and in elderly people we can only assume the demand for physiotherapist will also be increasing [1, 32] and therefore the physiotherapist could have great interest in digitizing part of the treatment to lessen their work burden, and gain time to treat additional patients. Furthermore the physiotherapist might be able to adjust the treatment with the knowledge that a software solution could give, as the patient cannot provide them with the same kind of data, that a digital solution might. Knowing if and when the patient is doing exercises could be a good thing for the physiotherapist, as that information is vital to figure out if and when a treatment is working. Using a digital solution to assist with home exercises could however also be an issue for the physiotherapist. This solution might be based around a motion-detection based technology, and this would necessitate the acquisition of whatever hardware, the solution used [appendix C]. 2.4 Developers If a digital solution was to be used in physiotherapy, it would require the development and maintenance of the required software, as well as the acquisition of the requisite hardware, which leads to the next group of stakeholders: Developers. Both hardware and software developers could have separate, large interests in a digital solution for physiotherapy. Hardware developers would be in charge of supplying the hardware, that the solution would use. Developers of this hardware would have an interest in the solution, as it would create additional demand for their products. Software developers might have an interest in adopting the idea and further develop upon motion-based technology used in physiotherapy. While these might have an interest they are not a concern for this project, as the software will be developed as a part of the project, and the project itself does not require a large amount of hardware. 10

21 Chapter 2: People and stakeholders 2.5 The healthcare sector The healthcare sector of the Danish state also has a large interest in ensuring the effectiveness of home exercises, as it spends a large amount of money paying for various types of therapy. In an average, doctor-referred visit to a physiotherapist, the healthcare sector pays approximately 40% of the fee for the patient[39]. The fact that it only pays for doctor-referred visits to physiotherapists, means that doctors have a major influence, on Danish physiotherapists, as a large amount of their revenue is dependent on them. Optimising the exercises performed at home could reduce the amount of visits the patient would need, and thus reduce the amount of money the healthcare sector would have to pay. As such, the healthcare sector is a major stakeholder in this project, as it could stand to save a large amount of money, if home exercises were made more efficient through a digital solution. Optimising the exercises people are capable of doing at home, would improve physiotherapists ability to keep up with demographic changes causing a greater demand for physiotherapy, as well as population growth. The healthcare sector would also have to consider the economic sides of the introduction of a digital solution, as development and distribution of systems can take large amounts of both time and money. Specifically in Denmark, the healthcare sector s interaction with physiotherapists in the country happens through the various regions. The regions have a collective agreement with a large amount of physiotherapists, that the state supports. Additionally, the regions also control a fund, that physiotherapists can apply for funds from, in order to further the physiotherapeutical discipline. This means that it is the regions, that manage the healthcare sector s interests, as well as the direct interaction with the physiotherapist. As such, it would most likely be the regions, that would need to be interacted with, if this projects solution was to be deployed. [32, 43] and [appendix D]. 2.6 Summary In this chapter we have been through the various people and stakeholders involved or affected by a solution that would digitally assist people with physiotherapy. Patients were found to be in need of help with the problem that lies in the exercises 11

22 Part I: Problem Analysis they have to do at home. While the exercises or the problem is not necessarily seen in a specific group of people, it is evident that there are some groupings that can be dealt with. These groupings are based around the type of injury, the patient has sustained. Additionally, the question of the patients motivation was raised, but seeing as motivation is a rather abstract topic, it will be examined greater detail later in this project. Also the physiotherapist have a great interest in lessening the work burden and potentially use a solution like ours to counter democratic changes and improve the course of treatment for patients. With the interview, that was conducted as a part of the examination, the physiotherapists have obtained the opportunity to affect the product and act as potential costumer for our product. The physiotherapist we were in contact with provided us with ideas and input that could help the development. Finally, the healthcare sector could have interest in delegating even more responsibility on to the patient and developers could have economic or work related interest too, but we have chosen to discard these interests. This is done based on a focus on the actual development of the solution, rather than the deployment of it. 12

23 3 Motivation These sections about motivation do not follow the structure of the Laudon & Laudon model. Instead, it serves to further explain the term motivation, and how motivation may impact this project. It has been concluded from what we have discussed earlier about people who use physiotherapy, that we need to know about motivation in order to potentially use that knowledge for a solution later on. It is shown through the research that has been made, that patients can lack proper motivation to do exercises at home in between sessions with the physiotherapist. We wish to motivate people to do their physiotherapeutic exercises. So we are going to look into the mechanics of how motivation works so that we can apply it to a possible solution later on. The following section will examine motivation as a psychological subject, its mechanics and how or when to apply these. We will also look into motivational techniques applied in the field of physiotherapy. 3.1 What is motivation A large amount of theories regarding motivation exist, and many of these theories are based on various factors, as the source of human motivation [17]. Many of these theories point to various types of needs or rewards as the source of human motivation, however, many of them disagree, as to the importance, of the various factors. For this project, we use a combination of various theories, that are primarily based on internal and external motivation from the theory of Self-Determination, that will be used in the following section. Motivation is defined as the process that initiates, guides and maintains goal-oriented behaviors [21]. Such behaviour could for example be a morning jog for the enjoyment of exercise(/intrinsic), or working a job to earn money(/extrinsic). Driving factors that regulate these kind of behaviours can be biological, emotional, social, moral and cognitive. Commonly, motivation or motive is the term used to describe why a person performs an activity[21]. There are three major components in motivation. Activation, persistence and intensity. Firstly, activation can be described as the decision to start an action or 13

24 Part I: Problem Analysis behaviour. Persistence is the continued effort towards a goal, in spite of obstacles and hindrances to reach it. Intensity is the amount of vigour, focus and effort put into reaching the goal. You can still remain persistent in reaching your goals, without much intensity. A person can be doing his or her physiotherapy exercises, but probably will not do them right if he or she is not putting any effort into it. While some believe in the splitup of motivation in only intrinsic and extrinsic motivation, others have widely different theories, like Aquired Needs, Cognitive Evaluation and Side Bet amongst others.[5] 3.2 Intrinsic and extrinsic motivation Intrinsic motivation describes, when one is motivated by internal rewards. When a person performs an activity for the sake of the activity itself, without any apparent external rewards, then it is because of intrinsic motivation. Extrinsic motivation stems from the promise of external rewards. These could be material, such as money or candy, or metaphysical such as praise, fame or grades. In order to provide an example, we will consider a person performing their physiotherapy exercises. If said person is doing it in order to get well again, it is extrinsic. If that person simply enjoys doing the exercises, then the motivation is intrinsic. In both cases, the person is motivated to perform the exercises, and as such, will likely perform them as he has been instructed by the physiotherapist. It follows that both types of motivation could be useful in this project. Intrinsic motivation vs. extrinsic Of course this poses the question, which is better: Intrinsic or extrinsic motivation? Experts suggest that people who are intrinsically motivated are also a great deal more creative. In a work setting, external rewards can positively affect productivity; however the actual quality of work is vastly improved by intrinsic motivation.[21] Extrinsic motivation can be used to engage people in an activity they previously did not have any interest in. Extrinsic rewards can motivate people to acquire skills or knowledge that, once learned, become internally rewarding on their own. When people receive feedback as an external reward, they know their performance has reached a standard deserving of reinforcement.[21] It is preferable to avoid attempting to extrinsically motivate people that are already 14

25 Chapter 3: Motivation intrinsically motivated so as to avoid attempting to motivate people for something they are motivated for, by offering rewards. If people are intrinsically motivated, they do not need the promise of external rewards, in order to perform the activity. 3.3 How to increase motivation? Increasing extrinsic motivation should be fairly straightforward, as it builds on constant reinforcement. Rewards, fear of punishment or fear of disappointment all contribute to increasing extrinsic motivation. Extrinsic motivation can be about being rewarded, or about pleasing others for various reasons. It has been observed that offering rewards for an action that already was intrinsically rewarding can make it less so [21]. It should be mentioned however, that while external rewards can decrease intrinsic motivation, unexpected rewards do not, the decrease is tied to coming to expect external rewards for the activity.[21] It has also been found that offering praise and positive feedback when individuals perform better in comparison to others actually affects intrinsic motivation positively [21]. If external rewards are greatly exaggerated for an activity, such as a simple task that deserves little to no reinforcement, it will decrease intrinsic motivation[21]. There are five factors that contribute to intrinsic motivation: Challenge: People enjoy opportunities to prove themselves and increase self esteem. This primarily applies when performance feedback is readily available, and attaining the goal is possible but not certain. Curiosity: Curiosity can be sensory, in that something in the environment catches the individuals attention. It can also be cognitive: The activity makes the individual wish to learn more. Both of these factors appeal to the individuals curiosity, by making him want to learn something, the difference is in, when the individuals curiosity is aroused. Control: Control, as in wanting to do activities without being forced, asked or commanded to do them. This could also be referred to as choice or autonomy, that allows a person to make an impact on their own life. Cooperation and Competition: Individuals take satisfaction from helping others, making cooperative activities intrinsically rewarding. Individuals also take satisfaction from knowing they have performed well in comparison to others. 15

26 Part I: Problem Analysis Recognition: People enjoy praise for their accomplishments. [21] These factors of motivation could potentially be used, if this project was to incorporate intrinsic motivation. While implementing external rewards, such as a score, might be relatively simple compared to implementing intrinsic motivation, these factors provide a reference point, that may make this implementation more simple. 3.4 Importance of motivation in physiotherapy For the purpose of this project, it is also very important to examine the effect, that motivation has on physiotherapy. This ensures that it is made certain, whether incorporating motivation into this projects digital solution, might have a positive effect on the patients that would end up using it. In the study performed by Vong et al. (2011) they prove that Motivational Enhancement Therapy (MET) plus physiotherapy has a greater effect on the patients than conventional physiotherapy alone in patients with lower back pain. This effect increase is both in their compliance with the physiotherapist and in their physical function.[46] The MET was designed to increase the patients motivation to engage in the treatment. It also included strategies for managing pain [46]. This information proves, that motivation can be a powerful supplement to physiotherapeutic exercises, and therefore is worth considering, when developing a digital solution. 3.5 Motivation using VR While motivation may be a powerful supplement to physiotherapeutic exercises, integrating something as abstract as motivation into a digital solution is a difficult task. One type of digital solution, that has been successful at motivating people, is Virtual Reality (VR) games, and this may serve as inspiration, for how to digitally motivate people for physiotherapy. Games can be highly motivating, and the use of combinations of learning and games have increased over the years. Kids want to have fun, and having fun 16

27 Chapter 3: Motivation learning makes learning easier, which is why more and more games are being made with teaching in mind. But learning is not the only thing that games can motivate people into doing, it could be exercising or even doing therapy exercises. According to several of the participating physiotherapists in Levac et al. (2012) VR games are capable of motivating kids to do their exercises. One participant even says that There s no other way that I can engage a patient for so long, [...] [20]. Another aspect is that it makes it more difficult to put the quality of movement performance in focus, seeing that children are more inclined to see it as a game and therefore end up playing [20]. However there has been trials focused on upper limb training, which were the only area that returned a statistically significant effect. There were not enough results to reveal any statistically significant effect on any other area of focus[19]. This information proves, that motivation can be increased digitally, and also has an effect on physiotherapeutic exercises. This allows us to conclude that it would be beneficial to our product, to include motivation, as it has been shown to have a positive effect on some types of exercises. 17

28

29 4 Technology Analysis In this chapter we will move on to the technology analysis, which is one of the parts of the Laudon & Laudon model. We will strive to bridge the gap from physiotherapy to technology. Physiotherapy is based on doing a set of exercises in order to heal or strengthen the body. This movement of the body has to be digitally registered to technologically assist physiotherapy exercises. There are various ways to register and model these movements digitally. One branch of technology, that could be used for this purpose, is motion capture. In this section we will take a look at the existing technologies used to create digital physiotherapy. Furthermore we will take a look at motion-based technologies, which could possibly be used to create a better digital solution. This section will give us an insight into the technologies currently used, as well as the technologies that could be used for the solution of the project. Furthermore we will conclude which of these technologies is the most suited for a digital solution, that assists people with physiotherapy. First we will take a look at how motion-based technologies is used to model the real world as a virtual reality. 4.1 Motion Capture To better understand how to model movements of the human body digitally, this section will take a look at motion capture. Motion capture allows for the interpretation of the movement of the human body, which can then be used as input for various types of software, that could be used in digitally improving physiotherapy. Motion Capture is a way to collect information about a subjects posture and location over time. The technical goal is to obtain information about specific points of interest. This information can then be used to calculate various other informations, such as movement or angles between joints. This could for instance be the location of the hand, elbow and shoulder. With the location of these three body parts, it will be possible to see how they are located according to each other, which could be used to determine if an exercise was done, and if it was done 19

30 Part I: Problem Analysis correctly. One of the greatest problems with this technology is to determinate where the points of interest are located, which is generally time-consuming to do with software. This can be solved by marking the points of interest on the subject, so it becomes easier for the software to recognise the points. [16] 4.2 Motion-based Technologies Motion capture and other motion-based technologies are used for many things, and one of the most common things is actually motion-based video-games. The idea of using motion-based technologies for video-games is relatively old, and can be traced back to a number of devices that have been released for various gaming systems, such as the Nintendo Powerpad, or Konami s Dance Dance Revolution, that used a specially designed dance platform to read the players movement. With the introduction of the Nintendo Wii in late 2006, motion based technologies became a large part of the home console market, and initially gave Nintendo a large share of said market, prompting Nintendo s main competitors, Sony and Microsoft, to develop their own motion-based technologies, the Playstation Move, and the Kinect [40]. In the following sections we will take a look at various methods to detect motion, and how these could be used with physiotherapy. While some of these will be part of the home console market for video games, others will be examples of the various things, motion-based technologies are used for. 4.3 The Intelligent Elastic Various digital solutions for improving physiotherapy exercises at home have already been produced,though in this case, it is simply by improving one of the training utilities already used. Michael Skovdal Rathleff, a Ph.d. student at the University Hospital in Aalborg have modified a training elastic into a more intelligent piece of technology. Their studies had revealed, that in a group of people, percent would lie and say they had done their physiotherapeutic exercises. Because of this, he developed the intelligent elastic that looks like any other training elastic used for physiotherapy, 20

31 Chapter 4: Technology Analysis however it has a sensor that can tell when it is being used, when the patient uses the device. An old method to ensure patients did their exercises, was to make them film themselves performing it in front of a computer with a webcam, however this made the patients feel watched. The elastic will simply measure how many times it has been stretched and send the data directly to the therapists computer. The patient can also view this data on their home tv-screen if they wish to do so. It can help them check if they are doing it with the right speed or if they are too slow [34, 30]. This technology is representation of the state of the art. However, looking at the information it provides, it does not provide information about how it was used, or if the exercise was performed correctly. 4.4 Kinovea We were made aware of Kinovea through our interview with Thomas appendix A. Kinovea is a video player for sport enthusiasts which allows the user to study videos of one self or other athletes, to help improve techniques. There are various tools in the program: Slow Enables the user to review their video in slow motion to enable them to check their technique in details. Show Refers to tools, which enables the user to add arrows, lines and descriptions to the video. Compare Allows the user to sync up two videos shown side by side to ease the detection of differences. Measure Gives the ability to measure distances or angles, this can be used to track angles over time to see how the movement of joints are during the exercise. Extending Allows the user to export the analysis of the video to a spreadsheet, which could be used for scientific study. [18] This functionality can be used to analyse a video of the patient doing an exercise, and to control how well the exercise was performed. The main concern here is the 21

32 Part I: Problem Analysis fact, that you have to manually do most of the work, and this will either put more work onto the physiotherapist or the patient. This video player is also a state of the art, as it provides the user with a lot of powerful tools, which enables the user to improve their techniques in various exercises. It does not provide any kind of advanced motion capture, and as it is done on a previously recorded video, it does not provide much use for the solution. 4.5 Photogrammetry Another piece of technology, that could be used for motion capture in relation to physiotherapy is photogrammetry. With a minimum of two cameras or pictures taken at minimum two positions, it is possible to build a model of a 3-dimensional space. An example of software that use multiple images from a single camera to construct such a model is MonoFusion [33]. Using two cameras simultaneously is called stereophotogrammetry, capturing the same subject from two different angles. Tim Greenfield illustrates how this can be achieved through MMP Player Framework, where he created his own software to generate a 3d video stream with inputs from two webcams located closely together [13]. Without a deeper investigation, we believed photogrammetry to be too difficult a subject for use in this project, focused on digital assistance in physiotherapy. 4.6 Nintendo Wii Announced in 2006, and launched later the same year, The Nintendo Wii was the first of the three major home consoles of the generation to incorporate motion based technology into video games. The console is based around the use of the Wii remote, a remote-like controller, that is capable of detecting three-dimensional acceleration using built-in gyroscopes, as well as being tracked through an infrared system. The Wii remote essentially allows the system to keep track of its movement through a three-dimensional space, which can then be interpreted by the system to represent the user s movement as input. While the Wii s motion tracking is rather simple, as it only tracks the location of the Wii remote, this also allows it to remain relatively robust and accurate. 22

33 Chapter 4: Technology Analysis Nonetheless, the fact that the Wii is only capable of tracking the Wii remote in the user s hand remains a major inconvenience in using it for physiotherapeutic exercises. The Wii does not have a publicly accessible software development kit, but it is possible to use input from the Wii remote in flash games, that the Wii is capable of playing [40]. There are several examples of the Nintendo Wii already being used to improve physiotherapy. One of these examples is Southern Illinois Healthcare [41]. They bought a Wii system to improve the willingness of the patients to do their exercises. This was generally improved through making the exercises part of a game, which created a competitive environment. Since most people want to win, they could shift their focus away from the exercises, and most people performed better. 4.7 Playstation Move Prior to Playstation Move, Sony had a motion based camera for Playstation 2 called EyeToy, which was released in This device used simple gesture recognition technology to see motions and needed a very well lit room to recognise the gestures [35]. EyeToy helped Sony understand what people wanted from the physical interaction with the game, which they used to develop the Playstation Move [22]. Announced in 2009, and launched in 2010, the Playstation move functions as a supplementary controller for the Playstation 3 home console. Much like the Nintendo Wii, the move uses a hand-held controller with built-in gyroscopes to track movement, while also visually tracking the controller using a camera. The controller also features a mounted orb, that the camera s software is capable of using as a reference to determine depth. Reminiscent of the Wii, the Playstation move only actively tracks the movement of the handheld controller, however, unlike the Wii, the Move also uses inverse kinematics to calculate the position of the rest of the users body. The Playstation move does not have an open software development kit, and as such, it would be very difficult to acquire the tools necessary to develop software for the Playstation move [40]. 23

34 Part I: Problem Analysis 4.8 Microsoft Kinect Announced in 2009, and launched in 2010, the Microsoft Kinect was originally released as a motion-based controller for the Xbox 360, and later released for Windows. The Kinect uses a depth camera, an RGB camera, an infra-red laser speckle pattern projector and a multi-array microphone to interpret the users body language. The combination of RGB and depth camera allows the Kinect to accurately track up to four different persons in its field of view, while also keeping track of their position in a three-dimensional space. Compared to the alternatives, the Kinect takes a much larger input, as it uses a series of joints to keep track of the person(s) movements purely through visual input provided by its cameras. The Kinect also outputs the data related to these points through its own software, removing the need for additional software to keep track of the joints. While this makes the Kinect less responsive, operating at 30Hz, it does enable the Kinect to use the movement of the user s entire body, allowing a much greater range of inputs than the other types of motion-based technologies used for video games [14, 40] Body Tracking The most important thing about the Kinect s software, is the very accurate and stable body tracking, which is a step forward compared to previous technology in the field. It allows the Kinect to recognize several people at the same time, and split them into specific body parts. This is achieved through locating the body parts based on local analysis of each pixel, which will represent the shape of the 3D area around the pixel. These shapes are then run through a decision forest, containing several decision trees. Each tree is trained to represent a specific body part. Once a pixel is run through this algorithm, it will return as the body part it has the largest probability to be a part of. Once the pixels have been assigned to a body part, the next step is suggesting joint positions. This process is illustrated in fig. 4.1 on the facing page, where the pixels from the depth image is assigned with the appropriate body part, which is colour coded in this figure, and from there the joint positions is suggested. 24

35 Chapter 4: Technology Analysis Figure 4.1: Kinect bodyparts Software Development Kit Shortly after the release of the Kinect for Xbox, people were immediately interested in the new technology. This resulted in a lot of hacks, which allowed to use the Kinect for a lot of various tasks. This resulted in Microsoft starting to develop a Software Development Kit, which opened up the use of the Kinect to everyone. The Software Development Kit (SDK) is integrated into Microsoft Visual Studio, which also integrates DirectX, and these combined makes it a functional tool for developing software for the Kinect [7]. 4.9 Summary Having examined the various types of technology, that could be used for physiotherapeutic purposes, we can make a decisive choice, regarding which technology we will use for this project. The intelligent elastic is an excellent example of a very specific piece of technology, 25

36 Part I: Problem Analysis that is used for a very specific type of physiotherapeutic assistance. While the elastic performs a specific operation very well, this focus on a specific type of operation (stretching the elastic) also means that it lacks the means to assist in a variety of ways. While a webcam can be used for motion tracking, this will require the aid of highly sophisticated software, as an average webcam does not receive any additional information, that can assist in accurately tracking the movement of the human body. Combined with the great differences, that may exist between various types of webcams and their output, which would make it highly difficult to ensure that any single webcam could be used for motion tracking. While the Wii, and to an even higher degree the Playstation move may be accurate, they are severely hindered by the fact they primarily track a handheld controller. The Kinect, on the other hand, tracks the user s entire body, and as such allows the programmer to use the user s entire body as input, as opposed to just the location and movement of a handheld controller. This allows for the design of programs, that are capable of tracking the human body s movement, which is an extremely useful tool for physiotherapy. This makes the Kinect the most useful technology for the purpose of physiotherapy, due to both the Kinect s capabilities, and the open SDK, which allows for easy development of software for the Kinect. 26

37 5 Scope of project In this chapter we will present, how the various aspects of this project, that have been analysed in the previous chapters, make up the scope of our project. The chapter will also include reasoning for this choice of scope, as well as a description of the base of our solution. 5.1 Kinect As mentioned in the summary of the technology chapter, we have chosen the Kinect for this project. The main reason why we are choosing the Kinect for our project is the SDK. The SDK contains great Application Programming Interface (API)s for analysing the output from the Kinect device. This will enable us to focus on making a working solution, rather than working on detecting a human. Another reason is the full body tracking that the Kinect is capable of. Compared to other devices, the Kinect has a more developed motion tracking system than some of the other systems that are currently available. The Kinect is also a viable choice, because it is cheap in comparison to the hardware it contains. This makes it an ideal choice in case the physiotherapist chooses to provide the devices themselves. 5.2 People and Stakeholders In this section, we will outlay, which of the respective groups of stakeholders, we will be considering in the development of our solution, as well as how their needs will be considered Patients We have analysed the patients and found that it is impossible to name a certain target group decided by who can use a solution based on motion based technology. The way that physiotherapy works, patients are in most cases grouped by the type of injury rather than age. This means that we have to make a decision on an injury group. 27

38 Part I: Problem Analysis People with specific types of injuries are also often assigned the same specific exercises to perform, and therefore, one of the most effective ways to narrow down the scope of our project, is to focus on a specific exercise. Through our interview we have found that shoulder abduction is a movement that should be easy to track and to correct. This means, that our target group, will initially consist of a people who are tasked with performing shoulder abductions Physiotherapists Much like the patients, the physiotherapist has a large interest in ensuring the effectiveness of the various exercises. This means that the physiotherapist will still be of interest to this project. The problems concerning distribution of the solution and the problems that stems from demographic changes will be discarded as focus points. However the physiotherapist will still be considered in the development of the solution due to his or her involvement with the patients and his expertise within the field Organisation As previously mentioned, we have decided to focus this project on the development of a solution for our problem, rather than the various logistical problems, that may arise in the implementation or distribution of such a solution. As such, we have decided not to focus on the economical aspects of the problem, as well as much of the interaction with various organisations, that would be necessary in order to implement the solution Motivation We have analysed the topic of motivation in the context of physiotherapy, and have found that motivation is a powerful supplement, which can ensure, that patients perform their assigned exercises. However, whilst age may be irrelevant to the exercises, that the patients are tasked with performing, this irrelevance does not apply to motivation. Any attempts at motivation in this project would not specifically have to be directed at a specific age group. We could pick any age group and therefore we have chosen not to pick a specific one. 28

39 Chapter 5: Scope of project Ideally, we would be able to provide intrinsic motivation for performing the exercises, these patients are assigned. However, ultimately motivation is not absolutely necessary for the project, and will therefore not be prioritized as highly in this project, as ensuring that our product is capable of recognizing exercises. 5.3 Problem Statement From what we have narrowed down, our problem consists of two major elements. We have chosen to attempt to assist people with physiotherapy through the use of technology, specifically the Microsoft Kinect, that is capable of motion-capture. Additionally, we have chosen to attempt to motivate people to do their exercises in order to ensure optimal effectiveness of exercises. While motivation may be a powerful supplement to ensure the effectiveness of the exercises, patients are assigned, they are not strictly necessary to the performance of the exercises. As such, the primary objective of this project will be to assist people with performance of the physiotherapeutic exercises. All that remains, is to state the final problem statement of this project: How can we assist patients with physiotherapeutic exercises at home through the use of motion-capture technology? This problem statement encapsulates the focus of our solution, and also describes, what the remainder of the project, the actual development of the solution, will be centred around. 29

40

41 6 Solution Specifications In this chapter, a number of criteria for our solution is specified, in order to serve as demands, that the product of this project must meet, for it to be regarded as a solution. These demands will be presented, and their importance explained, in order to further provide an overview of the criteria and expectations, for the product of this project. First the solution is split up into two groups. There are two different interest groups, one being the patients and the other being the physiotherapists. These two groups of people have different demands or different specifications that needs to be met. There are a few points which are key for the solution to work at all, these are: The solution must be capable of modelling the human body, and its movement. The solution must be capable of comparing various models of a moving human body. An interface, that will allow users to easily navigate the solution, and use its functions. Additionally the physiotherapist part must be capable of the following, Record a video that can then be used as a guideline for an exercise Tracking patients exercises The patients have different demands, listed here, A user experience in form of a user interface that can help motivate to further use Being able to choose an exercise or program to perform Being able to track how well exercises are done Being able to track how many repetitions are done or are left 31

42 Part I: Problem Analysis These criteria are the main demands, for the solutions functionality, in ensuring its ability to digitally assist patients with physiotherapy. Modelling will be achieved with the Microsoft Kinect, which will also act as a tool in achieving comparison and analysis of a moving human body. Also an interface will be necessary for the program to be a complete solution. This will ease the use of the solution, making it much more accessible to patients. 32

43 Part II Development 33

44

45 7 Theory All types of solutions are based around theories, that describe the problem, the solution aims to solve. Theories are assumptions based on collected data, that describe the world to the best of human knowledge. Using theoretical knowledge to describe a problem is typically done through modelling, using theory to create a virtual representation of the problem. This chapter will account for different types of theoretical knowledge, which has been deemed necessary to understand before a solution can be made. Mainly there will be explanations to the mathematics involved with modelling in 3D space. Linear algebra supports further theory in 3D modelling of movement and space, and 3D coordinate systems. There s also an account of what features the Kinect sensor and Kinect Software Development Kit have that are interesting to note, as they will help us in constructing our solution. 7.1 Modelling Movement, and other physical phenomena like it, occur in the real world. In order to represent these phenomena as data, that a computer is capable of processing, we must convert them according to some standard. This concept of converting something physical into something virtual, that is representative of it, is known as modelling. Modelling is typically done through a form of mathematics, that can be used to describe physical phenomena. Modelling allows computers to compute the representation of reality, giving developers the tools needed to solve problems related to the real world through the use of software Modelling of space Space can be modelled through various types of mathematics, that describe space, and the relationship between various objects in an area. Space can be modelled absolutely, or in relation to another factor, depending on the way you choose to represent it. 35

46 Part II: Development Relative versus absolute space For the purpose of the solution of this project, it is important to consider, whether it should be based around absolute or relative measurements of space. Absolute space will here be defined as space, that is interpreted exactly as it is viewed, without any alterations. In the context of this project, using absolute space would be relatively simple, as it would consist of allowing the solution to simply operate on the data, it received from the Kinect without any alterations to any of their aspects. In comparison, using relative space, which will be defined as space, that is interpreted in a specific way, would be more complicated, as it would require the data from the Kinect to be interpreted in a specific way, in order to establish, what our space is relative to. An example of relative space, that could be used in this project, could be splitting the data from the Kinect between the various joints in the body, and studying them by themselves, or in relation to their parent joints. Using absolute measurements of space would make less computer calculations needed for the solution, as the data from the Kinect would simply be used in the state they were obtained in. This would make the solution more efficient computationally, but this improvement would come at the price of flexibility. Operating on absolute measurements, the solution would not be capable of compensating for various oddities, such as rotation of the user, in the data from the Kinect, and as such, would require the user to maintain a relatively strict position and technique. On the contrary, using relative measurements of space would be more computationally intensive, but would allow the solution to compensate for some of the various oddities, that might arise in the execution of physiotherapeutic exercises. As such, we have decided to use relative measurements of space for this project, as it will give the user greater freedom in the performance of their exercises. This will impact the performance of our solution, but we consider the impact negligible, compared to the benefit, the user receives. 3D coordinate systems In this subsection, we will describe some of the basic functionality of threedimensional coordinate systems, as well as describing, how we they will apply to this project, through the modelling of space. In order to model movements in physical world in a mathematically based way, we need to be able to represent the three dimensions, that these movements 36

47 Chapter 7: Theory pass through. The three dimensions can be represented through the use of a three-dimensional coordinate system. A three-dimensional coordinate system is a Cartesian coordinate system with three separate axes, the length of which are measured in a constant unit, such as centimetres. These units represent the distance from the center of the coordinate system, known as the origin. As seen in fig The three axes are known as the x, y and z axes, and each span a plane in the threedimensional coordinate system, known as the coordinate planes. Any given point in the system, can be identified by a set of three coordinates corresponding to the axes, that denotes the points position along the axes. Figure 7.1: Three-dimensional cartesian coordinate system While the three dimensions of a coordinate allow for the representation of single points, it is also important, to be able to represent the distances between various points, such as the limbs between the joints of the human body. The distance between any two points in a coordinate system can be found in a multitude of ways. Given two points of the type P (x,y,z) and Q(x,y,z ), M(2,3,4) and N(1,2,3), subtracting the coordinates of the points gives the following result: M N = (2 1,3 2,4 3) = (1,1,1). This result describes a vector, indicating the distance along the various axes, one would have to travel, to get from point M to point N. As visualized in fig Subtracting the points in the reverse order would result in the distance one would have to travel from point N to M. While this would provide the amount of units, one would have to travel along the various axes, it does not provide the absolute amount of units between the two points. To obtain this number, one can use the Pythagorean theorem: The absolute distance between the points, M and N, is: MN = (2 1) 2 + (3 2) 2 + (4 3) 2 = = 3, Figure 7.2: The distance between two points 37

48 Part II: Development denoting the absolute amount of units, between the two points in the coordinate system. Given a specific point, P, and a specific distance, D, it is possible to calculate, whether a given point, Q, is within the distance of the original point. One simply needs to calculate, whether the distance between P, and the second point M P Q, is smaller than or equal to D. If the newly calculated distance is smaller than, or equal to the original distance, it is within the range of the original point. Visually, this can be represented by drawing a sphere in the coordinate system, with the original point in its centre, and a radius equal to the distance. Any point, whose distance to the original point is smaller than or equal to the distance, will be within or on the sphere, and therefore within the range D of point P. [38] Modelling of movement In order to develop a solution, we must first develop an understanding of, how we represent physical phenomena through various types of data and mathematical computations. When working with physiotherapeutic movements and exercises it is important to realise that these are done in the real world by a human being. Movements and exercises will vary depending on the person performing them and are limited by the human body. These concepts need to be modelled into something that can be worked out from a programmer s perspective. This means being able to model these movements done by the human body in the real world into something that can be seen in a mathematical way, something that can be analysed and computed. Modelling human motion can be done in various ways with the help of statistics, mathematics, data-analysis, machine learning, algorithms and computer picture analysis. All could be used in the general idea of using a monitor and the Kinect for physiotherapy. Depending on the complexity of the movement it can be difficult to predict the movement of the human body, also taking in mind that each person is different in size and ability to move. This makes it difficult to predict when a movement is done right or wrong and there can be doubt about what the optimal movement is. One can approach this problem in different ways, either using statistics, using averages or by setting different parameters in which the movement is considered correct or wrong. 38

49 Chapter 7: Theory It also needs to be acknowledged that not all joints of the human body are the same and there are limitations in certain joints, but this is where the Kinect SDK comes in handy as its calculations removes the various things that can go wrong when trying to read the movement in the human body. The Kinect SDK allows to tap into a predefined skeleton stream where the joints are already calculated in an absolute space, this deals with the problems related to the joint limitations. [31] The body will be approached with a skeleton or a robot like structure with joints and sticks or pins between the joints. It is not necessarily important to this project to know about the limitations of the body. It is mostly important to know the different joints correlations to each other, this means we can use the skeleton very well to do calculations such as angles of movement in joints and different calculations using vectors in a relative space. [11] Knowing about space and movement in coordinate systems in three dimensions will allow us to interpret a movement done by a person. It is possible to interpret points that represents joints in a space relative to the position of the body, this makes it possible to analyse the movement and change in position of these points, and thereby to represent movement. By comparing movements to a predefined reference, it can be used to detect right or wrong movements. Potentially each joint could have its own three dimensional coordinate system. But when considering simple movements such as a shoulder abduction it might not be necessary to go to such extends as it is possible to analyse the movement alone by calculating angles of movement in the joints involved in the movement. Another dimension, that will have to be represented in our solution, is time.time is essential to the correct execution of various physiotherapeutic exercises, as they can easily be performed either too rapidly or slowly. Any given movement can also be represented as a set of position as a function of time, the difference between each position representing the movement. For this projects solution, time will be managed by chronologically handling the data, that is received from the Kinect. How the Kinect acts as a tool The following is an account of some of the features that the Kinect and the Kinect Software Development Kit has available for us to use. The Kinect uses a feature known as skeletal tracking. It recognizes up to six people and tracks up to two players simultaneously and 20 joints for each player, with which it makes a skeleton. With active tracking, frames are captured and the 3D coordinates of the joints can be stored in meters. For the Kinect to track a person, it must be able to see their head and upper body. Skeletal tracking works both on 39

50 Part II: Development (a) UI notification[27] (b) Initializing Kinect[27] (c) Kinect is ready[27] Figure 7.3: Kinect status notifications standing or sitting, in the sitting mode, only the upper body joints are tracked. The user does not need a specific pose to be noticed by the Kinect sensor, the only other requirement is that the user must generally face the sensor to be tracked [28]. Skeletons are also accompanied by a floor-clipping-plane vector, which is made using an estimated floor plane. If the Kinect can not see the floor, then that vector is a zero-vector, this vector is estimated in each tracked frame [25].It is possible to access joint orientation in absolute space or through the hierarchical representation of the skeleton. The SDK for Kinect allows for calculations of bone orientation (of the virtual skeleton) as a rotation matrix or as a quaternion. For the skeleton a bone hierarchy is defined, where a bone is defined by it s parent joint and the child joint. The centre hip joint is the root of the skeleton, and extends through the various joints to end at the head, hands and feet. These bones are not explicitly defined but their rotations are in the child joints [26]. Applications that use the kinect usually include a Kinect Runtime setup to install dependencies and drivers. These include features that are useful for a good end-user experience. When you run a kinect enabled application, the notification from fig. 7.3a will be displayed if no Kinect is plugged in or the computer is unable to detect a plugged in Kinect. It will remain until a Kinect is detected. When a Kinect is discovered by Windows the notification from fig. 7.3b is displayed. After the Kinect and its drivers have been initialized, the notification in fig. 7.3c will be displayed and fade out. Afterwards the Kinect will be ready to be used by the application. This is useful feedback for the end-user to help show that they ve done the set-up correctly. 40

51 Chapter 7: Theory 7.2 Linear algebra Linear algebra is the branch of mathematics that deals with theory about systems of linear equations, matrices, vectors, vector spaces, and linear transformation. Some of these subjects, such as matrices and vectors, are highly relevant to determine the direction between two joints in our model. This section will take a look at the parts of linear algebra, that will be used in the solution of this project Vectors As mentioned in the section about modelling of space, the human body will be represented in a coordinate system, with points representing each joint. To be able to determine if an exercise is performed correctly, the direction between connected joints have to be known. This can be done by creating a 3d vector going from one joint to another. But first, what is a vector? A vector is a line segment running through a space with a starting and ending point. A 3d vector is u 1 usually represented as following: u u Figure 7.4: 3d Vector. 2 The vector u starts in origo and u 3 moves its length out on the X-, The line between these points is the length of the Y -, and Z-axis [44] vector, and the direction goes from the starting point to the ending point. This is illustrated in fig Each row represents how the vector moves on an axis, in this case, the first is X-axis, the second is Y -axis, and the third is Z-axis. The Microsoft Kinect provides information about the position of 20 joints, as illustrated in fig. 7.5 on the next page. With the knowledge of these joints, and how they are connected, it is possible to create vectors between them. The hip_center will be the root of the joint tree, and the rest of the joints will be connected to the hip_center. For instance the vector going from hip_center to spine, would be the spine-point subtracted by the hip_center-point. 41

52 Part II: Development Figure 7.5: Representation of the 20 joints provided by the Kinect [24] 42

53 Chapter 7: Theory Let: 0 hip_center = and spine = 4 0 then hip_center spine = 4 0 = Magnitude. The length of the line segment that defines a vector, is also called the vectors magnitude. This is always represented by a positive number, and the formula is as following [44]: u = u1 2 + u2 2 + u2 3. Example 7.1 (Calculate u and its magnitude.) 4 u = 3 = u = = 26 (7.1) 1 The magnitude of u is 26. Unit vector. A vector with a magnitude of 1 is called a unit vector, and this is what will be used to illustrate the position of a joint from its parent joint, see fig. 7.5 on the facing page. Any vector can be converted to a unit vector, by changing the magnitude to 1. This is called normalizing the vector, and is done as following: U u = u u. [44] Example 7.2 (Normalize u from previous example.) The magnitude was calculated to 26 which will be used. U u = (7.2) 43

54 Part II: Development To prove that U u is a unit vector, the magnitude is calculated again, and a result of 1 is expected. 4 U u = ( ) ( ) ( ) 2 = 1 (7.3) Unit vectors will be very important in ensuring that various people can perform the same exercise without having the same length of body parts. This can be done, since the important thing about doing the exercise is to maintain the correct position of a joint compared to another joint. For instance how is the elbow in relation to the shoulder. This relation can be described with a vector, which will show where the elbow is in relation to the shoulder. This is illustrated in fig. 7.6, where the shoulder point and elbow point is connected with the vector shoulder_elbow, which goes from the shoulder to the elbow. Since this relation can be described with a vector, we can normalize this vector into a unit vector, thereby ensuring that all the directions of the users joints can be compared to a normalized recording of the exercise. Figure 7.6: Vector going from shoulder to elbow Angle between two vectors. With all the vectors normalized, it will still be hard for a user to perform the movement in exactly the same way as the recorded exercise. To ensure that the exercises can still be done correctly without performing it to perfection, there is a need for a margin of error. Figure 7.7: Angle between two vectors This error could be measured in degrees, and this is where the angle between two vectors becomes interesting. Let us say that the user s vector is close to, but not equal to the recorded vector. This difference could in this example be 2. Now it is possible to set an allowed 44

55 Chapter 7: Theory error margin, and thereby determine if what the user is doing is corresponding to the recorded exercise. The angle between two vectors is measured as illustrated in fig. 7.7 on the facing page. The formula to calculate the angle between u and v is as following [45], where a is the angle in degrees: cos(a) = u 1 v 1 + u 2 v 2 + u 3 v 3 u1 2 + u2 2 + u2 3 v1 2 + v2 2 + v2 3 Which is the same as: a = cos u 1 1 v 1 + u 2 v 2 + u 3 v 3 u1 2 + u2 2 + u2 3 v1 2 + v2 2 + v2 3 Example 7.3 (Calculating the angle between vectors) Let: 3 8 u = 4 and v = (7.4) First the calculation of the dot product of u and v: u v = 3 ( 8) = 0 (7.5) Now the whole formula: ( ) a = cos ( 8) = cos 1 = 90 (7.6) So the angle between u and v is 90. Let the error margin be 10, and let the angle between current and recorded be 6. This would mean that the current vector is close enough to the recorded vector to be accepted as the same. This method will be used to determine when the user is doing something close enough to the recorded exercise. 45

56 Part II: Development Matrices A matrix is an array of numbers, as illustrated here: [ ] A = (7.7) Matrices can be noted by their size, where a matrix A can have the size m n and has a notation A m n. If we look at eq. (7.7) then we see that it has a size of 2 3. Matrices can be multiplied with each other, but only if the width of the first matrix, and the height of the second matrix are equivalent. This is why a 3 3-matrix can be used to transform/rotate a 3-dimensional vector.[37] e.g.: Example 7.4 (A transforming matrix) a 1,1 a 1,2 a 1,3 a 2,1 a 2,2 a 2,3 a 3,1 a 3,2 a 3,3 v x v y v z a 1,1 v x + a 1,2 v y + a 1,3 v z = a 2,1 v x + a 2,2 v y + a 2,3 v z a 3,1 v x + a 3,2 v y + a 3,3 v z (7.8) In the following one of the common uses of matrices in vector calculations, the rotation matrices, will be described. Rotation matrices In this solution we use matrices for rotating the vectors received for each joint. This is done in order to obtain comparable result, regardless of the rotation of the user and thereby creating a relative space. To rotate a vector we first need to find a rotation matrix, and then multiply it with the vector we wish to rotate. We previously went through the process of multiplying a matrix and a vector, so in the following text, we will primarily focus on finding the rotation matrix. To find a rotation matrix we use the general formula for determining a rotation around an axis. Let: 46

57 Chapter 7: Theory cosθ + u 2 x(1 cosθ) u x u y (1 cosθ) u z sinθ u x u z (1 cosθ) + u y sinθ R = u y u x (1 cosθ) + u z sinθ cosθ + uy 2 (1 cosθ) u y u z (1 1cosθ) u x sinθ u z u x (1 cosθ) u y sinθ u z u y (1 cosθ) + u x sinθ cosθ + uz 2 (1 cosθ) (7.9) [12] Where u is the unit vector u = (u x,u y,u z ). Since it is only necessary to rotate around the y axis in this solution, the equation can be optimized. Let the vector u be set to the vector u = (0,1,0), and fill it into the matrix in eq. (7.9): cosθ (1 cosθ) 0 1(1 cosθ) 0sinθ 0 0(1 cosθ) + 1sinθ R = 1 0(1 cosθ) + 0sinθ cosθ (1 cosθ) 1 0(1 1cosθ) 0sinθ 0 0(1 cosθ) 1sinθ 0 1(1 cosθ) + 0sinθ cosθ (1 cosθ) This can be reduced to: [37] cosθ 0 sinθ R = sinθ 0 cosθ An angle is needed to put this matrix to use, and this angle is calculated using the formula mentioned in example 7.3 on page

58

59 8 Solution In this chapter the different aspects of the solution development will be revised. There will be an explanation of general solution design ideas and how we use them. There will be data handling ethics and interface explanation. This chapter also includes the implementation where the theory is put to use, and lastly we will explain how we have achieved to motivate. This chapter is essentially the solution to the problem statement, which will then be tested and concluded upon. 8.1 Program design Program design is the plan of action when developing software. It consists of steps that should be done in the development process. When completed these steps will make the program easier to write and maintain. The waterfall program design which we have followed, can be described like this, Understanding the program through analysis Design phase using design tools. Implementation Testing Evaluation [15] Understanding the program When working on a software solution that has several programmers assigned to it, there is a need to create a mutual understanding of what the program needs to do. A documentation is created through an analysis. This has been done through the problem analysis in this project. Specifications are set from the result of this analysis. In general these often include layouts, descriptions, documentation of the process steps etc.[15] 49

60 Part II: Development Design phase Design tools could be things like pseudo-code or as in our case Model, View, Controller (MVC) and hierarchy, classes and methods. Object orientated programming also allows for design tools as class inheritance and polymorphism. It is expected that programmers are familiar with these different tools. In the end an information system is set up, which serves as a model of the solution. This way everyone has an understanding of how to approach programming. There will be further explanation on MVC and the class hierarchy used later. Also the approach used when handling data will be explained.[15] Implementation Implementation happens after the program has been analysed and the context understood. Implementation involves coding the solution and realising the classes and other design tools. Implementation will be explained in section 8.5 on page Testing In the waterfall approach development is usually done linearly, and therefore testing comes after implementation. Unlike other approaches such as test driven development where the tests are written before the code. Testing also has a chapter of its own (chapter 9 on page 73) and will therefore be explained later Evaluation Evaluation is done when the development process is finished. This phase is where the group evaluates, whether they have met the specifications that was set. The evaluation of this development process will be performed in part III on page Model, View, Controller The term Model, View, Controller describes a design pattern, used in the production of software. This pattern consists of a process, where the program is split into three separate components known as the model, the view, and the controller. The model contains the information, that the program is based around, which for this projects solution, would be the information received from the Kinect, as well 50

61 Chapter 8: Solution as the various ways it is manipulated. The model essentially handles most of the actual processing of the program, and is not dependent on any of the other parts of the program. The view is the part of the program that handles the output, and also handles any interfaces, the user is viewing. It allows for viewing of the data from the model, and allows the user to navigate the interface. The view is essentially the " front" of the program, that the user interacts with, and receives responses from. The controller handles input for the program. When a user interacts with the program, e.g. clicking a button, requesting information of some kind from the program, this activates the controller, which then passes the required commands along to the model, that handles the data. When the model has processed the requested data, the view is then updated, in order to display this information to the user. The controller itself is not dependent on any specific type of interface, which makes the interface easily replaceable. The strength of this design pattern is, that the three parts are separated, and as such, it emphasises code clarity. An educated examiner will always be able to tell, which part of the program a certain piece of code belongs to, as long as the program follows the pattern. Additionally, the separation between the different parts of the program ensures that it is relatively easy to test them[2]. 8.3 Data handling Before implementing the solution, it is important to understand how data is handled is handled by it. This refers to, how the input is managed and the form in which it comes, how loading of the data handled and how the programmer manages memory in C# Input It is important to realise that in this project there are different kinds of input, for example inputs from the Kinect, inputs in form button clicks in the Graphical User Interface (GUI) or input in form of XML files. The input from the Kinect comes in form of streams. There is the option in the Kinect SDK to access specific streams, and therefore it is possible to limit the data that needs to be handled. For example, it would be a waste of computing power and memory to manage the video stream from the Kinect, if the interesting information is really in the skeleton stream. The skeleton stream is the only stream that is accessed in this project and thus 51

62 Part II: Development this is the input that is received from the Kinect. The skeleton stream consists of positions of all joints that are in the view of the Kinect. A skeleton consists of 20 joints. The skeleton stream is handled in listing 8.1 in such a way that skeletons are loaded into an array of type skeleton. This skeleton can then be viewed with the method GetFirstSkeleton, as seen below. After this the first skeleton is retrieved and a frame class is created with new Exercise.Frame(skeleton); 1 void newsensor_skeletonframeready(object sender, SkeletonFrameReadyEventArgs e) 2 { 3 skeleton = GetFirstSkeleton(e); 4 5 if (skeleton == null) 6 { 7 return; 8 } 9 10 Exercise.Frame frame = new Exercise.Frame(skeleton); if (Recording) 13 { 14 ExerciseRecorder.AddFrame(frame); 15 } SkeletonStatus.Content = "We got a skeleton frame! Frames: " + ExerciseRecorder.frames.Count; 18 } Listing 8.1: Example of SkeletonFrameReady The frame class creates 19 vectors in its constructor, all based on the joint positions contained in the skeleton class. We now have a frame that contains all the information needed and a series of frames are loaded into an array, which is referred to as an exercise in the solution. This is only done when recording, which can be seen in the if (Recording) check listing 8.1, where the frame is added if the solution is recording. The exercise is loaded into an array instead of keeping it in a list, due to the fact that arrays have direct selections, whereas in a list you need to run through the list to find an element. The input from the Kinect can either be used directly from the frame class in the exercise, or saved to an XML file. When saved to the XML file, all vectors in every frame of the exercise is saved together with a list of focus vectors. The focus vectors are later used to determine which vectors to compare on, when comparing a recording and a live feed from the Kinect. An example of this can be seen in 52

63 Chapter 8: Solution listing <Exercise xmlns:xsi=" xmlns:xsd=" 2 <Name>Skulder Abduktion Højre</Name> 3 <Frames> 4 <Frame> 5 <framevectors> 6 <UnitVector3> 7 <x> </x> 8 <y> </y> 9 <z> </z> 10 </UnitVector3> 11 </framevectors> 12 </Frame> 13 </Frames> 14 <FocusVectors> 15 <VectorName>ShoulderRightToElbowRight</VectorName> 16 <VectorName>ElbowRightToWristRight</VectorName> 17 </FocusVectors> 18 </Exercise> Listing 8.2: Example of XML save file The XML file is also used as an input, as it is used in the part of the solution where you perform an exercise. The input file is chosen in the menu, where an exercise is chosen. The XML file corresponding to that exercise is then the input file that is compared on. The user interface also allows for inputs. There are simple inputs that fire events to change back and forth between pages, and these inputs are handled by the page switcher in listing 8.3. But there are also inputs that fire more complex events in the code. Some of these inputs are used to determine the name of files the user wants to save, such as listing 8.4. Others could for example be used to load the focus vectors into a list. 1 private void Button_Click_Optag(object sender, RoutedEventArgs e) 2 { 3 Switcher.Switch(new JointSelector()); 4 } Listing 8.3: Example of PageSwitcher 1 private void Gem_Click(object sender, RoutedEventArgs e) 2 { 3 string recordingname = RecordingName.Text; 53

64 Part II: Development 4 5 if (recordingname == "") 6 { 7 MessageBox.Show("Optagelse mangler navn"); 8 return; 9 } if (ExerciseRecorder.frames.Count == 0) 12 { 13 MessageBox.Show("Intet optaget"); 14 return; 15 } FileHandler handler = new FileHandler(recordingName); if 20 { (!handler.save(exerciserecorder.converttoexercise(recordingname))) 21 MessageBox.Show("Could not save " + recordingname + "!"); 22 } 23 else 24 { 25 ExerciseRecorder.frames.Clear(); 26 RecordingName.Text = ""; 27 } 28 } Listing 8.4: Example of save button Loading Loading can be considered in two ways, the loading that occurs when you open the solution and load, say for example the Kinect. It can also be considered strictly the loading that is done from files while the solution is running. The Kinect loading is handled by the KinectSensorChooser tool kit provided by Coding4Fun[6] which will not be explained further. The loading of the XML files is done by the solution s FileHandler class, that loads frames from the XML file and allocates space for them in an array and returns it as an exercise. 54

65 Chapter 8: Solution Memory management C# employs automatic memory management. This frees developers from manually allocating and freeing memory occupied by objects. Automatic memory management follows policies implemented by a garbage collector. The garbage collector follows certain rules about the memory management life cycle of an object.[23] These rules will not be further explained, as the information is uninteresting for the purpose of this section, the interesting part is that C# offers automatic memory management. While there is automatic memory management, there are still some considerations that have to be done when using data models. These considerations are not taken due to memory concerns but rather for the sake of computer performance. Like previously mentioned, an array could have certain pros compared to a list. For example when looking through an array or a list. So it is still important to be aware of what memory models to use. 8.4 Interface In this section the user interface and the ideas and choices behind the design will be explained. How the user interface is designed to accommodate the use of both a physiotherapist and a patient. The impact of the user interface, such as its effect on motivation will also be touched upon. Writing the user interface we have used the WPF tools in visual studio and used a Switcher interface adopted from another interface User interface A User Interface (UI) is the part of the solution, that allows the user to interact with the computer. This makes the UI everything that is designed into devices or machines that a human being can interact with, including things like computers, smart-phones, tablets and their software [36]. UIs have improved over time and have gone from command line input and report structures to GUIs. The GUI is a much more immersive UI that has, as the name indicates, a graphical part to it. This means that the GUI can be considered to include the whole user experience when using an application. This is where the user will give the application input and get output [8]. The idea behind a GUI is to make an interaction platform that makes it easy 55

66 Part II: Development or self-explanatory and seamless to use an application. The GUI is supposed to accommodate a wide spectre of people in all ages. The quote Keep it simple, Stupid is the idea behind the GUI of the application. To have as few modules as possible to make it easy to use, while still working fully with all of the functions. Ideally there should be no explanation needed to use the application due to the way the GUI is designed. A general idea of what the application does is needed, but once you know that the application is supposed to capture and help you with physiotherapeutic movements, you should be all set to go How the user interface motivates The GUI holds the user s experience as this is where interactions happens, therefore a good GUI can help in many ways. Motivation for example, which is of interest to this project, can be helped with a GUI. If the user finds the application easy and seamless and also finds the colours or tools to add an element of fun, a sense of motivation could be achieved. This way a well designed GUI could have a major impact on the application and how it will be used Our design choices Because we strive to have a feeling of motivation in our application we have chosen to have a GUI, which has video and a palette of colours implemented. The GUI has a blue background with buttons in the green colour typical of the XBOX, due to the Kinect s ties to the XBOX. We feel like this is an interesting mix of colours that could possibly add to the user experience. The application GUI also has very few buttons and only allow you to do necessary things and do not have any added features that are not directly needed when using the application. Ideas about having a game element have been thrown around, but will be mentioned in future works, and have in the current state not been implemented in the GUI. This results in a GUI that has two different components, one that aimed at recording new exercises and another one that is aimed at the performance of an exercise. The way this works is with a main menu, where the choice between recording and performing an exercise is presented. This is seen in fig. 8.1 on the facing page. Pressing the Optag (Record) button, causes the next page to be presented. This page allows for joint selection, and when this is done and Videre (Next) is pressed, 56

67 Chapter 8: Solution Figure 8.1: The main menu of the program Figure 8.2: The page where joint selection is done 57

68 Part II: Development Figure 8.3: The Kinect camera is setup for recording the page with the camera is opened. Here there are several options for recording and saving a new exercise. These pages can be seen in figs. 8.2 and 8.3 on the previous page and on this page. Pressing the Udfør (Perform) button causes a new page to be presented. On this page it is possible to select what exercise should be loaded, or in case exercise programs are developed, these can be chosen too. When an exercise is chosen, a new page is presented where exercises can be performed. This can be seen in figs. 8.4 and 8.5 on the next page. 8.5 Implementation In this section we will take a close look at the source code of the program and explain how we decided to deal with several of the problems related to the creation of the program. We will split this into several subsections with various relevant subjects for the program. 58

69 Chapter 8: Solution Figure 8.4: The page for exercise selection Figure 8.5: The Kinect camera is setup for exercise performance 59