Results from the piano tutor project

Results from the Piano Tutor Project1 Roger B. Dannenberg, Marta Sanchez, Annabelle Joseph, Robert Joseph, Ronald Saul, Peter Capell Carnegie Mellon University Pittsburgh, PA 15213 email: dannenberg@cs.cmu.edu ABSTRACT The Piano Tutor combines an expert system and multimedia technology to form an interactive piano teaching system. Important elements of the Piano Tutor are: (1) the use of score-following software to interpret student performances, (2) the use of extensive multimedia to create a natural dialog with the student, (3) an expert system to analyze student mistakes and give pertinent multimedia feedback, and (4) the use of Instructional Design theory to develop an extensive curriculum that can be tailored automatically to individual student needs. Results from a preliminary assessment of the Piano Tutor are presented. KEYWORDS: Intelligent Interactive, Accompaniment Tutor, Multimedia, Instructional Design, 1. Introduction The Piano Tutor Project began as an ambitious project to integrate multimedia [Blattner 92] and intelligent tutoring [Wenger 87, Sleeman 82] technologies with a sound and extensive piano instruction curriculum. Our goal, established by Drs. M. Sanchez and A. Joseph, was to use computers and multimedia technology to support piano instruction for beginners. After three years of concentrated effort, the project produced a working prototype that has achieved international recognition as the state-of-the-art in computer-based music instruction systems. Although the Piano Tutor research project has ended, a commercial implementation of the Piano Tutor and related technologies is now in progress. In a pilot study, the Piano Tutor was used to teach piano to a group of 18 subjects. This 1Published as: Dannenberg, Sanchez, Joseph, Joseph, Saul, and Capell, ‘‘Results from the Piano Tutor Project,’’ in Proceedings of the Fourth Biennial Arts and Technology Symposium, Connecticut College, (March 1993), pp. 143-150. study confirmed our belief that a computer-based instruction system could be very effective in teaching a substantial body of musical knowledge. Our subjects enjoyed using the Piano Tutor, they learned at a rapid pace, and they could transfer their new skills to sight-reading and performance tasks in standard piano instruction books. There have been at least four noteworthy achievements. The first is the application of scorefollowing technology in an instructional system. The Piano Tutor can follow a student’s (polyphonic) piano performance [Bloch 85, Dannenberg 88], turn pages automatically, and synchronize an accompaniment. The ability to follow student performances is important to understanding, analyzing, and interacting with students. Automatic page turning is an important technique for presenting music to students via computer screen, and accompaniment gives students ensemble experience. The second achievement is the integration of an extensive multimedia presentation system to create a natural dialog with the student. [Dannenberg 92] The Piano Tutor uses a combination of video, music notation, digitized voice, synthesized music, and graphics, all controlled by an expert system in response to student performances. Multimedia allows the Piano Tutor to effectively convey the visual, aural, and basic technical skills necessary for piano performance. Our third achievement is the development of an expert system component for the Piano Tutor. [Dannenberg 90a, Dannenberg 90b] The expert system analyzes student performances and gives pertinent feedback to guide the student. Initially, the student is encouraged to correct mistakes by trying again, but if this fails, the system automatically proposes simpler tasks that help the student concentrate on the source of difficulty. If problems persist, the system assumes that the the student needs to strengthen previously learned skills. Again, the system identifies relevant tasks for the student. The fourth achievement is a contribution to Instructional Design theory. We applied Instructional Design concepts to develop the extensive Piano Tutor curriculum. We found Instructional Design to be helpful in providing general principles and guidelines, but we needed much stronger analytic tools to develop the Piano Tutor curriculum. We invented a formal representation for curricula, and we developed computer programs to analyze these curricula for a broad range of properties. This new approach to curriculum design has farreaching implications for future computer-based instruction systems as well as for textbooks and hypermedia systems. [Capell 93] The Piano Tutor is closely related to a number of other intelligent tutoring systems [Sleeman 82, Anderson 89], but intelligent tutoring systems rarely incorporate multimedia. Two exceptions are a code inspection system [Stevens 89] and a CPR instruction system [Hon 82]. Only the CPR system teaches a psychomotor skill as does the Piano Tutor. A distinguishing feature of the Piano Tutor is its large number of lessons and its ability to assemble lessons into an individually tailored curriculum. Previous music tutoring systems have found some success in the areas of ear-training and theory, but performance-oriented instruction software has often suffered from inadequate or inflexible interfaces [Hofstetter 88]. Another effort to use score following software with students is the Artificially Intelligent Computer Performer (AICP) at Connecticut College [Zahler 91]. 2 The next section describes the Piano Tutor, and the following sections present in greater detail what we feel are our major achievements. The last section outlines other ‘‘spinoffs’’ of this research. 2. The Piano Tutor The Piano Tutor teaches beginners to play the piano. It consists of an electronic piano with a computer interface, a computer and monitor, a videodisc player and second monitor, and an audio mixer to combine piano sounds, digitized voice from the computer, and sound from the videodisc. The Piano Tutor assumes no initial musical knowledge, so the Piano Tutor must teach concepts such as notation, pitch, rhythm, and basic music structure in addition to piano performance skills. The basic approach of the Piano Tutor is to teach in units of short lessons that address one to a few concepts each. A lesson begins with a multimedia presentation to introduce the concept. Then, the Tutor asks the student to perform a task (usually a piece of music) that demonstrates mastery of the new concept. The Piano Tutor monitors the student’s performance and offers suggestions if the student makes mistakes. When the new concept is mastered, the Piano Tutor selects a new lesson and the teaching process continues. The Piano Tutor is a large project with many components. The greatest value of the project may be its successful integration of these components. In fact, one interesting aspect of the Piano Tutor is how well this complicated engine is hidden ‘‘under the hood’’ of the exterior interface. Nevertheless, we will describe the major components separately in the following sections. 3. The Human-Computer Interface We set out to make the Piano Tutor emulate human teachers, and this required the development of new interface techniques. We were attracted to the idea that a video could make the system more ‘‘friendly’’ and supportive of students. In addition, we anticipated that students would find it easier to learn the psychomotor skills needed for playing by seeing them demonstrated on the video by a real teacher. One pleasant surprise was that we could augment video with digitally recorded speech. This greatly extended our limited 30 minutes of video and still preserved the human touch we were after. Ultimately, we supplemented our presentations with graphics, computer generated music, and music notation. These helped us to communicate effectively with students. Another aspect of emulating teachers is that the system must take an active role in teaching. Many educational systems have been little more than active books, and we see this approach repeated in many hypermedia systems that leave navigation up to the student. In contrast, we feel that a primary function of the teacher is to guide the student toward instructional material and exercises. This is a difficult task for computers, but we have developed a number of techniques to support this goal, and we have demonstrated that they work effectively. A key to the kind of active instruction provided by the Piano Tutor is the ability to analyze student performances. This allows the system to build a model of what the student knows and 3 does not know, and that in turn guides the selection of new material. This analysis process is discussed in greater detail below. In addition, the curriculum must be organized according to some formal principles in order to simplify the search for appropriate lesson materials. This organization is discussed in Section 4. The Piano Tutor has succeeded in demonstrating that intelligent systems can play an active role in guiding students. Our effort of emulating teachers was given a considerable boost by the prior existence of real-time score-following software. Score following refers to the ability of a computer to track or follow a musical score while it is being performed by a student. The analogy to speech would be following a written text while listening to the spoken version. Score following is an essential step in listening to and understanding a student performance. As in a lesson with a human teacher, it allows the student to set his or her own tempo, leaving enough room to make minor timing or pitch errors without confusing the computer analysis software. Another function of score following allows for the synchronization of accompaniment. One of the teaching techniques incorporated into the system is to provide accompaniments for the pieces presented in each lesson to enhance the auditory skills of the students, to help them with rhythm, phrasing, and dynamics, and to give them the experience of ensemble playing. Score following also allows us to turn pages of music on the computer screen automatically. Again, this enhances our ability to provide a natural human-computer interface. 4. Instructional Design Our goal from the start was to design a system that could emulate the student/teacher relationship that exists in a one-on-one music lesson. This meant that we had to create a system capable of adapting to the needs of each student. The problem was not to design a single curriculum, but to build an intelligent system that could design a new curriculum for each student. Therefore, our task was to • Design a set of component lessons with explicit relationships to one another, so that lessons could be assembled into a curriculum, and • Develop and automate strategies for assembling a curriculum from component lessons. We feel the result was a breakthrough in Instructional Design. We developed a formal representation for lessons in which every lesson has a list of prerequisites and objectives. The student’s state of knowledge is modeled as a set of skills. When the student masters a new skill, the skill is added to the student’s skill set. Consistent with instructional design principles, a student is qualified to take a lesson when the prerequisites of the lesson are all contained in the student’s skill set. If the student is successful in taking the lesson (lessons have behaviorally described evaluation criteria), then the objectives of the lesson are considered to be mastered. They are therefore added to the student’s acquired set of skills. The strength of this model lies in the fact that computers can simulate a student’s progress through the curriculum. Computers can also analyze the network of lessons to determine whether certain desirable properties exist. For example, every skill should be taught by at least one lesson. Every lesson should teach something the student does not already know. There 4 should be a way, via a sequence of lessons, to learn every skill. With over 100 lessons in the Piano Tutor, these maxims would be extremely difficult to check by hand. We developed techniques whereby computers can check for these and many other useful properties automatically in minutes. When errors are found, it is a relatively simple matter to revise the design since no time has been invested in scripting, media production, or programming. These tasks are performed only after a complete design has been validated by computer. This formalized approach to curriculum design is not at all limited to the Piano Tutor, nor is it limited to intelligent teaching systems. We have discussed these ideas with other educators at Carnegie Mellon and found these ideas seem to formalize existing ideas of how computers might help teachers. One professor is interested in computer-assisted textbook authoring, and is hoping to apply our ideas. It also seems that hypermedia and hypertext systems, for which effective choices of links are an oft-cited problem [ACM 89], could benefit from the formal approach that we offer. 5. The Expert System The Piano Tutor is by far the most sophisticated music-teaching computer system known to us. To a great extent, the capabilities of the Piano Tutor are a result of its expert system component, which analyzes student performances and chooses how to proceed. In essence, this is where teaching knowledge is utilized. Some key aspects of the expert system are that it works in close conjunction with our curriculum design, and it operates at two levels: within lessons and outside of lessons. An important design decision in the Piano Tutor was to factor the curriculum into a global and a local level. At the global level, the problem is to select a lesson from a large set of possibilities. At the local level, the problem is to deliver the lesson, including specific suggestions in response to student performances. We found this two-level structure to work quite well. As noted above, the global level is more or less independent of the subject area, so much of our work here is relevant to other domains. Work at the global level focuses on curriculum design, and by dealing with fairly abstract lessons, we can more easily address curriculum planning issues. Within lessons, interaction tends to be much more focused. It has been well-known since the beginnings of artificial intelligence research that restricted domains are much easier to deal with than unrestricted ones. In the case of the Piano Tutor, the performance task for the student within a lesson is very specific, so the Piano Tutor can respond with very specific and helpful comments. One of the liabilities of a two-level scheme is that it may be difficult to notice patterns that span multiple lessons. To solve this problem, we attempt to update the student model with information that might be useful across lessons. For example, if a student has problems with rhythm, a corresponding skill might be decremented by a lesson to indicate that a potential problem exists. If several lessons do this, the cumulative effect will be to communicate to the expert system that more work is needed on rhythm. 5 6. Results We conducted a pilot study using students, staff, and faculty from the Carnegie Mellon community. The purpose of this study was not to perform a controlled objective experiment which would require a large student population and very careful design. Instead, we hoped to characterize the Piano Tutor as an instructional system: do students like using the system?, do they learn from it?, are aspects of the system confusing?, are there components missing?, do students need human assistance?, and so on. We started with just a few students and built up to a total of eighteen (18). Ten (10) students completed the Piano Tutor curriculum, and eight (8) withdrew, mostly for reasons of personal schedules such as academic pressures. We have collected data in several ways. The Piano Tutor automatically keeps a log of each student, so we know what lessons were taken and how much time was spent on each. We also asked students to fill out an evaluation form after their initial exposure to the Piano Tutor. Finally, we have tested students with standard piano instruction material to see if the knowledge and skills learned through the Piano Tutor are transferable to current curricula. The logs support our hypothesis that a system could automatically tailor a curriculum to the needs of each student. Of the 10 students completing the curriculum, the time-to-completion ranged from 17 calendar days to 159 days, and from 3.7 on-line hours to 21 hours. These figures show that quick students or students with some background are not held back by a rigid or inappropriate curriculum. One might argue that the quick students simply play everything correct on the first try and therefore take less time. However, our logs also show that the system gave more lessons to the slower students. For example, the number of presented music performance lessons ranges from 36 to 53. Clearly, the Piano Tutor is varying the lesson content according to the ability of the student. Our evaluation forms are more subjective but perhaps more useful. We received a number of constructive comments that can be applied to future systems. Overall, the forms are very positive. Here are some responses to the questions, ‘‘Do you enjoy working with the Piano Tutor? Are you learning what you want to learn?’’ • ‘‘Yes! Yes!’’ • ‘‘I appreciate the self-paced aspect and the fact that a computer is judging my progress rather than a person.’’ • ‘‘Yes. I had never learned Bass Clef before.’’ • ‘‘Very much! [and to the second question:] Not sure. It is teaching me to play the piano. (But the true judge would be a human teacher.)’’ • ‘‘I enjoyed it very much but I’d like to learn more.’’ • ‘‘I think it is fun working with the Tutor. The hardest thing for me is to learn how to keep time. The Tutor gets me on that all the time. So, yes, I am learning what I want and need to learn.’’ Our survey shows that students enjoy the style of teaching embodied in the Piano Tutor and they feel it is a good way to learn. 6 Every student spent a small amount of time with human teachers to assess their progress and supplement the Piano Tutor by checking on posture, ease of movement, and other physical components of piano playing. In general, we found students were learning rapidly, and they were able to apply their knowledge and skills in sight-reading tasks outside the Piano Tutor curriculum. One of the surprises was that students seem to have better than average ability to keep time and play in rhythm. We believe the constant feedback from the Piano Tutor promotes good practice habits, as illustrated by the last student comment listed above. In summary, we set out to evaluate the Piano Tutor by testing it with a small pool of students. As expected, we found many small problems and we fixed many of them by augmenting the curriculum when it was found lacking. Overall, though, the testing went very smoothly. We found no serious problems with our approach, and we were gratified that students were so positive about the system. To a great extent, the numbers speak for themselves: most of our students stayed with the system to the end. Those that finished mastered what would normally be considered at least a one-year curriculum. The average time was 69 calendar days and less than 20 on-line hours. 7. The Future There are now only two operational Piano Tutor systems. We intend to keep these systems functional on the Carnegie Mellon campus for the foreseeable future, and we welcome visitors to try the system for themselves. We expect the mere existence of these systems will inspire and inform a great deal of future work in music education and intelligent tutoring. We are hopeful that a commercial version of the Piano Tutor will make these ideas even more available. Meanwhile, the organization of the Piano Tutor seems applicable to other domains. We are working toward a framework for general intelligent multimedia instruction based on the system architecture that we created for the Piano Tutor. 8. Conclusion The Piano Tutor Project established a new plateau of excellence in computer-based music instruction. Having received world-wide recognition, we feel confident that we have had an impact on the designers of future educational systems. The main contributions of the Piano Tutor include the ideas of active, intelligent, multimedia instruction, and formal support for curriculum design. 9. Acknowledgments We are extremely indebted to the Markle Foundation for support, which made all of this possible. The Piano Tutor project began in the Center for Art and Technology in the College of Fine Arts, Carnegie Mellon University. The Center is now the Studio for Creative Inquiry. 7 References [ACM 89] ACM. Hypertext ’89 Proceedings. ACM, 1989. [Anderson 89] Anderson, J. R., F. G. Conrad, and A. T. Corbett. Skill acquisition and the LISP TUTOR. Cognitive Science 13(4):467-505, Oct-Dec, 1989. [Blattner 92] Blattner, M. M. and R. B. Dannenberg (editors). Multimedia Interface Design. ACM Press, 1992. [Bloch 85] Bloch, J. J. and R. B. Dannenberg. Real-Time Computer Accompaniment of Keyboard Performances. In B. Truax (editor), Proceedings of the International Computer Music Conference 1985, pages 279-290. International Computer Music Association, 1985. [Capell 93] Capell, P. and R. B. Dannenberg. Instructional Design and Intelligent Tutoring: Theory and the Precision of Design. Journal of Artificial Intelligence in Education 4(1):95-121, 1993. [Dannenberg 88] Dannenberg, R. B. and H. Mukaino. New Techniques for Enhanced Quality of Computer Accompaniment. In C. Lischka and J. Fritsch (editor), Proceedings of the 14th International Computer Music Conference, pages 243-249. International Computer Music Association, San Francisco, 1988. [Dannenberg 90a] Dannenberg, R. B., M. Sanchez, A. Joseph, P. Capell, R. Joseph, and R. Saul. A Computer-Based Multi-Media Tutor for Beginning Piano Students. Interface 19(2-3):155-73, 1990. [Dannenberg 90b] Dannenberg, R. B. An Expert System for Teaching Piano to Novices. In S. Arnold and G. Hair (editor), ICMC Glasgow 1990 Proceedings, pages 20-23. International Computer Music Association, San Francisco, 1990. [Dannenberg 92] Dannenberg, R. B. and R. L. Joseph. Human-Computer Interaction in the Piano Tutor. Multimedia Interface Design. In Blattner, M. M. and R. B. Dannenberg, ACM Press, 1992, pages 65-78. [Hofstetter 88] Hofstetter, F. T. Computer Literacy for Musicians. Prentice Hall, Englewood Cliffs, NJ, 1988. [Hon 82] Hon, D. Interactive Training in Cardiopulmonary Resuscitation. Byte Magazine 7(6), June, 1982. [Sleeman 82] Sleeman, D. and J. S. Brown (eds.). Intelligent Tutoring Systems. Academic Press, New York, 1982. [Stevens 89] Stevens, S. M. Intelligent Interactive Video Simulation of a Code Inspection. Communications of the ACM 32(7):832-43, July, 1989. [Wenger 87] Wenger, E. Artificial Intelligence and Tutoring Systems. Morgan Kaufmann Publishers, Inc., 1987. [Zahler 91] Zahler, N. CEDS TO GROW A CAT: A Progress/Studio Report. In B. Alphonce and B. Pennycook (editor), ICMC Montreal 1991 Proceedings, pages 455-458. International Computer Music Association, 1991. 8