LOGIN TO YOUR ACCOUNT

Username
Password
Remember Me
Or use your Academic/Social account:

CREATE AN ACCOUNT

Or use your Academic/Social account:

Congratulations!

You have just completed your registration at OpenAire.

Before you can login to the site, you will need to activate your account. An e-mail will be sent to you with the proper instructions.

Important!

Please note that this site is currently undergoing Beta testing.
Any new content you create is not guaranteed to be present to the final version of the site upon release.

Thank you for your patience,
OpenAire Dev Team.

Close This Message

CREATE AN ACCOUNT

Name:
Username:
Password:
Verify Password:
E-mail:
Verify E-mail:
*All Fields Are Required.
Please Verify You Are Human:
fbtwitterlinkedinvimeoflicker grey 14rssslideshare1
Martin, Charles P.; Ellefsen, Kai Olav; Torresen, Jim (2018)
Languages: English
Types: Preprint
Subjects: Computer Science - Sound, Electrical Engineering and Systems Science - Audio and Speech Processing, Computer Science - Artificial Intelligence, Computer Science - Human-Computer Interaction, Computer Science - Neural and Evolutionary Computing
Musical performance requires prediction to operate instruments, to perform in groups and to improvise. We argue, with reference to a number of digital music instruments (DMIs), including two of our own, that predictive machine learning models can help interactive systems to understand their temporal context and ensemble behaviour. We also discuss how recent advances in deep learning highlight the role of prediction in DMIs, by allowing data-driven predictive models with a long memory of past states. We advocate for predictive musical interaction, where a predictive model is embedded in a musical interface, assisting users by predicting unknown states of musical processes. We propose a framework for characterising prediction as relating to the instrumental sound, ongoing musical process, or between members of an ensemble. Our framework shows that different musical interface design configurations lead to different types of prediction. We show that our framework accommodates deep generative models, as well as models for predicting gestural states, or other high-level musical information. We apply our framework to examples from our recent work and the literature, and discuss the benefits and challenges revealed by these systems as well as musical use-cases where prediction is a necessary component.
  • The results below are discovered through our pilot algorithms. Let us know how we are doing!

    • [1] N. S. Altman. An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician, 46(3):175-185, 1992. doi:10.1080/00031305.1992.10475879.
    • [2] C. Ames. Automated composition in retrospect: 1956-1986. Leonardo, 20(2):169-185, 1987. doi:10.2307/1578334.
    • [3] C. Ames. The Markov process as a compositional model: A survey and tutorial. Leonardo, 22(2):175-187, 1989. doi:10.2307/1575226.
    • [4] J. A. Biles. Improvizing with genetic algorithms: Genjam. In E. R. Miranda and J. A. Biles, editors, Evolutionary Computer Music, pages 137-169. Springer London, London, 2007. doi:10.1007/978-1-84628- 600-1_7.
    • [5] S.-J. Blakemore, D. M. Wolpert, and C. D. Frith. Central cancellation of self-produced tickle sensation. Nature Neuroscience, 1(7):635-640, 1998. doi:10.1038/2870.
    • [6] A. R. Brown and T. Gifford. Prediction and proactivity in real-time interactive music systems. Int. Workshop on Musical Metacreation, pages 35-39, 2013. URL: http://eprints.qut.edu.au/64500/.
    • [7] J.-P. Cáceres, R. Hamilton, D. Iyer, C. Chafe, and G. Wang. To the edge with China: Explorations in network performance. In ARTECH 2008: Proc. 4th Int. Conf. Digital Arts, pages 61-66, 2008.
    • [8] B. Caramiaux, N. Montecchio, A. Tanaka, and F. Bevilacqua. Adaptive gesture recognition with variation estimation for interactive systems. ACM Transactions on Interactive Intelligent Systems, 4(4):18:1-18:34, 2014. doi:10.1145/2643204.
    • [9] B. Caramiaux and A. Tanaka. Machine learning of musical gestures. In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME '13, pages 513-518, 2013. URL: http://nime. org/proceedings/2013/nime2013_84.pdf.
    • [10] K. Chakraborty, K. Mehrotra, C. K. Mohan, and S. Ranka. Forecasting the behavior of multivariate time series using neural networks. Neural networks, 5(6):961-970, 1992.
    • [11] B. A. Clegg, G. J. DiGirolamo, and S. W. Keele. Sequence learning. Trends in Cognitive Sciences, 2(8):275-281, 2017/11/28 1998. URL: http://dx.doi.org/10.1016/S1364-6613(98)01202-9, doi:10.1016/ S1364-6613(98)01202-9.
    • [12] S. Davies. Themes in the Philosophy of Music. Oxford University Press, Oxford, UK, 2005.
    • [13] D. Eck and J. Schmidhuber. Finding temporal structure in music: Blues improvisation with LSTM recurrent networks. In Proc. 12th IEEE Workshop on Neural Networks for Signal Processing, pages 747-756, 2002. doi:10.1109/NNSP.2002.1030094.
    • [14] J. Engel, C. Resnick, A. Roberts, S. Dieleman, D. Eck, K. Simonyan, and M. Norouzi. Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders. ArXiv e-prints, Apr. 2017. URL: https://arxiv.org/abs/ 1704.01279.
    • [15] R. Fiebrink, P. R. Cook, and D. Trueman. Human model evaluation in interactive supervised learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '11, pages 147-156, New York, NY, USA, 2011. ACM. doi:10.1145/1978942.1978965.
    • [16] R. Fiebrink, D. Trueman, and P. R. Cook. A meta-instrument for interactive, on-the-fly machine learning. In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME '09, pages 280-285, 2009. URL: http://www.nime.org/proceedings/2009/ nime2009_280.pdf.
    • [17] A. Graves. Generating Sequences With Recurrent Neural Networks. ArXiv e-prints, Aug. 2013. URL: https://arxiv.org/abs/1308.0850, arXiv:1308.0850.
    • [18] D. Ha and D. Eck. A Neural Representation of Sketch Drawings. ArXiv e-prints, Apr. 2017. URL: https://arxiv.org/abs/1704.03477.
    • [19] G. Hadjeres, F. Pachet, and F. Nielsen. DeepBach: a steerable model for Bach chorales generation. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1362- 1371, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL: http://proceedings.mlr.press/v70/hadjeres17a.html.
    • [20] T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer-Verlag New York, New York, USA, 2009. doi: 10.1007/978-0-387-84858-7.
    • [21] J. Hawkins and S. Blakeslee. On Intelligence. Times Books, New York, NY, USA, 2004.
    • [22] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997. doi:10.1162/neco.1997.9.8. 1735.
    • [23] N. Jaques, S. Gu, D. Bahdanau, J. M. Hernández-Lobato, R. E. Turner, and D. Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with KL-control. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1645- 1654, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL: http://proceedings.mlr.press/v70/jaques17a.html.
    • [24] Z. Jin, R. Oda, A. Finkelstein, and R. Fiebrink. Mallo: A distributed synchronized musical instrument designed for internet performance. In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME '15, pages 293-298, 2015. URL: http://www. nime.org/proceedings/2015/nime2015_223.pdf.
    • [26] I. Malik and C. H. Ek. Neural translation of musical style. ArXiv e-prints, Aug. 2017. URL: https://arxiv.org/abs/1708.03535.
    • [27] Y. Mann. Ai duet. Interactive Web Page, 2016. //aiexperiments.withgoogle.com/ai-duet.
    • [28] C. Martin, H. Gardner, and B. Swift. Tracking ensemble performance on touch-screens with gesture classification and transition matrices. In E. Berdahl and J. Allison, editors, Proceedings of the International Conference on New Interfaces for Musical Expression, NIME '15, pages 359-364, Baton Rouge, LA, USA, 2015. Louisiana State University. URL: http://www.nime.org/proceedings/2015/nime2015_242.pdf.
    • [29] C. P. Martin, K. O. Ellefsen, and J. Torresen. Deep models for ensemble touch-screen improvisation. In Proceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music Experiences, AM '17, pages 4:1-4:4, New York, NY, USA, 2017. ACM. doi:10.1145/3123514.3123556.
    • [30] C. P. Martin and J. Torresen. Exploring social mobile music with tiny touch-screen performances. In T. Lokki, J. Pätynen, and V. Välimäki, editors, Proceedings of the 14th Sound and Music Computing Conference, SMC '17, pages 175-180, Espoo, Finland, 2017. Aalto University. URL: http://smc2017.aalto.fi/media/materials/proceedings/SMC17_ p175.pdf.
    • [31] C. P. Martin and J. Torresen. RoboJam: A musical mixture density network for collaborative touchscreen interaction. ArXiv e-prints, Nov 2017. ArXiv e-prints arXiv:1711.10746 [cs.HC]. URL: http://arxiv.org/ abs/1711.10746.
    • [32] K. A. McMillen. Stage-worthy sensor bows for stringed instruments. In Proceedings of the International Conference on New Interfaces for Musical Expression, pages 347-348, Genoa, Italy, 2008. URL: http: //www.nime.org/proceedings/2008/nime2008_347.pdf.
    • [33] M. C. Mozer. Neural network music composition by prediction: Exploring the benefits of psychoacoustic constraints and multi-scale processing. Connection Science, 6(2-3):247-280, 1994. doi:10.1080/ 09540099408915726.
    • [34] R. Oda, A. Finkelstein, and R. Fiebrink. Towards note-level prediction for networked music performance. In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME '13, pages 94-97, 2013. URL: http://nime.org/proceedings/2013/nime2013_258. pdf.
    • [35] F. Pachet. The continuator: Musical interaction with style. Journal of New Music Research, 32(3):333-341, 2003. doi:10.1076/jnmr.32.3. 333.16861.
    • [36] L. S. Petroa and L. Mucklia. The brain's predictive prowess revealed in primary visual cortex. Proceedings of the National Academy of Sciences, 113(5), 2016. doi:10.1073/pnas.1523834113.
    • [37] M. Puckette. Something digital. Computer Music Journal, 15(4):65-69, 1991. doi:10.2307/3681075.
    • [38] S. Ross and N. C. Hansen. Dissociating prediction failure: Considerations from music perception. Journal of Neuroscience, 36(11):3103-3105, 2016. doi:10.1523/JNEUROSCI.0053-16.2016.
    • [39] R. Rowe. Interactive Music Systems: Machine Listening and Composing. The MIT Press, 1993. URL: https://wp.nyu.edu/robert_rowe/text/ interactive-music-systems-1993/.
    • [40] M. Sarkar and B. Vercoe. Recognition and prediction in a network music performance system for indian percussion. In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME '07, pages 317-320, 2007. doi:10.1145/1279740.1279809.
    • [41] R. K. Sawyer. Group creativity: Musical performance and collaboration. Psychology of Music, 34(2):148-165, 2006. doi:10.1177/ 0305735606061850.
    • [42] M. Schedel, P. Perry, and R. Fiebrink. Wekinating 000000swan: Using machine learning to create and control complex artistic systems. In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME '11, pages 453-456, 2011. URL: http://www.nime.org/proceedings/2011/nime2011_453.pdf.
    • [43] G. Schillaci, V. V. Hafner, and B. Lara. Exploration behaviors, body representations, and simulation processes for the development of cognition in artificial agents. Frontiers in Robotics and AI, 3:39, 2016. doi:10.3389/frobt.2016.00039.
    • [44] I. Simon, D. Morris, and S. Basu. Mysong: Automatic accompaniment generation for vocal melodies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '08, pages 725-734, New York, NY, USA, 2008. ACM. URL: http://doi.acm.org/10.1145/ 1357054.1357169, doi:10.1145/1357054.1357169.
    • [45] I. Simon and S. Oore. Performance rnn: Generating music with expressive timing and dynamics. https://magenta.tensorflow.org/performancernn, 2017.
    • [46] B. L. Sturm, J. F. Santos, O. Ben-Tal, and I. Korshunova. Music transcription modelling and composition using deep learning. In Proceedings of the 1st Conference on Computer Simulation of Musical Creativity, 2016. URL: http://arxiv.org/abs/1604.08723.
    • [47] R. Sun and C. L. Giles. Sequence learning: From recognition and prediction to sequential decision making. IEEE Intelligent Systems, 16(4):67-70, 2001. URL: http://dx.doi.org/10.1109/MIS.2001.1463065, doi:10.1109/MIS.2001.1463065.
    • [48] A. Tidemann. An artificial intelligence architecture for musical expressiveness that learns by imitation. In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME '11, pages 268-271, 2011.
    • [49] A. Tidemann, P. Öztürk, and Y. Demiris. A groovy virtual drumming agent. In Z. Ruttkay, M. Kipp, A. Nijholt, and H. H. Vilhjálmsson, editors, Intelligent Virtual Agents, IVA 2009, volume 5773 of Lecture Notes in Computer Science, pages 104-117. Springer Berlin Heidelberg, Berlin, Heidelberg, 2009. URL: http://dx.doi.org/10.1007/978-3-642- 04380-2_14, doi:10.1007/978-3-642-04380-2_14.
    • [50] A. Van, B. Caramiaux, and A. Tanaka. PiaF: A tool for augmented piano performance using gesture variation following. In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME '14, pages 167-170, 2014. URL: http://www.nime.org/ proceedings/2014/nime2014_511.pdf.
    • [51] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. ArXiv e-prints, abs/1609.03499, Sept. 2016. URL: http://arxiv.org/abs/1609.03499, arXiv:1609.03499.
    • [52] B. Webb. Neural mechanisms for prediction: Do insects have forward models?, 2004. doi:10.1016/j.tins.2004.03.004.
    • [53] D. M. Wolpert, R. C. Miall, and M. Kawato. Internal models in the cerebellum. Trends in Cognitive Sciences, 2(9):338-347, 1998. doi: 10.1016/S1364-6613(98)01221-2.
    • [54] B. Yuksel, D. Afergan, E. Peck, G. Griffin, L. Harrison, N. Chen, R. Chang, and R. Jacob. BRAAHMS: A novel adaptive musical interface based on users' cognitive state. In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME '15, pages 136-139, 2015. URL: http://www.nime.org/proceedings/2015/ nime2015_243.pdf.
  • No related research data.
  • No similar publications.

Share - Bookmark

Cite this article

Collected from