LOT Winter School 2019

Multimodal language learning and social robots

Paul Vogt



Paul Vogt



Postal address

PO Box 90153

5000 LE Tilburg

The Netherlands



Course title

Multimodal language learning and social robots

Course level (e.g. introductory, intermediate, advanced)


Course description

Social robots (and other forms of socially intelligent agents) are entering our society and they need to communicate with us humans in a multimodal manner. In the first part of the course, I will present studies that investigate what sorts of mechanisms are necessary for robots to learn language and gesture from interacting with humans. In the second part, I will present recent studies investigating how robots can support children learning a second language. For both parts, we will discuss opportunities and limitations of these approaches for both the science and technology.

Day-to-day program

Monday: Issues in social symbol grounding

Tuesday: Robots learning language

Wednesday: Second language tutoring using social robots

Thursday: More on second language tutoring using social robots

Friday: Robots’ gesture learning

Reading list

Course readings:

Lecture 1:

Vogt, P. and Divina, F. (2007) Social symbol grounding and language evolution. Interaction Studies 8(1): 31-52.

Vogt, P. (2012). Exploring the robustness of cross-situational learning under Zipfian distributions. Cognitive Science 36(4): 726-739

(Optional: Aussems, S. and Vogt, P. (in press) Adults use cross-situational statistics for word learning in a conservative way. IEEE Transactions on Cognitive and Developmental Systems.)

Lecture 2:

Twomey, K. E., Morse, A. F., Cangelosi, A., & Horst, J. S. (2016). Children’s referent selection and word learning: insights from a developmental robotic system. Interaction Studies, 17(1), 93-119.

Vogt, P. and Mastin, J. D. (2013) Anchoring social symbol grounding in children's interactions. Künstlichen Intelligenz, 27(2): 145-151

(Optional: Mastin, J.D. and Vogt, P. (2016) Infant engagement and early vocabulary development: A naturalistic observation study of Mozambican infants from 1;1 to 2;1. Journal of Child Language, 43(2): 235-264)

Lecture 3:

Kanero, J., Geçkin, V., Oranç, C., Mamus, E., Küntay, A. C., & Göksun, T. (2018). Social Robots for Early Language Learning: Current Evidence and Future Directions. Child Development Perspectives.

Vogt, P., de Haas, M., de Jong, C., Baxter, P., and Krahmer, E. (2017). Child-robot interactions for second language tutoring to preschool children. Frontiers in Human Neuroscience. 11:73.

Lecture 4:

Belpaeme, T., Vogt, P., van den Berghe, R., Bergmann, K., Göskun, T., de Haas, M. et al. (2018). Guidelines for designing social robots as second language tutors. International Journal of Social Robotics, 10(3): 325-341

De Wit, J., Schodde, T., Willemsen, B., Bergmann, K., de Haas, M., Kopp, S., Krahmer, E.J., & Vogt, P. (2018). The Effect of a Robot’s Gestures and Adaptive Tutoring on Children’s Acquisition of Second Language Vocabularies. In Proceedings of the 13th Annual ACM International Conference on Human-Robot Interaction (HRI’18).

Lecture 5:

Cabrera, M. E., & Wachs, J. P. (2017). A human-centered approach to one-shot gesture learning. Frontiers in Robotics and AI, 4, 8