MODULE-BASED REINFORCEMENT LEARNING - EXPERIMENTS WITH A REAL ROBOT

The behavior of reinforcement learning (RL) algorithms is best underst ood in completely observable, discrete-rime controlled Markov chains w ith finite state and action spaces. In contrast, robot-learning domain s are inherently continuous both in time and space, and moreover are p artially observable. Here we suggest a systematic approach to solve su ch problems in which the available qualitative and quantitative knowle dge is used to reduce the complexity of learning task. The steps of th e design process are to: i) decompose the task into subtasks using the qualitative knowledge at hand; ii) design local controllers to solve the subtasks using the available quantitative knowledge and iii) learn a coordination of these controllers by means of reinforcement learnin g. It is argued that the approach enables fast, semi-automatic, but st ill high quality robot-control as no fine-tuning of the local controll ers is needed. The approach was verified on a non-trivial real-life ro bot task. Several RL algorithms were compared by ANOVA and it was foun d that the model-based approach worked significantly better than the m odel-free approach. The learnt switching strategy performed comparably to a handcrafted version. Moreover, the learnt strategy seemed to exp loit certain properties of the environment which were not foreseen in advance, thus supporting the view that adaptive algorithms are advanta geous to non-adaptive ones in complex environments.