Learning is one of the most important abilities of intelligent adaptive agents. The generalization capability and training efficiency of learning algorithms depend heavily on the abstract representations acquired. Planning, on the other hand, allows agents to anticipate the future consequences of their actions so as to act optimally at the now. The action-contingent predictive features generated by planning modules thereby provide a good abstract representation constituting the current state of the agent. From this insight, this thesis aims to integrate trainable planning modules for data-efficient learning in sequential decision making and manipulation problems, ranging from Go game to real-world robotic AI. Specifically, this thesis will investigate the effectiveness of such approach by trying to solve the key questions of (1) how to integrate planning modules into deep learning frameworks so as to train the whole system from data, and (2) how to exploit predictive, but possibly inaccurate, abstract features from planning modules to guide the learning process. The main contributions of this thesis are to answer these questions within a broad literature survey and incorporate the ideas in an algorithm that can be applied to learn to plan in visual navigation tasks in a completely unsupervised manner.