Advancements in sensing and the Internet of Things (IoT) technologies generate a huge amount of data. Mobility on demand (MoD) service benefits from the availability of big data in the intelligent transportation system. Given the future travel demand or origin-destination (OD) flows prediction, service providers can pre-allocate unoccupied vehicles to the customers' origins of service to reduce waiting time. Traditional approaches on future travel demand and the OD flows predictions rely on statistical or machine learning methods. Inspired by deep learning techniques for image and video processing, through regarding localized travel demands as image pixels, a novel deep learning model called multi-scale convolutional long short-term memory network (MultiConvLSTM) is developed in this paper. Rather than using the traditional OD matrix which may lead to loss of geographical information, we propose a new data structure, called OD tensor to represent OD flows, and a manipulation method, called OD tensor permutation and matricization, is introduced to handle the high dimensionality features of OD tensor. MultiConvLSTM considers both temporal and spatial correlations to predict the future travel demand and OD flows. Experiments on real-world New York taxi data of around 400 million records are performed. Our results show that the MultiConvLSTM achieves the highest accuracy in both one-step and multiple-step predictions and it outperforms the existing methods for travel demand and OD flow predictions.