Recently, various smart home services such as smart air-conditioning, monitoring of elderly/kids and energy-efficient appliance operations are emerging, thanks to technologies of indoor positioning of users and recognition of Activity of Daily Living (ADL). Meanwhile, to realize more convenient home services, it will become more important to be able to predict occurrence time of each ADL. ADL prediction is a challenging problem since it is difficult to train a prediction model by general machine learning algorithms which use only the data at a moment for classification. In this paper, taking into account temporal dependency of data (consumed power of appliances and position of users) collected during daily life, we propose a method for constructing models to predict ADL with LSTM (Long Short-Term Memory). In the proposed method, we construct LSTM-based models by setting occurrence time of each activity to an objective variable. First, we tried to construct a multi-class classification model which outputs one of several predefined time ranges (time elapsed from present) as the occurrence time of the activity. Through preliminary experiment, we found that this model results in low accuracy in predicting the occurrence time. Then, as the second approach, we constructed a before-or-after classification model which judges if the activity occurs within a specified time or not. We applied this model to our smart home data and confirmed that it achieves better prediction accuracy for all activities.