For the beneficial leisure life of the elderly, the purpose of this study is to analyze the determinants of time use for social activities and cultural activities, and the relationship between the two activities. By classifying the elderly as “the semi-elderly”and“the currently-elderly,”this paper investigates more about the differences in activity behaviors of two groups. We use‘2014 Time Use Survey’data for the empirical analysis and a multivariate tobit model for the correlation between dependent variables. The main results of this study are as follows. First, we found that the determinants of social activities have similar results for both the semi-elderly and the currently-elderly. The time spent on social activities has been shown to decrease if there are family members in the household that need care and are currently working, indicating that these factors can affect individual time constraints. Second, the time spent gap between urban and non-urban areas appeared to arise for the semi-elderly people, so that cultural activities for them should seek to mitigate the gap between regions by enhancing physical accessibility. On the other hand, for the currently-elderly, the result showed that it is urgent for low-educated and low-income people to come up with diverse programs to gain access to cultural activities. Also, we found that the interrelationship between social and cultural activities shows to be a substitute for the currently-elderly.