To detect fatigue of a desk worker, this paper focuses on fatigue hidden in smiling and neutral faces and employs a periodic short time monitoring setting. In contrast to continual monitoring, the setting assumes that each short-time monitoring (in this paper, it is called a task) is conducted only during a break time. However, there are two problems: the small number of data in each task and the increasing number of tasks. To detect fatigue, the authors propose a method which is a combination of multi-task learning, clustering and anomaly detection. For the first problem, the authors employ multi-task learning which builds a specific classifier to each task efficiently by using information shared among tasks. Since clustering gathers similar tasks into a cluster, it mitigates the influence of the second problem. Experiments show that the proposed method exhibits a high performance in a long-term monitoring.