We discuss the applicability of a fully convolutional network (FCN), which provides promising performance in semantic segmentation tasks, to plant segmentation tasks. The challenge lies in training the network with a small dataset because there are not many samples in plant image datasets, as compared to object image datasets such as ImageNet and PASCAL VOC datasets. The proposed method is inspired by transfer learning, but involves a two-step adaptation. In the first step, we apply transfer learning from a source domain that contains many objects with a large amount of labeled data to a major category in the plant domain. Then, in the second step, category adaptation is performed from the major category to a minor category with a few samples within the plant domain. With leaf segmentation challenge (LSC) dataset, the experimental results confirm the effectiveness of the proposed method such that F-measure criterion was, for instance, 0.953 for the A2 dataset, which was 0.355 higher than that of direct adaptation, and 0.527 higher than that of non-adaptation.