It is known that the tip clearance flow is dominant and very important flow phenomena in axial compressor aerodynamics because the tip clearance flow has a great influence on the stability as well as aerodynamic loss of compressors. Our goal is to clarify the behavior of tip clearance flow at near-stall condition in a transonic axial compressor rotor (NASA Rotor 37). In the present work, steady and unsteady RANS simulations were performed to investigate vortical flow structures and separated flow field near the tip for several different clearance cases. Boundary layer separation on the casing wall and blade suction surface was investigated in detail for near-stall and stall condition. In order to understand such complicated flow field, vortex cores were identified using the critical point theory and a topology of the three-dimensional separated and vortical flows was analyzed. In the nominal clearance case, the breakdown of tip leakage vortex has occurred at a near-stall operating condition because of the interaction of the vortex with the shock wave, leading to a large blockage and unsteadiness in the rotor tip. On the other hand, the calculation with no clearance suggested that the separation on the suction surface was different from that with the nominal clearance. Since the shock wave induced the boundary layer separation on the blade suction surface in the transonic axial compressor rotor, focal-type critical points appeared on the suction surface near the tip at near-stall condition.