In this paper a new algorithm is proposed for detecting and mitigating NLOS error in a TOA based wireless positioning system. Received signal strength (RSS) is employed to detect whether a TOA value contains NLOS error and mitigate it. When the relationship between TOA and RSS in LOS situation is known, a possibility of NLOS can be calculated using the TOA RSS measurement results which are provided by a base station. Then the TOA measurement results can be trimmed using the possibility mentioned above. Finally, the algorithm is implemented in a wireless positioning system. Experiment results show that the algorithm can mitigate the effect of NLOS.