Yes; when used as a relative pronoun, that can be used to refer to both singular and plural nouns.
Both of the following are correct:
This is the car that belongs to me.
These are the cars that belong to me.
However, when used as a Demonstrative pronoun, that can only refer to singular nouns.
So, following is correct:
That car is mine.
But following is not:
That cars are mine.