"Factors of -5 could be 1 and -5, which would satisfy the rule of 'factors of 1 and itself'
Is there something I'm forgetting?"
That's like saying factors of 20 could be 20 and 1 so 20 is prime.
Just remember that negative numbers can never be prime because the true definition of prime isn't a number with "factors of 1 and itself," but instead a number with "two distinct natural number divisors". A natural number is 1,2,3,4,5..... 0 and neg integers are not natural numbers.
So with -5 what are two unique positive integers other than 0 that can multiply and create -5? You can't do it with just two. The only way is 5*1*-1, which does not satisfy the rule of (1) only two factors and (2) the only two factors of the number must be natural.
This is why 1 is NOT prime. What two UNIQUE natural numbers can you multiply together and get 1? Well you might say 1*1, but they are not unique, they are the same number, so 1 is not prime.
My illustration in the previous post may not be sufficient. But hopefully this clarifies with a bit more definitional jargon.