Well, the square root of -1 makes sense: it makes every polynomial of degree n have n roots. It also makes a lot of physics calculations that get verifiable results possible (so long as you throw away the "imaginary" part at the end). It also simplifies a lot of things. It's not mathematically indefensible. It just completes some things.
There are, however, other problems, like two kinds of infinities. The first is a countable infinity, that you get if you count from 1 to the end of time. (An example is "Hilbert's Hotel, which has an infinite number of rooms. There's always room for one more--every guest just moves to the next room number up and the newbie goes into room 1.) The other is an uncountable infinity, with more "members" than can be labeled with an integer (you can prove it!). The first is the rational numbers. The second is the real numbers. And, as Feynman, said, there are "more numbers than numbers": 1, 2, 3, ... has just as many members as 3,6.9.... Both are countable, and both are infinite, but somehow one is smaller than the other. Then there is Cantor's "middle thirds" set. Start with numbers 0 or greater and 1 or less. Remove all the numbers after 1/3 and before 2/3. From what is left, remove numbers greater than 1/9 and less than 2/9, and greater than 7/9 and less than 8/9. Keep going, and you get an infinite number of members which can't be matched up with the counting numbers. (This is kind of like Xeno's turtle which starts ahead of Apollo and wins the race because Apollo requires x time to make up half the distance, x/4 to cover half the remaining distance, then x/8 to cover half the remainder, etc., and never catches up. Also, there is Bertrand Russel's "set of all sets which aren't members of themselves". Is it a member of itself? If it isn't, it's in the set of all sets which aren't members of themselves and therefore is a member of itself, so it is. If it is, then it's not in the set of all sets which aren't members of itself, so it isn't.
All this bothers some people, but one book I have says "so what? it works.".
There is obviously something wrong with our heads, with how we perceive things. But then, we imagine all kinds of magic stuff that isn't real, so maybe that's not so surprising. It's a fun thing to do in retirement, I guess. It's like the lunatic in Dana Sobel's book "Longitude" who does endless longitude astronomical calculations on the asylum walls before sea captains had good clocks. Or the guy who had "A Beautiful Mind" but needed some help to get back to reality.