Maybe it's the fiber. If the fiber being spun is harsh, or has a harsh part to it, then the resulting yarn will show up that harshness. Kemp, or hair fibers left in Navajo Churro, for example, or the britch of some breeds like Jacob can be fairly harsh and will make the resulting yarn harsh. Even a little bit of kemp or britch will have a big effect in the final feel. (That's a friend's Navajo Churro in the picture.)
How fine are you spinning your yarn? I've found that even soft fibers, spun finely, can seem harsh, since fine spinning needs more twist to hold together. The thicker the yarn, the less twist it needs to hold together.
Also -- are you spinning singles or plying? If you are spinning singles, consider plying -- the plying twist allows the singles to relax back out a bit, since it removes twist from the singles for the ply.
Maybe it's overtwist. In singles, you only want to put just enough twist that the fiber will hold together as yarn. It's tough to get this right; I do a tug test alot until I get a good feel for how much twist I'm putting in, and I continue to "tug test" all the way through, at semi-random intervals.
If you can't get less twist in your singles, but want that thickness, consider plying with thread -- there are cotton, silk, and nylon threads out there in a rainbow of colors.
If you are plying your yarn already, check the amount of plying twist, see if it is balanced, after the washing step -- perhaps it's overplied or underplied? (If the whole skein twists "S", then it is underplied, if the whole skein twists "Z" then it is overplied. Two or more full twists, that is -- less than that is close enough, for me anyway). For more on plying twist, see here.
If it's over or under plied you can quickly run it through a wheel (if you have one) even on a spindle it's not too onerous to add or remove plying twist to "fix" a skein. Rewash and all that after fixing the twist.
Perhaps the fiber needs washing and "whacking" to bloom and soften it a bit. Some fibers feel harsh until they are beaten on a counter to help them relax in the final yarn. I do this 3-4 times for a skein, after washing it, while it is rolled in a towel (to help keep the skein organized). This can lightly felt the fiber too, so be sure to check the strands of yarn in the skein to ensure they aren't sticking together afterward.
Some fibers benefit from a bit of hair conditioner in the final rinse, like llama.
(based on posts by me on spindlers, 12/2001 and today)