Verified answer

I use OpenOffice 2.3.1.

I could replicate your problem and solve it, too. Here's what I did:

I typed "qwerty" (or some other text). Then pressed Enter. This took me to the next line. Here I typed "-" (the minus sign) thrice (or more) and pressed Enter again. The string "---" got converted into a page-wide continuous horizontal line as you described.

To remove it, I placed the cursor next to "y" in "Qwerty" (that is, after the last character typed), and selected Format -> Default frmatting (Ctrl+Shift+Space) from the menu. That removed the separation line.

To get a string of three or more minus signs, I simply add a space after the last one, i.e. I type ---[space].

I use OpenOffice Writer for my word processing and I wasn't able to duplicate your issue.