Will saying yes allow this server to connect back to my computer and cause me harm?
Generally, no. Why would it? Much like visiting any random website, there is nothing that implicitly opens some kind of backdoor for it to visit back.
(Of course, there's a possibility that the ssh client could have a bug that makes this possible, and such bugs are discovered frequently in web browsers, but ssh is tiny in comparison and so its client code has been carefully reviewed over time.)
I don't care about what I send
You're sending your password, for one, and the various files that you mentioned earlier.
MITM attacks aren't that frequent in practice, but not too hard to perform for someone using the same LAN or working at your ISP, either, so it's wise to guard against them, because an attacker wouldn't merely receive the data, but could also use it against you, e.g. connect to your server with your password and destroy things.
(Remember Firesheep? Most people wouldn't say they send anything secret to Facebook either, but at the same time, nobody likes it when someone finds out their password. So HTTPS servers also have their own keys that must be verified; web browsers just use "certificate authorities" instead of asking every user.)
I am just afraid something might creep back into my computer through that connection.
The only way the server could control the client is if the client had such a feature (ssh doesn't) or a bug that made it possible (ssh doesn't, though web browsers frequently do).
That said, some other remote connection tools, such as Remote Desktop in Windows, do have features that would let the server access files kept on the client computer, but those have to be enabled manually, and a warning message is shown that explicitly mentions everything that the server could do.
