Yesterday I got my Raspberry Pi B+, which I use via SSH. Sometimes the SSH typing takes about 5s and more to respond with an visual feedback after a time of idleness, after that lag everything just works fine.

(I started typing the moment the gif starts.)
I know the Raspberry Pi is no super modern MacBook Pro, but it's still kind of strange. I checked the connection, and checked my terminal setting, tried reconnecting with the pi and checked if other servers I use respond similar, but everything works fine with other servers. Could anybody tell me if they have the same problem and if so how to solve it?