I have this issue. It is due to Windows 7 processing all of the files in your download folder for the media info.
In your downloads folder, you are likely to have lots of images, mp3 files, video clips and other items. Windows is opening all of those files to get the resolution and size information for the pictures, the size and bit rate for the video clips for the movies, and the MP3INFO for the mp3 files. IIRC, the MP3INFO is at the end of the file, so that requires that you seek to end of each file to get this information. Windows will also cache the image information, but each time the folder gets updated (each time you do another download), it must re-scan the folder looking for what changed.
Note, the details mode collects lots of information that it doesn't show, so if switching to that mode doesn't speed things up as compared to the icon mode, that is likely your problem.
I have a high end system with a fast SSD and see this problem. I also have hundreds of files taking gigs of disk. On another system that has a slower drive, only dozens of files, and almost no media files, opening the downloads folder happens without any delay.