As others have noted in the comments, an obvious reason for requiring root permissions is that pre-built packages install to shared system directories, which shouldn't be writable by just anyone. Why not "a different non-root user"? I'd say because enough packages need to make changes to shared system files and directories that it's not worth the complexity of keeping track of which packages really need root and which don't, and because not installing as root doesn't buy you all that much -- you're already putting your faith in the vendor for a lot of the core packages, so what's a few more. But ultimately, yes, there is a security trade-off, and it seems the vendors have decided that simplicity wins here.
This brings us to the question of why packages can't just be installed to, say, a user's home directory. Actually, in many cases, they can, but only if you use a source-based distribution, like Gentoo. In that case, you just compile the program with an appropriate --prefix or similar argument (many, but not all, pieces of software support such a concept). Most distros are binary-based, though, and in that case, the problem is that many Unix programs are written in such a way that various paths are hardcoded into the program at compile-time, and a ton of work and coordination would be required to change this when you're dealing with thousands of packages and their maintainers. Coordination is especially difficult in the Unix world, since there are relatively many system vendors, unlike Windows, where Microsoft can dictate changes to a great extent -- and even Microsoft has trouble getting everyone to fall in line, hence the need for compatibility measures like UAC Virtualization.
Going more into the hardcoded path issue, consider a program foo that needs to access its database of some sort. Where is this file? A plausible path is /usr/share/foo/foo.db, which would be hardcoded into the program, so that it knows exactly where to look. You might object that a program could simply try $HOME/usr/share/foo/foo.db or similar if it doesn't find its hardcoded file. This is true, and this would be nice, except there's no real standard or convention for this sort of thing, so most programs just don't implement such a fallback mechanism (that coordination problem, again). And arguably, it also adds complexity that benefits a relatively small part of the user population, but that's another can of worms.
A related problem is how a program loads shared libraries it depends on. The dynamic linker, ld.so, needs to know where to find all these shared libraries. Basically, a list of paths can be (again) hardcoded into the executable at compile-time, in which case ld.so will search these paths first; failing that, ld.so searches the directories configured in /etc/ld.so.conf. This problem is actually relatively easy to overcome, e.g., by setting the LD_LIBRARY_PATH environment variable appropriately.
So, the bottom line is just that a lot of programs aren't designed to be flexible (at runtime) about how they find their files, and consequently, the packaging developers don't find it worthwhile to offer the option of installing to alternate directories. It's not that this is technically impossible -- just that so far, there hasn't been the will to create a standard and make everyone follow it. For an example showing how it's not technically impossible, you might take a look at my answer for this question: Install git on server without sudo. Note how it's kind of an ugly workaround, and that's the sad state of things.
For the record, I actually share your frustration in not being able to easily install binary packages to alternate locations, and I've spent considerable time and effort over the years in either compiling packages manually or working around packaging issues (in ways similar to the question I linked to in the previous paragraph) when running on systems where the sysadmins aren't cooperative about installing packages I want to use.
