[mistwang]

I'm guessing its because the rights on the files in the admin panel are something like 604?

Change them to 644 then, it is caused by the umask setting of "root" user.

The reason i would opt for 640 is because i don't want people on my machine to access it, but i would like the web server to control the access via .htaccess.

Actually, as long as the permission of one of the parent directories set to 0750, other users can not access any directory/file under. It is not necessary to set 640 to all files at all.