"I'm root, and you're not!"

We once came across a very unusual problem: root could log in to this system, get a shell, do anything that root normally does. Non-root users however couldn't log in, those that were logged in could not run any commands including the basics, ls, cd, nothing. Only shell built-ins such as echo worked, and only if you were already logged in.

Our support contact finally told us to look at permissions for the root of the file system:

root@mahler:/# ls -ld /
drwx------ 21 root root 4096 2009-01-22 19:27 /
root@mahler:/# 

A big "WTF!" Clearly, someone fat fingered a chmod.

By now you have guessed the solution:

# chmod 755 /