This is related to the DRY principle and Totto's axiom on responsibility.
The concept is explained well in The Pragmatic Programmer:
In computing, the term has come to signify a kind of independence or decoupling. Two or more things are orthogonal if changes in one do not affect any of the others. In a well-designed system, the database code will be orthogonal to the user interface: you can change the interface without affecting the database, and swap databases without changing the interface.
Rather than elaborate on the concept here, we point to a discussion between Dave Thomas, Andy Hunt and Bill Venners on Orthogonality and DRY.
There's a small Wikipedia entry on the subject.