On a 4-channel amp, each of the + and - amps per channel are independently ground-referenced. It doesn't "know" what the other half is doing or what it should be doing relatively to the other.
When you connect a truly differential signal source (no ground reference) to the inputs of an amp like this, each of the + and - amps "see" a signal that isn't really referenced to a ground that it "knows" about, unless the ground from the source is also connected. At best it sort of "works", but the + and - halves may not output an equal but opposite phase signal. Now, if the source and the amp both have 3-prong AC cords and their grounds happen to be connected via the wall wiring, then it would "work" by happenstance. But relying on wall AC wiring to carry a ground reference is not ideal.
The possible solutions are:
1. Connect the source XLR ground to the amp ground. This establishes the "zero reference" between the two.
2. Use a input coupling transformer at the amp input (with a single primary and a center-tapped secondary, ground the secondary center tap). This performs the proper "Phase split" without requiring a ground connection from the source.
3. Design a truly differential amp that could support a floating ground balanced input without a connected ground reference. The β24, for example, has a common-mode feedback (CMFB) mechanism that establishes the reference. The OPA1632, THAT1646 and other fully-differential opamps also have a similar mechanism.