Do you have a headphone with a TRRS plug, like shown here? 

TRRS cables are used on earphones with integrated microphones. They don't work with TRS jacks, even with a size adapter—they will sound bad or cause problems. All of our amplifiers use TRS jacks.

If you want to use your headphones with a TRRS plug, here's a kit that adapts them to 1/8" or 1/4" TRS jacks.