its all about the gear. As simple as that. Also, this aint normal gameplay. The job is done by the garg, so put some fire dmg and +garg dmg on your gear, and you will be fine. Only when you start getting the real good gear you can go back a little into phys dmg gear.
I do it in 9-11s using STR T&T gloves and poison (useless) soj. I bet there are people who kill him in 6-7 seconds. No mara or unity needed at all. My gear is kinda "average" since I play all 6 classes: Aughild bracers (+phis) and shoulders (+fetish). rakoff necklace (+phys), Jeram (92%), hellcat, Blackthorne pants and boots (no skill dmg), poison soj, crapy RGOG, STR T&T, spite (+ garg dmg) and thats about it, nothing really good rolled, but more than enough for T6 op WD. If you dont have T&T, very good magefist do the work, consider that most of your dps = garg and firebats, which are fire spells.
I use face to face, with templar healing (use the access all skills relic, so he heals twice). Pot = not needed.
Witch Doctor - Game Guide - Diablo III
My advice here = dont use crusader for ghom. You dont need the gear, it seems, so I guess you want para levels. I recommend you to level up a WD and use it. In less than 48 hours you will be doing T6 faster than your crusader will ever be able to. If you want to still use crusader, then do T5 A4bounty+rift. which gives mutch better drops than "ghom the weapon dropper" and about the same XP/hour when you are properly geared, as you say you are.