Well, I just realized I can click the damn labels in the settings and there's an explanation that pops up, so that's helpful and entirely my own derp. I think I'm going to disable "Use Epic Followers to fill slots", while keeping "Boost XP with epic followers" enabled. That should make it to where XP missions are never filled with only Epic Followers, but can still be used to fill out missions with non-epic followers.
If that works the way I'm understanding it, then I should be able to set the "Max Number of Epic followers for Boost" back to the default setting of 2. I think I'm understanding this right, but it may take some time for me to know it works - but it's something for now