The answer is in your own post.
Your graphics card or RAM are likely the cause of the problem, provided it happens when you're in WoW regularly. This does not sound like an HB issue.
The problem is that your graphics card is now responsible for rendering 25 sets of armor, spell animations, movement and casting animations, etc. That just takes a lot more raw hardware power than 5 or 1 player. You'll find the same thing if your capital city is very crowded.
The more players being drawn on screen, the lower your FPS.