The developers were able to make the fight against Dhoulmagus hard because they understood the concept of economy of actions. I am borrowing the term from some of the talk coming out of the development of 4th edition Dungeons & Dragons. Essentially, economy of actions refers to how the number of actions a character or monster can make has a huge effect on its power. Even a monster with lots of hit points and powerful attacks will be fairly easy if it can only act once in a round, since it will be up against three, four, or five characters, each with their own action. That's about 4 actions to one. So, in order to make a boss fight against a major villain interesting, it is necessary to even up the number of actions between the two sides.
Dhoulmagus was such a big threat because of two key reasons: he evened up the odds by creating two clones of himself on the first turn, and he was capable of acting twice every round. So, he had four actions of his own every turn to match my four actions. If he had only acted once every round, my character's defense and healing would have easily been able to keep up with his damage output. Furthermore, the Doppelgangers forced me to spread out my damage more and use less powerful all-enemy attacks. As it was, I spent the entire two-part battle struggling to keep my characters alive. For the first time in the entire game I had to resurrect fallen comrades mid-battle.
Another thing that made the battle particularly hard was Dhoulmagus's use of all-character attacks, particularly in the second battle. Even though he had only two actions in the second half of the battle, Dhoulmagus was able to nearly exhaust my healing resources because of his constant use of strong all-character attacks. However, this only was a serious threat because I did not have full-party healing spells, and instead had to constantly alternate between healing individual characters using both of my healers. If I had full-party healing spells, Dhoulmagus's full-party attacks would not have been nearly as big of a threat. I guess this goes to show how powerful team healing spells are.
However, Dhoulmagus also proved to be difficult thanks to his wide array of action-denying attacks. In the first phase of the battle, he used single-character sleep spells to great effect, while in the second phase, he used an all-character one-round stun attack. In addition, he used an attack which nullified my entire teams buff spells and tension (an effect where a character uses a turn increasing their attack power). This forced my characters to waste more rounds restoring buff spells and rendered the turns I spent raising tension worthless.
In other words, the developers of DQ8 were able to make the Dhoulmagus fight very difficult by significantly increasing his number of actions, giving Dhoulmagus allies to soak up attacks, and giving Dhoulmagus the ability to deny or cancel out the actions of the characters. These are all tried and true methods of improving the difficulty of a major boss battle. Many of the earlier boss battles of the game used these same tools to lesser degrees in order to make them more challenging as well.
Beyond just what made the battle hard, the battle against Dhoulmagus was fun, thanks to the showmenship that went into the fight. First off, it felt a lot like the final battle of a game (particularly since Dhoulmagus has been the main villain so far). After Dhoulmagus's first form was defeated, he underwent the stereotypical transformation into a more powerful, more monstrous form. While it is an old trope, it still works, particularly since the first form felt like a full boss battle by itself. The backgrounds of both fights were also unique and very flashy. On top of that, Dhoulmagus was given a very large number of attacks and actions in both battles. In the first battle, Dhoulmagus used a basic melee attack, the "wind sickle" spell, the "thin air" spell, a full-party thorn attack, a multi-hit attack with thrown debris, a sleep attack, his debuff spell, and a full team heal spell, while still making time to laugh at the party. This is quite a few more attacks than is typical in a boss fight, and it adds to the spectacle of the fight considerably.
I am now really looking forward to the real final boss of the game, whatever that is.
Side note: I really hate how dead characters don't get experience in most RPGs. Jessica died mere moments before the final blow was struck against Dhoulmagus, and missed out on 12000 experience points. This is particularly a problem in DQ8, where boss battles give huge xp gains, but have a high chance of killing off a character or two. This causes really bothersome level discrepancies that are hard to close.