Why ISIS is difficult to defeat even with many states fighting them.
Power politics between nations always supersedes the threat of terrorist organizations
TL;DR: To almost every actor in the region, they are more useful alive than dead
To understand the answer, we must first understand some significant points:
1) Any answer that says "x is the main problem" is super simplistic. It may have part of a problem, but, like the rest of the grown up problems of the world, a complex issue like ISIS/terrorism cannot be reduced to a single point.
2) There are always more questions than clear answers. This is a significant problem when lives are at stake, but I do believe that, given the research over the last century or so, we have a fairly reasonable idea.
The single sentence answer, which necessarily needs to be broken down into a thousand different pieces, is as follows:
The vast majority of terrorism in the last two centuries has been committed due to the political interaction of identity and the state
To understand this completely, we have to understand the concept of the nation. Most people around the world feel nationality in at least three levels:
1) Lowest level - The state as a function of nation 2) Mid-level - the Ethnic group as a nation 3) Highest level - Religious group as a nation
There are other "nations" as well, but these three are the most common. Now, the marriage of Westphalian peace that designated (theoretically) standard borders and the increased centralization and power of governments, the challenge became to somehow synthesize these three. The idea of a political "people" is born. The most cohesive state, then, was one where people shared all three national identities. The least cohesive state was one that had neither 2nd or 3rd, and thus even the function of the state as a nation fall into peril.
Following the Peace of Westphalia, Europe was consumed by violence which was primarily nationalistic violence. Between revolutions, revolts, genocides, and ethnic cleansing, Europe was a blood bath from 1648 to 1945. When one portion of Europe reached stability, another portion was lit ablaze by nationalist politics.
The Arab world never had that moment. Even after the fall of colonial powers in the Arab world, the cold war politics of the US and Russia necessitated that the Middle East, a key region in the cold war, could not suffer the instability like that of Europe after Westphalia. So strong men were put in place who clamped down on the political negotiations which are required for the long-term stability of a state which functions as a nation (A nation state).
Now, fast forward to today. The strong men of the Arab world collapse (Saddam and Yemen forcefully, Mubarak, Qadhafi, and Assad by public pressure). This unleashes a huge tide of suppressed nationalist politics, which is always, always, always both disastrously bloody and excessively violent. Add in the Western exploitation of Pan-Islamism for combating the Soviet Union in Afghanistan, which leads to 9/11 (once the Soviets were defeated in Afghanistan, Al Qaeda and other mujahideen groups focused on two other Empires encroaching on "Muslim sovereignty," the US (Iraq, Palestine, the Gulf) and Russia (Chechnya)).
Now, how does this lead to terrorism against the West?
The West obviously has a terrible reputation in the Middle East, going back two centuries. But America's bad reputation starts in the 50s, with the opposition of the United States to Mossadeq, the United States propping up of Israel, the US opposition to Nasser (who was extremely popular in the Arab world), and the United States support of dictators. Now, add the Iraq sanctions and the Iraq War II, and you get just flaming hatred for the West.
That hatred is used by the Pan-Islamists in fighting the US wherever it has presence in the Muslim world. But, for nationalists like ISIS, it has a much more sinister use.
Attacking "the West" creates legitimacy for ISIS. Due to the involvement of the US during the cold war (and after) in the Middle East, hatred for the West is immense in the Arab world. Attacks against the "oppressors," then, serves as a legitimizing tool in a fight where manpower is at a premium.
This leads us to ISIS of today. It is a nationalist movement which gains legitimacy for showing how it is fighting the historic oppressors by bringing the war to them. "We are doing to them what they have been doing to us" is the line they use consistently in their recruitment material. The goal, though, is to swell their ranks LOCALLY, not to actually harm the West in any way. No one seriously thinks ISIS is creating any real national security risk for Western nation -- not ISIS, and not Western nations. This is just a really great propaganda tool for them.
Now, given all the above, why is ISIS so difficult to defeat:
The reason why ISIS in particular has not been defeated, is, primarily power politics:
1) Power politics between nations always supersedes the threat of terrorist organizations
2) ISIS is not the main concern of Middle Eastern nations. The power imbalance and vacuum between Iran, the Gulf allies, and Turkey is
3) The Syrian civil war adds to those complications significantly
4) Nations are more focused on fighting for power balance than against ISIS
5) In managing the power politics in the Middle East, ISIS is actually useful for most if not all powers in the area
Let's give a couple examples:
Turkey was fighting Assad in Syria, but the Kurds to the north were a serious threat to their territorial integrity. As such, they helped ISIS fight the Kurds to the north as well. But to assuage Western powers, they made a show of fighting ISIS as well.
Saudi is actively funding ISIS to push back Iranian-allied Syria and Iraq and force Iran into a perpetual war with ISIS. This is because the Gulf allies are terrified of being surrounded by the Iranian-led alliance of Iran, Syria, Iraq, and Yemen. It therefore needs to ally with groups like ISIS and Al Qaeda to force Iran into perpetual war and keep it from cementing that alliance.
Iran wants to fight ISIS. It really does. It's probably the most active in fighting ISIS. But it can't make a death blow to ISIS yet, because of Iraq and Russia.
ISIS is extremely useful to Russia. As long as ISIS remains a force, it provides Russia diplomatic cover for its activities in support of Assad as long as it can shove a few bombs towards ISIS and say that it's there for terrorism. Therefore, Russia NEEDS ISIS to assist its ally Assad.
Iraq is a key ally to Iran, and ISIS is an existential threat to the Shia regime there. BUT airpower is extremely limited in efficacy, Iraq doesn't have a strong enough ground force to take ISIS on itself, the US won't allow the only other real enemy of ISIS (Iran) into Iraq en masse to defeat ISIS, and the other members of the anti-ISIS coalition see ISIS as to useful to destroy just yet
6) The US and the Western Allies:
This is where things get really fun. The West wants to defeat ISIS. But to do so would effectively hand Syria to Russia\ and Iraq to Iran, making a giant power block from Iran to Syria and Yemen to the south. So it wants to use Iraqi ground forces to force ISIS into Syria and let them fight it out in the Syrian civil war, while putting diplo pressure on Iraq to break its ties with Iran and try to isolate Iran diplomatically.
The key to defeating ISIS:
1) End the Syrian civil war 2) Bring about political reproachment in Iraq and Yemen 3) The powers in the area will crush ISIS in a day
Edit: All the above is an interpretation consistent with the theories of the school of realism in international relations. For more information about the realist school, read "The Man, the State, and War" by Kenneth Waltz, "Theory of International Politics," by Waltz again. For my favorite sub school, offensive realism, read "Tragedy of Great Power Politics" by John Mearsheimer.