The answer is this: hackers aren't necessarily smart; the problem is that the victims are stupid. Hackers like LulzSec exploited "obvious" problems in the victim websites. But for all their obviousness, we (as an industry) still don't know how to fix them.
For example, take the most common problem, called "SQL injection". This problem is caused by the fact that websites treat input as "code" rather than "data". This is demonstrated by his recent vulnerability at CNN (which I got from a Tweet by Dave Aitel, it's still working last I checked). CNN set up a website that allows people to query information about colleges. A typical page looks like the following:
What we see in this picture is the webpage that is accessed with the URL http://cgi.money.cnn.com/tools/collegecost/collegecost.jsp?college_id=7966. Data for all the colleges are stored in a database. Each college is given a unique numeric identifier. In this case, the college_id of 7966 represents Oklahoma State University in CNN's database. The web-application takes this number from the URL and uses it to query the database, then formats the results in a web page.
The problem with CNN's website is that it doesn't treat 7966 as just data, but also as code. Hackers can replace that number with code, and CNN's database will run it. The easiest way to test this is to enter in faulty code, which generates an error message. The way that most of us do that is simply put a quote ' character at the end of the field. This allows us to test that this flaw exists, that the website will run our code, without actually running code on the database. An example is below:
vulnerability has an even better example that runs harmless code that grabs the version number. It's URL containing code looks like:
The thing about this vulnerability is that any teenager can exploit it. You would think that a guy with 10 years experience creating websites would know more about this problem than a teenage hacker, but the reverse is true. Most people who create websites don't really understand how they truly work, nor do they care. They care about the end result, about what the user sees, about pretty pictures. They rarely care about the boring technical details. Website designers are shockingly ignorant about the cybersecurity implications of their work. Conversely, teenagers are shockingly ignorant about how to create websites -- they only know how to break them.
The problem with this threat is that it's chronic. Everybody in the cybersecurity community understands this problem, but nobody knows how to fix it. Sure, we know how to fix this specific bug -- just change one line of code, and the problem goes away. The thing we don't understand is how to prevent a similar bug from appearing yet again. We have tools, like scanners that look for such bugs on websites, code analyzers to point out such bugs in the web-app code, and web-app firewalls to block some attacks, but none of these tools are wholly effective.