Firstly, there is the problem of usability. Unusable crypto isn't a valid option for most users. Most would rather just not communicate at all, or risk going to jail, rather than deal with the typical dependency hell of trying to get open-source to compile. Moreover, open-source apps are notoriously user-hostile, which is why the Linux desktop still hasn't made headway against Windows or Macintosh. The reason is that developers blame users for being stupid for not appreciating how easy their apps are, whereas Microsoft and Apple spend $billions in usability studies actually listening to users. Desktops like Ubuntu are pretty good -- but only when they exactly copy Windows/Macintosh. Ubuntu still doesn't invest in the usability studies that Microsoft/Apple do.
The second problem is deterministic builds. If I want to install an app on my iPhone or Android, the only usable way is through their app stores. This means downloading the binary, not the source. Without deterministic builds, there is no way to verify the downloaded binary matches the public source. The binary may, in fact, be compiled from different source containing a backdoor. This means a malicious company (or an FBI NSL letter) can backdoor open-source binaries as easily as closed-source binaries.
The third problem is code-review. People trust open-source because they can see for themselves if it has any bugs. Or, if not themselves, they have faith that others are looking at the code ("many eyes makes bugs shallow"). Yet, this rarely happens. We repeatedly see bugs giving backdoor access ('vulns') that remain undetected in open-source projects for years, such as the OpenSSL Heartbleed bug. The simple fact is that people aren't looking at open-source. Those qualified to review code would rather be writing their own code. The opposite is true for closed-source, where they pay people to review code. While engineers won't review code for fame/glory, they will for money. Given two products, one open and the other closed, it's impossible to guess which has had more "eyes" looking at the source -- in many case, it's the closed-source that has been better reviewed.
What's funny about this open-source bigotry is that it leads to very bad solutions. A lot of people I know use the libpurple open-source library and the jabber.ccc.de server (run by CCC hacking club). People have reviewed the libpurple source and have found it extremely buggy, and chat apps don't pin SSL certificates, meaning any SSL encryption to the CCC server can easily be intercepted. In other words, the open-source alternative is known to be incredibly insecure, yet people still use it, because "everyone knows" that open-source is more secure than closed-source.
Wickr and SilentCircle are two secure messaging/phone apps that I use, for the simple fact that they work both on Android and iPhone, and both are easy to use. I've read their crypto algorithms, so I have some assurance that they are doing things right. SilentCircle has open-sourced part of their code, which looks horrible, so it's probable they have some 0day lurking in there somewhere, but it's really no worse than equivalent code. I do know that both companies have spent considerable resources on code review, so I know at least as many "eyes" have reviewed their code as open-source. Even if they showed me their source, I'm not going to read it all -- I've got more important things to do, like write my own source.
Thus, I see no benefit to open-source in this case. Except for Cryptocat, all the open-source messaging apps I've used have been buggy and hard to use. But, you can easily change my mind: just demonstrate an open-source app where more eyes have reviewed the code, or a project that has deterministic builds, or a project that is easier to use, or some other measurable benefit.
Of course, I write this as if the argument was about the benefits of open-source. We all know this doesn't matter. As the EFF teaches us, it's not about benefits, but which is ideologically pure; that open-source is inherently more ethical than closed-source.