People love the whole idea of mashups with technology in terms of consuming APIs, using plugins or 3rd party widgets. However, as websites scale and grow more complex, depending on these 3rd party dependencies is actually problematic. This blog discusses the issues surrounding the situation.
The biggest issue in my mind in using 3rd party systems is that you lose control over what the vender can provide. A huge problem I’ve encountered is when another site goes down and inevitably can take your system down along with it. An example in my situation occurred when Livestrong.com died the night Michael Jackson passed away. What happened was that we pulled and cached data from Twitter, which went down as a result of a massive flood of users killing their system. In turn, our system, despite having caching, ended up making a large number of requests to the dead Twitter service. We used a get_file_contents php call, which is inefficient for things like this. What happened was we built up a lot of requests that got queued up and just halted the Apache request processes.
Later, we changed things around to use curl which has a timeout. But obviously, this still isn’t a great solution since you’re still keeping open connections and preventing users from proceeding to do something on the website. We added an iframe to it so a particular section with the external dependency would not affect the rest of the page if it went down. And we queued up the data as another layer of redundancy.
Still that doesn’t change the fact that relying on Twitter made our lives any easier. Having stuff like Facebook Connect as a core dependency introduces a level of complexity because you lose a lot of control over some functionality. Part of the issue is that sites frequently change API calls. What that means is that your site can inevitably go down depending on how well informed the maintainers of the API keep their customers.
Another problem is terrible documentation. Most engineers hate writing documentation and often are tasked with fixing their stuff. So documentation (along with testing sadly) often come at the expense of adding features and fixing bugs. Because documentation can go out of date frequently, your developers might often tear their hair out debugging issues.
One of the biggest offenders at the moment is the huge number of network calls being made. While there are ways to speed up HTTP requests (like caching certain requests), having too many means that the page will load slower for every network request being made. It’s been noted in front end optimization that your real bottleneck is the network. While CDNs and minification can help your own stuff, it’s hard to know how well a partner site might be handling optimizing their front end. It’s not uncommon these days to open up Firebug and see several dozens of network calls originated from all over the place. While you do get some benefit in extra content, ads, etc. from using these 3rd party services, your site visitors only get bogged down with excess garbage, reducing their experience and plaguing them with content that most likely they won’t care about.
And with these 3rd party sites you lose control to the type of content that they provide. Ads, for instance, might incidentally ship with malware. It’s not uncommon for ad companies to be unaware of the maliciousness in some of their ads or the ad networks they partner with. So there’s a chance that you might be inadvertently causing harm when you use these sites.
If anything even with the novelty of being able to integrate with other sites that might offer a new potential experience, you, at the very least, want to minimize their importance to the core business of your site. If you business strategy is creating a content farm, for instance, and you intend to mine Google using SEO tricks then make money through ad networks, you probably have a huge gap if suddenly they decide to shut you down.
Obviously, you don’t want to reinvent the wheel too. But understanding the balance between having too many unnecessary external dependencies and using ones that make sense to help your platform is the key. In general, if things become ugly, too difficult or the public just doesn’t want it, get rid of it. Use common sense. Try to be the user on the other end and feel their pain using your system. Be that developer trying to make an impossible solution work when an easier one is at hand.