It was one of "those" pushes performed by Facebook last night. Facebook push is a weekly update usually performed by Facebook updating their production servers with new code, fixes and features. Apparently Facebook does not perform integration tests, therefore each push can break applications depending on the platform. Because of the fact that each push is performed too frequently without enough testing the platform became unstable. I already mentioned this earlier here - Facebook platform on Decline.
So what happened yesterday ? It started when Facebook moved the push one day later:
"For this week only, we're moving the push to wednesday night, so hopefully you guys can breathe easy tonight."
Many application development folks didn't pay much attention for the last words, since everything was stable for the past 3 weeks or so. Man WTF a shit load of bugs. It took them almost entire day to fix all breaking points in various places. One example - all desktop applications were broken - users were not able to login. Also application directory desktop applications section was emptied entirely including applications developed by Facebook itself were cleared. There were much more issues. Much more than acceptable level of bugs.
Personally I have tolerance for application/platform bugs and issues. There is no application in the world 100% bug free. But there is also some acceptable threshold level of bugs and their severity.
Meanwhile some popular applications redirected their users to Facebook beta tier in order not to loose them because eventually the end user does not know the truth and usually thinks that the application is broken or crappy. Which resulted in the following announcement by Facebook folks the other day:
"Some apps instructed their users to go to the apps.beta.facebook.com/appname today instead of apps.facebook.com/appname.
Our beta tier is only 3 machines and is just for testing. It can't handle the load from popular apps.
If your app is crushing our beta tier by sending your users there, we'll have to disable your application. *. beta.facebook.com is just for testing and isn't set up to handle production load."
Despite all this I think that Facebook platform architectural design is great and of very high level.
Saturday, October 13, 2007 6:15 AM