I sit tonight with a heavy heart, staring at my monitor in uncomprehending disbelief. Could it happen again so soon? Is disaster a weekely event in an apocalyptic new year?
Where are the people? Where have they gone?
As most of you already know, early today a wave of current failure swept over Internap, the service provider for millions (yes,
millions) of LiveJournal bloggers. Reports have been few and scattershot at best, but the fact remains is that LiveJournal is off the map - and we don't know where these people are now, or what they're doing, or what's eating them, or if the blogs have even survived.
What we know is this: late this morning, an event occurred that caused Internap to lose power and its redundant backup power, forcing a hard shutdown of over 100 LiveJournal systems. When the power returned, a message appeared at www.livejournal.com:
We're currently dealing with bringing our 100+ servers back online. Not fun. We're not happy about this. Sorry... :-/ More details later.
At 7.35 PM PST, a second communique was released:
we have power again, and we're working to assess the state of the databases. The worst thing we could do right now is rush the site up in an unreliable state. We're checking all the hardware and data, making sure everything's consistent. Where it's not, we'll be restoring from recent backups and replaying all the changes since that time, to get to the current point in time, but in good shape. We'll be providing more technical details later, for those curious, on the power failure (when we learn more), the database details, and the recovery process. For now, please be patient. We'll be working all weekend on this if we have to.
A profile in courage - that's still being sketched.
While there is no doubt that LiveJournal will prevail and rise from the ashes of the unthinkable, one can't ignore the millions of bloggers, emerging dazed and confused from an unqualified disaster. Now is the time to put politics aside. Whether you're on Blogger, or use Moveable Type - no matter how you contribute to this fragile, fragile blogosphere - reach out to a LiveJournaler. Now's the time when they will need us most. And it's our duty to be there for them.
The losses may be unimaginable. Many will have lost entries, or portions of entries, or entire chapters of their copy. And we must prepare in the days ahead for the unthinkable: some blogs may have paid the ultimate price in this disaster, and we may not see them or their likes again.
In times like this it's hard not to ruminate about the awesome power of electricity over this Internet, our precious virtual menagerie. But it is always there, and when it goes away, the darkness overcomes us all. We must look to the survivors, those that come back from the abyss, with an open heart and open wallet. In the days ahead, consider a donation to a cheriable organization or NGO to help in their effort to get LiveJournalers back on their feet, and happily - and safely - blogging again.
There comes a time, to heed a certain call, when the world must come together as one. We are the Internet. Good luck, LiveJournal.
;-)
