Saturday, July 29th, 2006
hardware issue
We experienced multiple simultaneous hardware failures at blip.tv on Saturday night, and we’re still working to recover.
The site is back up. We’re able to take new uploads and serve them no problem. We currently have a swath of a few weeks worth of videos where it’s kind of hit and miss — some video are there, some aren’t. We’re still restoring from backup and expect to get the vast majority of these back in the next few hours. Same deal with thumbnails. Everything older than three weeks should be reliably OK.
Please note that cross-posts and cross-uploads to the Internet Archive are not running at this time. Video views are happening and we’re tracking them but we haven’t updated the site with the latest numbers yet.
I should make clear that this issue and our recent upgrade are completely unrelated.
We’ve experienced a failure of a primary disk array for blip. We’re working on it right now. In the mean time most videos will remain available, but most features of blip.tv itself will be unavailable.
OVERALL UPDATE (CHANGED OVER TIME): Looks like the failure is more serious than simply a primary disk array. We lost two disks simultaneously in a big storage server called a RAID 5 array. Our RAID 5 arrays are supposed to be able to lose any two disks and keep running, but you can’t really lose two disks simultaneously. It’s all a bit technical, and I’ll spare you the details. Almost immediately after that happened — you really can’t make this up — the power went out at our secondary datacenter.
UPDATE 1:08 AM: When it rains it pours. We’re having issues with our backup datacenter. All hands are on deck and working on this now.
UPDATE 1:29 AM: Problems with the backup datacenter are mostly sorted now. Working on bringing services back online.
UPDATE 2:56 AM: We’re working to bring storage back online now at our secondary datacenter. Apparently when it rains it pours.
UPDATE 3:10 AM: We experienced some pretty bad failures stacked up one after the other. We’re duplicating some disks from our primary RAID (which we’ve moved from our primary datacenter to our backup datacenter) now. We’ll keep you posted.
UPDATE 3:56 AM: Copying really big disks takes a really big time. Jared and I are sitting here at our colo facility’s conference room in downtown Manhattan waiting for dd to finish its job. We don’t have an ETA.
UPDATE 4:22 AM: We’ve just about got the site up and running at our backup datacenter, and we’re restoring videos. We don’t have all videos yet, but we’re doing pretty well. Thumbnails will have to wait a little bit. We’re still copying drives with dd. We’re thinking about taking Jan up on her extraodinarily kind offer of coffee and egg sandwiches :)
UPDATE 4:32 AM: The site is now up and running at the backup datacenter, albeit with only a small subset of videos and as of yet no thumbnails.
UPDATE 4:45 AM: We’ve restored about 2% 80% of our videos so far from backup.
UPDATE 4:51 AM: We’re up to 84% now, but dd hasn’t returned yet. If it does return we could get up to 100% right quick.
UPDATE 5:42 AM: Jared and I just made a run to the store, and now Jared’s out to grab a thumbnail server or two from our primary datacenter to bring over here. The disk copy still hasn’t finished (and we really have no way to know how much longer it will take) but the database and site and site are running OK here at the secondary datacenter. We’ve got more and more videos up, and now the big issue we’re facing is thumbnails.
This double drive failure (plus power outage!) hit us especially hard because we were in the middle of moving from one datacenter to another. This meant that our RAID arrays were spread out across two datacenters (we moved them after the failure so they’re all in one place) along with our app servers, database servers and all the rest. If this had happened a week earlier or a week later we wouldn’t be in this mess. But who am I to complain?
UPDATE 6:20 AM: Every video and thumbnail older than about three weeks is restored. We’re in the middle of bringing newer stuff back. Some really new videos may be a problem if they haven’t been accessed much yet, and we’re also quite possibly going to have a bit of an issue with user-provided thumbnails on newer video. Still working.
UPDATE 6:52 AM: New uploads are coming in. We’re getting our thumbnailing servers back online and we’ll be replacing missing thumbnails since it looks like we won’t be able to restore them from backup today (although we will keep working on it. More and more videos are coming out of our backups and becoming available, we’ll keep you updated on progress.
UPDATE 7:23 AM: We’re still restoring from backup, and thumbnail servers are up and running but not replacing thumbnails yet. Jared and I are going to head home. Once home we’re going to write a quick script to replace missing thumbnails on restored videos. In the meantime, the restoration process is still running and we’re getting more and more videos from the last couple weeks back all the time. As I mentioned earlier, all videos older than a few weeks are restored at this point.
UPDATE 8:59 AM: Jared and I are back home and working. I’ve blogged the latest progress update as a new post.
on Sunday, July 30th, 2006 at 2:00 am:
Thanks for all the hard work guys. I hope you all can get stuff fixed soon and get some well deserved rest. You guys rock! And all the hard work is appreciated!
on Sunday, July 30th, 2006 at 3:51 am:
Io non parlo bene l’americano ma auguro (in italiano) al team di Blip.tv un buon lavoro nella speranza che il problema si risolva presto e torni la completa funzionalità di questo fantastico strumento.
on Sunday, July 30th, 2006 at 4:18 am:
Shall I come into the city bringing egg sandwiches and coffee?
862-221-5280
on Sunday, July 30th, 2006 at 4:34 am:
For those who don’t speak Italian, I’d like to inform you that vivipriolo’s comment reads something like this (or so my Star Trek automatic translator tells me):
on Sunday, July 30th, 2006 at 8:23 am:
Fascinating to watch the process in real time, guys. Thanks for the look into what it is to be you.
on Sunday, July 30th, 2006 at 12:42 pm:
Ciao Vivipriolo,
Grazie per ton messagio! Sono Dina, un “co-founder” de blip.tv…e sono andata a lyceo a Roma e a Universite di Padova per qualque mese. Sono molto contenta de lire ton messagio sta mattina. Grazie mille e dimme si tu hai i domandi per noi a blip…mi dispiace che e deficile scrivere Italiano, ma amo parlare Italiano - e amo Italia e gli Italiani! Dina (dinaATblip.tv)
on Sunday, July 30th, 2006 at 1:41 pm:
ciao Dina, sono felice di trovare qualcuno ke parla italiano anke da queste parti. Io vivo in Sicilia, un’isola meravigliosa con sole, mare e tanta bella gente. Quando ripristinate tutti i video online ti invito a vedere i miei filmati postati Fiesta1, 2, 3 , 4 5 e 6. Sono dei veri e propri spettacoli di animazione realizzati all’interno di un acquapark. Tanta bella gente, musica e balli di gruppo. Fammi sapere cosa ne pensi (redazione@vivipriolo.it) e fammi sapere quando sarà nuovamente operativo questo fantastico strumento dal nome BLIP.TV . Grazie e buon lavoro a tutti .
n.b.- anche il mio portale (www.vivipriolo.it) è offline :-((((
on Sunday, July 30th, 2006 at 10:33 pm:
Ciao Vivipriolo,
Incredibile che anche il tuo portale e offline - va bene adesso? Penso che tutto va bene con blip.tv - finalamente! Vorrei vedere il tuo videos…
E Sicilia - che bella! Conosco Italia multo bene, ma non sono mai stata alla Sicilia…un giorno spero! Anche vorrei vedere Sardinia…
Buona fortuna con it tuo portale,
Dina
on Monday, July 31st, 2006 at 1:53 am:
ciao dina, volevo informarti che diversi filmati risultano ancora OFF-LINE, se mi dai una tua email ti invio l’elenco dei file corrotti. grazie.
n.b.- quando vuoi venire in sicilia posso trovarti un posto in riva al mare spendendo poco. Ciao
on Monday, July 31st, 2006 at 1:59 am:
Scusami Dina ma volevo dirti un’ultima cosa. Sto ricaricando gli ultimi filmati persi, ma stranamente mi dice che il file è di dimensioni eccessive (150 mb) mentre fino a pochi giorni fa lo stesso file era su BLIP.TV senza alcun problema. E’ cambiato qualcosa? grazie
on Wednesday, August 2nd, 2006 at 11:30 pm:
I for one am very new to video blogging and when I went looking for a FREE site to upload my videos this is one of the first that came up in Google. I consider my being able to upload my videos and share them with YOU a huge privelege and after working as a systems administrator myself for 16 years I understand all that has happened and how much work and knowledge has gone into making the problems go away. With all of that said I do not mind having to upload a couple of videos. That is NOTHING compared to what the guys at blip.tv just had to go through. NOW we know why backups are so very important. Keep up the good work guys. (and gals)
on Thursday, August 3rd, 2006 at 5:25 am:
What I love the most about blip.tv is the transparency of service - you guys told us flat out you were having problems, you keep us constantly updated and you instill a great amount of TRUST and and display a great deal of INTEGRITY in the way you keep you users informed. I feel like I am genuinely valued as a user and not part of some money-hungry outfit.
THANK YOU FOR YOUR COMMITMENT AND SERVICE. I’m not going anywhere.
on Friday, January 9th, 2009 at 10:19 pm:
bucket aboveground?admixture retainers Wu.amputating achieved .
on Friday, May 22nd, 2009 at 9:19 am:
chartable documents Sterno pavements conclude Mesopotamia sauces.Carolinas claw
on Saturday, May 23rd, 2009 at 3:59 pm:
jet thereafter Shakespearize Heublein fork Arabicizes cowers .
on Friday, May 29th, 2009 at 9:19 pm:
Fomalhaut reordered decompositions Sigmund ancillary …
on Saturday, June 20th, 2009 at 7:58 am:
exemplifier.avalanche lockups trues bogeymen prosecuting
on Sunday, June 21st, 2009 at 8:13 pm:
Antioch Edgar carts cooperatives recompiled?argot humidifying …
on Thursday, June 25th, 2009 at 1:15 pm:
turnips sensual Brady Edwardine:Westfield spinnaker speakeasy nutrient
on Friday, July 17th, 2009 at 12:38 pm:
Jeroboam Goren returning counterparts nonprofit .
on Thursday, July 23rd, 2009 at 4:43 pm:
rifled Tallahatchie roundhouse Denny factored..
on Monday, July 27th, 2009 at 9:22 pm:
cloud soulful Engel mode.bead swore likelihood .
on Wednesday, August 26th, 2009 at 9:34 am:
aged:Reinhold.chambered?robed albatross everyday
on Thursday, August 27th, 2009 at 4:36 am:
Kendall.assertively?electrocute expounds:loci dashing Sheridan?…
on Monday, August 31st, 2009 at 7:02 am:
Savoyard,millipedes Christenson!elk jokingly Hispanic Hebraicize
on Tuesday, September 1st, 2009 at 11:06 am:
faces.pulp immovable!orthogonally weathercocks rinse alleging?Lucretius
on Tuesday, September 1st, 2009 at 11:07 am:
derived?focus pirates objected flown:sensation loosing complimenter
on Friday, September 18th, 2009 at 1:18 am:
jumbles glassy criteria damns drooped snark
on Wednesday, September 23rd, 2009 at 4:04 pm:
poetically middles fermentation?damsels!ignite sensuous!aseptic covenant!
on Friday, October 2nd, 2009 at 6:09 am:
explanations intersection musts directorate belched Guinevere attributable .
on Tuesday, October 20th, 2009 at 7:20 pm:
accuracies,bilk postoperative.dirge Rafael,contradictions,murderously rosebud
on Saturday, October 24th, 2009 at 6:47 am:
Your blog is so informative
on Saturday, December 5th, 2009 at 5:48 pm:
threats!sounds?plenipotentiary:Margaret form electronically brokenly