My RAID 5 array died 2 days ago. And my heart just about stopped. Actually only a single drive died, but in the process of failing the drive in mdadm and adding a new one, somehow ext3 freaked out. Superblocks gone. filesystem, unmountable. Testdisk couldn't find any superblocks and insisted that the 320GB array I was pointing it contained a 13TB HFS+ partition. Holy Shit! Every document I wrote
Every piece of code I wrote
Documents and code I was legally required to destroy years ago
Every digital picture I had taken
Every email I had sent
All my music and videos Essentially the record of my life, both digital and non, since 1996 was gone. Holy Shit! Sure I had backups, but they were spotty at best, and some of these are on CDs of dubious quality that I had burned literally 11 years ago. Holy Shit! I was almost beyond hope and was about to call SE2600 friend Scott Moulton when I saw a passing reference to debugfs on a forum post. I fire it up, point it a /dev/md0 and at the prompt do an ls. ... ... and I can see my directories! ... ok ok [breathes] ... [checks man debugfs] ... debugfs: rdump publications /tmp/
debugfs:
Could it be? acidus@hatter:~$ cd /tmp/publications
HOLY SHIT! HOLY SHIT! Ode to joy and all that lot, I can recover my data! [HR inappropriate victory dance] Theodore Ts'o, I love, and if I ever meet you, I just might make sweet sweet love to you down by the fire. Serious. I crave my obsolete and poorly written C code that badly. PS: The only thing I couldn't recover with debugfs were some 4+ GB files that were flat text files and MySQL dumps. debugfs coredumps. Luckily the code which crawled the in-tar-web to assemble this data was still around and working, and so the data is reproduceable. | 
