It's going to be a busy day

[giza]

Seems that one of our developers didn't fully test something on Friday. I know, because I ran into this earlier:

drwxr-x--- 2 exim exim 9502720 Sep 10 09:37 input
drwxr-x--- 2 exim exim 4153344 Sep 10 09:37 msglog

Those aren't files, those are directory structures that are nearly 15 Megs in total...

The normal size for a directory entry is 4096 bytes...

Comments

[simbab.livejournal.com]

I had my mail server blow up on me (anti virus choked up and refused to allow messages through), and it was sending bounce notices to anyone and everyone who sent anything to any of the family addresses. To boot, I was in Syracuse at the time and was checking my mail through Gmail anyway. I decided that playing mail server admin was no fun anymore (if it ever was) and signed my parents up for Gmail accounts and forwarded their existing addresses there. Then I sat down and attempted to extricate the legitimate messages from the spam and get them somehow in the proper hands. That took quite a while, and yielded probably 20 legit messages from a spool of 250 or so. Your situation looks, um, considerably worse than that, considering the directory inode size.

Good luck.

[zorinlynx.livejournal.com]

I once had an errant script dump hundreds of thousands of files; probably millions into a directory. By the time I noticed it the directory entry had grown to 300 megabytes.

There were so many files in the directory that ls ran out of memory when trying to list it. I couldn't echo * because the shell would run out of memory globbing the directory contents.

I had to rm -rf the directory, recreate it, and run the script again to debug it before the directory got too large. rm -rf'ing it thankfully didn't run out of memory, but took forever.

(This was a few years ago when a typical workstation memory size was 256MB)

-Z

[giza.livejournal.com]

find . -type f -exec rm -f {} \;

This (hopefully) ensures that memory usage remains constant.

That's what I'm running right now...

[jlick.livejournal.com]

I hope your kernel is forking fast if you use that method.

[giza.livejournal.com]

Yeah... I forgot about the forking.

I tried doing the rm -rf method instead, and it didn't seem to be much faster. :-/

[two-pi-r.livejournal.com]

Go single-user, fsck, use the fs debugger to unlink the directory, run fsck again, and tell it to fuck off when it offers to recover all the files that lost their links. :)

[jlick.livejournal.com]

Yeah what he said. Even more 'fun' when your kernel has some insanely small directory entry cache so that everything takes forever or your filesystem type doesn't handle large directories well.

[netcrimes.livejournal.com]

Holy crap.

[greenreaper.livejournal.com]

"Only now in this late hour did Billy realize the folly of storing small files in the NTFS directory inode . . ."

(actually NTFS's method is not that bad: stores tiny files in the entry, if that's too big then stores pointers to clusters in the inode, if there's too many it stores pointers to pointers).

[giza.livejournal.com]

That's exactly what ext2/ext3 does for files.

Maybe it does something similar for directories, I have no idea. The real issue here is that with a directory that has hundreds of thousands of files, that's a lot of data to shuffle through, no matter how you store it. :-P

[greenreaper.livejournal.com]

Sounds like a job for bzip2! ;-)

yeeeek!

[coyoteden.livejournal.com]

The last time I saw something like that was when someone left snort running wide open on a VERY big pipe...

OC-48, to be precise.

IIRC, it actually filled the entire disk with directory trees.

[balinares.livejournal.com]

... Do I smell ext3 there? Because this sure smells like ext3. :)

[wildw0lf.livejournal.com]

So if it's supposed to be 4096 bytes, and is 15mb now, what all other crap is in those 15 megs?

[giza.livejournal.com]

Directory entries, aka file metadata.