cp: argument list too long

[giza]

Ever get the error mv: argument list too long or cp: argument list too long in Linux? Yeah, I know I shouldn't work with directories that have thousands of files in them, but sometimes that's not an option. Anyway, I did a little research and found this interesting workaround:

(cd $SOURCE && tar cfv - .) | (cd $TARGET && tar xfv -)

This command will change to your source directory, start tarring the contents, and write the tarball to stdout. The command on the right side of the pipe will change to your target directory and extract the tarball from stdin.

The reason for changing into the appropriate directories is because tar preserves the directory structure of whatever it archives, so you don't want it to include any parent directories of $SOURCE. The && characters mean that the command on the right side of them is only executed if the cd command on the left side of them executed successfully. Note that the above command requires you have enough disk space for a copy of the files to be made.

I love UNIX shell scripting. :-)

Comments

[lurene.livejournal.com]

you can use xargs too

[doco.livejournal.com]

We actually had that little trick in our Advanced Unix class at the community college a few years ago...but I forgot about that just until now.

Thanks for reminding me. :)

[giza.livejournal.com]

But that would call mv or cp for each file, right? I think the overhead for setting up and tearing down processes would be a bit of a performance killer...

[lurene.livejournal.com]

That depends on how large your tar files are - if they're larger than your ram, you'll be much slower that way than with xargs

[wolfspawn3.livejournal.com]

Find wants to be your friend as well!

find . -type f -exec mv {} $TARGET \;

[wolfspawn3.livejournal.com]

Actually depending on the situation tar might not be whats best at all, sometimes dump (if we are talking whole filesystems) is what is called for.

[lurene.livejournal.com]

dd!!!

[lurene.livejournal.com]

Slick!

[points.livejournal.com]

You can set up xargs to batch your requests as well.. i.e.: send out the commands using 100 files at a time, plus you can tell xargs to do the commands serially, or spawn multiple threads at a time.. so if you had some 5000 files to copy or so, and you wanted to do it in batches of 100, you could say always have five seperate copy processes running.

Anyhow... xargs, arcane, ludicrously powerful. Like so many other UNIX tools. ;)

[pandaguy.livejournal.com]

also look at "cpio -p" -- this uses cpio in "pipe" mode, which will accomplish the same without having to make a tarball in the middle of the process.

something like "find $SOURCE -print | cpio -pdvum $TARGET" I believe is the format, but it's been few years since I have used cpio that way.

[crywolf.livejournal.com]

Don't forget the p flag for tar, to keep permissions.

Also, rsync, while a bit more cpu overhead, especially for large file lists, is a nifty way to copy a bunch of files from one directory to another. It's especially slick if for some reason you have to interrupt the copy and resume later. "rsync -av $SOURCE $TARGET" (and add other flags if you want to delete files in $TARGET that are not in $SOURCE).

find/mv, as has been said, is also an option. I think that caching essentially eliminates the overhead of reloading mv each time.

[furahi.livejournal.com]

Linux Torvalds said, when Kernel 2.4 (or 2.2) was released that because of how filesystems work now you should never use dump.. I wish I had a link :X

[furahi.livejournal.com]

I have indeed gotten the argument list too long error, a lot.
However.... not with cp or mv; both can handle entire directories without having to have an argument for each file.
I usually deal with that using foreach; which is odd as foreach takes parameters anyway, but it'd seem to support more parameters, probably because it's not an actual command

like instead of mv * somewhere
foreach i in * ; do mv $i somewhere ; done

Yes, there is overhead for running mv with each file, and it /is/ noticeable

[giza.livejournal.com]

I was previously doing that when working on directories of tens of thousands of files. I was looking for a solution that was more efficient. :-)

[smoke-au.livejournal.com]

Reading this comment thread I swear there is more I have forgotten than I ever knew.. or something like that. Oh for the days when I worked with computers...

[stormydragon.livejournal.com]

I usually use: find $SOURCE -name $PATTERN -exec cp {} $TARGET \;

[taral.livejournal.com]

Yay cpio!

[taral.livejournal.com]

I often use cp -a $SOURCE $TARGET

[wolfspawn3.livejournal.com]

Ahh yes, you are quite right, dump is made to be used on ufs filesystems (bsd systems) there is an ext2dump around though but how good and stable it is I dont know.

[bigtig.livejournal.com]

I was kinda wondering why folks are putting thousands of files on the comman via shell substitutions when you can just move the dang directory.

Of course if you wanted to be sneak you could rsync and then suppress the various files you don't want to copy. The nice thing about using rsync as a local large copy is its resumable....

[giza.livejournal.com]

I should have clarified... I had two directories each with tens of thousands of files, and was moving their contents all to one directory. I did not want to just move things around either so that I could preserve the original directory structure.

[bigtig.livejournal.com]

Uhhh -- so why not this?

cd $SOURCE; cp -r ./ $TARGET

[darthgeek.livejournal.com]

I ran into rm: argument list too long once

I was working on a SCO Unix box (which might still be running) and somehow a directory was created that contained a copy of everything one directory above it, and then proceeded to, very likely, infinitely recurse. I was never able to find the "bottom" of it and it didn't seem to have any sort of performance impact so I just left it alone.