Best practice to continue mv
up vote
13
down vote
favorite
I used the terminal to copy files from one drive to another.
sudo mv -vi /location/to/drive1/ /location/to/drive2/
However that suddenly stopped, while some hours into it, and without an error, after creating a directory.
My own solution to that is often a mix of hashing and comparing which is mostly a time consuming mess as I now have to recover from an intermediate copy without really knowing which files are missing (written as very long one-liner for zsh — note that this script doesn't work in bash as written):
source_directory="/path/to/source_directory/";
target_directory="/path/to/target_directory/";
while read hash_and_file; do {
echo "${hash_and_file}" | read hash file;
echo "${file}" | sed "s/^/${source_directory}/g" | read copy_from;
echo "${copy_from}" | sed "s/${source_directory}/${target_directory}/g" | read copy_to;
mv -v "${copy_from}" "${copy_to}" | tee -a log;
rm -v "${copy_from}" | tee -a log; };
done <<<$(
comm -23 <( find ${source_directory} -type f -exec sha256sum "{}" ; |
sed "s: ${source_directory}: :g" | sort;
) <( find ${target_directory} -type f -exec sha256sum "{}" ; |
sed "s: ${target_directory}: :g" | sort; ) )
This is error prone if the name target directory or source_directory are part of the path, and delete files if they have not been moved because they were marked as duplicates. Also it does not source directory in the end.
Is there a best practice how to recover from interrupted mv?
data-recovery mv
|
show 3 more comments
up vote
13
down vote
favorite
I used the terminal to copy files from one drive to another.
sudo mv -vi /location/to/drive1/ /location/to/drive2/
However that suddenly stopped, while some hours into it, and without an error, after creating a directory.
My own solution to that is often a mix of hashing and comparing which is mostly a time consuming mess as I now have to recover from an intermediate copy without really knowing which files are missing (written as very long one-liner for zsh — note that this script doesn't work in bash as written):
source_directory="/path/to/source_directory/";
target_directory="/path/to/target_directory/";
while read hash_and_file; do {
echo "${hash_and_file}" | read hash file;
echo "${file}" | sed "s/^/${source_directory}/g" | read copy_from;
echo "${copy_from}" | sed "s/${source_directory}/${target_directory}/g" | read copy_to;
mv -v "${copy_from}" "${copy_to}" | tee -a log;
rm -v "${copy_from}" | tee -a log; };
done <<<$(
comm -23 <( find ${source_directory} -type f -exec sha256sum "{}" ; |
sed "s: ${source_directory}: :g" | sort;
) <( find ${target_directory} -type f -exec sha256sum "{}" ; |
sed "s: ${target_directory}: :g" | sort; ) )
This is error prone if the name target directory or source_directory are part of the path, and delete files if they have not been moved because they were marked as duplicates. Also it does not source directory in the end.
Is there a best practice how to recover from interrupted mv?
data-recovery mv
I wrote a similar script, which usescmp
instead of hashing. It has dependencies, and the same issues withwhile read
that Gilles mentioned. It is also slow and verbose. But it frees up disk-space sooner than the rsync method, because files are (re)moved from the source as it runs. It may serve as inspiration for the brave.
– joeytwiddle
Dec 1 at 14:46
3
@joeytwiddle rsync offers--delete-during receiver deletes during the transfer
and also several other useful alternatives:--delete --delete-before --delete-delay --delete-after --delete-excluded
. So, yes, rsync is the best alternative,
– Isaac
Dec 1 at 19:31
I must be missing something. Why doesn't just repeating the samemv
command work? Perhaps with*
appended to source path if the original source was a directory.
– jpa
Dec 2 at 11:04
@isaac No, I'm afraidrsync --delete*
would be a disaster! It will remove things fromdest
which are not currently insrc
, so all files which were successfully moved in the previous attempt will now be deleted! You were probably thinking ofrsync --remove-source-files
which I agree would be a good alternative. (more1, more2)
– joeytwiddle
Dec 3 at 3:29
@joeytwiddle No,rsync --delete
will only remove other files that are not part of the source. From [man rsync]()*delete extraneous files from dest dirs*. Understand what extraneous means: Not being synced. And yes, rsync also provides a way to remove source files after they have been correctly transmitted.
– Isaac
Dec 3 at 4:53
|
show 3 more comments
up vote
13
down vote
favorite
up vote
13
down vote
favorite
I used the terminal to copy files from one drive to another.
sudo mv -vi /location/to/drive1/ /location/to/drive2/
However that suddenly stopped, while some hours into it, and without an error, after creating a directory.
My own solution to that is often a mix of hashing and comparing which is mostly a time consuming mess as I now have to recover from an intermediate copy without really knowing which files are missing (written as very long one-liner for zsh — note that this script doesn't work in bash as written):
source_directory="/path/to/source_directory/";
target_directory="/path/to/target_directory/";
while read hash_and_file; do {
echo "${hash_and_file}" | read hash file;
echo "${file}" | sed "s/^/${source_directory}/g" | read copy_from;
echo "${copy_from}" | sed "s/${source_directory}/${target_directory}/g" | read copy_to;
mv -v "${copy_from}" "${copy_to}" | tee -a log;
rm -v "${copy_from}" | tee -a log; };
done <<<$(
comm -23 <( find ${source_directory} -type f -exec sha256sum "{}" ; |
sed "s: ${source_directory}: :g" | sort;
) <( find ${target_directory} -type f -exec sha256sum "{}" ; |
sed "s: ${target_directory}: :g" | sort; ) )
This is error prone if the name target directory or source_directory are part of the path, and delete files if they have not been moved because they were marked as duplicates. Also it does not source directory in the end.
Is there a best practice how to recover from interrupted mv?
data-recovery mv
I used the terminal to copy files from one drive to another.
sudo mv -vi /location/to/drive1/ /location/to/drive2/
However that suddenly stopped, while some hours into it, and without an error, after creating a directory.
My own solution to that is often a mix of hashing and comparing which is mostly a time consuming mess as I now have to recover from an intermediate copy without really knowing which files are missing (written as very long one-liner for zsh — note that this script doesn't work in bash as written):
source_directory="/path/to/source_directory/";
target_directory="/path/to/target_directory/";
while read hash_and_file; do {
echo "${hash_and_file}" | read hash file;
echo "${file}" | sed "s/^/${source_directory}/g" | read copy_from;
echo "${copy_from}" | sed "s/${source_directory}/${target_directory}/g" | read copy_to;
mv -v "${copy_from}" "${copy_to}" | tee -a log;
rm -v "${copy_from}" | tee -a log; };
done <<<$(
comm -23 <( find ${source_directory} -type f -exec sha256sum "{}" ; |
sed "s: ${source_directory}: :g" | sort;
) <( find ${target_directory} -type f -exec sha256sum "{}" ; |
sed "s: ${target_directory}: :g" | sort; ) )
This is error prone if the name target directory or source_directory are part of the path, and delete files if they have not been moved because they were marked as duplicates. Also it does not source directory in the end.
Is there a best practice how to recover from interrupted mv?
data-recovery mv
data-recovery mv
edited Dec 1 at 9:50
asked Dec 1 at 9:17
What
786
786
I wrote a similar script, which usescmp
instead of hashing. It has dependencies, and the same issues withwhile read
that Gilles mentioned. It is also slow and verbose. But it frees up disk-space sooner than the rsync method, because files are (re)moved from the source as it runs. It may serve as inspiration for the brave.
– joeytwiddle
Dec 1 at 14:46
3
@joeytwiddle rsync offers--delete-during receiver deletes during the transfer
and also several other useful alternatives:--delete --delete-before --delete-delay --delete-after --delete-excluded
. So, yes, rsync is the best alternative,
– Isaac
Dec 1 at 19:31
I must be missing something. Why doesn't just repeating the samemv
command work? Perhaps with*
appended to source path if the original source was a directory.
– jpa
Dec 2 at 11:04
@isaac No, I'm afraidrsync --delete*
would be a disaster! It will remove things fromdest
which are not currently insrc
, so all files which were successfully moved in the previous attempt will now be deleted! You were probably thinking ofrsync --remove-source-files
which I agree would be a good alternative. (more1, more2)
– joeytwiddle
Dec 3 at 3:29
@joeytwiddle No,rsync --delete
will only remove other files that are not part of the source. From [man rsync]()*delete extraneous files from dest dirs*. Understand what extraneous means: Not being synced. And yes, rsync also provides a way to remove source files after they have been correctly transmitted.
– Isaac
Dec 3 at 4:53
|
show 3 more comments
I wrote a similar script, which usescmp
instead of hashing. It has dependencies, and the same issues withwhile read
that Gilles mentioned. It is also slow and verbose. But it frees up disk-space sooner than the rsync method, because files are (re)moved from the source as it runs. It may serve as inspiration for the brave.
– joeytwiddle
Dec 1 at 14:46
3
@joeytwiddle rsync offers--delete-during receiver deletes during the transfer
and also several other useful alternatives:--delete --delete-before --delete-delay --delete-after --delete-excluded
. So, yes, rsync is the best alternative,
– Isaac
Dec 1 at 19:31
I must be missing something. Why doesn't just repeating the samemv
command work? Perhaps with*
appended to source path if the original source was a directory.
– jpa
Dec 2 at 11:04
@isaac No, I'm afraidrsync --delete*
would be a disaster! It will remove things fromdest
which are not currently insrc
, so all files which were successfully moved in the previous attempt will now be deleted! You were probably thinking ofrsync --remove-source-files
which I agree would be a good alternative. (more1, more2)
– joeytwiddle
Dec 3 at 3:29
@joeytwiddle No,rsync --delete
will only remove other files that are not part of the source. From [man rsync]()*delete extraneous files from dest dirs*. Understand what extraneous means: Not being synced. And yes, rsync also provides a way to remove source files after they have been correctly transmitted.
– Isaac
Dec 3 at 4:53
I wrote a similar script, which uses
cmp
instead of hashing. It has dependencies, and the same issues with while read
that Gilles mentioned. It is also slow and verbose. But it frees up disk-space sooner than the rsync method, because files are (re)moved from the source as it runs. It may serve as inspiration for the brave.– joeytwiddle
Dec 1 at 14:46
I wrote a similar script, which uses
cmp
instead of hashing. It has dependencies, and the same issues with while read
that Gilles mentioned. It is also slow and verbose. But it frees up disk-space sooner than the rsync method, because files are (re)moved from the source as it runs. It may serve as inspiration for the brave.– joeytwiddle
Dec 1 at 14:46
3
3
@joeytwiddle rsync offers
--delete-during receiver deletes during the transfer
and also several other useful alternatives: --delete --delete-before --delete-delay --delete-after --delete-excluded
. So, yes, rsync is the best alternative,– Isaac
Dec 1 at 19:31
@joeytwiddle rsync offers
--delete-during receiver deletes during the transfer
and also several other useful alternatives: --delete --delete-before --delete-delay --delete-after --delete-excluded
. So, yes, rsync is the best alternative,– Isaac
Dec 1 at 19:31
I must be missing something. Why doesn't just repeating the same
mv
command work? Perhaps with *
appended to source path if the original source was a directory.– jpa
Dec 2 at 11:04
I must be missing something. Why doesn't just repeating the same
mv
command work? Perhaps with *
appended to source path if the original source was a directory.– jpa
Dec 2 at 11:04
@isaac No, I'm afraid
rsync --delete*
would be a disaster! It will remove things from dest
which are not currently in src
, so all files which were successfully moved in the previous attempt will now be deleted! You were probably thinking of rsync --remove-source-files
which I agree would be a good alternative. (more1, more2)– joeytwiddle
Dec 3 at 3:29
@isaac No, I'm afraid
rsync --delete*
would be a disaster! It will remove things from dest
which are not currently in src
, so all files which were successfully moved in the previous attempt will now be deleted! You were probably thinking of rsync --remove-source-files
which I agree would be a good alternative. (more1, more2)– joeytwiddle
Dec 3 at 3:29
@joeytwiddle No,
rsync --delete
will only remove other files that are not part of the source. From [man rsync]()*delete extraneous files from dest dirs*. Understand what extraneous means: Not being synced. And yes, rsync also provides a way to remove source files after they have been correctly transmitted.– Isaac
Dec 3 at 4:53
@joeytwiddle No,
rsync --delete
will only remove other files that are not part of the source. From [man rsync]()*delete extraneous files from dest dirs*. Understand what extraneous means: Not being synced. And yes, rsync also provides a way to remove source files after they have been correctly transmitted.– Isaac
Dec 3 at 4:53
|
show 3 more comments
1 Answer
1
active
oldest
votes
up vote
44
down vote
accepted
Forget about trying to reinvent rsync, and use rsync.
sudo rsync -av /location/to/drive1/ /location/to/drive2/
Make sure you use a trailing slash on the source, otherwise it would copy to /location/to/drive2/drive1
.
Double-check that the command succeeded, then run rm -rf /location/to/drive1/
.
The command above will overwrite any preexisting file from drive2
. If you want to prompt the user to skip files that already existed in drive2
, as with mv -i
, it's more complicated, because you now need to distinguish files that have already been copied and files that haven't. You can pass the --ignore-existing
option to rsync to skip files that already exist on the destination regardless of their content. Note that if the original mv
was interrupted in the middle of creating a file, this file will remain in its half-copied state (whereas a bare rsync -a
would properly finish copying it).
If you want to reproduce the exact behavior of mv -i
, including the prompting, it could be done, but it's a lot more complicated.
Note that your one-giant-liner is very fragile. If there are file names containing backslashes or newlines, they may not be copied properly or they may even trick your script into removing arbitrary files. So do not use the code in the question unless you're sure that you can trust the file names not to contain backslashes or newlines.
For future reference, I recommend to never use mv
for large cross-drive moves, precisely because it's hard to control what happens if it gets interrupted. Use rsync to do the copying, and then remove the original.
What promises does rsync make that mv does not?
– What
Dec 1 at 9:48
4
well, for examplersync
does what you're trying to do, whilemv
does not. Also: copying between different machines; compression for transfer; skipping files existing at the destination based on timestamp- or hash-based equality; configurable handling of ownership, permissions, links and special files; etc. linux.die.net/man/1/rsync
– Silly Freak
Dec 1 at 11:00
1
@SillyFreak should I conclude from that, that I should always use rsync instead of mv, not only as Gilles said for cross-drive, but any operation, as the boundary of "too large" is relatively subjective and if it comes to a problem it would have been solved by rsync anyway?
– What
Dec 1 at 11:07
8
well, when I'm moving files or directories inside one partition, I usually usemv
(or the file manager) because it's only moving a reference to the file/directory. If I need to do actual data transfer, then I usersync
if one of the following is true: 1) I'm moving more files than I can check for correct transfer at a glance; 2) I anticipate that I'll need to keep files in sync; 3) I expect the transfer could be interrupted. My point is, for the use case you're presenting in the question,rsync
is simply the right tool, andmv
orcp
are not.
– Silly Freak
Dec 1 at 11:22
4
I would advise to always run any rsync command with -v and —dry-run first to confirm exactly what it’s going to do.
– Darren
Dec 1 at 18:49
|
show 1 more comment
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
44
down vote
accepted
Forget about trying to reinvent rsync, and use rsync.
sudo rsync -av /location/to/drive1/ /location/to/drive2/
Make sure you use a trailing slash on the source, otherwise it would copy to /location/to/drive2/drive1
.
Double-check that the command succeeded, then run rm -rf /location/to/drive1/
.
The command above will overwrite any preexisting file from drive2
. If you want to prompt the user to skip files that already existed in drive2
, as with mv -i
, it's more complicated, because you now need to distinguish files that have already been copied and files that haven't. You can pass the --ignore-existing
option to rsync to skip files that already exist on the destination regardless of their content. Note that if the original mv
was interrupted in the middle of creating a file, this file will remain in its half-copied state (whereas a bare rsync -a
would properly finish copying it).
If you want to reproduce the exact behavior of mv -i
, including the prompting, it could be done, but it's a lot more complicated.
Note that your one-giant-liner is very fragile. If there are file names containing backslashes or newlines, they may not be copied properly or they may even trick your script into removing arbitrary files. So do not use the code in the question unless you're sure that you can trust the file names not to contain backslashes or newlines.
For future reference, I recommend to never use mv
for large cross-drive moves, precisely because it's hard to control what happens if it gets interrupted. Use rsync to do the copying, and then remove the original.
What promises does rsync make that mv does not?
– What
Dec 1 at 9:48
4
well, for examplersync
does what you're trying to do, whilemv
does not. Also: copying between different machines; compression for transfer; skipping files existing at the destination based on timestamp- or hash-based equality; configurable handling of ownership, permissions, links and special files; etc. linux.die.net/man/1/rsync
– Silly Freak
Dec 1 at 11:00
1
@SillyFreak should I conclude from that, that I should always use rsync instead of mv, not only as Gilles said for cross-drive, but any operation, as the boundary of "too large" is relatively subjective and if it comes to a problem it would have been solved by rsync anyway?
– What
Dec 1 at 11:07
8
well, when I'm moving files or directories inside one partition, I usually usemv
(or the file manager) because it's only moving a reference to the file/directory. If I need to do actual data transfer, then I usersync
if one of the following is true: 1) I'm moving more files than I can check for correct transfer at a glance; 2) I anticipate that I'll need to keep files in sync; 3) I expect the transfer could be interrupted. My point is, for the use case you're presenting in the question,rsync
is simply the right tool, andmv
orcp
are not.
– Silly Freak
Dec 1 at 11:22
4
I would advise to always run any rsync command with -v and —dry-run first to confirm exactly what it’s going to do.
– Darren
Dec 1 at 18:49
|
show 1 more comment
up vote
44
down vote
accepted
Forget about trying to reinvent rsync, and use rsync.
sudo rsync -av /location/to/drive1/ /location/to/drive2/
Make sure you use a trailing slash on the source, otherwise it would copy to /location/to/drive2/drive1
.
Double-check that the command succeeded, then run rm -rf /location/to/drive1/
.
The command above will overwrite any preexisting file from drive2
. If you want to prompt the user to skip files that already existed in drive2
, as with mv -i
, it's more complicated, because you now need to distinguish files that have already been copied and files that haven't. You can pass the --ignore-existing
option to rsync to skip files that already exist on the destination regardless of their content. Note that if the original mv
was interrupted in the middle of creating a file, this file will remain in its half-copied state (whereas a bare rsync -a
would properly finish copying it).
If you want to reproduce the exact behavior of mv -i
, including the prompting, it could be done, but it's a lot more complicated.
Note that your one-giant-liner is very fragile. If there are file names containing backslashes or newlines, they may not be copied properly or they may even trick your script into removing arbitrary files. So do not use the code in the question unless you're sure that you can trust the file names not to contain backslashes or newlines.
For future reference, I recommend to never use mv
for large cross-drive moves, precisely because it's hard to control what happens if it gets interrupted. Use rsync to do the copying, and then remove the original.
What promises does rsync make that mv does not?
– What
Dec 1 at 9:48
4
well, for examplersync
does what you're trying to do, whilemv
does not. Also: copying between different machines; compression for transfer; skipping files existing at the destination based on timestamp- or hash-based equality; configurable handling of ownership, permissions, links and special files; etc. linux.die.net/man/1/rsync
– Silly Freak
Dec 1 at 11:00
1
@SillyFreak should I conclude from that, that I should always use rsync instead of mv, not only as Gilles said for cross-drive, but any operation, as the boundary of "too large" is relatively subjective and if it comes to a problem it would have been solved by rsync anyway?
– What
Dec 1 at 11:07
8
well, when I'm moving files or directories inside one partition, I usually usemv
(or the file manager) because it's only moving a reference to the file/directory. If I need to do actual data transfer, then I usersync
if one of the following is true: 1) I'm moving more files than I can check for correct transfer at a glance; 2) I anticipate that I'll need to keep files in sync; 3) I expect the transfer could be interrupted. My point is, for the use case you're presenting in the question,rsync
is simply the right tool, andmv
orcp
are not.
– Silly Freak
Dec 1 at 11:22
4
I would advise to always run any rsync command with -v and —dry-run first to confirm exactly what it’s going to do.
– Darren
Dec 1 at 18:49
|
show 1 more comment
up vote
44
down vote
accepted
up vote
44
down vote
accepted
Forget about trying to reinvent rsync, and use rsync.
sudo rsync -av /location/to/drive1/ /location/to/drive2/
Make sure you use a trailing slash on the source, otherwise it would copy to /location/to/drive2/drive1
.
Double-check that the command succeeded, then run rm -rf /location/to/drive1/
.
The command above will overwrite any preexisting file from drive2
. If you want to prompt the user to skip files that already existed in drive2
, as with mv -i
, it's more complicated, because you now need to distinguish files that have already been copied and files that haven't. You can pass the --ignore-existing
option to rsync to skip files that already exist on the destination regardless of their content. Note that if the original mv
was interrupted in the middle of creating a file, this file will remain in its half-copied state (whereas a bare rsync -a
would properly finish copying it).
If you want to reproduce the exact behavior of mv -i
, including the prompting, it could be done, but it's a lot more complicated.
Note that your one-giant-liner is very fragile. If there are file names containing backslashes or newlines, they may not be copied properly or they may even trick your script into removing arbitrary files. So do not use the code in the question unless you're sure that you can trust the file names not to contain backslashes or newlines.
For future reference, I recommend to never use mv
for large cross-drive moves, precisely because it's hard to control what happens if it gets interrupted. Use rsync to do the copying, and then remove the original.
Forget about trying to reinvent rsync, and use rsync.
sudo rsync -av /location/to/drive1/ /location/to/drive2/
Make sure you use a trailing slash on the source, otherwise it would copy to /location/to/drive2/drive1
.
Double-check that the command succeeded, then run rm -rf /location/to/drive1/
.
The command above will overwrite any preexisting file from drive2
. If you want to prompt the user to skip files that already existed in drive2
, as with mv -i
, it's more complicated, because you now need to distinguish files that have already been copied and files that haven't. You can pass the --ignore-existing
option to rsync to skip files that already exist on the destination regardless of their content. Note that if the original mv
was interrupted in the middle of creating a file, this file will remain in its half-copied state (whereas a bare rsync -a
would properly finish copying it).
If you want to reproduce the exact behavior of mv -i
, including the prompting, it could be done, but it's a lot more complicated.
Note that your one-giant-liner is very fragile. If there are file names containing backslashes or newlines, they may not be copied properly or they may even trick your script into removing arbitrary files. So do not use the code in the question unless you're sure that you can trust the file names not to contain backslashes or newlines.
For future reference, I recommend to never use mv
for large cross-drive moves, precisely because it's hard to control what happens if it gets interrupted. Use rsync to do the copying, and then remove the original.
answered Dec 1 at 9:38
Gilles
524k12610481578
524k12610481578
What promises does rsync make that mv does not?
– What
Dec 1 at 9:48
4
well, for examplersync
does what you're trying to do, whilemv
does not. Also: copying between different machines; compression for transfer; skipping files existing at the destination based on timestamp- or hash-based equality; configurable handling of ownership, permissions, links and special files; etc. linux.die.net/man/1/rsync
– Silly Freak
Dec 1 at 11:00
1
@SillyFreak should I conclude from that, that I should always use rsync instead of mv, not only as Gilles said for cross-drive, but any operation, as the boundary of "too large" is relatively subjective and if it comes to a problem it would have been solved by rsync anyway?
– What
Dec 1 at 11:07
8
well, when I'm moving files or directories inside one partition, I usually usemv
(or the file manager) because it's only moving a reference to the file/directory. If I need to do actual data transfer, then I usersync
if one of the following is true: 1) I'm moving more files than I can check for correct transfer at a glance; 2) I anticipate that I'll need to keep files in sync; 3) I expect the transfer could be interrupted. My point is, for the use case you're presenting in the question,rsync
is simply the right tool, andmv
orcp
are not.
– Silly Freak
Dec 1 at 11:22
4
I would advise to always run any rsync command with -v and —dry-run first to confirm exactly what it’s going to do.
– Darren
Dec 1 at 18:49
|
show 1 more comment
What promises does rsync make that mv does not?
– What
Dec 1 at 9:48
4
well, for examplersync
does what you're trying to do, whilemv
does not. Also: copying between different machines; compression for transfer; skipping files existing at the destination based on timestamp- or hash-based equality; configurable handling of ownership, permissions, links and special files; etc. linux.die.net/man/1/rsync
– Silly Freak
Dec 1 at 11:00
1
@SillyFreak should I conclude from that, that I should always use rsync instead of mv, not only as Gilles said for cross-drive, but any operation, as the boundary of "too large" is relatively subjective and if it comes to a problem it would have been solved by rsync anyway?
– What
Dec 1 at 11:07
8
well, when I'm moving files or directories inside one partition, I usually usemv
(or the file manager) because it's only moving a reference to the file/directory. If I need to do actual data transfer, then I usersync
if one of the following is true: 1) I'm moving more files than I can check for correct transfer at a glance; 2) I anticipate that I'll need to keep files in sync; 3) I expect the transfer could be interrupted. My point is, for the use case you're presenting in the question,rsync
is simply the right tool, andmv
orcp
are not.
– Silly Freak
Dec 1 at 11:22
4
I would advise to always run any rsync command with -v and —dry-run first to confirm exactly what it’s going to do.
– Darren
Dec 1 at 18:49
What promises does rsync make that mv does not?
– What
Dec 1 at 9:48
What promises does rsync make that mv does not?
– What
Dec 1 at 9:48
4
4
well, for example
rsync
does what you're trying to do, while mv
does not. Also: copying between different machines; compression for transfer; skipping files existing at the destination based on timestamp- or hash-based equality; configurable handling of ownership, permissions, links and special files; etc. linux.die.net/man/1/rsync– Silly Freak
Dec 1 at 11:00
well, for example
rsync
does what you're trying to do, while mv
does not. Also: copying between different machines; compression for transfer; skipping files existing at the destination based on timestamp- or hash-based equality; configurable handling of ownership, permissions, links and special files; etc. linux.die.net/man/1/rsync– Silly Freak
Dec 1 at 11:00
1
1
@SillyFreak should I conclude from that, that I should always use rsync instead of mv, not only as Gilles said for cross-drive, but any operation, as the boundary of "too large" is relatively subjective and if it comes to a problem it would have been solved by rsync anyway?
– What
Dec 1 at 11:07
@SillyFreak should I conclude from that, that I should always use rsync instead of mv, not only as Gilles said for cross-drive, but any operation, as the boundary of "too large" is relatively subjective and if it comes to a problem it would have been solved by rsync anyway?
– What
Dec 1 at 11:07
8
8
well, when I'm moving files or directories inside one partition, I usually use
mv
(or the file manager) because it's only moving a reference to the file/directory. If I need to do actual data transfer, then I use rsync
if one of the following is true: 1) I'm moving more files than I can check for correct transfer at a glance; 2) I anticipate that I'll need to keep files in sync; 3) I expect the transfer could be interrupted. My point is, for the use case you're presenting in the question, rsync
is simply the right tool, and mv
or cp
are not.– Silly Freak
Dec 1 at 11:22
well, when I'm moving files or directories inside one partition, I usually use
mv
(or the file manager) because it's only moving a reference to the file/directory. If I need to do actual data transfer, then I use rsync
if one of the following is true: 1) I'm moving more files than I can check for correct transfer at a glance; 2) I anticipate that I'll need to keep files in sync; 3) I expect the transfer could be interrupted. My point is, for the use case you're presenting in the question, rsync
is simply the right tool, and mv
or cp
are not.– Silly Freak
Dec 1 at 11:22
4
4
I would advise to always run any rsync command with -v and —dry-run first to confirm exactly what it’s going to do.
– Darren
Dec 1 at 18:49
I would advise to always run any rsync command with -v and —dry-run first to confirm exactly what it’s going to do.
– Darren
Dec 1 at 18:49
|
show 1 more comment
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f485321%2fbest-practice-to-continue-mv%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I wrote a similar script, which uses
cmp
instead of hashing. It has dependencies, and the same issues withwhile read
that Gilles mentioned. It is also slow and verbose. But it frees up disk-space sooner than the rsync method, because files are (re)moved from the source as it runs. It may serve as inspiration for the brave.– joeytwiddle
Dec 1 at 14:46
3
@joeytwiddle rsync offers
--delete-during receiver deletes during the transfer
and also several other useful alternatives:--delete --delete-before --delete-delay --delete-after --delete-excluded
. So, yes, rsync is the best alternative,– Isaac
Dec 1 at 19:31
I must be missing something. Why doesn't just repeating the same
mv
command work? Perhaps with*
appended to source path if the original source was a directory.– jpa
Dec 2 at 11:04
@isaac No, I'm afraid
rsync --delete*
would be a disaster! It will remove things fromdest
which are not currently insrc
, so all files which were successfully moved in the previous attempt will now be deleted! You were probably thinking ofrsync --remove-source-files
which I agree would be a good alternative. (more1, more2)– joeytwiddle
Dec 3 at 3:29
@joeytwiddle No,
rsync --delete
will only remove other files that are not part of the source. From [man rsync]()*delete extraneous files from dest dirs*. Understand what extraneous means: Not being synced. And yes, rsync also provides a way to remove source files after they have been correctly transmitted.– Isaac
Dec 3 at 4:53