Masked (hidden) failures copying large folder with File Explorer
up vote
0
down vote
favorite
When I start copying a very large folder (say 40 GB) in Windows 7 File Explorer, the rate of progress indicates that it may take hours to complete. So I leave it overnight, only to discover in the morning that it still claims to be trying to copy (it has failed but still appears to be in progress).
Is there a way to copy a large folder with high confidence that it will be successfully completed, or at least that it will conclude and display an error message if there is a problem?
windows-7 windows windows-explorer file-transfer
edited Nov 24 at 23:45
Scott
15.5k113789
asked Nov 21 at 19:19
zar
4281615
|
show 2 more comments
up vote
0
down vote
favorite
When I start copying a very large folder (say 40 GB) in Windows 7 File Explorer, the rate of progress indicates that it may take hours to complete. So I leave it overnight, only to discover in the morning that it still claims to be trying to copy (it has failed but still appears to be in progress).
Is there a way to copy a large folder with high confidence that it will be successfully completed, or at least that it will conclude and display an error message if there is a problem?
windows-7 windows windows-explorer file-transfer
edited Nov 24 at 23:45
Scott
15.5k113789
asked Nov 21 at 19:19
zar
4281615
Last time I used Windows, it didn't even know if the destination had sufficient space for the file until it returned a 'disk full' error, half way through. Hopefully that has improved since Win7... but ...
– Tetsujin
Nov 21 at 19:30
@Tetsujin it really is a pain to get a large folder to copy via file explorer. This issue is not acknoledged too well. Using File explorer for large folders should be a no no to put it blunetly.
– zar
Nov 21 at 19:53
1
You've got a legit question buried in this, but it's worded in an off-topic way. It invites discussion, invites dispute about whether File Explorer is actually problematic, and asks for opinion on what's best. I'm voting to close on that basis, but will retract the vote if you can improve the question. Just state the problem you're facing and ask for a solution to it.
– fixer1234
Nov 24 at 1:16
@fixer1234 I have changed the title since.
– zar
Nov 24 at 16:14
Copying from one drive to another drive will be much faster than copying to the same drive. When copying to the same drive, the drive needs to keep finding, and moving between, the source and destination locations. If the folder contains a huge number of tiny files, it also needs to store directory information for each one. So that will take much longer than copying a few extremely large files. Huge folders won't be copied instantaneously, but 40 GB shouldn't take hours, even under the worst conditions (unless you are doing another drive-intensive, (cont'd)
– fixer1234
Nov 24 at 23:52
|
show 2 more comments
up vote
0
down vote
favorite
up vote
0
down vote
favorite
When I start copying a very large folder (say 40 GB) in Windows 7 File Explorer, the rate of progress indicates that it may take hours to complete. So I leave it overnight, only to discover in the morning that it still claims to be trying to copy (it has failed but still appears to be in progress).
Is there a way to copy a large folder with high confidence that it will be successfully completed, or at least that it will conclude and display an error message if there is a problem?
windows-7 windows windows-explorer file-transfer
edited Nov 24 at 23:45
Scott
15.5k113789
asked Nov 21 at 19:19
zar
4281615
When I start copying a very large folder (say 40 GB) in Windows 7 File Explorer, the rate of progress indicates that it may take hours to complete. So I leave it overnight, only to discover in the morning that it still claims to be trying to copy (it has failed but still appears to be in progress).
Is there a way to copy a large folder with high confidence that it will be successfully completed, or at least that it will conclude and display an error message if there is a problem?
windows-7 windows windows-explorer file-transfer
windows-7 windows windows-explorer file-transfer
edited Nov 24 at 23:45
Scott
15.5k113789
asked Nov 21 at 19:19
zar
4281615
edited Nov 24 at 23:45
Scott
15.5k113789
asked Nov 21 at 19:19
zar
4281615
edited Nov 24 at 23:45
Scott
15.5k113789
edited Nov 24 at 23:45
Scott
15.5k113789
edited Nov 24 at 23:45
Scott
15.5k113789
15.5k113789
asked Nov 21 at 19:19
zar
4281615
asked Nov 21 at 19:19
zar
4281615
asked Nov 21 at 19:19
zar
4281615
4281615
Last time I used Windows, it didn't even know if the destination had sufficient space for the file until it returned a 'disk full' error, half way through. Hopefully that has improved since Win7... but ...
– Tetsujin
Nov 21 at 19:30
@Tetsujin it really is a pain to get a large folder to copy via file explorer. This issue is not acknoledged too well. Using File explorer for large folders should be a no no to put it blunetly.
– zar
Nov 21 at 19:53
1
You've got a legit question buried in this, but it's worded in an off-topic way. It invites discussion, invites dispute about whether File Explorer is actually problematic, and asks for opinion on what's best. I'm voting to close on that basis, but will retract the vote if you can improve the question. Just state the problem you're facing and ask for a solution to it.
– fixer1234
Nov 24 at 1:16
@fixer1234 I have changed the title since.
– zar
Nov 24 at 16:14
Copying from one drive to another drive will be much faster than copying to the same drive. When copying to the same drive, the drive needs to keep finding, and moving between, the source and destination locations. If the folder contains a huge number of tiny files, it also needs to store directory information for each one. So that will take much longer than copying a few extremely large files. Huge folders won't be copied instantaneously, but 40 GB shouldn't take hours, even under the worst conditions (unless you are doing another drive-intensive, (cont'd)
– fixer1234
Nov 24 at 23:52
|
show 2 more comments
Last time I used Windows, it didn't even know if the destination had sufficient space for the file until it returned a 'disk full' error, half way through. Hopefully that has improved since Win7... but ...
– Tetsujin
Nov 21 at 19:30
@Tetsujin it really is a pain to get a large folder to copy via file explorer. This issue is not acknoledged too well. Using File explorer for large folders should be a no no to put it blunetly.
– zar
Nov 21 at 19:53
1
You've got a legit question buried in this, but it's worded in an off-topic way. It invites discussion, invites dispute about whether File Explorer is actually problematic, and asks for opinion on what's best. I'm voting to close on that basis, but will retract the vote if you can improve the question. Just state the problem you're facing and ask for a solution to it.
– fixer1234
Nov 24 at 1:16
@fixer1234 I have changed the title since.
– zar
Nov 24 at 16:14
Copying from one drive to another drive will be much faster than copying to the same drive. When copying to the same drive, the drive needs to keep finding, and moving between, the source and destination locations. If the folder contains a huge number of tiny files, it also needs to store directory information for each one. So that will take much longer than copying a few extremely large files. Huge folders won't be copied instantaneously, but 40 GB shouldn't take hours, even under the worst conditions (unless you are doing another drive-intensive, (cont'd)
– fixer1234
Nov 24 at 23:52
Last time I used Windows, it didn't even know if the destination had sufficient space for the file until it returned a 'disk full' error, half way through. Hopefully that has improved since Win7... but ...
– Tetsujin
Nov 21 at 19:30
Last time I used Windows, it didn't even know if the destination had sufficient space for the file until it returned a 'disk full' error, half way through. Hopefully that has improved since Win7... but ...
– Tetsujin
Nov 21 at 19:30
@Tetsujin it really is a pain to get a large folder to copy via file explorer. This issue is not acknoledged too well. Using File explorer for large folders should be a no no to put it blunetly.
– zar
Nov 21 at 19:53
@Tetsujin it really is a pain to get a large folder to copy via file explorer. This issue is not acknoledged too well. Using File explorer for large folders should be a no no to put it blunetly.
– zar
Nov 21 at 19:53
1
1
You've got a legit question buried in this, but it's worded in an off-topic way. It invites discussion, invites dispute about whether File Explorer is actually problematic, and asks for opinion on what's best. I'm voting to close on that basis, but will retract the vote if you can improve the question. Just state the problem you're facing and ask for a solution to it.
– fixer1234
Nov 24 at 1:16
You've got a legit question buried in this, but it's worded in an off-topic way. It invites discussion, invites dispute about whether File Explorer is actually problematic, and asks for opinion on what's best. I'm voting to close on that basis, but will retract the vote if you can improve the question. Just state the problem you're facing and ask for a solution to it.
– fixer1234
Nov 24 at 1:16
@fixer1234 I have changed the title since.
– zar
Nov 24 at 16:14
@fixer1234 I have changed the title since.
– zar
Nov 24 at 16:14
Copying from one drive to another drive will be much faster than copying to the same drive. When copying to the same drive, the drive needs to keep finding, and moving between, the source and destination locations. If the folder contains a huge number of tiny files, it also needs to store directory information for each one. So that will take much longer than copying a few extremely large files. Huge folders won't be copied instantaneously, but 40 GB shouldn't take hours, even under the worst conditions (unless you are doing another drive-intensive, (cont'd)
– fixer1234
Nov 24 at 23:52
Copying from one drive to another drive will be much faster than copying to the same drive. When copying to the same drive, the drive needs to keep finding, and moving between, the source and destination locations. If the folder contains a huge number of tiny files, it also needs to store directory information for each one. So that will take much longer than copying a few extremely large files. Huge folders won't be copied instantaneously, but 40 GB shouldn't take hours, even under the worst conditions (unless you are doing another drive-intensive, (cont'd)
– fixer1234
Nov 24 at 23:52
|
show 2 more comments
1 Answer
1
active
oldest
votes
up vote
0
down vote
Using windows file explorer to copy a large folder is, indeed, not very helpful for two reasons.
- The main reason is that the file copy may fail for one reason or another, but it never returns and it will appear as if copying is still in progress where, in fact, it'ts just stuck. This can waste a lot of time of user.
- If it failed due to error, it doesn't tell why it failed, what was the problem.
For example a file copy may fail because amid 40 GB data, there was one invalid shortcut where the target was missing. This will fail the copy but user will neer know and it will appear as if the copying is still in progress. There could be other reasons of failures that we we wouldn't know.
Solutions:
- It's best to copy large folders on command line and use `xcopy'. The benfit is that if it fails, it will return immediatly and you will know. That said it will still fail with above situation (invalid shortcut) but will error out correctly.
- The best Selection is to deploy a blind copy which doesn't care about the files but copy bits to bits. The best tools are
scporrsyncbut they are not readily available on Windows.
Using SCP on Windows
Install cygwin on Windows and use scpcommand from its shell the following way assuming to copy c:BigFolder to a USB drive F:BigFolderBackup/
scp -r /cygdrive/c/BigFolder/ /cygdrive/f/BigFolderBackup/
Where -r is for recurrsive copy which means copy all subfolders and their files.
Update 1
As noted by Scott in comments, since its copy on local machine so cp will do as well. Syntax is essentially identical.
Another scenario I came across is when a folder contain a file where I didn't had permssions to read. Our IT department doesn't allow users to run CCleaner utility so read permission is disabled on it.
I used cp this time and it performed nicely. It reported appropriate error on this file (didn't copy it) but didn't halt the copy operation and it continued with copying everything else. Another reason that this approach wins.
Update 2
I discovered scp (and probably cp) modifies file ownership which can cause problem with read/write access. Adding p flag should address that as noted in this article. Another way is to use rsync with -o and -g flags as noted in this answer to preserve ownership of files.
answered Nov 21 at 19:19
zar
4281615
If you have Cygwin, and you're not copying from one machine to another, why no just usecp(instead ofscp)?
– Scott
Nov 25 at 20:38
@Scott I guess that too, might be better in this case. It just didn't come on my radar since I had only used scp in the past
– zar
Nov 27 at 19:27
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
Using windows file explorer to copy a large folder is, indeed, not very helpful for two reasons.
- The main reason is that the file copy may fail for one reason or another, but it never returns and it will appear as if copying is still in progress where, in fact, it'ts just stuck. This can waste a lot of time of user.
- If it failed due to error, it doesn't tell why it failed, what was the problem.
For example a file copy may fail because amid 40 GB data, there was one invalid shortcut where the target was missing. This will fail the copy but user will neer know and it will appear as if the copying is still in progress. There could be other reasons of failures that we we wouldn't know.
Solutions:
- It's best to copy large folders on command line and use `xcopy'. The benfit is that if it fails, it will return immediatly and you will know. That said it will still fail with above situation (invalid shortcut) but will error out correctly.
- The best Selection is to deploy a blind copy which doesn't care about the files but copy bits to bits. The best tools are
scporrsyncbut they are not readily available on Windows.
Using SCP on Windows
Install cygwin on Windows and use scpcommand from its shell the following way assuming to copy c:BigFolder to a USB drive F:BigFolderBackup/
scp -r /cygdrive/c/BigFolder/ /cygdrive/f/BigFolderBackup/
Where -r is for recurrsive copy which means copy all subfolders and their files.
Update 1
As noted by Scott in comments, since its copy on local machine so cp will do as well. Syntax is essentially identical.
Another scenario I came across is when a folder contain a file where I didn't had permssions to read. Our IT department doesn't allow users to run CCleaner utility so read permission is disabled on it.
I used cp this time and it performed nicely. It reported appropriate error on this file (didn't copy it) but didn't halt the copy operation and it continued with copying everything else. Another reason that this approach wins.
Update 2
I discovered scp (and probably cp) modifies file ownership which can cause problem with read/write access. Adding p flag should address that as noted in this article. Another way is to use rsync with -o and -g flags as noted in this answer to preserve ownership of files.
answered Nov 21 at 19:19
zar
4281615
If you have Cygwin, and you're not copying from one machine to another, why no just usecp(instead ofscp)?
– Scott
Nov 25 at 20:38
@Scott I guess that too, might be better in this case. It just didn't come on my radar since I had only used scp in the past
– zar
Nov 27 at 19:27
add a comment |
up vote
0
down vote
Using windows file explorer to copy a large folder is, indeed, not very helpful for two reasons.
- The main reason is that the file copy may fail for one reason or another, but it never returns and it will appear as if copying is still in progress where, in fact, it'ts just stuck. This can waste a lot of time of user.
- If it failed due to error, it doesn't tell why it failed, what was the problem.
For example a file copy may fail because amid 40 GB data, there was one invalid shortcut where the target was missing. This will fail the copy but user will neer know and it will appear as if the copying is still in progress. There could be other reasons of failures that we we wouldn't know.
Solutions:
- It's best to copy large folders on command line and use `xcopy'. The benfit is that if it fails, it will return immediatly and you will know. That said it will still fail with above situation (invalid shortcut) but will error out correctly.
- The best Selection is to deploy a blind copy which doesn't care about the files but copy bits to bits. The best tools are
scporrsyncbut they are not readily available on Windows.
Using SCP on Windows
Install cygwin on Windows and use scpcommand from its shell the following way assuming to copy c:BigFolder to a USB drive F:BigFolderBackup/
scp -r /cygdrive/c/BigFolder/ /cygdrive/f/BigFolderBackup/
Where -r is for recurrsive copy which means copy all subfolders and their files.
Update 1
As noted by Scott in comments, since its copy on local machine so cp will do as well. Syntax is essentially identical.
Another scenario I came across is when a folder contain a file where I didn't had permssions to read. Our IT department doesn't allow users to run CCleaner utility so read permission is disabled on it.
I used cp this time and it performed nicely. It reported appropriate error on this file (didn't copy it) but didn't halt the copy operation and it continued with copying everything else. Another reason that this approach wins.
Update 2
I discovered scp (and probably cp) modifies file ownership which can cause problem with read/write access. Adding p flag should address that as noted in this article. Another way is to use rsync with -o and -g flags as noted in this answer to preserve ownership of files.
answered Nov 21 at 19:19
zar
4281615
If you have Cygwin, and you're not copying from one machine to another, why no just usecp(instead ofscp)?
– Scott
Nov 25 at 20:38
@Scott I guess that too, might be better in this case. It just didn't come on my radar since I had only used scp in the past
– zar
Nov 27 at 19:27
add a comment |
up vote
0
down vote
up vote
0
down vote
Using windows file explorer to copy a large folder is, indeed, not very helpful for two reasons.
- The main reason is that the file copy may fail for one reason or another, but it never returns and it will appear as if copying is still in progress where, in fact, it'ts just stuck. This can waste a lot of time of user.
- If it failed due to error, it doesn't tell why it failed, what was the problem.
For example a file copy may fail because amid 40 GB data, there was one invalid shortcut where the target was missing. This will fail the copy but user will neer know and it will appear as if the copying is still in progress. There could be other reasons of failures that we we wouldn't know.
Solutions:
- It's best to copy large folders on command line and use `xcopy'. The benfit is that if it fails, it will return immediatly and you will know. That said it will still fail with above situation (invalid shortcut) but will error out correctly.
- The best Selection is to deploy a blind copy which doesn't care about the files but copy bits to bits. The best tools are
scporrsyncbut they are not readily available on Windows.
Using SCP on Windows
Install cygwin on Windows and use scpcommand from its shell the following way assuming to copy c:BigFolder to a USB drive F:BigFolderBackup/
scp -r /cygdrive/c/BigFolder/ /cygdrive/f/BigFolderBackup/
Where -r is for recurrsive copy which means copy all subfolders and their files.
Update 1
As noted by Scott in comments, since its copy on local machine so cp will do as well. Syntax is essentially identical.
Another scenario I came across is when a folder contain a file where I didn't had permssions to read. Our IT department doesn't allow users to run CCleaner utility so read permission is disabled on it.
I used cp this time and it performed nicely. It reported appropriate error on this file (didn't copy it) but didn't halt the copy operation and it continued with copying everything else. Another reason that this approach wins.
Update 2
I discovered scp (and probably cp) modifies file ownership which can cause problem with read/write access. Adding p flag should address that as noted in this article. Another way is to use rsync with -o and -g flags as noted in this answer to preserve ownership of files.
answered Nov 21 at 19:19
zar
4281615
Using windows file explorer to copy a large folder is, indeed, not very helpful for two reasons.
- The main reason is that the file copy may fail for one reason or another, but it never returns and it will appear as if copying is still in progress where, in fact, it'ts just stuck. This can waste a lot of time of user.
- If it failed due to error, it doesn't tell why it failed, what was the problem.
For example a file copy may fail because amid 40 GB data, there was one invalid shortcut where the target was missing. This will fail the copy but user will neer know and it will appear as if the copying is still in progress. There could be other reasons of failures that we we wouldn't know.
Solutions:
- It's best to copy large folders on command line and use `xcopy'. The benfit is that if it fails, it will return immediatly and you will know. That said it will still fail with above situation (invalid shortcut) but will error out correctly.
- The best Selection is to deploy a blind copy which doesn't care about the files but copy bits to bits. The best tools are
scporrsyncbut they are not readily available on Windows.
Using SCP on Windows
Install cygwin on Windows and use scpcommand from its shell the following way assuming to copy c:BigFolder to a USB drive F:BigFolderBackup/
scp -r /cygdrive/c/BigFolder/ /cygdrive/f/BigFolderBackup/
Where -r is for recurrsive copy which means copy all subfolders and their files.
Update 1
As noted by Scott in comments, since its copy on local machine so cp will do as well. Syntax is essentially identical.
Another scenario I came across is when a folder contain a file where I didn't had permssions to read. Our IT department doesn't allow users to run CCleaner utility so read permission is disabled on it.
I used cp this time and it performed nicely. It reported appropriate error on this file (didn't copy it) but didn't halt the copy operation and it continued with copying everything else. Another reason that this approach wins.
Update 2
I discovered scp (and probably cp) modifies file ownership which can cause problem with read/write access. Adding p flag should address that as noted in this article. Another way is to use rsync with -o and -g flags as noted in this answer to preserve ownership of files.
answered Nov 21 at 19:19
zar
4281615
edited 9 hours ago
answered Nov 21 at 19:19
zar
4281615
answered Nov 21 at 19:19
zar
4281615
answered Nov 21 at 19:19
zar
4281615
4281615
If you have Cygwin, and you're not copying from one machine to another, why no just usecp(instead ofscp)?
– Scott
Nov 25 at 20:38
@Scott I guess that too, might be better in this case. It just didn't come on my radar since I had only used scp in the past
– zar
Nov 27 at 19:27
add a comment |
If you have Cygwin, and you're not copying from one machine to another, why no just usecp(instead ofscp)?
– Scott
Nov 25 at 20:38
@Scott I guess that too, might be better in this case. It just didn't come on my radar since I had only used scp in the past
– zar
Nov 27 at 19:27
If you have Cygwin, and you're not copying from one machine to another, why no just use
cp (instead of scp)?– Scott
Nov 25 at 20:38
If you have Cygwin, and you're not copying from one machine to another, why no just use
cp (instead of scp)?– Scott
Nov 25 at 20:38
@Scott I guess that too, might be better in this case. It just didn't come on my radar since I had only used scp in the past
– zar
Nov 27 at 19:27
@Scott I guess that too, might be better in this case. It just didn't come on my radar since I had only used scp in the past
– zar
Nov 27 at 19:27
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1377391%2fmasked-hidden-failures-copying-large-folder-with-file-explorer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Last time I used Windows, it didn't even know if the destination had sufficient space for the file until it returned a 'disk full' error, half way through. Hopefully that has improved since Win7... but ...
– Tetsujin
Nov 21 at 19:30
@Tetsujin it really is a pain to get a large folder to copy via file explorer. This issue is not acknoledged too well. Using File explorer for large folders should be a no no to put it blunetly.
– zar
Nov 21 at 19:53
1
You've got a legit question buried in this, but it's worded in an off-topic way. It invites discussion, invites dispute about whether File Explorer is actually problematic, and asks for opinion on what's best. I'm voting to close on that basis, but will retract the vote if you can improve the question. Just state the problem you're facing and ask for a solution to it.
– fixer1234
Nov 24 at 1:16
@fixer1234 I have changed the title since.
– zar
Nov 24 at 16:14
Copying from one drive to another drive will be much faster than copying to the same drive. When copying to the same drive, the drive needs to keep finding, and moving between, the source and destination locations. If the folder contains a huge number of tiny files, it also needs to store directory information for each one. So that will take much longer than copying a few extremely large files. Huge folders won't be copied instantaneously, but 40 GB shouldn't take hours, even under the worst conditions (unless you are doing another drive-intensive, (cont'd)
– fixer1234
Nov 24 at 23:52