Backup files from server (static IP) to home server (Dynamic IP)
I am already familiar and use rsync, scp transfers, tar, 7z etc however none of which solve a backup I intend to do rather regularly.
I have a remote server with 80% used space on 100GB SSD that I would like to backup from. Server has 10,000+ small files so a basic tarball or some sort of intermediate compression (like rsync -z) will help.
I have a server at home that I would like to backup to but it uses Dynamic IP.
The trouble is to use rsync -z or scp and tar (via pipe), it has to be done on remote server but that server can't see my home server since its on Dynamic IP.
I can therefore only execute commands on my home server like scp user@remoteserver:/files/to/backup /home/user/backups but this can't use compression so the huge number of files takes very long time to transfer.
Is there a faster way?
P.S. I don't need to store file permissions etc that tarball does
Edit: Because of limited space on remote server, I can't make a tarball there and pull on home server. Therefore, it has to be realtime.
linux ssh backup rsync scp
migrated from serverfault.com Dec 1 at 9:30
This question came from our site for system and network administrators.
add a comment |
I am already familiar and use rsync, scp transfers, tar, 7z etc however none of which solve a backup I intend to do rather regularly.
I have a remote server with 80% used space on 100GB SSD that I would like to backup from. Server has 10,000+ small files so a basic tarball or some sort of intermediate compression (like rsync -z) will help.
I have a server at home that I would like to backup to but it uses Dynamic IP.
The trouble is to use rsync -z or scp and tar (via pipe), it has to be done on remote server but that server can't see my home server since its on Dynamic IP.
I can therefore only execute commands on my home server like scp user@remoteserver:/files/to/backup /home/user/backups but this can't use compression so the huge number of files takes very long time to transfer.
Is there a faster way?
P.S. I don't need to store file permissions etc that tarball does
Edit: Because of limited space on remote server, I can't make a tarball there and pull on home server. Therefore, it has to be realtime.
linux ssh backup rsync scp
migrated from serverfault.com Dec 1 at 9:30
This question came from our site for system and network administrators.
add a comment |
I am already familiar and use rsync, scp transfers, tar, 7z etc however none of which solve a backup I intend to do rather regularly.
I have a remote server with 80% used space on 100GB SSD that I would like to backup from. Server has 10,000+ small files so a basic tarball or some sort of intermediate compression (like rsync -z) will help.
I have a server at home that I would like to backup to but it uses Dynamic IP.
The trouble is to use rsync -z or scp and tar (via pipe), it has to be done on remote server but that server can't see my home server since its on Dynamic IP.
I can therefore only execute commands on my home server like scp user@remoteserver:/files/to/backup /home/user/backups but this can't use compression so the huge number of files takes very long time to transfer.
Is there a faster way?
P.S. I don't need to store file permissions etc that tarball does
Edit: Because of limited space on remote server, I can't make a tarball there and pull on home server. Therefore, it has to be realtime.
linux ssh backup rsync scp
I am already familiar and use rsync, scp transfers, tar, 7z etc however none of which solve a backup I intend to do rather regularly.
I have a remote server with 80% used space on 100GB SSD that I would like to backup from. Server has 10,000+ small files so a basic tarball or some sort of intermediate compression (like rsync -z) will help.
I have a server at home that I would like to backup to but it uses Dynamic IP.
The trouble is to use rsync -z or scp and tar (via pipe), it has to be done on remote server but that server can't see my home server since its on Dynamic IP.
I can therefore only execute commands on my home server like scp user@remoteserver:/files/to/backup /home/user/backups but this can't use compression so the huge number of files takes very long time to transfer.
Is there a faster way?
P.S. I don't need to store file permissions etc that tarball does
Edit: Because of limited space on remote server, I can't make a tarball there and pull on home server. Therefore, it has to be realtime.
linux ssh backup rsync scp
linux ssh backup rsync scp
asked Dec 1 at 9:13
Kapil802
migrated from serverfault.com Dec 1 at 9:30
This question came from our site for system and network administrators.
migrated from serverfault.com Dec 1 at 9:30
This question came from our site for system and network administrators.
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
Why can't use initiate the rsync from your backup target? There is no inherent restriction in rsync that would prevent this. Running
rsync -avz user@remote.example.com:/data /backupdir
on the backup target (home server) should do just that.
answered Dec 1 at 9:30
Sven
1,583910
I agree, I do exactly this kind of thing to backup my own server. Initiating the copy on the server opens a serious can of worms because your home server has to accept connections from the internet, it can of course be done but making sure it is (and remains) watertigh isn't so easy.
– xenoid
Dec 1 at 16:27
add a comment |
There are lots of options The easiest might be to just add the -C command to SSH or so which enables compression for the link.
A more sophisticated ,(and faster, be sure of lots f small files and latency) solution would be to drop scp and use a pipe to compress data on the fly. Something like tar -cz /path/to/backup | SSH user@remote "cd /path; tar -xz" will compress and decompress files using gzip compression. Of-course, there is nothing to stop you using a different command after ssh if, for example you just wanted to store the compressed file you could use "cat > /path/to/file.tar.gz" instead.
It's unclear to me why you are talking about dynamic IPs here. If this poses a problem, you can use dynamic DNS or set up a VPN with the server on the static or initiate a pull from the dynamic IP rather then push from the static.
answered Dec 1 at 9:44
davidgo
42k75086
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1379919%2fbackup-files-from-server-static-ip-to-home-server-dynamic-ip%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Why can't use initiate the rsync from your backup target? There is no inherent restriction in rsync that would prevent this. Running
rsync -avz user@remote.example.com:/data /backupdir
on the backup target (home server) should do just that.
answered Dec 1 at 9:30
Sven
1,583910
I agree, I do exactly this kind of thing to backup my own server. Initiating the copy on the server opens a serious can of worms because your home server has to accept connections from the internet, it can of course be done but making sure it is (and remains) watertigh isn't so easy.
– xenoid
Dec 1 at 16:27
add a comment |
Why can't use initiate the rsync from your backup target? There is no inherent restriction in rsync that would prevent this. Running
rsync -avz user@remote.example.com:/data /backupdir
on the backup target (home server) should do just that.
answered Dec 1 at 9:30
Sven
1,583910
I agree, I do exactly this kind of thing to backup my own server. Initiating the copy on the server opens a serious can of worms because your home server has to accept connections from the internet, it can of course be done but making sure it is (and remains) watertigh isn't so easy.
– xenoid
Dec 1 at 16:27
add a comment |
Why can't use initiate the rsync from your backup target? There is no inherent restriction in rsync that would prevent this. Running
rsync -avz user@remote.example.com:/data /backupdir
on the backup target (home server) should do just that.
answered Dec 1 at 9:30
Sven
1,583910
Why can't use initiate the rsync from your backup target? There is no inherent restriction in rsync that would prevent this. Running
rsync -avz user@remote.example.com:/data /backupdir
on the backup target (home server) should do just that.
answered Dec 1 at 9:30
Sven
1,583910
answered Dec 1 at 9:30
Sven
1,583910
answered Dec 1 at 9:30
Sven
1,583910
answered Dec 1 at 9:30
Sven
1,583910
1,583910
I agree, I do exactly this kind of thing to backup my own server. Initiating the copy on the server opens a serious can of worms because your home server has to accept connections from the internet, it can of course be done but making sure it is (and remains) watertigh isn't so easy.
– xenoid
Dec 1 at 16:27
add a comment |
I agree, I do exactly this kind of thing to backup my own server. Initiating the copy on the server opens a serious can of worms because your home server has to accept connections from the internet, it can of course be done but making sure it is (and remains) watertigh isn't so easy.
– xenoid
Dec 1 at 16:27
I agree, I do exactly this kind of thing to backup my own server. Initiating the copy on the server opens a serious can of worms because your home server has to accept connections from the internet, it can of course be done but making sure it is (and remains) watertigh isn't so easy.
– xenoid
Dec 1 at 16:27
I agree, I do exactly this kind of thing to backup my own server. Initiating the copy on the server opens a serious can of worms because your home server has to accept connections from the internet, it can of course be done but making sure it is (and remains) watertigh isn't so easy.
– xenoid
Dec 1 at 16:27
add a comment |
There are lots of options The easiest might be to just add the -C command to SSH or so which enables compression for the link.
A more sophisticated ,(and faster, be sure of lots f small files and latency) solution would be to drop scp and use a pipe to compress data on the fly. Something like tar -cz /path/to/backup | SSH user@remote "cd /path; tar -xz" will compress and decompress files using gzip compression. Of-course, there is nothing to stop you using a different command after ssh if, for example you just wanted to store the compressed file you could use "cat > /path/to/file.tar.gz" instead.
It's unclear to me why you are talking about dynamic IPs here. If this poses a problem, you can use dynamic DNS or set up a VPN with the server on the static or initiate a pull from the dynamic IP rather then push from the static.
answered Dec 1 at 9:44
davidgo
42k75086
add a comment |
There are lots of options The easiest might be to just add the -C command to SSH or so which enables compression for the link.
A more sophisticated ,(and faster, be sure of lots f small files and latency) solution would be to drop scp and use a pipe to compress data on the fly. Something like tar -cz /path/to/backup | SSH user@remote "cd /path; tar -xz" will compress and decompress files using gzip compression. Of-course, there is nothing to stop you using a different command after ssh if, for example you just wanted to store the compressed file you could use "cat > /path/to/file.tar.gz" instead.
It's unclear to me why you are talking about dynamic IPs here. If this poses a problem, you can use dynamic DNS or set up a VPN with the server on the static or initiate a pull from the dynamic IP rather then push from the static.
answered Dec 1 at 9:44
davidgo
42k75086
add a comment |
There are lots of options The easiest might be to just add the -C command to SSH or so which enables compression for the link.
A more sophisticated ,(and faster, be sure of lots f small files and latency) solution would be to drop scp and use a pipe to compress data on the fly. Something like tar -cz /path/to/backup | SSH user@remote "cd /path; tar -xz" will compress and decompress files using gzip compression. Of-course, there is nothing to stop you using a different command after ssh if, for example you just wanted to store the compressed file you could use "cat > /path/to/file.tar.gz" instead.
It's unclear to me why you are talking about dynamic IPs here. If this poses a problem, you can use dynamic DNS or set up a VPN with the server on the static or initiate a pull from the dynamic IP rather then push from the static.
answered Dec 1 at 9:44
davidgo
42k75086
There are lots of options The easiest might be to just add the -C command to SSH or so which enables compression for the link.
A more sophisticated ,(and faster, be sure of lots f small files and latency) solution would be to drop scp and use a pipe to compress data on the fly. Something like tar -cz /path/to/backup | SSH user@remote "cd /path; tar -xz" will compress and decompress files using gzip compression. Of-course, there is nothing to stop you using a different command after ssh if, for example you just wanted to store the compressed file you could use "cat > /path/to/file.tar.gz" instead.
It's unclear to me why you are talking about dynamic IPs here. If this poses a problem, you can use dynamic DNS or set up a VPN with the server on the static or initiate a pull from the dynamic IP rather then push from the static.
answered Dec 1 at 9:44
davidgo
42k75086
answered Dec 1 at 9:44
davidgo
42k75086
answered Dec 1 at 9:44
davidgo
42k75086
answered Dec 1 at 9:44
davidgo
42k75086
42k75086
add a comment |
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1379919%2fbackup-files-from-server-static-ip-to-home-server-dynamic-ip%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
