Are there drawbacks to sqlite as archivation format?
I have a few huge folders of small files I want to archive for long term storage (about 200GB).
Instead of just using something like gzip/tar/dar/7zip I am considering to use sqlite
sqlite3 Backup.db ".ar -cv Data"
Disregarding the compression, am I going towards some pitfalls? Are there any advantages old simple tar would have?
backup tar archiving sqlite file-archiver
asked Jan 25 at 13:59
KiriaKasisKiriaKasis
85
add a comment |
I have a few huge folders of small files I want to archive for long term storage (about 200GB).
Instead of just using something like gzip/tar/dar/7zip I am considering to use sqlite
sqlite3 Backup.db ".ar -cv Data"
Disregarding the compression, am I going towards some pitfalls? Are there any advantages old simple tar would have?
backup tar archiving sqlite file-archiver
asked Jan 25 at 13:59
KiriaKasisKiriaKasis
85
add a comment |
I have a few huge folders of small files I want to archive for long term storage (about 200GB).
Instead of just using something like gzip/tar/dar/7zip I am considering to use sqlite
sqlite3 Backup.db ".ar -cv Data"
Disregarding the compression, am I going towards some pitfalls? Are there any advantages old simple tar would have?
backup tar archiving sqlite file-archiver
asked Jan 25 at 13:59
KiriaKasisKiriaKasis
85
I have a few huge folders of small files I want to archive for long term storage (about 200GB).
Instead of just using something like gzip/tar/dar/7zip I am considering to use sqlite
sqlite3 Backup.db ".ar -cv Data"
Disregarding the compression, am I going towards some pitfalls? Are there any advantages old simple tar would have?
backup tar archiving sqlite file-archiver
backup tar archiving sqlite file-archiver
asked Jan 25 at 13:59
KiriaKasisKiriaKasis
85
asked Jan 25 at 13:59
KiriaKasisKiriaKasis
85
asked Jan 25 at 13:59
KiriaKasisKiriaKasis
85
asked Jan 25 at 13:59
KiriaKasisKiriaKasis
85
asked Jan 25 at 13:59
KiriaKasisKiriaKasis
85
85
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
The primary disadvantage: you need an sqlite3 executable with archive support when you need to recover your data. Depending on the circumstances, you may not have one handy, but you're far more likely to find a working gzip, tar et al.
Here's why I'm being particularly pedantic about this:
- macOS system
sqlite3versions are generally within spitting distance of up-to-date, but Apple only updates its software sporadically, and many Mac users may run backdated OSes for various reasons. Consequently, macOS systemsqlite3before Mojave does not support archiving, so for portability, you'll definitely want to ensure that the Homebrew sqlite3 package is installed and updated. - Homebrew on Linux defaults to installing a bottled (i.e. precompiled binary) sqlite v3.26 that somehow manages to not support archiving. I had to
brew install --build-from-source sqlite3to fix that.
Those are just two examples that I have personal experience with. The platforms on which you plan to sqlar may have similar gotchas. Contrast all that with "it just works everywhere" gzip/tar/etc.
So, if you want to use the sqlar format in earnest, carefully consider all the circumstances and environments in which you'll be doing so. A rescue disk, for instance, may not have an archiving sqlite3, and you really don't want to build sqlite from scratch during disaster recovery.
Aside from the availability of an archive-capable sqlite3, you also need to consider the issue of file sizes.
Since an sqlar is basically an SQLite DB, it inherits all the limits of this file format. In particular, each file's data is stored as a BLOB, which has a default maximum size of 1 billion bytes. Also, SQLite query processing leads to each row being handled as a single BLOB, further lowering the file size limit.
Compare that with ZIP (232 for "classic ZIP", 264 for ZIP64) or GNU tar (unlimited), and you may have yet another reason to take the traditional route instead.
answered Feb 11 at 3:42
AdrianAdrian
1413
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1398376%2fare-there-drawbacks-to-sqlite-as-archivation-format%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The primary disadvantage: you need an sqlite3 executable with archive support when you need to recover your data. Depending on the circumstances, you may not have one handy, but you're far more likely to find a working gzip, tar et al.
Here's why I'm being particularly pedantic about this:
- macOS system
sqlite3versions are generally within spitting distance of up-to-date, but Apple only updates its software sporadically, and many Mac users may run backdated OSes for various reasons. Consequently, macOS systemsqlite3before Mojave does not support archiving, so for portability, you'll definitely want to ensure that the Homebrew sqlite3 package is installed and updated. - Homebrew on Linux defaults to installing a bottled (i.e. precompiled binary) sqlite v3.26 that somehow manages to not support archiving. I had to
brew install --build-from-source sqlite3to fix that.
Those are just two examples that I have personal experience with. The platforms on which you plan to sqlar may have similar gotchas. Contrast all that with "it just works everywhere" gzip/tar/etc.
So, if you want to use the sqlar format in earnest, carefully consider all the circumstances and environments in which you'll be doing so. A rescue disk, for instance, may not have an archiving sqlite3, and you really don't want to build sqlite from scratch during disaster recovery.
Aside from the availability of an archive-capable sqlite3, you also need to consider the issue of file sizes.
Since an sqlar is basically an SQLite DB, it inherits all the limits of this file format. In particular, each file's data is stored as a BLOB, which has a default maximum size of 1 billion bytes. Also, SQLite query processing leads to each row being handled as a single BLOB, further lowering the file size limit.
Compare that with ZIP (232 for "classic ZIP", 264 for ZIP64) or GNU tar (unlimited), and you may have yet another reason to take the traditional route instead.
answered Feb 11 at 3:42
AdrianAdrian
1413
add a comment |
The primary disadvantage: you need an sqlite3 executable with archive support when you need to recover your data. Depending on the circumstances, you may not have one handy, but you're far more likely to find a working gzip, tar et al.
Here's why I'm being particularly pedantic about this:
- macOS system
sqlite3versions are generally within spitting distance of up-to-date, but Apple only updates its software sporadically, and many Mac users may run backdated OSes for various reasons. Consequently, macOS systemsqlite3before Mojave does not support archiving, so for portability, you'll definitely want to ensure that the Homebrew sqlite3 package is installed and updated. - Homebrew on Linux defaults to installing a bottled (i.e. precompiled binary) sqlite v3.26 that somehow manages to not support archiving. I had to
brew install --build-from-source sqlite3to fix that.
Those are just two examples that I have personal experience with. The platforms on which you plan to sqlar may have similar gotchas. Contrast all that with "it just works everywhere" gzip/tar/etc.
So, if you want to use the sqlar format in earnest, carefully consider all the circumstances and environments in which you'll be doing so. A rescue disk, for instance, may not have an archiving sqlite3, and you really don't want to build sqlite from scratch during disaster recovery.
Aside from the availability of an archive-capable sqlite3, you also need to consider the issue of file sizes.
Since an sqlar is basically an SQLite DB, it inherits all the limits of this file format. In particular, each file's data is stored as a BLOB, which has a default maximum size of 1 billion bytes. Also, SQLite query processing leads to each row being handled as a single BLOB, further lowering the file size limit.
Compare that with ZIP (232 for "classic ZIP", 264 for ZIP64) or GNU tar (unlimited), and you may have yet another reason to take the traditional route instead.
answered Feb 11 at 3:42
AdrianAdrian
1413
add a comment |
The primary disadvantage: you need an sqlite3 executable with archive support when you need to recover your data. Depending on the circumstances, you may not have one handy, but you're far more likely to find a working gzip, tar et al.
Here's why I'm being particularly pedantic about this:
- macOS system
sqlite3versions are generally within spitting distance of up-to-date, but Apple only updates its software sporadically, and many Mac users may run backdated OSes for various reasons. Consequently, macOS systemsqlite3before Mojave does not support archiving, so for portability, you'll definitely want to ensure that the Homebrew sqlite3 package is installed and updated. - Homebrew on Linux defaults to installing a bottled (i.e. precompiled binary) sqlite v3.26 that somehow manages to not support archiving. I had to
brew install --build-from-source sqlite3to fix that.
Those are just two examples that I have personal experience with. The platforms on which you plan to sqlar may have similar gotchas. Contrast all that with "it just works everywhere" gzip/tar/etc.
So, if you want to use the sqlar format in earnest, carefully consider all the circumstances and environments in which you'll be doing so. A rescue disk, for instance, may not have an archiving sqlite3, and you really don't want to build sqlite from scratch during disaster recovery.
Aside from the availability of an archive-capable sqlite3, you also need to consider the issue of file sizes.
Since an sqlar is basically an SQLite DB, it inherits all the limits of this file format. In particular, each file's data is stored as a BLOB, which has a default maximum size of 1 billion bytes. Also, SQLite query processing leads to each row being handled as a single BLOB, further lowering the file size limit.
Compare that with ZIP (232 for "classic ZIP", 264 for ZIP64) or GNU tar (unlimited), and you may have yet another reason to take the traditional route instead.
answered Feb 11 at 3:42
AdrianAdrian
1413
The primary disadvantage: you need an sqlite3 executable with archive support when you need to recover your data. Depending on the circumstances, you may not have one handy, but you're far more likely to find a working gzip, tar et al.
Here's why I'm being particularly pedantic about this:
- macOS system
sqlite3versions are generally within spitting distance of up-to-date, but Apple only updates its software sporadically, and many Mac users may run backdated OSes for various reasons. Consequently, macOS systemsqlite3before Mojave does not support archiving, so for portability, you'll definitely want to ensure that the Homebrew sqlite3 package is installed and updated. - Homebrew on Linux defaults to installing a bottled (i.e. precompiled binary) sqlite v3.26 that somehow manages to not support archiving. I had to
brew install --build-from-source sqlite3to fix that.
Those are just two examples that I have personal experience with. The platforms on which you plan to sqlar may have similar gotchas. Contrast all that with "it just works everywhere" gzip/tar/etc.
So, if you want to use the sqlar format in earnest, carefully consider all the circumstances and environments in which you'll be doing so. A rescue disk, for instance, may not have an archiving sqlite3, and you really don't want to build sqlite from scratch during disaster recovery.
Aside from the availability of an archive-capable sqlite3, you also need to consider the issue of file sizes.
Since an sqlar is basically an SQLite DB, it inherits all the limits of this file format. In particular, each file's data is stored as a BLOB, which has a default maximum size of 1 billion bytes. Also, SQLite query processing leads to each row being handled as a single BLOB, further lowering the file size limit.
Compare that with ZIP (232 for "classic ZIP", 264 for ZIP64) or GNU tar (unlimited), and you may have yet another reason to take the traditional route instead.
answered Feb 11 at 3:42
AdrianAdrian
1413
edited Feb 11 at 11:34
answered Feb 11 at 3:42
AdrianAdrian
1413
answered Feb 11 at 3:42
AdrianAdrian
1413
answered Feb 11 at 3:42
AdrianAdrian
1413
1413
add a comment |
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1398376%2fare-there-drawbacks-to-sqlite-as-archivation-format%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
