Re: [Duplicity-talk] why restart takes so long

On Thu, 12 May 2016 at 02:25 <address@hidden> wrote:

Well, here are the first few lines from the log when I did the restart.
Since I can't see my own posts, I can't answer the question about the
traceback.

Using archive dir: /root/.cache/duplicity/backup_linux
Using backup name: backup_linux
Import of duplicity.backends.acdclibackend Succeeded
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.b2backend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.copycombackend Succeeded
Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.lftpbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.mediafirebackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.multibackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Succeeded
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Reading globbing filelist /etc/azure_excludes.txt
Main action: inc
================================================================================
duplicity 0.7.07 (April 10, 2016)
Args: /usr/bin/duplicity -v info --volsize 300 --gpg-options
--pinentry-mode loopback --exclude-filelist /etc/azure_excludes.txt
--name backup_linux --encrypt-key address@hidden / azure://linux
Linux ccs.covici.com 4.1.17-gentoo #1 SMP PREEMPT Mon Feb 15 15:05:32
EST 2016 x86_64 Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
/usr/lib/python-exec/python2.7/python2 2.7.11 (default, Jan 29 2016,
22:32:51)
[GCC 4.9.3]
================================================================================
Using temporary directory /tmp/duplicity-K67URk-tempdir
Temp has 37554356224 available, backup will use approx 408944640.
Local and Remote metadata are synchronized, no sync needed.
Last full backup left a partial set, restarting.
Last full backup date: Fri May 6 16:24:16 2016
RESTART: Volumes 45955 to 45955 failed to upload before termination.
Restarting backup at volume 45955.
Deleting /tmp/duplicity-K67URk-tempdir/mktemp-gLMnDl-2
Restarting after volume 45954, file
var/www/covici.com/htdocs-secure/owncloud/data/mattguice/files/20151114Town
Hall Meeting (Choral Prelude).mp3, block 977
A .
A audio
A audio/.mp3crc
A audio/0015.ps

And so on and so forth.

Kenneth Loafman <address@hidden> wrote:

> One quick question about the format of the traceback you show... is that
> because of the email client you use, or does it show up that way on the
> console?
>
> I'd like to see the block of data at the first of the log, say the first
> 200 lines. Please dump in raw text format.
>
>
> On Tue, May 10, 2016 at 2:19 PM, <address@hidden> wrote:
>
> > I have thousands of files, some large and some small, its about a
> > terabytes worth. It stopped pretty near the end and it just stopped
> > again, so my emphasis wuld be to fix the error and maybe do something
> > about optimizing the program.
> >
> >
> > Kenneth Loafman <address@hidden> wrote:
> >
> > > Do you have a lot of very small files? This should not take too long.
> > >
> > >
> > > On Tue, May 10, 2016 at 12:02 PM, <address@hidden> wrote:
> > >
> > > > Well, what happens is that it says A and all the filenames, but no
> > > > reference to a tar file, so its not really backing up again, this is
> > why
> > > > I was wondering what it is doing. The part of the program which seems
> > > > to be involved is this:
> > > > def restart_position_iterator(tarblock_iter):
> > > > """
> > > > Fake writing to backend, but do go through all the source
> > paths.
> > > > Stop when we have processed the last file and block
> > from the
> > > > last backup. Normal backup will proceed at the
> > > > start of
> > > > the
> > > > next volume in the set.
> > > >
> > > > address@hidden tarblock_iter: tarblock_iter
> > > > address@hidden tarblock_iter: iterator for current tar block
> > > >
> > > > address@hidden: int
> > > > address@hidden: constant 0 (zero)
> > > > """
> > > > last_index = globals.restart.last_index
> > > > last_block = globals.restart.last_block
> > > > try:
> > > > # Just spin our wheels
> > > > iter_result =
> > > > tarblock_iter.next()
> > > > while iter_result:
> > > > if (tarblock_iter.previous_index
> > ==
> > > > last_index):
> > > > # If both the previous index and this index
> > are
> > > > done, exit now
> > > > # before we hit the next index, to prevent
> > > > skipping its first
> > > > # block.
> > > > if not last_block
> > and
> > > > not
> > > > tarblock_iter.previous_block:
> > > > break
> > > > # Only check
> > > > block number if
> > > > last_block is also a number
> > > > if last_block and
> > tarblock_iter.previous_block >
> > > > last_block:
> > > > break
> > > > if
> > > > tarblock_iter.previous_index
> > > > > last_index:
> > > > log.Warn(_("File %s complete in backup
> > set.\n"
> > > >
> > "Continuing
> > > > restart
> > > > on file %s.") %
> > > >
> > > > Kenneth Loafman <address@hidden> wrote:
> > > >
> > > > > It sounds like the backup restarted completely, rather than at the
> > last
> > > > > known spot. What do the first 50 or so lines of the log contain?
> > > > >
> > > > > ...Ken
> > > > >
> > > > >
> > > > > On Tue, May 10, 2016 at 10:48 AM, <address@hidden> wrote:
> > > > >
> > > > > > But the question is why does duplicity need to do that at all?
> > > > > > Couldn't it go through the last hundred or so or some reasonable
> > > > number?
> > > > > > Also, what about the original error which caused the restart in the
> > > > > > first place? What is up with that that?
> > > > > >
> > > > > >
> > > > > > Kenneth Loafman <address@hidden> wrote:
> > > > > >
> > > > > > > No, there is no way to get rid of the loop, whichever one you're
> > > > talking
> > > > > > > about, there are a lot of them.
> > > > > > >
> > > > > > > The main suggestion I have for anyone with multi-hour backups is
> > to:
> > > > > > >
> > > > > > > 1. Use a local filesystem or USB drive for backup,
> > > > > > > 2. Transfer the backup using rsync or similar tool.
> > > > > > >
> > > > > > > The second suggestion I have is to chunk the backup into
> > reasonably
> > > > sized
> > > > > > > portions, 200-500GB each.
> > > > > > >
> > > > > > > ...Ken
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 10, 2016 at 7:02 AM, <address@hidden> wrote:
> > > > > > >
> > > > > > > > Hi. I had to restart my full backup and its going through
> > every
> > > > file
> > > > > > it
> > > > > > > > had and it is taking 18 hours and its not done yet. I see a
> > loop
> > > > in
> > > > > > > > there and I wonder if there is any way to shorten the time or
> > get
> > > > rid
> > > > > > of
> > > > > > > > the loop altoggether?
> > > > > > > >
> > > > > > > > Thanks in advance for any suggestions.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Your life is like a penny. You're going to lose it. The
> > question
> > > > is:
> > > > > > > > How do
> > > > > > > > you spend it?
> > > > > > > >
> > > > > > > > John Covici
> > > > > > > > address@hidden
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > Duplicity-talk mailing list
> > > > > > > > address@hidden
> > > > > > > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > > > > > >
> > > > > > >
> > > > > > > ----------------------------------------------------
> > > > > > > Alternatives:
> > > > > > >
> > > > > > > ----------------------------------------------------
> > > > > > > _______________________________________________
> > > > > > > Duplicity-talk mailing list
> > > > > > > address@hidden
> > > > > > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > > > >
> > > > > > --
> > > > > > Your life is like a penny. You're going to lose it. The question
> > is:
> > > > > > How do
> > > > > > you spend it?
> > > > > >
> > > > > > John Covici
> > > > > > address@hidden
> > > > > >
> > > > > > _______________________________________________
> > > > > > Duplicity-talk mailing list
> > > > > > address@hidden
> > > > > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > > > >
> > > > >
> > > > > ----------------------------------------------------
> > > > > Alternatives:
> > > > >
> > > > > ----------------------------------------------------
> > > > > _______________________________________________
> > > > > Duplicity-talk mailing list
> > > > > address@hidden
> > > > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > >
> > > > --
> > > > Your life is like a penny. You're going to lose it. The question is:
> > > > How do
> > > > you spend it?
> > > >
> > > > John Covici
> > > > address@hidden
> > > >
> > > > _______________________________________________
> > > > Duplicity-talk mailing list
> > > > address@hidden
> > > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > >
> > >
> > > ----------------------------------------------------
> > > Alternatives:
> > >
> > > ----------------------------------------------------
> > > _______________________________________________
> > > Duplicity-talk mailing list
> > > address@hidden
> > > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >
> > --
> > Your life is like a penny. You're going to lose it. The question is:
> > How do
> > you spend it?
> >
> > John Covici
> > address@hidden
> >
> > _______________________________________________
> > Duplicity-talk mailing list
> > address@hidden
> > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >
>
> ----------------------------------------------------
> Alternatives:
>
> ----------------------------------------------------
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk

--
Your life is like a penny. You're going to lose it. The question is:
How do
you spend it?

John Covici
address@hidden

_______________________________________________
Duplicity-talk mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

From:	Scott McKenzie
Subject:	Re: [Duplicity-talk] why restart takes so long
Date:	Wed, 11 May 2016 21:12:09 +0000