Re: [Duplicity-talk] (Option to) Cache retrieved volumes between (fetching/restoring) runs locally?!

[edgar . soldin]

On 10.11.2010 00:22, Daniel Hahler wrote:
> Hello,
> 
>> It would be much easier to simply mirror you repository to a local path and 
>> restore from there.
> 
> It would be ~30GB, which would take a while to mirror, apart from that
> you might not have the space available.
> 

if you are lacking the space, you probably have no space for the cache either.

>> Can't you just fetch/restore all files/folders in one go?
> 
> Does "duply fetch" / duplicity support fetching multiple files? It does
> not look so from the man page.

Indeed it doesn't. I always expected duplicity to work with exclude/include 
also on restoring. It doesn't. Currently you can obviously only restore all or 
one file/folder. For the latter. It is the folder only without it's contents ;).

You could of course restore all to a temp folder and move what you need to 
wherever needed. That's what I did whenever I had to restore.

> If you're referring to get the meat of it only, that would have been the
> root directory of all virtual containers (which is >90% of the whole
> backup).

when it is 90% of the backup, the last 10% download really do not weigh out the 
speed improvement once you restore from your local 'cache'.

Eventually I think it might be more important to make restore respect 
in/exclude, as a user would expect it to, instead of working around it by 
looping/caching over the backup.
Would you create a feature request on launchpad for that?

regards ..ede/duply.net

> 
> 
> Thanks,
> Daniel
> 
>> On 09.11.2010 21:53, Daniel Hahler wrote:
>>> Hello,
>>>
>>> I would like to be able to cache retrieved files from the backend
>>> locally between multiple runs of duplicity, e.g. via some config or
>>> command line option.
>>>
>>> Use case: having accidentally overwritten a lot of my (virtual)
>>> containers files, I've used the following to restore the previous state:
>>>   for i in $=VCIDS; do
>>>     b=path/to/files
>>>     /bin/rm -rf /$b*
>>>     duply profile fetch $b /$b 1D
>>>     duply profile fetch ${b}.d /${b}.d 1D
>>>   done
>>>
>>> This makes up 60+ runs of duplicity (2 runs of duplicity per 30+
>>> containers, one for a single file, the other for a directory), and when
>>> looking at it with "--verbosity 9" it looks like a lot of the same
>>> volumes (with a size of 50M in my case) are downloaded every time.
>>>
>>> I think it would speed this (particular) use case dramatically up, if
>>> these files would get cached locally.
>>>
>>> I could imagine to configure something like "keep files for X hours",
>>> and when duplicity gets run and files are older than this, they get
>>> cleaned on shutdown.
>>> When being accessed, they would get touched to reset the timer.
>>>
>>> However, there should also be a maximum number of files to cache, since
>>> this might easily fill your local volume otherwise.
>>>
>>> I am thinking about caching the files encrypted (just as on the remote
>>> site), but maybe caching decrypted files would make sense, too?
>>>
>>> Obviously, this should take into account if this is a remote backup
>>> (maybe by looking at the transfer rate of the files?!), and not pollute
>>> the cache, if the backend is as fast as local transfers might be.
>>>
>>> What do you think?
>>>
>>>
>>> Cheers,
>>> Daniel
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk