Re: [Duplicity-talk] FTP not working

[Tim Riemenschneider]

Tim Riemenschneider schrieb:
> Hi,
>
> I think I have a problem with the ftp-backend. collection-status shows
> as empty, but there are backups on the server. I think ncftpls is not
> working correctly.
>   
Ok, trying further revealed the problem: the ftp-server seems to choke
on the default options of ls, since the first call of ls (long listing)
works, but the second doesn't:

SERVER:~/backup# ncftpls -f ../mkstemp-vC3HJk-1 -F -t 30 -o
useCLNT=0,useHELP_SITE=0 -l ftp://BACKUPHOST/PATH/
drwxr-xr-x    2 iwbackup iwbackup     4096 Apr  3 13:23 .
drwxr-xr-x    3 iwbackup iwbackup     4096 Mar 27 17:03 ..
-rw-r--r--    1 iwbackup iwbackup 11778590 Mar 28 00:49
duplicity-full-signatures.20090328T000007Z.sigtar.gpg
-rw-r--r--    1 iwbackup iwbackup      989 Mar 28 00:48
duplicity-full.20090328T000007Z.manifest.gpg
-rw-r--r--    1 iwbackup iwbackup 26192373 Mar 28 00:04
duplicity-full.20090328T000007Z.vol1.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26232496 Mar 28 00:39
duplicity-full.20090328T000007Z.vol10.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26180213 Mar 28 00:43
duplicity-full.20090328T000007Z.vol11.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26251300 Mar 28 00:47
duplicity-full.20090328T000007Z.vol12.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup  5266124 Mar 28 00:48
duplicity-full.20090328T000007Z.vol13.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26201358 Mar 28 00:07
duplicity-full.20090328T000007Z.vol2.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26246924 Mar 28 00:11
duplicity-full.20090328T000007Z.vol3.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26262286 Mar 28 00:15
duplicity-full.20090328T000007Z.vol4.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26261917 Mar 28 00:19
duplicity-full.20090328T000007Z.vol5.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26215495 Mar 28 00:22
duplicity-full.20090328T000007Z.vol6.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26254581 Mar 28 00:27
duplicity-full.20090328T000007Z.vol7.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26248036 Mar 28 00:31
duplicity-full.20090328T000007Z.vol8.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26176462 Mar 28 00:35
duplicity-full.20090328T000007Z.vol9.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26212310 Apr  3 12:41
duplicity-full.20090403T103847Z.vol1.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26252742 Apr  3 13:21
duplicity-full.20090403T103847Z.vol10.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26181848 Apr  3 13:23
duplicity-full.20090403T103847Z.vol11.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26180652 Apr  3 12:44
duplicity-full.20090403T103847Z.vol2.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26256169 Apr  3 12:46
duplicity-full.20090403T103847Z.vol3.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26254417 Apr  3 12:49
duplicity-full.20090403T103847Z.vol4.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26252116 Apr  3 13:00
duplicity-full.20090403T103847Z.vol5.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26219357 Apr  3 13:09
duplicity-full.20090403T103847Z.vol6.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26257988 Apr  3 13:13
duplicity-full.20090403T103847Z.vol7.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26201299 Apr  3 13:15
duplicity-full.20090403T103847Z.vol8.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup 26247716 Apr  3 13:18
duplicity-full.20090403T103847Z.vol9.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup      228 Mar 29 00:03
duplicity-inc.20090328T000007Z.to.20090329T000008Z.manifest.gpg
-rw-r--r--    1 iwbackup iwbackup  1417577 Mar 29 00:03
duplicity-inc.20090328T000007Z.to.20090329T000008Z.vol1.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup      228 Mar 30 00:02
duplicity-inc.20090329T000008Z.to.20090329T230007Z.manifest.gpg
-rw-r--r--    1 iwbackup iwbackup    28255 Mar 30 00:02
duplicity-inc.20090329T000008Z.to.20090329T230007Z.vol1.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup      216 Mar 31 00:03
duplicity-inc.20090329T230007Z.to.20090330T230008Z.manifest.gpg
-rw-r--r--    1 iwbackup iwbackup   279926 Mar 31 00:03
duplicity-inc.20090329T230007Z.to.20090330T230008Z.vol1.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup      229 Apr  1 00:02
duplicity-inc.20090330T230008Z.to.20090331T230007Z.manifest.gpg
-rw-r--r--    1 iwbackup iwbackup   146709 Apr  1 00:02
duplicity-inc.20090330T230008Z.to.20090331T230007Z.vol1.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup      230 Apr  2 00:03
duplicity-inc.20090331T230007Z.to.20090401T230008Z.manifest.gpg
-rw-r--r--    1 iwbackup iwbackup  2576211 Apr  2 00:03
duplicity-inc.20090331T230007Z.to.20090401T230008Z.vol1.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup      227 Apr  3 00:03
duplicity-inc.20090401T230008Z.to.20090402T230008Z.manifest.gpg
-rw-r--r--    1 iwbackup iwbackup  1630497 Apr  3 00:03
duplicity-inc.20090401T230008Z.to.20090402T230008Z.vol1.difftar.gpg
-rw-r--r--    1 iwbackup iwbackup    91236 Mar 29 00:03
duplicity-new-signatures.20090328T000007Z.to.20090329T000008Z.sigtar.gpg
-rw-r--r--    1 iwbackup iwbackup    41702 Mar 30 00:02
duplicity-new-signatures.20090329T000008Z.to.20090329T230007Z.sigtar.gpg
-rw-r--r--    1 iwbackup iwbackup    52597 Mar 31 00:03
duplicity-new-signatures.20090329T230007Z.to.20090330T230008Z.sigtar.gpg
-rw-r--r--    1 iwbackup iwbackup    50853 Apr  1 00:03
duplicity-new-signatures.20090330T230008Z.to.20090331T230007Z.sigtar.gpg
-rw-r--r--    1 iwbackup iwbackup    90746 Apr  2 00:03
duplicity-new-signatures.20090331T230007Z.to.20090401T230008Z.sigtar.gpg
-rw-r--r--    1 iwbackup iwbackup    92901 Apr  3 00:03
duplicity-new-signatures.20090401T230008Z.to.20090402T230008Z.sigtar.gpg
SERVER:~/backup# ncftpls -f ../mkstemp-vC3HJk-1 -F -t 30 -o
useCLNT=0,useHELP_SITE=0 ftp://BACKUPHOST/PATH/
SERVER:~/backup#

playing with the -x option of ncftpls:

SERVER:~/backup# ncftpls -f ../mkstemp-vC3HJk-1 -F -t 30 -o
useCLNT=0,useHELP_SITE=0 -x "-a" ftp://BACKUPHOST/PATH/
SERVER:~/backup# ncftpls -f ../mkstemp-vC3HJk-1 -F -t 30 -o
useCLNT=0,useHELP_SITE=0 -x "" ftp://BACKUPHOST/PATH/
.pureftpd-upload.49d5f216.15.2283.30d444d0
duplicity-full-signatures.20090328T000007Z.sigtar.gpg
duplicity-full.20090328T000007Z.manifest.gpg
duplicity-full.20090328T000007Z.vol1.difftar.gpg
duplicity-full.20090328T000007Z.vol10.difftar.gpg
duplicity-full.20090328T000007Z.vol11.difftar.gpg
duplicity-full.20090328T000007Z.vol12.difftar.gpg
duplicity-full.20090328T000007Z.vol13.difftar.gpg
duplicity-full.20090328T000007Z.vol2.difftar.gpg
duplicity-full.20090328T000007Z.vol3.difftar.gpg
duplicity-full.20090328T000007Z.vol4.difftar.gpg
duplicity-full.20090328T000007Z.vol5.difftar.gpg
duplicity-full.20090328T000007Z.vol6.difftar.gpg
duplicity-full.20090328T000007Z.vol7.difftar.gpg
duplicity-full.20090328T000007Z.vol8.difftar.gpg
duplicity-full.20090328T000007Z.vol9.difftar.gpg
duplicity-full.20090403T103847Z.vol1.difftar.gpg
duplicity-full.20090403T103847Z.vol10.difftar.gpg
duplicity-full.20090403T103847Z.vol11.difftar.gpg
duplicity-full.20090403T103847Z.vol2.difftar.gpg
duplicity-full.20090403T103847Z.vol3.difftar.gpg
duplicity-full.20090403T103847Z.vol4.difftar.gpg
duplicity-full.20090403T103847Z.vol5.difftar.gpg
duplicity-full.20090403T103847Z.vol6.difftar.gpg
duplicity-full.20090403T103847Z.vol7.difftar.gpg
duplicity-full.20090403T103847Z.vol8.difftar.gpg
duplicity-full.20090403T103847Z.vol9.difftar.gpg
duplicity-inc.20090328T000007Z.to.20090329T000008Z.manifest.gpg
duplicity-inc.20090328T000007Z.to.20090329T000008Z.vol1.difftar.gpg
duplicity-inc.20090329T000008Z.to.20090329T230007Z.manifest.gpg
duplicity-inc.20090329T000008Z.to.20090329T230007Z.vol1.difftar.gpg
duplicity-inc.20090329T230007Z.to.20090330T230008Z.manifest.gpg
duplicity-inc.20090329T230007Z.to.20090330T230008Z.vol1.difftar.gpg
duplicity-inc.20090330T230008Z.to.20090331T230007Z.manifest.gpg
duplicity-inc.20090330T230008Z.to.20090331T230007Z.vol1.difftar.gpg
duplicity-inc.20090331T230007Z.to.20090401T230008Z.manifest.gpg
duplicity-inc.20090331T230007Z.to.20090401T230008Z.vol1.difftar.gpg
duplicity-inc.20090401T230008Z.to.20090402T230008Z.manifest.gpg
duplicity-inc.20090401T230008Z.to.20090402T230008Z.vol1.difftar.gpg
duplicity-new-signatures.20090328T000007Z.to.20090329T000008Z.sigtar.gpg
duplicity-new-signatures.20090329T000008Z.to.20090329T230007Z.sigtar.gpg
duplicity-new-signatures.20090329T230007Z.to.20090330T230008Z.sigtar.gpg
duplicity-new-signatures.20090330T230008Z.to.20090331T230007Z.sigtar.gpg
duplicity-new-signatures.20090331T230007Z.to.20090401T230008Z.sigtar.gpg
duplicity-new-signatures.20090401T230008Z.to.20090402T230008Z.sigtar.gpg
IW-Watch:~/backup#

(yes, there is a backup currently in progress. the older backups where
done with duplicity 0.5.12, today I upgraded to 0.5.14, which reverted
to a full-backup because it thinks, the directory is empty)

So, after hacking ftpbackend.py:
--- ftpbackend.py.orig  2009-04-02 17:51:52.000000000 +0200
+++ ftpbackend.py       2009-04-03 13:33:50.000000000 +0200
@@ -109,7 +109,7 @@
         if not l:
             return l
         # if long list is not empty, get short list of names only
-        commandline = "ncftpls %s '%s'" % \
+        commandline = "ncftpls %s -x '' '%s'" % \
             (self.flags, self.url_string)
         l = self.popen_persist(commandline).split('\n')
         l = [x.split()[-1] for x in l if x]

it works (the backup is complete now ;-) ):
SERVER:~/backup# duplicity collection-status $TARGET
NcFTP version is 3.2.1
Warning, found an unnecessary signature chain
duplicity-full-signatures.20090328T000007Z.sigtar.gpg
Last full backup date: Fri Apr  3 12:38:47 2009
Collection Status
-----------------
Connecting with backend: FTPBackend
Archive dir: None

Found 1 backup chain without signatures.
Signature-less chain 1 of 1:
-------------------------
Chain start time: Sat Mar 28 01:00:07 2009
Chain end time: Fri Apr  3 01:00:08 2009
Number of contained backup sets: 7
Total number of contained volumes: 19
 Type of backup set:                            Time:      Num volumes:
                Full         Sat Mar 28 01:00:07 2009                13
         Incremental         Sun Mar 29 01:00:08 2009                 1
         Incremental         Mon Mar 30 01:00:07 2009                 1
         Incremental         Tue Mar 31 01:00:08 2009                 1
         Incremental         Wed Apr  1 01:00:07 2009                 1
         Incremental         Thu Apr  2 01:00:08 2009                 1
         Incremental         Fri Apr  3 01:00:08 2009                 1
-------------------------


Found a complete backup chain with matching signature chain:
-------------------------
Chain start time: Fri Apr  3 12:38:47 2009
Chain end time: Fri Apr  3 12:38:47 2009
Number of contained backup sets: 1
Total number of contained volumes: 13
 Type of backup set:                            Time:      Num volumes:
                Full         Fri Apr  3 12:38:47 2009                13
-------------------------
No orphaned or incomplete backup sets found.
SERVER:~/backup#

Can my change break something? (Otherwise it could probably go into the
source?)

cu
Tim

-- 
detco GbR
Tim Riemenschneider - Entwickler
Haferwende 27
28357 Bremen

Geschäftsführer: Sebastian Constantin, Jens Frischen
Phone:  0421 / 22 37 99 00
Fax:    0421 / 22 37 99 01
Mobil:  0179 / 21 27 285
Mail:   address@hidden


Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. 
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten 
haben, informieren Sie bitte sofort den Absender und löschen Sie diese Mail. 
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der 
darin enthaltenen Informationen sind nicht gestattet.


This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and delete this e-mail. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly forbidden.