Re: [Bug-ddrescue] Proposed ddrescue improvement

[Neil Steiner]

Hi address@hidden and Florian,

Thank you again for taking the time to reply to me.  I wouldn't have 
expected a development team to offer end-user advice, but I much 
appreciate it.

When I last saw the previous ddrescue process, it was at 259,995 MB out 
of 260,000 MB rescued.  I know for sure that the vast majority of the 
recovery file was allocated, including the very last one sector, and 
since HFS+ does not support sparse files, I believe Florian must be 
correct that the allocation blocks will not have changed.

I have now used the generate mode (thanks for the suggestion) to create 
a new map file, and I am going to use ddrescuelog (thanks for the 
suggestion) to OR the last resumed known-good map file (somewhere over 
100,000 MB known good now) with the generated map file.  When that 
completes, I will run the remainder of the recovery and then try an fsck 
on the image.  If I seem to have lost huge amounts of stuff, I will fall 
back to the 100,000 MB known good map and keep going from there.  But if 
fsck is happy or mostly happy, then this will be a success.

Again, many thanks for your suggestions and for opening my eyes to even 
more ddrescue features.  (And hello to Hungary!)

Neil

Florian Sedivy wrote:
> Hi Neil!
>
> All is not lost, maybe. It is possible that the HFS+ on your destination drive 
> had its journal rolled back after the power failure. That could lead to an 
> older version of the map file (allocated on different blocks) being shown 
> instead of the latest. But as you said the destination image file had been 
> fully allocated before, its block allocation hopefully did not change, so it's 
> content might be more current than the directory entry.
>
> You might scavenge unused blocks on the destination disk to rescue more current 
> versions of the map file. As you know how it starts, that should be easy. But 
> as OS X fills sparse files with zeros on closing, you would also get a pretty 
> good map file using ddrescue's -G mode to generate one from the destination 
> image file.
>
> Comparing the map file you found after the power failure (or, if you didn't 
> make a copy of that, the current map file) to the generated or rescued map file 
> using ddrescuelog should give you a better picture of how much is really left 
> to be rescued (some of it again). After that you will probably decide to 
> continue the rescue with the latter map file.
>
> Florian
>
> > Am 06.11.2015 um 18:30 schrieb Neil Steiner <address@hidden>:
> >
> > Hi Matt,
> >
> > Thanks for the response.  I agree that my machine couldn't possibly cache 180 
> > GB.  What I do know however is that the last modification date in the recovery 
> > directory was 11/1 at about 10:00am (as of 11/4 at 6:00pm), and the map file 
> > certainly seemed consistent with that.
> >
> > The actual recovery file must surely be more up-to-date as you say, but with a 
> > stale map file I wouldn't know what I can safely trust other than what it tells 
> > me.  (The recovery file was already fully allocated by 11/1 in my very 
> > nonlinear passes, so its size yield no clues.)
> >
> > I'll admit that I didn't think to look at the log file, but again the most 
> > recent modification date anywhere in the recovery directory was 1/11 so I doubt 
> > the log file would tell me more.
> >
> > The target filesystem is HFS+.  The source is an external USB drive.  I was 
> > running in single-user mode on an old Mac Mini because its patience made it far 
> > more effective than my MacBook Pro at recovering data.
> >
> > Neil
> >
> > Sent from my iPhone
> >
> > On Nov 6, 2015, at 9:03 AM, Matt Ruffalo <address@hidden> wrote:
> >
> > > Hi Neil-
> > >
> > > I think it's extremely unlikely that the OS would cache 180GB in RAM (of
> > > course, it's not even possible unless you were running the recovery on a
> > > machine with at least that much memory).
> > >
> > > What is telling you that only 80GB has been recovered? The size of the
> > > recovered image, or the state saved in the log file? Hopefully the data
> > > in the disk image is present, and the log file simply doesn't reflect
> > > everything that has been recovered.
> > >
> > > You mentioned synchronous writes to a journaled filesystem -- I assume
> > > that you were writing the log file to this filesystem, but were you also
> > > copying the disk to a file on this filesystem? Which filesystem, out of
> > > curiosity? I wouldn't be surprised if you lost a few hundred megabytes
> > > after a power outage but would be very surprised if 180GB disappeared.
> > >
> > > MMR...
> > >
> > > On 2015-11-05 19:50, Neil Steiner wrote:
> > > > Hello ddrescue Team,
> > > >
> > > > First of all, thank you for the wonderful tool that ddrescue is.  I
> > > > have a small suggestion that might benefit other ddrescue users as it
> > > > would have benefited me:  Adding a command line option to enable
> > > > periodic sync() calls, to force the OS to flush its buffers.
> > > >
> > > > My case is as follows:  I have been working on recovering data from a
> > > > friend's dying drive.  After much experimenting with both command line
> > > > arguments and the hardware (cooling/freezing the drive, using an older
> > > > more lenient computer, ...), I had finally managed to recover 259,994
> > > > MB of the requested 260,000 MB, and I was already planning my fschk on
> > > > the recovered data.  But today I apparently lost power for a brief
> > > > while, and even though I was using -y for synchronous writes on a
> > > > journaled filesystem, the OS had apparently been caching most
> > > > everything in memory.  I'm back down to only 80,000 MB recovered, and
> > > > hoping that I'll be able to regain the additional 80,000 MB that has
> > > > disappeared.  (Yes, it's my own fault for not having a suitable UPS on
> > > > that machine, and for being lulled into complacency by ddrescue's
> > > > remarkable ability to pick up where it left off.)
> > > >
> > > > I think this proposed additional command line switch should be fairly
> > > > simple to implement, and I expect that it would have very little
> > > > impact on the remainder of the tool functionality.  I am happy to
> > > > discuss this in greater detail if that would be helpful.
> > > >
> > > > Regards,
> > > >
> > > > Neil
> > > >
> > > >
> > > > _______________________________________________
> > > > Bug-ddrescue mailing list
> > > > address@hidden
> > > > https://lists.gnu.org/mailman/listinfo/bug-ddrescue
> > _______________________________________________
> > Bug-ddrescue mailing list
> > address@hidden
> > https://lists.gnu.org/mailman/listinfo/bug-ddrescue