Re: [PATCH] Add wipename option to shred

[Pádraig Brady]

On 06/13/2013 05:13 PM, Joseph D. Wagner wrote:
> On 06/13/2013 8:35 am, Pádraig Brady wrote:
> 
>> On 06/13/2013 12:51 AM, Joseph D. Wagner wrote:
>>
>>> ## perchar ##
>>> real    678m33.468s
>>> user    0m9.450s
>>> sys    3m20.001s
>>>
>>> ## once ##
>>> real    151m54.655s
>>> user    0m3.336s
>>> sys    0m32.357s
>>>
>>> ## none ##
>>> real    107m34.307s
>>> user    0m2.637s
>>> sys    0m21.825s
>>>
>>> perchar: 11 hours 18 minutes 33.468 seconds
>>> once: 2 hours 31 minutes 54.655 seconds
>>>  * a 346% improvement over perchar
>>> none: 1 hour 47 minutes 34.307 seconds
>>>  * a 530% improvement over perchar
>>>  * a 41% improvement over once
>>
>> Whoa, so this creates 23s CPU work
>> but waits for 1 hour 47 mins on the sync!
>> What file system and backing device are you using here
>> as a matter of interest?
> 
> ext4 data=ordered (default) + 7200 SATA
> 
> Just to be clear, the times also include shredding the data part of the files.
>
> For my test I used 16 character file names and 100,000 files each 4k in size,
> which comes to:
> perchar: (1 data fsync + 16 name fsync) * 100,000 files = 1,700,000 fsync
> once: (1 data fsync + 1 name fsync) * 100,000 files = 200,000 fsync
> none: (1 data fsync + 0 name fsync) * 100,000 files = 100,000 fsync
> 
> I included the exact script I used to generate those statistics in a previous
> email.  Feel free to replicate my experiment on your own equipment, using my
> patched version of shred of course.
> 
> Alternatively, if you still have reservations about adopting my patch,
> would you be more open to a --no-wipename option?  This would be the
> equivalent of my proposed --wipename=none.  It would not imply any
> additional security; to the contrary, it implies less security.  Yet, it
> would still give me the optional performance boost I am trying to
> achieve.

Yes these sync latencies really add up.

I timed this simple test script on ext4 on an SSD
and traditional disk in my laptop:

  import os
  d=os.open(".", os.O_DIRECTORY|os.O_RDONLY)
  for i in range(1000):
    os.fdatasync(d)

That gave 2ms and 12ms per sync operation respectively.
This seems to be independent of dir size and whether any
changes were made to the dir, which is a bit surprising.
Seems like there could be only sync on change optimizations possible.
Anyway...

So with the extra 1.6M syncs above on spinning rust,
that would add an extra 5.3 hours by my calc.
Your latencies seemed to be nearly double that,
but fair enough, same ball park.

Now we could handle this outside of shred, if we only
wanted to choose between wiping names and simple delete.
Given that the above latencies, the overhead of a couple
of microseconds to start a process per file is insignificant:

find /files | xargs -n1 -I{} sh -c 'shred "{}" && rm "{}"'

But yes this is a bit awkward.
Also if you did want to select wipe, but avoid the explicit syncs,
because you knew your file system had synchronous metadata updates
then we couldn't support that operation with this scheme.

So I'm leaning a bit towards adding control through shred options.
So how about this interface:

-u, --remove[=HOW]
    truncate and remove file after overwriting.
    HOW indicates how to remove the directory entry:
    unlink => just call unlink, wipe => also first obfuscate the name,
    wipesync => also sync each obfuscated character to disk (the default)

thanks,
Pádraig.