Re: [Qemu-devel] [PATCH] makefile: detect corrupted elf files

[Michael S. Tsirkin]

On Sun, May 26, 2013 at 08:29:35PM +0000, Blue Swirl wrote:
> On Sun, May 26, 2013 at 8:15 PM, Michael S. Tsirkin <address@hidden> wrote:
> > On Sun, May 26, 2013 at 07:28:40PM +0000, Blue Swirl wrote:
> >> On Sun, May 26, 2013 at 6:24 PM, Michael S. Tsirkin <address@hidden> wrote:
> >> > On Sun, May 26, 2013 at 06:20:17PM +0000, Blue Swirl wrote:
> >> >> On Sun, May 26, 2013 at 1:40 PM, Michael S. Tsirkin <address@hidden> 
> >> >> wrote:
> >> >> > On Sun, May 26, 2013 at 02:36:28PM +0100, Peter Maydell wrote:
> >> >> >> On 26 May 2013 13:31, Michael S. Tsirkin <address@hidden> wrote:
> >> >> >> > On Sun, May 26, 2013 at 10:12:21AM +0100, Peter Maydell wrote:
> >> >> >> >> I definitely think individual project makefiles are the wrong 
> >> >> >> >> place
> >> >> >> >> to fix this. If create-as-temp-and-rename is useful functionality
> >> >> >> >> it needs to go in the compiler so that everybody benefits.
> >> >> >> >
> >> >> >> > This will not help users on existing systems.
> >> >> >> > Also it's not just compiler. We'd have to do it in linker,
> >> >> >> > asm, ... lots of work.
> >> >> >>
> >> >> >> This is clearly less work than implementing it in the makefile
> >> >> >> of every single open source project in the world (or even every
> >> >> >> single open source project in Debian).
> >> >> >
> >> >> > You seem to have removed the part that explained that
> >> >> > 1. we run scripts in our makefiles so need to handle that anyway
> >> >> > 2. we care about users on existing systems
> >> >>
> >> >> A generic hook (default none, or maybe "test -s") after object
> >> >> production and before linkage should be enough but would scale to SHA
> >> >> producing/verifying tools.
> >> >>
> >> >> >
> >> >> > This means that we would need the fix in our makefiles even
> >> >> > if compiler and linker gain this feature.
> >> >>
> >> >> Depends on the feature. If the object files have robust checksums
> >> >> which are checked after output and before input, this should be
> >> >> transparent to the build system.
> >> >>
> >> >> >
> >> >> >> > You are wellcome to implement this in compiler/linker/etc if you 
> >> >> >> > like
> >> >> >> > but we will still want to handle it in our makefile as well.
> >> >> >>
> >> >> >> I specifically don't want it handled in our makefiles because
> >> >> >> it's the wrong place to fix the problem and it will make
> >> >> >> our build system more complicated.
> >> >>
> >> >> +1
> >> >>
> >> >> Also, what is the worst case scenario? The link fails and you have to
> >> >> clean up and rebuild? An automated build system can't produce the
> >> >> expected output if the build machine is unreliable?
> >> >
> >> > It's a simple issue.
> >> > Each time I reboot during build, I have to make clean and rebuild.
> >> > This wastes my time so I looked for ways to save the time.
> >>
> >> Compile under a stable kernel and test the bleeding edge kernel only
> >> as KVM guest? Get a different box for testing or compiling? Run 'sync'
> >> every time gcc finishes?
> >
> > What's the question here?
> 
> The question is if any of the suggestions solves the problem?
> 
> Also how about something this: post boot, find -name '*.o' | xargs -iF
> sh -c 'if test ! -s F; then rm F;fi'

Maybe. I don't know if it's the only kind one typically sees after a
power failure. I'll experiment when this happens next.

> >
> >> Don't you have bigger problems with file systems due to the crashes?
> >
> > As it happens, no. Maybe because I'm using ext4.
> > Maybe I'm lucky.
> >
> >> > On my system at least, it has no measureable cost,
> >> > likely also because size only looks at headers and metadata.
> >>
> >> For example on OpenBSD, 'size' does not seem to come from binutils, so
> >> there could be portability issues.
> >
> > True, I'm not saying it's perfect.
> >
> >> >
> >> > If others are not interested, I can keep it out of tree.
> >>
> >> I've had problems with disk close to full, so I'm semi-interested if
> >> the solution does not slow down others and it's not too ugly.
> >
> > I think the simplest way to do it is to change makefile to unlike, create
> > then rename. Then you are safe against abrupt killing or crashing make.
> > And with a journaled fs, if you also have e.g. linux ext4 and mount with
> > data=ordered, you are safe against power failures.
> >
> > It shouldn't be hard to do and I don't expect this to have any
> > measureable speed impact.  What do you think?
> 
> I'd prefer a more generic solution, like the hook.

What is meant by the hook?

> What you propose
> wouldn't protect from the disk full scenario.

Why not?
I think it will - renaming file in same directory doesn't need any space
so is almost sure to succeed even on disk full.

What I mean is this

gcc -o a.o a.c -> rm a.o && gcc -o a.tmp.o a.c && mv a.tmp.o a.o

> >
> >> >
> >> >> >>
> >> >> >> -- PMM
> >> >> >
> >> >> >