Re: general partition questions

[paddy]

On Sat, Nov 13, 2004 at 12:08:16PM +0100, B.Hakvoort wrote:
> Hi all :)
> 
> On Fri, 2004-11-12 at 14:52 +0000, paddy wrote: 
> <snip>
> > dd is a fundamental tool.  But, it really doesn't do much more than put 
> > bells and whistles around read(2) and write(2).  A few thoughts:
> > 
> Which makes me wonder.. What if i don't use 'dd' and implement something 
> simple myself:
> 
> -----PSEUDOCODE-----------------
> INPUT = /dev/hda1
> OUTPUT = /dev/hda2
> 
> while ( INPUT .read_byte( c ) )
>       OUTPUT .write_byte( c ) ;
> --------------------------------------
> 
> Afaics this should work.. any disadvantages i should be aware of?
> One huge advantage of this approach is the fact i can implement some 
> excellent progressfeedback.
> 
> > 
> >     1. large file support has been variable in the past
> How large is large? I've been able to test it with 10Gb (no bigger disks 
> available) without problems.
> Some googling didn't show up much usefull either. Maybe you could elaborate a 
> bit? Thanks! :)

Indeed, the suggestion is that you should at least take the take the time 
to familiarise yourself with the source of dd in general, and with your
chosen implementation in particular.  And of course, from what you were
saying, you may be thinking terms of working with whatever random dd
exists at the location you arrive at.  Given the simplicity of dd, you
may then prefer to have more control, by writing to read(2) and write(2)
as interfaces rather than dd.  Of course all this may be moot if there
is some interface in parted, or some other library, that is better for 
the job, but even then I would expect to find calls to read and write
at the bottom of it all.

You probably don't want to be doing too much in the way of sub-block
(byte size rather than integral number of disk blocks) manipulations
with dd.  Not only is it fidly, but the various overheads
could be considerable.  It can serve well for the occasional bit of 
cut'n'paste from the command line. Bear in mind that, for typical
IDE/SCSI hardware, the fundamental unit for reading an writing is a
block.

The key to getting maximum throughput is to be transfering data 
continuously on whichever interface is the bottleneck.  IIRC stock
gnu dd does a blocking read followed by a blocking write (rinse, repeat)
so you see at best half the possible throughput.  You can work round
this somehow ( dd if=/dev/hda | dd of=/dev/hdc ???), but I don't
remember the details.  By the same token I would usually set the block
size (when working on dos style partitions) to a track or a cylinder,
as this plays better with the mechanics further down the food chain.
Tools like hdparm and time are your friends here.

dd is not just a tool for block devices, it will deal with streams,
which are potentially unseekable.  It used to the case that gnu dd
on linux would use the 32-bit seek call, and then fall back to spooling
through the file. So, if you were to be moving an 8G partion up the
disk a cylinder (or whatever) at a time, you really wanted to have
a dd that could do the 64-bit seek.  This is probably not currently an 
issue on typical modern systems.  However, the stream oriented aspect 
of dd seems worth pointing out to anyone looking to use it for 
manipulating block devices.

In general I would say you can do quite a lot of basic parted type
stuff from the shell with nothing more than dd, but more complex tasks 
like filesystem resizing really call for 'a real programming language',
or at least helper applications.

> On Sat, 2004-11-13 at 00:51 +0100, Szakacsits Szabolcs wrote: 
> > Independently whatever you use for bit-to-bit copy, if the partition is
> > used for booting then people may end up having an unbootable system,
> > depending on the bootloader, filesystems. Solutions are different for the
> > different cases.
> > 
> I take it the 'source partition' remains unaffected? So this only happens 
> when people try to boot
> from a fresh copied partition? I think i'll settle with a warning then. 
> (Maybe a FAQ entry if too
> many complains start rolling in ;) )

In general, once you start messing with the bootstrap, if you don't
understand it, how can you know if you're going to break it?

Probably the most you can realistically do with a parted type program
is recognition-based.  Ie: you cover cases where you can specifically
recognize the exact software in use, and then perhaps even try your best
with a fuzzy match.

No program, short of an AI, is going to understand every possible
configuration, so the *best* thing is to enable a comptent admin to
operate the software safely, rather than going overboard trying to
understand the universe.

> > Probably you should be aware also that kernels earlier than 2.6 couldn't
> > read the last sectors of partitions if it was odd thus people lost that
> > sector, being filesystem data, gpt, ntfs backup boot sector or whatever
> > else.
> > 
> Hmmz, sounds more serious.. Another warning may be appropiate :)

Interesting, I don't recall this.  Probably I wasn't aware of it.

So, if you wanted to support such systems, (on those systems) you might
need to be able to work from /dev/hda rather hda1 and do the maths yourself
What a drag!

> Another question:
> When at least one partition on a device is mounted (or active in case of 
> swap) this device is considered 'busy'
> And it's not possible to do anything with newly created partitions on this 
> device.
> (they don't show up in /dev/ and /proc/partitions)
> At least.. this is the case with fdisk.. 
> I've noticed it's possible to create a new partition on a busy device with
> parted and it gets properly listed in /dev/ and /proc/partitions.
> 
> This is great, but makes me wonder how parted accomplishes this? And how 
> reliable is this?
> Will it also work on older kernels etc..
> Some insights would be appreciated

Okay, this is down to the kernel.  I'm a bit fuzzy on the details, but it
has changed over the years, so its not set in stone.  No doubt there is a 
also bigger picture beyond linux that would help to illuminate the problem.

Hopefully someone here can offer more specific details.  But look at
the system calls that fdisk or parted use to update the partition tables
(I'm imagining just plain read), and then to ask for a re-read of the
partition tables (BLKRRPART ??),  have a look in and around

        /usr/src/linux/devices/block/genhd.c
and     /usr/src/linux/fs

and then grep lkml for the history and discussion.

I generally opt for a reboot when I see that 'busy' error :)

Regards,
Paddy
-- 
Perl 6 will give you the big knob. -- Larry Wall