Re: [Evms] Questions about portability

[Andrew Clausen]

Christoph Hellwig wrote:
> 
> On Sun, Nov 12, 2000 at 10:30:33AM -0200, Andrew Clausen wrote:
> > "align" depends on the on-disk file system (not merely the file system
> > type).
> > For example, if you have an ext2 file system, that starts at sector 63
> > (512
> > byte unit), and the file system has a block size of 2k, then:
> >
> >       new_start = old_start + X * block_size
> 
> This _is_ a simple rule, isn't it?

Yes.  But the front-end doesn't know how to generate the rule.

> > max_size might be a function of certain properties of the file system,
> > like the block size.  (it is rather difficult to grow the block size,
> > and if the resizer can't handle it, then this is going to be a factor)
> > However, there is no way a front-end can know this, unless it asks
> > the resizer.
> 
> Right.  What I currently do in lvmadm is just ignoring that fact and let
> the LV in it's new, big size and the fs in it's all small.  This is
> not the right error recovery, thus my current working tree version resizes
> the LV back.

I think it's better to be able to plan ahead.

> > I disagree.  ext2 doesn't follow simple rules, and I don't think it's
> > braindead.  FAT is braindead, and it also doesn't follow simple rules :p
> 
> I'm not completly familar with the ext2 size rules, but I've mostly seen
> simple rules like X (some not to difficult to retain variable) * blocksize.
> 
> You are more familar with the limits, so please tell me the most difficult
> to calculate value, that has a real value for resizing ext2.
> 
> And of course ext2 is not braindead - also it has a few small designe flaws.

yep.  The rules are resizer dependent.  I intend to change the resizer,
so it duplicates all metadata it changes, so it can do resize the start
(which isn't important for LVM, but is important for partition tables)

So, in this case, it would be a function of the number of directory
blocks...  (Yes, it isn't exactly elegant, but it is a MUCH more
defensive way of doing FS resizing.  The idea of FS resizing is
that you don't lose data ;-)

> > >  And I don't think we should overdesign the whole system just to
> > > support completly braindead filesystems.
> >
> > I think we should!  The "overdesigned system" (libparted) is clean
> > enough
> > for me.  (Heck, people are always trying to make it uglier, but I don't
> > let them ;-)
> 
> I haven't said that libparted is overdesined - but I would lead to a over-
> design if you make it capable of dealing with each messy fs'es strange
> limitations.

Actually, I think I've achieved an elegant way of dealing with the
problems.  For example, you ask the file system for a resize
constraint with ped_file_system_get_resize_constraint()
(this function is polymorphic - i.e. each resizer has it's own
implementation)

struct _PedConstraint {
        PedAlignment*   start_align;
        PedAlignment*   end_align;
        PedGeometry*    start_range;
        PedGeometry*    end_range;
        PedSector       min_size;
};  

struct _PedGeometry {
        PedDisk*                disk;
        PedSector               start;
        PedSector               length;
        PedSector               end;
};

struct _PedAlignment {
        PedSector       offset;
        PedSector       grain_size;
};

So, the resizer can generate the constraint anyway it wants to...

Obviously, the above (PedConstraint) can be represented in a hints
file.  However, the PedConstraint only applies to a particular
file system (not all file systems of a particular file system
type).  The rules for generating the PedConstraint are non-trivial.

> > > These filesystem would usually be foreign fses and only occur on one 
> > > partition
> > > type, so the hints can be added. (Yes, that's _horribly_ ugly, but less 
> > > ugly
> > > than messing up the whole design).
> >
> > I don't follow.  How do the hints work?
> 
> If you have e.g. the mac partition table format the chance that a hfs(+) fs
> will be created on it is very big.  If hfs had a strange limitation, the
> mac partition table resizer would just deal with this strange limitation
> (for example - just hypothetically it must have a size of 7 times the block
> size) even if an ext2fs would be created on it.

Just, the limitations usually must be computed from hard-to-access (and
resizer-specific) stuff, like "2 * # directory blocks".
 
> > Maybe this is true, but the block size doesn't tell you anything about
> > the constraints.  It's the *resizer* that has the constraints.  Only the
> > resizer knows what it can and can't do.  My FAT resizer USED to be
> > unable
> > to shrink the cluster size, or move the start... but now it can.
> 
> As we agreed on before moving the start of a fs should not be the fs
> resizers job.

In LVM, this is the case.  For partitions, this is not the case.

> Shrinking the cluster size usually (always) happens by
> powers of two - so there shouldn't be a real problem either.

Yes, but it changes lots of other things... (changes the
amount of metadata, etc.)

> And btw I don't know about any operating system that has problems with
> filesystems that are smaller then the underlying partitions.

DOS/Windows have problems.  In some circumstances, DOS/Windows computes
information about the file system (like, the size of the FATs, the size
of
the file system, etc.) from the size of the partition.  This is VERY
braindead, I know, but spent a month reverse engineering it, to figure
out the algorithm.  (It's a VERY brain-damaged algorithm!  It took
me a day to prove that it never calculated an invalid file system)
 
> > (2) programs are easy to interface with users, but difficult to
> > interface
> > with other programs (i.e. difficult to be used by other programs).  When
> > I say "difficult", I mean for non-trivial stuff.
> 
> I can't agree 100% with you on this.
> Yes, the library interface is easier to use in programs in the first time,
> but not very much (when the interface to the program is designed right),

Program interfaces aren't good at giving back information.  It's only
good when program A calls program B, and A gives information to B,
but B doesn't give information to A.

> but the program interface is _much_ more flexible.

When B needs to give A information (like constraints, etc.), then I
disagree...

> Just for loading
> different libraries with the same interface you need to play tricks with
> libdl,

Nah, it's trivial!

(BTW, "same interface" usually means calling init() and done().
init() would just register the filesystem/disklabel/whatever it
implements.  eg: ped_file_system_type_register())

> and loading libraries written in another language

Depends on the language...

> or just as shell
> script are extremly diffcult if not impossible.

Shell scripts: should just use the program front-end.  Obviously,
the program front-end needs to have a good interface.  Parted
is a reasonable interface to libparted, IMHO.

> > (4) and (5) follow from (2) and (3):
> >
> > (4) libraries are easily wrapped by programs, to provide an interface to
> > users
> 
> True. (wether this is a good design principle or not is another question ...)

Hehe.  I think it is, because you get a nice library interface (for
other programs) and a nice user interface (for users).
 
> > Here's another interesting option: libparted is structured such that
> > it would be very trivial to write resize.ext2 and resize.fat front
> > ends.  VERY TRIVIAL (such frontends would be about 30 lines of code).
> 
> Good.  And when (if) you start doing this frontends you can simply move
> the code that is only used by one program into this program and make the
> 'parted' program call them.

This is not true.  The simple frontends can't communicate the
constraints.
(well, you could do somthing like: resize.ext2 --print-constraints, but
I don't like it!)

> I don't have any problem with multiple
> resizers sharing some common code - as long as they are separate programs,
> and one can code another such program without having to use the common code.

Well, at least for partitioning stuff, this is necessary.

> > Maybe, we should keep libparted as-is for partition table stuff,
> > but use the simpler front ends for LVM.
> 
> Yes! - but they could of course share code for e.g. fs resizing by moving
> them into programs use by both.
> 
> Eg. we could have:
> 
>         libresize       (common fs resizing functions)
>         libparted       (partition handling stuff)
>         liblvm          (lvm handling)
> 
>         parted          (frontend for partion handling)
>         lvmadm          (frontend for lvm handling)

Do you feel like becoming more familiar with libparted?  This
kind of thing might be possible, but I think we need to discuss
the "guts" a bit more ;-)

I suggest you look in include/parted/*, and look at how the
weird stuff (like ped_file_system_set_system()) is done.

There is API documentation, but it's really only useful as
reference.  The frontend's code isn't particularly elegant
(I'm working on it!), but it might help, also.

> > However, is this sufficient for conversion between LVM types,
> > or converting from file-systems-on-partitions to physical volumes
> > containing a logical volume with the old file system?
> 
> Yet another frontend just for this purpose Do one thing right ...)
> that uses both middle and backends.

OK.  I think this is possible...

> If you are interested in such a thing I would help out on the LVM part.

Of course :-)
 
> > In the case of FAT/Windows: if the file system type and the partition
> > type don't match, M$ chokes.  There are 12 different partition types
> > (numbers)
> > corresponding to different types of FAT partitions.  These correspond to
> > various permutations of:
> >
> > (1) FAT12/16/32
> > (2) the size of the file system (some brain-damage, WRT small file
> > systems)
> > (3) whether the partition is "hidden"
> > (4) whether M$ should use LBA
> >
> > (1) and (2) should be determined by the file system code.  You could
> > hack around this, by creating different file systems for fat12, fat16
> > and fat32, but this seems dubious to me.
> 
> I would really prefer that - fat{12,16,32} _are_ different filesystem,
> even if they are very similar - you might easily put all the code in
> one binary if that makes sense - and just make those behave differently
> when called as resizefs.fat12 and resizefs.fat32

What happens if another feature gets introduced, that can be enabled
on fat12/16/32, and affects the partition type?  Basically, I'm paranoid
of idiots doing stupid things (as has happened many times in the
past...)
 
> > (For example: progams
> > might get confused when the resizer automagically converts from fat16
> > to fat32, or whatever)
> 
> Right - but this is actually an argument for my standpoint ;)

Why?  No-one should care about a change from fat16 -> fat32.  If you
are doing a whole series of operations on a partition, should you
have to check the file system type every second, to make sure you're
calling the right function?  (compulsive obsessive disorder! yay!)

> > I think my Mac example was bogus... the partition types need to be
> > right,
> > but it is merely a function of the file system type.  I'm sure other
> > cases will pop-up... Sun disk labels for example, allow partitions
> > and the disk label metadata to overlap, so you have to be careful
> > about which file system you use, so as not to clobber the disk label.
> > (ext2, and new linux-swap don't use the first 1k of the partition)
> >
> 
> That's really odd (besides the BSDish slice X = whole partition thing ...)
> But a offset option for the hints file and the fs resizer could easi

I *really* don't like the idea of a hints file.  Hints file + program
is a more complicated interface than a library.

> > But, it's useful to know how much free space you need, before hand,
> > so you can plan ahead.  (Think: automatic partitioning)
> 
> I'm not really sure wether it does matter if the user gets 1024 or
> 950MB when he specifies 1GB.

It does, if you're trying to plan some complicated manuevre...
(which you often need to do... eg: swapping the order of two partitions,
which might be necessary, say, to get your computer to boot with
brain-damaged BIOSes, or to move free space together, or something!)

> > > Why should the generic resize frontend want to know about that?
> >
> > Automatic partitioning, and "planning ahead".  (Eg: figuring out how
> > you want to rearrange your partitions, and then clicking "go!".  It's
> > nicer to know before hand, if it's going to work)
> 
> And why does it help the automatic partitioning tool when it knows
> wether the filesystem uses sparse supers?

It doesn't.  But this information is necessary for computing some of
the constriants.  Hence, it makes sense to have the computations
performed by fs-specific code, not via hints.

> It will get a few megabytes
> more if the filesystem has sparse supers enabled - but that shouldn't
> really matter (IMHO).

It matters, IMHO.

> Actually the fake partition table is a cludge that would not be
> needed with separate programs...

The fake partition is a cludge, because PedGeometry and PedDevice
aren't polymorphic.  I could change that, though...

> On the other hand: why do I need a fake partition table to resize
> the LV? - Once parted has LVM support this should not be needed and
> I doubt it will do anything usefull before that point ;)

ROFL

Yeah, we can get rid of the fake partition table thing. But it
was a very simple hack.  (Also, it keeps the UI consistent,
although it isn't exactly intuitive...)

> >       # parted /dev/hda1 (or the LV device) resize 1 0 [new-size]
> >
> > (where 1 = the fake "partition" number, and 0 = the new start)
> 
> Isn't that the parted syntax for resizing the partion _and_ the fs?

No.

        # parted /dev/hda resize MINOR NEW-START NEW-END
 
> > libparted is easy to port to... (in fact, both the ext2 and fat
> > resizers were originally separate programs)  You could almost write
> > a sed script to do it!
> 
> But what is when the new resizer is not written in C?

Still shouldn't be that hard... besides, this is rather unlikely.
It's a pity this isn't a trivial job.

Andrew Clausen