Re: [Info-mtools] Bug: mformat does not format FAT32 correctly

[Pali Rohár]

On Sunday 23 September 2018 23:05:41 Alain Knaff wrote:
> Hi,
> 
> First, thanks for your patch, and sorry for the slow response. When I
> got your mails, I was rather busy, and eventually they ended up below
> the pile...
> 
> On 11/08/18 12:45, Pali Rohár wrote:
> [...]
> > On Tuesday 12 December 2017 10:41:26 Pali Rohár wrote:
> >> On Monday 04 December 2017 19:29:53 Pali Rohár wrote:
> >>> 2) mformat refuse to create FAT32 if disk size is less then 257 MB.
> >>>
> >>> Test to reproduce (-F means to format as FAT32):
> >>>
> >>>   $ truncate -s 100M fat.img
> >>>   $ mformat -F -h 64 -s 32 -t 100 -m 0xf8 :: -i fat.img
> >>>
> >>> mformay show just error message:
> >>>
> >>> Too few clusters for this fat size. Please choose a 16-bit fat in your
> >>> /etc/mtools.conf or .mtoolsrc file
> >>>
> >>> Fix is to set size of cluster (in number of sectors) to 1. So mformat
> >>> parameter: -c 1
> 
> Exactly.
> 
> >>>
> >>> It is pity that mformat cannot calculate cluster size for smaller FAT32
> >>> disks (when size is less then 257 MB). Note that mformat can calculate
> >>> it correctly for FAT16 (when -F is not used).
> 
> As some Microsoft documents (the cited knowledge base article) claimed
> that 4KB clusters where the default minimum size, this was used by
> default (... while still allowing to override it using -c if wanted).
> 
> [...]
> >>>   switch(abs(fat_bits)) {
> >>> ...
> >>>           case 32:
> >>>                   Fs->cluster_size = 8;
> >>>                   /* According to
> >>>                    * 
> >>> http://support.microsoft.com/support/kb/articles/q154/9/97.asp
> >>>                    * Micro$oft does not support FAT32 with less than 4K
> >>>                    */
> >>>                   return;
> >>> ...
> >>>
> >>> It means that cluster size is hardcoded to 8 sectors for FAT32. This is
> 
> 
> >>> also reason for problem 3). For FAT16 there is autodetection code. I do
> >>> not know what that comment mean
> 
> Really? Or are you joking? :-)

No. I understand "Implementation XYZ does not support feature ABC" as if
I prepare feature ABC, then whatever I do XYZ would not read or access
ABC. Therefore I understood above comment as Microsoft's FAT driver
cannot read FA532 filesystem with less then 4K.

But in reality it can.

> It gives the web address of a document published at Microsoft's support
> site. This document contains a sectence which could be understood to
> mean that FAT32 filesystems should not have cluster sizes less than 4K
> (i.e. 8 sectors).
> 
> 
> > or what it try to refer as link is dead.
> 
> Indeed, they moved it to a different address:
> https://support.microsoft.com/en-us/help/154997/description-of-the-fat32-file-system
> 
> The relevant sentence is:
> "The FAT32 file system allows for a default cluster size as small as 4 KB"
> 
> Ok, so this is the _default_ size, but smaller sizes are apparently
> permissible.

That is then something different. As it describe parameters for
formatting new filesystem. Not what filesystem driver can mount.

> >>>
> >>> So I would like to know, why is hardcoded 8 sectors as cluster size for
> >>> FAT32?
> 
> Actually, in mtools 4.0.18 this was not hardcoded, but overrideable
> using the -c option, as you noticed yourself in your earlier comment.
> 
> I'm not sure why you disabled this possibility in your patch, or maybe
> that was just some leftover from earlier testing.

Hm... I'm not aware of nothing that I disabled... Maybe bug?

> >>> I suspect this is a bug which needs fixing. At least to have
> >>> working mformat in FAT32 mode for small disk (or disk images).
> 
> I've now implemented the table you suggested (starting with 1 and going
> to 64), but I still kept the possibility to override with -c

Ok.

> >>
> >> Also please note that current versions of Windows (probably since 2000)
> >> allow user to specify cluster size in Format dialog also for FAT32. And
> 
> Exactly. And that's why I re-enabled the -c option.
> 
> >> default value depends on disk size, it is not hardcoded to 8 (=4kB).
> >>
> >> See also:
> >> https://support.microsoft.com/en-us/help/192322/description-of-default-cluster-sizes-for-fat32-file-system
> >> https://support.microsoft.com/en-us/help/140365/default-cluster-size-for-ntfs--fat--and-exfat
> 
> Oddly, these leave out the step with cluster_size = 1 but start with 4K
> (8 sectors)
> 
> >>
> >> So I would really suggest to change above hardcoded value 8 to either
> >> mapping table or use similar algorithm as is already present for FAT16
> >> in mformat.c
> >>
> >>> Because, in Microsoft FAT32 specification (fatgen103.doc) on page 20 for
> >>> FAT32 is written calculation table. For disk size there is cluster size
> >>> (in number of sectors), see table below:
> >>>
> >>> 32.5 MB - 260 MB   cluster_size =  1
> >>>  260 MB -   8 GB   cluster_size =  8
> >>>    8 GB -  16 GB   cluster_size = 16
> >>>   16 GB -  32 GB   cluster_size = 32
> >>>   32 GB -   2 TB   cluster_size = 64
> 
> Indeed, fatgen103.doc starts with cluster_size = 1.
> 
> >>>
> >>> Note that in that Microsoft FAT specification is written: "The rest of
> >>> this section is totally specific to drives that have 512 bytes per
> >>> sector.". So there is need to recalculate values when (logical) sector
> >>> size is not 512 bytes.
> > 
> > In attachment is a patch for mtools which calculates FAT32 cluster size
> > according to that Microsoft specification. With that patch I'm able to
> > correctly format small FAT32 disks (below 100M) and also big disks (2TB)
> > and even also very huge disks 16TB with 4k sectors.
> > 
> > Examples:
> > 
> > small 100 MB FAT32 disk image
> > $ truncate -s 100MB /tmp/fat
> > $ mformat -F -h 64 -s 32 -t $((100-1)) :: -i /tmp/fat
> > 
> > big 2 TB FAT32 disk image (this is maximal size for 512 byte sectors)
> > $ truncate -s 2TB /tmp/fat
> > $ mformat -F -h 64 -s 32 -t $((2*1024*1024-1)) :: -i /tmp/fat
> > 
> > very huge 16 TB FAT32 disk image with 4k sectors
> > $ truncate -s 16TB /tmp/fat
> > $ mformat -F -M 4096 -h 8 -s 32 -t $((16*1024*1024-1)) :: -i /tmp/fat
> > 
> > Everything is correctly detected and mounted on Linux.
> > 
> 
> This is now part of the development version of mtools, to be released in
> a week or so.

Ok. Is somewhere available that development version?

> Thanks,
> 
> Alain

-- 
Pali Rohár
address@hidden