Re: [Lynx-dev] File size limitation problem

[Nelson H. F. Beebe]

Recent discussions on this list concern the problem of handling files
larger than 2**31 - 1 bytes in size (the largest signed 32-bit
integer): that is the infamous 2GB limit that is grossly inadequate
for today's disks on desktop computers (e.g., 400GB EIDE disk for
about $400), and even for DVDs (up to 18GB in multilayer formats).

The 1989 and 1999 ISO C Standards define an OPAQUE datatype, fpos_t,
that can be used to address bytes within files.  However, neither
standard requires that it be implemented as an integer data type, or
even that it be a primitive data type: it can legally be a composite
type, such as a struct or a union, or even an opaque pointer into
kernel data structures.  Although all Unix systems implement fpos_t as
a primitive type as far as I know, it is legitimate to implement it
differently.  VAX VMS addressed files with a pair (file block number,
offset in block).

As some posters have noted, some systems implement fpos_t to be big
enough to address 2**63 - 1 bytes (the largest signed 64-bit integer):
that value, 9_223_372_036_854_775_807, is unlikely to be surpassed in
filesystem capacity in our lifetimes, or even our grandchildrens'.  At
100MB/sec, it takes more than 5000 years to write every byte of an
64-bit address space just once.

On another list that I follow, a similar discussion arose in
mid-October 2004, and I posted this summary of the current state of
Unix support for large filesystems:

>> ...
>> Date: Wed, 13 Oct 2004 18:06:16 -0600 (MDT)
>> From: "Nelson H. F. Beebe" <address@hidden>
>> Subject: Re: fseek/ftell and 32-bit limits
>> 
>> Sun Solaris provides these routines:
>> 
>> SYNOPSIS
>>      #include <stdio.h>
>> 
>>      int fseek(FILE *stream, long offset, int whence);
>> 
>>      int fseeko(FILE *stream, off_t offset, int whence);
>> 
>>      long ftell(FILE *stream);
>> 
>>      off_t ftello(FILE *stream);
>> 
>> The off_t type is defined like this in <sys/types.h>:
>> 
>> /*
>>  * The size of off_t and related types depends on the setting of
>>  * _FILE_OFFSET_BITS.  (Note that other system headers define other types
>>  * related to those defined here.)
>>  *
>>  * If _LARGEFILE64_SOURCE is defined, variants of these types that are
>>  * explicitly 64 bits wide become available.
>>  */
>> #ifndef _OFF_T
>> #define _OFF_T
>> 
>> #if defined(_LP64) || _FILE_OFFSET_BITS == 32
>> typedef long            off_t;          /* offsets within files */
>> #elif _FILE_OFFSET_BITS == 64
>> typedef longlong_t      off_t;          /* offsets within files */
>> #endif
>> 
>> #if defined(_LARGEFILE64_SOURCE)
>> #ifdef _LP64
>> typedef off_t           off64_t;        /* offsets within files */
>> #else
>> typedef longlong_t      off64_t;        /* offsets within files */
>> #endif
>> #endif  /* _LARGEFILE64_SOURCE */
>> 
>> #endif /* _OFF_T */
>> 
>> FreeBSD, HP-UX, NetBSD, and OpenBSD supply fseeko() and ftello() like
>> Solaris does.
>> 
>> IBM AIX offers fseeko(), ftello(), fseeko64(), and ftello64(): they
>> differ in having arguments of type off_t vs off64_t.
>> 
>> SGI IRIX offers fseek64() and ftell64() which use long long int.
>> 
>> HP/Compaq/DEC Alpha OSF/1 4.0 has fseek() and ftell() with long int
>> (64-bit) offsets.  Version 5.0 also has fseeko() and ftello().
>> 
>> The weak system in all of this is GNU/Linux: it has only the
>> traditional fseek()/ftell() with 32-bit long offsets on 32-bit CPUs,
>> and 64-bit long offsets on 64-bit CPUs (Alpha and Itanium), and ISO
>> Standard C fgetpos()/fsetpos() with 64-bit fpos_t.
>> 
>> Thus, on almost every system, except 32-bit GNU/Linux, you can get
>> what you are seeking, but not quite in the same system-independent
>> way.
>> ...

-------------------------------------------------------------------------------
- Nelson H. F. Beebe                    Tel: +1 801 581 5254                  -
- University of Utah                    FAX: +1 801 581 4148                  -
- Department of Mathematics, 110 LCB    Internet e-mail: address@hidden  -
- 155 S 1400 E RM 233                       address@hidden  address@hidden -
- Salt Lake City, UT 84112-0090, USA    URL: http://www.math.utah.edu/~beebe  -
-------------------------------------------------------------------------------