[Mldonkey-tasks] [task #4342] Access to disk data (free space, max file name length)

[spiralvoice]

URL:
  <http://savannah.nongnu.org/task/?func=detailitem&item_id=4342>

                 Summary: Access to disk data (free space, max file name
length)
                 Project: mldonkey, a multi-networks file-sharing client
            Submitted by: None
            Submitted on: Fre 01.07.2005 um 23:31
                Category: Core
                Priority: 8
                  Status: In Progress
             Assigned to: None
             Open/Closed: Open
         Should Start On: Fre 01.07.2005 um 00:00
   Should be Finished on: Fre 01.07.2005 um 00:00

    _______________________________________________________

Details:

it's not forbidden to download a super-long-name file in mldonkey. But when
it's done, it can't be commited.
I don't know the limit of the system exactly. But the only way to commit the
file is to commit it as a new short name. Maybe it's not bad to check the
filename length and auto-rename to a new filename with same ext name? ex:
blah-blah-blah.rar -> 1.rar or blah~1.rar

ps. I got this problem because of filename with a unicode character with value
0xff,for example, is sent to mldonkey as '\255'  or '%255' .

    _______________________________________________________

Follow-up Comments:


-------------------------------------------------------
Date: Son 17.07.2005 um 17:02       By: spiralvoice <spiralvoice>
http://mldonkey.berlios.de/modules.php?name=Forums&file=viewtopic&p=19907#19907

-------------------------------------------------------
Date: Mit 13.07.2005 um 21:47       By: spiralvoice <spiralvoice>
"I don`t see a sys.vfs on the OSX system, nor on FreeBSD 5.4. "

For FreeBSD it does not matter, there param.h and mount.h are used:

+#ifdef HAVE_SYS_PARAM_H
+#  include <sys/param.h>
+#  if (defined(__FreeBSD__) && __FreeBSD_version >= 503001) ||
defined(__OpenBSD__) || defined(__NetBSD__)
+#    include <sys/mount.h>

-------------------------------------------------------
Date: Mit 13.07.2005 um 21:39       By: Anonymous
Here is my os_stubs_c.c: http://pastebin.com/312849

-------------------------------------------------------
Date: Mit 13.07.2005 um 21:14       By: Anonymous
I applied your patch, added --disable-iconv and forced HAVE_STATS to 1, but
compiling of os_stubs_c.c fails:
/usr/local/bin/ocamlc.opt -verbose -ccopt "-I /byterun -o
src/utils/cdk/zlibstubs.o" -ccopt "-g -O2   -D_THREAD_SAFE " -ccopt " " -cclib
"" -ccopt " " -cclib "" -cclib -lz -ccopt "-D_THREAD_SAFE " -c
src/utils/cdk/zlibstubs.c
+ gcc -fno-defer-pop -no-cpp-precomp -Wall -D_FILE_OFFSET_BITS=64 -D_REENTRANT
 -c -I /byterun -o src/utils/cdk/zlibstubs.o -g -O2   -D_THREAD_SAFE     
-D_THREAD_SAFE   -I'/usr/local/lib/ocaml' 'src/utils/cdk/zlibstubs.c'
src/utils/cdk/zlibstubs.c: In function 'camlzip_deflate':
src/utils/cdk/zlibstubs.c:87: warning: pointer targets in assignment differ in
signedness
src/utils/cdk/zlibstubs.c:89: warning: pointer targets in assignment differ in
signedness
src/utils/cdk/zlibstubs.c: In function 'camlzip_inflate':
src/utils/cdk/zlibstubs.c:135: warning: pointer targets in assignment differ
in signedness
src/utils/cdk/zlibstubs.c:137: warning: pointer targets in assignment differ
in signedness
src/utils/cdk/zlibstubs.c: In function 'camlzip_update_crc32':
src/utils/cdk/zlibstubs.c:170: warning: pointer targets in passing argument 2
of 'crc32' differ in signedness
/usr/local/bin/ocamlc.opt -ccopt "-g -O2   -D_THREAD_SAFE   -o
src/utils/cdk/heap_c.o" -ccopt "" -c src/utils/cdk/heap_c.c
/usr/local/bin/ocamlc.opt -verbose -ccopt "-I /byterun -o
src/config/unix/os_stubs_c.o" -ccopt "-g -O2   -D_THREAD_SAFE " -ccopt " "
-cclib "" -ccopt " " -cclib "" -cclib -lz -ccopt "-D_THREAD_SAFE " -c
src/config/unix/os_stubs_c.c
+ gcc -fno-defer-pop -no-cpp-precomp -Wall -D_FILE_OFFSET_BITS=64 -D_REENTRANT
 -c -I /byterun -o src/config/unix/os_stubs_c.o -g -O2   -D_THREAD_SAFE     
-D_THREAD_SAFE   -I'/usr/local/lib/ocaml' 'src/config/unix/os_stubs_c.c'
src/config/unix/os_stubs_c.c: In function 'os_ftruncate':
src/config/unix/os_stubs_c.c:83: warning: format '%d' expects type 'int', but
argument 4 has type 'off_t'
src/config/unix/os_stubs_c.c: At top level:
src/config/unix/os_stubs_c.c:138: warning: 'struct statfs' declared inside
parameter list
src/config/unix/os_stubs_c.c:138: warning: its scope is only this definition
or declaration, which is probably not what you want
src/config/unix/os_stubs_c.c: In function 'copy_statfs':
src/config/unix/os_stubs_c.c:147: error: dereferencing pointer to incomplete
type
src/config/unix/os_stubs_c.c:149: error: dereferencing pointer to incomplete
type
src/config/unix/os_stubs_c.c:150: error: dereferencing pointer to incomplete
type
src/config/unix/os_stubs_c.c:151: error: dereferencing pointer to incomplete
type
src/config/unix/os_stubs_c.c:152: error: dereferencing pointer to incomplete
type
src/config/unix/os_stubs_c.c:153: error: dereferencing pointer to incomplete
type
src/config/unix/os_stubs_c.c:154: error: dereferencing pointer to incomplete
type
src/config/unix/os_stubs_c.c:175: error: dereferencing pointer to incomplete
type
src/config/unix/os_stubs_c.c: In function 'statfs_statfs':
src/config/unix/os_stubs_c.c:194: error: storage size of 'buf' isn't known
src/config/unix/os_stubs_c.c:195: warning: implicit declaration of function
'statfs'
src/config/unix/os_stubs_c.c:197: warning: implicit declaration of function
'strerror'
src/config/unix/os_stubs_c.c:197: warning: passing argument 1 of
'caml_failwith' makes pointer from integer without a cast
src/config/unix/os_stubs_c.c:194: warning: unused variable 'buf'
src/config/unix/os_stubs_c.c: In function 'statfs_fstatfs':
src/config/unix/os_stubs_c.c:212: error: storage size of 'buf' isn't known
src/config/unix/os_stubs_c.c:213: warning: implicit declaration of function
'fstatfs'
src/config/unix/os_stubs_c.c:215: warning: passing argument 1 of
'caml_failwith' makes pointer from integer without a cast
src/config/unix/os_stubs_c.c:212: warning: unused variable 'buf'
make: *** [src/config/unix/os_stubs_c.o] Error 2


-------------------------------------------------------
Date: Mit 13.07.2005 um 20:37       By: Anonymous
I don`t see a sys.vfs on the OSX system, nor on FreeBSD 5.4. There is a
<sys/statvfs.h>.

Relevant utput of `man statfs`:
     typedef struct { int32_t val[2]; } fsid_t;

     #define MFSNAMELEN   15 /* length of fs type name, not inc. nul */
     #define MNAMELEN     90 /* length of buffer for returned name */

     struct statfs {
         short   f_otype;    /* type of file system (reserved: zero) */
         short   f_oflags;   /* copy of mount flags (reserved: zero) */
         long    f_bsize;    /* fundamental file system block size */
         long    f_iosize;   /* optimal transfer block size */
         long    f_blocks;   /* total data blocks in file system */
         long    f_bfree;    /* free blocks in fs */
         long    f_bavail;   /* free blocks avail to non-superuser */
         long    f_files;    /* total file nodes in file system */
         long    f_ffree;    /* free file nodes in fs */
         fsid_t  f_fsid;     /* file system id (super-user only) */
         uid_t   f_owner;    /* user that mounted the file system */
         short   f_reserved1;        /* reserved for future use */
         short   f_type;     /* type of file system (reserved) */
         long    f_flags;    /* copy of mount flags (reserved) */
         long    f_reserved2[2];     /* reserved for future use */
         char    f_fstypename[MFSNAMELEN]; /* fs type name */
         char    f_mntonname[MNAMELEN];    /* directory on which mounted */
         char    f_mntfromname[MNAMELEN];  /* mounted file system */
         char    f_reserved3;        /* reserved for future use */
         long    f_reserved4[4];     /* reserved for future use */
     };


-------------------------------------------------------
Date: Mit 13.07.2005 um 20:25       By: spiralvoice <spiralvoice>
The new functions are only active if <sys/param.h> or <sys/vfs.h>
are present on the system. Only then HAVE_STATS is defined to 1.
For testing you could add this line

+#  define HAVE_STATS 1

after

+#ifdef HAVE_SYS_VFS_H
+#  include <sys/vfs.h>
+#  define HAVE_STATS 1

to force the compile of the new functions.

https://savannah.nongnu.org/task/index.php#comment10
"If some C cracks are reading, the case HAVE_STATS 0 is not coded
yet because I do not know how to return "-1" values if none of
the header files and data structures are present."

-------------------------------------------------------
Date: Mit 13.07.2005 um 20:22       By: spiralvoice <spiralvoice>
Besides the link error for iconv, which is a different story
(please try "--disable-iconv"), could you please post the output
of the man page of "statfs" or "statvfs"?

Maybe it looks like this:
http://developer.apple.com/documentation/Darwin/Reference/ManPages/man2/statfs.2.html

So you could extend the "#if defined" statements so Darwin is treated like
OpenBSD. This should be added:
#if defined(__APPLE__)

Maximal file name length is a bit more complicated:
http://developer.apple.com/documentation/Darwin/Reference/ManPages/man2/pathconf.2.html

This is from the patch:

+#  if defined(__OpenBSD__) || defined(__NetBSD__)
+#    include <sys/syslimits.h>
+  v = copy_int64 (NAME_MAX); caml_modify (&Field (bufv, 8), v);
+#  else
+  v = copy_int64 (buf->f_namemax); caml_modify (&Field (bufv, 8), v);
+#  endif

Please change this to:

+#  if defined(__OpenBSD__) || defined(__NetBSD__)
+#    include <sys/syslimits.h>
+  v = copy_int64 (NAME_MAX); caml_modify (&Field (bufv, 8), v);
+#  else
+#    if defined(__APPLE__)
+#      include <unistd.h>
+#      v = copy_int64 (_PC_NAME_MAX); caml_modify (&Field (bufv, 8), v);
+#    else
+       v = copy_int64 (buf->f_namemax); caml_modify (&Field (bufv, 8), v);
+#    endif
+#  endif

There is a MLDonkey command "disk <dir>" for tests and the display
of the "shares" command in HTML is extended to show some data.

-------------------------------------------------------
Date: Mit 13.07.2005 um 19:19       By: Anonymous
Unfortunately, OSX exits final step with the following error:
/usr/local/bin/ocamlopt.opt -inline 10 -linkall  -o mlnet       unix.cmxa
str.cmxa -ccopt " " -cclib "-lcharset  -liconv" -ccopt " " -cclib "" -cclib
-lz -ccopt "-D_THREAD_SAFE "   -I build    build/cdk.cmxa    build/common.cmxa
   build/client.cmxa    build/core.cmxa    build/driver.cmxa   
src/daemon/common/commonMain.cmx  
/usr/bin/ld: warning multiple definitions of symbol _locale_charset
/usr/lib/gcc/powerpc-apple-darwin8/4.0.0/../../../libcharset.dylib(localcharset.o)
definition of _locale_charset
/usr/lib/gcc/powerpc-apple-darwin8/4.0.0/../../../libiconv.dylib(localcharset.o)
definition of _locale_charset
/usr/bin/ld: Undefined symbols:
_statfs_statfs
collect2: ld returned 1 exit status
Error during linking
make: *** [mlnet] Error 2

-------------------------------------------------------
Date: Die 12.07.2005 um 18:30       By: spiralvoice <spiralvoice>
What about Mac OSX?

-------------------------------------------------------
Date: Die 12.07.2005 um 18:28       By: spiralvoice <spiralvoice>
Updated patch, should work on FreeBSD >= 5.3 and OpenBSD (tested on 5.3).
If there are older FreeBSD versions the version number in
os_stubs_c.c has to be changed, reports about older FreeBSD versions
where this patch works are welcome.

If some C cracks are reading, the case HAVE_STATS 0 is not coded
yet because I do not know how to return "-1" values if none of the
header files and data structures are present.

-------------------------------------------------------
Date: Mon 11.07.2005 um 23:32       By: Anonymous
Still doesn't work on FreeBSD. Datastructure now in mldonkey:

type statfs = {
  f_type : int64;   (* type of filesystem *)
  f_bsize : int64;  (* optimal transfer block size *)
  f_blocks : int64; (* total data blocks in file system *)
  f_bfree : int64;  (* free blocks in fs *)
  f_bavail : int64; (* free blocks avail to non-superuser *)
  f_files : int64;  (* total file nodes in file system *)
  f_ffree : int64;  (* free file nodes in fs *)
  f_fsid : unit;  (* See note in statfs(2) *)
  f_fnamelen : int64; (* maximum length of filenames *)
  f_basetype : string; (* type of filesystem - Solaris, (-1) on other systems,
use f_type there *)
  f_frsize : int64;  (* Fundamental file system block size, (-1) if not
provided by system *)
}

Datatype decl. in FreeBSD 
     struct statfs {
     uint32_t f_version;             /* structure version number */
     uint32_t f_type;                /* type of filesystem */
     uint64_t f_flags;               /* copy of mount exported flags */
     uint64_t f_bsize;               /* filesystem fragment size */
     uint64_t f_iosize;              /* optimal transfer block size */
     uint64_t f_blocks;              /* total data blocks in filesystem */
     uint64_t f_bfree;               /* free blocks in filesystem */
     int64_t  f_bavail;              /* free blocks avail to non-superuser
*/
     uint64_t f_files;               /* total file nodes in filesystem */
     int64_t  f_ffree;               /* free nodes avail to non-superuser */
     uint64_t f_syncwrites;          /* count of sync writes since mount */
     uint64_t f_asyncwrites;         /* count of async writes since mount */
     uint64_t f_syncreads;           /* count of sync reads since mount */
     uint64_t f_asyncreads;          /* count of async reads since mount */
     uint64_t f_spare[10];           /* unused spare */
     uint32_t f_namemax;             /* maximum filename length */
     uid_t     f_owner;              /* user that mounted the filesystem */
     fsid_t    f_fsid;               /* filesystem id */
     char      f_charspare[80];          /* spare string space */
     char      f_fstypename[MFSNAMELEN]; /* filesystem type name */
     char      f_mntfromname[MNAMELEN];  /* mounted filesystem */
     char      f_mntonname[MNAMELEN];    /* directory on which mounted */
     };

However statvfs should work on freebsd too starting from FreeBSD 5.0. I can't
see light at the end of the tunnel yet.

-------------------------------------------------------
Date: Mon 11.07.2005 um 20:16       By: spiralvoice <spiralvoice>
Updated for current CVS, should now work on Solaris, too.

-------------------------------------------------------
Date: Don 07.07.2005 um 16:38       By: spiralvoice <spiralvoice>
Updated patch according to
http://article.gmane.org/gmane.comp.lang.caml.general/29511

-------------------------------------------------------
Date: Son 03.07.2005 um 22:46       By: spiralvoice <spiralvoice>
Updated patch to compile on MinGW although no values are printed
because of the missing implementation.
Tried compiling on FreeBSD, it fails because header files are
different and the data structure is also different.

-------------------------------------------------------
Date: Son 03.07.2005 um 14:52       By: spiralvoice <spiralvoice>
Summary was: commit failed for super-long name files

I renamed this task because a broader topic will be addressed here.
Attached is a patch which demonstrates the features to be applied .
For a first impression call the "shares" command in the HTML
interface, it should print some data about the shared directories,
!!temp_directory and the core directory.

It lays the foundation for accessing this data:

type statfs = {
  f_type : int64;   (* type of filesystem *)
  f_bsize : int64;  (* optimal transfer block size *)
  f_blocks : int64; (* total data blocks in file system *)
  f_bfree : int64;  (* free blocks in fs *)
  f_bavail : int64; (* free blocks avail to non-superus *)
  f_files : int64;  (* total file nodes in file system *)
  f_ffree : int64;  (* free file nodes in fs *)
  f_fsid : unit;  (* See note in statfs(2) *)
  f_fnamelen : int64; (* maximum length of filenames *)
}

For the problem with the too-long-filenames something
like this can be coded (to be done):

if Unix32.fnamelen incoming_dir < String.length filename then
  String.shorten filename (Unix32.fnamelen incoming_dir)

I am also thinking about an option hdd_space_reserve (in MB,
default 50):

if (Unix32.diskfree !!temp_directoriy) < (!!hdd_space_reserve **
Int64.megabyte) then () (* pause all downloads here, 
print warnings on all interfaces, also in logfile and stop
logging *)

This should be done in the common module via add_infinite_timer.
These two things are most important and somewhat easy to code.

The rest is not so important as not all users are affected by
the problems described below.

The dllink command should also be made aware of this, if no
sparse files are used the file in !!temp_directory is created
with full length on Windows, so checking if enough space is
available and failing dllink is neccessary if not enough free
space is available. Currently only a zero-byte file and some
obscure error message is printed.

Something similar should be coded for committing if !!temp_directory
and the incoming directory are on different partitions, a check
on the incoming partition if enough space for copying the data
is available. But to reliable check if incoming and temp *are*
really on different physical partitions?
Unix2.rename tries renaming the file during commit, if it fails
with error code EXDEV
(see http://caml.inria.fr/pub/docs/manual-ocaml/libref/Unix.html)
it copies the file, but if this happens during multifile torrent
commit half of the files could already be copied as Unix2.rename
copies file by file and it is the only place where is notice that
different physical partitions are used.

If !!temp_directory and the incoming directory are on the same
physical partition this does not matter because the file is only
renamed so no more space is needed.

Further this patch currently only compiles on Linux,
Solaris/Sparc fails, did not test on other platforms yet.
Please report your findings, especially on Mac OSX, MinGW
(compile should fail there), Cygwin (might work) and *BSD.

-------------------------------------------------------
Date: Fre 01.07.2005 um 23:31       By: spiralvoice <spiralvoice>
This item has been reassigned from the project mldonkey, a multi-networks
file-sharing client bugs tracker to your tracker.

The original report is still available at bugs #10758

Following are the information included in the original report:

[field #0]                  Item ID: 10758<br />[field #1]                
Group ID: 1409<br />[field #2]              Open/Closed: Open<br />[field #3] 
               Severity: 3 - Normal<br />[field #4]                  Privacy:
Public<br />[field #5]                 Category: Core<br />[field #6]         
   Submitted by: None<br />[field #7]              Assigned to: None<br
/>[field #8]             Submitted on: Don 21.10.2004 um 17:37<br />[field #9]
                 Summary: commit failed for super-long name files<br />[field
#10]      Original Submission: it s not forbidden to download a
super-long-name file in mldonkey. But when it s done, it can t be commited.
I don t know the limit of the system exactly. But the only way to commit the
file is to commit it as a new short name. Maybe it s not bad to check the
filename length and auto-rename to a new filename with same ext name? ex:
blah-blah-blah.rar -> 1.rar or blah~1.rar

ps. I got this problem because of filename with a unicode character with value
0xff,for example, is sent to mldonkey as  \255   or  %255  .<br />[field #12] 
             Item Group: Feature request<br />[field #13]                  
Status: Postponed<br />[field #14]        Component Version: None<br />[field
#15]         Platform Version: Linux<br />[field #16]         
Reproducibility: None<br />[field #17]               Size (loc): None<br
/>[field #18]            Fixed Release: None<br />[field #19]          Planned
Release: None<br />[field #20]                   Effort: 0.00<br />[field #24]
                Priority: 5 - Normal<br />[field #27]         Percent
Complete: 0%<br />[field #29]                  Release: 2-5-28<br />[field
#54]          Binaries Origin: None<br />[field #55]                 CPU type:
Intel x86<br />[field #56]                      CPU: None<br />[field #57]    
Custom Select Box #4: None<br />[field #58]     Custom Select Box #5: None<br
/>[field #59]     Custom Select Box #6: None<br />[field #60]     Custom
Select Box #7: None<br />[field #61]     Custom Select Box #8: None<br
/>[field #62]     Custom Select Box #9: None<br />[field #63]    Custom Select
Box #10: None<br />

-------------------------------------------------------
Date: Fre 01.07.2005 um 23:31       By: spiralvoice <spiralvoice>
http://article.gmane.org/gmane.comp.lang.caml.general/29507

With this code MLDonkey is able to check fnamelen during runtime,
this bug is moved to the task tracker to remind me to implement this.

-------------------------------------------------------
Date: Fre 29.04.2005 um 09:48       By: Amorphous <amorphous>
I would guess the file-name length limit is also different per filesystem, so
i guess this would have to be an runtime check...
or an option with default 0 =unlimited .
But just printing the exception in the log would help, too :)

-------------------------------------------------------
Date: Die 12.04.2005 um 10:22       By: spiralvoice <spiralvoice>
Unicode stuff is now in MLDonkey, try 2-5-30-4 + latest patches from here. How
can we find find out the allowed length of filenames on the different systems
MLDonkey can be built?







    _______________________________________________________

File Attachments:


-------------------------------------------------------
Date: Die 12.07.2005 um 18:28  Name: free2.patch  Size: 15,61KB   By:
spiralvoice

<http://savannah.nongnu.org/task/download.php?item_id=4342&item_file_id=227>

    _______________________________________________________

Reply to this item at:

  <http://savannah.nongnu.org/task/?func=detailitem&item_id=4342>

_______________________________________________
  Nachricht geschickt von/durch Savannah
  http://savannah.nongnu.org/