[GNUnet-SVN] r12252 - in gnunet: . contrib src/dht

[gnunet]

Author: grothoff
Date: 2010-07-18 22:18:17 +0200 (Sun, 18 Jul 2010)
New Revision: 12252

Modified:
   gnunet/RATIONALE
   gnunet/README
   gnunet/TODO
   gnunet/contrib/report.sh
   gnunet/mkinstalldirs
   gnunet/src/dht/
Log:
update docs

Modified: gnunet/RATIONALE
===================================================================
--- gnunet/RATIONALE    2010-07-18 20:18:00 UTC (rev 12251)
+++ gnunet/RATIONALE    2010-07-18 20:18:17 UTC (rev 12252)
@@ -1,13 +1,14 @@
-This document is a summary of why we're moving to GNUnet NG and what
-this major redesign tries to address.
+This document is a summary of the changes made to GNUnet for version
+0.9.x (from 0.8.x) and what this major redesign tries to address.
 
 First of all, the redesign does not (intentionally) change anything
 fundamental about the application-level protocols or how files are
 encoded and shared.  However, it is not protocol-compatible due to
 other changes that do not relate to the essence of the application
-protocols.
+protocols.  This choice was made since productive development and
+readable code were considered more important than compatibility at
+this point.
 
-
 The redesign tries to address the following major problem groups
 describing isssues that apply more or less to all GNUnet versions
 prior to 0.9.x:
@@ -26,10 +27,10 @@
   mutexes and almost 1000 lines of lock/unlock operations.
   It is challenging for even good programmers to program or 
   maintain good multi-threaded code with this complexity.
-  The excessive locking essentially prevents GNUnet from
+  The excessive locking essentially prevents GNUnet 0.8 from
   actually doing much in parallel on multicores.
 * Despite efforts like Freeway, it was virtually 
-  impossible to contribute code to GNUnet that was not
+  impossible to contribute code to GNUnet 0.8 that was not
   writen in C/C++.
 * Changes to the configuration almost always required restarts
   of gnunetd; the existence of change-notifications does not
@@ -44,11 +45,11 @@
   days, result in really nasty and hard-to-find crashes.
 * structs of function pointers in service APIs were
   needlessly adding complexity, especially since in 
-  most cases there was no polymorphism
+  most cases there was no actual polymorphism
 
 SOLUTION:
 * Use multiple, lously-coupled processes and one big select
-  loop in each (supported by a powerful library to eliminate
+  loop in each (supported by a powerful util library to eliminate
   code duplication for each process).  
 * Eliminate all threads, manage the processes with a 
   master-process (gnunet-arm, for automatic restart manager) 
@@ -65,13 +66,15 @@
   => Process priorities can be used to schedule the CPU better
   Note that we can not just use one process with a big
   select loop because we have blocking operations (and the
-  blocking is outside of our control, thanks MySQL,
+  blocking is outside of our control, thanks to MySQL,
   sqlite, gethostbyaddr, etc.).  So in order to perform
   reasonably well, we need some construct for parallel
-  execution.
+  execution.  
 
   RULE: If your service contains blocking functions, it
-        MUST be a process by itself.
+        MUST be a process by itself.  If your service
+        is sufficiently complex, you MAY choose to make
+        it a separate process.
 * Eliminate structs with function pointers for service APIs;
   instead, provide a library (still ending in _service.h) API
   that transmits the requests nicely to the respective
@@ -121,6 +124,8 @@
   thing given the potential for bugs.
 * There is no more TIME API function to do anything
   with 32-bit seconds
+* There is now a bandwidth API to handle 
+  non-trivial bandwidth utilization calculations
 
 
 PROBLEM GROUP 3 (statistics):
@@ -237,18 +242,15 @@
 * FILENAME metadata is killed by ECRS/FSUI to avoid
   exposing HOME, but what if the user set it manually?
 
-SOLUTION (draft, not done yet, details missing...):
+SOLUTION:
 * Eliminate threads from FS-APIs
-  => Open question: how to best write the APIs to
-     allow integration with diverse event loops
-     of GUI libraries?
-* Store FS-state always also on disk
-  => Open question: how to do this without 
-     compromising state/scalability?
+* Incrementally store FS-state always also on disk using many
+  small files instead of one big file
 * Have API to manipulate sharing tree before
   upload; have auto-construction modify FILENAME
   but allow user-modifications afterwards
 
+
 PROBLEM GROUP 7 (User experience):
 * Searches often do not return a sufficient / significant number of
   results
@@ -281,11 +283,12 @@
   - network traffic monitors (goal: eliminate)
   - IPC semaphores (goal: d-bus? / eliminate?)
   - second timers
-  - DNS lookup (goal: have async service; issue: still need synchronous 
-                resolution in places, current code may not be portable)
-  => code shrunk from 61 files to 34, 22k LOC to 15k LOC, 470k to 330k (with 
symbols)
 * New features in util:
   - scheduler
   - service and program boot-strap code
+  - bandwidth and time APIs
+  - buffered IO API
+  - HKDF implementation (crypto)
 * Major changes in util:
   - more expressive server (replaces selector)
+  - DNS lookup replaced by async service

Modified: gnunet/README
===================================================================
--- gnunet/README       2010-07-18 20:18:00 UTC (rev 12251)
+++ gnunet/README       2010-07-18 20:18:17 UTC (rev 12252)
@@ -7,13 +7,13 @@
 GNUnet is peer-to-peer framework focusing on security.  The first and
 primary application for GNUnet is anonymous file-sharing.  GNUnet is
 currently developed by a worldwide group of independent free software
-developers.  GNUnet is a part of the GNU project (http://www.gnu.org/).
+developers.  GNUnet is a GNU package (http://www.gnu.org/).
 
 This is an ALPHA release.  There are known and significant bugs as 
 well as many missing features in this release.  
 
-For a longer description of the GNUnet System see our webpages
-http://www.gnu.org/software/gnunet/ and https://ng.gnunet.org/.
+Additional documentation about GNUnet can be found at
+https://gnunet.org/.
 
 
 Dependencies:
@@ -45,40 +45,41 @@
 
 The fastest way is to use a binary package if it is available for your
 system.  For a more detailed description, read the installation
-instructions on the webpage at https://ng.gnunet.org/installation.
+instructions on the webpage at https://gnunet.org/installation.
 
+GNUnet requires the GNU MP library (http://www.gnu.org/software/gmp/)
+and libgcrypt (http://www.gnupg.org/).  You can specify the path to
+libgcrypt by passing "--with-gcrypt=PATH" to configure.  You will also
+need either sqlite (http://www.sqlite.org/) or MySQL
+(http://www.mysql.org/).
+
 If you install from source, you need to install GNU libextractor first
 (download from http://www.gnu.org/software/libextractor/).  We also
-recommend installing GNU libmicrohttpd.  Then you can start the actual
-GNUnet compilation process with:
+recommend installing GNU libmicrohttpd (download from
+http://www.gnu.org/software/libmicrohttpd/).  Then you can start the
+actual GNUnet compilation process with:
 
 $ ./configure --prefix=$HOME --with-extractor=$HOME
 $ make
 # make install
-# sudo -u gnunet gnunet-setup # (note: does not yet exist!)
+# sudo -u gnunet mkdir ~/.gnunet/ 
+# sudo -u gnunet touch ~/.gnunet/gnunet.conf
 # sudo -u gnunet gnunet-arm -s
 
-This will compile and install GNUnet to ~/bin/, ~/lib/ and ~/man/.
-gnunet-setup will create the configuration; this step is interactive.
-You can run gnunet-setup as root for a system-wide installation or as
-a particular user to create a personal installation.  It is
-recommended that you add a user "gnunet" to run "gnunet-arm".  You can
-then still run the end-user applications as another user.  If you
+This will compile and install GNUnet to $HOME/bin/, $HOME/lib/ and
+$HOME/share/ and start the system with the default configuration.  It
+is recommended that you add a user "gnunet" to run "gnunet-arm".  You
+can then still run the end-user applications as another user.  If you
 create a user "gnunet", it is recommended that you edit the
 configuration file slightly so that data can be stored in
 "/var/lib/gnunet"; you may also want to use "/etc/gnunet.conf" for the
-configuration in this case.  
+location of the configuration file in this case.
 
 Note that additional, per-user configuration files
 (~/.gnunet/gnunet.conf) need to be created by each user (for example,
 by running gnunet-setup). For more general information about the GNU
 build process read the INSTALL file.
 
-GNUnet requires the GNU MP library (http://www.gnu.org/software/gmp/)
-and libgcrypt.  You can specify the path to libgcrypt by passing
-"--with-gcrypt=PATH" to configure.  You will also need either sqlite
-(version 3 or higher) or MySQL (version 5.0 or higher).
-
 If you are compiling the code from subversion, you have to run
 ". bootstrap" before ./configure.  If you receive an error during the
 running of ". bootstrap" that looks like "macro `AM_PATH_GTK' not
@@ -91,26 +92,22 @@
 Configuration
 =============
 
-// FIXME: update this section once we have gnunet-setup!  GNUnet uses
-two types of configuration files, one for the daemon (called
-gnunetd.conf) and one for each user (gnunet.conf).  You can create and
-edit these configuration files with the gnunet-setup tool.  The
-defaults that are created the first time you run gnunet-setup are
-usually ok, you may want to adjust the limitations (space consumption,
-bandwidth, etc.) though.  The configuration files are human-readable;
-GNUnet's configuration is typically located at
-"~/.gnunet/gnunet.conf".
+GNUnet uses two types of configuration files, one that specifies the
+system-wide defaults (typically located in /usr/share/gnunet/defaults
+.conf) and a second one that overrides default values with
+user-specific preferences.  The user-specific configuration file
+should be located in "~/.gnunet/gnunet.conf" or its location can be
+specified by giving the "-c" option to the respective GNUnet
+application.
 
-You MUST create "~/.gnunet/gnunet.conf" explicitly before starting
-GNUnet, and the recommended way to do this is to run gnunet-setup.
-You can also create an empty configuration file, in which case 
-default values will be used for everything.
+The defaults that are shipped with the installation are usually ok,
+you may want to adjust the limitations (space consumption, bandwidth,
+etc.) though.  The configuration files are human-readable.  Note that
+you MUST create "~/.gnunet/gnunet.conf" explicitly before starting
+GNUnet.  You can either copy "defaults.conf" or simply create an empty
+file.
 
-If you want to use a different configuration file, pass the name of
-the configuration file as an argument with the option "-c" to any
-GNUnet application.
 
-
 Usage
 =====
 
@@ -126,19 +123,19 @@
 
 and offering 'the_file' via your web server.  Alternatively, you can
 run the build-in web server by adding '-p' to the OPTIONS value
-in the "hostlist" section of gnunet.conf.
+in the "hostlist" section of gnunet.conf and opening the respective
+HTTPPORT to the public.
 
-If the solution with the URL is not feasible for your situation, you
-can also add hosts manually.  Simply copy the hostkeys to
-"$SERVICEHOME/data/hosts/" (where $SERVICEHOME is the directory
-specified in the /etc/gnunet.conf configuration file).
+If the solution with the hostlist URL is not feasible for your
+situation, you can also add hosts manually.  Simply copy the hostkeys
+to "$SERVICEHOME/data/hosts/" (where $SERVICEHOME is the directory
+specified in the gnunet.conf configuration file).
 
 Now start the local node using "gnunet-arm -s".  GNUnet should run 24/7 if
 you want to maximize your anonymity.  
 
-// FIXME: gnunet-gtk is not yet supported in 0.9.x
-The GTK user interface is shipped separately from GNUnet.  After
-downloading and installing gnunet-gtk, you can invoke the GUI with:
+The GTK user interface is shipped separately.  After downloading and
+installing gnunet-gtk, you can invoke the GUI with:
 
 $ gnunet-gtk
 
@@ -202,13 +199,13 @@
 =======================
 
 In order to hide GNUnet's HTTP traffic perfectly, you might consider
-running GNUnet's HTTP transport on port 80.  However, you should not
-run GNUnet as root.  Instead, forward port 80 to say 8080 with this
-command (as root, in your startup scripts):
+running GNUnet's HTTP transport on port 80.  However, we do not
+recommend running GNUnet as root.  Instead, forward port 80 to say
+8080 with this command (as root, in your startup scripts):
 
 # iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT 
--to-ports 8080
 
-Then set in the HTTP section of gnunetd.conf the "ADVERTISED-PORT"
+Then set in the HTTP section of gnunet.conf the "ADVERTISED-PORT"
 to "80" and "PORT" to 8080.   You can do the same trick for the
 TCP and UDP transports if you want to map them to a priviledged
 port (from the point of view of the network).
@@ -227,12 +224,11 @@
 Stay tuned
 ==========
 
-* http://www.gnu.org/software/gnunet/
-* http://gnunet.org/
+* https://gnunet.org/
 * https://gnunet.org/bugs/
-* https://ng.gnunet.org/
+* https://gnunet.org/svn/
+* http://www.gnu.org/software/gnunet/
 * http://mail.gnu.org/mailman/listinfo/gnunet-developers
 * http://mail.gnu.org/mailman/listinfo/help-gnunet
 * http://mail.gnu.org/mailman/listinfo/info-gnunet
 * http://mail.gnu.org/mailman/listinfo/gnunet-svn
-

Modified: gnunet/TODO
===================================================================
--- gnunet/TODO 2010-07-18 20:18:00 UTC (rev 12251)
+++ gnunet/TODO 2010-07-18 20:18:17 UTC (rev 12252)
@@ -4,6 +4,8 @@
     [On W32, we need to select after calling socket before doing connect etc.]
 * TRANSPORT:
   - HTTP backend [MW]
+  - UDP fails non-deterministically on niko; suspect that it uses 
hostname-derived,
+    invalid 10.0.0.x address for test-message (which creates transmission 
failure) [CG]
   - Jun 27 11:51:54 core-7670 ERROR Assertion failed at 
gnunet-service-core.c:3616.
     (transport notified us that we connected to ourselves!!!)
 * CORE:
@@ -11,25 +13,22 @@
     disconnect in a timely fashion (waits for connection timeout);
     need to figure a way to make it near-instant in those cases 
     (i.e. rapid reduction in timeout on transport-level disconnect) [CG]
+    => may have been fixed with instant-notification of disconnect
+       to core on session-oriented connection hick-up; not perfect but
+       likely good enough until we get ATS going; still should be tested...
   - derived key generation [Nils]
 * DHT: [Nate]
   - implement DHT service 
-  - implement testcases
   - implement performance tests
 * FS: [CG]
   - service:
     + trust: do not charge when "idle" / load considerations (migration, 
routing)
     + artificial delays
-    + active reply route caching design & implementation of service; gap 
extension!
+    + active reply route caching design & implementation; gap extension!
   - non-anonymous FS service (needs DHT)
     + DHT integration for search
     + CS-DHT-functions (DHT-put of LOC)
     + P2P-functions (DHT-get)
-  - implement FS performance tests
-    + insert
-    + download
-    + search
-    + unindex
 * GNUNET-GTK: [CG]
   - implement publish dialog details:
     + open (import directory structure)
@@ -56,9 +55,6 @@
   - right-clicking on NS list in search dialog should open menu that allows 
     + viewing full meta data 
     + deletion of namespace info
-  - extend peer dialog with green-yellow-red connectivity status lights
-  - extend peer dialog with country flags and names
-  - implement statistics tabs
   - do meaningful update to status line (starting up, peer running, 
#connections, shutdown, ...)
   - implement unindex operation (use dialog with all indexed files for 
selection)
   - implement download by URI dialog; figure out where to display those 
downloads!
@@ -74,10 +70,10 @@
 
 
 0.9.0pre3:
+* Determine RC bugs and fix those!
 * TRACEKIT: [MW]
   - good to have for DV/DHT evaluation!
-* Determine RC bugs and fix those!
-* DATASTORE:
+* DATASTORE: [NN]
   - GNUNET_DATASTORE_cancel method not tested
 * TESTING: [Nate]
   - test basic peer re-configure 
@@ -97,14 +93,29 @@
   - reconstruct IBLOCKS from DBLOCKS if possible (during download; see FIXME 
in fs_download)
   - add support for pushing "already seen" search results to FS service for 
bloomfilter (can wait)
   - use different 'priority' for probe downloads vs. normal downloads
+  - implement FS performance tests
+    + insert
+    + download
+    + search
+    + unindex
 * ARM: [CG/Safey]
   - better tracking of which config changes actually need to cause process 
restarts by ARM.
   - handle gnunet-arm -k in combination with auto-start magic (what is the 
right thing here?)
   - discover dependencies between services
-* STATISTICS:
+* STATISTICS: [NN]
   - test notification-based statistics API
+  - implement statistics GUI
+* PEERINFO: [NN]
+  - move peerinfo to new GUI?
+  - extend peer dialog with green-yellow-red connectivity status lights
+  - extend peer dialog with country flags and names
 * SETUP:
   - design & implement new setup tool
+* GNUNET-GTK:
+  - add tool bar
+* MONKEY: [Safey]
+  - '-m EMAIL' option for alternative e-mail TO
+  - '-f FILENAME' option to write  report to file instead of e-mail (for 
testing!)
 
 0.9.0:
 * new webpage:
@@ -145,13 +156,13 @@
     (Note: build library always, build service when libxml2/etc. are available)
 * FS: [CG]
   - Remove KBlocks in gnunet-unindex (see discussion with Kenneth Almquist on 
gnunet-devs in 9/2009)
-* PEERINFO: [CG]
+* PEERINFO: [NN]
   - expire 'ancient' HELLOs (those without valid addresses AND that 
     we have not 'used' (for their public keys) in a while; need a way
     to track actual 'use')
   - make sue we also trigger notifications whenever HELLOs expire
 * VPN
-* UTIL:
+* UTIL: [CG]
   - allow limiting UNIX socket access by UID/GID
 
 

Modified: gnunet/contrib/report.sh
===================================================================
--- gnunet/contrib/report.sh    2010-07-18 20:18:00 UTC (rev 12251)
+++ gnunet/contrib/report.sh    2010-07-18 20:18:17 UTC (rev 12252)
@@ -7,8 +7,6 @@
   WHICH=type
 fi
 
-echo "On some systems, you may need to change /bin/sh to point to bash"
-echo
 echo "Please submit the following information with your bug report: "
 echo "--------------------------------------------------------------"
 OS=`uname -s 2>/dev/null`
@@ -86,11 +84,19 @@
 TEST=`$WHICH gnunetd 2>/dev/null`
 if test -n "$TEST"; then
   gnunetd -v | sed -e "s/v//" 2>/dev/null |\
-    awk '{print "GNUnet         : "$2}'
+    awk '{print "GNUnet 0.8     : "$2 (may conflict!)}'
 else
-  echo "GNUnet         : Not Found"
+  echo "GNUnet 0.8     : Not Found (good)"
 fi
 
+TEST=`$WHICH gnunet-arm 2>/dev/null`
+if test -n "$TEST"; then
+  gnunet-arm -v | sed -e "s/v//" 2>/dev/null |\
+    awk '{print "GNUnet 0.9     : "$2}'
+else
+  echo "GNUnet 0.9     : Not Found"
+fi
+
 TEST=`$WHICH libgcrypt-config 2> /dev/null`
 if test -n "$TEST"; then
   libgcrypt-config --version 2> /dev/null | \
@@ -169,13 +175,6 @@
 fi
 
 
-TEST=`grep "Id:" /usr/include/adns.h 2> /dev/null`
-if test -n "$TEST"; then
-  echo "$TEST" | awk '{print "GNU adns       : "$4}'
-else
-  echo "GNU adns       : Not found"
-fi
-
 TEST=`which qmake 2> /dev/null`
 if test -x "$TEST"; then
   qmake --version | tail -n 1 | awk '{print "Qt             : "$4}'

Modified: gnunet/mkinstalldirs
===================================================================
--- gnunet/mkinstalldirs        2010-07-18 20:18:00 UTC (rev 12251)
+++ gnunet/mkinstalldirs        2010-07-18 20:18:17 UTC (rev 12252)
@@ -1,7 +1,7 @@
 #! /bin/sh
 # mkinstalldirs --- make directory hierarchy
 
-scriptversion=2006-05-11.19
+scriptversion=2009-04-28.21; # UTC
 
 # Original author: Noah Friedman <address@hidden>
 # Created: 1993-05-16
@@ -157,5 +157,6 @@
 # eval: (add-hook 'write-file-hooks 'time-stamp)
 # time-stamp-start: "scriptversion="
 # time-stamp-format: "%:y-%02m-%02d.%02H"
-# time-stamp-end: "$"
+# time-stamp-time-zone: "UTC"
+# time-stamp-end: "; # UTC"
 # End:


Property changes on: gnunet/src/dht
___________________________________________________________________
Modified: svn:ignore
   - test_dht_api
Makefile.in
Makefile
gnunet-service-dht
gnunet-dht-put
gnunet-dht-get-peer
gnunet-dht-get
.deps

   + test_dht_twopeer_put_get
test_dht_twopeer
test_dht_api
Makefile.in
Makefile
gnunet-service-dht
gnunet-dht-put
gnunet-dht-get-peer
gnunet-dht-get
.deps