[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[GNUnet-SVN] r18024 - in gnunet-update: doc gnunet_update
|
From: |
gnunet |
|
Subject: |
[GNUnet-SVN] r18024 - in gnunet-update: doc gnunet_update |
|
Date: |
Sat, 5 Nov 2011 16:12:13 +0100 |
Author: harsha
Date: 2011-11-05 16:12:13 +0100 (Sat, 05 Nov 2011)
New Revision: 18024
Modified:
gnunet-update/doc/metadata.txt
gnunet-update/gnunet_update/dependency.py
gnunet-update/gnunet_update/install.py
gnunet-update/gnunet_update/metadata.py
gnunet-update/gnunet_update/package.py
gnunet-update/gnunet_update/util.py
Log:
modified metadata format, added code for determining what external dependencies
need to be installed
Modified: gnunet-update/doc/metadata.txt
===================================================================
--- gnunet-update/doc/metadata.txt 2011-11-05 14:46:54 UTC (rev 18023)
+++ gnunet-update/doc/metadata.txt 2011-11-05 15:12:13 UTC (rev 18024)
@@ -1,34 +1,48 @@
-File: doc/metadata.txt
-Author: Sree Harsha Totakura
+#+TITLE: Metadata File Format Description
+#+AUTHOR: Sree Harsha Totakura
+#+EMAIL: address@hidden
* Metadata file:
- The metadata file consists of sections seperated by a line containing
+ The metadata file consists of sections separated by a line containing
`%%'. Currently there are 3 sections:
1. The metadata header which consists of information about the host (the
machine on which the package was built)
2. The metadata body which is a list of the dependencies of binary objects in
the package
- 3. A list of signatures of the binary objects to verify the authenticity of
- each binary object in the package
+ 3. A list of signatures of the binary objects and dependencies to verify the
+ authenticity. A list mapping dependency name and to the dependency file
+ and its signature
- IMPORTANT: The lists in metadata body and signatures should correspond to
- each other. For every object listed in metadata body, there should be a
+ IMPORTANT: For every object listed in metadata body, there should be a
signature associated with that file in signature list
** Metadata header
It consists of information about the host system. This is needed to identify
- the subset of machin e which have the capability to execute the binary
+ the subset of machine which have the capability to execute the binary
objects in the package.
The header consists of KEY:VALUE pairs with one such pair in each line.
The valid KEYs are:
- * MACHINE: Host machine type. e.g: i386
- * OS: Host operating system. e.g: Linux
- * PKEY: Public key of the host in hexadecimal digits
+ * MACHINE:
+ Host's machine hardware name. This is the output of the
+ command `uname --machine'.
+
+ E.g: On a system with AMD Opteron running Ubuntu Lucid 64-bit, this
+ key will have the value: `x86\_64'
+
+ * SYSTEM:
+ Host's kernel name. This is the output of the command `uname
--kernel-name'
+
+ E.g: On a system running GNU/Linux, this key will have the
+ value: `Linux'. On a system running FreeBSD, this key will have
+ the value: `FreeBSD'
+
+ * PKEY: Public key of the packager in hexadecimal digits
+
* RELEASE: Release number for the package. This is independent of Gnunet
release and is intended for packagers and maintainers to identify their
own release information
@@ -37,19 +51,75 @@
This is a list of dependencies for each binary object.
- If a binary object foo needs a shared library libbar.so.X where X is the
- major number for libbar, the dependency information for A is stored as
- follows:
+ E.g: If a binary object foo is linked to a shared library libbar.so, it is
+ represented as:
- A;libbar.so.X;X;Y;Z
+ foo;libbar.so
+
+** List of signatures of binary object and dependencies
+
+ This section has 2 lists:
- where Y, Z are the minor and revision numbers of libbar.so.X. In case minor
- and revision numbers for libbar cannot be determined, they are set to -1
+*** List of signatures of binary objects
-** List of signatures
+ This is a list of sha512 digest of each file in the package expressed in
+ hexadecimal format. For each file an entry begins with its file name
+ followed by a `:' and then the sha512 digest.
+
+#+BEGIN_EXAMPLE
+ file name;<sha512 digest of the file>
+#+END_EXAMPLE
+
+*** List of signatures of dependencies
- This is a list of sha512 digest of each file in the package expressed in
- hexadecimal format. For each file an entry begins with its file name
- followed by a `:' and then the sha512 digest.
+ This is listing that maps dependency name(soname) to a file
+ name(real name) and its signature. This is required to serve the purpose of
+ mapping symbolic links of shared libraries to the actual files. E.g: if
+ libfoo.so.1 is a dependency needed by a binary object and on the host's
file
+ system libfoo.so.1 is a symbolic link to libfoo.so.1.0.1, it is represented
+ in the list as:
- file_name: sha512 digest of the file
+#+BEGIN_EXAMPLE
+ libfoo.so.1;libfoo.so.1.0.1;<sha512 digest of file libfoo.so.1.0.1>
+#+END_EXAMPLE
+
+ For a brief explanation regarding soname and realname of libraries see
+ [[Library Naming Conventions]]
+
+ Note that in many cases the realname of a library contains information about
+ the library's major, minor and release numbers. Usually these are the last
+ three digits separated by `.' in the library's real name. When such
+ information is available it can be used to find the dependencies which are
+ already satisfied by the already installed shared libraries on the target
+ system. A dependency is said to be compatible if any of the following
+ conditions hold true:
+
+ * There exists a library already installed on the host system whose real
+ name matches the real name of the dependency
+
+ * There exists a library already installed on the host system which has the
+ same soname, same major and minor number but a greater release
+ number. Although a library with lower release number doesn't break binary
+ compatibility, a greater release number of the same library may have some
+ of the existing bugs fixed and using it in place of older library is
+ usually desirable
+
+ * There exists a library already installed on the host system which has the
+ same major number but a greater minor number
+
+* Appendix
+
+# <<Library Naming Conventions>>
+** Library Naming Conventions
+
+ On systems that support shared libraries, the library naming is divided into
+ a library name also called soname and a realname. A library's soname
+ essentially identifies the name of the library which is used by the dynamic
+ linker while to locate the needed library at run time. The realname of a
+ library is the actual file which contains the binary code of the
+ library. Usually, the soname of a library is a symbolic link to its
+ realname. E.g: on a GNU/Linux system the soname for foo library would be
+ libfoo.so.1 and the realname would be libfoo.so.1.0.1
+
+ For more information on shared library naming conventions, please refer:
+ http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html
Modified: gnunet-update/gnunet_update/dependency.py
===================================================================
--- gnunet-update/gnunet_update/dependency.py 2011-11-05 14:46:54 UTC (rev
18023)
+++ gnunet-update/gnunet_update/dependency.py 2011-11-05 15:12:13 UTC (rev
18024)
@@ -23,28 +23,37 @@
#
#File for holding the Dependency and BinaryObject classes
+import os
from operator import xor
-from hashlib import sha512
+import util
class Dependency:
"""Class for holding data for a dependency"""
- major = minor = rev = None
+ major = minor = rel = None
+ name = None
+ path = None
+ realname = None
+
def __init__(self, name, path=None):
"""Creates a new dependency object with name and path."""
self.name = name
self.path = path
+ if path is not None:
+ self.realname = os.path.basename(os.path.realpath(path))
+ self.hash = util.sha512_hexdigest(path)
def __eq__(self, other):
"""Compares two dependency objects. Returns True if both have same name
and path, false otherwise.
"""
- return (self.name == other.name and self.path == other.path)
+ return (self.name == other.name)
def __hash__(self):
- """Calculates the hashes of name and path. Returns XOR of hashes."""
- return xor(hash(self.name), hash(self.path))
+ """Calculates the hashes of dependency name."""
+ return hash(self.name)
+
class BinaryObject:
"""Class representing executable code."""
@@ -54,18 +63,8 @@
self.path = path
self._deps = list()
- if path != None:
- #Calculate the hash of this binary object
- hash_obj = sha512()
- object_file = open(path, "rb")
- while True:
- #read 512 bytes - suitable for sha512 blocksdepif
- data = object_file.read(512)
- if 0 == len(data): #End of file is reached
- self.hash = hash_obj.hexdigest()
- break
- hash_obj.update(data)
- object_file.close()
+ if path is not None:
+ self.hash = util.sha512_hexdigest(path)
def add_dependency(self, dep):
"""Adds dep object to the list of dependencies."""
@@ -77,10 +76,7 @@
def _dependency_ascii(self, dep):
"""Given a dependency, returns an ascii line describing it."""
- dep_str = self.name + ";" + dep.name + ";"
- dep_str += "-1;" if dep.major == None else dep.major + ";"
- dep_str += "-1;" if dep.minor == None else dep.minor + ";"
- dep_str += "-1" if dep.rev == None else dep.rev
+ dep_str = self.name + ";" + dep.name
return dep_str + "\n"
def dependency_listlines(self):
Modified: gnunet-update/gnunet_update/install.py
===================================================================
--- gnunet-update/gnunet_update/install.py 2011-11-05 14:46:54 UTC (rev
18023)
+++ gnunet-update/gnunet_update/install.py 2011-11-05 15:12:13 UTC (rev
18024)
@@ -58,7 +58,7 @@
proc.stdout.close()
parsed_dep_list = util.parse_ldconfig_output(ldconfig_output)
def create_dependency(parsed_dep_line):
- return Dependency(parsed_dep_line[0], parsed_dep_line[-1])
+ return Dependency(parsed_dep_line[0])
return map(create_dependency, parsed_dep_list)
def main():
@@ -94,11 +94,11 @@
package_tarfile.close()
sys.exit(2)
- #check whether the host os and machine architecture match
- host_os = platform.system()
+ #check whether the system and machine architecture match
+ host_system = platform.system()
host_machine = platform.machine()
- if metadata.os != host_os or metadata.machine != host_machine:
+ if metadata.system != host_system or metadata.machine != host_machine:
print "The given package is not suited for this platform."
sys.exit(1)
@@ -109,17 +109,16 @@
os.mkdir(install_dir, 0755)
installed_deps = get_installed_deps() # already available dependencies
- new_binary_objects = metadata.get_binary_objects() # To be installed
objects
+ new_binary_objects = metadata.binary_objects # To be installed objects
installed_deps.sort(key=(lambda dep: dep.name))
- needed_deps = list()
- for binary_object in new_binary_objects:
- for dep in binary_object.get_dependencies():
- if dep not in needed_deps:
- needed_deps.append(dep)
+ needed_deps = metadata.dependencies
+
+ to_be_installed_deps = [(dep) for dep in needed_deps if dep not in
installed_deps]
+
# TODO: Find the dependencies to be installed
- for dep in installed_deps:
+ for dep in to_be_installed_deps:
print "%s -- %s" % (dep.name, dep.path)
#FIXME: Security warning! Perhaps we should examin the contents of tarfile
Modified: gnunet-update/gnunet_update/metadata.py
===================================================================
--- gnunet-update/gnunet_update/metadata.py 2011-11-05 14:46:54 UTC (rev
18023)
+++ gnunet-update/gnunet_update/metadata.py 2011-11-05 15:12:13 UTC (rev
18024)
@@ -27,26 +27,19 @@
class Metadata:
"""Class for holding metadata information."""
machine = None
- os = None
+ system = None
pkey = None
release = None
- _binary_objects = None
+ binary_objects = None
+ dependencies = None
- def __init__(self, machine=None, os=None, pkey=None,
+ def __init__(self, machine=None, system=None, pkey=None,
release=None):
self.machine = machine
- self.os = os
+ self.system = system
self.pkey = pkey
self.release = release
- def set_binary_objects(self, binary_objects):
- """Setter for _binary_objects."""
- self._binary_objects = binary_objects
-
- def get_binary_objects(self):
- """Getter for binary_objects."""
- return self._binary_objects
-
def write_to_file(self, path=None):
"""Saves metadata to a file and returns the path of that file.
@@ -71,20 +64,24 @@
#write the header
if self.machine != None: writeln_("MACHINE:" + self.machine)
- if self.os != None: writeln_("OS:" + self.os)
+ if self.system != None: writeln_("SYSTEM:" + self.system)
if self.pkey != None: writeln_("PKEY:" + self.pkey)
if self.release != None: writeln_("RELEASE:" + self.release)
#write the metadata body
writeln_("%%") #section seperator
- for binary_object in self._binary_objects:
+ for binary_object in self.binary_objects:
f.writelines(binary_object.dependency_listlines())
-
- #write the signatures
+
+ #write the signatures of binary objects
writeln_("%%") #section seperator
- for binary_object in self._binary_objects:
- writeln_(binary_object.name + ":" + binary_object.hash)
+ for binary_object in self.binary_objects:
+ writeln_(binary_object.name + ";" + binary_object.hash)
+ # write the signatures of dependencies
+ for dep in self.dependencies:
+ writeln_(dep.name + ";" + dep.realname + ";" + dep.hash)
+
#close file
if None == path:
tmp_file.close()
@@ -115,8 +112,8 @@
value = tokens[1][:-1] # last character will be `\n'
if "MACHINE" == key:
self.machine = value
- elif "OS" == key:
- self.os = value
+ elif "SYSTEM" == key:
+ self.system = value
elif "PKEY" == key:
self.pkey = value
elif "RELEASE" == key:
@@ -124,7 +121,8 @@
# read until next `%%'
seen_bin_object = None
- self._binary_objects = list()
+ self.binary_objects = list()
+ self.dependencies = dict()
while True:
read_line = f.readline()
if len(read_line) == 0:
@@ -134,26 +132,26 @@
break
tokens = read_line.split(';')
bin_name = tokens[0]
- dep_name = tokens[1]
- dep_maj = int(tokens[2])
- dep_min = int(tokens[3])
- dep_rev = int(tokens[4][:-1])# last character will be `\n'
- if None == seen_bin_object or seen_bin_object.name != bin_name:
+ dep_name = tokens[1][:-1] # last character will be `\n'
+ if seen_bin_object is None or seen_bin_object.name != bin_name:
seen_bin_object = BinaryObject(name=bin_name)
- self._binary_objects.append(seen_bin_object)
+ self.binary_objects.append(seen_bin_object)
dep = Dependency(dep_name)
- if -1 != dep_maj: dep.maj = dep_maj
- if -1 != dep_min: dep.min = dep_min
- if -1 != dep_rev: dep.rev = dep_rev
- seen_bin_object.add_dependency(dep)
+ if dep not in self.dependencies:
+ self.dependencies[dep] = dep
+ else:
+ # use the one which is already existing
+ seen_bin_object.add_dependency(self.dependencies[dep])
+ # read until next `%%'
# read the hashes from signatures
- for binary in self._binary_objects:
+ for binary in self.binary_objects:
read_line = f.readline()
+ print read_line
if len(read_line) == 0:
print "Unrecognized metadata file"
error_exit()
- tokens = read_line.split(':')
+ tokens = read_line.split(';')
bin_name = tokens[0]
hash = tokens[1][:-1] # last character will be `\n'
# FIXME: We are reading based on the assumption that the order of
@@ -164,4 +162,23 @@
print "Unrecognized file format"
error_exit()
binary.hash = hash
+
+ # read the dependency name, file name(real name) and its hash
+ while True:
+ read_line = f.readline()
+ print read_line
+ if len(read_line) == 0:
+ break
+ tokens = read_line.split(';')
+ dep_name = tokens[0]
+ dep_realname = tokens[1]
+ dep_hash = tokens[2][:-1] # last character is `\n'
+ dep = Dependency(dep_name)
+ # dep should already be in self.dependencies
+ # if we get a keyerror here, then something went wrong
+ dep = self.dependencies[dep] # gets the correct object
+ dep.name = dep_name
+ dep.realname = dep_realname
+ dep.hash = dep_hash
+
f.close()
Modified: gnunet-update/gnunet_update/package.py
===================================================================
--- gnunet-update/gnunet_update/package.py 2011-11-05 14:46:54 UTC (rev
18023)
+++ gnunet-update/gnunet_update/package.py 2011-11-05 15:12:13 UTC (rev
18024)
@@ -115,7 +115,7 @@
def get_deps(install_dir):
"""Extract dependencies from ldd output."""
- regex =
re.compile(".*\.so\.(?P<major>\d+)(?:\.(?P<minor>\d+))?(?:\.(?P<rev>\d+))?$")
+# regex =
re.compile(".*\.so\.(?P<major>\d+)(?:\.(?P<minor>\d+))?(?:\.(?P<rev>\d+))?$")
for root, dirs, files in os.walk(install_dir):
for file in files:
file_path = os.path.join(root, file)
@@ -144,18 +144,18 @@
#check in cache if we already saw this dependency
if dep not in dependencies:
#Retrieve major number of the dependency
- match = regex.match(dep_data[-1])
- if match:
- match2 = None
- if os.path.islink(dep_data[-1]):
- match2 =
regex.match(os.path.realpath(dep_data[-1]))
+ # match = regex.match(dep_data[-1])
+ # if match:
+ # match2 = None
+ # if os.path.islink(dep_data[-1]):
+ # match2 =
regex.match(os.path.realpath(dep_data[-1]))
- (dep.major,
- dep.minor,
- dep.rev) = match2.groups() if match2 \
- else match.groups()
- else:
- raise Exception('Unhandled discrepancy.')
+ # (dep.major,
+ # dep.minor,
+ # dep.rel) = match2.groups() if match2 \
+ # else match.groups()
+ # else:
+ # raise Exception('Unhandled discrepancy.')
dependencies[dep] = dep
else:
@@ -188,9 +188,10 @@
test_dependency_collection()
metadata = Metadata(machine=platform.machine(),
- os=platform.system(),
+ system=platform.system(),
release="0")
- metadata.set_binary_objects(binary_objects)
+ metadata.binary_objects = binary_objects
+ metadata.dependencies = dependencies
#package the installed files
tar_file = tarfile.open(package_file + ".tgz", 'w:gz')
Modified: gnunet-update/gnunet_update/util.py
===================================================================
--- gnunet-update/gnunet_update/util.py 2011-11-05 14:46:54 UTC (rev 18023)
+++ gnunet-update/gnunet_update/util.py 2011-11-05 15:12:13 UTC (rev 18024)
@@ -21,10 +21,13 @@
#
# Utility function library
-def parse_ldd_output(ldd_output):
+from hashlib import sha512
+
+def parse_ldd_output(ldd_output, splitted_input=False):
"""Parses ldd output.
ldd_ouput : Output of `ldd <program>' to identify dependencies
+ splitted_input: Is the input already splitted into lines? Default is False
Returns a list of 2 element lists having the dependency name as the first
element and the path of the dependency as the second one
@@ -38,7 +41,8 @@
tokens[-1] = tokens[-1].rsplit(' ', 1)[0]
return tokens
- return map(extract_deps, ldd_output.splitlines())
+ return map(extract_deps,
+ ldd_output.splitlines() if not splitted_input else ldd_output)
def parse_ldconfig_output(ldconfig_output):
"""Parses ldconfig output.
@@ -55,5 +59,24 @@
def massage_head(dep_data):
dep_data[0] = dep_data[0].split(' ')[0] # take the first part
return dep_data
+ #ldconfig seems to print the total number of libraries found in the first
+ #line of its output
+ #FIXME: Dirty hack
+ return map(massage_head,
+ parse_ldd_output(ldconfig_output.splitlines()[1:],
+ splitted_input=True))
- return map(massage_head, parse_ldd_output(ldconfig_output))
+def sha512_hexdigest(path):
+ """Return the sha512 hexdigest of the file at path."""
+
+ hash_obj = sha512()
+ object_file = open(path, "rb")
+ while True:
+ #read 512 bytes - suitable for sha512 blocks
+ data = object_file.read(512)
+ if 0 == len(data): #End of file is reached
+ hexdigest = hash_obj.hexdigest()
+ break
+ hash_obj.update(data)
+ object_file.close()
+ return hexdigest
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [GNUnet-SVN] r18024 - in gnunet-update: doc gnunet_update,
gnunet <=