freetype-commit
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Git][freetype/freetype][download-test-fonts] tests/scripts/download-tes


From: David Turner (@david.freetype)
Subject: [Git][freetype/freetype][download-test-fonts] tests/scripts/download-test-fonts.sh rewrite in Python3
Date: Thu, 15 Jul 2021 11:21:19 +0000

David Turner pushed to branch download-test-fonts at FreeType / FreeType

Commits:

4 changed files:

Changes:

  • ChangeLog
    1
    +2021-07-15  David Turner  <david@freetype.org>
    
    2
    +
    
    3
    +	Replaces download-test-fonts.sh with download-test-fonts.py which
    
    4
    +	does the same work, and also avoids downloading anything if the
    
    5
    +	files are already installed with the right content.
    
    6
    +
    
    7
    +	Now uses the first 8 byte of each file's sha256 hash for the digest.
    
    8
    +
    
    9
    +	* tests/scripts/download-test-fonts.sh: Removed
    
    10
    +	* tests/scripts/download-test-fonts.py: New script
    
    11
    +	* tests/README.md: Updated
    
    12
    +
    
    1 13
     2021-07-15  Alex Richardson  <Alexander.Richardson@cl.cam.ac.uk>
    
    2 14
     
    
    3 15
     	Support architectures where `long` is smaller than pointers.
    

  • tests/README.md
    ... ... @@ -4,7 +4,7 @@
    4 4
     
    
    5 5
     ### Download test fonts
    
    6 6
     
    
    7
    -Run the `tests/scripts/download-fonts.sh` script, which will
    
    7
    +Run the `tests/scripts/download-fonts.py` script, which will
    
    8 8
     download test fonts to the `tests/data/` directory first.
    
    9 9
     
    
    10 10
     ### Build the test programs
    

  • tests/scripts/download-test-fonts.py
    1
    +#!/usr/bin/env python3
    
    2
    +
    
    3
    +"""Download test fonts used by the FreeType regression test programs.
    
    4
    +These will be copied to $FREETYPE/tests/data/ by default.
    
    5
    +"""
    
    6
    +
    
    7
    +import argparse
    
    8
    +import collections
    
    9
    +import hashlib
    
    10
    +import io
    
    11
    +import os
    
    12
    +import requests
    
    13
    +import sys
    
    14
    +import zipfile
    
    15
    +
    
    16
    +from typing import Callable, List, Optional, Tuple
    
    17
    +
    
    18
    +# The list of download items describing the font files to install.
    
    19
    +# Each download item is a dictionary with one of the following schemas:
    
    20
    +#
    
    21
    +# - File item:
    
    22
    +#
    
    23
    +#      file_url
    
    24
    +#        Type: URL string.
    
    25
    +#        Required: Yes.
    
    26
    +#        Description: URL to download the file from.
    
    27
    +#
    
    28
    +#      install_name
    
    29
    +#        Type: file name string
    
    30
    +#        Required: No
    
    31
    +#        Description: Installation name for the font file, only provided if it
    
    32
    +#          must be different from the original URL's basename.
    
    33
    +#
    
    34
    +#      hex_digest
    
    35
    +#        Type: hexadecimal string
    
    36
    +#        Required: No
    
    37
    +#        Description: Digest of the input font file.
    
    38
    +#
    
    39
    +# - Zip items:
    
    40
    +#
    
    41
    +#   These items correspond to one or more font files that are embedded in a
    
    42
    +#   remote zip archive. Each entry has the following fields:
    
    43
    +#
    
    44
    +#      zip_url
    
    45
    +#        Type: URL string.
    
    46
    +#        Required: Yes.
    
    47
    +#        Description: URL to download the zip archive from.
    
    48
    +#
    
    49
    +#      zip_files
    
    50
    +#        Type: List of file entries (see below)
    
    51
    +#        Required: Yes
    
    52
    +#        Description: A list of entries describing a single font file to be
    
    53
    +#          extracted from the archive
    
    54
    +#
    
    55
    +# Apart from that, some schemas are used for dictionaries used inside download
    
    56
    +# items:
    
    57
    +#
    
    58
    +# - File entries:
    
    59
    +#
    
    60
    +#   These are dictionaries describing a single font file to extract from an archive.
    
    61
    +#
    
    62
    +#      filename
    
    63
    +#        Type: file path string
    
    64
    +#        Required: Yes
    
    65
    +#        Description: Path of source file, relative to the archive's top-level directory.
    
    66
    +#
    
    67
    +#      install_name
    
    68
    +#        Type: file name string
    
    69
    +#        Required: No
    
    70
    +#        Description: Installation name for the font file, only provided if it must be
    
    71
    +#          different from the original filename value.
    
    72
    +#
    
    73
    +#      hex_digest
    
    74
    +#        Type: hexadecimal string
    
    75
    +#        Required: No
    
    76
    +#        Description: Digest of the input source file
    
    77
    +#
    
    78
    +_DOWNLOAD_ITEMS = [
    
    79
    +    {
    
    80
    +        "zip_url": "https://github.com/python-pillow/Pillow/files/6622147/As.I.Lay.Dying.zip",
    
    81
    +        "zip_files": [
    
    82
    +            {
    
    83
    +                "filename": "As I Lay Dying.ttf",
    
    84
    +                "install_name": "As.I.Lay.Dying.ttf",
    
    85
    +                "hex_digest": "ef146bbc2673b387",
    
    86
    +            },
    
    87
    +        ],
    
    88
    +    },
    
    89
    +]
    
    90
    +
    
    91
    +
    
    92
    +def digest_data(data: bytes):
    
    93
    +    """Compute the digest of a given input byte string, which are the first 8 bytes of its sha256 hash."""
    
    94
    +    m = hashlib.sha256()
    
    95
    +    m.update(data)
    
    96
    +    return m.digest()[:8]
    
    97
    +
    
    98
    +
    
    99
    +def check_existing(path: str, hex_digest: str):
    
    100
    +    """Return True if |path| exists and matches |hex_digest|."""
    
    101
    +    if not os.path.exists(path) or hex_digest is None:
    
    102
    +        return False
    
    103
    +
    
    104
    +    with open(path, "rb") as f:
    
    105
    +        existing_content = f.read()
    
    106
    +
    
    107
    +    return bytes.fromhex(hex_digest) == digest_data(existing_content)
    
    108
    +
    
    109
    +
    
    110
    +def install_file(content: bytes, dest_path: str):
    
    111
    +    """Write a byte string to a given destination file.
    
    112
    +
    
    113
    +    Args:
    
    114
    +      content: Input data, as a byte string
    
    115
    +      dest_path: Installation path
    
    116
    +    """
    
    117
    +    parent_path = os.path.dirname(dest_path)
    
    118
    +    if not os.path.exists(parent_path):
    
    119
    +        os.makedirs(parent_path)
    
    120
    +
    
    121
    +    with open(dest_path, "wb") as f:
    
    122
    +        f.write(content)
    
    123
    +
    
    124
    +
    
    125
    +def download_file(url: str, expected_digest: Optional[bytes] = None):
    
    126
    +    """Download a file from a given URL.
    
    127
    +
    
    128
    +    Args:
    
    129
    +      url: Input URL
    
    130
    +      expected_digest: Optional digest of the file
    
    131
    +        as a byte string
    
    132
    +    Returns:
    
    133
    +      URL content as binary string.
    
    134
    +    """
    
    135
    +    r = requests.get(url, allow_redirects=True)
    
    136
    +    content = r.content
    
    137
    +    if expected_digest is not None:
    
    138
    +        digest = digest_data(r.content)
    
    139
    +        if digest != expected_digest:
    
    140
    +            raise ValueError(
    
    141
    +                "%s has invalid digest %s (expected %s)"
    
    142
    +                % (url, digest.hex(), expected_digest.hex())
    
    143
    +            )
    
    144
    +
    
    145
    +    return content
    
    146
    +
    
    147
    +
    
    148
    +def extract_file_from_zip_archive(
    
    149
    +    archive: zipfile.ZipFile,
    
    150
    +    archive_name: str,
    
    151
    +    filepath: str,
    
    152
    +    expected_digest: Optional[bytes] = None,
    
    153
    +):
    
    154
    +    """Extract a file from a given zipfile.ZipFile archive.
    
    155
    +
    
    156
    +    Args:
    
    157
    +      archive: Input ZipFile objec.
    
    158
    +      archive_name: Archive name or URL, only used to generate a human-readable error
    
    159
    +        message.
    
    160
    +      filepath: Input filepath in archive.
    
    161
    +      expected_digest: Optional digest for the file.
    
    162
    +    Returns:
    
    163
    +      A new File instance corresponding to the extract file.
    
    164
    +    Raises:
    
    165
    +      ValueError if expected_digest is not None and does not match the extracted file.
    
    166
    +    """
    
    167
    +    file = archive.open(filepath)
    
    168
    +    if expected_digest is not None:
    
    169
    +        digest = digest_data(archive.open(filepath).read())
    
    170
    +        if digest != expected_digest:
    
    171
    +            raise ValueError(
    
    172
    +                "%s in zip archive at %s has invalid digest %s (expected %s)"
    
    173
    +                % (filepath, archive_name, digest.hex(), expected_digest.hex())
    
    174
    +            )
    
    175
    +    return file.read()
    
    176
    +
    
    177
    +
    
    178
    +def _get_and_install_file(
    
    179
    +    install_path: str,
    
    180
    +    hex_digest: Optional[str],
    
    181
    +    force_download: bool,
    
    182
    +    get_content: Callable[[], bytes],
    
    183
    +) -> bool:
    
    184
    +    if not force_download and hex_digest is not None and os.path.exists(install_path):
    
    185
    +        with open(install_path, "rb") as f:
    
    186
    +            content: bytes = f.read()
    
    187
    +        if bytes.fromhex(hex_digest) == digest_data(content):
    
    188
    +            return False
    
    189
    +
    
    190
    +    content = get_content()
    
    191
    +    install_file(content, install_path)
    
    192
    +    return True
    
    193
    +
    
    194
    +
    
    195
    +def download_and_install_item(
    
    196
    +    item: dict, install_dir: str, force_download: bool
    
    197
    +) -> List[Tuple[str, bool]]:
    
    198
    +    """Download and install one item.
    
    199
    +
    
    200
    +    Args:
    
    201
    +      item: Download item as a dictionary, see above for schema.
    
    202
    +      install_dir: Installation directory.
    
    203
    +      force_download: Set to True to force download and installation, even if
    
    204
    +        the font file is already installed with the right content.
    
    205
    +
    
    206
    +    Returns:
    
    207
    +      A list of (install_name, status) tuples, where 'install_name' is the file's
    
    208
    +      installation name under 'install_dir', and 'status' is a boolean that is True
    
    209
    +      to indicate that the file was downloaded and installed, or False to indicate that
    
    210
    +      the file is already installed with the right content.
    
    211
    +    """
    
    212
    +    if "file_url" in item:
    
    213
    +        file_url = item["file_url"]
    
    214
    +        install_name = item.get("install_name", os.path.basename(file_url))
    
    215
    +        install_path = os.path.join(install_dir, install_name)
    
    216
    +        hex_digest = item.get("hex_digest")
    
    217
    +
    
    218
    +        def get_content():
    
    219
    +            return download_file(file_url, hex_digest)
    
    220
    +
    
    221
    +        status = _get_and_install_file(
    
    222
    +            install_path, hex_digest, force_download, get_content
    
    223
    +        )
    
    224
    +        return [(install_name, status)]
    
    225
    +
    
    226
    +    if "zip_url" in item:
    
    227
    +        # One or more files from a zip archive.
    
    228
    +        archive_url = item["zip_url"]
    
    229
    +        archive = zipfile.ZipFile(io.BytesIO(download_file(archive_url)))
    
    230
    +
    
    231
    +        result = []
    
    232
    +        for f in item["zip_files"]:
    
    233
    +            filename = f["filename"]
    
    234
    +            install_name = f.get("install_name", filename)
    
    235
    +            hex_digest = f.get("hex_digest")
    
    236
    +
    
    237
    +            def get_content():
    
    238
    +                return extract_file_from_zip_archive(
    
    239
    +                    archive,
    
    240
    +                    archive_url,
    
    241
    +                    filename,
    
    242
    +                    bytes.fromhex(hex_digest) if hex_digest else None,
    
    243
    +                )
    
    244
    +
    
    245
    +            status = _get_and_install_file(
    
    246
    +                os.path.join(install_dir, install_name),
    
    247
    +                hex_digest,
    
    248
    +                force_download,
    
    249
    +                get_content,
    
    250
    +            )
    
    251
    +            result.append((install_name, status))
    
    252
    +
    
    253
    +        return result
    
    254
    +
    
    255
    +    else:
    
    256
    +        raise ValueError("Unknown download item schema: %s" % item)
    
    257
    +
    
    258
    +
    
    259
    +def main():
    
    260
    +    parser = argparse.ArgumentParser(description=__doc__)
    
    261
    +
    
    262
    +    # Assume this script is under tests/scripts/ and tests/data/
    
    263
    +    # is the default installation directory.
    
    264
    +    install_dir = os.path.normpath(
    
    265
    +        os.path.join(os.path.dirname(__file__), "..", "data")
    
    266
    +    )
    
    267
    +
    
    268
    +    parser.add_argument(
    
    269
    +        "--force",
    
    270
    +        action="store_true",
    
    271
    +        default=False,
    
    272
    +        help="Force download and installation of font files",
    
    273
    +    )
    
    274
    +
    
    275
    +    parser.add_argument(
    
    276
    +        "--install-dir",
    
    277
    +        default=install_dir,
    
    278
    +        help="Specify installation directory [%s]" % install_dir,
    
    279
    +    )
    
    280
    +
    
    281
    +    args = parser.parse_args()
    
    282
    +
    
    283
    +    for item in _DOWNLOAD_ITEMS:
    
    284
    +        for install_name, status in download_and_install_item(
    
    285
    +            item, args.install_dir, args.force
    
    286
    +        ):
    
    287
    +            print("%s %s" % (install_name, "INSTALLED" if status else "UP-TO-DATE"))
    
    288
    +
    
    289
    +    return 0
    
    290
    +
    
    291
    +
    
    292
    +if __name__ == "__main__":
    
    293
    +    sys.exit(main())

  • tests/scripts/download-test-fonts.sh deleted
    1
    -#!/usr/bin/bash
    
    2
    -# Download test fonts used by the FreeType regression test programs.
    
    3
    -# These will be copied to $FREETYPE/tests/data/
    
    4
    -# Each font file contains an 8-hexchar prefix corresponding to its md5sum
    
    5
    -
    
    6
    -set -e
    
    7
    -
    
    8
    -export LANG=C
    
    9
    -export LC_ALL=C
    
    10
    -
    
    11
    -PROGDIR=$(dirname "$0")
    
    12
    -PROGNAME=$(basename "$0")
    
    13
    -
    
    14
    -# Download a file from a given URL
    
    15
    -#
    
    16
    -# $1: URL
    
    17
    -# $2: Destination directory
    
    18
    -# $3: If not empty, destination file name. Default is to take
    
    19
    -# the URL's basename.
    
    20
    -#
    
    21
    -download_file () {
    
    22
    -  local URL=$1
    
    23
    -  local DST_DIR=$2
    
    24
    -  local DST_FILE=$3
    
    25
    -  if [[ -z "$DST_FILE" ]]; then
    
    26
    -    DST_FILE=$(basename "$URL")
    
    27
    -  fi
    
    28
    -  echo "URL: $URL"
    
    29
    -  wget -q -O "$DST_DIR/$DST_FILE" "$URL"
    
    30
    -}
    
    31
    -
    
    32
    -# $1: URL
    
    33
    -# $2: Destination directory
    
    34
    -# $3+: Optional file list, otherwise the full archive is extracted to $2
    
    35
    -download_and_extract_zip () {
    
    36
    -  local URL=$1
    
    37
    -  local DST_DIR=$2
    
    38
    -  shift
    
    39
    -  shift
    
    40
    -  TEMP_DST_DIR=$(mktemp -d)
    
    41
    -  TEMP_DST_NAME="a.zip"
    
    42
    -  download_file "$URL" "$TEMP_DST_DIR" "$TEMP_DST_NAME"
    
    43
    -  unzip -qo "$TEMP_DST_DIR/$TEMP_DST_NAME" -d "$DST_DIR" "$@"
    
    44
    -  rm -rf "$TEMP_DST_DIR"
    
    45
    -}
    
    46
    -
    
    47
    -# $1: File path
    
    48
    -# $2: Expected md5sum
    
    49
    -md5sum_check () {
    
    50
    -  local FILE=$1
    
    51
    -  local EXPECTED=$2
    
    52
    -  local HASH=$(md5sum "$FILE" | cut -d" " -f1)
    
    53
    -  if [[ "$EXPECTED" != "$HASH" ]]; then
    
    54
    -    echo "$FILE: Invalid md5sum $HASH expected $EXPECTED"
    
    55
    -    return 1
    
    56
    -  fi
    
    57
    -}
    
    58
    -
    
    59
    -INSTALL_DIR=$(cd $PROGDIR/.. && pwd)/data
    
    60
    -
    
    61
    -mkdir -p "$INSTALL_DIR"
    
    62
    -
    
    63
    -# See https://gitlab.freedesktop.org/freetype/freetype/-/issues/1063
    
    64
    -download_and_extract_zip "https://github.com/python-pillow/Pillow/files/6622147/As.I.Lay.Dying.zip" "$INSTALL_DIR"
    
    65
    -mv "$INSTALL_DIR/As I Lay Dying.ttf" "$INSTALL_DIR/As.I.Lay.Dying.ttf"
    
    66
    -md5sum_check "$INSTALL_DIR/As.I.Lay.Dying.ttf" e153d60e66199660f7cfe99ef4705ad7


  • reply via email to

    [Prev in Thread] Current Thread [Next in Thread]