[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[gnuastro-commits] master c4d8e33: Statistics can read from Standard inp
|
From: |
Mohammad Akhlaghi |
|
Subject: |
[gnuastro-commits] master c4d8e33: Statistics can read from Standard input |
|
Date: |
Fri, 2 Nov 2018 19:56:22 -0400 (EDT) |
branch: master
commit c4d8e3357d4fd61c3befd9e83908b7faa9925df9
Author: Mohammad Akhlaghi <address@hidden>
Commit: Mohammad Akhlaghi <address@hidden>
Statistics can read from Standard input
Until now, Gnuastro's programs would only take their input from a
file. However, atleast for plain text files, it is very annoying (and prone
to bugs) to be forced to create a new temporary file and break a series of
pipes.
One of the most common cases was the Statistics program: when you make a
column with something like `awk' and want to want to do some basic
statistical measurements on it. So to build the necessary infra-structure
(and use in all the programs), the Statistics program was chosen to start
with.
With this commit, an initial implementation of this system has been added
to Statistics, it will later be extended to any other program that takes
plain text files as input.
---
NEWS | 12 ++
bin/arithmetic/arithmetic.c | 2 +-
bin/arithmetic/operands.c | 2 +-
bin/arithmetic/ui.c | 1 +
bin/buildprog/ui.c | 1 +
bin/convertt/ui.c | 3 +-
bin/convolve/ui.c | 5 +-
bin/cosmiccal/ui.c | 1 +
bin/crop/ui.c | 6 +-
bin/fits/ui.c | 1 +
bin/gnuastro.conf | 1 +
bin/match/match.c | 4 +-
bin/match/ui.c | 5 +-
bin/mkcatalog/ui.c | 13 +-
bin/mknoise/ui.c | 3 +-
bin/mkprof/ui.c | 5 +-
bin/noisechisel/ui.c | 5 +-
bin/segment/ui.c | 11 +-
bin/statistics/ui.c | 60 ++++--
bin/table/ui.c | 8 +-
bin/warp/ui.c | 3 +-
doc/gnuastro.texi | 224 +++++++++++++++++------
lib/array.c | 22 ++-
lib/gnuastro-internal/commonopts.h | 15 +-
lib/gnuastro-internal/options.h | 7 +-
lib/gnuastro/array.h | 12 +-
lib/gnuastro/table.h | 10 +-
lib/gnuastro/txt.h | 17 +-
lib/options.c | 26 +++
lib/table.c | 19 +-
lib/txt.c | 364 ++++++++++++++++++++++++++-----------
tests/Makefile.am | 6 +-
tests/statistics/from-stdin.sh | 56 ++++++
33 files changed, 681 insertions(+), 249 deletions(-)
diff --git a/NEWS b/NEWS
index 0a8b68b..f097ddb 100644
--- a/NEWS
+++ b/NEWS
@@ -58,6 +58,7 @@ GNU Astronomy Utilities NEWS -*-
outline -*-
column.
- Sky estimation: new outlier estimation algorithm similar to
NoiseChisel.
+ - Input can be given using the standard input (for example a pipe).
Library:
- gal_blank_flag_apply: Set all flagged/masked elements to blank.
@@ -67,6 +68,7 @@ GNU Astronomy Utilities NEWS -*-
outline -*-
- gal_fits_key_write_version_in_ptr: old gal_fits_key_write_version.
- gal_fits_key_write_config: write key list and version as config header.
- gal_statistics_outlier_positive: Find the first positive outlier.
+ - gal_txt_stdin_read: Read lines in standard input into a list of strings.
** Removed features
@@ -105,6 +107,16 @@ GNU Astronomy Utilities NEWS -*-
outline -*-
- gal_fits_key_write: filename and hdu instead of FITS pointer.
- gal_fits_key_write_version: filename and hdu instead of FITS pointer.
- gal_fits_key_write_filename: write at the top or end of the input list.
+ - gal_array_read: list of strings (from standard input) acceptable.
+ - gal_array_read_to_type: list of strings (from stin) acceptable.
+ - gal_array_read_one_ch: list of strings (from stdin) acceptable.
+ - gal_array_read_one_ch_to_type: list of strings (from stdin) acceptable.
+ - gal_table_info: list of strings (from stdin) acceptable.
+ - gal_table_read: list of strings (from stdin) acceptable.
+ - gal_txt_table_info: list of strings (from stdin) acceptable.
+ - gal_txt_image_info: list of strings (from stdin) acceptable.
+ - gal_txt_table_read: list of strings (from stdin) acceptable.
+ - gal_txt_image_read: list of strings (from stdin) acceptable.
** Bugs fixed
bug #54493: Warp crashes when type isn't set.
diff --git a/bin/arithmetic/arithmetic.c b/bin/arithmetic/arithmetic.c
index 8afcb7b..d747e85 100644
--- a/bin/arithmetic/arithmetic.c
+++ b/bin/arithmetic/arithmetic.c
@@ -1139,7 +1139,7 @@ reversepolish(struct arithmeticparams *p)
filename=p->operands->filename;
if( gal_fits_name_is_fits(filename) )
{
- p->operands->data=gal_array_read_one_ch(filename, hdu,
+ p->operands->data=gal_array_read_one_ch(filename, hdu, NULL,
p->cp.minmapsize);
p->refdata.wcs=gal_wcs_read(filename, hdu, 0, 0, &p->refdata.nwcs);
if(!p->cp.quiet) printf(" - %s (hdu %s) is read.\n", filename, hdu);
diff --git a/bin/arithmetic/operands.c b/bin/arithmetic/operands.c
index 400a25d..2a00a29 100644
--- a/bin/arithmetic/operands.c
+++ b/bin/arithmetic/operands.c
@@ -335,7 +335,7 @@ operands_pop(struct arithmeticparams *p, char *operator)
filename=operands->filename;
/* Read the dataset. */
- data=gal_array_read_one_ch(filename, hdu, p->cp.minmapsize);
+ data=gal_array_read_one_ch(filename, hdu, NULL, p->cp.minmapsize);
/* Arithmetic changes the contents of a dataset, so the existing name
(in the FITS `EXTNAME' keyword) should not be passed on beyond
diff --git a/bin/arithmetic/ui.c b/bin/arithmetic/ui.c
index 9ca9e6e..61075cc 100644
--- a/bin/arithmetic/ui.c
+++ b/bin/arithmetic/ui.c
@@ -140,6 +140,7 @@ ui_initialize_options(struct arithmeticparams *p,
case GAL_OPTIONS_KEY_TYPE:
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_IGNORECASE:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
diff --git a/bin/buildprog/ui.c b/bin/buildprog/ui.c
index 46aef6e..8667aa5 100644
--- a/bin/buildprog/ui.c
+++ b/bin/buildprog/ui.c
@@ -125,6 +125,7 @@ ui_initialize_options(struct buildprogparams *p,
case GAL_OPTIONS_KEY_NUMTHREADS:
case GAL_OPTIONS_KEY_MINMAPSIZE:
case GAL_OPTIONS_KEY_TABLEFORMAT:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
cp->coptions[i].mandatory=GAL_OPTIONS_NOT_MANDATORY;
break;
diff --git a/bin/convertt/ui.c b/bin/convertt/ui.c
index 0df8390..97137ae 100644
--- a/bin/convertt/ui.c
+++ b/bin/convertt/ui.c
@@ -146,6 +146,7 @@ ui_initialize_options(struct converttparams *p,
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_IGNORECASE:
case GAL_OPTIONS_KEY_TABLEFORMAT:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
}
@@ -492,7 +493,7 @@ ui_make_channels_ll(struct converttparams *p)
/* Text: */
else
{
- data=gal_txt_image_read(name->v, p->cp.minmapsize);
+ data=gal_txt_image_read(name->v, NULL, p->cp.minmapsize);
gal_list_data_add(&p->chll, data);
++p->numch;
}
diff --git a/bin/convolve/ui.c b/bin/convolve/ui.c
index d2b3cf1..f8ed0e6 100644
--- a/bin/convolve/ui.c
+++ b/bin/convolve/ui.c
@@ -132,6 +132,7 @@ ui_initialize_options(struct convolveparams *p,
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_IGNORECASE:
case GAL_OPTIONS_KEY_TABLEFORMAT:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
case GAL_OPTIONS_KEY_INTERPNUMNGB:
case GAL_OPTIONS_KEY_INTERPONLYBLANK:
cp->coptions[i].flags=OPTION_HIDDEN;
@@ -296,7 +297,7 @@ ui_read_kernel(struct convolveparams *p)
/* Read the image into file. */
p->kernel = gal_array_read_one_ch_to_type(p->kernelname, p->khdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
/* Convert all the NaN pixels to zero if the kernel contains blank
@@ -343,7 +344,7 @@ ui_preparations(struct convolveparams *p)
/* Read the input image as a float64 array. */
- p->input=gal_array_read_one_ch_to_type(p->filename, cp->hdu,
+ p->input=gal_array_read_one_ch_to_type(p->filename, cp->hdu, NULL,
GAL_TYPE_FLOAT32, cp->minmapsize);
p->input->wcs=gal_wcs_read(p->filename, cp->hdu, 0, 0, &p->input->nwcs);
diff --git a/bin/cosmiccal/ui.c b/bin/cosmiccal/ui.c
index f792d5d..6b60557 100644
--- a/bin/cosmiccal/ui.c
+++ b/bin/cosmiccal/ui.c
@@ -138,6 +138,7 @@ ui_initialize_options(struct cosmiccalparams *p,
case GAL_OPTIONS_KEY_MINMAPSIZE:
case GAL_OPTIONS_KEY_IGNORECASE:
case GAL_OPTIONS_KEY_TABLEFORMAT:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
}
diff --git a/bin/crop/ui.c b/bin/crop/ui.c
index f7afc41..b4495dd 100644
--- a/bin/crop/ui.c
+++ b/bin/crop/ui.c
@@ -145,6 +145,10 @@ ui_initialize_options(struct cropparams *p,
case GAL_OPTIONS_KEY_IGNORECASE:
cp->coptions[i].group=UI_GROUP_CENTER_CATALOG;
break;
+
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
+ cp->coptions[i].group=OPTION_HIDDEN;
+ break;
}
/* Select by group. */
@@ -671,7 +675,7 @@ ui_read_cols(struct cropparams *p)
/* Read the desired columns from the file. */
- cols=gal_table_read(p->catname, p->cathdu, colstrs, p->cp.searchin,
+ cols=gal_table_read(p->catname, p->cathdu, NULL, colstrs, p->cp.searchin,
p->cp.ignorecase, p->cp.minmapsize, NULL);
if(cols==NULL)
error(EXIT_FAILURE, 0, "%s: is empty! No usable information "
diff --git a/bin/fits/ui.c b/bin/fits/ui.c
index 6482d17..7b5ef33 100644
--- a/bin/fits/ui.c
+++ b/bin/fits/ui.c
@@ -123,6 +123,7 @@ ui_initialize_options(struct fitsparams *p,
case GAL_OPTIONS_KEY_LOG:
case GAL_OPTIONS_KEY_MINMAPSIZE:
case GAL_OPTIONS_KEY_NUMTHREADS:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
diff --git a/bin/gnuastro.conf b/bin/gnuastro.conf
index c9a6d42..3652e6a 100644
--- a/bin/gnuastro.conf
+++ b/bin/gnuastro.conf
@@ -21,6 +21,7 @@
hdu 1
ignorecase 1
searchin name
+ stdintimeout 100000
# Tessellation
tilesize 30,30
diff --git a/bin/match/match.c b/bin/match/match.c
index 011de6c..c1461f7 100644
--- a/bin/match/match.c
+++ b/bin/match/match.c
@@ -66,8 +66,8 @@ match_catalog_read_write_all(struct matchparams *p, size_t
*permutation,
__func__, "numcolmatch");
/* Read the full table. */
- cat=gal_table_read(filename, hdu, cols, p->cp.searchin, p->cp.ignorecase,
- p->cp.minmapsize, *numcolmatch);
+ cat=gal_table_read(filename, hdu, NULL, cols, p->cp.searchin,
+ p->cp.ignorecase, p->cp.minmapsize, *numcolmatch);
origsize=cat->size;
diff --git a/bin/match/ui.c b/bin/match/ui.c
index a6dbcd6..1d2ad5b 100644
--- a/bin/match/ui.c
+++ b/bin/match/ui.c
@@ -119,6 +119,7 @@ ui_initialize_options(struct matchparams *p,
break;
case GAL_OPTIONS_KEY_TYPE:
case GAL_OPTIONS_KEY_NUMTHREADS:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
}
@@ -404,8 +405,8 @@ ui_read_columns_to_double(struct matchparams *p, char
*filename, char *hdu,
"numberes are the only identifiers guaranteed to be unique).";
/* Read the columns. */
- tout=gal_table_read(filename, hdu, cols, cp->searchin, cp->ignorecase,
- cp->minmapsize, NULL);
+ tout=gal_table_read(filename, hdu, NULL, cols, cp->searchin,
+ cp->ignorecase, cp->minmapsize, NULL);
/* A small sanity check. */
if(gal_list_data_number(tout)!=numcols)
diff --git a/bin/mkcatalog/ui.c b/bin/mkcatalog/ui.c
index 64c5cf0..f2de924 100644
--- a/bin/mkcatalog/ui.c
+++ b/bin/mkcatalog/ui.c
@@ -142,6 +142,7 @@ ui_initialize_options(struct mkcatalogparams *p,
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_IGNORECASE:
case GAL_OPTIONS_KEY_WORKOVERCH:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
case GAL_OPTIONS_KEY_INTERPNUMNGB:
case GAL_OPTIONS_KEY_INTERPONLYBLANK:
cp->coptions[i].flags=OPTION_HIDDEN;
@@ -540,7 +541,7 @@ ui_read_labels(struct mkcatalogparams *p)
gal_data_t *tmp, *keys=gal_data_array_calloc(2);
/* Read it into memory. */
- p->objects = gal_array_read_one_ch(p->objectsfile, p->cp.hdu,
+ p->objects = gal_array_read_one_ch(p->objectsfile, p->cp.hdu, NULL,
p->cp.minmapsize);
@@ -603,7 +604,7 @@ ui_read_labels(struct mkcatalogparams *p)
/* Read the clumps image. */
p->clumps = gal_array_read_one_ch(p->usedclumpsfile, p->clumpshdu,
- p->cp.minmapsize);
+ NULL, p->cp.minmapsize);
/* Check its size. */
if( gal_dimension_is_different(p->objects, p->clumps) )
@@ -908,7 +909,7 @@ ui_preparations_read_inputs(struct mkcatalogparams *p)
/* Read the values dataset. */
p->values=gal_array_read_one_ch_to_type(p->usedvaluesfile, p->valueshdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
/* Make sure it has the correct size. */
@@ -955,7 +956,7 @@ ui_preparations_read_inputs(struct mkcatalogparams *p)
/* Read the Sky dataset. */
p->sky=gal_array_read_one_ch_to_type(p->usedskyfile, p->skyhdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
/* Check its size and prepare tile structure. */
@@ -984,7 +985,7 @@ ui_preparations_read_inputs(struct mkcatalogparams *p)
/* Read the Sky standard deviation image into memory. */
p->std=gal_array_read_one_ch_to_type(p->usedstdfile, p->stdhdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
/* Check its size and prepare tile structure. */
@@ -1018,7 +1019,7 @@ ui_preparations_read_inputs(struct mkcatalogparams *p)
/* Read the mask image. */
p->upmask = gal_array_read_one_ch(p->upmaskfile, p->upmaskhdu,
- p->cp.minmapsize);
+ NULL, p->cp.minmapsize);
/* Check its size. */
if( gal_dimension_is_different(p->objects, p->upmask) )
diff --git a/bin/mknoise/ui.c b/bin/mknoise/ui.c
index 471328d..5221db0 100644
--- a/bin/mknoise/ui.c
+++ b/bin/mknoise/ui.c
@@ -131,6 +131,7 @@ ui_initialize_options(struct mknoiseparams *p,
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_TABLEFORMAT:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
}
@@ -282,7 +283,7 @@ void
ui_preparations(struct mknoiseparams *p)
{
/* Read the input image as a double type */
- p->input=gal_array_read_one_ch_to_type(p->inputname, p->cp.hdu,
+ p->input=gal_array_read_one_ch_to_type(p->inputname, p->cp.hdu, NULL,
GAL_TYPE_FLOAT64, p->cp.minmapsize);
p->input->wcs=gal_wcs_read(p->inputname, p->cp.hdu, 0, 0, &p->input->nwcs);
diff --git a/bin/mkprof/ui.c b/bin/mkprof/ui.c
index 8ad4520..159f21d 100644
--- a/bin/mkprof/ui.c
+++ b/bin/mkprof/ui.c
@@ -202,6 +202,7 @@ ui_initialize_options(struct mkprofparams *p,
break;
case GAL_OPTIONS_KEY_TABLEFORMAT:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
@@ -621,7 +622,7 @@ ui_read_cols(struct mkprofparams *p)
gal_list_str_reverse(&colstrs);
/* Read the desired columns from the file. */
- cols=gal_table_read(p->catname, p->cp.hdu, colstrs, p->cp.searchin,
+ cols=gal_table_read(p->catname, p->cp.hdu, NULL, colstrs, p->cp.searchin,
p->cp.ignorecase, p->cp.minmapsize, NULL);
/* Set the number of objects. */
@@ -1036,7 +1037,7 @@ ui_prepare_canvas(struct mkprofparams *p)
{
/* Read the image. */
p->out=gal_array_read_one_ch_to_type(p->backname, p->backhdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
p->out->wcs=gal_wcs_read(p->backname, p->backhdu, 0, 0,
&p->out->nwcs);
diff --git a/bin/noisechisel/ui.c b/bin/noisechisel/ui.c
index 2ce2c70..0719b63 100644
--- a/bin/noisechisel/ui.c
+++ b/bin/noisechisel/ui.c
@@ -126,6 +126,7 @@ ui_initialize_options(struct noisechiselparams *p,
case GAL_OPTIONS_KEY_TYPE:
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_IGNORECASE:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
@@ -543,7 +544,7 @@ ui_preparations_read_input(struct noisechiselparams *p)
/* Read the input as a single precision floating point dataset. */
p->input = gal_array_read_one_ch_to_type(p->inputname, p->cp.hdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
p->input->wcs = gal_wcs_read(p->inputname, p->cp.hdu, 0, 0,
&p->input->nwcs);
@@ -601,7 +602,7 @@ ui_preparations(struct noisechiselparams *p)
{
/* Read the input convolved image. */
p->conv = gal_array_read_one_ch_to_type(p->convolvedname, p->chdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
/* Make sure the convolved image is the same size as the input. */
diff --git a/bin/segment/ui.c b/bin/segment/ui.c
index 707b396..02cf063 100644
--- a/bin/segment/ui.c
+++ b/bin/segment/ui.c
@@ -131,6 +131,7 @@ ui_initialize_options(struct segmentparams *p,
case GAL_OPTIONS_KEY_TYPE:
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_IGNORECASE:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
@@ -404,7 +405,7 @@ ui_prepare_inputs(struct segmentparams *p)
/* Read the input as a single precision floating point dataset. */
p->input = gal_array_read_one_ch_to_type(p->inputname, p->cp.hdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
p->input->wcs = gal_wcs_read(p->inputname, p->cp.hdu, 0, 0,
&p->input->nwcs);
@@ -428,7 +429,7 @@ ui_prepare_inputs(struct segmentparams *p)
{
/* Read the input convolved image. */
p->conv = gal_array_read_one_ch_to_type(p->convolvedname, p->chdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
p->conv->wcs=gal_wcs_copy(p->input->wcs);
@@ -447,7 +448,7 @@ ui_prepare_inputs(struct segmentparams *p)
{
/* Read the dataset into memory. */
p->olabel = gal_array_read_one_ch(p->useddetectionname, p->dhdu,
- p->cp.minmapsize);
+ NULL, p->cp.minmapsize);
if( gal_dimension_is_different(p->input, p->olabel) )
error(EXIT_FAILURE, 0, "`%s' (hdu: %s) and `%s' (hdu: %s) have a"
"different dimension/size", p->useddetectionname, p->dhdu,
@@ -722,7 +723,7 @@ ui_read_std_and_sky(struct segmentparams *p)
/* Read the STD image. */
p->std=gal_array_read_one_ch_to_type(p->usedstdname, p->stdhdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
/* Make sure it has the correct size. */
@@ -780,7 +781,7 @@ ui_read_std_and_sky(struct segmentparams *p)
/* Read the Sky dataset. */
sky=gal_array_read_one_ch_to_type(p->skyname, p->skyhdu,
- GAL_TYPE_FLOAT32,
+ NULL, GAL_TYPE_FLOAT32,
p->cp.minmapsize);
/* Check its size. */
diff --git a/bin/statistics/ui.c b/bin/statistics/ui.c
index 21f43b4..3cedec1 100644
--- a/bin/statistics/ui.c
+++ b/bin/statistics/ui.c
@@ -27,6 +27,7 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
#include <error.h>
#include <stdio.h>
+#include <gnuastro/txt.h>
#include <gnuastro/wcs.h>
#include <gnuastro/fits.h>
#include <gnuastro/tile.h>
@@ -530,8 +531,6 @@ ui_check_options_and_arguments(struct statisticsparams *p)
"to tables.", p->inputname, p->cp.hdu);
}
}
- else
- error(EXIT_FAILURE, 0, "no input file is specified");
}
@@ -701,6 +700,8 @@ ui_read_columns(struct statisticsparams *p)
gal_list_str_t *column=NULL;
gal_data_t *cols, *tmp, *cinfo;
size_t size, ncols, nrows, counter=0;
+ gal_list_str_t *lines=gal_options_check_stdin(p->inputname,
+ p->cp.stdintimeout);
/* If a reference column is also given, add it to the list of columns to
read. */
@@ -713,32 +714,50 @@ ui_read_columns(struct statisticsparams *p)
if(p->column==NULL)
{
/* Get the basic table information. */
- cinfo=gal_table_info(p->inputname, p->cp.hdu, &ncols, &nrows, &tformat);
+ cinfo=gal_table_info(p->inputname, p->cp.hdu, lines, &ncols, &nrows,
+ &tformat);
gal_data_array_free(cinfo, ncols, 1);
/* See how many columns it has and take the proper action. */
- if(ncols==1)
- gal_checkset_allocate_copy("1", &p->column);
- else
- error(EXIT_FAILURE, 0, "%s is a table containing more than one "
- "column. However, the specific column to work on isn't "
- "specified.\n\n"
- "Please use the `--column' (`-c') option to specify a "
- "column. You can either give it the column number "
- "(couting from 1), or a match/search in its meta-data (e.g., "
- "column names).\n\n"
- "For more information, please run the following command "
- "(press the `SPACE' key to go down and `q' to return to the "
- "command-line):\n\n"
- " $ info gnuastro \"Selecting table columns\"\n",
- gal_checkset_dataset_name(p->inputname, p->cp.hdu));
+ switch(ncols)
+ {
+ case 0:
+ error(EXIT_FAILURE, 0, "%s contains no usable information",
+ ( p->inputname
+ ? gal_checkset_dataset_name(p->inputname, p->cp.hdu)
+ : "Standard input" ));
+ case 1:
+ gal_checkset_allocate_copy("1", &p->column);
+ break;
+ default:
+ error(EXIT_FAILURE, 0, "%s is a table containing more than one "
+ "column. However, the specific column to work on isn't "
+ "specified.\n\n"
+ "Please use the `--column' (`-c') option to specify a "
+ "column. You can either give it the column number "
+ "(couting from 1), or a match/search in its meta-data (e.g., "
+ "column names).\n\n"
+ "For more information, please run the following command "
+ "(press the `SPACE' key to go down and `q' to return to the "
+ "command-line):\n\n"
+ " $ info gnuastro \"Selecting table columns\"\n",
+ ( p->inputname
+ ? gal_checkset_dataset_name(p->inputname, p->cp.hdu)
+ : "Standard input" ));
+ }
}
gal_list_str_add(&column, p->column, 0);
/* Read the desired column(s). */
- cols=gal_table_read(p->inputname, p->cp.hdu, column, p->cp.searchin,
+ cols=gal_table_read(p->inputname, p->cp.hdu, lines, column, p->cp.searchin,
p->cp.ignorecase, p->cp.minmapsize, NULL);
+ gal_list_str_free(lines, 1);
+
+ /* If the input was from standard input, we can actually write this into
+ it (for future reporting). */
+ if(p->inputname==NULL)
+ gal_checkset_allocate_copy("Standard input", &p->inputname);
/* Put the columns into the proper gal_data_t. */
size=cols->size;
@@ -807,7 +826,8 @@ ui_preparations(struct statisticsparams *p)
if(p->isfits && p->hdu_type==IMAGE_HDU)
{
p->inputformat=INPUT_FORMAT_IMAGE;
- p->input=gal_array_read_one_ch(p->inputname, cp->hdu, cp->minmapsize);
+ p->input=gal_array_read_one_ch(p->inputname, cp->hdu, NULL,
+ cp->minmapsize);
p->input->wcs=gal_wcs_read(p->inputname, cp->hdu, 0, 0,
&p->input->nwcs);
}
diff --git a/bin/table/ui.c b/bin/table/ui.c
index 6a77393..cf219f1 100644
--- a/bin/table/ui.c
+++ b/bin/table/ui.c
@@ -129,6 +129,7 @@ ui_initialize_options(struct tableparams *p,
/* Options to ignore. */
case GAL_OPTIONS_KEY_TYPE:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
}
@@ -277,7 +278,7 @@ ui_print_info_exit(struct tableparams *p)
size_t i, numcols, numrows;
/* Read the table information for the number of columns and rows. */
- allcols=gal_table_info(p->filename, p->cp.hdu, &numcols,
+ allcols=gal_table_info(p->filename, p->cp.hdu, NULL, &numcols,
&numrows, &tableformat);
/* If there was no actual data in the file, then inform the user */
@@ -371,8 +372,9 @@ ui_preparations(struct tableparams *p)
ui_columns_prepare(p);
/* Read in the table columns. */
- p->table=gal_table_read(p->filename, cp->hdu, p->columns, cp->searchin,
- cp->ignorecase, cp->minmapsize, NULL);
+ p->table=gal_table_read(p->filename, cp->hdu, NULL, p->columns,
+ cp->searchin, cp->ignorecase, cp->minmapsize,
+ NULL);
/* If there was no actual data in the file, then inform the user and
abort. */
diff --git a/bin/warp/ui.c b/bin/warp/ui.c
index d767db8..c01c3a6 100644
--- a/bin/warp/ui.c
+++ b/bin/warp/ui.c
@@ -128,6 +128,7 @@ ui_initialize_options(struct warpparams *p,
case GAL_OPTIONS_KEY_SEARCHIN:
case GAL_OPTIONS_KEY_TABLEFORMAT:
+ case GAL_OPTIONS_KEY_STDINTIMEOUT:
cp->coptions[i].flags=OPTION_HIDDEN;
break;
}
@@ -340,7 +341,7 @@ ui_check_options_and_arguments(struct warpparams *p)
/* Read the input image as double type and its WCS structure. */
p->input=gal_array_read_one_ch_to_type(p->inputname, p->cp.hdu,
- GAL_TYPE_FLOAT64,
+ NULL, GAL_TYPE_FLOAT64,
p->cp.minmapsize);
p->input->wcs=gal_wcs_read(p->inputname, p->cp.hdu, p->hstartwcs,
p->hendwcs, &p->input->nwcs);
diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index a07dacb..b235d05 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -7493,6 +7493,31 @@ programs.
@vtable @option
address@hidden Timeout
address@hidden Standard input
address@hidden --stdintimeout
+Number of micro-seconds to wait for writing/typing in the @emph{first line}
+of standard input from the command-line. This is only relevant for programs
+that also accept input from the standard input, @emph{and} you want to
+manually write/type the contents on the terminal. When the standard input
+is already connected to a pipe (output of another program), there won't be
+any waiting (hence no timeout, thus making this option redundant).
+
+If the first line-break (for example with the @key{ENTER} key) is not
+provided before the timeout, the program will abort with an error that no
+input was given. Note that this time interval is @emph{only} for the first
+line that you type. Once the first line is given, the program will assume
+that more data will come and accept rest of your inputs without any time
+limit. You need to specify the ending of the standard input, for example by
+pressing @key{CTRL-D} after a new line.
+
+Note that any input you write/type into a program on the command-line with
+Standard input will be discarded (lost) once the program is finished. It is
+only recoverable manually from your command-line (where you actually typed)
+as long as the terminal is open. So only use this feature when you are sure
+that you don't need the dataset (or have a copy of it somewhere else).
+
+
@cindex HDU
@cindex Header data unit
@item -h STR/INT
@@ -15091,18 +15116,25 @@ $ aststatistics cat.fits -cMAG_F160W -g26 -l27
--sigmaclip=3,0.2
## Print the median value of all records in column MAG_F160W that
## have a value larger than 3 in column PHOTO_Z:
$ aststatistics tab.txt -rPHOTO_Z -g3 -cMAG_F160W --median
+
+## Calculate the median of the third column in the input table, but only
+## for rows where the mean of the first and second columns is >5.
+$ awk '($1+$2)/2 > 5 @{print address@hidden' table.txt | aststatistics --median
@end example
@noindent
-An input image or table is necessary when processing is to be done. If any
-output file is to be created, the value to the @option{--output} option, is
-used as the base name for the generated files. Without @option{--output},
-the input name will be used to generate an output name, see @ref{Automatic
-output}. The options described below are particular to Statistics, but for
-general operations, it shares a large collection of options with the other
-Gnuastro programs, see @ref{Common options} for the full list. Options can
-also be given in configuration files, for more, please see
address@hidden files}.
address@hidden Standard input
+Statistics can take its input dataset either from a file (image or table)
+or the Standard input. If any output file is to be created, the value to
+the @option{--output} option, is used as the base name for the generated
+files. Without @option{--output}, the input name will be used to generate
+an output name, see @ref{Automatic output}. The options described below are
+particular to Statistics, but for general operations, it shares a large
+collection of options with the other Gnuastro programs, see @ref{Common
+options} for the full list. For more on reading from standard input, please
+see the description of @code{--stdintimeout} option in @ref{Input output
+options}. Options can also be given in configuration files, for more,
+please see @ref{Configuration files}.
The input dataset may have blank values (see @ref{Blank pixels}), in this
case, all blank pixels are ignored during the calculation. Initially, the
@@ -24710,42 +24742,51 @@ types for reading arrays which may contain multiple
extensions (for example
FITS or TIFF) formats.
@end deftypefun
address@hidden gal_data_t gal_array_read (char @code{*filename}, char
@code{*extension}, size_t @code{minmapsize})
address@hidden gal_data_t gal_array_read (char @code{*filename}, char
@code{*extension}, gal_list_str_t @code{*lines}, size_t @code{minmapsize})
Read the array within the given extension (@code{extension}) of
address@hidden If the array is larger than @code{minmapsize} bytes, then
-it won't be read into RAM, but a file on the HDD/SSD (no difference for the
-programmer). @code{extension} will be ignored for files that don't support
-them (for example JPEG or text). For FITS files, @code{extension} can be a
-number or a string (name of the extension), but for TIFF files, it has to
-be number. In both cases, counting starts from zero.
address@hidden, or the @code{lines} list (see below). If the array is
+larger than @code{minmapsize} bytes, then it won't be read into RAM, but a
+file on the HDD/SSD (no difference for the programmer). @code{extension}
+will be ignored for files that don't support them (for example JPEG or
+text). For FITS files, @code{extension} can be a number or a string (name
+of the extension), but for TIFF files, it has to be number. In both cases,
+counting starts from zero.
For multi-channel formats (like RGB images in JPEG or TIFF), this function
will return a @ref{List of gal_data_t}: one data structure per
channel. Thus if you just want a single array (and want to check if the
user hasn't given a multi-channel input), you can check the @code{next}
pointer of the returned @code{gal_data_t}.
+
address@hidden is a list of strings with each node representing one line
+(including the new-line character), see @ref{List of strings}. It will
+mostly be the output of @code{gal_txt_stdin_read}, which is used to read
+the program's input as separate lines from the standard input (see
address@hidden files}). Note that @code{filename} and @code{lines} are mutually
+exclusive and one of them must be @code{NULL}.
@end deftypefun
address@hidden void gal_array_read_to_type (char @code{*filename}, char
@code{*extension}, uint8_t @code{type}, size_t @code{minmapsize})
address@hidden void gal_array_read_to_type (char @code{*filename}, char
@code{*extension}, gal_list_str_t @code{*lines}, uint8_t @code{type}, size_t
@code{minmapsize})
Similar to @code{gal_array_read}, but the output data structure(s) will
have a numeric data type of @code{type}, see @ref{Numeric data types}.
@end deftypefun
address@hidden void gal_array_read_one_ch (char @code{*filename}, char
@code{*extension}, size_t @code{minmapsize})
address@hidden void gal_array_read_one_ch (char @code{*filename}, char
@code{*extension}, gal_list_str_t @code{*lines}, size_t @code{minmapsize})
@cindex Channel
@cindex Color channel
Read the dataset within @code{filename} (extension/hdu/dir
address@hidden) and make sure it is only a single channel. Formats like
-JPEG or TIFF support multiple channels per input, but it may happen that
-your program only works on a single dataset. This function can be a
-convenient way to make sure that the data that comes into your program is
-only one channel. It is just a very simple wrapper around
address@hidden that checks if there was more than one dataset and
-aborts with an informative error if there is more than one channel in the
-dataset.
address@hidden) and make sure it is only a single channel. This is just a
+simple wrapper around @code{gal_array_read} that checks if there was more
+than one dataset and aborts with an informative error if there is more than
+one channel in the dataset.
+
+Formats like JPEG or TIFF support multiple channels per input, but it may
+happen that your program only works on a single dataset. This function can
+be a convenient way to make sure that the data that comes into your program
+is only one channel.
@end deftypefun
address@hidden void gal_array_read_one_ch_to_type (char @code{*filename}, char
@code{*extension}, uint8_t @code{type}, size_t @code{minmapsize})
address@hidden void gal_array_read_one_ch_to_type (char @code{*filename}, char
@code{*extension}, gal_list_str_t @code{*lines}, uint8_t @code{type}, size_t
@code{minmapsize})
Similar to @code{gal_array_read_one_ch}, but the output data structure will
has a numeric data type of @code{type}, see @ref{Numeric data types}.
@end deftypefun
@@ -24835,19 +24876,25 @@ string to match, or regular expression to search, be
in the @emph{name},
should be used for the @code{searchin} variables of the functions.
@end deffn
address@hidden {gal_data_t *} gal_table_info (char @code{*filename}, char
@code{*hdu}, size_t @code{*numcols}, size_t @code{*numrows}, int
@code{*tableformat})
-Store the information of each column in a table (either as a text file or
-as a FITS table) into an array of data structures with @code{numcols}
-structures (one data structure for each column). The number of rows is
-stored in the space that @code{numrows} points to. The format of the table
-(e.g., ascii text file, or FITS binary or ASCII table) will be put in
address@hidden (macros defined above). If the filename is not a FITS
-file, then @code{hdu} will not be used (can be @code{NULL}).
address@hidden {gal_data_t *} gal_table_info (char @code{*filename}, char
@code{*hdu}, gal_list_str_t @code{*lines}, size_t @code{*numcols}, size_t
@code{*numrows}, int @code{*tableformat})
+Store the information of each column in a table into an array of data
+structures with @code{numcols} datasets (one data structure for each
+column). The number of rows is stored in @code{numrows}. The format of the
+table (e.g., ascii text file, or FITS binary or ASCII table) will be put in
address@hidden (macros defined above). If the @code{filename} is not a
+FITS file, then @code{hdu} will not be used (can be @code{NULL}).
+
+The input must be either a file (specified by @code{filename}) or a list of
+strings (@code{lines}). @code{lines} is a list of strings with each node
+representing one line (including the new-line character), see @ref{List of
+strings}. It will mostly be the output of @code{gal_txt_stdin_read}, which
+is used to read the program's input as separate lines from the standard
+input (see @ref{Text files}). Note that @code{filename} and @code{lines}
+are mutually exclusive and one of them must be @code{NULL}.
-Note that other than the character strings (column name, units and
-comments), nothing in the data structure(s) will be allocated by this
-function for the actual data (e.g., the `array' or `dsize' elements). This
-function is just for column information (meta-data), not column contents.
+In the output datasets, only the meta-data strings (column name, units and
+comments), will be allocated and set. This function is just for column
+information (meta-data), not column contents.
@end deftypefun
@deftypefun void gal_table_print_info (gal_data_t @code{*allcols}, size_t
@code{numcols}, size_t @code{numrows})
@@ -24859,10 +24906,19 @@ $ asttable --info table.fits
@end example
@end deftypefun
address@hidden {gal_data_t *} gal_table_read (char @code{*filename}, char
@code{*hdu}, gal_list_str_t @code{*cols}, int @code{searchin}, int
@code{ignorecase}, size_t @code{minmapsize}, size_t @code{*colmatch})
-Read the specified columns in a text file (named @code{filename}) into a
-linked list of data structures. If the file is FITS, then @code{hdu} will
-also be used, otherwise, @code{hdu} is ignored.
address@hidden {gal_data_t *} gal_table_read (char @code{*filename}, char
@code{*hdu}, gal_list_str_t @code{*lines}, gal_list_str_t @code{*cols}, int
@code{searchin}, int @code{ignorecase}, size_t @code{minmapsize}, size_t
@code{*colmatch})
+
+Read the specified columns in a file (named @code{filename}), or list of
+strings (@code{lines}) into a linked list of data structures. If the file
+is FITS, then @code{hdu} will also be used, otherwise, @code{hdu} is
+ignored.
+
address@hidden is a list of strings with each node representing one line
+(including the new-line character), see @ref{List of strings}. It will
+mostly be the output of @code{gal_txt_stdin_read}, which is used to read
+the program's input as separate lines from the standard input (see
address@hidden files}). Note that @code{filename} and @code{lines} are mutually
+exclusive and one of them must be @code{NULL}.
@cindex AWK
@cindex GNU AWK
@@ -25631,11 +25687,19 @@ data line. The returned values are the macros that
start with
@code{GAL_TXT_LINESTAT}.
@end deftypefun
address@hidden {gal_data_t *} gal_txt_table_info (char @code{*filename}, size_t
@code{*numcols}, size_t @code{*numrows})
-Store the information of each column in @code{filename} into an array of
-data structures with @code{numcols} elements (one data structure for each
-column) see @ref{Arrays of datasets}. The total number of rows in the table
-is also put into the memory that @code{numrows} points to.
address@hidden {gal_data_t *} gal_txt_table_info (char @code{*filename},
gal_list_str_t @code{*lines}, size_t @code{*numcols}, size_t @code{*numrows})
+Store the information of each column in a text file @code{filename}, or
+list of strings (@code{lines}) into an array of data structures with
address@hidden elements (one data structure for each column) see
address@hidden of datasets}. The total number of rows in the table is also put
+into the memory that @code{numrows} points to.
+
address@hidden is a list of strings with each node representing one line
+(including the new-line character), see @ref{List of strings}. It will
+mostly be the output of @code{gal_txt_stdin_read}, which is used to read
+the program's input as separate lines from the standard input (see
+below). Note that @code{filename} and @code{lines} are mutually exclusive
+and one of them must be @code{NULL}.
This function is just for column information. Therefore it only stores
meta-data like column name, units and comments. No actual data (contents of
@@ -25647,13 +25711,20 @@ variety of table formats based on the filename (see
@ref{Table input
output}).
@end deftypefun
address@hidden {gal_data_t *} gal_txt_table_read (char @code{*filename}, size_t
@code{numrows}, gal_data_t @code{*colinfo}, gal_list_sizet_t @code{*indexll},
size_t @code{minmapsize})
-Read the columns given in the list @code{indexll} from a plain text table
-into a linked list of data structures, see @ref{List of size_t} and
address@hidden of gal_data_t}. If the necessary space for each column is larger
-than @code{minmapsize}, don't keep it in the RAM, but in a file on the
-HDD/SSD, see the description under the same name in @ref{Generic data
-container}.
address@hidden {gal_data_t *} gal_txt_table_read (char @code{*filename},
gal_list_str_t @code{*lines}, size_t @code{numrows}, gal_data_t
@code{*colinfo}, gal_list_sizet_t @code{*indexll}, size_t @code{minmapsize})
+Read the columns given in the list @code{indexll} from a plain text file
+(@code{filename}) or list of strings (@code{lines}), into a linked list of
+data structures (see @ref{List of size_t} and @ref{List of gal_data_t}). If
+the necessary space for each column is larger than @code{minmapsize}, don't
+keep it in the RAM, but in a file on the HDD/SSD, see the description under
+the same name in @ref{Generic data container}.
+
address@hidden is a list of strings with each node representing one line
+(including the new-line character), see @ref{List of strings}. It will
+mostly be the output of @code{gal_txt_stdin_read}, which is used to read
+the program's input as separate lines from the standard input (see
+below). Note that @code{filename} and @code{lines} are mutually exclusive
+and one of them must be @code{NULL}.
Note that this is a low-level function, so the output data list is the
inverse of the input indexs linked list. It is recommended to use
@@ -25661,11 +25732,44 @@ inverse of the input indexs linked list. It is
recommended to use
@ref{Table input output}.
@end deftypefun
address@hidden {gal_data_t *} gal_txt_image_read (char @code{*filename}, size_t
@code{minmapsize})
-Read the 2D plain text dataset in @code{filename} into a dataset and return
-the dataset. If the necessary space for the image is larger than
address@hidden, don't keep it in the RAM, but in a file on the HDD/SSD,
-see the description under the same name in @ref{Generic data container}.
address@hidden {gal_data_t *} gal_txt_image_read (char @code{*filename},
gal_list_str_t @code{*lines}, size_t @code{minmapsize})
+Read the 2D plain text dataset in file (@code{filename}) or list of strings
+(@code{lines}) into a dataset and return the dataset. If the necessary
+space for the image is larger than @code{minmapsize}, don't keep it in the
+RAM, but in a file on the HDD/SSD, see the description under the same name
+in @ref{Generic data container}.
+
address@hidden is a list of strings with each node representing one line
+(including the new-line character), see @ref{List of strings}. It will
+mostly be the output of @code{gal_txt_stdin_read}, which is used to read
+the program's input as separate lines from the standard input (see
+below). Note that @code{filename} and @code{lines} are mutually exclusive
+and one of them must be @code{NULL}.
address@hidden deftypefun
+
address@hidden {gal_list_str_t *} gal_txt_stdin_read (long
@code{timeout_microsec})
address@hidden Standard input
+Read the complete standard input and return a list of strings with each
+line (including the new-line character) as one node of that list. If the
+standard input is already filled (for example connected to another
+program's output with a pipe), then this function will parse the whole
+stream.
+
+If Standard input is not pre-configured and the @emph{first line} is
+typed/written in the terminal before @code{timeout_microsec} micro-seconds,
+it will continue parsing until reaches an end-of-file character
+(@key{CTRL-D} after a new-line on the keyboard) with no time limit. If
+nothing is entered before @code{timeout_microsec} micro-seconds, it will
+return @code{NULL}.
+
+All the functions that can read plain text tables will accept a filename as
+well as a list of strings (intended to be the output of this function for
+using Standard input). The reason for keeping the standard input is that
+once something is read from the standard input, it is hard to put it
+back. We often need to read a text file several times: once to count how
+many columns it has and which ones are requested, and another time to read
+the desired columns. So it easier to keep it all in allocated memory and
+pass it on from the start for each round.
@end deftypefun
@deftypefun void gal_txt_write (gal_data_t @code{*cols}, gal_list_str_t
@code{*comment}, char @code{*filename}, uint8_t @code{colinfoinstdout})
diff --git a/lib/array.c b/lib/array.c
index 6c63afc..c6db2f5 100644
--- a/lib/array.c
+++ b/lib/array.c
@@ -86,7 +86,8 @@ gal_array_name_recognized_multiext(char *name)
/* Read (all the possibly existing) color channels within each
extension/dir of the given file. */
gal_data_t *
-gal_array_read(char *filename, char *extension, size_t minmapsize)
+gal_array_read(char *filename, char *extension, gal_list_str_t *lines,
+ size_t minmapsize)
{
size_t ext;
@@ -107,7 +108,7 @@ gal_array_read(char *filename, char *extension, size_t
minmapsize)
/* Default: plain text. */
else
- return gal_txt_image_read(filename, minmapsize);
+ return gal_txt_image_read(filename, lines, minmapsize);
/* Control should not get to here, but just to avoid compiler warnings,
we'll return a NULL. */
@@ -123,11 +124,13 @@ gal_array_read(char *filename, char *extension, size_t
minmapsize)
/* Read the contents of the given file/extension to a specific type. */
gal_data_t *
-gal_array_read_to_type(char *filename, char *extension, uint8_t type,
+gal_array_read_to_type(char *filename, char *extension,
+ gal_list_str_t *lines, uint8_t type,
size_t minmapsize)
{
gal_data_t *out=NULL;
- gal_data_t *next, *in=gal_array_read(filename, extension, minmapsize);
+ gal_data_t *next, *in=gal_array_read(filename, extension, lines,
+ minmapsize);
/* Go over all the channels. */
while(in)
@@ -149,11 +152,12 @@ gal_array_read_to_type(char *filename, char *extension,
uint8_t type,
/* Read the input array and make sure it is only one channel. */
gal_data_t *
-gal_array_read_one_ch(char *filename, char *extension, size_t minmapsize)
+gal_array_read_one_ch(char *filename, char *extension, gal_list_str_t *lines,
+ size_t minmapsize)
{
char *fname;
gal_data_t *out;
- out=gal_array_read(filename, extension, minmapsize);
+ out=gal_array_read(filename, extension, lines, minmapsize);
if(out->next)
{
@@ -183,10 +187,12 @@ gal_array_read_one_ch(char *filename, char *extension,
size_t minmapsize)
/* Read a single-channel dataset into a specific type. */
gal_data_t *
-gal_array_read_one_ch_to_type(char *filename, char *extension, uint8_t type,
+gal_array_read_one_ch_to_type(char *filename, char *extension,
+ gal_list_str_t *lines, uint8_t type,
size_t minmapsize)
{
- gal_data_t *out=gal_array_read_one_ch(filename, extension, minmapsize);
+ gal_data_t *out=gal_array_read_one_ch(filename, extension, lines,
+ minmapsize);
return gal_data_copy_to_new_type_free(out, type);
}
diff --git a/lib/gnuastro-internal/commonopts.h
b/lib/gnuastro-internal/commonopts.h
index 300f498..50f3397 100644
--- a/lib/gnuastro-internal/commonopts.h
+++ b/lib/gnuastro-internal/commonopts.h
@@ -60,7 +60,7 @@ struct argp_option gal_commonopts_options[] =
GAL_OPTIONS_KEY_SEARCHIN,
"STR",
0,
- "Select column(s) in: `name', `unit', `comment'.",
+ "Select column(s): `name', `unit', `comment'.",
GAL_OPTIONS_GROUP_INPUT,
&cp->searchin,
GAL_TYPE_STRING,
@@ -82,6 +82,19 @@ struct argp_option gal_commonopts_options[] =
GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
+ {
+ "stdintimeout",
+ GAL_OPTIONS_KEY_STDINTIMEOUT,
+ "INT",
+ 0,
+ "Micro-seconds to wait for standard input.",
+ GAL_OPTIONS_GROUP_INPUT,
+ &cp->stdintimeout,
+ GAL_TYPE_LONG,
+ GAL_OPTIONS_RANGE_GE_0,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET
+ },
diff --git a/lib/gnuastro-internal/options.h b/lib/gnuastro-internal/options.h
index 37641a4..bc71b5c 100644
--- a/lib/gnuastro-internal/options.h
+++ b/lib/gnuastro-internal/options.h
@@ -107,7 +107,8 @@ enum options_common_keys
GAL_OPTIONS_KEY_REMAINDERFRAC = 'F',
/* Only long option (integers for keywords). */
- GAL_OPTIONS_KEY_MINMAPSIZE = 500,
+ GAL_OPTIONS_KEY_STDINTIMEOUT = 500,
+ GAL_OPTIONS_KEY_MINMAPSIZE,
GAL_OPTIONS_KEY_LOG,
GAL_OPTIONS_KEY_CITE,
GAL_OPTIONS_KEY_CONFIG,
@@ -183,6 +184,7 @@ struct gal_options_common_params
char *hdu; /* Image extension. */
uint8_t searchin; /* Column meta-data to match/search. */
uint8_t ignorecase; /* Ignore case when matching col info. */
+ long stdintimeout; /* Timeout (micro-seconds) for stdin. */
/* Output. */
char *output; /* Directory containg output. */
@@ -301,6 +303,9 @@ gal_options_set_from_key(int key, char *arg, struct
argp_option *options,
error_t
gal_options_common_argp_parse(int key, char *arg, struct argp_state *state);
+gal_list_str_t *
+gal_options_check_stdin(char *inputname, long stdintimeout);
+
diff --git a/lib/gnuastro/array.h b/lib/gnuastro/array.h
index faff5f2..9b92fc1 100644
--- a/lib/gnuastro/array.h
+++ b/lib/gnuastro/array.h
@@ -59,17 +59,21 @@ int
gal_array_name_recognized_multiext(char *name);
gal_data_t *
-gal_array_read(char *filename, char *extension, size_t minmapsize);
+gal_array_read(char *filename, char *extension, gal_list_str_t *lines,
+ size_t minmapsize);
gal_data_t *
-gal_array_read_to_type(char *filename, char *extension, uint8_t type,
+gal_array_read_to_type(char *filename, char *extension,
+ gal_list_str_t *lines, uint8_t type,
size_t minmapsize);
gal_data_t *
-gal_array_read_one_ch(char *filename, char *extension, size_t minmapsize);
+gal_array_read_one_ch(char *filename, char *extension, gal_list_str_t *lines,
+ size_t minmapsize);
gal_data_t *
-gal_array_read_one_ch_to_type(char *filename, char *extension, uint8_t type,
+gal_array_read_one_ch_to_type(char *filename, char *extension,
+ gal_list_str_t *lines, uint8_t type,
size_t minmapsize);
diff --git a/lib/gnuastro/table.h b/lib/gnuastro/table.h
index b92a340..68b2012 100644
--- a/lib/gnuastro/table.h
+++ b/lib/gnuastro/table.h
@@ -125,8 +125,8 @@ enum gal_table_where_to_search
/*************** Information about a table ***************/
/************************************************************************/
gal_data_t *
-gal_table_info(char *filename, char *hdu, size_t *numcols,
- size_t *numrows, int *tableformat);
+gal_table_info(char *filename, char *hdu, gal_list_str_t *lines,
+ size_t *numcols, size_t *numrows, int *tableformat);
void
gal_table_print_info(gal_data_t *allcols, size_t numcols, size_t numrows);
@@ -137,9 +137,9 @@ gal_table_print_info(gal_data_t *allcols, size_t numcols,
size_t numrows);
/*************** Read a table ***************/
/************************************************************************/
gal_data_t *
-gal_table_read(char *filename, char *hdu, gal_list_str_t *cols,
- int searchin, int ignorecase, size_t minmapsize,
- size_t *colmatch);
+gal_table_read(char *filename, char *hdu, gal_list_str_t *lines,
+ gal_list_str_t *cols, int searchin, int ignorecase,
+ size_t minmapsize, size_t *colmatch);
diff --git a/lib/gnuastro/txt.h b/lib/gnuastro/txt.h
index e26e7f5..4429b00 100644
--- a/lib/gnuastro/txt.h
+++ b/lib/gnuastro/txt.h
@@ -79,14 +79,23 @@ int
gal_txt_line_stat(char *line);
gal_data_t *
-gal_txt_table_info(char *filename, size_t *numcols, size_t *numrows);
+gal_txt_table_info(char *filename, gal_list_str_t *lines, size_t *numcols,
+ size_t *numrows);
gal_data_t *
-gal_txt_table_read(char *filename, size_t numrows, gal_data_t *colinfo,
- gal_list_sizet_t *indexll, size_t minmapsize);
+gal_txt_image_info(char *filename, gal_list_str_t *lines, size_t *numimg,
+ size_t *dsize);
gal_data_t *
-gal_txt_image_read(char *filename, size_t minmapsize);
+gal_txt_table_read(char *filename, gal_list_str_t *lines, size_t numrows,
+ gal_data_t *colinfo, gal_list_sizet_t *indexll,
+ size_t minmapsize);
+
+gal_data_t *
+gal_txt_image_read(char *filename, gal_list_str_t *lines, size_t minmapsize);
+
+gal_list_str_t *
+gal_txt_stdin_read(long timeout_microsec);
void
gal_txt_write(gal_data_t *input, gal_list_str_t *comment, char *filename,
diff --git a/lib/options.c b/lib/options.c
index 3e8eedc..8facca8 100644
--- a/lib/options.c
+++ b/lib/options.c
@@ -1358,6 +1358,32 @@ gal_options_common_argp_parse(int key, char *arg, struct
argp_state *state)
+/* Make the notice that is printed before program terminates, when no input
+ is given and Standard input is also available. */
+gal_list_str_t *
+gal_options_check_stdin(char *inputname, long stdintimeout)
+{
+ gal_list_str_t *lines=inputname ? NULL : gal_txt_stdin_read(stdintimeout);
+
+ /* See if atleast one of the two inputs is given. */
+ if(inputname==NULL && lines==NULL)
+ error( EXIT_FAILURE, 0, "no input!\n\n"
+ "The (first) input dataset can be read from a file "
+ "(specified as an argument) or the standard input. If "
+ "both are provided, a file takes precedence. Standard "
+ "input can come from a pipe (output of another program) "
+ "or typed on the command-line before %ld micro-seconds "
+ "(configurable with the `--stdintimeout' option).",
+ stdintimeout );
+
+ /* Return the output. */
+ return lines;
+}
+
+
+
+
+
diff --git a/lib/table.c b/lib/table.c
index 84bfdd5..69246ae 100644
--- a/lib/table.c
+++ b/lib/table.c
@@ -60,16 +60,16 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
comments), nothing in the data structure(s) will be allocated by this
function for the actual data (e.g., the `array' or `dsize' elements). */
gal_data_t *
-gal_table_info(char *filename, char *hdu, size_t *numcols, size_t *numrows,
- int *tableformat)
+gal_table_info(char *filename, char *hdu, gal_list_str_t *lines,
+ size_t *numcols, size_t *numrows, int *tableformat)
{
/* Get the table format and size (number of columns and rows). */
- if(gal_fits_name_is_fits(filename))
+ if(filename && gal_fits_name_is_fits(filename))
return gal_fits_tab_info(filename, hdu, numcols, numrows, tableformat);
else
{
*tableformat=GAL_TABLE_FORMAT_TXT;
- return gal_txt_table_info(filename, numcols, numrows);
+ return gal_txt_table_info(filename, lines, numcols, numrows);
}
/* Abort with an error if we get to this point. */
@@ -403,9 +403,9 @@ make_list_of_indexs(gal_list_str_t *cols, gal_data_t
*allcols,
So the first requested column is the first popped data structure and so
on. */
gal_data_t *
-gal_table_read(char *filename, char *hdu, gal_list_str_t *cols,
- int searchin, int ignorecase, size_t minmapsize,
- size_t *colmatch)
+gal_table_read(char *filename, char *hdu, gal_list_str_t *lines,
+ gal_list_str_t *cols, int searchin, int ignorecase,
+ size_t minmapsize, size_t *colmatch)
{
int tableformat;
gal_list_sizet_t *indexll;
@@ -413,7 +413,8 @@ gal_table_read(char *filename, char *hdu, gal_list_str_t
*cols,
gal_data_t *allcols, *out=NULL;
/* First get the information of all the columns. */
- allcols=gal_table_info(filename, hdu, &numcols, &numrows, &tableformat);
+ allcols=gal_table_info(filename, hdu, lines, &numcols, &numrows,
+ &tableformat);
/* If there was no actual data in the file, then return NULL. */
if(allcols==NULL) return NULL;
@@ -435,7 +436,7 @@ gal_table_read(char *filename, char *hdu, gal_list_str_t
*cols,
switch(tableformat)
{
case GAL_TABLE_FORMAT_TXT:
- out=gal_txt_table_read(filename, numrows, allcols, indexll,
+ out=gal_txt_table_read(filename, lines, numrows, allcols, indexll,
minmapsize);
break;
diff --git a/lib/txt.c b/lib/txt.c
index 6b16337..a9fe6b9 100644
--- a/lib/txt.c
+++ b/lib/txt.c
@@ -27,6 +27,7 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
#include <stdio.h>
#include <errno.h>
#include <error.h>
+#include <unistd.h>
#include <string.h>
#include <stdlib.h>
@@ -290,11 +291,21 @@ txt_info_from_comment(char *line, gal_data_t **datall,
char *comm_start)
given text file. If the file is a text table with string columns, the
contents of the string column will be counted as one token.*/
static size_t
-txt_info_from_first_row(char *line, gal_data_t **datall, int format)
+txt_info_from_first_row(char *in_line, gal_data_t **datall, int format,
+ int inplace)
{
gal_data_t *col, *prev, *tmp;
size_t n=0, maxcnum=0, numtokens;
- char *token, *end=line+strlen(line);
+ char *line, *token, *end, *aline=NULL;
+
+ /* Make a copy of the input line if necessary. */
+ if(inplace) line=in_line;
+ else
+ {
+ gal_checkset_allocate_copy(in_line, &line);
+ aline=line; /* We are going to change `line' during this function. */
+ }
+ end=line+strlen(line);
/* Remove the line termination character(s) from the end of the line. In
Unix, the line terminator is just the new-line character, however, in
@@ -307,8 +318,8 @@ txt_info_from_first_row(char *line, gal_data_t **datall,
int format)
the available space on the line, we don't want to have the line's
new-line character. Its better for it to actually be shorter than the
space. */
- if( *(end-2)==13 ) *(end-2)='\0';
- else *(end-1)='\0';
+ if( end>line+2 && *(end-2)==13 ) *(end-2)='\0';
+ else if( *(end-1)=='\n' ) *(end-1)='\0';
/* Get the maximum number of columns read from the comment
information. */
@@ -438,6 +449,7 @@ txt_info_from_first_row(char *line, gal_data_t **datall,
int format)
}
/* Return the total number of columns/second-img-dimension. */
+ if(inplace==0) free(aline);
return numtokens;
}
@@ -517,18 +529,58 @@ txt_infoll_to_array(gal_data_t *datall, size_t *numdata)
+static void
+txt_get_info_line(char *line, gal_data_t **datall, char *comm_start,
+ int *firstlinedone, int format, size_t *dsize, int inplace)
+{
+ size_t numtokens;
+
+ switch( gal_txt_line_stat(line) )
+ {
+ /* Line is a comment, see if it has formatted information. */
+ case GAL_TXT_LINESTAT_COMMENT:
+ txt_info_from_comment(line, datall, comm_start);
+ break;
+
+ /* Line is actual data, use it to fill in the gaps. */
+ case GAL_TXT_LINESTAT_DATAROW:
+ ++dsize[0];
+ if(*firstlinedone==0)
+ {
+ *firstlinedone=1;
+ numtokens=txt_info_from_first_row(line, datall, format, inplace);
+ if(format==TXT_FORMAT_IMAGE) dsize[1]=numtokens;
+ }
+ break;
+
+ /* We also have the case of GAL_TXT_LINESTAT_BLANK, but we don't
+ need to do anything about it. */
+ }
+}
+
+
+
+
+
/* Return the information about a text file table. If there were no
readable rows, it will return NULL.*/
static gal_data_t *
-txt_get_info(char *filename, int format, size_t *numdata, size_t *dsize)
+txt_get_info(char *filename, gal_list_str_t *lines, int format,
+ size_t *numdata, size_t *dsize)
{
FILE *fp;
- size_t numtokens;
- int firstlinedone=0;
- gal_data_t *datall=NULL, *dataarr;
+ gal_list_str_t *tmp;
+ gal_data_t *datall=NULL;
+ int test, firstlinedone=0;
char *line, *format_err="empty", *comm_start;
size_t linelen=10; /* `linelen' will be increased by `getline'. */
+ /* `filename' and `lines' cannot both be non-NULL. */
+ test = (filename!=NULL) + (lines!=NULL);
+ if( test!=1 )
+ error(EXIT_FAILURE, 0, "%s: one of the `filename' and `lines' "
+ "arguments must be NULL, but they are both %s", __func__,
+ test==2 ? "non-NULL" : "NULL");
/* Set the constant strings */
switch(format)
@@ -540,75 +592,59 @@ txt_get_info(char *filename, int format, size_t *numdata,
size_t *dsize)
__func__, format);
}
+ /* Initialize the first `dsize' element. */
+ dsize[0]=0;
- /* Open the file. */
- errno=0;
- fp=fopen(filename, "r");
- if(fp==NULL)
- error(EXIT_FAILURE, errno, "%s: couldn't open to read as a plain "
- "text %s (from Gnuastro's `%s')", filename, format_err, __func__);
+ /* Parse the file or go over the lines. */
+ if(filename)
+ {
+ /* Open the file. */
+ errno=0;
+ fp=fopen(filename, "r");
+ if(fp==NULL)
+ error(EXIT_FAILURE, errno, "%s: couldn't open to read as a plain "
+ "text %s (from Gnuastro's `%s')", filename, format_err,
+ __func__);
- /* Allocate the space necessary to keep each line as we parse it. Note
- that `getline' is going to later `realloc' this space to fit the line
- length. */
- errno=0;
- line=malloc(linelen*sizeof *line);
- if(line==NULL)
- error(EXIT_FAILURE, errno, "%s: allocating %zu bytes for line",
- __func__, linelen*sizeof *line);
+ /* Allocate the space necessary to keep each line as we parse
+ it. Note that `getline' is going to later `realloc' this space to
+ fit the line length. */
+ errno=0;
+ line=malloc(linelen*sizeof *line);
+ if(line==NULL)
+ error(EXIT_FAILURE, errno, "%s: allocating %zu bytes for line",
+ __func__, linelen*sizeof *line);
- /* Read the comments of the line for possible information about the
- lines, but also confirm/complete the info by parsing the first
- uncommented line. */
- dsize[0]=0;
- while( getline(&line, &linelen, fp) != -1 )
- switch( gal_txt_line_stat(line) )
- {
- /* Line is a comment, see if it has formatted information. */
- case GAL_TXT_LINESTAT_COMMENT:
- txt_info_from_comment(line, &datall, comm_start);
- break;
-
- /* Line is actual data, use it to fill in the gaps. */
- case GAL_TXT_LINESTAT_DATAROW:
- ++dsize[0];
- if(firstlinedone==0)
- {
- firstlinedone=1;
- numtokens=txt_info_from_first_row(line, &datall, format);
- if(format==TXT_FORMAT_IMAGE) dsize[1]=numtokens;
- }
- break;
-
- /* We also have the case of GAL_TXT_LINESTAT_BLANK. */
- }
+ /* Read the comments of the line for possible information about the
+ lines, but also confirm/complete the info by parsing the first
+ uncommented line. */
+ while( getline(&line, &linelen, fp) != -1 )
+ txt_get_info_line(line, &datall, comm_start, &firstlinedone, format,
+ dsize, 1);
+
+ /* Clean up and close the file. */
+ free(line);
+ errno=0;
+ if(fclose(fp))
+ error(EXIT_FAILURE, errno, "%s: couldn't close file after reading "
+ "plain text %s information in %s", filename, format_err,
+ __func__);
+ }
+ else
+ {
+ for(tmp=lines; tmp!=NULL; tmp=tmp->next)
+ txt_get_info_line(tmp->v, &datall, comm_start, &firstlinedone,
+ format, dsize, 0);
+ }
/* The final dataset linked list can have any order (depending on how the
user gave column information in tables for example). So here, we will
convert the list into a nicely sorted array, note that this function
frees list as part of the process. */
- dataarr=txt_infoll_to_array(datall, numdata);
-
-
- /* Clean up. Note that even if there were no usable columns, there might
- have been meta-data comments, so we need to free `colsll' in any
- case. If the list is indeed empty, then `gal_data_free_ll' won't do
- anything. */
- free(line);
-
-
- /* Close the file. */
- errno=0;
- if(fclose(fp))
- error(EXIT_FAILURE, errno, "%s: couldn't close file after reading plain "
- "text %s information in %s", filename, format_err, __func__);
-
-
- /* Return the array of column information. */
- return dataarr;
+ return txt_infoll_to_array(datall, numdata);
}
@@ -617,9 +653,10 @@ txt_get_info(char *filename, int format, size_t *numdata,
size_t *dsize)
/* Get the information of each column in a text file */
gal_data_t *
-gal_txt_table_info(char *filename, size_t *numcols, size_t *numrows)
+gal_txt_table_info(char *filename, gal_list_str_t *lines, size_t *numcols,
+ size_t *numrows)
{
- return txt_get_info(filename, TXT_FORMAT_TABLE, numcols, numrows);
+ return txt_get_info(filename, lines, TXT_FORMAT_TABLE, numcols, numrows);
}
@@ -628,9 +665,10 @@ gal_txt_table_info(char *filename, size_t *numcols, size_t
*numrows)
/* Get the information of a 2D array in a text file. */
gal_data_t *
-gal_txt_image_info(char *filename, size_t *numimg, size_t *dsize)
+gal_txt_image_info(char *filename, gal_list_str_t *lines, size_t *numimg,
+ size_t *dsize)
{
- return txt_get_info(filename, TXT_FORMAT_IMAGE, numimg, dsize);
+ return txt_get_info(filename, lines, TXT_FORMAT_IMAGE, numimg, dsize);
}
@@ -768,17 +806,27 @@ txt_read_token(gal_data_t *data, gal_data_t *info, char
*token,
static void
-txt_fill(char *line, char **tokens, size_t maxcolnum, gal_data_t *info,
- gal_data_t *out, size_t rowind, char *filename, size_t lineno)
+txt_fill(char *in_line, char **tokens, size_t maxcolnum, gal_data_t *info,
+ gal_data_t *out, size_t rowind, char *filename, size_t lineno,
+ int inplace)
{
size_t i, n=0;
gal_data_t *data;
int notenoughcols=0;
- char *end=line+strlen(line);
+ char *end, *line, *aline=NULL;
+
+ /* Make a copy of the input line if necessary. */
+ if(inplace) line=in_line;
+ else
+ {
+ gal_checkset_allocate_copy(in_line, &line);
+ aline=line; /* We are going to change `line' during this function. */
+ }
+ end=line+strlen(line);
/* See explanations in `txt_info_from_first_row'. */
- if( *(end-2)==13 ) *(end-2)='\0';
- else *(end-1)='\0';
+ if( end>line+2 && *(end-2)==13 ) *(end-2)='\0';
+ else if( *(end-1)=='\n' ) *(end-1)='\0';
/* Start parsing the line. Note that `n' and `maxcolnum' start from
one. */
@@ -848,6 +896,9 @@ txt_fill(char *line, char **tokens, size_t maxcolnum,
gal_data_t *info,
error(EXIT_FAILURE, 0, "%s: currently only 1 and 2 dimensional "
"datasets acceptable", __func__);
}
+
+ /* Clean up. */
+ if(inplace==0) free(aline);
}
@@ -855,23 +906,26 @@ txt_fill(char *line, char **tokens, size_t maxcolnum,
gal_data_t *info,
static gal_data_t *
-gal_txt_read(char *filename, size_t *dsize, gal_data_t *info,
- gal_list_sizet_t *indexll, size_t minmapsize, int format)
+txt_read(char *filename, gal_list_str_t *lines, size_t *dsize,
+ gal_data_t *info, gal_list_sizet_t *indexll, size_t minmapsize,
+ int format)
{
FILE *fp;
+ int test;
char *line;
char **tokens;
+ gal_list_str_t *tmp;
gal_data_t *out=NULL;
gal_list_sizet_t *ind;
size_t one=1, maxcolnum=0, rowind=0, lineno=0, ndim;
size_t linelen=10; /* `linelen' will be increased by `getline'. */
- /* Open the file. */
- errno=0;
- fp=fopen(filename, "r");
- if(fp==NULL)
- error(EXIT_FAILURE, errno, "%s: couldn't open to read as a text table "
- "in %s", filename, __func__);
+ /* `filename' and `lines' cannot both be non-NULL. */
+ test = (filename!=NULL) + (lines!=NULL);
+ if( test!=1 )
+ error(EXIT_FAILURE, 0, "%s: one of the `filename' and `lines' "
+ "arguments must be NULL, but they are both %s", __func__,
+ test==2 ? "non-NULL" : "NULL");
/* Allocate the space necessary to keep a copy of each line as we parse
it. Note that `getline' is going to later `realloc' this space to fit
@@ -947,24 +1001,42 @@ gal_txt_read(char *filename, size_t *dsize, gal_data_t
*info,
error(EXIT_FAILURE, errno, "%s: allocating %zu bytes for `tokens'",
__func__, (maxcolnum+1)*sizeof *tokens);
- /* Read the data columns. */
- while( getline(&line, &linelen, fp) != -1 )
+ if(filename)
{
- ++lineno;
- if( gal_txt_line_stat(line) == GAL_TXT_LINESTAT_DATAROW )
- txt_fill(line, tokens, maxcolnum, info, out, rowind++,
- filename, lineno);
+ /* Open the file. */
+ errno=0;
+ fp=fopen(filename, "r");
+ if(fp==NULL)
+ error(EXIT_FAILURE, errno, "%s: couldn't open to read as a text "
+ "table in %s", filename, __func__);
+
+ /* Read the data columns. */
+ while( getline(&line, &linelen, fp) != -1 )
+ {
+ ++lineno;
+ if( gal_txt_line_stat(line) == GAL_TXT_LINESTAT_DATAROW )
+ txt_fill(line, tokens, maxcolnum, info, out, rowind++,
+ filename, lineno, 1);
+ }
+
+ /* Clean up and close the file. */
+ errno=0;
+ if(fclose(fp))
+ error(EXIT_FAILURE, errno, "%s: couldn't close file after reading "
+ "ASCII table information in %s", filename, __func__);
+ free(line);
}
+ else
+ for(tmp=lines; tmp!=NULL; tmp=tmp->next)
+ {
+ ++lineno;
+ if( gal_txt_line_stat(tmp->v) == GAL_TXT_LINESTAT_DATAROW )
+ txt_fill(tmp->v, tokens, maxcolnum, info, out, rowind++,
+ filename, lineno, 0);
+ }
- /* Clean up and close the file. */
- errno=0;
- if(fclose(fp))
- error(EXIT_FAILURE, errno, "%s: couldn't close file after reading ASCII "
- "table information in %s", filename, __func__);
+ /* Clean up and return. */
free(tokens);
- free(line);
-
- /* Return the array of column information. */
return out;
}
@@ -973,11 +1045,12 @@ gal_txt_read(char *filename, size_t *dsize, gal_data_t
*info,
gal_data_t *
-gal_txt_table_read(char *filename, size_t numrows, gal_data_t *colinfo,
- gal_list_sizet_t *indexll, size_t minmapsize)
+gal_txt_table_read(char *filename, gal_list_str_t *lines, size_t numrows,
+ gal_data_t *colinfo, gal_list_sizet_t *indexll,
+ size_t minmapsize)
{
- return gal_txt_read(filename, &numrows, colinfo, indexll, minmapsize,
- TXT_FORMAT_TABLE);
+ return txt_read(filename, lines, &numrows, colinfo, indexll, minmapsize,
+ TXT_FORMAT_TABLE);
}
@@ -985,18 +1058,18 @@ gal_txt_table_read(char *filename, size_t numrows,
gal_data_t *colinfo,
gal_data_t *
-gal_txt_image_read(char *filename, size_t minmapsize)
+gal_txt_image_read(char *filename, gal_list_str_t *lines, size_t minmapsize)
{
size_t numimg, dsize[2];
gal_data_t *img, *imginfo;
gal_list_sizet_t *indexll=NULL;
/* Get the image information. */
- imginfo=gal_txt_image_info(filename, &numimg, dsize);
+ imginfo=gal_txt_image_info(filename, lines, &numimg, dsize);
/* Read the table. */
- img=gal_txt_read(filename, dsize, imginfo, indexll, minmapsize,
- TXT_FORMAT_IMAGE);
+ img=txt_read(filename, lines, dsize, imginfo, indexll, minmapsize,
+ TXT_FORMAT_IMAGE);
/* Clean up and return. */
gal_data_free(imginfo);
@@ -1006,6 +1079,87 @@ gal_txt_image_read(char *filename, size_t minmapsize)
+/* See if there is anything in the standard input already. This function is
+ modeled on the solution provided in:
+
+ https://stackoverflow.com/questions/3711830/set-a-timeout-for-reading-stdin
*/
+static int
+txt_stdin_has_contents(long timeout_microsec)
+{
+ fd_set fds;
+ struct timeval tv;
+
+ /* Set the timeout time. */
+ tv.tv_sec = 0;
+ tv.tv_usec = timeout_microsec;
+
+ /* Initialize `fd_set'. */
+ FD_ZERO(&fds);
+
+ /* Set standard input (STDIN_FILENO is 0) as the FD that must be read. */
+ FD_SET(STDIN_FILENO, &fds);
+
+ /* `select' takes the last file descriptor value + 1 in the fdset to
+ check, the fdset for reads, writes, and errors. We are only passing
+ in reads. the last parameter is the timeout. select will return if
+ an FD is ready or the timeout has occurred. */
+ select(STDIN_FILENO+1, &fds, NULL, NULL, &tv);
+
+ // return 0 if STDIN is not ready to be read.
+ return FD_ISSET(STDIN_FILENO, &fds);
+}
+
+
+
+
+/* Read each line of the standard input into a linked list of strings. */
+gal_list_str_t *
+gal_txt_stdin_read(long timeout_microsec)
+{
+ char *line;
+ gal_list_str_t *out=NULL;
+ size_t lineno=0, linelen=10;/* `linelen' will be increased by `getline'. */
+
+ /* If there is nothing */
+ if( txt_stdin_has_contents(timeout_microsec) )
+ {
+ /* Allocate the space necessary to keep a copy of each line as we
+ parse it. Note that `getline' is going to later `realloc' this
+ space to fit the line length. */
+ errno=0;
+ line=malloc(linelen*sizeof *line);
+ if(line==NULL)
+ error(EXIT_FAILURE, errno, "%s: allocating %zu bytes for `line'",
+ __func__, linelen*sizeof *line);
+
+ /* Read the whole standard input. We are using getline because it can
+ deal with a `NULL' in the input, while also handing allocation
+ issues while reading (allocating by line, not by a fixed buffer
+ size). */
+ while( getline(&line, &linelen, stdin) != -1 )
+ {
+ /* To help in reporting (when necessary), keep a count of how
+ many lines we have. */
+ ++lineno;
+
+ /* Add the line to the output list. */
+ gal_list_str_add(&out, line, 1);
+ }
+
+ /* Reverse the list (to be the same order as input). */
+ gal_list_str_reverse(&out);
+
+ /* Clean up. */
+ free(line);
+ }
+
+ /* Return the result. */
+ return out;
+}
+
+
+
+
diff --git a/tests/Makefile.am b/tests/Makefile.am
index e3c7379..1ca09b0 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -167,10 +167,12 @@ if COND_SEGMENT
segment/segment.sh: noisechisel/noisechisel.sh.log
endif
if COND_STATISTICS
- MAYBE_STATISTICS_TESTS = statistics/basicstats.sh statistics/estimate_sky.sh
+ MAYBE_STATISTICS_TESTS = statistics/basicstats.sh statistics/estimate_sky.sh
\
+ statistics/from-stdin.sh
- statistics/basicstats.sh: mknoise/addnoise.sh.log
+ statistics/basicstats.sh: mknoise/addnoise.sh.log
statistics/estimate_sky.sh: mknoise/addnoise.sh.log
+ statistics/from-stdin.sh: prepconf.sh.log
endif
if COND_TABLE
MAYBE_TABLE_TESTS = table/txt-to-fits-binary.sh \
diff --git a/tests/statistics/from-stdin.sh b/tests/statistics/from-stdin.sh
new file mode 100755
index 0000000..7105fdb
--- /dev/null
+++ b/tests/statistics/from-stdin.sh
@@ -0,0 +1,56 @@
+# Measure basic table statistics using standard input.
+#
+# See the Tests subsection of the manual for a complete explanation
+# (in the Installing gnuastro section).
+#
+# Original author:
+# Mohammad Akhlaghi <address@hidden>
+# Contributing author(s):
+#
+# Copying and distribution of this file, with or without modification,
+# are permitted in any medium without royalty provided the copyright
+# notice and this notice are preserved. This file is offered as-is,
+# without any warranty.
+
+
+
+
+
+# Preliminaries
+# =============
+#
+# Set the variables (The executable is in the build tree). Do the
+# basic checks to see if the executable is made or if the defaults
+# file exists (basicchecks.sh is in the source tree).
+prog=statistics
+execname=../bin/$prog/ast$prog
+in=$topsrc/tests/$prog/stdin-input.txt
+
+
+
+
+
+# Skip?
+# =====
+#
+# If the dependencies of the test don't exist, then skip it. There are two
+# types of dependencies:
+#
+# - The executable was not made (for example due to a configure option),
+#
+# - The input data was not made (for example the test that created the
+# data file failed).
+if [ ! -f $execname ]; then echo "$execname not created."; exit 77; fi
+if [ ! -f $in ]; then echo "$in does not exist."; exit 77; fi
+
+
+
+
+
+# Actual test script
+# ==================
+#
+# `check_with_program' can be something like `Valgrind' or an empty
+# string. Such programs will execute the command if present and help in
+# debugging when the developer doesn't have access to the user's system.
+cat $in | $check_with_program $execname
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [gnuastro-commits] master c4d8e33: Statistics can read from Standard input,
Mohammad Akhlaghi <=