[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[gnuastro-commits] master 1c6b577 2/2: Correction touches to NoiseChisel
|
From: |
Mohammad Akhlaghi |
|
Subject: |
[gnuastro-commits] master 1c6b577 2/2: Correction touches to NoiseChisel, Segment and MakeCatalog |
|
Date: |
Tue, 10 Apr 2018 20:07:01 -0400 (EDT) |
branch: master
commit 1c6b5773b8c4d009952f7226b5510a738a40b706
Author: Mohammad Akhlaghi <address@hidden>
Commit: Mohammad Akhlaghi <address@hidden>
Correction touches to NoiseChisel, Segment and MakeCatalog
This commit started with correcting the General program usage tutorial for
the new NoiseChisel, Segment and MakeCatalog programs. However, as I made
progress, I noticed some issues in all three programs that needed to be
corrected/added. Here is a short summary of the major corrections/changes
for each program.
- NoiseChisel: The default outout is now an 8-bit binary detection image
(to be fed into Segment). So it has a new `--label' option to mark
connected components.
- NoiseChisel: The default `--detgrowquant' option was increased to
`0.80'. `0.7' was good for the example mock image produced in `make
check', but on real datasets, it may be too much.
- MakeCatalog: By default the clumps catalog will also be sorted by object
ID. The new `--noclumpsort' option can be used to avoid this sorting
(after all, it takes some processing power).
- MakeCatalog: No more complaints about clump-specific columns if `--ids'
is used only for an objects catalog.
- MakeCatalog and Segment: If a Sky or Sky STD file aren't given, it will
look into the values file (if given), then the objects file (main input
argument).
- MakeCatalog and Segment: Like MakeCatalog, the Sky and its STD can also
be given as values (not just datasets).
- Segment: error can also be given as variance.
---
NEWS | 18 +-
bin/mkcatalog/args.h | 101 +-
bin/mkcatalog/columns.c | 21 +-
bin/mkcatalog/main.h | 3 +
bin/mkcatalog/mkcatalog.c | 103 +-
bin/mkcatalog/parse.c | 36 +-
bin/mkcatalog/ui.c | 190 +-
bin/mkcatalog/ui.h | 5 +-
bin/noisechisel/args.h | 13 +
bin/noisechisel/astnoisechisel.conf | 2 +-
bin/noisechisel/detection.c | 40 +-
bin/noisechisel/main.h | 1 +
bin/noisechisel/noisechisel.c | 25 +-
bin/noisechisel/threshold.c | 25 +-
bin/noisechisel/ui.c | 2 +-
bin/noisechisel/ui.h | 3 +-
bin/segment/args.h | 123 +-
bin/segment/clumps.c | 69 +-
bin/segment/main.h | 8 +-
bin/segment/segment.c | 59 +-
bin/segment/ui.c | 237 ++-
bin/segment/ui.h | 1 +
doc/gnuastro.texi | 1851 +++++++++++---------
lib/arithmetic-and.c | 2 +-
lib/arithmetic-or.c | 2 +-
tests/Makefile.am | 7 +-
.../arithmetic/{or.sh => connected-components.sh} | 7 +-
tests/arithmetic/or.sh | 4 +-
tests/arithmetic/snimage.sh | 2 +-
tests/arithmetic/where.sh | 2 +-
tests/mkcatalog/aperturephot.sh | 2 +-
tests/mkcatalog/detections.sh | 12 +-
tests/mkcatalog/objects-clumps.sh | 2 +-
tests/segment/segment.sh | 2 +-
34 files changed, 1837 insertions(+), 1143 deletions(-)
diff --git a/NEWS b/NEWS
index e5b7772..96ac855 100644
--- a/NEWS
+++ b/NEWS
@@ -32,9 +32,12 @@ GNU Astronomy Utilities NEWS -*-
outline -*-
the values file, but the object labels file. Please see the
"MakeCatalog inputs and basic settings" section of the book for
more. Here is the summary of the the options:
+ --insky: Sky value as a single value or file (dataset).
+ --instd: Sky standard deviation as a single value or file (dataset).
--valuesfile: filename containing the values dataset.
--valueshdu: HDU/extension containing the values dataset.
--clumpscat: Make a clumps catalog also (`WCLUMPS' keyword not used
anymore).
+ --noclumpsort: Don't sort the clumps catalog by host object ID.
--subtractsky: Subtract the given Sky from the values dataset.
--variance: input standard deviation image is actually variance.
--checkupperlimit: make a table for random positions and measurements.
@@ -49,6 +52,13 @@ GNU Astronomy Utilities NEWS -*-
outline -*-
NoiseChisel:
--rawoutput: only output the detection labels and Sky and its STD.
+ --label: label the connected detections. Until now this was the default
+ behavior. However, from this release, NoiseChisel is only in charge
+ of detection. Segmentation is done by a new program (Segment). Since
+ detection is ultimately just a binary operator, the default output
+ now has an 8-bit unsigned integer type with values of 0 or 1. With
+ this option, you can ask for it to label/count the connected
+ detections instead of the default binary output.
Statistics:
--manualbinrange: histogram or CFP range can be outside of distribution.
@@ -98,8 +108,10 @@ GNU Astronomy Utilities NEWS -*-
outline -*-
--comment: can be called/written multiple times in one run.
NoiseChisel:
- Many of the changes below are because NoiseChisel doesn't do
- segmentation any more.
+ From this release, NoiseChisel is only in charge of detection and won't
+ do segmentation any more. The new Segment program is in charge of
+ segmentation. Many of the changes below are due to this new limited
+ scope of NoiseChisel.
--kernel: value `none' will disable convolution.
- Renamed options:
--convolvedhdu ==> --chdu
@@ -107,6 +119,8 @@ GNU Astronomy Utilities NEWS -*-
outline -*-
--detsnminarea ==> --snminarea
--checkdetsn ==> --checksn
--detquant ==> --snquant
+ - By default the detection map is a binary image (values of 0 or 1).
+ - With no output name, the output has a `_detected.fits' suffix.
MakeCatalog:
- Estimation of noise-level is now done per-pixel over the whole
diff --git a/bin/mkcatalog/args.h b/bin/mkcatalog/args.h
index e6e95f4..6232f7e 100644
--- a/bin/mkcatalog/args.h
+++ b/bin/mkcatalog/args.h
@@ -33,63 +33,63 @@ struct argp_option program_options[] =
{
/* Input options. */
{
- "valuesfile",
- UI_KEY_VALUESFILE,
+ "clumpsfile",
+ UI_KEY_CLUMPSFILE,
"STR",
0,
- "Values/brightness dataset.",
+ "Dataset containing clump labels.",
GAL_OPTIONS_GROUP_INPUT,
- &p->valuesfile,
+ &p->clumpsfile,
GAL_TYPE_STRING,
GAL_OPTIONS_RANGE_ANY,
GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
{
- "valueshdu",
- UI_KEY_VALUESHDU,
+ "clumpshdu",
+ UI_KEY_CLUMPSHDU,
"STR",
0,
- "Name or number of extension containing values.",
+ "Clump labels extension name or number.",
GAL_OPTIONS_GROUP_INPUT,
- &p->valueshdu,
+ &p->clumpshdu,
GAL_TYPE_STRING,
GAL_OPTIONS_RANGE_ANY,
GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
{
- "clumpsfile",
- UI_KEY_CLUMPSFILE,
+ "valuesfile",
+ UI_KEY_VALUESFILE,
"STR",
0,
- "Dataset containing clump labels.",
+ "Values/brightness dataset.",
GAL_OPTIONS_GROUP_INPUT,
- &p->clumpsfile,
+ &p->valuesfile,
GAL_TYPE_STRING,
GAL_OPTIONS_RANGE_ANY,
GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
{
- "clumpshdu",
- UI_KEY_CLUMPSHDU,
+ "valueshdu",
+ UI_KEY_VALUESHDU,
"STR",
0,
- "Clump labels extension name or number.",
+ "Name or number of extension containing values.",
GAL_OPTIONS_GROUP_INPUT,
- &p->clumpshdu,
+ &p->valueshdu,
GAL_TYPE_STRING,
GAL_OPTIONS_RANGE_ANY,
GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
{
- "skyfile",
- UI_KEY_SKYFILE,
- "STR",
+ "insky",
+ UI_KEY_INSKY,
+ "STR/FLT",
0,
- "Image containing sky values.",
+ "Input Sky value or dataset.",
GAL_OPTIONS_GROUP_INPUT,
&p->skyfile,
GAL_TYPE_STRING,
@@ -111,11 +111,24 @@ struct argp_option program_options[] =
GAL_OPTIONS_NOT_SET
},
{
- "stdfile",
- UI_KEY_STDFILE,
- "STR",
+ "subtractsky",
+ UI_KEY_SUBTRACTSKY,
+ 0,
0,
- "Image containing sky STD values.",
+ "Subtract the Sky dataset from the values.",
+ GAL_OPTIONS_GROUP_INPUT,
+ &p->subtractsky,
+ GAL_OPTIONS_NO_ARG_TYPE,
+ GAL_OPTIONS_RANGE_0_OR_1,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET
+ },
+ {
+ "instd",
+ UI_KEY_INSTD,
+ "STR/FLT",
+ 0,
+ "Sky standard deviation value or dataset.",
GAL_OPTIONS_GROUP_INPUT,
&p->stdfile,
GAL_TYPE_STRING,
@@ -137,19 +150,6 @@ struct argp_option program_options[] =
GAL_OPTIONS_NOT_SET
},
{
- "zeropoint",
- UI_KEY_ZEROPOINT,
- "FLT",
- 0,
- "Zeropoint magnitude of input dataset.",
- GAL_OPTIONS_GROUP_INPUT,
- &p->zeropoint,
- GAL_TYPE_FLOAT32,
- GAL_OPTIONS_RANGE_ANY,
- GAL_OPTIONS_NOT_MANDATORY,
- GAL_OPTIONS_NOT_SET
- },
- {
"variance",
UI_KEY_VARIANCE,
0,
@@ -163,15 +163,15 @@ struct argp_option program_options[] =
GAL_OPTIONS_NOT_SET
},
{
- "subtractsky",
- UI_KEY_SUBTRACTSKY,
- 0,
+ "zeropoint",
+ UI_KEY_ZEROPOINT,
+ "FLT",
0,
- "Subtract the Sky dataset from the values.",
+ "Zeropoint magnitude of input dataset.",
GAL_OPTIONS_GROUP_INPUT,
- &p->subtractsky,
- GAL_OPTIONS_NO_ARG_TYPE,
- GAL_OPTIONS_RANGE_0_OR_1,
+ &p->zeropoint,
+ GAL_TYPE_FLOAT32,
+ GAL_OPTIONS_RANGE_ANY,
GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
@@ -193,6 +193,19 @@ struct argp_option program_options[] =
GAL_OPTIONS_NOT_SET
},
{
+ "noclumpsort",
+ UI_KEY_NOCLUMPSORT,
+ 0,
+ 0,
+ "Don't sort the clumps catalog by ID.",
+ GAL_OPTIONS_GROUP_OUTPUT,
+ &p->noclumpsort,
+ GAL_OPTIONS_NO_ARG_TYPE,
+ GAL_OPTIONS_RANGE_0_OR_1,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET
+ },
+ {
"sfmagnsigma",
UI_KEY_SFMAGNSIGMA,
"FLT",
diff --git a/bin/mkcatalog/columns.c b/bin/mkcatalog/columns.c
index 188b9ce..f2de605 100644
--- a/bin/mkcatalog/columns.c
+++ b/bin/mkcatalog/columns.c
@@ -1170,8 +1170,15 @@ columns_define_alloc(struct mkcatalogparams *p)
/* If this is a clumps-only column and no clumps image was
given. Add the column to the list of similar columns to inform
- the user. */
- else if(otype==GAL_TYPE_INVALID)
+ the user.
+
+ We'll just ignore the clump-specific ID-related columns,
+ because the `--ids' (generic for both objects and clumps) is a
+ simple generic solution for identifiers. Also, ultimately,
+ they aren't measurements. */
+ else if( otype==GAL_TYPE_INVALID
+ && colcode->v!=UI_KEY_HOSTOBJID
+ && colcode->v!=UI_KEY_IDINHOSTOBJ )
gal_list_str_add(&noclumpimg, name, 1);
}
}
@@ -1449,6 +1456,11 @@ columns_fill(struct mkcatalog_passparams *pp)
key=column->status;
colarr=column->array;
+ /* Put the number of clumps in the internal array which we will need
+ later to order the clump table by object ID. */
+ if(p->numclumps_c)
+ p->numclumps_c[oind]=pp->clumpsinobj;
+
/* Go over all the columns. */
switch(key)
{
@@ -1678,6 +1690,11 @@ columns_fill(struct mkcatalog_passparams *pp)
key = column->status;
ci = &pp->ci[ coind * CCOL_NUMCOLS ];
+ /* Put the object ID of this clump into the temporary array that we
+ will later need to sort the final clumps catalog. */
+ if(p->hostobjid_c)
+ p->hostobjid_c[cind]=pp->object;
+
/* Parse columns */
switch(key)
{
diff --git a/bin/mkcatalog/main.h b/bin/mkcatalog/main.h
index ebbe8e9..b8fe4a7 100644
--- a/bin/mkcatalog/main.h
+++ b/bin/mkcatalog/main.h
@@ -161,6 +161,7 @@ struct mkcatalogparams
char *stdhdu; /* HDU of sky STD image. */
uint8_t clumpscat; /* ==1: create clumps catalog. */
+ uint8_t noclumpsort; /* Don't sort the clumps catalog. */
float zeropoint; /* Zero-point magnitude of object. */
uint8_t variance; /* Input STD file is actually variance. */
uint8_t subtractsky; /* ==1: subtract the Sky from values. */
@@ -205,6 +206,8 @@ struct mkcatalogparams
size_t rngmin; /* Minimum possible value of RNG. */
size_t rngdiff; /* Difference of RNG max and min. */
uint8_t uprangewarning; /* A warning must be printed. */
+ size_t *hostobjid_c; /* To sort the clumps table by Obj.ID. */
+ size_t *numclumps_c; /* To sort the clumps table by Obj.ID. */
char *usedvaluesfile; /* Ptr to final name used for values. */
char *usedclumpsfile; /* Ptr to final name used for clumps. */
diff --git a/bin/mkcatalog/mkcatalog.c b/bin/mkcatalog/mkcatalog.c
index 300dc44..00dfb35 100644
--- a/bin/mkcatalog/mkcatalog.c
+++ b/bin/mkcatalog/mkcatalog.c
@@ -38,6 +38,7 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
#include <gnuastro/threads.h>
#include <gnuastro/dimension.h>
#include <gnuastro/statistics.h>
+#include <gnuastro/permutation.h>
#include <gnuastro-internal/timing.h>
@@ -308,8 +309,9 @@ mkcatalog_write_inputs_in_comments(struct mkcatalogparams
*p,
gal_list_str_t **comments, int withsky,
int withstd)
{
- char *str;
+ char *tmp, *str;
+ /* Basic classifiers for plain text outputs. */
if(p->cp.tableformat==GAL_TABLE_FORMAT_TXT)
{
if( asprintf(&str, "--------- Input files ---------")<0 )
@@ -317,10 +319,12 @@ mkcatalog_write_inputs_in_comments(struct mkcatalogparams
*p,
gal_list_str_add(comments, str, 0);
}
+ /* Object labels. */
if( asprintf(&str, "Objects: %s (hdu: %s).", p->objectsfile, p->cp.hdu)<0 )
error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
gal_list_str_add(comments, str, 0);
+ /* Clump labels. */
if(p->clumps)
{
if(asprintf(&str, "Clumps: %s (hdu: %s).", p->usedclumpsfile,
@@ -329,6 +333,7 @@ mkcatalog_write_inputs_in_comments(struct mkcatalogparams
*p,
gal_list_str_add(comments, str, 0);
}
+ /* Values dataset. */
if(p->values)
{
if( asprintf(&str, "Values: %s (hdu: %s).", p->usedvaluesfile,
@@ -337,22 +342,43 @@ mkcatalog_write_inputs_in_comments(struct mkcatalogparams
*p,
gal_list_str_add(comments, str, 0);
}
+ /* Sky dataset. */
if(withsky && p->sky)
{
- if( asprintf(&str, "Sky: %s (hdu: %s).", p->usedskyfile,
- p->skyhdu)<0 )
- error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
+ if(p->sky->size==1)
+ {
+ if( asprintf(&str, "Sky: %g.", *((float *)(p->sky->array)) )<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
+ }
+ else
+ {
+ if( asprintf(&str, "Sky: %s (hdu: %s).", p->usedskyfile,
+ p->skyhdu)<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
+ }
gal_list_str_add(comments, str, 0);
}
+ /* Sky standard deviation dataset. */
+ tmp = p->variance ? "VAR" : "STD";
if(withstd && p->std)
{
- if( asprintf(&str, "Sky STD: %s (hdu: %s).", p->usedstdfile,
- p->stdhdu)<0 )
- error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
+ if(p->std->size==1)
+ {
+ if( asprintf(&str, "Sky %s: %g.", tmp,
+ *((float *)(p->std->array)) )<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
+ }
+ else
+ {
+ if( asprintf(&str, "Sky %s: %s (hdu: %s).", tmp, p->usedstdfile,
+ p->stdhdu)<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
+ }
gal_list_str_add(comments, str, 0);
}
+ /* Upper limit mask. */
if(p->upmaskfile)
{
if( asprintf(&str, "Upperlimit mask: %s (hdu: %s).", p->upmaskfile,
@@ -509,6 +535,61 @@ mkcatalog_outputs_same_start(struct mkcatalogparams *p,
int o0c1,
+
+/* Since all the measurements were done in parallel (and we didn't know the
+ number of clumps per object a-priori), the clumps informtion is just
+ written in as they are measured. Here, we'll sort the clump columns by
+ object ID. There is an option to disable this. */
+static void
+sort_clumps_by_objid(struct mkcatalogparams *p)
+{
+ gal_data_t *col;
+ size_t o, i, j, *permute, *rowstart;
+
+ /* Make sure everything is fine. */
+ if(p->hostobjid_c==NULL || p->numclumps_c==NULL)
+ error(EXIT_FAILURE, 0, "%s: a bug! Please contact us at %s to fix the "
+ "problem. `p->hostobjid_c' and `p->numclumps_c' must not be "
+ "NULL.", __func__, PACKAGE_BUGREPORT);
+
+
+ /* Allocate the necessary arrays. */
+ rowstart=gal_data_malloc_array(GAL_TYPE_SIZE_T, p->numobjects, __func__,
+ "rowstart");
+ permute=gal_data_malloc_array(GAL_TYPE_SIZE_T, p->numclumps, __func__,
+ "permute");
+
+
+ /* The objects array is already sorted by object ID. So we should just
+ add up the number of clumps to find the row where each object's clumps
+ should start from in the final sorted clumps catalog. */
+ rowstart[0] = 0;
+ for(i=1;i<p->numobjects;++i)
+ rowstart[i] = p->numclumps_c[i-1] + rowstart[i-1];
+
+ /* Fill the permutation array. Note that WE KNOW that all the objects for
+ one clump are after each other.*/
+ i=0;
+ while(i<p->numclumps)
+ {
+ o=p->hostobjid_c[i]-1;
+ for(j=0; j<p->numclumps_c[o]; ++j)
+ permute[i++] = rowstart[o] + j;
+ }
+
+ /* Permute all the clump columns. */
+ for(col=p->clumpcols; col!=NULL; col=col->next)
+ gal_permutation_apply_inverse(col, permute);
+
+ /* Clean up */
+ free(permute);
+ free(rowstart);
+}
+
+
+
+
+
/* Write the produced columns into the output */
static void
mkcatalog_write_outputs(struct mkcatalogparams *p)
@@ -538,10 +619,9 @@ mkcatalog_write_outputs(struct mkcatalogparams *p)
============== */
if(p->clumps)
{
+ /* Make the comments. */
comments=mkcatalog_outputs_same_start(p, 1, "Clumps");
-
-
/* Write objects catalog
---------------------
@@ -604,6 +684,11 @@ mkcatalog(struct mkcatalogparams *p)
and Dec. */
mkcatalog_wcs_conversion(p);
+ /* If the columns need to be sorted (by object ID), then some adjustments
+ need to be made (possibly to both the objects and clumps catalogs). */
+ if(p->hostobjid_c)
+ sort_clumps_by_objid(p);
+
/* Write the filled columns into the output. */
mkcatalog_write_outputs(p);
diff --git a/bin/mkcatalog/parse.c b/bin/mkcatalog/parse.c
index e672a59..42cf27e 100644
--- a/bin/mkcatalog/parse.c
+++ b/bin/mkcatalog/parse.c
@@ -118,9 +118,9 @@ parse_objects(struct mkcatalog_passparams *pp)
int32_t *O, *OO, *C=NULL, *objects=p->objects->array;
float *std=p->std?p->std->array:NULL, *sky=p->sky?p->sky->array:NULL;
- /* If a any tile processing is necessary. */
- size_t tid = ( ( (p->sky && pp->st_sky == NULL )
- || ( p->std && pp->st_std == NULL ) )
+ /* If tile processing isn't necessary, set `tid' to a blank value. */
+ size_t tid = ( ( (p->sky && p->sky->size>1 && pp->st_sky == NULL )
+ || ( p->std && p->std->size>1 && pp->st_std == NULL ) )
? 0 : GAL_BLANK_SIZE_T );
/* Coordinate shift. */
@@ -258,13 +258,17 @@ parse_objects(struct mkcatalog_passparams *pp)
/* Sky value based measurements. */
if(p->sky)
if(oif[ OCOL_SUMSKY ])
- oi[ OCOL_SUMSKY ] += pp->st_sky ? *SK : sky[tid];
+ oi[ OCOL_SUMSKY ] += ( pp->st_sky
+ ? *SK /* Full array */
+ : ( p->sky->size>1
+ ? sky[tid] /* Tile */
+ : sky[0] ) ); /* Single value */
/* Sky standard deviation based measurements.*/
if(p->std)
{
- sval = pp->st_std ? *ST : std[tid];
+ sval = pp->st_std ? *ST : (p->std->size>1?std[tid]:std[0]);
st = p->variance ? sqrt(sval) : sval;
if(oif[ OCOL_SUMSTD ]) oi[ OCOL_SUMSTD ] += st;
/* For each pixel, we have a sky contribution to the
@@ -321,9 +325,9 @@ parse_clumps(struct mkcatalog_passparams *pp)
int32_t *objects=p->objects->array, *clumps=p->clumps->array;
float *std=p->std?p->std->array:NULL, *sky=p->sky?p->sky->array:NULL;
- /* If a any tile processing is necessary. */
- size_t tid = ( ( (p->sky && pp->st_sky == NULL )
- || ( p->std && pp->st_std == NULL ) )
+ /* If tile processing isn't necessary, set `tid' to a blank value. */
+ size_t tid = ( ( (p->sky && p->sky->size>1 && pp->st_sky == NULL )
+ || ( p->std && p->std->size>1 && pp->st_std == NULL ) )
? 0 : GAL_BLANK_SIZE_T );
/* Coordinate shift. */
@@ -432,13 +436,19 @@ parse_clumps(struct mkcatalog_passparams *pp)
/* Sky based measurements. */
if(p->sky)
if(cif[ CCOL_SUMSKY ])
- ci[ CCOL_SUMSKY ] += pp->st_sky ? *SK : sky[tid];
+ ci[ CCOL_SUMSKY ] += ( pp->st_sky
+ ? *SK /* Full */
+ : ( p->sky->size>1
+ ? sky[tid] /* Tile */
+ : sky[0] ) ); /* 1 value */
/* Sky Standard deviation based measurements, see
`parse_objects' for comments. */
if(p->std)
{
- sval = pp->st_std ? *ST : std[tid];
+ sval = ( pp->st_std
+ ? *ST
+ : (p->std->size>1 ? std[tid] : std[0]) );
st = p->variance ? sqrt(sval) : sval;
if(cif[ CCOL_SUMSTD ]) ci[ CCOL_SUMSTD ] += st;
if(cif[ CCOL_SUM_VAR ])
@@ -500,7 +510,11 @@ parse_clumps(struct mkcatalog_passparams *pp)
if(cif[ CCOL_RIV_SUM_VAR ])
{
- sval = pp->st_std ? *ST : std[tid];
+ sval = ( pp->st_std
+ ? *ST
+ : ( p->std->size>1
+ ? std[tid]
+ : std[0] ) );
cir[ CCOL_RIV_SUM_VAR ] += fabs(*V)
+ (p->variance ? sval : sval*sval);
}
diff --git a/bin/mkcatalog/ui.c b/bin/mkcatalog/ui.c
index 25da469..392125c 100644
--- a/bin/mkcatalog/ui.c
+++ b/bin/mkcatalog/ui.c
@@ -357,6 +357,10 @@ ui_check_upperlimit(struct argp_option *option, char *arg,
static void
ui_read_check_only_options(struct mkcatalogparams *p)
{
+ float tmp;
+ size_t one=1;
+ char *tailptr;
+
/* If an upper-limit check table is requested with a specific clump, but
no clump catalog has been requested, then abort and inform the
user. */
@@ -368,6 +372,38 @@ ui_read_check_only_options(struct mkcatalogparams *p)
"command calling MakeCatalog.\n\n"
"If you want the upperlimit check table for an object, only give "
"one value (the object's label) to `--checkupperlimit'.");
+
+ /* See if `--skyin' is a filename or a value. When the string is ONLY a
+ number (and nothing else), `tailptr' will point to the end of the
+ string (`\0'). */
+ if(p->skyfile)
+ {
+ tmp=strtod(p->skyfile, &tailptr);
+ if(*tailptr=='\0')
+ {
+ /* Allocate the data structure. */
+ p->sky=gal_data_alloc(NULL, GAL_TYPE_FLOAT32, 1, &one, NULL, 0, -1,
+ NULL, NULL, NULL);
+
+ /* Write the value inside it. */
+ *((float *)(p->sky->array))=tmp;
+ }
+ }
+
+ /* Similar to the case for Sky above. */
+ if(p->stdfile)
+ {
+ tmp=strtod(p->stdfile, &tailptr);
+ if(*tailptr=='\0')
+ {
+ /* Allocate the data structure. */
+ p->std=gal_data_alloc(NULL, GAL_TYPE_FLOAT32, 1, &one, NULL, 0, -1,
+ NULL, NULL, NULL);
+
+ /* Write the value inside it. */
+ *((float *)(p->std->array))=tmp;
+ }
+ }
}
@@ -512,8 +548,8 @@ ui_read_labels(struct mkcatalogparams *p)
ui_check_type_int(p->objectsfile, p->cp.hdu, p->objects->type);
- /* Conver it to `int32' type (if it already isn't). */
- gal_data_copy_to_new_type_free(p->objects, GAL_TYPE_INT32);
+ /* Convert it to `int32' type (if it already isn't). */
+ p->objects=gal_data_copy_to_new_type_free(p->objects, GAL_TYPE_INT32);
/* Currently MakeCatalog is only implemented for 2D images. */
@@ -522,7 +558,6 @@ ui_read_labels(struct mkcatalogparams *p)
"currently only supports 2D inputs", p->objectsfile, p->cp.hdu,
p->objects->ndim);
-
/* See if the total number of objects is given in the header keywords. */
keys[0].name="NUMLABS";
keys[0].type=GAL_TYPE_SIZE_T;
@@ -535,7 +570,6 @@ ui_read_labels(struct mkcatalogparams *p)
gal_data_free(tmp);
}
-
/* If there were no objects in the input, then inform the user with an
error (it is pointless to build a catalog). */
if(p->numobjects==0)
@@ -797,23 +831,24 @@ ui_preparation_check_size_read_tiles(struct
mkcatalogparams *p,
/* Subtract `sky' from the input dataset depending on its size (it may be
the whole array or a tile-values array).. */
static void
-ui_subtract_sky(gal_data_t *in, gal_data_t *sky,
- struct gal_tile_two_layer_params *tl)
+ui_subtract_sky(struct mkcatalogparams *p)
{
size_t tid;
gal_data_t *tile;
- float *f, *s, *sf, *skyarr=sky->array;
+ float *s, *f, *ff, *skyarr=p->sky->array;
+ struct gal_tile_two_layer_params *tl=&p->cp.tl;
- /* It is the same size as the input. */
- if( gal_data_dsize_is_different(in, sky)==0 )
+ /* It is the same size as the input or a single value. */
+ if( gal_data_dsize_is_different(p->values, p->sky)==0 || p->sky->size==1)
{
- f=in->array;
- sf=(s=sky->array)+sky->size;
- do *f++-=*s; while(++s<sf);
+ s=p->sky->array;
+ ff = (f=p->values->array) + p->values->size;
+ if(p->sky->size==1) { if(*s!=0.0) do *f-=*s; while(++f<ff); }
+ else do *f-=*s++; while(++f<ff);
}
/* It is the same size as the number of tiles. */
- else if( tl->tottiles==sky->size )
+ else if( tl->tottiles==p->sky->size )
{
/* Go over all the tiles. */
for(tid=0; tid<tl->tottiles; ++tid)
@@ -822,8 +857,7 @@ ui_subtract_sky(gal_data_t *in, gal_data_t *sky,
tile=&tl->tiles[tid];
/* Subtract the Sky value from the input image. */
- GAL_TILE_PO_OISET(int32_t, float, tile, in, 1, 1,
- {*o-=skyarr[tid];});
+ GAL_TILE_PARSE_OPERATE(tile, NULL, 0, 0, {*i-=skyarr[tid];});
}
}
@@ -898,53 +932,46 @@ ui_preparations_read_inputs(struct mkcatalogparams *p)
/* Read the Sky image and check its size. */
if(p->subtractsky || need_sky)
{
- /* Make sure the HDU is also given. */
- if(p->skyhdu==NULL)
- error(EXIT_FAILURE, 0, "%s: no HDU/extension provided for the "
- "SKY dataset. Atleast one column needs this dataset, or you "
- "have asked to subtract the Sky from the values.\n\n"
- "Please use the `--skyhdu' option to give a specific HDU "
- "using its number (counting from zero) or name. If the "
- "dataset is in another file, please use `--skyfile' to give "
- "the filename", p->usedskyfile);
-
- /* Read the sky image. */
- p->sky=gal_array_read_one_ch_to_type(p->usedskyfile, p->skyhdu,
- GAL_TYPE_FLOAT32,
- p->cp.minmapsize);
-
- /* Check its size. */
- ui_preparation_check_size_read_tiles(p, p->sky, p->usedskyfile,
- p->skyhdu);
-
- /* Subtract the Sky value. */
- if(p->subtractsky && need_sky==0)
+ /* If it wasn't a number, read the dataset into memory. */
+ if(p->sky==NULL)
{
- /* Subtract the Sky value. */
- ui_subtract_sky(p->values, p->sky, &p->cp.tl);
-
- /* If we don't need the Sky value, then free it here. */
- if(need_sky==0)
- {
- gal_data_free(p->sky);
- p->sky=NULL;
- }
+ /* Make sure the HDU is also given. */
+ if(p->skyhdu==NULL)
+ error(EXIT_FAILURE, 0, "%s: no HDU/extension provided for the "
+ "SKY dataset. Atleast one column needs this dataset, or "
+ "you have asked to subtract the Sky from the values.\n\n"
+ "Please use the `--skyhdu' option to give a specific HDU "
+ "using its number (counting from zero) or name. If the "
+ "dataset is in another file, please use `--skyin' to "
+ "give the filename", p->usedskyfile);
+
+ /* Read the Sky dataset. */
+ p->sky=gal_array_read_one_ch_to_type(p->usedskyfile, p->skyhdu,
+ GAL_TYPE_FLOAT32,
+ p->cp.minmapsize);
+
+ /* Check its size and prepare tile structure. */
+ ui_preparation_check_size_read_tiles(p, p->sky, p->usedskyfile,
+ p->skyhdu);
}
+
+ /* Subtract the Sky value. */
+ if(p->subtractsky) ui_subtract_sky(p);
}
- /* Read the Sky standard deviation image and check its size. */
- if(need_std)
+ /* Read the Sky standard deviation dataset (if it wasn't already given as
+ a number) and check its size. */
+ if(need_std && p->std==NULL)
{
/* Make sure the HDU is also given. */
if(p->stdhdu==NULL)
error(EXIT_FAILURE, 0, "%s: no HDU/extension provided for the "
- "SKY STANDARD DEVIATION dataset. Atleast one column needs "
- "this dataset. Please use the `--stdhdu' option to give "
- "a specific HDU using its number (counting from zero) or "
- "name. If the dataset is in another file, please use "
- "`--stdfile' to give the filename. If its actually the Sky's "
- "variance dataset, run MakeCatalog with `--variance'",
+ "SKY STANDARD DEVIATION dataset.\n\n"
+ "Atleast one column needs this dataset. Please use the "
+ "`--stdhdu' option to give a specific HDU using its number "
+ "(counting from zero) or name. If the dataset is in another "
+ "file, please use `--stdin' to give the filename",
p->usedstdfile);
/* Read the Sky standard deviation image into memory. */
@@ -952,7 +979,7 @@ ui_preparations_read_inputs(struct mkcatalogparams *p)
GAL_TYPE_FLOAT32,
p->cp.minmapsize);
- /* Check its size. */
+ /* Check its size and prepare tile structure. */
ui_preparation_check_size_read_tiles(p, p->std, p->usedstdfile,
p->stdhdu);
}
@@ -1029,7 +1056,8 @@ ui_preparations_read_keywords(struct mkcatalogparams *p)
gal_data_t *tmp;
gal_data_t *keys=NULL;
- if(p->std)
+ /* When a Sky standard deviation dataset (not number) is given. */
+ if(p->std && p->std->size>1)
{
/* Read the keywords from the standard deviation image. */
keys=gal_data_array_calloc(2);
@@ -1101,9 +1129,9 @@ ui_preparations_both_names(struct mkcatalogparams *p)
}
else
{
- /* Note that suffix is not used in the text table outputs, so it
+ /* Note that the suffix is not used in the text table outputs, so it
doesn't matter if the output table is not FITS. */
- suffix="_catalog.fits";
+ suffix="_cat.fits";
basename = p->objectsfile;
}
@@ -1328,28 +1356,36 @@ ui_preparations(struct mkcatalogparams *p)
error(EXIT_FAILURE, 0, "no columns requested, please run again with "
"`--help' for the full list of columns you can ask for");
+
/* Set the actual filenames to use. */
ui_set_filenames(p);
+
/* Read the main input (the objects image). */
ui_read_labels(p);
+
/* Prepare the output columns. */
columns_define_alloc(p);
+
/* Read the inputs. */
ui_preparations_read_inputs(p);
+
/* Read the helper keywords from the inputs and if they aren't present
then calculate the necessary parameters. */
ui_preparations_read_keywords(p);
+
/* Set the output filename(s). */
ui_preparations_outnames(p);
+
/* Make the tiles that cover each object. */
ui_one_tile_per_object(p);
+
/* Allocate the reference random number generator and seed values. It
will be cloned once for every thread. If the user hasn't called
`envseed', then we want it to be different for every run, so we need
@@ -1358,6 +1394,24 @@ ui_preparations(struct mkcatalogparams *p)
if( p->hasmag && isnan(p->zeropoint) )
error(EXIT_FAILURE, 0, "no zeropoint specified");
+
+
+ /* Prepare the two internal arrays necessary to sort the clumps catalog
+ by object and clump IDs. We are allocating and filling these in
+ separately (and not using the actual output columns that have the same
+ values), because playing with the output columns can cause bad
+ bugs. If the user wants performance, they are encouraged to run
+ MakeCatalog with `--noclumpsort' and avoid the whole process all
+ together. */
+ if(p->clumps && !p->noclumpsort && p->cp.numthreads>1)
+ {
+ p->hostobjid_c=gal_data_malloc_array(GAL_TYPE_SIZE_T,
+ p->clumpcols->size, __func__,
+ "p->hostobjid_c");
+ p->numclumps_c=gal_data_malloc_array(GAL_TYPE_SIZE_T,
+ p->objectcols->size, __func__,
+ "p->numclumps_c");
+ }
}
@@ -1385,6 +1439,7 @@ ui_preparations(struct mkcatalogparams *p)
void
ui_read_check_inputs_setup(int argc, char *argv[], struct mkcatalogparams *p)
{
+ char *tmp;
struct gal_options_common_params *cp=&p->cp;
@@ -1448,10 +1503,25 @@ ui_read_check_inputs_setup(int argc, char *argv[],
struct mkcatalogparams *p)
if(p->values)
printf(" - Values: %s (hdu: %s)\n", p->usedvaluesfile,
p->valueshdu);
+
if(p->subtractsky || p->sky)
- printf(" - Sky: %s (hdu: %s)\n", p->usedskyfile, p->skyhdu);
+ {
+ if(p->sky->size==1)
+ printf(" - Sky: %g\n", *((float *)(p->sky->array)) );
+ else
+ printf(" - Sky: %s (hdu: %s)\n", p->usedskyfile, p->skyhdu);
+ }
+
if(p->std)
- printf(" - Sky STD: %s (hdu: %s)\n", p->usedstdfile, p->stdhdu);
+ {
+ tmp = p->variance ? "VAR" : "STD";
+ if(p->std->size==1)
+ printf(" - Sky %s: %g\n", tmp, *((float *)(p->std->array)) );
+ else
+ printf(" - Sky %s: %s (hdu: %s)\n", tmp, p->usedstdfile,
+ p->stdhdu);
+ }
+
if(p->upmaskfile)
printf(" - Upper limit magnitude mask: %s (hdu: %s)\n",
p->upmaskfile, p->cp.hdu);
@@ -1526,6 +1596,8 @@ ui_free_report(struct mkcatalogparams *p, struct timeval
*t1)
free(p->valueshdu);
free(p->clumpsfile);
free(p->valuesfile);
+ free(p->hostobjid_c);
+ free(p->numclumps_c);
gal_data_free(p->sky);
gal_data_free(p->std);
gal_data_free(p->values);
diff --git a/bin/mkcatalog/ui.h b/bin/mkcatalog/ui.h
index 75b1309..c6562b8 100644
--- a/bin/mkcatalog/ui.h
+++ b/bin/mkcatalog/ui.h
@@ -56,8 +56,8 @@ enum option_keys_enum
UI_KEY_CLUMPSCAT = 'C', /* General settings. */
UI_KEY_VALUESFILE = 'v',
UI_KEY_CLUMPSFILE = 'l',
- UI_KEY_SKYFILE = 's',
- UI_KEY_STDFILE = 't',
+ UI_KEY_INSKY = 's',
+ UI_KEY_INSTD = 't',
UI_KEY_ENVSEED = 'e',
UI_KEY_IDS = 'i', /* Catalog columns. */
@@ -97,6 +97,7 @@ enum option_keys_enum
UI_KEY_UPSIGMACLIP,
UI_KEY_UPNSIGMA,
UI_KEY_CHECKUPPERLIMIT,
+ UI_KEY_NOCLUMPSORT,
UI_KEY_OBJID, /* Catalog columns. */
UI_KEY_IDINHOSTOBJ,
diff --git a/bin/noisechisel/args.h b/bin/noisechisel/args.h
index 0f3f033..2ff6901 100644
--- a/bin/noisechisel/args.h
+++ b/bin/noisechisel/args.h
@@ -149,6 +149,19 @@ struct argp_option program_options[] =
GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
+ {
+ "label",
+ UI_KEY_LABEL,
+ 0,
+ 0,
+ "Label/count detected pixels that are connected.",
+ GAL_OPTIONS_GROUP_OUTPUT,
+ &p->label,
+ GAL_OPTIONS_NO_ARG_TYPE,
+ GAL_OPTIONS_RANGE_0_OR_1,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET
+ },
diff --git a/bin/noisechisel/astnoisechisel.conf
b/bin/noisechisel/astnoisechisel.conf
index 320e74f..5126341 100644
--- a/bin/noisechisel/astnoisechisel.conf
+++ b/bin/noisechisel/astnoisechisel.conf
@@ -42,7 +42,7 @@
dthresh 0.0
snminarea 10
snquant 0.95
- detgrowquant 0.70
+ detgrowquant 0.80
detgrowmaxholesize 100
# Operating mode
diff --git a/bin/noisechisel/detection.c b/bin/noisechisel/detection.c
index 7203038..cf1f7d0 100644
--- a/bin/noisechisel/detection.c
+++ b/bin/noisechisel/detection.c
@@ -36,6 +36,7 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
#include <gnuastro/statistics.h>
#include <gnuastro-internal/timing.h>
+#include <gnuastro-internal/checkset.h>
#include "main.h"
@@ -1117,28 +1118,22 @@ detection(struct noisechiselparams *p)
/* Update the user on the progress, if necessary. */
- if(!p->cp.quiet)
+ if(!p->cp.quiet && p->detgrowquant!=1.0f && num_expanded!=GAL_BLANK_SIZE_T )
{
- if(p->detgrowquant==1.0f)
+ /* If the user hasn't asked for a labeled image, then don't confuse
+ them with the number of detections, just let them know that growth
+ is complete. */
+ if(p->label)
{
- if( asprintf(&msg, "%zu detections with no growth.",
- num_true_initial)<0 )
+ if( asprintf(&msg, "%zu detections after growth to %.3f "
+ "quantile.", num_true_initial, p->detgrowquant)<0 )
error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
}
else
{
- if(num_expanded==GAL_BLANK_SIZE_T)
- {
- if(asprintf(&msg, "%zu detections (no growth to %.3f "
- "quantile).", num_true_initial, p->detgrowquant)<0)
- error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
- }
- else
- {
- if( asprintf(&msg, "%zu detections after growth to %.3f "
- "quantile.", num_true_initial, p->detgrowquant)<0 )
- error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
- }
+ if( asprintf(&msg, "Growth to %.3f quantile complete.",
+ p->detgrowquant)<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
}
gal_timing_report(&t1, msg, 2);
free(msg);
@@ -1157,8 +1152,17 @@ detection(struct noisechiselparams *p)
: num_true_initial );
if(!p->cp.quiet)
{
- if( asprintf(&msg, "%zu final true detections.", p->numdetections)<0 )
- error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
+ /* If the user hasn't asked for a labeled image, then don't confuse
+ them with the number of detections, just let them know that growth
+ is complete. */
+ if(p->label)
+ {
+ if( asprintf(&msg, "%zu final true detections.",
+ p->numdetections)<0 )
+ error(EXIT_FAILURE, 0, "%s: asprintf allocation", __func__);
+ }
+ else
+ gal_checkset_allocate_copy("Detection complete.", &msg);
gal_timing_report(&t0, msg, 1);
free(msg);
}
diff --git a/bin/noisechisel/main.h b/bin/noisechisel/main.h
index 0cf1711..05eb729 100644
--- a/bin/noisechisel/main.h
+++ b/bin/noisechisel/main.h
@@ -53,6 +53,7 @@ struct noisechiselparams
uint8_t continueaftercheck; /* Don't abort after the check steps. */
uint8_t rawoutput; /* Only detection & 1 elem/tile output. */
+ uint8_t label; /* Label detections that are connected. */
float mirrordist; /* Maximum distance to check mode sym. */
float modmedqdiff; /* Difference between mode and median. */
diff --git a/bin/noisechisel/noisechisel.c b/bin/noisechisel/noisechisel.c
index 8f27ce9..04f6835 100644
--- a/bin/noisechisel/noisechisel.c
+++ b/bin/noisechisel/noisechisel.c
@@ -158,15 +158,24 @@ noisechisel_output(struct noisechiselparams *p)
/* Write the object labels and useful information into it's header. */
- gal_fits_key_list_add(&keys, GAL_TYPE_SIZE_T, "NUMLABS", 0,
- &p->numdetections, 0, "Total number of labels "
- "(inclusive)", 0, "counter");
gal_fits_key_list_add(&keys, GAL_TYPE_FLOAT32, "DETSN", 0, &p->detsnthresh,
0, "Minimum S/N of true pseudo-detections", 0,
"ratio");
- p->olabel->name = "DETECTIONS";
- gal_fits_img_write(p->olabel, p->cp.output, keys, PROGRAM_NAME);
- p->olabel->name=NULL;
+ if(p->label)
+ {
+ gal_fits_key_list_add(&keys, GAL_TYPE_SIZE_T, "NUMLABS", 0,
+ &p->numdetections, 0, "Total number of labels "
+ "(inclusive)", 0, "counter");
+ p->olabel->name = "DETECTIONS";
+ gal_fits_img_write(p->olabel, p->cp.output, keys, PROGRAM_NAME);
+ p->olabel->name=NULL;
+ }
+ else
+ {
+ p->binary->name = "DETECTIONS";
+ gal_fits_img_write(p->binary, p->cp.output, keys, PROGRAM_NAME);
+ p->binary->name=NULL;
+ }
keys=NULL;
@@ -192,6 +201,10 @@ noisechisel_output(struct noisechiselparams *p)
gal_tile_full_values_write(p->std, &p->cp.tl, 1, p->cp.output, keys,
PROGRAM_NAME);
p->std->name=NULL;
+
+ /* Let the user know that the output is written. */
+ if(!p->cp.quiet)
+ printf(" - Output written to `%s'.\n", p->cp.output);
}
diff --git a/bin/noisechisel/threshold.c b/bin/noisechisel/threshold.c
index 1168efd..de94e3a 100644
--- a/bin/noisechisel/threshold.c
+++ b/bin/noisechisel/threshold.c
@@ -762,18 +762,17 @@ threshold_quantile_find_apply(struct noisechiselparams *p)
"Thus don't loosen them too much. Recall that you can see all the "
"option values to Gnuastro's programs by appending `-P' to the "
"end of your command.\n\n"
- " - Decrease `--interpnumngb' to be smaller than the number "
- "mentioned above.\n"
- " - Slightly decrease `--tilesize' to have more tiles.\n"
- " - Slightly increase `--modmedqdiff' to accept more tiles.\n\n"
+ " * Decrease `--interpnumngb' to be smaller than %zu.\n"
+ " * Slightly decrease `--tilesize' to have more tiles.\n"
+ " * Slightly increase `--modmedqdiff' to accept more tiles.\n\n"
"Try appending your command with `--checkqthresh' to see the "
- "successful tiles (and get a feeling of the cause/solution, note "
- "that the output is a multi-extension FITS file). To better "
- "understand this important step, please run the following "
- "command (press `SPACE'/arrow-keys to navigate and `Q'-key to "
- "return back to the command-line):\n\n"
+ "successful tiles (and get a feeling of the cause/solution. Note "
+ "that the output is a multi-extension FITS file).\n\n"
+ "To better understand this important step, please run the "
+ "following command (press `SPACE'/arrow-keys to navigate and "
+ "`Q' to return back to the command-line):\n\n"
" $ info gnuastro \"Quantifying signal in a tile\"\n",
- nval, cp->interpnumngb);
+ nval, cp->interpnumngb, cp->interpnumngb);
/* Interpolate and smooth the derived values. */
@@ -789,7 +788,11 @@ threshold_quantile_find_apply(struct noisechiselparams *p)
/* Write the binary image if check is requested. */
if(p->qthreshname && !tl->oneelempertile)
- gal_fits_img_write(p->binary, p->qthreshname, NULL, PROGRAM_NAME);
+ {
+ p->binary->name="QTHRESH-APPLIED";
+ gal_fits_img_write(p->binary, p->qthreshname, NULL, PROGRAM_NAME);
+ p->binary->name=NULL;
+ }
/* Set the expansion quantile if necessary. */
diff --git a/bin/noisechisel/ui.c b/bin/noisechisel/ui.c
index 00fa9cb..66d8d61 100644
--- a/bin/noisechisel/ui.c
+++ b/bin/noisechisel/ui.c
@@ -347,7 +347,7 @@ ui_set_output_names(struct noisechiselparams *p)
}
else
p->cp.output=gal_checkset_automatic_output(&p->cp, p->inputname,
- "_labeled.fits");
+ "_detected.fits");
/* Tile check. */
if(p->cp.tl.checktiles)
diff --git a/bin/noisechisel/ui.h b/bin/noisechisel/ui.h
index 0722c38..0c5e7b2 100644
--- a/bin/noisechisel/ui.h
+++ b/bin/noisechisel/ui.h
@@ -50,7 +50,7 @@ enum program_args_groups
/* Available letters for short options:
- a b f g i j l n u v x y z
+ a b f g i j n u v x y z
A E G H J O W X Y
*/
enum option_keys_enum
@@ -71,6 +71,7 @@ enum option_keys_enum
UI_KEY_SNQUANT = 'c',
UI_KEY_DETGROWQUANT = 'd',
UI_KEY_CONTINUEAFTERCHECK = 'C',
+ UI_KEY_LABEL = 'l',
/* Only with long version (start with a value 1000, the rest will be set
diff --git a/bin/segment/args.h b/bin/segment/args.h
index b7a66e5..280a0cc 100644
--- a/bin/segment/args.h
+++ b/bin/segment/args.h
@@ -33,81 +33,94 @@ struct argp_option program_options[] =
{
/* Input options. */
{
- "detection",
- UI_KEY_DETECTION,
- "STR",
+ "sky",
+ UI_KEY_SKY,
+ "STR/FLT",
0,
- "Filename of detection image (to segment).",
+ "Filename of Sky values image to subtract.",
GAL_OPTIONS_GROUP_INPUT,
- &p->detectionname,
+ &p->skyname,
GAL_TYPE_STRING,
GAL_OPTIONS_RANGE_ANY,
GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
{
- "dhdu",
- UI_KEY_DHDU,
+ "skyhdu",
+ UI_KEY_SKYHDU,
"STR",
0,
- "HDU containing detection image.",
+ "HDU containing Sky value to subtract.",
GAL_OPTIONS_GROUP_INPUT,
- &p->dhdu,
+ &p->skyhdu,
GAL_TYPE_STRING,
GAL_OPTIONS_RANGE_ANY,
- GAL_OPTIONS_MANDATORY,
+ GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
{
- "sky",
- UI_KEY_SKY,
- "STR",
+ "std",
+ UI_KEY_STD,
+ "STR/FLT",
0,
- "Filename of Sky values image to subtract.",
+ "Filename of Sky standard deviation.",
GAL_OPTIONS_GROUP_INPUT,
- &p->skyname,
+ &p->stdname,
GAL_TYPE_STRING,
GAL_OPTIONS_RANGE_ANY,
GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
{
- "skyhdu",
- UI_KEY_SKYHDU,
+ "stdhdu",
+ UI_KEY_STDHDU,
"STR",
0,
- "HDU containing Sky value to subtract.",
+ "HDU containing Sky standard deviation.",
GAL_OPTIONS_GROUP_INPUT,
- &p->skyhdu,
+ &p->stdhdu,
GAL_TYPE_STRING,
GAL_OPTIONS_RANGE_ANY,
- GAL_OPTIONS_MANDATORY,
+ GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
{
- "std",
- UI_KEY_STD,
+ "variance",
+ UI_KEY_VARIANCE,
+ 0,
+ 0,
+ "STD input is actually variance.",
+ GAL_OPTIONS_GROUP_INPUT,
+ &p->variance,
+ GAL_OPTIONS_NO_ARG_TYPE,
+ GAL_OPTIONS_RANGE_0_OR_1,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET
+ },
+ {
+ "detection",
+ UI_KEY_DETECTION,
"STR",
0,
- "Filename of Sky standard deviation.",
+ "Filename of detection image (to segment).",
GAL_OPTIONS_GROUP_INPUT,
- &p->stdname,
+ &p->detectionname,
GAL_TYPE_STRING,
GAL_OPTIONS_RANGE_ANY,
GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
{
- "stdhdu",
- UI_KEY_STDHDU,
+ "dhdu",
+ UI_KEY_DHDU,
"STR",
0,
- "HDU containing Sky standard deviation.",
+ "HDU containing detection image.",
GAL_OPTIONS_GROUP_INPUT,
- &p->stdhdu,
+ &p->dhdu,
GAL_TYPE_STRING,
GAL_OPTIONS_RANGE_ANY,
- GAL_OPTIONS_MANDATORY,
+ GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
{
@@ -181,6 +194,32 @@ struct argp_option program_options[] =
GAL_OPTIONS_NOT_MANDATORY,
GAL_OPTIONS_NOT_SET
},
+ {
+ "grownclumps",
+ UI_KEY_GROWNCLUMPS,
+ 0,
+ 0,
+ "Save grown clumps instead of original.",
+ UI_GROUP_SEGMENTATION,
+ &p->grownclumps,
+ GAL_OPTIONS_NO_ARG_TYPE,
+ GAL_OPTIONS_RANGE_0_OR_1,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET
+ },
+ {
+ "continueaftercheck",
+ UI_KEY_CONTINUEAFTERCHECK,
+ 0,
+ 0,
+ "Continue processing after checks.",
+ GAL_OPTIONS_GROUP_OPERATING_MODE,
+ &p->continueaftercheck,
+ GAL_OPTIONS_NO_ARG_TYPE,
+ GAL_OPTIONS_RANGE_0_OR_1,
+ GAL_OPTIONS_NOT_MANDATORY,
+ GAL_OPTIONS_NOT_SET
+ },
@@ -342,19 +381,6 @@ struct argp_option program_options[] =
GAL_OPTIONS_NOT_SET
},
{
- "grownclumps",
- UI_KEY_GROWNCLUMPS,
- 0,
- 0,
- "Save grown clumps instead of original.",
- UI_GROUP_SEGMENTATION,
- &p->grownclumps,
- GAL_OPTIONS_NO_ARG_TYPE,
- GAL_OPTIONS_RANGE_0_OR_1,
- GAL_OPTIONS_NOT_MANDATORY,
- GAL_OPTIONS_NOT_SET
- },
- {
"checksegmentation",
UI_KEY_CHECKSEGMENTATION,
0,
@@ -373,19 +399,6 @@ struct argp_option program_options[] =
/* Operating mode options. */
- {
- "continueaftercheck",
- UI_KEY_CONTINUEAFTERCHECK,
- 0,
- 0,
- "Continue processing after checks.",
- GAL_OPTIONS_GROUP_OPERATING_MODE,
- &p->continueaftercheck,
- GAL_OPTIONS_NO_ARG_TYPE,
- GAL_OPTIONS_RANGE_0_OR_1,
- GAL_OPTIONS_NOT_MANDATORY,
- GAL_OPTIONS_NOT_SET
- },
diff --git a/bin/segment/clumps.c b/bin/segment/clumps.c
index 31a138f..90d1e36 100644
--- a/bin/segment/clumps.c
+++ b/bin/segment/clumps.c
@@ -70,8 +70,8 @@ clumps_grow_prepare_initial(struct clumps_thread_params
*cltprm)
size_t ndiffuse=0, coord[2], *dindexs;
double wcoord[2]={0.0f,0.0f}, sum=0.0f;
size_t *s, *sf, *dsize=input->dsize, ndim=input->ndim;
- float glimit, *imgss=input->array, *std=p->std->array;
int32_t *olabel=p->olabel->array, *clabel=p->clabel->array;
+ float glimit, *imgss=input->array, *std=p->std?p->std->array:NULL;
/* Find the flux weighted center (meaningful only for positive valued
@@ -107,12 +107,16 @@ clumps_grow_prepare_initial(struct clumps_thread_params
*cltprm)
coord[1] = GAL_DIMENSION_FLT_TO_INT(wcoord[1]/sum);
- /* Find the growth limit. Note that the STD image may be a full sized
- image or a tessellation. We'll check through the number of elements it
- has. */
- cltprm->std = ( p->std->size==p->input->size
- ? std[ gal_dimension_coord_to_index(ndim, dsize, coord) ]
- : std[ gal_tile_full_id_from_coord(&p->cp.tl, coord) ] );
+ /* Find the growth limit. Note that the STD may be a value, or a dataset
+ (which may be a full sized image or a tessellation). If its a dataset,
+ `stdval==NAN', and we'll check through the number of elements to see
+ what kind of dataset it is.. */
+ cltprm->std = ( p->std
+ ? ( p->std->size==p->input->size
+ ? std[gal_dimension_coord_to_index(ndim, dsize, coord)]
+ : std[gal_tile_full_id_from_coord(&p->cp.tl, coord)] )
+ : p->stdval );
+ if(p->variance) cltprm->std = sqrt(cltprm->std);
/* From the standard deviation, find the growth limit. */
@@ -235,9 +239,9 @@ clumps_get_raw_info(struct clumps_thread_params *cltprm)
double *row, *info=cltprm->info->array;
size_t nngb=gal_dimension_num_neighbors(ndim);
struct gal_tile_two_layer_params *tl=&p->cp.tl;
- float *arr=p->input->array, *std=p->std->array;
size_t *dinc=gal_dimension_increment(ndim, dsize);
int32_t lab, nlab, *ngblabs, *clabel=p->clabel->array;
+ float *arr=p->input->array, *std=p->std?p->std->array:NULL;
/* Allocate the array to keep the neighbor labels of river pixels. */
ngblabs=gal_data_malloc_array(GAL_TYPE_INT32, nngb, __func__, "ngblabs");
@@ -320,12 +324,15 @@ clumps_get_raw_info(struct clumps_thread_params *cltprm)
{
coord[0]=GAL_DIMENSION_FLT_TO_INT(row[INFO_X]/row[INFO_SFF]);
coord[1]=GAL_DIMENSION_FLT_TO_INT(row[INFO_Y]/row[INFO_SFF]);
- row[INFO_INSTD]=( p->std->size==p->input->size
- ? std[ gal_dimension_coord_to_index(ndim,
- dsize,
- coord) ]
- : std[ gal_tile_full_id_from_coord(tl,
- coord)] );
+ row[INFO_INSTD]=( p->std
+ ? ( p->std->size==p->input->size
+ ? std[gal_dimension_coord_to_index(ndim,
+ dsize, coord)]
+ : std[gal_tile_full_id_from_coord(tl,
+ coord)] )
+ : p->stdval );
+ if(p->variance) row[INFO_INSTD] = sqrt(row[INFO_INSTD]);
+
/* For a check
printf("---------\n");
printf("\t%f --> %zu\n", row[INFO_Y]/row[INFO_SFF], coord[1]);
@@ -900,11 +907,35 @@ clumps_true_find_sn_thresh(struct segmentparams *p)
if(clprm.sn[i].ndim) /* Only on tiles were an S/N was calculated. */
numsn+=clprm.sn[i].size;
if( numsn < p->minnumfalse )
- error(EXIT_FAILURE, 0, "only %zu clumps could be identified in the "
- "undetected regions. This is less than %zu (value to "
- "`--minnumfalse' option). Please either decrease this value or "
- "other options to change prior processing steps", numsn,
- p->minnumfalse);
+ error(EXIT_FAILURE, 0, "%zu usable clumps found in the undetected "
+ "regions. This is smaller than the requested minimum number of "
+ "false/reference clumps (%zu, value to the `--minnumfalse' "
+ "option).\n\n"
+ "There are several ways to address the problem. The best and most "
+ "highly recommended is to use a larger input if possible (when the "
+ "input is a crop from a larger dataset). If that is not the case, "
+ "or it doesn't solve the problem, you need to loosen the "
+ "parameters (and therefore cause more scatter/bias in the final "
+ "result). Thus don't loosen them too much. Recall that you can "
+ "see all the option values to Gnuastro's programs by appending "
+ "`-P' to the end of your command.\n\n"
+ " * Slightly decrease `--largetilesize' to have more tiles.\n"
+ " * Decrease `--minskyfrac' (currently %g) to look into more "
+ "tiles.\n"
+ " * Slightly decrease `--snminarea' (currently %zu) to "
+ "measure more clumps.\n"
+ " * If Segment already works on a dataset with similar noise "
+ "properties, you can directly pass the 'true' clump "
+ "signal-to-noise ratio found there to `--clumpsnthresh' and "
+ "avoid having to study the undetected regions any more.\n\n"
+ "Append your previous command with `--checksegmentation' to see "
+ "the steps and get a better feeling of the cause/solution. Note "
+ "that the output is a multi-extension FITS file).\n\n"
+ "To better understand the segmentation process and options, "
+ "please run the following command (press `SPACE'/arrow-keys to "
+ "navigate and `Q' to return back to the command-line):\n\n"
+ " $ info gnuastro \"Segmentation options\"\n",
+ numsn, p->minnumfalse, p->minskyfrac, p->snminarea);
/* Allocate the space to keep all the S/N values. */
diff --git a/bin/segment/main.h b/bin/segment/main.h
index 76e2335..62f3102 100644
--- a/bin/segment/main.h
+++ b/bin/segment/main.h
@@ -34,6 +34,9 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
#define PROGRAM_STRING PROGRAM_NAME" (" PACKAGE_NAME ") " PACKAGE_VERSION
+/* Macros */
+#define DETECTION_ALL "all"
+
@@ -56,6 +59,7 @@ struct segmentparams
char *skyhdu; /* Filename of Sky image. */
char *stdname; /* File name of Standard deviation image. */
char *stdhdu; /* HDU of Stanard deviation image. */
+ uint8_t variance; /* The input STD is actually variance. */
uint8_t rawoutput; /* Output only object and clump labels. */
float minskyfrac; /* Undetected area min. frac. in tile. */
@@ -64,6 +68,7 @@ struct segmentparams
size_t minnumfalse; /* Min No. of det/seg for true quantile. */
float snquant; /* Quantile of clumps in sky for true S/N.*/
uint8_t keepmaxnearriver; /* Keep clumps with a peak near a river. */
+ float clumpsnthresh; /* Clump S/N threshold. */
float gthresh; /* Multiple of STD to stop growing clumps.*/
size_t minriverlength; /* Min, len of good grown clump rivers. */
float objbordersn; /* Minimum S/N for grown clumps to be one.*/
@@ -83,12 +88,13 @@ struct segmentparams
gal_data_t *olabel; /* Object labels. */
gal_data_t *clabel; /* Clumps labels. */
gal_data_t *std; /* STD of undetected pixels, per tile. */
+ float stdval; /* Single value to use for std deviation. */
+ float skyval; /* Single value to use for Sky. */
float cpscorr; /* Counts/second correction. */
size_t numdetections; /* Number of final detections. */
size_t numclumps; /* Number of true clumps. */
size_t numobjects; /* Number of objects. */
- float clumpsnthresh; /* Clump S/N threshold. */
char *useddetectionname; /* Name of file USED for detection image. */
char *usedstdname; /* Name of file USED for sky STD image. */
diff --git a/bin/segment/segment.c b/bin/segment/segment.c
index caaca61..9fc1f5c 100644
--- a/bin/segment/segment.c
+++ b/bin/segment/segment.c
@@ -106,8 +106,7 @@ segment_initialize(struct segmentparams *p)
uint8_t *b;
float *f, minv;
gal_data_t *min;
- gal_data_t *forcc;
- int32_t *o, *c, *cf, maxlab=0;
+ int32_t *o, *c, *cf;
/* Allocate the clump labels image and the binary image. */
p->clabel=gal_data_alloc(NULL, p->olabel->type, p->olabel->ndim,
@@ -139,45 +138,25 @@ segment_initialize(struct segmentparams *p)
"Each non-zero pixel in this image must be positive (a "
"counter, counting from 1).", p->useddetectionname,
p->dhdu);
-
- /* Set the largest value. */
- if(*o>maxlab) maxlab=*o;
}
++o;
++b;
}
while(++c<cf);
- /* prepare the labeled image. */
- switch(maxlab)
- {
- /* No positive values. */
- case 0:
- error(EXIT_FAILURE, 0, "%s (hdu: %s): contains no detections "
- "(positive valued pixels)", p->useddetectionname, p->dhdu);
-
- /* This is a binary image, probably it was the result of a
- threshold. In this case, we need to separate the connected
- components. */
- case 1:
- forcc=gal_data_copy_to_new_type(p->olabel, GAL_TYPE_UINT8);
- p->numdetections=gal_binary_connected_components(forcc, &p->olabel,
- p->olabel->ndim);
- break;
-
- /* There is more than one label in the image, so just set the maximum
- number to the number of detections. */
- default:
- p->numdetections=maxlab;
- }
- /* If the minimum standard deviation is less than 1, then the units of
+ /* If the (minimum) standard deviation is less than 1, then the units of
the input are in units of counts/time. As described in the NoiseChisel
paper, we need to correct the S/N equation later. */
- min=gal_statistics_minimum(p->std);
- minv=*(float *)(min->array);
+ if(p->std)
+ {
+ min=gal_statistics_minimum(p->std);
+ minv=*(float *)(min->array);
+ gal_data_free(min);
+ }
+ else minv=p->stdval;
+ if(p->variance) minv=sqrt(minv);
p->cpscorr = minv>1 ? 1.0 : minv;
- gal_data_free(min);
}
@@ -1018,6 +997,7 @@ segment_detections(struct segmentparams *p)
void
segment_output(struct segmentparams *p)
{
+ float *f, *ff;
gal_fits_list_key_t *keys=NULL;
/* The Sky-subtracted input (if requested). */
@@ -1048,8 +1028,8 @@ segment_output(struct segmentparams *p)
keys=NULL;
- /* The Standard deviation image. */
- if(!p->rawoutput)
+ /* The Standard deviation image (if one was actually given). */
+ if( !p->rawoutput && p->std)
{
/* See if any keywords should be written (possibly inherited from the
detection program). */
@@ -1069,6 +1049,12 @@ segment_output(struct segmentparams *p)
"Median raw tile standard deviation", 0,
p->input->unit);
+ /* If the input was actually a variance dataset, we'll need to take
+ its square root before writing it. We want this output to be a
+ standard deviation dataset. */
+ if(p->variance)
+ { ff=(f=p->std->array)+p->std->size; do *f=sqrt(*f); while(++f<ff); }
+
/* Write the STD dataset into the output file. */
p->std->name="SKY_STD";
if(p->std->size == p->input->size)
@@ -1078,6 +1064,10 @@ segment_output(struct segmentparams *p)
PROGRAM_NAME);
p->std->name=NULL;
}
+
+ /* Let the user know that the output is written. */
+ if(!p->cp.quiet)
+ printf(" - Output written to `%s'.\n", p->cp.output);
}
@@ -1131,7 +1121,8 @@ segment(struct segmentparams *p)
p->olabel->name=NULL;
}
if(!p->cp.quiet)
- printf(" - Input number of detections: %zu\n", p->numdetections);
+ printf(" - Input number of connected components: %zu\n",
+ p->numdetections);
/* Find the clump S/N threshold. */
diff --git a/bin/segment/ui.c b/bin/segment/ui.c
index e8208df..0466525 100644
--- a/bin/segment/ui.c
+++ b/bin/segment/ui.c
@@ -31,8 +31,10 @@ along with Gnuastro. If not, see
<http://www.gnu.org/licenses/>.
#include <gnuastro/wcs.h>
#include <gnuastro/fits.h>
#include <gnuastro/array.h>
+#include <gnuastro/binary.h>
#include <gnuastro/threads.h>
#include <gnuastro/dimension.h>
+#include <gnuastro/statistics.h>
#include <gnuastro-internal/timing.h>
#include <gnuastro-internal/options.h>
@@ -113,6 +115,8 @@ ui_initialize_options(struct segmentparams *p,
cp->numthreads = gal_threads_number();
cp->coptions = gal_commonopts_options;
+ p->skyval = NAN;
+ p->stdval = NAN;
p->medstd = NAN;
p->minstd = NAN;
p->maxstd = NAN;
@@ -220,10 +224,47 @@ parse_opt(int key, char *arg, struct argp_state *state)
static void
ui_read_check_only_options(struct segmentparams *p)
{
+ char *tailptr;
+
+ /* If the full area is to be used as a single detection, we can't find
+ the S/N value from the un-detected regions, so the user must have
+ given the `clumpsnthresh' option. */
+ if( p->detectionname
+ && !strcmp(p->detectionname, DETECTION_ALL)
+ && isnan(p->clumpsnthresh) )
+ error(EXIT_FAILURE, 0, "`--clumpsnthresh' (`-%c') not given.\n\n"
+ "When `--detection=all' (the whole input dataset is assumed to "
+ "be a detection), Segment can't use the undetected pixels to find "
+ "the signal-to-noise ratio of true clumps. Therefore it is "
+ "mandatory to provide a signal-to-noise ratio manually",
+ UI_KEY_CLUMPSNTHRESH);
+
+ /* See if `--sky' is a filename or a value. When the string is only a
+ number (and nothing else), `tailptr' will point to the end of the
+ string (`\0'). When the string doesn't start with a number, it will
+ point to the start of the string. However, file names might also be
+ things like `1_sky.fits'. In such cases, `strtod' will return `1.0'
+ and `tailptr' will be `_std.fits', so we need to reset `p->stdval' to
+ NaN. */
+ if(p->skyname)
+ {
+ p->skyval=strtod(p->skyname, &tailptr);
+ if(tailptr==p->skyname || *tailptr!='\0')
+ p->skyval=NAN;
+ }
+
+ /* Similar to `--sky' above. */
+ if(p->stdname)
+ {
+ p->stdval=strtod(p->stdname, &tailptr);
+ if(tailptr==p->stdname || *tailptr!='\0')
+ p->stdval=NAN;
+ }
+
/* If the convolved HDU is given. */
if(p->convolvedname && p->chdu==NULL)
error(EXIT_FAILURE, 0, "no value given to `--convolvedhdu'. When the "
- "`--convolved' option is called (to specify a convolved image "
+ "`--convolved' option is called (to specify a convolved dataset "
"and avoid convolution) it is mandatory to also specify a HDU "
"for it");
@@ -308,7 +349,8 @@ ui_set_used_names(struct segmentparams *p)
p->usedstdname = ( p->stdname
? p->stdname
- : ( p->detectionname
+ : ( ( p->detectionname
+ && strcmp(p->detectionname, DETECTION_ALL) )
? p->detectionname
: p->inputname ) );
}
@@ -367,6 +409,9 @@ ui_set_output_names(struct segmentparams *p)
static void
ui_prepare_inputs(struct segmentparams *p)
{
+ int32_t *i, *ii;
+ gal_data_t *maxd, *ccin, *ccout=NULL;
+
/* Read the input as a single precision floating point dataset. */
p->input = gal_array_read_one_ch_to_type(p->inputname, p->cp.hdu,
GAL_TYPE_FLOAT32,
@@ -405,24 +450,68 @@ ui_prepare_inputs(struct segmentparams *p)
}
- /* Read the detected label image and check its type and size. */
- p->olabel = gal_array_read_one_ch(p->useddetectionname, p->dhdu,
- p->cp.minmapsize);
- if( gal_data_dsize_is_different(p->input, p->olabel) )
- error(EXIT_FAILURE, 0, "`%s' (hdu: %s) and `%s' (hdu: %s) have a"
- "different dimension/size", p->useddetectionname, p->dhdu,
- p->inputname, p->cp.hdu);
- if(p->olabel->type==GAL_TYPE_FLOAT32 || p->olabel->type==GAL_TYPE_FLOAT64)
- error(EXIT_FAILURE, 0, "%s (hdu: %s) has a `%s' type. The detection "
- "(labeled) map must have an integer type (labels/classes can only "
- "be integers). If the pixel values are integers, but only the "
- "numerical type of the image is floating-point, you can use the "
- "command below to convert it to a 32-bit (signed) integer type:\n\n"
- " $ astarithmetic %s int32 -h%s\n\n", p->useddetectionname,
- p->dhdu, gal_type_name(p->olabel->type, 1), p->useddetectionname,
- p->dhdu);
- p->olabel = gal_data_copy_to_new_type_free(p->olabel, GAL_TYPE_INT32);
- p->olabel->wcs=gal_wcs_copy(p->input->wcs);
+ /* Read the detected label image and check its size. When the user gives
+ `--detection=all', then the whole input is assumed to be a single
+ detection. */
+ if( strcmp(p->useddetectionname, DETECTION_ALL) )
+ {
+ /* Read the dataset into memory. */
+ p->olabel = gal_array_read_one_ch(p->useddetectionname, p->dhdu,
+ p->cp.minmapsize);
+ if( gal_data_dsize_is_different(p->input, p->olabel) )
+ error(EXIT_FAILURE, 0, "`%s' (hdu: %s) and `%s' (hdu: %s) have a"
+ "different dimension/size", p->useddetectionname, p->dhdu,
+ p->inputname, p->cp.hdu);
+
+ /* Make sure the detected labels are not floating point. */
+ if(p->olabel->type==GAL_TYPE_FLOAT32
+ || p->olabel->type==GAL_TYPE_FLOAT64)
+ error(EXIT_FAILURE, 0, "%s (hdu: %s) has a `%s' type. The detection "
+ "(labeled) map must have an integer type (labels/classes can "
+ "only be integers). If the pixel values are integers, but only "
+ "the numerical type of the image is floating-point, you can "
+ "use the command below to convert it to a 32-bit (signed) "
+ "integer type:\n\n"
+ " $ astarithmetic %s int32 -h%s\n\n", p->useddetectionname,
+ p->dhdu, gal_type_name(p->olabel->type, 1),
+ p->useddetectionname, p->dhdu);
+
+ /* Get the maximum value of the input (total number of labels if they
+ are separate). If the maximum is 1 (the image is a binary image),
+ then apply the connected components algorithm to separate the
+ connected regions. The user is allowed to supply a simple binary
+ image.*/
+ maxd=gal_statistics_maximum(p->olabel);
+ maxd=gal_data_copy_to_new_type_free(maxd, GAL_TYPE_INT64);
+ p->numdetections = *((uint64_t *)(maxd->array));
+ if( p->numdetections == 1 )
+ {
+ ccin=gal_data_copy_to_new_type_free(p->olabel, GAL_TYPE_UINT8);
+ p->numdetections=gal_binary_connected_components(ccin, &ccout,
+ ccin->ndim);
+ gal_data_free(ccin);
+ p->olabel=ccout;
+ }
+ else
+ p->olabel = gal_data_copy_to_new_type_free(p->olabel, GAL_TYPE_INT32);
+
+
+ /* Write the WCS into the objects dataset too. */
+ p->olabel->wcs=gal_wcs_copy(p->input->wcs);
+ }
+ else
+ {
+ /* Set the total number of detections to 1. */
+ p->numdetections=1;
+
+ /* Allocate the array. */
+ p->olabel=gal_data_alloc(NULL, GAL_TYPE_INT32, p->input->ndim,
+ p->input->dsize, p->input->wcs, 0,
+ p->cp.minmapsize, NULL, NULL, NULL);
+
+ /* Initialize it to 1. */
+ ii=(i=p->olabel->array)+p->olabel->size; do *i++=1; while(i<ii);
+ }
}
@@ -562,18 +651,19 @@ ui_check_size(gal_data_t *base, gal_data_t *comp, size_t
numtiles,
the whole array or a tile-values array).. */
static void
ui_subtract_sky(gal_data_t *in, gal_data_t *sky,
- struct gal_tile_two_layer_params *tl)
+ struct gal_tile_two_layer_params *tl)
{
size_t tid;
gal_data_t *tile;
- float *f, *s, *sf, *skyarr=sky->array;
+ float *s, *f, *ff, *skyarr=sky->array;
- /* It is the same size as the input. */
- if( gal_data_dsize_is_different(in, sky)==0 )
+ /* It is the same size as the input or a single value. */
+ if( gal_data_dsize_is_different(in, sky)==0 || sky->size==1)
{
- f=in->array;
- sf=(s=sky->array)+sky->size;
- do *f++-=*s; while(++s<sf);
+ s=sky->array;
+ ff=(f=in->array)+in->size;
+ if(sky->size==1) { if(*s!=0.0) do *f-=*s; while(++f<ff); }
+ else do *f-=*s++; while(++f<ff);
}
/* It is the same size as the number of tiles. */
@@ -606,29 +696,67 @@ ui_subtract_sky(gal_data_t *in, gal_data_t *sky,
image (only one element/pixel for a tile). So we need to do some extra
checks on them (after reading the tessellation). */
static void
-ui_read_std(struct segmentparams *p)
+ui_read_std_and_sky(struct segmentparams *p)
{
+ size_t one=1;
struct gal_tile_two_layer_params *tl=&p->cp.tl;
gal_data_t *sky, *keys=gal_data_array_calloc(3);
- /* The standard devitaion image. */
- p->std=gal_array_read_one_ch_to_type(p->usedstdname, p->stdhdu,
- GAL_TYPE_FLOAT32, p->cp.minmapsize);
- ui_check_size(p->input, p->std, tl->tottiles, p->inputname, p->cp.hdu,
- p->usedstdname, p->stdhdu);
+ /* Read the standard devitaion image (if it isn't a single number). */
+ if(isnan(p->stdval))
+ {
+ /* Make sure a HDU is also given. */
+ if(p->stdhdu==NULL)
+ error(EXIT_FAILURE, 0, "no value given to `--stdhdu'.\n\n"
+ "When the Sky standard deviation is a dataset, it is mandatory "
+ "specify which HDU/extension it is present in. The file can "
+ "be specified explicitly with `--std'. If not, segment will "
+ "use the file given to `--detection'. If that is also not "
+ "called, it will look into the main input file (with no "
+ "option)");
+
+ /* Read the STD image. */
+ p->std=gal_array_read_one_ch_to_type(p->usedstdname, p->stdhdu,
+ GAL_TYPE_FLOAT32,
+ p->cp.minmapsize);
+
+ /* Make sure it has the correct size. */
+ ui_check_size(p->input, p->std, tl->tottiles, p->inputname, p->cp.hdu,
+ p->usedstdname, p->stdhdu);
+ }
/* If a Sky image is given, subtract it from the values and convolved
- images, then free it. */
+ images immediately and free it. */
if(p->skyname)
{
- /* Read the Sky dataset. */
- sky=gal_array_read_one_ch_to_type(p->skyname, p->skyhdu,
- GAL_TYPE_FLOAT32, p->cp.minmapsize);
-
- /* Check its size. */
- ui_check_size(p->input, sky, tl->tottiles, p->inputname, p->cp.hdu,
- p->skyname, p->skyhdu);
+ /* If its a file, read it into memory. */
+ if( isnan(p->skyval) )
+ {
+ /* Make sure a HDU is also given. */
+ if(p->skyhdu==NULL)
+ error(EXIT_FAILURE, 0, "no value given to `--skyhdu'.\n\n"
+ "When the Sky is a dataset, it is mandatory specify "
+ "which HDU/extension it is present in. The file can be "
+ "specified explicitly with `--sky'. If it is a single "
+ "value, you can just pass the value to `--sky' and no "
+ "HDU will be necessary");
+
+ /* Read the Sky dataset. */
+ sky=gal_array_read_one_ch_to_type(p->skyname, p->skyhdu,
+ GAL_TYPE_FLOAT32,
+ p->cp.minmapsize);
+
+ /* Check its size. */
+ ui_check_size(p->input, sky, tl->tottiles, p->inputname, p->cp.hdu,
+ p->skyname, p->skyhdu);
+ }
+ else
+ {
+ sky=gal_data_alloc(NULL, GAL_TYPE_FLOAT32, 1, &one, NULL, 0, -1,
+ NULL, NULL, NULL);
+ *((float *)(sky->array)) = p->skyval;
+ }
/* Subtract it from the input. */
ui_subtract_sky(p->input, sky, tl);
@@ -639,6 +767,7 @@ ui_read_std(struct segmentparams *p)
/* Clean up. */
gal_data_free(sky);
+ p->skyval=NAN;
}
@@ -650,7 +779,7 @@ ui_read_std(struct segmentparams *p)
more accurate estimate of the noise level. So if they are present, we
will read them here and write them to the STD output (which is created
when `--rawoutput' is not given). */
- if(!p->rawoutput)
+ if(!p->rawoutput && p->std)
{
keys[0].next=&keys[1];
keys[1].next=&keys[2];
@@ -694,7 +823,7 @@ ui_preparations(struct segmentparams *p)
ui_prepare_tiles(p);
/* Prepare the (optional Sky, and) Sky Standard deviation image. */
- ui_read_std(p);
+ ui_read_std_and_sky(p);
}
@@ -721,6 +850,7 @@ ui_preparations(struct segmentparams *p)
void
ui_read_check_inputs_setup(int argc, char *argv[], struct segmentparams *p)
{
+ char *stdunit;
struct gal_options_common_params *cp=&p->cp;
@@ -773,12 +903,30 @@ ui_read_check_inputs_setup(int argc, char *argv[], struct
segmentparams *p)
/* Let the user know that processing has started. */
if(!p->cp.quiet)
{
+ /* Basic inputs. */
printf(PROGRAM_NAME" started on %s", ctime(&p->rawtime));
printf(" - Using %zu CPU thread%s\n", p->cp.numthreads,
p->cp.numthreads==1 ? "." : "s.");
printf(" - Input: %s (hdu: %s)\n", p->inputname, p->cp.hdu);
+
+ /* Sky value information. */
if(p->skyname)
- printf(" - Sky: %s (hdu: %s)\n", p->skyname, p->skyhdu);
+ {
+ if( isnan(p->skyval) )
+ printf(" - Sky: %s (hdu: %s)\n", p->skyname, p->skyhdu);
+ else
+ printf(" - Sky: %g\n", p->skyval);
+ }
+
+ /* Sky Standard deviation information. */
+ stdunit = p->variance ? "variance" : "STD";
+ if(p->std)
+ printf(" - Sky %s: %s (hdu: %s)\n", stdunit, p->usedstdname,
+ p->stdhdu);
+ else
+ printf(" - Sky %s: %g\n", stdunit, p->stdval);
+
+ /* Convolution information. */
if(p->convolvedname)
printf(" - Convolved input: %s (hdu: %s)\n", p->convolvedname,
p->chdu);
@@ -795,7 +943,6 @@ ui_read_check_inputs_setup(int argc, char *argv[], struct
segmentparams *p)
printf(" - Kernel: FWHM=2 pixel Gaussian.\n");
}
printf(" - Detection: %s (hdu: %s)\n", p->useddetectionname, p->dhdu);
- printf(" - Sky STD: %s (hdu: %s)\n", p->usedstdname, p->stdhdu);
}
}
diff --git a/bin/segment/ui.h b/bin/segment/ui.h
index 01a67ee..79a2cc5 100644
--- a/bin/segment/ui.h
+++ b/bin/segment/ui.h
@@ -78,6 +78,7 @@ enum option_keys_enum
UI_KEY_SKYHDU,
UI_KEY_STD,
UI_KEY_STDHDU,
+ UI_KEY_VARIANCE,
UI_KEY_RAWOUTPUT,
UI_KEY_MINNUMFALSE,
UI_KEY_GROWNCLUMPS,
diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index 55036c7..094ea0d 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -452,9 +452,9 @@ NoiseChisel
Invoking NoiseChisel
-* General NoiseChisel options:: General NoiseChisel preprocessing.
+* NoiseChisel input:: NoiseChisel's input options.
* Detection options:: Configure detection in NoiseChisel.
-* NoiseChisel output:: NoiseChisel's output format.
+* NoiseChisel output:: NoiseChisel's output options and format.
Segment
@@ -462,7 +462,7 @@ Segment
Invoking Segment
-* Segment inputs and general settings:: Input files and general options.
+* Segment input:: Input files and options.
* Segmentation options:: Parameters of the segmentation process.
* Segment output:: Outputs of Segment
@@ -2718,8 +2718,7 @@ shown. GNU AWK (the most common implementation) comes
with a free and
wonderful @url{https://www.gnu.org/software/gawk/manual/, book} in the same
format as this book which will allow you to master it nicely. Just like
this manual, you can also access GNU AWK's manual on the command-line
-whenever necessary without taking your hands off the keyboard as described
-above.
+whenever necessary without taking your hands off the keyboard.
@end cartouche
This takes us to the second question that you have probably asked yourself
@@ -3085,324 +3084,388 @@ $ rm *.fits
@end example
@noindent
-To detect the objects in the image, we'll run NoiseChisel:
+To detect the signal in the image (separate interesting pixels from noise),
+we'll run NoiseChisel (@ref{NoiseChisel}):
@example
-$ mkdir noisechisel
-$ astnoisechisel flat-ir/xdf-f160w.fits -onoisechisel/xdf-f160w.fits
+$ astnoisechisel flat-ir/xdf-f160w.fits --output=nc-test.fits
@end example
-Read what NoiseChisel writes on the command-line. The curious thing you
-will notice is that while there were more than 3000 pseudo detections to
-find the pseudo-detection S/N, but there were only slightly more than 100
-clumps to find the false clump S/N. We will see what caused this after a
-short review on the output of NoiseChisel.
-
NoiseChisel's output is a single file containing multiple extensions. You
can get basic information about the extensions in a FITS file with
-Gnuastro's Fits program (see @ref{Fits}) as shown below. It contains 6
-extensions and the first (counted as zero) is blank (has no data).
+Gnuastro's Fits program (see @ref{Fits}):
@example
-$ astfits noisechisel/xdf-f160w.fits
+$ astfits nc-test.fits
@end example
-NoiseChisel puts some general information on its outputs in the FITS header
-of the respective extension. To see the full list of keywords, you can
-again use the Fits program like above, but also give it your desired
-extension/HDU. You can also give the extension number (as listed in the
-output above), for example @option{-h2} instead of @option{-hOBJECTS}.
+From the output list, you see NoiseChisel's output contains 5 extensions
+and the first (counted as zero) is blank: it has value of @code{0} in the
+last/size column, showing that it contains no data, it just contains
+meta-data. All of Gnuastro's programs follow this convention of writing no
+data in the first HDU/extension. This allows the first extension to keep
+meta-data about all the extensions and is recommended by the FITS standard,
+see @ref{Fits}.
+
+The name of each extension describes its contents: the first
+(@code{INPUT-NO-SKY}) is the Sky-subtracted input that you provided. The
+second (NoiseChisel's main output, @code{DETECTIONS}) has a numeric data
+type of @code{uint8} with only two possible values for all pixels: 0 for
+noise and 1 for signal. The third and fourth (called @code{SKY} and
address@hidden), have the Sky and its standard deviation values of the
+input on a tessellation and were calculated over the un-detected regions.
+
address@hidden DS9
address@hidden GNOME
address@hidden SAO DS9
+To better understand NoiseChisel's output and the extensions described
+above, let's visually inspect it (here, we'll use SAO DS9). Since the file
+contains multiple related extensions, the easiest way to view all of them
+in DS9 is to open it as a ``Multi-extension data cube'' with the
address@hidden option as shown address@hidden can configure your
+graphic user interface to open DS9 in multi-extension cube mode by default
+when using the GUI (double clicking on the file). If your graphic user
+interface is GNOME (another GNU software, it is most common in GNU/Linux
+operating systems), a full description is given in @ref{Viewing
+multiextension FITS images}}.
@example
-$ astfits noisechisel/xdf-f160w.fits -hOBJECTS
+$ ds9 -mecube nc-test.fits -zscale -zoom to fit
@end example
address@hidden GNU Grep
-The @code{NUMLABS} keyword in NoiseChisel's @code{OBJECTS} extension
-contains the number of objects that was found by NoiseChisel in that
-run. Try to visually find it in the header keywords you saw above.
+The buttons and horizontal scroll bar in the ``cube'' window can be used to
+navigate between the extensions. In this mode, all DS9's settings (for
+example zoom or color-bar) will be identical between the extensions. Try
+zooming into to one part and seeing how the galaxies were detected along
+with the Sky and Sky standard deviation values. Just have in mind that
+NoiseChisel's job is @emph{only} detection (separating signal from noise),
+We'll segment this result later.
-To simplify the process, you can pipe the output of the command above into
address@hidden (a program for matching lines which is available on almost all
-Unix-like operating systems).
+One good way to see if you have missed any signal is to mask all the
+detected pixels and inspect the noise pixels. For this, you can use
+Gnuastro's Arithmetic program (in particular its @code{where} operator, see
address@hidden operators}). With the command below, all detected pixels
+(in the @code{DETECTIONS} extension) will be set to NaN in the output
+(@file{nc-masked.fits}). To make the command easier to read/write, let's
+just put the file name in a shell variable (@code{img}) first. A shell
+variable's value can be retrieved by adding a @code{$} before its name.
@example
-$ astfits noisechisel/xdf-f160w.fits -hOBJECTS | grep NUMLABS
+$ img=nc-test.fits
+$ astarithmetic $img $img nan where -hINPUT-NO-SKY -hDETECTIONS \
+ --output=nc-masked.fits
@end example
address@hidden GNU Grep
-If you just want the value of the keyword and not the full FITS keyword
-line, you can use AWK. In the example below, AWK will print the third word
-(separated by white space characters) in any line that has a first column
-value of @code{NUMLABS}. You can also try this for the third HDU (called
address@hidden) to see the number of clumps.
address@hidden
+To invert the result (only keep the values of detected pixels), you can
+flip the detected pixel values (from 0 to 1 and vice-versa) by adding a
address@hidden after the second @code{$img}:
@example
-$ astfits noisechisel/xdf-f160w.fits -h2 \
- | awk '$1=="NUMLABS" @{print address@hidden'
+$ astarithmetic $img $img not nan where -hINPUT-NO-SKY -hDETECTIONS \
+ --output=nc-masked.fits
@end example
-Grep and AWK are simple, but very powerful command-line software for
-processing text files. Learning them properly can greatly simplify your
-processing, while improving your creativity, productivity and speed. When
-you get time, it is highly recommended to master them. The most common
-implementation of both is from GNU. Like almost all GNU software, both GNU
-Grep and GNU AWK have wonderful manuals which come with the program for
-free. You don't have to read the whole manual at once, they both start with
-great generic introductions to get you going fast. As described above, you
-can read both manuals or refresh your memory on your command-line with
-these commands:
+NoiseChisel can produce ``Check images'' to help you visualize and inspect
+how each step is completed. You can see all the check images it can produce
+with this command.
@example
-$ info awk
-$ info grep
+$ astnoisechisel --help | grep check
@end example
address@hidden GNOME
-You can now open NoiseChisel's output with SAO DS9 and visually inspect
-it. Just note that since there are multiple extensions, the easiest way to
-view the whole file is to open it as a ``Multi-extension data cube'' with
-the @option{-mecube} option as shown below. If you use GNOME (another GNU
-software, most common graphic user interface in GNU/Linux operating
-systems), please see @ref{Viewing multiextension FITS images} to open DS9
-in multi-extension cube mode by default when using the GUI (double clicking
-on the file).
+Let's see how the quantile threshold (the first step after convolution) has
+been found and applied in our previous run of NoiseChisel:
@example
-$ ds9 -mecube noisechisel/xdf-f160w.fits -zscale -zoom to fit
+$ astnoisechisel flat-ir/xdf-f160w.fits --checkqthresh
@end example
-Using Gnuastro's Fits program, you can also copy a HDU from one FITS file
-to another (for more, see @ref{HDU manipulation}). So if you want to have a
-FITS file with only the detected objects, you can run this command:
+The check images/tables are also multi-extension FITS files. As you see,
+when check datasets are requested, NoiseChisel won't go to the end and
+abort as soon as all its extensions are ready. Try listing the extensions
+with @command{astfits} and then opening them with @command{ds9} as we done
+above. It is @emph{strongly} encouraged to play with the different
+parameters and use the respective check images to see which step is
+affected by your change.
+
+One major factor determining NoiseChisel's the quantile threshold is
+NoiseChisel's ability to identify signal in a tile (the threshold is only
+found on those tiles with no major signal). Therefore the larger the tiles
+are, number-statistics will cause less scatter, therefore helping
+NoiseChisel find the quantile threshold. However, if the tiles are too
+large, there might not be enough over the dataset or they won't be able to
+deal with possible gradients. Let's see what the default (small) tile size
+was with the following command.
@example
-$ astfits noisechisel/xdf-f160w.fits --copy=OBJECTS -oobjects.fits
+$ astnoisechisel -P | grep tilesize
@end example
-One good way to see if you have missed any signal is to mask all the
-detected pixels and inspect the noise pixels. For this, you can use
-Gnuastro's Arithmetic program (in particular its @code{where} operator as
-shown below, see @ref{Arithmetic operators}). With this command, all input
-pixels that have a value larger than zero in the @code{OBJECTS} extension
-will be set to NaN in the output (written in @file{det-masked.fits}). If
-you change the @code{gt} (for ``greater than'') operator to @code{eq} (for
-``equal''), all the un-detected (sky) pixels will be masked and you can see
-the detections.
+Its a 50 by 50 box. Flip back and forth between the @code{CONVOLVED} and
address@hidden extensions of the check image to get a feeling of
+which tiles succeeded (have non-blank values)@footnote{The other extensions
+don't matter much for the check here. They show the qthresh values for the
+other thresholds (on the same tiles), followed by the interpolated values
+for all the thresholds and afterwards the smoothed values that are used for
+the next steps. The final extension (@code{QTHRESH-APPLIED}, shows the
+result of applying the erode and no-erode thresholds.}. Since this is a
+relatively large image and we don't have any gradients, let's increase the
+tile size to 100 by 100
@example
-$ astarithmetic noisechisel/xdf-f160w.fits \
- noisechisel/xdf-f160w.fits 0 gt nan where -h1 -h2 \
- --output=nc-masked.fits
+$ astnoisechisel flat-ir/xdf-f160w.fits --tilesize=100,100 \
+ --checkqthresh
@end example
-In some cases, you might want to use a different kernel with NoiseChisel
-(not the default one). To do that, you can use MakeProfiles (see
address@hidden) in the following manner to build a 2D Gaussian kernel
-with a FWHM of 3 pixels that extends 5 times the FWHM. This new kernel can
-later be fed into NoiseChisel with the @option{--kernel} option.
+Inspecting the check image, you see that there are now only a handful of
+useful tiles in the central parts. This shows the field is too crowded, and
+we should slightly decrease the tile size for a more robust result that
+also covers more of the dataset. Let's set it to a 75 by 75 pixel box:
@example
-$ astmkprof --kernel=gaussian,3,5 --oversample=1 -okernel-g-3-5.fits
-$ astnoisechisel flat-ir/xdf-f160w.fits --kernel=kernel-g-3-5.fits \
- --output=nc-my-kernel.fits
+$ astnoisechisel flat-ir/xdf-f160w.fits --tilesize=75,75 \
+ --checkqthresh
@end example
-NoiseChisel can produce ``Check images'' to help you visualize how each
-step is completed. You can see all the check images it can produce with
-this command.
+The result seems reasonable now: we have a larger tile size than the
+default value, but less scatter, @emph{and} the tiles cover a sufficiently
+wide area of the dataset. So, we'll use this tile size for the next
+steps. But first, let's clean all the temporary files and make a directory
+for the NoiseChisel outputs:
@example
-$ astnoisechisel --help | grep check
+$ rm *.fits
+$ mkdir nc
+$ astnoisechisel flat-ir/xdf-f160w.fits --tilesize=75,75 \
+ --output=nc/xdf-f160w.fits
+$ astnoisechisel flat-ir/xdf-f105w.fits --tilesize=75,75 \
+ --output=nc/xdf-f105w.fits
@end example
-The check images are also multi-extension FITS files. After each check
-image is produced, open it with ds9 like NoiseChisel's output above and
-flip through the extensions to see each processing in detail. It is
address@hidden encouraged to play with the different parameters and use
-the respective check images to see which step is affected by your
-change. The three most useful check images are @option{--checkqthresh},
address@hidden, and @option{--checksegmentation}.
-
-We can now get back to the curious situation we noticed after running
-NoiseChisel: the number of false clumps to find an S/N limit was very small
-(given the extent of this image). This is bad, because we use quantiles in
-NoiseChisel and such a small number will result in a relatively large
-scatter. Since this is a segmentation issue, let's see why this happens with
address@hidden
+Before continuing with the higher-level processing of this dataset, we'll
+pause to use NoiseChisel's multi-extension output for showing how the Fits
+program can make it very easy to work with this wonderful data container
+(see @ref{Fits}). Let's say you need to copy a HDU/extension from one FITS
+file to another. Try the command below to make an @file{objects.fits} file
+that contains only NoiseChisel's binary detection map. There are similar
+options to conveniently cut or delete HDUs from a FITS file also.
@example
-$ astnoisechisel flat-ir/xdf-f160w.fits --checksegmentation
+$ astfits nc/xdf-f160w.fits --copy=DETECTIONS -odetections.fits
@end example
-To help you get a result faster, when check images are requested,
-NoiseChisel doesn't finish its processing (unless you also call
address@hidden). NoiseChisel aborts with an explanation of
-why it stopped without finishing and the file name of the check image that
-it produced. The first five extensions are: the input image, the convolved
-image, the initially labeled detected regions, all the sky region (false)
-clumps, and those that will be used for S/N. The sky clumps are found over
-NoiseChisel's ``large tiles'' independently. When inspecting the fourth
-extension of the check image, it is interesting how NoiseChisel has ignored
-most large tiles and only used the few that we see, mostly on the edge.
-
-The reason that almost all internal large tiles are ignored is that
-galaxies are extended and this is a very deep (and small) image. Thanks to
-the PSF (see @ref{PSF}), no object will have a sharp truncation. We have
-not seen any abrupt break in the light profile of any galaxy: galaxies are
-always larger when we get deeper datasets. Therefore, in a noisy image,
-some light will always be left un-detected. To be less affected by this
-un-detected light, NoiseChisel has the @option{--minskyfrac} option (see
address@hidden NoiseChisel options}): any tile that has a larger fraction of
-detected pixels will be ignored. So let's see what the default value of
-this option is (recall that with @option{-P}, you can list all the options
-with their values in Gnuastro's programs):
+NoiseChisel puts some general information on its outputs in the FITS header
+of the respective extension. To see the full list of keywords in an
+extension, you can again use the Fits program like above. But instead of
+HDU manipulation options, give it the HDU you are interested in with
address@hidden You can also give the HDU number (as listed in the output
+above), for example @option{-h2} instead of @option{-hDETECTIONS}.
@example
-$ astnoisechisel -P | grep minskyfrac
+$ astfits nc/xdf-f160w.fits -hDETECTIONS
@end example
address@hidden
-Try decreasing this value and re-running NoiseChisel to see the effect on
-the fraction of large tiles that will be used for finding false/sky
-clumps. Play a little with this parameter (give it different values and
-check the result).
address@hidden GNU Grep
+The @code{DETSN} keyword in NoiseChisel's @code{DETECTIONS} extension
+contains the true pseudo-detection signal-to-noise ratio that was found by
+NoiseChisel on the dataset. It is not easy to find it in the middle of all
+the other keywords printed by the command above (especially in files that
+have many more keywords). To fix the problem, you can pipe the output of
+the command above into @code{grep} (a program for matching lines which is
+available on almost all Unix-like operating systems).
@example
-$ astnoisechisel flat-ir/xdf-f160w.fits --minskyfrac=0.5 \
- --checksegmentation
+$ astfits nc/xdf-f160w.fits -hDETECTIONS | grep DETSN
@end example
-The smaller the value to @option{--minskyfrac}, the more probable that
-un-detected light in the wings of galaxies will bias/affect the derived
-false clump S/N. So it is always important to find a good balance between a
-larger @option{--minskyfrac} while having a sufficient number of resulting
-clumps (to avoid scatter).
-
-NoiseChisel doesn't just find the labeled pixels, it also finds the Sky
-value and the Sky standard deviation in the final two extensions of its
-output. To generate a catalog of the colors, we will be using the
-NoiseChisel labeled image from the F160W image. But the Sky and Sky
-standard deviation values for each different filter will also be
-necessary. So we'll run NoiseChisel with our finalized parameters value on
-both filters (you can also put this option's value in a configuration file
-to avoid repeating it).
address@hidden GNU Grep
+If you just want the value of the keyword and not the full FITS keyword
+line, you can use AWK. In the example below, AWK will print the third word
+(separated by white space characters) in any line that has a first column
+value of @code{DETSN}.
@example
-$ astnoisechisel flat-ir/xdf-f105w.fits -onoisechisel/xdf-f105w.fits \
- --minskyfrac=0.5
-$ astnoisechisel flat-ir/xdf-f160w.fits -onoisechisel/xdf-f160w.fits \
- --minskyfrac=0.5
+$ astfits nc/xdf-f160w.fits -h2 | awk '$1=="DETSN" @{print address@hidden'
@end example
-Now, we are ready to make a catalog. We want the object and clump labels
-from the F160W image. But the input, Sky and Sky standard deviation images
-should come from each filter. So, we'll run MakeCatalog on NoiseChisel's
-output differently for each filter. When making the F105W catalog, we'll
-use the @option{--objectsfile} and @option{--clumpsfile} options to tell
-MakeCatalog to read the object and clump labels from the F160W NoiseChisel
-output. When these options aren't given, MakeCatalog will look into the
-same input file for object and clump labels.
-
-For both filters, we'll ask for the ID, RA, Dec, Magnitude and
-signal-to-noise ratio (see @ref{Quantifying measurement limits}). To see a
-list of all the parameters that MakeCatalog can measure for you, run it
-with @option{--help} option.
+The main output of NoiseChisel is the binary detection map
+(@code{DETECTIONS} extension), which only has two values of 1 or 0. This is
+useful when studying the noise, but hardly of any use when you actually
+want to study the targets/galaxies in the image, especially in such a deep
+field where the detection map of almost everything is connected. To find
+the galaxies over the detections, we'll use Gnuastro's @ref{Segment}
+program:
@example
-$ mkdir catalog
-
-$ astmkcatalog noisechisel/xdf-f160w.fits --zeropoint=25.94 \
- --ids --ra --dec --magnitude --sn \
- --output=catalog/xdf-f160w.fits
-
-$ astmkcatalog noisechisel/xdf-f105w.fits --zeropoint=26.27 \
- --objectsfile=noisechisel/xdf-f160w.fits \
- --clumpsfile=noisechisel/xdf-f160w.fits \
- --ids --ra --dec --magnitude --sn \
- --output=catalog/xdf-f105w.fits
+$ rm *.fits
+$ mkdir segment
+$ astsegment nc/xdf-f160w.fits -osegment/xdf-f160w.fits
address@hidden example
+
+Segment's operation is very much like NoiseChisel (in fact, prior to
+version 0.6, it was part of NoiseChisel), for example the output is a
+multi-extension FITS file, it has check images and uses the signal-to-noise
+ratio of undetected regions as a reference. Please have a look at Segment's
+multi-extension output with @command{ds9} to get a good feeling of what it
+has done. Like NoiseChisel, the first extension is the input. The
address@hidden extension shows the true ``clumps'' with values that are
address@hidden, and the diffuse regions labeled as @mymath{-1}. In the
address@hidden extension, we see that the large detections of NoiseChisel
+(that may have contained many galaxies) are now broken up into separate
+labels. see @ref{Segment} for more.
+
+Having localized the regions of interest in the dataset, we are ready to do
+measurements on them with @ref{MakeCatalog}. Besides the IDs, we want to
+measure the Right Ascension (with @option{--ra}), Declination
+(@option{--dec}, magnitude (@option{--magnitude} and signal-to-noise ratio
+(@option{--sn}) of the objects and clumps. The following command will make
+these measurements on Segment's F160W output:
+
address@hidden Keep the `--zeropoint' on a single line, because later, we'll add
address@hidden `--valuesfile' in that line also, and it would be more clear if
both
address@hidden catalogs follow the same format.
address@hidden
+$ mkdir cat
+$ astmkcatalog seg/xdf-f160w.fits --ids --ra --dec --magnitude --sn \
+ --zeropoint=25.94 \
+ --clumpscat --output=cat/xdf-f160w.fits
@end example
-MakeCatalog can also produce catalogs in plain text format. Please run the
-MakeCatalog commands above again and replace the @file{.fits} suffix, in
-the value to @option{--output}, with @file{.txt} (we will also be using the
-text file outputs later).
-
-When a clumps image is also address@hidden will look at the
address@hidden keyword in the objects image to see if it should also use a
-clumps image or not.}, like this example, two catalogs will be made. If you
-asked for a plain text file output, two files will be made with the
address@hidden and @file{_o.txt} suffixes. If MakeCatalog's output is
-requested to be FITS, the two catalogs will be in separate extensions of a
-single file. You can inspect the separate extensions with the Fits program
-like before (as shown below). Afterwards, you can inspect the table in each
-extension with Gnuastro's Table program (see @ref{Table}) as shown below.
address@hidden
+From the printed statements on the command-line, you see that MakeCatalog
+read all the extensions in Segment's output for the various measurements it
+needed.
+
+The output of the MakeCatalog command above is a FITS table. The two clump
+and object catalogs are available in the two extensions of the single FITS
address@hidden can also output plain text tables. However, in
+the plain text format you can only have one table per file. Therefore, if
+you also request measurements on clumps, two plain text tables will be
+created (suffixed with @file{_o.txt} and @file{_c.txt}).}. Let's inspect
+the separate extensions with the Fits program like before (as shown
+below). Later, we'll inspect the table in each extension with Gnuastro's
+Table program (see @ref{Table}). Note that we could have used
address@hidden and @option{-hCLUMPS} instead of @option{-h1} and
address@hidden respectively.
+
address@hidden
+$ astfits cat/xdf-f160w.fits # Extension information
+$ asttable cat/xdf-f160w.fits -h1 --info # Objects catalog info.
+$ asttable cat/xdf-f160w.fits -h1 # Objects catalog columns.
+$ asttable cat/xdf-f160w.fits -h2 -i # Clumps catalog info.
+$ asttable cat/xdf-f160w.fits -h2 # Clumps catalog columns.
address@hidden example
+
+As you see above, when given a specific table (file name and extension),
+Table will print the full contents of all the columns. To see basic
+information about each column (for example name, units and comments),
+simply append a @option{--info} (or @option{-i}).
+
+To print the contents of special column(s), just specify the column
+number(s) (counting from @code{1}) or the column name(s) (if they have
+one). For example, if you just want the magnitude and signal-to-noise ratio
+of the clumps (in @option{-h2}), you can get it with any of the following
+commands
+
address@hidden
+$ asttable cat/xdf-f160w.fits -h2 -c5,6
+$ asttable cat/xdf-f160w.fits -h2 -c5,SN
+$ asttable cat/xdf-f160w.fits -h2 -c5 -c6
+$ asttable cat/xdf-f160w.fits -h2 -cMAGNITUDE -cSN
address@hidden example
+
+In the example above, the clumps catalog has two ID columns (one for the
+over-all clump ID and one for the ID of the clump in its host object),
+while the objects catalog only has one ID column. Therefore, the location
+of the magnitude column differs between the object and clumps catalog. So
+if you want to specify the columns by number, you will need to change the
+numbers when viewing the clump and objects catalogs. This is a useful
+advantage of having/using column address@hidden meta-data (including
+a name) can also be specified in plain text tables, see @ref{Gnuastro text
+table format}.}.
@example
-$ astfits catalog/xdf-f105w.fits # Extension information
-$ asttable catalog/xdf-f105w.fits -h1 --info # Objects catalog info.
-$ asttable catalog/xdf-f105w.fits -h1 # Objects catalog columns.
-$ asttable catalog/xdf-f105w.fits -h2 -i # Clumps catalog info.
-$ asttable catalog/xdf-f105w.fits -h2 # Clumps catalog columns.
+$ asttable catalog/xdf-f160w.fits -h1 -c4 -c5
+$ asttable catalog/xdf-f160w.fits -h2 -c5 -c6
@end example
-As you see above, to see the column contents of each table, you can just
-remove the @option{--info} (or @option{-i}) option. If you want to print
-the contents of special column(s), just specify the column number(s)
-(counting from @code{1}, same as output of the command above) or the column
-name(s) (if they have one). For example, if you just want the objects and
-clumps magnitude and signal-to-noise ratio in the F160W filter. You can get
-it with the following commands.
+Finally, the comments in MakeCatalog's output (@code{COMMENT} keywords in
+the FITS headers, or lines starting with @code{#} in plain text) contain
+some important information about the input dataset that can be useful (for
+example pixel area or per-pixel surface brightness limit). For example have
+a look at the output of this command:
@example
-$ asttable catalog/xdf-f160w.fits -h1 -cMAGNITUDE -cSN
-$ asttable catalog/xdf-f160w.fits -h2 -cMAGNITUDE -cSN
+$ astfits cat/xdf-f160w.fits -h1 | grep COMMENT
@end example
-The clumps catalog has two ID columns (one for the over-all clump ID and
-one for the ID of the clump in its host object), the magnitude column
-numbers differ between the object and clumps catalog. So if you want to
-specify the columns by number, you will need to change the numbers when
-viewing the clump and objects catalogs. This is a useful advantage of
-having/using column names.
+To calculate colors, we also need magnitude measurements on the F105W
+filter. However, the galaxy properties might differ between the filters
+(which is the whole purpose behind measuring colors). Also, the noise
+properties and depth of the datasets differ. Therefore, if we simply follow
+the same Segment and MakeCatalog calls above for the F105W filter, we are
+going to get a different number of objects and clumps. Matching the two
+catalogs is possible (for example with @ref{Match}), but the fact that the
+measurements will be done on different pixels, can bias the result. Since
+the Point Spread Function (PSF) of both images is very similar, an accurate
+color calculation can only be done when magnitudes are measured from the
+same pixels on both images.
+
+The F160W image is deeper, thus providing better detection/segmentation,
+and redder, thus observing smaller/older stars and representing more of the
+mass in the galaxies. We will thus use the pixel labels generated on the
+F160W filter, but do the measurements on the F105W filter (using the
address@hidden option) in the command below. Notice how the only
+difference between this call to MakeCatalog and the previous one is
address@hidden, the value given to @code{--zeropoint} and the output
+name.
@example
-$ asttable catalog/xdf-f160w.fits -h1 -c4 -c5
-$ asttable catalog/xdf-f160w.fits -h2 -c5 -c6
+$ astmkcatalog seg/xdf-f160w.fits --ids --ra --dec --magnitude --sn \
+ --valuesfile=nc/xdf-f105w.fits --zeropoint=26.27 \
+ --clumpscat --output=cat/xdf-f105w.fits
@end example
-Finally, the comments in MakeCatalog's output (in FITS headers or lines
-starting with @code{#} in plain text) contain some important information
-about the dataset that can be useful. Open @file{catalog/xdf-f160w_o.txt}
-in a text editor to see them.
+Look into what MakeCatalog printed on the command-line. You can see that
+(as requested) the object and clump labels were taken from the respective
+extensions in @file{seg/xdf-f160w.fits}, while the values and Sky standard
+deviation were done on @file{nc/xdf-f105w.fits}.
Since we used the same labeled image on both filters, the number of rows in
-both catalogs are the same. So, let's measure the colors of the objects in
-this image. We'll merge the two clump catalogs together into one using the
address@hidden program on the command-line. The output file will have each
-line of both catalogs merged into a single line.
+both catalogs are the same. The clumps are not affected by the
+hard-to-deblend and low signal-to-noise diffuse regions, they are more
+robust for calculating the colors (compared to objects). Therefore from
+this step onward, we'll continue with clumps.
+
+We can finally calculate the colors of the objects from these two
+datasets. We'll merge them into one table using the @command{paste} program
+on the command-line. But, we only want the magntitude from the F105W
+dataset, so we'll only pull out the @code{MAGNITUDE} and @code{SN}
+column. The output of @command{paste} will have each line of both catalogs
+merged into a single line.
@example
-$ paste catalog/xdf-f160w_c.txt catalog/xdf-f105w_c.txt \
- > xdf-f160w-f105w_c_p.txt
+$ asttable cat/xdf-f160w.fits -h2 > xdf-f160w.txt
+$ asttable cat/xdf-f105w.fits -h2 -cMAGNITUDE,SN > xdf-f105w.txt
+$ paste xdf-f160w.txt xdf-f105w.txt > xdf-f160w-f105w.txt
@end example
-Open @file{xdf-f160w-f105w_c_p.txt} after making it to see how
address@hidden has operated. We can now use AWK to find the colors. We'll
-ask AWK to only use lines that don't start with @code{#} and don't have a
-NaN magnitude in the 9th column (F105W address@hidden that the
-objects and clumps labels were made on the F160W image. On the F105W image,
-there might not be enough signal, so random scatter may give a negative
-total brightness and thus a NaN magnitude.}). We will also ignore columns
-which don't have reliable F105W magnitudes (with a S/N less than
address@hidden value of 7 is taken from the clump S/N threshold in F160W
-(where the clumps were defined).}). For the other lines, AWK will print the
-ID, positional columns and the difference between the respective magnitude
-columns.
+Open @file{xdf-f160w-f105w.txt} to see how @command{paste} has operated. We
+can now use AWK to find the colors. We'll ask AWK to only use lines that
+don't have a NaN magnitude in the 7th column (F105W
address@hidden that the objects and clumps labels were made on
+the F160W image. On the F105W image, there might not be enough signal, so
+random scatter may give a negative total brightness and thus a NaN
+magnitude.}). We will also ignore columns which don't have reliable F105W
+measurement (with a S/N less than address@hidden value of 7 is taken from
+the clump S/N threshold in F160W (where the clumps were defined).}). For
+the other lines, AWK will print the IDs, positional columns and the
+difference between the respective magnitude columns.
@example
-$ awk '!/^#/ && $11!="nan" && $12>7 @{print $1, $2, $3, $4, address@hidden' \
- xdf-f160w-f105w_c_p.txt > catalog/xdf-f105w-f160w_c.txt
+$ awk '$7!="nan" && $8>7 @{print $1, $2, $3, $4, address@hidden' \
+ xdf-f160w-f105w.txt > cat/xdf-f105w-f160w_c.txt
@end example
Gnuastro has a simple program for basic statistical analysis. The command
@@ -3416,14 +3479,15 @@ working on a server (where you may not have graphic
user interface), and
finally, its fast.
@example
-$ aststatistics catalog/xdf-f105w-f160w_c.txt -c5
+$ aststatistics cat/xdf-f105w-f160w_c.txt -c5
@end example
You can later use Gnuastro's Statistics program with the
@option{--histogram} option to build a much more fine-grained histogram as
a table to feed into your favorite plotting program for a much more
-accurate/appealing plot. If you just want a specific measure, for example
-the mean, median and standard deviation, you can ask for them specifically:
+accurate/appealing plot (for example with PGFPlots in @LaTeX{}). If you
+just want a specific measure, for example the mean, median and standard
+deviation, you can ask for them specifically:
@example
$ aststatistics catalog/xdf-f105w-f160w_c.txt -c5 --mean --median --std
@@ -3431,21 +3495,23 @@ $ aststatistics catalog/xdf-f105w-f160w_c.txt -c5
--mean --median --std
Some researchers prefer to have colors in a fixed aperture for all the
objects. The colors we calculated above used a different segmentation map
-for each object. This might not satisfy some science cases. To make a fixed
-aperture catalog, we should make a labeled image which has a fixed label
-for each aperture. That labeled image can be given to MakeCatalog instead
-of NoiseChisel's labeled detection image.
+for each object. This might not satisfy some science cases. So, let's make
+a fixed aperture catalog. To make an catalog from fixed apertures, we
+should make a labeled image which has a fixed label for each aperture. That
+labeled image can be given to MakeCatalog instead of Segment's labeled
+detection image.
@cindex GNU AWK
-We'll use the objects catalog in the F160W catalog we generated before for
-the positions and set the other parameters of each profile to be a fixed
-circle of radius 5 pixels (we want all apertures to be fixed
-after all). AWK is a wonderful tool for such jobs as the command below
-shows.
+To generate the apertures catalog, we'll first read the positions from
+F160W catalog we generated before for the positions and set the other
+parameters of each profile to be a fixed circle of radius 5 pixels (we want
+all apertures to be identical after all).
@example
-$ awk '!/^#/@{print $1, $2, $3, 5, 5, 0, 0, 1, $1, address@hidden'
\
- catalog/xdf-f160w_c.txt > catalog/apertures.txt
+$ rm *.txt
+$ $ asttable cat/xdf-f160w.fits -h2 -cRA,DEC \
+ | awk '!/^#/@{print NR, $1, $2, 5, 5, 0, 0, 1, NR,
address@hidden' \
+ > apertures.txt
@end example
We can now feed this catalog into MakeProfiles to build the apertures for
@@ -3458,15 +3524,14 @@ a @emph{magnitude} (in log-scale) of the value given in
that column (what
you would expect when simulating a galaxy for example).
@example
-$ astmkprof catalog/apertures.txt --background=flat-ir/xdf-f160w.fits \
- --clearcanvas --replace --type=int16 --mforflatpix \
+$ astmkprof apertures.txt --background=flat-ir/xdf-f160w.fits \
+ --clearcanvas --replace --type=int16 --mforflatpix \
--mode=wcs
@end example
The first thing you might notice in the printed information is that the
profiles are not built in order. This is because MakeProfiles works in
-parallel and parallel CPU operations are asynchronous. Without
address@hidden, the output is the same in any case. You can try running
+parallel, and parallel CPU operations are asynchronous. You can try running
MakeProfiles with one thread (using @option{--numthreads=1} to see how
order is respected in that case.
@@ -3478,99 +3543,102 @@ are interested, please join us in completing Gnuastro
with added
improvements like this (see task 14750
@address@hidden://savannah.gnu.org/task/index.php?14750}}).
address@hidden labeled image can now be fed input into
-MakeCatalog. Similar to how we used the F160W labels for the F105W catalog:
-the labels don't have to be produced by NoiseChisel. In comparison with the
-previous MakeCatalog call, notice how 1) we have no more clumps image, 2)
-that we have set @option{--objectshdu=1} (since @file{apertures.fits} is
-not the output of NoiseChisel where the labeled image is in the third
-extension), and 3) that we are now using @option{--objid} instead of
address@hidden to avoid warnings that some ID columns are only for clumps.
+We can now feed the labeled @file{apertures.fits} labeled image into
+MakeCatalog instead of Segment's output as shown below. In comparison with
+the previous MakeCatalog call, you will notice that there is no more
address@hidden option, since each aperture is treated as a separate
+object.
@example
-$ astmkcatalog noisechisel/xdf-f105w.fits --zeropoint=26.27 \
- --objectsfile=apertures.fits --objectshdu=1 \
- --objid --ra --dec --magnitude --sn \
- --output=catalog/xdf-f105w-aper.fits
+$ astmkcatalog apertures.fits -h1 --zeropoint=26.27 \
+ --valuesfile=nc/xdf-f105w.fits \
+ --ids --ra --dec --magnitude --sn \
+ --output=cat/xdf-f105w-aper.fits
@end example
-Change the filter name and zeropoint magnitudes and run this command again
-to have the fixed aperture magnitude in the F160W filter also. From this
-point on, you can follow the previous steps to derive the color in a fixed
-aperture.
+This catalog has the same number of rows as the catalog produced from
+clumps, therefore similar to how we found colors, you can compare the
+aperutre and clump magnitudes for example. You can also change the filter
+name and zeropoint magnitudes and run this command again to have the fixed
+aperture magnitude in the F160W filter and measure colors on apertures.
-We will now find some of the objects with the strongest color difference
-and make a cutout to inspect them visually: let's see what the objects with
-a color more than two magnitudes look like.
-
address@hidden AWK
-We'll use the @file{catalog/xdf-f105w-f160w_c.txt} file that we produced
-above. With the command below, all lines with a color value more than 2
-will be put in @file{reddest.txt} and inspect it using @command{cat}.
address@hidden GNU AWK
+Let's find some of the objects with the strongest color difference and make
+a cutout to inspect them visually: let's see what the objects with a color
+more than two magnitudes look like. We'll use the
address@hidden/xdf-f105w-f160w_c.txt} file that we made above. With the command
+below, all lines with a color more than 1.5 will be put in @file{reddest.txt}
@example
-$ awk '$5>1' catalog/xdf-f105w-f160w_c.txt > reddest.txt
-$ cat reddest.txt
+$ awk '$5>1.5' cat/xdf-f105w-f160w_c.txt > red.txt
@end example
-We can now feed @file{reddest.txt} into Gnuastro's crop to see what these
+We can now feed @file{red.txt} into Gnuastro's crop to see what these
objects look like. To keep things clean, we'll make a directory called
address@hidden and ask Crop to save the crops in this directory. We'll also
-add a @file{-f160w.fits} suffix to the crops.
address@hidden and ask Crop to save the crops in this directory. We'll
+also add a @file{-f160w.fits} suffix to the crops (to remind us which image
+they came from).
@example
-$ mkdir crop
-$ astcrop --mode=wcs --coordcol=3 --coordcol=4 flat-ir/xdf-f160w.fits \
- --catalog=reddest.txt --width=15/3600,15/3600 \
- --suffix=-f160w.fits --output=crop
+$ mkdir crop-red
+$ astcrop --mode=wcs --coordcol=3 --coordcol=4 flat-ir/xdf-f160w.fits \
+ --catalog=red.txt --width=15/3600,15/3600 \
+ --suffix=-f160w.fits --output=crop-red
@end example
Like the MakeProfiles command above, you might notice that the crops aren't
-made in order. Since each crop is independent of the rest, the crops are
-done in parallel and parallel operations are asynchronous. In the command
-above, change @file{f160w} to @file{f105w} to make the crops in both
-filters.
+made in order. This is because each crop is independent of the rest,
+therefore crops are done in parallel, and parallel operations are
+asynchronous. In the command above, you can change @file{f160w} to
address@hidden to make the crops in both filters.
To view the crops more easily (not having to open ds9 for each image), you
-can convert the FITS crops into the JPEG format.
+can convert the FITS crops into the JPEG format with a shell loop like
+below.
@example
+$ cd crop-red
$ for f in *.fits; do \
astconvertt $f --fluxlow=-0.001 --fluxhigh=0.005 --invert -ojpg; \
done
+$ cd ..
@end example
-The loop above is in series: each file is processed only after the previous
-ones are complete. If you have @url{https://www.gnu.org/software/parallel,
-GNU Parallel}, you can greatly speed up this conversion. GNU Parallel will
-run the separate commands simultaneously on different CPU threads in
-parallel. For more information on efficiently using your threads, see
address@hidden operations}.
+You can now easily use your general graphic user interface image viewer to
+flip through the images more easily. On GNOME, you can use the ``Eye of
+GNOME'' image viewer (with executable name of @file{eog}). Run the command
+below and by pressing the @key{<SPACE>} key, you can flip through the
+images and compare them visually more easily. Of course, the flux ranges
+have been chosen generically here for seeing the fainter parts. Therefore,
+brighter objects will be fully black.
@example
-$ parallel astconvertt --fluxlow=-0.001 --fluxhigh=0.005 --invert \
- -ojpg ::: *.fits
+$ eog 1-f160w.jpg
@end example
-You can now easily use your general GUI image viewer to flip through the
-images more easily. On GNOME, you can use the ``Eye of GNOME'' image viewer
-(with executable name of @file{eog}). Run the command below and by pressing
-the @key{<SPACE>} key, you can flip through the images and compare them
-visually more easily. Of course, the flux ranges have been chosen
-generically here for seeing the fainter parts. Therefore, brighter objects
-will be fully black.
address@hidden GNU Parallel
+The @code{for} loop above to convert the images will do the job in series:
+each file is converted only after the previous ones are complete. If you
+have @url{https://www.gnu.org/software/parallel, GNU Parallel}, you can
+greatly speed up this conversion. GNU Parallel will run the separate
+commands simultaneously on different CPU threads in parallel. For more
+information on efficiently using your threads, see @ref{Multi-threaded
+operations}. Here is a replacement for the shell @code{for} loop above
+using GNU Parallel.
@example
-$ eog 1-f105w.jpg
+$ cd crop-red
+$ parallel astconvertt --fluxlow=-0.001 --fluxhigh=0.005 --invert \
+ -ojpg ::: *.fits
+$ cd ..
@end example
Another thing that is commonly needed is to visually mark these objects on
-the image. DS9 has ``Region''s for this purpose. You just have to convert
-your catalog into a ``region file'' to feed into DS9. To do that, you can
-use AWK again as shown below.
+the image. DS9 has the ``Region''s concept for this purpose. You just have
+to convert your catalog into a ``region file'' to feed into DS9. To do
+that, you can use AWK again as shown below.
@example
-$ cd ..
$ awk 'address@hidden "# Region file format: DS9 version 4.1"; \
print "global color=green width=2"; \
print "fk5";@} \
@@ -3579,20 +3647,25 @@ $ awk 'address@hidden "# Region file format: DS9
version 4.1"; \
@end example
This region file can be loaded into DS9 with its @option{-regions} option
-as shown below (see above for the other options):
+to display over any image (that has world coordinate system). In the
+example below, we'll open Segment's output and load the regions over all
+the extensions (to see the image and the respective clump):
@example
-$ ds9 -mecube noisechisel/xdf-f160w.fits -zscale -zoom to fit \
+$ ds9 -mecube seg/xdf-f160w.fits -zscale -zoom to fit \
-regions load all reddest.reg
@end example
Finally, if this book or any of the programs in Gnuastro have been useful
-for your research, please cite the respective papers. All Gnuastro programs
-have a @option{--cite} option to help you cite the authors' work more
-easily. For example:
+for your research, please cite the respective papers and share your
+thoughts and suggestions with us (it can be very encouraging). All Gnuastro
+programs have a @option{--cite} option to help you cite the authors' work
+more easily. Just note that it may be necessary to cite additional papers
+for different programs, so please try it out for any program you use.
@example
$ astmkcatalog --cite
+$ astnoisechisel --cite
@end example
@@ -7709,7 +7782,7 @@ $ ls
ABC01.jpg ABC02.jpg
$ astnoisechisel /mnt/data/DEFproject/DEF01.fits # Prog 2
$ ls
-ABC01.jpg ABC02.jpg DEF01_labeled.fits
+ABC01.jpg ABC02.jpg DEF01_detected.fits
@end example
@@ -13641,60 +13714,78 @@ non-random factors (for example light from a distant
galaxy) are
present. This classification of a dataset is formally known as
@emph{detection}.
-In a dataset, signal is inherently mixed with noise: only mock/simulated
-datasets are free of noise. Therefore detection, or the process of
-separating signal from noise, determines the number of objects you study
-and the accuracy of any higher-level measurement you do on them. Detection
-is thus the most important step of any analysis and is not trivial. In
-particular, the most scientifically interesting astronomical targets are
-faint, can have a large variety of morphologies, along with a large
-distribution in brightness and size, and are deeply drowned in an ocean of
-noise. Therefore when noise is significant, detection is the uniquely
-decisive step of your scientific result.
+In an observational/experimental dataset, signal is always burried in
+noise: only mock/simulated datasets are free of noise. Therefore detection,
+or the process of separating signal from noise, determines the number of
+objects you study and the accuracy of any higher-level measurement you do
+on them. Detection is thus the most important step of any analysis and is
+not trivial. In particular, the most scientifically interesting
+astronomical targets are faint, can have a large variety of morphologies,
+along with a large distribution in brightness and size. Therefore when
+noise is significant, detection is the uniquely decisive step of your
+scientific result.
@cindex Erosion
-NoiseChisel is Gnuastro's program for detection. NoiseChisel uses a
-noise-based paradigm to signal detection that was initially introduced in
address@hidden://arxiv.org/abs/1505.01664, Akhlaghi and Ichikawa [2015]}. The
-name of NoiseChisel is derived from the first thing it does after
-thresholding the dataset: to erode it. In mathematical morphology, erosion
-on pixels can be pictured as carving off boundary pixels. Hence, what
-NoiseChisel does is similar to what a wood chisel or stone chisel do. It is
-just not a hardware, but a software. In fact, looking at it as a chisel and
-your dataset as a solid cube of rock will greatly help in best using it:
-with NoiseChisel you literally carve your targets out of the noise. Try
-running it with the @option{--checkdetection} option to see each step of
-the carving process on your input dataset.
+NoiseChisel is Gnuastro's program for detection of targets that don't have
+a sharp border (almost all astronomical objects). When the targets have a
+sharp edges/border, a simple threshold is enough to separate them from
+noise and each other (if they are not touching). For detection of such
+targets, you can use Gnuastro's Arithmetic program in a command like below
+(assuming the threshold is @code{100}, see @ref{Arithmetic}):
+
address@hidden
+$ astarithmetic in.fits 100 gt 2 connected-components
address@hidden example
+
+NoiseChisel uses a new noise-based paradigm for detection of very exteded
+and diffuse targets that are drowned deeply in the ocean of noise. It was
+initially introduced in @url{https://arxiv.org/abs/1505.01664, Akhlaghi and
+Ichikawa [2015]}. The name of NoiseChisel is derived from the first thing
+it does after thresholding the dataset: to erode it. In mathematical
+morphology, erosion on pixels can be pictured as carving-off boundary
+pixels. Hence, what NoiseChisel does is similar to what a wood chisel or
+stone chisel do. It is just not a hardware, but a software. In fact,
+looking at it as a chisel and your dataset as a solid cube of rock will
+greatly help in effectively understanding and optimally using it: with
+NoiseChisel you literally carve your targets out of the noise. Try running
+it with the @option{--checkdetection} option to see each step of the
+carving process on your input dataset.
@cindex Segmentation
-NoiseChisel's primary output is a dataset with the same size as the input
-but with integer values that identify each pixel's label (class). Pixels
-that don't harbor any detected signal (noise) are given a label (or value)
-of zero and those with a positive integer have been identified as hosting
-signal. Segmentation is the process of classifing the signal into
-higher-level constructs, for example if you have two separate galaxies in
-one image, you give them separate labels. NoiseChisel does a first order
-segmentation by labeling detections that aren't touching each other as
-separate detections. Each physically separate detection is thus given a
-different label (or counter, starting from 1).
-
-For more on NoiseChisel's particular output format and its benefits
-(especially in conjunction with @ref{MakeCatalog}), please see
address@hidden://arxiv.org/abs/1611.06387, Akhlaghi [2016]}. The output is
-designed to be generic enough to be easily used in any higher-level
-analysis: Higher-level segmentation (for example classifying a large
-detected body as multiple nearby galaxies), you can feed NoiseChisel's
-output to Gnuastro's Segment program (see @ref{Segment}). If NoiseChisel's
-first order segmentation is enough for your analysis, you can feed it
-directly into MakeCatalog to do measurements over the detections and
-produce a catalog (see @ref{MakeCatalog}).
+NoiseChisel's primary output is a binary detection map with the same size
+as the input but only with two values: 0 and 1. Pixels that don't harbor
+any detected signal (noise) are given a label (or value) of zero and those
+with a value of 1 have been identified as hosting signal. Segmentation is
+the process of classifing the signal into higher-level constructs, for
+example if you have two separate galaxies in one image, after segmentation,
+they will get separate labels. NoiseChisel is only focused on detection
+(separating signal from noise), to segment the signal (into separate
+galaxies for example), Gnuastro has a separate specialized program
address@hidden NoiseChisel's output can be directly/readily fed into
+Segment.
+
+For more on NoiseChisel's output format and its benefits (especially in
+conjunction with @ref{Segment} and later @ref{MakeCatalog}), please see
address@hidden://arxiv.org/abs/1611.06387, Akhlaghi [2016]}. Just note that
+when that paper was published, Segment was not yet spinned-off, and
+NoiseChisel done both detection and segmentation. The output is designed to
+be generic enough to be easily used in any higher-level analysis. If your
+targets are not touching after running NoiseChisel and you aren't
+interested in their sub-structure, you don't need the Segment program at
+all. You can ask NoiseChisel to find the connected pixels in the output
+with the @option{--label} option. In this case, the output won't be a
+binary image any more, the signal will have counters/labels starting from 1
+for each connected group of pixels. You can then directly feed
+NoiseChisel's output into MakeCatalog for measurements over the detections
+and the production of a catalog (see @ref{MakeCatalog}).
Thanks to the published papers mentioned above, there is no need to provide
-a more complete introduction in this book. However, published papers cannot
-be updated any more, but the software has evolved/changed. The changes
-since publication are documented in @ref{NoiseChisel changes after
-publication}. Afterwards, in @ref{Invoking astnoisechisel}, the details of
-running NoiseChisel and its options are discussed.
+a more complete introduction to NoiseChisel in this book. However,
+published papers cannot be updated any more, but the software has
+evolved/changed. The changes since publication are documented in
address@hidden changes after publication}. Afterwards, in @ref{Invoking
+astnoisechisel}, the details of running NoiseChisel and its options are
+discussed.
As discussed above, detection is the single most important step for your
scientific result. It is therefore very important to obtain a good
@@ -13703,15 +13794,23 @@ understanding of NoiseChisel (and afterwards
@ref{Segment} and
papers above and the respective sections of Gnuastro's book, you play a
little with the settings (in the order presented in @ref{Invoking
astnoisechisel}) on a dataset you are familiar with and inspect all the
-check images (produced with options starting with @option{--check}) to see
-the effect of each parameter.
+check images (options starting with @option{--check}) to see the effect of
+each parameter.
@ref{General program usage tutorial} is also a good place to get a feeling
-of how Gnuastro's programs are built to complement, and build upon, each
-other. This tutorial culminates culminating in using NoiseChisel to detect
-galaxies and use its outputs to find the galaxy colors. This is a common
-use case, so please also complete that tutorial for most effectively using
-NoiseChisel in conjunction with all of the other Gnuastro programs.
+of the modular principle behind Gnuastro's programs and how they are built
+to complement, and build upon, each other. The tutorial culminates in using
+NoiseChisel to detect galaxies and use its outputs to find the galaxy
+colors. Defining colors is a very common process in most
+science-cases. Therefore it is also recommended to (patiently) complete
+that tutorial for optimal usage of NoiseChisel in conjunction with all the
+other Gnuastro programs.
+
+In @ref{NoiseChisel changes after publication}, we'll review the changes in
+NoiseChisel since the publication of @url{https://arxiv.org/abs/1611.06387,
+Akhlaghi [2016]}. We will then review NoiseChisel's input, detection, and
+output options in @ref{NoiseChisel input}, @ref{Detection options}, and
address@hidden output}.
@menu
* NoiseChisel changes after publication:: Changes to the software after
publication.
@@ -13832,11 +13931,12 @@ of false detections, see the descriptions under this
option in
@node Invoking astnoisechisel, , NoiseChisel changes after publication,
NoiseChisel
@subsection Invoking NoiseChisel
-NoiseChisel will detect signal in noise producing a multi-extension labeled
-image. Its output can be readily used for input into @ref{Segment}, for
-higher-level segmentation, or @ref{MakeCatalog} to generate a catalog. The
-executable name is @file{astnoisechisel} with the following general
-template
+NoiseChisel will detect signal in noise producing a multi-extension dataset
+containing a binary detection map which is the same size as the input. Its
+output can be readily used for input into @ref{Segment}, for higher-level
+segmentation, or @ref{MakeCatalog} to do measurements and generate a
+catalog. The executable name is @file{astnoisechisel} with the following
+general template
@example
$ astnoisechisel [OPTION ...] InputImage.fits
@@ -13852,9 +13952,9 @@ $ astnoisechisel input.fits
## Inspect all the detection steps after changing a parameter.
$ astnoisechisel input.fits --qthresh=0.4 --checkdetection
-## Detect signal assuming input has 4 channels along first dimension
-## and 1 along the second. Also set the regular tile size to 100 along
-## both dimensions:
+## Detect signal assuming input has 4 amplifier channels along first
+## dimension and 1 along the second. Also set the regular tile size
+## to 100 along both dimensions:
$ astnoisechisel --numchannels=4,1 --tilesize=100,100 input.fits
@end example
@@ -13928,26 +14028,26 @@ $ astnoisechisel --help | grep check
@end cartouche
Below, we'll discuss NoiseChisel's options, classified into two general
-classes, to help in easy navigation. @ref{General NoiseChisel options}
-mainly discusses the options relating to inputs and prior to
-detection. Afterwards, @ref{Detection options} fully describes every
-configuration parameter (option) related to detection and how they affect
-the final result. The order of options in this section follow the logical
-order within NoiseChisel. On first reading (while you are still new to
-NoiseChisel), it is therefore strongly recommended to read the options in
-the given order below. The output of @option{--printparams} (or
+classes, to help in easy navigation. @ref{NoiseChisel input} mainly
+discusses the basic options relating to inputs and prior to the detection
+process detection. Afterwards, @ref{Detection options} fully describes
+every configuration parameter (option) related to detection and how they
+affect the final result. The order of options in this section follow the
+logical order within NoiseChisel. On first reading (while you are still new
+to NoiseChisel), it is therefore strongly recommended to read the options
+in the given order below. The output of @option{--printparams} (or
@option{-P}) also has this order. However, the output of @option{--help} is
sorted alphabetically. Finally, in @ref{NoiseChisel output} the format of
NoiseChisel's output is discussed.
@menu
-* General NoiseChisel options:: General NoiseChisel preprocessing.
+* NoiseChisel input:: NoiseChisel's input options.
* Detection options:: Configure detection in NoiseChisel.
-* NoiseChisel output:: NoiseChisel's output format.
+* NoiseChisel output:: NoiseChisel's output options and format.
@end menu
address@hidden General NoiseChisel options, Detection options, Invoking
astnoisechisel, Invoking astnoisechisel
address@hidden General NoiseChisel options
address@hidden NoiseChisel input, Detection options, Invoking astnoisechisel,
Invoking astnoisechisel
address@hidden NoiseChisel input
The options here can be used to configure the inputs and output of
NoiseChisel, along with some general processing options. Recall that you
@@ -14067,56 +14167,9 @@ sizes. Except for the tile size, all the other
parameters for this
tessellation are taken from the common options described in @ref{Processing
options}. The format is identical to that of the @option{--tilesize} option
that is discussed in that section.
-
address@hidden --continueaftercheck
-Continue NoiseChisel after any of the options starting with
address@hidden NoiseChisel involves many steps and as a result, there
-are many checks, allowing to inspect the status of the processing. The
-results of each step affect the next steps of processing. Therefore, when
-you want to check the status of the processing at one step, the time spent
-to complete NoiseChisel is just wasted/distracting time.
-
-To encourage easier experimentation with the option values, when you use
-any of the NoiseChisel options that start with @option{--check},
-NoiseChisel will abort once all the desired check file is complete. With
-the @option{--continueaftercheck} option, you can disable this behavior and
-ask NoiseChisel to continue with the rest of the processing, even after
-completing the requested check files are complete.
-
address@hidden --rawoutput
-Don't include the Sky-subtracted input image as the first extension of the
-output. By default, the Sky-subtracted input image is put in the first
-extension of the output. The next extensions are NoiseChisel's main output:
-a labeled image with the detected pixels having positive values and the Sky
-and Sky standard deviation images, see @ref{NoiseChisel output}.
-
-The extra Sky-subtracted input can help in checking NoiseChisel's
-configuration by comparing the detections with the input image: visually
-see if everything you expected is detected (reasonable completeness) and
-that you don't have too many false detections (reasonable purity). This
-visual inspection is simplified if you use SAO DS9 to view NoiseChisel's
-output as a multi-extension datacube, see @ref{Viewing multiextension FITS
-images}.
-
-However, when you are satisfied with your NoiseChisel configuration
-(therefore you don't need to check on every run), the datasets become
-large, or you are running NoiseChisel as part of a pipeline, this
-Sky-subtracted input image can be significant burden (take up a large
-volume). In such cases, @option{--rawoutput} can be used to avoid this
-extra extension. It is always possible to easily produce the Sky-subtracted
-image from the input (assuming it is in extension @code{1} of
address@hidden) and the @code{SKY} extension of NoiseChisel's output (let's
-call it @file{nc.fits}) with a command like below (assuming NoiseChisel
-wasn't run with @option{--oneelempertile}, see @ref{Tessellation}):
-
address@hidden
-$ astarithmetic in.fits nc.fits - -h1 -hSKY
address@hidden example
-
-
@end table
address@hidden Detection options, NoiseChisel output, General NoiseChisel
options, Invoking astnoisechisel
address@hidden Detection options, NoiseChisel output, NoiseChisel input,
Invoking astnoisechisel
@subsubsection Detection options
Detection is the process of separating the pixels in the image into two
@@ -14462,48 +14515,26 @@ have the same pixel size as the input, but with the
@node NoiseChisel output, , Detection options, Invoking astnoisechisel
@subsubsection NoiseChisel output
-The main output of NoiseChisel is the detected pixels (labeled with
-integers starting from 1) over the input dataset along with the Sky and Sky
-standard deviation that are all returned as multiple extensions of a FITS
-file. The name of the output file can be set by giving a value to
address@hidden If @option{--output} is not used, the output will be
-input name but with a @file{_labeled.fits} suffix, see @ref{Automatic
-output}. If any of the options starting with @option{--check*} are given,
-NoiseChisel won't complete and will abort as soon as the respective check
-images are created. See the descriptions in @ref{Detection options} for a
-description of the extensions in each of the check images when the
address@hidden options are called. With the
address@hidden option, NoiseChisel will produce the check
-images without aborting and will continue to finish the detection.
-
-The default output can have three or four extensions based on the
address@hidden option. If it is called on the command-line or given a
-value of 1 in any of the configuration files, then the output will only
-contain the three raw outputs (Labeled image, Sky and Sky Standard
-deviation). This can be useful when you are calling NoiseChisel as part of
-a large script and on large files where the extra extension is redundant,
-but will take up a large volume.
-
-When @option{--rawoutput} is not called, the first extension will be the
-input image subtracted by the Sky value. When playing-with/testing
-NoiseChisel on a new dataset, having this extra image can greatly simplify
-the inspection of the results. By flipping through the extensions, you can
-check how accurate or effective the change in parameters was.
-
-To inspect NoiseChisel's output, you can configure SAO DS9 in your Graphic
-User Interface (GUI) to open NoiseChisel's output as a multi-extension data
-cube. This will allow you to flip through the different extensions and
-visually inspect the results. This process has been described for the GNOME
-GUI (most common GUI in GNU/Linux operating systems) in @ref{Viewing
-multiextension FITS images}.
-
-The main output (present in any case) is thus the labeled dataset. In this
-image, the non-detected (Sky) pixels of the input are given a label of zero
-and the detected pixel are given positive label (starting from one, based
-on connectivity). The total number of labels is stored as the value to the
address@hidden keyword in the header of this extension. This number (along
-with other important information) is also printed by NoiseChisel in verbose
-mode (when @option{--quiet} is not called).
+NoiseChisel's output is a multi-extension FITS file. The main
+extension/dataset is a (binary) detection map. It has the same size as the
+input but with only two possible values for all pixels: 0 (for pixels
+identified as noise) and 1 (for those identified as signal/detections). The
+detection map is followed by a Sky and Sky standard deviation dataset
+(which are calculated from the binary image). By default (when
address@hidden isn't called), NoiseChisel will also subtract the Sky
+value from the input and save the sky-subtracted input as the first
+extension in the output.
+
+The name of the output file can be set by giving a value to
address@hidden (this is a common option between all programs and is
+therefore discussed in @ref{Input output options}). If @option{--output}
+isn't used, the input name will be suffixed with @file{_detected.fits} and
+used as output, see @ref{Automatic output}. If any of the options starting
+with @option{--check*} are given, NoiseChisel won't complete and will abort
+as soon as the respective check images are created. For more information on
+the different check images, see the description for the @option{--check*}
+options in @ref{Detection options} (this can be disabled with
address@hidden).
The last two extensions of the output are the Sky and its Standard
deviation, see @ref{Sky value} for a complete explanation. They are
@@ -14513,17 +14544,110 @@ pixels in one tile given one value. To be more
space-efficient (keep only
one pixel per tile), you can use the @option{--oneelempertile} option, see
@ref{Tessellation}.
address@hidden GNOME
+To inspect any of NoiseChisel's output files, assuming you use SAO DS9, you
+can configure your Graphic User Interface (GUI) to open NoiseChisel's
+output as a multi-extension data cube. This will allow you to flip through
+the different extensions and visually inspect the results. This process has
+been described for the GNOME GUI (most common GUI in GNU/Linux operating
+systems) in @ref{Viewing multiextension FITS images}.
+
+NoiseChisel's output configuration options are described in detail below.
+
address@hidden @option
address@hidden --continueaftercheck
+Continue NoiseChisel after any of the options starting with
address@hidden (see @ref{Detection options}. NoiseChisel involves many
+steps and as a result, there are many checks, allowing you to inspect the
+status of the processing. The results of each step affect the next steps of
+processing. Therefore, when you want to check the status of the processing
+at one step, the time spent to complete NoiseChisel is just
+wasted/distracting time.
+
+To encourage easier experimentation with the option values, when you use
+any of the NoiseChisel options that start with @option{--check},
+NoiseChisel will abort once its desired extensions have been written. With
address@hidden option, you can disable this behavior and ask
+NoiseChisel to continue with the rest of the processing, even after the
+requested check files are complete.
+
address@hidden -l
address@hidden --label
+Run a connected-components algorithm on the finally detected pixels to
+identify which pixels are connected to which. By default the main output is
+a binary dataset with only two values: 0 (for noise) and 1 (for
+signal/detections). See @ref{NoiseChisel output} for more.
+
+The purpose of Noisechisel is to detect targets that are extended and
+diffuse, with outer parts that sink into the noise very gradually (galaxies
+and stars for example). Since NoiseChisel digs down to extremely low
+surface brightness values, many such targets will commonly be detected
+together as a single large body of connected pixels.
+
+To properly separte connected objects, sophisticated segmentation methods
+are commonly necessary on NoiseChisel's output. Gnuastro has the dedicated
address@hidden program for this job. Since input images are commonly large
+and can take a significant volume, the extra volume necessary to store the
+labels of the connected components in the detection map (which will be
+created with this @option{--label} option, in 32-bit signed integer type)
+can thus be a major waste of space. Since the default output is just a
+binary dataset, an 8-bit unsigned dataset is enough.
+
+The binary output will also encourage users to segment the result
+separately prior to doing higher-level analysis. As an alternative to
address@hidden, if you have the binary detection image, you can use the
address@hidden operator in Gnuastro's Arithmetic program to
+identify regions that are connected with each other. For example with this
+command (assuming NoiseChisel's output is called @file{nc.fits}):
+
address@hidden
+$ astarithmetic nc.fits connected-components -hDETECTIONS
address@hidden example
+
address@hidden --rawoutput
+Don't include the Sky-subtracted input image as the first extension of the
+output. By default, the Sky-subtracted input is put in the first extension
+of the output. The next extensions are NoiseChisel's main outputs described
+above.
+
+The extra Sky-subtracted input can be convenient in checking NoiseChisel's
+output and comparing the detection map with the input: visually see if
+everything you expected is detected (reasonable completeness) and that you
+don't have too many false detections (reasonable purity). This visual
+inspection is simplified if you use SAO DS9 to view NoiseChisel's output as
+a multi-extension datacube, see @ref{Viewing multiextension FITS images}.
+
+When you are satisfied with your NoiseChisel configuration (therefore you
+don't need to check on every run), or you want to archive/transfer the
+outputs, or the datasets become large, or you are running NoiseChisel as
+part of a pipeline, this Sky-subtracted input image can be a significant
+burden (take up a large volume). The fact that the input is also noisy,
+makes it hard to compress it efficiently.
+
+In such cases, this @option{--rawoutput} can be used to avoid the extra
+sky-subtracted input in the output. It is always possible to easily produce
+the Sky-subtracted dataset from the input (assuming it is in extension
address@hidden of @file{in.fits}) and the @code{SKY} extension of NoiseChisel's
+output (let's call it @file{nc.fits}) with a command like below (assuming
+NoiseChisel wasn't run with @option{--oneelempertile}, see
address@hidden):
+
address@hidden
+$ astarithmetic in.fits nc.fits - -h1 -hSKY
address@hidden example
address@hidden table
+
@cartouche
@noindent
@cindex Compression
@strong{Save space:} with the @option{--rawoutput} and
address@hidden, NoiseChisel's output will only be one labeled
-datasets (only containing integers) and two much smaller arrays with one
-value per tile. Since they have no noise, integer datasets can be
-compressed very effectively (without any loss of data) with exceptionally
-high compression ratios. You can therefore use the following command to
-keep NoiseChisel's output in a much more efficient manner (take up less
-volume).
address@hidden, NoiseChisel's output will only be one binary
+detection map and two much smaller arrays with one value per tile. Since
+none of these have noise they can be compressed very effectively (without
+any loss of data) with exceptionally high compression ratios. This makes it
+easy to archive, or transfer, NoiseChisel's output even on huge
+datasets. To compress it with the most efficient method (take up less
+volume), run the following command:
@cindex GNU Gzip
@example
@@ -14532,14 +14656,9 @@ $ gzip --best noisechisel_output.fits
@noindent
The resulting @file{.fits.gz} file can then be fed into any of Gnuastro's
-programs directly, without having to decompress it separately (it will just
-take them a little longer, because they have to decompress it before use).
-
-Alternatively, you can also tell NoiseChisel to directly build a
address@hidden output (for example with @option{--output=nc.fits.gz}). But
-CFITSIO (which reads and writes FITS files for Gnuastro) puts more emphasis
-on compression/decompression speed. So it doesn't use the most efficient
-compression settings (which can take slightly longer).
+programs directly, or viewed in viewers like SAO DS9, without having to
+decompress it separately (they will just take a little longer, because they
+have to internally decompress it before starting).
@end cartouche
@@ -14557,53 +14676,89 @@ compression settings (which can take slightly longer).
Once signal is separated from noise (for example with @ref{NoiseChisel}),
you have a binary dataset: each pixel is either signal (1) or noise
(0). Signal (for example every galaxy in your image) has been ``detected'',
-but they all have a label of 1. This still doesn't answer our scientific
-questions, for example, to count how many galaxies there are in the image
-and measure their properties. At the lowest level, detection is a kind of
-segmentation (segmenting the the whole dataset into signal and noise, see
address@hidden). Here, we'll define segmentation only on signal: to
-separate and find sub-structure within the detections.
+but all detections have a label of 1. Therefore while we know which pixels
+contain signal, we still can't find out how many galaxies they contain or
+which detected pixels correspond to which galaxy. At the lowest (most
+generic) level, detection is a kind of segmentation (segmenting the the
+whole dataset into signal and noise, see @ref{NoiseChisel}). Here, we'll
+define segmentation only on signal: to separate and find sub-structure
+within the detections.
@cindex Connected component labeling
If the targets are clearly separated in the dataset (image), a simple
@emph{connected
address@hidden@url{https://en.wikipedia.org/wiki/Connected-component_labeling}}
algorithm (very basic segmentation) is enough to separate the regions that
-are touching/connected. This is such a basic form of segmentation that
-detection programs (for example NoiseChisel) do this for you. However, in
-most real situations our targets are not nicely separated: they touch (for
-example merging galaxies), or are simply in the same line-of-sight, causing
-their images to overlap. In particular, when you do your detection with
-NoiseChisel, you will detect signal to very low surface brightness limits
-(the faint wings of galaxies or bright stars). Therefore, it often happens
-that several galaxies are detected as one large detection. To continue your
-scientific analysis, it is therefore necessary to separate/segment each
-detection into multiple targets as best as possible.
+are touching/connected. This is such a basic and simple form of
+segmentation that Gnuastro's Arithmetic program has an operator for it: see
address@hidden in @ref{Arithmetic operators}. Assuming the
+binary dataset is called @file{binary.fits}, you can use it with a command
+like this:
+
address@hidden
+$ astarithmetic binary.fits 2 connected-components
address@hidden example
+
address@hidden
+You can even do a very basic detection (a threshold, say at value
address@hidden) @emph{and} segmentation in Arithmetic with a single command
+like below to apply:
+
address@hidden
+$ astarithmetic in.fits 100 gt 2 connected-components
address@hidden example
+
+However, in most astronomical situations our targets are not nicely
+separated or have a sharp boundary/edge (for a threshold to suffice): they
+touch (for example merging galaxies), or are simply in the same
+line-of-sight, causing their images to overlap. In particular, when you do
+your detection with NoiseChisel, you will detect signal to very low surface
+brightness limits: deep into the faint wings of galaxies or bright stars
+(which can extend very far and irregularly from their center). Therefore,
+it often happens that several galaxies are detected as one large
+detection. To continue your scientific analysis, a simple connected
+components algorithm will not suffice. It is therefore necessary to do a
+more sophisticated segmentation and break up the detected pixels (even
+those that are touching) into multiple target objects as accurately as
+possible.
Segment will use a detection map and its corresponding dataset to find
-sub-structure over the detected areas. Segment was originally part of
address@hidden To help in modularity and improved creativity,
-NoiseChisel's segmentation features have been brought into a separate
-program. Therefore, as with NoiseChisel, the best place to start reading
-about Segment and understanding what it does (with many illustrative
-figures) is Section 3.2 of @url{https://arxiv.org/abs/1505.01664, Akhlaghi
-and Ichikawa [2015]}.
-
-Segment will first find the @emph{true} local maxima over the detections in
-the given dataset along with all the pixels in their vicinity before
-reaching a local minimum. These regions over a detection are called
address@hidden Afterwards, the clumps are grown up to a certain
-threshold. Based on the connectivity of these grown clumps, they are
-considered parts of one @emph{object} or as separate @emph{object}s. See
-Section 3.2 of Akhlaghi and Ichikawa [2015] (link above) for more. In
-short, Segment's main output are two labeled datasets: 1) clumps, and 2)
-objects. The the clumps output will only have positive integers on the true
-clumps, the rest of the detected regions will get a fixed negative
-value. The clump labels start counting from 1 within their host object, so
-if two objects have two and four clumps respectively, the clumps in the
-first will will be labeled 1 and 2 and in the second, they will be labeled
-1, 2, 3, and 4. The objects output has a positive integer (counting from 1)
-for all detected pixels.
+sub-structure over the detected areas and use them for its
+segmentation. Until Gnuastro version 0.6 (released in 2018), Segment was
+part of @ref{NoiseChisel}. Therefore, similar to NoiseChisel, the best
+place to start reading about Segment and understanding what it does (with
+many illustrative figures) is Section 3.2 of
address@hidden://arxiv.org/abs/1505.01664, Akhlaghi and Ichikawa [2015]}.
+
+As a summary, Segment first finds true @emph{clump}s over the
+detections. Clumps are associated with local maxima and extend over the
+neighboring pixels until they reach a local minimum (@emph{river}). Segment
+will use the distribution of clump signal-to-noise ratios over the
+undetected regions as reference to find ``true'' clumps over the
+detections. Using the undetected regions can be disabled by directly giving
+a signal-to-noise ratio to @option{--clumpsnthresh}.
+
+The true clumps are then grown to a certain threshold over the
+detections. Based on the strength of the connections between the grown
+clumps, they are considered parts of one @emph{object} or as separate
address@hidden See Section 3.2 of Akhlaghi and Ichikawa [2015] (link
+above) for more. Segment's main output are thus two labeled datasets: 1)
+clumps, and 2) objects. See @ref{Segment output} for more.
+
+To start learning about Segment, especially in relation to detection
+(@ref{NoiseChisel}) and measurement (@ref{MakeCatalog}), the recommended
+references are @url{https://arxiv.org/abs/1505.01664, Akhlaghi and Ichikawa
+[2015]} and @url{https://arxiv.org/abs/1611.06387, Akhlaghi [2016]}. Just
+have in mind that Segment became a separate program in 2018 (after those
+papers). Therefore we don't currently need to extend the introduction any
+further and can directly dive into the invocation details in @ref{Invoking
+astsegment}.
+
+Those papers cannot be updated any more but the software will
+evolve. Therefore this book is the definitive reference. Major changes from
+those papers will be documented here when the software starts to diverge
+from them the future, similar what we have already done for NoiseChisel in
address@hidden changes after publication}.
@menu
@@ -14613,11 +14768,10 @@ for all detected pixels.
@node Invoking astsegment, , Segment, Segment
@subsection Invoking Segment
-Segment will identify substructure within a detected region using its
-corresponding image. Its output can be readily used for input into
address@hidden to generate a catalog of the objects and clumps. The
-executable name is @file{astsegment} with the following general
-template
+Segment will identify substructure within the detected regions of an input
+image. Segment's output labels can be directly used for measurements (for
+example with @ref{MakeCatalog}). The executable name is @file{astsegment}
+with the following general template
@example
$ astsegment [OPTION ...] InputImage.fits
@@ -14628,7 +14782,7 @@ One line examples:
@example
## Segment NoiseChisel's detected regions.
-$ astsegment full-noisechisel-output.fits
+$ astsegment default-noisechisel-output.fits
## Use a hand-input S/N value for keeping true clumps
## (avoid finding the S/N using the undetected regions).
@@ -14636,59 +14790,80 @@ $ astsegment nc-out.fits --clumpsnthresh=10
## Inspect all the segmentation steps after changing a parameter.
$ astsegment input.fits --snquant=0.9 --checksegmentaion
+
+## Use the fixed value of 0.01 for the input's Sky standard deviation
+## (in the units of the input), and assume all the pixels are a
+## detection (for example a large structure extending over the whole
+## image), and only keep clumps with S/N>10 as true clumps.
+$ astsegment in.fits --std=0.01 --detection=all --clumpsnthresh=10
@end example
@cindex Gaussian
@noindent
If Segment is to do processing (for example you don't want to get help, or
-see the values to each input parameter), an input image should be provided
-with the recognized extensions (see @ref{Arguments}). NoiseChisel shares a
+see the values of each option), at least one input dataset is necessary
+along with detection and error information, either as separate datasets
+(per-pixel) or fixed values, see @ref{Segment input}. Segment shares a
large set of common operations with other Gnuastro programs, mainly
regarding input/output, general processing steps, and general operating
modes. To help in a unified experience between all of Gnuastro's programs,
-these operations have the same command-line options, see @ref{Common
-options} for a full list/description (they are not repeated here).
+these common operations have the same names and defined in @ref{Common
+options}.
As in all Gnuastro programs, options can also be given to Segment in
-configuration files. For a thorough description on Gnuastro's configuration
+configuration files. For a thorough description of Gnuastro's configuration
file parsing, please see @ref{Configuration files}. All of Segment's
options with a short description are also always available on the
command-line with the @option{--help} option, see @ref{Getting help}. To
inspect the option values without actually running Segment, append your
command with @option{--printparams} (or @option{-P}).
-To help in easy navigation between Segment's options, they are separated
-discussed in two separate sub-sections: @ref{Segment inputs and general
-settings} discusses how you can customize the inputs to Segment along with
-other general Segment options. Another section (@ref{Segmentation options})
-is devoted to options specific to the segmentation process. Finally, in
address@hidden output}, we'll discuss the format of Segment's output file.
+To help in easy navigation between Segment's options, they are separately
+discussed in the three sub-sections below: @ref{Segment input} discusses
+how you can customize the inputs to Segment. @ref{Segmentation options} is
+devoted to options specific to the high-level segmentation
+process. Finally, in @ref{Segment output}, we'll discuss options that
+affect Segment's output.
@menu
-* Segment inputs and general settings:: Input files and general options.
+* Segment input:: Input files and options.
* Segmentation options:: Parameters of the segmentation process.
* Segment output:: Outputs of Segment
@end menu
address@hidden Segment inputs and general settings, Segmentation options,
Invoking astsegment, Invoking astsegment
address@hidden Segment inputs and general settings
-
-Segment needs atleast three datasets as input: 1) Sky-subtracted values
-(actual image), 2) detection labels, and 3) standard deviation. If the
-values dataset is not Sky-subtracted, you can ask Segment to subtract the
-Sky from it by giving a file name to the @option{--sky} option, see
-below. For the rest of this discussion, we'll assume it is already sky
-subtracted.
-
-These three necessary datasets must ofcourse be related: derived from the
-same basic input in steps prior to running Segment. By default, Segment
-will assume they are multiple extensions of a single file, for example see
address@hidden output}. When the Sky-subtracted values are in one file,
-and the detection and Sky standard deviation are in another, you just need
-to use @option{--detection}: in the absence of @option{--std}, Segment will
-look for both the detection labels and Sky standard deviation in the file
-given to @option{--detection}. Ultimately, if all three are in separate
-files, you need to call both @option{--detection} and @option{--std}.
address@hidden Segment input, Segmentation options, Invoking astsegment,
Invoking astsegment
address@hidden Segment input
+
+Besides the input dataset (for example astronomical image), Segment also
+needs to know the Sky standard deviation and the regions of the dataset
+that it should segment. The values dataset is assumed to be Sky subtracted
+by default. If it isn't, you can ask Segment to subtract the Sky internally
+by calling @option{--sky}. For the rest of this discussion, we'll assume it
+is already sky subtracted.
+
+The Sky and its standard deviation can be a single value (to be used for
+the whole dataset) or a separate dataset (for a separate value per
+pixel). If a dataset is used for the Sky and its standard deviation, they
+must either be the size of the input image, or have a single value per tile
+(generated with @option{--oneelempertile}, see @ref{Processing options} and
address@hidden).
+
+The detected regions/pixels can be specified as a detection map (for
+example see @ref{NoiseChisel output}). If @option{--detection=all}, Segment
+won't read any detection map and assume the whole input is a single
+detection. For example when the dataset is fully covered by a large nearby
+galaxy/globular cluster.
+
+When dataset are to be used for any of the inputs, Segment will assume they
+are multiple extensions of a single file by default (when @option{--std} or
address@hidden aren't called). For example NoiseChisel's default
+output @ref{NoiseChisel output}. When the Sky-subtracted values are in one
+file, and the detection and Sky standard deviation are in another, you just
+need to use @option{--detection}: in the absence of @option{--std}, Segment
+will look for both the detection labels and Sky standard deviation in the
+file given to @option{--detection}. Ultimately, if all three are in
+separate files, you need to call both @option{--detection} and
address@hidden
The extensions of the three mandatory inputs can be speicified with
@option{--hdu}, @option{--dhdu}, and @option{--stdhdu}. For a full
@@ -14704,73 +14879,117 @@ for HDUs on your system, run this command:
$ astsegment -P | grep hdu
@end example
-To help in identifying the true clumps, Segment can also work on a
-convolved image (see Section 3.2.1 and Figure 8 of
address@hidden://arxiv.org/abs/1505.01664, Akhlaghi and Ichikawa [2015]}). If
-you have already convolved your input image, you can use the
address@hidden and @option{--chdu} options to specify the file name
-and HDU/extension that it is stored in. Just don't forget that the
-convolved image must also be sky-subtracted before calling
-Segment. However, if a value/file is given to @option{--sky}, the convolved
-values will also be Sky subtracted internally. Alternatively, if you would
-prefer to give a kernel, so Segment does the convolution internally, you
-can use the @option{--kernel} and @option{--khdu}. To disable convolution,
-use @option{--kernel=none}.
+By default Segment will convolve the input with a kernel to improve the
+signal-to-noise ratio of true peaks. If you already have the convolved
+input dataset, you can pass it directly to Segment for faster processing
+(using the @option{--convolved} and @option{--chdu} options). Just don't
+forget that the convolved image must also be Sky-subtracted before calling
+Segment. If a value/file is given to @option{--sky}, the convolved values
+will also be Sky subtracted internally. Alternatively, if you prefer to
+give a kernel (with @option{--kernel} and @option{--khdu}), Segment can do
+the convolution internally. To disable convolution, use
address@hidden
@table @option
address@hidden --sky=STR
-The optional file name containing the Sky values dataset. The extension can
-be specified with @option{--skyhdu}. This dataset must either have the same
-size as the output or the same size as the tessellation (so there is one
-pixel per tile, see @ref{Tessellation}).
address@hidden --sky=STR/FLT
+The Sky value(s) to subtract from the input. This option can either be
+given a constant number or a file name containing a dataset (multiple
+values, per pixel or per tile). By default, Segment will assume the input
+dataset is Sky subtracted, so this option is not mandatory.
+
+If the value can't be read as a number, it is assumed to be a file
+name. When the value is a file, the extension can be specified with
address@hidden When its not a single number, the given dataset must
+either have the same size as the output or the same size as the
+tessellation (so there is one pixel per tile, see @ref{Tessellation}).
+
+When this option is given, its value(s) will be subtraced from the input
+and the (optional) convolved dataset (given to @option{--convolved}) prior
+to starting the segmentation process.
+
address@hidden --skyhdu=STR/INT
+The HDU/extension containing the Sky values. This is mandatory when the
+value given to @option{--sky} is not a number. Please see the description
+of @option{--hdu} in @ref{Input output options} for the different ways you
+can identify a special extension.
+
address@hidden --std=STR/FLT
+The Sky standard deviation value(s) corresponding to the input. The value
+can either be a constant number or a file name containing a dataset
+(multiple values, per pixel or per tile). The Sky standard deviation is
+mandatory for Segment to operate.
+
+If the value can't be read as a number, it is assumed to be a file
+name. When the value is a file, the extension can be specified with
address@hidden When its not a single number, the given dataset must
+either have the same size as the output or the same size as the
+tessellation (so there is one pixel per tile, see @ref{Tessellation}).
+
+When this option is not called, Segment will assume the standard deviation
+is a dataset and in a HDU/extension (@option{--stdhdu}) of another one of
+the input file(s). If a file is given to @option{--detection}, it will
+assume that file contains the standard deviation dataset, otherwise, it
+will look into input filename (the main argument, without any option).
-When this option is given, its values will be subtraced from the input and
-(the optional) dataset given to @option{--convolved} prior to starting the
-segmentation process.
address@hidden --stdhdu=INT/STR
+The HDU/extension containing the Sky standard deviation values, when the
+value given to @option{--std} is a file name. Please see the description of
address@hidden in @ref{Input output options} for the different ways you can
+identify a special extension.
address@hidden --skyhdu
-The HDU/extension containing the Sky value (when @option{--sky} is
-called). For acceptable values, please see the description of
address@hidden in @ref{Input output options}.
address@hidden --variance
+The input Sky standard deviation value/dataset is actually variance. When
+this option is called, the square root of input Sky standard deviation (see
address@hidden) is used internally, not its raw value(s).
@item -d STR
@itemx --detection=STR
-The file name containing the labeled dataset. The extension can be
-specified with @option{--dhdu}. Segmentation (clumps or objects) will only
-take place over the non-zero pixels of this image. The dataset must have
-the same size as the input image. Only datasets with an integer type are
-acceptable for the labeled image, see @ref{Numeric data types}.
-
-If your labeled image only has integer values, but it is stored in a
-floating point container, you can use Gnuastro's Arithmetic program (see
address@hidden) to convert it to an integer container, like the example
-below:
+Detection map to use for segmentation. If given a value of @option{all},
+Segment will assume the whole dataset must be segmented, see below. If a
+detection map is given, the extension can be specified with
address@hidden If not given, Segment will assume the desired
+HDU/extension is in the main input argument (input file specified with no
+option).
+
+Segmentation (clumps or objects) will only take place over the non-zero
+pixels of this detection map. The dataset must have the same size as the
+input image. Only datasets with an integer type are acceptable for the
+labeled image, see @ref{Numeric data types}. If your detection map only has
+integer values, but it is stored in a floating point container, you can use
+Gnuastro's Arithmetic program (see @ref{Arithmetic}) to convert it to an
+integer container, like the example below:
@example
$ astarithmetic float.fits int32 --output=int.fits
@end example
address@hidden --dhdu
-The HDU/extension containing the detection labels. For acceptable values,
-please see the description of @option{--hdu} in @ref{Input output options}.
+It may happen that the whole input dataset is covered by signal, for
+example when working on parts of the Andromeda galaxy, or nearby globular
+clusters (that cover the whole field of view). In such cases, segmentation
+is necessary over the complete dataset, not just specific regions
+(detections). By default Segment will first use the undetected regions as a
+reference to find the proper signal-to-noise ratio of ``true'' clumps (give
+a purity level specified with @option{--snquant}). Therefore, in such
+scenarios you also need to manually give a ``true'' clump signal-to-noise
+ratio with the @option{--clumpsnthresh} option to disable looking into the
+undetected regions, see @ref{Segmentation options}. In such cases, is
+possible to make a detection map that only has the value @code{1} for all
+pixels (for example using @ref{Arithmetic}), but for convenience, you can
+also use @option{--detection=all}.
address@hidden --std=STR
-The file name containing the Sky standard deviation dataset. The extension
-can be specified with @option{--stdhdu}. This dataset must either have the
-same size as the output or the same size as the tessellation (so there is
-one pixel per tile, see @ref{Tessellation}).
-
address@hidden --stdhdu=INT/STR
-The HDU/extension containing the Sky standard deviation. For acceptable
-values, please see the description of @option{--hdu} in @ref{Input output
-options}.
address@hidden --dhdu
+The HDU/extension containing the detection map given to
address@hidden Please see the description of @option{--hdu} in
address@hidden output options} for the different ways you can identify a
+special extension.
@item -k STR
@itemx --kernel=STR
The kernel used to convolve the input image. The usage of this option is
-identical to NoiseChisel's @option{--kernel} option (@ref{General
-NoiseChisel options}). Please see the descriptions there for more.
+identical to NoiseChisel's @option{--kernel} option (@ref{NoiseChisel
+input}). Please see the descriptions there for more. To disable
+convolution, you can give it a value of @option{none}.
@item --khdu
The HDU/extension containing the kernel used for convolution. For
@@ -14780,41 +14999,33 @@ acceptable values, please see the description of
@option{--hdu} in
@item --convolved
The convolved image to avoid internal convolution by Segment. The usage of
this option is identical to NoiseChisel's @option{--convolved} option
-(@ref{General NoiseChisel options}). Please see the descriptions there for
-more.
+(@ref{NoiseChisel input}). Please see the descriptions there for more.
@item --chdu
The HDU/extension containing the convolved image (given to
@option{--convolved}). For acceptable values, please see the description of
@option{--hdu} in @ref{Input output options}.
address@hidden --rawoutput
-Only output the clumps and objects labeled images. Without this option, the
-first and last extensions of the output will the sky-subtracted dataset and
-the Sky standard deviation dataset (that were inputs to Segment). When the
-datasets are small, this can help in conveniently using the output of
-Segment in @ref{MakeCatalog}. However, when the inputs are large, these
-extra dataset can be burden, and you can use this option to only make the
-raw images and provide the other necessary inputs to MakeCatalog (or any
-other software) with extra options. See @ref{Segment output} for more.
-
@item -L INT[,INT]
@itemx --largetilesize=INT[,INT]
-The size of the large tiles to use for identifying the clump S/N
-threshold. The usage of this option is identical to NoiseChisel's
address@hidden option (@ref{General NoiseChisel options}). Please
-see the descriptions there for more.
-
address@hidden --continueaftercheck
-Don't abort Segment after producing the check image(s). The usage of this
-option is identical to NoiseChisel's @option{--continueaftercheck} option
-(@ref{General NoiseChisel options}). Please see the descriptions there for
-more.
+The size of the large tiles to use for identifying the clump S/N threshold
+over the undetected regions. The usage of this option is identical to
+NoiseChisel's @option{--largetilesize} option (@ref{NoiseChisel
+input}). Please see the descriptions there for more.
+
+The undetected regions can be a significant fraction of the dataset and
+finding clumps requires sorting of the desired regions, which can be
+slow. To speed up the processing, Segment finds clumps in the undetected
+regions over separate large tiles. This allows it to have to sort a much
+smaller set of pixels and also to treat them independently and in
+parallel. Both these issues greatly speed it up. Just be sure to not
+decrease the large tile sizes too much (less than 100 pixels in each
+dimension). It is important for them to be much larger than the clumps.
@end table
address@hidden Segmentation options, Segment output, Segment inputs and general
settings, Invoking astsegment
address@hidden Segmentation options, Segment output, Segment input, Invoking
astsegment
@subsubsection Segmentation options
The options below can be used to configure every step of the segmentation
@@ -14952,13 +15163,6 @@ NoiseChisel. The `true' detections and clumps can be
objectively
identified from the noise characteristics of the image, so you don't
have to give any hand input Signal to noise ratio.
address@hidden --grownclumps
-In the output (see @ref{NoiseChisel output}) store the grown clumps (or
-full detected region if only one clump was present in that detection). By
-default the original clumps are stored as the third extension of the
-output, if this option is called, it is replaced with the grown clump
-labels.
-
@item --checksegmentation
A file with the suffix @file{_seg.fits} will be created. This file keeps
all the relevant steps in finding true clumps and segmenting the detections
@@ -14981,69 +15185,79 @@ The main output of Segment are two label datasets
(with integer types,
separating the dataset's elements into different classes) with
HDU/extension names of @code{CLUMPS} and @code{OBJECTS}. For a complete
definition of clumps and objects, please see Section 3.2 of
address@hidden://arxiv.org/abs/1505.01664, Akhlaghi and Ichikawa [2015]}.
address@hidden://arxiv.org/abs/1505.01664, Akhlaghi and Ichikawa [2015]} and
address@hidden options}.
The clumps are ``true'' local maxima and their surrounding pixels until a
local minimum (caused by noise fluctuations, or another ``true''
clump). Therefore it may happen that some of the input detections aren't
-covered by clumps at all (a very diffuse object without a strong central
-peak), while some objects may contain many clumps. Even in those that have
-clumps, there will be regions that are too diffuse. In other words, the
-peaks in those regions are too similar to the peaks in the un-detected
-regions. Those diffuse regions (within the input detected regions) are
-given a negative label (-1) to help you separate them from the undetected
-regions (with a value of zero).
-
-Each clump is labeled respective to its host object. Therefore, if an
+covered by clumps at all (very diffuse objects without any strong peak),
+while some objects may contain many clumps. Even in those that have clumps,
+there will be regions that are too diffuse. The diffuse regions (within the
+input detected regions) are given a negative label (-1) to help you
+separate them from the undetected regions (with a value of zero).
+
+Each clump is labeled with respect to its host object. Therefore, if an
object has three clumps for example, the clumps within it have labels 1, 2
-or 3. As a result, if an initial detected region has multiple objects, each
-with a single clump, all the clumps will have a label of 1. The total
+and 3. As a result, if an initial detected region has multiple objects,
+each with a single clump, all the clumps will have a label of 1. The total
number of clumps in the dataset is stored in the @code{NCLUMPS} keyword of
-the @code{CLUMPS} extension and printed in the verbose output of Segment.
-
-If the @option{--grownclumps} option is called (or a value of @code{1} is
-given to it in any of the configuration files) the grown clumps will be
-stored in the @code{CLUMPS} extension instead of the original clump
-regions. Note with @option{--grownclumps}, if there is only one clump, or
-no clumps, over a detected region, the whole detected region is given a
-label of 1.
+the @code{CLUMPS} extension and printed in the verbose output of Segment
+(when @option{--quiet} is not called).
The @code{OBJECTS} extension of the output will give a positive
-counter/label to every positive pixel in the input. As described in
-Akhlaghi and Ichikawa [2015], the clumps are grown and based on the
-signal-to-noise ratio of their connections, they may be considered as part
-of a single object.
+counter/label to every detected pixel in the input. As described in
+Akhlaghi and Ichikawa [2015], the true clumps are grown until a certain
+threshold. If the grown clumps touch other clumps and the connection is
+strong enough, they are considered part of the same @emph{object}. Once
+objects (grown clumps) are identified, they are grown to cover the whole
+detected area.
+
address@hidden DS9
address@hidden SAO DS9
+By default, besides the @code{CLUMPS} and @code{OBJECTS} extensions,
+Segment's output will also contain the (technically redundant) input
+dataset and the sky standard deviation dataset (if it wasn't a constant
+number). This can help in visually inspecting the result when viewing the
+images as a ``Multi-extension data cube'' in SAO DS9 for example (see
address@hidden multiextension FITS images}). You can simply flip through the
+exetensions and see the same region of the image and its corresponding
+clumps/object labels. It also makes it easy to feed the output (as one
+file) into MakeCatalog when you intend to make a catalog afterwards (see
address@hidden To remove these redundant extensions from the output
+(for example when designing a pipeline), you can use
address@hidden
The @code{OBJECTS} and @code{CLUMPS} extensions can be used as input into
@ref{MakeCatalog} to generate a catalog for higher-level analysis. If you
want to treat each clump separately, you can give a very large value (or
even a NaN, which will always fail) to the @option{--gthresh} option (for
example @code{--gthresh=1e10} or @code{--gthresh=nan}), see
address@hidden options}.
address@hidden options}. The options to configure the output of Segment
+are listed below:
address@hidden DS9
address@hidden SAO DS9
-By default, besides the @code{CLUMPS} and @code{OBJECTS} extensions, the
-output will be supplimented with the input dataset and the sky standard
-deviation dataset. This can help in visually inspecting the result when
-viewing the images as a ``Multi-extension data cube'' in SAO DS9 for
-example (see @ref{Viewing multiextension FITS images}). You can simply flip
-through the exetensions and see the same region of the image and its
-corresponding clumps/object labels. It also makes it easy to feed the
-output (as one file) into MakeCatalog when you intend to make a catalog
-afterwards (see @ref{MakeCatalog}.
-
-Therefore, these two default extra extensions of the output are redundant
-(you already had them before calling Segment) and just help in convenience
-when your dataset is small. However, as you start to expand your project to
-larger datasets or more images, the extra volume taken by these extra
-extensions can cause a large burden in your archives and processing. When
-this happens, you can use the @option{--rawoutput} option to avoid such
-excessive waste of precious space (see @ref{Segment inputs and general
-settings}). With this option, the output will only contain the
address@hidden and @code{OBJECTS} datasets. You can then use MakeCatalog's
address@hidden and @option{--clumpsfile} to let it know where to
-find these extensions.
address@hidden @option
address@hidden --continueaftercheck
+Don't abort Segment after producing the check image(s). The usage of this
+option is identical to NoiseChisel's @option{--continueaftercheck} option
+(@ref{NoiseChisel input}). Please see the descriptions there for more.
+
address@hidden --grownclumps
+In the output @code{CLUMPS} extension, store the grown clumps. If a
+detected region contains no clumps or only one clump, then it will be fully
+given a label of @code{1} (no negative valued pixels).
+
address@hidden --rawoutput
+Only write the @code{CLUMPS} and @code{OBJECTS} datasets in the output
+file. Without this option (by default), the first and last extensions of
+the output will the Sky-subtracted input dataset and the Sky standard
+deviation dataset (if it wasn't a number). When the datasets are small,
+these redundant extensions can make it convenient to inspect the results
+visually or feed the output to @ref{MakeCatalog} for
+measurements. Ultimately both the input and Sky standard deviation datasets
+are redundant (you had them before running Segment). When the inputs are
+large/numerous, these extra dataset can be a burden.
address@hidden table
@cartouche
@noindent
@@ -15062,14 +15276,8 @@ $ gzip --best segment_output.fits
@noindent
The resulting @file{.fits.gz} file can then be fed into any of Gnuastro's
programs directly, without having to decompress it separately (it will just
-take them a little longer, because they have to decompress it before use).
-
-Alternatively, you can also tell Segment to directly build a
address@hidden output (for example with
address@hidden). But CFITSIO (which reads and writes FITS
-files for Gnuastro) puts more emphasis on compression/decompression
-speed. So it doesn't use the most efficient compression settings (which can
-take slightly longer).
+take them a little longer, because they have to decompress it internally
+before use).
@end cartouche
@@ -15766,15 +15974,15 @@ $ astmkcatalog --ra --dec --magnitude --magnitudeerr
seg-out.fits
$ asmkcatalog -rdmG seg-out.fits
## Write the catalog to a text table:
-$ astmkcatalog -mpQ input_labeled.fits --output=cat.txt
+$ astmkcatalog -mpQ seg-out.fits --output=cat.txt
-## Read the columns to create from `columns.conf':
-$ astmkcatalog --config=columns.conf input_labeled.fits
+## Output columns specified in `columns.conf':
+$ astmkcatalog --config=columns.conf seg-out.fits
## Use object and clump labels from a K-band image, but pixel values
## from an i-band image.
-$ astmkcatalog K_labeled.fits --hdu=1 --clumpscat \
- --clumpsfile=K_labeled.fits --clumpshdu=2 \
+$ astmkcatalog K_segmented.fits --hdu=DETECTIONS --clumpscat \
+ --clumpsfile=K_segmented.fits --clumpshdu=CLUMPS \
--valuesfile=i_band.fits
@end example
@@ -15888,6 +16096,21 @@ options and general setting options are described
below.
@table @option
address@hidden -l STR
address@hidden --clumpsfile=STR
+The file containing the labeled clumps dataset when @option{--clumpscat} is
+called (see @ref{MakeCatalog output file}). When @option{--clumpscat} is
+called, but this option isn't, MakeCatalog will look into the main input
+file (given as an argument) for the required extension/HDU (value to
address@hidden).
+
address@hidden --clumpshdu=STR
+The HDU/extension of the clump labels dataset. Only pixels with values
+above zero will be considered. The clump lables dataset has to be an
+integer data type (see @ref{Numeric data types}) and only pixels with a
+value larger than zero will be used. See @ref{Segment output} for a
+description of the expected format.
+
@item -v STR
@itemx --valuesfile=STR
The file name of the (sky-subtracted) values dataset. When any of the
@@ -15904,91 +16127,57 @@ The name or number (counting from zero) of the
extension containing the
``values'' dataset, see the descriptions above and those in
@option{--valuesfile} for more.
address@hidden -l STR
address@hidden --clumpsfile=STR
-The file containing the labeled clumps dataset when @option{--clumpscat} is
-called. For more, see the description of @option{--clumpscat}. When
address@hidden is called, but this option isn't, MakeCatalog will
-look into the main input file (given as an argument) for the required
-(@option{--clumpshdu}).
address@hidden -s STR/FLT
address@hidden --insky=STR/FLT
+Sky value as a single number, or the file name containing a dataset
+(different values per pixel or tile). The Sky dataset is only necessary
+when @option{--subtractsky} is called or when a column directly related to
+the Sky value is requested (currently @option{--sky}).
address@hidden --clumpshdu=STR
-The HDU/extension of the clump labels dataset. Only pixels with values
-above zero will be considered. The clump lables dataset has to be an
-integer data type (see @ref{Numeric data types}) and only pixels with a
-value larger than zero will be used. See @ref{Segment output} for a
-description of the expected format.
+When the Sky dataset is necessary and this option is not called,
+MakeCatalog will assume it is a dataset first look into the
address@hidden (if it is given) and then the main input file (given
+as an argument). By default the values dataset is assumed to be already Sky
+subtracted, so this dataset is not necessary for many of the columns.
address@hidden -s STR
address@hidden --skyfile=STR
-File name of dataset keeping the Sky value for each pixel. The Sky dataset
-is only necessary when @option{--subtractsky} is called or when a column
-directly related to the Sky value is requested (currently
address@hidden). When the Sky dataset is necessary and this option is not
-called, MakeCatalog will first look into the @option{--valuesfile} (if it
-is given) and then the main input file (given as an argument). By default
-the values dataset is assumed to be already Sky subtracted, so this dataset
-is not necessary for many of the columns.
-
-This dataset may be a tessellation values dataset with one element per tile
-(see @option{--oneelempertile} of @ref{Processing options}).
+This dataset may be tessellation, with one element per tile (see
address@hidden of @ref{Processing options}).
@item --skyhdu=STR
HDU/extension of the Sky dataset, see @option{--skyfile}.
address@hidden -t STR
address@hidden --stdfile=STR
-File name of image keeping the @emph{Sky value} standard deviation for each
-pixel. With the @option{--variance} option you can tell MakeCatalog to
-interpret this image as a variance image, not standard deviation.
-
address@hidden note:} This must only be the Sky standard deviation or
-variance, without the signal's contribution. In other words, for each pixel
-that harbors signal, the final standard deviation is more than those
-without signal. MakeCatalog will find the amount of signal within each
-pixel using the Sky image and account for the extra error that it
-needs. Therefore if the input standard deviation (or variance) image also
-contains the contribution of signal to the error, then the final error
-measurements will be over-estimated.
address@hidden --subtractsky
+Subtract the sky value or dataset from the values file prior to any
+processing.
+
address@hidden -t STR/FLT
address@hidden --instd=STR/FLT
+Sky standard deviation value as a single number, or the file name
+containing a dataset (different values per pixel or tile). With the
address@hidden option you can tell MakeCatalog to interpret this
+value/dataset as a variance image, not standard deviation.
+
address@hidden note:} This must only be the SKY standard deviation or
+variance (not including the signal's contribution to the error). In other
+words, the final standard deviation of a pixel depends on how much signal
+there is in it. MakeCatalog will find the amount of signal within each
+pixel (while subtracting the Sky, if @option{--subtractsky} is called) and
+account for the extra error due to it's value (signal). Therefore if the
+input standard deviation (or variance) image also contains the contribution
+of signal to the error, then the final error measurements will be
+over-estimated.
@item --stdhdu=STR
The HDU of the Sky value standard deviation image.
address@hidden -z FLT
address@hidden --zeropoint=FLT
-The zero point magnitude for the input image, see @ref{Flux Brightness and
-magnitude}.
-
@item --variance
The dataset given to @option{--stdfile} (and @option{--stdhdu} has the Sky
variance of every pixel, not the Sky standard deviation.
address@hidden -C
address@hidden --clumpscat
-Produce a clumps catalog also. When this option is given, MakeCatalog will
-also look for secondary (sub-structure) labeled dataset and produce a
-catalog from that. If @option{--clumpsfile} isn't given, it will look into
-the main input argument/file for the HDU/extension containing the
-dataset. The extension can be specified with @option{--clumpshdu}. For more
-on the definition of ``clumps'', see @ref{Segment}.
-
address@hidden --sfmagnsigma=FLT
-The median standard deviation (from a @command{MEDSTD} keyword in the Sky
-standard deviation image) will be multiplied by the value to this option
-and its magnitude will be reported in the comments of the output
-catalog. This value is a per-pixel value, not per object/clump and is not
-found over an area or aperture, like the common @mymath{5\sigma} values
-that are commonly reported as a measure of depth or the upper-limit
-measurements (see @ref{Quantifying measurement limits}).
-
address@hidden --sfmagarea=FLT
-Area (in arcseconds squared) to convert the per-pixel estimation of
address@hidden in the comments section of the output tables. Note
-that this is just a unit conversion using the World Coordinate System (WCS)
-information in the input's header. It does not actually do any measurements
-on this area. For random measurements on any area, please use the
-upper-limit columns of MakeCatalog (see the discussion on upper-limit
-measurements in @ref{Quantifying measurement limits}).
address@hidden -z FLT
address@hidden --zeropoint=FLT
+The zero point magnitude for the input image, see @ref{Flux Brightness and
+magnitude}.
@end table
@@ -16473,28 +16662,86 @@ with the first FITS axis in degrees.
@node MakeCatalog output file, , MakeCatalog output columns, Invoking
astmkcatalog
@subsubsection MakeCatalog output file
-When a clump catalog is also desired, two catalogs will be made: one for
-the objects (suffixed with @file{_o.txt} or @file{_o.fits}) and another for
-the clumps (suffixed with @file{_c.txt} or @file{_c.fits}). Therefore if
-any value is given to the @option{--output} option, MakeCatalogs will
-replace these two suffixes with any existing suffix in the given value. If
-no output value is given, MakeCatalog will use the input name, see
address@hidden output}. The format of the output table is specified with
-the @option{--tableformat} option, see @ref{Input output options}.
-
-When MakeCatalog is run on multiple threads, the clumps catalog rows will
-not be sorted by object since each object is processed independently by one
-thread and threaded applications are asynchronous. The clumps in each
-object will be sorted based on their labels, but you will find lower-index
-objects come after higher-index ones (especially if they have more clumps
-and thus take more time). If the order is very important for you, you can
-run the following command to sort the rows by object ID (and clump ID with
-each object):
+When complete, MakeCatalog will store its measurements as a table. If an
+output filename is given (see @option{--output} in @ref{Input output
+options}), the format of the table will be deduced from the name. When it
+isn't given, the input name will be appended with a @file{_cat} suffix (see
address@hidden output}) and its format will be determined from the
address@hidden option, which is also discussed in @ref{Input output
+options}. @option{--tableformat} is also necessary when the requested
+output name is a FITS table (recall that FITS can accept ASCII and binary
+tables, see @ref{Table}).
+
+By default only a single catalog/table will be created for ``objects'',
+however, if @option{--clumpscat} is called, a secondary catalog/table will
+also be created. For more on ``objects'' and ``clumps'', see
address@hidden In short, if you only have one set of labeled images, you
+don't have to worry about clumps (they are deactivated by default). Here is
+a full description of MakeCatalog's output options:
+
address@hidden @option
address@hidden -C
address@hidden --clumpscat
+Do measurements on clumps and produce a second catalog (only devoted to
+clumps). When this option is given, MakeCatalog will also look for a
+secondary labeled dataset (identifying substructure) and produce a catalog
+from that. For more on the definition on ``clumps'', see @ref{Segment}.
+
+When the output is a FITS file, the objects and clumps catalogs/tables will
+be stored as multiple extensions of one FITS file. You can use @ref{Table}
+to inspect the column meta-data and contents in this case. However, in
+plain text format (see @ref{Gnuastro text table format}), it is only
+possible to keep one table per file. Therefore, if the output is a text
+file, two ouput files will be created, ending in @file{_o.txt} (for
+objects) and @file{_c.txt} (for clumps).
+
address@hidden --noclumpsort
+Don't sort the clumps catalog based on object ID (only relevant with
address@hidden). This option will benefit the
address@hidden performance boost due to @option{--noclumpsort}
+can only be felt when there are a huge number of objects. Therefore, by
+default the output is sorted to avoid miss-understandings or bugs in the
+user's scripts when the user forgets to sort the outputs.} of MakeCatalog
+when it is run on multiple threads @emph{and} the position of the rows in
+the clumps catalog is irrelevant (for example you just want the
+number-counts).
+
+MakeCatalog does all its measurements on each @emph{object} independently
+and in parallel. As a result, while it is writing the measurements on each
+object's clumps, it doesn't know how many clumps there were in previous
+objects. Each thread will just fetch the first available row and write the
+information of clumps (in order) starting from that row. After all the
+measurements are done, by default (when this option isn't called),
+MakeCatalog will reorder/permute the clumps catalog to have both the object
+and clump ID in an ascending order.
+
+If you would like to order the catalog later (when its a plain text file),
+you can run the following command to sort the rows by object ID (and clump
+ID within each object), assuming they are respectively the first and second
+columns:
@example
$ awk '!/^#/' out_c.txt | sort -g -k1,1 -k2,2
@end example
address@hidden --sfmagnsigma=FLT
+The median standard deviation (from a @command{MEDSTD} keyword in the Sky
+standard deviation image) will be multiplied by the value to this option
+and its magnitude will be reported in the comments of the output
+catalog. This value is a per-pixel value, not per object/clump and is not
+found over an area or aperture, like the common @mymath{5\sigma} values
+that are commonly reported as a measure of depth or the upper-limit
+measurements (see @ref{Quantifying measurement limits}).
+
address@hidden --sfmagarea=FLT
+Area (in arcseconds squared) to convert the per-pixel estimation of
address@hidden in the comments section of the output tables. Note
+that this is just a unit conversion using the World Coordinate System (WCS)
+information in the input's header. It does not actually do any measurements
+on this area. For random measurements on any area, please use the
+upper-limit columns of MakeCatalog (see the discussion on upper-limit
+measurements in @ref{Quantifying measurement limits}).
address@hidden table
diff --git a/lib/arithmetic-and.c b/lib/arithmetic-and.c
index bd00a61..3d144da 100644
--- a/lib/arithmetic-and.c
+++ b/lib/arithmetic-and.c
@@ -47,5 +47,5 @@ arithmetic_and(gal_data_t *l, gal_data_t *r, gal_data_t *o)
{
int checkblank=gal_arithmetic_binary_checkblank(l, r);
- BINARY_SET_LT( ARITHMETIC_BINARY_OUT_TYPE_UINT8, && );
+ BINARY_SET_LT( ARITHMETIC_BINARY_OUT_TYPE_INCR_SEP, && );
}
diff --git a/lib/arithmetic-or.c b/lib/arithmetic-or.c
index cd1dd41..716469e 100644
--- a/lib/arithmetic-or.c
+++ b/lib/arithmetic-or.c
@@ -47,5 +47,5 @@ arithmetic_or(gal_data_t *l, gal_data_t *r, gal_data_t *o)
{
int checkblank=gal_arithmetic_binary_checkblank(l, r);
- BINARY_SET_LT( ARITHMETIC_BINARY_OUT_TYPE_UINT8, || );
+ BINARY_SET_LT( ARITHMETIC_BINARY_OUT_TYPE_INCR_SEP, || );
}
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 31e4664..1d08315 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -53,12 +53,13 @@ if COND_HASGNULIBTOOL
endif
if COND_ARITHMETIC
MAYBE_ARITHMETIC_TESTS = arithmetic/snimage.sh arithmetic/onlynumbers.sh \
- arithmetic/where.sh arithmetic/or.sh
+ arithmetic/where.sh arithmetic/or.sh arithmetic/connected-components.sh
arithmetic/onlynumbers.sh: prepconf.sh.log
+ arithmetic/connected-components.sh: noisechisel/noisechisel.sh.log
arithmetic/snimage.sh: noisechisel/noisechisel.sh.log
arithmetic/where.sh: noisechisel/noisechisel.sh.log
- arithmetic/or.sh: noisechisel/noisechisel.sh.log
+ arithmetic/or.sh: segment/segment.sh.log
endif
if COND_BUILDPROG
MAYBE_BUILDPROG_TESTS = buildprog/simpleio.sh
@@ -128,7 +129,7 @@ if COND_MKCATALOG
mkcatalog/objects-clumps.sh mkcatalog/aperturephot.sh
mkcatalog/objects-clumps.sh: segment/segment.sh.log
- mkcatalog/detections.sh: noisechisel/noisechisel.sh.log
+ mkcatalog/detections.sh: arithmetic/connected-components.sh.log
mkcatalog/aperturephot.sh: noisechisel/noisechisel.sh.log \
mkprof/clearcanvas.sh.log
endif
diff --git a/tests/arithmetic/or.sh b/tests/arithmetic/connected-components.sh
similarity index 85%
copy from tests/arithmetic/or.sh
copy to tests/arithmetic/connected-components.sh
index 81efa6f..dfdf058 100755
--- a/tests/arithmetic/or.sh
+++ b/tests/arithmetic/connected-components.sh
@@ -1,4 +1,4 @@
-# Choose two detected regions with the `or' operator
+# Find the connected components in NoiseChisel's output.
#
# See the Tests subsection of the manual for a complete explanation
# (in the Installing gnuastro section).
@@ -24,7 +24,7 @@
# file exists (basicchecks.sh is in the source tree).
prog=arithmetic
execname=../bin/$prog/ast$prog
-img=convolve_spatial_noised_labeled.fits
+img=convolve_spatial_noised_detected.fits
@@ -49,4 +49,5 @@ if [ ! -f $img ]; then echo "$img does not exist.";
exit 77; fi
# Actual test script
# ==================
-$execname $img 6 eq $img 3 eq or -h2 -h2 --output=or.fits
+$execname $img 2 connected-components -hDETECTIONS \
+ --output=connected-components.fits
diff --git a/tests/arithmetic/or.sh b/tests/arithmetic/or.sh
index 81efa6f..9437258 100755
--- a/tests/arithmetic/or.sh
+++ b/tests/arithmetic/or.sh
@@ -24,7 +24,7 @@
# file exists (basicchecks.sh is in the source tree).
prog=arithmetic
execname=../bin/$prog/ast$prog
-img=convolve_spatial_noised_labeled.fits
+img=convolve_spatial_noised_detected_segmented.fits
@@ -49,4 +49,4 @@ if [ ! -f $img ]; then echo "$img does not exist.";
exit 77; fi
# Actual test script
# ==================
-$execname $img 6 eq $img 3 eq or -h2 -h2 --output=or.fits
+$execname $img 1 eq $img 3 eq or -gOBJECTS --output=or.fits
diff --git a/tests/arithmetic/snimage.sh b/tests/arithmetic/snimage.sh
index a42d95e..37dcb2d 100755
--- a/tests/arithmetic/snimage.sh
+++ b/tests/arithmetic/snimage.sh
@@ -25,7 +25,7 @@
prog=arithmetic
execname=../bin/$prog/ast$prog
imgin=convolve_spatial_noised.fits
-imgnc=convolve_spatial_noised_labeled.fits
+imgnc=convolve_spatial_noised_detected.fits
diff --git a/tests/arithmetic/where.sh b/tests/arithmetic/where.sh
index cb5acc1..a8948b0 100755
--- a/tests/arithmetic/where.sh
+++ b/tests/arithmetic/where.sh
@@ -24,7 +24,7 @@
# file exists (basicchecks.sh is in the source tree).
prog=arithmetic
execname=../bin/$prog/ast$prog
-img=convolve_spatial_noised_labeled.fits
+img=convolve_spatial_noised_detected.fits
diff --git a/tests/mkcatalog/aperturephot.sh b/tests/mkcatalog/aperturephot.sh
index 31466e7..c331029 100755
--- a/tests/mkcatalog/aperturephot.sh
+++ b/tests/mkcatalog/aperturephot.sh
@@ -25,7 +25,7 @@
prog=mkcatalog
objimg=clearcanvas.fits
execname=../bin/$prog/ast$prog
-img=convolve_spatial_noised_labeled.fits
+img=convolve_spatial_noised_detected.fits
diff --git a/tests/mkcatalog/detections.sh b/tests/mkcatalog/detections.sh
index d47c94a..5077dee 100755
--- a/tests/mkcatalog/detections.sh
+++ b/tests/mkcatalog/detections.sh
@@ -24,7 +24,8 @@
# file exists (basicchecks.sh is in the source tree).
prog=mkcatalog
execname=../bin/$prog/ast$prog
-img=convolve_spatial_noised_labeled.fits
+labels=connected-components.fits
+base=convolve_spatial_noised_detected.fits
@@ -40,8 +41,9 @@ img=convolve_spatial_noised_labeled.fits
#
# - The input data was not made (for example the test that created the
# data file failed).
-if [ ! -f $execname ]; then echo "$execname not created."; exit 77; fi
-if [ ! -f $img ]; then echo "$img does not exist."; exit 77; fi
+if [ ! -f $execname ]; then echo "$execname not created."; exit 77; fi
+if [ ! -f $labels ]; then echo "$labels does not exist."; exit 77; fi
+if [ ! -f $base ]; then echo "$base does not exist."; exit 77; fi
@@ -49,5 +51,5 @@ if [ ! -f $img ]; then echo "$img does not exist.";
exit 77; fi
# Actual test script
# ==================
-$execname $img --x --y --ra --dec --magnitude --upperlimitmag --sn \
- --tableformat=txt --hdu=DETECTIONS
+$execname $labels -h1 --valuesfile=$base --tableformat=txt \
+ --output=detections.txt --x --y --ra --dec --magnitude --sn
diff --git a/tests/mkcatalog/objects-clumps.sh
b/tests/mkcatalog/objects-clumps.sh
index 4fa0b46..987d5e7 100755
--- a/tests/mkcatalog/objects-clumps.sh
+++ b/tests/mkcatalog/objects-clumps.sh
@@ -24,7 +24,7 @@
# file exists (basicchecks.sh is in the source tree).
prog=mkcatalog
execname=../bin/$prog/ast$prog
-img=convolve_spatial_noised_labeled_segmented.fits
+img=convolve_spatial_noised_detected_segmented.fits
diff --git a/tests/segment/segment.sh b/tests/segment/segment.sh
index 6b9c038..995dfa3 100755
--- a/tests/segment/segment.sh
+++ b/tests/segment/segment.sh
@@ -24,7 +24,7 @@
# file exists (basicchecks.sh is in the source tree).
prog=segment
execname=../bin/$prog/ast$prog
-img=convolve_spatial_noised_labeled.fits
+img=convolve_spatial_noised_detected.fits