[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[gnuastro-commits] master b1e2eb54 68/69: Book: Extended PSF tutorial mo
|
From: |
Mohammad Akhlaghi |
|
Subject: |
[gnuastro-commits] master b1e2eb54 68/69: Book: Extended PSF tutorial moved to Tutorials chapter |
|
Date: |
Wed, 26 Jan 2022 12:39:17 -0500 (EST) |
branch: master
commit b1e2eb54d2c1226b6db1a59969bdc1d1ec456b1e
Author: Mohammad Akhlaghi <mohammad@akhlaghi.org>
Commit: Mohammad Akhlaghi <mohammad@akhlaghi.org>
Book: Extended PSF tutorial moved to Tutorials chapter
Until now, this tutorial was in the Installed scripts section. But it is
now growing into a much more complete set of instructions for new users.
With this commit, it has been moved into the Tutorials chapter, as the
fourth Gnuastro tutorial.
---
doc/gnuastro.texi | 11097 ++++++++++++++++++++++++++--------------------------
1 file changed, 5550 insertions(+), 5547 deletions(-)
diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index fccf084f..880aa18b 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -242,6 +242,7 @@ Tutorials
* Sufi simulates a detection:: Simulating a detection.
* General program usage tutorial:: Tutorial on many programs in generic
scenario.
* Detecting large extended targets:: NoiseChisel for huge extended targets.
+* Building the extended PSF::
Sufi simulates a detection
@@ -279,6 +280,15 @@ Detecting large extended targets
* Achieved surface brightness level:: Calculate the outer surface brightness.
* Extract clumps and objects:: Find sub-structure over the detections.
+Building the extended PSF
+
+* Preparing input for extended PSF:: Which stars should be used?
+* Saturated pixels and Segment's clumps:: Saturation is a major hurdle!
+* Building outer part of PSF:: Building the outermost PSF wings.
+* Inner parts of the PSF:: Going towards the PSF center.
+* Uniting the different PSF components:: Merging all the components into one
PSF.
+* Subtracting the PSF:: Having the PSF, we now want to subtract it.
+
Installation
* Dependencies:: Necessary packages for Gnuastro.
@@ -658,22 +668,12 @@ SAO DS9 region files from table
PSF construction and subtraction
* Overview of the PSF scripts:: Summary of concepts and methods
-* Tutorial on building the extended PSF:: Tutorial with real-world usage of
scripts
* Invoking astscript-psf-create-select-stars:: Select good starts within an
image.
* Invoking astscript-psf-create-make-stamp:: Make a stamp of each star to
stack.
* Invoking astscript-psf-create-junction:: Merge stacks of different regions
of PSF.
* Invoking astscript-psf-model-flux-factor:: Calculate factor to scale PSF to
star.
* Invoking astscript-psf-model-scattered-light:: Put the PSF in the image to
subtract.
-Tutorial on building the extended PSF
-
-* Preparing input for extended PSF:: Which stars should be used?
-* Saturated pixels and Segment's clumps:: Saturation is a major hurdle!
-* Building outer part of PSF:: Building the outermost PSF wings.
-* Inner parts of the PSF:: Going towards the PSF center.
-* Uniting the different PSF components:: Merging all the components into one
PSF.
-* Subtracting the PSF:: Having the PSF, we now want to subtract it.
-
Library
* Review of library fundamentals:: Guide on libraries and linking.
@@ -1673,6 +1673,7 @@ For an explanation of the conventions we use in the
example codes through the bo
* Sufi simulates a detection:: Simulating a detection.
* General program usage tutorial:: Tutorial on many programs in generic
scenario.
* Detecting large extended targets:: NoiseChisel for huge extended targets.
+* Building the extended PSF::
@end menu
@@ -4332,7 +4333,7 @@ $ astnoisechisel --cite
-@node Detecting large extended targets, , General program usage tutorial,
Tutorials
+@node Detecting large extended targets, Building the extended PSF, General
program usage tutorial, Tutorials
@section Detecting large extended targets
The outer wings of large and extended objects can sink into the noise very
gradually and can have a large variety of shapes (for example due to tidal
interactions).
@@ -5371,5194 +5372,5663 @@ See @ref{Segmentation and making a catalog} and
@ref{Segment} for more on using
+@node Building the extended PSF, , Detecting large extended targets, Tutorials
+@section Building the extended PSF
-@node Installation, Common program behavior, Tutorials, Top
-@chapter Installation
-
-@c This link is put here because the `Quick start' section of the first
-@c chapter is not the most eye-catching part of the manual and some users
-@c were seen to follow this ``Installation'' chapter title in search of the
-@c tarball and fast instructions.
-@cindex Installation
-The latest released version of Gnuastro source code is always available at the
following URL:
-
-@url{http://ftpmirror.gnu.org/gnuastro/gnuastro-latest.tar.gz}
-
-@noindent
-@ref{Quick start} describes the commands necessary to configure, build, and
install Gnuastro on your system.
-This chapter will be useful in cases where the simple procedure above is not
sufficient, for example your system lacks a mandatory/optional dependency (in
other words, you can't pass the @command{$ ./configure} step), or you want
greater customization, or you want to build and install Gnuastro from other
random points in its history, or you want a higher level of control on the
installation.
-Thus if you were happy with downloading the tarball and following @ref{Quick
start}, then you can safely ignore this chapter and come back to it in the
future if you need more customization.
-
-@ref{Dependencies} describes the mandatory, optional and bootstrapping
dependencies of Gnuastro.
-Only the first group are required/mandatory when you are building Gnuastro
using a tarball (see @ref{Release tarball}), they are very basic and low-level
tools used in most astronomical software, so you might already have them
installed, if not they are very easy to install as described for each.
-@ref{Downloading the source} discusses the two methods you can obtain the
source code: as a tarball (a significant snapshot in Gnuastro's history), or
the full history@footnote{@ref{Bootstrapping dependencies} are required if you
clone the full history.}.
-The latter allows you to build Gnuastro at any random point in its history
(for example to get bug fixes or new features that are not released as a
tarball yet).
-
-The building and installation of Gnuastro is heavily customizable, to learn
more about them, see @ref{Build and install}.
-This section is essentially a thorough explanation of the steps in @ref{Quick
start}.
-It discusses ways you can influence the building and installation.
-If you encounter any problems in the installation process, it is probably
already explained in @ref{Known issues}.
-In @ref{Other useful software} the installation and usage of some other free
software that are not directly required by Gnuastro but might be useful in
conjunction with it is discussed.
-
+Deriving the extended PSF of an image is very important in many aspects of the
analysis of the objects within it.
+Gnuastro has a set of installed scripts, designed to simplify the process
following the recipe of Infante-Sainz et al. (2020,
@url{https://arxiv.org/abs/1911.01430}); for more, see @ref{PSF construction
and subtraction}.
+An overview of the process is given in @ref{Overview of the PSF scripts}.
@menu
-* Dependencies:: Necessary packages for Gnuastro.
-* Downloading the source:: Ways to download the source code.
-* Build and install:: Configure, build and install Gnuastro.
+* Preparing input for extended PSF:: Which stars should be used?
+* Saturated pixels and Segment's clumps:: Saturation is a major hurdle!
+* Building outer part of PSF:: Building the outermost PSF wings.
+* Inner parts of the PSF:: Going towards the PSF center.
+* Uniting the different PSF components:: Merging all the components into one
PSF.
+* Subtracting the PSF:: Having the PSF, we now want to subtract it.
@end menu
+@node Preparing input for extended PSF, Saturated pixels and Segment's clumps,
Building the extended PSF, Building the extended PSF
+@subsection Preparing input for extended PSF
+In @ref{Overview of the PSF scripts}, we presented an overview of using the
separate scripts in sequence, to build an extended PSF.
+Before digging into the details of each script, we'll start here with a fully
working tutorial on how to use the scripts in order to construct a PSF, model
the scattered light field and subtract it from the image.
+For this particular example, an image of the M51 galaxy group in the r (SDSS)
band of the Javalambre Photometric Local Universe Survey (J-PLUS) used.
+For more information on J-PLUS, and its unique features visit:
@url{http://www.j-plus.es}.
+First, let's download the image from the J-PLUS webpage using @code{wget}.
+But to have a generalize-able, and easy to read command, we'll define some
base variables (in all-caps) first.
+After the download is complete, open the image with SAO DS9 (or any other FITS
viewer you prefer!) to have a feeling of the data (and of course, enjoy the
beauty of M51!).
+@example
+$ IMAGEID="jplus-dr2/get_fits?id=67510"
+$ URL="http://archive.cefca.es/catalogues/vo/siap/";
+$ wget $URL$IMAGEID -O image.fits.fz
+$ ds9 image.fits.fz -zoom to fit -zscale
+@end example
-@node Dependencies, Downloading the source, Installation, Installation
-@section Dependencies
-
-A minimal set of dependencies are mandatory for building Gnuastro from the
standard tarball release.
-If they are not present you cannot pass Gnuastro's configuration step.
-The mandatory dependencies are therefore very basic (low-level) tools which
are easy to obtain, build and install, see @ref{Mandatory dependencies} for a
full discussion.
-
-If you have the packages of @ref{Optional dependencies}, Gnuastro will have
additional functionality (for example converting FITS images to JPEG or PDF).
-If you are installing from a tarball as explained in @ref{Quick start}, you
can stop reading after this section.
-If you are cloning the version controlled source (see @ref{Version controlled
source}), an additional bootstrapping step is required before configuration and
its dependencies are explained in @ref{Bootstrapping dependencies}.
-
-Your operating system's package manager is an easy and convenient way to
download and install the dependencies that are already pre-built for your
operating system.
-In @ref{Dependencies from package managers}, we'll list some common operating
system package manager commands to install the optional and mandatory
dependencies.
-
-@menu
-* Mandatory dependencies:: Gnuastro will not install without these.
-* Optional dependencies:: Adding more functionality.
-* Bootstrapping dependencies:: If you have the version controlled source.
-* Dependencies from package managers:: Installing from OS package managers.
-@end menu
+Have a closer look at the edges of the image: zoom in to the corners.
+You see that on the edges the pixel values are either zero, or with
significantly different values than the main body of the image.
+This is due to the dithering pattern that was used to make this image.
+To avoid potential issues or problems that these regions may cause, we will
first crop out the main body of the image with the command below.
+To keep the top-level directory clean, let's also put the crop in a directory
called @file{flat}.
-@node Mandatory dependencies, Optional dependencies, Dependencies, Dependencies
-@subsection Mandatory dependencies
+@example
+$ mkdir flat
+$ astcrop image.fits.fz --mode=img --section=225:9275,150:9350 \
+ -oflat/image.fits
+$ ds9 flat/image.fits -zoom to fit -zscale
+@end example
-@cindex Dependencies, Gnuastro
-@cindex GNU build system
-The mandatory Gnuastro dependencies are very basic and low-level tools.
-They all follow the same basic GNU based build system (like that shown in
@ref{Quick start}), so even if you don't have them, installing them should be
pretty straightforward.
-In this section we explain each program and any specific note that might be
necessary in the installation.
+@noindent
+Please zoom into the edges again, you will see that they now have the same
noise-level as the rest of the image (the problematic parts are now gone).
+@node Saturated pixels and Segment's clumps, Building outer part of PSF,
Preparing input for extended PSF, Building the extended PSF
+@subsection Saturated pixels and Segment's clumps
+As explained in @ref{Overview of the PSF scripts}, it is important to mask
other sources in the image.
+Therefore, before going onto selecting stars, let's detect all significant
signal, and separate the clumps over the detections.
+However, the saturated pixels of the bright stars are going to cause problems.
+To see this problem, let's make a @mymath{1000\times1000} crop around a bright
star to speed up the test (and its solution):
-@menu
-* GNU Scientific Library:: Installing GSL.
-* CFITSIO:: C interface to the FITS standard.
-* WCSLIB:: C interface to the WCS standard of FITS.
-@end menu
+@example
+$ astcrop flat/image.fits --mode=wcs --widthinpix --width=1000 \
+ --center=203.3916736,46.7968652 --output=saturated.fits
+$ astnoisechisel saturated.fits --output=sat-nc.fits
+$ astsegment sat-nc.fits --output=sat-seg.fits
+$ ds9 -mecube -zscale sat-seg.fits -zoom to fit
+@end example
-@node GNU Scientific Library, CFITSIO, Mandatory dependencies, Mandatory
dependencies
-@subsubsection GNU Scientific library
+Have a look at the @code{CLUMPS} extension, you will see that instead of a
single clump at the center of the star, we have multiple!
+This is because of the saturated pixels!
+When saturation occurs, the peak of the profile is lost (like cutting off the
tip of a mountain!) and all saturated pixels get a noisy value close to the
saturation level.
+This disrupts Segment's assumption to expand clumps from local maxima (the
noise at the very high S/N level, results in each one being treated as a
separate clump).
+To have the center identified as a single clump, we should mask these
saturated pixels in a way that suites Segment's non-parametric methodology.
-@cindex GNU Scientific Library
-The @url{http://www.gnu.org/software/gsl/, GNU Scientific Library}, or GSL, is
a large collection of functions that are very useful in scientific
applications, for example integration, random number generation, and Fast
Fourier Transform among many others.
-To install GSL from source, you can run the following commands after you have
downloaded @url{http://ftpmirror.gnu.org/gsl/gsl-latest.tar.gz,
@file{gsl-latest.tar.gz}}:
+To find the saturation level, let's make a smaller crop of @mymath{50\times50}
pixels around the star with the command below, then have a look at the
distribution of pixels with a value larger than 100 (above the noise level):
@example
-$ tar xf gsl-latest.tar.gz
-$ cd gsl-X.X # Replace X.X with version number.
-$ ./configure
-$ make -j8 # Replace 8 with no. CPU threads.
-$ make check
-$ sudo make install
+$ astcrop image-crop.fits --mode=wcs --widthinpix --width=50 \
+ --center=203.3916736,46.7968652 --output=saturated-center.fits
+$ aststatistics saturated-center.fits --greaterequal=100
+Histogram:
+ |*
+ |*
+ |*
+ |* *
+ |** *
+ |*** *
+ |*** *
+ |***** **
+ |******** ****
+ |********** * * ** ***** *
+ |************************* ** * ***** * * * ** *** * *************
+ |----------------------------------------------------------------------
@end example
-@node CFITSIO, WCSLIB, GNU Scientific Library, Mandatory dependencies
-@subsubsection CFITSIO
-
-@cindex CFITSIO
-@cindex FITS standard
-@url{http://heasarc.gsfc.nasa.gov/fitsio/, CFITSIO} is the closest you can get
to the pixels in a FITS image while remaining faithful to the
@url{http://fits.gsfc.nasa.gov/fits_standard.html, FITS standard}.
-It is written by William Pence, the principal author of the FITS
standard@footnote{Pence, W.D. et al. Definition of the Flexible Image Transport
System (FITS), version 3.0. (2010) Astronomy and Astrophysics, Volume 524,
id.A42, 40 pp.}, and is regularly updated.
-Setting the definitions for all other software packages using FITS images.
+The peak you see in the right end (larger values) of the histogram shows the
saturated pixels.
+If there was no saturation, you would expect the number of pixels at
increasing values to simply decrease until reaching the maximum value of the
profile.
+But that is not the case here.
+Please try this experiment on a non-saturated star (decreasing the crop width)
to see what we mean.
+Finding the saturation level is easy with Statistics (by using the
@option{--lessthan} option until the histogram becomes as expected: only
decreasing).
+First let's try @option{--lessthan=3000} and then @option{--lessthan=2500}:
-@vindex --enable-reentrant
-@cindex Reentrancy, multiple file opening
-@cindex Multiple file opening, reentrancy
-Some GNU/Linux distributions have CFITSIO in their package managers, if it is
available and updated, you can use it.
-One problem that might occur is that CFITSIO might not be configured with the
@option{--enable-reentrant} option by the distribution.
-This option allows CFITSIO to open a file in multiple threads, it can thus
provide great speed improvements.
-If CFITSIO was not configured with this option, any program which needs this
capability will warn you and abort when you ask for multiple threads (see
@ref{Multi-threaded operations}).
+@example
+$ aststatistics saturated-center.fits --greaterequal=100 --lessthan=3000
+-------
+Histogram:
+ |*
+ |*
+ |*
+ |*
+ |**
+ |***
+ |*** *
+ |*** *
+ |******* **
+ |*********** * * * **
+ |********************* *** * *** * ***** *** * * *** * * ********
+ |----------------------------------------------------------------------
-To install CFITSIO from source, we strongly recommend that you have a look
through Chapter 2 (Creating the CFITSIO library) of the CFITSIO manual and
understand the options you can pass to @command{$ ./configure} (they aren't too
much).
-This is a very basic package for most astronomical software and it is best
that you configure it nicely with your system.
-Once you download the source and unpack it, the following configure script
should be enough for most purposes.
-Don't forget to read chapter two of the manual though, for example the second
option is only for 64bit systems.
-The manual also explains how to check if it has been installed correctly.
-CFITSIO comes with two executable files called @command{fpack} and
@command{funpack}.
-From their manual: they ``are standalone programs for compressing and
uncompressing images and tables that are stored in the FITS (Flexible Image
Transport System) data format.
-They are analogous to the gzip and gunzip compression programs except that
they are optimized for the types of astronomical images that are often stored
in FITS format''.
-The commands below will compile and install them on your system along with
CFITSIO.
-They are not essential for Gnuastro, since they are just wrappers for
functions within CFITSIO, but they can come in handy.
-The @command{make utils} command is only available for versions above 3.39, it
will build these executable files along with several other executable test
files which are deleted in the following commands before the installation
(otherwise the test files will also be installed).
+$ aststatistics saturated-center.fits --greaterequal=100 --lessthan=2500
+-------
+Histogram:
+ |*
+ |*
+ |*
+ |**
+ |**
+ |**
+ |****
+ |**** **
+ |*********
+ |************** ** * * * *
+ |******************** ***** *** * *** * *** ** * ** * * * **
+ |----------------------------------------------------------------------
+@end example
-The commands necessary to decompress, build and install CFITSIO from source
are described below.
-Let's assume you have downloaded
@url{http://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c/cfitsio_latest.tar.gz,
@file{cfitsio_latest.tar.gz}} and are in the same directory:
+@noindent
+We still see an increase in the histogram around 3000.
+Therefore, We can set the saturation level in this image@footnote{In raw
exposures, this value is usually around 65000 (close to @mymath{2^{16}}, since
most CCDs have 16-bit pixels). But that is not the case here, because this is a
processed/stacked image that has been calibrated.} to 2500.
+Let's mask all such pixels with the command below:
@example
-$ tar xf cfitsio_latest.tar.gz
-$ cd cfitsio-X.XX # Replace X.XX with version
-$ ./configure --prefix=/usr/local --enable-sse2 --enable-reentrant
-$ make
-$ make utils
-$ ./testprog > testprog.lis # See below if this has an error
-$ diff testprog.lis testprog.out # Should have no output
-$ cmp testprog.fit testprog.std # Should have no output
-$ rm cookbook fitscopy imcopy smem speed testprog
-$ sudo make install
+$ astarithmetic saturated.fits set-i i i 2500 gt nan where \
+ --output=sat-masked.fits
+$ ds9 sat-masked.fits
@end example
-In the @code{./testprog > testprog.lis} step, you may confront an error,
complaining that it can't find @file{libcfitsio.so.AAA} (where @code{AAA} is an
integer).
-This is the library that you just built and haven't yet installed.
-But unfortunately some versions of CFITSIO don't account for this on some OSs.
-To fix the problem, you need to tell your OS to also look into current CFITSIO
build directory with the first command below, afterwards, the problematic
command (second below) should run properly.
+Zoom into the center: you will see that the staturated pixels are indeed
masked.
+However, there is still another problem: on the edges of the vertical
``bleeding'' saturated pixels, there are strong negative values (almost like
``waves'').
+Such that the pixels just touching the saturated pixels have strong negative
values.
+These will also cause problems!
+So with the same command, let's dilate the saturated regions by four times.
@example
-$ export LD_LIBRARY_PATH="$(pwd):$LD_LIBRARY_PATH"
-$ ./testprog > testprog.lis
+$ astarithmetic saturated.fits set-i i i 2500 gt \
+ 2 dilate 2 dilate 2 dilate 2 dilate nan where \
+ --output=sat-masked.fits
+$ ds9 sat-masked.fits
@end example
-Recall that the modification above is ONLY NECESSARY FOR THIS STEP.
-@emph{Don't} put the @code{LD_LIBRARY_PATH} modification command in a
permanent place (like your bash startup file).
-After installing CFITSIO, close your terminal and continue working on a new
terminal (so @code{LD_LIBRARY_PATH} has its default value).
-For more on @code{LD_LIBRARY_PATH}, see @ref{Installation directory}.
-
+Now that saturated pixels (and their problematic neighbors) have been masked,
we can convolve the image (recall that Segment will use the convolved image for
identifing clumps) with the command below.
+However, we will use the Spatial Domain convolution which can account for
blank pixels (see @ref{Spatial vs. Frequency domain}).
+We will also create a Gaussian kernel with @mymath{\rm{FWHM}=2} pixels,
truncated at @mymath{5\times\rm{FWHM}}.
+@example
+$ astmkprof --kernel=gaussian,2,5 --oversample=1 -okernel.fits
+$ astconvolve sat-masked.fits --kernel=kernel.fits --domain=spatial \
+ --output=sat-masked-conv.fits
+@end example
+@noindent
+After convolution, the problematic pixels are still NaN.
+But Segment requires the profile to start with a maximum value and decrease.
+So before feeding into Segment, we'll fill the blank values with the maximum
value of the neighboring pixels (see @ref{Interpolation operators}):
+@example
+$ astarithmetic sat-masked-conv.fits 2 interpolate-maxofregion \
+ --output=sat-conv.fits
+$ ds9 sat-conv.fits
+@end example
-@node WCSLIB, , CFITSIO, Mandatory dependencies
-@subsubsection WCSLIB
+@noindent
+Open the output and have a look;
+The output image doesn't have blank pixels any more and each blank region (in
the centers of stars) is now filled with the largest value that is touching
that particular region.
+We can now feed this image to Segment as the convolved image:
-@cindex WCS
-@cindex WCSLIB
-@cindex World Coordinate System
-@url{http://www.atnf.csiro.au/people/mcalabre/WCS/, WCSLIB} is written and
maintained by one of the authors of the World Coordinate System (WCS)
definition in the @url{http://fits.gsfc.nasa.gov/fits_standard.html, FITS
standard}@footnote{Greisen E.W., Calabretta M.R. (2002) Representation of world
coordinates in FITS.
-Astronomy and Astrophysics, 395, 1061-1075.}, Mark Calabretta.
-It might be already built and ready in your distribution's package management
system.
-However, here the installation from source is explained, for the advantages of
installation from source please see @ref{Mandatory dependencies}.
-To install WCSLIB you will need to have CFITSIO already installed, see
@ref{CFITSIO}.
+@example
+$ astsegment sat-nc.fits --convolved=sat-conv.fits --output=sat-seg.fits
+$ ds9 -mecube -zscale sat-seg.fits -zoom to fit
+@end example
-@vindex --without-pgplot
-WCSLIB also has plotting capabilities which use PGPLOT (a plotting library for
C).
-If you wan to use those capabilities in WCSLIB, @ref{PGPLOT} provides the
PGPLOT installation instructions.
-However PGPLOT is old@footnote{As of early June 2016, its most recent version
was uploaded in February 2001.}, so its installation is not easy, there are
also many great modern WCS plotting tools (mostly in written in Python).
-Hence, if you will not be using those plotting functions in WCSLIB, you can
configure it with the @option{--without-pgplot} option as shown below.
+@noindent
+Please look at the @code{CLUMPS} extension.
+Do you see how the whole center of the star has indeed been identified as a
single clump?
+We thus achieved our aim and didn't let the saturated pixels harm the
identification of the center.
-If you have the cURL library @footnote{@url{https://curl.haxx.se}} on your
system and you installed CFITSIO version 3.42 or later, you will need to also
link with the cURL library at configure time (through the @code{-lcurl} option
as shown below).
-CFITSIO uses the cURL library for its HTTPS (or HTTP
Secure@footnote{@url{https://en.wikipedia.org/wiki/HTTPS}}) support and if it
is present on your system, CFITSIO will depend on it.
-Therefore, if @command{./configure} command below fails (you don't have the
cURL library), then remove this option and rerun it.
+Now we can extend it to the whole image.
+To detect signal, we can run NoiseChisel using the command below.
+In short:
+@itemize
+@item
+Since the image is so large, we increase @option{--interpnumngb} to get better
outlier statistics on the tiles.
+@item
+Since the image not the result of too many exposures, it doesn't have strong
correlated noise, so we'll decrease @option{--detgrowquant} and increase
@option{--detgrowmaxholesize}.
+@end itemize
+@noindent
+See @ref{Detecting large extended targets} for more on optimizing NoiseChisel.
+Furthermore, since both NoiseChisel and Segment need a convolved image, we'll
do the convolution before and feed it to both (to save running time).
+But first, let's delete all the temporary files we made above.
-Let's assume you have downloaded
@url{ftp://ftp.atnf.csiro.au/pub/software/wcslib/wcslib.tar.bz2,
@file{wcslib.tar.bz2}} and are in the same directory, to configure, build,
check and install WCSLIB follow the steps below.
@example
-$ tar xf wcslib.tar.bz2
-
-## In the `cd' command, replace `X.X' with version number.
-$ cd wcslib-X.X
-
-## If `./configure' fails, remove `-lcurl' and run again.
-$ ./configure LIBS="-pthread -lcurl -lm" --without-pgplot \
- --disable-fortran
-$ make
-$ make check
-$ sudo make install
+$ rm *.fits
+$ mkdir label
+$ astmkprof --kernel=gaussian,2,5 --oversample=1 \
+ -olabel/kernel.fits
+$ astarithmetic flat/image.fits set-i i i 2500 gt \
+ 2 dilate 2 dilate 2 dilate 2 dilate nan where \
+ --output=flat/sat-masked.fits
+$ astconvolve flat/sat-masked.fits --kernel=kernel.fits \
+ --domain=spatial --output=label/sat-masked-conv.fits
+$ astarithmetic label/sat-masked-conv.fits 2 interpolate-maxofregion \
+ --output=label/image-conv.fits
+$ astnoisechisel label/image-conv.fits --interpnumngb=100 \
+ --detgrowquant=0.8 --detgrowmaxholesize=100000 \
+ --convolved=label/image-conv.fits \
+ --output=label/nc.fits
+$ astsegment label/nc.fits --convolved=label/image-conv.fits \
+ --output=label/seg.fits
+$ ds9 -mecube -zscale label/nc.fits -zoom to fit
@end example
+Looking in the @code{SKY} extension of NoiseChisel, we see that there is no
footprint of the bright stars or of M51, so the detection was good!
+Furthermore, the saturated pixels haven't caused any problems and the central
clumps of bright stars are now a single clump.
+We can now proceed to estimating the outer PSF in @ref{Building outer part of
PSF}.
+But first, let's clean up behind our selves (we only need the Segment output)
+@example
+$ rm label/image-conv.fits label/kernel.fits label/sat-masked.fits \
+ label/sat-masked-conv.fits
+@end example
-@node Optional dependencies, Bootstrapping dependencies, Mandatory
dependencies, Dependencies
-@subsection Optional dependencies
-
-The libraries listed here are only used for very specific applications,
therefore they are optional and Gnuastro can be built without them (with only
those specific features disabled).
-Since these are pretty low-level tools, they are not too hard to install from
source, but you can also use your operating system's package manager to easily
install all of them.
-For more, see @ref{Dependencies from package managers}.
-
-@cindex GPL Ghostscript
-If the @command{./configure} script can't find any of these optional
dependencies, it will notify you of the operation(s) you can't do due to not
having them.
-If you continue the build and request an operation that uses a missing
library, Gnuastro's programs will warn that the optional library was missing at
build-time and abort.
-Since Gnuastro was built without that library, installing the library
afterwards won't help.
-The only way is to re-build Gnuastro from scratch (after the library has been
installed).
-However, for program dependencies (like cURL or GhostScript) things are
easier: you can install them after building Gnuastro also.
-This is because libraries are used to build the internal structure of
Gnuastro's executables.
-However, a program dependency is called by Gnuastro's programs at run-time and
has no effect on their internal structure.
-So if a dependency program becomes available later, it will be used next time
it is requested.
+@node Building outer part of PSF, Inner parts of the PSF, Saturated pixels and
Segment's clumps, Building the extended PSF
+@subsection Building outer part of PSF
+In @ref{Preparing input for extended PSF}, we described how to create a
Segment clump and object map, while accounting for saturated stars.
+So we are now ready to start building the outer parts of the PSF.
-@table @asis
+First we will build the outer parts of the PSF, so we want the brightest stars.
+You will see we have several bright stars in this very large field of view,
but we don't yet have a feeling how how many they are, and at what magnitudes.
+So let's use Gnuastro's Query program to find the magnitudes of the brightest
stars (those brighter than magnitude 12).
+For more on Query, see @ref{Query}.
-@item GNU Libtool
-@cindex GNU Libtool
-Libtool is a program to simplify managing of the libraries to build an
executable (a program).
-GNU Libtool has some added functionality compared to other implementations.
-If GNU Libtool isn't present on your system at configuration time, a warning
will be printed and @ref{BuildProgram} won't be built or installed.
-The configure script will look into your search path (@code{PATH}) for GNU
Libtool through the following executable names: @command{libtool} (acceptable
only if it is the GNU implementation) or @command{glibtool}.
-See @ref{Installation directory} for more on @code{PATH}.
+@example
+$ astquery gaia --dataset=edr3 --overlapwith=flat/image.fits \
+ --range=phot_g_mean_mag,-inf,12 --output=gaia.fits
+@end example
-GNU Libtool (the binary/executable file) is a low-level program that is
probably already present on your system, and if not, is available in your
operating system package manager@footnote{Note that we want the
binary/executable Libtool program which can be run on the command-line.
-In Debian-based operating systems which separate various parts of a package,
you want want @code{libtool-bin}, the @code{libtool} package won't contain the
executable program.}.
-If you want to install GNU Libtool's latest version from source, please visit
its @url{https://www.gnu.org/software/libtool/, web page}.
+Now, we can easily visualize the magnitude and positions of these stars using
@command{astscript-ds9-region} and the command below (for more on this script,
see @ref{SAO DS9 region files from table})
-Gnuastro's tarball is shipped with an internal implementation of GNU Libtool.
-Even if you have GNU Libtool, Gnuastro's internal implementation is used for
the building and installation of Gnuastro.
-As a result, you can still build, install and use Gnuastro even if you don't
have GNU Libtool installed on your system.
-However, this internal Libtool does not get installed.
-Therefore, after Gnuastro's installation, if you want to use
@ref{BuildProgram} to compile and link your own C source code which uses the
@ref{Gnuastro library}, you need to have GNU Libtool available on your system
(independent of Gnuastro).
-See @ref{Review of library fundamentals} to learn more about libraries.
+@example
+$ astscript-ds9-region gaia.fits -cra,dec \
+ --namecol=phot_g_mean_mag \
+ --command="ds9 flat/image.fits -zoom to fit -zscale"
+@end example
-@item libgit2
-@cindex Git
-@pindex libgit2
-@cindex Version control systems
-Git is one of the most common version control systems (see @ref{Version
controlled source}).
-When @file{libgit2} is present, and Gnuastro's programs are run within a
version controlled directory, outputs will contain the version number of the
working directory's repository for future reproducibility.
-See the @command{COMMIT} keyword header in @ref{Output FITS files} for a
discussion.
+You can see that we have several stars between magnitudes 6 to 10.
+Let's use @file{astscript-psf-create-select-stars} in the command below to
select the relevant stars in the image (the brightest; with a magnitude between
6 to 8).
+Since this will select very bright stars, we will also increase the distance
to nearby neighbors with brighter or similar magnitudes (the default value is 1
arcmin).
+To do this, we will set @option{--mindistdeg=0.05}, which corresponds to 3
arcmin (or 3/60 degrees).
-@item libjpeg
-@pindex libjpeg
-@cindex JPEG format
-libjpeg is only used by ConvertType to read from and write to JPEG images, see
@ref{Recognized file formats}.
-@url{http://www.ijg.org/, libjpeg} is a very basic library that provides tools
to read and write JPEG images, most Unix-like graphic programs and libraries
use it.
-Therefore you most probably already have it installed.
-@url{http://libjpeg-turbo.virtualgl.org/, libjpeg-turbo} is an alternative to
libjpeg.
-It uses Single instruction, multiple data (SIMD) instructions for ARM based
systems that significantly decreases the processing time of JPEG compression
and decompression algorithms.
+@example
+$ mkdir outer
+$ astscript-psf-create-select-stars flat/image.fits \
+ --magnituderange=6,8 \
+ --mindistdeg=0.05 \
+ --output=outer/stars.fits
+@end example
-@item libtiff
-@pindex libtiff
-@cindex TIFF format
-libtiff is used by ConvertType and the libraries to read TIFF images, see
@ref{Recognized file formats}.
-@url{http://www.simplesystems.org/libtiff/, libtiff} is a very basic library
that provides tools to read and write TIFF images, most Unix-like operating
system graphic programs and libraries use it.
-Therefore even if you don't have it installed, it must be easily available in
your package manager.
+@noindent
+Let's have a look at the selected stars in the image (it is very important to
visually check every step when you are first discovering a new dataset).
-@item cURL
-@cindex cURL (downloading tool)
-cURL's executable (@command{curl}) is called by @ref{Query} for submitting
queries to remote datasets and retrieving the results.
-It isn't necessary for the build of Gnuastro from source (only a warning will
be printed if it can't be found at configure time), so if you don't have it at
build-time there is no problem.
-Just be sure to have it when you run @command{astquery}, otherwise you'll get
an error about not finding @command{curl}.
+@example
+$ astscript-ds9-region outer/stars.fits -cra,dec \
+ --namecol=phot_g_mean_mag \
+ --command='ds9 flat/image.fits -zoom to fit -zscale'
+@end example
-@item GPL Ghostscript
-@cindex GPL Ghostscript
-GPL Ghostscript's executable (@command{gs}) is called by ConvertType to
compile a PDF file from a source PostScript file, see @ref{ConvertType}.
-Therefore its headers (and libraries) are not needed.
+Now that the catalog of good stars is ready, it is time to construct the
individual stamps for each star of the catalog.
+To do that, we'll use @file{astscript-psf-create-make-stamp}.
+One of the most important parameters for this script is the normalization
radii @code{--normradii}.
+This parameter defines a ring for the flux normalization of each star stamp.
+The normalization of the flux is necessary because each star has a different
brightness, and consequently, it is crucial for having all the stamps with the
same flux level in the same region.
+Otherwise the final stack of the different stamps would have no sense.
+Depending on the PSF shape, internal reflections, ghosts, saturated pixels,
and other systematics, it would be necessary to choose the @code{--normradii}
appropriately.
-@end table
+The selection of the normalization radii is something that requires a good
understanding of the data.
+To do that, let's use two useful parameters that will help us in the checking
of the data: @code{--tmpdir} and @code{--keeptmp}.
+With @code{--tmpdir=checking-normradii} all temporary files, including the
radial profiles, will be save in that directory (instead of an
internally-created name).
+With @code{--keeptmp} we won't remove the temporal files, so it is possible to
have a look at them (by default the temporary directory gets deleted at the
end).
+It is necessary to specify the @code{--normradii} even if we don't know yet
the final values.
+Otherwise the script will not generate the radial profile.
+As a consequence, in this step we put the normalization radii equal to the
size of the stamps.
+By doing this, the script will generate the radial profile of the entire stamp.
+In this particular step we set it to @code{--normradii=150,160}.
+@example
+$ counter=1
+$ mkdir finding-normradii
+$ asttable outer/stars.fits \
+ | while read -r ra dec mag; do
+ astscript-psf-create-make-stamp flat/sat-masked.fits \
+ --mode=wcs \
+ --stampwidth=500 \
+ --center=$ra,$dec \
+ --normradii=250,260 \
+ --segment=label/seg.fits \
+ --output=finding-normradii/$counter.fits \
+ --tmpdir=finding-normradii --keeptmp; \
+ counter=$((counter+1)); \
+ done
+@end example
+Now, by having a look at all radial profiles at the same time, it is possible
to choose a good normalization region where the profiles are similar.
+For example using Topcat or any other plotting tool.
+In the same way, it is always good to check the different stamps to ensure the
quality and posible two dimensional features that are difficult to detect from
the radial profiles (i.e., ghosts, internal reflections, etc.).
+@example
+$ topcat finding-normradii/rprofile*.fits
+$ ds9 finding-normradii/cropped-masked*.fits
+@end example
-@node Bootstrapping dependencies, Dependencies from package managers, Optional
dependencies, Dependencies
-@subsection Bootstrapping dependencies
+After some study of this data, we could say that a good normalization ring is
those pixels between R=50 and R=60 pixels.
+Such a ring ensures having a high number of pixels so the estimation of the
flux normalization will be robust.
+Also, at such distance from the center the signal to noise is high and there
are not obvious features that can affect the normalization.
-Bootstrapping is only necessary if you have decided to obtain the full version
controlled history of Gnuastro, see @ref{Version controlled source} and
@ref{Bootstrapping}.
-Using the version controlled source enables you to always be up to date with
the most recent development work of Gnuastro (bug fixes, new functionalities,
improved algorithms, etc).
-If you have downloaded a tarball (see @ref{Downloading the source}), then you
can ignore this subsection.
+We later need the normalization radii in the next steps also.
+Therefore, to avoid typos or chances of a mistake, we'll define the two
@code{NORMRADII_INNER} and @code{NORMRADII_OUTER} variables.
+Furthermore, since there are several stars (as you saw from the output of the
previous command), we iterate over each row of the catalog.
-To successfully run the bootstrapping process, there are some additional
dependencies to those discussed in the previous subsections.
-These are low level tools that are used by a large collection of Unix-like
operating systems programs, therefore they are most probably already available
in your system.
-If they are not already installed, you should be able to easily find them in
any GNU/Linux distribution package management system (@command{apt-get},
@command{yum}, @command{pacman}, etc).
-The short names in parenthesis in @command{typewriter} font after the package
name can be used to search for them in your package manager.
-For the GNU Portability Library, GNU Autoconf Archive and @TeX{} Live, it is
recommended to use the instructions here, not your operating system's package
manager.
-
-@table @asis
-
-@item GNU Portability Library (Gnulib)
-@cindex GNU C library
-@cindex Gnulib: GNU Portability Library
-@cindex GNU Portability Library (Gnulib)
-To ensure portability for a wider range of operating systems (those that don't
include GNU C library, namely glibc), Gnuastro depends on the GNU portability
library, or Gnulib.
-Gnulib keeps a copy of all the functions in glibc, implemented (as much as
possible) to be portable to other operating systems.
-The @file{bootstrap} script can automatically clone Gnulib (as a
@file{gnulib/} directory inside Gnuastro), however, as described in
@ref{Bootstrapping} this is not recommended.
+@example
+$ counter=1
+$ IMAGE=image-masked.fits
+$ NORMRADII_INNER=50
+$ NORMRADII_OUTER=60
+$ mkdir outer/stamps
+$ asttable outer/stars.fits \
+ | while read -r ra dec mag; do
+ astscript-psf-create-make-stamp flat/sat-masked.fits \
+ --mode=wcs \
+ --stampwidth=500 \
+ --center=$ra,$dec \
+ --segment=label/seg.fits \
+ --output=outer/stamps/$counter.fits \
+ --normradii=$NORMRADII_INNER,$NORMRADII_OUTER; \
+ counter=$((counter+1)); \
+ done
+@end example
-The recommended way to bootstrap Gnuastro is to first clone Gnulib and the
Autoconf archives (see below) into a local directory outside of Gnuastro.
-Let's call it @file{DEVDIR}@footnote{If you are not a developer in Gnulib or
Autoconf archives, @file{DEVDIR} can be a directory that you don't backup.
-In this way the large number of files in these projects won't slow down your
backup process or take bandwidth (if you backup to a remote server).} (which
you can set to any directory).
-Currently in Gnuastro, both Gnulib and Autoconf archives have to be cloned in
the same top directory@footnote{If you already have the Autoconf archives in a
separate directory, or can't clone it in the same directory as Gnulib, or you
have it with another directory name (not @file{autoconf-archive/}), you can
follow this short step.
-Set @file{AUTOCONFARCHIVES} to your desired address.
-Then define a symbolic link in @file{DEVDIR} with the following command so
Gnuastro's bootstrap script can find it:@*@command{$ ln -s $AUTOCONFARCHIVES
$DEVDIR/autoconf-archive}.} like the case here@footnote{If your internet
connection is active, but Git complains about the network, it might be due to
your network setup not recognizing the git protocol.
-In that case use the following URL for the HTTP protocol instead (for Autoconf
archives, replace the name): @command{http://git.sv.gnu.org/r/gnulib.git}}:
+After the stamps are created, we need to stack them together with a simple
Arithmetic command (see @ref{Stacking operators}).
+The stack is done using the median operator and the stacked image is the outer
PSF that we are looking for: @file{psf-outer.fits}.
@example
-$ DEVDIR=/home/yourname/Development
-$ cd $DEVDIR
-$ git clone https://git.sv.gnu.org/git/gnulib.git
-$ git clone https://git.sv.gnu.org/git/autoconf-archive.git
+$ astarithmetic outer/stamps/*.fits -g1 \
+ $(ls outer/stamps/*.fits | wc -l) median \
+ --output=outer/stack.fits
@end example
-Gnulib is a source-based dependency of Gnuastro's bootstrapping process, so
simply having it is enough on your computer, there is no need to install, and
thus check anything.
-
@noindent
-You now have the full version controlled source of these two repositories in
separate directories.
-Both these packages are regularly updated, so every once in a while, you can
run @command{$ git pull} within them to get any possible updates.
-
-@item GNU Automake (@command{automake})
-@cindex GNU Automake
-GNU Automake will build the @file{Makefile.in} files in each sub-directory
using the (hand-written) @file{Makefile.am} files.
-The @file{Makefile.in}s are subsequently used to generate the @file{Makefile}s
when the user runs @command{./configure} before building.
-
-To check that you have a working GNU Automake in your system, you can try this
command:
+With the command below, we'll compare this stacked PSF with the images of the
individual stars that were used to create it.
+You clearly see that the number of masked pixels is significantly decreased
and the PSF is much more ``cleaner'').
@example
-$ automake --version
+$ ds9 outer/stack.fits outer/stamps/*.fits
@end example
-@item GNU Autoconf (@command{autoconf})
-@cindex GNU Autoconf
-GNU Autoconf will build the @file{configure} script using the configurations
we have defined (hand-written) in @file{configure.ac}.
+However, the bleeding, vertical saturation in the center still remains.
+Also, because we didn't have too many images (only three!), some small regions
still remain that were (by chance!) masked in all three.
+If we had more bright stars in our selected magnitude range, we could have
filled those outer remaining patches.
-To check that you have a working GNU Autoconf in your system, you can try this
command:
+@c -------------------------------------------
+@c ------- This part may be removed ----------
+To fill those regions, we can use the radial profile of the stacked PSF.
+Obtaining the radial profile for a simple (square, with a single, circular
object), image is very easy:
@example
-$ autoconf --version
+$ astscript-radial-profile outer/stack.fits -oouter/stack-radial.fits
+$ echo "1 251 251 8 300 0 0 1 1 1" \
+ | astmkprof --background=outer/stack.fits --clearcanvas \
+ --customtable=outer/stack-radial.fits
+
@end example
+@c -------------------------------------------
-@item GNU Autoconf Archive
-@cindex GNU Autoconf Archive
-These are a large collection of tests that can be called to run at
@command{./configure} time.
-See the explanation under GNU Portability Library (Gnulib) above for
instructions on obtaining it and keeping it up to date.
+@node Inner parts of the PSF, Uniting the different PSF components, Building
outer part of PSF, Building the extended PSF
+@subsection Inner parts of the PSF
-GNU Autoconf Archive is a source-based dependency of Gnuastro's bootstrapping
process, so simply having it is enough on your computer, there is no need to
install, and thus check anything.
-Just don't forget that it has to be in the same directory as Gnulib (described
above).
+@node Uniting the different PSF components, Subtracting the PSF, Inner parts
of the PSF, Building the extended PSF
+@subsection Uniting the different PSF components
-@item GNU Texinfo (@command{texinfo})
-@cindex GNU Texinfo
-GNU Texinfo is the tool that formats this manual into the various output
formats.
-To bootstrap Gnuastro you need all of Texinfo's command-line programs.
-However, some operating systems package them separately, for example in
Fedora, @command{makeinfo} is packaged in the @command{texinfo-tex} package.
-To check that you have a working GNU Texinfo in your system, you can try this
command:
+@node Subtracting the PSF, , Uniting the different PSF components, Building
the extended PSF
+@subsection Subtracting the PSF
+With the commands above PSF has been constructed and now it is going to be
used for modeling and subtracting each individual star.
+By construction, the pixel values of the PSF came from the normalization of
the individual stamps.
+As a consequence, when modeling each individual star, it is necessary to
compute what is the factor by which the PSF has to be scaled.
+This is done with the script @file{astscript-psf-model-flux-factor}.
+For each star, a single value for the flux factor is saved into a plain text
file.
+In this case, the parameters where the computation of the flux factor is done
are the same to those that were used for creating the stamps.
@example
-$ makeinfo --version
+$ asttable catalog.fits | while read -r ra dec mag; do
+ astscript-psf-model-flux-factor $IMAGE \
+ --psf=psf.fits \
+ --mode=wcs \
+ --center=$ra,$dec \
+ --normradii=$NORMRADII_INNER,$NORMRADII_OUTER \
+ --output=fluxfactor-"$ra"_"$dec".txt; done
@end example
-@item GNU Libtool (@command{libtool})
-@cindex GNU Libtool
-GNU Libtool is in charge of building all the libraries in Gnuastro.
-The libraries contain functions that are used by more than one program and are
installed for use in other programs.
-They are thus put in a separate directory (@file{lib/}).
-
-To check that you have a working GNU Libtool in your system, you can try this
command (and from the output, make sure it is GNU's libtool)
+Once the flux factors have been computed, it is possible to scale the PSF to
have the desired flux level.
+Of course, it is not only to scale the PSF flux but also allocate it in the
correct position on the sky to model the star.
+To do this, we use the script @file{astscript-psf-model-scattered-light}.
+It will put the PSF into the coordinates specified by the user with the
appropiate size and flux level.
+For those pixels that there is no PSF (it is normal that the size of the image
is bigger than the size of the PSF), the script sets them automatically to zero.
@example
-$ libtool --version
+$ asttable catalog.fits | while read -r ra dec mag; do
+ fluxfactor=$(cat fluxfactor-"$ra"_"$dec".txt)
+ astscript-psf-model-scattered-light $IMAGE \
+ --psf=psf.fits \
+ --mode=wcs \
+ --fluxfactor $fluxfactor \
+ --center=$ra,$dec \
+ --output=starmodel-"$ra"_"$dec".fits; done
@end example
-@item GNU help2man (@command{help2man})
-@cindex GNU help2man
-GNU help2man is used to convert the output of the @option{--help} option
-(@ref{--help}) to the traditional Man page (@ref{Man pages}).
-
-To check that you have a working GNU Help2man in your system, you can try this
command:
+After all stars have been modeled using the PSF, the next step is to sum all
of them together.
+In this case, it is necessary to combine them using the operator sum because
each individual star has been flux calibrated, otherwise they would be averaged.
+The result is the complete scattered light field of the image using the PSF:
@file{scattered-light-field.fits}.
@example
-$ help2man --version
+$ astarithmetic starmodel*.fits -g1 \
+ $(ls starmodel*.fits | wc -l) sum \
+ --output=scattered-light-field.fits
@end example
-
-@item @LaTeX{} and some @TeX{} packages
-@cindex @LaTeX{}
-@cindex @TeX{} Live
-Some of the figures in this book are built by @LaTeX{} (using the PGF/TikZ
package).
-The @LaTeX{} source for those figures is version controlled for easy
maintenance not the actual figures.
-So the @file{./boostrap} script will run @LaTeX{} to build the figures.
-The best way to install @LaTeX{} and all the necessary packages is through
@url{https://www.tug.org/texlive/, @TeX{} live} which is a package manager for
@TeX{} related tools that is independent of any operating system.
-It is thus preferred to the @TeX{} Live versions distributed by your operating
system.
-
-To install @TeX{} Live, go to the web page and download the appropriate
installer by following the ``download'' link.
-Note that by default the full package repository will be downloaded and
installed (around 4 Giga Bytes) which can take @emph{very} long to download and
to update later.
-However, most packages are not needed by everyone, it is easier, faster and
better to install only the ``Basic scheme'' (consisting of only the most basic
@TeX{} and @LaTeX{} packages, which is less than 200 Mega bytes)@footnote{You
can also download the DVD iso file at a later time to keep as a backup for when
you don't have internet connection if you need a package.}.
-
-After the installation, be sure to set the environment variables as suggested
in the end of the outputs.
-Any time you confront (need) a package you don't have, simply install it with
a command like below (similar to how you install software from your operating
system's package manager)@footnote{After running @TeX{}, or @LaTeX{}, you might
get a warning complaining about a @file{missingfile}.
-Run `@command{tlmgr info missingfile}' to see the package(s) containing that
file which you can install.}.
-To install all the necessary @TeX{} packages for a successful Gnuastro
bootstrap, run this command:
+This is the final step.
+Once the scattered light field model has been obtained (i.e., each individual
star has been modeled using the PSF and they are in one single image), it is
possible to subtract this model from the original image.
+By doing this the scattered light field is removed.
+This step is done with a simple Arithmetic model to subtract two images.
@example
-$ su
-# tlmgr install epsf jknapltx caption biblatex biber iftex \
- etoolbox logreq xstring xkeyval pgf ms \
- xcolor pgfplots times rsfs ps2eps epspdf
+$ astarithmetic $IMAGE --hdu=1 \
+ scattered-light-field.fits --hdu=1 \
+ - --output subtracted.fits
@end example
-To check that you have a working @LaTeX{} executable in your system, you can
try this command (this just checks if @LaTeX{} exists, as described above, if
you have a missing package, you can easily identify it from the output and
install it with @command{tlmgr}.
+As always, check that each step has been done properly.
+It is normal that on the final image there are residuals and several parts
that have been not properly corrected.
+Take into account that the number of the stars used in this example is small.
+With more stars the PSF would have been better determined.
+Note also that during this process, any mask of contaminant sources have been
applied (either in the creation of the PSF nor in the calibration of the flux
of the PSF).
+This tutorial is intendeed to be a basic example from which the reader can
start to construct the PSF.
+To obtain better results it is necessary to study in details the peculiarities
of the datasets and consider other options that here they have not been used.
-@example
-$ latex --version
-@end example
-@item ImageMagick (@command{imagemagick})
-@cindex ImageMagick
-ImageMagick is a wonderful and robust program for image manipulation on the
command-line.
-@file{bootstrap} uses it to convert the book images into the formats necessary
for the various book formats.
-To check that you have a working ImageMagick in your system, you can try this
command:
-@example
-$ convert --version
-@end example
-@end table
+@node Installation, Common program behavior, Tutorials, Top
+@chapter Installation
-@node Dependencies from package managers, , Bootstrapping dependencies,
Dependencies
-@subsection Dependencies from package managers
+@c This link is put here because the `Quick start' section of the first
+@c chapter is not the most eye-catching part of the manual and some users
+@c were seen to follow this ``Installation'' chapter title in search of the
+@c tarball and fast instructions.
+@cindex Installation
+The latest released version of Gnuastro source code is always available at the
following URL:
-@cindex Package managers
-@cindex Source code building
-@cindex Building from source
-@cindex Compiling from source
-@cindex Source code compilation
-@cindex Distributions, GNU/Linux
-The most basic way to install a package on your system is to build the
packages from source yourself.
-Alternatively, you can use your operating system's package manager to download
pre-compiled files and install them.
-The latter choice is easier and faster.
-However, we recommend that you build the @ref{Mandatory dependencies} yourself
from source (all necessary commands and links are given in the respective
section).
-Here are some basic reasons behind this recommendation.
+@url{http://ftpmirror.gnu.org/gnuastro/gnuastro-latest.tar.gz}
-@enumerate
+@noindent
+@ref{Quick start} describes the commands necessary to configure, build, and
install Gnuastro on your system.
+This chapter will be useful in cases where the simple procedure above is not
sufficient, for example your system lacks a mandatory/optional dependency (in
other words, you can't pass the @command{$ ./configure} step), or you want
greater customization, or you want to build and install Gnuastro from other
random points in its history, or you want a higher level of control on the
installation.
+Thus if you were happy with downloading the tarball and following @ref{Quick
start}, then you can safely ignore this chapter and come back to it in the
future if you need more customization.
-@item
-Your operating system's pre-built software might not be the most recent
release.
-For example, Gnuastro itself is also packaged in some package managers.
-For the list see: @url{https://repology.org/project/gnuastro/versions}.
-You will notice that Gnuastro's version in some operating systems is more than
10 versions old!
-It is the same for all the dependencies of Gnuastro.
+@ref{Dependencies} describes the mandatory, optional and bootstrapping
dependencies of Gnuastro.
+Only the first group are required/mandatory when you are building Gnuastro
using a tarball (see @ref{Release tarball}), they are very basic and low-level
tools used in most astronomical software, so you might already have them
installed, if not they are very easy to install as described for each.
+@ref{Downloading the source} discusses the two methods you can obtain the
source code: as a tarball (a significant snapshot in Gnuastro's history), or
the full history@footnote{@ref{Bootstrapping dependencies} are required if you
clone the full history.}.
+The latter allows you to build Gnuastro at any random point in its history
(for example to get bug fixes or new features that are not released as a
tarball yet).
-@item
-For each package, Gnuastro might preform better (or require) certain
configuration options that your distribution's package managers didn't add for
you.
-If present, these configuration options are explained during the installation
of each in the sections below (for example in @ref{CFITSIO}).
-When the proper configuration has not been set, the programs should complain
and inform you.
+The building and installation of Gnuastro is heavily customizable, to learn
more about them, see @ref{Build and install}.
+This section is essentially a thorough explanation of the steps in @ref{Quick
start}.
+It discusses ways you can influence the building and installation.
+If you encounter any problems in the installation process, it is probably
already explained in @ref{Known issues}.
+In @ref{Other useful software} the installation and usage of some other free
software that are not directly required by Gnuastro but might be useful in
conjunction with it is discussed.
-@item
-For the libraries, they might separate the binary file from the header files
which can cause confusion, see @ref{Known issues}.
-@item
-Like any other tool, the science you derive from Gnuastro's tools highly
depend on these lower level dependencies, so generally it is much better to
have a close connection with them.
-By reading their manuals, installing them and staying up to date with
changes/bugs in them, your scientific results and understanding (of what is
going on, and thus how you interpret your scientific results) will also
correspondingly improve.
-@end enumerate
+@menu
+* Dependencies:: Necessary packages for Gnuastro.
+* Downloading the source:: Ways to download the source code.
+* Build and install:: Configure, build and install Gnuastro.
+@end menu
-Based on your package manager, you can use any of the following commands to
install the mandatory and optional dependencies.
-If your package manager isn't included in the list below, please send us the
respective command, so we add it.
-As discussed above, we recommend installing the @emph{mandatory} dependencies
manually from source (see @ref{Mandatory dependencies}).
-Therefore, in each command below, first the optional dependencies are given.
-The mandatory dependencies are included after an empty line.
-If you would also like to install the mandatory dependencies with your package
manager, just ignore the empty line.
-For better archivability and compression ratios, Gnuastro's recommended
tarball compression format is with the @url{http://lzip.nongnu.org/lzip.html,
Lzip} program, see @ref{Release tarball}.
-Therefore, the package manager commands below also contain Lzip.
-@table @asis
-@item @command{apt-get} (Debian-based OSs: Debian, Ubuntu, Linux Mint, etc)
-@cindex Debian
-@cindex Ubuntu
-@cindex Linux Mint
-@cindex @command{apt-get}
-@cindex Advanced Packaging Tool (APT, Debian)
-@url{https://en.wikipedia.org/wiki/Debian,Debian} is one of the oldest
-GNU/Linux
-distributions@footnote{@url{https://en.wikipedia.org/wiki/List_of_Linux_distributions#Debian-based}}.
-It thus has a very extended user community and a robust internal structure and
standards.
-All of it is free software and based on the work of volunteers around the
world.
-Many distributions are thus derived from it, for example Ubuntu and Linux Mint.
-This arguably makes Debian-based OSs the largest, and most used, class of
GNU/Linux distributions.
-All of them use Debian's Advanced Packaging Tool (APT, for example
@command{apt-get}) for managing packages.
-@example
-$ sudo apt-get install ghostscript libtool-bin libjpeg-dev \
- libtiff-dev libgit2-dev curl lzip \
- \
- libgsl0-dev libcfitsio-dev wcslib-dev
-@end example
+@node Dependencies, Downloading the source, Installation, Installation
+@section Dependencies
-@noindent
-Gnuastro is @url{https://tracker.debian.org/pkg/gnuastro,packaged} in Debian
(and thus some of its derivate operating systems).
-Just make sure it is the most recent version.
+A minimal set of dependencies are mandatory for building Gnuastro from the
standard tarball release.
+If they are not present you cannot pass Gnuastro's configuration step.
+The mandatory dependencies are therefore very basic (low-level) tools which
are easy to obtain, build and install, see @ref{Mandatory dependencies} for a
full discussion.
-@item @command{dnf}
-@itemx @command{yum} (Red Hat-based OSs: Red Hat, Fedora, CentOS, Scientific
Linux, etc)
-@cindex RHEL
-@cindex Fedora
-@cindex CentOS
-@cindex Red Hat
-@cindex @command{dnf}
-@cindex @command{yum}
-@cindex Scientific Linux
-@url{https://en.wikipedia.org/wiki/Red_Hat,Red Hat Enterprise Linux} (RHEL) is
released by Red Hat Inc.
-RHEL requires paid subscriptions for use of its binaries and support.
-But since it is free software, many other teams use its code to spin-off their
own distributions based on RHEL.
-Red Hat-based GNU/Linux distributions initially used the ``Yellowdog Updated,
Modifier'' (YUM) package manager, which has been replaced by ``Dandified yum''
(DNF).
-If the latter isn't available on your system, you can use @command{yum}
instead of @command{dnf} in the command below.
-@example
-$ sudo dnf install ghostscript libtool libjpeg-devel \
- libtiff-devel libgit2-devel lzip curl \
- \
- gsl-devel cfitsio-devel wcslib-devel
-@end example
+If you have the packages of @ref{Optional dependencies}, Gnuastro will have
additional functionality (for example converting FITS images to JPEG or PDF).
+If you are installing from a tarball as explained in @ref{Quick start}, you
can stop reading after this section.
+If you are cloning the version controlled source (see @ref{Version controlled
source}), an additional bootstrapping step is required before configuration and
its dependencies are explained in @ref{Bootstrapping dependencies}.
-@item @command{brew} (macOS)
-@cindex macOS
-@cindex Homebrew
-@cindex MacPorts
-@cindex @command{brew}
-@url{https://en.wikipedia.org/wiki/MacOS,macOS} is the operating system used
on Apple devices.
-macOS does not come with a package manager pre-installed, but several widely
used, third-party package managers exist, such as Homebrew or MacPorts.
-Both are free software.
-Currently we have only tested Gnuastro's installation with Homebrew as
described below.
+Your operating system's package manager is an easy and convenient way to
download and install the dependencies that are already pre-built for your
operating system.
+In @ref{Dependencies from package managers}, we'll list some common operating
system package manager commands to install the optional and mandatory
dependencies.
-If not already installed, first obtain Homebrew by following the instructions
at @url{https://brew.sh}.
-Homebrew manages packages in different `taps'.
-To install WCSLIB (discussed in @ref{Mandatory dependencies}) via Homebrew you
will need to @command{tap} into @command{brewsci/science} first (the tap may
change in the future, but can be found by calling @command{brew search wcslib}).
-@example
-$ brew install ghostscript libtool libjpeg libtiff \
- libgit2 curl lzip \
- \
- gsl cfitsio
-$ brew tap brewsci/science
-$ brew install wcslib
-@end example
+@menu
+* Mandatory dependencies:: Gnuastro will not install without these.
+* Optional dependencies:: Adding more functionality.
+* Bootstrapping dependencies:: If you have the version controlled source.
+* Dependencies from package managers:: Installing from OS package managers.
+@end menu
-@item @command{pacman} (Arch Linux)
-@cindex Arch Linux
-@cindex @command{pacman}
-@url{https://en.wikipedia.org/wiki/Arch_Linux,Arch Linux} is a smaller
GNU/Linux distribution, which follows the KISS principle (``keep it simple,
stupid'') as a general guideline.
-It ``focuses on elegance, code correctness, minimalism and simplicity, and
expects the user to be willing to make some effort to understand the system's
operation''.
-Arch Linux uses ``Package manager'' (Pacman) to manage its packages/components.
-@example
-$ sudo pacman -S ghostscript libtool libjpeg libtiff \
- libgit2 curl lzip \
- \
- gsl cfitsio wcslib
-@end example
+@node Mandatory dependencies, Optional dependencies, Dependencies, Dependencies
+@subsection Mandatory dependencies
-@item @command{zypper} (openSUSE and SUSE Linux Enterprise Server)
-@cindex openSUSE
-@cindex SUSE Linux Enterprise Server
-@cindex @command{zypper}, OpenSUSE package manager
-SUSE Linux Enterprise
Server@footnote{@url{https://www.suse.com/products/server}} (SLES) is the
commercial offering which shares code and tools.
-Many additional packages are offered in the Build
Service@footnote{@url{https://build.opensuse.org}}.
-openSUSE and SLES use @command{zypper} (cli) and YaST (GUI) for managing
repositories and
-packages.
+@cindex Dependencies, Gnuastro
+@cindex GNU build system
+The mandatory Gnuastro dependencies are very basic and low-level tools.
+They all follow the same basic GNU based build system (like that shown in
@ref{Quick start}), so even if you don't have them, installing them should be
pretty straightforward.
+In this section we explain each program and any specific note that might be
necessary in the installation.
-@example
-$ sudo zypper install ghostscript_any libtool pkgconfig \
- libcurl-devel libgit2-devel libjpeg62-devel \
- libtiff-devel curl \
- \
- gsl-devel cfitsio-devel wcslib-devel
-@end example
-@noindent
-When building Gnuastro, run the configure script with the following
@code{CPPFLAGS} environment variable:
-@example
-$ ./configure CPPFLAGS="-I/usr/include/cfitsio"
-@end example
+@menu
+* GNU Scientific Library:: Installing GSL.
+* CFITSIO:: C interface to the FITS standard.
+* WCSLIB:: C interface to the WCS standard of FITS.
+@end menu
-@c Gnuastro is @url{https://software.opensuse.org/package/gnuastro,packaged}
-@c in @command{zypper}. Just make sure it is the most recent version.
-@end table
+@node GNU Scientific Library, CFITSIO, Mandatory dependencies, Mandatory
dependencies
+@subsubsection GNU Scientific library
-Usually, when libraries are installed by operating system package managers,
there should be no problems when configuring and building other programs from
source (that depend on the libraries: Gnuastro in this case).
-However, in some special conditions, problems may pop-up during the
configuration, building, or checking/running any of Gnuastro's programs.
-The most common of such problems and their solution are discussed below.
+@cindex GNU Scientific Library
+The @url{http://www.gnu.org/software/gsl/, GNU Scientific Library}, or GSL, is
a large collection of functions that are very useful in scientific
applications, for example integration, random number generation, and Fast
Fourier Transform among many others.
+To install GSL from source, you can run the following commands after you have
downloaded @url{http://ftpmirror.gnu.org/gsl/gsl-latest.tar.gz,
@file{gsl-latest.tar.gz}}:
-@cartouche
-@noindent
-@strong{Not finding library during configuration:} If a library is installed,
but during Gnuastro's @command{configure} step the library isn't found, then
configure Gnuastro like the command below (correcting @file{/path/to/lib}).
-For more, see @ref{Known issues} and @ref{Installation directory}.
@example
-$ ./configure LDFLAGS="-L/path/to/lib"
+$ tar xf gsl-latest.tar.gz
+$ cd gsl-X.X # Replace X.X with version number.
+$ ./configure
+$ make -j8 # Replace 8 with no. CPU threads.
+$ make check
+$ sudo make install
@end example
-@end cartouche
-@cartouche
-@noindent
-@strong{Not finding header (.h) files while building:} If a library is
installed, but during Gnuastro's @command{make} step, the library's header
(file with a @file{.h} suffix) isn't found, then configure Gnuastro like the
command below (correcting @file{/path/to/include}).
-For more, see @ref{Known issues} and @ref{Installation directory}.
+@node CFITSIO, WCSLIB, GNU Scientific Library, Mandatory dependencies
+@subsubsection CFITSIO
+
+@cindex CFITSIO
+@cindex FITS standard
+@url{http://heasarc.gsfc.nasa.gov/fitsio/, CFITSIO} is the closest you can get
to the pixels in a FITS image while remaining faithful to the
@url{http://fits.gsfc.nasa.gov/fits_standard.html, FITS standard}.
+It is written by William Pence, the principal author of the FITS
standard@footnote{Pence, W.D. et al. Definition of the Flexible Image Transport
System (FITS), version 3.0. (2010) Astronomy and Astrophysics, Volume 524,
id.A42, 40 pp.}, and is regularly updated.
+Setting the definitions for all other software packages using FITS images.
+
+@vindex --enable-reentrant
+@cindex Reentrancy, multiple file opening
+@cindex Multiple file opening, reentrancy
+Some GNU/Linux distributions have CFITSIO in their package managers, if it is
available and updated, you can use it.
+One problem that might occur is that CFITSIO might not be configured with the
@option{--enable-reentrant} option by the distribution.
+This option allows CFITSIO to open a file in multiple threads, it can thus
provide great speed improvements.
+If CFITSIO was not configured with this option, any program which needs this
capability will warn you and abort when you ask for multiple threads (see
@ref{Multi-threaded operations}).
+
+To install CFITSIO from source, we strongly recommend that you have a look
through Chapter 2 (Creating the CFITSIO library) of the CFITSIO manual and
understand the options you can pass to @command{$ ./configure} (they aren't too
much).
+This is a very basic package for most astronomical software and it is best
that you configure it nicely with your system.
+Once you download the source and unpack it, the following configure script
should be enough for most purposes.
+Don't forget to read chapter two of the manual though, for example the second
option is only for 64bit systems.
+The manual also explains how to check if it has been installed correctly.
+
+CFITSIO comes with two executable files called @command{fpack} and
@command{funpack}.
+From their manual: they ``are standalone programs for compressing and
uncompressing images and tables that are stored in the FITS (Flexible Image
Transport System) data format.
+They are analogous to the gzip and gunzip compression programs except that
they are optimized for the types of astronomical images that are often stored
in FITS format''.
+The commands below will compile and install them on your system along with
CFITSIO.
+They are not essential for Gnuastro, since they are just wrappers for
functions within CFITSIO, but they can come in handy.
+The @command{make utils} command is only available for versions above 3.39, it
will build these executable files along with several other executable test
files which are deleted in the following commands before the installation
(otherwise the test files will also be installed).
+
+The commands necessary to decompress, build and install CFITSIO from source
are described below.
+Let's assume you have downloaded
@url{http://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c/cfitsio_latest.tar.gz,
@file{cfitsio_latest.tar.gz}} and are in the same directory:
+
@example
-$ ./configure CPPFLAGS="-I/path/to/include"
+$ tar xf cfitsio_latest.tar.gz
+$ cd cfitsio-X.XX # Replace X.XX with version
+$ ./configure --prefix=/usr/local --enable-sse2 --enable-reentrant
+$ make
+$ make utils
+$ ./testprog > testprog.lis # See below if this has an error
+$ diff testprog.lis testprog.out # Should have no output
+$ cmp testprog.fit testprog.std # Should have no output
+$ rm cookbook fitscopy imcopy smem speed testprog
+$ sudo make install
@end example
-@end cartouche
-@cartouche
-@noindent
-@strong{Gnuastro's programs don't run during check or after install:}
-If a library is installed, but the programs don't run due to linking problems,
set the @code{LD_LIBRARY_PATH} variable like below (assuming Gnuastro is
installed in @file{/path/to/installed}).
-For more, see @ref{Known issues} and @ref{Installation directory}.
+In the @code{./testprog > testprog.lis} step, you may confront an error,
complaining that it can't find @file{libcfitsio.so.AAA} (where @code{AAA} is an
integer).
+This is the library that you just built and haven't yet installed.
+But unfortunately some versions of CFITSIO don't account for this on some OSs.
+To fix the problem, you need to tell your OS to also look into current CFITSIO
build directory with the first command below, afterwards, the problematic
command (second below) should run properly.
+
@example
-$ export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/path/to/installed/lib"
+$ export LD_LIBRARY_PATH="$(pwd):$LD_LIBRARY_PATH"
+$ ./testprog > testprog.lis
@end example
-@end cartouche
+Recall that the modification above is ONLY NECESSARY FOR THIS STEP.
+@emph{Don't} put the @code{LD_LIBRARY_PATH} modification command in a
permanent place (like your bash startup file).
+After installing CFITSIO, close your terminal and continue working on a new
terminal (so @code{LD_LIBRARY_PATH} has its default value).
+For more on @code{LD_LIBRARY_PATH}, see @ref{Installation directory}.
+@node WCSLIB, , CFITSIO, Mandatory dependencies
+@subsubsection WCSLIB
+@cindex WCS
+@cindex WCSLIB
+@cindex World Coordinate System
+@url{http://www.atnf.csiro.au/people/mcalabre/WCS/, WCSLIB} is written and
maintained by one of the authors of the World Coordinate System (WCS)
definition in the @url{http://fits.gsfc.nasa.gov/fits_standard.html, FITS
standard}@footnote{Greisen E.W., Calabretta M.R. (2002) Representation of world
coordinates in FITS.
+Astronomy and Astrophysics, 395, 1061-1075.}, Mark Calabretta.
+It might be already built and ready in your distribution's package management
system.
+However, here the installation from source is explained, for the advantages of
installation from source please see @ref{Mandatory dependencies}.
+To install WCSLIB you will need to have CFITSIO already installed, see
@ref{CFITSIO}.
+@vindex --without-pgplot
+WCSLIB also has plotting capabilities which use PGPLOT (a plotting library for
C).
+If you wan to use those capabilities in WCSLIB, @ref{PGPLOT} provides the
PGPLOT installation instructions.
+However PGPLOT is old@footnote{As of early June 2016, its most recent version
was uploaded in February 2001.}, so its installation is not easy, there are
also many great modern WCS plotting tools (mostly in written in Python).
+Hence, if you will not be using those plotting functions in WCSLIB, you can
configure it with the @option{--without-pgplot} option as shown below.
-@node Downloading the source, Build and install, Dependencies, Installation
-@section Downloading the source
+If you have the cURL library @footnote{@url{https://curl.haxx.se}} on your
system and you installed CFITSIO version 3.42 or later, you will need to also
link with the cURL library at configure time (through the @code{-lcurl} option
as shown below).
+CFITSIO uses the cURL library for its HTTPS (or HTTP
Secure@footnote{@url{https://en.wikipedia.org/wiki/HTTPS}}) support and if it
is present on your system, CFITSIO will depend on it.
+Therefore, if @command{./configure} command below fails (you don't have the
cURL library), then remove this option and rerun it.
-Gnuastro's source code can be downloaded in two ways.
-As a tarball, ready to be configured and installed on your system (as
described in @ref{Quick start}), see @ref{Release tarball}.
-If you want official releases of stable versions this is the best, easiest and
most common option.
-Alternatively, you can clone the version controlled history of Gnuastro, run
one extra bootstrapping step and then follow the same steps as the tarball.
-This will give you access to all the most recent work that will be included in
the next release along with the full project history.
-The process is thoroughly introduced in @ref{Version controlled source}.
+Let's assume you have downloaded
@url{ftp://ftp.atnf.csiro.au/pub/software/wcslib/wcslib.tar.bz2,
@file{wcslib.tar.bz2}} and are in the same directory, to configure, build,
check and install WCSLIB follow the steps below.
+@example
+$ tar xf wcslib.tar.bz2
+## In the `cd' command, replace `X.X' with version number.
+$ cd wcslib-X.X
+## If `./configure' fails, remove `-lcurl' and run again.
+$ ./configure LIBS="-pthread -lcurl -lm" --without-pgplot \
+ --disable-fortran
+$ make
+$ make check
+$ sudo make install
+@end example
-@menu
-* Release tarball:: Download a stable official release.
-* Version controlled source:: Get and use the version controlled source.
-@end menu
-@node Release tarball, Version controlled source, Downloading the source,
Downloading the source
-@subsection Release tarball
-A release tarball (commonly compressed) is the most common way of obtaining
free and open source software.
-A tarball is a snapshot of one particular moment in the Gnuastro development
history along with all the necessary files to configure, build, and install
Gnuastro easily (see @ref{Quick start}).
-It is very straightforward and needs the least set of dependencies (see
@ref{Mandatory dependencies}).
-Gnuastro has tarballs for official stable releases and pre-releases for
testing.
-See @ref{Version numbering} for more on the two types of releases and the
formats of the version numbers.
-The URLs for each type of release are given below.
-
-@table @asis
-
-@item Official stable releases (@url{http://ftp.gnu.org/gnu/gnuastro}):
-This URL hosts the official stable releases of Gnuastro.
-Always use the most recent version (see @ref{Version numbering}).
-By clicking on the ``Last modified'' title of the second column, the files
will be sorted by their date which you can also use to find the latest version.
-It is recommended to use a mirror to download these tarballs, please visit
@url{http://ftpmirror.gnu.org/gnuastro/} and see below.
+@node Optional dependencies, Bootstrapping dependencies, Mandatory
dependencies, Dependencies
+@subsection Optional dependencies
-@item Pre-release tar-balls (@url{http://alpha.gnu.org/gnu/gnuastro}):
-This URL contains unofficial pre-release versions of Gnuastro.
-The pre-release versions of Gnuastro here are for enthusiasts to try out
before an official release.
-If there are problems, or bugs then the testers will inform the developers to
fix before the next official release.
-See @ref{Version numbering} to understand how the version numbers here are
formatted.
-If you want to remain even more up-to-date with the developing activities,
please clone the version controlled source as described in @ref{Version
controlled source}.
+The libraries listed here are only used for very specific applications,
therefore they are optional and Gnuastro can be built without them (with only
those specific features disabled).
+Since these are pretty low-level tools, they are not too hard to install from
source, but you can also use your operating system's package manager to easily
install all of them.
+For more, see @ref{Dependencies from package managers}.
-@end table
+@cindex GPL Ghostscript
+If the @command{./configure} script can't find any of these optional
dependencies, it will notify you of the operation(s) you can't do due to not
having them.
+If you continue the build and request an operation that uses a missing
library, Gnuastro's programs will warn that the optional library was missing at
build-time and abort.
+Since Gnuastro was built without that library, installing the library
afterwards won't help.
+The only way is to re-build Gnuastro from scratch (after the library has been
installed).
+However, for program dependencies (like cURL or GhostScript) things are
easier: you can install them after building Gnuastro also.
+This is because libraries are used to build the internal structure of
Gnuastro's executables.
+However, a program dependency is called by Gnuastro's programs at run-time and
has no effect on their internal structure.
+So if a dependency program becomes available later, it will be used next time
it is requested.
-@cindex Gzip
-@cindex Lzip
-Gnuastro's official/stable tarball is released with two formats: Gzip (with
suffix @file{.tar.gz}) and Lzip (with suffix @file{.tar.lz}).
-The pre-release tarballs (after version 0.3) are released only as an Lzip
tarball.
-Gzip is a very well-known and widely used compression program created by GNU
and available in most systems.
-However, Lzip provides a better compression ratio and more robust archival
capacity.
-For example Gnuastro 0.3's tarball was 2.9MB and 4.3MB with Lzip and Gzip
respectively, see the @url{http://www.nongnu.org/lzip/lzip.html, Lzip web page}
for more.
-Lzip might not be pre-installed in your operating system, if so, installing it
from your operating system's package manager or from source is very easy and
fast (it is a very small program).
+@table @asis
-The GNU FTP server is mirrored (has backups) in various locations on the globe
(@url{http://www.gnu.org/order/ftp.html}).
-You can use the closest mirror to your location for a more faster download.
-Note that only some mirrors keep track of the pre-release (alpha) tarballs.
-Also note that if you want to download immediately after and announcement (see
@ref{Announcements}), the mirrors might need some time to synchronize with the
main GNU FTP server.
+@item GNU Libtool
+@cindex GNU Libtool
+Libtool is a program to simplify managing of the libraries to build an
executable (a program).
+GNU Libtool has some added functionality compared to other implementations.
+If GNU Libtool isn't present on your system at configuration time, a warning
will be printed and @ref{BuildProgram} won't be built or installed.
+The configure script will look into your search path (@code{PATH}) for GNU
Libtool through the following executable names: @command{libtool} (acceptable
only if it is the GNU implementation) or @command{glibtool}.
+See @ref{Installation directory} for more on @code{PATH}.
+GNU Libtool (the binary/executable file) is a low-level program that is
probably already present on your system, and if not, is available in your
operating system package manager@footnote{Note that we want the
binary/executable Libtool program which can be run on the command-line.
+In Debian-based operating systems which separate various parts of a package,
you want want @code{libtool-bin}, the @code{libtool} package won't contain the
executable program.}.
+If you want to install GNU Libtool's latest version from source, please visit
its @url{https://www.gnu.org/software/libtool/, web page}.
-@node Version controlled source, , Release tarball, Downloading the source
-@subsection Version controlled source
+Gnuastro's tarball is shipped with an internal implementation of GNU Libtool.
+Even if you have GNU Libtool, Gnuastro's internal implementation is used for
the building and installation of Gnuastro.
+As a result, you can still build, install and use Gnuastro even if you don't
have GNU Libtool installed on your system.
+However, this internal Libtool does not get installed.
+Therefore, after Gnuastro's installation, if you want to use
@ref{BuildProgram} to compile and link your own C source code which uses the
@ref{Gnuastro library}, you need to have GNU Libtool available on your system
(independent of Gnuastro).
+See @ref{Review of library fundamentals} to learn more about libraries.
+@item libgit2
@cindex Git
-@cindex Version control
-The publicly distributed Gnuastro tar-ball (for example
@file{gnuastro-X.X.tar.gz}) does not contain the revision history, it is only a
snapshot of the source code at one significant instant of Gnuastro's history
(specified by the version number, see @ref{Version numbering}), ready to be
configured and built.
-To be able to develop successfully, the revision history of the code can be
very useful to track when something was added or changed, also some updates
that are not yet officially released might be in it.
+@pindex libgit2
+@cindex Version control systems
+Git is one of the most common version control systems (see @ref{Version
controlled source}).
+When @file{libgit2} is present, and Gnuastro's programs are run within a
version controlled directory, outputs will contain the version number of the
working directory's repository for future reproducibility.
+See the @command{COMMIT} keyword header in @ref{Output FITS files} for a
discussion.
-We use Git for the version control of Gnuastro.
-For those who are not familiar with it, we recommend the
@url{https://git-scm.com/book/en, ProGit book}.
-The whole book is publicly available for online reading and downloading and
does a wonderful job at explaining the concepts and best practices.
+@item libjpeg
+@pindex libjpeg
+@cindex JPEG format
+libjpeg is only used by ConvertType to read from and write to JPEG images, see
@ref{Recognized file formats}.
+@url{http://www.ijg.org/, libjpeg} is a very basic library that provides tools
to read and write JPEG images, most Unix-like graphic programs and libraries
use it.
+Therefore you most probably already have it installed.
+@url{http://libjpeg-turbo.virtualgl.org/, libjpeg-turbo} is an alternative to
libjpeg.
+It uses Single instruction, multiple data (SIMD) instructions for ARM based
systems that significantly decreases the processing time of JPEG compression
and decompression algorithms.
-Let's assume you want to keep Gnuastro in the @file{TOPGNUASTRO} directory
(can be any directory, change the value below).
-The full version controlled history of Gnuastro can be cloned in
@file{TOPGNUASTRO/gnuastro} by running the following commands@footnote{If your
internet connection is active, but Git complains about the network, it might be
due to your network setup not recognizing the Git protocol.
-In that case use the following URL which uses the HTTP protocol instead:
@command{http://git.sv.gnu.org/r/gnuastro.git}}:
+@item libtiff
+@pindex libtiff
+@cindex TIFF format
+libtiff is used by ConvertType and the libraries to read TIFF images, see
@ref{Recognized file formats}.
+@url{http://www.simplesystems.org/libtiff/, libtiff} is a very basic library
that provides tools to read and write TIFF images, most Unix-like operating
system graphic programs and libraries use it.
+Therefore even if you don't have it installed, it must be easily available in
your package manager.
-@example
-$ TOPGNUASTRO=/home/yourname/Research/projects/
-$ cd $TOPGNUASTRO
-$ git clone git://git.sv.gnu.org/gnuastro.git
-@end example
+@item cURL
+@cindex cURL (downloading tool)
+cURL's executable (@command{curl}) is called by @ref{Query} for submitting
queries to remote datasets and retrieving the results.
+It isn't necessary for the build of Gnuastro from source (only a warning will
be printed if it can't be found at configure time), so if you don't have it at
build-time there is no problem.
+Just be sure to have it when you run @command{astquery}, otherwise you'll get
an error about not finding @command{curl}.
-@noindent
-The @file{$TOPGNUASTRO/gnuastro} directory will contain hand-written (version
controlled) source code for Gnuastro's programs, libraries, this book and the
tests.
-All are divided into sub-directories with standard and very descriptive names.
-The version controlled files in the top cloned directory are either mainly in
capital letters (for example @file{THANKS} and @file{README}) or mainly written
in small-caps (for example @file{configure.ac} and @file{Makefile.am}).
-The former are non-programming, standard writing for human readers containing
high-level information about the whole package.
-The latter are instructions to customize the GNU build system for Gnuastro.
-For more on Gnuastro's source code structure, please see @ref{Developing}.
-We won't go any deeper here.
+@item GPL Ghostscript
+@cindex GPL Ghostscript
+GPL Ghostscript's executable (@command{gs}) is called by ConvertType to
compile a PDF file from a source PostScript file, see @ref{ConvertType}.
+Therefore its headers (and libraries) are not needed.
-The cloned Gnuastro source cannot immediately be configured, compiled, or
installed since it only contains hand-written files, not automatically
generated or imported files which do all the hard work of the build process.
-See @ref{Bootstrapping} for the process of generating and importing those
files (its not too hard!).
-Once you have bootstrapped Gnuastro, you can run the standard procedures (in
@ref{Quick start}).
-Very soon after you have cloned it, Gnuastro's main @file{master} branch will
be updated on the main repository (since the developers are actively working on
Gnuastro), for the best practices in keeping your local history in sync with
the main repository see @ref{Synchronizing}.
+@end table
+@node Bootstrapping dependencies, Dependencies from package managers, Optional
dependencies, Dependencies
+@subsection Bootstrapping dependencies
-@menu
-* Bootstrapping:: Adding all the automatically generated files.
-* Synchronizing:: Keep your local clone up to date.
-@end menu
+Bootstrapping is only necessary if you have decided to obtain the full version
controlled history of Gnuastro, see @ref{Version controlled source} and
@ref{Bootstrapping}.
+Using the version controlled source enables you to always be up to date with
the most recent development work of Gnuastro (bug fixes, new functionalities,
improved algorithms, etc).
+If you have downloaded a tarball (see @ref{Downloading the source}), then you
can ignore this subsection.
-@node Bootstrapping, Synchronizing, Version controlled source, Version
controlled source
-@subsubsection Bootstrapping
+To successfully run the bootstrapping process, there are some additional
dependencies to those discussed in the previous subsections.
+These are low level tools that are used by a large collection of Unix-like
operating systems programs, therefore they are most probably already available
in your system.
+If they are not already installed, you should be able to easily find them in
any GNU/Linux distribution package management system (@command{apt-get},
@command{yum}, @command{pacman}, etc).
+The short names in parenthesis in @command{typewriter} font after the package
name can be used to search for them in your package manager.
+For the GNU Portability Library, GNU Autoconf Archive and @TeX{} Live, it is
recommended to use the instructions here, not your operating system's package
manager.
-@cindex Bootstrapping
-@cindex GNU Autoconf Archive
+@table @asis
+
+@item GNU Portability Library (Gnulib)
+@cindex GNU C library
@cindex Gnulib: GNU Portability Library
@cindex GNU Portability Library (Gnulib)
-@cindex Automatically created build files
-@noindent
-The version controlled source code lacks the source files that we have not
written or are automatically built.
-These automatically generated files are included in the distributed tar ball
for each distribution (for example @file{gnuastro-X.X.tar.gz}, see @ref{Version
numbering}) and make it easy to immediately configure, build, and install
Gnuastro.
-However from the perspective of version control, they are just bloatware and
sources of confusion (since they are not changed by Gnuastro developers).
-
-The process of automatically building and importing necessary files into the
cloned directory is known as @emph{bootstrapping}.
-After bootstrapping is done you are ready to follow the default GNU build
steps that you normally run on the tarball (@command{./configure && make} for
example, described more in @ref{Quick start}).
-Some known issues with bootstrapping may occur during the process, to see how
to fix them, please see @ref{Known issues}.
-
-All the instructions for an automatic bootstrapping are available in
@file{bootstrap} and configured using @file{bootstrap.conf}.
-@file{bootstrap} and @file{COPYING} (which contains the software copyright
notice) are the only files not written by Gnuastro developers but under version
control to enable simple bootstrapping and legal information on usage
immediately after cloning.
-@file{bootstrap.conf} is maintained by the GNU Portability Library (Gnulib)
and this file is an identical copy, so do not make any changes in this file
since it will be replaced when Gnulib releases an update.
-Make all your changes in @file{bootstrap.conf}.
+To ensure portability for a wider range of operating systems (those that don't
include GNU C library, namely glibc), Gnuastro depends on the GNU portability
library, or Gnulib.
+Gnulib keeps a copy of all the functions in glibc, implemented (as much as
possible) to be portable to other operating systems.
+The @file{bootstrap} script can automatically clone Gnulib (as a
@file{gnulib/} directory inside Gnuastro), however, as described in
@ref{Bootstrapping} this is not recommended.
-The bootstrapping process has its own separate set of dependencies, the full
list is given in @ref{Bootstrapping dependencies}.
-They are generally very low-level and used by a very large set of commonly
used programs, so they are probably already installed on your system.
-The simplest way to bootstrap Gnuastro is to simply run the bootstrap script
within your cloned Gnuastro directory as shown below.
-However, please read the next paragraph before doing so (see @ref{Version
controlled source} for @file{TOPGNUASTRO}).
+The recommended way to bootstrap Gnuastro is to first clone Gnulib and the
Autoconf archives (see below) into a local directory outside of Gnuastro.
+Let's call it @file{DEVDIR}@footnote{If you are not a developer in Gnulib or
Autoconf archives, @file{DEVDIR} can be a directory that you don't backup.
+In this way the large number of files in these projects won't slow down your
backup process or take bandwidth (if you backup to a remote server).} (which
you can set to any directory).
+Currently in Gnuastro, both Gnulib and Autoconf archives have to be cloned in
the same top directory@footnote{If you already have the Autoconf archives in a
separate directory, or can't clone it in the same directory as Gnulib, or you
have it with another directory name (not @file{autoconf-archive/}), you can
follow this short step.
+Set @file{AUTOCONFARCHIVES} to your desired address.
+Then define a symbolic link in @file{DEVDIR} with the following command so
Gnuastro's bootstrap script can find it:@*@command{$ ln -s $AUTOCONFARCHIVES
$DEVDIR/autoconf-archive}.} like the case here@footnote{If your internet
connection is active, but Git complains about the network, it might be due to
your network setup not recognizing the git protocol.
+In that case use the following URL for the HTTP protocol instead (for Autoconf
archives, replace the name): @command{http://git.sv.gnu.org/r/gnulib.git}}:
@example
-$ cd TOPGNUASTRO/gnuastro
-$ ./bootstrap # Requires internet connection
+$ DEVDIR=/home/yourname/Development
+$ cd $DEVDIR
+$ git clone https://git.sv.gnu.org/git/gnulib.git
+$ git clone https://git.sv.gnu.org/git/autoconf-archive.git
@end example
-Without any options, @file{bootstrap} will clone Gnulib within your cloned
Gnuastro directory (@file{TOPGNUASTRO/gnuastro/gnulib}) and download the
necessary Autoconf archives macros.
-So if you run bootstrap like this, you will need an internet connection every
time you decide to bootstrap.
-Also, Gnulib is a large package and cloning it can be slow.
-It will also keep the full Gnulib repository within your Gnuastro repository,
so if another one of your projects also needs Gnulib, and you insist on running
bootstrap like this, you will have two copies.
-In case you regularly backup your important files, Gnulib will also slow down
the backup process.
-Therefore while the simple invocation above can be used with no problem, it is
not recommended.
-To do better, see the next paragraph.
+Gnulib is a source-based dependency of Gnuastro's bootstrapping process, so
simply having it is enough on your computer, there is no need to install, and
thus check anything.
-The recommended way to get these two packages is thoroughly discussed in
@ref{Bootstrapping dependencies} (in short: clone them in the separate
@file{DEVDIR/} directory).
-The following commands will take you into the cloned Gnuastro directory and
run the @file{bootstrap} script, while telling it to copy some files (instead
of making symbolic links, with the @option{--copy} option, this is not
mandatory@footnote{The @option{--copy} option is recommended because some
backup systems might do strange things with symbolic links.}) and where to look
for Gnulib (with the @option{--gnulib-srcdir} option).
-Please note that the address given to @option{--gnulib-srcdir} has to be an
absolute address (so don't use @file{~} or @file{../} for example).
+@noindent
+You now have the full version controlled source of these two repositories in
separate directories.
+Both these packages are regularly updated, so every once in a while, you can
run @command{$ git pull} within them to get any possible updates.
+
+@item GNU Automake (@command{automake})
+@cindex GNU Automake
+GNU Automake will build the @file{Makefile.in} files in each sub-directory
using the (hand-written) @file{Makefile.am} files.
+The @file{Makefile.in}s are subsequently used to generate the @file{Makefile}s
when the user runs @command{./configure} before building.
+
+To check that you have a working GNU Automake in your system, you can try this
command:
@example
-$ cd $TOPGNUASTRO/gnuastro
-$ ./bootstrap --copy --gnulib-srcdir=$DEVDIR/gnulib
+$ automake --version
@end example
-@cindex GNU Texinfo
-@cindex GNU Libtool
+@item GNU Autoconf (@command{autoconf})
@cindex GNU Autoconf
-@cindex GNU Automake
-@cindex GNU C library
-@cindex GNU build system
-Since Gnulib and Autoconf archives are now available in your local
directories, you don't need an internet connection every time you decide to
remove all untracked files and redo the bootstrap (see box below).
-You can also use the same command on any other project that uses Gnulib.
-All the necessary GNU C library functions, Autoconf macros and Automake inputs
are now available along with the book figures.
-The standard GNU build system (@ref{Quick start}) will do the rest of the job.
+GNU Autoconf will build the @file{configure} script using the configurations
we have defined (hand-written) in @file{configure.ac}.
-@cartouche
-@noindent
-@strong{Undoing the bootstrap:}
-During the development, it might happen that you want to remove all the
automatically generated and imported files.
-In other words, you might want to reverse the bootstrap process.
-Fortunately Git has a good program for this job: @command{git clean}.
-Run the following command and every file that is not version controlled will
be removed.
+To check that you have a working GNU Autoconf in your system, you can try this
command:
@example
-git clean -fxd
+$ autoconf --version
@end example
-@noindent
-It is best to commit any recent change before running this command.
-You might have created new files since the last commit and if they haven't
been committed, they will all be gone forever (using @command{rm}).
-To get a list of the non-version controlled files instead of deleting them,
add the @option{n} option to @command{git clean}, so it becomes @option{-fxdn}.
-@end cartouche
-
-Besides the @file{bootstrap} and @file{bootstrap.conf}, the
@file{bootstrapped/} directory and @file{README-hacking} file are also related
to the bootstrapping process.
-The former hosts all the imported (bootstrapped) directories.
-Thus, in the version controlled source, it only contains a @file{REAME} file,
but in the distributed tar-ball it also contains sub-directories filled with
all bootstrapped files.
-@file{README-hacking} contains a summary of the bootstrapping process
discussed in this section.
-It is a necessary reference when you haven't built this book yet.
-It is thus not distributed in the Gnuastro tarball.
-
+@item GNU Autoconf Archive
+@cindex GNU Autoconf Archive
+These are a large collection of tests that can be called to run at
@command{./configure} time.
+See the explanation under GNU Portability Library (Gnulib) above for
instructions on obtaining it and keeping it up to date.
-@node Synchronizing, , Bootstrapping, Version controlled source
-@subsubsection Synchronizing
+GNU Autoconf Archive is a source-based dependency of Gnuastro's bootstrapping
process, so simply having it is enough on your computer, there is no need to
install, and thus check anything.
+Just don't forget that it has to be in the same directory as Gnulib (described
above).
-The bootstrapping script (see @ref{Bootstrapping}) is not regularly needed:
you mainly need it after you have cloned Gnuastro (once) and whenever you want
to re-import the files from Gnulib, or Autoconf
archives@footnote{@url{https://savannah.gnu.org/task/index.php?13993} is
defined for you to check if significant (for Gnuastro) updates are made in
these repositories, since the last time you pulled from them.} (not too common).
-However, Gnuastro developers are constantly working on Gnuastro and are
pushing their changes to the official repository.
-Therefore, your local Gnuastro clone will soon be out-dated.
-Gnuastro has two mailing lists dedicated to its developing activities (see
@ref{Developing mailing lists}).
-Subscribing to them can help you decide when to synchronize with the official
repository.
+@item GNU Texinfo (@command{texinfo})
+@cindex GNU Texinfo
+GNU Texinfo is the tool that formats this manual into the various output
formats.
+To bootstrap Gnuastro you need all of Texinfo's command-line programs.
+However, some operating systems package them separately, for example in
Fedora, @command{makeinfo} is packaged in the @command{texinfo-tex} package.
-To pull all the most recent work in Gnuastro, run the following command from
the top Gnuastro directory.
-If you don't already have a built system, ignore @command{make distclean}.
-The separate steps are described in detail afterwards.
+To check that you have a working GNU Texinfo in your system, you can try this
command:
@example
-$ make distclean && git pull && autoreconf -f
+$ makeinfo --version
@end example
-@noindent
-You can also run the commands separately:
+@item GNU Libtool (@command{libtool})
+@cindex GNU Libtool
+GNU Libtool is in charge of building all the libraries in Gnuastro.
+The libraries contain functions that are used by more than one program and are
installed for use in other programs.
+They are thus put in a separate directory (@file{lib/}).
+
+To check that you have a working GNU Libtool in your system, you can try this
command (and from the output, make sure it is GNU's libtool)
@example
-$ make distclean
-$ git pull
-$ autoreconf -f
+$ libtool --version
@end example
-@cindex GNU Autoconf
-@cindex Mailing list: info-gnuastro
-@cindex @code{info-gnuastro@@gnu.org}
-If Gnuastro was already built in this directory, you don't want some outputs
from the previous version being mixed with outputs from the newly pulled work.
-Therefore, the first step is to clean/delete all the built files with
@command{make distclean}.
-Fortunately the GNU build system allows the separation of source and built
files (in separate directories).
-This is a great feature to keep your source directory clean and you can use it
to avoid the cleaning step.
-Gnuastro comes with a script with some useful options for this job.
-It is useful if you regularly pull recent changes, see @ref{Separate build and
source directories}.
-
-After the pull, we must re-configure Gnuastro with @command{autoreconf -f}
(part of GNU Autoconf).
-It will update the @file{./configure} script and all the
@file{Makefile.in}@footnote{In the GNU build system, @command{./configure} will
use the @file{Makefile.in} files to create the necessary @file{Makefile} files
that are later read by @command{make} to build the package.} files based on the
hand-written configurations (in @file{configure.ac} and the @file{Makefile.am}
files).
-After running @command{autoreconf -f}, a warning about @code{TEXI2DVI} might
show up, you can ignore that.
-
-The most important reason for re-building Gnuastro's build system is to
generate/update the version number for your updated Gnuastro snapshot.
-This generated version number will include the commit information (see
@ref{Version numbering}).
-The version number is included in nearly all outputs of Gnuastro's programs,
therefore it is vital for reproducing an old result.
-
-As a summary, be sure to run `@command{autoreconf -f}' after every change in
the Git history.
-This includes synchronization with the main server or even a commit you have
made yourself.
+@item GNU help2man (@command{help2man})
+@cindex GNU help2man
+GNU help2man is used to convert the output of the @option{--help} option
+(@ref{--help}) to the traditional Man page (@ref{Man pages}).
-If you would like to see what has changed since you last synchronized your
local clone, you can take the following steps instead of the simple command
above (don't type anything after @code{#}):
+To check that you have a working GNU Help2man in your system, you can try this
command:
@example
-$ git checkout master # Confirm if you are on master.
-$ git fetch origin # Fetch all new commits from server.
-$ git log master..origin/master # See all the new commit messages.
-$ git merge origin/master # Update your master branch.
-$ autoreconf -f # Update the build system.
+$ help2man --version
@end example
-@noindent
-By default @command{git log} prints the most recent commit first, add the
@option{--reverse} option to see the changes chronologically.
-To see exactly what has been changed in the source code along with the commit
message, add a @option{-p} option to the @command{git log}.
-
-If you want to make changes in the code, have a look at @ref{Developing} to
get started easily.
-Be sure to commit your changes in a separate branch (keep your @code{master}
branch to follow the official repository) and re-run @command{autoreconf -f}
after the commit.
-If you intend to send your work to us, you can safely use your commit since it
will be ultimately recorded in Gnuastro's official history.
-If not, please upload your separate branch to a public hosting service, for
example @url{https://gitlab.com, GitLab}, and link to it in your report/paper.
-Alternatively, run @command{make distcheck} and upload the output
@file{gnuastro-X.X.X.XXXX.tar.gz} to a publicly accessible web page so your
results can be considered scientific (reproducible) later.
+@item @LaTeX{} and some @TeX{} packages
+@cindex @LaTeX{}
+@cindex @TeX{} Live
+Some of the figures in this book are built by @LaTeX{} (using the PGF/TikZ
package).
+The @LaTeX{} source for those figures is version controlled for easy
maintenance not the actual figures.
+So the @file{./boostrap} script will run @LaTeX{} to build the figures.
+The best way to install @LaTeX{} and all the necessary packages is through
@url{https://www.tug.org/texlive/, @TeX{} live} which is a package manager for
@TeX{} related tools that is independent of any operating system.
+It is thus preferred to the @TeX{} Live versions distributed by your operating
system.
+To install @TeX{} Live, go to the web page and download the appropriate
installer by following the ``download'' link.
+Note that by default the full package repository will be downloaded and
installed (around 4 Giga Bytes) which can take @emph{very} long to download and
to update later.
+However, most packages are not needed by everyone, it is easier, faster and
better to install only the ``Basic scheme'' (consisting of only the most basic
@TeX{} and @LaTeX{} packages, which is less than 200 Mega bytes)@footnote{You
can also download the DVD iso file at a later time to keep as a backup for when
you don't have internet connection if you need a package.}.
+After the installation, be sure to set the environment variables as suggested
in the end of the outputs.
+Any time you confront (need) a package you don't have, simply install it with
a command like below (similar to how you install software from your operating
system's package manager)@footnote{After running @TeX{}, or @LaTeX{}, you might
get a warning complaining about a @file{missingfile}.
+Run `@command{tlmgr info missingfile}' to see the package(s) containing that
file which you can install.}.
+To install all the necessary @TeX{} packages for a successful Gnuastro
bootstrap, run this command:
+@example
+$ su
+# tlmgr install epsf jknapltx caption biblatex biber iftex \
+ etoolbox logreq xstring xkeyval pgf ms \
+ xcolor pgfplots times rsfs ps2eps epspdf
+@end example
+To check that you have a working @LaTeX{} executable in your system, you can
try this command (this just checks if @LaTeX{} exists, as described above, if
you have a missing package, you can easily identify it from the output and
install it with @command{tlmgr}.
+@example
+$ latex --version
+@end example
+@item ImageMagick (@command{imagemagick})
+@cindex ImageMagick
+ImageMagick is a wonderful and robust program for image manipulation on the
command-line.
+@file{bootstrap} uses it to convert the book images into the formats necessary
for the various book formats.
+To check that you have a working ImageMagick in your system, you can try this
command:
+@example
+$ convert --version
+@end example
+@end table
-@node Build and install, , Downloading the source, Installation
-@section Build and install
-This section is basically a longer explanation to the sequence of commands
given in @ref{Quick start}.
-If you didn't have any problems during the @ref{Quick start} steps, you want
to have all the programs of Gnuastro installed in your system, you don't want
to change the executable names during or after installation, you have root
access to install the programs in the default system wide directory, the Letter
paper size of the print book is fine for you or as a summary you don't feel
like going into the details when everything is working, you can safely skip
this section.
-If you have any of the above problems or you want to understand the details
for a better control over your build and install, read along.
-The dependencies which you will need prior to configuring, building and
installing Gnuastro are explained in @ref{Dependencies}.
-The first three steps in @ref{Quick start} need no extra explanation, so we
will skip them and start with an explanation of Gnuastro specific configuration
options and a discussion on the installation directory in @ref{Configuring},
followed by some smaller subsections: @ref{Tests}, @ref{A4 print book}, and
@ref{Known issues} which explains the solutions to known problems you might
encounter in the installation steps and ways you can solve them.
+@node Dependencies from package managers, , Bootstrapping dependencies,
Dependencies
+@subsection Dependencies from package managers
+@cindex Package managers
+@cindex Source code building
+@cindex Building from source
+@cindex Compiling from source
+@cindex Source code compilation
+@cindex Distributions, GNU/Linux
+The most basic way to install a package on your system is to build the
packages from source yourself.
+Alternatively, you can use your operating system's package manager to download
pre-compiled files and install them.
+The latter choice is easier and faster.
+However, we recommend that you build the @ref{Mandatory dependencies} yourself
from source (all necessary commands and links are given in the respective
section).
+Here are some basic reasons behind this recommendation.
-@menu
-* Configuring:: Configure Gnuastro
-* Separate build and source directories:: Keeping derivate/build files
separate.
-* Tests:: Run tests to see if it is working.
-* A4 print book:: Customize the print book.
-* Known issues:: Issues you might encounter.
-@end menu
+@enumerate
+@item
+Your operating system's pre-built software might not be the most recent
release.
+For example, Gnuastro itself is also packaged in some package managers.
+For the list see: @url{https://repology.org/project/gnuastro/versions}.
+You will notice that Gnuastro's version in some operating systems is more than
10 versions old!
+It is the same for all the dependencies of Gnuastro.
+@item
+For each package, Gnuastro might preform better (or require) certain
configuration options that your distribution's package managers didn't add for
you.
+If present, these configuration options are explained during the installation
of each in the sections below (for example in @ref{CFITSIO}).
+When the proper configuration has not been set, the programs should complain
and inform you.
+@item
+For the libraries, they might separate the binary file from the header files
which can cause confusion, see @ref{Known issues}.
+@item
+Like any other tool, the science you derive from Gnuastro's tools highly
depend on these lower level dependencies, so generally it is much better to
have a close connection with them.
+By reading their manuals, installing them and staying up to date with
changes/bugs in them, your scientific results and understanding (of what is
going on, and thus how you interpret your scientific results) will also
correspondingly improve.
+@end enumerate
-@node Configuring, Separate build and source directories, Build and install,
Build and install
-@subsection Configuring
+Based on your package manager, you can use any of the following commands to
install the mandatory and optional dependencies.
+If your package manager isn't included in the list below, please send us the
respective command, so we add it.
-@pindex ./configure
-@cindex Configuring
-The @command{$ ./configure} step is the most important step in the build and
install process.
-All the required packages, libraries, headers and environment variables are
checked in this step.
-The behaviors of make and make install can also be set through command line
options to this command.
+As discussed above, we recommend installing the @emph{mandatory} dependencies
manually from source (see @ref{Mandatory dependencies}).
+Therefore, in each command below, first the optional dependencies are given.
+The mandatory dependencies are included after an empty line.
+If you would also like to install the mandatory dependencies with your package
manager, just ignore the empty line.
-@cindex Configure options
-@cindex Customizing installation
-@cindex Installation, customizing
-The configure script accepts various arguments and options which enable the
final user to highly customize whatever she is building.
-The options to configure are generally very similar to normal program options
explained in @ref{Arguments and options}.
-Similar to all GNU programs, you can get a full list of the options along with
a short explanation by running
+For better archivability and compression ratios, Gnuastro's recommended
tarball compression format is with the @url{http://lzip.nongnu.org/lzip.html,
Lzip} program, see @ref{Release tarball}.
+Therefore, the package manager commands below also contain Lzip.
+@table @asis
+@item @command{apt-get} (Debian-based OSs: Debian, Ubuntu, Linux Mint, etc)
+@cindex Debian
+@cindex Ubuntu
+@cindex Linux Mint
+@cindex @command{apt-get}
+@cindex Advanced Packaging Tool (APT, Debian)
+@url{https://en.wikipedia.org/wiki/Debian,Debian} is one of the oldest
+GNU/Linux
+distributions@footnote{@url{https://en.wikipedia.org/wiki/List_of_Linux_distributions#Debian-based}}.
+It thus has a very extended user community and a robust internal structure and
standards.
+All of it is free software and based on the work of volunteers around the
world.
+Many distributions are thus derived from it, for example Ubuntu and Linux Mint.
+This arguably makes Debian-based OSs the largest, and most used, class of
GNU/Linux distributions.
+All of them use Debian's Advanced Packaging Tool (APT, for example
@command{apt-get}) for managing packages.
@example
-$ ./configure --help
+$ sudo apt-get install ghostscript libtool-bin libjpeg-dev \
+ libtiff-dev libgit2-dev curl lzip \
+ \
+ libgsl0-dev libcfitsio-dev wcslib-dev
@end example
@noindent
-@cindex GNU Autoconf
-A complete explanation is also included in the @file{INSTALL} file.
-Note that this file was written by the authors of GNU Autoconf (which builds
the @file{configure} script), therefore it is common for all programs which use
the @command{$ ./configure} script for building and installing, not just
Gnuastro.
-Here we only discuss cases where you don't have super-user access to the
system and if you want to change the executable names.
-But before that, a review of the options to configure that are particular to
Gnuastro are discussed.
-
-@menu
-* Gnuastro configure options:: Configure options particular to Gnuastro.
-* Installation directory:: Specify the directory to install.
-* Executable names:: Changing executable names.
-* Configure and build in RAM:: For minimal use of HDD or SSD, and clean
source.
-@end menu
-
-@node Gnuastro configure options, Installation directory, Configuring,
Configuring
-@subsubsection Gnuastro configure options
-
-@cindex @command{./configure} options
-@cindex Configure options particular to Gnuastro
-Most of the options to configure (which are to do with building) are similar
for every program which uses this script.
-Here the options that are particular to Gnuastro are discussed.
-The next topics explain the usage of other configure options which can be
applied to any program using the GNU build system (through the configure
script).
-
-@vtable @option
+Gnuastro is @url{https://tracker.debian.org/pkg/gnuastro,packaged} in Debian
(and thus some of its derivate operating systems).
+Just make sure it is the most recent version.
-@item --enable-debug
-@cindex Valgrind
-@cindex Debugging
-@cindex GNU Debugger
-Compile/build Gnuastro with debugging information, no optimization and without
shared libraries.
+@item @command{dnf}
+@itemx @command{yum} (Red Hat-based OSs: Red Hat, Fedora, CentOS, Scientific
Linux, etc)
+@cindex RHEL
+@cindex Fedora
+@cindex CentOS
+@cindex Red Hat
+@cindex @command{dnf}
+@cindex @command{yum}
+@cindex Scientific Linux
+@url{https://en.wikipedia.org/wiki/Red_Hat,Red Hat Enterprise Linux} (RHEL) is
released by Red Hat Inc.
+RHEL requires paid subscriptions for use of its binaries and support.
+But since it is free software, many other teams use its code to spin-off their
own distributions based on RHEL.
+Red Hat-based GNU/Linux distributions initially used the ``Yellowdog Updated,
Modifier'' (YUM) package manager, which has been replaced by ``Dandified yum''
(DNF).
+If the latter isn't available on your system, you can use @command{yum}
instead of @command{dnf} in the command below.
+@example
+$ sudo dnf install ghostscript libtool libjpeg-devel \
+ libtiff-devel libgit2-devel lzip curl \
+ \
+ gsl-devel cfitsio-devel wcslib-devel
+@end example
-In order to allow more efficient programs when using Gnuastro (after the
installation), by default Gnuastro is built with a 3rd level (a very high
level) optimization and no debugging information.
-By default, libraries are also built for static @emph{and} shared linking (see
@ref{Linking}).
-However, when there are crashes or unexpected behavior, these three features
can hinder the process of localizing the problem.
-This configuration option is identical to manually calling the configuration
script with @code{CFLAGS="-g -O0" --disable-shared}.
+@item @command{brew} (macOS)
+@cindex macOS
+@cindex Homebrew
+@cindex MacPorts
+@cindex @command{brew}
+@url{https://en.wikipedia.org/wiki/MacOS,macOS} is the operating system used
on Apple devices.
+macOS does not come with a package manager pre-installed, but several widely
used, third-party package managers exist, such as Homebrew or MacPorts.
+Both are free software.
+Currently we have only tested Gnuastro's installation with Homebrew as
described below.
-In the (rare) situations where you need to do your debugging on the shared
libraries, don't use this option.
-Instead run the configure script by explicitly setting @code{CFLAGS} like this:
+If not already installed, first obtain Homebrew by following the instructions
at @url{https://brew.sh}.
+Homebrew manages packages in different `taps'.
+To install WCSLIB (discussed in @ref{Mandatory dependencies}) via Homebrew you
will need to @command{tap} into @command{brewsci/science} first (the tap may
change in the future, but can be found by calling @command{brew search wcslib}).
@example
-$ ./configure CFLAGS="-g -O0"
+$ brew install ghostscript libtool libjpeg libtiff \
+ libgit2 curl lzip \
+ \
+ gsl cfitsio
+$ brew tap brewsci/science
+$ brew install wcslib
@end example
-@item --enable-check-with-valgrind
-@cindex Valgrind
-Do the @command{make check} tests through Valgrind.
-Therefore, if any crashes or memory-related issues (segmentation faults in
particular) occur in the tests, the output of Valgrind will also be put in the
@file{tests/test-suite.log} file without having to manually modify the check
scripts.
-This option will also activate Gnuastro's debug mode (see the
@option{--enable-debug} configure-time option described above).
-
-Valgrind is free software.
-It is a program for easy checking of memory-related issues in programs.
-It runs a program within its own controlled environment and can thus identify
the exact line-number in the program's source where a memory-related issue
occurs.
-However, it can significantly slow-down the tests.
-So this option is only useful when a segmentation fault is found during
@command{make check}.
-
-@item --enable-progname
-Only build and install @file{progname} along with any other program that is
enabled in this fashion.
-@file{progname} is the name of the executable without the @file{ast}, for
example @file{crop} for Crop (with the executable name of @file{astcrop}).
-
-Note that by default all the programs will be installed.
-This option (and the @option{--disable-progname} options) are only relevant
when you don't want to install all the programs.
-Therefore, if this option is called for any of the programs in Gnuastro, any
program which is not explicitly enabled will not be built or installed.
+@item @command{pacman} (Arch Linux)
+@cindex Arch Linux
+@cindex @command{pacman}
+@url{https://en.wikipedia.org/wiki/Arch_Linux,Arch Linux} is a smaller
GNU/Linux distribution, which follows the KISS principle (``keep it simple,
stupid'') as a general guideline.
+It ``focuses on elegance, code correctness, minimalism and simplicity, and
expects the user to be willing to make some effort to understand the system's
operation''.
+Arch Linux uses ``Package manager'' (Pacman) to manage its packages/components.
+@example
+$ sudo pacman -S ghostscript libtool libjpeg libtiff \
+ libgit2 curl lzip \
+ \
+ gsl cfitsio wcslib
+@end example
-@item --disable-progname
-@itemx --enable-progname=no
-Do not build or install the program named @file{progname}.
-This is very similar to the @option{--enable-progname}, but will build and
install all the other programs except this one.
+@item @command{zypper} (openSUSE and SUSE Linux Enterprise Server)
+@cindex openSUSE
+@cindex SUSE Linux Enterprise Server
+@cindex @command{zypper}, OpenSUSE package manager
+SUSE Linux Enterprise
Server@footnote{@url{https://www.suse.com/products/server}} (SLES) is the
commercial offering which shares code and tools.
+Many additional packages are offered in the Build
Service@footnote{@url{https://build.opensuse.org}}.
+openSUSE and SLES use @command{zypper} (cli) and YaST (GUI) for managing
repositories and
+packages.
-@cartouche
+@example
+$ sudo zypper install ghostscript_any libtool pkgconfig \
+ libcurl-devel libgit2-devel libjpeg62-devel \
+ libtiff-devel curl \
+ \
+ gsl-devel cfitsio-devel wcslib-devel
+@end example
@noindent
-@strong{Note:} If some programs are enabled and some are disabled, it is
equivalent to simply enabling those that were enabled.
-Listing the disabled programs is redundant.
-@end cartouche
-
-@item --enable-gnulibcheck
-@cindex GNU C library
-@cindex Gnulib: GNU Portability Library
-@cindex GNU Portability Library (Gnulib)
-Enable checks on the GNU Portability Library (Gnulib).
-Gnulib is used by Gnuastro to enable users of non-GNU based operating systems
(that don't use GNU C library or glibc) to compile and use the advanced
features that this library provides.
-We make extensive use of such functions.
-If you give this option to @command{$ ./configure}, when you run @command{$
make check}, first the functions in Gnulib will be tested, then the Gnuastro
executables.
-If your operating system does not support glibc or has an older version of it
and you have problems in the build process (@command{$ make}), you can give
this flag to configure to see if the problem is caused by Gnulib not supporting
your operating system or Gnuastro, see @ref{Known issues}.
+When building Gnuastro, run the configure script with the following
@code{CPPFLAGS} environment variable:
-@item --disable-guide-message
-@itemx --enable-guide-message=no
-Do not print a guiding message during the GNU Build process of @ref{Quick
start}.
-By default, after each step, a message is printed guiding the user what the
next command should be.
-Therefore, after @command{./configure}, it will suggest running @command{make}.
-After @command{make}, it will suggest running @command{make check} and so on.
-If Gnuastro is configured with this option, for example
@example
-$ ./configure --disable-guide-message
+$ ./configure CPPFLAGS="-I/usr/include/cfitsio"
@end example
-Then these messages will not be printed after any step (like most programs).
-For people who are not yet fully accustomed to this build system, these
guidelines can be very useful and encouraging.
-However, if you find those messages annoying, use this option.
-
-@item --without-libgit2
-@cindex Git
-@pindex libgit2
-@cindex Version control systems
-Build Gnuastro without libgit2 (for including Git commit hashes in output
files), see @ref{Optional dependencies}.
-libgit2 is an optional dependency, with this option, Gnuastro will ignore any
possibly existing libgit2 that may already be on the system.
-
-@item --without-libjpeg
-@pindex libjpeg
-@cindex JPEG format
-Build Gnuastro without libjpeg (for reading/writing to JPEG files), see
@ref{Optional dependencies}.
-libjpeg is an optional dependency, with this option, Gnuastro will ignore any
possibly existing libjpeg that may already be on the system.
-
-@item --without-libtiff
-@pindex libtiff
-@cindex TIFF format
-Build Gnuastro without libtiff (for reading/writing to TIFF files), see
@ref{Optional dependencies}.
-libtiff is an optional dependency, with this option, Gnuastro will ignore any
possibly existing libtiff that may already be on the system.
-
-@end vtable
-
-The tests of some programs might depend on the outputs of the tests of other
programs.
-For example MakeProfiles is one the first programs to be tested when you run
@command{$ make check}.
-MakeProfiles' test outputs (FITS images) are inputs to many other programs
(which in turn provide inputs for other programs).
-Therefore, if you don't install MakeProfiles for example, the tests for many
the other programs will be skipped.
-To avoid this, in one run, you can install all the programs and run the tests
but not install.
-If everything is working correctly, you can run configure again with only the
programs you want.
-However, don't run the tests and directly install after building.
-
-
-
-@node Installation directory, Executable names, Gnuastro configure options,
Configuring
-@subsubsection Installation directory
-@vindex --prefix
-@cindex Superuser, not possible
-@cindex Root access, not possible
-@cindex No access to super-user install
-@cindex Install with no super-user access
-One of the most commonly used options to @file{./configure} is
@option{--prefix}, it is used to define the directory that will host all the
installed files (or the ``prefix'' in their final absolute file name).
-For example, when you are using a server and you don't have administrator or
root access.
-In this example scenario, if you don't use the @option{--prefix} option, you
won't be able to install the built files and thus access them from anywhere
without having to worry about where they are installed.
-However, once you prepare your startup file to look into the proper place (as
discussed thoroughly below), you will be able to easily use this option and
benefit from any software you want to install without having to ask the system
administrators or install and use a different version of a software that is
already installed on the server.
+@c Gnuastro is @url{https://software.opensuse.org/package/gnuastro,packaged}
+@c in @command{zypper}. Just make sure it is the most recent version.
+@end table
-The most basic way to run an executable is to explicitly write its full file
name (including all the directory information) and run it.
-One example is running the configuration script with the @command{$
./configure} command (see @ref{Quick start}).
-By giving a specific directory (the current directory or @file{./}), we are
explicitly telling the shell to look in the current directory for an executable
file named `@file{configure}'.
-Directly specifying the directory is thus useful for executables in the
current (or nearby) directories.
-However, when the program (an executable file) is to be used a lot, specifying
all those directories will become a significant burden.
-For example, the @file{ls} executable lists the contents in a given directory
and it is (usually) installed in the @file{/usr/bin/} directory by the
operating system maintainers.
-Therefore, if using the full address was the only way to access an executable,
each time you wanted a listing of a directory, you would have to run the
following command (which is very inconvenient, both in writing and in
remembering the various directories).
+Usually, when libraries are installed by operating system package managers,
there should be no problems when configuring and building other programs from
source (that depend on the libraries: Gnuastro in this case).
+However, in some special conditions, problems may pop-up during the
configuration, building, or checking/running any of Gnuastro's programs.
+The most common of such problems and their solution are discussed below.
+@cartouche
+@noindent
+@strong{Not finding library during configuration:} If a library is installed,
but during Gnuastro's @command{configure} step the library isn't found, then
configure Gnuastro like the command below (correcting @file{/path/to/lib}).
+For more, see @ref{Known issues} and @ref{Installation directory}.
@example
-$ /usr/bin/ls
+$ ./configure LDFLAGS="-L/path/to/lib"
@end example
+@end cartouche
-@cindex Shell variables
-@cindex Environment variables
-To address this problem, we have the @file{PATH} environment variable.
-To understand it better, we will start with a short introduction to the shell
variables.
-Shell variable values are basically treated as strings of characters.
-For example, it doesn't matter if the value is a name (string of
@emph{alphabetic} characters), or a number (string of @emph{numeric}
characters), or both.
-You can define a variable and a value for it by running
-@example
-$ myvariable1=a_test_value
-$ myvariable2="a test value"
-@end example
+@cartouche
@noindent
-As you see above, if the value contains white space characters, you have to
put the whole value (including white space characters) in double quotes
(@key{"}).
-You can see the value it represents by running
+@strong{Not finding header (.h) files while building:} If a library is
installed, but during Gnuastro's @command{make} step, the library's header
(file with a @file{.h} suffix) isn't found, then configure Gnuastro like the
command below (correcting @file{/path/to/include}).
+For more, see @ref{Known issues} and @ref{Installation directory}.
@example
-$ echo $myvariable1
-$ echo $myvariable2
+$ ./configure CPPFLAGS="-I/path/to/include"
@end example
-@noindent
-@cindex Environment
-@cindex Environment variables
-If a variable has no value or it wasn't defined, the last command will only
print an empty line.
-A variable defined like this will be known as long as this shell or terminal
is running.
-Other terminals will have no idea it existed.
-The main advantage of shell variables is that if they are exported@footnote{By
running @command{$ export myvariable=a_test_value} instead of the simpler case
in the text}, subsequent programs that are run within that shell can access
their value.
-So by changing their value, you can change the ``environment'' of a program
which uses them.
-The shell variables which are accessed by programs are therefore known as
``environment variables''@footnote{You can use shell variables for other
actions too, for example to temporarily keep some names or run loops on some
files.}.
-You can see the full list of exported variables that your shell recognizes by
running:
+@end cartouche
+@cartouche
+@noindent
+@strong{Gnuastro's programs don't run during check or after install:}
+If a library is installed, but the programs don't run due to linking problems,
set the @code{LD_LIBRARY_PATH} variable like below (assuming Gnuastro is
installed in @file{/path/to/installed}).
+For more, see @ref{Known issues} and @ref{Installation directory}.
@example
-$ printenv
+$ export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/path/to/installed/lib"
@end example
+@end cartouche
-@cindex @file{HOME}
-@cindex @file{HOME/.local/}
-@cindex Environment variable, @code{HOME}
-@file{HOME} is one commonly used environment variable, it is any user's (the
one that is logged in) top directory.
-Try finding it in the command above.
-It is used so often that the shell has a special expansion (alternative) for
it: `@file{~}'.
-Whenever you see file names starting with the tilde sign, it actually
represents the value to the @file{HOME} environment variable, so @file{~/doc}
is the same as @file{$HOME/doc}.
-@vindex PATH
-@pindex ./configure
-@cindex Setting @code{PATH}
-@cindex Default executable search directory
-@cindex Search directory for executables
-Another one of the most commonly used environment variables is @file{PATH}, it
is a list of directories to search for executable names.
-Its value is a list of directories (separated by a colon, or `@key{:}').
-When the address of the executable is not explicitly given (like
@file{./configure} above), the system will look for the executable in the
directories specified by @file{PATH}.
-If you have a computer nearby, try running the following command to see which
directories your system will look into when it is searching for executable
(binary) files, one example is printed here (notice how @file{/usr/bin}, in the
@file{ls} example above, is one of the directories in @command{PATH}):
-@example
-$ echo $PATH
-/usr/local/sbin:/usr/local/bin:/usr/bin
-@end example
-By default @file{PATH} usually contains system-wide directories, which are
readable (but not writable) by all users, like the above example.
-Therefore if you don't have root (or administrator) access, you need to add
another directory to @file{PATH} which you actually have write access to.
-The standard directory where you can keep installed files (not just
executables) for your own user is the @file{~/.local/} directory.
-The names of hidden files start with a `@key{.}' (dot), so it will not show up
in your common command-line listings, or on the graphical user interface.
-You can use any other directory, but this is the most recognized.
-The top installation directory will be used to keep all the package's
components: programs (executables), libraries, include (header) files, shared
data (like manuals), or configuration files (see @ref{Review of library
fundamentals} for a thorough introduction to headers and linking).
-So it commonly has some of the following sub-directories for each class of
installed components respectively: @file{bin/}, @file{lib/}, @file{include/}
@file{man/}, @file{share/}, @file{etc/}.
-Since the @file{PATH} variable is only used for executables, you can add the
@file{~/.local/bin} directory (which keeps the executables/programs or more
generally, ``binary'' files) to @file{PATH} with the following command.
-As defined below, first the existing value of @file{PATH} is used, then your
given directory is added to its end and the combined value is put back in
@file{PATH} (run `@command{$ echo $PATH}' afterwards to check if it was added).
-@example
-$ PATH=$PATH:~/.local/bin
-@end example
-@cindex GNU Bash
-@cindex Startup scripts
-@cindex Scripts, startup
-Any executable that you installed in @file{~/.local/bin} will now be usable
without having to remember and write its full address.
-However, as soon as you leave/close your current terminal session, this
modified @file{PATH} variable will be forgotten.
-Adding the directories which contain executables to the @file{PATH}
environment variable each time you start a terminal is also very inconvenient
and prone to errors.
-Fortunately, there are standard `startup files' defined by your shell
precisely for this (and other) purposes.
-There is a special startup file for every significant starting step:
-@table @asis
-@cindex GNU Bash
-@item @file{/etc/profile} and everything in @file{/etc/profile.d/}
-These startup scripts are called when your whole system starts (for example
after you turn on your computer).
-Therefore you need administrator or root privileges to access or modify them.
+@node Downloading the source, Build and install, Dependencies, Installation
+@section Downloading the source
-@item @file{~/.bash_profile}
-If you are using (GNU) Bash as your shell, the commands in this file are run,
when you log in to your account @emph{through Bash}.
-Most commonly when you login through the virtual console (where there is no
graphic user interface).
+Gnuastro's source code can be downloaded in two ways.
+As a tarball, ready to be configured and installed on your system (as
described in @ref{Quick start}), see @ref{Release tarball}.
+If you want official releases of stable versions this is the best, easiest and
most common option.
+Alternatively, you can clone the version controlled history of Gnuastro, run
one extra bootstrapping step and then follow the same steps as the tarball.
+This will give you access to all the most recent work that will be included in
the next release along with the full project history.
+The process is thoroughly introduced in @ref{Version controlled source}.
-@item @file{~/.bashrc}
-If you are using (GNU) Bash as your shell, the commands here will be run each
time you start a terminal and are already logged in.
-For example, when you open your terminal emulator in the graphic user
interface.
-@end table
-For security reasons, it is highly recommended to directly type in your
@file{HOME} directory value by hand in startup files instead of using variables.
-So in the following, let's assume your user name is `@file{name}' (so @file{~}
may be replaced with @file{/home/name}).
-To add @file{~/.local/bin} to your @file{PATH} automatically on any startup
file, you have to ``export'' the new value of @command{PATH} in the startup
file that is most relevant to you by adding this line:
+@menu
+* Release tarball:: Download a stable official release.
+* Version controlled source:: Get and use the version controlled source.
+@end menu
-@example
-export PATH=$PATH:/home/name/.local/bin
-@end example
+@node Release tarball, Version controlled source, Downloading the source,
Downloading the source
+@subsection Release tarball
-@cindex GNU build system
-@cindex Install directory
-@cindex Directory, install
-Now that you know your system will look into @file{~/.local/bin} for
executables, you can tell Gnuastro's configure script to install everything in
the top @file{~/.local} directory using the @option{--prefix} option.
-When you subsequently run @command{$ make install}, all the install-able files
will be put in their respective directory under @file{~/.local/} (the
executables in @file{~/.local/bin}, the compiled library files in
@file{~/.local/lib}, the library header files in @file{~/.local/include} and so
on, to learn more about these different files, please see @ref{Review of
library fundamentals}).
-Note that tilde (`@key{~}') expansion will not happen if you put a `@key{=}'
between @option{--prefix} and @file{~/.local}@footnote{If you insist on using
`@key{=}', you can use @option{--prefix=$HOME/.local}.}, so we have avoided the
@key{=} character here which is optional in GNU-style options, see
@ref{Options}.
+A release tarball (commonly compressed) is the most common way of obtaining
free and open source software.
+A tarball is a snapshot of one particular moment in the Gnuastro development
history along with all the necessary files to configure, build, and install
Gnuastro easily (see @ref{Quick start}).
+It is very straightforward and needs the least set of dependencies (see
@ref{Mandatory dependencies}).
+Gnuastro has tarballs for official stable releases and pre-releases for
testing.
+See @ref{Version numbering} for more on the two types of releases and the
formats of the version numbers.
+The URLs for each type of release are given below.
-@example
-$ ./configure --prefix ~/.local
-@end example
+@table @asis
-@cindex @file{MANPATH}
-@cindex @file{INFOPATH}
-@cindex @file{LD_LIBRARY_PATH}
-@cindex Library search directory
-@cindex Default library search directory
-You can install everything (including libraries like GSL, CFITSIO, or WCSLIB
which are Gnuastro's mandatory dependencies, see @ref{Mandatory dependencies})
locally by configuring them as above.
-However, recall that @command{PATH} is only for executable files, not
libraries and that libraries can also depend on other libraries.
-For example WCSLIB depends on CFITSIO and Gnuastro needs both.
-Therefore, when you installed a library in a non-recognized directory, you
have to guide the program that depends on them to look into the necessary
library and header file directories.
-To do that, you have to define the @command{LDFLAGS} and @command{CPPFLAGS}
environment variables respectively.
-This can be done while calling @file{./configure} as shown below:
+@item Official stable releases (@url{http://ftp.gnu.org/gnu/gnuastro}):
+This URL hosts the official stable releases of Gnuastro.
+Always use the most recent version (see @ref{Version numbering}).
+By clicking on the ``Last modified'' title of the second column, the files
will be sorted by their date which you can also use to find the latest version.
+It is recommended to use a mirror to download these tarballs, please visit
@url{http://ftpmirror.gnu.org/gnuastro/} and see below.
-@example
-$ ./configure LDFLAGS=-L/home/name/.local/lib \
- CPPFLAGS=-I/home/name/.local/include \
- --prefix ~/.local
-@end example
+@item Pre-release tar-balls (@url{http://alpha.gnu.org/gnu/gnuastro}):
+This URL contains unofficial pre-release versions of Gnuastro.
+The pre-release versions of Gnuastro here are for enthusiasts to try out
before an official release.
+If there are problems, or bugs then the testers will inform the developers to
fix before the next official release.
+See @ref{Version numbering} to understand how the version numbers here are
formatted.
+If you want to remain even more up-to-date with the developing activities,
please clone the version controlled source as described in @ref{Version
controlled source}.
-It can be annoying/buggy to do this when configuring every software that
depends on such libraries.
-Hence, you can define these two variables in the most relevant startup file
(discussed above).
-The convention on using these variables doesn't include a colon to separate
values (as @command{PATH}-like variables do), they use white space characters
and each value is prefixed with a compiler option@footnote{These variables are
ultimately used as options while building the programs, so every value has be
an option name followed be a value as discussed in @ref{Options}.}: note the
@option{-L} and @option{-I} above (see @ref{Options}), for @option{-I} see
@ref{Headers}, and for @optio [...]
-Therefore we have to keep the value in double quotation signs to keep the
white space characters and adding the following two lines to the startup file
of choice:
+@end table
-@example
-export LDFLAGS="$LDFLAGS -L/home/name/.local/lib"
-export CPPFLAGS="$CPPFLAGS -I/home/name/.local/include"
-@end example
+@cindex Gzip
+@cindex Lzip
+Gnuastro's official/stable tarball is released with two formats: Gzip (with
suffix @file{.tar.gz}) and Lzip (with suffix @file{.tar.lz}).
+The pre-release tarballs (after version 0.3) are released only as an Lzip
tarball.
+Gzip is a very well-known and widely used compression program created by GNU
and available in most systems.
+However, Lzip provides a better compression ratio and more robust archival
capacity.
+For example Gnuastro 0.3's tarball was 2.9MB and 4.3MB with Lzip and Gzip
respectively, see the @url{http://www.nongnu.org/lzip/lzip.html, Lzip web page}
for more.
+Lzip might not be pre-installed in your operating system, if so, installing it
from your operating system's package manager or from source is very easy and
fast (it is a very small program).
-@cindex Dynamic libraries
-Dynamic libraries are linked to the executable every time you run a program
that depends on them (see @ref{Linking} to fully understand this important
concept).
-Hence dynamic libraries also require a special path variable called
@command{LD_LIBRARY_PATH} (same formatting as @command{PATH}).
-To use programs that depend on these libraries, you need to add
@file{~/.local/lib} to your @command{LD_LIBRARY_PATH} environment variable by
adding the following line to the relevant start-up file:
+The GNU FTP server is mirrored (has backups) in various locations on the globe
(@url{http://www.gnu.org/order/ftp.html}).
+You can use the closest mirror to your location for a more faster download.
+Note that only some mirrors keep track of the pre-release (alpha) tarballs.
+Also note that if you want to download immediately after and announcement (see
@ref{Announcements}), the mirrors might need some time to synchronize with the
main GNU FTP server.
-@example
-export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/name/.local/lib
-@end example
-If you also want to access the Info (see @ref{Info}) and man pages (see
@ref{Man pages}) documentations add @file{~/.local/share/info} and
@file{~/.local/share/man} to your @command{INFOPATH}@footnote{Info has the
following convention: ``If the value of @command{INFOPATH} ends with a colon
[or it isn't defined] ..., the initial list of directories is constructed by
appending the build-time default to the value of @command{INFOPATH}.'' So when
installing in a non-standard directory and if [...]
+@node Version controlled source, , Release tarball, Downloading the source
+@subsection Version controlled source
-@cindex Search directory order
-@cindex Order in search directory
-A final note is that order matters in the directories that are searched for
all the variables discussed above.
-In the examples above, the new directory was added after the system specified
directories.
-So if the program, library or manuals are found in the system wide
directories, the user directory is no longer searched.
-If you want to search your local installation first, put the new directory
before the already existing list, like the example below.
+@cindex Git
+@cindex Version control
+The publicly distributed Gnuastro tar-ball (for example
@file{gnuastro-X.X.tar.gz}) does not contain the revision history, it is only a
snapshot of the source code at one significant instant of Gnuastro's history
(specified by the version number, see @ref{Version numbering}), ready to be
configured and built.
+To be able to develop successfully, the revision history of the code can be
very useful to track when something was added or changed, also some updates
that are not yet officially released might be in it.
+
+We use Git for the version control of Gnuastro.
+For those who are not familiar with it, we recommend the
@url{https://git-scm.com/book/en, ProGit book}.
+The whole book is publicly available for online reading and downloading and
does a wonderful job at explaining the concepts and best practices.
+
+Let's assume you want to keep Gnuastro in the @file{TOPGNUASTRO} directory
(can be any directory, change the value below).
+The full version controlled history of Gnuastro can be cloned in
@file{TOPGNUASTRO/gnuastro} by running the following commands@footnote{If your
internet connection is active, but Git complains about the network, it might be
due to your network setup not recognizing the Git protocol.
+In that case use the following URL which uses the HTTP protocol instead:
@command{http://git.sv.gnu.org/r/gnuastro.git}}:
@example
-export LD_LIBRARY_PATH=/home/name/.local/lib:$LD_LIBRARY_PATH
+$ TOPGNUASTRO=/home/yourname/Research/projects/
+$ cd $TOPGNUASTRO
+$ git clone git://git.sv.gnu.org/gnuastro.git
@end example
@noindent
-This is good when a library, for example CFITSIO, is already present on the
system, but the system-wide install wasn't configured with the correct
configuration flags (see @ref{CFITSIO}), or you want to use a newer version and
you don't have administrator or root access to update it on the whole
system/server.
-If you update @file{LD_LIBRARY_PATH} by placing @file{~/.local/lib} first
(like above), the linker will first find the CFITSIO you installed for yourself
and link with it.
-It thus will never reach the system-wide installation.
+The @file{$TOPGNUASTRO/gnuastro} directory will contain hand-written (version
controlled) source code for Gnuastro's programs, libraries, this book and the
tests.
+All are divided into sub-directories with standard and very descriptive names.
+The version controlled files in the top cloned directory are either mainly in
capital letters (for example @file{THANKS} and @file{README}) or mainly written
in small-caps (for example @file{configure.ac} and @file{Makefile.am}).
+The former are non-programming, standard writing for human readers containing
high-level information about the whole package.
+The latter are instructions to customize the GNU build system for Gnuastro.
+For more on Gnuastro's source code structure, please see @ref{Developing}.
+We won't go any deeper here.
-There are important security problems with using local installations first:
all important system-wide executables and libraries (important executables like
@command{ls} and @command{cp}, or libraries like the C library) can be replaced
by non-secure versions with the same file names and put in the customized
directory (@file{~/.local} in this example).
-So if you choose to search in your customized directory first, please @emph{be
sure} to keep it clean from executables or libraries with the same names as
important system programs or libraries.
+The cloned Gnuastro source cannot immediately be configured, compiled, or
installed since it only contains hand-written files, not automatically
generated or imported files which do all the hard work of the build process.
+See @ref{Bootstrapping} for the process of generating and importing those
files (its not too hard!).
+Once you have bootstrapped Gnuastro, you can run the standard procedures (in
@ref{Quick start}).
+Very soon after you have cloned it, Gnuastro's main @file{master} branch will
be updated on the main repository (since the developers are actively working on
Gnuastro), for the best practices in keeping your local history in sync with
the main repository see @ref{Synchronizing}.
-@cartouche
-@noindent
-@strong{Summary:} When you are using a server which doesn't give you
administrator/root access AND you would like to give priority to your own built
programs and libraries, not the version that is (possibly already) present on
the server, add these lines to your startup file.
-See above for which startup file is best for your case and for a detailed
explanation on each.
-Don't forget to replace `@file{/YOUR-HOME-DIR}' with your home directory (for
example `@file{/home/your-id}'):
-@example
-export PATH="/YOUR-HOME-DIR/.local/bin:$PATH"
-export LDFLAGS="-L/YOUR-HOME-DIR/.local/lib $LDFLAGS"
-export MANPATH="/YOUR-HOME-DIR/.local/share/man/:$MANPATH"
-export CPPFLAGS="-I/YOUR-HOME-DIR/.local/include $CPPFLAGS"
-export INFOPATH="/YOUR-HOME-DIR/.local/share/info/:$INFOPATH"
-export LD_LIBRARY_PATH="/YOUR-HOME-DIR/.local/lib:$LD_LIBRARY_PATH"
-@end example
+
+
+@menu
+* Bootstrapping:: Adding all the automatically generated files.
+* Synchronizing:: Keep your local clone up to date.
+@end menu
+
+@node Bootstrapping, Synchronizing, Version controlled source, Version
controlled source
+@subsubsection Bootstrapping
+
+@cindex Bootstrapping
+@cindex GNU Autoconf Archive
+@cindex Gnulib: GNU Portability Library
+@cindex GNU Portability Library (Gnulib)
+@cindex Automatically created build files
@noindent
-Afterwards, you just need to add an extra
@option{--prefix=/YOUR-HOME-DIR/.local} to the @file{./configure} command of
the software that you intend to install.
-Everything else will be the same as a standard build and install, see
@ref{Quick start}.
-@end cartouche
+The version controlled source code lacks the source files that we have not
written or are automatically built.
+These automatically generated files are included in the distributed tar ball
for each distribution (for example @file{gnuastro-X.X.tar.gz}, see @ref{Version
numbering}) and make it easy to immediately configure, build, and install
Gnuastro.
+However from the perspective of version control, they are just bloatware and
sources of confusion (since they are not changed by Gnuastro developers).
-@node Executable names, Configure and build in RAM, Installation directory,
Configuring
-@subsubsection Executable names
+The process of automatically building and importing necessary files into the
cloned directory is known as @emph{bootstrapping}.
+After bootstrapping is done you are ready to follow the default GNU build
steps that you normally run on the tarball (@command{./configure && make} for
example, described more in @ref{Quick start}).
+Some known issues with bootstrapping may occur during the process, to see how
to fix them, please see @ref{Known issues}.
-@cindex Executable names
-@cindex Names of executables
-At first sight, the names of the executables for each program might seem to be
uncommonly long, for example @command{astnoisechisel} or @command{astcrop}.
-We could have chosen terse (and cryptic) names like most programs do.
-We chose this complete naming convention (something like the commands in
@TeX{}) so you don't have to spend too much time remembering what the name of a
specific program was.
-Such complete names also enable you to easily search for the programs.
+All the instructions for an automatic bootstrapping are available in
@file{bootstrap} and configured using @file{bootstrap.conf}.
+@file{bootstrap} and @file{COPYING} (which contains the software copyright
notice) are the only files not written by Gnuastro developers but under version
control to enable simple bootstrapping and legal information on usage
immediately after cloning.
+@file{bootstrap.conf} is maintained by the GNU Portability Library (Gnulib)
and this file is an identical copy, so do not make any changes in this file
since it will be replaced when Gnulib releases an update.
+Make all your changes in @file{bootstrap.conf}.
-@cindex Shell auto-complete
-@cindex Auto-complete in the shell
-To facilitate typing the names in, we suggest using the shell auto-complete.
-With this facility you can find the executable you want very easily.
-It is very similar to file name completion in the shell.
-For example, simply by typing the letters below (where @key{[TAB]} stands for
the Tab key on your keyboard)
+The bootstrapping process has its own separate set of dependencies, the full
list is given in @ref{Bootstrapping dependencies}.
+They are generally very low-level and used by a very large set of commonly
used programs, so they are probably already installed on your system.
+The simplest way to bootstrap Gnuastro is to simply run the bootstrap script
within your cloned Gnuastro directory as shown below.
+However, please read the next paragraph before doing so (see @ref{Version
controlled source} for @file{TOPGNUASTRO}).
@example
-$ ast[TAB][TAB]
+$ cd TOPGNUASTRO/gnuastro
+$ ./bootstrap # Requires internet connection
@end example
-@noindent
-you will get the list of all the available executables that start with
@command{ast} in your @command{PATH} environment variable directories.
-So, all the Gnuastro executables installed on your system will be listed.
-Typing the next letter for the specific program you want along with a Tab,
will limit this list until you get to your desired program.
+Without any options, @file{bootstrap} will clone Gnulib within your cloned
Gnuastro directory (@file{TOPGNUASTRO/gnuastro/gnulib}) and download the
necessary Autoconf archives macros.
+So if you run bootstrap like this, you will need an internet connection every
time you decide to bootstrap.
+Also, Gnulib is a large package and cloning it can be slow.
+It will also keep the full Gnulib repository within your Gnuastro repository,
so if another one of your projects also needs Gnulib, and you insist on running
bootstrap like this, you will have two copies.
+In case you regularly backup your important files, Gnulib will also slow down
the backup process.
+Therefore while the simple invocation above can be used with no problem, it is
not recommended.
+To do better, see the next paragraph.
-@cindex Names, customize
-@cindex Customize executable names
-In case all of this does not convince you and you still want to type short
names, some suggestions are given below.
-You should have in mind though, that if you are writing a shell script that
you might want to pass on to others, it is best to use the standard name
because other users might not have adopted the same customization.
-The long names also serve as a form of documentation in such scripts.
-A similar reasoning can be given for option names in scripts: it is good
practice to always use the long formats of the options in shell scripts, see
@ref{Options}.
+The recommended way to get these two packages is thoroughly discussed in
@ref{Bootstrapping dependencies} (in short: clone them in the separate
@file{DEVDIR/} directory).
+The following commands will take you into the cloned Gnuastro directory and
run the @file{bootstrap} script, while telling it to copy some files (instead
of making symbolic links, with the @option{--copy} option, this is not
mandatory@footnote{The @option{--copy} option is recommended because some
backup systems might do strange things with symbolic links.}) and where to look
for Gnulib (with the @option{--gnulib-srcdir} option).
+Please note that the address given to @option{--gnulib-srcdir} has to be an
absolute address (so don't use @file{~} or @file{../} for example).
-@cindex Symbolic link
-The simplest solution is making a symbolic link to the actual executable.
-For example let's assume you want to type @file{ic} to run Crop instead of
@file{astcrop}.
-Assuming you installed Gnuastro executables in @file{/usr/local/bin} (default)
you can do this simply by running the following command as root:
+@example
+$ cd $TOPGNUASTRO/gnuastro
+$ ./bootstrap --copy --gnulib-srcdir=$DEVDIR/gnulib
+@end example
+
+@cindex GNU Texinfo
+@cindex GNU Libtool
+@cindex GNU Autoconf
+@cindex GNU Automake
+@cindex GNU C library
+@cindex GNU build system
+Since Gnulib and Autoconf archives are now available in your local
directories, you don't need an internet connection every time you decide to
remove all untracked files and redo the bootstrap (see box below).
+You can also use the same command on any other project that uses Gnulib.
+All the necessary GNU C library functions, Autoconf macros and Automake inputs
are now available along with the book figures.
+The standard GNU build system (@ref{Quick start}) will do the rest of the job.
+
+@cartouche
+@noindent
+@strong{Undoing the bootstrap:}
+During the development, it might happen that you want to remove all the
automatically generated and imported files.
+In other words, you might want to reverse the bootstrap process.
+Fortunately Git has a good program for this job: @command{git clean}.
+Run the following command and every file that is not version controlled will
be removed.
@example
-# ln -s /usr/local/bin/astcrop /usr/local/bin/ic
+git clean -fxd
@end example
@noindent
-In case you update Gnuastro and a new version of Crop is installed, the
-default executable name is the same, so your custom symbolic link still
-works.
+It is best to commit any recent change before running this command.
+You might have created new files since the last commit and if they haven't
been committed, they will all be gone forever (using @command{rm}).
+To get a list of the non-version controlled files instead of deleting them,
add the @option{n} option to @command{git clean}, so it becomes @option{-fxdn}.
+@end cartouche
-@vindex --program-prefix
-@vindex --program-suffix
-@vindex --program-transform-name
-The installed executable names can also be set using options to @command{$
./configure}, see @ref{Configuring}.
-GNU Autoconf (which configures Gnuastro for your particular system), allows
the builder to change the name of programs with the three options
@option{--program-prefix}, @option{--program-suffix} and
@option{--program-transform-name}.
-The first two are for adding a fixed prefix or suffix to all the programs that
will be installed.
-This will actually make all the names longer! You can use it to add versions
of program names to the programs in order to simultaneously have two executable
versions of a program.
+Besides the @file{bootstrap} and @file{bootstrap.conf}, the
@file{bootstrapped/} directory and @file{README-hacking} file are also related
to the bootstrapping process.
+The former hosts all the imported (bootstrapped) directories.
+Thus, in the version controlled source, it only contains a @file{REAME} file,
but in the distributed tar-ball it also contains sub-directories filled with
all bootstrapped files.
+@file{README-hacking} contains a summary of the bootstrapping process
discussed in this section.
+It is a necessary reference when you haven't built this book yet.
+It is thus not distributed in the Gnuastro tarball.
-@cindex SED, stream editor
-@cindex Stream editor, SED
-The third configure option allows you to set the executable name at install
time using the SED program.
-SED is a very useful `stream editor'.
-There are various resources on the internet to use it effectively.
-However, we should caution that using configure options will change the actual
executable name of the installed program and on every re-install (an update for
example), you have to also add this option to keep the old executable name
updated.
-Also note that the documentation or configuration files do not change from
their standard names either.
-@cindex Removing @file{ast} from executables
-For example, let's assume that typing @file{ast} on every invocation of every
program is really annoying you! You can remove this prefix from all the
executables at configure time by adding this option:
+@node Synchronizing, , Bootstrapping, Version controlled source
+@subsubsection Synchronizing
+
+The bootstrapping script (see @ref{Bootstrapping}) is not regularly needed:
you mainly need it after you have cloned Gnuastro (once) and whenever you want
to re-import the files from Gnulib, or Autoconf
archives@footnote{@url{https://savannah.gnu.org/task/index.php?13993} is
defined for you to check if significant (for Gnuastro) updates are made in
these repositories, since the last time you pulled from them.} (not too common).
+However, Gnuastro developers are constantly working on Gnuastro and are
pushing their changes to the official repository.
+Therefore, your local Gnuastro clone will soon be out-dated.
+Gnuastro has two mailing lists dedicated to its developing activities (see
@ref{Developing mailing lists}).
+Subscribing to them can help you decide when to synchronize with the official
repository.
+
+To pull all the most recent work in Gnuastro, run the following command from
the top Gnuastro directory.
+If you don't already have a built system, ignore @command{make distclean}.
+The separate steps are described in detail afterwards.
@example
-$ ./configure --program-transform-name='s/ast/ /'
+$ make distclean && git pull && autoreconf -f
@end example
+@noindent
+You can also run the commands separately:
+@example
+$ make distclean
+$ git pull
+$ autoreconf -f
+@end example
-@node Configure and build in RAM, , Executable names, Configuring
-@subsubsection Configure and build in RAM
-
-@cindex File I/O
-@cindex Input/Output, file
-Gnuastro's configure and build process (the GNU build system) involves the
creation, reading, and modification of a large number of files (input/output,
or I/O).
-Therefore file I/O issues can directly affect the work of developers who need
to configure and build Gnuastro numerous times.
-Some of these issues are listed below:
+@cindex GNU Autoconf
+@cindex Mailing list: info-gnuastro
+@cindex @code{info-gnuastro@@gnu.org}
+If Gnuastro was already built in this directory, you don't want some outputs
from the previous version being mixed with outputs from the newly pulled work.
+Therefore, the first step is to clean/delete all the built files with
@command{make distclean}.
+Fortunately the GNU build system allows the separation of source and built
files (in separate directories).
+This is a great feature to keep your source directory clean and you can use it
to avoid the cleaning step.
+Gnuastro comes with a script with some useful options for this job.
+It is useful if you regularly pull recent changes, see @ref{Separate build and
source directories}.
-@itemize
-@cindex HDD
-@cindex SSD
-@item
-I/O will cause wear and tear on both the HDDs (mechanical failures) and
-SSDs (decreasing the lifetime).
+After the pull, we must re-configure Gnuastro with @command{autoreconf -f}
(part of GNU Autoconf).
+It will update the @file{./configure} script and all the
@file{Makefile.in}@footnote{In the GNU build system, @command{./configure} will
use the @file{Makefile.in} files to create the necessary @file{Makefile} files
that are later read by @command{make} to build the package.} files based on the
hand-written configurations (in @file{configure.ac} and the @file{Makefile.am}
files).
+After running @command{autoreconf -f}, a warning about @code{TEXI2DVI} might
show up, you can ignore that.
-@cindex Backup
-@item
-Having the built files mixed with the source files can greatly affect backing
up (synchronization) of source files (since it involves the management of a
large number of small files that are regularly changed.
-Backup software can of course be configured to ignore the built files and
directories.
-However, since the built files are mixed with the source files and can have a
large variety, this will require a high level of customization.
-@end itemize
+The most important reason for re-building Gnuastro's build system is to
generate/update the version number for your updated Gnuastro snapshot.
+This generated version number will include the commit information (see
@ref{Version numbering}).
+The version number is included in nearly all outputs of Gnuastro's programs,
therefore it is vital for reproducing an old result.
-@cindex tmpfs file system
-@cindex file systems, tmpfs
-One solution to address both these problems is to use the
@url{https://en.wikipedia.org/wiki/Tmpfs, tmpfs file system}.
-Any file in tmpfs is actually stored in the RAM (and possibly SWAP), not on
HDDs or SSDs.
-The RAM is built for extensive and fast I/O.
-Therefore the large number of file I/Os associated with configuring and
building will not harm the HDDs or SSDs.
-Due to the volatile nature of RAM, files in the tmpfs file-system will be
permanently lost after a power-off.
-Since all configured and built files are derivative files (not files that have
been directly written by hand) there is no problem in this and this feature can
be considered as an automatic cleanup.
+As a summary, be sure to run `@command{autoreconf -f}' after every change in
the Git history.
+This includes synchronization with the main server or even a commit you have
made yourself.
-@cindex Linux kernel
-@cindex GNU C library
-@cindex GNU build system
-The modern GNU C library (and thus the Linux kernel) defines the
@file{/dev/shm} directory for this purpose in the RAM (POSIX shared memory).
-To build in it, you can use the GNU build system's ability to build in a
separate directory (not necessarily in the source directory) as shown below.
-Just set @file{SRCDIR} as the address of Gnuastro's top source directory (for
example, the unpacked tarball).
+If you would like to see what has changed since you last synchronized your
local clone, you can take the following steps instead of the simple command
above (don't type anything after @code{#}):
@example
-$ mkdir /dev/shm/tmp-gnuastro-build
-$ cd /dev/shm/tmp-gnuastro-build
-$ SRCDIR/configure --srcdir=SRCDIR
-$ make
+$ git checkout master # Confirm if you are on master.
+$ git fetch origin # Fetch all new commits from server.
+$ git log master..origin/master # See all the new commit messages.
+$ git merge origin/master # Update your master branch.
+$ autoreconf -f # Update the build system.
@end example
-Gnuastro comes with a script to simplify this process of configuring and
building in a different directory (a ``clean'' build), for more see
@ref{Separate build and source directories}.
-
-@node Separate build and source directories, Tests, Configuring, Build and
install
-@subsection Separate build and source directories
+@noindent
+By default @command{git log} prints the most recent commit first, add the
@option{--reverse} option to see the changes chronologically.
+To see exactly what has been changed in the source code along with the commit
message, add a @option{-p} option to the @command{git log}.
-The simple steps of @ref{Quick start} will mix the source and built files.
-This can cause inconvenience for developers or enthusiasts following the the
most recent work (see @ref{Version controlled source}).
-The current section is mainly focused on this later group of Gnuastro users.
-If you just install Gnuastro on major releases (following
@ref{Announcements}), you can safely ignore this section.
+If you want to make changes in the code, have a look at @ref{Developing} to
get started easily.
+Be sure to commit your changes in a separate branch (keep your @code{master}
branch to follow the official repository) and re-run @command{autoreconf -f}
after the commit.
+If you intend to send your work to us, you can safely use your commit since it
will be ultimately recorded in Gnuastro's official history.
+If not, please upload your separate branch to a public hosting service, for
example @url{https://gitlab.com, GitLab}, and link to it in your report/paper.
+Alternatively, run @command{make distcheck} and upload the output
@file{gnuastro-X.X.X.XXXX.tar.gz} to a publicly accessible web page so your
results can be considered scientific (reproducible) later.
-@cindex GNU build system
-When it is necessary to keep the source (which is under version control), but
not the derivative (built) files (after checking or installing), the best
solution is to keep the source and the built files in separate directories.
-One application of this is already discussed in @ref{Configure and build in
RAM}.
-To facilitate this process of configuring and building in a separate
directory, Gnuastro comes with the @file{developer-build} script.
-It is available in the top source directory and is @emph{not} installed.
-It will make a directory under a given top-level directory (given to
@option{--top-build-dir}) and build Gnuastro in there directory.
-It thus keeps the source completely separated from the built files.
-For easy access to the built files, it also makes a symbolic link to the built
directory in the top source files called @file{build}.
-When run without any options, default values will be used for its
configuration.
-As with Gnuastro's programs, you can inspect the default values with
@option{-P} (or @option{--printparams}, the output just looks a little
different here).
-The default top-level build directory is @file{/dev/shm}: the shared memory
directory in RAM on GNU/Linux systems as described in @ref{Configure and build
in RAM}.
-@cindex Debug
-Besides these, it also has some features to facilitate the job of developers
or bleeding edge users like the @option{--debug} option to do a fast build,
with debug information, no optimization, and no shared libraries.
-Here is the full list of options you can feed to this script to configure its
operations.
-@cartouche
-@noindent
-@strong{Not all Gnuastro's common program behavior usable here:}
-@file{developer-build} is just a non-installed script with a very limited
scope as described above.
-It thus doesn't have all the common option behaviors or configuration files
for example.
-@end cartouche
-@cartouche
-@noindent
-@strong{White space between option and value:} @file{developer-build}
-doesn't accept an @key{=} sign between the options and their values.
-It also needs at least one character between the option and its value.
-Therefore @option{-n 4} or @option{--numthreads 4} are acceptable, while
@option{-n4}, @option{-n=4}, or @option{--numthreads=4} aren't.
-Finally multiple short option names cannot be merged: for example you can say
@option{-c -n 4}, but unlike Gnuastro's programs, @option{-cn4} is not
acceptable.
-@end cartouche
-@cartouche
-@noindent
-@strong{Reusable for other packages:} This script can be used in any software
which is configured and built using the GNU Build System.
-Just copy it in the top source directory of that software and run it from
there.
-@end cartouche
-@table @option
-@item -b STR
-@itemx --top-build-dir STR
-The top build directory to make a directory for the build.
-If this option isn't called, the top build directory is @file{/dev/shm} (only
available in GNU/Linux operating systems, see @ref{Configure and build in RAM}).
-@item -V
-@itemx --version
-Print the version string of Gnuastro that will be used in the build.
-This string will be appended to the directory name containing the built files.
-@item -a
-@itemx --autoreconf
-Run @command{autoreconf -f} before building the package.
-In Gnuastro, this is necessary when a new commit has been made to the project
history.
-In Gnuastro's build system, the Git description will be used as the version,
see @ref{Version numbering} and @ref{Synchronizing}.
-@item -c
-@itemx --clean
-@cindex GNU Autoreconf
-Delete the contents of the build directory (clean it) before starting the
configuration and building of this run.
-This is useful when you have recently pulled changes from the main Git
repository, or committed a change your self and ran @command{autoreconf -f},
see @ref{Synchronizing}.
-After running GNU Autoconf, the version will be updated and you need to do a
clean build.
+@node Build and install, , Downloading the source, Installation
+@section Build and install
-@item -d
-@itemx --debug
-@cindex Valgrind
-@cindex GNU Debugger (GDB)
-Build with debugging flags (for example to use in GNU Debugger, also known as
GDB, or Valgrind), disable optimization and also the building of shared
libraries.
-Similar to running the configure script of below
+This section is basically a longer explanation to the sequence of commands
given in @ref{Quick start}.
+If you didn't have any problems during the @ref{Quick start} steps, you want
to have all the programs of Gnuastro installed in your system, you don't want
to change the executable names during or after installation, you have root
access to install the programs in the default system wide directory, the Letter
paper size of the print book is fine for you or as a summary you don't feel
like going into the details when everything is working, you can safely skip
this section.
-@example
-$ ./configure --enable-debug
-@end example
+If you have any of the above problems or you want to understand the details
for a better control over your build and install, read along.
+The dependencies which you will need prior to configuring, building and
installing Gnuastro are explained in @ref{Dependencies}.
+The first three steps in @ref{Quick start} need no extra explanation, so we
will skip them and start with an explanation of Gnuastro specific configuration
options and a discussion on the installation directory in @ref{Configuring},
followed by some smaller subsections: @ref{Tests}, @ref{A4 print book}, and
@ref{Known issues} which explains the solutions to known problems you might
encounter in the installation steps and ways you can solve them.
-Besides all the debugging advantages of building with this option, it will
also be significantly speed up the build (at the cost of slower built programs).
-So when you are testing something small or working on the build system itself,
it will be much faster to test your work with this option.
-@item -v
-@itemx --valgrind
-@cindex Valgrind
-Build all @command{make check} tests within Valgrind.
-For more, see the description of @option{--enable-check-with-valgrind} in
@ref{Gnuastro configure options}.
+@menu
+* Configuring:: Configure Gnuastro
+* Separate build and source directories:: Keeping derivate/build files
separate.
+* Tests:: Run tests to see if it is working.
+* A4 print book:: Customize the print book.
+* Known issues:: Issues you might encounter.
+@end menu
-@item -j INT
-@itemx --jobs INT
-The maximum number of threads/jobs for Make to build at any moment.
-As the name suggests (Make has an identical option), the number given to this
option is directly passed on to any call of Make with its @option{-j} option.
-@item -C
-@itemx --check
-After finishing the build, also run @command{make check}.
-By default, @command{make check} isn't run because the developer usually has
their own checks to work on (for example defined in @file{tests/during-dev.sh}).
-@item -i
-@itemx --install
-After finishing the build, also run @command{make install}.
-@item -D
-@itemx --dist
-Run @code{make dist-lzip pdf} to build a distribution tarball (in
@file{.tar.lz} format) and a PDF manual.
-This can be useful for archiving, or sending to colleagues who don't use Git
for an easy build and manual.
-@item -u STR
-@item --upload STR
-Activate the @option{--dist} (@option{-D}) option, then use secure copy
(@command{scp}, part of the SSH tools) to copy the tarball and PDF to the
@file{src} and @file{pdf} sub-directories of the specified server and its
directory (value to this option).
-For example @command{--upload my-server:dir}, will copy the tarball in the
@file{dir/src}, and the PDF manual in @file{dir/pdf} of @code{my-server} server.
-It will then make a symbolic link in the top server directory to the tarball
that is called @file{gnuastro-latest.tar.lz}.
+@node Configuring, Separate build and source directories, Build and install,
Build and install
+@subsection Configuring
-@item -p
-@itemx --publish
-Short for @option{--autoreconf --clean --debug --check --upload STR}.
-@option{--debug} is added because it will greatly speed up the build.
-It will have no effect on the produced tarball.
-This is good when you have made a commit and are ready to publish it on your
server (if nothing crashes).
-Recall that if any of the previous steps fail the script aborts.
+@pindex ./configure
+@cindex Configuring
+The @command{$ ./configure} step is the most important step in the build and
install process.
+All the required packages, libraries, headers and environment variables are
checked in this step.
+The behaviors of make and make install can also be set through command line
options to this command.
-@item -I
-@item --install-archive
-Short for @option{--autoreconf --clean --check --install --dist}.
-This is useful when you actually want to install the commit you just made (if
the build and checks succeed).
-It will also produce a distribution tarball and PDF manual for easy access to
the installed tarball on your system at a later time.
+@cindex Configure options
+@cindex Customizing installation
+@cindex Installation, customizing
+The configure script accepts various arguments and options which enable the
final user to highly customize whatever she is building.
+The options to configure are generally very similar to normal program options
explained in @ref{Arguments and options}.
+Similar to all GNU programs, you can get a full list of the options along with
a short explanation by running
-Ideally, Gnuastro's Git version history makes it easy for a prepared system to
revert back to a different point in history.
-But Gnuastro also needs to bootstrap files and also your collaborators might
(usually do!) find it too much of a burden to do the bootstrapping themselves.
-So it is convenient to have a tarball and PDF manual of the version you have
installed (and are using in your research) handily available.
+@example
+$ ./configure --help
+@end example
-@item -h
-@itemx --help
-@itemx -P
-@itemx --printparams
-Print a description of this script along with all the options and their
-current values.
+@noindent
+@cindex GNU Autoconf
+A complete explanation is also included in the @file{INSTALL} file.
+Note that this file was written by the authors of GNU Autoconf (which builds
the @file{configure} script), therefore it is common for all programs which use
the @command{$ ./configure} script for building and installing, not just
Gnuastro.
+Here we only discuss cases where you don't have super-user access to the
system and if you want to change the executable names.
+But before that, a review of the options to configure that are particular to
Gnuastro are discussed.
-@end table
+@menu
+* Gnuastro configure options:: Configure options particular to Gnuastro.
+* Installation directory:: Specify the directory to install.
+* Executable names:: Changing executable names.
+* Configure and build in RAM:: For minimal use of HDD or SSD, and clean
source.
+@end menu
+@node Gnuastro configure options, Installation directory, Configuring,
Configuring
+@subsubsection Gnuastro configure options
+@cindex @command{./configure} options
+@cindex Configure options particular to Gnuastro
+Most of the options to configure (which are to do with building) are similar
for every program which uses this script.
+Here the options that are particular to Gnuastro are discussed.
+The next topics explain the usage of other configure options which can be
applied to any program using the GNU build system (through the configure
script).
-@node Tests, A4 print book, Separate build and source directories, Build and
install
-@subsection Tests
+@vtable @option
-@cindex @command{make check}
-@cindex @file{mock.fits}
-@cindex Tests, running
-@cindex Checking tests
-After successfully building (compiling) the programs with the @command{$ make}
command you can check the installation before installing.
-To run the tests, run
-
-@example
-$ make check
-@end example
-
-For every program some tests are designed to check some possible operations.
-Running the command above will run those tests and give you a final report.
-If everything is OK and you have built all the programs, all the tests should
pass.
-In case any of the tests fail, please have a look at @ref{Known issues} and if
that still doesn't fix your problem, look that the
@file{./tests/test-suite.log} file to see if the source of the error is
something particular to your system or more general.
-If you feel it is general, please contact us because it might be a bug.
-Note that the tests of some programs depend on the outputs of other program's
tests, so if you have not installed them they might be skipped or fail.
-Prior to releasing every distribution all these tests are checked.
-If you have a reasonably modern terminal, the outputs of the successful tests
will be colored green and the failed ones will be colored red.
-
-These scripts can also act as a good set of examples for you to see how the
programs are run.
-All the tests are in the @file{tests/} directory.
-The tests for each program are shell scripts (ending with @file{.sh}) in a
sub-directory of this directory with the same name as the program.
-See @ref{Test scripts} for more detailed information about these scripts in
case you want to inspect them.
+@item --enable-debug
+@cindex Valgrind
+@cindex Debugging
+@cindex GNU Debugger
+Compile/build Gnuastro with debugging information, no optimization and without
shared libraries.
+In order to allow more efficient programs when using Gnuastro (after the
installation), by default Gnuastro is built with a 3rd level (a very high
level) optimization and no debugging information.
+By default, libraries are also built for static @emph{and} shared linking (see
@ref{Linking}).
+However, when there are crashes or unexpected behavior, these three features
can hinder the process of localizing the problem.
+This configuration option is identical to manually calling the configuration
script with @code{CFLAGS="-g -O0" --disable-shared}.
+In the (rare) situations where you need to do your debugging on the shared
libraries, don't use this option.
+Instead run the configure script by explicitly setting @code{CFLAGS} like this:
+@example
+$ ./configure CFLAGS="-g -O0"
+@end example
+@item --enable-check-with-valgrind
+@cindex Valgrind
+Do the @command{make check} tests through Valgrind.
+Therefore, if any crashes or memory-related issues (segmentation faults in
particular) occur in the tests, the output of Valgrind will also be put in the
@file{tests/test-suite.log} file without having to manually modify the check
scripts.
+This option will also activate Gnuastro's debug mode (see the
@option{--enable-debug} configure-time option described above).
-@node A4 print book, Known issues, Tests, Build and install
-@subsection A4 print book
+Valgrind is free software.
+It is a program for easy checking of memory-related issues in programs.
+It runs a program within its own controlled environment and can thus identify
the exact line-number in the program's source where a memory-related issue
occurs.
+However, it can significantly slow-down the tests.
+So this option is only useful when a segmentation fault is found during
@command{make check}.
-@cindex A4 print book
-@cindex Modifying print book
-@cindex A4 paper size
-@cindex US letter paper size
-@cindex Paper size, A4
-@cindex Paper size, US letter
-The default print version of this book is provided in the letter paper size.
-If you would like to have the print version of this book on paper and you are
living in a country which uses A4, then you can rebuild the book.
-The great thing about the GNU build system is that the book source code which
is in Texinfo is also distributed with the program source code, enabling you to
do such customization (hacking).
+@item --enable-progname
+Only build and install @file{progname} along with any other program that is
enabled in this fashion.
+@file{progname} is the name of the executable without the @file{ast}, for
example @file{crop} for Crop (with the executable name of @file{astcrop}).
-@cindex GNU Texinfo
-In order to change the paper size, you will need to have GNU Texinfo installed.
-Open @file{doc/gnuastro.texi} with any text editor.
-This is the source file that created this book.
-In the first few lines you will see this line:
+Note that by default all the programs will be installed.
+This option (and the @option{--disable-progname} options) are only relevant
when you don't want to install all the programs.
+Therefore, if this option is called for any of the programs in Gnuastro, any
program which is not explicitly enabled will not be built or installed.
-@example
-@@c@@afourpaper
-@end example
+@item --disable-progname
+@itemx --enable-progname=no
+Do not build or install the program named @file{progname}.
+This is very similar to the @option{--enable-progname}, but will build and
install all the other programs except this one.
+@cartouche
@noindent
-In Texinfo, a line is commented with @code{@@c}.
-Therefore, un-comment this line by deleting the first two characters such that
it changes to:
-
-@example
-@@afourpaper
-@end example
+@strong{Note:} If some programs are enabled and some are disabled, it is
equivalent to simply enabling those that were enabled.
+Listing the disabled programs is redundant.
+@end cartouche
-@noindent
-Save the file and close it.
-You can now run the following command
+@item --enable-gnulibcheck
+@cindex GNU C library
+@cindex Gnulib: GNU Portability Library
+@cindex GNU Portability Library (Gnulib)
+Enable checks on the GNU Portability Library (Gnulib).
+Gnulib is used by Gnuastro to enable users of non-GNU based operating systems
(that don't use GNU C library or glibc) to compile and use the advanced
features that this library provides.
+We make extensive use of such functions.
+If you give this option to @command{$ ./configure}, when you run @command{$
make check}, first the functions in Gnulib will be tested, then the Gnuastro
executables.
+If your operating system does not support glibc or has an older version of it
and you have problems in the build process (@command{$ make}), you can give
this flag to configure to see if the problem is caused by Gnulib not supporting
your operating system or Gnuastro, see @ref{Known issues}.
+@item --disable-guide-message
+@itemx --enable-guide-message=no
+Do not print a guiding message during the GNU Build process of @ref{Quick
start}.
+By default, after each step, a message is printed guiding the user what the
next command should be.
+Therefore, after @command{./configure}, it will suggest running @command{make}.
+After @command{make}, it will suggest running @command{make check} and so on.
+If Gnuastro is configured with this option, for example
@example
-$ make pdf
+$ ./configure --disable-guide-message
@end example
+Then these messages will not be printed after any step (like most programs).
+For people who are not yet fully accustomed to this build system, these
guidelines can be very useful and encouraging.
+However, if you find those messages annoying, use this option.
-@noindent
-and the new PDF book will be available in @file{SRCdir/doc/gnuastro.pdf}.
-By changing the @command{pdf} in @command{$ make pdf} to @command{ps} or
@command{dvi} you can have the book in those formats.
-Note that you can do this for any book that is in Texinfo format, they might
not have @code{@@afourpaper} line, so you can add it close to the top of the
Texinfo source file.
+@item --without-libgit2
+@cindex Git
+@pindex libgit2
+@cindex Version control systems
+Build Gnuastro without libgit2 (for including Git commit hashes in output
files), see @ref{Optional dependencies}.
+libgit2 is an optional dependency, with this option, Gnuastro will ignore any
possibly existing libgit2 that may already be on the system.
+@item --without-libjpeg
+@pindex libjpeg
+@cindex JPEG format
+Build Gnuastro without libjpeg (for reading/writing to JPEG files), see
@ref{Optional dependencies}.
+libjpeg is an optional dependency, with this option, Gnuastro will ignore any
possibly existing libjpeg that may already be on the system.
+@item --without-libtiff
+@pindex libtiff
+@cindex TIFF format
+Build Gnuastro without libtiff (for reading/writing to TIFF files), see
@ref{Optional dependencies}.
+libtiff is an optional dependency, with this option, Gnuastro will ignore any
possibly existing libtiff that may already be on the system.
+@end vtable
-@node Known issues, , A4 print book, Build and install
-@subsection Known issues
+The tests of some programs might depend on the outputs of the tests of other
programs.
+For example MakeProfiles is one the first programs to be tested when you run
@command{$ make check}.
+MakeProfiles' test outputs (FITS images) are inputs to many other programs
(which in turn provide inputs for other programs).
+Therefore, if you don't install MakeProfiles for example, the tests for many
the other programs will be skipped.
+To avoid this, in one run, you can install all the programs and run the tests
but not install.
+If everything is working correctly, you can run configure again with only the
programs you want.
+However, don't run the tests and directly install after building.
-Depending on your operating system and the version of the compiler you are
using, you might confront some known problems during the configuration
(@command{$ ./configure}), compilation (@command{$ make}) and tests (@command{$
make check}).
-Here, their solutions are discussed.
-@itemize
-@cindex Configuration, not finding library
-@cindex Development packages
-@item
-@command{$ ./configure}: @emph{Configure complains about not finding a library
even though you have installed it.}
-The possible solution is based on how you installed the package:
-@itemize
-@item
-From your distribution's package manager.
-Most probably this is because your distribution has separated the header files
of a library from the library parts.
-Please also install the `development' packages for those libraries too.
-Just add a @file{-dev} or @file{-devel} to the end of the package name and
re-run the package manager.
-This will not happen if you install the libraries from source.
-When installed from source, the headers are also installed.
+@node Installation directory, Executable names, Gnuastro configure options,
Configuring
+@subsubsection Installation directory
-@item
-@cindex @command{LDFLAGS}
-From source.
-Then your linker is not looking where you installed the library.
-If you followed the instructions in this chapter, all the libraries will be
installed in @file{/usr/local/lib}.
-So you have to tell your linker to look in this directory.
-To do so, configure Gnuastro like this:
+@vindex --prefix
+@cindex Superuser, not possible
+@cindex Root access, not possible
+@cindex No access to super-user install
+@cindex Install with no super-user access
+One of the most commonly used options to @file{./configure} is
@option{--prefix}, it is used to define the directory that will host all the
installed files (or the ``prefix'' in their final absolute file name).
+For example, when you are using a server and you don't have administrator or
root access.
+In this example scenario, if you don't use the @option{--prefix} option, you
won't be able to install the built files and thus access them from anywhere
without having to worry about where they are installed.
+However, once you prepare your startup file to look into the proper place (as
discussed thoroughly below), you will be able to easily use this option and
benefit from any software you want to install without having to ask the system
administrators or install and use a different version of a software that is
already installed on the server.
+
+The most basic way to run an executable is to explicitly write its full file
name (including all the directory information) and run it.
+One example is running the configuration script with the @command{$
./configure} command (see @ref{Quick start}).
+By giving a specific directory (the current directory or @file{./}), we are
explicitly telling the shell to look in the current directory for an executable
file named `@file{configure}'.
+Directly specifying the directory is thus useful for executables in the
current (or nearby) directories.
+However, when the program (an executable file) is to be used a lot, specifying
all those directories will become a significant burden.
+For example, the @file{ls} executable lists the contents in a given directory
and it is (usually) installed in the @file{/usr/bin/} directory by the
operating system maintainers.
+Therefore, if using the full address was the only way to access an executable,
each time you wanted a listing of a directory, you would have to run the
following command (which is very inconvenient, both in writing and in
remembering the various directories).
@example
-$ ./configure LDFLAGS="-L/usr/local/lib"
+$ /usr/bin/ls
@end example
-If you want to use the libraries for your other programming projects, then
-export this environment variable in a start-up script similar to the case
-for @file{LD_LIBRARY_PATH} explained below, also see @ref{Installation
-directory}.
-@end itemize
-
-@item
-@vindex --enable-gnulibcheck
-@cindex Gnulib: GNU Portability Library
-@cindex GNU Portability Library (Gnulib)
-@command{$ make}: @emph{Complains about an unknown function on a non-GNU based
operating system.}
-In this case, please run @command{$ ./configure} with the
@option{--enable-gnulibcheck} option to see if the problem is from the GNU
Portability Library (Gnulib) not supporting your system or if there is a
problem in Gnuastro, see @ref{Gnuastro configure options}.
-If the problem is not in Gnulib and after all its tests you get the same
complaint from @command{make}, then please contact us at
@file{bug-gnuastro@@gnu.org}.
-The cause is probably that a function that we have used is not supported by
your operating system and we didn't included it along with the source tar ball.
-If the function is available in Gnulib, it can be fixed immediately.
-
-@item
-@cindex @command{CPPFLAGS}
-@command{$ make}: @emph{Can't find the headers (.h files) of installed
libraries.}
-Your C pre-processor (CPP) isn't looking in the right place.
-To fix this, configure Gnuastro with an additional @code{CPPFLAGS} like below
(assuming the library is installed in @file{/usr/local/include}:
+@cindex Shell variables
+@cindex Environment variables
+To address this problem, we have the @file{PATH} environment variable.
+To understand it better, we will start with a short introduction to the shell
variables.
+Shell variable values are basically treated as strings of characters.
+For example, it doesn't matter if the value is a name (string of
@emph{alphabetic} characters), or a number (string of @emph{numeric}
characters), or both.
+You can define a variable and a value for it by running
+@example
+$ myvariable1=a_test_value
+$ myvariable2="a test value"
+@end example
+@noindent
+As you see above, if the value contains white space characters, you have to
put the whole value (including white space characters) in double quotes
(@key{"}).
+You can see the value it represents by running
+@example
+$ echo $myvariable1
+$ echo $myvariable2
+@end example
+@noindent
+@cindex Environment
+@cindex Environment variables
+If a variable has no value or it wasn't defined, the last command will only
print an empty line.
+A variable defined like this will be known as long as this shell or terminal
is running.
+Other terminals will have no idea it existed.
+The main advantage of shell variables is that if they are exported@footnote{By
running @command{$ export myvariable=a_test_value} instead of the simpler case
in the text}, subsequent programs that are run within that shell can access
their value.
+So by changing their value, you can change the ``environment'' of a program
which uses them.
+The shell variables which are accessed by programs are therefore known as
``environment variables''@footnote{You can use shell variables for other
actions too, for example to temporarily keep some names or run loops on some
files.}.
+You can see the full list of exported variables that your shell recognizes by
running:
@example
-$ ./configure CPPFLAGS="-I/usr/local/include"
+$ printenv
@end example
-If you want to use the libraries for your other programming projects, then
export this environment variable in a start-up script similar to the case for
@file{LD_LIBRARY_PATH} explained below, also see @ref{Installation directory}.
+@cindex @file{HOME}
+@cindex @file{HOME/.local/}
+@cindex Environment variable, @code{HOME}
+@file{HOME} is one commonly used environment variable, it is any user's (the
one that is logged in) top directory.
+Try finding it in the command above.
+It is used so often that the shell has a special expansion (alternative) for
it: `@file{~}'.
+Whenever you see file names starting with the tilde sign, it actually
represents the value to the @file{HOME} environment variable, so @file{~/doc}
is the same as @file{$HOME/doc}.
-@cindex Tests, only one passes
-@cindex @file{LD_LIBRARY_PATH}
-@item
-@command{$ make check}: @emph{Only the first couple of tests pass, all the
rest fail or get skipped.} It is highly likely that when searching for shared
libraries, your system doesn't look into the @file{/usr/local/lib} directory
(or wherever you installed Gnuastro or its dependencies).
-To make sure it is added to the list of directories, add the following line to
your @file{~/.bashrc} file and restart your terminal.
-Don't forget to change @file{/usr/local/lib} if the libraries are installed in
other (non-standard) directories.
+@vindex PATH
+@pindex ./configure
+@cindex Setting @code{PATH}
+@cindex Default executable search directory
+@cindex Search directory for executables
+Another one of the most commonly used environment variables is @file{PATH}, it
is a list of directories to search for executable names.
+Its value is a list of directories (separated by a colon, or `@key{:}').
+When the address of the executable is not explicitly given (like
@file{./configure} above), the system will look for the executable in the
directories specified by @file{PATH}.
+If you have a computer nearby, try running the following command to see which
directories your system will look into when it is searching for executable
(binary) files, one example is printed here (notice how @file{/usr/bin}, in the
@file{ls} example above, is one of the directories in @command{PATH}):
@example
-export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib"
+$ echo $PATH
+/usr/local/sbin:/usr/local/bin:/usr/bin
@end example
-You can also add more directories by using a colon `@code{:}' to separate them.
-See @ref{Installation directory} and @ref{Linking} to learn more on the
@code{PATH} variables and dynamic linking respectively.
-
-@cindex GPL Ghostscript
-@item
-@command{$ make check}: @emph{The tests relying on external programs (for
example @file{fitstopdf.sh} fail.)} This is probably due to the fact that the
version number of the external programs is too old for the tests we have
preformed.
-Please update the program to a more recent version.
-For example to create a PDF image, you will need GPL Ghostscript, but older
versions do not work, we have successfully tested it on version 9.15.
-Older versions might cause a failure in the test result.
-
-@item
-@cindex @TeX{}
-@cindex GNU Texinfo
-@command{$ make pdf}: @emph{The PDF book cannot be made.}
-To make a PDF book, you need to have the GNU Texinfo program (like any
program, the more recent the better).
-A working @TeX{} program is also necessary, which you can get from Tex
Live@footnote{@url{https://www.tug.org/texlive/}}.
-
-@item
-@cindex GNU Libtool
-After @code{make check}: do not copy the programs' executables to another (for
example, the installation) directory manually (using @command{cp}, or
@command{mv} for example).
-In the default configuration@footnote{If you configure Gnuastro with the
@option{--disable-shared} option, then the libraries will be statically linked
to the programs and this problem won't exist, see @ref{Linking}.}, the program
binaries need to link with Gnuastro's shared library which is also built and
installed with the programs.
-Therefore, to run successfully before and after installation, linking
modifications need to be made by GNU Libtool at installation time.
-@command{make install} does this internally, but a simple copy might give
linking errors when you run it.
-If you need to copy the executables, you can do so after installation.
+By default @file{PATH} usually contains system-wide directories, which are
readable (but not writable) by all users, like the above example.
+Therefore if you don't have root (or administrator) access, you need to add
another directory to @file{PATH} which you actually have write access to.
+The standard directory where you can keep installed files (not just
executables) for your own user is the @file{~/.local/} directory.
+The names of hidden files start with a `@key{.}' (dot), so it will not show up
in your common command-line listings, or on the graphical user interface.
+You can use any other directory, but this is the most recognized.
-@cindex Tests, error in converting images
-@item
-@command{$ make} (when bootstrapping): After you have bootstrapped Gnuastro
from the version-controlled source, you may confront the following (or a
similar) error when converting images (for more on bootstrapping, see
@ref{Bootstrapping}):
+The top installation directory will be used to keep all the package's
components: programs (executables), libraries, include (header) files, shared
data (like manuals), or configuration files (see @ref{Review of library
fundamentals} for a thorough introduction to headers and linking).
+So it commonly has some of the following sub-directories for each class of
installed components respectively: @file{bin/}, @file{lib/}, @file{include/}
@file{man/}, @file{share/}, @file{etc/}.
+Since the @file{PATH} variable is only used for executables, you can add the
@file{~/.local/bin} directory (which keeps the executables/programs or more
generally, ``binary'' files) to @file{PATH} with the following command.
+As defined below, first the existing value of @file{PATH} is used, then your
given directory is added to its end and the combined value is put back in
@file{PATH} (run `@command{$ echo $PATH}' afterwards to check if it was added).
@example
-@code{convert: attempt to perform an operation not allowed by the
-security policy `gs' @ error/delegate.c/ExternalDelegateCommand/378.}
+$ PATH=$PATH:~/.local/bin
@end example
-This error is a known
issue@footnote{@url{https://wiki.archlinux.org/title/ImageMagick}} with
@code{ImageMagick} security policies in some operating systems.
-In short, @code{imagemagick} uses Ghostscript for PDF, EPS, PS and XPS parsing.
-However, because some security vulnerabilities have been found in
Ghostscript@footnote{@url{https://security.archlinux.org/package/ghostscript}},
by default, ImageMagick may be compiled without Ghostscript library.
-In such cases, if allowed, ImageMagick will fall back to the external
@command{gs} command instead of the library.
-But this may be disabled with the following (or a similar) lines in
@code{/etc/ImageMagick-7/policy.xml} (anything related to PDF, PS, or
GhostScript).
+@cindex GNU Bash
+@cindex Startup scripts
+@cindex Scripts, startup
+Any executable that you installed in @file{~/.local/bin} will now be usable
without having to remember and write its full address.
+However, as soon as you leave/close your current terminal session, this
modified @file{PATH} variable will be forgotten.
+Adding the directories which contain executables to the @file{PATH}
environment variable each time you start a terminal is also very inconvenient
and prone to errors.
+Fortunately, there are standard `startup files' defined by your shell
precisely for this (and other) purposes.
+There is a special startup file for every significant starting step:
-@example
-<policy domain="delegate" rights="none" pattern="gs" />
-<policy domain="module" rights="none" pattern="@{PS,PDF,XPS@}" />
-@end example
+@table @asis
-To fix this problem, simply comment such lines (by placing a @code{<!--}
before each statement/line and @code{-->} at the end of that statement/line).
+@cindex GNU Bash
+@item @file{/etc/profile} and everything in @file{/etc/profile.d/}
+These startup scripts are called when your whole system starts (for example
after you turn on your computer).
+Therefore you need administrator or root privileges to access or modify them.
-@end itemize
+@item @file{~/.bash_profile}
+If you are using (GNU) Bash as your shell, the commands in this file are run,
when you log in to your account @emph{through Bash}.
+Most commonly when you login through the virtual console (where there is no
graphic user interface).
-@noindent
-If your problem was not listed above, please file a bug report (@ref{Report a
bug}).
+@item @file{~/.bashrc}
+If you are using (GNU) Bash as your shell, the commands here will be run each
time you start a terminal and are already logged in.
+For example, when you open your terminal emulator in the graphic user
interface.
+@end table
+For security reasons, it is highly recommended to directly type in your
@file{HOME} directory value by hand in startup files instead of using variables.
+So in the following, let's assume your user name is `@file{name}' (so @file{~}
may be replaced with @file{/home/name}).
+To add @file{~/.local/bin} to your @file{PATH} automatically on any startup
file, you have to ``export'' the new value of @command{PATH} in the startup
file that is most relevant to you by adding this line:
+@example
+export PATH=$PATH:/home/name/.local/bin
+@end example
+@cindex GNU build system
+@cindex Install directory
+@cindex Directory, install
+Now that you know your system will look into @file{~/.local/bin} for
executables, you can tell Gnuastro's configure script to install everything in
the top @file{~/.local} directory using the @option{--prefix} option.
+When you subsequently run @command{$ make install}, all the install-able files
will be put in their respective directory under @file{~/.local/} (the
executables in @file{~/.local/bin}, the compiled library files in
@file{~/.local/lib}, the library header files in @file{~/.local/include} and so
on, to learn more about these different files, please see @ref{Review of
library fundamentals}).
+Note that tilde (`@key{~}') expansion will not happen if you put a `@key{=}'
between @option{--prefix} and @file{~/.local}@footnote{If you insist on using
`@key{=}', you can use @option{--prefix=$HOME/.local}.}, so we have avoided the
@key{=} character here which is optional in GNU-style options, see
@ref{Options}.
+@example
+$ ./configure --prefix ~/.local
+@end example
+@cindex @file{MANPATH}
+@cindex @file{INFOPATH}
+@cindex @file{LD_LIBRARY_PATH}
+@cindex Library search directory
+@cindex Default library search directory
+You can install everything (including libraries like GSL, CFITSIO, or WCSLIB
which are Gnuastro's mandatory dependencies, see @ref{Mandatory dependencies})
locally by configuring them as above.
+However, recall that @command{PATH} is only for executable files, not
libraries and that libraries can also depend on other libraries.
+For example WCSLIB depends on CFITSIO and Gnuastro needs both.
+Therefore, when you installed a library in a non-recognized directory, you
have to guide the program that depends on them to look into the necessary
library and header file directories.
+To do that, you have to define the @command{LDFLAGS} and @command{CPPFLAGS}
environment variables respectively.
+This can be done while calling @file{./configure} as shown below:
+@example
+$ ./configure LDFLAGS=-L/home/name/.local/lib \
+ CPPFLAGS=-I/home/name/.local/include \
+ --prefix ~/.local
+@end example
+It can be annoying/buggy to do this when configuring every software that
depends on such libraries.
+Hence, you can define these two variables in the most relevant startup file
(discussed above).
+The convention on using these variables doesn't include a colon to separate
values (as @command{PATH}-like variables do), they use white space characters
and each value is prefixed with a compiler option@footnote{These variables are
ultimately used as options while building the programs, so every value has be
an option name followed be a value as discussed in @ref{Options}.}: note the
@option{-L} and @option{-I} above (see @ref{Options}), for @option{-I} see
@ref{Headers}, and for @optio [...]
+Therefore we have to keep the value in double quotation signs to keep the
white space characters and adding the following two lines to the startup file
of choice:
+@example
+export LDFLAGS="$LDFLAGS -L/home/name/.local/lib"
+export CPPFLAGS="$CPPFLAGS -I/home/name/.local/include"
+@end example
+@cindex Dynamic libraries
+Dynamic libraries are linked to the executable every time you run a program
that depends on them (see @ref{Linking} to fully understand this important
concept).
+Hence dynamic libraries also require a special path variable called
@command{LD_LIBRARY_PATH} (same formatting as @command{PATH}).
+To use programs that depend on these libraries, you need to add
@file{~/.local/lib} to your @command{LD_LIBRARY_PATH} environment variable by
adding the following line to the relevant start-up file:
+@example
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/name/.local/lib
+@end example
+If you also want to access the Info (see @ref{Info}) and man pages (see
@ref{Man pages}) documentations add @file{~/.local/share/info} and
@file{~/.local/share/man} to your @command{INFOPATH}@footnote{Info has the
following convention: ``If the value of @command{INFOPATH} ends with a colon
[or it isn't defined] ..., the initial list of directories is constructed by
appending the build-time default to the value of @command{INFOPATH}.'' So when
installing in a non-standard directory and if [...]
+@cindex Search directory order
+@cindex Order in search directory
+A final note is that order matters in the directories that are searched for
all the variables discussed above.
+In the examples above, the new directory was added after the system specified
directories.
+So if the program, library or manuals are found in the system wide
directories, the user directory is no longer searched.
+If you want to search your local installation first, put the new directory
before the already existing list, like the example below.
+@example
+export LD_LIBRARY_PATH=/home/name/.local/lib:$LD_LIBRARY_PATH
+@end example
+@noindent
+This is good when a library, for example CFITSIO, is already present on the
system, but the system-wide install wasn't configured with the correct
configuration flags (see @ref{CFITSIO}), or you want to use a newer version and
you don't have administrator or root access to update it on the whole
system/server.
+If you update @file{LD_LIBRARY_PATH} by placing @file{~/.local/lib} first
(like above), the linker will first find the CFITSIO you installed for yourself
and link with it.
+It thus will never reach the system-wide installation.
+There are important security problems with using local installations first:
all important system-wide executables and libraries (important executables like
@command{ls} and @command{cp}, or libraries like the C library) can be replaced
by non-secure versions with the same file names and put in the customized
directory (@file{~/.local} in this example).
+So if you choose to search in your customized directory first, please @emph{be
sure} to keep it clean from executables or libraries with the same names as
important system programs or libraries.
+@cartouche
+@noindent
+@strong{Summary:} When you are using a server which doesn't give you
administrator/root access AND you would like to give priority to your own built
programs and libraries, not the version that is (possibly already) present on
the server, add these lines to your startup file.
+See above for which startup file is best for your case and for a detailed
explanation on each.
+Don't forget to replace `@file{/YOUR-HOME-DIR}' with your home directory (for
example `@file{/home/your-id}'):
+@example
+export PATH="/YOUR-HOME-DIR/.local/bin:$PATH"
+export LDFLAGS="-L/YOUR-HOME-DIR/.local/lib $LDFLAGS"
+export MANPATH="/YOUR-HOME-DIR/.local/share/man/:$MANPATH"
+export CPPFLAGS="-I/YOUR-HOME-DIR/.local/include $CPPFLAGS"
+export INFOPATH="/YOUR-HOME-DIR/.local/share/info/:$INFOPATH"
+export LD_LIBRARY_PATH="/YOUR-HOME-DIR/.local/lib:$LD_LIBRARY_PATH"
+@end example
-@node Common program behavior, Data containers, Installation, Top
-@chapter Common program behavior
+@noindent
+Afterwards, you just need to add an extra
@option{--prefix=/YOUR-HOME-DIR/.local} to the @file{./configure} command of
the software that you intend to install.
+Everything else will be the same as a standard build and install, see
@ref{Quick start}.
+@end cartouche
-All the programs in Gnuastro share a set of common behavior mainly to do with
user interaction to facilitate their usage and development.
-This includes how to feed input datasets into the programs, how to configure
them, specifying the outputs, numerical data types, treating columns of
information in tables, etc.
-This chapter is devoted to describing this common behavior in all programs.
-Because the behaviors discussed here are common to several programs, they are
not repeated in each program's description.
+@node Executable names, Configure and build in RAM, Installation directory,
Configuring
+@subsubsection Executable names
-In @ref{Command-line}, a very general description of running the programs on
the command-line is discussed, like difference between arguments and options,
as well as options that are common/shared between all programs.
-None of Gnuastro's programs keep any internal configuration value (values for
their different operational steps), they read their configuration primarily
from the command-line, then from specific files in directory, user, or
system-wide settings.
-Using these configuration files can greatly help reproducible and robust usage
of Gnuastro, see @ref{Configuration files} for more.
+@cindex Executable names
+@cindex Names of executables
+At first sight, the names of the executables for each program might seem to be
uncommonly long, for example @command{astnoisechisel} or @command{astcrop}.
+We could have chosen terse (and cryptic) names like most programs do.
+We chose this complete naming convention (something like the commands in
@TeX{}) so you don't have to spend too much time remembering what the name of a
specific program was.
+Such complete names also enable you to easily search for the programs.
-It is not possible to always have the different options and configurations of
each program on the top of your head.
-It is very natural to forget the options of a program, their current default
values, or how it should be run and what it did.
-Gnuastro's programs have multiple ways to help you refresh your memory in
multiple levels (just an option name, a short description, or fast access to
the relevant section of the manual.
-See @ref{Getting help} for more for more on benefiting from this very
convenient feature.
+@cindex Shell auto-complete
+@cindex Auto-complete in the shell
+To facilitate typing the names in, we suggest using the shell auto-complete.
+With this facility you can find the executable you want very easily.
+It is very similar to file name completion in the shell.
+For example, simply by typing the letters below (where @key{[TAB]} stands for
the Tab key on your keyboard)
-Many of the programs use the multi-threaded character of modern CPUs, in
@ref{Multi-threaded operations} we'll discuss how you can configure this
behavior, along with some tips on making best use of them.
-In @ref{Numeric data types}, we'll review the various types to store numbers
in your datasets: setting the proper type for the usage context@footnote{For
example if the values in your dataset can only be integers between 0 or 65000,
store them in a unsigned 16-bit type, not 64-bit floating point type (which is
the default in most systems).
-It takes four times less space and is much faster to process.} can greatly
improve the file size and also speed of reading, writing or processing them.
+@example
+$ ast[TAB][TAB]
+@end example
-We'll then look into the recognized table formats in @ref{Tables} and how
large datasets are broken into tiles, or mesh grid in @ref{Tessellation}.
-Finally, we'll take a look at the behavior regarding output files:
@ref{Automatic output} describes how the programs set a default name for their
output when you don't give one explicitly (using @option{--output}).
-When the output is a FITS file, all the programs also store some very useful
information in the header that is discussed in @ref{Output FITS files}.
+@noindent
+you will get the list of all the available executables that start with
@command{ast} in your @command{PATH} environment variable directories.
+So, all the Gnuastro executables installed on your system will be listed.
+Typing the next letter for the specific program you want along with a Tab,
will limit this list until you get to your desired program.
-@menu
-* Command-line:: How to use the command-line.
-* Configuration files:: Values for unspecified variables.
-* Getting help:: Getting more information on the go.
-* Multi-threaded operations:: How threads are managed in Gnuastro.
-* Numeric data types:: Different types and how to specify them.
-* Memory management:: How memory is allocated (in RAM or HDD/SSD).
-* Tables:: Recognized table formats.
-* Tessellation:: Tile the dataset into non-overlapping bins.
-* Automatic output:: About automatic output names.
-* Output FITS files:: Common properties when outputs are FITS.
-@end menu
+@cindex Names, customize
+@cindex Customize executable names
+In case all of this does not convince you and you still want to type short
names, some suggestions are given below.
+You should have in mind though, that if you are writing a shell script that
you might want to pass on to others, it is best to use the standard name
because other users might not have adopted the same customization.
+The long names also serve as a form of documentation in such scripts.
+A similar reasoning can be given for option names in scripts: it is good
practice to always use the long formats of the options in shell scripts, see
@ref{Options}.
-@node Command-line, Configuration files, Common program behavior, Common
program behavior
-@section Command-line
+@cindex Symbolic link
+The simplest solution is making a symbolic link to the actual executable.
+For example let's assume you want to type @file{ic} to run Crop instead of
@file{astcrop}.
+Assuming you installed Gnuastro executables in @file{/usr/local/bin} (default)
you can do this simply by running the following command as root:
-Gnuastro's programs are customized through the standard Unix-like command-line
environment and GNU style command-line options.
-Both are very common in many Unix-like operating system programs.
-In @ref{Arguments and options} we'll start with the difference between
arguments and options and elaborate on the GNU style of options.
-Afterwards, in @ref{Common options}, we'll go into the detailed list of all
the options that are common to all the programs in Gnuastro.
+@example
+# ln -s /usr/local/bin/astcrop /usr/local/bin/ic
+@end example
-@menu
-* Arguments and options:: Different ways to specify inputs and
configuration.
-* Common options:: Options that are shared between all programs.
-* Shell TAB completion:: Customized TAB completion in Gnuastro.
-* Standard input:: Using output of another program as input.
-@end menu
+@noindent
+In case you update Gnuastro and a new version of Crop is installed, the
+default executable name is the same, so your custom symbolic link still
+works.
-@node Arguments and options, Common options, Command-line, Command-line
-@subsection Arguments and options
+@vindex --program-prefix
+@vindex --program-suffix
+@vindex --program-transform-name
+The installed executable names can also be set using options to @command{$
./configure}, see @ref{Configuring}.
+GNU Autoconf (which configures Gnuastro for your particular system), allows
the builder to change the name of programs with the three options
@option{--program-prefix}, @option{--program-suffix} and
@option{--program-transform-name}.
+The first two are for adding a fixed prefix or suffix to all the programs that
will be installed.
+This will actually make all the names longer! You can use it to add versions
of program names to the programs in order to simultaneously have two executable
versions of a program.
-@cindex Shell
-@cindex Options to programs
-@cindex Command-line options
-@cindex Arguments to programs
-@cindex Command-line arguments
-When you type a command on the command-line, it is passed onto the shell (a
generic name for the program that manages the command-line) as a string of
characters.
-As an example, see the ``Invoking ProgramName'' sections in this manual for
some examples of commands with each program, like @ref{Invoking asttable},
@ref{Invoking astfits}, or @ref{Invoking aststatistics}.
+@cindex SED, stream editor
+@cindex Stream editor, SED
+The third configure option allows you to set the executable name at install
time using the SED program.
+SED is a very useful `stream editor'.
+There are various resources on the internet to use it effectively.
+However, we should caution that using configure options will change the actual
executable name of the installed program and on every re-install (an update for
example), you have to also add this option to keep the old executable name
updated.
+Also note that the documentation or configuration files do not change from
their standard names either.
-The shell then brakes up your string into separate @emph{tokens} or
@emph{words} using any @emph{metacharacters} (like white-space, tab,
@command{|}, @command{>} or @command{;}) that are in the string.
-On the command-line, the first thing you usually enter is the name of the
program you want to run.
-After that, you can specify two types of tokens: @emph{arguments} and
@emph{options}.
-In the GNU-style, arguments are those tokens that are not preceded by any
hyphens (@command{-}, see @ref{Arguments}).
-Here is one example:
+@cindex Removing @file{ast} from executables
+For example, let's assume that typing @file{ast} on every invocation of every
program is really annoying you! You can remove this prefix from all the
executables at configure time by adding this option:
@example
-$ astcrop --center=53.162551,-27.789676 -w10/3600 --mode=wcs udf.fits
+$ ./configure --program-transform-name='s/ast/ /'
@end example
-In the example above, we are running @ref{Crop} to crop a region of width 10
arc-seconds centered at the given RA and Dec from the input Hubble Ultra-Deep
Field (UDF) FITS image.
-Here, the argument is @file{udf.fits}.
-Arguments are most commonly the input file names containing your data.
-Options start with one or two hyphens, followed by an identifier for the
option (the option's name, for example, @option{--center}, @option{-w},
@option{--mode} in the example above) and its value (anything after the option
name, or the optional @key{=} character).
-Through options you can configure how the program runs (interprets the data
you provided).
-
-@vindex --help
-@vindex --usage
-@cindex Mandatory arguments
-Arguments can be mandatory and optional and unlike options, they don't have
any identifiers.
-Hence, when there multiple arguments, their order might also matter (for
example in @command{cp} which is used for copying one file to another location).
-The outputs of @option{--usage} and @option{--help} shows which arguments are
optional and which are mandatory, see @ref{--usage}.
-As their name suggests, @emph{options} can be considered to be optional and
most of the time, you don't have to worry about what order you specify them in.
-When the order does matter, or the option can be invoked multiple times, it is
explicitly mentioned in the ``Invoking ProgramName'' section of each program
(this is a very important aspect of an option).
-@cindex Metacharacters on the command-line In case your arguments or option
values contain any of the shell's meta-characters, you have to quote them.
-If there is only one such character, you can use a backslash (@command{\})
before it.
-If there are multiple, it might be easier to simply put your whole argument or
option value inside of double quotes (@command{"}).
-In such cases, everything inside the double quotes will be seen as one token
or word.
+@node Configure and build in RAM, , Executable names, Configuring
+@subsubsection Configure and build in RAM
-@cindex HDU
-@cindex Header data unit
-For example, let's say you want to specify the header data unit (HDU) of your
FITS file using a complex expression like `@command{3; images(exposure > 100)}'.
-If you simply add these after the @option{--hdu} (@option{-h}) option, the
programs in Gnuastro will read the value to the HDU option as `@command{3}' and
run.
-Then, the shell will attempt to run a separate command
`@command{images(exposure > 100)}' and complain about a syntax error.
-This is because the semicolon (@command{;}) is an `end of command' character
in the shell.
-To solve this problem you can simply put double quotes around the whole string
you want to pass to @option{--hdu} as seen below:
-@example
-$ astcrop --hdu="3; images(exposure > 100)" image.fits
-@end example
+@cindex File I/O
+@cindex Input/Output, file
+Gnuastro's configure and build process (the GNU build system) involves the
creation, reading, and modification of a large number of files (input/output,
or I/O).
+Therefore file I/O issues can directly affect the work of developers who need
to configure and build Gnuastro numerous times.
+Some of these issues are listed below:
+@itemize
+@cindex HDD
+@cindex SSD
+@item
+I/O will cause wear and tear on both the HDDs (mechanical failures) and
+SSDs (decreasing the lifetime).
+@cindex Backup
+@item
+Having the built files mixed with the source files can greatly affect backing
up (synchronization) of source files (since it involves the management of a
large number of small files that are regularly changed.
+Backup software can of course be configured to ignore the built files and
directories.
+However, since the built files are mixed with the source files and can have a
large variety, this will require a high level of customization.
+@end itemize
+@cindex tmpfs file system
+@cindex file systems, tmpfs
+One solution to address both these problems is to use the
@url{https://en.wikipedia.org/wiki/Tmpfs, tmpfs file system}.
+Any file in tmpfs is actually stored in the RAM (and possibly SWAP), not on
HDDs or SSDs.
+The RAM is built for extensive and fast I/O.
+Therefore the large number of file I/Os associated with configuring and
building will not harm the HDDs or SSDs.
+Due to the volatile nature of RAM, files in the tmpfs file-system will be
permanently lost after a power-off.
+Since all configured and built files are derivative files (not files that have
been directly written by hand) there is no problem in this and this feature can
be considered as an automatic cleanup.
+@cindex Linux kernel
+@cindex GNU C library
+@cindex GNU build system
+The modern GNU C library (and thus the Linux kernel) defines the
@file{/dev/shm} directory for this purpose in the RAM (POSIX shared memory).
+To build in it, you can use the GNU build system's ability to build in a
separate directory (not necessarily in the source directory) as shown below.
+Just set @file{SRCDIR} as the address of Gnuastro's top source directory (for
example, the unpacked tarball).
-@menu
-* Arguments:: For specifying the main input files/operations.
-* Options:: For configuring the behavior of the program.
-@end menu
+@example
+$ mkdir /dev/shm/tmp-gnuastro-build
+$ cd /dev/shm/tmp-gnuastro-build
+$ SRCDIR/configure --srcdir=SRCDIR
+$ make
+@end example
-@node Arguments, Options, Arguments and options, Arguments and options
-@subsubsection Arguments
-In Gnuastro, arguments are almost exclusively used as the input data file
names.
-Please consult the first few paragraph of the ``Invoking ProgramName'' section
for each program for a description of what it expects as input, how many
arguments, or input data, it accepts, or in what order.
-Everything particular about how a program treats arguments, is explained under
the ``Invoking ProgramName'' section for that program.
+Gnuastro comes with a script to simplify this process of configuring and
building in a different directory (a ``clean'' build), for more see
@ref{Separate build and source directories}.
-@cindex Filename suffix
-@cindex Suffix (filename)
-@cindex FITS filename suffixes
-Generally, if there is a standard file name suffix for a particular format,
that filename extension is checked to identify their format.
-In astronomy (and thus Gnuastro), FITS is the preferred format for inputs and
outputs, so the focus here and throughout this book is on FITS.
-However, other formats are also accepted in special cases, for example
@ref{ConvertType} also accepts JPEG or TIFF inputs, and writes JPEG, EPS or PDF
files.
-The recognized suffixes for these formats are listed there.
+@node Separate build and source directories, Tests, Configuring, Build and
install
+@subsection Separate build and source directories
-The list below shows the recognized suffixes for FITS data files in Gnuastro's
programs.
-However, in some scenarios FITS writers may not append a suffix to the file,
or use a non-recognized suffix (not in the list below).
-Therefore if a FITS file is expected, but it doesn't have any of these
suffixes, Gnuastro programs will look into the contents of the file and if it
does conform with the FITS standard, the file will be used.
-Just note that checking about 5 characters at the end of a name string is much
more efficient than opening and checking the contents of a file, so it is
generally recommended to have a recognized FITS suffix.
+The simple steps of @ref{Quick start} will mix the source and built files.
+This can cause inconvenience for developers or enthusiasts following the the
most recent work (see @ref{Version controlled source}).
+The current section is mainly focused on this later group of Gnuastro users.
+If you just install Gnuastro on major releases (following
@ref{Announcements}), you can safely ignore this section.
-@itemize
+@cindex GNU build system
+When it is necessary to keep the source (which is under version control), but
not the derivative (built) files (after checking or installing), the best
solution is to keep the source and the built files in separate directories.
+One application of this is already discussed in @ref{Configure and build in
RAM}.
-@item
-@file{.fits}: The standard file name ending of a FITS image.
+To facilitate this process of configuring and building in a separate
directory, Gnuastro comes with the @file{developer-build} script.
+It is available in the top source directory and is @emph{not} installed.
+It will make a directory under a given top-level directory (given to
@option{--top-build-dir}) and build Gnuastro in there directory.
+It thus keeps the source completely separated from the built files.
+For easy access to the built files, it also makes a symbolic link to the built
directory in the top source files called @file{build}.
-@item
-@file{.fit}: Alternative (3 character) FITS suffix.
+When run without any options, default values will be used for its
configuration.
+As with Gnuastro's programs, you can inspect the default values with
@option{-P} (or @option{--printparams}, the output just looks a little
different here).
+The default top-level build directory is @file{/dev/shm}: the shared memory
directory in RAM on GNU/Linux systems as described in @ref{Configure and build
in RAM}.
-@item
-@file{.fits.Z}: A FITS image compressed with @command{compress}.
+@cindex Debug
+Besides these, it also has some features to facilitate the job of developers
or bleeding edge users like the @option{--debug} option to do a fast build,
with debug information, no optimization, and no shared libraries.
+Here is the full list of options you can feed to this script to configure its
operations.
-@item
-@file{.fits.gz}: A FITS image compressed with GNU zip (gzip).
+@cartouche
+@noindent
+@strong{Not all Gnuastro's common program behavior usable here:}
+@file{developer-build} is just a non-installed script with a very limited
scope as described above.
+It thus doesn't have all the common option behaviors or configuration files
for example.
+@end cartouche
-@item
-@file{.fits.fz}: A FITS image compressed with @command{fpack}.
+@cartouche
+@noindent
+@strong{White space between option and value:} @file{developer-build}
+doesn't accept an @key{=} sign between the options and their values.
+It also needs at least one character between the option and its value.
+Therefore @option{-n 4} or @option{--numthreads 4} are acceptable, while
@option{-n4}, @option{-n=4}, or @option{--numthreads=4} aren't.
+Finally multiple short option names cannot be merged: for example you can say
@option{-c -n 4}, but unlike Gnuastro's programs, @option{-cn4} is not
acceptable.
+@end cartouche
-@item
-@file{.imh}: IRAF format image file.
+@cartouche
+@noindent
+@strong{Reusable for other packages:} This script can be used in any software
which is configured and built using the GNU Build System.
+Just copy it in the top source directory of that software and run it from
there.
+@end cartouche
-@end itemize
+@table @option
-Through out this book and in the command-line outputs, whenever we want to
generalize all such astronomical data formats in a text place holder, we will
use @file{ASTRdata}, we will assume that the extension is also part of this
name.
-Any file ending with these names is directly passed on to CFITSIO to read.
-Therefore you don't necessarily have to have these files on your computer,
they can also be located on an FTP or HTTP server too, see the CFITSIO manual
for more information.
+@item -b STR
+@itemx --top-build-dir STR
+The top build directory to make a directory for the build.
+If this option isn't called, the top build directory is @file{/dev/shm} (only
available in GNU/Linux operating systems, see @ref{Configure and build in RAM}).
-CFITSIO has its own error reporting techniques, if your input file(s)
-cannot be opened, or read, those errors will be printed prior to the
-final error by Gnuastro.
+@item -V
+@itemx --version
+Print the version string of Gnuastro that will be used in the build.
+This string will be appended to the directory name containing the built files.
+@item -a
+@itemx --autoreconf
+Run @command{autoreconf -f} before building the package.
+In Gnuastro, this is necessary when a new commit has been made to the project
history.
+In Gnuastro's build system, the Git description will be used as the version,
see @ref{Version numbering} and @ref{Synchronizing}.
+@item -c
+@itemx --clean
+@cindex GNU Autoreconf
+Delete the contents of the build directory (clean it) before starting the
configuration and building of this run.
-@node Options, , Arguments, Arguments and options
-@subsubsection Options
+This is useful when you have recently pulled changes from the main Git
repository, or committed a change your self and ran @command{autoreconf -f},
see @ref{Synchronizing}.
+After running GNU Autoconf, the version will be updated and you need to do a
clean build.
-@cindex GNU style options
-@cindex Options, GNU style
-@cindex Options, short (@option{-}) and long (@option{--})
-Command-line options allow configuring the behavior of a program in all
GNU/Linux applications for each particular execution on a particular input data.
-A single option can be called in two ways: @emph{long} or @emph{short}.
-All options in Gnuastro accept the long format which has two hyphens an can
have many characters (for example @option{--hdu}).
-Short options only have one hyphen (@key{-}) followed by one character (for
example @option{-h}).
-You can see some examples in the list of options in @ref{Common options} or
those for each program's ``Invoking ProgramName'' section.
-Both formats are shown for those which support both.
-First the short is shown then the long.
+@item -d
+@itemx --debug
+@cindex Valgrind
+@cindex GNU Debugger (GDB)
+Build with debugging flags (for example to use in GNU Debugger, also known as
GDB, or Valgrind), disable optimization and also the building of shared
libraries.
+Similar to running the configure script of below
-Usually, the short options are for when you are writing on the command-line
and want to save keystrokes and time.
-The long options are good for shell scripts, where you aren't usually rushing.
-Long options provide a level of documentation, since they are more descriptive
and less cryptic.
-Usually after a few months of not running a program, the short options will be
forgotten and reading your previously written script will not be easy.
+@example
+$ ./configure --enable-debug
+@end example
-@cindex On/Off options
-@cindex Options, on/off
-Some options need to be given a value if they are called and some don't.
-You can think of the latter type of options as on/off options.
-These two types of options can be distinguished using the output of the
@option{--help} and @option{--usage} options, which are common to all GNU
software, see @ref{Getting help}.
-In Gnuastro we use the following strings to specify when the option needs a
value and what format that value should be in.
-More specific tests will be done in the program and if the values are out of
range (for example negative when the program only wants a positive value), an
error will be reported.
+Besides all the debugging advantages of building with this option, it will
also be significantly speed up the build (at the cost of slower built programs).
+So when you are testing something small or working on the build system itself,
it will be much faster to test your work with this option.
-@vtable @option
+@item -v
+@itemx --valgrind
+@cindex Valgrind
+Build all @command{make check} tests within Valgrind.
+For more, see the description of @option{--enable-check-with-valgrind} in
@ref{Gnuastro configure options}.
-@item INT
-The value is read as an integer.
+@item -j INT
+@itemx --jobs INT
+The maximum number of threads/jobs for Make to build at any moment.
+As the name suggests (Make has an identical option), the number given to this
option is directly passed on to any call of Make with its @option{-j} option.
-@item FLT
-The value is read as a float.
-There are generally two types, depending on the context.
-If they are for fractions, they will have to be less than or equal to unity.
+@item -C
+@itemx --check
+After finishing the build, also run @command{make check}.
+By default, @command{make check} isn't run because the developer usually has
their own checks to work on (for example defined in @file{tests/during-dev.sh}).
-@item STR
-The value is read as a string of characters.
-For example column names in a table, or HDU names in a multi-extension FITS
file.
-Other examples include human-readable settings by some programs like the
@option{--domain} option of the Convolve program that can be either
@code{spatial} or @code{frequency} (to specify the type of convolution, see
@ref{Convolve}).
+@item -i
+@itemx --install
+After finishing the build, also run @command{make install}.
-@item FITS @r{or} FITS/TXT
-The value should be a file (most commonly FITS).
-In many cases, other formats may also be accepted (for example input tables
can be FITS or plain-text, see @ref{Recognized table formats}).
+@item -D
+@itemx --dist
+Run @code{make dist-lzip pdf} to build a distribution tarball (in
@file{.tar.lz} format) and a PDF manual.
+This can be useful for archiving, or sending to colleagues who don't use Git
for an easy build and manual.
-@end vtable
+@item -u STR
+@item --upload STR
+Activate the @option{--dist} (@option{-D}) option, then use secure copy
(@command{scp}, part of the SSH tools) to copy the tarball and PDF to the
@file{src} and @file{pdf} sub-directories of the specified server and its
directory (value to this option).
+For example @command{--upload my-server:dir}, will copy the tarball in the
@file{dir/src}, and the PDF manual in @file{dir/pdf} of @code{my-server} server.
+It will then make a symbolic link in the top server directory to the tarball
that is called @file{gnuastro-latest.tar.lz}.
-@noindent
-@cindex Values to options
-@cindex Option values
-To specify a value in the short format, simply put the value after the option.
-Note that since the short options are only one character long, you don't have
to type anything between the option and its value.
-For the long option you either need white space or an @option{=} sign, for
example @option{-h2}, @option{-h 2}, @option{--hdu 2} or @option{--hdu=2} are
all equivalent.
+@item -p
+@itemx --publish
+Short for @option{--autoreconf --clean --debug --check --upload STR}.
+@option{--debug} is added because it will greatly speed up the build.
+It will have no effect on the produced tarball.
+This is good when you have made a commit and are ready to publish it on your
server (if nothing crashes).
+Recall that if any of the previous steps fail the script aborts.
-The short format of on/off options (those that don't need values) can be
concatenated for example these two hypothetical sequences of options are
equivalent: @option{-a -b -c4} and @option{-abc4}.
-As an example, consider the following command to run Crop:
-@example
-$ astcrop -Dr3 --wwidth 3 catalog.txt --deccol=4 ASTRdata
-@end example
-@noindent
-The @command{$} is the shell prompt, @command{astcrop} is the program name.
-There are two arguments (@command{catalog.txt} and @command{ASTRdata}) and
four options, two of them given in short format (@option{-D}, @option{-r}) and
two in long format (@option{--width} and @option{--deccol}).
-Three of them require a value and one (@option{-D}) is an on/off option.
+@item -I
+@item --install-archive
+Short for @option{--autoreconf --clean --check --install --dist}.
+This is useful when you actually want to install the commit you just made (if
the build and checks succeed).
+It will also produce a distribution tarball and PDF manual for easy access to
the installed tarball on your system at a later time.
-@vindex --printparams
-@cindex Options, abbreviation
-@cindex Long option abbreviation
-If an abbreviation is unique between all the options of a program, the long
option names can be abbreviated.
-For example, instead of typing @option{--printparams}, typing @option{--print}
or maybe even @option{--pri} will be enough, if there are conflicts, the
program will warn you and show you the alternatives.
-Finally, if you want the argument parser to stop parsing arguments beyond a
certain point, you can use two dashes: @option{--}.
-No text on the command-line beyond these two dashes will be parsed.
+Ideally, Gnuastro's Git version history makes it easy for a prepared system to
revert back to a different point in history.
+But Gnuastro also needs to bootstrap files and also your collaborators might
(usually do!) find it too much of a burden to do the bootstrapping themselves.
+So it is convenient to have a tarball and PDF manual of the version you have
installed (and are using in your research) handily available.
-@cindex Repeated options
-@cindex Options, repeated
-Gnuastro has two types of options with values, those that only take a single
value are the most common type.
-If these options are repeated or called more than once on the command-line,
the value of the last time it was called will be assigned to it.
-This is very useful when you are testing/experimenting.
-Let's say you want to make a small modification to one option value.
-You can simply type the option with a new value in the end of the command and
see how the script works.
-If you are satisfied with the change, you can remove the original option for
human readability.
-If the change wasn't satisfactory, you can remove the one you just added and
not worry about forgetting the original value.
-Without this capability, you would have to memorize or save the original value
somewhere else, run the command and then change the value again which is not at
all convenient and is potentially cause lots of bugs.
+@item -h
+@itemx --help
+@itemx -P
+@itemx --printparams
+Print a description of this script along with all the options and their
+current values.
-On the other hand, some options can be called multiple times in one run of a
program and can thus take multiple values (for example see the
@option{--column} option in @ref{Invoking asttable}.
-In these cases, the order of stored values is the same order that you
specified on the command-line.
+@end table
-@cindex Configuration files
-@cindex Default option values
-Gnuastro's programs don't keep any internal default values, so some options
are mandatory and if they don't have a value, the program will complain and
abort.
-Most programs have many such options and typing them by hand on every call is
impractical.
-To facilitate the user experience, after parsing the command-line, Gnuastro's
programs read special configuration files to get the necessary values for the
options you haven't identified on the command-line.
-These configuration files are fully described in @ref{Configuration files}.
-@cartouche
-@noindent
-@cindex Tilde expansion as option values
-@strong{CAUTION:} In specifying a file address, if you want to use the shell's
tilde expansion (@command{~}) to specify your home directory, leave at least
one space between the option name and your value.
-For example use @command{-o ~/test}, @command{--output ~/test} or
@command{--output= ~/test}.
-Calling them with @command{-o~/test} or @command{--output=~/test} will disable
shell expansion.
-@end cartouche
-@cartouche
-@noindent
-@strong{CAUTION:} If you forget to specify a value for an option which
requires one, and that option is the last one, Gnuastro will warn you.
-But if it is in the middle of the command, it will take the text of the next
option or argument as the value which can cause undefined behavior.
-@end cartouche
-@cartouche
-@noindent
-@cindex Counting from zero.
-@strong{NOTE:} In some contexts Gnuastro's counting starts from 0 and in
others 1.
-You can assume by default that counting starts from 1, if it starts from 0 for
a special option, it will be explicitly mentioned.
-@end cartouche
-@node Common options, Shell TAB completion, Arguments and options, Command-line
-@subsection Common options
+@node Tests, A4 print book, Separate build and source directories, Build and
install
+@subsection Tests
-@cindex Options common to all programs
-@cindex Gnuastro common options
-To facilitate the job of the users and developers, all the programs in
Gnuastro share some basic command-line options for the options that are common
to many of the programs.
-The full list is classified as @ref{Input output options}, @ref{Processing
options}, and @ref{Operating mode options}.
-In some programs, some of the options are irrelevant, but still recognized
(you won't get an unrecognized option error, but the value isn't used).
-Unless otherwise mentioned, these options are identical between all programs.
+@cindex @command{make check}
+@cindex @file{mock.fits}
+@cindex Tests, running
+@cindex Checking tests
+After successfully building (compiling) the programs with the @command{$ make}
command you can check the installation before installing.
+To run the tests, run
-@menu
-* Input output options:: Common input/output options.
-* Processing options:: Options for common processing steps.
-* Operating mode options:: Common operating mode options.
-@end menu
+@example
+$ make check
+@end example
-@node Input output options, Processing options, Common options, Common options
-@subsubsection Input/Output options
+For every program some tests are designed to check some possible operations.
+Running the command above will run those tests and give you a final report.
+If everything is OK and you have built all the programs, all the tests should
pass.
+In case any of the tests fail, please have a look at @ref{Known issues} and if
that still doesn't fix your problem, look that the
@file{./tests/test-suite.log} file to see if the source of the error is
something particular to your system or more general.
+If you feel it is general, please contact us because it might be a bug.
+Note that the tests of some programs depend on the outputs of other program's
tests, so if you have not installed them they might be skipped or fail.
+Prior to releasing every distribution all these tests are checked.
+If you have a reasonably modern terminal, the outputs of the successful tests
will be colored green and the failed ones will be colored red.
-These options are to do with the input and outputs of the various
-programs.
+These scripts can also act as a good set of examples for you to see how the
programs are run.
+All the tests are in the @file{tests/} directory.
+The tests for each program are shell scripts (ending with @file{.sh}) in a
sub-directory of this directory with the same name as the program.
+See @ref{Test scripts} for more detailed information about these scripts in
case you want to inspect them.
-@vtable @option
-@cindex Timeout
-@cindex Standard input
-@item --stdintimeout
-Number of micro-seconds to wait for writing/typing in the @emph{first line} of
standard input from the command-line (see @ref{Standard input}).
-This is only relevant for programs that also accept input from the standard
input, @emph{and} you want to manually write/type the contents on the terminal.
-When the standard input is already connected to a pipe (output of another
program), there won't be any waiting (hence no timeout, thus making this option
redundant).
-If the first line-break (for example with the @key{ENTER} key) is not provided
before the timeout, the program will abort with an error that no input was
given.
-Note that this time interval is @emph{only} for the first line that you type.
-Once the first line is given, the program will assume that more data will come
and accept rest of your inputs without any time limit.
-You need to specify the ending of the standard input, for example by pressing
@key{CTRL-D} after a new line.
-Note that any input you write/type into a program on the command-line with
Standard input will be discarded (lost) once the program is finished.
-It is only recoverable manually from your command-line (where you actually
typed) as long as the terminal is open.
-So only use this feature when you are sure that you don't need the dataset (or
have a copy of it somewhere else).
+@node A4 print book, Known issues, Tests, Build and install
+@subsection A4 print book
+@cindex A4 print book
+@cindex Modifying print book
+@cindex A4 paper size
+@cindex US letter paper size
+@cindex Paper size, A4
+@cindex Paper size, US letter
+The default print version of this book is provided in the letter paper size.
+If you would like to have the print version of this book on paper and you are
living in a country which uses A4, then you can rebuild the book.
+The great thing about the GNU build system is that the book source code which
is in Texinfo is also distributed with the program source code, enabling you to
do such customization (hacking).
-@cindex HDU
-@cindex Header data unit
-@item -h STR/INT
-@itemx --hdu=STR/INT
-The name or number of the desired Header Data Unit, or HDU, in the FITS image.
-A FITS file can store multiple HDUs or extensions, each with either an image
or a table or nothing at all (only a header).
-Note that counting of the extensions starts from 0(zero), not 1(one).
-Counting from 0 is forced on us by CFITSIO which directly reads the value you
give with this option (see @ref{CFITSIO}).
-When specifying the name, case is not important so @command{IMAGE},
@command{image} or @command{ImAgE} are equivalent.
+@cindex GNU Texinfo
+In order to change the paper size, you will need to have GNU Texinfo installed.
+Open @file{doc/gnuastro.texi} with any text editor.
+This is the source file that created this book.
+In the first few lines you will see this line:
-CFITSIO has many capabilities to help you find the extension you want, far
beyond the simple extension number and name.
-See CFITSIO manual's ``HDU Location Specification'' section for a very
complete explanation with several examples.
-A @code{#} is appended to the string you specify for the HDU@footnote{With the
@code{#} character, CFITSIO will only read the desired HDU into your memory,
not all the existing HDUs in the fits file.} and the result is put in square
brackets and appended to the FITS file name before calling CFITSIO to read the
contents of the HDU for all the programs in Gnuastro.
+@example
+@@c@@afourpaper
+@end example
-@item -s STR
-@itemx --searchin=STR
-Where to match/search for columns when the column identifier wasn't a number,
see @ref{Selecting table columns}.
-The acceptable values are @command{name}, @command{unit}, or @command{comment}.
-This option is only relevant for programs that take table columns as input.
+@noindent
+In Texinfo, a line is commented with @code{@@c}.
+Therefore, un-comment this line by deleting the first two characters such that
it changes to:
-@item -I
-@itemx --ignorecase
-Ignore case while matching/searching column meta-data (in the field specified
by the @option{--searchin}).
-The FITS standard suggests to treat the column names as case insensitive,
which is strongly recommended here also but is not enforced.
-This option is only relevant for programs that take table columns as input.
+@example
+@@afourpaper
+@end example
-This option is not relevant to @ref{BuildProgram}, hence in that program the
short option @option{-I} is used for include directories, not to ignore case.
+@noindent
+Save the file and close it.
+You can now run the following command
-@item -o STR
-@itemx --output=STR
-The name of the output file or directory. With this option the automatic
output names explained in @ref{Automatic output} are ignored.
+@example
+$ make pdf
+@end example
-@item -T STR
-@itemx --type=STR
-The data type of the output depending on the program context.
-This option isn't applicable to some programs like @ref{Fits} and will be
ignored by them.
-The different acceptable values to this option are fully described in
@ref{Numeric data types}.
+@noindent
+and the new PDF book will be available in @file{SRCdir/doc/gnuastro.pdf}.
+By changing the @command{pdf} in @command{$ make pdf} to @command{ps} or
@command{dvi} you can have the book in those formats.
+Note that you can do this for any book that is in Texinfo format, they might
not have @code{@@afourpaper} line, so you can add it close to the top of the
Texinfo source file.
-@item -D
-@itemx --dontdelete
-By default, if the output file already exists, Gnuastro's programs will
silently delete it and put their own outputs in its place.
-When this option is activated, if the output file already exists, the programs
will not delete it, will warn you, and will abort.
-@item -K
-@itemx --keepinputdir
-In automatic output names, don't remove the directory information of the input
file names.
-As explained in @ref{Automatic output}, if no output name is specified (with
@option{--output}), then the output name will be made in the existing directory
based on your input's file name (ignoring the directory of the input).
-If you call this option, the directory information of the input will be kept
and the automatically generated output name will be in the same directory as
the input (usually with a suffix added).
-Note that his is only relevant if you are running the program in a different
directory than the input data.
-@item -t STR
-@itemx --tableformat=STR
-The output table's type.
-This option is only relevant when the output is a table and its format cannot
be deduced from its filename.
-For example, if a name ending in @file{.fits} was given to @option{--output},
then the program knows you want a FITS table.
-But there are two types of FITS tables: FITS ASCII, and FITS binary.
-Thus, with this option, the program is able to identify which type you want.
-The currently recognized values to this option are:
-@item --wcslinearmatrix=STR
-Select the linear transformation matrix of the output's WCS.
-This option only takes two values: @code{pc} (for the @code{PCi_j} formalism)
and @code{cd} (for @code{CDi_j}).
-For more on the different formalisms, please see Section 8.1 of the FITS
standard@footnote{@url{https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf}},
version 4.0.
+@node Known issues, , A4 print book, Build and install
+@subsection Known issues
-@cindex @code{CDELT}
-In short, in the @code{PCi_j} formalism, we only keep the linear rotation
matrix in these keywords and put the scaling factor (or the pixel scale in
astronomical imaging) in the @code{CDELTi} keywords.
-In the @code{CDi_j} formalism, we blend the scaling into the rotation into a
single matrix and keep that matrix in these FITS keywords.
-By default, Gnuastro uses the @code{PCi_j} formalism, because it greatly helps
in human readability of the raw keywords and is also the default mode of WCSLIB.
-However, in some circumstances it may be necessary to have the keywords in the
CD format; for example when you need to feed the outputs into other software
that don't follow the full FITS standard and only recognize the @code{CDi_j}
formalism.
+Depending on your operating system and the version of the compiler you are
using, you might confront some known problems during the configuration
(@command{$ ./configure}), compilation (@command{$ make}) and tests (@command{$
make check}).
+Here, their solutions are discussed.
-@table @command
-@item txt
-A plain text table with white-space characters between the columns (see
-@ref{Gnuastro text table format}).
-@item fits-ascii
-A FITS ASCII table (see @ref{Recognized table formats}).
-@item fits-binary
-A FITS binary table (see @ref{Recognized table formats}).
-@end table
+@itemize
+@cindex Configuration, not finding library
+@cindex Development packages
+@item
+@command{$ ./configure}: @emph{Configure complains about not finding a library
even though you have installed it.}
+The possible solution is based on how you installed the package:
-@end vtable
+@itemize
+@item
+From your distribution's package manager.
+Most probably this is because your distribution has separated the header files
of a library from the library parts.
+Please also install the `development' packages for those libraries too.
+Just add a @file{-dev} or @file{-devel} to the end of the package name and
re-run the package manager.
+This will not happen if you install the libraries from source.
+When installed from source, the headers are also installed.
+@item
+@cindex @command{LDFLAGS}
+From source.
+Then your linker is not looking where you installed the library.
+If you followed the instructions in this chapter, all the libraries will be
installed in @file{/usr/local/lib}.
+So you have to tell your linker to look in this directory.
+To do so, configure Gnuastro like this:
-@node Processing options, Operating mode options, Input output options, Common
options
-@subsubsection Processing options
+@example
+$ ./configure LDFLAGS="-L/usr/local/lib"
+@end example
-Some processing steps are common to several programs, so they are defined as
common options to all programs.
-Note that this class of common options is thus necessarily less common between
all the programs than those described in @ref{Input output options}, or
@ref{Operating mode options} options.
-Also, if they are irrelevant for a program, these options will not display in
the @option{--help} output of the program.
+If you want to use the libraries for your other programming projects, then
+export this environment variable in a start-up script similar to the case
+for @file{LD_LIBRARY_PATH} explained below, also see @ref{Installation
+directory}.
+@end itemize
-@table @option
+@item
+@vindex --enable-gnulibcheck
+@cindex Gnulib: GNU Portability Library
+@cindex GNU Portability Library (Gnulib)
+@command{$ make}: @emph{Complains about an unknown function on a non-GNU based
operating system.}
+In this case, please run @command{$ ./configure} with the
@option{--enable-gnulibcheck} option to see if the problem is from the GNU
Portability Library (Gnulib) not supporting your system or if there is a
problem in Gnuastro, see @ref{Gnuastro configure options}.
+If the problem is not in Gnulib and after all its tests you get the same
complaint from @command{make}, then please contact us at
@file{bug-gnuastro@@gnu.org}.
+The cause is probably that a function that we have used is not supported by
your operating system and we didn't included it along with the source tar ball.
+If the function is available in Gnulib, it can be fixed immediately.
-@item --minmapsize=INT
-The minimum size (in bytes) to memory-map a processing/internal array as a
file (on the non-volatile HDD/SSD), and not use the system's RAM.
-Before using this option, please read @ref{Memory management}.
-By default processing arrays will only be memory-mapped to a file when the RAM
is full.
-With this option, you can force the memory-mapping, even when there is enough
RAM.
-To ensure this default behavior, the pre-defined value to this option is an
extremely large value (larger than any existing RAM).
+@item
+@cindex @command{CPPFLAGS}
+@command{$ make}: @emph{Can't find the headers (.h files) of installed
libraries.}
+Your C pre-processor (CPP) isn't looking in the right place.
+To fix this, configure Gnuastro with an additional @code{CPPFLAGS} like below
(assuming the library is installed in @file{/usr/local/include}:
-Please note that using a non-volatile file (in the HDD/SDD) instead of RAM can
significantly increase the program's running time, especially on HDDs (where
read/write is slower).
-Also, note that the number of memory-mapped files that your kernel can support
is limited.
-So when this option is necessary, it is best to give it values larger than 1
megabyte (@option{--minmapsize=1000000}).
-You can then decrease it for a specific program's invocation on a large input
after you see memory issues arise (for example an error, or the program not
aborting and fully consuming your memory).
-If you see randomly named files remaining in this directory when the program
finishes normally, please send us a bug report so we address the problem, see
@ref{Report a bug}.
+@example
+$ ./configure CPPFLAGS="-I/usr/local/include"
+@end example
-@cartouche
-@noindent
-@strong{Limited number of memory-mapped files:} The operating system kernels
usually support a limited number of memory-mapped files.
-Therefore never set @code{--minmapsize} to zero or a small number of bytes (so
too many files are created).
-If the kernel capacity is exceeded, the program will crash.
-@end cartouche
+If you want to use the libraries for your other programming projects, then
export this environment variable in a start-up script similar to the case for
@file{LD_LIBRARY_PATH} explained below, also see @ref{Installation directory}.
-@item --quietmmap
-Don't print any message when an array is stored in non-volatile memory
-(HDD/SSD) and not RAM, see the description of @option{--minmapsize} (above)
-for more.
+@cindex Tests, only one passes
+@cindex @file{LD_LIBRARY_PATH}
+@item
+@command{$ make check}: @emph{Only the first couple of tests pass, all the
rest fail or get skipped.} It is highly likely that when searching for shared
libraries, your system doesn't look into the @file{/usr/local/lib} directory
(or wherever you installed Gnuastro or its dependencies).
+To make sure it is added to the list of directories, add the following line to
your @file{~/.bashrc} file and restart your terminal.
+Don't forget to change @file{/usr/local/lib} if the libraries are installed in
other (non-standard) directories.
-@item -Z INT[,INT[,...]]
-@itemx --tilesize=[,INT[,...]]
-The size of regular tiles for tessellation, see @ref{Tessellation}.
-For each dimension an integer length (in units of data-elements or pixels) is
necessary.
-If the number of input dimensions is different from the number of values given
to this option, the program will stop with an error.
-Values must be separated by commas (@key{,}) and can also be fractions (for
example @code{4/2}).
-If they are fractions, the result must be an integer, otherwise an error will
be printed.
+@example
+export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib"
+@end example
-@item -M INT[,INT[,...]]
-@itemx --numchannels=INT[,INT[,...]]
-The number of channels for larger input tessellation, see @ref{Tessellation}.
-The number and types of acceptable values are similar to @option{--tilesize}.
-The only difference is that instead of length, the integers values given to
this option represent the @emph{number} of channels, not their size.
+You can also add more directories by using a colon `@code{:}' to separate them.
+See @ref{Installation directory} and @ref{Linking} to learn more on the
@code{PATH} variables and dynamic linking respectively.
-@item -F FLT
-@itemx --remainderfrac=FLT
-The fraction of remainder size along all dimensions to add to the first tile.
-See @ref{Tessellation} for a complete description.
-This option is only relevant if @option{--tilesize} is not exactly divisible
by the input dataset's size in a dimension.
-If the remainder size is larger than this fraction (compared to
@option{--tilesize}), then the remainder size will be added with one regular
tile size and divided between two tiles at the start and end of the given
dimension.
+@cindex GPL Ghostscript
+@item
+@command{$ make check}: @emph{The tests relying on external programs (for
example @file{fitstopdf.sh} fail.)} This is probably due to the fact that the
version number of the external programs is too old for the tests we have
preformed.
+Please update the program to a more recent version.
+For example to create a PDF image, you will need GPL Ghostscript, but older
versions do not work, we have successfully tested it on version 9.15.
+Older versions might cause a failure in the test result.
-@item --workoverch
-Ignore the channel borders for the high-level job of the given application.
-As a result, while the channel borders are respected in defining the small
tiles (such that no tile will cross a channel border), the higher-level program
operation will ignore them, see @ref{Tessellation}.
+@item
+@cindex @TeX{}
+@cindex GNU Texinfo
+@command{$ make pdf}: @emph{The PDF book cannot be made.}
+To make a PDF book, you need to have the GNU Texinfo program (like any
program, the more recent the better).
+A working @TeX{} program is also necessary, which you can get from Tex
Live@footnote{@url{https://www.tug.org/texlive/}}.
-@item --checktiles
-Make a FITS file with the same dimensions as the input but each pixel is
replaced with the ID of the tile that it is associated with.
-Note that the tile IDs start from 0.
-See @ref{Tessellation} for more on Tiling an image in Gnuastro.
+@item
+@cindex GNU Libtool
+After @code{make check}: do not copy the programs' executables to another (for
example, the installation) directory manually (using @command{cp}, or
@command{mv} for example).
+In the default configuration@footnote{If you configure Gnuastro with the
@option{--disable-shared} option, then the libraries will be statically linked
to the programs and this problem won't exist, see @ref{Linking}.}, the program
binaries need to link with Gnuastro's shared library which is also built and
installed with the programs.
+Therefore, to run successfully before and after installation, linking
modifications need to be made by GNU Libtool at installation time.
+@command{make install} does this internally, but a simple copy might give
linking errors when you run it.
+If you need to copy the executables, you can do so after installation.
-@item --oneelempertile
-When showing the tile values (for example with @option{--checktiles}, or when
the program's output is tessellated) only use one element for each tile.
-This can be useful when only the relative values given to each tile compared
to the rest are important or need to be checked.
-Since the tiles usually have a large number of pixels within them the output
will be much smaller, and so easier to read, write, store, or send.
+@cindex Tests, error in converting images
+@item
+@command{$ make} (when bootstrapping): After you have bootstrapped Gnuastro
from the version-controlled source, you may confront the following (or a
similar) error when converting images (for more on bootstrapping, see
@ref{Bootstrapping}):
-Note that when the full input size in any dimension is not exactly divisible
by the given @option{--tilesize} in that dimension, the edge tile(s) will have
different sizes (in units of the input's size), see @option{--remainderfrac}.
-But with this option, all displayed values are going to have the (same) size
of one data-element.
-Hence, in such cases, the image proportions are going to be slightly different
with this option.
+@example
+@code{convert: attempt to perform an operation not allowed by the
+security policy `gs' @ error/delegate.c/ExternalDelegateCommand/378.}
+@end example
-If your input image is not exactly divisible by the tile size and you want one
value per tile for some higher-level processing, all is not lost though.
-You can see how many pixels were within each tile (for example to weight the
values or discard some for later processing) with Gnuastro's Statistics (see
@ref{Statistics}) as shown below.
-The output FITS file is going to have two extensions, one with the median
calculated on each tile and one with the number of elements that each tile
covers.
-You can then use the @code{where} operator in @ref{Arithmetic} to set the
values of all tiles that don't have the regular area to a blank value.
+This error is a known
issue@footnote{@url{https://wiki.archlinux.org/title/ImageMagick}} with
@code{ImageMagick} security policies in some operating systems.
+In short, @code{imagemagick} uses Ghostscript for PDF, EPS, PS and XPS parsing.
+However, because some security vulnerabilities have been found in
Ghostscript@footnote{@url{https://security.archlinux.org/package/ghostscript}},
by default, ImageMagick may be compiled without Ghostscript library.
+In such cases, if allowed, ImageMagick will fall back to the external
@command{gs} command instead of the library.
+But this may be disabled with the following (or a similar) lines in
@code{/etc/ImageMagick-7/policy.xml} (anything related to PDF, PS, or
GhostScript).
@example
-$ aststatistics --median --number --ontile input.fits \
- --oneelempertile --output=o.fits
-$ REGULAR_AREA=1600 # Check second extension of `o.fits'.
-$ astarithmetic o.fits o.fits $REGULAR_AREA ne nan where \
- -h1 -h2
+<policy domain="delegate" rights="none" pattern="gs" />
+<policy domain="module" rights="none" pattern="@{PS,PDF,XPS@}" />
@end example
-Note that if @file{input.fits} also has blank values, then the median on
-tiles with blank values will also be ignored with the command above (which
-is desirable).
+To fix this problem, simply comment such lines (by placing a @code{<!--}
before each statement/line and @code{-->} at the end of that statement/line).
+@end itemize
-@item --inteponlyblank
-When values are to be interpolated, only change the values of the blank
-elements, keep the non-blank elements untouched.
+@noindent
+If your problem was not listed above, please file a bug report (@ref{Report a
bug}).
-@item --interpmetric=STR
-@cindex Radial metric
-@cindex Taxicab metric
-@cindex Manhattan metric
-@cindex Metric: Manhattan, Taxicab, Radial
-The metric to use for finding nearest neighbors.
-Currently it only accepts the Manhattan (or taxicab) metric with
@code{manhattan}, or the radial metric with @code{radial}.
-The Manhattan distance between two points is defined with
@mymath{|\Delta{x}|+|\Delta{y}|}.
-Thus the Manhattan metric has the advantage of being fast, but at the expense
of being less accurate.
-The radial distance is the standard definition of distance in a Euclidean
space: @mymath{\sqrt{\Delta{x}^2+\Delta{y}^2}}.
-It is accurate, but the multiplication and square root can slow down the
processing.
-@item --interpnumngb=INT
-The number of nearby non-blank neighbors to use for interpolation.
-@end table
-@node Operating mode options, , Processing options, Common options
-@subsubsection Operating mode options
-Another group of options that are common to all the programs in Gnuastro are
those to do with the general operation of the programs.
-The explanation for those that are not only limited to Gnuastro but are common
to all GNU programs start with (GNU option).
-@vtable @option
-@item --
-(GNU option) Stop parsing the command-line.
-This option can be useful in scripts or when using the shell history.
-Suppose you have a long list of options, and want to see if removing some of
them (to read from configuration files, see @ref{Configuration files}) can give
a better result.
-If the ones you want to remove are the last ones on the command-line, you
don't have to delete them, you can just add @option{--} before them and if you
don't get what you want, you can remove the @option{--} and get the same
initial result.
-@item --usage
-(GNU option) Only print the options and arguments and abort.
-This is very useful for when you know the what the options do, and have just
forgot their long/short identifiers, see @ref{--usage}.
-@item -?
-@itemx --help
-(GNU option) Print all options with an explanation and abort.
-Adding this option will print all the options in their short and long formats,
also displaying which ones need a value if they are called (with an @option{=}
after the long format followed by a string specifying the format, see
@ref{Options}).
-A short explanation is also given for what the option is for.
-The program will quit immediately after the message is printed and will not do
any form of processing, see @ref{--help}.
-@item -V
-@itemx --version
-(GNU option) Print a short message, showing the full name, version, copyright
information and program authors and abort.
-On the first line, it will print the official name (not executable name) and
version number of the program.
-Following this is a blank line and a copyright information.
-The program will not run.
-@item -q
-@itemx --quiet
-Don't report steps.
-All the programs in Gnuastro that have multiple major steps will report their
steps for you to follow while they are operating.
-If you do not want to see these reports, you can call this option and only
error/warning messages will be printed.
-If the steps are done very fast (depending on the properties of your input)
disabling these reports will also decrease running time.
-@item --cite
-Print all necessary information to cite and acknowledge Gnuastro in your
published papers.
-With this option, the programs will print the Bib@TeX{} entry to include in
your paper for Gnuastro in general, and the particular program's paper (if that
program comes with a separate paper).
-It will also print the necessary acknowledgment statement to add in the
respective section of your paper and it will abort.
-For a more complete explanation, please see @ref{Acknowledgments}.
-Citations and acknowledgments are vital for the continued work on Gnuastro.
-Gnuastro started, and is continued, based on separate research projects.
-So if you find any of the tools offered in Gnuastro to be useful in your
research, please use the output of this command to cite and acknowledge the
program (and Gnuastro) in your research paper.
-Thank you.
-Gnuastro is still new, there is no separate paper only devoted to Gnuastro yet.
-Therefore currently the paper to cite for Gnuastro is the paper for
NoiseChisel which is the first published paper introducing Gnuastro to the
astronomical community.
-Upon reaching a certain point, a paper completely devoted to describing
Gnuastro's many functionalities will be published, see @ref{GNU Astronomy
Utilities 1.0}.
-@item -P
-@itemx --printparams
-With this option, Gnuastro's programs will read your command-line options and
all the configuration files.
-If there is no problem (like a missing parameter or a value in the wrong
format or range) and immediately before actually running, the programs will
print the full list of option names, values and descriptions, sorted and
grouped by context and abort.
-They will also report the version number, the date they were configured on
your system and the time they were reported.
-As an example, you can give your full command-line options and even the input
and output file names and finally just add @option{-P} to check if all the
parameters are finely set.
-If everything is OK, you can just run the same command (easily retrieved from
the shell history, with the top arrow key) and simply remove the last two
characters that showed this option.
-No program will actually start its processing when this option is called.
-The otherwise mandatory arguments for each program (for example input image or
catalog files) are no longer required when you call this option.
-@item --config=STR
-Parse @option{STR} as a configuration file name, immediately when this option
is confronted (see @ref{Configuration files}).
-The @option{--config} option can be called multiple times in one run of any
Gnuastro program on the command-line or in the configuration files.
-In any case, it will be immediately read (before parsing the rest of the
options on the command-line, or lines in a configuration file).
-If the given file doesn't exist or can't be read for any reason, the program
will print a warning and continue its processing.
-The warning can be suppressed with @option{--quiet}.
-Note that by definition, options on the command-line still take precedence
over those in any configuration file, including the file(s) given to this
option if they are called before it.
-Also see @option{--lastconfig} and @option{--onlyversion} on how this option
can be used for reproducible results.
-You can use @option{--checkconfig} (below) to check/confirm the parsing of
configuration files.
+@node Common program behavior, Data containers, Installation, Top
+@chapter Common program behavior
-@item --checkconfig
-Print options and their values, within the command-line or configuration
files, as they are parsed (see @ref{Configuration file precedence}).
-If an option has already been set, or is ignored by the program, this option
will also inform you with special values like @code{--ALREADY-SET--}.
-Only options that are parsed after this option are printed, so to see the
parsing of all input options, it is recommended to put this option immediately
after the program name before any other options.
+All the programs in Gnuastro share a set of common behavior mainly to do with
user interaction to facilitate their usage and development.
+This includes how to feed input datasets into the programs, how to configure
them, specifying the outputs, numerical data types, treating columns of
information in tables, etc.
+This chapter is devoted to describing this common behavior in all programs.
+Because the behaviors discussed here are common to several programs, they are
not repeated in each program's description.
-@cindex Debug
-This is a very good option to confirm where the value of each option is has
been defined in scenarios where there are multiple configuration files (for
debugging).
+In @ref{Command-line}, a very general description of running the programs on
the command-line is discussed, like difference between arguments and options,
as well as options that are common/shared between all programs.
+None of Gnuastro's programs keep any internal configuration value (values for
their different operational steps), they read their configuration primarily
from the command-line, then from specific files in directory, user, or
system-wide settings.
+Using these configuration files can greatly help reproducible and robust usage
of Gnuastro, see @ref{Configuration files} for more.
-@item -S
-@itemx --setdirconf
-Update the current directory configuration file for the Gnuastro program and
quit.
-The full set of command-line and configuration file options will be parsed and
options with a value will be written in the current directory configuration
file for this program (see @ref{Configuration files}).
-If the configuration file or its directory doesn't exist, it will be created.
-If a configuration file exists it will be replaced (after it, and all other
configuration files have been read).
-In any case, the program will not run.
+It is not possible to always have the different options and configurations of
each program on the top of your head.
+It is very natural to forget the options of a program, their current default
values, or how it should be run and what it did.
+Gnuastro's programs have multiple ways to help you refresh your memory in
multiple levels (just an option name, a short description, or fast access to
the relevant section of the manual.
+See @ref{Getting help} for more for more on benefiting from this very
convenient feature.
-This is the recommended method@footnote{Alternatively, you can use your
favorite text editor.} to edit/set the configuration file for all future calls
to Gnuastro's programs.
-It will internally check if your values are in the correct range and type and
save them according to the configuration file format, see @ref{Configuration
file format}.
-So if there are unreasonable values to some options, the program will notify
you and abort before writing the final configuration file.
+Many of the programs use the multi-threaded character of modern CPUs, in
@ref{Multi-threaded operations} we'll discuss how you can configure this
behavior, along with some tips on making best use of them.
+In @ref{Numeric data types}, we'll review the various types to store numbers
in your datasets: setting the proper type for the usage context@footnote{For
example if the values in your dataset can only be integers between 0 or 65000,
store them in a unsigned 16-bit type, not 64-bit floating point type (which is
the default in most systems).
+It takes four times less space and is much faster to process.} can greatly
improve the file size and also speed of reading, writing or processing them.
-When this option is called, the otherwise mandatory arguments, for
-example input image or catalog file(s), are no longer mandatory (since
-the program will not run).
+We'll then look into the recognized table formats in @ref{Tables} and how
large datasets are broken into tiles, or mesh grid in @ref{Tessellation}.
+Finally, we'll take a look at the behavior regarding output files:
@ref{Automatic output} describes how the programs set a default name for their
output when you don't give one explicitly (using @option{--output}).
+When the output is a FITS file, all the programs also store some very useful
information in the header that is discussed in @ref{Output FITS files}.
-@item -U
-@itemx --setusrconf
-Update the user configuration file and quit (see @ref{Configuration files}).
-See explanation under @option{--setdirconf} for more details.
+@menu
+* Command-line:: How to use the command-line.
+* Configuration files:: Values for unspecified variables.
+* Getting help:: Getting more information on the go.
+* Multi-threaded operations:: How threads are managed in Gnuastro.
+* Numeric data types:: Different types and how to specify them.
+* Memory management:: How memory is allocated (in RAM or HDD/SSD).
+* Tables:: Recognized table formats.
+* Tessellation:: Tile the dataset into non-overlapping bins.
+* Automatic output:: About automatic output names.
+* Output FITS files:: Common properties when outputs are FITS.
+@end menu
-@item --lastconfig
-This is the last configuration file that must be read.
-When this option is confronted in any stage of reading the options (on the
command-line or in a configuration file), no other configuration file will be
parsed, see @ref{Configuration file precedence} and @ref{Current directory and
User wide}.
-Like all on/off options, on the command-line, this option doesn't take any
values.
-But in a configuration file, it takes the values of @option{0} or @option{1},
see @ref{Configuration file format}.
-If it is present in a configuration file with a value of @option{0}, then all
later occurrences of this option will be ignored.
+@node Command-line, Configuration files, Common program behavior, Common
program behavior
+@section Command-line
+Gnuastro's programs are customized through the standard Unix-like command-line
environment and GNU style command-line options.
+Both are very common in many Unix-like operating system programs.
+In @ref{Arguments and options} we'll start with the difference between
arguments and options and elaborate on the GNU style of options.
+Afterwards, in @ref{Common options}, we'll go into the detailed list of all
the options that are common to all the programs in Gnuastro.
-@item --onlyversion=STR
-Only run the program if Gnuastro's version is exactly equal to @option{STR}
(see @ref{Version numbering}).
-Note that it is not compared as a number, but as a string of characters, so
@option{0}, or @option{0.0} and @option{0.00} are different.
-If the running Gnuastro version is different, then this option will report an
error and abort as soon as it is confronted on the command-line or in a
configuration file.
-If the running Gnuastro version is the same as @option{STR}, then the program
will run as if this option was not called.
+@menu
+* Arguments and options:: Different ways to specify inputs and
configuration.
+* Common options:: Options that are shared between all programs.
+* Shell TAB completion:: Customized TAB completion in Gnuastro.
+* Standard input:: Using output of another program as input.
+@end menu
-This is useful if you want your results to be exactly reproducible and not
mistakenly run with an updated/newer or older version of the program.
-Besides internal algorithmic/behavior changes in programs, the existence of
options or their names might change between versions (especially in these
earlier versions of Gnuastro).
+@node Arguments and options, Common options, Command-line, Command-line
+@subsection Arguments and options
-Hence, when using this option (probably in a script or in a configuration
file), be sure to call it before other options.
-The benefit is that, when the version differs, the other options won't be
parsed and you, or your collaborators/users, won't get errors saying an option
in your configuration doesn't exist in the running version of the program.
+@cindex Shell
+@cindex Options to programs
+@cindex Command-line options
+@cindex Arguments to programs
+@cindex Command-line arguments
+When you type a command on the command-line, it is passed onto the shell (a
generic name for the program that manages the command-line) as a string of
characters.
+As an example, see the ``Invoking ProgramName'' sections in this manual for
some examples of commands with each program, like @ref{Invoking asttable},
@ref{Invoking astfits}, or @ref{Invoking aststatistics}.
-Here is one example of how this option can be used in conjunction with the
@option{--lastconfig} option.
-Let's assume that you were satisfied with the results of this command:
@command{astnoisechisel image.fits --snquant=0.95} (along with various options
set in various configuration files).
-You can save the state of NoiseChisel and reproduce that exact result on
@file{image.fits} later by following these steps (the extra spaces, and
@key{\}, are only for easy readability, if you want to try it out, only one
space between each token is enough).
+The shell then brakes up your string into separate @emph{tokens} or
@emph{words} using any @emph{metacharacters} (like white-space, tab,
@command{|}, @command{>} or @command{;}) that are in the string.
+On the command-line, the first thing you usually enter is the name of the
program you want to run.
+After that, you can specify two types of tokens: @emph{arguments} and
@emph{options}.
+In the GNU-style, arguments are those tokens that are not preceded by any
hyphens (@command{-}, see @ref{Arguments}).
+Here is one example:
@example
-$ echo "onlyversion X.XX" > reproducible.conf
-$ echo "lastconfig 1" >> reproducible.conf
-$ astnoisechisel image.fits --snquant=0.95 -P \
- >> reproducible.conf
+$ astcrop --center=53.162551,-27.789676 -w10/3600 --mode=wcs udf.fits
@end example
-@option{--onlyversion} was available from Gnuastro 0.0, so putting it
immediately at the start of a configuration file will ensure that later, you
(or others using different version) won't get a non-recognized option error in
case an option was added/removed.
-@option{--lastconfig} will inform the installed NoiseChisel to not parse any
other configuration files.
-This is done because we don't want the user's user-wide or system wide option
values affecting our results.
-Finally, with the third command, which has a @option{-P} (short for
@option{--printparams}), NoiseChisel will print all the option values visible
to it (in all the configuration files) and the shell will append them to
@file{reproduce.conf}.
-Hence, you don't have to worry about remembering the (possibly) different
options in the different configuration files.
+In the example above, we are running @ref{Crop} to crop a region of width 10
arc-seconds centered at the given RA and Dec from the input Hubble Ultra-Deep
Field (UDF) FITS image.
+Here, the argument is @file{udf.fits}.
+Arguments are most commonly the input file names containing your data.
+Options start with one or two hyphens, followed by an identifier for the
option (the option's name, for example, @option{--center}, @option{-w},
@option{--mode} in the example above) and its value (anything after the option
name, or the optional @key{=} character).
+Through options you can configure how the program runs (interprets the data
you provided).
-Afterwards, if you run NoiseChisel as shown below (telling it to read this
configuration file with the @file{--config} option).
-You can be sure that there will either be an error (for version mismatch) or
it will produce exactly the same result that you got before.
+@vindex --help
+@vindex --usage
+@cindex Mandatory arguments
+Arguments can be mandatory and optional and unlike options, they don't have
any identifiers.
+Hence, when there multiple arguments, their order might also matter (for
example in @command{cp} which is used for copying one file to another location).
+The outputs of @option{--usage} and @option{--help} shows which arguments are
optional and which are mandatory, see @ref{--usage}.
+
+As their name suggests, @emph{options} can be considered to be optional and
most of the time, you don't have to worry about what order you specify them in.
+When the order does matter, or the option can be invoked multiple times, it is
explicitly mentioned in the ``Invoking ProgramName'' section of each program
(this is a very important aspect of an option).
+
+@cindex Metacharacters on the command-line In case your arguments or option
values contain any of the shell's meta-characters, you have to quote them.
+If there is only one such character, you can use a backslash (@command{\})
before it.
+If there are multiple, it might be easier to simply put your whole argument or
option value inside of double quotes (@command{"}).
+In such cases, everything inside the double quotes will be seen as one token
or word.
+@cindex HDU
+@cindex Header data unit
+For example, let's say you want to specify the header data unit (HDU) of your
FITS file using a complex expression like `@command{3; images(exposure > 100)}'.
+If you simply add these after the @option{--hdu} (@option{-h}) option, the
programs in Gnuastro will read the value to the HDU option as `@command{3}' and
run.
+Then, the shell will attempt to run a separate command
`@command{images(exposure > 100)}' and complain about a syntax error.
+This is because the semicolon (@command{;}) is an `end of command' character
in the shell.
+To solve this problem you can simply put double quotes around the whole string
you want to pass to @option{--hdu} as seen below:
@example
-$ astnoisechisel --config=reproducible.conf
+$ astcrop --hdu="3; images(exposure > 100)" image.fits
@end example
-@item --log
-Some programs can generate extra information about their outputs in a log file.
-When this option is called in those programs, the log file will also be
printed.
-If the program doesn't generate a log file, this option is ignored.
-@cartouche
-@noindent
-@strong{@option{--log} isn't thread-safe}: The log file usually has a fixed
name.
-Therefore if two simultaneous calls (with @option{--log}) of a program are
made in the same directory, the program will try to write to he same file.
-This will cause problems like unreasonable log file, undefined behavior, or a
crash.
-@end cartouche
-@cindex CPU threads, set number
-@cindex Number of CPU threads to use
-@item -N INT
-@itemx --numthreads=INT
-Use @option{INT} CPU threads when running a Gnuastro program (see
@ref{Multi-threaded operations}).
-If the value is zero (@code{0}), or this option is not given on the
command-line or any configuration file, the value will be determined at
run-time: the maximum number of threads available to the system when you run a
Gnuastro program.
-Note that multi-threaded programming is only relevant to some programs.
-In others, this option will be ignored.
-@end vtable
+@menu
+* Arguments:: For specifying the main input files/operations.
+* Options:: For configuring the behavior of the program.
+@end menu
+@node Arguments, Options, Arguments and options, Arguments and options
+@subsubsection Arguments
+In Gnuastro, arguments are almost exclusively used as the input data file
names.
+Please consult the first few paragraph of the ``Invoking ProgramName'' section
for each program for a description of what it expects as input, how many
arguments, or input data, it accepts, or in what order.
+Everything particular about how a program treats arguments, is explained under
the ``Invoking ProgramName'' section for that program.
+@cindex Filename suffix
+@cindex Suffix (filename)
+@cindex FITS filename suffixes
+Generally, if there is a standard file name suffix for a particular format,
that filename extension is checked to identify their format.
+In astronomy (and thus Gnuastro), FITS is the preferred format for inputs and
outputs, so the focus here and throughout this book is on FITS.
+However, other formats are also accepted in special cases, for example
@ref{ConvertType} also accepts JPEG or TIFF inputs, and writes JPEG, EPS or PDF
files.
+The recognized suffixes for these formats are listed there.
+The list below shows the recognized suffixes for FITS data files in Gnuastro's
programs.
+However, in some scenarios FITS writers may not append a suffix to the file,
or use a non-recognized suffix (not in the list below).
+Therefore if a FITS file is expected, but it doesn't have any of these
suffixes, Gnuastro programs will look into the contents of the file and if it
does conform with the FITS standard, the file will be used.
+Just note that checking about 5 characters at the end of a name string is much
more efficient than opening and checking the contents of a file, so it is
generally recommended to have a recognized FITS suffix.
-@node Shell TAB completion, Standard input, Common options, Command-line
-@subsection Shell TAB completion (highly customized)
+@itemize
-@cartouche
-@noindent
-@strong{Under development:} Gnuastro's TAB completion in Bash already greatly
improves usage of Gnuastro on the command-line, but still under development and
not yet complete.
-If you are interested to try it out, please go ahead and activate it (as
described below), we encourage this.
-But please have in mind that there are known
issues@footnote{@url{http://savannah.gnu.org/bugs/index.php?group=gnuastro&category_id=128}}
and you may find new issues.
-If you do, please get in touch with us as described in @ref{Report a bug}.
-TAB completion is currently only implemented in the following programs:
Arithmetic, BuildProgram, ConvertType, Convolve, CosmicCalculator, Crop, Fits
and Table.
-For progress on this task, please see Task
15799@footnote{@url{https://savannah.gnu.org/task/?15799}}.
-@end cartouche
+@item
+@file{.fits}: The standard file name ending of a FITS image.
-@cindex Bash auto-complete
-@cindex Completion in the shell
-@cindex Bash programmable completion
-@cindex Autocomplete (in the shell/Bash)
-Bash provides a built-in feature called @emph{programmable
completion}@footnote{@url{https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion.html}}
to help increase interactive workflow efficiency and minimize the number of
key-strokes @emph{and} the need to memorize things.
-It is also known as TAB completion, bash completion, auto-completion, or word
completion.
-Completion is activated by pressing @key{[TAB]} while you are typing a command.
-For file arguments this is the default behavior already and you have probably
used it a lot with any command-line program.
+@item
+@file{.fit}: Alternative (3 character) FITS suffix.
-Besides this simple/default mode, Bash also enables a high level of
customization features for its completion.
-These features have been extensively used in Gnuastro to improve your work
efficiency@footnote{To learn how Gnuastro implements TAB completion in Bash,
see @ref{Bash programmable completion}.}.
-For example if you are running @code{asttable} (which only accepts files
containing a table), and you press @key{[TAB]}, it will only suggest files
containing tables.
-As another example, if an option needs image HDUs within a FITS file, pressing
@key{[TAB]} will only suggest the image HDUs (and not other possibly existing
HDUs that contain tables, or just metadata).
-Just note that the file name has to be already given on the command-line
before reaching such options (that look into the contents of a file).
+@item
+@file{.fits.Z}: A FITS image compressed with @command{compress}.
-But TAB completion is not limited to file types or contents.
-Arguments/Options that take certain fixed string values will directly suggest
those strings with TAB, and completely ignore the file structure (for example
spectral line names in @ref{Invoking astcosmiccal})!
-As another example, the option @option{--numthreads} option (to specify the
number of threads to use by the program), will find the number of available
threads on the system, and suggest the possible numbers with a TAB!
+@item
+@file{.fits.gz}: A FITS image compressed with GNU zip (gzip).
-To activate Gnuastro's custom TAB completion in Bash, you need to put the
following line in one of your Bash startup files (for example @file{~/.bashrc}).
-If you installed Gnuastro using the steps of @ref{Quick start}, you should
have already done this (the command just after @command{sudo make install}).
-For a list of (and discussion on) Bash startup files and installation
directories see @ref{Installation directory}.
-Of course, if Gnuastro was installed in a custom location, replace the
`@file{/usr/local}' part of the line below to the value that was given to
@option{--prefix} during Gnuastro's configuration@footnote{In case you don't
know the installation directory of Gnuastro on your system, you can find out
with this command: @code{which astfits | sed -e"s|/bin/astfits||"}}.
+@item
+@file{.fits.fz}: A FITS image compressed with @command{fpack}.
-@example
-# Enable Gnuastro's TAB completion
-source /usr/local/share/gnuastro/completion.bash
-@end example
+@item
+@file{.imh}: IRAF format image file.
-After adding the line above in a Bash startup file, TAB completion will always
be activated in any new terminal.
-To see if it has been activated, try it out with @command{asttable [TAB][TAB]}
and @command{astarithmetic [TAB][TAB]} in a directory that contains tables and
images.
-The first will only suggest the files with a table, and the second, only those
with an image.
+@end itemize
-@cartouche
-@noindent
-@strong{TAB completion only works with long option names:}
-As described above, short options are much more complex to generalize,
therefore TAB completion is only available for long options.
-But don't worry!
-TAB completion also involves option names, so if you just type
@option{--a[TAB][TAB]}, you will get the list of options that start with an
@option{--a}.
-Therefore as a side-effect of TAB completion, your commands will be far more
human-readable with minimal key strokes.
-@end cartouche
+Through out this book and in the command-line outputs, whenever we want to
generalize all such astronomical data formats in a text place holder, we will
use @file{ASTRdata}, we will assume that the extension is also part of this
name.
+Any file ending with these names is directly passed on to CFITSIO to read.
+Therefore you don't necessarily have to have these files on your computer,
they can also be located on an FTP or HTTP server too, see the CFITSIO manual
for more information.
+CFITSIO has its own error reporting techniques, if your input file(s)
+cannot be opened, or read, those errors will be printed prior to the
+final error by Gnuastro.
-@node Standard input, , Shell TAB completion, Command-line
-@subsection Standard input
-@cindex Standard input
-@cindex Stream: standard input
-The most common way to feed the primary/first input dataset into a program is
to give its filename as an argument (discussed in @ref{Arguments}).
-When you want to run a series of programs in sequence, this means that each
will have to keep the output of each program in a separate file and re-type
that file's name in the next command.
-This can be very slow and frustrating (mis-typing a file's name).
-@cindex Standard output stream
-@cindex Stream: standard output
-To solve the problem, the founders of Unix defined pipes to directly feed the
output of one program (its ``Standard output'' stream) into the ``standard
input'' of a next program.
-This removes the need to make temporary files between separate processes and
became one of the best demonstrations of the Unix-way, or Unix philosophy.
+@node Options, , Arguments, Arguments and options
+@subsubsection Options
-Every program has three streams identifying where it reads/writes non-file
inputs/outputs: @emph{Standard input}, @emph{Standard output}, and
@emph{Standard error}.
-When a program is called alone, all three are directed to the terminal that
you are using.
-If it needs an input, it will prompt you for one and you can type it in.
-Or, it prints its results in the terminal for you to see.
+@cindex GNU style options
+@cindex Options, GNU style
+@cindex Options, short (@option{-}) and long (@option{--})
+Command-line options allow configuring the behavior of a program in all
GNU/Linux applications for each particular execution on a particular input data.
+A single option can be called in two ways: @emph{long} or @emph{short}.
+All options in Gnuastro accept the long format which has two hyphens an can
have many characters (for example @option{--hdu}).
+Short options only have one hyphen (@key{-}) followed by one character (for
example @option{-h}).
+You can see some examples in the list of options in @ref{Common options} or
those for each program's ``Invoking ProgramName'' section.
+Both formats are shown for those which support both.
+First the short is shown then the long.
-For example, say you have a FITS table/catalog containing the B and V band
magnitudes (@code{MAG_B} and @code{MAG_V} columns) of a selection of galaxies
along with many other columns.
-If you want to see only these two columns in your terminal, can use Gnuastro's
@ref{Table} program like below:
+Usually, the short options are for when you are writing on the command-line
and want to save keystrokes and time.
+The long options are good for shell scripts, where you aren't usually rushing.
+Long options provide a level of documentation, since they are more descriptive
and less cryptic.
+Usually after a few months of not running a program, the short options will be
forgotten and reading your previously written script will not be easy.
-@example
-$ asttable cat.fits -cMAG_B,MAG_V
-@end example
+@cindex On/Off options
+@cindex Options, on/off
+Some options need to be given a value if they are called and some don't.
+You can think of the latter type of options as on/off options.
+These two types of options can be distinguished using the output of the
@option{--help} and @option{--usage} options, which are common to all GNU
software, see @ref{Getting help}.
+In Gnuastro we use the following strings to specify when the option needs a
value and what format that value should be in.
+More specific tests will be done in the program and if the values are out of
range (for example negative when the program only wants a positive value), an
error will be reported.
-Through the Unix pipe mechanism, when the shell confronts the pipe character
(@key{|}), it connects the standard output of the program before the pipe, to
the standard input of the program after it.
-So it is literally a ``pipe'': everything that you would see printed by the
first program on the command (without any pipe), is now passed to the second
program (and not seen by you).
+@vtable @option
-@cindex AWK
-@cindex GNU AWK
-To continue the previous example, let's say you want to see the B-V color.
-To do this, you can pipe Table's output to AWK (a wonderful tool for
processing things like plain text tables):
+@item INT
+The value is read as an integer.
-@example
-$ asttable cat.fits -cMAG_B,MAG_V | awk '@{print $1-$2@}'
-@end example
-
-But understanding the distribution by visually seeing all the numbers under
each other is not too useful! You can therefore feed this single column
information into @ref{Statistics} to give you a general feeling of the
distribution with the same command:
+@item FLT
+The value is read as a float.
+There are generally two types, depending on the context.
+If they are for fractions, they will have to be less than or equal to unity.
-@example
-$ asttable cat.fits -cMAG_B,MAG_V | awk '@{print $1-$2@}' | aststatistics
-@end example
+@item STR
+The value is read as a string of characters.
+For example column names in a table, or HDU names in a multi-extension FITS
file.
+Other examples include human-readable settings by some programs like the
@option{--domain} option of the Convolve program that can be either
@code{spatial} or @code{frequency} (to specify the type of convolution, see
@ref{Convolve}).
-Gnuastro's programs that accept input from standard input, only look into the
Standard input stream if there is no first argument.
-In other words, arguments take precedence over Standard input.
-When no argument is provided, the programs check if the standard input stream
is already full or not (output from another program is waiting to be used).
-If data is present in the standard input stream, it is used.
+@item FITS @r{or} FITS/TXT
+The value should be a file (most commonly FITS).
+In many cases, other formats may also be accepted (for example input tables
can be FITS or plain-text, see @ref{Recognized table formats}).
-When the standard input is empty, the program will wait
@option{--stdintimeout} micro-seconds for you to manually enter the first line
(ending with a new-line character, or the @key{ENTER} key, see @ref{Input
output options}).
-If it detects the first line in this time, there is no more time limit, and
you can manually write/type all the lines for as long as it takes.
-To inform the program that Standard input has finished, press @key{CTRL-D}
after a new line.
-If the program doesn't catch the first line before the time-out finishes, it
will abort with an error saying that no input was provided.
+@end vtable
-@cartouche
@noindent
-@strong{Manual input in Standard input is discarded:}
-Be careful that when you manually fill the Standard input, the data will be
discarded once the program finishes and reproducing the result will be
impossible.
-Therefore this form of providing input is only good for temporary tests.
-@end cartouche
+@cindex Values to options
+@cindex Option values
+To specify a value in the short format, simply put the value after the option.
+Note that since the short options are only one character long, you don't have
to type anything between the option and its value.
+For the long option you either need white space or an @option{=} sign, for
example @option{-h2}, @option{-h 2}, @option{--hdu 2} or @option{--hdu=2} are
all equivalent.
-@cartouche
+The short format of on/off options (those that don't need values) can be
concatenated for example these two hypothetical sequences of options are
equivalent: @option{-a -b -c4} and @option{-abc4}.
+As an example, consider the following command to run Crop:
+@example
+$ astcrop -Dr3 --wwidth 3 catalog.txt --deccol=4 ASTRdata
+@end example
@noindent
-@strong{Standard input currently only for plain text:}
-Currently Standard input only works for plain text inputs like the example
above.
-We will later allow FITS files into the programs through standard input also.
-@end cartouche
+The @command{$} is the shell prompt, @command{astcrop} is the program name.
+There are two arguments (@command{catalog.txt} and @command{ASTRdata}) and
four options, two of them given in short format (@option{-D}, @option{-r}) and
two in long format (@option{--width} and @option{--deccol}).
+Three of them require a value and one (@option{-D}) is an on/off option.
+@vindex --printparams
+@cindex Options, abbreviation
+@cindex Long option abbreviation
+If an abbreviation is unique between all the options of a program, the long
option names can be abbreviated.
+For example, instead of typing @option{--printparams}, typing @option{--print}
or maybe even @option{--pri} will be enough, if there are conflicts, the
program will warn you and show you the alternatives.
+Finally, if you want the argument parser to stop parsing arguments beyond a
certain point, you can use two dashes: @option{--}.
+No text on the command-line beyond these two dashes will be parsed.
-@node Configuration files, Getting help, Command-line, Common program behavior
-@section Configuration files
+@cindex Repeated options
+@cindex Options, repeated
+Gnuastro has two types of options with values, those that only take a single
value are the most common type.
+If these options are repeated or called more than once on the command-line,
the value of the last time it was called will be assigned to it.
+This is very useful when you are testing/experimenting.
+Let's say you want to make a small modification to one option value.
+You can simply type the option with a new value in the end of the command and
see how the script works.
+If you are satisfied with the change, you can remove the original option for
human readability.
+If the change wasn't satisfactory, you can remove the one you just added and
not worry about forgetting the original value.
+Without this capability, you would have to memorize or save the original value
somewhere else, run the command and then change the value again which is not at
all convenient and is potentially cause lots of bugs.
+
+On the other hand, some options can be called multiple times in one run of a
program and can thus take multiple values (for example see the
@option{--column} option in @ref{Invoking asttable}.
+In these cases, the order of stored values is the same order that you
specified on the command-line.
-@cindex @file{etc}
@cindex Configuration files
-@cindex Necessary parameters
@cindex Default option values
-@cindex File system Hierarchy Standard
-Each program needs a certain number of parameters to run.
-Supplying all the necessary parameters each time you run the program is very
frustrating and prone to errors.
-Therefore all the programs read the values for the necessary options you have
not given in the command line from one of several plain text files (which you
can view and edit with any text editor).
-These files are known as configuration files and are usually kept in a
directory named @file{etc/} according to the file system hierarchy
-standard@footnote{@url{http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard}}.
+Gnuastro's programs don't keep any internal default values, so some options
are mandatory and if they don't have a value, the program will complain and
abort.
+Most programs have many such options and typing them by hand on every call is
impractical.
+To facilitate the user experience, after parsing the command-line, Gnuastro's
programs read special configuration files to get the necessary values for the
options you haven't identified on the command-line.
+These configuration files are fully described in @ref{Configuration files}.
-@vindex --output
-@vindex --numthreads
-@cindex CPU threads, number
-@cindex Internal default value
-@cindex Number of CPU threads to use
-The thing to have in mind is that none of the programs in Gnuastro keep any
internal default value.
-All the values must either be stored in one of the configuration files or
explicitly called in the command-line.
-In case the necessary parameters are not given through any of these methods,
the program will print a missing option error and abort.
-The only exception to this is @option{--numthreads}, whose default value is
determined at run-time using the number of threads available to your system,
see @ref{Multi-threaded operations}.
-Of course, you can still provide a default value for the number of threads at
any of the levels below, but if you don't, the program will not abort.
-Also note that through automatic output name generation, the value to the
@option{--output} option is also not mandatory on the command-line or in the
configuration files for all programs which don't rely on that value as an
input@footnote{One example of a program which uses the value given to
@option{--output} as an input is ConvertType, this value specifies the type of
the output through the value to @option{--output}, see @ref{Invoking
astconvertt}.}, see @ref{Automatic output}.
+@cartouche
+@noindent
+@cindex Tilde expansion as option values
+@strong{CAUTION:} In specifying a file address, if you want to use the shell's
tilde expansion (@command{~}) to specify your home directory, leave at least
one space between the option name and your value.
+For example use @command{-o ~/test}, @command{--output ~/test} or
@command{--output= ~/test}.
+Calling them with @command{-o~/test} or @command{--output=~/test} will disable
shell expansion.
+@end cartouche
+@cartouche
+@noindent
+@strong{CAUTION:} If you forget to specify a value for an option which
requires one, and that option is the last one, Gnuastro will warn you.
+But if it is in the middle of the command, it will take the text of the next
option or argument as the value which can cause undefined behavior.
+@end cartouche
+@cartouche
+@noindent
+@cindex Counting from zero.
+@strong{NOTE:} In some contexts Gnuastro's counting starts from 0 and in
others 1.
+You can assume by default that counting starts from 1, if it starts from 0 for
a special option, it will be explicitly mentioned.
+@end cartouche
+@node Common options, Shell TAB completion, Arguments and options, Command-line
+@subsection Common options
+@cindex Options common to all programs
+@cindex Gnuastro common options
+To facilitate the job of the users and developers, all the programs in
Gnuastro share some basic command-line options for the options that are common
to many of the programs.
+The full list is classified as @ref{Input output options}, @ref{Processing
options}, and @ref{Operating mode options}.
+In some programs, some of the options are irrelevant, but still recognized
(you won't get an unrecognized option error, but the value isn't used).
+Unless otherwise mentioned, these options are identical between all programs.
@menu
-* Configuration file format:: ASCII format of configuration file.
-* Configuration file precedence:: Precedence of configuration files.
-* Current directory and User wide:: Local and user configuration files.
-* System wide:: System wide configuration files.
+* Input output options:: Common input/output options.
+* Processing options:: Options for common processing steps.
+* Operating mode options:: Common operating mode options.
@end menu
-@node Configuration file format, Configuration file precedence, Configuration
files, Configuration files
-@subsection Configuration file format
+@node Input output options, Processing options, Common options, Common options
+@subsubsection Input/Output options
-@cindex Configuration file suffix
-The configuration files for each program have the standard program executable
name with a `@file{.conf}' suffix.
-When you download the source code, you can find them in the same directory as
the source code of each program, see @ref{Program source}.
+These options are to do with the input and outputs of the various
+programs.
-@cindex White space character
-@cindex Configuration file format
-Any line in the configuration file whose first non-white character is a
@key{#} is considered to be a comment and is ignored.
-An empty line is also similarly ignored.
-The long name of the option should be used as an identifier.
-The parameter name and parameter value have to be separated by any number of
`white-space' characters: space, tab or vertical tab.
-By default several space characters are used.
-If the value of an option has space characters (most commonly for the
@option{hdu} option), then the full value can be enclosed in double quotation
signs (@key{"}, similar to the example in @ref{Arguments and options}).
-If it is an option without a value in the @option{--help} output (on/off
option, see @ref{Options}), then the value should be @option{1} if it is to be
`on' and @option{0} otherwise.
+@vtable @option
-In each non-commented and non-blank line, any text after the first two words
(option identifier and value) is ignored.
-If an option identifier is not recognized in the configuration file, the name
of the file, the line number of the unrecognized option, and the unrecognized
identifier name will be reported and the program will abort.
-If a parameter is repeated more more than once in the configuration files,
accepts only one value, and is not set on the command-line, then only the first
value will be used, the rest will be ignored.
+@cindex Timeout
+@cindex Standard input
+@item --stdintimeout
+Number of micro-seconds to wait for writing/typing in the @emph{first line} of
standard input from the command-line (see @ref{Standard input}).
+This is only relevant for programs that also accept input from the standard
input, @emph{and} you want to manually write/type the contents on the terminal.
+When the standard input is already connected to a pipe (output of another
program), there won't be any waiting (hence no timeout, thus making this option
redundant).
-@cindex Writing configuration files
-@cindex Automatic configuration file writing
-@cindex Configuration files, writing
-You can build or edit any of the directories and the configuration files
yourself using any text editor.
-However, it is recommended to use the @option{--setdirconf} and
@option{--setusrconf} options to set default values for the current directory
or this user, see @ref{Operating mode options}.
-With these options, the values you give will be checked before writing in the
configuration file.
-They will also print a set of commented lines guiding the reader and will also
classify the options based on their context and write them in their logical
order to be more understandable.
+If the first line-break (for example with the @key{ENTER} key) is not provided
before the timeout, the program will abort with an error that no input was
given.
+Note that this time interval is @emph{only} for the first line that you type.
+Once the first line is given, the program will assume that more data will come
and accept rest of your inputs without any time limit.
+You need to specify the ending of the standard input, for example by pressing
@key{CTRL-D} after a new line.
+Note that any input you write/type into a program on the command-line with
Standard input will be discarded (lost) once the program is finished.
+It is only recoverable manually from your command-line (where you actually
typed) as long as the terminal is open.
+So only use this feature when you are sure that you don't need the dataset (or
have a copy of it somewhere else).
-@node Configuration file precedence, Current directory and User wide,
Configuration file format, Configuration files
-@subsection Configuration file precedence
-@cindex Configuration file precedence
-@cindex Configuration file directories
-@cindex Precedence, configuration files
-The option values in all the programs of Gnuastro will be filled in the
following order.
-If an option only takes one value which is given in an earlier step, any value
for that option in a later step will be ignored.
-Note that if the @option{lastconfig} option is specified in any step below, no
other configuration files will be parsed (see @ref{Operating mode options}).
+@cindex HDU
+@cindex Header data unit
+@item -h STR/INT
+@itemx --hdu=STR/INT
+The name or number of the desired Header Data Unit, or HDU, in the FITS image.
+A FITS file can store multiple HDUs or extensions, each with either an image
or a table or nothing at all (only a header).
+Note that counting of the extensions starts from 0(zero), not 1(one).
+Counting from 0 is forced on us by CFITSIO which directly reads the value you
give with this option (see @ref{CFITSIO}).
+When specifying the name, case is not important so @command{IMAGE},
@command{image} or @command{ImAgE} are equivalent.
-@enumerate
-@item
-Command-line options, for a particular run of ProgramName.
+CFITSIO has many capabilities to help you find the extension you want, far
beyond the simple extension number and name.
+See CFITSIO manual's ``HDU Location Specification'' section for a very
complete explanation with several examples.
+A @code{#} is appended to the string you specify for the HDU@footnote{With the
@code{#} character, CFITSIO will only read the desired HDU into your memory,
not all the existing HDUs in the fits file.} and the result is put in square
brackets and appended to the FITS file name before calling CFITSIO to read the
contents of the HDU for all the programs in Gnuastro.
-@item
-@file{.gnuastro/astprogname.conf} is parsed by ProgramName in the current
directory.
+@item -s STR
+@itemx --searchin=STR
+Where to match/search for columns when the column identifier wasn't a number,
see @ref{Selecting table columns}.
+The acceptable values are @command{name}, @command{unit}, or @command{comment}.
+This option is only relevant for programs that take table columns as input.
-@item
-@file{.gnuastro/gnuastro.conf} is parsed by all Gnuastro programs in the
current directory.
+@item -I
+@itemx --ignorecase
+Ignore case while matching/searching column meta-data (in the field specified
by the @option{--searchin}).
+The FITS standard suggests to treat the column names as case insensitive,
which is strongly recommended here also but is not enforced.
+This option is only relevant for programs that take table columns as input.
-@item
-@file{$HOME/.local/etc/astprogname.conf} is parsed by ProgramName in the
user's home directory (see @ref{Current directory and User wide}).
+This option is not relevant to @ref{BuildProgram}, hence in that program the
short option @option{-I} is used for include directories, not to ignore case.
-@item
-@file{$HOME/.local/etc/gnuastro.conf} is parsed by all Gnuastro programs in
the user's home directory (see @ref{Current directory and User wide}).
+@item -o STR
+@itemx --output=STR
+The name of the output file or directory. With this option the automatic
output names explained in @ref{Automatic output} are ignored.
-@item
-@file{prefix/etc/astprogname.conf} is parsed by ProgramName in the system-wide
installation directory (see @ref{System wide} for @file{prefix}).
+@item -T STR
+@itemx --type=STR
+The data type of the output depending on the program context.
+This option isn't applicable to some programs like @ref{Fits} and will be
ignored by them.
+The different acceptable values to this option are fully described in
@ref{Numeric data types}.
-@item
-@file{prefix/etc/gnuastro.conf} is parsed by all Gnuastro programs in the
system-wide installation directory (see @ref{System wide} for @file{prefix}).
+@item -D
+@itemx --dontdelete
+By default, if the output file already exists, Gnuastro's programs will
silently delete it and put their own outputs in its place.
+When this option is activated, if the output file already exists, the programs
will not delete it, will warn you, and will abort.
-@end enumerate
+@item -K
+@itemx --keepinputdir
+In automatic output names, don't remove the directory information of the input
file names.
+As explained in @ref{Automatic output}, if no output name is specified (with
@option{--output}), then the output name will be made in the existing directory
based on your input's file name (ignoring the directory of the input).
+If you call this option, the directory information of the input will be kept
and the automatically generated output name will be in the same directory as
the input (usually with a suffix added).
+Note that his is only relevant if you are running the program in a different
directory than the input data.
-The basic idea behind setting this progressive state of checking for parameter
values is that separate users of a computer or separate folders in a user's
file system might need different values for some parameters.
+@item -t STR
+@itemx --tableformat=STR
+The output table's type.
+This option is only relevant when the output is a table and its format cannot
be deduced from its filename.
+For example, if a name ending in @file{.fits} was given to @option{--output},
then the program knows you want a FITS table.
+But there are two types of FITS tables: FITS ASCII, and FITS binary.
+Thus, with this option, the program is able to identify which type you want.
+The currently recognized values to this option are:
-@cartouche
-@noindent
-@strong{Checking the order:}
-You can confirm/check the order of parsing configuration files using the
@option{--checkconfig} option with any Gnuastro program, see @ref{Operating
mode options}.
-Just be sure to place this option immediately after the program name, before
any other option.
-@end cartouche
+@item --wcslinearmatrix=STR
+Select the linear transformation matrix of the output's WCS.
+This option only takes two values: @code{pc} (for the @code{PCi_j} formalism)
and @code{cd} (for @code{CDi_j}).
+For more on the different formalisms, please see Section 8.1 of the FITS
standard@footnote{@url{https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf}},
version 4.0.
-As you see above, there can also be a configuration file containing the common
options in all the programs: @file{gnuastro.conf} (see @ref{Common options}).
-If options specific to one program are specified in this file, there will be
unrecognized option errors, or unexpected behavior if the option has different
behavior in another program.
-On the other hand, there is no problem with @file{astprogname.conf} containing
common options@footnote{As an example, the @option{--setdirconf} and
@option{--setusrconf} options will also write the common options they have read
in their produced @file{astprogname.conf}.}.
-
-@cartouche
-@noindent
-@strong{Manipulating the order:} You can manipulate this order or add new
files with the following two options which are fully described in
-@ref{Operating mode options}:
-@table @option
-@item --config
-Allows you to define any file to be parsed as a configuration file on the
command-line or within the any other configuration file.
-Recall that the file given to @option{--config} is parsed immediately when
this option is confronted (on the command-line or in a configuration file).
+@cindex @code{CDELT}
+In short, in the @code{PCi_j} formalism, we only keep the linear rotation
matrix in these keywords and put the scaling factor (or the pixel scale in
astronomical imaging) in the @code{CDELTi} keywords.
+In the @code{CDi_j} formalism, we blend the scaling into the rotation into a
single matrix and keep that matrix in these FITS keywords.
+By default, Gnuastro uses the @code{PCi_j} formalism, because it greatly helps
in human readability of the raw keywords and is also the default mode of WCSLIB.
+However, in some circumstances it may be necessary to have the keywords in the
CD format; for example when you need to feed the outputs into other software
that don't follow the full FITS standard and only recognize the @code{CDi_j}
formalism.
-@item --lastconfig
-Allows you to stop the parsing of subsequent configuration files.
-Note that if this option is given in a configuration file, it will be fully
read, so its position in the configuration doesn't matter (unlike
@option{--config}).
+@table @command
+@item txt
+A plain text table with white-space characters between the columns (see
+@ref{Gnuastro text table format}).
+@item fits-ascii
+A FITS ASCII table (see @ref{Recognized table formats}).
+@item fits-binary
+A FITS binary table (see @ref{Recognized table formats}).
@end table
-@end cartouche
-One example of benefiting from these configuration files can be this: raw
telescope images usually have their main image extension in the second FITS
extension, while processed FITS images usually only have one extension.
-If your system-wide default input extension is 0 (the first), then when you
want to work with the former group of data you have to explicitly mention it to
the programs every time.
-With this progressive state of default values to check, you can set different
default values for the different directories that you would like to run
Gnuastro in for your different purposes, so you won't have to worry about this
issue any more.
+@end vtable
-The same can be said about the @file{gnuastro.conf} files: by specifying a
behavior in this single file, all Gnuastro programs in the respective
directory, user, or system-wide steps will behave similarly.
-For example to keep the input's directory when no specific output is given
(see @ref{Automatic output}), or to not delete an existing file if it has the
same name as a given output (see @ref{Input output options}).
+@node Processing options, Operating mode options, Input output options, Common
options
+@subsubsection Processing options
-@node Current directory and User wide, System wide, Configuration file
precedence, Configuration files
-@subsection Current directory and User wide
+Some processing steps are common to several programs, so they are defined as
common options to all programs.
+Note that this class of common options is thus necessarily less common between
all the programs than those described in @ref{Input output options}, or
@ref{Operating mode options} options.
+Also, if they are irrelevant for a program, these options will not display in
the @option{--help} output of the program.
-@cindex @file{$HOME}
-@cindex @file{./.gnuastro/}
-@cindex @file{$HOME/.local/etc/}
-For the current (local) and user-wide directories, the configuration files are
stored in the hidden sub-directories named @file{.gnuastro/} and
@file{$HOME/.local/etc/} respectively.
-Unless you have changed it, the @file{$HOME} environment variable should point
to your home directory.
-You can check it by running @command{$ echo $HOME}.
-Each time you run any of the programs in Gnuastro, this environment variable
is read and placed in the above address.
-So if you suddenly see that your home configuration files are not being read,
probably you (or some other program) has changed the value of this environment
variable.
+@table @option
-@vindex --setdirconf
-@vindex --setusrconf
-Although it might cause confusions like above, this dependence on the
@file{HOME} environment variable enables you to temporarily use a different
directory as your home directory.
-This can come in handy in complicated situations.
-To set the user or current directory configuration files based on your
command-line input, you can use the @option{--setdirconf} or
@option{--setusrconf}, see @ref{Operating mode options}.
+@item --minmapsize=INT
+The minimum size (in bytes) to memory-map a processing/internal array as a
file (on the non-volatile HDD/SSD), and not use the system's RAM.
+Before using this option, please read @ref{Memory management}.
+By default processing arrays will only be memory-mapped to a file when the RAM
is full.
+With this option, you can force the memory-mapping, even when there is enough
RAM.
+To ensure this default behavior, the pre-defined value to this option is an
extremely large value (larger than any existing RAM).
+Please note that using a non-volatile file (in the HDD/SDD) instead of RAM can
significantly increase the program's running time, especially on HDDs (where
read/write is slower).
+Also, note that the number of memory-mapped files that your kernel can support
is limited.
+So when this option is necessary, it is best to give it values larger than 1
megabyte (@option{--minmapsize=1000000}).
+You can then decrease it for a specific program's invocation on a large input
after you see memory issues arise (for example an error, or the program not
aborting and fully consuming your memory).
+If you see randomly named files remaining in this directory when the program
finishes normally, please send us a bug report so we address the problem, see
@ref{Report a bug}.
+@cartouche
+@noindent
+@strong{Limited number of memory-mapped files:} The operating system kernels
usually support a limited number of memory-mapped files.
+Therefore never set @code{--minmapsize} to zero or a small number of bytes (so
too many files are created).
+If the kernel capacity is exceeded, the program will crash.
+@end cartouche
-@node System wide, , Current directory and User wide, Configuration files
-@subsection System wide
+@item --quietmmap
+Don't print any message when an array is stored in non-volatile memory
+(HDD/SSD) and not RAM, see the description of @option{--minmapsize} (above)
+for more.
-@cindex @file{prefix/etc/}
-@cindex System wide configuration files
-@cindex Configuration files, system wide
-When Gnuastro is installed, the configuration files that are shipped with the
distribution are copied into the (possibly system wide) @file{prefix/etc/}
directory.
-For more details on @file{prefix}, see @ref{Installation directory} (by
default it is: @file{/usr/local}).
-This directory is the final place (with the lowest priority) that the programs
in Gnuastro will check to retrieve parameter values.
+@item -Z INT[,INT[,...]]
+@itemx --tilesize=[,INT[,...]]
+The size of regular tiles for tessellation, see @ref{Tessellation}.
+For each dimension an integer length (in units of data-elements or pixels) is
necessary.
+If the number of input dimensions is different from the number of values given
to this option, the program will stop with an error.
+Values must be separated by commas (@key{,}) and can also be fractions (for
example @code{4/2}).
+If they are fractions, the result must be an integer, otherwise an error will
be printed.
-If you remove an option and its value from the system wide configuration
files, you either have to specify it in more immediate configuration files or
set it each time in the command-line.
-Recall that none of the programs in Gnuastro keep any internal default values
and will abort if they don't find a value for the necessary parameters (except
the number of threads and output file name).
-So even though you might never expect to use an optional option, it safe to
have it available in this system-wide configuration file even if you don't
intend to use it frequently.
+@item -M INT[,INT[,...]]
+@itemx --numchannels=INT[,INT[,...]]
+The number of channels for larger input tessellation, see @ref{Tessellation}.
+The number and types of acceptable values are similar to @option{--tilesize}.
+The only difference is that instead of length, the integers values given to
this option represent the @emph{number} of channels, not their size.
-Note that in case you install Gnuastro from your distribution's repositories,
@file{prefix} will either be set to @file{/} (the root directory) or
@file{/usr}, so you can find the system wide configuration variables in
@file{/etc/} or @file{/usr/etc/}.
-The prefix of @file{/usr/local/} is conventionally used for programs you
install from source by your self as in @ref{Quick start}.
+@item -F FLT
+@itemx --remainderfrac=FLT
+The fraction of remainder size along all dimensions to add to the first tile.
+See @ref{Tessellation} for a complete description.
+This option is only relevant if @option{--tilesize} is not exactly divisible
by the input dataset's size in a dimension.
+If the remainder size is larger than this fraction (compared to
@option{--tilesize}), then the remainder size will be added with one regular
tile size and divided between two tiles at the start and end of the given
dimension.
+@item --workoverch
+Ignore the channel borders for the high-level job of the given application.
+As a result, while the channel borders are respected in defining the small
tiles (such that no tile will cross a channel border), the higher-level program
operation will ignore them, see @ref{Tessellation}.
+@item --checktiles
+Make a FITS file with the same dimensions as the input but each pixel is
replaced with the ID of the tile that it is associated with.
+Note that the tile IDs start from 0.
+See @ref{Tessellation} for more on Tiling an image in Gnuastro.
+@item --oneelempertile
+When showing the tile values (for example with @option{--checktiles}, or when
the program's output is tessellated) only use one element for each tile.
+This can be useful when only the relative values given to each tile compared
to the rest are important or need to be checked.
+Since the tiles usually have a large number of pixels within them the output
will be much smaller, and so easier to read, write, store, or send.
+Note that when the full input size in any dimension is not exactly divisible
by the given @option{--tilesize} in that dimension, the edge tile(s) will have
different sizes (in units of the input's size), see @option{--remainderfrac}.
+But with this option, all displayed values are going to have the (same) size
of one data-element.
+Hence, in such cases, the image proportions are going to be slightly different
with this option.
+If your input image is not exactly divisible by the tile size and you want one
value per tile for some higher-level processing, all is not lost though.
+You can see how many pixels were within each tile (for example to weight the
values or discard some for later processing) with Gnuastro's Statistics (see
@ref{Statistics}) as shown below.
+The output FITS file is going to have two extensions, one with the median
calculated on each tile and one with the number of elements that each tile
covers.
+You can then use the @code{where} operator in @ref{Arithmetic} to set the
values of all tiles that don't have the regular area to a blank value.
+@example
+$ aststatistics --median --number --ontile input.fits \
+ --oneelempertile --output=o.fits
+$ REGULAR_AREA=1600 # Check second extension of `o.fits'.
+$ astarithmetic o.fits o.fits $REGULAR_AREA ne nan where \
+ -h1 -h2
+@end example
+Note that if @file{input.fits} also has blank values, then the median on
+tiles with blank values will also be ignored with the command above (which
+is desirable).
+@item --inteponlyblank
+When values are to be interpolated, only change the values of the blank
+elements, keep the non-blank elements untouched.
-@node Getting help, Multi-threaded operations, Configuration files, Common
program behavior
-@section Getting help
+@item --interpmetric=STR
+@cindex Radial metric
+@cindex Taxicab metric
+@cindex Manhattan metric
+@cindex Metric: Manhattan, Taxicab, Radial
+The metric to use for finding nearest neighbors.
+Currently it only accepts the Manhattan (or taxicab) metric with
@code{manhattan}, or the radial metric with @code{radial}.
-@cindex Help
-@cindex Book formats
-@cindex Remembering options
-@cindex Convenient book formats
-Probably the first time you read this book, it is either in the PDF or HTML
formats.
-These two formats are very convenient for when you are not actually working,
but when you are only reading.
-Later on, when you start to use the programs and you are deep in the middle of
your work, some of the details will inevitably be forgotten.
-Going to find the PDF file (printed or digital) or the HTML web page is a
major distraction.
+The Manhattan distance between two points is defined with
@mymath{|\Delta{x}|+|\Delta{y}|}.
+Thus the Manhattan metric has the advantage of being fast, but at the expense
of being less accurate.
+The radial distance is the standard definition of distance in a Euclidean
space: @mymath{\sqrt{\Delta{x}^2+\Delta{y}^2}}.
+It is accurate, but the multiplication and square root can slow down the
processing.
-@cindex Online help
-@cindex Command-line help
-GNU software have a very unique set of tools for aiding your memory on the
command-line, where you are working, depending how much of it you need to
remember.
-In the past, such command-line help was known as ``online'' help, because they
were literally provided to you `on' the command `line'.
-However, nowadays the word ``online'' refers to something on the internet, so
that term will not be used.
-With this type of help, you can resume your exciting research without taking
your hands off the keyboard.
+@item --interpnumngb=INT
+The number of nearby non-blank neighbors to use for interpolation.
+@end table
-@cindex Installed help methods
-Another major advantage of such command-line based help routines is that they
are installed with the software in your computer, therefore they are always in
sync with the executable you are actually running.
-Three of them are actually part of the executable.
-You don't have to worry about the version of the book or program.
-If you rely on external help (a PDF in your personal print or digital archive
or HTML from the official web page) you have to check to see if their versions
fit with your installed program.
+@node Operating mode options, , Processing options, Common options
+@subsubsection Operating mode options
-If you only need to remember the short or long names of the options,
@option{--usage} is advised.
-If it is what the options do, then @option{--help} is a great tool.
-Man pages are also provided for those who are use to this older system of
documentation.
-This full book is also available to you on the command-line in Info format.
-If none of these seems to resolve the problems, there is a mailing list which
enables you to get in touch with experienced Gnuastro users.
-In the subsections below each of these methods are reviewed.
+Another group of options that are common to all the programs in Gnuastro are
those to do with the general operation of the programs.
+The explanation for those that are not only limited to Gnuastro but are common
to all GNU programs start with (GNU option).
+@vtable @option
-@menu
-* --usage:: View option names and value formats.
-* --help:: List all options with description.
-* Man pages:: Man pages generated from --help.
-* Info:: View complete book in terminal.
-* help-gnuastro mailing list:: Contacting experienced users.
-@end menu
+@item --
+(GNU option) Stop parsing the command-line.
+This option can be useful in scripts or when using the shell history.
+Suppose you have a long list of options, and want to see if removing some of
them (to read from configuration files, see @ref{Configuration files}) can give
a better result.
+If the ones you want to remove are the last ones on the command-line, you
don't have to delete them, you can just add @option{--} before them and if you
don't get what you want, you can remove the @option{--} and get the same
initial result.
-@node --usage, --help, Getting help, Getting help
-@subsection @option{--usage}
-@vindex --usage
-@cindex Usage pattern
-@cindex Mandatory arguments
-@cindex Optional and mandatory tokens
-If you give this option, the program will not run.
-It will only print a very concise message showing the options and arguments.
-Everything within square brackets (@option{[]}) is optional.
-For example here are the first and last two lines of Crop's @option{--usage}
is shown:
+@item --usage
+(GNU option) Only print the options and arguments and abort.
+This is very useful for when you know the what the options do, and have just
forgot their long/short identifiers, see @ref{--usage}.
-@example
-$ astcrop --usage
-Usage: astcrop [-Do?IPqSVW] [-d INT] [-h INT] [-r INT] [-w INT]
- [-x INT] [-y INT] [-c INT] [-p STR] [-N INT] [--deccol=INT]
- ....
- [--setusrconf] [--usage] [--version] [--wcsmode]
- [ASCIIcatalog] FITSimage(s).fits
-@end example
+@item -?
+@itemx --help
+(GNU option) Print all options with an explanation and abort.
+Adding this option will print all the options in their short and long formats,
also displaying which ones need a value if they are called (with an @option{=}
after the long format followed by a string specifying the format, see
@ref{Options}).
+A short explanation is also given for what the option is for.
+The program will quit immediately after the message is printed and will not do
any form of processing, see @ref{--help}.
-There are no explanations on the options, just their short and long names
shown separately.
-After the program name, the short format of all the options that don't require
a value (on/off options) is displayed.
-Those that do require a value then follow in separate brackets, each
displaying the format of the input they want, see @ref{Options}.
-Since all options are optional, they are shown in square brackets, but
arguments can also be optional.
-For example in this example, a catalog name is optional and is only required
in some modes.
-This is a standard method of displaying optional arguments for all GNU
software.
+@item -V
+@itemx --version
+(GNU option) Print a short message, showing the full name, version, copyright
information and program authors and abort.
+On the first line, it will print the official name (not executable name) and
version number of the program.
+Following this is a blank line and a copyright information.
+The program will not run.
-@node --help, Man pages, --usage, Getting help
-@subsection @option{--help}
+@item -q
+@itemx --quiet
+Don't report steps.
+All the programs in Gnuastro that have multiple major steps will report their
steps for you to follow while they are operating.
+If you do not want to see these reports, you can call this option and only
error/warning messages will be printed.
+If the steps are done very fast (depending on the properties of your input)
disabling these reports will also decrease running time.
-@vindex --help
-If the command-line includes this option, the program will not be run.
-It will print a complete list of all available options along with a short
explanation.
-The options are also grouped by their context.
-Within each context, the options are sorted alphabetically.
-Since the options are shown in detail afterwards, the first line of the
@option{--help} output shows the arguments and if they are optional or not,
similar to @ref{--usage}.
+@item --cite
+Print all necessary information to cite and acknowledge Gnuastro in your
published papers.
+With this option, the programs will print the Bib@TeX{} entry to include in
your paper for Gnuastro in general, and the particular program's paper (if that
program comes with a separate paper).
+It will also print the necessary acknowledgment statement to add in the
respective section of your paper and it will abort.
+For a more complete explanation, please see @ref{Acknowledgments}.
-In the @option{--help} output of all programs in Gnuastro, the options for
each program are classified based on context.
-The first two contexts are always options to do with the input and output
respectively.
-For example input image extensions or supplementary input files for the inputs.
-The last class of options is also fixed in all of Gnuastro, it shows operating
mode options.
-Most of these options are already explained in @ref{Operating mode options}.
+Citations and acknowledgments are vital for the continued work on Gnuastro.
+Gnuastro started, and is continued, based on separate research projects.
+So if you find any of the tools offered in Gnuastro to be useful in your
research, please use the output of this command to cite and acknowledge the
program (and Gnuastro) in your research paper.
+Thank you.
-@cindex Long outputs
-@cindex Redirection of output
-@cindex Command-line, long outputs
-The help message will sometimes be longer than the vertical size of your
terminal.
-If you are using a graphical user interface terminal emulator, you can scroll
the terminal with your mouse, but we promised no mice distractions! So here are
some suggestions:
+Gnuastro is still new, there is no separate paper only devoted to Gnuastro yet.
+Therefore currently the paper to cite for Gnuastro is the paper for
NoiseChisel which is the first published paper introducing Gnuastro to the
astronomical community.
+Upon reaching a certain point, a paper completely devoted to describing
Gnuastro's many functionalities will be published, see @ref{GNU Astronomy
Utilities 1.0}.
-@itemize
-@item
-@cindex Scroll command-line
-@cindex Command-line scroll
-@cindex @key{Shift + PageUP} and @key{Shift + PageDown}
-@key{Shift + PageUP} to scroll up and @key{Shift + PageDown} to scroll down.
-For most help output this should be enough.
-The problem is that it is limited by the number of lines that your terminal
keeps in memory and that you can't scroll by lines, only by whole screens.
+@item -P
+@itemx --printparams
+With this option, Gnuastro's programs will read your command-line options and
all the configuration files.
+If there is no problem (like a missing parameter or a value in the wrong
format or range) and immediately before actually running, the programs will
print the full list of option names, values and descriptions, sorted and
grouped by context and abort.
+They will also report the version number, the date they were configured on
your system and the time they were reported.
-@item
-@cindex Pipe
-@cindex @command{less}
-Pipe to @command{less}.
-A pipe is a form of shell re-direction.
-The @command{less} tool in Unix-like systems was made exactly for such outputs
of any length.
-You can pipe (@command{|}) the output of any program that is longer than the
screen to it and then you can scroll through (up and down) with its many tools.
-For example:
-@example
-$ astnoisechisel --help | less
-@end example
-@noindent
-Once you have gone through the text, you can quit @command{less} by pressing
the @key{q} key.
+As an example, you can give your full command-line options and even the input
and output file names and finally just add @option{-P} to check if all the
parameters are finely set.
+If everything is OK, you can just run the same command (easily retrieved from
the shell history, with the top arrow key) and simply remove the last two
characters that showed this option.
+No program will actually start its processing when this option is called.
+The otherwise mandatory arguments for each program (for example input image or
catalog files) are no longer required when you call this option.
-@item
-@cindex Save output to file
-@cindex Redirection of output
-Redirect to a file.
-This is a less convenient way, because you will then have to open the file in
a text editor!
-You can do this with the shell redirection tool (@command{>}):
-@example
-$ astnoisechisel --help > filename.txt
-@end example
-@end itemize
+@item --config=STR
+Parse @option{STR} as a configuration file name, immediately when this option
is confronted (see @ref{Configuration files}).
+The @option{--config} option can be called multiple times in one run of any
Gnuastro program on the command-line or in the configuration files.
+In any case, it will be immediately read (before parsing the rest of the
options on the command-line, or lines in a configuration file).
+If the given file doesn't exist or can't be read for any reason, the program
will print a warning and continue its processing.
+The warning can be suppressed with @option{--quiet}.
-@cindex GNU Grep
-@cindex Searching text
-@cindex Command-line searching text
-In case you have a special keyword you are looking for in the help, you don't
have to go through the full list.
-GNU Grep is made for this job.
-For example if you only want the list of options whose @option{--help} output
contains the word ``axis'' in Crop, you can run the following command:
+Note that by definition, options on the command-line still take precedence
over those in any configuration file, including the file(s) given to this
option if they are called before it.
+Also see @option{--lastconfig} and @option{--onlyversion} on how this option
can be used for reproducible results.
+You can use @option{--checkconfig} (below) to check/confirm the parsing of
configuration files.
-@example
-$ astcrop --help | grep axis
-@end example
+@item --checkconfig
+Print options and their values, within the command-line or configuration
files, as they are parsed (see @ref{Configuration file precedence}).
+If an option has already been set, or is ignored by the program, this option
will also inform you with special values like @code{--ALREADY-SET--}.
+Only options that are parsed after this option are printed, so to see the
parsing of all input options, it is recommended to put this option immediately
after the program name before any other options.
-@cindex @code{ARGP_HELP_FMT}
-@cindex Argp argument parser
-@cindex Customize @option{--help} output
-@cindex @option{--help} output customization
-If the output of this option does not fit nicely within the confines of your
terminal, GNU does enable you to customize its output through the environment
variable @code{ARGP_HELP_FMT}, you can set various parameters which specify the
formatting of the help messages.
-For example if your terminals are wider than 70 spaces (say 100) and you feel
there is too much empty space between the long options and the short
explanation, you can change these formats by giving values to this environment
variable before running the program with the @option{--help} output.
-You can define this environment variable in this manner:
-@example
-$ export ARGP_HELP_FMT=rmargin=100,opt-doc-col=20
-@end example
-@cindex @file{.bashrc}
-This will affect all GNU programs using GNU C library's @file{argp.h}
facilities as long as the environment variable is in memory.
-You can see the full list of these formatting parameters in the ``Argp User
Customization'' part of the GNU C library manual.
-If you are more comfortable to read the @option{--help} outputs of all GNU
software in your customized format, you can add your customization (similar to
the line above, without the @command{$} sign) to your @file{~/.bashrc} file.
-This is a standard option for all GNU software.
+@cindex Debug
+This is a very good option to confirm where the value of each option is has
been defined in scenarios where there are multiple configuration files (for
debugging).
-@node Man pages, Info, --help, Getting help
-@subsection Man pages
-@cindex Man pages
-Man pages were the Unix method of providing command-line documentation to a
program.
-With GNU Info, see @ref{Info} the usage of this method of documentation is
highly discouraged.
-This is because Info provides a much more easier to navigate and read
environment.
+@item -S
+@itemx --setdirconf
+Update the current directory configuration file for the Gnuastro program and
quit.
+The full set of command-line and configuration file options will be parsed and
options with a value will be written in the current directory configuration
file for this program (see @ref{Configuration files}).
+If the configuration file or its directory doesn't exist, it will be created.
+If a configuration file exists it will be replaced (after it, and all other
configuration files have been read).
+In any case, the program will not run.
-However, some operating systems require a man page for packages that are
installed and some people are still used to this method of command line help.
-So the programs in Gnuastro also have Man pages which are automatically
generated from the outputs of @option{--version} and @option{--help} using the
GNU help2man program.
-So if you run
-@example
-$ man programname
-@end example
-@noindent
-You will be provided with a man page listing the options in the
-standard manner.
+This is the recommended method@footnote{Alternatively, you can use your
favorite text editor.} to edit/set the configuration file for all future calls
to Gnuastro's programs.
+It will internally check if your values are in the correct range and type and
save them according to the configuration file format, see @ref{Configuration
file format}.
+So if there are unreasonable values to some options, the program will notify
you and abort before writing the final configuration file.
+
+When this option is called, the otherwise mandatory arguments, for
+example input image or catalog file(s), are no longer mandatory (since
+the program will not run).
+@item -U
+@itemx --setusrconf
+Update the user configuration file and quit (see @ref{Configuration files}).
+See explanation under @option{--setdirconf} for more details.
+@item --lastconfig
+This is the last configuration file that must be read.
+When this option is confronted in any stage of reading the options (on the
command-line or in a configuration file), no other configuration file will be
parsed, see @ref{Configuration file precedence} and @ref{Current directory and
User wide}.
+Like all on/off options, on the command-line, this option doesn't take any
values.
+But in a configuration file, it takes the values of @option{0} or @option{1},
see @ref{Configuration file format}.
+If it is present in a configuration file with a value of @option{0}, then all
later occurrences of this option will be ignored.
+@item --onlyversion=STR
+Only run the program if Gnuastro's version is exactly equal to @option{STR}
(see @ref{Version numbering}).
+Note that it is not compared as a number, but as a string of characters, so
@option{0}, or @option{0.0} and @option{0.00} are different.
+If the running Gnuastro version is different, then this option will report an
error and abort as soon as it is confronted on the command-line or in a
configuration file.
+If the running Gnuastro version is the same as @option{STR}, then the program
will run as if this option was not called.
-@node Info, help-gnuastro mailing list, Man pages, Getting help
-@subsection Info
+This is useful if you want your results to be exactly reproducible and not
mistakenly run with an updated/newer or older version of the program.
+Besides internal algorithmic/behavior changes in programs, the existence of
options or their names might change between versions (especially in these
earlier versions of Gnuastro).
-@cindex GNU Info
-@cindex Command-line, viewing full book
-Info is the standard documentation format for all GNU software.
-It is a very useful command-line document viewing format, fully equipped with
links between the various pages and menus and search capabilities.
-As explained before, the best thing about it is that it is available for you
the moment you need to refresh your memory on any command-line tool in the
middle of your work without having to take your hands off the keyboard.
-This complete book is available in Info format and can be accessed from
anywhere on the command-line.
+Hence, when using this option (probably in a script or in a configuration
file), be sure to call it before other options.
+The benefit is that, when the version differs, the other options won't be
parsed and you, or your collaborators/users, won't get errors saying an option
in your configuration doesn't exist in the running version of the program.
-To open the Info format of any installed programs or library on your system
which has an Info format book, you can simply run the command below (change
@command{executablename} to the executable name of the program or library):
+Here is one example of how this option can be used in conjunction with the
@option{--lastconfig} option.
+Let's assume that you were satisfied with the results of this command:
@command{astnoisechisel image.fits --snquant=0.95} (along with various options
set in various configuration files).
+You can save the state of NoiseChisel and reproduce that exact result on
@file{image.fits} later by following these steps (the extra spaces, and
@key{\}, are only for easy readability, if you want to try it out, only one
space between each token is enough).
@example
-$ info executablename
+$ echo "onlyversion X.XX" > reproducible.conf
+$ echo "lastconfig 1" >> reproducible.conf
+$ astnoisechisel image.fits --snquant=0.95 -P \
+ >> reproducible.conf
@end example
-@noindent
-@cindex Learning GNU Info
-@cindex GNU software documentation
-In case you are not already familiar with it, run @command{$ info info}.
-It does a fantastic job in explaining all its capabilities its self.
-It is very short and you will become sufficiently fluent in about half an hour.
-Since all GNU software documentation is also provided in Info, your whole
GNU/Linux life will significantly improve.
-
-@cindex GNU Emacs
-@cindex GNU C library
-Once you've become an efficient navigator in Info, you can go to any part of
this book or any other GNU software or library manual, no matter how long it
is, in a matter of seconds.
-It also blends nicely with GNU Emacs (a text editor) and you can search
manuals while you are writing your document or programs without taking your
hands off the keyboard, this is most useful for libraries like the GNU C
library.
-To be able to access all the Info manuals installed in your GNU/Linux within
Emacs, type @key{Ctrl-H + i}.
+@option{--onlyversion} was available from Gnuastro 0.0, so putting it
immediately at the start of a configuration file will ensure that later, you
(or others using different version) won't get a non-recognized option error in
case an option was added/removed.
+@option{--lastconfig} will inform the installed NoiseChisel to not parse any
other configuration files.
+This is done because we don't want the user's user-wide or system wide option
values affecting our results.
+Finally, with the third command, which has a @option{-P} (short for
@option{--printparams}), NoiseChisel will print all the option values visible
to it (in all the configuration files) and the shell will append them to
@file{reproduce.conf}.
+Hence, you don't have to worry about remembering the (possibly) different
options in the different configuration files.
-To see this whole book from the beginning in Info, you can run
+Afterwards, if you run NoiseChisel as shown below (telling it to read this
configuration file with the @file{--config} option).
+You can be sure that there will either be an error (for version mismatch) or
it will produce exactly the same result that you got before.
@example
-$ info gnuastro
+$ astnoisechisel --config=reproducible.conf
@end example
-@noindent
-If you run Info with the particular program executable name, for
-example @file{astcrop} or @file{astnoisechisel}:
-
-@example
-$ info astprogramname
-@end example
+@item --log
+Some programs can generate extra information about their outputs in a log file.
+When this option is called in those programs, the log file will also be
printed.
+If the program doesn't generate a log file, this option is ignored.
+@cartouche
@noindent
-you will be taken to the section titled ``Invoking ProgramName'' which
explains the inputs and outputs along with the command-line options for that
program.
-Finally, if you run Info with the official program name, for example Crop or
NoiseChisel:
+@strong{@option{--log} isn't thread-safe}: The log file usually has a fixed
name.
+Therefore if two simultaneous calls (with @option{--log}) of a program are
made in the same directory, the program will try to write to he same file.
+This will cause problems like unreasonable log file, undefined behavior, or a
crash.
+@end cartouche
-@example
-$ info ProgramName
-@end example
+@cindex CPU threads, set number
+@cindex Number of CPU threads to use
+@item -N INT
+@itemx --numthreads=INT
+Use @option{INT} CPU threads when running a Gnuastro program (see
@ref{Multi-threaded operations}).
+If the value is zero (@code{0}), or this option is not given on the
command-line or any configuration file, the value will be determined at
run-time: the maximum number of threads available to the system when you run a
Gnuastro program.
-@noindent
-you will be taken to the top section which introduces the program.
-Note that in all cases, Info is not case sensitive.
+Note that multi-threaded programming is only relevant to some programs.
+In others, this option will be ignored.
+@end vtable
-@node help-gnuastro mailing list, , Info, Getting help
-@subsection help-gnuastro mailing list
-@cindex help-gnuastro mailing list
-@cindex Mailing list: help-gnuastro
-Gnuastro maintains the help-gnuastro mailing list for users to ask any
questions related to Gnuastro.
-The experienced Gnuastro users and some of its developers are subscribed to
this mailing list and your email will be sent to them immediately.
-However, when contacting this mailing list please have in mind that they are
possibly very busy and might not be able to answer immediately.
-@cindex Mailing list archives
-@cindex @code{help-gnuastro@@gnu.org}
-To ask a question from this mailing list, send a mail to
@code{help-gnuastro@@gnu.org}.
-Anyone can view the mailing list archives at
@url{http://lists.gnu.org/archive/html/help-gnuastro/}.
-It is best that before sending a mail, you search the archives to see if
anyone has asked a question similar to yours.
-If you want to make a suggestion or report a bug, please don't send a mail to
this mailing list.
-We have other mailing lists and tools for those purposes, see @ref{Report a
bug} or @ref{Suggest new feature}.
+@node Shell TAB completion, Standard input, Common options, Command-line
+@subsection Shell TAB completion (highly customized)
+@cartouche
+@noindent
+@strong{Under development:} Gnuastro's TAB completion in Bash already greatly
improves usage of Gnuastro on the command-line, but still under development and
not yet complete.
+If you are interested to try it out, please go ahead and activate it (as
described below), we encourage this.
+But please have in mind that there are known
issues@footnote{@url{http://savannah.gnu.org/bugs/index.php?group=gnuastro&category_id=128}}
and you may find new issues.
+If you do, please get in touch with us as described in @ref{Report a bug}.
+TAB completion is currently only implemented in the following programs:
Arithmetic, BuildProgram, ConvertType, Convolve, CosmicCalculator, Crop, Fits
and Table.
+For progress on this task, please see Task
15799@footnote{@url{https://savannah.gnu.org/task/?15799}}.
+@end cartouche
+@cindex Bash auto-complete
+@cindex Completion in the shell
+@cindex Bash programmable completion
+@cindex Autocomplete (in the shell/Bash)
+Bash provides a built-in feature called @emph{programmable
completion}@footnote{@url{https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion.html}}
to help increase interactive workflow efficiency and minimize the number of
key-strokes @emph{and} the need to memorize things.
+It is also known as TAB completion, bash completion, auto-completion, or word
completion.
+Completion is activated by pressing @key{[TAB]} while you are typing a command.
+For file arguments this is the default behavior already and you have probably
used it a lot with any command-line program.
+Besides this simple/default mode, Bash also enables a high level of
customization features for its completion.
+These features have been extensively used in Gnuastro to improve your work
efficiency@footnote{To learn how Gnuastro implements TAB completion in Bash,
see @ref{Bash programmable completion}.}.
+For example if you are running @code{asttable} (which only accepts files
containing a table), and you press @key{[TAB]}, it will only suggest files
containing tables.
+As another example, if an option needs image HDUs within a FITS file, pressing
@key{[TAB]} will only suggest the image HDUs (and not other possibly existing
HDUs that contain tables, or just metadata).
+Just note that the file name has to be already given on the command-line
before reaching such options (that look into the contents of a file).
+But TAB completion is not limited to file types or contents.
+Arguments/Options that take certain fixed string values will directly suggest
those strings with TAB, and completely ignore the file structure (for example
spectral line names in @ref{Invoking astcosmiccal})!
+As another example, the option @option{--numthreads} option (to specify the
number of threads to use by the program), will find the number of available
threads on the system, and suggest the possible numbers with a TAB!
-@node Multi-threaded operations, Numeric data types, Getting help, Common
program behavior
-@section Multi-threaded operations
-
-@pindex nproc
-@cindex pthread
-@cindex CPU threads
-@cindex GNU Coreutils
-@cindex Using CPU threads
-@cindex CPU, using all threads
-@cindex Multi-threaded programs
-@cindex Using multiple CPU cores
-@cindex Simultaneous multithreading
-Some of the programs benefit significantly when you use all the threads your
computer's CPU has to offer to your operating system.
-The number of threads available can be larger than the number of physical
(hardware) cores in the CPU (also known as Simultaneous multithreading).
-For example, in Intel's CPUs (those that implement its Hyper-threading
technology) the number of threads is usually double the number of physical
cores in your CPU.
-On a GNU/Linux system, the number of threads available can be found with the
command @command{$ nproc} command (part of GNU Coreutils).
-
-@vindex --numthreads
-@cindex Number of threads available
-@cindex Available number of threads
-@cindex Internally stored option value
-Gnuastro's programs can find the number of threads available to your system
internally at run-time (when you execute the program).
-However, if a value is given to the @option{--numthreads} option, the given
number will be used, see @ref{Operating mode options} and @ref{Configuration
files} for ways to use this option.
-Thus @option{--numthreads} is the only common option in Gnuastro's programs
with a value that doesn't have to be specified anywhere on the command-line or
in the configuration files.
-
-@menu
-* A note on threads:: Caution and suggestion on using threads.
-* How to run simultaneous operations:: How to run things simultaneously.
-@end menu
-
-@node A note on threads, How to run simultaneous operations, Multi-threaded
operations, Multi-threaded operations
-@subsection A note on threads
-
-@cindex Using multiple threads
-@cindex Best use of CPU threads
-@cindex Efficient use of CPU threads
-Spinning off threads is not necessarily the most efficient way to run an
application.
-Creating a new thread isn't a cheap operation for the operating system.
-It is most useful when the input data are fixed and you want the same
operation to be done on parts of it.
-For example one input image to Crop and multiple crops from various parts of
it.
-In this fashion, the image is loaded into memory once, all the crops are
divided between the number of threads internally and each thread cuts out those
parts which are assigned to it from the same image.
-On the other hand, if you have multiple images and you want to crop the same
region(s) out of all of them, it is much more efficient to set
@option{--numthreads=1} (so no threads spin off) and run Crop multiple times
simultaneously, see @ref{How to run simultaneous operations}.
-
-@cindex Wall-clock time
-You can check the boost in speed by first running a program on one of the data
sets with the maximum number of threads and another time (with everything else
the same) and only using one thread.
-You will notice that the wall-clock time (reported by most programs at their
end) in the former is longer than the latter divided by number of physical CPU
cores (not threads) available to your operating system.
-Asymptotically these two times can be equal (most of the time they aren't).
-So limiting the programs to use only one thread and running them independently
on the number of available threads will be more efficient.
-
-@cindex System Cache
-@cindex Cache, system
-Note that the operating system keeps a cache of recently processed data, so
usually, the second time you process an identical data set (independent of the
number of threads used), you will get faster results.
-In order to make an unbiased comparison, you have to first clean the system's
cache with the following command between the two runs.
+To activate Gnuastro's custom TAB completion in Bash, you need to put the
following line in one of your Bash startup files (for example @file{~/.bashrc}).
+If you installed Gnuastro using the steps of @ref{Quick start}, you should
have already done this (the command just after @command{sudo make install}).
+For a list of (and discussion on) Bash startup files and installation
directories see @ref{Installation directory}.
+Of course, if Gnuastro was installed in a custom location, replace the
`@file{/usr/local}' part of the line below to the value that was given to
@option{--prefix} during Gnuastro's configuration@footnote{In case you don't
know the installation directory of Gnuastro on your system, you can find out
with this command: @code{which astfits | sed -e"s|/bin/astfits||"}}.
@example
-$ sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
+# Enable Gnuastro's TAB completion
+source /usr/local/share/gnuastro/completion.bash
@end example
+After adding the line above in a Bash startup file, TAB completion will always
be activated in any new terminal.
+To see if it has been activated, try it out with @command{asttable [TAB][TAB]}
and @command{astarithmetic [TAB][TAB]} in a directory that contains tables and
images.
+The first will only suggest the files with a table, and the second, only those
with an image.
+
@cartouche
@noindent
-@strong{SUMMARY: Should I use multiple threads?} Depends:
-@itemize
-@item
-If you only have @strong{one} data set (image in most cases!), then yes, the
more threads you use (with a maximum of the number of threads available to your
OS) the faster you will get your results.
-
-@item
-If you want to run the same operation on @strong{multiple} data sets, it is
best to set the number of threads to 1 and use Make, or GNU Parallel, as
explained in @ref{How to run simultaneous operations}.
-@end itemize
+@strong{TAB completion only works with long option names:}
+As described above, short options are much more complex to generalize,
therefore TAB completion is only available for long options.
+But don't worry!
+TAB completion also involves option names, so if you just type
@option{--a[TAB][TAB]}, you will get the list of options that start with an
@option{--a}.
+Therefore as a side-effect of TAB completion, your commands will be far more
human-readable with minimal key strokes.
@end cartouche
+@node Standard input, , Shell TAB completion, Command-line
+@subsection Standard input
+@cindex Standard input
+@cindex Stream: standard input
+The most common way to feed the primary/first input dataset into a program is
to give its filename as an argument (discussed in @ref{Arguments}).
+When you want to run a series of programs in sequence, this means that each
will have to keep the output of each program in a separate file and re-type
that file's name in the next command.
+This can be very slow and frustrating (mis-typing a file's name).
+@cindex Standard output stream
+@cindex Stream: standard output
+To solve the problem, the founders of Unix defined pipes to directly feed the
output of one program (its ``Standard output'' stream) into the ``standard
input'' of a next program.
+This removes the need to make temporary files between separate processes and
became one of the best demonstrations of the Unix-way, or Unix philosophy.
-@node How to run simultaneous operations, , A note on threads, Multi-threaded
operations
-@subsection How to run simultaneous operations
+Every program has three streams identifying where it reads/writes non-file
inputs/outputs: @emph{Standard input}, @emph{Standard output}, and
@emph{Standard error}.
+When a program is called alone, all three are directed to the terminal that
you are using.
+If it needs an input, it will prompt you for one and you can type it in.
+Or, it prints its results in the terminal for you to see.
-There are two@footnote{A third way would be to open multiple terminal emulator
windows in your GUI, type the commands separately on each and press @key{Enter}
once on each terminal, but this is far too frustrating, tedious and prone to
errors.
-It's therefore not a realistic solution when tens, hundreds or thousands of
operations (your research targets, multiplied by the operations you do on each)
are to be done.} approaches to simultaneously execute a program: using GNU
Parallel or Make (GNU Make is the most common implementation).
-The first is very useful when you only want to do one job multiple times and
want to get back to your work without actually keeping the command you ran.
-The second is usually for more important operations, with lots of dependencies
between the different products (for example a full scientific research).
+For example, say you have a FITS table/catalog containing the B and V band
magnitudes (@code{MAG_B} and @code{MAG_V} columns) of a selection of galaxies
along with many other columns.
+If you want to see only these two columns in your terminal, can use Gnuastro's
@ref{Table} program like below:
-@table @asis
+@example
+$ asttable cat.fits -cMAG_B,MAG_V
+@end example
-@item GNU Parallel
-@cindex GNU Parallel
-When you only want to run multiple instances of a command on different threads
and get on with the rest of your work, the best method is to use GNU parallel.
-Surprisingly GNU Parallel is one of the few GNU packages that has no Info
documentation but only a Man page, see @ref{Info}.
-So to see the documentation after installing it please run
+Through the Unix pipe mechanism, when the shell confronts the pipe character
(@key{|}), it connects the standard output of the program before the pipe, to
the standard input of the program after it.
+So it is literally a ``pipe'': everything that you would see printed by the
first program on the command (without any pipe), is now passed to the second
program (and not seen by you).
+
+@cindex AWK
+@cindex GNU AWK
+To continue the previous example, let's say you want to see the B-V color.
+To do this, you can pipe Table's output to AWK (a wonderful tool for
processing things like plain text tables):
@example
-$ man parallel
+$ asttable cat.fits -cMAG_B,MAG_V | awk '@{print $1-$2@}'
@end example
-@noindent
-As an example, let's assume we want to crop a region fixed on the pixels (500,
600) with the default width from all the FITS images in the @file{./data}
directory ending with @file{sci.fits} to the current directory.
-To do this, you can run:
+
+But understanding the distribution by visually seeing all the numbers under
each other is not too useful! You can therefore feed this single column
information into @ref{Statistics} to give you a general feeling of the
distribution with the same command:
@example
-$ parallel astcrop --numthreads=1 --xc=500 --yc=600 ::: \
- ./data/*sci.fits
+$ asttable cat.fits -cMAG_B,MAG_V | awk '@{print $1-$2@}' | aststatistics
@end example
-@noindent
-GNU Parallel can help in many more conditions, this is one of the simplest,
see the man page for lots of other examples.
-For absolute beginners: the backslash (@command{\}) is only a line breaker to
fit nicely in the page.
-If you type the whole command in one line, you should remove it.
+Gnuastro's programs that accept input from standard input, only look into the
Standard input stream if there is no first argument.
+In other words, arguments take precedence over Standard input.
+When no argument is provided, the programs check if the standard input stream
is already full or not (output from another program is waiting to be used).
+If data is present in the standard input stream, it is used.
-@item Make
-@cindex Make
-Make is a program for building ``targets'' (e.g., files) using ``recipes'' (a
set of operations) when their known ``prerequisites'' (other files) have been
updated.
-It elegantly allows you to define dependency structures for building your
final output and updating it efficiently when the inputs change.
-It is the most common infra-structure to build software today.
+When the standard input is empty, the program will wait
@option{--stdintimeout} micro-seconds for you to manually enter the first line
(ending with a new-line character, or the @key{ENTER} key, see @ref{Input
output options}).
+If it detects the first line in this time, there is no more time limit, and
you can manually write/type all the lines for as long as it takes.
+To inform the program that Standard input has finished, press @key{CTRL-D}
after a new line.
+If the program doesn't catch the first line before the time-out finishes, it
will abort with an error saying that no input was provided.
-Scientific research methodology is very similar to software development: you
start by testing a hypothesis on a small sample of objects/targets with a
simple set of steps.
-As you are able to get promising results, you improve the method and use it on
a larger, more general, sample.
-In the process, you will confront many issues that have to be corrected (bugs
in software development jargon).
-Make a wonderful tool to manage this style of development.
-It has been used to make reproducible papers, for example see
@url{https://gitlab.com/makhlaghi/NoiseChisel-paper, the reproduction pipeline}
of the paper introducing @ref{NoiseChisel} (one of Gnuastro's programs).
+@cartouche
+@noindent
+@strong{Manual input in Standard input is discarded:}
+Be careful that when you manually fill the Standard input, the data will be
discarded once the program finishes and reproducing the result will be
impossible.
+Therefore this form of providing input is only good for temporary tests.
+@end cartouche
-@cindex GNU Make
-GNU Make@footnote{@url{https://www.gnu.org/software/make/}} is the most common
implementation which (similar to nearly all GNU programs, comes with a
wonderful manual@footnote{@url{https://www.gnu.org/software/make/manual/}}).
-Make is very basic and simple, and thus the manual is short (the most
important parts are in the first roughly 100 pages) and easy to read/understand.
+@cartouche
+@noindent
+@strong{Standard input currently only for plain text:}
+Currently Standard input only works for plain text inputs like the example
above.
+We will later allow FITS files into the programs through standard input also.
+@end cartouche
-Make comes with a @option{--jobs} (@option{-j}) option which allows you to
specify the maximum number of jobs that can be done simultaneously.
-For example if you have 8 threads available to your operating system.
-You can run:
-@example
-$ make -j8
-@end example
+@node Configuration files, Getting help, Command-line, Common program behavior
+@section Configuration files
-With this command, Make will process your @file{Makefile} and create all the
targets (can be thousands of FITS images for example) simultaneously on 8
threads, while fully respecting their dependencies (only building a file/target
when its prerequisites are successfully built).
-Make is thus strongly recommended for managing scientific research where
robustness, archiving, reproducibility and speed@footnote{Besides its
multi-threaded capabilities, Make will only re-build those targets that depend
on a change you have made, not the whole work.
-For example, if you have set the prerequisites properly, you can easily test
the changing of a parameter on your paper's results without having to re-do
everything (which is much faster).
-This allows you to be much more productive in easily checking various
ideas/assumptions of the different stages of your research and thus produce a
more robust result for your exciting science.} are important.
+@cindex @file{etc}
+@cindex Configuration files
+@cindex Necessary parameters
+@cindex Default option values
+@cindex File system Hierarchy Standard
+Each program needs a certain number of parameters to run.
+Supplying all the necessary parameters each time you run the program is very
frustrating and prone to errors.
+Therefore all the programs read the values for the necessary options you have
not given in the command line from one of several plain text files (which you
can view and edit with any text editor).
+These files are known as configuration files and are usually kept in a
directory named @file{etc/} according to the file system hierarchy
+standard@footnote{@url{http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard}}.
-@end table
+@vindex --output
+@vindex --numthreads
+@cindex CPU threads, number
+@cindex Internal default value
+@cindex Number of CPU threads to use
+The thing to have in mind is that none of the programs in Gnuastro keep any
internal default value.
+All the values must either be stored in one of the configuration files or
explicitly called in the command-line.
+In case the necessary parameters are not given through any of these methods,
the program will print a missing option error and abort.
+The only exception to this is @option{--numthreads}, whose default value is
determined at run-time using the number of threads available to your system,
see @ref{Multi-threaded operations}.
+Of course, you can still provide a default value for the number of threads at
any of the levels below, but if you don't, the program will not abort.
+Also note that through automatic output name generation, the value to the
@option{--output} option is also not mandatory on the command-line or in the
configuration files for all programs which don't rely on that value as an
input@footnote{One example of a program which uses the value given to
@option{--output} as an input is ConvertType, this value specifies the type of
the output through the value to @option{--output}, see @ref{Invoking
astconvertt}.}, see @ref{Automatic output}.
+@menu
+* Configuration file format:: ASCII format of configuration file.
+* Configuration file precedence:: Precedence of configuration files.
+* Current directory and User wide:: Local and user configuration files.
+* System wide:: System wide configuration files.
+@end menu
+@node Configuration file format, Configuration file precedence, Configuration
files, Configuration files
+@subsection Configuration file format
-@node Numeric data types, Memory management, Multi-threaded operations, Common
program behavior
-@section Numeric data types
+@cindex Configuration file suffix
+The configuration files for each program have the standard program executable
name with a `@file{.conf}' suffix.
+When you download the source code, you can find them in the same directory as
the source code of each program, see @ref{Program source}.
-@cindex Bit
-@cindex Type
-At the lowest level, the computer stores everything in terms of @code{1} or
@code{0}.
-For example, each program in Gnuastro, or each astronomical image you take
with the telescope is actually a string of millions of these zeros and ones.
-The space required to keep a zero or one is the smallest unit of storage, and
is known as a @emph{bit}.
-However, understanding and manipulating this string of bits is extremely hard
for most people.
-Therefore, different standards are defined to package the bits into separate
@emph{type}s with a fixed interpretation of the bits in each package.
+@cindex White space character
+@cindex Configuration file format
+Any line in the configuration file whose first non-white character is a
@key{#} is considered to be a comment and is ignored.
+An empty line is also similarly ignored.
+The long name of the option should be used as an identifier.
+The parameter name and parameter value have to be separated by any number of
`white-space' characters: space, tab or vertical tab.
+By default several space characters are used.
+If the value of an option has space characters (most commonly for the
@option{hdu} option), then the full value can be enclosed in double quotation
signs (@key{"}, similar to the example in @ref{Arguments and options}).
+If it is an option without a value in the @option{--help} output (on/off
option, see @ref{Options}), then the value should be @option{1} if it is to be
`on' and @option{0} otherwise.
-@cindex Byte
-@cindex Signed integer
-@cindex Unsigned integer
-@cindex Integer, Signed
-To store numbers, the most basic standard/type is for integers (@mymath{...,
-2, -1, 0, 1, 2, ...}).
-The common integer types are 8, 16, 32, and 64 bits wide (more bits will give
larger limits).
-Each bit corresponds to a power of 2 and they are summed to create the final
number.
-In the integer types, for each width there are two standards for reading the
bits: signed and unsigned.
-In the `signed' convention, one bit is reserved for the sign (stating that the
integer is positive or negative).
-The `unsigned' integers use that bit in the actual number and thus contain
only positive numbers (starting from zero).
+In each non-commented and non-blank line, any text after the first two words
(option identifier and value) is ignored.
+If an option identifier is not recognized in the configuration file, the name
of the file, the line number of the unrecognized option, and the unrecognized
identifier name will be reported and the program will abort.
+If a parameter is repeated more more than once in the configuration files,
accepts only one value, and is not set on the command-line, then only the first
value will be used, the rest will be ignored.
-Therefore, at the same number of bits, both signed and unsigned integers can
allow the same number of integers, but the positive limit of the
@code{unsigned} types is double their @code{signed} counterparts with the same
width (at the expense of not having negative numbers).
-When the context of your work doesn't involve negative numbers (for example
counting, where negative is not defined), it is best to use the @code{unsigned}
types.
-For the full numerical range of all integer types, see below.
+@cindex Writing configuration files
+@cindex Automatic configuration file writing
+@cindex Configuration files, writing
+You can build or edit any of the directories and the configuration files
yourself using any text editor.
+However, it is recommended to use the @option{--setdirconf} and
@option{--setusrconf} options to set default values for the current directory
or this user, see @ref{Operating mode options}.
+With these options, the values you give will be checked before writing in the
configuration file.
+They will also print a set of commented lines guiding the reader and will also
classify the options based on their context and write them in their logical
order to be more understandable.
-Another standard of converting a given number of bits to numbers is the
floating point standard, this standard can @emph{approximately} store any real
number with a given precision.
-There are two common floating point types: 32-bit and 64-bit, for single and
double precision floating point numbers respectively.
-The former is sufficient for data with less than 8 significant decimal digits
(most astronomical data), while the latter is good for less than 16 significant
decimal digits.
-The representation of real numbers as bits is much more complex than integers.
-If you are interested to learn more about it, you can start with the
@url{https://en.wikipedia.org/wiki/Floating_point, Wikipedia article}.
-Practically, you can use Gnuastro's Arithmetic program to convert/change the
type of an image/datacube (see @ref{Arithmetic}), or Gnuastro Table program to
convert a table column's data type (see @ref{Column arithmetic}).
-Conversion of a dataset's type is necessary in some contexts.
-For example the program/library, that you intend to feed the data into, only
accepts floating point values, but you have an integer image/column.
-Another situation that conversion can be helpful is when you know that your
data only has values that fit within @code{int8} or @code{uint16}.
-However it is currently formatted in the @code{float64} type.
-
-The important thing to consider is that operations involving wider, floating
point, or signed types can be significantly slower than smaller-width, integer,
or unsigned types respectively.
-Note that besides speed, a wider type also requires much more storage space
(by 4 or 8 times).
-Therefore, when you confront such situations that can be optimized and want to
store/archive/transfer the data, it is best to use the most efficient type.
-For example if your dataset (image or table column) only has positive integers
less than 65535, store it as an unsigned 16-bit integer for faster processing,
faster transfer, and less storage space.
-
-The short and long names for the recognized numeric data types in Gnuastro are
listed below.
-Both short and long names can be used when you want to specify a type.
-For example, as a value to the common option @option{--type} (see @ref{Input
output options}), or in the information comment lines of @ref{Gnuastro text
table format}.
-The ranges listed below are inclusive.
+@node Configuration file precedence, Current directory and User wide,
Configuration file format, Configuration files
+@subsection Configuration file precedence
-@table @code
-@item u8
-@itemx uint8
-8-bit unsigned integers, range:@*
-@mymath{[0\rm{\ to\ }2^8-1]} or @mymath{[0\rm{\ to\ }255]}.
+@cindex Configuration file precedence
+@cindex Configuration file directories
+@cindex Precedence, configuration files
+The option values in all the programs of Gnuastro will be filled in the
following order.
+If an option only takes one value which is given in an earlier step, any value
for that option in a later step will be ignored.
+Note that if the @option{lastconfig} option is specified in any step below, no
other configuration files will be parsed (see @ref{Operating mode options}).
-@item i8
-@itemx int8
-8-bit signed integers, range:@*
-@mymath{[-2^7\rm{\ to\ }2^7-1]} or @mymath{[-128\rm{\ to\ }127]}.
+@enumerate
+@item
+Command-line options, for a particular run of ProgramName.
-@item u16
-@itemx uint16
-16-bit unsigned integers, range:@*
-@mymath{[0\rm{\ to\ }2^{16}-1]} or @mymath{[0\rm{\ to\ }65535]}.
+@item
+@file{.gnuastro/astprogname.conf} is parsed by ProgramName in the current
directory.
-@item i16
-@itemx int16
-16-bit signed integers, range:@* @mymath{[-2^{15}\rm{\ to\ }2^{15}-1]} or
-@mymath{[-32768\rm{\ to\ }32767]}.
+@item
+@file{.gnuastro/gnuastro.conf} is parsed by all Gnuastro programs in the
current directory.
-@item u32
-@itemx uint32
-32-bit unsigned integers, range:@* @mymath{[0\rm{\ to\ }2^{32}-1]} or
-@mymath{[0\rm{\ to\ }4294967295]}.
+@item
+@file{$HOME/.local/etc/astprogname.conf} is parsed by ProgramName in the
user's home directory (see @ref{Current directory and User wide}).
-@item i32
-@itemx int32
-32-bit signed integers, range:@* @mymath{[-2^{31}\rm{\ to\ }2^{31}-1]} or
-@mymath{[-2147483648\rm{\ to\ }2147483647]}.
+@item
+@file{$HOME/.local/etc/gnuastro.conf} is parsed by all Gnuastro programs in
the user's home directory (see @ref{Current directory and User wide}).
-@item u64
-@itemx uint64
-64-bit unsigned integers, range@* @mymath{[0\rm{\ to\ }2^{64}-1]} or
-@mymath{[0\rm{\ to\ }18446744073709551615]}.
+@item
+@file{prefix/etc/astprogname.conf} is parsed by ProgramName in the system-wide
installation directory (see @ref{System wide} for @file{prefix}).
-@item i64
-@itemx int64
-64-bit signed integers, range:@* @mymath{[-2^{63}\rm{\ to\ }2^{63}-1]} or
-@mymath{[-9223372036854775808\rm{\ to\ }9223372036854775807]}.
+@item
+@file{prefix/etc/gnuastro.conf} is parsed by all Gnuastro programs in the
system-wide installation directory (see @ref{System wide} for @file{prefix}).
-@item f32
-@itemx float32
-32-bit (single-precision) floating point types.
-The maximum (minimum is its negative) possible value is
@mymath{3.402823\times10^{38}}.
-Single-precision floating points can accurately represent a floating point
number up to @mymath{\sim7.2} significant decimals.
-Given the heavy noise in astronomical data, this is usually more than
sufficient for storing results.
+@end enumerate
-@item f64
-@itemx float64
-64-bit (double-precision) floating point types.
-The maximum (minimum is its negative) possible value is @mymath{\sim10^{308}}.
-Double-precision floating points can accurately represent a floating point
number @mymath{\sim15.9} significant decimals.
-This is usually good for processing (mixing) the data internally, for example
a sum of single precision data (and later storing the result as @code{float32}).
-@end table
+The basic idea behind setting this progressive state of checking for parameter
values is that separate users of a computer or separate folders in a user's
file system might need different values for some parameters.
@cartouche
@noindent
-@strong{Some file formats don't recognize all types.} For example the FITS
standard (see @ref{Fits}) does not define @code{uint64} in binary tables or
images.
-When a type is not acceptable for output into a given file format, the
respective Gnuastro program or library will let you know and abort.
-On the command-line, you can convert the numerical type of an image, or table
column into another type with @ref{Arithmetic} or @ref{Table} respectively.
-If you are writing your own program, you can use the
@code{gal_data_copy_to_new_type()} function in Gnuastro's library, see
@ref{Copying datasets}.
+@strong{Checking the order:}
+You can confirm/check the order of parsing configuration files using the
@option{--checkconfig} option with any Gnuastro program, see @ref{Operating
mode options}.
+Just be sure to place this option immediately after the program name, before
any other option.
@end cartouche
+As you see above, there can also be a configuration file containing the common
options in all the programs: @file{gnuastro.conf} (see @ref{Common options}).
+If options specific to one program are specified in this file, there will be
unrecognized option errors, or unexpected behavior if the option has different
behavior in another program.
+On the other hand, there is no problem with @file{astprogname.conf} containing
common options@footnote{As an example, the @option{--setdirconf} and
@option{--setusrconf} options will also write the common options they have read
in their produced @file{astprogname.conf}.}.
+@cartouche
+@noindent
+@strong{Manipulating the order:} You can manipulate this order or add new
files with the following two options which are fully described in
+@ref{Operating mode options}:
+@table @option
+@item --config
+Allows you to define any file to be parsed as a configuration file on the
command-line or within the any other configuration file.
+Recall that the file given to @option{--config} is parsed immediately when
this option is confronted (on the command-line or in a configuration file).
-@node Memory management, Tables, Numeric data types, Common program behavior
-@section Memory management
-
-@cindex Memory management
-@cindex Non-volatile memory
-@cindex Memory, non-volatile
-In this section we'll review how Gnuastro manages your input data in your
system's memory.
-Knowing this can help you optimize your usage (in speed and memory
consumption) when the data volume is large and approaches, or exceeds, your
available RAM (usually in various calls to multiple programs simultaneously).
-But before diving into the details, let's have a short basic introduction to
memory in general and in particular the types of memory most relevant to this
discussion.
-
-Input datasets (that are later fed into programs for analysis) are commonly
first stored in @emph{non-volatile memory}.
-This is a type of memory that doesn't need a constant power supply to keep the
data and is therefore primarily aimed for long-term storage, like HDDs or SSDs.
-So data in this type of storage is preserved when you turn off your computer.
-But by its nature, non-volatile memory is much slower, in reading or writing,
than the speeds that CPUs can process the data.
-Thus relying on this type of memory alone would create a bad bottleneck in the
input/output (I/O) phase of any processing.
-
-@cindex RAM
-@cindex Volatile memory
-@cindex Memory, volatile
-The first step to decrease this bottleneck is to have a faster storage space,
but with a much limited storage volume.
-For this type of storage, computers have a Random Access Memory (or RAM).
-RAM is classified as a @emph{volatile memory} because it needs a constant flow
of electricity to keep the information.
-In other words, the moment power is cut-off, all the stored information in
your RAM is gone (hence the ``volatile'' name).
-But thanks to that constant supply of power, it can access any random address
with equal (and very high!) speed.
+@item --lastconfig
+Allows you to stop the parsing of subsequent configuration files.
+Note that if this option is given in a configuration file, it will be fully
read, so its position in the configuration doesn't matter (unlike
@option{--config}).
+@end table
+@end cartouche
-Hence, the general/simplistic way that programs deal with memory is the
following (this is general to almost all programs, not just Gnuastro's):
-1) Load/copy the input data from the non-volatile memory into RAM.
-2) Use the copy of the data in RAM as input for all the internal processing as
well as the intermediate data that is necessary during the processing.
-3) Finally, when the analysis is complete, write the final output data back
into non-volatile memory, and free/delete all the used space in the RAM (the
initial copy and all the intermediate data).
-Usually the RAM is most important for the data of the intermediate steps (that
you never see as a user of a program!).
+One example of benefiting from these configuration files can be this: raw
telescope images usually have their main image extension in the second FITS
extension, while processed FITS images usually only have one extension.
+If your system-wide default input extension is 0 (the first), then when you
want to work with the former group of data you have to explicitly mention it to
the programs every time.
+With this progressive state of default values to check, you can set different
default values for the different directories that you would like to run
Gnuastro in for your different purposes, so you won't have to worry about this
issue any more.
-When the input dataset(s) to a program are small (compared to the available
space in your system's RAM at the moment it is run) Gnuastro's programs and
libraries follow the standard series of steps above.
-The only exception is that deleting the intermediate data is not only done at
the end of the program.
-As soon as an intermediate dataset is no longer necessary for the next
internal steps, the space it occupied is deleted/freed.
-This allows Gnuastro programs to minimize their usage of your system's RAM
over the full running time.
+The same can be said about the @file{gnuastro.conf} files: by specifying a
behavior in this single file, all Gnuastro programs in the respective
directory, user, or system-wide steps will behave similarly.
+For example to keep the input's directory when no specific output is given
(see @ref{Automatic output}), or to not delete an existing file if it has the
same name as a given output (see @ref{Input output options}).
-The situation gets complicated when the datasets are large (compared to your
available RAM when the program is run).
-For example if a dataset is half the size of your system's available RAM, and
the program's internal analysis needs three or more intermediately processed
copies of it at one moment in its analysis.
-There won't be enough RAM to keep those higher-level intermediate data.
-In such cases, programs that don't do any memory management will crash.
-But fortunately Gnuastro's programs do have a memory management plans for such
situations.
-@cindex Memory-mapped file
-When the necessary amount of space for an intermediate dataset cannot be
allocated in the RAM, Gnuastro's programs will not use the RAM at all.
-They will use the ``memory-mapped file'' concept in modern operating systems
to create a randomly-named file in your non-volatile memory and use that
instead of the RAM.
-That file will have the exact size (in bytes) of that intermediate dataset.
-Any time the program needs that intermediate dataset, the operating system
will directly go to that file, and bypass your RAM.
-As soon as that file is no longer necessary for the analysis, it will be
deleted.
-But as mentioned above, non-volatile memory has much slower I/O speed than the
RAM.
-Hence in such situations, the programs will become noticeably slower
(sometimes by factors of 10 times slower, depending on your non-volatile memory
speed).
+@node Current directory and User wide, System wide, Configuration file
precedence, Configuration files
+@subsection Current directory and User wide
-Because of the drop in I/O speed (and thus the speed of your running program),
the moment that any to-be-allocated dataset is memory-mapped, Gnuastro's
programs and libraries will notify you with a descriptive statement like below
(can happen in any phase of their analysis).
-It shows the location of the memory-mapped file, its size, complemented with a
small description of the cause, a pointer to this section of the book for more
information on how to deal with it (if necessary), and what to do to suppress
it.
+@cindex @file{$HOME}
+@cindex @file{./.gnuastro/}
+@cindex @file{$HOME/.local/etc/}
+For the current (local) and user-wide directories, the configuration files are
stored in the hidden sub-directories named @file{.gnuastro/} and
@file{$HOME/.local/etc/} respectively.
+Unless you have changed it, the @file{$HOME} environment variable should point
to your home directory.
+You can check it by running @command{$ echo $HOME}.
+Each time you run any of the programs in Gnuastro, this environment variable
is read and placed in the above address.
+So if you suddenly see that your home configuration files are not being read,
probably you (or some other program) has changed the value of this environment
variable.
-@example
-astarithmetic: ./gnuastro_mmap/Fu7Dhs: temporary memory-mapped file
-(XXXXXXXXXXX bytes) created for intermediate data that is not stored
-in RAM (see the "Memory management" section of Gnuastro's manual for
-optimizing your project's memory management, and thus speed). To
-disable this warning, please use the option '--quiet-mmap'
-@end example
+@vindex --setdirconf
+@vindex --setusrconf
+Although it might cause confusions like above, this dependence on the
@file{HOME} environment variable enables you to temporarily use a different
directory as your home directory.
+This can come in handy in complicated situations.
+To set the user or current directory configuration files based on your
command-line input, you can use the @option{--setdirconf} or
@option{--setusrconf}, see @ref{Operating mode options}.
-@noindent
-Finally, when the intermediate dataset is no longer necessary, the program
will automatically delete it and notify you with a statement like this:
-@example
-astarithmetic: ./gnuastro_mmap/Fu7Dhs: deleted
-@end example
-@noindent
-To disable these messages, you can run the program with @code{--quietmmap}, or
set the @code{quietmmap} variable in the allocating library function to be
non-zero.
+@node System wide, , Current directory and User wide, Configuration files
+@subsection System wide
-An important component of these messages is the name of the memory-mapped file.
-Knowing that the file has been deleted is important for the user if the
program crashes for any reason: internally (for example a parameter is given
wrongly) or externally (for example you mistakenly kill the running job).
-In the event of a crash, the memory-mapped files will not be deleted and you
have to manually delete them because they are usually large and they may soon
fill your full storage if not deleted in a long time due to successive crashes.
+@cindex @file{prefix/etc/}
+@cindex System wide configuration files
+@cindex Configuration files, system wide
+When Gnuastro is installed, the configuration files that are shipped with the
distribution are copied into the (possibly system wide) @file{prefix/etc/}
directory.
+For more details on @file{prefix}, see @ref{Installation directory} (by
default it is: @file{/usr/local}).
+This directory is the final place (with the lowest priority) that the programs
in Gnuastro will check to retrieve parameter values.
-This brings us to managing the memory-mapped files in your non-volatile memory.
-In other words: knowing where they are saved, or intentionally placing them in
different places of your file system, or deleting them when necessary.
-As the examples above show, memory-mapped files are stored in a sub-directory
of the the running directory called @file{gnuastro_mmap}.
-If this directory doesn't exist, Gnuastro will automatically create it when
memory mapping becomes necessary.
-Alternatively, it may happen that the @file{gnuastro_mmap} sub-directory
exists and isn't writable, or it can't be created.
-In such cases, the memory-mapped file for each dataset will be created in the
running directory with a @file{gnuastro_mmap_} prefix.
+If you remove an option and its value from the system wide configuration
files, you either have to specify it in more immediate configuration files or
set it each time in the command-line.
+Recall that none of the programs in Gnuastro keep any internal default values
and will abort if they don't find a value for the necessary parameters (except
the number of threads and output file name).
+So even though you might never expect to use an optional option, it safe to
have it available in this system-wide configuration file even if you don't
intend to use it frequently.
-Therefore one easy way to delete all memory-mapped files in case of a crash,
is to delete everything within the sub-directory (first command below), or all
files stating with this prefix:
+Note that in case you install Gnuastro from your distribution's repositories,
@file{prefix} will either be set to @file{/} (the root directory) or
@file{/usr}, so you can find the system wide configuration variables in
@file{/etc/} or @file{/usr/etc/}.
+The prefix of @file{/usr/local/} is conventionally used for programs you
install from source by your self as in @ref{Quick start}.
-@example
-rm -f gnuastro_mmap/*
-rm -f gnuastro_mmap_*
-@end example
-A much more common issue when dealing with memory-mapped files is their
location.
-For example, you may be running a program in a partition that is hosted by an
HDD.
-But you also have another partition on an SSD (which has much faster I/O).
-So you want your memory-mapped files to be created in the SSD to speed up your
processing.
-In this scenario, you want your project source directory to only contain your
plain-text scripts and you want your project's built products (even the
temporary memory-mapped files) to be built in a different location because they
are large; thus I/O speed becomes important.
-To host the memory-mapped files in another location (with fast I/O), you can
set (@file{gnuastro_mmap}) to be a symbolic link to it.
-For example, let's assume you want your memory-mapped files to be stored in
@file{/path/to/dir/for/mmap}.
-All you have to do is to run the following command before your Gnuastro
analysis command(s).
-@example
-ln -s /path/to/dir/for/mmap gnuastro_mmap
-@end example
-The programs will delete a memory-mapped file when it is no longer needed, but
they won't delete the @file{gnuastro_mmap} directory that hosts them.
-So if your project involves many Gnuastro programs (possibly called in
parallel) and you want your memory-mapped files to be in a different location,
you just have to make the symbolic link above once at the start, and all the
programs will use it if necessary.
-Another memory-management scenario that may happen is this: you don't want a
Gnuastro program to allocate internal datasets in the RAM at all.
-For example the speed of your Gnuastro-related project doesn't matter at that
moment, and you have higher-priority jobs that are being run at the same time
which need to have RAM available.
-In such cases, you can use the @option{--minmapsize} option that is available
in all Gnuastro programs (see @ref{Processing options}).
-Any intermediate dataset that has a size larger than the value of this option
will be memory-mapped, even if there is space available in your RAM.
-For example if you want any dataset larger than 100 megabytes to be
memory-mapped, use @option{--minmapsize=100000000} (8 zeros!).
-@cindex Linux kernel
-@cindex Kernel, Linux
-You shouldn't set the value of @option{--minmapsize} to be too small,
otherwise even small intermediate values (that are usually very numerous) in
the program will be memory-mapped.
-However the kernel can only host a limited number of memory-mapped files at
every moment (by all running programs combined).
-For example in the default@footnote{If you need to host more memory-mapped
files at one moment, you need to build your own customized Linux kernel.} Linux
kernel on GNU/Linux operating systems this limit is roughly 64000.
-If the total number of memory-mapped files exceeds this number, all the
programs using them will crash.
-Gnuastro's programs will warn you if your given value is too small and may
cause a problem later.
-Actually, the default behavior for Gnuastro's programs (to only use
memory-mapped files when there isn't enough RAM) is a side-effect of
@option{--minmapsize}.
-The pre-defined value to this option is an extremely large value in the
lowest-level Gnuastro configuration file (the installed @file{gnuastro.conf}
described in @ref{Configuration file precedence}).
-This value is larger than the largest possible available RAM.
-You can check by running any Gnuastro program with a @option{-P} option.
-Because no dataset will be larger than this, by default the programs will
first attempt to use the RAM for temporary storage.
-But if writing in the RAM fails (for any reason, mainly due to lack of
available space), then a memory-mapped file will be created.
+@node Getting help, Multi-threaded operations, Configuration files, Common
program behavior
+@section Getting help
+@cindex Help
+@cindex Book formats
+@cindex Remembering options
+@cindex Convenient book formats
+Probably the first time you read this book, it is either in the PDF or HTML
formats.
+These two formats are very convenient for when you are not actually working,
but when you are only reading.
+Later on, when you start to use the programs and you are deep in the middle of
your work, some of the details will inevitably be forgotten.
+Going to find the PDF file (printed or digital) or the HTML web page is a
major distraction.
+@cindex Online help
+@cindex Command-line help
+GNU software have a very unique set of tools for aiding your memory on the
command-line, where you are working, depending how much of it you need to
remember.
+In the past, such command-line help was known as ``online'' help, because they
were literally provided to you `on' the command `line'.
+However, nowadays the word ``online'' refers to something on the internet, so
that term will not be used.
+With this type of help, you can resume your exciting research without taking
your hands off the keyboard.
-@node Tables, Tessellation, Memory management, Common program behavior
-@section Tables
-
-``A table is a collection of related data held in a structured format within a
database.
-It consists of columns, and rows.'' (from Wikipedia).
-Each column in the table contains the values of one property and each row is a
collection of properties (columns) for one target object.
-For example, let's assume you have just ran MakeCatalog (see
@ref{MakeCatalog}) on an image to measure some properties for the labeled
regions (which might be detected galaxies for example) in the image.
-For each labeled region (detected galaxy), there will be a @emph{row} which
groups its measured properties as @emph{columns}, one column for each property.
-One such property can be the object's magnitude, which is the sum of pixels
with that label, or its center can be defined as the light-weighted average
value of those pixels.
-Many such properties can be derived from the raw pixel values and their
position, see @ref{Invoking astmkcatalog} for a long list.
-
-As a summary, for each labeled region (or, galaxy) we have one @emph{row} and
for each measured property we have one @emph{column}.
-This high-level structure is usually the first step for higher-level analysis,
for example finding the stellar mass or photometric redshift from magnitudes in
multiple colors.
-Thus, tables are not just outputs of programs, in fact it is much more common
for tables to be inputs of programs.
-For example, to make a mock galaxy image, you need to feed in the properties
of each galaxy into @ref{MakeProfiles} for it do the inverse of the process
above and make a simulated image from a catalog, see @ref{Sufi simulates a
detection}.
-In other cases, you can feed a table into @ref{Crop} and it will crop out
regions centered on the positions within the table, see @ref{Finding reddest
clumps and visual inspection}.
-So to end this relatively long introduction, tables play a very important role
in astronomy, or generally all branches of data analysis.
-
-In @ref{Recognized table formats} the currently recognized table formats in
Gnuastro are discussed.
-You can use any of these tables as input or ask for them to be built as output.
-The most common type of table format is a simple plain text file with each row
on one line and columns separated by white space characters, this format is
easy to read/write by eye/hand.
-To give it the full functionality of more specific table types like the FITS
tables, Gnuastro has a special convention which you can use to give each column
a name, type, unit, and comments, while still being readable by other plain
text table readers.
-This convention is described in @ref{Gnuastro text table format}.
+@cindex Installed help methods
+Another major advantage of such command-line based help routines is that they
are installed with the software in your computer, therefore they are always in
sync with the executable you are actually running.
+Three of them are actually part of the executable.
+You don't have to worry about the version of the book or program.
+If you rely on external help (a PDF in your personal print or digital archive
or HTML from the official web page) you have to check to see if their versions
fit with your installed program.
-When tables are input to a program, the program reading it needs to know which
column(s) it should use for its desired purposes.
-Gnuastro's programs all follow a similar convention, on the way you can select
columns in a table.
-They are thoroughly discussed in @ref{Selecting table columns}.
+If you only need to remember the short or long names of the options,
@option{--usage} is advised.
+If it is what the options do, then @option{--help} is a great tool.
+Man pages are also provided for those who are use to this older system of
documentation.
+This full book is also available to you on the command-line in Info format.
+If none of these seems to resolve the problems, there is a mailing list which
enables you to get in touch with experienced Gnuastro users.
+In the subsections below each of these methods are reviewed.
@menu
-* Recognized table formats:: Table formats that are recognized in Gnuastro.
-* Gnuastro text table format:: Gnuastro's convention plain text tables.
-* Selecting table columns:: Identify/select certain columns from a table
+* --usage:: View option names and value formats.
+* --help:: List all options with description.
+* Man pages:: Man pages generated from --help.
+* Info:: View complete book in terminal.
+* help-gnuastro mailing list:: Contacting experienced users.
@end menu
-@node Recognized table formats, Gnuastro text table format, Tables, Tables
-@subsection Recognized table formats
-
-The list of table formats that Gnuastro can currently read from and write to
are described below.
-Each has their own advantage and disadvantages, so a short review of the
format is also provided to help you make the best choice based on how you want
to define your input tables or later use your output tables.
+@node --usage, --help, Getting help, Getting help
+@subsection @option{--usage}
+@vindex --usage
+@cindex Usage pattern
+@cindex Mandatory arguments
+@cindex Optional and mandatory tokens
+If you give this option, the program will not run.
+It will only print a very concise message showing the options and arguments.
+Everything within square brackets (@option{[]}) is optional.
+For example here are the first and last two lines of Crop's @option{--usage}
is shown:
-@table @asis
+@example
+$ astcrop --usage
+Usage: astcrop [-Do?IPqSVW] [-d INT] [-h INT] [-r INT] [-w INT]
+ [-x INT] [-y INT] [-c INT] [-p STR] [-N INT] [--deccol=INT]
+ ....
+ [--setusrconf] [--usage] [--version] [--wcsmode]
+ [ASCIIcatalog] FITSimage(s).fits
+@end example
-@item Plain text table
-This is the most basic and simplest way to create, view, or edit the table by
hand on a text editor.
-The other formats described below are less eye-friendly and have a more formal
structure (for easier computer readability).
-It is fully described in @ref{Gnuastro text table format}.
+There are no explanations on the options, just their short and long names
shown separately.
+After the program name, the short format of all the options that don't require
a value (on/off options) is displayed.
+Those that do require a value then follow in separate brackets, each
displaying the format of the input they want, see @ref{Options}.
+Since all options are optional, they are shown in square brackets, but
arguments can also be optional.
+For example in this example, a catalog name is optional and is only required
in some modes.
+This is a standard method of displaying optional arguments for all GNU
software.
-@cindex FITS Tables
-@cindex Tables FITS
-@cindex ASCII table, FITS
-@item FITS ASCII tables
-The FITS ASCII table extension is fully in ASCII encoding and thus easily
readable on any text editor (assuming it is the only extension in the FITS
file).
-If the FITS file also contains binary extensions (for example an image or
binary table extensions), then there will be many hard to print characters.
-The FITS ASCII format doesn't have new line characters to separate rows.
-In the FITS ASCII table standard, each row is defined as a fixed number of
characters (value to the @code{NAXIS1} keyword), so to visually inspect it
properly, you would have to adjust your text editor's width to this value.
-All columns start at given character positions and have a fixed width (number
of characters).
+@node --help, Man pages, --usage, Getting help
+@subsection @option{--help}
-Numbers in a FITS ASCII table are printed into ASCII format, they are not in
binary (that the CPU uses).
-Hence, they can take a larger space in memory, loose their precision, and take
longer to read into memory.
-If you are dealing with integer type columns (see @ref{Numeric data types}),
another issue with FITS ASCII tables is that the type information for the
column will be lost (there is only one integer type in FITS ASCII tables).
-One problem with the binary format on the other hand is that it isn't portable
(different CPUs/compilers) have different standards for translating the zeros
and ones.
-But since ASCII characters are defined on a byte and are well recognized, they
are better for portability on those various systems.
-Gnuastro's plain text table format described below is much more portable and
easier to read/write/interpret by humans manually.
+@vindex --help
+If the command-line includes this option, the program will not be run.
+It will print a complete list of all available options along with a short
explanation.
+The options are also grouped by their context.
+Within each context, the options are sorted alphabetically.
+Since the options are shown in detail afterwards, the first line of the
@option{--help} output shows the arguments and if they are optional or not,
similar to @ref{--usage}.
-Generally, as the name implies, this format is useful for when your table
mainly contains ASCII columns (for example file names, or descriptions).
-They can be useful when you need to include columns with structured ASCII
information along with other extensions in one FITS file.
-In such cases, you can also consider header keywords (see @ref{Fits}).
+In the @option{--help} output of all programs in Gnuastro, the options for
each program are classified based on context.
+The first two contexts are always options to do with the input and output
respectively.
+For example input image extensions or supplementary input files for the inputs.
+The last class of options is also fixed in all of Gnuastro, it shows operating
mode options.
+Most of these options are already explained in @ref{Operating mode options}.
-@cindex Binary table, FITS
-@item FITS binary tables
-The FITS binary table is the FITS standard's solution to the issues discussed
with keeping numbers in ASCII format as described under the FITS ASCII table
title above.
-Only columns defined as a string type (a string of ASCII characters) are
readable in a text editor.
-The portability problem with binary formats discussed above is mostly solved
thanks to the portability of CFITSIO (see @ref{CFITSIO}) and the very long
history of the FITS format which has been widely used since the 1970s.
+@cindex Long outputs
+@cindex Redirection of output
+@cindex Command-line, long outputs
+The help message will sometimes be longer than the vertical size of your
terminal.
+If you are using a graphical user interface terminal emulator, you can scroll
the terminal with your mouse, but we promised no mice distractions! So here are
some suggestions:
-In the case of most numbers, storing them in binary format is more memory
efficient than ASCII format.
-For example, to store @code{-25.72034} in ASCII format, you need 9
bytes/characters.
-But if you keep this same number (to the approximate precision possible) as a
4-byte (32-bit) floating point number, you can keep/transmit it with less than
half the amount of memory.
-When catalogs contain thousands/millions of rows in tens/hundreds of columns,
this can lead to significant improvements in memory/band-width usage.
-Moreover, since the CPU does its operations in the binary formats, reading the
table in and writing it out is also much faster than an ASCII table.
+@itemize
+@item
+@cindex Scroll command-line
+@cindex Command-line scroll
+@cindex @key{Shift + PageUP} and @key{Shift + PageDown}
+@key{Shift + PageUP} to scroll up and @key{Shift + PageDown} to scroll down.
+For most help output this should be enough.
+The problem is that it is limited by the number of lines that your terminal
keeps in memory and that you can't scroll by lines, only by whole screens.
-When you are dealing with integer numbers, the compression ratio can be even
better, for example if you know all of the values in a column are positive and
less than @code{255}, you can use the @code{unsigned char} type which only
takes one byte! If they are between @code{-128} and @code{127}, then you can
use the (signed) @code{char} type.
-So if you are thoughtful about the limits of your integer columns, you can
greatly reduce the size of your file and also the speed at which it is
read/written.
-This can be very useful when sharing your results with collaborators or
publishing them.
-To decrease the file size even more you can name your output as ending in
@file{.fits.gz} so it is also compressed after creation.
-Just note that compression/decompressing is CPU intensive and can slow down
the writing/reading of the file.
+@item
+@cindex Pipe
+@cindex @command{less}
+Pipe to @command{less}.
+A pipe is a form of shell re-direction.
+The @command{less} tool in Unix-like systems was made exactly for such outputs
of any length.
+You can pipe (@command{|}) the output of any program that is longer than the
screen to it and then you can scroll through (up and down) with its many tools.
+For example:
+@example
+$ astnoisechisel --help | less
+@end example
+@noindent
+Once you have gone through the text, you can quit @command{less} by pressing
the @key{q} key.
-Fortunately the FITS Binary table format also accepts ASCII strings as column
types (along with the various numerical types).
-So your dataset can also contain non-numerical columns.
-@end table
+@item
+@cindex Save output to file
+@cindex Redirection of output
+Redirect to a file.
+This is a less convenient way, because you will then have to open the file in
a text editor!
+You can do this with the shell redirection tool (@command{>}):
+@example
+$ astnoisechisel --help > filename.txt
+@end example
+@end itemize
-@menu
-* Gnuastro text table format:: Reading plain text tables
-@end menu
+@cindex GNU Grep
+@cindex Searching text
+@cindex Command-line searching text
+In case you have a special keyword you are looking for in the help, you don't
have to go through the full list.
+GNU Grep is made for this job.
+For example if you only want the list of options whose @option{--help} output
contains the word ``axis'' in Crop, you can run the following command:
-@node Gnuastro text table format, Selecting table columns, Recognized table
formats, Tables
-@subsection Gnuastro text table format
+@example
+$ astcrop --help | grep axis
+@end example
-Plain text files are the most generic, portable, and easiest way to (manually)
create, (visually) inspect, or (manually) edit a table.
-In this format, the ending of a row is defined by the new-line character (a
line on a text editor).
-So when you view it on a text editor, every row will occupy one line.
-The delimiters (or characters separating the columns) are white space
characters (space, horizontal tab, vertical tab) and a comma (@key{,}).
-The only further requirement is that all rows/lines must have the same number
of columns.
+@cindex @code{ARGP_HELP_FMT}
+@cindex Argp argument parser
+@cindex Customize @option{--help} output
+@cindex @option{--help} output customization
+If the output of this option does not fit nicely within the confines of your
terminal, GNU does enable you to customize its output through the environment
variable @code{ARGP_HELP_FMT}, you can set various parameters which specify the
formatting of the help messages.
+For example if your terminals are wider than 70 spaces (say 100) and you feel
there is too much empty space between the long options and the short
explanation, you can change these formats by giving values to this environment
variable before running the program with the @option{--help} output.
+You can define this environment variable in this manner:
+@example
+$ export ARGP_HELP_FMT=rmargin=100,opt-doc-col=20
+@end example
+@cindex @file{.bashrc}
+This will affect all GNU programs using GNU C library's @file{argp.h}
facilities as long as the environment variable is in memory.
+You can see the full list of these formatting parameters in the ``Argp User
Customization'' part of the GNU C library manual.
+If you are more comfortable to read the @option{--help} outputs of all GNU
software in your customized format, you can add your customization (similar to
the line above, without the @command{$} sign) to your @file{~/.bashrc} file.
+This is a standard option for all GNU software.
-The columns don't have to be exactly under each other and the rows can be
arbitrarily long with different lengths.
-For example the following contents in a file would be interpreted as a table
with 4 columns and 2 rows, with each element interpreted as a @code{double}
type (see @ref{Numeric data types}).
+@node Man pages, Info, --help, Getting help
+@subsection Man pages
+@cindex Man pages
+Man pages were the Unix method of providing command-line documentation to a
program.
+With GNU Info, see @ref{Info} the usage of this method of documentation is
highly discouraged.
+This is because Info provides a much more easier to navigate and read
environment.
+However, some operating systems require a man page for packages that are
installed and some people are still used to this method of command line help.
+So the programs in Gnuastro also have Man pages which are automatically
generated from the outputs of @option{--version} and @option{--help} using the
GNU help2man program.
+So if you run
@example
-1 2.234948 128 39.8923e8
-2 , 4.454 792 72.98348e7
+$ man programname
@end example
+@noindent
+You will be provided with a man page listing the options in the
+standard manner.
-However, the example above has no other information about the columns (it is
just raw data, with no meta-data).
-To use this table, you have to remember what the numbers in each column
represent.
-Also, when you want to select columns, you have to count their position within
the table.
-This can become frustrating and prone to bad errors (getting the columns
wrong) especially as the number of columns increase.
-It is also bad for sending to a colleague, because they will find it hard to
remember/use the columns properly.
-To solve these problems in Gnuastro's programs/libraries you aren't limited to
using the column's number, see @ref{Selecting table columns}.
-If the columns have names, units, or comments you can also select your columns
based on searches/matches in these fields, for example see @ref{Table}.
-Also, in this manner, you can't guide the program reading the table on how to
read the numbers.
-As an example, the first and third columns above can be read as integer types:
the first column might be an ID and the third can be the number of pixels an
object occupies in an image.
-So there is no need to read these to columns as a @code{double} type (which
takes more memory, and is slower).
-In the bare-minimum example above, you also can't use strings of characters,
for example the names of filters, or some other identifier that includes
non-numerical characters.
-In the absence of any information, only numbers can be read robustly.
-Assuming we read columns with non-numerical characters as string, there would
still be the problem that the strings might contain space (or any delimiter)
character for some rows.
-So, each `word' in the string will be interpreted as a column and the program
will abort with an error that the rows don't have the same number of columns.
-To correct for these limitations, Gnuastro defines the following convention
for storing the table meta-data along with the raw data in one plain text file.
-The format is primarily designed for ease of reading/writing by eye/fingers,
but is also structured enough to be read by a program.
-When the first non-white character in a line is @key{#}, or there are no
non-white characters in it, then the line will not be considered as a row of
data in the table (this is a pretty standard convention in many programs, and
higher level languages).
-In the former case, the line is interpreted as a @emph{comment}.
-If the comment line starts with `@code{# Column N:}', then it is assumed to
contain information about column @code{N} (a number, counting from 1).
-Comment lines that don't start with this pattern are ignored and you can use
them to include any further information you want to store with the table in the
text file.
-A column information comment is assumed to have the following format:
+@node Info, help-gnuastro mailing list, Man pages, Getting help
+@subsection Info
+
+@cindex GNU Info
+@cindex Command-line, viewing full book
+Info is the standard documentation format for all GNU software.
+It is a very useful command-line document viewing format, fully equipped with
links between the various pages and menus and search capabilities.
+As explained before, the best thing about it is that it is available for you
the moment you need to refresh your memory on any command-line tool in the
middle of your work without having to take your hands off the keyboard.
+This complete book is available in Info format and can be accessed from
anywhere on the command-line.
+
+To open the Info format of any installed programs or library on your system
which has an Info format book, you can simply run the command below (change
@command{executablename} to the executable name of the program or library):
@example
-# Column N: NAME [UNIT, TYPE, BLANK] COMMENT
+$ info executablename
@end example
-@cindex NaN
@noindent
-Any sequence of characters between `@key{:}' and `@key{[}' will be interpreted
as the column name (so it can contain anything except the `@key{[}' character).
-Anything between the `@key{]}' and the end of the line is defined as a comment.
-Within the brackets, anything before the first `@key{,}' is the units
(physical units, for example km/s, or erg/s), anything before the second
`@key{,}' is the short type identifier (see below, and @ref{Numeric data
types}).
-If the type identifier isn't recognized, the default 64-bit floating point
type will be used.
-
-Finally (still within the brackets), any non-white characters after the second
`@key{,}' are interpreted as the blank value for that column (see @ref{Blank
pixels}).
-The blank value can either be in the same type as the column (for example
@code{-99} for a signed integer column), or any string (for example @code{NaN}
in that same column).
-In both cases, the values will be stored in memory as Gnuastro's fixed blank
values for each type.
-For floating point types, Gnuastro's internal blank value is IEEE NaN
(Not-a-Number).
-For signed integers, it is the smallest possible value and for unsigned
integers its the largest possible value.
+@cindex Learning GNU Info
+@cindex GNU software documentation
+In case you are not already familiar with it, run @command{$ info info}.
+It does a fantastic job in explaining all its capabilities its self.
+It is very short and you will become sufficiently fluent in about half an hour.
+Since all GNU software documentation is also provided in Info, your whole
GNU/Linux life will significantly improve.
-When a formatting problem occurs, or when the column was already given
meta-data in a previous comment, or when the column number is larger than the
actual number of columns in the table (the non-commented or empty lines), then
the comment information line will be ignored.
+@cindex GNU Emacs
+@cindex GNU C library
+Once you've become an efficient navigator in Info, you can go to any part of
this book or any other GNU software or library manual, no matter how long it
is, in a matter of seconds.
+It also blends nicely with GNU Emacs (a text editor) and you can search
manuals while you are writing your document or programs without taking your
hands off the keyboard, this is most useful for libraries like the GNU C
library.
+To be able to access all the Info manuals installed in your GNU/Linux within
Emacs, type @key{Ctrl-H + i}.
-When a comment information line can be used, the leading and trailing white
space characters will be stripped from all of the elements.
-For example in this line:
+To see this whole book from the beginning in Info, you can run
@example
-# Column 5: column name [km/s, f32,-99] Redshift as speed
+$ info gnuastro
@end example
-The @code{NAME} field will be `@code{column name}' and the @code{TYPE} field
will be `@code{f32}'.
-Note how all the white space characters before and after strings are not used,
but those in the middle remained.
-Also, white space characters aren't mandatory.
-Hence, in the example above, the @code{BLANK} field will be given the value of
`@code{-99}'.
-
-Except for the column number (@code{N}), the rest of the fields are optional.
-Also, the column information comments don't have to be in order.
-In other words, the information for column @mymath{N+m} (@mymath{m>0}) can be
given in a line before column @mymath{N}.
-Furthermore, you don't have to specify information for all columns.
-Those columns that don't have this information will be interpreted with the
default settings (like the case above: values are double precision floating
point, and the column has no name, unit, or comment).
-So these lines are all acceptable for any table (the first one, with nothing
but the column number is redundant):
+@noindent
+If you run Info with the particular program executable name, for
+example @file{astcrop} or @file{astnoisechisel}:
@example
-# Column 5:
-# Column 1: ID [,i8] The Clump ID.
-# Column 3: mag_f160w [AB mag, f32] Magnitude from the F160W filter
+$ info astprogramname
@end example
@noindent
-The data type of the column should be specified with one of the following
values:
-
-@itemize
-@item
-For a numeric column, you can use any of the numeric types (and their
-recognized identifiers) described in @ref{Numeric data types}.
-@item
-`@code{strN}': for strings.
-The @code{N} value identifies the length of the string (how many characters it
has).
-The start of the string on each row is the first non-delimiter character of
the column that has the string type.
-The next @code{N} characters will be interpreted as a string and all leading
and trailing white space will be removed.
-
-If the next column's characters, are closer than @code{N} characters to the
start of the string column in that line/row, they will be considered part of
the string column.
-If there is a new-line character before the ending of the space given to the
string column (in other words, the string column is the last column), then
reading of the string will stop, even if the @code{N} characters are not
complete yet.
-See @file{tests/table/table.txt} for one example.
-Therefore, the only time you have to pay attention to the positioning and
spaces given to the string column is when it is not the last column in the
table.
-
-The only limitation in this format is that trailing and leading white space
characters will be removed from the columns that are read.
-In most cases, this is the desired behavior, but if trailing and leading
white-spaces are critically important to your analysis, define your own
starting and ending characters and remove them after the table has been read.
-For example in the sample table below, the two `@key{|}' characters (which are
arbitrary) will remain in the value of the second column and you can remove
them manually later.
-If only one of the leading or trailing white spaces is important for your
work, you can only use one of the `@key{|}'s.
+you will be taken to the section titled ``Invoking ProgramName'' which
explains the inputs and outputs along with the command-line options for that
program.
+Finally, if you run Info with the official program name, for example Crop or
NoiseChisel:
@example
-# Column 1: ID [label, u8]
-# Column 2: Notes [no unit, str50]
-1 leading and trailing white space is ignored here 2.3442e10
-2 | but they will be preserved here | 8.2964e11
+$ info ProgramName
@end example
-@end itemize
-
-Note that the FITS binary table standard does not define the @code{unsigned
int} and @code{unsigned long} types, so if you want to convert your tables to
FITS binary tables, use other types.
-Also, note that in the FITS ASCII table, there is only one integer type
(@code{long}).
-So if you convert a Gnuastro plain text table to a FITS ASCII table with the
@ref{Table} program, the type information for integers will be lost.
-Conversely if integer types are important for you, you have to manually set
them when reading a FITS ASCII table (for example with the Table program when
reading/converting into a file, or with the @file{gnuastro/table.h} library
functions when reading into memory).
+@noindent
+you will be taken to the top section which introduces the program.
+Note that in all cases, Info is not case sensitive.
-@node Selecting table columns, , Gnuastro text table format, Tables
-@subsection Selecting table columns
-At the lowest level, the only defining aspect of a column in a table is its
number, or position.
-But selecting columns purely by number is not very convenient and, especially
when the tables are large it can be very frustrating and prone to errors.
-Hence, table file formats (for example see @ref{Recognized table formats})
have ways to store additional information about the columns (meta-data).
-Some of the most common pieces of information about each column are its
@emph{name}, the @emph{units} of data in it, and a @emph{comment} for
longer/informal description of the column's data.
+@node help-gnuastro mailing list, , Info, Getting help
+@subsection help-gnuastro mailing list
-To facilitate research with Gnuastro, you can select columns by matching, or
searching in these three fields, besides the low-level column number.
-To view the full list of information on the columns in the table, you can use
the Table program (see @ref{Table}) with the command below (replace
@file{table-file} with the filename of your table, if its FITS, you might also
need to specify the HDU/extension which contains the table):
+@cindex help-gnuastro mailing list
+@cindex Mailing list: help-gnuastro
+Gnuastro maintains the help-gnuastro mailing list for users to ask any
questions related to Gnuastro.
+The experienced Gnuastro users and some of its developers are subscribed to
this mailing list and your email will be sent to them immediately.
+However, when contacting this mailing list please have in mind that they are
possibly very busy and might not be able to answer immediately.
-@example
-$ asttable --information table-file
-@end example
+@cindex Mailing list archives
+@cindex @code{help-gnuastro@@gnu.org}
+To ask a question from this mailing list, send a mail to
@code{help-gnuastro@@gnu.org}.
+Anyone can view the mailing list archives at
@url{http://lists.gnu.org/archive/html/help-gnuastro/}.
+It is best that before sending a mail, you search the archives to see if
anyone has asked a question similar to yours.
+If you want to make a suggestion or report a bug, please don't send a mail to
this mailing list.
+We have other mailing lists and tools for those purposes, see @ref{Report a
bug} or @ref{Suggest new feature}.
-Gnuastro's programs need the columns for different purposes, for example in
Crop, you specify the columns containing the central coordinates of the crop
centers with the @option{--coordcol} option (see @ref{Crop options}).
-On the other hand, in MakeProfiles, to specify the column containing the
profile position angles, you must use the @option{--pcol} option (see
@ref{MakeProfiles catalog}).
-Thus, there can be no unified common option name to select columns for all
programs (different columns have different purposes).
-However, when the program expects a column for a specific context, the option
names end in the @option{col} suffix like the examples above.
-These options accept values in integer (column number), or string (metadata
match/search) format.
-If the value can be parsed as a positive integer, it will be seen as the
low-level column number.
-Note that column counting starts from 1, so if you ask for column 0, the
respective program will abort with an error.
-When the value can't be interpreted as an a integer number, it will be seen as
a string of characters which will be used to match/search in the table's
meta-data.
-The meta-data field which the value will be compared with can be selected
through the @option{--searchin} option, see @ref{Input output options}.
-@option{--searchin} can take three values: @code{name}, @code{unit},
@code{comment}.
-The matching will be done following this convention:
-@itemize
-@item
-If the value is enclosed in two slashes (for example @command{-x/RA_/}, or
@option{--coordcol=/RA_/}, see @ref{Crop options}), then it is assumed to be a
regular expression with the same convention as GNU AWK.
-GNU AWK has a very well written
@url{https://www.gnu.org/software/gawk/manual/html_node/Regexp.html, chapter}
describing regular expressions, so we will not continue discussing them here.
-Regular expressions are a very powerful tool in matching text and useful in
many contexts.
-We thus strongly encourage reviewing this chapter for greatly improving the
quality of your work in many cases, not just for searching column meta-data in
Gnuastro.
-@item
-When the string isn't enclosed between `@key{/}'s, any column that exactly
matches the given value in the given field will be selected.
-@end itemize
-Note that in both cases, you can ignore the case of alphabetic characters with
the @option{--ignorecase} option, see @ref{Input output options}.
-Also, in both cases, multiple columns may be selected with one call to this
function.
-In this case, the order of the selected columns (with one call) will be the
same order as they appear in the table.
+@node Multi-threaded operations, Numeric data types, Getting help, Common
program behavior
+@section Multi-threaded operations
+@pindex nproc
+@cindex pthread
+@cindex CPU threads
+@cindex GNU Coreutils
+@cindex Using CPU threads
+@cindex CPU, using all threads
+@cindex Multi-threaded programs
+@cindex Using multiple CPU cores
+@cindex Simultaneous multithreading
+Some of the programs benefit significantly when you use all the threads your
computer's CPU has to offer to your operating system.
+The number of threads available can be larger than the number of physical
(hardware) cores in the CPU (also known as Simultaneous multithreading).
+For example, in Intel's CPUs (those that implement its Hyper-threading
technology) the number of threads is usually double the number of physical
cores in your CPU.
+On a GNU/Linux system, the number of threads available can be found with the
command @command{$ nproc} command (part of GNU Coreutils).
+@vindex --numthreads
+@cindex Number of threads available
+@cindex Available number of threads
+@cindex Internally stored option value
+Gnuastro's programs can find the number of threads available to your system
internally at run-time (when you execute the program).
+However, if a value is given to the @option{--numthreads} option, the given
number will be used, see @ref{Operating mode options} and @ref{Configuration
files} for ways to use this option.
+Thus @option{--numthreads} is the only common option in Gnuastro's programs
with a value that doesn't have to be specified anywhere on the command-line or
in the configuration files.
+@menu
+* A note on threads:: Caution and suggestion on using threads.
+* How to run simultaneous operations:: How to run things simultaneously.
+@end menu
+@node A note on threads, How to run simultaneous operations, Multi-threaded
operations, Multi-threaded operations
+@subsection A note on threads
-@node Tessellation, Automatic output, Tables, Common program behavior
-@section Tessellation
+@cindex Using multiple threads
+@cindex Best use of CPU threads
+@cindex Efficient use of CPU threads
+Spinning off threads is not necessarily the most efficient way to run an
application.
+Creating a new thread isn't a cheap operation for the operating system.
+It is most useful when the input data are fixed and you want the same
operation to be done on parts of it.
+For example one input image to Crop and multiple crops from various parts of
it.
+In this fashion, the image is loaded into memory once, all the crops are
divided between the number of threads internally and each thread cuts out those
parts which are assigned to it from the same image.
+On the other hand, if you have multiple images and you want to crop the same
region(s) out of all of them, it is much more efficient to set
@option{--numthreads=1} (so no threads spin off) and run Crop multiple times
simultaneously, see @ref{How to run simultaneous operations}.
-It is sometimes necessary to classify the elements in a dataset (for example
pixels in an image) into a grid of individual, non-overlapping tiles.
-For example when background sky gradients are present in an image, you can
define a tile grid over the image.
-When the tile sizes are set properly, the background's variation over each
tile will be negligible, allowing you to measure (and subtract) it.
-In other cases (for example spatial domain convolution in Gnuastro, see
@ref{Convolve}), it might simply be for speed of processing: each tile can be
processed independently on a separate CPU thread.
-In the arts and mathematics, this process is formally known as
@url{https://en.wikipedia.org/wiki/Tessellation, tessellation}.
+@cindex Wall-clock time
+You can check the boost in speed by first running a program on one of the data
sets with the maximum number of threads and another time (with everything else
the same) and only using one thread.
+You will notice that the wall-clock time (reported by most programs at their
end) in the former is longer than the latter divided by number of physical CPU
cores (not threads) available to your operating system.
+Asymptotically these two times can be equal (most of the time they aren't).
+So limiting the programs to use only one thread and running them independently
on the number of available threads will be more efficient.
-The size of the regular tiles (in units of data-elements, or pixels in an
image) can be defined with the @option{--tilesize} option.
-It takes multiple numbers (separated by a comma) which will be the length
along the respective dimension (in FORTRAN/FITS dimension order).
-Divisions are also acceptable, but must result in an integer.
-For example @option{--tilesize=30,40} can be used for an image (a 2D dataset).
-The regular tile size along the first FITS axis (horizontal when viewed in SAO
DS9) will be 30 pixels and along the second it will be 40 pixels.
-Ideally, @option{--tilesize} should be selected such that all tiles in the
image have exactly the same size.
-In other words, that the dataset length in each dimension is divisible by the
tile size in that dimension.
+@cindex System Cache
+@cindex Cache, system
+Note that the operating system keeps a cache of recently processed data, so
usually, the second time you process an identical data set (independent of the
number of threads used), you will get faster results.
+In order to make an unbiased comparison, you have to first clean the system's
cache with the following command between the two runs.
-However, this is not always possible: the dataset can be any size and every
pixel in it is valuable.
-In such cases, Gnuastro will look at the significance of the remainder length,
if it is not significant (for example one or two pixels), then it will just
increase the size of the first tile in the respective dimension and allow the
rest of the tiles to have the required size.
-When the remainder is significant (for example one pixel less than the size
along that dimension), the remainder will be added to one regular tile's size
and the large tile will be cut in half and put in the two ends of the
grid/tessellation.
-In this way, all the tiles in the central regions of the dataset will have the
regular tile sizes and the tiles on the edge will be slightly larger/smaller
depending on the remainder significance.
-The fraction which defines the remainder significance along all dimensions can
be set through @option{--remainderfrac}.
+@example
+$ sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
+@end example
-The best tile size is directly related to the spatial properties of the
property you want to study (for example, gradient on the image).
-In practice we assume that the gradient is not present over each tile.
-So if there is a strong gradient (for example in long wavelength ground based
images) or the image is of a crowded area where there isn't too much blank
area, you have to choose a smaller tile size.
-A larger mesh will give more pixels and and so the scatter in the results will
be less (better statistics).
+@cartouche
+@noindent
+@strong{SUMMARY: Should I use multiple threads?} Depends:
+@itemize
+@item
+If you only have @strong{one} data set (image in most cases!), then yes, the
more threads you use (with a maximum of the number of threads available to your
OS) the faster you will get your results.
-@cindex CCD
-@cindex Amplifier
-@cindex Bias current
-@cindex Subaru Telescope
-@cindex Hyper Suprime-Cam
-@cindex Hubble Space Telescope (HST)
-For raw image processing, a single tessellation/grid is not sufficient.
-Raw images are the unprocessed outputs of the camera detectors.
-Modern detectors usually have multiple readout channels each with its own
amplifier.
-For example the Hubble Space Telescope Advanced Camera for Surveys (ACS) has
four amplifiers over its full detector area dividing the square field of view
to four smaller squares.
-Ground based image detectors are not exempt, for example each CCD of Subaru
Telescope's Hyper Suprime-Cam camera (which has 104 CCDs) has four amplifiers,
but they have the same height of the CCD and divide the width by four parts.
+@item
+If you want to run the same operation on @strong{multiple} data sets, it is
best to set the number of threads to 1 and use Make, or GNU Parallel, as
explained in @ref{How to run simultaneous operations}.
+@end itemize
+@end cartouche
-@cindex Channel
-The bias current on each amplifier is different, and initial bias subtraction
is not perfect.
-So even after subtracting the measured bias current, you can usually still
identify the boundaries of different amplifiers by eye.
-See Figure 11(a) in Akhlaghi and Ichikawa (2015) for an example.
-This results in the final reduced data to have non-uniform amplifier-shaped
regions with higher or lower background flux values.
-Such systematic biases will then propagate to all subsequent measurements we
do on the data (for example photometry and subsequent stellar mass and star
formation rate measurements in the case of galaxies).
-Therefore an accurate analysis requires a two layer tessellation: the top
layer contains larger tiles, each covering one amplifier channel.
-For clarity we'll call these larger tiles ``channels''.
-The number of channels along each dimension is defined through the
@option{--numchannels}.
-Each channel is then covered by its own individual smaller tessellation (with
tile sizes determined by the @option{--tilesize} option).
-This will allow independent analysis of two adjacent pixels from different
channels if necessary.
-If the image is processed or the detector only has one amplifier, you can set
the number of channels in both dimension to 1.
-The final tessellation can be inspected on the image with the
@option{--checktiles} option that is available to all programs which use
tessellation for localized operations.
-When this option is called, a FITS file with a @file{_tiled.fits} suffix will
be created along with the outputs, see @ref{Automatic output}.
-Each pixel in this image has the number of the tile that covers it.
-If the number of channels in any dimension are larger than unity, you will
notice that the tile IDs are defined such that the first channels is covered
first, then the second and so on.
-For the full list of processing-related common options (including tessellation
options), please see @ref{Processing options}.
+@node How to run simultaneous operations, , A note on threads, Multi-threaded
operations
+@subsection How to run simultaneous operations
+There are two@footnote{A third way would be to open multiple terminal emulator
windows in your GUI, type the commands separately on each and press @key{Enter}
once on each terminal, but this is far too frustrating, tedious and prone to
errors.
+It's therefore not a realistic solution when tens, hundreds or thousands of
operations (your research targets, multiplied by the operations you do on each)
are to be done.} approaches to simultaneously execute a program: using GNU
Parallel or Make (GNU Make is the most common implementation).
+The first is very useful when you only want to do one job multiple times and
want to get back to your work without actually keeping the command you ran.
+The second is usually for more important operations, with lots of dependencies
between the different products (for example a full scientific research).
+@table @asis
-@node Automatic output, Output FITS files, Tessellation, Common program
behavior
-@section Automatic output
-
-@cindex Standard input
-@cindex Automatic output file names
-@cindex Output file names, automatic
-@cindex Setting output file names automatically
-All the programs in Gnuastro are designed such that specifying an output file
or directory (based on the program context) is optional.
-When no output name is explicitly given (with @option{--output}, see
@ref{Input output options}), the programs will automatically set an output name
based on the input name(s) and what the program does.
-For example when you are using ConvertType to save FITS image named
@file{dataset.fits} to a JPEG image and don't specify a name for it, the JPEG
output file will be name @file{dataset.jpg}.
-When the input is from the standard input (for example a pipe, see
@ref{Standard input}), and @option{--output} isn't given, the output name will
be the program's name (for example @file{converttype.jpg}).
-
-@vindex --keepinputdir
-Another very important part of the automatic output generation is that all the
directory information of the input file name is stripped off of it.
-This feature can be disabled with the @option{--keepinputdir} option, see
@ref{Input output options}.
-It is the default because astronomical data are usually very large and
organized specially with special file names.
-In some cases, the user might not have write permissions in those
directories@footnote{In fact, even if the data is stored on your own computer,
it is advised to only grant write permissions to the super user or root.
-This way, you won't accidentally delete or modify your valuable data!}.
-
-Let's assume that we are working on a report and want to process the FITS
images from two projects (ABC and DEF), which are stored in the sub-directories
named @file{ABCproject/} and @file{DEFproject/} of our top data directory
(@file{/mnt/data}).
-The following shell commands show how one image from the former is first
converted to a JPEG image through ConvertType and then the objects from an
image in the latter project are detected using NoiseChisel.
-The text after the @command{#} sign are comments (not typed!).
+@item GNU Parallel
+@cindex GNU Parallel
+When you only want to run multiple instances of a command on different threads
and get on with the rest of your work, the best method is to use GNU parallel.
+Surprisingly GNU Parallel is one of the few GNU packages that has no Info
documentation but only a Man page, see @ref{Info}.
+So to see the documentation after installing it please run
@example
-$ pwd # Current location
-/home/usrname/research/report
-$ ls # List directory contents
-ABC01.jpg
-$ ls /mnt/data/ABCproject # Archive 1
-ABC01.fits ABC02.fits ABC03.fits
-$ ls /mnt/data/DEFproject # Archive 2
-DEF01.fits DEF02.fits DEF03.fits
-$ astconvertt /mnt/data/ABCproject/ABC02.fits --output=jpg # Prog 1
-$ ls
-ABC01.jpg ABC02.jpg
-$ astnoisechisel /mnt/data/DEFproject/DEF01.fits # Prog 2
-$ ls
-ABC01.jpg ABC02.jpg DEF01_detected.fits
+$ man parallel
@end example
+@noindent
+As an example, let's assume we want to crop a region fixed on the pixels (500,
600) with the default width from all the FITS images in the @file{./data}
directory ending with @file{sci.fits} to the current directory.
+To do this, you can run:
+@example
+$ parallel astcrop --numthreads=1 --xc=500 --yc=600 ::: \
+ ./data/*sci.fits
+@end example
+@noindent
+GNU Parallel can help in many more conditions, this is one of the simplest,
see the man page for lots of other examples.
+For absolute beginners: the backslash (@command{\}) is only a line breaker to
fit nicely in the page.
+If you type the whole command in one line, you should remove it.
+@item Make
+@cindex Make
+Make is a program for building ``targets'' (e.g., files) using ``recipes'' (a
set of operations) when their known ``prerequisites'' (other files) have been
updated.
+It elegantly allows you to define dependency structures for building your
final output and updating it efficiently when the inputs change.
+It is the most common infra-structure to build software today.
+Scientific research methodology is very similar to software development: you
start by testing a hypothesis on a small sample of objects/targets with a
simple set of steps.
+As you are able to get promising results, you improve the method and use it on
a larger, more general, sample.
+In the process, you will confront many issues that have to be corrected (bugs
in software development jargon).
+Make a wonderful tool to manage this style of development.
+It has been used to make reproducible papers, for example see
@url{https://gitlab.com/makhlaghi/NoiseChisel-paper, the reproduction pipeline}
of the paper introducing @ref{NoiseChisel} (one of Gnuastro's programs).
-@node Output FITS files, , Automatic output, Common program behavior
-@section Output FITS files
-
-@cindex FITS
-@cindex Output FITS headers
-@cindex CFITSIO version on outputs
-The output of many of Gnuastro's programs are (or can be) FITS files.
-The FITS format has many useful features for storing scientific datasets
(cubes, images and tables) along with a robust features for archivability.
-For more on this standard, please see @ref{Fits}.
-
-As a community convention described in @ref{Fits}, the first extension of all
FITS files produced by Gnuastro's programs only contains the meta-data that is
intended for the file's extension(s).
-For a Gnuastro program, this generic meta-data (that is stored as FITS keyword
records) is its configuration when it produced this dataset: file name(s) of
input(s) and option names, values and comments.
-Note that when the configuration is too trivial (only input filename, for
example the program @ref{Table}) no meta-data is written in this extension.
-
-FITS keywords have the following limitations in regards to generic option
names and values which are described below:
-
-@itemize
-@item
-If a keyword (option name) is longer than 8 characters, the first word in the
record (80 character line) is @code{HIERARCH} which is followed by the keyword
name.
-
-@item
-Values can be at most 75 characters, but for strings, this changes to 73
(because of the two extra @key{'} characters that are necessary).
-However, if the value is a file name, containing slash (@key{/}) characters to
separate directories, Gnuastro will break the value into multiple keywords.
+@cindex GNU Make
+GNU Make@footnote{@url{https://www.gnu.org/software/make/}} is the most common
implementation which (similar to nearly all GNU programs, comes with a
wonderful manual@footnote{@url{https://www.gnu.org/software/make/manual/}}).
+Make is very basic and simple, and thus the manual is short (the most
important parts are in the first roughly 100 pages) and easy to read/understand.
-@item
-Keyword names ignore case, therefore they are all in capital letters.
-Therefore, if you want to use Grep to inspect these keywords, use the
@option{-i} option, like the example below.
+Make comes with a @option{--jobs} (@option{-j}) option which allows you to
specify the maximum number of jobs that can be done simultaneously.
+For example if you have 8 threads available to your operating system.
+You can run:
@example
-$ astfits image_detected.fits -h0 | grep -i snquant
+$ make -j8
@end example
-@end itemize
-The keywords above are classified (separated by an empty line and title) as a
group titled ``ProgramName configuration''.
-This meta-data extension, as well as all the other extensions (which contain
data), also contain have final group of keywords to keep the basic date and
version information of Gnuastro, its dependencies and the pipeline that is
using Gnuastro (if its under version control).
+With this command, Make will process your @file{Makefile} and create all the
targets (can be thousands of FITS images for example) simultaneously on 8
threads, while fully respecting their dependencies (only building a file/target
when its prerequisites are successfully built).
+Make is thus strongly recommended for managing scientific research where
robustness, archiving, reproducibility and speed@footnote{Besides its
multi-threaded capabilities, Make will only re-build those targets that depend
on a change you have made, not the whole work.
+For example, if you have set the prerequisites properly, you can easily test
the changing of a parameter on your paper's results without having to re-do
everything (which is much faster).
+This allows you to be much more productive in easily checking various
ideas/assumptions of the different stages of your research and thus produce a
more robust result for your exciting science.} are important.
-@table @command
+@end table
-@item DATE
-The creation time of the FITS file.
-This date is written directly by CFITSIO and is in UT format.
-@item COMMIT
-Git's commit description from the running directory of Gnuastro's programs.
-If the running directory is not version controlled or @file{libgit2} isn't
installed (see @ref{Optional dependencies}) then this keyword will not be
present.
-The printed value is equivalent to the output of the following command:
-@example
-git describe --dirty --always
-@end example
-If the running directory contains non-committed work, then the stored value
will have a `@command{-dirty}' suffix.
-This can be very helpful to let you know that the data is not ready to be
shared with collaborators or submitted to a journal.
-You should only share results that are produced after all your work is
committed (safely stored in the version controlled history and thus
reproducible).
-At first sight, version control appears to be mainly a tool for software
developers.
-However progress in a scientific research is almost identical to progress in
software development: first you have a rough idea that starts with handful of
easy steps.
-But as the first results appear to be promising, you will have to extend, or
generalize, it to make it more robust and work in all the situations your
research covers, not just your first test samples.
-Slowly you will find wrong assumptions or bad implementations that need to be
fixed (`bugs' in software development parlance).
-Finally, when you submit the research to your collaborators or a journal, many
comments and suggestions will come in, and you have to address them.
+@node Numeric data types, Memory management, Multi-threaded operations, Common
program behavior
+@section Numeric data types
-Software developers have created version control systems precisely for this
kind of activity.
-Each significant moment in the project's history is called a ``commit'', see
@ref{Version controlled source}.
-A snapshot of the project in each ``commit'' is safely stored away, so you can
revert back to it at a later time, or check changes/progress.
-This way, you can be sure that your work is reproducible and track the
progress and history.
-With version control, experimentation in the project's analysis is greatly
facilitated, since you can easily revert back if a brainstorm test procedure
fails.
+@cindex Bit
+@cindex Type
+At the lowest level, the computer stores everything in terms of @code{1} or
@code{0}.
+For example, each program in Gnuastro, or each astronomical image you take
with the telescope is actually a string of millions of these zeros and ones.
+The space required to keep a zero or one is the smallest unit of storage, and
is known as a @emph{bit}.
+However, understanding and manipulating this string of bits is extremely hard
for most people.
+Therefore, different standards are defined to package the bits into separate
@emph{type}s with a fixed interpretation of the bits in each package.
-One important feature of version control is that the research result (FITS
image, table, report or paper) can be stamped with the unique commit
information that produced it.
-This information will enable you to exactly reproduce that same result later,
even if you have made changes/progress.
-For one example of a research paper's reproduction pipeline, please see the
@url{https://gitlab.com/makhlaghi/NoiseChisel-paper, reproduction pipeline} of
the @url{https://arxiv.org/abs/1505.01664, paper} describing @ref{NoiseChisel}.
+@cindex Byte
+@cindex Signed integer
+@cindex Unsigned integer
+@cindex Integer, Signed
+To store numbers, the most basic standard/type is for integers (@mymath{...,
-2, -1, 0, 1, 2, ...}).
+The common integer types are 8, 16, 32, and 64 bits wide (more bits will give
larger limits).
+Each bit corresponds to a power of 2 and they are summed to create the final
number.
+In the integer types, for each width there are two standards for reading the
bits: signed and unsigned.
+In the `signed' convention, one bit is reserved for the sign (stating that the
integer is positive or negative).
+The `unsigned' integers use that bit in the actual number and thus contain
only positive numbers (starting from zero).
-@item CFITSIO
-The version of CFITSIO used (see @ref{CFITSIO}).
+Therefore, at the same number of bits, both signed and unsigned integers can
allow the same number of integers, but the positive limit of the
@code{unsigned} types is double their @code{signed} counterparts with the same
width (at the expense of not having negative numbers).
+When the context of your work doesn't involve negative numbers (for example
counting, where negative is not defined), it is best to use the @code{unsigned}
types.
+For the full numerical range of all integer types, see below.
-@item WCSLIB
-The version of WCSLIB used (see @ref{WCSLIB}).
-Note that older versions of WCSLIB do not report the version internally.
-So this is only available if you are using more recent WCSLIB versions.
+Another standard of converting a given number of bits to numbers is the
floating point standard, this standard can @emph{approximately} store any real
number with a given precision.
+There are two common floating point types: 32-bit and 64-bit, for single and
double precision floating point numbers respectively.
+The former is sufficient for data with less than 8 significant decimal digits
(most astronomical data), while the latter is good for less than 16 significant
decimal digits.
+The representation of real numbers as bits is much more complex than integers.
+If you are interested to learn more about it, you can start with the
@url{https://en.wikipedia.org/wiki/Floating_point, Wikipedia article}.
-@item GSL
-The version of GNU Scientific Library that was used, see @ref{GNU Scientific
Library}.
+Practically, you can use Gnuastro's Arithmetic program to convert/change the
type of an image/datacube (see @ref{Arithmetic}), or Gnuastro Table program to
convert a table column's data type (see @ref{Column arithmetic}).
+Conversion of a dataset's type is necessary in some contexts.
+For example the program/library, that you intend to feed the data into, only
accepts floating point values, but you have an integer image/column.
+Another situation that conversion can be helpful is when you know that your
data only has values that fit within @code{int8} or @code{uint16}.
+However it is currently formatted in the @code{float64} type.
-@item GNUASTRO
-The version of Gnuastro used (see @ref{Version numbering}).
-@end table
+The important thing to consider is that operations involving wider, floating
point, or signed types can be significantly slower than smaller-width, integer,
or unsigned types respectively.
+Note that besides speed, a wider type also requires much more storage space
(by 4 or 8 times).
+Therefore, when you confront such situations that can be optimized and want to
store/archive/transfer the data, it is best to use the most efficient type.
+For example if your dataset (image or table column) only has positive integers
less than 65535, store it as an unsigned 16-bit integer for faster processing,
faster transfer, and less storage space.
-Here is one example of the last few lines of an example output.
+The short and long names for the recognized numeric data types in Gnuastro are
listed below.
+Both short and long names can be used when you want to specify a type.
+For example, as a value to the common option @option{--type} (see @ref{Input
output options}), or in the information comment lines of @ref{Gnuastro text
table format}.
+The ranges listed below are inclusive.
-@example
- / Versions and date
-DATE = '...' / file creation date
-COMMIT = 'v0-8-g547f6eb' / Commit description in running dir.
-CFITSIO = '3.45 ' / CFITSIO version.
-WCSLIB = '5.19 ' / WCSLIB version.
-GSL = '2.5 ' / GNU Scientific Library version.
-GNUASTRO= '0.7 ' / GNU Astronomy Utilities version.
-END
-@end example
+@table @code
+@item u8
+@itemx uint8
+8-bit unsigned integers, range:@*
+@mymath{[0\rm{\ to\ }2^8-1]} or @mymath{[0\rm{\ to\ }255]}.
+@item i8
+@itemx int8
+8-bit signed integers, range:@*
+@mymath{[-2^7\rm{\ to\ }2^7-1]} or @mymath{[-128\rm{\ to\ }127]}.
+@item u16
+@itemx uint16
+16-bit unsigned integers, range:@*
+@mymath{[0\rm{\ to\ }2^{16}-1]} or @mymath{[0\rm{\ to\ }65535]}.
+@item i16
+@itemx int16
+16-bit signed integers, range:@* @mymath{[-2^{15}\rm{\ to\ }2^{15}-1]} or
+@mymath{[-32768\rm{\ to\ }32767]}.
+@item u32
+@itemx uint32
+32-bit unsigned integers, range:@* @mymath{[0\rm{\ to\ }2^{32}-1]} or
+@mymath{[0\rm{\ to\ }4294967295]}.
+@item i32
+@itemx int32
+32-bit signed integers, range:@* @mymath{[-2^{31}\rm{\ to\ }2^{31}-1]} or
+@mymath{[-2147483648\rm{\ to\ }2147483647]}.
+@item u64
+@itemx uint64
+64-bit unsigned integers, range@* @mymath{[0\rm{\ to\ }2^{64}-1]} or
+@mymath{[0\rm{\ to\ }18446744073709551615]}.
+@item i64
+@itemx int64
+64-bit signed integers, range:@* @mymath{[-2^{63}\rm{\ to\ }2^{63}-1]} or
+@mymath{[-9223372036854775808\rm{\ to\ }9223372036854775807]}.
+@item f32
+@itemx float32
+32-bit (single-precision) floating point types.
+The maximum (minimum is its negative) possible value is
@mymath{3.402823\times10^{38}}.
+Single-precision floating points can accurately represent a floating point
number up to @mymath{\sim7.2} significant decimals.
+Given the heavy noise in astronomical data, this is usually more than
sufficient for storing results.
+@item f64
+@itemx float64
+64-bit (double-precision) floating point types.
+The maximum (minimum is its negative) possible value is @mymath{\sim10^{308}}.
+Double-precision floating points can accurately represent a floating point
number @mymath{\sim15.9} significant decimals.
+This is usually good for processing (mixing) the data internally, for example
a sum of single precision data (and later storing the result as @code{float32}).
+@end table
+@cartouche
+@noindent
+@strong{Some file formats don't recognize all types.} For example the FITS
standard (see @ref{Fits}) does not define @code{uint64} in binary tables or
images.
+When a type is not acceptable for output into a given file format, the
respective Gnuastro program or library will let you know and abort.
+On the command-line, you can convert the numerical type of an image, or table
column into another type with @ref{Arithmetic} or @ref{Table} respectively.
+If you are writing your own program, you can use the
@code{gal_data_copy_to_new_type()} function in Gnuastro's library, see
@ref{Copying datasets}.
+@end cartouche
+@node Memory management, Tables, Numeric data types, Common program behavior
+@section Memory management
-@node Data containers, Data manipulation, Common program behavior, Top
-@chapter Data containers
+@cindex Memory management
+@cindex Non-volatile memory
+@cindex Memory, non-volatile
+In this section we'll review how Gnuastro manages your input data in your
system's memory.
+Knowing this can help you optimize your usage (in speed and memory
consumption) when the data volume is large and approaches, or exceeds, your
available RAM (usually in various calls to multiple programs simultaneously).
+But before diving into the details, let's have a short basic introduction to
memory in general and in particular the types of memory most relevant to this
discussion.
-@cindex File operations
-@cindex Operations on files
-@cindex General file operations
-The most low-level and basic property of a dataset is how it is stored.
-To process, archive and transmit the data, you need a container to store it
first.
-From the start of the computer age, different formats have been defined to
store data, optimized for particular applications.
-One format/container can never be useful for all applications: the storage
defines the application and vice-versa.
-In astronomy, the Flexible Image Transport System (FITS) standard has become
the most common format of data storage and transmission.
-It has many useful features, for example multiple sub-containers (also known
as extensions or header data units, HDUs) within one file, or support for
tables as well as images.
-Each HDU can store an independent dataset and its corresponding meta-data.
-Therefore, Gnuastro has one program (see @ref{Fits}) specifically designed to
manipulate FITS HDUs and the meta-data (header keywords) in each HDU.
-
-Your astronomical research does not just involve data analysis (where the FITS
format is very useful).
-For example you want to demonstrate your raw and processed FITS images or
spectra as figures within slides, reports, or papers.
-The FITS format is not defined for such applications.
-Thus, Gnuastro also comes with the ConvertType program (see @ref{ConvertType})
which can be used to convert a FITS image to and from (where possible) other
formats like plain text and JPEG (which allow two way conversion), along with
EPS and PDF (which can only be created from FITS, not the other way round).
+Input datasets (that are later fed into programs for analysis) are commonly
first stored in @emph{non-volatile memory}.
+This is a type of memory that doesn't need a constant power supply to keep the
data and is therefore primarily aimed for long-term storage, like HDDs or SSDs.
+So data in this type of storage is preserved when you turn off your computer.
+But by its nature, non-volatile memory is much slower, in reading or writing,
than the speeds that CPUs can process the data.
+Thus relying on this type of memory alone would create a bad bottleneck in the
input/output (I/O) phase of any processing.
-Finally, the FITS format is not just for images, it can also store tables.
-Binary tables in particular can be very efficient in storing catalogs that
have more than a few tens of columns and rows.
-However, unlike images (where all elements/pixels have one data type), tables
contain multiple columns and each column can have different properties:
independent data types (see @ref{Numeric data types}) and meta-data.
-In practice, each column can be viewed as a separate container that is grouped
with others in the table.
-The only shared property of the columns in a table is thus the number of
elements they contain.
-To allow easy inspection/manipulation of table columns, Gnuastro has the Table
program (see @ref{Table}).
-It can be used to select certain table columns in a FITS table and see them as
a human readable output on the command-line, or to save them into another plain
text or FITS table.
+@cindex RAM
+@cindex Volatile memory
+@cindex Memory, volatile
+The first step to decrease this bottleneck is to have a faster storage space,
but with a much limited storage volume.
+For this type of storage, computers have a Random Access Memory (or RAM).
+RAM is classified as a @emph{volatile memory} because it needs a constant flow
of electricity to keep the information.
+In other words, the moment power is cut-off, all the stored information in
your RAM is gone (hence the ``volatile'' name).
+But thanks to that constant supply of power, it can access any random address
with equal (and very high!) speed.
-@menu
-* Fits:: View and manipulate extensions and keywords.
-* ConvertType:: Convert data to various formats.
-* Table:: Read and Write FITS tables to plain text.
-* Query:: Import data from external databases.
-@end menu
+Hence, the general/simplistic way that programs deal with memory is the
following (this is general to almost all programs, not just Gnuastro's):
+1) Load/copy the input data from the non-volatile memory into RAM.
+2) Use the copy of the data in RAM as input for all the internal processing as
well as the intermediate data that is necessary during the processing.
+3) Finally, when the analysis is complete, write the final output data back
into non-volatile memory, and free/delete all the used space in the RAM (the
initial copy and all the intermediate data).
+Usually the RAM is most important for the data of the intermediate steps (that
you never see as a user of a program!).
+When the input dataset(s) to a program are small (compared to the available
space in your system's RAM at the moment it is run) Gnuastro's programs and
libraries follow the standard series of steps above.
+The only exception is that deleting the intermediate data is not only done at
the end of the program.
+As soon as an intermediate dataset is no longer necessary for the next
internal steps, the space it occupied is deleted/freed.
+This allows Gnuastro programs to minimize their usage of your system's RAM
over the full running time.
+The situation gets complicated when the datasets are large (compared to your
available RAM when the program is run).
+For example if a dataset is half the size of your system's available RAM, and
the program's internal analysis needs three or more intermediately processed
copies of it at one moment in its analysis.
+There won't be enough RAM to keep those higher-level intermediate data.
+In such cases, programs that don't do any memory management will crash.
+But fortunately Gnuastro's programs do have a memory management plans for such
situations.
+@cindex Memory-mapped file
+When the necessary amount of space for an intermediate dataset cannot be
allocated in the RAM, Gnuastro's programs will not use the RAM at all.
+They will use the ``memory-mapped file'' concept in modern operating systems
to create a randomly-named file in your non-volatile memory and use that
instead of the RAM.
+That file will have the exact size (in bytes) of that intermediate dataset.
+Any time the program needs that intermediate dataset, the operating system
will directly go to that file, and bypass your RAM.
+As soon as that file is no longer necessary for the analysis, it will be
deleted.
+But as mentioned above, non-volatile memory has much slower I/O speed than the
RAM.
+Hence in such situations, the programs will become noticeably slower
(sometimes by factors of 10 times slower, depending on your non-volatile memory
speed).
+Because of the drop in I/O speed (and thus the speed of your running program),
the moment that any to-be-allocated dataset is memory-mapped, Gnuastro's
programs and libraries will notify you with a descriptive statement like below
(can happen in any phase of their analysis).
+It shows the location of the memory-mapped file, its size, complemented with a
small description of the cause, a pointer to this section of the book for more
information on how to deal with it (if necessary), and what to do to suppress
it.
-@node Fits, ConvertType, Data containers, Data containers
-@section Fits
+@example
+astarithmetic: ./gnuastro_mmap/Fu7Dhs: temporary memory-mapped file
+(XXXXXXXXXXX bytes) created for intermediate data that is not stored
+in RAM (see the "Memory management" section of Gnuastro's manual for
+optimizing your project's memory management, and thus speed). To
+disable this warning, please use the option '--quiet-mmap'
+@end example
-@cindex Vatican library
-The ``Flexible Image Transport System'', or FITS, is by far the most common
data container format in astronomy and in constant use since the 1970s.
-Archiving (future usage, simplicity) has been one of the primary design
principles of this format.
-In the last few decades it has proved so useful and robust that the Vatican
Library has also chosen FITS for its ``long-term digital preservation''
project@footnote{@url{https://www.vaticanlibrary.va/home.php?pag=progettodigit}}.
+@noindent
+Finally, when the intermediate dataset is no longer necessary, the program
will automatically delete it and notify you with a statement like this:
-@cindex IAU, international astronomical union
-Although the full name of the standard invokes the idea that it is only for
images, it also contains complete and robust features for tables.
-It started off in the 1970s and was formally published as a standard in 1981,
it was adopted by the International Astronomical Union (IAU) in 1982 and an IAU
working group to maintain its future was defined in 1988.
-The FITS 2.0 and 3.0 standards were approved in 2000 and 2008 respectively,
and the 4.0 draft has also been released recently, please see the
@url{https://fits.gsfc.nasa.gov/fits_standard.html, FITS standard document web
page} for the full text of all versions.
-Also see the @url{https://doi.org/10.1051/0004-6361/201015362, FITS 3.0
standard paper} for a nice introduction and history along with the full
standard.
+@example
+astarithmetic: ./gnuastro_mmap/Fu7Dhs: deleted
+@end example
-@cindex Meta-data
-Many common image formats, for example a JPEG, only have one image/dataset per
file, however one great advantage of the FITS standard is that it allows you to
keep multiple datasets (images or tables along with their separate meta-data)
in one file.
-In the FITS standard, each data + metadata is known as an extension, or more
formally a header data unit or HDU.
-The HDUs in a file can be completely independent: you can have multiple images
of different dimensions/sizes or tables as separate extensions in one file.
-However, while the standard doesn't impose any constraints on the relation
between the datasets, it is strongly encouraged to group data that are
contextually related with each other in one file.
-For example an image and the table/catalog of objects and their measured
properties in that image.
-Other examples can be images of one patch of sky in different colors
(filters), or one raw telescope image along with its calibration data (tables
or images).
+@noindent
+To disable these messages, you can run the program with @code{--quietmmap}, or
set the @code{quietmmap} variable in the allocating library function to be
non-zero.
-As discussed above, the extensions in a FITS file can be completely
independent.
-To keep some information (meta-data) about the group of extensions in the FITS
file, the community has adopted the following convention: put no data in the
first extension, so it is just meta-data.
-This extension can thus be used to store Meta-data regarding the whole file
(grouping of extensions).
-Subsequent extensions may contain data along with their own separate meta-data.
-All of Gnuastro's programs also follow this convention: the main output
dataset(s) are placed in the second (or later) extension(s).
-The first extension contains no data the program's configuration (input file
name, along with all its option values) are stored as its meta-data, see
@ref{Output FITS files}.
+An important component of these messages is the name of the memory-mapped file.
+Knowing that the file has been deleted is important for the user if the
program crashes for any reason: internally (for example a parameter is given
wrongly) or externally (for example you mistakenly kill the running job).
+In the event of a crash, the memory-mapped files will not be deleted and you
have to manually delete them because they are usually large and they may soon
fill your full storage if not deleted in a long time due to successive crashes.
-The meta-data contain information about the data, for example which region of
the sky an image corresponds to, the units of the data, what telescope, camera,
and filter the data were taken with, it observation date, or the software that
produced it and its configuration.
-Without the meta-data, the raw dataset is practically just a collection of
numbers and really hard to understand, or connect with the real world (other
datasets).
-It is thus strongly encouraged to supplement your data (at any level of
processing) with as much meta-data about your processing/science as possible.
+This brings us to managing the memory-mapped files in your non-volatile memory.
+In other words: knowing where they are saved, or intentionally placing them in
different places of your file system, or deleting them when necessary.
+As the examples above show, memory-mapped files are stored in a sub-directory
of the the running directory called @file{gnuastro_mmap}.
+If this directory doesn't exist, Gnuastro will automatically create it when
memory mapping becomes necessary.
+Alternatively, it may happen that the @file{gnuastro_mmap} sub-directory
exists and isn't writable, or it can't be created.
+In such cases, the memory-mapped file for each dataset will be created in the
running directory with a @file{gnuastro_mmap_} prefix.
-The meta-data of a FITS file is in ASCII format, which can be easily viewed or
edited with a text editor or on the command-line.
-Each meta-data element (known as a keyword generally) is composed of a name,
value, units and comments (the last two are optional).
-For example below you can see three FITS meta-data keywords for specifying the
world coordinate system (WCS, or its location in the sky) of a dataset:
+Therefore one easy way to delete all memory-mapped files in case of a crash,
is to delete everything within the sub-directory (first command below), or all
files stating with this prefix:
@example
-LATPOLE = -27.805089 / [deg] Native latitude of celestial pole
-RADESYS = 'FK5' / Equatorial coordinate system
-EQUINOX = 2000.0 / [yr] Equinox of equatorial coordinates
+rm -f gnuastro_mmap/*
+rm -f gnuastro_mmap_*
@end example
-However, there are some limitations which discourage viewing/editing the
keywords with text editors.
-For example there is a fixed length of 80 characters for each keyword (its
name, value, units and comments) and there are no new-line characters, so on a
text editor all the keywords are seen in one line.
-Also, the meta-data keywords are immediately followed by the data which are
commonly in binary format and will show up as strange looking characters on a
text editor, and significantly slowing down the processor.
-
-Gnuastro's Fits program was designed to allow easy manipulation of FITS
extensions and meta-data keywords on the command-line while conforming fully
with the FITS standard.
-For example you can copy or cut (copy and remove) HDUs/extensions from one
FITS file to another, or completely delete them.
-It also has features to delete, add, or edit meta-data keywords within one HDU.
-
-@menu
-* Invoking astfits:: Arguments and options to Header.
-@end menu
-
-@node Invoking astfits, , Fits, Fits
-@subsection Invoking Fits
+A much more common issue when dealing with memory-mapped files is their
location.
+For example, you may be running a program in a partition that is hosted by an
HDD.
+But you also have another partition on an SSD (which has much faster I/O).
+So you want your memory-mapped files to be created in the SSD to speed up your
processing.
+In this scenario, you want your project source directory to only contain your
plain-text scripts and you want your project's built products (even the
temporary memory-mapped files) to be built in a different location because they
are large; thus I/O speed becomes important.
-Fits can print or manipulate the FITS file HDUs (extensions), meta-data
keywords in a given HDU.
-The executable name is @file{astfits} with the following general template
+To host the memory-mapped files in another location (with fast I/O), you can
set (@file{gnuastro_mmap}) to be a symbolic link to it.
+For example, let's assume you want your memory-mapped files to be stored in
@file{/path/to/dir/for/mmap}.
+All you have to do is to run the following command before your Gnuastro
analysis command(s).
@example
-$ astfits [OPTION...] ASTRdata
+ln -s /path/to/dir/for/mmap gnuastro_mmap
@end example
+The programs will delete a memory-mapped file when it is no longer needed, but
they won't delete the @file{gnuastro_mmap} directory that hosts them.
+So if your project involves many Gnuastro programs (possibly called in
parallel) and you want your memory-mapped files to be in a different location,
you just have to make the symbolic link above once at the start, and all the
programs will use it if necessary.
-@noindent
-One line examples:
-
-@example
-## View general information about every extension:
-$ astfits image.fits
+Another memory-management scenario that may happen is this: you don't want a
Gnuastro program to allocate internal datasets in the RAM at all.
+For example the speed of your Gnuastro-related project doesn't matter at that
moment, and you have higher-priority jobs that are being run at the same time
which need to have RAM available.
+In such cases, you can use the @option{--minmapsize} option that is available
in all Gnuastro programs (see @ref{Processing options}).
+Any intermediate dataset that has a size larger than the value of this option
will be memory-mapped, even if there is space available in your RAM.
+For example if you want any dataset larger than 100 megabytes to be
memory-mapped, use @option{--minmapsize=100000000} (8 zeros!).
-## Print the header keywords in the second HDU (counting from 0):
-$ astfits image.fits -h1
+@cindex Linux kernel
+@cindex Kernel, Linux
+You shouldn't set the value of @option{--minmapsize} to be too small,
otherwise even small intermediate values (that are usually very numerous) in
the program will be memory-mapped.
+However the kernel can only host a limited number of memory-mapped files at
every moment (by all running programs combined).
+For example in the default@footnote{If you need to host more memory-mapped
files at one moment, you need to build your own customized Linux kernel.} Linux
kernel on GNU/Linux operating systems this limit is roughly 64000.
+If the total number of memory-mapped files exceeds this number, all the
programs using them will crash.
+Gnuastro's programs will warn you if your given value is too small and may
cause a problem later.
-## Only print header keywords that contain `NAXIS':
-$ astfits image.fits -h1 | grep NAXIS
+Actually, the default behavior for Gnuastro's programs (to only use
memory-mapped files when there isn't enough RAM) is a side-effect of
@option{--minmapsize}.
+The pre-defined value to this option is an extremely large value in the
lowest-level Gnuastro configuration file (the installed @file{gnuastro.conf}
described in @ref{Configuration file precedence}).
+This value is larger than the largest possible available RAM.
+You can check by running any Gnuastro program with a @option{-P} option.
+Because no dataset will be larger than this, by default the programs will
first attempt to use the RAM for temporary storage.
+But if writing in the RAM fails (for any reason, mainly due to lack of
available space), then a memory-mapped file will be created.
-## Only print the WCS standard PC matrix elements
-$ astfits image.fits -h1 | grep 'PC._.'
-## Copy a HDU from input.fits to out.fits:
-$ astfits input.fits --copy=hdu-name --output=out.fits
-## Update the OLDKEY keyword value to 153.034:
-$ astfits --update=OLDKEY,153.034,"Old keyword comment"
-## Delete one COMMENT keyword and add a new one:
-$ astfits --delete=COMMENT --comment="Anything you like ;-)."
-## Write two new keywords with different values and comments:
-$ astfits --write=MYKEY1,20.00,"An example keyword" --write=MYKEY2,fd
-@end example
+@node Tables, Tessellation, Memory management, Common program behavior
+@section Tables
-@cindex HDU
-@cindex HEALPix
-When no action is requested (and only a file name is given), Fits will print a
list of information about the extension(s) in the file.
-This information includes the HDU number, HDU name (@code{EXTNAME} keyword),
type of data (see @ref{Numeric data types}, and the number of data elements it
contains (size along each dimension for images and table rows and columns).
-Optionally, a comment column is printed for special situations (like a 2D
HEALPix grid that is usually stored as a 1D dataset/table).
-You can use this to get a general idea of the contents of the FITS file and
what HDU to use for further processing, either with the Fits program or any
other Gnuastro program.
+``A table is a collection of related data held in a structured format within a
database.
+It consists of columns, and rows.'' (from Wikipedia).
+Each column in the table contains the values of one property and each row is a
collection of properties (columns) for one target object.
+For example, let's assume you have just ran MakeCatalog (see
@ref{MakeCatalog}) on an image to measure some properties for the labeled
regions (which might be detected galaxies for example) in the image.
+For each labeled region (detected galaxy), there will be a @emph{row} which
groups its measured properties as @emph{columns}, one column for each property.
+One such property can be the object's magnitude, which is the sum of pixels
with that label, or its center can be defined as the light-weighted average
value of those pixels.
+Many such properties can be derived from the raw pixel values and their
position, see @ref{Invoking astmkcatalog} for a long list.
-Here is one example of information about a FITS file with four extensions: the
first extension has no data, it is a purely meta-data HDU (commonly used to
keep meta-data about the whole file, or grouping of extensions, see @ref{Fits}).
-The second extension is an image with name @code{IMAGE} and single precision
floating point type (@code{float32}, see @ref{Numeric data types}), it has 4287
pixels along its first (horizontal) axis and 4286 pixels along its second
(vertical) axis.
-The third extension is also an image with name @code{MASK}.
-It is in 2-byte integer format (@code{int16}) which is commonly used to keep
information about pixels (for example to identify which ones were saturated, or
which ones had cosmic rays and so on), note how it has the same size as the
@code{IMAGE} extension.
-The third extension is a binary table called @code{CATALOG} which has 12371
rows and 5 columns (it probably contains information about the sources in the
image).
+As a summary, for each labeled region (or, galaxy) we have one @emph{row} and
for each measured property we have one @emph{column}.
+This high-level structure is usually the first step for higher-level analysis,
for example finding the stellar mass or photometric redshift from magnitudes in
multiple colors.
+Thus, tables are not just outputs of programs, in fact it is much more common
for tables to be inputs of programs.
+For example, to make a mock galaxy image, you need to feed in the properties
of each galaxy into @ref{MakeProfiles} for it do the inverse of the process
above and make a simulated image from a catalog, see @ref{Sufi simulates a
detection}.
+In other cases, you can feed a table into @ref{Crop} and it will crop out
regions centered on the positions within the table, see @ref{Finding reddest
clumps and visual inspection}.
+So to end this relatively long introduction, tables play a very important role
in astronomy, or generally all branches of data analysis.
-@example
-GNU Astronomy Utilities X.X
-Run on Day Month DD HH:MM:SS YYYY
------
-HDU (extension) information: `image.fits'.
- Column 1: Index (counting from 0).
- Column 2: Name (`EXTNAME' in FITS standard).
- Column 3: Image data type or `table' format (ASCII or binary).
- Column 4: Size of data in HDU.
------
-0 n/a uint8 0
-1 IMAGE float32 4287x4286
-2 MASK int16 4287x4286
-3 CATALOG table_binary 12371x5
-@end example
+In @ref{Recognized table formats} the currently recognized table formats in
Gnuastro are discussed.
+You can use any of these tables as input or ask for them to be built as output.
+The most common type of table format is a simple plain text file with each row
on one line and columns separated by white space characters, this format is
easy to read/write by eye/hand.
+To give it the full functionality of more specific table types like the FITS
tables, Gnuastro has a special convention which you can use to give each column
a name, type, unit, and comments, while still being readable by other plain
text table readers.
+This convention is described in @ref{Gnuastro text table format}.
-If a specific HDU is identified on the command-line with the @option{--hdu}
(or @option{-h} option) and no operation requested, then the full list of
header keywords in that HDU will be printed (as if the @option{--printallkeys}
was called, see below).
-It is important to remember that this only occurs when @option{--hdu} is given
on the command-line.
-The @option{--hdu} value given in a configuration file will only be used when
a specific operation on keywords requested.
-Therefore as described in the paragraphs above, when no explicit call to the
@option{--hdu} option is made on the command-line and no operation is requested
(on the command-line or configuration files), the basic information of each
HDU/extension is printed.
+When tables are input to a program, the program reading it needs to know which
column(s) it should use for its desired purposes.
+Gnuastro's programs all follow a similar convention, on the way you can select
columns in a table.
+They are thoroughly discussed in @ref{Selecting table columns}.
-The operating mode and input/output options to Fits are similar to the other
programs and fully described in @ref{Common options}.
-The options particular to Fits can be divided into two groups:
-1) those related to modifying HDUs or extensions (see @ref{HDU information and
manipulation}), and
-2) those related to viewing/modifying meta-data keywords (see @ref{Keyword
inspection and manipulation}).
-These two classes of options cannot be called together in one run: you can
either work on the extensions or meta-data keywords in any instance of Fits.
@menu
-* HDU information and manipulation:: Learn about the HDUs and move them.
-* Keyword inspection and manipulation:: Manipulate metadata keywords in a HDU
+* Recognized table formats:: Table formats that are recognized in Gnuastro.
+* Gnuastro text table format:: Gnuastro's convention plain text tables.
+* Selecting table columns:: Identify/select certain columns from a table
@end menu
+@node Recognized table formats, Gnuastro text table format, Tables, Tables
+@subsection Recognized table formats
+The list of table formats that Gnuastro can currently read from and write to
are described below.
+Each has their own advantage and disadvantages, so a short review of the
format is also provided to help you make the best choice based on how you want
to define your input tables or later use your output tables.
+@table @asis
+@item Plain text table
+This is the most basic and simplest way to create, view, or edit the table by
hand on a text editor.
+The other formats described below are less eye-friendly and have a more formal
structure (for easier computer readability).
+It is fully described in @ref{Gnuastro text table format}.
-@node HDU information and manipulation, Keyword inspection and manipulation,
Invoking astfits, Invoking astfits
-@subsubsection HDU information and manipulation
-Each FITS file header data unit, or HDU (also known as an extension) is an
independent dataset (data + meta-data).
-Multiple HDUs can be stored in one FITS file, see @ref{Fits}.
-The general HDU-related options to the Fits program are listed below as two
general classes:
-the first group below focus on HDU information while the latter focus on
manipulating (moving or deleting) the HDUs.
-
-The options below print information about the given HDU on the command-line.
-Thus they cannot be called together in one command (each has its own
independent output).
+@cindex FITS Tables
+@cindex Tables FITS
+@cindex ASCII table, FITS
+@item FITS ASCII tables
+The FITS ASCII table extension is fully in ASCII encoding and thus easily
readable on any text editor (assuming it is the only extension in the FITS
file).
+If the FITS file also contains binary extensions (for example an image or
binary table extensions), then there will be many hard to print characters.
+The FITS ASCII format doesn't have new line characters to separate rows.
+In the FITS ASCII table standard, each row is defined as a fixed number of
characters (value to the @code{NAXIS1} keyword), so to visually inspect it
properly, you would have to adjust your text editor's width to this value.
+All columns start at given character positions and have a fixed width (number
of characters).
-@table @option
-@item -n
-@itemx --numhdus
-Print the number of extensions/HDUs in the given file.
-Note that this option must be called alone and will only print a single number.
-It is thus useful in scripts, for example when you need to do check the number
of extensions in a FITS file.
+Numbers in a FITS ASCII table are printed into ASCII format, they are not in
binary (that the CPU uses).
+Hence, they can take a larger space in memory, loose their precision, and take
longer to read into memory.
+If you are dealing with integer type columns (see @ref{Numeric data types}),
another issue with FITS ASCII tables is that the type information for the
column will be lost (there is only one integer type in FITS ASCII tables).
+One problem with the binary format on the other hand is that it isn't portable
(different CPUs/compilers) have different standards for translating the zeros
and ones.
+But since ASCII characters are defined on a byte and are well recognized, they
are better for portability on those various systems.
+Gnuastro's plain text table format described below is much more portable and
easier to read/write/interpret by humans manually.
-For a complete list of basic meta-data on the extensions in a FITS file, don't
use any of the options in this section or in @ref{Keyword inspection and
manipulation}.
-For more, see @ref{Invoking astfits}.
+Generally, as the name implies, this format is useful for when your table
mainly contains ASCII columns (for example file names, or descriptions).
+They can be useful when you need to include columns with structured ASCII
information along with other extensions in one FITS file.
+In such cases, you can also consider header keywords (see @ref{Fits}).
-@item --hastablehdu
-Print @code{1} (on standard output) if at least one table HDU (ASCII or
binary) exists in the FITS file.
-Otherwise (when no table HDU exists in the file), print @code{0}.
+@cindex Binary table, FITS
+@item FITS binary tables
+The FITS binary table is the FITS standard's solution to the issues discussed
with keeping numbers in ASCII format as described under the FITS ASCII table
title above.
+Only columns defined as a string type (a string of ASCII characters) are
readable in a text editor.
+The portability problem with binary formats discussed above is mostly solved
thanks to the portability of CFITSIO (see @ref{CFITSIO}) and the very long
history of the FITS format which has been widely used since the 1970s.
-@item --listtablehdus
-Print the names or numbers (when a name doesn't exist, counting from zero) of
HDUs that contain a table (ASCII or Binary) on standard output, one per line.
-Otherwise (when no table HDU exists in the file) nothing will be printed.
+In the case of most numbers, storing them in binary format is more memory
efficient than ASCII format.
+For example, to store @code{-25.72034} in ASCII format, you need 9
bytes/characters.
+But if you keep this same number (to the approximate precision possible) as a
4-byte (32-bit) floating point number, you can keep/transmit it with less than
half the amount of memory.
+When catalogs contain thousands/millions of rows in tens/hundreds of columns,
this can lead to significant improvements in memory/band-width usage.
+Moreover, since the CPU does its operations in the binary formats, reading the
table in and writing it out is also much faster than an ASCII table.
-@item --hasimagehdu
-Print @code{1} (on standard output) if at least one image HDU exists in the
FITS file.
-Otherwise (when no image HDU exists in the file), print @code{0}.
+When you are dealing with integer numbers, the compression ratio can be even
better, for example if you know all of the values in a column are positive and
less than @code{255}, you can use the @code{unsigned char} type which only
takes one byte! If they are between @code{-128} and @code{127}, then you can
use the (signed) @code{char} type.
+So if you are thoughtful about the limits of your integer columns, you can
greatly reduce the size of your file and also the speed at which it is
read/written.
+This can be very useful when sharing your results with collaborators or
publishing them.
+To decrease the file size even more you can name your output as ending in
@file{.fits.gz} so it is also compressed after creation.
+Just note that compression/decompressing is CPU intensive and can slow down
the writing/reading of the file.
-In the FITS standard, any array with any dimensions is called an ``image'',
therefore this option includes 1, 3 and 4 dimensional arrays too.
-However, an image HDU with zero dimensions (which is usually the first
extension and only contains metadata) is not counted here.
+Fortunately the FITS Binary table format also accepts ASCII strings as column
types (along with the various numerical types).
+So your dataset can also contain non-numerical columns.
-@item --listimagehdus
-Print the names or numbers (when a name doesn't exist, counting from zero) of
HDUs that contain an image on standard output, one per line.
-Otherwise (when no image HDU exists in the file) nothing will be printed.
+@end table
-In the FITS standard, any array with any dimensions is called an ``image'',
therefore this option includes 1, 3 and 4 dimensional arrays too.
-However, an image HDU with zero dimensions (which is usually the first
extension and only contains metadata) is not counted here.
+@menu
+* Gnuastro text table format:: Reading plain text tables
+@end menu
-@item --listallhdus
-Print the names or numbers (when a name doesn't exist, counting from zero) of
all HDUs within the input file on the standard output, one per line.
+@node Gnuastro text table format, Selecting table columns, Recognized table
formats, Tables
+@subsection Gnuastro text table format
-@item --pixelscale
-Print the HDU's pixel-scale (change in world coordinate for one pixel along
each dimension) and pixel area or voxel volume.
-Without the @option{--quiet} option, the output of @option{--pixelscale} has
multiple lines and explanations, thus being more human-friendly.
-It prints the file/HDU name, number of dimensions, and the units along with
the actual pixel scales.
-Also, when any of the units are in degrees, the pixel scales and area/volume
are also printed in units of arc-seconds.
-For 3D datasets, the pixel area (on each 2D slice of the 3D cube) is printed
as well as the voxel volume.
+Plain text files are the most generic, portable, and easiest way to (manually)
create, (visually) inspect, or (manually) edit a table.
+In this format, the ending of a row is defined by the new-line character (a
line on a text editor).
+So when you view it on a text editor, every row will occupy one line.
+The delimiters (or characters separating the columns) are white space
characters (space, horizontal tab, vertical tab) and a comma (@key{,}).
+The only further requirement is that all rows/lines must have the same number
of columns.
-However, in scripts (that are to be run automatically), this human-friendly
format is annoying, so when called with the @option{--quiet} option, only the
pixel-scale value(s) along each dimension is(are) printed in one line.
-These numbers are followed by the pixel area (in the raw WCS units).
-For 3D datasets, this will be area on each 2D slice.
-Finally, for 3D datasets, a final number (the voxel volume) is printed.
-As a summary, in @option{--quiet} mode, for 2D datasets three numbers are
printed and for 3D datasets, 5 numbers are printed.
-If the dataset has more than 3 dimensions, only the pixel-scale values are
printed (no area or volume will be printed).
+The columns don't have to be exactly under each other and the rows can be
arbitrarily long with different lengths.
+For example the following contents in a file would be interpreted as a table
with 4 columns and 2 rows, with each element interpreted as a @code{double}
type (see @ref{Numeric data types}).
-@item --skycoverage
-@cindex Image's sky coverage
-@cindex Coverage of image over sky
-Print the rectangular area (or 3D cube) covered by the given image/datacube
HDU over the Sky in the WCS units.
-The covered area is reported in two ways:
-1) the center and full width in each dimension,
-2) the minimum and maximum sky coordinates in each dimension.
-This is option is thus useful when you want to get a general feeling of a new
image/dataset, or prepare the inputs to query external databases in the region
of the image (for example with @ref{Query}).
+@example
+1 2.234948 128 39.8923e8
+2 , 4.454 792 72.98348e7
+@end example
-If run without the @option{--quiet} option, the values are given with a
human-friendly description.
-For example here is the output of this option on an image taken near the star
Castor:
+However, the example above has no other information about the columns (it is
just raw data, with no meta-data).
+To use this table, you have to remember what the numbers in each column
represent.
+Also, when you want to select columns, you have to count their position within
the table.
+This can become frustrating and prone to bad errors (getting the columns
wrong) especially as the number of columns increase.
+It is also bad for sending to a colleague, because they will find it hard to
remember/use the columns properly.
-@example
-$ astfits castor.fits --skycoverage
-Input file: castor.fits (hdu: 1)
+To solve these problems in Gnuastro's programs/libraries you aren't limited to
using the column's number, see @ref{Selecting table columns}.
+If the columns have names, units, or comments you can also select your columns
based on searches/matches in these fields, for example see @ref{Table}.
+Also, in this manner, you can't guide the program reading the table on how to
read the numbers.
+As an example, the first and third columns above can be read as integer types:
the first column might be an ID and the third can be the number of pixels an
object occupies in an image.
+So there is no need to read these to columns as a @code{double} type (which
takes more memory, and is slower).
-Sky coverage by center and (full) width:
- Center: 113.9149075 31.93759664
- Width: 2.41762045 2.67945253
+In the bare-minimum example above, you also can't use strings of characters,
for example the names of filters, or some other identifier that includes
non-numerical characters.
+In the absence of any information, only numbers can be read robustly.
+Assuming we read columns with non-numerical characters as string, there would
still be the problem that the strings might contain space (or any delimiter)
character for some rows.
+So, each `word' in the string will be interpreted as a column and the program
will abort with an error that the rows don't have the same number of columns.
-Sky coverage by range along dimensions:
- RA 112.7235592 115.1411797
- DEC 30.59262123 33.27207376
-@end example
+To correct for these limitations, Gnuastro defines the following convention
for storing the table meta-data along with the raw data in one plain text file.
+The format is primarily designed for ease of reading/writing by eye/fingers,
but is also structured enough to be read by a program.
-With the @option{--quiet} option, the values are more machine-friendly (easy
to parse).
-It has two lines, where the first line contains the center/width values and
the second line shows the coordinate ranges in each dimension.
+When the first non-white character in a line is @key{#}, or there are no
non-white characters in it, then the line will not be considered as a row of
data in the table (this is a pretty standard convention in many programs, and
higher level languages).
+In the former case, the line is interpreted as a @emph{comment}.
+If the comment line starts with `@code{# Column N:}', then it is assumed to
contain information about column @code{N} (a number, counting from 1).
+Comment lines that don't start with this pattern are ignored and you can use
them to include any further information you want to store with the table in the
text file.
+A column information comment is assumed to have the following format:
@example
-$ astfits castor.fits --skycoverage --quiet
-113.9149075 31.93759664 2.41762045 2.67945253
-112.7235592 115.1411797 30.59262123 33.27207376
+# Column N: NAME [UNIT, TYPE, BLANK] COMMENT
@end example
-Note that this is a simple rectangle (cube in 3D) definition, so if the image
is rotated in relation to the celestial coordinates a general polygon is
necessary to exactly describe the coverage.
-Hence when there is rotation, the reported area will be larger than the actual
area containing data, you can visually see the area with the @option{--align}
option of @ref{Warp}.
+@cindex NaN
+@noindent
+Any sequence of characters between `@key{:}' and `@key{[}' will be interpreted
as the column name (so it can contain anything except the `@key{[}' character).
+Anything between the `@key{]}' and the end of the line is defined as a comment.
+Within the brackets, anything before the first `@key{,}' is the units
(physical units, for example km/s, or erg/s), anything before the second
`@key{,}' is the short type identifier (see below, and @ref{Numeric data
types}).
+If the type identifier isn't recognized, the default 64-bit floating point
type will be used.
-@item --datasum
-@cindex @code{DATASUM}: FITS keyword
-Calculate and print the given HDU's "datasum" to stdout.
-The given HDU is specified with the @option{--hdu} (or @option{-h}) option.
-This number is calculated by parsing all the bytes of the given HDU's data
records (excluding keywords).
-This option ignores any possibly existing @code{DATASUM} keyword in the HDU.
-For more on @code{DATASUM} in the FITS standard, see @ref{Keyword inspection
and manipulation} (under the @code{checksum} component of @option{--write}).
+Finally (still within the brackets), any non-white characters after the second
`@key{,}' are interpreted as the blank value for that column (see @ref{Blank
pixels}).
+The blank value can either be in the same type as the column (for example
@code{-99} for a signed integer column), or any string (for example @code{NaN}
in that same column).
+In both cases, the values will be stored in memory as Gnuastro's fixed blank
values for each type.
+For floating point types, Gnuastro's internal blank value is IEEE NaN
(Not-a-Number).
+For signed integers, it is the smallest possible value and for unsigned
integers its the largest possible value.
-You can use this option to confirm that the data in two different HDUs
(possibly with different keywords) is identical.
-Its advantage over @option{--write=datasum} (which writes the @code{DATASUM}
keyword into the given HDU) is that it doesn't require write permissions.
-@end table
+When a formatting problem occurs, or when the column was already given
meta-data in a previous comment, or when the column number is larger than the
actual number of columns in the table (the non-commented or empty lines), then
the comment information line will be ignored.
-The following options manipulate (move/delete) the HDUs in one FITS file or to
another FITS file.
-These options may be called multiple times in one run.
-If so, the extensions will be copied from the input FITS file to the output
FITS file in the given order (on the command-line and also in configuration
files, see @ref{Configuration file precedence}).
-If the separate classes are called together in one run of Fits, then first
@option{--copy} is run (on all specified HDUs), followed by @option{--cut}
(again on all specified HDUs), and then @option{--remove} (on all specified
HDUs).
+When a comment information line can be used, the leading and trailing white
space characters will be stripped from all of the elements.
+For example in this line:
-The @option{--copy} and @option{--cut} options need an output FITS file
(specified with the @option{--output} option).
-If the output file exists, then the specified HDU will be copied following the
last extension of the output file (the existing HDUs in it will be untouched).
-Thus, after Fits finishes, the copied HDU will be the last HDU of the output
file.
-If no output file name is given, then automatic output will be used to store
the HDUs given to this option (see @ref{Automatic output}).
+@example
+# Column 5: column name [km/s, f32,-99] Redshift as speed
+@end example
-@table @option
+The @code{NAME} field will be `@code{column name}' and the @code{TYPE} field
will be `@code{f32}'.
+Note how all the white space characters before and after strings are not used,
but those in the middle remained.
+Also, white space characters aren't mandatory.
+Hence, in the example above, the @code{BLANK} field will be given the value of
`@code{-99}'.
-@item -C STR
-@itemx --copy=STR
-Copy the specified extension into the output file, see explanations above.
+Except for the column number (@code{N}), the rest of the fields are optional.
+Also, the column information comments don't have to be in order.
+In other words, the information for column @mymath{N+m} (@mymath{m>0}) can be
given in a line before column @mymath{N}.
+Furthermore, you don't have to specify information for all columns.
+Those columns that don't have this information will be interpreted with the
default settings (like the case above: values are double precision floating
point, and the column has no name, unit, or comment).
+So these lines are all acceptable for any table (the first one, with nothing
but the column number is redundant):
-@item -k STR
-@itemx --cut=STR
-Cut (copy to output, remove from input) the specified extension into the
-output file, see explanations above.
+@example
+# Column 5:
+# Column 1: ID [,i8] The Clump ID.
+# Column 3: mag_f160w [AB mag, f32] Magnitude from the F160W filter
+@end example
-@item -R STR
-@itemx --remove=STR
-Remove the specified HDU from the input file.
+@noindent
+The data type of the column should be specified with one of the following
values:
-The first (zero-th) HDU cannot be removed with this option.
-Consider using @option{--copy} or @option{--cut} in combination with
@option{primaryimghdu} to not have an empty zero-th HDU.
-From CFITSIO: ``In the case of deleting the primary array (the first HDU in
the file) then [it] will be replaced by a null primary array containing the
minimum set of required keywords and no data.''.
-So in practice, any existing data (array) and meta-data in the first extension
will be removed, but the number of extensions in the file won't change.
-This is because of the unique position the first FITS extension has in the
FITS standard (for example it cannot be used to store tables).
+@itemize
+@item
+For a numeric column, you can use any of the numeric types (and their
+recognized identifiers) described in @ref{Numeric data types}.
+@item
+`@code{strN}': for strings.
+The @code{N} value identifies the length of the string (how many characters it
has).
+The start of the string on each row is the first non-delimiter character of
the column that has the string type.
+The next @code{N} characters will be interpreted as a string and all leading
and trailing white space will be removed.
-@item --primaryimghdu
-Copy or cut an image HDU to the zero-th HDU/extension a file that doesn't yet
exist.
-This option is thus irrelevant if the output file already exists or the
copied/cut extension is a FITS table.
-For example with the commands below, first we make sure that @file{out.fits}
doesn't exist, then we copy the first extension of @file{in.fits} to the
zero-th extension of @file{out.fits}.
+If the next column's characters, are closer than @code{N} characters to the
start of the string column in that line/row, they will be considered part of
the string column.
+If there is a new-line character before the ending of the space given to the
string column (in other words, the string column is the last column), then
reading of the string will stop, even if the @code{N} characters are not
complete yet.
+See @file{tests/table/table.txt} for one example.
+Therefore, the only time you have to pay attention to the positioning and
spaces given to the string column is when it is not the last column in the
table.
+
+The only limitation in this format is that trailing and leading white space
characters will be removed from the columns that are read.
+In most cases, this is the desired behavior, but if trailing and leading
white-spaces are critically important to your analysis, define your own
starting and ending characters and remove them after the table has been read.
+For example in the sample table below, the two `@key{|}' characters (which are
arbitrary) will remain in the value of the second column and you can remove
them manually later.
+If only one of the leading or trailing white spaces is important for your
work, you can only use one of the `@key{|}'s.
@example
-$ rm -f out.fits
-$ astfits in.fits --copy=1 --primaryimghdu --output=out.fits
+# Column 1: ID [label, u8]
+# Column 2: Notes [no unit, str50]
+1 leading and trailing white space is ignored here 2.3442e10
+2 | but they will be preserved here | 8.2964e11
@end example
-If we hadn't used @option{--primaryimghdu}, then the zero-th extension of
@file{out.fits} would have no data, and its second extension would host the
copied image (just like any other output of Gnuastro).
+@end itemize
-@end table
+Note that the FITS binary table standard does not define the @code{unsigned
int} and @code{unsigned long} types, so if you want to convert your tables to
FITS binary tables, use other types.
+Also, note that in the FITS ASCII table, there is only one integer type
(@code{long}).
+So if you convert a Gnuastro plain text table to a FITS ASCII table with the
@ref{Table} program, the type information for integers will be lost.
+Conversely if integer types are important for you, you have to manually set
them when reading a FITS ASCII table (for example with the Table program when
reading/converting into a file, or with the @file{gnuastro/table.h} library
functions when reading into memory).
-@node Keyword inspection and manipulation, , HDU information and
manipulation, Invoking astfits
-@subsubsection Keyword inspection and manipulation
-The meta-data in each header data unit, or HDU (also known as extension, see
@ref{Fits}) is stored as ``keyword''s.
-Each keyword consists of a name, value, unit, and comments.
-The Fits program (see @ref{Fits}) options related to viewing and manipulating
keywords in a FITS HDU are described below.
+@node Selecting table columns, , Gnuastro text table format, Tables
+@subsection Selecting table columns
-First, let's review the @option{--keyvalue} option which should be called
separately from the rest of the options described in this section.
-Also, unlike the rest of the options in this section, with
@option{--keyvalue}, you can give more than one input file.
+At the lowest level, the only defining aspect of a column in a table is its
number, or position.
+But selecting columns purely by number is not very convenient and, especially
when the tables are large it can be very frustrating and prone to errors.
+Hence, table file formats (for example see @ref{Recognized table formats})
have ways to store additional information about the columns (meta-data).
+Some of the most common pieces of information about each column are its
@emph{name}, the @emph{units} of data in it, and a @emph{comment} for
longer/informal description of the column's data.
-@table @option
-@item -l STR[,STR[,...]
-@itemx --keyvalue=STR[,STR[,...]
-Only print the value of the requested keyword(s): the @code{STR}s.
-@option{--keyvalue} can be called multiple times, and each call can contain
multiple comma-separated keywords.
-If more than one file is given, this option uses the same HDU/extension for
all of them (value to @option{--hdu}).
-For example, you can get the number of dimensions of the three FITS files in
the running directory, as well as the length along each dimension, with this
command:
+To facilitate research with Gnuastro, you can select columns by matching, or
searching in these three fields, besides the low-level column number.
+To view the full list of information on the columns in the table, you can use
the Table program (see @ref{Table}) with the command below (replace
@file{table-file} with the filename of your table, if its FITS, you might also
need to specify the HDU/extension which contains the table):
@example
-$ astfits *.fits --keyvalue=NAXIS,NAXIS1 --keyvalue=NAXIS2
-image-a.fits 2 774 672
-image-b.fits 2 774 672
-image-c.fits 2 387 336
+$ asttable --information table-file
@end example
-If only one input is given, and the @option{--quiet} option is activated, the
file name is not printed on the first column, only the values of the requested
keywords.
+Gnuastro's programs need the columns for different purposes, for example in
Crop, you specify the columns containing the central coordinates of the crop
centers with the @option{--coordcol} option (see @ref{Crop options}).
+On the other hand, in MakeProfiles, to specify the column containing the
profile position angles, you must use the @option{--pcol} option (see
@ref{MakeProfiles catalog}).
+Thus, there can be no unified common option name to select columns for all
programs (different columns have different purposes).
+However, when the program expects a column for a specific context, the option
names end in the @option{col} suffix like the examples above.
+These options accept values in integer (column number), or string (metadata
match/search) format.
-@example
-$ astfits image-a.fits --keyvalue=NAXIS,NAXIS1 \
- --keyvalue=NAXIS2 --quiet
-2 774 672
-@end example
+If the value can be parsed as a positive integer, it will be seen as the
low-level column number.
+Note that column counting starts from 1, so if you ask for column 0, the
respective program will abort with an error.
+When the value can't be interpreted as an a integer number, it will be seen as
a string of characters which will be used to match/search in the table's
meta-data.
+The meta-data field which the value will be compared with can be selected
through the @option{--searchin} option, see @ref{Input output options}.
+@option{--searchin} can take three values: @code{name}, @code{unit},
@code{comment}.
+The matching will be done following this convention:
-The output is internally stored (and finally printed) as a table (with one
column per keyword).
-Therefore just like the Table program, you can use @option{--colinfoinstdout}
to print the metadata like the example below (also see @ref{Invoking asttable}).
-The keyword metadata (comments and units) are extracted from the comments and
units of the keyword in the input files (first file that has a comment or unit).
-Hence if the keyword doesn't have units or comments in any of the input files,
they will be empty.
-For more on Gnuastro's plain-text metadata format, see @ref{Gnuastro text
table format}.
+@itemize
+@item
+If the value is enclosed in two slashes (for example @command{-x/RA_/}, or
@option{--coordcol=/RA_/}, see @ref{Crop options}), then it is assumed to be a
regular expression with the same convention as GNU AWK.
+GNU AWK has a very well written
@url{https://www.gnu.org/software/gawk/manual/html_node/Regexp.html, chapter}
describing regular expressions, so we will not continue discussing them here.
+Regular expressions are a very powerful tool in matching text and useful in
many contexts.
+We thus strongly encourage reviewing this chapter for greatly improving the
quality of your work in many cases, not just for searching column meta-data in
Gnuastro.
-@example
-$ astfits *.fits --keyvalue=NAXIS,NAXIS1,NAXIS2 \
- --colinfoinstdout
-# Column 1: FILENAME [name,str10,] Name of input file.
-# Column 2: NAXIS [ ,u8 ,] number of data axes
-# Column 3: NAXIS1 [ ,u16 ,] length of data axis 1
-# Column 4: NAXIS2 [ ,u16 ,] length of data axis 2
-image-a.fits 2 774 672
-image-b.fits 2 774 672
-image-c.fits 2 387 336
-@end example
+@item
+When the string isn't enclosed between `@key{/}'s, any column that exactly
matches the given value in the given field will be selected.
+@end itemize
-Another advantage of a table output is that you can directly write the the
table to a file.
-For example if you add @option{--output=fileinfo.fits}, the information above
will be printed into a FITS table.
-You can also pipe it into @ref{Table} to select files based on certain
properties, to sort them based on another property, or any other operation that
can be done with Table (including @ref{Column arithmetic}).
-For example with the command below, you can select all the files that have a
size larger than 500 pixels in both dimensions.
+Note that in both cases, you can ignore the case of alphabetic characters with
the @option{--ignorecase} option, see @ref{Input output options}.
+Also, in both cases, multiple columns may be selected with one call to this
function.
+In this case, the order of the selected columns (with one call) will be the
same order as they appear in the table.
-@example
-$ astfits *.fits --keyvalue=NAXIS,NAXIS1,NAXIS2 \
- --colinfoinstdout \
- | asttable --range=NAXIS1,500,inf \
- --range=NAXIS2,500,inf -cFILENAME
-image-a.fits
-image-b.fits
-@end example
-Note that @option{--colinfoinstdout} is necessary to use column names when
piping to other programs (like @command{asttable} above).
-Also, with the @option{-cFILENAME} option, we are asking Table to only print
the final file names (we don't need the sizes any more).
-The commands with multiple files above used @file{*.fits}, which is only
useful when all your FITS files are in the same directory.
-However, in many cases, your FITS files will be scattered in multiple
sub-directories of a certain top-level directory, or you may only want those
with more particular file name patterns.
-A more powerful way to list the input files to @option{--keyvalue} is to use
the @command{find} program in Unix-like operating systems.
-For example, with the command below you can search all the FITS files in all
the sub-directories of @file{/TOP/DIR}.
-@example
-astfits $(find /TOP/DIR/ -name "*.fits") --keyvalue=NAXIS2
-@end example
-@item -O
-@itemx --colinfoinstdout
-Print column information (or metadata) above the column values when writing
keyword values to standard output with @option{--keyvalue}.
-You can read this option as column-information-in-standard-output.
-@end table
+@node Tessellation, Automatic output, Tables, Common program behavior
+@section Tessellation
-Below we will discuss the options that can be used to manipulate keywords.
-To see the full list of keywords in a FITS HDU, you can use the
@option{--printallkeys} option.
-If any of the keyword modification options below are requested (for example
@option{--update}), the headers of the input file/HDU will be changed first,
then printed.
-Keyword modification is done within the input file.
-Therefore, if you want to keep the original FITS file or HDU intact, it is
easiest to create a copy of the file/HDU first and then run Fits on that (for
copying a HDU to another file, see @ref{HDU information and manipulation}.
-In the FITS standard, keywords are always uppercase.
-So case does not matter in the input or output keyword names you specify.
+It is sometimes necessary to classify the elements in a dataset (for example
pixels in an image) into a grid of individual, non-overlapping tiles.
+For example when background sky gradients are present in an image, you can
define a tile grid over the image.
+When the tile sizes are set properly, the background's variation over each
tile will be negligible, allowing you to measure (and subtract) it.
+In other cases (for example spatial domain convolution in Gnuastro, see
@ref{Convolve}), it might simply be for speed of processing: each tile can be
processed independently on a separate CPU thread.
+In the arts and mathematics, this process is formally known as
@url{https://en.wikipedia.org/wiki/Tessellation, tessellation}.
-@cartouche
-@noindent
-@strong{@code{CHECKSUM} automatically updated, when present:} the keyword
modification options will change the contents of the HDU.
-Therefore, if a @code{CHECKSUM} is present in the HDU, after all the keyword
modification options have been complete, Fits will also update @code{CHECKSUM}
before closing the file.
-@end cartouche
+The size of the regular tiles (in units of data-elements, or pixels in an
image) can be defined with the @option{--tilesize} option.
+It takes multiple numbers (separated by a comma) which will be the length
along the respective dimension (in FORTRAN/FITS dimension order).
+Divisions are also acceptable, but must result in an integer.
+For example @option{--tilesize=30,40} can be used for an image (a 2D dataset).
+The regular tile size along the first FITS axis (horizontal when viewed in SAO
DS9) will be 30 pixels and along the second it will be 40 pixels.
+Ideally, @option{--tilesize} should be selected such that all tiles in the
image have exactly the same size.
+In other words, that the dataset length in each dimension is divisible by the
tile size in that dimension.
-Most of the options can accept multiple instances in one command.
-For example you can add multiple keywords to delete by calling
@option{--delete} multiple times, since repeated keywords are allowed, you can
even delete the same keyword multiple times.
-The action of such options will start from the top most keyword.
+However, this is not always possible: the dataset can be any size and every
pixel in it is valuable.
+In such cases, Gnuastro will look at the significance of the remainder length,
if it is not significant (for example one or two pixels), then it will just
increase the size of the first tile in the respective dimension and allow the
rest of the tiles to have the required size.
+When the remainder is significant (for example one pixel less than the size
along that dimension), the remainder will be added to one regular tile's size
and the large tile will be cut in half and put in the two ends of the
grid/tessellation.
+In this way, all the tiles in the central regions of the dataset will have the
regular tile sizes and the tiles on the edge will be slightly larger/smaller
depending on the remainder significance.
+The fraction which defines the remainder significance along all dimensions can
be set through @option{--remainderfrac}.
-The precedence of operations are described below.
-Note that while the order within each class of actions is preserved, the order
of individual actions is not.
-So irrespective of what order you called @option{--delete} and
@option{--update}.
-First, all the delete operations are going to take effect then the update
operations.
-@enumerate
-@item
-@option{--delete}
-@item
-@option{--rename}
-@item
-@option{--update}
-@item
-@option{--write}
-@item
-@option{--asis}
-@item
-@option{--history}
-@item
-@option{--comment}
-@item
-@option{--date}
-@item
-@option{--printallkeys}
-@item
-@option{--verify}
-@item
-@option{--copykeys}
-@end enumerate
-@noindent
-All possible syntax errors will be reported before the keywords are actually
written.
-FITS errors during any of these actions will be reported, but Fits won't stop
until all the operations are complete.
-If @option{--quitonerror} is called, then Fits will immediately stop upon the
first error.
+The best tile size is directly related to the spatial properties of the
property you want to study (for example, gradient on the image).
+In practice we assume that the gradient is not present over each tile.
+So if there is a strong gradient (for example in long wavelength ground based
images) or the image is of a crowded area where there isn't too much blank
area, you have to choose a smaller tile size.
+A larger mesh will give more pixels and and so the scatter in the results will
be less (better statistics).
-@cindex GNU Grep
-If you want to inspect only a certain set of header keywords, it is easiest to
pipe the output of the Fits program to GNU Grep.
-Grep is a very powerful and advanced tool to search strings which is precisely
made for such situations.
-For example if you only want to check the size of an image FITS HDU, you can
run:
+@cindex CCD
+@cindex Amplifier
+@cindex Bias current
+@cindex Subaru Telescope
+@cindex Hyper Suprime-Cam
+@cindex Hubble Space Telescope (HST)
+For raw image processing, a single tessellation/grid is not sufficient.
+Raw images are the unprocessed outputs of the camera detectors.
+Modern detectors usually have multiple readout channels each with its own
amplifier.
+For example the Hubble Space Telescope Advanced Camera for Surveys (ACS) has
four amplifiers over its full detector area dividing the square field of view
to four smaller squares.
+Ground based image detectors are not exempt, for example each CCD of Subaru
Telescope's Hyper Suprime-Cam camera (which has 104 CCDs) has four amplifiers,
but they have the same height of the CCD and divide the width by four parts.
-@example
-$ astfits input.fits | grep NAXIS
-@end example
+@cindex Channel
+The bias current on each amplifier is different, and initial bias subtraction
is not perfect.
+So even after subtracting the measured bias current, you can usually still
identify the boundaries of different amplifiers by eye.
+See Figure 11(a) in Akhlaghi and Ichikawa (2015) for an example.
+This results in the final reduced data to have non-uniform amplifier-shaped
regions with higher or lower background flux values.
+Such systematic biases will then propagate to all subsequent measurements we
do on the data (for example photometry and subsequent stellar mass and star
formation rate measurements in the case of galaxies).
-@cartouche
-@noindent
-@strong{FITS STANDARD KEYWORDS:}
-Some header keywords are necessary for later operations on a FITS file, for
example BITPIX or NAXIS, see the FITS standard for their full list.
-If you modify (for example remove or rename) such keywords, the FITS file
extension might not be usable any more.
-Also be careful for the world coordinate system keywords, if you modify or
change their values, any future world coordinate system (like RA and Dec)
measurements on the image will also change.
-@end cartouche
+Therefore an accurate analysis requires a two layer tessellation: the top
layer contains larger tiles, each covering one amplifier channel.
+For clarity we'll call these larger tiles ``channels''.
+The number of channels along each dimension is defined through the
@option{--numchannels}.
+Each channel is then covered by its own individual smaller tessellation (with
tile sizes determined by the @option{--tilesize} option).
+This will allow independent analysis of two adjacent pixels from different
channels if necessary.
+If the image is processed or the detector only has one amplifier, you can set
the number of channels in both dimension to 1.
+The final tessellation can be inspected on the image with the
@option{--checktiles} option that is available to all programs which use
tessellation for localized operations.
+When this option is called, a FITS file with a @file{_tiled.fits} suffix will
be created along with the outputs, see @ref{Automatic output}.
+Each pixel in this image has the number of the tile that covers it.
+If the number of channels in any dimension are larger than unity, you will
notice that the tile IDs are defined such that the first channels is covered
first, then the second and so on.
+For the full list of processing-related common options (including tessellation
options), please see @ref{Processing options}.
-@noindent
-The keyword related options to the Fits program are fully described below.
-@table @option
-@item -d STR
-@itemx --delete=STR
-Delete one instance of the @option{STR} keyword from the FITS header.
-Multiple instances of @option{--delete} can be given (possibly even for the
same keyword, when its repeated in the meta-data).
-All keywords given will be removed from the headers in the same given order.
-If the keyword doesn't exist, Fits will give a warning and return with a
non-zero value, but will not stop.
-To stop as soon as an error occurs, run with @option{--quitonerror}.
-@item -r STR,STR
-@itemx --rename=STR,STR
-Rename a keyword to a new value (for example @option{--rename=OLDNAME,NEWNAME}.
-@option{STR} contains both the existing and new names, which should be
separated by either a comma (@key{,}) or a space character.
-Note that if you use a space character, you have to put the value to this
option within double quotation marks (@key{"}) so the space character is not
interpreted as an option separator.
-Multiple instances of @option{--rename} can be given in one command.
-The keywords will be renamed in the specified order.
-If the keyword doesn't exist, Fits will give a warning and return with a
non-zero value, but will not stop.
-To stop as soon as an error occurs, run with @option{--quitonerror}.
-@item -u STR
-@itemx --update=STR
-Update a keyword, its value, its comments and its units in the format
described below.
-If there are multiple instances of the keyword in the header, they will be
changed from top to bottom (with multiple @option{--update} options).
-@noindent
-The format of the values to this option can best be specified with an
-example:
+@node Automatic output, Output FITS files, Tessellation, Common program
behavior
+@section Automatic output
-@example
---update=KEYWORD,value,"comments for this keyword",unit
-@end example
+@cindex Standard input
+@cindex Automatic output file names
+@cindex Output file names, automatic
+@cindex Setting output file names automatically
+All the programs in Gnuastro are designed such that specifying an output file
or directory (based on the program context) is optional.
+When no output name is explicitly given (with @option{--output}, see
@ref{Input output options}), the programs will automatically set an output name
based on the input name(s) and what the program does.
+For example when you are using ConvertType to save FITS image named
@file{dataset.fits} to a JPEG image and don't specify a name for it, the JPEG
output file will be name @file{dataset.jpg}.
+When the input is from the standard input (for example a pipe, see
@ref{Standard input}), and @option{--output} isn't given, the output name will
be the program's name (for example @file{converttype.jpg}).
-If there is a writing error, Fits will give a warning and return with a
non-zero value, but will not stop.
-To stop as soon as an error occurs, run with @option{--quitonerror}.
+@vindex --keepinputdir
+Another very important part of the automatic output generation is that all the
directory information of the input file name is stripped off of it.
+This feature can be disabled with the @option{--keepinputdir} option, see
@ref{Input output options}.
+It is the default because astronomical data are usually very large and
organized specially with special file names.
+In some cases, the user might not have write permissions in those
directories@footnote{In fact, even if the data is stored on your own computer,
it is advised to only grant write permissions to the super user or root.
+This way, you won't accidentally delete or modify your valuable data!}.
-@noindent
-The value can be any numerical or string value@footnote{Some tricky situations
arise with values like `@command{87095e5}', if this was intended to be a number
it will be kept in the header as @code{8709500000} and there is no problem.
-But this can also be a shortened Git commit hash.
-In the latter case, it should be treated as a string and stored as it is
written.
-Commit hashes are very important in keeping the history of a file during your
research and such values might arise without you noticing them in your
reproduction pipeline.
-One solution is to use @command{git describe} instead of the short hash alone.
-A less recommended solution is to add a space after the commit hash and Fits
will write the value as `@command{87095e5 }' in the header.
-If you later compare the strings on the shell, the space character will be
ignored by the shell in the latter solution and there will be no problem.}.
-Other than the @code{KEYWORD}, all the other values are optional.
-To leave a given token empty, follow the preceding comma (@key{,}) immediately
with the next.
-If any space character is present around the commas, it will be considered
part of the respective token.
-So if more than one token has space characters within it, the safest method to
specify a value to this option is to put double quotation marks around each
individual token that needs it.
-Note that without double quotation marks, space characters will be seen as
option separators and can lead to undefined behavior.
-
-@item -w STR
-@itemx --write=STR
-Write a keyword to the header.
-For the possible value input formats, comments and units for the keyword, see
the @option{--update} option above.
-The special names (first string) below will cause a special behavior:
-
-@table @option
-
-@item /
-Write a ``title'' to the list of keywords.
-A title consists of one blank line and another which is blank for several
spaces and starts with a slash (@key{/}).
-The second string given to this option is the ``title'' or string printed
after the slash.
-For example with the command below you can add a ``title'' of `My keywords'
after the existing keywords and add the subsequent @code{K1} and @code{K2}
keywords under it (note that keyword names are not case sensitive).
+Let's assume that we are working on a report and want to process the FITS
images from two projects (ABC and DEF), which are stored in the sub-directories
named @file{ABCproject/} and @file{DEFproject/} of our top data directory
(@file{/mnt/data}).
+The following shell commands show how one image from the former is first
converted to a JPEG image through ConvertType and then the objects from an
image in the latter project are detected using NoiseChisel.
+The text after the @command{#} sign are comments (not typed!).
@example
-$ astfits test.fits -h1 --write=/,"My keywords" \
- --write=k1,1.23,"My first keyword" \
- --write=k2,4.56,"My second keyword"
-$ astfits test.fits -h1
-[[[ ... truncated ... ]]]
-
- / My keywords
-K1 = 1.23 / My first keyword
-K2 = 4.56 / My second keyword
-END
+$ pwd # Current location
+/home/usrname/research/report
+$ ls # List directory contents
+ABC01.jpg
+$ ls /mnt/data/ABCproject # Archive 1
+ABC01.fits ABC02.fits ABC03.fits
+$ ls /mnt/data/DEFproject # Archive 2
+DEF01.fits DEF02.fits DEF03.fits
+$ astconvertt /mnt/data/ABCproject/ABC02.fits --output=jpg # Prog 1
+$ ls
+ABC01.jpg ABC02.jpg
+$ astnoisechisel /mnt/data/DEFproject/DEF01.fits # Prog 2
+$ ls
+ABC01.jpg ABC02.jpg DEF01_detected.fits
@end example
-Adding a ``title'' before each contextually separate group of header keywords
greatly helps in readability and visual inspection of the keywords.
-So generally, when you want to add new FITS keywords, its good practice to
also add a title before them.
-The reason you need to use @key{/} as the keyword name for setting a title is
that @key{/} is the first non-white character.
-The title(s) is(are) written into the FITS with the same order that
@option{--write} is called.
-Therefore in one run of the Fits program, you can specify many different
titles (with their own keywords under them).
-For example the command below that builds on the previous example and adds
another group of keywords named @code{A1} and @code{A2}.
-@example
-$ astfits test.fits -h1 --write=/,"My keywords" \
- --write=k1,1.23,"My first keyword" \
- --write=k2,4.56,"My second keyword" \
- --write=/,"My second group of keywords" \
- --write=a1,7.89,"First keyword" \
- --write=a2,0.12,"Second keyword"
-@end example
-@item checksum
-@cindex CFITSIO
-@cindex @code{DATASUM}: FITS keyword
-@cindex @code{CHECKSUM}: FITS keyword
-When nothing is given afterwards, the header integrity keywords @code{DATASUM}
and @code{CHECKSUM} will be calculated and written/updated.
-The calculation and writing is done fully by CFITSIO, therefore they comply
with the FITS standard
4.0@footnote{@url{https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf}}
that defines these keywords (its Appendix J).
+@node Output FITS files, , Automatic output, Common program behavior
+@section Output FITS files
-If a value is given (e.g., @option{--write=checksum,MyOwnCheckSum}), then
CFITSIO won't be called to calculate these two keywords and the value (as well
as possible comment and unit) will be written just like any other keyword.
-This is generally not recommended since @code{CHECKSUM} is a reserved FITS
standard keyword.
-If you want to calculate the checksum with another hashing standard manually
and write it into the header, its is recommended to use another keyword name.
+@cindex FITS
+@cindex Output FITS headers
+@cindex CFITSIO version on outputs
+The output of many of Gnuastro's programs are (or can be) FITS files.
+The FITS format has many useful features for storing scientific datasets
(cubes, images and tables) along with a robust features for archivability.
+For more on this standard, please see @ref{Fits}.
-In the FITS standard, @code{CHECKSUM} depends on the HDU's data @emph{and}
header keywords, it will therefore not be valid if you make any further changes
to the header after writing the @code{CHECKSUM} keyword.
-This includes any further keyword modification options in the same call to the
Fits program.
-However, @code{DATASUM} only depends on the data section of the HDU/extension,
so it is not changed when you add, remove or update the header keywords.
-Therefore, it is recommended to write these keywords as the last keywords that
are written/modified in the extension.
-You can use the @option{--verify} option (described below) to verify the
values of these two keywords.
+As a community convention described in @ref{Fits}, the first extension of all
FITS files produced by Gnuastro's programs only contains the meta-data that is
intended for the file's extension(s).
+For a Gnuastro program, this generic meta-data (that is stored as FITS keyword
records) is its configuration when it produced this dataset: file name(s) of
input(s) and option names, values and comments.
+Note that when the configuration is too trivial (only input filename, for
example the program @ref{Table}) no meta-data is written in this extension.
-@item datasum
-Similar to @option{checksum}, but only write the @code{DATASUM} keyword (that
doesn't depend on the header keywords, only the data).
-@end table
+FITS keywords have the following limitations in regards to generic option
names and values which are described below:
-@item -a STR
-@itemx --asis=STR
-Write the given @code{STR} @emph{exactly} as it is, into the given FITS file
header with no modifications.
-If the contents of @code{STR} does not conform to the FITS standard for
keywords, then it may (most probably: it will) corrupt your file and you may
not be able to open it any more.
-So please be @strong{very careful} with this option.
-If you want to define the keyword from scratch, it is best to use the
@option{--write} option (see below) and let CFITSIO worry about complying with
the FITS standard.
+@itemize
+@item
+If a keyword (option name) is longer than 8 characters, the first word in the
record (80 character line) is @code{HIERARCH} which is followed by the keyword
name.
-The best way to use this option is when you want to add a keyword from one
FITS file to another, unchanged and untouched.
-Below is an example of such a case that can be very useful sometimes (on the
command-line or in scripts):
+@item
+Values can be at most 75 characters, but for strings, this changes to 73
(because of the two extra @key{'} characters that are necessary).
+However, if the value is a file name, containing slash (@key{/}) characters to
separate directories, Gnuastro will break the value into multiple keywords.
+
+@item
+Keyword names ignore case, therefore they are all in capital letters.
+Therefore, if you want to use Grep to inspect these keywords, use the
@option{-i} option, like the example below.
@example
-$ key=$(astfits firstimage.fits | grep KEYWORD)
-$ astfits --asis="$key" secondimage.fits
+$ astfits image_detected.fits -h0 | grep -i snquant
@end example
+@end itemize
-@cindex GNU Bash
-In particular note the double quotation signs (@key{"}) around the shell
variable (@command{$key}).
-This is because FITS keyword strings usually have lots of space characters, if
this variable is not quoted, the shell will only give the first word in the
full keyword to this option, which will definitely be a non-standard FITS
keyword and will make it hard to work on the file afterwords.
-See the ``Quoting'' section of the GNU Bash manual for more information if
your keyword has the special characters @key{$}, @key{`}, or @key{\}.
+The keywords above are classified (separated by an empty line and title) as a
group titled ``ProgramName configuration''.
+This meta-data extension, as well as all the other extensions (which contain
data), also contain have final group of keywords to keep the basic date and
version information of Gnuastro, its dependencies and the pipeline that is
using Gnuastro (if its under version control).
-You can also use @option{--asis} to copy multiple keywords from one file to
another.
-But the process will be a little more complicated. So we'll show the process
as the simple shell script below.
-You can customize it in the first block of variable definitions:
-1) set the names of the keywords you want to copy: it can be as many keys as
you want, just put a `@code{\|}' between them.
-2) The input FITS file (where keywords should be read from) and its HDU.
-3) The output FITS file (where keywords should be written to) and its HDU.
-4) Set a ``title'' for the newly added keywords in the output (so they are
visually separate from the existing keywords in the output).
+@table @command
-@example
-#!/bin/bash
+@item DATE
+The creation time of the FITS file.
+This date is written directly by CFITSIO and is in UT format.
-# Customizations (input, output and key names).
-# NOTE: put a '\|' between each keyword name.
-keys="KEYa\|KEYb\|KEYc\|KEYd"
-ifits=original.fits; ihdu=1
-ofits=to_write.fits; ohdu=1
-title="Keys from $ifits (hdu $ihdu)"
+@item COMMIT
+Git's commit description from the running directory of Gnuastro's programs.
+If the running directory is not version controlled or @file{libgit2} isn't
installed (see @ref{Optional dependencies}) then this keyword will not be
present.
+The printed value is equivalent to the output of the following command:
-# Read keywords from input and write in output.
-oIFS=$IFS
-IFS=''
-c="astfits $ofits -h$ohdu --write=/,\"$title\""
-astfits $ifits -h$ihdu \
- | grep $keys \
- | (while read line; \
- do c="$c --asis=\"$line\""; \
- done; eval $c); \
-IFS=$oIFS
+@example
+git describe --dirty --always
@end example
-Since its not too long, you can also simply put the variable values of the
first block into the second, and write it directly on the command line (if its
just for one time).
+If the running directory contains non-committed work, then the stored value
will have a `@command{-dirty}' suffix.
+This can be very helpful to let you know that the data is not ready to be
shared with collaborators or submitted to a journal.
+You should only share results that are produced after all your work is
committed (safely stored in the version controlled history and thus
reproducible).
-@item -H STR
-@itemx --history STR
-Add a @code{HISTORY} keyword to the header with the given value. A new
@code{HISTORY} keyword will be created for every instance of this option. If
the string given to this option is longer than 70 characters, it will be
separated into multiple keyword cards. If there is an error, Fits will give a
warning and return with a non-zero value, but will not stop. To stop as soon as
an error occurs, run with @option{--quitonerror}.
+At first sight, version control appears to be mainly a tool for software
developers.
+However progress in a scientific research is almost identical to progress in
software development: first you have a rough idea that starts with handful of
easy steps.
+But as the first results appear to be promising, you will have to extend, or
generalize, it to make it more robust and work in all the situations your
research covers, not just your first test samples.
+Slowly you will find wrong assumptions or bad implementations that need to be
fixed (`bugs' in software development parlance).
+Finally, when you submit the research to your collaborators or a journal, many
comments and suggestions will come in, and you have to address them.
-@item -c STR
-@itemx --comment STR
-Add a @code{COMMENT} keyword to the header with the given value.
-Similar to the explanation for @option{--history} above.
+Software developers have created version control systems precisely for this
kind of activity.
+Each significant moment in the project's history is called a ``commit'', see
@ref{Version controlled source}.
+A snapshot of the project in each ``commit'' is safely stored away, so you can
revert back to it at a later time, or check changes/progress.
+This way, you can be sure that your work is reproducible and track the
progress and history.
+With version control, experimentation in the project's analysis is greatly
facilitated, since you can easily revert back if a brainstorm test procedure
fails.
-@item -t
-@itemx --date
-Put the current date and time in the header.
-If the @code{DATE} keyword already exists in the header, it will be updated.
-If there is a writing error, Fits will give a warning and return with a
non-zero value, but will not stop.
-To stop as soon as an error occurs, run with @option{--quitonerror}.
+One important feature of version control is that the research result (FITS
image, table, report or paper) can be stamped with the unique commit
information that produced it.
+This information will enable you to exactly reproduce that same result later,
even if you have made changes/progress.
+For one example of a research paper's reproduction pipeline, please see the
@url{https://gitlab.com/makhlaghi/NoiseChisel-paper, reproduction pipeline} of
the @url{https://arxiv.org/abs/1505.01664, paper} describing @ref{NoiseChisel}.
-@item -p
-@itemx --printallkeys
-Print the full meta data (keywords, values, units and comments) in the
specified FITS extension (HDU).
-If this option is called along with any of the other keyword editing commands,
as described above, all other editing commands take precedence to this.
-Therefore, it will print the final keywords after all the editing has been
done.
+@item CFITSIO
+The version of CFITSIO used (see @ref{CFITSIO}).
-@item --printkeynames
-Print only the keyword names of the specified FITS extension (HDU), one line
per name.
-This option must be called alone.
+@item WCSLIB
+The version of WCSLIB used (see @ref{WCSLIB}).
+Note that older versions of WCSLIB do not report the version internally.
+So this is only available if you are using more recent WCSLIB versions.
-@item -v
-@itemx --verify
-Verify the @code{DATASUM} and @code{CHECKSUM} data integrity keywords of the
FITS standard.
-See the description under the @code{checksum} (under @option{--write}, above)
for more on these keywords.
+@item GSL
+The version of GNU Scientific Library that was used, see @ref{GNU Scientific
Library}.
-This option will print @code{Verified} for both keywords if they can be
verified.
-Otherwise, if they don't exist in the given HDU/extension, it will print
@code{NOT-PRESENT}, and if they cannot be verified it will print
@code{INCORRECT}.
-In the latter case (when the keyword values exist but can't be verified), the
Fits program will also return with a failure.
+@item GNUASTRO
+The version of Gnuastro used (see @ref{Version numbering}).
+@end table
-By default this function will also print a short description of the
@code{DATASUM} AND @code{CHECKSUM} keywords.
-You can suppress this extra information with @code{--quiet} option.
+Here is one example of the last few lines of an example output.
-@item --copykeys=INT:INT
-Copy the input's keyword records in the given range (inclusive) to the
-output HDU (specified with the @option{--output} and @option{--outhdu}
-options, for the filename and HDU/extension respectively).
+@example
+ / Versions and date
+DATE = '...' / file creation date
+COMMIT = 'v0-8-g547f6eb' / Commit description in running dir.
+CFITSIO = '3.45 ' / CFITSIO version.
+WCSLIB = '5.19 ' / WCSLIB version.
+GSL = '2.5 ' / GNU Scientific Library version.
+GNUASTRO= '0.7 ' / GNU Astronomy Utilities version.
+END
+@end example
-The given string to this option must be two integers separated by a colon
(@key{:}).
-The first integer must be positive (counting of the keyword records starts
from 1).
-The second integer may be negative (zero is not acceptable) or an integer
larger than the first.
-A negative second integer means counting from the end.
-So @code{-1} is the last copy-able keyword (not including the @code{END}
keyword).
-To see the header keywords of the input with a number before them, you can
pipe the output of the FITS program (when it prints all the keywords in an
extension) into the @command{cat} program like below:
-@example
-$ astfits input.fits -h1 | cat -n
-@end example
-@item --outhdu
-The HDU/extension to write the output keywords of @option{--copykeys}.
-@item -Q
-@itemx --quitonerror
-Quit if any of the operations above are not successful.
-By default if an error occurs, Fits will warn the user of the faulty keyword
and continue with the rest of actions.
-@item -s STR
-@itemx --datetosec STR
-@cindex Unix epoch time
-@cindex Time, Unix epoch
-@cindex Epoch, Unix time
-Interpret the value of the given keyword in the FITS date format (most
generally: @code{YYYY-MM-DDThh:mm:ss.ddd...}) and return the corresponding Unix
epoch time (number of seconds that have passed since 00:00:00 Thursday, January
1st, 1970).
-The @code{Thh:mm:ss.ddd...} section (specifying the time of day), and also the
@code{.ddd...} (specifying the fraction of a second) are optional.
-The value to this option must be the FITS keyword name that contains the
requested date, for example @option{--datetosec=DATE-OBS}.
-@cindex GNU C Library
-This option can also interpret the older FITS date format
(@code{DD/MM/YYThh:mm:ss.ddd...}) where only two characters are given to the
year.
-In this case (following the GNU C Library), this option will make the
following assumption: values 68 to 99 correspond to the years 1969 to 1999, and
values 0 to 68 as the years 2000 to 2068.
-This is a very useful option for operations on the FITS date values, for
example sorting FITS files by their dates, or finding the time difference
between two FITS files.
-The advantage of working with the Unix epoch time is that you don't have to
worry about calendar details (for example the number of days in different
months, or leap years, etc).
-@item --wcscoordsys=STR
-@cindex Galactic coordinate system
-@cindex Ecliptic coordinate system
-@cindex Equatorial coordinate system
-@cindex Supergalactic coordinate system
-@cindex Coordinate system: Galactic
-@cindex Coordinate system: Ecliptic
-@cindex Coordinate system: Equatorial
-@cindex Coordinate system: Supergalactic
-Convert the coordinate system of the image's world coordinate system (WCS) to
the given coordinate system (@code{STR}) and write it into the file given to
@option{--output} (or an automatically named file if no @option{--output} has
been given).
-For example with the command below, @file{img-eq.fits} will have an identical
dataset (pixel values) as @file{image.fits}.
-However, the WCS coordinate system of @file{img-eq.fits} will be the
equatorial coordinate system in the Julian calendar epoch 2000 (which is the
most common epoch used today).
-Fits will automatically extract the current coordinate system of
@file{image.fits} and as long as its one of the recognized coordinate systems
listed below, it will do the conversion.
-@example
-$ astfits image.fits --coordsys=eq-j2000 --output=img-eq.fits
-@end example
-The currently recognized coordinate systems are listed below (the most common
one today is @code{eq-j2000}):
-@table @code
-@item eq-j2000
-2000.0 (Julian-year) equatorial coordinates.
-@item eq-b1950
-1950.0 (Besselian-year) equatorial coordinates.
-@item ec-j2000
-2000.0 (Julian-year) ecliptic coordinates.
-@item ec-b1950
-1950.0 (Besselian-year) ecliptic coordinates.
-@item galactic
-Galactic coordinates.
-@item supergalactic
-Supergalactic coordinates.
-@end table
+@node Data containers, Data manipulation, Common program behavior, Top
+@chapter Data containers
-The Equatorial and Ecliptic coordinate systems are defined by the mean equator
and equinox epoch: either the Besselian year 1950.0, or the Julian year 2000.
-For more on their difference and links for further reading about epochs in
astronomy, please see the description in
@url{https://en.wikipedia.org/wiki/Epoch_(astronomy), Wikipedia}.
+@cindex File operations
+@cindex Operations on files
+@cindex General file operations
+The most low-level and basic property of a dataset is how it is stored.
+To process, archive and transmit the data, you need a container to store it
first.
+From the start of the computer age, different formats have been defined to
store data, optimized for particular applications.
+One format/container can never be useful for all applications: the storage
defines the application and vice-versa.
+In astronomy, the Flexible Image Transport System (FITS) standard has become
the most common format of data storage and transmission.
+It has many useful features, for example multiple sub-containers (also known
as extensions or header data units, HDUs) within one file, or support for
tables as well as images.
+Each HDU can store an independent dataset and its corresponding meta-data.
+Therefore, Gnuastro has one program (see @ref{Fits}) specifically designed to
manipulate FITS HDUs and the meta-data (header keywords) in each HDU.
-@item --wcsdistortion=STR
-@cindex WCS distortion
-@cindex Distortion, WCS
-@cindex SIP WCS distortion
-@cindex TPV WCS distortion
-If the argument has a WCS distortion, the output (file given with the
@option{--output} option) will have the distortion given to this option (for
example @code{SIP}, @code{TPV}).
-The output will be a new file (with a copy of the image, and the new WCS), so
if it already exists, the file will be delete (unless you use the
@code{--dontdelete} option, see @ref{Input output options}).
+Your astronomical research does not just involve data analysis (where the FITS
format is very useful).
+For example you want to demonstrate your raw and processed FITS images or
spectra as figures within slides, reports, or papers.
+The FITS format is not defined for such applications.
+Thus, Gnuastro also comes with the ConvertType program (see @ref{ConvertType})
which can be used to convert a FITS image to and from (where possible) other
formats like plain text and JPEG (which allow two way conversion), along with
EPS and PDF (which can only be created from FITS, not the other way round).
-With this option, the FITS program will read the minimal set of keywords from
the input HDU and the HDU data, it will then write them into the file given to
the @option{--output} option but with a newly created set of WCS-related
keywords corresponding to the desired distortion standard.
+Finally, the FITS format is not just for images, it can also store tables.
+Binary tables in particular can be very efficient in storing catalogs that
have more than a few tens of columns and rows.
+However, unlike images (where all elements/pixels have one data type), tables
contain multiple columns and each column can have different properties:
independent data types (see @ref{Numeric data types}) and meta-data.
+In practice, each column can be viewed as a separate container that is grouped
with others in the table.
+The only shared property of the columns in a table is thus the number of
elements they contain.
+To allow easy inspection/manipulation of table columns, Gnuastro has the Table
program (see @ref{Table}).
+It can be used to select certain table columns in a FITS table and see them as
a human readable output on the command-line, or to save them into another plain
text or FITS table.
-If no @option{--output} file is specified, an automatically generated output
name will be used which is composed of the input's name but with the
@file{-DDD.fits} suffix, see @ref{Automatic output}.
-Where @file{DDD} is the value given to this option (desired output distortion).
+@menu
+* Fits:: View and manipulate extensions and keywords.
+* ConvertType:: Convert data to various formats.
+* Table:: Read and Write FITS tables to plain text.
+* Query:: Import data from external databases.
+@end menu
-Note that all possible conversions between all standards are not yet supported.
-If the requested conversion is not supported, an informative error message
will be printed.
-If this happens, please let us know and we'll try our best to add the
respective conversions.
-For example with the command below, you can be sure that if @file{in.fits} has
a distortion in its WCS, the distortion of @file{out.fits} will be in the SIP
standard.
-@example
-$ astfits in.fits --wcsdistortion=SIP --output=out.fits
-@end example
-@end table
+@node Fits, ConvertType, Data containers, Data containers
+@section Fits
+@cindex Vatican library
+The ``Flexible Image Transport System'', or FITS, is by far the most common
data container format in astronomy and in constant use since the 1970s.
+Archiving (future usage, simplicity) has been one of the primary design
principles of this format.
+In the last few decades it has proved so useful and robust that the Vatican
Library has also chosen FITS for its ``long-term digital preservation''
project@footnote{@url{https://www.vaticanlibrary.va/home.php?pag=progettodigit}}.
+@cindex IAU, international astronomical union
+Although the full name of the standard invokes the idea that it is only for
images, it also contains complete and robust features for tables.
+It started off in the 1970s and was formally published as a standard in 1981,
it was adopted by the International Astronomical Union (IAU) in 1982 and an IAU
working group to maintain its future was defined in 1988.
+The FITS 2.0 and 3.0 standards were approved in 2000 and 2008 respectively,
and the 4.0 draft has also been released recently, please see the
@url{https://fits.gsfc.nasa.gov/fits_standard.html, FITS standard document web
page} for the full text of all versions.
+Also see the @url{https://doi.org/10.1051/0004-6361/201015362, FITS 3.0
standard paper} for a nice introduction and history along with the full
standard.
+@cindex Meta-data
+Many common image formats, for example a JPEG, only have one image/dataset per
file, however one great advantage of the FITS standard is that it allows you to
keep multiple datasets (images or tables along with their separate meta-data)
in one file.
+In the FITS standard, each data + metadata is known as an extension, or more
formally a header data unit or HDU.
+The HDUs in a file can be completely independent: you can have multiple images
of different dimensions/sizes or tables as separate extensions in one file.
+However, while the standard doesn't impose any constraints on the relation
between the datasets, it is strongly encouraged to group data that are
contextually related with each other in one file.
+For example an image and the table/catalog of objects and their measured
properties in that image.
+Other examples can be images of one patch of sky in different colors
(filters), or one raw telescope image along with its calibration data (tables
or images).
+As discussed above, the extensions in a FITS file can be completely
independent.
+To keep some information (meta-data) about the group of extensions in the FITS
file, the community has adopted the following convention: put no data in the
first extension, so it is just meta-data.
+This extension can thus be used to store Meta-data regarding the whole file
(grouping of extensions).
+Subsequent extensions may contain data along with their own separate meta-data.
+All of Gnuastro's programs also follow this convention: the main output
dataset(s) are placed in the second (or later) extension(s).
+The first extension contains no data the program's configuration (input file
name, along with all its option values) are stored as its meta-data, see
@ref{Output FITS files}.
+The meta-data contain information about the data, for example which region of
the sky an image corresponds to, the units of the data, what telescope, camera,
and filter the data were taken with, it observation date, or the software that
produced it and its configuration.
+Without the meta-data, the raw dataset is practically just a collection of
numbers and really hard to understand, or connect with the real world (other
datasets).
+It is thus strongly encouraged to supplement your data (at any level of
processing) with as much meta-data about your processing/science as possible.
+The meta-data of a FITS file is in ASCII format, which can be easily viewed or
edited with a text editor or on the command-line.
+Each meta-data element (known as a keyword generally) is composed of a name,
value, units and comments (the last two are optional).
+For example below you can see three FITS meta-data keywords for specifying the
world coordinate system (WCS, or its location in the sky) of a dataset:
+@example
+LATPOLE = -27.805089 / [deg] Native latitude of celestial pole
+RADESYS = 'FK5' / Equatorial coordinate system
+EQUINOX = 2000.0 / [yr] Equinox of equatorial coordinates
+@end example
+However, there are some limitations which discourage viewing/editing the
keywords with text editors.
+For example there is a fixed length of 80 characters for each keyword (its
name, value, units and comments) and there are no new-line characters, so on a
text editor all the keywords are seen in one line.
+Also, the meta-data keywords are immediately followed by the data which are
commonly in binary format and will show up as strange looking characters on a
text editor, and significantly slowing down the processor.
+Gnuastro's Fits program was designed to allow easy manipulation of FITS
extensions and meta-data keywords on the command-line while conforming fully
with the FITS standard.
+For example you can copy or cut (copy and remove) HDUs/extensions from one
FITS file to another, or completely delete them.
+It also has features to delete, add, or edit meta-data keywords within one HDU.
+@menu
+* Invoking astfits:: Arguments and options to Header.
+@end menu
+@node Invoking astfits, , Fits, Fits
+@subsection Invoking Fits
+Fits can print or manipulate the FITS file HDUs (extensions), meta-data
keywords in a given HDU.
+The executable name is @file{astfits} with the following general template
+@example
+$ astfits [OPTION...] ASTRdata
+@end example
+@noindent
+One line examples:
+@example
+## View general information about every extension:
+$ astfits image.fits
+## Print the header keywords in the second HDU (counting from 0):
+$ astfits image.fits -h1
-@node ConvertType, Table, Fits, Data containers
-@section ConvertType
+## Only print header keywords that contain `NAXIS':
+$ astfits image.fits -h1 | grep NAXIS
-@cindex Data format conversion
-@cindex Converting data formats
-@cindex Image format conversion
-@cindex Converting image formats
-@pindex @r{ConvertType (}astconvertt@r{)}
-The FITS format used in astronomy was defined mainly for archiving,
transmission, and processing.
-In other situations, the data might be useful in other formats.
-For example, when you are writing a paper or report, or if you are making
slides for a talk, you can't use a FITS image.
-Other image formats should be used.
-In other cases you might want your pixel values in a table format as plain
text for input to other programs that don't recognize FITS.
-ConvertType is created for such situations.
-The various types will increase with future updates and based on need.
+## Only print the WCS standard PC matrix elements
+$ astfits image.fits -h1 | grep 'PC._.'
-The conversion is not only one way (from FITS to other formats), but two ways
(except the EPS and PDF formats@footnote{Because EPS and PDF are vector, not
raster/pixelated formats}).
-So you can also convert a JPEG image or text file into a FITS image.
-Basically, other than EPS/PDF, you can use any of the recognized formats as
different color channel inputs to get any of the recognized outputs.
-So before explaining the options and arguments (in @ref{Invoking
astconvertt}), we'll start with a short description of the recognized files
types in @ref{Recognized file formats}, followed a short introduction to
digital color in @ref{Color}.
+## Copy a HDU from input.fits to out.fits:
+$ astfits input.fits --copy=hdu-name --output=out.fits
-@menu
-* Recognized file formats:: Recognized file formats
-* Color:: Some explanations on color.
-* Aligning images with small WCS offsets:: When the WCS slightly differs.
-* Annotations for figure in paper:: Adding coordinates or physical scale.
-* Invoking astconvertt:: Options and arguments to ConvertType.
-@end menu
+## Update the OLDKEY keyword value to 153.034:
+$ astfits --update=OLDKEY,153.034,"Old keyword comment"
-@node Recognized file formats, Color, ConvertType, ConvertType
-@subsection Recognized file formats
+## Delete one COMMENT keyword and add a new one:
+$ astfits --delete=COMMENT --comment="Anything you like ;-)."
-The various standards and the file name extensions recognized by ConvertType
are listed below.
-Currently Gnuastro uses the file name's suffix to identify the format.
+## Write two new keywords with different values and comments:
+$ astfits --write=MYKEY1,20.00,"An example keyword" --write=MYKEY2,fd
+@end example
-@table @asis
-@item FITS or IMH
-@cindex IRAF
-@cindex Astronomical data format
-Astronomical data are commonly stored in the FITS format (or the older data
IRAF @file{.imh} format), a list of file name suffixes which indicate that the
file is in this format is given in @ref{Arguments}.
+@cindex HDU
+@cindex HEALPix
+When no action is requested (and only a file name is given), Fits will print a
list of information about the extension(s) in the file.
+This information includes the HDU number, HDU name (@code{EXTNAME} keyword),
type of data (see @ref{Numeric data types}, and the number of data elements it
contains (size along each dimension for images and table rows and columns).
+Optionally, a comment column is printed for special situations (like a 2D
HEALPix grid that is usually stored as a 1D dataset/table).
+You can use this to get a general idea of the contents of the FITS file and
what HDU to use for further processing, either with the Fits program or any
other Gnuastro program.
-Each image extension of a FITS file only has one value per pixel/element.
-Therefore, when used as input, each input FITS image contributes as one color
channel.
-If you want multiple extensions in one FITS file for different color channels,
you have to repeat the file name multiple times and use the @option{--hdu},
@option{--hdu2}, @option{--hdu3} or @option{--hdu4} options to specify the
different extensions.
+Here is one example of information about a FITS file with four extensions: the
first extension has no data, it is a purely meta-data HDU (commonly used to
keep meta-data about the whole file, or grouping of extensions, see @ref{Fits}).
+The second extension is an image with name @code{IMAGE} and single precision
floating point type (@code{float32}, see @ref{Numeric data types}), it has 4287
pixels along its first (horizontal) axis and 4286 pixels along its second
(vertical) axis.
+The third extension is also an image with name @code{MASK}.
+It is in 2-byte integer format (@code{int16}) which is commonly used to keep
information about pixels (for example to identify which ones were saturated, or
which ones had cosmic rays and so on), note how it has the same size as the
@code{IMAGE} extension.
+The third extension is a binary table called @code{CATALOG} which has 12371
rows and 5 columns (it probably contains information about the sources in the
image).
-@item JPEG
-@cindex JPEG format
-@cindex Raster graphics
-@cindex Pixelated graphics
-The JPEG standard was created by the Joint photographic experts group.
-It is currently one of the most commonly used image formats.
-Its major advantage is the compression algorithm that is defined by the
standard.
-Like the FITS standard, this is a raster graphics format, which means that it
is pixelated.
+@example
+GNU Astronomy Utilities X.X
+Run on Day Month DD HH:MM:SS YYYY
+-----
+HDU (extension) information: `image.fits'.
+ Column 1: Index (counting from 0).
+ Column 2: Name (`EXTNAME' in FITS standard).
+ Column 3: Image data type or `table' format (ASCII or binary).
+ Column 4: Size of data in HDU.
+-----
+0 n/a uint8 0
+1 IMAGE float32 4287x4286
+2 MASK int16 4287x4286
+3 CATALOG table_binary 12371x5
+@end example
-A JPEG file can have 1 (for gray-scale), 3 (for RGB) and 4 (for CMYK) color
channels.
-If you only want to convert one JPEG image into other formats, there is no
problem, however, if you want to use it in combination with other input files,
make sure that the final number of color channels does not exceed four.
-If it does, then ConvertType will abort and notify you.
+If a specific HDU is identified on the command-line with the @option{--hdu}
(or @option{-h} option) and no operation requested, then the full list of
header keywords in that HDU will be printed (as if the @option{--printallkeys}
was called, see below).
+It is important to remember that this only occurs when @option{--hdu} is given
on the command-line.
+The @option{--hdu} value given in a configuration file will only be used when
a specific operation on keywords requested.
+Therefore as described in the paragraphs above, when no explicit call to the
@option{--hdu} option is made on the command-line and no operation is requested
(on the command-line or configuration files), the basic information of each
HDU/extension is printed.
-@cindex Suffixes, JPEG images
-The file name endings that are recognized as a JPEG file for input are:
-@file{.jpg}, @file{.JPG}, @file{.jpeg}, @file{.JPEG}, @file{.jpe},
@file{.jif}, @file{.jfif} and @file{.jfi}.
+The operating mode and input/output options to Fits are similar to the other
programs and fully described in @ref{Common options}.
+The options particular to Fits can be divided into two groups:
+1) those related to modifying HDUs or extensions (see @ref{HDU information and
manipulation}), and
+2) those related to viewing/modifying meta-data keywords (see @ref{Keyword
inspection and manipulation}).
+These two classes of options cannot be called together in one run: you can
either work on the extensions or meta-data keywords in any instance of Fits.
-@item TIFF
-@cindex TIFF format
-TIFF (or Tagged Image File Format) was originally designed as a common format
for scanners in the early 90s and since then it has grown to become very
general.
-In many aspects, the TIFF standard is similar to the FITS image standard: it
can allow data of many types (see @ref{Numeric data types}), and also allows
multiple images to be stored in a single file (each image in the file is called
a `directory' in the TIFF standard).
-However, unlike FITS, it can only store images, it has no constructs for
tables.
-Another (inconvenient) difference with the FITS standard is that keyword names
are stored as numbers, not human-readable text.
+@menu
+* HDU information and manipulation:: Learn about the HDUs and move them.
+* Keyword inspection and manipulation:: Manipulate metadata keywords in a HDU
+@end menu
-However, outside of astronomy, because of its support of different numeric
-data types, many fields use TIFF images for accurate (for example 16-bit
-integer or floating point for example) imaging data.
-Currently ConvertType can only read TIFF images, if you are interested in
-writing TIFF images, please get in touch with us.
-@item EPS
-@cindex EPS
-@cindex PostScript
-@cindex Vector graphics
-@cindex Encapsulated PostScript
-The Encapsulated PostScript (EPS) format is essentially a one page PostScript
file which has a specified size.
-PostScript also includes non-image data, for example lines and texts.
-It is a fully functional programming language to describe a document.
-Therefore in ConvertType, EPS is only an output format and cannot be used as
input.
-Contrary to the FITS or JPEG formats, PostScript is not a raster format, but
is categorized as vector graphics.
-@cindex PDF
-@cindex Adobe systems
-@cindex PostScript vs. PDF
-@cindex Compiled PostScript
-@cindex Portable Document format
-@cindex Static document description format
-The Portable Document Format (PDF) is currently the most common format for
documents.
-Some believe that PDF has replaced PostScript and that PostScript is now
obsolete.
-This view is wrong, a PostScript file is an actual plain text file that can be
edited like any program source with any text editor.
-To be able to display its programmed content or print, it needs to pass
through a processor or compiler.
-A PDF file can be thought of as the processed output of the compiler on an
input PostScript file.
-PostScript, EPS and PDF were created and are registered by Adobe Systems.
-@cindex @TeX{}
-@cindex @LaTeX{}
-With these features in mind, you can see that when you are compiling a
document with @TeX{} or @LaTeX{}, using an EPS file is much more low level than
a JPEG and thus you have much greater control and therefore quality.
-Since it also includes vector graphic lines we also use such lines to make a
thin border around the image to make its appearance in the document much better.
-No matter the resolution of the display or printer, these lines will always be
clear and not pixelated.
-In the future, addition of text might be included (for example labels or
object IDs) on the EPS output.
-However, this can be done better with tools within @TeX{} or @LaTeX{} such as
PGF/Tikz@footnote{@url{http://sourceforge.net/projects/pgf/}}.
+@node HDU information and manipulation, Keyword inspection and manipulation,
Invoking astfits, Invoking astfits
+@subsubsection HDU information and manipulation
+Each FITS file header data unit, or HDU (also known as an extension) is an
independent dataset (data + meta-data).
+Multiple HDUs can be stored in one FITS file, see @ref{Fits}.
+The general HDU-related options to the Fits program are listed below as two
general classes:
+the first group below focus on HDU information while the latter focus on
manipulating (moving or deleting) the HDUs.
-@cindex Binary image
-@cindex Saving binary image
-@cindex Black and white image
-If the final input image (possibly after all operations on the flux explained
below) is a binary image or only has two colors of black and white (in
segmentation maps for example), then PostScript has another great advantage
compared to other formats.
-It allows for 1 bit pixels (pixels with a value of 0 or 1), this can decrease
the output file size by 8 times.
-So if a gray-scale image is binary, ConvertType will exploit this property in
the EPS and PDF (see below) outputs.
+The options below print information about the given HDU on the command-line.
+Thus they cannot be called together in one command (each has its own
independent output).
-@cindex Suffixes, EPS format
-The standard formats for an EPS file are @file{.eps}, @file{.EPS},
@file{.epsf} and @file{.epsi}.
-The EPS outputs of ConvertType have the @file{.eps} suffix.
+@table @option
+@item -n
+@itemx --numhdus
+Print the number of extensions/HDUs in the given file.
+Note that this option must be called alone and will only print a single number.
+It is thus useful in scripts, for example when you need to do check the number
of extensions in a FITS file.
-@item PDF
-@cindex Suffixes, PDF format
-@cindex GPL Ghostscript
-As explained above, a PDF document is a static document description format,
viewing its result is therefore much faster and more efficient than PostScript.
-To create a PDF output, ConvertType will make a PostScript page description
and convert that to PDF using GPL Ghostscript.
-The suffixes recognized for a PDF file are: @file{.pdf}, @file{.PDF}.
-If GPL Ghostscript cannot be run on the PostScript file, it will remain and a
warning will be printed.
+For a complete list of basic meta-data on the extensions in a FITS file, don't
use any of the options in this section or in @ref{Keyword inspection and
manipulation}.
+For more, see @ref{Invoking astfits}.
-@item @option{blank}
-@cindex @file{blank} color channel
-This is not actually a file type! But can be used to fill one color channel
with a blank value.
-If this argument is given for any color channel, that channel will not be used
in the output.
+@item --hastablehdu
+Print @code{1} (on standard output) if at least one table HDU (ASCII or
binary) exists in the FITS file.
+Otherwise (when no table HDU exists in the file), print @code{0}.
-@item Plain text
-@cindex Plain text
-@cindex Suffixes, plain text
-Plain text files have the advantage that they can be viewed with any text
editor or on the command-line.
-Most programs also support input as plain text files.
-As input, each plain text file is considered to contain one color channel.
+@item --listtablehdus
+Print the names or numbers (when a name doesn't exist, counting from zero) of
HDUs that contain a table (ASCII or Binary) on standard output, one per line.
+Otherwise (when no table HDU exists in the file) nothing will be printed.
-In ConvertType, the recognized extensions for plain text files are @file{.txt}
and @file{.dat}.
-As described in @ref{Invoking astconvertt}, if you just give these extensions,
(and not a full filename) as output, then automatic output will be preformed to
determine the final output name (see @ref{Automatic output}).
-Besides these, when the format of a file cannot be recognized from its name,
ConvertType will fall back to plain text mode.
-So you can use any name (even without an extension) for a plain text input or
output.
-Just note that when the suffix is not recognized, automatic output will not be
preformed.
+@item --hasimagehdu
+Print @code{1} (on standard output) if at least one image HDU exists in the
FITS file.
+Otherwise (when no image HDU exists in the file), print @code{0}.
-The basic input/output on plain text images is very similar to how tables are
read/written as described in @ref{Gnuastro text table format}.
-Simply put, the restrictions are very loose, and there is a convention to
define a name, units, data type (see @ref{Numeric data types}), and comments
for the data in a commented line.
-The only difference is that as a table, a text file can contain many datasets
(columns), but as a 2D image, it can only contain one dataset.
-As a result, only one information comment line is necessary for a 2D image,
and instead of the starting `@code{# Column N}' (@code{N} is the column
number), the information line for a 2D image must start with `@code{# Image 1}'.
-When ConvertType is asked to output to plain text file, this information
comment line is written before the image pixel values.
+In the FITS standard, any array with any dimensions is called an ``image'',
therefore this option includes 1, 3 and 4 dimensional arrays too.
+However, an image HDU with zero dimensions (which is usually the first
extension and only contains metadata) is not counted here.
-When converting an image to plain text, consider the fact that if the image is
large, the number of columns in each line will become very large, possibly
making it very hard to open in some text editors.
+@item --listimagehdus
+Print the names or numbers (when a name doesn't exist, counting from zero) of
HDUs that contain an image on standard output, one per line.
+Otherwise (when no image HDU exists in the file) nothing will be printed.
-@item Standard output (command-line)
-This is very similar to the plain text output, but instead of creating a file
to keep the printed values, they are printed on the command line.
-This can be very useful when you want to redirect the results directly to
another program in one command with no intermediate file.
-The only difference is that only the pixel values are printed (with no
information comment line).
-To print to the standard output, set the output name to `@file{stdout}'.
+In the FITS standard, any array with any dimensions is called an ``image'',
therefore this option includes 1, 3 and 4 dimensional arrays too.
+However, an image HDU with zero dimensions (which is usually the first
extension and only contains metadata) is not counted here.
-@end table
+@item --listallhdus
+Print the names or numbers (when a name doesn't exist, counting from zero) of
all HDUs within the input file on the standard output, one per line.
-@node Color, Aligning images with small WCS offsets, Recognized file formats,
ConvertType
-@subsection Color
+@item --pixelscale
+Print the HDU's pixel-scale (change in world coordinate for one pixel along
each dimension) and pixel area or voxel volume.
+Without the @option{--quiet} option, the output of @option{--pixelscale} has
multiple lines and explanations, thus being more human-friendly.
+It prints the file/HDU name, number of dimensions, and the units along with
the actual pixel scales.
+Also, when any of the units are in degrees, the pixel scales and area/volume
are also printed in units of arc-seconds.
+For 3D datasets, the pixel area (on each 2D slice of the 3D cube) is printed
as well as the voxel volume.
-@cindex RGB
-@cindex CMYK
-@cindex Image
-@cindex Color
-@cindex Pixels
-@cindex Colormap
-@cindex Primary colors
-Color is defined by mixing various measurements/filters.
-In digital monitors or common digital cameras, colors are displayed/stored by
mixing the three basic colors of red, green and blue (RGB) with various
proportions.
-When printing on paper, standard printers use the cyan, magenta, yellow and
key (CMYK, key=black) color space.
-In other words, for each displayed/printed pixel of a color image, the
dataset/image has three or four values.
+However, in scripts (that are to be run automatically), this human-friendly
format is annoying, so when called with the @option{--quiet} option, only the
pixel-scale value(s) along each dimension is(are) printed in one line.
+These numbers are followed by the pixel area (in the raw WCS units).
+For 3D datasets, this will be area on each 2D slice.
+Finally, for 3D datasets, a final number (the voxel volume) is printed.
+As a summary, in @option{--quiet} mode, for 2D datasets three numbers are
printed and for 3D datasets, 5 numbers are printed.
+If the dataset has more than 3 dimensions, only the pixel-scale values are
printed (no area or volume will be printed).
-@cindex Color channel
-@cindex Channel, color
-To store/show the three values for each pixel, cameras and monitors allocate a
certain fraction of each pixel's area to red, green and blue filters.
-These three filters are thus built into the hardware at the pixel level.
-However, because measurement accuracy is very important in scientific
instruments, and we want to do measurements (take images) with various/custom
filters (without having to order a new expensive detector!), scientific
detectors use the full area of the pixel to store one value for it in a
single/mono channel dataset.
-To make measurements in different filters, we just place a filter in the light
path before the detector.
-Therefore, the FITS format that is used to store astronomical datasets is
inherently a mono-channel format (see @ref{Recognized file formats} or
@ref{Fits}).
+@item --skycoverage
+@cindex Image's sky coverage
+@cindex Coverage of image over sky
+Print the rectangular area (or 3D cube) covered by the given image/datacube
HDU over the Sky in the WCS units.
+The covered area is reported in two ways:
+1) the center and full width in each dimension,
+2) the minimum and maximum sky coordinates in each dimension.
+This is option is thus useful when you want to get a general feeling of a new
image/dataset, or prepare the inputs to query external databases in the region
of the image (for example with @ref{Query}).
-When a subject has been imaged in multiple filters, you can feed each
different filter into the red, green and blue channels and obtain a colored
visualization.
-In ConvertType, you can do this by giving each separate single-channel dataset
(for example in the FITS image format) as an argument (in the proper order),
then asking for the output in a format that supports multi-channel datasets
(for example JPEG or PDF, see the examples in @ref{Invoking astconvertt}).
+If run without the @option{--quiet} option, the values are given with a
human-friendly description.
+For example here is the output of this option on an image taken near the star
Castor:
-@cindex Grayscale
-@cindex Visualization
-@cindex Colormap, HSV
-@cindex Colormap, gray-scale
-@cindex HSV: Hue Saturation Value
-As discussed above, color is not defined when a dataset/image contains a
single value for each pixel.
-However, we interact with scientific datasets through monitors or printers
(which allow multiple values per pixel and produce color with them).
-As a result, there is a lot of freedom in visualizing a single-channel dataset.
-The most basic is to use shades of black (because of its strong contrast with
white).
-This scheme is called grayscale.
-To help in visualization, more complex mappings can be defined.
-For example, the values can be scaled to a range of 0 to 360 and used as the
``Hue'' term of the @url{https://en.wikipedia.org/wiki/HSL_and_HSV,
Hue-Saturation-Value} (HSV) color space (while fixing the ``Saturation'' and
``Value'' terms).
-In ConvertType, you can use the @option{--colormap} option to choose between
different mappings of mono-channel inputs, see @ref{Invoking astconvertt}.
+@example
+$ astfits castor.fits --skycoverage
+Input file: castor.fits (hdu: 1)
-Since grayscale is a commonly used mapping of single-valued datasets, we'll
continue with a closer look at how it is stored.
-One way to represent a gray-scale image in different color spaces is to use
the same proportions of the primary colors in each pixel.
-This is the common way most FITS image viewers work: for each pixel, they fill
all the channels with the single value.
-While this is necessary for displaying a dataset, there are downsides when
storing/saving this type of grayscale visualization (for example in a paper).
+Sky coverage by center and (full) width:
+ Center: 113.9149075 31.93759664
+ Width: 2.41762045 2.67945253
-@itemize
+Sky coverage by range along dimensions:
+ RA 112.7235592 115.1411797
+ DEC 30.59262123 33.27207376
+@end example
-@item
-Three (for RGB) or four (for CMYK) values have to be stored for every pixel,
this makes the output file very heavy (in terms of bytes).
+With the @option{--quiet} option, the values are more machine-friendly (easy
to parse).
+It has two lines, where the first line contains the center/width values and
the second line shows the coordinate ranges in each dimension.
-@item
-If printing, the printing errors of each color channel can make the printed
image slightly more blurred than it actually is.
+@example
+$ astfits castor.fits --skycoverage --quiet
+113.9149075 31.93759664 2.41762045 2.67945253
+112.7235592 115.1411797 30.59262123 33.27207376
+@end example
-@end itemize
+Note that this is a simple rectangle (cube in 3D) definition, so if the image
is rotated in relation to the celestial coordinates a general polygon is
necessary to exactly describe the coverage.
+Hence when there is rotation, the reported area will be larger than the actual
area containing data, you can visually see the area with the @option{--align}
option of @ref{Warp}.
-@cindex PNG standard
-@cindex Single channel CMYK
-To solve both these problems when storing grayscale visualization, the best
way is to save a single-channel dataset into the black channel of the CMYK
color space.
-The JPEG standard is the only common standard that accepts CMYK color space.
+@item --datasum
+@cindex @code{DATASUM}: FITS keyword
+Calculate and print the given HDU's "datasum" to stdout.
+The given HDU is specified with the @option{--hdu} (or @option{-h}) option.
+This number is calculated by parsing all the bytes of the given HDU's data
records (excluding keywords).
+This option ignores any possibly existing @code{DATASUM} keyword in the HDU.
+For more on @code{DATASUM} in the FITS standard, see @ref{Keyword inspection
and manipulation} (under the @code{checksum} component of @option{--write}).
-The JPEG and EPS standards set two sizes for the number of bits in each
channel: 8-bit and 12-bit.
-The former is by far the most common and is what is used in ConvertType.
-Therefore, each channel should have values between 0 to @math{2^8-1=255}.
-From this we see how each pixel in a gray-scale image is one byte (8 bits)
long, in an RGB image, it is 3 bytes long and in CMYK it is 4 bytes long.
-But thanks to the JPEG compression algorithms, when all the pixels of one
channel have the same value, that channel is compressed to one pixel.
-Therefore a Grayscale image and a CMYK image that has only the K-channel
filled are approximately the same file size.
+You can use this option to confirm that the data in two different HDUs
(possibly with different keywords) is identical.
+Its advantage over @option{--write=datasum} (which writes the @code{DATASUM}
keyword into the given HDU) is that it doesn't require write permissions.
+@end table
+The following options manipulate (move/delete) the HDUs in one FITS file or to
another FITS file.
+These options may be called multiple times in one run.
+If so, the extensions will be copied from the input FITS file to the output
FITS file in the given order (on the command-line and also in configuration
files, see @ref{Configuration file precedence}).
+If the separate classes are called together in one run of Fits, then first
@option{--copy} is run (on all specified HDUs), followed by @option{--cut}
(again on all specified HDUs), and then @option{--remove} (on all specified
HDUs).
-@node Aligning images with small WCS offsets, Annotations for figure in paper,
Color, ConvertType
-@subsection Aligning images with small WCS offsets
+The @option{--copy} and @option{--cut} options need an output FITS file
(specified with the @option{--output} option).
+If the output file exists, then the specified HDU will be copied following the
last extension of the output file (the existing HDUs in it will be untouched).
+Thus, after Fits finishes, the copied HDU will be the last HDU of the output
file.
+If no output file name is given, then automatic output will be used to store
the HDUs given to this option (see @ref{Automatic output}).
-In order to have nice color images, it is important that the images be
properly aligned.
-This is usually the case in many scenarios, but it some times happens that the
images have a small WCS offset, even though they have the same size.
-In such cases you can use the script below to align the images into
approximately the same pixel grid (to within about 0.5 pixels which is
sufficient in many color-image usage scenarios).
+@table @option
-The script below does the job using Gnuastro's @ref{Warp} and @ref{Crop}
programs.
-Simply copy the lines below into a plain-text file with your favorite text
editor and save it as @file{my-align.sh}.
-Don't forget to set the variables of the first three lines to specify the file
names (without the @file{.fits} suffix) and the HDUs of your inputs.
-These four lines are all you need to edit, leave the rest unchanged.
-Also, if you are copy/pasting the script from a PDF, be careful that the
single-quotes used in AWK may need to be corrected.
+@item -C STR
+@itemx --copy=STR
+Copy the specified extension into the output file, see explanations above.
-@example
-#!/bin/sh
+@item -k STR
+@itemx --cut=STR
+Cut (copy to output, remove from input) the specified extension into the
+output file, see explanations above.
-# Set the input names (without the '.fits' suffix),
-# and their HDUs.
-r=RED_IMAGE_NO_SUFFIX; rhdu=1
-g=GREEN_IMAGE_NO_SUFFIX; ghdu=1
-b=BLUE_IMAGE_NO_SUFFIX; bhdu=1
+@item -R STR
+@itemx --remove=STR
+Remove the specified HDU from the input file.
-# To stop the script if there is a crash
-set -e
+The first (zero-th) HDU cannot be removed with this option.
+Consider using @option{--copy} or @option{--cut} in combination with
@option{primaryimghdu} to not have an empty zero-th HDU.
+From CFITSIO: ``In the case of deleting the primary array (the first HDU in
the file) then [it] will be replaced by a null primary array containing the
minimum set of required keywords and no data.''.
+So in practice, any existing data (array) and meta-data in the first extension
will be removed, but the number of extensions in the file won't change.
+This is because of the unique position the first FITS extension has in the
FITS standard (for example it cannot be used to store tables).
-# Align all the images to the celestial poles.
-astwarp $r.fits --align -h$rhdu -o $r-aligned.fits
-astwarp $g.fits --align -h$ghdu -o $g-aligned.fits
-astwarp $b.fits --align -h$bhdu -o $b-aligned.fits
-
-# Calculate the final WCS-based center and image-based width based on
-# the G-band (in RGB) image.
-centerwcs=$(astfits $g-aligned.fits --skycoverage --quiet \
- | awk 'NR==1@{printf "%g %g", $1,$2@}')
-widthpix=$(astfits $g-aligned.fits -h1 --quiet \
- --keyvalue=NAXIS1,NAXIS2 \
- | awk '@{printf "%d,%d", $1, $2@}')
-
-# Crop all the images around the desired center and width.
-for f in $r $g $b; do
- centerpix=$(echo $centerwcs \
- | asttable -c'arith $1 $2 wcs-to-img' \
- --wcsfile=$f-aligned.fits \
- | awk '@{printf "%g,%g", $1, $2@}')
- astcrop $f-aligned.fits --mode=img --width=$widthpix \
- --center=$centerpix -o$f-use.fits
- rm $f-aligned.fits
-done
-@end example
-
-Once you have have saved the file and come back to your command-line you can
run the script like this:
+@item --primaryimghdu
+Copy or cut an image HDU to the zero-th HDU/extension a file that doesn't yet
exist.
+This option is thus irrelevant if the output file already exists or the
copied/cut extension is a FITS table.
+For example with the commands below, first we make sure that @file{out.fits}
doesn't exist, then we copy the first extension of @file{in.fits} to the
zero-th extension of @file{out.fits}.
@example
-$ chmod +x my-align.sh
-$ ./my-align.sh
+$ rm -f out.fits
+$ astfits in.fits --copy=1 --primaryimghdu --output=out.fits
@end example
-@noindent
-Of course, feel free to hack it and modify it to fit your datasets, like the
rest of Gnuastro, this script is released under GNU GPLv.3 and above, see
@ref{Your rights}.
-
-@node Annotations for figure in paper, Invoking astconvertt, Aligning images
with small WCS offsets, ConvertType
-@subsection Annotations for figure in paper
+If we hadn't used @option{--primaryimghdu}, then the zero-th extension of
@file{out.fits} would have no data, and its second extension would host the
copied image (just like any other output of Gnuastro).
-@cindex Image annotation
-@cindex Annotation of images for paper
-To make a nice figure from your FITS images, it is important to show more than
merely the raw image (converted to a printer friendly format like PDF or JPEG).
-Annotations (or visual metadata) over the raw image greatly help the readers
clearly see your argument and put the image/result in a larger context.
-Examples include:
-@itemize
-@item
-Coordinates (Right Ascension and Declination) on the edges of the image, so
viewers of your paper or presentation slides can get a physical feeling of the
field's sky coverage.
-@item
-Thick line that has a fixed tangential size (for example in kilo parsecs) at
the redshift/distance of interest.
-@item
-Contours over the image to show radio/X-ray emission, over an optical image
for example.
-@item
-Text or arrows or etc, over certain parts of the image.
-@end itemize
+@end table
-@cindex PGFPlots
-Because of the modular philosophy of Gnuastro, ConvertType is only focused on
converting your FITS images to printer friendly formats like JPEG or PDF.
-But to present your results in a slide or paper, you will often need to
annotate the raw JPEG or PDF with some of the features above.
-The good news is that there are many powerful plotting programs that you can
use to add such annotations.
-As a result, there is no point in making a new one, specific to Gnuastro.
-In this section, we'll demonstrate this using the very powerful
PGFPlots@footnote{@url{http://mirrors.ctan.org/graphics/pgf/contrib/pgfplots/doc/pgfplots.pdf}}
package of @LaTeX{}.
-@cartouche
-@noindent
-@strong{Single script for easy running:} In this section we are reviewing the
reason and details of every step which is good for educational purposes.
-But when you know the steps already, these separate code blocks can be
annoying.
-Therefore the full script (except for the data download step) is available in
@ref{Full script of annotations on figure}.
-@end cartouche
+@node Keyword inspection and manipulation, , HDU information and
manipulation, Invoking astfits
+@subsubsection Keyword inspection and manipulation
+The meta-data in each header data unit, or HDU (also known as extension, see
@ref{Fits}) is stored as ``keyword''s.
+Each keyword consists of a name, value, unit, and comments.
+The Fits program (see @ref{Fits}) options related to viewing and manipulating
keywords in a FITS HDU are described below.
-@cindex TiKZ
-@cindex Matplotlib
-PGFPlots uses the same @LaTeX{} graphic engine that typesets your paper/slide.
-Therefore when you build your plots and figures using PGFPlots (and its
underlying package
PGF/TiKZ@footnote{@url{http://mirrors.ctan.org/graphics/pgf/base/doc/pgfmanual.pdf}})
your plots will blend beautifully within your text: same fonts, same colors,
same line properties and etc.
-Since most papers (and presentation slides@footnote{To build slides, @LaTeX{}
has packages like Beamer, see
@url{http://mirrors.ctan.org/macros/latex/contrib/beamer/doc/beameruserguide.pdf}})
are made with @LaTeX{}, PGFPlots is therefore the best tool for those who use
@LaTeX{} to create documents.
-PGFPlots also doesn't need any extra dependencies beyond a basic/minimal
@TeX{}-live installation, so it is much more reliable than tools like
Matplotlib in Python that have hundreds of fast-evolving
dependencies@footnote{See Figure 1 of Alliez et al. 2020 at
@url{https://arxiv.org/pdf/1905.11123.pdf}}.
+First, let's review the @option{--keyvalue} option which should be called
separately from the rest of the options described in this section.
+Also, unlike the rest of the options in this section, with
@option{--keyvalue}, you can give more than one input file.
-To demonstrate this, we'll create a surface brightness image of a galaxy in
the F160W filter of the ABYSS
survey@footnote{@url{http://research.iac.es/proyecto/abyss}}.
-In the code-block below, let's make a ``build'' directory to keep intermediate
files and avoid populating the source.
-Afterwards, we'll download the full image and crop out a 20 arcmin wide image
around the galaxy with the commands below.
-You can run these commands in an empty directory.
+@table @option
+@item -l STR[,STR[,...]
+@itemx --keyvalue=STR[,STR[,...]
+Only print the value of the requested keyword(s): the @code{STR}s.
+@option{--keyvalue} can be called multiple times, and each call can contain
multiple comma-separated keywords.
+If more than one file is given, this option uses the same HDU/extension for
all of them (value to @option{--hdu}).
+For example, you can get the number of dimensions of the three FITS files in
the running directory, as well as the length along each dimension, with this
command:
@example
-$ mkdir build
-$ wget http://cdsarc.u-strasbg.fr/ftp/J/A+A/621/A133/fits/ah_f160w.fits
-$ astcrop ah_f160w.fits --center=53.1616278,-27.7802446 --mode=wcs \
- --width=20/3600 --output=build/crop.fits
+$ astfits *.fits --keyvalue=NAXIS,NAXIS1 --keyvalue=NAXIS2
+image-a.fits 2 774 672
+image-b.fits 2 774 672
+image-c.fits 2 387 336
@end example
-To better show the low surface brightness (LSB) outskirts, we'll warp the
image, then convert the pixel units to surface brightness with the commands
below.
-It is very important that the warping is done @emph{before} the conversion to
surface brightness (in units of mag/arcsec@mymath{^2}), because the definition
of surface brightness is non-linear.
-For more, see the Surface brightness topic of @ref{Brightness flux magnitude}.
+If only one input is given, and the @option{--quiet} option is activated, the
file name is not printed on the first column, only the values of the requested
keywords.
@example
-$ zeropoint=25.94
-$ astwarp build/crop.fits --centeroncorner --scale=1/3 \
- --output=build/scaled.fits
-$ pixarea=$(astfits build/scaled.fits --pixelscale --quiet \
- | awk '@{print $1*3600*$2*3600@}')
-$ astarithmetic build/scaled.fits abs $zeropoint counts-to-mag \
- $pixarea log10 2.5 x + --output=build/sb.fits
+$ astfits image-a.fits --keyvalue=NAXIS,NAXIS1 \
+ --keyvalue=NAXIS2 --quiet
+2 774 672
@end example
-We are now ready to convert the surface brightness image into a PDF.
-To better show the LSB features, we'll also limit the color range with the
@code{--fluxlow} and @option{--fluxhigh} options: all pixels with a surface
brightness brighter than 22 mag/arcsec@mymath{^2} will be shown as black, and
all pixels with a surface brightness fainter than 30 mag/arcsec@mymath{^2} will
be white.
-These thresholds are being defined as variables, because we will also need
them below (to pass into PGFPlots).
-We will also set @option{--borderwidth=0}, because the coordinate system we
will add over the image will effectively be a border for the image (separating
it from the background).
+The output is internally stored (and finally printed) as a table (with one
column per keyword).
+Therefore just like the Table program, you can use @option{--colinfoinstdout}
to print the metadata like the example below (also see @ref{Invoking asttable}).
+The keyword metadata (comments and units) are extracted from the comments and
units of the keyword in the input files (first file that has a comment or unit).
+Hence if the keyword doesn't have units or comments in any of the input files,
they will be empty.
+For more on Gnuastro's plain-text metadata format, see @ref{Gnuastro text
table format}.
@example
-$ sblow=22
-$ sbhigh=30
-$ astconvertt build/sb.fits --colormap=gray --borderwidth=0 \
- --fluxhigh=$sbhigh --fluxlow=$sblow --output=build/sb.pdf
+$ astfits *.fits --keyvalue=NAXIS,NAXIS1,NAXIS2 \
+ --colinfoinstdout
+# Column 1: FILENAME [name,str10,] Name of input file.
+# Column 2: NAXIS [ ,u8 ,] number of data axes
+# Column 3: NAXIS1 [ ,u16 ,] length of data axis 1
+# Column 4: NAXIS2 [ ,u16 ,] length of data axis 2
+image-a.fits 2 774 672
+image-b.fits 2 774 672
+image-c.fits 2 387 336
@end example
-Please open @file{sb.pdf} and have a look.
-Also, please open @file{sb.fits} in DS9 (or any other FITS viewer) and play
with the color range.
-Can the surface brightness limits be changed to better show the LSB structure?
-If so, you are free to change the limits above.
-
-We now have the printable PDF representation of the image, but as discussed
above, its not enough for a paper.
-We'll add 1) a thick line showing the size of 20 kpc (kilo parsecs) at the
redshift of the central galaxy, 2) coordinates and 3) a color bar, showing the
surface brightness level of each grayscale level.
-
-To get the first job done, we first need to know the redshift of the central
galaxy.
-To do this, we can use Gnuastro's Query program to look into all the objects
in NED within this image (only asking for the RA, Dec and redshift columns).
-We will then use the Match program to find the NED entry that corresponds to
our galaxy.
+Another advantage of a table output is that you can directly write the the
table to a file.
+For example if you add @option{--output=fileinfo.fits}, the information above
will be printed into a FITS table.
+You can also pipe it into @ref{Table} to select files based on certain
properties, to sort them based on another property, or any other operation that
can be done with Table (including @ref{Column arithmetic}).
+For example with the command below, you can select all the files that have a
size larger than 500 pixels in both dimensions.
@example
-$ astquery ned --dataset=objdir --overlapwith=build/sb.fits \
- --column=ra,dec,z --output=ned.fits
-$ astmatch ned.fits -h1 --coord=53.1616278,-27.7802446 \
- --ccol1=RA,Dec --aperture=1/3600
-$ redshift=$(asttable ned_matched.fits -cz)
-$ echo $redshift
+$ astfits *.fits --keyvalue=NAXIS,NAXIS1,NAXIS2 \
+ --colinfoinstdout \
+ | asttable --range=NAXIS1,500,inf \
+ --range=NAXIS2,500,inf -cFILENAME
+image-a.fits
+image-b.fits
@end example
-Now that we know the redshift of the central object, we can define the
coordinates of the thick line that will show the length of 20 kpc at that
redshift.
-It will be a horizontal line (fixed Declination) across a range of RA.
-The start of this thick line will be located at the top edge of the image (at
the 95-percent of the width and height of the image).
-With the commands below we'll find the three necessary parameters (one
declination and two RAs).
-Just note that in astronomical images, RA increases to the left/east, which is
the reason we are using the minimum and @code{+} to find the RA starting point.
-
-@example
-$ scalelineinkpc=20
-$ coverage=$(astfits build/sb.fits --skycoverage --quiet | awk 'NR==2')
-$ scalelinedec=$(echo $coverage | awk '@{print $4-($4-$3)*0.05@}')
-$ scalelinerastart=$(echo $coverage | awk '@{print $1+($2-$1)*0.05@}')
-$ scalelineraend=$(astcosmiccal --redshift=$redshift --arcsectandist \
- | awk '@{start='$scalelinerastart'; \
- width='$scalelineinkpc'/$1/3600; \
- print start+width@}')
-@end example
+Note that @option{--colinfoinstdout} is necessary to use column names when
piping to other programs (like @command{asttable} above).
+Also, with the @option{-cFILENAME} option, we are asking Table to only print
the final file names (we don't need the sizes any more).
-To draw coordinates over the image, we need to feed these values into PGFPlots.
-But manually entering numbers into the PGFPlots source will be very
frustrating and prone to many errors!
-Fortunately there is an easy way to do this: @LaTeX{} macros.
-New macros are defined by this @LaTeX{} command:
-@example
-\newcommand@{\macroname@}@{value@}
-@end example
-@noindent
-Anywhere that @LaTeX{} confronts @code{\macroname}, it will replace
@code{value} when building the output.
-We will have one file called @file{macros.tex} in the build directory and
define macros based on those values.
-We will use the shell's @code{printf} command to write these macro definition
lines into the macro file.
-We just have to use double backslashes in the @code{printf} command, because
backslash is a meaningful character for @code{printf}, but we want to keep one
of them.
-Also, we put a @code{\n} at the end of each line, otherwise, all the commands
will go into a single line of the macro file.
-We will also place the random `@code{ma}' string at the start of all our
@LaTeX{} macros to help identify the macros for this plot.
+The commands with multiple files above used @file{*.fits}, which is only
useful when all your FITS files are in the same directory.
+However, in many cases, your FITS files will be scattered in multiple
sub-directories of a certain top-level directory, or you may only want those
with more particular file name patterns.
+A more powerful way to list the input files to @option{--keyvalue} is to use
the @command{find} program in Unix-like operating systems.
+For example, with the command below you can search all the FITS files in all
the sub-directories of @file{/TOP/DIR}.
@example
-$ macros=build/macros.tex
-$ printf '\\newcommand@{\\maScaleDec@}'"@{$scalelinedec@}\n" > $macros
-$ printf '\\newcommand@{\\maScaleRAa@}'"@{$scalelinerastart@}\n" >> $macros
-$ printf '\\newcommand@{\\maScaleRAb@}'"@{$scalelineraend@}\n" >> $macros
-$ printf '\\newcommand@{\\maScaleKpc@}'"@{$scalelineinkpc@}\n" >> $macros
-$ printf '\\newcommand@{\\maCenterZ@}'"@{$redshift@}\n" >> $macros
+astfits $(find /TOP/DIR/ -name "*.fits") --keyvalue=NAXIS2
@end example
-Please open the macros file after these commands and have a look to see if
they do conform to the expected format above.
-Another set of macros we will need to feed into PGFPlots is the coordinates of
the image corners.
-Fortunately the @code{coverage} variable found above is also useful here.
-We just need to extract each item before feeding it into the macros.
-To do this, we'll use AWK and keep each value with the temporary shell
variable `@code{v}'.
-
-@example
-$ v=$(echo $coverage | awk '@{print $1@}')
-$ printf '\\newcommand@{\\maCropRAMin@}'"@{$v@}\n" >> $macros
-$ v=$(echo $coverage | awk '@{print $2@}')
-$ printf '\\newcommand@{\\maCropRAMax@}'"@{$v@}\n" >> $macros
-$ v=$(echo $coverage | awk '@{print $3@}')
-$ printf '\\newcommand@{\\maCropDecMin@}'"@{$v@}\n" >> $macros
-$ v=$(echo $coverage | awk '@{print $4@}')
-$ printf '\\newcommand@{\\maCropDecMax@}'"@{$v@}\n" >> $macros
-@end example
+@item -O
+@itemx --colinfoinstdout
+Print column information (or metadata) above the column values when writing
keyword values to standard output with @option{--keyvalue}.
+You can read this option as column-information-in-standard-output.
+@end table
-Finally, we also need to pass some other numbers to PGFPlots: 1) the major
tick distance (in the coordinate axes that will be printed on the edge of the
image).
-We'll assume 7 ticks for this image.
-2) The minimum and maximum surface brightness values that we gave to
ConvertType when making the PDF; PGFPlots will define its color-bar based on
these two values.
+Below we will discuss the options that can be used to manipulate keywords.
+To see the full list of keywords in a FITS HDU, you can use the
@option{--printallkeys} option.
+If any of the keyword modification options below are requested (for example
@option{--update}), the headers of the input file/HDU will be changed first,
then printed.
+Keyword modification is done within the input file.
+Therefore, if you want to keep the original FITS file or HDU intact, it is
easiest to create a copy of the file/HDU first and then run Fits on that (for
copying a HDU to another file, see @ref{HDU information and manipulation}.
+In the FITS standard, keywords are always uppercase.
+So case does not matter in the input or output keyword names you specify.
-@example
-$ v=$(echo $coverage | awk '@{print ($2-$1)/7@}')
-$ printf '\\newcommand@{\\maTickDist@}'"@{$v@}\n" >> $macros
-$ printf '\\newcommand@{\\maSBlow@}'"@{$sblow@}\n" >> $macros
-$ printf '\\newcommand@{\\maSBhigh@}'"@{$sbhigh@}\n" >> $macros
-@end example
+@cartouche
+@noindent
+@strong{@code{CHECKSUM} automatically updated, when present:} the keyword
modification options will change the contents of the HDU.
+Therefore, if a @code{CHECKSUM} is present in the HDU, after all the keyword
modification options have been complete, Fits will also update @code{CHECKSUM}
before closing the file.
+@end cartouche
-All the necessary numbers are now ready.
-Please copy the contents below into a file called @file{my-figure.tex}.
-This is the PGFPlots source for this particular plot.
-Besides the coordinates and scale-line, we will also add some text over the
image and an orange arrow pointing to the central object with its redshift
printed over it.
-The parameters are generally human-readable, so you should be able to get a
good feeling of every line.
-There are also comments which will show up as a different color when you copy
this into a plain-text editor.
+Most of the options can accept multiple instances in one command.
+For example you can add multiple keywords to delete by calling
@option{--delete} multiple times, since repeated keywords are allowed, you can
even delete the same keyword multiple times.
+The action of such options will start from the top most keyword.
-@verbatim
-\begin{tikzpicture}
+The precedence of operations are described below.
+Note that while the order within each class of actions is preserved, the order
of individual actions is not.
+So irrespective of what order you called @option{--delete} and
@option{--update}.
+First, all the delete operations are going to take effect then the update
operations.
+@enumerate
+@item
+@option{--delete}
+@item
+@option{--rename}
+@item
+@option{--update}
+@item
+@option{--write}
+@item
+@option{--asis}
+@item
+@option{--history}
+@item
+@option{--comment}
+@item
+@option{--date}
+@item
+@option{--printallkeys}
+@item
+@option{--verify}
+@item
+@option{--copykeys}
+@end enumerate
+@noindent
+All possible syntax errors will be reported before the keywords are actually
written.
+FITS errors during any of these actions will be reported, but Fits won't stop
until all the operations are complete.
+If @option{--quitonerror} is called, then Fits will immediately stop upon the
first error.
+
+@cindex GNU Grep
+If you want to inspect only a certain set of header keywords, it is easiest to
pipe the output of the Fits program to GNU Grep.
+Grep is a very powerful and advanced tool to search strings which is precisely
made for such situations.
+For example if you only want to check the size of an image FITS HDU, you can
run:
+
+@example
+$ astfits input.fits | grep NAXIS
+@end example
+
+@cartouche
+@noindent
+@strong{FITS STANDARD KEYWORDS:}
+Some header keywords are necessary for later operations on a FITS file, for
example BITPIX or NAXIS, see the FITS standard for their full list.
+If you modify (for example remove or rename) such keywords, the FITS file
extension might not be usable any more.
+Also be careful for the world coordinate system keywords, if you modify or
change their values, any future world coordinate system (like RA and Dec)
measurements on the image will also change.
+@end cartouche
+
+
+@noindent
+The keyword related options to the Fits program are fully described below.
+@table @option
+
+@item -d STR
+@itemx --delete=STR
+Delete one instance of the @option{STR} keyword from the FITS header.
+Multiple instances of @option{--delete} can be given (possibly even for the
same keyword, when its repeated in the meta-data).
+All keywords given will be removed from the headers in the same given order.
+If the keyword doesn't exist, Fits will give a warning and return with a
non-zero value, but will not stop.
+To stop as soon as an error occurs, run with @option{--quitonerror}.
+
+@item -r STR,STR
+@itemx --rename=STR,STR
+Rename a keyword to a new value (for example @option{--rename=OLDNAME,NEWNAME}.
+@option{STR} contains both the existing and new names, which should be
separated by either a comma (@key{,}) or a space character.
+Note that if you use a space character, you have to put the value to this
option within double quotation marks (@key{"}) so the space character is not
interpreted as an option separator.
+Multiple instances of @option{--rename} can be given in one command.
+The keywords will be renamed in the specified order.
+If the keyword doesn't exist, Fits will give a warning and return with a
non-zero value, but will not stop.
+To stop as soon as an error occurs, run with @option{--quitonerror}.
+
+@item -u STR
+@itemx --update=STR
+Update a keyword, its value, its comments and its units in the format
described below.
+If there are multiple instances of the keyword in the header, they will be
changed from top to bottom (with multiple @option{--update} options).
+
+@noindent
+The format of the values to this option can best be specified with an
+example:
+
+@example
+--update=KEYWORD,value,"comments for this keyword",unit
+@end example
+
+If there is a writing error, Fits will give a warning and return with a
non-zero value, but will not stop.
+To stop as soon as an error occurs, run with @option{--quitonerror}.
+
+@noindent
+The value can be any numerical or string value@footnote{Some tricky situations
arise with values like `@command{87095e5}', if this was intended to be a number
it will be kept in the header as @code{8709500000} and there is no problem.
+But this can also be a shortened Git commit hash.
+In the latter case, it should be treated as a string and stored as it is
written.
+Commit hashes are very important in keeping the history of a file during your
research and such values might arise without you noticing them in your
reproduction pipeline.
+One solution is to use @command{git describe} instead of the short hash alone.
+A less recommended solution is to add a space after the commit hash and Fits
will write the value as `@command{87095e5 }' in the header.
+If you later compare the strings on the shell, the space character will be
ignored by the shell in the latter solution and there will be no problem.}.
+Other than the @code{KEYWORD}, all the other values are optional.
+To leave a given token empty, follow the preceding comma (@key{,}) immediately
with the next.
+If any space character is present around the commas, it will be considered
part of the respective token.
+So if more than one token has space characters within it, the safest method to
specify a value to this option is to put double quotation marks around each
individual token that needs it.
+Note that without double quotation marks, space characters will be seen as
option separators and can lead to undefined behavior.
+
+@item -w STR
+@itemx --write=STR
+Write a keyword to the header.
+For the possible value input formats, comments and units for the keyword, see
the @option{--update} option above.
+The special names (first string) below will cause a special behavior:
+
+@table @option
+
+@item /
+Write a ``title'' to the list of keywords.
+A title consists of one blank line and another which is blank for several
spaces and starts with a slash (@key{/}).
+The second string given to this option is the ``title'' or string printed
after the slash.
+For example with the command below you can add a ``title'' of `My keywords'
after the existing keywords and add the subsequent @code{K1} and @code{K2}
keywords under it (note that keyword names are not case sensitive).
+
+@example
+$ astfits test.fits -h1 --write=/,"My keywords" \
+ --write=k1,1.23,"My first keyword" \
+ --write=k2,4.56,"My second keyword"
+$ astfits test.fits -h1
+[[[ ... truncated ... ]]]
+
+ / My keywords
+K1 = 1.23 / My first keyword
+K2 = 4.56 / My second keyword
+END
+@end example
+
+Adding a ``title'' before each contextually separate group of header keywords
greatly helps in readability and visual inspection of the keywords.
+So generally, when you want to add new FITS keywords, its good practice to
also add a title before them.
+
+The reason you need to use @key{/} as the keyword name for setting a title is
that @key{/} is the first non-white character.
+
+The title(s) is(are) written into the FITS with the same order that
@option{--write} is called.
+Therefore in one run of the Fits program, you can specify many different
titles (with their own keywords under them).
+For example the command below that builds on the previous example and adds
another group of keywords named @code{A1} and @code{A2}.
+
+@example
+$ astfits test.fits -h1 --write=/,"My keywords" \
+ --write=k1,1.23,"My first keyword" \
+ --write=k2,4.56,"My second keyword" \
+ --write=/,"My second group of keywords" \
+ --write=a1,7.89,"First keyword" \
+ --write=a2,0.12,"Second keyword"
+@end example
+
+@item checksum
+@cindex CFITSIO
+@cindex @code{DATASUM}: FITS keyword
+@cindex @code{CHECKSUM}: FITS keyword
+When nothing is given afterwards, the header integrity keywords @code{DATASUM}
and @code{CHECKSUM} will be calculated and written/updated.
+The calculation and writing is done fully by CFITSIO, therefore they comply
with the FITS standard
4.0@footnote{@url{https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf}}
that defines these keywords (its Appendix J).
+
+If a value is given (e.g., @option{--write=checksum,MyOwnCheckSum}), then
CFITSIO won't be called to calculate these two keywords and the value (as well
as possible comment and unit) will be written just like any other keyword.
+This is generally not recommended since @code{CHECKSUM} is a reserved FITS
standard keyword.
+If you want to calculate the checksum with another hashing standard manually
and write it into the header, its is recommended to use another keyword name.
+
+In the FITS standard, @code{CHECKSUM} depends on the HDU's data @emph{and}
header keywords, it will therefore not be valid if you make any further changes
to the header after writing the @code{CHECKSUM} keyword.
+This includes any further keyword modification options in the same call to the
Fits program.
+However, @code{DATASUM} only depends on the data section of the HDU/extension,
so it is not changed when you add, remove or update the header keywords.
+Therefore, it is recommended to write these keywords as the last keywords that
are written/modified in the extension.
+You can use the @option{--verify} option (described below) to verify the
values of these two keywords.
+
+@item datasum
+Similar to @option{checksum}, but only write the @code{DATASUM} keyword (that
doesn't depend on the header keywords, only the data).
+@end table
+
+@item -a STR
+@itemx --asis=STR
+Write the given @code{STR} @emph{exactly} as it is, into the given FITS file
header with no modifications.
+If the contents of @code{STR} does not conform to the FITS standard for
keywords, then it may (most probably: it will) corrupt your file and you may
not be able to open it any more.
+So please be @strong{very careful} with this option.
+If you want to define the keyword from scratch, it is best to use the
@option{--write} option (see below) and let CFITSIO worry about complying with
the FITS standard.
+
+The best way to use this option is when you want to add a keyword from one
FITS file to another, unchanged and untouched.
+Below is an example of such a case that can be very useful sometimes (on the
command-line or in scripts):
+
+@example
+$ key=$(astfits firstimage.fits | grep KEYWORD)
+$ astfits --asis="$key" secondimage.fits
+@end example
+
+@cindex GNU Bash
+In particular note the double quotation signs (@key{"}) around the shell
variable (@command{$key}).
+This is because FITS keyword strings usually have lots of space characters, if
this variable is not quoted, the shell will only give the first word in the
full keyword to this option, which will definitely be a non-standard FITS
keyword and will make it hard to work on the file afterwords.
+See the ``Quoting'' section of the GNU Bash manual for more information if
your keyword has the special characters @key{$}, @key{`}, or @key{\}.
+
+You can also use @option{--asis} to copy multiple keywords from one file to
another.
+But the process will be a little more complicated. So we'll show the process
as the simple shell script below.
+You can customize it in the first block of variable definitions:
+1) set the names of the keywords you want to copy: it can be as many keys as
you want, just put a `@code{\|}' between them.
+2) The input FITS file (where keywords should be read from) and its HDU.
+3) The output FITS file (where keywords should be written to) and its HDU.
+4) Set a ``title'' for the newly added keywords in the output (so they are
visually separate from the existing keywords in the output).
+
+@example
+#!/bin/bash
+
+# Customizations (input, output and key names).
+# NOTE: put a '\|' between each keyword name.
+keys="KEYa\|KEYb\|KEYc\|KEYd"
+ifits=original.fits; ihdu=1
+ofits=to_write.fits; ohdu=1
+title="Keys from $ifits (hdu $ihdu)"
+
+# Read keywords from input and write in output.
+oIFS=$IFS
+IFS=''
+c="astfits $ofits -h$ohdu --write=/,\"$title\""
+astfits $ifits -h$ihdu \
+ | grep $keys \
+ | (while read line; \
+ do c="$c --asis=\"$line\""; \
+ done; eval $c); \
+IFS=$oIFS
+@end example
+
+Since its not too long, you can also simply put the variable values of the
first block into the second, and write it directly on the command line (if its
just for one time).
+
+@item -H STR
+@itemx --history STR
+Add a @code{HISTORY} keyword to the header with the given value. A new
@code{HISTORY} keyword will be created for every instance of this option. If
the string given to this option is longer than 70 characters, it will be
separated into multiple keyword cards. If there is an error, Fits will give a
warning and return with a non-zero value, but will not stop. To stop as soon as
an error occurs, run with @option{--quitonerror}.
+
+@item -c STR
+@itemx --comment STR
+Add a @code{COMMENT} keyword to the header with the given value.
+Similar to the explanation for @option{--history} above.
+
+@item -t
+@itemx --date
+Put the current date and time in the header.
+If the @code{DATE} keyword already exists in the header, it will be updated.
+If there is a writing error, Fits will give a warning and return with a
non-zero value, but will not stop.
+To stop as soon as an error occurs, run with @option{--quitonerror}.
+
+@item -p
+@itemx --printallkeys
+Print the full meta data (keywords, values, units and comments) in the
specified FITS extension (HDU).
+If this option is called along with any of the other keyword editing commands,
as described above, all other editing commands take precedence to this.
+Therefore, it will print the final keywords after all the editing has been
done.
+
+@item --printkeynames
+Print only the keyword names of the specified FITS extension (HDU), one line
per name.
+This option must be called alone.
+
+@item -v
+@itemx --verify
+Verify the @code{DATASUM} and @code{CHECKSUM} data integrity keywords of the
FITS standard.
+See the description under the @code{checksum} (under @option{--write}, above)
for more on these keywords.
+
+This option will print @code{Verified} for both keywords if they can be
verified.
+Otherwise, if they don't exist in the given HDU/extension, it will print
@code{NOT-PRESENT}, and if they cannot be verified it will print
@code{INCORRECT}.
+In the latter case (when the keyword values exist but can't be verified), the
Fits program will also return with a failure.
+
+By default this function will also print a short description of the
@code{DATASUM} AND @code{CHECKSUM} keywords.
+You can suppress this extra information with @code{--quiet} option.
+
+@item --copykeys=INT:INT
+Copy the input's keyword records in the given range (inclusive) to the
+output HDU (specified with the @option{--output} and @option{--outhdu}
+options, for the filename and HDU/extension respectively).
+
+The given string to this option must be two integers separated by a colon
(@key{:}).
+The first integer must be positive (counting of the keyword records starts
from 1).
+The second integer may be negative (zero is not acceptable) or an integer
larger than the first.
+
+A negative second integer means counting from the end.
+So @code{-1} is the last copy-able keyword (not including the @code{END}
keyword).
+
+To see the header keywords of the input with a number before them, you can
pipe the output of the FITS program (when it prints all the keywords in an
extension) into the @command{cat} program like below:
+
+@example
+$ astfits input.fits -h1 | cat -n
+@end example
+
+@item --outhdu
+The HDU/extension to write the output keywords of @option{--copykeys}.
+
+@item -Q
+@itemx --quitonerror
+Quit if any of the operations above are not successful.
+By default if an error occurs, Fits will warn the user of the faulty keyword
and continue with the rest of actions.
+
+@item -s STR
+@itemx --datetosec STR
+@cindex Unix epoch time
+@cindex Time, Unix epoch
+@cindex Epoch, Unix time
+Interpret the value of the given keyword in the FITS date format (most
generally: @code{YYYY-MM-DDThh:mm:ss.ddd...}) and return the corresponding Unix
epoch time (number of seconds that have passed since 00:00:00 Thursday, January
1st, 1970).
+The @code{Thh:mm:ss.ddd...} section (specifying the time of day), and also the
@code{.ddd...} (specifying the fraction of a second) are optional.
+The value to this option must be the FITS keyword name that contains the
requested date, for example @option{--datetosec=DATE-OBS}.
+
+@cindex GNU C Library
+This option can also interpret the older FITS date format
(@code{DD/MM/YYThh:mm:ss.ddd...}) where only two characters are given to the
year.
+In this case (following the GNU C Library), this option will make the
following assumption: values 68 to 99 correspond to the years 1969 to 1999, and
values 0 to 68 as the years 2000 to 2068.
+
+This is a very useful option for operations on the FITS date values, for
example sorting FITS files by their dates, or finding the time difference
between two FITS files.
+The advantage of working with the Unix epoch time is that you don't have to
worry about calendar details (for example the number of days in different
months, or leap years, etc).
+
+@item --wcscoordsys=STR
+@cindex Galactic coordinate system
+@cindex Ecliptic coordinate system
+@cindex Equatorial coordinate system
+@cindex Supergalactic coordinate system
+@cindex Coordinate system: Galactic
+@cindex Coordinate system: Ecliptic
+@cindex Coordinate system: Equatorial
+@cindex Coordinate system: Supergalactic
+Convert the coordinate system of the image's world coordinate system (WCS) to
the given coordinate system (@code{STR}) and write it into the file given to
@option{--output} (or an automatically named file if no @option{--output} has
been given).
+
+For example with the command below, @file{img-eq.fits} will have an identical
dataset (pixel values) as @file{image.fits}.
+However, the WCS coordinate system of @file{img-eq.fits} will be the
equatorial coordinate system in the Julian calendar epoch 2000 (which is the
most common epoch used today).
+Fits will automatically extract the current coordinate system of
@file{image.fits} and as long as its one of the recognized coordinate systems
listed below, it will do the conversion.
+
+@example
+$ astfits image.fits --coordsys=eq-j2000 --output=img-eq.fits
+@end example
+
+The currently recognized coordinate systems are listed below (the most common
one today is @code{eq-j2000}):
+
+@table @code
+@item eq-j2000
+2000.0 (Julian-year) equatorial coordinates.
+@item eq-b1950
+1950.0 (Besselian-year) equatorial coordinates.
+@item ec-j2000
+2000.0 (Julian-year) ecliptic coordinates.
+@item ec-b1950
+1950.0 (Besselian-year) ecliptic coordinates.
+@item galactic
+Galactic coordinates.
+@item supergalactic
+Supergalactic coordinates.
+@end table
+
+The Equatorial and Ecliptic coordinate systems are defined by the mean equator
and equinox epoch: either the Besselian year 1950.0, or the Julian year 2000.
+For more on their difference and links for further reading about epochs in
astronomy, please see the description in
@url{https://en.wikipedia.org/wiki/Epoch_(astronomy), Wikipedia}.
+
+@item --wcsdistortion=STR
+@cindex WCS distortion
+@cindex Distortion, WCS
+@cindex SIP WCS distortion
+@cindex TPV WCS distortion
+If the argument has a WCS distortion, the output (file given with the
@option{--output} option) will have the distortion given to this option (for
example @code{SIP}, @code{TPV}).
+The output will be a new file (with a copy of the image, and the new WCS), so
if it already exists, the file will be delete (unless you use the
@code{--dontdelete} option, see @ref{Input output options}).
+
+With this option, the FITS program will read the minimal set of keywords from
the input HDU and the HDU data, it will then write them into the file given to
the @option{--output} option but with a newly created set of WCS-related
keywords corresponding to the desired distortion standard.
+
+If no @option{--output} file is specified, an automatically generated output
name will be used which is composed of the input's name but with the
@file{-DDD.fits} suffix, see @ref{Automatic output}.
+Where @file{DDD} is the value given to this option (desired output distortion).
+
+Note that all possible conversions between all standards are not yet supported.
+If the requested conversion is not supported, an informative error message
will be printed.
+If this happens, please let us know and we'll try our best to add the
respective conversions.
+
+For example with the command below, you can be sure that if @file{in.fits} has
a distortion in its WCS, the distortion of @file{out.fits} will be in the SIP
standard.
+
+@example
+$ astfits in.fits --wcsdistortion=SIP --output=out.fits
+@end example
+@end table
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+@node ConvertType, Table, Fits, Data containers
+@section ConvertType
+
+@cindex Data format conversion
+@cindex Converting data formats
+@cindex Image format conversion
+@cindex Converting image formats
+@pindex @r{ConvertType (}astconvertt@r{)}
+The FITS format used in astronomy was defined mainly for archiving,
transmission, and processing.
+In other situations, the data might be useful in other formats.
+For example, when you are writing a paper or report, or if you are making
slides for a talk, you can't use a FITS image.
+Other image formats should be used.
+In other cases you might want your pixel values in a table format as plain
text for input to other programs that don't recognize FITS.
+ConvertType is created for such situations.
+The various types will increase with future updates and based on need.
+
+The conversion is not only one way (from FITS to other formats), but two ways
(except the EPS and PDF formats@footnote{Because EPS and PDF are vector, not
raster/pixelated formats}).
+So you can also convert a JPEG image or text file into a FITS image.
+Basically, other than EPS/PDF, you can use any of the recognized formats as
different color channel inputs to get any of the recognized outputs.
+So before explaining the options and arguments (in @ref{Invoking
astconvertt}), we'll start with a short description of the recognized files
types in @ref{Recognized file formats}, followed a short introduction to
digital color in @ref{Color}.
+
+@menu
+* Recognized file formats:: Recognized file formats
+* Color:: Some explanations on color.
+* Aligning images with small WCS offsets:: When the WCS slightly differs.
+* Annotations for figure in paper:: Adding coordinates or physical scale.
+* Invoking astconvertt:: Options and arguments to ConvertType.
+@end menu
+
+@node Recognized file formats, Color, ConvertType, ConvertType
+@subsection Recognized file formats
+
+The various standards and the file name extensions recognized by ConvertType
are listed below.
+Currently Gnuastro uses the file name's suffix to identify the format.
+
+@table @asis
+@item FITS or IMH
+@cindex IRAF
+@cindex Astronomical data format
+Astronomical data are commonly stored in the FITS format (or the older data
IRAF @file{.imh} format), a list of file name suffixes which indicate that the
file is in this format is given in @ref{Arguments}.
+
+Each image extension of a FITS file only has one value per pixel/element.
+Therefore, when used as input, each input FITS image contributes as one color
channel.
+If you want multiple extensions in one FITS file for different color channels,
you have to repeat the file name multiple times and use the @option{--hdu},
@option{--hdu2}, @option{--hdu3} or @option{--hdu4} options to specify the
different extensions.
+
+@item JPEG
+@cindex JPEG format
+@cindex Raster graphics
+@cindex Pixelated graphics
+The JPEG standard was created by the Joint photographic experts group.
+It is currently one of the most commonly used image formats.
+Its major advantage is the compression algorithm that is defined by the
standard.
+Like the FITS standard, this is a raster graphics format, which means that it
is pixelated.
+
+A JPEG file can have 1 (for gray-scale), 3 (for RGB) and 4 (for CMYK) color
channels.
+If you only want to convert one JPEG image into other formats, there is no
problem, however, if you want to use it in combination with other input files,
make sure that the final number of color channels does not exceed four.
+If it does, then ConvertType will abort and notify you.
+
+@cindex Suffixes, JPEG images
+The file name endings that are recognized as a JPEG file for input are:
+@file{.jpg}, @file{.JPG}, @file{.jpeg}, @file{.JPEG}, @file{.jpe},
@file{.jif}, @file{.jfif} and @file{.jfi}.
+
+@item TIFF
+@cindex TIFF format
+TIFF (or Tagged Image File Format) was originally designed as a common format
for scanners in the early 90s and since then it has grown to become very
general.
+In many aspects, the TIFF standard is similar to the FITS image standard: it
can allow data of many types (see @ref{Numeric data types}), and also allows
multiple images to be stored in a single file (each image in the file is called
a `directory' in the TIFF standard).
+However, unlike FITS, it can only store images, it has no constructs for
tables.
+Another (inconvenient) difference with the FITS standard is that keyword names
are stored as numbers, not human-readable text.
+
+However, outside of astronomy, because of its support of different numeric
+data types, many fields use TIFF images for accurate (for example 16-bit
+integer or floating point for example) imaging data.
+
+Currently ConvertType can only read TIFF images, if you are interested in
+writing TIFF images, please get in touch with us.
+
+@item EPS
+@cindex EPS
+@cindex PostScript
+@cindex Vector graphics
+@cindex Encapsulated PostScript
+The Encapsulated PostScript (EPS) format is essentially a one page PostScript
file which has a specified size.
+PostScript also includes non-image data, for example lines and texts.
+It is a fully functional programming language to describe a document.
+Therefore in ConvertType, EPS is only an output format and cannot be used as
input.
+Contrary to the FITS or JPEG formats, PostScript is not a raster format, but
is categorized as vector graphics.
+
+@cindex PDF
+@cindex Adobe systems
+@cindex PostScript vs. PDF
+@cindex Compiled PostScript
+@cindex Portable Document format
+@cindex Static document description format
+The Portable Document Format (PDF) is currently the most common format for
documents.
+Some believe that PDF has replaced PostScript and that PostScript is now
obsolete.
+This view is wrong, a PostScript file is an actual plain text file that can be
edited like any program source with any text editor.
+To be able to display its programmed content or print, it needs to pass
through a processor or compiler.
+A PDF file can be thought of as the processed output of the compiler on an
input PostScript file.
+PostScript, EPS and PDF were created and are registered by Adobe Systems.
+
+@cindex @TeX{}
+@cindex @LaTeX{}
+With these features in mind, you can see that when you are compiling a
document with @TeX{} or @LaTeX{}, using an EPS file is much more low level than
a JPEG and thus you have much greater control and therefore quality.
+Since it also includes vector graphic lines we also use such lines to make a
thin border around the image to make its appearance in the document much better.
+No matter the resolution of the display or printer, these lines will always be
clear and not pixelated.
+In the future, addition of text might be included (for example labels or
object IDs) on the EPS output.
+However, this can be done better with tools within @TeX{} or @LaTeX{} such as
PGF/Tikz@footnote{@url{http://sourceforge.net/projects/pgf/}}.
+
+@cindex Binary image
+@cindex Saving binary image
+@cindex Black and white image
+If the final input image (possibly after all operations on the flux explained
below) is a binary image or only has two colors of black and white (in
segmentation maps for example), then PostScript has another great advantage
compared to other formats.
+It allows for 1 bit pixels (pixels with a value of 0 or 1), this can decrease
the output file size by 8 times.
+So if a gray-scale image is binary, ConvertType will exploit this property in
the EPS and PDF (see below) outputs.
+
+@cindex Suffixes, EPS format
+The standard formats for an EPS file are @file{.eps}, @file{.EPS},
@file{.epsf} and @file{.epsi}.
+The EPS outputs of ConvertType have the @file{.eps} suffix.
+
+@item PDF
+@cindex Suffixes, PDF format
+@cindex GPL Ghostscript
+As explained above, a PDF document is a static document description format,
viewing its result is therefore much faster and more efficient than PostScript.
+To create a PDF output, ConvertType will make a PostScript page description
and convert that to PDF using GPL Ghostscript.
+The suffixes recognized for a PDF file are: @file{.pdf}, @file{.PDF}.
+If GPL Ghostscript cannot be run on the PostScript file, it will remain and a
warning will be printed.
+
+@item @option{blank}
+@cindex @file{blank} color channel
+This is not actually a file type! But can be used to fill one color channel
with a blank value.
+If this argument is given for any color channel, that channel will not be used
in the output.
+
+@item Plain text
+@cindex Plain text
+@cindex Suffixes, plain text
+Plain text files have the advantage that they can be viewed with any text
editor or on the command-line.
+Most programs also support input as plain text files.
+As input, each plain text file is considered to contain one color channel.
+
+In ConvertType, the recognized extensions for plain text files are @file{.txt}
and @file{.dat}.
+As described in @ref{Invoking astconvertt}, if you just give these extensions,
(and not a full filename) as output, then automatic output will be preformed to
determine the final output name (see @ref{Automatic output}).
+Besides these, when the format of a file cannot be recognized from its name,
ConvertType will fall back to plain text mode.
+So you can use any name (even without an extension) for a plain text input or
output.
+Just note that when the suffix is not recognized, automatic output will not be
preformed.
+
+The basic input/output on plain text images is very similar to how tables are
read/written as described in @ref{Gnuastro text table format}.
+Simply put, the restrictions are very loose, and there is a convention to
define a name, units, data type (see @ref{Numeric data types}), and comments
for the data in a commented line.
+The only difference is that as a table, a text file can contain many datasets
(columns), but as a 2D image, it can only contain one dataset.
+As a result, only one information comment line is necessary for a 2D image,
and instead of the starting `@code{# Column N}' (@code{N} is the column
number), the information line for a 2D image must start with `@code{# Image 1}'.
+When ConvertType is asked to output to plain text file, this information
comment line is written before the image pixel values.
+
+When converting an image to plain text, consider the fact that if the image is
large, the number of columns in each line will become very large, possibly
making it very hard to open in some text editors.
+
+@item Standard output (command-line)
+This is very similar to the plain text output, but instead of creating a file
to keep the printed values, they are printed on the command line.
+This can be very useful when you want to redirect the results directly to
another program in one command with no intermediate file.
+The only difference is that only the pixel values are printed (with no
information comment line).
+To print to the standard output, set the output name to `@file{stdout}'.
+
+@end table
+
+@node Color, Aligning images with small WCS offsets, Recognized file formats,
ConvertType
+@subsection Color
+
+@cindex RGB
+@cindex CMYK
+@cindex Image
+@cindex Color
+@cindex Pixels
+@cindex Colormap
+@cindex Primary colors
+Color is defined by mixing various measurements/filters.
+In digital monitors or common digital cameras, colors are displayed/stored by
mixing the three basic colors of red, green and blue (RGB) with various
proportions.
+When printing on paper, standard printers use the cyan, magenta, yellow and
key (CMYK, key=black) color space.
+In other words, for each displayed/printed pixel of a color image, the
dataset/image has three or four values.
+
+@cindex Color channel
+@cindex Channel, color
+To store/show the three values for each pixel, cameras and monitors allocate a
certain fraction of each pixel's area to red, green and blue filters.
+These three filters are thus built into the hardware at the pixel level.
+However, because measurement accuracy is very important in scientific
instruments, and we want to do measurements (take images) with various/custom
filters (without having to order a new expensive detector!), scientific
detectors use the full area of the pixel to store one value for it in a
single/mono channel dataset.
+To make measurements in different filters, we just place a filter in the light
path before the detector.
+Therefore, the FITS format that is used to store astronomical datasets is
inherently a mono-channel format (see @ref{Recognized file formats} or
@ref{Fits}).
+
+When a subject has been imaged in multiple filters, you can feed each
different filter into the red, green and blue channels and obtain a colored
visualization.
+In ConvertType, you can do this by giving each separate single-channel dataset
(for example in the FITS image format) as an argument (in the proper order),
then asking for the output in a format that supports multi-channel datasets
(for example JPEG or PDF, see the examples in @ref{Invoking astconvertt}).
+
+@cindex Grayscale
+@cindex Visualization
+@cindex Colormap, HSV
+@cindex Colormap, gray-scale
+@cindex HSV: Hue Saturation Value
+As discussed above, color is not defined when a dataset/image contains a
single value for each pixel.
+However, we interact with scientific datasets through monitors or printers
(which allow multiple values per pixel and produce color with them).
+As a result, there is a lot of freedom in visualizing a single-channel dataset.
+The most basic is to use shades of black (because of its strong contrast with
white).
+This scheme is called grayscale.
+To help in visualization, more complex mappings can be defined.
+For example, the values can be scaled to a range of 0 to 360 and used as the
``Hue'' term of the @url{https://en.wikipedia.org/wiki/HSL_and_HSV,
Hue-Saturation-Value} (HSV) color space (while fixing the ``Saturation'' and
``Value'' terms).
+In ConvertType, you can use the @option{--colormap} option to choose between
different mappings of mono-channel inputs, see @ref{Invoking astconvertt}.
+
+Since grayscale is a commonly used mapping of single-valued datasets, we'll
continue with a closer look at how it is stored.
+One way to represent a gray-scale image in different color spaces is to use
the same proportions of the primary colors in each pixel.
+This is the common way most FITS image viewers work: for each pixel, they fill
all the channels with the single value.
+While this is necessary for displaying a dataset, there are downsides when
storing/saving this type of grayscale visualization (for example in a paper).
+
+@itemize
+
+@item
+Three (for RGB) or four (for CMYK) values have to be stored for every pixel,
this makes the output file very heavy (in terms of bytes).
+
+@item
+If printing, the printing errors of each color channel can make the printed
image slightly more blurred than it actually is.
+
+@end itemize
+
+@cindex PNG standard
+@cindex Single channel CMYK
+To solve both these problems when storing grayscale visualization, the best
way is to save a single-channel dataset into the black channel of the CMYK
color space.
+The JPEG standard is the only common standard that accepts CMYK color space.
+
+The JPEG and EPS standards set two sizes for the number of bits in each
channel: 8-bit and 12-bit.
+The former is by far the most common and is what is used in ConvertType.
+Therefore, each channel should have values between 0 to @math{2^8-1=255}.
+From this we see how each pixel in a gray-scale image is one byte (8 bits)
long, in an RGB image, it is 3 bytes long and in CMYK it is 4 bytes long.
+But thanks to the JPEG compression algorithms, when all the pixels of one
channel have the same value, that channel is compressed to one pixel.
+Therefore a Grayscale image and a CMYK image that has only the K-channel
filled are approximately the same file size.
+
+
+@node Aligning images with small WCS offsets, Annotations for figure in paper,
Color, ConvertType
+@subsection Aligning images with small WCS offsets
+
+In order to have nice color images, it is important that the images be
properly aligned.
+This is usually the case in many scenarios, but it some times happens that the
images have a small WCS offset, even though they have the same size.
+In such cases you can use the script below to align the images into
approximately the same pixel grid (to within about 0.5 pixels which is
sufficient in many color-image usage scenarios).
+
+The script below does the job using Gnuastro's @ref{Warp} and @ref{Crop}
programs.
+Simply copy the lines below into a plain-text file with your favorite text
editor and save it as @file{my-align.sh}.
+Don't forget to set the variables of the first three lines to specify the file
names (without the @file{.fits} suffix) and the HDUs of your inputs.
+These four lines are all you need to edit, leave the rest unchanged.
+Also, if you are copy/pasting the script from a PDF, be careful that the
single-quotes used in AWK may need to be corrected.
+
+@example
+#!/bin/sh
+
+# Set the input names (without the '.fits' suffix),
+# and their HDUs.
+r=RED_IMAGE_NO_SUFFIX; rhdu=1
+g=GREEN_IMAGE_NO_SUFFIX; ghdu=1
+b=BLUE_IMAGE_NO_SUFFIX; bhdu=1
+
+# To stop the script if there is a crash
+set -e
+
+# Align all the images to the celestial poles.
+astwarp $r.fits --align -h$rhdu -o $r-aligned.fits
+astwarp $g.fits --align -h$ghdu -o $g-aligned.fits
+astwarp $b.fits --align -h$bhdu -o $b-aligned.fits
+
+# Calculate the final WCS-based center and image-based width based on
+# the G-band (in RGB) image.
+centerwcs=$(astfits $g-aligned.fits --skycoverage --quiet \
+ | awk 'NR==1@{printf "%g %g", $1,$2@}')
+widthpix=$(astfits $g-aligned.fits -h1 --quiet \
+ --keyvalue=NAXIS1,NAXIS2 \
+ | awk '@{printf "%d,%d", $1, $2@}')
+
+# Crop all the images around the desired center and width.
+for f in $r $g $b; do
+ centerpix=$(echo $centerwcs \
+ | asttable -c'arith $1 $2 wcs-to-img' \
+ --wcsfile=$f-aligned.fits \
+ | awk '@{printf "%g,%g", $1, $2@}')
+ astcrop $f-aligned.fits --mode=img --width=$widthpix \
+ --center=$centerpix -o$f-use.fits
+ rm $f-aligned.fits
+done
+@end example
+
+Once you have have saved the file and come back to your command-line you can
run the script like this:
+
+@example
+$ chmod +x my-align.sh
+$ ./my-align.sh
+@end example
+
+@noindent
+Of course, feel free to hack it and modify it to fit your datasets, like the
rest of Gnuastro, this script is released under GNU GPLv.3 and above, see
@ref{Your rights}.
+
+@node Annotations for figure in paper, Invoking astconvertt, Aligning images
with small WCS offsets, ConvertType
+@subsection Annotations for figure in paper
+
+@cindex Image annotation
+@cindex Annotation of images for paper
+To make a nice figure from your FITS images, it is important to show more than
merely the raw image (converted to a printer friendly format like PDF or JPEG).
+Annotations (or visual metadata) over the raw image greatly help the readers
clearly see your argument and put the image/result in a larger context.
+Examples include:
+@itemize
+@item
+Coordinates (Right Ascension and Declination) on the edges of the image, so
viewers of your paper or presentation slides can get a physical feeling of the
field's sky coverage.
+@item
+Thick line that has a fixed tangential size (for example in kilo parsecs) at
the redshift/distance of interest.
+@item
+Contours over the image to show radio/X-ray emission, over an optical image
for example.
+@item
+Text or arrows or etc, over certain parts of the image.
+@end itemize
+
+@cindex PGFPlots
+Because of the modular philosophy of Gnuastro, ConvertType is only focused on
converting your FITS images to printer friendly formats like JPEG or PDF.
+But to present your results in a slide or paper, you will often need to
annotate the raw JPEG or PDF with some of the features above.
+The good news is that there are many powerful plotting programs that you can
use to add such annotations.
+As a result, there is no point in making a new one, specific to Gnuastro.
+In this section, we'll demonstrate this using the very powerful
PGFPlots@footnote{@url{http://mirrors.ctan.org/graphics/pgf/contrib/pgfplots/doc/pgfplots.pdf}}
package of @LaTeX{}.
+
+@cartouche
+@noindent
+@strong{Single script for easy running:} In this section we are reviewing the
reason and details of every step which is good for educational purposes.
+But when you know the steps already, these separate code blocks can be
annoying.
+Therefore the full script (except for the data download step) is available in
@ref{Full script of annotations on figure}.
+@end cartouche
+
+@cindex TiKZ
+@cindex Matplotlib
+PGFPlots uses the same @LaTeX{} graphic engine that typesets your paper/slide.
+Therefore when you build your plots and figures using PGFPlots (and its
underlying package
PGF/TiKZ@footnote{@url{http://mirrors.ctan.org/graphics/pgf/base/doc/pgfmanual.pdf}})
your plots will blend beautifully within your text: same fonts, same colors,
same line properties and etc.
+Since most papers (and presentation slides@footnote{To build slides, @LaTeX{}
has packages like Beamer, see
@url{http://mirrors.ctan.org/macros/latex/contrib/beamer/doc/beameruserguide.pdf}})
are made with @LaTeX{}, PGFPlots is therefore the best tool for those who use
@LaTeX{} to create documents.
+PGFPlots also doesn't need any extra dependencies beyond a basic/minimal
@TeX{}-live installation, so it is much more reliable than tools like
Matplotlib in Python that have hundreds of fast-evolving
dependencies@footnote{See Figure 1 of Alliez et al. 2020 at
@url{https://arxiv.org/pdf/1905.11123.pdf}}.
+
+To demonstrate this, we'll create a surface brightness image of a galaxy in
the F160W filter of the ABYSS
survey@footnote{@url{http://research.iac.es/proyecto/abyss}}.
+In the code-block below, let's make a ``build'' directory to keep intermediate
files and avoid populating the source.
+Afterwards, we'll download the full image and crop out a 20 arcmin wide image
around the galaxy with the commands below.
+You can run these commands in an empty directory.
+
+@example
+$ mkdir build
+$ wget http://cdsarc.u-strasbg.fr/ftp/J/A+A/621/A133/fits/ah_f160w.fits
+$ astcrop ah_f160w.fits --center=53.1616278,-27.7802446 --mode=wcs \
+ --width=20/3600 --output=build/crop.fits
+@end example
+
+To better show the low surface brightness (LSB) outskirts, we'll warp the
image, then convert the pixel units to surface brightness with the commands
below.
+It is very important that the warping is done @emph{before} the conversion to
surface brightness (in units of mag/arcsec@mymath{^2}), because the definition
of surface brightness is non-linear.
+For more, see the Surface brightness topic of @ref{Brightness flux magnitude}.
+
+@example
+$ zeropoint=25.94
+$ astwarp build/crop.fits --centeroncorner --scale=1/3 \
+ --output=build/scaled.fits
+$ pixarea=$(astfits build/scaled.fits --pixelscale --quiet \
+ | awk '@{print $1*3600*$2*3600@}')
+$ astarithmetic build/scaled.fits abs $zeropoint counts-to-mag \
+ $pixarea log10 2.5 x + --output=build/sb.fits
+@end example
+
+We are now ready to convert the surface brightness image into a PDF.
+To better show the LSB features, we'll also limit the color range with the
@code{--fluxlow} and @option{--fluxhigh} options: all pixels with a surface
brightness brighter than 22 mag/arcsec@mymath{^2} will be shown as black, and
all pixels with a surface brightness fainter than 30 mag/arcsec@mymath{^2} will
be white.
+These thresholds are being defined as variables, because we will also need
them below (to pass into PGFPlots).
+We will also set @option{--borderwidth=0}, because the coordinate system we
will add over the image will effectively be a border for the image (separating
it from the background).
+
+@example
+$ sblow=22
+$ sbhigh=30
+$ astconvertt build/sb.fits --colormap=gray --borderwidth=0 \
+ --fluxhigh=$sbhigh --fluxlow=$sblow --output=build/sb.pdf
+@end example
+
+Please open @file{sb.pdf} and have a look.
+Also, please open @file{sb.fits} in DS9 (or any other FITS viewer) and play
with the color range.
+Can the surface brightness limits be changed to better show the LSB structure?
+If so, you are free to change the limits above.
+
+We now have the printable PDF representation of the image, but as discussed
above, its not enough for a paper.
+We'll add 1) a thick line showing the size of 20 kpc (kilo parsecs) at the
redshift of the central galaxy, 2) coordinates and 3) a color bar, showing the
surface brightness level of each grayscale level.
+
+To get the first job done, we first need to know the redshift of the central
galaxy.
+To do this, we can use Gnuastro's Query program to look into all the objects
in NED within this image (only asking for the RA, Dec and redshift columns).
+We will then use the Match program to find the NED entry that corresponds to
our galaxy.
+
+@example
+$ astquery ned --dataset=objdir --overlapwith=build/sb.fits \
+ --column=ra,dec,z --output=ned.fits
+$ astmatch ned.fits -h1 --coord=53.1616278,-27.7802446 \
+ --ccol1=RA,Dec --aperture=1/3600
+$ redshift=$(asttable ned_matched.fits -cz)
+$ echo $redshift
+@end example
+
+Now that we know the redshift of the central object, we can define the
coordinates of the thick line that will show the length of 20 kpc at that
redshift.
+It will be a horizontal line (fixed Declination) across a range of RA.
+The start of this thick line will be located at the top edge of the image (at
the 95-percent of the width and height of the image).
+With the commands below we'll find the three necessary parameters (one
declination and two RAs).
+Just note that in astronomical images, RA increases to the left/east, which is
the reason we are using the minimum and @code{+} to find the RA starting point.
+
+@example
+$ scalelineinkpc=20
+$ coverage=$(astfits build/sb.fits --skycoverage --quiet | awk 'NR==2')
+$ scalelinedec=$(echo $coverage | awk '@{print $4-($4-$3)*0.05@}')
+$ scalelinerastart=$(echo $coverage | awk '@{print $1+($2-$1)*0.05@}')
+$ scalelineraend=$(astcosmiccal --redshift=$redshift --arcsectandist \
+ | awk '@{start='$scalelinerastart'; \
+ width='$scalelineinkpc'/$1/3600; \
+ print start+width@}')
+@end example
+
+To draw coordinates over the image, we need to feed these values into PGFPlots.
+But manually entering numbers into the PGFPlots source will be very
frustrating and prone to many errors!
+Fortunately there is an easy way to do this: @LaTeX{} macros.
+New macros are defined by this @LaTeX{} command:
+@example
+\newcommand@{\macroname@}@{value@}
+@end example
+@noindent
+Anywhere that @LaTeX{} confronts @code{\macroname}, it will replace
@code{value} when building the output.
+We will have one file called @file{macros.tex} in the build directory and
define macros based on those values.
+We will use the shell's @code{printf} command to write these macro definition
lines into the macro file.
+We just have to use double backslashes in the @code{printf} command, because
backslash is a meaningful character for @code{printf}, but we want to keep one
of them.
+Also, we put a @code{\n} at the end of each line, otherwise, all the commands
will go into a single line of the macro file.
+We will also place the random `@code{ma}' string at the start of all our
@LaTeX{} macros to help identify the macros for this plot.
+
+@example
+$ macros=build/macros.tex
+$ printf '\\newcommand@{\\maScaleDec@}'"@{$scalelinedec@}\n" > $macros
+$ printf '\\newcommand@{\\maScaleRAa@}'"@{$scalelinerastart@}\n" >> $macros
+$ printf '\\newcommand@{\\maScaleRAb@}'"@{$scalelineraend@}\n" >> $macros
+$ printf '\\newcommand@{\\maScaleKpc@}'"@{$scalelineinkpc@}\n" >> $macros
+$ printf '\\newcommand@{\\maCenterZ@}'"@{$redshift@}\n" >> $macros
+@end example
+
+Please open the macros file after these commands and have a look to see if
they do conform to the expected format above.
+Another set of macros we will need to feed into PGFPlots is the coordinates of
the image corners.
+Fortunately the @code{coverage} variable found above is also useful here.
+We just need to extract each item before feeding it into the macros.
+To do this, we'll use AWK and keep each value with the temporary shell
variable `@code{v}'.
+
+@example
+$ v=$(echo $coverage | awk '@{print $1@}')
+$ printf '\\newcommand@{\\maCropRAMin@}'"@{$v@}\n" >> $macros
+$ v=$(echo $coverage | awk '@{print $2@}')
+$ printf '\\newcommand@{\\maCropRAMax@}'"@{$v@}\n" >> $macros
+$ v=$(echo $coverage | awk '@{print $3@}')
+$ printf '\\newcommand@{\\maCropDecMin@}'"@{$v@}\n" >> $macros
+$ v=$(echo $coverage | awk '@{print $4@}')
+$ printf '\\newcommand@{\\maCropDecMax@}'"@{$v@}\n" >> $macros
+@end example
+
+Finally, we also need to pass some other numbers to PGFPlots: 1) the major
tick distance (in the coordinate axes that will be printed on the edge of the
image).
+We'll assume 7 ticks for this image.
+2) The minimum and maximum surface brightness values that we gave to
ConvertType when making the PDF; PGFPlots will define its color-bar based on
these two values.
+
+@example
+$ v=$(echo $coverage | awk '@{print ($2-$1)/7@}')
+$ printf '\\newcommand@{\\maTickDist@}'"@{$v@}\n" >> $macros
+$ printf '\\newcommand@{\\maSBlow@}'"@{$sblow@}\n" >> $macros
+$ printf '\\newcommand@{\\maSBhigh@}'"@{$sbhigh@}\n" >> $macros
+@end example
+
+All the necessary numbers are now ready.
+Please copy the contents below into a file called @file{my-figure.tex}.
+This is the PGFPlots source for this particular plot.
+Besides the coordinates and scale-line, we will also add some text over the
image and an orange arrow pointing to the central object with its redshift
printed over it.
+The parameters are generally human-readable, so you should be able to get a
good feeling of every line.
+There are also comments which will show up as a different color when you copy
this into a plain-text editor.
+
+@verbatim
+\begin{tikzpicture}
%% Define the coordinates and colorbar
\begin{axis}[
@@ -22047,1918 +22517,1451 @@ With this option, the output image noise is
always going to be identical (or rep
Save the output in the double precision floating point format that was used
internally.
This option will be most useful if the input images were of integer types.
-@end table
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-@node High-level calculations, Installed scripts, Modeling and fittings, Top
-@chapter High-level calculations
-
-After the reduction of raw data (for example with the programs in @ref{Data
manipulation}) you will have reduced images/data ready for processing/analyzing
(for example with the programs in @ref{Data analysis}).
-But the processed/analyzed data (or catalogs) are still not enough to derive
any scientific result.
-Even higher-level analysis is still needed to convert the observed magnitudes,
sizes or volumes into physical quantities that we associate with each catalog
entry or detected object which is the purpose of the tools in this section.
-
-
-
-
-
-@menu
-* CosmicCalculator:: Calculate cosmological variables
-@end menu
-
-@node CosmicCalculator, , High-level calculations, High-level calculations
-@section CosmicCalculator
-
-To derive higher-level information regarding our sources in extra-galactic
astronomy, cosmological calculations are necessary.
-In Gnuastro, CosmicCalculator is in charge of such calculations.
-Before discussing how CosmicCalculator is called and operates (in
@ref{Invoking astcosmiccal}), it is important to provide a rough but mostly
self sufficient review of the basics and the equations used in the analysis.
-In @ref{Distance on a 2D curved space} the basic idea of understanding
distances in a curved and expanding 2D universe (which we can visualize) are
reviewed.
-Having solidified the concepts there, in @ref{Extending distance concepts to
3D}, the formalism is extended to the 3D universe we are trying to study in our
research.
-
-The focus here is obtaining a physical insight into these equations (mainly
for the use in real observational studies).
-There are many books thoroughly deriving and proving all the equations with
all possible initial conditions and assumptions for any abstract universe,
interested readers can study those books.
-
-@menu
-* Distance on a 2D curved space:: Distances in 2D for simplicity
-* Extending distance concepts to 3D:: Going to 3D (our real universe).
-* Invoking astcosmiccal:: How to run CosmicCalculator
-@end menu
-
-@node Distance on a 2D curved space, Extending distance concepts to 3D,
CosmicCalculator, CosmicCalculator
-@subsection Distance on a 2D curved space
-
-The observations to date (for example the Planck 2015 results), have not
measured@footnote{The observations are interpreted under the assumption of
uniform curvature.
-For a relativistic alternative to dark energy (and maybe also some part of
dark matter), non-uniform curvature may be even be more critical, but that is
beyond the scope of this brief explanation.} the presence of significant
curvature in the universe.
-However to be generic (and allow its measurement if it does in fact exist), it
is very important to create a framework that allows non-zero uniform curvature.
-However, this section is not intended to be a fully thorough and
mathematically complete derivation of these concepts.
-There are many references available for such reviews that go deep into the
abstract mathematical proofs.
-The emphasis here is on visualization of the concepts for a beginner.
-
-As 3D beings, it is difficult for us to mentally create (visualize) a picture
of the curvature of a 3D volume.
-Hence, here we will assume a 2D surface/space and discuss distances on that 2D
surface when it is flat and when it is curved.
-Once the concepts have been created/visualized here, we will extend them, in
@ref{Extending distance concepts to 3D}, to a real 3D spatial @emph{slice} of
the Universe we live in and hope to study.
-
-To be more understandable (actively discuss from an observer's point of view)
let's assume there's an imaginary 2D creature living on the 2D space (which
@emph{might} be curved in 3D).
-Here, we will be working with this creature in its efforts to analyze
distances in its 2D universe.
-The start of the analysis might seem too mundane, but since it is difficult to
imagine a 3D curved space, it is important to review all the very basic
concepts thoroughly for an easy transition to a universe that is more difficult
to visualize (a curved 3D space embedded in 4D).
-
-To start, let's assume a static (not expanding or shrinking), flat 2D surface
similar to @ref{flatplane} and that the 2D creature is observing its universe
from point @mymath{A}.
-One of the most basic ways to parameterize this space is through the Cartesian
coordinates (@mymath{x}, @mymath{y}).
-In @ref{flatplane}, the basic axes of these two coordinates are plotted.
-An infinitesimal change in the direction of each axis is written as
@mymath{dx} and @mymath{dy}.
-For each point, the infinitesimal changes are parallel with the respective
axes and are not shown for clarity.
-Another very useful way of parameterizing this space is through polar
coordinates.
-For each point, we define a radius (@mymath{r}) and angle (@mymath{\phi}) from
a fixed (but arbitrary) reference axis.
-In @ref{flatplane} the infinitesimal changes for each polar coordinate are
plotted for a random point and a dashed circle is shown for all points with the
same radius.
-
-@float Figure,flatplane
-@center@image{gnuastro-figures/flatplane, 10cm, , }
-
-@caption{Two dimensional Cartesian and polar coordinates on a flat
-plane.}
-@end float
-
-Assuming an object is placed at a certain position, which can be parameterized
as @mymath{(x,y)}, or @mymath{(r,\phi)}, a general infinitesimal change in its
position will place it in the coordinates @mymath{(x+dx,y+dy)}, or
@mymath{(r+dr,\phi+d\phi)}.
-The distance (on the flat 2D surface) that is covered by this infinitesimal
change in the static universe (@mymath{ds_s}, the subscript signifies the
static nature of this universe) can be written as:
-
-@dispmath{ds_s^2=dx^2+dy^2=dr^2+r^2d\phi^2}
-
-The main question is this: how can the 2D creature incorporate the (possible)
curvature in its universe when it's calculating distances? The universe that it
lives in might equally be a curved surface like @ref{sphereandplane}.
-The answer to this question but for a 3D being (us) is the whole purpose to
this discussion.
-Here, we want to give the 2D creature (and later, ourselves) the tools to
measure distances if the space (that hosts the objects) is curved.
-
-@ref{sphereandplane} assumes a spherical shell with radius @mymath{R} as the
curved 2D plane for simplicity.
-The 2D plane is tangent to the spherical shell and only touches it at
@mymath{A}.
-This idea will be generalized later.
-The first step in measuring the distance in a curved space is to imagine a
third dimension along the @mymath{z} axis as shown in @ref{sphereandplane}.
-For simplicity, the @mymath{z} axis is assumed to pass through the center of
the spherical shell.
-Our imaginary 2D creature cannot visualize the third dimension or a curved 2D
surface within it, so the remainder of this discussion is purely abstract for
it (similar to us having difficulty in visualizing a 3D curved space in 4D).
-But since we are 3D creatures, we have the advantage of visualizing the
following steps.
-Fortunately the 2D creature is already familiar with our mathematical
constructs, so it can follow our reasoning.
-
-With the third axis added, a generic infinitesimal change over @emph{the full}
3D space corresponds to the distance:
-
-@dispmath{ds_s^2=dx^2+dy^2+dz^2=dr^2+r^2d\phi^2+dz^2.}
-
-@float Figure,sphereandplane
-@center@image{gnuastro-figures/sphereandplane, 10cm, , }
-
-@caption{2D spherical shell (centered on @mymath{O}) and flat plane (light
gray) tangent to it at point @mymath{A}.}
-@end float
-
-It is very important to recognize that this change of distance is for
@emph{any} point in the 3D space, not just those changes that occur on the 2D
spherical shell of @ref{sphereandplane}.
-Recall that our 2D friend can only do measurements on the 2D surfaces, not the
full 3D space.
-So we have to constrain this general change to any change on the 2D spherical
shell.
-To do that, let's look at the arbitrary point @mymath{P} on the 2D spherical
shell.
-Its image (@mymath{P'}) on the flat plain is also displayed. From the dark
gray triangle, we see that
-
-@dispmath{\sin\theta={r\over R},\quad\cos\theta={R-z\over R}.}These relations
allow the 2D creature to find the value of @mymath{z} (an abstract dimension
for it) as a function of r (distance on a flat 2D plane, which it can
visualize) and thus eliminate @mymath{z}.
-From @mymath{\sin^2\theta+\cos^2\theta=1}, we get @mymath{z^2-2Rz+r^2=0} and
solving for @mymath{z}, we find:
-
-@dispmath{z=R\left(1\pm\sqrt{1-{r^2\over R^2}}\right).}
-
-The @mymath{\pm} can be understood from @ref{sphereandplane}: For each
@mymath{r}, there are two points on the sphere, one in the upper hemisphere and
one in the lower hemisphere.
-An infinitesimal change in @mymath{r}, will create the following infinitesimal
change in @mymath{z}:
-
-@dispmath{dz={\mp r\over R}\left(1\over
-\sqrt{1-{r^2/R^2}}\right)dr.}Using the positive signed equation instead of
@mymath{dz} in the @mymath{ds_s^2} equation above, we get:
-
-@dispmath{ds_s^2={dr^2\over 1-r^2/R^2}+r^2d\phi^2.}
-
-The derivation above was done for a spherical shell of radius @mymath{R} as a
curved 2D surface.
-To generalize it to any surface, we can define @mymath{K=1/R^2} as the
curvature parameter.
-Then the general infinitesimal change in a static universe can be written as:
-
-@dispmath{ds_s^2={dr^2\over 1-Kr^2}+r^2d\phi^2.}
-
-Therefore, when @mymath{K>0} (and curvature is the same everywhere), we have a
finite universe, where @mymath{r} cannot become larger than @mymath{R} as in
@ref{sphereandplane}.
-When @mymath{K=0}, we have a flat plane (@ref{flatplane}) and a negative
@mymath{K} will correspond to an imaginary @mymath{R}.
-The latter two cases may be infinite in area (which is not a simple concept,
but mathematically can be modeled with @mymath{r} extending infinitely), or
finite-area (like a cylinder is flat everywhere with @mymath{ds_s^2={dx^2 +
dy^2}}, but finite in one direction in size).
-
-@cindex Proper distance
-A very important issue that can be discussed now (while we are still in 2D and
can actually visualize things) is that @mymath{\overrightarrow{r}} is tangent
to the curved space at the observer's position.
-In other words, it is on the gray flat surface of @ref{sphereandplane}, even
when the universe if curved: @mymath{\overrightarrow{r}=P'-A}.
-Therefore for the point @mymath{P} on a curved space, the raw coordinate
@mymath{r} is the distance to @mymath{P'}, not @mymath{P}.
-The distance to the point @mymath{P} (at a specific coordinate @mymath{r} on
the flat plane) over the curved surface (thick line in @ref{sphereandplane}) is
called the @emph{proper distance} and is displayed with @mymath{l}.
-For the specific example of @ref{sphereandplane}, the proper distance can be
calculated with: @mymath{l=R\theta} (@mymath{\theta} is in radians).
-Using the @mymath{\sin\theta} relation found above, we can find @mymath{l} as
a function of @mymath{r}:
-
-@dispmath{\theta=\sin^{-1}\left({r\over R}\right)\quad\rightarrow\quad
-l(r)=R\sin^{-1}\left({r\over R}\right)}
-
-
-@mymath{R} is just an arbitrary constant and can be directly found from
@mymath{K}, so for cleaner equations, it is common practice to set
@mymath{R=1}, which gives: @mymath{l(r)=\sin^{-1}r}.
-Also note that when @mymath{R=1}, then @mymath{l=\theta}.
-Generally, depending on the curvature, in a @emph{static} universe the proper
distance can be written as a function of the coordinate @mymath{r} as (from now
on we are assuming @mymath{R=1}):
-
-@dispmath{l(r)=\sin^{-1}(r)\quad(K>0),\quad\quad
-l(r)=r\quad(K=0),\quad\quad l(r)=\sinh^{-1}(r)\quad(K<0).}With
-@mymath{l}, the infinitesimal change of distance can be written in a
-more simpler and abstract form of
-
-@dispmath{ds_s^2=dl^2+r^2d\phi^2.}
-
-@cindex Comoving distance
-Until now, we had assumed a static universe (not changing with time).
-But our observations so far appear to indicate that the universe is expanding
(it isn't static).
-Since there is no reason to expect the observed expansion is unique to our
particular position of the universe, we expect the universe to be expanding at
all points with the same rate at the same time.
-Therefore, to add a time dependence to our distance measurements, we can
include a multiplicative scaling factor, which is a function of time:
@mymath{a(t)}.
-The functional form of @mymath{a(t)} comes from the cosmology, the physics we
assume for it: general relativity, and the choice of whether the universe is
uniform (`homogeneous') in density and curvature or inhomogeneous.
-In this section, the functional form of @mymath{a(t)} is irrelevant, so we can
avoid these issues.
-
-With this scaling factor, the proper distance will also depend on time.
-As the universe expands, the distance between two given points will shift to
larger values.
-We thus define a distance measure, or coordinate, that is independent of time
and thus doesn't `move'.
-We call it the @emph{comoving distance} and display with @mymath{\chi} such
that: @mymath{l(r,t)=\chi(r)a(t)}.
-We have therefore, shifted the @mymath{r} dependence of the proper distance we
derived above for a static universe to the comoving distance:
-
-@dispmath{\chi(r)=\sin^{-1}(r)\quad(K>0),\quad\quad
-\chi(r)=r\quad(K=0),\quad\quad \chi(r)=\sinh^{-1}(r)\quad(K<0).}
-
-Therefore, @mymath{\chi(r)} is the proper distance to an object at a specific
reference time: @mymath{t=t_r} (the @mymath{r} subscript signifies
``reference'') when @mymath{a(t_r)=1}.
-At any arbitrary moment (@mymath{t\neq{t_r}}) before or after @mymath{t_r},
the proper distance to the object can be scaled with @mymath{a(t)}.
-
-Measuring the change of distance in a time-dependent (expanding) universe only
makes sense if we can add up space and time@footnote{In other words, making our
space-time consistent with Minkowski space-time geometry.
-In this geometry, different observers at a given point (event) in space-time
split up space-time into `space' and `time' in different ways, just like people
at the same spatial position can make different choices of splitting up a map
into `left--right' and `up--down'.
-This model is well supported by twentieth and twenty-first century
observations.}.
-But we can only add bits of space and time together if we measure them in the
same units: with a conversion constant (similar to how 1000 is used to convert
a kilometer into meters).
-Experimentally, we find strong support for the hypothesis that this conversion
constant is the speed of light (or gravitational waves@footnote{The speed of
gravitational waves was recently found to be very similar to that of light in
vacuum, see @url{https://arxiv.org/abs/1710.05834, arXiv:1710.05834}.}) in a
vacuum.
-This speed is postulated to be constant@footnote{In @emph{natural units},
speed is measured in units of the speed of light in vacuum.} and is almost
always written as @mymath{c}.
-We can thus parameterize the change in distance on an expanding 2D surface as
-
-@dispmath{ds^2=c^2dt^2-a^2(t)ds_s^2 = c^2dt^2-a^2(t)(d\chi^2+r^2d\phi^2).}
-
-
-@node Extending distance concepts to 3D, Invoking astcosmiccal, Distance on a
2D curved space, CosmicCalculator
-@subsection Extending distance concepts to 3D
-
-The concepts of @ref{Distance on a 2D curved space} are here extended to a 3D
space that @emph{might} be curved.
-We can start with the generic infinitesimal distance in a static 3D universe,
but this time in spherical coordinates instead of polar coordinates.
-@mymath{\theta} is shown in @ref{sphereandplane}, but here we are 3D beings,
positioned on @mymath{O} (the center of the sphere) and the point @mymath{O} is
tangent to a 4D-sphere.
-In our 3D space, a generic infinitesimal displacement will correspond to the
following distance in spherical coordinates:
-
-@dispmath{ds_s^2=dx^2+dy^2+dz^2=dr^2+r^2(d\theta^2+\sin^2{\theta}d\phi^2).}
-
-Like the 2D creature before, we now have to assume an abstract dimension which
we cannot visualize easily.
-Let's call the fourth dimension @mymath{w}, then the general change in
coordinates in the @emph{full} four dimensional space will be:
-
-@dispmath{ds_s^2=dr^2+r^2(d\theta^2+\sin^2{\theta}d\phi^2)+dw^2.}
-
-@noindent
-But we can only work on a 3D curved space, so following exactly the same steps
and conventions as our 2D friend, we arrive at:
-
-@dispmath{ds_s^2={dr^2\over 1-Kr^2}+r^2(d\theta^2+\sin^2{\theta}d\phi^2).}
-
-@noindent
-In a non-static universe (with a scale factor a(t)), the distance can be
written as:
-
-@dispmath{ds^2=c^2dt^2-a^2(t)[d\chi^2+r^2(d\theta^2+\sin^2{\theta}d\phi^2)].}
-
-
+@end table
-@c@dispmath{H(z){\equiv}\left(\dot{a}\over a\right)(z)=H_0E(z) }
-@c@dispmath{E(z)=[ \Omega_{\Lambda,0} + \Omega_{C,0}(1+z)^2 +
-@c\Omega_{m,0}(1+z)^3 + \Omega_{r,0}(1+z)^4 ]^{1/2}}
-@c Let's take @mymath{r} to be the radial coordinate of the emitting
-@c source, which emitted its light at redshift $z$. Then the comoving
-@c distance of this object would be:
-@c@dispmath{ \chi(r)={c\over H_0a_0}\int_0^z{dz'\over E(z')} }
-@c@noindent
-@c So the proper distance at the current time to that object is:
-@c @mymath{a_0\chi(r)}, therefore the angular diameter distance
-@c (@mymath{d_A}) and luminosity distance (@mymath{d_L}) can be written
-@c as:
-@c@dispmath{ d_A={a_0\chi(r)\over 1+z}, \quad d_L=a_0\chi(r)(1+z) }
-@node Invoking astcosmiccal, , Extending distance concepts to 3D,
CosmicCalculator
-@subsection Invoking CosmicCalculator
-CosmicCalculator will calculate cosmological variables based on the input
parameters.
-The executable name is @file{astcosmiccal} with the following general template
-@example
-$ astcosmiccal [OPTION...] ...
-@end example
-@noindent
-One line examples:
-@example
-## Print basic cosmological properties at redshift 2.5:
-$ astcosmiccal -z2.5
-## Only print Comoving volume over 4pi stradian to z (Mpc^3):
-$ astcosmiccal --redshift=0.8 --volume
-## Print redshift and age of universe when Lyman-alpha line is
-## at 6000 angstrom (another way to specify redshift).
-$ astcosmiccal --obsline=lyalpha,6000 --age
-## Print luminosity distance, angular diameter distance and age
-## of universe in one row at redshift 0.4
-$ astcosmiccal -z0.4 -LAg
+@node High-level calculations, Installed scripts, Modeling and fittings, Top
+@chapter High-level calculations
-## Assume Lambda and matter density of 0.7 and 0.3 and print
-## basic cosmological parameters for redshift 2.1:
-$ astcosmiccal -l0.7 -m0.3 -z2.1
+After the reduction of raw data (for example with the programs in @ref{Data
manipulation}) you will have reduced images/data ready for processing/analyzing
(for example with the programs in @ref{Data analysis}).
+But the processed/analyzed data (or catalogs) are still not enough to derive
any scientific result.
+Even higher-level analysis is still needed to convert the observed magnitudes,
sizes or volumes into physical quantities that we associate with each catalog
entry or detected object which is the purpose of the tools in this section.
-## Print wavelength of all pre-defined spectral lines when
-## Lyman-alpha is observed at 4000 Angstroms.
-$ astcosmiccal --obsline=lyalpha,4000 --listlinesatz
-@end example
-The input parameters (for example current matter density, etc) can be given as
command-line options or in the configuration files, see @ref{Configuration
files}.
-For a definition of the different parameters, please see the sections prior to
this.
-If no redshift is given, CosmicCalculator will just print its input parameters
and abort.
-For a full list of the input options, please see @ref{CosmicCalculator input
options}.
-Without any particular output requested (and only a given redshift),
CosmicCalculator will print all basic cosmological calculations (one per line)
with some explanations before each.
-This can be good when you want a general feeling of the conditions at a
specific redshift.
-Alternatively, if any specific calculation(s) are requested (its possible to
call more than one), only the requested value(s) will be calculated and printed
with one character space between them.
-In this case, no description or units will be printed.
-See @ref{CosmicCalculator basic cosmology calculations} for the full list of
these options along with some explanations how when/how they can be useful.
-Another common operation in observational cosmology is dealing with spectral
lines at different redshifts.
-CosmicCalculator also has features to help in such situations, please see
@ref{CosmicCalculator spectral line calculations}.
@menu
-* CosmicCalculator input options:: Options to specify input conditions.
-* CosmicCalculator basic cosmology calculations:: Like distance modulus,
distances and etc.
-* CosmicCalculator spectral line calculations:: How they get affected by
redshift.
+* CosmicCalculator:: Calculate cosmological variables
@end menu
-@node CosmicCalculator input options, CosmicCalculator basic cosmology
calculations, Invoking astcosmiccal, Invoking astcosmiccal
-@subsubsection CosmicCalculator input options
-
-The inputs to CosmicCalculator can be specified with the following options:
-@table @option
-
-@item -z FLT
-@itemx --redshift=FLT
-The redshift of interest.
-There are two other ways that you can specify the target redshift:
-1) Spectral lines and their observed wavelengths, see @option{--obsline}.
-2) Velocity, see @option{--velocity}.
-Hence this option cannot be called with @option{--obsline} or
@option{--velocity}.
+@node CosmicCalculator, , High-level calculations, High-level calculations
+@section CosmicCalculator
-@item -y FLT
-@itemx --velocity=FLT
-Input velocity in km/s.
-The given value will be converted to redshift internally, and used in any
subsequent calculation.
-This option is thus an alternative to @code{--redshift} or @code{--obsline},
it cannot be used with them.
-The conversion will be done with the more general and accurate relativistic
equation of @mymath{1+z=\sqrt{(c+v)/(c-v)}}, not the simplified
@mymath{z\approx v/c}.
+To derive higher-level information regarding our sources in extra-galactic
astronomy, cosmological calculations are necessary.
+In Gnuastro, CosmicCalculator is in charge of such calculations.
+Before discussing how CosmicCalculator is called and operates (in
@ref{Invoking astcosmiccal}), it is important to provide a rough but mostly
self sufficient review of the basics and the equations used in the analysis.
+In @ref{Distance on a 2D curved space} the basic idea of understanding
distances in a curved and expanding 2D universe (which we can visualize) are
reviewed.
+Having solidified the concepts there, in @ref{Extending distance concepts to
3D}, the formalism is extended to the 3D universe we are trying to study in our
research.
-@item -H FLT
-@itemx --H0=FLT
-Current expansion rate (in km sec@mymath{^{-1}} Mpc@mymath{^{-1}}).
+The focus here is obtaining a physical insight into these equations (mainly
for the use in real observational studies).
+There are many books thoroughly deriving and proving all the equations with
all possible initial conditions and assumptions for any abstract universe,
interested readers can study those books.
-@item -l FLT
-@itemx --olambda=FLT
-Cosmological constant density divided by the critical density in the current
Universe (@mymath{\Omega_{\Lambda,0}}).
+@menu
+* Distance on a 2D curved space:: Distances in 2D for simplicity
+* Extending distance concepts to 3D:: Going to 3D (our real universe).
+* Invoking astcosmiccal:: How to run CosmicCalculator
+@end menu
-@item -m FLT
-@itemx --omatter=FLT
-Matter (including massive neutrinos) density divided by the critical density
in the current Universe (@mymath{\Omega_{m,0}}).
+@node Distance on a 2D curved space, Extending distance concepts to 3D,
CosmicCalculator, CosmicCalculator
+@subsection Distance on a 2D curved space
-@item -r FLT
-@itemx --oradiation=FLT
-Radiation density divided by the critical density in the current Universe
(@mymath{\Omega_{r,0}}).
+The observations to date (for example the Planck 2015 results), have not
measured@footnote{The observations are interpreted under the assumption of
uniform curvature.
+For a relativistic alternative to dark energy (and maybe also some part of
dark matter), non-uniform curvature may be even be more critical, but that is
beyond the scope of this brief explanation.} the presence of significant
curvature in the universe.
+However to be generic (and allow its measurement if it does in fact exist), it
is very important to create a framework that allows non-zero uniform curvature.
+However, this section is not intended to be a fully thorough and
mathematically complete derivation of these concepts.
+There are many references available for such reviews that go deep into the
abstract mathematical proofs.
+The emphasis here is on visualization of the concepts for a beginner.
-@item -O STR/FLT,FLT
-@itemx --obsline=STR/FLT,FLT
-@cindex Rest-frame wavelength
-@cindex Wavelength, rest-frame
-Find the redshift to use in next steps based on the rest-frame and observed
wavelengths of a line.
-This option is thus an alternative to @code{--redshift} or @code{--velocity},
it cannot be used with them.
-Wavelengths are assumed to be in Angstroms.
-The first argument identifies the line.
-It can be one of the standard names below, or any rest-frame wavelength in
Angstroms.
-The second argument is the observed wavelength of that line.
-For example @option{--obsline=lyalpha,6000} is the same as
@option{--obsline=1215.64,6000}.
+As 3D beings, it is difficult for us to mentally create (visualize) a picture
of the curvature of a 3D volume.
+Hence, here we will assume a 2D surface/space and discuss distances on that 2D
surface when it is flat and when it is curved.
+Once the concepts have been created/visualized here, we will extend them, in
@ref{Extending distance concepts to 3D}, to a real 3D spatial @emph{slice} of
the Universe we live in and hope to study.
-The pre-defined names are listed below, sorted from red (longer wavelength) to
blue (shorter wavelength).
-You can get this list on the command-line with the @option{--listlines}.
+To be more understandable (actively discuss from an observer's point of view)
let's assume there's an imaginary 2D creature living on the 2D space (which
@emph{might} be curved in 3D).
+Here, we will be working with this creature in its efforts to analyze
distances in its 2D universe.
+The start of the analysis might seem too mundane, but since it is difficult to
imagine a 3D curved space, it is important to review all the very basic
concepts thoroughly for an easy transition to a universe that is more difficult
to visualize (a curved 3D space embedded in 4D).
-@table @code
-@item siired
-[6731@AA{}] SII doublet's redder line.
+To start, let's assume a static (not expanding or shrinking), flat 2D surface
similar to @ref{flatplane} and that the 2D creature is observing its universe
from point @mymath{A}.
+One of the most basic ways to parameterize this space is through the Cartesian
coordinates (@mymath{x}, @mymath{y}).
+In @ref{flatplane}, the basic axes of these two coordinates are plotted.
+An infinitesimal change in the direction of each axis is written as
@mymath{dx} and @mymath{dy}.
+For each point, the infinitesimal changes are parallel with the respective
axes and are not shown for clarity.
+Another very useful way of parameterizing this space is through polar
coordinates.
+For each point, we define a radius (@mymath{r}) and angle (@mymath{\phi}) from
a fixed (but arbitrary) reference axis.
+In @ref{flatplane} the infinitesimal changes for each polar coordinate are
plotted for a random point and a dashed circle is shown for all points with the
same radius.
-@item sii
-@cindex Doublet: SII
-@cindex SII doublet
-[6724@AA{}] SII doublet's mean center at .
+@float Figure,flatplane
+@center@image{gnuastro-figures/flatplane, 10cm, , }
-@item siiblue
-[6717@AA{}] SII doublet's bluer line.
+@caption{Two dimensional Cartesian and polar coordinates on a flat
+plane.}
+@end float
-@item niired
-[6584@AA{}] NII doublet's redder line.
+Assuming an object is placed at a certain position, which can be parameterized
as @mymath{(x,y)}, or @mymath{(r,\phi)}, a general infinitesimal change in its
position will place it in the coordinates @mymath{(x+dx,y+dy)}, or
@mymath{(r+dr,\phi+d\phi)}.
+The distance (on the flat 2D surface) that is covered by this infinitesimal
change in the static universe (@mymath{ds_s}, the subscript signifies the
static nature of this universe) can be written as:
-@item nii
-@cindex Doublet: NII
-@cindex NII doublet
-[6566@AA{}] NII doublet's mean center.
+@dispmath{ds_s^2=dx^2+dy^2=dr^2+r^2d\phi^2}
-@item halpha
-@cindex H-alpha
-[6562.8@AA{}] H-@mymath{\alpha} line.
+The main question is this: how can the 2D creature incorporate the (possible)
curvature in its universe when it's calculating distances? The universe that it
lives in might equally be a curved surface like @ref{sphereandplane}.
+The answer to this question but for a 3D being (us) is the whole purpose to
this discussion.
+Here, we want to give the 2D creature (and later, ourselves) the tools to
measure distances if the space (that hosts the objects) is curved.
-@item niiblue
-[6548@AA{}] NII doublet's bluer line.
+@ref{sphereandplane} assumes a spherical shell with radius @mymath{R} as the
curved 2D plane for simplicity.
+The 2D plane is tangent to the spherical shell and only touches it at
@mymath{A}.
+This idea will be generalized later.
+The first step in measuring the distance in a curved space is to imagine a
third dimension along the @mymath{z} axis as shown in @ref{sphereandplane}.
+For simplicity, the @mymath{z} axis is assumed to pass through the center of
the spherical shell.
+Our imaginary 2D creature cannot visualize the third dimension or a curved 2D
surface within it, so the remainder of this discussion is purely abstract for
it (similar to us having difficulty in visualizing a 3D curved space in 4D).
+But since we are 3D creatures, we have the advantage of visualizing the
following steps.
+Fortunately the 2D creature is already familiar with our mathematical
constructs, so it can follow our reasoning.
-@item oiiired-vis
-[5007@AA{}] OIII doublet's redder line in the visible.
+With the third axis added, a generic infinitesimal change over @emph{the full}
3D space corresponds to the distance:
-@item oiii-vis
-@cindex Doublet: OIII (visible)
-@cindex OIII doublet in visible
-[4983@AA{}] OIII doublet's mean center in the visible.
+@dispmath{ds_s^2=dx^2+dy^2+dz^2=dr^2+r^2d\phi^2+dz^2.}
-@item oiiiblue-vis
-[4959@AA{}] OIII doublet's bluer line in the visible.
+@float Figure,sphereandplane
+@center@image{gnuastro-figures/sphereandplane, 10cm, , }
-@item hbeta
-@cindex H-beta
-[4861.36@AA{}] H-@mymath{\beta} line.
+@caption{2D spherical shell (centered on @mymath{O}) and flat plane (light
gray) tangent to it at point @mymath{A}.}
+@end float
-@item heii-vis
-[4686@AA{}] HeII doublet's redder line in the visible.
+It is very important to recognize that this change of distance is for
@emph{any} point in the 3D space, not just those changes that occur on the 2D
spherical shell of @ref{sphereandplane}.
+Recall that our 2D friend can only do measurements on the 2D surfaces, not the
full 3D space.
+So we have to constrain this general change to any change on the 2D spherical
shell.
+To do that, let's look at the arbitrary point @mymath{P} on the 2D spherical
shell.
+Its image (@mymath{P'}) on the flat plain is also displayed. From the dark
gray triangle, we see that
-@item hgamma
-@cindex H-gamma
-[4340.46@AA{}] H-@mymath{\gamma} line.
+@dispmath{\sin\theta={r\over R},\quad\cos\theta={R-z\over R}.}These relations
allow the 2D creature to find the value of @mymath{z} (an abstract dimension
for it) as a function of r (distance on a flat 2D plane, which it can
visualize) and thus eliminate @mymath{z}.
+From @mymath{\sin^2\theta+\cos^2\theta=1}, we get @mymath{z^2-2Rz+r^2=0} and
solving for @mymath{z}, we find:
-@item hdelta
-@cindex H-delta
-[4101.74@AA{}] H-@mymath{\delta} line.
+@dispmath{z=R\left(1\pm\sqrt{1-{r^2\over R^2}}\right).}
-@item hepsilon
-@cindex H-epsilon
-[3970.07@AA{}] H-@mymath{\epsilon} line.
+The @mymath{\pm} can be understood from @ref{sphereandplane}: For each
@mymath{r}, there are two points on the sphere, one in the upper hemisphere and
one in the lower hemisphere.
+An infinitesimal change in @mymath{r}, will create the following infinitesimal
change in @mymath{z}:
-@item neiii
-[3869@AA{}] NEIII line.
+@dispmath{dz={\mp r\over R}\left(1\over
+\sqrt{1-{r^2/R^2}}\right)dr.}Using the positive signed equation instead of
@mymath{dz} in the @mymath{ds_s^2} equation above, we get:
-@item oiired
-[3729@AA{}] OII doublet's redder line.
+@dispmath{ds_s^2={dr^2\over 1-r^2/R^2}+r^2d\phi^2.}
-@item oii
-@cindex Doublet: OII
-@cindex OII doublet
-[3727.5@AA{}] OII doublet's mean center.
+The derivation above was done for a spherical shell of radius @mymath{R} as a
curved 2D surface.
+To generalize it to any surface, we can define @mymath{K=1/R^2} as the
curvature parameter.
+Then the general infinitesimal change in a static universe can be written as:
-@item oiiblue
-[3726@AA{}] OII doublet's bluer line.
+@dispmath{ds_s^2={dr^2\over 1-Kr^2}+r^2d\phi^2.}
-@item blimit
-@cindex Balmer limit
-[3646@AA{}] Balmer limit.
+Therefore, when @mymath{K>0} (and curvature is the same everywhere), we have a
finite universe, where @mymath{r} cannot become larger than @mymath{R} as in
@ref{sphereandplane}.
+When @mymath{K=0}, we have a flat plane (@ref{flatplane}) and a negative
@mymath{K} will correspond to an imaginary @mymath{R}.
+The latter two cases may be infinite in area (which is not a simple concept,
but mathematically can be modeled with @mymath{r} extending infinitely), or
finite-area (like a cylinder is flat everywhere with @mymath{ds_s^2={dx^2 +
dy^2}}, but finite in one direction in size).
-@item mgiired
-[2803@AA{}] MgII doublet's redder line.
+@cindex Proper distance
+A very important issue that can be discussed now (while we are still in 2D and
can actually visualize things) is that @mymath{\overrightarrow{r}} is tangent
to the curved space at the observer's position.
+In other words, it is on the gray flat surface of @ref{sphereandplane}, even
when the universe if curved: @mymath{\overrightarrow{r}=P'-A}.
+Therefore for the point @mymath{P} on a curved space, the raw coordinate
@mymath{r} is the distance to @mymath{P'}, not @mymath{P}.
+The distance to the point @mymath{P} (at a specific coordinate @mymath{r} on
the flat plane) over the curved surface (thick line in @ref{sphereandplane}) is
called the @emph{proper distance} and is displayed with @mymath{l}.
+For the specific example of @ref{sphereandplane}, the proper distance can be
calculated with: @mymath{l=R\theta} (@mymath{\theta} is in radians).
+Using the @mymath{\sin\theta} relation found above, we can find @mymath{l} as
a function of @mymath{r}:
-@item mgii
-@cindex Doublet: MgII
-@cindex MgII doublet
-[2799.5@AA{}] MgII doublet's mean center.
+@dispmath{\theta=\sin^{-1}\left({r\over R}\right)\quad\rightarrow\quad
+l(r)=R\sin^{-1}\left({r\over R}\right)}
-@item mgiiblue
-[2796@AA{}] MgII doublet's bluer line.
-@item ciiired
-[1909@AA{}] CIII doublet's redder line.
+@mymath{R} is just an arbitrary constant and can be directly found from
@mymath{K}, so for cleaner equations, it is common practice to set
@mymath{R=1}, which gives: @mymath{l(r)=\sin^{-1}r}.
+Also note that when @mymath{R=1}, then @mymath{l=\theta}.
+Generally, depending on the curvature, in a @emph{static} universe the proper
distance can be written as a function of the coordinate @mymath{r} as (from now
on we are assuming @mymath{R=1}):
-@item ciii
-@cindex Doublet: CIII
-@cindex CIII doublet
-[1908@AA{}] CIII doublet's mean center.
+@dispmath{l(r)=\sin^{-1}(r)\quad(K>0),\quad\quad
+l(r)=r\quad(K=0),\quad\quad l(r)=\sinh^{-1}(r)\quad(K<0).}With
+@mymath{l}, the infinitesimal change of distance can be written in a
+more simpler and abstract form of
-@item ciiiblue
-[1907@AA{}] CIII doublet's bluer line.
+@dispmath{ds_s^2=dl^2+r^2d\phi^2.}
-@item si_iiired
-[1892@AA{}] SiIII doublet's redder line.
+@cindex Comoving distance
+Until now, we had assumed a static universe (not changing with time).
+But our observations so far appear to indicate that the universe is expanding
(it isn't static).
+Since there is no reason to expect the observed expansion is unique to our
particular position of the universe, we expect the universe to be expanding at
all points with the same rate at the same time.
+Therefore, to add a time dependence to our distance measurements, we can
include a multiplicative scaling factor, which is a function of time:
@mymath{a(t)}.
+The functional form of @mymath{a(t)} comes from the cosmology, the physics we
assume for it: general relativity, and the choice of whether the universe is
uniform (`homogeneous') in density and curvature or inhomogeneous.
+In this section, the functional form of @mymath{a(t)} is irrelevant, so we can
avoid these issues.
-@item si_iii
-@cindex Doublet: SiIII
-@cindex SiIII doublet
-[1887.5@AA{}] SiIII doublet's mean center.
+With this scaling factor, the proper distance will also depend on time.
+As the universe expands, the distance between two given points will shift to
larger values.
+We thus define a distance measure, or coordinate, that is independent of time
and thus doesn't `move'.
+We call it the @emph{comoving distance} and display with @mymath{\chi} such
that: @mymath{l(r,t)=\chi(r)a(t)}.
+We have therefore, shifted the @mymath{r} dependence of the proper distance we
derived above for a static universe to the comoving distance:
-@item si_iiiblue
-[1883@AA{}] SiIII doublet's bluer line.
+@dispmath{\chi(r)=\sin^{-1}(r)\quad(K>0),\quad\quad
+\chi(r)=r\quad(K=0),\quad\quad \chi(r)=\sinh^{-1}(r)\quad(K<0).}
-@item oiiired-uv
-[1666@AA{}] OIII doublet's redder line in the ultra-violet.
+Therefore, @mymath{\chi(r)} is the proper distance to an object at a specific
reference time: @mymath{t=t_r} (the @mymath{r} subscript signifies
``reference'') when @mymath{a(t_r)=1}.
+At any arbitrary moment (@mymath{t\neq{t_r}}) before or after @mymath{t_r},
the proper distance to the object can be scaled with @mymath{a(t)}.
-@item oiii-uv
-@cindex Doublet: OIII (in UV)
-@cindex OIII doublet in UV
-[1663.5@AA{}] OIII doublet's mean center in the ultra-violet.
+Measuring the change of distance in a time-dependent (expanding) universe only
makes sense if we can add up space and time@footnote{In other words, making our
space-time consistent with Minkowski space-time geometry.
+In this geometry, different observers at a given point (event) in space-time
split up space-time into `space' and `time' in different ways, just like people
at the same spatial position can make different choices of splitting up a map
into `left--right' and `up--down'.
+This model is well supported by twentieth and twenty-first century
observations.}.
+But we can only add bits of space and time together if we measure them in the
same units: with a conversion constant (similar to how 1000 is used to convert
a kilometer into meters).
+Experimentally, we find strong support for the hypothesis that this conversion
constant is the speed of light (or gravitational waves@footnote{The speed of
gravitational waves was recently found to be very similar to that of light in
vacuum, see @url{https://arxiv.org/abs/1710.05834, arXiv:1710.05834}.}) in a
vacuum.
+This speed is postulated to be constant@footnote{In @emph{natural units},
speed is measured in units of the speed of light in vacuum.} and is almost
always written as @mymath{c}.
+We can thus parameterize the change in distance on an expanding 2D surface as
-@item oiiiblue-uv
-[1661@AA{}] OIII doublet's bluer line in the ultra-violet.
+@dispmath{ds^2=c^2dt^2-a^2(t)ds_s^2 = c^2dt^2-a^2(t)(d\chi^2+r^2d\phi^2).}
-@item heii-uv
-[1640@AA{}] HeII doublet's bluer line in the ultra-violet.
-@item civred
-[1551@AA{}] CIV doublet's redder line.
+@node Extending distance concepts to 3D, Invoking astcosmiccal, Distance on a
2D curved space, CosmicCalculator
+@subsection Extending distance concepts to 3D
-@item civ
-@cindex Doublet: CIV
-@cindex CIV doublet
-[1549@AA{}] CIV doublet's mean center.
+The concepts of @ref{Distance on a 2D curved space} are here extended to a 3D
space that @emph{might} be curved.
+We can start with the generic infinitesimal distance in a static 3D universe,
but this time in spherical coordinates instead of polar coordinates.
+@mymath{\theta} is shown in @ref{sphereandplane}, but here we are 3D beings,
positioned on @mymath{O} (the center of the sphere) and the point @mymath{O} is
tangent to a 4D-sphere.
+In our 3D space, a generic infinitesimal displacement will correspond to the
following distance in spherical coordinates:
-@item civblue
-[1548@AA{}] CIV doublet's bluer line.
+@dispmath{ds_s^2=dx^2+dy^2+dz^2=dr^2+r^2(d\theta^2+\sin^2{\theta}d\phi^2).}
-@item nv
-[1240@AA{}] NV (four times ionized Sodium).
+Like the 2D creature before, we now have to assume an abstract dimension which
we cannot visualize easily.
+Let's call the fourth dimension @mymath{w}, then the general change in
coordinates in the @emph{full} four dimensional space will be:
-@item lyalpha
-@cindex Lyman-alpha
-[1215.67@AA{}] Lyman-@mymath{\alpha} line.
+@dispmath{ds_s^2=dr^2+r^2(d\theta^2+\sin^2{\theta}d\phi^2)+dw^2.}
-@item lybeta
-@cindex Lyman-beta
-[1025.7@AA{}] Lyman-@mymath{\beta} line.
+@noindent
+But we can only work on a 3D curved space, so following exactly the same steps
and conventions as our 2D friend, we arrive at:
-@item lygamma
-@cindex Lyman-gamma
-[972.54@AA{}] Lyman-@mymath{\gamma} line.
+@dispmath{ds_s^2={dr^2\over 1-Kr^2}+r^2(d\theta^2+\sin^2{\theta}d\phi^2).}
-@item lydelta
-@cindex Lyman-delta
-[949.74@AA{}] Lyman-@mymath{\delta} line.
+@noindent
+In a non-static universe (with a scale factor a(t)), the distance can be
written as:
-@item lyepsilon
-@cindex Lyman-epsilon
-[937.80@AA{}] Lyman-@mymath{\epsilon} line.
+@dispmath{ds^2=c^2dt^2-a^2(t)[d\chi^2+r^2(d\theta^2+\sin^2{\theta}d\phi^2)].}
-@item lylimit
-@cindex Lyman limit
-[912@AA{}] Lyman limit.
-@end table
-@end table
+@c@dispmath{H(z){\equiv}\left(\dot{a}\over a\right)(z)=H_0E(z) }
+@c@dispmath{E(z)=[ \Omega_{\Lambda,0} + \Omega_{C,0}(1+z)^2 +
+@c\Omega_{m,0}(1+z)^3 + \Omega_{r,0}(1+z)^4 ]^{1/2}}
+@c Let's take @mymath{r} to be the radial coordinate of the emitting
+@c source, which emitted its light at redshift $z$. Then the comoving
+@c distance of this object would be:
-@node CosmicCalculator basic cosmology calculations, CosmicCalculator spectral
line calculations, CosmicCalculator input options, Invoking astcosmiccal
-@subsubsection CosmicCalculator basic cosmology calculations
-By default, when no specific calculations are requested, CosmicCalculator will
print a complete set of all its calculators (one line for each calculation, see
@ref{Invoking astcosmiccal}).
-The full list of calculations can be useful when you don't want any specific
value, but just a general view.
-In other contexts (for example in a batch script or during a discussion), you
know exactly what you want and don't want to be distracted by all the extra
information.
+@c@dispmath{ \chi(r)={c\over H_0a_0}\int_0^z{dz'\over E(z')} }
-You can use any number of the options described below in any order.
-When any of these options are requested, CosmicCalculator's output will just
be a single line with a single space between the (possibly) multiple values.
-In the example below, only the tangential distance along one arc-second (in
kpc), absolute magnitude conversion, and age of the universe at redshift 2 are
printed (recall that you can merge short options together, see @ref{Options}).
+@c@noindent
+@c So the proper distance at the current time to that object is:
+@c @mymath{a_0\chi(r)}, therefore the angular diameter distance
+@c (@mymath{d_A}) and luminosity distance (@mymath{d_L}) can be written
+@c as:
-@example
-$ astcosmiccal -z2 -sag
-8.585046 44.819248 3.289979
-@end example
+@c@dispmath{ d_A={a_0\chi(r)\over 1+z}, \quad d_L=a_0\chi(r)(1+z) }
-Here is one example of using this feature in scripts: by adding the following
two lines in a script to keep/use the comoving volume with varying redshifts:
-@example
-z=3.12
-vol=$(astcosmiccal --redshift=$z --volume)
-@end example
-@cindex GNU Grep
-@noindent
-In a script, this operation might be necessary for a large number of objects
(several of galaxies in a catalog for example).
-So the fact that all the other default calculations are ignored will also help
you get to your result faster.
-If you are indeed dealing with many (for example thousands) of redshifts,
using CosmicCalculator is not the best/fastest solution.
-Because it has to go through all the configuration files and preparations for
each invocation.
-To get the best efficiency (least overhead), we recommend using Gnuastro's
cosmology library (see @ref{Cosmology library}).
-CosmicCalculator also calls the library functions defined there for its
calculations, so you get the same result with no overhead.
-Gnuastro also has libraries for easily reading tables into a C program, see
@ref{Table input output}.
-Afterwards, you can easily build and run your C program for the particular
processing with @ref{BuildProgram}.
+@node Invoking astcosmiccal, , Extending distance concepts to 3D,
CosmicCalculator
+@subsection Invoking CosmicCalculator
-If you just want to inspect the value of a variable visually, the description
(which comes with units) might be more useful.
-In such cases, the following command might be better.
-The other calculations will also be done, but they are so fast that you will
not notice on modern computers (the time it takes your eye to focus on the
result is usually longer than the processing: a fraction of a second).
+CosmicCalculator will calculate cosmological variables based on the input
parameters.
+The executable name is @file{astcosmiccal} with the following general template
@example
-$ astcosmiccal --redshift=0.832 | grep volume
+$ astcosmiccal [OPTION...] ...
@end example
-The full list of CosmicCalculator's specific calculations is present below in
two groups: basic cosmology calculations and those related to spectral lines.
-In case you have forgot the units, you can use the @option{--help} option
which has the units along with a short description.
-@table @option
+@noindent
+One line examples:
-@item -e
-@itemx --usedredshift
-The redshift that was used in this run.
-In many cases this is the main input parameter to CosmicCalculator, but it is
useful in others.
-For example in combination with @option{--obsline} (where you give an observed
and rest-frame wavelength and would like to know the redshift) or with
@option{--velocity} (where you specify the velocity instead of redshift).
-Another example is when you run CosmicCalculator in a loop, while changing the
redshift and you want to keep the redshift value with the resulting calculation.
+@example
+## Print basic cosmological properties at redshift 2.5:
+$ astcosmiccal -z2.5
-@item -Y
-@itemx --usedvelocity
-The velocity (in km/s) that was used in this run.
-The conversion from redshift will be done with the more general and accurate
relativistic equation of @mymath{1+z=\sqrt{(c+v)/(c-v)}}, not the simplified
@mymath{z\approx v/c}.
+## Only print Comoving volume over 4pi stradian to z (Mpc^3):
+$ astcosmiccal --redshift=0.8 --volume
-@item -G
-@itemx --agenow
-The current age of the universe (given the input parameters) in Ga (Giga
annum, or billion years).
+## Print redshift and age of universe when Lyman-alpha line is
+## at 6000 angstrom (another way to specify redshift).
+$ astcosmiccal --obsline=lyalpha,6000 --age
-@item -C
-@itemx --criticaldensitynow
-The current critical density (given the input parameters) in grams per
centimeter-cube (@mymath{g/cm^3}).
+## Print luminosity distance, angular diameter distance and age
+## of universe in one row at redshift 0.4
+$ astcosmiccal -z0.4 -LAg
-@item -d
-@itemx --properdistance
-The proper distance (at current time) to object at the given redshift in
Megaparsecs (Mpc).
-See @ref{Distance on a 2D curved space} for a description of the proper
distance.
+## Assume Lambda and matter density of 0.7 and 0.3 and print
+## basic cosmological parameters for redshift 2.1:
+$ astcosmiccal -l0.7 -m0.3 -z2.1
-@item -A
-@itemx --angulardimdist
-The angular diameter distance to object at given redshift in Megaparsecs (Mpc).
+## Print wavelength of all pre-defined spectral lines when
+## Lyman-alpha is observed at 4000 Angstroms.
+$ astcosmiccal --obsline=lyalpha,4000 --listlinesatz
+@end example
-@item -s
-@itemx --arcsectandist
-The tangential distance covered by 1 arc-seconds at the given redshift in
kiloparsecs (Kpc).
-This can be useful when trying to estimate the resolution or pixel scale of an
instrument (usually in units of arc-seconds) at a given redshift.
+The input parameters (for example current matter density, etc) can be given as
command-line options or in the configuration files, see @ref{Configuration
files}.
+For a definition of the different parameters, please see the sections prior to
this.
+If no redshift is given, CosmicCalculator will just print its input parameters
and abort.
+For a full list of the input options, please see @ref{CosmicCalculator input
options}.
-@item -L
-@itemx --luminositydist
-The luminosity distance to object at given redshift in Megaparsecs (Mpc).
+Without any particular output requested (and only a given redshift),
CosmicCalculator will print all basic cosmological calculations (one per line)
with some explanations before each.
+This can be good when you want a general feeling of the conditions at a
specific redshift.
+Alternatively, if any specific calculation(s) are requested (its possible to
call more than one), only the requested value(s) will be calculated and printed
with one character space between them.
+In this case, no description or units will be printed.
+See @ref{CosmicCalculator basic cosmology calculations} for the full list of
these options along with some explanations how when/how they can be useful.
-@item -u
-@itemx --distancemodulus
-The distance modulus at given redshift.
+Another common operation in observational cosmology is dealing with spectral
lines at different redshifts.
+CosmicCalculator also has features to help in such situations, please see
@ref{CosmicCalculator spectral line calculations}.
-@item -a
-@itemx --absmagconv
-The conversion factor (addition) to absolute magnitude.
-Note that this is practically the distance modulus added with
@mymath{-2.5\log{(1+z)}} for the desired redshift based on the input parameters.
-Once the apparent magnitude and redshift of an object is known, this value may
be added with the apparent magnitude to give the object's absolute magnitude.
+@menu
+* CosmicCalculator input options:: Options to specify input conditions.
+* CosmicCalculator basic cosmology calculations:: Like distance modulus,
distances and etc.
+* CosmicCalculator spectral line calculations:: How they get affected by
redshift.
+@end menu
-@item -g
-@itemx --age
-Age of the universe at given redshift in Ga (Giga annum, or billion years).
+@node CosmicCalculator input options, CosmicCalculator basic cosmology
calculations, Invoking astcosmiccal, Invoking astcosmiccal
+@subsubsection CosmicCalculator input options
-@item -b
-@itemx --lookbacktime
-The look-back time to given redshift in Ga (Giga annum, or billion years).
-The look-back time at a given redshift is defined as the current age of the
universe (@option{--agenow}) subtracted by the age of the universe at the given
redshift.
+The inputs to CosmicCalculator can be specified with the following options:
+@table @option
-@item -c
-@itemx --criticaldensity
-The critical density at given redshift in grams per centimeter-cube
(@mymath{g/cm^3}).
+@item -z FLT
+@itemx --redshift=FLT
+The redshift of interest.
+There are two other ways that you can specify the target redshift:
+1) Spectral lines and their observed wavelengths, see @option{--obsline}.
+2) Velocity, see @option{--velocity}.
+Hence this option cannot be called with @option{--obsline} or
@option{--velocity}.
-@item -v
-@itemx --onlyvolume
-The comoving volume in Megaparsecs cube (Mpc@mymath{^3}) until the desired
redshift based on the input parameters.
+@item -y FLT
+@itemx --velocity=FLT
+Input velocity in km/s.
+The given value will be converted to redshift internally, and used in any
subsequent calculation.
+This option is thus an alternative to @code{--redshift} or @code{--obsline},
it cannot be used with them.
+The conversion will be done with the more general and accurate relativistic
equation of @mymath{1+z=\sqrt{(c+v)/(c-v)}}, not the simplified
@mymath{z\approx v/c}.
-@end table
+@item -H FLT
+@itemx --H0=FLT
+Current expansion rate (in km sec@mymath{^{-1}} Mpc@mymath{^{-1}}).
+@item -l FLT
+@itemx --olambda=FLT
+Cosmological constant density divided by the critical density in the current
Universe (@mymath{\Omega_{\Lambda,0}}).
+@item -m FLT
+@itemx --omatter=FLT
+Matter (including massive neutrinos) density divided by the critical density
in the current Universe (@mymath{\Omega_{m,0}}).
+@item -r FLT
+@itemx --oradiation=FLT
+Radiation density divided by the critical density in the current Universe
(@mymath{\Omega_{r,0}}).
-@node CosmicCalculator spectral line calculations, , CosmicCalculator basic
cosmology calculations, Invoking astcosmiccal
-@subsubsection CosmicCalculator spectral line calculations
+@item -O STR/FLT,FLT
+@itemx --obsline=STR/FLT,FLT
+@cindex Rest-frame wavelength
+@cindex Wavelength, rest-frame
+Find the redshift to use in next steps based on the rest-frame and observed
wavelengths of a line.
+This option is thus an alternative to @code{--redshift} or @code{--velocity},
it cannot be used with them.
+Wavelengths are assumed to be in Angstroms.
+The first argument identifies the line.
+It can be one of the standard names below, or any rest-frame wavelength in
Angstroms.
+The second argument is the observed wavelength of that line.
+For example @option{--obsline=lyalpha,6000} is the same as
@option{--obsline=1215.64,6000}.
-@cindex Rest frame wavelength
-At different redshifts, observed spectral lines are shifted compared to their
rest frame wavelengths with this simple relation:
@mymath{\lambda_{obs}=\lambda_{rest}(1+z)}.
-Although this relation is very simple and can be done for one line in the head
(or a simple calculator!), it slowly becomes tiring when dealing with a lot of
lines or redshifts, or some precision is necessary.
-The options in this section are thus provided to greatly simplify usage of
this simple equation, and also helping by storing a list of pre-defined
spectral line wavelengths.
+The pre-defined names are listed below, sorted from red (longer wavelength) to
blue (shorter wavelength).
+You can get this list on the command-line with the @option{--listlines}.
-For example if you want to know the wavelength of the @mymath{H\alpha} line
(at 6562.8 Angstroms in rest frame), when @mymath{Ly\alpha} is at 8000
Angstroms, you can call CosmicCalculator like the first example below.
-And if you want the wavelength of all pre-defined spectral lines at this
redshift, you can use the second command.
+@table @code
+@item siired
+[6731@AA{}] SII doublet's redder line.
-@example
-$ astcosmiccal --obsline=lyalpha,8000 --lineatz=halpha
-$ astcosmiccal --obsline=lyalpha,8000 --listlinesatz
-@end example
+@item sii
+@cindex Doublet: SII
+@cindex SII doublet
+[6724@AA{}] SII doublet's mean center at .
-Bellow you can see the printed/output calculations of CosmicCalculator that
are related to spectral lines.
-Note that @option{--obsline} is an input parameter, so its discussed (with the
full list of known lines) in @ref{CosmicCalculator input options}.
+@item siiblue
+[6717@AA{}] SII doublet's bluer line.
-@table @option
+@item niired
+[6584@AA{}] NII doublet's redder line.
-@item --listlines
-List the pre-defined rest frame spectral line wavelengths and their names on
standard output, then abort CosmicCalculator.
-When this option is given, other operations on the command-line will be
ignored.
-This is convenient when you forget the specific name of the spectral line used
within Gnuastro, or when you forget the exact wavelength of a certain line.
+@item nii
+@cindex Doublet: NII
+@cindex NII doublet
+[6566@AA{}] NII doublet's mean center.
-These names can be used with the options that deal with spectral lines, for
example @option{--obsline} and @option{--lineatz} (@ref{CosmicCalculator basic
cosmology calculations}).
+@item halpha
+@cindex H-alpha
+[6562.8@AA{}] H-@mymath{\alpha} line.
-The format of the output list is a two-column table, with Gnuastro's text
table format (see @ref{Gnuastro text table format}).
-Therefore, if you are only looking for lines in a specific range, you can pipe
the output into Gnuastro's table program and use its @option{--range} option on
the @code{wavelength} (first) column.
-For example, if you only want to see the lines between 4000 and 6000
Angstroms, you can run this command:
+@item niiblue
+[6548@AA{}] NII doublet's bluer line.
-@example
-$ astcosmiccal --listlines \
- | asttable --range=wavelength,4000,6000
-@end example
+@item oiiired-vis
+[5007@AA{}] OIII doublet's redder line in the visible.
-@noindent
-And if you want to use the list later and have it as a table in a file, you
can easily add the @option{--output} (or @option{-o}) option to the
@command{asttable} command, and specify the filename, for example
@option{--output=lines.fits} or @option{--output=lines.txt}.
+@item oiii-vis
+@cindex Doublet: OIII (visible)
+@cindex OIII doublet in visible
+[4983@AA{}] OIII doublet's mean center in the visible.
-@item --listlinesatz
-Similar to @option{--listlines} (above), but the printed wavelength is not in
the rest frame, but redshifted to the given redshift.
-Recall that the redshift can be specified by @option{--redshift} directly or
by @option{--obsline}, see @ref{CosmicCalculator input options}.
+@item oiiiblue-vis
+[4959@AA{}] OIII doublet's bluer line in the visible.
-@item -i STR/FLT
-@itemx --lineatz=STR/FLT
-The wavelength of the specified line at the redshift given to CosmicCalculator.
-The line can be specified either by its name or directly as a number (its
wavelength).
-To get the list of pre-defined names for the lines and their wavelength, you
can use the @option{--listlines} option, see @ref{CosmicCalculator input
options}.
-In the former case (when a name is given), the returned number is in units of
Angstroms.
-In the latter (when a number is given), the returned value is the same units
of the input number (assuming its a wavelength).
+@item hbeta
+@cindex H-beta
+[4861.36@AA{}] H-@mymath{\beta} line.
-@end table
+@item heii-vis
+[4686@AA{}] HeII doublet's redder line in the visible.
+@item hgamma
+@cindex H-gamma
+[4340.46@AA{}] H-@mymath{\gamma} line.
+@item hdelta
+@cindex H-delta
+[4101.74@AA{}] H-@mymath{\delta} line.
+@item hepsilon
+@cindex H-epsilon
+[3970.07@AA{}] H-@mymath{\epsilon} line.
+@item neiii
+[3869@AA{}] NEIII line.
+@item oiired
+[3729@AA{}] OII doublet's redder line.
+@item oii
+@cindex Doublet: OII
+@cindex OII doublet
+[3727.5@AA{}] OII doublet's mean center.
+@item oiiblue
+[3726@AA{}] OII doublet's bluer line.
-@node Installed scripts, Library, High-level calculations, Top
-@chapter Installed scripts
+@item blimit
+@cindex Balmer limit
+[3646@AA{}] Balmer limit.
-Gnuastro's programs (introduced in previous chapters) are designed to be
highly modular and thus contain lower-level operations on the data.
-However, in many contexts, certain higher-level are also shared between many
contexts.
-For example a sequence of calls to multiple Gnuastro programs, or a special
way of running a program and treating the output.
-To facilitate such higher-level data analysis, Gnuastro also installs some
scripts on your system with the (@code{astscript-}) prefix (in contrast to the
other programs that only have the @code{ast} prefix).
+@item mgiired
+[2803@AA{}] MgII doublet's redder line.
-@cindex GNU Bash
-@cindex Portable shell
-@cindex Shell, portable
-Like all of Gnuastro's source code, these scripts are also heavily commented.
-They are written in portable shell scripts (command-line environments), which
doesn't need compilation.
-Therefore, if you open the installed scripts in a text editor, you can
actually read them@footnote{Gnuastro's installed programs (those only starting
with @code{ast}) aren't human-readable.
-They are written in C and need to be compiled before execution.
-Compilation optimizes the steps into the low-level hardware CPU
instructions/language to improve efficiency.
-Because compiled programs don't need an interpreter like Bash on every run,
they are much faster and more independent than scripts.
-To read the source code of the programs, look into the @file{bin/progname}
directory of Gnuastro's source (@ref{Downloading the source}).
-If you would like to read more about why C was chosen for the programs, please
see @ref{Why C}.}.
-For example with this command (just replace @code{nano} with your favorite
text editor, like @command{emacs} or @command{vim}):
+@item mgii
+@cindex Doublet: MgII
+@cindex MgII doublet
+[2799.5@AA{}] MgII doublet's mean center.
-@example
-$ nano $(which astscript-NAME)
-@end example
+@item mgiiblue
+[2796@AA{}] MgII doublet's bluer line.
-Shell scripting is the same language that you use when typing on the
command-line.
-Therefore shell scripting is much more widely known and used compared to C
(the language of other Gnuastro programs).
-Because Gnuastro's installed scripts do higher-level operations, customizing
these scripts for a special project will be more common than the programs.
+@item ciiired
+[1909@AA{}] CIII doublet's redder line.
-These scripts also accept options and are in many ways similar to the programs
(see @ref{Common options}) with some minor differences:
+@item ciii
+@cindex Doublet: CIII
+@cindex CIII doublet
+[1908@AA{}] CIII doublet's mean center.
-@itemize
-@item
-Currently they don't accept configuration files themselves.
-However, the configuration files of the Gnuastro programs they call are indeed
parsed and used by those programs.
+@item ciiiblue
+[1907@AA{}] CIII doublet's bluer line.
-As a result, they don't have the following options: @option{--checkconfig},
@option{--config}, @option{--lastconfig}, @option{--onlyversion},
@option{--printparams}, @option{--setdirconf} and @option{--setusrconf}.
+@item si_iiired
+[1892@AA{}] SiIII doublet's redder line.
-@item
-They don't directly allocate any memory, so there is no @option{--minmapsize}.
+@item si_iii
+@cindex Doublet: SiIII
+@cindex SiIII doublet
+[1887.5@AA{}] SiIII doublet's mean center.
-@item
-They don't have an independent @option{--usage} option: when called with
@option{--usage}, they just recommend running @option{--help}.
+@item si_iiiblue
+[1883@AA{}] SiIII doublet's bluer line.
-@item
-The output of @option{--help} is not configurable like the programs (see
@ref{--help}).
+@item oiiired-uv
+[1666@AA{}] OIII doublet's redder line in the ultra-violet.
-@item
-@cindex GNU AWK
-@cindex GNU SED
-The scripts will commonly use your installed shell and other basic
command-line tools (for example AWK or SED).
-Different systems have different versions and implementations of these basic
tools (for example GNU/Linux systems use GNU Bash, GNU AWK and GNU SED which
are far more advanced and up to date then the minimalist AWK and SED of most
other systems).
-Therefore, unexpected errors in these tools might come up when you run these
scripts on non-GNU/Linux operating systems.
-If you do confront such strange errors, please submit a bug report so we fix
it as soon as possible (see @ref{Report a bug}).
+@item oiii-uv
+@cindex Doublet: OIII (in UV)
+@cindex OIII doublet in UV
+[1663.5@AA{}] OIII doublet's mean center in the ultra-violet.
-@end itemize
+@item oiiiblue-uv
+[1661@AA{}] OIII doublet's bluer line in the ultra-violet.
-@menu
-* Sort FITS files by night:: Sort many files by date.
-* Generate radial profile:: Radial profile of an object in an image.
-* SAO DS9 region files from table:: Create ds9 region file from a table.
-* PSF construction and subtraction::
-@end menu
+@item heii-uv
+[1640@AA{}] HeII doublet's bluer line in the ultra-violet.
-@node Sort FITS files by night, Generate radial profile, Installed scripts,
Installed scripts
-@section Sort FITS files by night
+@item civred
+[1551@AA{}] CIV doublet's redder line.
-@cindex Calendar
-FITS images usually contain (several) keywords for preserving important dates.
-In particular, for lower-level data, this is usually the observation date and
time (for example, stored in the @code{DATE-OBS} keyword value).
-When analyzing observed datasets, many calibration steps (like the dark, bias
or flat-field), are commonly calculated on a per-observing-night basis.
+@item civ
+@cindex Doublet: CIV
+@cindex CIV doublet
+[1549@AA{}] CIV doublet's mean center.
-However, the FITS standard's date format (@code{YYYY-MM-DDThh:mm:ss.ddd}) is
based on the western (Gregorian) calendar.
-Dates that are stored in this format are complicated for automatic processing:
a night starts in the final hours of one calendar day, and extends to the early
hours of the next calendar day.
-As a result, to identify datasets from one night, we commonly need to search
for two dates.
-However calendar peculiarities can make this identification very difficult.
-For example when an observation is done on the night separating two months
(like the night starting on March 31st and going into April 1st), or two years
(like the night starting on December 31st 2018 and going into January 1st,
2019).
-To account for such situations, it is necessary to keep track of how many days
are in a month, and leap years, etc.
+@item civblue
+[1548@AA{}] CIV doublet's bluer line.
-@cindex Unix epoch time
-@cindex Time, Unix epoch
-@cindex Epoch, Unix time
-Gnuastro's @file{astscript-sort-by-night} script is created to help in such
important scenarios.
-It uses @ref{Fits} to convert the FITS date format into the Unix epoch time
(number of seconds since 00:00:00 of January 1st, 1970), using the
@option{--datetosec} option.
-The Unix epoch time is a single number (integer, if not given in sub-second
precision), enabling easy comparison and sorting of dates after January 1st,
1970.
+@item nv
+[1240@AA{}] NV (four times ionized Sodium).
-You can use this script as a basis for making a much more highly customized
sorting script.
-Here are some examples
+@item lyalpha
+@cindex Lyman-alpha
+[1215.67@AA{}] Lyman-@mymath{\alpha} line.
-@itemize
-@item
-If you need to copy the files, but only need a single extension (not the whole
file), you can add a step just before the making of the symbolic links, or
copies, and change it to only copy a certain extension of the FITS file using
the Fits program's @option{--copy} option, see @ref{HDU information and
manipulation}.
+@item lybeta
+@cindex Lyman-beta
+[1025.7@AA{}] Lyman-@mymath{\beta} line.
-@item
-If you need to classify the files with finer detail (for example the purpose
of the dataset), you can add a step just before the making of the symbolic
links, or copies, to specify a file-name prefix based on other certain keyword
values in the files.
-For example when the FITS files have a keyword to specify if the dataset is a
science, bias, or flat-field image.
-You can read it and to add a @code{sci-}, @code{bias-}, or @code{flat-} to the
created file (after the @option{--prefix}) automatically.
+@item lygamma
+@cindex Lyman-gamma
+[972.54@AA{}] Lyman-@mymath{\gamma} line.
-For example, let's assume the observing mode is stored in the hypothetical
@code{MODE} keyword, which can have three values of @code{BIAS-IMAGE},
@code{SCIENCE-IMAGE} and @code{FLAT-EXP}.
-With the step below, you can generate a mode-prefix, and add it to the
generated link/copy names (just correct the filename and extension of the first
line to the script's variables):
+@item lydelta
+@cindex Lyman-delta
+[949.74@AA{}] Lyman-@mymath{\delta} line.
-@example
-modepref=$(astfits infile.fits -h1 \
- | sed -e"s/'/ /g" \
- | awk '$1=="MODE"@{ \
- if($3=="BIAS-IMAGE") print "bias-"; \
- else if($3=="SCIENCE-IMAGE") print "sci-"; \
- else if($3==FLAT-EXP) print "flat-"; \
- else print $3, "NOT recognized"; exit 1@}')
-@end example
+@item lyepsilon
+@cindex Lyman-epsilon
+[937.80@AA{}] Lyman-@mymath{\epsilon} line.
-@cindex GNU AWK
-@cindex GNU Sed
-Here is a description of it.
-We first use @command{astfits} to print all the keywords in extension @code{1}
of @file{infile.fits}.
-In the FITS standard, string values (that we are assuming here) are placed in
single quotes (@key{'}) which are annoying in this context/use-case.
-Therefore, we pipe the output of @command{astfits} into @command{sed} to
remove all such quotes (substituting them with a blank space).
-The result is then piped to AWK for giving us the final mode-prefix: with
@code{$1=="MODE"}, we ask AWK to only consider the line where the first column
is @code{MODE}.
-There is an equal sign between the key name and value, so the value is the
third column (@code{$3} in AWK).
-We thus use a simple @code{if-else} structure to look into this value and
print our custom prefix based on it.
-The output of AWK is then stored in the @code{modepref} shell variable which
you can add to the link/copy name.
+@item lylimit
+@cindex Lyman limit
+[912@AA{}] Lyman limit.
-With the solution above, the increment of the file counter for each night will
be independent of the mode.
-If you want the counter to be mode-dependent, you can add a different counter
for each mode and use that counter instead of the generic counter for each
night (based on the value of @code{modepref}).
-But we'll leave the implementation of this step to you as an exercise.
+@end table
-@end itemize
+@end table
-@menu
-* Invoking astscript-sort-by-night:: Inputs and outputs to this script.
-@end menu
-@node Invoking astscript-sort-by-night, , Sort FITS files by night, Sort FITS
files by night
-@subsection Invoking astscript-sort-by-night
-This installed script will read a FITS date formatted value from the given
keyword, and classify the input FITS files into individual nights.
-For more on installed scripts please see (see @ref{Installed scripts}).
-This script can be used with the following general template:
+@node CosmicCalculator basic cosmology calculations, CosmicCalculator spectral
line calculations, CosmicCalculator input options, Invoking astcosmiccal
+@subsubsection CosmicCalculator basic cosmology calculations
+By default, when no specific calculations are requested, CosmicCalculator will
print a complete set of all its calculators (one line for each calculation, see
@ref{Invoking astcosmiccal}).
+The full list of calculations can be useful when you don't want any specific
value, but just a general view.
+In other contexts (for example in a batch script or during a discussion), you
know exactly what you want and don't want to be distracted by all the extra
information.
+
+You can use any number of the options described below in any order.
+When any of these options are requested, CosmicCalculator's output will just
be a single line with a single space between the (possibly) multiple values.
+In the example below, only the tangential distance along one arc-second (in
kpc), absolute magnitude conversion, and age of the universe at redshift 2 are
printed (recall that you can merge short options together, see @ref{Options}).
@example
-$ astscript-sort-by-night [OPTION...] FITS-files
+$ astcosmiccal -z2 -sag
+8.585046 44.819248 3.289979
@end example
-@noindent
-One line examples:
+Here is one example of using this feature in scripts: by adding the following
two lines in a script to keep/use the comoving volume with varying redshifts:
@example
-## Use the DATE-OBS keyword
-$ astscript-sort-by-night --key=DATE-OBS /path/to/data/*.fits
-
-## Make links to the input files with the `img-' prefix
-$ astscript-sort-by-night --link --prefix=img- /path/to/data/*.fits
+z=3.12
+vol=$(astcosmiccal --redshift=$z --volume)
@end example
-This script will look into a HDU/extension (@option{--hdu}) for a keyword
(@option{--key}) in the given FITS files and interpret the value as a date.
-The inputs will be separated by "night"s (11:00a.m to next day's 10:59:59a.m,
spanning two calendar days, exact hour can be set with @option{--hour}).
+@cindex GNU Grep
+@noindent
+In a script, this operation might be necessary for a large number of objects
(several of galaxies in a catalog for example).
+So the fact that all the other default calculations are ignored will also help
you get to your result faster.
-The default output is a list of all the input files along with the following
two columns: night number and file number in that night (sorted by time).
-With @option{--link} a symbolic link will be made (one for each input) that
contains the night number, and number of file in that night (sorted by time),
see the description of @option{--link} for more.
-When @option{--copy} is used instead of a link, a copy of the inputs will be
made instead of symbolic link.
+If you are indeed dealing with many (for example thousands) of redshifts,
using CosmicCalculator is not the best/fastest solution.
+Because it has to go through all the configuration files and preparations for
each invocation.
+To get the best efficiency (least overhead), we recommend using Gnuastro's
cosmology library (see @ref{Cosmology library}).
+CosmicCalculator also calls the library functions defined there for its
calculations, so you get the same result with no overhead.
+Gnuastro also has libraries for easily reading tables into a C program, see
@ref{Table input output}.
+Afterwards, you can easily build and run your C program for the particular
processing with @ref{BuildProgram}.
-Below you can see one example where all the @file{target-*.fits} files in the
@file{data} directory should be separated by observing night according to the
@code{DATE-OBS} keyword value in their second extension (number @code{1},
recall that HDU counting starts from 0).
-You can see the output after the @code{ls} command.
+If you just want to inspect the value of a variable visually, the description
(which comes with units) might be more useful.
+In such cases, the following command might be better.
+The other calculations will also be done, but they are so fast that you will
not notice on modern computers (the time it takes your eye to focus on the
result is usually longer than the processing: a fraction of a second).
@example
-$ astscript-sort-by-night -pimg- -h1 -kDATE-OBS data/target-*.fits
-$ ls
-img-n1-1.fits img-n1-2.fits img-n2-1.fits ...
+$ astcosmiccal --redshift=0.832 | grep volume
@end example
-The outputs can be placed in a different (already existing) directory by
including that directory's name in the @option{--prefix} value, for example
@option{--prefix=sorted/img-} will put them all under the @file{sorted}
directory.
-
-This script can be configured like all Gnuastro's programs (through
command-line options, see @ref{Common options}), with some minor differences
that are described in @ref{Installed scripts}.
-The particular options to this script are listed below:
+The full list of CosmicCalculator's specific calculations is present below in
two groups: basic cosmology calculations and those related to spectral lines.
+In case you have forgot the units, you can use the @option{--help} option
which has the units along with a short description.
@table @option
-@item -h STR
-@itemx --hdu=STR
-The HDU/extension to use in all the given FITS files.
-All of the given FITS files must have this extension.
-@item -k STR
-@itemx --key=STR
-The keyword name that contains the FITS date format to classify/sort by.
+@item -e
+@itemx --usedredshift
+The redshift that was used in this run.
+In many cases this is the main input parameter to CosmicCalculator, but it is
useful in others.
+For example in combination with @option{--obsline} (where you give an observed
and rest-frame wavelength and would like to know the redshift) or with
@option{--velocity} (where you specify the velocity instead of redshift).
+Another example is when you run CosmicCalculator in a loop, while changing the
redshift and you want to keep the redshift value with the resulting calculation.
-@item -H FLT
-@itemx --hour=FLT
-The hour that defines the next ``night''.
-By default, all times before 11:00a.m are considered to belong to the previous
calendar night.
-If a sub-hour value is necessary, it should be given in units of hours, for
example @option{--hour=9.5} corresponds to 9:30a.m.
+@item -Y
+@itemx --usedvelocity
+The velocity (in km/s) that was used in this run.
+The conversion from redshift will be done with the more general and accurate
relativistic equation of @mymath{1+z=\sqrt{(c+v)/(c-v)}}, not the simplified
@mymath{z\approx v/c}.
-@cartouche
-@noindent
-@cindex Time zone
-@cindex UTC (Universal time coordinate)
-@cindex Universal time coordinate (UTC)
-@strong{Dealing with time zones:}
-The time that is recorded in @option{--key} may be in UTC (Universal Time
Coordinate).
-However, the organization of the images taken during the night depends on the
local time.
-It is possible to take this into account by setting the @option{--hour} option
to the local time in UTC.
+@item -G
+@itemx --agenow
+The current age of the universe (given the input parameters) in Ga (Giga
annum, or billion years).
-For example, consider a set of images taken in Auckland (New Zealand, UTC+12)
during different nights.
-If you want to classify these images by night, you have to know at which time
(in UTC time) the Sun rises (or any other separator/definition of a different
night).
-For example if your observing night finishes before 9:00a.m in Auckland, you
can use @option{--hour=21}.
-Because in Auckland the local time of 9:00 corresponds to 21:00 UTC.
-@end cartouche
+@item -C
+@itemx --criticaldensitynow
+The current critical density (given the input parameters) in grams per
centimeter-cube (@mymath{g/cm^3}).
-@item -l
-@itemx --link
-Create a symbolic link for each input FITS file.
-This option cannot be used with @option{--copy}.
-The link will have a standard name in the following format (variable parts are
written in @code{CAPITAL} letters and described after it):
+@item -d
+@itemx --properdistance
+The proper distance (at current time) to object at the given redshift in
Megaparsecs (Mpc).
+See @ref{Distance on a 2D curved space} for a description of the proper
distance.
-@example
-PnN-I.fits
-@end example
+@item -A
+@itemx --angulardimdist
+The angular diameter distance to object at given redshift in Megaparsecs (Mpc).
-@table @code
-@item P
-This is the value given to @option{--prefix}.
-By default, its value is @code{./} (to store the links in the directory this
script was run in).
-See the description of @code{--prefix} for more.
-@item N
-This is the night-counter: starting from 1.
-@code{N} is just incremented by 1 for the next night, no matter how many
nights (without any dataset) there are between two subsequent observing nights
(its just an identifier for each night which you can easily map to different
calendar nights).
-@item I
-File counter in that night, sorted by time.
-@end table
+@item -s
+@itemx --arcsectandist
+The tangential distance covered by 1 arc-seconds at the given redshift in
kiloparsecs (Kpc).
+This can be useful when trying to estimate the resolution or pixel scale of an
instrument (usually in units of arc-seconds) at a given redshift.
-@item -c
-@itemx --copy
-Make a copy of each input FITS file with the standard naming convention
described in @option{--link}.
-With this option, instead of making a link, a copy is made.
-This option cannot be used with @option{--link}.
+@item -L
+@itemx --luminositydist
+The luminosity distance to object at given redshift in Megaparsecs (Mpc).
-@item -p STR
-@itemx --prefix=STR
-Prefix to append before the night-identifier of each newly created link or
copy.
-This option is thus only relevant with the @option{--copy} or @option{--link}
options.
-See the description of @option{--link} for how its used.
-For example, with @option{--prefix=img-}, all the created file names in the
current directory will start with @code{img-}, making outputs like
@file{img-n1-1.fits} or @file{img-n3-42.fits}.
+@item -u
+@itemx --distancemodulus
+The distance modulus at given redshift.
-@option{--prefix} can also be used to store the links/copies in another
directory relative to the directory this script is being run (it must already
exist).
-For example @code{--prefix=/path/to/processing/img-} will put all the
links/copies in the @file{/path/to/processing} directory, and the files (in
that directory) will all start with @file{img-}.
+@item -a
+@itemx --absmagconv
+The conversion factor (addition) to absolute magnitude.
+Note that this is practically the distance modulus added with
@mymath{-2.5\log{(1+z)}} for the desired redshift based on the input parameters.
+Once the apparent magnitude and redshift of an object is known, this value may
be added with the apparent magnitude to give the object's absolute magnitude.
+
+@item -g
+@itemx --age
+Age of the universe at given redshift in Ga (Giga annum, or billion years).
+
+@item -b
+@itemx --lookbacktime
+The look-back time to given redshift in Ga (Giga annum, or billion years).
+The look-back time at a given redshift is defined as the current age of the
universe (@option{--agenow}) subtracted by the age of the universe at the given
redshift.
+
+@item -c
+@itemx --criticaldensity
+The critical density at given redshift in grams per centimeter-cube
(@mymath{g/cm^3}).
+
+@item -v
+@itemx --onlyvolume
+The comoving volume in Megaparsecs cube (Mpc@mymath{^3}) until the desired
redshift based on the input parameters.
-@item --stdintimeout=INT
-Number of micro-seconds to wait for standard input within this script.
-This does not correspond to general inputs into the script, inputs to the
script should always be given as a file.
-However, within the script, pipes are often used to pass the output of one
program to another.
-The value given to this option will be passed to those internal pipes.
-When running this script, if you confront an error, saying ``No input!'', you
should be able to fix it by giving a larger number to this option (the default
value is 10000000 micro-seconds or 10 seconds).
@end table
+@node CosmicCalculator spectral line calculations, , CosmicCalculator basic
cosmology calculations, Invoking astcosmiccal
+@subsubsection CosmicCalculator spectral line calculations
+
+@cindex Rest frame wavelength
+At different redshifts, observed spectral lines are shifted compared to their
rest frame wavelengths with this simple relation:
@mymath{\lambda_{obs}=\lambda_{rest}(1+z)}.
+Although this relation is very simple and can be done for one line in the head
(or a simple calculator!), it slowly becomes tiring when dealing with a lot of
lines or redshifts, or some precision is necessary.
+The options in this section are thus provided to greatly simplify usage of
this simple equation, and also helping by storing a list of pre-defined
spectral line wavelengths.
+
+For example if you want to know the wavelength of the @mymath{H\alpha} line
(at 6562.8 Angstroms in rest frame), when @mymath{Ly\alpha} is at 8000
Angstroms, you can call CosmicCalculator like the first example below.
+And if you want the wavelength of all pre-defined spectral lines at this
redshift, you can use the second command.
+
+@example
+$ astcosmiccal --obsline=lyalpha,8000 --lineatz=halpha
+$ astcosmiccal --obsline=lyalpha,8000 --listlinesatz
+@end example
+Bellow you can see the printed/output calculations of CosmicCalculator that
are related to spectral lines.
+Note that @option{--obsline} is an input parameter, so its discussed (with the
full list of known lines) in @ref{CosmicCalculator input options}.
+@table @option
+@item --listlines
+List the pre-defined rest frame spectral line wavelengths and their names on
standard output, then abort CosmicCalculator.
+When this option is given, other operations on the command-line will be
ignored.
+This is convenient when you forget the specific name of the spectral line used
within Gnuastro, or when you forget the exact wavelength of a certain line.
+These names can be used with the options that deal with spectral lines, for
example @option{--obsline} and @option{--lineatz} (@ref{CosmicCalculator basic
cosmology calculations}).
+The format of the output list is a two-column table, with Gnuastro's text
table format (see @ref{Gnuastro text table format}).
+Therefore, if you are only looking for lines in a specific range, you can pipe
the output into Gnuastro's table program and use its @option{--range} option on
the @code{wavelength} (first) column.
+For example, if you only want to see the lines between 4000 and 6000
Angstroms, you can run this command:
+@example
+$ astcosmiccal --listlines \
+ | asttable --range=wavelength,4000,6000
+@end example
+@noindent
+And if you want to use the list later and have it as a table in a file, you
can easily add the @option{--output} (or @option{-o}) option to the
@command{asttable} command, and specify the filename, for example
@option{--output=lines.fits} or @option{--output=lines.txt}.
+@item --listlinesatz
+Similar to @option{--listlines} (above), but the printed wavelength is not in
the rest frame, but redshifted to the given redshift.
+Recall that the redshift can be specified by @option{--redshift} directly or
by @option{--obsline}, see @ref{CosmicCalculator input options}.
+@item -i STR/FLT
+@itemx --lineatz=STR/FLT
+The wavelength of the specified line at the redshift given to CosmicCalculator.
+The line can be specified either by its name or directly as a number (its
wavelength).
+To get the list of pre-defined names for the lines and their wavelength, you
can use the @option{--listlines} option, see @ref{CosmicCalculator input
options}.
+In the former case (when a name is given), the returned number is in units of
Angstroms.
+In the latter (when a number is given), the returned value is the same units
of the input number (assuming its a wavelength).
+@end table
-@node Generate radial profile, SAO DS9 region files from table, Sort FITS
files by night, Installed scripts
-@section Generate radial profile
-@cindex Radial profile
-@cindex Profile, profile
-The 1 dimensional radial profile of an object is an important parameter in
many aspects of astronomical image processing.
-For example, you want to study how the light of a galaxy is distributed as a
function of the radial distance from the center.
-In other cases, the radial profile of a star can show the PSF (see @ref{PSF}).
-Gnuastro's @file{astscript-radial-profile} script is created to obtain such
radial profiles for one object within an image.
-This script uses @ref{MakeProfiles} to generate elliptical apertures with the
values equal to the distance from the center of the object and
@ref{MakeCatalog} for measuring the values over the apertures.
-@menu
-* Invoking astscript-radial-profile:: How to call astscript-radial-profile
-@end menu
+@node Installed scripts, Library, High-level calculations, Top
+@chapter Installed scripts
-@node Invoking astscript-radial-profile, , Generate radial profile, Generate
radial profile
-@subsection Invoking astscript-radial-profile
+Gnuastro's programs (introduced in previous chapters) are designed to be
highly modular and thus contain lower-level operations on the data.
+However, in many contexts, certain higher-level are also shared between many
contexts.
+For example a sequence of calls to multiple Gnuastro programs, or a special
way of running a program and treating the output.
+To facilitate such higher-level data analysis, Gnuastro also installs some
scripts on your system with the (@code{astscript-}) prefix (in contrast to the
other programs that only have the @code{ast} prefix).
-This installed script will measure the radial profile of an object within an
image.
-For more on installed scripts please see (see @ref{Installed scripts}).
-This script can be used with the following general template:
+@cindex GNU Bash
+@cindex Portable shell
+@cindex Shell, portable
+Like all of Gnuastro's source code, these scripts are also heavily commented.
+They are written in portable shell scripts (command-line environments), which
doesn't need compilation.
+Therefore, if you open the installed scripts in a text editor, you can
actually read them@footnote{Gnuastro's installed programs (those only starting
with @code{ast}) aren't human-readable.
+They are written in C and need to be compiled before execution.
+Compilation optimizes the steps into the low-level hardware CPU
instructions/language to improve efficiency.
+Because compiled programs don't need an interpreter like Bash on every run,
they are much faster and more independent than scripts.
+To read the source code of the programs, look into the @file{bin/progname}
directory of Gnuastro's source (@ref{Downloading the source}).
+If you would like to read more about why C was chosen for the programs, please
see @ref{Why C}.}.
+For example with this command (just replace @code{nano} with your favorite
text editor, like @command{emacs} or @command{vim}):
@example
-$ astscript-radial-profile [OPTION...] FITS-file
+$ nano $(which astscript-NAME)
@end example
-@noindent
-Examples:
-
-@example
-## Generate the radial profile with default options (assuming the
-## object is in the center of the image, and using the mean).
-$ astscript-radial-profile image.fits
-
-## Generate the radial profile centered at x=44 and y=37 (in pixels),
-## up to a radial distance of 19 pixels, use the mean value.
-$ astscript-radial-profile image.fits --center=44,37 --rmax=19
-
-## Generate the radial profile centered at x=44 and y=37 (in pixels),
-## up to a radial distance of 100 pixels, compute sigma clipped
-## mean and standard deviation (sigclip-mean and sigclip-std) using
-## 5 sigma and 0.1 tolerance (default is 3 sigma and 0.2 tolerance).
-$ astscript-radial-profile image.fits --center=44,37 --rmax=100 \
- --sigmaclip=5,0.1 \
- --measure=sigclip-mean,sigclip-std
-
-## Generate the radial profile centered at RA=20.53751695,
-## DEC=0.9454292263, up to a radial distance of 88 pixels,
-## axis ratio equal to 0.32, and position angle of 148 deg.
-## Name the output table as `radial-profile.fits'
-$ astscript-radial-profile image.fits --mode=wcs \
- --center=20.53751695,0.9454292263 \
- --rmax=88 --axisratio=0.32 \
- --positionangle=148 -oradial-profile.fits
+Shell scripting is the same language that you use when typing on the
command-line.
+Therefore shell scripting is much more widely known and used compared to C
(the language of other Gnuastro programs).
+Because Gnuastro's installed scripts do higher-level operations, customizing
these scripts for a special project will be more common than the programs.
-## Generate the radial profile centered at RA=40.062675270971,
-## DEC=-8.1511992735126, up to a radial distance of 20 pixels,
-## and calculate the SNR using the INPUT-NO-SKY and SKY-STD
-## extensions of the NoiseChisel output file.
-$ astscript-radial-profile image_detected.fits -hINPUT-NO-SKY \
- --mode=wcs --measure=sn \
- --center=40.062675270971,-8.1511992735126 \
- --rmax=20 --stdhdu=SKY_STD
+These scripts also accept options and are in many ways similar to the programs
(see @ref{Common options}) with some minor differences:
-## Generate the radial profile centered at RA=40.062675270971,
-## DEC=-8.1511992735126, up to a radial distance of 20 pixels,
-## and compute the SNR with a fixed value for std, std=10.
-$ astscript-radial-profile image.fits -h1 --mode=wcs --rmax=20 \
- --center=40.062675270971,-8.1511992735126 \
- --measure=sn --instd=10
+@itemize
+@item
+Currently they don't accept configuration files themselves.
+However, the configuration files of the Gnuastro programs they call are indeed
parsed and used by those programs.
-## Generate the radial profile centered at X=1201, Y=1201 pixels, up
-## to a radial distance of 20 pixels and compute the median and the
-## SNR using the first extension of sky-std.fits as the dataset for std
-## values.
-$ astscript-radial-profile image.fits -h1 --mode=img --rmax=20 \
- --center=1201,1201 --measure=median,sn \
- --instd=sky-std.fits
-@end example
+As a result, they don't have the following options: @option{--checkconfig},
@option{--config}, @option{--lastconfig}, @option{--onlyversion},
@option{--printparams}, @option{--setdirconf} and @option{--setusrconf}.
-This installed script will read a FITS image and will use it as the basis for
constructing the radial profile.
-The output radial profile is a table (FITS or plain-text) containing the
radial distance from the center in the first row and the specified measurements
in the other columns (mean, median, sigclip-mean, sigclip-median, etc.).
+@item
+They don't directly allocate any memory, so there is no @option{--minmapsize}.
-To measure the radial profile, this script needs to generate temporary files.
-All these temporary files will be created within the directory given to the
@option{--tmpdir} option.
-When @option{--tmpdir} is not called, a temporary directory (with a name based
on the inputs) will be created in the running directory.
-If the directory doesn't exist at run-time, this script will create it.
-After the output is created, this script will delete the directory by default,
unless you call the @option{--keeptmp} option.
+@item
+They don't have an independent @option{--usage} option: when called with
@option{--usage}, they just recommend running @option{--help}.
-With the default options, the script will generate a circular radial profile
using the mean value and centered at the center of the image.
-In order to have more flexibility, several options are available to configure
for the desired radial profile.
-In this sense, you can change the center position, the maximum radius, the
axis ratio and the position angle (elliptical apertures are considered), the
operator for obtaining the profiles, and others (described below).
+@item
+The output of @option{--help} is not configurable like the programs (see
@ref{--help}).
-@cartouche
-@noindent
-@strong{Debug your profile:} to debug your results, especially close to the
center of your object, you can see the radial distance associated to every
pixel in your input.
-To do this, use @option{--keeptmp} to keep the temporary files, and compare
@file{crop.fits} (crop of your input image centered on your desired coordinate)
with @file{apertures.fits} (radial distance of each pixel).
-@end cartouche
+@item
+@cindex GNU AWK
+@cindex GNU SED
+The scripts will commonly use your installed shell and other basic
command-line tools (for example AWK or SED).
+Different systems have different versions and implementations of these basic
tools (for example GNU/Linux systems use GNU Bash, GNU AWK and GNU SED which
are far more advanced and up to date then the minimalist AWK and SED of most
other systems).
+Therefore, unexpected errors in these tools might come up when you run these
scripts on non-GNU/Linux operating systems.
+If you do confront such strange errors, please submit a bug report so we fix
it as soon as possible (see @ref{Report a bug}).
-@cartouche
-@noindent
-@strong{Finding properties of your elliptical target: } you want to measure
the radial profile of a galaxy, but don't know its exact location, position
angle or axis ratio.
-To obtain these values, you can use @ref{NoiseChisel} to detect signal in the
image, feed it to @ref{Segment} to do basic segmentation, then use
@ref{MakeCatalog} to measure the center (@option{--x} and @option{--y} in
MakeCatalog), axis ratio (@option{--axisratio}) and position angle
(@option{--positionangle}).
-@end cartouche
+@end itemize
-@cartouche
-@noindent
-@strong{Masking other sources:} The image of an astronomical object will
usually have many other sources with your main target.
-A crude solution is to use sigma-clipped measurements for the profile.
-However, sigma-clipped measurements can easily be biased when the number of
sources at each radial distance increases at larger distances.
-Therefore a robust solution is to mask all other detections within the image.
-You can use @ref{NoiseChisel} and @ref{Segment} to detect and segment the
sources, then set all pixels that don't belong to your target to blank using
@ref{Arithmetic} (in particular, its @code{where} operator).
-@end cartouche
+@menu
+* Sort FITS files by night:: Sort many files by date.
+* Generate radial profile:: Radial profile of an object in an image.
+* SAO DS9 region files from table:: Create ds9 region file from a table.
+* PSF construction and subtraction::
+@end menu
-@table @option
-@item -h STR
-@itemx --hdu=STR
-The HDU/extension of the input image to use.
+@node Sort FITS files by night, Generate radial profile, Installed scripts,
Installed scripts
+@section Sort FITS files by night
-@item -o STR
-@itemx --output=STR
-Filename of measured radial profile.
-It can be either a FITS table, or plain-text table (determined from your given
file name suffix).
+@cindex Calendar
+FITS images usually contain (several) keywords for preserving important dates.
+In particular, for lower-level data, this is usually the observation date and
time (for example, stored in the @code{DATE-OBS} keyword value).
+When analyzing observed datasets, many calibration steps (like the dark, bias
or flat-field), are commonly calculated on a per-observing-night basis.
-@item -c FLT[,FLT[,...]]
-@itemx --center=FLT[,FLT[,...]]
-The central position of the radial profile.
-This option is used for placing the center of the profiles.
-This parameter is used in @ref{Crop} to center and crop the region.
-The positions along each dimension must be separated by a comma (@key{,}) and
fractions are also acceptable.
-The number of values given to this option must be the same as the dimensions
of the input dataset.
-The units of the coordinates are read based on the value to the
@option{--mode} option, see below.
+However, the FITS standard's date format (@code{YYYY-MM-DDThh:mm:ss.ddd}) is
based on the western (Gregorian) calendar.
+Dates that are stored in this format are complicated for automatic processing:
a night starts in the final hours of one calendar day, and extends to the early
hours of the next calendar day.
+As a result, to identify datasets from one night, we commonly need to search
for two dates.
+However calendar peculiarities can make this identification very difficult.
+For example when an observation is done on the night separating two months
(like the night starting on March 31st and going into April 1st), or two years
(like the night starting on December 31st 2018 and going into January 1st,
2019).
+To account for such situations, it is necessary to keep track of how many days
are in a month, and leap years, etc.
-@item -O STR
-@itemx --mode=STR
-Interpret the center position of the object (values given to
@option{--center}) in image or WCS coordinates.
-This option thus accepts only two values: @option{img} or @option{wcs}.
-By default, it is @option{--mode=img}.
+@cindex Unix epoch time
+@cindex Time, Unix epoch
+@cindex Epoch, Unix time
+Gnuastro's @file{astscript-sort-by-night} script is created to help in such
important scenarios.
+It uses @ref{Fits} to convert the FITS date format into the Unix epoch time
(number of seconds since 00:00:00 of January 1st, 1970), using the
@option{--datetosec} option.
+The Unix epoch time is a single number (integer, if not given in sub-second
precision), enabling easy comparison and sorting of dates after January 1st,
1970.
-@item -R FLT
-@itemx --rmax=FLT
-Maximum radius for the radial profile (in pixels).
-By default, the radial profile will be computed up to a radial distance equal
to the maximum radius that fits into the image (assuming circular shape).
+You can use this script as a basis for making a much more highly customized
sorting script.
+Here are some examples
-@item -Q FLT
-@itemx --axisratio=FLT
-The axis ratio of the apertures (minor axis divided by the major axis in a 2D
ellipse).
-By default (when this option isn't given), the radial profile will be circular
(axis ratio of 1).
-This parameter is used as the option @option{--qcol} in the generation of the
apertures with @command{astmkprof}.
+@itemize
+@item
+If you need to copy the files, but only need a single extension (not the whole
file), you can add a step just before the making of the symbolic links, or
copies, and change it to only copy a certain extension of the FITS file using
the Fits program's @option{--copy} option, see @ref{HDU information and
manipulation}.
-@item -p FLT
-@itemx --positionangle=FLT
-The position angle (in degrees) of the profiles relative to the first FITS
axis (horizontal when viewed in SAO DS9).
-By default, it is @option{--positionangle=0}, which means that the semi-major
axis of the profiles will be parallel to the first FITS axis.
+@item
+If you need to classify the files with finer detail (for example the purpose
of the dataset), you can add a step just before the making of the symbolic
links, or copies, to specify a file-name prefix based on other certain keyword
values in the files.
+For example when the FITS files have a keyword to specify if the dataset is a
science, bias, or flat-field image.
+You can read it and to add a @code{sci-}, @code{bias-}, or @code{flat-} to the
created file (after the @option{--prefix}) automatically.
-@item -m STR
-@itemx --measure=STR
-The operator for measuring the values over each radial distance.
-The values given to this option will be directly passed to @ref{MakeCatalog}.
-As a consequence, all MakeCatalog measurements like the magnitude, magnitude
error, median, mean, signal-to-noise ratio (S/N), std, surface brightness,
sigclip-mean, sigclip-number, etc. can be used here.
-For a full list of MakeCatalog's measurements, please run
@command{astmkcatalog --help}.
-Multiple values can be given to this option, each separated by a comma.
-This option can also be called multiple times.
+For example, let's assume the observing mode is stored in the hypothetical
@code{MODE} keyword, which can have three values of @code{BIAS-IMAGE},
@code{SCIENCE-IMAGE} and @code{FLAT-EXP}.
+With the step below, you can generate a mode-prefix, and add it to the
generated link/copy names (just correct the filename and extension of the first
line to the script's variables):
-Some measurements by MakeCatalog require a per-pixel sky standard deviation
(for example magnitude error or S/N).
-Therefore when asking for such measurements, use the @option{--instd} option
(described below) to specify the per-pixel sky standard deviation over each
pixel.
-For other measurements like the magnitude or surface brightness, MakeCatalog
will need a Zeropoint, which you can set with the @option{--zeropoint} option.
+@example
+modepref=$(astfits infile.fits -h1 \
+ | sed -e"s/'/ /g" \
+ | awk '$1=="MODE"@{ \
+ if($3=="BIAS-IMAGE") print "bias-"; \
+ else if($3=="SCIENCE-IMAGE") print "sci-"; \
+ else if($3==FLAT-EXP) print "flat-"; \
+ else print $3, "NOT recognized"; exit 1@}')
+@end example
-For example, by setting @option{--measure=mean,sigclip-mean --measure=median},
the mean, sigma-clipped mean and median values will be computed.
-The output radial profile will have 4 columns in this order: radial distance,
mean, sigma-clipped and median.
-By default (when this option isn't given), the mean of all pixels at each
radial position will be computed.
+@cindex GNU AWK
+@cindex GNU Sed
+Here is a description of it.
+We first use @command{astfits} to print all the keywords in extension @code{1}
of @file{infile.fits}.
+In the FITS standard, string values (that we are assuming here) are placed in
single quotes (@key{'}) which are annoying in this context/use-case.
+Therefore, we pipe the output of @command{astfits} into @command{sed} to
remove all such quotes (substituting them with a blank space).
+The result is then piped to AWK for giving us the final mode-prefix: with
@code{$1=="MODE"}, we ask AWK to only consider the line where the first column
is @code{MODE}.
+There is an equal sign between the key name and value, so the value is the
third column (@code{$3} in AWK).
+We thus use a simple @code{if-else} structure to look into this value and
print our custom prefix based on it.
+The output of AWK is then stored in the @code{modepref} shell variable which
you can add to the link/copy name.
-@item -s FLT,FLT
-@itemx --sigmaclip=FLT,FLT
-Sigma clipping parameters: only relevant if sigma-clipping operators are
requested by @option{--measure}.
-For more on sigma-clipping, see @ref{Sigma clipping}.
-If given, the value to this option is directly passed to the
@option{--sigmaclip} option of @ref{MakeCatalog}, see @ref{MakeCatalog inputs
and basic settings}.
-By default (when this option isn't given), the default values within
MakeCatalog will be used.
-To see the default value of this option in MakeCatalog, you can run this
command:
+With the solution above, the increment of the file counter for each night will
be independent of the mode.
+If you want the counter to be mode-dependent, you can add a different counter
for each mode and use that counter instead of the generic counter for each
night (based on the value of @code{modepref}).
+But we'll leave the implementation of this step to you as an exercise.
-@example
-$ astmkcatalog -P | grep " sigmaclip "
-@end example
+@end itemize
-@item -z FLT
-@itemx --zeropoint=FLT
-The Zeropoint of the input dataset.
-This is necessary when you request measurements like Magnitude, or Surface
brightness.
+@menu
+* Invoking astscript-sort-by-night:: Inputs and outputs to this script.
+@end menu
-@item -v INT
-@itemx --oversample=INT
-Oversample the input dataset to the fraction given to this option.
-Therefore if you set @option{--rmax=20} for example and
@option{--oversample=5}, your output will have 100 rows (without
@option{--oversample} it will only have 20 rows).
-Unless the object is heavily undersampled (the pixels are larger than the
actual object), this method provides a much more accurate result and there are
sufficient number of pixels to get the profile accurately.
+@node Invoking astscript-sort-by-night, , Sort FITS files by night, Sort FITS
files by night
+@subsection Invoking astscript-sort-by-night
-@item -u INT
-@itemx --undersample=INT
-Undersample the input dataset by the number given to this option.
-This option is for considering larger apertures than the original pixel size
(aperture size is equal to 1 pixel).
-For example, if a radial profile computed by default has 100 different radii
(apertures of 1 pixel width), by considering @option{--undersample=2} the
radial profile will be computed over apertures of 2 pixels, so the final radial
profile will have 50 different radii.
-This option is good to measure over a larger number of pixels to improve the
measurement.
+This installed script will read a FITS date formatted value from the given
keyword, and classify the input FITS files into individual nights.
+For more on installed scripts please see (see @ref{Installed scripts}).
+This script can be used with the following general template:
-@item -i FLT/STR
-@itemx --instd=FLT/STR
-Sky standard deviation as a single number (FLT) or as the filename (STR)
containing the image with the std value for each pixel (the HDU within the file
should be given to the @option{--stdhdu} option mentioned below).
-This is only necessary when the requested measurement (value given to
@option{--measure}) by MakeCatalog needs the Standard deviation (for example
the signal-to-noise ratio or magnitude error).
-If your measurements don't require a standard deviation, it is best to ignore
this option (because it will slow down the script).
+@example
+$ astscript-sort-by-night [OPTION...] FITS-files
+@end example
-@item -d INT/STR
-@itemx --stdhdu=INT/STR
-HDU/extension of the sky standard deviation image specified with
@option{--instd}.
+@noindent
+One line examples:
-@item -t STR
-@itemx --tmpdir=STR
-Several intermediate files are necessary to obtain the radial profile.
-All of these temporal files are saved into a temporal directory.
-With this option, you can directly specify this directory.
-By default (when this option isn't called), it will be built in the running
directory and given an input-based name.
-If the directory doesn't exist at run-time, this script will create it.
-Once the radial profile has been obtained, this directory is removed.
-You can disable the deletion of the temporary directory with the
@option{--keeptmp} option.
+@example
+## Use the DATE-OBS keyword
+$ astscript-sort-by-night --key=DATE-OBS /path/to/data/*.fits
-@item -k
-@itemx --keeptmp
-Don't delete the temporary directory (see description of @option{--tmpdir}
above).
-This option is useful for debugging.
-For example, to check that the profiles generated for obtaining the radial
profile have the desired center, shape and orientation.
-@end table
+## Make links to the input files with the `img-' prefix
+$ astscript-sort-by-night --link --prefix=img- /path/to/data/*.fits
+@end example
+This script will look into a HDU/extension (@option{--hdu}) for a keyword
(@option{--key}) in the given FITS files and interpret the value as a date.
+The inputs will be separated by "night"s (11:00a.m to next day's 10:59:59a.m,
spanning two calendar days, exact hour can be set with @option{--hour}).
+The default output is a list of all the input files along with the following
two columns: night number and file number in that night (sorted by time).
+With @option{--link} a symbolic link will be made (one for each input) that
contains the night number, and number of file in that night (sorted by time),
see the description of @option{--link} for more.
+When @option{--copy} is used instead of a link, a copy of the inputs will be
made instead of symbolic link.
+Below you can see one example where all the @file{target-*.fits} files in the
@file{data} directory should be separated by observing night according to the
@code{DATE-OBS} keyword value in their second extension (number @code{1},
recall that HDU counting starts from 0).
+You can see the output after the @code{ls} command.
+@example
+$ astscript-sort-by-night -pimg- -h1 -kDATE-OBS data/target-*.fits
+$ ls
+img-n1-1.fits img-n1-2.fits img-n2-1.fits ...
+@end example
+The outputs can be placed in a different (already existing) directory by
including that directory's name in the @option{--prefix} value, for example
@option{--prefix=sorted/img-} will put them all under the @file{sorted}
directory.
+This script can be configured like all Gnuastro's programs (through
command-line options, see @ref{Common options}), with some minor differences
that are described in @ref{Installed scripts}.
+The particular options to this script are listed below:
+@table @option
+@item -h STR
+@itemx --hdu=STR
+The HDU/extension to use in all the given FITS files.
+All of the given FITS files must have this extension.
+@item -k STR
+@itemx --key=STR
+The keyword name that contains the FITS date format to classify/sort by.
+@item -H FLT
+@itemx --hour=FLT
+The hour that defines the next ``night''.
+By default, all times before 11:00a.m are considered to belong to the previous
calendar night.
+If a sub-hour value is necessary, it should be given in units of hours, for
example @option{--hour=9.5} corresponds to 9:30a.m.
+@cartouche
+@noindent
+@cindex Time zone
+@cindex UTC (Universal time coordinate)
+@cindex Universal time coordinate (UTC)
+@strong{Dealing with time zones:}
+The time that is recorded in @option{--key} may be in UTC (Universal Time
Coordinate).
+However, the organization of the images taken during the night depends on the
local time.
+It is possible to take this into account by setting the @option{--hour} option
to the local time in UTC.
+For example, consider a set of images taken in Auckland (New Zealand, UTC+12)
during different nights.
+If you want to classify these images by night, you have to know at which time
(in UTC time) the Sun rises (or any other separator/definition of a different
night).
+For example if your observing night finishes before 9:00a.m in Auckland, you
can use @option{--hour=21}.
+Because in Auckland the local time of 9:00 corresponds to 21:00 UTC.
+@end cartouche
+@item -l
+@itemx --link
+Create a symbolic link for each input FITS file.
+This option cannot be used with @option{--copy}.
+The link will have a standard name in the following format (variable parts are
written in @code{CAPITAL} letters and described after it):
+@example
+PnN-I.fits
+@end example
+@table @code
+@item P
+This is the value given to @option{--prefix}.
+By default, its value is @code{./} (to store the links in the directory this
script was run in).
+See the description of @code{--prefix} for more.
+@item N
+This is the night-counter: starting from 1.
+@code{N} is just incremented by 1 for the next night, no matter how many
nights (without any dataset) there are between two subsequent observing nights
(its just an identifier for each night which you can easily map to different
calendar nights).
+@item I
+File counter in that night, sorted by time.
+@end table
+@item -c
+@itemx --copy
+Make a copy of each input FITS file with the standard naming convention
described in @option{--link}.
+With this option, instead of making a link, a copy is made.
+This option cannot be used with @option{--link}.
+@item -p STR
+@itemx --prefix=STR
+Prefix to append before the night-identifier of each newly created link or
copy.
+This option is thus only relevant with the @option{--copy} or @option{--link}
options.
+See the description of @option{--link} for how its used.
+For example, with @option{--prefix=img-}, all the created file names in the
current directory will start with @code{img-}, making outputs like
@file{img-n1-1.fits} or @file{img-n3-42.fits}.
+@option{--prefix} can also be used to store the links/copies in another
directory relative to the directory this script is being run (it must already
exist).
+For example @code{--prefix=/path/to/processing/img-} will put all the
links/copies in the @file{/path/to/processing} directory, and the files (in
that directory) will all start with @file{img-}.
+@item --stdintimeout=INT
+Number of micro-seconds to wait for standard input within this script.
+This does not correspond to general inputs into the script, inputs to the
script should always be given as a file.
+However, within the script, pipes are often used to pass the output of one
program to another.
+The value given to this option will be passed to those internal pipes.
+When running this script, if you confront an error, saying ``No input!'', you
should be able to fix it by giving a larger number to this option (the default
value is 10000000 micro-seconds or 10 seconds).
+@end table
-@node SAO DS9 region files from table, PSF construction and subtraction,
Generate radial profile, Installed scripts
-@section SAO DS9 region files from table
-Once your desired catalog (containing the positions of some objects) is
created (for example with @ref{MakeCatalog}, @ref{Match}, or @ref{Table}) it
often happens that you want to see your selected objects on an image for a
feeling of the spatial properties of your objects.
-For example you want to see their positions relative to each other.
-In this section we describe a simple installed script that is provided within
Gnuastro for converting your given columns to an SAO DS9 region file to help in
this process.
-SAO DS9@footnote{@url{http://ds9.si.edu}} is one of the most common FITS image
visualization tools in astronomy and is free software.
-@menu
-* Invoking astscript-ds9-region:: How to call astscript-ds9-region
-@end menu
-@node Invoking astscript-ds9-region, , SAO DS9 region files from table, SAO
DS9 region files from table
-@subsection Invoking astscript-ds9-region
-This installed script will read two positional columns within an input table
and generate an SAO DS9 region file to visualize the position of the given
objects over an image.
-For more on installed scripts please see (see @ref{Installed scripts}).
-This script can be used with the following general template:
-@example
-## Use the RA and DEC columns of 'table.fits' for the region file.
-$ astscript-ds9-region table.fits --column=RA,DEC \
- --output=ds9.reg
-## Select objects with a magnitude between 18 to 20, and generate the
-## region file directly (through a pipe), each region with radius of
-## 0.5 arcseconds.
-$ asttable table.fits --range=MAG,18:20 --column=RA,DEC \
- | astscript-ds9-region --column=1,2 --radius=0.5
-## With the first command, select objects with a magnitude of 25 to 26
-## as red regions in 'bright.reg'. With the second command, select
-## objects with a magnitude between 28 to 29 as a green region and
-## show both.
-$ asttable cat.fits --range=MAG_F160W,25:26 -cRA,DEC \
- | ./astscript-ds9-region -c1,2 --color=red -obright.reg
-$ asttable cat.fits --range=MAG_F160W,28:29 -cRA,DEC \
- | ./astscript-ds9-region -c1,2 --color=green \
- --command="ds9 image.fits -regions bright.reg"
-@end example
-The input can either be passed as a named file, or from standard input (a
pipe).
-Only the @option{--column} option is mandatory (to specify the input table
columns): two columns from the input table must be specified, either by name
(recommended) or number.
-You can optionally also specify the region's radius, width and color of the
regions with the @option{--radius}, @option{--width} and @option{--color}
options, otherwise default values will be used for these (described under each
option).
-The created region file will be written into the file name given to
@option{--output}.
-When @option{--output} isn't called, the default name of @file{ds9.reg} will
be used (in the running directory).
-If the file exists before calling this script, it will be overwritten, unless
you pass the @option{--dontdelete} option.
-Optionally you can also use the @option{--command} option to give the full
command that should be run to execute SAO DS9 (see example above and
description below).
-In this mode, the created region file will be deleted once DS9 is closed
(unless you pass the @option{--dontdelete} option).
-A full description of each option is given below.
-@table @option
-@item -h INT/STR
-@item --hdu INT/STR
-The HDU of the input table when a named FITS file is given as input.
-The HDU (or extension) can be either a name or number (counting from zero).
-For more on this option, see @ref{Input output options}.
-@item -c STR,STR
-@itemx --column=STR,STR
-Identifiers of the two positional columns to use in the DS9 region file from
the table.
-They can either be in WCS (RA and Dec) or image (pixel) coordinates.
-The mode can be specified with the @option{--mode} option, described below.
-@item -n STR
-@itemx --namecol=STR
-The column containing the name (or label) of each region.
-The type of the column (numeric or a character-based string) is irrelevant:
you can use both types of columns as a name or label for the region.
-This feature is useful when you need to recognize each region with a certain
ID or property (for example magnitude or redshift).
-@item -m wcs|img
-@itemx --mode=wcs|org
-The coordinate system of the positional columns (can be either
@option{--mode=wcs} and @option{--mode=img}).
-In the WCS mode, the values within the columns are interpreted to be RA and
Dec.
-In the image mode, they are interpreted to be pixel X and Y positions.
-This option also affects the interpretation of the value given to
@option{--radius}.
-When this option isn't explicitly given, the columns are assumed to be in WCS
mode.
-@item -C STR
-@itemx --color=STR
-The color to use for created regions.
-These will be directly interpreted by SAO DS9 when it wants to open the region
file so it must be recognizable by SAO DS9.
-As of SAO DS9 8.2, the recognized color names are @code{black}, @code{white},
@code{red}, @code{green}, @code{blue}, @code{cyan}, @code{magenta} and
@code{yellow}.
-The default color (when this option is not called) is @code{green}
-@item -w INT
-@itemx --width=INT
-The line width of the regions.
-These will be directly interpreted by SAO DS9 when it wants to open the region
file so it must be recognizable by SAO DS9.
-The default value is @code{1}.
-@item -r FLT
-@itemx --radius=FLT
-The radius of all the regions.
-In WCS mode, the radius is assumed to be in arc-seconds, in image mode, it is
in pixel units.
-If this option is not explicitly given, in WCS mode the default radius is 1
arc-seconds and in image mode it is 3 pixels.
+@node Generate radial profile, SAO DS9 region files from table, Sort FITS
files by night, Installed scripts
+@section Generate radial profile
-@item --dontdelete
-If the output file name exists, abort the program and don't over-write the
contents of the file.
-This option is thus good if you want to avoid accidentally writing over an
important file.
-Also, don't delete the created region file when @option{--command} is given
(by default, when @option{--command} is given, the created region file will be
deleted after SAO DS9 closes).
+@cindex Radial profile
+@cindex Profile, profile
+The 1 dimensional radial profile of an object is an important parameter in
many aspects of astronomical image processing.
+For example, you want to study how the light of a galaxy is distributed as a
function of the radial distance from the center.
+In other cases, the radial profile of a star can show the PSF (see @ref{PSF}).
+Gnuastro's @file{astscript-radial-profile} script is created to obtain such
radial profiles for one object within an image.
+This script uses @ref{MakeProfiles} to generate elliptical apertures with the
values equal to the distance from the center of the object and
@ref{MakeCatalog} for measuring the values over the apertures.
-@item -o STR
-@itemx --output=STR
-Write the created SAO DS9 region file into the name given to this option.
-If not explicitly given on the command-line, a default name of @file{ds9.reg}
will be used.
-If the file already exists, it will be over-written, you can avoid the
deletion (or over-writing) of an existing file with the @option{--dontdelete}.
+@menu
+* Invoking astscript-radial-profile:: How to call astscript-radial-profile
+@end menu
-@item --command="STR"
-After creating the region file, run the string given to this option as a
command-line command.
-The SAO DS9 region command will be appended to the end of the given command.
-Because the command will mostly likely contain white-space characters it is
recommended to put the given string in double quotations.
+@node Invoking astscript-radial-profile, , Generate radial profile, Generate
radial profile
+@subsection Invoking astscript-radial-profile
-For example, let's assume @option{--command="ds9 image.fits -zscale"}.
-After making the region file (assuming it is called @file{ds9.reg}), the
following command will be executed:
+This installed script will measure the radial profile of an object within an
image.
+For more on installed scripts please see (see @ref{Installed scripts}).
+This script can be used with the following general template:
@example
-ds9 image.fits -zscale -regions ds9.reg
+$ astscript-radial-profile [OPTION...] FITS-file
@end example
-You can customize all aspects of SAO DS9 with its command-line options,
therefore the value of this option can be as long and complicated as you like.
-For example if you also want the image to fit into the window, this option
will be: @command{--command="ds9 image.fits -zscale -zoom to fit"}.
-You can see the SAO DS9 command-line descriptions by clicking on the ``Help''
menu and selecting ``Reference Manual''.
-In the opened window, click on ``Command Line Options''.
-@end table
+@noindent
+Examples:
+
+@example
+## Generate the radial profile with default options (assuming the
+## object is in the center of the image, and using the mean).
+$ astscript-radial-profile image.fits
+## Generate the radial profile centered at x=44 and y=37 (in pixels),
+## up to a radial distance of 19 pixels, use the mean value.
+$ astscript-radial-profile image.fits --center=44,37 --rmax=19
+## Generate the radial profile centered at x=44 and y=37 (in pixels),
+## up to a radial distance of 100 pixels, compute sigma clipped
+## mean and standard deviation (sigclip-mean and sigclip-std) using
+## 5 sigma and 0.1 tolerance (default is 3 sigma and 0.2 tolerance).
+$ astscript-radial-profile image.fits --center=44,37 --rmax=100 \
+ --sigmaclip=5,0.1 \
+ --measure=sigclip-mean,sigclip-std
+## Generate the radial profile centered at RA=20.53751695,
+## DEC=0.9454292263, up to a radial distance of 88 pixels,
+## axis ratio equal to 0.32, and position angle of 148 deg.
+## Name the output table as `radial-profile.fits'
+$ astscript-radial-profile image.fits --mode=wcs \
+ --center=20.53751695,0.9454292263 \
+ --rmax=88 --axisratio=0.32 \
+ --positionangle=148 -oradial-profile.fits
+## Generate the radial profile centered at RA=40.062675270971,
+## DEC=-8.1511992735126, up to a radial distance of 20 pixels,
+## and calculate the SNR using the INPUT-NO-SKY and SKY-STD
+## extensions of the NoiseChisel output file.
+$ astscript-radial-profile image_detected.fits -hINPUT-NO-SKY \
+ --mode=wcs --measure=sn \
+ --center=40.062675270971,-8.1511992735126 \
+ --rmax=20 --stdhdu=SKY_STD
+## Generate the radial profile centered at RA=40.062675270971,
+## DEC=-8.1511992735126, up to a radial distance of 20 pixels,
+## and compute the SNR with a fixed value for std, std=10.
+$ astscript-radial-profile image.fits -h1 --mode=wcs --rmax=20 \
+ --center=40.062675270971,-8.1511992735126 \
+ --measure=sn --instd=10
+## Generate the radial profile centered at X=1201, Y=1201 pixels, up
+## to a radial distance of 20 pixels and compute the median and the
+## SNR using the first extension of sky-std.fits as the dataset for std
+## values.
+$ astscript-radial-profile image.fits -h1 --mode=img --rmax=20 \
+ --center=1201,1201 --measure=median,sn \
+ --instd=sky-std.fits
+@end example
+This installed script will read a FITS image and will use it as the basis for
constructing the radial profile.
+The output radial profile is a table (FITS or plain-text) containing the
radial distance from the center in the first row and the specified measurements
in the other columns (mean, median, sigclip-mean, sigclip-median, etc.).
+To measure the radial profile, this script needs to generate temporary files.
+All these temporary files will be created within the directory given to the
@option{--tmpdir} option.
+When @option{--tmpdir} is not called, a temporary directory (with a name based
on the inputs) will be created in the running directory.
+If the directory doesn't exist at run-time, this script will create it.
+After the output is created, this script will delete the directory by default,
unless you call the @option{--keeptmp} option.
+With the default options, the script will generate a circular radial profile
using the mean value and centered at the center of the image.
+In order to have more flexibility, several options are available to configure
for the desired radial profile.
+In this sense, you can change the center position, the maximum radius, the
axis ratio and the position angle (elliptical apertures are considered), the
operator for obtaining the profiles, and others (described below).
+@cartouche
+@noindent
+@strong{Debug your profile:} to debug your results, especially close to the
center of your object, you can see the radial distance associated to every
pixel in your input.
+To do this, use @option{--keeptmp} to keep the temporary files, and compare
@file{crop.fits} (crop of your input image centered on your desired coordinate)
with @file{apertures.fits} (radial distance of each pixel).
+@end cartouche
+@cartouche
+@noindent
+@strong{Finding properties of your elliptical target: } you want to measure
the radial profile of a galaxy, but don't know its exact location, position
angle or axis ratio.
+To obtain these values, you can use @ref{NoiseChisel} to detect signal in the
image, feed it to @ref{Segment} to do basic segmentation, then use
@ref{MakeCatalog} to measure the center (@option{--x} and @option{--y} in
MakeCatalog), axis ratio (@option{--axisratio}) and position angle
(@option{--positionangle}).
+@end cartouche
+@cartouche
+@noindent
+@strong{Masking other sources:} The image of an astronomical object will
usually have many other sources with your main target.
+A crude solution is to use sigma-clipped measurements for the profile.
+However, sigma-clipped measurements can easily be biased when the number of
sources at each radial distance increases at larger distances.
+Therefore a robust solution is to mask all other detections within the image.
+You can use @ref{NoiseChisel} and @ref{Segment} to detect and segment the
sources, then set all pixels that don't belong to your target to blank using
@ref{Arithmetic} (in particular, its @code{where} operator).
+@end cartouche
+@table @option
+@item -h STR
+@itemx --hdu=STR
+The HDU/extension of the input image to use.
+@item -o STR
+@itemx --output=STR
+Filename of measured radial profile.
+It can be either a FITS table, or plain-text table (determined from your given
file name suffix).
+@item -c FLT[,FLT[,...]]
+@itemx --center=FLT[,FLT[,...]]
+The central position of the radial profile.
+This option is used for placing the center of the profiles.
+This parameter is used in @ref{Crop} to center and crop the region.
+The positions along each dimension must be separated by a comma (@key{,}) and
fractions are also acceptable.
+The number of values given to this option must be the same as the dimensions
of the input dataset.
+The units of the coordinates are read based on the value to the
@option{--mode} option, see below.
+@item -O STR
+@itemx --mode=STR
+Interpret the center position of the object (values given to
@option{--center}) in image or WCS coordinates.
+This option thus accepts only two values: @option{img} or @option{wcs}.
+By default, it is @option{--mode=img}.
+@item -R FLT
+@itemx --rmax=FLT
+Maximum radius for the radial profile (in pixels).
+By default, the radial profile will be computed up to a radial distance equal
to the maximum radius that fits into the image (assuming circular shape).
+@item -Q FLT
+@itemx --axisratio=FLT
+The axis ratio of the apertures (minor axis divided by the major axis in a 2D
ellipse).
+By default (when this option isn't given), the radial profile will be circular
(axis ratio of 1).
+This parameter is used as the option @option{--qcol} in the generation of the
apertures with @command{astmkprof}.
+@item -p FLT
+@itemx --positionangle=FLT
+The position angle (in degrees) of the profiles relative to the first FITS
axis (horizontal when viewed in SAO DS9).
+By default, it is @option{--positionangle=0}, which means that the semi-major
axis of the profiles will be parallel to the first FITS axis.
-@node PSF construction and subtraction, , SAO DS9 region files from table,
Installed scripts
-@section PSF construction and subtraction
+@item -m STR
+@itemx --measure=STR
+The operator for measuring the values over each radial distance.
+The values given to this option will be directly passed to @ref{MakeCatalog}.
+As a consequence, all MakeCatalog measurements like the magnitude, magnitude
error, median, mean, signal-to-noise ratio (S/N), std, surface brightness,
sigclip-mean, sigclip-number, etc. can be used here.
+For a full list of MakeCatalog's measurements, please run
@command{astmkcatalog --help}.
+Multiple values can be given to this option, each separated by a comma.
+This option can also be called multiple times.
-@c Alternative names
-@c astscript-psf-create-select-stars --> astscript-psf-select
-@c astscript-psf-create-make-stamp --> astscript-psf-stamp
-@c astscript-psf-create-junction --> astscript-psf-unite
+Some measurements by MakeCatalog require a per-pixel sky standard deviation
(for example magnitude error or S/N).
+Therefore when asking for such measurements, use the @option{--instd} option
(described below) to specify the per-pixel sky standard deviation over each
pixel.
+For other measurements like the magnitude or surface brightness, MakeCatalog
will need a Zeropoint, which you can set with the @option{--zeropoint} option.
-The point spread function (PSF) describes how the light of a point-like source
is affected by several optical scattering effects (atmosphere, telescope,
instrument, etc.).
-The role of the PSF is key in astronomical analysis (for small and large
objects), and consequently, having a good characterization of the PSF is
fundamental.
-In many situations the PSF can be obtained by modeling it with analytical
functions (Gaussian, Moffat, etc.), see @ref{PSF}.
-However, in other scenarios, it is necessary to obtain an empirical
(non-parametric) and extended PSF.
-For this reason, here we have developed a set of scripts with the aim of
constructing the PSF using point-like sources from the same astronomical images
that the science is derived from.
+For example, by setting @option{--measure=mean,sigclip-mean --measure=median},
the mean, sigma-clipped mean and median values will be computed.
+The output radial profile will have 4 columns in this order: radial distance,
mean, sigma-clipped and median.
+By default (when this option isn't given), the mean of all pixels at each
radial position will be computed.
-The scripts are based on the concepts described in Infante-Sainz et al. (2020,
@url{https://arxiv.org/abs/1911.01430}).
-But to be complete, we first give a summary of the logic and overview of their
combined usage in @ref{Overview of the PSF scripts}.
-Furthermore, before going into the technical details of each script, in
@ref{Tutorial on building the extended PSF}, we'll start with a tutorial that
will use a real-world example to use the scripts in action!
+@item -s FLT,FLT
+@itemx --sigmaclip=FLT,FLT
+Sigma clipping parameters: only relevant if sigma-clipping operators are
requested by @option{--measure}.
+For more on sigma-clipping, see @ref{Sigma clipping}.
+If given, the value to this option is directly passed to the
@option{--sigmaclip} option of @ref{MakeCatalog}, see @ref{MakeCatalog inputs
and basic settings}.
+By default (when this option isn't given), the default values within
MakeCatalog will be used.
+To see the default value of this option in MakeCatalog, you can run this
command:
-@menu
-* Overview of the PSF scripts:: Summary of concepts and methods
-* Tutorial on building the extended PSF:: Tutorial with real-world usage of
scripts
-* Invoking astscript-psf-create-select-stars:: Select good starts within an
image.
-* Invoking astscript-psf-create-make-stamp:: Make a stamp of each star to
stack.
-* Invoking astscript-psf-create-junction:: Merge stacks of different regions
of PSF.
-* Invoking astscript-psf-model-flux-factor:: Calculate factor to scale PSF to
star.
-* Invoking astscript-psf-model-scattered-light:: Put the PSF in the image to
subtract.
-@end menu
+@example
+$ astmkcatalog -P | grep " sigmaclip "
+@end example
-@node Overview of the PSF scripts, Tutorial on building the extended PSF, PSF
construction and subtraction, PSF construction and subtraction
-@subsection Overview of the PSF scripts
-The fundamental ideas of the following methodology are thoroughly described in
Infante-Sainz et al. (2020, @url{https://arxiv.org/abs/1911.01430}).
-Once the PSF has been obtained, it can be scaled to the magnitude of the
point-like sources to subtract.
-We therefore also have a script for this.
-We describe the technical implementation of some scripts developed with the
aim of constructing an empirical PSF, and then, use that PSF for correcting the
scattered light.
-All scripts are independent of each other, meaning this that you are free to
use all of them as you wish (for example only some of them, using them for
other purposes, or running independent parts in parallel).
+@item -z FLT
+@itemx --zeropoint=FLT
+The Zeropoint of the input dataset.
+This is necessary when you request measurements like Magnitude, or Surface
brightness.
-For constructing the PSF, the basic idea is to crop images of stars (with
other sources masked), scale them properly, and finally stack all of them
together.
-As a consequence, the first step would be to obtain a catalog of stars within
the datasets.
-In general, not all stars are good to construct a PSF.
-For example we don't want contamination from other bright, and nearby objects.
-The first script below is therefore designed for selecting only good candidate
stars in your image.
-It will use different criteria, for example good parallax (where available, to
avoid confusion with galaxies), not near bright stars, axis ratio, etc.
-For more on this script, see @ref{Invoking astscript-psf-create-select-stars}.
+@item -v INT
+@itemx --oversample=INT
+Oversample the input dataset to the fraction given to this option.
+Therefore if you set @option{--rmax=20} for example and
@option{--oversample=5}, your output will have 100 rows (without
@option{--oversample} it will only have 20 rows).
+Unless the object is heavily undersampled (the pixels are larger than the
actual object), this method provides a much more accurate result and there are
sufficient number of pixels to get the profile accurately.
-Once the catalog of stars is constructed, another script is in charge of
making appropiate stamps of the stars.
-Each stamp is a cropped image of the star with the desired size, normalization
of the flux, and mask of the contaminant objects.
-For more on this script, see @ref{Invoking astscript-psf-create-make-stamp}
-After obtaining a set of star stamps, they can be stacked for obtaining the
combined PSF from many stars (for example with @ref{Stacking operators}).
+@item -u INT
+@itemx --undersample=INT
+Undersample the input dataset by the number given to this option.
+This option is for considering larger apertures than the original pixel size
(aperture size is equal to 1 pixel).
+For example, if a radial profile computed by default has 100 different radii
(apertures of 1 pixel width), by considering @option{--undersample=2} the
radial profile will be computed over apertures of 2 pixels, so the final radial
profile will have 50 different radii.
+This option is good to measure over a larger number of pixels to improve the
measurement.
-In the combined PSF, the masked background objects of each star's image will
be covered and the signal-to-noise ratio will increase, giving a very nice view
of the ``clean'' PSF.
-However, is usually necessary to obtain different regions of the same PSF from
different stars.
-For example, to construct the far wings it is necessary to consider very
bright stars.
-However, these stars will be saturated in the most inner part, and immediately
outside of the saturation level, they will be deformed due to non-linearity
effects.
-Consequently, fainter stars are necessary for the inner regions.
+@item -i FLT/STR
+@itemx --instd=FLT/STR
+Sky standard deviation as a single number (FLT) or as the filename (STR)
containing the image with the std value for each pixel (the HDU within the file
should be given to the @option{--stdhdu} option mentioned below).
+This is only necessary when the requested measurement (value given to
@option{--measure}) by MakeCatalog needs the Standard deviation (for example
the signal-to-noise ratio or magnitude error).
+If your measurements don't require a standard deviation, it is best to ignore
this option (because it will slow down the script).
-Therefore, you need to repeat the steps above for certain stars (in a certain
magnitude range) to obtain the PSF in certain radial ranges.
-For example, in Infante-Sainz et al. (2020,
@url{https://arxiv.org/abs/1911.01430}), the final PSF was constructed from
three regions (and thus, using stars from three ranges in magnitude).
-In other cases, we even needed four groups of stars!
-Once clean stacks of different parts of the PSF have been constructed through
the steps above, it is therefore necessary to blend them all into one.
-This is done by finding a common radial region in both, and scaling the inner
region by a factor to add with the outer region.
-This is not trivial, therefore, a third script is in charge of it, see
@ref{Invoking astscript-psf-create-junction}.
+@item -d INT/STR
+@itemx --stdhdu=INT/STR
+HDU/extension of the sky standard deviation image specified with
@option{--instd}.
-Having constructed the PSF as described above (or by any other procedure), it
can be scaled to the brightness of the various stars in the image to get
subtracted (and thus remove the extended/bright wings; better showing the
background objects of interest).
-Note that the absolute flux of a PSF is meaningless (and in fact, it is
usually normalized to have a total sum of unity!), so it should be scaled.
-We therefore have another script that will calculate the scale
(multiplication) factor of the PSF for each star.
-For more on the scaling script, see @ref{Invoking
astscript-psf-model-flux-factor}.
+@item -t STR
+@itemx --tmpdir=STR
+Several intermediate files are necessary to obtain the radial profile.
+All of these temporal files are saved into a temporal directory.
+With this option, you can directly specify this directory.
+By default (when this option isn't called), it will be built in the running
directory and given an input-based name.
+If the directory doesn't exist at run-time, this script will create it.
+Once the radial profile has been obtained, this directory is removed.
+You can disable the deletion of the temporary directory with the
@option{--keeptmp} option.
-Once the flux factor has been computed, a final script is in charge of placing
the scaled PSF over the proper location in the image and subtract it.
-For more on the scaling and positioning script, see @ref{Invoking
astscript-psf-model-scattered-light}.
+@item -k
+@itemx --keeptmp
+Don't delete the temporary directory (see description of @option{--tmpdir}
above).
+This option is useful for debugging.
+For example, to check that the profiles generated for obtaining the radial
profile have the desired center, shape and orientation.
+@end table
-As mentioned above, in the following sections, each script has its own
documentation and list of options for very detailed customization (if
necessary).
-But before going into them, we continue with a practical tutorial to show the
usage of scripts in practice; see @ref{Tutorial on building the extended PSF}.
-@node Tutorial on building the extended PSF, Invoking
astscript-psf-create-select-stars, Overview of the PSF scripts, PSF
construction and subtraction
-@subsection Tutorial on building the extended PSF
-Deriving the extended PSF of an image is very important in many aspects of the
analysis of the objects within it.
-Gnuastro has a set of installed scripts, designed to simplify the process
following the recipe of Infante-Sainz et al. (2020,
@url{https://arxiv.org/abs/1911.01430}); for more, see @ref{PSF construction
and subtraction}.
-An overview of the process is given in @ref{Overview of the PSF scripts}.
-@menu
-* Preparing input for extended PSF:: Which stars should be used?
-* Saturated pixels and Segment's clumps:: Saturation is a major hurdle!
-* Building outer part of PSF:: Building the outermost PSF wings.
-* Inner parts of the PSF:: Going towards the PSF center.
-* Uniting the different PSF components:: Merging all the components into one
PSF.
-* Subtracting the PSF:: Having the PSF, we now want to subtract it.
-@end menu
-@node Preparing input for extended PSF, Saturated pixels and Segment's clumps,
Tutorial on building the extended PSF, Tutorial on building the extended PSF
-@subsubsection Preparing input for extended PSF
-In @ref{Overview of the PSF scripts}, we presented an overview of using the
separate scripts in sequence, to build an extended PSF.
-Before digging into the details of each script, we'll start here with a fully
working tutorial on how to use the scripts in order to construct a PSF, model
the scattered light field and subtract it from the image.
-For this particular example, an image of the M51 galaxy group in the r (SDSS)
band of the Javalambre Photometric Local Universe Survey (J-PLUS) used.
-For more information on J-PLUS, and its unique features visit:
@url{http://www.j-plus.es}.
-First, let's download the image from the J-PLUS webpage using @code{wget}.
-But to have a generalize-able, and easy to read command, we'll define some
base variables (in all-caps) first.
-After the download is complete, open the image with SAO DS9 (or any other FITS
viewer you prefer!) to have a feeling of the data (and of course, enjoy the
beauty of M51!).
-@example
-$ IMAGEID="jplus-dr2/get_fits?id=67510"
-$ URL="http://archive.cefca.es/catalogues/vo/siap/";
-$ wget $URL$IMAGEID -O image.fits.fz
-$ ds9 image.fits.fz -zoom to fit -zscale
-@end example
-Have a closer look at the edges of the image: zoom in to the corners.
-You see that on the edges the pixel values are either zero, or with
significantly different values than the main body of the image.
-This is due to the dithering pattern that was used to make this image.
-To avoid potential issues or problems that these regions may cause, we will
first crop out the main body of the image with the command below.
-To keep the top-level directory clean, let's also put the crop in a directory
called @file{flat}.
-@example
-$ mkdir flat
-$ astcrop image.fits.fz --mode=img --section=225:9275,150:9350 \
- -oflat/image.fits
-$ ds9 flat/image.fits -zoom to fit -zscale
-@end example
-@noindent
-Please zoom into the edges again, you will see that they now have the same
noise-level as the rest of the image (the problematic parts are now gone).
-@node Saturated pixels and Segment's clumps, Building outer part of PSF,
Preparing input for extended PSF, Tutorial on building the extended PSF
-@subsubsection Saturated pixels and Segment's clumps
-As explained in @ref{Overview of the PSF scripts}, it is important to mask
other sources in the image.
-Therefore, before going onto selecting stars, let's detect all significant
signal, and separate the clumps over the detections.
-However, the saturated pixels of the bright stars are going to cause problems.
-To see this problem, let's make a @mymath{1000\times1000} crop around a bright
star to speed up the test (and its solution):
-@example
-$ astcrop flat/image.fits --mode=wcs --widthinpix --width=1000 \
- --center=203.3916736,46.7968652 --output=saturated.fits
-$ astnoisechisel saturated.fits --output=sat-nc.fits
-$ astsegment sat-nc.fits --output=sat-seg.fits
-$ ds9 -mecube -zscale sat-seg.fits -zoom to fit
-@end example
-Have a look at the @code{CLUMPS} extension, you will see that instead of a
single clump at the center of the star, we have multiple!
-This is because of the saturated pixels!
-When saturation occurs, the peak of the profile is lost (like cutting off the
tip of a mountain!) and all saturated pixels get a noisy value close to the
saturation level.
-This disrupts Segment's assumption to expand clumps from local maxima (the
noise at the very high S/N level, results in each one being treated as a
separate clump).
-To have the center identified as a single clump, we should mask these
saturated pixels in a way that suites Segment's non-parametric methodology.
-To find the saturation level, let's make a smaller crop of @mymath{50\times50}
pixels around the star with the command below, then have a look at the
distribution of pixels with a value larger than 100 (above the noise level):
-@example
-$ astcrop image-crop.fits --mode=wcs --widthinpix --width=50 \
- --center=203.3916736,46.7968652 --output=saturated-center.fits
-$ aststatistics saturated-center.fits --greaterequal=100
-Histogram:
- |*
- |*
- |*
- |* *
- |** *
- |*** *
- |*** *
- |***** **
- |******** ****
- |********** * * ** ***** *
- |************************* ** * ***** * * * ** *** * *************
- |----------------------------------------------------------------------
-@end example
-The peak you see in the right end (larger values) of the histogram shows the
saturated pixels.
-If there was no saturation, you would expect the number of pixels at
increasing values to simply decrease until reaching the maximum value of the
profile.
-But that is not the case here.
-Please try this experiment on a non-saturated star (decreasing the crop width)
to see what we mean.
-Finding the saturation level is easy with Statistics (by using the
@option{--lessthan} option until the histogram becomes as expected: only
decreasing).
-First let's try @option{--lessthan=3000} and then @option{--lessthan=2500}:
-@example
-$ aststatistics saturated-center.fits --greaterequal=100 --lessthan=3000
--------
-Histogram:
- |*
- |*
- |*
- |*
- |**
- |***
- |*** *
- |*** *
- |******* **
- |*********** * * * **
- |********************* *** * *** * ***** *** * * *** * * ********
- |----------------------------------------------------------------------
+@node SAO DS9 region files from table, PSF construction and subtraction,
Generate radial profile, Installed scripts
+@section SAO DS9 region files from table
+Once your desired catalog (containing the positions of some objects) is
created (for example with @ref{MakeCatalog}, @ref{Match}, or @ref{Table}) it
often happens that you want to see your selected objects on an image for a
feeling of the spatial properties of your objects.
+For example you want to see their positions relative to each other.
-$ aststatistics saturated-center.fits --greaterequal=100 --lessthan=2500
--------
-Histogram:
- |*
- |*
- |*
- |**
- |**
- |**
- |****
- |**** **
- |*********
- |************** ** * * * *
- |******************** ***** *** * *** * *** ** * ** * * * **
- |----------------------------------------------------------------------
-@end example
+In this section we describe a simple installed script that is provided within
Gnuastro for converting your given columns to an SAO DS9 region file to help in
this process.
+SAO DS9@footnote{@url{http://ds9.si.edu}} is one of the most common FITS image
visualization tools in astronomy and is free software.
-@noindent
-We still see an increase in the histogram around 3000.
-Therefore, We can set the saturation level in this image@footnote{In raw
exposures, this value is usually around 65000 (close to @mymath{2^{16}}, since
most CCDs have 16-bit pixels). But that is not the case here, because this is a
processed/stacked image that has been calibrated.} to 2500.
-Let's mask all such pixels with the command below:
+@menu
+* Invoking astscript-ds9-region:: How to call astscript-ds9-region
+@end menu
-@example
-$ astarithmetic saturated.fits set-i i i 2500 gt nan where \
- --output=sat-masked.fits
-$ ds9 sat-masked.fits
-@end example
+@node Invoking astscript-ds9-region, , SAO DS9 region files from table, SAO
DS9 region files from table
+@subsection Invoking astscript-ds9-region
-Zoom into the center: you will see that the staturated pixels are indeed
masked.
-However, there is still another problem: on the edges of the vertical
``bleeding'' saturated pixels, there are strong negative values (almost like
``waves'').
-Such that the pixels just touching the saturated pixels have strong negative
values.
-These will also cause problems!
-So with the same command, let's dilate the saturated regions by four times.
+This installed script will read two positional columns within an input table
and generate an SAO DS9 region file to visualize the position of the given
objects over an image.
+For more on installed scripts please see (see @ref{Installed scripts}).
+This script can be used with the following general template:
@example
-$ astarithmetic saturated.fits set-i i i 2500 gt \
- 2 dilate 2 dilate 2 dilate 2 dilate nan where \
- --output=sat-masked.fits
-$ ds9 sat-masked.fits
-@end example
+## Use the RA and DEC columns of 'table.fits' for the region file.
+$ astscript-ds9-region table.fits --column=RA,DEC \
+ --output=ds9.reg
-Now that saturated pixels (and their problematic neighbors) have been masked,
we can convolve the image (recall that Segment will use the convolved image for
identifing clumps) with the command below.
-However, we will use the Spatial Domain convolution which can account for
blank pixels (see @ref{Spatial vs. Frequency domain}).
-We will also create a Gaussian kernel with @mymath{\rm{FWHM}=2} pixels,
truncated at @mymath{5\times\rm{FWHM}}.
+## Select objects with a magnitude between 18 to 20, and generate the
+## region file directly (through a pipe), each region with radius of
+## 0.5 arcseconds.
+$ asttable table.fits --range=MAG,18:20 --column=RA,DEC \
+ | astscript-ds9-region --column=1,2 --radius=0.5
-@example
-$ astmkprof --kernel=gaussian,2,5 --oversample=1 -okernel.fits
-$ astconvolve sat-masked.fits --kernel=kernel.fits --domain=spatial \
- --output=sat-masked-conv.fits
+## With the first command, select objects with a magnitude of 25 to 26
+## as red regions in 'bright.reg'. With the second command, select
+## objects with a magnitude between 28 to 29 as a green region and
+## show both.
+$ asttable cat.fits --range=MAG_F160W,25:26 -cRA,DEC \
+ | ./astscript-ds9-region -c1,2 --color=red -obright.reg
+$ asttable cat.fits --range=MAG_F160W,28:29 -cRA,DEC \
+ | ./astscript-ds9-region -c1,2 --color=green \
+ --command="ds9 image.fits -regions bright.reg"
@end example
-@noindent
-After convolution, the problematic pixels are still NaN.
-But Segment requires the profile to start with a maximum value and decrease.
-So before feeding into Segment, we'll fill the blank values with the maximum
value of the neighboring pixels (see @ref{Interpolation operators}):
+The input can either be passed as a named file, or from standard input (a
pipe).
+Only the @option{--column} option is mandatory (to specify the input table
columns): two columns from the input table must be specified, either by name
(recommended) or number.
+You can optionally also specify the region's radius, width and color of the
regions with the @option{--radius}, @option{--width} and @option{--color}
options, otherwise default values will be used for these (described under each
option).
-@example
-$ astarithmetic sat-masked-conv.fits 2 interpolate-maxofregion \
- --output=sat-conv.fits
-$ ds9 sat-conv.fits
-@end example
+The created region file will be written into the file name given to
@option{--output}.
+When @option{--output} isn't called, the default name of @file{ds9.reg} will
be used (in the running directory).
+If the file exists before calling this script, it will be overwritten, unless
you pass the @option{--dontdelete} option.
+Optionally you can also use the @option{--command} option to give the full
command that should be run to execute SAO DS9 (see example above and
description below).
+In this mode, the created region file will be deleted once DS9 is closed
(unless you pass the @option{--dontdelete} option).
+A full description of each option is given below.
-@noindent
-Open the output and have a look;
-The output image doesn't have blank pixels any more and each blank region (in
the centers of stars) is now filled with the largest value that is touching
that particular region.
-We can now feed this image to Segment as the convolved image:
+@table @option
-@example
-$ astsegment sat-nc.fits --convolved=sat-conv.fits --output=sat-seg.fits
-$ ds9 -mecube -zscale sat-seg.fits -zoom to fit
-@end example
+@item -h INT/STR
+@item --hdu INT/STR
+The HDU of the input table when a named FITS file is given as input.
+The HDU (or extension) can be either a name or number (counting from zero).
+For more on this option, see @ref{Input output options}.
-@noindent
-Please look at the @code{CLUMPS} extension.
-Do you see how the whole center of the star has indeed been identified as a
single clump?
-We thus achieved our aim and didn't let the saturated pixels harm the
identification of the center.
+@item -c STR,STR
+@itemx --column=STR,STR
+Identifiers of the two positional columns to use in the DS9 region file from
the table.
+They can either be in WCS (RA and Dec) or image (pixel) coordinates.
+The mode can be specified with the @option{--mode} option, described below.
-Now we can extend it to the whole image.
-To detect signal, we can run NoiseChisel using the command below.
-In short:
-@itemize
-@item
-Since the image is so large, we increase @option{--interpnumngb} to get better
outlier statistics on the tiles.
-@item
-Since the image not the result of too many exposures, it doesn't have strong
correlated noise, so we'll decrease @option{--detgrowquant} and increase
@option{--detgrowmaxholesize}.
-@end itemize
-@noindent
-See @ref{Detecting large extended targets} for more on optimizing NoiseChisel.
-Furthermore, since both NoiseChisel and Segment need a convolved image, we'll
do the convolution before and feed it to both (to save running time).
-But first, let's delete all the temporary files we made above.
+@item -n STR
+@itemx --namecol=STR
+The column containing the name (or label) of each region.
+The type of the column (numeric or a character-based string) is irrelevant:
you can use both types of columns as a name or label for the region.
+This feature is useful when you need to recognize each region with a certain
ID or property (for example magnitude or redshift).
-@example
-$ rm *.fits
-$ mkdir label
-$ astmkprof --kernel=gaussian,2,5 --oversample=1 \
- -olabel/kernel.fits
-$ astarithmetic flat/image.fits set-i i i 2500 gt \
- 2 dilate 2 dilate 2 dilate 2 dilate nan where \
- --output=flat/sat-masked.fits
-$ astconvolve flat/sat-masked.fits --kernel=kernel.fits \
- --domain=spatial --output=label/sat-masked-conv.fits
-$ astarithmetic label/sat-masked-conv.fits 2 interpolate-maxofregion \
- --output=label/image-conv.fits
-$ astnoisechisel label/image-conv.fits --interpnumngb=100 \
- --detgrowquant=0.8 --detgrowmaxholesize=100000 \
- --convolved=label/image-conv.fits \
- --output=label/nc.fits
-$ astsegment label/nc.fits --convolved=label/image-conv.fits \
- --output=label/seg.fits
-$ ds9 -mecube -zscale label/nc.fits -zoom to fit
-@end example
+@item -m wcs|img
+@itemx --mode=wcs|org
+The coordinate system of the positional columns (can be either
@option{--mode=wcs} and @option{--mode=img}).
+In the WCS mode, the values within the columns are interpreted to be RA and
Dec.
+In the image mode, they are interpreted to be pixel X and Y positions.
+This option also affects the interpretation of the value given to
@option{--radius}.
+When this option isn't explicitly given, the columns are assumed to be in WCS
mode.
-Looking in the @code{SKY} extension of NoiseChisel, we see that there is no
footprint of the bright stars or of M51, so the detection was good!
-Furthermore, the saturated pixels haven't caused any problems and the central
clumps of bright stars are now a single clump.
-We can now proceed to estimating the outer PSF in @ref{Building outer part of
PSF}.
-But first, let's clean up behind our selves (we only need the Segment output)
+@item -C STR
+@itemx --color=STR
+The color to use for created regions.
+These will be directly interpreted by SAO DS9 when it wants to open the region
file so it must be recognizable by SAO DS9.
+As of SAO DS9 8.2, the recognized color names are @code{black}, @code{white},
@code{red}, @code{green}, @code{blue}, @code{cyan}, @code{magenta} and
@code{yellow}.
+The default color (when this option is not called) is @code{green}
+
+@item -w INT
+@itemx --width=INT
+The line width of the regions.
+These will be directly interpreted by SAO DS9 when it wants to open the region
file so it must be recognizable by SAO DS9.
+The default value is @code{1}.
+
+@item -r FLT
+@itemx --radius=FLT
+The radius of all the regions.
+In WCS mode, the radius is assumed to be in arc-seconds, in image mode, it is
in pixel units.
+If this option is not explicitly given, in WCS mode the default radius is 1
arc-seconds and in image mode it is 3 pixels.
+
+@item --dontdelete
+If the output file name exists, abort the program and don't over-write the
contents of the file.
+This option is thus good if you want to avoid accidentally writing over an
important file.
+Also, don't delete the created region file when @option{--command} is given
(by default, when @option{--command} is given, the created region file will be
deleted after SAO DS9 closes).
-@example
-$ rm label/image-conv.fits label/kernel.fits label/sat-masked.fits \
- label/sat-masked-conv.fits
-@end example
+@item -o STR
+@itemx --output=STR
+Write the created SAO DS9 region file into the name given to this option.
+If not explicitly given on the command-line, a default name of @file{ds9.reg}
will be used.
+If the file already exists, it will be over-written, you can avoid the
deletion (or over-writing) of an existing file with the @option{--dontdelete}.
-@node Building outer part of PSF, Inner parts of the PSF, Saturated pixels and
Segment's clumps, Tutorial on building the extended PSF
-@subsubsection Building outer part of PSF
-In @ref{Preparing input for extended PSF}, we described how to create a
Segment clump and object map, while accounting for saturated stars.
-So we are now ready to start building the outer parts of the PSF.
+@item --command="STR"
+After creating the region file, run the string given to this option as a
command-line command.
+The SAO DS9 region command will be appended to the end of the given command.
+Because the command will mostly likely contain white-space characters it is
recommended to put the given string in double quotations.
-First we will build the outer parts of the PSF, so we want the brightest stars.
-You will see we have several bright stars in this very large field of view,
but we don't yet have a feeling how how many they are, and at what magnitudes.
-So let's use Gnuastro's Query program to find the magnitudes of the brightest
stars (those brighter than magnitude 12).
-For more on Query, see @ref{Query}.
+For example, let's assume @option{--command="ds9 image.fits -zscale"}.
+After making the region file (assuming it is called @file{ds9.reg}), the
following command will be executed:
@example
-$ astquery gaia --dataset=edr3 --overlapwith=flat/image.fits \
- --range=phot_g_mean_mag,-inf,12 --output=gaia.fits
+ds9 image.fits -zscale -regions ds9.reg
@end example
-Now, we can easily visualize the magnitude and positions of these stars using
@command{astscript-ds9-region} and the command below (for more on this script,
see @ref{SAO DS9 region files from table})
+You can customize all aspects of SAO DS9 with its command-line options,
therefore the value of this option can be as long and complicated as you like.
+For example if you also want the image to fit into the window, this option
will be: @command{--command="ds9 image.fits -zscale -zoom to fit"}.
+You can see the SAO DS9 command-line descriptions by clicking on the ``Help''
menu and selecting ``Reference Manual''.
+In the opened window, click on ``Command Line Options''.
+@end table
-@example
-$ astscript-ds9-region gaia.fits -cra,dec \
- --namecol=phot_g_mean_mag \
- --command="ds9 flat/image.fits -zoom to fit -zscale"
-@end example
-You can see that we have several stars between magnitudes 6 to 10.
-Let's use @file{astscript-psf-create-select-stars} in the command below to
select the relevant stars in the image (the brightest; with a magnitude between
6 to 8).
-Since this will select very bright stars, we will also increase the distance
to nearby neighbors with brighter or similar magnitudes (the default value is 1
arcmin).
-To do this, we will set @option{--mindistdeg=0.05}, which corresponds to 3
arcmin (or 3/60 degrees).
-@example
-$ mkdir outer
-$ astscript-psf-create-select-stars flat/image.fits \
- --magnituderange=6,8 \
- --mindistdeg=0.05 \
- --output=outer/stars.fits
-@end example
-@noindent
-Let's have a look at the selected stars in the image (it is very important to
visually check every step when you are first discovering a new dataset).
-@example
-$ astscript-ds9-region outer/stars.fits -cra,dec \
- --namecol=phot_g_mean_mag \
- --command='ds9 flat/image.fits -zoom to fit -zscale'
-@end example
-Now that the catalog of good stars is ready, it is time to construct the
individual stamps for each star of the catalog.
-To do that, we'll use @file{astscript-psf-create-make-stamp}.
-One of the most important parameters for this script is the normalization
radii @code{--normradii}.
-This parameter defines a ring for the flux normalization of each star stamp.
-The normalization of the flux is necessary because each star has a different
brightness, and consequently, it is crucial for having all the stamps with the
same flux level in the same region.
-Otherwise the final stack of the different stamps would have no sense.
-Depending on the PSF shape, internal reflections, ghosts, saturated pixels,
and other systematics, it would be necessary to choose the @code{--normradii}
appropriately.
-The selection of the normalization radii is something that requires a good
understanding of the data.
-To do that, let's use two useful parameters that will help us in the checking
of the data: @code{--tmpdir} and @code{--keeptmp}.
-With @code{--tmpdir=checking-normradii} all temporary files, including the
radial profiles, will be save in that directory (instead of an
internally-created name).
-With @code{--keeptmp} we won't remove the temporal files, so it is possible to
have a look at them (by default the temporary directory gets deleted at the
end).
-It is necessary to specify the @code{--normradii} even if we don't know yet
the final values.
-Otherwise the script will not generate the radial profile.
-As a consequence, in this step we put the normalization radii equal to the
size of the stamps.
-By doing this, the script will generate the radial profile of the entire stamp.
-In this particular step we set it to @code{--normradii=150,160}.
-@example
-$ counter=1
-$ mkdir finding-normradii
-$ asttable outer/stars.fits \
- | while read -r ra dec mag; do
- astscript-psf-create-make-stamp flat/sat-masked.fits \
- --mode=wcs \
- --stampwidth=500 \
- --center=$ra,$dec \
- --normradii=250,260 \
- --segment=label/seg.fits \
- --output=finding-normradii/$counter.fits \
- --tmpdir=finding-normradii --keeptmp; \
- counter=$((counter+1)); \
- done
-@end example
-Now, by having a look at all radial profiles at the same time, it is possible
to choose a good normalization region where the profiles are similar.
-For example using Topcat or any other plotting tool.
-In the same way, it is always good to check the different stamps to ensure the
quality and posible two dimensional features that are difficult to detect from
the radial profiles (i.e., ghosts, internal reflections, etc.).
-@example
-$ topcat finding-normradii/rprofile*.fits
-$ ds9 finding-normradii/cropped-masked*.fits
-@end example
-After some study of this data, we could say that a good normalization ring is
those pixels between R=50 and R=60 pixels.
-Such a ring ensures having a high number of pixels so the estimation of the
flux normalization will be robust.
-Also, at such distance from the center the signal to noise is high and there
are not obvious features that can affect the normalization.
-We later need the normalization radii in the next steps also.
-Therefore, to avoid typos or chances of a mistake, we'll define the two
@code{NORMRADII_INNER} and @code{NORMRADII_OUTER} variables.
-Furthermore, since there are several stars (as you saw from the output of the
previous command), we iterate over each row of the catalog.
-@example
-$ counter=1
-$ IMAGE=image-masked.fits
-$ NORMRADII_INNER=50
-$ NORMRADII_OUTER=60
-$ mkdir outer/stamps
-$ asttable outer/stars.fits \
- | while read -r ra dec mag; do
- astscript-psf-create-make-stamp flat/sat-masked.fits \
- --mode=wcs \
- --stampwidth=500 \
- --center=$ra,$dec \
- --segment=label/seg.fits \
- --output=outer/stamps/$counter.fits \
- --normradii=$NORMRADII_INNER,$NORMRADII_OUTER; \
- counter=$((counter+1)); \
- done
-@end example
-After the stamps are created, we need to stack them together with a simple
Arithmetic command (see @ref{Stacking operators}).
-The stack is done using the median operator and the stacked image is the outer
PSF that we are looking for: @file{psf-outer.fits}.
-@example
-$ astarithmetic outer/stamps/*.fits -g1 \
- $(ls outer/stamps/*.fits | wc -l) median \
- --output=outer/stack.fits
-@end example
-@noindent
-With the command below, we'll compare this stacked PSF with the images of the
individual stars that were used to create it.
-You clearly see that the number of masked pixels is significantly decreased
and the PSF is much more ``cleaner'').
-@example
-$ ds9 outer/stack.fits outer/stamps/*.fits
-@end example
-However, the bleeding, vertical saturation in the center still remains.
-Also, because we didn't have too many images (only three!), some small regions
still remain that were (by chance!) masked in all three.
-If we had more bright stars in our selected magnitude range, we could have
filled those outer remaining patches.
-@c -------------------------------------------
-@c ------- This part may be removed ----------
-To fill those regions, we can use the radial profile of the stacked PSF.
-Obtaining the radial profile for a simple (square, with a single, circular
object), image is very easy:
-@example
-$ astscript-radial-profile outer/stack.fits -oouter/stack-radial.fits
-$ echo "1 251 251 8 300 0 0 1 1 1" \
- | astmkprof --background=outer/stack.fits --clearcanvas \
- --customtable=outer/stack-radial.fits
+@node PSF construction and subtraction, , SAO DS9 region files from table,
Installed scripts
+@section PSF construction and subtraction
-@end example
-@c -------------------------------------------
+@c Alternative names
+@c astscript-psf-create-select-stars --> astscript-psf-select
+@c astscript-psf-create-make-stamp --> astscript-psf-stamp
+@c astscript-psf-create-junction --> astscript-psf-unite
-@node Inner parts of the PSF, Uniting the different PSF components, Building
outer part of PSF, Tutorial on building the extended PSF
-@subsubsection Inner parts of the PSF
+The point spread function (PSF) describes how the light of a point-like source
is affected by several optical scattering effects (atmosphere, telescope,
instrument, etc.).
+The role of the PSF is key in astronomical analysis (for small and large
objects), and consequently, having a good characterization of the PSF is
fundamental.
+In many situations the PSF can be obtained by modeling it with analytical
functions (Gaussian, Moffat, etc.), see @ref{PSF}.
+However, in other scenarios, it is necessary to obtain an empirical
(non-parametric) and extended PSF.
+For this reason, here we have developed a set of scripts with the aim of
constructing the PSF using point-like sources from the same astronomical images
that the science is derived from.
-@node Uniting the different PSF components, Subtracting the PSF, Inner parts
of the PSF, Tutorial on building the extended PSF
-@subsubsection Uniting the different PSF components
+The scripts are based on the concepts described in Infante-Sainz et al. (2020,
@url{https://arxiv.org/abs/1911.01430}).
+But to be complete, we first give a summary of the logic and overview of their
combined usage in @ref{Overview of the PSF scripts}.
+Furthermore, before going into the technical details of each script, in
@ref{Tutorial on building the extended PSF}, we'll start with a tutorial that
will use a real-world example to use the scripts in action!
+@menu
+* Overview of the PSF scripts:: Summary of concepts and methods
+* Invoking astscript-psf-create-select-stars:: Select good starts within an
image.
+* Invoking astscript-psf-create-make-stamp:: Make a stamp of each star to
stack.
+* Invoking astscript-psf-create-junction:: Merge stacks of different regions
of PSF.
+* Invoking astscript-psf-model-flux-factor:: Calculate factor to scale PSF to
star.
+* Invoking astscript-psf-model-scattered-light:: Put the PSF in the image to
subtract.
+@end menu
-@node Subtracting the PSF, , Uniting the different PSF components, Tutorial
on building the extended PSF
-@subsubsection Subtracting the PSF
-With the commands above PSF has been constructed and now it is going to be
used for modeling and subtracting each individual star.
-By construction, the pixel values of the PSF came from the normalization of
the individual stamps.
-As a consequence, when modeling each individual star, it is necessary to
compute what is the factor by which the PSF has to be scaled.
-This is done with the script @file{astscript-psf-model-flux-factor}.
-For each star, a single value for the flux factor is saved into a plain text
file.
-In this case, the parameters where the computation of the flux factor is done
are the same to those that were used for creating the stamps.
+@node Overview of the PSF scripts, Invoking astscript-psf-create-select-stars,
PSF construction and subtraction, PSF construction and subtraction
+@subsection Overview of the PSF scripts
+The fundamental ideas of the following methodology are thoroughly described in
Infante-Sainz et al. (2020, @url{https://arxiv.org/abs/1911.01430}).
+Once the PSF has been obtained, it can be scaled to the magnitude of the
point-like sources to subtract.
+We therefore also have a script for this.
+We describe the technical implementation of some scripts developed with the
aim of constructing an empirical PSF, and then, use that PSF for correcting the
scattered light.
+All scripts are independent of each other, meaning this that you are free to
use all of them as you wish (for example only some of them, using them for
other purposes, or running independent parts in parallel).
-@example
-$ asttable catalog.fits | while read -r ra dec mag; do
- astscript-psf-model-flux-factor $IMAGE \
- --psf=psf.fits \
- --mode=wcs \
- --center=$ra,$dec \
- --normradii=$NORMRADII_INNER,$NORMRADII_OUTER \
- --output=fluxfactor-"$ra"_"$dec".txt; done
-@end example
+For constructing the PSF, the basic idea is to crop images of stars (with
other sources masked), scale them properly, and finally stack all of them
together.
+As a consequence, the first step would be to obtain a catalog of stars within
the datasets.
+In general, not all stars are good to construct a PSF.
+For example we don't want contamination from other bright, and nearby objects.
+The first script below is therefore designed for selecting only good candidate
stars in your image.
+It will use different criteria, for example good parallax (where available, to
avoid confusion with galaxies), not near bright stars, axis ratio, etc.
+For more on this script, see @ref{Invoking astscript-psf-create-select-stars}.
-Once the flux factors have been computed, it is possible to scale the PSF to
have the desired flux level.
-Of course, it is not only to scale the PSF flux but also allocate it in the
correct position on the sky to model the star.
-To do this, we use the script @file{astscript-psf-model-scattered-light}.
-It will put the PSF into the coordinates specified by the user with the
appropiate size and flux level.
-For those pixels that there is no PSF (it is normal that the size of the image
is bigger than the size of the PSF), the script sets them automatically to zero.
+Once the catalog of stars is constructed, another script is in charge of
making appropiate stamps of the stars.
+Each stamp is a cropped image of the star with the desired size, normalization
of the flux, and mask of the contaminant objects.
+For more on this script, see @ref{Invoking astscript-psf-create-make-stamp}
+After obtaining a set of star stamps, they can be stacked for obtaining the
combined PSF from many stars (for example with @ref{Stacking operators}).
-@example
-$ asttable catalog.fits | while read -r ra dec mag; do
- fluxfactor=$(cat fluxfactor-"$ra"_"$dec".txt)
- astscript-psf-model-scattered-light $IMAGE \
- --psf=psf.fits \
- --mode=wcs \
- --fluxfactor $fluxfactor \
- --center=$ra,$dec \
- --output=starmodel-"$ra"_"$dec".fits; done
-@end example
+In the combined PSF, the masked background objects of each star's image will
be covered and the signal-to-noise ratio will increase, giving a very nice view
of the ``clean'' PSF.
+However, is usually necessary to obtain different regions of the same PSF from
different stars.
+For example, to construct the far wings it is necessary to consider very
bright stars.
+However, these stars will be saturated in the most inner part, and immediately
outside of the saturation level, they will be deformed due to non-linearity
effects.
+Consequently, fainter stars are necessary for the inner regions.
-After all stars have been modeled using the PSF, the next step is to sum all
of them together.
-In this case, it is necessary to combine them using the operator sum because
each individual star has been flux calibrated, otherwise they would be averaged.
-The result is the complete scattered light field of the image using the PSF:
@file{scattered-light-field.fits}.
+Therefore, you need to repeat the steps above for certain stars (in a certain
magnitude range) to obtain the PSF in certain radial ranges.
+For example, in Infante-Sainz et al. (2020,
@url{https://arxiv.org/abs/1911.01430}), the final PSF was constructed from
three regions (and thus, using stars from three ranges in magnitude).
+In other cases, we even needed four groups of stars!
+Once clean stacks of different parts of the PSF have been constructed through
the steps above, it is therefore necessary to blend them all into one.
+This is done by finding a common radial region in both, and scaling the inner
region by a factor to add with the outer region.
+This is not trivial, therefore, a third script is in charge of it, see
@ref{Invoking astscript-psf-create-junction}.
-@example
-$ astarithmetic starmodel*.fits -g1 \
- $(ls starmodel*.fits | wc -l) sum \
- --output=scattered-light-field.fits
-@end example
+Having constructed the PSF as described above (or by any other procedure), it
can be scaled to the brightness of the various stars in the image to get
subtracted (and thus remove the extended/bright wings; better showing the
background objects of interest).
+Note that the absolute flux of a PSF is meaningless (and in fact, it is
usually normalized to have a total sum of unity!), so it should be scaled.
+We therefore have another script that will calculate the scale
(multiplication) factor of the PSF for each star.
+For more on the scaling script, see @ref{Invoking
astscript-psf-model-flux-factor}.
-This is the final step.
-Once the scattered light field model has been obtained (i.e., each individual
star has been modeled using the PSF and they are in one single image), it is
possible to subtract this model from the original image.
-By doing this the scattered light field is removed.
-This step is done with a simple Arithmetic model to subtract two images.
+Once the flux factor has been computed, a final script is in charge of placing
the scaled PSF over the proper location in the image and subtract it.
+For more on the scaling and positioning script, see @ref{Invoking
astscript-psf-model-scattered-light}.
-@example
-$ astarithmetic $IMAGE --hdu=1 \
- scattered-light-field.fits --hdu=1 \
- - --output subtracted.fits
-@end example
+As mentioned above, in the following sections, each script has its own
documentation and list of options for very detailed customization (if
necessary).
+But before going into them, we continue with a practical tutorial to show the
usage of scripts in practice; see @ref{Tutorial on building the extended PSF}.
-As always, check that each step has been done properly.
-It is normal that on the final image there are residuals and several parts
that have been not properly corrected.
-Take into account that the number of the stars used in this example is small.
-With more stars the PSF would have been better determined.
-Note also that during this process, any mask of contaminant sources have been
applied (either in the creation of the PSF nor in the calibration of the flux
of the PSF).
-This tutorial is intendeed to be a basic example from which the reader can
start to construct the PSF.
-To obtain better results it is necessary to study in details the peculiarities
of the datasets and consider other options that here they have not been used.
-In what follows, all options of each script are described in detail.
-For more on installed scripts please see (see @ref{Installed scripts}).
-@node Invoking astscript-psf-create-select-stars, Invoking
astscript-psf-create-make-stamp, Tutorial on building the extended PSF, PSF
construction and subtraction
+@node Invoking astscript-psf-create-select-stars, Invoking
astscript-psf-create-make-stamp, Overview of the PSF scripts, PSF construction
and subtraction
@subsection Invoking astscript-psf-create-select-stars
This installed script will select good star candidates for constructing a PSF.
It will consider stars within a given range of brightness without nearby
contaminant objects.
- [gnuastro-commits] master cd14a366 44/69: PSF select-stars: including the make check step for this script, (continued)
- [gnuastro-commits] master cd14a366 44/69: PSF select-stars: including the make check step for this script, Mohammad Akhlaghi, 2022/01/26
- [gnuastro-commits] master 85648ac3 45/69: PSF select-stars: The script has been changed for use in general scenario, Mohammad Akhlaghi, 2022/01/26
- [gnuastro-commits] master 0facf365 16/69: Book: adding documentation of 'psf-model-scattered-light' script, Mohammad Akhlaghi, 2022/01/26
- [gnuastro-commits] master b1e3da80 34/69: PSF junction: fixed bug when centering the different parts of the PSF, Mohammad Akhlaghi, 2022/01/26
- [gnuastro-commits] master 7c398567 36/69: Book: Correct the some parts of the making the PSF, Mohammad Akhlaghi, 2022/01/26
- [gnuastro-commits] master b82adcfc 46/69: Book: some essential option was add to astscript-psf-select-stars section, Mohammad Akhlaghi, 2022/01/26
- [gnuastro-commits] master ff3bb03f 38/69: PSF select-stars: removing not used options and polishing the comments, Mohammad Akhlaghi, 2022/01/26
- [gnuastro-commits] master b5525411 48/69: PSF select-stars: delete the value of some options and ask them from the user, Mohammad Akhlaghi, 2022/01/26
- [gnuastro-commits] master 075bc680 58/69: PSF select-stars: remove the stars that have stars., Mohammad Akhlaghi, 2022/01/26
- [gnuastro-commits] master 9aae5f5c 65/69: Book: Edits to the tutorial on creating the extended PSF, Mohammad Akhlaghi, 2022/01/26
- [gnuastro-commits] master b1e2eb54 68/69: Book: Extended PSF tutorial moved to Tutorials chapter,
Mohammad Akhlaghi <=
- [gnuastro-commits] master 9db9e205 11/69: PSF junction: new script for joining two PSF images, Mohammad Akhlaghi, 2022/01/26