[bug #63815] Manual should mention that only essential FITS keywords are in output

[Mohammad Akhlaghi]

Update of bug #63815 (project gnuastro):

                  Status:                    None => In Progress            
             Assigned to:                    None => makhlaghi              
                 Summary: Warp removes essential keys from header => Manual
should mention that only essential FITS keywords are in output

    _______________________________________________________

Follow-up Comment #1:

Thanks For the comment,

All Gnuastro programs only keep essential keywords in their output, it is not
just Warp. But there is a simple solution to this issue that is described
below.

Let me first explain the reason for this behavior: we usually need more than
one program to complete a task and moving around all the non-essential
keywords in between intermediate data products is just an extra overhead and
can cause conflicts. 

This overhead can be significant because in the FITS standard, the keywords
are kept in blocks of 2880 bytes at the start of the file (which translates to
36 lines of 80 bytes: the length of each keyword). When you initially define
the output array in CFITSIO there is only one of these blocks for keywords. As
you add more keywords such that a new block will be necessary, CFITSIO is
forced to shift the whole data array also (because these blocks should be at
the start of the file). This shift can slow-down the write operation on larger
images.

Furthermore, searching for keywords that are not affected by the operation of
each program will also add a lot of overhead. Especially as the number of
keywords increase to very large number of keywords (I have seen files with
thousands of keywords).

Preserving all extra keywords can also potentially add many bugs/annoyances
for:
* Users who may have keywords that are meaningful for any program (making them
worry about loosing information). 
* Developers, who will have to keep track of the keywords that are created
during the operation of a program (sometimes they are not known a-priori: they
depend on the processing).  

All this complexity is really against the Unix philosophy
<https://en.wikipedia.org/wiki/Unix_philosophy>; which has a term explicitly
saying: "Don't clutter output with extraneous information".

Since the Job of the Warp program is to only Warp the pixel grid, any keyword
that is not directly related to the pixel grid is "extraneous".  

*The solution* however is simple (again following the Unix philosopy: "Make
each program do one thing well. To do a new job, build afresh rather than
complicate old programs by adding new 'features'."): 

Gnuastro has the 'astfits' program which you can use (through its '--copykeys'
option) to copy keywords from one file to your final output (possibly after
many operations through any number of programs). 

Here is an example following your original example (just replace 'KEY1',
'KEY2', 'KEY3' with the keywords you want in the end). 


$ astwarp --hdu=1 --rotate=0 -o warped.fits original.fits

$ astfits original.fits --copykeys=KEY1,KEY2,KEY3 \
          --output=warped.fits --outhdu=1


You can even add a "title" key entry for the new keywords you want to preserve
in the final output (of many operations) through the '--write' option of
'astfits' like below.


$ astwarp --hdu=1 --rotate=0 -o warped.fits original.fits

$ astfits warped.fits --write=/,"Pipeline keywords"

$ astfits original.fits --copykeys=KEY1,KEY2,KEY3 \
          --output=warped.fits --outhdu=1


You can curate the keywords in your final output from many separate files and
in any order and any grouping (with different titles) to be more clear for the
final recipients of your pipeline's data products.

For more on the keyword editing features of the 'astfits' program, see
https://www.gnu.org/software/gnuastro/manual/html_node/Keyword-manipulation.html


I hope this helps clarify that this issue is not a bug, but a feature of
Gnuastro ;-). 

However, to avoid confusion for users that are not use to this behavior I am
keeping this bug open to add all the discussion here inside the manual to be
clear for the users. 


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?63815>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/