FpackandFunpackUser'sGuide

W.D.Pence,NASA/GSFC
R.Seaman,NOAO
R.LWhite,STScI
14November2013

1.Introduction
Fpackandfunpack1arestandaloneprogramsforcompressinganduncompressingimagesandtablesthatare
storedintheFITS(FlexibleImageTransportSystem)dataformat.Theyareanalogoustothegzipandgunzip
compressionprogramsexceptthattheyareoptimizedforthetypesofastronomicalimagesthatareoftenstored
inFITSformat.
ThecompressedimagesthataregeneratedbyfpackarestoredinaFITSbinarytableinaformatthatis
definedbytheFITStiledimagecompressionconvention(seehttp://fits.gsfc.nasa.gov/fits_registry.html).This
conventionallowseachimagetobedividedintoagridofrectangulartiles(usuallyrowbyrow),andtheneach
tileisindividuallycompressedandstoredinavariablelengtharraycolumnintheFITSbinarytable.FITS
binarytablesarecompressedandstoredbackintoabinarytablethathasaslightlydifferentstructurethanthe
originaltable(seehttp://fits.gsfc.nasa.gov/registry/tiletablecompression/tiletable.pdf)
FpackandfunpackarerelativelysimplewrappersthatcallroutinesintheCFITSIOlibrary
(http://heasarc.gsfc.nasa.gov/fitsio)toreadtheinputFITSfile(s),performthecompressionoruncompression,and
thenwritetheoutputFITSfiles.Currentlysupportedcompressionalgorithmsforimagesare:Rice,GZIP,
Hcompress,andthePLIOIRAFpixellistcompressionalgorithms.TablecolumnscanbecompressedwithRice
orGZIP,dependingonthecolumndatatype.Otheralgorithmsmaybeaddedinthefuture.
Fpackoffersanumberofadvantagesoverusingageneralpurposeprogramlikegzipofbzip2tocompressthe
wholeFITSfile:
1. fpackisoptimizedforastronomicalimagesandtablesandgenerallyachieveshighercompressionratios
andfastercompressionspeeds.
2. ThecompressedfileproducedbyfpackisitselfavalidFITSfilethatcanbedirectlymanipulatedby
othergeneralpurposeFITSsoftware.
3. EachHeaderDataUnit(HDU)ofamultiextensionFITSfileiscompressedseparately,soeachimage
ortablecanbedirectlyaccessedwithouthavingtouncompressthewholeFITS file.
4. FpackcompressesonlythedataandnottheheadersineachHDU,sosoftwarecanreadandwritethe
headerkeywordswithoutanyaddedcomputationaloverhead.
5. Dividingtheimageintotilesbeforecompressionenablesfasteraccesstosmallsectionsoftheimage
becauseonlythosetilescoveringtheareaofinterestneedbeuncompressed.
6. Fpacksupportslosslesscompressionofbothintegerandfloatingpointformatimages,andalsosupports
anew'lossy'compressiontechniqueforfloatingpointimagesthatproducesmuchhighercompression
byreducingtheamountofuncompressable'noise'intheimagewhilestillpreservingthescientific
informationcontentoftheimage.
7. FpackandFunpackautomaticallyupdatetheCHECKSUMkeywordsinthecompressedand
uncompressedfilestohelpverifytheintegrityoftheFITSfiles.
8. Manysoftwareapplications(suchasds9andprogramsthatuseCFITSIOtoreadandwriteFITSfiles)
1ThefpackandfunpackprogramswereoriginallydevelopedbyRobSeaman.WilliamPenceaddedfurther
enhancementsandmaintainstheimagecompressionalgorithmsintheunderlyingCFITSIOlibrary.RickWhite
wrotetheRiceandHcompressalgorithmsthatareusedbyCFITSIO.

candirectlyread(andwrite)thecompressedFITSimageformat,thuseliminatingtheneedtocreatean
uncompressedversionofthefile.

2.Obtainingandbuildingfpackandfunpack
ThelatestversionsofthefpackandfunpackCsourcecodeareincludedintheCFITSIOsourcefile
distributionsavailableathttp://heasarc.gsfc.nasa.gov/fitsio. Binaryexecutablesforafewcommon
platformsarealsoavailablefromhttp://heasarc.gsfc.nasa.gov/fitsio/fpack.ThisPDFusersguideisavailable
frombothlocations.
TobuildthesoftwareonlinuxandMacOSsystems,firstdownloadandbuildtheCFITSIOlibraryby
followingtheinstructionsintheincludedReadmefile.Thenenterthecommands
makefpack
makefunpack
inthatdirectorytocreatethefpackandfunpackexecutablefiles.
OnWindowsPCs,onecanbuildthefpackandfunpackprogramsusingtheVisualC++compilerwiththe
followingcommandlines(afterfirstbuildingtheCFITSIOlibraryfollowingtheinstructionsinthe
README.win32file):
> cl /MD fpack.c fpackutil.c cfitsio.lib /link setargv.obj
> cl /MD funpack.c fpackutil.c cfitsio.lib /link setargv.obj

Note:the"/linksetargv.obj"argumentisoptional.Itenablessupportforwildcardcharacterswhenspecifyingthe
namesoftheinputfilestofpackandfunpack.

3.fpackcompressionoptions
OnecanspecifyhowtheinputFITSimagesandtablesaretobecompressedbyfpackin2ways:
1. Thecompressionoptionsmaybespecifiedonthefpackcommandline.Theseglobaloptionswillbe
usedbydefaultwhencompressingalltheinputfiles.
2. 'Compressiondirective'keywordsmaybeaddedtotheheaderofanyHDUtospecifyhowthatHDU
istobecompressed.Thesekeywordswilloverridetheglobaloptionsthatwerespecifiedonthe
fpackcommandlineandcanbeusedtocustomizethecompressiononanHDUbyHDUbasis.
Eachofthesemethodsaredescribedinthefollowingsubsections.

3.1fpackcommandlinesyntax
Thefpackcommandlinehasthefollowingform:
fpack [OPTIONS]... [FILES]...
Variouscompressionoptionsmaybespecifiedpriortothelistoffilestobecompressed.Thenamesofthe
filestobecompressedmaycontaintheusualwildcardcharactersthatwillbeexpandedbytheUnixorWindows
shell.TheinputFITSfilecanalsobepipedintofpackonthestdinfilestreambyspecifyingahyphen('')as
thefilename.
A.Compressionalgorithmoptions:

r
h
gorg1

Rice[default],or
Hcompress,or
GZIP(pertile),or

g2
GZIP(pertile)afterfirstshufflingthebytesinthepixelvaluesintodecreasingorder
ofsignificance(firstbyteofeverypixel,followedby2 ndbyte,etc.)
p
IRAFpixellistcompressionalgorithm.Thiscanonlybeappliedtoimageswhose
pixelvaluesalllieintherange0to224(16777216).

d
tilesarenotcompressed(debuggingmode)

B.Imagetilepatternoptions:
WhenusingtheRice,GZIP,orPLIOcompressionalgorithms,thedefaultactionistotreateachrowof
theimageasatile(i.e.,thetilesareonedimensionandcontainNAXIS1pixels).TheHcompress
algorithmisinherently2dimensionalinnature,thereforethedefaultistouse16rowsoftheimageper
tile,exceptifthiswouldcausethelasttileoftheimagetoonlycontainasmallnumberofrows,inwhich
casethetilesizeisadjustedslightlysothatthelasttileismorenearlyequalinsizetotheothertiles.The
defaulttilesizecanbeoverriddenwithoneofthefollowingcommandlineoptions:
wcompressthewholeimageasasinglelargetile
t<axes>commaseparatedlistoftiledimensions(e.g.,t200,100willproducetiles
thatare200pixelswide(intheNAXIS1dimension)by100pixelshigh
C.Floatingpointimagecompressionoptions:
FloatingpointformatFITSimages(withBITPIX=32or64)areusuallycompressedbyfirstlinearly
scalingthefloatingpointvaluesintoquantizedintegerlevels,andthentheintegervaluesarecompressed
usingtheRicealgorithmbydefault.Thequantizationfactorisspecificwiththeqoption:
q<level>(default=4.0;valueof0.0meansdonotquantizeandlosslesslycompresswithGZIP)
PositiveqvaluesspecifyhowfinelythequantizedlevelsarespacedrelativetothemeasuredR.M.S
noiseinthetileoftheimagethatisbeingcompressed.Thedefaultqvalueof4.0meansthatthespacing
betweenquantizedlevelsisequaltotheRMSnoisevaluedividedby4.Largerqvalueswillresultin
morefinelyspacedlevels(andthusthequantizedvaluesmorecloselymatchtheoriginalfloatingpoint
values)butthiswillresultinlesscompressionoftheimage(asshowninTable2).Refertothelater
sectiononCompressionversusNoiseforamoredetaileddiscussiononthetheoryandpracticeof
imagequantization.
Insomecasesitmaybedesirabletospecifytheexactspacingbetweenthequantizedlevels(i.e.,not
relativetothemeasurednoise),forexample,sothatallthetilesinaparticularimage,oralltheimagesin
adataset,arequantizedbyexactlythesamefactor,Thiscanbedonebyspecifyingthenegativeofthe
desiredqvalue.Inthiscase,smallerabsolutevaluesofq(e.g.,.001insteadof.01),willmoreclosely
preservetheoriginalpixelvaluesattheexpenseofaloweramountofimagecompression.TheT
option(describedbelow)canbeusedtoestimatethenoiselevelintheimageasanaidinestimatingan
appropriateabsoluteqvalue.
Quantizationisinherentlya'lossy'compressionprocessbecausetheoriginalpixelvaluesarenotexactly
preserved.Whenappliedjudiciously,nosignificantscientificinformationislostandmuchhigher
compressionwillbeachieved.However,onecanturnoffthequantizationprocessandlosslessly
compressthefloatingpointvaluesbyspecifyingaqvalueof0.0.OnemustalsospecifytheGZIP
compressionalgorithminthiscase(withtheg1org2option).
Whenthefloatingpointpixelsarequantized,aditheringoptionisalsousuallyapplied.Thisservesto

randomizethequantizedvaluesandhelpstominimizeanysystematicbiasesinthemeasuredpositions
orintensitiesofobjectsinthequantizedimage.Bydefault,alltheimagepixelsaredithered,but2other
ditheringoptionsmaybespecified:
qz<level>onlyditherthenonzeropixels;thezerovalueswillbeexactlypreserved
q0<level>donotditheranyofthepixelvalues
Ifditheringisturnedoffwithq0,,thentheqlevelshouldbeincreasedbyaboutafactorof4inorderto
preservethesamelevelofscientificprecisionintheimageaswhenditheringisperformed.
Finally,theditheringoptionalsorequiresthataninitialrandomnumberseedvaluebedefined,withan
integervaluerangingfrom1to10000.Bydefault,theinitialseedvalueisrandomlychosenbasedon
thecurrentsystemclocktime.Thismethodhelpstoensurethatthesameditheringpatternisnotapplied
tosuccessiveimage,whichwouldnotbedesirablewhenaddingorsubtracting2compressedimages
(otherwisethebenefitsofditheringwillbelost).Notethatwhenusingthismethodforcomputingthe
seedvalue,thecompressedimagepixelvalueswillbeslightlydifferenteachtimetheimageis
compressed.
There2alsootherwaystospecifyrandomnumberseedvalue:
qt<level>computetheseedfromthechecksumofthefirsttileofimagepixels
qN<level>whereNisanintegerbetween1and10000useseedN(e.g.,q30104).
Onecancombinethese2optionswiththe'z'ditheringoptionbyspecifying'qzt'or'qzN'.
D.Optionsrelatedtolossycompressionofintegerimages:
i2fThisoption(whichstandsforintegertofloat)forcesfpacktointernallyconvertimageswith
integerpixelvaluesintofloatingpointpixels,whicharethencompressedusingthequantizationmethod
thatisnormallyusedforactualfloatingpointFITSimages.Thislossycompressionmethodmay
achievehighercompression,withoutsignificantlossofinformation,especiallyforlongexposure
imagesthathaverelativelylargenumberofcountsperpixel(andhencealargeamountofPoissonian
noise).Thisparameterwillbeignorediftheamountofnoiseintheimageislessthantheabsoluteand
relativenoisethresholdssetbythen3minandn3ratioparametersdescribedbelow.
n3min<noise>(defaultvalue=6.)Thisparameterisusedinconjunctionwiththei2fparameterto
specifytheminimumallowedvalueoftheRMSbackgroundnoiseintheimageinorderforthei2f
parametertotakeeffect.Iftheimagehaslessnoise,thenthei2fparameterisignoredandtheinteger
imageislosslesslycompressed.
n3ratio<ratio>(defaultvalue=2.0)Thisparameterisusedinconjunctionwiththei2fparameterto
specifytheminimumallowedratiooftheRMSbackgroundnoiseintheimagedividedbytheq
parametervalue.Iftheimagehaslessnoise,thenthei2fparameterisignoredandtheintegerimageis
losslesslycompressed.
n<noise>Thisrarelyusedparameter(thei2fparameteroffersabettercompressionmethodinmany
cases)rescalesthepixelvaluesinapreviouslyscaledimagetoimprovethecompressionratioby

reducingtheR.M.S.noiseintheimage.ThisoptionisintendedforusewithFITSimagesthatuse
scaledintegerstorepresentfloatingpointpixelvalues,andinwhichthescalingwaschosensothatthe
rangeofthescaledintegervaluescoverstheentireallowedrangeforthatintegerdatatype(e.g.,32768
to+32767for16bitintegersand2147483648to+2147483647for32bitintegers).Whenimagesare
scaledinthismanner,themeasuredR.M.S.noiseintheintegerimagesistypicallysolargethatthey
cannotbeeffectivelylosslesslycompressed.ThisnoptionrescalesthepixelvaluessothattheR.M.S.
noisewillbeequaltothespecifiedvalue.Appropriatevaluesofnwilllikelybeintherangefrom2(for
lowprecisionandthehighcompression)to16(forthehighprecisionandlowercompression).Users
shouldreadthesectiononcompressingfloatingpointimages,forguidelinesonchoosinganappropriate
valuefornthatdoesnotlosesignificantinformationintheimage.
s<scale>ScalefactorforlossycompressionwhenusingHcompress.Thedefaultvalueis0.0which
implieslosslesscompression.PositivescalevaluesareinterpretedasrelativetotheR.M.S.noiseinthe
image.Forreference,scalevaluesof1.0,4.0,and10.0willtypicallyproducecompressionfactorsof
about4,10,and25,respectively,whenappliedto16bitintegerimages.Insomeinstancesitmaybe
desirabletospecifytheexactscalevalue(notrelativetothemeasurednoise),sothatallthetilesinthe
image,andalltheimagesinadataset,arecompressedwiththeidenticalscalevalue,regardlessofslight
variationsinthemeasurednoiselevelbetweentiles.Thisisdonebyspecifyingthenegativeofthe
desiredvalue(e.g.30.,whichwouldbeequivalenttospecifyingascalevalueof2.0inanimagethat
hasRMSnoise=15.).
Itisimportanttorealizethatthisoptionachievesthehighcompressionratiosattheexpenseofnot
exactlypreservingtheoriginalpixelvaluesintheimage.Usersshouldcarefullyevaluatethe
compressedimages(e.g.,byuncompressingthemwithfunpack)tomakesurethatanyessential
informationintheimagehasnotbeenlost.

E.ParametersforcompressingFITSbinarytableextensions(experimental)
table Thisturnsonthetablecompressionfeatureinfpack(imagesarealsocompressed)
tableonlyThisturnsonthetablecompressionfeatureinfpackandturnsoffimagecompression.
F.Parametersthataffecttheinputandoutputfiles:
Thecompressedoutputfilenameisusuallyconstructedbyappending.fztotheinputfilename,and
theinputfileisnotdeleted,butthisbehaviormaybemodifiedwiththefollowingparameters:
F

forcetheinputfiletobeoverwrittenbythecompressedfilewiththesamename.Thisisonly
allowedwhenalosslesscompressionalgorithmisused.
Ddeletetheinputfileaftercreatingthecompressedoutputfile.
YsuppressthepromptstoconfirmtheForDoptions
SoutputthecompressedFITSfiletotheSTDOUTstream(tobepipedtoanothertask)
G.Othermiscellaneousparameters:
v
L
C
H
V
T

verbosemode;listeachfileasitisprocessed
listalltheextensionsinalltheinput,files.Nocompressionisperformed.
don'tupdateFITSchecksumkeywords
displayasummaryhelpfilethatdescribestheavailablefpackoptions
displaytheprogramversionnumber
produceareportthatcomparestheimagecompressionratioandthecompressionand

uncompressiontimesforeachofthemaincompressionalgorithms.Theinputfile
remainsunchangedandisnotcompressed.WhenusedwithBETAtableoption,the
reportshowsthecompressionratioforeachcolumnaswellasforthetableasawhole.
Theinputfileremainsunchangedandnocompressedoutputfileisproduced.
Theformatofthisreportisshownintheappendix.
R<filename>savethecomparisontestreport(producedbytheToption)tothenamedASCIIfile.

3.2.funpackcommandlineparameters
Funpacksharesmanyofthesameparametersasfpackasshownbelow:
A.Parametersthataffecttheinputandoutputfiles:
F

forcetheinputfiletobeoverwrittenbytheuncompressedfilewiththesamename.Thisisonly
allowedwhenalosslesscompressionalgorithmisused.
D
deletetheinputfileaftercreatingthecompressedoutputfile.
P<pre>nameoftheuncompressedoutputfileisconstructedbyprependingthe<pre>stringtothe
nameoftheinputfile
O<name>usedtospecifythefullnameoftheuncompressedoutputfile.
S
outputtheuncompressedFITSfiletotheSTDOUTstream(tobepipedtoanothertask)
Z
recompresstheunpackedoutputfilewiththehostgzipprogram
B.Othermiscellaneousfunpackparameters:
v
L
C
H
V

verbosemode;listeachfileasitisprocessed
listalltheextensionsinalltheinput,files.Nouncompressionisperformed.
don'tupdateFITSchecksumkeywords
displayasummaryhelpfilethatdescribestheavailablefunpackoptions
displaytheprogramversionnumber

3.3Fpackcompressiondirectivekeywords
OnecanspecifyhoweachHDUinaFITSfileistobecompressbyaddingcompressiondirectivekeywords
totheheaderofthatHDU.Thevalueofthesekeywordswilltakeprecedenceoverwhatevercompression
parameterswherespecifiedonthecommandlinewhenfpackwasexecuted.Dataproviderscanusethese
keywordstocontrolthewayinwhicheachindividualHDUinthefileiscompressed.Thefollowingtablelists
theallowedvaluesforallthecompressiondirectivekeywordsinimageHDUs:
Keyword

Description

AllowedValues

FZALGOR

CompressionAlgorithm 'RICE_1' (default)

'GZIP_1'
'GZIP_2' - bytes are shuffled in order of decreasing significance
before being compressed
'HCOMPRESS_1'
'PLIO_1'
'NONE' the HDU remains uncompressed

FZTILE

Tilingpattern

'ROW' row-by-row tile pattern (default)

'WHOLE' - treat entire image as a single tile
'(n,m)' tile dimensions, for example, '(250,100)'

FZQVALUE

QuantizationFactor

Float value - default = 4; a value of 0 means do not quantize, and

instead losslessly compress the floating-point image (must use GZIP)

FZQMETHD QuantizationMethod

'SUBTRACTIVE_DITHER_1' (default)
'SUBTRACTIVE_DITHER_2' zero-valued pixels are not dithered
'NO_DITHER' turns off dithering completely

FZDTHRSD DitheringSeedValue

'CLOCK' seed is randomly chosen based on the system clock time

'CHECKSUM' seed is calculated from checksum of the first tile
'1' through '10000' specifies which seed value to use

FZI2F

Convertintstofloats?

T, F convert integer images into floats and then quantize?

FZHSCALE

Hcompressscalefactor Float value (default = 0.0 = lossless compression)

CompressiondirectivekeywordsallowedinFITSbinarytables:
Keyword

Description

AllowedValues

FZALGOR

DefaultCompression
Algorithmtobeapplied
toeverycolumn,if
possible

'RICE_1'
'GZIP_1'
'GZIP_2' - bytes are shuffled in order of decreasing significance
before being compressed
'NONE' the HDU remains uncompressed

FZALGn

Compressionalgorithm RICE_1' , 'GZIP_1', or 'GZIP_2'

forcolumnnofthe
If the data type of the column is incompatible with the specified
table.Thisoverrides
algorithm, then a suitable default algorithm will be used instead.
theFZALGORvalue

FZTILELN

RowsperTile

Integer number of rows to be compressed as a group

4.TheoryandPracticeofImageCompression
Thissectionprovidesbackgroundinformationonthetheoryandpracticeofimagecompression.Most
ofthismaterialcomesfromour2papersonimagecompression:
PaperI:Pence,Seaman,andWhite,2009,PASP121,414;preprint:http://arxiv.org/abs/0903.21401
PaperII:Pence,White,&Seaman,PASP122,1096(2010);preprint:http://arxiv.org/abs/1007.1179)

4.1Compressionversusnoise
Whenlosslesslycompressingtypicalastronomicalimages,theamountofcompressionthatcanbeachieved
dependsalmostcompletelyononesimplefactor:theamountofthenoise(ormoreprecisely,theentropy)that
ispresentinthepixelvalues.Thenoise,bydefinition,cannotbecompressed,sothecompressionratioofan
imagewillbeinverselyproportionaltothetotalnumberofnoisebitsintheimage.Asisdiscussedingreater
detailinPaperI,theamountofnoiseinanimagecanbeestimatedfromthemeasuredstandarddeviation
(sigma)ofthepixelsinthe"background"areasoftheimage(e.g.,excludingbrightstarsofotherobjectsinthe
image)whichtypicallycloselyapproximatesaGaussiandistribution.Itcanbeshownthattheaveragenumber
ofnoisebitsperpixelisgivenby:
Nbits=log2(sigma)+1.792
Sincethenoisebitscannotbecompressed,theupperlimitonthecompressionratio,intheidealcasewhereall
theremainingbitsarecompressedtozerosize,issimplygivenbytheratioBITPIX/Nbits(whereBITPIXisthe
numberofbitsineachpixelvalueandiseither8,16,or32inFITSimages).Sincenoactualcompression
algorithmcanachievethistheoreticallimit,inpracticethecompressionratiocanbeexpressedas
R=BITPIX/(Nbits+K)
whereKisanempiricalmeasureoftheefficiencyoftheparticularcompressionalgorithm.FortheRice
algorithm,Khasavalueofabout1.2,andforHcompressitisabout0.9.ThekvalueforGZIPismorevariable
anddependsonthenumberanddistributionofdifferentbitpatternsinthedata,but,typicallyhasavalueof
about4or5intypicalastronomicalimages.
TheamountofPoissoniannoiseinastronomicalimagetypicallyscaleswiththesquarerootofthenumberof
detectedphotons.(Theremaybeothersourcesofnoiseaswell).Thepracticalimplicationofthisfactisthatthe
differenttypesofexposurestakenduringaastronomicalobservingsession(e.g.,biasframes,shortcalibration
exposures,deepskyexposures,andflatfieldimages)allhavedistinctlydifferentamountsofnoiseandhence
willcompressbydifferingamounts.Forexample,thebiasframes,withverylownumberofcounts,will
compressmuchbetterthantheflatfieldimagesorlongexposuresofthenightskythatcontainmuchmorenoise.

4.2.LosslesscompressionofintegerFITSimages.
InPaperI,weexaminedalargesetofdirectimagingCCDexposuresfromNOAOtakenofstarfieldsinthe
nightsky,plustheassociatedcalibrationexposures,tocomparethecompressionspeedsandfilecompression
ratiosforthe3differentgeneralpurposecompressionalgorithmsthatarecurrentlysupportedbyfpack,namely,
Rice,GZIP,andHcompress.WealsocomparedthesetothewidelyusedmethodofcompressingtheentireFITS
filewiththehostlevelGZIPfilecompressionprogram.
ThemeanfilecompressionratiosandtherelativecompressionanduncompressionelapsedCPUtimesfor
these4differentcompressionmethodsareshowninTable1.Thesevaluesarethemeanforall163216bit

integerimagesinthesampledataset;theCPUtimesineachcasearerelativetothosewhenusingtheRice
algorithm.

Table1.CompressionStatisticsfor16bitIntegerImages
Rice

Hcompress

GZIP

HostGZIP

CompressionRatio

2.11

2.18

1.53

1.6

RelativecompressionCPUtime

1.0

2.8

5.6

2.6

RelativeuncompressionCPUtime

1.0

3.1

1.9

0.9

Asshowninthefirstrow,theRiceandHcompressmethodsachievesignificantlylargercompressionofthese
astronomicalimagesthanGZIP.TheGZIPcompressedfilesareonaverageabout1.4timeslargerthantheRice
orHcompressedfiles.Thisdependsslightlyontheamountofnoiseintheimage:theratioisabout1.3forthe
imageswiththemostamountofnoiseandabout1.5fortheleastnoisyimages.Hcompressproducesslightly
bettercompressionthanRice(about3%smaller),butformostapplicationsthissmallgainisnotworththemuch
greaterCPUtimesrequiredtocompressanduncompresstheimageswithHcompress.
ThesecondandthirdrowsofthetableshowthattheRicecompressionalgorithmisgenerallyconsiderably
fasterthanHcompressorGZIP.Notethatthefactorofapproximately2timingdifferencebetweenusingthe
hostlevelGZIPprogramtocompressanimageandusingthetiledimageimplementationofthesameGZIP
algorithmwithinfpack/CFITSIOisduetothefactthatthehostlevelGZIPprogramcanreadandwritethefiles
moreefficientlyassimplecontinuousstreams,whereasthefpackimplementationrequiresthattheinputand
outputfilesbecopiedtoandfromintermediatestoragebuffersinmemory.Asabenchmarkpointofreference,a
Linuxmachinewitha2.4GHzAMDOpteron250dualcoreprocessorcancompressoruncompressa50MB
16bitintegerimagein1secondofCPUtimewhenusingtheRicecompressionalgorithm.
Similartrendsareseenwhencompressing32bitintegerimages,onlythecompressionfactorsthatare
achievedaretypicallytwicethatofa16bitimage,giventhesamenoiselevel.SeePaperIformoredetails.

4.3.CompressionoffloatingpointFITSimages
Itisgenerallynotefficient(nornecessary)tolosslesslycompressfloatingpointformatFITSimages(which

haveBITPIX=32or64).Thisisbecausealargefractionofthebitsinthemantissaoftheimagepixelvalues
areoftenfilledwithuncompressiblenoise,whichseverelyreducesthemaximumpossiblefilecompression
ratios.Notethata32bitfloatingpointvaluecanrepresent67decimalplacesofprecision,whichoftenfar
exceedsthenecessaryprecisionneededtorepresentindividualimagepixelvalues.Forthisreason,fpackusually
quantizesthepixelvaluesinto32bitintegersusingalinearscalingfunction:
integer_value=(floating_point_valueZERO_POINT)/SCALE_FACTOR
Thisarrayofscaledintegersisthencompressedusingoneofthesupportedcompressionalgorithms(thedefault
algorithmisRice).Whentheimageissubsequentlyuncompressed,theintegervaluesareinversescaledto
closely(butnotexactly)reproducetheoriginalfloatingpointpixelvalues.Separatescaleandzeropointvalues
arecomputedforeachtileoftheimage.Theseandotherissuesrelatedtocompressingfloatingpointimagesare
discussedingreaterdetailinaPaperII.
ThevalueofSCALE_FACTORintheabovescalingfunctioncontrolshowcloselytheinversescaledvalues

approximatetheoriginalfloatingpointvalues:decreasingSCALE_FACTORreducesthespacingbetweenthe
quantizedlevelsintheinversescaledvaluesandthusmorecloselyreproducestheoriginalpixelvalues.
However,thisalsomagnifiesthedynamicrangeandthenoiselevelintheintegerarraythatistobecompressed,
whichadverselyaffectstheamountofcompressionthatisachieved.Thus,thereisadirecttradeoffbetween
providingmoreprecisionorachievinggreatercompression.

Onerefinementtothequantizationproceduredescribedaboveistoditherthepixelvaluesbyaddingasmall

amountofrandomnoisetothefloatingpointvaluebeforescalingittoaninteger;thatsamerandomvalueis
thensubtractedwhenconvertingbacktothefloatingpointvalue.Thissubtractiveditheringtechniquehelps
preservelowamplitudesignalsinthequantizedimagethroughaneffectknownasstochasticresonance.This
isespeciallyimportantforpreservingthemeanvalueofthebackgroundskylevelwhenmeasuringthefluxof
faintsourcesintheimage.
ItisnoteasytodirectlycalculateanappropriateSCALE_FACTORvaluetousewithagivenimage,therefore
fpackprovidesinsteadaquantizationparametercalled"q"forspecifyinghowcloselytheinversescaledinteger
pixelvaluesmustapproximatetheoriginalfloatingpointpixelvalues,relativetothemeasurednoisein
backgroundareasintheimage.Theimagepixelvalueswillbequantizedsothatthespacingbetweenthe
adjacentdiscretelevelsisequaltothemeasuredsigmaoftheR.M.S.noiseinthebackgroundregionsofthe
imagedividedbyq.Themaximumdeviationbetweenthepixelvaluesinthecompressedimageandinthe
originalimagewillbehalfthisvalue.Formally,thenumberofnoisebitsthatarepreservedineachpixelvalue
isgivenbylog2(q)+1.792.Forexample,ifq=4,thenthequantizedlevelsarespacedatintervalsofsigma/4
andabout3.8bitsofnoisearepreservedineachpixelvalue.Increasingthevalueofqwillproduce
compressedimagesthatmorecloselyapproximatethepixelvaluesintheoriginalfloatingpointimage,butwill
alsoincreasethesizeofthecompressedimagefile,asshowninTable2.Thethirdcolumninthistableshows
howmuchthebackgroundnoisewillincreaseinthequantizedimages,whichisgivenbysqrt(1+1/(12q 2)).
ThenumericalexperimentsdescribedinPaperIIdemonstratethatthestatisticalnoiseonthemeasured
magnitudesandpositionsoffaintstarsinthequantizedimagescanbeexpectedtoincreasebyaboutthesame
percentageamount.Unlessotherwisespecified,fpackcurrentlyusesadefaultqvalue=4whensubtractive
ditheringisalsoperfomed,orq=16whenditheringisnotperformed.

Table2.CompressionofFloatingPointImages
q

Compression
Ratio

Noiseincrease
%

9.5

4.1

6.5

0.26

5.3

0.07

Usersoffpackareurgedtoperformquantitativetestsonthereownfloatingpointimagedatasetsusing
differentvaluesofqtodeterminetheappropriatevaluefortheirparticularapplication.
Insomesituations,orfortestpurposes,itmaybedesirabletolosslesslycompressfloatingpointFITSimages.
ThiscanbeaccomplishedbyspecifyingtheGZIPcompressionalgorithmwithg,g1,org2(becausetheother
algorithmscanonlycompressintegers)andaqvalueof0(e.g.,fpackg2q0).Thiswillexactlypreserve
everybitinthefloatingpointimageattheexpenseofmuchlowercompressionratiosthangivenbythe

10

quantizationtechnique.Preliminarytestsindicatethattheg2variant(wherehthebytesinthepixelvaluesare
shuffledsothatallthemostsignificantbytesoccurfirst,followedbythenextmostsignificantbytes,andsoon
beforetheentirebytestreamiscompressedwiththeGZIPalgorithm)oftenproducesbettercompressionof
floatingpointimagesthantheplaing1compressionmethod.

4.4.LossycompressionofintegerFITSimages
SomeintegerFITSimagescontaintoomuchnoisetobelosslesslycompressedveryeffectively.InourPaper
I,forexample,Figure3showsacaseofdeepCCDexposureswhichcontain78bitsofnoiseperpixelthat
canonlybelosslesslycompressedbylessthanafactorof2.Justaswithtypicalfloatingpointimages,
removingsomeofthenoisefromtheseintegerimagescanincreasetheimagecompressionratiowithout
significantlydegradingthescientificaccuracyofmeasurementsintheimage.
Fpackhasanoption(specifiedwiththei2fcommandlineparameter)tointernallyconvertintegerFITS
imagesintofloatingpointformat,andthenapplythesamequantizationprocedurethatitusesforactual
floatingpointFITSimages.Thecompressionratiowilldependprimarilyonthespecifiedqquantization
parameter,asshowninTable2,exceptthatthecompressionratiowillonlybeoftheamountshowninthe
table,sincetheoriginal16bitintegerpixelsarethesizeofthe32bitfloatingpointpixels.Notethatifthese
compressedimagesaresubsequentlyuncompressed,theywillhaveafloatingpoint,notinteger,datatype.
Ifanintegerimagecontainstoolittlenoise,thenitdoesnotmakesensetousethislossycompressionoption
becausethecompressedfilewillactuallybelargerthanifthestandardlosslesscompressionmethodwasused.
Topreventthis,thereare2otherfpackparametersthatspecifytheminimumthresholdfortheamountofnoisein
animageforthislossycompressionmethodtobeapplied.Ifthesethresholdsarenotmet,thenthei2foptionis
ignoredandtheintegerimageislosslesslycompressed.Forfurtherinformation,seethedescriptionofthe
n3ratioandn3minfpackparametersinthefollowingsection.

5.CompressionofFITSbinarytables(Experimentalfeature)
AnexperimentalcapabilitytolosslesslycompressFITSbinarytablesisavailableinfpack.Thisusesa
prototypeFITSconvention(http://fits.gsfc.nasa.gov/tiletable.pdf)forcompressingeachcolumnofthetableand
storingthecompressedbytesinavariablelengtharraycolumninthecompressedtable.
Thistablecompressionmethodisinvokedbyaddingthetableortableonlyoptiononthecommandline.
Funpackwillautomaticallyuncompressthesetables(nocommandlineswitchisrequired).
NotethatwhileCFITSIOcandirectlyreadandwriteFITSimagesintheircompressedform,thisiscurrently
notpossibleforFITSbinarytables.OnemustfirstcreatetheuncompressedFITSbinarytablefile,then
compressitusingfpack.Similarly,onemustuncompressthetablewithfunpackbeforeCFITSIOcanreadthe
contentsofthetable.

11

AppendixI:ExampleofthefpackTtestreport
File:ct655046_13.fits
ExtBITPIXDimens.NullsMinMaxMeanSigmaNoise3NbitsMaxR
016(1112,4096)0315032596726679.32.5e+0356.87.62.10
TypeRatioSize(MB)Pk(Sec)UnPkExactElpNCPUNElp1CPU1
Native0.0240.0160.0130.010
RICE1.839.11>4.980.570.55Yes0.0530.0470.0450.040
HCOMP1.859.11>4.921.911.56Yes0.1750.1590.1790.162
GZIP1.359.11>6.733.071.09Yes0.1140.1060.1080.101
NONE0.999.11>9.180.350.31Yes0.0220.0210.0150.013

ThefirstlineofthereportgivesthenameoftheFITSfile;the3 rdlinegivesthefollowingparameters:
Extextensionnumberwithinthefile(zerobased)
BITPIXFITSdatatypeoftheimage(8,16,32,32or64)
Dimensimagedimensions
Nullsnumberofundefinedornullpixelsintheimage
Min,Maxtheminimumandmaximumvaluesintheimage
Meanmeanvalueofallthenonnullpixels
Sigmastandarddeviationofallthenonnullpixels
Noise3ameasureofthenoiseinthebackgroundregionsoftheimage
Nbitsnumberofnoisebitsperpixel=log2(noise3)+1.792
MaxRtheoreticalmaximumpossiblecompressionratio=BITPIX/Nbits

Thisisfollowedbyatablewiththefollowingcolumns:
Typenameofcompressionmethod,ifany
Ratiofilecompressionratio
Sizeuncompressedandcompressedsizesofthefiles,inMB
PktheCPUtimeinsecondstocompresstheimagewithfpack
UnPktheCPUtimeinsecondstouncompresstheimagewithfunpack
Exactisthecompressionlossless(i.e.,doesitexactlypreservethepixelvalues)?
Thefollowing4parametersgivethemeasuredimagereadrates,inunitsofseconds/MB
ElpNelapsedtimetoreadtheentireimagewithasinglesubroutinecall
CPUNCPUtimetoreadtheentireimagewithasinglesubroutinecall
Elp1elapsedtimetoreadthewholeimage,onerowatatime
CPU1CPUtimetoreadthewholeimage,onerowatatime
Therowsinthistablecorrespondtothefollowingcases:
Nativethisjustgivesthereadspeedoftheinputuncompressedimage
RicewhenusingtheRicecompressionalgorithm
HcompwhenusingtheHcompressalgorithm
GZIPwhenusingthegzipalgorithm(withintheFITStiledimagecompressionformat)
NonetheimageissimplytiledandpackedintotheFITStiledimageformat,withoutperforming
anycompressiononthetiles.

12