6. IASI Level 1 Product Formats and Dissemination

6. IASI Level 1 Product Formats and Dissemination

A description of the dissemination means for EPS products and formats is provided in the following paragraphs, focusing down on IASI products and their formats.

6.1 EPS products available dissemination means

Note that this section about dissemination means of EPS products in general could be removed when that info is available on the EPS website.

6.1.1 Satellite Direct Broadcast Service

Instrument and ancillary data acquired by the Metop satellites will be broadcast and received by authorised users in real-time via:

• High Resolution Picture Transmission (HRPT) - transmission of data from all Metop instruments in full resolution

The data will be received by local reception stations. It is the responsibility of the user to procure and install a local reception station. Specification documentation for a EUMETSAT-based HRPT Reference User Station is available for information on the EUMETSAT webpage Metop - AHRPT.

The output format of the EUMETSAT HRPT Reference User Station is Level 0 products in the EPS Native format [RD11], [RD23] .

The broadcast data are encrypted. To get authorisation to access the data, users need to register with the EUMETSAT User Services and will receive the data decryption information.

Data from the NOAA payload are also broadcast and received by local users via the HRPT mechanism. For details on the NOAA HRPT system, the reader is referred to the NOAA KLM User's Guide [RD47].

6.1.2 EUMETCast

Global EPS products at different levels will be distributed in near real-time via EUMETSAT's Data Distribution System (EUMETCast). EUMETCast utilises the services of a satellite operator and telecommunications provider to distribute data files using Digital Video Broadcast (DVB) to a wide audience located within the geographical coverage zone which includes most of Europe and certain areas in Africa.

Within the current EUMETCast configuration, the multicast system is based upon a client/server system with the server side implemented at the EUMETCast uplink site (Usingen, Germany) and the client side installed on the individual EUMETCast reception stations. The telecommunications suppliers provide the DVB multicast distribution mechanism. Data/product files are transferred via a dedicated communications line from EUMETSAT to the uplink facility. These files are encoded and transmitted to a geostationary communications satellite for broadcast to user receiving stations. Each receiving station decodes the signal and recreates the data/products according to a defined directory and file name structure. A single reception station can receive any combination of the provided services.

A typical EUMETCast reception station comprises a standard PC with DVB card inserted and a satellite off-set antenna fitted with a digital universal V/H LNB. In addition, users require the multicast client software, which can be obtained via the EUMETSAT User Services.

More detailed information on this service can be found in the EUMETSAT webpage EUMETCast Dissemination Scheme.

Products distributed on EUMETCast can be formatted in a variety of formats, including EPS native format and the WMO formats (BUFR and GRIB).

6.1.3 GTS/RMDCN

A subset of EPS products will be disseminated additionally in near real-time via the Global Telecommunication System (GTS). GTS is the World Meteorological Organization integrated network of point-to-point circuits, and multi-point circuits which interconnect meteorological telecommunication centres. Its purpose is to enable an efficient exchange of meteorological data and products in a timely and reliable way to meet the needs of World, Regional and National Meteorological Centres. The circuits of the GTS are composed of a combination of terrestrial and satellite telecommunication links. Meteorological Telecommunication Centres are responsible for receiving data and relaying them selectively on GTS circuits. The GTS is organised on a three-level basis, namely:

• The Main Telecommunication Network, linking together 3 World meteorological centres and 15 regional telecommunication hubs.
• The Regional Meteorological Telecommunication Networks, consisting of an integrated network of circuits interconnecting meteorological centres in a region, which are complemented by radio broadcasts where necessary. In Europe, the GTS network is supported by the Regional Meteorological Data Communication Network (RMDCN).
• The National Meteorological Telecommunication Networks, which extend the GTS network down to national level.

More detailed information on this service can be found in Section 11 and in the WMO Manual on Codes [RD46].

Products distributed on the GTS are in official WMO formats, namely BUFR or GRIB.

6.1.4 EUMETSAT Data Centre

All EPS products and auxiliary data are normally archived and made available to users from the EUMETSAT Data Centre (formerly known as the UMARF or Archive Services) upon request.

The Data Centre can be accessed through the EUMETSAT webpage EUMETSAT Data Centre. Access is through a Web interface, the Online Ordering Application, through which the users are able to browse and order products, manage their user profile, retrieve products, documentation and software libraries, get help, etc.

The Data Centre features include geographical and time sub-setting and image preview. EPS products archived in the Data Centre can be accessed in a variety of formats, including EPS native format and HDF5.

6.2 IASI products dissemination

Table 6.1 summarises the different dissemination means and formats for all IASI Level 1 products available to users.

Format	Real-Time Direct Broadcast	Near-Real-Time dissemination on EUMETCast (timeliness)	Near-Real-Time dissemination on GTS (timeliness)	EUMETSAT Data Centre retrieval (timeliness)
Metop raw data format	IASI HRPT raw data stream and Metop Admin message	--	--	--
EPS native format	--	--	--	IASI Level 1a IASI Level 1b IASI Level 1c IASI Verification data IASI Engineering data (8-9 h)
HDF5	--	--	--	IASI Level 1c
WMO (BUFR)	--	IASI Level 1c (full product) (2 h 15 min)	IASI Level 1c (366 channel subset only)	--

'Timeliness' refers to the elapsed time between sensing and dissemination.

Table 6.1: Summary of dissemination means and formats for IASI Level 1 products

Real-time broadcast of IASI raw data is not covered in this guide. It is noted though for information that the raw data streams mentioned in the table above indicate what is broadcast by the platform. Depending on the reception system used (i.e., the HRPT/LRPT local reception system), different formats of this raw data stream are produced. This depends on the local reception station provider. For Metop HRPT/LRPT stations, the Reference User Station has been developed to produce EPS Native Level 0 format products.

Although available through the EUMETSAT Data Centre, IASI Level 0 products are not considered as an end-user product, hence they are not addressed in this guide either.

6.2.1 Near-real-time dissemination

The IASI Products disseminated to users in near real-time are:

• IASI Level 1c products, with a timeliness of 2 h 15 min from sensing

The dissemination granularity of the data is 3 minutes for Level 1c.

6.2.2 Archive retrieval

The IASI Level 1 Products available from the EUMETSAT Data Centre are:

• IASI Level 1a
• IASI Level 1b
• IASI Level 1c

The products are archived as full-dump products, but sub-setting capabilities are provided to the user in the retrieval step. The products are available for the users in the EUMETSAT Data Centre 8 to 9 hours after sensing.

6.3 IASI EPS native product formats

6.3.1 The EPS native formats

6.3.1.1 General overview of the EPS generic product format

All products in EPS native format are structured and defined according to an EPS Generic Product Format. This format is not IASI specific. The general product section breakdown is given, and the following sections will focus on how this generic format is further applied to IASI products.

This description is not aimed at supporting the writing of reader software for the IASI or other EPS products, because readers and product extraction tools are already available (see Section 5). The intention of this and the following sections is to provide enough information to be able to use such available tools and to interpret the retrieved information.

For users interested in writing their own product readers for one or several IASI products in EPS native format, we refer them to the detailed format specifications provided in [RD11] and [RD12].

The general structure of the products is broken down in sections, which contain one or more records of different classes. Every single record is accompanied by a Generic Record Header (GRH), which contains the metadata necessary to uniquely identify the record type and occurrence within the product. The following general structure is followed by all EPS products, where all the sections occur always in the given order.

Header Section, containing metadata applicable to the entire product. The header section may contain two records, the Main Product Header Record (MPHR) and the Secondary Product Header Record (SPHR). This is the only section that contains ASCII records, the rest of the product is in binary.

Pointer Section, containing pointer information to navigate within the product. It consists of a series of Internal Pointer Records (IPR), which include pointers to records within the Global Auxiliary Data, Variable Auxiliary Data and Body Sections that follow.

Global Auxiliary Data Section, containing information on the auxiliary data that have been used or produced during the process of the product and applies to the whole length of the product. There can be zero or more records in this section, and they can be of two classes: Global External Auxiliary Data Record (GEADR), containing an ASCII pointer to the source of the auxiliary data used, and Global Internal Auxiliary Data Record (GIADR), containing the auxiliary data used itself.

Variable Auxiliary Data Section, containing information on the auxiliary data that have been used or produced during the process of the product and may vary within a product, but with a frequency in any case less than the measurement data itself. There can be zero or more records in this section, and they can be of two classes: Variable External Auxiliary Data Record (VEADR), containing an ASCII pointer to the source of the auxiliary data used, and Variable Internal Auxiliary Data Record (VIADR), containing the auxiliary data used itself.

Body Section, which is usually the main bulk of the product and contains the raw or processed instrument data and associated information. This section contains time-ordered Measurement Data Records (MDR). A particular type of MDR can occur to indicate the location of an unexpected data gap within any product, the Dummy Measurement Data Record (DMDR).

The format of the MPHR, IPRs, GEADR, VEADR and DMDRs is common to all products, while the other records can be of different formats and contents, and identified as of different sub-classes for different products. Every record consists of a series of fields, which can have different data types. See Appendix C for all possible data types.

It is important to note that GEADR and VEADR records are included in the products to support processing configuration control for EUMETSAT at product level. They point to the name of auxiliary data files used in the processing, but they are not of any interest or use to the end-user for the utilisation of the products.

Two types of records deserve special description, because they are key to navigate within the products, namely the GRH and the IPR. Their format and the meaning of their fields are detailed in Appendix D.

Table 6.2 gives an example of the general structure of the Generic Product Format.

Section	RECORD CLASS	RECORD SUBCLASS	START TIME	STOP TIME
HEADER SECTION	MAIN PRODUCT HEADER RECORD SECONDARY PRODUCT HEADER RECORD		T1 T1	T6 T6
INTERNAL POINTER SECTION	INTERNAL POINTER RECORD (GEADR Subclass A) INTERNAL POINTER RECORD (GEADR Subclass B) INTERNAL POINTER RECORD (GIADR Subclass A) INTERNAL POINTER RECORD (GIADR Subclass B) INTERNAL POINTER RECORD (GIADR Subclass C) INTERNAL POINTER RECORD (VEADR Subclass A) INTERNAL POINTER RECORD (VEADR Subclass B) INTERNAL POINTER RECORD (VEADR Subclass C) INTERNAL POINTER RECORD (VIADR Subclass A) INTERNAL POINTER RECORD (VIADR Subclass B) INTERNAL POINTER RECORD (VIADR Subclass C) INTERNAL POINTER RECORD (MDR Subclass A) INTERNAL POINTER RECORD (MDR Subclass B) INTERNAL POINTER RECORD (MDR DUMMY) INTERNAL POINTER RECORD (MDR Subclass A) INTERNAL POINTER RECORD (MDR Subclass B)		T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1 T1	T6 T6 T6 T6 T6 T6 T6 T6 T6 T6 T6 T6 T6 T6 T6 T6
GLOBAL AUXILIARY DATA SECTION	GLOBAL INTERNAL AUXILIARY DATA RECORD GLOBAL INTERNAL AUXILIARY DATA RECORD GLOBAL INTERNAL AUXILIARY DATA RECORD GLOBAL INTERNAL AUXILIARY DATA RECORD GLOBAL INTERNAL AUXILIARY DATA RECORD	SUBCLASS A SUBCLASS B SUBCLASS A SUBCLASS B SUBCLASS C	T1 T1 T1 T1 T1	T6 T6 T6 T6 T6
VARIABLE AUXILIARY DATA SECTION	VARIABLE INTERNAL AUXILIARY DATA RECORD VARIABLE INTERNAL AUXILIARY DATA RECORD VARIABLE INTERNAL AUXILIARY DATA RECORD VARIABLE INTERNAL AUXILIARY DATA RECORD VARIABLE INTERNAL AUXILIARY DATA RECORD VARIABLE INTERNAL AUXILIARY DATA RECORD VARIABLE INTERNAL AUXILIARY DATA RECORD VARIABLE INTERNAL AUXILIARY DATA RECORD VARIABLE INTERNAL AUXILIARY DATA RECORD VARIABLE INTERNAL AUXILIARY DATA RECORD	SUBCLASS A SUBCLASS B SUBCLASS B SUBCLASS C SUBCLASS C SUBCLASS A SUBCLASS A SUBCLASS A SUBCLASS B SUBCLASS C	T1 T1 T3 T1 T5 T1 T2 T4 T1 T1	T6 T3 T6 T5 T6 T2 T4 T6 T6 T6
BODY SECTION	MEASUREMENT DATA RECORD MEASUREMENT DATA RECORD MEASUREMENT DATA RECORD MEASUREMENT DATA RECORD MEASUREMENT DATA RECORD	SUBCLASS A SUBCLASS B DUMMY SUBCLASS A SUBCLASS B	T1 T2 T3 T4 T5	T2 T3 T4 T5 T6

Table 6.2: Generalised schematic of the generic product format

6.3.1.2 Granularity of the EPS products

The Full EPS product is produced by processing a dump of data. This is the product size used to archive in the EUMETSAT Data Centre.

In addition, the Regional EPS product is a full product that has been passed through a geographical filter. This may happen, for example, during the retrieval of the product from the Data Centre.

Finally, the Product Dissemination Unit (PDU) is the near-real-time dissemination of the full product, and it is typically of 3 minutes duration. A PDU is often referred to as product 'granule'.

The EPS Generic Product Format has been defined to apply to any length of sensing. That means that the same generic format described above applies to a 3-minute duration granule, half an orbit or a full dump of data. The length in time of the product is contained in the MPHR.

6.3.1.3 Product format version control

Every record class and sub-class has an associated record version number contained in its corresponding GRH. In addition, each product has a format version number, which is stored in the MPHR.

6.3.1.4 Product naming convention

File naming convention for EPS products in EPS native format provides a product name that uniquely identifies any product and provides a summary of its contents. The field contents in a product name correspond to those in the MPHR.

<INSTRUMENT_ID>_<PRODUCT_TYPE>_<PROCESSING_LEVEL>_<SPACECRAFT_ID>_
_<SENSING_START>_<SENSING_END>_<PROCESSING_MODE>_<DISPOSITION_MODE>_
_<PROCESSING_ TIME>

Product Name Field / MPHR Field	Description	Size in Characters
INSTRUMENT_ID	Instrument identification	4
PRODUCT_TYPE	Product Type	3
PROCESSING_LEVEL	Processing Level Identification	2
SPACECRAFT_ID	Spacecraft identification	3
SENSING_START	UTC Time of start of Sensing Data	15
SENSING_END	UTC Time of end of Sensing Data	15
PROCESSING_MODE	Identification of the mode of processing	1
DISPOSITION_MODE	Identification of the type of processing	1
PROCESSING_TIME	UTC time at start of processing for the product	15

Table 6.3: EPS product name fields and their correspondence with MPHR fields

For the IASI products, the resulting product file names are as follows:

Product	Product name
IASI Level L1a	IASI_xxx_1A_Mnn_< ...>
IASI Level L1b	IASI_xxx_1B_Mnn_< ...>
IASI Level L1c	IASI_xxx_1C_Mnn_< ...>

Table 6.4: Generic IASI product names

6.3.2 The IASI Level 1 product formats

Records to be found in the IASI Level 1 a/b/c products are:

Record name	Description	Usage	Subclass ID
MPHR	Main Product Header Record	Main product identification details	0
GIADR-QUALITY	Global Internal Auxiliary Data Record - Quality flags	General Product quality information	0
GIADR-SCALEFACTORS	Global Internal Auxiliary Data Record - Scaling factors for the Level 1a/b/c spectra	Scaling factors are needed for the calculation of IASI Level 1 radiance spectra	1
GIADR-ENGINEERING	Global Internal Auxiliary Data Record	Quality parameters for conversion of on board time to UTC	2
MDR-1A	Measurement Data Record Level 1a		0
MDR-1B	Measurement Data Record Level 1b		1
MDR-1C	Measurement Data Record Level 1c		2

Table 6.5: The different records in the IASI L1 products

The IASI Level 1 products are organised as successive scan lines. Each IASI Level 1 product contains one IASI scan line. The IFOVs within one scan line are referenced by the geolocation. The START/STOP timeshare is indicated in the MPHR.

6.3.2.1 IASI Level 1a

The MDR-1A contains the decoded and radiometrically calibrated IIS images for the 30 IASI EFOVs. A radiometric post-calibration has been applied to the IASI spectra. Additionally, the spectral calibration is appended to the spectra, and the geolocation based on the collocation of the IIS with AVHRR/3 is performed.

The occurrence of the different records in the Level 1a products is as follows:

Record	Occurrence
MPHR	Once per product
GIADRs	Once per product
MDR-1A	Once per scan line

Table 6.6: The occurrence of the different records in the IASI L1a product

See Section 10.1 for more details on the contents and format of the IASI Level 1a products.

The occurrence of the different records in the Level 1b full resolution product is as follows:

Record	Occurrence
MPHR	Once per product
GIADRs	Once per product
MDR-1B	Once per scan line

Table 6.7: The occurrence of the different records in the IASI L1b product

See Section 10.2 for more details on the contents and format of the IASI Level 1b products.

6.3.2.3 IASI Level 1c

The occurrence of the different records in the Level 1c full resolution product is as follows:

Record	Occurrence
MPHR	Once per product
GIADRs	Once per product
MDR-1C	Once per scan line

Table 6.8: The occurrence of the different records in the IASI L1c product

See Section 10.3 for more details on the contents and format of the IASI Level 1c products.

6.3.3 Decoding of IASI spectra using scaling factors

The scaling factors to be used for the calculation of the IASI radiance spectra vary according to the interferometer band. They are to be read from the GIADR-SCALEFACTORS records which are detailed in Sections 10.1, 10.2 and 10.3 for IASI 1a, 1b and 1c respectively. The scaling factors are used to decode the spectra via the following algorithm:

FOR numScale=1 to IDefScaleSondNbScale DO
          SF = IDefScaleSondScaleFactor(numScale)
          FOR(chanNb=IDefScaleSondNsfirst(numscale) TO IDefScaleSondNslast(numscale) DO
                    w=chanNb - IDefNsfirst + 1
                    SpectDecoded(w) = Spect(w).10^-SF

where:

Spect = GSmcSpect in the case of Level 1a,
Spect = GS1bSpect in the case of Level 1b,
Spect = GS1cSpect in the case of Level 1c.

For more details on decoding IASI spectra and images, please refer to Section 2.8 of the IASI Level 1 PFS [RD12].

6.3.4 The computation of wavenumbers

The spectral position of the IIS filter function can be calculated based on the Spectral Response Function (SRF), which is provided in the product. The array can contain up to 100 samples. The calculation can be performed as follows:

wavenumber_IIS(k) = IDefIISSrfDWn * (IDefIISSrfNsfirst + k - 2)

The data IDefIISSrfNslast determines the actual size of the SRF description.

6.3.4.1 IASI Level 1a spectra

The wavenumber of Level 1a spectra at sample number k is computed:

wavenumber_1a(k) = IDefSpectrDWn * (IDefNsfirst + k - 2) * fcs(k)

where the spectral calibration function fcs(k) must be computed from information extracted from the IASI OPS spectral database (see section 6.3.4.3).

6.3.4.2 IASI Level 1b/c spectra

IASI Level 1b and 1c spectra are provided with a constant sampling. The wavenumber associated with the sample number k is given by the formula:

wavenumber_1b(k) = IDefSpectDWn1b * (IDefNsfirst1b + k - 2)

The data IDefNslast1b determines the actual size of the spectrum.

6.3.4.3 Interpolation in the sounder spectral database

The spectral calibration function fcs(w,p,CCD) and the IASI instrument spectral response function isrf(w,p,CCD) may be calculated from the sounder spectra database. The fcs() and isrf() depend on the the IFOV (pixel) number (p=1,2,3,4), the corner cube direction (CCD=0,1) and on wavenumber (w(k)).

The spectral calibration function fcs(w,p,CCD) is calculated using the following equation:

fcs(k, p, CCD) = p1 * fcs1 + p2 * fsc2 + p3 * fcs3 + p4 * fsc4

where:

p1 = GIsfPds1, fcs1 = ISdbGridFcs(k, p, CCD, i0, j0)

p2 = GIsfPds2, fcs2 = ISdbGridFcs(k, p, CCD, i0, j0+1)

p3 = GIsfPds3, fcs3 = ISdbGridFcs(k, p, CCD, i0+1, j0+1)

p4 = GIsfPds4, fcs4 = ISdbGridFcs(k, p, CCD, i0+1, j0)

with i0 = GIsfLinOrigin, j0 = GIsfColOrigin

IsdbGridFcs and ISdbGridIsrf are extracted from the IASI spectral database provided to the users, while GIsfPds1, GIsfPds2, GIsfPds3, GIsfPds4, GIsfLinOrigin and GIsfColOrigin are part of the product itself.

The same computations can be used to derive the ISRF() function if ISdbGridIsrf() is used instead of IsdbGridFcs().

This computation provides an under-sampled spectral calibration function (typically sampled every 15 cm^-1). The wavenumber w(k) associated with fcs(k) is given by:

w(k) = IDefSafDWn * (IDefSafNsfirst + k - 2)

This smooth function can now be used to oversample every needed wavenumber. IDefSafDWn and IDefSafNsfirst are part of the IASI spectral database.

6.4 The HDF format

The contents and formats of the individual fields of the IASI Level 1c HDF5 products are the same as for the EPS native format. The organisation of the data is different. Typically, the EPS native format presents each scan and corresponding parameters as one complete sequence, stored in a Measurement Data Record (MDR), which is successively repeated until the whole swath is completed. In conversion to HDF5 the measurement values and associated parameters are grouped into separate arrays.

‘.h5’ for HDF5 formatted products
‘.nat’ for products in the native EPS format