6. GRAS Product Formats and Dissemination

Table of Contents

6. GRAS Product Formats and Dissemination

A description of the dissemination means for EPS products and formats is provided in the following paragraphs, focusing down on GRAS products and their formats.

6.1 EPS products available dissemination means

Note that this section about dissemination means of EPS products in general could be removed when that info is available on the EPS website.

6.1.1 Satellite Direct Broadcast Service

Instrument and ancillary data acquired by the Metop satellites will be broadcast and received by authorised users in real-time via:

The data will be received by local reception stations. It is the responsibility of the user to procure and install a local reception station. Specification documentation for a EUMETSAT-based HRPT Reference User Station is available for information on the EUMETSAT webpage Metop - AHRPT.

The output format of the EUMETSAT HRPT Reference User Station is Level 0 products in the EPS Native format [RD3].

The broadcast data are encrypted. To get authorisation to access the data, users need to register with the EUMETSAT User Services and will receive the data decryption information.

Data from the NOAA payload are also broadcast and received by local users via the HRPT mechanism. For details on the NOAA HRPT system, the reader is referred to the NOAA KLM User's Guide [SCD5].

6.1.2 EUMETCast

Global EPS products at different processing levels are distributed in near real-time via EUMETSAT's Data Distribution System (EUMETCast). EUMETCast utilises the services of a satellite operator and telecommunications provider to distribute data files using Digital Video Broadcast (DVB) to a wide audience located within the geographical coverage zone which includes most of Europe and certain areas in Africa.

Within the current EUMETCast configuration, the multicast system is based upon a client/server system with the server side implemented at the EUMETCast uplink site (Usingen, Germany) and the client side installed on the individual EUMETCast reception stations. The telecommunications suppliers provide the DVB multicast distribution mechanism. Data/product files are transferred via a dedicated communications line from EUMETSAT to the uplink facility. These files are encoded and transmitted to a geostationary communications satellite for broadcast to user receiving stations. Each receiving station decodes the signal and recreates the data/products according to a defined directory and file name structure. A single reception station can receive any combination of the provided services.

A typical EUMETCast reception station comprises a standard PC with DVB card inserted and a satellite off-set antenna fitted with a digital universal V/H LNB. In addition, users require the multicast client software, which can be obtained via the EUMETSAT User Services.

More detailed information on this service can be found on the EUMETSAT webpage EUMETCast Dissemination Scheme.

Products distributed on EUMETCast can be formatted in a variety of formats, including EPS native format and the WMO formats (e.g. BUFR and GRIB).

6.1.3 GTS/RMDCN

A subset of EPS products is disseminated additionally in near real-time via the Global Telecommunication System (GTS). The GTS is the World Meteorological Organization integrated network of point-to-point circuits and multi-point circuits which interconnect meteorological telecommunication centres. Its purpose is to enable an efficient exchange of meteorological data and products in a timely and reliable way to meet the needs of World, Regional and National Meteorological Centres. The circuits of the GTS are composed of a combination of terrestrial and satellite telecommunication links. Meteorological Telecommunication Centres are responsible for receiving data and relaying them selectively on GTS circuits. The GTS is organised on a three-level basis, namely:

More detailed information on this service can be found on the WMO website www.wmo.int.

Products distributed on the GTS are in official WMO formats, e.g. BUFR or GRIB.

6.1.4 UMARF

All EPS products and auxiliary data are normally archived and made available to users from the EUMETSAT Unified Meteorological Archive and Retrieval Facility (UMARF) upon request.

The UMARF can be accessed through the EUMETSAT webpage Archive Services. Access is through a web interface through which the users are able to browse and order products, manage their user profile, retrieve products, documentation and software libraries, get help, etc.

UMARF features include geographical and time sub-setting and image preview. EPS products archived in the UMARF can be accessed in a variety of formats, including EPS native format and HDF.

6.2 GRAS products dissemination

Table 6.1 summarises the different dissemination means and formats for all GRAS Level 1 products available to users.

    Format Real-Time Direct Broadcast Near-Real-Time dissemination on EUMETCast (timeliness) Near-Real-Time dissemination on GTS (timeliness) UMARF retrieval (timeliness)
    Metop raw data format
    GRAS HRPT raw data stream and Metop Admin message
    --
    --
    --
    EPS native format
    --
    GRAS Level 1b
    (2 h 15 min)
    --
    GRAS Level 1a (8-9 h)
    GRAS Level 1b (8-9 h)
    HDF
    --
    --
    --
    (TBC)
    WMO (BUFR)
    --
    GRAS Level 1b
    (2 h 15 min)

    GRAS Level 1b (thinned)
    (2 h 15 min)
    GRAS Level 1b (thinned)
    (2 h 15 min)
    --

    'Timeliness' refers to the elapsed time between sensing and dissemination.

Table 6.1: Summary of dissemination means and formats for GOME-2 products

Note: The BUFR thinned data on GTS is much smaller in quantity than the full data on EUMETCast.

Real-time broadcast of GRAS raw data is not covered in this guide. It is noted though for information that the raw data streams mentioned in the table above indicate what is broadcast by the platform. Depending on the reception system used (i.e., the HRPT local reception system), different formats of this raw data stream are produced. This depends on the local reception station provider. For Metop HRPT stations, the Reference User Station has been developed to produce EPS Native Level 0 format products.

Although available through the UMARF, GRAS Level 0 products are not considered as an end-user product, hence they are not addressed in this guide either.

6.2.1 Near real-time dissemination

The GRAS Level 1 Products disseminated to users in near real-time are:

The dissemination granularity for Level 1 data covers one to several occultations within a time window of about 3 minutes.

6.2.2 Archive retrieval

The GRAS Level 1 & 2 Products available from the UMARF are:

The products are archived as full-dump products, but sub-setting capabilities are provided by the UMARF to the user in the retrieval step. The products are available for the users in the UMARF 8 to 9 hours after sensing.

6.3 GRAS EPS product format

6.3.1 The EPS native formats

6.3.1.1 General overview of the EPS generic product format

All products in EPS native format are structured and defined according to an EPS Generic Product Format. This format is not GRAS specific. The general product section breakdown is given, and the following sections will focus on how this generic format is further applied to GRAS products.

This description is not aimed at supporting the writing of reader software for the GRAS or other EPS products, because readers and product extraction tools are already available (see Section 5). The intention of this and the following sections is to provide enough information to be able to use such available tools and to interpret the retrieved information.

For users interested in writing their own product readers for one or several GRAS products in EPS native format, we refer them to the detailed format specifications provided in [RD6].

The general structure of the products is broken down in sections, which contain one or more records of different classes. Every single record is accompanied by a Generic Record Header (GRH), which contains the metadata necessary to uniquely identify the record type and occurrence within the product. The following general structure is followed by all EPS products, where all the sections occur always in the given order.

Header Section, containing metadata applicable to the entire product. The header section may contain two records, the Main Product Header Record (MPHR) and the Secondary Product Header Record (SPHR). This is the only section that contains ASCII records; the rest of the product is in binary.

Pointer Section, containing pointer information to navigate within the product. It consists of a series of Internal Pointer Records (IPR), which include pointers to records within the Global Auxiliary Data, Variable Auxiliary Data and Body Sections that follow.

Global Auxiliary Data Section, containing information on the auxiliary data that have been used or produced during the processing of the product and applies to the whole length of the product. There can be zero or more records in this section, and they can be of two classes: Global External Auxiliary Data Record (GEADR), containing an ASCII pointer to the source of the auxiliary data used, and Global Internal Auxiliary Data Record (GIADR), containing the auxiliary data used itself. Note: not used for GRAS.

Variable Auxiliary Data Section, containing information on the auxiliary data that have been used or produced during the process of the product and may vary within a product, but with a frequency in any case less than the measurement data itself. There can be zero or more records in this section, and they can be of two classes: Variable External Auxiliary Data Record (VEADR), containing an ASCII pointer to the source of the auxiliary data used, and Variable Internal Auxiliary Data Record (VIADR), containing the auxiliary data used itself. Note: VEADR not used for GRAS.

Body Section, which is usually the main bulk of the product and contains the raw or processed instrument data and associated information. This section contains time-ordered Measurement Data Records (MDR). A particular type of MDR can occur to indicate the location of an unexpected data gap within any product, the Dummy Measurement Data Record (DMDR).

The format of the MPHR, IPRs, GEADR, VEADR and DMDRs is common to all products, while the other records can be of different formats and contents, and identified as of different sub-classes for different products. Every record consists of a series of fields, which can have different data types. See Appendix C for all possible data types.

It is important to note that GEADR and VEADR records are included in the products to support processing configuration control for EUMETSAT at product level. They point to the name of auxiliary data files used in the processing, but they are not of any interest or use to the end-user for the utilisation of the products.

Two types of records deserve special description because they are key to navigating within the products, namely the GRH and the IPR. Their format and the meaning of their fields are detailed in Appendix D.

Table 6.2 gives an example of the general structure of the Generic Product Format.

    Section RECORD CLASS RECORD SUBCLASS START TIME STOP TIME
    HEADER SECTION
    MAIN PRODUCT HEADER RECORD
    SECONDARY PRODUCT HEADER RECORD
     
    T1
    T1
    T6
    T6
    INTERNAL POINTER SECTION
    INTERNAL POINTER RECORD (GEADR Subclass A)
    INTERNAL POINTER RECORD (GEADR Subclass B)
    INTERNAL POINTER RECORD (GIADR Subclass A)
    INTERNAL POINTER RECORD (GIADR Subclass B)
    INTERNAL POINTER RECORD (GIADR Subclass C)
    INTERNAL POINTER RECORD (VEADR Subclass A)
    INTERNAL POINTER RECORD (VEADR Subclass B)
    INTERNAL POINTER RECORD (VEADR Subclass C)
    INTERNAL POINTER RECORD (VIADR Subclass A)
    INTERNAL POINTER RECORD (VIADR Subclass B)
    INTERNAL POINTER RECORD (VIADR Subclass C)
    INTERNAL POINTER RECORD (MDR Subclass A)
    INTERNAL POINTER RECORD (MDR Subclass B)
    INTERNAL POINTER RECORD (MDR DUMMY)
    INTERNAL POINTER RECORD (MDR Subclass A)
    INTERNAL POINTER RECORD (MDR Subclass B)
     
    T1
    T1
    T1
    T1
    T1
    T1
    T1
    T1
    T1
    T1
    T1
    T1
    T1
    T1
    T1
    T1
    T6
    T6
    T6
    T6
    T6
    T6
    T6
    T6
    T6
    T6
    T6
    T6
    T6
    T6
    T6
    T6
    GLOBAL AUXILIARY DATA SECTION
    GLOBAL INTERNAL AUXILIARY DATA RECORD
    GLOBAL INTERNAL AUXILIARY DATA RECORD
    GLOBAL INTERNAL AUXILIARY DATA RECORD
    GLOBAL INTERNAL AUXILIARY DATA RECORD
    GLOBAL INTERNAL AUXILIARY DATA RECORD
    SUBCLASS A
    SUBCLASS B
    SUBCLASS A
    SUBCLASS B
    SUBCLASS C
    T1
    T1
    T1
    T1
    T1
    T6
    T6
    T6
    T6
    T6
    VARIABLE AUXILIARY DATA SECTION
    VARIABLE INTERNAL AUXILIARY DATA RECORD
    VARIABLE INTERNAL AUXILIARY DATA RECORD
    VARIABLE INTERNAL AUXILIARY DATA RECORD
    VARIABLE INTERNAL AUXILIARY DATA RECORD
    VARIABLE INTERNAL AUXILIARY DATA RECORD
    VARIABLE INTERNAL AUXILIARY DATA RECORD
    VARIABLE INTERNAL AUXILIARY DATA RECORD
    VARIABLE INTERNAL AUXILIARY DATA RECORD
    VARIABLE INTERNAL AUXILIARY DATA RECORD
    VARIABLE INTERNAL AUXILIARY DATA RECORD
    SUBCLASS A
    SUBCLASS B
    SUBCLASS B
    SUBCLASS C
    SUBCLASS C
    SUBCLASS A
    SUBCLASS A
    SUBCLASS A
    SUBCLASS B
    SUBCLASS C
    T1
    T1
    T3
    T1
    T5
    T1
    T2
    T4
    T1
    T1
    T6
    T3
    T6
    T5
    T6
    T2
    T4
    T6
    T6
    T6
    BODY SECTION
    MEASUREMENT DATA RECORD
    MEASUREMENT DATA RECORD
    MEASUREMENT DATA RECORD
    MEASUREMENT DATA RECORD
    MEASUREMENT DATA RECORD
    SUBCLASS A
    SUBCLASS B
    DUMMY
    SUBCLASS A
    SUBCLASS B
    T1
    T2
    T3
    T4
    T5
    T2
    T3
    T4
    T5
    T6

Table 6.2: Generalised schematic of the generic product format

6.3.1.2 Granularity of the EPS products

The Full EPS product is produced by processing a dump of data. This is the product size used to archive in the UMARF.

In addition, the Regional EPS product is a full product that has been passed through a geographical filter. This may happen, for example, during the retrieval of the product from the UMARF.

Finally, the Product Dissemination Unit (PDU) is the near-real-time dissemination of the full product, and is typically of 3 minutes duration. A PDU is often referred to as a product 'granule'.

The EPS Generic Product Format has been defined to apply to any length of sensing. That means that the same generic format described above applies to a 3-minute duration granule, half an orbit or a full dump of data. The length in time of the product is contained in the MPHR.

6.3.1.3 Product format version control

Every record class and sub-class has an associated record version number contained in its corresponding GRH. In addition, each product has a format version number, which is stored in the MPHR.

6.3.1.4 Product naming convention

The file naming convention for EPS products in EPS native format provides a product name that uniquely identifies any product and provides a summary of its contents. The field contents in a product name correspond to those in the MPHR.

<INSTRUMENT_ID>_<PRODUCT_TYPE>_<PROCESSING_LEVEL>_<SPACECRAFT_ID>_
_<SENSING_START>_<SENSING_END>_<PROCESSING_MODE>_<DISPOSITION_MODE>_
_<PROCESSING_ TIME>

    Product Name Field / MPHR Field Description
    Size in Characters
    INSTRUMENT_ID Instrument identification
    4
    PRODUCT_TYPE Product Type
    3
    PROCESSING_LEVEL Processing Level identification
    2
    SPACECRAFT_ID Spacecraft identification
    3
    SENSING_START UTC Time of start of Sensing Data
    15
    SENSING_END UTC Time of end of Sensing Data
    15
    PROCESSING_MODE Identification of the mode of processing
    1
    DISPOSITION_MODE Identification of the type of processing
    1
    PROCESSING_TIME UTC time at start of processing for the product
    15

    Table 6.3: EPS Product name fields and their correspondence with MPHR fields

For the GRAS products, the resulting product file names are as follows:

    Product Product name
    GRAS Level 1a GRAS_xxx_1A_Mnn_< ...>
    GRAS Level 1b GRAS_xxx_1B_Mnn_< ...>

    Table 6.4: Generic GRAS product names

6.3.2 GRAS Level 1a products

GRAS Level 1a products (see [RD5]) contain reformatted raw instrument data along with all supplementary data needed for further processing, including geolocation and quality flags. The full information is available in Section 9.

    Section Record class Record subclasses / Remarks
    Header MPHR  
    SPHR  
    Pointer IPR (One per target class)
    Global Aux Data GEADR None for GRAS
    GIADR None for GRAS
    Variable Aux Data VIADR Occulting GPS NRT orbit arc
    Earth orientation parameters
    Tropospheric delay product
    Ground station clock bias estimate
    Ground station clock offset estimates
    Sounding support measurements
    GPS tracking data
    Occultation table
    GSN quality report
    Metop NRT orbit arc
    Metop attitude data
    Metop clock offset estimates
    Body MDR Information generally on a per PDU basis, see [MDR-1a]
    (Granularity: generally one per PDU)

    Table 6.5: Generalised format of GRAS Level 1a products

Typical product size for one full orbit is about 250 MB.

6.3.3 GRAS Level 1b products

GRAS Level 1b products (see [RD5]) contain information on a per occultation basis, along with auxiliary information such as geolocation, quality flags, etc. The full information is available in Section 10.

    Section Record class Record subclasses / Remarks
    Header MPHR  
    SPHR (same as in Level 1a product)
    Pointer IPR One per target class
    Global Aux Data GEADR as defined [RD3]
    GIADR as defined [RD3]
    Variable Aux Data VIADR GPS orbit arc
    Occulting GPS clock bias estimate
    Tropospheric delay products for fiducial stations
    Clock offset estimates for fiducial stations
    Metop NRT orbit arc
    Metop clock bias estimation
    Earth orientation parameters
    Metop attitude information
    Body MDR Information on a per occultation basis, see [MDR-1b]
    (Granularity: one per occultation)

    Table 6.6: Generalised format of GRAS Level 1b products

Typical product size for one full orbit is about 180 MB.

6.4 The HDF format

The contents and formats of the individual fields of the GRAS-2 Level 1b HDF5 products are the same as for the EPS native format.

Detailed format descriptions are provided in [RD10]. The products retrieved from the UMARF have the same name as the original EPS formatted ones, with the extension appended:

‘.h5’ for HDF5 formatted products
‘.nat’ for products in the native EPS format

Tools to read HDF formats are TBD, but it is intended that the products can be read using standard HDF libraries. For more information on HDF5 formats in general, see the HDF5 webpages.

6.5 The WMO formats

The GRAS Level 1b products available in WMO (BUFR) format are summarised in the table below.

     Product
    Bulletin header
    Originating station
    Descriptor sequence
    GRAS Level 1b
    N/A
    N/A
    3-10-026
    [Descriptor 3-10-026 is the standard for radio occultation data.]
    GRAS Level 1b thinned
    IEGXii (ii from 01 to 89)
    [ii is incremented for successive occultations ending in the same minute.]
    EUMP
    3-10-026

    Table 6.7: GRAS Level 1b products available in WMO (BUFR) format

The full format description of these products is available in the WMO Manual on Codes [SCD4].

The names of the GRAS Level 1b products distributed on EUMETCast are specified in [RD11]. They follow the pattern:

gras_yyyymmdd_hhmmss_metopa_nnnnn_eps_o[_thn].l1_bufr

where:

yyyymmdd stands for the UTC year, month, day of the data end sensing time
hhmmss stands for the UTC hour, minute, second of the data end sensing time
nnnnn is the orbit number
_thn denotes a thinned product

GRAS vertical profiles regularly contain several thousand samples because the instrument samples at a frequency of 50 Hz and an occultation can often last more than 60 seconds. The GTS is currently not able to cope with the resulting large BUFR files, hence a thinned BUFR product is generated and distributed only on GTS. The thinning reduces the number of vertical levels to currently 247, using a simple linear interpolation in log(bending angle). The same thinning algorithm is also used at the GRAS SAF to thin Level 2 products.

There will be one occultation per GTS bulletin, and the GTS data will not be compressed. The full data on EUMETCast may have multiple occultations within a BUFR message and will be compressed.