6.
GRAS Product Formats and Dissemination
|
|
|
6.
GRAS Product Formats and Dissemination |
A description of the dissemination means for EPS products and formats is provided in the following paragraphs, focusing down on GRAS products and their formats.
Note that this section about dissemination means of EPS products in general could be removed when that info is available on the EPS website.
Instrument and ancillary data acquired by the Metop satellites will be broadcast and received by authorised users in real-time via:
The data will be received by local reception stations. It is the responsibility of the user to procure and install a local reception station. Specification documentation for a EUMETSAT-based HRPT Reference User Station is available for information on the EUMETSAT webpage Metop - AHRPT.
The output format of the EUMETSAT HRPT Reference User Station is Level 0 products in the EPS Native format [RD3].
The broadcast data are encrypted. To get authorisation to access the data, users need to register with the EUMETSAT User Services and will receive the data decryption information.
Data from the NOAA payload are also broadcast and received by local users via the HRPT mechanism. For details on the NOAA HRPT system, the reader is referred to the NOAA KLM User's Guide [SCD5].
Global EPS products at different processing levels are distributed in near real-time via EUMETSAT's Data Distribution System (EUMETCast). EUMETCast utilises the services of a satellite operator and telecommunications provider to distribute data files using Digital Video Broadcast (DVB) to a wide audience located within the geographical coverage zone which includes most of Europe and certain areas in Africa.
Within the current EUMETCast configuration, the multicast system is based upon a client/server system with the server side implemented at the EUMETCast uplink site (Usingen, Germany) and the client side installed on the individual EUMETCast reception stations. The telecommunications suppliers provide the DVB multicast distribution mechanism. Data/product files are transferred via a dedicated communications line from EUMETSAT to the uplink facility. These files are encoded and transmitted to a geostationary communications satellite for broadcast to user receiving stations. Each receiving station decodes the signal and recreates the data/products according to a defined directory and file name structure. A single reception station can receive any combination of the provided services.
A typical EUMETCast reception station comprises a standard PC with DVB card inserted and a satellite off-set antenna fitted with a digital universal V/H LNB. In addition, users require the multicast client software, which can be obtained via the EUMETSAT User Services.
More detailed information on this service can be found on the EUMETSAT webpage EUMETCast Dissemination Scheme.
Products distributed on EUMETCast can be formatted in a variety of formats, including EPS native format and the WMO formats (e.g. BUFR and GRIB).
A subset of EPS products is disseminated additionally in near real-time via the Global Telecommunication System (GTS). The GTS is the World Meteorological Organization integrated network of point-to-point circuits and multi-point circuits which interconnect meteorological telecommunication centres. Its purpose is to enable an efficient exchange of meteorological data and products in a timely and reliable way to meet the needs of World, Regional and National Meteorological Centres. The circuits of the GTS are composed of a combination of terrestrial and satellite telecommunication links. Meteorological Telecommunication Centres are responsible for receiving data and relaying them selectively on GTS circuits. The GTS is organised on a three-level basis, namely:
More detailed information on this service can be found on the WMO website www.wmo.int.
Products distributed on the GTS are in official WMO formats, e.g. BUFR or GRIB.
All EPS products and auxiliary data are normally archived and made available to users from the EUMETSAT Unified Meteorological Archive and Retrieval Facility (UMARF) upon request.
The UMARF can be accessed through the EUMETSAT webpage Archive Services. Access is through a web interface through which the users are able to browse and order products, manage their user profile, retrieve products, documentation and software libraries, get help, etc.
UMARF features include geographical and time sub-setting and image preview. EPS products archived in the UMARF can be accessed in a variety of formats, including EPS native format and HDF.
Table 6.1 summarises the different dissemination means and formats for all GRAS Level 1 products available to users.
| Format | Real-Time Direct Broadcast | Near-Real-Time dissemination on EUMETCast (timeliness) | Near-Real-Time dissemination on GTS (timeliness) | UMARF retrieval (timeliness) |
| Metop raw data format | GRAS HRPT raw data stream and Metop
Admin message |
-- |
-- |
-- |
| EPS native format | -- |
GRAS Level 1b (2 h 15 min) |
-- |
GRAS Level 1a (8-9 h) GRAS Level 1b (8-9 h) |
| HDF | -- |
-- |
-- |
(TBC) |
| WMO (BUFR) | -- |
GRAS Level 1b (2 h 15 min) GRAS Level 1b (thinned) (2 h 15 min) |
GRAS Level 1b (thinned) (2 h 15 min) |
-- |
'Timeliness' refers to the elapsed time between sensing and dissemination.
Table 6.1: Summary of dissemination means and formats for GOME-2 products
Note: The BUFR thinned data on GTS is much smaller in quantity than the full data on EUMETCast.
Real-time broadcast of GRAS raw data is not covered in this guide. It is noted though for information that the raw data streams mentioned in the table above indicate what is broadcast by the platform. Depending on the reception system used (i.e., the HRPT local reception system), different formats of this raw data stream are produced. This depends on the local reception station provider. For Metop HRPT stations, the Reference User Station has been developed to produce EPS Native Level 0 format products.
Although available through the UMARF, GRAS Level 0 products are not considered as an end-user product, hence they are not addressed in this guide either.
The GRAS Level 1 Products disseminated to users in near real-time are:
The dissemination granularity for Level 1 data covers one to several occultations within a time window of about 3 minutes.
The GRAS Level 1 & 2 Products available from the UMARF are:
The products are archived as full-dump products, but sub-setting capabilities are provided by the UMARF to the user in the retrieval step. The products are available for the users in the UMARF 8 to 9 hours after sensing.
6.3.1.1 General overview of the EPS generic product format
All products in EPS native format are structured and defined according to an EPS Generic Product Format. This format is not GRAS specific. The general product section breakdown is given, and the following sections will focus on how this generic format is further applied to GRAS products.
This description is not aimed at supporting the writing of reader software for the GRAS or other EPS products, because readers and product extraction tools are already available (see Section 5). The intention of this and the following sections is to provide enough information to be able to use such available tools and to interpret the retrieved information.
For users interested in writing their own product readers for one or several GRAS products in EPS native format, we refer them to the detailed format specifications provided in [RD6].
The general structure of the products is broken down in sections, which contain one or more records of different classes. Every single record is accompanied by a Generic Record Header (GRH), which contains the metadata necessary to uniquely identify the record type and occurrence within the product. The following general structure is followed by all EPS products, where all the sections occur always in the given order.
Header Section, containing metadata applicable to the entire product. The header section may contain two records, the Main Product Header Record (MPHR) and the Secondary Product Header Record (SPHR). This is the only section that contains ASCII records; the rest of the product is in binary.
Pointer Section, containing pointer information to navigate within the product. It consists of a series of Internal Pointer Records (IPR), which include pointers to records within the Global Auxiliary Data, Variable Auxiliary Data and Body Sections that follow.
Global Auxiliary Data Section, containing information on the auxiliary data that have been used or produced during the processing of the product and applies to the whole length of the product. There can be zero or more records in this section, and they can be of two classes: Global External Auxiliary Data Record (GEADR), containing an ASCII pointer to the source of the auxiliary data used, and Global Internal Auxiliary Data Record (GIADR), containing the auxiliary data used itself. Note: not used for GRAS.
Variable Auxiliary Data Section, containing information on the auxiliary data that have been used or produced during the process of the product and may vary within a product, but with a frequency in any case less than the measurement data itself. There can be zero or more records in this section, and they can be of two classes: Variable External Auxiliary Data Record (VEADR), containing an ASCII pointer to the source of the auxiliary data used, and Variable Internal Auxiliary Data Record (VIADR), containing the auxiliary data used itself. Note: VEADR not used for GRAS.
Body Section, which is usually the main bulk of the product and contains the raw or processed instrument data and associated information. This section contains time-ordered Measurement Data Records (MDR). A particular type of MDR can occur to indicate the location of an unexpected data gap within any product, the Dummy Measurement Data Record (DMDR).
The format of the MPHR, IPRs, GEADR, VEADR and DMDRs is common to all products, while the other records can be of different formats and contents, and identified as of different sub-classes for different products. Every record consists of a series of fields, which can have different data types. See Appendix C for all possible data types.
It is important to note that GEADR and VEADR records are included in the products to support processing configuration control for EUMETSAT at product level. They point to the name of auxiliary data files used in the processing, but they are not of any interest or use to the end-user for the utilisation of the products.
Two types of records deserve special description because they are key to navigating within the products, namely the GRH and the IPR. Their format and the meaning of their fields are detailed in Appendix D.
Table 6.2 gives an example of the general structure of the Generic Product Format.
| Section | RECORD CLASS | RECORD SUBCLASS | START TIME | STOP TIME | ||||||||||||||||||||||||||||||||||||||||||||||||
| HEADER SECTION |
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||
| INTERNAL POINTER SECTION |
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||
| GLOBAL AUXILIARY DATA SECTION |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||
| VARIABLE AUXILIARY DATA SECTION |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||
| BODY SECTION |
|
|
|
|
Table 6.2: Generalised schematic of the generic product format
6.3.1.2 Granularity of the EPS products
The Full EPS product is produced by processing a dump of data. This is the product size used to archive in the UMARF.
In addition, the Regional EPS product is a full product that has been passed through a geographical filter. This may happen, for example, during the retrieval of the product from the UMARF.
Finally, the Product Dissemination Unit (PDU) is the near-real-time dissemination of the full product, and is typically of 3 minutes duration. A PDU is often referred to as a product 'granule'.
The EPS Generic Product Format has been defined to apply to any length of sensing. That means that the same generic format described above applies to a 3-minute duration granule, half an orbit or a full dump of data. The length in time of the product is contained in the MPHR.
6.3.1.3 Product format version control
Every record class and sub-class has an associated record version number contained in its corresponding GRH. In addition, each product has a format version number, which is stored in the MPHR.
6.3.1.4 Product naming convention
The file naming convention for EPS products in EPS native format provides a product name that uniquely identifies any product and provides a summary of its contents. The field contents in a product name correspond to those in the MPHR.
<INSTRUMENT_ID>_<PRODUCT_TYPE>_<PROCESSING_LEVEL>_<SPACECRAFT_ID>_
_<SENSING_START>_<SENSING_END>_<PROCESSING_MODE>_<DISPOSITION_MODE>_
_<PROCESSING_ TIME>
| Product Name Field / MPHR Field | Description | Size in Characters |
| INSTRUMENT_ID | Instrument identification | 4 |
| PRODUCT_TYPE | Product Type | 3 |
| PROCESSING_LEVEL | Processing Level identification | 2 |
| SPACECRAFT_ID | Spacecraft identification | 3 |
| SENSING_START | UTC Time of start of Sensing Data | 15 |
| SENSING_END | UTC Time of end of Sensing Data | 15 |
| PROCESSING_MODE | Identification of the mode of processing | 1 |
| DISPOSITION_MODE | Identification of the type of processing | 1 |
| PROCESSING_TIME | UTC time at start of processing for the product | 15 |
Table 6.3: EPS Product name fields and their correspondence with MPHR fields
For the GRAS products, the resulting product file names are as follows:
| Product | Product name |
| GRAS Level 1a | GRAS_xxx_1A_Mnn_< ...> |
| GRAS Level 1b | GRAS_xxx_1B_Mnn_< ...> |
Table 6.4: Generic GRAS product names
GRAS Level 1a products (see [RD5]) contain reformatted raw instrument data along with all supplementary data needed for further processing, including geolocation and quality flags. The full information is available in Section 9.
| Section | Record class | Record subclasses / Remarks |
| Header | MPHR | |
| SPHR | ||
| Pointer | IPR | (One per target class) |
| Global Aux Data | GEADR | None for GRAS |
| GIADR | None for GRAS | |
| Variable Aux Data | VIADR | Occulting GPS NRT orbit arc Earth orientation parameters Tropospheric delay product Ground station clock bias estimate Ground station clock offset estimates Sounding support measurements GPS tracking data Occultation table GSN quality report Metop NRT orbit arc Metop attitude data Metop clock offset estimates |
| Body | MDR | Information generally on a per PDU basis,
see [MDR-1a] (Granularity: generally one per PDU) |
Table 6.5: Generalised format of GRAS Level 1a products
Typical product size for one full orbit is about 250 MB.
GRAS Level 1b products (see [RD5]) contain information on a per occultation basis, along with auxiliary information such as geolocation, quality flags, etc. The full information is available in Section 10.
| Section | Record class | Record subclasses / Remarks |
| Header | MPHR | |
| SPHR | (same as in Level 1a product) | |
| Pointer | IPR | One per target class |
| Global Aux Data | GEADR | as defined [RD3] |
| GIADR | as defined [RD3] | |
| Variable Aux Data | VIADR | GPS orbit arc Occulting GPS clock bias estimate Tropospheric delay products for fiducial stations Clock offset estimates for fiducial stations Metop NRT orbit arc Metop clock bias estimation Earth orientation parameters Metop attitude information |
| Body | MDR | Information on a per occultation basis,
see [MDR-1b] (Granularity: one per occultation) |
Table 6.6: Generalised format of GRAS Level 1b products
Typical product size for one full orbit is about 180 MB.
The contents and formats of the individual fields of the GRAS-2 Level 1b HDF5 products are the same as for the EPS native format.
Detailed format descriptions are provided in [RD10]. The products retrieved from the UMARF have the same name as the original EPS formatted ones, with the extension appended:
‘.h5’ for HDF5 formatted products
‘.nat’ for products in the native EPS format
Tools to read HDF formats are TBD, but it is intended that the products can be read using standard HDF libraries. For more information on HDF5 formats in general, see the HDF5 webpages.
The GRAS Level 1b products available in WMO (BUFR) format are summarised in the table below.
Product |
Bulletin header |
Originating station |
Descriptor sequence
|
GRAS Level 1b |
N/A |
N/A |
3-10-026 [Descriptor 3-10-026 is the standard for radio occultation data.] |
GRAS Level 1b thinned |
IEGXii (ii from 01 to 89) [ii is incremented for successive occultations ending in the same minute.] |
EUMP |
3-10-026 |
Table 6.7: GRAS Level 1b products available in WMO (BUFR) format
The full format description of these products is available in the WMO Manual on Codes [SCD4].
The names of the GRAS Level 1b products distributed on EUMETCast are specified in [RD11]. They follow the pattern:
gras_yyyymmdd_hhmmss_metopa_nnnnn_eps_o[_thn].l1_bufr
where:
yyyymmdd stands for the UTC year, month, day of the data end sensing time hhmmss stands for the UTC hour, minute, second of the data end sensing time nnnnn is the orbit number _thn denotes a thinned product
GRAS vertical profiles regularly contain several thousand samples because the instrument samples at a frequency of 50 Hz and an occultation can often last more than 60 seconds. The GTS is currently not able to cope with the resulting large BUFR files, hence a thinned BUFR product is generated and distributed only on GTS. The thinning reduces the number of vertical levels to currently 247, using a simple linear interpolation in log(bending angle). The same thinning algorithm is also used at the GRAS SAF to thin Level 2 products.
There will be one occultation per GTS bulletin, and the GTS data will not be compressed. The full data on EUMETCast may have multiple occultations within a BUFR message and will be compressed.