GOME-2 Products Guide - Product Formats and Dissemination

A description of the dissemination means for EPS products and formats is provided in the following paragraphs, focusing down on GOME-2 products and their formats.

6.1 EPS products available dissemination means

Note that this section about dissemination means of EPS products in general could be removed when that info is available on the EPS website.

6.1.1 Satellite Direct Broadcast Service

Instrument and ancillary data acquired by the Metop satellites will be broadcast and received by authorised users in real-time via:

The data will be received by local reception stations. It is the responsibility of the user to procure and install a local reception station. Specification documentation for a EUMETSAT-based HRPT/LRPT Reference User Station is available for information on the EUMETSAT webpage Metop - AHRPT/LRPT.

The output format of the EUMETSAT HRPT/LRPT Reference User Station is Level 0 products in the EPS Native format [RD3].

The broadcast data are encrypted. To get authorisation to access the data, users need to register with the EUMETSAT User Services and will receive the data decryption information.

Data from the NOAA payload are also broadcast and received by local users via the HRPT mechanism. For details on the NOAA HRPT system, the reader is referred to the NOAA KLM User's Guide [SCD39].

6.1.2 EUMETCast

Global EPS products at different levels will be distributed in near real-time via EUMETSAT's Data Distribution System (EUMETCast). EUMETCast utilises the services of a satellite operator and telecommunications provider to distribute data files using Digital Video Broadcast (DVB) to a wide audience located within the geographical coverage zone which includes most of Europe and certain areas in Africa.

Within the current EUMETCast configuration, the multicast system is based upon a client/server system with the server side implemented at the EUMETCast uplink site (Usingen, Germany) and the client side installed on the individual EUMETCast reception stations. The telecommunications suppliers provide the DVB multicast distribution mechanism. Data/product files are transferred via a dedicated communications line from EUMETSAT to the uplink facility. These files are encoded and transmitted to a geostationary communications satellite for broadcast to user receiving stations. Each receiving station decodes the signal and recreates the data/products according to a defined directory and file name structure. A single reception station can receive any combination of the provided services.

A typical EUMETCast reception station comprises of a standard PC with DVB card inserted and a satellite off-set antenna fitted with a digital universal V/H LNB. In addition, users require the multicast client software, which can be obtained via the EUMETSAT User Services.

Products distributed on EUMETCast can be formatted in a variety of formats, including EPS native format and the WMO formats (BUFR and GRIB).

6.1.3 GTS/RMDCN

A subset of EPS products will be disseminated additionally in near real-time via the Global Telecommunication System (GTS). GTS is the World Meteorological Organization integrated network of point-to-point circuits, and multi-point circuits which interconnect meteorological telecommunication centres. Its purpose is to enable an efficient exchange of meteorological data and products in a timely and reliable way to meet the needs of World, Regional and National Meteorological Centres. The circuits of the GTS are composed of a combination of terrestrial and satellite telecommunication links. Meteorological Telecommunication Centres are responsible for receiving data and relaying them selectively on GTS circuits. The GTS is organised on a three-level basis, namely:

More detailed information on this service can be found on the WMO website www.wmo.int.

Products distributed on the GTS are in official WMO formats, namely BUFR or GRIB.

6.1.4 UMARF

All EPS products and auxiliary data will be archived and available to users from the EUMETSAT Unified Meteorological Archive and Retrieval Facility (UMARF) upon request.

The UMARF can be accessed through the EUMETSAT webpage Archive Services. Access is through a web interface through which the users are able to browse and order products, manage their user profile, retrieve products, documentation and software libraries, get help, etc.

UMARF features include geographical and time sub-setting and image preview. EPS products archived in the UMARF can be accessed in a variety of formats, including EPS native format and HDF5.

6.2 GOME-2 products dissemination

Table 6.1 summarises the different dissemination means and formats for all GOME-2 Level 1 products available to users.

Real-time broadcast of GOME-2 raw data is not covered in this guide. It is noted though for information that the raw data streams mentioned in the table above indicate what is broadcast by the platform. Depending on the reception system used (i.e., the HRPT/LRPT local reception system), different formats of this raw data stream are produced. This depends on the local reception station provider. For Metop HRPT/LRPT stations, the Reference User Station has been developed to produce EPS Native Level 0 format products.

Although available through the UMARF, GOME-2 Level 0 products are not considered as an end-user product, hence they are not addressed in this guide either.

6.2.1 Near-real-time dissemination

6.2.2 Archive retrieval

The products are archived as full-dump products, but sub-setting capabilities are provided by the UMARF to the user in the retrieval step. The products are available for the users in the UMARF 8 to 9 hours after sensing.

6.3 GOME-2 EPS product format

6.3.1 The EPS native formats

All products in EPS native format are structured and defined according to an EPS Generic Product Format. This format is not GOME-2 specific. The general product section breakdown is given, and the following sections will focus on how this generic format is further applied to GOME-2 products.

This description is not aimed at supporting the writing of reader software for the GOME-2 or other EPS products, because readers and product extraction tools are already available. The intention of this and the following sections is to provide enough information to be able to use such available tools and to interpret the retrieved information.

For users interested in writing their own product readers for one or several GOME-2 products in EPS native format, we refer them to the detailed format specifications provided in [RD8], GOME-2 Level 1 Product Format Specification, EPS.MIS.SPE.97232.

The general structure of the products is broken down in sections, which contain one or more records of different classes. Every single record is accompanied by a Generic Record Header (GRH), which contains the metadata necessary to uniquely identify the record type and occurrence within the product. The following general structure is followed by all EPS products, where all the sections occur always in the given order.

Header Section, containing metadata applicable to the entire product. The header section may contain two records, the Main Product Header Record (MPHR) and the Secondary Product Header Record (SPHR). This is the only section that contains ASCII records, the rest of the product is in binary.

Pointer Section, containing pointer information to navigate within the product. It consists of a series of Internal Pointer Records (IPR), which include pointers to records within the Global Auxiliary Data, Variable Auxiliary Data and Body Sections that follow.

Global Auxiliary Data Section, containing information on the auxiliary data that have been used or produced during the process of the product and applies to the whole length of the product. There can be zero or more records in this section, and they can be of two classes: Global External Auxiliary Data Record (GEADR), containing an ASCII pointer to the source of the auxiliary data used, and Global Internal Auxiliary Data Record (GIADR), containing the auxiliary data used itself.

Variable Auxiliary Data Section, containing information on the auxiliary data that have been used or produced during the process of the product and may vary within a product, but with a frequency in any case less than the measurement data itself. There can be zero or more records in this section, and they can be of two classes: Variable External Auxiliary Data Record (VEADR), containing an ASCII pointer to the source of the auxiliary data used, and Variable Internal Auxiliary Data Record (VIADR), containing the auxiliary data used itself.

Body Section, which is usually the main bulk of the product and contains the raw or processed instrument data and associated information. This section contains time-ordered Measurement Data Records (MDR). A particular type of MDR can occur to indicate the location of an unexpected data gap within any product, the Dummy Measurement Data Record (DMDR).

The format of the MPHR, IPRs, GEADR, VEADR and DMDRs is common to all products, while the other records can be of different formats and contents, and identified as of different sub-classes for different products. Every record consists of a series of fields, which can have different data types. See Annex 4 for all possible data types.

It is important to note that GEADR and VEADR records are included in the products to support processing configuration control for EUMETSAT at product level. They point to the name of auxiliary data files used in the processing, but they are not of any interest or use to the end-user for the utilisation of the products.

Two types of records deserve special description, because they are key to navigating within the products, namely the GHR and the IPR. Their format and the meaning of their fields are detailed in Annex 5.

Table 6.2 gives an example of the general structure of the Generic Product Format.

The Full EPS product is produced by processing a dump of data. This is the product size used to archive in the UMARF.

In addition, the Regional EPS product is a full product that has been passed through a geographical filter. This may happen, for example, during the retrieval of the product from the UMARF.

Finally, a Product Dissemination Unit (PDU) is the near-real-time dissemination of the full product, and it is typically of 3 minutes. A PDU is often referred to as a product 'granule'.

The EPS Generic Product Format has been defined to apply to any length of sensing. That means that the same generic format described above applies to a 3-minute duration granule, half an orbit or a full dump of data. The length in time of the product is contained in the MPHR.

Every record class and sub-class has an associated record version number contained in its corresponding GHR. In addition, each product has a format version number, which is stored in the MPHR.

File naming convention for EPS products in EPS native format provides a product name that uniquely identifies any product and provides a summary of its contents. The field contents in a product name correspond to those in the MPHR.

<INSTRUMENT_ID>_<PRODUCT_TYPE>_<PROCESSING_LEVEL>_<SPACECRAFT_ID>_
_<SENSING_START>_<SENSING_END>_<PROCESSING_MODE>_<DISPOSITION_MODE>_
_<PROCESSING_ TIME>

6.3.2 GOME-2 Level 1a products

GOME-2 Level 1a products (see [RD7], GOME-2 Level 1 Product Generation Specification) contain reformatted raw instrument data along with all supplementary data needed for further processing, including geolocation and quality flags.

Section	Record class	Record subclasses / Remarks
Header	MPHR
Header	SPHR
Pointer	IPR	(One per target class)
Global Aux Data	GEADR	Time correlation information Orbit parameters Surface elevation Land/sea mask In-flight calibration parameters Configuration parameters Initialisation parameters Calibration key data Correction factors (for long-term effects)
Global Aux Data	GIADR	Bands: start pixel and number of pixels Calibration steps applied Müller matrix elements Channels: start/end pixel numbers/wavelengths
Variable Aux Data	VIADR	Dark signals PPG correction Etalon correction Spectral calibration parameters Sun mean reference spectrum
Body	MDR	Earth Calibration Sun Moon Other (Granularity: one scan (6 s))

Table 6.5: Generalised format of GOME-2 Level 1a products

Typical product size for one full orbit: 250 MB. The Level 1a product is smaller than the Level 0 product because spectral data from unfinished readouts (hex FFFF) have been removed.

6.3.3 GOME-2 Level 1b products

GOME-2 Level 1b products (see [RD7], GOME-2 Level 1 Product Generation Specification) contain radiometrically and spectrally calibrated (ir)radiances, along with auxiliary information such as geolocation, quality flags, cloud parameters and polarisation information.

Section	Record class	Record subclasses / Remarks
Header	MPHR
Header	SPHR	(same as in Level 1a product)
Pointer	IPR	One per target class
Global Aux Data	GEADR	Surface elevation Configuration parameters Initialisation parameters Calibration key data Transmittance database (for Fresco) Surface reflectance database (for Fresco) TOMS UV reflectance database (for Fresco)
Global Aux Data	GIADR	Channels: start/end pixel numbers/wavelengths Bands: start pixel and number of pixels, start/end wavelengths Calibration steps applied
Variable Aux Data	VIADR	Sun mean reference spectrum
Body	MDR	Earth Calibration Sun Moon (Granularity: one scan (6 s))

Table 6.6: Generalised format of GOME-2 Level 1b products

Typical product size for one full orbit of Level 1b data: 1.0 GB.

Format	Real-Time Direct Broadcast	Near-Real-Time dissemination on EUMETCast (timeliness)	Near-Real-Time dissemination on GTS (timeliness)	UMARF retrieval (timeliness)
Metop raw data format	GOME-2 HRPT raw data stream and Metop Admin message	--	--	--
EPS native format	--	GOME-2 Level 1b (2 h 15 min)	--	GOME-2 Level 1a (8-9 h) GOME-2 Level 1b (8-9 h)
WMO (BUFR)	--	--	--	--

Product Name Field / MPHR Field	Description	Size in Characters
INSTRUMENT_ID	Instrument identification	4
PRODUCT_TYPE	Product Type	3
PROCESSING_LEVEL	Processing Level Identification	2
SPACECRAFT_IUD	Spacecraft identification	3
SENSING_START	UTC Time of start of Sensing Data	15
SENSING_END	UTC Time of end of Sensing Data	15
PROCESSING_MODE	Identification of the mode of processing	1
DISPOSITION_MODE	Identification of the type of processing	1
PROCESSING_TIME	UTC time at start of processing for the product	15

Product	Product name
GOME-2 Level L1a	GOME_xxx_1A_Mnn_< ...>
GOME-2 Level L1b	GOME_xxx_1B_Mnn_< ...>