数据查找

reki.data_finder.find_local_file(data_type: str, start_time: str | Timestamp | datetime, forecast_time: str | Timedelta = '0', data_level: str | Iterable | None = ('archive', 'storage'), path_type: str = 'local', data_class: str = 'od', config_dir: str | Path = None, obs_time: str | Timestamp = None, debug: bool = False, **kwargs) → Path | None

Find local data path using config files in config dir.

参数:

• data_type – data type, relative path of config file to config_dir without suffix. For example grapes_gfs_gmf/grib2/orig means using config file {config_dir}/grapes_gfs_gmf/grib2/orig.yaml.

• start_time – start time of production. YYYYMMDDHH if str.

• forecast_time – forecast time of production. A string (such as 3h) will be parsed by pd.to_timedelta.

• data_level – data storage level, [“archive”, “runtime”, “storage”, … ], default is ("archive", "storage").

• path_type – path type, [“local”, “storage”, …], for future usage.

• data_class – data class, od means operation systems.

• config_dir – config root directory. If None, use embedded config files in conf directory.

• obs_time – time for observation data.

• debug – show debug info.

• **kwargs – other options needed by path template. All of them will be added into query_vars.

返回:

file path if found or None if not.

返回类型:

Path or None

示例

Find an existing orig grib2 file of CMA-GFS in CMA-PAI HPC.

>>> find_local_file(
...     "cma_gfs_gmf/grib2/orig",
...     start_time="2023122000",
...     forecast_time="3h",
... )
PosixPath('/g1/COMMONDATA/OPER/NWPC/GRAPES_GFS_GMF/Prod-grib/2023122000/ORIG/gmf.gra.2023122000003.grb2')

Find a non-existing orig grib2 file of CMA-GFS.

>>> find_local_file(
...     "cma_gfs_gmf/grib2/orig",
...     start_time="2020032100",
...     forecast_time="1h",
... )
None

Find a grib2 file in storage for CMA-MESO 3km.

>>> find_local_file(
...     "cma_meso_3km/grib2/orig",
...     start_time="2020032100",
...     forecast_time="1h",
...     data_level="storage",
... )
PosixPath('/sstorage1/COMMONDATA/OPER/NWPC/GRAPES_MESO_3KM/Prod-grib/2020032100/ORIG/rmf.hgra.2020032100001.grb2')

Find a grib2 file in storage for CMA-GEPS.

>>> find_local_file(
...     "cma_geps/grib2/orig",
...     start_time="2023032100",
...     forecast_time="3h",
...     data_level="storage",
...     number=1,
... )
PosixPath('/sstorage1/COMMONDATA/OPER/NWPC/GRAPES_GEPS/Prod-grib/2023032100/grib2/gef.gra.001.2023032100003.grb2')

Find postvar ctl file for CMA-TYM in Windows mount storage.

>>> find_local_file(
...     "cma_tym/bin/postvar_ctl",
...     start_time="2021080200",
...     storage_base="M:"
... )
WindowsPath('M:/GRAPES_TYM/Fcst-main/2021080200/post.ctl_2021080200')

内部函数

reki.data_finder._util.find_file(config: Dict, data_level: str | List, start_time: datetime | Timestamp, forecast_time: Timedelta, obs_time: Timedelta | None = None, debug: bool = False, **kwargs) → Path | None

Find a file according to config.

参数:

• config –

  data source config dictionary. An example of config file:

  query:
    system: cma_gfs_gmf
    stream: oper
    type: grib2
    name: modelvar
  
  file_name: 'modelvar{{ time_vars.Year }}{{ time_vars.Month }}{{ time_vars.Day }}{{ time_vars.Hour }}{{ time_vars.Forecast }}.grb2'
  
  paths:
    - type: local
      level: runtime
      path: "/some/path/to/{{ time_vars.Year }}{{ time_vars.Month }}{{ time_vars.Day }}{{ time_vars.Hour }}/data/output/grib2_orig/"
  
    - type: local
      level: storage
      path: '{{ query_vars.storage_base }}/GRAPES_GFS_GMF/Prod-grib/{{ time_vars.Year }}{{ time_vars.Month }}{{ time_vars.Day }}{{ time_vars.Hour }}/MODELVAR'
  

• data_level –

• start_time –

• forecast_time –

• obs_time –

• debug – show debug info

• kwargs –

返回类型:

Path if found, None otherwise

reki.data_finder._config.find_config(config_dir: str | Path, data_type: str, data_class: str = 'od') → Path | None

Find config YAML for some data type by combine config_dir, data_type and “.yaml”.

参数:

• config_dir –

• data_type –

• data_class –

返回:

config file path, or None if not found.

返回类型:

Optional[Path]

示例

Load config file for operation CMA-MESO 3KM grib2 orig product.

>>> find_config("/path/to/config_dir", "cma_meso_3km/grib2/orig", "od")
/path/to/config_dir/od/cma_meso_3km/grib2/orig.yaml

If config YAML file is not found, return None.

>>> find_config("/path/to/config_dir", "no_system/grib2/orig", "od")
None