OpenDatasetBase Class

Open Dataset Base Class for inherit.

Construct open datasets.

Inheritance

PublicData

OpenDatasetBase

Constructor

OpenDatasetBase(cols: List[str] | None = None, enable_telemetry: bool = True, **kwargs)

Parameters

Name	Description
cols	list[str] A list of columns names to load from the dataset, defaults to None Default value: None
enable_telemetry	bool Whether to enable telemetry on this dataset, defaults to True Default value: True
kwargs Required	dict args for filter

Methods

get_file_dataset	Get the file dataset for open dataset.
get_tabular_dataset	Initialize AbstractTabularOpenDataset with blob url.
to_pandas_dataframe	To pandas dataframe.
to_spark_dataframe	To spark dataframe.

get_file_dataset

Get the file dataset for open dataset.

get_file_dataset(start_date: datetime = None, end_date: datetime = None, enable_telemetry: bool = True, **kwargs) -> FileDataset

Parameters

Name	Description
cls Required	type current class
start_date Required	datetime start date, defaults to None
end_date Required	datetime end date, defaults to None
enable_telemetry Required	bool enable telemetry or not, defaults to True

Returns

Type	Description
FileDataset	file dataset

get_tabular_dataset

Initialize AbstractTabularOpenDataset with blob url.

get_tabular_dataset(start_date: datetime = None, end_date: datetime = None, cols: List[str] = None, enable_telemetry: bool = True, **kwargs) -> TabularDataset

Parameters

Name	Description
cls Required	type type name of the Open Dataset.
start_date Required	datetime The start date to query inclusively.
end_date Required	datetime The end date to query inclusively.
cols Required	list[str] A list of column names to retrieve. None will get all columns.
enable_telemetry Required	bool Whether to enable telemetry, disabled for UT only.

Returns

Type	Description
TabularDataset	TabularDataset

to_pandas_dataframe

To pandas dataframe.

to_pandas_dataframe() -> DataFrame

to_spark_dataframe

To spark dataframe.

to_spark_dataframe()

Attributes

cols

Get the column name list to retrieve.

data

Get the data of the OpenDataset Object.

id

Get the location ID of the open data.

log_properties

Get log properties.

registry_id

Get the registry ID of this public dataset registered at the backend.

This registry ID is used to get latest metadata like storage location. Expect all public data sub classes to assign _registry_id.

Returns

Type	Description
str	Registry ID string.

time_column_name

Time column name.