6.7.1. pilotscope.DataManager.DataManager

class DataManager(config: pilotscope.PilotConfig.PilotConfig)

Bases: object

This class provides methods to interact with a database to store and retrieve data.

__init__(config: pilotscope.PilotConfig.PilotConfig) → None

Initialize the DataManager.

Parameters

config – The configuration of PilotScope.

read_all(table_name)

Read all rows from a given table.

Parameters

table_name – The name of the table to read from.

Returns

All rows in the specified table as a pandas DataFrame.

read_update(table_name)

Read the new rows from a given table since the last visit. This function will also update the last visit record in the database.

Parameters

table_name – The name of the table to read .

Returns

The new rows in the specified table since the last visit as a pandas DataFrame.

save_data(table_name, column_2_value)

Save data to the specified table in the database.

Parameters

• table_name – The name of the table to insert the data into.

• column_2_value – A dictionary of column names and their corresponding values, e.g., {‘id’: 1, ‘name’: ‘John Doe’}

save_data_batch(table_name, column_2_value_list)

Save data batch.

Parameters

• table_name – The name of the table to be saved.

• column_2_value_list – A list of dictionaries, where each dictionary represents one row of data, e.g., [{‘id’: 1, ‘name’: ‘John Doe’}, {‘id’: 2, ‘name’: ‘Jane Doe’}]

remove_table_and_tracker(table_name)

Drop the table and its corresponding tracker which is used to track the last visit id.

Parameters

table_name – The name of the table to be dropped.