Nok’s experiment with nbdev and DebugRunner

Demo

source

DebugRunner

 DebugRunner (is_async:bool=False)

SequentialRunner is an AbstractRunner implementation. It can be used to run the Pipeline in a sequential manner using a topological sort of provided nodes.

source

DebugRunner

 DebugRunner (is_async:bool=False)

SequentialRunner is an AbstractRunner implementation. It can be used to run the Pipeline in a sequential manner using a topological sort of provided nodes.

# `DebugRunner` has to be used in a different way since `session.run` don't support additional argument, so we are going to use a lower level approach and construct `Runner` and `Pipeline` and `DataCatalog` ourselves.

# Testing Kedro Project: https://github.com/noklam/kedro_gallery/tree/master/kedro-debug-runner-demo
The kedro.ipython extension is already loaded. To reload it, use:
  %reload_ext kedro.ipython
[10/06/22 15:32:15] INFO     Updated path to Kedro project:                                          __init__.py:54
                             /Users/Nok_Lam_Chan/dev/kedro_gallery/kedro-debug-runner-demo                         

[10/06/22 15:32:17] INFO     Kedro project kedro_debug_runner_demo                                   __init__.py:77

                    INFO     Defined global variable 'context', 'session', 'catalog' and 'pipelines' __init__.py:78

runner = DebugRunner()
default_pipeline = pipelines["__default__"]
run_1 = runner.run(default_pipeline, catalog)
                    INFO     Updated path to Kedro project:                                          __init__.py:54
                             /Users/Nok_Lam_Chan/dev/kedro_gallery/kedro-debug-runner-demo                         

[10/06/22 15:32:19] INFO     Kedro project kedro_debug_runner_demo                                   __init__.py:77

                    INFO     Defined global variable 'context', 'session', 'catalog' and 'pipelines' __init__.py:78

                    INFO     Loading data from 'example_iris_data' (CSVDataSet)...              data_catalog.py:343

                    INFO     Loading data from 'parameters' (MemoryDataSet)...                  data_catalog.py:343

                    INFO     Running node: split: split_data([example_iris_data,parameters]) ->         node.py:327
                             [X_train,X_test,y_train,y_test]                                                       

                    INFO     Saving data to 'X_train' (MemoryDataSet)...                        data_catalog.py:382

                    INFO     Saving data to 'X_test' (MemoryDataSet)...                         data_catalog.py:382

                    INFO     Saving data to 'y_train' (MemoryDataSet)...                        data_catalog.py:382

                    INFO     Saving data to 'y_test' (MemoryDataSet)...                         data_catalog.py:382

                    INFO     Loading data from 'X_train' (MemoryDataSet)...                     data_catalog.py:343

                    INFO     Loading data from 'X_test' (MemoryDataSet)...                      data_catalog.py:343

                    INFO     Loading data from 'y_train' (MemoryDataSet)...                     data_catalog.py:343

                    INFO     Running node: make_predictions: make_predictions([X_train,X_test,y_train]) node.py:327
                             -> [y_pred]                                                                           

                    INFO     Saving data to 'y_pred' (MemoryDataSet)...                         data_catalog.py:382

                    INFO     Loading data from 'y_pred' (MemoryDataSet)...                      data_catalog.py:343

                    INFO     Loading data from 'y_test' (MemoryDataSet)...                      data_catalog.py:343

                    INFO     Running node: report_accuracy: report_accuracy([y_pred,y_test]) -> None    node.py:327

                    INFO     Model has accuracy of 0.933 on test data.                                  nodes.py:74

runner = DebugRunner()
default_pipeline = pipelines["__default__"]
run_2 = runner.run(default_pipeline, catalog, dataset_names=["example_iris_data"])
                    INFO     Loading data from 'example_iris_data' (CSVDataSet)...              data_catalog.py:343

                    INFO     Loading data from 'parameters' (MemoryDataSet)...                  data_catalog.py:343

                    INFO     Running node: split: split_data([example_iris_data,parameters]) ->         node.py:327
                             [X_train,X_test,y_train,y_test]                                                       

                    INFO     Saving data to 'X_train' (MemoryDataSet)...                        data_catalog.py:382

                    INFO     Saving data to 'X_test' (MemoryDataSet)...                         data_catalog.py:382

                    INFO     Saving data to 'y_train' (MemoryDataSet)...                        data_catalog.py:382

                    INFO     Saving data to 'y_test' (MemoryDataSet)...                         data_catalog.py:382

                    INFO     Loading data from 'X_train' (MemoryDataSet)...                     data_catalog.py:343

                    INFO     Loading data from 'X_test' (MemoryDataSet)...                      data_catalog.py:343

                    INFO     Loading data from 'y_train' (MemoryDataSet)...                     data_catalog.py:343

                    INFO     Running node: make_predictions: make_predictions([X_train,X_test,y_train]) node.py:327
                             -> [y_pred]                                                                           

                    INFO     Saving data to 'y_pred' (MemoryDataSet)...                         data_catalog.py:382

                    INFO     Loading data from 'y_pred' (MemoryDataSet)...                      data_catalog.py:343

                    INFO     Loading data from 'y_test' (MemoryDataSet)...                      data_catalog.py:343

                    INFO     Running node: report_accuracy: report_accuracy([y_pred,y_test]) -> None    node.py:327

                    INFO     Model has accuracy of 0.933 on test data.                                  nodes.py:74

                    INFO     Loading data from 'example_iris_data' (CSVDataSet)...              data_catalog.py:343

runner = DebugRunner()
default_pipeline = pipelines["__default__"]
run_3 = runner.run(default_pipeline, catalog, dataset_names=["X_train"]) # Input datasets
                    INFO     Loading data from 'example_iris_data' (CSVDataSet)...              data_catalog.py:343

                    INFO     Loading data from 'parameters' (MemoryDataSet)...                  data_catalog.py:343

                    INFO     Running node: split: split_data([example_iris_data,parameters]) ->         node.py:327
                             [X_train,X_test,y_train,y_test]                                                       

                    INFO     Saving data to 'X_train' (MemoryDataSet)...                        data_catalog.py:382

                    INFO     Saving data to 'X_test' (MemoryDataSet)...                         data_catalog.py:382

                    INFO     Saving data to 'y_train' (MemoryDataSet)...                        data_catalog.py:382

                    INFO     Saving data to 'y_test' (MemoryDataSet)...                         data_catalog.py:382

                    INFO     Loading data from 'X_train' (MemoryDataSet)...                     data_catalog.py:343

                    INFO     Loading data from 'X_test' (MemoryDataSet)...                      data_catalog.py:343

                    INFO     Loading data from 'y_train' (MemoryDataSet)...                     data_catalog.py:343

                    INFO     Running node: make_predictions: make_predictions([X_train,X_test,y_train]) node.py:327
                             -> [y_pred]                                                                           

                    INFO     Saving data to 'y_pred' (MemoryDataSet)...                         data_catalog.py:382

                    INFO     Loading data from 'y_pred' (MemoryDataSet)...                      data_catalog.py:343

                    INFO     Loading data from 'y_test' (MemoryDataSet)...                      data_catalog.py:343

                    INFO     Running node: report_accuracy: report_accuracy([y_pred,y_test]) -> None    node.py:327

                    INFO     Model has accuracy of 0.933 on test data.                                  nodes.py:74

                    INFO     Loading data from 'X_train' (MemoryDataSet)...                     data_catalog.py:343

run_1
{}

run_2
{
    'example_iris_data':      sepal_length  sepal_width  petal_length  petal_width    species
0             5.1          3.5           1.4          0.2     setosa
1             4.9          3.0           1.4          0.2     setosa
2             4.7          3.2           1.3          0.2     setosa
3             4.6          3.1           1.5          0.2     setosa
4             5.0          3.6           1.4          0.2     setosa
..            ...          ...           ...          ...        ...
145           6.7          3.0           5.2          2.3  virginica
146           6.3          2.5           5.0          1.9  virginica
147           6.5          3.0           5.2          2.0  virginica
148           6.2          3.4           5.4          2.3  virginica
149           5.9          3.0           5.1          1.8  virginica

[150 rows x 5 columns]
}

run_3
{
    'X_train':      sepal_length  sepal_width  petal_length  petal_width
47            4.6          3.2           1.4          0.2
3             4.6          3.1           1.5          0.2
31            5.4          3.4           1.5          0.4
25            5.0          3.0           1.6          0.2
15            5.7          4.4           1.5          0.4
..            ...          ...           ...          ...
28            5.2          3.4           1.4          0.2
78            6.0          2.9           4.5          1.5
146           6.3          2.5           5.0          1.9
49            5.0          3.3           1.4          0.2
94            5.6          2.7           4.2          1.3

[120 rows x 4 columns]
}

source

GreedySequentialRunner

 GreedySequentialRunner (is_async:bool=False)

SequentialRunner is an AbstractRunner implementation. It can be used to run the Pipeline in a sequential manner using a topological sort of provided nodes.

source

GreedySequentialRunner

 GreedySequentialRunner (is_async:bool=False)

SequentialRunner is an AbstractRunner implementation. It can be used to run the Pipeline in a sequential manner using a topological sort of provided nodes.