core

Fill in a module description here

source

DebugRunner

 DebugRunner (is_async:bool=False)

SequentialRunner is an AbstractRunner implementation. It can be used to run the Pipeline in a sequential manner using a topological sort of provided nodes.

source

DebugRunner

 DebugRunner (is_async:bool=False)

SequentialRunner is an AbstractRunner implementation. It can be used to run the Pipeline in a sequential manner using a topological sort of provided nodes.

# `DebugRunner` has to be used in a different way since `session.run` don't support additional argument, so we are going to use a lower level approach and construct `Runner` and `Pipeline` and `DataCatalog` ourselves.

# Testing Kedro Project: https://github.com/noklam/kedro_gallery/tree/master/kedro-debug-runner-demo
The kedro.ipython extension is already loaded. To reload it, use:
  %reload_ext kedro.ipython
[10/06/22 14:45:20] INFO     Updated path to Kedro project:       __init__.py:54
                             /Users/Nok_Lam_Chan/dev/kedro_galler               
                             y/kedro-debug-runner-demo                          
[10/06/22 14:45:22] INFO     Kedro project                        __init__.py:77
                             kedro_debug_runner_demo                            
                    INFO     Defined global variable 'context',   __init__.py:78
                             'session', 'catalog' and 'pipelines'               
runner = DebugRunner()
default_pipeline = pipelines["__default__"]
run_1 = runner.run(default_pipeline, catalog)
                    INFO     Updated path to Kedro project:       __init__.py:54
                             /Users/Nok_Lam_Chan/dev/kedro_galler               
                             y/kedro-debug-runner-demo                          
[10/06/22 14:45:24] INFO     Kedro project                        __init__.py:77
                             kedro_debug_runner_demo                            
                    INFO     Defined global variable 'context',   __init__.py:78
                             'session', 'catalog' and 'pipelines'               
                    INFO     Loading data from               data_catalog.py:343
                             'example_iris_data'                                
                             (CSVDataSet)...                                    
                    INFO     Loading data from 'parameters'  data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Running node: split:                    node.py:327
                             split_data([example_iris_data,parameter            
                             s]) -> [X_train,X_test,y_train,y_test]             
                    INFO     Saving data to 'X_train'        data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Saving data to 'X_test'         data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Saving data to 'y_train'        data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Saving data to 'y_test'         data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'X_train'     data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'X_test'      data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'y_train'     data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Running node: make_predictions:         node.py:327
                             make_predictions([X_train,X_test,y_trai            
                             n]) -> [y_pred]                                    
                    INFO     Saving data to 'y_pred'         data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'y_pred'      data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'y_test'      data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Running node: report_accuracy:          node.py:327
                             report_accuracy([y_pred,y_test]) ->                
                             None                                               
                    INFO     Model has accuracy of 0.933 on test     nodes.py:74
                             data.                                              
runner = DebugRunner()
default_pipeline = pipelines["__default__"]
run_2 = runner.run(default_pipeline, catalog, dataset_names=["example_iris_data"])
[10/06/22 14:45:27] INFO     Loading data from               data_catalog.py:343
                             'example_iris_data'                                
                             (CSVDataSet)...                                    
                    INFO     Loading data from 'parameters'  data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Running node: split:                    node.py:327
                             split_data([example_iris_data,parameter            
                             s]) -> [X_train,X_test,y_train,y_test]             
                    INFO     Saving data to 'X_train'        data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Saving data to 'X_test'         data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Saving data to 'y_train'        data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Saving data to 'y_test'         data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'X_train'     data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'X_test'      data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'y_train'     data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Running node: make_predictions:         node.py:327
                             make_predictions([X_train,X_test,y_trai            
                             n]) -> [y_pred]                                    
                    INFO     Saving data to 'y_pred'         data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'y_pred'      data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'y_test'      data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Running node: report_accuracy:          node.py:327
                             report_accuracy([y_pred,y_test]) ->                
                             None                                               
                    INFO     Model has accuracy of 0.933 on test     nodes.py:74
                             data.                                              
                    INFO     Loading data from               data_catalog.py:343
                             'example_iris_data'                                
                             (CSVDataSet)...                                    
runner = DebugRunner()
default_pipeline = pipelines["__default__"]
run_3 = runner.run(default_pipeline, catalog, dataset_names=["X_train"]) # Input datasets
[10/06/22 14:46:01] INFO     Loading data from               data_catalog.py:343
                             'example_iris_data'                                
                             (CSVDataSet)...                                    
                    INFO     Loading data from 'parameters'  data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Running node: split:                    node.py:327
                             split_data([example_iris_data,parameter            
                             s]) -> [X_train,X_test,y_train,y_test]             
                    INFO     Saving data to 'X_train'        data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Saving data to 'X_test'         data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Saving data to 'y_train'        data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Saving data to 'y_test'         data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'X_train'     data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'X_test'      data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'y_train'     data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Running node: make_predictions:         node.py:327
                             make_predictions([X_train,X_test,y_trai            
                             n]) -> [y_pred]                                    
                    INFO     Saving data to 'y_pred'         data_catalog.py:382
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'y_pred'      data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Loading data from 'y_test'      data_catalog.py:343
                             (MemoryDataSet)...                                 
                    INFO     Running node: report_accuracy:          node.py:327
                             report_accuracy([y_pred,y_test]) ->                
                             None                                               
                    INFO     Model has accuracy of 0.933 on test     nodes.py:74
                             data.                                              
                    INFO     Loading data from 'X_train'     data_catalog.py:343
                             (MemoryDataSet)...                                 
run_1
{}
run_2
{'example_iris_data':      sepal_length  sepal_width  petal_length  petal_width    species
 0             5.1          3.5           1.4          0.2     setosa
 1             4.9          3.0           1.4          0.2     setosa
 2             4.7          3.2           1.3          0.2     setosa
 3             4.6          3.1           1.5          0.2     setosa
 4             5.0          3.6           1.4          0.2     setosa
 ..            ...          ...           ...          ...        ...
 145           6.7          3.0           5.2          2.3  virginica
 146           6.3          2.5           5.0          1.9  virginica
 147           6.5          3.0           5.2          2.0  virginica
 148           6.2          3.4           5.4          2.3  virginica
 149           5.9          3.0           5.1          1.8  virginica
 
 [150 rows x 5 columns]}
run_3
{'X_train':      sepal_length  sepal_width  petal_length  petal_width
 47            4.6          3.2           1.4          0.2
 3             4.6          3.1           1.5          0.2
 31            5.4          3.4           1.5          0.4
 25            5.0          3.0           1.6          0.2
 15            5.7          4.4           1.5          0.4
 ..            ...          ...           ...          ...
 28            5.2          3.4           1.4          0.2
 78            6.0          2.9           4.5          1.5
 146           6.3          2.5           5.0          1.9
 49            5.0          3.3           1.4          0.2
 94            5.6          2.7           4.2          1.3
 
 [120 rows x 4 columns]}

source

GreedySequentialRunner

 GreedySequentialRunner (is_async:bool=False)

SequentialRunner is an AbstractRunner implementation. It can be used to run the Pipeline in a sequential manner using a topological sort of provided nodes.

source

GreedySequentialRunner

 GreedySequentialRunner (is_async:bool=False)

SequentialRunner is an AbstractRunner implementation. It can be used to run the Pipeline in a sequential manner using a topological sort of provided nodes.