[Fast Data Analysis Using C++ and Python]
Date: Sep 2018
Technologies used: C++ 11, C++ Python API, Python 3, Bash scripts, Git
C++ and Python code that was applied to analyse data from the Grow-When-Required Neural Network simulation experiments.
This code repository includes:
Apart from offering superior computational speed of C++, the gaData_module also caches data in the memory, decreasing the number of times text files need to be processed. Tests have shown that using C++ to process data makes the Python scripts execute more than 10 times faster, compared to when text files are loaded and processed by Python alone.
It is assumed that data is organised in various folders, representing different experiments. Each folder contains data files for N runs. A python script specifies which folder to take data from and how many experimental runs to consider.
When a python script loads the gaData_module, a static instance of the module is created in the memory. This allows gaData_module to persist data, making it possible to process text files from individual runs only once and return information to various places in the python script via different function calls. In order to ensure that data from different experiments is not stored and analysed together, gaData_module also keeps track of which folder it loaded text files from and loads new data when the requested folder path changes.
C++ and Python code that was applied to analyse data from the Grow-When-Required Neural Network simulation experiments.
This code repository includes:
- C++ code that is compiled into a python module (gaData_module) and that reads and processes text data files
- Python scripts that request data from gaData_module for various experiment sets and visualise it using pyCreeper
Apart from offering superior computational speed of C++, the gaData_module also caches data in the memory, decreasing the number of times text files need to be processed. Tests have shown that using C++ to process data makes the Python scripts execute more than 10 times faster, compared to when text files are loaded and processed by Python alone.
How it works
It is assumed that data is organised in various folders, representing different experiments. Each folder contains data files for N runs. A python script specifies which folder to take data from and how many experimental runs to consider.When a python script loads the gaData_module, a static instance of the module is created in the memory. This allows gaData_module to persist data, making it possible to process text files from individual runs only once and return information to various places in the python script via different function calls. In order to ensure that data from different experiments is not stored and analysed together, gaData_module also keeps track of which folder it loaded text files from and loads new data when the requested folder path changes.
Comments
[07/06/2020]
[22/08/2019]
[22/08/2019]
[18/08/2019]
{Please enable JavaScript in order to post comments}