Update main to v0.1.1
This merge request adds some minor usability improvements as well as bugfixes. Scripts used to run policies are not affected and will be updated in a separate request.
Changes:
- removed dependence on
sifforconvert-to-parquet.py's environment. Instead,run-convert-to-parquetwill automatically source a virtual environment stored in/data/rc/gpfs-policywith the necessary packages and Python interpreter. (Fixes #23 (closed)) - Set the default path for split files to be in a directory with the same name as the log file being split. This directory will be in the same place as the log file. (Fixed #19 (closed))
- The output directory can also now be specified by the user
- Added and improved documentation for each step in the current pipeline from running a policy to converting split logs to parquet
Bugfixes:
-
tlddefaults toNonewhen the path is not in/data/user,/data/project,/home, or/scratch(Fixes #22 (closed))
Edited by Matthew K Defenderfer