Update main to v0.1.1
This merge request adds some minor usability improvements as well as bugfixes. Scripts used to run policies are not affected and will be updated in a separate request.
Changes:
- removed dependence on
sif
forconvert-to-parquet.py
's environment. Instead,run-convert-to-parquet
will automatically source a virtual environment stored in/data/rc/gpfs-policy
with the necessary packages and Python interpreter. (Fixes #23 (closed)) - Set the default path for split files to be in a directory with the same name as the log file being split. This directory will be in the same place as the log file. (Fixed #19 (closed))
- The output directory can also now be specified by the user
- Added and improved documentation for each step in the current pipeline from running a policy to converting split logs to parquet
Bugfixes:
-
tld
defaults toNone
when the path is not in/data/user
,/data/project
,/home
, or/scratch
(Fixes #22 (closed))
Edited by Matthew K Defenderfer