3 releases

0.1.3	Nov 30, 2024
0.1.2	Nov 30, 2024
0.1.1	Nov 30, 2024

#388 in Filesystem

394 downloads per month

Custom license

130KB
2K SLoC

file utils

Intro
Installation
- python
- rust
Before starting
Arguments-explaination

Examples:

WithEOL: python:
- Example-file
- Examples:
WithCustomDelims: python:
How to use the rust crate?
Python class
Rust Structure
Structure

Intro

This package allows to read/parse a file in python. When should we use this package? If your file is really big (> 100 000 lines), because if you want to parse a file in python you'll write:

f = open("my_file", "r")
buffer: str = f.read()
...

or:

f = open("my_file", "r")
for line in f.readlines():
    ...

With the first one, there is a memory issue because you must save the entire file into a buffer.
With the second one, there is a time issue because a loop can be very slow in python.

So, this package gives tools to easily read a file with efficiently. It's based on Linux tools like grep, sed, cat, head, tail and tested with them.
WithEOL class as the same memory problem as the first example. If you want to resolve it, you must use WithCustomDelims with the "\n" delimiter.
So, why I keep WithEOL?
WithEOL is helping me to test the code, it's using a built in rust function and I'm using it as a reference to compare with WithCustomDelims.

Installation

python

With pypi:

pip install file-utils-operations

From source:

maturin develop

rust

cargo add file_utils_operations

Before-starting

This package is ASCII/UTF-8 compliant, all others encoded files will not work...

Arguments-explaination

path: the path to the file
remove_empty_string: ignore the empty string "[ ]*"
n: get n lines with tail/head
n1: the beginning line to take with between
n2: the last line to take with between
restrict: if enable, if we have last N lines, it just keep the regex in those lines. If not enable, it takes last N regex

with regex:

regex_keep: list of regex to keep
regex_pass: list of regex to pass/ignore

WithEOL-python

Example-file

We will use this example file test.txt

With cat -e test.txt:

[Warning]:Entity not found$
[Error]:Unable to recover data$
[Info]:Segfault$
[Warning]:Indentation$
[Error]:Memory leaks$
[Info]:Entity not found$
[Warning]:Unable to recover data$
  $
[Error]:Segfault$
[Info]:Indentation$
[Warning]:Memory leaks$

Example-simple-head-python

1\ Simple head (can be change to tail) Code:

import file_utils_operations_lib

path: str = "my_path_to_file"
n: int = 2 # Number of lines to read

try:
    head: list = file_utils_operations_lib.WithEOL.head(path=path, n=n)
    print(head)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Entity not found', '[Error]:Unable to recover data']

Example-simple-tail-python

Code:

import file_utils_operations_lib

path: str = "my_path_to_file"
n: int = 2 # Number of lines to read

try:
    tail: list = file_utils_operations_lib.WithEOL.tail(path=path, n=n)
    print(tail)
except:
    print("Unable to open/read the file")

Stdout:

['[Info]:Indentation', '[Warning]:Memory leaks']

Example-simple-between-python

Code:

import file_utils_operations_lib

path: str = "my_path_to_file"
n1: int = 2 # First line to read
n2: int = 4 # Last line to read

try:
    between: list = file_utils_operations_lib.WithEOL.between(path=path, n1=n1, n2=n2)
    print(between)
except:
    print("Unable to open/read the file")

Stdout:

['[Error]:Unable to recover data', '[Info]:Segfault', '[Warning]:Indentation']

Example-simple-parse-python

Code:

import file_utils_operations_lib

path: str = "my_path_to_file"

try:
    parse: list = file_utils_operations_lib.WithEOL.parse(path=path)
    print(parse)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Entity not found', '[Error]:Unable to recover data', '[Info]:Segfault', '[Warning]:Indentation', '[Error]:Memory leaks', '[Info]:Entity not found', '[Warning]:Unable to recover data', '  ', '[Error]:Segfault', '[Info]:Indentation', '[Warning]:Memory leaks']

Example-simple-count_lines-python

Code:

import file_utils_operations_lib

path: str = "my_path_to_file"

try:
    count: list = file_utils_operations_lib.WithEOL.count_lines(path=path)
    print(count)
except:
    print("Unable to open/read the file")

Stdout:

Example-remove_empty_string-python

With remove_empty_string enable:
Code:

import file_utils_operations_lib

path: str = "my_path_to_file"
n: int = 4 # First line to read

try:
    tail: list = file_utils_operations_lib.WithEOL.tail(path=path, n=n, remove_empty_string=True)
    print(tail)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Unable to recover data', '[Error]:Segfault', '[Info]:Indentation', '[Warning]:Memory leaks']

With remove_empty_string disable (default option):
Code:

import file_utils_operations_lib

path: str = "my_path_to_file"
n: int = 4 # First line to read

try:
    tail: list = file_utils_operations_lib.WithEOL.tail(path=path, n=n, remove_empty_string=False)
    print(tail)
except:
    print("Unable to open/read the file")

Stdout:

['  ', '[Error]:Segfault', '[Info]:Indentation', '[Warning]:Memory leaks']

Example-regex_keep-python

Code:

import file_utils_operations_lib

path: str = "my_path_to_file"
n: int = 4 # First line to read

try:
    head: list = file_utils_operations_lib.WithEOL.head(path=path, n=n, remove_empty_string=False, regex_keep=["\[Warning\]:*", "\[Error\]:*"])
    print(head)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Entity not found', '[Error]:Unable to recover data', '[Warning]:Indentation']

Why there is just 3 elements instead of 4? You should look at the restrict option

Example-regex_pass-python

Code:

import file_utils_operations_lib

path: str = "my_path_to_file"
n: int = 4 # First line to read

try:
    head: list = file_utils_operations_lib.WithEOL.head(path=path, n=n, remove_empty_string=False, regex_pass=["\[Warning\]:*", "\[Error\]:*"])
    print(head)
except:
    print("Unable to open/read the file")

Stdout:

['[Info]:Segfault']

Why there is just 3 elements instead of 4? You should look at the restrict option

Example-restrict-python

With restrict disable:
Code:

import file_utils_operations_lib

path: str = "my_path_to_file"
n: int = 4 # First line to read

try:
    head: list = file_utils_operations_lib.WithEOL.head(path=path, n=4, remove_empty_string=False, regex_keep=["\[Warning\]:*", "\[Error\]:*"], restrict=False)
    print(head)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Entity not found', '[Error]:Unable to recover data', '[Warning]:Indentation', '[Error]:Memory leaks']

With restrict enbale(default):
Code:

import file_utils_operations_lib

path: str = "my_path_to_file"
n: int = 4 # First line to read

try:
    head: list = file_utils_operations_lib.WithEOL.head(path=path, n=4, remove_empty_string=False, regex_keep=["\[Warning\]:*", "\[Error\]:*"], restrict=True)
    print(head)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Entity not found', '[Error]:Unable to recover data', '[Warning]:Indentation']

WithCustomDelims-python

How-to-use-it-python

It it like WithEOL but with a list of custom delimiter. For example:

import file_utils_operations_lib

path: str = "my_path_to_file"
n: int = 2 # Number of lines to read

try:
    head: list = file_utils_operations_lib.WithEOL.head(path=path, n=n)
    print(head)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Entity not found', '[Error]:Unable to recover data']

has the same behavious as

import file_utils_operations_lib

path: str = "my_path_to_file"
n: int = 2 # Number of lines to read

try:
    head: list = file_utils_operations_lib.WithCustomDelims.head(path=path, n=n, delimiter=['\n])
    print(head)
except:
    print("Unable to open/read the file")

Stdout:

['[Warning]:Entity not found', '[Error]:Unable to recover data']

So, you use it as same as WithEOL but with a list of custom delimiter.

What-delim-can-be-used

All string can be used like:

";"
"abc"
"éà"
::
"小六号"
"毫"

With-more-than-one-delimiter

If my file contains:

;À ;la ;;
pêche éèaux moules, @moules, ::小六号moules::Je n'veux小六号 plus ::y 
aller éèmaman小六号

We'll have with ";", "\n", "éè", "@", "小六号", "::"

import file_utils_operations_lib

path: str = "my_path_to_file"

try:
    parse: list = file_utils_operations_lib.WithCustomDelims.parse(path=path, delimiter=[";", "\n", "éè", "@", "::"])
    print(parse)
except:
    print("Unable to open/read the file")

Stdout

['', 'À ', 'la ', '', '', 'pêche ', 'aux moules, ', 'moules, ', '', 'moules', "Je n'veux", ' plus ', 'y ', 'aller ', 'maman', '']

How-to-use-the-rust-crate?

You must import the library with

use file_utils_operations_lib::with_custom_delims::WithCustomDelims;

use file_utils_operations_lib::with_eol::WithEOL;

Then, you can use the same functions as python because there are the same behavious.
Example:

use file_utils_operations_lib::with_custom_delims::WithCustomDelims;

fn main() {
    let mut delimiters: Vec<String> = Vec::new();
    delimiters.push("\n".to_string());
    let n: usize = 10;
    let res: Vec<String> = WithCustomDelims::head(
        "my path".to_string(),
        n,
        delimiters,
        false,
        Vec::new(),
        Vec::new(),
        true,
        1024,
    );
}

has the same behaviour as

import file_utils_operations_lib

path: str = "my_path_to_file"
n: int = 2 # Number of lines to read

try:
    head: list = file_utils_operations_lib.WithEOL.head(path=path, n=n)
    print(head)
except:
    print("Unable to open/read the file")

Python-class

If we translate the rust into python, we'll have:

class WithEOL:
    # head: Read the n first lines
    # if n > (numbers of lines in the file) => return the whole file
    def head(path: str, n: int, \
                remove_empty_string: bool = False, \
                regex_keep: list = [] \
                regex_pass: list = [] \
                restrict: bool = True):
        ...

    # between: Read the lines [n1, n2]
    # if n1 > n2 => return an empty list
    # if n1 > (numbers of lines in the file) => return an empty list
    def between(path: str, n1: int, n2: int \
                remove_empty_string: bool = False, \
                regex_keep: list = [] \
                regex_pass: list = [] \
                restrict: bool = True):
        ...
    
    # tail: Read the n last lines
    # if n > (numbers of lines in the file) => return the whole file
    def tail(path: str, n: int, \
                remove_empty_string: bool = False, \
                regex_keep: list = [] \
                regex_pass: list = [] \
                restrict: bool = True):
        ...
    
    # parse: Read the whole file
    def parse(path: str, \ 
                remove_empty_string: bool = False \
                regex_keep: list = [] \
                regex_pass: list = []):
        ...

    # Count the number of lines
    def count_lines(path: str \
                    remove_empty_string: bool = False, \
                    regex_keep: list = [] \
                    regex_pass: list = []):
        ...

class WithCustomDelims:
    # head: Read the n first lines
    # if n > (numbers of lines in the file) => return the whole file
    def head(path: str, n: int, delimiter: list \
                remove_empty_string: bool = False, \
                regex_keep: list = [] \
                regex_pass: list = [] \
                restrict: bool = True \
                buffer_size: int = 1024):
        ...

    # between: Read the lines [n1, n2]
    # if n1 > n2 => return an empty list
    # if n1 > (numbers of lines in the file) => return an empty list
    def between(path: str, n1: int, n2: int, delimiter: list \
                remove_empty_string: bool = False, \
                regex_keep: list = [] \
                regex_pass: list = [] \
                restrict: bool = True \
                buffer_size: int = 1024):
        ...
    
    # tail: Read the n last lines
    # if n > (numbers of lines in the file) => return the whole file
    def tail(path: str, n: int, delimiter: list \
                remove_empty_string: bool = False, \
                regex_keep: list = [] \
                regex_pass: list = [] \
                restrict: bool = True \
                buffer_size: int = 1024):
        ...
    
    # parse: Read the whole file
    def parse(path: str, delimiter: list \
                remove_empty_string: bool = False \
                regex_keep: list = [] \
                regex_pass: list = [] \
                buffer_size: int = 1024):
        ...

    # Count the number of lines
    def count_lines(path: str, delimiter: list \
                    remove_empty_string: bool = False, \
                    regex_keep: list = [] \
                    regex_pass: list = [] \
                    buffer_size: int = 1024):
        ...

Rust-Structure

Take a look at https://docs.rs/file_utils_operations/latest/file_utils_operations_lib/

Structure

src/: all sources files
tests/: all tests for rust
tests_files/: all files used for tests
tests_python/: a python script to test

Dependencies

~5.5MB
~103K SLoC

3 releases

file utils

Table of contents

Intro

Installation

python

rust

Before-starting

Arguments-explaination

WithEOL-python

Example-file

Example-simple-head-python

Example-simple-tail-python

Example-simple-between-python

Example-simple-parse-python

Example-simple-count_lines-python

Example-remove_empty_string-python

Example-regex_keep-python

Example-regex_pass-python

Example-restrict-python

WithCustomDelims-python

How-to-use-it-python

What-delim-can-be-used

With-more-than-one-delimiter

How-to-use-the-rust-crate?

Python-class

Rust-Structure

Structure

Dependencies