Parallel Image Crop

A simple rust library that crops a set of images in parallel and that can be called from python.

Description

The library requires a mini-batch of paths, x-coordinates, y-coordinates, scales and a window-size. Using this it returns a crop per image. Parallelism takes place automatically using rayon

Performance Statistics

These are using FFI for the rust library. There are probably better python implementations, but this implementation is almost a 1:1 between Python & Rust. The following tests compare the Rust implementation against PIL-SIMD, pyvips using threading and multiprocessing in Python.

Gray Scale 4000 x 4000 JPEG Image:

(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=0 --use-threading=0 --use-grayscale=1
python crop average over 100 trials : 0.48177589893341066 +/- 0.06522766741342648 sec
rust crop average over 100 trials : 0.5541290354728698 +/- 0.01701503452857914 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=0 --use-threading=1 --use-grayscale=1
python crop average over 100 trials : 0.46666148900985716 +/- 0.04960818629061339 sec
rust crop average over 100 trials : 0.5545140480995179 +/- 0.018513324366932055 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=1 --use-threading=0 --use-grayscale=1
python crop average over 100 trials : 0.4660675573348999 +/- 0.057874179663163314 sec
rust crop average over 100 trials : 0.5588112926483154 +/- 0.02093116503601568 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=1 --use-threading=1 --use-grayscale=1
python crop average over 100 trials : 0.4722147536277771 +/- 0.0485672093840053 sec
rust crop average over 100 trials : 0.5598368859291076 +/- 0.023788753163516176 sec

Best: PIL-SIMD or vips

Gray Scale 4000 x 4000 PNG Image:

(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=0 --use-threading=0 --use-grayscale=1
python crop average over 100 trials : 0.48749495029449463 +/- 0.08113528509347652 sec
rust crop average over 100 trials : 0.5558233976364135 +/- 0.016653053525873318 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=0 --use-threading=1 --use-grayscale=1
python crop average over 100 trials : 0.47109495639801025 +/- 0.04871502077173173 sec
rust crop average over 100 trials : 0.5580063557624817 +/- 0.03254635395283061 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=1 --use-threading=0 --use-grayscale=1
python crop average over 100 trials : 0.45097975969314574 +/- 0.05091062232673361 sec
rust crop average over 100 trials : 0.5570903038978576 +/- 0.02757379330167939 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=1 --use-threading=1 --use-grayscale=1
python crop average over 100 trials : 0.46305535078048704 +/- 0.05080726901263223 sec
rust crop average over 100 trials : 0.5532968330383301 +/- 0.024264257976699295 sec

Best: PIL-SIMD or vips

Gray Scale 4000 x 4000 BMP Image:

(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=0 --use-threading=0 --use-grayscale=1
python crop average over 100 trials : 0.41575870513916013 +/- 0.2293008182224854 sec
rust crop average over 100 trials : 0.41494714736938476 +/- 0.008218232557933018 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=0 --use-threading=1 --use-grayscale=1
python crop average over 100 trials : 0.40679455041885376 +/- 0.1453958218374439 sec
rust crop average over 100 trials : 0.4185313296318054 +/- 0.01249058083031065 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=1 --use-threading=0 --use-grayscale=1
python crop average over 100 trials : 0.4151870512962341 +/- 0.14371236661889436 sec
rust crop average over 100 trials : 0.4274453210830689 +/- 0.019399714659261984 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=1 --use-threading=1 --use-grayscale=1
python crop average over 100 trials : 0.4173280334472656 +/- 0.15052042382476047 sec
rust crop average over 100 trials : 0.4264633345603943 +/- 0.021683232996469272 sec

Best: all almost equal

Color 512 x 512 PNG Image:

(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=0 --use-threading=0
python crop average over 100 trials : 0.1581390118598938 +/- 0.07718986836579676 sec
rust crop average over 100 trials : 0.049066624641418456 +/- 0.0063537388027109275 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=0 --use-threading=1
python crop average over 100 trials : 0.16133457899093628 +/- 0.08104974713972524 sec
rust crop average over 100 trials : 0.04776606798171997 +/- 0.002771486956986403 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=1 --use-threading=0
python crop average over 100 trials : 0.1862180781364441 +/- 0.08092567829068359 sec
rust crop average over 100 trials : 0.048760356903076174 +/- 0.004486362361121834 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=1 --use-threading=1
python crop average over 100 trials : 0.15620680570602416 +/- 0.07046950472514979 sec
rust crop average over 100 trials : 0.0478854775428772 +/- 0.0029215781261751495 sec

Best: parallel_image_crop Rust Library

Color 512 x 512 JPEG Image:

(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=0 --use-threading=0
python crop average over 100 trials : 0.2412877368927002 +/- 0.08093420929641125 sec
rust crop average over 100 trials : 0.05020998954772949 +/- 0.0029470025228493205 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=0 --use-threading=1
python crop average over 100 trials : 0.24630839586257935 +/- 0.08689070644503785 sec
rust crop average over 100 trials : 0.053718960285186766 +/- 0.005535220006405835 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=1 --use-threading=0
python crop average over 100 trials : 0.25572638273239134 +/- 0.07592489302786763 sec
rust crop average over 100 trials : 0.056830999851226804 +/- 0.0025679050981689214 sec
(base) ➜  parallel_image_crop git:(master) ✗ python benchmarks/test.py --batch-size=32 --num-trials=100 --use-vips=1 --use-threading=1
python crop average over 100 trials : 0.275831139087677 +/- 0.06285512636507674 sec
rust crop average over 100 trials : 0.05264364004135132 +/- 0.004437379466866233 sec

Best: parallel_image_crop Rust Library

Takeaway

• Small Images (512x512): parallel_image_crop use Rust library (ours)
• Large Images (4000x4000): use PIL-SIMD or vips
• Image Format: no discernible performance gain b/w PNG or JPEG. BMP is slowest.