About bevy_webcam_facial

Plugin for Bevy game engine. Captures webcam image, finds face and provides all available data (face rectangle coordinates, face probability) to Bevy game engine via events for further use in Bevy game engine.

Features

• Webcam capture using camera_capture
• Face position recognition using rustface
• Realtime and lightweight SeetaFace Detection model
• Runs in separate Bevy AsyncTaskpool task without blocking
• 2 data smoothing/denoising filters

Plans

• MacOSX webcam support
• Several AI face recognition models to choose by default (simple frame, with face features like eyes/nose/mouth, full face mesh recognition, emotion detection...)

Supported Platforms

• Linux
• MacOSX
• Windows

Available for use in Bevy:

Plugin config

Several parameters when including in .add_plugins or use default camera and settings .add_plugins(WebcamFacialPlugin::default()):

.add_plugins(WebcamFacialPlugin {
    config_webcam_device: 0,
    config_webcam_width: 640,
    config_webcam_height: 480,
    config_webcam_framerate: 15,
    config_webcam_autostart: true,
    config_filter_type: SmoothingFilterType::LowPass(0.1),
    config_filter_length: 10,
})

Parameters:

• Webcamera device number (0-first default) ex.0,1,2...
  • Linux: Number get appended to /dev/video{number}
  • Windows: Device number
• Width of frame: 640
• Width of frame: 480
• Frames per second: 15
• Start capturing and sending events instantly after plugin activation: true/false (can be enabled/disabled anytime at runtime via ResMut<WebcamFacialController>)
• Smoothing filter for coordinates (currently: MeanMedian, LowPass(f32), NoFilter)
• From how many frames take data for smoothing 5-10 optimal (more frames - less noisy data, but slower response)

Resources:

Enable/disable webcam capture and recognition from Bevy via mutable resource ResMut<WebcamFacialController>

pub struct WebcamFacialController {
...
    pub control: bool,
...
}

Event with captured data

<Event>WebcamFacialDataEvent

Data struct returned via Event

pub struct WebcamFacialData {
    pub center_x: f32,
    pub center_y: f32,
    pub x: f32,
    pub y: f32,
    pub width: f32,
    pub height: f32,
    pub score: f32,
}

Coordinates are mapped as floating point number in range of -50.0 .. 50.0, camera resolution doesn't matter

• [center_x) Face center point x coordinate
• (center_y) Face center point y coordinate
• (x) Face rectangle frame x coordinate
• (y) Face rectangle frame y coordinate
• (width) Face rectangle frame width
• (height) Face rectangle frame height
• (score) Probability of a detected object being a true face 0-30..

Some ideas and use cases of data comming from plugin:

• Controlling game object transformations (transform, rotate, scale)
• Object control (car driving, player movement...)
• Background scene movement in 2D games or background scene movement in 3D top/side view games for better depth perception or 'looking around'
• Camera FPS like movement
• Rotation around scenes, player or other objects
• Zooming in scenes (map zoom, scene zoom, sniper zoom...)
• Scaring horror games to pop beasts on detected face closeup
• Your imagination...

*Note: Use some interpolation for transforms for smoother transforms like "bevy_easings" or "bevy_mod_interp"

Examples

Three examples are provided in [examples] folder: (under construction)

• object_mover - simplest example to move object using raw unfiltered/noisy data
• camera_control - control bevy camera view using filtered data
• neck_trainer - train you neck :) most complex example with filtered data + bone animation and skin

Unchecked - not finished

Versions

Reference Material

The following were used for coding of plugin:

• Bevy The Book
• Unofficial Bevy Cheat Book
• Bevy Third Party Plugin Guidelines
• rustface

Additional interesting sources for future research:

• Tensorflow rust
• Face Detection with Tensorflow Rust (Was succesfully tested earlier)
• Tensorflow model implementing the mtcnn face detector
• https://github.com/cetra3/mtcnn/
• tfjs-model web demo of face mesh
• Awesome-Rust-MachineLearning Docs/Projects

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.