Xfbs Player

The idea is to write me a tool that I can use to create videos about programming. It should be able to accept a script (which defines all the scenes, consisting of animations, with things like:

• text
• animations
• code being typed
• terminal session replay
• background audio
• forground audio (narration), per scene or time offset

This should be able to be played back in a Gtk player application for review, or further processed into a video file (whatever format desired). Also, should be able to have a CLI, and load pre-parsed serialised data structures.

The base should revolve around a data structure that holds the audio (with time index) as well as the animations (with type, paramters, z-value, and time index). The player would then go through this data structure and generate things as it sees them coming.

The data structure has to be able to support fractional time, so that the frame rate can be set to anything desired (24, 30, 60, max fps).

Scripting Language (REPL):

• https://github.com/jonathandturner/rhai
• https://github.com/PistonDevelopers/dyon
• JSON?
• https://github.com/gluon-lang/gluon

Sound?

• Probably want to support AAC and Vorbis (for mp4 and webm)
• Need some library that can handle decoding of arbitrary sound, mixing, and re-encoding back to some format.

Video Generation?

• Some kind of h.264 encoder as well as webm?

Animations?

• cairo and gtk, married together

Binary Format?

• would be cool to be able to run a script, create all the data structures, and then use serde to serialise that along with the audio into some kind of format that can be shared.

Syntax Highlighter?

• ???

Features

• focus: ffmpeg encoding
• focus: ttyrec playback
• focus: latex support?
• focus: text type-in
• focus: fade in, out
• migrate everything from box to rc (although it doesn't help during serialisation)
• finish audio implementations
• finish animation implementations
• background color fade with HSLuv or something like that?
• load and show images
• implement wrapper types
• implement italic and stuff
• implement custom colorschemes
• TEX display (render via svg?)
• simple math display with mathjax and quick-js?
• audio encoding
• on-screen display of position
• lazy png rendering
• render to JPEGs
• render to APNG
• render to webm
• render to mp4
• implement other easing functions
• fade in
• groups of renderable items
• CI testing
• write unit tests for drawables
• write unit tests for animateds
• custom easing function
• implement scene
• SVG display with resvg
• render to GIF
• move sinus (?)
• move cubic
• move quad
• implement audio architecture
• make highlight lazy load
• make highlight return error
• fix parsing slow speed
• make colorschemes loaded lazily
• audio decoding
  • mp3 decoding
  • wav decoding
  • ogg decoding
• audio playback with rodio
• live playback controls
• pause, resume
• animations
  • move linear
  • move smooth
• make colored text scale
• source code display (highlighted) with crate
• implement new architecture
• scripting language interface
• text display
• colored text display
• render to PNGs
• live playback
• background color setting
• come up with new architecture for traits

Usage

You write a "movie" as a rhai script. It looks somewhat like this:

let movie = movie();

let bg = background(...);

movie

You can compile this script into a movie (uses a custom binary format) with the compile subcommand.

$ xfpl compile script.rhai -o movie.xfpl

This gives you an .xfpl binary file, which you can then play using the play subcommand:

$ xfpl play movie.xfpl

For convenience, the compilation script can be omitted and the player can be told to parse the script itself, using the --parse flag.

$ xfpl play --parse script.rhai

Finally, the movie can be exported into a video file using the render subcommand.

$ xfpl render script.rhai --format apkg --output out.png --resolution fullhd

Note that this can take some time, viewing it with the player is usually better for local viewing.

I want to implement my own custom drawing function

You need to decide if you're drawing something that is static or something that is animated. If you want to draw something static (as in, it doesn't change), you need to implement the Drawable trait. For example:

struct MyDrawing;

impl MyDrawing {
    pub fn new() -> MyDrawing {
        MyDrawing
    }
}

#[typetag::serde]
impl Drawing for MyDrawing {
    pub fn draw(&self, context: cairo::Context) {
        // draw
        context.set_source.rgba(1.0, 0.0, 1.0, 0.5);
        context.move_to(0.1, 0.1);
        context.draw();
    }

    pub fn size(&self) -> Size {
        Size::new(0.1, 0.1)
    }
}

In the draw() function, you can basically call any Cairo drawing function. If you're going to call something that alters the context, you should wrap your drawing code in a save/restore block, like this:

context.save();
// your code here
context.restore();

Otherwise you may break things. The size() function should return the size of whatever you're drawing.

If you notice that what you actually want to draw is something animated, then what you need to do is implement the Animated trait. This is similar to the Drawable trait, but when drawing, it gets the current time stamp, it has a duration and it can signal to play audio files.