#dwarf-fortress #json #parsing

dfraw_json_parser

Library which parses Dwarf Fortress raw files into JSON

27 releases (10 breaking)

0.12.0 Jan 24, 2023
0.10.1 Jan 18, 2023
0.7.5 Dec 29, 2022
0.2.1 Aug 9, 2022

#1 in #dwarf-fortress

Download history 2/week @ 2022-10-20 4/week @ 2022-11-03 1/week @ 2022-11-17 2/week @ 2022-11-24 1/week @ 2022-12-01 106/week @ 2022-12-08 34/week @ 2022-12-15 116/week @ 2022-12-22 124/week @ 2022-12-29 230/week @ 2023-01-05 69/week @ 2023-01-12 100/week @ 2023-01-19 31/week @ 2023-01-26

442 downloads per month

MIT license

330KB
6K SLoC

dfraw_json_parser

A solution to present information about what raws are in your save game, in a searchable format.

I made this because I was playing with Splint's Vanilla Expanded Mod and Primal and at the prepare carefully screen I had no way to figure out what some of the animals were.

Yes there is the raw explorer program but I found that difficult to search with and the information was displayed in basically the same format as the raw file itself, so it was hard to read.

There is a Typescript type definition file.

Current Functionality

Creates JSON from Dwarf Fortress raw files. This JSON can be used with the small web client to search through, be plopped into Algolia and search or, you could simply CTRL+F or grep for what you are looking for in the file itself. I find the JSON easier to read than the RAW txt files, and it currently doesn't include a lot of items that were not important to me when looking up creatures. I was most concerned with the description of the animal, if they laid eggs, if they were milkable, and how big they were.

  • Parses raw files for creatures

Outputs

There is a Typescript definition file for the format of the generated JSON. Here are detailed descriptions of Rust structs used when performing the serde_json serialization.

DF Info File

It parses the info.txt file in each raw module directory it goes through.

Property Description Type
identifier id of the raw module String
sourced_directory directory found in, vanilla, mods or installed_mods String
numeric_version version of raw module represented as a number u32
displayed_version human-readable display for numeric_version String
earliest_compatible_numeric_version earliest "upgrade-compatible" version of this raw module, as a number u32
earliest_compatible_displayed_version human readable display for earliest_compatible_numeric_version String
author author of the raw module String
name human readable name of the module String
description description of the module String
display_title built-in formatting for {name} v{displayed_version} String

Creature Token

Creature data and its castes are included. However, currently doesn't apply COPY_TAG_FROM.

Property Description Type
identifier defined in CREATURE token String
parent_raw name of the raw file its located String
raw_module id of the raws module raw is from String
raw_module_version version of the raws module raw is from String
dfraw_display human readable "Name Version" of the parent module String
objectId unique id for creature String
name species name String
names_map names by castes HashMap<String, Vec<String>>
descriptions descriptions by castes HashMap<String, String>
max_age max age by castes HashMap<String, [u16; 2]>
clutch_size clutch size by castes HashMap<String, [u16; 2]>
based_on defined by copy_tags_from token String
biomes biomes creature found in Vec<String>
cluster_range cluster range (how many appear at once) [u16; 2]
underground_depth depth found [u16; 2]
body_size body size by castes HashMap<String, Vec<DFBodySize>>
grown_at age when adult by castes HashMap<String, u32>
child_at age when adolescent by castes HashMap<String, u32>
egg_sizes egg size by castes HashMap<String, u32>
pet_value pet value by castes HashMap<String, u16>
intelligence intelligence by castes HashMap<String, [bool; 2]>
flier flier by castes HashMap<String, bool>
gnawer gnawer by castes HashMap<String, bool>
trainable trainability by castes HashMap<String, u8>
active_time active time by castes HashMap<String, u8>
inactive_season NO_SEASON by castes HashMap<String, u8>
creature_class creature class by castes HashMap<String, Vec<String>>
tags tags on the creature Vec<CreatureTag>
caste_tags tags on each caste HashMap<String, Vec<CasteTag>>
difficulty difficulty by castes HashMap<String, u32>
grass_trample grass trample by castes HashMap<String, u8>
grazer grazer by castes HashMap<String, u32>
low_light_vision low light vision by castes HashMap<String, u32>
pop_ratio population ratio by castes HashMap<String, u16>
milkable milk production by castes HashMap<String, DFMilkable>
pref_string preference string for creature Vec<String>
population_number pop num (how many exist per valid world tile) [u16; 2]

Rust Program

An example rust program which will parse a directory for DF raw files and then output the raws as JSON to be consumed by the simple web client in www/ for simple search and display functionality. The rust program is capable of serving the web client itself.

You can run this program after cloning this repository:

cargo run --example cli -- --help

cli.rs

Library which parses Dwarf Fortress raw files into JSON

Usage: cli.exe [OPTIONS]

Options:
  -g, --game-dir <GAME_DIR>
          Specify the directory where Dwarf Fortress is installed.

          This directory will likely include the 'gamelog.txt' file, and it should have a 'data' subdirectory.

          [default: ]

  -o, --out-dir <OUT_DIR>
          Specify the directory that the JSON database should be saved into.

          If raw files are parsed, a JSON database (an array of objects) is
          saved to disk in a location specified by this argument. This will
          create an 'raws.json' file in the directory specified by this argument.

          Alongside raws.json will be a modules.json which is a JSON database for the
          raw modules that were found and parsed.

          [default: ./www/]

  -h, --help
          Print help information (use `-h` for a summary)

  -V, --version
          Print version information

How to use

The example is a usable tool to write the parsed raws into JSON, this could then be uploaded to Algolia search for searching through, or can be easily ingested by a website with javascript.

There is a Typescript definition file for the format of the generated JSON.

Existing Projects

These are projects which use this library.

Perl script

This all started with a perl script, I've archived that to a gist since it lost feature parity (and object similarity) to this project.

Dependencies

~5–43MB
~812K SLoC