fy-tool¶

Synopsis¶

fy-tool [OPTIONS] [<file> …]

fy-dump [OPTIONS] [<file> …]

fy-testsuite [OPTIONS] <file>

fy-filter [OPTIONS] [-f*FILE*] [<path> …]

fy-join [OPTIONS] [-T*PATH*] [-F*PATH*] [-t*PATH*] [<file> …]

Description¶

fy-tool is a general YAML/JSON manipulation tool using libfyaml. Its operation is different depending on how it’s called, either via aliases named fy-dump, fy-testsuite, fy-filter, fy-join, or via the –dump, –testsuite, –filter, –join options.

In dump mode it will parse YAML/JSON input files and output YAML/JSON according to the output format options.
In testsuite mode it will parse a single YAML input file and output yaml testsuite reference output.
In filter mode it will parse YAML/JSON files and output YAML/JSON output filtered according to the given option.
In join mode it will parse YAML/JSON files and join them into a single YAML/JSON document according to the given command line options.

Options¶

A number of options are common for all the different modes of operation and they are as follows:

Common options

-q, --quiet¶: Quiet operation, does not output informational messages at all.

-h, --help¶: Display usage information.

-v, --version¶: Display version information.

-I DIR, --include=DIR¶: Add the DIR directory to the search path which will be used to locate a YAML/JSON file. The default path is set to “”

-d LEVEL, --debug-level=LEVEL¶

Set the minimum numeric debug level value of the library to LEVEL. The numeric value must be in the range of 0 to 4 and their meaning is as follows:

0 (DEBUG)

Internal library debugging messages. No output is produced when the library was compiled with –disable-debug
1 (INFO)

Informational messages about the internal operation of the library.
2 (NOTICE)

Messsages that could require attention.
3 (WARNING)

A warning message, something is wrong, but operation can continue. This is the default value.
4 (ERROR)

A fatal error occured, it is impossible to continue.

The default level is 3 (WARNING), which means that messages with level 3 and higher will be displayed.

Parser Options

-j JSONOPT, --json=JSONOPT¶

Marks the input files as JSON or YAML accordingly to:

no

The input files are always in YAML mode.
yes

The input files are always set to JSON mode.
auto

The input files are set to JSON mode automatically when the file’s extension is .json. This is the default.

Note that although libfyaml’s YAML 1.2 support is complete, that simply means that a valid JSON file will parse as YAML 1.2. However there are instances where one requires strict JSON spec adherence, and for invalid JSON to be rejected even when it’s valid YAML 1.2.

--streaming¶

Only valid when in dump mode, enables streaming mode. This means that no in-memory graph tree is constructed, so indefinite and arbitrary large YAML input streams can be processed.

Note that in streaming mode:

Key duplication checks are disabled.
No reording of key order is possible when emitting (i.e. –sort is not available).
Alias resolution is not available (i.e. –resolve).

Resolver Options

-r, --resolve¶: Perform anchor and merge key resolution. By default this option is disabled.

-l, --follow¶: Follow aliases when performing path traversal. By default this option is disabled.

Emitter Options

-i INDENT, --indent=INDENT¶: Sets the emitter indent (in spaces). Default is 2.

-w WIDTH, --width=WIDTH¶: Sets the preferred output width of the emitter. It is generally impossible to strictly adhere to this limit so this is treated as a hint at best. It not valid in any oneline output modes (i.e. flow-oneline or json-oneline). Default value is 80.

-m MODE, --mode=MODE¶

Sets the output mode of the YAML emitted. Possible values are:

original

The original formatting used in the input. This is default mode.
block

The output is forced to be in block mode. All flow constructs will be converted to block mode.
flow

The output is forced to be in flow mode. All block constructs will be converted to flow mode.
flow-oneline

The output is forced to be in flow mode, but no newlines will be emitted; the output is going to be a (potentially very) long line.
json

The output is forced to be in JSON mode. Note that it is impossible to output an arbitrary YAML file as JSON, so this may fail.
json-oneline

The output is forced to be in JSON mode and in a single line.
dejson

Output is in block YAML mode but with special care to convert JSON quoted strings in as non-idiomatic YAML as possible. For example { foo: “this is a test” } will be emitted as foo: this is a test. YAML can handle scalars without using excessive quoting.

-C MODE, --color=MODE¶

It is possible to colorize output using ANSI color escape sequences, and the mode can be one of:

off

Never colorize output.
on

Always colorize output.
auto

Automatically colorize output when the output is a terminal. This is the default.

-V, --visible¶: Make all whitespace (spaces, unicode spaces and linebreaks) visible. Note that this is performed using UTF8 characters so it will not work on non-UTF8 terminals, or a non-UTF8 complete font.

-s, --sort¶: Sort keys on output. This option is disabled by default.

-c, --comment¶: Experimental output comments option. Enabled output while preserving comments. Disabled by default.

--strip-labels¶: Strip labels on output. Disabled by default.

--strip-tags¶: Strip tags on output. Disabled by default.

--strip-doc¶: Strip document indicators on output. Disabled by default.

Tool mode select options

--dump¶

Select dump mode of operation. This is the default. This mode is also enabled when the called binary is aliased to fy-dump.

In this mode, all files provided in the command line will be dumped in one continuous stream, to the standard output, using document start indicators to mark the start of end new file.

If the file provided is - then the input is the standard input.

--testsuite¶

Select testsuite mode of operation. This mode is also enabled when the called binary is aliased to fy-testsuite.

In this mode, a single YAML file is read and an event stream is generated which is the format used for yaml-testsuite compliance.

If the file provided is - then the input is the standard input.

--filter¶

Select filter mode of operation. This mode is also enabled when the called binary is aliased to fy-filter.

In this mode, a single YAML file is read from the standard input for each path that is provided in the command line a document will be produced to the standard output. To use file instead of standard input use the -f/–file option.

If the file provided is - then the input is the standard input.

-f FILE, --file=FILE¶

Use the given file as input instead of standard input.

If first character of FILE is > the the input is the content of the option that follows. For example –file “>foo: bar” is as –file file.yaml with file.yaml “foo: bar”

--join¶

Select join mode of operation. This mode is also enabled when the called binary is aliased to fy-join.

In this mode, multiple YAML files are joined into a single document, emitted to the standard output.

If the file provided is - then the input is the standard input.

-T PATH, --to=PATH¶

The target path of the join. By default this is the root /.

If first character of FILE is > the the input is the content of the option that follows.

-F PATH, --from=PATH¶

The origin path of the join (for each input). By default this is the root /.

If first character of FILE is > the the input is the content of the option that follows.

-t PATH, --trim=PATH¶

Trim path of the output of the join. By default this is the root /.

If first character of FILE is > the the input is the content of the option that follows.

Examples¶

Example input files

We’re going to be using a couple of YAML files in our examples.

invoice.yaml¶

# invoice.yaml
invoice: 34843
date   : !!str 2001-01-23
bill-to: &id001
given  : Chris
family : Dumars
address:
        lines: |
        458 Walkman Dr.
        Suite #292

simple-anchors.yaml¶

# simple-anchors.yaml
foo: &label { bar: frooz }
baz: *label

mergekeyspec.yaml¶

---
- &CENTER { x: 1, y: 2 }
- &LEFT { x: 0, y: 2 }
- &BIG { r: 10 }
- &SMALL { r: 1 }

# All the following maps are equal:

- # Explicit keys
  x: 1
  y: 2
  r: 10
  label: center/big

- # Merge one map
  << : *CENTER
  r: 10
  label: center/big

- # Merge multiple maps
  << : [ *CENTER, *BIG ]
  label: center/big

- # Override
  << : [ *BIG, *LEFT, *SMALL ]
  x: 1
  label: center/big

bomb.yaml¶

a: &a ["lol","lol","lol","lol","lol","lol","lol","lol","lol"]
b: &b [*a,*a,*a,*a,*a,*a,*a,*a,*a]
c: &c [*b,*b,*b,*b,*b,*b,*b,*b,*b]
d: &d [*c,*c,*c,*c,*c,*c,*c,*c,*c]
e: &e [*d,*d,*d,*d,*d,*d,*d,*d,*d]
f: &f [*e,*e,*e,*e,*e,*e,*e,*e,*e]
g: &g [*f,*f,*f,*f,*f,*f,*f,*f,*f]

fy-dump examples.

Parse and dump generated YAML document tree in the original YAML form

$ fy-dump invoice.yaml

invoice: 34843
date: !!str 2001-01-23
bill-to: &id001
given: Chris
family: Dumars
  address:
  lines: |
    458 Walkman Dr.
    Suite #292

Parse and dump generated YAML document tree in flow YAML form

$ fy-dump -mflow invoice.yaml

{
  invoice: 34843,
  date: !!str 2001-01-23,
  bill-to: &id001 {
    given: Chris,
    family: Dumars,
    address: {
      lines: "458 Walkman Dr.\nSuite #292\n"
    }
  }
}

Parse and dump generated YAML document from the input string

$ fy-dump -mjson ">foo: bar"

{
  "foo": "bar"
}

Using the resolve option on the simple-anchors.yaml

$ fy-dump -r simple-anchor.yaml

foo: &label {
    bar: frooz
  }
baz: {
    bar: frooz
  }

Stripping the labels too:

$ fy-dump -r --strip-label simple-anchor.yaml

foo: {
    bar: frooz
  }
baz: {
    bar: frooz
  }

Merge key support:

$ fy-dump -r --strip-label mergekeyspec.yaml

---
- {
    x: 1,
    y: 2
  }
- {
    x: 0,
    y: 2
  }
- {
    r: 10
  }
- {
    r: 1
  }
- x: 1
  y: 2
  r: 10
  label: center/big
- y: 2
  x: 1
  r: 10
  label: center/big
- r: 10
  y: 2
  x: 1
  label: center/big
- y: 2
  r: 10
  x: 1
  label: center/big

Sorting option:

$ fy-dump -s invoice.yaml

bill-to: &id001
  address:
    lines: |
      458 Walkman Dr.
      Suite #292
  family: Dumars
  given: Chris
date: !!str 2001-01-23
invoice: 34843

fy-testsuite example.

An example using the testsuite mode generates the following event stream from invoice.yaml

Parse and dump test-suite event format

$ fy-testsuite invoice.yaml

fy-filter examples.

Filter out from the /bill-to path of invoice.yaml

$ cat invoice.yaml | fy-filter /bill-to

&id001
given: Chris
family: Dumars
address:
  lines: |
    458 Walkman Dr.
    Suite #292

Filter example with arrays (and use the –file option)

$ fy-filter --file=mergekeyspec.yaml /5

---
<<: *CENTER
r: 10
label: center/big

Follow anchors example

$ fy-filter --file=simple-anchors.yaml /baz/bar

frooz

Handle YAML bombs (if you can spare the memory and cpu time)

$ fy-filter --file=bomb.yaml -r / | wc -l
6726047

You don’t have to, you can just follow links to retrieve data.

$ fy-filter --file=stuff/bomb.yaml -l --strip-label /g/0/1/2/3/4/5/6

"lol"

Following links works with merge keys too:

$ fy-filter --file=mergekeyspec.yaml -l --strip-label /5/x

--- 1

fy-join examples.

Joining two YAML files that have root mappings.

$ fy-join simple-anchors.yaml invoice.yaml

foo: &label {
    bar: frooz
  }
baz: *label
invoice: 34843
date: !!str 2001-01-23
bill-to: &id001
  given: Chris
  family: Dumars
  address:
    lines: |
      458 Walkman Dr.
      Suite #292

Join two files with sequences at root:

$ fy-join -mblock ">[ foo, bar ]" ">[ baz ]"

- foo
- bar
- baz

Author¶

Pantelis Antoniou <pantelis.antoniou@konsulko.com>

Bugs¶

The only supported input and output character encoding is UTF8.
Sorting does not respect language settings.
There is no way for the user to specific a different coloring scheme.

fy-tool¶

Synopsis¶

Description¶

Options¶

Examples¶

Author¶

Bugs¶

See also¶