Isolate Functions From main

No law requires moving the main program into a library file. Instead, jq allows you to have the benefits of “load this code from a file” with the convenience of “here is the main program to run”.

echo "\"CAD 175.62\"" | jq 'include "assets/jq/parsing"; parse_monetary_amount'

Here’s what the file $PWD/assets/jq/parsing.jq looks like:

def parse_monetary_amount:
  capture("(?<currency>\\p{alpha}{3}) (?<value>[^\\p{space}]+)");

You’ll notice the improved name. That will only become more important as I add more code.

Danger

You can try jq --from-file assets/jq/parsing.jq 'parse_monetary_amount', but it won’t work. I know, because I tried.

The keyword include behaves like #include does in C: it dumps the contents of the file you’re including at the point you include it, just as if you’d pasted the contents right there.

Note

When you include a file, you don’t specify the extension .jq, but you may specify a relative path or even an absolute path to the location of the file.

Multiple Library Files

If you’d like to split your code base into multiple library files, it’s enough to write multiple include commands before the main program.

cat input.json | jq 'include "relative/path/to/library1"; include "relative/path/to/library2"; include "/absolute/path/to/library3"; lets_go'

Even so, once you have three such includes, you might prefer to make a library file out of including those three libraries.

# Save this to './libraries.jq'
include "relative/path/to/library1";
include "relative/path/to/library2";
include "/absolute/path/to/library3";

cat input.json | jq 'include "libraries"; lets_go'

So much nicer… but now I don’t love the relative and absolute paths. That way lies trouble… eventually.

Finding Library Files Reliably

You can specify the library path, which lists the directories in which to look for library files. This way you don’t need to scatter that knowledge throughout your code base, creating resistance to reorganizing your library files as your projects grow. If you remove the resistance now, then you’re more likely to do what you know you ought to do when the time comes.

cat input.json | jq -L 'relative/path/to' -L '/absolute/path/to' 'include "library1"; include "library2"; include "library3"; lets_go'

I’ll let you decide whether it’s still worth extracting those three include statements into their own file. When I’m not forced to read long paths, the resulting code becomes much easier to skim, and I start to prefer seeing which libraries are included here. You can choose.

Extra Added Bonus?

If you collect a handful of jq commands into a shell script, you can use a shell variable to remove duplication in the -L switches.

#!/usr/bin/env bash

# If you know your shell better than I do, you can produce this command fragment from an array of paths!
LIBRARY_PATH="-L 'relative/path/to' -L '/absolute/path/to'"

# INPUT comes from stdin

jq $LIBRARY_PATH 'include "library1"; step_1' | \
  jq $LIBRARY_PATH 'include "library1"; including "library3"; step_2' | \
  jq $LIBRARY_PATH 'include "library2"; including "library5"; step_3' | \
  jq $LIBRARY_PATH 'include "library4"; step_4' | \

You might even push the library path up to the top of the call stack, so that clients can provide them.

#!/usr/bin/env bash

# INPUT comes from stdin

# This is probably unsafe shell scripting.
# In a hostile environment, I would handle these parameters with more care and attention,
# but when it's just me, I worry less. I rarely try to hack me.
jq "$@" 'include "library1"; step_1' | \
  jq "$@" 'include "library1"; including "library3"; step_2' | \
  jq "$@" 'include "library2"; including "library5"; step_3' | \
  jq "$@" 'include "library4"; step_4' | \

Naming Collisions

This is already quite nice, but we have one more risk to consider: the global namespace. All these libraries are dumping functions into a single global namespace and that means naming collisions are inevitable. Since we can’t see inside those libraries, that only increases the chance of a collision. Moreover, as our familiarity with the contents of each library fades, we will struggle to remember which function comes from which library.

We need explicit namespaces. And jq has you covered.