Post by Screwtapello in Lil Programming Questions

Viewing post in Lil Programming Questions

I have a directory with files named like:

snare-2.wav
snare-3.wav
snare.wav

I think the files are in that order because that’s how naïve sorting works (the - sorts before .) so that’s the order that Lilt’s dir[] built-in returns them.

I would like to process them according to the obvious order:

snare
snare-2
snare-3

…so I think I need to do two things: trim the last four characters off the filename, and sort by the result.

My first attempt was:

select basename:(-4 drop name)
where type=".wav"
orderby basename asc
from dir["samples"]

…but this doesn’t work. For starters, -4 drop name drops the last 4 item from the name column, not the last 4 characters of each value in the column.

My first attempt was to try -4 drop @ name to “push” the drop deeper into the array, in the same way that sum @ items sums each item individually. Unfortunately, Lil doesn’t seem to understand that syntax, and the docs suggest it should only work for unary operators like sum, not binary operators like drop.

My second attempt was to do the dropping inside a loop:

each r in rows select name
  where type=".wav"
  from dir["samples"]
 -4 drop r.name
end

…but then I went to figure out how to sort the resulting list, and it seems like there’s no way to sort lists, only tables? So this seems to do what I want:

select
orderby name asc
from table (list "name") dict list
 each r in rows
  select name
    where type=".wav"
    from dir["samples"]
  -4 drop r.name
 end

…but that seems like a mess. Surely there’s a better way?

Internet Janitor8 days ago(+1)

Let's say we have a table of directory information that looks like this, for consistency of the following examples:

files:insert dir name type with
 0 "snare-2.wav" ".wav"
 0 "foo.txt"     ".txt"
 0 "snare-3.wav" ".wav"
 0 "snare.wav"   ".wav"
end

Lil query clauses are executed right-to-left, like primitive operators within an expression. We can't define a result column and then reference it in clauses to its right, but we can chain queries, as I'll demonstrate shortly.

You're correct that @ is only useful for spreading unary operations down to the elements of a list, not spreading binary operators. Many binary operators in Lil such as "like", "in", "parse", and "format" generalize to listy left and right arguments in order make them easier to use within queries, but for "drop" this would be ambiguous, so there's no free lunch.

If I wanted to shave off the last four characters of each string in a list of strings I might use an explicit "each" loop:

each x in files.name -4 drop x end
# ("snare-2","foo","snare-3","snare")

In context (since loops are expressions),

select basename:each x in name -4 drop x end type from files
# +-----------+--------+
# | basename  | type   |
# +-----------+--------+
# | "snare-2" | ".wav" |
# | "foo"     | ".txt" |
# | "snare-3" | ".wav" |
# | "snare"   | ".wav" |
# +-----------+--------+

Or, I could factor the basename extraction into a unary helper function, and subsequently use "@"; this is a good approach if "data cleaning" operations like this are ever used in multiple places, and can also clarify intent by giving the operation an explicit name:

on baseof x do -4 drop x end
baseof @ files.name
# ("snare-2","foo","snare-3","snare")

In context,

select basename:baseof @ name type from files
# +-----------+--------+
# | basename  | type   |
# +-----------+--------+
# | "snare-2" | ".wav" |
# | "foo"     | ".txt" |
# | "snare-3" | ".wav" |
# | "snare"   | ".wav" |
# +-----------+--------+

If the filenames you're working with only contain a single "." you could also consider using "parse", which automatically spreads itself to a rightward list of strings and greedily stops consuming characters for the "%s" pattern when it encounters the next literal in the format string:

"%s." parse files.name
# ("snare-2","foo","snare-3","snare")
"%s." parse ("one.wav","two.wav","three.four.wav")
# ("one","two","three")

I'll optimistically use this last strategy going forward; your mileage may vary:

select basename:"%s." parse name type from files
# +-----------+--------+
# | basename  | type   |
# +-----------+--------+
# | "snare-2" | ".wav" |
# | "foo"     | ".txt" |
# | "snare-3" | ".wav" |
# | "snare"   | ".wav" |
# +-----------+--------+

I can then write another query against that table to sort and filter it. Note that by not specifying result columns I get all the columns of the input table:

select orderby basename asc where type=".wav" from select basename:"%s." parse name type from files
# +-----------+--------+
# | basename  | type   |
# +-----------+--------+
# | "snare"   | ".wav" |
# | "snare-2" | ".wav" |
# | "snare-3" | ".wav" |
# +-----------+--------+

If I was only interested in that first column as a list, I could use "extract" instead of "select". If you don't specify a result column for "extract", it defaults to peeling out the first column:

extract orderby basename asc where type=".wav" from select basename:"%s." parse name type from files
# ("snare","snare-2","snare-3")

If that's a daunting query to look at, I could name the subquery and write it like this instead:

bases:select basename:"%s." parse name type from files
extract orderby basename asc where type=".wav" from bases

You can use queries to filter, sort, and aggregate tables, lists, or dicts; the input will be "widened" into a table in the process as needed:

select from "One","Two","Three"
# +---------+
# | value   |
# +---------+
# | "One"   |
# | "Two"   |
# | "Three" |
# +---------+

Thus, if you want to sort a plain list you can query against it and then use "extract". Here's a different formulation to the original problem:

extract "%s." parse name where type=".wav" from files
# ("snare-2","snare-3","snare")
extract orderby value asc from extract "%s." parse name where type=".wav" from files
# ("snare","snare-2","snare-3")

Does that help clear things up?

Screwtapello7 days ago(+1)

Since Lil queries look like SQL queries, I’ve been trying to use them as SQL queries with all the confusing execution order that implies. It hadn’t occurred to me to think about them as “a pipeline of operations executing right-to-left” but now that you mention it, not only does that make a lot of sense, it’s how I always wished SQL worked anyway.

Having worked with Decker transitions and on loop (both of which have strict limits on how much calculation you can do) I’ve learned to fear each expressions as they can consume a lot of the quota. Since this is a Lilt script those restrictions don’t apply, so I should probably relax and not worry so much about each.

I think I was also worried about chaining queries for similar reasons - calculation quota and my first attempt being clunky and awkward. I guess I need to trust Lil’s automatic conversions more - your chained extract example is much tidier than what I came up with, and basically what I wanted.

You’ve given me a lot to think about, thanks!

itch.io

Viewing post in Lil Programming Questions