Zombie Zen

Ross's Blog

Introducing pql

Posted at by Ross Light

I’m happy to announce a project I’ve been working on for the last month: pql. pql is a pipelined query language inspired by the syntax of Microsoft’s KQL and Splunk’s Search Processing Language that I developed in collaboration with the fine folks at RunReveal. pql takes queries like:

logs
	| where eventName != ''
	| summarize AllEventNames=groupUniqArray(eventName) by sourceType

and translates them into SQL like:

WITH "__subquery0" AS (SELECT * FROM "logs" WHERE coalesce("eventName" <> '', FALSE))
SELECT "sourceType" AS "sourceType", groupUniqArray("eventName") AS "AllEventNames" FROM "__subquery0" GROUP BY "sourceType";

I personally find the pql query easier to read and modify, while still retaining the power of what a relational database has to offer. pql supports all the basics of filtering, sorting, and aggregating, as well as complex joins.

Under the hood, pql builds a full Abstract Syntax Tree (AST) of its input and transforms it into a sequence of chained subqueries (similar to Static Single-Assignment form). While the generated SQL is sometimes more verbose than hand-written SQL, the queries are straightforward for database query planners to optimize out. This means that the generated queries are just as efficient as the equivalent, more condensed queries.

pql can be used with nearly any SQL-based database. If you’re interested in learning more, check out pql.dev and the announcement on RunReveal’s blog. pql is packaged as a Go library and small CLI that’s available on GitHub under an Apache 2.0 license.

If you're looking for someone to help with a complex software problem like this for your business, I'm available for consulting and contract work! Read about how I can help you and contact me at consulting@zombiezen.com.

Posted at
Permalink

Today I Learned: rsync

Posted at by Ross Light

rsync is a wonderfully useful utility for copying files both locally and between machines. It makes a reasonable effort to copy only files that have changed, which can save a tremendous amount of time in many scenarios. However, I’ve always been a bit daunted by its plethora of command-line options, and would always resort to its man page for every invocation. Thus, it would never stick in my set of tools I would commonly use. After doing some work where rsync really was the only tool for the job, I decided to finally make a little cheatsheet on how to use rsync.

Read more…
Posted at
Permalink

Ross on Cup O' Go Podcast

Posted at by Ross Light

I was on the Cup O’ Go podcast this last week! I talked about my SQLite package and the history behind it. Go check it out!

Posted at
Permalink

Bundling Scripts with Nix

Posted at by Ross Light

I write a lot of shell scripts. Many are one-offs or specific to a project, but every so often, I’ll have a script that transcends to become a part of my toolbelt. For example, nix-op-key is a script I wrote to generate new Nix signing keys and place them in 1Password. It’s not a task that requires a dedicated program, it just needs to glue two existing programs together: nix key generate-secret and op (the 1Password CLI). These sorts of scripts are great, but if you want to share them with someone else (or even just use it on a different computer), how do you do it? Scripts like these depend on specific programs (or maybe even specific versions) being installed and Bash does not have a package manager like pip or the go tool.

As it turns out, Nix is such a package manager. And with flakes, there’s built-in support for installing and running scripts with well-specified dependencies in a single command. For example, you can run my nix-op-key script I mentioned earlier (pinned to a specific version) with:

COMMIT=25e9bd52e977cca415df84ea91028efd92d3da92
nix run "github:zombiezen/dotfiles?dir=nix&rev=$COMMIT#nix-op-key" -- --help

Furthermore, you can install the script using the nix profile install command:

nix profile install "github:zombiezen/dotfiles?dir=nix&rev=$COMMIT#nix-op-key"

(If you try this out yourself, you can uninstall the script with nix profile remove '.*.nix-op-key').

In this blog post, I’ll show you how you can package your own shell scripts with Nix to make them more reliable and easier to share. This article assumes familiarity with Unix command line and Bash shell scripting. I’m using Nix 2.17.1. All of the source code in this post is released under the Unlicense and is available on GitHub.

Read more…
Posted at
Permalink

zombiezen.com/go/sqlite reaches 1.0

Posted at by Ross Light

I’m proud to announce that my CGo-less SQLite Go package, zombiezen.com/go/sqlite, has finally reached version 1.0. This has been the culmination of almost three years of work: I started on this project in March of 2021 as a fork of David Crawshaw’s crawshaw.io/sqlite to build on top of Jan Mercl’s amazing modernc.org/sqlite. I’ve built a number of features on top of this package such as:

  • A simple schema migration framework
  • A basic REPL
  • User-defined functions, including windows and aggregates
  • Custom virtual tables
  • Utilities for running embedded SQL scripts
  • A go fix-like tool for migrating existing code using crawshaw.io/sqlite
  • Support for running restricted SQL

Over this time, the project has been used in about a dozen open source projects and has over 350 stars on GitHub. I’ve successfully used it in a number of personal projects, including a personal accounting program and my Nix caching layer.

With the 1.0 release, I’m proud to call the API stable so that Go applications can continue to build on it for their storage needs. If you’re using zombiezen.com/go/sqlite for something interesting, let me know about it, and consider supporting me on GitHub.

Posted at
Permalink
Next Page →