Russ McKendrick

GitHub - russmckendrick/tokenuse: A Rust TUI and Desktop app for exploring AI coding tool token and cost usage

That’s the last seven days. £1538, 8,308 calls, 61 sessions. Graph, Coach, and Scrollback in the sidebar are all new since the 1.0.0 post.

Coach mode, and the extension that started it

A colleague at work pointed me at AI-Engineering-Coach↗, a VS Code extension from Microsoft that watches how you work with AI coding tools and grades you on it. Not on your code, on your practice: are your prompts specific enough, are your sessions running long past the point where context has gone stale, are you reviewing what the model writes before you accept it. It’s MIT licensed, and the rules are readable.

The reason it stuck with me is that it’s answering a question Token Use was already sitting on the data for and never asking. I had every prompt, every response, every edited file, every timestamp, sat in a local archive being summed into a cost figure. Cost tells you what you spent. It tells you nothing about whether you spent it well.

So Coach is a report card for how you work with your AI tools. It landed in 1.1.0 “Report Card” as a desktop sidebar page, and the TUI got the same grade, findings, and Setup panel a couple of releases later under k.

The detection is a Rust-native port of the extension’s rule catalogue, vibe-coded in Rust using Claude Code like the rest of Token Use. Worth being clear about the split, given what the rest of this post is about: Claude Code wrote the port, and the calls about what to carry over, what to leave out, and where to deviate were mine.

It’s all evaluated locally. No network calls, no model in the loop, which is the same line I drew when I cut Insights: I’m happy for the tool to count things, I’m not happy for it to start another model to tell you what the count means. There are 28 rules now across four groups (prompt quality, session hygiene, code review, tool mastery), each with a severity that costs you points when it fires, and the composite grade maps onto A+ through F. Attribution is in NOTICE at the repo root and in docs/development/coach.md. The port is rules-as-code rather than the reference’s user-editable markdown DSL, and thresholds are Microsoft’s defaults unless there’s a documented deviation.

Below the grade there’s a Findings tab that turns each triggered rule into a filterable evidence list, an AI Output tab counting what the models actually produced, and an Activity tab with work hours, a calendar strip, and per-project breakdowns. The Flow and Pace panels sit at the bottom of the Report tab, and mine puts my average longest unbroken block at 222 minutes.

What I deliberately skipped

Sixteen of the reference rules don’t work here, and the reason is worth saying out loud because it’s the whole shape of the port. Rules like yolo-mode, no-plan-mode, no-slash-commands and auto-approve-terminal are all detectable from VS Code’s own chat logs. Token Use reads the usage files that Claude Code, Codex, Cursor, Copilot, and Gemini write to disk, and those signals simply aren’t in there. Rather than approximate them and hand you a grade built on guesses, they’re skipped.

The reference also has a profanity rule. That one I skipped on a judgement call: scanning someone’s prompts for swearing reads as surveillance in a usage dashboard, whatever the intent. And the extension’s LLM-backed features (Skill Finder, Learning Center, the Context Health review) are permanently out of scope under the no-network rule.

Going the other way, there’s one rule that isn’t in the reference set at all. Retry Loops borrows codeburn’s one-shot/retry idea and points it at the archive: it flags periods where a high share of edited files needed a re-edit after a shell run, which is the pattern where a change didn’t survive its own verification. It only fires for tools that actually report edited files, so Claude Code, Codex, and Cursor.

The other thing I care about here is that rows with missing history stay NULL and are excluded from rule denominators. If half your archive predates the enrichment columns, that half doesn’t get to skew your grade. A grading tool that quietly marks you down for data it never had is worse than useless.

The Setup panel

A couple of releases later, in 1.2.0, Coach picked up an advisory panel alongside the grade that’s kept separate from it, deliberately. It looks at your local Claude Code configuration and flags configured MCP servers you never call, CLAUDE.md files over 200 lines once @imports are expanded, files being re-read repeatedly inside a single session, and reads under node_modules and friends. Each one comes with a rough token-savings estimate.

None of it affects your practice grade, because none of it is really about practice. It’s the stuff that’s quietly costing you input tokens on every single turn.

Everything else that shipped

Coach was the headline, but four more releases went out in the same week.

1.1.1 “Ground Truth” was a correction. Cursor changed where it stores things, and Token Use had been reading a shape of the data that no longer reflected what Cursor actually does. Ingestion now goes through the current Agent store.db alongside the older state bubbles, rebuilt around a single canonical reconstruction of each user turn. Cursor’s first-party model pricing went in too, and model lookups became tool-aware so Cursor models get Cursor rates instead of falling through to a global table. Codex Desktop was also silently dropping usage events where the credit balance arrived as a string rather than a number, which is fixed and backfilled. The archive now carries provenance labels for interaction mode, token quality, and timestamp quality, and Coach’s timing rules only trust exact timestamps.

1.2.0 “Total Recall” is the big one after Coach. Scrollback is full-text search across every archived transcript, backed by an SQLite FTS5 index, available as a TUI tab on / and a desktop page. There’s a new MCP server (tokenuse mcp) exposing read-only status, overview, projects, and scrollback tools over stdio, and it will also serve over a loopback HTTP endpoint so it can run alongside the desktop app rather than fighting it for the archive. By Activity classifies every call into one of thirteen task categories using deterministic rules rather than a model. And there’s tokenuse doctor for per-tool diagnostics, plus tokenuse status and tokenuse overview for scriptable summaries with --json.

Buried in that release is a parser fix I’m quietly pleased with. Claude Code writes streamed JSONL where a single assistant turn can span multiple lines, and the old parser took the first line and dropped the rest, which silently lost tool calls. Incremental sync also went byte-level, so on a ~50MB project a one-file append now resyncs in about 0.04s rather than about 0.4s.

doctor is the one I keep going back to. It walks every probed location for every tool and tells you what it found, what’s missing, and how many calls it managed to parse, which turns “why is Cursor showing nothing” from a guess into a list.

The Projects page also became a two-level thing in this release: an index, then a page per project. This is the Token Use repo itself over the same seven days. £1098, and 45,749 lines of Rust that Claude Code wrote rather than me, which is the honest accounting behind a post otherwise full of “I added”.

1.2.1 “Model Citizen” made the Models catalogue clickable. Selecting a row opens a page for that model with its own KPIs, session list, token-composition split, per-tool donut, and an effective-pricing panel. That last one says so explicitly when a model is being billed through the pricing book’s fallback row, rather than presenting a guess as a price. The TUI got m for a typeable model picker, mirroring the existing p project picker.

It’s also where the shared model registry from 1.0.0 earns its keep. Every dated pin, vendor prefix, and router alias in the raw logs folds down into one row per model, which is why that table reads as a list rather than the same model three times under slightly different names.

1.2.2 “Constellation” shipped yesterday and adds a 3D Graph page: projects wired to the tools and models they use, or the same thing from the other end. Node size and edge strength follow whichever measure you pick (calls, spend, or tokens), and selected nodes drill straight into their detail pages. Large graphs get capped, and when that happens the view tells you how many entities it left out instead of pretending it’s showing you everything.

Note the line above the canvas: 16 of 62 entities, 33 relationships. Drawing the top slice and letting you assume that’s everything would have been easier, and I’ve used enough dashboards that do exactly that.

Installing

Installing - Token Use

# The terminal UI
brew install russmckendrick/tap/tokenuse
tokenuse

# The macOS desktop app
brew install --cask russmckendrick/tap/tokenuse-desktop
open -a "Token Use"

Linux, Windows, and manual desktop downloads are on GitHub Releases↗, and there’s more at tokenuse.app↗.

Wrapping up

Adding a report card, a search index, an MCP server, and a 3D graph a week after a post about cutting features is not a great look. But Coach still passes the test I set for 1.0.0: it counts what’s already in your data and shows you the count, rather than starting a model to tell you what the count means.

My grade is a C. 77 out of 100, 19 of the 28 rules clean. Session hygiene came out at 66, which is a D, mostly down to 51 runaway agent loops and an average unbroken block of nearly four hours.

Coach has also flagged me for Vibe Coding eight times, with the advice to write a small spec first and review the output line by line. A tool I vibe-coded, telling me off for vibe-coding. I did briefly wonder about retuning that threshold.

Thanks to the team at Microsoft who wrote those rules and put them out under MIT, and to my colleague for pointing the extension out. Coach exists because someone else had already done the thinking about what good looks like.

A Rust TUI and Desktop app for exploring AI coding tool token and cost usage - russmckendrick/tokenuse

GitHub - microsoft/AI-Engineering-Coach: better agentic engineering

better agentic engineering. Contribute to microsoft/AI-Engineering-Coach development by creating an account on GitHub.

GitHub - russmckendrick/terminal-svg: Pixel-perfect SVG screenshots of terminal output, from a single self-contained binary.

If you give it a go and the grading feels wrong for how you work, that’s worth an issue. The rules are code, and thresholds tuned against VS Code chat logs won’t be right for everyone. Pull requests and issues are welcome.

A Catch-Up: terminal-svg and Token Use v1

web.site@mckendrick.email (Russ McKendrick) — Sun, 12 Jul 2026 15:27:44 GMT

It’s been a busy few months of evenings and weekends, and two of my projects have reached a point where they’re worth writing about. One is brand new: terminal-svg, a tool for turning terminal output into SVG screenshots that actually look right everywhere. The other is Token Use, which has been through more releases than I can count and has finally landed on 1.0.0.

Rather than spin these into two posts, I’ll take them one after the other. Both were vibe-coded in Rust using Claude Code, so if you’ve read the AI Commit, ssl-toolkit or HostsButler posts, the shape of the work will be familiar.

terminal-svg

If you’ve ever tried to put a terminal screenshot in a README or a blog post, you’ll know the two ways it usually goes wrong. Either the tool regex-strips the ANSI codes, which breaks the moment your output has a progress bar or a spinner or anything that redraws a line, or it renders text against a system font, which means the box-drawing characters and powerline segments line up perfectly on your machine and nowhere else. I’d been hitting both often enough — most of the screenshots on this blog are of terminal output — that I decided to build something that did neither.

The result is terminal-svg. Point it at a command, a file, or a pipe and it gives you back an SVG with window chrome, your choice of colour scheme, and the JetBrains Mono Nerd Font it used subsetted and embedded in the file itself. That last part is the bit I care about most: the SVG renders identically on GitHub, in a browser on a machine with no fonts installed, anywhere an <img> tag works.

Pixel-perfect SVG screenshots of terminal output, from a single self-contained binary. - russmckendrick/terminal-svg

GitHub - russmckendrick/tokenuse: A Rust TUI and Desktop app for exploring AI coding tool token and cost usage

How it avoids the usual problems

The output goes through a proper VT state machine (avt↗, the engine behind asciinema) rather than a regex. Carriage-return progress bars, ESC[K clears, and cursor-up repaints all resolve to exactly the final screen a real terminal would show. For each screenshot it collects only the glyphs actually used and embeds those as WOFF2, which usually adds a few KB rather than a whole font file. Box-drawing aligns, powerline segments work, and wide CJK characters take up exactly two cells. Colour emoji can’t live in a monochrome font subset, so those get emitted as their own positioned text runs and render through the viewer’s native emoji font without knocking the columns after them out of line.

Screenshots

# Run a command in a PTY (it sees a real TTY, so colours are on)
terminal-svg -- lsd -la

# Pipe ANSI output through it
ls --color=always | terminal-svg -o ls.svg

# Render a captured ANSI dump with a different theme
terminal-svg dump.ansi -t nord -o dump.svg

Recording animated SVGs

The part I didn’t expect to enjoy as much as I did is terminal-svg rec. It drops you into your shell, records everything you do, and when you exit it renders an animated SVG replaying the session. It’s a GIF-quality demo at a fraction of the size, the text stays real and selectable, and like everything else it plays anywhere an <img> tag does — GitHub READMEs included, no JavaScript.

Recording a session

# Record an interactive session; exit the shell to finish
terminal-svg rec -o demo.svg

# rec keeps an asciicast next to the SVG, so you can retheme
# or re-time the replay later without re-recording
terminal-svg demo.cast -t github-dark --speed 2 --no-loop -o demo-fast.svg

Recordings are standard asciicast v2↗ files, so asciinema play demo.cast works, and existing asciinema recordings render straight through terminal-svg.

The SVG is its own source

Every SVG carries its own recording, compressed inside a <metadata> block that browsers ignore, along with the options it was rendered with. Lose the .cast and nothing is lost — the SVG re-renders and the recording extracts byte-for-byte. It means you can re-theme a screenshot you saved months ago without keeping the original around.

Round-tripping

# Re-render an existing SVG with different options
terminal-svg demo.svg -t nord --speed 2 -o demo-nord.svg

# Recover the recording from the SVG
terminal-svg extract demo.svg -o demo.cast

Because the embedded source contains everything that was captured, including anything a program echoed, there’s a terminal-svg edit for cleaning a take up first (masking a leaked token, say, or taming a run of long pauses), and a --no-embed-source flag if you’d rather it carried nothing at all. There’s also a browser-based visual editor (terminal-svg editor) with a live preview and a timeline panel for fixing typos and retiming events after the fact, but that’s a rabbit hole for another post.

Installing

It’s a single self-contained binary with no runtime dependencies.

Installing - macOS and Linux

brew install russmckendrick/tap/terminal-svg

Installing - Linux (direct binary)

ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
curl -LO "https://github.com/russmckendrick/terminal-svg/releases/latest/download/terminal-svg-linux-${ARCH}"
chmod +x terminal-svg-linux-${ARCH}
sudo mv terminal-svg-linux-${ARCH} /usr/local/bin/terminal-svg

Installing - Windows

Invoke-WebRequest -Uri "https://github.com/russmckendrick/terminal-svg/releases/latest/download/terminal-svg-windows-amd64.exe" -OutFile "terminal-svg.exe"

On Windows, drop the .exe somewhere on your PATH. The rec subcommand is macOS and Linux only for now; rendering .cast files works everywhere. All builds are on the GitHub releases page↗, each with a .sha256 next to it, and there’s a theme gallery at terminal-svg.dev↗.

Token Use hits v1.0.0

If you’ve read the Token Use post before, you’ll know it started as a small TUI for reading the token and cost data that Claude Code, Codex, Cursor, and Copilot already write to disk. It’s had a long run of small releases since then, and 1.0.0, named Clear Focus, is the biggest change since I started it.

A Rust TUI and Desktop app for exploring AI coding tool token and cost usage - russmckendrick/tokenuse

GitHub - russmckendrick/terminal-svg: Pixel-perfect SVG screenshots of terminal output, from a single self-contained binary.

Two things happened in this release. The first is that the desktop app got rebuilt properly. It used to be more or less a copy of the terminal layout wrapped in a window; now it’s a sidebar application with its own screens — Overview, Analytics, Tools, Models, Projects, and Config — plus direct pages for each supported tool. It still shares the same Rust data layer as the TUI, so the numbers can’t drift between the two, but it stopped pretending to be a terminal. The Projects screen now does a proper local drill-down: pick a project to see its sessions, pick a session to see its calls, open a call for its full token breakdown. The TUI got the same attention from the other direction and is now tightened down to three views: Overview, Deep Dive, and Usage.

The second thing is that I took some features out. Earlier versions had an Insights area that generated recommendations by shelling out to a local codex, claude, or gemini and asking it what your usage meant. There was an Agent Setup page doing something similar for config auditing. Both sat outside the job the tool is actually good at, added a second layer of generated interpretation on top of your data, and threw confusing errors when one of those CLIs wasn’t installed. So they’re gone, end to end — the advice engine, the prompts, the snapshots, all of it.

I’ll be honest, cutting features on the way to a 1.0.0 felt slightly backwards while I was doing it. But it left the tool much easier to describe: Token Use reads your local usage records, normalises them, prices them, and shows you the result. It doesn’t start another model to tell you what the numbers mean. Everything still stays local — there’s no platform API key, no proxy, no telemetry, and the only network calls are the confirmed pricing and currency downloads and the opt-in quota sync, exactly as before.

The other change worth mentioning is a shared model registry. Real tool logs are full of awkward model ids: dated Claude pins, vendor-prefixed Gemini paths, Cursor’s auto and default, Copilot routers that hide which provider actually served the request. All of that now folds into one set of canonical rows through src/models/registry.json, shared by the TUI, the desktop app, and the visual reports, so a model shows up the same way everywhere instead of three slightly different ways.

Installing - Token Use

# The terminal UI
brew install russmckendrick/tap/tokenuse
tokenuse

# The macOS desktop app
brew install --cask russmckendrick/tap/tokenuse-desktop
open -a "Token Use"

Linux, Windows, and manual desktop downloads are on GitHub Releases↗, and there’s more at tokenuse.app↗.

Wrapping up

Two fairly different projects, then. terminal-svg scratches a very specific itch I’ve had for years: screenshots that look the same for everyone who loads the page. It’s already rendering most of the terminal output on this blog, including its own README. Token Use reaching 1.0.0 is less about new features and more about deciding what the tool is for and cutting the rest, which turned out to be the more useful exercise of the two.

Both are on GitHub, both are open source, and both have rough edges I’m still working through. If you give either a go and something breaks or you have an idea, pull requests and issues are welcome.

Pixel-perfect SVG screenshots of terminal output, from a single self-contained binary. - russmckendrick/terminal-svg

GitHub - russmckendrick/tokenuse: A Rust TUI and Desktop app for exploring AI coding tool token and cost usage

A Rust TUI and Desktop app for exploring AI coding tool token and cost usage - russmckendrick/tokenuse

GitHub - headroomlabs-ai/headroom: Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

Playing with Headroom: Compressing the Context Going to My AI Coding Agents

web.site@mckendrick.email (Russ McKendrick) — Sat, 13 Jun 2026 10:10:02 GMT

I spend real money on AI coding tools. Enough that I built Token Use↗ to keep an eye on where it all goes across Claude Code, Codex, and the rest. So when a project turns up promising “60-95% fewer tokens, same answers,” it gets my attention, even if the part of me that has read a few too many README files starts squinting straightaway.

The project is Headroom↗, and I’ve been running it for a few days against my normal Claude Code workflow to see what those numbers actually look like once they hit my own usage rather than a benchmark. The short version: there are savings, they’re worth having, but almost none of them come from the bit the headline is selling. Let me walk through what I found.

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server. - headrooml...

What Headroom actually is

Headroom is a context-compression layer for AI agents. In practice that means a local proxy: it sits between your coding agent and the model provider, intercepts everything the agent reads on its way to the LLM, and tries to shrink it before forwarding. Tool outputs, build logs, JSON blobs, file reads, RAG chunks, and the growing conversation history all get inspected and, where it can, compressed.

It’s the work of Tejas Chopra, it’s Apache-2.0 licensed, and it’s mostly Python with a Rust core. It can run as a Python library, an MCP server, or the proxy I’ve been using. I went with the proxy because it needs no changes to how I work: point Claude Code at it and carry on.

The compression itself is the clever-sounding part. A content router sniffs each payload, works out what kind of data it is, and hands it to a matching compressor. Arrays of near-identical JSON get crushed down to a representative sample. Logs get de-duplicated. The original is kept locally so nothing is genuinely lost, and the model gets a headroom_retrieve tool it can call to pull the full version back if it actually needs it. That last detail is what separates this from just truncating output and hoping, and it’s the bit I liked most on paper.

Getting it running

Installation is a pip install and a handful of subcommands. The full extras pull in the ML models for the heavier text compression, which means a chunky first-run download, so be on a decent connection for the first go.

Installing Headroom

pip install "headroom-ai[all]"
headroom tools install
headroom init

headroom init wires it into the agents it can find on your machine, registers its MCP retrieve tool with Claude Code and Codex, and tells you to restart them so the hooks take effect. From there, the way you run an agent through the proxy is headroom wrap:

Running Claude Code through Headroom

headroom wrap claude

That starts the proxy, points Claude Code’s API base URL at it, and launches Claude Code as normal. As far as the day-to-day goes, that’s it. You work exactly as before, and the proxy quietly does its thing in the background on port 8787.

The dashboard

The proxy serves a live dashboard at http://127.0.0.1:8787/dashboard, and it’s a nicely put-together thing. Proxy dollars saved, token-savings percentage, request overhead, a prefix-cache panel, and an Anthropic subscription-window tracker that watches your 5-hour and 7-day usage windows and the overage on top.

It’s the sort of dashboard that makes you feel like you’re getting away with something. After a few longer sessions mine was reporting 20.7% token savings on roughly 1.9 million tokens, $9.23 saved on compression, and $47.36 on top from the prefix cache. Fifty-odd dollars for the price of a pip install. Lovely.

Except that’s the number you stop reading at, and the interesting part is what happens when you don’t.

Reading the numbers properly

Two things are worth pulling apart here, because the headline figure quietly blends them.

The first is that the big dollar number isn’t really Headroom’s. That $47.36 is the prefix-cache discount, which is Anthropic’s own prompt caching doing its job: cached input tokens are billed at about a tenth of the normal rate, and over 10.5 million cache reads at an 88% hit rate that adds up fast. Headroom’s contribution here is to keep the prefix of each request byte-stable so the cache actually hits, which is a real and useful thing to do. But Claude Code already caches aggressively on its own, so most of that discount is money I’d have saved anyway. Crediting it to the proxy is generous accounting.

The second is more interesting, and you only see it if you run headroom perf, which breaks down where the savings came from.

headroom perf (trimmed)

Requests:     152
Tokens:       8,956,380 -> 7,105,767 (20.7% reduction)
Total saved:  1,850,613 tokens

Per-Model Breakdown
----------------------------------------
  claude-haiku-4-5: 34 reqs, 6,688 tokens saved (2%)    ~$0.01 at list price
  claude-opus-4-8:  116 reqs, 1,843,925 tokens saved (22%)  ~$9.22 at list price
  * Actual bill savings depend on provider caching behavior

Optimization Overhead
----------------------------------------
  Average:  304ms
  Max:      24722ms
  >500ms:   8 requests

Transform Effectiveness
----------------------------------------
  content_router: 0.8% avg reduction, 152 uses, 97,239 saved

Content Router Routing
----------------------------------------
  Compressed:  92 (2%)
  Excluded:    1356 (28%) — Read/Glob outputs
  Skipped:     3047 (63%) — <50 words
  Unchanged:   311 (6%) — ratio too high

So the proxy reports 1.85 million tokens saved, and the actual content compressor (the clever JSON-and-logs router that’s the whole pitch) accounts for 97,239 of them. That’s about five percent of the saving. The rest is the proxy trimming stale conversation history on long sessions, which is a perfectly reasonable thing to do, but it’s not the headline feature. The dashboard files all of it under “compression,” and the perf breakdown is the more honest reading.

Look at the routing lines and it’s clear why. Sixty-three percent of what passed through was too short to bother with. Twenty-eight percent was Read and Glob output, which Headroom deliberately excludes. Only two percent actually got compressed. My sessions are mostly conversational back-and-forth and file reads, and that’s precisely the shape of work where the content compressor has the least to chew on. The tool even tells me so in its own recommendations, suggesting I let it compress stale Read outputs for a bigger win.

None of this is Headroom being broken. It’s Headroom being honest about a workload it isn’t ideally suited to. If I were running tool-heavy agents that hammer JSON APIs or wade through enormous log files, those compression numbers would look very different. For someone doing the kind of mixed coding-and-chatting I do, the compressor is a rounding error and the real saving is the history trimming plus a cache discount I mostly already had.

One more thing the dashboard glosses over: it isn’t free. The proxy adds about 300ms of overhead per request on average, with the occasional outlier into the tens of seconds while the ML text model warms up. Against the multi-second latency of an actual model response that’s mostly noise, but it’s not nothing, and it’s the kind of cost that doesn’t show up next to the dollar signs.

The bit I didn’t expect to like

The feature that earned its keep wasn’t the compression at all. It was headroom learn.

It reads back through your past agent sessions, the logs Claude Code and Codex leave behind, finds the places where the agent went wrong and correlates them with whatever eventually worked, and proposes corrections to write into your CLAUDE.md or AGENTS.md. It runs as a dry run by default and shows you exactly what it would add before touching anything.

headroom learn

headroom learn

headroom learn (trimmed)

Detected agents: Claude Code, OpenAI Codex CLI, Google Gemini CLI

============================================================
[codex] codex
Path: /Users/russ.mckendrick
============================================================
  Analyzing with claude-cli...

  Sessions: 181  |  Calls: 26599  |  Failures: 956 (3.6%)
  Recommendations: 5

  [WOULD WRITE] /Users/russ.mckendrick/.codex/AGENTS.md
  ──────────────────────────────────────────────────

  ## Headroom Learned Patterns
  *Auto-generated by `headroom learn` on 2026-06-13 — do not edit manually*

  ### dvr file map
  *~2,500 tokens/session saved*
  Large files — read targeted ranges, never cat whole: `src/recorder.rs`
  (~2400 lines) and `frontend/App.jsx` (~2300 lines). Recordings live at
  `/Volumes/vids/_dvr/` locally and `/mnt/_dvr/` on the server; they are
  NOT under the repo.

  ### Projects & Paths
  *~1,500 tokens/session saved*
  Active work is in two Rust projects, not the session root (`~`):
  `~/Code/dvr` and `~/Code/aicommit`. Both have an AGENTS.md. cwd often
  starts at `~` — cd into the project or use absolute paths.

  ### Build & Validation (Rust)
  *~1,200 tokens/session saved*
  Standard gate in both projects:
  `cargo fmt && cargo check && cargo clippy --all-targets --all-features
  -- -D warnings && cargo test` (clippy is run `-D warnings`, so warnings
  fail). dvr's frontend builds only with `pnpm run build`.

  ### aicommit facts
  *~1,000 tokens/session saved*
  Installed binary is `aic` (the `aicommit` bin was removed). Config env
  vars use the `AIC_` prefix; AI engine code in `src/ai/`, prompts in
  `prompts/commit-system.md`; tests in `tests/cli.rs` and `tests/provider.rs`.

  ### Environment (macOS)
  *~700 tokens/session saved*
  macOS host: `timeout`/`gtimeout` and `go` are NOT installed — don't use
  `timeout` to bound commands and don't attempt `go build`/`go test`.
  Running the dvr server with `cargo run` binds a port and is blocked by
  the sandbox; rely on unit/integration tests rather than launching the server.

  ──────────────────────────────────────────────────

  Dry run — use --apply to write.

On my machine it correctly spotted my two active Rust projects, worked out which files are too large to read whole, learned the cargo fmt && cargo check && cargo clippy && cargo test gate I run before every commit, and noted that this Mac has no timeout or go installed so the agents should stop trying to use them. It also slotted in a block for codebase-memory-mcp↗, the knowledge-graph indexer, telling the agents to prefer graph queries over blind file searching.

That’s the sort of context I’d normally write into an AGENTS.md by hand over a few weeks of being mildly annoyed. Having a tool mine it out of my own session history and hand it back is properly useful, and it’s the part of Headroom I’ll keep using regardless of what the compression numbers do.

What I made of it

Headroom is at v0.25.0 and the author is upfront that it’s early and not battle-tested against every edge case, which matches what I saw. The pieces all work, the dashboard is good, and nothing fell over.

Whether it’s worth running comes down to what your agents actually do. If you’re running tool-heavy, JSON-and-log-heavy agent loops, the content compression has real material to work on and the savings should be the genuine article. If you’re doing mixed interactive coding like me, the compressor barely registers, the history trimming does a bit, and the headline dollar figure is mostly the provider’s own cache discount wearing Headroom’s badge. The two things are complementary rather than competing, so it’s not an argument against running it, just an argument for reading the perf breakdown instead of the front page.

The lesson, if there is one, is the same one that applies to any dashboard with a big green number on it: the number is the start of the question, not the answer. Headroom’s perf command tells you the truth if you ask it, which is more than a lot of tools manage. And the learn feature alone makes the install worth it.

GitHub - DeusData/codebase-memory-mcp: High-performance code intelligence MCP server. Indexes codebases into a persistent knowledge graph — average repo in milliseconds. 158 languages, sub-ms queries, 99% fewer tokens. Single static binary, zero dependencies.

If you spend enough on tokens to care, give it a go, run headroom perf after a few real sessions, and decide based on what it shows you rather than the headline. Your mileage will depend entirely on what your agents spend their day reading.

codebase-memory-mcp: Giving Claude Code (and Codex) a Map

web.site@mckendrick.email (Russ McKendrick) — Sun, 10 May 2026 16:13:23 GMT

Anyone who has spent a few hours with Claude Code or Codex on a growing project will have watched the agent grep its way around the same files over and over. Ask it “what calls this function?” and you can almost see it picking through the codebase a directory at a time, burning tokens as it goes. It works, but it’s not exactly elegant.

codebase-memory-mcp↗ takes a different approach. It’s an MCP server that parses your source with tree-sitter and indexes everything (functions, classes, call chains, even cross-service HTTP links) into a SQLite-backed knowledge graph. Your agent then queries the graph instead of grepping the filesystem.

High-performance code intelligence MCP server. Indexes codebases into a persistent knowledge graph — average repo in milliseconds. 158 languages, sub-ms queries, 99% fewer tokens. Single static bin...

What it does

The README claims a 99.2% reduction in tokens for structural queries on a multi-service test project (~3,400 tokens via the graph versus ~412,000 via file-by-file exploration), and there’s an arXiv paper↗ backing the design with benchmarks across 31 real-world repositories: 83% answer quality, 10× fewer tokens, and 2.1× fewer tool calls than file-by-file exploration. I was sceptical, but after a couple of sessions on one of my own projects, I’m prepared to believe it.

It supports 66 languages - the usual suspects (Python, Go, TypeScript, Rust, Java, C/C++, C#, PHP, Ruby) plus a long tail that includes Zig, Elixir, OCaml, Haskell, and infrastructure formats like Dockerfile, Kubernetes manifests, and HCL. The Linux kernel (28M LOC, 75K files) gets indexed in three minutes on an M3 Pro, and most queries return in under a millisecond. 14 MCP tools cover indexing, call-path tracing, dead code detection, git-diff impact analysis, ADR management, and Cypher-like graph queries against the graph. REST routes from FastAPI, Gin, Express and friends are first-class graph nodes too, so you can match HTTP call sites to their handlers across services.

It’s a single statically-linked binary written in C with no runtime dependencies, backed by SQLite in WAL mode. The cache lives at ~/.cache/codebase-memory-mcp/ and persists across restarts.

Installation

The repo provides a one-liner installer that grabs the right pre-built binary for your platform, sorts macOS code signing, and registers the MCP server with whichever coding agents it can find. I went with the --ui variant, which bundles a graph visualisation frontend on top of the MCP server itself:

Install on macOS / Linux with the UI variant

curl -fsSL https://raw.githubusercontent.com/DeusData/codebase-memory-mcp/main/install.sh | bash -s -- --ui

Windows users get a PowerShell equivalent (install.ps1 from the same repo) - the README’s Quick Start↗ has the three-line download-inspect-run sequence.

The script drops the binary into ~/.local/bin and then runs codebase-memory-mcp install to wire it into any agents it detects. It auto-configures eleven of them out of the box (Claude Code, Codex CLI, Gemini CLI, Zed, OpenCode, Antigravity, Aider, KiloCode, VS Code, OpenClaw, and Kiro), adding MCP server entries, instruction files, skills, and pre-tool hooks where each agent supports them. In my case it picked up both Claude Code and Codex without me having to point at anything, and it’ll warn you if ~/.local/bin isn’t on your PATH yet.

After it finishes, restart your coding agent. In Claude Code, /mcp will show codebase-memory-mcp in the list of available servers; in Codex, the new tools turn up on the next session.

Indexing a project

Once it’s connected, point your agent at a project:

Index a project

Index this project

The agent calls index_repository against your current directory. The first run does a full parse; after that, it’s incremental, and only files whose content hashes have changed get re-parsed.

Asking questions

Once the graph is built, the agent has something better than grep to reach for. The kind of questions that used to take a flurry of greps now resolve to a single tool call:

“What calls process_order?”
“Find all functions with no callers”
“Show me the cross-service HTTP calls between the API and the worker”

Under the hood, those become trace_call_path, search_graph, and query_graph calls. Both Claude Code and Codex translate the natural language into the right tool without any extra prompting on my part.

Here it is answering a real one against this blog’s repo:

Behind that single prompt the agent ran one search_graph call (16 results, top hit getCFImageUrl), pulled the implementation with get_code_snippet, and skim-read a few of the related docs files. Total work for a question that would otherwise have been a parade of greps and reads: “Explored 5 files, 1 search”. The summary it produced (this repo uses Cloudflare Image Transformations and the key helper is src/utils/cloudflare-images.ts at line 32) is the kind of concise, file-referenced answer I want from an agent, and the kind that’s hard to get when it’s forced to crawl the filesystem one read at a time.

For the curious, here’s what that whole Codex session actually cost - pulled out of Token Use↗, the dashboard I wrote about last week:

The whole session, including the one-off index_repository call upfront (the most expensive of the seven by some margin), came to £0.299 - call it 30p. The question-answering portion alone, once the index was in place, was about 22p across six smaller calls. The standout number is the cache reads: 198.3K of the 238.9K input tokens were served from cache, with the actual output a tiny 3.5K. Codex is doing very little talking and a lot of querying, which is precisely the shape you want.

The graph UI

If you went with the --ui variant, the binary will also start a small web app on demand that lets you browse the indexed graph visually. Run it with the --ui=true flag and a port of your choosing:

Start the graph UI

codebase-memory-mcp --ui=true --port=9749

Then point your browser at http://localhost:9749. Here’s what it looks like for this blog:

That’s 10,929 nodes and 10,916 edges, which sounds like a lot until you remember Astro turns every MDX file into a tree of Section nodes (7,885 of them in my case, alongside 542 Files, 542 Modules, 94 Functions, and a handful of Routes). The sidebar filters by node label and edge type, you can search and click into individual nodes, and the whole thing lays itself out as a big force-directed graph that’s quietly fun to poke around in.

I haven’t reached for the UI much during day-to-day work - the value flows through the agent - but it’s useful as a sanity check on what the indexer has actually picked up, and it makes for a decent screensaver if someone is looking over your shoulder.

Teaching your agent to actually use it

One thing the README is upfront about: without a hint, agents will sometimes default to grep and Read for code questions even with the MCP server connected. The good news is that install handles this for you - it adds skills, instruction blocks, and pre-tool hooks for whichever agents it detects, and those nudge the agent toward graph queries for structural questions rather than letting it reach for grep out of habit.

If you want to know what kind of guidance gets added - or you’re using an agent that isn’t on the auto-config list and want to roll your own - here’s a sensible starting block. Drop it into ~/.claude/CLAUDE.md for Claude Code, or ~/.codex/AGENTS.md for Codex:

CLAUDE.md / AGENTS.md

## Codebase Memory (codebase-memory-mcp)

When this MCP server is available, **prefer graph tools over grep/Explore for structural code questions**.
Graph queries return precise results in a single tool call (~500 tokens) vs file-by-file exploration (~80K tokens).

- **Before exploration/planning**: Run `index_repository` to ensure the graph is current
- **"Who calls X?"**: `trace_call_path(function_name="X", direction="inbound")`
- **"What does X call?"**: `trace_call_path(function_name="X", direction="outbound")`
- **Find functions by pattern**: `search_graph(label="Function", name_pattern=".*Pattern.*")`
- **Dead code**: `search_graph(label="Function", relationship="CALLS", direction="inbound", max_degree=0, exclude_entry_points=true)`
- **Cross-service calls**: `search_graph(relationship="HTTP_CALLS")` or `query_graph` with Cypher
- **REST routes**: `search_graph(label="Route")`
- **Understand structure first**: `get_graph_schema` before writing complex queries
- **Read source**: `get_code_snippet(qualified_name="...")` after finding functions via search
- **Complex patterns**: `query_graph` with Cypher for multi-hop graph traversals

Use grep/Glob for text search (string literals, error messages, config values) - the graph doesn't index text content.

I dropped this into my global CLAUDE.md and ~/.codex/AGENTS.md, and the difference is noticeable in both. The agent leans on trace_call_path and search_graph for anything structural, and falls back to grep only for text content the graph doesn’t index, like string literals or error messages.

A few caveats

The graph indexes structure, not text content, so don’t expect it to find that one config string you misspelt three months ago - that’s still grep territory. The recommended block above is explicit about this, which is why the agent will still happily reach for grep when it’s the right tool for the job.

That said, for the “where is this used?” and “what’s actually dead?” questions that come up constantly when refactoring, it’s quietly become part of my default setup in both Claude Code and Codex.

Worth a try?

If you’re using Claude Code or Codex on anything bigger than a single-file script and you’ve felt the friction of the agent re-reading the same files for the same questions, this is worth ten minutes of your time. The install is painless thanks to the script doing the agent configuration for you, and once a project is indexed the token savings on structural queries are substantial enough to notice on the bill at the end of the month.

Token Use - AI Token Usage & Cost Tracker for Coding Tools

Counting the Cost of Vibe Coding

web.site@mckendrick.email (Russ McKendrick) — Fri, 01 May 2026 07:53:04 GMT

Over the last year or so, every AI coding provider seems to have settled on the same pricing direction: token-based billing. Claude Code, Codex, Cursor, Copilot, and Gemini all now meter you on input, output, cache reads, cache writes, and (in some cases) reasoning tokens, and the bill at the end of the month reflects how chatty you’ve been with the model rather than a flat subscription.

Which is fine, until you start wondering where the money actually went.

Vibe coding makes that worse, not better. When you’re cheerfully throwing prompts at the model and pasting in screenshots, the per-turn cost is the last thing on your mind, but it does add up.

I’d been using a couple of existing tools to keep an eye on this, and they did the job for the basics (totals per day, totals per project) but I kept hitting limitations. I wanted to see costs in pounds rather than always having to do the conversion in my head. I wanted to drill into a specific session and see the prompt that triggered a particularly expensive turn. I wanted to compare Claude Code and Codex side by side on the same project, because I tend to use both depending on the task. None of the existing tools quite did all of that, so I built Token Use.

Track AI coding token usage, costs, models, projects, and quotas across Claude Code, Codex, Cursor, GitHub Copilot, and Gemini with a local TUI and desktop app.

tokenuse.app

What it does

Token Use reads the session files that AI coding tools already write to your local disk, normalises the data into a single archive, and shows you what you’re spending. There’s no API key, no proxy, no telemetry endpoint, no daemon, and no live file watcher. It just parses the JSONL (and one SQLite database, in Cursor’s case) that your tools have already written, then renders the results.

It currently supports five sources:

Tool	Source format	Token quality
Claude Code	JSONL session files under `~/.claude/projects/`	Exact, including cache reads and writes
Codex	JSONL rollouts under `~/.codex/sessions/`	Exact per-turn token deltas, plus rate-limit snapshots
Cursor	SQLite `state.vscdb` and Agent transcripts under `~/.cursor/projects/`	Exact when present, estimated otherwise
GitHub Copilot	Legacy CLI events and VS Code Copilot Chat transcripts	Exact for legacy, estimated for transcripts
Gemini CLI	Chat files under `~/.gemini/tmp/<project_hash>/chats/`	Exact, including cached input reads and thought tokens

It comes in two flavours: a Rust-based TUI for the terminal, and a Tauri desktop app for when you’d rather have a window with a sidebar and clickable rows. Both share the same local archive, so whichever you launch shows the same data, and they share the same embedded keymap so the shortcuts work the same in either place.

The meta moment

Before we get into installing and using it, I want to show you the obvious thing: I built Token Use using Claude Code and Codex, which means Token Use has a very detailed record of what Token Use cost to build.

The dashboard above shows the project filtered to tokens (which is the name of the folder on my local machine), with the breakdown split across the tools I used while building it. That 96.3% cache hit rate is doing a lot of the heavy lifting on the Claude Code side. Anthropic prices cache reads at roughly 10% of the input rate, so without that ratio the Claude column would be considerably more eye-watering.

The orange and cyan chart in the middle is the Activity Pulse: the orange bars are spend per bucket, the cyan area is call volume over the same window. Hovering a bucket shows the period label, cost, and call count. Short windows like 24 Hours and 7 Days use hourly buckets; 30 Days and All Time use daily buckets. This Month is a hybrid that stays hourly for the first two weeks and switches to daily from the 15th onward, which avoids the start of a month collapsing into one or two oversized bars.

Drilling into a single session shows every call: when it happened, which model handled it, how many tokens went in and out, what tools were used, and the start of the prompt. The Codex session above cost £55.37 across 616 calls, with 86.9M of the 91.4M input tokens served from cache.

I’ll leave the absolute number for the GitHub README (the cover image above isn’t entirely satirical), but seeing it broken down by day, by tool, and by session was the thing that finally convinced me the tool was actually useful rather than just an interesting weekend project.

Installation

macOS

Both the TUI and the desktop app are available through my Homebrew tap↗:

Installing the TUI - macOS

brew install russmckendrick/tap/tokenuse

Installing the desktop app - macOS

brew install --cask russmckendrick/tap/tokenuse-desktop
open -a "Token Use"

The desktop DMG is Apple Silicon only, signed with a Developer ID Application certificate, and notarised through App Store Connect, so it should open without any Gatekeeper drama.

Linux

The TUI works on AMD64 and ARM64. The script below detects your architecture and grabs the right binary:

Installing the TUI - Linux

ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
curl -sL "https://github.com/russmckendrick/tokenuse/releases/latest/download/tokenuse-linux-${ARCH}" -o tokenuse
chmod +x tokenuse
sudo mv tokenuse /usr/local/bin/

The desktop app for Linux is published as unsigned AppImage, deb, and rpm assets for both architectures. Verify the matching .sha256 file before running. For the AppImage:

Installing the desktop app - Linux

curl -L -O https://github.com/russmckendrick/tokenuse/releases/latest/download/tokenuse-desktop-linux-amd64.AppImage
curl -L -O https://github.com/russmckendrick/tokenuse/releases/latest/download/tokenuse-desktop-linux-amd64.AppImage.sha256
sha256sum -c tokenuse-desktop-linux-amd64.AppImage.sha256
chmod +x tokenuse-desktop-linux-amd64.AppImage
./tokenuse-desktop-linux-amd64.AppImage

AppImage installs can check for newer desktop releases from the Config tab. If you used the .deb or .rpm package, update manually with the matching asset from GitHub Releases.

Windows

The Windows TUI is a single executable:

Installing the TUI - Windows

Invoke-WebRequest -Uri "https://github.com/russmckendrick/tokenuse/releases/latest/download/tokenuse-windows-amd64.exe" -OutFile "tokenuse.exe"

The desktop app ships as both an NSIS installer and an MSI for environments that prefer Windows Installer. Both are unsigned for now, so verify the .sha256 before installing:

Installing the desktop app - Windows

Invoke-WebRequest -Uri "https://github.com/russmckendrick/tokenuse/releases/latest/download/tokenuse-desktop-windows-amd64-setup.exe" -OutFile "tokenuse-setup.exe"
Get-FileHash .\tokenuse-setup.exe -Algorithm SHA256
Start-Process .\tokenuse-setup.exe

Like the AppImage build, the installed Windows desktop app can check for later NSIS installer releases from the Config tab. All releases are on the GitHub releases page↗.

The TUI

Run tokenuse and it’ll scan your local session files, build the archive, and drop you into the Overview page:

The summary tiles up top cover cost, calls, sessions, cache hit rate, and input/output token totals. Below that, the Activity Pulse is chronological and ignores whatever sort mode you’ve picked, so the line keeps showing usage over time even when the ranked tables underneath are sorted by spend, latest date, or token use. The ranked tables themselves cover project and project/tool spend, model spend, shell commands, and MCP servers.

The filters work the way you’d expect:

Key	Action
`1` – `5`	Period: rolling 24 hours, 7 days, 30 days, this month, all time
`t`	Cycle the tool filter
`p`	Open the project picker
`s`	Open the session picker
`g`	Cycle sort mode: spend, latest date, token use
`Tab` / `Shift-Tab`	Cycle between Overview, Deep Dive, and Usage
`Shift-D`	Toggle between live and bundled sample data
`r`	Resync the archive
`e`	Generate a report
`q`	Quit

The 1 period is rolling from the current time rather than a calendar-day midnight cutoff, which I find more useful when you’ve just finished a session and want to see what it actually cost.

The Deep Dive page swaps the compact Activity Pulse for a larger Activity Trend graph, and adds two panels that aren’t on Overview: Top Sessions, ranking the most expensive sessions in the active filter, and Core Tools, which counts every AI tool call (Bash, Edit, Read, update_plan, and so on).

Sessions and call detail

The session page is the bit I find myself in most often. Press s to open a searchable picker of every session in the active filter; start typing to narrow it down by date, tool, or project name:

Pick one and the drill-down opens, showing every turn in the session along with its timestamp, model, cost, tokens, tools, and the start of the prompt:

Pressing Enter on any row opens a detail modal with the full stored prompt and metadata for that turn. It’s useful for tracking down which question produced the expensive answer.

Usage and rate limits

The Usage page is a rolling 24-hour view across every supported tool. Opening the page automatically selects the 24 Hours period so the visible filter matches what each per-tool console is showing, and the page ignores the project filter so each tool gets its own section regardless of where you’ve been working.

Each tool console shows a 24-hour pulse graph, totals for calls, tokens, cost, and last seen, plus the top three models for that tool’s slice. If you’re using Codex, the Usage page also shows the rate-limit snapshots Codex writes to its session files, so you can see how close you are to the 5-hour and weekly windows without having to do the maths yourself:

Token Use doesn’t make any live API calls to fetch limits, it just surfaces the values your tools have already written locally. That’s a deliberate trade-off: the figures lag slightly behind whatever the tool is showing in real time, but there’s no extra surface area to worry about.

Currency and pricing

Costs are calculated and stored as USD at import time, then converted for display. The Config page lets you pick a different display currency:

The currency snapshot is fetched from Frankfurter↗ and the pricing table is derived from LiteLLM↗. Both are embedded in the binary so the tool works completely offline by default. If you want fresher rates, the Config page has explicit “download latest” buttons that need confirmation each time. There are no background fetches.

The Config page also has a clear-data action that asks for confirmation, deletes the local archive.db, and immediately reimports from your tool history. Your config, downloaded rates, pricing snapshots, and previously generated reports are kept. It’s handy if you want to rebuild the archive against a fresher pricing snapshot, or just to check the importer is doing what you expect.

The desktop app

The desktop app is a Tauri v2 + Svelte frontend over the same Rust core. It reads the same archive, writes the same config, and exposes the same dashboard pages, with clickable tabs and dropdowns alongside the keyboard shortcuts. Here’s the Deep Dive view, which mirrors the layout of the TUI Deep Dive page:

The Usage tab is the same view as in the TUI, with per-tool consoles for each of the five supported tools and the Codex rate-limit windows visible alongside the 24-hour totals:

If you have both installed, you can use whichever suits the moment. I tend to leave the desktop app open on a second monitor when I’m doing a lot of AI-assisted work, and reach for the TUI when I’m already in a terminal and want to spot-check spend on a specific project.

Tray popover and background alerts

Closing the desktop window doesn’t quit the app: Token Use keeps running in the background, and on macOS the Dock icon will reopen the main window if you click it again. Left-clicking the tray or menu-bar icon opens a compact 24-hour popover with KPI tiles for cost, calls, tokens, and cache hit rate, an Activity sparkline, and a top-models breakdown. There’s an Open button if you want the full dashboard, a Refresh button if you want to force a sync, and right-clicking the icon gives you the native menu with Show Token Use and Quit Token Use.

While it’s running in the background, the desktop app keeps polling the archive for new usage. If an automatic refresh imports a chunk of new activity that crosses configurable thresholds, you get a native desktop notification. The defaults are conservative (notify after at least $1.00, 100k tokens, or 25 calls of new usage, with a 30-minute cooldown) and they live in the same config.json the TUI reads:

config.json

{
  "currency": "GBP",
  "background_alerts": {
    "enabled": true,
    "min_cost_usd": 1.0,
    "min_tokens": 100000,
    "min_calls": 25,
    "cooldown_minutes": 30
  },
  "desktop": {
    "open_at_login": false,
    "show_dock_or_taskbar_icon": true
  }
}

The desktop-only Open at Login and Dock/taskbar icon toggles are also exposed in the Config tab, so you can opt into a tray-first setup without hand-editing JSON if you’d rather. On Windows and Linux, the Config tab additionally includes an explicit update check against GitHub Releases, which is the supported path for those platforms; macOS desktop updates continue to come through Homebrew Cask.

Reports

This is the bit that’s changed the most since the first release. The earlier snapshot exports have been replaced with a richer Reports workflow. Press e on Overview, Deep Dive, Usage, or Session (or click the report button in the desktop app) and you’ll get a modal that asks for the period, project scope, format, and whether to redact prompts, commands, raw paths, session IDs, and dedup keys before generation. Reports always include all tools for the chosen scope, regardless of whatever tool filter you have set on the dashboard.

The available formats cover most use cases:

Format	Output
HTML	One self-contained executive report deck
PDF	One A4 landscape render of the same deck (browserless)
SVG	One 16:9 visual summary with KPI strip, heatmap, and top highlights
PNG	Same one-page visual summary, rasterised
JSON	One pretty-printed full report dataset
Excel	One multi-sheet `.xlsx` workbook with summary and raw data sheets
CSV	One folder with one CSV per report area

The HTML and PDF decks are intended to be client-ready, with cover metadata, KPI ribbons, insight tiles, activity pages, and breakdown pages.

The SVG and PNG outputs are the one-page version for when you just want a single visual summary. JSON, Excel, and CSV are the raw data formats for if you want to do your own analysis. Output defaults to your Downloads folder, falling back to ~/Downloads and then the local config directory, with timestamped, slugged filenames so re-running doesn’t overwrite the previous report.

There’s also a guided CLI for when you want a report without opening the dashboard:

Generating reports from the CLI

tokenuse report

This walks through the same period, project, format, output folder, redaction, and confirmation steps as the modal, then exits. It only writes from live local sessions, so it bails out cleanly if there’s nothing to report on.

Privacy

Token Use is local-only by design. There is no telemetry or analytics, and no API keys to provide for any of the supported tools. The only outbound network requests are the explicit Config-page downloads for currency rates and the pricing snapshot, the optional update checks on the Windows and Linux desktop apps, and (on tagged builds) the maintainer release paths. Each of those needs confirmation before it fires.

If you’re security-conscious about a tool that reads your session transcripts, the source is on GitHub. The parsers live under src/tools/<name>/ if you want to verify exactly what’s being read, and the documentation on tokenuse.app↗ has a parser-by-parser breakdown covering each tool’s quirks.

Token Use - AI Token Usage & Cost Tracker for Coding Tools

Track AI coding token usage, costs, models, projects, and quotas across Claude Code, Codex, Cursor, GitHub Copilot, and Gemini with a local TUI and desktop app.

tokenuse.app

Summary

The source code for Token Use lives at:

GitHub - russmckendrick/tokenuse: A Rust TUI and Desktop app for exploring AI coding tool token and cost usage

A Rust TUI and Desktop app for exploring AI coding tool token and cost usage - russmckendrick/tokenuse

Is Your Site Agent-Ready?

It’s still early in its life, so there are bound to be edge cases I haven’t hit yet, particularly across the five different tool formats now that Gemini CLI has joined the lineup. If you spot something off in the parsing, or if there’s a feature that would make the tool more useful for your workflow, issues and pull requests are welcome.

If nothing else, having the data has changed how I think about my AI tool usage. Knowing that a particular kind of refactoring task tends to cost a few quid, while a quick code review costs almost nothing, is the sort of thing that’s hard to internalise without seeing the numbers laid out next to each other. Whether you end up doing more of the cheap things or fewer of the expensive ones is up to you.

Is my blog Agent-Ready?

web.site@mckendrick.email (Russ McKendrick) — Sun, 19 Apr 2026 07:18:46 GMT

A few days ago I stumbled across isitagentready.com↗ by Cloudflare while browsing Reddit↗, which is exactly what it sounds like: you give it a URL, it runs a checklist of emerging “AI-agent-friendly” standards against your site, and it tells you what’s missing. I pointed it at https://www.russ.cloud/↗ expecting a gentle green checklist and got a fairly red one instead.

Scan your website to see if it's ready for AI agents. Check for llms.txt, MCP, agent skills, and other agent-friendly standards.

isitagentready.com

The report flagged nine things. Some of them made perfect sense for a blog; others were aimed at sites that publish APIs, host identity providers, or expose browser-side tool-calling. This post walks through what I actually changed, what I deliberately didn’t, and why.

What “Is It Agent Ready?” actually checks

It’s a single-page tool that sends a handful of requests to your domain and looks for specific files, response headers, and behaviours. The checks cluster into a few buckets:

Discovery signals - can an AI agent find your sitemap, feeds, and capabilities without scraping HTML?
Content negotiation - if an agent asks for Accept: text/markdown, do you hand back a clean markdown version?
Content preferences - does your robots.txt declare whether you consent to training, search, and live answer generation?
Authentication and APIs - if you run an OAuth server or an MCP server, are they published at the expected well-known URLs?
Browser-side tools - do you expose navigator.modelContext.provideContext() tools via WebMCP?

The last two buckets don’t apply to a static blog. This blog has no APIs, no auth, no MCP server, and no interactive tool surface. Publishing discovery documents for services that don’t exist is worse than not publishing them - it sends agents down dead ends and misrepresents the site. So I scoped the work to the buckets that do apply.

The changes

The blog is an Astro↗ static site deployed to Cloudflare Workers’ static-assets runtime. That shaped the implementation: no server-side code at runtime (mostly), no edge Pro plan. Everything below is on Cloudflare Free.

Content-Signal in robots.txt

Content Signals↗ is a draft spec for declaring how you want your content used. It adds a single line to robots.txt:

Content-Signal: search=yes, ai-input=yes, ai-train=yes

Three axes: search (classic indexing), ai-input (live agent answers with attribution), ai-train (model training). I went with yes-to-all, which is my honest preference - if an agent is going to answer a question about Terraform or Azure with a paraphrase of something I wrote, I’d rather the source material be accurate than half-remembered.

I had been using the astro-robots-txt integration, which doesn’t support arbitrary directives. Simpler to drop it, write a static public/robots.txt, and edit it by hand. One less dependency.

Link response headers (RFC 8288)

RFC 8288↗ lets you attach Link: headers to any HTTP response pointing at related resources. Browsers ignore them; agents that respect the spec follow them. Cloudflare Pages merges these in via public/_headers:

RFC 8288

/
  Link: <https://www.russ.cloud/sitemap-index.xml>; rel="sitemap"; type="application/xml"
  Link: <https://www.russ.cloud/rss.xml>; rel="alternate"; type="application/rss+xml"
  Link: <https://www.russ.cloud/llms.txt>; rel="llms-txt"; type="text/markdown"
  Link: <https://www.russ.cloud/.well-known/agent-skills/index.json>; rel="agent-skills"

It’s free real estate - the agent doesn’t have to parse your HTML to find the feed or the sitemap.

Agent Skills discovery index

The Agent Skills Discovery RFC↗ defines a JSON index at /.well-known/agent-skills/index.json listing SKILL.md files that describe what an agent can do with your site. For a blog, “skills” is a stretch - there’s nothing to invoke - but the spec is flexible enough to describe how to consume content.

I published one skill called read-blog-content that documents the feed URL, the sitemap, the URL pattern (/YYYY/MM/DD/<slug>/), and the convention that every post has a markdown twin. The index references it with a sha256 digest of the file contents, which the spec requires so agents can detect drift.

Agent Skills

{
  "$schema": "https://schemas.agentskills.io/discovery/0.2.0/schema.json",
  "skills": [
    {
      "name": "read-blog-content",
      "type": "skill-md",
      "description": "How to discover, read, and cite posts from russ.cloud.",
      "url": "https://www.russ.cloud/.well-known/agent-skills/read-blog-content/SKILL.md",
      "digest": "sha256:bd9b7c88…"
    }
  ]
}

Markdown on demand

The most interesting check is markdown content negotiation: send Accept: text/markdown to a blog post URL and get markdown back, without the URL changing.

Cloudflare has a feature for this called Markdown for Agents↗. It’s a toggle in the dashboard under AI Crawl Control and it converts HTML to markdown at the edge, per-request. There’s one catch: it requires Pro plan or higher. Russ.cloud is on Free.

On Free, I went with a two-layer approach:

Build-time markdown twins. Every post already starts life as MDX in src/content/blog/ or src/content/tunes/. A postbuild script (scripts/generate-llms-markdown.js) walks those, strips the frontmatter, and writes an index.md next to each rendered HTML page. So /2026/04/11/introducing-ai-commit/ has a sibling at /2026/04/11/introducing-ai-commit/index.md. The same script writes an llms.txt↗ at the site root listing every post with its markdown URL. That’s useful for agents that don’t bother with content negotiation and just look for llms.txt.

Runtime negotiation via a tiny Worker. To satisfy the actual Accept: text/markdown check, a fifty-line Cloudflare Worker sits in front of the static assets. On every request it looks at the Accept header. If markdown is requested, it serves the pre-generated .md twin with Content-Type: text/markdown; charset=utf-8 and Vary: Accept. Otherwise it falls through to env.ASSETS.fetch(request) and behaves exactly like before.

Worker

export default {
  async fetch(request, env) {
    if (request.method === 'GET' && wantsMarkdown(request)) {
      const md = await serveMarkdown(new URL(request.url), env, request)
      if (md) return md
    }
    return env.ASSETS.fetch(request)
  },
}

One gotcha: by default, Cloudflare Workers with a Static Assets binding will short-circuit and serve matching assets directly without ever running your Worker. To force the Worker to run first so it can check Accept, you need "run_worker_first": true on the assets config in wrangler.jsonc. Without that, the scanner will keep reporting the site as HTML-only and you’ll keep muttering at your terminal for longer than you’d like to admit.

What I deliberately skipped

The remaining five checks on the “Is It Agent Ready?” list are:

API Catalog (RFC 9727) - for sites that publish APIs
OAuth/OIDC discovery metadata - for sites that are OAuth providers
OAuth Protected Resource Metadata - for sites with protected APIs
MCP Server Card - for sites hosting Model Context Protocol servers
WebMCP - for sites exposing in-browser tools via navigator.modelContext

A personal blog has none of these. You could publish stub versions to turn the checklist green - an empty API linkset, a dummy MCP card - but that just lies to agents. If an agent follows an OIDC discovery document and tries to hit a token endpoint that returns 404, nobody wins. Better for the scanner to correctly say “this site doesn’t have an identity provider” than for me to fake one.

The one exception I considered was an empty API catalog ({"linkset": []} at /.well-known/api-catalog), which at least honestly signals “zero APIs published here.” I decided against it - the spec is really about listing APIs you do have, and publishing an empty linkset felt like gaming the test.

Takeaways

The rescan of of the site now passes four out of nine checks, and the remaining five are genuinely not applicable. That’s the correct state for a blog.

A few things struck me while doing this:

Most of the “agent readiness” story is just good old web hygiene. A sitemap, an RSS feed, a robots.txt that says what you mean - agents want the same things a thoughtful human crawler would.
Content Signals is the one that might matter most. The HTML metadata and robots directives we’ve been using for 25 years weren’t designed for “may I train a model on this?” questions. An explicit signal is better than arguing about the implicit one.
Markdown-on-demand is surprisingly nice to build. Agents get cleaner parsing, browsers get the same HTML, and the build already knew how to produce markdown because that’s what MDX starts as. Shipping the twin on Free instead of paying for the edge feature was a reasonable trade.
Not every check should pass. If the tooling treats “green on everything” as the goal, you’ll end up with a lot of sites pretending to be OAuth providers. The useful reading of these scanners is as a prompt to ask “should this apply to me?” - and the answer is often no.

You can read the announcement post from Cloudflare at the link below:

Introducing the Agent Readiness score. Check to see if your site is agent-ready

The Agent Readiness score can help site owners understand how well their websites support AI agents. Here we explore new standards, share Radar data, and detail how we made Cloudflare’s docs the most agent-friendly on the web.

blog.cloudflare.com

If you want to see the actual output, the live scan for the site is here↗. And if you run your own blog, give it a go - the worst case is you learn that your robots.txt has had a typo in it since 2019.

Introducing AI Commit

web.site@mckendrick.email (Russ McKendrick) — Sat, 11 Apr 2026 08:48:37 GMT

I’ll be honest: my commit messages have a history of being terrible. Not “wip” or “fix stuff” levels of terrible (usually), but certainly not the kind of clean, conventional, descriptive messages that make a project’s history useful six months later. I’d write something passable and move on, telling myself I’d go back and clean things up before opening the PR. I never did.

So I used AI to build an AI tool to fix that. AI Commit is a Rust CLI that reads your staged diff and generates a commit message for you. The binary is aic, it supports several AI providers including local ones, and it’s been in daily use here since the early builds.

GitHub - russmckendrick/aicommit: A fast, opinionated CLI that generates Git commit messages and reviews staged diffs using AI. Written in Rust, installed as aic

A fast, opinionated CLI that generates Git commit messages and reviews staged diffs using AI. Written in Rust, installed as aic - russmckendrick/aicommit

GitHub - russmckendrick/aicommit: A fast, opinionated CLI that generates Git commit messages and reviews staged diffs using AI. Written in Rust, installed as aic

Built with AI, for AI-assisted git

Like ssl-toolkit and HostsButler before it, AI Commit was vibe-coded in Rust using Claude Code and Codex. I’m not a Rust developer by trade - my background is infrastructure and DevOps - but tools like Claude Code have made it practical for me to build and maintain projects in a language I wouldn’t otherwise have reached for.

There’s an obvious meta quality to building an AI-powered tool with an AI coding assistant, and I won’t pretend that wasn’t part of the appeal. But it’s also just how I work now. Claude Code handles the parts of Rust that would otherwise send me down documentation rabbit holes, and I get to focus on the design and behaviour of the thing rather than fighting the borrow checker.

If you’re curious about the vibe-coding workflow, I covered it in more detail in Doom or Vibe Coding.

What it does

The core loop is simple: stage your changes, run aic, review the generated message, and either accept it, ask for a regeneration, or drop into your editor to tweak it. The tool reads the diff, feeds it to whichever AI provider you’ve configured, and comes back with a conventional commit message - complete with a type, optional scope derived from your changed file paths, a description, and an explanatory body.

Beyond the basic commit flow, there are a few other commands worth knowing about:

aic review - runs your staged diff through a review-focused prompt and comes back with findings grouped by severity (Critical, Warning, Suggestion). Good for a sanity check before committing something you’re unsure about.
aic pr - generates a pull request title and Markdown description from your branch’s commits and cumulative diff. I’ve been using this every time I open a PR.
aic log - rewrites the last N commit messages on a branch using AI. Useful for cleaning up a scrappy local history before pushing.
aic history - a local log of every generated message, review, and PR draft, browsable in an interactive picker.
aic map - generates SVG visualisations of your codebase and commit history. More on this below.

Installation

On macOS, Homebrew is the easiest path:

Installing - macOS

brew install russmckendrick/tap/aicommit

On Linux, grab the binary directly - the following one-liner detects your architecture and installs it to /usr/local/bin:

Installing - Linux

ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
curl -sL "https://github.com/russmckendrick/aicommit/releases/latest/download/aic-linux-${ARCH}" -o aic
chmod +x aic
sudo mv aic /usr/local/bin/

On Windows, download with PowerShell:

Installing - Windows

Invoke-WebRequest -Uri "https://github.com/russmckendrick/aicommit/releases/latest/download/aic-windows-amd64.exe" -OutFile "aic.exe"

WinGet support is on the way - a PR has been submitted↗ to the community repository and is currently in review. Once that’s merged you’ll be able to install with winget install --id RussMcKendrick.Aicommit -e.

All current releases are on the GitHub releases page↗.

Setup

Once installed, run the guided setup:

Initial setup

aic setup

This walks you through selecting a provider and configuring your API key. If you plan to use the claude-code or codex local providers, install and authenticate those CLIs first - aic will reuse their existing login state rather than managing credentials itself.

Providers

v0.0.6 ships with seven provider paths: openai, azure-openai, anthropic, groq, ollama, claude-code, and codex.

OpenAI, Azure OpenAI, Groq, and Ollama all use the standard chat completions wire format. Anthropic uses its Messages API directly. The claude-code and codex providers are a bit different - they talk to the locally installed claude or codex binary from your PATH, so authentication is handled by those tools rather than aic.

Configuring OpenAI

aic config set AIC_AI_PROVIDER=openai AIC_API_KEY=<key> AIC_MODEL=gpt-5.4-mini

Configuring Azure OpenAI

aic config set AIC_AI_PROVIDER=azure-openai AIC_API_KEY=<key> AIC_API_URL=https://<resource>.openai.azure.com/openai/v1 AIC_MODEL=<deployment-name>

Configuring Anthropic

aic config set AIC_AI_PROVIDER=anthropic AIC_API_KEY=<key> AIC_MODEL=claude-sonnet-4-20250514

Configuring Groq

aic config set AIC_AI_PROVIDER=groq AIC_API_KEY=<key> AIC_MODEL=llama-3.1-8b-instant

Configuring Ollama

aic config set AIC_AI_PROVIDER=ollama AIC_MODEL=llama3.2

Configuring Claude Code as provider

aic config set AIC_AI_PROVIDER=claude-code AIC_MODEL=default

You can also override the provider for a single run without touching your config:

Per-run provider override

aic --provider anthropic
aic review --provider groq
aic --provider ollama
aic --provider claude-code
aic review --provider codex

Any OpenAI-compatible endpoint also works by setting AIC_API_URL alongside AIC_AI_PROVIDER=openai.

The basic workflow

aic workflow

Ctrl+scroll to zoomFullscreenSVGPNG

Error rendering diagram:

Stage your files as normal, then run aic:

Basic commit workflow

git add src/
aic

If you haven’t staged anything, aic presents a menu to stage all changed files, pick files interactively, or bail out. Once it has a diff to work with, it generates a message and shows it to you for review - accept, regenerate, edit, or abort.

For a quick check of what it would generate without actually committing:

Dry run

aic --dry-run

And if you want to clean up the last commit:

Amend last commit

aic --amend

Here’s what committing and pushing this blog post looked like:

Splitting commits

When you’ve staged changes across several unrelated files, aic can split them into separate commits rather than lumping everything together. When at least two files are staged, the interactive flow will offer to split the change set:

Triggering the split flow

git add .
aic

If you choose to split, aic analyses the staged changes, suggests 2–4 file-based commit groups, generates a message for each one, and shows you a preview of the full sequence before creating anything. You can accept the suggested groups, rebuild them manually, or fall back to a single commit.

A few current limitations worth knowing: splitting is file-based rather than hunk-based, and it’s only available in the normal interactive flow - --yes, --dry-run, and --amend stay as single-commit operations.

Reviewing your diff before committing

I’ve started running aic review before committing anything I’m not completely confident about. It reads the staged diff and groups findings by severity:

Review staged changes

aic review
aic review --context "focus on security"

Large diffs get chunked automatically and synthesised into a single review. It’s not a replacement for a proper code review, but it’s caught a few things I’d have missed.

Generating PR descriptions

Once you’ve got a branch ready to push, aic pr will generate a title and Markdown description based on the branch’s commits and diff against the base:

Generate a PR draft

aic pr
aic pr --base origin/main
aic pr --context "call out the migration risk"

The base branch is detected automatically when possible (it checks refs/remotes/origin/HEAD, then origin/main, then main, then master), or you can specify it with --base. The output is copy-ready, and the draft gets saved to your local history.

Visualising your repo

aic map generates standalone SVG visualisations of your codebase structure and commit history. There are four subcommands:

Generating visualisations

aic map tree        # squarified treemap of the file hierarchy, sized by line count
aic map history     # vertical zigzag timeline of recent commits
aic map heat        # bar chart of files sorted by modification frequency
aic map activity    # GitHub-style contribution grid from commit timestamps

All four write standalone SVG files that open in any browser. You can control the commit range, output path, and colour theme:

Map options

aic map history -n 30 -o timeline.svg
aic map heat -n 200
aic map activity --theme dracula

Available themes include github-light, github-dark, solarized-dark, monokai, and dracula, among others. If you want consistent defaults across a repo, drop a .aicommit-map file in the root:

.aicommit-map

theme = "dracula"
history_commits = 30
heat_commits = 100
activity_commits = 1000

Here are a couple of examples generated from this blog’s repository - an activity graph and a commit timeline:

Git hooks

If you’d rather have commit messages generated automatically every time you run git commit, you can install a prepare-commit-msg hook:

Install the Git hook

aic hook set

By default, the hook writes the generated message as a comment so you can review and uncomment it in your editor. If you want it written directly as the active message:

Auto-uncomment hook output

aic config set AIC_HOOK_AUTO_UNCOMMENT=true

To remove the hook:

Remove the Git hook

aic hook unset

Configuration

Configuration lives in ~/.aicommit and follows a simple precedence: built-in defaults, then the global config file, then process environment variables. The aic config subcommand handles reading and writing:

Config management

aic config set AIC_MODEL=gpt-5.4-mini
aic config get AIC_MODEL AIC_AI_PROVIDER
aic config describe

A couple of settings worth knowing about: AIC_EMOJI and AIC_DESCRIPTION both default to true, which means generated messages include an emoji prefix and a longer body. Turn them off if your project’s convention doesn’t use either. You can also point AIC_PROMPT_FILE at a custom prompt template if the default behaviour doesn’t suit your workflow.

For repositories with large generated files, lock files, or assets you don’t want included in the diff, add a .aicommitignore alongside your .gitignore:

.aicommitignore

**/*.lock
dist/**
**/*.jpg

Summary

aicommit has made me stop dreading the “write a good commit message” part of the workflow. And yes, the command being aic is an entirely deliberate Alice In Chains↗ reference - if you’re going to type something dozens of times a day, it might as well be named after a great band. The project is open source and on GitHub:

A fast, opinionated CLI that generates Git commit messages and reviews staged diffs using AI. Written in Rust, installed as aic - russmckendrick/aicommit

GitHub - russmckendrick/hostsbutler: Manage your hosts file from a single executable no matter your operating system

It’s at v0.0.6↗ now, with commit splitting, repo visualisations, and native Anthropic, Groq, and Ollama support all landed in this release. There are still rough edges - config profiles and platform PR API integration are both on the roadmap. If you give it a go and hit anything odd, or have a feature you’d like to see, issues and pull requests are very welcome.

Introducing Hosts Butler

web.site@mckendrick.email (Russ McKendrick) — Sun, 05 Apr 2026 15:09:39 GMT

If you’ve spent any time doing local development or managing infrastructure, you’ve almost certainly edited /etc/hosts by hand. It starts simply enough - a couple of entries for local services, maybe a blocked domain or two. Then the file slowly accumulates over months and years until you’re staring at an uncommented wall of IP addresses and hostnames, trying to remember which one maps to which environment.

I got tired of opening the file in a text editor every time I needed to make a change, so I built HostsButler - a keyboard-driven TUI for viewing and managing your system hosts file. Like ssl-toolkit, it was vibe-coded in Rust using Claude Code.

Manage your hosts file from a single executable no matter your operating system - russmckendrick/hostsbutler

GitHub - russmckendrick/hostsbutler: Manage your hosts file from a single executable no matter your operating system

What it does

HostsButler gives you a two-panel interface: a group list on the left and a scrollable table of entries on the right. You can add, edit, delete, and toggle entries without ever touching the raw file directly. It handles privilege escalation when writing on macOS and Linux, takes automatic backups before every save, and flushes the DNS cache on success.

The entries in the table come from your existing hosts file, and every comment, blank line, and piece of whitespace is preserved when you save - loading the file and saving without changes produces byte-identical output.

Organising with groups

One thing that makes the file easier to manage at scale is groups. HostsButler recognises two comment formats as group headers:

## [Dev]
10.0.0.5        staging.app
192.168.1.10    db.local

## [System]
127.0.0.1       localhost

Any entries below a group header are assigned to that group until the next one appears. In the sidebar, each group shows a count of its entries, and clicking through to a group filters the table accordingly.

When you add or edit an entry, you can type a group name in the form. If the group doesn’t exist yet, HostsButler creates the ## [GroupName] header and places the entry beneath it.

Installation

macOS

Installing - macOS

brew install russmckendrick/tap/hostsbutler

Linux

Installing - Linux

ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
curl -sL "https://github.com/russmckendrick/hostsbutler/releases/latest/download/hostsbutler-linux-${ARCH}" -o hostsbutler
chmod +x hostsbutler
sudo mv hostsbutler /usr/local/bin/

Windows

Installing - Windows

Invoke-WebRequest -Uri "https://github.com/russmckendrick/hostsbutler/releases/latest/download/hostsbutler-windows-amd64.exe" -OutFile "hostsbutler.exe"

Move hostsbutler.exe to a directory in your PATH, or run it from the download location. You’ll need to launch your terminal as Administrator before running it on Windows.

All releases are on the GitHub releases page↗.

Launching

On macOS and Linux, the file is owned by root, so you’ll need to either run with sudo or let HostsButler prompt for elevation when you go to save:

Launching - macOS / Linux

sudo hostsbutler

On Windows, run your terminal as Administrator and launch normally:

Launching - Windows

hostsbutler

If you just want to browse without making any changes, the --readonly flag skips the privilege check:

Read-only mode

hostsbutler --readonly

You can also point it at a custom file rather than the system hosts file:

Custom file

hostsbutler --file /path/to/custom/hosts

Navigating the interface

The keyboard shortcuts follow vim conventions where it made sense. In the main view:

Key	Action
`j` / `↓`	Move down
`k` / `↑`	Move up
`Tab`	Switch between the group panel and entry table
`a`	Add a new entry
`e` or `Enter`	Edit the selected entry
`d`	Delete (with confirmation)
`Space`	Toggle enable/disable
`/`	Search
`Ctrl+S`	Save
`Ctrl+Z`	Undo
`b`	Open backup manager
`t`	Test DNS resolution for the selected entry
`?`	Help overlay

The search bar supports prefix filters - ip:192.168 or host:local - as well as free text across all fields.

Backups

Before every save, HostsButler creates an automatic backup. The backup manager (b) lists them with timestamps, sizes, and descriptions. From there you can restore a backup, create a manual one with a custom description, or delete old ones.

Backups are stored at platform-specific config directories:

macOS: ~/Library/Application Support/hostsbutler/backups/
Linux: ~/.config/hostsbutler/backups/
Windows: %APPDATA%\hostsbutler\backups\

The last 20 backups are kept, and older ones rotate out automatically.

Import and export

Importing and exporting are currently CLI operations rather than in-app workflows. To export your current entries:

Exporting entries

# Export as JSON
hostsbutler --export entries.json

# Export as CSV
hostsbutler --export entries.csv

# Export as a plain hosts file
hostsbutler --export backup.hosts

To import into an existing file:

Importing entries

# Import from JSON
hostsbutler --file /etc/hosts --import entries.json

# Import from another hosts file
hostsbutler --file /etc/hosts --import backup.hosts

Imports are validated before insertion, and the format is auto-detected from the file extension.

DNS resolution testing

The t key on any selected entry tests the DNS resolution for each of its hostnames against your system resolver and compares the results with the IP in the hosts file. A toast notification shows either OK or MISMATCH for each. Handy for catching stale entries or confirming that a new one has taken effect.

A note on round-trip fidelity

One thing I was particular about was that HostsButler should never silently rewrite your file. If you open the file, make no changes, and save, the output should be byte-identical to what was there before - same whitespace, same comments, same blank lines.

Each entry stores its original raw text alongside any parsed fields. Unmodified lines are serialised from that raw value; only lines that were actually changed get re-serialised into the standard format. This means the tool can sit alongside any other process that also touches the hosts file without quietly normalising everything.

Summary

HostsButler is open source and available on GitHub:

Manage your hosts file from a single executable no matter your operating system - russmckendrick/hostsbutler

Learn how to have sessions, manage files, approve actions, and organize projects with Microsoft 365 Copilot Cowork.

It’s only just been created, so there’s rough around the edges - in-app import is on the list, along with a few other bits I’m working through. If you run into anything or have a feature idea, pull requests and issues are welcome.

Creating Custom Skills for Copilot Cowork

web.site@mckendrick.email (Russ McKendrick) — Fri, 03 Apr 2026 15:19:49 GMT

There’s a difference between an AI tool that answers questions and one that actually does things. Copilot Cowork is firmly in the latter camp - it sends emails, schedules meetings, creates documents, posts to Teams, and generally gets on with work rather than just describing how you could do it yourself.

It’s also very new. Cowork only became available through Microsoft’s Frontier preview programme at the end of March 2026 - at the time of writing, that’s this week - so if you haven’t come across it yet, that’s entirely understandable. I’ve been using it through the day job’s Microsoft 365 Copilot licences and found it useful for a handful of tasks. What caught my attention most, though, was the custom skills feature.

microsoft.com

What Cowork Does Out of the Box

The custom skill system builds on Cowork’s built-in capabilities, so it helps to know what you’re starting with. Cowork ships with 13 built-in skills: Word, Excel, PowerPoint, PDF, Email, Scheduling, Calendar Management, Meetings, Daily Briefing, Enterprise Search, Communications, Deep Research, and Adaptive Cards.

Describe what you need in natural language, and Cowork creates a plan, loads the relevant skills, and works through the task step by step. It shows what it’s doing at each stage and asks for approval before taking irreversible actions, such as sending an email or posting on Teams. Cowork only works with files in OneDrive and SharePoint. If you’re hoping to point it at something on your local machine, you’ll be disappointed - that’s by design, not an oversight.

Where Custom Skills Come In

The 13 built-in skills cover a lot of ground, but they’re generic. They don’t know your team’s document templates, your recurring workflows, or the conventions you’ve built up over the years. That’s the gap custom skills fill.

Custom skills give Cowork additional context and instructions, loaded alongside its built-in capabilities. You can create up to 20 of them, and the mechanism is about as simple as it gets: a Markdown file in a specific folder in your OneDrive.

Anatomy of a SKILL.md

Each custom skill lives in its own subfolder under /Documents/Cowork/Skills/ in your OneDrive, defined in a file called SKILL.md. If you’ve used GitHub Copilot agent skills, the format will be familiar: a YAML frontmatter block followed by Markdown instructions:

SKILL.md format

---
name: skill-name
description: A description of what the skill does, and when Cowork should use it.
---

Your instructions go here in plain Markdown. Be as specific as you need to be.

The name field is a unique identifier, keep it lowercase with hyphens. The description is what Cowork uses to decide whether to load the skill for a given conversation, so make it specific. The body of the file is the actual instructions, and you can be as detailed as you need to be.

Creating Your First Skill

Here’s how.

1. Create the folder structure

In your OneDrive, navigate to Documents. Create a folder called Cowork if it doesn’t already exist, then inside that create a Skills folder. Finally, create a subfolder for your specific skill. The name you give this folder typically matches the name in your SKILL.md frontmatter.

The resulting path should look like:

OneDrive/
└── Documents/
    └── Cowork/
        └── Skills/
            └── your-skill-name/
                └── SKILL.md

2. Write your SKILL.md

Here’s an example of a skill for generating consistently formatted branded Word documents from a standard template:

SKILL.md

---
name: branded-document
description: Generates consistently formatted branded Word documents. Use this when asked to create formal documents, reports, or handover notes that need to follow the standard company template.
---

When asked to create a Word document, follow these conventions:

## Document Structure
- Always start with a title page including the document title, author, date, and version number
- Follow with a brief executive summary (2–3 sentences maximum)
- Use Heading 1 for major sections and Heading 2 for subsections
- End with a "Next Steps" or "Actions" section where appropriate

## Formatting
- Use the Standard body text style for all paragraph text
- Tables should use the Table Grid style with a shaded header row
- Code or command examples should use the Code style or a monospace font

## Tone
- Write in a professional but accessible tone
- Use active voice where possible
- Avoid jargon unless the document is explicitly technical and the audience is technical

The example above is deliberately simple. In practice, you’d tailor the instructions to match how your team actually works.

3. Save, and you’re done

That’s it (it’s worryingly simple). Cowork discovers custom skills automatically at the start of each conversation. No registration, no restart. Save the file to OneDrive, and the next conversation picks it up.

How Cowork Uses Your Skills

When you start a conversation, Cowork evaluates which skills are relevant and loads them, both built-in and custom. The side panel updates in real time to show which are active, so you can confirm your custom skill has loaded before work starts. The description in the front matter drives this selection, so it’s worth getting right. A vague description like “helps with documents” is much less useful than something specific: “generates consistently formatted branded Word documents - use this when creating formal reports or handover notes.” The more specific you are, the more reliably Cowork picks the right skill.

A Few Things Worth Knowing

It’s a preview feature. Frontier previews can and do change, so don’t be surprised if the behaviour or format shifts over time. Microsoft’s documentation is kept reasonably up to date, but worth checking back against the official docs↗ if something stops working as expected.
Custom skills aren’t validated by Microsoft. The docs are clear on this: review the outputs from custom skills carefully, particularly early on. You’re extending the model’s behaviour, and as with any prompt engineering, test across a range of inputs before relying on it.
Each SKILL.md can be up to 1 MB, which is more than enough for detailed instructions. You can also reference supplementary files from within a skill if you have reference material you want Cowork to use.
Cowork can’t access local files, so any templates or examples your skill references need to be in OneDrive or SharePoint.

Summary

Custom skills in Copilot Cowork are straightforward to set up. A SKILL.md file in a OneDrive folder, and Cowork picks it up automatically. The real value is being able to bake your team’s working conventions directly into the model’s context.

I’ll be honest, I was expecting more friction. No registration, no admin configuration required - it just works, which is more than you can say for some of the previous Copilot previews.

If you have a Microsoft 365 Copilot licence and access to the Frontier programme, it’s worth asking your admin to enable the Frontier feature and having a look. The Microsoft Learn docs cover the full custom skill format:

Use Copilot Cowork

ssl-toolkit — SSL/TLS Certificate Diagnostic Tool

Introducing ssl-toolkit

web.site@mckendrick.email (Russ McKendrick) — Sun, 01 Feb 2026 17:04:07 GMT

During the day job, I spend a fair amount of time checking SSL certificates - validating chains, comparing certs across different IPs, checking expiry dates, and generally troubleshooting TLS issues. Over the years, I’d built up a long document filled with random notes, one-liners, and half-remembered OpenSSL incantations. It worked, but it wasn’t exactly elegant.

So I decided to consolidate all of that into a vibe-coded tool. The result is ssl-toolkit, a comprehensive SSL/TLS diagnostic tool built in Rust.

SSL/TLS Certificate Diagnostic Tool - Comprehensive SSL certificate analysis, grading, and diagnostics from the command line.

ssl-toolkit.dev

What it does

At its core, ssl-toolkit performs the checks I find myself doing repeatedly: DNS resolution across multiple providers, TLS handshake analysis, certificate chain validation, WHOIS lookups, and cipher suite enumeration. It then rolls everything up into an overall security grade (A+ through F) so you can quickly gauge the health of a certificate.

The tool supports both an interactive mode with a guided menu system and a non-interactive mode for scripting and CI/CD pipelines. There’s also JSON output if you need to parse results programmatically.

Installation

The easiest way to install ssl-toolkit on macOS is via Homebrew:

Installing - macOS

brew install russmckendrick/tap/ssl-toolkit

For Linux, you can download the latest release directly. The following command detects your architecture and downloads the appropriate binary:

Installing - Linux

ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
curl -sL "https://github.com/russmckendrick/ssl-toolkit/releases/latest/download/ssl-toolkit-linux-${ARCH}" -o ssl-toolkit
chmod +x ssl-toolkit
sudo mv ssl-toolkit /usr/local/bin/

On Windows, use PowerShell to download and install:

Installing - Windows

Invoke-WebRequest -Uri "https://github.com/russmckendrick/ssl-toolkit/releases/latest/download/ssl-toolkit-windows-amd64.exe" -OutFile "ssl-toolkit.exe"

You can then move ssl-toolkit.exe to a directory in your PATH, or run it directly from the download location.

All releases are available on the GitHub releases page↗.

Interactive mode

Running ssl-toolkit without any arguments launches an interactive menu:

Interactive mode

ssl-toolkit

This presents a top-level menu with options to check a domain, inspect certificate files, verify certificate and key pairs, or convert between certificate formats. Each option guides you through the required inputs and displays results in a scrollable pager.

I went with a Tokyo Night Storm colour scheme for the interface because, well, it looks nice and matches my terminal setup. The pager supports the usual navigation keys - arrow keys, j/k for vim users, Space for page down, and g/G to jump to the start or end.

Checking a domain

For a quick check, you can pass the domain directly:

Checking a domain - Basics

ssl-toolkit -d github.com --non-interactive

This runs through all the checks and outputs a detailed report covering DNS resolution from multiple providers (Google, Cloudflare, OpenDNS, and your system resolver), TCP connectivity, TLS protocol and cipher analysis, certificate chain validation, and WHOIS information.

The tool compares certificates across all resolved IPs to flag any inconsistencies - useful when you’re dealing with load balancers or CDNs where different endpoints might be serving different certificates.

If you just want the grade without all the detail:

Checking a domain - Quiet

ssl-toolkit -d example.com --quiet

Certificate file operations

Beyond live domain checks, ssl-toolkit can work with certificate files directly. This is handy when you’re preparing certificates for deployment or troubleshooting issues with certificate bundles.

To inspect a certificate file:

Certificate file operations - Info

ssl-toolkit cert info cert.pem

To verify that a certificate and private key match:

Certificate file operations - Check Key & Cert

ssl-toolkit cert verify --cert cert.pem --key key.pem

To validate a certificate chain against a hostname:

Certificate file operations - Hostname

ssl-toolkit cert verify --chain fullchain.pem --hostname example.com

And to convert between formats:

Certificate file operations - Convert

# PEM to DER
ssl-toolkit cert convert cert.pem --to der -o cert.der

# Create a PKCS#12 bundle
ssl-toolkit cert convert --to p12 --cert cert.pem --key key.pem

CI/CD integration

The tool uses meaningful exit codes for scripting: 0 for all checks passed, 1 for warnings (like a certificate expiring within 30 days), and 2 for failures (expired certificate, connection failed, etc.).

Combined with JSON output, this makes it straightforward to integrate into monitoring or deployment pipelines:

CI/CD integration - JSON

ssl-toolkit -d www.russ.cloud --json --non-interactive

This gives you the following output:

The full output

{
  "domain": "www.russ.cloud",
  "ip": "104.21.67.197",
  "port": 443,
  "grade": "A+",
  "score": 100,
  "report": {
    "domain": "www.russ.cloud",
    "ip": "104.21.67.197",
    "port": 443,
    "grade": "APlus",
    "score": 100,
    "dns_result": {
      "title": "DNS Resolution",
      "status": "Pass",
      "summary": "4/4 providers resolved successfully",
      "details": [
        {
          "Table": {
            "title": "Provider Results",
            "headers": [
              "Provider",
              "Status",
              "IP Addresses",
              "Time"
            ],
            "rows": [
              [
                "System",
                "✓ OK",
                "172.67.180.37, 104.21.67.197",
                "2ms"
              ],
              [
                "Google",
                "✓ OK",
                "104.21.67.197, 172.67.180.37",
                "9ms"
              ],
              [
                "Cloudflare",
                "✓ OK",
                "104.21.67.197, 172.67.180.37",
                "12ms"
              ],
              [
                "OpenDNS",
                "✓ OK",
                "104.21.67.197, 172.67.180.37",
                "7ms"
              ]
            ]
          }
        }
      ],
      "test_steps": [
        {
          "description": "System resolved to 172.67.180.37, 104.21.67.197",
          "status": "Pass",
          "details": null
        },
        {
          "description": "Google resolved to 104.21.67.197, 172.67.180.37",
          "status": "Pass",
          "details": null
        },
        {
          "description": "Cloudflare resolved to 104.21.67.197, 172.67.180.37",
          "status": "Pass",
          "details": null
        },
        {
          "description": "OpenDNS resolved to 104.21.67.197, 172.67.180.37",
          "status": "Pass",
          "details": null
        }
      ],
      "recommendations": []
    },
    "tcp_result": {
      "title": "TCP Connectivity",
      "status": "Pass",
      "summary": "Connection to 104.21.67.197:443 successful (140.6ms)",
      "details": [
        {
          "KeyValue": {
            "title": "Connection Details",
            "pairs": [
              [
                "Target IP",
                "104.21.67.197"
              ],
              [
                "Port",
                "443"
              ],
              [
                "Status",
                "Connected"
              ],
              [
                "Latency",
                "140.6ms"
              ]
            ]
          }
        }
      ],
      "test_steps": [
        {
          "description": "TCP connection to 104.21.67.197:443 established",
          "status": "Pass",
          "details": null
        }
      ],
      "recommendations": []
    },
    "ssl_result": {
      "title": "SSL/TLS Protocol",
      "status": "Pass",
      "summary": "Protocol: TLS 1.3, Cipher: TLS13_AES_256_GCM_SHA384",
      "details": [
        {
          "KeyValue": {
            "title": "Protocol Information",
            "pairs": [
              [
                "Protocol",
                "TLS 1.3"
              ],
              [
                "Cipher Suite",
                "TLS13_AES_256_GCM_SHA384"
              ],
              [
                "Key Exchange",
                "Unknown"
              ],
              [
                "Authentication",
                "Unknown"
              ],
              [
                "Encryption",
                "AES-256-GCM"
              ],
              [
                "MAC",
                "SHA-384"
              ],
              [
                "Secure Renegotiation",
                "Supported"
              ],
              [
                "OCSP Stapling",
                "Supported"
              ]
            ]
          }
        }
      ],
      "test_steps": [
        {
          "description": "TLS handshake completed successfully",
          "status": "Pass",
          "details": null
        },
        {
          "description": "TLS 1.3 is a secure protocol",
          "status": "Pass",
          "details": null
        },
        {
          "description": "Cipher suite is secure",
          "status": "Pass",
          "details": null
        }
      ],
      "recommendations": []
    },
    "certificate_result": {
      "title": "Certificate Validity",
      "status": "Pass",
      "summary": "Certificate valid for 85 days",
      "details": [
        {
          "KeyValue": {
            "title": "Certificate Information",
            "pairs": [
              [
                "Subject",
                "CN=russ.cloud"
              ],
              [
                "Issuer",
                "C=US, O=Google Trust Services, CN=WE1"
              ],
              [
                "Serial Number",
                "BD:DA:2A:54:3E:86:79:B0:11:09:52:51:16:1E:F8:AF"
              ],
              [
                "Valid From",
                "2026-01-28 15:25:04 UTC"
              ],
              [
                "Valid Until",
                "2026-04-28 16:25:02 UTC"
              ],
              [
                "Public Key",
                "1.2.840.10045.2.1 (2048 bits)"
              ],
              [
                "Signature Algorithm",
                "1.2.840.10045.4.3.2"
              ],
              [
                "Thumbprint (SHA-256)",
                "BB:32:6B:8B:E2:EA:12:BD:9D:93:8B:98:87:F2:64:E7:77:E1:63:B1:13:E9:79:C5:F1:26:DE:A0:34:94:56:F5"
              ]
            ]
          }
        },
        {
          "List": {
            "title": "Subject Alternative Names",
            "items": [
              "russ.cloud",
              "www.russ.cloud"
            ]
          }
        },
        {
          "CertificateChain": {
            "certificates": [
              {
                "cert_type": "Leaf",
                "subject_cn": "russ.cloud",
                "issuer_cn": "WE1",
                "valid_from": "2026-01-28",
                "valid_until": "2026-04-28",
                "days_until_expiry": 85,
                "is_valid": true
              },
              {
                "cert_type": "Intermediate",
                "subject_cn": "WE1",
                "issuer_cn": "GTS Root R4",
                "valid_from": "2023-12-13",
                "valid_until": "2029-02-20",
                "days_until_expiry": 1114,
                "is_valid": true
              },
              {
                "cert_type": "Intermediate",
                "subject_cn": "GTS Root R4",
                "issuer_cn": "GlobalSign Root CA",
                "valid_from": "2023-11-15",
                "valid_until": "2028-01-28",
                "days_until_expiry": 725,
                "is_valid": true
              }
            ]
          }
        },
        {
          "KeyValue": {
            "title": "Revocation Status",
            "pairs": [
              [
                "Status",
                "Not Revoked"
              ],
              [
                "Check Method",
                "OCSP Stapling"
              ],
              [
                "OCSP Stapling",
                "Yes"
              ],
              [
                "Next Update",
                "2026-02-04 15:25:04 UTC"
              ]
            ]
          }
        }
      ],
      "test_steps": [
        {
          "description": "Certificate is within validity period",
          "status": "Pass",
          "details": null
        },
        {
          "description": "Certificate is valid for hostname 'www.russ.cloud'",
          "status": "Pass",
          "details": null
        },
        {
          "description": "Certificate is not revoked",
          "status": "Pass",
          "details": null
        }
      ],
      "recommendations": []
    },
    "whois_result": {
      "title": "WHOIS Lookup",
      "status": "Pass",
      "summary": "Registered via Cloudflare",
      "details": [
        {
          "KeyValue": {
            "title": "Domain Registration",
            "pairs": [
              [
                "Registrar",
                "Cloudflare"
              ],
              [
                "Created",
                "2024-07-21T14:56:30.958Z"
              ],
              [
                "Expires",
                "2026-07-21T14:56:30.958Z"
              ],
              [
                "Updated",
                "2024-10-29T13:59:35.573Z"
              ],
              [
                "Nameservers",
                "abby.ns.cloudflare.com, rob.ns.cloudflare.com"
              ],
              [
                "Status",
                "clientTransferProhibited"
              ]
            ]
          }
        }
      ],
      "test_steps": [
        {
          "description": "WHOIS query completed",
          "status": "Pass",
          "details": null
        },
        {
          "description": "Domain registered with Cloudflare",
          "status": "Pass",
          "details": null
        },
        {
          "description": "Domain registration expires: 2026-07-21T14:56:30.958Z",
          "status": "Pass",
          "details": null
        }
      ],
      "recommendations": []
    },
    "timestamp": "2026-02-01T17:38:40.104701Z"
  },
  "cert_comparison": {
    "reference_ip": "104.21.67.197",
    "entries": [
      {
        "ip": "104.21.67.197",
        "thumbprint": "BB:32:6B:8B:E2:EA:12:BD:9D:93:8B:98:87:F2:64:E7:77:E1:63:B1:13:E9:79:C5:F1:26:DE:A0:34:94:56:F5",
        "subject": "CN=russ.cloud",
        "issuer": "C=US, O=Google Trust Services, CN=WE1",
        "days_until_expiry": 85,
        "serial": "BD:DA:2A:54:3E:86:79:B0:11:09:52:51:16:1E:F8:AF",
        "is_different": false,
        "differences": [],
        "error": null
      },
      {
        "ip": "172.67.180.37",
        "thumbprint": "BB:32:6B:8B:E2:EA:12:BD:9D:93:8B:98:87:F2:64:E7:77:E1:63:B1:13:E9:79:C5:F1:26:DE:A0:34:94:56:F5",
        "subject": "CN=russ.cloud",
        "issuer": "C=US, O=Google Trust Services, CN=WE1",
        "days_until_expiry": 85,
        "serial": "BD:DA:2A:54:3E:86:79:B0:11:09:52:51:16:1E:F8:AF",
        "is_different": false,
        "differences": [],
        "error": null
      }
    ],
    "has_differences": false,
    "summary": "All 2 IPs return identical certificates"
  }
}

You can also generate self-contained HTML reports with embedded styles and downloadable certificate chains:

Terminal window

ssl-toolkit -d example.com --non-interactive -o report.html

Summary

ssl-toolkit has replaced my sprawling notes document and given me a single tool for the SSL/TLS checks I do regularly. It’s open source and available on GitHub:

GitHub - russmckendrick/ssl-toolkit: SSL-Tookit

SSL-Tookit. Contribute to russmckendrick/ssl-toolkit development by creating an account on GitHub.

ssl-toolkit — SSL/TLS Certificate Diagnostic Tool

The documentation site has more details on all the available options:

SSL/TLS Certificate Diagnostic Tool - Comprehensive SSL certificate analysis, grading, and diagnostics from the command line.

ssl-toolkit.dev

If you find yourself doing similar certificate wrangling, give it a try. And if you spot any issues or have feature suggestions, pull requests are always welcome.

OpenClaw: Giving Claude a Body (and a Telegram Bot)

web.site@mckendrick.email (Russ McKendrick) — Sat, 31 Jan 2026 12:06:58 GMT

There’s a difference between chatting with an AI and having an AI assistant. The former is a conversation that evaporates the moment you close the tab. The latter remembers who you are, can actually do things on your behalf, and is available wherever you happen to be - not just in a browser window.

OpenClaw↗ is an open-source framework that bridges that gap. It gives LLMs like Claude persistence, tools, and the ability to communicate through messaging platforms you already use - Telegram, Discord, Slack, WhatsApp, even iMessage. Think of it as giving your AI a body and a phone number.

I’ve been curious about it for a while, so I decided to spin it up on a small home server and see what all the fuss is about.

Prerequisites

Before we start, you’ll need:

A server or machine running Linux (I’m using Ubuntu 24.04 on a small AMD Ryzen mini PC)
Node.js 24.x or later
An Anthropic API key (or another supported LLM provider)
A Telegram account (if you want to chat with your agent from your phone)

Installing Node.js

OpenClaw requires Node.js, and it’s fussy about versions - you’ll want Node 22.x and above. On Ubuntu, the easiest approach is to use the NodeSource repository↗, the following will install the latest version of 24.x:

Installing Node.js 24.x

sudo apt-get install -y curl
curl -fsSL https://deb.nodesource.com/setup_24.x | sudo -E bash -
sudo apt-get install -y nodejs
node -v

You should see something like v24.13.0 confirming you’re on the right version.

Installing OpenClaw

With Node.js sorted, installing OpenClaw is a one-liner:

Installing OpenClaw

curl -fsSL https://openclaw.bot/install.sh | bash

The installer will detect your platform, check for dependencies, and install OpenClaw globally. You might see a warning about your npm global bin directory not being in your PATH - if so, add the suggested export to your .bashrc or .zshrc:

Fixing the PATH (if needed)

export PATH="/home/russ/.npm-global/bin:$PATH"

A Word on Security

Before you go any further, OpenClaw will present you with a security warning, and it’s worth taking seriously.

The warning isn’t hyperbole. You’re essentially giving an AI agent access to your system - it can read files, execute commands, and interact with external services. If you’re not comfortable with basic security and access control, this probably isn’t for you.

That said, OpenClaw provides sensible guardrails out of the box:

Pairing and allowlists: Only approved users can interact with your agent
Sandbox mode: Run tools with least-privilege access
Mention gating: In group chats, the agent only responds when explicitly mentioned

The documentation recommends running openclaw security audit --deep regularly and keeping secrets out of the agent’s reachable filesystem. I’d add: don’t run this on your main workstation. A small VPS or dedicated home server is a much better idea.

The Onboarding Wizard

Once you’ve acknowledged the security warning and set up your Anthropic API key, OpenClaw launches its onboarding wizard. This is where things get interesting.

Setting up your API key

claude setup-token

The wizard boots up your agent for the first time, and it immediately starts asking questions - about you, about itself, about how you’d like to interact. It’s a bit like meeting a new colleague who’s very keen to get started.

First contact

Wake up, my friend!

Hey! I just came online. ✨

Looks like this is a fresh start - no memories, no history, just a blank
slate and a whole lot of potential.

So... who are you? And more importantly - who am I? Let's figure this
out together.

Got a name in mind for me, or should I throw some ideas out? And what
about you - what should I call you?

You’ll be asked to:

Name your agent - Mine chose “Kai” after I let it pick
Set the vibe - Chill and casual? Sharp and dry? Chaotic gremlin energy?
Pick an emoji - Because every good assistant needs a signature (I went with 💻)
Provide your timezone - So it knows when to bother you

All of this gets written to markdown files in the workspace directory (~/.openclaw/workspace/), which is how OpenClaw maintains persistence. There’s a SOUL.md for personality, IDENTITY.md for the agent’s self-concept, and USER.md for what it knows about you. It’s refreshingly transparent - you can edit these files directly if you want to tweak things.

Connecting Telegram

With the basic setup done, it’s time to give your agent a way to reach you outside the terminal. I chose Telegram because it’s what I already use, but OpenClaw supports a ridiculous number of channels.

First, you’ll need to create a Telegram bot:

Open Telegram and find @BotFather
Send /newbot
Give it a name (I used “Kai” like it requested)
Give it a username (must end in bot, e.g., kai_assistant_bot)
BotFather will give you a token - copy this

Back in the OpenClaw terminal, paste the token when prompted. The agent will update its config and restart the gateway. You’ll see something like:

Telegram configured

Done! Token's updated and the gateway is restarting. 🎉

Now go find your bot on Telegram and send it a message - say /start
or just "hey". The first time you message it, it'll pair with you
so only you can use it.

Open Telegram, find your bot, and send it a message. The first time you do this, you’ll need to approve the pairing from the terminal:

Approving the pairing

openclaw pairing approve telegram ZK9FFEWK

And that’s it. Your AI agent is now available in your pocket.

What Can It Actually Do?

Once everything’s wired up, you’ve got a persistent AI assistant that can do rather more than just chat. Out of the box, OpenClaw supports web browsing, file operations, shell commands, and a memory system that retains context across sessions. There’s also a heartbeat feature that lets the agent proactively check in with you - useful for monitoring tasks or gentle reminders.

I’m still exploring the edges of what’s possible, so I’ll save the deep dive for a future post. For now, I’m enjoying having an assistant that remembers our conversations and doesn’t require me to re-explain context every time I open a new chat window.

Where to Go From Here

If you want to dig deeper, here are some useful links:

OpenClaw — Personal AI Assistant

OpenClaw — the open-source AI assistant that runs on your machine and works from the chat apps you already use.

openclaw.ai

OpenClaw Docs

OpenClaw is a multi-channel gateway for AI agents that runs on any OS.

docs.openclaw.ai

Security · OpenClaw

Security considerations and threat model for running an AI gateway with shell access

docs.openclaw.ai

The documentation covers skills, cron jobs, browser automation, and a whole ecosystem of community-built extensions on ClawHub. There’s also an active Discord if you get stuck.

First Impressions

OpenClaw is the kind of project that makes you rethink what “using AI” means. It’s not just a chatbot - it’s a framework for building something closer to an actual assistant. The onboarding experience is surprisingly charming, the Telegram integration works flawlessly, and the transparent markdown-based memory system means you’re never left wondering what the agent knows or why it’s behaving a certain way.

That said, the security warnings are there for a reason. This is beta software that can execute commands on your system. Treat it accordingly - run it on isolated hardware, keep your secrets elsewhere, and don’t give it access to anything you wouldn’t hand to a keen but occasionally confused intern.

I’m looking forward to seeing where this goes.

Day to Day Tools, the 2025 edition

web.site@mckendrick.email (Russ McKendrick) — Mon, 29 Dec 2025 11:07:03 GMT

As 2025 draws to a close, I find myself reflecting on the tools I use day-to-day. Comparing this to my 2024 edition, it’s remarkable how much the landscape has shifted, particularly around AI tooling. What was experimental 18 months ago has now become central to how I work. 🤔

Desktop Apps 💻

Antigravity↗: At the start of 2025, I was using Cursor as a replacement for VSCode (and still do at the day job), but as the year progressed, Cursor’s pricing and limit changes started to get in the way. Luckily, towards the end of the year, Google released Antigravity - a free AI-powered IDE built on a VSCode fork. It supports multiple AI models including Gemini 3, Claude, and others, with generous rate limits on the Gemini models. Combined with the Claude Code plugin, it’s become my IDE of choice - essentially what Cursor should have been. 🚀
Paste↗: Still going strong after years of use. Having my clipboard history synced across all my devices remains invaluable - I genuinely don’t know how I’d function without [CMD]+[Shift]+V at this point. 📋
Ghostty↗: This is the big change for 2025. After years with iTerm2, I’ve switched to Ghostty as my terminal emulator of choice. Built by one of the original HashiCorp founders, it’s fast, native, and just feels right. The configuration is straightforward, and the performance is noticeably snappier. 👻
Sublime Text↗: For raw text editing, Sublime Text remains unbeatable. Quick to launch, no bloat, and still the best text editor around after all these years. 📝
Reeder↗: The new Reeder has become my go-to for keeping up with RSS feeds, podcasts, videos, and social media posts - all in a unified timeline. It syncs your position across devices via iCloud, so you can pick up exactly where you left off. Clean, focused, and does one thing exceptionally well. 📖

Terminal Apps ⌨️

Claude Code↗: This has genuinely changed how I work. Anthropic’s CLI tool for agentic coding lets me delegate coding tasks directly from my terminal, with Claude understanding context from my codebase. I split my time between Claude Code in the terminal and Antigravity with its Claude Code plugin - between them, they’ve replaced VSCode entirely. 🤖
Homebrew↗: Some things never change. Homebrew remains the only real package manager for macOS, and it’s still the first thing I install on any new machine. 🍺
Starship↗: A minimal, blazing-fast, and highly customisable prompt for any shell. I covered my setup in detail here. It works beautifully with Ghostty. ✨
Oh My Zsh↗: I’ve been using Oh My Zsh for quite a while now. It’s a framework for managing Zsh configuration that comes with helpful plugins, themes, and sensible defaults. It makes Zsh genuinely pleasant to use. 🐚
zoxide↗: A smarter cd command that remembers which directories you use most frequently, letting you jump to them in just a few keystrokes. Once you’ve used it, going back to regular cd feels painful. 📂
Conda↗: Still my go-to for Python environment management. I wrote about my setup back in 2024 and it’s remained largely unchanged. 🐍
PNPM↗: For Node.js package management, I’ve fully switched to PNPM. It’s faster than npm, more efficient with disk space thanks to its content-addressable storage, and just works. This blog is built with it, for details on why I switched from NPM see this post 😱 📦

AI Services 🤖

Claude↗: If it wasn’t obvious from Claude Code making the list, Anthropic’s Claude has become my primary AI assistant. The reasoning capabilities, the longer context windows, and the ability to work through complex problems make it my first port of call. 🧠
Perplexity↗: For research-style queries where I want sources and citations, Perplexity fills a nice gap. It’s particularly useful when I need to verify information or get a quick overview of a topic with references. 📚
Google Gemini↗: Between the desktop app and the web interface, Gemini has earned its place in my toolkit, especially for multimodal tasks and when I need tight integration with Google’s ecosystem. ♊
ChatGPT↗: Still useful, particularly for image generation and for specific tasks where GPT-5 shines. It’s dropped from being my primary AI tool, but it hasn’t left the toolkit entirely. 💬
Fal↗: A great collection of API driven image generation tools, I use it for generating images for the blog, see this post for details. 📸

Services 🌐

Kagi↗: My primary search engine. No ads, no tracking, and genuinely better results. It’s become indispensable for actual research rather than wading through SEO-optimised garbage. 🔍
Proton↗: Privacy-focused email, calendar, and VPN. I’ve been gradually moving more of my digital life to Proton’s ecosystem. In a world where everything wants to harvest your data, having services that actively protect your privacy feels increasingly important. 🔒
GitHub↗: Where all my code↗ lives. From this blog↗ to various side projects and infrastructure-as-code repos, GitHub remains central to how I work and share code. 🐙
Cloudflare Pages↗: This blog and my record collection page↗ both run on Cloudflare Pages. Fast deployments, great CDN, and the price is right (free for my usage). 🌩️

The biggest theme of 2025? AI tooling has moved from “interesting experiment” to “core workflow.” Between Claude Code, multiple AI assistants, and AI-enhanced search, these tools have fundamentally changed how I approach work and various day-to-day tasks.

Oh, and I got a 3D printer↗ - which has already sparked a minor Gridfinity↗ obsession.

I wonder what next year’s list will look like - at this pace of change, it could be dramatically different again. 🚀

Save money by pausing Microsoft Fabric Capacities with Azure Logic Apps

web.site@mckendrick.email (Russ McKendrick) — Sun, 21 Dec 2025 11:37:26 GMT

Back in April 2024, I wrote a post about Saving money with Azure Logic Apps where I covered automating the stop/start of Azure Virtual Machines and Application Gateways. Since then, I’ve had a few requests to add support for Microsoft Fabric capacities to the repo, so here we are.

If you’re not familiar with Microsoft Fabric, it’s Microsoft’s unified analytics platform that brings together data engineering, data science, real-time analytics, and business intelligence. It’s a powerful platform, but like many Azure services, it can get expensive if you’re not careful about managing your capacity.

Why Pause Fabric Capacities?

Microsoft Fabric uses a capacity-based pricing model where you pay for Capacity Units (CUs) by the hour. The costs add up quickly:

An F2 capacity runs about $0.36 per hour, or roughly $262 per month if left running 24/7
An F64 capacity can set you back around $5,000+ per month on a reserved instance

The good news is that with pay-as-you-go (PAYG) capacities, you can pause them when they’re not in use. If your team only uses Fabric during working hours, pausing the capacity overnight and at weekends could save you around 65% of your costs. That’s a significant saving, especially for dev and test environments where you may not want to pay for a reserved capacity that’s running 24/7.

The Logic App

The Logic App I’ve added to the repo follows the same pattern as the existing Virtual Machine and Application Gateway workflows. It uses a user-managed identity to authenticate against the Azure REST API and targets resources based on tags.

You can find the Logic App definition in the GitHub repo:

GitHub - russmckendrick/money-saving-azure-logic-apps: Discover how Azure Logic Apps can help you save money by automating tasks like stopping and starting Virtual Machines and Application Gateways. This post provides a step-by-step guide on deploying Logic Apps and walks through the workflows for cost-saving automations, ensuring your Azure resources only run when needed.

Discover how Azure Logic Apps can help you save money by automating tasks like stopping and starting Virtual Machines and Application Gateways. This post provides a step-by-step guide on deploying ...

Azure API Management Soft-Delete

What It Does

The workflow is straightforward:

Recurrence Trigger: By default, it runs at 6pm GMT, Monday to Friday
Get Tagged Resources: Queries the subscription for resources tagged with fabricCapacityPause=included
Filter to Fabric Capacities: Filters the results to only include Microsoft.Fabric/capacities resources
Check Status: For each capacity, it checks if the state is “Active”
Suspend: If active, it suspends the capacity

Unlike the VM and Application Gateway Logic Apps which toggle between start and stop, this one is designed specifically to pause capacities at the end of the working day.

Deploying the Logic App

Let’s walk through deploying the Logic App. If you’ve followed the original post, this will feel very familiar.

Cloning the Repo

First, clone the repo containing the Logic App definitions:

Cloning the repo

git clone https://github.com/russmckendrick/money-saving-azure-logic-apps.git
cd money-saving-azure-logic-apps

Setting Up the Environment

Set some environment variables for the deployment:

Setting environment variables

export RESOURCE_GROUP_NAME="rg-logicapps-fabric-uks"
export REGION="uksouth"
export SUBSCRIPTION_ID=$(az account show --query id --output tsv)
export MANAGED_ID_NAME="mi-logicapps-fabric-uks"
export LOGIC_APP_NAME="la-fabricCapacityPause-uks"

Creating the Resources

Create the resource group:

Creating the resource group

az group create \
    --name $RESOURCE_GROUP_NAME \
    --location $REGION

Create the user-managed identity:

Creating the managed identity

az identity create \
    --resource-group $RESOURCE_GROUP_NAME \
    --name $MANAGED_ID_NAME

Tagging Your Fabric Capacities

Before we grant permissions, we need to tag the Fabric capacities we want to manage. The Logic App targets resources based on tags, and we can also use those same tags to discover which resources need permissions.

Add the following tag to any capacity you want to be automatically paused:

Tag Name: fabricCapacityPause
Tag Value: included

You can do this through the Azure Portal or using the Azure CLI:

Tagging a Fabric capacity

az resource tag \
    --tags fabricCapacityPause=included \
    --ids "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/YOUR_FABRIC_RESOURCE_GROUP/providers/Microsoft.Fabric/capacities/YOUR_CAPACITY_NAME"

Granting Permissions

The managed identity needs Reader access to list resources across the subscription, and permissions to manage the Fabric capacities. According to Microsoft’s documentation↗, the following permissions are required to pause and resume capacities:

Microsoft.Fabric/capacities/read
Microsoft.Fabric/capacities/write
Microsoft.Fabric/capacities/suspend/action
Microsoft.Fabric/capacities/resume/action

Unfortunately, there isn’t a dedicated Fabric-specific built-in role that provides just these permissions, so Contributor is the minimum required role. However, we can scope it to just the Fabric capacity resources rather than the entire subscription.

First, grant Reader access to the subscription so the Logic App can query for tagged resources:

Granting Reader access to the subscription

az role assignment create \
    --assignee-principal-type "ServicePrincipal" \
    --assignee-object "$(az identity show --resource-group $RESOURCE_GROUP_NAME --name $MANAGED_ID_NAME --query principalId --output tsv)" \
    --role "Reader" \
    --scope "/subscriptions/$SUBSCRIPTION_ID"

Now, we can loop through all the tagged Fabric capacities and grant Contributor access to each one:

Granting Contributor access to tagged Fabric capacities

for CAPACITY_ID in $(az resource list \
    --tag fabricCapacityPause=included \
    --query "[?type=='Microsoft.Fabric/capacities'].id" \
    --output tsv); do
    echo "Granting Contributor access to: $CAPACITY_ID"
    az role assignment create \
        --assignee-principal-type "ServicePrincipal" \
        --assignee-object "$(az identity show --resource-group $RESOURCE_GROUP_NAME --name $MANAGED_ID_NAME --query principalId --output tsv)" \
        --role "Contributor" \
        --scope "$CAPACITY_ID"
done

This approach means you only grant permissions to the specific capacities you’ve tagged, following the principle of least privilege.

Deploying the Logic App

Now deploy the Logic App itself:

Deploying the Logic App

az logic workflow create \
    --resource-group $RESOURCE_GROUP_NAME \
    --location $REGION \
    --name $LOGIC_APP_NAME \
    --mi-user-assigned "$(az identity show --resource-group $RESOURCE_GROUP_NAME --name $MANAGED_ID_NAME --query id --output tsv)" \
    --state "Disabled" \
    --definition "fabricCapacityPause.json"

Updating the Parameters

Update the Logic App parameters with your managed identity and subscription IDs:

Setting the managedId parameter

az logic workflow update \
    --resource-group $RESOURCE_GROUP_NAME \
    --name $LOGIC_APP_NAME \
    --set "definition.parameters.managedId.defaultValue=$(az identity show --resource-group $RESOURCE_GROUP_NAME --name $MANAGED_ID_NAME --query id --output tsv)"

Setting the subscriptionId parameter

az logic workflow update \
    --resource-group $RESOURCE_GROUP_NAME \
    --name $LOGIC_APP_NAME \
    --set "definition.parameters.subscriptionId.defaultValue=$SUBSCRIPTION_ID"

Enabling the Logic App

Once everything is configured, enable the Logic App:

Enabling the Logic App

az logic workflow update \
    --resource-group $RESOURCE_GROUP_NAME \
    --name $LOGIC_APP_NAME \
    --state "Enabled"

You can also trigger a manual run from the Azure Portal to test that everything is working as expected before waiting for the scheduled run.

Customising the Schedule

The default schedule pauses capacities at 6pm GMT, Monday to Friday. If you need a different schedule, you can update the recurrence trigger in the Logic App designer or by modifying the JSON directly.

For example, if you wanted to pause at 7pm instead, you would update the hours array in the trigger:

Modifying the schedule

"recurrence": {
    "frequency": "Week",
    "interval": 1,
    "schedule": {
        "hours": [
            "19"
        ],
        "minutes": [
            0
        ],
        "weekDays": [
            "Monday",
            "Tuesday",
            "Wednesday",
            "Thursday",
            "Friday"
        ]
    },
    "timeZone": "GMT Standard Time"
}

Resuming Capacities

This Logic App only pauses capacities; it doesn’t resume them. For resuming, you have a few options:

Manual Resume: Start the capacity manually from the Azure Portal when you need it
Separate Logic App: Create a similar Logic App that runs in the morning and calls the resume endpoint instead of suspend
On-Demand: Use Azure Automation or a simple script to resume capacities as needed

The resume endpoint follows the same pattern as the suspend endpoint, just swap /suspend for /resume in the API call.

Cleanup

To remove the Logic App and supporting resources:

Removing the resources

export RESOURCE_GROUP_NAME="rg-logicapps-fabric-uks"
az group delete --name $RESOURCE_GROUP_NAME

Don’t forget to clean up any lingering RBAC assignments in the Azure Portal.

Summary

Adding Fabric capacity management to the money-saving Logic Apps collection was a natural extension. The pricing model makes it an obvious candidate for automation, especially for development and test environments where 24/7 availability isn’t required.

If you’re running Fabric capacities and not already pausing them outside of working hours, you’re potentially leaving significant savings on the table. A few minutes of setup could save you hundreds (or thousands) of pounds each month.

As always, the full Logic App definition is available in the GitHub repo, and I’ll continue adding new resource types as I encounter them in the wild.

Azure Soft Delete: Recovering Resources and Preventing Accidental Deletions

web.site@mckendrick.email (Russ McKendrick) — Fri, 05 Dec 2025 09:37:37 GMT

Earlier this week, I had one of those moments that makes your stomach drop – an Azure API Management instance had been deleted. These things happen; someone clicks the wrong button, a Terraform state gets out of sync, or an automated cleanup script gets a bit too enthusiastic. Whatever the cause, the result is the same: a critical resource is gone, and you need it back.

Fortunately, some Azure services, including API Management, support soft delete. Many Azure services retain deleted resources for a period before permanently removing them, giving you a window to recover from these situations. This post covers my experience recovering the APIM instance and provides a broader look at which Azure services support soft delete and how to use it.

Recovering a Deleted Azure API Management Instance

When an Azure API Management instance is deleted (via the Azure Portal or REST API version 2020-06-01-preview or later), it enters a soft-deleted state for 48 hours before being permanently purged. This gives you a window to recover it.

Finding the Deleted Service

First, let’s confirm the service is in a soft-deleted state and get the details we need. Using the Azure CLI:

Check the soft-deleted APIM service

az apim deletedservice show \
  --location uksouth \
  --service-name apim-that-was-deleted-uks

This returns something like:

Soft-deleted service details (JSON)

{
  "deletionDate": "2025-12-01T15:04:29.570042+00:00",
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/providers/Microsoft.ApiManagement/locations/uksouth/deletedservices/apim-that-was-deleted-uks",
  "location": "UK South",
  "name": "apim-that-was-deleted-uks",
  "scheduledPurgeDate": "2025-12-03T15:02:32.189403+00:00",
  "serviceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/rg-apim-production-uks/providers/Microsoft.ApiManagement/service/apim-that-was-deleted-uks",
  "type": "Microsoft.ApiManagement/deletedservices"
}

The key things to note here are the scheduledPurgeDate (you have until then to recover) and the serviceId which tells us the original resource group. In my case, the resource group had also been deleted, so I needed to recreate that first.

Restoring the Service

To restore the APIM instance, you need to use the Azure REST API directly – there’s no native Azure CLI command for this yet. Set up your variables:

Set restoration variables

apimName="apim-that-was-deleted-uks"
subscriptionId=$(az account show --query id --output tsv)
location="uksouth"
resourceGroupName="rg-apim-production-uks"

If your resource group was also deleted, recreate it first:

Recreate the resource group (if needed)

az group create \
  --name $resourceGroupName \
  --location $location

Now restore the APIM service using the REST API:

Restore the APIM service

az rest \
  --method PUT \
  --header "Accept=application/json" \
  --uri "https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.ApiManagement/service/${apimName}?api-version=2021-08-01" \
  --body "{\"location\":\"${location}\",\"properties\": {\"restore\" : true} }"

The restoration process will take several minutes. You’ll see the provisioningState change from Deleted to Activating and eventually to Succeeded. Once complete, your APIM instance is back, complete with all its APIs, policies, and configurations.

Soft-delete allows you to recover a recently deleted API Management instance.

Lock your Azure resources to protect your infrastructure - Azure Resource Manager

Other Azure Services with Soft Delete

My APIM adventure got me thinking about what other Azure services support soft delete. Turns out, quite a few do – and it’s worth knowing about them before you need them.

Azure Key Vault

Key Vault is probably the most well-known example. Soft delete is enabled by default and cannot be disabled. When you delete a secret, key, or certificate, it’s retained for a configurable period (minimum 7 days, up to 90 days) before permanent deletion.

List soft-deleted secrets

az keyvault secret list-deleted --vault-name my-keyvault

Recover a deleted secret

az keyvault secret recover --vault-name my-keyvault --name my-secret

You can also enable purge protection to prevent even administrators from permanently deleting items before the retention period expires – useful in production environments where you really can’t afford to lose things.

Azure Storage

Azure Blob Storage supports soft delete for both blobs and containers, with configurable retention periods from 1 to 365 days.

Enable blob soft delete

az storage account blob-service-properties update \
  --account-name mystorageaccount \
  --enable-delete-retention true \
  --delete-retention-days 30

Azure File Shares also support soft delete, allowing you to recover entire file shares and their contents:

List deleted file shares

az storage share-rm list \
  --storage-account mystorageaccount \
  --include-deleted

Azure SQL Database

Deleted Azure SQL databases can be restored within the backup retention period (up to 35 days depending on your tier). Deleted SQL servers themselves can also be recovered if the backup is still available.

List deleted databases

az sql db list-deleted \
  --resource-group myresourcegroup \
  --server myserver

Azure Backup

Azure Backup enforces soft delete by default as part of its “secure by default” posture. When you stop backup and delete data, the recovery points are retained for 14 days (configurable up to 180 days). This applies to VM backups, SQL Server in Azure VM backups, and SAP HANA backups.

Azure Container Registry

Currently in preview, ACR supports soft delete for accidentally deleted container images and tags with retention periods from 1 to 90 days.

Azure Log Analytics Workspace

Deleted workspaces are retained in a soft-deleted state for 14 days, during which they can be recovered with all their data intact.

Microsoft Entra ID (Azure AD)

The Recycle Bin feature retains deleted users, groups, and other directory objects for 30 days before permanent deletion.

Quick Reference Table

Service	Default Retention	Configurable	Recovery Method
API Management	48 hours	No	REST API
Key Vault	7-90 days	Yes	CLI/Portal
Blob Storage	1-365 days	Yes	CLI/Portal
File Shares	1-365 days	Yes	CLI/Portal
SQL Database	Up to 35 days	Service tier	CLI/Portal
Azure Backup	14-180 days	Yes	Portal
Container Registry	1-90 days	Yes	CLI/Portal
Log Analytics	14 days	No	CLI
Entra ID	30 days	No	Portal

Prevention is Better Than Cure: Delete Locks

While soft delete is a great safety net, it’s better to prevent accidental deletions in the first place. Azure Resource Locks allow you to protect resources from accidental modification or deletion.

There are two types of locks:

CanNotDelete: Resources can be read and modified, but not deleted
ReadOnly: Resources can only be read, not modified or deleted

Applying a Delete Lock

To protect a critical resource from deletion:

Apply a delete lock to a resource

az lock create \
  --name "PreventDeletion" \
  --lock-type CanNotDelete \
  --resource-group rg-production-uks \
  --resource-name apim-production-uks \
  --resource-type Microsoft.ApiManagement/service

Or protect an entire resource group:

Apply a delete lock to a resource group

az lock create \
  --name "PreventDeletion" \
  --lock-type CanNotDelete \
  --resource-group rg-production-uks

Managing Locks with Infrastructure as Code

If you’re using Terraform, you can apply locks as part of your deployment:

Terraform resource lock (HCL)

resource "azurerm_management_lock" "apim_lock" {
  name       = "PreventDeletion"
  scope      = azurerm_api_management.main.id
  lock_level = "CanNotDelete"
  notes      = "Protected production resource"
}

For ARM templates or Bicep:

Bicep resource lock

resource apimLock 'Microsoft.Authorization/locks@2020-05-01' = {
  name: 'PreventDeletion'
  scope: apimService
  properties: {
    level: 'CanNotDelete'
    notes: 'Protected production resource'
  }
}

Learn how to lock all users and roles to protect your Azure resources from updates or deletions.

Summary

Getting that sinking feeling when a critical resource disappears is never pleasant, but knowing that Azure has soft delete capabilities across many services can turn a potential disaster into a minor inconvenience. The key takeaways:

Know your retention periods: Different services have different soft delete windows. API Management gives you 48 hours, while Key Vault can give you up to 90 days.
Test recovery procedures before you need them: It’s much less stressful to learn the recovery process in a dev environment than during an incident.
Use delete locks on production resources: A few minutes setting up locks can save hours of recovery work and stress.
Document your critical resources: Keep a list of resources that would cause significant impact if deleted, along with their soft delete capabilities and recovery procedures.

The next time someone accidentally deletes something important, you’ll know exactly where to look and what to do. And if you put those delete locks in place, hopefully you won’t need to deal with it at all.

When the Sandworm Came for My Secrets: Lessons from Shai-Hulud 2.0

web.site@mckendrick.email (Russ McKendrick) — Sat, 29 Nov 2025 00:00:00 GMT

Earlier this week, I discovered that this very repository had fallen victim to Shai-Hulud 2.0 - a sophisticated supply chain attack that compromised npm packages and exfiltrated secrets from thousands of developers and CI/CD pipelines. My API keys for services like FAL.ai, OpenAI, Perplexity, and others were published to attacker-controlled repositories before I even knew what had happened.

This post covers what the attack is, how it works, exactly how I got caught, and the changes I’ve made to prevent this from happening again.

How I Found Out

On November 27th, I received an email from Omar Hammou↗, a security researcher who monitors GitHub for exposed secrets:

I have identified several leaked credentials associated with your accounts, including OpenAI and Perplexity tokens. I operate a GitHub monitoring system that continuously flags exposed secrets, and although some repositories were deleted, the leaked data is still present in GitHub’s historical snapshots.

The credentials had been exfiltrated to a repository at github.com/dp5iveme/zf0jxl7wqn14g9enav (since deleted by GitHub). As Omar explained, even when repositories are deleted, forks and clones preserve the repository state as archived snapshots - meaning the leaked data can persist indefinitely. Omar was kind enough to share the full payload that was captured - a JSON dump of my entire shell environment, including every environment variable that was set at the time of the attack.

What is Shai-Hulud 2.0?

Named after the giant sandworms from Frank Herbert’s Dune, Shai-Hulud 2.0 is an ongoing malware campaign that targets npm package maintainers. Attackers compromise maintainer accounts and publish trojanized versions of legitimate packages.

According to Wiz’s analysis↗, the scale is staggering:

25,000+ repositories affected
350+ unique users compromised
1,000 new repositories being created every 30 minutes during the initial outbreak
Packages from major companies including Zapier, ENS Domains, PostHog, and Postman were trojanized

How the Attack Works

The malware is deceptively simple but devastatingly effective:

Preinstall Hook Execution: The malicious code runs during npm’s preinstall phase via scripts like setup_bun.js and bun_environment.js. This ensures execution before any code review or static analysis can occur.
Credential Harvesting: The malware systematically steals:
- Local configuration files (.npmrc, .env, cloud CLI configs)
- Environment variables
- Cloud metadata service tokens
- Secrets from AWS Secrets Manager, Google Secret Manager, and Azure Key Vault
Triple Base64 Encoding: Stolen data is encoded three times before exfiltration, evading simple detection rules.
Exfiltration to GitHub: Credentials are published to attacker-controlled repositories with the description “Sha1-Hulud: The Second Coming.”
Persistent Backdoor: The malware injects a discussion.yaml GitHub Actions workflow that registers infected machines as self-hosted runners named “SHA1HULUD”, enabling future remote command execution.
Destructive Fallback: If credential exfiltration fails, the malware attempts to delete the victim’s entire home directory - converting espionage into sabotage.

The Smoking Gun: kill-port@2.0.3

Thanks to the exfiltrated payload Omar shared, I was able to identify exactly which package compromised my system. The evidence was right there in the environment variables:

The Smoking Gun

{
  "npm_lifecycle_script": "node setup_bun.js",
  "npm_lifecycle_event": "preinstall",
  "npm_package_name": "kill-port",
  "npm_package_version": "2.0.3",
  "PWD": "/Users/russ.mckendrick/Code/blog/node_modules/kill-port"
}

The culprit was kill-port - a utility package for killing processes on specific ports. According to the GitLab Advisory (GMS-2025-498)↗, versions 2.0.2 and 2.0.3 were compromised on November 24th, 2025. The vulnerability is rated HIGH severity (CVSS 8.6) and classified as CWE-506 (Embedded Malicious Code).

The npm_lifecycle_script value of node setup_bun.js is the exact malicious script identified in the Wiz security report - it’s designed to look innocuous (who wouldn’t trust something called “setup_bun.js” in 2025?), but its sole purpose is credential theft.

The package maintainer confirmed on GitHub↗ that npm support has since removed the infected versions (2.0.2 and 2.0.3) from the registry. Version 2.0.1 is safe and does not contain the malicious files.

What Was Stolen

The payload captured my entire environment, including:

Service	Credential Type
OpenAI	API Key
Perplexity	API Key
FAL.ai	API Key
Tavily	API Key
Exa	API Key
Last.fm	API Key

Plus my username, home directory path, shell configuration paths, and various other environment details that could be useful for further attacks.

Fortunately, I don’t store AWS, GCP, or Azure credentials in environment variables (they use CLI-based authentication), but many developers aren’t so lucky.

The Global Damage

According to Wiz’s preliminary analysis, attackers obtained:

Credential Type	Count
GitHub Access Tokens	775
AWS Credentials	373
GCP Credentials	300
Azure Credentials	115

These numbers only represent what was discovered in the attacker’s public repositories before GitHub took action.

What I’ve Changed

After rotating all my compromised credentials (a fun few hours, I must say), I’ve implemented several changes to reduce the attack surface:

Migrated from npm to pnpm

Beyond being faster and more disk-efficient, pnpm has better security defaults. I’ve updated all my GitHub Actions workflows to use pnpm with the --frozen-lockfile flag:

Github Action

- name: 🌸 Setup PNPM
  uses: pnpm/action-setup@v6

- name: 📦 Install dependencies
  run: pnpm install --frozen-lockfile

The --frozen-lockfile flag ensures CI/CD installs exactly what’s in the lockfile - no surprises from updated packages.

I’ve also added aliases to my .zshrc to prevent accidental npm usage locally:

Terminal window

# 📦 pnpm aliases in .zshrc
alias npm='echo "Error: Use pnpm"; false'
alias n="pnpm"
alias ni="pnpm install"

This ensures I can’t accidentally run npm install out of muscle memory.

Explicitly Approve Build Scripts

pnpm 10+ has a security feature that blocks lifecycle scripts by default, showing warnings like:

Ignore build scripts !!!

Ignored build scripts: esbuild, sharp, workerd.
Run "pnpm approve-builds" to pick which dependencies should be allowed to run scripts.

Rather than blanket-allowing all scripts, you can explicitly whitelist only the packages you trust. I’ve added this to my package.json:

package.json

{
  "pnpm": {
    "onlyBuiltDependencies": [
      "esbuild",
      "sharp",
      "workerd"
    ]
  }
}

This means:

Package	Script	Purpose
`sharp`	`install`	Downloads native image processing binaries
`esbuild`	`postinstall`	Downloads platform-specific bundler binary
`workerd`	`postinstall`	Downloads Cloudflare Workers runtime

Any other package trying to run lifecycle scripts will be blocked. This is a much better approach than --ignore-scripts (which breaks legitimate packages) or allowing everything (which is what npm does by default).

Regular Security Audits

I’ve added pnpm audit to all my GitHub Actions workflows:

pnpm audit

- name: 🔒 Security audit
  run: pnpm audit

This checks dependencies against the npm security advisory database and will fail the build if any known vulnerabilities are found. You can also run it locally:

Terminal window

pnpm audit

Pinned Dependencies

Rather than allowing automatic updates, I’m now pinning dependencies to specific versions and reviewing changes manually:

package.json

{
  "dependencies": {
    "astro": "5.1.1"
  }
}

Auditing for Compromise

If you’re concerned your environment may have been affected, check for:

Suspicious workflow files: Look in .github/workflows/ for shai-hulud-workflow.yml, discussion.yaml, or unexpected branches
Self-hosted runner registrations: Check for runners named “SHA1HULUD” in your GitHub repository settings
Unexpected npm packages: Review your package-lock.json or pnpm-lock.yaml for packages added between November 21-25, 2025

Credential Rotation

If you installed any npm packages during the attack window (November 21-25, 2025), you should:

Clear your npm/pnpm cache: pnpm store prune
Delete node_modules and reinstall
Rotate all credentials that were accessible to your development environment
Review GitHub for any unauthorized workflow changes

Lessons Learned

This attack is a stark reminder of several realities in modern development:

Supply chain attacks are the new normal: We implicitly trust thousands of packages and their maintainers every time we run npm install.
CI/CD environments are high-value targets: They often have access to production credentials, cloud resources, and deployment capabilities.
Lifecycle scripts are a security risk: The ability for packages to execute arbitrary code during installation is powerful but dangerous.
Defense in depth matters: No single measure would have prevented this, but layers of security (restricted CI/CD permissions, pinned dependencies, disabled scripts) reduce the blast radius.
The security community is watching: I’m grateful to researchers like Omar who actively monitor for exposed secrets and alert affected developers. Without his email, I might not have known for weeks.

Acknowledgments

Thanks to Omar Hammou↗ for the heads-up and for sharing the exfiltrated payload so I could identify the attack vector. Security researchers who responsibly disclose findings like this are invaluable to the community.

Playing with Nano Banana Pro: AI-Powered Album Cover Collages

web.site@mckendrick.email (Russ McKendrick) — Sun, 23 Nov 2025 10:28:53 GMT

Every week, I generate a “Listened to This Week” blog post that showcases my top albums from Last.fm. The header of each post is a custom cover collage-a visually stunning blend of album artwork that serves as the hero image. Until recently, I was using a local Sharp-based “torn paper strip” generator. But I wanted something more dynamic, more artistic, and more… AI-powered.

Enter Nano Banana Pro↗, Googles latest image editing model:

dotfiles/starship.toml at main · russmckendrick/dotfiles

It is now available on the FAL.ai↗ API and after implementing this I now have a sophisticated AI-powered collage generator that intelligently selects the most vibrant album covers, analyzes them with GPT-4 Vision, and composites them into quirky music masterpieces.

The Challenge

I found that creating collages is using well known album covers isn’t as simple as throwing images together. There are several challenges:

Content Policy Violations: Album covers can contain imagery that triggers AI model safety filters
Visual Quality: Not all album covers are equally vibrant or interesting
Text Removal: Album covers have text/typography that needs to be removed
Composition: The AI needs intelligent prompts to create cohesive artistic compositions
Reliability: The system needs to handle failures gracefully and retry with alternative images

The Solution Architecture

The collage generator that I prompted using Claude Code↗ uses a multi-stage pipeline that combines computer vision analysis, GPT-4 Vision↗ prompting, and Nano Banana Pro’s↗ image editing capabilities - the work flow looks something like this …

Process flow diagram

Ctrl+scroll to zoomFullscreenSVGPNG

Error rendering diagram:

Image Selection Intelligence

The start of the process is the image selection algorithm. Instead of randomly picking albums, it analyzes each cover using three key metrics:

1. Color Saturation Analysis

Color Analysis

// Calculate average saturation (max - min of RGB)
const max = Math.max(r, g, b)
const min = Math.min(r, g, b)
saturationSum += (max - min)
const avgSaturation = saturationSum / pixelCount

High saturation = vibrant, eye-catching colors that work well in composites.

2. Color Variance Detection

Color Variance

// Calculate RGB variance for visual complexity
const totalVariance = (rVariance + gVariance + bVariance) / 3

High variance = rich color diversity and visual interest.

3. Text Detection & Penalty

This is the clever bit. Album covers typically have text in the top and bottom regions (album title, artist name). The system:

Extracts the top 20% and bottom 20% of each image
Applies edge detection using a convolution kernel
Counts high-intensity pixels (edges indicate text)
Penalizes covers with heavy text

Edge detection kernel

// Edge detection kernel
kernel: [-1, -1, -1, -1, 8, -1, -1, -1, -1]

// Text score: higher = more text = worse for collage
const textScore = (totalEdgePixels / totalPixels) * 100

Weighted Scoring Formula

The final score combines all three metrics:

Final Score

const colorScore = (avgSaturation * 0.4) + (Math.sqrt(totalVariance) * 0.3)
const textPenalty = (100 - textScore) * 0.3
const finalScore = colorScore + textPenalty

Weights (configurable in fal-collage-config.json):

Saturation: 40% - Vibrant colors are most important
Variance: 30% - Visual complexity matters
Text Penalty: 30% - Avoid text-heavy covers

The GPT-4 Vision Advantage

Once the best albums are selected, GPT-4 Vision analyzes the actual visual content and generates a custom compositing prompt. This is crucial because generic prompts produce generic results.

The system prompt instructs GPT-4 to:

Identify specific visual elements: “the portrait from the yellow cover”, “the building from the blue cover”
Describe composition techniques: double-exposure, color grading, blend modes
Create unified scenes: Not separate album covers, but one cohesive image
Specify color treatment: Cohesive palette across the entire composition

Example Generated Prompt:

“Create a cinematic music blog header featuring the vibrant abstract faces from the colorful cover, blending them into the architecture of the church-like building with musicians from the second image. Add the spilled flower pot from the third cover in the foreground, and place the painter from the fourth cover painting this scene. Overlay the dreamy curtains from the fifth cover to add a soft, ethereal layer, using double-exposure techniques. Integrate the collage of faces from the sixth cover subtly into the background, and place the penguins from the seventh cover on the grassy field. Finally, position the black and white group from the eighth cover in the middle, tying the scene together with a cohesive blend of warm and cool tones. Remove all text and typography.”

This context-aware prompting produces far better results than static prompts.

Image Preprocessing Pipeline

Before sending images to Nano Banana Pro, they go through preprocessing to remove text regions:

Image Preprocessing Pipeline

Ctrl+scroll to zoomFullscreenSVGPNG

Error rendering diagram:

+−⟲

This aggressive cropping removes most album text while preserving the core visual content.

Content Policy & Retry Logic

AI models have content policies. Some album covers trigger safety filters (violence, nudity, etc.). The solution: intelligent retry with alternative images.

Image Preprocessing Pipeline

Ctrl+scroll to zoomFullscreenSVGPNG

Error rendering diagram:

+−⟲

The system:

Pre-ranks ALL available albums by vibrancy/text score
Attempts with top batch (albums 1-8)
On policy violation, selects the NEXT batch (albums 7-12)
Regenerates GPT-4 Vision prompt with new images
Retries up to 3 times before failing

This approach has a high success rate because it systematically tries different image combinations.

Blacklist Management

Some albums are persistent offenders (I’m looking at you, Is This It↗ and Með Suð Í Eyrum Við Spilum Endalaust↗). The config file includes a blacklist:

Blacklist

{
  "blacklist": {
    "albums": [
      "Is This It",
      "Med-sud-i-eyrum-vid-spilum-endalaust"
    ],
    "artists": []
  }
}

Blacklisted items are filtered out before scoring, ensuring they never make it to the API and I don’t shouted for submitting pictures of bums !!!.

Configuration-Driven Design

Everything is configurable via fal-collage-config.json:

Config

{
  "model": {
    "name": "fal-ai/nano-banana-pro/edit",
    "fallback": "fal-ai/reve/fast/remix"
  },
  "output": {
    "aspectRatio": "16:9",
    "numImages": 1,
    "format": "png",
    "resolution": "2K"
  },
  "images": {
    "minCount": 2,
    "maxCount": 8,
    "cropRegions": {
      "top": 0.15,
      "bottom": 0.15,
      "left": 0.05,
      "right": 0.05
    }
  },
  "scoring": {
    "saturationWeight": 0.4,
    "varianceWeight": 0.3,
    "textPenaltyWeight": 0.3
  },
  "retry": {
    "maxAttempts": 3,
    "contentPolicyViolation": {
      "enabled": true,
      "excludeProblematicImages": true
    }
  }
}

Want to change the model? Update model.name. Want more aggressive text removal? Adjust images.cropRegions. Want to prioritize color variance over saturation? Tweak scoring weights.

Model Selection & Fallback Strategy

The collage generator is built around Nano Banana Pro as the primary model, with WAN 2.5 configured as a fallback option. This design provides reliability while leveraging the best available technology.

Why Nano Banana Pro?

When designing the system, I chose Nano Banana Pro for several key reasons:

Photographic Quality: Produces photorealistic, cinematic results rather than painterly effects
High Resolution: Native 2K (2048px) output without upscaling artifacts
Better Composition: Stronger understanding of spatial relationships and scene composition
Text Removal: More effective at removing typography while preserving visual elements
Color Preservation: Maintains original album artwork vibrancy better

Configuration-Driven Model Selection

The system uses a flexible config-driven approach defined in fal-collage-config.json:

Model selection

{
  "model": {
    "name": "fal-ai/nano-banana-pro/edit",
    "fallback": "fal-ai/reve/fast/remix"
  }
}

The implementation loads the model dynamically:

Model loading

const modelName = config.model?.name || "fal-ai/nano-banana-pro/edit"

const apiInput = {
  prompt: smartPrompt || defaultPrompt,
  image_urls: uploadedUrls,
  aspect_ratio: "16:9",
  num_images: 1,
  output_format: "png",
  resolution: "2K"
}

const result = await fal.subscribe(modelName, {
  input: apiInput,
  logs: debug,
  onQueueUpdate: (update) => {
    if (debug && update.status === "IN_PROGRESS") {
      update.logs?.map(log => log.message).forEach(msg => console.log(`[FAL] ${msg}`))
    }
  }
})

Fallback Strategy

While Nano Banana Pro is the primary model, the configuration includes a fallback model (fal-ai/reve/fast/remix) that can be activated by simply changing the config file. This provides:

Flexibility: Switch models without code changes
Reliability: Alternative if Nano Banana Pro has availability issues
Experimentation: Easy A/B testing between different models
Future-proofing: Simple to add new models as FAL.ai releases them

The result? Photorealistic context aware music blog headers that look professionally designed-all generated automatically each week.

The Results

I tested with the post from Nov 17, 2025 post↗, which contained these albums:

As you can see from the output below and the covers above it certainly made the right choice for which 8 covers to use for the collage, it correctly skipped the all text Stop Making Sense↗, and the not very colorful Songs from the Big Chair↗ and A Secret Wishhttps://www.russ.fm/album/a-secret-wish-35663980/↗) …

Running the script

Creating FAL.ai music collage...\n
Input folder: /Users/russ.mckendrick/Code/blog/public/assets/2025-11-17-listened-to-this-week/albums
Found 11 album images\n
Creating FAL.ai collage...\n
  Creating FAL.ai collage (1400x800)...
  Analyzing 11 images (color vibrancy + text detection + blacklist filtering)...
  Top 11 best images (vibrant + low text):
    1. Zeitgeist.jpg [color: 48.0, text: 23.7%, final: 70.9]
    2. So-Here-We-Are-Best-Of-Doves.jpg [color: 40.4, text: 9.0%, final: 67.7]
    3. De-La-Soul-Is-Dead.jpg [color: 37.1, text: 7.7%, final: 64.8]
    4. The-Big-Lad-In-The-Windmill.jpg [color: 35.8, text: 19.5%, final: 59.9]
    5. Lost-in-the-Dream.jpg [color: 31.3, text: 10.9%, final: 58.1]
    6. Metropolis-Pt-2-Scenes-From-a-Memory.jpg [color: 31.1, text: 11.1%, final: 57.8]
    7. Best-of-1969-1974.jpg [color: 31.8, text: 21.0%, final: 55.5]
    8. Specials.jpg [color: 28.4, text: 15.7%, final: 53.7]
    9. A-Secret-Wish.jpg [color: 27.8, text: 18.0%, final: 52.4]
    10. Songs-from-the-Big-Chair.jpg [color: 25.8, text: 12.1%, final: 52.2]
    11. Stop-Making-Sense-Music-from-a-film-by-Jonathan-Demme-and-Talking-Heads.jpg [color: 22.7, text: 13.8%, final: 48.6]
  Strategy: Send 8 individual album covers to Gemini 3 Pro model (2-8 range)
  Ranked 11 images by vibrancy + text score
  Will retry up to 3 times if content policy issues occur
  Attempt 1: Selected 8 albums:
    1. Zeitgeist.jpg
    2. So-Here-We-Are-Best-Of-Doves.jpg
    3. De-La-Soul-Is-Dead.jpg
    4. The-Big-Lad-In-The-Windmill.jpg
    5. Lost-in-the-Dream.jpg
    6. Metropolis-Pt-2-Scenes-From-a-Memory.jpg
    7. Best-of-1969-1974.jpg
    8. Specials.jpg
  Pre-processing and uploading 8 albums:
    Processing Zeitgeist.jpg...
    Cropping Zeitgeist.jpg: 1500x1500 → 1349x1050
      → Uploaded: https://v3b.fal.media/files/b/zebra/qh45oC-X8U5EHI4IDnTTl_Zeitgeist.jpg
    Processing So-Here-We-Are-Best-Of-Doves.jpg...
    Cropping So-Here-We-Are-Best-Of-Doves.jpg: 1500x1500 → 1349x1050
      → Uploaded: https://v3b.fal.media/files/b/monkey/qfWi_zFOwABweDyDQE1Sd_So-Here-We-Are-Best-Of-Doves.jpg
    Processing De-La-Soul-Is-Dead.jpg...
    Cropping De-La-Soul-Is-Dead.jpg: 1024x1024 → 921x716
      → Uploaded: https://v3b.fal.media/files/b/tiger/nCSgsSHYN-3SLQ5Cq0qUd_De-La-Soul-Is-Dead.jpg
    Processing The-Big-Lad-In-The-Windmill.jpg...
    Cropping The-Big-Lad-In-The-Windmill.jpg: 1024x1024 → 921x716
      → Uploaded: https://v3b.fal.media/files/b/penguin/HXmy8r0L24DUM52p2bASd_The-Big-Lad-In-The-Windmill.jpg
    Processing Lost-in-the-Dream.jpg...
    Cropping Lost-in-the-Dream.jpg: 1425x1425 → 1282x997
      → Uploaded: https://v3b.fal.media/files/b/rabbit/SngKAtV8avD-tNCCGxtbH_Lost-in-the-Dream.jpg
    Processing Metropolis-Pt-2-Scenes-From-a-Memory.jpg...
    Cropping Metropolis-Pt-2-Scenes-From-a-Memory.jpg: 1024x1024 → 921x716
      → Uploaded: https://v3b.fal.media/files/b/elephant/4dBAexdI-O2phSnBGUe0V_Metropolis-Pt-2-Scenes-From-a-Memory.jpg
    Processing Best-of-1969-1974.jpg...
    Cropping Best-of-1969-1974.jpg: 1024x1024 → 921x716
      → Uploaded: https://v3b.fal.media/files/b/rabbit/buCRs63ekxrIZteMYXkBZ_Best-of-1969-1974.jpg
    Processing Specials.jpg...
    Cropping Specials.jpg: 1024x1024 → 921x716
      → Uploaded: https://v3b.fal.media/files/b/penguin/dymm0RrVm-ncPUfDHgrFo_Specials.jpg
  Analyzing album covers with GPT-4 Vision...
  ✓ Generated smart prompt: "Create a cinematic music blog header featuring the vibrant abstract faces from the colorful cover, blending them into the architecture of the church-like building with musicians from the second image. Add the spilled flower pot from the third cover in the foreground, and place the painter from the fourth cover painting this scene. Overlay the dreamy curtains from the fifth cover to add a soft, ethereal layer, using double-exposure techniques. Integrate the collage of faces from the sixth cover subtly into the background, and place the penguins from the seventh cover on the grassy field. Finally, position the black and white group from the eighth cover in the middle, tying the scene together with a cohesive blend of warm and cool tones. Remove all text and typography."
  Generating collage with 8 images using Gemini 3 Pro Image...
  Using AI-generated prompt
  Prompt: "Create a cinematic music blog header featuring the vibrant abstract faces from the colorful cover, blending them into the architecture of the church-like building with musicians from the second image. Add the spilled flower pot from the third cover in the foreground, and place the painter from the fourth cover painting this scene. Overlay the dreamy curtains from the fifth cover to add a soft, ethereal layer, using double-exposure techniques. Integrate the collage of faces from the sixth cover subtly into the background, and place the penguins from the seventh cover on the grassy field. Finally, position the black and white group from the eighth cover in the middle, tying the scene together with a cohesive blend of warm and cool tones. Remove all text and typography."
  Aspect ratio: 16:9
  API payload: {
  "prompt": "Create a cinematic music blog header featuring the vibrant abstract faces from the colorful cover, blending them into the architecture of the church-like building with musicians from the second image. Add the spilled flower pot from the third cover in the foreground, and place the painter from the fourth cover painting this scene. Overlay the dreamy curtains from the fifth cover to add a soft, ethereal layer, using double-exposure techniques. Integrate the collage of faces from the sixth cover subtly into the background, and place the penguins from the seventh cover on the grassy field. Finally, position the black and white group from the eighth cover in the middle, tying the scene together with a cohesive blend of warm and cool tones. Remove all text and typography.",
  "image_urls": [
    "https://v3b.fal.media/files/b/zebra/qh45oC-X8U5EHI4IDnTTl_Zeitgeist.jpg",
    "https://v3b.fal.media/files/b/monkey/qfWi_zFOwABweDyDQE1Sd_So-Here-We-Are-Best-Of-Doves.jpg",
    "https://v3b.fal.media/files/b/tiger/nCSgsSHYN-3SLQ5Cq0qUd_De-La-Soul-Is-Dead.jpg",
    "https://v3b.fal.media/files/b/penguin/HXmy8r0L24DUM52p2bASd_The-Big-Lad-In-The-Windmill.jpg",
    "https://v3b.fal.media/files/b/rabbit/SngKAtV8avD-tNCCGxtbH_Lost-in-the-Dream.jpg",
    "https://v3b.fal.media/files/b/elephant/4dBAexdI-O2phSnBGUe0V_Metropolis-Pt-2-Scenes-From-a-Memory.jpg",
    "https://v3b.fal.media/files/b/rabbit/buCRs63ekxrIZteMYXkBZ_Best-of-1969-1974.jpg",
    "https://v3b.fal.media/files/b/penguin/dymm0RrVm-ncPUfDHgrFo_Specials.jpg"
  ],
  "aspect_ratio": "16:9",
  "num_images": 1,
  "output_format": "png",
  "resolution": "2K"
}
  Using model: fal-ai/nano-banana-pro/edit
  Generated image URL: https://v3b.fal.media/files/b/penguin/Bs5Dx01KgUcnveOxhN5am.png
  ✓ Created FAL.ai collage with 8 vibrant album covers
\n✓ FAL.ai collage complete!
  Output: /Users/russ.mckendrick/Code/blog/test-output/tunes-cover-2025-11-17-listened-to-this-week.png
  Dimensions: 1400×800
  Selected images: 8

This gave us the following:

I also went back and did the rest of the year, here are some more examples:

Try It Yourself

The collage generator is part of my blog’s codebase and runs automatically via GitHub Actions every week. You can find the key files below:

Main script: scripts/fal-tunes-cover.js↗
Config: scripts/fal-collage-config.json↗
Tunes generator: scripts/generate-tunes-post.js (orchestrator)↗

Required environment variables:

Terminal window

FAL_KEY=your-fal-api-key
OPENAI_API_KEY=your-openai-key

Run manually:

Terminal window

node scripts/fal-tunes-cover.js --input=./albums --output=./collage.png --debug

Summary

Migrating to Nano Banana Pro transformed my weekly music blog post workflow, what started as a simple image processing task evolved into a AI-powered pipeline that combines:

Computer vision (color analysis, text detection)
GPT-4 Vision (intelligent prompt generation)
Nano Banana Pro (photographic composition)
Smart retry logic (content policy handling)

The result? Unique cover collages that make each weekly post visually distinctive-and all generated automatically.

Next time you see a “Listened to This Week” post, you’ll know there’s a lot more than meets the eye behind that cover image.

Resources

My Starship Prompt Setup

web.site@mckendrick.email (Russ McKendrick) — Sat, 15 Nov 2025 13:48:35 GMT

I spend alot of time in the terminal on both my home and work machines so have taken the time to get my terminal prompt looking good and I had been using PowerLine10k↗ for a few years.

After trying to debug why my terminal was opening slowly I spotted the warning below at the top of the projects repo:

I decided it was time for a change.

Why Starship?

After some quick Googling I decided upon Starship↗, it is a cross-shell prompt written in Rust that’s fast, customizable, and incredibly feature-rich. Its repo boasts that unlike traditional shell prompts, Starship:

Works across multiple shells (zsh, bash, fish, PowerShell)
Is blazingly fast (written in Rust)
Provides context-aware information
Supports extensive customization through a simple TOML file
Has minimal dependencies

That ticks a lot of boxes so I decided to give it a try.

The Design Philosophy

My prompt design follows a few key principles:

Information density without clutter: Show relevant context without overwhelming
Visual hierarchy: Use color and positioning to prioritize information
Gruvbox aesthetics: Maintain a consistent retro-groove color scheme
Left-to-right narrative: Left side shows “where I am,” right side shows “what’s happening”
Look good: I take a lot of screen shots and do screen shares - so it needs to look good !!!

Now this is a little zoomed in, but this is what my prompt looks like:

The Color Palette: Gruvbox Dark

I’m using the Gruvbox↗ Dark color scheme, which provides a warm, retro aesthetic that’s easy on the eyes during long coding sessions. The palette includes:

Orange (#d65d0e): System and user context
Yellow (#d79921): Current directory and status
Aqua (#689d6a): Git information
Blue (#458588): Programming language indicators
Purple (#b16286): Time display
Red (#cc241d): Battery and critical indicators

The Prompt Layout

My prompt is divided into segments, connected with powerline-style arrows. Here’s the breakdown:

Left Side: Contextual Information

OS and Username Segment (Orange)

[os]
disabled = false
style = "bg:color_orange fg:color_fg0"

[username]
show_always = true
format = '[ $user ]($style)'

The prompt starts with the OS icon (on macOS) and username. While some people hide the username, I keep it visible-it’s a quick visual confirmation of which system I’m on, especially when dealing with remote sessions.

Directory Segment (Yellow)

[directory]
format = "[ $path ]($style)"
truncation_length = 3
truncation_symbol = " "
truncate_to_repo = true

The current directory is shown with intelligent truncation. If I’m in a git repository, it truncates relative to the repo root. I’ve also added custom substitutions with icons for common directories, such as ~/Code/, ~/Documents and ~/Downloads:

Git Metrics Segment (Aqua)

[git_metrics]
disabled = false
format = '[[+$added]($added_style)/[-$deleted]($deleted_style) ]'

This is where things get interesting. The git segment shows four pieces of information:

Branch name with an icon
Git status showing staged/modified/untracked files
Git state for operations like rebasing or merging
Git metrics showing additions and deletions

The metrics feature is particularly useful-at a glance, I can see +7/-7 to understand the scope of my changes.

Language Environment Segment (Blue)

This segment automatically detects and displays icons for active programming environments:

The segment only appears when relevant files are detected in the current directory, keeping the prompt clean when you’re not in a project.

Right Side: Status Information

Using the $fill variable, I push status information to the right side of the terminal:

Time (Purple)

[time]
disabled = false
time_format = "%R"
format = '[[ 󰥔 $time ]]'

A simple 24-hour clock (e.g., 14:30). Some find this redundant, but I appreciate having timestamps in my terminal sessions.

Battery (Red background)

[battery]
full_symbol = "󰁹"
charging_symbol = "󰂄"
discharging_symbol = "󰂃"

Battery status with appropriate icons. The red background serves as a visual alert when I’m unplugged.

Command Duration (Aqua)

[cmd_duration]
min_time = 2000
format = '[[  $duration ]]'

Shows execution time for commands that take longer than 2 seconds. This is invaluable for understanding which operations are slow and need optimization.

Exit Status (Yellow)

Displays an error indicator (✖) when the last command failed, along with the exit code.

Background Jobs (Orange)

Shows a icon with a count when there are background jobs running.

The Command Line

Finally, after all the context, comes the actual command line:

[character]
success_symbol = '[](bold fg:color_bright_green)'
error_symbol = '[](bold fg:color_bright_red)'

A simple ❯ that turns red when the previous command failed. I’ve also configured vim mode indicators (since I use vi-mode in zsh):

Normal mode (green)
Replace mode (purple)
Visual mode (yellow)

Performance Considerations

Starship is fast, but I’ve made a few tweaks to ensure it stays that way:

scan_timeout = 35
command_timeout = 700

The scan_timeout controls how long Starship spends looking for context (like git repos), while command_timeout limits how long individual modules can run. These values ensure the prompt never feels laggy.

Installation

Want to use this configuration? Here’s how:

Install Starship:
Terminal window
```
brew install starship
```
Install a Nerd Font: The icons require a Nerd Font. I use Hack Nerd Font Mono:
Terminal window
```
brew tap homebrew/cask-fonts
brew install font-hack-nerd-font
```
Configure your shell: Add to your ~/.zshrc:
Terminal window
```
eval "$(starship init zsh)"
```
Edit your configuration: Find the configuration at ~/.config/starship.toml

Customization Tips

The beauty of Starship is how easy it is to customize. Here are some ideas:

Minimize for remote sessions: Use different configs based on $SSH_CONNECTION
Add custom modules: Starship supports custom commands and environment variables
Theme variations: Try different palettes like Tokyo Night, Nord, or Dracula
Context-aware layouts: Use different formats for different directories

Summary

Your terminal prompt is personal - it should reflect how you work and what information you need at a glance. My Starship configuration has evolved from when I was using PowerLine10k and will continue to be added to as time goes on.

The key is to start with something functional, then iterate. Pay attention to which information you actually use and which just becomes noise. A great prompt should feel invisible when you don’t need it and invaluable when you do.

You can find my current and full starship.toml configuration at the following link:

My dotfiles - you probably don't need to use these. - russmckendrick/dotfiles

Install n8n locally using Cloudflare

Rotating Azure DevOps SSH Keys: How to Update Your Git Remotes and SSH Config

web.site@mckendrick.email (Russ McKendrick) — Mon, 27 Oct 2025 19:11:06 GMT

We’ve all been there. Your SSH key for Azure DevOps expires, you generate a new one, and suddenly you’re faced with the task of updating all your local repositories. While it’s not complicated, it’s easy to forget the exact steps when you’re in the moment. Here’s a straightforward guide to get you back up and running quickly.

The Problem: Rotating Azure DevOps SSH Keys

SSH keys expire (and they should – it’s good security practice). When you generate a new SSH key pair and add the public key to Azure DevOps, your local Git repositories still point to the old configuration. You need to update two things:

Your SSH configuration file
Your Git remote URLs to use the new SSH config

If you haven’t already, you’ll need to generate your new SSH key:

Generate your SSH key

ssh-keygen -t rsa-sha2-512 -C "user@domain.com"

When you have your key, make a note of the path, as we’ll need it for the next step.

Step 1: Configure Your SSH Config for Azure DevOps

First, update your ~/.ssh/config file to include your new Azure DevOps SSH configuration:

SSH Config

Host azure-devops
  Hostname ssh.dev.azure.com
  User git
  IdentityFile ~/.ssh/ado_id_rsa
  IdentitiesOnly yes

This creates a convenient alias (azure-devops) that you can use instead of typing out the full hostname every time. The IdentitiesOnly yes directive is particularly important, as it ensures SSH only uses the specified identity file, preventing authentication issues when you have multiple SSH keys.

Step 2: Check Your Current Git Remotes

Before making changes, let’s see what we’re working with:

Check the remotes

git remote -v

You’ll likely see something like:

The results

origin  git@ssh.dev.azure.com:v3/YourOrg/YourProject/YourRepo (fetch)
origin  git@ssh.dev.azure.com:v3/YourOrg/YourProject/YourRepo (push)

Step 3: Update Your Git Remote URLs for Azure DevOps

Now update your Git remote to use the new SSH config host:

Set the new remote

git remote set-url origin git@azure-devops:v3/YourOrg/YourProject/YourRepo

The key change here is replacing the hostname (ssh.dev.azure.com or vs-ssh.visualstudio.com) with your SSH config host alias (azure-devops).

Step 4: Verify the Remote Configuration

Check that the remote was updated correctly:

Confirm the new remotes

git remote -v

You should now see:

The results

origin  git@azure-devops:v3/YourOrg/YourProject/YourRepo (fetch)
origin  git@azure-devops:v3/YourOrg/YourProject/YourRepo (push)

Step 5: Test Your Azure DevOps SSH Connection

Before you start working, verify that SSH authentication is working:

Check the connection

ssh -T git@azure-devops

You should see a response like:

The result of the test connection

remote: Shell access is not supported.

Don’t worry – this is exactly what you want to see! It means SSH successfully authenticated, but Azure DevOps doesn’t provide shell access (which is expected and secure).

Finally, test with an actual Git operation:

Run a git command

git fetch

If this completes without prompting for credentials or throwing errors, you’re all set!

Handling Multiple Git Remotes

If you work with multiple remotes (for example, origin and upstream in a fork workflow), you’ll need to update each one:

Terminal window

git remote set-url origin git@azure-devops:v3/MainOrg/MainProject/MainRepo
git remote set-url upstream git@azure-devops:v3/ForkOrg/ForkProject/ForkRepo

Automating SSH Key Rotation Across Multiple Repositories

If you have many local repositories that need updating, you can create a simple script to automate this process:

#!/bin/bash

# Find all Git repositories in your projects directory
find ~/projects -name ".git" -type d | while read gitdir; do
    repo=$(dirname "$gitdir")
    echo "Updating $repo"
    cd "$repo"

    # Get current remote URL
    current_url=$(git remote get-url origin 2>/dev/null)

    # Check if it's an Azure DevOps SSH URL
    if [[ $current_url == *"dev.azure.com"* ]] || [[ $current_url == *"visualstudio.com"* ]]; then
        # Extract the path (everything after the hostname)
        path=$(echo "$current_url" | sed 's/.*://')
        new_url="git@azure-devops:$path"

        echo "  Old: $current_url"
        echo "  New: $new_url"

        git remote set-url origin "$new_url"
    fi
    echo ""
done

Save this as update-ado-remotes.sh, make it executable with chmod +x update-ado-remotes.sh, and run it from your projects directory.

Best Practices for Azure DevOps SSH Key Management

While we’re on the topic, here are some best practices for managing SSH keys effectively:

Use strong passphrases: Always protect your SSH private keys with a strong passphrase. Use ssh-agent to avoid typing it repeatedly.
Set expiration dates: Azure DevOps allows you to set expiration dates on SSH keys. Use this feature – it’s a great way to enforce key rotation.
Keep your config organised: As you accumulate more SSH configurations, keep your ~/.ssh/config file well-organised with comments. Future you will thank present you.
Document your setup: Keep notes about which keys are used where. When you have multiple keys for different services or organisations, this documentation is invaluable.

Summary

In short:

Rotate your Azure DevOps SSH keys regularly.
Update your SSH config and Git remotes to match.
Test connections before committing changes.
Use automation for bulk updates.
Keep keys protected and your config tidy.

A few small steps now will save you a lot of frustration later.

A new coat of paint

web.site@mckendrick.email (Russ McKendrick) — Sun, 12 Oct 2025 11:52:51 GMT

While there are posts in this blog dating back to 2014, nearly the last 5 years it has been built and deployed using a combination of Hugo↗, Cloudflare Pages↗ and the excellent PaperMod↗ theme.

Why the change?

For my other sites …

russ.fm

russ.tools

… I have been using various Javascript tools, and I wanted to use something similar for this blog. The Hugo setup served me well, but I wanted to modernize the stack and bring it in line with the tools I’m using for my other projects.

The Hugo pain points

Don’t get me wrong - Hugo is an excellent static site generator. It’s incredibly fast, has a great community, and PaperMod is a beautiful theme. However, after five years, I started feeling some friction:

Template language limitations: Hugo’s Go templating syntax, while powerful, felt limited from what I had been using for my other projects
Theme customization: Making changes to PaperMod was challenging, I had to make changes to the underlying HTML and CSS and then figure out a way of applying those changes to the using a sub-theme
Tooling ecosystem: My other projects use modern JavaScript tooling (TypeScript, ESLint, Prettier) and I wanted that same developer experience for my blog
Content scripting: The automated weekly music posts I generate required Python scripts that felt disconnected from the blog itself
Component reusability: I had built custom components for my other sites that I couldn’t easily reuse in Hugo

Evaluating JavaScript static site generators

While the other sites are both single page web apps, I wanted to stick with a static site generator for the blog. When it comes to JavaScript static site generators, I evaluated several options:

Next.js↗ - The obvious choice given its popularity, but felt like overkill for a simple blog. The app router and server components, while powerful, added complexity I didn’t need. Plus, the build times for a large blog can get hefty.
Gatsby↗ - I had used Gatsby in the past and appreciated its GraphQL data layer, but the framework has lost momentum in recent years. The plugin ecosystem felt stagnant, and I wanted something with a more active community.
Eleventy (11ty)↗ - A strong contender with great flexibility and performance. However, it felt too minimal - I’d need to build a lot of structure myself. The template language agnosticism, while nice, meant less cohesive tooling.
VitePress↗ - Excellent for documentation sites but too opinionated for a personal blog. The Vue-centric approach also meant I’d be writing components in a framework I don’t use elsewhere.
Astro↗ - After working through a “Hello World” blog implementation, Astro emerged as the clear winner.

Here’s why it clicked with me:

Content Collections: Type-safe content management with Zod schemas felt like a massive upgrade
Framework agnostic: While I used React components where needed, I wasn’t locked into any framework
Island architecture: Ship only the JavaScript you need, when you need it
MDX first-class support: Unlike Hugo’s shortcodes, MDX components are just JavaScript/React
Excellent DX: TypeScript throughout, great error messages, fast dev server
Active ecosystem: Integrations for everything I needed (Tailwind, image optimization, sitemap, etc.)
Performance: Nearly perfect Lighthouse scores out of the box

The tipping point

The final push came when I wanted to overhaul my automated weekly music posts. The Python/CrewAI implementation worked, but required context switching between languages and tools. With Astro and JavaScript, I could:

Keep all automation scripts in the same language as the blog
Share types and utilities between the blog and automation
Use the same content collections API to generate posts
Leverage the Node.js ecosystem for API integrations

It just made sense to consolidate around a single stack.

The new Astro-based tech stack

The blog is now built with a modern Astro and JavaScript-first stack:

Astro↗ - The core framework, providing excellent performance and flexibility
Tailwind CSS v4↗ - For utility-first styling with dark mode support
Expressive Code↗ - Beautiful syntax highlighting with GitHub Dark/Light themes
Pagefind↗ - Fast, client-side search functionality
Swup↗ - Smooth page transitions
Tabler Icons↗ - Icon library
Cloudflare Pages↗ - Hosting and deployment

Key features

Content Collections

The site uses Astro’s content collections for type-safe content management. Blog posts are stored in src/content/blog/ as MDX files with Zod schema validation for frontmatter. This provides compile-time safety while maintaining backward compatibility with Hugo-style frontmatter.

Auto-generated OpenGraph images

Using astro-og-canvas, every blog post automatically gets a custom OpenGraph image generated at build time. These images feature the site logo, gradient background, and consistent branding - perfect for social media sharing.

For example, here is what the OpenGraph image looks like for the post I made last week:

Learn to install n8n locally with Docker and Cloudflare Tunnel. Includes PostgreSQL setup and Zero Trust security for home lab deployment.

russ.cloud

MDX component system

Global MDX components are automatically available in all blog posts without imports. This includes:

Media embeds: YouTube, Instagram, Giphy, Reddit
Audio: MP3/OGG/WAV player, Apple Music
Content: LinkPreview, ChatMessage
Callouts: Note, Tip, Info, Important, Warning, Caution
LightGallery: Image galleries with a custom meta file plugin for captions

The LightGallery component maintains compatibility with Hugo’s .meta file system, making the migration seamless.

Weekly music posts automation

One of the most exciting updates was to the automated “Weekly Tunes” generator. This system:

Fetches listening stats from Last.fm for the previous week
Retrieves album/artist metadata and images from russ.fm↗
Uses AI (LangChain with OpenAI GPT-5) to research each album and write engaging content
Generates a complete MDX post with galleries

This was migrated from a Python/CrewAI implementation to JavaScript/LangChain.js, providing better integration with the Astro ecosystem, and also it gave me chance to roll my sleeves up and play with LangChain.

For more information on the old implementation see the following post:

russ.cloud

You can view the results of the new implementation at this post.

SEO and performance

The site includes comprehensive SEO optimization via astro-seo-plugin:

Automatic sitemap generation (excluding draft posts)
robots.txt generation
RSS feed at /rss.xml
Canonical URLs and meta tags
Twitter Card and OpenGraph support

Performance is excellent with nearly perfect Lighthouse scores, thanks to Astro’s island architecture and static site generation.

Dark mode and styling

Dark mode is implemented via a .dark class on the <html> element, with Tailwind CSS v4 providing theme-aware utilities. The theme switcher persists preferences and integrates with Expressive Code for syntax highlighting.

Page transitions

Swup provides smooth fade transitions between pages while maintaining performance. Careful cleanup hooks prevent DOM artifacts from libraries like LightGallery that append to document.body.

Migration considerations

The migration involved several key decisions:

Hugo shortcodes → MDX components: All custom shortcodes were reimplemented as Astro components
URL structure: Maintained the date-based pattern /[year]/[month]/[day]/[slug] for backward compatibility
Frontmatter: Support for both Astro and Hugo-style fields ensures old posts work without modification
Image handling: Assets moved from static/ to src/assets/ for Astro’s image optimization
Build optimization: @playform/compress handles compression of all static assets

What’s next?

While I am happy with the new setup, there are still a few things I want to improve - but I am holding off on those for now as I slowly bring my site score on Lighthouse up to 100↗ and work through the issues which ahrefs↗ is flagging.

Summary

After five years running on Hugo and the PaperMod theme, this blog has been completely rebuilt using a modern JavaScript-first stack centered around Astro. The migration brings significant improvements including type-safe content collections with Zod validation, automatically generated OpenGraph images for every post, and a global MDX component system that eliminates the need for imports. Key features include seamless media embeds, a migrated weekly music automation system (now using JavaScript/LangChain.js instead of Python/CrewAI), comprehensive SEO optimization, and smooth page transitions via Swup. The new stack maintains backward compatibility with Hugo-style frontmatter and URL structures while leveraging Astro’s island architecture for excellent performance. The migration required reimplementing all Hugo shortcodes as Astro components and reorganizing assets, but the result is a more maintainable codebase that aligns with modern web development practices and the author’s other projects.

Install n8n locally using Cloudflare

web.site@mckendrick.email (Russ McKendrick) — Sat, 04 Oct 2025 17:52:55 GMT

I finally got around to getting a small AMD Ryzen 7 Mini PC with 32GB of RAM to use as part of my home lab to replace some aging Raspberry Pis.

While it came with Windows 11 (which i am sure was fine and not dodgy in any way) I instead opted to run the server version of Ubuntu 24.04 LTS and immediately trashed the Windows install. Given everyone is making videos about running n8n↗ on VPS servers I decided to do the same and run it locally on my new machine.

Installing Docker

As I was running a fresh Ubuntu 24.04 install I needed to install Docker before I could run n8n locally, To do this I first needed to install the prerequisites and then setup the official Docker repository:

Installing the prerequisites and setting up the Docker repository

sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update

Once the repository is setup I could install Docker itself and configure it to run as the current user:

Installing Docker

sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y
sudo usermod -aG docker $USER
newgrp docker

Finally, I checked the Docker version to ensure it was installed correctly:

Checking the Docker version

docker --version
Docker version 28.5.0, build 887030f

With that done I could start to install n8n locally.

Preparing the environment

As I will be running a few different services locally on the same machine I decided to use Docker Compose and keep all of the services needed for n8n in one file. We will be running the following containers:

docker.n8n.io/n8nio/n8n↗: The main n8n service
postgres:16↗: The PostgreSQL database for n8n
cloudflare/cloudflared:latest↗: The Cloudflare Tunnel for n8n

Setting up Cloudflare Tunnel

I host all of my domains via Cloudflare (as well as this blog) so using their Tunnel↗ service for routing the traffic to my local n8n instance was a no-brainer.

To do this signup for their Zero Trust service, using the free tier is fine, once you have signed up you need to create a new tunnel, you can this by going Network > Tunnels > Create a Tunnel.

On the Select your tunnel type screen select Cloudflare. Next you will be asked to name the tunnel, in the screens below I have called mine n8nBlogPost.

Next you will be taken to the Install and run connectors screen, here select Docker, copy and page the command you get given as this contains the --token parameter you will need to use later the command will look something like this:

Copying the Cloudflare Tunnel token

docker run cloudflare/cloudflared:latest tunnel \
  --no-autoupdate run \
  --token eyJhIjoiMzQwODksoTHingRealyLong5YzNotaR4alTok3N

We won’t be bringing the tunnel up here, we will do that later when we setup the Docker Compose file so just move onto the next screen.

Here, we will be setting up the domain and ports for the tunnel, in the screens below I have called mine n8n.domain.com and have used the service we will be tunnelling to:

As you can see from the screenshot above I have set the Hostname as n8n.wibble.foo (Yes wibble.foo↗ is a real domain I own) and as we will be proxying to our local n8n instance on port 5678 I set the type to HTTP and URL as n8n:5678.

Once setup we can click on Complete setup and you will be taken back to the Cloudflare Tunnel screen where you will see your new tunnel listed, as we haven’t brought up the connector you will see it as INACTIVE, don’t worry about this yet.

Setting the folders and .env file

First we need to generate a secure random key for which will be used for encryption, you can do this by running the following command:

Generating a secure random key for encryption

openssl rand -hex 32

Make a note of the random key and we can move on to creating the folders and setting the permissions:

Setting the folders

sudo mkdir -p /opt/n8n/local-files
sudo chown -R $USER:$USER /opt/n8n
sudo chown -R 1000:1000 /opt/n8n/local-files
cd /opt/n8n/
vi /opt/n8n/.env

Inside the .env file we need to set the following variables (make sure you use your own values):

Environment variables

# Domain Configuration
PUBLIC_DOMAIN=n8n.domain.com

# PostgreSQL Configuration
POSTGRES_USER=root
POSTGRES_PASSWORD=some-secure-password-here
POSTGRES_DB=n8n
POSTGRES_NON_ROOT_USER=n8n
POSTGRES_NON_ROOT_PASSWORD=some-secure-password-here

# Timezone
GENERIC_TIMEZONE=Europe/London

# n8n Configuration
# IMPORTANT: Generate a secure random key for encryption
# Use: openssl rand -hex 32
N8N_ENCRYPTION_KEY=random-key-here

# Cloudflare Tunnel Token (get from Cloudflare dashboard)
CLOUDFLARE_TUNNEL_TOKEN=your-cloudflare-tunnel-token

Enter the values you generated for the encryption key, set some passwords for the PostgreSQL database and add the Cloudflare Tunnel Token we grabbed earlier (make sure the domain matches the one you set up in Cloudflare) and save the file.

Download the Database setup script

Next we need to create a script to setup the database:

Creating the database setup script

curl -L https://raw.githubusercontent.com/russmckendrick/local-n8n-with-cloudflare/refs/heads/main/init-data.sh > /opt/n8n/init-data.sh

This will download the script from my local-n8n-with-cloudflare↗ GitHub repository.

Setup Docker Compose file

Next we need to create a Docker Compose file:

Creating the Docker Compose file

curl -L https://raw.githubusercontent.com/russmckendrick/local-n8n-with-cloudflare/refs/heads/main/docker-compose.yml > /opt/n8n/docker-compose.yml

This will download the following file:

Docker Compose file

volumes:
  db_storage:
    driver: local
  n8n_data:
    driver: local

networks:
  backend:
    driver: bridge
  cloudflare:
    driver: bridge

services:
  postgres:
    image: postgres:16
    restart: unless-stopped
    networks:
      - backend
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
      POSTGRES_NON_ROOT_USER: ${POSTGRES_NON_ROOT_USER}
      POSTGRES_NON_ROOT_PASSWORD: ${POSTGRES_NON_ROOT_PASSWORD}
    volumes:
      - db_storage:/var/lib/postgresql/data
      - ./init-data.sh:/docker-entrypoint-initdb.d/init-data.sh
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -h localhost -U ${POSTGRES_USER} -d ${POSTGRES_DB}']
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 10s

  n8n:
    image: docker.n8n.io/n8nio/n8n
    container_name: n8n
    restart: unless-stopped
    networks:
      - backend
      - cloudflare
    environment:
      DB_TYPE: postgresdb
      DB_POSTGRESDB_HOST: postgres
      DB_POSTGRESDB_PORT: 5432
      DB_POSTGRESDB_DATABASE: ${POSTGRES_DB}
      DB_POSTGRESDB_USER: ${POSTGRES_NON_ROOT_USER}
      DB_POSTGRESDB_PASSWORD: ${POSTGRES_NON_ROOT_PASSWORD}
      EXECUTIONS_MODE: regular
      N8N_HOST: ${PUBLIC_DOMAIN}
      N8N_PROTOCOL: https
      WEBHOOK_URL: https://${PUBLIC_DOMAIN}
      GENERIC_TIMEZONE: ${GENERIC_TIMEZONE}
      TZ: ${GENERIC_TIMEZONE}
      N8N_ENCRYPTION_KEY: ${N8N_ENCRYPTION_KEY}
      N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS: true
    volumes:
      - n8n_data:/home/node/.n8n
      - ./local-files:/files
    depends_on:
      postgres:
        condition: service_healthy
    healthcheck:
      test: ['CMD-SHELL', 'wget --spider -q http://localhost:5678/healthz || exit 1']
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s

  cloudflared:
    image: cloudflare/cloudflared:latest
    container_name: cloudflared
    restart: unless-stopped
    networks:
      - cloudflare
    command: tunnel --no-autoupdate run --token ${CLOUDFLARE_TUNNEL_TOKEN}
    depends_on:
      - n8n

Bringing up the n8n containers

We now have everything in place to bring up the containers, to do this run the following command:

Bringing up the containers

docker compose -f /opt/n8n/docker-compose.yml pull

This will pull the images for the containers, next we can bring them up:

Bringing up the containers

docker compose -f /opt/n8n/docker-compose.yml up -d

This should give you the following output:

Bringing up the containers

[+] Running 5/5
 ✔ Volume n8n_n8n_data       Created       0.1s
 ✔ Volume n8n_db_storage     Created       0.0s
 ✔ Container n8n-postgres-1  Healthy       5.8s
 ✔ Container n8n             Started       5.9s
 ✔ Container cloudflared     Started       6.0s

Once the containers are up and running we can check the logs to ensure everything is working correctly:

Checking the logs

docker compose -f /opt/n8n/docker-compose.yml logs -f

You should see something like the following:

Logs

n8n          |
n8n          | [license SDK] Skipping renewal on init: license cert is not initialized
n8n          | [Data table] The Data table module is experimental and subject to change.
n8n          | Any tables added before the official release may become inaccessible at any point. Use at your own risk.
n8n          | Version: 1.113.3
n8n          |
n8n          | Editor is now accessible via:
n8n          | https://n8n.wibble.foo

If you open your browser and navigate your domain, for example https://n8n.wibble.foo↗ you should see the n8n editor, if you click on the Sign in button you should be taken to the login screen, enter the credentials you set up in the .env file and you should be logged in.

That’s it, you now have a fully working n8n instance running locally and accessible via Cloudflare Tunnel. Install the free license, grab a coffee and get started with n8n - but thats another post.

Housekeeping

If you need to update the n8n instance you can do so by running the following command:

Updating the n8n instance

docker compose -f /opt/n8n/docker-compose.yml down
docker compose -f /opt/n8n/docker-compose.yml pull
docker compose -f /opt/n8n/docker-compose.yml up -d

This will stop the current instance, pull the latest images and bring the containers back up.

If you need to remove the instance you can do so by running the following command:

Removing the n8n instance

docker compose -f /opt/n8n/docker-compose.yml down

This will remove the containers and networks, to remove the volumes you can run the following command:

Removing the volumes

docker volume rm n8n_db_storage n8n_n8n_data

Making it a little more secure

Given that we are running the n8n instance on a local resource, and the tunnel is exposing it to the internet, we can make it a little more secure by fronting the site with a Cloudflare Zero Trust login. This will prevent anyone from accessing the n8n instance without being validated via Cloudflare.

To do this go to the Cloudflare dashboard and navigate to Zero Trust > Access > Applications > Add an Application; the first screen will ask you to select the application type, select Self hosted and click on Next.

Next give your application a name, in the screens below I have called mine n8nBlogPost, then in the public hostname section enter the URL of your n8n instance, in the screens below I have entered n8n.wibble.foo:

Now we need to add a policy to the application, click on + Create new policy where you will be taken to the Add policy screen, here enter a name for the policy, I have called mine n8nBlogPostPolicy. Then scroll down to the Add rules section, by default there will an Include rule waiting to be added, in the Selector select Emails ending in and enter the domain name of your email provider, in my example I have entered wibble.email. Now click on + Add require, in the Selector this time select Country and enter the country(s) you want to allow access from, in my example I have entered United Kingdom and click on Next.

This will create a policy that will allow access to the n8n instance from the United Kingdom for users with an email ending in wibble.email. Click on Save and return to the tab where you are configuring the application, here click on Select existing policy and select the policy you just created.

Under Login methods select Accept all available identity providers and click on Next. The next two screens are both optional and can be left at their defaults. Finally, click on Save and you will be taken to the Application screen where you will see your new application listed.

Once configured and goto your n8n domain you should be prompted to login with Cloudflare, once you have done this you will be taken to the n8n editor:

If you were to try and access the n8n instance from outside the United Kingdom you would be denied access:

Same for not using the correct email ending in wibble.email. The advantage of this is that unless you meet the criteria you will not be able to access the n8n instance; which means no traffic to your local machine from random people or bots on the internet.

Summary

In this post, we explored how to install n8n on a local resource using Docker and route it via Cloudflare. We saw how easy it is to setup a local n8n instance and how to use Cloudflare Tunnel to route the traffic to it.

Here are the links to n8n and the repository we downloaded the files from:

n8n.io - AI workflow automation platform

n8n is a workflow automation platform that uniquely combines AI capabilities with business process automation, giving technical teams the flexibility of code with the speed of no-code.

n8n.io

cloudflare.com

GitHub - russmckendrick/local-n8n-with-cloudflare: n8n + Docker + Cloudflare

n8n + Docker + Cloudflare. Contribute to russmckendrick/local-n8n-with-cloudflare development by creating an account on GitHub.

Your browser does not support the audio element.

Doom or Vibe Coding

web.site@mckendrick.email (Russ McKendrick) — Mon, 26 May 2025 10:29:35 GMT

This week was an interesting one in the world of AI - we had all of Google’s announcements at Google I/O↗, some announcements from Microsoft at their Build↗ conference, and the most exciting of all - the launch of Claude 4↗ from Anthropic; and to top it off, there is a new Doom↗ game out!!!

Gaming or coding

While I am not much of a gamer, game releases like Doom prick up my ears - as most of you will know I am a macOS user, so Doom isn’t out for that. This is one of the reasons why I subscribe to Nvidia’s GeForce Now↗ service, so I started playing Doom: The Dark Ages, but, I do like puzzles, so when Anthropic announced Claude 4 I decided to park Doom and give it a go.

Russ.fm - Personal Record Collection

RussTools

A few weeks ago I used Cursor↗ to create a single-page app to help with my day job. The first of the two tools was a network design tool↗, the second was an Azure Resource Naming tool↗. I decided to make use of one of the nearly fifty domains I’ve accumulated since 2001 and stuck them on russ.tools↗. To create these tools I have used a combination of Claude 3.5 Sonnet and Google’s Gemini 2.5 Pro, and while the two tools were functional, they were very obviously created by two different models - there wasn’t much consistency, both style-wise and with the code itself.

Finishing off some existing code

After the announcement of Claude 4 and after seeing a few posts about its code improvements, I decided to use it to finish off the third tool, which was a Cron Job Expression Builder/Validator↗. While the code I already had in place was mostly non-functional and full of placeholders, this was enough for Claude 4 Sonnet to pick up where Gemini left off and finish the tool off with a single prompt; not only that, it rebuilt the interface to be more user-friendly and added a few extra features without me asking it to do anything!

More tools

Colour me impressed - so with a long weekend I decided to add some more tools: first an SSL Certificate Checker↗, then a DNS Lookup Tool↗, and then a WHOIS Lookup Tool↗.

What is missing

At this point, I had quite a good little collection of tools. I fed Claude the README↗ file and asked it what other tools would complement the existing ones, and we came up with …

Data Format Converter↗; Convert between JSON, YAML, and TOML formats with validation and formatting
Base64 Encoder/Decoder↗; Encode and decode text and files using Base64 encoding with multiple variants
JWT Decoder/Validator↗; Decode JWT tokens and validate signatures without sending to external services

Bringing it all together

While everything was pretty consistent, there were a few design issues. Not a problem, though - I asked Claude to review screenshots of each of the tools and write then implement a style guide↗ giving them a consistent look and feel across the site.

Why?

While the new Doom game is fun, I actually find puzzles more interesting. Not being a developer by trade, the whole vibe coding trend is quite a lot of fun, and I find it a lot more relaxing than Doom - errors are interesting to debug, and finding the right prompt is very satisfying when you get it right first time.

Summary

While the new Doom game offers its own kind of thrill, I found myself drawn to the intellectual puzzles of “vibe coding” with these cutting-edge AI tools. I’ve been building out my ‘RussTools’ collection, a series of single-page apps for my day job, using a mix of AI models. It’s been an interesting journey, especially seeing the inconsistencies when different models were involved. However, Claude 4 truly impressed me, not only finishing off a tricky bit of code with a single prompt but also intuitively improving the UI and adding features I hadn’t even asked for!

This success spurred me on to create even more tools, like an SSL Certificate Checker and DNS/WHOIS lookup tools. The final touch was having Claude help implement a consistent style guide across the entire site, bringing everything together beautifully. Ultimately, for me, this ‘vibe coding’ isn’t just about building; it’s about the satisfying challenge of debugging errors and crafting the perfect prompt – a far more relaxing and engaging pursuit than the chaos of Doom!

Your browser does not support the audio element.

Personal Project Updates and AI Editors

web.site@mckendrick.email (Russ McKendrick) — Sun, 12 Jan 2025 11:16:46 GMT

Over the holidays, I decided to revisit a personal project I worked on in October.

Vinyl Scrobbler Recap

The project was a very basic Python script prompted using AI. The application sits in the macOS menu bar and allows you to enter a Discogs↗ release ID to scrobble the record you are listening to in real-time to LastFM↗.

The reason for the application was not only to dip my toes into the then newly released Claude 3.5 Sonnet model↗ but also to have the artwork for the album I am listening to show up on my Tuneshine↗, as existing services didn’t quite have the functionality needed for it to work.

The end of the post came with this warning:

But that didn’t stop Tobias Butler, the creator of Tuneshine, from posting about it:

View this post on Instagram

I fixed a few annoying issues and tidied up the code a little over November and December. I was happy using it in the background as a personal project.

Enter AI-Powered Code Editors

I started talking with a colleague about AI code editors before the Christmas break. Since I had some free time, I decided to take a look at two of them.

Windsurf and a Test Project

The first editor I looked at was Windsurf by Codeium↗. My initial idea, rather than touching the Vinyl Scrobbler app, was to create a Spotify Wrapped-style end-of-year report on the albums I had purchased. As I catalog everything I buy using my Discogs account↗, which is then indexed on my own website↗, I had all the data I needed to create something.

I first ended up asking Windsurf to create a set of Python scripts, as I am getting used to working with and debugging Python code. While that was great for fetching data from Discogs, I quickly hit limitations when it came to creating the HTML for the reports. Rather than spending too much time trying to fix it, I scrapped the code and started again. This time, I used Python to generate JSON for each year and then asked Windsurf to create a single-page web app.

It suggested creating a Vue 3-powered SPA that read the JSON files and created the site, which is hosted at:

Explore my personal vinyl and music collection. Browse albums, artists, and genres from my curated record collection.

russ.fm

… As you can see, it eventually did a good job. I say “eventually” because while it worked, I got stuck in a few loops, which annoyingly ate into my prompt allowance (more on that later).

The full code and commit history can be found at https://github.com/russmckendrick/unwrapped↗.

This gave me the confidence to try something a little more challenging.

Revisiting Vinyl Scrobbler

The code I have, which produces the website at https://www.russ.fm/↗, interacts with the Apple Music API to get album covers and the links to embed the Apple Music player on the page.

Because accessing the Apple Music API requires a developer key, I had to sign up to be an Apple Developer. This meant that if I wanted to, I could properly sign the Vinyl Scrobbler application so that it didn’t warn the user that the application is not trusted when they first opened it.

This then got me thinking: maybe I could use Windsurf to port the Python code to Swift and make the application a fully native macOS app.

Windsurf + Vinyl Scrobbler

With Xcode installed and a new project created, I dropped a copy of the Python script into the project and asked Windsurf to reproduce the functionality in Swift. To my surprise, after a little back and forth about build errors, I had something that pretty much recreated the original Python script in Swift. It didn’t work initially because it appeared unable to reach the necessary APIs. However, it successfully built and launched an application that visually resembled the Python version.

Now that I had a base to build on, I could start adding functionality back in. This went as follows:

Sorting out permissions: By default, the application was being blocked from accessing the internet, which made interacting with the Discogs and LastFM APIs a little troublesome. :rofl:
Clearing up some misunderstandings: The original Python code was documented, but the documentation mostly dealt with what to do with each function rather than why it was there. This meant I had to fix some of the logic related to how it was scrobbling tracks to Last.FM.
Build errors: With each change came build errors, most of which were simple to fix. However, I did find that some fixes broke previously working functionality, which resulted in quite a bit of back and forth.

Now I had a working application, so I decided it was time to add some functionality. This was partly inspired by my experience using AI-powered tools like Windsurf for other projects and my curiosity about how far I could take this app with similar assistance.

The first new functionality I tried to add was a player window, and this is where I became a little unstuck with Windsurf.

While prompting the functionality, things started off great, but I soon found myself fighting Windsurf due to technical issues and limitations. Specifically, it started overcomplicating the code when making small changes and, worse, removed large chunks of existing working code for no apparent reason. as it started over complicating code when making small changes and worse, it removed large chunks of existing working code for no apparent reason. Normally, when an LLM starts to do that,t I start new chats with a detailed prompt; in this case, I used:

This Swift app is building and running great, but would like to add more functionality.

The files we are working with are as follows …

ConfigurationManager.swift = Loads in the configuration file from ~/.vinyl-scrobbler-config.json and sets everything up
AppDelegate.swift = The bulk of the main code
DiscogsService.swift = The Discogs part of the application
LastFMService.swift = The LastFM parts of the application

Please concentrate on only adding this functionality, which is detailed below …

< rest of the prompt here >

While this approach normally works when dealing with Claude directly I was still having problems with Windsurf - at this stage I decided to look at one of Windsurf’s competitors, Cursor, as I had used pretty much all of the tokens I had paid for and felt like I was going around in circles.

A new challenger enters the ring, Cursor.

I had used Cursor↗ when it was first launched, and I am not entirely sure why, but I didn’t get on with the interface. It felt like having a whole separate IDE when VS Code was fine seemed like overkill. Given my experience of Windsurf I decided to sign-up for the trial and give it a second chance to see if it could pick up from where Windsurf left off.

Now given that both tools use Claude 3.5 Sonnet by default, the difference was night and day. The quality of the interactions felt a lot better, and within no time, I had a player window, with artwork and track listing within the application.

Sure, it had problems when adding code that caused build errors-but these were nowhere near as dramatic as the ones Windsurf caused in that it was a problem with the new code that was added rather than the tool’s decision to remove entire functions without explanation or reason.

99% of the time, these errors were resolved by passing the line number and the error itself, and if Xcode suggested a fix, I passed that along, too. This gave me the confidence to add more functionality.

Last.FM authentication

One of the things which I had been doing, since it was only really me using the application was putting my Last.FM credentials in a text file, it was very hacky, but functional. If I was to make a little more user friendly I had to get the end users to properly authenticate against using their own Last.FM credentials. After a little looking around I found ScrobbleKit, it would handle both the authentication as well as scrobbling the tracks to Last.FM:

GitHub - tfmart/ScrobbleKit: A Swift library for interacting with the Last.fm API on Apple platforms and Linux

A Swift library for interacting with the Last.fm API on Apple platforms and Linux - tfmart/ScrobbleKit

Purz/choose-your-own-adventure · Hugging Face

With a few prompts and a little back and forth with Cursor and a little Googling I had removed my method of handling the authentication and was using ScrobbleKit to sign into Last.FM and saving the credntials to the Key Chain so I could use them each time the application launched. I had also switched over to using ScrobbleKit to send the actual scrobbles.

Searching Discogs

Up until now I was passing in the Discogs release ID to search for an album and load its details, this was good for me as I index my collection on Russ.FM↗ and one of the features I added a while ago was a button which copies the Discogs ID to the clipboard, which you can see [here](↗).

Again, with a few prompts I had the application searching Discogs and presenting the results back to the user so that can select which release to Scrobble.

Time to submit to Apple

After a little bit of tidying up and removing some of the leftover debug code, I was ready to submit the application to Apple. The application was rejected a few times because I had not removed some of the security entitlements that had allowed the application to access the configuration file in the user’s folder, which was a good catch. There was also a redundant entitlement file that had been created at some point and was doubling up on some information.

Once those issues were resolved, the application was accepted, and I published it; you can find it on the Mac App Store by clicking the banner below:

So, with that, I have what was a simple app that I would only use, converted to native Swift, and published on the Mac Store. That’s not bad, considering I am not a developer by trade, nor had I ever used Swift or Xcode before starting this project.

Things I wish I knew before starting

There are a few things I should have been doing, but I didn’t and regretted not doing so.

Branch and commit often

Create a “feature” branch in Git, commit often and don’t be afraid to roll back.

Plan and feedback

Take baby steps, it is tempting to ask the LLM to just make one big feature change in one go. Don’t do this, instead explain your plan and ask the LLM to break it down into steps, here is the initial prompt I have used to implement an upcoming feature.

We are working on a feature branch. I want to add the option to use ShazamKit to listen to what the user is playing and then use the results to search Discogs. As ShazamKit only returns the Artist and track title, we need to think carefully about how to pass this information onto Discogs. Let’s use the LastFM API to get more information on the track using track.getInfo and then the album information from there to pass onto the Discogs search we already have using the Album details there.

The interface should be a new slider activated from a new “Listen” menu item below “Load Album.” As per the rest of the application, this new slider should be modern and animated, providing user feedback on what is happening. It should also, as always, be written and implemented using pure SwiftUI and Apple’s best practices regarding creating new Views, models, and, if requested, Services.

Please do not change any existing functionality.

How does this sound?

Your approach sounds well thought out and aligns perfectly with macOS design patterns and SwiftUI best practices. Let me outline the plan and then we can implement it step by step:

Architecture Plan:

Create a ShazamService to handle audio recognition
Add a new ListenView for the sliding panel
Create a ListenViewModel to manage the state and business logic 4, Integrate with existing LastFMService and DiscogsService

The flow would be:

User activates “Listen” from menu
Sliding panel appears with animation
ShazamKit starts listening
On match:
- Get track/artist from Shazam
- Use LastFM API to get album info
- Pass album info to Discogs search
- Show results in existing Discogs view

Let me first check the existing codebase structure to ensure we maintain consistency.

< agent action taken here >

Perfect. The project follows a clean architecture. Here’s what we’ll need to create:

New Service:
- ShazamService.swift - Handles ShazamKit integration
New Views:
- ListenView.swift - Main sliding panel view
- ListenButtonView.swift - Reusable button component for listening state
New ViewModel:
- ListenViewModel.swift - Manages listening state and coordinates services
Model updates:
- ShazamMatch.swift - Model for Shazam results

Before we start implementing, I should check the entitlements file to ensure we have microphone access:

< agent action taken here >

We’ll need to add microphone access entitlements. Would you like me to proceed with:

Adding the necessary entitlements for microphone access and ShazamKit
Creating the ShazamService with basic recognition functionality
Implementing the ListenView with a sliding panel design
Integrating with existing LastFM and Discogs services
The UI will follow macOS design patterns with:
A subtle sliding panel from the right
Clear visual feedback during listening
SF Symbols for status indicators
Smooth animations for state transitions
Proper error handling and user feedback

Does this align with your vision? I can start with any of these components first, but I recommend beginning with the entitlements and ShazamService as they’re foundational to the feature.

A preview of this (and the new GUI) can been seen below:

Going around in circles

Sometimes, you will find yourself going around in a loop. You will ask to fix an error, and the LLM will try to cause another problem. You manage to prompt a fix for the new issue, but all that does is recreate the original problem, and your prompt to fix that recreates the new problem you just fixed.

Ask the LLM this one simple thing:

Are we using the simplest approach and best practices for the scale of our project structure?

A lot of LLMs like Clade 3.5 and GPT-4o love to overcomplicate their solutions and sometimes need to be told to calm things down and take a simpler approach.

Forgetfulness

At some point, the LLM will behave like it has forgotten what it is doing. At the first sign of this, start a new session and give the LLM a prompt to refresh its context of where you are in the task. To do this, you can use the plan it originally gave. Just let us know what steps it has done and which step you would like it to work on next. Ensure that you tell it to do it one step at a time.

Summary

This blog post chronicles my journey of revisiting the Vinyl Scrobbler app, initially written in Python, and transforming it into a fully native macOS app using Swift. Using AI-assisted development tools like Windsurf and Cursor, I explored new functionality, tackled technical challenges, and learned valuable lessons about working with AI-driven tools. Finally, the project culminated in the successful publication of Vinyl Scrobbler on the Mac App Store, which was fun considering my lack of prior experience with Swift or Xcode. The lessons learned during this process apply to anyone venturing into AI-assisted app development or embarking on similar projects.

Audio Summary

Your browser does not support the audio element.

My 2024 Top 20(ish)

web.site@mckendrick.email (Russ McKendrick) — Fri, 27 Dec 2024 10:03:22 GMT

This year has been particularly notable for the long-awaited return of Tears For Fears with “Songs For A Nervous Planet” to The Cure’s “Songs Of A Lost World,” we’ve seen legendary artists continuing to push boundaries.

The progressive rock scene has been especially vibrant with strong releases from Frost*, Pure Reason Revolution, and The Pineapple Thief. What’s particularly striking about 2024’s releases is the diversity of sounds - from Underworld’s electronic landscapes in “Strawberry Hotel” to Orville Peck’s unique take on country music with “Stampede.” The year has demonstrated that established artists can still innovate while staying true to their core sound.

If you’re interested in exploring my complete vinyl collection and purchases throughout 2024, don’t forget to check out my detailed tracking at this link https://www.russ.fm/wrapped/2024↗.

In no particular order, here are what I dug in 2024 …

Running Flux on macOS

web.site@mckendrick.email (Russ McKendrick) — Sun, 24 Nov 2024 14:16:40 GMT

When I last wrote about FLUX.1 from Black Forest Labs, I mentioned, “however, it quickly became clear that my M3 MacBook Pro with its 36GB of RAM wasn’t going to cut it.” Since then, I’ve upgraded to an M4 MacBook Pro with 48GB of RAM. Alongside this, substantial improvements have been made to run FLUX.1 smoothly on macOS machines via the MFLUX project↗.

In this quick post, I’ll walk you through installing and using MFLUX. Let’s dive in!

Installing MFLUX

As I’ve shared over the past few months, I use Conda to manage Python environments on my macOS machine. The first step is to create a new environment, switch to it, and install MFLUX:

Installing MFLUX

conda create -n mflux python=3.11
conda activate mflux
pip install mflux

With that, MFLUX is installed, and we’re almost ready to generate our first image.

Generating images with FLUX.1 [schnell]

With MFLUX installed, the mflux-generate command becomes available. On its first run, it will download the models and necessary files-over 30GB in total, so be prepared for a bit of a wait:

Something here

mflux-generate \
--prompt "Luxury food photograph" \
--model schnell \
--steps 2 \
--seed 2 \
--height 1024 \
--width 1024

Once done, you’ll receive an image like this:

Sample FLUX.1 [schnell] images

Here are more samples using FLUX.1 [Schnell], generated by altering only the prompts:

The prompts for each of the images are below:

Each image adhered closely to its prompt and rendered text beautifully, taking about 30 seconds to generate.

Generating images with FLUX.1 [dev]

Switching to FLUX.1 [Dev] only requires changing the --model flag to dev. However, this model requires login credentials for Hugging Face. Install the CLI and log in:

Install the Hugging Face CLI and login

brew install hugging face-cli
huggingface-cli login

After logging in, download the FLUX.1 [Dev] model (another 30GB). Running the same command with adjusted steps yields improved results.

Oh, well that didn’t work as expected, what happened? Well, FLUX.1 [schnell] is optimized for speed and is capable of generating images in just a small number of steps, like the 2 we used. This makes it significantly faster than FLUX.1 [dev], which typically requires 20 steps or more for image generation. The trade-off for this speed is a somewhat lower image quality compared to FLUX.1 dev.

Lets add some more steps:

Running with more steps

mflux-generate \
--prompt "Luxury food photograph" \
--model dev \
--steps 20 \
--seed 20 \
--height 1024 \
--width 1024

What difference did increase the number of steps to 20 have? Well it now took just over 5 minutes to generate then image, but as you can see the quality is a lot higher.

Sample FLUX.1 [dev] images

Here are some samples of the same prompts we used for FLUX.1 [schnell], but this time the number of steps was updated to 20 and also the so was the seed.

I think this is one of the only times I have taxed my M4 (so far) that the fans have kicked in while running the command. I think that FLUX.1 [dev] did a way better job on the first three images than FLUX.1 [schnell], what do you think?

Adding LoRAs

Now we have FLUX.1 [dev] running we can also add LoRAs:

LoRAs (Low-Rank Adaptations) for Flux are specialized fine-tuning modules that enhance the capabilities of the Flux image generation model. These adaptations allow users to customize Flux’s output for specific styles, concepts, or characters without retraining the entire model.

In the example below I am using to the following LoRA:

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

All you need to do it download purz-cy04-choose_your_own_adventure.safetensors file and then add the path to your command:

Something here

mflux-generate \
--prompt 'cy04, a book titled "Paths of the Vale". A breathtaking fantasy scene unfolds on the illustrated cover. A lone adventurer clad in a flowing cloak stands at a crossroads surrounded by glowing ethereal pathways that twist and ascend into a star-streaked sky. Each path is lined with unique details: one shimmering with golden light and blooming flowers, another cloaked in shadow with ominous ruins, and a third sparkling with icy frost and towering crystalline spires. In the distance, a mysterious, floating vale hovers, its surface glowing with pulsating, magical runes. The adventurer holds an ancient, glowing compass, its needle spinning wildly, reflecting the uncertainty of the journey ahead.' \
--model dev \
--steps 30 \
--seed 20 \
--height 1024 \
--width 720 \
--lora-paths purz-cy04-choose_your_own_adventure.safetensors

In this case the LoRA is referenced using cy04 in your prompt, this gave me the following:

As you can see from the prompt, I used a few more steps for this one and adjusted the size to make it more book shaped - the other over you may have noticed is that image above is a little larger than 1024x720. To upscale the image is used Upscayl↗.

Summary

In this post, I’ve explored installing and running FLUX.1 with MFLUX on macOS, demonstrated the differences between the FLUX.1 [Schnell] and [Dev] models, and showcased how LoRAs enhance customization. Whether you’re focused on speed or quality, these tools deliver impressive results on macOS.

For more information on MFLUX see the following GitHub repo:

GitHub - filipstrand/mflux: MLX native implementations of state-of-the-art generative image models

MLX native implementations of state-of-the-art generative image models - filipstrand/mflux

Zsh Conda Environment Selector Function

web.site@mckendrick.email (Russ McKendrick) — Sun, 17 Nov 2024 12:43:53 GMT

Managing Python environments can sometimes feel like navigating a jungle 🌴🐍, especially when dealing with dependency conflicts. To streamline this process, I created a handy Zsh function to add to your dotfile for switching Conda environments quickly and efficiently. 🚀 Let me walk you through it!

Why I Needed This Function 🛠️

After months of using Conda to manage Python environments on my macOS machines, I realized I had built up quite a collection. Switching between them manually became tedious. So, I decided to create a quick, interactive function to simplify the process. 🎯

How It Works 🔧

Here’s the magic in action ✨:

Something here (YAML)

# Conda Select function with colors and styling
function cs() {
    # Colors and formatting
    local BLUE='\033[0;34m'
    local GREEN='\033[0;32m'
    local YELLOW='\033[1;33m'
    local CYAN='\033[0;36m'
    local BOLD='\033[1m'
    local NC='\033[0m' # No Color

    # Get list of conda environments
    local environments=($(conda env list | grep -v '^#' | awk '{print $1}' | grep -v '^$'))

    # Print header with styling
    echo "\n${BOLD}${BLUE}🐍 Available Conda Environments:${NC}\n"

    # Print environments with numbers and colors
    for i in {1..${#environments[@]}}; do
        if [ "${environments[$i]}" = "base" ]; then
            echo "  ${YELLOW}$i)${NC} ${CYAN}${environments[$i]}${NC} ${GREEN}(base)${NC}"
        else
            echo "  ${YELLOW}$i)${NC} ${CYAN}${environments[$i]}${NC}"
        fi
    done

    # Get user selection with styled prompt
    echo "\n${BOLD}${BLUE}Enter environment number (${GREEN}1-${#environments[@]}${BLUE}):${NC} "
    read selection

    # Validate input
    if [[ "$selection" =~ ^[0-9]+$ ]] && [ "$selection" -ge 1 ] && [ "$selection" -le "${#environments[@]}" ]; then
        echo "${GREEN}✓ Activating ${CYAN}${environments[$selection]}${GREEN} environment...${NC}"
        conda activate "${environments[$selection]}"
    else
        echo "${YELLOW}⚠️  Invalid selection${NC}"
    fi
}

Using the Function ⚡

Running the cs command brings up a list of your Conda environments with some stylish colors and formatting. 🎨 All you have to do is pick a number, and the script takes care of the rest!

And there you have it! A quick and stylish way to manage your Python environments in Zsh 🐍. Have fun coding and keep your workflows clean and efficient! 🚀✨

Vinyl Scrobbling macOS App

web.site@mckendrick.email (Russ McKendrick) — Mon, 28 Oct 2024 13:47:28 GMT

Like many techies on their day off, I completely abandoned my planned tasks and embarked on an entirely unplanned project: creating a macOS application. Well, to be more precise, I decided to prompt one using Claude 3.5 Sonnet↗.

With Anthropic’s recent updates to their AI model, I figured it was the perfect opportunity to test its capabilities while solving a personal need.

The Catalyst: My New Tuneshine Device

Recently, I acquired a Tuneshine↗ - a clever device that arrived last week. It works wonderfully with my streaming services of choice, including Apple Music↗ and Spotify↗, and offers integration with LastFM↗.

For those unfamiliar with the device, Techmoan provides an excellent overview in their review:

GitHub - russmckendrick/vinyl-scrobbler: A very small and very basic status bar app to scrobble records. See the linked blog post for more details.

For those that know me, you will know that the Tuneshine ticks a lot of geeky boxes ✅.

The Challenge

Given the Tuneshine’s LastFM support, I assumed I could continue using OpenScrobbler↗ to log my vinyl listening sessions and have the album artwork display on the device. I was mistaken - but not due to any fault in either the Tuneshine or OpenScrobbler.

The issue lies in how these services interact with LastFM. The Tuneshine is designed for online players and monitors tracks that are “Scrobbling now” through LastFM’s API. In contrast, OpenScrobbler manually logs tracks after they’ve been played, which means they don’t trigger the “now playing” status that the Tuneshine looks for.

After searching for an alternative - specifically, an application that would run in the background and send real-time “now playing” requests to the LastFM API↗ - I came up empty-handed. That’s when I decided to prompt my own solution.

Where to start?

The place to start was prompting a quick MVP, could I get Clade to give me something that

I would like to have python script that scrobles to Last.FM in real-time so it shows on now playing - can you help?

Let me help you create a script for Last.FM scrobbling. I’ll create a Python script that can handle real-time scrobbling.

Claude returned the following code (I filled in the track details at the bottom):

Something here (YAML)

import pylast
import time
from datetime import datetime
import logging
import json
import os

class LastFMScrobbler:
    def __init__(self, config_path='config.json'):
        self.load_config(config_path)
        self.setup_logging()
        self.initialize_network()

    def load_config(self, config_path):
        """Load configuration from JSON file"""
        try:
            with open(config_path, 'r') as f:
                config = json.load(f)
                self.api_key = config['api_key']
                self.api_secret = config['api_secret']
                self.username = config['username']
                self.password_hash = config['password_hash']
        except FileNotFoundError:
            raise Exception("Config file not found. Please create a config.json file.")

    def setup_logging(self):
        """Setup logging configuration"""
        logging.basicConfig(
            level=logging.INFO,
            format='%(asctime)s - %(levelname)s - %(message)s',
            handlers=[
                logging.FileHandler('scrobbler.log'),
                logging.StreamHandler()
            ]
        )
        self.logger = logging.getLogger(__name__)

    def initialize_network(self):
        """Initialize Last.FM network connection"""
        try:
            self.network = pylast.LastFMNetwork(
                api_key=self.api_key,
                api_secret=self.api_secret,
                username=self.username,
                password_hash=self.password_hash
            )
            self.logger.info("Successfully connected to Last.FM network")
        except Exception as e:
            self.logger.error(f"Failed to initialize Last.FM network: {str(e)}")
            raise

    def scrobble_track(self, artist, title, album=""):
        """Scrobble a track to Last.FM"""
        timestamp = int(time.time())
        try:
            self.network.scrobble(
                artist=artist,
                title=title,
                timestamp=timestamp,
                album=album
            )
            self.logger.info(f"Scrobbled: {artist} - {title}")
            return True
        except Exception as e:
            self.logger.error(f"Failed to scrobble track: {str(e)}")
            return False

    def update_now_playing(self, artist, title, album=""):
        """Update now playing status"""
        try:
            self.network.update_now_playing(
                artist=artist,
                title=title,
                album=album
            )
            self.logger.info(f"Updated now playing: {artist} - {title}")
            return True
        except Exception as e:
            self.logger.error(f"Failed to update now playing: {str(e)}")
            return False

def main():
    # Example configuration file format (config.json):
    # {
    #     "api_key": "your_api_key",
    #     "api_secret": "your_api_secret",
    #     "username": "your_username",
    #     "password_hash": "your_password_hash"
    # }

    scrobbler = LastFMScrobbler()

    # Example usage:
    artist = "Tori Amos"
    title = "God"
    album = "Under The Pink"

    # Update now playing
    scrobbler.update_now_playing(artist, title, album)

    # Scrobble the track
    scrobbler.scrobble_track(artist, title, album)

if __name__ == "__main__":
    main()

After installing the pylast↗ module using the command below:

Something here

pip install pylast

I ran the code and saw that “God” by Tori Amos from the album Under The Pink had appeared in LastFM and glancing over at the Tuneshine I saw the following …

Result !!! Now that I proved it could work I could start prompting away.

Prompting the full application

I won’t fill this post with every prompt, because there were a lot of them, but after a few hours I had a Python powered stand-alone macOS application. The key fetaures I prompted were …

Search albums using their Discogs ID
Automatic track duration detection (with fallback options)
Real-time “Now Playing” status with a timer
Automatic track progression
Clean configuration management as I didn’t want to hard code any of my keys or credentials into the application
Detailed logging system

Its uses the rumps↗ to do the bulk of heavy lifting around being a status bar application and then py2app↗ to take the rumps powered status bar application and make it standalone,

Seeing it in action

It is a really basic application so lets dive straight in scrobble Damned Damned Damned↗ by The Damned↗- first off we need the Discogs ID which is “24589202”. Now we have that we can open the application and search for it:

Checking the logs at ~/.vinyl-scrobbler/vinyl_scrobbler.log I can see the following:

Something here

tail -f ~/.vinyl-scrobbler/vinyl_scrobbler.log
2024-10-28 16:45:46,156 - INFO - No Discogs duration for I Feel Alright, trying Last.fm...
2024-10-28 16:45:46,156 - INFO - Fetching duration from Last.fm for: The Damned - I Feel Alright
2024-10-28 16:45:46,156 - INFO - track.getInfo
2024-10-28 16:45:46,570 - INFO - HTTP Request: POST https://ws.audioscrobbler.com/2.0/ "HTTP/1.1 200 OK"
2024-10-28 16:45:46,573 - INFO - Found duration on Last.fm: 4:27
2024-10-28 16:45:46,573 - INFO - Menu updated successfully
2024-10-28 16:46:04,573 - INFO - Loaded album: Damned Damned Damned
2024-10-28 16:46:54,528 - INFO - track.updateNowPlaying
2024-10-28 16:46:54,773 - INFO - HTTP Request: POST https://ws.audioscrobbler.com/2.0/ "HTTP/1.1 200 OK"
2024-10-28 16:46:54,774 - INFO - Updated now playing: The Damned - Neat Neat Neat
2024-10-28 16:49:36,780 - INFO - track.scrobble
2024-10-28 16:49:36,941 - INFO - HTTP Request: POST https://ws.audioscrobbler.com/2.0/ "HTTP/1.1 200 OK"
2024-10-28 16:49:36,943 - INFO - Scrobbled: The Damned - Neat Neat Neat
2024-10-28 16:49:36,944 - INFO - track.updateNowPlaying
2024-10-28 16:49:37,093 - INFO - HTTP Request: POST https://ws.audioscrobbler.com/2.0/ "HTTP/1.1 200 OK"
2024-10-28 16:49:37,095 - INFO - Updated now playing: The Damned - Fan Club

Finally, checking the Tuneshine we can see:

Summary

What started as an off the cuff idea after waking up turned into a successful experiment in AI-assisted development using Claude 3.5 Sonnet. The challenge was to create a solution that would enable real-time LastFM scrobbling for vinyl records to work with the Tuneshine device. Through iterative prompting and development, I was were able to create a functional macOS status bar application that not only solves the original problem but includes several useful features.

The project demonstrates the potential of modern AI tools in practical application development. By leveraging Claude 3.5 Sonnet’s capabilities alongside libraries like rumps and py2app, I were was to transform a simple Python script into a fully-functional macOS application with next to no knowlege of either developing Python or native macOS applications.

For those interested in trying it out or building upon this project, the complete source code is available on GitHub at:

A very small and very basic status bar app to scrobble records. See the linked blog post for more details. - russmckendrick/vinyl-scrobbler

Now its not perfect, so I should probably add the following:

But this adhoc project served as an example of how AI can be effectively used to rapidly prototype and develop solutions for specific use cases, even when working with multiple APIs and system-level integration requirements.

Audio Summary

Your browser does not support the audio element.

Dynamic Network Config in Ansible: Local to Azure VNet

web.site@mckendrick.email (Russ McKendrick) — Sat, 21 Sep 2024 08:00:00 GMT

If like me you have been guilty of hard coding all of your network configuration in Ansible variables like this …

Hard Code Values (YAML)

  vars:
    network:
      full_range: "10.0.0.0/24"
      subnets:
        - name: "subnet1"
          address_range: "10.0.0.0/27"
        - name: "subnet2"
          address_range: "10.0.0.32/27"
        - name: "subnet3"
          address_range: "10.0.0.64/27"
        - name: "subnet4"
          address_range: "10.0.0.96/27"

… then this post is for you. Not that I am saying that is wrong, but it does mean if you want to the network range you are using them you also have to update it in four other places.

Working with network ranges dynamically in Ansible

Instead of hardcoding subnet addresses, we can leverage Ansible’s powerful filters and plugins to calculate network and subnet information dynamically. This approach enhances the flexibility and maintainability of your playbooks.

A Basic Playbook

Let’s start by creating a playbook that calculates subnet addresses based on a given network range and subnet sizes.

site.yml (YAML)

---

- name: "Network and Subnet Information Playbook"
  hosts: localhost
  gather_facts: true
  vars:
    cidr_range: "10.0.0.0/24"
    subnets:
      - name: "subnet1"
        cidr_size: 27
      - name: "subnet2"
        cidr_size: 27
      - name: "subnet3"
        cidr_size: 27
      - name: "subnet4"
        cidr_size: 27

  tasks:
    - name: "Generate network information"
      ansible.builtin.set_fact:
        network_info:
          cidr: "{{ cidr_range }}"
          available_ips: "{{ (cidr_range | ansible.utils.ipaddr('size')) - 2 }}"

    - name: "Generate subnet information"
      ansible.builtin.set_fact:
        subnet_info: "{{ subnet_info | default([]) + [{'name': item.0.name, 'cidr': cidr_range | ansible.utils.ipsubnet(item.0.cidr_size, item.1)}] }}"
      loop: "{{ subnets | zip_longest(range(subnets|length), fillvalue='')|list }}"

    - name: "Calculate detailed subnet information"
      ansible.builtin.set_fact:
        subnet_details: "{{ subnet_details | default([]) + [subnet | combine({
          'network': (subnet.cidr | ansible.utils.ipaddr('network')),
          'netmask': (subnet.cidr | ansible.utils.ipaddr('netmask')),
          'broadcast': (subnet.cidr | ansible.utils.ipaddr('broadcast')),
          'first_ip': (subnet.cidr | ansible.utils.ipaddr('1') | ansible.utils.ipaddr('address')),
          'last_ip': (subnet.cidr | ansible.utils.ipaddr('-2') | ansible.utils.ipaddr('address'))
        })] }}"
      loop: "{{ subnet_info }}"
      loop_control:
        loop_var: subnet

    - name: "Display network information"
      vars:
        network_info_output: |
          ===============================
          Network Information
          ===============================
          CIDR Range: {{ network_info.cidr }}
          Available IP Addresses: {{ network_info.available_ips }}

          Subnet Overview:
          {% for subnet in subnet_info %}
          - {{ subnet.name }}: {{ subnet.cidr }}
          {% endfor %}
      ansible.builtin.debug:
        msg: "{{ network_info_output.split('\n') }}"

    - name: "Display detailed subnet information"
      vars:
        detailed_info: |
          ===============================
          Detailed Subnet Information
          ===============================
          {% for subnet in subnet_details %}
          {{ subnet.name }}:
            CIDR: {{ subnet.cidr }}
            Network: {{ subnet.network }}
            Netmask: {{ subnet.netmask }}
            Broadcast: {{ subnet.broadcast }}
            First Usable IP: {{ subnet.first_ip }}
            Last Usable IP: {{ subnet.last_ip }}
          {% if not loop.last %}

          {% endif %}
          {% endfor %}
      ansible.builtin.debug:
        msg: "{{ detailed_info.split('\n') }}"

Breaking Down the Playbook

The playbook is designed to generate and display network and subnet information based on a given CIDR range. Let’s break it down:

Playbook Header:
- Name: “Network and Subnet Information Playbook”
- Hosts: localhost (runs on the local machine)
- gather_facts: true (collects system information)
Variables:
- cidr_range: Defines the main network CIDR (10.0.0.0/24)
- subnets: A list of 4 subnets, each with a name and CIDR size (27 bits)
Tasks: The playbook contains several tasks that generate and display network information.

Tasks

Generate network information:
- Creates a ‘network_info’ fact containing the CIDR range and available IP addresses
- Uses the ‘ansible.utils.ipaddr’ filter to calculate the number of available IPs
Generate subnet information:
- Creates a ‘subnet_info’ list containing each subnet’s name and CIDR
- Uses ‘ansible.utils.ipsubnet’ to calculate subnet CIDRs
- Utilizes a loop with zip_longest to iterate over subnets
Calculate detailed subnet information:
- Expands on ‘subnet_info’ to create ‘subnet_details’
- Adds network, netmask, broadcast, first IP, and last IP for each subnet
- Uses various ‘ansible.utils.ipaddr’ filters to calculate these values
Display network information:
- Creates a formatted output string with network overview
- Uses ansible.builtin.debug to display the information
Display detailed subnet information:
- Creates a formatted output string with detailed subnet information
- Uses ansible.builtin.debug to display the information

Understanding the ansible.utils.ipsubnet function

The ‘ansible.utils.ipsubnet’ function is used to calculate subnet information based on a given network and subnet size. In this playbook, it’s used as follows:

Dynamic Network Configuration (YAML)

cidr_range | ansible.utils.ipsubnet(item.0.cidr_size, item.1)

cidr_range: The base network (10.0.0.0/24)
item.0.cidr_size: The size of the subnet (27 in this case)
item.1: The index of the current subnet (0, 1, 2, 3)

This function calculates the CIDR for each subnet within the main network. For example:

Subnet 1: 10.0.0.0/27
Subnet 2: 10.0.0.32/27
Subnet 3: 10.0.0.64/27
Subnet 4: 10.0.0.96/27

Understanding the ansible.utils.ipaddr function

The ‘ansible.utils.ipaddr’ function is a versatile filter for IP address manipulation. In this playbook, it’s used in several ways, as you can see from the examples below the ‘ansible.utils.ipaddr’ function accepts various options to perform different operations on IP addresses and networks, making it a powerful tool for network-related tasks in Ansible playbooks.

Calculating available IPs

Dynamic Network Configuration (YAML)

 cidr_range | ansible.utils.ipaddr('size')

This calculates the total number of IP addresses in the CIDR range.

Getting network address

Dynamic Network Configuration (YAML)

```yaml {linenos=true}
 subnet.cidr | ansible.utils.ipaddr('network')

This extracts the network address from a CIDR.

#### Getting netmask
```yaml frame="code" title="Dynamic Network Configuration (YAML)" showLineNumbers
subnet.cidr | ansible.utils.ipaddr('netmask')

This calculates the subnet mask for a given CIDR.

Getting broadcast address

Dynamic Network Configuration (YAML)

subnet.cidr | ansible.utils.ipaddr('broadcast')

This calculates the broadcast address for a subnet.

Calculating first and last usable IPs

Dynamic Network Configuration (YAML)

subnet.cidr | ansible.utils.ipaddr('1') | ansible.utils.ipaddr('address')
subnet.cidr | ansible.utils.ipaddr('-2') | ansible.utils.ipaddr('address')

These calculate the first and last usable IP addresses in a subnet.

Running the Playbook

Running the site.yml playbook using the following:

Running the playbook

ansible-playbook site.yml

Gives the following output:

The Full Output

[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit
localhost does not match 'all'

PLAY [Network and Subnet Information Playbook] *************************************

TASK [Gathering Facts] *************************************************************
ok: [localhost]

TASK [Generate network information] ************************************************
ok: [localhost]

TASK [Generate subnet information] *************************************************
ok: [localhost] => (item=[{'name': 'subnet1', 'cidr_size': 27}, 0])
ok: [localhost] => (item=[{'name': 'subnet2', 'cidr_size': 27}, 1])
ok: [localhost] => (item=[{'name': 'subnet3', 'cidr_size': 27}, 2])
ok: [localhost] => (item=[{'name': 'subnet4', 'cidr_size': 27}, 3])

TASK [Calculate detailed subnet information] ***************************************
ok: [localhost] => (item={'name': 'subnet1', 'cidr': '10.0.0.0/27'})
ok: [localhost] => (item={'name': 'subnet2', 'cidr': '10.0.0.32/27'})
ok: [localhost] => (item={'name': 'subnet3', 'cidr': '10.0.0.64/27'})
ok: [localhost] => (item={'name': 'subnet4', 'cidr': '10.0.0.96/27'})

TASK [Display network information] *************************************************
ok: [localhost] => {
    "msg": [
        "===============================",
        "Network Information",
        "===============================",
        "CIDR Range: 10.0.0.0/24",
        "Available IP Addresses: 254",
        "",
        "Subnet Overview:",
        "- subnet1: 10.0.0.0/27",
        "- subnet2: 10.0.0.32/27",
        "- subnet3: 10.0.0.64/27",
        "- subnet4: 10.0.0.96/27",
        ""
    ]
}

TASK [Display detailed subnet information] ****************************************
ok: [localhost] => {
    "msg": [
        "===============================",
        "Detailed Subnet Information",
        "===============================",
        "subnet1:",
        "  CIDR: 10.0.0.0/27",
        "  Network: 10.0.0.0",
        "  Netmask: 255.255.255.224",
        "  Broadcast: 10.0.0.31",
        "  First Usable IP: 10.0.0.1",
        "  Last Usable IP: 10.0.0.30",
        "",
        "subnet2:",
        "  CIDR: 10.0.0.32/27",
        "  Network: 10.0.0.32",
        "  Netmask: 255.255.255.224",
        "  Broadcast: 10.0.0.63",
        "  First Usable IP: 10.0.0.33",
        "  Last Usable IP: 10.0.0.62",
        "",
        "subnet3:",
        "  CIDR: 10.0.0.64/27",
        "  Network: 10.0.0.64",
        "  Netmask: 255.255.255.224",
        "  Broadcast: 10.0.0.95",
        "  First Usable IP: 10.0.0.65",
        "  Last Usable IP: 10.0.0.94",
        "",
        "subnet4:",
        "  CIDR: 10.0.0.96/27",
        "  Network: 10.0.0.96",
        "  Netmask: 255.255.255.224",
        "  Broadcast: 10.0.0.127",
        "  First Usable IP: 10.0.0.97",
        "  Last Usable IP: 10.0.0.126",
        ""
    ]
}

PLAY RECAP ************************************************************************
localhost: ok=6  changed=0 unreachable=0  failed=0  skipped=0  rescued=0  ignored=0

As you can see, we are getting quite alot of information about the network with only telling Ansible the CIDR range to use and the number of subnets and their size.

Deploying an Azure Virtual Network

This Ansible playbook automates the creation of an Azure Virtual Network (VNet) and multiple subnets. It leverages the same network calculation concepts as our previous example, applying them to real Azure resources. The playbook creates a VNet with a specified CIDR range and dynamically generates subnets based on given parameters. It uses Azure Resource Manager modules to interact with Azure, and concludes by displaying detailed information about the created network infrastructure - which you probably would never include in your own Playbooks - I have included it here so we can see the output.

This automation streamlines the process of setting up complex network configurations in Azure, ensuring consistency and reducing manual errors.

The Playbook itself looks like the following:

The Azure Vnet Playbook

azure.yml (YAML)

---

- name: "Create Azure Virtual Network and Subnets"
  hosts: localhost
  connection: local
  gather_facts: false

  vars:
    location: "uksouth"
    resource_group: "rg-ansible-demo-uks"
    vnet_name: "vnet-ansible-demo-uks"
    cidr_range: "10.0.0.0/24"
    subnets:
      - name: "subnet1"
        cidr_size: 27
      - name: "subnet2"
        cidr_size: 27
      - name: "subnet3"
        cidr_size: 27
      - name: "subnet4"
        cidr_size: 27

  tasks:
    - name: "Ensure resource group exists"
      azure.azcollection.azure_rm_resourcegroup:
        name: "{{ resource_group }}"
        location: "{{ location }}"

    - name: "Calculate subnet CIDRs"
      ansible.builtin.set_fact:
        subnet_info: "{{ subnet_info | default([]) + [{'name': item.0.name, 'cidr': cidr_range | ansible.utils.ipsubnet(item.0.cidr_size, item.1)}] }}"
      loop: "{{ subnets | zip_longest(range(subnets|length), fillvalue='')|list }}"

    - name: "Create virtual network"
      azure.azcollection.azure_rm_virtualnetwork:
        resource_group: "{{ resource_group }}"
        name: "{{ vnet_name }}"
        address_prefixes_cidr:
          - "{{ cidr_range }}"

    - name: "Create subnets"
      azure.azcollection.azure_rm_subnet:
        resource_group: "{{ resource_group }}"
        name: "{{ item.name }}"
        address_prefix_cidr: "{{ item.cidr }}"
        virtual_network_name: "{{ vnet_name }}"
      loop: "{{ subnet_info }}"

    - name: "Gather VNet facts"
      azure.azcollection.azure_rm_virtualnetwork_info:
        resource_group: "{{ resource_group }}"
        name: "{{ vnet_name }}"
      register: vnet_facts

    - name: "Gather VNet facts"
      azure.azcollection.azure_rm_virtualnetwork_info:
        resource_group: "{{ resource_group }}"
        name: "{{ vnet_name }}"
      register: vnet_facts

    - name: "Calculate detailed subnet information"
      ansible.builtin.set_fact:
        subnet_details: "{{ subnet_details | default([]) + [subnet | combine({
          'network': (subnet.address_prefix | ansible.utils.ipaddr('network')),
          'netmask': (subnet.address_prefix | ansible.utils.ipaddr('netmask')),
          'broadcast': (subnet.address_prefix | ansible.utils.ipaddr('broadcast')),
          'first_ip': (subnet.address_prefix | ansible.utils.ipaddr('1') | ansible.utils.ipaddr('address')),
          'last_ip': (subnet.address_prefix | ansible.utils.ipaddr('-2') | ansible.utils.ipaddr('address'))
        })] }}"
      loop: "{{ vnet_facts.virtualnetworks[0].subnets }}"
      loop_control:
        loop_var: subnet

    - name: "Display detailed VNet and Subnet information"
      vars:
        detailed_info: |
          ===============================
          Virtual Network Information
          ===============================
          Name: {{ vnet_name }}
          Resource Group: {{ resource_group }}
          Location: {{ location }}
          Address Space: {{ vnet_facts.virtualnetworks[0].address_prefixes[0] }}

          ===============================
          Detailed Subnet Information
          ===============================
          {% for subnet in subnet_details %}
          {{ subnet.name }}:
            CIDR: {{ subnet.address_prefix }}
            Network: {{ subnet.network }}
            Netmask: {{ subnet.netmask }}
            Broadcast: {{ subnet.broadcast }}
            First Usable IP: {{ subnet.first_ip }}
            Last Usable IP: {{ subnet.last_ip }}
          {% if not loop.last %}

          {% endif %}
          {% endfor %}
      ansible.builtin.debug:
        msg: "{{ detailed_info.split('\n') }}"

Running the Playbook

Running the azure.yml playbook using the following:

Running the playbook

ansible-playbook site.yml

Gives the following output:

The Full Output

PLAY [Create Azure Virtual Network and Subnets] ************************************

TASK [Ensure resource group exists] ************************************************
changed: [localhost]

TASK [Calculate subnet CIDRs] ******************************************************
ok: [localhost] => (item=[{'name': 'subnet1', 'cidr_size': 27}, 0])
ok: [localhost] => (item=[{'name': 'subnet2', 'cidr_size': 27}, 1])
ok: [localhost] => (item=[{'name': 'subnet3', 'cidr_size': 27}, 2])
ok: [localhost] => (item=[{'name': 'subnet4', 'cidr_size': 27}, 3])

TASK [Create virtual network] ******************************************************
changed: [localhost]

TASK [Create subnets] ***************************************************************
changed: [localhost] => (item={'name': 'subnet1', 'cidr': '10.0.0.0/27'})
changed: [localhost] => (item={'name': 'subnet2', 'cidr': '10.0.0.32/27'})
changed: [localhost] => (item={'name': 'subnet3', 'cidr': '10.0.0.64/27'})
changed: [localhost] => (item={'name': 'subnet4', 'cidr': '10.0.0.96/27'})

TASK [Gather VNet facts] ************************************************************
ok: [localhost]

TASK [Gather VNet facts] *************************************************************
ok: [localhost]

TASK [Calculate detailed subnet information] *****************************************
ok: [localhost] => (item={'id': '/subscriptions/ce7aa0b9-3545-4104-99dc-d4d082339a05/resourceGroups/rg-ansible-demo-uks/providers/Microsoft.Network/virtualNetworks/vnet-ansible-demo-uks/subnets/subnet1', 'name': 'subnet1', 'provisioning_state': 'Succeeded', 'address_prefix': '10.0.0.0/27', 'address_prefixes': None, 'network_security_group': None, 'route_table': None})
ok: [localhost] => (item={'id': '/subscriptions/ce7aa0b9-3545-4104-99dc-d4d082339a05/resourceGroups/rg-ansible-demo-uks/providers/Microsoft.Network/virtualNetworks/vnet-ansible-demo-uks/subnets/subnet2', 'name': 'subnet2', 'provisioning_state': 'Succeeded', 'address_prefix': '10.0.0.32/27', 'address_prefixes': None, 'network_security_group': None, 'route_table': None})
ok: [localhost] => (item={'id': '/subscriptions/ce7aa0b9-3545-4104-99dc-d4d082339a05/resourceGroups/rg-ansible-demo-uks/providers/Microsoft.Network/virtualNetworks/vnet-ansible-demo-uks/subnets/subnet3', 'name': 'subnet3', 'provisioning_state': 'Succeeded', 'address_prefix': '10.0.0.64/27', 'address_prefixes': None, 'network_security_group': None, 'route_table': None})
ok: [localhost] => (item={'id': '/subscriptions/ce7aa0b9-3545-4104-99dc-d4d082339a05/resourceGroups/rg-ansible-demo-uks/providers/Microsoft.Network/virtualNetworks/vnet-ansible-demo-uks/subnets/subnet4', 'name': 'subnet4', 'provisioning_state': 'Succeeded', 'address_prefix': '10.0.0.96/27', 'address_prefixes': None, 'network_security_group': None, 'route_table': None})

TASK [Display detailed VNet and Subnet information] **********************************
ok: [localhost] => {
    "msg": [
        "===============================",
        "Virtual Network Information",
        "===============================",
        "Name: vnet-ansible-demo-uks",
        "Resource Group: rg-ansible-demo-uks",
        "Location: uksouth",
        "Address Space: 10.0.0.0/24",
        "",
        "===============================",
        "Detailed Subnet Information",
        "===============================",
        "subnet1:",
        "  CIDR: 10.0.0.0/27",
        "  Network: 10.0.0.0",
        "  Netmask: 255.255.255.224",
        "  Broadcast: 10.0.0.31",
        "  First Usable IP: 10.0.0.1",
        "  Last Usable IP: 10.0.0.30",
        "",
        "subnet2:",
        "  CIDR: 10.0.0.32/27",
        "  Network: 10.0.0.32",
        "  Netmask: 255.255.255.224",
        "  Broadcast: 10.0.0.63",
        "  First Usable IP: 10.0.0.33",
        "  Last Usable IP: 10.0.0.62",
        "",
        "subnet3:",
        "  CIDR: 10.0.0.64/27",
        "  Network: 10.0.0.64",
        "  Netmask: 255.255.255.224",
        "  Broadcast: 10.0.0.95",
        "  First Usable IP: 10.0.0.65",
        "  Last Usable IP: 10.0.0.94",
        "",
        "subnet4:",
        "  CIDR: 10.0.0.96/27",
        "  Network: 10.0.0.96",
        "  Netmask: 255.255.255.224",
        "  Broadcast: 10.0.0.127",
        "  First Usable IP: 10.0.0.97",
        "  Last Usable IP: 10.0.0.126",
        ""
    ]
}

PLAY RECAP ***************************************************************************
localhost:   ok=8  changed=3  unreachable=0  failed=0  skipped=0  rescued=0  ignored=0

As you can see, we have the VNet and subnets all created as well.

Extending the Playbook further

We could extend the playbook further, for example we could deploy an Azure Application Gateway into one of the subnets and get its static internal IP address by figuring out the forth IP address in subnet1 by using:

Adding an Azure Application Gateway (YAML)

- name: "Calculate 4th IP address of subnet1"
  ansible.builtin.set_fact:
    appgw_ip: "{{ subnet_details | selectattr('name', 'equalto', 'subnet1') | map(attribute='cidr') | first | ansible.utils.ipaddr('4') | ansible.utils.ipaddr('address') }}"

- name: "Create public IP address for Application Gateway"
  azure.azcollection.azure_rm_publicipaddress:
    resource_group: "{{ resource_group }}"
    name: "pip-agw-ansible-demo-uks"
    allocation_method: Static
    sku: Standard

- name: "Create Application Gateway"
  azure.azcollection.azure_rm_applicationgateway:
    resource_group: "{{ resource_group }}"
    name: "pip-agw-ansible-demo-uks"
    sku:
      name: Standard_v2
      tier: Standard_v2
    gateway_ip_configurations:
      - subnet:
          id: "{{ vnet_facts.virtualnetworks[0].subnets | selectattr('name', 'equalto', 'subnet1') | map(attribute='id') | first }}"
        name: appGatewayIpConfig
    frontend_ip_configurations:
      - public_ip_address: "pip-agw-ansible-demo-uks"
        name: appGatewayFrontendIP
    frontend_ports:
      - port: 80
        name: appGatewayFrontendPort
    backend_address_pools:
      - name: appGatewayBackendPool
    backend_http_settings_collection:
      - port: 80
        protocol: Http
        cookie_based_affinity: Disabled
        name: appGatewayBackendHttpSettings
    http_listeners:
      - frontend_ip_configuration: appGatewayFrontendIP
        frontend_port: appGatewayFrontendPort
        name: appGatewayHttpListener
    request_routing_rules:
      - rule_type: Basic
        http_listener: appGatewayHttpListener
        backend_address_pool: appGatewayBackendPool
        backend_http_settings: appGatewayBackendHttpSettings
        name: rule1
    ip_configurations:
      - subnet:
          id: "{{ vnet_facts.virtualnetworks[0].subnets | selectattr('name', 'equalto', 'subnet1') | map(attribute='id') | first }}"
        private_ip_address: "{{ appgw_ip }}"
        private_ip_allocation_method: Static

Summary

In this post, we’ve explored the power of dynamic network configuration in Ansible, moving beyond hardcoded values to create more flexible and maintainable playbooks. We started with a local playbook that calculates subnet information based on a given CIDR range, utilizing Ansible’s powerful ansible.utils.ipsubnet and ansible.utils.ipaddr functions.

We then applied these concepts to a real-world scenario, creating an Azure Virtual Network with multiple subnets. This approach not only simplifies our code but also makes it more adaptable to changes in network requirements.

Key takeaways include:

Using ansible.utils.ipsubnet to dynamically calculate subnet CIDRs
Leveraging ansible.utils.ipaddr for detailed IP address manipulation
Applying these techniques to Azure VNet and subnet creation
Extending the playbook to include advanced resources like Azure Application Gateway

By adopting these practices, you can create more robust, flexible, and error-resistant network configurations in your Ansible playbooks. Whether you’re working with local environments or cloud providers like Azure, these techniques will help you manage network infrastructure more effectively. I would encourage you to experiment with these methods in your own projects. As you become more comfortable with dynamic network calculations, you’ll find that your infrastructure-as-code becomes more powerful and adaptable to changing requirements.

Audio Summary

Your browser does not support the audio element.

How to Install Ansible on a Mac: A Modern Approach

web.site@mckendrick.email (Russ McKendrick) — Sun, 15 Sep 2024 10:15:29 GMT

As the DevOps landscape evolves, so do our methods for managing tools like Ansible. This post outlines my current preferred approach to installing and managing Ansible on macOS.

The Traditional Homebrew Approach

For years, many macOS users, including myself, relied on Homebrew to install and maintain Ansible. The process was straightforward:

Installing Ansible using Homebrew

brew install ansible

While this method effectively managed the core Ansible installation, it became problematic as Ansible’s ecosystem grew more complex.

The Dependency Dilemma

As more collections moved away from the core Ansible distribution, managing prerequisites for various collections became increasingly challenging. Some collections had specific dependency requirements, leading to potential conflicts and a maintenance headache - especially when doing other Python development which have the same dependency requirements but you need to use other versions than those defined by the requirements.txt which ships with a collection.

A Modern Solution: Conda for Ansible Management

Having experienced the pitfalls of Python dependency management, I’ve transitioned to using Conda for Ansible. This approach allows for a dedicated, isolated Ansible environment, mitigating the risks of “dependency hell.”

Step-by-Step Installation Process

Create and activate a Conda environment for Ansible:

Setting up Ansible environment with Conda

conda create -n ansible python=3.12
conda activate ansible

Install Ansible within the environment:

Installing Ansible in Conda environment

python -m pip install ansible

Managing Ansible Collections

Once Ansible is installed, you can add collections and their dependencies using Ansible Galaxy and pip.

For example, to install the Azure Collection and its dependencies:

Adding the Azure collection

ansible-galaxy collection install azure.azcollection
python -m pip install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements.txt

Upgrading Ansible and Collections

Keeping Ansible and its collections up-to-date is crucial. Here’s how to do it:

Upgrade Ansible:

Upgrading Ansible

python -m pip install ansible --upgrade

Upgrade a specific collection (e.g., Azure):

Upgrading Azure Collection

ansible-galaxy collection install azure.azcollection --upgrade
python -m pip install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements.txt

Summary and Best Practices

Adopting Conda for Ansible management on macOS offers several advantages:

Isolation: Your Ansible setup exists in its own environment, preventing conflicts with other Python applications.
Flexibility: Easily manage different versions of Ansible for various projects.
Dependency Control: Better handle complex dependencies required by different Ansible collections.
Consistency: Ensure a consistent environment across different machines or team members.

To make the most of this setup:

Regularly update both Ansible and its collections to benefit from the latest features and security patches.
Explore Conda’s environment export↗ features to share your exact setup with team members.

By embracing this modern approach to Ansible management on macOS, you’ll find yourself with a more maintainable, flexible, and robust Ansible environment. Happy automating!

Audio Summary

Your browser does not support the audio element.

Unlocking Image Creation with Flux and GPT-4o

web.site@mckendrick.email (Russ McKendrick) — Sun, 11 Aug 2024 12:00:00 GMT

This week I decided to see how many buzzwords I could get into one post, given the recent release of Flux by Black Forest Labs I had originally intended on trying to get it up and running locally - however it quickly became clear that my M3 MacBook Pro with its 36GB of RAM wasn’t going to cut it. Because of this, I decided to look at one of the many online services that offer access to the various Flux models via their APIs which led me to Fal.AI which in turn gave me the idea for this post.

The Tools

Before we dive into the idea I had, let’s quickly get ourselves up to speed with the tools we’ll be using in this post. Each of these technologies plays a crucial role in our AI-powered image generation project.

Flux.1: The new kid on the block

A team of AI researchers and engineers, renowned for their work on foundational generative AI models like VQGAN and Latent Diffusion across academic, industrial, and open-source platforms, surprised everyone with the announcement of Black Forest Labs↗.

Alongside their Series Seed funding round of $31 million, they also revealed the immediate availability of three variations of their Flux.1 model:

FLUX.1 [pro]: The premier version of the FLUX.1 model, offering state-of-the-art image generation with exceptional prompt following, visual quality, image detail, and output diversity. FLUX.1 [pro] is accessible via API, and is also available through platforms such as Replicate and fal.ai. Additionally, it supports dedicated and customized enterprise solutions.
FLUX.1 [dev]: An open-weight, guidance-distilled model intended for non-commercial applications. Derived directly from FLUX.1 [pro], FLUX.1 [dev] provides similar quality and prompt adherence capabilities while being more efficient than standard models of the same size. The model’s weights are available top download from Hugging Face↗, and it can be executed on Replicate↗ and fal.ai↗.
FLUX.1 [schnell]: Designed as the fastest model in the lineup, FLUX.1 [schnell] is optimized for local development and personal use. It is freely available under an Apache 2.0 license, with weights hosted on Hugging Face. Inference code can be found on GitHub and in Hugging Face’s Diffusers, and the model is integrated with ComfyUI from day one.

For an idea of what you can produce have a look at the example images below, which were all generated using the tools we are going to be covering in this post:

As you can see, it can not only handle text extremely well, but it also produces high-quality photo-realistic images as well as some abstract ones.

fal.ai: Doing the heavy lifting

Also already mentioned, running even the small model locally was out of the question so I decided to look at one of the two original partners providing all three models in the Flux.1 family, I chose to focus on fal.ai↗ as I had used them previously to test another model earlier in the year.

Creating an image using cURL

export FAL_KEY="YOUR_API_KEY"
curl --request POST \
  --url https://fal.run/fal-ai/flux-pro \
  --header "Authorization: Key $FAL_KEY" \
  --header "Content-Type: application/json" \
  --data '{
     "prompt": "Extreme close-up of a single tiger eye, direct frontal view. Detailed iris and pupil. Sharp focus on eye texture and color. Natural lighting to capture authentic eye shine and depth. The word \"FLUX\" is painted over it in big, white brush strokes with visible texture."
   }'

This returned the following JSON response:

The results of the image (JSON)

{
    "images": [
        {
            "url": "https://fal.media/files/panda/oGSuf_FrMn7im_0DGnSdg_4378270c342f4170bb0f55e14cbb0636.jpg",
            "width": 1024,
            "height": 768,
            "content_type": "image/jpeg"
        }
    ],
    "timings": {},
    "seed": 932194184,
    "has_nsfw_concepts": [
        false
    ],
    "prompt": "Extreme close-up of a single tiger eye, direct frontal view. Detailed iris and pupil. Sharp focus on eye texture and color. Natural lighting to capture authentic eye shine and depth. The word \"FLUX\" is painted over it in big, white brush strokes with visible texture."
}

Opening the image in the response gave us the following:

This got me thinking - there is a Python library for fal.ai so I could use that, also what other tools have Python libraries I can use alongside fai.ai?

OpenAI: Building a Better Prompt

OpenAI↗ shouldn’t need an introduction, we will be using the GPT4o models↗ to help with prompting.

Streamlit: Bring it all together

Streamlit↗ is a Python framework from Snowflake↗ that quickly lets you build data and AI-driven web applications with a really low entry point, see the following for a quick overview:

There was just one problem with this, I am not a developer - so how could I do this?

Claude: Your personal development team

The final tool used in this post is Claude↗ from Anthropic↗, while I may not be a developer - I can prompt and debug code and since the release of Claude 3.5 Sonnet↗ I have been using it more and more for development tasks.

So let’s give it a go !!!

The Idea

The idea I had was to use Streamlit to build an interface to interact with the fal.ai API and the Flux family of models from my local machine - as part of this it should also use GPT4o to help tune the prompt to give us the best shot at getting a good result from Flux.

Running the application

Before we dive into the code, let’s just download and run the app.

Getting the keys

To run the application you will need two API keys, one for fal.ai and the other for the OpenAI API, the links below:

Make a note of the keys and keep them very safe.

Creating the environment

Next up we need to create an environment to run the application in and grab the code from GitHub↗.

Run the commands below to create the conda environment, switch to it, check the code out from GitHub↗ and install the required Python packages:

Preparing the environment

conda create -n streamlit python=3.12
conda activate streamlit
git clone https://github.com/russmckendrick/flux-fal-openai-streamlit.git
cd flux-fal-openai-streamlit
pip install -r requirements.txt

Launching the application

We are nearly ready to launch the application, just one more thing to do and that is the expose our API keys as an environment variable for our application to read:

Exporting the API keys

export FAL_KEY="<put-your-fal.ai-key-here>"
export OPENAI_API_KEY='<put-your-openai-key-here>'

With the keys in place we can launch the application by running the command below:

Exporting the API keys

streamlit run app.py

This will open the application window, which should look something like the following:

Using the application

Now we have the application open, let’s generate an image, first let’s write a basic prompt:

Just clicking “☁️ Generate image” at this point will give you an impressive-looking image without any tuning, below is the image which was generated using the basic prompt above:

Lets now get GPT-4o to expand on our basic prompt. First, click on “Use OpenAI for prompt tuning” and then select your chosen GPT-4o model from the drop-down, in this example, I will use the default of gpt-4o rather than gpt-4o-mini, then click on the “✏️ Tune Prompt” button.

This will send your original prompt to the model and not only return a new prompt, which in this run was:

You will also get if you click on “See explanation and suggestions” more details on why the prompt has been updated:

As well as more suggestions on how you could further improve the prompt:

If you are happy with the new prompt, click on ”✅ Accept Tuned Prompt” and then click the “☁️ Generate image” button, after a minute you will get your image back:

The screens below show how the process above looks within the application:

As you can see from the last picture, a copy of the image and a markdown summary have been saved to your local machine.

Code Highlights

Let’s take a closer look at some key parts of our application’s code. These snippets highlight the core functionality and demonstrate how we’re integrating the various tools we’ve discussed. By examining these, you’ll get a better understanding of how the different components work together to create our AI-powered image generation app.

tune_prompt_with_openai (PYTHON)

def tune_prompt_with_openai(prompt, model):
    openai_api_key = os.getenv("OPENAI_API_KEY")
    if not openai_api_key:
        raise ValueError("OPENAI_API_KEY environment variable is not set")

    client = openai.OpenAI(api_key=openai_api_key)
    response = client.chat.completions.create(
        model=model,
        messages=[
            {
                "role": "system",
                "content": "You are an advanced AI assistant specialized in refining and enhancing image generation prompts. Your goal is to help users create more effective, detailed, and creative prompts for high-quality images. Respond with: 1) An improved prompt (prefix with 'PROMPT:'), 2) Explanation of changes (prefix with 'EXPLANATION:'), and 3) Additional suggestions (prefix with 'SUGGESTIONS:'). Each section should be on a new line."
            },
            {
                "role": "user",
                "content": f"Improve this image generation prompt: {prompt}"
            }
        ]
    )
    return response.choices[0].message.content.strip()

As you can see from the system prompt below, we are instructing the model to not only generate the prompt but also everything else and how it should format its response:

This means we only make one call to the model and also we have sections we can identify when we receive the response.

As the images may take a little while to generate I had Claude add some messages to cycle through rather than us just looking at a boring spinner:

cycle_spinner_messages (PYTHON)

def cycle_spinner_messages():
    messages = [
        "🎨 Mixing colors...",
        "✨ Sprinkling creativity dust...",
        "🖌️ Applying artistic strokes...",
        "🌈 Infusing with vibrant hues...",
        "🔍 Focusing on details...",
        "🖼️ Framing the masterpiece...",
        "🌟 Adding that special touch...",
        "🎭 Bringing characters to life...",
        "🏙️ Building the scene...",
        "🌅 Setting the perfect mood...",
    ]
    return itertools.cycle(messages)

When I said Streamlit made creating web applications really easy, I wasn’t kidding. The code below generates the bulk of what you see on the initial screen:

main() (PYTHON)

def main():
    st.title("🤖 Image Generation with fal.ai & Flux")

    # Check for environment variables
    if not os.getenv("FAL_KEY"):
        st.error("FAL_KEY environment variable is not set. Please set it before running the app.")
        return

    # Model selection dropdown
    model_options = {
        "Flux Pro": "fal-ai/flux-pro",
        "Flux Dev": "fal-ai/flux/dev",
        "Flux Schnell": "fal-ai/flux/schnell",
        "Flux Realism": "fal-ai/flux-realism"
    }
    selected_model = st.selectbox("Select Model:", list(model_options.keys()), index=0)

    # Basic parameters
    image_size = st.selectbox("Image Size:", ["square_hd", "square", "portrait_4_3", "portrait_16_9", "landscape_4_3", "landscape_16_9"], index=0)
    num_inference_steps = st.slider("Number of Inference Steps:", min_value=1, max_value=50, value=28)

    # Advanced configuration in an expander
    with st.expander("Advanced Configuration", expanded=False):
        guidance_scale = st.slider("Guidance Scale:", min_value=1.0, max_value=20.0, value=3.5, step=0.1)
        safety_tolerance = st.selectbox("Safety Tolerance:", ["1", "2", "3", "4"], index=1)

    # Initialize session state
    if 'user_prompt' not in st.session_state:
        st.session_state.user_prompt = ""
    if 'tuned_prompt' not in st.session_state:
        st.session_state.tuned_prompt = ""
    if 'prompt_accepted' not in st.session_state:
        st.session_state.prompt_accepted = False

    # User input for the prompt
    user_prompt = st.text_input("Enter your image prompt:", value=st.session_state.user_prompt)

    # Update session state when user types in the input field
    if user_prompt != st.session_state.user_prompt:
        st.session_state.user_prompt = user_prompt
        st.session_state.prompt_accepted = False

    # OpenAI prompt tuning options
    use_openai_tuning = st.checkbox("Use OpenAI for prompt tuning", value=False)

    openai_model_options = ["gpt-4o", "gpt-4o-mini"]
    selected_openai_model = st.selectbox("Select OpenAI Model:", openai_model_options, index=0, disabled=not use_openai_tuning)

The drop-downs, text input boxes, sliders and the hidden advanced settings are all native Streamlit components which “just work”, also being all native it gives the application a nice consistent look when other things happen like:

Generating image (PYTHON)

        # Display the prompt being used
        st.subheader("☁️ Generating image with the following prompt:")
        st.info(user_prompt)

The rest of the code can be found on the application’s GitHub repo↗.

Some more images

Before wrapping up this post, let’s take some of the prompts I have used to generate previous blog post covers in Midjouney and run them through Flux - I will leave you to make your own mind up on which is better 😊.

Generating an Azure Storage Account SAS token using Azure Logic and Function apps

Midjourney …

Flux.1 [pro] …

Azure Firewall KQL Query

Midjourney …

Flux.1 [pro] …

Azure Virtual Desktop KQL Queries

Midjourney …

Flux.1 [pro] …

Azure DevOps Ansible Pipeline; Boosting Efficiency with Caching

Midjourney …

Flux.1 [pro] …

Day to Day Tools, the 2024 edition

Midjourney …

Flux.1 [pro] …

Conda for Python environment management on macOS

Midjourney …

Flux.1 [pro] …

GitHub - russmckendrick/discogs-scraper: A basic scraper for generating files for my website 🎸.

Conclusion

Now that you’ve seen the potential of combining Flux, fal.ai, OpenAI, and Streamlit, why not give it a try yourself? Clone the repository, set up your environment, and start experimenting with your own prompts. Whether you’re a developer looking to build on this framework or a creative professional curious about AI-assisted image generation, there’s plenty of room for exploration and innovation. Don’t forget to share your experiences or any cool images you generate – I’d love to see what you come up with!

Behind the Scenes; More than just a background

web.site@mckendrick.email (Russ McKendrick) — Sun, 04 Aug 2024 10:47:42 GMT

If you’ve ever joined me on a Teams call from my home office, you’ve likely noticed the wall of vinyl records behind me. This post aims to answer the most common questions I get on calls, offering a glimpse into my music collection and the technology used to play it.

The Collection

My vinyl collection is more than just a backdrop; let me give you a tour:

The Main Attraction

The wall you see behind me during video calls is just the beginning:

What you don’t see on calls

Adjacent to my desk, a second wall of records adds to the collection:

Box Set Bonanza

For those special editions and compilations, my bookshelves house an array of box sets:

Exploring the Collection Digitally

With over 2,827 items cataloged on Discogs↗; the collection caters to every mood and taste and I don’t find myself with nothing to listen to. A while ago I got bored during a winter weekend and using ChatGPT created a custom website that brings my collection to life digitally. Using data from Discogs, Wikipedia, Apple Music, and Spotify, https://www.russ.fm/↗ offers an interactive exploration of my vinyl collection.

Discover a Random Album

If I ever get stuck and can’t decide what to listen I added a feature that allows you virtually randomly grab an album from the shelves until you find something you want to listen to:

Click “Album Details” to learn more, or “Grab Another Album” to explore further. Hover over the cover for more information on the album if you can’t recognise from the sleeve cover!

The Tech Behind the Tunes

For those interested in the development side, here are links to the code that does the scrapping and serves the website:

A basic scraper for generating files for my website 🎸. - russmckendrick/discogs-scraper

GitHub - russmckendrick/records: This is the source code for russFM, a site dedicated to showcasing my personal record collection. 🎼🔥

This is the source code for russFM, a site dedicated to showcasing my personal record collection. 🎼🔥 - russmckendrick/records

Azure Blob Storage - Connectors

Don’t forget to check out my weekly listening habits in the “What have I been listening to?” section of this website.

The Audio Setup

To truly appreciate these records, a quality audio setup is crucial. Here’s what brings my vinyl to life:

Turntable: Denon DP-400↗.
Cartridge: Nagaoka MP-110↗.
Pre-amp: Cambridge Audio Alva Solo↗.
Speakers: Sonos Five (Stereo pair)↗ & Sonos Sub Gen3↗.
Record Cleaner: Pro-ject VC-S MKII↗.

This setup is complemented by additional Sonos speakers throughout the house, a system I’ve been building for about a decade.

Summary

The result of all this? It can get loud!!! 🔊

Generating an Azure Storage Account SAS token using Azure Logic and Function apps

web.site@mckendrick.email (Russ McKendrick) — Sat, 27 Jul 2024 17:03:45 GMT

Don’t you just love it when you open your laptop first thing on a Monday morning, a fresh cup of coffee at the ready and you think to yourself …

This should be a straight-forward five-minute job to start the week with

… and it turns out to be anything but?

That is exactly what happened to me this week, I won’t go into too much detail, but I was working on an Azure Logic App that had to interact with an external service and pass it a a fully signed SAS URL for an Azure Storage Account so that it could send some quite large files there. You would assume that the Azure Blob connector↗ would be able to do this, and you would be correct as there is a “Create SAS URI by path (V2)” action:

Learn-Ansible-Second-Edition/Chapter15/azure-pipelines.yml at main · PacktPublishing/Learn-Ansible-Second-Edition

This worked great in generating a SAS URL but when I tried to use it the service I was targeting returned the following message:

LocalFunctionProj/getSAS/run.ps1 (JSON)

{
  "code": "ExportFailed",
  "message": "The SAS token requires access to the 'Blob' service (ss=b)"
}

After a few hours of making tweaks, searching forums and community posts during and between calls I could not get it to work. After a few aborted Plan B’s which included trying to make all of the REST requests needed to generate the SAS token, that failed as I could not encode the connection string correctly using the built-in Logic App functions. I eventually settled on deploying an Azure Function App to run some Powershell which generates the SAS token and passes it back to the Logic App.

Launching the Function App

First of all, we need a Function App, for this, we will be deploying a consumption plan-based App with the Powershell runtime. You can use the commands below to deploy this using the Azure CLI↗.

First, we need to set some variables;

Set some variables

export RESOURCE_GROUP_NAME="rg-generate-sas-uks"
export REGION="uksouth"
export FUNCATION_APP_NAME="func-generate-sas-uks"
export FUNCATION_APP_STORAGE_ACCOUNT="safusasuks"

Now that we have some variables set we can add the Resource Group;

Add the resource group using the Azure CLI

az group create \
  --name $RESOURCE_GROUP_NAME \
  --location $REGION

With somewhere to place the resources next up we need to add the storage account which will be used by the Function App to store its files;

Deploy the Storage Account using the Azure CLI

az storage account create \
  --resource-group $RESOURCE_GROUP_NAME \
  --name $FUNCATION_APP_STORAGE_ACCOUNT \
  --kind StorageV2

Finally, we can launch the Azure Function App;

Deploy the Function App using the Azure CLI

az functionapp create \
  --resource-group $RESOURCE_GROUP_NAME \
  --name "$FUNCATION_APP_NAME" \
  --consumption-plan-location "$REGION" \
  --storage-account "$FUNCATION_APP_STORAGE_ACCOUNT" \
  --functions-version "4" \
  --runtime "powershell" \
  --runtime-version "7.4"

Deploying the Function

Now we have the Azure Function App deployed, our next task is to create and deploy our actual function. Being a macOS user I use func to create the function locally and then publish it, you can download the tool from here↗, once installed, run the following commands to create the local function:

Deploy to the Function App using the Azure CLI

func init LocalFunctionProj --worker-runtime powershell
cd LocalFunctionProj
func new --name getSAS --template "HTTP trigger"

This will create all of the files needed for the function, using your IDE of choice replace the contents of LocalFunctionProj/requirements.psd1 with the code below:

LocalFunctionProj/requirements.psd1 (POWERSHELL)

@{
    'Az' = '12.*'
}

Then the contents of LocalFunctionProj/getSAS/run.ps1 with:

LocalFunctionProj/getSAS/run.ps1 (POWESHELL)

# Import required namespaces
using namespace System.Net
using namespace Microsoft.WindowsAzure.Storage.Blob

# Input bindings are passed in via param block.
# $Request: Contains the request data
# $TriggerMetadata: Contains metadata about the trigger
param($Request, $TriggerMetadata)

# Log the request body for debugging purposes
Write-Host "Request body type: $($Request.Body.GetType().Name)"
Write-Host "Request body content: $($Request.Body | ConvertTo-Json -Depth 10)"

# The body is already deserialized, so we can use it directly
$requestBody = $Request.Body

# Check if we received the expected data (StorageAccountName and StorageAccountKey)
if ($requestBody -and $requestBody.ContainsKey('StorageAccountName') -and $requestBody.ContainsKey('StorageAccountKey')) {
    # Extract the storage account details from the request body
    $storageAccountName = $requestBody.StorageAccountName
    $storageAccountKey = $requestBody.StorageAccountKey

    # Generate the SAS token
    try {
        # Create a new Azure Storage context using the provided credentials
        $ctx = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey

        # Generate a new SAS token with the following parameters:
        # - Service: Blob
        # - ResourceType: Service, Container, Object
        # - Permission: Create, Read, Write, Delete, List
        # - ExpiryTime: 2 days from now
        $sasToken = New-AzStorageAccountSASToken -Service Blob -ResourceType Service,Container,Object -Permission "crwdl" -ExpiryTime (Get-Date).AddDays(2) -Context $ctx

        # Prepare the success response
        $responseBody = @{
            "message" = "SAS Token generated successfully"
            "sasToken" = $sasToken
        } | ConvertTo-Json
        Write-Host "SAS Token generated successfully"
    } catch {
        # If an error occurs, prepare an error response
        $responseBody = @{
            "error" = "Failed to generate SAS Token"
            "details" = $_.Exception.Message
        } | ConvertTo-Json
        Write-Host "Failed to generate SAS Token: $_"
    }
}
else {
    # If the required data is missing or invalid, prepare an error response
    $responseBody = @{
        "error" = "Invalid or missing JSON data"
        "receivedData" = $requestBody
    } | ConvertTo-Json
    Write-Host "Invalid or missing JSON data"
}

# Associate values to output bindings by calling 'Push-OutputBinding'.
# This sends the HTTP response back to the client.
Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
    StatusCode = [HttpStatusCode]::OK
    Body = $responseBody
    Headers = @{'Content-Type' = 'application/json'}
})

As you can see, the code above is commented which should give you a good gist of what happening but basically, we post a JSON file, the body of which looks like the following:

The JSON body (JSON)

{
  "StorageAccountName": "<name-of-your-target-storage-account>",
  "StorageAccountKey": "<the-primary-key-of-the-storage-account-above>"
}

It then uses the New-AzStorageAccountSASToken cmdlet to generate the SAS token. With the updates to the files in place, we can publish the local function to Azure by running the following code:

Deploy to the Function App using the Azure CLI

func azure functionapp publish $FUNCATION_APP_NAME

Once deployed, you can test if it is working as expected using cURL↗ locally.

Testing the Azure Function App

First, let’s re-export our variables adding one that contains the name of the storage account you want to target:

Set some variables

export RESOURCE_GROUP_NAME="rg-generate-sas-uks"
export REGION="uksouth"
export FUNCATION_APP_NAME="func-generate-sas-uks"
export FUNCATION_APP_STORAGE_ACCOUNT="safusasuks"
export TARGRET_STORAGE_ACCOUNT_NAME="sabackupuks"

We can now use the Azure CLI to grab the primary storage key and use that to populate our JSON payload by running the command below:

Set some variables

json_data=$(cat <<EOF
{
  "StorageAccountName": "$TARGRET_STORAGE_ACCOUNT_NAME",
  "StorageAccountKey": "$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $TARGRET_STORAGE_ACCOUNT_NAME --query '[0].value' -o tsv)"
}
EOF
)

With our JSON body populated, we can now use the Azure CLI to grab the details on the function we have just deployed (i.e. the URL and access key) and then post to JSON body to it by running:

Set some variables

curl -X POST \
  -H "Content-Type: application/json" \
  -d "$json_data" \
  "$(az functionapp function show --resource-group $RESOURCE_GROUP_NAME --name "$FUNCATION_APP_NAME" --function-name getSAS --query "invokeUrlTemplate" --output tsv)?code=$(az functionapp function keys list --resource-group $RESOURCE_GROUP_NAME --name "$FUNCATION_APP_NAME" --function-name getSAS --query "default" --output tsv)"

It will take a few seconds to return something, but if everything goes as planned you should see something like the following response:

The JSON result (JSON)

{
  "message": "SAS Token generated successfully",
  "sasToken": "sv=2023-08-03&ss=b&srt=sco&se=2024-07-30T08%3A12%3A06Z&sp=rwdlc&sig=7RsOYC6nEHCT9NaHm4t34RNqSLrt4GRiefZclzkHWgc%3D"
}

As you can see, this worked as expected and returned the signed token which is valid for 48 hours.

Adding the Function to your Logic App

I am not going to into details of the Logic App here, suffice to say that it needed to be integrated with an Azure Key Vault to store the primary Storage Account key and several other secrets for the service I was interacting with via HTTP requests. The Azure Function App was called using the Function workflow which is detailed here↗. There was a little bit of messing about with the results and getting the right information to construct the URL so here those details:

Getting the information into the LogicApp (JSON)

"Parse_JSON_(SAS)": {
    "inputs": {
        "content": "@outputs('Generate_SAS_Token')",
        "schema": {
            "properties": {
                "body": {
                    "properties": {
                        "message": {
                            "type": "string"
                        },
                        "sasToken": {
                            "type": "string"
                        }
                    },
                    "type": "object"
                },
                "headers": {
                    "properties": {
                        "Content-Length": {
                            "type": "string"
                        },
                        "Content-Type": {
                            "type": "string"
                        },
                        "Date": {
                            "type": "string"
                        },
                        "Request-Context": {
                            "type": "string"
                        },
                        "Transfer-Encoding": {
                            "type": "string"
                        },
                        "Vary": {
                            "type": "string"
                        }
                    },
                    "type": "object"
                },
                "statusCode": {
                    "type": "integer"
                }
            },
            "type": "object"
        }
    },
    "runAfter": {
        "Generate_SAS_Token": [
            "Succeeded"
        ]
    },
    "type": "ParseJson"
},

Here is the string I used to generate the fully signed URL:

Calling the third-part API (JSON)

"Trigger_the_external_API": {
    "inputs": {
        "body": {
            "blob": "@{variables('dateTime')}.somefiletype",
            "container": "@{parameters('saBlobContainer')}",
            "storageAccountSasUri": "https://@&#123;parameters('saAccountName')&#125;.blob.core.windows.net/?@&#123;body('Parse_JSON_(SAS)')?['body']?['sasToken']&#125;"
        },
        "headers": {
            "Authorization": "Bearer @{body('Parse_JSON_(AuthToken)')?['access_token']}",
            "Content-Type": "application/json"
        },
        "method": "POST",
        "uri": "<some URL for an external endpoint>"
    },
    "runAfter": {
        "Parse_JSON_(SAS)": [
            "Succeeded"
        ]
    },
},

Conclusion

What started as a “straightforward five-minute job” turned into an insightful journey through Azure’s intricacies. While the Azure Blob connector’s built-in “Create SAS URI by path (V2)” action seemed promising initially, it fell short of providing the specific SAS token required by the external service. This challenge led us to explore alternative solutions, ultimately landing on a robust and flexible approach using an Azure Function App.

By leveraging PowerShell within an Azure Function, we created a custom solution that generates the exact SAS token needed, complete with the required ‘Blob’ service access. This approach not only solved our immediate problem but also provided several advantages:

Flexibility: The function can be easily modified to accommodate different SAS token requirements in the future.
Security: By using Azure Key Vault in conjunction with the Logic App, we maintained secure storage and access to sensitive information like storage account keys.
Reusability: The function can be called from various Logic Apps or other Azure services, making it a versatile tool in your Azure toolkit.
Scalability: Being serverless, the Function App can handle varying loads without manual intervention.

This solution demonstrates the power of combining different Azure services to overcome limitations and create tailored solutions. Remember, when working with Azure or any complex system, what seems like a simple task can often lead to valuable learning experiences and innovative solutions. Embrace these challenges as opportunities to deepen your understanding and expand your Azure expertise.

As you implement this solution in your projects, consider documenting any additional hurdles you encounter and the solutions you devise. I hope you found this useful and also that appears high in your search results so don’t have to spend lots of frustrating hours scratching your head like I did.

Audio Summary

Your browser does not support the audio element.

Azure Firewall KQL Query

web.site@mckendrick.email (Russ McKendrick) — Sat, 20 Jul 2024 12:42:01 GMT

This is a quick post to share a useful KQL query for Azure Firewalls that I’ve been using. It’s based on the default Firewall query from Microsoft, but with some added variables to allow for a more focused analysis of source and destination traffic. I’m putting it here for my future reference and in case it’s helpful for anyone else.

Azure Firewall Query with Variables

This query allows you to analyze Azure Firewall logs, focusing on specific source and destination IP addresses within a defined time range. It’s particularly useful for troubleshooting and security analysis.

The Query (KQL)

let timeAgo = 15m; // how far back to look in the logs
let sourceIP = "192.168.0.1"; // the source IP address
let targetIP = "10.0.0.1"; // the desination IP address
let theLimit = 100; // number of results to return
AzureDiagnostics
| where Category == "AzureFirewallNetworkRule" or Category == "AzureFirewallApplicationRule"
| extend msg_original = msg_s
| extend msg_s = replace(@'. Action: Deny. Reason: SNI TLS extension was missing.', @' to no_data:no_data. Action: Deny. Rule Collection: default behavior. Rule: SNI TLS extension missing', msg_s)
| extend msg_s = replace(@'No rule matched. Proceeding with default action', @'Rule Collection: default behavior. Rule: no rule matched', msg_s)
| parse msg_s with * " Web Category: " WebCategory
| extend msg_s = replace(@'(. Web Category:).*', '', msg_s)
| parse msg_s with * ". Rule Collection: " RuleCollection ". Rule: " Rule
| extend msg_s = replace(@'(. Rule Collection:).*', '', msg_s)
| parse msg_s with * ". Rule Collection Group: " RuleCollectionGroup
| extend msg_s = replace(@'(. Rule Collection Group:).*', '', msg_s)
| parse msg_s with * ". Policy: " Policy
| extend msg_s = replace(@'(. Policy:).*', '', msg_s)
| parse msg_s with * ". Signature: " IDSSignatureIDInt ". IDS: " IDSSignatureDescription ". Priority: " IDSPriorityInt ". Classification: " IDSClassification
| extend msg_s = replace(@'(. Signature:).*', '', msg_s)
| parse msg_s with * " was DNAT'ed to " NatDestination
| extend msg_s = replace(@"( was DNAT'ed to ).*", ". Action: DNAT", msg_s)
| parse msg_s with * ". ThreatIntel: " ThreatIntel
| extend msg_s = replace(@'(. ThreatIntel:).*', '', msg_s)
| extend URL = extract(@"(Url: )(.*)(\. Action)", 2, msg_s)
| extend msg_s=replace(@"(Url: .*)(Action)", @"\2", msg_s)
| parse msg_s with Protocol " request from " SourceIP " to " Target ". Action: " Action
| extend
    SourceIP = iif(SourceIP contains ":", strcat_array(split(SourceIP, ":", 0), ""), SourceIP),
    SourcePort = iif(SourceIP contains ":", strcat_array(split(SourceIP, ":", 1), ""), ""),
    Target = iif(Target contains ":", strcat_array(split(Target, ":", 0), ""), Target),
    TargetPort = iif(SourceIP contains ":", strcat_array(split(Target, ":", 1), ""), ""),
    Action = iif(Action contains ".", strcat_array(split(Action, ".", 0), ""), Action),
    Policy = case(RuleCollection contains ":", split(RuleCollection, ":")[0], Policy),
    RuleCollectionGroup = case(RuleCollection contains ":", split(RuleCollection, ":")[1], RuleCollectionGroup),
    RuleCollection = case(RuleCollection contains ":", split(RuleCollection, ":")[2], RuleCollection),
    IDSSignatureID = tostring(IDSSignatureIDInt),
    IDSPriority = tostring(IDSPriorityInt)
| project
    msg_original,
    TimeGenerated,
    Protocol,
    SourceIP,
    SourcePort,
    Target,
    TargetPort,
    URL,
    Action,
    NatDestination,
    OperationName,
    ThreatIntel,
    IDSSignatureID,
    IDSSignatureDescription,
    IDSPriority,
    IDSClassification,
    Policy,
    RuleCollectionGroup,
    RuleCollection,
    Rule,
    WebCategory
| order by TimeGenerated
| where TimeGenerated > ago(timeAgo)
| where Target == targetIP and SourceIP == sourceIP
| limit theLimit

Key Features of the Query

Variable Declarations: The query starts with variables for time range, source IP, target IP, and result limit. This makes it easy to customize without changing the main query logic.
Log Filtering: It focuses on Azure Firewall Network and Application rules.
Data Parsing: The query uses a series of parse and extend operations to extract structured information from the log messages.
Field Extraction: Important fields like Protocol, Source/Target IP and Port, URL, Action, and various rule details are extracted.
Custom Filtering: The query filters results based on the specified source and target IP addresses.

How to Use This Query

Adjust the timeAgo, sourceIP, targetIP, and theLimit variables at the beginning of the query to suit your needs.
Run the query in your Azure Log Analytics workspace.
The results will show you detailed information about firewall actions, including source and destination details, protocols used, and the specific rules applied.

Wrapping Up

This query has been a handy tool for me when working with Azure Firewall logs. It’s particularly useful for:

Troubleshooting connectivity issues between specific hosts
Investigating potential security incidents
Auditing Firewall Rule Effectiveness
Monitoring traffic patterns

Remember, this is just a starting point. Feel free to modify the query to better fit your specific needs. As you work more with Azure Firewall and KQL, you’ll probably come up with your own variations that work even better for your environment.

A few final thoughts:

Keep an eye on performance, especially if you’re looking at a large time range or have a busy environment.
Always test your queries, especially after making modifications.
Don’t forget to update the placeholders (like sourceIP and targetIP) with actual values when you use this query.

Hopefully, this query will save future me, (👋 Hello again future Russ, and maybe you sometime when digging into Azure Firewall logs.

Happy querying!

Azure Virtual Desktop KQL Queries

web.site@mckendrick.email (Russ McKendrick) — Sun, 07 Jul 2024 14:52:12 GMT

This is a quick post collating some of the notes for KQL queries for Azure Virtual Desktop I have in various documents into a single place, just in case they are helpful for anyone and also for my reference, 👋 hello future Russ.

Find out all the IP addresses in the last 30 days

This query looks at the past 30 days, counting connections by username and client IP address, then sorts these counts in descending order, adds geolocation data for each IP, and finally outputs the client IP, connection count, and location details (country, state, and city) to provide insights into connection patterns and frequencies across different users and locations in the Windows Virtual Desktop environment.

Find out all the IP addresses in the last 30 days (KQL)

let daysAgo = 30d;
WVDConnections
| where TimeGenerated > ago(daysAgo)
| summarize NumberOfConnections = count() by UserName, ClientIPAddress
| order by NumberOfConnections desc
| extend ip_location = parse_json(geo_info_from_ip_address(ClientIPAddress))
| extend
    Country = tostring(ip_location.country),
    State = tostring(ip_location.state),
    City = tostring(ip_location.city)
| project ClientIPAddress, NumberOfConnections, Country, State, City

Find out all Users and their IP addresses in the last 30 days

This query extends the previous query and analyses AVD connections over the last 30 days, counting connections by username and client IP address, sorting these counts in descending order, enriching the data with geo-location information for each IP, and finally outputting the username, client IP, connection count, and location details (country, state, and city). This provides a comprehensive view of the Azure Virtual Desktop environment’s connection patterns, frequencies, and geographical distribution for each user.

Find out all Users and their IP addresses in the last 30 days (KQL)

let daysAgo = 30d;
WVDConnections
| where TimeGenerated > ago(daysAgo)
| summarize NumberOfConnections = count() by UserName, ClientIPAddress
| order by NumberOfConnections desc
| extend ip_location = parse_json(geo_info_from_ip_address(ClientIPAddress))
| extend
  Country = tostring(ip_location.country),
  State = tostring(ip_location.state),
  City = tostring(ip_location.city)
| project UserName, ClientIPAddress, NumberOfConnections, Country, State, City

Find out the IP addresses of where a user is connecting from

This KQL query is similar to the previous ones but introduces a new variable and focuses on filtering by username. Here’s a summary: The query analyses WVDConnections over the past 30 days, filters for usernames containing a specified search string (currently empty), and counts connections by client IP address. It orders these counts in descending order, adds geolocation data for each IP, and outputs the client IP, connection count, and location details (country, state, and city).

This query allows for targeted analysis of connection patterns for specific users or user groups, providing insights into the geographical distribution of their connections to the Windows Virtual Desktop environment.

Find out the IP addresses of where a user is connecting from (KQL)

let userSearch = "<replace with the UPN of a user>";
let daysAgo = 30d;
WVDConnections
| where TimeGenerated > ago(daysAgo)
| where UserName contains userSearch
| summarize NumberOfConnections = count() by ClientIPAddress
| order by NumberOfConnections desc
| extend ip_location = parse_json(geo_info_from_ip_address(ClientIPAddress))
| extend
  Country = tostring(ip_location.country),
  State = tostring(ip_location.state),
  City = tostring(ip_location.city)
| project ClientIPAddress, NumberOfConnections, Country, State, City

Get all errors for all users from a single IP address

The query first identifies users connected from a specified IP address (currently set to an empty string) within the last 30 days using the WVDConnections table. It then uses this list of users to filter the WVDErrors table, summarising the count of errors for each user and the error code. The results are ordered by error count in descending order. This query allows for targeted error analysis, linking connection data with error occurrences to help identify potential issues affecting users connecting from specific IP addresses in the Windows Virtual Desktop environment.

Get all errors for all users from a single IP address (KQL)

let ipAddress = "<replace with the IP address you are interested in>";
let daysAgo = 30d;
let users =
    WVDConnections
    | where TimeGenerated > ago(daysAgo)
    | where ClientSideIPAddress contains ipAddress
    | summarize by UserName;
WVDErrors
| where TimeGenerated > ago(daysAgo)
| where UserName in (users)
| summarize ErrorCount = count() by UserName, CodeSymbolic
| order by ErrorCount desc

Total session time

This KQL query analyses Azure Virtual Desktop connections over the past month, matching ‘Connected’ and ‘Completed’ session states to calculate total session durations for each user and connection type, then presents the results in both hours and days, sorted to show the most active users first, thereby providing insights into user engagement patterns and system usage within the WVD environment.

Total session time (KQL)

let daysAgo = 31d;
WVDConnections
| where TimeGenerated > ago(daysAgo)
| where State == "Connected"
| project CorrelationId, UserName, ConnectionType, StartTime=TimeGenerated
| join (WVDConnections
    | where State == "Completed"
    | project EndTime=TimeGenerated, CorrelationId)
    on CorrelationId
| extend SessionDuration = EndTime - StartTime
| summarize TotalDuration = sum(SessionDuration) by UserName, ConnectionType
| extend
    DurationHours = round(TotalDuration / 1h, 2),
    DurationDays = round(TotalDuration / 1d, 2)
| project UserName, ConnectionType, DurationHours, DurationDays
| sort by DurationHours desc

Total number of sessions per host pool

This KQL query analyses Windows Virtual Desktop connections from the past 31 days, focusing on successful connections to count distinct users per host pool, then formats the host pool names by combining specific segments of the resource ID, ultimately providing a concise summary of user diversity across different WVD host pools.

Total number of sessions per host pool (KQL)

let daysAgo = 31d;
WVDConnections
| where TimeGenerated > ago(daysAgo)
| where State == "Connected"
| project _ResourceId, UserName
| project-rename Hostpool = _ResourceId
| summarize DistinctUsers= dcount(UserName) by Hostpool
| extend HostPool=toupper(strcat(split(Hostpool, "/")[4], ".", split(Hostpool, "/")[8]))
| project HostPool, DistinctUsers

Total Sessions

This KQL query analyses Windows Virtual Desktop connections from the past day, focusing on completed sessions, and extracts key session information including timestamp, user, client details, connection type, and a reformatted host pool identifier, providing a detailed snapshot of recent WVD activity across different host pools with an emphasis on successfully concluded sessions.

Total Sessions (KQL)

let daysAgo = 1d;
WVDConnections
| where TimeGenerated > ago(daysAgo) and State contains "Completed"
| project-rename Hostpool = _ResourceId
| extend HostPool=toupper(strcat(split(Hostpool, "/")[4], ".", split(Hostpool, "/")[8]))
| project TimeGenerated, UserName, ClientOS, ClientType, ConnectionType, HostPool

Errors per host pool

This KQL query examines Azure Virtual Desktop errors from the past day, extracting the error code and reformatting the host pool identifier from the resource ID, then summarises the data by counting the occurrences of each error code per host pool, providing a concise overview of recent error patterns across different AVD environments.

Errors per host pool (KQL)

let daysAgo = 1d;
WVDErrors
| where TimeGenerated > ago(daysAgo)
| project _ResourceId, CodeSymbolic
| project-rename Hostpool = _ResourceId
| extend HostPool=toupper(strcat(split(Hostpool, "/")[4], ".", split(Hostpool, "/")[8]))
| summarize Count=count() by CodeSymbolic, HostPool

Wrapping Up

So there you have it - a handful of KQL queries I’ve found useful when working with Azure Virtual Desktop. These aren’t exhaustive by any means, just a collection of queries I’ve used and wanted to keep handy.

Remember, these queries are starting points. Feel free to tweak them to fit your specific needs. As you work more with AVD and KQL, you’ll probably come up with your own variations that work even better for your environment.

A few final thoughts:

Keep an eye on performance. Some of these queries look at 30 days of data, which might be heavy if you have a large environment.
Always test your queries, especially if you modify them.
Don’t forget to update the placeholders (like <replace with the IP address you are interested in>) with actual values when you use these queries.

Hopefully, these notes will save future me (and maybe you) time when digging into AVD logs. If you’ve got any cool AVD queries of your own, I’d love to hear about them in the comments.

Happy querying!

Azure DevOps Ansible Pipeline; Boosting Efficiency with Caching

web.site@mckendrick.email (Russ McKendrick) — Fri, 28 Jun 2024 13:24:15 GMT

Back in May 2020 I published what would be one of the more popular posts on this blog Azure DevOps Ansible Pipeline, it covered the Azure DevOps pipeline I had settled upon for running my Ansible playbooks which built and interacted with workloads running in Microsoft Azure.

Fast forward four years, I updated the pipeline for Chapter 15 of the second edition of Learn Ansible↗; this version of the pipeline, which is covered in detail in the book, can be found in the repo linked below;

Learn Ansible, Second Edition, published by Packt. Contribute to PacktPublishing/Learn-Ansible-Second-Edition development by creating an account on GitHub.

GitHub - russmckendrick/DevOpsAnsiblePipeline2024: Discover how to optimize your Azure DevOps Ansible pipeline with caching techniques. Learn to reduce execution time, improve efficiency, and maintain security checks in your infrastructure as code deployments.

After the book was published, it occurred to me that something was missing from the pipeline, and that is caching !!!

The Learn Ansible Pipeline

For those of you who haven’t read the second edition of Learn Ansible↗, this updated Azure DevOps pipeline automates the secure deployment of Ansible playbooks by first running a KICS↗ (Keeping Infrastructure as Code Secure) scan to identify potential security vulnerabilities in the infrastructure code. It then parses the scan results to determine whether it’s safe to proceed with the deployment, setting a flag to either continue or halt the pipeline based on the severity of any issues found.

If the security checks pass, the pipeline sets up the necessary Azure credentials, installs the required Ansible collections and dependencies, and finally executes the Ansible playbook, capturing and publishing the deployment output for easy review.

The only downside was that it could take quite a while to execute. To circumvent this for another project I am working on, I have added a small level of caching to the pipeline code.

Introducing Caching

All of the work to introduce caching happens in the Run Ansible stage of the pipeline; the first of the changes happens just after the task, which sets up the Azure CLI and exposes the service principles credentials as environment variables.

Before we get there, though, a few additional variables are introduced at the top of the file. These are …

The Variables (YAML)

- name: PATH_COLLECTION
  value: /home/vsts/.ansible/collections/ansible_collections/azure/azcollection
- name: PATH_VENV
  value: $(Pipeline.Workspace)/venv
- name: CACHE_VERSION
  value: v001
- name: REQUIREMENTS_FILE_NAME
  value: requirements.txt

While they mainly deal with paths, the last one is important as the Azure Collection maintainers at Microsoft kindly changed the name of the requirements file we are using to install the Python modules we need for the pipeline to work. This changed the name of the requirements file, which broke all the builds I had that used it; thanks for that grrrrrrr 😠.

Also, as a bonus, should you ever need to update the cache, change the CACHE_VERSION variable, which will force a new cache to be created when the pipeline subsequently runs.

Set up the virtual environment.

This task configures a virtual environment; this is important as we will be taking a copy of the virtual environment for future runs of the pipelines:

Set up the virtual environment (YAML)

- task: Bash@3
  name: "setup_environment"
  displayName: "Setup Python Environment"
  inputs:
    targetType: "inline"
    script: |
      echo "##[group]Setup Python Virtual Environment"
          python3 -m venv $(PATH_VENV)
          source $(PATH_VENV)/bin/activate
          pip install --upgrade pip
      echo "##[endgroup]"

The next task is as per the original pipeline, which grabs a copy of the private and public SSH keys and then copies them to the host running the Azure DevOps agent.

Check if there is an active cache

Next up, we have the Cache@2 task; with this task, we both check to see if there is already an active cache and if there isn’t, at the end of the pipeline run, a cache will be created will save us time and network bandwidth by caching files between pipeline runs.

Check if there is an active cache (YAML)

- task: Cache@2
  name: "cache_venv"
  displayName: "Cache virtual environment"
  inputs:
    key: 'venv$(CACHE_VERSION) | "$(Agent.OS)" | $(PATH_COLLECTION)/$(REQUIREMENTS_FILE_NAME)'
    path: $(PATH_VENV)
    cacheHitVar: CACHE_RESTORED

The inputs needed for the task to run are;

Key: The key input is crucial for identifying the cache. It’s composed of:
- A CACHE_VERSION variable (allowing you to invalidate all caches by changing this version)
- The operating system of the agent (Agent.OS)
- The path to the requirements file (PATH_COLLECTION/REQUIREMENTS_FILE_NAME) This combination ensures that the cache is specific to the current environment and dependencies.
Path: The path input specifies what should be cached, in this case, the virtual environment directory (PATH_VENV).
Cache Hit Variable: The cacheHitVar is set to CACHE_RESTORED. This variable will be set to ‘true’ if the cache is successfully restored, allowing subsequent tasks to know whether they need to reinstall dependencies or can use the cached version.

Install the requirements if there is no active cache

This next task is responsible for installing dependencies when the cache isn’t available.

Install the requirements if there is no active cache (YAML)

- task: Bash@3
  name: "install_dependencies"
  displayName: "Install Dependencies"
  condition: ne(variables.CACHE_RESTORED, 'true')
  inputs:
    targetType: "inline"
    script: |
      echo "##[group]Install pip requirements"
          source $(PATH_VENV)/bin/activate
          pip install ansible[azure]
          ansible-galaxy collection install --force azure.azcollection
          pip install -r $(PATH_COLLECTION)/$(REQUIREMENTS_FILE_NAME)
          pip freeze > $(PATH_VENV)/$(REQUIREMENTS_FILE_NAME)
      echo "##[endgroup]"

Here’s an explanation of what’s happening:

Task Type: It’s using the Bash@3 task, which allows running Bash scripts in the pipeline.
Purpose: The task installs necessary dependencies, including Ansible and its Azure-related packages, as well as any other requirements specified in a requirements file.
Condition: The condition field specifies that this task only runs if CACHE_RESTORED is not ‘true’. This means it will skip the installation if the previous caching task successfully restored the virtual environment.
Script Content:
- It activates the virtual environment.
- Installs Ansible with Azure support using pip.
- Uses ansible-galaxy to install the Azure collection, forcing an update if it already exists.
- Installs additional requirements from a specified requirements file.
- Generates a new requirements file based on the current state of the virtual environment.
Output Grouping: The ##[group] and ##[endgroup] syntax is used to group the output in Azure DevOps, making the logs more readable.

This task ensures that all necessary dependencies are installed when the cache isn’t available or is outdated. By conditionally running this task, we are instructing the pipeline to use cached dependencies when possible, falling back to a fresh installation only when needed. This approach balances between speed (using cached dependencies) and reliability (ensuring up-to-date packages when necessary).

Run the Ansible Playbook.

This task is responsible for running the Ansible playbook and handling its output; there have been some changes to the task since the Learn Ansible version, these changes load the virtual environment and also confirm that the Azure modules are available:

Run the Ansible Playbook (YAML)

 - task: Bash@3
   name: "ansible"
   displayName: "Run Ansible"
   env:
     AZURE_CLIENT_ID: $(ARM_CLIENT_ID)
     AZURE_SECRET: $(ARM_CLIENT_SECRET)
     AZURE_TENANT: $(ARM_TENANT_ID)
     AZURE_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
     ANSIBLE_HOST_KEY_CHECKING: "False"
   inputs:
       targetType: "inline"
       script: |
         echo "##[group]Add SSH key"
             echo "$(SSH_PUBLIC_KEY)" > ~/.ssh/id_rsa.pub
             chmod 644 ~/.ssh/id_rsa.pub
         echo "##[endgroup]"
         echo "##[group]Verify Azure module installation"
             source $(PATH_VENV)/bin/activate
             python -c "import azure; print('Azure module is available')"
         echo "##[endgroup]"
         echo "##[group]Run the Ansible Playbook"
             ansible-playbook -i inv site.yml 2>&1 | tee $(System.DefaultWorkingDirectory)/ansible_output.log
         echo "##[endgroup]"
         echo "##[group]Create the markdown file for the Ansible Playbook Output"
             mkdir -p $(System.DefaultWorkingDirectory)/markdown
             echo "# Ansible Playbook Output" > $(System.DefaultWorkingDirectory)/markdown/summary.md
             echo "<details><summary>Click to expand</summary>" >> $(System.DefaultWorkingDirectory)/markdown/summary.md
             echo "" >> $(System.DefaultWorkingDirectory)/markdown/summary.md
             echo "\`\`\`" >> $(System.DefaultWorkingDirectory)/markdown/summary.md
             cat $(System.DefaultWorkingDirectory)/ansible_output.log >> $(System.DefaultWorkingDirectory)/markdown/summary.md
             echo "\`\`\`" >> $(System.DefaultWorkingDirectory)/markdown/summary.md
             echo "</details>" >> $(System.DefaultWorkingDirectory)/markdown/summary.md
         echo "##[endgroup]"

Let’s take a deeper dive; as you can see, it uses the Bash@3 task to execute a series of bash commands. The task begins by setting up Azure-related environment variables for authentication and disables Ansible host key checking for non-interactive execution.

The script content is divided into several sections. First, it sets up an SSH key by adding a public key to the pipeline environment and setting appropriate permissions. Next, it verifies the Azure module’s availability by activating the virtual environment and running a Python command to import the Azure module.

The core of this task is executing the Ansible playbook. It runs the site.yml playbook using a specified inventory file (inv) and captures the output in both the console and a log file. This step likely configures or manages Azure resources as defined in the playbook.

Following the playbook execution, the task creates a markdown summary of the Ansible output. It generates a markdown file with an expandable section containing the full Ansible output within code blocks. This summary allows quick access to the Ansible output directly in the Azure DevOps UI, particularly useful for troubleshooting or auditing pipeline runs.

Throughout the script, the task uses ##[group] and ##[endgroup] commands to organize the pipeline logs into clear, collapsible sections. This improves readability in the Azure DevOps interface.

Seeing it in action

As you can see from the screens below, the first run took nearly 6 minutes; however, the second run took nearly half the time and was just under 3 minutes now that the cache is in place:

Conclusion

The introduction of caching in our Azure DevOps Ansible pipeline has significantly improved its efficiency and reduced execution time. By implementing the Cache@2 task and conditionally installing dependencies, we’ve created a pipeline that balances speed and reliability. This optimization not only saves time but also reduces network bandwidth usage, making our DevOps processes more streamlined and cost-effective.

You can find the full code below:

Discover how to optimize your Azure DevOps Ansible pipeline with caching techniques. Learn to reduce execution time, improve efficiency, and maintain security checks in your infrastructure as code ...