Software engineer, technology enthusiast, and occasional writer. Exploring Rust, web development, and creative solutions to technical challenges. Zola 2026-06-02T00:00:00+00:00 https://clay.fyi/atom.xml 2026-06-02T00:00:00+00:00 2026-06-02T00:00:00+00:00 Unknown https://clay.fyi/blog/duckdb-is-all-you-need-for-opentelemetry-data/ <h4 id="is-duckdb-all-you-need-for-storing-and-querying-otel">Is duckdb all you need for storing and querying OTel?</h4> <p>In a <a href="/blog/cheap-opentelemetry-lakehouses-parquet-duckdb-iceberg/">previous post</a> about OpenTelemetry, parquet, and data lakehouses, I mentioned difficulties when you want to push lots of metrics, logs, or traces to object storage with an Iceberg catalog like Amazon S3 Tables. A couple people messaged me, shared expletives, and said the same.</p> <p>Some good news: there has been progress around this in the DuckDB ecosystem with the new <a href="https://ducklake.select/2026/04/13/ducklake-10/">DuckLake 1.0</a> standard, <a href="https://duckdb.org/2026/05/29/new-iceberg-features">Iceberg extension features</a>, and the <a href="https://github.com/smithclay/canardstack">Quack Protocol</a>. To explore some of these ideas in action, I updated the <a href="https://github.com/smithclay/duckdb-otlp">DuckDB OpenTelemetry extension</a> to write metrics, logs and traces via the OpenTelemetry Protocol (OTLP/HTTP) directly to duckdb. This makes it easy to store observability data in anything with existing duckdb support: lakehouses, normal and <a href="https://duckdb.org/community_extensions/extensions/gsheets">not-normal</a> filesystems, or <a href="https://duckdb.org/docs/current/core_extensions/aws">object storage</a>. Everything runs in a single process: no extra dependencies or orchestration needed.</p> <p>DuckDB is evolving into very capable data glue, even for streaming data. Below is some discussion of storing ~gigabytes to low terabytes of metrics, logs and traces streamed into a DuckLake using the new extension. Early benchmarks (see below) suggest if you're working with a modest amount of OpenTelemetry data, it's promising.</p> <h3 id="streaming-observability-data-into-ducklake-and-other-catalogs">Streaming observability data into DuckLake and other catalogs</h3> <p>The idea was to do the following with a couple lines of SQL statements typed into duckdb on a mac:</p> <ul> <li>Accept OpenTelemetry data streamed over HTTP (metrics, logs, traces)</li> <li>Write compressed OTel data to object storage (Amazon S3, Cloudflare R2, etc)</li> <li>Keep track of what data you write in a metadata catalog (DuckLake, Iceberg, etc)</li> <li>Query the data in SQL with duckdb remotely (Quack Protocol)</li> <li>Align to open standards/open source/vendor-neutral</li> </ul> <p>The main reason to do all of this is to get reasonable performance when you want to do SQL queries like "show errors last Monday for the checkout service." If not: <a href="https://smithclay.github.io/duckdb-otlp/guides/exporting-to-parquet/">just throw parquet files in a bucket</a>, which is <a href="/blog/cheap-opentelemetry-lakehouses-parquet-duckdb-iceberg/">what the last post partly explored</a>.</p> <p>For #1-5, all you do is <a href="https://smithclay.github.io/duckdb-otlp/quickstart/serve/">type a couple SQL commands</a> into the duckdb shell to load and configure a handful of extensions. That gets repetitive so I packaged it up in a docker image with duckdb embedded and pre-configured inside:</p> <pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">docker run --rm --name duckdb-otlp \ -p 4318:4318 \ -v &quot;$(pwd):&#x2F;data&quot; \ ghcr.io&#x2F;smithclay&#x2F;duckdb-otlp:latest </code></pre> <p>POST some OTel data to :4318, it gets added to local DuckLake, then you can query it with SQL (<code>SELECT * FROM otel_logs</code>).</p> <p>The trouble is what happens when you have hundreds of thousands of log lines arriving, which without optimizations means many small files being written to object storage. It kills query performance and costs more due to the intricacies of cloud object storage billing:</p> <figure> <a href="/images/ducklake-small-files-problem.png"> <img src="/images/ducklake-small-files-problem.png" alt="Diagram of the duckdb-otlp pipeline showing the small-file problem"> </a> <figcaption>The duckdb-otlp pipeline, and its catch.</figcaption> </figure> <p>Every telemetry pipeline that writes to object storage has ways to mitigate this by batching data to reduce the number of writes. This involves a tradeoff that varies widely depending on what you’re doing: how fresh do you need the data to be? You end up with configuration knobs that might dial the bill way down, but fresh data might not be queryable for a while.</p> <p>DuckLake is building in automatic-ish maintenance features and making it easy to use <a href="https://ducklake.select/docs/stable/duckdb/maintenance/checkpoint">via a single command</a> that can merge small files together (compaction) or reduce metadata. Iceberg catalogs like S3 Tables or R2 Data Catalog have had variations of this as well. If tuned correctly you might get reasonable query performance and a cheap object storage bill.</p> <p>Until then, after some basic synthetic log ingest benchmarks and the official <a href="https://duckdb.org/docs/current/guides/performance/benchmarks">duckdb benchmarks</a>: it seems promising. The duckdb-otlp extension was able to write ~100k log rows per second to remote cloud catalogs, which comes out to ~2.5 TB/day of data <em>for a single duckdb process</em>.</p> <table><thead><tr><th>Catalog</th><th>Durable rows/s</th><th>Accepted MiB/s</th><th>Drain Time</th></tr></thead><tbody> <tr><td><a href="https://smithclay.github.io/duckdb-otlp/guides/stream-to-local-ducklake/">Local DuckLake</a></td><td>97,545</td><td>29.9</td><td>0.21s</td></tr> <tr><td>Local DuckLake, R2 Storage</td><td>96,911</td><td>30.2</td><td>0.62s</td></tr> <tr><td><a href="https://smithclay.github.io/duckdb-otlp/guides/stream-to-remote-ducklake/">Neon DuckLake, R2 Storage</a></td><td>90,850</td><td>30.1</td><td>0.56s</td></tr> <tr><td><a href="https://smithclay.github.io/duckdb-otlp/guides/stream-to-s3-tables/">S3 Tables Catalog (Iceberg)</a></td><td>90,390</td><td>29.5</td><td>2.82s</td></tr> <tr><td><a href="https://smithclay.github.io/duckdb-otlp/guides/stream-to-r2-data-catalog/">R2 Data Catalog (Iceberg)</a></td><td>88,128</td><td>30.0</td><td>4.80s</td></tr> </tbody></table> <p>Yes, more benchmarking is needed. That's a future post.</p> <h3 id="grafana-goes-swimming-in-a-ducklake">Grafana goes swimming in a DuckLake</h3> <p>Performance aside, there's a major UX problem: open-source observability tools use their own special-purpose databases and query languages (PromQL, etc)... not SQL. Visualization tools like Grafana can be <a href="https://grafana.com/grafana/plugins/mssql/">extended to support SQL</a> via plugins, but it's not out of the box.</p> <p>There are other people working on this, recently learned about <a href="https://gigapipe.com/">Gigapipe</a> and vibe-weekend-coded a basic prototype <a href="https://github.com/smithclay/canardstack">here</a>. For simple queries in PromQL/Loki/Tempo, you can get reasonable-looking dashboards in Grafana, powered by SQL queries in a duckdb instance fronted by compatibility APIs:</p> <figure> <a href="/images/grafana-ducklake-dashboard.png"> <img src="/images/grafana-ducklake-dashboard.png" alt="Grafana dashboard rendering logs, traces, and service metrics from a DuckLake"> </a> <figcaption>A weekend-prototype Grafana dashboard, activity totals, service memory, recent logs, and frontend traces, rendered entirely from SQL queries against a DuckLake behind PromQL/Loki/Tempo-compatible APIs.</figcaption> </figure> <p>There's also the parallel universe of visualization and analytics tools in the analytics world that work well with duckdb. Perspective is doing really cool things inside of browsers with wasm, <a href="https://perspective-dev.github.io/block.html?example=duckdb">for example</a>.</p> <p>Quack is also opening up new integration pathways. As a standard, it's just an official and fast way for duckdbs to talk to each other over HTTP (credit to extensions like <a href="https://duckdb.org/community_extensions/extensions/airport">Airport</a> that have been exploring the idea for a while). <a href="https://smithclay.github.io/duckdb-otlp/guides/query-with-quack/">Currently in the duckdb-otlp extension, it's an easy way to connect to a remote DuckLake and query it</a>:</p> <pre data-lang="sql" class="language-sql "><code class="language-sql" data-lang="sql">ATTACH &#x27;quack:localhost:9494&#x27; AS otel_writer (TYPE quack); FROM otel_writer.query( $$ SELECT * FROM lake.main.otlp_logs WHERE service_name = &#x27;quack-local-ducklake-demo&#x27; ORDER BY time_unix_nano DESC LIMIT 5 $$ ); </code></pre> <p>What people start building with Quack and how that changes duckdb architecture will be interesting to watch. For example, there's a <a href="https://github.com/Query-farm/quackscale">new extension that runs Tailscale directly in duckdb</a>, making it easy to transfer data and queries across networks and firewalls. The tip of the iceberg (pun intended).</p> <h3 id="it-s-promising-but-early-more-data-needed">It's promising but early, more data needed</h3> <p>I think people are excited in the analytics space right now because there seems to be momentum in the direction of a vendor-neutral and cheap way to store and structured data, even if there's a lot of it.</p> <p>In many ways, if you consider <a href="https://clickhouse.com/docs/use-cases/observability/clickstack">ClickHouse</a>, <a href="https://quickwit.io/">Quickwit</a>, <a href="https://turbopuffer.com/">turbopuffer</a>, or <a href="https://www.langchain.com/blog/introducing-smithdb">LangChain's new tracing database</a> it technically already exists already minus the vendor-neutral and open standard part. The promise of Iceberg, DuckLake, or Delta Lake getting traction is that almost any major closed-source or open-source analytics tool or AI agent can query and read the data with SQL, even if it doesn't know what an OpenTelemetry trace is.</p> <p>DuckLake and the new quack protocol have some emerging capabilities that could be compelling for ops teams stuck between running their own complex observability infrastructure (Kafka-esque) and paying huge monthly bills to vendors (canine-esque). More data and better benchmarks to validate that and understand how far you can scale up.</p> <p>I hope this post gets people excited about working on open-source at the intersection of observability and analytics, which has the vibe right now of "something is happening". If you aggree, say hi on <a href="https://x.com/duckdb/status/1471806748814106624?lang=en">duckdb discord</a> or <a href="https://opentelemetry.io/community/end-user/#slack">OpenTelemetry Slack</a> or the other channels.</p> <p>PRs, feedback and benchmark ideas for <a href="https://github.com/smithclay/duckdb-otlp">duckdb-otlp</a> are welcome.</p> <p>Thanks for reading about my duck-pilled open-source project.</p> 2026-01-27T00:00:00+00:00 2026-01-27T00:00:00+00:00 Unknown https://clay.fyi/blog/build-your-own-claude-code-cloud-swarm/ <p>Personal background agents in dangerous mode are a few hundred lines of code away with the 2.0.19 release of Claude Code. This matters because running 4+ coding agents at a time, <a href="https://www.theregister.com/2026/01/22/cursor_ai_wrote_a_browser/">including fully autonomous agents that run for multiple days</a>, is quickly becoming a reality.</p> <p>This post is about some emerging patterns when you want to connect Claude Code sessions together in multiple cloud VMs <a href="https://steve-yegge.medium.com/welcome-to-gas-town-4f25ee16dd04">without going full-on Gastown</a>. The urgency behind all of this is related to <a href="https://medium.com/@peltomakiw/how-a-single-email-turned-my-clawdbot-into-a-data-leak-1058792e783a">security and prompt injection attacks when running locally</a>. As <a href="https://simonwillison.net/2025/Oct/22/living-dangerously-with-claude/">Simon Willison said in October 2025</a>:</p> <blockquote> <p>On the one hand I'm getting enormous value from running coding agents with as few restrictions as possible. On the other hand I'm <a href="https://www.ibm.com/think/topics/prompt-injection">deeply concerned by the risks</a> that accompany that freedom.</p> </blockquote> <p>The (imperfect) technical solution is a sandbox—ideally one that's not on your computer or network. This is already built into the web interfaces of Claude Code for Web and ChatGPT, but let's make three more assumptions:</p> <ol> <li>You're a developer using a coding agent on a command line.</li> <li>You're starting to run 4-10 coding sessions at a time, and it's getting chaotic.</li> <li>You want a way to centrally manage and monitor those sessions.</li> </ol> <p>In other words, if you're adopting sandboxes: you now get to manage a small fleet of cloud VMs or try to use some sort of container orchestration thing. This is a nightmare for non-experts using the major cloud providers as of early 2026, and even Opus 4.5 struggles with AWS IAM policies and the Azure CLI.</p> <p>Enter a new category of "sandbox-as-a-service" products from <a href="https://exe.dev/">exe.dev</a>, <a href="https://sprites.dev">sprites.dev</a> (fly.io), <a href="https://modal.com/">Modal</a>, <a href="https://developers.cloudflare.com/sandbox/">Cloudflare Sandbox SDK</a>, and <a href="https://e2b.dev/">E2B</a>. You get a remote, fully stateful server that spins up in a few seconds: a perfect and isolated home for Claude Code, Gemini, or Codex.</p> <p>My rough CLI implementation of doing this with sprites and exe.dev is at <a href="http://github.com/smithclay/ocaptain">github.com/smithclay/ocaptain</a>, but expect there to be 10,000 orchestrators created in the next few months before Claude Code or Codex releases something that obsoletes them.</p> <h3 id="get-a-full-linux-vm-in-a-couple-seconds-seriously">Get a full Linux VM in a couple seconds, seriously</h3> <p>With exe.dev or sprites.dev, the idea is simple. You use a CLI (or ssh session, a cool twist on how exe.dev does it), and type something like this:</p> <pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">$ ssh exe.dev new $ sprite create your-sprite </code></pre> <p>A few seconds later you get a full Linux VM you can do whatever you want with: run Docker images, run Claude Code, run Codex, install services, whatever. It won't go away. The pricing is reasonable (cost of a coffee and croissant in San Francisco per month) and it's fast.</p> <h3 id="living-dangerously-and-obscure-claude-code-token-settings">Living dangerously and obscure Claude Code token settings</h3> <p>Once you go multi-Claude Code, the immediate hurdle is logging into your subscription on multiple VMs or containers: a fresh start of Claude asks you to log in. If you're an API user it's easy to generate a new key, but for subscription users it's a different story.</p> <p>I lost four hours on a sunny and warm Saturday vibing elaborate token injection proxies before learning something critical: there's just a poorly-documented command to get an OAuth token tied to your subscription that lives for one year. Run <code>claude get-token</code>, and set that token using a special environment variable (<code>CLAUDE_CODE_OAUTH_TOKEN</code>).</p> <p>You can revoke or manage the token at <a href="https://claude.ai/settings/claude-code">https://claude.ai/settings/claude-code</a>. That's it.</p> <p>When launching an autonomous session in interactive mode, it's also helpful to disable the onboarding flow:</p> <pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">echo &#x27;{&quot;hasCompletedOnboarding&quot;:true}&#x27; &gt; ~&#x2F;.claude.json </code></pre> <p>Since you're in an isolated VM with none of your data, it's relatively okay to run Claude with <code>--dangerously-skip-permissions</code> – that means no prompts will pop up for permissions.</p> <p>It's also a good idea to set up some outbound firewall rules on the VM while you're at it to specific domains only.</p> <h3 id="task-orchestration-without-external-dependencies">Task orchestration without external dependencies</h3> <p>Before Friday, January 24—the release of Claude Code 2.1.19—task orchestration across sessions was largely a roll-your-own sort of thing, the most popular solution being Steve Yegge's <a href="https://github.com/steveyegge/beads">beads</a>. But now, with an <a href="https://x.com/trq212/status/2014480496013803643?s=20">upgrade to the task management system in Claude</a>, it's straightforward to generate and distribute tasks, dependencies between those tasks (e.g. write API, then write API integration tests) and status across sessions.</p> <p>All you need are some JSON files that you can generate yourself <a href="https://github.com/smithclay/ocaptain/blob/main/.claude/skills/voyage-plan/SKILL.md">or with a skill</a>, Claude Code handles the rest. Each instance of Claude Code gets passed a special environment variable called <code>CLAUDE_CODE_TASK_LIST_ID</code> that identifies a shared task list that lives in a collection of JSON files in <code>~/.claude/tasks/&lt;task id&gt;</code>. After the session starts, the task list pops up and updates automatically in each session: just ask Claude to pick up an item that needs work and that isn't blocked. You can set up a <a href="https://code.claude.com/docs/en/hooks">hook</a> that fires to notify you when all sessions end and there are no tasks left.</p> <p>Here's what it looks like, note the task list below that has been picked up by multiple "ships" (separate VMs, my orchestration CLI has a nautical theme). You can attach via tmux to the live Claude Code session at any time to see what's going on, but probably won't:</p> <p><a href="/images/ship-task-coordination.png"><img src="/images/ship-task-coordination.png" alt="ohcaptain task orchestration" /></a></p> <p>In terms of sharing the <code>~/.claude/tasks</code> directory across VMs, I initially used <a href="https://github.com/libfuse/sshfs">sshfs</a> to mount a shared directory across my exe.dev machines, but many approaches are possible: it's just a regular Linux VM so choose your favorite network filesystem. Claude Code handles the file locks between sessions. For VMs without SSH access, creating your own Tailscale <a href="https://tailscale.com/kb/1136/tailnet">tailnet</a> is also compelling.</p> <p>There are some sharp edges, but initial testing for semi-complex tasks is promising. The capabilities around this in Claude Code are likely to grow, <a href="https://news.ycombinator.com/item?id=46743908">as recent hidden feature flag investigations</a> have shown.</p> <h3 id="trust-but-verify-with-logs-and-metrics-from-sessions">Trust but verify with logs and metrics from sessions</h3> <p>When you're scaling up to even a modest number of VMs, what's going on inside those sessions is a quasi-black box: you're no longer babysitting, approving permissions, or seeing tool calls go by. All you see is the end result of the sessions. Some level of logging and analytics is needed, and fortunately Claude Code and Codex <a href="https://code.claude.com/docs/en/monitoring-usage">can output relevant events and metrics in OpenTelemetry format</a>.</p> <p>Nowadays, it's easy enough to <a href="https://github.com/smithclay/otlp2parquet">store those logs and metrics in parquet format</a> on disk or in cheap object storage (see the <a href="https://github.com/smithclay/otlp2parquet">otlp2parquet</a> or <a href="https://github.com/smithclay/otlp2pipeline">otlp2pipeline</a> projects). Several weeks of heavy sessions for a single user take less than 100 MB of storage. It's a low enough volume you can easily query and do some meaningful analytics on it with duckdb on a laptop or even in a web browser. Potential queries include: understanding cost per session, token count/cached token count over time, tool call errors, tool calls to external domains, etc.</p> <p>The OTel-to-parquet/Iceberg workflow works nicely with AI-assisted notebook tools like <a href="https://sidequery.dev/">Sidequery</a> or <a href="https://marimo.io/">Marimo</a>: you don't even have to write the SQL or pandas code and you get some nice visualizations.</p> <h3 id="what-we-wanted-serverless-to-be-in-the-late-2010s">What we wanted serverless to be in the late 2010s</h3> <p>Looking back: this is probably what we always wanted serverless compute to be in the first place… just took us a while to get here.</p> <p><a href="https://clay.fyi/blog/ssh-ing-into-aws-lambda-functions/">In 2017, I wrote a small Go program</a> so I could ssh into an AWS Lambda function as a joke for a conference talk: back then, there were—and still are—many people who didn't like functions-as-a-service architecture ("it's still a server!", "containers are better", "the cost model doesn't make sense", etc).</p> <p>This new wave of infrastructure providers settles the architecture debate: you get all the benefits of a serverless function (fast, ephemeral, cheap) but it's <em>also</em> a full VM. This pattern is also more relevant than ever with millions of developers starting to use coding agents in more sophisticated ways that require remote and secure sandboxes.</p> <p>It's a fun time to build stuff using cloud sandbox VMs, and it's likely to evolve a lot in the next 6 months. Just don't run YOLO mode on your personal laptop.</p> <h3 id="links-and-acknowledgements">Links and acknowledgements</h3> <ul> <li>exe.dev docs: <a href="https://exe.dev/docs/what-is-exe">https://exe.dev/docs/what-is-exe</a></li> <li>Sprites docs: <a href="https://docs.sprites.dev/">https://docs.sprites.dev/</a></li> <li>My own CLI impementation of this post: <a href="https://github.com/smithclay/ocaptain">https://github.com/smithclay/ocaptain</a></li> <li>Simon's blog is great and he has been convering sandboxes extensively: <a href="https://simonwillison.net/2026/Jan/9/sprites-dev/">https://simonwillison.net/2026/Jan/9/sprites-dev/</a></li> <li>OTLP to parquet converter, useful for monitoring: <a href="https://github.com/smithclay/otlp2parquet">https://github.com/smithclay/otlp2parquet</a></li> </ul> 2025-12-02T00:00:00+00:00 2025-12-02T00:00:00+00:00 Unknown https://clay.fyi/blog/cheap-opentelemetry-lakehouses-parquet-duckdb-iceberg/ <h4 id="what-if-all-your-observability-data-was-just-parquet-files">What if all your observability data was just parquet files?</h4> <p>If you work in SRE, DevOps, or observability it might be time to introduce yourself to the data engineers and understand what's going on with lakehouses, columnar storage, and a <a href="https://projects.apache.org/projects.html?category#big-data">bewildering number of projects</a> from the Apache Foundation. Some of the most interesting developments in observability in late 2025 are coming from analytics.</p> <p>For people who have tuned out everything related to data warehouses since Hadoop:</p> <ol> <li>Columnar storage formats, <a href="https://arrow.apache.org/">Apache Arrow</a>, and <a href="https://github.com/open-telemetry/otel-arrow">emerging schemas for OpenTelemetry</a> are making it more realistic to do analytics on logs, metrics, and traces quickly and cheaply in object storage (S3, R2, <a href="https://www.min.io/">MinIO</a>, etc).</li> <li><a href="https://duckdb.org/">duckdb</a> lets you do analytics on all this data using regular SQL, including from <a href="https://smithclay.github.io/otlp2parquet/query-demo/">inside your browser</a> (feat. WebAssembly).</li> <li>Iceberg-based lakehouse<sup class="footnote-reference"><a href="#1">1</a></sup> architecture, now available via managed services like Cloudflare <a href="https://developers.cloudflare.com/r2/data-catalog/">R2 Data Catalog</a> and AWS <a href="https://aws.amazon.com/s3/features/tables/">S3 Tables</a><sup class="footnote-reference"><a href="#2">2</a></sup> that implement the <a href="https://iceberg.apache.org/">Apache Iceberg</a> spec, add convenient analytics, metadata and performance features on top of parquet files in object storage without needing to run a distributed compute cluster.</li> <li>The monitoring world is currently obsessed with AI agents and anomaly detection. By having OTel data in a lakehouse, Python/Pandas/PyTorch workloads run directly against Parquet without moving data.</li> </ol> <p>Observability isn't a typical analytics workload: lots of small, highly concurrent writes present many technical challenges that have traditionally been left to specialized vendors ($$$). This is at odds with the overall lakehouse philosophy: stop shuttling data through multiple storage tiers and put it once into a managed table format on object storage so tools read from a single source of truth.</p> <p>This post explores a prototype of how far you could get with cheap storage, <a href="https://github.com/smithclay/otlp2parquet">some glue code written in rust</a>, and open table formats for observability to understand if the big data ecosystem has arrived at the point where lakehouses could cost-effectively augment observability tools for longer-term analytics.</p> <div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup> <p>In this post, 'lakehouse' means what <a href="https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf">Armbrust et al. defined in their 2021 CIDR paper</a>: data on cheap object storage in open formats (Parquet), with a metadata layer that adds transactions, schema evolution, versioning, and query optimization.</p> </div> <div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup> <p><strong>12/3/2025 update:</strong> The day after this post was published, <a href="https://aws.amazon.com/blogs/aws/amazon-cloudwatch-introduces-unified-data-management-and-analytics-for-operations-security-and-compliance/">Amazon released an automatic integration for S3 Tables from CloudWatch Logs</a>. It makes all of the glue code below redundant for log sources in AWS.</p> </div> <h3 id="teaching-duckdb-to-speak-opentelemetry">Teaching duckdb to speak OpenTelemetry</h3> <p>A straightforward way to export OpenTelemetry data is to configure a collector with a <a href="https://opentelemetry.io/docs/specs/otel/protocol/file-exporter/">file exporter</a> and output a bunch of protobuf or JSON files to a local directory. See this <a href="https://www.otelbin.io/?#config=receivers%3A*N__otlp%3A*N____protocols%3A*N______grpc%3A*N________endpoint%3A_0.0.0.0%3A4317*N______http%3A*N________endpoint%3A_0.0.0.0%3A4318*N*Nprocessors%3A*N__batch%3A_%7B%7D*N*Nexporters%3A*N__file%2Fproto-logs%3A*N____path%3A_%2Fexport%2Fproto%2Flogs.proto*N____format%3A_proto*N____rotation%3A*N______max*_megabytes%3A_128*N______max*_days%3A_7*N______max*_backups%3A_100*N__file%2Fproto-metrics%3A*N____path%3A_%2Fexport%2Fproto%2Fmetrics.proto*N____format%3A_proto*N____rotation%3A*N______max*_megabytes%3A_128*N______max*_days%3A_7*N______max*_backups%3A_100*N__file%2Fproto-traces%3A*N____path%3A_%2Fexport%2Fproto%2Ftraces.proto*N____format%3A_proto*N____rotation%3A*N______max*_megabytes%3A_128*N______max*_days%3A_7*N______max*_backups%3A_100*N*Nextensions%3A*N__health*_check%3A*N__pprof%3A*N__zpages%3A*N*Nservice%3A*N__extensions%3A_%5Bhealth*_check%2C_pprof%2C_zpages%5D*N__pipelines%3A*N____traces%3A*N______receivers%3A_%5Botlp%5D*N______processors%3A_%5Bbatch%5D*N______exporters%3A_%5Bfile%2Fproto-traces%5D*N____metrics%3A*N______receivers%3A_%5Botlp%5D*N______processors%3A_%5Bbatch%5D*N______exporters%3A_%5Bfile%2Fproto-metrics%5D*N____logs%3A*N______receivers%3A_%5Botlp%5D*N______processors%3A_%5Bbatch%5D*N______exporters%3A_%5Bfile%2Fproto-logs%5D%7E&amp;distro=otelcol-contrib%7E&amp;distroVersion=v0.139.0%7E">example on OTelBin</a> using the latest collector version (v0.139.0).</p> <p>With JSON/protobuf files, querying this data remains neither easy nor fast with standard analytics tools. This is what <a href="https://duckdb.org/docs/stable/extensions/overview">duckdb extensions</a> were designed for: taking external data and converting it into rows you can easily query with SQL.</p> <p>I recently wrote and published a <a href="https://github.com/smithclay/duckdb-otlp">community extension</a> that imports telemetry data from JSON or protobuf files. It uses a schema inspired by the <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/clickhouseexporter">Clickhouse OpenTelemetry exporter.</a> Here's what it looks like in the duckdb CLI (v1.4.2 or later). duckdb makes reading multiple files or reading from HTTP/S3/cloud storage easy:</p> <pre data-lang="sql" class="language-sql "><code class="language-sql" data-lang="sql">INSTALL otlp FROM community; LOAD otlp; LOAD httpfs; -- Query slow traces SELECT TraceId, SpanName, Duration &#x2F; 1000000 AS duration_ms FROM read_otlp_traces(&#x27;https:&#x2F;&#x2F;raw.githubusercontent.com&#x2F;smithclay&#x2F;otlp2parquet&#x2F;refs&#x2F;heads&#x2F;main&#x2F;testdata&#x2F;traces.jsonl&#x27;) WHERE Duration &gt; 1000000000 -- over 1 second ORDER BY Duration DESC LIMIT 5; </code></pre> <p>This is enough to do some powerful analytics: it's trivial to join across signal types or correlate with non-observability data. You can also save any data (errors, slow traces, interesting logs) to parquet format with a SQL one-liner:</p> <pre data-lang="sql" class="language-sql "><code class="language-sql" data-lang="sql">-- Query slow traces, write to Parquet COPY ( SELECT TraceId, SpanName, Duration &#x2F; 1000000 AS duration_ms FROM read_otlp_traces( &#x27;https:&#x2F;&#x2F;raw.githubusercontent.com&#x2F;smithclay&#x2F;otlp2parquet&#x2F;refs&#x2F;heads&#x2F;main&#x2F;testdata&#x2F;traces.jsonl&#x27; ) WHERE Duration &gt; 1000000000 ORDER BY Duration DESC LIMIT 5 ) TO &#x27;slow_traces.parquet&#x27; (FORMAT PARQUET); </code></pre> <p>So that's cool. There are two significant problems:</p> <ol> <li><strong>Needs streaming support:</strong> OpenTelemetry data isn't usually accessible from JSON/protobuf files in production scenarios. It's streamed in real time from instrumented applications through various collectors and pipelines to specialized databases like Elastic or Tempo.</li> <li><strong>Wrong formats:</strong> Naively writing a lot of metrics, logs and traces to small JSON/protobuf files kills the price and performance benefits of a data lake: object storage gets more expensive and queries get slower.</li> </ol> <p>Next is what happened when I tried to convert OpenTelemetry to parquet in a Lambda function/Cloudflare worker and hoped for the best. Spoiler: object stores are great at cheap blobs, but less great at "thousands of tiny writes per second, forever."</p> <h3 id="streaming-opentelemetry-to-parquet-files">Streaming OpenTelemetry to Parquet files</h3> <p>No duckdb extension is needed if metrics, logs, and traces are stored in parquet instead of JSON or protobuf. The OTel schema, especially for metrics, is far from trivial, but there are emerging projects exploring what it could look like including <a href="https://github.com/open-telemetry/otel-arrow">OpenTelemetry Arrow</a> and <a href="https://www.rakirahman.me/otel-arrow-delta-lake/">OpenTelemetry on Delta Lake</a>. These take advantage of the columnar, semi-structured, highly repetitive and compression-friendly aspects of OTel data: you could see file size reduction of up to 90%.</p> <p>Thanks to the robust Arrow/rust/Apache ecosystem and Claude Code, I wrote a simple Rust library called <a href="https://github.com/smithclay/otlp2parquet"><code>otlp2parquet</code></a> that handles these boring parts: decoding OTLP, mapping it into Arrow, writing Parquet, and dealing with cloud storage. It's profoundly cheap in terms of compute costs: log-to-parquet conversion using demo data came out to about $0.01/per uncompressed GB. That's a rounding error on cloud bills and doesn't even account for free tiers on Workers or Lambda.</p> <p><a href="/images/otlp2parquet-architecture.png"><img src="/images/otlp2parquet-architecture.png" alt="otlp2parquet architecture diagram" /></a></p> <p>Parquet format means query engines benefit from only reading—and transferring over the wire—a significantly smaller amount of data when running a query like: "give me auth service logs in us-east-2 for the past hour." Only portions of the file are scanned.</p> <p>Here's what a simple logs query looks like now. For maximum fun, you can even run this completely in your browser <a href="https://smithclay.github.io/otlp2parquet/query-demo/">here</a>.</p> <pre data-lang="sql" class="language-sql "><code class="language-sql" data-lang="sql">LOAD httpfs; SELECT Timestamp, ServiceName, SeverityText, Body FROM read_parquet(&#x27;https:&#x2F;&#x2F;github.com&#x2F;smithclay&#x2F;otlp2parquet&#x2F;raw&#x2F;refs&#x2F;heads&#x2F;main&#x2F;testdata&#x2F;parquet&#x2F;logs.parquet&#x27;) ORDER BY Timestamp DESC LIMIT 5; </code></pre> <p>So imagine you actually do this. A few months later, you've got 100k+ small Parquet files (a "data swamp") sitting in a bucket. How do you query it, what happens when you want to change the schema, add partitions for faster queries, or consolidate (<a href="https://www.dremio.com/blog/compaction-in-apache-iceberg-fine-tuning-your-iceberg-tables-data-files/">compact</a>) small files and metadata into larger files?</p> <h3 id="writing-to-managed-apache-iceberg-s3-tables-and-r2-data-catalog">Writing to managed Apache Iceberg: S3 Tables and R2 Data Catalog</h3> <p>Enter catalogs like Apache Iceberg or Delta Lake, available as <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html">fully managed service on AWS</a> and <a href="https://developers.cloudflare.com/r2/data-catalog/">Cloudflare</a> (among <a href="https://docs.cloud.google.com/biglake/docs/about-blms">others</a>). Data warehouses have traditionally been big, expensive, and complex. This is starting to change with more or less "managed bucket" cloud services based on Iceberg that provide both the cheap storage and catalog/metadata management service in one.</p> <p>Iceberg handles snapshots, partitions and schema changes so your query engine doesn't have to crawl a bucket listing every time. These catalogs somewhat help wrangle the million tiny files problem, but they don't magically erase it. Someone still has to pay for compaction and decide how aggressively to merge things.</p> <p>(Fun fact: Cloudflare R2 Data Catalog has no additional cost for egress or using it over regular storage while in beta.)</p> <p>With managed Iceberg services, you get lakehouse-like semantics for working with OpenTelemetry in addition to performance benefits, like more efficient reads. In lakehouse terms, otlp2parquet plus R2 Data Catalog/S3 Tables are the metadata and transaction layer over Parquet. DuckDB is one of potentially many SQL/ML engines on top.</p> <p>Here's what it looks like connected to a Cloudflare R2 Data Catalog in duckdb. Once the connection is made, you query the data using regular SQL.</p> <pre data-lang="sql" class="language-sql "><code class="language-sql" data-lang="sql">--- Credentials to read R2 buckets CREATE SECRET r2_secret ( TYPE S3, KEY_ID &#x27;ACCESS_KEY_ID&#x27;, SECRET &#x27;SECRET_KEY&#x27;, ENDPOINT &#x27;ACCOUNT_ID.r2.cloudflarestorage.com&#x27;, REGION &#x27;auto&#x27;); --- Credentials to read catalog metadata CREATE SECRET r2_catalog_secret ( TYPE ICEBERG, TOKEN &#x27;API_TOKEN&#x27;); ATTACH &#x27;ACCOUNT_ID_BUCKET_NAME&#x27; as iceberg_catalog (TYPE ICEBERG, ENDPOINT &#x27;https:&#x2F;&#x2F;catalog.cloudflarestorage.com&#x2F;ACCOUNT_ID&#x2F;BUCKET_NAME&#x27;); --- Attach catalog ATTACH &#x27;account_bucket&#x27; AS iceberg_catalog ( TYPE ICEBERG, ENDPOINT &#x27;https:&#x2F;&#x2F;catalog.cloudflarestorage.com&#x2F;&lt;account-id&gt;&#x2F;&lt;bucket-name&gt;&#x27; ); --- Show available metric&#x2F;log&#x2F;trace tables SHOW tables FROM iceberg_catalog; </code></pre> <p>There's a big caveat: this pattern assumes batch commits, but observability forces you to confront what freshness means for streaming telemetry. Lakehouse systems assume a small number of writers doing big transactional commits... OTel looks more like a million tiny writers. A production scenario needs well-designed queues and aggregators (potentially just a well-tuned OTel collector) so that only a small number of writers update the Iceberg metadata.</p> <p>Thanks on a helpful pointer from a GitHub issue, am also exploring improved batching in <code>otlp2parquet</code> using <a href="https://github.com/smithclay/otlp2parquet/pull/48">Cloudflare Durable Objects</a>.</p> <p>There are several open source projects and startups trying to solve the "streaming database" problem in a more production-ready way: see <a href="https://fluss.apache.org/">Apache Fluss</a>, <a href="https://risingwave.com/">Risingwave</a> or <a href="https://github.com/Mooncake-Labs/moonlink">moonlink</a>. An observability startup called Parsable, built on top of object storage and parquet, details their solution for this <a href="https://www.parseable.com/docs/architecture">here</a>.</p> <h3 id="observability-s-data-engineering-moment">Observability's data engineering moment?</h3> <p>The observability team and the data engineering team may see each other at the company holiday party this year but probably don't eat lunch together. This is too bad: there's genuinely interesting stuff happening in the big data ecosystem.</p> <p>I don't think a lakehouse replaces your existing observability stack, but may eat its cold storage. A cheap, analytics-friendly back-end for long-term retention, weird one-off questions, and joining telemetry with the rest of your data is where it gets interesting. Youn own data storage you own also simplifies complex and growing regulatory requirements (PII, GDPR, etc).</p> <p>It's already shaking up how people do analysis on large amounts of <a href="https://geoiceberg.org/">geospatial data</a>, maybe logs and traces will be next.</p> <p>If a standard schema and <a href="https://duckdb.org/2025/10/13/duckdb-streaming-patterns">streaming patterns can be figured out</a>, the combination of cheap and queryable observability data, located entirely within the cloud of your choice, might make data lakes for metrics, logs, and traces much more common and usable.</p> <p>The data engineers might be using the components of your next observability stack.</p> <h3 id="related-projects">Related projects</h3> <ul> <li><a href="https://github.com/open-telemetry/otel-arrow">https://github.com/open-telemetry/otel-arrow</a></li> <li><a href="https://github.com/Mooncake-Labs/moonlink">https://github.com/Mooncake-Labs/moonlink</a></li> <li><a href="https://www.rakirahman.me/otel-arrow-delta-lake/">https://www.rakirahman.me/otel-arrow-delta-lake/</a></li> </ul> 2025-07-29T00:00:00+00:00 2025-07-29T00:00:00+00:00 Unknown https://clay.fyi/blog/iphone-claude-code-context-coding/ <p>You might be surprised to learn that <a href="https://docs.anthropic.com/en/docs/claude-code/overview">Claude Code</a> on your phone is fun, not frustrating. After experimenting several weeks, including using it during a two hour flight delay to build this blog and deploy it on <a href="https://vercel.com/">Vercel</a>: I'm now that person on the BART train <del>vibe</del> <a href="https://x.com/karpathy/status/1937902205765607626">context coding</a> from their iPhone.</p> <p>Claude Code, when combined with push notifications, some iOS apps, <a href="https://docs.anthropic.com/en/docs/claude-code/slash-commands">slash commands</a> and <a href="https://docs.anthropic.com/en/docs/claude-code/hooks">hooks</a> is surprisingly mobile-friendly. Thanks to <a href="https://tailscale.com/">Tailscale</a> and some <a href="https://www.wireguard.com/">mesh VPN crypto magic</a>, it now takes about 5 minutes to set up a private, always-on connection between your phone and home laptop or desktop.</p> <p>If inspiration strikes in the subway: you get a new superpower. You can also go along with your regular commute and get pinged by a notification when Claude Code needs your input. It's better for your brain than social media apps and you get to make cool stuff in unexpected places.</p> <p>Here's how I used Claude Code on my phone and you can too (with caveats). The hooks, commands, containers, and terminal layouts I used are <a href="https://github.com/smithclay/claudetainer">on Github</a>, where I've tried to package them in an "easy-mode" CLI that automates this setup inside a <a href="https://containers.dev/">devcontainer</a>.</p> <h2 id="the-minimal-mobile-app-stack">The minimal mobile app stack</h2> <img src="/images/cc-iphone/image1.png" alt="Mobile setup diagram" class="mobile-image"> <p>The basic ingredients for iPhone are:</p> <ul> <li>an iOS terminal app that supports mosh (recommend <a href="https://apps.apple.com/us/app/blink-shell-build-code/id1594898306">Blink Shell</a> or <a href="https://apps.apple.com/us/app/termius-modern-ssh-client/id549039908">Termimus</a>)</li> <li><a href="https://apps.apple.com/us/app/ntfy/id1625396347">ntfy</a> app for free push notifications</li> <li><a href="https://tailscale.com/">Tailscale</a> if connecting to your laptop/desktop, otherwise any remote server</li> <li><a href="https://zellij.dev/">zellij</a> or <a href="https://github.com/tmux/tmux/wiki">tmux</a> for switching between your desktop and phone easily and nice UI</li> <li>The <a href="https://apps.apple.com/us/app/github/id1477376905">GitHub app</a> for merging PRs, seeing CI output, etc</li> <li>a <a href="https://www.anthropic.com/pricing">Claude subscription</a></li> </ul> <p>The tech stack is pretty much identical for Android, but assume if you're a developer and Android user you've been running a custom shell on your phone since the early Obama administration.</p> <p><strong>Caveat #1:</strong> I don't recommend this workflow at all if you're new to Claude Code, even if you've been using similar tools like Cursor for a while. The idea is to be able to keep ideas and inspiration going on your mobile devices once you find a groove on a real computer with a keyboard.</p> <p>If you're past that stage, happy to report that this technical combination of apps and mobile-friendly remote protocols is surprisingly resilient. It works on a 3G connection. It also worked well on United's wifi somewhere over the North Atlantic.</p> <p><strong>Caveat #2:</strong> You're obviously not going to be doing much typing, and it's a good idea to bootstrap your project on your laptop first. However, you can make a lot of typos and use terrible, shortened English. Claude always understands what you mean when you ask it to "gut" commit your files. Consider adding a keyboard shortcut that types "<a href="https://claudelog.com/faqs/what-is-ultrathink/">ultrathink</a>" in the prompt.</p> <h2 id="setting-up-your-phone-to-laptop-pipeline">Setting up your phone-to-laptop pipeline</h2> <p>Before we start context engineering, some prerequisites are needed. Let's assume you have a personal macOS laptop with a reliable Wifi connection, and <a href="https://tailscale.com/">Tailscale</a> installed on both your phone and laptop. In my case, I've also connected my iPad and a cloud Linux server on DigitalOcean — everything can talk to everything else over the same private network.</p> <img src="/images/cc-iphone/image4.png" alt="Tailscale network diagram" class="mobile-image"> <p>This means, using an iPhone app like Blink Shell, you can run <code>mosh you@&lt;private Tailscale IP&gt;</code> (because <a href="https://mosh.org/">mosh</a> is much better for mobile roaming) and get a shell on your laptop anywhere in the world. That by itself is pretty cool.</p> <p><strong>Caveat #3:</strong> Tailscale is optional if you have a development server somewhere... potentially even <a href="https://github.com/features/codespaces">GitHub Codespaces</a> could be an interesting option.</p> <p>With the network and server in place: let's now build a fully functional Python app on our iPhone.</p> <h2 id="context-coding-on-the-go">Context coding on the go</h2> <p>While it's completely possible to do everything via your phone, it's much less painful to do the initial bootstrapping with a real keyboard. In my case, I used an <a href="https://github.com/fpgmaas/cookiecutter-uv">excellent python cookiecutter template</a> to create a modern python app scaffold with quality checks and a build system in a directory on my personal laptop. Good scaffolding with things like lint rules and code formatting saves hours (trust me on this). My example python project also has access to my systemwide collection of Claude Code hooks, commands and (soon) <a href="https://github.com/fpgmaas/cookiecutter-uv">sub-agents</a>.</p> <p>On my iPhone, I then open up a MOSH session to my laptop over my private Tailscale network inside of the Blink Shell app:</p> <pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">blink&gt; mosh clay@&lt;redacted-tailscale-ip-of-my-laptop&gt; </code></pre> <p>At this point: a lot happens behind the scenes. My shell profile detects a remote session and spins up (or resumes) a <a href="https://zellij.dev/">Zellij</a> layout that gives me a terminal-based Claude Code workspace optimized for mobile: I have my first tab to run Claude Code, then others to monitor usage of Claude Code itself using <a href="https://github.com/ryoppippi/ccusage">ccusage</a>, a view of my git history, and system performance info. I can switch between tabs by tapping on them, and my primary input device is the iOS keyboard with many shortened shell aliases (gs = "git status", etc).</p> <img src="/images/cc-iphone/image6.png" alt="Mobile terminal interface" class="mobile-image"> <p>It's normal/regular Claude Code at this point, with one more critical mobile addition: push notifications when Claude Code has been waiting for your input more than 60 seconds, which is just a lightweight integration between a hook and the free ntfy app:</p> <img src="/images/cc-iphone/image3.png" alt="Push notification example" class="mobile-image"> <p>The push notifications are really nice because, for especially long running Claude Code tasks, you can switch to other tabs or tasks on your phone (or life).</p> <p><strong>Caveat #4:</strong> Figuring out ways to reduce typing is, well, key. You <strong>must</strong> have pre-defined Claude Code commands (like /check and /test) that do meaningful repetitive work and automated quality hooks that validate the output of Claude.</p> <p>The other addition I'll highlight is monitoring of Claude Code itself ... when it's available on your phone, you may find yourself using it even more. I keep an eye on it in a separate tab:</p> <img src="/images/cc-iphone/image5.png" alt="Claude Code monitoring interface" class="mobile-image"> <p>If this seems like a lot to set up, it's mostly copying files to your ~/.claude directory and setting up some aliases: Claude Code can actually help you with both. Paste in your bash/shell history for some recommendations. If you don't want to use Tailscale, ask it to write some terraform and deploy a new server to the cloud of your choice.</p> <h2 id="why-this-matters-or-not">Why this matters (or not)</h2> <p>There's a possible near-future where the morning commute is full of people typing, yelling, or whispering to their phones about the quality checks, PR approvals, and research reports that were running overnight. But who knows what the future workflow might be? As Scott Werner said, <a href="https://worksonmymachine.substack.com/p/the-great-flood-of-adequate-software?utm_source=substack&amp;utm_medium=email&amp;utm_content=share">we're all junior AI engineers now</a>.</p> <p>I don't use Claude Code for work, but for fun non-professional projects. Here's what happened to my personal GitHub commit history after I got it working on my phone:</p> <img src="/images/cc-iphone/image2.png" alt="GitHub commit history showing increased activity" class="mobile-image"> <p>This is <a href="https://en.wikipedia.org/wiki/Jevons_paradox">Jevons' paradox</a> in action: I made this thing to use Claude Code on my iPhone, and I am now committing much more code for projects that I've wanted to build for a long time. The mobile constraint also had the effect of making me focus on what matters in this new workflow: good commands, well-defined <a href="http://claude.md">CLAUDE.md</a> files, and hooks to get the best results the first time without many keystrokes.</p> <p>The barrier between ideas and actually building a prototype of it is dissolving faster than the time it takes to take BART from Embarcadero to 24th Street. This setup isn't for everyone, but generally works for me and the types of projects I like to build for fun (python APIs, web apps, mobile apps, shell scripts).</p> <p>With <a href="https://support.anthropic.com/en/articles/11145838-using-claude-code-with-your-pro-or-max-plan">new Claude Code limits coming soon</a>, it's a good time to be focused on context and prompt efficiency. And you know what? It's way more fun than doom scrolling.</p> <p><em>Check out <a href="https://github.com/smithclay/claudetainer">https://github.com/smithclay/claudetainer</a> for the code.</em></p> 2024-07-24T00:00:00+00:00 2024-07-24T00:00:00+00:00 Unknown https://clay.fyi/blog/emperor-norton-meets-genai/ <blockquote> <p>Oyez, oyez! We, Norton I, Emperor of America and Chief Wrangler of Cantankerous Seagulls, do hereby proclaim: Our Royal persona shall be reborn in the realm of thinking machines! All citizens must converse with Our digital doppelgänger or face a tax of 37 pineapples.</p> </blockquote> <p><img src="/images/1_YLnC7GFt1FjaSwrKndmLUw.png" alt="San Francisco Panorama circa 1880" /></p> <p><em>San Francisco Panorama circa 1880</em></p> <p>The genteel alleys of Jackson Square north of San Francisco’s Transamerica Pyramid are full of boutique venture capital firms, design studios, and the occasional Michelin-starred restaurant. As one of the few neighborhoods not obliterated by the 1906 earthquake, it also has a direct connection to the city’s Barbary Coast origins. If you take a historical walking tour, you’ll hear about the gold and silver rushes, financial panics, political corruption, violent gangs of Australian convicts, and an eccentric named <a href="https://en.wikipedia.org/wiki/Emperor_Norton">Joshua Norton</a> who proclaimed himself Emperor of the United States and Protector of Mexico.</p> <p>The most recent San Francisco boom cycle involves generative artificial intelligence, not precious metals in the Sierras. The peculiar mix of local culture, technology, and money is the same. Developers are writing code on the same street where Emperor Norton wrote a proclamation with the then-absurd idea of a bridge across the Bay. He was the original mascot, progressive, oddball, and minor celebrity who helped generations of locals reflect on what kind of a place San Francisco was or could be.</p> <p>Last year, I decided to use generative AI to recreate Norton’s voice and have it engage with people on social media. I was interested in how an AI model of Norton, which I’ll refer to as the Emperor Bot, could make us more creative. Much of what is written about AI seems to be about civilization-ending scenarios or startup pitches for new ways to summarize your emails. I wanted to know if generative AI could help people reflect on the past… and possibly laugh.</p> <h4 id="building-an-emperor-bot-that-challenges-injustice"><strong>Building an Emperor Bot that challenges injustice</strong></h4> <p>The first thing I had to do was build the bot, which was the most fun I’ve had on a technical project in a long time. How it works is straightforward, even for a non-programmer: you instruct a large language model (LLM), like OpenAI’s GPT-4, to reply as Emperor Norton. This instruction is called the prompt.</p> <p>Some facts about Norton have been incorporated into the base LLM models. The result is cutting-edge technology approximating how to speak in the voice of someone from the 19th century. After I made dozens of small adjustments to the prompt — it was difficult to get the replies concise — here’s the bot’s reply to a tweet about seagulls and a locally famous Taco Bell:</p> <blockquote> <p>Emperor Norton I has decreed that all seagulls must be removed from Pacifica &amp; its environs. The Board of Supervisors must award damages to the owners of the desecrated Cathedral of Taco Bell.</p> </blockquote> <p>The reply captures three things that are essential to the historical Norton: it challenges injustice (albeit of a Taco Bell), holds politicians accountable, and does so with the authority of someone with absolute political power.</p> <p>Here’s another reply to a tweet from a local supervisor proposing renovation of some crumbling piers that doesn’t hit the mark:</p> <blockquote> <p>We decree that Piers 30–32 in San Francisco shall be rebuilt as a mixed-use development with public recreation, retail, office, and housing!</p> </blockquote> <p>The pro-development tweet was it seemed inauthentic in the worst possible way. It didn’t capture the historical voice. It was like seeing an ad for breakfast cereal featuring an Emperor Norton proclamation on whole grains. I wanted to understand if this was truly a limitation of the current models, or something that could be improved.</p> <h4 id="finding-norton-s-authentic-voice-in-the-llm"><strong>Finding Norton’s authentic voice in the LLM</strong></h4> <p>One of the issues with recreating Norton’s voice is that, while everything he wrote is in the public domain, there is a thin historical record for someone who continues to be famous in the modern era. The proclamations he published were short, and no journal or diary survived. With the help of local historians I became more familiar with gold rush era San Francisco. John Lumea of the<a href="https://emperornortontrust.org/">Emperor Norton Trust</a>, the world’s foremost expert on Norton, sent me several helpful emails, and Taryn Edwards of the <a href="https://www.milibrary.org/">Mechanics’ Institute</a> (where the real Norton was a member) shared a reading list with a warning: <em>San Francisco history is rife with lore and legend</em>.</p> <p>I can never know Norton’s views on the correct amount to tip at restaurants, but did know he was always against corruption and wanted to keep politicians accountable… to the point of calling for Congress to be abolished. Using more AI, I generated a map of the themes and topics Norton wrote about and hoped it would guide better responses.</p> <p>For example, Norton had a proclamation that referred to sailing ships — but his writing on the ships thematically touched on justice, authority, responsibility and safety. When he wrote about being refused entrance to an event by “snobs in charge of the door” — the map pointed to his concerns around the abuse of power and social hierarchies.</p> <p>Instead of parroting or sounding like a shill, the bot’s replies seemed to improve with the new context. Here’s the reply to the news of a local brewery closing:</p> <blockquote> <p>Great Scott! Anchor Brewing, a city gem, faces peril? We’ll rally, oh we will! Workers, I stand in solidarity with you! Citizens, let us preserve our city’s heritage!</p> </blockquote> <p>It was around this time a controversial local billionaire changed policies and most bots, including the Emperor Bot, were kicked off Twitter.</p> <h4 id="bluesky-social-hackerspaces-and-vegetables"><strong>BlueSky Social, hackerspaces and vegetables</strong></h4> <p>Elon Musk kicked the bots off Twitter because he wanted money for the type of access the bots needed to post, which was previously free. This resulted in the Emperor Bot moving to the newer social network <a href="https://bsky.app/profile/emperornorton.bsky.social">BlueSky</a>.</p> <p>Despite being a friendlier and invite-only version of Twitter at the time, I was concerned what might happen. Despite profile disclosures and a prominent robot icon, my impression is that most people assumed the text was written by a cosplaying San Francisco history nerd pretending to be a digital version of Norton.</p> <p>Non-public tests with the bot commenting on trending topics went badly. The bot never generated outright offensive content, but anything related to politics was inflammatory. Even positive replies, like support for the Hollywood writers’ strike, were problematic. People striking for reasons related to losing their livelihoods over AI-generated content do not want a supportive yet eccentric 19th century bot to join the conversation. The fully-automated social media version was never released, so I reviewed replies and posted the unaltered text every few days.</p> <p>The most engaging conversations with the bot were in private channels on Discord, the popular chat app. A local hackerspace added it to their AI channel and started interacting with it. Using the latest tricks being spread around the LLM space, bot replies seemed to improve when I added an “therapy” mode where the bot would articulate its sensations, fears and desires before replying.</p> <p>Here’s the bot’s inner monologue for replying to something related political corruption:</p> <blockquote> <p>Sensations: A harsh, sneering laughter rings in my ears, leaving an unpleasant taste in my mouth.</p> </blockquote> <blockquote> <p>Fears: Anarchy and corruption taking over my beloved city.</p> </blockquote> <blockquote> <p>Desires: To enact justice and maintain the order of my realm.</p> </blockquote> <p>Somebody also asked the bot what vegetable it found the most threatening.</p> <blockquote> <p>I would say it is the artichoke. Its sharp exterior might be seen as a symbol of uprising, but in truth, it’s the tastiest rebellion I’ve ever faced.</p> </blockquote> <h4 id="a-folk-hero-emerges-on-bluesky-social"><strong>A folk hero emerges on BlueSky Social</strong></h4> <p>The bot was finding a certain groove on Discord, but on BlueSky it was limited to replys related to a very small number of topics: mostly pets and Norton himself. I was surprised how culturally relevant Norton, who died in 1880, is to modern-day San Francisco and how deep that appreciation and fandom goes. An award-winning author was one of the first followers.</p> <p>My impression is that Norton is a folk hero on social media or, depending on who you ask, the world’s first crypto bro. (He printed his own money and passed it off as legal tender.) People imply they would love to vote for him in the current election. He’s also considered a minor saint in a belief system that started in 1960s Berkeley called <a href="https://en.wikipedia.org/wiki/Discordianism">Discordianism</a> (no relation to the app Discord).</p> <p>The most liked posts from the bot, after some trial and error, were those that acted as a kind of promoter for San Francisco and the historical Norton. Anything else tended to annoy or confuse people, like John Adams showing up on Instagram and liking your vacation photo. When someone skeeted (that’s BlueSky for tweet) that Norton was cool and they wished he could save everyone now, it replied:</p> <blockquote> <p>Indeed, time unceasingly marches forward. While I can’t step forth in flesh, my philosophies persist. Lead with benevolence, act with authenticity.</p> </blockquote> <p>In another instance, the bot replied to a post memorializing Norton with a familiar sounding phrase: “death is but a door, time a window.” It’s a quote from the villain in the 1989 movie <em>Ghostbusters II</em>… a 16th century historical figure that comes back to life in an evil painting.</p> <h4 id="what-are-the-chances-of-the-emperor-bot-taking-over-the-world"><strong>What are the chances of the Emperor Bot taking over the world?</strong></h4> <p>The 1906 earthquake destroyed much of Norton’s San Francisco. The murkiness of the late 19th century also means that it’s easier for every generation to repurpose Norton to fit their mood and anxieties. In present day San Francisco and the United States: there are plenty of issues to choose from.</p> <p>Of course, one of the largest local conversations in technology is the promise and perils of generative AI itself.</p> <p>It’s not hard to imagine these sorts of bots finding a place in educational settings or museums to make history more interactive and conversational, at least for well-liked and popular historical figures. The bot would be an excellent companion in a self-guided walking tour of the Emperor’s old neighborhood.</p> <p>There’s also the obvious danger of these bots on social media, which has been written about at length in many different places. My assessment is the risk of the Emperor Bot causing global conflict or taking over governments as a digital Emperor is effectively zero. However, it’s hard to think of any scenario where more advanced gen AI bots will improve the overall quality of conversation in the near-term on social media.</p> <p>There has been an enormous amount of investment from major companies to make sure the output of LLMs is inoffensive and not harmful, but that is changing as individuals and groups create their own models. Using a collection of scanned 19th century newspapers, I started on an experiment with fine-tuning the bot to make it sound more like actual writers from the 1880s. That was paused once I realized that I did not want my bot to pick up on the views that existed in the vast majority of late 19th-century American newspapers.</p> <h4 id="norton-generative-ai-and-a-way-forward"><strong>Norton, generative AI, and a way forward</strong></h4> <p>Joshua Norton is buried in Colma, the cemetery city just south of San Francisco. There’s a marble headstone that says <em>Norton I Emperor</em> in large letters. E Clampus Vitus, a 19th-century fraternal organization (sometimes called in the press a “historical drinking society”) throws a party honoring his legacy once a year on “Norton Day” at his gravesite. There’s a convenient Irish bar a short stroll away.</p> <p>My theory on why Norton still has a large number of admirers who toast him is because, in his own chaotic and eccentric way, he is remembered for fighting against injustice. He also was an immigrant to San Francisco who made and lost a fortune but still figured out a way to get free lunches and had a popular newspaper column for twenty years.</p> <p>Generative AI will never be able to capture the full context, intricacies, and contradictions of the human Norton but it can be strange and unexpected. Despite the peril of thousands of trolling bots taking over social media, in the right context this technology can be fun and creative — without taking over the world or your job.</p> <p>Uncertainty hangs over the air right now for both AI and San Francisco. There are plenty of doom loop scenarios being discussed for both. Economically, the success or failure of generative AI will also have a direct impact on The City for years to come. The open question for local entrepreneurs depositing checks from venture capitalists is what happens next. At one extreme: collapse from malevolent AI. Another direction is the mediocrity of bots that make life worse via annoying assistants that get in the way.</p> <p>A happy outcome would be technology — and a city — where people are able to be thoughtful, explore new ideas, and maybe have fun while holding those in power accountable. It might also get a little weird and off-putting at times. We should, in other words, take more inspiration from Emperor Norton.</p> <p><em>Thanks to John Lumea of the Emperor Norton Trust, Tayrn Edwards of the San Francisco Mechanics Institute, the weekend staff at The San Francisco History Center, the Prelinger Library, The Ancient and Honorable Order of E Clampus Vitus, the Misalignment Museum, and various adherents of Discordianism active on BlueSky and Discord.</em></p> <p><em>You can follow the Emperor Bot at</em><a href="https://bsky.app/profile/emperornorton.bsky.social"><em>https://bsky.app/profile/emperornorton.bsky.social</em></a></p> 2019-07-02T00:00:00+00:00 2019-07-02T00:00:00+00:00 Unknown https://clay.fyi/blog/neural-net-apple-zealot-wired-training/ <p><img src="/images/1_MV3CFoPXFNieMtAmHfBzew.png" alt="Generated text using OpenAI’s GPT-2 and OCR text from mid-1990s WIRED issues" /></p> <p><em>Generated text using OpenAI’s GPT-2 and OCR text from mid-1990s WIRED issues</em></p> <p>I trained a bleeding-edge machine learning (ML) model with a dozen issues of WIRED magazine from the mid-1990s and things got weird. The new model started generating a unique blend of 90s-era techno-optimism, product advertisements for microprocessors and gaming systems that never existed, odd interviews with tech luminaries and… a profound love for all things Apple.</p> <p>There’s a popular ML technique going around where people generate text using neural networks based on different source materials. The technology isn’t advanced enough to make us think that a sensible person actually wrote the generated text—a serious problem we’re going to have to deal with—but it’s often good enough to produce bizarre statements that are funny in context.</p> <p>I first tried it around <a href="https://medium.com/clog/using-ai-to-generate-an-ico-white-paper-d328f26577f3">two years ago to generate an ICO whitepaper</a> and there have been many other developers, data scientists, and researchers who have continued to explore. <a href="https://aiweirdness.com/aboutme">Janelle Shane’s posts on this subject</a> are wonderful and inspired this work. In the past two years, the effectiveness of the models have significantly improved.</p> <p>This post is about what happened when I wanted to train one of the newer models going around (<a href="https://openai.com/">OpenAI’s</a> GPT-2) with mid-1990s WIRED magazine issues.</p> <h4 id="wired-nostalgia-and-machine-learning">WIRED, Nostalgia and Machine Learning</h4> <p>I grew up reading WIRED magazine, and I’m not immune to the nostalgia for the period of the 1990s when the internet and personal computers were a new and amazing thing for many people in the United States.</p> <p>Generating text from the model is surprisingly addictive—with a click of a button, it’s like reading something a WIRED journalist might write after not sleeping for three days.</p> <p><img src="/images/1_dT1Xoxn_UesQcpnZ2_y_OA.png" alt="" /></p> <p>There is also a strange blend of marketing and journalism is impacted by the training data—WIRED issues in the 90s, after all, had plenty of print ads. As a result you get serious-sounding statements about history mixed with product marketing speak:</p> <p><img src="/images/1_MRMxUNJjri33TuHdsUeXtQ.png" alt="" /></p> <p>Sometimes this veers off into editorialization. After a quote around how you can get the latest PowerBook “when it’s available” (which every tech company would write in announcements if they could get away with it), this paragraph ends with something that sounds like a melody:</p> <p><img src="/images/1_hWcqId3l-SgVdJ3fIPdzFg.png" alt="" /></p> <p>When it comes to technology luminaries of the mid-1990s, even seemingly critical quotes of Jobs end up making him look good:</p> <p><img src="/images/1_xwyX_GIO6lFeOtGMrr3qUw.png" alt="" /></p> <p>Bill Gates, in contrast, basically has to plead that Microsoft isn’t actually in trouble when seeking an ambitious target of “download hits”:</p> <p><img src="/images/1_XOZmjNKiRNTk3zQZzD-7RQ.png" alt="" /></p> <h4 id="the-machine-learning-hype-is-real">The machine learning hype is real?</h4> <p><img src="/images/1_KCP6IU0uWk9GrwZHp2HC5Q.png" alt="The hype, man, it’s real." /></p> <p><em>The hype, man, it’s real.</em></p> <p>This started as a quick AI experiment to see how text generation models were advancing. From start to finish (including training time), it took only about two hours before text started coming out, including some text that wouldn’t look out of place in an experimental poetry journal.</p> <p>While it was fun and fascinating to see what was generated, it also doesn’t seem unrealistic that more advanced versions of the model would probably be good enough to generate entirely convincing WIRED articles that seemed like they were written in the 90s. That’s disturbing for anyone who reads digitized historical sources of anything on the internet.</p> <p>The creator of the GPT-2 model, OpenAI, <a href="https://openai.com/blog/better-language-models/#update">has committed to working with research and governmental institutions on the societal impact of widespread use of models like this</a>, including an advanced version of GPT-2 that hasn’t been publicly released yet.</p> <p>As a creative tool, however, my experiments with WIRED and neural networks were unexpectedly inspiring. Strange alternate histories, quotes, bizarre products, and koan-like statements around the future emerged.</p> <p>My deep concern for the future of these language models is combined with a desire to read the first few chapters of a book featuring Larry Ellison, his wife FmmL, and Denny Dallas. Maybe they’re all Apple fans.</p> <p><img src="/images/1_vRa0PhaxK1OrvpC-brN3ww.png" alt="The plot of 2025’s bestselling tech thriller?" /></p> <p><em>The plot of 2025’s bestselling tech thriller?</em></p> <p><em>Thanks for reading! Might get around to turning this model into some kind of chat bot in the next few weeks. Find me on twitter: <em><a href="https://twitter.com/smithclay"><em>@smithclay</em></a></em>.</em></p> 2019-05-02T00:00:00+00:00 2019-05-02T00:00:00+00:00 Unknown https://clay.fyi/blog/tcp-ip-over-cloudwatch-logs/ <h4 id="running-network-services-inside-aws-lambda-functions">Running network services inside AWS Lambda Functions</h4> <p>You can’t ping an AWS Lambda function. You can’t SSH into a running function <a href="https://medium.com/clog/ssh-ing-into-your-aws-lambda-functions-c940cebf7646">without reverse tunneling</a>. You would never run nginx or Rails inside of a function because there’s little point — accepting an inbound TCP connection on a port like a Docker container isn’t supported by function-as-a-service (FaaS) platforms. It is possible, however, to write your own userspace networking stack and do what you want with it —including using Amazon Cloudwatch Logs (or even AWS Lambda tags) as the data link layer.</p> <p>This post is about implementing TCP/IP over Amazon Cloudwatch Logs using Go, which enables you to access network services running inside of AWS Lambda functions.</p> <p>It’s slow and not very useful, but it was a fun way to learn more about Linux networking and using AWS services in a way that might horrify some AWS engineering teams. I’m calling this proof-of-concept “<strong>Richard Linklayer”</strong> and <a href="https://github.com/smithclay/rlinklayer">published the full Github code today.</a></p> <h3 id="a-serverless-architecture-darkly">A Serverless Architecture Darkly</h3> <p>This experiment was the escalation of:</p> <ul> <li>Trying to run progressively stranger things in AWS Lambda.</li> <li>Interest in exploring netstack, an IPv4/IPv6 userland networking stack from Google.</li> <li>Hearing contradictory and confusing things from some clients and engineering teams around different limitations of serverless functions.</li> </ul> <p>I attempted to check off all three of those things using Go. With netstack, it’s possible to do some exotic things with networking in unexpected places, no kernel hacking (i.e. writing C code) required. Any Go program with read and write access to reliable-ish bi-directional communication channel — like logs, tags or <a href="https://en.wikipedia.org/wiki/IP_over_Avian_Carriers">carrier pigeon</a> — can implement full in-process TCP/IP networking. There are many AWS services that meet this criteria.</p> <p>Network tunneling is not a new idea, but inside the AWS Lambda execution environment a userspace solution is needed because it’s not possible to change routing tables, modify network interfaces, or change OS-level networking (unlike container-based network overlays or service meshes).</p> <p>Running in a standard Go process, Richard Linklayer tunnels IP packets over Amazon Cloudwatch Log Streams that follow a special naming convention — the stream and log group names are just MAC addresses. Using a tun or tap interface, I can bridge my AWS Lambda network endpoints to my local development machine:</p> <p><img src="/images/0_sYCn3BhpTon89Nt8.png" alt="High-level architecture of Richard Linklayer where a Linux host communicates with a TCP network service running inside of an AWS Lambda function. The process polls Amazon Cloudwatch for new inbound packets." /></p> <p><em>High-level architecture of Richard Linklayer where a Linux host communicates with a TCP network service running inside of an AWS Lambda function. The process polls Amazon Cloudwatch for new inbound packets.</em></p> <p>This design ignores normal AWS Lambda event-driven patterns for processing inbound packets and sending outbound packets —only a single instance of a function starts, reads and writes from Cloudwatch using polling, then processes packets until it times out. There is only one instance of a function running a unique network service running at any given time, and it can only accept network connections for up to 15 minutes, the maximum timeout of an AWS Lambda function in early 2019.</p> <p><img src="/images/0_wPTklPyIuEI3LoXT.png" alt="Lifecycle of an AWS Lambda function that accepts inbound TCP requests via Cloudwatch. Don’t do this." /></p> <p><em>Lifecycle of an AWS Lambda function that accepts inbound TCP requests via Cloudwatch. Don’t do this.</em></p> <p>When you put it all together, the result is that you can ping a Lambda function.</p> <p><img src="/images/0_2wJktNatuiupDEbN.png" alt="Pinging an AWS Lambda function on Mac OS X via a Linux container. It’s kind of slow but remember everything is running over Amazon Cloudwatch." /></p> <p><em>Pinging an AWS Lambda function on Mac OS X via a Linux container. It’s kind of slow but remember everything is running over Amazon Cloudwatch.</em></p> <p>The underlying implementation over Amazon Cloudwatch is at <a href="https://en.wikipedia.org/wiki/Data_link_layer">layer 2,</a> so it’s reading and writing ethernet packets delivered to MAC addresses. MAC address discovery happens using <a href="https://en.wikipedia.org/wiki/Address_Resolution_Protocol">ARP</a> with a special-purpose log stream that acts as a “broadcast address” for the entire network.</p> <p>Since it supports anything that runs over TCP, you can also cURL a node.js HTTP server running inside of a function, or anything else that listens on a TCP port. This server returns some text after a few seconds.</p> <p><img src="/images/0_pVFK-lYuXlqFkNXn.png" alt="Making an HTTP request to a node.js server running inside of an AWS Lambda function." /></p> <p><em>Making an HTTP request to a node.js server running inside of an AWS Lambda function.</em></p> <p>The latency means that this would likely never work for anything production-level, but TCP is resilient enough to handle the slow connection.</p> <p>We have full transparency into the network traffic since it’s just data inside of log streams. It looks like this, which is the stream of IPv4 traffic from <em>74:74:74:74:74</em> to <em>42:42:42:42:42</em>:</p> <p><img src="/images/0_biGVnazegJoe0ZG8.png" alt="Layer 2 Observability? Traffic from one MAC address to another that uses Cloudwatch as the channel." /></p> <p><em>Layer 2 Observability? Traffic from one MAC address to another that uses Cloudwatch as the channel.</em></p> <p>An example on how to get this working with a <a href="https://github.com/smithclay/rlinklayer/tree/master/examples">simple network running an HTTP server is on Github</a>. While not described in this post, there is also an additional link layer that uses AWS Lambda function tags as a transport that has slightly better performance results.</p> <h3 id="dazed-and-lambfused">Dazed and Lambfused</h3> <p>There’s a lot of interesting open-source networking projects in the container space right now: <a href="https://github.com/envoyproxy/envoy">Envoy</a>, <a href="https://linkerd.io/">Linkerd</a>, and <a href="https://github.com/cilium/cilium">Cilium</a> to name a few. With support for custom AWS Lambda runtimes released at re:Invent in 2018, I’m curious how emerging control or data planes are going to impact serverless functions beyond additional language support and monitoring and security solutions.</p> <p>Beyond that, there’s a few interesting ideas to explore beyond this proof-of-concept, including:</p> <ul> <li>UDP services (DNS over AWS Lambda, anyone?)</li> <li>Any kind of performance optimizations (everything is based on polling APIs and could be tuned)</li> <li>Exploration of other link layer transports in AWS (DynamoDB streams? SNS? SQS? S3?)</li> </ul> <p><a href="https://github.com/smithclay/rlinklayer">Pull requests and ideas welcome.</a></p> 2017-11-16T00:00:00+00:00 2017-11-16T00:00:00+00:00 Unknown https://clay.fyi/blog/ai-generate-ico-white-paper/ <p><img src="/images/1_rH9Lh2RZprmbUp4FXM6U0w.png" alt="The white paper generated using a recurrent neural network." /></p> <p><em>The white paper generated using a recurrent neural network.</em></p> <p>With ICOs, or token sales of cryptocurrencies, <a href="https://hackernoon.com/insider-reflections-on-the-ico-bubble-6304c4d25ed8">having reportedly raised $2.3 billion in 2017,</a> the time seemed right to use AI to generate a fake ICO white paper programmatically using a <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">recurrent neural network</a> (RNN) in the cloud.</p> <p>I’ve titled it “RNN-Coin” <a href="https://github.com/smithclay/char-embeddings-ml-engine/blob/master/whitepaper.pdf">and have published the full white paper here (pdf)</a>.</p> <p>RNNs are a machine learning technique that have frequently and hilariously been used to generate arbitrary text from text-based training data. Some of my favorite examples are generating <a href="https://medium.com/@hondanhon/i-trained-a-neural-net-to-generate-british-placenames-9460e907e4e9">new British town names</a> or <a href="https://github.com/nihitx/game-of-thrones-">a Game of Thrones</a> book. However, unlike most town names or the work of George R.R. Martin, there are plenty of people that claim that ICO whitepapers are mostly incomprehensible anyway. This post shares some the technical background and some of the excerpts from the paper (it’s painful to read in full).</p> <h4 id="building-the-model">Building the model</h4> <p>Using <a href="https://www.tensorflow.org/">Tensorflow</a>, <a href="https://cloud.google.com/ml-engine/">Google Cloud ML engine</a>, <a href="https://github.com/mfclarke/char-embeddings-ml-engine">and a modified RNN example I came across on Github from Maximilian Clarke</a>, I created a model using a GPU-powered cloud training job that ran for a few hours (my MacBook Pro was too slow and RNNs are resource-intensive to train). The trained model could then output text that eerily echoed many ICO white papers.</p> <p>For the published white paper, I generated around 2,000 characters at a time from 40-character sentence seeds I pulled from real white papers (<a href="https://bitcoin.org/bitcoin.pdf">like the Bitcoin white paper</a>). I then manually organized the output into sections — formatted in LaTeX for full white paper legitimacy, of course.</p> <p>One of the first outputs became the basis of the white paper abstract:</p> <blockquote> <p>A purely peer-to-peer version of the company system with the price of the system to develop and the contract agent servers and the price of the system of the system which is a simple decentralized system which is the contract to provide the contract […] can be considered as a result of the blockchain.</p> </blockquote> <p>The algorithm created peer-to-peer, blockchain-backed alternative to corporate structure! It’s just a simple decentralized system. The price is still developing, however.</p> <h4 id="the-local-revolution-of-experience-is-here">The local revolution of experience is here</h4> <p>RNNs have been described by<a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">some people as “magical” or “uncanny”</a> and after going through this exercise I agree. There are points in the white paper that sound perfectly reasonable — and even decent marketing. What millennial, including myself, <em>wouldn’t</em>want to be part of “the local revolution of experience”?</p> <p>It often generates language constructions that don’t sound natural. This could probably be tuned further by trying cleaner training data and different model parameters. Yet strange-sounding sentences also have gems:</p> <blockquote> <p>This is the highest commitment wallet that a confident of money for medical currencies of the project in the initiative instruments of the project is the farmer of the cost of the system.</p> </blockquote> <p>A “highest commitment wallet” might be a reassuring concept for an ICO investor in medical currencies.</p> <h4 id="a-pledge-to-the-smart-token-and-having-fun-with-ml">“A pledge to the smart token” (and having fun with ML)</h4> <blockquote> <p>The first social advance is a problem for control of the platform.</p> </blockquote> <p>I don’t see RNNs replacing real white paper writers or creating ICOs, but the experience of building and training fairly complex machine learning models that were previously the exclusive domain of academics and researchers does seem to be getting much easier for the average developer.</p> <p>Much of the code for sample tensorflow projects using different ML techniques is freely available on Github and it seems cloud providers are now <a href="https://aws.amazon.com/ec2/elastic-gpus/">racing to rent GPUs</a> to developers for model training purposes. I think as more tensorflow projects get shared the potential for collaboration also goes up: if a real AI/ML researcher comes across this post, p<a href="https://github.com/smithclay/char-embeddings-ml-engine">ull requests are welcome</a> to refine the model.</p> <p>There is much speculation and uncertainty around the future of ICOs, but it also seems like there has never been a better time to generate fake ICO white papers with AI.</p> <p><a href="https://github.com/smithclay/char-embeddings-ml-engine/raw/master/whitepaper.pdf">The full paper is available for download here (pdf).</a></p> 2017-10-17T00:00:00+00:00 2017-10-17T00:00:00+00:00 Unknown https://clay.fyi/blog/ethereum-smart-contract-19th-century/ <p>Thought it’d be fun to explore blockchain hype and write a simple <a href="https://www.coindesk.com/information/ethereum-smart-contracts-work/">Ethereum smart contract</a> based on an short story from the 19th century.</p> <p>In middle school, I remember reading Robert Louis Stevenson’s <a href="https://en.wikipedia.org/wiki/The_Bottle_Imp"><em><strong>The Bottle Imp</strong></em></a>, written in 1891. The story describes the the struggles of a person who buys a magic bottle from an elderly man who lives in a “fine house on a hill” in San Francisco. It grants unlimited wishes with certain <em>evil</em> conditions: you can’t sell it to anyone else for more than you paid for it and if you can’t find a willing buyer before you expire <strong>you’re doomed</strong>. You also have to pay in real minted coins: no paper money is allowed so there’s some physical limit to how cheaply you can sell it.</p> <p>Remembering this story while reading about the blockchain, the obvious thing to do was attempt to implement it as an <a href="https://ethereum.org/">Ethereum</a> smart contract on a sunny Sunday afternoon. <a href="https://github.com/smithclay/etherimp">I’ve published the code on Github here</a>.</p> <h4 id="my-attempt-at-sounding-like-i-know-about-what-i-m-talking-about-about-with-blockchain">My attempt at sounding like I know about what I’m talking about about with blockchain</h4> <p>This was my first attempt and building a ÐApp (<a href="https://ethereum.stackexchange.com/questions/383/what-is-a-dapp">decentralized app</a>). I feel like including this diagram of distributed networks from a 1962 academic paper gives this post some additional gravitas and is completely self-explanatory on the benefits of decentralized applications, the nature of distributed ledgers, and inherent security considerations of applications built using this technology:</p> <p><img src="/images/0_tbaizJv1spbNUhG2.png" alt="On Distributed Communications Networks, Paul Baran (1962)" /></p> <p><em>On Distributed Communications Networks, Paul Baran (1962)</em></p> <h4 id="now-that-we-have-a-full-and-complete-understanding-of-ethereum-and-distributed-applications-how-does-the-contract-work">Now that we have a full and complete understanding of Ethereum and distributed applications, how does the contract work?</h4> <blockquote> <p>As for why I sell so cheap, I must explain to you there is a peculiarity about the bottle. — The Bottle Imp</p> </blockquote> <p>In the process of writing this smart contract, I was terrified. Outside of the usual pains of figuring out how to write, deploy, and build things using a completely new toolchain (the <a href="https://solidity.readthedocs.io/en/develop/">Solidity</a> language, <a href="https://github.com/ethereumjs/testrpc">testrpc</a>, and the <a href="https://github.com/ethereum/wiki/wiki/JavaScript-API">web3</a> APIs), it was unsettling to read though a <a href="https://github.com/ConsenSys/smart-contract-best-practices">popular resource on potential security vulnerabilities from ConsenSys</a>.</p> <p>The contract I wrote (my first) does only one trivial thing by design: <em>transfer ownership of the (virtual) bottle if they pay less than the previous owner</em>.</p> <p>However, from best practices I found it seems clear that production-grade smart contracts need to go through a lot of testing and analysis. Specifically, it’s a good idea to:</p> <ul> <li>Really understand the Solidity language and know the danger of external calls, transfer calls, and reentrancy vulnerabilities.</li> <li>Integrate rate-limiting and “pause” functionality if things go wrong.</li> <li>Use static analysis and linting to catch vulnerabilities.</li> <li>Explore different patterns to upgrade your contract as new vulnerabilities are discovered.</li> <li>Start a bug bounty program to thoroughly test it (with rewards to those who find issues) before the contract goes live on the main ethereum network. Maybe hire some consultants for auditing purposes, too (they were all over my Google search results for Solidity development questions).</li> <li>Consider using frameworks like truffle or OpenZepplin when you realize you’re probably doing it wrong.</li> </ul> <p>Attempting to, uh, “implement” some of those best practices over a couple hours, <a href="https://github.com/smithclay/etherimp/blob/master/contracts/EtherImp.sol#L24">I came up with this code for the key logic in the contract to transfer the bottle</a>. If the price is less than the price the previous owner paid and it’s non transferred back to the same account— the bottle has a new owner and the price paid is transferred back the previous owner:</p> <pre data-lang="solidity" class="language-solidity "><code class="language-solidity" data-lang="solidity">function buyBottle() payable public { &#x2F;&#x2F; Conditions require(msg.sender != currentOwner); require(msg.value &gt; 0); require(msg.value &lt; lastPricePaid); &#x2F;&#x2F; Effects previousOwner = currentOwner; currentOwner = msg.sender; lastPricePaid = msg.value; LogTransfer(previousOwner, currentOwner, lastPricePaid); &#x2F;&#x2F; Interactions &#x2F;&#x2F; *Security alert (correction on October 27, 2017):* &#x2F;&#x2F; This implementation isn&#x27;t safe! See GitHub and comments. previousOwner.transfer(msg.value); } </code></pre> <p>The “conditions”, “effects” and “interactions” comments are only for reassurance based on a <a href="https://ethereum.stackexchange.com/questions/19341/address-send-vs-address-transfer-best-practice-usage">solidity question I read on StackOverflow</a>. Otherwise, I’m conservatively ~40% sure it works without a major structural flaw or security vulnerability.</p> <p><strong>Like the original short story, there’s ultimately a loser in this contract: if the value paid is 1 wei — the smallest allowable unit in Ethereum, you’re stuck with ownership of this smart contract forever.</strong></p> <p>After some local testing and <a href="https://www.rinkeby.io/">trying it out on the Rinkeby test network</a>, decided it was time to ship it.</p> <h4 id="it-s-live-on-the-main-ethereum-network-beware">It’s live on the main Ethereum network, beware</h4> <blockquote> <p>“This is the bottle,” said the man; and, when Keawe laughed, “You do not believe me?” he added. “Try, then, for yourself. See if you can break it.” — The Bottle Imp</p> </blockquote> <p>The contract/virtual bottle has been published to the main Ethereum network and is able to be transferred to anyone who wants it— as long as you pay less than the previous account did. I’m curious to see how long it takes for the value to get down to 1 wei and what happens (if anything).</p> <p>To kick things off, <em>I’ve started the price of the contract at the going rate of a of a typical sandwich in the Financial District of San Francisco: 0.05 ETH</em> (~16 USD).</p> <p>If you’re new to Ethereum, I’d recommend downloading the <a href="https://github.com/ethereum/mist/releases">Mist browser</a> or <a href="https://chrome.google.com/webstore/detail/metamask/nkbihfbeogaeaoehlefnkodbefgpgknn?hl=en">Metamask extension</a> and then visiting the following address: <a href="https://wallet.ethereum.org/account/0x9fcd683584173c5df0ee47852f5592f68ed72a0c">https://wallet.ethereum.org/account/0x9fcd683584173c5df0ee47852f5592f68ed72a0c</a></p> <p><img src="/images/1_qChPwEBf0lLVbFIfjP40uQ.png" alt="The contract visualized in the Mist browser. The &quot;buy bottle&quot; function transfers ownership." /></p> <p><em>The contract visualized in the Mist browser. The “buy bottle” function transfers ownership.</em></p> <p>Feedback and vulnerability assessments welcome in the comments, on <a href="https://twitter.com/smithclay">twitter</a>, <a href="https://github.com/smithclay/EtherImp">or on the Github project page</a>.</p> <p>If you’re new to all this can also recommend <a href="https://www.coinbase.com/what-is-ethereum?locale=en-US">Coinbase’s general introduction to Ethereum (and how it’s different from Bitcoin)</a>. For getting started with development, the <a href="https://www.reddit.com/r/ethereum/">Reddit community</a> was useful too.</p> <p>Regardless of where the blockchain hype goes — and who ends up stuck with the smart contract I created when they buy inevitably buy it for 1 wei — it was really interesting to learn more about the process of building and testing smart contracts. Last word, as it does in <em><strong>The Bottle Imp</strong></em>, goes to the boatswain:</p> <blockquote> <p>“I don’t value any of your talk,” replied the boatswain. “You thought I was a flat; now you see I’m not; and there’s an end. If you won’t have a swallow of the rum, I’ll have one myself. Here’s your health, and good-night to you!”</p> </blockquote> 2017-07-28T00:00:00+00:00 2017-07-28T00:00:00+00:00 Unknown https://clay.fyi/blog/running-selenium-headless-chrome-aws-lambda/ <p>There seems to be a small community of developers building interesting browser things on AWS Lambda with Headless Chrome.</p> <p>Around a month ago, I was also prototyping a small <a href="https://aws.amazon.com/lambda/">AWS Lambda</a> project that enabled running <a href="http://www.seleniumhq.org/projects/webdriver/">Selenium Webdriver</a> with <a href="https://developers.google.com/web/updates/2017/04/headless-chrome">Headless Chrome</a> inside a Lambda function called <a href="https://github.com/smithclay/lambdium">lambdium</a> (disclaimer: no relation to laudanum).</p> <p>I wanted to share the general architecture of the app and some things I learned while building it to help any other developers wanting to give serverless and Headless Chrome a try.</p> <p>There’s also great post <a href="https://medium.com/@kensoh/chromeless-chrominator-chromy-navalia-lambdium-ghostjs-autogcd-ef34bcd26907">by Ken Soh for an overview</a> of other projects experimenting with Headless Chrome and AWS Lambda. <a href="https://news.ycombinator.com/item?id=14859084">The comments in a recent Hacker News post about the Chromeless project</a> are worth a read, too.</p> <h4 id="binaries-not-included">Binaries not included</h4> <p>In brief, AWS Lambda is just an event-driven Linux container on EC2 with some <a href="https://www.kernel.org/doc/Documentation/cgroup-v1/freezer-subsystem.txt">special cgroup sauce</a> — <a href="https://speakerdeck.com/smithclay/faas-measurement-fundamentals">see my Gluecon talk for more on that</a>. Because of that, it’s perfectly acceptable to run almost any non-privileged process you want inside Lambda as you would in a container. You just invoke a 64-bit Linux binary that doesn’t require special privileges using something like Node.js’s <a href="https://nodejs.org/api/child_process.html">child_process module</a> or <a href="https://docs.python.org/2/library/subprocess.html">Python’s subprocess library</a> in code in the main function handler.</p> <p>Complex Linux binaries like chromium, though, need lots of shared libraries and assume a couple things about the Linux environment that don’t play perfectly with Lambda. <a href="https://medium.com/@marco.luethy?source=post_header_lockup">Marco Lüthy</a> came to the rescue in early March and figured out how exactly to get the chromium binary working. His <a href="https://github.com/adieuadieu/serverless-chrome">serverless-chrome</a> project is now the reference implementation for many projects, including <a href="https://github.com/graphcool/chromeless">chromeless</a> and lambdium.</p> <p>With the ability to run Headless Chrome in AWS Lambda, I was curious if I could get <a href="http://www.seleniumhq.org/projects/webdriver/">Selenium</a> tests to run using <a href="https://sites.google.com/a/chromium.org/chromedriver/home">ChromeDriver</a>. Then, using one of the <a href="http://docs.seleniumhq.org/download/">many Selenium client libraries</a>, I could run pre-existing tests in different languages. After many iterations of copying shared libraries around — I <a href="https://gist.github.com/smithclay/7852bfdbf565ba6fe0601b0e2d01b56b">detailed the general recipe for getting untested binaries working on Github</a> — ChromeDriver was able to run and interact with Headless Chrome inside a Lambda function. With ChromeDriver running, I just had to connect a Selenium client library to it.</p> <h4 id="finally-running-selenium-tests">Finally: running Selenium tests</h4> <p>By going through the process above, I was able to get ChromeDriver and Headless Chrome running inside an AWS Lambda function — barely making it under the compressed 50mb size limit for functions (the Chrome binary is around 80% of this).</p> <p>Surprisingly, running the Selenium tests using the Node.js <a href="https://www.npmjs.com/package/selenium-webdriver">selenium-webdriver</a> module ended up being possible with just a few changes to the default options:</p> <ul> <li>There are a bunch of Headless Chrome-specific command line options you need to specify.</li> <li>You have to tell Selenium where to find the Headless Chrome binary (because it’s in AWS Lambda, it’s not in the PATH). With that done, there was a <a href="https://github.com/smithclay/lambdium/issues">mostly working</a> Selenium Webdriver session with Headless Chrome running in AWS Lambda ready to go. The output of a script that visits Google.com and checks the title looks like this in AWS Cloudwatch(again, not much to see because it’s headless after all):</li> </ul> <p><img src="/images/1_8R-I5RmiPeFz1Ozy2cKuOQ.png" alt="Output of a Selenium script running in Headless Chrome on AWS Lambda" /></p> <p><em>Output of a Selenium script running in Headless Chrome on AWS Lambda</em></p> <p>According to the <a href="https://s3.amazonaws.com/lambda-tools/pricing-calculator.html">AWS Lambda calculator</a>, if I ran that same function 1,000 more times, my monthly bill (<em>not</em> including the free tier of the service) would come to $0.04. I increased the function memory to 1152mb—running Chrome takes a lot more than the default memory size in AWS Lambda.</p> <h4 id="project-setup-and-known-issues">Project setup and known issues</h4> <p>The prototype project, which initially tried to design as “framework-less” and as boing as possible (it’s just regular Node.js, a <code>Makefile</code> , and some shell-scripts), is<a href="https://github.com/smithclay/lambdium">up on Github</a>. It shouldn’t be too hard to port to, say, Python, C# or Java (all AWS Lambda-supported languages). As of early January 2018, I migrated the project to use <a href="https://github.com/awslabs/aws-sam-local">AWS SAM Local</a> and in March 2018 <a href="https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:156280089524:applications~lambdium">I ported it the AWS Serverless Application Repository (one click install!)</a>.</p> <p>I’m still working on a new branch that should improve startup times, but that’s a subject for another post.</p> <h4 id="will-headless-chrome-shake-things-up-in-the-browser-qa-test-world">Will Headless Chrome shake things up in the browser QA/test world?</h4> <p>The duo of Headless Chrome and AWS Lambda seems like it has a real possibility to shake up automated browser testing — due in no small part to how cost effective serverless is for low-volume traffic like occasional browser testing in QA. With the generous Lambda free tier, you could probably run low-volume browser tests on AWS infrastructure for free.</p> <p>Also, there’s no reason why Headless Chrome couldn’t also work on other serverless compute platforms, and there seems to be plenty of interest <a href="https://github.com/yukinying/chrome-headless-browser-docker">running it in Docker as well</a>. I give it 6 months, max, before someone writes some sort of Kubernetes cluster that runs Headless Chrome at scale.</p> <p>Think it’s a pretty fun time to be hacking on serverless compute platforms or even containers in general — especially with the recent release of Headless Chrome and the cool things that are going to be built with it.</p> <p>Pull requests and questions welcome. Also highly recommend you check out the <a href="https://medium.com/@kensoh/chromeless-chrominator-chromy-navalia-lambdium-ghostjs-autogcd-ef34bcd26907">other projects in Ken Soh’s post</a>.</p> 2017-05-02T00:00:00+00:00 2017-05-02T00:00:00+00:00 Unknown https://clay.fyi/blog/ssh-ing-into-aws-lambda-functions/ <blockquote> <p>Finally proof that serverless has servers?</p> </blockquote> <p>I spoke at <a href="https://dotscale.io">dotScale</a> in Paris last week <a href="https://speakerdeck.com/smithclay/searching-for-the-server-in-serverless">about some stuff I’ve been working on to learn more about Function-as-a-Service (FaaS) performance</a>.</p> <p>While my talk was mostly about cold starts, function warming, and understanding the internal architecture of FaaS, lots of people were curious how I was able to SSH into a running AWS Lambda function. This post describes how to turn Lambda functions into a short-lived Linux servers with some help from <a href="https://godoc.org/golang.org/x/crypto/ssh">Go’s SSH crypto libraries</a>.</p> <h4 id="adventures-in-missing-dependencies">Adventures in missing dependencies</h4> <p>Many popular FaaS services are built on Linux containers—Azure seems to use IIS voodoo I don’t understand. From an serverless app developer perspective, you write a handler function in a supported language that’s bundled with its dependencies. When a predefined event like an HTTP request occurs, the handler function is invoked inside the container and your code runs (or has an error, or times out).</p> <p>Because the environment is just a container, you’re allowed to execute binaries using standard language features like <a href="https://nodejs.org/api/child_process.html">exec() in node.js</a> or <a href="https://docs.python.org/2/library/subprocess.html">python subprocess</a>. This is also how open-source projects like <a href="https://github.com/apex/apex">Apex</a> let developers run Go in AWS Lambda.</p> <p>With a 12-line lambda function, I tried running the sshd process on a non-privileged port with a host key and configuration I bundled with my the function. This was a bad idea for several reasons:</p> <ul> <li>AWS Lambda’s container environment is missing several libraries that the ssh daemon requires.</li> <li>AWS Lambda functions do not seem to allow any inbound port access (yes, even if running in a VPC with a security group that explicitly allows it).</li> <li>Exotic ssh configuration files are hard to write and filled with weird options.</li> <li>There’s no easy way to automatically determine the external IP address of a Lambda function.</li> </ul> <p>It was time for some SSH tunneling and copy-and-paste Golang code from Github.</p> <h4 id="writing-an-ssh-server-that-creates-a-tunnel-in-go">Writing an SSH server that creates a tunnel in Go</h4> <p>Go has some nice high-level SSH libraries for creating servers, clients, and tunnels. Unfortunately, my first <a href="https://gist.github.com/jpillora/b480fde82bff51a06238">copy-and-paste attempt</a> had a dependency on an OS feature (<a href="https://en.wikipedia.org/wiki/Pseudoterminal">unix pseudoterminals</a>) that did not exist in the AWS Lambda environment. Digging around a bit more in the Go documentation, I found some libraries designed for interacting with terminals that didn’t have a <a href="https://github.com/kr/pty">dependency on the pty package</a>.</p> <p>Cobbling it all together, I had a simple go service that:</p> <ul> <li>Runs an SSH server capable of executing commands in bash on a non-privileged port.</li> <li>Creates a tunnel to a remote host (i.e. my laptop via ngrok or an EC2 host running inside my VPC) and requests a port to be opened that forwards to the SSH server running inside the Lambda function.</li> </ul> <p>I then wrote a Lambda function that:</p> <ul> <li>Is configured with a 5-minute timeout (the maximum allowed value).</li> <li>Executes the Go SSH binary I built for 64-bit Linux (an under-appreciated feature of Go development on macs).</li> <li>Used Lambda environment variables to connect to a specific host and port number (i.e. my laptop via a jump proxy).</li> </ul> <p>With this in place, using the tunnel I can access my go SSH server and run shell commands from another host using a standard SSH client until the Lambda function times out:</p> <p><img src="/images/1_LwTTzR9xRfeelnvPBKdVkw.png" alt="Interacting with the AWS Lambda Environment using faasssh" /></p> <p><em>Interacting with the AWS Lambda Environment using faasssh</em></p> <h4 id="this-isn-t-a-good-idea-but-it-was-fun-to-build">This isn’t a good idea (but it was fun to build)</h4> <p>I don’t think creating SSH tunnels and running random servers inside AWS Lambda is a particularly good way to take advantage of serverless, but it was fun to learn more about the execution environment by building it. Couple other possibilities I’m thinking of now that there’s a proof on concept:</p> <ul> <li>Running other types of servers in Lambda that allow inbound connections through tunneling.</li> <li>Integration with a key-management services for SSH authentication (KMS or Vault perhaps?)</li> <li>Support for concurrent sessions or integration with a TCP load balancer.</li> </ul> <p>The code is up on Github (project name: <a href="https://github.com/smithclay/faassh">faassh</a>). Pull requests and questions welcome.</p> <p>In the next few weeks, I’ll be at<a href="http://www.devopsdaysaustin.com/">DevOpsDays Austin</a> and <a href="http://gluecon.com/">Gluecon</a> speaking more about FaaS if you want to chat in person.</p> <h4 id="links">Links</h4> <ul> <li>“Searching for the Server in Serverless” talk (dotScale Paris 2017)</li> <li>“Serverless Measurement Limits” talk (DevOpsDays Austin)</li> <li>Measuring Cold Starts in AWS Lambda (New Relic Blog Post)</li> <li>faash source code</li> </ul>