Blog

For awhile, devrel was the best work I could do. The world changed, and now it isn’t.

Back in high school, I built robots with a group of exactly the sort of teenagers you’d expect to find in a robotics club. I was physically weaker than most of my peers, so I had to use tools and leverage to complete routine tasks that others could do with their bare hands. This gave me more practice with those tools than others were getting. When the tasks scaled up in difficulty – a stuck bolt turning out to be very stuck, a thicker sheet of metal turning out to be unusually reluctant to bend – my tool use tended to scale up quickly thanks to my constant practice with it, whereas peers who’d been brute forcing the task before would face the initial learning curve of the tool at that moment.

In my present role, I get to work with colleagues of delightfully diverse skill sets. Sometimes I'm learning directly from experts, and other times I get to learn by teaching in areas where I happen to have the local maximum of experience. One delightful colleague has a coding bootcamp background which rapidly exposed them to a lot of important topics, and I find myself helping to fill in the gaps where a couple hours of instruction don't quite get the point across like years of practice.

The Pacific Northwest is dangerously frozen at the moment. Other regions handle conditions like this just fine. But a freeze like this is especially dangerous to us because most people are unprepared for it.

If you're here for pretty pictures of dice, prepare to be disappointed. Making adequate dice is easy, but taking good photos of them exceeds my current skills.

Retroreflectors are fun. I finally got around to picking up some cheap glass blast media today (mine's the 40/70 grit recycled bottle glass from Harbor Freight / Central Pneumatic) and did some testing with various paints and glues that I had lying around. I'm using it as retroreflective beads. When I hear "beads" I think of things with holes in them for putting on a string, but in this case it means more like beads of condensation -- tiny round blobs. They feel gritty like beach sand, and being made from clear glass, they look like unusually sparkly white sand as well.

Over the past couple weeks, my schedule has had a higher than usual concentration of the kind of meetings where one sits off-camera and listens to a presenter talk. Like many engineers who knit in meetings, I find that keeping my hands busy helps me focus. Knitting puts me on the losing side of a battle between "don't drop any stitches" and the laws of physics, however, so instead I've been hand sewing quite a bit.

About a year ago, I found out about lumenators. The theory is that if you put sunshine-ish amounts of light onto a creature, the creature reacts as it would in sunlight.

Some friends showed me a post investigating whether Starlink dishes still work when decorated in various ways. They asked whether I was able to reproduce the results. So I pulled the dish down off my roof and tested it with a few things that I had lying around the house and yard.

How to get rid of the top bar in firefox after installing tree style tab:

Helping a neighbor with a windows update issue today, I explained that asking a Linux admin to use Windows is like asking a Latin speaker to translate a document from Spanish. Most of the concepts are similar enough that they'll be more helpful than a monolingual English speaker, but good guessing is not the same as fluency.

I recently wanted to manually transcribe an audio recording. I prefer to type into LibreOffice Writer for this purpose. Writer has an audio player plugin for transcription, but unfortunately its keyboard shortcuts didn't work when I tried it.

I'm in some channels that are moving from Freenode to Libera.

This time last year, my living room was occupied by a cotton mask production facility of my own devising. I had reverse engineered a leftover surgical mask to get the approximate dimensions, consulted pictures of actual surgeons' masks, and contrived a mask design which was easy-enough to sew in bulk, durable-enough to wash with one's linens, and wearable-enough to fit most faces.

The speakers in my living room are hooked up to a raspberry pi that runs Volumio. It's a nice way to play music from various sources without having to physically reconfigure the speakers between inputs.

Today -- Friday, May 22nd, 2020 -- is within days of my 5-year anniversary with Mozilla, and it's also my last day there for a while. Working at Mozilla has been an amazing experience, and I'd recommend it to anyone.

Turns out that 5 years at a place gets you a bit of a pile of digital detritus. Future me might want notes on what-all steps I took to remove myself from everything, so here goes:

My team has a repo where we keep all our terraform modules, but we had a separate module off in its own repo for reasons that are no longer relevant.

I happen to administer a tiny, mostly-vanilla Minecraft server. The other day, I was playing there with some friends at a location out in the middle of nowhere. I slept in a bed at the base, thinking that would suffice to get me back again later.

I think hypercubes are neat, so I tried to make one out of string to play with. In the process, I discovered that there are surprisingly many ways to fail to trace every edge of a drawing of a hypercube exactly once with a single continuous line.

Or, Stupid Error Of The Day. I'm talking to a GCP's Kubernetes engine through several layers of intermediate tooling, and kubectl is failing:

I took last week off work and attended CCC camp, which was wonderful on a variety of axes. I packed light, but through the week I noted some things it'd be worth packing less-lightly for.

Last year, I wrote about some of the aspirations which motivated my move from Mozilla Research to the CloudOps team. At the recent Mozilla All Hands in Whistler, I had the "how's the new team going?" conversation with many old and new friends, and that repetition helped me reify some ideas about what I really meant by "I'd like better mentorship".

Based on feedback from the crab plushie pattern, I took more pictures this time.

This blog has seen a dearth of posts lately, in part because my standard post formula is "a public thing had a poorly documented problem whose solution seems worth exposing to search engines". In my present role, the tools I troubleshoot are more often private or so local that the best place to put such docs has been an internal wiki or their own READMEs.

I just wrote a thing in a private repo that I suspect I'll want to use again later, so I'll drop it here.

Or, "how to make a selection committee do one of the hard parts of your job as a speaker for you". For values of "hard parts" that include fine-tuning your talk for your audience.

I’ve recently identified some ways in which my former role wasn’t setting me up for career success, and taken steps to remedy them. Since not everybody lucks into this kind of process like I did, I’d like to write a bit about what I’ve learned in case it offers some reader a useful new framework for thinking about their skills and career growth.

Disclaimers: Not all tasks that come to a person in an ops role meet my definition of ops tasks. Advanced ops teams move on from simple problems and choose more complex problems to solve, for a variety of reasons. This post contains generalizations, and all generalizations have counter-examples. This post also refers to feelings, and humans often experience different feelings in response to similar stimuli, so yours might not be like mine.

The Rust Community Team has recently been having a conversation about what a team member's "retirement" can or should look like. I used to be quite active on the team but now find myself without the time to contribute much, so I’m helping pioneer the “retirement” process. I’ve been talking with our subteam lead extensively about how to best do this, in a way that sets the right expectations and keeps the team membership experience great for everyone.

UPDATE: SLACK DECIDED THIS SHOULD NO LONGER BE POSSIBLE AND IT WILL NOT WORK ANY MORE

Somebody asked me recently about what conferences a developer in the pacific northwest looking to attend more FOSS events should consider. Here's an incomplete list of conferences I've attended or hear good things about, plus the approximate times of year to expect their CFPs.

Summary: A user who is otherwise authorized to change their password may get this error when attempting to change their password to a string which violates the Password Policy in your IAM Account Settings.

Agile development's artifact of the daily stand-up meeting is a great idea. In theory, the whole team should stand together (sitting or eating makes meetings take too long) for about 5 minutes every morning. Each person should comment on:

There's an idiom on IRC, and to a lesser extent other more modern communication media, where people indicate interest in performing a real-time conversation with someone by saying "ping" to them. This effectively translates to "I would like to converse with you as soon as you are available".

Some of my IRC friends are job hunting at the moment, so I've been proofreading resumes. These friends are several years into their professional careers at this point, and I've found it really interesting to see what they include and exclude to make the best use of their resumes' space.

The Mozilla All-Hands this week gave me the opportunity to explore an exhibit about the "Mozilla Worldview" that Mitchell Baker has been working on. The exhibit sparked some interesting and sometimes heated discussion (as direct result of my decision to express unpopular-sounding opinions), and helped me refine my opinions on what it means for someone to be "safe" on the internet.

Today I was setting up some new OSX hosts on Macstadium for Servo's build cluster. The hosts are managed with SaltStack.

I saw an excellent question get some excellent infosec advice on IRC recently. I'm quoting the discussion here because I expect that I'll want to reference it when answering others' questions in the future.

A question from a computer science student crossed one of my IRC channels recently:

Here's the text version, with clickable links, of my Automacon lightning talk today.

Some recent flooding in a Freenode channel sent me on a quest to discover whether the network's services were capable of setting a custom message rate limit for each channel. As far as I can tell, they are not.

I adore Git, but have needed to ramp up my Mercurial (Hg) skills recently to dig prior work related to my current tasks out of a repo's history. Here are some things I'm learning:

As has been mentioned on Reddit, the Thinkpad 13 trackpoint settings aren't in the same place as those of older thinkpads. Despite some troubleshooting, I haven't yet found what files to edit to adjust the trackpoint's speed and sensitivity in Ubuntu 16.04.

In setting up virtualenvs for my slides and blog repos on my new laptop, I've been reminded that a variety of Sphinx-based tools require system dependencies as well as the ones in their virtualenvs.

Check out this post by Lucy Bain about how to come up with an idea for what to talk about at a conference. I blogged last year about how I turn abstracts into talks, as well. Now that the SeaGL CFP is open, it's time to look in a bit more detail about the process of going from a talk idea to a compelling abstract.

I recently got a Thinkpad 13 to try replacing my X230 as a personal laptop. Here's the relevant specs from my order confirmation:

This morning, I participated in the O'Reilly Emerging Languages Webcast with my "Rust from a Scripting Background" talk. Here's how it went.

Although its front page claims we've deprecated persona, it's the only way to log into the statusboard and Air Mozilla. For a long time, I was unable to log into any site using Persona from Firefox 43 and 44 because of an error about my browser not being configured to accept third-party cookies.

I made a Rustacean. He's cute. You can make one too.

The short answer: No.

Servo has some Salt configs, hosted on GitHub, for which changes are smoke-tested on TravisCI before they're deployed. Travis only shows the first 10k lines of log output, so I want to minimize the amount of extraneous information that the states print.

The first symptom that I had made a terrible mistake showed up in an Ansible playbook:

Related to this bug, the Vagrant Ansible provisioner seems to ignore some system settings.

Mozilla uses Vidyo for virtual meetings across distributed teams. If it doesn't work on your laptop, you can use the mobile client or book a meeting room in an office, but neither of those solutions is optimal when working from home.

The Rust community, being founded and enjoyed by a variety of Mozilians, seems to have inherited the tradition of tracking top-level progress metrics using are we sites.

Today, I copied an existing command from a Buildbot configuration and then modified it to print a date into a file.:

I recently got the "Hey, you're a Rust Person!" question of how to install notty and interact with it.

I was recently asked an excellent question when I promoted the LFNW CFP on IRC:

More SEO-bait, after tracking down an poorly documented problem:

If you're using open source as a portfolio to make yourself a more competitive job candidate, it can feel like you have to start your own project to show off your skills.

Today I've learned a few things aout how stunnel works. The main takeaway is that Googling for specific errors in the stunnel log is incredibly unhelpful, resulting in a variety of mailing list posts with no replies. Tracking an error message through the source of the program doesn't lead to any useful comments, either. So here's some SEO bait with concrete troubleshooting suggestions.

Today, I posed a question to some professional UI and UX designers:

I'm examining a Linode account in order to figure out how to switch the application its instances are running to AWS. The first challenge is that instance types in the main dashboard are described by arbitrary numbers ("UI Name" in the chart below), rather than a statistic about their resources or pricing. Here's how those magic numbers line up to hourly rates and their corresponding monthly price caps:

It's the day before Thanksgiving here in the US, and the time of year when we're culturally conditioned to be a bit more public than usual in giving thanks for things.

$ sudo apt-get install docker
$ which docker
$ docker
The program 'docker' is currently not installed. You can install it by typing: 
apt-get install docker
$ apt-get install docker
Reading package lists... Done
Building dependency tree       
Reading state information... Done
docker is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.

Oh, you wanted to run a docker container? The docker package in Ubuntu is some window manager dock thingy. The docker binary that runs containers comes from the docker.io system package.

I just got a good question from a friend on IRC: "Should I ask my university's administration to install Rust on our shared servers?" The answer is "you don't have to".

Today I noticed an assumption which was making my life unnecessarily difficult: I assumed that if my .travis.yml said language: ruby on the first line, I was supposed to only run Ruby code from it.

Update: I'm now maintaining the issue aggregator list at http://edunham.net/pages/issue_aggregators.html

Today, the website's build broke. We made no changes to the tests, yet a wild dependency error emerged:

As well as nominally helping organize the event, I attended and spoke at SeaGL 2015 this weekend. The slides from my talk are here.

Here are some quick notes on upgrading Buildbot.

This scenario is simplified for purposes of demonstration.

I've recently been working on an Ansible role that applies to both Ubuntu and OSX hosts. It has some dependencies which are only needed on OSX. There doesn't seem to be a central document on all the options available for solving this problem, so here are my notes.

At the bottom of the Apache 2.0 License file, there's an appendix:

The screen on my X1 Carbon gave out after a couple months, and my loaner laptop in the meantime is an X240.

I was a student employee at the OSU Open Source Lab, on and off between internships and other jobs, for 4 years. Being part of the lab helped shape my life and career, in almost overwhelmingly positive ways. However, the farther I get from the lab the more clearly I notice how being part of it changed the way I form expectations about my own technical skills.

The videos from RustCamp are available here.

It was a lazy Sunday afternoon and I wanted to play Don't Starve. This actually ended up meaning about 3 hours of intermittent troubleshooting and 1 hour of games, because Linux.

I'm using search.mibbit.com to count these. All have at least one user in them as of 4pm PST 2015-07-31.

People sometimes say "morning" or "evening" on IRC for a time zone unlike my own. Here's a bash one-liner that emits the correct time-of-day generalization based on the datetime settings of the machine you run it on.

The office printers have instructions for setting them up under Windows, Mac, and Ubuntu. I had forgotten how to wrangle printers, since the last time I had to set up new ones was half a decade ago when I first joined the OSL.

At the end of the day on July 14th, 2015, the certificate that Rust's buildbot slaves were using to communicate with the buildmaster expired. This broke things. The problem started at midnight on July 15th, and was only fully resolved at the end of July 16th. Much of the reason for this outage's duration was that I was learning about Buildbot as I went along.

Many "free" wifi hotspots give you a limited time per computer. If you're traveling light and forgot to bring extra devices, it's easy to give a Linux laptop multiple personalities:

Rust by Example has a little box where readers can interact with some example Rust code, run it using the playground, and see the results in the page. As a sysadmin I'm loath to recommend that anybody trust the playground for anything, but as a nerd and coder I recognize that it's super cool and people want to use it.

One of many questions facing the Rust infrastructure team right now is "What's our packaging situation?". We don't have a centralized source of information on what version of Rust is available in which systems' package managers, and we don't even know where to find that information.

On April 16th, I threw an application toward this job posting on careers.mozilla.org. I doubted whether I'd be qualified, but I reminded myself that most people apply to jobs where they meet only 80% of the critiera. I could, with some creative redefinition ("of course an internship at Intel is a year of relevant experience!"), meet every listed criterion. So I applied, since the worst they could say was no.

UPDATE: The ticket has been claimed by a local student! Thanks everyone for helping boost the signal about this.

Today I got a seemingly ordinary request from a community member who volunteers a build slave for Rust's buildbot:

The rust-lang.org web site used to be hosted on GitHub Pages. This gave it excellent uptime and made deploying changes easy, but did not support HTTPS.

Today, the security alias for a site I administer got an automated message pointing out that we lacked a DMARC record. Here's what I learned about how to set up and test them.

Since I'm curious about how often files are downloaded from S3, I enabled logging on the buckets serving them and directed the logs into a bucket which I created to hold them. Then I wanted to move everything on that logs bucket to my local machine, so I could poke around in the logs and ascertain the best way to turn them into useful information.

Some spammers got onto the Mozilla network, scraped a major channel's user list, and PMed everybody requests to join their network from almost 1,000 different nicks. Here's how I tidied up afterwards.

One of the more egregious inconsistencies in Rust's architecture is that play.rust-lang.org lives on an Arch box, while everything else is Ubuntu. Before the team has a dedicated operations person, the argument for using Arch was that playpen comes pre-packaged for it, whereas one has to build it oneself on Ubuntu.

I've been running Arch on my work laptop and it's pretty much working. However, I have a nice external monitor on my desk, and I keep having to manually configure the output to it with arandr. Here's how I made it configure itself by default when X starts.

Although I currently expect that I'll end up choosing Salt for work, I've gotten nerdsniped by the apparent simplicity and power of Ansible. Since I'm trying to make a habit of narrating my first encounters with various tools, here's a short novel of 0 through cloning a repo.

Let's just say that it's pretty clear why my team at Mozilla decided to hire an operations specialist when they did. For the infrastructure which supports the Rust programming language, I get the relatively rare (compared to just hacking on an existing deployment) privilege of deploying configuration management from the ground up.

Today I learned that there's a trick to getting sound to work on Arch using recordmydesktop.

I got a shiny new Thinkpad X1 Carbon 3rd Gen for my new job. It came with Windows pre-installed. Out of morbid curiosity and willingness to consider giving this shiny new allegedly-less-terrible Win8 thing a chance, I booted it up into the default Windows installation before wiping everything to install arch.