The System

In transitioning to involuntarily remote companies, I noticed a decline in the quality of my work-related social life compared to when I was at in-person or intentionally remote organizations. The intentionally remote culture of Mozilla was built around twice-yearly week-long offsites for the entire organization, which served in part as summer-camp-like friendship accelerators. In-person workspaces offer richer and clearer signaling about social interest: “we’re going to lunch, wanna join us?” grows into invitations to mutually interesting events outside of work as common interests are discovered.

I like working with my friends, and many other people do too. To get more of that, I’ve trial-and-errored my way into a system of making friends at work that’s working well for me so far.

Identify Likeminded Individuals

First, I taught myself to notice when I’ve met someone at work whose company I enjoy. Are meetings more interesting when they weigh in with their perspectives? Do we get to chatting about ordinary topics when we cross paths in larger meetings, and feel disappointed instead of relieved when the conversation returns to the meeting’s actual agenda? Are they interesting to speak with socially when we meet at in-person work events? Is there something about their work, or their approach to work, which I find especially interesting and want to learn more about? Do they express reluctance or regret at ending conversations about topics which interest us both, beyond the mere rote pleasantries?

In short, if we worked in the same office and I saw that they were hanging out in a common area, would their presence make me more likely to go hang out in that area too? These are all ways that my brain tells me someone might be a good candidate to upgrade from acquaintance to friend. Other brains may signal the same potential through a different set of feelings; the trick to this step is discovering what your pattern-match on promising individuals feels like.

If you’re recreating my system for yourself, it would probably be helpful to reflect on other times when you’ve developed friendships with colleagues. What was different about interacting with them, versus interacting with the comparable colleagues who didn’t end up as friends? That’s what you’re looking for.

At this step of the process, being on the same team as me counts as a big enough shared interest to make the system worth trying.

Check Compatibility

To find out how others feel, I rely heavily on the words that they say. Some people are better at picking up subtler cues, but in my experience, even relying on tone of voice can be hit-or-miss.

People can say one thing while meaning another for many reasons, like manners and people-pleasing, and can have wonderful friendships despite that trait! They just don’t tend to have those wonderful friendships with me, because the more we interact socially, the more likely I am to harm them by taking actions which are appropriate based on the content of their speech but inappropriate based on its context.

In the same vein, I like to wait on aggressively pursuing friendship with someone until I’ve seen how they say “no” to optional tasks they don’t want to do. Once I’ve seen them say “no” when a “yes” would have been easier, I know they’re the kind of person who will tell me to back off if I’m being annoying or burdensome.

Those are just my personal compatibility checks. Yours might be similar, or they might be opposite – you might prefer people who will notice subtle changes to your mood without being told, for instance!

Ask the Awkward Question

Once I identify a prospective work friend and verify that their social habits are compatible with mine, I ask them if they’d like to meet more often. This sounds like the most awkward thing ever, but it’s actually not so bad.

The best time to ask if they’d like to meet more often is when we’re already having a good conversation for some other reason. When they are clearly enjoying a conversation with me, it’s unambiguous that they at least sometimes enjoy my company, so it’s a perfect moment to open an invitation to make interactions more frequent.

The question goes something like this: “I wish we did this more often! How would you feel about having a recurring 1:1 chat? Maybe every few weeks?”

The frequency depends on your schedule and how much overlap there is in your work. I have some work friends that I catch up with quarterly, others monthly, others fortnightly.

Set Expectations

I like to set a recurring calendar invite for 1:1s, and be sure that the invitee has “modify event” permissions on it.

My event description usually reads something like, “If we worked in the same office, I would enjoy chatting with you in the kitchen from time to time. But we’re remote, so a call is the next best thing. Please feel free to delete or reschedule this meeting any time it would make your day more difficult.”

I don’t personally use agendas for these meetings. If developing an agenda together is a fun social activity to share with a new friend, though, there’s nothing wrong with that!

Leave a Graceful Exit Open

I don’t like worrying about whether I’m pressuring someone to hang out when they’d prefer not to, so if they cancel several times, I check in about the cadence. “It seems like your schedule has been packed recently; how about I cancel this invite? If you have more time later and want to meet, I’d still be interested!”

For the first few 1:1s with someone, I make sure to have something scheduled to start immediately after the calendar invite’s end time, even if it’s just an alarm on my phone. I know that when I get really into a fun conversation, I can lose track of time and then have a more stressful day as a result. I don’t want to make my friends’ days worse, so as the instigator, I take ownership of ending at the agreed end time as a default.

Some people turn out to tolerate or even prefer these meetings running long, sometimes very long. One work friend who shares my interests in career strategy and loves mentoring will enjoy letting our 30 minutes become 2 hours if we’re both free. Regardless, I consider it polite to check in at the scheduled end time and see when we each need to be someplace else.

In The Meeting

So, you’ve got this person in a call. What do you say?

I personally ask some variant of “how are you doing?” or “how is it going?”. This is their excuse to start talking about whatever is on their mind.

Sometimes people will reflexively respond “good, and you?”. The “and you?” is my invitation to lay out several options for conversation topic, like trying different fishing lures to see what’s of interest today.

To prepare for the meeting, I like to think about 1-2 personal things and 2-3 work things which seem like they might interest the person. Personal things might be “I’ve finally figured out how to get my cat to tolerate existing in the same room as me”, or “I’m looking forward to finally planting tomatoes in my garden”, or “I’m finally getting around to watching such and such a show”. Professional things might be “I’m super heads down in such and such a project”, or “I just found out that this change is coming so now I get to figure out how that impacts my other work”, or similar.

Answering “how are you?” with concise references to 1 personal thing and 1-2 work things gives the person a ton of options. They can request more information on something that they would enjoy hearing about, or offer a related story of their own, or change the topic to something they find more compelling than anything I brought up.

Then again, I don’t hate small talk like some people do. I find that small talk is a challenging game of identifying common interests, almost like wiki-racing. When someone shares information about the weather, it’s a thinly veiled offer to share more about their hobbies and interests: What is it too hot, or too cold, or just right to do? I hold the opinion that small talk is the bootloader for interesting conversation.

Another excellent topic of conversation is to lend and borrow expertise. One of my work friendships is with a product designer, and part of the fun of talking to her is how our areas of greater and lesser expertise line up so compatibly. She answers my “why is it like this?” inquiries, while I answer her “what’s it like to actually use this thing?”, to great mutual benefit.

Afterwards

If you had a nice time, do it again, the next time it shows up on your calendar at a time that works for you both! If you didn’t have a nice time, change plans as necessary so that next time will be better.

If a work friend or even just a friendly collaborator moves on to another opportunity, I like to ask them who else in their part of the organization I should know. Then I can reach out to those people: “such and so told me you’re a good person to know; would you mind a quick call?”. The topic of conversation for that call is pre-established: What you both like about the person who’s leaving, and what mutually beneficial involvements you had with them that you’d like to replace. Replacement is most politely framed as “I’ll really miss how…”, but that’s basically an offer to invite them into your little part of the vacuum left by the departing individual.

I try to remember to intermittently ask the “who else should I know?” question of my work friends, but I rarely remember to. It’s ultimately not essential while someone is still a colleague, but it’s a very important aspect of a good goodbye.

Hi, Future Me

I hope that someday I’ll look back on this post and go “wow, you sure were doing things the difficult and inefficient way!”. If there’s better ways of accomplishing the outcomes of this system that are working for you, please don’t regress to my way of doing things.

But several people in the past few weeks have expressed admiration at the level of connectedness I’ve built within my present workplace, so it seemed worth writing up my current state of the art on the topic.

Git Concepts

• Know where you are. Care constantly about what branch you are presently on, because many commands make changes relative to your current location.
• Speak with pedantic precision about precisely what you’re referring to. When referring to somewhere you might push to or pull from, a description has 3 parts: the location or owner (a github org, a github user, or your local checkout), the repository name, and the branch name. If I say “there’s cake in the kitchen”, you’ll have to guess which kitchen I mean: mine? yours? the office? Similarly, if you say “on the main branch”, I’ll have to guess from context which main branch: your laptop? your fork of the repo on github? the upstream repository you forked from?
• Commit IDs do not matter to you, but they matter to other people. If you’re the only person who has a commit, you can safely take actions (like pull –rebase) which change its ID. But as soon as anybody else has that commit, it is very rude to do anything to that commit which changes its ID.
• There is a social distinction between expectations for a kitchen and a dining room, a factory floor and a showroom, a studio and an art gallery, which parallels the distinction between your fork of a repository versus the origin. Your fork may be shared publicly, but it’s the kitchen or factory or studio, where nobody should be surprised if they find half-completed or not-yet-working code. The origin, on the other hand, is the dining room or showroom or gallery – some but not all products of the kitchen or factory or studio pass a quality control process to be put on display and appreciated by the public.
• Version control gives you time travel powers. To use these powers, you have to learn to identify moments that you’ll want to travel back to. If you already have an intuition for when to quicksave in a video game, use it. Every commit is a quicksave, and you get an unlimited number of them. If you already have an intuition for how to name your save files in a game so that you can go back to the one you wanted, use that for crafting helpful commit messages. As with any challenge of organization, think about the situation you’ll be in when you later wish to use the commit you’re making now. What will future-you search for when trying to find this commit?
• To take useful notes for yourself, it’s important to be more pedantically precise than you’d expect. Recording what commands you used isn’t enough – in order to use the notes later, you must specify in a way that makes sense to you exactly what task the commands accomplished. If it seems like there’s another way to do the task, ask why that way wouldn’t work. Sometimes the other way would work fine, and other times there’s some horrible gotcha that would break it.

Unix Concepts

• Know where you are. Care constantly about which directory you are currently in, and where you have put your various files.
• When someone spells out a command verbally including flags, do not expect that they’ll want you to type a space after the “dash” or “tac”. Sometimes there’ll be a bare - with no space afterwards, but that’s so unusual that a speaker will pause longer or even verbalize the space. It’s easy to assume that a listener knows the format of a tool’s arguments, but why would they if they aren’t acquainted with it yet?
• Changes made in one shell session will not automatically propagate to others. The point of the config file (~/.bashrc, ~/.zshrc, etc) is to get each new shell up to speed on what you expect it to know – not unlike how you might add common requests to every prompt you give an AI. (the config file is just a list of commands that run automatically every time a shell is opened)
• It’s all files. Turtles all the way down. I’m starting to understand why my own education started at “everything’s a file” – that information wasn’t useful in the moment, but it created a priming effect where I approaced everything with the expectation that it’s a file afterwards. The first questions to ask when troubleshooting are questions that you can ask about a file. You cannot ask all that many questions about files: Where is it? What’s in it? What are its permissions? Addressing those fundamentals catches annoyingly many problems.

Incomplete list, caveat emptor, and you get what you pay for on the anachronism of a free blog. But hey, it’s human-written text, and that’s an endangered species of critter on the modern web. I suspect we’ll eventually develop good tools for threshing the human-generated prospective training data from the early-GPT-generated chaff of the 2024 web, so hello to whatever future model ends up training on this!

Mental Model

Here’s how I think about bead-type retroreflectors. Physics means that the glass beads basically bounce the light straight back toward where it came from, so the stick figure is holding the flashlight close to its eyes for maximum testing effectiveness over short distances. The beads are obviously not to scale.

The beads have a different index of refraction from the surrounding air. That just means that light bends at a certain angle when it goes from air to glass, and it bends at that angle again when it goes from glass to air on its way out.

The surface of the bead that’s covered in paint reflects light, perhaps only some of the light depending on the color of the paint. White paint reflects the most light, of course.

The light goes into the bead, bends a little, bounces off the paint, goes almost straight back through the bead the way it came, bends a little when it hits the air, and ends up going more or less straight back toward the light source.

I’ve drawn the light in orange before it bounces off the paint and in yellow after it bounces off, to kind of illustrate how it behaves differently hitting the painted floor when it’s gone through a retroreflective bead versus when it’s just gone through the air.

Storing the blast media

The one problem with even 5lbs of ground glass is that it does its utmost to get everywhere. If the bottle is ever inverted with the lid on and the lid gets bumped, the sand-like glass ends up everywhere the next time you take the lid off.

Plastic bottles like this come with a “safety” seal when I buy food stuff like spices in them, and I’ve never had a problem with those getting all up in the cap and spilling everywhere.

I tried ironing a scrap of mylar food packaging (a fruit snacks bag was most conveniently accessible) onto the plastic bottle, and it stuck nicely! I used my clothes iron on the Cotton setting. Now the beads will stay in the jar until I want to use them again.

Paint/Glue Testing

I put various sorts of paint and glue onto popsicle sticks and sprinkled them with the blast media. I then put them on the door of a room with good blackout curtains, took a photo, turned out the lights, and took a flash photo.

I used a commercial plastic retroreflector in the first two photos as a control for how well the DIY ones were working.

Here’s how it looked with the lights on:

From top to bottom, the samples are:

• Dark green puff paint / fabric paint
• Black puff paint / fabric paint
• Superglue
• modpodge
• clear nail polish
• dark red nail polish
• 1 coat white nail polish, then half white half clear nail polish with beads on the second coat
• white nail polish
• black nail polish
• black acrylic paint
• white acrylic paint
• metallic gold-ish paint stuff

Everything except the two-coat enamel test had the glass beads poured directly onto the one layer of the paint or glue. Yes, I labeled modpodge upside down. I wasn’t planning to take photos at first but then I realized if I was going to post about it I probably should.

Here’s how that setup looks in a flash photo from maybe 10’ away, in a dark room:

Here’s how it looks from the same distance, still a dark room, but I zoomed in more:

And here’s how it looks, still zoomed in, but with the commercial reflector removed:

Conclusions

The cheapest glass sandblast media that I could get ahold of works surprisingly well as a retroreflector when applied to white paint. The performance difference between beads on white and beads on darker colors was surprising. The clear glues (superglue and mod podge) worked better than the darker colored paints, which was a bit surprising.

My biggest surprise was how the white sample almost blended in with the white door in ambient lighting, yet stood out so much in the dark.

Further stuff to test with this includes doing the glue thing on a darker background color, and trying various types of clear topcoat over the bead layer. I don’t think those will work, but then again I did think the black paint might work, so it’s worth testing.

Sourcing materials

The plywood, paint, and romex were left over from other projects.

I had a tough time sourcing affordable e26 sockets. Apparently people don’t build ridiculous light fixtures from scratch on the cheap very much? I ended up using adapters meant to help you use e26 bulbs in GU24 sockets. The pins meant to interface with the GU24 socket were easy to solder wires onto later, and the cylindrical shape of the adapters made them easy to friction fit into holes in the plywood.

I made a big spreadsheet of price per lumen of e26 LED bulbs available various places, ruled out those whose specs were incomplete or self-contradictory, and got the cheapest bulbs that seemed likely to add up to Too Much Light.

The switches, 6-gang switchplate, and electrical plug came from Habitat For Humanity.

Build Process

I chose the size based on what plywood I had around. On the plywood, I drew a grid of 96 rectangles. I drilled holes in alternating rectangles of the grid, to maximize space between bulbs, because I was frightened of making the kind of mistake that would cause the whole thing to overheat and catch on fire.

The holes were drilled with a bit just a hair smaller than the GU24 adapter sockets, so that the sockets would be held in place by friction. A few holes came out too small for the sockets, but were easily enlarged with a rasp.

I also rasped off any noticeable splinters. If I’d been doing fancy carpentry, I would have sanded the whole thing at this point. I wasn’t, so I didn’t.

I painted the board white, for maximum light reflection. I thought about trying to do some kind of mirror or chrome finish behind the bulbs, but that seemed way too hard. If I was building it again, I would look for a plastic mirror and secure it to the plywood before drilling the holes. Or I might try gluing a sheet of mylar onto the wood after drilling the holes but before installing the sockets.

The sockets installed flush with the board, by pushing them through the holes from the front side. I made sure to install all of them in the same orientation for each row, though this was actually unnecessary because alternating current doesn’t care about polarity. To get the sockets flush with the board, I found that it was easiest to fit a row in and then place the whole board face-down on a flat surface and walk on it between the sockets. That way everything ended up even with the floor. If you don’t like walking on your projects, you could probably achieve a similar effect by carefully hitting it with a hammer.

I then wired the rows of lights to the switches. 3 of the switches control 12 bulbs each, and 2 control 6 bulbs each, because this seemed like the best compromise between using all 5 available switches and getting it to work with the limited quantity of romex that I had available.

I test fit all the wires to make sure I had enough, then soldered them onto the pins of the GU24 adapters. Since I needed to strip off 1” of wire insulation from the middle of a wire a bunch of times, I built a tool for the job, by gluing razor blades to each side of the jaws of a pair of clothes pins. That way I’d just clip the tool onto the cable where I needed to strip it, spin it around so the blades cut through the insulation, and cut off the inch of insulation with another knife.

You’ll notice in the photos that I secured the switches by screwing them directly to the switch plate instead of putting them in a box. You’re normally supposed to put switches in boxes, and I might regret this shortcut someday, but I didn’t have a 6-gang box and it’s been fine so far.

Mind-numbing amounts of soldering later, I tested the assembly to make sure nothing was shorted, then powered it on to make sure it could light. Before installing it on the wall, I covered all the exposed wiring in electircal tape, mostly because I dislike exposed wires. Also, safety. But you’d get just as zapped from connecting the poles of any of those exposed sockets on the front as you would by connecting a pair of wires on the back, so worrying about hiding exposed conductors on the back but not the front is mostly aesthetic.

I then affixed it to the wall above my desk, using a piece of 2x4 at each short end of the plywood to hold everything well away from the wall. I left the top and bottom open to improve air circulation in case it ran into thermal problems, but in a year of use it hasn’t caught on fire.

It looks underwhelming in photos, because very bright lights are hard to photograph. This shows how I us it on an ordinary day like today, with a set of 12 5000K bulbs and a set of 6 3000K bulbs lit and the rest remaining off.

Costs

If you built one of these today, it’d cost you:

• about $20 for a 2’x4’ chunk of plywood
• about $40 for a pack of 50 GU24 to E26 adapters
• about $160 for 48 2500-lumen e26 bulbs
• about $30 for 15 feet of 12 gauge 2-wire romex
• maybe $10 for switches and a 6-gang plate at your local Habitat For Humanity?

So you’d be looking at around $260 for the whole build, unless you had materials on hand or found better deals on materials.

Was it worth it?

I like having my false sun in my office, because I feel like being in bright light helps my brain and body agree that it’s actually daytime. It also helps me fake a summer sleep schedule when the weather outside suggests wintry hibernation. I notice that being in natural daylight at dusk (even when dusk is at 6pm!) causes me to feel sleepy, whereas being in midday-sunlight light levels circumvents that process.

The false sun was somewhat inconvenient to build and install, and would not be accessible for everyone. I was able to make it because I have woodworking and soldering equipment already available, and because I was able to screw it directly to the wall of my office without penalty.

If I was renting my home, I would have mounted the lightbulb array in a bookcase or other tall piece of furniture instead of attaching it onto the wall. If I was building one of these for use in a rental, I’d probably keep the holes-in-plywood scheme for supporting the lamp sockets, but I would size it to mount over the top couple shelves of an Ikea Billy or similar cheap bookcase, including some room for airflow.

For other locations in my house, I’ve become partial to the 5,000-lumen 4’ LED “shop” lights that my local Harbor Freight sometimes has on sale for $20 apiece. I find them trivially easy to install and plenty bright to send most of that “it’s daytime!” signal. However, they’re over twice as expensive per lumen compared to the DIY version.

You probably shouldn’t build exactly what I did, as it’s bulky and inconvenient and not all that aesthetically pleasing. But I hope this project demonstrates that it’s perfectly achieveable to create a light fixture brighter than the sun! When playing with electricity, please learn what you’re doing beforehand, be mindful that alternating current can kill you, and generally use common sense. If you try to build or modify something like this without having a basic idea of electrical safety, you’re just sticking a fork in an outlet but with extra steps.

Methodology Notes:

I gathered data in my camera roll: a snapshot of the setup, followed by a screencap of the “router <-> internet” pane of the speedtest within the Starlink Android app. I left the dish set up in the same spot, which was at ground level but reported no obstructions when I did the sky scan thing in the app. I power cycled the dish once during the experiment, roughly halfway through the control tests for the “is it slower?” question, because I needed to plug something else into the extension cord that it was using for a minute.

I’m located south of the 45th parallel.

Will It Send?

For my first batch of tests, I was curious whether I could get any data in and out via Starlink with various configurations of stuff on and around the dish. I only ran one speedtest each for these.

I started by suspending things over Starlink, because I knew some of the materials would get heavy (such as wet cotton cloth) and I was scared of hurting the little motors inside that it uses to move itself around. I draped a canvas tarp over the north side of a fence to make an impromptu photo studio for the dish. The dish had an unobstructed view of the sky upward and to the north. I placed metal folding chairs to the east and west of the dish, with their backs closest to the dish, to hold up the various covers. I used some old brake pads to weigh down the things that I draped over the chairs so they wouldn’t blow away.

1. No cover, control to make sure Starlink works in this position. It sends. 146 Mbps down, 15 Mbps up.
2. Covered with clear-ish greenhouse plastic. It sends. 148 Mbps down, 15 Mbps up.
3. Same as (2) but with a cotton bedspread over the plastic. It sends. 170 Mbps down, 8 Mbps up.
4. Same as (3) but with the bedspread soaking wet. It does not send. The app reports “offline, obstructed”.
5. Plastic from (2) plus a single layer of corrugated cardboard box. It sends. 62 Mbps down, 9 Mbps up.
6. Same as (5) but I dumped some water on the box. It mostly ran off. It sends. 65 Mbps down, 9 Mbps up.
7. Double layer of row cover, like you put on plants in the garden to keep the frost from hurting them. It sends. 94 Mbps down, 11 Mbps up.
8. Single layer of ratty old woven plastic tarp. It sends. 109 Mbps down, 6 Mbps up.

For the final 2 tests, I got a bit braver about putting things directly on the dish, instead of just suspending them above it.

9. Dish in a black plastic trash bag like you’re throwing it out. It sends. 147 Mbps down, 10 Mbps up.
10. Same tarp as from (8), but directly on the dish instead of hanging above it. It sends. 207 Mbps down, 20 Mbps up.

Is It Slower?

As you’ll notice in the numbers that I was getting in the above trials, sometimes covering the dish yields a faster single speed test than having it exposed. That seems wrong. I suspect that this is normal variance based on what satellites are available, but I don’t actually know what behavior is normal. So I did a few more tests in configuration 10, and a bunch of tests in configuration 1, to make sure we’re not in a timeline where obstructing a radio link somehow makes it perform better.

I left it all out overnight, and the canvas tarp that I’d been using as a visual backdrop blew off of the fence so it was covering the dish. I made a video call on the connection with the canvas over the dish and didn’t notice any subjective degradation of service compared to what I’m accustomed to getting from it.

Conclusions

Starlink definitely still works when I put dry textiles between the dish and the satellite. It doesn’t seem to matter if the textiles are directly on the dish or suspended a couple feet above it.

I’m surprised that Eric Kuhnke’s Starlink worked with 2 layers of wet bedsheet over it. The only case in which I managed to cause my Starlink to report itself as being obstructed was when I had a wet bedspread suspended in the air over it. I don’t really want to start piling wet cloth directly on the dish, though, because wet cloth is heavy and I’m scared of overloading the actuators.

Starlink gets kind of warm during normal operation, as demonstrated by the cats. I wouldn’t want to leave mine in a dark colored trash bag or under a dark colored tarp on a sunny day in case it overheated. And there’s no need to – if you don’t want people to know you’re using one, you can just suspend an opaque and waterproof tarp above it and there’ll be better airflow around the dish itself and thus less risk of overheating.

Have fun!

Volumio as a pulseaudio output

For awhile, my laptop was able to treat Volumio as just another output device, based on the following setup:

• the package pulseaudio-module-zeroconf was installed on the pi and on every laptop that wants to output audio through the living room speakers
• the lines load-module module-zeroconf-publish and load-module module-native-protocol-tcp were added to /etc/pulseaudio/default.pa on the pi
• the line load-module module-zeroconf-discover was added to /etc/pulse/default.pa on my Ubuntu laptop
• pulseaudio was restarted on both devices after these changes (pulseaudio -k to kill, pulseaudio to start it)

starting pulseaudio on boot on Volumio

And then as long as the laptop was connected to the same wifi network as the pi, it Just Worked. Until, in the course of troubleshooting an issue that turned out to involve the laptop having chosen the wrong wifi, I power cycled the pi and it stopped working, because pulseaudio was not yet configured to start on boot.

The solution was to add the following to /etc/systemd/system/pulseaudio.service on the pi:

[Unit]
Description=PulseAudio system server

[Service]
Type=notify
Exec=pulseaudio --daemonize=no --system --realtime --log-target=journal
User=volumio
ExecStart=/usr/bin/pulseaudio

[Install]

And then enabling, starting, and troubleshooting any failures to start:

systemctl --system enable pulseaudio.service
systemctl --system start pulseaudio.service
systemctl status pulseaudio.service -l # explain what went wrong
systemctl daemon-reload # run after editing the .service file

Thanks to Rudd-O’s blog post, which got me 90% of the way to the “start pulseaudio on boot” solution. Apparently systemctl started caring more about having an ExecStart directive since that post was written, which meant I had to inspect the resulting errors, which means I’m writing down the resulting tidbit of knowledge so that I can find it again later.

future work

Nobody in my household has yet found a good way to persuade the Windows computer who lives under the TV to speak pulseaudio yet. If I ever figure that out, I’ll update here.

Why go?

When I try to predict what my 2025 or 2030 self will wish I’d done with my career around now, I anticipate that I’ll want access to opportunities which build on a background of technical leadership and mentoring junior engineers.

It wouldn’t be impossible to create these opportunities within Mozilla, but from talking with trusted mentors both inside and outside the company, I’ve concluded that I would get a lot more impact for the same effort if I was working within a growing organization.

As a mature organization, Mozilla’s internal leadership needs are very different from those of a younger and more actively growing company. There’s a far higher bar at Moz for what it takes to be the best person for a task, because the saturation of “best people” is quite high and the prevalence of entirely new tasks is relatively low in comparison. Technical leadership here seems to often require creating a need as well as filling it. At a growing organization, on the other hand, the types and availabilities of such opportunities are very different.

I’m especially looking forward to leveling up on a different stack in my next role, to improve my understanding of the nuances of the underlying problems our technolgies address. I think it’s a bit like learning a second language: only through comparing and contrasting multiple solutions to the same sort of problem can one understand what traits corrolate to all those solutions’ strengths versus which details are simply incidental.

Why now?

I’ll be the first to admit that May 2020 is a really strange time to be changing jobs. But I have an annual tradition of interviewing at several places, learning what their stacks and cultures and unique fractals of tech debt look like, and then turning down an offer or two because changing roles would be a step backwards for both my career development and overall quality of life.

Shortly before the global conference circuit ground to a halt along with everything else, I started interviewing from a DevOps Advocate position, just to explore what it might look like to turn my teaching hobby into a day job. By the time those interviews were complete, the tech evangelism space had been turned inside out and was rapidly reinventing itself, and the skills that qualified me for the old world of devrel were looking less and less like the kind of expertise that might be needed to succeed in the new one. However, a SRE from the technical interviews suggested that I interview for her team, and upon taking that advice I discovered an organization that keeps most of the stuff I loved about Mozilla while also offering the other opportunities that I was looking for.

As with anywhere, there are a few aspects of my new role that I suspect may not be as great yet as where I’m leaving, but these areas of improvement look like things that I’ll be able to have some influence over. Or at least there’ll be room to push the Overton Window in a good direction!

Want more details on the new role? I’ll be writing more about it after I start on June 1st!

Mathematical materials

To construct a toy hypercube or any other graph, you need the graph. To make it from a single piece of string, every vertex should have an even number of edges.

Knowing the number of edges in the graph will be useful later, when marking the string.

Physical materials

For the edges of the toy, I wanted something that’s a bit flexible but can sort of stand up on its own. I found that cotton clothesline rope worked well: it’s easy to mark, easy to pin vertex numbers onto, and sturdy but still flexible. I realized after completing the construction that it would have been clever to string items like beads onto the edges to make the toy prettier and identify which edge is which.

For the vertices, I pierced jump rings through the rope, then soldered them shut, to create flexible attachment points. This worked better than a previous prototype in which I used flimsier string and made the vertices from beads.

Vertices could be knotted, glued, sewn, or safety pinned. A bookbinding awl came in handy for making holes in the rope for the rings to go through.

Mathematical construction

First, I drew the graph of the shape I was trying to make – in this case, a hypercube. I counted its edges per vertex, 4. I made sure to draw each vertex with spots to write numbers in, half as many numbers as there are edges, because each time the string passes through the vertex it makes 2 edges. So in this case, every vertex needs room to write 2 numbers on it.

Here’s the graph I started with. I drew the edges in a lighter color so I could see which had already been visited when drawing in the euler cycle.

Then I started from an arbitrary vertex and drew in the line. Any algorithm for finding euler paths will suffice to draw the line. The important part of tracing the line on the graph is to mark each vertex it encounters, sequentially. So the vertex I start at is 1, the first vertex I visit is 2, and so forth.

Since the euler path visits every vertex of my particular hypercube twice, every vertex will have 2 numbers (the one I started at will have 3) when I finish the math puzzle. These pairs of numbers are what tell me which part of the string to attach to which other part.

Here’s what my graph looked like once I found an euler cycle in it and numbered the vertices that the cycle visited:

Physical construction

Since my graph has 32 edges, I made 33 evenly spaced marks on the string. I used an index card to measure them because that seemed like an ok size, but in retrospect it would have been fine if I’d made it smaller.

I then numbered each mark in sequence, from 1 to 33. I numbered them by writing the numbers on slips of paper and pinning the papers to the rope, but if I was using a ribbon or larger rope, the numbers could have been written directly on it. If you’re doing this at home, you could mark the numbers on masking tape on the rope just as well.

The really tedious step is applying the vertices. I just went through the graph, one vertex at a time, and attached the right points on the string together for it.

The first vertex had numbers 1, 25, and 33 on it for the euler cycle I drew and numbered on the graph, so I attached the string’s points 1, 25, and 33 together with a jump ring. The next vertex on the drawing had the numbers 2 and 18 on it, so I pierced together the points on the string that were labeled 2 and 18.

I don’t think it matters what order the vertices are assembled in, as long as the process ultimately results in all the vertices on the graph being represented by rings affixing the corresponding points on the string together.

I also soldered the rings shut, because after all that work I don’t want them falling out.

That’s all there is to it!

I’m going to have to find a faster way to apply the vertices before attempting a 6D hypercube. An ideal vertex would allow all edges to rotate and reposition themselves freely, but failing that, a lighter weight string and crimp fasteners large enough to hold 6 pieces of that string might do the trick.

The finished toy is not much to look at, but quite amusing to try to flatten into 3-space.

Clothing

The site is dusty, extremely hot through the day, and quite cold at night. Fashion ranges from “generic nerd” to hippie, rave, and un-labelably eccentric. There is probably no wrong thing to wear, though I didn’t see a single suit or tie. A full base layer and a silk sleeping bag liner improve comfort at night. A big hat, or even an umbrella, offers protection from the day star.

I was glad to have 3 pairs of shoes: Lightweight waterproof sandals for showering, sturdier sandals for walking around in all day, and boots for early mornings and late nights. I saw quite a few long coats and even cloaks at night, and their inhabitants all looked very comfortably warm.

Doing sink laundry was more inconvenient at camp than for ordinary travel, and I was glad to have packed to minimize it.

A small comfortable bag, or large pockets in every outfit, are essential for keeping track of one’s wallet, phone, map, and water bottle.

I occasionally found myself wishing that I’d brought a washable dust mask, usually around midafternoon when camp became one big dust cloud.

Campsite Amenities

Covering a tent in space blankets makes it look like a baked potato, but keeps it warm and dark at night and cool through early afternoon. Space blankets are super cheap online, but difficult to find locally.

For a particularly opulent tent experience, consider putting a doormat outside the entrance as a place to remove shoes or clean dusty feet before going inside. I improvised a doormat with a trash bag, which was alright but the real thing would have been nicer to sit on.

Biertisch tables and benches are prevalent around camp, so you can usually find somewhere to sit, but it doesn’t hurt to bring a camp chair of the folding or inflatable variety. Inflatable stuff, from furniture to swimming pools, tended to survive fine on the ground.

I was glad to have brought a full sized towel rather than a tiny travel one. A shower caddy or bag to carry soap, washcloth, hair stuff, and clean clothes would have been handy, though I improvised one from another bag that I had available.

String and duct tape came in predictably handy in customizing my campsite.

Electronics

DECT phones are very fun at camp, but easy to pocket dial with. This is solved by finding the lock feature on the keypad, or picking a flip phone. I was shy about publishing my number and location in the phonebook, but after seeing how helpful the directory was for people to get ahold of new acquaintances for important reasons, I would be more public about my temporary number in the future.

Electricity is a limited resource but sunlight isn’t. Many tents sport portable solar panels. For those whose electronics have non-European plugs, a power strip from home is a good idea.

I packed a small headlamp and used it pretty much every day. Even with it, I found myself occasionally wishing that I’d brought a small LED lantern as well.

A battery to recharge cell phones is good to have as well, especially if you don’t run power to your tent. A battery can be left charging unattended in all kinds of public places where one would never leave one’s phone or laptop.

Food

Potable water is free, both still and sparkling. Perhaps I’ve been spoiled by the quality of the tap water at my home, but I wished that I’d brought water flavorings to mask the local combination of minerals.

I brought a small medical kit, from which I ended up using or sharing some aspirin, ibuprofin, antihistamines, and lots of oral rehydration salt packets.

Meals were available for free (with donations gratefully accepted) at several camps for everyone, and at the Heaven kitchen for volunteers. There were also a variety of food carts with varyingly priced dishes. The food carts outside the gates in front of the venue were good for an icecream or fresh veggie snack, which were harder to find within camp.

Savory meals and all kinds of drinks were everywhere, but there didn’t seem to be any place nearby to just pick up straight chocolate. Small, nonperishable snacks like that are worth getting at a grocery before arrival, since they’re not readily available on the grounds.

Other

If any of the special skills that your nerd friends ask you for help with require tools, bring them. I happen to always carry a needle and thread when traveling, and ended up using them to repair a giant inflatable computer-controlled sculpture.

A hammock, and something to shade it with, came in very handy and would be worth bringing again. There were lots of trees, and it might have been entertaining to set up a slackline for passers-by to fall off of, but I don’t think it’d be worth the weight of carrying one internationally.

Night time is basically a futuristic art show as well as a party. There’s no such thing as too much electroluminescent wire or too many LEDs, whether for decorating your camp or yourself. As a music party, it’s also extremely loud, so I was glad to have brought earplugs. Comfortable earplugs also improve sleep; music goes till 3 or 4 AM in many places and early risers start making noise around 8 or 9.

Camp has a lake, in which it’s popular to float large inflatable animals, especially unicorns. I saw more big inflatable unicorn floaties being used around camps as extra seating than being used in the lake, though.

There’s a railway that goes around camp, and sometimes runs a steam train. I won’t say you should rig a little electric cart to fit its rails and drive around on it, but somebody did and looked like they were having a really wonderful time.

Bikes, and lots of folding bikes, were everywhere. Scooters, skateboards, and all sorts of other wheeled contrivances, often electric, were also prevalent. The only rolling transportation that I didn’t see at all around camp were roller blades and skates, because the ground is probably too rough for them.

I ran out of stickers, and wished I’d brought more. I didn’t see as many pins as some conferences have.

A small notebook also came in handy. Each day, I checked both the stage schedule and the calendar to find the official and unofficial events which looked interesting, and noted their times on paper. It was consistently convenient to have a means of jotting down notes which didn’t risk running out of battery. Flipping through the book afterwards, about 1/4 of its contents is actually pictures I drew to explain various concepts to people I was chatting with, a few pages are daily schedule notes, and the rest is about half notes on things that presenters said and half ideas I jotted down to do something with later.

I was glad to have brought cash rather than just cards, not only for food but also because many workshops had a small fee to cover the cost of the materials that they provided.

Camp Advice

Nobody even tries to maintain a normal sleep schedule. People sleep when they’re tired, and do stuff when they aren’t. Talks and events tend to be scheduled from around noon to around midnight. I don’t think it would be possible to attend camp with a rigorous plan for what to every day and both stick to that plan and get the most out of the experience.

In shared spaces, people pick the lowest common denominator of language – at several workshops, even those initially scheduled to be held in German, presenters proactively asked if any attendees needed it to be in English then switched to English if asked. Behind-the-scenes, such as in the volunteers’ kitchen, I found that this was reversed: Everyone speaks German, and only switches to give you instructions if you specifically ask for English. Plenty of attendees have no German at all and get along fine.

Volunteer! If something isn’t happening how it should, fix it, or ask “how can I help?”. Volunteering an hour or two for filing badges or washing dishes is a great way to make new friends and see another side of how camp works.

Mentorship across industries

Mentors from outside my own industry are valuable for high level perspectives, and for advice on general life and human topics that aren’t specialized to a single field. Additionally, specialists in other industries often represent the consumers of my own industry’s products. Wise and thoughtful people who share little to none of my domain knowledge can provide constructive feedback on why my industry’s work gets particular reactions from the people it affects – just as someone who’s never read a particular book before is likely to catch more spelling errors than its own author, who’s been poring over the same manuscript for many hours a day for several years.

However, for more concrete problems within my particular career (“this program is running slower than expected”, or even “how should I describe that role on my resume?”), observers from outside of it can rarely offer a well tested recommendation of a path forward.

Mentorship across companies within an industry

Similarly, mentors from other companies within my own industry are my go-to source of insight on general trends and technologies. A colleague in a distant corner of my field can tell me about the frustrations they encountered when using a piece of technology that I’m considering, and I can use that advice to make better-informed choices in my daily work.

But advice on a particular company’s peculiarities rarely translates well across organizations. A certain frequency of reorganization might be perfectly ordinary at my company, but a re-org might indicate major problems at another. This type of education, while difficult to get from someone at a different company, is perfectly feasible to pick up from anyone on another team within one’s own organization.

Mentorship across teams within a company

When I switched roles, I had trial-and-errored my way into the observation that there’s a large class of problems with which mentors from different teams within the same company cannot effectively help. I’d tentatively call these “junior engineer problems”, as having overcome their general cases seems to correlate strongly to seniority. In my own expeience, honing the improvement of code-adjacent skills such as the intuition for what problems should be effectively solvable from the docs versus when and whom to ask for help, how deeply to explore a prospective course of action before committing to it, and when to write off an experiment as “effectively impossible”, are all questions whose answers one derives from experience and observing expert peers rather than from just asking them with words.

Mentorship across projects or specialties within a team

I had assumed that simply being on the same team as people capable of imparting that highly specialized variant of common sense would suffice to expose me to it. However, my first few projects on my new team have clearly shown, in both the positive and the negative cases, that working on the same project as an expert is far more useful to my own growth than simply chancing to be bureaucracied into the same group.

The negative case was my first pair of projects: The migration of 2 small, simple services from my team’s AWS infrastructure to GCP. Although I was on the same team as experts in this process, the particular projects were essentially mine alone, and it was up to me to determine how far to proceed on each problem by myself before escalating it to interrupt a busy senior engineer. My heuristics for that process weren’t great, and I knew that at the outset, but my bias toward asking for help later than was optimal slowed the process of improving my ability to draw that line – how can one enhance one’s discrimination between “too soon”, “just right”, and “too late” when all the data points one gathers are in the same one of those categories?

Mentorship within a project

Finally, however, I’m in the midst of a project that demonstrates a positive case for the type of mentorship I switched teams to seek. I’m in the case labeled A on the diagram up above – I’m working with a more-experienced teammate on a project which also includes close collaboration with members of another team within our organization. In examining why this is working so much better for me than my prior tasks, I’ve noticed some differences: First, I’m getting constant feedback on my own expectations for my work. This is no serious nor bureaucratic process, but simply a series of tiny interactions – expressions of surprise when I complete a task effectively, or recommendations to move on to a different approach when something seems to take too long. Similarly, code review from someone immersed in the same problem that I’m working on is indescribably more constructive than review from someone who’s less familiar with the nuances of whatever objective my code is trying to achieve.

Another reason that I suspect I’m improving more quickly than before in this particular task is the opportunity to observe my teammate modeling the skills that I’m learning in his interactions with our colleagues from another team (those in position C on that chart). There’s always a particular trick to asking a question in a way that elicits the category of answer one actually wanted, and watching this trick done frequently in circumstances where I’m up to date on all the nuances and details is a great way to learn.

The FOSS loophole

I suspect I may have been slower to notice these differences than I otherwise might have been, because the start of my career included a lot of fantastic, same-project mentorship from individuals on other teams, at other companies, and even in other industries. This is because my earliest work was on free and open source software and infrastructure. In FOSS, anyone who can pay with their time and computer usage buys access to a cross-company, often cross-industry web of professionals and can derive all the benefits of working directly with mentors on a single project. I was particularly fortunate to draw a wage from the OSU Open Source Lab while doing that work, because the opportunity cost of a hours spent on FOSS by a student who also needs to spend those hours on work is far from free.

Materials required

• About 1/4 yard or 1/4 meter of orange fabric. Maybe more if it’s particularly narrow. Polar fleece is good because it stretches a little and does not fray near seams.
• A measuring device. You can just use a piece of string and mark it.
• Scissors, a sewing machine, pins, orange thread
• Scraps of black and white cloth to make the face
• The measurements of the hat wearer’s head. I’m using a hat to guess the measurements from.
• A pen or something to mark the fabric with is handy.

Constructing the pattern pieces

If you’re using polar fleece, you don’t have to pre-wash it. Fold it in half. In these pictures, I have the fold on the left and the selvedges on the right.

The first step is to chop off a piece from the bottom of the fleece. We’ll use it to make the legs and spines later. Basically like this:

Next, measure the circumference you want the hat to be. I’ve measured on a hat to show you.

Find 1/4 of that circumference. If you measured with a string, you can just fold it, like I folded the tape measure. Or you could use maths.

That quarter-of-the-circumference is the distance that you fold over the left side of the big piece of fabric. Like so:

Leave it folded over, we’ll get right back to it. Guesstimate the height that a hat piece might need to be, so that we can sketch a piece of the hat on it. I do this by measuring front to back on a hat and folding the string, I mean tape measure, in half:

Back on the piece we folded over, put down the measurement so we make sure not to cut the hat too short. That measurement tells you roughly where to draw a curvy triangle on the folded fabric, just like this:

Now cut it out. Make sure not to cut off that folded edge. Like this:

Congratulations, you just cut out the lining of the hat! It should be all one piece. If we unfold the bit we just cut and the bit we cut it from, it’ll look like this:

Now we’re going to use that lining piece as a template to cut the outside pieces. Set it down on the fabric like so:

And cut around it. Afterwards you have 1 lining piece and 2 outer pieces:

Now grab that black and white scrap fabric and cut a couple eye sized black circles, and a couple bits of white for the light glints on the eyes. Also cut a black D shape to be the mouth if you want your hat to have a happy little mouth as well as just eyes.

Construction

Put the black and white together, and sew an eye glint kind of shape in the same spot on both, like so:

Pull the top threads to the back so the stitching looks all tidy. Then cut off the excess white fabric so it looks all pretty:

Now the fun part: Grab one of those outside pieces we cut before. Doesn’t matter which. Sew the eyes andmouth onto it like so:

Now it’s time to give the hat some shape. On both outside pieces – the one with the face and also the one without – sew that little V shaped gap shut. Like so:

Now they look kind of 3D, like so:

Let’s sew up the lining piece next. It’s the bit we cut off of the fold earlier. Fold then sew the Vs shut, thusly:

Next, sew most of the remaining seam of the lining, but leave a gap at the top so we can turn the whole thing inside out later:

Now that the lining is sewn, let’s sew 10 little legs. Grab that big rectangular strip we cut out at the very beginning, and sew its layers together into a bunch of little triangles with open bottoms. Then cut them apart and turn them inside out to get legs. Here’s how I did those steps:

Those little legs should have taken up maybe 1/3 of big rectangular strip. With the rest of it, let’s make some spines to go across Ferris’s back. They’re little triangles, wider than the legs, sewn up the same way.

Now put those spines onto one of the outside hat pieces. Leave some room at the bottom, because that’s where we’ll attach the claws that we’ll make later. The spines will stick toward the face when you pin them out, so when the whole thing turns right-side-out after sewing they’ll stick out.

Put the back of the outside onto this spine sandwich you’re building. Make sure the seam that sticks out is on the outside, because the outsides of this sandwich will end up inside the hat.

Pin and sew around the edge:

Note how the bottoms of the spines make the seam very bulky. Trim them closer to the seam, if you’re using a fabric which doesn’t fray, such as polar fleece.

The outer layer of the hat is complete!

At this point, we remember that Ferris has some claws that we haven’t accounted for yet. That’s ok because there was some extra fabric left over when we cut out the lining and outer for the hat. On that extra fabric, draw two claws. A claw is just an oval with a pie slice misisng, plus a little stem for the arm. Make sure the arms are wide enough to turn the claw inside out through later. It’s ok to draw them straight onto the fabric with a pen, since the pen marks will end up inside the claw later.

Then sew around the claws. It doesn’t have to match the pen lines exactly; nobody will ever know (except the whole internet in this case). Here are the front and back of the cloth sandwich that I sewed claws with:

Cut them out, being careful not to snip through the stitching when cutting the bit that sticks inward, and turn them right-side out:

Now it’s time to attach the liner and the hat outer together. First we need to pin the arms and legs in, making another sandwich kind of like we did with the spines along the back. I like to pin the arms sticking straight up and covering the outer’s side seams, like so:

Remember those 10 little legs we sewed earlier? Well, we need those now. And I used an extra spine from when we sewed the spines along Ferris’s back, in the center back, as a tail. Pin them on, 5 on each side, like little legs.

And finally, remember that liner we sewed, with a hole in the middle? Go find that one real quick:

Now we’re going to put the whole hat outer inside of the lining, creating Ferris The Bowl. All the pretty sides of things are INSIDE the sandwich, so all the seam allowances are visible.

Rearrange your pins to allow sewing, then sew around the entire rim of Ferris The Bowl.

Snip off the extra bits of the legs and stuff, just like we snipped off the extra bits of the spines before, like this:

Now Ferris The Bowl is more like Ferris The Football:

Reach in through the hole in the end of Ferris The Football, grab the other end, and pull. First it’ll look like this...

And then he’ll look like this:

Sew shut that hole in the bottom of the lining...

Stuff that lining into the hat, to make the whole thing hat-shaped, and you’re done!

Find the command that yielded the error

First, I identify what tool I’m trying to use. Ops tools are often an amalgam of several disparate tools glued together by a script or automation. This alias invokes that call to SSH, this internal tool wraps that API with an appropriate set of arguments by ascertaining them from its context. If I think that SSH, or the API, is having a problem, the first troubleshooting step is to figure out exactly what my toolchain fed into it. Then I can run that from my own terminal, and either observe a more actionable error or have something that can be compared against some reliable documentation.

Wrappers often elide some or all of the actual error messages that they receive. I ran into this quite recently when a multi-part shell command run by a script was silently failing, but running the ssh portion of that command in isolation yielded a helpful and familiar error that prompted me to add the appropriate key to my ssh-agent, which in turn allowed the entire script to run properly.

Make sure the version “should” work

Identifying the tool also lets me figure out where that tool’s source lives. Finding the source is essential for the next troubleshooting steps that I take.:

$ which toolname
$ toolname -version #

I look for hints about whether the version of the tool that I’m using is supposed to be able to do the thing I’m asking it to do. Sometimes my version of the tool might be too new. This can be the case when the dates on all the docs that suggest it’s supposed to work the way it’s failing are more than a year or so old. If I suspect I might be on too new a version, I can find a list of releases near the tool’s source and try one from around the date of the docs.

More often, my version of a custom tool has fallen behind. If the date of the docs claiming the tool should work is recent, and the date of my local version is old, updating is an obvious next step.

If the tool was installed in a way other than my system package manager, I also check its README for hints about the versions of any dependencies it might expect, and make sure that it has those available on the system I’m running it from.

Look for interference from settings

Once I have something that seems like the right version of the tool, I check the way its README or other docs looked as of the installed version, and note any config files that might be informing its behavior. Some tooling cares about settings in an individual file; some cares about certain environment variables; some cares about a dotfile nearby on the file system; some cares about configs stored somewhere in the homedir of the user invoking it. Many heed several of the above, usually prioritizing the nearest (env vars and local settings) over the more distant (system-wide settings).

Check permissions

Issues where the user running a script has inappropriate permissions are usually obvious on the local filesystem, but verifying that you’re trying to do a task as a user allowed to do it is more complicated in the cloud. Especially when trying to do something that’s never worked before, it can be helpful to attempt to do the same task as your script manually through the cloud service’s web interface. If it lets you, you narrow down the possible sources of the problem; if it fails, it often does so with a far more human-friendly message than when you get the same failure through an API.

Trace the error through the source

I know where the error came from, I have the right versions of the tool and its dependencies, no settings are interfering with the tool’s operation, and permissions are set such that the tool should be able to succeed. When all this normal, generic troubleshooting has failed, it’s time to trace the error through the tool’s source.

This is straightforward when I’m fortunate enough to have a copy of that source: I pick some string from the error message that looks like it’ll always be the same for that particular error, and search it in the source. If there are dozens of hits, either the tool is aflame with technical debt or I picked a bad search string.

Locating what ran right before things broke leads to the part of the source that encodes the particular assumptions that the program makes about its environment, which can sometimes point out that I failed to meet one. Sometimes, I find that the error looked unfamiliar because it was actually escalated from some other program wrapped by the tool that showed it to me, in which case I restart this troubleshooting process from the beginning on that tool.

Sometimes, when none of the aforementioned problems is to blame, I discover that the problem arose from a mismatch between documentation and the program’s functionality. In these cases, it’s often the docs that were “right”, and the proper solution is to point out the issue to the tool’s developers and possibly offer a patch. When the code’s behavior differs from the docs’ claims, a patch to one or the other is always necessary.

Why choosing an audience is hard

Deciding who you’re speaking to is one of the trickiest bits of writing an abstract, because a good abstract is tailored to bring in people who will be interested in and benefit from your talk. One of the reasons that it’s extra hard for a speaker to choose the right audience, especially at a conference that they haven’t attended before, is because they’re not sure who’ll be at the conference or track.

Knowing your audience lets you write an abstract full of relevant and interesting questions that your talk will answer. Not only do these questions show that you can teach your subject matter, but they’re an invaluable resource for assessing your own slides to make sure your talk delivers everything that you promised it would!

Tricks for choosing an audience

Some strategies I’ve recommended in the past for dealing with this include looking at the conference’s marketing materials to imagine who they would interest, and examining the abstracts of past years’ talks.

Make the committee choose by submitting multiple proposals

Once you narrow down the possible audiences, a good way to get the right talk in is to offload the final choice onto the selection committee! A classic example is to offer both a “Beginner” and an “Advanced” talk, on the same topic, so that the committee can pick whichever they think will be a better fit for the audience they’re targeting and the track they choose to schedule you for.

If the CFP allows notes to the committee, it can be helpful to add a note about how your talks are different, especially if their titles are similar: “This is an introduction to Foo, whereas my other proposal is a deep dive into Foo’s Bar and Baz”.

Use the organizers’ own words

I always encourage resume writers to use the same buzzwords as the job posting to which they’re applying when possible. This shows that you’re paying attention, and makes it easy for readers to tell that you meet their criteria.

In the same way, if your talk concept ties into a buzzword that the organizers have used to describe their conference, or directly answers a question that their marketing materials claim the conference will answer, don’t be afraid to repeat those words!

When choosing between several possible talk titles, keep in mind that any jargon you use can show off your ability, or lack thereof, to relate to the conference’s target audience. For instance, a talk with “Hacking” in the title may be at an advantage in an infosec conference but at a disadvantage in a highly professional corporate conf. Another example is that spinning my Rust Community Automation talk to “Life is Better with Rust’s Community Automation” worked great for a conference whose tagline and theme was “Life is Better with Linux”, but would not have been as successful elsewhere.

Good luck with your talks!

Managers as Mentors

Oddly enough, I can trace the series of causes and effects that ultimately culminated in my decision to switch roles straight back to a company that I’ve never even worked at. By shaping my manager’s leadership skills, Microsoft’s policies have trickled down into benefits to my open source focused career at Mozilla.

Apparently, managers at Microsoft have targets to meet for their reports’ career growth and advancement, as well as the usual product and performance metrics. I’ve heard that a manager there who meets all their other deliverables but fails to promote the people under them can be demoted for that negligence! I’ve learned about this culture because it showed up in my own ex-Microsoft manager as a refusal to take “I’m happy where I am and don’t think I particularly need to go seek a promotion” as a complete answer when he asked about my goals for career growth and progression.

I have many peers whose managers seem fine with “I’m happy where I am” as an answer to career goal questions. Replacing an employee for their lower-level duties as they graduate to higher-level ones can be between inconvenient and almost impossible for a boss, so it can seem to be in everybody’s best interests to leave someone for a long time when they find a role where they’re content. But my own manager’s ingrained distaste for allowing that kind of stagnation led to a series of conversations that forced me to look at the bigger picture of my career, and realize the ways that sticking with a role that’s “good enough for now” could cause me serious problems years down the road.

Complacency and the Same Year of Work

I’ve been warned over and over by mentors through the industry that it’s a terrible idea to do the “same year of work” over and over. I’ve even repeated that advice to others – “don’t tolerate a job that isn’t helping you grow! Keep moving roles till you find something you love!”. This advice is especially easy to follow when the “same year of work” you’re repeating is unpleasant in some way.

However, I found myself in a role at Mozilla Research where the “same year of work” is totally amazing! 3 times in a row, I came into a team with major ops needs, and delivered value in a way that I’m great at to a team of awesome people with expertise in areas where my knowledge is only superficial. And it feels great to help them out and solve problems with knowledge that’s old hat for me but novel to my colleagues. For each role, my progress eventually reached a point where I’d solved all the problems that I could fix in a timely manner – I’d hit the edge of my skills at the time. Although it’s educational, working outside my skill set eventually slowed down my performance and work quality to the point where it was better to have me shift focus to a group whose needs were within what I could execute rapidly and with high quality.

The parts of this cycle in which I’m attempting a load of tasks primarily outside my skill set (and often outside the set of skills that I have the prerequisite skills to bootstrap myself into in a timely manner) can be stressful, but on the whole it’s been a rewarding experience. I’ve felt like I’ve gotten a “new role” every year or so, on about the timeline that a peer starting with similar skills to my own might have had to switch companies to find new opportunities. Without my manager’s guidance, I might have continued in “same year of work” cycle indefinitely, but our conversations about career progression helped me improve my understanding of how I personally grow as an engineer.

Self-Teachable Skills

What’s that “set of skills that I have the prerequisite skills to bootstrap myself into in a timely manner” that I mentioned earlier? They’re those challenges which I can identify and frame in a way which lets me ask the kind of questions that get actionable answers. They’re knowledge gaps directly adjacent to my established areas of expertise, about which I can concisely frame questions whose answers I can use to accomplish a task. Despite being a “team of one”, I’ve successfully self-taught skills such as learning a new configuration management system and improving my FOSS community management knowledge to solve particular challenges. Since I was already more skilled than the average entry-level engineer in those areas when I started my solo ops role, I had good heuristics for framing questions in a way that was likely to yield the answers I wanted, and evaluating the credibility and applicability of sources to learn from.

In other words, I’ve found that I can be successful at self-teaching a skill when I can write out a template for what I know the correct answer to a given question will look like. I might not know precisely which config management system will meet my needs, but I already know a variety of pain points to look out for and how to assess whether an example given in a tool’s docs is similar to the task I’m looking to apply the tool on. I might not know exactly what community management strategy is best for a given FOSS challenge, but I know enough about community management in general to frame a question about what strategy to use with all the relevant information and propose several viable options to ask an expert for their opinions in selecting between.

Non-Self-Teachable skills

I’ve found that, in working as a team of one, I’ve done pretty well at growing the skills that I had the appropriate foundation to “bootstrap” or “self-teach”. However, there are other skills for which this hasn’t been the case. Although these types of skills can be self-taught very slowly and inefficiently, I’m going to nickname them “non-self-teachable” for purposes of this discussion. “Infeasible-to-self-teach”, while technically more accurate, is just too much of a tangle to type out.

For skills with whose problem space I’m not fully acquainted, I find it extremely challenging to improve in a timely manner in isolation. In areas like prioritizing work across projects, and project planning, I find that my ability to self-teach is limited when I don’t have sufficient expertise to ask for help efficiently. When I don’t know what sort of answer would be actionable, I struggle to frame general questions and figure out where to ask them. And when I lack the expertise to identify what specific tasks I should be asking for help to improve at, I miss many opportunities to seek help.

How did I discover these “unknown unknowns” of skills that I lack the foundation to teach myself efficiently? I started noticing them when seeking resources to understand my career growth options in order to refine my career goals. In an internal wiki, I found a document which gives broad outlines for and descriptions of the competence and impact of employees at each pay grade. Mozilla’s is called a Job Family Architecture; other companies have similar documents with different names. Comparing my performance to the descriptions of higher levels helped me identify and articulate some particular skill areas that I haven’t improved at the same rate as those where I’ve successfully self-taught. Quantifying these areas that I’d like to work on has helped me figure out what changes I could make to get into circumstances more like those where I’ve improved rapidly at the “problem” skills in the past.

Bootstrapping Non-Self-Teachable Skills

When an engineer enters their very first job or internship, or touches a programming language for the first time, it’s reasonable to assume that all the skills they need to build are in the category that I’m describing as non-self-teachable. Skills I’m currently expert at, and even teach to others, started out as non-self-teachable for me as well. So what changed? How did I go from having no idea what questions to ask about Git, to being able to solve even its gnarliest problems with only a handful of StackOverflow queries or man page checks?

When I look back at my career, the common denominators among times I’ve built any skill from non-self-teachable to self-teachable have been peers and mentors. Watching the way that a teammate addresses a challenge and comparing it to the way I would have approached it gives me new insights at a rate that no amount of struggling with the skill by myself ever could. And managers or mentors who are subject matter experts in my field can compare the way I approach a task to the way they would have, and point out differences to yield skill-improving feedback. When I work closely enough with an expert for a while , I build a mental model of the way they think, and for the rest of my career I can ask myself “How would they have architected this? How would they have tackled this problem?”.

These mental models can simulate a team in circumstances that are similar to those where I worked with the experts, but as I advance into novel work in isolation, the models become less and less useful because I can’t predict how the expert would have approached a task unlike anything I’ve seen them encounter.

Bridging My Skill Gaps

Once I identified my skill gaps, I first attempted to seek better mentorship in my existing role. After several different attempts, I determined that a mentor’s intimacy with the particular task they’re giving feedback on is integral to their ability to help me refine a technical skill. In both individual and group mentorship, I found it easy to refine skills that I could already self-teach, but between difficult and impossible to get good help on skills about which I wasn’t yet expert enough to frame good questions. Doing my best to improve within my existing role helped me figure out which skills it was reasonable to try to improve in place, and which others were infeasible to build in a timely manner under the circumstances.

I talked with my manager about what I’d learned about myself from the various attempts to build the skills I regard as needing work. We evaluated our options for putting me into a situation where I had the peers and mentorship that I’m looking for: What would it look like to change my current role so it had more peers? What would it look like to move me into a role on a team of ops folks elsewhere? When we took into account some other skill gaps that I’m interested in addressing, such as working with experts on infrastructure at larger scale, we concluded that the best step to address my current concerns was to explore my options for shifting to a different team.

At this point, I literally wrote down a list of relevant skills in my notebook. I brainstormed a section of skills where I feel I’m below where I’d like to be and would learn best from peers. I also outlined the skills I’ve succeeded with in my present role, and the skills at which I consider myself above average and thus am not worried about aggressively growing while I focus on improvement in other areas.

I summarized my lists into the 2 key reasons I’m interested in a role change. Those reasons are: I want to work with peers and mentors who can offer detailed technical feedback based on their expertise at the problems I’m solving, and I want to refine my prioritization and planning skills by being more closely exposed to good examples in work like mine.

As I sought other teams that might meet these needs for me and interviewed with them, I kept my lists and summary on my desk. I felt that having them in sight made a real difference in my ability to clearly articulate my interest in a role change, and helped me ask the right questions to determine whether any team looked like a good match for my priorities. As I interviewed, I paid attention to not only the technical topics that we discussed, but the prospective colleagues’ attitude toward the answers I got wrong. On the team I ended up selecting, I was especially impressed by the way that my future peers ended each question by filling in any gaps between my answer and the complete or best possible answer that they were “looking for”. This made the interviews feel like a constructive conversation, and even if I hadn’t taken the role, I would have left with a better understanding of the technologies we discussed.

New Team

With all that, I’m excited to announce that I’m transitioning from Research to the Cloud Ops team within Mozilla! This team supports a multitude of projects, and there are always at least 2 ops engineers in a “buddy system” for every project they support. The new role is similar to my old one in that it juggles supporting many projects at once, but very different in that I’ll be working directly with expert colleagues to learn their best ways to do it.

And yeah, I’m staying at Mozilla. It might net me more cash to jump ship to another company, but monetary compensation is not what this move is optimizing for. The drawbacks I would experience if I chose a team at another company include greater uncertainty about what the team is actually like, and having to re-learn all the specialized bureaucracy that comes with onboarding anywhere. I also encounter very few other companies whose cultures are as closely aligned with my own values, expecially pertaining to open source, and none of them currently have openings that I can confirm are as good a match to my current skill building goals as the Cloud Ops team.

Visualization

Drawing a picture is one of my favorite ways to gain control of and ask the right questions about new knowledge. Here’s an ugly little chart into which I’ve thrown a handful of skill areas and my approximate levels at them before and after my old role… and of course, the skill emphases that have been important to me in picking the right place to go next.

If you want to make a similar chart, I made that one with the net chart type in LibreOffice.

Why join a team?

When I joined the Rust community subteam, its purpose was defined vaguely. It was a small group of “people who do community stuff”, and needed all the extra hands it could get. A lot of my time was devoted explicitly to Rust and Rust-related tasks. The tasks that I was doing anyways seemed closely aligned with the community team’s work, so stepping up as a team contributor made a lot of sense. Additionally, the team was so new that the only real story for “how to work with this team and contribute to its work” was “join the team” . We hadn’t yet pioneered the subteams and collaboration with community organizers outside the official community team which are now multiplying the team’s impact.

Why leave?

I’m grateful to the people who have the bandwidth and interest to put consistent work into participating on Rust’s community team today. As the team has grown and matured, its role has transitioned from “do community tasks” to “support and coordinate the many people doing those tasks”. I neither enjoy nor excel at such coordination tasks. Not only do I have less time to devote to Rust stuff, but the community team’s work has naturally grown into higher-impact categories that I personally find less fulfilling and more exhausting to work on.

Teams and people change

In a way, the team’s growth and refinement over the years reminds me of a microcosm of what I saw while working at a former startup as it built up into an enterprise company. Some peoples’ working style had been excellently suited to the 5-person company they originally joined, but clashed with the 50-person company into which that startup grew. Others who would never have thrived in a company of only 10 people were hiring on and having a fantastic impact scaling the company up to 1,000. And some were fine when the company was small and didn’t mind being part of a larger organization either. That experience reminds me that the fit between a person and organization at some point in the past does not guarantee that they’ll remain a good fit for each other over time, and neither is necessarily to blame for the eventual mismatch as both grow and change.

Does leaving harm anyone?

When you’re appreciated and valued for the work you do on a team, it’s easy to get the idea that the team would be harmed if you left. The tyres on my bike are a Very Important Part of the bike, and if I took them off, the bike wouldn’t be rideable. But a team isn’t just a machine – a team’s impact is an emergent phenomenon that comes out of many factors, not a static item. If a sports team has a really excellent coach, they’ll retain the lessons they learned from that coach’s mentorship even after the coach moves away. Older players will pass along the coach’s lessons to younger ones, and their ideas will stick around and improve the group even long after the original players’ retirement. When a team is coordinated well, one member leaving doesn’t hurt it. And if I leave on good terms rather than sticking around till I burn out or burn bridges, I can always be available for remaining members to consult when if need advice that only I can provide.

Would staying harm anyone?

I think that in the case of the Rust community team, it would reflect poorly on the community as a whole if the exact same people comprised the community team for the entire life of the language.

If nobody new ever joins the team, we wouldn’t get new ideas and tactics, nor the priceless infusion of fresh patience and optimism that new team members bring to our perennial challenges and frustrations. So, new team members are essential. If new people joined on a regular basis but nobody ever left, the team would grow unboundedly large as time went on, and have you ever tried to get anything done with a hundred- or thousand-person committee? In my opinion, having established team members retire every now and then is an essential factor in preventing either of those undesirable hypotheticals.

The team selects for members who’ll step up and accomplish tasks when they need to. I think establishing turnover in a healthy and sustainable way is one of the most essential tasks for the team to build its skills at. The best way to get a healthy amount of turnover – not too much, but not too little either – is for every team member to step up to the personal challenge of identifying the best time to retire from active involvement. And for me, that happens to look like right now.

Aspirational Clutter

Do you have stuff in your house that you don’t use, and it’s taking up space, and you’re kind of annoyed at it for taking up space, but you don’t feel like you can get rid of it because you think you really should use it, or you’re sure you’re just going to make some personal change that will cause you to use it someday? I call that stuff aspirational clutter: It doesn’t belong to you, it belongs to some imaginary person who doesn’t exist but you aspire to become them someday.

A team meeting every week on your agenda can be aspirational clutter in the same way as a jumbled shelf of planners or a pile of sports gear covering a treadmill: It not only isn’t a good fit for who you are right now, but by wasting time or space it actually gets in the way of the habits and changes that would make you more like that person you aspire to be.

I find few experiences more existentially miserable than feeling obliged to promise work that I know I’ll lack the resources of time or energy to deliver. Sticking around on a team that I’m no longer a good fit for puts me in a situation where get to choose between feeling guilty if I don’t promise to get any work done, or feeling like a disappointment for letting others down if I commit to more than I’m able to deliver. Those aren’t feelings I want to experience, and I can avoid them easily by being honest with myself about the amount of time and energy I have available to commit to the team.

The benefits of contributing from a non-team-member role

One scary idea that comes up when leaving a team is the question: “if I’m not on the team, how can I help with the team’s work?”.

In my opinion, it builds a healthier community if people who are good at a given team’s work spend some time interfacing with the team from the perspective of non-team-members. If I know how the community team gets stuff done and I go “undercover” as a non-team-member coming to them for help, I can give them essential feedback to improve the experience and processes that non-team-members encounter.

When I wear my non-team-member hat and try to get stuff done, I learn what it’s like for everyone else who tries to interface with the team. I can then use the skills that I built on by participating on the team to remedy any challenges that a non-team-member encounters. Those changes create a better experience for every community member who interacts with the team afterwards.

What next?

As a community team alum, I’ll keep doing the Rust outreach – the meetup organizing, the conference talks, the cute swag, the stickers – that I’ve been doing all along. Stepping down from the official team member list just formalizes the state that my involvement has been in for the past year or so: Although I get the community team’s support for my endeavors when I need it, I’m not invested in the challenges of supporting others’ work which the team is now tackling.

I’m proud of the impact that the team has had while I’ve been a part of it, and I look forward to seeing what it will continue to accomplish. I’m grateful for all the leadership and hard work that have gone into making the Rust community subteam an organization from which I can step back while remaining confident that it will keep excelling and evolving.

Why blog all that?

I’m publishing my thoughts on leaving in the hopes that they can help you, dear reader, gain some perspective on your own commitments and curate them in whatever way is best for you.

If you read this and feel pressured to leave something you love and find fulfilling, please try to forget you ever saw in this post.

If you read this hoping it would give you some excuse to quit a burdensome commitment and feel disappointed that I didn’t provide one, here it is now: You don’t need a fancy eloquent excuse to stop doing something if you don’t want to any more. Replace unfulfilling pursuits with better ones.

Do you need to Slack from irssi?

If you feel ambivalent toward the web UI, going to the trouble of setting up an IRC client for Slack will likely be more hassle than reward.

If you loathe the Slack UI but don’t care much for IRC, you might be better off considering the Matrix bridge or seeking out other clients. I have not tried them and can’t vouch for any.

If you use WeeChat, you can use Slack IRC directly, or try the WeeChat Slack gateway. A friend at a larger company informs me that the latter doesn’t hold up well on Slack workspaces in the tens of thousands of users, so if you’re on a particularly large Slack you might be better off treating it as an IRC server.

If you’re looking for a terminal-based client to get started with IRC, consider choosing WeeChat over Irssi. WeeChat is extensible in more languages and allegedly has fewer edge cases like the saga detailed here.

The IRC bridge won’t work on all Slacks

First, a word of warning: You can only Slack from IRC if a workspace owner enables the IRC/XMPP gateway for a given instance, which is disabled by default because Slack distrusts users’ ability to make good security decisions. They’re not necessarily wrong, but generally users who care deeply enough to slog through their misleading instructions also know a thing or two about SSL.

This will work on all IRC clients, But...

The good news is that once a Slack instance is exposing the IRC gateway, it looks to an IRC client just like a single standalone IRC server.

The bad news is that the instructions Slack provides will only work if you do them on an IRC client where you weren’t connected to any other IRC yet. This is because (at time of writing) they tell you only to add Slack as a server in your client.

Irssi is “special”.

Let’s step back to the basics of how IRC works for a minute: On IRC, a network is a group of servers, and on a given network you can join various channels. Being on a network is intentionally agnostic of what server you’re connected to, so that if one server goes away, you can connect to another and keep right on chatting.

All modern IRC clients that I’m aware of allow you to be connected to several networks at once. For instance I’m on the Freenode IRC network to talk to people about FOSS projects I use, and also the Mozilla IRC network because most of my work channels are there. Every command you issue to irssi is done in the context of some network – do you want to auth to services? Join a new channel? Add a server? Irssi assumes that the server of the buffer in which you issue a command is the server you want the command to apply to, though some might require you to explicitly specify the network.

So, if you’re already on a network and you tell your client to add a server, what will happen? That’s right, your client will do the smart thing and add the server to the network. It will also likely connect you to that server, and do all the things on that server which you’ve asked it to do when you first connect to that particular network.

What happens if that new server is not part of the network in question at all, but is instead Slack? irssi will do the automatic things, like joining channels and trying to auth to services, that it’s supposed to do when it joins that network anyways, because you as a human told the client that the server was on the network. Even if that’s because some mean ol’ documentation tricked you into it, irssi doesn’t know any better.

What happens when irssi autojoins a bunch of channels, but issues those join commands to the Slack server? Well, on Slack just as on IRC, the first person to join a channel creates it. So, you have just revealed to your whole Slack workspace exactly the names of the channels you were in on the IRC server from which you issued the /server command.

Spuriously creating a bunch of channels isn’t the end of the world, you can just delete them, right? Well, if you have owner or admin permissions on the Slack workspace, absolutely! If you are not an owner or admin, you will have to go find someone who is and ask them to clean up the mess.

Well, at least that’s what happens when an active IRC user blindly assumes that whoever wrote the Slack IRC connection instructions had tried them in an irssi instance they were actually using for IRC. My bad.

Don’t follow the Slack docs verbatim from Irssi.

When you’re looking at https://my.slack.com/account/gateways, it has instructions like the following:

1. Ensure that your IRC client is configured with your normal Slack username as your nick.
2. If you are connecting through a raw /server command, your command will be: /server myserver.irc.slack.com 6667 myserver.Nosh5Neevot5Efua
3. If you have a more UI-oriented setup, your IRC server is myserver.irc.slack.com, and the server password is myserver.Nosh5Neevot5Efua. Accepted ports are 6667, 6697, and 8000.

That Nosh5Neevot5Efua bit is a password you shouldn’t share with anyone – for this post I’m using a string from pwgen so it looks more like the actual config.

To avoid the tale of woe that I outlined above, if you’re slacking from Irssi, you need to add a network before adding the server. This changes the steps to:

1. Ensure that your IRC client is configured with your normal Slack username as your nick.
2. Add a network with the command /network add myslack
3. If you are connecting through a raw /server command, your command will be: /server add -auto -network myslack myserver.irc.slack.com 6667 myserver.Nosh5Neevot5Efua

See the irssi docs for more options. Join the desired channels on the Slack network just as you would in IRC. When you’re done, remember to /save, and your .irssi/config should contain something like:

servers = (
  ...
  {
    address = "myserver.irc.slack.com";
    chatnet = "myslack";
    port = "6697";
    use_ssl = "yes";
    ssl_verify = "no";
    autoconnect = "yes";
    password = "mozilla.Nosh5Neevot5Efua";
  }
);

chatnets = {
  ...
  myslack = { type = "IRC"; };
};

channels = (
  ...
  { name = "#slackchannel"; chatnet = "myslack"; autojoin = "yes"; },
  ...
)

Now Slack is almost IRC

With the bridge set up, Slack behaves mostly like IRC. There remain some outstanding differences:

• You cannot leave the workspace’s default channel. You can mute the channel or turn off notifications but Slack won’t let you leave.
• When someone uses @here in a channel, Slack appends your username to the end of the message when forwarding it along to IRC to make sure you get pinged. The person did not actually type your nick when it occurrs in this context.
• If you want to hilight a Slack user in a message, you must inlcude the @ in their username. If you just say the string of their name, they won’t get notified. This is the opposite of IRC, where it’s a newbie mistake to include someone’s hat when addressing them.
• Slack has message threading and allows editing and deleting messages, neither of which are really a thing on IRC. Remember that Slack sends the first version of each message to the IRC bridge. Messages in a thread will look like they were sent to the channel. Messages that were later deleted will persist in your IRC logs. Edits won’t show up; IRC bridge users see only the first version of each. If you need to view an edited message or edit or delete your message, you have to use the Slack UI.

Have fun!

Problems with in-person standups

When I’ve participated in stand-up meetings in person, I’ve noticed a few major flaws:

• Context switching into and out of the meeting impacts a maker’s schedule by substantially more than the meeting’s planned 5 minutes.
• People naturally tend to problem-solve during the meeting, and overcoming this urge to be helpful can be difficult and frustrating. However, allowing this problem-solving is a waste of most attendees’ time and can drag the meeting out to over an hour if left unchecked.
• The content of the meeting isn’t recorded. If I run into an issue that I think I recall Jane saying she was blocked on last week, I can either interrupt her work to ask her about it, send her an email and wait for her to reply, or just fight it myself. I can’t look it up anywhere.
• If a team decides to keep notes, this “office housework may be distributed inequitably among team members. Or if everyone takes turns taking notes, well... not everyone is necessarily skilled at note-taking, so there’s little guarantee that the notes will be consistently useful.

When an international team decides to pursue standup meetings through a synchronous medium like a phone or video call, it keeps all of these drawbacks while adding the problem of time zones. Let’s say your San Francisco-based company holds your daily standup at the perfectly sensible hour of 10am. Colleagues in New York City may love this, as it’s 1pm their time so they have plenty of time to prepare for the meeting. But your “perfectly reasonable” 10am standup isn’t so reasonable internationally: it’s 5pm for a colleague in London, 6pm in Paris, 6am the next day in Auckland, and sleep-worthy hours like 4AM the next day in Sydney and 2am, also the next day, in Tokyo.

Is demanding that some team members stay late at the office every day at the expense of family and personal commitments, or wake up before sunrise, the way that you want to treat your team? Is making a request like this, which disproportionately impacts your international colleagues, consistent with your values?

The better way: Robots!

If you’re familiar with my talks about community automation, you won’t be surprised at my excitement to share another robot which makes life better.

A team I’m on has recently started using a Slack app called Geekbot to perform asynchronous, inherently logged, on-task standup meetings. The only thing special about Geekbot is that somebody else has already done the coding, testing, and debugging – if your team uses IRC or another chat client, the basic “ask questions of each team member and post their answers, once per day” functionality is trivial to implement on any extensible platform.

These distributed, asynchronous standups are the best standup meetings I’ve ever participated in. Why?

• They stay on task. You answer specific questions to the robot in PM, the robot posts your answers to a channel, and anyone who wants to chat about what you said has to do so outside the main thread of that conversation.
• They’re asynchronous. If we added colleagues in any time zone, they could configure the bot to ping them at a time they find convenient, and the rest of the team could still keep up with their progress as long as everyone keeps reporting roughly every 24 hours.
• Others’ updates minimize interruption. Rather than dropping what I’m doing to take a call for a meeting, I can use my ordinary chat-multitasking skills to read my colleagues’ updates while waiting on another task to complete.
• They’re self-recording. I can look back after lunch and see what I claimed I’d get done today; I can search my chat logs for “who was working on that component?”. I strongly prefer to answer easy questions like this myself instead of interrupting others – I save that for the difficult, interesting questions – so I deeply appreciate this ability to solve my own problems with the meeting’s inherently perfect notes.

Basically, these robot-powered, distributed standup check-ins are showing all of the benefits, and none of the major drawbacks, of in-person standup meetings.

The catch: Culture

Why is it working? We’ve had similar standup type platforms before, and they work poorly if at all. I believe these standups are working better than prior attempts to automate the process for 2 main reasons:

• The standups are performed in a convenient location. Rather than having to remember to log into some service which exists only for the standup, it comes to you in the chat medium where you were doing the rest of your team communication.
• The team’s culture values filling them out. If someone skips a standup, others will ask them where they were or what was going on. If you added a bot without adding a culture of appreciating standups, everyone would simply ignore or block the bot and nothing would change.

So, assess how your standups are working. Do they take your target amount of time and stay focused? Can you refer to their contents later as you need to? If they could use improvement, it’s worth investigating how a robot could help you out.

No wasted space

I’ve also stumbled into a rule of thumb that I like a lot better than the “1 page rule”: The rule of No Wasted Space.

If you spread 1 page worth of stuff across 2 pages, you’re wasting a page worth of space. This forces any reader to spend 2 pages worth of time on your 1 page worth of content, which is an act of disrespect to them.

Big blank areas are a waste of space if they push your resume to more pages than it needs to be. On the other side of that same coin, however, a bunch of cramped small text wastes its space if it sacrifices easy legibility and scannability for the sake of cramming everything relevant onto a single sheet. And excess words where fewer would have communicated just as well are a waste of ink, as well as interviewer time!

Number those pages

Resumes that run to multiple pages are vastly improved by a little note on the corner of each: “Surname, Page X of Y”.

“I existed!”

I’m still surprised by how often people share awards and titles as “got X award”, “was President of Organization”. I hold the opinion that identical achievements sound much cooler with active rather than passive verbs: “Earned X award”, “Led Organization”, etc.

Stuff in more numbers

I’m a little annoyed by whatever cognitive bias is in play with this one: People sound better at what they do when they stuff more numbers into their descriptions, even when the numbers aren’t objectively very useful or necessary. “Tutored students” vs “Tutored 30 students”, “Improved performance” vs “Doubled performance”, etc. This is another compelling reason to instrument your systems and measure them before and after making major changes, which is something I personally need to improve at work as well :)

The Mirror Trick

When you’ve been staring at a document for hours, it’s really hard to take a step back and tell what kind of first impression its formatting is going to make. To counter this, make your resume in the format you expect an interviewer to see it. Printed out or fullscreened on a laptop are common choices. Then either make your computer flip it horizontally, or just hold it up to a mirror and look at the reflection. This makes it look just new enough that you can spot glaring formatting errors instead of just reading the individual words – “What’s that enormous white space doing there?”, “Wait why is that indented to a different level from everything else?”.

Safety to Consume Desired Information

I believe that the fundamental layer of safety that someone can have online is to be able to safely consume information. Even at this basic level, a lot of things can go wrong. To safely consume information, people need internet access. This might mean free public WiFi, or a cell phone data plan. Safety here means that the user won’t come to harm solely as a result of what they choose to learn. “Desired information” means that the person gets a chance to find the “best” answer to their question that’s available.

How could someone come to harm as a result of choosing to learn something? If you’ve ever joked about a particular search getting you “put on a watch list”, I’m sure you can guess. I happen to hold the opinion that knowledge is an amoral tool, and it’s the actions that people take for which they should be held accountable – if you believe that there exist facts that are inherently unethical to know, we’ll necessarily differ on the importance of this safety.

How might someone fail to get the information they desired? Imagine someone searching for the best open source social networking tools on a “free” internet connection that’s provided and monitored by a social networking giant. Do you think the articles that turn up in their search results would be comparable to what they’d get on a connection provided by a less biased organization?

Why “desired information”, and not “truth”? My reason here is selfish. I enjoy learning about different viewpoints held by groups who each insist that the other is completely wrong. If somebody tried to moderate what information is “true” and what’s “false”, I would probably only be allowed to access the propaganda of at most one of those groups.

Sadly, if your ISP is monitoring your internet connection or tampering with the content you’re trying to view, there’s not a whole lot that you can do about it. The usual solution is to relocate – either physically, or feign relocation by using an onion router or proxy. By building better tools, legislation, and localization, it’s plausible that we could extend this safety to almost everyone in the world within our lifetimes.

Safety to Produce Information Anonymously

I think the next layer of internet safety that people need is the ability to produce information anonymously. The caveat here is that, of course, nobody else is obligated to choose to host your content for you. The safety of hosting providers, especially coupled with their ability to take financial payment while maintaining user anonymity, is a whole other can of worms.

Why does producing information anonymously come before producing information with attribution? Consider the types of danger that accompany producing content online. Attackers often choose their victims based on characteristics that the victims have in the physical world. Attempted attacks often cause harm because the attacker could identify the victim’s physical location or social identity. While the best solution would of course be to prevent the attackers from behaving harmfully at all, a less ambitious but more attainable project is to simply prevent them from being able to find targets for their aggression. Imagine an attacker determined to harm all people in a certain group, on an internet where nobody discloses whether or not they’re a member of that group: The attacker is forced to go for nobody or everybody, neither of which is as effective as an individually targeted attack. And that’s just for verbal or digital assaults – it is extremely difficult to threaten or enact physical harm upon someone whose location you do not know.

Systems that support anonymity and arbitrary account creation open themselves to attempted abuse, but they also provide people with extremely powerful tools to avoid being abused. There are of course tradeoffs – it takes a certain amount of mental overhead, and might feel duplicitous, to use separate accounts for discussing your unfashionable polticical views and planning the local block party – but there’s no denying how much less harm it is possible to come to when behaving anonymously than when advertising your physical identity and location.

How do you produce information anonymously? First, you access the internet in a way that won’t make it easy to trace your activity to your house. This could mean booting from a LiveCD and accessing a public internet connection, or booting from a LiveCD and using a proxy or onion router to connect to the sites you wish to access in order to mask your IP address. A LiveCD is safer than using your day-to-day computer profile because browsers store information from sites you visit, and some information about your operating system is sometimes visible to sites you visit. Using a brand-new copy of your operating system, which forgets everything when you shut down, is an easy way to avoid revealing those identifying pieces of information.

Proof read anything that you want to post anonymously to make sure it doesn’t contain any details about where you live, or facts that only someone with your experiences would know.

How do you put information online anonymously? Once you have a connection that’s hard to trace to your real-world self, it’s pretty simple to set up free accounts on mail and web hosting sites under some placeholder name.

Be aware that the vocabulary you use and the way you structure your sentences can sometimes be identifying, as well. A good way to strip all of the uniqueness from your writing voice is to run a piece of writing through http://hemingwayapp.com/ and fix everything that it calls an error. After that, use a thesaurus to add some words you don’t usually use anywhere else. Alternately, you could run it through a couple different translation tools to make it sound less like you wrote it.

How do you share something you wrote anonymously with your friends? Here’s the hard part: You don’t. If you’re not careful, the way that you distribute a piece of information that you wrote anonymously can make it clear that it came from you. Anonymously posted information generally has to be shared publicly or to an entire forum, because to pick and choose exactly which individuals get to see a piece of content reveals a lot about the identity of the person posting it.

Doing these things can enable you to produce a piece of information on the internet that would be a real nuisance to trace back to you in real life. It’s not impossible, of course – there are sneaky tricks like comparing the times when you use a proxy to the times when material shows up online – but someone would only attempt such tricks if they already had a high level of technical knowledge and a grudge against you in particular.

Long story short, in most places with internet access, it is possible but inconvenient to exercise your safety to produce information anonymously. By building better online tools and hosting options, we can extend this safety to more people who have internet access.

Safety to Produce Information Pseudonymously

An important thing to note about producing information anonymously is that if you step up and take credit for another piece of information you posted, you’re less anonymous. Add another attribution, and you’re easier still to track. It’s most anonymous to produce every piece of information under a different throwaway identity, and least anonymous to produce everything under a single identity even if it’s made up.

Producing information pseudonymously is when you use a fake name and biography, but otherwise go about the internet as the same person from day to day. The technical mechanics of producing a single pseudonymous post are identical to what I described for acting “anonymously”, but I differentiate psyedonymity from anonymity in that the former is continuous – you can form friendships with other humans under a pseudonym.

The major hazard to a pseudonymous online presence is that if you aggregate enough details about your physical life under a single account, someone reading all those details might use them to figure out who you are offline. This is addressed by private forums and boards, which limit the number of possible attackers who can see your posts, as well as by being careful of what information you disclose. Beware, however, that any software vulnerability in a private forum may mean its contents suddenly becomes public.

In my opinion, pseudonymous identity is an excellent compromise between the social benefits of always being the same person, and physical safety from hypothetical attackers. I find that behaving pseudonymously rather than anonymously helps me build friendships with people whom I’m not sure at first whether to trust, while maintaining a sense of accountability for my reputation that’s absent in strictly anonymous communication. But hey, I’m biased – you probably don’t know my full name or home address from my web presence, so I’m on the pseudonymity spectrum too.

Safety to Produce Information with Accurate Attribution

The “safety” to produce information with attribution is extremely complex, and the one on which I believe that most social justice advocates tend to focus on. It is as it sounds: Will someone come to harm if they choose to post their opinions and location under their real name?

For some people, this is the easiest safety to acquire: If you’re in a group that’s not subject to hate crimes in your area, and your content is only consumed by people who agree with you or feel neutrally toward your views, you have this freedom by default.

For others, this safety is almost impossible to obtain. If the combination of your appearance and the views you’re discussing would get you hurt if you said it in public, extreme social change would be required before you had even a chance at being comparably safe online.

I hold the opinion that solving the general case of linking created content to real-world identities is not a computer problem. It’s a social problem, requiring a world in which no person offended by something on the internet and aware of where its creator lives is physically able to take action against the content’s creator. So it’d be great, but we are not there yet, and the only fictional worlds I’ve encountered in which this safety can be said to exist are impossibly unrealistic, totalitarian dystopias, or both.

In Summary

In other words, I view misuse of the internet as a pattern of the form “Creator posts content -> attacker views content -> attacker identifies creator -> attacker harms creator”. This chain can break, with varying degrees of difficulty, at several points:

First, this chain of outcomes won’t begin if the creator doesn’t post the content at all. This is the easiest solution, and I point out the “safety to consume desired content” because even someone who never posts online can derive major benefits from the information available on the internet. It’s easy, but it’s not good enough: Producing as well as consuming content is part of what sets the internet apart from TV or books.

The next essential link in the chain is the attacker identifying the content’s creator. If someone has no way to contact you physically or digitally, all they can do is shout nasty things to the world online, and you’re free to either ignore them or shout right back. Having nasty things shouted about your work isn’t optimal, but it is difficult to feel that your physical or social wellbeing is jeopardized by someone when they have no idea who you are. This is why I believe that the safety to produce information anonymously is so important: It uses software to change the outcome even in circumstances where the attacker’s behavior cannot be modified. Perfect pseudonymity also breaks this link, but any software mishap or accidental over-sharing can invalidate it instantly. The link is broken with fewer potential points of failure by creating content anonymously.

The third solution is what I alluded to when discussing the safety of pseudonymity: Prevent the attacker from viewing the content. This is what private, interest-specific forums accomplish reasonably well. There are hazards here, especially if a forum’s contents become public unintentionally, or if a dedicated attacker masquerades as a member of the very group they wish to harm. So it helps, and can be improved technologically through proper security practices by forum administrators, and socially via appropriate moderation. It’s better, from the perspective that assuming the same online identity each day allows creators to build social bonds with one another, but it’s still not optimal.

The fourth and ideal solution is to break the cycle right at the very end, by preventing the attacker from harming the content creator. This seems to be where most advocates argue we should jump straight into, because it’s really perfect – it requires no change or compromise from content creators, and total change from those who might be out to harm them. It’s the only solution in which people of all appearances and beliefs and locations are equally safe online. However, it’s also the most difficult place to break the cycle, and a place at which any error of implementation would create the potential for incalculable abuse.

I’ve listed these safeties in an order that I regard as how feasible they are to implement with today’s social systems and technologies. I think it’s possible to recognize the 4th safety as the top of the heap, without using that as an excuse to neglect the benefits which can come from bringing more of the world its 3 lesser but far more attainable cousins.

Philosophy of Job Hunting

My opinion on job hunting, especially for early-career technologists, is that it’s important to get multiple offers whenever possible. Only once one has a viable alternative can one be said to truly choose a role, rather than being forced into it by financial necessity.

In my personal experience, cultivating multiple offers was an important step in disentangling impostor syndrome from my career choices. Multiple data points about one’s skills being valued by others can help balance out an internal monologue about how much one has yet to learn.

If you disagree that cultivating simulataneous opportunities then politely declining all but the best is a viable internship hunting strategy, the rest of this post may not be relevant or interesting to you.

Identifying Your Options

To get an internship offer, you need to make a compelling application to a company which might hire you. I find that a useful first step is to come up with a list of such companies, so you can study their needs and determine what will make your application interest them.

Use your social network. Ask your peers about what internships they’ve had or applied for. Ask your mentors whether they or their friends and colleagues hire interns.

When you ask someone about their experience with a company, remember to ask for their opinion of it. To put that opinion into perspective, it’s useful to also ask about their personal preferences for what they enjoy or hate about a workplace. Knowing that someone who prefers to work with a lot of background noise enjoyed a company’s busy open-plan office can be extremely useful if you need silence to concentrate! Listening with interest to a person’s opinions also strengthens your social bond with them, which never hurts if it turns out they can help you get into a company that you feel might be a good fit.

Use LinkedIn, Hacker News, Glassdoor, and your city’s job boards. The broader a net you cast to start with, the better your chances of eventually finding somewhere that you enjoy. If your job hunt includes certain fields (web dev, DevOps, big data, whatever), investigate whether there’s a meetup for professionals in that field in your region. If you have the opportunity to give a short talk on a personal project at such a meetup, do it and make sure to mention that you’re looking for an internship.

Identify your own priorities

Now that you have a list of places which might concievably want to hire you, it’s time to do some introspection. For each field that you’ve found a prospective company in, try to answer the question “What makes you excited about working here?”.

You do not have to have know what you want to do with your life to know that, right now, you think DevOps or big data or frontend development is cool.

You do not have to personally commit to a single passion at the expense of all others – it’s perfectly fine to be interested in several different languages or frameworks, even if the tech media tries to pit them against each other.

However, for each application, it’s prudent to only emphasize your interests in that particular field. It’s a bit of a faux pas to show up to a helpdesk interview and focus the whole time on your passion for building robots, or vice versa. And acting equally interested in every other field will cause an employer to doubt that you’re necessarily the best fit for a specialized role... So in an interview, try not to stray too far from the value that you’re able to deliver to that company.

This is also a good time to identify any deal-breakers that would cause you to decline a prospective internship. Are you ok with relocating? Is there some tool or technology that would cause you to dread going to work every day?

I personally think that it’s worth applying even to a role that you know you wouldn’t accept an offer from when you’re early in your career. If they decide to interview you, you’ll get practice experiencing a real interview without the pressure of “I’ll lose my chance at my dream job if I mess this up!”. Plus if they extend an offer to you, it can help you calibrate the financial value of your skills and negotiate with employers that you’d actually enjoy.

Craft an excellent resume

I talk about this elsewhere.

There are a couple extra notes if you’re applying for an internship:

1) Emphasize the parts of your experience that relate most closely to what each employer values. If you can, it’s great to use the same words for skills that were used in the job description.

2) The bar for what skills go on your resume is lower when you have less experience. Did you play with Docker for a weekend recently and use it to deploy a toy app? Make sure to include that experience.

Practice, Practice, Practice

If you’re uncomfortable with interviewing, do it until it becomes second nature. If your current boss supports your internship search, do some mock interviews with them. If you’re nervous about things going wrong, have a friend roleplay as a really bad interview with you to help you practice coping strategies. If you’ll be in front of a panel of interviewers, try to get a panel of friends to gang up on you and ask tough questions!

To some readers this may be obvious, but to others it’s worth pointing out that you should also practice wearing the clothes that you’ll wear to an interview. If you wear a tie, learn to tie it well. If you wear shirts or pants that need to be ironed, learn to iron them comptently. If you wear shoes that need to be shined, learn to shine them. And if your interview will include lunch, learn to eat with good table manners and avoid spilling food on yourself.

Yes, the day-to-day dress codes of many tech offices are solidly in the “sneakers, jeans, and t-shirt” category for employees of all levels and genders. But many interviewers, especially mid- to late-career folks, grew up in an age when dressing casually at an interview was a sign of incompetence or disrespect. Although some may make an effort to overcome those biases, the subconscious conditioning is often still there, and you can take advantage of it by wearing at least business casual.

Apply Everywhere

If you know someone at a company where you’re applying, try to get their feedback on how you can tailor your resume to be the best fit for the job you’re looking at! They might even be able to introduce you personally to your potential future boss.

I think it’s worth submitting a good resume to every company which you identify as being possibly interested in your skills, even the ones you don’t currently think you want to work for. Interview practice is worth more in potential future salary than the hours of your time it’ll take at this point in your career.

Follow Up

If you don’t hear back from a company for a couple weeks, a polite note is order. Restate your enthusiasm for their company or field, express your understanding that there are a lot of candidates and everything is busy, and politely solicit any feedback that they may be able to offer about your application. A delayed reply does not always mean rejection.

If you’re rejected, follow up to thank HR for their time.

If you’re invited to interview, reply promptly and set a time and date. For a virtual or remote interview, only offer times when you’ll have access to a quiet room with a good network connection.

Interview Excellently

I don’t have any advice that you won’t find a hundred times over on the rest of the web. The key points are:

• Show up on time, looking respectable
• Let’s hope you didn’t lie on your resume
• Restate each question in your answer
• It’s ok not to know an answer – state what you would do if you encountered the problem at work. Would you Google a certain phrase? Ask a colleague? Read the manual?
• Always ask questions at the end. When in doubt, ask your interviewer what they enjoy about working for the company.

Keep Following Up

After your interview, write to whoever arranged it and thank the interviewers for their time. For bonus points, mention something that you talked about in the interview, or include the answer to a question that you didn’t know off the top of your head at the time.

Getting an Offer

Recruiters don’t usually like to disclose the details of offers in writing right away. They’ll often phone you to talk about it. You do not have to accept or decline during that first call – if you’re trying to stall for a bit more time for another company to get back to you, an excuse like “I’ll have to run that by my family to make sure those details will work” is often safe.

Remember, though, that no offer is really a job until both you and the employer have signed a contract.

Declining Offers

If you’ve applied to enough places with a sufficiently compelling resume, you’ll probably have multiple offers. If you’re lucky, they’ll all arrive around the same time.

If you wish to decline an offer from a company whom you’re certain you don’t want to work for, you can practice your negotiation skills. Read up on salary negotiation, try to talk the company into making you a better offer, and observe what works and what doesn’t. It’s not super polite to invest a bunch of their time in negotiations and then turn them down anyway, which is why I suggest only doing this to a place that you’re not very fond of.

To decline an offer without burning any bridges, be sure to thank them again for their time and regretfully inform them that you’ll be pursuing other opportunities at this time. It never hurts to also do them a favor like recommending a friend who’s job hunting and might be a good fit.

Again, though, don’t decline an offer until you have your actual job’s contract in writing.

Intro

I’m a DevOps engineer at Mozilla Research and a member of the Rust Community subteam, but the conclusions and suggestions in this talk are my own observations and opinions.

The slides are a result of trying to write my own CSS for sliderust... Sorry about the ugliness.

I have 10 minutes, so this is not the time to teach you any Rust. Check out rust-lang.org, the Rust Community Resources, or your city’s Rust meetup to get started with the language.

What we are going to cover is how Rust automates some community tasks, and what you can learn from our automation.

Community

I define “community”, in this context, as “the human interaction work necessary for a project’s success”. This work is done by a wide variety of people in many situations. Every interaction, from helping a new contributor to discussing a proposed code change to criticizing someone’s behavior, affects the overall climate of a project’s community.

Automation

To me, “automation” means “offloading peoples’ work onto a system”. This can be a computer system, but I think it can also mean changes to the social systems that guide peoples’ behavior.

So, community automation is a combination of:

• Building tools to do things the humans used to have to
• Tweaking the social systems to minimize the overhead they create

Scoping the Problem

While not all things can be automated and not all factors of the community are under the project leadership’s control, it’s not totally hopeless.

Choices made and automation deployed by project leaders can help control:

• Which contributors feel welcome or unwelcome in a project
• What code makes it into the project’s tree
• Robots!

Moderation

Our robots and social systems to improve workflow and contributor experience all rely on community members’ cooperation. To create a community of people who want to work constructively together and not be jerks to each other, Rust defines behavior expectations code of conduct. The important thing to note about the CoC is that half the document is a clear explanation of how the policies in it will be enforced. This would be impossible without the dedication of the amazing mod team.

The process of moderation cannot and should not be left to a computer, but we can use technology to make our mods’ work as easy as possible. We leave the human tasks to humans, but let our technologies do the rest.

In this case, while the mods need to step in when a human has a complaint about something, we can automate the process of telling peole that the rules exist. You can’t join the IRC channel, post on the Discourse forums, or even read the Rust subreddit without being made aware that you’re expected to follow the CoC’s guidelines in official Rust spaces.

Depending on the forums where your project communicates, try to automate the process of excluding obvious spammers and trolls. Not everybody has the skills or interest to be an excellent moderator, so when you find them, avoid wasting their time on things that a computer could do for them!

It didn’t fit in the talk, but this Slashdot post is one of my favorite examples of somebody being filtered out of participating in the Rust community due to their personal convictions about how project leadership should work. While we do miss out on that person’s potential technical contributions, we also save all of the time that might be spent hashing out our disagreements with them if we had a less clear set of community guideines.

Robots

This lightning talk highlighted 4 categories of robots:

• Maintaining code quality
• Engaging in social pleasantries
• Guiding new contributors
• Widening the contributor pipeline

Longer versions of this talk also touch on automatically testing compiler releases, but that’s more than 10 minutes of content on its own.

Set arbirary metadata with /msg chanserv set #channel property key value

The commands /msg chanserv set #channel email a@b.com and /msg chanserv set #channel property email a@b.com appear to function identically, with the former being a convenient wrapper around the latter.

So that’s how #atheme got their fancy cryllic taxonomy: Someone with the appropriate permissions issued the command /msg chanserv set #atheme property ОХЯЕБУ лололол.

Behaviors of channel properties

I’ve attempted to deduce the rules governing custom metadata items, because I couldn’t find them documented anywhere.

0. Issuing a set property command with a property name but no value deletes the property, removing it from the taxonomy.
1. A property is overwritten each time someone with the appropriate permissions issues a /set command with a matching property name (more on the matching in a moment). The property name and value are stored with the same capitalization as the command issued.
2. The algorithm which decides whether to overwrite an existing property or create a new one is not case sensitive. So if you set ##test email test@example.com and then set ##test EMAIL foo, the final taxonomy will show no field called email and one field called EMAIL with the value foo.
3. When displayed, taxonomy items are sorted first in alphabetical order (case insensitively), then by length. For instance, properties with the names a, AA, and aAa would appear in that order, because the initial alphebetization is case-insensitive.
4. Attempting to place mIRC color codes in the property name results in the error “Parameters are too long. Aborting.” However, placing color codes in the value of a custom property works just fine.

Other uses

As a final note, you can also do basically the same thing with Freenode’s NickServ, to set custom information about your nickname instead of about a channel.

Command Equivalences

As this tutorial so helpfully explains, the two VCSes aren’t all that dissimilar under their hoods. I condensed the command comparison table into a single page and printed it out for quick reference; a PDF is here.

Clone

The thing I want to clone lives at http://hg.mozilla.org/hgcustom/version-control-tools/file/tip/autoland.

Trying to clone the full URL yields a 404, but snipping the URL back to the top-level directory gets me the repo:

$ hg clone http://hg.mozilla.org/hgcustom/version-control-tools/
destination directory: version-control-tools
requesting all changes
adding changesets
adding manifests
adding file changes
added 4574 changesets with 10874 changes to 1971 files
updating to bookmark @
1428 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ ls
version-control-tools

Examine Log

hg log | less shows me that each commit’s summary in this repo includes the part of the codebase it touches, and a bug number.

hg log | grep autoland: | less gives me the summaries of every commit that touched autoland, but I cannot show a commit from summary alone.

The Hg book helped me construct a filter that will show a unique revision ID onthe same line as each description.

hg log --template '{rev} {desc}\n' | grep autoland: is much more useful. It gives me the local ID of each changeset whose description included “autoland:”.

From here, I can use a bit more grep to narrow down the list of matching messages, then I’m ready to examine commits.

Examining Commits

That {rev} in my filter was the “repository-local changeset revision number”. For these examples I’ll examine revision 2589.

hg status --change 2589 lists the files that were touched by that revision, and hg export 2589 yields a full diff of the changes introduced.

This gets me enough information to make an appropriate set of changes, run the tests, and create my own commits!

Hieroglyph dependency issues

The error resulting from pip install -r requirements.txt ended with:

Command ".../virtualenv/bin/python2 -u -c
"import setuptools,
tokenize;__file__='/tmp/pip-build-lzbk_r/Pillow/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
install --record /tmp/pip-BNDc_6-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/home/edunham/repos/slides/rustcommunity/v/include/site/python2.7/Pillow"
failed with error code 1 in /tmp/pip-build-lzbk_r/Pillow/

Its fix, from stackoverflow, was:

$ sudo apt-get install libtiff5-dev libjpeg8-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python-tk
$ pip install -r requirements.txt

Tinkerer depencencies, too!

pip install -r requirements.txt over in my blog repo yielded:

Command ".../virtualenv/bin/python2 -u -c "import setuptools,
tokenize;__file__='/tmp/pip-build-NVLSBY/lxml/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
install --record /tmp/pip-qD5QIe-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/home/edunham/repos/site/v/include/site/python2.7/lxml" failed with error code
1 in /tmp/pip-build-NVLSBY/lxml/

The fix is again to install the missing system deps, on Ubuntu:

$ sudo apt-get install libxml2-dev libxslt-dev
$ pip install -r requirements.txt

That’s it! I’m writing this down for SEO on the specific errors at hand, since the first several useful hits are currently stackoverflow.

If you’re a pip developer reading this, please briefly contemplate whether it’d be worthwhile to have some built-in mechanism to translate common dependency errors to the appropriate system package names needed based on the OS on which the command is run.

Getting ready to write your abstract

Your abstract is a promise about what your talk will deliver. Have you ever gotten your hopes up for a talk based on its abstract, then attended only to hear something totally unrelated? You can save your audience from that disappointment by making sure that you present what your abstract says you will.

I find the abstract to be the hardest part of the talk to write, because it sets the stage for every other part of it. If your abstract is thorough and clear about what your talk will deliver, you can refer back to it throughout the writing process to make sure you’re including the information that your audience is there for!

For both you and your audience to get the most out of your talk, the following questions can help you refine your talk idea before you even start to write its abstract.

The Anatomy of an Abstract

There are many ways to structure a good abstract. Here are the 7 elements that I try to always include:

1. Set the scene with a strong introductory sentence, which reminds your target audience of your topic’s relevance to them. Some of mine have included:

   • “Rust is a systems programming language that runs blazingly fast, prevents segfaults, and guarantees thread safety.”
   • “Git is the most popular source code management and version control system in the open source community.”
   • “When you’re new to programming, or self-taught with an emphasis on those topics that are directly relevant to your current project, it’s easy to skip learning about analyzing the complexity of algorithms.”

2. Ask some questions, which the talk promises to answer. These questions should be asked from the perspective of your target audience, which you identified earlier. This is the least essential piece of an abstract, and can be skipped if you make sure your exposition clearly shows that you understand your target audience in some other way. Here are a couple of questions I’ve used in abstracts that were accepted to conferences:

   • “Do you know how to control what information people can discover about you on an IRC network?”
   • “Is the project of your dreams ignoring your pull requests?”

3. Drop some hints about the format that the talk will take. This shows the selection commitee that you’ve planned ahead, and helps audience members select session that’re a good fit for their learning styles. Useful words here include:

   • “Overview of”
   • “Case study”
   • “Demonstration”
   • “Deep dive into”
   • “Outline X principles for”
   • “Live coding”

4. Identify what background knowledge the audience will need to get the talk’s benefit, if applicable. Being specific about this helps welcome audience members who’re undecided about whether the talk is applicable to them. Useful phrases include:

   • “This talk will assume no background knowledge of...”
   • “If you’ve used ____ to ____, ...”
   • “If you’ve completed the ____ tutorial...”

5. State a specific benefit that audience members will get from having attended the talk. Benefits can include:

   • “Halve your Django website’s page load times”
   • “Get help on IRC”
   • “Learn from ____‘s mistakes”
   • “Ask the right questions about ____“

6. Reinforce and quantify your credibility. If you’re presenting a case study into how your company deployed a specific tool, be sure to mention your role on the team! For instance, you might say:

   • “Presented by [the original author | a developer | a maintainer | a long-term user] of [the project], this talk will...”

7. End with a recap of the talk’s basic promise, and welcome audience members to attend.

These 7 pieces of information don’t have to each be in their own sentence – for instance, establishing your credibility and indicating the talk’s format often fit together nicely in a single sentence.

Once you’ve got all of the essential pieces of an abstract, munge them around until it sounds like concise, fluent English. Get some feedback on helpmeabstract.com if you’d like assistance!

Give it a title

Naming things is hard. Here are some assorted tips:

• Keep it under about 50 characters, or it might not fit on the program
• Be polite. Rude puns or metaphors might be eye-catching, but probably violate your conference or community’s code of conduct, and will definitely alienate part of your prospective audience.
• For general talks, it’s hard to go wrong with “Intro to ___” or “___ for ___ users”.
• The form “[topic]: A [history|overview|melodrama|case study|love story]” is generally reliable. Well, I’m kidding about “melodrama” and “love story”... Mostly.
• Clickbait is underhanded, but it works. “___ things I wish I’d known about ___”, anyone?

Good luck, and happy conferencing!

Why the 13?

I had a few hard requirements for my next personal laptop:

• Trackpoint with buttons
• Decent key travel (the X1 carbon has 1.86mm and typing on it for too long made my hands hurt)
• USBC port
• Under $1,000

Plus a few nice-to-haves:

• Small and light are nice, including charger
• Screen not much worse than 1920x1080
• Good battery life
• Metal case and design for durability make me happy
• My house is already full of Thinkpad chargers, so a laptop that uses them helps reduce additional clutter

I’ll be the first to admit that this is an atypical set of priorities. My laptop is home to Git, Vim, and a variety of tools for interacting with the internet, so the superficial I/O differences matter more to me than the machine’s internal specs.

Things I like about the 13

• 2.1mm key travel is everything I hoped for. At least, I’ve used it all day and my hands don’t hurt.
• Battery life is pretty decent, and battery will be easy to replace when it starts to fail.
• Light-enough weight. Lighter charger than other Thinkpads I’ve had.
• Smallest Thinkpad charger that I’ve ever seen.
• Case screws are all captive.
• Mystery hole in the bottom case turns out to be a highly convenient hard shutdown button.
• Hinges feel pretty solid and hold the screen up nicely.
• No keyboard backlight. I dislike them.
• 4GB of RAM is a single stick, easy to add more (and I’ll need to for a smoother web browsing experience; neither Firefox nor Chromium is particularly happy with only 4GB)
• A vanilla Ubuntu 16.04 iso Just Worked for installing Linux. It must have shipped with whatever magic signatures were required to play nice with the new security measures, because the install process was delightfully non-thought-provoking.
• ~7mm plastic bezel between buttons and trackpad reduces likelihood of accidentally moving cursor when clicking.

• Screen’s the same as my X240, xrandr calls it 1920x1080 294mm x 165mm. This fits 211 columns of a default font, or 127 columns of a font that’s comfortably legible when the laptop is on the other side of my desk.

Nitpicks about the 13

• Color.

When I purchased mine at the end of April, only the silver chassis had a metal lid and shipped with a nice screen by default (the higher-res screen is available in the all-plastic black model for an additional charge). So now I own a non-black laptop for the first time since my Dell Latitude D410 in high school. The screen bezel and keys are black, though, and if I really cared I could probably paint the rest of it.

• Power button.

It feels horribly... squishy? There’s no satisfying click to tell me when I’ve pushed it far enough. Holding it for 10 seconds only sometimes shuts the laptop off (though there’s a reset switch on the mobo accessible by a paperclip-hole in the bottom panel which forces shutdown instantly when pushed). There’s a circle on the pwoer button that looks like it might be an LED, but it never lights up.

• Cutesy font.

This is a tiny nitpick, but they’ve changed the Lenovo logo on the lid, pre-BIOS boot screen, and screen bezel from the already-mediocre font to a super condescending, childish, roundy one. Fortunately the lid one is easily hidden under some stickers.

• Bottom panel held on by clips as well as screws.

More on this one in the disassembly section below, but I’m afraid they’ll break with how often I take my laptops apart.

• Mouse buttons feel cheap and plastic-ey.

They feel like thin plastic shells instead of solid buttons like on older Thinkpads. I’m not sure precisely why they feel that way, but it’s a reminder that you’re using a lower-end machine.

• Longest side is about 1cm greater than the short side of a TSA xray tub.

My X240 fits perfectly along the short end of the tub, leaving room for my shoes beside it. I have to use two tubs or separate my pair of shoes when putting the 13 through the scanner. (see, I wasn’t kidding when I said “nitpicks”)

• The Trackpoint top is not interchangeable with those of older Thinkpads.

The round part is the same size, but the square hole in the bottom is about 2mm to a side rather than the 4mm of the one on an x220 keyboard. Plus the cylinder bit is about 2mm long rather than the x220’s 3.5mm, so even with an adapter for the square hole, older trackpoints would risk leaving marks on the screen.

• The fan is a little loud.

I anticipate that this will get a lot less annoying when I upgrade to 16 or 32GB of ram and maybe tune it in software using thinkfan.

Thinkpad 13 partial disassembly photos

To get the bottom case off, pull all the visible screws and also remove the 3 tiny rubber feet from under the palm rest. I stuck my tiny rubber feet in a plastic bag and filed it away, because repeated removal would eventually destroy the glue and get them lost.

The bottom case comes off with a combination of sliding and prying. Getting it back on again requires sliding the palmrest edge just right, then snapping the sides and back on before the palm rest slips out of place. It’s tricky.

The battery is easily removed by pulling out a single (non-captive) screw. It seems to be a thin plastic wrapper around 3 cell phone batteries. The battery has no glue holding it in, just screws.

Here’s its guts, with battery removed.

Note the convenient hard power cycle button (accessible via a tiny hole in the bottom case when assembled), pair of RAM slots and SSD form factor, and airspace compartment that almost looks intended for hiding half a dozen very small items. The coin cell battery (in sky blue shrink wrap) flaps around awkwardly when the machine is disassembled, but at least it’s not glued down.

Preparation

I was contacted about a month before my webcast and asked to present my OSCON talk as part of the event. I explained why my “How to learn Rust” talk didn’t sound like a good fit for the emerging languages webcast, and suggested the “Starting Rust from a Scripting Background” talk that I gave at my local Rust meetup recently as an alternative.

After we agreed on what talk would be suitable, O’Reilly’s Online Events Producers emailed me a contract to e-sign. The contract gives O’Reilly the opportunity to reuse and redistribute my talk from the webcast, and promises me a percentage of the proceeds if my recording is sold, licensed, or otherwise used to make them money.

During the week before the webcast, I did a test call in which an O’Reilly representative walked me through how to use the webcast software and verified that my audio was good on the phone I planned to use for the webcast.

A final copy of the slides for the webcast, in my option of PDF or Powerpoint, was due at 5pm the day before the event.

The Software (worked for me on Ubuntu)

O’Reilly Media provided an application called “Presentation Manager XD” from on24.com that presenters log into during the event.

According to my email from O’Reilly, the requirements for the event are:

• Slides - PowerPoint or PDF only please with no embedded video or audio. Screen ratio of 4:3
• Robust Internet connection
• Clear, reliable phone line.
• Windows 7 or 8 running IE 8+, Firefox 22+ or Chrome 27+
• Mac OS 10.6+ running Firefox 22+ or Chrome 27+
• Latest version of Flash Player
• If you plan to share your screen, you will need to install a small application - you will be prompted to install it the first time you log into the platform.

Some of these requirements are lies. I used Firefox 46.0 on Ubuntu 14.04. I did rewrite my slides in LibreOffice because it emits better PDFs than the HTML tools I normally use, but I was also looking for an excuse to rewrite them to clean up their organization.

I clicked around in the “Presentation Manager XD” UI and downloaded a file called “ON24-ScreenShare-plugin”, then chmod +x‘d it and executed it with ./ON24-ScreenShare-plugin. This caused Wine to run, install some Gecko stuff, and start the screenshare plugin sufficiently well to share my screen to the webcast tool.

I had to re-run the plugin in Wine after logging out of and back into my window manager, of course. Additionally, the screenshare window’s resizing is finicky. It’s fine to grab and drag the highlighted parts of the window’s border with the mouse, but the meta+click command with which one usually moves windows in i3 causes the sides of the screenshare window to move independently of each other.

Here’s what the webcast UI looked like during streaming, just at the end of the Kotlin talk while I was getting ready to start mine:

The Talk

As previously mentioned, I rewrote my talk in LibreOffice Impress – ostensibly to get a prettier PDF, but also because it’s been a month or two since I last prepared for it and re-writing helps me refresh my memory and verify that all my facts are up to date.

GUI-based slide editing is downright painful after using rst-based tools for so long, especially because LibreOffice has no good way to embed code samples out of the box. I ended up going with screenshots from the Rust playground, which were probably better than embedded code samples, but relearning how to edit slides like a regular person wasn’t a pleasant experience.

I took more notes than I normally do, since nobody on the webcast could see whether I was reciting or reading. I’m glad I did, as having the notes on a physical page in front of me was reassuring and helped me avoid missing any important points.

I rehearsed the timing of each section of my slides individually, since it naturally broke down into 7 or so discrete parts and I had previously calculated how much of my hour to allocate to each section. Most sections ran consistently over time when preparing, yet under time during the actual talk. The lesson here is to rehearse until I’m happy with a section and can make it the same duration twice in a row.

The experience of presenting a talk in a subjectively empty room made me realize just what high-bandwidth communication regular conferences are.

Pros:

• No need to worry about eye contact
• All the notes you want
• Can’t see anyone sleeping
• Chat channel allowed instant distribution of links
• Chat channel allowed expert attendees to help answer questions
• Presentation software allowed gracefully skipping slides, rather than the usual paging back and forth with the arrow keys

Cons:

• Can’t take quick surveys by show of hands
• Negligible feedback on how many people are there and their body language of engagement/disengagement
• Silences are super awkward
• Can’t see the shy attendees, in order to encourage participation

The audience asked fewer questions during the talk than I expected. Fortunately, they came up with plenty of questions at the end – extra fortunate because I overcompensated on time and finished my slides about 15 minutes before the end of my speaking slot!

Q&A was surprisingly relaxing, as it was totally up to me which questions to answer. I’ll admit that I did select in favor of those that I could answer concisely and eloquently, deferring the questions that didn’t make as much sense to think about them while I answered easier ones.

tl;dr

In my experience, presenting a webcast was lower-stress and comparably impactful to a conference talk.

For would-be presenters concerned about their or the audience’s appearance, the visual anonymity of a webcast could be a great place to start a speaking career.

Speakers accustomed to presenting in rooms full of humans should expect subtle feedback, like nods, smiles, and laughter, to be totally invisible in a webcast environment.

And if O’Reilly asks you to do a webcast with them, I’d say go for it – they made the whole experience as seamless and easy as possible.

Facets of DevOps

In my opinion, 3 distinct areas of focus have made me the kind of person from whom others solicit DevOps advice:

• Cultural background
• Technical skills
• Self-promotion

I place “cultural background” first because many people with all the skills to succeed at “DevOps” roles choose or get stuck with other job titles, and everyone touches a slightly different point on the metaphorical elephant of what “DevOps” even means.

Cultural Background

What does “DevOps” mean to you?

• Sysadmins who aren’t afraid of automation?
• 2 sets of job requirements for the price of 1 engineer?
• Developers who understand what the servers are actually doing?
• Reducing the traditional divide between “development” and “operations” silos?
• A buzzword that increases your number of weekly recruiter emails?
• People who use configuration management, aka “infrastructure as code”?

From my experiences starting Oregon State University’s DevOps Bootcamp training program, speaking on DevOps related topics at a variety of industry conferences, and generally being a professional in the field, I’ve seen the term defined all of those ways and more.

Someone switching from “sysadmin” to “devops” should clearly define how they want their day-to-day job duties to change, and how their skills will need to change as a result.

Technical Skills

The best way to figure out the technical skills required for your dream job will always be to talk to people in the field, and read a lot of job postings to see what you’ll need on your resume and LinkedIn to catch a recruiter’s eye.

In my opinion, the bare minimum of technical skills that an established sysadmin would need in order to apply for DevOps roles are:

• Use a configuration management tool – Chef, Puppet, Salt, or Ansible – to set up a web server in a VM.
• Write a script in Python to do something more than Hello World – an IRC bot or tool to gather data from an API is fine.
• Know enough about Git and GitHub to submit a pull request to fix something about an open source tool that other sysadmins use, even if it’s just a typo in the docs.
• Understand enough about continuous integration testing to use it on a personal project, such as TravisCI on a GitHub repo, and appreciate the value of unit and integration tests for software.
• Be able to tell a story about successfully troubleshooting a problem on a Linux or BSD server, and what you did to prevent it from happening again.

Keep in mind that your job in an interview is to represent what you know and how well you can learn new things. If you’re missing one of the above skills, go ask for help on how to build it.

Once you have all the experiences that I listed, you are no longer allowed to skip applying for an interesting role because you don’t feel you know enough. It’s the employer’s job to decide whether they want to grow you into the candiate of their dreams, and your job to give them a chance. Remember that a job posting describes the person leaving a role, and if you started with every skill listed, you’d probably be bored and not challenged to your full potential.

Self Promotion

“DevOps” is a label that engineers apply to themselves, then justify with various experiences and qualifications.

The path to becoming a community leader begins at engaging with the community. Look up DevOps-related conferences – find video recordings of talks from recent DevOps Days events, and see what names are on the schedules of upcoming DevOps conferences.

Look at which technologies the recent conferences have discussed, then look up talks about them from other events. Get into the IRC or Slack channels of the tools you want to become more expert at, listen until you know the answers to common questions, then start helping beginners newer than yourself.

Reach out to speakers whose talks you’ve enjoyed, and don’t be afraid to ask them for advice. Remember that they’re often extremely busy, so a short message with a compliment on their talk and a specific request for a suggestion is more likely to get a reply than overly vague requests. This type of networking will make your name familiar when their companies ask them to help recruit DevOps engineers, and can build valuable professional friendships that provide job leads and other assistance.

Contribute to the DevOps-related projects that you identify as having healthy communities. For configuration management, I’ve found that SaltStack is a particularly welcoming group. Find the source code on GitHub, examine the issue tracker, pick something easy, and submit a pull request fixing the bug. As you graduate to working on more challenging or larger issues, remember to advertise your involvment with the project on your LinkedIn profile!

Additionally, help others out by blogging what you learn during these adventures. If you ever find that Google doesn’t have useful results for an error message that you searched, write a blog post with the message and how you fixed it. If you’re tempted to bikeshed over which blogging platform to use, default to GitHub Pages, as a static site hosted there is easy to move to your own hosting later if you so desire.

Examine job postings for roles like you want, and make sure the key buzzwords appear on your LinkedIn profile wherever appropriate. A complete LinkedIn profile for even a relatively new DevOps engineer draws a surprising number of recruiters for DevOps-related roles. If you’re just starting out in the field, I’d recommend expressing interest in every opportunity that you’re contacted about, progressing to at least a phone interview if possible, and getting all the feedback you can about your performance afterwards. It’s especially important to interview at companies that you can’t see yourself enjoying a job at, because you can practice asking probing questions that tell you whether an employer will be a good fit for you. (check out this post for ideas).

Another trick for getting to an interview is to start something with DevOps in the name. It could be anything from a curated blog to a meetup to an online “book club” for DevOps-related papers, but leading something with a cool name seems to be highly attractive to recruiters. Another way to increase your visibility in the field is to give a talk at any local conference, especially LinuxFest and DevOpsDays events. Putting together an introductory talk on a useful technology only requires intermediate proficiency, and is a great way to build your personal brand.

To summarize, there are really 4 tricks to getting DevOps interviews, and you should interview as much as you can to get a feeling for what DevOps means to different parts of the industry:

• Contribute back to the open source tools that you use
• Network with established professionals
• Optimize your LinkedIn and other professional profiles to draw recruiters
• Be the founder of something.

Questions?

I collect interesting job search and interview advice links at the bottom of my resume repo readme.

I bolded each paragraph’s key points in the hopes of making them easier to read.

You’re welcome to reach out to me at blog @ edunham.net or @qedunham on Twitter if you have other questions. If I made a dumb typo or omitted some information in this post, either tell me about it or just throw a pull request at the repo to fix it and give yourself credit.

You’ll Need

• A couple square feet of orange polar fleece, or any other orange fabric that won’t stretch or fray too much
• A handful of stuffing. I cannibalized a throw pillow.
• A needle and some orange thread
• Black and white fabric scraps and thread, or black and white embroidery floss, for making the face.
• Intermediate sewing skills
• This pattern

The Pattern

Get yourself a front, back, underside, and claw drawn on paper, either by printing them out or tracing from a screen. The front, back, and underside should have horizontal symmetry, except for the face placement. Make sure the points marked in red and blue on this pattern are noted on your paper.

Mine measure about 6” wide between the points marked in red.

Sewing vocabulary

• The right side of a fabric is what ends up on the outside of the finished item. The wrong side ends up where you can’t see it. Some fabrics have both sides the same; in that case, the wrong side is whichever one you feel like tracing the pattern onto.
• seam allowance is some extra fabric that ends up on the inside of the item when you’re done. The pattern above does not include seam allowance. This means that if you cut the fabric along the lines in the pattern, your finished rustacean will be tiny and sad and shaped wrong. You cut the paper along the lines, then trace it onto the fabric, then sew along the lines.
• applique is where you sew one piece of fabric onto the surface of another to make a design.
• There are a bunch of great youtube videos on basic sewing skills. Watch whichever ones you need.

Assembly

1. Trace a front, a back, an underside, and the 2 claws onto the wrong side of your fabric with whatever will write on it without bleeding through. Make sure to transfer the blue centerline marks and the red three-point join marks.
2. Cut out the shapes you just traced, leaving about 1” of margin around them. We’ll trim the seams properly later, so don’t worry about getting it exact.
3. Find a couple claw-sized chunks of leftover fabric and pin one to the back of each claw (right sides together, of course).
4. Sew around both claws, leaving the arm ends open so you can turn them. I find it’s easiest to backstitch, and you can get away with stitches up to about 1.5mm apart with normal weight polar fleece.
5. Trim around the outside of the seams on the claws to leave about 1/4” seam allowance, and clip right up to the stitches in the concave spot. If you backstitched, make sure flip them over before trimming the seams so you don’t accidentally cut through the longer stitches.
6. Turn the claws so the right side of the fabric is out and the seams are on the inside, and stuff them with stuffing or fabric scraps. A pair of wooden chopsticks from a fast food place are a great tool for turning and stuffing.
7. Put the front and back pieces right sides together so the points marked in red and blue on the pattern line up. Pin them together.
8. Sew from one red mark to the other along Ferris’s spiky back.
9. Trim around the spikes leaving about 1/4” seam allowance, clipping right up to the seam in the concave spots.
10. Figure out which side is front (hint, it has only 2 legs rather than 4). Imagine where Ferris’s little face will go when he’s finished. Now, pin both claws onto the right side of the front piece, so they’ll be oriented correctly when he’s done. If in doubt, pin the bottom front in place and turn the whole thing inside out to make sure the claws are right.
11. Match the center front of the underside with the center of Ferris’s front (both have a blue + on the pattern). Be sure the pieces have their right sides together and the claws are sandwiched between them.
12. Match the points marked with red triangles on each side of the front and underside together and pin them. If the claws are sticking out at this point, go back to step 10 and try again
13. Sew from one red mark to the other to join Ferris’s front to the front of his underside. Put a few extra stitches in the part of the seam where his “arms”/claws are attached, to make sure they can’t be pulled out.
14. Trim around the 2 tiny legs that you’ve sewn so far, with about 1/8” seam allowance.
15. Now you can applique his face onto the right side of his front. Or embroider it if you know how. Cut the black and white felt scraps into face-shaped pieces and sew them down, giving Ferris whatever expression you want.
16. Line up the 4 back legs on the underside and back pieces, and pin them right sides toether. Sew everything except the part marked in green – that’s the hole through which you’ll turn him inside out.
17. Trim around those last 4 legs, leaving at least 1/8” seam allowance. Don’t cut away any more fabric from the bit marked in green. If you leave a bit of extra fabric around the leg seams, they’ll be harder to turn but require less stuffing.
18. Turn Ferris right side out. Again, chopsticks or the non-pointy end of a barbeque skewer are useful for getting the pointy bits to do the right thing.
19. Stuff Ferris with the filling. I filled mine quite loosely, because it makes him softer and more huggable. If you overfill his body, his spikes will look silly. If you overfill his legs, they’ll stick out in funny directions and not bend right.
20. Tuck the seam allowance back into the hole through which you stuffed Ferris and sew it shut. Congratulations, you have your own toy crab!

The Finished Product

He’s cute, cuddly, and palm-sized. Lego dude for scale.

What happened with left-pad?

The Node community had a lot of drama this week when a developer unpublished a package on which a lot of the world depended.

This was fundamentally possible because NPM offers an unpublish feature. Although the docs for unpublish admonish users that “It is generally considered bad behavior to remove versions of a library that others are depending on!” in large bold print, the feature is available.

What’s the Rust equivalent?

The Rust package manager, Cargo, is similar to NPM in that it helps users get the libraries on which their projects depend. Rust’s analog to the NPM index is crates.io.

The best explanation of Cargo’s robustness against unpublish exploits is the docs themselves:

cargo yank

Occasions may arise where you publish a version of a crate that actually ends up being broken for one reason or another (syntax error, forgot to include a file, etc.). For situations such as this, Cargo supports a “yank” of a version of a crate.:

$ cargo yank --vers 1.0.1
$ cargo yank --vers 1.0.1 --undo

A yank does not delete any code. This feature is not intended for deleting accidentally uploaded secrets, for example. If that happens, you must reset those secrets immediately.

The semantics of a yanked version are that no new dependencies can be created against that version, but all existing dependencies continue to work. One of the major goals of crates.io is to act as a permanent archive of crates that does not change over time, and allowing deletion of a version would go against this goal. Essentially a yank means that all projects with a Cargo.lock will not break, while any future Cargo.lock files generated will not list the yanked version.

As Cargo author Alex Crichton clarified in a GitHub comment yesterday, the only way that it’s possible to remove code from crates.io is to compel the Rust tools team to edit the database and S3 bucket.

Even if a crate maintainer leaves the community in anger or legal action is taken against a crate, this workflow ensures that code deletion is only possible by a small group of people with the motivation and authority to do it in the way that’s least problematic for users of the Rust language.

For more information on the crates.io package and copyright policies, see this internals thread.

But I just want to left pad a string in Rust??

Although a left-pad crate was created as a joke, you should probably just use the format! built-in from the standard library.

terse outputter settings

The outputter takes several state_output setting options. The terse option summarizes the result of each state into a single line.

There are a couple places you can set this:

• Invoke Salt with salt --state-output=terse hostname state.highstate
• Add the line state_output: terse to /etc/salt/minion, if you’re using salt-call
• Setting state_output_terse is apparently an option, though I can’t find any example of a real-world salt config that uses it

Setting the terse option in /etc/salt/minion dropped the output of a highstate from over 10,000 lines to about 2500.

mod + shift + space

Mod was alt by default last time I installed i3, but I’ve since remapped it to the window key (as IRC clients use alt for switching windows). Some people use caps lock as their mod key.

mod + shift + space makes the window floating, which allows it to pop up the confirmation dialog when you try to join a call.

Float windows by default

Alternately, stick the line:

for_window [class="VidyoDesktop"] floating enable

in your ~/.i3/config.

Installing Vidyo despite the libqt4-gui error

Edited as of May 2017: Recent Vidyos depend on a package that’s not available in Ubuntu’s repos. The easiest workaround is:

sudo dpkg -i --ignore-depends=libqt4-gui path/to/VidyoInstaller.deb

Easy Mode

Get some Rust:

curl -sf https://raw.githubusercontent.com/brson/multirust/master/blastoff.sh | sh
multirust update nightly

Get the system dependencies:

sudo apt-get install libcairo2-dev libgdk-pixbuf2.0 libatk1.0 libsdl-pango-dev libgtk-3-dev

Run Notty:

git clone https://github.com/withoutboats/notty.git
cd notty/scaffolding
multirust run nightly cargo run

And there you have it! As mentioned in the notty README, “This terminal is buggy and feature poor and not intended for general use”. Notty is meant as a library for building graphical terminals, and scaffolding is only a minimal proof of concept.

Explanation: Getting Rust

Since the Rust language is still under active development, many features are available in the Nightly version of the compiler which are not yet available in Stable. If you got Rust from your package manager, you probably are using Stable. To check, run rustc --version and see whether the result says “nightly” in it.

Notty uses some features that’re available in Nightly but not Stable. If you try to compile it with Stable, you’ll get an error that makes this obvious:

Compiling notty v0.1.0 (file:///home/edunham/code/notty)
src/lib.rs:16:1: 16:16 error: #[feature] may not be used on the stable release channel
src/lib.rs:16 #![feature(io)]
              ^~~~~~~~~~~~~~~
error: aborting due to previous error
Could not compile `notty`.

When you need to switch between Rust versions frequently, multirust is the tool for the job.

Explanation: Getting system dependencies

I’ve reproduced the following error messages in full to help out any confused new Rustaceans Googling for them:

Cairo is a graphics library that you can get from your system package manager. If you try to compile notty’s dependencies without it, you’ll get an error:

Build failed, waiting for other jobs to finish...
failed to run custom build command for `cairo-sys-rs v0.2.1`
Process didn't exit successfully:
`/home/edunham/code/notty/notty-cairo/target/release/build/cairo-sys-rs-1d0cf50d5d2dab2f/build-script-build`
(exit code: 101)
--- stderr
thread '<main>' panicked at '`"pkg-config" "--libs" "--cflags" "cairo"` did
not exit successfully: exit code: 1
--- stderr
Package cairo was not found in the pkg-config search path.
Perhaps you should add the directory containing `cairo.pc'
to the PKG_CONFIG_PATH environment variable
No package 'cairo' found
', /home/edunham/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/cairo-sys-rs-0.2.1/build.rs:9
note: Run with `RUST_BACKTRACE=1` for a backtrace.

The only other gotcha about the dependencies is that errors about gdk actually mean you need to install the libgtk-3-dev package:

failed to run custom build command for `gdk-sys v0.2.1`
Process didn't exit successfully:
`/home/edunham/code/notty/scaffolding/target/release/build/gdk-sys-e1b0a13b32593729/build-script-build`
(exit code: 101)
--- stderr
thread '<main>' panicked at '`"pkg-config" "--libs" "--cflags" "gdk-3.0"` did
not exit successfully: exit code: 1
--- stderr
Package gdk-3.0 was not found in the pkg-config search path.
Perhaps you should add the directory containing `gdk-3.0.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gdk-3.0' found
', /home/edunham/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/gdk-sys-0.2.1/build.rs:17
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Running notty

Compiling and running scaffolding necessarily builds a bunch of dependencies, some of which throw various warnings. You also might be able to crash scaffolding with an error such as:

thread '<main>' panicked at 'not yet implemented', .../notty/src/datatypes/mod.rs:160

This, along with everywhere else that unimplemented!() occurrs in the notty source code, is an opportunity for you to contribute and help improve the project!

[!] Error resolving “127.0.0.1”: Neither nodename nor servname known

This was the biggest wat, and the hardest to track down because the solution is so obvious.

“Error resolving” sounds like the machine hasn’t been informed of localhost’s existance, so let’s check:

$ cat /etc/hosts
127.0.0.1   localhost
255.255.255.255 broadcasthost
::1             localhost

And I can even ping 127.0.0.1 successfully. So the message and I have different ideas about what it means to “resolve” an IP.

I found the fix here by diffing the stunnel.conf against that on a working machine, and learned that I’d neglected to specify the correct port number on the destination host.

The solution to the “Error resolving localhost” turned out to be specifying the correct port for the other end of the stunnel:

$ cat stunnel.conf
pid =

[9987]
client = yes
accept = 127.0.0.1:9987
cafile = ./cert.pem
verify = 3
connect = 01.23.456.789:9988

Wow. Painfully obvious after you realize what’s wrong, and just plain painful before.

[!] Error binding service [9987] to 127.0.0.1:9987

The “already running?” hint is correct here. This error means stunnel didn’t let go of the port despite failing to start on a previous attempt.

Easy fix; check whether it’s really stunnel hogging the port:

$ lsof -i tcp:9987
COMMAND PID      USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
stunnel 363        me    6u  IPv4 0x20f17e5e0dd35277      0t0  TCP localhost:dsm-scm-target (LISTEN)

and if so, whack it with a metaphorical hammer:

$ sudo killall stunnel

Tada!

After getting the destination IP+port combination specified correctly and the old broken stunnel killed, the stunnel starts successfully.

AWS “Equivalents”

AWS T2 instances have burstable performance. M* instances are general-purpose; C* are compute-optimized; R* are memory-optimized. *3 instances run on slightly older Ivy Bridge or Sandy Bridge processors, while *4 instances run on the newer Haswells. I’m disergarding the G2 (GPU-optimized), D2 (dense-storage), and I2 (IO-optmized) instance types from this analysis.

Note that the AWS specs page has memory in GiB rather than GB. I’ve converted everything into GB in the following table, since the Linode specs are in GB and the AWS RAM amounts don’t seem to follow any particular pattern that would lose information in the conversion.

Hourly price is the Linux/UNIX rate for US West (Northern California) on 2015-12-03. Monthly price estimate is the hourly price multiplied by 730.

Comparison

Linode and AWS do not compare cleanly at all. The smallest AWS instance to match a given Linode type’s RAM typically has fewer vCPUs and costs more in the region where I compared them. Conversely, the smallest AWS instance to match a Linode type’s number of cores often has almost double the RAM of the Linode, and costs substantially more.

Switching from Linode to AWS

When I examine the Servo build machines’ utilization graphs via the Linode dashboard, it becomes clear that even their load spikes aren’t fully utilizing the available CPUs. To view memory usage stats on Linode, it’s necessary to configure hosts to run the longview client. After installation, the client begins reporting data to Linode immediately.

After a few days, these metrics can be used to find the smallest AWS instance whose specs exceed what your application is actually using on Linode.

Sources:

• linode specs
• linode pricing
• AWS specs

Without multirust

If you just want one version of Rust, this blog post by Valérian Galliat has a fix in 7 lines:

cd ~/.rust
wget https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz
tar xf rust-nightly-x86_64-unknown-linux-gnu.tar.gz
mv rust-nightly-x86_64-unknown-linux-gnu rust
export LD_LIBRARY_PATH=~/opt/rust/rustc/lib:$LD_LIBRARY_PATH
export PATH=~/.rust/rust/rustc/bin:$PATH
export PATH=~/.rust/rust/cargo/bin:$PATH

If you want rust stable instead of rust nightly, use the URL https://static.rust-lang.org/dist/rust-stable-x86_64-unknown-linux-gnu.tar.gz in the wget step to download the latest stable release.

If you’re security-conscious, you might want to verify the integrity of the tarball before inflating it and running its contents. We provide a GPG signature of every tarball, and sha256 sums of the tarballs and signatures.

You can construct the URL for shasum or GPG signature by adding the desired extension to the tarball’s URL, so for nightly:

https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz
https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz.sha256
https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz.asc
https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz.asc.sha256

To verify the GPG signature, you’ll also need a copy of the Rust project’s public key. This key is available through several channels:

• on the Rust website, available only over HTTPS.
• on keybase.io, correlated to Rust’s Twitter account and URL. Don’t worry, we authenticated the key by signing a string from Keybase with it locally. We don’t trust them to ever see our private key.
• on GitHub, in the website’s repository.

Remember, verifying the signature only guarantees that the tarball you downloaded matches the one that was produced by the Rust project’s build infrastructure. As with any piece of software, there exist a variety of threat models from which verifying the signatures cannot completely protect you.

Multirust without root

Multirust is a tool that makes it easy to use multiple Rust versions on the same system. Although the absolute easiest way to use it is curl -sf https://raw.githubusercontent.com/brson/multirust/master/blastoff.sh | sh (which will interactively request a sudo password partway through), it can be installed without root as well:

git clone --recursive https://github.com/brson/multirust && cd multirust
./build.sh # create install.sh
mkdir ~/.rust
./install.sh --prefix=~/.rust/
echo "PATH=~/.rust/bin:$PATH" >> ~/.bashrc; source ~/.bashrc

If you run into an error like:

install: WARNING: failed to run ldconfig. this may happen when not installing
as root. run with --verbose to see the error

or in verbose mode,:

install: running ldconfig
/sbin/ldconfig.real: Can't create temporary cache file /etc/ld.so.cache~:
Permission denied
install: WARNING: failed to run ldconfig. this may happen when not installing
as root. run with --verbose to see the error

It means you don’t have permissions to write to /etc/ld.so.cache. Until this issue gets fixed, the easiest workaround to lacking those permissions is to change the script called by the installer to pass -C to ldconfig:

sed -i 's/   ldconfig/   ldconfig -C ~\/.rust\/ld.so.cache/' build/work/multirust-0.7.0/install.sh

Then you should be able to ./install.sh --prefix=~/.rust without the prior warning. Nasty hack, but the easiest way to get it working today.

This technically breaks rustc (since it’s dynamically linked), but if you’re building a Rust project or library, you’ll be using the statically linked cargo tool and thus won’t be affected.

By the way, this is an example of why people who write system utilities like ldconfig should make them able to read their serttings out of environment variables as well as just command-line arguments.

Now you can multirust default nightly to install rust-nightly and configure it as the default, and you’re ready to roll!

Testing your Rust installation

You can now make a package that says “Hello World” in just 5 commands, using a workflow that will scale to packaging and distributing larger projects:

cargo new hello --bin
echo "fn main(){println!(\"Hello World\");}" > hello/src/main.rs
cd hello
cargo build
cargo run

Congratulations, you’re running Rust!

GitHub Tricks

If you’re looking for open issues across all repos owned by a particular user or organization, you can use the search at https://github.com/pulls and specify the “user” (or org) in the search bar. For instance, this search will find all the unassigned, easy-tagged issues in the rust-lang org. Breaking down the search:

• user:rust-lang searches all repos owned by github.com/rust-lang. It could also be someone’s github username.
• is:open searches only open issues.
• no:assignee will filter out the issues which are obviously claimed. Note that some issues without an assignee set may still have a comment saying “I’ll do this!”, if it was claimed by a user who did not have permissions to set assignees and then not triaged.
• label:E-Easy uses my prior knowledge that most repos within rust-lang annotate introductory bugs with the E-easy tag. When in doubt, check the contributing.md file at the top level in the org’s most popular repository for an explanation of what various issue labels mean. If that information isn’t in the contributing file or the README, file a bug!

Am I missing your favorite introductory issue aggregator? Shoot me an email to ___@edunham.net (fill in the blank with anything; the email will get to me) with a link, and I’ll add it here if it looks good!

System Dependencies

There are more now. In order to successfully install all of Buildbot’s dependencies with Pip, I needed a few more apt packages:

python-dev
python-openssl
libffi-dev
libssl-dev

Then for sanity’s sake make a virtualenv, and install the following packages. Note that having too new a sqlalchemy will break things.:

buildbot==0.8.12
boto
pyopenssl
cryptography
SQLAlchemy<=0.7.10

Virtualenvs

Troubleshooting compatibility issues with system packages on a host that runs several Python services with various dependency versions is predictably terrible.

The potential problem with switching to running Buildbot only from a virtualenv is that developers with access to the buildmaster might want to restart it and miss the extra step of activating the virtualenv. I addressed this by adding the command to activate the virtualenv (using the virtualenv’s absolute path) to the ~/.bashrc of the user that we run Buildbot as. This way, we’ve gained the benefits of having our dependencies consolidated without adding the cost of an extra workflow step to remember.

Template changes

Most of Buildbot’s status pages worked fine after the upgrade, but the console view threw a template error because it couldn’t find any variable named “categories”. The fix was to simply copy the new template from venv/local/lib/python2.7/site-packages/buildbot/status/web/templates/console.html to my-buildbot/master/templates/console.html.

That’s it!

Rust currently has these updates on the development buildmaster, but not yet (as of 10/14/2015) in prod.

SSH Agent Forwarding

I have keys AB and BC on host A, where I start. Host A is running ssh-agent, which is installed by default on most Linux distributions.

me@A$ ssh-add ~/.ssh/AB     # Add keypair AB to ssh-agent's keychain
me@A$ ssh-add ~/.ssh/BC     # Add keypair BC to the keychain
me@A$ ssh -A me@B           # Forward my ssh-agent

Now I’m logged into host B and have access to the AB and BC keypairs. An attacker who gains access to B after I log out will have no way to steal the BC keypair, unlike what would happen if that keypair was stored on B.

See here for pretty pictures explaining in more detail how agent forwarding works.

Anyways, I could now ssh me@C with no problem. But if I sudo su user, my agent is no longer forwarded, so I can’t then use the key that I added back on A!

Switch user while preserving environment variables

me@B$ sudo -E su user
user@B$ sudo -E ssh user@C

What?

The -E flag to sudo preserves the environment variables of the user you’re logged in as. ssh-agent uses a socket whose name is of the form /tmp/ssh-AbCdE/agent.12345 to call back to host A when it’s time to do the handshake involving key BC, and the socket’s name is stored in me‘s SSH_AUTH_SOCK environment variable. So by telling sudo to preserve environment variables when switching user, we allow user to pass ssh handshake stuff back to A, where the BC key is available.

Why is sudo -E required to ssh to C? Because /tmp/sshAbCdE/agent.12345 is owned by me:me, and only the file’s owner may read, write, or execute it. Additionally, the socket itself (agent.12345) is owned by me:me, and is not writable by others.

If you must run ssh on B without sudo, chown -R /tmp/ssh-AbCdE to the user who needs to end up using the socket. Making them world read/writable would allow any user on the system to use any key currently added to the ssh-agent on A, which is a terrible idea.

For what it’s worth, the actual value of /tmp/ssh-AbCdE/agent.12345 is available at any time in this workflow as the result of printenv | grep SSH_AUTH_SOCK | cut -f2 -d =.

The Catch

Did you see what just happened there? An arbitrary user with sudo on B just gained access to all the keys added to ssh-agent on A. Simon pointed out that the right way address this issue is to use ProxyCommand instead of agent forwarding.

No, I really don’t want my keys accessible on B

See man ssh_config for more of the details on ProxyCommand. In ~/.ssh/config on A, I can put:

Host B
    User me
    Hostname 111.222.333.444

Host C
    User user
    Hostname 222.333.444.555
    Port 2222
    ProxyCommand ssh -q -w %h:%p B

So then, on A, I can ssh C and be forwarded through B transparently.

Scenario

The role which must apply to both Ubuntu and OSX hosts builds a Rust compiler capable of cross-compiling to Android, so I call it crosscompiler. To run the crosscompiler role on a Mac, you need the xcode role installed, but applying the xcode role to an Ubuntu host will fail.

A simplified version of this setup looks like:

ansible-configs/
├── galaxy_roles.yaml
├── hosts
├── roles
│   ├── crosscompiler
│   │   ├── defaults
│   │   │   └── main.yaml
│   │   ├── meta
│   │   │   └── main.yaml
│   │   └── tasks
│   │       └── main.yaml
│   └── xcode
│       ├── defaults
│       │   └── main.yaml
│       ├── meta
│       │   └── main.yaml
│       └── tasks
│           └── main.yaml
└── site.yaml

Here are a bunch of different ways to avoid applying a Mac-specific task to an Ubuntu host, or vice versa. Note that any of the following steps in isolation will solve the problem – it should not be necessary to use more than one of them.

Check OS on each task of the role

Add the line when: ansible_os_family == 'Darwin' at the end of each task in roles/xcode/tasks/main.yaml.

This needlessly bloats the code and makes it more difficult to read.

Refactor depended role to ignore non-target platforms

Move the entire contents of roles/xcode/main.yaml into roles/xcode/osx.yaml, then create a new main.yaml containing:

---
- include: osx.yaml
  when: ansible_os_family == 'Darwin'

This avoids the bloat induced by running the conditional on each task, while accomplishing the same goal. Now the xcode role looks like:

xcode
├── defaults
│   └── main.yaml
├── meta
│   └── main.yaml
└── tasks
    ├── main.yaml
    └── osx.yaml

This is the best solution for a role which might later expand to support additional platforms.

Make the dependency conditional in meta/main.yaml of depending role

Edit ansible-configs/roles/crosscompiler/main.yaml so that the dependency on xcode reads:

---
dependencies:
  - { role: 'xcode', when: ansible_os_family == 'Darwin' }

This is the best solution when the inner role will only ever target one platform, as is the case with xcode.

Install role conditionally from site.yaml

Edit ansible-configs/site.yaml to read:

- name: Provision cross-compile hosts
  hosts: xcompilehosts
  roles:
    - { role: xcode, when: ansible_os_family == 'Darwin' }
    - crosscompiler

This is problematic because if I was to distribute the crosscompiler role on the Ansible Galaxy, its dependency logic would not be distributed to other users correctly.

TL;DR

You can conditionally include dependencies in your roles. It’s helpful to end users when galaxy roles only try to apply platform-specific tasks to their target platforms, since you can’t be sure how others will use your code.

Epilogue

Once upon a somewhat later time, Jumping School had a reunion and all of the former students attended. Claire and Dave were a little afraid of meeting Aggie, since they’d heard such impressive stories of how far she could jump. All 4 kangaroos tested how far they could jump, just for old times’ sake, and they found that they could all jump about 6 feet! They compared stories about their experiences since leaving school, and found that their rates of improvement had slowed down as they got closer to the limit of how far their species was able to jump.

In reality, red kangaroos only jump about 1.8 meters, which is the only factually accurate part of this entire story.

The Moral

I think that a similar effect distorted my perception of my own competence when I compared myself to past OSL students. One can spot the fallacy pretty easily when everything is spelled out with cute photos: Relative skill levels don’t translate reliably into absolute ones over time. It’s tricker to spot the same fallacy in real life, but you might have an easier time now that you’ve seen the pattern once before.

Photo credits, in order:

• https://en.wikipedia.org/wiki/File:Kangaroo_Island_kangaroos.jpg
• https://www.flickr.com/photos/chrissamuel/5366854530
• https://commons.wikimedia.org/wiki/File:Flying-kangaroo.jpg
• https://www.flickr.com/photos/kaptainkobold/3007004396
• https://en.wikipedia.org/wiki/File:-_fighting_red_kangaroos_1.jpg
• http://opencage.info/pics/files/800_11526.jpg
• https://www.flickr.com/photos/moonfish/3147269483
• https://www.flickr.com/photos/chrissamuel/5480184535

Get the files

I bought Don’t Starve from the Humble Bundle store, although there are other methods of obtaining it which strike a different balance between cost and convenience.

The downloaded file is dontstarve_x64_july21.tar.gz.

This Just Works

$ yaourt -S libcurl-compat
$ tar -xvf dontstarve_x64_july21.tar.gz
$ cd dontstarve/bin
$ LD_PRELOAD=libcurl.so.3 ./dontstarve

Below the fold is the troubleshooting process I went through to make it look so easy. Hopefully it’ll be of assistance to those searching for the errors that I ran into!

This part has SEO for all the intermediate errors

$ cd dontstarve
$ ./dontstarve.sh
sh: symbol lookup error: sh: undefined symbol: rl_signal_event_hook
sh: symbol lookup error: sh: undefined symbol: rl_signal_event_hook
sh: symbol lookup error: sh: undefined symbol: rl_signal_event_hook
sh: symbol lookup error: sh: undefined symbol: rl_signal_event_hook
sh: symbol lookup error: sh: undefined symbol: rl_signal_event_hook
Fontconfig error: "/etc/fonts/conf.d/10-scale-bitmap-fonts.conf", line 70: non-double matrix element
Fontconfig error: "/etc/fonts/conf.d/10-scale-bitmap-fonts.conf", line 70: non-double matrix element
Fontconfig warning: "/etc/fonts/conf.d/10-scale-bitmap-fonts.conf", line 78: saw unknown, expected number

(updater:2135): Pango-WARNING **: failed to choose a font, expect ugly output. engine-type='PangoRenderFc', script='latin'

(updater:2135): Pango-WARNING **: failed to choose a font, expect ugly output. engine-type='PangoRenderFc', script='common'
./dontstarve.sh: line 2: unexpected EOF while looking for matching ``'
./dontstarve.sh: line 4: syntax error: unexpected end of file

$ bin/dontstarve.sh
$ bin/dontstarve.sh: line 3: ./dontstarve: No such file or directory

$ cd bin
$ ./dontstarve.sh
./dontstarve: /usr/lib/libcurl.so.4: version `CURL_OPENSSL_3' not found (required by ./dontstarve)

$ bin/dontstarve # Spoiler, this will still be broken later
bin/dontstarve: /usr/lib/libcurl.so.4: version `CURL_OPENSSL_3' not found (required by bin/dontstarve)

$ yaourt -S libcurl-compat

Sometimes you have to preload library!
 e.g. if you see this message:
 /usr/lib/libcurl.so.4: version `CURL_OPENSSL_3' not found'
 Do this:
 LD_PRELOAD=libcurl.so.3 youprogname

$ LD_PRELOAD=libcurl.so.3 bin/dontstarve

ERROR: Missing Shader 'shaders/font.ksh'.
Assert failure '0' at ../source/renderlib/OpenGL/HWEffect.cpp(86)

Assert failure 'BREAKPT:' at ../source/renderlib/OpenGL/HWEffect.cpp(86)

Assert failure 'datasize + mReadHead <= mBufferLength' at ../source/util/reader.h(28)

Assert failure 'BREAKPT:' at ../source/util/reader.h(28)

Segmentation fault (core dumped)

$ cd bin
$ LD_PRELOAD=libcurl.so.3 ./dontstarve

~/Downloads/dontstarve/bin $ LD_PRELOAD=libcurl.so.3 ./dontstarve
bash: ./dontstarve: No such file or directory

How?

First, check if the feature is already implemented. man date and try not to giggle. Search for morning. It’s not there.

So we need a switch/case:

case EXPRESSION in CASE1) COMMAND-LIST;; CASE2) COMMAND-LIST;; ... CASEN) COMMAND-LIST;; esac

And the expression we’re switching on will be the current hour:

date +%H

My first attempt does not work because I expect too much of Bash:

case $(date +%H) in [0-12]) m="morning";;[13-18]) m="afternoon";;[19-21])m="evening";;*)m="night";;esac; echo $m

It fails because the “ranges” are actually just shell patterns.

I could either expand my script to handle all hours, or compress ranges of hours down into something that can be expressed by patterns. The latter sounds shorter and easier. I want to divide the current hour by 6, to tell which quarter of the day I’m in.

A bit of trial and error reveals that a syntax that allows me to do math on the result of date is:

$(( $(date +%H)/6 ))

because it’s shorthand for assigning the result of the math into a variable and using it immediately. This only adds a few characters to the one-liner:

case $(($(date +%H)/6)) in 0|1)m="morning";;2)m="afternoon";;3)m="night";;esac; echo $m

That’s it!

Troubleshooting

Dave Huseby pointed out the problem on IRC when the slaves that he runs were unable to connect to the buildmaster:

16:48:23 <&brson> edunham: dhuseby said this earlier <huseby> it seems like the verify=3 in the stunnel config is the problem
16:48:39 <&brson> if he changed 'verify' to some other value in the stunnel config it worked

A quick check fo the stunnel docs shows that verify=3 is the strictest setting, and will fail if the locally installed cert isn’t right. This supports the hypothesis that our cert might be expired. On the buildmaster, I found the cert and examined its metadata:

$ find . -type f -name "*.pem"
$ openssl x509 -noout -issuer -subject -dates -in certname.pem

On the old cert, the results contained:

$ openssl x509 -noout -issuer -subject -dates -in rust-bot-cert.pem
issuer= /
    O=Rust Project/
    OU=Bot/
    CN=bot.rust-lang.org/
    emailAddress=admin@rust-lang.org
subject= /
    O=Rust Project/
    OU=Bot/
    CN=bot.rust-lang.org/
    emailAddress=admin@rust-lang.org
notBefore=Jul 14 02:28:50 2012 GMT
notAfter=Jul 14 02:28:50 2015 GMT

This tells me that the cert was created in 2012 and had its expiry set for the seemingly distant future of 2015.

Make a New Cert

To determine whether the old key had a passphrase on it, go openssl rsa -check -in keyname.pem. It writes the private key to your terminal if there’s no password, or prompt for a passphrase if the key has one.

The stunnel docs give most of the relevant incantation. Since no file in our buildbot directory is named precisely stunnel.conf, make cert doesn’t quite work right. But it works fine to manually run a variant of the command given in the docs:

$ openssl req -new -x509 -days 3650 -nodes -out cert.pem -keyout key.pem

That prompted me for a variety of information, which I entered where applicable and left blank where it wasn’t. The metadata is primarily for the benefit of others verifying that a cert belongs to the correct person, which isn’t a relevant concern in our use case.

I then backed up the old key and cert (although they’re no longer usable, they contain a bunch of metadata that I didn’t know whether I’d need later) and moved the new key and cert to match the old ones’ original file names.

Finally, I updated the repository with the new certificate.

Make Buildbot spin up AMIs with the new cert

This was the tricky bit. Since the slave image does not pull updates to its copy of the Rust Buildbot github repo when it boots, the file had to be statically edited and then the AMIs re-saved. But Buildbot makes its instance requests based on AMI ID, and the IDs are unique to a particular image. So the workflow goes:

• Figure out which AMI Buildbot will spin up for a given job
• Spin up an instance of the AMI
• Remote into it and manually update the cert
• Save the instance into a new AMI, noting its ID
• Update Buildbot’s configuration with the new ID

ami-b74fa1f3 -- windows
ami-7fd23e3b -- generic linux
ami-381e197d -- android
ami-dbac5f9f -- centos5 (builds snapshots that work with ancient Linux)

Spin up an instance of the AMI

Since there were only 4, I did this manually. If it was a recurring task or there had been more AMIs, I would have automated this part of the process.

In the AWS Console, go to EC2, then click the AMIs link under “Images” at the left.

Search for the AMI ID in the search box. Only one AMI is found, because IDs are unique. Then click the big blue Launch button up at the left.

The only “gotcha” in the ensuing 7-step process is making sure to put the instance into the correct security group. I spun my temporary instances up into the same group as a host I know I can get to from the Bastion server with my credentials, to reduce the number of steps I’d have to troubleshoot if they were difficult to access.

It also helps to tag the spot request with the name of the AMI it was created from, when processing several at once.

Kick Buildbot a bit

After the AMIs and FreeBSD, Bitrig, and Mac builders all had the new cert, I restarted Buildbot on the buildmaster and reran its script for creating the stunnels. Althoug it didn’t gracefully pick up where it had left off on partially built pull requests, closing then re-opening the PRs caused it to notice them and resume building successfully.

Hopefully TaskCluster gets OSX support soon, so we can start switching off of Buildbot.

The Easy Way

There are these cool things called iframes that basically let you put websites in your websites...

All you have to do is stick this one line in your page’s source, and modern browsers will go load and inject the specified page – in this case, the playpen. The exact technique for injecting raw HTML will differ based on your blogging platform. With Tinkerer, the source will look like this:

.. raw:: html

    <iframe src="https://play.rust-lang.org/" style="width:100%; height:400px;"></iframe>

That’ll render with the default contents of the playpen. Note that it’ll someties try to be clever and load the code that the viewer most recently opened in it when loaded with no arguments: