Adventures in Mercurial

I adore Git, but have needed to ramp up my Mercurial (Hg) skills recently to dig prior work related to my current tasks out of a repo’s history. Here are some things I’m learning:

Command Equivalences

As this tutorial so helpfully explains, the two VCSes aren’t all that dissimilar under their hoods. I condensed the command comparison table into a single page and printed it out for quick reference; a PDF is here.

Clone

The thing I want to clone lives at http://hg.mozilla.org/hgcustom/version-control-tools/file/tip/autoland.

Trying to clone the full URL yields a 404, but snipping the URL back to the top-level directory gets me the repo:

$ hg clone http://hg.mozilla.org/hgcustom/version-control-tools/
destination directory: version-control-tools
requesting all changes
adding changesets
adding manifests
adding file changes
added 4574 changesets with 10874 changes to 1971 files
updating to bookmark @
1428 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ ls
version-control-tools

Examine Log

hg log | less shows me that each commit’s summary in this repo includes the part of the codebase it touches, and a bug number.

hg log | grep autoland: | less gives me the summaries of every commit that touched autoland, but I cannot show a commit from summary alone.

The Hg book helped me construct a filter that will show a unique revision ID onthe same line as each description.

hg log --template '{rev} {desc}\n' | grep autoland: is much more useful. It gives me the local ID of each changeset whose description included “autoland:”.

From here, I can use a bit more grep to narrow down the list of matching messages, then I’m ready to examine commits.

Examining Commits

That {rev} in my filter was the “repository-local changeset revision number”. For these examples I’ll examine revision 2589.

hg status --change 2589 lists the files that were touched by that revision, and hg export 2589 yields a full diff of the changes introduced.

This gets me enough information to make an appropriate set of changes, run the tests, and create my own commits!

Thinkpad 13 Trackpoint slowdown in i3 window manager

As has been mentioned on Reddit, the Thinkpad 13 trackpoint settings aren’t in the same place as those of older thinkpads. Despite some troubleshooting, I haven’t yet found what files to edit to adjust the trackpoint’s speed and sensitivity in Ubuntu 16.04.

The trackpoint has been slightly sluggish/unresponsive when I use the i3 window manager, and has additional intermittent slowdowns when using Chromium and Firefox in i3.

Although I don’t yet know the right way to fix trackpoint sensitivity on this machine, I accidentally discovered a highly effective workaround today:

  • Log into Unity (the default desktop that Ubuntu ships with) and configure the mouse and input settings as desired
  • Log out, and get back into i3wm
  • Launch unity-settings-daemon
  • And suddenly, the mouse works correctly the way it did in Unity!

I fully realize that this is a nasty hack around identifying and solving the actual problem, but it succeeds at making the mouse responsive while minimizing time spent troubleshooting.

Hieroglyph and Tinkerer Dependencies

In setting up virtualenvs for my slides and blog repos on my new laptop, I’ve been reminded that a variety of Sphinx-based tools require system dependencies as well as the ones in their virtualenvs.

Hieroglyph dependency issues

The error resulting from pip install -r requirements.txt ended with:

Command ".../virtualenv/bin/python2 -u -c
"import setuptools,
tokenize;__file__='/tmp/pip-build-lzbk_r/Pillow/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
install --record /tmp/pip-BNDc_6-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/home/edunham/repos/slides/rustcommunity/v/include/site/python2.7/Pillow"
failed with error code 1 in /tmp/pip-build-lzbk_r/Pillow/

Its fix, from stackoverflow, was:

$ sudo apt-get install libtiff5-dev libjpeg8-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python-tk
$ pip install -r requirements.txt

Tinkerer depencencies, too!

pip install -r requirements.txt over in my blog repo yielded:

Command ".../virtualenv/bin/python2 -u -c "import setuptools,
tokenize;__file__='/tmp/pip-build-NVLSBY/lxml/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
install --record /tmp/pip-qD5QIe-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/home/edunham/repos/site/v/include/site/python2.7/lxml" failed with error code
1 in /tmp/pip-build-NVLSBY/lxml/

The fix is again to install the missing system deps, on Ubuntu:

$ sudo apt-get install libxml2-dev libxslt-dev
$ pip install -r requirements.txt

That’s it! I’m writing this down for SEO on the specific errors at hand, since the first several useful hits are currently stackoverflow.

If you’re a pip developer reading this, please briefly contemplate whether it’d be worthwhile to have some built-in mechanism to translate common dependency errors to the appropriate system package names needed based on the OS on which the command is run.

CFPs Made Easier

Check out this post by Lucy Bain about how to come up with an idea for what to talk about at a conference. I blogged last year about how I turn abstracts into talks, as well. Now that the SeaGL CFP is open, it’s time to look in a bit more detail about the process of going from a talk idea to a compelling abstract.

In this post, I’ll walk you through some exercises to clarify your understanding of your talk idea and find its audience, then help you use that information to outline the 7 essential parts of a complete abstract.

Getting ready to write your abstract

Your abstract is a promise about what your talk will deliver. Have you ever gotten your hopes up for a talk based on its abstract, then attended only to hear something totally unrelated? You can save your audience from that disappointment by making sure that you present what your abstract says you will.

I find the abstract to be the hardest part of the talk to write, because it sets the stage for every other part of it. If your abstract is thorough and clear about what your talk will deliver, you can refer back to it throughout the writing process to make sure you’re including the information that your audience is there for!

For both you and your audience to get the most out of your talk, the following questions can help you refine your talk idea before you even start to write its abstract.

Why do you love this idea?

Start working on your abstract by taking some quick notes on why you’re excited about speaking on this topic. There are no wrong answers! Your reasons might include:

  • Document a topic you care about in a format that works well for those who learn by listening and watching
  • Impress a potential employer with your knowledge and skills
  • Meet others in the community who’ve solved similar problems before, to advise you
  • Recruit contributors to a project
  • Force yourself to finish a project or learn more detail about a tool
  • Save novices from a pitfall that you encountered
  • Travel to a conference location that you’ve always wanted to visit
  • Build your resume
  • Or something else entirely!

Starting out by identifying what you personally hope to gain from giving the talk will help ensure that you make the right promises in your abstract, and get the right people into the room.

What’s your idea’s scope?

Make 2 quick little lists:

  • Topics I really want this presentation to cover
  • Topics I do not want this presentation to cover

Once you think that you have your abstract all sorted out, come back to these lists and make sure that you included enough topics from the first list, and excluded those from the second.

Who’s the conference’s target audience?

Keynotes and single-track conferences are special, but generally your talk does not have to appeal to every single person at the conference.

Write down all the major facts you know about the people who attend the conference to which you’re applying. How young or old might they be? How technically expert or inexperienced? What are their interests? Why are they there?

For example, here are some statements that I can make about the audience at SeaGL:

  • Expertise varies from university students and random community members to long-time contributors who’ve run multiple FOSS projects.
  • Age varies from a few school-aged kids (usually brought by speakers and attendees) to retirees.
  • The audience will contain some long-term FOSS contributors who don’t program, and some relatively expert programmers who might have minimal involvement in their FOSS communities
  • Most attendees will be from the vicinity of Seattle. It will be some attendees’ first tech conference. A handful of speakers are from other parts of the US and Canada; international attendees are a tiny minority.
  • The audience comes from a mix of socioeconomic backgrounds, and many attendees have day jobs in fields other than tech.
  • Attendees typically come to SeaGL because they’re interested in FOSS community and software.

Where’s your niche?

Now that you’ve taken some guesses about who will be reading your abstract, think about which subset of the conference’s attendees would get the most benefit out of the topic that you’re planning to talk about.

Write down which parts of the audience will get the most from your talk – novices to open source? Community leaders who’ve found themselves in charge of an IRC channel but aren’t sure how to administer it? Intermediate Bash users looking to learn some new tricks?

If your talk will appeal to multiple segments of the community (developers interested in moving into DevOps, and managers wondering what their operations people do all day?), write one question that your talk will answer for each segment.

You’ll use this list to customize your abstract and help get the right people into the room for your talk.

Still need an idea?

Conferences with a diverse audience often offer an introductory track to help enthusiastic newcomers get up to speed. If you have intermediate skills in a technology like Bash, Git, LaTeX, or IRC, offer an introductory talk to help newbies get started with it! Can you teach a topic that you learned recently in a way that’s useful to newbies?

If you’re an expert in a field that’s foreign to most attendees (psychology? beekeeping? Cray Supercomputer assembly language?), consider an intersection talk: “What you can learn from X about Y”. Can you combine your hobby, background, or day job with a theme from the conference to come up with something unique?

The Anatomy of an Abstract

There are many ways to structure a good abstract. Here are the 7 elements that I try to always include:

  1. Set the scene with a strong introductory sentence, which reminds your target audience of your topic’s relevance to them. Some of mine have included:

    • “Rust is a systems programming language that runs blazingly fast, prevents segfaults, and guarantees thread safety.”
    • “Git is the most popular source code management and version control system in the open source community.”
    • “When you’re new to programming, or self-taught with an emphasis on those topics that are directly relevant to your current project, it’s easy to skip learning about analyzing the complexity of algorithms.”
  2. Ask some questions, which the talk promises to answer. These questions should be asked from the perspective of your target audience, which you identified earlier. This is the least essential piece of an abstract, and can be skipped if you make sure your exposition clearly shows that you understand your target audience in some other way. Here are a couple of questions I’ve used in abstracts that were accepted to conferences:

    • “Do you know how to control what information people can discover about you on an IRC network?”
    • “Is the project of your dreams ignoring your pull requests?”
  3. Drop some hints about the format that the talk will take. This shows the selection commitee that you’ve planned ahead, and helps audience members select session that’re a good fit for their learning styles. Useful words here include:

    • “Overview of”
    • “Case study”
    • “Demonstration”
    • “Deep dive into”
    • “Outline X principles for”
    • “Live coding”
  4. Identify what background knowledge the audience will need to get the talk’s benefit, if applicable. Being specific about this helps welcome audience members who’re undecided about whether the talk is applicable to them. Useful phrases include:

    • “This talk will assume no background knowledge of...”
    • “If you’ve used ____ to ____, ...”
    • “If you’ve completed the ____ tutorial...”
  5. State a specific benefit that audience members will get from having attended the talk. Benefits can include:

    • “Halve your Django website’s page load times”
    • “Get help on IRC”
    • “Learn from ____‘s mistakes”
    • “Ask the right questions about ____
  6. Reinforce and quantify your credibility. If you’re presenting a case study into how your company deployed a specific tool, be sure to mention your role on the team! For instance, you might say:

    • “Presented by [the original author | a developer | a maintainer | a long-term user] of [the project], this talk will...”
  7. End with a recap of the talk’s basic promise, and welcome audience members to attend.

These 7 pieces of information don’t have to each be in their own sentence – for instance, establishing your credibility and indicating the talk’s format often fit together nicely in a single sentence.

Once you’ve got all of the essential pieces of an abstract, munge them around until it sounds like concise, fluent English. Get some feedback on helpmeabstract.com if you’d like assistance!

Give it a title

Naming things is hard. Here are some assorted tips:

  • Keep it under about 50 characters, or it might not fit on the program
  • Be polite. Rude puns or metaphors might be eye-catching, but probably violate your conference or community’s code of conduct, and will definitely alienate part of your prospective audience.
  • For general talks, it’s hard to go wrong with “Intro to ___” or “___ for ___ users”.
  • The form “[topic]: A [history|overview|melodrama|case study|love story]” is generally reliable. Well, I’m kidding about “melodrama” and “love story”... Mostly.
  • Clickbait is underhanded, but it works. “___ things I wish I’d known about ___”, anyone?

Good luck, and happy conferencing!

2ish weeks with the Thinkpad 13

I recently got a Thinkpad 13 to try replacing my X230 as a personal laptop. Here’s the relevant specs from my order confirmation:

Battery 3cell 42Wh
System Unit 13&S2 IG i5-6200U NvPro
Camera 720p HD Camera
AC Adapter and Power Cord 45W 2pin US
Processor Intel Core i5-6200U MB
Hard drive 128GB SSD SATA3
Keyboard Language KYB SR ENG
Publication Language PUB ENG
Total memory 4GB DDR4 2133 SoDIMM
OS DPK W10 Home
Pointing device 3+2BCP NoFPR SR
Preload Language W10 H64-ENG
Preload OS Windows 10 Home 64
Preload Type Standard Image
TPM Setting Software TPM Enabled
Display Panel 13&S2 FHD IPS AG AL SR
WiFi wireless LAN adapters Intel 8260AC+BT 2x2 vPro

I picked mediocre CPU and RAM because the RAM’s easy to upgrade, and I’m curious about whether I actually need top-of-the-line CPUs to have an acceptable experience on a personal laptop.

Why the 13?

I had a few hard requirements for my next personal laptop:

  • Trackpoint with buttons
  • Decent key travel (the X1 carbon has 1.86mm and typing on it for too long made my hands hurt)
  • USBC port
  • Under $1,000

Plus a few nice-to-haves:

  • Small and light are nice, including charger
  • Screen not much worse than 1920x1080
  • Good battery life
  • Metal case and design for durability make me happy
  • My house is already full of Thinkpad chargers, so a laptop that uses them helps reduce additional clutter

I’ll be the first to admit that this is an atypical set of priorities. My laptop is home to Git, Vim, and a variety of tools for interacting with the internet, so the superficial I/O differences matter more to me than the machine’s internal specs.

Things I like about the 13

  • 2.1mm key travel is everything I hoped for. At least, I’ve used it all day and my hands don’t hurt.
  • Battery life is pretty decent, and battery will be easy to replace when it starts to fail.
  • Light-enough weight. Lighter charger than other Thinkpads I’ve had.
  • Smallest Thinkpad charger that I’ve ever seen.
  • Case screws are all captive.
  • Mystery hole in the bottom case turns out to be a highly convenient hard shutdown button.
  • Hinges feel pretty solid and hold the screen up nicely.
  • No keyboard backlight. I dislike them.
  • 4GB of RAM is a single stick, easy to add more (and I’ll need to for a smoother web browsing experience; neither Firefox nor Chromium is particularly happy with only 4GB)
  • A vanilla Ubuntu 16.04 iso Just Worked for installing Linux. It must have shipped with whatever magic signatures were required to play nice with the new security measures, because the install process was delightfully non-thought-provoking.
  • ~7mm plastic bezel between buttons and trackpad reduces likelihood of accidentally moving cursor when clicking.
../../../_images/13-button-bezel.jpg
  • Screen’s the same as my X240, xrandr calls it 1920x1080 294mm x 165mm. This fits 211 columns of a default font, or 127 columns of a font that’s comfortably legible when the laptop is on the other side of my desk.

Nitpicks about the 13

  • Color.

When I purchased mine at the end of April, only the silver chassis had a metal lid and shipped with a nice screen by default (the higher-res screen is available in the all-plastic black model for an additional charge). So now I own a non-black laptop for the first time since my Dell Latitude D410 in high school. The screen bezel and keys are black, though, and if I really cared I could probably paint the rest of it.

  • Power button.
../../../_images/13-power-button.jpg

It feels horribly... squishy? There’s no satisfying click to tell me when I’ve pushed it far enough. Holding it for 10 seconds only sometimes shuts the laptop off (though there’s a reset switch on the mobo accessible by a paperclip-hole in the bottom panel which forces shutdown instantly when pushed). There’s a circle on the pwoer button that looks like it might be an LED, but it never lights up.

  • Cutesy font.
../../../_images/13-lenovo-font.jpg

This is a tiny nitpick, but they’ve changed the Lenovo logo on the lid, pre-BIOS boot screen, and screen bezel from the already-mediocre font to a super condescending, childish, roundy one. Fortunately the lid one is easily hidden under some stickers.

  • Bottom panel held on by clips as well as screws.

More on this one in the disassembly section below, but I’m afraid they’ll break with how often I take my laptops apart.

  • Mouse buttons feel cheap and plastic-ey.

They feel like thin plastic shells instead of solid buttons like on older Thinkpads. I’m not sure precisely why they feel that way, but it’s a reminder that you’re using a lower-end machine.

  • Longest side is about 1cm greater than the short side of a TSA xray tub.

My X240 fits perfectly along the short end of the tub, leaving room for my shoes beside it. I have to use two tubs or separate my pair of shoes when putting the 13 through the scanner. (see, I wasn’t kidding when I said “nitpicks”)

  • The Trackpoint top is not interchangeable with those of older Thinkpads.
../../../_images/13-trackpoints.jpg

The round part is the same size, but the square hole in the bottom is about 2mm to a side rather than the 4mm of the one on an x220 keyboard. Plus the cylinder bit is about 2mm long rather than the x220’s 3.5mm, so even with an adapter for the square hole, older trackpoints would risk leaving marks on the screen.

  • The fan is a little loud.

I anticipate that this will get a lot less annoying when I upgrade to 16 or 32GB of ram and maybe tune it in software using thinkfan.

Thinkpad 13 partial disassembly photos

To get the bottom case off, pull all the visible screws and also remove the 3 tiny rubber feet from under the palm rest. I stuck my tiny rubber feet in a plastic bag and filed it away, because repeated removal would eventually destroy the glue and get them lost.

../../../_images/13-slide-and-pry.jpg

The bottom case comes off with a combination of sliding and prying. Getting it back on again requires sliding the palmrest edge just right, then snapping the sides and back on before the palm rest slips out of place. It’s tricky.

../../../_images/13-bendy-battery.jpg

The battery is easily removed by pulling out a single (non-captive) screw. It seems to be a thin plastic wrapper around 3 cell phone batteries. The battery has no glue holding it in, just screws.

../../../_images/13-mobo.jpg

Here’s its guts, with battery removed.

../../../_images/13-mobo-annotated.jpg

Note the convenient hard power cycle button (accessible via a tiny hole in the bottom case when assembled), pair of RAM slots and SSD form factor, and airspace compartment that almost looks intended for hiding half a dozen very small items. The coin cell battery (in sky blue shrink wrap) flaps around awkwardly when the machine is disassembled, but at least it’s not glued down.

Reflections on my first live webcast

This morning, I participated in the O’Reilly Emerging Languages Webcast with my “Rust from a Scripting Background” talk. Here’s how it went.

Preparation

I was contacted about a month before my webcast and asked to present my OSCON talk as part of the event. I explained why my “How to learn Rust” talk didn’t sound like a good fit for the emerging languages webcast, and suggested the “Starting Rust from a Scripting Background” talk that I gave at my local Rust meetup recently as an alternative.

After we agreed on what talk would be suitable, O’Reilly’s Online Events Producers emailed me a contract to e-sign. The contract gives O’Reilly the opportunity to reuse and redistribute my talk from the webcast, and promises me a percentage of the proceeds if my recording is sold, licensed, or otherwise used to make them money.

During the week before the webcast, I did a test call in which an O’Reilly representative walked me through how to use the webcast software and verified that my audio was good on the phone I planned to use for the webcast.

A final copy of the slides for the webcast, in my option of PDF or Powerpoint, was due at 5pm the day before the event.

The Software (worked for me on Ubuntu)

O’Reilly Media provided an application called “Presentation Manager XD” from on24.com that presenters log into during the event.

According to my email from O’Reilly, the requirements for the event are:

  • Slides - PowerPoint or PDF only please with no embedded video or audio. Screen ratio of 4:3
  • Robust Internet connection
  • Clear, reliable phone line.
  • Windows 7 or 8 running IE 8+, Firefox 22+ or Chrome 27+
  • Mac OS 10.6+ running Firefox 22+ or Chrome 27+
  • Latest version of Flash Player
  • If you plan to share your screen, you will need to install a small application - you will be prompted to install it the first time you log into the platform.

Some of these requirements are lies. I used Firefox 46.0 on Ubuntu 14.04. I did rewrite my slides in LibreOffice because it emits better PDFs than the HTML tools I normally use, but I was also looking for an excuse to rewrite them to clean up their organization.

I clicked around in the “Presentation Manager XD” UI and downloaded a file called “ON24-ScreenShare-plugin”, then chmod +x‘d it and executed it with ./ON24-ScreenShare-plugin. This caused Wine to run, install some Gecko stuff, and start the screenshare plugin sufficiently well to share my screen to the webcast tool.

I had to re-run the plugin in Wine after logging out of and back into my window manager, of course. Additionally, the screenshare window’s resizing is finicky. It’s fine to grab and drag the highlighted parts of the window’s border with the mouse, but the meta+click command with which one usually moves windows in i3 causes the sides of the screenshare window to move independently of each other.

Here’s what the webcast UI looked like during streaming, just at the end of the Kotlin talk while I was getting ready to start mine:

../../../_images/oreillywebcast.png

The Talk

As previously mentioned, I rewrote my talk in LibreOffice Impress – ostensibly to get a prettier PDF, but also because it’s been a month or two since I last prepared for it and re-writing helps me refresh my memory and verify that all my facts are up to date.

GUI-based slide editing is downright painful after using rst-based tools for so long, especially because LibreOffice has no good way to embed code samples out of the box. I ended up going with screenshots from the Rust playground, which were probably better than embedded code samples, but relearning how to edit slides like a regular person wasn’t a pleasant experience.

I took more notes than I normally do, since nobody on the webcast could see whether I was reciting or reading. I’m glad I did, as having the notes on a physical page in front of me was reassuring and helped me avoid missing any important points.

I rehearsed the timing of each section of my slides individually, since it naturally broke down into 7 or so discrete parts and I had previously calculated how much of my hour to allocate to each section. Most sections ran consistently over time when preparing, yet under time during the actual talk. The lesson here is to rehearse until I’m happy with a section and can make it the same duration twice in a row.

The experience of presenting a talk in a subjectively empty room made me realize just what high-bandwidth communication regular conferences are.

Pros:

  • No need to worry about eye contact
  • All the notes you want
  • Can’t see anyone sleeping
  • Chat channel allowed instant distribution of links
  • Chat channel allowed expert attendees to help answer questions
  • Presentation software allowed gracefully skipping slides, rather than the usual paging back and forth with the arrow keys

Cons:

  • Can’t take quick surveys by show of hands
  • Negligible feedback on how many people are there and their body language of engagement/disengagement
  • Silences are super awkward
  • Can’t see the shy attendees, in order to encourage participation

The audience asked fewer questions during the talk than I expected. Fortunately, they came up with plenty of questions at the end – extra fortunate because I overcompensated on time and finished my slides about 15 minutes before the end of my speaking slot!

Q&A was surprisingly relaxing, as it was totally up to me which questions to answer. I’ll admit that I did select in favor of those that I could answer concisely and eloquently, deferring the questions that didn’t make as much sense to think about them while I answered easier ones.

tl;dr

In my experience, presenting a webcast was lower-stress and comparably impactful to a conference talk.

For would-be presenters concerned about their or the audience’s appearance, the visual anonymity of a webcast could be a great place to start a speaking career.

Speakers accustomed to presenting in rooms full of humans should expect subtle feedback, like nods, smiles, and laughter, to be totally invisible in a webcast environment.

And if O’Reilly asks you to do a webcast with them, I’d say go for it – they made the whole experience as seamless and easy as possible.

Paths Into DevOps

../../../_images/twitter.png

Today, Carol asked me about how a current sysadmin can pivot into a junior “devops” role. 10 tweets into the reply, it became obvious that my thoughts on that type of transition won’t fit well into 140-character blocks.

My goal with this post is to catalog what I’ve done to reach my current level of success in the field, and outline the steps that a reader could take to mimic me.

Facets of DevOps

In my opinion, 3 distinct areas of focus have made me the kind of person from whom others solicit DevOps advice:

  • Cultural background
  • Technical skills
  • Self-promotion

I place “cultural background” first because many people with all the skills to succeed at “DevOps” roles choose or get stuck with other job titles, and everyone touches a slightly different point on the metaphorical elephant of what “DevOps” even means.

Cultural Background

What does “DevOps” mean to you?

  • Sysadmins who aren’t afraid of automation?
  • 2 sets of job requirements for the price of 1 engineer?
  • Developers who understand what the servers are actually doing?
  • Reducing the traditional divide between “development” and “operations” silos?
  • A buzzword that increases your number of weekly recruiter emails?
  • People who use configuration management, aka “infrastructure as code”?

From my experiences starting Oregon State University’s DevOps Bootcamp training program, speaking on DevOps related topics at a variety of industry conferences, and generally being a professional in the field, I’ve seen the term defined all of those ways and more.

Someone switching from “sysadmin” to “devops” should clearly define how they want their day-to-day job duties to change, and how their skills will need to change as a result.

Technical Skills

The best way to figure out the technical skills required for your dream job will always be to talk to people in the field, and read a lot of job postings to see what you’ll need on your resume and LinkedIn to catch a recruiter’s eye.

In my opinion, the bare minimum of technical skills that an established sysadmin would need in order to apply for DevOps roles are:

  • Use a configuration management tool – Chef, Puppet, Salt, or Ansible – to set up a web server in a VM.
  • Write a script in Python to do something more than Hello World – an IRC bot or tool to gather data from an API is fine.
  • Know enough about Git and GitHub to submit a pull request to fix something about an open source tool that other sysadmins use, even if it’s just a typo in the docs.
  • Understand enough about continuous integration testing to use it on a personal project, such as TravisCI on a GitHub repo, and appreciate the value of unit and integration tests for software.
  • Be able to tell a story about successfully troubleshooting a problem on a Linux or BSD server, and what you did to prevent it from happening again.

Keep in mind that your job in an interview is to represent what you know and how well you can learn new things. If you’re missing one of the above skills, go ask for help on how to build it.

Once you have all the experiences that I listed, you are no longer allowed to skip applying for an interesting role because you don’t feel you know enough. It’s the employer’s job to decide whether they want to grow you into the candiate of their dreams, and your job to give them a chance. Remember that a job posting describes the person leaving a role, and if you started with every skill listed, you’d probably be bored and not challenged to your full potential.

Self Promotion

“DevOps” is a label that engineers apply to themselves, then justify with various experiences and qualifications.

The path to becoming a community leader begins at engaging with the community. Look up DevOps-related conferences – find video recordings of talks from recent DevOps Days events, and see what names are on the schedules of upcoming DevOps conferences.

Look at which technologies the recent conferences have discussed, then look up talks about them from other events. Get into the IRC or Slack channels of the tools you want to become more expert at, listen until you know the answers to common questions, then start helping beginners newer than yourself.

Reach out to speakers whose talks you’ve enjoyed, and don’t be afraid to ask them for advice. Remember that they’re often extremely busy, so a short message with a compliment on their talk and a specific request for a suggestion is more likely to get a reply than overly vague requests. This type of networking will make your name familiar when their companies ask them to help recruit DevOps engineers, and can build valuable professional friendships that provide job leads and other assistance.

Contribute to the DevOps-related projects that you identify as having healthy communities. For configuration management, I’ve found that SaltStack is a particularly welcoming group. Find the source code on GitHub, examine the issue tracker, pick something easy, and submit a pull request fixing the bug. As you graduate to working on more challenging or larger issues, remember to advertise your involvment with the project on your LinkedIn profile!

Additionally, help others out by blogging what you learn during these adventures. If you ever find that Google doesn’t have useful results for an error message that you searched, write a blog post with the message and how you fixed it. If you’re tempted to bikeshed over which blogging platform to use, default to GitHub Pages, as a static site hosted there is easy to move to your own hosting later if you so desire.

Examine job postings for roles like you want, and make sure the key buzzwords appear on your LinkedIn profile wherever appropriate. A complete LinkedIn profile for even a relatively new DevOps engineer draws a surprising number of recruiters for DevOps-related roles. If you’re just starting out in the field, I’d recommend expressing interest in every opportunity that you’re contacted about, progressing to at least a phone interview if possible, and getting all the feedback you can about your performance afterwards. It’s especially important to interview at companies that you can’t see yourself enjoying a job at, because you can practice asking probing questions that tell you whether an employer will be a good fit for you. (check out this post for ideas).

Another trick for getting to an interview is to start something with DevOps in the name. It could be anything from a curated blog to a meetup to an online “book club” for DevOps-related papers, but leading something with a cool name seems to be highly attractive to recruiters. Another way to increase your visibility in the field is to give a talk at any local conference, especially LinuxFest and DevOpsDays events. Putting together an introductory talk on a useful technology only requires intermediate proficiency, and is a great way to build your personal brand.

To summarize, there are really 4 tricks to getting DevOps interviews, and you should interview as much as you can to get a feeling for what DevOps means to different parts of the industry:

  • Contribute back to the open source tools that you use
  • Network with established professionals
  • Optimize your LinkedIn and other professional profiles to draw recruiters
  • Be the founder of something.

Questions?

I collect interesting job search and interview advice links at the bottom of my resume repo readme.

I bolded each paragraph’s key points in the hopes of making them easier to read.

You’re welcome to reach out to me at blog @ edunham.net or @qedunham on Twitter if you have other questions. If I made a dumb typo or omitted some information in this post, either tell me about it or just throw a pull request at the repo to fix it and give yourself credit.

Persona and third-party cookies in Firefox

Although its front page claims we’ve deprecated persona, it’s the only way to log into the statusboard and Air Mozilla. For a long time, I was unable to log into any site using Persona from Firefox 43 and 44 because of an error about my browser not being configured to accept third-party cookies.

The support article on the topic says that checking the “always accept cookies” box should fix the problem. I tried setting “accept third-party cookies” to “Always”, and yet the error persisted. (setting the top-level history configuration to “always remember history” didn’t affect the error either).

Fortunately, there’s also an “Exceptions” button by the “Accept cookies from sites” checkbox. Editing the exceptions list to universally allow “http://persona.org” lets me use Persona in Firefox normally.

_static/persona-exception.png

That’s the fix, but I don’t know whose bug it is. Did Firefox mis-balance privacy against convenience? Is the “always accept third-party cookies” setting’s failure to accept a cookie without an exception some strange edge case of a broken regex? Is Persona in the wrong for using a design that requires third-party cookies at all? Who knows!

Plushie Rustacean Pattern

I made a Rustacean. He’s cute. You can make one too.

../../../_images/ferris-on-pattern.jpg

You’ll Need

  • A couple square feet of orange polar fleece, or any other orange fabric that won’t stretch or fray too much
  • A handful of stuffing. I cannibalized a throw pillow.
  • A needle and some orange thread
  • Black and white fabric scraps and thread, or black and white embroidery floss, for making the face.
  • Intermediate sewing skills
  • This pattern

The Pattern

../../../_images/ferris-pattern-color.png

Get yourself a front, back, underside, and claw drawn on paper, either by printing them out or tracing from a screen. The front, back, and underside should have horizontal symmetry, except for the face placement. Make sure the points marked in red and blue on this pattern are noted on your paper.

Mine measure about 6” wide between the points marked in red.

Sewing vocabulary

  • The right side of a fabric is what ends up on the outside of the finished item. The wrong side ends up where you can’t see it. Some fabrics have both sides the same; in that case, the wrong side is whichever one you feel like tracing the pattern onto.
  • seam allowance is some extra fabric that ends up on the inside of the item when you’re done. The pattern above does not include seam allowance. This means that if you cut the fabric along the lines in the pattern, your finished rustacean will be tiny and sad and shaped wrong. You cut the paper along the lines, then trace it onto the fabric, then sew along the lines.
  • applique is where you sew one piece of fabric onto the surface of another to make a design.
  • There are a bunch of great youtube videos on basic sewing skills. Watch whichever ones you need.

Assembly

  1. Trace a front, a back, an underside, and the 2 claws onto the wrong side of your fabric with whatever will write on it without bleeding through. Make sure to transfer the blue centerline marks and the red three-point join marks.
  2. Cut out the shapes you just traced, leaving about 1” of margin around them. We’ll trim the seams properly later, so don’t worry about getting it exact.
  3. Find a couple claw-sized chunks of leftover fabric and pin one to the back of each claw (right sides together, of course).
  4. Sew around both claws, leaving the arm ends open so you can turn them. I find it’s easiest to backstitch, and you can get away with stitches up to about 1.5mm apart with normal weight polar fleece.
  5. Trim around the outside of the seams on the claws to leave about 1/4” seam allowance, and clip right up to the stitches in the concave spot. If you backstitched, make sure flip them over before trimming the seams so you don’t accidentally cut through the longer stitches.
  6. Turn the claws so the right side of the fabric is out and the seams are on the inside, and stuff them with stuffing or fabric scraps. A pair of wooden chopsticks from a fast food place are a great tool for turning and stuffing.
  7. Put the front and back pieces right sides together so the points marked in red and blue on the pattern line up. Pin them together.
  8. Sew from one red mark to the other along Ferris’s spiky back.
  9. Trim around the spikes leaving about 1/4” seam allowance, clipping right up to the seam in the concave spots.
  10. Figure out which side is front (hint, it has only 2 legs rather than 4). Imagine where Ferris’s little face will go when he’s finished. Now, pin both claws onto the right side of the front piece, so they’ll be oriented correctly when he’s done. If in doubt, pin the bottom front in place and turn the whole thing inside out to make sure the claws are right.
  11. Match the center front of the underside with the center of Ferris’s front (both have a blue + on the pattern). Be sure the pieces have their right sides together and the claws are sandwiched between them.
  12. Match the points marked with red triangles on each side of the front and underside together and pin them. If the claws are sticking out at this point, go back to step 10 and try again
  13. Sew from one red mark to the other to join Ferris’s front to the front of his underside. Put a few extra stitches in the part of the seam where his “arms”/claws are attached, to make sure they can’t be pulled out.
  14. Trim around the 2 tiny legs that you’ve sewn so far, with about 1/8” seam allowance.
  15. Now you can applique his face onto the right side of his front. Or embroider it if you know how. Cut the black and white felt scraps into face-shaped pieces and sew them down, giving Ferris whatever expression you want.
  16. Line up the 4 back legs on the underside and back pieces, and pin them right sides toether. Sew everything except the part marked in green – that’s the hole through which you’ll turn him inside out.
  17. Trim around those last 4 legs, leaving at least 1/8” seam allowance. Don’t cut away any more fabric from the bit marked in green. If you leave a bit of extra fabric around the leg seams, they’ll be harder to turn but require less stuffing.
  18. Turn Ferris right side out. Again, chopsticks or the non-pointy end of a barbeque skewer are useful for getting the pointy bits to do the right thing.
  19. Stuff Ferris with the filling. I filled mine quite loosely, because it makes him softer and more huggable. If you overfill his body, his spikes will look silly. If you overfill his legs, they’ll stick out in funny directions and not bend right.
  20. Tuck the seam allowance back into the hole through which you stuffed Ferris and sew it shut. Congratulations, you have your own toy crab!

The Finished Product

../../../_images/ferris-plushie-montage.jpg

He’s cute, cuddly, and palm-sized. Lego dude for scale.

Could Rust have a left-pad incident?

The short answer: No.

What happened with left-pad?

The Node community had a lot of drama this week when a developer unpublished a package on which a lot of the world depended.

This was fundamentally possible because NPM offers an unpublish feature. Although the docs for unpublish admonish users that “It is generally considered bad behavior to remove versions of a library that others are depending on!” in large bold print, the feature is available.

What’s the Rust equivalent?

The Rust package manager, Cargo, is similar to NPM in that it helps users get the libraries on which their projects depend. Rust’s analog to the NPM index is crates.io.

The best explanation of Cargo’s robustness against unpublish exploits is the docs themselves:

cargo yank

Occasions may arise where you publish a version of a crate that actually ends up being broken for one reason or another (syntax error, forgot to include a file, etc.). For situations such as this, Cargo supports a “yank” of a version of a crate.:

$ cargo yank --vers 1.0.1
$ cargo yank --vers 1.0.1 --undo

A yank does not delete any code. This feature is not intended for deleting accidentally uploaded secrets, for example. If that happens, you must reset those secrets immediately.

The semantics of a yanked version are that no new dependencies can be created against that version, but all existing dependencies continue to work. One of the major goals of crates.io is to act as a permanent archive of crates that does not change over time, and allowing deletion of a version would go against this goal. Essentially a yank means that all projects with a Cargo.lock will not break, while any future Cargo.lock files generated will not list the yanked version.

As Cargo author Alex Crichton clarified in a GitHub comment yesterday, the only way that it’s possible to remove code from crates.io is to compel the Rust tools team to edit the database and S3 bucket.

Even if a crate maintainer leaves the community in anger or legal action is taken against a crate, this workflow ensures that code deletion is only possible by a small group of people with the motivation and authority to do it in the way that’s least problematic for users of the Rust language.

For more information on the crates.io package and copyright policies, see this internals thread.

But I just want to left pad a string in Rust??

Although a left-pad crate was created as a joke, you should probably just use the format! built-in from the standard library.

Reducing SaltStack log verbosity for TravisCI

Servo has some Salt configs, hosted on GitHub, for which changes are smoke-tested on TravisCI before they’re deployed. Travis only shows the first 10k lines of log output, so I want to minimize the amount of extraneous information that the states print.

My salt state looks like::

android-sdk:
  archive.extracted:
    - name: {{ common.homedir }}/android/sdk/{{ android.sdk.version }}
    - source: https://dl.google.com/android/android-sdk_{{
      android.sdk.version }}-linux.tgz
    - source_hash: sha512={{ android.sdk.sha512 }}
    - archive_format: tar
    - archive_user: user
    - if_missing: {{ common.homedir }}/android/sdk/{{ android.sdk.version
      }}/android-sdk-linux
    - require:
      - user: user

The output in TravisCI is::

      ID: android-sdk
Function: archive.extracted
    Name: /home/user/android/sdk/r24.4.1
  Result: True
 Comment: https://dl.google.com/android/android-sdk_r24.4.1-linux.tgz extracted in /home/user/android/sdk/r24.4.1/
 Started: 17:46:25.900436
Duration: 19540.846 ms
 Changes:
          ----------
          directories_created:
              - /home/user/android/sdk/r24.4.1/
              - /home/user/android/sdk/r24.4.1/android-sdk-linux

          extracted_files:
              ... 2755 lines listing one file per line that I don't want to see in the log

https://docs.saltstack.com/en/latest/ref/states/all/salt.states.archive.html has useful guidance on how to increase the tar state’s verbosity, but not to decrease it. This is because the extra 2755 lines aren’t coming from tar itself, but from Salt assuming that we want to know.

terse outputter settings

The outputter takes several state_output setting options. The terse option summarizes the result of each state into a single line.

There are a couple places you can set this:

  • Invoke Salt with salt --state-output=terse hostname state.highstate
  • Add the line state_output: terse to /etc/salt/minion, if you’re using salt-call
  • Setting state_output_terse is apparently an option, though I can’t find any example of a real-world salt config that uses it

Setting the terse option in /etc/salt/minion dropped the output of a highstate from over 10,000 lines to about 2500.

Fixing sudo errors from the command line on OSX

The first symptom that I had made a terrible mistake showed up in an Ansible playbook:

GATHERING FACTS
***************************************************************
fatal: [...] => ssh connection closed waiting for a privilege escalation password prompt
fatal: [...] => ssh connection closed waiting for a privilege escalation password prompt
fatal: [...] => ssh connection closed waiting for sudo password prompt
fatal: [...] => ssh connection closed waiting for sudo password prompt

That looks like the sudo binary might be broken. To rule out Ansible problems, remote into the machine and try to use sudo:

administrators-Mac-mini:~ administrator$ sudo ls
sudo: effective uid is not 0, is sudo installed setuid root?

This meant that there was a file permissions problem:

working-host administrator$ ls -al /usr/bin/sudo
-r-s--x--x  1 root  wheel  164560 Sep  9  2014 /usr/bin/sudo

broken-host administrator$ ls -al /usr/bin/sudo
-rwxrwxr-x  1 root  wheel  164560 Sep  9  2014 /usr/bin/sudo

Now the problem is reduced to fixing the permissions. One does not simply sudo to root, because there’s no working sudo. However, Apple provides a utility which allows you to enable root login using only the administrator account’s permissions:

broken-host administrator$ dsenableroot
username = administrator
user password:
root password:
verify root password:

dsenableroot:: ***Successfully enabled root user.

The first password is the current one for the administrator account, and the other two should be the same string and will become the root account’s password.

After enabling root login, disconnect then SSH into the host as root:

broken-host root# chmod 4411 /usr/bin/sudo

And test that the fix fixed it:

broken-host root# su administrator
broken-host administrator$ sudo ls

Finally, clean up after yourself to inconvenience any future attackers:

broken-host administrator$ dsenableroot -d

Moral of the story: Errant chowns of /usr/bin are just as bad when they come from automation as when they come from humans.

Ansible, Vagrant, and changed host keys

Related to this bug, the Vagrant Ansible provisioner seems to ignore some system settings.

The symptom is that when you update a previously used Vagrant box, or otherwise change its host key, Ansible provisioning fails with the error:

fatal: [hostname] => SSH Error: Host key verification failed.
    while connecting to 127.0.0.1:2200
It is sometimes useful to re-run the command using -vvvv, which prints SSH
debug output to help diagnose the issue.

The standard solution would be to forget about the old host key with ssh-keygen -R 127.0.0.1:2200 or ignore the change with export ANSIBLE_HOST_KEY_CHECKING=false.

If you trust the box not to be evil and expect its host key to change frequently due to your testing, a fix which the Ansible provisioner does respect is to add ansible.host_key_checking = false to the Vagrantfile, like:

Vagrant.configure(2) do |config|
...
    config.vm.define "hostname" do |prodmaster|
        hostname.vm.provision "ansible" do |ansible|
            ansible.playbook = "provision/hostname.yaml"
            ansible.sudo = true
            ansible.host_key_checking = false
            ansible.verbose = 'vvvv'
            ansible.extra_vars = { ansible_ssh_user: 'vagrant'}
        end
    end
...
end

Vidyo with Ubuntu and i3wm

Mozilla uses Vidyo for virtual meetings across distributed teams. If it doesn’t work on your laptop, you can use the mobile client or book a meeting room in an office, but neither of those solutions is optimal when working from home.

Vidyo users within Mozilla can download a .deb or .rpm installer from v.mozilla.org. On Ubuntu, it’s easy to install the downloaded package with sudo dpkg -i path/to/the/file.deb.

The issue is that when you invoke VidyoDesktop from your launcher of choice (dmenu for me), i3 does what’s usually the right thing and makes the client fullscreen in a tile. This doesn’t allow the interface to pop up a floating window with the confirm dialog when you try to join a room, so you can’t.

mod + shift + space

Mod was alt by default last time I installed i3, but I’ve since remapped it to the window key (as IRC clients use alt for switching windows). Some people use caps lock as their mod key.

mod + shift + space makes the window floating, which allows it to pop up the confirmation dialog when you try to join a call.

Float windows by default

Alternately, stick the line:

for_window [class="VidyoDesktop"] floating enable

in your ~/.i3/config.

Are we ‘are we’ yet?

The Rust community, being founded and enjoyed by a variety of Mozilians, seems to have inherited the tradition of tracking top-level progress metrics using are we sites.

  • Are we concurrent yet? tracks the progres of Rust’s concurrency ecosystem
  • Are we web yet? tracks the status of Rust’s HTTP stack, web frameworks, and related libraries
  • Are we IDE yet? provides a list of what features are supported for Rust per IDE, and links to the relevant tracking issues and RFCs

If this blog post was an ‘are we’ page itself, the big text at the top would probably say “Getting There”.

Buildbot WithProperties

Today, I copied an existing command from a Buildbot configuration and then modified it to print a date into a file.:

...
if "cargo" in component:
    cargo_date_cmd = "echo `date +'%Y-%m-%d'` > " + final_dist_dir + "/cargo-build-date.txt"
    f.addStep(MasterShellCommand(name="Write date to cargo-build-date.txt",
                             command=["sh", "-c", WithProperties(cargo_date_cmd)] ))
...

It broke:

Failure: twisted.internet.defer.FirstError: FirstError[#8, [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.defer.FirstError'>: FirstError[#2, [Failure instance: Traceback: <type 'exceptions.ValueError'>: unsupported format character 'Y' (0x59) at index 14

Why? WithProperties.

It turns out that WithProperties should only be used when you need to interpolate strings into an argument, using either %s, %d, or %(propertyname)s syntax in the string.

The lesson here is Buildbot will happily accept WithProperties("echo 'this command uses no interpolation'") in a command argument, and then blow up at you if you ever change the command to have a % in it.

However, it appears that build steps run as MasterShellCommand``s without ``WithProperties do not display their name in the waterfall, but rather say “running” or “ran”.

Using Notty

I recently got the “Hey, you’re a Rust Person!” question of how to install notty and interact with it.

A TTY was originally a teletypewriter. Linux users will have most likely encountered the concept of TTYs in the context of the TTY1 interface where you end up if your distro fails to start its window manager. Since you use ctrl + alt + f[1,2,...] to switch between these interfaces, it’s easy to assume that “TTY” refers to an interactive workspace.

Notty itself is only a virtual terminal. Think of it as a library meant as a building block for creating graphical terminal emulators. This means that a user who saw it on Hacker News and wants to play around should not ask “how do I install notty”, but rather “how do I run a terminal emulator built on notty?”.

Easy Mode

Get some Rust:

curl -sf https://raw.githubusercontent.com/brson/multirust/master/blastoff.sh | sh
multirust update nightly

Get the system dependencies:

sudo apt-get install libcairo2-dev libgdk-pixbuf2.0 libatk1.0 libsdl-pango-dev libgtk-3-dev

Run Notty:

git clone https://github.com/withoutboats/notty.git
cd notty/scaffolding
multirust run nightly cargo run

And there you have it! As mentioned in the notty README, “This terminal is buggy and feature poor and not intended for general use”. Notty is meant as a library for building graphical terminals, and scaffolding is only a minimal proof of concept.

Read more...

How much knowledge do you need to give a conference talk?

I was recently asked an excellent question when I promoted the LFNW CFP on IRC:

As someone who has never done a talk, but wants to, what kind of knowledge do you need about a subject to give a talk on it?

If you answer “yes” to any of the following questions, you know enough to propose a talk:

  • Do you have a hobby that most tech people aren’t experts on? Talk about applying a lesson or skill from that hobby to tech! For instance, I turned a habit of reading about psychology into my Human Hacking talk.
  • Have you ever spent a bunch of hours forcing two tools to work with each other, because the documentation wasn’t very helpful and Googling didn’t get you very far, and built something useful? “How to build ___ with ___” makes a catchy talk title, if the thing you built solves a common problem.
  • Have you ever had a mentor sit down with you and explain a tool or technique, and the new understanding improved the quality of your work or code? Passing along useful lessons from your mentors is a valuable talk, because it allows others to benefit from the knowledge without taking as much of your mentor’s time.
  • Have you seen a dozen newbies ask the same question over the course of a few months? When your answer to a common question starts to feel like a broken record, it’s time to compose it into a talk then link the newbies to your slides or recording!
  • Have you taken a really interesting class lately? Can you distill part of it into a 1-hour lesson that would appeal to nerds who don’t have the time or resources to take the class themselves? (thanks lucyw for adding this to the list!)
  • Have you built a cool thing that over a dozen other people use? A tutorial talk can not only expand your community, but its recording can augment your documentation and make the project more accessible for those who prefer to learn directly from humans!
  • Did you benefit from a really great introductory talk when you were learning a tool? Consider doing your own tutorial! Any conference with beginners in their target audience needs at least one Git lesson, an IRC talk, and some discussions of how to use basic Unix utilities. These introductory talks are actually better when given by someone who learned the technology relatively recently, because newer users remember what it’s like not to know how to use it. Just remember to have a more expert user look over your slides before you present, in case you made an incorrect assumption about the tool’s more advanced functionality.

I personally try to propose talks I want to hear, because the dealine of a CFP or conference is great motivation to prioritize a cool project over ordinary chores.

Buildbot and EOFError

More SEO-bait, after tracking down an poorly documented problem:

# buildbot start master
Following twistd.log until startup finished..
2016-01-17 04:35:49+0000 [-] Log opened.
2016-01-17 04:35:49+0000 [-] twistd 14.0.2 (/usr/bin/python 2.7.6) starting up.
2016-01-17 04:35:49+0000 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2016-01-17 04:35:49+0000 [-] Starting BuildMaster -- buildbot.version: 0.8.12
2016-01-17 04:35:49+0000 [-] Loading configuration from '/home/user/buildbot/master/master.cfg'
2016-01-17 04:35:53+0000 [-] error while parsing config file:
    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155, in gotResult
        _inlineCallbacks(r, g, deferred)
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
        result = g.send(result)
      File "/usr/local/lib/python2.7/dist-packages/buildbot/master.py", line 189, in startService
        self.configFileName)
    --- <exception caught here> ---
      File "/usr/local/lib/python2.7/dist-packages/buildbot/config.py", line 156, in loadConfig
        exec f in localDict
      File "/home/user/buildbot/master/master.cfg", line 415, in <module>
        extra_post_params={'secret': HOMU_BUILDBOT_SECRET},
      File "/usr/local/lib/python2.7/dist-packages/buildbot/status/status_push.py", line 404, in __init__
        secondaryQueue=DiskQueue(path, maxItems=maxDiskItems))
      File "/usr/local/lib/python2.7/dist-packages/buildbot/status/persistent_queue.py", line 286, in __init__
        self.secondaryQueue.popChunk(self.primaryQueue.maxItems()))
      File "/usr/local/lib/python2.7/dist-packages/buildbot/status/persistent_queue.py", line 208, in popChunk
        ret.append(self.unpickleFn(ReadFile(path)))
    exceptions.EOFError:

2016-01-17 04:35:53+0000 [-] Configuration Errors:
2016-01-17 04:35:53+0000 [-]   error while parsing config file:  (traceback in logfile)
2016-01-17 04:35:53+0000 [-] Halting master.
2016-01-17 04:35:53+0000 [-] Main loop terminated.
2016-01-17 04:35:53+0000 [-] Server Shut Down.

This happened after the buildmaster’s disk filled up and a bunch of stuff was manually deleted. There were no changes to master.cfg since it worked perfectly.

The fix was to examine master.cfg to see where the HttpStatusPush was created, of the form:

c['status'].append(HttpStatusPush(
    serverUrl='http://build.servo.org:54856/buildbot',
    extra_post_params={'secret': HOMU_BUILDBOT_SECRET},
))

Digging in the Buildbot source reveals that persistent_queue.py wants to unpickle a cache file from /events_build.servo.org/-1 if there was nothing in /events_build.servo.org/. To fix this the right way, create that file and make sure Buildbot has +rwx on it.

Alternately, you can give up on writing your status push cache to disk entirely by adding the line maxDiskItems=0 to the creation of the HttpStatusPush, giving you:

c['status'].append(HttpStatusPush(
   serverUrl='http://build.servo.org:54856/buildbot',
   maxDiskItems=0,
   extra_post_params={'secret': HOMU_BUILDBOT_SECRET},
))

The real moral of the story is “remember to use logrotate.

Who would you hire?

If you’re using open source as a portfolio to make yourself a more competitive job candidate, it can feel like you have to start your own project to show off your skills.

In the words of one job seeker I chatted with recently, “I feel like most of my contributions [to other peoples’ projects] aren’t that significant or noteworthy”. Here’s a thought experiment to justify including projects to which you contribute, even without a leadership role, on your resume:

Imagine you want to hire a coder.

Candidate A always works alone and refuses to contribute to a project if it doesn’t make her look like a rockstar.

Candidate B triages the unglamorous issues that affect multiple users, and steadily produces small, self-contained fixes that avoid introducing new bugs.

When the situation is framed in these terms, I hope that it’s obvious which coder you’d want on your team.

When writing your resume, there’s only space to include a few of the many activities in which you invest your time. It’s tempting to only include your biggest, highest-profile solo projects, while disregarding those projects to which you’ve made a small but steady stream of useful contributions.

Reread your resume from the perspective of someone who hasn’t met you yet and has only the information in that document available to form a first impression of your character. Which of the 2 hypothetical coders does it make you sound like? Is that how you really are?

Troubleshooting stunnel

Today I’ve learned a few things aout how stunnel works. The main takeaway is that Googling for specific errors in the stunnel log is incredibly unhelpful, resulting in a variety of mailing list posts with no replies. Tracking an error message through the source of the program doesn’t lead to any useful comments, either. So here’s some SEO bait with concrete troubleshooting suggestions.

Read more...

Questions about Open Source and Design

Today, I posed a question to some professional UI and UX designers:

How can an open source project without dedicated design experts collaborate with amateur, volunteer designers to produce a well-designed product?

They revealed that they’ve faced similar collaboration challenges, but knew of neither a specific process to solve the problem nor an organization that had overcome it in the past.

Have you solved this problem? Have you tried some process or technique and learned that it’s not able to solve the problem? Email me (design@edunham.net) if you know of an open source project that’s succeeded at opening their design as well, and I’ll update back here with what I learn!

In no particular order, here are some of the problems that we were talking about:

  • Non-designers struggle to give constructive feedback on design. I can say “that’s ugly” or “that’s hard to use” more easily than I can say “here’s how you can make it better”.
  • Projects without designers in the main decision-making team can have a hard time evaluating the quality of a proposed design.
  • Non-designers struggle to articulate the objective design needs of their projects, so design remains a single monolithic problem rather than being decomposed into bite-sized, introductory issues the way code problems are.
  • Volunteer designers have a difficult time finding open source projects to get involved with.
  • Non-designers don’t know the difference between different types of design, and tend to bikeshed on superficial, obvious traits like colors when they should be focusing on more subtle parts of the user experience. We as non-designers are like clients who ask for a web site without knowing that there’s a difference between frontend development, back end development, database administration, and systems administration.
  • The tests which designers apply to their work are often almost impossible to automate. For instance, I gather that a lot of user interaction testing involves watching new users attempt to complete a task using a given design, and observing the challenges they encounter.

Again, if you know of an open source project that’s overcome any of these challenges, please email me at design@edunham.net and tell me about it!

Linode vs AWS

I’m examining a Linode account in order to figure out how to switch the application its instances are running to AWS. The first challenge is that instance types in the main dashboard are described by arbitrary numbers (“UI Name” in the chart below), rather than a statistic about their resources or pricing. Here’s how those magic numbers line up to hourly rates and their corresponding monthly price caps:

RAM Hourly $ Monthly $ UI Name Cores GB SSD
1GB $0.015/hr $10/mo 1024 1 24
2GB $0.03/hr $20/mo 2048 2 48
4GB $0.06/hr $40/mo 4096 4 96
8GB $0.12/hr $80/mo 8192 6 192
16GB $0.24/hr $160/mo 16384 8 384
32GB $0.48/hr $320/mo 32768 12 768
48GB $0.72/hr $480/mo 49152 16 1152
64GB $0.96/hr $640/mo 65536 20 1536
96GB $1.44/hr $960/mo 98304 20 1920

AWS “Equivalents”

AWS T2 instances have burstable performance. M* instances are general-purpose; C* are compute-optimized; R* are memory-optimized. *3 instances run on slightly older Ivy Bridge or Sandy Bridge processors, while *4 instances run on the newer Haswells. I’m disergarding the G2 (GPU-optimized), D2 (dense-storage), and I2 (IO-optmized) instance types from this analysis.

Note that the AWS specs page has memory in GiB rather than GB. I’ve converted everything into GB in the following table, since the Linode specs are in GB and the AWS RAM amounts don’t seem to follow any particular pattern that would lose information in the conversion.

Hourly price is the Linux/UNIX rate for US West (Northern California) on 2015-12-03. Monthly price estimate is the hourly price multiplied by 730.

Instance vCPU GB RAM $/hr $/month
t2.micro 1 1.07 .017 12.41
t2.small 1 2.14 .034 24.82
t2.medium 2 4.29 .068 49.64
t2.large 2 8.58 .136 99.28
m4.large 2 8.58 .147 107.31
m4.xlarge 4 17.18 .294 214.62
m4.2xlarge 8 34.36 .588 429.24
m4.4xlarge 16 68.72 1.176 858.48
m4.10xlarge 40 171.8 2.94 2146.2
m3.medium 1 4.02 .077 56.21
m3.large 2 8.05 .154 112.42
m3.xlarge 4 16.11 .308 224.84
m3.2xlarge 8 32.21 .616 449.68
c4.large 2 4.02 .138 100.74
c4.xlarge 4 8.05 .276 201.48
c4.2xlarge 8 16.11 .552 402.96
c4.4xlarge 16 32.21 1.104 805.92
c4.8xlarge 36 64.42 2.208 1611.84
c3.large 2 4.02 .12 87.6
c3.xlarge 4 8.05 .239 174.47
c3.2xlarge 8 16.11 .478 348.94
c3.4xlarge 16 32.21 .956 697.88
c3.8xlarge 32 64.42 1.912 1395.76
r3.large 2 16.37 .195 142.35
r3.xlarge 4 32.75 .39 284.7
r3.2xlarge 8 65.50 .78 569.4
r3.4xlarge 16 131 1.56 1138.8
r3.8xlarge 32 262 3.12 2277.6

Comparison

Linode and AWS do not compare cleanly at all. The smallest AWS instance to match a given Linode type’s RAM typically has fewer vCPUs and costs more in the region where I compared them. Conversely, the smallest AWS instance to match a Linode type’s number of cores often has almost double the RAM of the Linode, and costs substantially more.

Switching from Linode to AWS

When I examine the Servo build machines’ utilization graphs via the Linode dashboard, it becomes clear that even their load spikes aren’t fully utilizing the available CPUs. To view memory usage stats on Linode, it’s necessary to configure hosts to run the longview client. After installation, the client begins reporting data to Linode immediately.

After a few days, these metrics can be used to find the smallest AWS instance whose specs exceed what your application is actually using on Linode.

Sources:

Giving Thanks to Rust Contributors

It’s the day before Thanksgiving here in the US, and the time of year when we’re culturally conditioned to be a bit more public than usual in giving thanks for things.

As always, I’m grateful that I’m working in tech right now, because almost any job in the tech industry is enough to fulfill all of one’s tangible needs like food and shelter and new toys. However, plenty of my peers have all those material needs met and yet still feel unsatisfied with the impact of their work. I’m grateful to be involved with the Rust project because I know that my work makes a difference to a project that I care about.

Rust is satisfying to be involved with because it makes a difference, but that would not be true without its community. To say thank you, I’ve put together a little visualization for insight into one facet of how that community works its magic:

../../../_images/orglog_deploy_teaser.png

The stats page is interactive and available at http://edunham.github.io/rust-org-stats/. The pretty graphs take a moment to render, since they’re built in your browser.

There’s a whole lot of data on that page, and you can scroll down for a list of all authors. It’s especially great to see the high impact that the month’s new contributors have had, as shown in the group comparison at the bottom of the “natural log of commits” chart!

It’s made with the little toy I wrote a while ago called orglog, which builds on gitstat to help visualize how many people contribute code to a GitHub organization. It’s deployed to GitHub Pages with TravisCI (eww) and nightli.es so that the Rust’s organization-wide contributor stats will be automatically rebuilt and updated every day.

If you’d like to help improve the page, you can contribute to gitstat or orglog!

PSA: Docker on Ubuntu

$ sudo apt-get install docker
$ which docker
$ docker
The program 'docker' is currently not installed. You can install it by typing:
apt-get install docker
$ apt-get install docker
Reading package lists... Done
Building dependency tree
Reading state information... Done
docker is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.

Oh, you wanted to run a docker container? The docker package in Ubuntu is some window manager dock thingy. The docker binary that runs containers comes from the docker.io system package.

$ sudo apt-get install docker.io
$ which docker
/usr/bin/docker

Also, if it can’t connect to its socket:

FATA[0000] Post http:///var/run/docker.sock/v1.18/containers/create: dial
unix /var/run/docker.sock: permission denied. Are you trying to connect to a
TLS-enabled daemon without TLS?

you need to make sure you’re in the right group:

sudo usermod -aG docker <username>; newgrp docker

(thanks, stackoverflow!)

Installing Rust without root

I just got a good question from a friend on IRC: “Should I ask my university’s administration to install Rust on our shared servers?” The answer is “you don’t have to”.

Pick one of the two following sets of directions. I’d recommend using Multirust, because it automatically checks the packages it downloads and lets you switch between Rust versions trivially.

Read more...

Multiple languages on TravisCI

Today I noticed an assumption which was making my life unnecessarily difficult: I assumed that if my .travis.yml said language: ruby on the first line, I was supposed to only run Ruby code from it.

Travis lets you run code much more arbitrary than that.

I did a bunch of tests on a toy repo to see what would happen if I ignored my preconceptions about how you can and can’t test stuff, and learned some interesting things:

  • You can install PyPI packages in a test suite that’s technically Ruby, or gems in a test suite that’s technically Python.
  • If your project is language:ruby, you need to sudo pip install dependencies. If it’s language:python, you can just gem install dependencies without sudo.
  • If I specify multiple instances of language: or multiple build matrices, Travis uses the language whose build matrix occurs last. If I specify a Python matrix and then a Ruby one, the Ruby matrix will be run.

This is especially useful when testing or deployment requires hitting an API whose libraries are most up to date in a language other than that of the project.

Beyond Openhatch

Update: I’m now maintaining the issue aggregator list at http://edunham.net/pages/issue_aggregators.html

OpenHatch is a wonderful place to help new contributors find their first open source issues to work on. Their training materials are unparalleled, and the “projects submit easy bugs with mentors” model makes their list of introductory issues reliably high-quality.

However, once you know the basics of how to engage with an open source project, you’re no longer in the target audience for OpenHatch’s list. Where should you look for introductory issues when you want to get involved with a new project, but you’re already familiar with open source in general?

An excellent slide deck by Josh Matthews contains several answers to this question:

  • issuehub.io scrapes GitHub by labels and language
  • up-for-grabs has an opt-in list of projects looking for new contributors, and scrapes their issue trackers for their “jump in”, “up for grabs” or other “new contributors welcome” tags.
  • If you’re looking for Mozilla-specific contributions outside of just code, What can I do for Mozilla? can help direct you into any of Mozilla’s myriad opportunities for involvement.

Additionally, the servo-starters page has a custom view of easy issues sorted by Servo’s project-specific tags.

GitHub Tricks

If you’re looking for open issues across all repos owned by a particular user or organization, you can use the search at https://github.com/pulls and specify the “user” (or org) in the search bar. For instance, this search will find all the unassigned, easy-tagged issues in the rust-lang org. Breaking down the search:

  • user:rust-lang searches all repos owned by github.com/rust-lang. It could also be someone’s github username.
  • is:open searches only open issues.
  • no:assignee will filter out the issues which are obviously claimed. Note that some issues without an assignee set may still have a comment saying “I’ll do this!”, if it was claimed by a user who did not have permissions to set assignees and then not triaged.
  • label:E-Easy uses my prior knowledge that most repos within rust-lang annotate introductory bugs with the E-easy tag. When in doubt, check the contributing.md file at the top level in the org’s most popular repository for an explanation of what various issue labels mean. If that information isn’t in the contributing file or the README, file a bug!

Am I missing your favorite introductory issue aggregator? Shoot me an email to ___@edunham.net (fill in the blank with anything; the email will get to me) with a link, and I’ll add it here if it looks good!

PSA: Pin Versions

Today, the website’s build broke. We made no changes to the tests, yet a wild dependency error emerged:

Generating...

  Dependency Error: Yikes! It looks like you don't have redcarpet or one of
its dependencies installed. In order to use Jekyll as currently configured,
you'll need to install this gem. The full error message from Ruby is: 'cannot
load such file -- redcarpet' If you run into trouble, you can find helpful
resources at http://jekyllrb.com/help/!

  Conversion error: Jekyll::Converters::Markdown encountered an error while
converting 'conduct.md':

                    redcarpet

             ERROR: YOUR SITE COULD NOT BE BUILT:

                    ------------------------------------

                    redcarpet

The command "jekyll build" exited with 1.

Although Googling the error was unhelpful, a bit more digging revealed that our last working build had been on Jekyll 2.5.3 and the builds breaking on a Redcarpet error all used 3.0.0.

The moral of the story is that where the .travis.yml said - gem install jekyll, it should have said - gem install jekyll -v 2.5.3.

SeaGL 2015 Retrospective

As well as nominally helping organize the event, I attended and spoke at SeaGL 2015 this weekend. The slides from my talk are here.

My talk drew an audience of perhaps a dozen people on Friday afternoon. I didn’t record this instance of the talk, but will probably give it at least one more time and be sure to record then.

One of the more useful tools I learned about is called myrepos. It lets you update all of the Git repositories on a machine at the same time, as well as other neat tricks like replaying actions that failed due to network problems. Its author has written a variety of other useful Git wrappers, as well.

Additionally, VCSH seems to be the “I knew somebody else wrote that already!” tool for keeping parts of a home directory in Git.

Upgrading Buildbot 0.8.6 to 0.8.12

Here are some quick notes on upgrading Buildbot.

System Dependencies

There are more now. In order to successfully install all of Buildbot’s dependencies with Pip, I needed a few more apt packages:

python-dev
python-openssl
libffi-dev
libssl-dev

Then for sanity’s sake make a virtualenv, and install the following packages. Note that having too new a sqlalchemy will break things.:

buildbot==0.8.12
boto
pyopenssl
cryptography
SQLAlchemy<=0.7.10

Virtualenvs

Troubleshooting compatibility issues with system packages on a host that runs several Python services with various dependency versions is predictably terrible.

The potential problem with switching to running Buildbot only from a virtualenv is that developers with access to the buildmaster might want to restart it and miss the extra step of activating the virtualenv. I addressed this by adding the command to activate the virtualenv (using the virtualenv’s absolute path) to the ~/.bashrc of the user that we run Buildbot as. This way, we’ve gained the benefits of having our dependencies consolidated without adding the cost of an extra workflow step to remember.

Template changes

Most of Buildbot’s status pages worked fine after the upgrade, but the console view threw a template error because it couldn’t find any variable named “categories”. The fix was to simply copy the new template from venv/local/lib/python2.7/site-packages/buildbot/status/web/templates/console.html to my-buildbot/master/templates/console.html.

That’s it!

Rust currently has these updates on the development buildmaster, but not yet (as of 10/14/2015) in prod.

Carrying credentials between environments

This scenario is simplified for purposes of demonstration.

I have 3 machines: A, B, and C. A is my laptop, B is a bastion, and C is a server that I only access through the bastion.

I use an SSH keypair helpfully named AB to get from me@A to me@B. On B, I su to user. I then use an SSH keypair named BC to get from user@B to user@C.

I do not wish to store the BC private key on host B.

SSH Agent Forwarding

I have keys AB and BC on host A, where I start. Host A is running ssh-agent, which is installed by default on most Linux distributions.

me@A$ ssh-add ~/.ssh/AB     # Add keypair AB to ssh-agent's keychain
me@A$ ssh-add ~/.ssh/BC     # Add keypair BC to the keychain
me@A$ ssh -A me@B           # Forward my ssh-agent

Now I’m logged into host B and have access to the AB and BC keypairs. An attacker who gains access to B after I log out will have no way to steal the BC keypair, unlike what would happen if that keypair was stored on B.

See here for pretty pictures explaining in more detail how agent forwarding works.

Anyways, I could now ssh me@C with no problem. But if I sudo su user, my agent is no longer forwarded, so I can’t then use the key that I added back on A!

Switch user while preserving environment variables

me@B$ sudo -E su user
user@B$ sudo -E ssh user@C

What?

The -E flag to sudo preserves the environment variables of the user you’re logged in as. ssh-agent uses a socket whose name is of the form /tmp/ssh-AbCdE/agent.12345 to call back to host A when it’s time to do the handshake involving key BC, and the socket’s name is stored in me‘s SSH_AUTH_SOCK environment variable. So by telling sudo to preserve environment variables when switching user, we allow user to pass ssh handshake stuff back to A, where the BC key is available.

Why is sudo -E required to ssh to C? Because /tmp/sshAbCdE/agent.12345 is owned by me:me, and only the file’s owner may read, write, or execute it. Additionally, the socket itself (agent.12345) is owned by me:me, and is not writable by others.

If you must run ssh on B without sudo, chown -R /tmp/ssh-AbCdE to the user who needs to end up using the socket. Making them world read/writable would allow any user on the system to use any key currently added to the ssh-agent on A, which is a terrible idea.

For what it’s worth, the actual value of /tmp/ssh-AbCdE/agent.12345 is available at any time in this workflow as the result of printenv | grep SSH_AUTH_SOCK | cut -f2 -d =.

The Catch

Did you see what just happened there? An arbitrary user with sudo on B just gained access to all the keys added to ssh-agent on A. Simon pointed out that the right way address this issue is to use ProxyCommand instead of agent forwarding.

No, I really don’t want my keys accessible on B

See man ssh_config for more of the details on ProxyCommand. In ~/.ssh/config on A, I can put:

Host B
    User me
    Hostname 111.222.333.444

Host C
    User user
    Hostname 222.333.444.555
    Port 2222
    ProxyCommand ssh -q -w %h:%p B

So then, on A, I can ssh C and be forwarded through B transparently.

Ansible: Conditional role dependencies

I’ve recently been working on an Ansible role that applies to both Ubuntu and OSX hosts. It has some dependencies which are only needed on OSX. There doesn’t seem to be a central document on all the options available for solving this problem, so here are my notes.

Read more...

Apache Licenses

At the bottom of the Apache 2.0 License file, there’s an appendix:

APPENDIX: How to apply the Apache License to your work.

...

Copyright [yyyy] [name of copyright owner]

...

Does that look like an invitation to fill in the blanks to you? It sure does to me, and has for others in the Rust community as well.

Today I was doing some licensing housekeeping and made the same embarrassing mistake.

This is a PSA to double check whether inviting blanks are part of the appendix before filling them out in Apache license texts.

X240 trackpoint speed

The screen on my X1 Carbon gave out after a couple months, and my loaner laptop in the meantime is an X240.

The worst thing about this laptop is how slowly the trackpoint moves with a default Ubuntu installation. However, it’s fixable:

cat /sys/devices/platform/i8042/serio1/serio2/speed
cat /sys/devices/platform/i8042/serio1/serio2/sensitivity

Note the starting values in case anything goes wrong, then fiddle around:

echo 255 | sudo tee /sys/devices/platform/i8042/serio1/serio2/sensitivity
echo 255 | sudo tee /sys/devices/platform/i8042/serio1/serio2/speed

Some binary search themed prodding and a lot of tee: /sys/devices/platform/i8042/serio1/serio2/sensitivity: Numerical result out of range has confirmed that both files accept values between 0-255. Interestingly, setting them to 0 does not seem to disable the trackpoint completely.

If you’re wondering why the configuration settings look like ordinary files but choke on values bigger or smaller than a short, go read about sysfs.

Folklore and fallacy

I was a student employee at the OSU Open Source Lab, on and off between internships and other jobs, for 4 years. Being part of the lab helped shape my life and career, in almost overwhelmingly positive ways. However, the farther I get from the lab the more clearly I notice how being part of it changed the way I form expectations about my own technical skills.

To show you the fallacy that I noticed myself falling into, I’d like to tell you a completely made-up story about some alphabetically named kangaroos. Below the fold, there’ll be pictures!

Read more...

RustCamp videos are available

The videos from RustCamp are available here.

I asked Gankro what was up with the milkshake thing in his talk, and learned about this meme.

Don’t Starve

It was a lazy Sunday afternoon and I wanted to play Don’t Starve. This actually ended up meaning about 3 hours of intermittent troubleshooting and 1 hour of games, because Linux.

Get the files

I bought Don’t Starve from the Humble Bundle store, although there are other methods of obtaining it which strike a different balance between cost and convenience.

The downloaded file is dontstarve_x64_july21.tar.gz.

This Just Works

$ yaourt -S libcurl-compat
$ tar -xvf dontstarve_x64_july21.tar.gz
$ cd dontstarve/bin
$ LD_PRELOAD=libcurl.so.3 ./dontstarve

Below the fold is the troubleshooting process I went through to make it look so easy. Hopefully it’ll be of assistance to those searching for the errors that I ran into!

Read more...

How many Rust channels are there?

I’m using search.mibbit.com to count these. All have at least one user in them as of 4pm PST 2015-07-31.

There are 53 Rust-related channels on irc.mozilla.org.

List below the fold.

Read more...

Good times

People sometimes say “morning” or “evening” on IRC for a time zone unlike my own. Here’s a bash one-liner that emits the correct time-of-day generalization based on the datetime settings of the machine you run it on.

case $(($(date +%H)/6)) in 0|1)m="morning";;2)m="afternoon";;3)m="night";;esac; echo good $m

Read more...

Printing

The office printers have instructions for setting them up under Windows, Mac, and Ubuntu. I had forgotten how to wrangle printers, since the last time I had to set up new ones was half a decade ago when I first joined the OSL.

Setting up printers on Arch is easy once you know the right incantations, but can waste some time if you try to do it by skimming the huge wiki page rather than either reading it thoroughly or just following these steps:

Install the CUPS client:

$ yaourt -S libcups

Add a magic line to /etc/cups/cups-files.conf:

SystemGroup username

With your username on the system, assuming you have root and will log in as yourself in the dialog it prompts for. That line can go anywhere in the file.

Make the daemon go:

$ sudo systemctl enable org.cups.cupsd.service
$ sudo systemctl start org.cups.cupsd.service

Visit the web interface at http://localhost:631.

Then you have a GUI sufficiently similar to the one in the instructions for Ubuntu!

There is no GUI client for CUPS to install. If you find yourself mucking about with gpr, xpp, kdeprint, or /etc/cups/client.conf, you have gone way too far down the wrong rabbit hole.

Outage postmortem: Replacing Rust Buildbot’s outdated cert

At the end of the day on July 14th, 2015, the certificate that Rust’s buildbot slaves were using to communicate with the buildmaster expired. This broke things. The problem started at midnight on July 15th, and was only fully resolved at the end of July 16th. Much of the reason for this outage’s duration was that I was learning about Buildbot as I went along.

Here’s how the outage got resolved, just in case anyone (especially future-me) finds themself Googling a similar problem.

Read more...

Airport Wifi

Many “free” wifi hotspots give you a limited time per computer. If you’re traveling light and forgot to bring extra devices, it’s easy to give a Linux laptop multiple personalities:

$ ip link
    1: lo
    2: wlp4s0
    3: enp0s25
$ ip link set dev wlp4s0 down
$ macchanger -r wlp4s0
$ ip link set dev wlp4s0 up

... And then connect to the wifi and jump through its silly captive portal hoops again!

Changing your MAC address occasionally can be part of a healthy security diet, making your device slightly more difficult to track, as well.

Interactive Rust Examples in Static Pages

Rust by Example has a little box where readers can interact with some example Rust code, run it using the playground, and see the results in the page. As a sysadmin I’m loath to recommend that anybody trust the playground for anything, but as a nerd and coder I recognize that it’s super cool and people want to use it.

There are 2 ways to stuff a Playground into your website: The easy way, and the “right” way. Here’s how to do it the easy way, and where to look for examples of the hard way.

Read more...

Rust’s Packaging Status Across Distros

One of many questions facing the Rust infrastructure team right now is “What’s our packaging situation?”. We don’t have a centralized source of information on what version of Rust is available in which systems’ package managers, and we don’t even know where to find that information.

This post is the notes I’ve taken in researching Rust’s packaging status across distributions.

I last updated this post on 8/17/2015.

Read more...

Mozilla Onboarding

On April 16th, I threw an application toward this job posting on careers.mozilla.org. I doubted whether I’d be qualified, but I reminded myself that most people apply to jobs where they meet only 80% of the critiera. I could, with some creative redefinition (“of course an internship at Intel is a year of relevant experience!”), meet every listed criterion. So I applied, since the worst they could say was no.

Here’s a rundown my experience getting interviewed and onboarded, which might be of interest to current Mozilla employees curious about how things have changed since they joined, and to anyone interested in working there.

Read more...

One free ticket to Rust Camp

UPDATE: The ticket has been claimed by a local student! Thanks everyone for helping boost the signal about this.

Read more...

How to find a Buildbot slave’s IP

Today I got a seemingly ordinary request from a community member who volunteers a build slave for Rust’s buildbot:

my builder is behind a firewall that just cycled IP's
and I don't know what it is
edunham: can you get the IP address of the bitrig builder for me?
I have admin access to the builders website but it doesn't list the IP addresses of builders

Read more...

Moving a Jekyll site from GitHub Pages to Amazon S3

The rust-lang.org web site used to be hosted on GitHub Pages. This gave it excellent uptime and made deploying changes easy, but did not support HTTPS.

Read more...

Downloading an S3 bucket

Since I’m curious about how often files are downloaded from S3, I enabled logging on the buckets serving them and directed the logs into a bucket which I created to hold them. Then I wanted to move everything on that logs bucket to my local machine, so I could poke around in the logs and ascertain the best way to turn them into useful information.

Read more...

DMARC

Today, the security alias for a site I administer got an automated message pointing out that we lacked a DMARC record. Here’s what I learned about how to set up and test them.

Read more...

Deleting spam logs

Some spammers got onto the Mozilla network, scraped a major channel’s user list, and PMed everybody requests to join their network from almost 1,000 different nicks. Here’s how I tidied up afterwards.

Read more...

Installing Playpen on Ubuntu

One of the more egregious inconsistencies in Rust’s architecture is that play.rust-lang.org lives on an Arch box, while everything else is Ubuntu. Before the team has a dedicated operations person, the argument for using Arch was that playpen comes pre-packaged for it, whereas one has to build it oneself on Ubuntu.

But standardizing the infrastructure to one OS is a really cool thing to do, since it requires that much less thought and effort to do any update that affects every system (I’m looking at you, security patches).

Read more...

Display Defaults

I’ve been running Arch on my work laptop and it’s pretty much working. However, I have a nice external monitor on my desk, and I keep having to manually configure the output to it with arandr. Here’s how I made it configure itself by default when X starts.

Read more...

Playing with Ansible

Although I currently expect that I’ll end up choosing Salt for work, I’ve gotten nerdsniped by the apparent simplicity and power of Ansible. Since I’m trying to make a habit of narrating my first encounters with various tools, here’s a short novel of 0 through cloning a repo.

Read more...

Configuration Management Comparison

Let’s just say that it’s pretty clear why my team at Mozilla decided to hire an operations specialist when they did. For the infrastructure which supports the Rust programming language, I get the relatively rare (compared to just hacking on an existing deployment) privilege of deploying configuration management from the ground up.

Read more...

Recording Screencasts on Arch

Today I learned that there’s a trick to getting sound to work on Arch using recordmydesktop.

Read more...

Oh, Windows...

I got a shiny new Thinkpad X1 Carbon 3rd Gen for my new job. It came with Windows pre-installed. Out of morbid curiosity and willingness to consider giving this shiny new allegedly-less-terrible Win8 thing a chance, I booted it up into the default Windows installation before wiping everything to install arch.

Read more...

Open Infrastructure

My New Ops Job

Next week, I’m starting work as the only DevOps Engineer on the Mozilla Research team. While I’ve held a variety of superficially similar jobs in the past few years, this one offers a special opportunity to apply the values I appreciate as a software developer to the infrastructure design and maintenance which represents my work as an ops guy.

Read more...

Starting Rust: Introduction

When discussing my experience (or lack thereof) with the Rust programming language during a recent interview, I learned that experienced developers might be interested in a stream-of-consciousness, “let’s-play” type narration of my learning curve through the language.

There’s an excrutiatingly detailed account of it below the fold. You’ve been warned.

Read more...

Command-line Keyboard Shortcuts

Just another installment of “How did I not know that already?”

Read more...

The life cycle of a conference talk

Although lines of code are the most convenient metric of success in open source, I’m proud of those of my own contributions that have taken the form of documentation, teaching, and outreach.

One such skill that I’ve been working on improving over the past couple years is public speaking, specifically educational talks on a variety of open source related topics. At my current skill level, I’m finding it increasingly important to understand what works and what doesn’t in the process that I use when creating my talks. Below the fold will be a few pages of my thinking “out loud” about the topic.

Read more...

Culling the GitHub notification spam

Long ago, I set up an email filter to make GitHub notifications skip my inbox. This made it easy to ignore the volume of irrelevant notifications that I was getting, and I only noticed them when they got in the way of a search for the one or two useful emails that GitHub sends.

However, it also means that I won’t find out if something legitimately notification-worthy happens, such as a new contributor opening a pull request to one of the handful of projects I actually care about.

Here are the steps one can take to improve that signal-to-noise ratio.

Read more...

Removing Blackarch

I installed Blackarch on top of my ordinary Arch install for the Oregon CTF a few weeks ago.

Now it’s taking up a lot of space on my system and I no longer need it around.

None of the Google queries I’ve tried have discussed how to get rid of it, and I’ve had to resort to reading the install script to figure out how to make its repositories go away. Here’s what I learned.

Read more...

SSHFS to public_html

Today, someone asked the #osu-lug channel how to mount their public_html directory on OSU’s shell server to a location on their local Linux machine using sshfs.

Read more...

Optional Arguments in LaTeX Macros

As I’ve mentioned before, I use LaTeX to typeset my resume. I recently found a convenient workaround to handle formatting which differs based on whether or not a macro’s argument is present.

Read more...

The Case of the Mixed-Case Tags

I accidentally discovered the “feature” that Tinkerer tags are somewhat case-sensitive.

Read more...

Fixing Mplayer

When trying to fix some errors related to installing BlackArch last week, I made a poorly thought out decision and deleted some kernel modules that I didn’t think were necessary. This rendered my system impossible to boot or chroot into for a bit.

Read more...

hieroglyph2beamer with Pandoc

I’ve played with getting Sphinx to generate PDFs before, and while rst2pdf generates a PDF with all the notes and pictures present, the results aren’t as beautifully typeset as I’ve come to expect from LaTeX.

This made me wonder whether any tool exists to convert Hieroglyph slides into Beamer presentations.

Read more...

The Magic of Extundelete

Last night, I was just falling asleep when my roommate knocked on my door with a Linux problem. Our CS480 assignment was due at midnight and she’d finished the work, then accidentally overwritten the most important file in her program with a malformed tar command.

Read more...

Code Coverage in the Monte Tests

My schoolwork is particularly bad this weekend, so I’m procrastinating by learning to analyze the test coverage of a moderately complex Python codebase. Specifically, the reference implementation of the Monte programming language. This effort is hampered only slightly by the fact that I’ve never done much with code coverage tools before.

Read more...

Hardware Saga, part 1

At around 1pm on 2/26/2015, my ThinkPad X230’s screen died without warning. I put the laptop away to take a quiz in class, walked home, tried to boot, an it remained black. The LEDs and CPU fan went through their familiar boot process, but the screen remained stubbornly blank.

Below the fold you’ll find several pages of excrutiating detail on the situation, because I enjoy writing these things down.

Read more...

The SomeoneWhoCares Hosts File

I’ve been having intermittent problems with Firefox freezing up, pegging a CPU, eating all my memory, and freezing my desktop environment over the past few weeks. The only apparent trend in their occurrence was that they happened while I was active on HabitRPG and other ridiculously Javascript-heavy sites.

I disabled AdBlock Plus, and haven’t had a similar freeze since then. I also haven’t seen an increase in the amount of annoying ads, even when I visit questionable websites. This is because I took 30 seconds when first setting up my Arch installation to take an incredibly easy and effective action against ads.

Read more...

Making a PDF of Hieroglyph slides

I build slides for my presentations with Hieroglyph, which makes beautiful HTML presentations out of raw ReStructuredText. However, HTML slides with my speaker notes in a JavaScript console are not ideal for redistribution.

Read more...

Scale13x Retrospective

This year I had the pleasure of attending and speaking at the Southern California Linux Expo. Here’s what I observed about the location, the conference’s organization, and my own talk.

Read more...

Automating Irssi

I use Irssi as my IRC client because I don’t need to write my own plugins for it, and its documentation is excellent even when you’re not sure what questions to ask. If I needed plugins that haven’t already been written by others, and didn’t mind either reading the entire manual or constantly asking for help to solve simple problems, I would switch to Weechat.

I’ve configured Irssi to automatically join the networks that I want, authenticate me to services, then join my preferred channels. The Irssi beginner docs are a great reference for client commands to set these functions up, but if I was doing it over again, it would save a lot of keystrokes to simply hand-write the config file which those commands end up generating. Here’s how you can do that.

Read more...

Starting screen+irssi at boot

Installing security updates isn’t always enough to apply them; you have to reboot from time to time if you want the benefit of any kernel upgrades that don’t use live patching.

There’s also a small chance that my hosting provider might reboot my VPS unexpectedly (though I’ve never had that happen in almost a year of using DigitalOcean).

In both cases, I would like my IRC client to get back online as soon as it can after reboot.

Read more...

Résumé Improvement with LaTeX Macros

It’s career fair season at OSU, which means that I’m taking my annual dive into LaTeX programming in order to improve my résumé.

I’m discovering how powerful and easy to use the macros in LaTeX are. Let’s talk about why you might want to convert your résumé to LaTeX, and how to do it easily.

Read more...

My Second Steps with Javascript

There are many things to hate about Javascript. I’m not a fan of the language, and I’ve been known to laugh at people when they make unqualified claims that it’s “good”.

However, I sometimes find myself wanting to build toys and share them with everyone who has a web browser. Compared to the security and scaling implications of making my poor little VPS run a bunch of other people’s code for them, Javascript becomes the only viable option.

Read more...

Interview on opensource.com

As part of my upcoming talk at SCALE, I was interviewed on opensource.com.

Nitish Tiwari interviewed me by email about my talk, then edited my comments into what I feel like ended up as a very nice publication. I appreciate the CC-BY-SA license under which he released the article.

Installing gavrasm on Arch

Avra, the AVR assembler that I have been using for my ECE375 assignments, started throwing cryptic error messages such as “PRAGMA directives currently ignored” from an include file which had previously been working fine.

In order to sanity check whether the problem is in my code or the compiler, I installed gavrasm, the assembler which the ECE375 lab website recommends for Mac users.

Read more...

Mailman and Multiple Addresses

“Your message to the list requires moderator approval”, replies Mailman when you try to post to your mailing list.

“But I’m on the list!” you complain, then waste a few minutes of your day finding the administrative password and releasing your message.

Read more...

Automating Arch, I3, and Terminator to do the right thing at startup

I’m currently on my second clean install of Arch Linux. On the whole I’ve been glad that I ditched most of my old configuration when I installed Arch to the new SSD I got for my laptop at the beginning of the school year. However, I’d been procrastinating on re-implementing a few things which worked last install without me really knowing which of my many attempts had fixed them. This time, I know what I’m doing and what questions to ask.

Read more...

Please License Your Code

Code without a license isn’t open source. It isn’t free software, either.

Posting your code publicly doesn’t inherently apply a license to it.

Read more...

20 HabitRPG levels in 5 days

I’ve been having fun with HabitRPG lately, and discovered this One Neat Trick:

Checklists.

Read more...

The Trouble with Toctrees

It’s a couple weeks and nearly a dozen posts into this Tinkerer experiment, I’m mostly delighted with it. It’s fulfilling its original promise of “write RST, push button, get pretty blog”... Mostly. There’s one problem, though. I constantly forget to add master.rst when committing.

Read more...

Gamifying Adulthood with HabitRPG

After creating an account and abandoning it some time before December 2013 (since I have never yet subscribed and yet had a Trapper Santa scroll in my inventory), I have returned to HabitRPG. Here’s a quick examination of why I think I left and then came back.

Read more...

Arbitrary Python versions on Flip

A question in the #osu-lug channel about running python 2.7 on a school server (which only has python 2.6.6) made me realize I should know how to do that but have never tried it. Stackoverflow provides instructions for solving a similar problem, so I’m testing them out to make sure they work for Python 2.7.7 and Python 3.

Read more...

Vim: Open file with cursor at the end

As part of a recent quest to automate everything and learn more Vim tricks, I’ve been identifying patterns in my use of the editor and attempting to get them done with fewer keystrokes.

Read more...

What Makes a Good Mailing List Post?

As part of my student club officer duties, I send a lot of emails. One game that makes this chore less onerous is to try to optimize each email’s quality. I do this by observing my own reaction to others’ postings, and others’ reaction to my posts. Here are a few trends I’ve noticed.

Read more...

Those funny characters in Vim

../../../_images/vim_encoding.png

I copied and pasted some of the lines from a PDF, and now I have a problem which is nearly impossible to Google.

Read more...

Searching a FOSS project’s history

I’m curious about whether anyone has tried to build a predictive analytics plugin for Heka before. To find out, I’m going to stalk the project’s entire recorded history. Since it’s a relatively young project (only in its third year of having a public mailing list), the history is small enough for basic Linux command-line utilities to handle in a timely manner.

Here are all the places one can look for project history, and how I used them.

Read more...

ECE375: Using an Arduino Uno as a programmer

I have an atmega128 development board for the ECE375 class at Oregon State University. I believe the board is good, because it runs the test program that it came with when I picked it up from TekBots. I also have an Olimex AVR-ISP-MK2 programmer inherited from an ECE major friend, which I have come to conclude is bad, because despite testing and rebuilding all of the connections between it and my atmega128 board, despite passing avrdude all of the force and override-warnings flags at my disposal, it consistently refuses to program.

Since my assignment is due tomorrow, I am configuring an Arduino Uno to stand in as a programmer. Here’s how.

Read more...

Don’t rename Tinkerer posts

Or if you must, at least do it correctly. Here’s how.

Read more...

Floating-point Forth

The first assignment for CS480 (Translators) requests that we use Forth as a pocket calculator, rather than teaching the immensely powerful composition strategies for which it’s valued in the real world.

Since our first exposure to the language is a deep dive into the syntax of floating-point computation, no single tutorial on the web answers all the strange assortment of introductory and advanced questions that my classmates and I are running into.

Read more...

Blogging with Tinkerer

I had a wok site here for a while, but I rarely (okay, never) updated it. My experience with blogging platforms has been limited to Wordpress (both self-hosted and on wordpress.com), Wok, Pelican, and an abomination of a Trac plugin that I’d prefer to forget.

Here’s how and why I am now trying Tinkerer.

Read more...