rjbs forgot what he was saying

not logged in (root) | by date | tagcloud | help | login

recent tags

2   journal  
1   rpg  

RSS feed entries

collapse entry bodies

personal code review practices, mark Ⅱ (body)

by rjbs, created 2013-08-28 10:34
last modified 2013-08-28 10:51

Just about two months ago, I posted about my revived "work through some tickets in each queue then rotate" strategy. When I had first tried to do it, I hadn't had enough discipline, and it failed. After a month, it seemed to be going very well, because of two minor changes:

  1. I replaced "remember where I was in the list" with "keep the list in a text file."
  2. I used The Daily Practice to keep track of whether I was actually getting work done on the list regularly.

About a month later, I automated step 2. I just had my cron job keep track of the SHA1 of the file in git. If it changed, I must have done some work.

Yesterady, as I start month three of the regime, I've invested a bit more time in improving it, and I expect this to pay big dividends.

My process is something like this, if you don't want to go read old posts:

  1. keep a list of all my projects
  2. group them into "bug queue cleared" and "project probably complete" and "work still to do"
  3. sort the lists by last-review date, descending; fall back to alphabetical order
  4. every time I sit down to work on projects, start with the first project on the "work to do" list, which has been touched least recently
  5. when new bugs show up for the other two lists, put them into the "work to do" list at the right position

This was not a big problem. I kept the data in a Markdown table and when I'd finish review, I'd delete a line from the top, add it to bottom, and add today's date. The step that looked like it would be irritating was #5. I'd have to keep an eye on incoming bug reports, reorder lists, and do stupid maintenance work. Clearly this is something a computer should be doing for me.

So, the first question was: can I get the count of open issues in GitHub? Answer: yes, trivially. That wasn't enough, though. Sometimes, I have older projects with their tickets still in rt.cpan.org. Could I find out which projects used which bugtracker? Yes, trivially. What if the project uses GitHub Issues, but has tickets left in its RT queues? Yes, I can get that.

Those are the big things, but once you pick up the data you need for figuring them out, there are other things that you can check almost for free: is my GitHub repo case-flattened? If so, I want to fix it. Is the project a CPAN dist, but not built by Dist::Zilla? Did I forget to enable Issues at GitHub? Am I missing any "Kwalitee point" on the CPANTS game scoreboard?

code-review

Writing the whole program took an hour, or maybe two, and it will clearly save me a fair bit of time whenever I do project review. I even added a --project switch so that I can say "I just did a round of work on Some::Project, please update my last reviewed date." It rebuilds the YAML file and commits the change to the repo. Since it's making a commit, I also added -m so I can specify my own commit message, in case there's something more to say than "I did some work."

This leaves my Markdown file in the lurch. That wouldn't bother me, really, except that I've been pointing people at the Markdown file to keep track of when I might get to that not-very-urgent bug report they filed. (I work on urgent stuff immediately, but not much is urgent.) Well, no problem here: I just have the program also regenerate the Markdown file. This eliminates the grouping of projects into those three groups, above. This is good! I only did that so I could avoid wasting time checking whether there were any bugs to review. Now that my program checks for me, there's no cost, so I might as well check every time it comes up in the queue. (Right now, it will still prompt me to review things with absolutely no improvements to be made. I doubt this will actually happen, but if it does, I'll deal with it then.)

The only part of the list that mattered to me was the list of "stuff I don't really plan to look at at all." With the automation done, the list shrinks from "a bunch of inherited or Acme modules" into one thing: Email-Store. I just marked it as "never review" and I'm done.

So, finally, this is my new routine:

  1. If The Daily Practice tells me that I have to do a code review session…
  2. …or I just feel like doing one…
  3. …I ask code-review what to work on next.
  4. It tells me what to work on, and what work to do.
  5. I go and do that work.
  6. When I'm done, I run code-review --project That-Project and push to github.
  7. Later, a cron job notices that I've done a review and updates my daily score.

Note that the only part of this where I have to make any decisions is #5, where I'm actually doing work. My code-review program (a mere 200 lines) is doing the same thing for me that Dist::Zilla did. It's taking care of all the stuff that doesn't actually engage my brain, so that I can focus on things that are interesting and useful applications of my brain!

My code-review program is on GitHub.

the stupidest profiler I could write (body)

by rjbs, created 2013-08-23 23:10
last modified 2013-08-23 23:11

There's a stupid program I rewrite every few months. It goes like this:

perl -pe 'BEGIN{$t=time} $_ = sprintf "%0.4f: $_", time - $t; $t = time;'

It prints every line of the input, along with how long it had to wait to get it. It can be useful for tailing a log file, for example. I wanted to write something similar, but to just tell me how long each line of my super-simple program took to run. I decide it would be fun to do this with a Devel module that would get loaded by the -d switch to perl.

I wrote one, and it's pretty dumb, but it was useful and it did, in the end, do the job I wanted.

When you pass -d, perl sets $^P to a certain value (on my perl it's 0x073F) and loads perl5db.pl. That library is the default perl debugger. You can replace it with your own "debugger," though, by providing an argument to -d like this:

$ perl -d:SomeThing ...

When you do that, perl loads Devel::SomeThing instead of perl5db.pl. That module can do all kinds of weird stuff, but the simplest thing for it to do is define a subroutine in the DB package called DB. &DB::DB is then called just before each statement runs, and can get information about just what is being run by looking at caller's return values.

One of the bits set on $^P tell it to make the contents of each loaded file available in a global array with a funky name. For example, the contents of foo.pl are in @{"::_<foo.pl"}. Woah.

My stupid timer keeps track of the amount of time taken between statements and prints your program back at you, telling you how long was spent on each line, without measuring the breakdown of time spent calling subroutines loaded from elsewhere. It expects an incredibly simple program. If you execute code on any line more than once, it will screw up.

Still, it was a fun little exercise, and maybe demonstrative of how things work. The code documentation for this stuff is a bit lacking, and I hope to fix that.

use strict;
use warnings;
package Devel::LineTimer;
use Time::HiRes;

my %next;
my %seen;
my $code;
sub emit_trace {
  my ($filename, $line) = @_;
  $code ||= do { no strict; \@{"::_<$filename"}; };
  my $now = Time::HiRes::time();

  $line = @$code if $line == -1;

  warn "Program has run line $line more than once.  Output will be weird.\n"
    if $seen{$line}++ == 1;

  unless (keys %next) {
    %next = (start => $now, line => 1, hunk => 0);
  }

  my @code = @$code[ $next{line} .. $line - 1 ];

  my $dur = $now - $next{start};

  printf STDERR "%03u %04u %8.04f %s",
    $next{hunk}, $next{line}, $dur, shift @code;

  my $n = $next{line};
  printf STDERR "%03u %04u %8s %s",
    $next{hunk}, ++$n, '.' x 8, $_ for @code;

  %next = (start => $now, line => $line, hunk => $next{hunk}+1);
}

package DB {
  sub DB {
    my ($package, $filename, $line) = caller;
    return unless $filename eq $0;
    Devel::LineTimer::emit_trace($filename, $line);
  }
}

END { Devel::LineTimer::emit_trace($0, -1) }

1;

With the module above installed in @INC somewhere, you can then run:

$ perl -d:LineTimer my-program

...and get output like...

000 0001   0.0000 #!perl
000 0002 ........ use 5.16.0;
000 0003 ........ use warnings;
000 0004 ........ use Email::MessageID;
000 0005 ........ use Email::MIME;
000 0006 ........ use Email::Sender::Transport::SMTP;
000 0007 ........
001 0008   0.0093 my $email = Email::MIME->create(
001 0009 ........   header_str => [
001 0010 ........     From => 'Ricardo <rjbs@cpan.org>',
001 0011 ........     To   => 'Mr. Signes <devnull@pobox.com>',
001 0012 ........     Subject => 'This is a speed test.',
001 0013 ........     'Message-Id' => Email::MessageID->new->in_brackets,
001 0014 ........   ],
001 0015 ........   body => "There is nothing much to say here.\n"
001 0016 ........ );
001 0017 ........
002 0018   0.0028 my $smtp = Email::Sender::Transport::SMTP->new({
002 0019 ........   host => 'mx-all.pobox.com',
002 0020 ........ });
002 0021 ........
003 0022   1.7395 $smtp->send($email, {
003 0023 ........   to   => 'devnull@pobox.com',
003 0024 ........   from => 'rjbs@cpan.org',
003 0025 ........ });

Yahoo!'s family accounts still stink (body)

by rjbs, created 2013-08-22 16:57
tagged with: @markup:md journal

It is amazing how bad Yahoo!'s "family account" experience is. I want to make an account for my six year old daughter to use to upload her photos to Flickr. Googling for Yahoo! family accounts and flickr finds this text:

Yahoo! Family Accounts allow a parent or legal guardian to give consent before their child under 13 creates an account with Yahoo!. A child is someone who indicates to us that they are under the age of 13. [...] Your child may have access to and use of all of Yahoo!'s products and services, including Mail, Messenger, Answers, mobile apps, Flickr, Search, Groups, Games, and others. To learn more about our privacy practices for specific products, please visit the Products page of our Privacy Policy.

(Emphasis mine, of course.)

I had to search around to find where to sign up. I had to use the normal signup page. I filled the form out and kept getting rejected. "The alternate email address you provided is invalid." After trying and trying, I finally realized that they have proscribed the use of "yahoo" inside the local part. So, com.yahoo.mykid@mydomain.com was not going to work. Fine, I replaced yahoo with ybang. Idiotic, but fine.

After I hit submit, I was, utterly without explanation, given a sign in page. I tried to sign in several times with the account I just requested, but was told "no such account exists."

Instead, I tried to log in with my own account, and that worked. I was taken to a page saying, "Do you want to create a Family Account for your child?" Yes, I do! Unfortunately, the CAPTCHA test that Yahoo! uses is utterly awful. It took me half a dozen tries to get one right, and I've been human since birth. Worse, the form lost data when I'd resubmit. It lost my credit card number — which is excusable — but also my state. Actually, it was worse: it kept my state but said "this data is required!" next to it. I had to change my country to UK, then back to USA, then re-pick my state. Then it was satisfied.

Finally, I got the account set up. I was dropped to the Yahoo! home page, mostly showing me the daily news. (To Yahoo!'s credit, none of this was horrible scandal rag stuff for my six year old. Less to their credit, the sidebar offered me Dating.) I verified my email address and went to log her in to Flickr. Result?

We're sorry, you need to be at least 13 years of age to share your photos and videos on Flickr.

So, what now? Now I create a second account as an adult, upload all her photos there, and give her the account when she's older, I guess. Or maybe I'll use something other than Flickr, since right now I'm pretty sick of the many ways that Yahoo! has continued to make Flickr worse.

getting stuff accomplished, next steps (body)

by rjbs, created 2013-08-16 22:57
last modified 2014-02-17 18:35

I've got nearly every goal on my big board lit up. So, now I'm getting into a routine of getting all the regular things done. Next up, I'm going to try to get better at doing the one-off tasks I have to do, like file my expenses, arrange a piano tuning, and that sort of thing. For this, I'm going to try using Remember the Milk. I've used it in the past and liked it fine, but I didn't stick with it. I think that if I integrate it into my new routine, it'll work.

I've put in a few tasks already, and gotten some done. Today, I added some tasks with the "nag" tag, telling me that they're things I need to bug other people about until they get done. Other tasks, I'm creating with due dates. Yet others are just general tasks.

My next step will be to use the Remember the Milk API (with WebService::RTMAgent, probably) to help deal with these in three ways:

  1. require that I never have any task more than one day overdue (I'm cutting myself a little slack, okay?)
  2. require a new note on any "nag" task every once in a while
  3. require that ... well, I'm not sure

That lousy #3 needs to be something about getting tasks done. I think it will be something like "one task in the oldest 25% of tasks has to get done every week." I think I won't know how to tweak it until I get more tasks into RTM.

Maybe I'll do that on vacation next week. That sounds relaxing, right?

the perils of the Daily Practice (body)

by rjbs, created 2013-08-09 20:26

Warning: This is sort of rambling.

I have no doubt that automating a bunch of my goals on The Daily Practice has helped me keep up with doing them. As I keep working to stay on top of my goals, though, I'm finding that the effects of TDP on my activity are more complex and subtle than I had anticipated.

The goals that are getting the most activity are:

  • automated
  • already started
  • achievable with a small amount of work performed frequently

My best streak, for example, is "review p5p commits." All I have to do, each day, is not have any unread commit notifications more than a week old. Every day, we have under two dozen notices, generally, so I can just read the ones that come in each day and I'm okay. If I miss a day, I'm still good for a while. After that comes "catch up with p5p," which is the same.

The next goals are in the form "do work on things which you will then record by making commits in git." For example, I try to keep more on top of bug reports lately. So far, so good. These goals are still going strong, and have been going strong for as long as my other automated goals. The score is lower, though, because they don't show up as done each day, but only on days I do the work. Despite that, the structure of the goals is the same: make sure the work is done before each safe period is over. This suggets an improvement to TDP: I'd like my goals' scores to be their streak lengths, in days, rather than the number of times I've performed something. This seems obvious to me, in retrospect.

The goal that trails all of these is "spend an hour on technical reading." I didn't get started on that immediately. Once I did, though, I've been motivated to keep the chain going. My strong suspicion, though, is that I only felt motivated because I had already established streaks with my easier to perform, automaticaly-measured goals. Still, my intuition here is that it's much easier to get going once at least a single instance is on the big board. Unless there's a streak at all, there's no streak to break. This suggests another improvement, though a more minor one. Right now, scores are only displayed for streaks with more than one completion. You don't see a score until you've done something twice. I think it would be better to keep the streaks looking visually similar, to give them all equal value. After all, the value isn't that I did something 100 times in a row, but that for 100 days, it was getting done.

Then come the goals that I haven't started at all. These goals are just sitting there, waiting for me to start a streak. Once I do start, I think I'll probably stick to it, but I have to overcome my initial inertia. Once I get it started, I get my nice solid line, and then I have a reason to keep it going. On the other hand, if I have no streak, there is no incentive to get started. I think this is a place to make improvements: just like I'd rather see scoring mean "every day in the streak is worth one point," I'd like to see "every day that a goal is not safe counts as a cumulative negative point." Now I can't just put in goals that I might start eventually. Leaving a goal undone for a long time costs me. I think there's something more to be found here, around the idea that something done irregularly, even if not meeting the goal, is better than something utterly ignored. Right now, that isn't reflected. Maybe that's for the best, though.

These aren't the real "dangers" to my productivity that I've seen in using TDP. There are two main things that I've worried about.

First, TDP sometimes squashes the value in doing more than one's goal. For example, my bug-fixing task says I have to do an hour of work every three days. On a certain day, I might feel motivated to do more than one hour of work. I may feel like I'm on a roll. I will not be rewarded for doing so. In theory, I could get two points for the day instead of one, but it won't actually extend my streak, which is what really counts. That is: if my streak is extended, I'm earning a day off from the task, so I have more time to do other work. This is what should happen if I do extra work today. It isn't what happens, though, which makes it a strange economy.

A related phenomenon is that if I were to write two journal entries today, I would benefit from saving one to publish later, because then the streak would extend from that day. It feels like a disincentive to actually do the extra work today, although this may be a problem with me that I need to work out on my own. In fact, there is a flip-side to this problem: if I do extra work now to extend my streak beyond its usual lenght, I'm breaking the regularity of my schedule, which might not fit in with the idea of getting into a schedule.

I don't really buy that, though.

The other problem is that once you buy into the idea that you must keep your streaks going — which is a pretty motivating idea — you're prioritizing things for which goals have been created over things for which they have not. Possibly you're heavily prioritizing them. It's important to remain aware of this fact, because there's a danger that any other work will be neglected only because you haven't thought to put it on the big board.

There are categories of tasks, too, that I've been struggling not to unconsciously deprioritize because they can't be usefully made into long-term goals. I'm trying to learn new Decktet games, to make plans to see friends more often, to work on spare time code projects, and so on. These are more "to do" items, and TDP is not a todo list. I think I'm going to end up having to write automation to connect it to a todo list manager, much as I did for my code review todo. Otherwise, I'll chug along with my current routine, but will stagnate by never doing new things.

These are good problems to have. They're the problems I get to have once I'm making reliable progress at keeping up with clear responsibilities or promises. Nonetheless, they are problems, and I need to recognize them, keep them in mind, and figure out how to overcome them.

the Random Wizard's Top 10 Troll Questions (body)

by rjbs, created 2013-08-02 14:15
last modified 2013-08-02 17:47
tagged with: @markup:md dnd journal rpg

I've not usually a big fan of blog-propagated questionnaires, but this one looked good, because it will force me to articulate a few of my thoughts on my D&D game. These are Random Wizard's Top 10 Troll Questions. I already posted answers to the 20 Quick Questions on Rules that he mentions, for my D&D game. My answers to the 20 Quick Questions are in the game's GitHub repo.

1. Race (Elf, Dwarf, Halfling) as a class? Yes or no?

Yes. I like the idea of matching classes up with monster manual entries. I have a Fighter class for now, but will probably break it up to Bandit and Soldier, eventually, to match my game's monster manual. So, I match elf or dwarf up with the monster manual entry. Goblins, in my manual, break down into a number of very distinct groups. If someone was to play a goblin, I'd make a per-group class, or at least have rules for specialization within the goblin class, the same way I customize clerics per-church.

2. Do demi-humans have souls?

The nature of the personal life force of a sentient creature is sort of blurry in my game, but the rough answer is, "Yes, but some demi-humans have different kinds of souls." Elves are often unable to interact with the technology of the ancient empire, for example, because it doesn't consider them to be alive at all.

3. Ascending or descending armor class?

Descending. I really like using THAC0, I find it very easy to do the math in my head. In fact, everyone I've played with does, once I get them to stop reacting violently to "this stupid thing I hate from 2E" and see how simple the rule is.

4. Demi-human level limits?

Probably, it hasn't come up. In fact, human level limits, too. I don't see anybody PC breaking past level 12-14 in my game, ever.

5. Should Thief be a class?

Yes. Actually, a few classes. When I get around to it, I want to break Thief into Assassin, Burglar, and Dungeoneer. Or, the other way to put this is: I do like the idea of classes for skill-based archetypes, but I think that Thief, as written, is not a very good such class. I'm not sure who it best represents in the fiction? With its d4 hit dice, it's neither the Grey Mouser nor Conan, both of whom would otherwise be decent candidates.

6. Do characters get non-weapon skills?

Kinda. I should really codify it. Basically, I assume that characters are good at the stuff related their archetype. (This is part of why I like more specialized classes than "Fighter.") If the player wants to declare that his or her character has an unusual skill for some reason, I'll allow it at least a few times.

I don't like skill lists.

7. Are magic-users more powerful than fighters (and, if yes, what level do they take the lead)?

We're using pretty basic fighter and magic-user classes, most of the time. Even the tweaks I'd like to make won't change the balance much I think. So, at low levels, the fighters are more powerful. So far, we haven't seen any magic-user survive long enough to overtake the fighters.

I've been slowly tweaking the rules to try to change the balance just a little.

8. Do you use alignment languages?

No.

I have publicly stated my bafflement by alignment languages before, and although I was glad to get a pretty clear answer as to why they existed, I didn't think they were really justified. When different cults have secret languages, they're just secret languages.

9. XP for gold, or XP for objectives (thieves disarming traps, etc...)?

Yeah, sure, XP for all kinds of stuff. Gold, monsters, traps, fast-talking, whatever. I wrote about gold as experience before.

10. Which is the best edition?

Heh.

Right now, I use the Moldvay Basic Set as the go-to reference, with plenty of stuff from Cook's Expert Set. I'd like to read Holmes, as I have read good things, and it looks like at least I should steal some of its rules for stuff, but I don't have a copy. I stole some of the psionics rules from 2E, and 1E has tons of tables and stuff to steal. I'm hacking in something like Action Points when I backport my 4E campaign to Basic. They're all fun, but I think Moldvay is a great framework from which to start hacking, and that's what I've done.

Bonus Question: Unified XP level tables or individual XP level tables for each class?

Individual tables. I really like unified XP, in theory, because it can make multiclassing a lot simpler. In practice, I've never really liked how it works.

being a polyglot programmer (barely) (body)

by rjbs, created 2013-07-31 22:03
tagged with: @markup:md journal programming

I like learning new programming languages. Unfortunately, I rarely make the time to get any good at them. I'm hoping to figure out how to force myself to write something non-trivial in something at least relatively unlike what I do all day.

I did some hacking on a Python implementation of statsd, and I started on a tool to build Z-machine programs in Perl 6. I took an online course in Scala and got this:

I know Scala!

That was fun, and I did write some things more complex that I might have written if I was working through a book. Speaking of working through books, I read a bunch of Introducing Erlang and Erlang Programming, and of Haskell: the Craft of FP. Now I'm back to working through Starting Forth — thankfully I have a print edition, as the web one is dreadful. I really enjoyed Programming in Prolog, too, and hope to someday get around to The Craft of Prolog. There are a number of other languages or books I could mention, but it all comes down to the same thing: I'm very good at dabbling, but I've found it very challenging to become proficient at foreign languages because I've found no motivation in the form of code I want to write. Or, more specifically, code that I want to write, but that I can't write faster in Perl, or where I don't mind suffering getting it done more laboriously.

What I need to do is look for existing programs I like, and want to hack on, and then do so. I think it will probably be really painful, but worth it.

What I wish I could do is become good friends with somebody expert in these languages, interested in helping me learn, possibly while hanging out over chips and salsa. Actually, it seems to turn out that part of the reason I'm not getting as much Erlang programming as I'd like is the same reason I'm not playing as much D&D as I like. Making friends isn't so hard, but finding friends with the exact set of skills and interests you hope they'll have can be a pretty tough challenge!

Dream Codex : Fonts

more old school computer fonts, including TI99/4A
by rjbs, created 2013-07-29 22:16
tagged with: fonts

Apple II Fonts

old school computer fonts
by rjbs, created 2013-07-29 22:16
tagged with: fonts

I get points for blogging this! (body)

by rjbs, created 2013-07-25 12:47
last modified 2014-02-17 18:36

I feel like I'm always struggling with productivity. I don't get the things done that I want to get done, and I'm never sure where I lost my momentum, or why, or how I can keep with it. I've tried a bunch of productivity tools, and most of them have failed. For a while, now, I've had an on-again-off-again relationship with The Daily Practice, which I think is great. Even though I think it's great, I don't always manage to keep up with it, which means it doesn't actually do me much good.

I'm on-again, though, and I'm trying to use it to combine some of my other recent changes to my routine. For example, I wrote about using a big queue to many all my projects' bug queues and forcing a reading bottleneck to avoid reading too many books at once, and thus getting none read. I also want to try to get sustained momentum on keeping my email read and replied-to. I'm not sure whether TDP will help me stay on target, but I think it will help me have a single place to see whether I'm on a roll.

The Daily Practice is a calendar for your goals. You tell it what things you want to do, and how often. Then, when you've done the thing, you tell TDP that you did. As long as you keep doing things often enough, you rack up points every time that you extend the chain. If you fail to keep it going, you lose all your points. It looks like this:

The Daily Practice

I started to think about what kind of goals would be useful to demonstrate momentum. My list looked something like:

  • get some email-reading done
  • clear out old mail that's marked for reply
  • spend time working on RT tickets and GH issues
  • catch up with reading p5p posts
  • review commits to perl.git
  • keep processing my long-lived perl issues list
  • write blog posts
  • read books
  • read things long-ago marked "to read" in Instapaper
  • keep up with RSS reader
  • keep losing weight

I started to think about how I'd track these. It was easy to track my email catch-up on my last big push. I was headed to Massachusettes with my family and while Gloria drove, I read email. Every state or so, I'd tweet my new email count. Doing this from day to day sounded annoying, though. If I just finished reading two hundred email messages, I want to reward myself with a candy. I don't want to go log into my productivity site and say that I've done it.

Fortunately, I realized that just about all the goals I had could be measured for me. I just needed to write the code and post results to TDP. I made doe eyes at TDP support and was given access to the beta API. Then I didn't do anything about it for a while… until getting to OSCON. I've been trying to use the conference as a time to work through some outstanding tasks. So far, so good.

I've written a program, review-life-stats, which measures the things I care about, when possible, and reports progress to TDP. Some of the measurements are a little iffy, or ripe for gaming, or don't measure exactly what I pretend they do. I think it will be okay. The program keeps track of the last measurement for each thing, storing them in a single-table SQLite file. It will only do a new measurement every 24 hours.

The way to think of these monitors is that they only report success. It has to say "today was a good day" or nothing. There is no reason (or means) to say "today was bad," and my monitors don't consider whether there's a streak going on (but they could be made to if it would help).

Here are the ones I've written so far:

  • the total count of flagged messages must be below the last measurement (or 10, whichever is higher)
  • the total count of unread messages must be below the last measurement (or 25, whichever is higher)
  • I must have written a new journal entry
  • I must have made a new commit to perlball.git; any commit will do. I check this by seeing whether the SHA1 of the master branch has changed, using the GitHub API.
  • There must be no unread messages from the perl.git commit bot over three days old.
  • The count of unread perl5-porters messages over two weeks old must be below the last measurement or zero.

I won't be able to automate "did I spend time reading?" as far as I can predict. I'm also probably going to have to do something stupid to track whether I'm catching up on my bug backlog. Probably I'll have to make sure that the SHA1 of cpan-review.mkdn has changed. I'm also not sure about my taks to keep reviewing smoke reports or to plan my upcoming D&D games.

The other goals that I can automate, somehow, are going to be doing semi-regular exercise, keeping my weight descending, and working through my backlog of Instapaper stuff. Those will require talking to the RunKeeper, Withings, and Instapaper APIs, none of which look trivial. Hopefully I can get to each one, in time, though, and hopefully they'll all turn out to be simple enough to be worth doing.

dealing with my half-read book problem (body)

by rjbs, created 2013-07-05 22:47
last modified 2013-07-05 22:48
tagged with: @markup:md journal reading

I just recently wrote about trying to deal with my backlog of bug reports and feature requests. It is not, sad to say, the only backlog of stuff I've been meaning, but failing, to do. There's also my backlog of reading.

my overgrown reading queue

(and that's only the physical books)

I need to get through these! Lately, I'm only getting any reading done on new books, and I'm buying a lot of new books. Also, my birthday is coming up soon, and my wishlist is full of books. I don't want to stop wishing for books… I just want to feel like I have a plan to read the books I get!

Too often, I started reading a book, then put it aside to read something else, then never get back to the thing I was reading to start with. No more! Or, at least, not as often anymore! I don't mind giving up on a book, or even saying, explicitly, that I'll try again in a year. I just don't like the idea that I'm putting books aside "for a little while" and then never getting back to them. Perhaps ironically, the book with which I've done this the most often is The Magic Mountain, which I've been reading off and on since 1997. For that book, I think it is a good plan. For almost any other book I've been trying to read, not so much.

So, the new plan is this: I keep big list of all the books I'm really meaning to read. This doesn't mean everything I own but haven't read. It's all the stuff I've asked for or purchased in order to read "soon." It's not a queue, either, because I'll pick what to read as I go. Who wants to pick, a year in advance, what book to read around Christmas 2014? Not me, man. Not me.

I categorized each book in the list as either fiction, humanities, or technical. (I was sorely tempted to replace "technical" with "quadrivium" just to keep the tables nicely aligned without spaces, but I resisted.) I will work on no more than one book of each category at a time. When I finish one, I won't pick its replacement until I feel like it. Once I will either finish reading the replacement or clearly decide to either give up on it or put it away for a good long while, while I read something else.

To start, I picked the most recent story collection I was working on, a nice short technical book, and a thick and difficult book of history that I was really, really enjoying (in a way) when I was making real progress in it.

 Literature: Moon Moth and Other Stories, The
 Technical : Starting Forth
 Humanities: Rise and Fall of the Third Reich

I have no idae whether this plan will work, but I feel like it's better than continuing along the random non-plan that I've been following. I put the big list of book on GitHub along with my previous big list of software libraries. Hopefully keeping them in one place will help me tie them together as "things to keep updated" in my head.

once again trying to keep up with the tickets (body)

by rjbs, created 2013-07-03 22:42
last modified 2013-07-27 12:39

I maintain a bunch of published code. I probably wrote more than half of it, and I've been the sole maintainer for years on most of the rest. I inherited a lot of bug reports, I get new bug reports, and I get feature requests. I used to try to respond to everything immediately, or at least within a few days.

This doesn't happen anymore.

Now, I've got piles and piles of suggestions and patches, some of which are obviously good, some of which I'd like to see happen but don't want to write, some of which are okay but need work, and some of which just aren't a good idea. Any time I want to try to clear out some of these tickets, I have to pick which ones. This takes too much time, especially because I'm often feeling sort of run down and like I want to cruise through some easy work, so I spent a lot of time looking at lists of tickets, burning up the time I could be spending just doing something.

A while back — maybe six months or a year ago — I thought that what I'd do was work alphabetically through my module list. I'd keep working on the next target until every ticket was stalled or closed. Eventually I got burned out or distracted.

I also didn't announce this plan publicly enough, so failing to keep on it brought me insufficient shame.

Now I'm back to the plan, roughly as follows:

I made a big list of all my code. (I'm sure I missed some stuff, but probably not much. Hopefully those omissions will become clear over time.) I started with everything in alphabetical order. Every time that I want to get some work done on my backlog, I go to the list, start at the top, and clear as many tickets as I can. If I clear the queue, or when I'm done doing work for the evening (or afternoon, or whatever) I file the dist away.

If there are still tickets left, I put it at the bottom of the "main sequence," where I keep working on stuff over and over. Otherwise it goes into either "maintenance," meaning that it'll pop to the top of the queue once it actually gets bugs, or "deep freeze," meaning that I seriously doubt any future development will happen. The deep freeze is also where I put code that lives primarily in the perl core but gets CPAN releases.

While I'm going through my very first pass through the main sequence, things that have never been reviewed are special. If a bug shows up in a "maintenance" item, it will go to the top of the already-reviewed stuff in the main sequence, but below everything still requiring review. I'm also checking each module to make sure that it points to GitHub Issues as its bug tracker and that it doesn't use Module::Install.

What I should do next is write a few tools:

  • something to check the bugtracker and build tool of all my dists at once
  • something to check whether there are tickets left in the rt.cpan.org queue
  • something to check whether there are non-stalled tickets for dists in maintenance or deep freeze

I don't know whether or when I'll get to those.

In the meantime, I'm making some progress for now. I'm hoping that once I finish my first pass, I'll be able to do a better job of clearing the backlog and keeping up, and then I'll feel pretty awesome!

I may try to publish more interesting data, but for now my maintenance schedule is published, and I'm keeping it up to date as I go.

Template Toolkit 2: still making me crazy (body)

by rjbs, created 2013-07-03 12:19

Template Toolkit 2, aka TT2, has long been a thorn in my side. Once upon a time, I really liked it, but the more I used it, the more it frustrated me. In almost every case, my real frustrations stem from the following set of facts:

  • TT2 is a templating system for Perl.
  • TT2 provides a language for use when adding logic to the templates.
  • The language is inferior to Perl. It may be useful to be inferior in some ways, to encourage programmers to move complex logic out of templates, but…
  • The language has significant conceptual mismatches with Perl.

I'll start with this object:

  package Thing {
    sub new      { bless { state => 0 } }
    sub name     { my $self = shift; $self->{name} // 'Default' }
    sub set_name { my $self = shift; $self->{name} = shift }

    sub next  { my $self = shift; return $self->{state}++ }
    sub error { my $self = shift; $self->{error} }
  }

…and I'll write this code…

  my $tt2   = Template->new;
  my $thing = Thing->new;

  my $template = <<'END';
  Got  : [% thing.next  %]
  Error: [% thing.error %]
  State: [% thing.state %]
  END

  $tt2->process(\$template, { thing => Thing->new })
    or die $tt2->error;

The output I get is:

  Got  : 0
  Error: 
  State: 1

We've got one problem, already: I was able to look at the object's guts, and not because I obviously dereferenced the reference as a hash, but because I forgot that state was not a method. There is, as far as I can tell, no way to prohibit fallback from methods to dereference by configuring TT2.

There's another problem: we stringified an undef, where we might have wanted some kind of default to display. In Perl we'd get a warning, but we don't here. We probably wanted to write:

Error: [% thing.error.defined ? thing.error : "(none)" %]

That works. That also calls error twice, so maybe:

Error: [% SET error = thing.error; error.defined ? error : "(none)" %]

…but that won't work, because it sets error to an empty string, which is defined. Why? Because TT2 doesn't really have a concept of an undefined value. This can really screw you up if you need to pass undef to an object API that was designed for use by Perl code.

This should be obvious:

Name : [% thing.name %]

You get "Default" as the name.

Name : [% CALL thing.set_name("Bob"); thing.name %]

…and we get Bob. If we ever needed to clear it again, though,

Name : [% CALL thing.set_name("Bob"); CALL thing.set_name(UNDEF); thing.name %]

Well, this won't work, because UNDEF isn't really a thing. It isn't declared, so it defaults to meaning an empty string. I thought you could, once upon a time, do something like this:

[% CALL thing.set_name( thing.error ) %]

…and that the undef returned by error would be passed as an argument. I may be mistaken. It doesn't work now, anyway.

We need to detect these errors, anyway, right? In Perl, we'd have use warnings 'uninitialized' to tell us that we did print $undef. In TT2, there's STRICT. We update our $tt2 to look like:

my $tt2   = Template->new(STRICT => 1);

Now, undefs in our templates are fatal. It's important to note that the error isn't stringifying undef, but evaluating something that results in undef. Our original template:

  my $template = <<'END';
  Got  : [% thing.next  %]
  Error: [% thing.error %]
  State: [% thing.state %]
  END

…now fails to process. The error is: var.undef error - undefined variable: thing.error. In other words, thing.error is undefined, so we can't use it. If we try to use our earlier solution:

  my $template = <<'END';
  Got  : [% thing.next  %]
  Error: [% thing.error.defined ? thing.error : "(none)" %]
  State: [% thing.state %]
  END

We still get an error:

  var.undef error - undefined variable: thing.error.defined

So, we can't check whether anything is defined, because if it isn't, it would've been illegal to evaluate it that far. You can always pass in a helper:

  my $undef_or = sub {
    my ($obj, $method, $default) = @_;
    $obj->$method // $default;
  };

  my $template = <<'END';
  Got  : [% thing.next  %]
  Error: [% undef_or(thing, "error", "(none)") %]
  State: [% thing.state %]
  END

  $tt2->process(\$template, { thing => Thing->new, undef_or => $undef_or })
    or die $tt2->error;

This, of course, still doesn't solve the inability to pass an undefined value to a Perl interface. In fact, it doesn't deal with any kind of variable passing.

I like the idea of discouraging templates from including too much program logic. On the other hand, I loathe the idea of providing a large and complex language in the templates that can still be used to put too much logic in there, but without making as much sense as Perl or working well with existing Perl interfaces.

I'll take Text::Template or HTML::Mason any day of the week, instead.

Notes from YAPC in Austin (body)

by rjbs, created 2013-07-01 10:34
tagged with: @markup:md journal perl yapc

I'm posting this much later than I started writing it. I thought I'd get back to it and fill in details, but that basically didn't happen. So it goes.

This year's YAPC was in Austin. A lot of people complained about the weather, but it was pretty much the same weather we had at home when I left home, so I wasn't bothered. This was good planning on the part of the YAPC organizers, and I thank them for thinking of me.

I'm just going to toss down notes on what I did, for future memory.

I landed on Sunday, having flown with Peter Martini and Andrew Rodland. Walt picked us up at the airport and we went to Rudy's for barbecue… but first we had to check in. I was worried, because it was after 19:00, and it sounded like nobody would be at the B&B to let me in. I called, and nobody was there. I wandered around the back of the building and found a note for me. It told me where to find my key and how to get in. "…and help yourself to the soda, lemonade, and wine in the fridge." Nice. I really liked the place, The Star of Texas Inn, and would stay there again, if the opportunity arose.

Rudy's was fantastic. I had some very, very good brisket and declared that I needed nothing else. I tried some of the turkey and pork, too, though, and they were superb. The jalapeño sausage, I could take or leave. The sides were great, too: creamed corn, new potatoes, potato salad. The bread was a distraction. I also had a Shiner bock, because I was in Texas.

From Rudy's we went to the DoubleTree, where lots of other attendees were staying, and I said hello to a ton of people. Eventually, though, Peter and I headed back to our respective lodgings. I worked a little on my slides and a lot on notes for the RPG that I planned to run on Thursday night.

Monday morning, I caught up with Thomas Sibley, who was staying at the same B&B. We had breakfast (which was fine) and headed to the conference. I attended:

  • Mark Keating's history of Perl, which was good, except that he seems to think that my name is "Ricky." I think he's been talking to my mom too much.
  • Sterling Hanenkamp's telecommuting panel discussion, on which I was a panel member. I think it went pretty well, although I wonder whether we needed an aggressive interviewer to push us harder.
  • John Anderson's automation talk, which was good, but to which I must admit I payed limited attention. I forget what I got distracted by, but something.

For lunch, we had a "p5p meetup" at the Red River Diner. The food was fine and the company was good, but we ended up quite a few more people than I'd expected, and it sort of became a generic conference lunch. Jim Keenan presented me with a copy of the Vertigo score, which is sitting on my desk waiting for a good 45 minute slot in which to be played. Sawyer was keen to get anything with blueberries in it. "We don't have these things in Israel, man! They're incredible!" I was tickled.

In the next slot, I spent most of my time in the hallway, talking to people who were interested in the state of Perl 5 development. The big questions that arose in these discussions, and similar ones later in the week: how can Perl 5 get more regular core contributors, and how can interested people start helping? For the second one, I need to boil things down to a really tight answer with a memorable URL. I'm not sure it will help, but it might.

I attended the MoarVM talk, which was interesting, but which I can't judge very well. At any rate, I'm excited to see the Rakudo team doing more cool stuff. After that, Larry spoke. It was good, and I was glad to be there. The lightning talks were good, and then there was the "VIP mixer." That's basically free drinks and an opportunity to meet all kinds of new people. I did! I would've met more, but it was loud in there. If we could've done it outside, I bet I would've stayed much longer, but I was losing my voice within the hour.

After that, we were off to Torchy's Tacos. Walt had previously described their tacos as "a revelation." They were definitely the two best tacos I'd ever eaten. Especially amazing was the "tuk tuk," a Thai-inspired delicacy. I went back to Torchy's twice more before I left town, and regret nothing. I'll definitely go again, if I go back to Austin.

Tuesday, in fact, Walt, Tom and I headed to Torchy's for breakfast. It was a good plan. We got to the venue in time for Walt to give his talk about OS X hackery (phew!). I saw a live demo of Tim Bunce's memory profiler, which is clearly something I'll be using in the future, though it looks like it will take significant wisdom to apply effectively. Before lunch, I took in Mark Allen's talk on DTrace, which provided more incentive for me to finally learn how to use the thing. I've been working on the giant DTrace book since YAPC. I also managed, during the talk, to predict and find a stupid little bug when DTrace and lexical subs interact.

For lunch, Walt suggested we eat Bacon, so we did. Peter, Walt, and I piled into his rental and got over there. The Cobb salad was very good, the bacon fries okay. I was very glad to have a selection of local beers beyond Shiner and Austin Amber, and the guy behind the counter suggested Fire Eagle, which I enjoyed.

After lunch, Reini's talk on p2, Karen's TPF update, Matt S Trout on automation, and then Stevan's talk about the state of Perl. Despite calling Perl "the Detroit of scripting languages," it made no mention of RoboCop, nor did it liken anyone to The Old Man, Clarence Boddicker, or Dick Jones. It was a good talk, but I was understandably let down.

For dinner, the whole conference (or much of it) headed out for barbecue. The barbecue was made by The Salt Lick, and while good, it did not beat out Rudy's. Dinner ended with "game night," and I ran a hacked up D&D game, set in what I'm calling Trekpocalypse. More on that in another entry.

Wednesday started with my last trip to Torchy's. It was good.

We took our time getting to the conference, and then I killed a bunch of time in the hallway track. The first talk I got to was Sawyer's talk about async. The talk was good, and one to learn from, as a speaker. I think he did a great job keeping people involved, especially with a hunk of "spot the error" slides in the middle. By the end, he had built a program that did a bunch of parallel queries against IMDB, and then showed results to the audience. He spent a fair hunk of time just commenting on the character actors he dug up, and this went over well, as he'd paid for the digression with strong technical content until that point. I was pleased!

After that, I was obliged to go to Andrew Rodland's talk on StatsD, as I knew that I needed to start using it at work. It was useful, and I've been graphing more stuff, now, which has also been cool. In fact, this talk led to me finding a bug in py-statsd, which has now been fixed. Woo!

After that, it was time for me to give my talk on perl 5. I think it went quite well! I had been worried about it, since I was editing and reworking it until the last day. I was happy with it, though, and will not be making major changes before giving it at OSCON. I look forward to seeing the attendee feedback, once it's in. After that, it was a lot of post-talk chat in the hallways, then Peter Martini's talk about his work on adding subroutine signatures to Perl. Everyone was excited, and rightly so.

After that, Matt S Trout and the lightning talks brought things to a conclusion, and I spent some time saying goodbyes. With everyone heading his or her own way, Tom Sibley and I decided that our way was "toward cocktails." I'd identified a place on Yelp that looked good, the Firehouse Lounge and we headed down. The drinks were okay, but it was amazingly loud, so we headed out. Actually, I should qualify the drink review: my drink was pretty good. I ordered the drinks for both of us, yelling the order across the bar, and I ordered the wrong thing for Tom, forgetting which one he'd settled on. I was mortified. Tom graciously let it go.

We hadn't eaten yet, and we were still hoping for some more drinks, so I consulted Yelp and it suggested Bar Congress, which was probably the best food and drink advice I've ever gotten from a website. I wrote a review which I won't repeat here, but: I would go there again in a heartbeat. If I get back to Austin, for some reason, I will make a point of getting there.

After dinner, we headed back to the inn and I turned in. I'd have to get up early for my flight, so I packed and went right to sleep. In the morning, I used the "Catch a Cab in Austin" iOS app that people at YAPC had been talking about. It worked, and I got to the airport with plenty of time, and my flights home were uneventful. As always, I'd had a great time, but I was ready to get home to my family.

Next year's YAPC::NA will be in Orlando, and although it won't be easy to be as good as this year's, I'm pretty sure it will do just fine.

print-ing to UDP sockets: not so good (body)

by rjbs, created 2013-06-27 20:21

We've been rolling out more and more metrics at work using Graphite and StatsD. I am in heaven. I'm not very good at doing data analysis, but fortunately there are some very, very obvious things I can pick out from our current visualizations, and I'm finding all kinds of things to improve based on these.

I'm using Net::Statsd::Client, as it looked convenient. Under the hood, for now, it uses Etsy::StatsD. I found a very confusing bug and when I told the author of Net::Statsd::Client, he confirmed that he'd seen it. I've worked out the details, and it has made me grumpy! The moral of this story will be: don't use print to send to a UDP socket. (I doubt I'll print to a socket again, after this.)

As a rule, I was sending very simple measurements to StatsD. They'd look like this:

  pobox.host.mx-1.messages|1c

This means: increment the counter with the given name.

StatsD listens for UDP. In theory, you can send a bunch of these in one datagram, and they're separated by newlines. In practice, though, I was sending exactly one measurement per datagram. Sometimes, though, the server was receiving mangled data. The metric names would be wrong, or the whole string would be mangled. I fired up a network sniffer and saw things like this:

  ost.mx-1.messages|1c␤pobox.host.mx-1.messages|1c␤pobox.host.
  mx-1.messages|1c␤pobox.host.mx-1.messages|1c␤pobox.host.mx-1.
  messages|1c␤pobox.host.mx-1.messages|1c␤pobox.host.mx-1.
  messages|1c␤

Okay, it's a bunch of +1 operations run together… but what's up with the first one being truncated? And, more importantly, what was sending them in one datagram!? A review of the StatsD libraries will show that they don't do any buffering. All that Etsy::StatsD does is open a UDP socket and print to it. You can send multiple metrics at once, if you want, but you have to go out of your way to do it, and I wasn't.

Further, sockets don't buffer their output in Perl! When you connect to a socket, it's set to auto-flush. Why was there buffering happening? Andrew Rodland, author of Net::Statsd::Client said that it only happened while the StatsD server was local and unavailable. Immediately, things fell into place.

If you're running Linux, you can try this fun experiment. First, run this server:

my $sock = IO::Socket::INET->new(
  Proto      => 'udp',
  LocalHost  => 0,
  LocalPort  => 3030,
  MultiHomed => 1,
);

while (1) {
  my $data;
  my $addr = $sock->recv($data, 1024);

  print "<<$data>>\n";
}

Then, run this client:

  use IO::Socket::INET;
  use Time::HiRes qw(sleep);

  my $sock = IO::Socket::INET->new(
    Proto    => 'udp',
    PeerHost => 'localhost',
    PeerPort => 3030,
  );

  for (1 .. 20) {
    print $sock "1234567890";
    sleep 0.5;
  }

You'll see the server print out the datagrams it's receiving. It all looks good.

If you start the server after the client, though, or kill it and restart it during the client's run, you'll see the server receive datagrams with the number sequence more than once. This is bad enough. My belief, which I haven't put hard to the test, is that when the buffer to send is full, data is lost from the left. Even if your data were capable of being safely concatenated, it wouldn't be safe.

This is, at least in part, a product of the fact that Linux tries much harder to deliver UDP datagrams to the local interface. They are, to some extent, guaranteed. I'm not yet sure whether the behavior of print in Perl with such a socket is a bug, or merely a horrible behavior emerging from the circumstances around it. Fortunately, no matter which, it's easy to avoid: just replace print with send:

  send $sock, "0123456789", 0;

With Etsy::StatsD patched to do that, my problems have gone away.

the 2013 Perl QA Hackathon in Lancaster (body)

by rjbs, created 2013-04-15 19:34
last modified 2014-03-18 17:46

I got home from Lancaster, this morning. I'd been there for the sixth annual Perl QA Hackathon. As usual, it was a success and made me feel pretty productive. Here's an account of most of the things I did:

Meetings!

There were many discussant-hours spent in room C-7C hashing out a bunch of things that needed hashing out. Although I was very interested in the outcome, and had some strong opinions about one or two things, I didn't want to get too tangled up in the discussion, so I split my attention between (mostly) coding and (a little) joining in. Others will surely write up the decisions of these meetings, so I won't, but in the end I felt that it was useful for me to be there and that I wasn't unhappy with any of the resolutions.

PAUSE!

At my first two QA Hackathons, I worked mostly on the CPAN Testers Metabase. Since then, my most common recurring project has been PAUSE. More specifically, I've mostly worked on the indexer. That's the program that looks at new files to decide whether it should put their contents into the 02packages file used by CPAN clients to install packages by name. It's a really important program, and I remain very interested in improving its maintainability. Once again, though, I wasn't just adding tests, but also doing some tweaks to how the PAUSE indexer works.

A few months ago at the NY Perl Hackathon, David Golden and I got to work fixing letter case behavior in PAUSE. The problem was that PAUSE treated package permissions case-insensitively even though not all supported Perl platforms would. The most commonly discussed problem was the conflict between File::stat, a core module, and File::Stat, a non-core module. If a user ran cpan File::Stat on Win32, he or she would end up with File/Stat.pm installed, and use File::stat would pick that up instead of the core module.

We'd done about three quarters of the work for this in New York, but I didn't send a pull request. I knew that we had an untested case: what happens when someone who owns Foo::Bar now uploads Foo::bar? We decided that it would replace the old entry. I wrote tests, which showed it didn't work, then David made it work. We also wrote a few more tests for other edge cases, and were pleased to find them all handled the way we wanted.

We also discussed problems with the non-uniqueness of distribution names on the CPAN. In short, non-maintainer of Text::Soundex should not be able to upload a dist called Text-Soundex and have it indexed. I implemented this, which ended up being a bit tricker to do than I expected, although the code changes weren't too bad. It was just getting there that took time. Unfortunately, over 1,000 distributions on the CPAN have names that don't match a contained package, so those had to be grandfathered in. I may send out a "consider changing your dist name" email, but I haven't decided. It isn't really such a big deal, but it nags at me.

I also did some work on the code that generates 03modlist.data, the registered module list. The future of this feature is unclear, and will have to get sorted out soon, probably over the next month or two.

Pod!

The thing I came to the hackathon knowing that I had to work on was Pod::Checker. In fact, most of the PAUSE work I did had to wait until I finished dealing with Pod::Checker. I was really not looking forward to the work, but I didn't want it to continue to languish, undone. I'm glad I started with it, because it only took about a day, and finishing it made me feel excited for the rest of my time: I'd be able to work on other things!

In 2011, there was a Perl project in the Google Summer of Code whose goal was to replace all core uses of Pod::Parser with Pod::Simple. Pod::Html was overhauled, but Pod::Usage and Pod::Checker weren't completed. Pod::Checker was mostly done, but not quite, and unfortunately languished in that state for some time. Since I'm keen to get Pod::Parser out of the core distribution, and since I know that nobody really wants to do this work, I decided it would be a good task to force myself to do while stuck in a room with nothing but my laptop and a bunch of sympathetic ears.

There were n kinds of Pod::Checker checks that needed to be implemented, reimplemented, or moved to Pod::Simple itself:

  • tests needed updating for the new mismatched =item type check from Pod::Simple
  • the totally broken "unescaped <" warning had to go
  • a warning for "no closing =back" got put into Pod::Simple, eliminating use of Pod-Parser's Pod::List
  • warnings for ambiguous constructs in L<> like leading spaces, unquoted slashes, and so on
  • the check for internal hyperlink resolution had to be reimplemented

...and a few other little things, like hash randomization bugs. I've filed a pull request with Pod-Simple for the patches that would go there, and my branch of Pod-Checker based on Marc Green's original work is also now on GitHub, waiting to get a trial release.

Once this is done, we'll get Pod::Usage converted, then we're done with everything but the actual warnings and subsequent removals!

Other Stuff!

I made a new release of CPAN::Mini, closing quite a few very old tickets. I also went ahead and made --remote optional. Maybe in the future, I might make --local optional, too! The biggest outstanding question is whether I will add any alternate configuration filename and location for Win32, rather than ~/.minicpanrc. I'm still conflicted.

I applied some patches to Router::Dumb, exposing (I think) an annoying missing behavior in Test::Deep. I'd like to figure out how to fix that Test::Deep problem soon, but it didn't happen this weekend.

I made a few other releases, including a release to Dist::Zilla that will make it always upload to PAUSE using HTTPS. I decided not to try to tackle anything bigger at the hackathon.

Free-Floating Helping!

This year, I think I spent less time than ever looking at other people's code to be a fresh set of eyes. On the other hand, I spent more time answering questions related to coordinating changes with blead and other future releases. "Is this a blocker?" was asked quite a few times as the rest of the room found some interesting bugs in bleadperl. "Shall I commit this to 5.19.0?" came up often, too.

The End!

I'm hoping to get some more work done on the Pod and PAUSE fronts, hopefully very soon, but maybe at YAPC. I'm looking forward to seeing the fruits of all the labors performed by the other hackers at the hackathon, too. (I started, here, to list "especially abc, xyz, …" but the list got far too long. Lots of good stuff is coming!) I also clearly found plenty of things I'd like to do, but not just yet. In other words, I'm ready for next year already!

I might write up some of the social bits of the trip a bit later. The short version of that is that I had a great time, enjoyed seeing old friends and making some new ones, and ate four servings of black pudding.

DKIM, Email::Simple, and heartache (body)

by rjbs, created 2013-04-09 14:06
last modified 2013-04-09 14:11

Header folding

These email headers are all supposed to be equivalent:

1 Foo: bar baz

2 Foo:  bar baz

3 Foo: bar
   baz

4 Foo: bar
     baz

It's part of the "folding white space" thing that is just one of the reasons that email is such an irritating format. In any of these cases, when your program has received the message and you as, "What's in the Foo header?" the answer should generally be "bar baz" and not anything else.

Representing headers in memory

Email::Simple uses a pretty simple mechanism for representing headers: a list of pairs. When a message is built in memory, its header object stores a list of name/value pairs. Since the above forms are all equivalent, they are reduced to the first form when parsed. If you read in form 4, above, your header will store:

@pair = (Foo => 'bar baz')

DKIM

DKIM is a mechanism for digitally signing parts of an email message to provide reliable evidence that some sender has taken responsibility for the post. Parts of the message are digested and signed by a private key. The public key can be found in DNS, and the message's recipient can verify the signature.

Here's a DKIM signature from a message I got from Twitter recently:

1 DKIM-Signature: v=1; a=rsa-sha1; c=simple/simple; d=twitter.com; s=dkim;
2     t=1365174665; i=@twitter.com; bh=LmB+XG63ICs3ubpceGdSzYEPG4o=;
3     h=Date:From:Reply-To:To:Subject:Mime-Version:Content-Type;
4     b=BRBSHqSmznpsZOEC1tbOtGdZu+YX20jL9NiEIAsepmOaazRCpzTYVfUMC9oEoomok
5      /X0HVHkBgrkYfp9sWTGcCrDHr+7zntfykKwDWNrgTx9+t64wTrASvcBlUD4lGxTw1T
6      +JPJtdI17YtTg7pvpsHYYOMmbNZCLNSFTpClo0RQ=

Sometimes messages change in flight. For example, "Received" headers are just about always added. For that reason, only some of the headers are signed. If the whole header was signed, it would be guaranteed to fail once somebody added Received (or did lots of other things). The "h" attribute in the signature says which headers were signed.

There are other ways to break a header. For example, the DKIM-Signature above references the Reply-To header. Twitter might set the header to:

To: "Twitter Customer Support"
  <support+F33D3AAE-A13D-11E2-85CF-F379509576E5@some.site.twitter.com>

...and then later someone emits the header as:

To: "Twitter Customer Support"
    <support+F33D3AAE-A13D-11E2-85CF-F379509576E5@some.site.twitter.com>

There's a change in the representation of the header, even if not the effective value. Does it matter? Maybe.

The "c" attribute in the signature says how values were "canonicalized" before signing. In relaxed canonicalization, changes to things like whitespace won't affect the signature. In simple canonicalization, they will. If the header is re-wrapped, the signature will be broken.

Simple canonicalization is the default canonicalization.

Finally, the DMARC standard is providing senders with a way to assert that DKIM signatures are a reliable test of a message's legitimacy. If a DKIM signature is broken, the message is not trusted. Breaking DKIM matters now more than it did before, because DKIM is taken more seriously.

Email::Simple and DKIM

Email::Simple stores the normalized form. When round-tripping a message, unless the header is folded exactly how Email::Simple would fold it, Email::Simple will re-fold the header. In other words, Email::Simple breaks DKIM signatures pretty often, even in simplest pass-through program that doesn't try to affect the message's content at all.

It amused and frustrated me, in fact, that with the Twitter message I posted above, the only change affected in the message was to the DKIM-Signature itself. The signature was rewritten to:

1 DKIM-Signature: v=1; a=rsa-sha1; c=simple/simple; d=twitter.com; s=dkim;
2  t=1365174665; i=@twitter.com; bh=LmB+XG63ICs3ubpceGdSzYEPG4o=;
3  h=Date:From:Reply-To:To:Subject:Mime-Version:Content-Type;
4  b=BRBSHqSmznpsZOEC1tbOtGdZu+YX20jL9NiEIAsepmOaazRCpzTYVfUMC9oEoomok
5  /X0HVHkBgrkYfp9sWTGcCrDHr+7zntfykKwDWNrgTx9+t64wTrASvcBlUD4lGxTw1T
6  +JPJtdI17YtTg7pvpsHYYOMmbNZCLNSFTpClo0RQ=

The only difference? Five tabs replaced with spaces and the omission of two extra spaces. This broke the signature.

Fixing Email::Simple

Email::Simple 2.200_01 is now on the CPAN, and it fixes this problem. When it reads a message in from a string, it keeps track of the exact lines that were used, and it will emit them again, unless the header is deleted or changed. If a header it set from within the program, it will be folded however Email::Simple likes. If you need to set a header field within your program and specify how it will be folded, you'll need to use another library.

If you're using Email::Simple (or Email::MIME) for forwarding or delivering mail (and that includes using Email::Filter), you should test with this new trial release right now. It will become a stable release soon. Probably as soon as I'm back from the QA Hackathon next week.

money into code: Perl 5 code bounties (body)

by rjbs, created 2013-02-24 22:44
tagged with: @markup:md journal perl

If there's money earmarked to be spent improving Perl 5, one seemingly obvious thing to do is to try to use it to directly to improve perl. In other words, the mission is to "turn money into code." The most successful expression of this strategy, I think, has been in Nick and Dave's grants. On the other hand, it's an expression that succeeded because of very specific and felicitous circumstances. Dave and Nick were both well-established, trusted participants in perl's development, known as experts and conscientious workers. They were given, by and large, free rein to pick the topics on which they would work. The foundation trusted them to pick things of value, though with a means for TPF to call shenanigans if needed. That trust has been well-placed, in my opinion.

For whom else, though, would the foundation be willing to do this? I began to write "precious few" before deciding that it's probably more likely to say "nobody." Or, rather, nobody who has any chance of applying. What we need is a confirmed expert at working on the core, with a broad knowledge of many of its pieces, including its build options, portability concerns, and so on. Then there would be the question of picking topics for work: they'd have to be things on which the candidate could work, since no one seems quite comfortable with every single piece of the codebase. They'd also have to be things with value, since they'd have to get approval from grant managers, and possibly code reviewers.

I think I speak for most, if not all, of the regular contributors when I say that we'd love to see such a person apply for and receive a grant to work on core improvements. It just doesn't seem likely, in part because it hasn't already happened.

If we can't fund the work of a highly skilled, self-starting factotum, what's next? One common refrain goes something like, "Now that TPF has such-and-such quantity of money, they should use it to get such-and-such feature added." This is something that we haven't really tried doing in recent memory, in part because it's very unclear how we'd do it.

First, we'd need to make a list of things we want done. That part is pretty easy, especially if we don't scruple to list even the very difficult things like "untangle PerlIO" or "replace string exceptions with objects without breaking things." In a sense, we have had this for some time in the perltodo document, but its placement does have a bit of "beware of the leopard" to it.

With a list of tasks that we'd like to see done, the next step would be to post bounties or request bids. That is, either we'd attach a prize to seeing the task completed or we'd let each applicant name his or her price. Either way, we'd end up (assuming any interest) with a list matching up people, tasks, and dollar amounts. Through some selection criteria, applications for grants would be approved and work would begin.

In fact, we already have a mechanism for some of this in the grants committee of the Perl Foundation. Four times a year, they post a public call for grant applications. Lately, there have been precious few applications, though. It's not clear why this is, but one reason that's been cited in the past is that the amount paid for the grants is not sufficient to warrant the time required, at least for many. The maximum value paid for a grant from the committee is $3,000 by their rules (although the most recent call for proposals set a $2,000 cap). This is about a week and a half at $50 per hour, the rate used for Dave Mitchell and Nicholas Clark's grants.

Being able to accept a $3,000 grant and work on it steadily probably means that an applicant must already be unemployed, working as a contractor, or willing and able to take off the stretch of time from one's day job. With those conditions met, the next question is whether any of the problems we'd really like to see sorted out can be done in ten days of work. It seems quite unlikely, from here.

What we see, instead, from most grants is that the work gets performed in off hours: evenings and weekends, while the grantee continues on his or her day job. The grant money isn't necessary income, but a bonus, and is now at odds with the grantee's enjoyment of leisure time — hardly a great motivator.

I think that in order to achieve results on any task that can't already be done in someone's leisure time, any bounty or grant is probably going to have to be for a much greater amount, sufficient to become the grantee's livelihood or chief income for the duration. Too many grants have lingered on for too long, often fizzling out, because they were hobby projects with an optional prize at the end. For the grantee to abandon the grant cost nothing but face.

The next problem is keeping the work moving. This isn't necessarily a question of keeping the grantee on task. After all, we've tried to make it possible for this work to be the primary employment of the grantee, so there should be some built-in incentive. The problem here is removing roadblocks to the work getting done and getting accepted. That means that the programmer needs access to code review to answer the questions, "why doesn't this work?" and "does this changeset look good?"

That last one is particularly important because at some point any work being done will have to land on perl5.git's blead branch, and if it's problematic, it could be very problematic. Several years ago, for example, a grant was paid for to have perl produce an AST. The work was done and the grant paid, but the code was rejected for inclusion in the core for numerous reasons that would not have occurred had there been regular code review.

Code review and expert advice are insufficently available on the perl5-porters mailing list. The reasons for this include, but are not limited to, the strictly low number of experts on the core and the low availability of those experts who exist. Some potential reviewers have gone so far as to say that providing code review can be depressing, because they have sometimes spent a lot of time trying to transfer knowledge to would-be contributors, only to have the contributors later go away, never to contribute again. (This, after both successful and failed attempts to get changes into the core. Difficulty in retaining contributors is problem for another day.)

If availability of code review is critical to making the money-to-code machine work, how do we improve that availability? One option is to pay reviewers, but I think this is no good. It means that potential aid might be withheld because the potential reviewer knows that someone else could be getting paid to do what he or she would be doing for free. In other words: if somebody is supposed to be getting paid to do this, why should I do it for free? Further: if I'm getting paid to perform review for this set of changes, why should I do it for free for that set of changes?

This concern applies, really, to the grant itself. Why should anyone contribute to the core "for fun" when others are doing it "for profit." I don't quite think that this argument is a reason to avoid cash-for-code altogether, but I do think it's a reason to limit grant-funded work to problems that are clearly understood to be very difficult and long-languishing. Dave's work on (??{}) is probably the paragon of this sort of problem.

Of course, fixing (??{}) is a big task, plausibly on the order of "untangle PerlIO" or "replace string exceptions with objects without breaking things." It took time, code review, design review, and time. It was possible, in part, because Dave was on a fairly open-ended grant, and could work on it until it was finished. If coding grants are given based on bids or bounties, and if the grant is meant to be the primary work of the grantee, the time in which they must be concluded is much more strictly bounded.

For me, on the topic of turning money into code, I'm left with a few thoughts:

  1. Grants like Dave's and Nick's amount in many practical ways to simple employment contracts, and work well because of the freedom afforded by that arrangement. Finding more suitable people willing to take on such a commission would be a boon. Of course, this might mean minting (read: mentoring) new experts in the perl internals, and so far, this seems to be the real mystery of the age.

  2. Grants for thankless, short tasks likely to take two weeks or less might be suitable for bounties. One example from the top of my head would be "convert Pod::Usage to Pod::Simple." These would (presumably) require less code review than longer-scale tasks. If the best way to try to turn money into code is to pay for short, awful, thankless tasks that will benefit future development, I think the best thing that can be done is for such tasks to be clearly listed and described somewhere, possibly rt.perl.org, and to point at them from any future call for grant proposals. I think the foundation should probably try to offer $3000 (or more) grants again, to try its best to make grants "real income" rather than a bonus for work in spare time.

  3. Code review needs to become more universally available — whether it's wanted or not, in some cases. It probably can't be brought about with the direct application of money, and I don't see how money can indirectly help just yet.

naming and numbering perl (body)

by rjbs, created 2013-02-19 22:21
tagged with: @markup:md journal perl

Matt S Trout wrote a very reasonable suggestion to brand the current Perl 5 implementation as Pumpkin Perl. The gist is something like, "take the emphasis off of 5, which sounds like one less than 6, and put it on the thing itself: the really nice language, all plump and ready to be used in a pie." I can get behind that.

The part that starts to wrankle me is this:

So what if next year's release, instead of saying

  perl revision 5, version 20

said

  pumpkin perl, version 20

It's not that I think Matt's proposal says we should drop the five as its first, key point. It's a bit more: "Look, everybody knows this is five. Focus on the thing that does keep changing and marking nice improvements!" (I will state for the record that I do not want to remove the five from many places, although moving it around a little might happen.) The problem is the huge influx of expectation that this is really is about dropping the five and using this as some sort of breaking point with history. This frustrates me for a few reasons.

There's an occasionally repeated refrain that "if only we could break backward compatibility," Perl 5 would surge forward with new innovations. "We'd finally throw off the yoke of some feature the speaker doesn't use and be free!" The problem, of course, is that somebody else uses that feature. Pretty often the speaker is his own somebody else, and just doesn't realize it. Prototypes? Test::More. Tied variables? DBI and Config. AUTOLOAD? CPAN and Encode. Typeglobs? Much of the Net namespace (not to mention anything that exports). Other times, the feature is old and goofy, but not really in the way of anything.

So there's one blocker to breaking backward compatibility: you'll make it a nice language in which you'll get to reimplement all the stuff you love about using the language. Whoops!

That's not the most important blocker, either. The more important blocker is that nobody is actually coming forward and saying, "If we can break X, we can get a big improvement to Y." Maybe this is because there is a feeling that backcompat is so deeply entrenched, and so pervasive about the smallest foibles of the language that there is no point. I think this would be a shame, because I can pretty confidently say that we can break backward compatibility for the right win. How much, for what? I don't know. We'd need to see an offer, and then a patch.

Of course, there are limits. Perl is used to power multi-billion-dollar businesses. This constrains its paths forward. It won't cease to exist, nor will it be abandoned, but it can't break the code bases of those businesses. Also, note that I'm speaking in the plural. If there was one massive enterprise that owned Perl and drove it forward, there would be a very clear set of guidelines for what could break: anything but the code making billions of dollars for the sponsors. Instead, there are a bunch of enterprises and upgrading them all in sync and keeping all their code working forever is not a simple matter.

This is the problem with success. As a language grows successful, it loses agility. That's one of Perl 6's strengths: it hasn't yet become a big success, so it can change anything it wants whenever a design flaw is made. If we want to regain that kind of agility, all we have to do is agree to give up our success.

That's what forking a project is about: you get a whole bunch of code (warts and all) for free, without the burden of success. Then again, maybe after a brief and extremely liberating romp through the free prairies of obscurity, you can try to steal the success of your ancestor. Remember: you'll be giving up that agility again.

This is where I get back to liking Pumpkin Perl.

If you want to break backward compatibility, you can sketch out your plan and say, "Hey, I figured out that if we drop support for reset, we can get a 4% speedup to local!" This will result in a response of "no, never!" or "yes, surely!" or "hm, show us a patch?" If it goes in, great. There's a deprecation period to ease everybody off of reset, and then local gets faster. If it doesn't, what do you do?

Either you grin and bear it, or you go work on another Perl. The Perl you work on doesn't care about reset. Heck, it doesn't care about dump, either, so you can save another 2% on something there. You won't be working on Pumpkin Perl, of course, but on Antelope Perl, or Hubbard Perl, or Kurila Perl.

Are these forks viable? Of course. They are viable as long as they have people using them, just like Pumpkin Perl. Perl is free software, so anybody's fork can continue to incorporate changes from the mainline, while it's possible, and changes determined to be massive improvements can be brought back to Pumpkin Perl after a proving period. GitHub showed us all that forking is good, because a fork is just a branch. That works here, too, and naming "the" perl5 is a way of saying, "This is one branch: the most conservative, commonly relied-upon one." The distinction it creates from Perl 6 is, to me, a minor side benefit.

Matt's posts have all been very clearly trying to avoid talking about forking perl and breaking backcompat, so I hope he isn't bothered to see me going directly to those two topics in this post. A lot of other responses went there, though, and I think those topics really need to be addressed.

If "Pumpkin Perl" is going to be a thing, it's going to be a very low-key change: we'll call the thing at perl5.git.perl.org "pumpkin perl," and it will answer to use 5.x.y, and it will still say "revision 5, version X," more or less. The freedom we get is a freedom of expression, granted by having a clearer way to refer to one branch of perl as an equal amongst others. By giving the first fork a name, we make room for future forks to exist and have their own names, without having to "break" this one.

spending somebody else's money… for Perl! (body)

by rjbs, created 2013-02-17 22:21
last modified 2013-02-18 09:54
tagged with: @markup:md journal perl

Over the past few years, the Perl Foundation received a bunch of nice big donations to be used for Perl 5. Some of this money has been used to pay for work by Dave Mitchell and Nicholas Clark to work on difficult problems in the Perl 5 core. This has, in my opinion, been money well spent. Dave and Nick know the Perl core very well, and they've worked seemingly tirelessly to make progress where progress is not easy, and to fix things that nobody else wanted to touch.

There are problems with this kind of spending, though. One of them is that Dave and Nick are human resources, and not permanent assets. We can keep spending money on them for as long as they let us, and it will almost certainly keep being a good investment, but it can't go on forever. Another problem is that the rate at which we can fund Nick and Dave is limited, and we're not going to burn through all the money any time soon doing that.

Do we want to be in a rush to spend all that money? Well, maybe not a big rush, and maybe not all of it, but I think it sends a bad sign to donators when we don't spend the money we're given. Specifically, it sends the sign that we don't have any need for money, because we're not even really using what we have.

Then again, maybe we don't. Maybe the only things we should be spending money on are the YAPCs, legal issues, and some service hosting. There's an argument to be made for that, too. It's been said that when TPF spends money on some coding, it indicates that there are multiple strata of people in the Perl community: those whose work is blessed by "the powers that be" and those whose work isn't. Does this create a real disincentive for "outsiders" to contribute?

It's a big complicated question, all of which boils down to something like, "What ought TPF to do?" Maybe the answer is, "just what it's doing now." If that's the case, though, I want to feel convinced of it. Right now, I'm not.

I think I'm going to write down a bunch of ideas for how TPF could spend money other than conferences and paying for Dave and Nick. Implicit or explicit in these ideas will be my internal list of problems that seem worth solving but without obvious solutions that can be carried out with just some free time and good intentions.

The Great Infocom Replay: Deadline (body)

by rjbs, created 2013-02-16 17:24
last modified 2013-09-15 07:17

Last weekend, I went to Las Vegas for a one-day trip to attend a party in honor of my father's many years coaching rugby. It was a great event, but the travel was, as usual, not a big bowl of cherries. I decided to make the most of it, though, and play Deadline on the plane.

I loaded the story file into Frotz on my iPad and put the manual into GoodReader. Using my iPad to play was a great plan, once I also started using my wireless keyboard (presumably in violation of some sort of regulation). Using my laptop would've been impossible in that cramped seat, but the iPad worked a charm, apart from occasional glitches in Frotz.

I was looking forward to Deadline, because it was the first Infocom game I'd be "replaying" that I hadn't actually played before. I had probably started it up and poked around for a few minutes, but I hadn't played the game in earnest. This was my chance to pretend that I was living in 1982, playing a brand new Infocom game ... in flight, on a touchscreen tablet. Well, the illusion of novelty wasn't the important thing.

Spoilers follow.

I enjoy a good murder mystery, and I decided I'd really try to solve this one. The Deadline feelies include a number of witness interviews. From those, I produced a timeline of events, noting what was probably fact and what was only testimony. Arriving at the scene of the crime, I made a survey of the grounds (one again cursing the asymmetry of the map exits) and figured out which windows led to which part of the estate. This would prove useful later.

I intercepted mail, rifled through medicine cabinets, conducted thorough interviews, snooped on phone calls, and tried to figure out where characters were sneaking off to. I felt like a real detective… almost. There were problems. For one, it was clear that I'd have to follow some characters around to see what they were doing, but it wasn't clear whom to follow. It meant that I had to do that most boring of adventure game chores: start over, over and over and over.

I didn't mind all that much, actually, because in many ways it reminded me of Suspended, a game I love. As I played, I expanded my timeline of events to include events that would happen during the day. Nine fifteen, a phone call. Ten o'clock, the mail arrives. Eleven thirty, Angus gets angry about holes in his garden. Noon, a reading of the will. As I built up more of this timeline, by playing and failing over and over, I began to find the critical path to being everywhere I needed to me. Every time I played, I could do every required action to learn every important fact. This was satisfying.

Learning these facts, on the other hand, was often very dissatisfying. I was often correct in my assumptions about what was going on. I often knew what I had to do. I just didn't know how to do it. It was a "guess the verb" puzzle on a very frustrating level.

dig in dirt
Although everything is coming up roses, you haven't found anything unusual except for a few pieces of a hard substance which fall back to the ground.

look in holes
There are two holes here, each about two inches by four inches. They are at least three inches deep and the soil is compacted around them.

look near holes
Ouch! You cut your finger on a sharp edge as you dig. You search carefully in the dirt, now that you are sure something is there, and pull up a piece of porcelain, covered with dirt and dried mud.

There were a number of drugs around the house, and many of them were labeled with warnings about drug interactions. They also had the names of the dispensing pharmacy. I couldn't call the pharmacist to ask, "How would Allergone interact with Ebullion?" I couldn't even ask the coroner to check for these drugs in the victim's system:

analyze Mr. Robner for loblo

Duffy appears in an instant. "Well, I might be able to analyze the Mr. Robner, but you don't even have it with you!" With that, he discreetly leaves.

Well, Duffy, ask at the morgue!

Later, I'd find traces of a drug on a piece of ceramic. Even if I'd already had that drug in particular analyzed, the lab would just say "well, it's not a common medication, anyway."

Later, I would resort to InvisiClues. This was almost as frustrating as trying to pieces things together myself, at least when it came to figuring out why on earth George would stand around outside doing nothing for an hour. The clue urged me to do something, but I couldn't guess the verb.

In the end, I played the game until I knew who had killed Mr. Robner, and why, and how. It's possible I could have solved the game, had I stuck with it for another few hours. I realize that a single five hour flight is not how one might usually expect to play Deadline. Then again, it did get five hours of my time, which seemed like quite a lot. More importantly, giving it more time wasn't going to lessen my frustration, but only increase it.

I think that an IF game could be a very good way to present a murder mystery, and I hope to find another one that I love. This one, though, is not one I'll recommend to friends. Maybe Witness...

the great new Email::Sender (body)

by rjbs, created 2013-02-07 09:33
last modified 2013-02-07 11:56
Yesterday I released Email: :Sender 1.300003. This is a pretty important release!

First, it is the first production release of Email::Sender to use Moo instead of Moose. This doesn't affect my code much, because I use Moose all over the place already. On the other hand, your code might speed right up. On my laptop, for example, the test suite now runs in 20% the time it used to. I'm hoping this helps people feel freer to move to Email::Sender, which really does make life easier for writing email-related code than previous email sending libraries did.

So, I'm delighted to have the work of Justin Hunter and Christian Walde in place, making the Moo-ification possible, not to mention that of Dagfinn Ilmari Mannsåker, Frew Schmidt, and Matt S Trout on porting Throwable to Moo, which was essential to making the rest of the conversion possible. Thanks!

Like I said, though, the Moo-ification doesn't change most of my code very much (yet?), but I'm still very excited. Why is that?

Right now, I've got a large stack of "technical debt payment" tasks scheduled at work. Many of these are quite old, and many are in the form "we switched 90% of our code to some new system, but 10% remains on the old system; convert the final 10% so we can reduce our total code complexity." We talk about this at the office as "killing snowflakes." Look, that subsystem is unique and special and not like anything else! It is a delicate, beautiful snowflake! Kill it!

Amongst those snowflakes being killed is our use of Email::Send. (Note the lack of the -er. This is the precursor to Email::Sender.) We don't actually use Email::Send, exactly, because it's so broken. Instead, we use an internal fork, the design of which strongly influenced the eventual creation of Email::Sender. It really had to go!

Unfortunately, it wasn't quite possible yet.

One of the great strengths of Email::Sender::Simple is that you can redirect all mail by setting an environment variable. This is very useful in tests, where you can say something like:

$ENV{EMAIL_SENDER_TRANSPORT} = 'Test';

maybe_send_some_mail;

my @deliveries = Email::Sender::Simple->default_transport->deliveries;

...and then inspect what mail would have been sent. Of course, doing this via an environment variable within one process isn't that compelling. You could just assign a global. On the other hand, this is a big deal:

$ENV{EMAIL_SENDER_TRANSPORT} = 'SQLite';

fork_and_maybe_send_mail_in_any_process;
wait;

my @deliveries = deliveries_from_db('email.db');

This is useful for testing something like an exploder that forks to do its work. The next step is testing how code behaves when the email transport fails. This has always been possible with the Failable transport, which wraps another transport and forces failures however the programmer likes. Unfortunately, it works via code references, which can't be easily passed in the environment. What's worse is that it turned out that no configuration could be passed to a wrapped transport via the environment. Oops!

This has been fixed! So, imagine this extremely simple (but quite useful) wrapper:

package Test::FailMailer;
use Moo;
extends 'Email::Sender::Transport::Wrapper';

use MooX::Types::MooseLike::Base qw(Int);

has fail_every    => (is => 'ro', isa => Int, required => 1);
has current_count => (is => 'rw', isa => Int, default  => sub { 0 });

around send_email => sub {
  my ($orig, $self, $email, $env, @rest) = @_;

  my $count = $self->current_count + 1;
  $self->current_count($count);

  my $f = $self->fail_every;
  if ($f and $count % $f == 0) {
    Email::Sender::Failure->throw("programmed to fail every $f message(s)");
  }

  return $self->$orig($email, $env, @rest);
};

This makes it easy to say "every third message fails." Then, to make your test configure it for spawned processes:

$ENV{EMAIL_SENDER_TRANSPORT} = 'Test::FailMailer';
$ENV{EMAIL_SENDER_TRANSPORT_fail_every}      = '3';
$ENV{EMAIL_SENDER_TRANSPORT_transport_class} = 'SQLite';
$ENV{EMAIL_SENDER_TRANSPORT_transport_arg_db_file} = 'failtest.db';

fork_and_maybe_send_mail_in_any_process;
wait;

my @deliveries = deliveries_from_db('failtest.db');

Done!

Using these tools together (in their internal-Email::Send-like version) was instrumental to allowing us to confidently refactor our code, because we could test it in ways we never could before. Now that everything has been moved to one email library, it's even easier to rely on these testing systems. I'm looking forward to improving them even more.

The Great Infocom Replay: Zork Ⅱ (body)

by rjbs, created 2013-02-02 22:15
last modified 2013-09-15 07:17

In my memory, before I "came back" to adventure games in the mid-nineties, Zork Ⅰ was an important classic and Zork Ⅲ was the serious, thoughtful capstone to the trilogy. Zork Ⅱ was, in my mind, an afterthought. It was something I had to get through before I could get to the trilogy's endgame.

Having played through Zork Ⅰ and Ⅱ in the last week, I can say that my childhood view was dumb.

The first thing I found was that the map made much more sense. I could keep the whole thing, more or less, in my head. Only a bit of the northern ledge confused me reliably. The Carousel Room made for a really useful hub, and helped me keep things divided into memorable segments. I found its puzzles much clearer, except for the widely hated two: the Bank of Zork and the Oddly-Angled Room. Even the Bank of Zork didn't bother me so much, except for the really lousy description of the Small Room.

The Wizard was much less annoying than the thief, but he still irritated me. Actually, it wasn't that he irritated me, it was that his spells were more trouble than I felt they should've been. More than once, I thought I'd made it through the duration of a spell, only to trip and fall, fatally. Oops? Still, he didn't scatter the contents of the dungeon far and wide, and that's worth something. I liked that the wizard had more personality than the thief, and the eventual interaction with the demon, who was also fun. It made the whole game feel a bit more story-oriented than Zork Ⅰ.

I enjoyed the robot, although every part of the robot puzzle was of the "you will have to die to figure this out" variety. The button puzzle, like the button puzzle in Zork Ⅰ, was tedious, and to be solved much in the same way as a maze. Its pay-off was much more enjoyable, though. I felt cleverer, and making the Carousel Room behave normally was a real win.

One thing I never understood in Zork Ⅱ: what's up with the underground volcano? What does that look like? It's still pretty gonzo, and my brain rejects it. I know this is my problem and not the game's, and it's silly since I enjoy old-school D&D, which is pretty rife with gonzo nonsense, but there it is.

Finally, I think the princess is a pretty underrated character in this game. Who is she? How long has she been stuck in that dragon's lair? Where does she go? Who are her parents? Why does the Wizard of Frobozz care about her?

I hope she gets her own game, someday.

The Great Infocom Replay: Zork Ⅰ (body)

by rjbs, created 2013-02-01 22:23
last modified 2013-09-15 07:17

Zork Ⅰ is a really important game to work through, if you're going to try to understand where interactive fiction came from. It's not the first, but it's a major early milestone of the golden age of commercial IF, and its book is alluded to repeatedly throughout later works. I'm really glad that I've played Zork Ⅰ, but my feeling after playing it again is that once was probably enough.

In fact, I didn't really replay the entire game. I remember it fairly well, despite having played it over twenty-five years ago. I played as much of it as I could from memory, plus the puzzles I could solve again, and I skipped the parts that I knew I would find painful. I'll list them as I go.

I'd forgotten, before this replay, that so many of the early games were very, very sparse of text. Zork is actually quite wordy compared to some, but it's still quite minimal. One of its most memorable locations, Flood Control Dam #3, is described like this:

Dam
You are standing on the top of the Flood Control Dam #3, which was quite a tourist attraction in times far distant. There are paths to the north, south, and west, and a scramble down.
The sluice gates on the dam are closed. Behind the dam, there can be seen a wide reservoir. Water is pouring over the top of the now abandoned dam.
There is a control panel here, on which a large metal bolt is mounted.
Directly above the bolt is a small green plastic bubble.

This led me to go play the first few rooms of one or two later games, as well as some popular amateur games, and decided that I have been failing to appreciate the economy of prose in many of these games. I also went to look at some of my never-completed projects and was not surprised to decide that I probably had too much text. (This realization, I must admit, came after feeling some surprise at Jon Ingold's remark in "Thinking Into the Box" that "players are not often in it for the prose.")

I was amazed by how much of the map was still in my memory. I didn't remember how the zones fit together, but I remembered them. I remembered how to solve the most hateful puzzles (like the awful Loud Room) and most importantly, I remembered many secret passages and exits to the surface. I knew to make my first task: get to the temple, get the torch, and pray my way out.

I managed to collect quite a few treasures, but I knew I didn't want to shoot for everything. My secret internal goal was to get the thief to open the egg, and then to retrieve it. This meant getting enough other treasure to have a chance at defeating the thief. I managed to do it all in one night, skipping only two things: the maze and the river. Skipping the maze was a no-brainer. I remember mapping the maze the first time. I realized how to do it, I felt like a genius, and then I never, ever, ever wanted to do it again. I also knew that I wouldn't need to go in there to accomplish my goals. I do not regret skipping the maze.

As for the river, I knew exactly what I would have to do, and executing all the steps just didn't seem like it would be worth it.

Finally, I also didn't drain the reservoir. The "trial and error save, push button, restore, repeat" puzzle drains my will to play faster than it would drain the reservoir. I used the magic word to get the platinum bar out of the Loud Room. As a side note, I think that is my least favorite non-maze puzzle in the game. My favorite is probably the coal mine, except for the part where it's also a maze.

I enjoyed making a map of the game, as I thought I would, but I'd forgotten how often rooms were not joined symmetrically. That is, very often you'd leave a room by walking east, but to get back, you'd walk south. The official maps make it very clear why the exits looked that way, but it made everything much more tedious. Just to draw a new room, I'd have to walk through many of its exits, figuring out the angle of the passage. That would waste battery life on the lamp. I ended up pursuing a strategy that felt like it hurt my enjoyment of the game. I worked on each zone's map and puzzles without concern for treasure or time. Once I had a plan of attack, I restarted the game and went through my checklist.

Even this would not have been much of a pain, if not for random aspects of the game. At one point, when things were going quite well, the thief wandered by and stole my torch. Oops. Fighting the thief also led to some "restore and try again" moments. I don't mind randomness that provides atmosphere, but randomness that can ruin my game tends to, well, ruin my game.

Finally, I had a hard time forming a mental picture of the dungeon. Once again, the official map helps, but it doesn't help enough for me. The problem here is mostly with me. Zork is a gonzo setting, where anything that will be neat is allowed to exist. I kept trying to figure out how it fit together, and why there's an Egyptian temple bordering a huge dam and a tiny coal mine. It was mostly a collection of puzzles, which is what I knew to expect.

In the end, I'm glad I replayed Zork Ⅰ, but my replay was really tainted by the fact that I had so much of the game still in my memory. I blew through many puzzles that I might have enjoyed, and there were puzzles that I might have enjoyed if I wasn't prejudiced by my previous play. I'm hoping I get more out of later games.

The Great Infocom Replay: Foreword (body)

by rjbs, created 2013-02-01 09:48
last modified 2013-09-15 07:17

Quite a while ago, I decided that I had too many petty interests, and that I should pick one and pursue it. I thought I'd work on running some really good D&D, but basically it hasn't worked out. I have been unable to establish a regular-enough group of players, and have not felt entirely compatible with some of the players who I was able to attract. It has been a big let down.

Something recently reminded me of Suspended, and I thought I'd give it another play. I found that I could still beat it from memory, which made me happy, and I played around with the custom and advanced scenarios, too. I also finally learned how to get a good score. (I don't think I ever got a better score than 7 in the past.) Then I started looking at my big pile of IF games to play, and my big pile of IF games to write. (I count seven abandoned projects in my code tree, each with a CVS directory in it.)

So, I'm puttering about again and trying to determine whether I can get motivated enough to do any real work on this. I finally wrote ZMachine::ZSCII, getting me one more letter in The CPAN Alphabet Game, and I've been poking at hand-assembling Z-Machine programs. I want to get a better handle on how to construct IF, though, and part of that means playing more. (I'm also really enjoying the IF Theory Reader.)

I've gone back and played a bunch of the games I liked, and I want to play a bunch of the games I never did. I gave some thought to trying to organize an interactive fiction "book club," but I decided it would probably not pan out. (If you think I am wrong, you can make a comment or something. I am not made of stone.)

Included in the games I want to play (or play again): the Infocom canon. I've only played about a third of it, and some of it I barely remember. I've decided to try playing all of it. To avoid any indecision, I'm going to do them in order. I haven't decided how to limit my play time, but I think it will be something like "at least three sessions, unless I finish the game earlier than that."

Playlist:

  • Zork I ✓
  • Zork II ✓
  • Deadline
  • Zork III ✓
  • Starcross
  • Suspended ✓
  • The Witness
  • Planetfall ✓
  • Enchanter
  • Infidel
  • Sorcerer
  • Seastalker
  • Cutthroats
  • HHGG ✓
  • Suspect
  • Wishbringer
  • A Mind Forever Voyaging ✓
  • Spellbreaker
  • Ballyhoo
  • Trinity
  • Leather Goddesses of Phobos
  • Moonmist
  • Hollywood Hijinx
  • Bureaucracy ✓
  • Stationfall
  • The Lurking Horror ✓
  • Nord and Bert Couldn't Make Head or Tail of It ✓
  • Plundered Hearts
  • Beyond Zork ✓
  • Sherlock

Not found in my collection, although I'll see if I can get them:

  • Zork Zero
  • Shōgun
  • Journey
  • Arthur

Checkmarks, above, are games I've played in the past, although I haven't completed all of them.

Wish me luck!

prev page
next page
page 3 of 82
2039 entries, 25 per page