rjbs forgot what he was saying

not logged in (root) | by date | tagcloud | help | login

what is pod weaver? (pt. 1: secret origins)

by rjbs, created 2009-10-30 00:54
last modified 2012-12-21 16:46
tagged with: @markup:md journal perl pod programming

One or two people who write Pod regularly said, "Yeah, I saw you blogging about that Pod thing. I had no idea what you were talking about." A few other people said, "neat, but how do I use it?" Its documentation is getting better, but here's a crash course in its history. Tomorrow I'll write about its application (and maybe later I can turn this into some real docs).

Pod is Perl's in-source documentation system. It has a very simple syntax, muddy simple-but-weird semantics, and a lot of conventions as to how Perl module documentation is written for distribution through the CPAN. Its syntax is usually tolerable, but some constructs are way too verbose. Its conventions can be tiresome because they involve lots of extra typing all over the place.

I wrote Dist::Zilla as a way to greatly reduce the amount of typing and boilerplate needed to build CPAN distributions, and part of what I wanted to do was eliminate boilerplate documentation. My first passes through this was mostly regex-based. It pulled all the Pod out of the Perl source with PPI and then played around with them as strings. It did a pretty reasonable job on a lot of things, but adding more stuff to it was nightmare because it really just was a pile of code.

I knew I needed a real Pod parser, so I looked at Pod::Simple and Pod::Parser but both were way more than I needed. They were hard (for me) to understand and they seemed to really focus on the semantics rather than the syntax. For example, I needed to think about how to do things like add the notion of =method to the parser rather than just get a "some kinda command." So I decided that it would be really easy to make an event-driven parser for Pod, especially if I stuck to the linewise layer (rather than the inline layer where things like B<bold> live). I pitched my idea to Dieter, who told me I was not crazy, and that led to Pod::Eventual. It only understood two kinds of events: command and everything else. Now there are four: command, text, non-Pod, and blank line. The only command it actually understands beyond "it's a command" is =cut, which is effectively the only Pod command that has to be understood differently no matter what.

The first thing I built was something to just give me a reified event stream. With that, I could muck with the document more easily than strings, but I knew I'd want structure. That is, when I moved the thing containing =head1 METHODS, a top-level header, I wanted to move all the plain text it included as well as all the lower-level headers.

The problem is that Pod doesn't really have much in the way of native structure. I agonized over exactly what nested when, delaying progress for a long time. Finally, I decided I wanted something like a document tree that I could put together based on whatever fancy struck me, rather than something where the nesting of elements was dictated by the parser.

This led me to the kind of design Pod::Elemental has now:

...and now you have an array of paragraphs, probably collected in a Document object. You can already turn them back into a Pod string, but probably you want to apply just a little amount of standard meaning: you want =begin regions marked off and you want verbatim paragraphs marked as such, so:

The Pod5-transformed document tree is much easier to play around with because it eliminates the need to track blank lines and estabishes structure by marking off begin/end regions. With that done, other transformations are really easy to apply, like:

Once you've done all your transformation, you can just turn things into a Pod string again.

The next step in the toolchain is Pod::Weaver. It's a system that lets you describe all the transformations you want to perform on an input document as well as what kind of output you want to guarantee in an output document. For example you can say "the input document will have its =method commands gathered up and put under a =head1 METHODS command." Then you can say "at this point in the output document, we will put a =head1 METHODS section, if we found it in the input."

What I was really pleased to discover was that once Pod::Elemental was all working, Pod::Weaver was almost trivial. It helped that I had a lot of code I could re-use from other places to build its plugin and configuration system. It helped that I'd spent a lot of time thinking about the way sections would work. Even so, there's just so little to it. All most section generating plugins have to do is create a Pod::Elemental::Element and push it onto the output document's contents.

Now that it's so easy to write new section generators and add them to my templates, the future of Pod::Weaver is probably all in writing plugins. I really want to write one to generate a "thanks" section for sponsors and contributors. I'm hoping to get code that will generate documentation by inspecting the Class::MOP and Moose meta-object systems. I want to write a few more special commands like =list and I want to write a few more dialects so I can =begin Markdown and maybe =begin KindaPod6.

So, tomorrow I'll show how you can use and configure Pod::Weaver either using Dist::Zilla or by writing your own document-munging tool. For now, it is way, way past my bedtime.