December 3, 2011

Announcing Augeas 0.10.0

Augeas 0.10.0 has just been released, and it brings a good lot of changes with it.

Among them, the aug_to_xml method has been added. This is essentially an integration of the XML export function initially introduced in the Config::Augeas::Exporter Perl module. The port in C has seen some improvements in the XML schema, as well as in speed. There is not yet an aug_from_xml method, and it might take some time to come, as it brings up a lot of merging issues. This change introduces a dependency on libxml2.

Dominic Cleal has been working on the new aug_srun method, which he wishes for his  Augeas module for Puppet. In order to achieve this, he introduced a way to set the context of XPath expressions in /augeas/context in order to use relative paths. This is a well-known feature for the users of the Augeas module for Puppet, and it is now available for all Augeas users.

A lot of lenses have been fixed and improved. As a matter of fact, when I wrote the Config::Augeas::Validator Perl module, the goal was initially to test our configuration files against the Augeas lenses. But after testing some 120.000 files, I've obviously found a few flaws in the lenses and had to fix them.

The package for Augeas 0.10.0 has entered Ubuntu Precise this afternoon. Versions for older releases are available on my Augeas PPA. Feedback is most welcome!

Unit testing your configuration files

At work, we have a lot of different services running and we end up with hundreds of thousands of configuration files in our configuration repository (especially since we don't use templates - more on that in another upcoming post...).

In order to ensure the quality of these files (avoiding syntax errors and insecure configurations alike), we had the idea of writing a system to unit test the configuration files. Being involved in Augeas, I thought it would make a great parsing backend to write the tests.

This gave birth to what is currently a Perl module called Config::Augeas::Validator, which comes with a Perl script and an SVN wrapper (this is what we used, but wrappers for other VCS are welcome) to plug it to pre- and post-commit hooks.

The module relies on Augeas (and its lenses) to parse the configuration files. On top of that, you need to specify rules (which are essentially unit test scenarii), at least one for each type of file you wish to test. A minimal rule might look like this:

[DEFAULT]
lens=Fstab
pattern=.*/fstab


This rule will tell augeas-validator to test all files whose full path matches .*/fstab against the Fstab.lns Augeas lens. If the file contains syntax errors (that is, the lens fails to parse it), the program will report it and exit with a status of 1.

Here is another simple example adding two unit tests with warnings:

[DEFAULT]
lens=PHP
pattern=.*/(php\.ini|php[45]/conf\.d/.*\.ini)

[file_uploads]
name=file_uploads
explanation=file_uploads should be set to 'Off'
type=count
expr=$file//file_uploads[. != 'Off']
value=0
level=warning
tags=security

[expose_php]
name=expose_php
explanation=expose_php should be set to 'Off'
type=count
expr=$file//expose_php[. != 'Off']
value=0
level=warning
tags=security


This test checks for php.ini files. Not only does it fail with status 1 if there is a syntax error, it also applies two unit tests called file_uploads and expose_php, which will make the program output a warning and exit with status 2 if they are not met. The essential part of the rules is the expr parameter, which is an Augeas XPath expression. In this case, the expressions must not match any nodes in order to pass the tests (hence value=0).

These are just simple examples of what this module can do. You can find more examples shipped with the module itself. As a last example, here is one for Apache configuration files (in Debian/Ubuntu):

[DEFAULT]
lens=Httpd
pattern=.*/(sites-available/.*)|(apache2/.*\.conf)

[one_servername]
name=One ServerName per VirtualHost *
explanation=There should be only one ServerName per VirtualHost * entry
type=count
expr=$file[label() != "default"]/VirtualHost[arg =~ regexp("\*(:80)?")][count(directive[. = "ServerName"]) != 1]
value=0
level=warning

[bufferedlogs]
name=BufferedLogs
explanation=BufferedLogs must be set to Off
type=count
expr=$file//directive[. = "BufferedLogs"][arg = "On"]
value=0
level=warning
tags=security

[timeout]
name=Timeout
explanation=Timeout must be at least 45
type=count
expr=$file//directive[. = "Timeout"][int(arg) < 45]
value=0
level=warning
tags=security

September 6, 2011

Bible verses in Mint fortunes

For my most recent computer, I've installed Linux Mint 11. In Mint, bash is set to display a fortune cookie every time you start a shell session.


This is a nice idea, however I would rather see daily Bible verses than random fortunes. Here is how to do that. First, install the verse package:


$ sudo apt-get install verse


Then, edit /usr/bin/mint-fortune with sudo, and replace the line:


/usr/games/fortune | $command -f $cow


with


/usr/bin/verse | sed -e 's@^ *@@' | $command -f $cow


To be sure that the file won't be overridden when you upgrade your distribution, set it as a local diversion:


$ sudo dpkg-divert --local /usr/bin/mint-fortune


and you're done!

August 26, 2011

Colored initials in LaTeX

Georg Duffner, the creator of the EB Garamond open-source font, has been working on an initials font for EB Garamond based on a 16th century French Bible lately. A few days ago, he was thinking about producing colored initials with it, and had the idea of splitting the font in two: one font for the background ornament, and one font for the foreground letter to superimpose on it.

Base on this idea, I have hacked a little LaTeX module to typeset the initials in a simple way. The module can be found on github and is based on the lettrine LaTeX module.

It makes use of fontspec to load the fonts so it only works with XeTeX and LuaTeX.

The alphabet only contains 3 letters for now, so not really much can be achieved with it so far, but the code is there.

Here is an example reproducing parts of the 16th century Bible used to get the samples of the initials:


May 11, 2011

Keeping the Law — for whose sake?

Ye hypocrites, well did Esaias prophesy of you, saying,

This people draweth nigh unto me with their mouth, and honoureth me with their lips; but their heart is far from me.

Matthew 15:7—8


Many times, Jesus called people — especially the Pharisees — hypocrites. This puzzled His disciples quite a bit, because the Pharisees where people who kept the Law strictly. How could people who kept the Law be wrong? Well, the problem is that they kept the Law outwardly but not inwardly.


I heard many comments on the subject over the years, but something about this struck me today. Who were the Pharisees keeping the Law for? For whose sake do we act the way we do? Jesus said that when you desire a woman, you've already committed adultery with her in your heart, and that when you insult someone — even in your heart — you've already committed murder.


What is it that keeps me from acting the thoughts I have? Why do I sometimes let myself desire a woman, but I won't commit adultery in real life? Let's face it: even a lot of non believers have never committed adultery, let alone murder. I think the problem is here: for whose sake do you act the way you do? For God's sake, or for your own?


Many people act properly for their own sake ; they are afraid of what people will think about them, or do to them, if they don't act nicely. They keep the Law for their own sake — because of fear, or because of pride — and they have no reason to keep their thoughts as clean as their acts.


Jesus called us to go much further than that. He said we should actually keep even our thoughts clean. Why? For God's sake! It's not so much that I should look like a perfect person, that I should care about what people think about me — if they will judge me, hate me, stone me — if I don't act properly. No, there's someone — God — who knows more about myself than I even do, and for His sake, I should be perfect, because He said : "Be ye therefore perfect, even as your Father which is in heaven is perfect" (Matthew 5:48).


As Christians, we so often tend to think that we can do it the easy way: starting by cleaning ourselves outwardly, and finishing with the inside. It doesn't work that way, because the goal is not the same. If you care to clean the outside first, you're doing it for your own sake, but cleaning the inside requires doing it for God's sake, for the inward cleaning is the work of the Holy Spirit in our lives.


Your motivation defines where you're heading. If you act the way you do for your own sake, you will stay stuck with the outward issues of your life. If you want to give it all to God and change for His sake, asking the Holy Spirit to cleanse you from the inside — even if it might take longer and be more painful — He will do this work in you, and it will end up showing up outwardly in your life.



Father Almighty,

I realize my own efforts to purify myself are vain because they are only motivated by fear and pride, and they do not cleanse me internally — for Your sake.

I pray Lord that you send your Holy Spirit in my life to achieve the work within me, that I may be perfected to your image inwardly, and shine of your glory outwardly.

In Jesus' name. Amen.

April 14, 2011

The fancytabs LaTeX package

Lately, I've been using LaTeX quite a bit, and defining my own macros. Today is the occasion for me to release my first package: fancytabs.

This is a package that allows to add fancy tabs on the border of pages in a document.

The package can be downloaded from the CTAN.


Installing


After downloading and unzipping the file, run latex on the fancytabs.ins file. This will produce a fancytabs.sty file which provides the package. You can also build the PDF documentation by running pdflatex on the fancytabs.dtx file.


What does it look like?


Here is an example of a page featuring a fancy tab:

March 29, 2011

Syntax highlighting in LaTeX

Lately, I have been busy making a book on Augeas. I wanted something that would look nice, and that would use only LaTeX.

A colleague of mine told me to have a look at the Pygments syntax highlighter and its LaTeX module minted. I'm very happy with them so far.

Installing Pygments


Installing Pygments on Ubuntu is as easy as can be:

$ sudo apt-get install python-pygments

This provides a /usr/bin/pygmentize command, which is then used by minted.


Installing minted


I didn't find a package for minted, so I downloaded the files from the CTAN and made a minted directory in my source.

When building with pdflatex, I make sure to export the TEXINPUTS=minted: variable so that pdflatex uses the minted directory for inclusions.


Adding syntax highlighting


Now that all the tools are installed, we need to actually use them in the LaTeX source. Here is a simple example:

\mint{console}|$ sudo apt-get install augeas-tools|


A multi-line example:

\begin{minted}{console}
$ sudo yum install readline-devel
$ sudo apt-get install libreadline-dev
\end{minted}


And even a listing example:

\begin{listing}
  \begin{minted}{augtool-shell}
/files/etc/foo.conf
/files/etc/foo.conf/#include = /etc/foo.d/*
  \end{minted}
  \caption{Augeas does not interpret include statements}
  \label{lst:intro_include_tree}
\end{listing}


There is much that can be done, and the documentation is well made.


I'm missing lexers


I would love my book to be full of colours. Not only for languages that are well known, but also, and especially, for the languages and commands that are specific to Augeas.

Of course, Pygments doesn't provide lexers for Augeas syntax (yet), so I need to write them.

The first lexer I would like to have is the augtool-shell lexer, which will put syntax highlighting on commands from an augtool session.

My lexer looks like this:

import re

try:
    set
except NameError:
    from sets import Set as set
from bisect import bisect

from pygments.lexer import Lexer, LexerContext, RegexLexer, ExtendedRegexLexer, \
     bygroups, include, using, this, do_insertions
from pygments.token import Punctuation, Text, Comment, Keyword, Name, String, \
     Generic, Operator, Number, Whitespace, Literal
from pygments.util import get_bool_opt
from pygments.lexers.other import BashLexer

__all__ = ['AugtoolShellLexer']


class AugtoolShellLexer(RegexLexer):
    """
    Lexer for Augtool shell sessions.
    """

    name = 'AugtoolShell'
    aliases = ['augtool-shell']
    filenames = ['*.augtoolshell']
    mimetypes = ['text/x-augtool-shell']

    tokens = {
        'root': [
            (r'^\s+$', Text),           # empty line
            (r'^[;#].*?$', Comment),    # comment
            (r'^(rm\s+:.*)', Text),     # removed nodes
            (r'^(Saved.*)', Text),      # saved
            (r'^(augtool\>)(\s+)(\S+)(?:(\s+)(.*))?$',   # augtool prompt
             bygroups(Generic.Prompt, Whitespace, Keyword, Whitespace, String)),
            (r'^([^=]+)(?:(\s+)(=)(\s+)(.*))?$',    # ls/get/print
             bygroups(String, Whitespace, Operator, Whitespace, String)),
            (r'^(\S+)(\s+)(label)(=)(\S+)(\s+)(value)(=)(\S+)(\s+)(span)(=)(\S+)$',  # span output
             bygroups(String, Whitespace, Keyword, Operator, String, Whitespace,
                                    Keyword, Operator, String, Whitespace,
                                    Keyword, Operator, String)),
        ]
    }


It is located in the augeas-lexer/augeaslexer/augeaslexer.py file, and the augeas-lexer directory contains the following files:

augeas-lexer/
├── augeaslexer
│   ├── augeaslexer.py
│   └── __init__.py
└── setup.py

__init__.py is an empty file, and setup.py contains the following:

"""
Augeas syntax highlighting for Pygments.
"""

from setuptools import setup

entry_points = """
[pygments.lexers]
augtool-shell = augeaslexer.augeaslexer:AugtoolShellLexer
"""

setup(
    name         = 'augeaslexer',
    version      = '0.1',
    description  = __doc__,
    author       = "Raphael Pinson",
    packages     = ['augeaslexer'],
    entry_points = entry_points
)


In order to build with this new lexer, I need to install it in my system (if someone knows how to do without this step, I'll be very happy to hear it!). The following command does that:

$ sudo python setup.py install

Now when I run pygmentize, I should see my lexer:

$ pygmentize -L | grep augtool
* augtool-shell:
    AugtoolShell (filenames *.augtoolshell)


I've actually added (or begun to add) 3 more lexers to my module: AugtoolLexer, AugeasLexer and PuppetAugeasLexer.

The result can be see in the PDF version of the book (which is still a work in progress).


Defining macros


One thing LaTeX users love is its flexibility. On big projects such as books, it is often useful to define macros and new environments.

Here is an example of a listing containing several languages with line numbering. The macros take care of applying the line numbering options (linenos and firstnumber) and defining the standard options for my document (fontsize=\footnotesize and bgcolor=bg).

The macros look like this:
% minted commands

\newcommand{\mymint}[3][]{\mint[fontsize=\footnotesize,bgcolor=bg,#1]{#2}#3}
\newcommand{\consolecode}[2][]{\mymint[#1]{console}#2}
\newcommand{\augtoolshcode}[2][]{\mymint[#1]{augtool-shell}#2}

% inputminted

\newcommand{\myinputminted}[3][]{\inputminted[fontsize=\footnotesize,bgcolor=bg,#1]{#2}{../listings/#3}}

And the listing:
\begin{listing}
  \consolecode[linenos]|$ augtool --backup --root myroot|
  \myinputminted[linenos,firstnumber=2]{augtool-shell}{rm_fstab_opt.augtoolshell}
  \consolecode[linenos,firstnumber=15]|$ diff -u myroot/etc/fstab myroot/etc/fstab.augsave|
  \myinputminted[linenos,firstnumber=16]{diff}{fstab_opt.diff}
  \caption{Removing an option in fstab}
  \label{lst:rm_fstab_opt}
\end{listing}

This produces the following output: