Oct 24, 2015 Tag: T3Docs

A New Task For An Old Server

Server srv123 For Insiders

Work in progress - not finished yet!

Once there was a server srv123. This post is for insiders and describes what needs to be done to get this rather old server ready for a new task in October 2015: Switch to the new Sphinx theme t3SphinxThemeRtd for TYPO3 documentation.

Updated on Nov 01, 2015

Navigate this page:

Summary For The User

As a normal user stop reading here. The following is for insiders!

Summary For Existing Documentation

What needs to be done?

Here we address authors and owners of existing documentation in reStructuredText format. According to our conventions it should look like this:

PROJECT/                            <- your project files
PROJECT/Documentation/              <- subfolder
PROJECT/Documentation/Index.rst     <- master document
PROJECT/Documentation/Settings.yml  <- metadata and build information in a Yaml file

New, needed now:

PROJECT/Documentation/Settings.cfg  <- add this settings file in INI-style

Simply ADD this file. Later the old Settings.yml file will not be needed At the moment in the time of transition I recommend to keep the Yaml file.

Tip: How do you make your Settings.cfg file?

Follow these steps!

Steps:

Here are the steps to create your your Settings.cfg file.

  1. Begin with this Starter Version of Settings.cfg

  2. You may want to look into this real live example to get a feeling for the old and the new version.

  3. All lines beginning with ‘#’ or ‘;’ are comments. No comment is needed. You may remove them all or add your own.

  4. Fill in sensible values for project, version, release, t3author, copyright and description. t3author usually is your name, so probably it’s Max Mustermann or John Doe :-)

  5. It is nice and useful to have “project links” in the documentation like in:

    RESOLVEME://files/001-project-links.png

    Insert and edit the section [html_theme_options] to get those project links.

  6. You CAN do intersphinx crossreferencing in your documentation. If you do see you need to add the definition of abbrev = URL in the [intersphinx_mapping] section.

  7. Now you’re ready to go. Reasonable starter settings for other output formats like PDF, Techinfo, Helptext, Manpages are automatically derived from the given settings. You may of course do further specifications.

    Check this full reference version of Settings.cfg to find out about the possibilities.

This is the end of the chapter about what authors and owners of existing documentation need to know.

Enjoy!

Introduction

Once there was a Debian server srv123. When he was young he served for development and experiments. He soon became very valuable to people and more and more features were added. And, though never planned, he took over the serious job of hosting, serving and building TYPO3 documentation. In general that worked amazingly well and required little maintainance - in relation to what the server offered.

Since years a young and shiny “follow up server” is on its way to take over the work. But it still hasn’t arrived.

Now it is October 2015 and we are ready to deploy a new Sphinx theme for the documentation. Will srv123 be able to cope with the task? Sure, as you’ll already have guessed. But how? That’s what the following story is about.

October 2015: Switch to the new Sphinx theme

Hint: Corresponding files are named like conf-2015-10.py!

t3SphinxThemeRtd is the rising star and new Sphinx theme for TYPO3. The time has come to put the new theme to action.

About development of the theme:

What has to be done to switch to the new theme? That’s what the following is about.

In words: What’s happening?

The new Sphinx theme is ready for deployment. To do so we start by adding some more Python modules to the system to have them available. This can be done without problem allthough the Python version is still 2.6.

Whenever a new version of the theme is available it can be easily updated. This command should never hurt:

sudo pip install --upgrade t3SphinxThemeRtd

conf.py

Next thing is that we switch to a completely new version of the Sphinx configuration conf.py. Note: This file is invariant and exactly the same for ALL projects, no matter whether the rendering is done on the server or locally at home.

Warning: Don’t modify conf.py! Use MAKEDIR/Overrides.cfg to inject settings.

conf-2015-10.py is specifically tailored to the new theme and provides all the necessary defaults. It only needs to be changed for bugfixes or further developments. To work locally at home add a MAKEDIR/Overrides.cfg with setting [html_theme_options]docstypo3org to False. Example: Overrides.cfg of the TypoScript Reference.

Party!

  • So we are NOT using the old conf.py any more.
  • We don’t use t3sphinx any more.
  • We don’t use RestTools.git with the exception of the Pygment’s TypoScript Lexer (one file).
  • We don’t use Yaml any more. So we don’t need to install the Python PyYAML module. Instead we make use of standard Python modules only.

Saying goodbye to that old t3sphinx module means that we are getting rid of the buggy t3html builder which had caused quite some headaches. So: drop the t3html in the Makefile and use the standard html builder instead. This is exactly what has changed in Makefile-2015-10.

INI-files

New: PROJECT/Settings.cfg, MAKEDIR/Defaults.cfg, MAKEDIR/Overrides.cfg

Gone: PROJECT/Settings.yml

The reason that we have configuration files is that we don’t want to have different versions of conf-2015-10.py.

In the documentation project itself we now use an INI-style textfile Settings.cfg instead of the Yaml file Settings.yml.

Compiling the effective settings for Sphinx rendering works like this:

  1. conf-2015-10.py comes with sensible defaults
  2. The user settings from Settings.cfg are added.
  3. IF Defaults.cfg is present then settings from there are added but ONLY IF they don’t exist.
  4. In the next step some hardcoded overrides in conf-2015-10.py are applied.
  5. Finally Overrides.cfg is applied without further checks and “has the last word”. Note that Overrides.cfg is under control of the administrator.

Tip: The effective settings are logged to LOGDIR/Settings.pprinted.txt

All of the three files Settings.cfg, Defaults.cfg, Overrides.cfg are in INI-style and could have the same contents. INI-file format doesn’t allow all Python constructs that are possible in conf.py. There is a reference file conf-2015-10.Reference.Settings that should visualize what the current conf-2015-10.py file understands.

Other than before array data is now ADDED to what is already existing. This means that an Intersphinx mapping or an extra Sphinx extension can be specified by the user and ADDS to whatelse is defined already.

Sphinx extensions will only be used if they are member of a white-list in conf-2015-10.py.

New: Automatic Defaults for other formats

Another new feature is that conf-2015-10.py now automatically defines default settings for the output formats latex_documents, man_pages, texinfo_documents and epub. The required values are derived from the existing ones. This means:

Tip: ALL manuals are automatically prepared for PDF-generation.

The defaults can of course be overridden by the user in Settings.cfg or by the administrator in Overrides.cfg.

Add Python Modules To The Server

On the server we need to add some Python modules. The installation commands can be repeatedly given. The same can be done on a local machine.

Added:

sudo pip install --upgrade t3fieldlisttable
sudo pip install --upgrade t3SphinxThemeRtd
sudo pip install --upgrade t3tablerows
sudo pip install --upgrade t3targets

Not needed any more, but not removed:

pip show pyyaml

Common Files Used By All Documentation Projects

To install the latest version:

srv123$  cd ~/public_html/services
srv123$  git pull

New: WEBROOT/services/ajaxdownloads.php

ajaxdownloads.php replaces pdfchoices.php. In the end pdfchoices.php will not be needed any more.

New: WEBROOT/services/ajaxversions.php

ajaxversions.php replaces versionchoices.php. In the end versionchoices.php will not be needed any more.

New: USERROOT/scripts/bin/conf-2015-10.py

Now we have a single, “one and only” conf.py file that is going to be used for ALL of our Sphinx projects that are using the t3SphinxThemeRtd theme. It doesn’t matter if you’re using the file at home or on the server: Don’t make any modifications, just use it. Customizations are done in the *.cfg INI-files.

And here it is: conf-2015-10.py Preferrably you will want to create a symlink to it: ln -s conf-2015-10.py conf.py.

conf-2015-10.py requires buildsettings.sh

Files that need to live together in MAKEDIR:

MAKEDIR/buildsettings.sh  # New! Needed! Is sourced into Bash AND read as INI-file
MAKEDIR/conf.py           # Make it a symlink to *conf-2015-10.py*!
MAKEDIR/Makefile          # In general: just what 'sphinx-quickstart' proposes.
MAKEDIR/Defaults.cfg      # Optional! Not used at the moment.
MAKEDIR/Overrides.cfg     # Optional!

Example of a minimal buildsetting.sh file:

# buildsetting.sh

# absolute path, or relative to conf.py, without suffix (.rst)
MASTERDOC=../Index

# absolute path, or relative to conf.py
BUILDDIR=./build

# absolute path, or relative to conf.py
LOGDIR=./_not_versioned

New: USERROOT/scripts/bin/cron_rebuild-2015-10.sh

cron_rebuild-2015-10.sh replaces cron_rebuild.sh.

Please compare the new file with the old one to find out about the differences. .. note:

- Please keep BINDIRECTORY a plain assignment to facilitate debugging of the script

- The Latex target_name is now set to the constant 'PROJECT', which always
  gives *PROJECT.tex*

New: USERROOT/scripts/config/Makefile-2015-10

Makefile-2015-10 replaces Makefile. In the end the older Makefile will not be needed any more.

Here we only have a tiny difference: As html builder we have reverted to the standard html builder and dropped the buggy t3html builder.

Project Files

These files are individual and specific for each documentation project.

A documentation project needs to have at least these two files:

  • PROJECT/Documentation/Index.rst - the master document (MASTERDOC)
  • PROJECT/Documentation/Settings.cfg - settings in INI-style

At the moment most projects have:

  • PROJECT/Documentation/Settings.yml - doesn’t hurt, will be dropped in the future

Project files are maintained and edited by the project owner or author. The Settings.cfg file is mandatory and maintained by the project owner. He is responsible - only! - for the NON TECHNICAL metadata of his project, as there are title, copyright, author and so on.

New: PROJECT/Documentation/Settings.cfg

Example: Settings.cfg

Settings.cfg is a text file in INI-style. It is easy to use and secure. Details about the syntax are given here.. Settings.cfg replaces the Yaml file Settings.yml we have been using up to now. Yaml isn’t used any in the build chain.

Relevant rules for Settings.cfg in TYPO3 context:

  1. There needs to be a section [general] in Settings.cfg.

  2. It is expected that the documentation project owner supplies these settings in the [genera] section:

    • project: A line with the title of your project
    • version: Should be a version number like 1.2
    • release: Should be a release number like 1.2.3
    • t3author: As line with author information
    • copyright: Notes about the copyright
    • description: A possibly multiline but still short description of you project.

    The user is not restricted to these settings. All simple definitions that would have otherwise gone into conf.py can be given here.

Please refer to this REFERENCE OF Settings.cfg to find out what’s possible.

In the end Settings.yml will not be needed any more.

Project-specific Build Files in MAKEDIR/

Building a documentation project is controlled by several files that live together in a project specific MAKEDIR.

New: MAKEDIR/buildsettings.sh

MAKEDIR/buildsettings.sh replaces MAKEDIR/cron_rebuild.conf.

Example:

# buildsettings.sh

# absolute path, or relative to conf.py, without suffix (.rst)
MASTERDOC=../ExtbaseFluid.git/Documentation/Index

# absolute path, or relative to conf.py
LOGDIR=.

PROJECT=t3extbasebook
VERSION=latest

# Where to publish documentation
BUILDDIR=/home/user/public_html/typo3cms/ExtbaseFluidBook/latest

# If GITURL is empty then GITDIR is expected to be "ready" to be processed
GITURL=https://github.com/TYPO3-Documentation/TYPO3CMS-Book-ExtbaseFluid.git
GITDIR=/home/user/HTDOCS/github.com/TYPO3-Documentation/TYPO3/Book/ExtbaseFluid.git
GITBRANCH=latest

# Path to the documentation within the Git repository
T3DOCDIR=$GITDIR/Documentation

# Packaging information
PACKAGE_ZIP=0
PACKAGE_KEY=typo3cms.books.t3extbasebook
PACKAGE_LANGUAGE=default

The insider will notice that nothing much has changed. Only two settings are new: MASTERDOC and LOGDIR. Note that both can be in absolute form or relative to the location of conf.py. For local rendering only MASTERDOC, BUILDDIR and LOGDIR are needed.

In the end cron_rebuild.conf will not be needed any more.

Logging

The new conf-2015-10.py will dump the final summary of effective settings to LOGDIR/Settings.pprinted.txt. It can be made public as it doesn’t contain sercurity relevant information.

Steps to migrate an existing documentation project

2 Update the MAKEDIR

Example:

  1. Copy this codeblock as a whole into your clipboard:

    # cd /path/to/MAKEDIR
    
    # make backup
    mv  conf.py            OLD-conf.py
    mv  cron_rebuild.conf  OLD-cron_rebuild.conf
    mv  cron_rebuild.sh    OLD-cron_rebuild.sh
    mv  Makefile           OLD-Makefile
    
    # find old MASTERDOC value
    LINE=$(cat OLD-conf.py | grep relpath_to_master_doc\']\ \=)
    MASTERDOC=$(echo $LINE | awk '{print $(NF)}' | sed s/[\"\']//g)
    
    # find old LOGDIR value
    LINE=$(cat OLD-conf.py | grep relpath_to_logdir\']\ \=)
    LOGDIR=$(echo $LINE | awk '{print $(NF)}' | sed s/[\"\']//g)
    
    # make buildsettings.sh
    # Verify MASTERDOC and LOGDIR afterwards!
    echo "# buildsettings.sh"         >  buildsettings.sh
    echo ""                           >> buildsettings.sh
    echo "# absolute path, or relative to conf.py, without suffix (.rst)" >> buildsettings.sh
    echo "MASTERDOC=$MASTERDOC/Index" >> buildsettings.sh
    echo ""                           >> buildsettings.sh
    echo "# absolute path, or relative to conf.py" >> buildsettings.sh
    echo "LOGDIR=$LOGDIR"             >> buildsettings.sh
    echo ""                           >> buildsettings.sh
    cat  OLD-cron_rebuild.conf        >> buildsettings.sh
    
    # Switch to version 2015-10
    ln -s /home/mbless/scripts/bin/cron_rebuild-2015-10.sh cron_rebuild.sh
    ln -s /home/mbless/scripts/bin/conf-2015-10.py         conf.py
    ln -s /home/mbless/scripts/config/Makefile-2015-10     Makefile
    
    # schedule a rebuild
    rm build.checksum; php request_rebuild.php
    
    # Be happy!
    
    # clean up more?!:
    rm 10* 20* 30* warnings.txt
    rm dirs-of-last-build.txt
    rm included-files-check.log.txt
    
    ls -la
    cat buildsettings.sh | grep MASTERDOC
    
    #
    
  2. Go to a MAKEDIR in old style on server srv123 and insert your clipboard at the commandline.

  3. You may want to have a look at your files with ls -la and cat buildsettings.sh.

  4. Don’t forget to add the Settings.cfg file to each branch of your project! As an example you can look at the Settings.cfg file of the TypoScript Reference

  5. Wait for the next rendering!

3 Supply a _make folder

As a service for users that want to render the documentation project locally at home provide a PROJECT/Documentation/_make folder. For this to do simply copy the complete `\_make folder of the TypoScript reference <>`__.

It is a clever idea to check whether the conf-2015-10.py in there is still the latest version of conf-2015-10.py. Otherwise download the latest version of *conf-2015-10.py*.

ToDo

  • A development task: Do NOT duplicate all theme files for each documentation on the server! Instead: run a postprocessor to replace links to css-, js- and font-files by links to ONE file on the server. By doing that the user uses THE SAME resource files for multiple documentation projects. This is better than using fdupes.
  • Not yet checked: What happens to manuals that only have a README.rst?
  • Not yet checked: Does cron_rebuild-2015-10.sh work for manuals with translations, like we have in ext:sphinx?
  • Check: Is use_opensearch working?

Tips

  1. Disable checksum check once for all versions of a manual (example):

    find . -wholename "*Typoscript.git*build.checksum" -exec rm {} +
    
    # or (probably)
    
    find . -wholename "*Typoscript.git*build.checksum" --delete
    
  2. Pitfall!

    To create ‘false’ values leave the right side empty in *.cfg files:

    ; here's is what you want:
    this_is_a_falsy_value =
    
    ; the following values may or may not be counted as "True",
    ; depending on how the value is treated in the theme.
    ; 'anything', '0', 'false', 'no' are treated as strings
    ; with length > 0 and therefore are "True".
    this_is_a_truthy_value = anything
    this_is_a_truthy_value = 0
    this_is_a_truthy_value = false
    this_is_a_truthy_value = no
    

Why INI-files?

It had taken some effort to make people use the PROJECT/Settings.yml file in Yaml-format. What sense does it make to switch to INI-style and PROJECT/Settings.cfg?

Reasons

Easiness: INI-files are human-friendly and easy to use. You can insert comments much better. The format is widely spread and much older. In general it’s just about writing key value pairs as assignment. Compare with Rfc-822 (1982). It is a lot less frightning to non programmers than Yaml files. It is the same format that Sphinx itself uses for the configuration of themes. A PHP parser is available. A Python parser is available.

Security: INI-files are secure, Yaml-files can be dangerous. See:

Maintenance costs To make Yaml secure an extra Python module is needed which we would have to create and maintain ourselves. It needs to be documented and people have to install it.

Aside: It was me myself who proposed the Yaml format back then in 2012. It was a decision I regretted soon. But until recently I didn’t have a better idea.

I had taken security measures in the Yaml script in t3sphinx. But that code has never been approved by somebody else and I wouldn’t stand in for it with my life.

Migration Issues

For new documentation it is easy to start with a PROJECT/Settings.cfg right away.

For existing documentation there are two ways that come into my mind. Maybe there are more.

  1. We could implement an automatic creation of the PROJECT/Settings.cfg file on the server and read from PROJECT/Settings.yml. This means that we at least have to have our own MakeYamlSecure module on the server. Or can we use PHP for it without running into security issues?
  2. Writing an INI-file if you have a Yaml file already is trivial. And how many developers are actually affected? So another solution would be: Yes, DO render the documentation even if Settings.cfg is missing. And insert a <a href="link/to/SettingsCfgHowTo">((missing))</a> where otherwise title, author and copyright appear.

I would prefer solution two as

  • I’m sure documentation maintainers are happier with that
  • it will soon lead to a transition everywhere
  • it requires the least effort on all sides.

At the moment rendering stops when Settings.cfg is missing with the new build chain. So I still would have to change conf-2015-10.py for that.

More thoughts: When people switch to Settings.cfg its a good moment to have them check their entries, add a description setting and update the copyright setting.

Comments

comments powered by Disqus

Previous topic

Drag And Drop From Jetbrains IDE

Next topic

TYPO3 Usergroup Meeting in Münster

Tags

Archives

Languages

Recent Posts

This Page