Bash

My Scripting Workflow

blue 1 Comment LSL, Tutorials

Introduction

A while ago, I described my switch to a new scripting workflow. That article explained to my customers why I was re-releasing a lot of my products with no functional change, and that due to my changed workflow, solving bugs they might encounter could be slower than was previously the case.

This article is for my SL scripter colleagues.  It lays out in precise detail just how my scripting workflow works, and includes a functional makefile and bash scripts to implement it.

Required Software

I use Linux (specifically Ubuntu) as my operating system of choice.  You will need the following software installed to implement this workflow.

  • Mercurial
  • GNU Make
  • GCC
  • Firestorm
  • LSL-PyOptimizer
  • Python
  • TortoiseHG (optional)

All my LSL projects are in Mercurial repositories, one repo per project.  Yes, yes, I know that the world seems hellbent on standardizing on Git, but 1) I’ve been using Mercurial forever, and 2) Git is far more complex that it needs to be for a single developer.  If you prefer Git (or some other VCS), feel free to take what follows and replace the hg commands.

LSL-PyOptimizer and the GNU Make utility are also a crucial part of this workflow, as is the Firestorm Preprocessor.

If you’re a scripter and unaware of LSL-PyOptimizer, you’re missing out.  This program, written in Python, takes your LSL as input and spits out LSL that is memory optimized.  Depending on your script, you can save large percentages of your original memory footprint by using this utility.  LSL-PyOptimizer also supports some sorely missed features in the LSL language, which can make coding easier and the end result more readable.  Get it, use it.

One note on LSL-PyOptimizer.  As the package is delivered, the main python script is called main.py, which is not very useful.  I copy main.py to lslpo and include the directory in my path so I can invoke it with a non-generic name.  Additionally, if you are running Python3 (which you should be in this day and age), you need to change the hash bang at the start of the file to remove the trailing “2” from “Python2”.

Python is required to run LSL-PyOptimizer.

GNU Make is a dependency build system.  You write instructions to tell Make how to build your project.  Because LSL-PyOptimizer recognizes most C preprocessor directives, and LSL is C-like syntax, you can use the GCC Preprocessor to automatically build dependencies for you, which Make will then use to build your project.  Make will also be responsible for automatically generating a version number for the resultant code (based on Mercurial tags).

Using Firestorm enables you to use its preprocessor so you can #include files.  The output of LSL-PyOptimizer will be included in its entirety to eliminate the need of copy/pasting from a file on your PC to the edit window in SL.

TortoiseHG is a graphical user interface to Mercurial.  In the following, I will use the Mercurial command line interface, but normally I will use TortoiseHG to manage my repos.

Step by Step

The easiest way to demonstrate this workflow is to work a real example, and additionally it can work as a test to see if you have everything set up correctly.  Let’s create a dead simple project from scratch, assuming nothing exists.


$ cd ~
$ mkdir src           # Standard source directory
$ cd src
$ mkdir include       # Dir containing included scripts
$ mkdir script        # We will place a supporting bash script here
$ mkdir test          # The name of our test project
$ hg init test        # Initialize the project dir as a Mercurial repo
$ cd test
$ mkdir Script        # This is where our original LSL will live
$ mkdir build         # This is where the LSL-PyOptimizer output will end up
$ mkdir stable        # This is where the current release code will end up

This sets up a source directory (/home/username/src/ is the defacto source directory name on Linux, and the makefile assumes that’s where it is). It also sets up a directory to contain included files such as debug.lsl, which I include in every LSL script I write. And finally it creates the basic structure for a project called “test” and initializes it as a Mercurial repository.

To follow along, visit the source for debug.lsl, copy and paste it into ~/src/include/debug.lsl

Now let’s create a basic LSL script in ~/src/test/Script/test.lsl.


#define DEBUG
#include "debug.lsl"

default {

    state_entry () {
        llSetObjectDesc (VERSION);
        llOwnerSay ("State_entry");
        debug ("Done");
    }
}

We initialized the test directory as a Mercurial repository already, so we can just add our new script as a tracked file and commit our changes. If you haven’t set a username for Mercurial, it will complain about that. Just follow its instructions to correct the issue.


$ hg add Script/test.lsl
$ hg commit -m 'Initial version'

And now the makefile. Even though a copy of this needs be in the project directory, they are not project specific, so when you create a new project you can just copy an existing makefile to the new project’s directory. I’ll list it out here and you can copy and paste it into ~/src/test/makefile


BUILD_DIR := ./build
SRC_DIR := Script/
INC_DIR := $${HOME}/src/include/

SRCS := $(shell find $(SRC_DIR) -not -path '*/.*' -name '*.lsl' -printf '%f\n')

OBJS := $(SRCS:.lsl=.olsl)
OBJS := $(OBJS:%=$(BUILD_DIR)/%)
DEPS := $(OBJS:.olsl=.d)

VERSION := $(shell bash $${HOME}/src/script/version.sh)

.PHONY: all
all: $(DEPS) $(OBJS)

$(BUILD_DIR)/%.d: $(SRC_DIR)%.lsl
	@echo "Rebuild" $@
	@mkdir -p $(dir $@)
	@set -e; rm -f $@; \
	cpp -MM -nostdinc -iquote$(INC_DIR) -iquote$(dir $(abspath $@)) $< > $@.$$$$; \
	sed 's,\($*\)\.o[ :]*,$(BUILD_DIR)/\1.olsl $@ : ,g' < $@.$$$$ > $@; \
	rm -f $@.$$$$

include $(DEPS)

$(BUILD_DIR)/%.olsl: $(SRC_DIR)%.lsl version.txt
	@echo "Rebuild" $@
	@mkdir -p $(dir $@)
	@lslpo \
                --preproc=gcpp \
	        --postarg='-DVERSION="$(VERSION)"' \
                --postarg="-nostdinc" \
		--postarg="-iquote$(INC_DIR)" \
		--postarg="-iquote$(dir $(abspath $<))" \
		--timestamp \
		$< -o $@

.PHONY: clean
clean:
	rm -rf $(BUILD_DIR)

STABLE_DIR := ./stable

.PHONY: stable
stable:
	mkdir -p $(STABLE_DIR)
	rm -f $(STABLE_DIR)/*.olsl
	cp $(BUILD_DIR)/*.olsl $(STABLE_DIR)

The makefile executes a bash script to automatically generate a version based on the last Mercurial tag in the repo of the form “VM.m”, which M is the major version and m is the minor version. For example, V1.0.

The script also appends a patch number to the version string if there are committed changes to the working directory after the last tag. For example, if the last tag is “V1.0”, and there are 5 commits after that tag (not including the commit that created/moved the tag), then the generated version will be “V1.0.5”. If there are no tags in the repo, the version number will start at “V0.1”.

Here’s the bash script. Copy and paste this to ~/src/script/version.sh.


#!/bin/bash
if [ ! -f version.txt ]
then
    echo "V0.1" > version.txt
fi
FILE_VERSION=$(<version.txt)
REGEX="re:^(V\d+\.)?(\d)$"
VERSION=$(hg log -r "." --template "{latesttag('$REGEX')}")
CHANGES=$(hg log -r ". and not keyword('added tag')" --template "{changessincelatesttag}")
CHANGES=$(($CHANGES-1))

if [ ${VERSION} == 'null' ]
then
    VERSION="V0.1"
    CHANGES=$(hg log -r "." --template "{rev}")
fi

PLUS=""
if [ $CHANGES -gt 0 ]
then
    PLUS="."$CHANGES
fi

if [ $FILE_VERSION != $VERSION$PLUS ]
then
    echo $VERSION$PLUS > version.txt
fi

echo "$VERSION$PLUS"

With all of this in place, we can finally run the make command to build the project.


$ make
Rebuild build/test.d
Rebuild build/test.olsl
$ ls build/
test.d  test.olsl
$ cat build/test.d
./build/test.olsl build/test.d : Script/test.lsl /home/blue/src/include/debug.lsl
$ cat build/test.olsl
// Generated on 2024-11-17T19:51:03.056390Z
debug(string text)
{
    llOwnerSay(text);
}

default
{
    state_entry()
    {
        llSetObjectDesc("V0.1");
        llOwnerSay("State_entry");
        debug("Done");
    }
}

Note that in the dependency description file test.d, the build targets not only depend on the LSL in the Script directory, but also debug.lsl in our include directory. This means that if we ever change debug.lsl, make will rebuild the project for us.

Also note that because we didn’t add a version tag to the repo, we get the default version of V0.1 in the generated code. Let’s fix that.


$ hg tag V1.0
$ touch Script/test.lsl    # Force make to rebuild even though the code didn't change
$ make
Rebuild build/test.d
Rebuild build/test.olsl
$ cat build/test.olsl
// Generated on 2024-11-17T20:12:33.719787Z
debug(string text)
{
    llOwnerSay(text);
}

default
{
    state_entry()
    {
        llSetObjectDesc("V1.0");
        llOwnerSay("State_entry");
        debug("Done");
    }
}

The makefile has a stable target too. It simply copies the generated LSL currently in the build directory to the stable directory. When I tag a changeset as a release version, I’ll simply run this.


$ make stable
mkdir -p ./stable
rm -f ./stable/*.olsl
cp ./build/*.olsl ./stable
$

Lastly, the repo is a bit of a mess.


$ hg status
? build/test.d
? build/test.olsl
? makefile
? stable/test.olsl
? version.txt

You can choose how to deal with untracked files to suit your style. I use an .hgingore file that contains the following standard lines.


$ cat .hgignore
syntax: glob
.DS_Store
build/**
stable/**
version.txt
makefile

And we should add .hgignore as a tracked file and move our tag to the last changeset (or tag with V1.1 if you prefer).


$ hg add .hgignore
$ hg commit -m 'Track .hgignore'
$ hg tag -f V1.0

All this work has so far been done on our PC. As I’m developing/testing the script, to get the results in-world, I will create a script in the target object that looks like this.


#include "/home/blue/src/test/build/test.olsl"

Note you need to enable the Firestorm preprocessor in the scripts editor window settings panel to do this.

Once I’m happy with a release and copy the generated code to the stable directory, the production version of the product will have the include path changed from “build” to “stable”.

Conclusion

It can take a little while to get used to editing the script on your PC, issuing the make command, and then switching to SL and hitting the “Recompile” button in the script editor, but it’s worth it.  Everything is tracked, every version is labeled, and you get memory optimized code, and automatic versioning so you never have to guess which version of script is in the twenty different copies of the object you generated while testing 🙂

I hope this helps some of you manage your code better. If you have questions or comments about any of this, feel free to email me or contact me in-world, or leave a comment here on the blog.