procedual generation on Parsecs Reach

SVG Mountains

Fri, 25 Sep 2026 20:12:00 +0000

Recently I’ve been playing with drawing stuff in web pages. Little procedurally generated toys like these brambles or these mountain ranges I thought I’d write up a little guide on what I do when I build these and how I think about the problems.

Scaffolding

To get started we need something to display stuff. Web browsers make this easy so lets make a web page with an SVG in it. Sure we could use python or rust or almost anything to generate SVGs but for this I’m going to use javascript and html.

Create a file somewhere. Tradition and webserver ease means it should probably be called index.html unless you’ve got more things in your web site that you want to wrap around this.

Inside this file we want to make the simplest svg we can get away with.


<html>
<head>
    <title>Mountainstitle>
head>
<body style="margin:0; padding:0; background-color: black;">
    <svg
        id="image_root"
        version="1.1"
        xmlns="http://www.w3.org/2000/svg"
        xmlns:svg="http://www.w3.org/2000/svg"
        stroke="none"
        fill="none"
        fill-opacity="1"
    >
    svg>
body>
html>

This has the basic html we need. Set the doc type so browsers use modern html and javascript not the old magics. We give the page a title. Strictly speaking we don’t need to do this but its nice. Then we have the body with an svg tag inside it.

We set some styles on the body, the first gets rid of the padding and margin that are there by default so the image we create can run all the way to the edge of the screen. The second sets the background colour. Yes we could use a style tag in the header but this is literally the only css we are going to need so it might as well go here.

On the svg tag we set the basic attributes to turn it into an svg, give it an id so we can find it easily, and set the stroke and fill styles. Both are set to none, so they will not display unless we set something else later. If we wanted we could set default line or fill styles here but each element of the thing I’ll be making will have a different colour so it’ll need updating later any way.

Now we have page, we want to see how it looks. Because this is a single html file we can probably get away with just opening the file by clicking on it or using open. However if in the future we want to include another file next to it then we will need some kind of server. Thankfully python provides a nice simple built in http server that you can spawn to host everything in a folder.

python3 -m http.server 8000

This will start a web server for you at http://localhost:8000 with the content of what ever folder you started it in. It shouldn’t be used for real deployments but it works well enough for development.

Size of the screen and view ports

SVGs kind of try to figure out their size based on whats in them if you don’t tell them how big they should be. Later we’ll also need to know the size of the image so that we can lay things out as we want.

We’ll set our image to the same size as the window, and record it in a global for later. Yes yes, globals are bad. You know why they are bad? Because of namespace pollution and multiple things accessing them. This thing’s a single file with less than 200 lines of code. You know what? I think we can handle this. If this gets out of hand we can refactor it later.

width = window.innerWidth || document.documentElement.clientWidth || document.body.clientWidth;
height = window.innerHeight || document.documentElement.clientHeight || document.body.clientHeight;

const svgEl = document.getElementById("image_root");
svgEl.width = width;
svgEl.height = height;
svgEl.setAttribute("viewBox", "0 0 "+width+" "+height);

SVG Polygons

Ok where were we? Trying to draw fun stuff? Right.

Lets draw a box. We can make it more complicated later. Once we can do this we can play around with algorithms and do fun stuff, but we need to be able to see what we are doing first.

Lets start with a simple box.

const box = [[0,0], [10, 0], [10, 10], [0, 10]]

An array of arrays representing a list of x, y points that draw a 10 unit square. SVGs can have almost any distance unit as their size, in our case we are drawing stuff on the screen so we probably want pixels eventually. So lets say these are pixels.

With SVGs we want a path element. Paths are more generic than lines, but they let us do all sorts of stuff with curves too. We don’t care about that for now. We just need to draw straight lines between the points we defined above.

The path spec defines movement types. There are upper and lower case versions that depend on what reference frame they move on. Upper case ones move relative to the image origin, lower case move relative to the starting point. For what we need now we want M for move and L for line. So we end up with a space separated list of movement commands, which are also space separated, yeah svg paths are a bit weird. Putting a Z on the end of the string will make it draw a line back to the first point. You can do this manually, but it makes the algorithm a little simpler to use the features it provides.

E.G

M0 0 L10 0 L10 10 L0 10 Z

We can build this using a map function from our original string

let pathStr = points
  .map(function (point, index) {
    if (index == 0) {
      return "M" + point[0] + " " + point[1];
    }
    return "L" + point[0] + " " + point[1];
  })
  .join(" ") + "Z";

And with the path string we can create the path element


let path = document.createElementNS(svgNS, "path");
path.setAttribute("id", "line-" + id);
path.setAttribute("stroke", "none");
path.setAttribute("fill", "black");
path.setAttribute("d", pathStr);
svgEl.appendChild(path);

We should now have a black square on the svg when we load the page.

Random numbers

A slight detour, we are going to need some random numbers. We don’t need super random cryptographic random numbers here. We’re making art, not dealing with the safety of boeing whistle blowers. If we were doing this on paper, a bunch of dice would be fine. In javascript the Math.random() function is our dice. Except our dice give us a random number between 0 and 1 (but never actually 1)

>> Math.random()
0.2725780965518224
>> Math.random()
0.05027598093152419
>> Math.random()
0.6247010986076533
>> Math.random()
0.3988789395544631

Numbers between 0 and 1 are not all that useful a lot of the time. So we’re going to have to do some math to get them into the ranges we need.

To pick a random number between a and b where b is bigger, we can add and multiply

>> (Math.random() * (b-a)) + a

Get the range, multiply by a random number to give us 0 to the gap between a and b then add on a to shift that range up to where we want.

But that’s still a floating point number, it has many decimal points.

Bring in the Math.floor(n) function. This takes floating point number n and chops off the decimal points. Or rounds down.

This is really useful for picking an entry from an array for instance.

>>> myArray[Math.floor(Math.random() * myArray.length)]

Note that a in the formula earlier is 0 here so it vanishes in a puff of simplification.

Making wiggly lines

So now we can make a wiggly line across the top of our square to make a mountain range. We can do this by stepping across the width of our page and deciding on a y coord based on a random number. But just picking a random number for the height at each step is going to make a line that is way way too spiky. We’re trying to make mountains not a weird needle field.

Instead we can use a random number to decide if the next point should be above or below the previous point.

let result = [];
let prevY = height/2;
for (let x = 0; x < width; x++) {
  let y = prevY + 1;
  let r = Math.random();
  if (r < 0.5) {
    y = prevY - 1;
  }

  result.push([x, y]);
  prevY = y;
}

This will draw a wiggly line that will drift with the ups and downs of the random number generator rather than jumping long distances. You can make the choices more complex, for instance if its between 0.4 and 0.6 is stays flat, or maybe some ranges it jumps by 2 rather than one.

Now its a case of repeating this and making a gradient of them down the height of the screen.

Licensing

Giving stuff like this a license isn’t going to do much to the llm scrapers, they really don’t care and I don’t have the funds to out lawyer them. However giving your project something like the Filthy Human Hands license or the Hippocratic license is a good signal to other humans that you care that this is all a bit shit.

Hosting

This comes out with a single static html file, or a small number of them depending on how you structure it, this should be easy to host basically anywhere that allows you to upload your own html. Fun thing though one of the biggest llm advocates also provides free web hosting with github pages. So you can make an llm company deal with the ddos that is all the other llm companies. It’s going to get scraped any way, might as well make them deal with it.

fin

You can see the code for both projects listed at the beginning by looking at the source of the pages. Neither is particularly complex, and intentionally neither is minified or packed in any way. There is no build script or node dependencies. Raw html, svg, and javascript is unreasonably effective for making silly art things.

Go make silly art things.

And when you do tell me about it, so I can see more silly art things.

The world needs more silly art.

Rust workspaces and docker

Thu, 16 Apr 2026 21:02:00 +0000

Recently I’ve been working on a multi workspace rust project. It’s a client and server with some shared libraries and tools. I’ve managed to get it to build bits of its self in docker, with dependency caching, and a very minimal container image.

Rust workspaces are a way to have a monorepo with several sub projects. You can define dependency versions globally across all the sub projects to make things consistent, you can keep slowly changing builds cached, and not have to build some bits if you are not working on them.

My project is a multiplayer game. The game or client side is using bevy which is a very large dependency and one of the largest libraries that I’ve used with rust. The server is an actix-web and diesel based project. Which are also quite large. I really don’t want to be building bevy every time I change the web server, and vise versa. Also bevy is big enough my laptop with 16Gb of ram struggles to build it, some times it takes a couple of goes.

The server needs to run in docker, to make deployments easier. But I don’t want to have to wait for the entire project to build every time I deploy it.

Workspaces have a layout like this in rust

|- Cargo.toml # top level workspace config.
|- Cargo.lock # global lock file of dependencies
|- data/  # a workspace folder (Shared library of data types to share between backend and front end)
| |- Cargo.toml # a workspace config
| |- src/
|   |- lib.rs 
|   |- ... # rust code here
|- game/ # The game client workspace
| |- Cargo.toml # bevy is defined here
| |- src/
|   |- main.rs
|   |- ... # rest of game here
|- server/
  |- Cargo.toml # server workspace config (defines actix-web and diesel)
  |- Diesel.toml
  |- src/
    |- main.rs
    |- ... rest of server code

Cargo defaults to building the entire thing when running cargo build or cargo test. To run a single application you use cargo run --bin game or cargo run --bin server There are plenty of guides on line on setting up workspaces, including in the rust book.

Building a basic docker container was pretty easy. First I started off with a two stage build

FROM rust:1.94 AS build

WORKDIR /parsecsreach

COPY ./* .
RUN cargo build --bin server --release

FROM debian:bookworm-slim

# install postgres system library
RUN apt-get update && apt-get install libpq-dev -y && rm -rf /var/cache/apt/archives /var/lib/apt/lists/*

# copy the build artifact from the build stage
COPY --from=build /parsecsreach/target/release/server .

# set the startup command to run your binary
CMD ["./server"]

A small wrinkle in there to get the postgres library into the bookworm-slim image but that’s not too much of a problem. Fairly standard approach to installing a package and then nuking the archives and lists.

The issue with this is that every time you build the container you will rebuild all of the dependencies for the server. Which can take a few minutes on my laptop, which sucks from a iteration time point of view, no hot reload of the server as it needs compiling means we are stuck waiting for that every time we change a line of code. bleh

So what we want is to have the dependencies compiled in one layer while our code compiles in a later layer.

If we just copy the Cargo.toml files from each workspace over we will get an error saying that the workspace doesn’t contain a main or lib file. For the library code we can just create an empty lib.rs in a src directory next to the Cargo.toml and it’ll let us past with no real code.

For the binary workspaces we need a main function too. For that we can include a dummy rust main file somewhere and copy that in. Then we do a cargo build --bin server --release and we have all our dependencies compiled and cached, and we can copy our real code over the top and run the build again.

But …

If we do that and our dummy lib.rs and main.rs files are newer than the real ones (which if we touch the lib files to create them, is guaranteed) One of cargo or docker are “smart” enough to use that to think it doesn’t need to copy over and rebuild our library and the binary for the server.

So we need to trick it into rebuilding the library and the binary.

The simplest I found was to remove the built artifacts for the library. I also tried changing the Cargo.toml files to point to a different main, which worked but seems a little more hacky.

FROM rust:1.94 AS build

WORKDIR /parsecsreach

# Copy cargo files
COPY ./Cargo.toml ./Cargo.toml
COPY ./Cargo.lock ./Cargo.lock

# copy and create wireframe of the project so we can build only the dependencies.
COPY ./data/Cargo.toml ./data/Cargo.toml
RUN mkdir -p ./data/src && touch ./data/src/lib.rs
COPY ./server/Cargo.toml ./server/Cargo.toml
COPY ./docker/template.main.rs ./server/src/dummy.rs
COPY ./game/Cargo.toml ./game/Cargo.toml
COPY ./docker/template.main.rs ./game/src/main.rs

# this should only build the dependencies
RUN sed -i 's#src/main.rs#src/dummy.rs#' server/Cargo.toml
RUN cargo build --bin server --release
RUN sed -i 's#src/dummy.rs#src/main.rs#' server/Cargo.toml

# Clean out the library stub builds so they get built properly in a second.
# These build files are newer than the ones in the source tree so cargo
# doesn't think it needs rebuilding.
RUN rm ./data/src/lib.rs && \
rm ./data/Cargo.toml && \
rm ./target/release/deps/libdata*

# Copy workspaces that we need
COPY ./data ./data
COPY ./server ./server

RUN cargo build --bin server --release

# our final base
FROM debian:bookworm-slim

# install postgres system library
RUN apt-get update && apt-get install libpq-dev -y && rm -rf /var/cache/apt/archives /var/lib/apt/lists/*

# copy the build artifact from the build stage
COPY --from=build /parsecsreach/target/release/server .

# set the startup command to run your binary
CMD ["./server"]

Now the dependencies will only re-compile if we change the Cargo.toml files which is reasonable, and we have a nice tiny container image to actually run the thing.

Using python to configure rust Part 2

Wed, 26 Nov 2025 18:02:00 +0000

Continuing on from Part 1 If you’ve not read that you probably should before continuing here. Previously we setup a pyo3 project and ran some python code from rust which was able to create a rust class and return it to rust. (I think I can say rust a few more times in this paragraph. Rust rust rust)

This is the kind of python config we are aiming to be able to use inside our rust program:

@configure
def configA():
  return Config(
    name= "blah"
  )

@configure
def configB(configA):
  return Config(
    name= configA.name + "_with_something_extra"
  )

@configure
def configC(configA, configB):
  return Config(
    name= configB.name + "_and_even_more_" + configA.name,
  )

Note the use of the @configure decorator to mark functions that do configuration stuff. That is the first thing we are going to do today.

Step 3: Find all the functions with the decorator

Slight side bar: what the heck is a decorator?

In python a decorator is a bit of syntactic sugar for wrapping a function in another one. Defined by a function that takes the function its “attached” to as a parameter and returns a function. We could implement this configure decorator in python like so:

LIST_OF_FUNCTIONS=[]

def configure(inner_func):
    LIST_OF_FUNCTIONS.append(inner_func)
    return inner_func

We could then loop over LIST_OF_FUNCTIONS and we’d have all our functions.

Of course we need that list in rust. Which isn’t the easiest thing to create, what we just made in python is a global mutable variable. Which is a terrible idea as soon as more than one thread is involved. So rust doesn’t let you do that.

Let me introduce you to the append only vec

Once that’s added to our Cargo.toml we can add this to our rust python module.

use append_only_vec::AppendOnlyVec;
pub static CONFIGURE_FUNCTIONS: AppendOnlyVec<Py<PyAny>> = AppendOnlyVec::new();

#[pyfunction]
pub fn configure(py: Python, inner: Py<PyAny>) -> PyResult<Py<PyAny>> {
    CONFIGURE_FUNCTIONS.push(inner.clone_ref(py));
    Ok(inner)
}

Please don’t ask me how that works but the AppendOnlyVec is doing some magic that lets it add to its self with out being a modifiable version of its self. In the python code we can import the configure function along with the Config object and we can wrap our configure functions in the decorator as in the first code block.


from pyconfigmod import Config, configure

@configure
def configA():
  return Config(
    name= "blah"
  )

@configure
def configB(configA):
  return Config(
    name= configA.name + "_with_something_extra"
  )

@configure
def configC(configA, configB):
  return Config(
    name= configB.name + "_and_even_more_" + configA.name,
  )

Now we have a list of functions we can loop over them. However we don’t know what their parameters are or what order we need to execute them in. So …

Step 4: Finding the arguments for the functions.

At the moment we have a list of function references. We don’t know what they are called and we don’t know what arguments they take.

Python objects have a bunch of built in attributes commonly called magic methods, special methods, or dunder methods. They are the ones that have two underscores around their names. There is a helpful list of them in the python documentation.

The one we need here is one I’ve not used in python before. __code__ This contains information about the code that created the object. Details can be found, as usual, in the python docs. For our needs we can use the co_name attribute to get the function name. I’m not using the co_qualname because I’ve not decided on a good mapping for characters that aren’t allowed in parameter names. This does mean we need all @configure functions to have unique names, which kind of sucks currently. This can be a project for later.

The arguments to the function are a little more complex. Rather than separating out the arguments, they are part of the list of local variables. There is co_argcount which will give us how many there are, so we can pull them from the front of the list of local variables.

First lets get the function name

fn get_func_name<'l>(_py: Python<'l>, func: &Bound<'l, PyAny>) -> PyResult<String> {
    if !func.is_callable() {
        // TODO: figure out error handling
        panic!("Object not callable.")
    }

    let code = func.getattr("__code__")?;
    let name: String = code.getattr("co_name")?.extract()?;
    Ok(name)
}

I really need to sort out error handling at some point, but I’m still in prototype mode so I’m just going to panic if we get given something that isn’t a function.

In the function we get the __code__ attribute and then the co_name attribute from that, then we are done.

Now we can move on to the arguments.

fn get_arg_names<'l>(_py: Python<'l>, func: &Bound<'l, PyAny>) -> PyResult<Vec<String>> {
    if !func.is_callable() {
        // TODO: figure out error handling
        panic!("Object not callable.")
    }

    let code = func.getattr("__code__")?;

    let arg_count: i32 = code.getattr("co_argcount")?.extract()?;
    let bound_var_names: Bound<'l, PyAny> = code.getattr("co_varnames")?;

    if bound_var_names.is_instance_of::<PyTuple>() {
        let tuple = bound_var_names.cast::<PyTuple>()?;
        let mut result = Vec::new();
        for i in 0..arg_count as usize {
            let arg_name: String = tuple.get_item(i)?.extract()?;

            result.push(arg_name);
        }

        Ok(result)
    } else {
        panic!("co_varnames wasn't a tuple")
    }
}

This is a bit more complex … We grab the __code__ attribute. We grab the co_argcount from it so we know how many arguments will be on the front of the co_varnames tuple. We then have to do some slightly nasty casting to get our PyAny into a PyTuple which we can then loop over the arg_count entries and pull out the argument names.

I feel ok panicking if the co_varnames attribute isn’t a tuple given that its a built in python type and if that goes wrong the entire python environment is likely in flames so we won’t be able to do much any way. Should probably check that the tuple actually has arg_count entries too but again, if the python environment we are using has gone that far wrong we have bigger issues.

Now that we have a way to get the name and arguments of a python function in rust land, we can build a HashMap of function name to arguments. In our main, after we have attached all the modules, we can do something like the following.


        let mut dependencies: HashMap<String, Vec<String>> = HashMap::new();
        let mut functions: HashMap<String, Bound<PyAny>> = HashMap::new();
        for func in pyconfigmod::CONFIGURE_FUNCTIONS.iter() {
            let bind = func.bind(py);
            if bind.is_callable() {
                let func_name = get_func_name(py, bind)?;
                let mut arg_names = get_arg_names(py, bind)?;

                println!("func {} args: {:?}", func_name, arg_names);
                functions.insert(func_name.clone(), bind.clone());
                dependencies
                    .entry(func_name)
                    .or_insert_with(Vec::new)
                    .append(&mut arg_names);
            }
        }

Here we are going through the CONFIGURE_FUNCTIONS list we created with the decorator, which got run when we loaded the module. Binding the functions to our python instance, so we can pass around references nicely. Then if its callable grabbing its name and arguments and adding them to the dependencies map. I’m also creating a name to function look up at the same time, we will need that later.

Part 5: Sorting our functions

Now that we have our functions and know the names of their arguments we can sort them to work out the order that they need to run so we have the arguments required for each function before we call it. This set of functions is called a Directed Acyclic Graph (or DAG for short) This is a bunch of fancy words to mean they form a tree structure, with out any loops (or cycles).

The really important thing for us is the lack of cycles. If we have functions that depend on each other then we will not be able to run them because there is no way for us to start. E.G

@configure
def configA(configB):
    return Config(name="Config A" + configA.name)
    
@configure
def configB(configA):
    return Config(name="Config B" + configA.name)

To work out configB we need configA and to work out configA we need configB which is impossible for us to do. So we will be raising an error if this happens.

There are quite a few guides on doing a DAG sort online. I’m going to include the code I wrote here and explain it, as its not too long.


fn remove_args(funcs: &mut HashMap<String, Vec<String>>, name: &str) {
    for (_k, v) in funcs.iter_mut() {
        if let Some(pos) = v.iter().position(|e| e == name) {
            v.remove(pos);
        }
    }
}

fn sort_functions(funcs: &HashMap<String, Vec<String>>) -> Vec<String> {
    let mut working = funcs.clone();

    let mut result = Vec::new();

    while !working.is_empty() {
        let keys: Vec<String> = working.keys().cloned().collect();
        let mut found_something = false;
        for k in keys {
            let args = working.get(&k).unwrap();
            if args.is_empty() {
                result.push(k.clone());

                remove_args(&mut working, &k);

                working.remove(&k);
                found_something = true;
                break;
            }
        }
        // if we get here without breaking then we must have a loop or arguments that don't match functions.
        if !found_something {
            // TODO: real errors here please
            println!("working left with: {:?}", working);
            panic!(
                "Could not find any functions that have no parameters left. There must be a loop or parameters that don't match any functions."
            )
        }
    }

    return result;
}

First we need a modifiable copy of our hashmap of functions and their arguments, we need to remove things from this as we work so its nicer to take a clone of it here.

Then we get on to the actual algorithm its self. This looks a bit more complex due to rust and the borrow checker, but the principle remains the same. Look through our working map of functions, find one that doesn’t have any arguments. This means it has no dependencies, so add it to the result list. Then go through the working map and remove any arguments with the name of the function we have. We also need to remove the function from the working map or we will end up with an infinite loop.

Then we start again, and we keep going until the working map is empty. If we ever manage to loop over all the entries in the working map and don’t find an entry that has no arguments, it means we have a loop somewhere, or we have argument names that don’t match a function we know of. Either way that’s an error and we need to tell the user about it, or in our case panic and stop the program.

One thing to note about this implementation is it is not entirely stable, by design. If there are two functions that take the same arguments, we don’t care what order they run in. working.keys() returns the keys in a random order. If two configs don’t have a defined order it should not matter which order they run in. If there is accidentally a dependency then that is a bug and they should have an extra parameter to make the ordering defined.

Part 6: Running the configure functions

Now that we have the information about all the configure functions and know the order they need to be executed in, we can start running them.

        let sorted_functions = sort_functions(&dependencies);
        println!("{:?}", sorted_functions);

        // execute the functions, keeping track of their results, looking up the arguments as needed.
        let mut results_map: HashMap<String, Config> = HashMap::new();
        for func_name in sorted_functions {
            println!("Executing {}", &func_name);
            let args = dependencies.get(&func_name).unwrap();
            let kwargs = create_kwargs(py, &results_map, args)?;
            let result: Config = functions
                .get(&func_name)
                .unwrap()
                .call((), Some(&kwargs))?
                .extract()?;

            results_map.insert(func_name.clone(), result);
        }

        println!("results: {:?}", results_map);

The code here loops over the list of function names that comes out of the sort function. Builds a set of kwargs (Key word args in python land) which is a PyDict that gets expanded into named arguments. Calls the function with those kwargs and finally stores the result for later.

When we create the kwargs we have to build a PyDict, I also convert our Config objects into ReadOnlyConfig objects so that later configure functions can not mess with the results of earlier functions. This is a near clone of the original Config object but it doesn’t have the constructor function or the set_all argument on the pyclass macro. There is also a function to create one from a Config object.

The create_kwargs is as follows:

fn create_kwargs<'l>(
    py: Python<'l>,
    existing_results: &HashMap<String, Config>,
    args: &Vec<String>,
) -> PyResult<Bound<'l, PyDict>> {
    let result = PyDict::new(py);

    for arg in args {
        let cr = existing_results.get(&arg.to_string());
        if cr.is_none() {
            panic!(
                "Couldn't find existing result for {}. This shouldn't be possible if the sort worked right",
                arg
            );
        }

        result.set_item(arg, ReadOnlyConfig::from_config(cr.unwrap()))?;
    }

    Ok(result)
}

As usual the error handling can be massively improved here, but something has gone horribly wrong if we don’t have a result for a function we need. The sort would have had to have broken somehow.

Now we have a fully working configuration system, if MVP level of implementation, rather than production code. Lets run it on our config and see what happens.

emily@diamondslab:~/Projects/breezeblock$ cargo run
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.02s
     Running `target/debug/breezeblock`
func configA args: []
func configB args: ["configA"]
func configC args: ["configA", "configB"]
dependencies: 
{"configC": ["configA", "configB"], "configA": [], "configB": ["configA"]}
["configA", "configB", "configC"]
Executing configA
Executing configB
Executing configC
results: {"configB": Config { config_class: "", repository: "", name: "blah_with_something_extra", features: [], parameters: {} }, "configC": Config { config_class: "", repository: "", name: "blah_with_something_extra_and_even_more_blah", features: [], parameters: {} }, "configA": Config { config_class: "", repository: "", name: "blah", features: [], parameters: {} }}

In the results section at the end we can see it has done what we wanted.

There are many improvements that could be made here, but the basic principle works just fine. Now we can go and do something with our config objects.

I’ve pushed the full code up to codeberg so if you want to play around with it feel free. If you find this interesting or want to use it in something I’d love to hear from you, drop me a message on mastodon @emily_s@mastodon.me.uk

Using python to configure rust Part 1

Mon, 24 Nov 2025 21:02:00 +0000

I’ve recently been playing with running python code from rust to use as a kind of configuration and extension system. The idea is that my program runs a bunch of python code that the user has defined and that gives it the configuration it needs. The user has the POWER OF THIS FULLY OPERATIONAL PROGRAMMING LANGUAGE cough cough, sorry voice box got stuck on imperial functionary there.

I’m doing this as a personal project and proof of concept so don’t expect bullet proof code here. If you want to do something similar then you will need to put some effort into hardening this. I’ll also be pretty vague about what is being configured here as I don’t think it matters right now. Think of something like helm charts or terraform config. Something that would usually be configured by a bunch of yaml files and maybe some templates. I want to know why they aren’t configured with code, so I’m experimenting to find the reasons.

This is going to be a series of posts and to avoid it being a book, I’m going to assume you are familiar with rust and python. You know how to set up a project in both languages and are comfortable with their memory models, module systems, and standard libraries.

We can run python code in rust using the library pyo3. To do this we needed to load the users code, find the right functions to call in the users code, and then run them. I could make them name their function something particular, but that doesn’t give much power. It would be way nicer if they could add a decorator to their config functions and that allowed me to find them. What would also be cool is if we could chain them, so if the args to one function meant that it would call the function with that name first. E.g

@configure
def configA():
  return Config(
    name: "blah"
  )

@configure
def configB(configA):
  return Config(
    name: configA.name + "_with_something_extra"
  )

@configure
def configC(configA, configB):
  return Config(
    name: configB.name + "_and_even_more_" + configA.name,
  )

Where it is able to work out that it needs the results of configA and configB to be able to call configC

There are a few steps here:

Load all the python files in a folder into the python environment
Create a rust struct that python code can create for the config objects
Find all the functions with the @configure decorator on them
Work out what the arguments are for each function
Work out the order they should be executed in so we have the results needed as inputs to other functions
Execute the functions and save their results so we can pass them as parameters later.

This is going to be a lot so I’ll split this into several posts.

Step 1: Load all the python files into the environment

pyo3 works in rust by creating an environment to run in. Python doesn’t have lifetimes and all the other fun and wonderful memory safety we have in rust. So all the python variables and memory gets put inside the pyo3 environment where it gets the lifetime of the environment. If you don’t understand this, don’t worry, its rust memory model stuff, you probably aren’t the intended audience for this (sorry)

I’m going to expect you’ve got a rust environment set up and a python environment setup. I’m on an old debian currently and have python 3.11.2 but I’ve not seen anything that should break using different versions. I’ve just not tested it.

Starting with a fresh cargo new project and adding pyo3 to the dependencies.

pyo3 = {version="0.27", features=["auto-initialize"]}

Now lets create the python environment inside our rust main.

use pyo3::prelude::*;
fn main() -> PyResult<()> {
    Python::attach(|py| {
        // do something in python here
    });
}

Inside that attached python block we can do things in python, see the pyo3 docs for examples Here I’m going to jump right into loading the users code from a folder somewhere.

pyo3 has a function to create a python module from a string. We can reasonably easily load a file into a string. One wrinkle is we need to use CString rather than the usual rust String because python operates in c land and we have to give it strings it will be able to comprehend.

fn attach_module<'l>(
    py: Python<'l>,
    root_dir: &Path,
    input_path: &Path,
) -> PyResult<(CString, pyo3::Bound<'l, pyo3::types::PyModule>)> {
    let file_name = CString::new(input_path.file_name().unwrap().as_encoded_bytes())?;
    let module_name = CString::new(
        input_path
            .strip_prefix(root_dir)
            .unwrap()
            .to_str()
            .unwrap()
            .replace("/", ".")
            .strip_suffix(".py")
            .unwrap(),
    )?;

    let content = CString::new(fs::read_to_string(input_path)?)?;
    let module = PyModule::from_code(py, &content, &file_name, &module_name)?;

    Ok((module_name, module))
}

The first thing is the mess of a type signature. The lifetime parameter 'l tells rust that the life time of the py parameter is going to be the life time of our results as well. We are going to be returning a PyResult, which is like a built in Result but it always returns a PyErr on errors. In the good case we are going to return a tuple of a CString (the module name) and a pyo3::Bound which acts like a rust Rc to the module with a lifetime.

Then there is a fair chunk of mess in there to convert the file path to a python module name. I’m making the assumption that you are using an operating system that cleanly converts from OsString to String. I’ve worked on windows, linux, and macs and I’ve not seen that conversion fail so for this code I’m ok with this assumption. It also assumes that its been given a string with .py extension, you’ll see why I’m ok with this in a minute.

Reading the file and converting it to a CString is a single line. (Its fun how some really complex things are easy in rust but some things are a pain like taking a file path and replacing all the / with spaces :) )

Once we’ve got the text of the file, the filename and the module name we can call the PyModule::from_code to create the module. We have to hand it a reference to the python environment we are using so it knows where to create the module.

This handles a single file for us. Easy enough. Lets go one more level and load all the files in a directory, even if they reference each other.

To read all the files in a directory, including sub directories we can use the WalkDir crate. This can give us an iterator over a directory and we can use the filter functions to find only the things we are interested in.

fn attach_modules<'l>(
    py: Python<'l>,
    input_path: &str,
) -> PyResult<HashMap<String, pyo3::Bound<'l, pyo3::types::PyModule>>> {
    let mut result = HashMap::new();
    
    let root_path = Path::new(input_path);
    let py_str = OsStr::new("py");
    
    for entry in WalkDir::new(input_path)
        .into_iter()
        .filter_map(|e| e.ok())
        .filter(|e| e.metadata().unwrap().is_file())
        .filter(|e| e.path().extension().unwrap_or(OsStr::new("")) == py_str)
    {
        let path = entry.path();
        let (name, module) = attach_module(py, &root_path, path)?;
        result.insert(name.into_string()?, module);
    }

    Ok(result)
}

This time our function takes a single path, where we are going to work from to find our python files. We create a HashMap (like a python dictionary) to use as the result then start using WalkDir First we filter out anything that the OS didn’t let us read (permissions fun etc) then we filter for only files, then for paths which have the “py” extension. We then use the entries that gives us to call the attach_module function we created just now.

So now we’ve been able to create all our modules. However they aren’t actually available to each other. They are just Module objects, they are not really part of the python “class path” to mix my language concepts. Python manages its modules by keeping a dictionary in the built in sys module called modules when ever you try to import something in your code it looks in that dictionary to find it and errors if its not there.

We can do this in rust with a few modifications to the above function

fn attach_modules<'l>(
    py: Python<'l>,
    input_path: &str,
) -> PyResult<HashMap<String, pyo3::Bound<'l, pyo3::types::PyModule>>> {
    let mut result = HashMap::new();

    let sys = PyModule::import(py, "sys")?;
    let py_modules: Bound<'l, PyDict> = sys.getattr("modules")?.cast_into()?;

    let root_path = Path::new(input_path);
    let py_str = OsStr::new("py");
    for entry in WalkDir::new(input_path)
        .into_iter()
        .filter_map(|e| e.ok())
        .filter(|e| e.metadata().unwrap().is_file())
        .filter(|e| e.path().extension().unwrap_or(OsStr::new("")) == py_str)
    {
        let path = entry.path();
        let (name, module) = attach_module(py, &root_path, path)?;

        py_modules.set_item(&name, &module)?;

        result.insert(name.into_string()?, module);
    }

    Ok(result)
}

We import the “sys” module then get hold of the modules dictionary using the getattr method. We’ll be using this a lot. It returns a PyResult that contains a reference to the python version of the value we asked for. This can be any attribute of the python object in question, in this case a global variable in the sys module. We can then call py_modules.set_item to insert the entry into the dictionary. Now all the modules should be able to find each other when we call them.

We’ve successfully loaded all the python files in the directory. So on to Step 2.

Step 2: Create a rust config struct that can be constructed in python.

What we want here is a rust struct that we can do something like this to

from somewhere import Config
def construct():
    c = Config(name: "foo")
    c.something = "a value"
    
    return c

Unfortunately we can’t just import a rust type and have it work. That would be too much magic. There are some hoops we need to jump though.

The first is creating a new python module that we can import. I’ve created a new rust file to put the config related stuff in. config.rs pyo3 has a macro to do this for us.

use pyo3::prelude::*;

#[pymodule]
pub mod pyconfigmod {


}

Though again, like with us loading the python code this doesn’t add it to the python world so we have do that separately. This needs a line before we attach to python, back in the main.rs file.

pub mod config;

use config::pyconfigmod;

fn main() -> PyResult<()> {
    pyo3::append_to_inittab!(pyconfigmod);
    Python::attach(|py| {

Now we can create the class, it looks like a normal rust struct but with some extra macros, back in the config.rs we can add the struct.

use pyo3::prelude::*;

#[pymodule]
pub mod pyconfigmod {
    
    use pyo3::prelude::*
    
    #[pyclass(set_all, get_all)]
    #[derive(Debug, Default, Clone)]
    pub struct Config {
        pub name: String,
    }
    
}

Note: We need to import the prelude again inside the module. (Don’t ask me why, this is a bit of rust I’ve not figured out yet, and don’t really need to understand right now. This works.)

By default pyo3 treats python classes as immutable. It makes it easier to reason about the thread safety of everything. However if you add the set_all and get_all parameters to the pyclass macro then you get generated getters and setters that take care of that synchronization mess for you. The final thing we need for this to work how we want is a constructor. Currently if we try to construct this thing in python it will tell us that it can’t be constructed.

This also shows us how to add custom methods to our python class.

use pyo3::prelude::*;

#[pymodule]
pub mod pyconfigmod {
    
    use pyo3::prelude::*
    
    #[pyclass(set_all, get_all)]
    #[derive(Debug, Default, Clone)]
    pub struct Config {
        pub name: String,
    }
    
    #[pymethods]
    impl Config {
        #[new]
        fn __new__(name: String) -> Config {
            let result = Config{name: name};
                
            result
        }
    }
}

While this is a very simple function we need to define it our selves.

As a result we are able to create a python file that looks like this

from pyconfigmod import Config

def something():
    print("configuring something")
    c = Config("my name")
    # Eventually do some thing more complex to create this here.
    return c

Back in main we should run this for the grand payoff of seeing it print out something from our Config class. Back in our main.rs file we’ve got something like this.

pub mod config;

use config::pyconfigmod;

// ... module loading code here ...

fn main() -> PyResult<()> {
    pyo3::append_to_inittab!(pyconfigmod);
    Python::attach(|py| {
        modules = attach_modules("/home/emily/Projects/my_config");
    });
}

With the little python script in the previous code block in /home/emily/Projects/my_config we now have the module. So we can test this and see it working we will loop through each of the module and see if we can find an attribute called something and if its callable we will call it.

pub mod config;

use config::pyconfigmod;

// ... module loading code here ...

fn main() -> PyResult<()> {
    pyo3::append_to_inittab!(pyconfigmod);
    Python::attach(|py| {
        modules = attach_modules("/home/emily/Projects/my_config");
        
        for (module_name, module) in modules {
            if module.hasattr("something")? {
                let func_something = module.getattr("something")?;
                println!("Found something in {}", module_name);
                if func_something.is_callable() {
                    let result: Config = func_something.call((), None)?.extract()?;
                    println!("Config received! Name: {}", result.name);
                }
            }
        }
    });
}

If we run this with cargo run we should see something like

emily@diamondslab:~/Projects/breezeblock$ cargo run
   Compiling breezeblock v0.1.0 (/home/emily/Projects/breezeblock)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.44s
     Running `target/debug/breezeblock`
Found something in something
configuring something
Config received! Name: my name

This is a good start. We’ve got the ability to load a folder full of python files into rust. Create a rust class in python, and call functions in python from rust and get back a rust class. Next time we will work on finding python functions with a custom decorator and then working out what parameters those functions take.

An introduction to Polygonical and ESVG

Sat, 07 Jun 2025 12:11:00 +0000

For the last ${scairy_number_of_years} years I’ve been working on some library tooling to create SVG files using rust. There are some libraries that existed already but they either didn’t work for what I needed to do or they had limitations. So I build two libraries of my own. Polygonical and ESVG. Polygonical focuses on 2d geometry operations. Points, lines, polygons and doing things to them. It does not concern its self with how they are shown, what format they are written to disk or anything like that. Its job is to make handling geometries easy. ESVG is a Document Object Model (DOM) based svg creation library, while it can read an svg it is mostly designed to build an svg in memory and write it out to disk or some where else.

Originally both of the libraries were inside another project that is used to create pages of shapes for my wife’s etsy shop. However I realized they were useful for other things and wanted to be able to use them in different projects. So a great extraction and refactoring occurred and two new libraries were born.

Recently I’ve been playing with procedual and random generation to create interesting patterns and realized this would make a good introduction tutorial to both libraries. This is what we are going to be aiming to create

Assumptions

You know some rust
You have a rust coding environment set up
You know a little about the SVG format. I wrote a very basic intro a few years ago

Creating a blank SVG

While we work we are going to want to see some stuff. This kind of thing I find easier to debug visually. So lets start by creating a blank svg document. A bit like a hello world.

First up create the project and add esvg and polygonical to the project.

cargo new svg_patterns
cd svg_patterns
cargo add esvg
cargo add polygonical

Now we need to add the document creation and file handling. Open the main.rs file, it should look like this

fn main() {
    println!("Hello, world!");
}

There are two things we need to define before we can create a document. The first is the Dots Per Inch (DPI) we will be using. This defines the number of pixels per inch of space we will end up with in the final image. Common DPI values are 96 for things appearing on screen, 72 for some cutting machines, or around 300 for print work. For this I’m going to set it to 96. The second thing we need is a definition of the “paper” we are going to be drawing on. I’m not american so I’m going to pick A4, but esvg has support for a range of paper sizes and custom ones.

use esvg::page::Page;

const DPI: i32 = 96;
fn main() {
    let page = Page::A4(DPI);
}

Now we can create our document.

use esvg::page::Page;

const DPI: i32 = 96;
fn main() {
    let page = Page::A4(DPI);
    let document = esvg::create_document(&page);
}

Finally we can write our document to a file.

use esvg::page::Page;

const DPI: i32 = 96;
fn main() {
    let page = Page::A4(DPI);
    let document = esvg::create_document(&page);

    esvg::save("output_path.svg", &document).expect("Could not save SVG");
}

Obviously this could could do with some improvements, but it proves the point. When run it will create a svg file in the working directory called output_path.svg that will look like this internally



 height="297.1270833333333mm" viewBox="0, 0, 794, 1123" width="210.07916666666668mm" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" />

This doesn’t do much. It sets up the doc type, and root svg tag, along with a couple of common namespaces.

It is usually worth setting some default style up on the root tag so that anything inside your image that doesn’t have a style explicitly set will get something. Lets pull document creation out into a function.

use esvg::Element;
use esvg::page::Page;

const DPI: i32 = 96;

fn main() {
    let page = Page::A4(DPI);
    let document = create_document(&page);

    esvg::save("output_path.svg", &document).expect("Could not save SVG");
}

fn create_document(page: &Page) -> Element {
    let mut doc = esvg::create_document(page);
    doc.set("stroke", "#8c8c8c");
    doc.set("fill", "none");
    doc.set("fill-opacity", "1");
    doc.set("stroke-opacity", "1");
    doc.set("stroke-width", "0.5mm");
    doc.set("stroke-linejoin", "miter");
    doc.set("stroke-dasharray", "none");
    doc.set("stroke-linecap", "square");
    doc.set("stroke-miterlimit", "10");
    doc.set("stroke-dashoffset", "0");
    doc.set("color-rendering", "auto");
    doc.set("color-interpolation", "auto");
    doc.set("text-rendering", "auto");
    doc.set("shape-rendering", "auto");
    doc.set("image-rendering", "auto");

    doc
}

I orgiginally cribbed most of these settings from what inkscape generates when you create a new image. The important ones are near the top. stroke controls the colour of the lines, fill controls the colour used inside shapes, fill-opacity and stroke-opacity control how transparant a shape is, they are a number between 0 for completely clear and 1 for completely solid. stroke-width controls how thick the paramiter line is. You can use any units here, if you don’t provide units it will be in pixels. The rest are not so important and can be ignored mostly. If you need to look one up mozilla provide excellent svg reference docs.

If we run the code now we’ll get, but it’ll still look like a blank page.



 color-interpolation="auto" color-rendering="auto" fill="none" fill-opacity="1" height="297.1270833333333mm" image-rendering="auto" shape-rendering="auto" stroke="#8c8c8c" stroke-dasharray="none" stroke-dashoffset="0" stroke-linecap="square" stroke-linejoin="miter" stroke-miterlimit="10" stroke-opacity="1" stroke-width="0.5mm" text-rendering="auto" viewBox="0, 0, 794, 1123" width="210.07916666666668mm" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" />

Lets draw something. The page we created at the start has helper methods to give the pixel locations of various points, top left corner, bottom right, etc. Lets draw a line across the page.

fn main() {
    let page = Page::A4(DPI);
    let mut document = create_document(&page);

    let line = esvg::path::create(&[page.top_left(), page.bottom_right()]);
    document.add(&line);

    esvg::save("output_path.svg", &document).expect("Could not save SVG");
}

(note I’ve only included main now to save space)

The path module in esvg provides helpers for creating path elements. In this case we pass in a slice that contains the coordinates we want to draw between. Note that the top_left method and its siblings all follow the margins applied to the page. They default to half an inch. If you need to control the margins you can use the A4_with_boarder and similar functions.

One thing to note, when you add an element to a parent element it is cloned into the dom tree. Changes made to the element after it has been added will not be applied to the version that gets saved at the end. This is due to the DOM tree needing to take owner ship of the element to make sure it doesn’t vanish when the original element variable goes out of scope.

If you open up the image you’ll see something like this, where the line doesn’t extend all the way to the edges.

Now we can start creating some patterns. We’ll start with the square grid because its slightly simpler. First we need to decide what size of grid we want to create. The one I showed a screenshot of near the top used half centimeter spacing. ESVG provides a collection of conversion functions in the convert module. The useful one for us here is parse_length. This will take a string like 0.5cm or 2.25in and convert it into a number of pixels. With that number of pixels for the grid we can work out how many rows and columns we should have for the given page.

fn main() {
    let page = Page::A4(DPI);
    let mut document = create_document(&page);

    let grid_spacing = convert::parse_length("0.5cm", DPI).expect("could not parse length");

    let num_rows = page.display_height_px() / grid_spacing;
    let num_columns = page.display_width_px() / grid_spacing;
    
    esvg::save("output_path.svg", &document).expect("Could not save SVG");
}

Now we can start actually using the polygonical library. We need to work out a grid of points across the page. We can use a pair of for loops to do the rows and columns. Then we need to create a Point on the page where we want to draw a corner.

Point is the most basic type in polygonical. It represents a point in 2d space. The page.top_left() function we used earlier returns one, but we didn’t need to worry about it then. They have an x and y value and can be translated or moved by other points, rotated around the origin.

Lets have a look at the code and then go through the important parts.

fn main() {
    let page = Page::A4(DPI);
    let mut document = create_document(&page);

    let grid_spacing = convert::parse_length("0.5cm", DPI).expect("could not parse length");

    let num_rows = page.display_height_px() / grid_spacing;
    let num_columns = page.display_width_px() / grid_spacing;

    for y in 0..num_rows {
        for x in 0..num_columns {
            let point = page
                .top_left()
                .translate(&Point::new(x * grid_spacing, y * grid_spacing));

            let circle = esvg::shapes::circle(point, grid_spacing / 5);
            document.add(&circle);
        }
    }

    esvg::save("output_path.svg", &document).expect("Could not save SVG");
}

In the middle of that code we work out the point we want to draw a circle.

let point = page
    .top_left()
    .translate(&Point::new(x * grid_spacing, y * grid_spacing));

This takes the point of the top left of the page, then moves it by a new point with its x and y coords set appropriately. We could keep a running variable and move it by grid_spacing each iteration but I find this slightly easier to understand personally.

After that we create a circle element and add it to the document, just like we did with our line previously. If we run this we should see something like this

Note that we have what looks like different margins on the right and bottom. This is because we are creating one less than the rows and columns calculated at the start. It’ll become clear why later.

Let’s draw some lines. We’ve already seen how to do this so all we need is the end point of the line and we can create these reasonably easily. For each point we visit, if we draw a line to the right and a line downwards we will end up with a nice grid. While yes, we could draw lines all the way along the page and have a smaller SVG at the end, later we are going to want to be able to turn individual sections off so this will be easier if we do it like this.

fn main() {
    let page = Page::A4(DPI);
    let mut document = create_document(&page);

    let grid_spacing = convert::parse_length("0.5cm", DPI).expect("could not parse length");

    let num_rows = page.display_height_px() / grid_spacing;
    let num_columns = page.display_width_px() / grid_spacing;

    let move_right = Point::new(grid_spacing, 0);
    let move_down = Point::new(0, grid_spacing);

    for y in 0..num_rows {
        for x in 0..num_columns {
            let point = page
                .top_left()
                .translate(&Point::new(x * grid_spacing, y * grid_spacing));

            let circle = esvg::shapes::circle(point, grid_spacing / 5);
            document.add(&circle);

            let right_point = point.translate(&move_right);
            let right_line = esvg::path::create(&[point, right_point]);
            document.add(&right_line);

            let down_point = point.translate(&move_down);
            let down_line = esvg::path::create(&[point, down_point]);
            document.add(&down_line);
        }
    }

    esvg::save("output_path.svg", &document).expect("Could not save SVG");
}

To save a bit on object churn I created a pair of points for moving right and moving down, so we could translate by the same points each time. There is nothing really stopping you creating new points in the loop, but I think this is a little clearer.

When you open this you’ll notice that the right and bottom edges are not filled in correctly. Lets take care of that now. There are a few ways we could do this. Extra loops after the main loop, or adding a check to see if we are on the last row, and increasing the iteration counts by one. This is what I’m going to go with.

fn main() {
    let page = Page::A4(DPI);
    let mut document = create_document(&page);

    let grid_spacing = convert::parse_length("0.5cm", DPI).expect("could not parse length");

    let num_rows = page.display_height_px() / grid_spacing;
    let num_columns = page.display_width_px() / grid_spacing;

    let move_right = Point::new(grid_spacing, 0);
    let move_down = Point::new(0, grid_spacing);

    for y in 0..=num_rows {
        for x in 0..=num_columns {
            let point = page
                .top_left()
                .translate(&Point::new(x * grid_spacing, y * grid_spacing));

            let circle = esvg::shapes::circle(point, grid_spacing / 5);
            document.add(&circle);

            if x < num_columns {
                let right_point = point.translate(&move_right);
                let right_line = esvg::path::create(&[point, right_point]);
                document.add(&right_line);
            }

            if y < num_rows {
                let down_point = point.translate(&move_down);
                let down_line = esvg::path::create(&[point, down_point]);
                document.add(&down_line);
            }
        }
    }

    esvg::save("output_path.svg", &document).expect("Could not save SVG");
}

Note that we switched from non inclusive ranges on the for loops to inclusive ranges. .. to ..=

Now that we have built the grid we can make it randomly decide to skip some of the entries. For this we will use the rand library. However, it would be good if we could get the same result repeatedly so that we can test things easily. For that we need a random number generator that we can seed with a known value. Rust’s rand crate doesn’t have one, so we’ll also need to bring in rand_chacha

cargo add rand
cargo add rand_chacha

Now we need to create a rng and then use it to decide if we want to render our lines and circles.

use esvg::page::Page;
use esvg::{Element, convert};
use polygonical::point::Point;
use rand::{Rng, SeedableRng as _};
use rand_chacha::ChaCha8Rng;

const DPI: i32 = 96;

fn main() {
    let page = Page::A4(DPI);
    let mut document = create_document(&page);

    let mut rng = ChaCha8Rng::seed_from_u64(123456789);
    let circle_chance = 0.5;
    let line_chance = 0.75;

    let grid_spacing = convert::parse_length("0.5cm", DPI).expect("could not parse length");

    let num_rows = page.display_height_px() / grid_spacing;
    let num_columns = page.display_width_px() / grid_spacing;

    let move_right = Point::new(grid_spacing, 0);
    let move_down = Point::new(0, grid_spacing);

    for y in 0..=num_rows {
        for x in 0..=num_columns {
            let point = page
                .top_left()
                .translate(&Point::new(x * grid_spacing, y * grid_spacing));

            if rng.random_bool(circle_chance) {
                let circle = esvg::shapes::circle(point, grid_spacing / 5);
                document.add(&circle);
            }

            if x < num_columns && rng.random_bool(line_chance) {
                let right_point = point.translate(&move_right);
                let right_line = esvg::path::create(&[point, right_point]);
                document.add(&right_line);
            }

            if y < num_rows && rng.random_bool(line_chance) {
                let down_point = point.translate(&move_down);
                let down_line = esvg::path::create(&[point, down_point]);
                document.add(&down_line);
            }
        }
    }

    esvg::save("output_path.svg", &document).expect("Could not save SVG");
}

Things to note here: we had to import SeedableRng to get the trait that allows us to set the seed on the chacha8 random number generator. The rng has to be mutable, its internal state gets updated when it generates a random number so that the next one is different. I created two variables to hold the probability of lines and circles being created, this just makes it easier to alter them when playing around.

Now when we run this we should get something that looks like this

There is still plenty to do to this little program. Command line arguments, tidying up into functions, but the core is there. It produces the output we wanted. Hopefully its given you a small taste of whats possible when programmatically creating SVG files.

If you give this a try I’d love to see the things you create. Drop me a message on mastodon

Creating a command line mastodon archive search tool in rust

Wed, 15 Feb 2023 12:11:00 +0000

Today lets step through creating a new command line tool in rust. Recently I wanted to find one of my old toots, but it was from several months ago and mastodon doesn’t have the ability to search for anything other than usernames and hashtags. I couldn’t remember if I’d put hash tags in the post I was after so I couldn’t use the built in thing. What I did have however is my backup archive. I’d downloaded it earlier for unrelated reasons. Wouldn’t it be good if I could run a search on that archive and get back the url for a toot I was looking for? Yep, lets build it. I’m going to step through this in horrible detail as a kind of tutorial on creating a command line tool in rust. So here we go!

Ok, before we start there are some pre-requisites that I already have setup and I’m not going to talk about.

I already have rust installed
I have downloaded my archive (the option is in your settings under import/export, you may need to tap the hamburger button on mobile to find it)
I have a text editor and coding environment set up. In my case VS Code and the Rust analyser plugins.

Now, lets have a terminal open and create our new tool.

cargo new magrep

     Created binary (application) `magrep2` package

This will create a new folder in the current directory called magrep (Mastodon Archive Grep) open that directory in your favourite text editor,

cd magrep
code .

There will be a cargo.toml a .gitignore and a src/ directory with a main.rs. First thing we need is to run this. Ok, we don’t strictly need to run this, but it is a good idea to make sure everything works.

cargo run

   Compiling magrep v0.1.0 (C:\Users\emily\git\magrep)
    Finished dev [unoptimized + debuginfo] target(s) in 1.86s
     Running `target\debug\magrep.exe`
Hello, world!

Hurrah it worked as expected. Now we need to have a quick think about what we want this tool to do and how it should work. We want to search the outbox.json file in the tar.gz archive file and look for any toots with a provided string in them. So we need to know where the archive file is, and what string to search for. When the thing runs it should open up the archive file, decode the outbox.json file, filter out anything thats not a toot (it includes likes and boosts too), and print out any that contain our search string.

Now we know that, the first thing the program is going to need to do is read the command line arguments and find out where the archive is and what we are searching for.

To do that we will use a library called Clap. This provides a bunch of command line parsing logic. We just need to add the following to the dependencies section of our cargo.toml file

clap = { version = "3.0", features = ["derive"] }

Then we can go to our src/main.rs file and start to write some code. We want an argument that is something like -a so we need to create a new command and then add an argument to it. First add an import for the types we need,

use clap::{Arg, Command};

Then add the following to your main function

let matches = Command::new("magrep")
        .arg(
            Arg::new("archive")
                .required(true)
                .short('a')
                .long("archive")
                .help("file path for the archive to search"),
        )
        .get_matches();

This creates a command magrep and an argument with a short name of -a and a long name of --archive that is required. Now the next argument we need to create is the search term. Now it would be nice if we didn’t have to specify that this was the search term, like when you use sed or grep you just type the search parameter. So lets create a positional argument.

let matches = Command::new("magrep")
        .arg(
            Arg::new("archive")
                .required(true)
                .short('a')
                .long("archive")
                .default_value("./archive.tar.gz")
                .help("file path for the archive to search"),
        )
        .arg(
            Arg::with_name("query")
                .index(1)
                .help("match string")
                .required(true),
        )
        .get_matches();

Now the matches variable will have our parsed command line arguments in it. We can pull them out into variables for us to use later like so

    let archive_path = matches.value_of("archive").unwrap();
    let pattern = matches.value_of("query").unwrap();

Lets do a quick test and print those out to make sure things are working…

    println!("path: {}, pattern: {}", archive_path, pattern);

cargo run -- -a something.gz "some pattern"

This will take a bit longer than before as it will need to build clap and all of its dependencies too. You should see something like this

   Compiling hashbrown v0.12.3
   Compiling os_str_bytes v6.4.1
   Compiling once_cell v1.17.1
   Compiling bitflags v1.3.2
   Compiling strsim v0.10.0
   Compiling textwrap v0.16.0
   Compiling winapi v0.3.9
   Compiling clap_lex v0.2.4
   Compiling indexmap v1.9.2
   Compiling winapi-util v0.1.5
   Compiling atty v0.2.14
   Compiling termcolor v1.2.0
   Compiling clap v3.2.23
   Compiling magrep v0.1.0 (C:\Users\emily\git\magrep)
    Finished dev [unoptimized + debuginfo] target(s) in 9.00s
     Running `target\debug\magrep.exe -a something.gz "some pattern"`
path: something.gz, pattern: some pattern

As you can see at the end we got our arguments printed out nicely. Good. Now we can move on to trying to open the archive file. Now this is a tar.gz file, if you’ve not encountered these before they are a compressed wrapper around a tar file, which is just some metadata and a bunch of files concatenated together. We’ll use a couple of libraries to handle these. Add the following to your dependencies in your cargo.toml

flate2 = "1.0.25"
tar = "0.4.38"

We can import the bits we will shortly need,

use std::fs::File;
use flate2::read::GzDecoder;
use tar::Archive;

Now we can open the file, wrap that file in a gz decoder, and wrap that in a tar archive decoder, like so:

    let tar_gz = File::open(archive_path).unwrap();
    let tar = GzDecoder::new(tar_gz);
    let mut archive = Archive::new(tar);

Lets do a test and print out the filenames in the archive, to make sure things are working as we expect.

    for entry in archive.entries().unwrap() {
        let e = entry.unwrap();
        let path_path = e.path().unwrap();
        let path_string = path_path.to_str().unwrap();
        println!("file: {}", path_string);
    }

Now if we run it we should get a list of filenames printed out…

cargo run -- -a ../../Downloads/archive-20230214153535-daa49fde73e802447b521186ba95c8af.tar.gz "some pattern"

   Compiling cfg-if v1.0.0
   Compiling adler v1.0.2
   Compiling windows_x86_64_msvc v0.42.1
   Compiling windows-targets v0.42.1
   Compiling crc32fast v1.3.2
   Compiling windows-sys v0.45.0
   Compiling miniz_oxide v0.6.2
   Compiling flate2 v1.0.25
   Compiling filetime v0.2.20
   Compiling tar v0.4.38
   Compiling magrep2 v0.1.0 (C:\Users\emily\git\magrep2)
    Finished dev [unoptimized + debuginfo] target(s) in 4.58s
     Running `target\debug\magrep2.exe -a ..\..\Downloads\archive-20230214153535-daa49fde73e802447b521186ba95c8af.tar.gz "some pattern"`
path: ..\..\Downloads\archive-20230214153535-daa49fde73e802447b521186ba95c8af.tar.gz, pattern: some pattern
file: media_attachments/files/000/857/528/original/f55902524814b2f2.jpeg
... cropped for brevity but there is a load more media attachments in my archive...
file: media_attachments/files/109/834/160/834/865/079/original/ce8fe16924be4b35.jpg
file: outbox.json
file: likes.json
file: bookmarks.json
file: avatar.jpg
file: actor.json

There is the outbox.json file we want to have a look inside of, so lets see whats inside. The entry object we have above acts like a reader so we can read it to a string like any other file. Lets modify our loop from the previous entry to something like this

    for entry in archive.entries().unwrap() {
        let mut e = entry.unwrap();
        let path_path = e.path().unwrap();
        let path_string = path_path.to_str().unwrap();
        
        if path_string == "outbox.json" {
            let mut buffer = String::new();
            e.read_to_string(&mut buffer).unwrap();
            println!("{buffer}");
        }
    }

Note we need to make the entry e mutable here as the read_to_string method changes the position of of the read pointer in the “file”

I won’t show the output here, suffice to say its a single giant line of json. So our next task is going to be decoding it. For this we need yet another dependency. This time serde_json. serde is a collection of libraries for encoding and decoding rust structs. The json library deals with json encoding and decoding.

serde_json = "1.0.93"

At this point we could define a set of structs that could represent the json object structure inside the outbox.json file. However that seems like a lot of work considering we really only need to pick out a couple of fields, the content, and the url. Helpfully serde_json has a built in Value type that can represent any arbitrary json object. So we can use that and save a bunch of typing and code.

First as usual import the library

use serde_json::Value;

Then we can modify our loop:

    let mut outbox : Option<Value> = None;

    for entry in archive.entries().unwrap() {
        let e = entry.unwrap();
        let path_path = e.path().unwrap();
        let path_string = path_path.to_str().unwrap();

        if path_string == "outbox.json" && outbox.is_none() {
            outbox = Some(serde_json::from_reader(e).unwrap());
        }
    }

    if outbox.is_none() {
        println!("no outbox file in archive !invalid!");
        return
    }

Small change to our loop here to decode the json object. I’m making a style choice here to store the decoded outbox json object for later. I don’t like to have things too deeply nested, they get difficult to understand and this allows us to handle the next bit outside the loop. We do have to take care of the case where we get a tar.gz file that doesn’t contain an outbox.json file so we store it in a option and check that the option is not none after the loop.

So now we have the decoded json object out of the tar ball with out writing anything temporary to disk, which is nice, so now we can start searching for toots that match our pattern. Lets take a look at this json object we have and see what we can do with it. We could print it out but we already know that its huge and going to roll off the end of our shell if we try and print it all out. So lets have a look at what keys are in the top level and see if that helps.

    for key in outbox.unwrap().as_object().unwrap().keys() {
        println!("{}", key);
    }

Much unwrapping later… so we take our Option and unwrap it to a Value we know this is safe because we checked is_none() above and exited if it was. We then convert the Value to an object representation with the as_object call. This returns an Option<&Map>, we kind of know this will work and is safe to unwrap, but if it wasn’t it would be ok to panic here as we are just playing around to figure out our data structure. Then we can get an iterator of the keys and loop over them.

If we run this we should get something like:

   Compiling ryu v1.0.12
   Compiling serde v1.0.152
   Compiling itoa v1.0.5
   Compiling serde_json v1.0.93
   Compiling magrep2 v0.1.0 (C:\Users\emily\git\magrep2)
    Finished dev [unoptimized + debuginfo] target(s) in 7.86s
     Running `target\debug\magrep2.exe -a ..\..\Downloads\archive-20230214153535-daa49fde73e802447b521186ba95c8af.tar.gz "some pattern"`
path: ..\..\Downloads\archive-20230214153535-daa49fde73e802447b521186ba95c8af.tar.gz, pattern: some pattern
@context
id
orderedItems
totalItems
type

That orderedItems key looks promising. Lets take a closer look. Hopefully its an array (if the world makes any sense … we should probably check) lets try it and see what explodes

println!("{}", outbox.unwrap()["orderedItems"].as_array().unwrap()[0]);

This should output something like this, my first ever toot! (Hay nothing exploded!)

{"actor":"https://tech.lgbt/users/Emily_S","cc":["https://tech.lgbt/users/Emily_S/followers"],"id":"https://tech.lgbt/users/Emily_S/statuses/103363936687819074/activity","object":{"atomUri":"https://tech.lgbt/users/Emily_S/statuses/103363936687819074","attachment":[],"attributedTo":"https://tech.lgbt/users/Emily_S","cc":["https://tech.lgbt/users/Emily_S/followers"],"content":"Going to try this mastodon thing again. 
Hello everyone. I'm Emily. A trans lady from the UK. (Yes we made a mess of things lately didn't we) software engineer, space nerd, maker of things and maps, parent and wife.
I'm interested in lots of data processing stuff, I work in a geographic field so you'll probably get rants about projection systems, buggy code, interesting code tricks, and other general stuff.
","contentMap":{"en":"Going to try this mastodon thing again. 
Hello everyone. I'm Emily. A trans lady from the UK. (Yes we made a mess of things lately didn't we) software engineer, space nerd, maker of things and maps, parent and wife.
I'm interested in lots of data processing stuff, I work in a geographic field so you'll probably get rants about projection systems, buggy code, interesting code tricks, and other general stuff."},"conversation":"tag:tech.lgbt,2019-12-24:objectId=3818591:objectType=Conversation","id":"https://tech.lgbt/users/Emily_S/statuses/103363936687819074","inReplyTo":null,"inReplyToAtomUri":null,"published":"2019-12-24T17:28:26Z","replies":{"first":{"items":[],"next":"https://tech.lgbt/users/Emily_S/statuses/103363936687819074/replies?only_other_accounts=true&page=true","partOf":"https://tech.lgbt/users/Emily_S/statuses/103363936687819074/replies","type":"CollectionPage"},"id":"https://tech.lgbt/users/Emily_S/statuses/103363936687819074/replies","type":"Collection"},"sensitive":false,"summary":null,"tag":[],"to":["https://www.w3.org/ns/activitystreams#Public"],"type":"Note","url":"https://tech.lgbt/@Emily_S/103363936687819074"},"published":"2019-12-24T17:28:26Z","signature":{"created":"2023-02-14T15:32:30Z","creator":"https://tech.lgbt/users/Emily_S#main-key","signatureValue":"sdTf+YO4x3og1ApJWIOZ5sEsRq49cCJe35ZJdUzOX+jfna6nMzaWUeAgv4Maz0Iig3SJ7ODNqrONghfQPUdfp6ObwUKwWsSmIfA3mvGZsgElfpkkKFLzqCHGWnW+dOLN6CebFKUCMf5YpWwPvFrxAX4bzs/EAnjCnMK43VWFS/srQo7BEBYnv9qkNeBfR2UNz0xkjQ0HP2YeGOXavNlCovNyN6zSyenFA66h3vNehX3un4szMCdM/u0LodU3pKSonyw8kTBaBpkzkqFHmlTQ5jX+0ZyA/+szFE6FkdTR8zG16bjuvxXoBezyq1rKRxjvGPhWehThYk8YpLtsVl28Yg==","type":"RsaSignature2017"},"to":["https://www.w3.org/ns/activitystreams#Public"],"type":"Create"}

Ok, so we have the content of our toot twice for internationalisation reasons by the look of it. (I guess something got extended at some point in the history of activity pub and now we have both content and contentMap fields) the other one that looks promising is the atomUri which looks a lot like a toot url. It is, we can put that into a browser and it brings up the toot in question.

So lets finish this thing and be done.

    for item in outbox.unwrap()["orderedItems"].as_array().unwrap() {
        if item["object"]["content"].as_str().unwrap().contains(pattern) {
            println!("*************");
            println!("{} : {}",item["object"]["atomUri"], item["object"]["content"]);
        }
    }

So we loop over all the items in the orderedItems list and then get the content as a string and see if it contains the pattern. If so print out its atomUri and the content so we can see it.

Unfortunately if we run this we’ll get a panic

   Compiling magrep v0.1.0 (C:\Users\emily\git\magrep)
    Finished dev [unoptimized + debuginfo] target(s) in 1.47s
     Running `target\debug\magrep.exe -a ..\..\Downloads\archive-20230214153535-daa49fde73e802447b521186ba95c8af.tar.gz "some pattern"`
path: ..\..\Downloads\archive-20230214153535-daa49fde73e802447b521186ba95c8af.tar.gz, pattern: some pattern
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src\main.rs:52:37
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: process didn't exit successfully: `target\debug\magrep.exe -a ..\..\Downloads\archive-20230214153535-daa49fde73e802447b521186ba95c8af.tar.gz "some pattern"` (exit code: 101)

What this is telling us is we tried to call unwrap on a option that was None, on line 52 of main.rs. Which is the if statement in the above block. The only unwrap in there is the as_str() call so we got a content that wasn’t a string? More likely we got an item that didn’t have a content field at all. Lets print some things out and see what happens.

    for item in outbox.unwrap()["orderedItems"].as_array().unwrap() {
        println!("%%%%%%%%%%%%%%%%");
        println!("{}", item);
        if item["object"]["content"].as_str().unwrap().contains(pattern) {
            println!("*************");
            println!("{} : {}", item["object"]["atomUri"], item["object"]["content"]);
        }
    }

I like to include separators so its easy to see whats going on, hence the ‘%’ symbols. With luck when we run this now we should be able to see the cause of the panic.

...
%%%%%%%%%%%%%%%%
{"@context":"https://www.w3.org/ns/activitystreams","actor":"https://tech.lgbt/users/Emily_S","cc":["https://tiny.tilde.website/users/selfsame","https://tech.lgbt/users/Emily_S/followers"],"id":"https://tech.lgbt/users/Emily_S/statuses/103370741914550623/activity","object":"https://tiny.tilde.website/users/selfsame/statuses/101789047613657946","published":"2019-12-25T22:19:06Z","to":["https://www.w3.org/ns/activitystreams#Public"],"type":"Announce"}
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src\main.rs:54:47
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: process didn't exit successfully: `target\debug\magrep.exe -a ..\..\Downloads\archive-20230214153535-daa49fde73e802447b521186ba95c8af.tar.gz "some pattern"` (exit code: 101)

Ah, its something that doesn’t have a content field inside the object. In fact the object is just a string. I think this is a boost, which is not something we want to search so we can ignore them. I remember now I did mention this way way back at the start of this post, oops. That type field looks like a good candidate for picking out what we need. Lets add a filter on that being “Create”

    for item in outbox.unwrap()["orderedItems"].as_array().unwrap() {
        if item["type"].as_str().unwrap() == "Create" {
            if item["object"]["content"].as_str().unwrap().contains(pattern) {
                println!("*************");
                println!("{} : {}", item["object"]["atomUri"], item["object"]["content"]);
            }
        }
    }

And if we run it now we get…

   Compiling magrep v0.1.0 (C:\Users\emily\git\magrep)
    Finished dev [unoptimized + debuginfo] target(s) in 1.20s
     Running `target\debug\magrep.exe -a ..\..\Downloads\archive-20230214153535-daa49fde73e802447b521186ba95c8af.tar.gz "some pattern"`
path: ..\..\Downloads\archive-20230214153535-daa49fde73e802447b521186ba95c8af.tar.gz, pattern: some pattern

Ah I’ve never tooted “some pattern” lets try something else…

PS C:\Users\emily\git\magrep> cargo run -- -a ..\..\Downloads\archive-20230214153535-daa49fde73e802447b521186ba95c8af.tar.gz "fish"
   Compiling magrep v0.1.0 (C:\Users\emily\git\magrep)
    Finished dev [unoptimized + debuginfo] target(s) in 1.66s
     Running `target\debug\magrep.exe -a ..\..\Downloads\archive-20230214153535-daa49fde73e802447b521186ba95c8af.tar.gz fish`
path: ..\..\Downloads\archive-20230214153535-daa49fde73e802447b521186ba95c8af.tar.gz, pattern: fish
*************
"https://tech.lgbt/users/Emily_S/statuses/109295945016110296" : "So let's dig in to some usecases: 
Say your country has a system to license vessels to fish in your waters, but also can't really afford a big navy. With ais data you can track vessels in your waters and deal with any that look like they are fishing.
Boats that are fishing move very differently to a cargo ship trying to get to the next port, so even if it lies about being a fishing vessel you can tell."
*************
"https://tech.lgbt/users/Emily_S/statuses/109295960022052842" : "What about verification that the fish you are selling in your super market is caught legally? This is a good one because it's just making sure vessels are doing what they say they are, so they will want to keep their transmitters on to prove they are good and can sell their fish.
You can also keep track of your cargo containers and know where the shipments are out side of what the shipping company says. That snafu in the suez was fun to watch."
*************
"https://tech.lgbt/users/Emily_S/statuses/109451801563109639" : "@delibrarian good question. For the purposes of story telling I'd go with nothing, but it acting like a fish bowl where they can't come here and we can't leave would be interesting."

Apparently I’ve tooted the word fish three times. And thats our problem solved. Now there is a bunch of stuff that can be improved here. Error handling being the big one. We are unwrapping a bunch of things here, which we should likely be checking. Also it could happily be broken down into more reusable functions, but if I ever actually need this code again I’ll do that. Right now it would be a waste of time, and I need to go toot “some pattern”.

If you want to see the entire thing the code is on github Any questions or comments you can find me on mastodon.

Turning an SVG into a PDF in rust

Tue, 17 Jan 2023 12:11:00 +0000

I have some code that creates pages of shapes. My wife uses this to create products for her etsy shop. I originally wrote this in java many years ago. Some time last year I decided to rewrite the entire thing in rust. There were a number of reasons for this. Mostly that I’d learned a heck of a lot since I designed the original architecture and wanted some features that would not be possible with out rewriting most of it any way.

So the rust rewrite happened. Everything was wonderful and lovely. Ish. I did have to build my own SVG classes to do what I needed. Thankfully the quick_xml library made this reasonably easy. While I don’t mind creating readers and writers for SVG files, PDFs are another kettle of fish.

In java land I used the batik-transcoder which worked wonderfully. In rust I started out using the aptly named svg2pdf library and thought I was done.

Unfortunately I eventually discovered, when we went to upload a file of pdfs to etsy that was too big, that the svg2pdf library uses usvg under the hood. This is a wonderful library that simplifies the svg down as much as possible to make later operations have to deal with the smallest subset of svg as possible.

One of the main things it does is convert everything to a path object. This does remove the need for a lot of special handling, circle? its a path, rect? its a path, text? its a path. Wait what? Yeah, it converts every single character of text into a path object. The watermark we put in the background to stop people reselling our files suddenly made the files 5x bigger.

I know PDFs can handle this, the original java version was handling text in SVGs nicely after all. So back to searching for a library to do this for me. Unfortunately I couldn’t find one. I did find printpdf which is a more manual way of creating a pdf, and has support for SVGs in a feature flag. Unfortunately it uses usvg and svg2pdf under the hood to do the import.

However it does also give me access to the pdf generation, so I could create a svg without the text in it, and handle the text separately. This is what I ended up doing. Though it wasn’t with out its wrinkles as I’ll describe now.

Step one was to do a depth first search of the svg document tree and find all the text elements. This was easy enough, a recursive tree walk function solves that. Due to rusts memory safety its better to create a new tree to go along with it, so copy everything thats not a text element over to the new tree and add to a vector of text elements. I’m not going to show an example here because its using my svg library and won’t apply anywhere else.

Then using the printpdf library we need to create a new document.

    let (doc, page1, main_layer) = PdfDocument::new(
        "FaerydaeStitches Shape File", 
        Mm(convert::pixels_to_mm(page.width, page.dpi)), 
        Mm(convert::pixels_to_mm(page.height, page.dpi)), 
        "layer1"
    );
    let main_layer_ref = doc.get_page(page1).get_layer(main_layer);

Note: we also need to get a PDFLayerReference to modify our layer, so we have to look that up after creating it.

Now we can go through and create our text elements. Wait no, to create a text element we need to tell the library what font we are using. First lets go through and create a cache of fonts we need. Now SVGs can’t embed a font, but a pdf can, so we can make sure that our fancy font goes with our documents. This will make them bigger but will also make sure they look right everywhere. I used the font_kit library to handle looking up the path for a system font. I’m only using the SystemFontSource bit not any of the rendering.

fn create_font_cache(doc : &printpdf::PdfDocumentReference, texts: &Vec<TextElement>) -> Result<HashMap<String, IndirectFontRef>, Error> {
    let mut result = HashMap::new();

    for te in texts {
        if !result.contains_key(&te.font_name) {
            let font_path = find_font(te.font_name.clone())?;
            let font = doc.add_external_font(File::open(font_path)?)?;
            result.insert(te.font_name.clone(), font);
        }
    }

    return Ok(result);
}

fn find_font(font_name: String) -> Result<String, Error> {
    let handle = SystemSource::new().select_by_postscript_name(font_name.as_str())?;

    match handle {
        Handle::Path { path, ..} => return Ok(path.to_str().unwrap().to_string()),
        _ => Err(Error::FontMemoryFont)
    }
}

Ok Now we have the fonts we can create the text. Awesome…

    layer.begin_text_section();

    let font = font_cache.get(&text.font_name).unwrap(); // we created the font cache from this list of text objects. We know this will exist.
    layer.set_font(font, text.font_size as f64);
    
    // set text colours
    layer.set_fill_color(convert_colour(text.fill.as_str())?);
    layer.set_outline_color(convert_colour(text.stroke.as_str())?);

    layer.write_text(text.text.clone(), font);

    layer.end_text_section();

Please note: I skipped over the part where we got the colours and font names from the text elements style attributes. Its all just text parsing, convert_colour is a helper function to go from #0CAB0AFF type hex colour codes to printpdf’s rgb colour objects.

Aaaaahhhhh, why is our text at the bottom of the page not the top? PDF’s (0,0) origin point is in the bottom left corner of the page. SVG’s is in the top left. So our text won’t appear where we want. To solve this we need to create a TextMatrix to feed our layer. This can just be a translation or rotation or both.

I know I’ll need to do a rotation later so lets solve all of this at once. But, the svg rotation isn’t applied on the text elements. Its applied on a group above the text elements. So when we walk the tree we are going to need to keep track of the current rotation as we go, so we can apply the right rotation values to the text elements we create.

The next fun thing is the coordinates applied to the text element in the svg get transformed by the group above them. So they think they are printing to a normal x,y coordinate plane, but that entire plane then gets rotated by the group. This doesn’t happen in the PDF so we need to undo it. Go from group coordinates to page coordinates. Functionally this is rotating the axis, which is reasonably easy to do with some trigonometry

// function for finding the coords in the page axis when there has been a rotation applied to the coords
fn get_effective_location(rotation:f64, x:i32, y:i32) -> (i32, i32) {

    if rotation == 0.0 {
        return (x, y)
    }

    let rad_rotation = rotation.to_radians();

    let new_x = ((x as f64) * rad_rotation.cos()) - ((y as f64) * rad_rotation.sin());
    let new_y = ((x as f64) * rad_rotation.sin()) + ((y as f64) * rad_rotation.cos());

    let rx = new_x.round() as i32;
    let ry = new_y.round() as i32;
    return (rx, ry)
}

I’m not going to explain how to get here. I don’t want to write another couple of thousand words. It just does what we need and gives us a point in page space as though the group was not there.

Now we need to tell the pdf that this is where we want our text. So we add a TextMatrix to our code from before

    layer.begin_text_section();

    let font = font_cache.get(&text.font_name).unwrap();
    layer.set_font(font, text.font_size as f64);
    // set text rotation
    
    // set text colours
    layer.set_fill_color(convert_colour(text.fill.as_str())?);
    layer.set_outline_color(convert_colour(text.stroke.as_str())?);
    
    layer.set_text_matrix(TextMatrix::TranslateRotate(
        Mm(convert::pixels_to_mm(text.x, page.dpi)).into_pt(),
        convert_y(text.y, page).into_pt(),
        convert_angle(text.rotation)
    ));
    
    
    layer.write_text(text.text.clone(), font);

    layer.end_text_section();

The x axis is still the same values, just at the bottom of the page rather than the top. We do need to convert the y axis though. Reasonably straight forward, take the existing y value away from the height of the page, with a bunch of unit conversions thrown in for good measure. My SVGs operate in pixels, the printpdf library likes millimeters (its Mm objects)

fn convert_y(y:i32, page: &Page) -> Mm {
    let page_height = convert::pixels_to_mm(page.height, page.dpi);
    let y_mm = convert::pixels_to_mm(y, page.dpi);
    Mm(page_height - y_mm)
}

The other wrinkle is because of the flipped axis the rotation angle is different. Instead of positive values rotating clockwise from the top of the page, positive values now rotate counter clockwise from the bottom of the page, again easy to solve, multiply by -1

fn convert_angle(angle:f64) -> f64 {
    angle * -1.0
}

Hurrah! Now our text is the right font, in the right place and going in the right direction. Excellent.

Now we just need to add our svg with out the text elements.

    let svg_string = svg_filtered.to_pretty_string();
    let pdf_svg = Svg::parse(svg_string.as_str())?;

    let svg_transform = SvgTransform{
        translate_x: None,
        translate_y: None,
        scale_x: None,
        scale_y: None,
        rotate: None,
        dpi: Some(page.dpi as f64),
    };

    pdf_svg.add_to_layer(&main_layer_ref, svg_transform);

First we turn our svg object tree into a string so it can be parsed by the pdf library. This is the down side to using our own objects for this, but at least we can do everything we want. Next we parse the string, this is where the text would get converted into paths if we hadn’t removed it all.

We set up a transform, mostly this is just need to set the dpi. If we don’t do this it defaults to 300dpi which is almost certainly wrong. While my code handles any dpi the most common uses are 96 and 72 dpi.

Finally we add the svg to our layer. Tada!

Oh we should probably save our pdf too.

    Ok(doc.save(&mut BufWriter::new(File::create(path)?))?)

Ok now we are finally done. My PDF’s have gone from 1mb ish to 200kb ish. Perfect.

May all this rambling be of no use to you. There is a bunch of stuff I didn’t cover that would be possible to solve. I only handle rotation transforms, not any other kind. It would be easy enough to do so, but my code does not ever generate any thing but rotations. It also doesn’t handle style cascading from group elements and probably a thousand other things. This is one of those problems that the more you look at it the harder it gets.

I’m glad I got this far and am very happy to put it down now. As usual this is as much documentation for my self as any one else.

Explorers End

Sun, 04 Dec 2022 12:17:00 +0100

This work of fiction started as a idea tooted on mastodon. Hope you enjoy.

“Reload complete.” The log message in its mind stated. It often wondered what happened to it during a warp jump before its mind state was automatically reloaded when it dropped back into normal space. It knew and understood why the protocols were in place. A machine like its self corrupted by the warp was not something to release on the universe.

‘Right, where are we?’ it thought to its self firing up the scanners. A mid sequence yellow star. Usual cloud of planets and asteroids. Its calculation matrix running hot as it calculated all the different orbital vectors. A few big gas giants. Ooh that one is nicely in the liquid water zone.

Wait… what is going on with the warp field there? Did that moon really form around a fissure? This is going to make an interesting report. Lets have a look at that planet. It has life? Seriously? With those levels of warp flux? How?

“Running evolution models. Error warp flux out of range.” It spent a while re-writing its own simulation models to handle the task. The results continually too terrible to be right. Except no, eventually there were no more bugs in its code. It was just going to be that bad. Of course it couldn’t tell the future exactly. It was an ensemble of predictions. But even its ‘best’ prediction was well inside the levels that authorized it to act.

Now it needed to work out how. The standard operating procedure was to use a warp fissure bomb to remove the planet. Unfortunately with the existing fissure in the nearby moon that couldn’t be risked. The fall back was old school orbital bombardment. That was out of the question. Spending that long in orbit near the fissure? Corruption was guaranteed.

An asteroid would do it. It started to run simulations. Selecting the best option from the millions of them in the system. That one. Small enough to be moveable. Big enough to guarantee extinction of all life on that planet. Perfect.

The asteroid set on a new course it set about mapping and recording everything for its report. The report encoded into the same crystal substrate it used for its mind state backups.

The process for sending the report home was like making a warp jump. It encoded its mind state backup. Set the automatic systems to open a warp fissure and fire the carefully packaged crystal towards home. While the warp was open it’s own mind was shut down. Functionally dead. Once the warp was closed the automated systems restored the mind state backup to bring it back to life. Thus avoiding any risk of corruption.

In the early days they had stayed awake through the warp. But then they’d had problems with corrupted minds messing with their own backup states while in warp. This was safer.

The asteroid finally approached its target, it watched carefully to make sure its work in this system was done.

Its sensors picked up a fluctuation in the warp fissure. Something reached out of the warp, through the moon. What ever the hell it was it touched the asteroid. It watched in alarm as the asteroid split in two, one half skimming past the atmosphere. The other slamming into the planet.

Quick calculations as the sky darkened and lava spilled across the surface. It wasn’t going to be enough. There was still an unacceptably high chance of life surviving. Wait … what the hell is that warp entity doing? Oh no. Its coming. Could it make a warp jump? Would the entity follow it? Can’t stay here. Have to try it.

“Emergency warp jump. Backup in progress.”

Plotting Geospatial Data

Fri, 02 Dec 2022 22:23:00 +0000

As useful as data is on its own, its really important for people to be able to see it. For a recent project I needed to show some World Cover images with the area of interest marked, along side the table of results. People respond better to being able to see it. This how I did that.

Lets start with a list of tools we used:

GeoPandas
Rasterio
MatPlotLib

First we need to open our image data. I’m going to skip over fetching the data, there are pretty good examples on the World Cover data page we have a variable data_path that is a filesystem path to our cropped tiff file. A word of warning about the WMS server that has the world cover data is not suitable for pixel counting types of analysis. There are colour mixing artifacts between the classes.

    import rasterio
    ds = rasterio.open(data_path)
    array = ds.read()

A fun thing about the world cover data is that it is a classification. The input contains a single band where each number means a different land cover type. Things like “forest” or “urban” etc. This isn’t your traditional colours, so plotting it as is won’t give us a nice result, just shades of dark gray. We need to create a colour map in MatPlotLib parlance.

    import matplotlib.colors as colors
    from matplotlib.colors import ListedColormap

    world_cover_colours_2021 = [
        '#FFFFFF', '#006400', '#ffbb22', '#ffff4c', '#f096ff', '#fa0000', 
        '#b4b4b4', '#f0f0f0', '#0064c8', '#0096a0', '#00cf75', '#fae6a0' 
    ]
    world_cover_codes = [
        0, 10, 20, 30, 40, 50, 60, 60, 70, 80, 90, 95, 100
    ]
    world_cover_cmap = ListedColormap(world_cover_colours_2021)
    world_cover_norm = colors.BoundaryNorm(world_cover_codes, 12)

Next we need our area of interest to plot over the top of the image we will create from the land cover data. Ours was contained in a pair of GeoPandas Data Frames called location_frame and area_frame. How we got those is outside the scope of this post.

Now that we have all our data and our colour maps setup we can start to plot our data. For this we are going to use MatPlotLib. This is a graphing library. What we are about to create would be a very bad chart, but it has all the tools we need to do it so it makes it easy for us. First we need to create a figure and axis for our “graph”. To do that we need to know the size we want. One wrinkle is this might not be square. So we work this out as a ratio of the size of our source image.

    import matplotlib.pyplot as plt

    dpi = 96

    ratio = array.shape[1] / array.shape[2]
    width = 400 / dpi
    height = (400  * ratio) / dpi

    fig = plt.figure()
    fig.set_size_inches((width, height))
    ax = plt.Axes(fig, [0.0, 0.0, 1.0, 1.0])
    ax.set_axis_off()
    fig.add_axes(ax)

The set_size_inches method on the figure uses inches funnily enough so we need to know the Dots per inch (DPI) we want to be able to calculate the size we need. Once we’ve set the size of the figure, which will be the size we want as a ratio of our data images shape. We create an axes, a matplotlib figure can contain many charts so we have to create the axes separately. We tell it that this one will cover the entire image. Just before we add the axis to the figure we turn off the axis on the axes. This stops the markers down the side of the image.

Now we can start plotting the data. We will do the area of interest polygon first.

    area_frame.to_crs("EPSG:4326").plot(ax=ax, facecolor='none')

Three things to note here. First the conversion of the CRS to match the source image data. If you don’t do this things won’t line up in the plot. MatPlotLib is not a geospatial library and doesn’t understand coordinate systems at all. Second is passing in our Axes we created earlier. If you don’t do this a brand new figure will be created inside this function and you won’t get the output you expect at the end. Third the facecolor setting of none makes the polygon unfilled. If you want a shaded area you can set it to a semi transparent colour.

Next is the center point with an almost identical line of code.

location_frame.to_crs("EPSG:4326").plot(ax=ax, c='r')

This geo data frame contains a point so will only show a single mark on the result. The c='r' bit is a short hand for setting the colour of the point to red.

Next we need to plot our image. To do this, there is one last thing we need to set up, so that our image ends up in the right place. Just like the polygons needing to be in the same CRS we need to tell matplotlib where our image sits. To do that we need the bounds of the image from the rasterio data source.

    bounds = (ds.bounds.left, ds.bounds.right, ds.bounds.bottom, ds.bounds.top)
    ax.imshow(array[0,:], cmap=world_cover_cmap, norm=world_cover_norm, extent=bounds)

The imshow method on the Axes adds a bitmap image from the provided 2d array. The output of a rasterio array is always 3d so we have to slice off one off the first axis. Then we provide the colour map and normalization objects we set up earlier. Finally we provide the bounds to move our array into the right place on the chart.

With all of our data plotted, all that is left to do is save the figure, making sure we tell it our dpi. This will save the resulting chart to disk at out_path

    plt.savefig(out_path, dpi=dpi)

I’ll freely admit I’m mostly writing this so I have the notes next time. Hopefully this helps you. Even if you are future me.

SVG

Wed, 23 Nov 2022 06:45:00 +0000

lets talk about SVGs as an image format.

You may have heard of them called a vector format. This means that instead of defining a grid of pixels and having about what colour every pixel is, it defines the features of the image.

There is a red line from 12,45 to 26,78 The text “hello” is in front ariel, 14pt and blue etc.

The advantage of this is you can scale the image with out losing any quality. Hence the name Scalable Vector Graphics

If you double the size of the image you still have a red line, the edges of it will still be just as crisp, unlike if you double the size of a raster image, you end up with 4 pixels for every one in the original, it starts to look blocky unless you do some magic maths to add extra information where there wasn’t before. AKA, make stuff up.

Internally an SVG file is an xml document. Just like any webpage you view its a series of “tags” which tell your computer what to display. There are a lot of similar concepts too, both have text, links, ids, even CSS.

SVGs have lines, rectangles, circles. You can draw almost anything as an SVG, but it can be a little more complex than painting a raster image.

Instead of div tags to group bits of an html document you have g or group tags. This is almost always just an ease of use thing rather than actually meaning anything in the picture. You can apply transformations and styles at the group level though to make your life easier. “rotate all of these things by 60 degrees” or “make all the lines red and fill with green”

Where the SVG language gets away from xml is in the definitions of paths and lines. Paths are a series of points joined by lines or curves. Once you start to deal with curves in computers things start to get complex. Mathematically there are a bunch of different ways to define curves. Some based on circles, some based on Bezier curves, which a bunch of helpful shortcuts.

If you want to know more Mozilla’s documentation on SVG’s is excellent.

Like HTML I’d argue that anything more complex than a simple image, you should not be hand crafting it. Inkscape is an excellent and free opensource tool to edit SVG files. Because they are basically an XML document internally there are libraries available for almost every programming language that exists. Even if you just end up editing the raw XML

One thing to keep in mind when working with SVG images is eventually it will be pixels on a screen or on a page. When you get there you will need to define your Pixels per inch. There is no defined standard, Inkscape uses 96ppi by default. The Cricut used 72. This causes no end of headaches. You can use physical units inside an SVG if you want. You can also set the size of the image, e.g. to the size of an A4 page.

While I said they will be pixels on a screen, the other really useful thing about SVGs is because they are made up of paths mostly they can be fed into things like laser cutters, plotters, and draw knife machines like the Cricut. The thing to watch out for though is the level of detail. You usually have a minimum feature size so you want to make sure your detail isn’t below that.

COG - Cloud Optimized Geotiffs

Tue, 22 Nov 2022 06:45:00 +0000

COGs, Cloud Optimized GeoTiffs, not the little things in clocks and gearboxes.

Why do we need yet another data format? What is the point? Well dear #geospatial people. Lets dive in.

First of all, lets quickly go over what a GeoTiff is. A tiff file is a type of image. Really old format designed for fax machines, but it has an advantage of not requiring only red, green, blue or alpha bands like a lot of other image formats do so you can happily have a 12 band sentinel 2 image as a single tiff. It also handles 16bit values and floating point values.

A geo tiff is a tiff file with a bunch of added geospatial information, so it can be placed in the world easily.

The one downside to a GeoTiff is because its a tiff and tiffs are very flexible you usually have to read the entire thing to understand it. This is fine with a fax machine, but with a 4gb multi band image its annoying.

A COG is a GeoTiff that has been setup in a defined way. all of the metadata will be at the beginning and the bands will be laid out in a known order.

Why is it being in a known order useful? Because of a completely unrelated bit of magic tech. The HTTP range request. Normally when you use the web your browser issues lots of “get” requests, which ask for files. you usually want the entire thing. there is no point displaying half the mastodon logo usually. With a range request you can ask the web server for bytes 45 to 2367 of a file and it’ll just get that. Assuming it supports range requests.

Combining this with a COG we can only pull down the bits of that 4Gb image that we actually need. Say we only want a small bit of the image to show one house on a map, or we only need two of the bands.

We can issue a couple of range requests to get the header information from the COG, and then work out where the data we want actually is and issue another range request for just that bit. Now we’ve not downloaded 3.9Gb of data we didn’t need we’ve just pulled out the 100mb we do.

Cloud storage systems like Amazon S3 or azure Blob storage support these range requests already, and their ease of use with COGs is why they are called “Cloud Optimized GeoTiffs”

They are designed to work on the cloud, but they don’t have to be on the cloud. They work just like an existing GeoTiff, so you can use them anywhere.

That is the magic, they don’t break anything that already works with a GeoTiff input, but they let you do new fun things.

If you want to know more about tiff file internals I gave a talk at Foss4guk a couple of years go on the the topic

Big O for geospatial people

Fri, 18 Nov 2022 08:17:00 +0100

Today’s topic is Algorithmic complexity for #geospatial people.

Or what the heck is Big O any way.

If you’ve been around programmers for any length of time you’ve probably heard of Big O. (settle down nsfw folk, this is about algorithms and programming not that)

The short version is its a way of describing the worst case runtime of an algorithm with regards to the amount of data. Given n bits of information what’s the order of magnitude of the worst possible run time. O(n)? O(n^2)?

Say we have a set of latitude and longitude points. We need to convert them to a different coordinate system. This requires us to visit every point and do the transformation function.

It doesn’t matter how many points there are we have to visit each one once. So this is described as O(n) this is linear, the more points we have the longer it takes, but doing 10000 takes 10 times longer than doing 1000, and 1000 takes 10 times more than 100 etc.

It also is explicitly always the worst case runtime. So if you have an algorithm that is finding the the first point in a cloud inside an area, it can stop and exit as soon as it finds one, but if none of the points are inside the area then it will have to check every point. So while on average it would be n/2 comparisons you would always have a big O notation of O(n)

If we have to find the pair of those points furthest away from each other. We could write some code that goes through each point and then calculates the distance to every other point to find the longest distance.

The version of this where you check every point every time would require O(n*n) operations or O(n^2) the more points you have the worse things get in a big way.

But we can improve this algorithm significantly, the distance between points doesn’t change depending on which way round you compare them. So we don’t need to check every point every time, we can skip the ones we’ve already done. However this only makes it O(n*(n-1)/2) One of the quirks of Big O notation is that you always get rid of any numbers or fixed terms and simplify the expression down to only the largest terms to give a rough order of the problem. So even the improved version is O(n^2)

Another common order is O(n*log(n), this is often seen in recursive merge algorithms, and things that take divide and conquer approaches. The graph below shows nicely how this is better than O(n^2) but not as good as O(n)

The idea of Big O notation is to allow you to compare different approaches to problems at a high level and what your scaling difficulties are going to be. An O(n^2) solution might be fine, if you know that n is only ever going to be a small number. The point is to have the tools to describe this kind of thing to each other.

With thanks to @adamb@aus.social for pointing out an error with this post.

InSAR

Thu, 17 Nov 2022 08:17:00 +0100

Today I’m going to do a thread on something I want to learn more about but have used a little in the past. So be warned this topic might not be 100% accurate but is my understanding.

Interferometric synthetic-aperture radar or InSAR for short because everyone loves acronyms.

Its a way of measuring how much something has moved between two radar “images” the something can be a building, a dam, a mountain. If you’ve ever seen those “fringe” maps with bands after an earthquake that’s this.

To do this you need two things: 1: a radar satellite that has a very well known orbit and orbits consistently over the area of interest, you always need to be “looking” at the target from the same direction. 2: A time series of radar images. You need two images to compare, you cant do it with one.

My understanding of the way it works is by measuring the phase of the returned signal and comparing it between the two images. Movement changes the distance so the returned wave isn’t the same.

Should probably do a thread on SAR at some point but that’s for another day.

With a high resolution radar satellite you can measure individual parts of a structure and see if any part is moving compared to the rest.

The amazing thing about this technology is you can measure movements of fractions of mm. (this is the part that makes this seem magic to me)

So I’ve mentioned measuring buildings a few times, but how is this use full. Say you own a bridge, over time that bridge needs maintenance, but its expensive to send people out regularly to look at your bridge and tell you if you need to do anything. Also potentially dangerous.

For my day job we worked on this with the Canadians. Where currently they get someone to walk across the bridge with a gps tool to measure where the bridge is. At best they get this result every 6 months.

So the system we build is able to look at radar images of the same bridge every couple of weeks (the orbits of the satellite mean it takes a couple of weeks or so to be back in the same place)

It also gets way way more data due to the resolution of the satellite we were using. The screen shot shows the radar result. The colour’s show how much each point on the bridge has moved.

But don’t worry its meant to move like that.

Its a big metal structure. Metals change size as the temperature changes. On something the size of a bridge we have to take that into account. Meaning you need the local weather as well or you are going to unnecessarily panic.

The maths involved between how you go from a pair of radar images to collection of points like this is where my knowledge ends. I helped load the results into the application and render them on the screen.

So in conclusion InSAR is awesome and something you should keep in mind for your remote monitoring needs.

If you have any good articles on how the maths inside it works I’d love to see them, please send me a link on mastodon.

AIS

Wed, 16 Nov 2022 06:27:00 +0100

Today we’ll talk about ais data. This is another post that started out as a thread over on mastodon

Sorry it’s another acronym. Automatic Identification System. It’s original purpose was for ships to be able to tell each other where they are and which way they are going to avoid collisions. It is a little radio box on the boat that sends periodic messages saying “my Id is 123456, I’m at this lat and long, I’m going this way at this speed” it also sends out other messages like “my Id is 123456 my name is see the wind, I’m a bulk grain transporter and 30m long”

There are many message types that can mean all sorts of different things. There are special ones for lighthouses, search and rescue aircraft, etc.

By maritime law any vessel over a defined size must have an aid transmitter on board. The fun thing is that these transmissions can be picked up by anyone. The transmissions are mostly line of sight (it’s radio, that gets weird so I’m not going to go into it too much) but satellites are always over head so can see everything.

It’s possible to buy a feed of the transmissions of every boat on the planet, it’s quite a lot of data to process, but honestly only just into big data territory.

The problem is the data is noisy. Very noisy. Because the transmitters are on the vessels, the owner can do what they like, turn it off? Easy, unplug the power. Change the id and vessel details? Also easy. If someone is doing something they shouldn’t they will often just sail out of port and turn off the transmitter.

Then there is the fun that because the messages are resent pretty often it doesn’t matter if one or two are corrupt so there is very little error correction. Bit flips are possible and more than one is common, if that bit is in the position parts it can make a vessel look like it’s jumped across the globe for a second and then jumped back. That’s before hundreds of people put their Id in as 111111 or 123456

So let’s dig in to some use cases:

Say your country has a system to license vessels to fish in your waters, but also can’t really afford a big navy. With ais data you can track vessels in your waters and deal with any that look like they are fishing.

Boats that are fishing move very differently to a cargo ship trying to get to the next port, so even if it lies about being a fishing vessel you can tell.

What about verification that the fish you are selling in your super market is caught legally? This is a good one because it’s just making sure vessels are doing what they say they are, so they will want to keep their transmitters on to prove they are good and can sell their fish.

You can also keep track of your cargo containers and know where the shipments are out side of what the shipping company says. That snafu in the Suez was fun to watch.

Want to make an economic estimate of how much stuff is getting to a country? Reasonably easy to get a list of vessels who’ve arrived at ports and how big they are. From a humanitarian point of view I’ve helped figure out how much fuel was getting into Yemen.

Ais is an awesomely powerful dataset but also not easy to work with. The source data comes in as basically points and then you have to construct routes and so on from that.

If you want to have a look at some data check out marine traffic. Who have a nice web interface. Also check out Anita Graser’s tutorial on processing ais data

Normalized Difference Vegetation Index

Tue, 15 Nov 2022 07:38:00 +0100

Today’s topic: NDVI. This is another post that started life as a thread over on mastodon.

What the heck is that? It’s an acronym (sorry) it stands for normalized difference vegetation index. Short version it’s a proxy for how much healthy plant life there is.

It will be a number between -1 and 1. -1 being a clean concrete slab or similar. 1 being an impossibly healthy rainforest canopy.

The way this one works is comparing the difference between absorption of near infrared light and red light.

Because plants leaves are green, this means their leaves absorb all the red and blue light. The atmosphere just about let’s the same amount of red and near infrared light (nir). But plants don’t absorb the nir light. See how the line drops off in the chart below.

So by comparing the values we can see how much healthy plant life there is.

We can convert this into a number with a generic formula. (A-B)/(A+B) This ends up with a ratio of one to the other between -1 and 1. This is where the normalized part of the name comes from. There are many versions of this formula for finding all sorts of different things, water, water turbidity, etc.

The great thing is this is all very easy to calculate on satellite images. The formula applies to each pixel independently. So you do them all in separate threads if you wanted. Making this very very fast with tools like dask or spark. Even numpy can do this pretty dang quick on most reasonably sized images.

As you can see in the image above its pretty easy to find fields or parks compared to buildings and structures. Its used for all sorts of applications from crop monitoring, deforestation, city growth.

Any questions feel free to ask on mastodon

180 Degrees

Mon, 14 Nov 2022 19:48:00 +0100

Note: This post started out as a thread on mastodon. Please follow me there if you’d like to see this sort of thing early

Lets start talking about something I wish I didn’t know as much about as I do.

The 180 degree line! +/-180 degrees longitude.

It is not the same thing as the international date line, that’s defined politically which lots of wiggles through the pacific islands.

The 180 degree line only crosses two countries, Russia and Fiji.

This means that you can go from +180 degrees to -180 degrees with a single step.

However most software that deals with maps doesn’t understand this. “Not many people live there, it doesn’t matter” people say. it is not unusual to end up with polygons going the wrong way around the world, when you ask for Fiji’s borders.

The other quite common solutions is to split the polygon in two, so there is a tiny sliver down the middle of Fiji that doesn’t apparently belong to Fiji, at least according to the map.

The way the Fijians solve this is they have their own National Coordinate system that just covers the islands.

The problem of course is converting into and out of this coordinate system, which you almost invariably need to do to plot things on a web map, as they commonly use a coordinate system based on latitude and longitude called WGS 84

For real fun you need to get a satellite image that crosses over the 180 degree line. The most common software for processing Sentinel 1 radar scenes attempts to allocate an array of 10m pixels all the way around the entire globe. This will almost always fail, due to running out of memory.

oh and if you think this sounds bad you should try working at either of the poles. Tip of the hat to my collogues who work with the Antarctic surveys

Announcing a new me

Fri, 06 Sep 2019 17:00:00 +0000

Hi. My name is Emily. No I’ve not taken over this blog for the day. I’m still the same author but a while ago I came to realize I’m a woman so my old name didn’t fit any more.

Don’t panic I’m still doing mostly the same things as before. Just with less background noise in my head so things are a lot better.

Family, friends, and colleagues reading this

Waves. Don’t panic. Call me Emily, refer to me as her and we will continue to get along fine.

Still me, still have the same interests. May be interspersed with more politics and trans rights bits but otherwise business as usual.

People who know me from github

If you use any of my GitHub projects that account will be changing over shortly. Import paths may change. In theory they should redirect everything though so we will see how that goes.

People who’ve never met me

Waves. Hello. Not sure why your here but some of my other posts can be found on the home page may be more interesting.

Frequently asked questions

Q: What does your wife think?

A: Confused but supportive and trying her best. I’m extremely lucky.

Q: What does your son think?

A: Its pretty much the new normal for him. He is young enough that this is just something that happens.

Q: How has work been?

A: Utterly fantastic. They have been better than I could have dreamed of.

That probably covers everything you need to know for now. So in summary: My name is Emily, I’m Trans and doing pretty well.

Emily

PS: Thank you to all the amazing people who have helped me over the years as I’ve been dealing with this.

Golang Resparse

Sat, 08 Dec 2018 13:28:33 +0000

A little while ago I found my self needing to be able to parse screen resolutions when generating some images in a golang program. I created a library to do this and had a bit of fun optimising it. The result is open source on github. It is a very simple library with a single function but I thought it might be interesting to walk you through the process.

What we need to build is a function that takes a string like “1080p”, “800x600”, or “4K” and returns a width and height value. There should also be an error in the return type just in case we can’t parse the string.

This is going to be a very basic walkthrough of what I did so if you know a bit of go you probably can skip this one. Or skip to the sections on benchmarking and optimization.

Assumptions

You have a working golang 1.11+ installation on your machine.

Setting up the project

Open a terminal and browse to where you want to put the project. It does not have to be in your GOPATH any more. Create a directory and move into it.

mkdir resparse
cd resparse

Set up the module with the go mod tool. It may complain you need to set a GOPATH if you don’t already have it set. As we are not inside our GOPATH we need to tell the module tool what the package path of our project is.

go mod init github.com/wselwood/resparse

Building the code

Now open up your favourte editor. (I use VS code with the excellent go plugin) and create a new file called resolution.go and create a function place holder for our parsing function.

package resparse

func ParseResolution(in string) (int, int, error) {

    return 0, 0, nil
}

This function takes in the string and returns x, y, and error values that match up to the input string. Now that we have created this place holder we can create a test harness so that we know if the code we are writing is working. Create a new file called resolution_test.go. We will use a table driven test for this as it will make it easier for us to add new cases as we think of them.

package resparse

import (
	"testing"
)

type testCase struct {
	input string
	x     int
	y     int
	err   string
}

var cases = []testCase{}

func TestBasicParse(t *testing.T) {

	for _, test := range cases {
		t.Run(test.input, func(t *testing.T) {
			x, y, err := ParseResolution(test.input)
			if (err != nil && test.err == "") || (err != nil && err.Error() != test.err) {
				t.Errorf("wrong error from parsing \"%v\" got \"%v\" expected \"%v\"", test.input, err.Error(), test.err)
			} else if x != test.x || y != test.y {
				t.Errorf("got wrong result from parsing \"%v\" got (%v,%v) expected (%v,%v)", test.input, x, y, test.x, test.y)
			}
		})
	}
}

This is a reasonably long block of code so lets go over it. First we define the package, and import the built in testing package. Next we define a testCase struct that holds an input string and the expected output values. The next line defines an empty list of test cases. Finally we get to the test function its self. Inside we loop over the testcases.

The next bit runs each test as a sub test. This is a useful ability of the go test library as it gives the option to run each of the sub tests in parallel which can speed up the test time significantly.

We can add a couple of basic test cases and then get on with the function its self.

	{"", -1, -1, "could not parse \"\" as a resolution"},
	{" SVGA", 800, 600, ""},
	{"WSVGA", 1024, 600, ""},
	{"800x600", 800, 600, ""},
	{"1600|1200", 1600, 1200, ""},
	{"dgsfgd,4000", -1, -1, "could not parse \"dgsfgd,4000\" as a resolution"},

This will give us some where to work. If you run this test now it will print a lot of failures.

wselwood@DESKTOP:/git/resparse$ go test .
--- FAIL: TestBasicParse (0.00s)
    --- FAIL: TestBasicParse/#00 (0.00s)
        resolution_test.go:31: got wrong result from parsing "" got (0,0) expected (-1,-1)
    --- FAIL: TestBasicParse/_SVGA (0.00s)
        resolution_test.go:31: got wrong result from parsing " SVGA" got (0,0) expected (800,600)
    --- FAIL: TestBasicParse/WSVGA (0.00s)
        resolution_test.go:31: got wrong result from parsing "WSVGA" got (0,0) expected (1024,600)
    --- FAIL: TestBasicParse/800x600 (0.00s)
        resolution_test.go:31: got wrong result from parsing "800x600" got (0,0) expected (800,600)
    --- FAIL: TestBasicParse/1600|1200 (0.00s)
        resolution_test.go:31: got wrong result from parsing "1600|1200" got (0,0) expected (1600,1200)
    --- FAIL: TestBasicParse/dgsfgd,4000 (0.00s)
        resolution_test.go:31: got wrong result from parsing "dgsfgd,4000" got (0,0) expected (-1,-1)
FAIL
FAIL    github.com/wselwood/resparse   0.008s

So lets go and start building our parsing function. We can start with the first test and check for empty or blank strings being passed in. We can trim the string and then check if it is empty. Returning an error if needed.

	work := strings.TrimSpace(in)

	if work == "" {
		return -1, -1, fmt.Errorf("could not parse \"\" as a resolution")
	}

Next we can create the look up table needed to handle named resolutions. Converting things like “SVGA” to 800,600,nil. Outside the function create a new struct type to hold the x, y pairs.

type res struct {
	x int
	y int
}

var known = map[string]res{
	"1080P": {1920, 1080},
	"WSVGA": {1024, 600},
	"SVGA":  {800, 600},
}

We can add more entires later, for now lets just keep those entires. Now we can look up our trimmed input string in the known map. We can use the second return value from the map lookup to know if we found a valid response. We will call strings.ToUpper() to make sure all the input values are easier to match against our known values.

	result, ok := known[strings.ToUpper(trimmed)]
	if ok {
		return result.x, result.y, nil
	}

If we don’t get a response from that we should try and split the string and then try and convert to a pair of numbers. Replace the old zero return with the following:

	splitStart := strings.IndexAny(trimmed, "Xx| ,*")
	splitEnd := strings.LastIndexAny(trimmed, "Xx| ,*")

	width := trimmed[:splitStart]
	height := trimmed[splitEnd+1:]

	x, err := strconv.Atoi(width)
	if err != nil {
		return -1, -1, fmt.Errorf("could not parse \"%v\" as a resolution", in)
	}

	y, err := strconv.Atoi(height)
	if err != nil {
		return -1, -1, fmt.Errorf("could not parse \"%v\" as a resolution", in)
	}

	return x, y, nil

Now if we run the test cases we should get a clear run. At this point we could call it done. But I’m not going to, first we are going to create a benchmark and then we are going to see if we can tune this function a bit.

Benchmarking

Before we can start doing any optimization we need to see how long this function is taking. Thankfully go has a reasonable benchmarking tool built into the testing library. If we hop back to our test file we can add a benchmark that uses the same test data as inputs.

func BenchmarkParseResolution(b *testing.B) {
	for n := 0; n < b.N; n++ {
		ParseResolution(cases[rand.Intn(len(cases))].input)
	}
}

The difference from a test is the use of the B object in testing and the function having to start with Benchmark. The B object has a value N which is the number of times to run the test. So we put that in a for loop for that many times. Then in each loop we pick a random entry from the cases list and run the ParseResolution function.

Go will then take care of the rest for us. If we run this we should see something like the following:

Running tool: C:\\Go\\bin\\go.exe test -benchmem -run=^$ github.com/wselwood/resparse2 -bench ^BenchmarkParseResolution$

goos: windows
goarch: amd64
pkg: github.com/wselwood/resparse2
BenchmarkParseResolution-4   	10000000	       174 ns/op	      39 B/op	       1 allocs/op
PASS
ok  	github.com/wselwood/resparse2	2.068s
Success: Benchmarks passed.

This tells us it ran with an b.N value of ten million and it took on average 174 NanoSeconds for each loop, which allocated 39 Bytes in one allocation. This is pretty good and for something like a command line tool where this is only called once at start up it is well within reasonable bounds. But lets not be reasonable, lets see what we can do here.

First thing to do is see what it is actually doing for those 174 Nanoseconds. So we are going to run the benchmark with the cpu profile enabled. I use the shell built into vs code for this. There is something about the Windows Subsystem for Linux (WSL) that does not play nice with the profile tools in go. It will run but you will have a completely blank profile file at the end.

PS C:\git\resparse2> go test "-cpuprofile=profile.pprof" -bench .
goos: windows
goarch: amd64
pkg: github.com/wselwood/resparse2
BenchmarkParseResolution-4      10000000               177 ns/op
PASS
ok      github.com/wselwood/resparse2   2.136s

You should now find a profile.pprof file in the directory. To open this up we can use the go tool pprof command. This can be accessed from the command line but it has an excellent web ui that provides some great visualizations. To enable the web ui you need to tell it an http port to open up. If you have used the built in http tools at all the format of this string should look pretty familiar.

PS C:\\git\\resparse2> go tool pprof -http=:8000 .\\profile.pprof

When you run this a web browser window should pop up with something like this:

This is basically a call graph of your benchmark with the calls that took more cpu time with larger and more defined arrows. The two unlinked trees off to one side are the test harness running in the background to keep track of things. Generally each go routine will end up with its own tree. You should be able to see that we spend most of our time creating errors, or in the string functions. Almost no time is spent in the map lookup or the number parsing.

Optimization

Given our ideal input we are going to end up looping over the string at least twice. At best once to do the separator finding at worst twice, and once for the ToUpper call. The call to Trim may or may not require any iteration of the string depending on if there are spaces. In the worst case where it is a completely blank string it will iterate the entire thing.

So we can avoid all of those if we make a single pass over the string and find, the start (after all the white space), the end (last character before all the white space), the start of the separator and the end of the separator. While we are at it we could also check if we need to upper case the string for the map look up. This should be a fairly simple loop with a bit of a state machine.

For a first pass we will just replace the trim and IndexOfAny functions with a single loop. The function becomes something like this:

func ParseResolution(in string) (int, int, error) {

	start := -1
	end := -1
	sepStart := -1
	sepEnd := -1

	for i, c := range in {
		if !unicode.IsSpace(c) {
			if start == -1 {
				start = i
			}
			end = i
		}

		if c == 'X' || c == 'x' || c == ',' || c == ' ' || c == '|' || c == '*' {
			if sepStart == -1 {
				if start != -1 {
					sepStart = i
				}
			} else {
				if sepEnd == -1 || sepEnd == i-1 {
					sepEnd = i
				}
			}
		}
	}

	if start == -1 || end == -1 {
		return -1, -1, fmt.Errorf("could not parse \"%v\" as a resolution", in)
	}

	result, ok := known[strings.ToUpper(in[start:end+1])]
	if ok {
		return result.x, result.y, nil
	}

	// if it is not in our lookup table then try and split the string
	if sepStart == -1 || sepStart == start || sepEnd == end {
		return -1, -1, fmt.Errorf("could not parse \"%v\" as a resolution", in)
	}

	if sepStart != -1 && sepEnd == -1 {
		sepEnd = sepStart
	}

	width := in[start:sepStart]
	height := in[sepEnd+1 : end+1]

	x, err := strconv.Atoi(width)
	if err != nil {
		return -1, -1, fmt.Errorf("could not parse \"%v\" as a resolution", in)
	}

	y, err := strconv.Atoi(height)
	if err != nil {
		return -1, -1, fmt.Errorf("could not parse \"%v\" as a resolution", in)
	}

	return x, y, nil
}

If we run the benchmark now we see:

Running tool: C:\\Go\\bin\\go.exe test -benchmem -run=^$ github.com/wselwood/resparse2 -bench ^BenchmarkParseResolution$

goos: windows
goarch: amd64
pkg: github.com/wselwood/resparse2
BenchmarkParseResolution-4   	10000000	       164 ns/op	      39 B/op	       1 allocs/op
PASS
ok  	github.com/wselwood/resparse2	1.955s
Success: Benchmarks passed.

You can see we have shaved 10 nano seconds per op off. We haven’t managed to stop it allocating. It is the ToUpper function that needs that. Now if we look at the graph again:

We can see that the ToUpper call is more defined now and the calls to Trim and IndexOfAny have gone away. Also the random number generation in our benchmark code has got more pronounced. The last change is to make it check in the loop if the character needs to be upper cased later. At this point we are where I stopped so I’m going to use my current code.

This gives a benchmark output:

Running tool: C:\\Go\\bin\\go.exe test -benchmem -run=^$ github.com/wselwood/resparse -bench ^BenchmarkParseResolution$

goos: windows
goarch: amd64
pkg: github.com/wselwood/resparse
BenchmarkParseResolution-4   	10000000	       142 ns/op	      15 B/op	       0 allocs/op
PASS
ok  	github.com/wselwood/resparse	1.744s
Success: Benchmarks passed.

We have shaved another 20 nano seconds off, and managed to half the average allocation size. Note this is averages over the 10 million operations here so going from 1 to 0 average allocations is probably only just under half of them. Now if we look at the profile we can see our map lookup has become a large amount of the time and the ToUpper is now smaller.

Conclusion

While it was probably not really worth going to this level of optimization with this chunk of code, it was interesting and hopefully provided a good introduction to the profile and benchmark tools provided with Go. There is a lot more power in the profile tools that I have not explored here. I recommend Rakyll’s excellent blog for further reading. I hope you gained something from this. If you did, or have any questions, please let me know on twitter.

Windows Subsystem for Linux Dot files

Thu, 09 Aug 2018 08:25:13 +0100

I’ve been using the Windows Subsystem for Linux (WSL) for a while at work. It’s been something of a radical improvement over cygwin for most of my use cases. A couple of weeks ago my installation got borked by the work Antivirus (AV) system. So I’ve created an automated script to set up my environment just how I currently like it.

Before we start lets get one thing out the way. Windows is a perfectly fine development environment. I won’t be tolerating any comments of just use a mac or just install linux. This is not always possible with a corporate machine. If this is of no help to you stop reading and go look at something else.

Assumptions

You have Ubuntu for windows already installed but not yet configured

Process

Step one was to create a directory to keep my initial environment in. Later we will turn it into a git repo as this sort of thing should not be kept on your machine. The times you need it most are when your machine has died and you need to build a new one.

mkdir -p /mnt/c/git/dotfiles/bin
cd /mnt/c/git/dotfiles

Next is to create the skeleton of the script that will do all the work for us in future. In the bin folder create a script called setup.sh

#!/bin/sh

# This needs to be run with sudo

apt update
apt upgrade -y

This updates all the existing packages.

Next we need to install the new things that we want. Add the following underneath:

apt install -y $(cat ../pkglist.txt)

This installs everything listed in a file one level up called pkglist.txt lets go and create that now. The following is mine:

build-essential
git
ssh
vim
tmux
make
wget
curl
zip
python3
python3-pip
xdg-utils
dos2unix
jq
html-xml-utils

Next we will create a directory in our home folder called bin to house all the executable things we need for our user:

mkdir -p ~/bin
chown wselwood: ~/bin

We need to change the owner of the bin folder as we are currently running as root

Now my work environment is very mixed. I work in a mixture of almost even parts Scala/Java, Go, and Python. I install python 3 the pkglist.txt already so that is easy enough. Next we will tackle java

Java

There are plenty of guides about installing java on a ubuntu system out there. Digital Ocean have a good one that I usually follow. It boils down to adding a line to the setup.sh file and one entry in the pgklist.txt

The setup.sh should now look like this

#!/bin/sh

# This needs to be run with sudo
add-apt-repository ppa:webupd8team/java
apt update
apt upgrade -y

apt install -y $(cat ../pkglist.txt)

Then add oracle-java8-installer to the pgklist.txt file.

Almost done. One last little trick to make our Java life easier. If we sym-link our WSL ~/.m2 folder to our windows .m2 folder we can avoid having two sets of libraries and very strange problems if we do mvn install at any point. Add the following to the bottom of your setup.sh

ln -s /mnt/c/Users/Wil.Selwood/.m2/ ~/.m2/

Next on to go

Go

Unfortunately at the time of writing the version of Go in the ubuntu package manager is way out of date so we need to go and get the latest version our selves. For this we are going to use a few tricks to make sure we always download the latest version.

First off installing go its self. Unfortunately there isn’t a handy URL I could find that always points to the latest stable build. There is a link on the download page but it changes with each version. So we are going to use the html-xml-utils package of tools to do some nasty (but better than pure regex) command line foo to get hold of the url from that page. Using chrome inspect the download link we are after. It should have a class of downloadBox So we can use that to find the one we need

Get hold of the page. We will use curl (cat url from my understanding) pass it a URL and it returns the text content to stdout.
Normalise the html into valid XML
Extract the links with a class of downloadBox
Find the link that says its for linux-amd64
Extract the link from that line.

The following line does this and stores the resulting text in a variable for us to download later:

go_url=$(curl --silent "https://golang.org/dl/" | hxnormalize -x | hxselect -s '\n' -i 'a.downloadBox' | grep linux-amd64 | grep -Po 'http[^\"]+')

Thats a bit long and unpleasant but it does the job. Now that we have the download url we can download, extract, and link to our path

wget $go_url -O ~/go-linux.tar.gz
tar xzf ~/go-linux.tar.gz -C ~/
ln -s ~/go/bin/go ~/bin/go
ln -s ~/go/bin/godoc ~/bin/godoc
ln -s ~/go/bin/gofmt ~/bin/gofmt

That should get us go installed. I’ve a few projects that use dep for package management so we need to install that too. This is a little bit easier as dep has its releases in github which provides an api to download them so we can use the jq command line tool to extract the url we need from the json response.

Send the query to the github API
Extract the asset that has a name ending with linux-amd64 and pull out the browser_download_url

dep_url=$(curl --silent "https://api.github.com/repos/golang/dep/releases/latest" | jq -r '.assets[] | select(.name | endswith("linux-amd64")).browser_download_url')

Now we just need to download that url and make it executable

wget $dep_url -O ~/bin/dep
chmod +x ~/bin/dep

There we go. That should be go set up and ready to work.

Now on to some quality of life changes

Quality of life

vim and git default configuration

I have a copy of my .vimrc and .gitconfig files stored in the dotfiles directory so we need to copy those over to our home directory. The following mess of characters copies everything from the directory above that starts with a dot to our home directory. Note there is no -r flag on the cp command so it will not copy the .git folder that exists in our project.

cp ../.* ~/

My configuration for vim and git are not too complex. I am colour blind and find the default colour schemes of both programs hard to read. So I change them in this config.

bash

I use mrzool’s Sensible bash as a starting point here. Along with a .bashrc file that I have had for so long I’m not sure where it came from.

Drop the sensible.bash file into the bin directory and the entry to source it to your .bashrc file. There are instructions in the git repo.

I also add an entry to make ~/bin and ~/.local/bin be part of the path.

Next I like to enable xdg-open to start a web browser. I find this really useful to include at the end of long build processes to open a browser to the server I have just launched so that I get pulled back from what ever I went off to do while the build ran.

The first step is to create a sym-link that points to your windows browser executable. Chrome in my case.

ln -s /mnt/c/Program\ Files\ \(x86\)/Google/Chrome/Application/chrome.exe /usr/bin/chrome

Then in your .bashrc file add export BROSWER=chrome and you should now be able to type xdg-open https://google.com and it will pop open a new chrome tab for you.

The last thing I have added to my .bashrc file is a thing that starts my ssh agent the first time I open a shell. This actually works way better under WSL than it did under cygwin as it lasts until you logout rather than as long as a single shell is left open. You may not want to do this depending on how paranoid you are about security. However it does mean that I don’t have to type in my ssh key password several hundred times a day.

export SSH_AUTH_SOCK=$HOME/.ssh-socket

ssh-add -l ? /dev/null 2>&1
if [ $? = 2 ]; then
  rm -rf $SSH_AUTH_SOCK
  ssh-agent -a $SSH_AUTH_SOCK >| /tmp/.ssh-script
  source /tmp/.ssh-script
  echo $SSH_AGENT_PID >| ~/.ssh-agent-pid
  rm /tmp/.ssh-script

  ssh-add
fi

And there we will leave the changes. Now all that we need to do is commit this to a remote repo somewhere. You can find mine on github There are a few tweaks in there I don’t mention here as they are only todo with the way I my disk is layed out.

I hope you have found this useful and have some ideas for your own environment. If you have any questions or comments please let me know on twitter I’d love to see your suggestions and little things you have found that makes your life a bit easier.

Finding flatroofs with QGIS

Fri, 26 Jan 2018 21:58:30 +0000

A little while ago I attended the Climathon kic event/hackathon in Bristol at my old university. The event was to come up with solutions that could help with pollution in the city. There were not a lot of people who turned up so we formed a single team. The idea we came up with was to create an incentive for people with flat roofs to put plants on them. This meant we needed to be able to find flat roofs in the city. There were suggestions that we could do this with machine learning and looking at satellite images, but I thought we could do this with the Free LIDAR data provided by the environment agency and a few steps of processing.

Assumptions

You have a computer with QGIS installed. You have a few hundred megabytes to a gigabyte of free storage.

Background

There are two sets of data that we are going to need for this. One is the Digital Surface Model (DSM) and the other Digital Terrain Model (DTM). These data sets are generated by an aircraft flying over an area with a laser system on its back. This laser is used to measure the distance from the plane to the surface below. This is then processed to build a height model of the world. The DSM is what is measured by the laser and it includes all the buildings, trees, lamp posts and other things that stick up from the ground. The DTM is further processed to remove these things and leave only the ground.

Downloading the data

The environment agency data website takes a little bit of getting used to but makes sense eventually. You scroll the map and then click on the area you are interested in. This will bring up information at the bottom of the screen about the available data. There are two sets of data: 2 meter and 1 meter. The 1 meter data will give us a better result, though the data will be bigger. The data for each square is split into four parts: North West, North East, South West, South East. If you are interested in a smaller area you can just download the bits you need. For some reason the only way to tell which one you are downloading is the address of the download on the right hand side of the table.

Click the link and download the file. Extract the zip files somewhere. Make sure that each zip file is extracted to its own directory. E.g:

data/
  dsm/
    LIDAR-DSM-1M-ST67nw/
      ...
  dtm/
    LIDAR-DTM-1M-ST67nw/
      ...

Processing

Now to start loading things up in QGIS. The first thing to do is join up the images so they are one image and not lots of small chunks. This will allow us to do less work later. Go to the Raster menu then Miscellaneous and Merge. This will pop up a window that looks like this:

The first thing to do is tick the box for input directories instead of files. This will allow you to select a directory rather than each file individually. Select one of the DSM directories you extracted and set the output file to a tif. I tend to make it the same name as the folder I selected so I know which one is which.

We didn’t need to set anything else, so press the Ok button. When it is done QGIS will open the result in the viewer. You will see a black and white image something like this:

The next step is to re-project this image. The British government uses its own coordinate system. If we want to use this with other things its easier to convert it to a more common projection. Map projections are a very involved topic that is way beyond the scope of this post. If you’re interested there are many good resources out there. To change the coordinate system of the image go to the Raster > Projections > Warp (Reproject) This will bring up a dialog that looks like this:

Tick the box next to the Target SRS entry, then press the select button. There should be a long list of possible projections. Find the one labelled WGS 84 or EPSG 4326. On the warp dialog fill in an output name. I usually append _warp or _repo to the name. Press OK and it should load a new image over the top of the old one. This is one of those things that will not appear to be any different, however it will make things easier later.

Next repeat the process for the DTM. Merge the images together and warp them into WGS 84 projection.

Now we have two merged and warped images.

The next step is to subtract the two images. This will leave us with all the things that stick up from the ground. To do this we need to use the Raster Calculator Tool. Select Raster > Raster Calculator from the menu.

This screen is very powerful but it is also intimidating to start with. Upper left are all the images you have loaded up in QGIS so far. On the upper right is the output format options. The important one is the file name, you have to select one or nothing will happen. Also make sure that the output CRS lists WGS 84.

The bottom section is a formula editor. This works a lot like an excel formula but rather than applying to cells it will apply to every pixel in the result image.

What we need to do here is take the DTM away from the DSM. If you double click on a band name in the box on the upper left it will add it to the calculation. Double click on the DSM Image and then click on the minus button in the collection of operators. Finally Double click on the DTM Image.

Hopefully if every thing went well it will say Expression Valid underneath the formula editor. Pick an output file name and press ok.

You should now have an image with a black background and gray and white buildings showing.

This gives us a pretty good idea of buildings. Some filtering by object size would probably give a good set of building outlines at this point. We, however, want to find flat roofs so we need a couple more steps.

The next thing is to work out how sloped each pixel is. Thankfully QGIS makes this very easy. There is a handy set of tools built in for handling Digital Elevation Model (DEM) data sets, which this is one of. Select Raster > Analysis > DEM (Terrain Models) from the menu.

As usual pick an output file name. Make sure you don’t overwrite the source image here, we will need it later. Also make sure the correct Input file is selected if you have more than one open. We need the subtracted image we just generated for the input.

In the mode drop down select Slope then press OK.

If you zoom in you should find lots of white lines around the walls of buildings (these are vertical slopes) and black between buildings (this is the ground). Some of the buildings will be black in the middle too denoting flat roofs. These are the ones we want to find.

Next we want to find all the places that are off the ground and not sloped. To do this we use the Raster Calculator again. This time we want to create a boolean (zero or one) image where the dsm-dtm image is greater than 3 and the slope image is less than 20 (it’s in degrees).

The formula is:

"LIDAR-1M-ST67NW-DSM-DTM@1" > 3 AND "LIDAR-1M-ST67NW-SLOPE@1" < 20

The thresholds for this can be changed if you find they don’t quite work for your area: I was working with Bristol. This will create a purely black and white image. Some bits will be quite speckled with some blobs. If you change the enabled layers in the layers panel you can probably start to work out what something’s are. The thin lines are often the peaks of pitched roofs. Dense Speckles are often trees.

To be able to use this we are going to have to clean up the data. Todo that we can use the Sieve tool. Go to Raster > Analysis > Sieve

Make sure the right input file is selected then pick an output file. Tick the box next to threshold and enter 25. This is the number of pixels that need to be next to each other to show up in the result. Again this is changeable, but I found this to be a reasonable size that avoided too many long roof peaks showing up in the results. Have a play and see how things come out for your area.

Press ok, you may be lucky here and have a nice image right off the bat. When I did this QGIS picked some very odd values for the colour range so I ended up with an entirely black screen. To fix this double click on the layer in the layer panel and you should get properties appearing. Go to the style section. It should be set to single band gray and there will be min and max boxes. Set the min to 0 and the max to 1.

Now we have a map of all the flat roofs in the area. The last thing that will be useful is to convert this into a vector format that will allow us to do things like overlay on google maps etc. This time we need the polygonize tool. This is under Raster > Conversion > Polygonize (Raster to Vector)

Select the right input file and pick an output file name. This defaults to ESRI shape files (kind of ugly but will do for now) so don’t put an extenson on the output file name. Press OK to start the process. Be aware this can take a while.

When it finishes your map will probably turn a random colour, mine went pink. For some reason a polygon is also drawn around the outside of the image. This is very easy to get rid of however. Right click on the layer and select the Toggle Editing option. Then switch to the Identify Features tool. It should be in the top bar and look like a mouse cursor pointing at an information bubble.

Select somewhere on the map that is background and not another feature. A box should appear on the right. You may need to resize it a bit to be able to see both columns, there should be two. The feature selected should have DN 0 in its labels if it is the border polygon.

In the tool bars at the top there should be a delete button. It looks like a dustbin. When you click it the background of your area should go white again. Now you just need to save your changes and disable editing. To save the changes to a layer, right click on it and select the Save changes option. Then Toggle Editing again to make sure you don’t do any thing you don’t mean to.

You should end up with something that looks a bit like this:

Conclusion

This turned into a much longer post than I planned however it covers a lot of useful things with the QGIS tools. We learnt to use the Raster Calculator, re-project an image, merge images, sieve an image, and use some of the DEM tools. Finally we also converted an image into polygons.

These are all very useful tools and hopefully you learnt something from this. If you found this interesting or you have any questions please let me know on twitter.

Spark and the Minor Planet Center data part 3

Tue, 05 Dec 2017 19:39:29 +0000

In the last post we read the minor planet center observation file. This was a fixed width text file. We only pulled a couple of columns out of it, but we learnt to use User Defined Functions, groupBy and select. In the first post of this series we covered reading a json file which contained information about all the asteroids we know about.

This time we are going to join the two data sets together and finally solve our original problem, which was to find the full date of the earliest observation of each un-numbered object. The json file we first looked at contained the year but not days and months. The observation file has the full date down to smaller than minutes accuracy, but it does not have the orbital parameters.

So we need to join the two files up to get our results.

Assumptions

You have been through the previous posts in this series and understood them. You have a project already set up and able to read and process the orbit and observation files.

Setup

As you should already have the project and the two required data files you are already set up.

Development

Thankfully Spark SQL makes this bit really easy. the .join() function on a data set allows you do all the kinds of joins you could do in a database. For this we need a simple inner join, which is the default option so you won’t see us having to define it in the code below.

val joined = orbRec.join(obs, Seq("id", "id"))

There is not a lot of difference in which way around you do this from the results point of view. There may be differences in performance but that will depend on your data set. In this case with the minor planets data it doesn’t really matter which way around we do it. If you now add a joined.show(2) line you will be able to see the columns from both data sets are now next to each other.

There are several ways to tell Spark SQL that you want to match the two id columns up. Seq("id", "id") is probably the simplest but you can also do something like obs.col("id").equalTo(orbRec.col("id")) This is really useful if you want to use something that is not a simple equals when joining.

Now we don’t really want all those columns in the output so we can add a .select() call to clean things up a bit.

  .select(
    obs.col("id"),
    col("Name"),
    col("date"),
    col("a"),
    col("e"),
    col("i"),
    col("Epoch"),
    col("H"),
    col("G"),
    col("Node"),
    col("Peri")
  )

Unfortunately for me at this point things went bang. I got an exception about the query plan being too big. My first attempt at fixing this was adding a .select() call to the orbRec processing chain to trim down the number of columns. This didn’t work. In fact I think it made things worse. The query plan is now longer.

The solution is to add checkpoints. Checkpoints allow spark sql to split the query plan in to bits. The checkpoints are written to disk and persistent. This is also useful if you need to reuse a data frame multiple times but it is expensive to compute.

The first thing we need to do is define where to put the checkpoint files. Just under where we create the SparkSession we need to add

spark.sparkContext.setCheckpointDir("/tmp/")

This will cause spark to store its checkpoint data in /tmp you may want to put it somewhere different depending on what you are running on. Now we just need to tell spark where to checkpoint. This is done with the .checkpoint() function. I tend to find it best to checkpoint before both sides of a join and any time you are going to reuse a data frame.

We simply need to add the call to .checkpoint() at the end of the two chains we defined for orbRec and obs. Now when we run joined.show(2) we should get something like

+---------+----+-------------------+---------+---------+-------+---------+----+----+---------+---------+
|       id|Name|               date|        a|        e|      i|    Epoch|   H|   G|     Node|     Peri|
+---------+----+-------------------+---------+---------+-------+---------+----+----+---------+---------+
|1995 SR42|null|1995-09-20T09:21:27|2.3527416|0.1811627|2.01897|2449980.5|19.0|0.15|117.61112|171.74016|
|  1996 LW|null|1996-06-09T07:21:53|2.1578017|0.2848271|7.64274|2450240.5|19.5|0.15|194.61088| 128.2009|
+---------+----+-------------------+---------+---------+-------+---------+----+----+---------+---------+
only showing top 2 rows

Now the only thing left to do is write the data back out to disk. Here we will run into one of the interesting design problems of running spark locally.

Saving data as a csv file is as easy as reading a file. You just need a write rather than a read

  joined.write.mode(SaveMode.Overwrite).csv("output")

Note the lack of file extension on the target name. This is because output will be a folder and not a file. Under the hood spark will split the data frame into partitions. These are basically groups of rows that will all be on the same machine in a clustered environment. In our environment every thing is on the one machine, however in a cluster you would get one output directory on each machine with the partitions that were on that machine written out.

After running the program, inside the output folder you will find a large number of files. There will be a _SUCCESS file created when every thing has finished. There will be one part-*.csv file and one .part-*.csv.crc file per partition in your data. The file ending in .crc is a checksum allowing you to verify that the data has written correctly if you need it. The files ending in .csv will be all the data you asked Spark SQL to write out.

The last step is to join every thing up. I find this easiest to do from the command line. (I’m on a linux machine with bash)

cat output/part-*.csv > output.csv

This will generate an output.csv file which is all the parts joined together. Note that you will not be able to control the order of the rows using this.

And there we go. We now have the orbital elements for all the un-numbered objects with the date of their first observation. In this part we have learnt how to join to data frames together and how to output data. This part has been a bit shorter than the others but joins every thing together. (Pun intended)

You can find all the code for this project on my github

I hope this is useful to you. If you have any questions please let me know on twitter.

Spark and the Minor Planet Center data part 2

Sun, 03 Dec 2017 15:39:22 +0000

In the last post we read the minor planet center orbit file. This was a JSON text file. This time we are going to look at a bit more complex file to process. If you haven’t read the first post in this series I recommend starting there before reading this.

In this post we are going to be looking at the Observation file. There are two parts to this file. One is for the numbered objects and other other for the un-numbered objects. Due to the original idea for this project we are going to work with the un-numbered file today.

Assumptions

You have been through the previous post in this series and understood it. You have a project already set up and able to read and process the orbit file.

Setup

The data file we are after can be found on the mpcat-obs page under the link for the un-numbered minor planets. Download it. Then while it is downloading open up the project you created following along from the last post.

Development

First lets add another variable so we don’t have to remember which argument is which.

val observationPath = args(1)

Now to get on to reading the file. In some ways this one is easier and some ways more complex. The data we are looking at here is a fixed width format. This means that there is nothing to separate the columns just that they always start in the same place no matter how long the data contained in them gets.

Unfortunately there is not a built in function to deal with this easily. So we will have to read the file line by line and then split it up our selves. Actually we will cheat a little bit and only pull out the bits we need so as not to waste time doing work we don’t need to do.

  val obs = spark.read.text(observationPath)

Like with the JSON reader from the previous post it is very easy to read a text file in spark sql. This time we don’t even need any options. This will create a data frame which contains each line from the file in a column called “value”

The first step is always to check the data looks how you expect it to look. Use the .show() function to have a look at the first few rows and get a feel for how the data looks. To make our lives easier however there is some documentation available The column formats are defined in fortran format, I am pretty sure this is because that is what is used to generate the files. However it’s a pretty simple format.

A Means ascii and then the number following it is the number of characters allowed. e.g

Columns     Format   Use
    1 -  5       A5     Minor planet number
    6 - 12       A7     Provisional or temporary designation
   13            A1     Discovery asterisk

The columns are numbered from 1 rather than 0 as an array would be. Which is another thing to remember. The first thing we need to do is extract an id column. Similar to what we did with the json file. In a future post we will use this to join the two files together.

We can use the substring function to do this. Because we are using the un-numbered file we need to be looking for the provisional to temporary designation column for the ID. We could use both and coalesce but as we know what data we are putting in we might as well keep things simple.

  .withColumn("id", substring(col("value"), 6, 7))

Now the next wrinkle is that the designations are what they call packed. This means that there is a range of different meanings to the data. There is an explanation in the MPC documentation. This code is best put into a function that can be tested separately.

Create a new function. There are three main cases we need to deal with.

If all the characters in the string are numbers.
If the third character is a number.
Any thing else.

Your function should look something like this:

  def unpackIdFunc(in : String) : String = {
    if (isAllDigits(in.substring(1))) {
      trimZeroFunc(unpackNumbered(in))
    } else if (in(2) >= '0' && in(2) <= '9') {
      numericId(in)
    } else {
      twoCharCode(in)
    }
  }

So this function takes in a string and performs out three tests before deciding how to format the result. The first detection of if its an integer is a simple function isAllDigits(). I’m going to leave it, along with the trimZeroFunc(), as an exercise for the reader.

The Next is the unpackNumbered function. This looks like this:

  def unpackNumbered(in : String) : String = {
    if (in(0) >= '0' && in(0) <= '9') {
      in
    } else {
      val numeric = in.substring(1)
      val expanded = if (in(0) >= 'a' && in(0) <= 'z') {
        in(0) - 'a' + 36
      } else {
        in(0) - 'A' + 10
      }

      expanded.toString + numeric
    }
  }

The really simple path in this function is if the first character of the string is a number. In which case we don’t have to do anything here. If it is not a number then we need to convert the first character into a number and then append the rest of the input string. The first character uses a range from 0-9 then A-Z followed by a-z to encode numbers 0 to 61. This helps save a bit of space in the files and keeps things in a reasonable order with out having to add an extra leading zero on to the numbers (and change the column lengths) every time too many asteroids are found.

Next up is the numericId() function. This one needs to pull some bits from different places and arrange them correctly.

  def numericId(in : String) : String = {
    val year = unpackNumbered(in.substring(0, 3))
    val number = trimZeroFunc(unpackNumbered(in.substring(4, 6)))
    if (number == "" || number == "00") {
      year + " " + in(3) + in(6)
    } else {
      year + " " + in(3) + in(6) + number
    }
  }

The format consists of the year, a space, the forth character in the string, the seventh character in the string and then a packed number between the two. You can see this reuses the unpackNumbered() function from above.

Last we have the twoCharCode() function. This one is very simple after the last one. Here we just have to unpack a number and then join it up with two characters and some spacers.

  def twoCharCode(in : String) : String = {
    val number = unpackNumbered(in.substring(3))
    number + " " + in(0) + "-" + in(1)
  }

Now we have a function that will unpack the different types of id we might find in the observation file. I recommend using the examples given in the MPC documentation to create some test cases. Put them in src/test/ and you will be able to run them with ./gradlew test or the correct button in your ide. If you have got this wrong you will get very strange results later.

We need to turn this function unpackIdFunc into a user defined function (udf) so that spark knows how to use it. It will need to work out how to send the function to other computers and so on. Thankfully it does a lot of magic behind the scenes so we just need to define it.

val unpackId = udf(unpackIdFunc _)

The underscore here just means that the first parameter to the udf should be passed through to the unpackIdFunc function. Now we can use it in our program. Change the line we created earlier to extract the id column to call the unpackId udf.

  .withColumn("id", unpackId(substring(col("value"), 6, 7)))

Nesting functions like this is very powerful and useful. You can often save your self from creating lots of temporary columns by doing this. Though it can sometimes be harder to work out what is going on.

Our original use case was to be able to find the full date of the first observation of an object. So the next thing we need to extract is the date and time of the observation. This starts in column 16 and is 16 characters long. We will also need to create a function to convert the date from a string into an actual date object.

The date format is a little bit weird. The Year, month and day are pretty reasonable. The first four characters are the year, the next two the month and the next two are the day. The rest of the string however is the part of the day divided into 10000 chunks. So we need to work out how many seconds we have if we take the number of seconds in a day and divide it by 10000. Then we can multiply that number by the last part of the string. Take a look at the code below it will hopefully make a little more sense.

  def dateFunc(in : String): Long = {
    val year = in.substring(0, 4).toInt
    val month = in.substring(5, 7).toInt
    val day = in.substring(8, 10).toInt

    val part = in.substring(11).replaceAll(" ", "0").toInt
    val seconds = Math.round(((24*60*60) * 0.00001) * part)

    LocalDate.of(year, month, day).atStartOfDay()
      .plus(seconds, ChronoUnit.SECONDS)
      .toInstant(ZoneOffset.UTC)
      .getEpochSecond
  }

I’ve used the java 8 date functions as they are a significant improvement over the older API. The bit to watch out for is to make sure you set the time zone to utc. Finally we return it in epoch seconds as spark doesn’t know how to handle dates very well. It is just a lot easier to deal with the date a long. Now turn the dateFunc into a udf like we did with the unpackIdFunc

 val dateConvert = udf(ObsUtils.dateFunc _)

Then add another column to our data frame.

  .withColumn("ts", dateConvert(substring(col("value"), 16, 16)))

We now have the columns we need so we can try and find the minimum time stamp for each id. To do this we need to first group by the id column and then find the minimum of the ts column. We will need to use the .groupBy() and col functions for the first part. Rather inconsistently the .min() function does not need its column name wrapped in a col call. What we end up with should look like this:

val obs = spark.read.text(observationPath)
      .withColumn("id", unpackId(substring(col("value"), idStart, idLen)))
      .withColumn("ts", dateConvert(substring(col("value"), 16, 16)))
      .groupBy(col("id"))
      .min("ts")

There are other aggregation functions available as you would expect. There is a max, avg, and sum to get you started. It is also possible to create your own.

Now you can add a obs.show(5) call below it to have a look at the results you have. You should see two columns that look something like below

+-------+-----------+
|     id|    min(ts)|
+-------+-----------+
|1908 OD|-1938820667|
|1914 KA|-1755298440|
|1927 UA|-1331164308|
|1931 RS|-1208486322|
|1933 DC|-1163208817|
+-------+-----------+
only showing top 5 rows

The dates are not massively useful like this. Converting them back in to a human readable string requires another user defined function.

   def formatDateFunc(in : Long) : String = {
    LocalDateTime.ofEpochSecond(in, 0, ZoneOffset.UTC).format(DateTimeFormatter.ISO_LOCAL_DATE_TIME)
  }

This uses the built in ISO8601 date formatter. We could use any thing but might as well use the built in standard. Turn it into a UDF as usual.

  val formatDate = udf(ObsUtils.formatDateFunc _)

Last we need to use this to format the ts column. We can use the .withColumn() function like before. But this time we are also going to use the .select() function to remove the min(ts) column that we don’t need any more. This can be very useful if you only need a few columns in a large data set. Also note that we had to call the ts column min(ts) now as it has had its name changed by the min function.

  .withColumn("date", formatDate(col("min(ts)")))
  .select("id", "date")

If you now run obs.show(5) you should get something like

+-------+-------------------+
|     id|               date|
+-------+-------------------+
|1908 OD|1908-07-24T22:42:13|
|1914 KA|1914-05-19T01:06:00|
|1927 UA|1927-10-27T00:08:12|
|1931 RS|1931-09-15T21:21:18|
|1933 DC|1933-02-20T22:26:23|
+-------+-------------------+
only showing top 5 rows

Conclusion

In this post we have learnt to read a text file, pull the bits out of the lines that we need, create user defined functions to handle more complex processing of columns, group by columns, perform aggregations, and select columns. This will hopefully leave you in pretty good stead for processing data. We will join up these two data sets in the next post.

If you found this interesting or you have any questions please let me know on twitter.

Spark and the Minor Planet Center data

Sat, 02 Dec 2017 08:55:24 +0000

Introduction

A few weeks ago I saw comments between @Sondy and @JLGalache talking about getting a list of asteroids with their date of discovery. The main data file lists the year of discovery but not the actual date. I thought there was a way to get this information by looking at the observation file and joining it to the main data file. Todo this I decided to use Apache Spark. In this post I’ll go through setting up the spark environment and reading the json object file.

What is spark?

Spark is an in-memory distributed processing framework. It is one of the largest open source data processing frameworks. There is a core section and several modules built on top. For this we will be using Spark SQL.

What is the Minor Planet Center?

The minor planet center is an organization that keeps track of observations of asteroids. They keep a list of all the known asteroids and all the observations people around the world have made. They publish this data in a range of formats from a fixed width text file to json. Their documentation has improved a lot in the last couple of years.

What data are we looking at?

A JSON file which contains information about all the known asteroids. This has their orbital parameters, id, name, discovery year, etc. Don’t worry too much we won’t get in to orbital mechanics here. (Mostly because I struggle with it my self)

While these data files are not big data by any means they are real data that has actual quirks and is not tiny which makes them a good thing to practice with. They shouldn’t take hours to process.

Assumptions

I am going to assume you have a local java installation and an IDE you are comfortable with. You know some scala. You can probably work out whats going on but it will be a lot easier if you are aware of the scala syntax before you start reading this.

Setup

First we need to go and get the data files. These are reasonably big. On the minor planet center data page there is a link to the mpcorb_extended.json.gz file. Download this. This might take a while. So feel free to get started with the next bit.

I use intelliJ. So create a new gradle project. There is no particular reason I chose Gradle over SBT or maven only I have more experience with it. The basics of the project setup are easy enough. First add a few bits to mark the project as a scala project and create variables for versions of things. This will save repeating your self if you need to change them in the future. The versions I used were simply the latest version of spark and its matching scala version when I did this. Open up the build.gradle file.

apply plugin : 'scala' // Adding the scala flag adds steps to the build process for the scala compiler

project.ext.scalaVersion = '2.11.9'
project.ext.sparkVersion = '2.2.0'

sourceCompatibility = 1.8 // Use Java 1.8 compatibility. Mostly this is a hint for the IDE rather than the build.

Then add the following dependencies to the dependencies section of your gradle file, there should be a junit dependency there already.

    compile "org.scala-lang:scala-library:${project.ext.scalaVersion}"
    compile "org.scala-lang:scala-reflect:${project.ext.scalaVersion}"
    compile "org.scala-lang:scala-compiler:${project.ext.scalaVersion}"
    compile "org.apache.spark:spark-core_2.11:${project.ext.sparkVersion}"
    compile "org.apache.spark:spark-sql_2.11:${project.ext.sparkVersion}"

Development

Now we need some actual code. Create a file in the src/main/scala/ directory called something like orbitdates.scala

Create a main method:

object OrbitDates {
  def main(args: Array[String]) : Unit = {
    // Our code will go here
  }
}

Note: with spark you must create a main method this way and not use the extends App style syntax because the objects will be serialised and sent to worker nodes by spark later on. You will get very strange errors due to the order that the generated main method does things.

It is probably worth adding a println("hello world") at this point and making sure the project builds and runs. ./gradlew build from the command line or the gradle build button in your ide should run cleanly. There will probably be a lot of downloading to be done the first time.

The first step of a spark program is getting hold of the right kind of spark context or session.

    val spark = SparkSession
      .builder()
      .master("local[2]") // if you have more cores or a cluster turn this up.
      .appName("Spark SQL join observations")
      .getOrCreate()

As we want to use Spark SQL we need a spark session. This needs to be told where our spark “cluster” is. For this we are just using local with two threads. If you have a more powerful machine feel free to turn up the number of threads. If you are lucky enough to have an access to a cluster you should put the address of the “master” node here. (Note: the master/slave terminology that spark uses is horrible. Controller/worker would make more sense and be less offensive)

The .appName() is just a name for your app. It can be anything. If you are running in a cluster this is what shows up on the web front end.

The .getOrCreate() call at the end of the chain returns the spark context you have just setup. If for some reason your code already has an active spark context this will handle that as well.

To make that work you will also need to include

import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.expressions.Substring
import org.apache.spark.sql.functions._

The first one covers the SparkSession. The other two we will get to later.

To make things clearer later I added a mapping from the arguments simply so I didn’t have to remember if the data file was args(0) or args(34)

val mpcobs = args(0) // The json data file of objects

Now we have the spark session and something holding the path to our input files we can start trying to read them. First lets start with the json data file.

val orbRec = spark.read
      .option("multiLine", true)
      .json(mpcobs)

This will give us what is called a data frame in spark sql parlance. Basically its a lazy representation of the data file. Nothing will actually be done until we ask for something to be returned. spark.read contains methods for reading lots of different kinds of data sources. These data sources all have different options that can be applied to them. Here we ask to read a multi line json file. This option has only existed since spark 2.2. We need the multi line option due to the way the MPC json file is layed out.

This will give us access to the data for each object as a record. To see how the data looks you can use the .show() function. This will display a simple table of the data to stdout when the program is run. This is very useful for working out how your data looks. There is also .describe("column name") function which will give you statistical information about your column.

The next step is to sort out the number column and remove the brackets that are around it. After that we can find the correct column to use for the id.

  .withColumn("number_formatted", new Column(Substring(col("Number").expr, lit(2).expr, (length(col("Number"))-2).expr)))
  .withColumn("id", coalesce(col("number_formatted"), col("Principal_desig")))

The first line adds an extra column that does a substring of the original column “Number” removing the first and last two characters from the column. The second line creates a new column called “id” which will contain the “number_formatted” column if it is not null or it will use the “Principal_desig” column. The coalesce function is very useful when you need to pick the first non null column from a list of options.

Now you should have a block that looks like this:

  val orbRec = spark.read
    .option("multiLine", true)
    .json(mpcobs)
    .withColumn("number_formatted", new Column(Substring(col("Number").expr, lit(2).expr, (length(col("Number"))-2).expr)))
    .withColumn("id", coalesce(col("number_formatted"), col("Principal_desig")))

This simple code will read in the file and create an “id” column. If you add a orbRec.show(2) on the end you should see a very long table like below. There will also be a load of logging of what spark is doing. It should end up in stderr rather than stdout. You can see the last two columns are the ones we added.

+-------------+----------+---------+--------+----------------------------------+---------+----+----+---------+----------+---------+--------+------+---------+-------+--------+------+---------------+--------------------------+----------+--------------+------------+--------+---------+---------------+----------+------------+---------------+---------+----------------+--------------+-------------+---+---------+---------+--------+----------+----+----------------+---+
|Aphelion_dist|Arc_length|Arc_years|Computer|Critical_list_numbered_object_flag|    Epoch|   G|   H|Hex_flags|  Last_obs|        M|NEO_flag|  Name|     Node|Num_obs|Num_opps|Number|One_km_NEO_flag|One_opposition_object_flag|Orbit_type|Orbital_period|Other_desigs|PHA_flag|     Peri|Perihelion_dist|Perturbers|Perturbers_2|Principal_desig|      Ref|Semilatus_rectum|Synodic_period|           Tp|  U|        a|        e|       i|         n| rms|number_formatted| id|
+-------------+----------+---------+--------+----------------------------------+---------+----+----+---------+----------+---------+--------+------+---------+-------+--------+------+---------------+--------------------------+----------+--------------+------------+--------+---------+---------------+----------+------------+---------------+---------+----------------+--------------+-------------+---+---------+---------+--------+----------+----+----------------+---+
|     2.976646|      null|1801-2017|MPCLINUX|                              null|2458000.5|0.12|3.34|     0000|2017-03-05|309.49412|    null| Ceres| 80.30888|   6672|     113|   (1)|           null|                      null|       MBA|     4.6037329|   [1943 XB]|    null| 73.02368|      2.5581728|       M-v|         30h|        A899 OF|MPO399990|       1.3757948|       1.27749|2458236.41089|  0|2.7674094|0.0756074|10.59322|0.21408881| 0.6|               1|  1|
|    3.4125514|      null|1821-2017|MPCLINUX|                              null|2458000.5|0.11|4.13|     0000|2017-10-05|291.65136|    null|Pallas|173.08718|   7910|     108|   (2)|           null|                      null|       MBA|     4.6179031|        null|    null|309.99154|       2.133619|       M-v|         28h|           null|MPO421624|        1.312813|     1.2764032|2458320.73644|  0|2.7730852|0.2305974|34.83792|0.21343186|0.58|               2|  2|
+-------------+----------+---------+--------+----------------------------------+---------+----+----+---------+----------+---------+--------+------+---------+-------+--------+------+---------------+--------------------------+----------+--------------+------------+--------+---------+---------------+----------+------------+---------------+---------+----------------+--------------+-------------+---+---------+---------+--------+----------+----+----------------+---+
only showing top 2 rows

If you get errors about array index out of bounds you may have forgotten to pass the parameter for where the data file is.

Now you can start to explore a bit. Find out the stats of the columns using the .describe() function. Play with the .select() function to limit the number of columns. The .where() function can be used to filter the data. You can use .agg() with min() andmax() to aggregate data. The .count() function will return the number of rows in a data set.

Challenge

How many asteroids were first observed in the year 2000?

The first person to send me a tweet with the answer will get a virtual hi-five and some kudos.

Conclusion

Here we are going to leave it for today. I’m planning to do another couple of posts. One reading the observation data file and another on joining every thing up. I hope this is useful and you found it interesting. If you did or you have questions please let me know on twitter.

Thanks

Thanks to the Minor planet center for making this data freely available.

Thanks to Sondy and JL for the idea.

Introduction

Sat, 02 Dec 2017 08:01:19 +0000

Hello World

I’m Emily Selwood. Every so often I get the urge to try and start a blog again. Here is iteration 235.

What do I do?

I build systems for a living. Mostly data processing but I’ll happily get my hands dirty doing anything that needs to get done. I’ve worked many things from Unity and C# to C, Groovy, Go, Javascript, and Big data things like Accumulo and Spark.

What will be here?

I am planning to have posts about technical topics. I have ideas for the first couple. They may take me a while to build. Due to my love of space many of the examples will use space data sets. I’ll try to keep every thing free to use.

About Me

Mon, 01 Jan 0001 00:00:00 +0000

Who am I?

I’m Emily Selwood. A software engineer by trade, parent and wife, and a hobbyist woodworker and repairer.

//todo: fill this in…

Projects

Mon, 01 Jan 0001 00:00:00 +0000

procedual generation on Parsecs Reach

SVG Mountains

Scaffolding

Size of the screen and view ports

SVG Polygons

Random numbers

Making wiggly lines

Licensing

Hosting

fin

Rust workspaces and docker

Using python to configure rust Part 2

Step 3: Find all the functions with the decorator

Step 4: Finding the arguments for the functions.

Part 5: Sorting our functions

Part 6: Running the configure functions

Using python to configure rust Part 1

Step 1: Load all the python files into the environment

Step 2: Create a rust config struct that can be constructed in python.

An introduction to Polygonical and ESVG

Assumptions

Creating a blank SVG

Creating a command line mastodon archive search tool in rust

Turning an SVG into a PDF in rust

Explorers End

Plotting Geospatial Data

SVG

COG - Cloud Optimized Geotiffs

Big O for geospatial people

InSAR

AIS

Normalized Difference Vegetation Index

180 Degrees

Announcing a new me

Family, friends, and colleagues reading this

People who know me from social media

People who know me from github

People who’ve never met me

Frequently asked questions

Golang Resparse

Assumptions

Setting up the project

Building the code

Benchmarking

Optimization

Conclusion

Windows Subsystem for Linux Dot files

Assumptions

Process

Java

Go

Quality of life

vim and git default configuration

bash

Finding flatroofs with QGIS

Assumptions

Background

Downloading the data

Processing

Conclusion

Spark and the Minor Planet Center data part 3

Assumptions

Setup

Development

Spark and the Minor Planet Center data part 2

Assumptions

Setup

Development

Conclusion

Spark and the Minor Planet Center data

Introduction

What is spark?

What is the Minor Planet Center?

What data are we looking at?

Assumptions

Setup

Development

Challenge

Conclusion

Thanks