Inkscape has amazing svg path boolean logic functionality contained in the “livarot” source directory. My pylivarot project is an attempt to make this functionality within python.

For this project, I don’t want to distribute a source distribution, because pylivarot has many dependencies which can be challenging for a non-C++ developer to set up. I ran into a few issues creating binary distributions:

Cmake, setup.py and multiple versions of python

Martino Pilia has a great blog post about how to use cmake with setuptools, and my setup.py file is largely based on the example he worked with. However, I added a few extra hacks. The first is that the cmake FindPython cmake module doesn’t have enough granularity about python versions, so I wanted to get the library, include and executable from whatever python interpreter is currently running setup.py:

            extra_config_args = []
            extra_config_args.append(f"-DPYTHON_LIBRARIES={sysconfig.get_config_var('LIBDEST')}")
            extra_config_args.append(f"-DPYTHON_INCLUDE_DIRS={sysconfig.get_config_var('INCLUDEPY')}") # this might need to be the subdir
            extra_config_args.append(f"-DPYTHON_EXECUTABLE={sys.executable}")
            subprocess.check_call(['cmake', "-S", extdir]+extra_config_args, cwd=self.build_temp, env=env)

FindPython still needs to be called in order for pybind11_add_module and python3_add_library to be defined:

find_package(Python3 REQUIRED COMPONENTS Development)

C++17 symbols

• pypi no longer accepts binary distributions that have the “linux” platform tag (it must be manylinux)
• the manylinux debian image is docker 9, which only has GCC 6
• lib2geom, one of my dependencies, uses std::optional, which is only available with C++17 forward
• C++17 requires at least GCC 8, and will require the CXXABI_1.3.11 symbol
• CXXABI_1.3.11 is not available for glibc 2.24, which is the latest version of glibc available through PEP 600

The solution here is to:

1. install gcc-8 in the manylinux_2_24 docker image:

RUN apt install -y dirmngr && apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 1E9377A2BA9EF27F && \
  echo "deb http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu xenial main" >> /etc/apt/sources.list && \
  apt-get update && apt-get install -y --no-install-recommends gcc-8 g++-8

1. statically compile libg++:

set(CMAKE_CXX_FLAGS "-static-libstdc++")

1. make sure that GCC 8 is being used within the pip virtual environment by putting in setup.py

            if os.path.exists('/usr/bin/gcc-8'):
                env['CC']='/usr/bin/gcc-8'
            if os.path.exists('/usr/bin/g++-8'):
                env['CXX'] = '/usr/bin/g++-8'
			subprocess.check_call(['cmake', "-S", extdir]+extra_config_args, cwd=self.build_temp, env=env)

1. dump any extra shared objects into the whl directory before it is packaged:

        # copy all the built files into the lib dir. Not sure why this is needed; it feels like setuptools should 
        # copy the built files into the bdist by default
        lib_dir = os.path.join(self.build_lib, "pylivarot")
        for _file in glob(os.path.join(self.build_temp, f"*{sys.version_info.major}{sys.version_info.minor}*.so")):
            print("copying ", _file," to ", os.path.join(lib_dir, os.path.basename(_file)))
            shutil.move(_file, os.path.join(lib_dir, os.path.basename(_file)))

Cricut Design Space, the software used to send designs to Cricut cutting plotters, does not handle the transform element on svg very well, which is frustrating because Inkscape loves to render paths with that attribute, rather than applying the transform directly to the path or object.

For example, here’s a design I made in inkscape. It has one path object representing the cuts I want to make in a piece of yellow cardstock on the bottom layer, and one group of paths to be traced out with a marker on the top layer:

However, this fails to render correctly in Cricut Design Space:

The telltale sign about why this is rendered like this is to look at whether a transformation matrix is being applied to this element:

    <g
       id="g6018"
       transform="matrix(0.3071252,0.01878458,-0.01878458,0.3071252,92.391111,151.00958)"
       style="stroke-width:0.831982;stroke-miterlimit:4;stroke-dasharray:none">

There are also transforms applied to layers:

  <g
     inkscape:label="Layer 1"
     inkscape:groupmode="layer"
     id="layer1"
     transform="translate(-24.058,-19.5223)"
     style="display:inline">

Some transforms Cricut Design Space can handle okay, but some it cannot. A way I fixed the transforms on this project was:

1. put all elements on the same layer
2. ungroup the line patterns, then group them again

Now it renders as expected:

Cricut also can’t handle the rotate transform. For example, I created a tiling that uses the rotate transform to put the trapezoids in the right place. But in Cricut, the trapezoids ignore this rotation, and all stack on top of each other:

Unfortunately, there’s no ungrouping as a way to get out of this. Fortunately there’s a really nice extension called Apply Transforms that will take the the transform and apply it to the path itself. Now the tiling renders correctly:

My great-grandmother Eudora was a calculus instructor at Ohio State and a fan of Martin Gardner’s Mathematical Games column in Scientific American. One of the concepts that Gardner introduced to a larger audience were Polyominos, extrapolation of dominos with larger numbers of squares. The shapes in Tetris are pentominos; they are shapes made up of 5 squares.

Eudora introduced Gardner’s column to my mother, Donna, who was also became a fan of recreational mathematics. In November, 1987, she saw this article in Science News:

Pieces of a Polyomino Puzzle

Polyominoes, which are the basis for thousands of mathematical puzzles, are shapes that cover connected squares on a checkerboard. One of the most intriguing of such puzzles involves proving that polyominoes of a certain shape can be laid down to form a complete rectangle. Recently. software engineer Karl A. Dahtke of AT&T Bell Laboratoties in Naperville, Ill., combining perseverance with clever computer programming, managed to solve two particularly perplexing versions of this problem — ones that for nearly 20 years had defeated the best efforts of scores of amateur and professional mathematicians. "I was amazed," says mathematician Solomon W. Golomb of the University of Southern California in Los Angeles, who in 1954 introduced the term "polyomino". Golomb has been exploring polyomino properties and proposing polyomino puzzles ever since. "A lot of very bright people have worked on [the problem]" he says. "This is a noteworthy accomplishment."

Dahlke, who is blind, first learned of the puzzles earlier this year in an audio edition of Science News. Dahlke spent several months working on the puzzles in his spare time. First, he tried proving that the problem has no solution. Because that approach didn't seem to lead anywhere, he started looking for an answer but kept running into dead ends. "I took so many different avenues," he says.

Finally, Dahlke decided to program his personal computer to search for an answer systematically. Dahlke's computer is equipped with a speech synthesizer that converts the computer's output into sound. It took the addition of several programming tricks designed to circumvent time-consuming situations, in which the computer was trapped in endlessly repeating patterns, before Dahlke found his two minimum-area rectangles (see illustrations).

"It turns out that the size of the solutions is clearly a little bit beyond what people could easily do by hand," says Golomb, "but fortunately, it's within the range of what you can find on a personal computer"

"I'm no Einstein," says Dahlke. "Maybe anyone with a micro, some perseverance and a little bit of geometric knowledge could have done it. But I did it, and it's a pretty thing."

Dahlke is ready to try more polyomino puzzles, He also dreams of the day when he'll get achance to go back to college to study mathematics at the graduate level.

My mom made quilts of these two tilings between 1987 and 1989, which was impressive because I was born between the two quilts.

The quilts hung in the dining room of my childhood home. I spent many hours staring at them instead of practicing piano (sorry mom). When my parents downsized after retirement, they gave me the quilts, and they now hang in my bedroom. My baby loves looking at the quilts because they are bold colours with sharp edges. As a project while my baby sleeps in my lap, I have been translating/recreating Dahlke’s original C code into Python so that I can generate vectors of the tiling for use with a die cutter, embroiderer, etc.,

Dahlke has hosted his pentomino packing code on his website, but not, as far as I know, the code that originally generated the heptomino/hexonimo tilings above. I used the packing algorithm in Dahlke’s pentomino code but with the 8 orientations of the piece instead of all permutations of the pentominos and their orientations. The packing algorithm is:

1. create a grid with a margin of size 2 around the edges. Fill the margins with a negative character.
2. find the topmost leftmost unoccupied location in the grid, and see if any pieces, defined my their topmost leftmost square, can be placed in the board. The margins prevent pieces from being placed partially off the edge of the board.
3. if no piece can fit, remove the last placed piece, and repeat step 2 with the next unused piece
4. if there are no more empty locations in the grid, the board must be filled, therefore we have a solution

Using a single thread of my core i7 7700k, this algorithm takes 3.5 hours to find the hexomino solution, and 7 hours to find the heptomino solution. Given that this code took 3 days in the late eighties, and consumer grade computing hardware has grown exponentially in the meantime, this time is a bit disappointing. Granted, I have not studied the problem for nearly as long as Dahlke, so there are probably several things I could do to constrain the search space, which I suspect is what Dahlke did.

Here’s somethings I tried to cut down on the execution time:

• parallelization: This problem does not have an obvious way to divide the search space evenly, so does not parallelize easily. I could use an evaluation queue for potential solutions, but the time cost of passing information between processes eclipses the time to evaluate each placement.

• CSPs: I tried formulating the problem as a constraint satisfaction problem, to take advantage of numerical optimization techniques that have been developed since the 80s, but this ran for days without finding a solution. I suspect I haven’t set up all the necessary constraints.

• removing holes: there are some placements that create holes that obviously can’t be filled by another shape. For example, if you place this heptomino sunny-side-up will create a hole in the next column that can’t be filled in by any piece:

Thus, if the grid location two up and one to the right of the current location to be filled is occupied, this sunny-side-up piece can’t be placed. This cuts down the search space. I identified at least one grid location to check per piece orientation for the heptominos. This cut down the time to find the first solution to 1 hour. However, identifying and coding these rules is time consuming. It feels like there should be a way for the computer to automatically detect hole-creating placements, but I haven’t found how.

My code is available on github. Dahlke requested that his pentomino packing code be used only for personal use, thus these scripts are distributed with the Creative Commons Attribution-NonCommercial 3.0 Unported (CC BY-NC 3.0) license.

freesewing.org maintains a node/react based editor for drafting parametric sewing patterns with JavaScript and JSON. Here are three ways to get it working on Windows:

Run on WSL

• Download Ubuntu for Windows,
• install node and npm via nvm, because the debian package for node is a bit old.
• add

   watch: {
    chokidar: {
   usePolling: true
    }
  }
  

  to rollup.config.js in order for file changes to be picked up

• If you want to use an editor in Windows to modify your patterns, open the files in Ubuntu file system mount point, which is going to be in *C:\Users<your windows username>\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_\LocalState\rootfs\home\\\* . Once you edit these files in Windows, you will need to add linux file permissions back to your file, i.e. by running `sudo chmod 777 /src/index.js`

Things that didn’t work for me

running natively

For some reason npm link does not correctly create symlinks on Windows. Between running npm start in the main pattern folder, and npm start on the example folder, manually create the symlink, i.e.:

npm start
mklink /D "C:\<path to your pattern>\example\node_modules\pattern" "C:\<path to your pattern>"
cd example
npm start

This leads to some weird errors with stale code.

Running in a Docker container

Running in a docker container prompts some interesting problems:

• nvm requires several environment variables to be set in order to work, but due to the idempotency of commands in dockerfiles, we can’t preserve these environment variables between steps. This means we need to explicitly state the node version and installation directory:

  ENV NVM_DIR /usr/local/nvm
  ENV NODE_VERSION 12.16.2
  RUN mkdir $NVM_DIR
  RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.34.0/install.sh | bash
  RUN [ -s "$NVM_DIR/nvm.sh" ] && \
    \. "$NVM_DIR/nvm.sh" && \
    [ -s "$NVM_DIR/bash_completion" ] && \
    \. "$NVM_DIR/bash_completion" && \
    nvm install $NODE_VERSION && \
    nvm alias default $NODE_VERSION && \
    nvm use default
  ENV PATH=$PATH:$NVM_DIR/versions/node/v$NODE_VERSION/bin
  

• should we install npm dependencies at every run of the image? This will significantly increase the runtime. To save time, in my dockerfile I install all the dependencies globally:

  RUN npm install -g create-freesewing-pattern
  RUN curl -o example/package.json https://raw.githubusercontent.com/freesewing/freesewing/master/packages/create-freesewing-pattern/template/default/example/package.json
  RUN cd example && cat package.json | jq '.devDependencies' | sed 's/: /@/' | sed 's/"//g' | sed 's/,//' |sed 's/{//'| sed 's/}//' |xargs npm install -g --loglevel=error
  

• do we run the server for both the editor front-end and pattern generator in the same line? I’ve never attempted to attach to a docker process in two different windows, so I run both in the same line, which puts all output in the same window.

For all of these complications, I ended up using the WSL instead.

If you want a python script to automatically shut itself down if it uses a certain amount of memory, the internet will tell you to use the setrlimit function in the resource library:

import resource
MAX_MEMORY = 10_737_418_240 # the maximum memory in bytes that this process can use
rsrc = resource.RLIMIT_DATA
soft, hard = resource.getrlimit(rsrc)
resource.setrlimit(rsrc, (MAX_MEMORY, MAX_MEMORY))

This works fine on Solaris, but not on Linux or AIX systems. AIX does not have an RLIMIT_DATA metric. Instead, the RLIMIT_AS is a kind of similar measure of all heap allocated memory:

rsrc = resource.RLIMIT_AS if sys.platform.find('aix') == 0 else resource.RLIMIT_DATA

Linux is able to bypass the ulimits when allocating heap memory using mmap. If your script has a running event loop, for example, if it reads in a file line by line, then you can periodically check the memory:

usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
if usage > MAX_MEMORY/1024:
	raise ValueError(f'{sys.argv[0]} has exceeded the memory limit: {usage} > {MAX_MEMORY/1024}')

This is not ideal because it only alarms on the resident size, but there’s no fast way to check the virtual memory size: reading the maxrss takes microseconds, and opening and reading /proc/pid files takes tens of milliseconds. However, in most python programs, the difference between virtual and resident memory size is not that great.

For Windows, unless you compile a library that can load in and access the system DLs, there’s no fast way to check the memory. A slow way to check the memory is using the psutils library.

How much memory does a python list require? This is easy to answer. Python has a function called getsizeof that will tell you directly:

>>> from sys import getsizeof
>>> getsizeof([1,2,3])
88

Okay, great, thanks Python! But why does this list take up 88 bytes? Also, if you run:

from sys import getsizeof
test_str = []
test_int = []
for i in range(100):
    test_int.append(i)
    test_str.append(str(i))
    print(i, getsizeof(test_int), getsizeof(test_str))

and plot the results, you’ll get:

Why does a list of integers have the same reported size as a list of strings, when ‘1’ is larger than 1?

>>> getsizeof('1')
50
>>> getsizeof(1)
28

Why does the size increase in steps at a time, instead of strictly linear?

In the cpython listobject.c list_resize code, you can see that the memory for a list is allocated using the integer variable new_allocated :

    num_allocated_bytes = new_allocated * sizeof(PyObject *);
    items = (PyObject **)PyMem_Realloc(self->ob_item, num_allocated_bytes);

new_allocated is calculated using this formula:

    new_allocated = (size_t)newsize + (newsize >> 3) + (newsize < 9 ? 3 : 6);

newsize here is the requested resize of the list. When using append, this is going to be len(list)+1. new_allocated is always going to be larger than new_size. If you append to a list, and new_size is less than or equal to the already allocated size, the function returns early and memory is not reallocated. This is a time-saving feature; if you’re appending to a list, there’s a good chance you’re going to append to it again soon, so it doesn’t make sense to do the computationally expensive step of reallocating memory every time. This is why there’s a discrete rather than continuous trend between list size and allocated memory.

Let’s go over this step by step:

• [].append(0) (1 + 1 » 3 + (1 < 9 ? 3 : 6) => (1 + 0 + 3) => 4
• [0].append(1) 2<=4, do nothing => 4
• [0, 1].append(2) 3<=4, do nothing => 4
• [0, 1, 2].append(3) 4<=4, do nothing => 4
• [0, 1, 2, 3].append(4) (5 + 5 » 3 + (5 < 9 ? 3 : 6) => (5 + 0 + 3) => 8

and so on.

We can also see that list_resize is allocating the size of a PyObject pointer, rather than the size of the object that you are appending to the list. Each item in a list exists elsewhere in memory, and the list just contains pointers to those items. If you want the true size of a list in python, you need to do recursively descend through each item in the list.

Finally, if you look at the cpython source code for getsizeof, you will see:

    if (PyObject_IS_GC(o))
        return ((size_t)size) + sizeof(PyGC_Head);

getsizeof adds on the memory cost of garbage collection. So, finally, the breakdown of that 88 bytes for the size of [1,2,3] on 64 bit systems is:

• 40 bytes for the size of the list PyObject
• 4 allocated size * 6 byte size of a PyObject pointer = 24
• 24 bytes for garbage collection

On 32 bit systems, this is:

• 20 bytes for the size of the list PyObject
• 4 allocated size * 4 byte size of a PyObject pointer = 16
• 12 bytes for garbage collection

Wow, garbage collection is expensive! Examining why is a separate post.

My python automatic digitizer can now turn pngs, text, and a SVGs produced by a variety of software into computerized sewing machine embroidery patterns. However, I am hesitant to release and advertise this project because every other pattern ends up breaking a needle when loaded onto a sewing machine. Though needles are 25 cents each, it breaks the thread, and pieces of the need can end up lost under the base plate. My sewing machine manual suggests that this happens because there are three layers of threads.

I wrote some python code to measure the density of stitches. The heatmap looks like:

Some locations in the pattern have more than 7 points - and that location corresponds to the point where the needle breaks on the sewing machine. My strategy for reducing the density is discretize the pattern into a grid, and move the fourth stitch on every grid location to the next nearest grid location with fewer than three stitches. To do this, I create a generator:

# spiral around a point until you find the next available location
class NextAvailableGrid(object):
    def __init__(self, i, j):
        self.i = i # i is the x coordinate
        self.j = j # j is the y coordinate
        self.direction = "left"
        self.stepsize = 1
        self.current_step = 0

    def __iter__(self):
        return self

    def __next__(self):
        return self.next()

    def next(self):
        directions = ["left", "down", "right", "up"] # go counter-clockwise
        # do the transforms
        if self.direction == "left":
            self.i += 1
        if self.direction == "down":
            self.j += 1
        if self.direction == "right":
            self.i -= 1
        if self.direction == "up":
            self.j -= 1
        self.current_step += 1
        if self.current_step == self.stepsize:
            self.direction = directions[(directions.index(self.direction) + 1)
                                        % len(directions)]
            self.current_step = 0
            if self.direction in ["right", "left"]:
                self.stepsize += 1

        return self.i, self.j

This will spiral out from the grid location i, j forever:

Then, the code to move the stitches is:

def de_densify(pattern):
    density, boundx, boundy, x_bins, y_bins = initialize_grid(pattern)
    # convert the density list of lists to a dict
    density = {i: {j: block for j, block in enumerate(density[i])}
               for i in range(len(density))}
    for block_i, block in enumerate(pattern.blocks):
        for stitch_i, stitch in enumerate(block.stitches):
            i = int((stitch.x - boundx[1]) / minimum_stitch)
            j = int((stitch.y - boundy[1]) / minimum_stitch)
            # if there is room for that stitch, continue
            if density[j][i] <= MAX_STITCHES:
                density[j][i] += 1
                continue
            for next_i, next_j in NextAvailableGrid(i, j):
                if density[next_j][next_i] >= MAX_STITCHES:
                    continue
                print("moving stitch from {} {}".format(stitch.x, stitch.y))
                pattern.blocks[block_i].stitches[stitch_i].x = next_i * minimum_stitch + boundx[1]
                pattern.blocks[block_i].stitches[stitch_i].y = next_j * minimum_stitch + \
                                                               boundy[1]
                print("to {} {}".format(pattern.blocks[block_i].stitches[stitch_i].x, pattern.blocks[block_i].stitches[stitch_i].y))
                density[next_j][next_i] += 1
                break
    return pattern

The density map after moving the stitches is:

And this gets stitched out without breaking a needle:

It’s still not where I want it - the jumps are too large and the threads cross over each other too frequently, but stitching without needle breaks is progress.

I generate cairo tilings because I’m working on generating some custom printed fabric. Cairo tilings are an ideal pattern for clothing prints because they can be made from the same pentagon rotated to four different angles, yet still have a semi-regular repetition, meaning that you can clip the tiling at certain places, and copy that clip up and down to make an infinite pattern.

This post goes over my method of generating cairo tilings with minimal math using my fork of svgpathtools, and svgwrite. The full code to generate the tilings is available in my python-sewing repository.

Drawing the base pentagon

According to wikipedia, a cairo tiling can be made with a pentagon with the interior angles of 120, 90, 120, 120, and 90 radians. So I’ll start off with with an angle of 120 radians, and a length of 300 radians. I’m going to lay out my pentagon like this:

angle01 is the angle from horizontal to point 1 from point 0. I start with point 0 at 0,0:

def cexp(x):
    return pow(exp(1), x)
angle01 = 60.0
length1 = 300
# start with the top corner at 0, 0
points = [0, None, None, None, None]

Next, points 1 and 4 are at the projection up and down of angle01 from point 0:

points[1] = points[0] + length1 * cexp(1j * radians(angle01))
points[4] = points[0] + length1 * cexp(-1j * radians(angle01))

points 1 and 4 have interior angles of 90 degrees. To project up from these points, I subtract angle01 from 90:

angle12 = -(90 - angle01)
points[2] = points[1] + length1 * cexp(1j * radians(angle12))
points[3] = points[4] + length1 * cexp(-1j * radians(angle12))

I’ve drawn a single pentagon with minimal math - I only had to calculate the second projection angle, and the rest came from looking at the pentagon. I’m going to draw the four-groups of cairo tiling the same way.

Drawing the four-pentagon group

Next, I generate the 4-group:

Like before with projecting from already-calculated points, I let svgpathtools do the work - use it to rotate the pentagons, snap them into place, then use the difference between where the points ended up for a list of transforms that I can use later.

def new_pentagon():
	return Path(*[Line(start=points[i - 1], end=points[i]) for i in range(len(points))])

transforms = [[0, 0]]
cairo_group = [new_pentagon()]
# point 1 of pentagon 1 needs to be attached to point 1 of pentagon 0
cairo_group.append(transform_path(rotate_transform(90), new_pentagon()))
diff = cairo_group[0][1].end - cairo_group[1][1].end
transforms.append([90, diff])
cairo_group[1] = cairo_group[1].translated(diff)
cairo_group.append(transform_path(rotate_transform(180), new_pentagon()))
# point 3 of pentagon 2 needs to be attached to point 2 of pentagon 0
diff = cairo_group[0][2].end - cairo_group[2][3].end
transforms.append([180, diff])
cairo_group[2] = cairo_group[2].translated(diff)
cairo_group.append(transform_path(rotate_transform(-90), new_pentagon()))
# point 4 of pentagon 3 needs to be attached to point 1 of pentagon 0
diff = cairo_group[0][4].end - cairo_group[3][4].end
transforms.append([-90, diff])
cairo_group[3] = cairo_group[3].translated(diff)

Finally, I want to calculate the width of the 4-group so that I know where to repeat the 4-group in the next step:

column_offset = cairo_group[0][0].end - cairo_group[1][2].end

Drawing the full pattern

The final trick to generating the full tiling is that every other 4-group in each row needs to be moved down by half of the height of the 4-group:

dwg = Drawing("{}/tiling2.svg".format(output_folder), profile="tiny")

current_color = 0
rep_spacing = pent_width * 2 + bottom_length

for y in range(num_down):
    transform = "translate({}, {})".format(0, rep_spacing * y)
    dgroup = dwg.add(dwg.g(transform=transform))
    for x in range(num_across):
        # if x is odd, point 0 of pent 0 needs to be attached to point 2 of pent 1
        if x % 2 == 1:
            dx = int(x / 2) * rep_spacing + pent_width * 2 + column_offset.real
            transform = "translate({}, {})".format(dx, column_offset.imag)
        else:
            transform = "translate({}, {})".format(int(x / 2) * rep_spacing, 0)
        group = dgroup.add(dwg.g(transform=transform))
        for pent in cairo_group:
            group.add(
                dwg.path(**{'d': pent.d(), 'fill': _colors[current_color % len(_colors)],
                            'stroke-width': 4, 'stroke': rgb(0, 0, 0)}))
            current_color += 1
dwg.save(pretty=True)

This generates the pattern:

The common way to make modifications to sewing patterns is to trace out the cut lines on tracing paper, then cut and tape the pieces of paper together to form a new shape to cut out of fabric. However, as a software developer, every problem is a programming one, so in this post I’m going to show my code for extracting the cut lines from sewing patterns as SVGs, so that you can manipulate them in Inkscape.

The pattern I’m going to use in this demo is the Cheyenne Tunic by Hey June which I recommend purchasing. I use the pdfrw library for reading in the pattern, and investigating a pdf using this library can really help your understanding of the format.

Extracting data from PDFs is quite challenging. If you’re unfamiliar with PDFs, I recommend the O’Reilly book Developing with PDF.

Getting the Right Size

Most sewing patterns store the cut lines for different sizes in original content groups (known as layers in most PDF readers). This allows you to turn off the sizes you don’t want before printing.

I set up a command argument parser to define the filename of the PDF and get the size to extract.

parser = argparse.ArgumentParser(
    description='Generate new pattern pieces from existing patterns')
parser.add_argument('--filename', type=str, help='The filename of the pdf pattern.')
parser.add_argument('--size', type=str, help="The size of the pattern to analyze.")

The content groups are defined both in the PDF document’s root, and on each page. The group keys for the content groups are defined under /Resources -> /Properties.

    args = parser.parse_args()
    x = PdfReader(args.filename, decompress=True)
    name = '(' + args.size + ')'
    shapes = []
    paths = []
    for page_num, page in enumerate(x.pages):
        if '/Resources' not in page:
            continue
        if '/Properties' not in page['/Resources']:
            continue
        oc_keyname = [key for key in page['/Resources']['/Properties']
                      if page['/Resources']['/Properties'][key]['/Name'] == name]

If there no original content groups matching the size we want, skip to the next page:

        if len(oc_keyname) == 0:
            continue

Another thing to extract out of the /Resources are the graphics states. If you want a faithful color representation of the graphics when converting PDF to SVG, you need these values. However, since these rules are a bit tricky, and I only care about the shapes, I am not doing anything meaningful with this information at the moment:

        gstates = {}
        if '/ExtGState' in page['/Resources']:
            gstates = page['/Resources']['/ExtGState']

The paths themselves will be in the page’s content stream. You want to start reading at the group key name, and then end at the end of that block, which is typically indicated by the EMC keyword:

        lines = page.Contents.stream.split('\n')
        start_index = [i for i, l in enumerate(lines) if l.find(oc_keyname) >= 0][0]
        end_index = \
        [i for i, l in enumerate(lines) if l.find('EMC') >= 0 and i > start_index][0]
        shape = "\n".join(lines[start_index:end_index])

PostScript Graphics to SVG

PDF graphics use a format similar to SVG for expressing shapes and paths, but uses a PostScript state machine for updating stylistic elements (like fill and stroke color) and for appending to paths. The instructions are written in reverse Polish notation, where the q operation pushes values onto the state machine and Q pops off the stack. I wrote a PostScript interpreter that goes over the instructions line by line, and adds shapes to a svgwrite Drawing. It’s by no means complete, but it’s good enough for my purposes. Some operators, like n, define clipping paths so that the cut lines render within the page window, and I ignore these.

def parse_shape(shape, i, gstates):
    # see https://www.adobe.com/content/dam/acom/en/devnet/acrobat/pdfs/PDF32000_2008.pdf
    output_filename = "page%s.svg" % i
    dwg = Drawing(output_filename, profile='tiny')
    fill = "none"
    stroke = rgb(0, 0, 0)
    stroke_width = 4
    transform = (1, 0, 0, 1, 0, 0)
    shapes_stack = []
    d = ""
    paths = []
    for line in shape.split("\n"):
        line = line.strip()
        parts = line.split(" ")
        nums = []
        for part in parts:
            try:
                nums.append(float(part))
            except ValueError:
                pass
        operation = parts[-1]
        if operation == 'BDC':
            continue
        elif operation == 'q':
            # q - start stack
            continue
        elif operation == 're':
            # rectangle
            vals = {'insert': (nums[0], nums[1]), 'size': (nums[2], nums[3])}
            if fill:
                vals['fill'] = fill
            if stroke:
                vals['stroke'] = stroke
            shapes_stack.append(dwg.rect(**vals))
        elif operation == 'n':
            # - clipping path
            continue
        elif operation == 'RG':
            # set stroke color
            stroke = rgb(*nums[0:3])
        elif operation == 'J':
            # not sure how to implement cap styles
            continue
        elif operation == 'cm':
            # current transformation matrix
            transform = nums[0:6]
        elif operation == 'F' or operation == 'f':
            # fill
            fill = rgb(*nums[0:3])
        elif operation == 'm':
            # move to
            d += "M " + format_pointstr(parts[0:2])
        elif operation == 'c':
            # curve
            d += " C " + format_pointstr(parts[0:6])
        elif operation == 'v':
            # append to bezier curve
            d += " S " + format_pointstr(parts[0:4])
        elif operation == 'y':
            d += " C " + format_pointstr(parts[0:4])+" "+format_pointstr(parts[2:4])
        elif operation == 'l':
            # line to
            d += " L " + format_pointstr(parts[0:2])
        elif operation == 'h':
            # make sure it's a closed path
            continue
        elif operation == 'S':
            # stroke to 4-unit width
            continue
        elif operation == 'Q':
            # end stack (draw)
            # apply transformation...
            for shape in shapes_stack:
                dwg.add(shape)
            if len(d) > 0:
                paths.append(d + " Z")
                d = " ".join([transform_str(p, transform) for p in d.split(" ")])
                vals = {'d': d}
                vals['stroke-width'] = stroke_width
                if fill:
                    vals['fill'] = fill
                if stroke:
                    vals['stroke'] = stroke

                dwg.add(dwg.path(**vals))
            d = ''
            shapes_stack = []
        elif operation == 'gs':
            key = parts[0]
            if key not in gstates:
                print("could not find state %s in dictionary")
            state = gstates[key]
            # color blending not yet implemented
            pass
        elif operation == 'w':
            stroke_width = nums[0]
        else:
            print("not sure what to do with %s %s" % (operation, line))
    dwg.save()
    return paths

PDF uses the same format as SVG for expressing transformations. Here’s my code for applying these transformations, which is copied from my fork of svgpathtools

def transform_point(point, matrix=(1, 0, 0, 1, 0, 0), format="float", relative=False):
    a, b, c, d, e, f = matrix
    if isinstance(point, list):
        x, y = point
    else:
        point_parts = point.split(',')
        if len(point_parts) >= 2:
            x, y = [float(x) for x in point_parts]
        else:
            # probably got a letter describing the point, i.e., m or z
            return point
    # if the transform is relative, don't apply the translation
    if relative:
        x, y = a * x + c * y, b * x + d * y
    else:
        x, y = a * x + c * y + e, b * x + d * y + f
    if format == "float":
        return x, y
    else:
        return "%s%s%s" % (x, point_separator, y)

Since the same cut lines are spread accross multiple pages, I want to eliminate duplicate cut lines, which I can do using set, since the shapes are all path strings:

paths = sorted(list(set(paths)))

Here’s all the unique cut lines for the XS shape:

Now that I have the cut lines in SVG format, I can programmatically manipulate them and then print them out again, but that’s a post for another day.

Within five minutes of releasing the first version of my web interface to my python-embroidery package, embroidery-maker.com, it was broken by my first user uploading a PNG image instead of an SVG. Though SVGs are much easier to work with, as the list of stitches that gets sent to the sewing machine is much closer in data to a vector than a grid of pixels, I recognize that my user-base is more familiar with opening and editing PNGs than SVGs. Luckily adding support for PNGs was relatively painless thanks to the Potrace contour-tracing tool, and the python bindings.

First, I convert the individual pixels in the image into valid thread colors. At this point, it might also be useful to restrict the total number of colors in the image, which can be done using the Python Imaging Library’s convert method. My sewing machine seems willing to support an arbitrarily large number of different thread colors in patterns, but it can be a pain to have to take out and re-thread the machine many times.

def posturize(_image):
    pixels = defaultdict(list)
    for i, pixel in enumerate(_image.getdata()):
        x = i % _image.size[0]
        y = int(i/_image.size[0])
        if len(pixel) > 3:
            if pixel[3] == 255:
                pixels[nearest_color(pixel)].append((x,y, pixel))
        else:
            pixels[nearest_color(pixel)].append((x, y, pixel))
    return pixels

Next, I trace the image. Potrace can only handle two-color images, so I keep track of all of the thread colors, then go make a masked image with only the pixels of that thread color.

for color in pixels:
    data = zeros(_image.size, uint32)
    for pixel in pixels[color]:
        data[pixel[0], pixel[1]] = 1
    # Create a bitmap from the array
    bmp = potrace.Bitmap(data)
    # Trace the bitmap to a path
    path = bmp.trace()

Next, I want to convert this traced path into the svgpathtools curve objects, so that they can be used in the rest of the digitizer code as if it was parsed in from an SVG. I also need to keep track of the start locations of each segment, so that I can check whether the segment is a closed loop. If it is, I can add a fill and stroke color.

# Iterate over path curves
for curve in path:
    svg_paths = []
    start_point = curve.start_point
    true_start = curve.start_point
    for segment in curve:
        if true_start is None:
            true_start = segment.start_point
        if start_point is None:
            start_point = segment.start_point
        if isinstance(segment, BezierSegment):
            svg_paths.append(
                CubicBezier(start=start_point[1] + 1j * start_point[0],
                            control1=segment.c1[1] + segment.c1[0] * 1j,
                            control2=segment.c2[1] + segment.c2[0] * 1j,
                            end=segment.end_point[1] + 1j * segment.end_point[0]))
        elif isinstance(segment, CornerSegment):
            svg_paths.append(Line(start=start_point[1] + 1j * start_point[0],
                                  end=segment.c[1] + segment.c[0] * 1j))
            svg_paths.append(Line(start=segment.c[1] + segment.c[0] * 1j,
                                  end=segment.end_point[1] + 1j *
                                                             segment.end_point[
                                                                 0]))
        else:
            print("not sure what to do with: ", segment)
        start_point = segment.end_point
        # is the path closed?
        if true_start == start_point:
            output_paths.append(Path(*svg_paths))
            color = pixel[2]
            rgb = "#%02x%02x%02x" % (color[0], color[1], color[2])
            fill = rgb
            attributes.append({"fill": fill, "stroke": rgb})
            true_start = None
            start_point = None
            svg_paths = []

Then, I need to cut the shapes above out of the shapes below so that the SVG is only one layer deep.

Feeding these paths into my digitizer, it can turn PNG images like¹:

into:

1: I’ve replaced the lion rampant on the flag of Nova Scotia with the emoji cat because it has too many curves to render nicely in stitches, and also because it amuses me

SVGs will display the last added shape on top. This means you can very easily create the Canadian flag by layering a white rectangle on top of a red rectangle, but that correctly representing merged SVGs is difficult. It also means that, if you’re using SVGs to generate things in meatspace, like with a 3D printer, laser cutter or computerized sewing machine, you’re going to waste material and reduce the quality of the output. Before sending the SVG to the device, you’ll probably want to cut out the later shapes out of the lower shapes. Doing this is kind of tricky, and I’ll go over two methods that work, with trade-offs and one complicated method that I haven’t completed implementing. The code for this post is in my python-embroidery library.

Subtract the Paths

In SVGs, if you add one path string to another after the first string, SVG will render the second d string as a cut-out of the first d string. This means that you can go through the SVG paths and add the next two together:

for i in range(1, len(all_paths)):
    remove_path = all_paths[i].d()
    current_path = all_paths[i-1].d()
    all_paths[i-1] = parse_path(current_path+" "+remove_path)

This works well for shapes without the holes, so for example the shape:

will be cut into the shapes:

However, for shapes that already have holes, like the shape:

will be cut into the shapes:

Cutting a path on an existing negative space turns it into a positive space. This makes sense if you have only an on or off setting for each fill area. So this method works well for simple shapes, but not for shapes that already have negative spaces. As soon as you try to stack multiple shapes, you’re going to be double-negating lots of shapes.

Use Computational Geometry

Computational Geometry is a tricky subject, luckily we can leverage that is used extensively in GIS code to avoid doing our own calculations. The Shapely python package has functions to perform all set operations on polygon points, including differencing. It doesn’t support curves, so you must first convert the SVG paths into polygons, do the difference, then convert the polygon into an SVG path:

from svgpathtools import Line, Path
from shapely.geometry import Polygon

def path_difference_shapely(path1, path2):
    # convert both paths to polygons
    def path_to_poly(inpath):
        points = []
        for path in inpath:
            if isinstance(path, Line):
                points.append([path.end.real, path.end.imag])
            else:
                num_segments = ceil(path.length() / minimum_stitch)
                for seg_i in range(int(num_segments + 1)):
                    points.append([path.point(seg_i / num_segments).real,
                                    path.point(seg_i / num_segments).imag])
        return Polygon(points)
    poly1 = path_to_poly(path1)
    poly2 = path_to_poly(path2)
    diff_poly = poly1.difference(poly2)
    points = diff_poly.exterior.coords
    new_path = []
    for i in range(len(points)-1):
        new_path.append(Line(start=points[i-1][0]+points[i-1][1]*1j,
                             end=points[i][0]+points[i][1]*1j))
    new_path.append(Line(start=points[-1][0]+points[-1][1]*1j,
                             end=points[0][0]+points[0][1]*1j))
    # make a new path from these points
    return Path(*new_path)

Here I’m sampling along any non-Line segments in the curve at intervals of minimum_stitch, which is the smallest stitch I’m going to tell my computerized sewing machine to make.

This cuts out our negative space the way we’d like:

Which is what we want. Unfortunately, doing this requires transforming our wonderful continuous Bezier curves into gross discrete polygons. This isn’t a problem if you know the lower resolution of the device that is going to render your SVG, like the minimum stitch of my sewing machine.

Use Inkscape

So what if you want to preserve your curves? Inkscape is able to do differences between shapes remarkably well, unfortunately their code for doing this is a thousand lines that is not linted, has no comments, and has methods and variables written in Franglish. Eventually, I’d like to understand what it is doing. Inkscape is a really wonderful tool and I understand that it is difficult to make a professional-level codebase on 0 income. If you want to just use Inkscape’s operations in your python code, you can use the inx-pathops package.

If you want to use google’s cloud libraries so that you can take advantage bigquery or cloudstorage, you need to add them to the code you deploy with your app. Typically, I check out the repository to a lib directory, then put the lines:

import os
import sys
# path to lib directory
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib'))

in the file appengine_config.py.

This works when the code is deployed, but it doesn’t work locally, where the lib/google folder conflicts with the google-cloud-sdk/platform/google_appengine/google folder, resulting in the error:

ImportError: No module named google.cloud.bigquery

or

ImportError: No module named google.api_core

The solution I’ve found is to initialize the files lib/google/__init__.py and lib/google/cloud/__init__.py (so that the version of python running appengine can recognize these folders as packages) and then reload the google module in appengine_config.py:

import os
import sys
# path to lib directory
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib'))
# google needs to be reloaded so that it is taken from the 'lib' folder, and not the
# sandbox environment. Google needs to be loaded once to trigger python to look for the
# module
import google
reload(google)
# after reloading, trigger importing sub modules so that they are in the sys.modules
# dictionary in the correct location on future handlers
import google.api_core
import google.cloud
import google.cloud.bigquery

I’m not sure why this works, but it does.

Most paper cutters, laser engravers, and 3D printers can’t handle fonts, and so ignore 〈text〉 tags in SVGs. Instead, text elements need to first be converted to paths by a program like Inkscape. This post describes how to automate text to path conversion in python using the freetype, svgpathtools, and svgwrite libraries.

I used the freetype rendering in matplotlib example as a guide.

First, load the svgpathtools paths, and define a tuple_to_imag function to convert between freetype’s tupple point representation to svgpathtools’ imaginary numbers:

from svgpathtools import wsvg, Line, QuadraticBezier, Path
def tuple_to_imag(t):
    return t[0] + t[1] * 1j

Load your font and initialize a character:

from freetype import Face
face = Face('./Vera.ttf')
face.set_char_size(48 * 64)
face.load_char('a')

We’ll be converting a string character by character. After converting this character to a path, you use the same method to convert the next character to a path, offset by the kerning:

face.get_kerning('a', 'b')

You’ll need to flip the y values of the points in order to render the characters right-side-up:

outline = face.glyph.outline
y = [t[1] for t in outline.points]
# flip the points
outline_points = [(p[0], max(y) - p[1]) for p in outline.points]

The face has three lists of interest: the points, the tags, and the contours. The points are the x/y coordinates of the start and end points of lines and control points. The tags indicate what type of point it is, where tag values of 0 are control points. Finally, the contours are the end point list index for each shape. Characters like i or ! have two shapes, most others have only one contour. So, for each contour, we want to pick out only the tags and points for that contour.

start, end = 0, 0
paths = []

for i in range(len(outline.contours)):
    end = outline.contours[i]
    points = outline_points[start:end + 1]
    points.append(points[0])
    tags = outline.tags[start:end + 1]
    tags.append(tags[0])

Next, we want to split the points up into path segments, using the tags. If the tags are 0, add the point to the current segment, else create a new segment, so that control points stay with their path segments:

    segments = [[points[0], ], ]
    for j in range(1, len(points)):
        segments[-1].append(points[j])
        if tags[j] and j < (len(points) - 1):
            segments.append([points[j], ])

Then convert the segments to lines. For lines with two control points (segment length 4), I could use the CubicBezier, but I find that breaking it into two Quadratic Beziers where the end point for the first and the start point of the second curve is the average of the control points, is more attractive:

    for segment in segments:
        if len(segment) == 2:
            paths.append(Line(start=tuple_to_imag(segment[0]),
                              end=tuple_to_imag(segment[1])))
        elif len(segment) == 3:
            paths.append(QuadraticBezier(start=tuple_to_imag(segment[0]),
                                         control=tuple_to_imag(segment[1]),
                                         end=tuple_to_imag(segment[2])))
        elif len(segment) == 4:
            C = ((segment[1][0] + segment[2][0]) / 2.0,
                 (segment[1][1] + segment[2][1]) / 2.0)

            paths.append(QuadraticBezier(start=tuple_to_imag(segment[0]),
                                         control=tuple_to_imag(segment[1]),
                                         end=tuple_to_imag(C)))
            paths.append(QuadraticBezier(start=tuple_to_imag(C),
                                         control=tuple_to_imag(segment[2]),
                                         end=tuple_to_imag(segment[3])))

Set the start location to the end location and continue. You can use the svgpathtools Path to merge the paths:

    start = end + 1

path = Path(*paths)
wsvg(path, filename="text.svg")

text.svg looks like:

You can see the full code in this gist.

If you want to create or manipulate SVGs, there several great JavaScript libraries, such as d3 or Snap.js. However, right now I just want to stack a set of already-existing SVGs. I’ve created a gist with my code to stack SVGs, I’m going to go over it in this post.

This post assumes that the SVGs to stack are the strings of the file contents of the SVGs. You can acquire these strings by either reading in a user-supplied input file, or by making an HTTP request, but that is beyond the scope of this post.

Quick and Dirty

To extract the svg contents out of a string, you could do some string manipulations to strip everything up to the opening <svg> tag and after the closing </svg> tag. But since JavaScript was created with the goal of being able to manipulate XML files, this seems unnecessary. Instead, I use the DOMParser object to extract the SVG contents:

function getSVGContents(inputString){
    let domParser = new DOMParser();
    let svgDOM = domParser.parseFromString(inputString, 'text/xml')
        .getElementsByTagName('svg')[0];
    return svgDOM.innerHTML
}

Then, you can simply concatenate the strings:

document.getElementById("quickSVG").innerHTML =
         getSVGContents(circle)+getSVGContents(star);

SVGs have no z-index, like CSS. The elements that appear on the top are just the last elements to be added. So the above code would create the svg:

Whereas if you change the order of ‘circle’ and ‘star’ in the string concatenation:

document.getElementById("quickSVG").innerHTML =
         getSVGContents(star)+getSVGContents(circle);

The star will now be under the circle:

Less Quick, more OOPy

The above method is okay if you want to just layer the SVGs without manipulating them. But since we already have our strings parsed to DOM elements, we can combine the elements directly. We can initialize a SVG element, then use appendChild to move the child nodes from one element to another:

function addSVGs(inputStrings){
    // takes a list of strings of SVGs to merge together into one large element
    let svgMain = document.createElement("svg");
    for(let stringI=0;stringI<inputStrings.length;stringI++){
        let domParser = new DOMParser();
        let svgDOM = domParser.parseFromString(inputStrings[stringI], 'text/xml')
            .getElementsByTagName('svg')[0];
        while(svgDOM.childNodes.length>0){
            svgMain.appendChild(svgDOM.childNodes[0]);
        }
    }
    return svgMain

}
var svgMain = addSVGs([circle, star]);
document.getElementById("appendSVG").innerHTML = svgMain.innerHTML;

The things I found confusing about this method were:

• first, unlike other XML parsers, DOMParser does not ignore whitespace. The tabs I put in to make my SVG strings more readable get read in as #text nodes. These strings will get rendered as whitespace, which don’t effect the final SVG.
• appendChild removes the element from the input element’s parent, hence why it’s the first childNodes added to the mainSVG, and why I keep appending until there are no more childNodes in the svgDOM.

If you’re building a Big Data application on AppEngine, you’ll probably need to grab more than a thousand entries out of a search index. It turns out this is not as straightforward as expected. I’ve built a sample project to demo how to do this.

In this demo we initialize our index to have 3000 entries of the numbers between 0 and 3000:

search_index = search.Index(name=INDEX_NAME, namespace=NAMESPACE)
for i in range(3000):
    fields = [search.NumberField(name='number_value', value=i),
              search.NumberField(name='last_modified', value=int(time()))]
    doc = search.Document(fields=fields)
    results = search_index.put(doc)

A first attempt at building a handler to query this index would be:

class Search(webapp2.RedirectHandler):
    def get(self):
        try:
            limit = int(self.request.get('limit'))
        except:
            limit = DOCS_PER_PAGE_LIMIT
        try:
            offset = int(self.request.get('offset'))
        except:
            offset = 0

        expr_list = [
            search.SortExpression(
                expression='number_value',
                default_value=int(0),
                direction=search.SortExpression.ASCENDING
            )
        ]
        sort_opts = search.SortOptions(expressions=expr_list)
        index = search.Index(name=INDEX_NAME, namespace=NAMESPACE)
        search_query = search.Query(
            query_string='',
            options=search.QueryOptions(
                limit=limit, sort_options=sort_opts, offset=offset, ids_only=False)
        )

This works well for values under 1000, but if you try to run the query /search?limit=3000, you’ll get the error:

Traceback (most recent call last):
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__
    rv = self.handle_exception(request, response, e)
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__
    rv = self.router.dispatch(request, response)
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/home/catherine/python_projects/appengine-large-offset-example/main.py", line 55, in get
    limit=limit, sort_options=sort_opts, offset=offset, ids_only=False)
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/google/appengine/api/search/search.py", line 3073, in __init__
    self._limit = _CheckLimit(limit)
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/google/appengine/api/search/search.py", line 2858, in _CheckLimit
    upper_bound=MAXIMUM_DOCUMENTS_RETURNED_PER_SEARCH)
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/google/appengine/api/search/search.py", line 496, in _CheckInteger
    raise ValueError('%s, %d must be <= %d' % (name, value, upper_bound))
ValueError: limit, 3000 must be <= 1000

We can’t get all 3000 entries in one query, so we’ll have to spread the acquisition over three queries:

• /search?limit=1000
• /search?limit=1000&offset=1000
• /search?limit=1000&offset=2000

The first two queries work, but if you set offset to 2000, you’ll get the error:

Traceback (most recent call last):
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__
    rv = self.handle_exception(request, response, e)
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__
    rv = self.router.dispatch(request, response)
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/home/catherine/python_projects/appengine-large-offset-example/main.py", line 55, in get
    limit=limit, sort_options=sort_opts, offset=offset, ids_only=False)
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/google/appengine/api/search/search.py", line 3082, in __init__
    self._offset = _CheckOffset(offset)
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/google/appengine/api/search/search.py", line 2895, in _CheckOffset
    upper_bound=MAXIMUM_SEARCH_OFFSET)
  File "/home/catherine/installed/google-cloud-sdk/platform/google_appengine/google/appengine/api/search/search.py", line 496, in _CheckInteger
    raise ValueError('%s, %d must be <= %d' % (name, value, upper_bound))
ValueError: offset, 2000 must be <= 1000

Darn. Because of these limitations, we’re going to need a concept from Databases usually reserved for operations that span multiple computers: a cursor.

You can set as a parameter to the query options:

search_query = search.Query(
    query_string='',
    options=search.QueryOptions(
        limit=self.limit,
        sort_options=self.sort_opts,
        cursor=search.Cursor(per_result=False),
        ids_only=False,
    )
)

you can access the cursor for this search query after getting the results:

search_results = self.index.search(search_query)
cursor = search_results.cursor        

The strategy for pulling out the entries is this:

1. create a search with no offset and a new cursor
2. initialize current_offset to 0
3. while current_offset is less than the desired offset, query the search index again with the cursor of the last query, and add the limit to the current_offset
4. output the last search results

It would be nice if we could start our loop at the maximum offset in order to limit the number of repetitions of step 3, but if you do, you’ll get the error:

ValueError: cannot set cursor and offset together

Also, make sure that you set the per_result value of search.Cursor to False, or else the cursor will be None:

  File "/home/catherine/python_projects/appengine-large-offset-example/main.py", line 109, in query_with_cursors
    cursor_cache[current_offset] = search_results.cursor.web_safe_string
AttributeError: 'NoneType' object has no attribute 'web_safe_string'

Here’s the implementation:

current_offset = self.limit
search_query = search.Query(query_string='',
    options=search.QueryOptions(
        limit=self.limit,
        sort_options=self.sort_opts,
        cursor=search.Cursor(per_result=False),
        ids_only=False,
    )
)

search_results = self.index.search(search_query)
if self.offset >= search_results.number_found:
    return self.render_search_doc([])
while current_offset < self.offset:
    current_offset += self.limit
    search_query = search.Query(query_string='',
        options=search.QueryOptions(
            limit=self.limit,
            sort_options=self.sort_opts,
            cursor=search_results.cursor
        )
    )
    search_results = self.index.search(search_query)
self.render_search_doc(search_results)

Now, if we’re going to be querying an offset of up to 100000, it will take a while to do 1000 queries. So, it would be better if we could cache these cursors. We can stick the web_safe_string in memcache:

cursor_cache = {}
current_offset = self.limit
search_query = search.Query(query_string='',
    options=search.QueryOptions(
        limit=self.limit,
        sort_options=self.sort_opts,
        cursor=search.Cursor(per_result=False),
        ids_only=False,
    )
)

search_results = self.index.search(search_query)
cursor_cache[current_offset] = search_results.cursor.web_safe_string
if self.offset >= search_results.number_found:
    return self.render_search_doc([])
while current_offset < self.offset:
    current_offset += self.limit
    search_query = search.Query(query_string='',
        options=search.QueryOptions(
            limit=self.limit,
            sort_options=self.sort_opts,
            cursor=search_results.cursor
        )
    )
    search_results = self.index.search(search_query)
    cursor_cache[current_offset] = search_results.cursor.web_safe_string
memcache.set(cursor_cache_key, json.dumps(cursor_cache))
self.render_search_doc(search_results)

And then the next time, we can pull out the cursor for that offset:

cursor_cache_key = 'cursors'
cursor_cache = memcache.get(cursor_cache_key)
if cursor_cache:
    cursor_cache = json.loads(cursor_cache)
    offset_key = str(self.offset)
    if offset_key in cursor_cache:
        cursor_cache = cursor_cache[offset_key]
    else:
        logging.info("%s not in %s" %(offset_key, cursor_cache))
        cursor_cache = None

if cursor_cache:
    logging.info("found cursor cache string %s " % cursor_cache)

    # construct the sort options
    search_query = search.Query(
        query_string='',
        options=search.QueryOptions(
            limit=self.limit,
            sort_options=self.sort_opts,
            cursor=search.Cursor(per_result=False, web_safe_string=cursor_cache),
            ids_only=False,
        )
    )
    return self.render_search_doc(self.index.search(search_query))

A next step would be to find the nearest cursor offset, and start from there. But that would be another blog post.

I have a dilemma: I don’t want to buy stuff for holidays, but I also don’t want to be the only house on my street without Christmas lights. So I made myself some Christmas lights from:

• the leftover 5050 RGB LEDs from my last PC build
• the 12 V DC power supply from my old wifi router
• the leftover canvas from making a bike bag
• a leftover dowel rod from making a bookcase

I made a model of an individual LED strip in Inkscape and experimented with a few designs. The one I ultimately chose was these snowflakes:

I chose this design because it had relatively few angles, which meant I could free-hand layout the LEDs without needing to project the design onto the fabric:

I soldered the 12V and Blue Grounds on each snowflake, and then hot-glued some of the longer wires down to the fabric:

This is the end result:

I recently purchased a Computerized Sewing Machine. Designs can be uploaded to internal storage via USB, which ubuntu automatically recognizes and mounts:

This compatibility with linux out of the box is fantastic; unfortunately designs must be in Brother’s undocumented binary file format, and the sewing machine gives no feedback regarding parsing errors. This means that I’ve been creating binaries for embroidery patterns with a lot of trial and error. In order to script some of the process, I’d like to upload and test new designs without human intervention.

To find where ubuntu has mounted the sewing machine, I could either:

1. write some udev rules
2. get python to figure it out from linux commands

In this case, method 2 is a bit nicer since I’m already using Python to generate the binary files.

First, I use the output of df to generate a mapping of where ubuntu has mounted the external drives:

mount_points = {}
out = check_output(["df"]).split("\n")
for line in out:
    if line.find("/media") >= 0:
        mount_location = line.split("%")[-1]
        line = line.split(" ")
        mount_points[line[0]] = mount_location.strip()

Next, get the vendor ids for each device, and if it matches, call that my mount location:

mount_destination = None
for mount_point in mount_points:
    out = check_output(["udevadm", "info", "--name="+mount_point, "--attribute-walk"])
    if out.find('ATTRS{idVendor}=="04f9"')> 0:
        print("found brother device, assuming embroidery machine")
        mount_destination = mount_points[mount_point]
        break

So I can now use shutil.copyfile() to move the files over to mount_destination. This is kind of a hacky solution because it may also catch Brother printers, or fail to recognize embroidery machines from other manufacturers, but since I have neither of these situations, it’s good enough for now.

My husband and I disagree about whether it’s more energy efficient to use the toaster oven or regular oven for baking. He believes the toaster oven consumes less energy, since the internal volume is much smaller. I believe the oven consumes less because it is better insulated.

I did some back-of-the-envelope calculations to see if there was an obvious conclusion. I estimate that the total power consumed is the work needed to heat up the volume of air inside the oven to the desired temperature, plus the work lost to heat transfer out of the door. It’s also possible that the heat is lost through the other sides, but I didn’t take that into account. From the ideal gas law, the work to heat up the air inside the oven is:

\[ W = D_mVR\triangle T \]

Where \(D_m\) is the molar density of air, \(V\) is the volume, \(R\) is the ideal gas constant, and \(\triangle T\) is the temperature change.

The heat lost through the oven door is:

\[ Q = kA\triangle Tt \]

Where \(k\) is the heat transfer coefficient, \(A\) is the area of the door, and \(t\) is the time that the oven is on.

Since both ovens have the same depth, the ratio of the ovens’ volume, 2.8, is the same for the door area. Some mechanical engineers estimate that the \(k\) of double-paned oven doors is ~3, whereas the toaster oven’s single pane of glass is 5.8. The ratio of these coefficients almost exactly cancels out the difference in volume. Because the heat is constantly being leaked out of the ovens, the work due to lost heat is nearly an order of magnitude larger than work required to heat up the oven, so this contribution can be mostly ignored. Since we don’t exactly know \(k\) for both ovens, we’re going to need to get experimental.

I bought an Aeotek Smart Energy Meter (currently on Amazon for 32 USD) and an Aeotek Z-stick (currently on Amazon for 45 USD)¹. The Z-Stick is compatible with open-zwave which has a python interface.

My code to grab the power data on linux is:

import time
from openzwave.option import ZWaveOption
from libopenzwave import PyManager
import csv
from matplotlib import pyplot

device="/dev/ttyUSB0"
options = ZWaveOption(device, config_path="/etc/openzwave/", user_path=".", cmd_line="")
options.set_console_output(False)
options.lock()


class PowerMeter(PyManager):
    def __init__(self):
        super(PowerMeter, self).__init__()
        self.create()
        self.addWatcher(self.callback)
        time.sleep(1.0)
        self.addDriver(device)
        self.meter_values = []
        self.start_time = None
        self.units = None
        self.value_id = None

    def get_meter_values(self, num_seconds):
        self.start_time = time.time()
        while time.time() - self.start_time < num_seconds:
            time.sleep(0.10)
            if self.value_id:
                assert self.refreshValue(self.value_id)

    def callback(self, zwargs):
        # get measurement value from the first message tagged with the sensor we want
        if "valueId" in zwargs:
            if zwargs["valueId"]["commandClass"] == "COMMAND_CLASS_METER":
                if zwargs["valueId"]["label"] == "Power" and \
                                zwargs["valueId"]["instance"] == 1 \
                        and self.start_time is not None:
                    self.value_id = zwargs["valueId"]["id"]
                    self.units = zwargs["valueId"]["units"]
                    self.meter_values.append((time.time() - self.start_time,
                                              zwargs["valueId"]["value"]))
                    print("Power", zwargs["valueId"]["value"])


manager = PowerMeter()
# collect data for an hour
manager.get_meter_values(60*60)
# add to a csv
with open('meter_values_%s.csv' % manager.start_time, 'wb') as csvfile:
    spamwriter = csv.writer(csvfile, delimiter=',', quoting=csv.QUOTE_MINIMAL)
    spamwriter.writerow(['time (s)', 'meter (%s)' % manager.units])
    for row in manager.meter_values:
        spamwriter.writerow(row)

fig, ax = pyplot.subplots()
ax.plot([point[0]/60.0 for point in manager.meter_values],
            [point[1] for point in manager.meter_values])
ax.set_xlabel("Time (minutes)")
ax.set_ylabel("Meter Value (%s)" % manager.units)
fig.savefig("power_meter_%s.png" % manager.start_time)

I believe that instance 1 is the power on the first probe, which is attached to the hot wire of my house’s main power. Since probe 2 is attached to the neutral wire, I believe it’s possible to cancel out most of the noise by subtracting the power on instance 2 from the power on instance 1. I have no idea how these map to European/Australian three phase systems.

With the script running, I collected three sets of data: one while the toaster cooked a sheet of cookies, one while the oven cooked a sheet of cookies, and one with neither oven running, just the house lights, routers, and refridgerator. The data, from the start of pre-heating to when the cookies came out, looks like this:

The toaster took longer to pre-heat, so it had a longer data collection time. It looks like both ovens and the refridgerator (in the base usage) are driven by square forcing functions that turn off when a desired temperature is reached. The oven’s forcing function is a bit more sophisticated than the toaster; it looks like it can heat at two different wattage levels. I think the long period of the oven been off while cooking is proof that the oven does have a lower heat transfer coefficient than the toaster oven.

I can integrate the data I collected using scipy’s integrate and numpy’s interpolate functions:

from scipy import integrate
from numpy import interp

result = integrate.quad(lambda x: interp(x, data_array[0], data_array[1]),
                        min(data_array[0]), max(data_array[0]))[0] 

I also subtract off the base usage in each case. After integrating, I estimate that the toaster uses 0.45 kWh to bake a single sheet of cookies, and the oven uses 0.70 kWh.

The toaster is more efficient at baking a single sheet of cookies, so I’d conclude that the heat transfer coefficient of the toaster is not as bad as expected. However, the oven can handle two sheets at once, and is faster. If you want to shave 8 minutes off of your cook time, it’s only an extra 0.15 kWh, which in Florida costs two cents. However, the oven has an advantage over the toaster in that it can cook more than a single sheet at a time, and then the energy cost per cookie favors the oven.

So depending on the situation, we’re both right.

1: It is my personal opinion that Z-Wave compatible devices are 2-3 times more expensive than they should be, given the underlying hardware. I think this is because Z-Wave is not open source, so development takes longer and costs more. I would not invest in a Z-Wave system now. It would be nice to be able to turn off my overhead light from a smartphone app so that I don’t have to get out of bed, but just putting a long piece of twine on the pull chain seems to be an adequate solution.

Using matplotlib’s axes.transData and get_sample_data, it’s possible to replace data markers with images, such as the poo emoji:

This is fun and silly, but it’s also important for accessibility for people with colorblindness or with shitty printers, like me.

axes.transData can transform x/y coordinates into the plot’s pixel location on the plot output. In order to then convert the pixel location to the figure’s 0.0-1.0 scale, divide by these locations by the size of the figure’s bounding box.

from os.path import join, dirname, abspath
from matplotlib import pyplot
from matplotlib.cbook import get_sample_data
from numpy import linspace
from numpy.core.umath import pi
from numpy.ma import sin

# poo-mark came from emojipedia:
# https://emojipedia-us.s3.amazonaws.com/thumbs/120/apple/96/pile-of-poo_1f4a9.png
poo_img = pyplot.imread(get_sample_data(join(dirname(abspath(__file__)), "poo-mark.png")))

x = linspace(0, 2*pi, num=10)
y = sin(x)

fig, ax = pyplot.subplots()
plot = ax.plot(x, y, linestyle="-")

ax_width = ax.get_window_extent().width
fig_width = fig.get_window_extent().width
fig_height = fig.get_window_extent().height
poo_size = ax_width/(fig_width*len(x))
poo_axs = [None for i in range(len(x))]
for i in range(len(x)):
    loc = ax.transData.transform((x[i], y[i]))
    poo_axs[i] = fig.add_axes([loc[0]/fig_width-poo_size/2, loc[1]/fig_height-poo_size/2,
                               poo_size, poo_size], anchor='C')
    poo_axs[i].imshow(poo_img)
    poo_axs[i].axis("off")
fig.savefig("poo_plot.png")

I recently came back to the Robot Operating System after not working with it for nearly 2 years. The great thing about ROS is that everything is a debian package and most of it is written in python, a language that I feel very comfortable with. The bad thing about ROS is that everything is a debian package and most of it is written in a language that I feel comfortable in. In my experience, 90\% of ROS problems are due to not having the right debian packages, including the first problem I encountered out of the gate.

I wanted to get my ros_realtime_plotter that I made for PyCon Canada 2015 working again. I followed my previous instructions starting with a fresh install of Ubuntu 16.04, but installed Kinetic Kame (instead of Indigo Igloo) and Gazebo 8.0.0 (instead of Gazebo 5.something). The first problem I encountered was some incompatibility between Kinetic Kame and Gazebo 8, so in future I would start off with gazebo 7.

My robot wouldn’t move. The only warning or error message was:

[WARN] [1508501276.314154, 1090.261000]: Controller Spawner couldn't find the expected controller_manager ROS interface.

This error message was not particularly useful to me. I didn’t understand what was meant by ROS interface. To learn more about this error message, I put ros_control in my workspace and modified the spawner script to print out the actual exception:

        rospy.logwarn("Controller Spawner couldn't find the expected controller_manager ROS interface. %s" % e)

The error message was a timeout waiting for the service /raz_bot/controller_manager/load_controller. Looking at rosservice, I could confirm that the service did not exist. I confirmed that my URDF had the code to load the gazebo ros controller:

  <!-- Gazebo plugin for ROS Control -->
  <gazebo>
    <plugin name="gazebo_ros_control" filename="libgazebo_ros_control.so">
      <robotNamespace>/</robotNamespace>
    </plugin>
  </gazebo>

It seemed like the robot spawner was just dying without a warning. So I set gazebo to have verbose output:

  <!-- Start the Gazebo simulator, loading a world -->
  <include file="$(find gazebo_ros)/launch/empty_world.launch">
    <arg name="world_name" value="$(find ros_realtime_plotter)/worlds/camera_above.world"/> <!-- world_name is wrt GAZEBO_RESOURCE_PATH environment variable -->
    <arg name="verbose" value="true" />
  </include>

This gave the error:

terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >'
  what():  boost: mutex lock failed in pthread_mutex_lock: Invalid argument

Not particularly helpful. But, in a semi-related github issue Michael Koval demonstrated how you could isolate problems with the URDF spawner shared object library by loading it in python.

I did that, and had no issues. The next step was to check the gazebo ros control shared object library. I looking for it showed me the source of my problem:

$ find / 2>/dev/null | grep "gazebo_ros_control" 
/home/cholloway/ros/ros_control_ws/src/ros_control/ros_control/documentation/gazebo_ros_control.png
/home/cholloway/ros/ros_control_ws/src/ros_control/ros_control/documentation/gazebo_ros_control.pdf
/home/cholloway/ros/ros_control_ws/src/ros_control/ros_control/documentation/gazebo_ros_control.odg

The only files related to gazebo_ros_control were guides as to how to use ros_control with gazebo in the ros_control repository! I had assumed that the gazebo package came with these, and I was wrong.

The packages I needed were:

sudo apt-get install ros-kinetic-gazebo-ros-control ros-kinetic-diff-drive-controller

after that, the ros_realtime_plotter worked.

Six months ago I moved to Florida and started working at GradientOne, a startup building a web-based platform for interacting with scientific and testing instruments and performing data analysis. It’s directly in the center of my interests and has kept me pretty occupied. Between my new job and working on the electrical wiring of my new house, I haven’t had much time for side-projects or reviewing the music I’ve been listening to.

However, I have been blogging for dollars about the things I’ve built for GradientOne, and I’m going to use this blog post as an opportunity to show off what I’ve built. If any of this functionality interests you, you should contact GradientOne to get API access or talk about your specific required functionality.

CANOpen Support

I added support for CANOpen devices, which was quite different from the SCPI-compliant devices GradientOne had previously supported. Unlike SCPI, I had never really worked on CAN protocols before, which was much more like TCP/IP than a serial communication protocol. I created a web-based CANOpen frame interpreter, so others learning CANOpen might find it easier in the future.

Trace Pattern Matching

There have been many occasions where in the process of debugging a piece of equiment, I have observed a plot and had it trigger a sense of deja-vu. Shapes can stick in memory a lot easier than a list of numbers. The trace pattern matching tool I created allows sections of a set of x-y data to be tagged and then searched against all previous trace data. This means that if there is a consistent blip in an experiment that happens one in a thousand times, the blip can be identified, even if the blip appears at different times and locations, or even multiple times. Once patterns have been defined, they can be used to define other measurements, so you could do something like calculate the time between the start of a clock signal and the appearance of a specific byte.

Pass/Fail Criteria definition

Most modern factories operate on PLCs, where each individual process boils down to true/false boolean values when determining future branching processes. To that end, I created an interface for defining tests to add pass/fail criteria to collected data. The Pass/Fail criteria can come at the end of a series of multiple different measurements. Calculated pass/fail measurements are added to the search index so that existing measurements can be quickly sorted based on passes and failures.

GradientOne Command Language

Knowing exactly how to communicate with a new device once you get it is an annoying pain. Although devices that are SCPI compliant are a bit easier to use than those that use some new communications protocol developed by an engineer throwing darts at a keyboard, as a lot of devices developed outside of the US seem to do, many device manufacturers fail to make their SCPI protocols SCPI-compliant. The GradientOne Command Language uses python-like syntax and tries to provide a consistent grammar for writing and querying instruments. Once a device’s functionality has been added to GradientOne, incorporating it into the Command Language means that the web editor can hint the user about all available functionality.

Those are the four big projects I’ve shipped in the past six months. I’m currently working on using machine learning for decoding digital signals, and have planned support for more devices.

Consider a python project with the directory structure:

├── inone
│   └── myfuncs.py
└── test.py

where myfuncs.py is:

def hello_world():
    print("hello world")

and test.py is:

from inone.myfuncs import hello_world

hello_world()

Running test.py with python 3.+ works as expected, but running in python 2 results in the error:

$ python2 test.py 
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    from inone.myfuncs import hello_world
ImportError: No module named inone.myfuncs

The solution is to add the file __init__.py to the inone directory with the code:

import myfuncs

However, this causes the error when running with python 3.+:

$ python3 test.py 
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    from inone.myfuncs import hello_world
  File "/home/catherine/python_projects/import_test/inone/__init__.py", line 1, in <module>
    import myfuncs
ImportError: No module named 'myfuncs'

The proposed solutions to this on the Stack Overflow question are unsatisfactory; they are either to add this compatibility to the setup script, which is useless for running code in the directory, or using importlib or appending to the PYTHONPATH, which seems unnecessarily complicated.

My own solution is to simply check for python3 in __init__.py:

import sys
if sys.version_info[0] < 3:
    import myfuncs

Of course, this is also a bit hacky and annoying, because these two lines need to be added to each directory’s __init__.py. Are there any better solutions out there?

Modules that are patched with mock’s patch decorator must be re-imported. This is a simple thing but I wasted a couple of hours this morning on it, so I’m documenting this for future reference.

Suppose you have two files:

parent.py

class MyParent(object):
    parent_name = "me"

child.py:

from parent import MyParent


class MyChild(MyParent):
    pass

And you would like to run some tests, one where the MyParent object is patched, and one where it is not. If you use this code for your tests:

from mock import patch


class MockParent(object):
    parent_name = "someone else"


def test_child():
    import child
    assert child.MyChild().parent_name == "me"


@patch("parent.MyParent", new=MockParent)
def test_mock():
    import child
    assert child.MyChild().parent_name == "someone else"

If you run these tests, only one test will work, depending on the order. I had assumed that imports were local to the scope of the function, but it appears that they aren’t. They aren’t even local to the file - if you split these tests up, only one will work, and it will be the one in the alphabetically first named file.

To run the tests as expected, you will need to re-import the child module. First, in order for this to work with both python2 and python3, import either imp or importlib

if sys.version_info >= (3, 0):
    import importlib
else:
    import imp as importlib

Then, after patching your code, re-import the patched library:

@patch("parent.MyParent", new=MockParent)
def test_mock():
    import child
    importlib.reload(child)
    assert child.MyChild().parent_name == "someone else"

I wasted a lot of time correctly defining angular modules. Here’s the problem and the solution.

Consider the following angular app:

<html ng-app="myApp">
<head>
<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.5.7/angular.min.js"></script>
<script type="text/javascript">
    var myApp = angular.module('myApp', []);
    myApp.controller('MyController', ['$scope', function($scope) {  
        $scope.result = 'test';
    }]);
    var myApp2 = angular.module('myApp');
    myApp2.controller('MyController2', ['$scope','$window', function($scope,$window) {
        $scope.result2 = 'test2';
    }])
</script>
</head>
<body>
  <div ng-controller="MyController">
    {{result}}    
  </div>
  <div ng-controller="MyController2">
    {{result2}}
  </div>
</body>
</html>

While creating both myApp and myApp2 is redundant, it is used in this example because eventually MyController and MyController2 will be separated into two separate files.

If you change the line:

var myApp = angular.module('myApp', []);

to:

var myApp = angular.module('myApp');

or change the line:

var myApp2 = angular.module('myApp');

to:

var myApp2 = angular.module('myApp', []);

The example code will no longer work. If you look at the console, you may see the nomod error. In the description, it says that adding the empty array of dependencies to angular.module defines a new module of name myApp, and calling angular.module without that argument returns a reference to an existing module of name myApp.

In the first change, removing the empty array results in an error due to angular looking up a module that does not exist. In the second change, adding the empty array results in an error due to angular attempting to define a module that already exists.

This implementation is confusing to me - I think that angular should instead accept angular.module(‘appname’) without a second argument when there are no dependencies, and have a separate function for returning references to the module. If angular.module was called more than once, it could then return an error message that the module is already defined. However, I’m a newcomer to Javascript and angular so there’s probably a good reason it isn’t implemented this way that I’m unaware of.

This implementation means the developer must take care to know which dependencies are needed accross all controllers they wish to use, and also to make sure that the module is created first, and any subsequent controllers get attached to a reference, rather than re-defining the module.

We’re not quite yet a week into 2017, so I’d better put down my thoughts on some albums that came out in 2016, but that didn’t quite fit any other posts, before they go stale.

Clarence Clarity - SAME and Vapid Feels are Vapid

Clarence Clarity is a British Progressive R&B artist. His 2015 debut album No Now is one of my favourites (he is currently ranked 47 in my music battle royale off of only one album). Clarence Clarity combines soulful singing with absurd lyrics, experimental sampling and production while still having a danceable beat. It’s the type of pop music I keep coming back to.

This year he released two songs - SAME (which was packaged as an album on soundcloud where every song is the same) and Vapid Feels are Vapid. SAME expands the repetoir of samples that can be incorporated into a functional R&B song, and the opening lines of the chorus are a nice mix of catchy rhyming and absurdity - Help! I been assaulted - I let the sun come and eat me up. Wait, no, I been exalted (Not really) I am a godless pyramid. It’s great, but I’m not sure it will stick around for long because it has an overly meandering ending.

Vapid Feels are Vapid in contrast, ends abruptly. It is closer to a traditional R&B song, and the singing and sentimentality are great. But it’s not as interesting as SAME. I like the direction he’s going in, but I hope that the songs on Clarence Clarity’s next album have more polished endings.

Be’lakor - Vessels

Be’lakor is an Australian Melodic Death Metal band beloved of /r/music. I liked their last albums on first listen, but soon got bored of them. I like the singing and the guitar solos, but dislike the overall composition, drumming and the repetitiveness of the guitar and bass lines. But, I had 10$ left in my bandcamp budget at the end of November so I decided to buy their latest album, Vessels.

I have two issues that keep me from liking Be’lakor. The first is that they can write metal epics, like An Ember’s Arc and Roots to Sever, but you have to listen to two minutes of the boring guitar lines and drumming first. The second issue is that the melodies on several songs, such as Withering Strands and Whelm, aren’t quite fleshed out in composition.

In short, I still don’t agree with /r/music.

Ghosts of Jupiter - The Great Bright Horses

)

Ghosts of Jupiter are a Boston-based psychadelic/progressive rock band. I first heard them when they were the opening act to Blue Öyster Cult, and ended up buying their 2011 debut album Ghosts of Jupiter.

The Great Bright Horses, their kickstarted followup, is leans into psychadelia and out of hard rock of their last album. The production quality is much better, and I appreciate the addition of the piano and flute. The lyrics and reverby melodies paint a picture of looking over a beautiful vista, like an early morning in a national park. It’s the perfect album to do yoga to. This album is a too sleepy - the song Lyra in particular has too many false starts and sounds like a Chopin piano lullaby. When the tempo does pick up, like on The Golden Age, the hard rock melodies are replaced by annoying 60s-style pop rock melodies that ruined the latest Purson album for me. This album is an improvement over their last album, but the absense of the harder grooves keeps it from being great.

FOES - The Summit Lies Skyward

FOES (Fall of Every Sparrow) are a British progressive band. They have the angelic vocals and alternative-rock drumming of a band like Keane crossed with the occasional harsh vocals and djent sounds of bands like Textures and TesseracT. I liked the single off of this album Beautiful Fiction, as well as their EP Ophir. The things I liked about the single generally apply to the rest of this album - I like the emotion, the nearly imperceptable counter-melodies. They blend styles really well. The lyrics are sometimes too wordy for the accompanying melody, like on the verses of Sworn Host and The Choir Invisible. But that complaint is minor. This band is something special.

CλaSH is a neat cross compiler from Haskell to VHDL, Verilog and SystemVerilog. Most examples were tested on an Altera FPGA (the DE0-Nano), and not Xilinx (the other major FPGA producer). This blog post demonstrates the procedure for getting Haskell code running on a Xilinx FPGA, which is as (relatively) straightforward as the Altera versions.

Specifically, these instructions are for the Embedded Micro Mojo, which is by far the best Xilinx development board.

Tools Setup

• Follow the CλaSH instructions for installing haskell, cabal and clash. CλaSH needs a specific version of haskell, so if you don’t follow their instructions you’re going to have a bad time.
• Follow Embedded Micro’s instructions for installing the Xilinx verilog compiler
• On linux, add the mojo’s udev rules
• Download and install the Mojo loader

Project Setup

First, generate some verilog from the blinker.hs example.

Start from the embedded micro sample base project. Remove the mojo_top.v file and add the verilog generated by CλaSH.

It will complain that there is no code defining the altpll50 module! The verilog written by CλaSH expects a clock generated by the Altera ALTPLL MegaFunction. To generate a clock that is similar in Xilinx, follow these steps:

• Open up the Xilinx Core Generator (Tools -> Core Generator)
• In Core Generator, open your current project (under File -> Open Project and picking the mojo_top.prj file.
• Make sure the chip is set to your spartan-6 chip. (for the mojo 2 board, this is XC6SLX9-2tqg144)
• Double click on Clocking Wizard in the IP catalog.
• Call the component altpll50, and change the input clock frequency to 50 MHz. On the next screen, change the output clock to 50 MHz
• On page 5 of the process, change the names of the inputs and outputs to match the Altera version. CLK_IN1 -> inclk0, CLK_OUT1-> c0, RESET -> areset, and LOCKED -> locked
• Wait for the Core Generator to make your verilog, then add it to your project using Project -> Add Sources. The altpll50.v should be in your top project directory. Do not add the UCF file!

Change the User Constraints File (.ucf) should be:

NET "CLOCK_50<0>" TNM_NET = clk;
TIMESPEC TS_clk = PERIOD "CLOCK_50<0>" 50 MHz HIGH 50%;
NET "CLOCK_50<0>" LOC = P56 | IOSTANDARD = LVTTL;
NET "KEY1" LOC = P38 | IOSTANDARD = LVTTL;
NET "LED<0>" LOC = P134 | IOSTANDARD = LVTTL;
NET "LED<1>" LOC = P133 | IOSTANDARD = LVTTL;
NET "LED<2>" LOC = P132 | IOSTANDARD = LVTTL;
NET "LED<3>" LOC = P131 | IOSTANDARD = LVTTL;
NET "LED<4>" LOC = P127 | IOSTANDARD = LVTTL;
NET "LED<5>" LOC = P126 | IOSTANDARD = LVTTL;
NET "LED<6>" LOC = P124 | IOSTANDARD = LVTTL;
NET "LED<7>" LOC = P123 | IOSTANDARD = LVTTL;

Notice that KEY0 is missing. The Mojo board has only one button, instead of the DE0-Nano’s two. In the original example, button 0 is used to reset the clock and button 1 is used to invert the state. I’ve decided to set the reset to constant low and set the Mojo’s one button to button 1:

altpll50_2 altpll50_inst
  (.inclk0 (CLOCK_50[0])
  ,.c0 (system1000)
  ,.areset (1'b0)
  ,.locked (altpll50_locked));

In addition, there are a few changes that need to be made because of the way Xilinx handles verilog. Hopefully these can be incorporated into CλaSH in the future:

• change the round brackets around CLOCK_50(0) to CLOCK_50[0], same for KEY1
• Xilinx does not like using NOT in wire assignments. If you want to use an inverted signal, simply create an inverted wire, i.e.:

wire not_reset = ~reset;
my_mod my_mod_inst(.reset(not_reset));

Compilation and Upload

Now your project is ready to run! Under processes, right click on Generate Programming File and select run. If you can’t find this option, make sure that you’ve selected the file blinker.v and that it is set as the top module (right-click on blinker.v and set as top module).

If you get a translation error about Symbol ‘MMCM_ADV’ is not supported in target ‘spartan 6’, it means the wrong chip was selected in the clock generation step.

Once it completes successfully, load the generated *.bin file onto the mojo using the Mojo Loader

End Result

Thanks for putting up with the camera shake, my tripod is in Florida

In my quest for a simple way to draw with haskell led me to struggle with curses, but I simultaneously discovered and struggled with gtk2hs, a package with haskell bindings to the gtk libraries. Gtk has integrated support for Cairo, a mature graphics library and has a lot of documentation and support.

However, the way things are drawn to the screen changed a lot between Gtk2 and Gtk3. Some functionality has been deprecated, so most gtk2hs demos do not work with Gtk3. Gtk2 is no longer supported in recent versions of Ubuntu, and I have no desire to compile unsupported libraries on my own, so I decided instead to try to make the gtk2hs demos work with Gtk3. The effort was going pretty well until I tried to update keyboard event bindings, for example, that pressing the Escape key closes the window:

window `on` keyPressEvent $ tryEvent $ do
	"Escape" <- eventKeyName
	liftIO mainQuit

This results in the error:

    Couldn't match type ‘[Char]’ with ‘Data.Text.Internal.Text’
    Expected type: System.Glib.UTFString.DefaultGlibString
      Actual type: [Char]
    In the pattern: "Escape"
    In a stmt of a 'do' block: "Escape" <- eventKeyName
    In the second argument of ‘($)’, namely
      ‘do { "Escape" <- eventKeyName;
            liftIO mainQuit }’

At some point between when this demo was written and modern versions of gtk2hs and haskell, the required type of eventKeyName changed from [Char] to Text. My first guess was to simply attempt to construct a new Text, using:

import Data.Text
...
    (Text "Escape") <- eventKeyName

But the error informs me that I don’t understand how the Text constructor works:

    Constructor ‘Text’ should have 3 arguments, but has been given 1
    In the pattern: Text "Escape"

The description of the Text constructor is:

Construct a Text without invisibly pinning its byte array in memory if its length has dwindled to zero.

No explanation of what the two first integers are, but seeing the mention of memory manipulation makes me freak out a little. Hopefully there’s some other method for converting [Char] to Text. It looks like pack is what I need:

    (pack "Escape") <- eventKeyName

This results in the error:

Parse error in pattern: pack
Possibly caused by a missing 'do'?

I don’t understand why I’m getting this error, but I’m guessing it’s because I can’t call a function in a case situation like that. So what if I take it out of the event monad?

  escText <- pack "Escape"
  window `on` keyPressEvent $ tryEvent $ do
    escText <- eventKeyName
    liftIO mainQuit

This results in the error:

Couldn't match expected type ‘IO t0’ with actual type ‘Text’
In a stmt of a 'do' block: escText <- pack "Escape"

Since I’m getting deeper and deeper down a rabbit hole of string conversions, maybe there’s a way for haskell to interpret string literals in the way that eventKeyName needs. A similar question on stack overflow suggests that I need to add:

{-# LANGUAGE OverloadedStrings #-}

to the top of the script. Doing that clears the error around eventKeyName, but it creates errors on every other string literal:

    No instance for (Data.String.IsString b0)
      arising from the literal ‘"Gtk2Hs Cairo Clock"’
    The type variable ‘b0’ is ambiguous
    Note: there are several potential instances:
      instance Data.String.IsString
                 Data.ByteString.Builder.Internal.Builder
        -- Defined in ‘Data.ByteString.Builder’
      instance Data.String.IsString Text -- Defined in ‘Data.Text’
      instance Data.String.IsString [Char] -- Defined in ‘Data.String’
    In the second argument of ‘(:=)’, namely ‘"Gtk2Hs Cairo Clock"’
    In the expression: windowTitle := "Gtk2Hs Cairo Clock"
    In the second argument of ‘set’, namely
      ‘[containerChild := canvas, windowDecorated := False,
        windowResizable := True, windowTitle := "Gtk2Hs Cairo Clock"]’

But now I know that I can convert strings to Text with pack, so that should make haskell less ambiguous:

set window [ containerChild := canvas, windowDecorated := False,
               windowResizable := True, windowTitle := (pack "Gtk2Hs Cairo Clock") ]

And it works!

But I can’t help feel like I’ve cheated somehow. There’s probably a better way to define strings.

A nameless Ghoul predicts the release of a new Ghost album in the fall of 2017, which feels like ages away. Here are some heavy-metal albums I bought recently to relieve my cravings:

Purson - Desire’s Magic Theatre

Purson is a progish, psychadelic heavy-metal band. Their fantastic 2013 debut album The Circle and the Blue Door was critically acclaimed by many prog and metal blogs. They opened for Ghost in their 2015 show in Toronto, and sound like Black Sabbath crossed with Coven.

Desire’s Magic Theater (guess which drug this album is about) came out in 2016 and is much tamer in sound than The Circle and the Blue Door. The Way It Is sounds like a reject from the Beatles’ Help. Electric Landlady starts out fantastic, but has a corny major key change in the chorus.

Not quite Ghostly enough because the song and melodies are too repetitive and familiar. This album is a disappointment.

Arcana 13 - Danza Macabra

Arcana 13 is a talented Italian heavy metal band that combines the soundtracks of old horror movies with a early 00s alt-rock singing style. Their debut album, Danza Macabra, came out in March 2016.

Like Ghost, they employ a lot of dramatic unison playing in between heavy guitar solos, with deliciously occult lyrics. Unlike Ghost, they have a tendency to stick to only one style. Despite having killer hook melodies, songs like Dread Ritual tend to go on a little too long. The vocals are constantly distorted, and it would be nice if there was a change in tonality or style.

Not quite Ghostly enough because the singing sticks to only one, heavily distorted style, but great in their own right.

Khemmis - Hunted and Absolution

Khemmis is a doom/heavy metal rock band. Hunted, an album they released in October of 2016, shortly after 2015’s Absolution, has been called one of the best of 2016 by many metalheads.

Most of their singing is operatic but distorted, like Arcana 13, with frequent growling sections, which is a nice balance. They also strike a good mix between atmospheric and melodic sections with heavier meat, like on the song The Bereaved. The production quality and technical ability on Hunted has improved over Absolution, and I look forward to further improvement. My only complaint with either of these albums certain songs, like Antedeluvian, and Candlelight are slightly too doom-y for my taste.

Not quite Ghostly enough because of the slower doom-y playing, but great in their own right.

Spellcaster - Night Rides the World

Spellcaster is a modern power/heavy band. Their latest album, Night Hides the World, was released in July of 2016, is full of Ghost-like theatrical melodies and complex songs that inspire a large range of emotions without edging into the campiness of most power metal. Their guitar and drumming style is NWOBHM-ish, but the singing style is like a late 00’s pop-punk band.

Not quite Ghostly enough because of the constant near-power-metal speed, but great in their own right.

I’m currently learning Haskell by working on stupid projects. In this post, I’m going to go over my process for trying to generate a rainbow terminal. I’m going to use the haskell-ncurses bindings to draw a rainbow of colored squares to the terimal.

Haskell has two types with monad instances: Curses and Update. Update is used whenever the window needs to be redrawn, like on resize. Curses is A small wrapper around IO, to ensure the ncurses library is initialized while running. but I don’t understand what is meant by this. To create a green square on the screen this is the minimal code:

import UI.NCurses
main = runCurses $ do
     w <- defaultWindow
     cid <- newColorID ColorGreen ColorBlack 1
     updateWindow w $ do setColor cid
                         drawString "■"
     render
     waitFor w (\ev -> ev == EventCharacter 'q' || ev == EventCharacter 'Q')

where waitFor is taken from the basic ncurses example:

waitFor :: Window -> (Event -> Bool) -> Curses ()
waitFor w p = loop where
    loop = do
        ev <- getEvent w Nothing
        case ev of
            Nothing -> loop
            Just ev' -> if p ev' then return () else loop

Next step is to create a field of squares. I’m going to use the forM_ Control Monad. There’s probably a better way of doing this, but as a Haskell newbie, this method feels the most comfortable.

import Control.Monad (forM_)
num_accross = 7
num_down = 7

...
     updateWindow w $ forM_ [(x,y) | x<-[0..num_accross], y<-[0..num_down]] $ \(x,y) ->
                            do  setColor cid
                                moveCursor y x
                                drawString "■"
...

haskell-ncurses has nine pre-defined colors. I want to assign each row to a different foreground color and each column to a different background color. Let’s try:

...

colors = [ColorMagenta, ColorRed, ColorYellow, ColorGreen, ColorBlue, ColorCyan, ColorWhite, ColorBlack]
...
     updateWindow w $ forM_ [(x,y) | x<-[0..num_accross], y<-[0..num_down]] $ \(x,y) ->
                            do  
                                cid <- newColorID (colors !! x) (colors !! y) 0
                                setColor cid
                                moveCursor y x
                                drawString "■"
...

But this leads to the error:

    Couldn't match type ‘Curses’ with ‘Update’
    Expected type: Update ()
      Actual type: Curses ()

This error occurs because newColorID returns an instance of Curses, but I’ve put it in a Monad bound to the an instance of the Update type. I think I can solve this by switching the order of operations:

     forM_ [(x,y) | x<-[0..num_accross], y<-[0..num_down]] $ \(x,y) ->
                            do
                                cid <- newColorID (colors !! x) (colors !! y) 1
                                updateWindow w $ do
                                                   setColor cid
                                                   moveCursor y x
                                                   drawString "■"

That clears up the type error. Let’s move on to the other errors:

    Couldn't match expected type ‘Integer’ with actual type ‘Int’
    In the first argument of ‘moveCursor’, namely ‘y’
    In a stmt of a 'do' block: moveCursor y x

This error is interesting because it goes away if I remove the newColorID line. The newColorID line must be casting the x and y values to Integers, but moveCursor is unable to cast them back to Int. I can force it to cast to Int using the fromIntegral function:

moveCursor (fromIntegral y) (fromIntegral x)

This code works without error, but results in a screen full of white squares:

This is because I’m naming all the colors in my pallette as 1. Instead, let’s name them based on their position in the grid:

cid <- newColorID (colors !! x) (colors !! y) ((x*(num_accross+1))+y+1)

This leads to the error:

Couldn't match expected type ‘Integer’ with actual type ‘Int’
    In the first argument of ‘(*)’, namely ‘x’
    In the first argument of ‘(+)’, namely ‘(x * num_accross)’

OMG Haskell just pick a type and stick with it! I can cast back to Int using toInteger:

cid <- newColorID (colors !! x) (colors !! y) (toInteger ((x*(num_accross+1))+y+1))

This works!

Okay, but now I want to paint with all the colors of the wind. We can define custom colors using Color.

First, let’s expand the window to more rows and columns. To avoid the error:

ncurses_minimal.hs:15:1: Warning: Tab character
ncurses_minimal.hs: CursesException "moveCursor: rc == ERR"

We need to change the allowable size of the window:

updateWindow w $ resizeWindow num_accross num_down

haskell-ncurses has a defineColor function that allows for custom colors, but like newColorID, it can only be applied to colors in the colors in the terminal’s list of colors. The number of colors can be checked using the code:

maxid <- maxColorID
updateWindow w $ drawString (show maxid)

On GNOME-terminal, this is 255, which is far less than most displays can output, but more than enough to make some pretty pictures. Like ColorPairs, colors need to be assigned to specific addresses in the table. Addresses should start at 1 so that 0 can be reserved for black. I’m not sure why Colors can start at 0 but ColorPairs need to start at 1, but this means that we can use the same equation for the indices of both.

...
num_accross = 24
num_down = 9
...
let colIndex = x*(num_down+1)+y+1
let my_color = Color (fromIntegral colIndex :: Int16)
defineColor my_color (x*36) 0 (y*110)
cid <- newColorID my_color ColorBlack (toInteger colIndex)

Which leads to another pretty picture:

I don’t quite understand why I need to use let in certain cases and single-assignment operators in others. Maybe it’s because the single-assignment is used in a nested monad?

The code for this exercise is available as a gist.

Unwrap My Heart is a Twilight parody written by Portland Standup Comic Alex Falcone and comedy writer Ezra Fox. The authors also created the Read it and Weep Podcast, which started by reviewing Twilight, then tackled other young adult fiction, and have more recently moved on to mostly reviewing bad movies. I first listened their review of Gossip Girl¹ with guest reviewer Amanda Leinbaugh of Skepchick and have been a fan since.

Through their podcast, the writers have developed an intimate knowledge of understanding of the genre. But Unwrap My Heart can’t be lumped in with other parodies like Barry Trotter and Bored of the Rings because in addition to hitting all of the obvious flaws in Twilight, the meat between the parody is also funny. A lot of the jokes were said first on Read it and Weep, but they’re even funnier in a fictional context. The ancillary characters are adorable. I would read a sequel that was just Sofia’s weatherman father and her failed 90s comic teacher Mr. Graff taking over her effort to get her cat internet famous while she’s off fighting mummies.

The book isn’t perfect. It’s a little over-written in a few places, but it has far fewer stylistic flaws than other famous first books like The Martian or Ready Player One. Sofia is smart yet can’t figure out that her crush is a mummy or that her friend is upset because he’s part of a love angle. It’s never fully explained why the mummy has a porche (at least it’s not a volvo?) or has to attend high school. And I’m having a really hard time imagining a handsome face without eyeballs.

It feels weird to nitpick a parody. This book is the funniest thing I’ve ever read, perhaps because it also has a lot of heart. Sofia is one of the most accurate written depictions of what it’s like to be an awkward teenage girl. Who would have thought that it would come from two 30-something men writing a Twilight parody?

1: Gossip Girl, along with Ashlee Simpson and sexual inuendo vintage graphic tees from American Eagle formed the triumvirate of things other girls at my high school loved but that I hated. Why read descriptions about what shitty rich people are wearing? Why drive 2.5 hours to buy one of 5 shirts at the MicMac Mall when at least one other person will also be wearing when you go to school on Monday?

I thought that Erik Meijer’s edX class on functional programming was super easy until the lecture on functional parsers and monads, and now I’m so confused and anxious about my inability to become unconfused that I think I’ll have to quit the course for now.

The example monads on wikipedia make sense to me, but I can’t figure out how to call them in hugs.

For example, the course provided some code with the following parsers defined:

> many                          :: Parser a -> Parser [a]
> many p                        =  many1 p +++ return []
> 
> many1                         :: Parser a -> Parser [a]
> many1 p                       =  do v  <- p
>                                     vs <- many p
>                                     return (v:vs)