19 January 2018

If you’re building a Big Data application on AppEngine, you’ll probably need to grab more than a thousand entries out of a search index. It turns out this is not as straightforward as expected. I’ve built a sample project to demo how to do this.

In this demo we initialize our index to have 3000 entries of the numbers between 0 and 3000:

search_index = search.Index(name=INDEX_NAME, namespace=NAMESPACE)
for i in range(3000):
fields = [search.NumberField(name='number_value', value=i),
search.NumberField(name='last_modified', value=int(time()))]
doc = search.Document(fields=fields)
results = search_index.put(doc)


A first attempt at building a handler to query this index would be:

class Search(webapp2.RedirectHandler):
def get(self):
try:
limit = int(self.request.get('limit'))
except:
limit = DOCS_PER_PAGE_LIMIT
try:
offset = int(self.request.get('offset'))
except:
offset = 0

expr_list = [
search.SortExpression(
expression='number_value',
default_value=int(0),
direction=search.SortExpression.ASCENDING
)
]
sort_opts = search.SortOptions(expressions=expr_list)
index = search.Index(name=INDEX_NAME, namespace=NAMESPACE)
search_query = search.Query(
query_string='',
options=search.QueryOptions(
limit=limit, sort_options=sort_opts, offset=offset, ids_only=False)
)


This works well for values under 1000, but if you try to run the query /search?limit=3000, you’ll get the error:

Traceback (most recent call last):
rv = self.handle_exception(request, response, e)
rv = self.router.dispatch(request, response)
return handler.dispatch()
return self.handle_exception(e, self.app.debug)
return method(*args, **kwargs)
File "/home/catherine/python_projects/appengine-large-offset-example/main.py", line 55, in get
limit=limit, sort_options=sort_opts, offset=offset, ids_only=False)
self._limit = _CheckLimit(limit)
upper_bound=MAXIMUM_DOCUMENTS_RETURNED_PER_SEARCH)
raise ValueError('%s, %d must be <= %d' % (name, value, upper_bound))
ValueError: limit, 3000 must be <= 1000


We can’t get all 3000 entries in one query, so we’ll have to spread the acquisition over three queries:

• /search?limit=1000
• /search?limit=1000&offset=1000
• /search?limit=1000&offset=2000

The first two queries work, but if you set offset to 2000, you’ll get the error:

Traceback (most recent call last):
rv = self.handle_exception(request, response, e)
rv = self.router.dispatch(request, response)
return handler.dispatch()
return self.handle_exception(e, self.app.debug)
return method(*args, **kwargs)
File "/home/catherine/python_projects/appengine-large-offset-example/main.py", line 55, in get
limit=limit, sort_options=sort_opts, offset=offset, ids_only=False)
self._offset = _CheckOffset(offset)
upper_bound=MAXIMUM_SEARCH_OFFSET)
raise ValueError('%s, %d must be <= %d' % (name, value, upper_bound))
ValueError: offset, 2000 must be <= 1000


Darn. Because of these limitations, we’re going to need a concept from Databases usually reserved for operations that span multiple computers: a cursor.

You can set as a parameter to the query options:

search_query = search.Query(
query_string='',
options=search.QueryOptions(
limit=self.limit,
sort_options=self.sort_opts,
cursor=search.Cursor(per_result=False),
ids_only=False,
)
)


you can access the cursor for this search query after getting the results:

search_results = self.index.search(search_query)
cursor = search_results.cursor


The strategy for pulling out the entries is this:

1. create a search with no offset and a new cursor
2. initialize current_offset to 0
3. while current_offset is less than the desired offset, query the search index again with the cursor of the last query, and add the limit to the current_offset
4. output the last search results

It would be nice if we could start our loop at the maximum offset in order to limit the number of repetitions of step 3, but if you do, you’ll get the error:

ValueError: cannot set cursor and offset together


Also, make sure that you set the per_result value of search.Cursor to False, or else the cursor will be None:

  File "/home/catherine/python_projects/appengine-large-offset-example/main.py", line 109, in query_with_cursors
cursor_cache[current_offset] = search_results.cursor.web_safe_string
AttributeError: 'NoneType' object has no attribute 'web_safe_string'


Here’s the implementation:

current_offset = self.limit
search_query = search.Query(query_string='',
options=search.QueryOptions(
limit=self.limit,
sort_options=self.sort_opts,
cursor=search.Cursor(per_result=False),
ids_only=False,
)
)

search_results = self.index.search(search_query)
if self.offset >= search_results.number_found:
return self.render_search_doc([])
while current_offset < self.offset:
current_offset += self.limit
search_query = search.Query(query_string='',
options=search.QueryOptions(
limit=self.limit,
sort_options=self.sort_opts,
cursor=search_results.cursor
)
)
search_results = self.index.search(search_query)
self.render_search_doc(search_results)


Now, if we’re going to be querying an offset of up to 100000, it will take a while to do 1000 queries. So, it would be better if we could cache these cursors. We can stick the web_safe_string in memcache:

cursor_cache = {}
current_offset = self.limit
search_query = search.Query(query_string='',
options=search.QueryOptions(
limit=self.limit,
sort_options=self.sort_opts,
cursor=search.Cursor(per_result=False),
ids_only=False,
)
)

search_results = self.index.search(search_query)
cursor_cache[current_offset] = search_results.cursor.web_safe_string
if self.offset >= search_results.number_found:
return self.render_search_doc([])
while current_offset < self.offset:
current_offset += self.limit
search_query = search.Query(query_string='',
options=search.QueryOptions(
limit=self.limit,
sort_options=self.sort_opts,
cursor=search_results.cursor
)
)
search_results = self.index.search(search_query)
cursor_cache[current_offset] = search_results.cursor.web_safe_string
memcache.set(cursor_cache_key, json.dumps(cursor_cache))
self.render_search_doc(search_results)


And then the next time, we can pull out the cursor for that offset:

cursor_cache_key = 'cursors'
cursor_cache = memcache.get(cursor_cache_key)
if cursor_cache:
offset_key = str(self.offset)
if offset_key in cursor_cache:
cursor_cache = cursor_cache[offset_key]
else:
logging.info("%s not in %s" %(offset_key, cursor_cache))
cursor_cache = None

if cursor_cache:
logging.info("found cursor cache string %s " % cursor_cache)

# construct the sort options
search_query = search.Query(
query_string='',
options=search.QueryOptions(
limit=self.limit,
sort_options=self.sort_opts,
cursor=search.Cursor(per_result=False, web_safe_string=cursor_cache),
ids_only=False,
)
)
return self.render_search_doc(self.index.search(search_query))


A next step would be to find the nearest cursor offset, and start from there. But that would be another blog post.

14 December 2017

I have a dilemma: I don’t want to buy stuff for holidays, but I also don’t want to be the only house on my street without Christmas lights. So I made myself some Christmas lights from:

• the leftover 5050 RGB LEDs from my last PC build
• the 12 V DC power supply from my old wifi router
• the leftover canvas from making a bike bag
• a leftover dowel rod from making a bookcase

I made a model of an individual LED strip in Inkscape and experimented with a few designs. The one I ultimately chose was these snowflakes:

I chose this design because it had relatively few angles, which meant I could free-hand layout the LEDs without needing to project the design onto the fabric:

I soldered the 12V and Blue Grounds on each snowflake, and then hot-glued some of the longer wires down to the fabric:

This is the end result:

26 November 2017

I recently purchased a Computerized Sewing Machine. Designs can be uploaded to internal storage via USB, which ubuntu automatically recognizes and mounts:

This compatibility with linux out of the box is fantastic; unfortunately designs must be in Brother’s undocumented binary file format, and the sewing machine gives no feedback regarding parsing errors. This means that I’ve been creating binaries for embroidery patterns with a lot of trial and error. In order to script some of the process, I’d like to upload and test new designs without human intervention.

To find where ubuntu has mounted the sewing machine, I could either:

1. write some udev rules
2. get python to figure it out from linux commands

In this case, method 2 is a bit nicer since I’m already using Python to generate the binary files.

First, I use the output of df to generate a mapping of where ubuntu has mounted the external drives:

mount_points = {}
out = check_output(["df"]).split("\n")
for line in out:
if line.find("/media") >= 0:
mount_location = line.split("%")[-1]
line = line.split(" ")
mount_points[line[0]] = mount_location.strip()


Next, get the vendor ids for each device, and if it matches, call that my mount location:

mount_destination = None
for mount_point in mount_points:
out = check_output(["udevadm", "info", "--name="+mount_point, "--attribute-walk"])
if out.find('ATTRS{idVendor}=="04f9"')> 0:
print("found brother device, assuming embroidery machine")
mount_destination = mount_points[mount_point]
break


So I can now use shutil.copyfile() to move the files over to mount_destination. This is kind of a hacky solution because it may also catch Brother printers, or fail to recognize embroidery machines from other manufacturers, but since I have neither of these situations, it’s good enough for now.

28 October 2017

My husband and I disagree about whether it’s more energy efficient to use the toaster oven or regular oven for baking. He believes the toaster oven consumes less energy, since the internal volume is much smaller. I believe the oven consumes less because it is better insulated.

I did some back-of-the-envelope calculations to see if there was an obvious conclusion. I estimate that the total power consumed is the work needed to heat up the volume of air inside the oven to the desired temperature, plus the work lost to heat transfer out of the door. It’s also possible that the heat is lost through the other sides, but I didn’t take that into account. From the ideal gas law, the work to heat up the air inside the oven is:

$W = D_mVR\triangle T$

Where $$D_m$$ is the molar density of air, $$V$$ is the volume, $$R$$ is the ideal gas constant, and $$\triangle T$$ is the temperature change.

The heat lost through the oven door is:

$Q = kA\triangle Tt$

Where $$k$$ is the heat transfer coefficient, $$A$$ is the area of the door, and $$t$$ is the time that the oven is on.

Since both ovens have the same depth, the ratio of the ovens’ volume, 2.8, is the same for the door area. Some mechanical engineers estimate that the $$k$$ of double-paned oven doors is ~3, whereas the toaster oven’s single pane of glass is 5.8. The ratio of these coefficients almost exactly cancels out the difference in volume. Because the heat is constantly being leaked out of the ovens, the work due to lost heat is nearly an order of magnitude larger than work required to heat up the oven, so this contribution can be mostly ignored. Since we don’t exactly know $$k$$ for both ovens, we’re going to need to get experimental.

I bought an Aeotek Smart Energy Meter (currently on Amazon for 32 USD) and an Aeotek Z-stick (currently on Amazon for 45 USD)1. The Z-Stick is compatible with open-zwave which has a python interface.

My code to grab the power data on linux is:

import time
from openzwave.option import ZWaveOption
from libopenzwave import PyManager
import csv
from matplotlib import pyplot

device="/dev/ttyUSB0"
options = ZWaveOption(device, config_path="/etc/openzwave/", user_path=".", cmd_line="")
options.set_console_output(False)
options.lock()

class PowerMeter(PyManager):
def __init__(self):
super(PowerMeter, self).__init__()
self.create()
time.sleep(1.0)
self.meter_values = []
self.start_time = None
self.units = None
self.value_id = None

def get_meter_values(self, num_seconds):
self.start_time = time.time()
while time.time() - self.start_time < num_seconds:
time.sleep(0.10)
if self.value_id:
assert self.refreshValue(self.value_id)

def callback(self, zwargs):
# get measurement value from the first message tagged with the sensor we want
if "valueId" in zwargs:
if zwargs["valueId"]["commandClass"] == "COMMAND_CLASS_METER":
if zwargs["valueId"]["label"] == "Power" and \
zwargs["valueId"]["instance"] == 1 \
and self.start_time is not None:
self.value_id = zwargs["valueId"]["id"]
self.units = zwargs["valueId"]["units"]
self.meter_values.append((time.time() - self.start_time,
zwargs["valueId"]["value"]))
print("Power", zwargs["valueId"]["value"])

manager = PowerMeter()
# collect data for an hour
manager.get_meter_values(60*60)
with open('meter_values_%s.csv' % manager.start_time, 'wb') as csvfile:
spamwriter = csv.writer(csvfile, delimiter=',', quoting=csv.QUOTE_MINIMAL)
spamwriter.writerow(['time (s)', 'meter (%s)' % manager.units])
for row in manager.meter_values:
spamwriter.writerow(row)

fig, ax = pyplot.subplots()
ax.plot([point[0]/60.0 for point in manager.meter_values],
[point[1] for point in manager.meter_values])
ax.set_xlabel("Time (minutes)")
ax.set_ylabel("Meter Value (%s)" % manager.units)
fig.savefig("power_meter_%s.png" % manager.start_time)


I believe that instance 1 is the power on the first probe, which is attached to the hot wire of my house’s main power. Since probe 2 is attached to the neutral wire, I believe it’s possible to cancel out most of the noise by subtracting the power on instance 2 from the power on instance 1. I have no idea how these map to European/Australian three phase systems.

With the script running, I collected three sets of data: one while the toaster cooked a sheet of cookies, one while the oven cooked a sheet of cookies, and one with neither oven running, just the house lights, routers, and refridgerator. The data, from the start of pre-heating to when the cookies came out, looks like this:

The toaster took longer to pre-heat, so it had a longer data collection time. It looks like both ovens and the refridgerator (in the base usage) are driven by square forcing functions that turn off when a desired temperature is reached. The oven’s forcing function is a bit more sophisticated than the toaster; it looks like it can heat at two different wattage levels. I think the long period of the oven been off while cooking is proof that the oven does have a lower heat transfer coefficient than the toaster oven.

I can integrate the data I collected using scipy’s integrate and numpy’s interpolate functions:

from scipy import integrate
from numpy import interp

result = integrate.quad(lambda x: interp(x, data_array[0], data_array[1]),
min(data_array[0]), max(data_array[0]))[0]


I also subtract off the base usage in each case. After integrating, I estimate that the toaster uses 0.45 kWh to bake a single sheet of cookies, and the oven uses 0.70 kWh.

The toaster is more efficient at baking a single sheet of cookies, so I’d conclude that the heat transfer coefficient of the toaster is not as bad as expected. However, the oven can handle two sheets at once, and is faster. If you want to shave 8 minutes off of your cook time, it’s only an extra 0.15 kWh, which in Florida costs two cents. However, the oven has an advantage over the toaster in that it can cook more than a single sheet at a time, and then the energy cost per cookie favors the oven.

So depending on the situation, we’re both right.

1: It is my personal opinion that Z-Wave compatible devices are 2-3 times more expensive than they should be, given the underlying hardware. I think this is because Z-Wave is not open source, so development takes longer and costs more. I would not invest in a Z-Wave system now. It would be nice to be able to turn off my overhead light from a smartphone app so that I don’t have to get out of bed, but just putting a long piece of twine on the pull chain seems to be an adequate solution.

24 October 2017

Using matplotlib’s axes.transData and get_sample_data, it’s possible to replace data markers with images, such as the poo emoji:

This is fun and silly, but it’s also important for accessibility for people with colorblindness or with shitty printers, like me.

axes.transData can transform x/y coordinates into the plot’s pixel location on the plot output. In order to then convert the pixel location to the figure’s 0.0-1.0 scale, divide by these locations by the size of the figure’s bounding box.

from os.path import join, dirname, abspath
from matplotlib import pyplot
from matplotlib.cbook import get_sample_data
from numpy import linspace
from numpy.core.umath import pi
from numpy.ma import sin

# poo-mark came from emojipedia:
# https://emojipedia-us.s3.amazonaws.com/thumbs/120/apple/96/pile-of-poo_1f4a9.png

x = linspace(0, 2*pi, num=10)
y = sin(x)

fig, ax = pyplot.subplots()
plot = ax.plot(x, y, linestyle="-")

ax_width = ax.get_window_extent().width
fig_width = fig.get_window_extent().width
fig_height = fig.get_window_extent().height
poo_size = ax_width/(fig_width*len(x))
poo_axs = [None for i in range(len(x))]
for i in range(len(x)):
loc = ax.transData.transform((x[i], y[i]))
poo_size, poo_size], anchor='C')
poo_axs[i].imshow(poo_img)
poo_axs[i].axis("off")
fig.savefig("poo_plot.png")

20 October 2017

I recently came back to the Robot Operating System after not working with it for nearly 2 years. The great thing about ROS is that everything is a debian package and most of it is written in python, a language that I feel very comfortable with. The bad thing about ROS is that everything is a debian package and most of it is written in a language that I feel comfortable in. In my experience, 90\% of ROS problems are due to not having the right debian packages, including the first problem I encountered out of the gate.

I wanted to get my ros_realtime_plotter that I made for PyCon Canada 2015 working again. I followed my previous instructions starting with a fresh install of Ubuntu 16.04, but installed Kinetic Kame (instead of Indigo Igloo) and Gazebo 8.0.0 (instead of Gazebo 5.something). The first problem I encountered was some incompatibility between Kinetic Kame and Gazebo 8, so in future I would start off with gazebo 7.

My robot wouldn’t move. The only warning or error message was:

[WARN] [1508501276.314154, 1090.261000]: Controller Spawner couldn't find the expected controller_manager ROS interface.


This error message was not particularly useful to me. I didn’t understand what was meant by ROS interface. To learn more about this error message, I put ros_control in my workspace and modified the spawner script to print out the actual exception:

        rospy.logwarn("Controller Spawner couldn't find the expected controller_manager ROS interface. %s" % e)


The error message was a timeout waiting for the service /raz_bot/controller_manager/load_controller. Looking at rosservice, I could confirm that the service did not exist. I confirmed that my URDF had the code to load the gazebo ros controller:

  <!-- Gazebo plugin for ROS Control -->
<gazebo>
<plugin name="gazebo_ros_control" filename="libgazebo_ros_control.so">
<robotNamespace>/</robotNamespace>
</plugin>
</gazebo>


It seemed like the robot spawner was just dying without a warning. So I set gazebo to have verbose output:

  <!-- Start the Gazebo simulator, loading a world -->
<include file="$(find gazebo_ros)/launch/empty_world.launch"> <arg name="world_name" value="$(find ros_realtime_plotter)/worlds/camera_above.world"/> <!-- world_name is wrt GAZEBO_RESOURCE_PATH environment variable -->
<arg name="verbose" value="true" />
</include>


This gave the error:

terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >'
what():  boost: mutex lock failed in pthread_mutex_lock: Invalid argument


Not particularly helpful. But, in a semi-related github issue Michael Koval demonstrated how you could isolate problems with the URDF spawner shared object library by loading it in python.

I did that, and had no issues. The next step was to check the gazebo ros control shared object library. I looking for it showed me the source of my problem:

$find / 2>/dev/null | grep "gazebo_ros_control" /home/cholloway/ros/ros_control_ws/src/ros_control/ros_control/documentation/gazebo_ros_control.png /home/cholloway/ros/ros_control_ws/src/ros_control/ros_control/documentation/gazebo_ros_control.pdf /home/cholloway/ros/ros_control_ws/src/ros_control/ros_control/documentation/gazebo_ros_control.odg  The only files related to gazebo_ros_control were guides as to how to use ros_control with gazebo in the ros_control repository! I had assumed that the gazebo package came with these, and I was wrong. The packages I needed were: sudo apt-get install ros-kinetic-gazebo-ros-control ros-kinetic-diff-drive-controller  after that, the ros_realtime_plotter worked. 09 August 2017 Six months ago I moved to Florida and started working at GradientOne, a startup building a web-based platform for interacting with scientific and testing instruments and performing data analysis. It’s directly in the center of my interests and has kept me pretty occupied. Between my new job and working on the electrical wiring of my new house, I haven’t had much time for side-projects or reviewing the music I’ve been listening to. However, I have been blogging for dollars about the things I’ve built for GradientOne, and I’m going to use this blog post as an opportunity to show off what I’ve built. If any of this functionality interests you, you should contact GradientOne to get API access or talk about your specific required functionality. # CANOpen Support I added support for CANOpen devices, which was quite different from the SCPI-compliant devices GradientOne had previously supported. Unlike SCPI, I had never really worked on CAN protocols before, which was much more like TCP/IP than a serial communication protocol. I created a web-based CANOpen frame interpreter, so others learning CANOpen might find it easier in the future. # Trace Pattern Matching There have been many occasions where in the process of debugging a piece of equiment, I have observed a plot and had it trigger a sense of deja-vu. Shapes can stick in memory a lot easier than a list of numbers. The trace pattern matching tool I created allows sections of a set of x-y data to be tagged and then searched against all previous trace data. This means that if there is a consistent blip in an experiment that happens one in a thousand times, the blip can be identified, even if the blip appears at different times and locations, or even multiple times. Once patterns have been defined, they can be used to define other measurements, so you could do something like calculate the time between the start of a clock signal and the appearance of a specific byte. # Pass/Fail Criteria definition Most modern factories operate on PLCs, where each individual process boils down to true/false boolean values when determining future branching processes. To that end, I created an interface for defining tests to add pass/fail criteria to collected data. The Pass/Fail criteria can come at the end of a series of multiple different measurements. Calculated pass/fail measurements are added to the search index so that existing measurements can be quickly sorted based on passes and failures. # GradientOne Command Language Knowing exactly how to communicate with a new device once you get it is an annoying pain. Although devices that are SCPI compliant are a bit easier to use than those that use some new communications protocol developed by an engineer throwing darts at a keyboard, as a lot of devices developed outside of the US seem to do, many device manufacturers fail to make their SCPI protocols SCPI-compliant. The GradientOne Command Language uses python-like syntax and tries to provide a consistent grammar for writing and querying instruments. Once a device’s functionality has been added to GradientOne, incorporating it into the Command Language means that the web editor can hint the user about all available functionality. Those are the four big projects I’ve shipped in the past six months. I’m currently working on using machine learning for decoding digital signals, and have planned support for more devices. 24 February 2017 Consider a python project with the directory structure: ├── inone │ └── myfuncs.py └── test.py  where myfuncs.py is: def hello_world(): print("hello world")  and test.py is: from inone.myfuncs import hello_world hello_world()  Running test.py with python 3.+ works as expected, but running in python 2 results in the error: $ python2 test.py
Traceback (most recent call last):
File "test.py", line 1, in <module>
from inone.myfuncs import hello_world
ImportError: No module named inone.myfuncs


The solution is to add the file __init__.py to the inone directory with the code:

import myfuncs


However, this causes the error when running with python 3.+:

$python3 test.py Traceback (most recent call last): File "test.py", line 1, in <module> from inone.myfuncs import hello_world File "/home/catherine/python_projects/import_test/inone/__init__.py", line 1, in <module> import myfuncs ImportError: No module named 'myfuncs'  The proposed solutions to this on the Stack Overflow question are unsatisfactory; they are either to add this compatibility to the setup script, which is useless for running code in the directory, or using importlib or appending to the PYTHONPATH, which seems unnecessarily complicated. My own solution is to simply check for python3 in __init__.py: import sys if sys.version_info[0] < 3: import myfuncs  Of course, this is also a bit hacky and annoying, because these two lines need to be added to each directory’s __init__.py. Are there any better solutions out there? 14 February 2017 Modules that are patched with mock’s patch decorator must be re-imported. This is a simple thing but I wasted a couple of hours this morning on it, so I’m documenting this for future reference. Suppose you have two files: parent.py class MyParent(object): parent_name = "me"  child.py: from parent import MyParent class MyChild(MyParent): pass  And you would like to run some tests, one where the MyParent object is patched, and one where it is not. If you use this code for your tests: from mock import patch class MockParent(object): parent_name = "someone else" def test_child(): import child assert child.MyChild().parent_name == "me" @patch("parent.MyParent", new=MockParent) def test_mock(): import child assert child.MyChild().parent_name == "someone else"  If you run these tests, only one test will work, depending on the order. I had assumed that imports were local to the scope of the function, but it appears that they aren’t. They aren’t even local to the file - if you split these tests up, only one will work, and it will be the one in the alphabetically first named file. To run the tests as expected, you will need to re-import the child module. First, in order for this to work with both python2 and python3, import either imp or importlib if sys.version_info >= (3, 0): import importlib else: import imp as importlib  Then, after patching your code, re-import the patched library: @patch("parent.MyParent", new=MockParent) def test_mock(): import child importlib.reload(child) assert child.MyChild().parent_name == "someone else"  31 January 2017 I wasted a lot of time correctly defining angular modules. Here’s the problem and the solution. Consider the following angular app: <html ng-app="myApp"> <head> <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.5.7/angular.min.js"></script> <script type="text/javascript"> var myApp = angular.module('myApp', []); myApp.controller('MyController', ['$scope', function($scope) {$scope.result = 'test';
}]);
var myApp2 = angular.module('myApp');
myApp2.controller('MyController2', ['$scope','$window', function($scope,$window) {
$scope.result2 = 'test2'; }]) </script> </head> <body> <div ng-controller="MyController"> {{result}} </div> <div ng-controller="MyController2"> {{result2}} </div> </body> </html>  While creating both myApp and myApp2 is redundant, it is used in this example because eventually MyController and MyController2 will be separated into two separate files. If you change the line: var myApp = angular.module('myApp', []);  to: var myApp = angular.module('myApp');  or change the line: var myApp2 = angular.module('myApp');  to: var myApp2 = angular.module('myApp', []);  The example code will no longer work. If you look at the console, you may see the nomod error. In the description, it says that adding the empty array of dependencies to angular.module defines a new module of name myApp, and calling angular.module without that argument returns a reference to an existing module of name myApp. In the first change, removing the empty array results in an error due to angular looking up a module that does not exist. In the second change, adding the empty array results in an error due to angular attempting to define a module that already exists. This implementation is confusing to me - I think that angular should instead accept angular.module(‘appname’) without a second argument when there are no dependencies, and have a separate function for returning references to the module. If angular.module was called more than once, it could then return an error message that the module is already defined. However, I’m a newcomer to Javascript and angular so there’s probably a good reason it isn’t implemented this way that I’m unaware of. This implementation means the developer must take care to know which dependencies are needed accross all controllers they wish to use, and also to make sure that the module is created first, and any subsequent controllers get attached to a reference, rather than re-defining the module. 05 January 2017 We’re not quite yet a week into 2017, so I’d better put down my thoughts on some albums that came out in 2016, but that didn’t quite fit any other posts, before they go stale. # Clarence Clarity - SAME and Vapid Feels are Vapid Clarence Clarity is a British Progressive R&B artist. His 2015 debut album No Now is one of my favourites (he is currently ranked 47 in my music battle royale off of only one album). Clarence Clarity combines soulful singing with absurd lyrics, experimental sampling and production while still having a danceable beat. It’s the type of pop music I keep coming back to. This year he released two songs - SAME (which was packaged as an album on soundcloud where every song is the same) and Vapid Feels are Vapid. SAME expands the repetoir of samples that can be incorporated into a functional R&B song, and the opening lines of the chorus are a nice mix of catchy rhyming and absurdity - Help! I been assaulted - I let the sun come and eat me up. Wait, no, I been exalted (Not really) I am a godless pyramid. It’s great, but I’m not sure it will stick around for long because it has an overly meandering ending. Vapid Feels are Vapid in contrast, ends abruptly. It is closer to a traditional R&B song, and the singing and sentimentality are great. But it’s not as interesting as SAME. I like the direction he’s going in, but I hope that the songs on Clarence Clarity’s next album have more polished endings. # Be’lakor - Vessels Be’lakor is an Australian Melodic Death Metal band beloved of /r/music. I liked their last albums on first listen, but soon got bored of them. I like the singing and the guitar solos, but dislike the overall composition, drumming and the repetitiveness of the guitar and bass lines. But, I had 10$ left in my bandcamp budget at the end of November so I decided to buy their latest album, Vessels.

I have two issues that keep me from liking Be’lakor. The first is that they can write metal epics, like An Ember’s Arc and Roots to Sever, but you have to listen to two minutes of the boring guitar lines and drumming first. The second issue is that the melodies on several songs, such as Withering Strands and Whelm, aren’t quite fleshed out in composition.

In short, I still don’t agree with /r/music.

# Ghosts of Jupiter - The Great Bright Horses

)

Ghosts of Jupiter are a Boston-based psychadelic/progressive rock band. I first heard them when they were the opening act to Blue Öyster Cult, and ended up buying their 2011 debut album Ghosts of Jupiter.

The Great Bright Horses, their kickstarted followup, is leans into psychadelia and out of hard rock of their last album. The production quality is much better, and I appreciate the addition of the piano and flute. The lyrics and reverby melodies paint a picture of looking over a beautiful vista, like an early morning in a national park. It’s the perfect album to do yoga to. This album is a too sleepy - the song Lyra in particular has too many false starts and sounds like a Chopin piano lullaby. When the tempo does pick up, like on The Golden Age, the hard rock melodies are replaced by annoying 60s-style pop rock melodies that ruined the latest Purson album for me. This album is an improvement over their last album, but the absense of the harder grooves keeps it from being great.

# FOES - The Summit Lies Skyward

FOES (Fall of Every Sparrow) are a British progressive band. They have the angelic vocals and alternative-rock drumming of a band like Keane crossed with the occasional harsh vocals and djent sounds of bands like Textures and TesseracT. I liked the single off of this album Beautiful Fiction, as well as their EP Ophir. The things I liked about the single generally apply to the rest of this album - I like the emotion, the nearly imperceptable counter-melodies. They blend styles really well. The lyrics are sometimes too wordy for the accompanying melody, like on the verses of Sworn Host and The Choir Invisible. But that complaint is minor. This band is something special.

26 December 2016

CλaSH is a neat cross compiler from Haskell to VHDL, Verilog and SystemVerilog. Most examples were tested on an Altera FPGA (the DE0-Nano), and not Xilinx (the other major FPGA producer). This blog post demonstrates the procedure for getting Haskell code running on a Xilinx FPGA, which is as (relatively) straightforward as the Altera versions.

Specifically, these instructions are for the Embedded Micro Mojo, which is by far the best Xilinx development board.

# Project Setup

First, generate some verilog from the blinker.hs example.

Start from the embedded micro sample base project. Remove the mojo_top.v file and add the verilog generated by CλaSH.

It will complain that there is no code defining the altpll50 module! The verilog written by CλaSH expects a clock generated by the Altera ALTPLL MegaFunction. To generate a clock that is similar in Xilinx, follow these steps:

• Open up the Xilinx Core Generator (Tools -> Core Generator)
• In Core Generator, open your current project (under File -> Open Project and picking the mojo_top.prj file.
• Make sure the chip is set to your spartan-6 chip. (for the mojo 2 board, this is XC6SLX9-2tqg144)
• Double click on Clocking Wizard in the IP catalog.
• Call the component altpll50, and change the input clock frequency to 50 MHz. On the next screen, change the output clock to 50 MHz
• On page 5 of the process, change the names of the inputs and outputs to match the Altera version. CLK_IN1 -> inclk0, CLK_OUT1-> c0, RESET -> areset, and LOCKED -> locked
• Wait for the Core Generator to make your verilog, then add it to your project using Project -> Add Sources. The altpll50.v should be in your top project directory. Do not add the UCF file!

Change the User Constraints File (.ucf) should be:

NET "CLOCK_50<0>" TNM_NET = clk;
TIMESPEC TS_clk = PERIOD "CLOCK_50<0>" 50 MHz HIGH 50%;
NET "CLOCK_50<0>" LOC = P56 | IOSTANDARD = LVTTL;
NET "KEY1" LOC = P38 | IOSTANDARD = LVTTL;
NET "LED<0>" LOC = P134 | IOSTANDARD = LVTTL;
NET "LED<1>" LOC = P133 | IOSTANDARD = LVTTL;
NET "LED<2>" LOC = P132 | IOSTANDARD = LVTTL;
NET "LED<3>" LOC = P131 | IOSTANDARD = LVTTL;
NET "LED<4>" LOC = P127 | IOSTANDARD = LVTTL;
NET "LED<5>" LOC = P126 | IOSTANDARD = LVTTL;
NET "LED<6>" LOC = P124 | IOSTANDARD = LVTTL;
NET "LED<7>" LOC = P123 | IOSTANDARD = LVTTL;


Notice that KEY0 is missing. The Mojo board has only one button, instead of the DE0-Nano’s two. In the original example, button 0 is used to reset the clock and button 1 is used to invert the state. I’ve decided to set the reset to constant low and set the Mojo’s one button to button 1:

altpll50_2 altpll50_inst
(.inclk0 (CLOCK_50[0])
,.c0 (system1000)
,.areset (1'b0)
,.locked (altpll50_locked));


In addition, there are a few changes that need to be made because of the way Xilinx handles verilog. Hopefully these can be incorporated into CλaSH in the future:

• change the round brackets around CLOCK_50(0) to CLOCK_50[0], same for KEY1
• Xilinx does not like using NOT in wire assignments. If you want to use an inverted signal, simply create an inverted wire, i.e.:
wire not_reset = ~reset;
my_mod my_mod_inst(.reset(not_reset));


Now your project is ready to run! Under processes, right click on Generate Programming File and select run. If you can’t find this option, make sure that you’ve selected the file blinker.v and that it is set as the top module (right-click on blinker.v and set as top module).

If you get a translation error about Symbol ‘MMCM_ADV’ is not supported in target ‘spartan 6’, it means the wrong chip was selected in the clock generation step.

Once it completes successfully, load the generated *.bin file onto the mojo using the Mojo Loader

# End Result

Thanks for putting up with the camera shake, my tripod is in Florida

25 December 2016

In my quest for a simple way to draw with haskell led me to struggle with curses, but I simultaneously discovered and struggled with gtk2hs, a package with haskell bindings to the gtk libraries. Gtk has integrated support for Cairo, a mature graphics library and has a lot of documentation and support.

However, the way things are drawn to the screen changed a lot between Gtk2 and Gtk3. Some functionality has been deprecated, so most gtk2hs demos do not work with Gtk3. Gtk2 is no longer supported in recent versions of Ubuntu, and I have no desire to compile unsupported libraries on my own, so I decided instead to try to make the gtk2hs demos work with Gtk3. The effort was going pretty well until I tried to update keyboard event bindings, for example, that pressing the Escape key closes the window:

window on keyPressEvent $tryEvent$ do
"Escape" <- eventKeyName
liftIO mainQuit


This results in the error:

    Couldn't match type ‘[Char]’ with ‘Data.Text.Internal.Text’
Expected type: System.Glib.UTFString.DefaultGlibString
Actual type: [Char]
In the pattern: "Escape"
In a stmt of a 'do' block: "Escape" <- eventKeyName
In the second argument of ‘($)’, namely ‘do { "Escape" <- eventKeyName; liftIO mainQuit }’  At some point between when this demo was written and modern versions of gtk2hs and haskell, the required type of eventKeyName changed from [Char] to Text. My first guess was to simply attempt to construct a new Text, using: import Data.Text ... (Text "Escape") <- eventKeyName  But the error informs me that I don’t understand how the Text constructor works:  Constructor ‘Text’ should have 3 arguments, but has been given 1 In the pattern: Text "Escape"  The description of the Text constructor is: Construct a Text without invisibly pinning its byte array in memory if its length has dwindled to zero. No explanation of what the two first integers are, but seeing the mention of memory manipulation makes me freak out a little. Hopefully there’s some other method for converting [Char] to Text. It looks like pack is what I need:  (pack "Escape") <- eventKeyName  This results in the error: Parse error in pattern: pack Possibly caused by a missing 'do'?  I don’t understand why I’m getting this error, but I’m guessing it’s because I can’t call a function in a case situation like that. So what if I take it out of the event monad?  escText <- pack "Escape" window on keyPressEvent$ tryEvent $do escText <- eventKeyName liftIO mainQuit  This results in the error: Couldn't match expected type ‘IO t0’ with actual type ‘Text’ In a stmt of a 'do' block: escText <- pack "Escape"  Since I’m getting deeper and deeper down a rabbit hole of string conversions, maybe there’s a way for haskell to interpret string literals in the way that eventKeyName needs. A similar question on stack overflow suggests that I need to add: {-# LANGUAGE OverloadedStrings #-}  to the top of the script. Doing that clears the error around eventKeyName, but it creates errors on every other string literal:  No instance for (Data.String.IsString b0) arising from the literal ‘"Gtk2Hs Cairo Clock"’ The type variable ‘b0’ is ambiguous Note: there are several potential instances: instance Data.String.IsString Data.ByteString.Builder.Internal.Builder -- Defined in ‘Data.ByteString.Builder’ instance Data.String.IsString Text -- Defined in ‘Data.Text’ instance Data.String.IsString [Char] -- Defined in ‘Data.String’ In the second argument of ‘(:=)’, namely ‘"Gtk2Hs Cairo Clock"’ In the expression: windowTitle := "Gtk2Hs Cairo Clock" In the second argument of ‘set’, namely ‘[containerChild := canvas, windowDecorated := False, windowResizable := True, windowTitle := "Gtk2Hs Cairo Clock"]’  But now I know that I can convert strings to Text with pack, so that should make haskell less ambiguous: set window [ containerChild := canvas, windowDecorated := False, windowResizable := True, windowTitle := (pack "Gtk2Hs Cairo Clock") ]  And it works! But I can’t help feel like I’ve cheated somehow. There’s probably a better way to define strings. 23 December 2016 A nameless Ghoul predicts the release of a new Ghost album in the fall of 2017, which feels like ages away. Here are some heavy-metal albums I bought recently to relieve my cravings: # Purson - Desire’s Magic Theatre Purson is a progish, psychadelic heavy-metal band. Their fantastic 2013 debut album The Circle and the Blue Door was critically acclaimed by many prog and metal blogs. They opened for Ghost in their 2015 show in Toronto, and sound like Black Sabbath crossed with Coven. Desire’s Magic Theater (guess which drug this album is about) came out in 2016 and is much tamer in sound than The Circle and the Blue Door. The Way It Is sounds like a reject from the Beatles’ Help. Electric Landlady starts out fantastic, but has a corny major key change in the chorus. Not quite Ghostly enough because the song and melodies are too repetitive and familiar. This album is a disappointment. # Arcana 13 - Danza Macabra Arcana 13 is a talented Italian heavy metal band that combines the soundtracks of old horror movies with a early 00s alt-rock singing style. Their debut album, Danza Macabra, came out in March 2016. Like Ghost, they employ a lot of dramatic unison playing in between heavy guitar solos, with deliciously occult lyrics. Unlike Ghost, they have a tendency to stick to only one style. Despite having killer hook melodies, songs like Dread Ritual tend to go on a little too long. The vocals are constantly distorted, and it would be nice if there was a change in tonality or style. Not quite Ghostly enough because the singing sticks to only one, heavily distorted style, but great in their own right. # Khemmis - Hunted and Absolution Khemmis is a doom/heavy metal rock band. Hunted, an album they released in October of 2016, shortly after 2015’s Absolution, has been called one of the best of 2016 by many metalheads. Most of their singing is operatic but distorted, like Arcana 13, with frequent growling sections, which is a nice balance. They also strike a good mix between atmospheric and melodic sections with heavier meat, like on the song The Bereaved. The production quality and technical ability on Hunted has improved over Absolution, and I look forward to further improvement. My only complaint with either of these albums certain songs, like Antedeluvian, and Candlelight are slightly too doom-y for my taste. Not quite Ghostly enough because of the slower doom-y playing, but great in their own right. # Spellcaster - Night Rides the World Spellcaster is a modern power/heavy band. Their latest album, Night Hides the World, was released in July of 2016, is full of Ghost-like theatrical melodies and complex songs that inspire a large range of emotions without edging into the campiness of most power metal. Their guitar and drumming style is NWOBHM-ish, but the singing style is like a late 00’s pop-punk band. Not quite Ghostly enough because of the constant near-power-metal speed, but great in their own right. 14 December 2016 I’m currently learning Haskell by working on stupid projects. In this post, I’m going to go over my process for trying to generate a rainbow terminal. I’m going to use the haskell-ncurses bindings to draw a rainbow of colored squares to the terimal. Haskell has two types with monad instances: Curses and Update. Update is used whenever the window needs to be redrawn, like on resize. Curses is A small wrapper around IO, to ensure the ncurses library is initialized while running. but I don’t understand what is meant by this. To create a green square on the screen this is the minimal code: import UI.NCurses main = runCurses$ do
w <- defaultWindow
cid <- newColorID ColorGreen ColorBlack 1
updateWindow w $do setColor cid drawString "■" render waitFor w (\ev -> ev == EventCharacter 'q' || ev == EventCharacter 'Q')  where waitFor is taken from the basic ncurses example: waitFor :: Window -> (Event -> Bool) -> Curses () waitFor w p = loop where loop = do ev <- getEvent w Nothing case ev of Nothing -> loop Just ev' -> if p ev' then return () else loop  Next step is to create a field of squares. I’m going to use the forM_ Control Monad. There’s probably a better way of doing this, but as a Haskell newbie, this method feels the most comfortable. import Control.Monad (forM_) num_accross = 7 num_down = 7 ... updateWindow w$ forM_ [(x,y) | x<-[0..num_accross], y<-[0..num_down]] $\(x,y) -> do setColor cid moveCursor y x drawString "■" ...  haskell-ncurses has nine pre-defined colors. I want to assign each row to a different foreground color and each column to a different background color. Let’s try: ... colors = [ColorMagenta, ColorRed, ColorYellow, ColorGreen, ColorBlue, ColorCyan, ColorWhite, ColorBlack] ... updateWindow w$ forM_ [(x,y) | x<-[0..num_accross], y<-[0..num_down]] $\(x,y) -> do cid <- newColorID (colors !! x) (colors !! y) 0 setColor cid moveCursor y x drawString "■" ...  But this leads to the error:  Couldn't match type ‘Curses’ with ‘Update’ Expected type: Update () Actual type: Curses ()  This error occurs because newColorID returns an instance of Curses, but I’ve put it in a Monad bound to the an instance of the Update type. I think I can solve this by switching the order of operations:  forM_ [(x,y) | x<-[0..num_accross], y<-[0..num_down]]$ \(x,y) ->
do
cid <- newColorID (colors !! x) (colors !! y) 1
updateWindow w $do setColor cid moveCursor y x drawString "■"  That clears up the type error. Let’s move on to the other errors:  Couldn't match expected type ‘Integer’ with actual type ‘Int’ In the first argument of ‘moveCursor’, namely ‘y’ In a stmt of a 'do' block: moveCursor y x  This error is interesting because it goes away if I remove the newColorID line. The newColorID line must be casting the x and y values to Integers, but moveCursor is unable to cast them back to Int. I can force it to cast to Int using the fromIntegral function: moveCursor (fromIntegral y) (fromIntegral x)  This code works without error, but results in a screen full of white squares: This is because I’m naming all the colors in my pallette as 1. Instead, let’s name them based on their position in the grid: cid <- newColorID (colors !! x) (colors !! y) ((x*(num_accross+1))+y+1)  This leads to the error: Couldn't match expected type ‘Integer’ with actual type ‘Int’ In the first argument of ‘(*)’, namely ‘x’ In the first argument of ‘(+)’, namely ‘(x * num_accross)’  OMG Haskell just pick a type and stick with it! I can cast back to Int using toInteger: cid <- newColorID (colors !! x) (colors !! y) (toInteger ((x*(num_accross+1))+y+1))  This works! Okay, but now I want to paint with all the colors of the wind. We can define custom colors using Color. First, let’s expand the window to more rows and columns. To avoid the error: ncurses_minimal.hs:15:1: Warning: Tab character ncurses_minimal.hs: CursesException "moveCursor: rc == ERR"  We need to change the allowable size of the window: updateWindow w$ resizeWindow num_accross num_down


haskell-ncurses has a defineColor function that allows for custom colors, but like newColorID, it can only be applied to colors in the colors in the terminal’s list of colors. The number of colors can be checked using the code:

maxid <- maxColorID
updateWindow w drawString (show maxid)  On GNOME-terminal, this is 255, which is far less than most displays can output, but more than enough to make some pretty pictures. Like ColorPairs, colors need to be assigned to specific addresses in the table. Addresses should start at 1 so that 0 can be reserved for black. I’m not sure why Colors can start at 0 but ColorPairs need to start at 1, but this means that we can use the same equation for the indices of both. ... num_accross = 24 num_down = 9 ... let colIndex = x*(num_down+1)+y+1 let my_color = Color (fromIntegral colIndex :: Int16) defineColor my_color (x*36) 0 (y*110) cid <- newColorID my_color ColorBlack (toInteger colIndex)  Which leads to another pretty picture: I don’t quite understand why I need to use let in certain cases and single-assignment operators in others. Maybe it’s because the single-assignment is used in a nested monad? The code for this exercise is available as a gist. 09 December 2016 Unwrap My Heart is a Twilight parody written by Portland Standup Comic Alex Falcone and comedy writer Ezra Fox. The authors also created the Read it and Weep Podcast, which started by reviewing Twilight, then tackled other young adult fiction, and have more recently moved on to mostly reviewing bad movies. I first listened their review of Gossip Girl1 with guest reviewer Amanda Leinbaugh of Skepchick and have been a fan since. Through their podcast, the writers have developed an intimate knowledge of understanding of the genre. But Unwrap My Heart can’t be lumped in with other parodies like Barry Trotter and Bored of the Rings because in addition to hitting all of the obvious flaws in Twilight, the meat between the parody is also funny. A lot of the jokes were said first on Read it and Weep, but they’re even funnier in a fictional context. The ancillary characters are adorable. I would read a sequel that was just Sofia’s weatherman father and her failed 90s comic teacher Mr. Graff taking over her effort to get her cat internet famous while she’s off fighting mummies. The book isn’t perfect. It’s a little over-written in a few places, but it has far fewer stylistic flaws than other famous first books like The Martian or Ready Player One. Sofia is smart yet can’t figure out that her crush is a mummy or that her friend is upset because he’s part of a love angle. It’s never fully explained why the mummy has a porche (at least it’s not a volvo?) or has to attend high school. And I’m having a really hard time imagining a handsome face without eyeballs. It feels weird to nitpick a parody. This book is the funniest thing I’ve ever read, perhaps because it also has a lot of heart. Sofia is one of the most accurate written depictions of what it’s like to be an awkward teenage girl. Who would have thought that it would come from two 30-something men writing a Twilight parody? 1: Gossip Girl, along with Ashlee Simpson and sexual inuendo vintage graphic tees from American Eagle formed the triumvirate of things other girls at my high school loved but that I hated. Why read descriptions about what shitty rich people are wearing? Why drive 2.5 hours to buy one of 5 shirts at the MicMac Mall when at least one other person will also be wearing when you go to school on Monday? 30 November 2016 I thought that Erik Meijer’s edX class on functional programming was super easy until the lecture on functional parsers and monads, and now I’m so confused and anxious about my inability to become unconfused that I think I’ll have to quit the course for now. The example monads on wikipedia make sense to me, but I can’t figure out how to call them in hugs. For example, the course provided some code with the following parsers defined: > many :: Parser a -> Parser [a] > many p = many1 p +++ return [] > > many1 :: Parser a -> Parser [a] > many1 p = do v <- p > vs <- many p > return (v:vs)  I don’t understand what these parsers do. My typical way of figuring out what code does is to execute it on some test input and reason about what it must be doing from the output. My first guess at how to execute this parser was incorrect: Parsing> many "hello" ERROR - Type error in application *** Expression : many "hello" *** Term : "hello" *** Type : String *** Does not match : Parser a  Okay, so many needs the input of something of type Parser a instead of a string. Looking through the sample code, it looks like I could use digit: > digit :: Parser Char > digit = sat isDigit  This gives me a different error: Parsing> many digit ERROR - Cannot find "show" function for: *** Expression : many digit *** Of type : Parser [Char]  I interpret this to mean that the output of many digit is a Parser [Char], a type which doesn’t have a way to defined way to print characters to the terminal. The definition of the type is: > newtype Parser a = P (String -> [(a,String)])  So if I feed the parser a string, I should get a list of tuples where the first entry is the parser input and the second entry is the resulting parsed string? However, feeding it a string argument does not work: Parsing> many digit "hello" ERROR - Type error in application *** Expression : many digit "hello" *** Term : many *** Type : Parser d -> Parser [d] *** Does not match : a -> b -> c  Okay, so many is complaining again. I don’t understand this error. I think my fundamental confusion is that I don’t understand what the type: Parser a -> Parser [a]  means. My first guess is that it takes a parser that takes input a and creates a parser with input list of a. But I don’t understand how this could be useful, since it seems like this would just generate an infinite amount of information. Or maybe my understanding is reversed, that this takes a list of things and allows them to be parsed item by item. However, I’ve also tried: Parsing> many digit ['a', '0', '1'] ERROR - Type error in application *** Expression : many digit ['a','0','1'] *** Term : many *** Type : Parser d -> Parser [d] *** Does not match : a -> b -> c  Parsing> many (digit) ["abc", "120", "12"] ERROR - Type error in application *** Expression : many digit ["abc","120","12"] *** Term : many *** Type : Parser d -> Parser [d] *** Does not match : a -> b -> c  At this point I’m going to give up and play video games for a bit and hope that my frustration and confusion abates enough to make more progress. # Clarification Thanks to Aaron Levin, and re-reading the course material, I have become unstuck. First I need a function that actually executes the parser, like this: > parse :: Parser a -> String -> [(a,String)] > parse (P p) inp = p inp  Next, I have to define the behavior of the Parser type when invoked as a monad (I think this is what this is for): > instance Monad Parser where > return v = P (\inp -> [(v,inp)]) > p >>= f = P (\ inp -> > case parse p inp of > [(v, out)] -> parse (f v) out > [] -> [])  Then, I can finally get the parser to work on a string with: Parsing> parse (many digit) "0234a" [("0234","a")]  Note that the brackets are required for order of operations, else the compiler will interpret many and digit as both being arguments to parse: Parsing> parse many digit "b0234a" ERROR - Type error in application *** Expression : parse many digit "b0234a" *** Term : parse *** Type : Parser e -> String -> [(e,String)] *** Does not match : a -> b -> c -> d  26 November 2016 This post covers some albums I purchased by bands I encountered as opening acts at concerts I’ve recently attended. # Fallujah San Francisco’s Fallujah opened for Between the Buried and Me. Fallujah is progressive djenty metal crossed with Aphex-Twin, which is refreshing mix. Their style oscillates between being an emo Fleshgod Apocalypse or what Jean Michel Jarre’s metalcore experiments might have sounded like. So far I like this band a lot, but I have two complaints. The changes between technical death metal and melodic parts are very abrupt, especially on Harvest Womb’s Enslaved Eternal Phenomenon. Their ambiance verges on filler - such as Nomadic EP’s Silent. I’m not sure if I can think of a situation appropriate for some of their songs - it’s too mellow to be programming or exercising music, but too metal for an intimate party or reading in bed. Similarly, Venom on the Blade is another track that has a great first two minutes but becomes annoyingly lethargic by minute 5. Fallujah is not the most progressive band, but they encorporate new sounds and their execution is competent. If you want some fresh-sounding technical, check them out. # Pallbearer Pallbearer opened for Baroness. Though classified as doom metal, the slowness of their playing reminds me more of stoner metal bands like Sleep, though with Baroness’s chords. They’re good, but have neither the melodic or rhythmic complexity of Sleep nor the emotional breadth or depth of *Baroness. The singing is overly mournful, such as on their cover of Black Sabbath’s Over and Over, or songs take way too long to get started, such as on Sorrow and Extinction’s Foreigner. Still, tracks that don’t have much singing are great. If you like Baroness, check them out. # Helen Money Helen Money is a cellist and composer who opened for MAGMA. Unlike the previous two bands, she shares nothing with the act she opened for, except for an appreciation of orchestral instruments. I would classify her as being in the category of metal-adjacent experimental musicians like Chelsea Wolfe. Her music would be the perfect soundtrack to a purposefully tonally jarring Sci-Fi movie like Under the Skin or Melancholia. Her music may not be something to mosh to, but it is great music for writing. # Tony MacAlpine Tony MacAlpine opened for Steve Vai. Like Vai, MacAlpine is a virtuoso guitarist who became famous in the 80s by playing for heavy metal bands. Unlike Vai, MacAlpine’s early musical influences were classical piano rather than popular rock music. MacAlpine’s music is more technical and less gimmicky than Vai’s as a result. Electronic keyboard covers of Chopin crop up several times in his albums, but are included as an expression of admiration rather than as schtick, like E.L.P.’s Hammond organ covers of Mussorsky. Having now heard both discographies, I prefer MacAlpine to Vai. MacAlpine dabbled in several musical groups throughout his discography, and a lot of it is skipable. His time in the supergroup M.A.R.S. produced some great solos, but also exhibited the worst aspects of New Wave of British Heavy Metal bands. Similarly, his work in CAB is also great, but I’m not into light jazz. The fake trumpets, slow, funky rhythms and repetitive melodies are ear-grating. Also skip the albums where MacAlpine sings, such as Master of Paradise I would highly recommend the albums Edge of Insanity (1986), Madness (1993), and Chromaticity (2001), which each consist of the best in metal composition trends and styles of the decade prior to their release date. # Brain Tentacles Brian Tentacles opened for Gorguts, and are a death metal band with a baritone and bass saxophone. The saxophone works better as a metal instrument on certain songs than on others. It has a tendency to slow down the music, for example, Gassed and The Sadist sound like a Captain Beefheart songs with a grating and short repetitive melody. By contrast, The Spoiler and Death Rules work better because they let other instruments breath, or have a longer melody. Despite the few misses, this band is really neat and fun both live and on their debut album. I’m hooked. # Intronaut Intronaut also opened for Gorguts. They’re a technical death metal in a more subdued folk-metal style like Nechochwen rather than the aggressive style used by Between the Buried and Me on Colors and The Great Misdirect. My only complaint with their style is that, like a lot of modern progressive bands, they go for overkill on the samples - such as The Unlikely Event after landing. An almost minute-long weird sample intro leads into an otherwise nice song. I wish these bands would cut their samples into interlude tracks, so they can be deleted. Believing that these sample interludes truly add to the song is pretentious. 25 November 2016 Out of the long list of cross-platform python GUI options, the two that I’m most familiar with are Tkinter and PySide. The purpose of this post is to demonstrate how to create the same application with a separate data acquisition thread using both of these libraries and discuss the differences between them. The code for these sample applications is available in my github repository python-gui-demos. # The Program The demo program measures the amount of waking day elapsed, calculated by comparing the current time against the a set wake time and bed time in the config file. The waking day elapsed is displayed using a progress bar and a label, a pause button prevents these elements from updating, and a quit button closes the application. This is accomplished by defining a GuiPart object and a ThreadedClient object which can communicate between each other. For the sake of comparison, I’ve used the same names for both implementations: The GUI uses a grid layout and looks like this: The grid layout has 3 rows and 2 columns, and the progress bar spans two rows. # Implementation ## Setting Up Objects Through Qt, PySide has a thread object - the QtCore QThread. Tkinter does not provide a tk-flavoured thread object, so we’ll use the standard library thread object. In python 2, the tkinter Tk object is implemented as a classobj instead of a type, so in order for our GUI to inherit it be have to make it a metaclass of Tk and object.  PySide Tkinter class GuiPart(QtGui.QWidget): ... class ThreadedClient(QtCore.QThread): ...  class GuiPart(tk.Tk, object): ... class ThreadedClient(threading.Thread): ...  ## Widget Layout In PySide, the layout is a separate object from the main window’s QWidget called QGridLayout. GUI widgets are then added to this main layout with the row/column/rowspan and columnspan information. In Tkinter, the main window is passed as the first variable to each GUI widget, and then can be packed into a grid using the .grid() method. You can also add elements without a grid layout using .pack(), but combining calls to .pack() and .grid() will result in errors.  PySide Tkinter self.main_layout = QtGui.QGridLayout() self.day_bar = QtGui.QProgressBar(self) self.day_label = QtGui.QLabel(self) self.pause_button = QtGui.QPushButton("Pause", self) self.quit_button = QtGui.QPushButton("Quit", self) program_label = QtGui.QLabel(cfg.get("variables", "main_label"), self) self.main_layout.addWidget(program_label, 0, 0, 1, 1) self.main_layout.addWidget(self.day_label, 1, 0, 1, 1) self.main_layout.addWidget(self.day_bar, 0, 1, 2, 1) self.main_layout.addWidget(self.pause_button, 2, 0, 1, 1) self.main_layout.addWidget(self.quit_button, 2, 1, 1, 1) self.setLayout(self.main_layout)  self.day_bar = ttk.Progressbar(self) self.day_bar.grid(row=0, column=1, rowspan=2) self.day_label = tk.Label(self) self.day_label.grid(row=1, column=0) program_label = ttk.Label(self, text=cfg.get("variables", "main_label")) program_label.grid(row=0, column=0) self.pause_button = ttk.Button(self) self.pause_button.grid(row=2, column=0) self.quit_button = ttk.Button(self, text="Quit") self.quit_button.grid(row=2, column=1)  ## Slots, Signals and Callbacks In order to have a responsive GUI, signals emitted from user’s interaction to the trigger the desired action. PySide Widgets come with several pre-defined signals, or it is possible to define your own. Signals can be connected to the desired method using the .connect() method. In Tkinter, buttons have a slot for specifying the action on clicking called command. Here’s how to define button callbacks:  PySide Tkinter self.pause_button.clicked.connect(self.pause) self.quit_button.clicked.connect(self.quit_click)  self.pause_button.configure(command=self.pause) self.quit_button.configure(command=self.quit_click)  Updating the appearance of Widgets does not easily lend itself to side-by-side comparison, because PySide allows signals to be emitted from any defined type in Python, whereas Tkinter allows for only 3 variables - StringVar, IntVar and DoubleVar. In PySide, we can create a custom signal called current_datetime, PySide class ThreadedClient(QtCore.QThread): current_time = QtCore.Signal(datetime) ... def run(self): while True: if not self.parent.paused: self.current_time.emit(datetime.now()) sleep(1)  and hook it up to the .update(current_datetime) method: class GuiPart(QtGui.QWidget): def __init__(self): ... self.thread.current_time.connect(self.update) ... @QtCore.Slot(datetime) def update(self, current_datetime): percent_elapsed_value = percent_elapsed(current_datetime) self.day_bar.setValue(percent_elapsed_value) self.day_label.setText(str(percent_elapsed_value))  Note that the update(current_datetime) needs a decorator to describe that the incoming signals are of type datetime. In Tkinter, Widget attributes must be defined by variables in order to be updated. For example: Tkinter class GuiPart(tk.Tk, object): def __init__(self): ... self.day_bar_value = tk.IntVar() self.day_label_value = tk.StringVar() self.day_bar.configure(variable=self.day_bar_value) self.day_label.configure(textvariable=self.day_label_value) ... def update(self, current_datetime): percent_elapsed_value = percent_elapsed(current_datetime) self.day_bar_value.set(percent_elapsed_value) self.day_label_value.set(str(percent_elapsed_value))  We could use bind to create a custom event to trigger the GUI update, however, it would not communicate the datetime information back to the GUI. Since we kept a reference to the GUI in the ThreadedClient object, we can call the method directly. class ThreadedClient(threading.Thread): ... def run(self): while True: ... if not self.parent.paused: self.parent.update(datetime.now()) sleep(1)  ## Styling PySide allows elements to be styled using CSS, and can be applied to the object using setStyleSheet. Like webpage style sheets, styles inherit the style of the encapsulating element unless otherwise specified, so applying the style sheet to the main window will result in the same style sheet being applied to both buttons and the program_label. Tkinter elements can either be styled using the .configure() method, if they were tk widgets, or using ttk.Style() objects. In this example I will style one element using the tk style and the others using ttk.  PySide Tkinter self.setStyleSheet("QWidget { " "background-color: " "\""+cfg.get('colors', 'background')+"\";" "font-family:"+cfg.get('font', 'face')+"; " "font-size: "+cfg.get('font', 'size')+"pt;" "color: \""+cfg.get('colors', 'text')+"\";" "}") self.day_label.setStyleSheet("QLabel { " "background-color: " "\""+cfg.get('colors', 'text')+"\";" "color: \""+cfg.get('colors', 'sub_text')+"\";" "border: "+cfg.get('layout', 'border_width')+"px" " solid \""+cfg.get('colors', 'border')+"\";" "}") self.day_bar.setStyleSheet("QProgressBar{ " "background-color: " "\""+cfg.get('colors', 'text')+"\";" "border: "+cfg.get('layout', 'border_width')+"px" " solid \""+cfg.get('colors', 'border')+"\";" " } " "QProgressBar::chunk { " " background-color: " "\""+cfg.get('colors', 'sub_text')+"\";} ")  self.configure(background=cfg.get('colors', 'background')) s = ttk.Style() s.configure('TButton', background=cfg.get('colors', 'background')) s.configure('TButton', activebackground=cfg.get('colors', 'background')) s.configure('TButton', foreground=cfg.get('colors', 'text')) s.configure('TButton', highlightbackground=cfg.get('colors', 'background')) s.configure('TButton', font=(cfg.get('font', 'face'), int(cfg.get('font', 'size')))) s.configure('TLabel', background=cfg.get('colors', 'background')) s.configure('TLabel', foreground=cfg.get('colors', 'text')) s.configure('TLabel', highlightbackground=cfg.get('colors', 'background')) s.configure('TLabel', font=(cfg.get('font', 'face'), int(cfg.get('font', 'size')))) s.configure('Vertical.TProgressbar', background=cfg.get('colors', 'sub_text')) s.configure('Vertical.TProgressbar', troughcolor=cfg.get('colors', 'text')) s.configure('Vertical.TProgressbar', highlightbackground=cfg.get('colors', 'border')) s.configure('Vertical.TProgressbar', highlightthickness=int(cfg.get('layout', 'border_width'))) self.day_label.configure(background=cfg.get('colors', 'text')) self.day_label.configure(foreground=cfg.get('colors', 'sub_text')) self.day_label.configure(highlightbackground= cfg.get('colors', 'border')) self.day_label.configure(highlightthickness=2) self.day_label.configure(font=(cfg.get('font', 'face'), int(cfg.get('font', 'size'))))  ## Filling the screen PySide spaces elements to fill the entire screen by default, but the main window must be set to fullscreen. Tkinter occupies only the minimal amount of space required by default. To get Tkinter elements to expand, certain rows and columns must be given weight:  PySide Tkinter self.showFullScreen()  self.attributes("-fullscreen", True) self.columnconfigure(0, weight=1) self.rowconfigure(0, weight=1) self.rowconfigure(1, weight=1)  # Final Result After all of that styling, here’s the final result:  PySide Tkinter Things that make this not quite a fair comparison are: • I didn’t bother with stickies in my grid layout in Tkinter, hence the difference in sizes in elements. • I was not able to figure out how to change the border of TProgressBar in Tkinter, though it may still be an option I haven’t found • text is not aligned to center in the PySide version # Which library should you pick? Tkinter has the same GNU General Public License as Python. PySide is licensed under the GNU Lesser General Public License. Both of these licenses allow for their use in proprietary software, however the LGPL is a bit friendlier to commercial development than GPL as the requirements to exactly copy the license are less strict. If you’re going to use PySide, the other parts of your code will almost certainly fall under Python’s GPL license, so this point seems moot. Though both libraries can be employed in object-oriented and procedural fashions, the Tkinter examples tend to use procedural programming and the PySide examples tend to use object-oriented programming, meaning that object-oriented programming neophytes may find PySide intimidating. However, PySide has a greater selection of functionality than Tkinter, in terms of variety of widgets, signals, slots and display options. PySide uses the styles defined in your installed version of Qt, which means that by default PySide programs tend to look less dated than Tkinter programs. PySide is a bit like using LaTeX over Word - though more complicated, the default layout rules are better than Tkinter, allowing design newbs like me to trade technical skill for aesthetics. In conclusion, my advice would be that Tkinter is good for prototyping simple applications, but if you plan on adding functionality in the future or presenting a polished product, start with PySide. 23 November 2016 In order to use a Teensy through a Raspberry Pi’s USB ports, you must change the supplied PJRC rule from: SUBSYSTEMS=="usb", ATTRS{idVendor}=="16c0", ATTRS{idProduct}=="04[789ABCD]?", MODE:="0666"  to: ATTRS{idVendor}=="16c0", ATTRS{idProduct}=="04[789ABCD]?", MODE:="0666"  I’m really not sure why. dmesg looks pretty much the same to me on my Ubuntu 16.04 (xenial) machine: [93358.756212] usb 6-1: new full-speed USB device number 7 using xhci_hcd [93358.915929] usb 6-1: New USB device found, idVendor=16c0, idProduct=0483 [93358.915939] usb 6-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [93358.915945] usb 6-1: Product: USB Serial [93358.915949] usb 6-1: Manufacturer: Teensyduino [93358.915953] usb 6-1: SerialNumber: 1578410 [93358.918104] cdc_acm 6-1:1.0: ttyACM0: USB ACM device  as on my Rasbian 6 (jessie): [ 88.988423] usb 1-1.4: new full-speed USB device number 5 using dwc_otg [ 89.091861] usb 1-1.4: New USB device found, idVendor=16c0, idProduct=0483 [ 89.091882] usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 89.091895] usb 1-1.4: Product: USB Serial [ 89.091908] usb 1-1.4: Manufacturer: Teensyduino [ 89.091921] usb 1-1.4: SerialNumber: 1578410 [ 89.133811] cdc_acm 1-1.4:1.0: ttyACM0: USB ACM device [ 89.135216] usbcore: registered new interface driver cdc_acm [ 89.135229] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters  15 November 2016 I’ve written some basic sample code for communicating between an FPGA and a raspberry pi, with both serial and parallel examples. The demo project pulls x, y, and z accelerometer data off of the DE0-Nano Terasic Altera Cyclone IV development board and prints it to the terminal in the Raspberry Pi. This post goes over setting up the examples and explains some of the code. # Functionality Common to Both Examples The verilog and tcl common to both DE0-Nano projects is in de0nano/common The accelerometer puts out 16 bits of data for each of the three dimensions. The basic functionality of both examples is that the Raspberry Pi puts out one of the bytes for ‘x’, ‘y’ and ‘z’, and the DE0-Nano responds with two sequential bytes, one for the lower half of the 16 bits, one for the upper half. The accelerometer verilog code comes from Terasic, but with a modified module which includes the 2-bit input register called dimension. When dimension is 0, the axis to be read is x, 1 is y and 2 is z. This is accomplished by setting the hex values sent to the accelerometer using ternary operators in spi_ee_config.v: wire[5:0] LB = (dimension==0) ? X_LB: ((dimension==1) ? Y_LB: Z_LB); wire[5:0] HB = (dimension==0) ? X_HB: ((dimension==1) ? Y_HB: Z_HB);  The 50 MHz clock, accelerometer pins, reset button, and LEDs are all defined in the file common_pins.tcl. Both examples use a simple state machine to handle responding to incoming signals. It looks like this: The check signal for each example is different for both examples; in the serial example, it’s either read in data being ready, or the transmission line being free, in the parallel example, it’s the incoming clock pin from the Raspberry Pi. Here’s the verilog for this state machine for the serial example: always @(posedge (TxD_busy | RxD_data_ready)) if(write_state == 0) begin if(120 <= RxD_data <= 122) write_state = 1; if(RxD_data == 120) dimension = 0; else if(RxD_data == 121) dimension = 1; else if(RxD_data == 122) dimension = 2; end else if(write_state == 3) write_state = 0; else write_state = write_state + 1;  # Serial Example ## Setup • Plug GPIO03 on the DE0-Nano into GPIO 15 on the Raspberry Pi. • Plug GPIO05 on the DE0-Nano into GPIO 14 on the Raspberry Pi. To load the DE0-Nano, can either import the verilog and tcl files into your own quartus project, or use my pyquartus tool. To compile and upload using pyquartus, plug your DE0-Nano into your computer, and run: cd de0-nano-raspi-serial-demo pyquartus -c -u -i de0nano/common -p de0nano/serial  On the raspberry pi side, install the requirements and run the program by opening up a terminal and running the commands: cd de0-nano-raspi-serial-demo/raspi pip install -r requirements.txt python serial/main.py  ## Code The serial verilog code was adapted from Jean P Nicolle at fpga4fun. I’ve made some modifications to it, including separating the modules and creating a header file to define a common baud rate in parameters.h: parameter BAUD_RATE = 460800; parameter CLK_FREQUENCY = 50000000;  The example requires two state machines - one to handle the transitions between read to write, one to handle setting the output data. This is because the TxD_data_ready is used as a signal for transitions between read and write and thus can’t be set by the same state machine it drives. A second state machine drives TxD_data_ready and the data: always @(write_state) if (~TxD_busy) if(write_state == 1) begin TxD_data_ready = 1'b1; TxD_data <= data[7:0]; end else if(write_state == 2) begin TxD_data_ready = 1'b1; TxD_data <= data[15:0]; end else if(write_state == 3) begin TxD_data_ready = 1'b0; end else if(TxD_busy) begin TxD_data_ready = 1'b0; end  The Raspberry Pi side is driven by python using the pyserial library: from time import time from sys import argv from serial import Serial rate = argv[1] if rate != 0 and rate != 1: rate = 0 rates = [460800, 115200] print("setting rate to: ", rates[rate]) conn = Serial('/dev/ttyAMA0', baudrate=rates[rate], timeout=2) def read_dimension(dimension): global conn failure_count = 0 # the serial connection often fails to read two single bytes while True: try: conn.write(dimension) value = ord(conn.read(1)) value += ord(conn.read(1)) << 8 return value except Exception as e: failure_count += 1 if __name__ == "__main__": while True: start = time() x_val = read_dimension(b'x') y_val = read_dimension(b'y') z_val = read_dimension(b'z') print(x_val, y_val, z_val, time()-start)  # Parallel Example ## Setup • Plug GPIO 133 on the DE0-Nano into GPIO 8 on the Raspberry Pi • Plug GPIO 131 on the DE0-Nano into GPIO 10 on the Raspberry Pi • Plug GPIO 129 on the DE0-Nano into GPIO 24 on the Raspberry Pi • Plug GPIO 127 on the DE0-Nano into GPIO 4 on the Raspberry Pi • Plug GPIO 125 on the DE0-Nano into GPIO 17 on the Raspberry Pi • Plug GPIO 132 on the DE0-Nano into GPIO 22 on the Raspberry Pi • Plug GPIO 130 on the DE0-Nano into GPIO 9 on the Raspberry Pi • Plug GPIO 128 on the DE0-Nano into GPIO 25 on the Raspberry Pi • Plug GPIO 126 on the DE0-Nano into GPIO 18 on the Raspberry Pi • Plug GPIO 124 on the DE0-Nano into GPIO 23 on the Raspberry Pi To load the DE0-Nano, can either import the verilog and tcl files into your own quartus project, or use my pyquartus tool. To compile and upload using pyquartus, plug your DE0-Nano into your computer, and run: cd de0-nano-raspi-serial-demo pyquartus -c -u -i de0nano/common -p de0nano/parallel  On the raspberry pi side, install the requirements and run the program: cd de0-nano-raspi-serial-demo/raspi pip install -r requirements.txt python parallel/main.py  ## Code On the parallel example, the Raspberry Pi does the heavy lifting. It uses a pin for clock, a pin to indicate whether the data pins are being used for reading and writing, and then 8 pins for data. This is accomplished using the RPi.GPIO python library: def send_byte(byte_out): """ Send a single byte. """ GPIO.output(clock_pin, 0) # set the chip select to write GPIO.output(chip_select, 1) # send the byte values = [(ord(byte_out) >> i) % 2 for i in range(0, 8)] GPIO.setup(data_pins, GPIO.OUT) GPIO.output(data_pins, values) # flash the clock pin GPIO.output(clock_pin, 1) GPIO.output(clock_pin, 0) def get_byte(): """ Get a single byte. """ GPIO.setup(data_pins, GPIO.IN) # read the data pins GPIO.output(chip_select, 0) GPIO.output(clock_pin, 1) GPIO.output(clock_pin, 0) value = 0 for i in range(0, 8): value += GPIO.input(data_pins[i]) << i return value  In verilog, the code is pretty simple and can be accomplished with a ternary operator and some logic triggered by the clock: module parallel_txrx( input clock, input chip_select, inout [7:0] data_pins, output reg [7:0] data_in, input [7:0] data_out ); reg [7:0] data_out_internal; assign data_pins = chip_select ? 8'bZ : data_out_internal; always @(negedge clock) // if chip select is high, the fpga is reading in data if(chip_select) data_in <= data_pins; else data_out_internal <= data_out; endmodule  I use the negative edge of the clock signal to ensure the data pins on the Raspberry Pi are synchronized and ready. # Performance The parallel example takes ~60 microseconds to send and receive a byte, and ~25 microseconds to receive a byte. That results in a transmission speed of ~133kHz if sending and receiving is required, and ~320kHz if only receiving data. The serial example can transmit at 460800 baud, however, after about a second of communication, the Raspberry Pi and FPGA stop communicating. I’m not exactly sure why this happens, but running the verilog code with a teensy serial interface is much more stable. In both of these cases, it might be possible to get faster and more reliable performance by switching to c. 08 November 2016 I’m a dual-Canadian-American citizen who grew up in Nova Scotia, spent her early adult life in Ontario, and recently moved to California. Today I voted in-person for the first time in a US election. I love both my countries of citizenship, but I think both countries could learn from each other. Based on my personal experience, I would say that the Canadian voting experience is superior. Here’s why: # Registration In Ontario and Nova Scotia, I was automatically registered to vote through my government health card. While I was a student, volunteers came to my dorm room to help me update my address from my parent’s house to my dorm address. In California, I was prompted for fill out and send in a separate form to the San Diego registrar when I applied for my driver’s license, which I did almost a year ago. A month before the election, the registrar sent a letter to my address stating that no one at my address was registered to vote. This gave me a minor panic attack before I could get online and confirm that I was registered. I don’t know how this mistake happened. There were at least three voters in the polling station in California who thought they were registered but were not. # Meeting Candidates In Canada, it has been impossible not to meet someone running for office during an election year. Elizabeth May showed up at a work party at my first job. Peter MacKay used to show up at after-school activities. One Liberal candidate in Montreal was waiting outside the door to a grocery store to introduce me to his daughter when I was covered in mud and sweat from mountain biking and carrying a case of beer. I have not met any candidate running for office in this year’s US election, even those running for city council. My closest encounter was that my husband, who currently lives in Florida, was passed by Kaine’s motorcade. # Voting Location In every Canadian election I have voted in, my polling place has been in my apartment building or housing complex. I live in a similarly-sized apartment complex in California, but my polling location is a 15 minute walk away, with no bike parking. It is right off of a freeway exit, however. # Accessibility In Canada, voting booths have been at sitting height, or locations have had both standing and sitting height booths. In California, all voting booths in the main voting area were at standing height. I observed that I voter in a wheelchair had to go into one building to check-in, then exit the building and re-enter through another entrance to vote in private. # Privacy In Canada, there is only one name on the ballot. Since it is small, it is easy to fold over and shove in the ballot box in your hand. In California, poll workers were having difficulty shoving the two large pieces of paper without opening the ballot protector. Several voters discarded the shield to speed up the process. # Voting Time In Canada, in 2015, I voted after work at 5 pm. There was no line, and it took me 3 minutes to vote. It would have taken less time, but one of the observers had chosen that time to go for a bathroom break. In every election, there have been as many poll workers as voters at the polling station when I’ve voted. In California, I went to the poll ten minutes after it opened and there was a 25 minute line. I would attribute wait to there only being 4 poll workers for a crowd of 50 people. # Conclusion Neither country’s voting experience is terrible, but there was a lot less friction in voting in Canada. With a few changes, i.e., having a better checked list of registered voters and addresses, putting at least one polling booth at wheelchair height, and hiring more election workers, the American voting experience could be as good as the Canadian experience. 08 November 2016 Suppose you want to annoy the people around you by running as many speakers or HDMI displays as possible at the same time off of your DE0-Nano Cyclone-IV based FPGA development board. Figuring out where to plug in your HDMI pins is stricky because is tricky because Altera provides little documentation on exactly which pins in which banks on the Cyclone IV E chip can support LVDS, many pins are already routed to the peripherals on the DE0 Nano, some pins are input-only, and LVDS pins must be at least 4 spaces from any other signal pins. In this post, through a lot of trial and error and fussing with the pin planner, I’ll document everything I know about which pins can be used. # Assigning LVDS Pins Quartus’ pin planner is frustrating and often behaves unpredictably, especially when assigning pins in pairs. I would not recommend using it. You can get around it by adding the following lines: set_instance_assignment -name IO_STANDARD LVDS -to HDMI[0] set_location_assignment PIN_K15 -to HDMI[0] set_location_assignment PIN_K16 -to "HDMI[0](n)"  to a .tcl file, which you can then add to your project by adding the line: set_global_assignment -name SOURCE_TCL_SCRIPT_FILE pins.tcl  to your project’s .qsf file. Alternatively, you can always directly copy and paste these lines into the .qsf file, but I like to keep them separate for readability. # Pin Survivor Out of the 256 pins on the Cyclone IV 4 EP4CE22 device, here are the 47 LVDS pairs:  (A10, B10) (A11, B11) (A12, B12) (A13, B13) (A2, A3) (A4, B4) (A5, B5) (A6, B6) (A7, B7) (A8, B8) (A9, B9) (C1, C2) (C14, D14) (C16, C15) (C9, D9) (D1, D2) (D11, D12) (E16, E15) (E8, F8) (F1, F2) (F16, F15) (G1, G2) (G16, G15) (J1, J2) (J16, J15) (K1, K2) (K16, K15) (L1, L2) (L16, L15) (M1, M2) (M16, M15) (N1, N2) (N16, N15) (N6, N5) (P1, P2) (P16, R16) (P3, N3) (T10, R10) (T11, R11) (T12, R12) (T13, R13) (T15, T14) (T3, R3) (T5, R5) (T6, R6) (T7, R7) (T9, R9) Removing all pins not accessible through JP1, JP2, and JP3 on the DE0-Nano leaves us with 19 pairs:  (A10, B10) (A11, B11) (A12, B12) (A13, B13) (A2, A3) (A4, B4) (A5, B5) (A6, B6) (A7, B7) (A8, B8) (A9, B9) (C1, C2) (C14, D14) (C16, C15) (C9, D9) (D1, D2) (D11, D12) (E16, E15) (E8, F8) (F1, F2) (F16, F15) (G1, G2) (G16, G15) (J1, J2) (J16, J15) (K1, K2) (K16, K15) (L1, L2) (L16, L15) (M1, M2) (M16, M15) (N1, N2) (N16, N15) (N6, N5) (P1, P2) (P16, R16) (P3, N3) (T10, R10) (T11, R11) (T12, R12) (T13, R13) (T15, T14) (T3, R3) (T5, R5) (T6, R6) (T7, R7) (T9, R9) Certain pins are input only. Assuming that you want output, the table reduces only six outputs:  (A10, B10) (A11, B11) (A12, B12) (A13, B13) (A2, A3) (A4, B4) (A5, B5) (A6, B6) (A7, B7) (A8, B8) (A9, B9) (C1, C2) (C14, D14) (C16, C15) (C9, D9) (D1, D2) (D11, D12) (E16, E15) (E8, F8) (F1, F2) (F16, F15) (G1, G2) (G16, G15) (J1, J2) (J16, J15) (K1, K2) (K16, K15) (L1, L2) (L16, L15) (M1, M2) (M16, M15) (N1, N2) (N16, N15) (N6, N5) (P1, P2) (P16, R16) (P3, N3) (T10, R10) (T11, R11) (T12, R12) (T13, R13) (T15, T14) (T3, R3) (T5, R5) (T6, R6) (T7, R7) (T9, R9) Which means you can ultimately only run one HDMI cable + 2 speakers off of your DE0-Nano. I’ve highlighted which pins these are here: White borders indicate p, black borders indicate n. The colours correspond to K15, K16, N15, N16, R16, P16, L15, L16 , F15, F16 and G15, G16. Finally, LVDS pins must be at least 4 pins away from any other used outputs. That means you may have trouble using KEY[0] and the (K15, K16) LVDS pins simultaneously, and as well as using (F15, F16) or (G15, G16) pin pairs simultaneously. Sometimes the fitter is able to cram both all pins on the same design, sometimes it can’t. Several of the DRAM pins look close to the LVDS ports, but I haven’t extensively studied which ones. # Re-Assigning pin F16 If you use pin F15/F16, you’ll have to re-assign it off of the programming pin, or else you’ll get the error: Error (169014): Cannot place node ~ALTERA_nCEO~ in location F16 because location already occupied by node HDMI[0](n)  You can fix this by adding the line: set_global_assignment -name CYCLONEII_RESERVE_NCEO_AFTER_CONFIGURATION "USE AS REGULAR IO"  to your tcl or qsf file. # The TCL File Here’s the tcl file for declaring these pins: set_global_assignment -name CYCLONEII_RESERVE_NCEO_AFTER_CONFIGURATION "USE AS REGULAR IO" set_instance_assignment -name IO_STANDARD LVDS -to LVDSO[0] set_location_assignment PIN_G15 -to LVDSO[0] set_location_assignment PIN_G16 -to "LVDSO[0](n)" set_instance_assignment -name IO_STANDARD LVDS -to LVDSO[1] set_location_assignment PIN_N15 -to LVDSO[1] set_location_assignment PIN_N16 -to "LVDSO[1](n)" set_instance_assignment -name IO_STANDARD LVDS -to LVDSO[2] set_location_assignment PIN_R16 -to LVDSO[2] set_location_assignment PIN_P16 -to "LVDSO[2](n)" set_location_assignment PIN_R8 -to clock_50 set_instance_assignment -name IO_STANDARD LVDS -to LVDSO[3] set_location_assignment PIN_L15 -to LVDSO[3] set_location_assignment PIN_L16 -to "LVDSO[3](n)" set_instance_assignment -name IO_STANDARD LVDS -to LVDSO[4] set_location_assignment PIN_K15 -to LVDSO[4] set_location_assignment PIN_K16 -to "LVDSO[4](n)" set_instance_assignment -name IO_STANDARD LVDS -to LVDSO[5] set_location_assignment PIN_F15 -to LVDSO[5] set_location_assignment PIN_F16 -to "LVDSO[5](n)"  04 November 2016 This post describes my software and scripting pipeline for compiling hex images for the teensy and uploading them to teensies. If you’re playing around with arduino or a teensy for the first time, I would not recommend doing this. Instead, follow PJRC’s instructions on using teensyduino. I went with my own pipeline for a few reasons: • the arduino IDE is ugly and does not have many features • I want compile and upload my hex files the same way every time • Since I often have multiple teensies plugged into a computer at a time, I want to be able target specific teensies by serial number • I don’t want to have to memorize different steps based on whether I’m programming in python, arduino or c • I want to be able to do all of the above with one command. # The Components All components in this pipeline have been compiled for both Windows and Linux, so I have been able to follow the same pipeline on both operating systems. My pipeline requires the following programs and binaries to be installed on your system for operation: For arduino code compilation: • Arduino Software IDE, creates the directory structure, and includes the arduino-builder, used for compiling the hex files. • Teensyduino beta, installs the teensy libraries in the arduino folder, as well as the binaries teensy (the little window that pops up when you upload hex files) to Arduino folder/hardware/tools/. If you are using the latest version of Arduino, which I would strongly advise, make sure you download the teensyduino beta instead of the version posted on the main PJRC website. For C/C++ code and MicroPython: • gcc-arm-embedded For MicroPython code compilation: For all code: • TyQt is a GUI that offers a few more debug options than the Arduino IDE. It also allows teensies to be targeted by serial number. pyteensy, my python scripts for compilation/uploading hexs. To install, run: git clone git@github.com:CatherineH/teensy-python-makefile.git cd teensy-python-makefile python setup.py install  Make sure the following directories are added to your PATH variable: •arduino folder$•$arduino folder$/hardware/tools •$tyQt folder$/ty/build/$your OS$on Windows, the arduino folder is usually under C:\Program Files\Arduino. # Using pyteensy There are two ways to use pyteensy. You can either import it as a module, for example, if you want to encorporate the functionality into your own scripts: from pyteensy import list_devices latest_hex = find_hexes() teensies = list_devices() for teensy in teensies: upload_latest(teensy, latest_hex)  or, you can use it from the command line: cd teensy-python-makefile pyteensy -u -d teensyLC -p test_project_arduino  The project test_project_arduino in the teensy-python-makefile repository contains three common pieces of functionality: digital writing, ADC read, and USB and hardware serial. If it compiles and uploads, then most functionality will work. 04 October 2016 No computing cluster? No problem, python’s multiprocessing library makes it easy to melt your CPU by parallelizing numerical integration. Here’s an example. Suppose we want to reproduce this graph: It is plotting the spatial mode overlap between the pump and idler beams of an SPDC process defined by the aggregate focus (xi) and the phase matching between the two beams (phi), which is: If it weren’t for that nasty integration variable in the denominator, we could use SymPy to find the exact analytical solution, then substitute our range of phi and xi’s, easy-peasy. Instead, we have to use numerical integration. Luckily scipy has an already-optimized integration module, so you don’t need to go digging out your undergraduate scientific computing assignments. The code to run our range of values through numerical integration could look like: from numpy import sqrt, exp, arange, pi from scipy.integrate import quad def integrate(args): phi = args[0] xi = args[1] return quad(lambda l: sqrt(xi)*exp(1j*phi*l/2.0)/(1.0-1j*xi*l), -1.0, 1.0) if __name__ == "__main__": num_points = 100.0 phase_mismatches = arange(-7.0, 3.0, 10.0/num_points) aggregate_focus_logs = arange(-1.5, 2.5, 4.0/num_points) parameters = [] for phase_mismatch in phase_mismatches: phi = float(phase_mismatch*pi) for aggregate_focus_log in aggregate_focus_logs: xi = float(10.0**aggregate_focus_log) parameters.append([phi, xi]) results = [] for parameter in parameters: results.append(integrate(parameter))  My laptop has a quad-core AMD A10-7300: not a chip for serious computations, but acceptable for most of what I do. With python 2.7, this code takes an average of 2m21 seconds to complete (on python 3.4 it takes 2m31s). Although not a lot of time, it’s enough so that my focus is broken and my mind will drift to other things, which is bad for flow. Because the integration of each of these points is independent, this task is an excellent candidate for parallelization. Python has several libraries for parallelization, but since I’m running this example on my laptop and not a distributed or cluster computer, using the standard library multiprocessing is the easiest. This requires only a slight modification of the code; instead of using a for loop to iterate over the parameters, we simply need to create a process pool and map the parameters onto the result:  p = Pool(4) results_pooled = p.map(integrate, parameters)  To make sure that the process of parallelization did not change the result somehow, we can check the serialized and parallelized results:  for i in range(len(results)): assert results_pooled[i] == results[i]  For this example, at least, the results are the same. The parallelized computation takes an average of 47.5 seconds on both python 2.7 and python 3.4. Long enough to be frustrating, but not long enough to get distracted. This is what the CPU usage looks like when the calculations are run one after the other: During serial execution, only one CPU is used to 100%; during parallel computation, all 4 CPUs are occupied at 100%. No heat alarms yet :p. # Troubleshooting Once executed in a pool, error messages become less helpful. For example, you may get something like this: Traceback (most recent call last): File "~/multi_processing_example.py", line 30, in <module> results = p.map(integrate, factors) File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map return self.map_async(func, iterable, chunksize).get() File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get raise self._value multiprocessing.pool.MaybeEncodingError: Error sending result: 'error('Supplied function does not return a valid float.',)'. Reason: 'PicklingError("Can't pickle <class 'quadpack.error'>: import of module quadpack failed",)'  It seems like the scipy integration module hit an error, but the worker process was unable to pack it into a pickle to send back to the main process. An easy way to troubleshoot this is by sending only the first parameter to the function outside of the pool. There we can see the error message of: Traceback (most recent call last): File "multi_processing_example.py", line 33, in <module> results.append(integrate(factor)) File "multi_processing_example.py", line 15, in integrate result = quad(lambda l: sqrt(xi)*exp(1j*phi*l/2.0)/(1.0-1j*xi*l), -1.0, 1.0) File "/usr/local/lib/python2.7/dist-packages/scipy/integrate/quadpack.py", line 315, in quad points) File "/usr/local/lib/python2.7/dist-packages/scipy/integrate/quadpack.py", line 380, in _quad return _quadpack._qagse(func,a,b,args,full_output,epsabs,epsrel,limit) quadpack.error: Supplied function does not return a valid float.  Which occured because we gave one parameter as a non-float. The multiprocessing library is capable of more complex parallel computations, but often a simple worker pool is sufficient to get a lot of improvement. 30 September 2016 The title of this post is a bit of a cheat; two of these albums actually came from Google Play. There are very few bands I’m willing to make a Google Play exception for. # Splendor & Misery - Clipping. Clipping’s latest album is an afrofuturist rap concept album about an escaped slave adrift in space. It’s great. The subject matter should make this an accessible album for fans of genres outside of rap where nerdy concept albums are common, such as musicals or prog rock. The instrumentals, lyrics and volume levels are all a bit toned down from Clipping’s previous albums. Although concerned with very different subject matter, I’ve heard that there are references to their previous albums that have likely been documented on rap genius. I haven’t looked through the annotations yet, as finding new references after multiple listens is part of Clipping’s appeal and I don’t want spoilers. # Popestar - Ghost Ghost are still on their post-Meliora tour, though they are already planning their next album and will be in the studio once their tour ends in 2017. In the meantime they released an EP with one new song and four covers, and I was one of the 21,000 people who bought it. The new song is great and seems the suited, out of most of any of the Ghost discography, to Papa Emeritus III’s vocal range. The covers are not great, though I am biased against covers. Ghost puts their own, different and talented version of all three songs, but become stale on second listen. The problem is the composition. A nameless ghoul called The Echo and the Bunnyman album Ocean Drive one of his favourite albums, so the Nocturnal Me cover is unsurprising. It’s clear that The Echo and the Bunnyman, though in a different genre, influenced the richness and dourness of Ghost’s sound. But unlike most Ghost songs, Nocturnal Me has no clear developement and ending. Two of the other covers, I Believe and Bible also suffer from lack of structure. Missionary Man doesn’t suffer from lack of structure, but like all Eurthymics songs, it uses cheap pop song tricks. I’ll only be keeping Square Hammer, but I’m so eager for more Ghost material that I’ll take what I can get. # The Prelude Implicit - Kansas Kansas has a new album 14 years and after arguably the two most important members, Livgren and Walsh, left the band. It’s okay. It’s better than their last album because Billy Greer, whose voice is past its prime, is no longer singing lead vocals. His replacement, Ronnie Platt is a bit nasal and has trouble enunciating. Now that the band’s strongest songwriters have retired, the lyrics are really boring. Visibility Zero is a political protest song, which is a different subject to execute in this era, but they do it adequately. The Unsung Heroes sounds like something Bon Jovi could have released, and Refugee is an overly simplistic consciousness-raising song. I don’t disagree with the message, but I think that a little more art could be employed in the delivery. On other songs seem to be trying to update the band’s sound. Camouflage uses a few baby diatonic rotations that Dream Theater pioneered. Rhythm in the Spirit uses the distorted but still poppy vocals that came out of nu-metal. However, both of these updates seem like too little, too late. If they’re not going to push the envelope, I would have preferred a return to organs and violins. The two covers at the end are the worst things on the album. The Home on the Range cover is weird, and simultaneously too slow and too fast. The instrumental breaks are neat but out of place. The Oh Shenandoah cover is a bit more listenable because there are no lyrics, but it’s still boring - it sounds like a song that might play on an Aaron Sworkin movie. Despite all of its flaws, the solos are as good as ever. If you’re a Kansas fan, it’s worth a listen, but I wouldn’t recommend it as an entry point to their discography. 29 September 2016 I feel the same way about these three bands/acts: I would probably enjoy them more if the lyrics were in another language. # Woods of Ypres Jeff suggested I listen to Woods of Ypres, which at first glance seemed like something would love: Canadian doom metal. Their last album won the 2013 Juno for Metal/Hard album of the year. Unfortunately, I’m not a fan. Their composition isn’t particularly interesting, but their lyrics are awful. If you enjoy self-help books and misogyny, this band might be for you. This band is aptly tagged as “depressive metal”. Every song deals with a mundane personal trajedy or anxiety, but deals with it using the coping mechanisms of over-generalization and platitudes. For example: the song “Move On!” deal with the end of a relationship, but asserts that “The woman will always leave the man. Women move on, men love forever.” Only people with an exteme inability to empathize with women, or believe that women are sub-human, would believe this. As a sensitive woman who can still remember how painful and awkward unrequited love, these lyrics make me want to shut down on the entire discography. There are ways to express your sadness about the end of a relationship without alienating an entire gender. Woods of Ypres should perhaps be commended for targetting the anxieties of 20 and 30-somethings rather than those of teenagers like a lot of rock bands, but the idea is better in theory than in implementation. Many songs contain unsolicited advice, like “Career Suicide (Is Not Real Suicide)”, and although they aren’t wrong, the end product is extremely lame. I looked up the rest of the recent Metal/Hard Junos while writing this - I’m disappointed that the 2014 Juno was given to Protest the Hero’s Volition over GorgutsColoured Sands (Beyond Creation’s Earthborn Evolution wasn’t even nominated), and that this year the Juno went to Kataklysm over Fuck the Facts and KEN Mode. At least it didn’t go to Diemonds. # Joe Bonamassa My grandfather informed me that he reads my blog and then suggested I listen to Joe Bonamassa after watching a PBS special. If you think I’m going to make some comment making fun of PBS for not being hardcore, you’re wrong; PBS music specials are great. Bonamassa is an amazing guitarist who can rip through fast licks effortlessly. He also has similar taste in music to me: his solo debut album had a Jethro Tull cover, and one of his live albums has a cover of Yes’s Heart of the Sunrise. Bonamassa’s discography primarily consists of covers of much older blues rock, a genre I am unfortunately not that familiar with. Listening to Bonamassa’s discography sent me down a blues-rock rabbit hole. After climbing out, I’ve come to the conclusion that the originals were almost always better for two reasons: Bonamassa’s awesome technique often overpowers any other instrumentation, and his voice is a little too corney compared to the original blues singers, and thus some of the emotional core of the music is lost. His original music holds a little more appeal to me, as it has nothing to be compared to. The exception is So, It’s Like That, where he copied the songwriting and singing style popular at the time. If you ever wondered what Avril Lavigne or Hoobastank might have sounded like if they went bluesy, that’s the album for you. The other complaint about Blues-rock was that a lot of the songs deal with male lust or longing, which are difficult to empathize with. It’s not intolerable like Woods of Ypres, who seem to want to take their own sadness out on all women. But outside of that album and a few songs, Bonamassa’s discography is fantastic. I’ll have to check out more of the 90s/00s blues-rock revival. # Kansas Kansas has a new album and a cinematic emo cover of Dust in the Wind was used to advertise the latest World of Warcraft expansion, so I decided to finally go back and listen to all of Kansas’ discography. I don’t have much to say, Kansas is still awesome. The only thing has changed is that I hadn’t listened to their last album, 2000’s Somewhere to Elsewhere, and my interpretation of their lyrics has changed. Their last album wasn’t very good - Billy Greer’s voice has aged too much, and Walsh didn’t contribute to the lyrics. Livgren’s references to Christianity were more noticeable on this listen. And it’s not the absurdist or thoughtful flavour of prog Christianity like The Flower Kings, it’s closer to an American Evangelical version that promises that your life is terrible and you know that they only way to be happy is love Jesus (though thankfully Jesus is never mentioned, so it could be Tom Hiddleston). Other songs have dated lyrics. The Cheyenne Anthem lyric “We will share it with you, no man owns this earth we’re on” advances a harmful stereotype that they Cheyenne Indians didn’t see themselves as owners of Kansas and Nebraska. Andi, a song where an older man assures a young girl that all her anxieties will go away once she becomes a woman with boobs, is patronizing. That being said, Kansas’ discography has far more hits than misses. Given that Kansas’ current lineup does not include Walsh, or Livgren, it’s not really Kansas anymore. At least Greer isn’t doing all lead vocals. 26 September 2016 My mind defaults to one of three questions while idle, like when I’m running or biking to work. These are: • What technology would I be able to recreate in the Roman Republic if I fell through a time vortex right now? • How can the physical laws governing vampires and other paranormal creatures be exploited to build a perpetual motion machine? • What is the best musical act of all time? Though the situations are hypothetical, at least the first two have concrete answers. It’s hard for me to decide on the best music even when limited to my own personal taste. Plus, the music taste that represents how I see myself as a person is different than the amount of enjoyment derived while listening. Thanks to a sloppy python script, I can at least answer what musical acts I seem to enjoy listening to the most. For the past six years I’ve been running a musical battle royale: I’ve limited my playlist to 16 Gb and new songs can only be added when old music is deleted. I determine which songs to delete by playing my entire collection on shuffle. In order to skip to the next song, it must be deleted, and once deleted it can never be re-added. I like this system because it requires me to contemplate why a song is annoying, boring, or bad before being issued to the void. The most obvious measure of how much I like a band is the number of tracks by that act on my playlist. By number of tracks, the ranking is: 1. Joe Bonamassa (86) 2. Spock’s Beard (60) 3. Jethro Tull (59) 4. Steve Vai (54) 5. Ayreon (46) 6. The Flower Kings (46) 7. The Moody Blues (42) 8. Katatonia (39) 9. King Crimson (39) 10. Yes (38) This ranking is surprising considering that I panned Katatonia last month. The high track count is mostly an indication that it hasn’t been in the battle royale long enough to be cut down. After seeing this ranking, I decided that a better metric might be the date created, as older tracks mean that I still enjoy them depite being played many times. Unfortunately, this playlist is stored on a linux system so I only have access to the date the file was last modified, and not the date the file was created. By oldest track modification date, the ranking is: 1. Marduk 2. Sourvein 3. Vektor 4. Tribulation 5. Into Another 6. BADBADNOTGOOD 7. Tengger Cavalry 8. Ketzer 9. Van der Graaf Generator 10. MisþYrming This list also seemed off: I was listening to more paletable prog rock before venturing into Van der Graaf Generator. Then I remembered that a six months ago I spent a lot of time programmatically fixing the broken mp3 tags in the playlist. This metric is thus a proxy measurement for how broken the tags were when they were added to the list, which usually only indicates whether I bought the music off of bandcamp or acquired it through other means. Many of these tracks have modification dates that differ by only a few seconds. I decided to change this ranking to classify all tracks modified on the same day as having the same age. I can then come up with a combined ranking where the age is weighted at 0.5 as the counts. By count ranking + 0.5*modification date ranking, the ranking is: 1. Jethro Tull (3.5) 2. Ayreon (5.5) 3. The Flower Kings (6.5) 4. Dream Theater (13.5) 5. Rush (14.5) 6. Baroness (15.5) 7. Vektor (17.5) 8. Spock’s Beard (20.0) 9. The Alan Parsons Project (21.5) 10. Sasquatch (22.5) This combined metric ranking is the closest to the song that play in my head in the shower, so it feels more accurate than the other two rankings. It’s also extremely embarrassing. As much as I’d like to think of myself as having a modern and diversified pallette, it turns out that I really just like middle-aged white guys who sing about aliens and robots. I enjoy rap music or R&B, but the battle royale is hard for tracks that are distracting or without replay value. The ranking also skews away from genres I love but haven’t been listening to recently, such as classical, folk, jazz and singer/songwriters of the 60s and 70s. It will be interesting to see how it ranking changes over time. 21 September 2016 Mitutoyo dial indicators can measure small displacements down to micron levels. I use them to calibrate motors used in quantum optics experiments, where every micron is several femtoseconds in the life of a photon. Modifying the instructions and source code in this instructable on communicating with the Mitutoyo communication protocol from arduino, I was able to use a teensy to automate this calibration process. My modifications/improvements to the original instructable are: • creating a PCB for the teensy • handling units and sign • adding a SCPI interface • creating a python example script using InstrumentKit The code and Eagle files for this project are available on github. # Electrical Inferface The electrical interface between the teensy and the Mitutoyo cable are relatively unchanged. The teensy footprint was adapted from footprints teensy originally created by PRJC forum user Constatin. The PCB was printed using OSH Park. I use a Teensy LC, but I believe the PCB and source code will work with all Teensy models. The other parts required to complete the board are: • a 10 kOhm R0805 surface mount resistor • a 2N2222 transistor • an 8 pin header (digikey part number ED1543-ND) # Teensy (embedded) Code The main.ino for this project is similar to the code in the instructable, but with a few bug fixes and code organization. #include "scpi_comm.h" #include "struct_definitions.h" mitutoyo_interface _interface = {.acknowledge = 0, .number = 0L, .sign = 0, .decimal = 0, .units = 0}; int req = 5; //mic REQ line goes to pin 5 through q1 (arduino high pulls request line low) int dat = 2; //mic Data line goes to pin 2 int clk = 3; //mic Clock line goes to pin 3 byte mydata[13]; void setup() { Serial.begin(9600); pinMode(req, OUTPUT); pinMode(clk, INPUT_PULLUP); pinMode(dat, INPUT_PULLUP); digitalWrite(req, LOW); // set request at LOW } void loop() { // get data from mic digitalWrite(req, HIGH); // generate set request for(int i = 0; i < 13; i++ ) { int k = 0; for (int j = 0; j < 4; j++) { while( digitalRead(clk) == LOW) { } // hold until clock is high while( digitalRead(clk) == HIGH) { } // hold until clock is low bitWrite(k, j, (digitalRead(dat) & 0x1)); // read data bits, and reverse order } // extract data mydata[i] = k; _interface.sign = mydata[4]; _interface.decimal = mydata[12]; _interface.units = mydata[11]; } // assemble measurement from bytes char buf[7]; for(int lp = 0;lp < 6; lp++ ) { buf[lp]=mydata[lp+5]+'0'; } buf[6] = 0; _interface.number = atol(buf); if (Serial.available() > 0) { comm_protocol(Serial.read(), &_interface); } }  My SCPI interface code, in scip_comm.h is similar to the code for my optical switch. Also like the optical switch code, I store variables generated in the main loop to an interface struct: typedef struct{ bool acknowledge; long number; int sign; int decimal; int units; } mitutoyo_interface;  # Python (Computer) Code The serial communication is handled using an InstrumentKit instrument. The dial indicator will respond with either millimeters or thousandths of an inch depending on an internal setting. Though with the original code, this might have resulted in unit conversion errors. However, by converting the reading string to a quantities Quantity, we can use someone else’s unit conversion code.  @property def reading(self): """ Get the current reading off of the indicator. :type: ~quantities.Quantity :units: millimeters or mils """ query = self.query("READ?").split(" ") response = pq.Quantity(float(query[0]), query[1]) return response  Finally, we can grab readings whenever we want using: if __name__ == "__main__": mi = MitutoyoInterface.open_serial(vid=5824, pid=1155, baud=9600) while True: print(mi.reading)  18 September 2016 This morning I was able to compile and upload micropython and my first main.py to the Teensy 3.2 development board. Once micropython was compiled and uploaded, it wasn’t too much effort to get my teensy to drive the upper matrix of an Adafruit 9x16 charlieplexed LED matrix. This post is intended as a recipe for getting to this point on debian linux machines. # Download the required toolchains In order to compile micropython for the teensy, three tools are needed: the gcc-arm-embedded toolchain, arduino and the teensyduino project. To install the gcc-arm-embedded, add the ppa and install with apt-get: sudo add-apt-repository ppa:team-gcc-arm-embedded/ppa sudo apt-get update sudo apt-get install gcc-arm-embedded  Download arduino from arduino.cc and install using the install.sh script. Download teensyduino from pjrc.com and install by making the TeensyduinoInstall.linux32 binary executable and then running it. You must have arduino previously installed in order for teensyduino to properly install. # Running MicroPython Download git and clone the micropython repository: sudo apt-get install git git clone https://github.com/micropython/micropython.git  Change directory into teensy: cd teensy  Compile the .elf file and .hex file into the build directory: ARDUINO=~/arduino-1.6.11 make  Upload the hex file to your teensy. You will need to run this as sudo unless you installed the teensy udev rules. sudo ARDUINO=~/arduino-1.6.11 make deploy  The teensy port of micropython will automatically set the teensy’s USB protocol to support a virtual serial port. Once the device is reset, it should show up in lsusb as a teensyduino serial port, whereas initially it was listed as Van Ooijen Technische Informatica Teensy. $ lsusb
Bus 003 Device 110: ID 16c0:0483 Van Ooijen Technische Informatica Teensyduino Serial


It should also now appear in /dev as a ttyACM device:

$sudo pip install --upgrade matplotlib  ## Custom Data Labels Seaborn will take the keys from the dataframe as the x and y axes labels, and assign labels only if the subplots are around the left and bottom sides of the grid. Replacing them means dipping down to the axes level: import seaborn as sns; sns.set(style="ticks", color_codes=True) iris = sns.load_dataset("iris") g = sns.pairplot(iris, hue='species') replacements = {'sepal_length': r'$\alpha$', 'sepal_width': 'sepal', 'petal_length': r'$\beta$', 'petal_width': 'petal', 'versicolor': 'bloop'} for i in range(4): for j in range(4): xlabel = g.axes[i][j].get_xlabel() ylabel = g.axes[i][j].get_ylabel() if xlabel in replacements.keys(): g.axes[i][j].set_xlabel(replacements[xlabel]) if ylabel in replacements.keys(): g.axes[i][j].set_ylabel(replacements[ylabel])  Similarly, the hue categories are taken from the values in the dataframe. Most of the template graphs (such as pairplot), add a legend to the top level figure on being called, and this makes it difficult to edit. In addition, the matplotlib Figure object does not provide a get_legend() method. Instead, it is possible to access via get_children(). The legend is the last artist added, so it will be at index -1: for i in range(len(g.fig.get_children()[-1].texts)): label = g.fig.get_children()[-1].texts[i].get_text() if label in replacements.keys(): g.fig.get_children()[-1].texts[i].set_text(replacements[label])  This is the resulting graph: ## Moving the Legend There are plenty of ways to place a legend in a figure when it is created, but moving the legend once it is already in the figure is a different story. The one method I’ve found is to move the legend’s anchor using set_bbox_to_anchor. These numbers seem to be expressed as a function of the size of the total figure, and are expressed as (left, bottom, width, height) . Since the figure expands to fill the required space of all the artists, width and height can be zero. So, the following code will put the legend in the middle in the y direction but over right edge of the figure: g.fig.get_children()[-1].set_bbox_to_anchor((1.1, 0.5, 0, 0))  20 May 2016 This month I bought three fantastic albums an an EP on my favourite DRM-free platform: # Sunburst - Fragments of Creation I am aware of only three things that my husband loves that I think are worse than subterranean trash fires: Slaughterhouse Five, Jupiter Ascending, and Dream Theater’s latest album, The Astonishing. I love Dream Theater, and think that their other two albums since Portnoy’s departure rank just under Images and Words, but listening to The Astonishing gave me DT fatigue. So I was hesitant to listen to Fragments of Creation by Sunburst when I read that it was marketed as an album for DT fans. Fragments of Creation was worth waking up for - it’s like the frenetic playing of Fleshgod Apocalypse with triumphant DT melodies, guitar work and Labrie’s singing style. There are quite a few other progressive and metal influences: the fast synthesizer arpeggios that sound like songs off Asia’s Aura, TesseracT djent, and Rhapsody of Fire choral breaks. There are still some fresh ideas - I really like the xylophones in the background. The singing often verges on Celine Dion-level corniness with all of the tremolos, or descends into Disturbed whining. The lyrics aren’t worth paying attention to: a lot of it is generic nu-metal stuff about pain and abandonment. The key change in Lullaby attempts to keep a song that had already expressed all of its ideas going for another 30 seconds. Finally, the playing isn’t quite tight, but the songs are fun enough that it doesn’t detract from the whole product. The album is not perfect, but it is a much better DT album than DT’s last. Ravenskill who? # Vektor - Terminal Redux Vektor, the band that made me a thrash-metal convert, released Terminal Redux, a gothic space-opera concept album. It is as technical, ferocious and fascinating as their last two albums, but the addition of a cohesive storyline makes this album special. The story is about some unspeakable evil plotting revenge on the civilization that punished it for previous crimes. Like the last Ghost album, several tracks have gospel choir singing in a positive key about the reincarnation of this great evil, which is utterly delightful. Vektor venture into slower territory on Collapse, which don’t quite hold my attention at slower speeds, though the interaction between counter melodies are still interesting. Collapse is the one weak song on a substantial album. # Bent Knee - Say So Say So is a more pop, accessible, and fantastic sequel to Bent Knee’s 2014 album Shiny Eyed Babies. Although Bent Knee are called art-rock, I think of them as more like progressive pop/jazz music. The lead singer Courtney Swain can channel powerful Chanson emotions like Edith Piaf, but with all of the avant-garde, modern flourishes of a singer like Kimbra. The album starts off with a deeply unsettling song reminiscent of Shiny Eyed Babies’ Way Too Long, about foreign liquids covering or flowing uncontrollably out of your body, except this time tar instead of oil and toxic gas, then delves into female-centric themes. There are songs about the pre-pubescent angst of female childhood: I think the song Leak Water is about feeling like one of the peeing dolls that were popular in the 90s and the song Counsellor evokes memories of the emotional distance and bullying of my childhood peers. Other songs cover dissatisfaction with domestic life; lyrics on multiple songs mention dreaming of love and adventure while doing mundane things like driving kids around. Commercial and Good Girl are deeply sympathetic feminist anthems that should show Meghan Trainor and Jennifer Lopez how it’s done. This album should be taken as a blueprint for a modern pop masterpiece - the songs can make you feel as excited as a Carly Rae Jepsen song, but the lyrics are as clever as something written by Sloan. This album combines the melancholy plaints of Melanie Martinez with the experimentation of Zappa, and the end result is simply flawless. If nothing else, the album is worth it to hear a sung five-note arpeggio of the word hyperloop. # Gorguts - Pleiades’ Dust All your favourite Gorguts elements are back in this 30-minute track: discordant guitar chords, growling vocals, and anxious drumming. Gorguts have been pretty consistent with their sound and quality over the past 25 years, if you liked Obscura, you’ll like this. The track runs a gamut of emotions, or at least emotions defined by Gorguts, which makes up for it not being a full album. 17 May 2016 Being a snobby music fan, I love music recommendations even when inevitably don’t love it as much as the recommender. Here are my thoughts on music people have recommended to me recently; please send me more recommendations! # Ensiferum In response to my Bathory review, Alex M. suggested Finnish Viking metal band Ensiferum. Their debut album appeared in 2001, a full 13 years after Bathory’s Blood Fire Death. Their sound doesn’t explore new territory, but it is cleaner and more listenable than Bathory. Though the Ensiferum discography is fairly consistent it quality, there were a few albums I didn’t like. Dragonheads (2006) was too ambitious, and is not quite polished. They used synthesized wind instruments, the singing is out of tune, the songs change tone all the time, the volumes aren’t balanced in a way to make the hard groves work. I thought their latest album, One Man Army (2015), which alternates between some lame country-western style and rage songs, was boring. However, the albums where Ensiferum embraces the camp and fun of pretending to be vikings with electric guitars are really fun. I normally don’t like covers, but their cover of Battery has usurped the original in my brain. It’s amusing to hear such traditional pop melodies mixed with power metal - the melody of Vanderum reminds me of the chorus of Doesn’t Really Matter by Janet Jackson, but so far no one agrees with me: I’m definitely going to see this band live the next time they’re in SoCal. # Loxley Thomas E. recommended a one-EP heavy metal band out of Ottawa called Loxley. Thomas bought their 1991 EP “Better Late than Never” off one of his coworkers, who was related to someone in the band. It pains me to say, because I want to support Canadian music, it’s not good. It’s not inventive for the time, and it sounds like it was recorded in a cathedral. If any members of Loxley are reading this, though I didn’t like this EP, Thomas clearly loved it and listened to the EP many times over the years, enough to recommend it in 2016. # Steve Vai I probably should have known about guitar prodigy Steve Vai prior to Ben C.’s recommendation, but 80s and 90s solo acts are a big gap in my musical awareness. Although I was initially dismissive of Vai, I’m glad I revisited his discography. My favourite album in Vai’s discography is Fire Garden (1996), though its greatness is due to pulling in the best ideas of other contemporary bands. Aching Hunger uses the synthetic music sampling Yes introduced in the 80s. Brother is a dramatic ballad that could have come from Toto. The oddest cut is an accidental cover of One Night in Bangkok from the ABBA/Tim Rice musical Chess. According to the internet, David Lee Roth asked Vai to transcribe the guitar solo intro, which he didn’t end up using until a decade later when he dug it out, forgot that he hadn’t composed the solo, and turned it into a full song. The resulting song is honestly better than the original because it isn’t immediately followed by campy sing-talking about liking chess more than sex. Although Vai’s instrumentals are fantastic, there’s a lot of garbage in his discography. Vai’s managers need to lock the sound samples out of his mixing computer. Vai is a fan of letting animals (like on Bad Horsie), children (like on Ya Yo Gek), and people with bad vocal fry (like on Kill the Guy) onto his tracks. On his interludes, he either regresses to the maturity level of a ten year old, or espouses an un-metal philosophy of positive thinking. He preaches being happy and nice all the time, then makes fun of Britney Spears while playing live. I guess negativity is only bad when it’s not directed at you? His singing is passable, but at its worst is like the singing of the B52’s Fred Schneider without any of the charm. Steve Vai is one of these talented guys with a career since the early 80s who appears in all sorts of interesting places. He played with Zappa, collaborated with Ozzy, and has appeared most recently on the new M83 album Junk. Listening to his discography is like a talented, curated tour of rock guitar of the 80s and 90s, and I hope he continues to pop up in unusual places well into the future. 29 April 2016 Here’s the stuff I bought on Bandcamp for the month of April: # Tengger Cavalry - Mountain Side EP Tengger Cavalry is a New York-based metal band that incorporates traditional mongolian instruments and throat singing into some fast metal. I first heard of them when their 2014 album Ancient Call was released, instantly fell in love with the band, and forced it on anyone in my social circle with even a passing interest in metal. Although I skipped their 2015 album Blood Sacrifice Shaman, their new Mountain Side EP, and it fit within my bandcamp budget for April. For 9 USD you get 11 tracks, but only two are full new songs. Both new songs focus more on the folk side of folk-metal Tengger Cavalry’s previous albums, and the recording and production quality is better. However, the songs sound a little too similar, and don’t have a good ending: Mountain Side ends abruptly and Krutaya Gora fades out. The other songs in the EP are not even worth including: there’s a club mix of Mountain Side that isn’t fast enough to dance to, an instrumental version which adds nothing because the throat singing on Tengger Cavalry tracks is practically a sixth instrument, a live version with a distracting yell of “everybody fucking move”, and an acoustic version that turns what was metal into country-western. The highlights on this album are the new version of War Horse, which sounds to me like a completely different (but better!) song, and the bonus jam tracks. The Mountain Side EP is not a particularly good value for money, but I have gotten so much enjoyment out of Ancient Call and I like the direction Tengger Cavalry seems to be heading in so I don’t mind contributing towards their next release. # Sourvein - Aquatic Occult Sourvein is a sludge metal band (which as far as I can tell is slower, more distorted stoner metal) from North Carolina. I loved their last album Black Fangs. Aquatic Occult is 14 nautical-themed track. The singing style on this album is different than Sourvein’s older albums, it seems more anemic and doesn’t quite match the tone or beat of the instrumentals. However, there are few tracks with harsher vocals which are quite good. Ocypuss and especially Aquanaut are absolutely hypnotic. Urchins contains some awesome atmospheric guitar melodies like those that Baroness has become famous for, but the track is far too short. Aquatic Occult has a few highlights, but Black Fangs was better. # Fall of Every Sparrow - Ophir EP & Beautiful Fiction Single Last month the music video for Beautiful Fiction by a band called FOES (Fall of Every Sparrow) caught my eye, so this month I bought the single and FOES other bandcamp release, an EP released in 2014 called Ophir. This EP is an unexpected gem. It reminds me of post-Porcupine Tree Steve Wilson, though as a Canadian that might be influenced by the accent. Like post-Porcupine Tree Steve Wilson, this music is introspective and immediately emotionally relatable, which is uncommon from an alternative progressive rock band. The tracks sound a bit sonically full, verging on shoe-gazy. The unclean vocal bits effectively emotionally compliment the clean bits and are used sparingly. The melodies are accessible, but there’s enough complexity in the counter melodies and rhythm to keep the songs interesting. However, this emotional accessibility occasionally goes overboard. The opening lyrics on The First Rook To… are trite. It’s a shame, because the rest of the song’s lyrics are bit more subtle and poetic. I think the song would have been better without lyrics in the first section, because the rest of the song is fantastic. I especially liked the triplets in the piano counter melody. Their latest output Beautiful Fiction is also great, it sounds like the band are experimenting a bit with syncopation, but kept a lot of the elements that I liked from Ophir. I’m looking forward to whatever comes next from FOES. On another note, it drives me crazy when bands change the punctuation in their titles. I take great, possibly neurotic, care in keeping my music organized and now the FOES tracks don’t stack up nicely in VLC. 27 April 2016 Aside from fingerprints, there is nothing more annoying, or more damaging to optics equipment than unplugging and plugging in fiber optics. Every time a fiber tip or port is exposed to air, there’s a chance of getting gross human skin cells on places where tightly focused high-power light might incinerate them, yet, so many experiments require routing light. Wouldn’t it be nicer if a robot did the switching for you at 2 a.m. while your gross dust and sweat producing body is sleeping? In experiments, I accomplish this using a teensy LC and a DiCon optical switch. # Electronics Side The optical switch I use is a DiCon 2×2 prism switch, which is essentially a little piece of glass on a motor. This switch will operate in several modes, but in every mode the first two (blue and purple) pins are used to set the position and the second two (red and green) pins are used to read the position of the switch. I chose to operate the switch in Non-latching 2-pin Control mode, because it was the least complicated. I chose to put the two pins that should never change in control mode (purple should always be ground, red should always be +5V) and the two pins with changing values (blue and green) on the other. This led to a lot of initial confusion, but I believe it was a good decision to because it allows me to keep the two pins that need to be hooked up to some external logic together. The threshold voltage for switching is above 3.3V, which is the maximum that the Teensy’s output pins can supply. Thus, I chose to use a solid state relay to bump the signal up to 5V. I use Sharp Microelectronics R22MA1 because they’re super cheap (25.65$ for 50) and I have a tendency to accidentally blow electronics up.

My prototype board looks like this:

Imagine the Sharp SSR on the breadboard instead of the Omron SSR. You’ll notice that I have left the green pin unconnected. The switch has always activated when 5V, thus the position information isn’t useful right now.

# Arduino Code

This teensy is also used to control several other pieces of experimental hardware, so I’ve excerpted it here.

I want to be able to read the current position state and write a new position state using a SCPI-compliant serial connection.

My .ino code looks like this:

#include "scpi_comm.h"
#include "optical_switch.h"
int switch_led_pin = 13;
int out_pin = 9;
settings _settings = {.switch_setting = 1, .motor = _state};

void setup()
{
Serial.begin(9600);
pinMode(switch_led_pin, OUTPUT);
pinMode(out_pin, OUTPUT);
}

void loop()
{
update_optical_switch(_settings.switch_setting, out_pin, switch_led_pin);
if (Serial.available() > 0) {
}
}


My scpi_comm.h contains an internal state machine which collects characters into a string until the termination character is received, then attempts to parse the string:

typedef struct {
int switch_setting;
} settings;

char terminationByte = '\r';
int incomingByte = 0;
int string_pos = 0;
int current_state = 1;
char line[15];
char serialsecond[5];
char input_str[8];
char output_str[8];

void comm_protocol(byte incomingByte, settings *settings){
line[string_pos] = incomingByte;
string_pos += 1;
if(incomingByte == terminationByte)
{
if(strncmp(line, ":OUTP", 5)==0)
{
char * loc = strchr(line, ':');
loc = strchr(loc+1, ' ');
memcpy(serialsecond, loc+1, 3);
if(strncmp(serialsecond, "1", 1) == 0)
{
settings->switch_setting = 1;
}
else
{
settings->switch_setting = 0;
}
sprintf(output_str, "Set Switch to %d%c", settings->switch_setting, terminationByte);
Serial.write(output_str);
}
else if(strncmp(line, "OUTP", 4)==0 && strpbrk(line, "?") != 0)
{
sprintf(output_str, "%d%c",  settings->switch_setting, terminationByte);
Serial.write(output_str);
}
else{
sprintf(output_str, "Unknown command%c", terminationByte);
Serial.write(output_str);
}
// reset the string
string_pos = 0;
line[0] = '\0';
}
}


Lastly, my optical_switch.h code simply reads the settings and makes the Teensy’s indicator LED also go high when the switch position is high:

void update_optical_switch(int optical_state, int switch_pin, int led_pin)
{
if(optical_state==1)
{
digitalWrite(switch_pin, HIGH);
digitalWrite(led_pin, HIGH);
}
else
{
digitalWrite(switch_pin, LOW);
digitalWrite(led_pin, LOW);
}
}


# Python (Computer) Side

I used the fantastic InstrumentKit to incorporate the control into my experimental software:

#!/usr/bin/env python
from instruments.abstract_instruments import Instrument
from time import sleep
from sys import platform

class Switch(Instrument):
"""
An interface to the teensy-controlled optical switch
"""
def __init__(self, filelike, increment=200):
super(Switch, self).__init__(filelike)
self.terminator = "\r"
self.increment = increment

@property
def setting(self):
"""
Get the current output setting
:return: int, representing the currently active channel
"""
response = self.query("OUTP?")
return int(response)

@setting.setter
def setting(self, new_val):
"""
Set the current output setting
:param new_val: the output channel number, either 0 or 1
:return:
"""
if new_val == 0:
response = self.query(":OUTP 0")
elif new_val == 1:
response = self.query(":OUTP 1")

def main():
# Runs the switch program on its own, as a test
if platform == "linux" or platform == "linux2":
port = "/dev/ttyACM0"
else:
port = "COM11"
switch = Switch.open_serial(port, 9600, timeout=1)
print("switch tests")

iteration = 0
for i in range(0, 50):
print("Iteration: "+str(iteration))
iteration += 1
switch.setting = 0
sleep(2)
switch.setting = 1
sleep(2)

if __name__ == "__main__":
main()


While the coding side of this application may seem a bit over-engineered, it is because I want to apply the principle of least astonishment to everything I do; by labeling everything I don’t have to guess at some convention I decided on several months ago. This switch is a simple piece of a much more complex testing organism, and I want it to be the least complicated organ.

25 April 2016

Six months ago I participated in a hackathon for the City of Waterloo. I built a WebGL application that built a little model of Waterloo that you could tilt and zoom in a browser. I used the building footprint data to find the base of the models, then used the latest aerial laser scan data minus the elevation data to estimate the height of the buildings. I coloured the buildings based on what their reported purpose and size were - one color for commercial buildings, one for municipal buildings, and one for residential buildings, then within that sub-shades for the types of buildings - apartment buildings, duplexes, single family dwellings.

My desire to build this model came from my own curiosity - between the time that I moved to Waterloo in 2010 and left in 2015, an entire neighborhood transformed from 1950’s style bungalows to high-rise student housing, and I wanted to be able to see those buildings spring out of the ground. Also, I wanted an excuse to learn WebGL. The end result was a stylized version of google maps:

I was proud of what I’d put together in 48 hours, but it didn’t win anything. An app that for tracking trees won. It wasn’t technically impressive - Apple and Android both provide sample apps that allow you to add data to maps, and the tree data was provided by the city. City maintenance apps like this have been implemented by cities larger than Waterloo since 2010. But, the guys who won were funny teenagers, and it was impressive for their level of experience.

After that hackathon I came to a fairly obvious conclusion: hackathons are about showmanship rather than technical ability, but I would rather impress myself than impress a judge. I’ve won hackathons on the basis of good presentations, but they’re not the things I brag about later.

Last weekend was SpaceApps 2016 - a yearly coding challenge organized by NASA. Communities around the world organize events for SpaceApps, and in previous years I’d attended the ones in Kitchener-Waterloo and Toronto. There is no event - at least open to the public - in San Diego, so I signed up for the “everywhere” event. Knowing that there was no competition with a pitch at the end, I decided to go for the most technically challenging project I could find: procedurally generating game environments using web map tile service files of Mars.

The most obvious way to solve the challenge - even cited in the challenge description, was to build something in Minecraft, but I went with the Unreal Engine instead. My reasons were two-fold: modding Minecraft is so easy that it is taught to elementary-aged kids, and secondly: I don’t like Minecraft. I have plenty of creative outlets in my everyday life, so when I sit down to play a video game I want to indulge my destructive impulses. I’d prefer to play first person shooters, hence my interest in building something in the Unreal Engine.

# Getting the data

NASA is gradually improving their data services. My first encounter with them was in high school, where for a science fair project I downloaded their database of asteroid impact crafters to correlate the size and age using deeply flawed statistics to validate a hypothesis about Jupiter being the solar system’s asteroid dust-buster. Back then, I had to use ftp and some text macros to extract the desired data. It was painful.

These days they’re moving to web-based APIs, but it’s still not as good as it could be. They claim that their datasets are WTMS-compliant, but I could not get a python WTMS library working with any of the datasets I wanted. I had two possible options - use the HiRISE data, which involves downloading large binary files via FTP, and then writing my own scripts to parse the binary data to a format I could use just as I had in high school, or going through the NASA API to pull down much lower-resolution data sets and pulling them down one image at a time. This being a hackathon, I went with the second option.

I used the Mars Orbital Laser Altimeter to get grayscale png images for elevation, then pulled down the images from the Viking missions for colors and texture information. The highest resolution for both datasets were different, so I had to stitch together images, but thanks to the urllib and pillow python libraries, this was trivially easy.

Unreal Editor can turn grayscale images into level landscapes, and then generate the textures from images. Within four hours of starting my git repo, I had this:

Great! Challenge solved, time to hand in my solution and spend the rest of the weekend playing Borderlands. I just needed to programmatically spawn the process to generate landscapes and since Unreal Engine and Editor is open-source, I thought it couldn’t be that hard.

Spoiler Alert: I was super wrong.

# Failure 1: Automating Landscape Generation

The problem is that landscapes are built into the level on build, and are impossible to change at run time. I’m trying to think of any FPS that have terrain that changes while you’re playing through a level or environment, and I can’t really think of any. Unlike top-down RPGs like Diablo, FPS’s rely on a lot of physics and not falling through things that are much harder to guarantee with a procedurally generated world. Also, the load times on new levels are much faster if you have a pre-baked, compressed description of the landscape.

At this point I considered giving up, but I kept going because the solution was so tantalizingly close: I could extract the compressed files that contained the landscape information, and somewhere in the source code for the Unreal Editor was the functionality to generate these files.

After three hours of poking through the Unreal Editor in Visual Studio with IntelliSense (I don’t recommend this, Visual Studio crashed five times during this time), I identified the sections that did the importing and then writing to a file. I attempted to pull them out and build a standalone C++ application that could convert grayscale images to Unreal Editor map files.

However, this code wasn’t intended to operate on its own. As time went on, I was pulling in more and more of the unreal editor - first the menus for generating landscapes, then the new level editor, then the editor gui itself. Basically, I was re-implementing the Unreal Editor with no end in sight.

At this point, 13 hours in, I considered giving up again. I was tired of Visual Studio crashing, and feeling cheated out of what I thought was a simple operation. But John sent me a tutorial on procedurally generating meshes at runtime and I had a crazy idea - why don’t I just delete the landscape and use a spawned mesh object instead?

# Success 1: Generating Meshes

For reasons my sleep-deprived post-hackathon brain can’t remember, the mesh tutorial code didn’t work with Unreal Engine 4.11, but did with 4.7. So, after spending an hour removing 4.11/Visual Studio 2015 and installing 4.7/Visual Studio 2013, I was back in business.

I intended that my Unreal Engine mesh generator code would check the tile cache for the desired section of Mars to render, and if it didn’t exist, execute the python script for pulling down the missing tiles. I encountered a problem implementing this: the windows C SDK does not play nicely with the Unreal Engine API. This is for good reason - Unreal Engine wants to be cross-platform, so they have re-implemented all the standard functionality of C independent of each operating system. If you attempt to include both the Unreal Engine and the Windows SDK in the same application, Visual Studio will bug out because it’s not sure which version of int you want to use.

I scrapped that idea. Luckily, reading and writing files isn’t the only way to get data into the Unreal Engine at runtime because networking is required for multiplayer games. After another four hours, I had a python server communicating with my Unreal Engine application. My first ground meshes looked like this:

It’s off the ground because I hadn’t sorted out the math yet, and it suddenly turns into a giant vertical wall because it turns out a TCP packet is not large enough to store the raw data required to render a 400 km-across section of Mars. So, I wrote the first compression method I could think of, and was able to serve mars up 3 Kbytes at a time. The result looked like this:

That red planet sure is… red.

# Failure 2: Colouring the ground

Like landscape files, textures are compressed and baked into the levels prior to runtime. No problem, I’d solve that issue like I had with the meshes: change the color of the triangles in the mesh to match the corresponding pixel in the images.

I had all of this implemented two hours before the deadline, but then Unreal started crashing. I was hitting a breakpoint in the memory allocation functionality of Unreal Engine. In addition, color images aren’t as smooth as surface data, so my poorly implemented compression method wasn’t cutting it anymore. I would guess that these two issues are related. At this point, frustrated and exhausted, I decided to pack up and submit what I had.

# Conclusion

So that was it. In the end, I tracked 29 hours working on SpaceApps this year, compared to 15 hours in 2015, and this year was way more stick-your-head-through-your-keyboard inducing than last year.

For reference, this is what John and I together were able to create at the end of SpaceApps 2015:

That project was much less frustrating than this one because it involved machine-learning analysis of experimental data in python, which is pretty much what John and I do for work. All of the code was built on existing tools used by scientists, who, despite what people may think, maintain their code better than video game developers. Game developers are under constant pressure to release the next big game and thus don’t have much time or incentive to contribute back to the community, whereas (good) scientists’ reputation depends on other people being able to use the frameworks they create.

Throughout this challenge, I was often attempting to use things in ways they weren’t meant to be used: using meshes as landscapes the player could walk on, using the networking protocol to serve massive texture data instead of the locations of other players.

Since my project crashed with two hours left on the clock, I wasn’t able to make a video of what it looks like to walk around in my generated Mars. This means the project likely won’t win anything. That’s fine: like with the Waterloo hackathon, impressing myself was more important than impressing a judge, and in that goal I succeeded. Not because of what I learned: though I learned a lot about how the unity engine and editor work, the real takeaway was that I really don’t want to be a AAA game developer if these are the tools they have to use. I’m impressed with myself because several times this weekend in the face of a problem - like automating the landscape generation - I was miserable and had decided to quit. But every time I did, I would go nap on the couch for a bit, wake up, and go right back to banging my head through the keyboard.

19 April 2016

In high school I was the one girl in a group of seven students who would hang out at the town ‘science center’ after school twice a week. ‘Science Center’ is in quotes because it was the disorganized office of a retired experimental physicist with some no-budget-physics demonstrations. I often left the ‘science center’ more confused than educated, but I loved physics so I stubbornly kept attending.

The primary factor in my confusion was that the experiments were not well explained, but there were other contributing factors. Maybe it was the learning style: as a teenager I would read undergrad-level physics textbooks, daydream about how to turn what I’d learned into an episode of Star Trek, and then struggle with the end chapter problems, and this worked better than a lecture and a demonstration. Maybe it was the non-physics content: more time was spent arguing over politics and religion than learning about physics, and I, a liberal atheist in a town referred to as the ‘little vatican’ and in the riding of Peter MacKay, was always wrong. Or maybe it was the weird analogies: electrons were boys, protons were girls, because boys have no mass and can’t stand to be in the same state together? No homo.

The men running the science center had the best of intentions, but they were quirky. The retired physicist was devoted to an obscure interpretation of quantum mechanics that had its last death throes in the early nineties. Explanations subtly included this interpretation, which didn’t mesh with the things I was reading. There were religious (Catholic) and cultish undertones: he insinuated that if I studied physics I would realize the importance of all life and never get an abortion. Studying physics at school was ‘mickey mouse’ compared to the education I was promised at the science center.

And they were unnecessarily critical. One afternoon I gave a summary of a textbook on cosmology I’d recently read. I was so excited about how the heavier elements fused together in the sun that my mouth went faster than my brain and I repeatedly said carbon monoxide instead of carbon dioxide. One of the men running the science center picked up on this, and interrupted to correct me using the socratic method. The socratic method is a fantastic way to introduce new students to knowledge just outside their understanding, but if you do this to a 16 year old girl who has 98-99% grades in her high school chemistry classes and surely knows the difference between carbon dioxide and monoxide, it is extremely condescending.

My reaction was not, Oh right, carbon dioxide, whatever but rather to think he thinks I’m dumb and it’s true what am I doing here and then hide under the table to cry. This memory is not a metaphor for how I felt at the time: I actually hid under the table and cried.

The boys in the group tried to cheer me up by saying things like “more men are geniuses but women are smarter on average”, and “you may not be good at math but you’re really good at explaining things”, the implication being: you’re not special like us but you’re still smarter than most people we know. It didn’t make me feel any much better.

I wish I could say that these experiences made me stronger, but there’s no evidence they did. I had many more hiding-under-the-table incidents in undergrad, through graduate school, and in my post-school career, and will probably continue reacting to things this way because I am a coward who would rather read physics papers alone and not have to prove I’m smart to anyone.

Over a decade on, only one of those seven students ended up in a career in physics.

It was me.

15 April 2016

My life revolves around three boards: the digi rabbit, an altera FPGA, and a teensy (an arduino-compatible microcontroller).

The provided IDE’s for each of these boards suck, but teensy’s sucks the least. They are ugly and they get in the way of my productivity. Since the code to interface with the boards written in python I’ve been migrating my digi rabbit and altera development to PyCharm. This post documents the errors encountered moving from the arduino IDE to the arduino-builder.

# index out of range on Board Resolver

A merged pull request on arduino-builder should prevent this error from occuring in future.

The arduino-builder requires the full board name to be specified as something like package_name:platform_name:board_name . The colons are essential and the arduino-builder will not check to make sure the full board name follows this pattern. If you don’t specify the board in that form, you will get the error:

\$ arduino-builder -fqbn teensyLC -hardware ~/arduino-1.6.7/hardware/ -tools ~/arduino-1.6.7/tools experiment_control.ino
panic: runtime error: index out of range

goroutine 1 [running]:
arduino.cc/builder.(*TargetBoardResolver).Run(0x6c3050, 0xc20803c4e0, 0x0, 0x0)
~/arduino-builder/src/arduino.cc/builder/target_board_resolver.go:46 +0xf44
arduino.cc/builder.(*ContainerSetupHardwareToolsLibsSketchAndProps).Run(0x6c3050, 0xc20803c4e0, 0x0, 0x0)
~/arduino-builder/src/arduino.cc/builder/container_setup.go:59 +0x6b8
arduino.cc/builder.runCommands(0xc20803c4e0, 0xc20802db90, 0x1d, 0x1d, 0x418001, 0x0, 0x0)
~/arduino-builder/src/arduino.cc/builder/builder.go:187 +0x139
arduino.cc/builder.(*Builder).Run(0xc20802dd88, 0xc20803c4e0, 0x0, 0x0)
~/arduino-builder/src/arduino.cc/builder/builder.go:118 +0xef7
arduino.cc/builder.RunBuilder(0xc20803c4e0, 0x0, 0x0)
~/arduino-builder/src/arduino.cc/builder/builder.go:218 +0x49
main.main()
~/arduino-builder/main.go:333 +0x1eb2



## Resolution

Change the -fqbn parameter to the package_name:platform_name:board_name format. You can guess what this needs to be by looking at the arduino folder structure. For example, my folder structure has:

arduino-1.6.7
└── hardware
├── arduino
└── teensy
└── avr
└── boards.txt



The boards.txt specifies the available boards for that platform. Opening up the file, I can see teensy30, teensy31, and so on. If I change the board name to teensy:avr:teensyLC, the error no longer occurs. Note that this string is case sensitive!

# Not finding the arm compiler

You may get the error:

fork/exec /../arm/bin/arm-none-eabi-g++: no such file or directory


## Resolution

Set the -tools argument to:

 -tools ~/arduino-1.6.7/hardware/tools/


# Missing build variables

Because we’re no longer in the arduino IDE, a bunch of menu-specified variables won’t be set. This will result in errors like:

<command-line>:0:1: error: macro names must be identifiers
In file included from ~/arduino-1.6.7/hardware/teensy/avr/cores/teensy3/core_pins.h:34:0,
from ~/arduino-1.6.7/hardware/teensy/avr/cores/teensy3/wiring.h:33,
from ~/arduino-1.6.7/hardware/teensy/avr/cores/teensy3/WProgram.h:15,
from ~/arduino-1.6.7/hardware/teensy/avr/cores/teensy3/Arduino.h:1,
from /tmp/arduino-sketch-55293F9A6EDF8EF849C232A18F1833A8/sketch/experiment_control.ino.cpp:1:
~/arduino-1.6.7/hardware/teensy/avr/cores/teensy3/kinetis.h:568:12: error: operator '==' has no left operand
#if (F_CPU == 180000000)
^


or, alternatively, you may see something like these errors:

arm-none-eabi-g++: error: {build.flags.optimize}: No such file or directory


## Resolution

This is a hack (build systems people avert your eyes), but I’ve added:

teensyLC.build.fcpu=48000000
teensyLC.build.flags.optimize=-Os
teensyLC.build.flags.ldspecs=--specs=nano.specs
teensyLC.build.keylayout=US_ENGLISH
teensyLC.build.usbtype=USB_SERIAL


to my boards.txt file.

# Ctags file does not exist

The following error may occur when compiling for arduino-branded (i.e., uno, mega, etc) boards:

exec: "{runtime.tools.ctags.path}/ctags": file does not exist


## Resolution

When executing arduino-builder, add the argument:

-tools ~/arduino-1.6.7/tools-builder

13 April 2016

I’m increasing the code coverage on my pyglet_helper project prior to adding new functionality. As of right now it is:

If this is green, I have succeeded in my task. Go me!

Before I got this spiffy number, I had to tackle an issue: pyglet_helper project is built on top of OpenGL, but OpenGL needs a display to draw to. The continuous integration system I am using (Travis) does not have a display.

After embarking on a fool’s errand to get Xdummy working in a docker container, my friend Steven pointed to an easier solution: simply create a fakeGL module and then run the tests using that instead of OpenGL. This is not an ideal solution, as my unit tests will only check to make sure that the math is correct, and not that things are being drawn to the screen without glitching, but at the moment I’m okay with that. I’m not trying to test the functionality of OpenGL; I want to test that my math and the inheritance of the objects in pyglet_helper works out. My own math mistakes, and not OpenGL, are responsible for 99% of the weird visual glitches in pyglet_helper.

This post details how to replace an entire module in python unit tests, since I didn’t find it in my initial reading of the mock documentation.

As an example, suppose we have some math to be tested on a Windows AMD machine1. Thus, we would like to mock out numpy.

The function to be tested is in the file one_deep.py:

import numpy

def sum_array(lower, upper):
return sum(numpy.arange(lower, upper))


This module uses the arange function in numpy, so the file fake_numpy.py contains the code:

def arange(lower, upper):
return range(lower, upper)


Essentially, the range is now a list instead of a numpy array.

The unit test, which replaces numpy with fake_numpy is:

from mock import patch
import fake_numpy

@patch('one_deep.numpy', new=fake_numpy)
def test_sum_to_hundred():
from one_deep import sum_array
result = sum_array(4, 16)
assert result == 114


Now, suppose we need to go deeper. A second function is in the file two_deep.py:

from one_deep import sum_array
import numpy

def sum_array_again(lower, upper):
return sum(numpy.arange(lower, sum_array(lower, upper)))


In our unit tests, if only two_deep is patched, when sum_array is called, it will still use numpy.arange instead of fake_numpy.arange. This can produce some interesting errors if numpy is expecting to operate on numpy types.

Thus, the module must be patched all the way down:

from mock import patch
import fake_numpy

@patch('one_deep.numpy', new=fake_numpy)
@patch('two_deep.numpy', new=fake_numpy)
def test_sum_to_hundred():
from two_deep import sum_array_again
result = sum_array_again(4, 16)
assert result == 6435


Unfortunately, I haven’t figured out a good way of making sure that numpy gets patched all in every place where it is invoked yet, leading to a lot of failed Travis builds as I encover another layer of a pyglet_helper object’s dependencies which rely on OpenGL.

1. Numpy does not support Windows running on AMD chips, as I recently learned.

11 April 2016

Six months ago, one of my co-workers (Matthew from Clearpath) enlightened me to the existence of Viking metal. Since then I’d watched the occasional viking metal youtube video for the lols, but was not further tempted. However, a few weeks ago I went to an Abbath concert and the Bathory logo had almost as much real estate on patches and T-shirts as Immortal, so I caved and listened to their entire discography. These are my thoughts on Bathory.

Bathory1’s discography can be split into four categories: the extreme metal 80s, proto-Viking metal early 90s, generic thrash metal in the mid 90s, and Quorthon’s (the lead singer) one man viking concept albums of the early 00s. Of these periods, the extreme metal and the generic thrash metal albums are totally skipable. The thrash metal albums were dated when they came out; Pantera had already done it better. The lyrics from one song in this period merely list political or religious ideologies ending with the y-sound.

I don’t like their extreme metal phase because of the lo-fi recording and the drumming that is played too fast to land the hits on target. I’m mildly anxious about calling their first 4 albums skipable, because the technical things I don’t like are probably part of the attraction for fans of 80’s death metal.

The proto-Viking metal albums are what the band are most known for, and they are worth a listen. The epic lyrics, the folk-style singing, and the acoustic guitars all make for a (mostly) pleasant listening experience. However, the albums are flawed. Quorthon was not a strong singer and often ventures out of his range. The ballads are a little boring, and the songs are often too verbose to let the melodies breath. The song Hammerheart from Twilight of the Gods is a particularly egregious example. It’s a ‘‘cover’’ of Holst’s Jupiter with mismatched lyrics sung out of key. I don’t think I’ll be going back to these albums.

In the early 00s, Quorthon produced three albums alone under the Bathory name. The first, Destroyer of Worlds, is an interesting mix of contemporary influences. Some songs have guitar licks reminiscent of Frusciante, another song sounds a lot like stoner metal. There’s a male rage-type thrash metal song that would have fit right in on Reinventing the Steel, and Quorthon’s rough vocals lend it an authenticity that make it a lot more tolerable than Pantera. Although I like this album, there are a few obvious flaws, the largest being that Quorthon needed to step away from the sound effect machine2. I wasn’t expecting a hockey arena organ in metal.

Quorthon’s next two albums, Nordland I and II, revisit viking metal and are better this time around: the lyrics mesh with the rhythm, the melodies have space to shine, and the ballads don’t get boring. Quorthon even makes synthetic horns pleasant, which I thought was impossible. My favourite Bathory song, Vinterbolt, is off Nordland I:

In my opinion, Bathory, along with Gentle Giant and Van der Graaf Generator, are examples of bands who didn’t quite have the technical chops or creativity to perfect their pioneering ideas. Viking metal was a novel style in the early 90s, but unfortunately a twenty year back catalog clouded my first listen of Hammerheart.

Of the four styles in the Bathory discography, the Quorthon solo-albums at the end of his career hold up the best against the test of time. He was experimenting with the latest sounds at the time across multiple genres, and seemed to be trying to fix issues from record to record. Quorthon planned another two albums to his Nordland series but died young before they were completed, and that is a shame. He was just getting good.

1. Countess Bathory is to Metal and Gothic Fantasy RPGs as Ada Lovelace is to computing: they are interesting women who are not commonly known to the general public, but their story has been retold ad nauseum. Bathory’s song about Countess Bathory is one of the better exemplars, but surely the genres can find other lady serial killers to idolize.

2. Panzer Division Marduk used the same World War 2 bombing sound two years earlier; Quorthon should have known better.

31 March 2016

2000+ line main functions are no fun, but Dynamic C does not make modularization easy. In this post, I’ll document all of the details of using library functions and the problems I’ve encountered using them.

The Dynamic C compiler will look for libraries in the directory structure detailed in the file C:\DCRABBIT_XX\LIB.DIR. After a default installation, this file will point towards C:\DCRABBIT_XX\LIB and certain SAMPLES directories. Since C:\DCRABBIT_XX is read-only, I prefer to create a link between the location of my source and the C:\DCRABBIT_XX\LIB directory, rather than either copying the files to this directory or editing the LIB.DIR. On Windows, this can be done using:

mklink C:\DCRabbit_9.62\Lib\starship.lib
C:\Users\catherine\Rabbit3400\lib\starship.lib


This needs to be done as administrator, which can either be done by running the command prompt as administrator, or by using runas:

runas /noprofile /user:pink-beast-windows\catherine "mklink C:\DCRabbit_9
.62\Lib\starship.lib C:\Users\catherine\Rabbit3400\lib\starship.lib"


However, this results in the error:

The system cannot find the file specified.


and my windows-file-system-fu is not strong enough to figure out what I’m doing wrong. Maybe when ubuntu comes to windows 10 I’ll be able to ignore my ignorance.

# A Sample Library

The following is an example project in Dynamic C with two libraries. The first library is called starship.lib and contains the source:

/*** BeginHeader StarshipCrew */
struct StarshipCrew{
char captain[10];
char first_officer[10];
char chief_engineer[10];
char chief_medical_officer[10];
};


The second library is called starship_functions.lib and contains the source:

/*** BeginHeader assignCaptain */
void assignCaptain(struct StarshipCrew * starship, char * captain);
void assignCaptain(struct StarshipCrew * starship, char * captain)
{
sprintf(starship->captain, "%s", captain);
}


And main looks like this:

#use "starship.lib"
#use "starship_functions.lib"

main(){
struct StarshipCrew enterprise;
char captain[10];
sprintf(captain, "Picard");
assignCaptain(&enterprise, captain);
printf("Captain of the Enterprise is %s\n", enterprise.captain);
}


This program compiles correctly once the links to the libraries are created in the library directory.

# Troubleshooting

Although I’ve just shown you works, there are many, many ways that libraries can be broken, and the compiler messages are not terribly useful.

## #use statements do not chain

If you move the #use “starship.lib” line to the starship_functions.lib, the main won’t be able to find the struct definition. The compiler will report the error:

line    5 : ERROR a_starship_test.c   : Struct use before definition.
line    9 : ERROR a_starship_test.c   : ) is missing/expected.
line    9 : ERROR a_starship_test.c   : Invalid struct reference.


Solution

Keep all use statements in your main() file.

Although they look superfluous, the BeginHeader statements are essential. Without the BeginHeader statement on the struct, this will create the error:

line    4 : ERROR a_starship_test.c   : Struct use before definition.
line    8 : ERROR a_starship_test.c   : ) is missing/expected.
line    8 : ERROR a_starship_test.c   : Invalid struct reference.


Without the BeginHeader statement on the function, the error is:

line    7 : WARNING a_starship_test.c   : Reference to 'assignCaptain' has no corresponding prototype.
line    7 : ERROR a_starship_test.c   : Undefined (but used) global label assignCaptain


It is important that the function name matches the function. For example, if the BeginHeader statement is changed to:

/*** BeginHeader assignCap */
void assignCaptain(struct StarshipCrew * starship, char * captain);


Then the compiler error is:

line    2 : ERROR STARSHIP_FUNCTIONS.LIB   : Undefined (but used) global label assignCaptain


Solution

All structs, typedefs and functions need BeginHeader functionName statement around the prototype, and functionName must match the struct or function name.

## Cleaning out compiled libraries

When a program with the #use statement is compiled, the compiler will look for the compiled library files for the used library. These are the files with the extension HX1 and MD1. If it can’t find these files, it will first compile the library, if it does find the files, it will ignore the lib file. This means that if you fix an error in your lib file, the next time you compile your project, you will get the super helpful error of:

line    2 : ERROR STARSHIP_FUNCTIONS.LIB   : Need function definition or declaration.
line    2 : ERROR STARSHIP_FUNCTIONS.LIB   : Syntax error - or garbage at end of program.


Solution

Before compiling your program, delete the HX1 and MD1 files for each library changed.

## Order of Import Errors

I’m not sure exactly how to replicate this error, but I have occasionally encountered compiler errors like this:

line  333 : ERROR LCD_FUNCTIONS.LIB   : Redefinition of parameter 1 different.


I’m not sure what causes this, but it typically happens when a library function uses a struct defined in another library, and this library is imported before the library with the function is imported in the main() function.

Solution

I’ve fixed this error by switching the order of imports from:

#use "lcd_functions.lib"
#use "library_with_struct.lib"


to:

#use "library_with_struct.lib"
#use "lcd_functions.lib"

30 March 2016

Although AVR now dominates the microcontroller landscape, some hardware projects still rely on older chips with quirky, proprietary versions of C. When working with these chips, many of C standard library functions are either unavailable or implemented… creatively. In this case: how to use arrays of strings in Digi’s Dynamic C.

Dynamic C has no malloc function. Instead, memory is allocated by declaring variables, such as:

char my_string[8];


All variables must be declared at the start of functions. For example, the perfectly valid C99 code:

main(){
int a;
a = 5;
int b;
b = a;
printf("b is: %d", b);
}


Results in the error:

line    4 : ERROR a_test.c     : int is out of scope/ not declared.
line    4 : ERROR a_test.c     : b is out of scope/ not declared.


The working code for Dynamic C is:

main(){
int a;
int b;
a = 5;
b = a;
printf("b is: %d", b);
}


This is frustrating, but not totally unworkable. Now let’s move on to something more complicated. Suppose you want an array of strings. Thanks to this StackOverflow answer we know that this code will work in C99:

main(){
char * point_names[5];
int i;
for(i=0;i<5;i++) {
point_names[i] = malloc(100);
sprintf(point_names[i], "number %d", i);
}
for(i=0;i<5;i++){
puts(point_names[i]);
}
}


Note that without the malloc() statement, the program will seg fault. In Dynamic C, the program won’t seg fault, but it will print out gibberish. Since malloc() does not exist in Dynamic C, we’ll have to declare a variable with the size we want, put the data there, and then copy the pointer to our array of pointers:

main(){
char names[5][100];
char * point_names[5];
int i;
for(i=0;i<5;i++) {
sprintf(names[i], "number %d", i);
point_names[i] = names[i];
}
for(i=0;i<5;i++){
puts(point_names[i]);
}
}


If point_names is declared right after names, then sprintf’ing right to point_names will work. This is getting into risky territory, however.