Jonathan Hamberg

Raising Quail and Building a Hutch

Mon, 19 Jan 2026 00:00:00 GMT

Building the Coop

This last summer I decided I wanted to raise quail in the back yard. I think YouTube algorithm started showing me a bunch of homestead videos about raising quail. They looked easy to raise and gave you eggs that you could eat or sell. So I decided to give it ago. I also used it as opportunity to learn how to build a quail hutch by hand.

I followed a YouTube tutorial by the Slightly Redneck channel. The design was simple to build. Just required some firring strips, hardware cloth, and some cheap plywood to build. I bought a Skill Saw and all the supplies from Lowes

Took me quite a few weekends over the summer, mostly because this is the first time I was building something like this and using all the tools. I learned a lot about actually building things. Overall I'm happy with how it turned out. I learned where you should put your extra focus to make sure things are square and line up. Like the doors. It's really important that you make everything square so that it lines up when you close the doors.

Here's a picture of the final build hutch.

We bought 36 Egyptian Jumbo Coturnix quail hatching eggs from https://www.juddsquail.com/

Due to the inexpensive incubator we used, only one chick hatched shown below. After the fact we learned that incobating quail eggs is more difficult than chicken eggs since they have tigher temperature and humidity tolerances. We didn't want the quail to be alone since they are social animals. I ended up finding someone on Craigslist selling some Coturnix quail and ended up with 12 all together.

Here's a picture of the quail in the hutch once they've gotten old enough to stay outside instead of the garage where it was warmer.

Overall Experience

It's fun having Quail. They are super cute looking. I love how rotund and round they are. They are pretty skittish by nature and don't really enjoy being held. Although I still do it sometimes because they are cute and make the funniest face when they are protesting you holding them. They have a light on a timer so that they produce eggs throughout the year since they required ~14 hours of light per day, which only happen in the summer in Washington.

We have some male and female quail so we may consider trying to incubate some of their fertilized eggs to have more quail in the future. They only live ~2-3 years so we'll need to figure out how to get more. They hatching eggs and chicks are relativly expensive so hatching our own would be a good option.

Conclusion

They are fun pets to have and provide us with eggs to eat. They are fun to watch run around and play in their hutch. It was also a great learning experience for me to learn how to use tools to build something with my hands. As a software engineer it's almost expected that i'd like to try my hand at carpentry to actually build something physical.

Mou'aputa Trip Report

Thu, 04 Sep 2025 00:00:00 GMT

I just got back from a trip to Tahiti and Moorea. On Sunday, August 31st, I hiked Mou'Aputa.

According to local legend, the 30-foot hole in the top of the mountain was caused by a spear piercing the mountain.

I started at 7:30 in the morning. The hiking trail begins through a residential neighborhood, so you can park anywhere along the main island road.

Here's a link to the activity on Strava. It was quite a steep hike, and since I hadn't done any hiking recently, I maintained a very slow and steady pace on the way up. https://www.strava.com/activities/15654698923

Here's a picture from when the hike first starts after leaving the local neighborhoods.

On a short detour on the way to the top, you can visit Vaioro Waterfall, which is shown below.

One of the things that was different from most of the Washington hiking I've done before is that this is jungle terrain. The dense jungle is unlike anything I've hiked before.

Most sections of the hike before the ridgeline were through dense forest, and the middle half of the hike followed a riverbed.

There was even some pretty interesting foliage along the way. This one, for example, has square leaves!

Here's a picture from when I first started hiking along the ridge to the top of the mountain.

Here's a quick selfie with the view of the mountain and ocean below. Once you reach the final ridge to the top, it becomes very steep. There are steep dropoffs on each side of the trail as you follow the ridge to the top. About half of the ascent required ropes to assist the climb. The AllTrails description advised against hiking this trail when it rains because it becomes muddy and slippery. Just my luck - it rained for 20 minutes right as I was getting to the top. This turned all the mud very slippery. I decided to turn around at this point even though I was only several hundred feet from the top. If the terrain was anything like the steep sections I'd already climbed, it would have been almost impossible to get back down safely in such slippery conditions.

The way back down was pretty miserable. It was so slippery I slipped and fell 4-5 times. My boots, hands, and shorts were covered in mud by the time I got to the bottom. The descent was even slower than my ascent because every step needed to be calculated to ensure I didn't lose my grip. If I were to do this hike again, I'd definitely make sure it was on a completely sunny day with no rain. I'd also bring some spikes for my boots to get more traction in the mud.

Even though I didn't make it to the top, the views of the valley and ocean made it worth it!

Self Hosting Service with Cloudflare Tunnels

Mon, 28 Jul 2025 00:00:00 GMT

Introduction

I'd like to show you how I host personal services using Cloudflare tunnels. For this post, I'll walk through setting up a service called GoatCounter, a privacy-focused web analytics platform that allows you to track website traffic without compromising user privacy. I use it to monitor how many people visit my blog.

First, I own the domain jonathanhamberg.com, which I purchased from NameCheap, but I've configured the DNS through Cloudflare. This setup is important for making Cloudflare tunnels easier to configure.

I also have a home NAS running TrueNAS that hosts these services.

TrueNAS Setup

From the Apps page, click on "Discover Apps":

You'll see an option to "Install via YAML" where you can paste the Docker compose file available from the GoatCounter GitHub page:

services:
  goatcounter:
    image: arp242/goatcounter
    ports:
      - '20236:8080'
    volumes:
      - goatcounter-data:/home/goatcounter/goatcounter-data
version: '3'
volumes:
  goatcounter-data:

Once you've added the service, you should see the following, indicating that GoatCounter is now running on TrueNAS:

Cloudflare Tunnel Setup

Now that GoatCounter is running on the local NAS, we can set up the tunnel to expose it to the internet. First, log in to Cloudflare and navigate to "Zero Trust":

Next, navigate to Network → Tunnels. Here you can see a list of tunnels for your account. I've already set up one for my NAS called "truenas":

You'll need to install the Cloudflare TrueNAS app, which should be available from the Apps page in TrueNAS. Simply follow the instructions from Cloudflare on how to connect TrueNAS to Cloudflare:

Once the tunnel is configured, Cloudflare is ready to forward traffic from your local machine to the internet.

Now click "Add a public hostname." This forwards a public subdomain to your local service running on your NAS:

By navigating to goatcounter.jonathanhamberg.com, you can now see the page served from your local network:

For services that shouldn't be publicly accessible to everyone on the internet, you can set up a Cloudflare policy to only allow certain authorized email addresses to access the site. These email addresses can be authenticated through an email with a PIN code, or you can set up Google Login for easier authentication. For domains that need public API access, these policies should remain open. It's also possible to add policies for certain routes while having bypass policies for publicly available routes.

Cloudflare Security Policy

When self-hosting web services, I prefer to keep publicly accessible routes to a minimum for security. In GoatCounter's case, the dashboard is exposed at goatcounter.jonathanhamberg.com, which could allow unauthorized login attempts. Ideally, only the HTTP API and count.js script should be publicly available. This is where Cloudflare applications and policies become useful.

Here you can see that I've created two applications for GoatCounter: one general policy that requires authentication and covers the entire website, and a specific application for API usage that bypasses authentication requirements for public access:

I've created one application for goatcounter.jonathanhamberg.com/* and attached an authorization policy to it. This requires authentication via email or Google OAuth to view these routes. I trust Cloudflare's authentication more than anything I might set up on my local NAS. I feel more secure knowing that the application is protected before requests even reach my NAS:

According to the GoatCounter API documentation (https://www.goatcounter.com/help/api), only the /count.js and /count HTTP endpoints need public access for telemetry. All other routes are for the dashboard and internal APIs that shouldn't be exposed to the internet. This approach makes me feel much safer, exposing only the required routes rather than the entire GoatCounter subdomain.

For public access, I've added the goatcounter_api application with the /count.js and /count HTTP endpoints. I've assigned a BYPASS policy to this application, meaning these routes don't require authentication. One confusing aspect of setting up this policy and application for GoatCounter is that you can't accomplish this with a single application. That's why two applications are required: one for blocking access and one for bypassing authentication on public routes. Once both applications are configured, everything works as expected:

This is the page you'll see when trying to access goatcounter.jonathanhamberg.com, but you'll still have access to goatcounter.jonathanhamberg.com/count.js:

Summary

PS. Obviously, a cloud VPS offers more reliability than a home server since you don't have to worry about power or internet outages. However, for running a service for a simple blog, this setup is sufficient. I also like the idea of maximizing the use of the expensive hardware I bought for the NAS to make the investment more worthwhile.

Switching to Astro

Sun, 27 Jul 2025 00:00:00 GMT

I've gotten the itch to switch up my website recently. Previously it was build with hugo. I think hugo is pretty great. I really like the idea of generating a static website for a personal blog because it's super easy to host for free using a variety of different hosting services. Now of days there are even integrations that will build from the source code and directly publish the website so you don't need to do this step manually.

Things I had a hard time with Hugo

One of my biggest complaints about hugo was that the themes were pretty low level and how to modify as a end user. One of the reasons hugo is so fast is it's templating language. But it was complicated for me to use. Whenver switching to a new theme I'd have to git clone the theme template. And then every theme had it's own options to configure in ithe website's config.toml. This made it hard to try out new themes because they could not be tried with low friction. Once all of the settingcs were configured then you could correctly look at the website, but that took a long time to get configured just to see an example of it applied to your website.

Another complains I had about hugo templates was that the templates were all located in a git submodule. This made it hard to configurate the template for my websie's usecase since it required modifying a vendor's dependency. I'd have to fork the git repo if I wanted to make any changes. This was pretty high effort just to tweak a theme that is used for the blog.

Also from my limited knowledge of hugo it seemed that templating of hugo was pretty low level. Editing html, javascript, css files manually, etc... I don't know a lot about web development, but this was very intimidating to make the changes I wanted to make. I had to have deep knowledge of the web frameworks the hugo templates used in order to make any useful changes.

Reasons I picked Astro

I've looked into a couple options and Astro stood out to me.

I liked that it was a static website bulider so I could host it for free on a variety of hosting platforms.

I liked how the site generation was custamizable using a variety of front end systems like React or Vue, etc... Gives the ability to custimize the frontend. Hugo seemed to be limited to using whatever framework the theme specified so there wasn't a lot of room to add custom logic to the site.

Astro supported converting markdown posts into posts which is covienient for authoring pages from my local machine. RSS generating was also pretty easy to set up as a route.

I'm pretty behind the times since Astro seems pretty popular now. I haden't really even heard of it until switching my blog over to it. I've choosen the basic theme that comes with base template. In the future I'll look into using a more iteresting theme. But for now I've started by just porting over the basics.

That's it for now. Just wanted to share an update why the site looks different now.

AOSP builds from fish shell using just

Sun, 11 Aug 2024 00:00:00 GMT

Introduction

I like to use the fish shell. It's easy to use, has most thingns I like to use configured with no issues. The problem I've run into recently is that building AOSP is only supported by running bash. This is because the build scripts define a bunch of bash functions that cannot be converted over to fish shell.

Building on Android

cd android-14-gsi
source build/envsetup.sh
lunch aosp_cf_x86_64_phone-eng
# m is a bash function
m sync

I use just on my development machines. I'd like to be able to just specify just aosp m sync to run whatever command I'd like. Typically I'd just create a bash script runner command in just and source the commands from above. But some android variant's can take a little while to run the envsetup.sh and lunch commands, so you wouldn't want to run them every time you run a command.

Solution

The solution I came up with is to source the build enviroment for running AOSP commands, and then saving this state to file. Then whenever I'd like to run a AOSP build command from just, I'll just source this saved state and run the command directly.

In bash this is how you'd save the state.

 declare -p > aosp-state.sh && declare -f >> aosp-state.sh

Here's the docs for -p display the attributes and value of each NAME. This is required. Because by default the declare statement will only save bash variables, and will not distinguise between a regular bash variable and an environment variable. These types are needed because the AOSP needs these environment variables.

The second part saves all functions declaration in a bash shell to the file so that they can be sourced later.

Now to run an AOSP build you'd source the saved state like so.

source aosp-state.sh
bash: declare: BASHOPTS: readonly variable
bash: BASH_VERSINFO: readonly variable
bash: declare: EUID: readonly variable
bash: declare: PPID: readonly variable
bash: declare: SHELLOPTS: readonly variable
bash: declare: UID: readonly variable

Only problem is that you get a bunch of warning about read only variables. There are some variables that are not actually variables, but information provided directly by bash to the user. If you don't want to see the warning every time you source this script you should manually delete these entries from aosp-state.sh

If you look in the file they are anything that is specified by the declare -r option before the variable name. Make sure to ignore the - when searching for matches. You only care about the r part of the declare specification.

Now to build you just need to

source aosp-state.sh
m sync

This is much faster than having to run the setup steps.

Justfile integration

.justfile

aosp target:
    #!/usr/bin/env bash
    source aosp-state.sh
    cd ~/android/android-14-gsi
    {{target}}

Now an just command can be run from wherever to start a AOSP build. Like so

just aosp m sync

Because we are not actually running the envsetup.sh and lunch this command starts very quickly. The only downside of this is that every time there is a change to the envsetup.sh or the lunch target the state will need to be saved again to aosp-state.sh because this is not automatically updated. For now that's something that I'm willing to live with. Hope this helps anybody who uses the fish shell, but would like to start AOSP builds without leaving the fish shell environment.

Creating Programs for Lab Machines with No Internet

Mon, 25 Jul 2022 00:00:00 GMT

Introduction

At the office I work with a lot of remote lab machines. There are different types of these machines. Some running an old version of ubunt. Some running an old version of CentOS. Some have no admin permissions to install packages. So have no internet which makes it hard to install packages.

There are several programs that make my development life easier like fish, rg, fd which I use everyday. If I have to use old outdated tools that really gets in the way of my flow. So in this article I'm going to describe how I got around the issue of running my software on a lab machine that doesn't have internet or admin privledges using Docker.

Getting Started

Basically a lab machine does not have the ability to install new packages. This means you cannot just install the desired package using apt or dnf because you don't have admin privledges. Sometimes building the application from source is an option, but the machine may be missing build dependencies that you cannot just install with apt or dnf. You also cannot just build the applications on your local machine because it has different GLIBC dependencies which make it a mess when moving binaries to a different system.

So in this article I'm going to describe how to compile these applications for the target system using docker. Docker allows the user to specify distro that is used to run all of the commands in. So to get this to work, docker just has to specify the base system to have the same system image as the remote target system. If this is done the docker image can install all the build dependencies it wants and can just install every package it wants without needing admin or an internet connection on the target machine.

Docker Script

FROM ubuntu
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Etc/UTC
WORKDIR /home

# Update base system, then install packages that are available in the package
# manager
RUN apt-get update && \
        apt-get -y upgrade && \
        apt-get install -y build-essential curl libpcre2-32-0 python3 man-db gettext-base cargo direnv neovim picocom python3-sphinx cmake git ncurses5-dev pkg-config

# Create .local directory that will contain all the desired applications
# that will be moved to the remote target system.
RUN mkdir /home/jonathanhamberg && mkdir /home/jonathanhamberg/.local
# Install my desired rust programs.
RUN cargo install exa du-dust fd-find ripgrep bat tokei
# Clone and build fish-shell
RUN git clone https://github.com/fish-shell/fish-shell.git
RUN cd fish-shell && cmake -DCMAKE_INSTALL_PREFIX=/home/jonathanhamberg/.local/ . -DBUILD_DOCS=True && make -j32 && make install
# Move all binaries to the .local folder which is going to be copied
# to the remote target system.
RUN cp /root/.cargo/bin/* /home/jonathanhamberg/.local/bin/
RUN cp $(which nvim) /home/jonathanhamberg/.local/bin/
RUN cp $(which picocom) /home/jonathanhamberg/.local/bin/
RUN cp $(which direnv) /home/jonathanhamberg/.local/bin/

sudo docker build . -t myubuntu

Once the image has been build just copy the .local directory out of the docker image. Then copy the .local image to the remote target server.

# Copy folder from docker image to the current directory.
sudo docker  run -v(pwd):/data -it myubuntu cp -r /home/jonathanhamberg/.local/ /data/.local/
# Copy the local .local to the remote target.
rsync -a .local/ remote_system:~/.local/

Now that this is done you can run all your modern applications that make you development environment feel like home. You don't need admin or internet to make this work. And whenever there is a different version of a lab machine you just need to modify the docker file to specify a different base image.

Generate ASCII Network Diagrams with protocol

Thu, 10 Dec 2020 00:00:00 GMT

Protocol

I've recently found a cool little command line utility called protocol written by Luis MartinGarcia. The protocol program is used to generate network protocol diagrams using ASCII art. The ASCII art style is the same style that is used in the IETF RFC documents. Here's an example of the tcp packet structure.

>> protocol tcp
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Source Port          |        Destination Port       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Sequence Number                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Acknowledgment Number                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Offset|  Res. |     Flags     |             Window            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            Checksum           |         Urgent Pointer        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Options                    |    Padding    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The protocol application can be used to generate your own diagrams as well. Simple usage is just to specify the field name and the field width in bits. The fields are separated by commas and the bit segment is specified by a : after the field name. Here's an example of creating a custom diagram.

protocol "field 1:8,field 2:24, field 3: 32"
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    field 1    |                    field 2                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                             field 3                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Conclusion

The protocol application is really neat program to generate ASCII art representations of packet structures. This can be quite useful for debugging byte packing and unpacket when you have a nice visual packet field representation.

Embedding Binary Data in Executable with CMake

Mon, 30 Nov 2020 00:00:00 GMT

Embedding Arbitrary Data in Executable with CMake

Sometimes you need to include some binary data into an executable. It's a lot easier to distribute 1 executable than it is to distribute an executable with supporting files. There are many times that embedding files into an executable would be useful. For example you could embed some simple image files into the executable, or any resource you would like to use in the executable. Also sometimes a target that you are compiling for doesn't have file system support like an embedded target for example. In this article I'm going to show how to embed arbitrary binary data into an executable.

There are a couple requirements to this system that I would like to discuss before we get starting. This method should be cross platform. There are several ways of embedding binary data into an executable using the linker, but that would not be cross platform across the many C++ compilers like GCC, Clang, and MSVC and others. Also the embedded files should take advantage of CMake's features and automatically keep the binary data up to date if the source file changes.

Getting Started in CMake

The first thing we are interested in is figuring out how to get the data from an arbitrary file into CMake. This is where the file(READ) function comes in. The file(READ) takes in a file as an argument and a variable that the content of the file should be stored in. It also has a HEX command which converts the file into HEX which allows us to work with binary data.

We'll be working with a file called message.txt which contains the following text.

This mesage is from message.txt

Let's try out the file(READ) function.

file(READ message.txt content HEX)
message(${content})

This file outputs the following message 54686973206d6573736167652069732066726f6d206d6573736167652e747874 This is the content from the message.txt file that has been encoded using HEX. Every 2 hex characters corresponds to 1 byte of the source file. We now have the raw data of the file that we want to embed into the executable. Now we want to be able to store each byte as a separate element in a CMake list so that we can work with the data easier. We can do that using the string(REGEX) function to separate each of the bytes. Here's the syntax: string(REGEX MATCHALL "([A-Fa-f0-9][A-Fa-f0-9])" SEPARATED_HEX ${content}) This uses the regex of 2 hex digits and matches all occurrences of the matched regex into a list called SEPARATED_HEX.

Now that we have all of the data in accessible in CMake we just need a way to store that data in an executable. The method I am going to use is to create a .c file that corresponds to the binary data that we want to embed. Once the source file has been created we are going to create an array and initialize that array to contain the contents from the file that we would like to embed in the executable. Here's the code to do just that.

# Create a counter so that we only have 16 hex bytes per line
set(counter 0)
# Iterate through each of the bytes from the source file
foreach (hex IN LISTS SEPARATED_HEX)
	# Write the hex string to the line with an 0x prefix
	# and a , postfix to seperate the bytes of the file.
    string(APPEND output_c "0x${hex},")
    # Increment the element counter before the newline.
    math(EXPR counter "${counter}+1")
    if (counter GREATER 16)
    	# Write a newline so that all of the array initializer
    	# gets spread across multiple lines.
        string(APPEND output_c "\n    ")
        set(counter 0)
    endif ()
endforeach ()

# Generate the contents that will be contained in the source file.
set(output_c "
#include \"${c_name}.h\"
uint8_t ${c_name}_data[] = {
    ${output_c}
}\;
unsigned ${c_name}_size = sizeof(${c_name}_data)\;
")

# Generate the contents that will be contained in the header file.
set(output_h "
#ifndef ${c_name}_H
#define ${c_name}_H
#include \"stdint.h\"
extern uint8_t ${c_name}_data[]\;
extern unsigned ${c_name}_size\;
#endif // ${c_name}_H
    ")

After the code from above executes here will be the contents of the .h/.c file.

message_txt.h

#ifndef message_txt_H
#define message_txt_H
#include "stdint.h"
extern uint8_t message_txt_data[];
extern unsigned message_txt_size;
#endif // message_txt_H

message_txt.c

#include "message_txt.h"
uint8_t message_txt_data[] = {
    0x54,0x68,0x69,0x73,0x20,0x6d,0x65,0x73,0x73,0x61,0x67,0x65,0x20,0x69,0x73,0x20,0x66,
    0x72,0x6f,0x6d,0x20,0x6d,0x65,0x73,0x73,0x61,0x67,0x65,0x2e,0x74,0x78,0x74,
};
unsigned message_txt_size = sizeof(message_txt_data);

Here you can see that the message_txt.h exposes the name of the data that has been embedded into the executable. The name of the object is named after the source file name. message.txt has been transformed into message_txt_data and message_txt_size. The data is a uint8_t point to the actual data that has been embedded. The size is the size in bytes of the data that has been embedded. Now that the source files have been generated in cmake we need to create a CMake target to build the generated source files.

# This function is used to create the file_embed target which is used to build
# the generated source files from the imported binary file.
function(FileEmbedSetup)
	# Make sure the directory exists where the generated source files end up
    if (NOT EXISTS ${CMAKE_BINARY_DIR}/file_embed)
        file(MAKE_DIRECTORY ${CMAKE_BINARY_DIR}file_embed)
    endif ()
	# CMake does not allow libraries with no source files.
	# So create an empty source file if necessary to create the library with.
    if (NOT EXISTS ${CMAKE_BINARY_DIR}/file_embed/file_embed_empty.c)
        file(WRITE ${CMAKE_BINARY_DIR}/file_embed/file_embed_empty.c "")
    endif ()

	# Create the file_embed library which will be used to compile all the
	# generated source files.
    add_library(file_embed ${CMAKE_BINARY_DIR}/file_embed/file_embed_empty.c)
    # When linking against this library make sure the include directory is
    # added to the library.
    target_include_directories(file_embed PUBLIC ${CMAKE_BINARY_DIR}/file_embed)
endfunction()

Now all that's left is to make CMake aware of the files that we would like to import into the project. I've created a function FileEmbedAdd(file) that does just that which is defined below.

function(FileEmbedAdd file)
	# Contains all the code from above to read in a file contents
	# and write the array to a generated .h/.c file.
    FileEmbedGenerate(${file} var)
    # target sources linkes a source file to the specified library.
    # the var varibale contains the name of the source file that
    # contains the generated array for the embedded data.
    target_sources(file_embed PUBLIC ${var})

	# This command adds a custom command that indicates that the
	# generated source file is dependent on the source binary
	# file.  THis means that if the source file changes then
	# the generated embedded file should also be updated.
    add_custom_command(
            OUTPUT ${var}
            COMMAND ${CMAKE_COMMAND}
            -DRUN_FILE_EMBED_GENERATE=1
            -DFILE_EMBED_GENERATE_PATH=${file}
            -P ${CMAKE_SOURCE_DIR}/cmake/FileEmbed.cmake
            MAIN_DEPENDENCY ${file}
    )
endfunction()

Here is what the user would see if they would use these CMake functions. here's the user's CMakeLists.txt where the project is set up and the file is registered using the FileEmbedAdd function.

cmake_minimum_required(VERSION 3.11)
project(file-embed)

add_executable(main main.cpp)

include(../cmake/FileEmbed.cmake)
FileEmbedSetup()
FileEmbedAdd(${CMAKE_SOURCE_DIR}/message.txt)

target_link_libraries(main file_embed)

// Include the generated file that is used to access the embedded data.
#include "message_txt.h"
#include <iostream>

int main() {
    // Iterate over each character of the imported embedded data.
    for(unsigned i = 0;i < message_txt_size;i++) {
        // Print the message to the console.
        std::cout << message_txt_data[i];
    }
    std::cout << "\n";
}

Here's the output of the example program.

make
./main
This message is from message.txt

Conclusion

An example of this file embedding code is available on GitHub. Now you can embed files into an executable and access the data without needing to use a file. This method also has the advantage of being compatible between multiple compilers.

Embedding Git Hash with CMake

Fri, 27 Nov 2020 00:00:00 GMT

Using Git Commit Hash in CMake Project

Often times it's very useful to include the version number into the software that you are building. Even better than a version number is the git hash of the commit that was used to build the software release. In this article I'm going to describe my function that I came up with to do just that.

Let's start with the code to read the git hash from the current source directory. This command reads the current git commit hash and stores the result in the variable GIT_HASH.

    # Get the latest abbreviated commit hash of the working branch
    execute_process(
        COMMAND git log -1 --format=%h
        WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
        OUTPUT_VARIABLE GIT_HASH
        OUTPUT_STRIP_TRAILING_WHITESPACE
        )

This commit hash can then be passed on to the rest of the CMake project. It seems like we this should be all we need. Unfortunately there are several issues with using just this method alone. This command is only run during the CMake configuration stage. So if the user configure the CMake project, then commits some changes, and then builds again the GIT_HASH variable will not be updated because the CMake project was not re-configured. This will lead to the wrong hash being used in building the project.

So lets talk about how we are going to expose the git commit hash to the build code. The first option would be to pass the git hash as a define statement to the project being build. The down-side of this is that if the git commit hash changes then the whole project has to be rebuilt, because the command line arguments have changed. The second option would be be put the git hash in a header file that could be accessed by the rest of the project. This is better, because the whole project would not have to be recompiled, but only the files that included the header that contained the git commit hash. Another option would to be to have a header file that contains a external reference to the git commit hash. The git commit hash would then be written to a source .c or .cpp file. this has the advantage of only having to re-compile one file when the git commit has has changed. This is the approach I have chosen for this function.

Here is the source of the git_version.h.

#ifndef GIT_VERSION_H
#define GIT_VERSION_H

extern const char *kGitHash;

#endif // GIT_VERSION_H

Here is the source of the git_version.h.in

#include "git_version.h"
const char *kGitHash = "@GIT_HASH@";

This is not the file that will actually be compiled. First it must be configured by CMake to include the actual git commit hash that should be used. CMake includes a function called configure_file that can be used to configure a file and update all of the CMake variables that are referenced in the configuration file.

configure_file(git_version.cpp.in git_version.cpp @ONLY)

Okay now we have a way to access the git commit hash, but we still have the problem of the hash not being updated unless the project is re-configured. So let's add a custom target that is run every time the project is build. this custom target will determine if the git commit hash has changed and update it if necessary. Here's how to add a custom target that runs a CMake command.

add_custom_target(AlwaysCheckGit COMMAND ${CMAKE_COMMAND}
    -DRUN_CHECK_GIT_VERSION=1
    -Dpre_configure_dir=${pre_configure_dir}
    -Dpost_configure_file=${post_configure_dir}
    -DGIT_HASH_CACHE=${GIT_HASH_CACHE}
    -P ${CURRENT_LIST_DIR}/CheckGit.cmake
    BYPRODUCTS ${post_configure_file}
    )

We are adding a target called AlwaysCheckGit. The COMMAND ${CMAKE_COMMAND} indicates that we should run the cmake executable for this command. The -P indicates that we should run cmake in a script mode which means that it will run the cmake file, not modify the cache at all. We must also pass any variables we want the script to have access through the -D arguments. the -DRUN_CHECK_GIT_VERSION=1 tells the script that we should run the CheckGitVersion function. The -Dpre_configure_file and -Dpost_configure_file include the directories that the configuration files are from and the generated data should go. These must be passed along since a CMake script file does not have access to the cache so we must pass these values along. We also have to specify the BYPRODUCTS of the command. This allows the byproducts to be used in other parts of the CMake script otherwise there would be an error indicating that the file is missing, but in reality the file has just not been generated yet, which isn't a problem.

We almost have all of the pieces together. The only thing left is that we want to reduce the amount of work done. We only want to regenerate the git_version.cpp if the git hash has actually changed. Since we don't have access to the CMake cache when running in CMake's script mode, we have to save this information into an external file called git-state.txt.

We can write the git commit hash using the file(WRITE) command and then we can file(STRINGS) command to read the content of the file back into CMake. This method of saving the state between between runs of the CMake command comes from the CMake-git-version-tracking repository.

# This command is used to save git commit hash.
file(WRITE ${CMAKE_BINARY_DIR}/git-state.txt ${git_hash})
# This following command is used to retreive the git commit hash from the file.
if (EXISTS ${CMAKE_BINARY_DIR}/git-state.txt)
    file(STRINGS ${CMAKE_BINARY_DIR}/git-state.txt CONTENT)
    LIST(GET CONTENT 0 var)
    set(${git_hash} ${var} PARENT_SCOPE)
endif ()

Here is how the the command is expected to be used in a normal CMakeLists.txt

cmake_minimum_required(VERSION 3.11)
project(git-hash)

include(../cmake/CheckGit.cmake)
CheckGitSetup()

add_executable(main main.cpp)
target_link_libraries(main git_version)

Here's a example cpp file that uses the git commit hash and prints it to the console. This file will update with the new git hash every time a new git commit is mode.

#include <iostream>
#include "git_version.h"

int main()
{
    std::cout << "Git Hash: " << kGitHash << "\n";
    return 0;
}

In review, I believe this is one of the best methods of including a git commit hash in a CMake project. It is able to update the git commit hash without re-configuring the project. It only updates the git commit hash when it changes reducing the amount of source files that have to be recompiled between git commits.

Here's a link to the complete version of the code. I've glossed over some of the finer implementation details since some of them are not super important and is just fighting with CMake paths.

GCC Archive Linker Oddity

Wed, 25 Nov 2020 00:00:00 GMT

Problem Setup

I've recently been trying to compile a simple hello world program on a STM32F466 Nucleo development. The STM32 MCU development kit comes with a excellent HAL. I've been trying to get this HAL integrated well with the CMake build system. I didn't want to specify all the HAL source files for every executable that I was going to add to the project. So I naively decided to create a static library containing all of the

cmake_minimum_required(VERSION 3.11)
project(hello C CXX ASM)

# Create library called hal that contains the compiled STM32 HAL sources.
add_library(hal
        Core/Src/system_stm32f4xx.c
        startup_stm32f446xx.s
        Core/Src/stm32f4xx_hal_msp.c
        Core/Src/stm32f4xx_it.c
        Core/Src/syscalls.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_tim.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_tim_ex.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_uart.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_rcc.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_rcc_ex.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_flash.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_flash_ex.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_flash_ramfunc.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_gpio.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_dma_ex.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_dma.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_pwr.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_pwr_ex.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_cortex.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal.c
        Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_exti.c
        )

# Add CPU configuration compiler options.
target_compile_options(hal PUBLIC
        -mcpu=cortex-m4
        -mthumb
        -mfpu=fpv4-sp-d16
        -mfloat-abi=hard
        -fdata-sections
        -ffunction-sections
        $<IF:$<CONFIG:Release>,-Os,-Og>)
# Add linker options.
target_link_options(hal PUBLIC
        -mcpu=cortex-m4
        -mthumb-mfpu=fpv4-sp-d16
        -mfloat-abi=hard
        -T${CMAKE_SOURCE_DIR}/STM32F446RETx_FLASH.ld
        --specs=nano.specs
        -Wl,--gc-sections
        )
# Add compiler define options.
target_compile_definitions(hal PUBLIC -DUSE_HAL_DRIVER -DSTM32F446xx)

# Add include directorys necessary to compile the HAL.
target_include_directories(hal PUBLIC
        Core/Inc
        Drivers/STM32F4xx_HAL_Driver/Inc
        Drivers/STM32F4xx_HAL_Driver/Inc/Legacy
        Drivers/CMSIS/Device/ST/STM32F4xx/Include
        Drivers/CMSIS/Include
        Drivers/CMSIS/Include)

# Create sample application.
add_executable(hello
        Core/Src/main.c
        )
# Link the HAL agains the sample application.
target_link_libraries(hello hal)

Ideally we would have a successful compilation with the following commands. To my suprise I get these errors thrown in my face instead.

FAILED: hello
: && /home/jhamberg/src/companion-firmware/toolchain/gcc-arm-none-eabi-9-2019-q4-major/bin/arm-none-eabi-gcc -O3 -DNDEBUG -mcpu=cortex-m4 -mthumb -mfpu=fpv4-sp-d16 -mfloat-abi=hard -T/home/jhamberg/src/embedded-cmake-course/04-08-library-dependencies/STM32F446RETx_FLASH.ld --specs=nano.specs -Wl,--gc-sections CMakeFiles/hello.dir/Core/Src/main.c.obj -o hello  libhal.a && :
/home/jhamberg/src/companion-firmware/toolchain/gcc-arm-none-eabi-9-2019-q4-major/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/bin/ld: /home/jhamberg/src/companion-firmware/toolchain/gcc-arm-none-eabi-9-2019-q4-major/bin/../lib/gcc/arm-none-eabi/9.2.1/../../../../arm-none-eabi/lib/thumb/v7e-m+fp/hard/libc_nano.a(lib_a-sbrkr.o): in function `_sbrk_r':
sbrkr.c:(.text._sbrk_r+0xc): undefined reference to `_sbrk'
...

Normally linker errors are no big deal. You just find the undefined reference and make sure that it's included in the build and properly linked in with the end result. So I know that I'm defining the _sbrk function in the syscalls.c file. This file should define all of functions required by the C lib to run on an embedded system. In this case most of them are stub functions, other than the _write, and _sbrk functions which write to the console and update the heap respectivly. If we take a look at the syscalls.c file we can see indeed that the _sbrk() function is implemented.

caddr_t _sbrk(int incr)
{
	extern char end asm("end");
	static char *heap_end;
	char *prev_heap_end;

	if (heap_end == 0)
		heap_end = &end;

	prev_heap_end = heap_end;
	if (heap_end + incr > stack_ptr)
	{
//		write(1, "Heap and stack collision\n", 25);
//		abort();
		errno = ENOMEM;
		return (caddr_t) -1;
	}

	heap_end += incr;

	return (caddr_t) prev_heap_end;
}

Maybe the symbol didn't actually get compiled successfully into the hal library. So let's search for the _sbrk() function in the libhal.a static library file.

nm -g libhal.a | rg sbrk
00000001 T _sbrk

Interesting we can see that the _sbrk() function is indeed present in the libhal.a library file. So what is the issue then. On the previous part of this project I didn't get a linker error when I compiled all of the source files in one executable.

In the previous step the linker command looked something like this. Quite a few of the sources file were omitted for brevity. The command below compiled and linked perfectly fine.

arm-none-eabi-gcc -g --specs=nosys.specs -mcpu=cortex-m4 -mthumb -mfpu=fpv4-sp-d16 -mfloat-abi=hard -T/home/jhamberg/src/embedded-cmake-course/04-07-programming-bin/STM32F446RETx_FLASH.ld -Wl,--gc-sections CMakeFiles/hello.dir/Core/Src/system_stm32f4xx.c.obj
...
CMakeFiles/hello.dir/Core/Src/syscalls.c.obj
CMakeFiles/hello.dir/Drivers/STM32																																																																																																																														ffmpeg-rtmp-duplication (copy).md																																																																																																									F4xx_HAL_Driver/Src/stm32f4xx_hal.c.obj -o hello

Here was the linker command using the hal static library that didn't pass.

arm-none-eabi-gcc -mcpu=cortex-m4 -mthumb -mfpu=fpv4-sp-d16 -mfloat-abi=hard -T/home/jhamberg/src/embedded-cmake-course/04-08-library-dependencies/STM32F446RETx_FLASH.ld --specs=nano.specs -Wl,--gc-sections CMakeFiles/hello.dir/Core/Src/main.c.obj -o hello  libhal.a

But when we ran this command we still have the undefined reference to _sbrk() and others even though we clearly see libhal.a is being included, and that libhal.a has the _sbrk() function. The answer to what is going on is located deep in the ld documentaion shown below.

-l namespec
--library=namespec
...
The linker will search an archive only once, at the location where it is specified on the command line.  If the
archive defines a symbol which was undefined in some object which appeared before the archive on the command
line, the linker will include the appropriate file(s) from the archive.  However, an undefined symbol in an
object appearing later on the command line will not cause the linker to search the archive again.
...

This says that the linker will only import symbols from the archive that are currently undefined. The problem with the code is that our application was not using those functions defined in the syscalls.c. This makes sense since those functions are provided for the C standard library to use during the final stage of the linker when we link against the nano specs implementation of the standard C library. Again since our library never used those functions, there were no undefined references to _sbrk() so according to the ld documentation the functions were not imported into the next linker steps. Whereas when we are linking object files individually, all of the symbols are imported by default regardless of if there the symbols are currently unresolved or simply missing. This is the different between linking a bunch of object files together and linking against an archive of object files.

Solution

The solution is to use a command that forces the linker to include all of the symbols from the archive, even if they are not currently used by the application. This argument is called the --whole-archive and --no-whole-archive. The whole-archive argument tells the linker that it should include all of the symbols in the remaining static library archives. The no-whole-archive tells the linker to return to the default behavior for the remaining static library archives passed through the command line.

So in CMake we add the whole-archive and no-whole-archive into the link options like so.

target_link_options(hal PUBLIC
        -mcpu=cortex-m4
        -mthumb -mfpu=fpv4-sp-d16
        -mfloat-abi=hard
        -T${CMAKE_SOURCE_DIR}/STM32F446RETx_FLASH.ld
        --specs=nano.specs
        -Wl,--whole-archive libhal.a -Wl,--no-whole-archive
        -Wl,--gc-sections
        )

This does result in adding the libhal.a twice during the linker step. This is unavoidable at the moment because CMake does not provide a mechanism for setting individual static library link options. The link options can only be specified at a global level.

This problem caused me to rack my brains for days. Because I could not figure out why it would not link against a function symbol that was obviously present in the static library I was trying to link against. I'm glad someone hinted at this behavior in a StackOverflow post. I would have struggled with this for many more days if they had not mentioned what was happening.

FFMPEG RTMP Stream Duplication

Wed, 30 Sep 2020 00:00:00 GMT

Desired Outcome

I recently ran into a situation where I needed to stream a video source to two destinations. One destination was for YouTube streaming, and the other destination was a local video display server to duplicate the video display across the building.

Typically you would use a service like Restream to achomplish this goal. Restream allows you to setup a stream to a single RTMP server which then forwards the stream to a variety of other streaming destinations like YouTube, Facebook, Twitch, etc...

The problem with restream is that it only supported streaming to external destinations and would not work to restream the the local network.

NGINX

One populare way to achieve this outcome of taking 1 RTMP stream and duplicating the stream to multiple destinations is using NGINX. NGINX is a fully featured web searver that can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache. One of it's third party modules adds support for RTMP. So to get the desired outcome you would host a NGINX web server instance, and then in the configuration file you would indicate the destinations of the duplicated stream. This is a lot of work for a seemingly simple task of duplicating a RTMP stream. It may be more reliable, but there is a simpler way using another program.

FFMPEG

This is where ffmpeg comes into play. ffmpeg is a complete, cross-fortform solution to record, convert and stream audio and video. Originally I though ffmpeg was just used for transcoding video and audio from one format to another format. I recently learned that ffmpeg is very versitile with it's inputs and destinations. The destination can be a file or in our case a RTMP server.

In this use case we would like ffmpeg to gather video from a RTMP server as an input and then forward that input to 2 RTMP servers. This is as simple as setting up 2 destinations from mpeg instead of using just one.

Here is the command used to stream to YouTube and also to a RTMP server on the local network.

ffmpeg -listen 1 -i rtmp://127.0.0.1:1935 \
    -f flv -c copy rtmp://a.rtmp.youtube.com/live2/aaaa-aaaa-aaaa-aaaa-aaaa
    -f flv -c copy rtmp://192.168.1.50/bbbb-bbbb-bbbb-bbbb-bbbb

-listen 1 -i rtmp://127.0.0.1:1935 This indicates that ffmpeg should host a RTMP server for devices to be able to stream to. This data is then ingested by ffmpeg and then redirected to other streaming destinations.

-f flv -c copy rtmp://a.rtmp.youtube.com/live/aaaa-aaaa-aaaa-aaaa-aaaa This takes the input video and then copies the input data to the destination rtmp server. In this case the data is streamed to the YouTube server. The last aaaa bit of the url is YouTube's stream key so that the user is authenticated to the YouTube channel. The -c copy field indicates that the data should just be copied and no other operations will be performed. ffmpeg has the flexibility to perform many operations on the source video/audio before transmitting the data to it's destination. Server operations would be to change the container format or change the video bitrate to something smaller which is more suitable for streaming.

Tailscale Hosts

Tue, 19 May 2020 00:00:00 GMT

Tailscale Hosts

Last post I talked about how to set up a mesh network of WireGuard peers using a service called Tailscale. One of my only complaints is that sometimes it's hard to keep track of the IP address assigned to all the peers. Tailscale automatically keeps track of all the connected peers with their IP address and hostname. This is where my program tailscale-hosts comes in. It parses the output of the tailscale status --json command append the hostname and IP address to the /etc/hosts for easy access. Here is an example of what information is included in the tailscale status command.

tailscale status
[ERwrT] linux   100.125.175.83  serenity           .
[MNw8B] linux   100.101.102.103 hello.ipn.dev      ...
[Ws5E9] windows 100.123.119.83  prometheus         ...
[ZZbvo] linux   100.76.104.117  awing              ...

The tailscale status command has all the information we need to be able to start using the human readable peer names. Here you can see that all my PC are named after science fiction starship. Below is an example of how the tailscale-hosts command works.

# here we can see that we are not able to connect to the peer by it's hostname
ping awing
ping: awing: Name or service not known

# Lets install the tailscale-hosts package
pip install git+https://www.gitlab.com/jhamberg/tailscale-hosts.git

# Update the hosts file with the tailscale peers
tailscale_hosts update

# Ping now successfully works with tailscale peer hostname
ping awing
PING awing (100.76.104.117) 56(84) bytes of data.
64 bytes from awing (100.76.104.117): icmp_seq=1 ttl=64 time=2.51 ms
64 bytes from awing (100.76.104.117): icmp_seq=2 ttl=64 time=1.73 ms

# Remove the tailscale peer hostnames from the hosts file.
tailscale_hosts remove

# Now we can see that the hostname no longer works.
ping awing
ping: awing: Name or service not known

Right now only GNU/Linux is supported, because that's the only platform that has a tailscale client that programatically exposes the peers hostnames. Eventually the tailscale-hosts package will get put on pip to make the instalation to new computers easier.

Conclusion

tailscale-hosts makes it very simple to add human readable hostnames to access remote peers instead of hard coded IP addresses. Which reduces the mental load needed to connect to these services.

Easy WireGuard setup using TailScale

Sat, 16 May 2020 00:00:00 GMT

WireGuard

I'm pretty sure that most people have heard about the networking tool called WireGuard. It's become very popular in the past couple of years for good reason. It is dead simple and only supports 1 encryption protocol. This prevents security flaws by accidentally choosing the wrong encryption parameters, which is easy to do if the tool you are using offers many options. It easily meets or exceeded the throughput of all other available VPN solutions. It is also build in to the Linux Kernel as of version 5.6 so there is a likely change that you already have access to it on your computer. These are just the basic. If you want to learn more I would suggest a visit to their website which explains things much better then I can.

WireGuard has a very simple interface to setup compared to all the other VPN's I've tried using in the past. Some parts can be considered tedious though. For every client a new public/private key pair has to be generated. Then for every preexisting WireGuard client the newly generated public key had to be manually added to the config file. This was very time consuming and error prone if one of the other WireGuard endpoints was mis-configured. Also for every new WireGuard endpoint a static IP had to be assigned which could have the possibility of colliding with a preexisting WireGuard endpoint.

tailscale

This is where a new technology called tailscale comes in. It allows a user to create a tailscale account using a Google or Microsoft account. This account is then used to manage a network of tailscale clients which are essentially just WireGuard clients that can be used to communicate with all endpoints in the tailscale network.

Using tailscale removes a lot of the pain points of using WireGuard that I've experienced in the past. Tailscale sets up the following thing automatically for WireGuard.

Public/Private key pairs that are automatically synced between endpoints in the tailscale network
Automatic key rotation which minimizes damage done by leaked private key
Automatic firewall traversal using NAT Punching with STUN and ICE
If firewall blocks WireGuard traffic a relay server called DERP is used to bypass firewall

Setting up a new tailscale client is as easy as downloading the executable and following the instructions to log on to your tailscale account and then everything else is automatically taken care of. tailscale then provision a IP for the new client and displays the IPs of all the other clients in the tailscale network.

jhamberg@falcon ~> tailscale status
[ERwrT] linux   100.xxx.xxx.xxA  serenity           ...
[MNw8B] linux   100.101.102.103 hello.ipn.dev       ...
[Ws5E9] windows 100.xxx.xxx.xxB  prometheus         ...
[ZZbvo] linux   100.xxx.xxx.xxC  awing              ...

Here you can see all of my endpoints are accessible by the IP listed in the tailscale status command. Tailscale is cross platform so you can have endpoints on Windows, Linux, Mac, iOS and soon Android.

I've glossed over some of the more advanced features of tailscale which include an audit trail for resource access. tailscale also has a very detailed Access Control List (ACLs) which allows defined rules to determine which endpoints have access to other endpoints. This would be very useful for a larger network that contains many different compute resources.

Conclusion

Tailscale is the tool that I've wanted for a long time that fixes a lot of the paint points I've had with WireGuard in the past. Tailscale builds of off WireGuard and makes a network of interconnected clients a breeze to setup. Right now it's free for individual users which has everything that I could need.

Added STM32F families to stm32-meson

Mon, 30 Mar 2020 00:00:00 GMT

The stm32-meson project now supports more STM32 chips. Previously the the project only supported STM32G0 family of ST microprocessors. I've gone through and added support for STM32F0, STM32F1, STM32F2, STM32F3, and STM32F4 families. In total 321 ST chips have been added in this update.

To build the stm32-blinky project run the following syntax. The cross file must be specified which includes the proper arguments to supply to the Arm GCC compiler. The cross file also contains a list of all the supported STM32 chips in the selected ST family. The Cube directory must also be specified in order for stm32-meson to find all the HAL libraries.

cd stm32-meson
meson build --cross-file stm32-meson/stm32f1.build -Dstm32_chip=STM32F100RE -Dstm32_cube_dir=$HOME/STM32Cube/Repository/STM32Cube_FW_F1_V1.8.0
ninja -C build

I've also added tests to all the ST chips that have been added. This way it can be easily determined if any of the individual ST chip are broken with any of the other additions to the stm32-meson project. The STM32 Cube HAL libraries have been hard coded to the default location used by STM32CubeMX software.

cd stm32-meson
sh scripts/test_all.sh

There is a slight disclaimer that the stm32-blink project may not work on all development boards. I have not verified that the output GPIO and timer are connected to a LED on all the development boards. The project in it's current state is more meant to be a boiler plate project to build upon for another project.

This project is meant to expedite STM32 development by being able to start from a preexisting project.

Dotfile Managment using dotdrop

Wed, 25 Mar 2020 00:00:00 GMT

Every developer has a customized system that they are familiar with to make coding easier. For a Linux system this involves modifying the dotfiles for all the software being used on their system. One of the dis-advantages of all these dotfiles is that they take a long time to get setup on a system. So switching between different PC's can be a bit of a time sync to get everything setup to your liking.

There are several options available to sharing these files between PC's, but they are often cumbersome and finicky and still require a lot of work to get set up. My previous method was to include all my dot files in a git repository. This worked fairly well, but felt a bit hacky since my home directory was now a git repository and every once in a while conflicted with the normal operation of my system.

The solution I've ended up moving to is a program called dotdrop. Dotdrop is a program used to save your dotfiles once, and deploy them everywhere. It allows for customization per system. In my case I can distribute the required dotfiles on my Windows PC, without cluttering my home directory with all the dotfiles for programs I'm not using.

Setup is really easy now as well. Dotdrop is written in python and is available from the pip repository which allows for easy setup. All that is required is cloning my git repository with all my dotdrop configuration and run the following commands.

pip3 install dotdrop
cd ~/.config
git clone https://gitlab.com/jhamberg/dotdrop.git
dotdrop install -p linux

Meson Python Modules

Mon, 24 Feb 2020 00:00:00 GMT

The meson build system is very versatile and can be integrated with a wide variety of technology. Meson can be used to build C modules that link in as a python module. Sometimes this is required to allow Python to integrate with a system library written in C. Let start by creating a Python module called mymath which implements a function called add written in C. This library contains all the definitions for the methods to interface with the extension module. You can see how it looks exactly like regular python code, except the mymath.add() function is implemented in C. Here is how the module would be used in Python. You can see that it looks exactly like any other regular Python code.

#!/usr/bin/env python3
import mymath
import sys

result = mymath.add(5, 3)
print('5 + 3 =', result)

Now lets look at the C file required to define the Python module. All the function that are required to interface with Python are included from a system header file called "Python.h"

// This includes all the requried Python definitions.
#include <Python.h>

// Definition of the add function in the external module interface.
static PyObject* add(PyObject *self, PyObject *args) {
    long long a, b, result;
    // PyArg_ParseTuple is a var args type function that behaves
    // similarlly to scanf by passing in a format string and the
    // pointers to the variables that are requsted.
    if(!PyArg_ParseTuple(args, "LL", &a, &b))
        return NULL;

    // Calculate the result of the add function.
    result = a + b;

    // Use Python's helper function to convert long long to
    // python return type.
    return PyLong_FromLong(result);
}

// Array that contains all the methods defined by this externel module.
static PyMethodDef MymathMethods[] = {
    {"add",  add, METH_VARARGS,
     "Add two numbers."},
    {NULL, NULL, 0, NULL}
};

// Structure that contains the definition of the external module.
static struct PyModuleDef mymathmodule = {
   PyModuleDef_HEAD_INIT,
   "mymath",
   NULL,
   -1,
   MymathMethods
};

// Function that gets called on load.  This creates a module with
// the PyModule_Create function.
PyMODINIT_FUNC PyInit_mymath(void) {
    return PyModule_Create(&mymathmodule);
}

Now all we need is the meson.build project to build the C extension module.

# Create normal meson project.
project('Mymath python extension', 'c',
  default_options : ['buildtype=release'])

# Import python3 meson module which is used to find the Python dependencies.
py3_mod = import('python3')
# Locate the python executable.
py3 = py3_mod.find_python()
# Create the Meson python3 dependency from the python3 module.
py3_dep = dependency('python3', required : false)

if py3_dep.found()
  # Create the external C module using the python3 module helper function.
  pylib = py3_mod.extension_module('mymath',
    'mymath_module.c',
    dependencies : py3_dep,
  )

  # Pathdir contains the dynamic library module.
  pypathdir = meson.current_build_dir()

  # Create a test script that runs a python script that uses
  # the C external module.  Must populate PYTHONPATH to include
  # a path that includes the mymath.so dynamic library.
  test('extmod',
    py3,
    args : files('mymath_test.py'),
    env : ['PYTHONPATH=' + pypathdir])

  # Check we can apply a version constraint
  # dependency('python3', version: '>=@0@'.format(py3_dep.version()))
else
  error('MESON_SKIP_TEST: Python3 libraries not found, skipping test.')
endif

Let's try out building the external python module using meson. This should start looking familiar by now if you've worked with Meson at all. Meson can be used to run unit tests defined for the project. By default the output of the unit test is not visible on the command line. If the -v argument is passed to the meson test command then the output of the unit test will be visible on the console.


$ meson build
$ ninja -C build
$ cd build
$ meson test -v
...
5 + 3 = 8
...

$ env PYTHONPATH=build python3 mymath_test.py
5 + 3 = 8

If you don't mind writing a bit of C code, extending Python can be a useful way to integrate existing C code in a very Pythonic way. Meson does a great job of finding all the dependencies and linking them together in a easy to use way.

Meson Polyglot - Rust and C

Wed, 12 Feb 2020 00:00:00 GMT

The definition of polyglot is "knowing or using several languages." With that definition Meson could be considered a polyglot. Meson is able to work with multiple languages. Currently c, cpp, d, objc, objcpp, fortran, java, cs, vala and rust are supported by Meson.

Rust is a language that emphasize on memory safety and speed at the same time. Let's start with Rust's version of Hello World! and see about compiling it using meson.

fn main() {
    println!("Hello from Rust!");
}

Let's setup meson to compile this rust file. The only different between the rust meson.build and a c meson.build is that rust is listed as one of the project supported languages and the executable source is a .rs file instead of a .c file. Meson is able to handle the rest.

project('rust_project', 'rust',
       version : '0.1',
       default_options : ['warning_level=3'])

executable('rust_exe', 'main.rs')

Now let's go ahead and create the build files just like we normally would.

$ meson build
$ ninja -C build
$ ./build/rust_exe
Hello from Rust!

Sometimes if you are using C code it can be helpful to use code that was written in Rust. We can do this by linking a rust library into a c project.

Lets say we have a library that defines some math functions. Like the add function which takes in two parameters and returns the sum of the two parameters. Here is the Rust file that implements this math function.

// This is required to prevent name mangling which exporting symbols.
#[no_mangle]

// Extern is required to make the symbol visibly from the library
pub extern fn add(a: i32, b: i32) -> i32 {
    a + b
}

Now lets create a c file that uses the add function that is defined in the Rust lib.rs file.

// Used for printf()
#include <stdio.h>

// Function prototype, so the compiler doesn't complain.
int add(int a, int b);

int main() {
    // Print the result of 5 + 3 calculated in Rust
    printf("5 + 3 = %d\n", add(5, 3));
}

Here's the meson.build file that is used to link the two languages together.

# Notice that both c and rust are listed as project languages.
project('rust_project', ['c', 'rust'],
  version : '0.1',
  default_options : ['warning_level=3'])

# Create a rust static library.
# Must specify rust_crate_type otherwise the generated output
# won't be compatibily with the c linker.
rust_add_lib = static_library('add_rust', 'lib.rs', rust_crate_type : 'staticlib')

# Create a list of dependencies required by Rust.
# The dl library is required by Rust to link the library.
deps = [
  meson.get_compiler('c').find_library('dl', required: false),
  dependency('threads'),
]

# Create a c executable that links with a rust library.
# Specify the link dependency as you would with any other library.
executable('link_with_rust', 'link_with_rust.c', link_with: rust_add_lib, dependencies: deps)

Let's run the newly compiled program.

# Here's the output of the program.
$ ./build/
5 + 3 = 8

Now Lets see if we can do the reverse operation and call c code from rust. Sometimes it can be useful to call code that is written in C from a Rust program. This can be used to gradually convert a program to Rust by converting small sections of code to Rust at a time. Lets compile the same example of a math library. Here is the add function, but written in the C language this time. The function is very simple. It just adds a and b and returns the result. Because C is used as the standard for external symbol names there is nothing special that needs to be done to allow the c code to be called from an external program.

// A dirt simple function to add two numbers in C.
int add(int a, int b)
{
    return a + b;
}

Here is the rust code that can be used to call the C library function.

// The name of the library is required to find the external function.
#[link(name = "add_c")]

// Provides the function prototype to the rust program.
extern {
    fn add(a: i32, b: i32) -> i32;
}

// Use the math library from the main function of the Rust program.
fn main() {
    // Unsafe keyword is required because C is not gaurenteed
    // to follow all the memory safty rules that Rust does.
    unsafe {
        println!("5 + 3 = {}", add(5, 3))
    }
}

Here is the meson.build glue that is required to build the following code.

# Create a project that uses the C and rust languages.
project('rust_project', ['c', 'rust'],
  version : '0.1',
  default_options : ['warning_level=3'])

# Create a c static library.
c_add_lib = static_library('add_c', 'lib.c')

# Create a rust executable that links with a c library.
executable('link_with_c', 'link_with_c.rs', link_with: c_add_lib)

These are the basics of linking Rust with C and C with Rust. All of these examples have been found from test cases directory of the Meson source code. They have been very valuable in understanding all the features of the meson build system.

stm32-meson

Thu, 30 Jan 2020 00:00:00 GMT

STM32 Meson

Ive been a big fan of stm32-cmake which simplifies the setup of a stm32 embedded project. All that is required is the STMCubeMX directory and the chip being used. All the rest is automatically generated by stm32-cmake. I've recently learned about meson and have decided to create the stm32-meson project as a way to learn how to get something useful done is meson.

First I'll show the expected usage of the project.

jhamberg@serenity ~/s/stm32-meson> meson build \
-Dstm32_cube_dir=$HOME/STM32Cube/Repository/STM32Cube_FW_G0_V1.3.0/ \
-Dstm32_chip=STM32G071RB \
--cross-file=stm32-meson/stm32.build

-Dstm32_cube_dir points to to the CubeMX directory that contains all the build files for the STM32G0 family. This directory contains all the HAL source files. The $HOME variable must be used because it doesn't currently seem that meson is able to expand the ~ character into the user's home directory.
-Dstm32_chip option specifies which STM chip is being used and will be used to configure the build system with the correct flash size and ram size.
--cross-fileThe cross-file must also be specified. This contains the name of the ARM compiler that will be used. It also contains all the proper linker arguments for compiling the project.

Here is the meson build code required to setup a simply "blinky" application.

# Normal setup of meson project.
project('stm32_meson', 'c',
  version : '0.1',
  default_options : ['warning_level=3'])

# Include the required modules to compile the projects.
stm32_hal_components = ['uart', 'uart_ex', 'dma', 'gpio', 'tim', 'tim_ex']

# Include to setup all the board dependencies.
subdir('stm32-meson')

# Define STM32 executable.
executable('stm32-blinky',
  'main.c',
  dependencies : [stm32_cube_dep, stm32_cmsis_dep]
  )

The stm32_hal_components variable contains a list of all the HAL components that are required by the project. This allows for shorter build times by only including the required HAL components. The stm32_chip can also be specified in the meson build script instead of being passed along at configuration time as well. All the custom setup logic code in included in the stm32-meson/meson.build script. The output of this script are two dependencies called stm32_cube_dep and stm32_cmsis_dep. stm32_cube_dep contains the linker script dependency and the HAL sources to be included into the stm32-blinky executable. The stm32_cmsis_dep is the dependency that includes the startup files for the particular STM32 chip and CMSIS includes which allow certain abstractions to be used across all ARM architectures.

Future Goals

My future goals for this project are to try to achieve as much feature parity with stm32-cmake as possible. Right now the project only supports the STM32G0 family and has been only tested with the STM32G071RB evaluation board. I would like to add support for other stm32 chip families.

Experience with Meson

One of the first things that took getting used to is that there are no functions. My first though was to create a meson function to setup all the dependencies for each executable by simply calling stm32_meson_setup_executable or something like that. Because I didn't have access to functions I had to do it the more canonical way which is actually better. I setup a dependency which is included by the executable that contains all the supporting resources. I am still trying to figure out a way to easily setup download targets for executables. It would be really convenient to have a stm32_meson_setup_download_target to automatically, but I haven't yet figured out an easy way to do this without forcing the user to manually set this up for each executable.

Coming from a background of CMake, the meson language is pleasant to work with. All the data structures and simplicity of the language is very much appreciated. It did take a little getting used to the way things are done in CMake. I ran into a couple of things that I thought should have been easier, but had to find a workaround. This is to be expected for learning a new technology.

RTL-SDR basic setup

Sat, 11 Jan 2020 00:00:00 GMT

What is a Software Defined Radio (SDR)

I recently got my hands on a RTL-SDR which is a software defined radio (SDR). A software-defined radio (SDR) is a radio communication system where components that have been traditionally implemented in hardware (e.g. mixers, filters, amplifiers, modulators/demodulators, detectors, etc.) are instead implemented by a means of software on a personal computer or embedded system.

In recent years SDRs have become very affordable using commedy hardware. SDRs used to only be available to academics and industrial applications in the past. I have chosen the RTL-SDR which is a very modest $20 dollars for just the SDR and $30 including extension cables and multiple different antennas.

SDRs can be used to decode a miriad of protocols. Here's a short list of the possibilities.

AM/FM radio
Iridium satallite tracking
Aircraft tracking beacons
HAM radio
lighting detection

Driver Setup

In order to use the RTL-SDR on your system the drivers must be installed. These instructions assume the suer is running a flavor of Linux. Make sure the following applications are installed.

git, cmake, build-essential, and libusb-1-0-0-dev

# Clone SDR-RTL linux drivers.
git clone git://git.osmocom.org/rtl-sdr.git
cd rtl-sdr/
mkdir build
cd build
cmake ../ -DINSTALL_UDUV_RULES=ON
make
sudo make install

# On my Manjaro system /usr/local/lib is not in the ldconfig path, so the following command fixed the issue
echo "/usr/local/lib" | sudo tee -a /etc/ld.so.conf

# Add the shared libraries into the system cache.
sudo ldconfig

# Copy the udev rules which are used to recognized the RTL-SDR USB device.
sudo cp ../rtl-sdr.rules /etc/udev/rules.d/

# This command allows the device to be recognized without restarting your computer.
# Although if it doesn't work restarting may be required to get the device to be recognized.
udevadm control --reload-rules && udevadm trigger

These instructions allow the SDR-RTL to be accessed from the system using the library that was just installed. This allows other applications to access the SDR.

There is one more step required to get the RTL-SDR to work properly on the system. Currently the original drivers are being used for the RTL-SDR dongle which will conflict with the new drivers that were just installed. To fix this the original drivers need to be blacklisted, so that our new correct drivers are used for the RTL-SDR.

# Blacklist instructions
# Navigate to the /etc/modprobe.d directory
cd /etc/modprobe.d
# Create new file called blacklist-rtl.conf
# Add this one line.
# blacklist dvb_usb_rtl28xxu
echo "blacklist dvb_usb_rtl28xxu" | sudo tee -a /etc/modprobe.d/blacklist-rtl.conf

The RTL-SDR dongle can now be tested by the rtl_test executable. The prompt below shows the expected output of the test executable. The system may have to be restarted after blacklisting the drivers in order to get the correct drivers loaded.


jhamberg@falcon /e/ld.so.conf.d> rtl_test -t
Found 1 device(s):
  0:  Realtek, RTL2838UHIDIR, SN: 00000001

Using device 0: Generic RTL2832U OEM
Found Rafael Micro R820T tuner
Supported gain values (29): 0.0 0.9 1.4 2.7 3.7 7.7 8.7 12.5 14.4 15.7 16.6 19.7 20.7 22.9 25.4 28.0 29.7 32.8 33.8 36.4 37.2 38.6 40.2 42.1 43.4 43.9 44.5 48.0 49.6
[R82XX] PLL not locked!
Sampling at 2048000 S/s.
No E4000 tuner found, aborting.

Cubic SDR (FM Radio)

There are multiple applications that can now take advantage of the RTL-SDR cababilities. One of the first one's I tried is called CubicSDR. It has the ability to decode multiple different kinds of radio protocols, but I am currently using it to listen to FM radio. There is an excellent Wikipedia article explaining how FM radio works.

https://en.wikipedia.org/wiki/FM_broadcasting

In America the frequency range from 87.5 MHZ to 108.0 MHZ is used. FM stands for frequency modulation which describes the modulation technique used to encode audio into radio waves to transmit over the air. Frequency modulation encodes the time series amplitude values of the audio in a way that is proportional to the frequency being transmitted. FM radio has a bandwidth of 200 kHz which means if a radio station is transmitting at 89. 5 MHz it is actually transmitting at a range of 89.4 MHz to 89.6 MHz.

When using the Cubic SDR application to listen to FM radio, choose the center frequency that aligns with the desired radio station. For FM radio 89.5 select 89,500,000 as the center frequency and 200 kHz as the bandwidth. The Cubic SDR application will decode this frequency into the audio that is being transmitted from station's radio tower.

Here is an example of what the CubicSDR application looks like when listening to a FM radio station.

Here is a 30 second audio sample from my favorite EDM station in Seattle, C89.5 that was captured from my apartment. Happy listening!

Automatic CMake Header Dependency

Fri, 27 Dec 2019 00:00:00 GMT

Often times 3rd party dependencies can be the hardest part of setting up a C++ project. CMake has eased this pain, but it is still often quite difficult to deal with with the numerous different types of build system. CMake includes the FetchContent library call which allows a dependency to either be cloned from a git repository or tar.gz archive to be included into the project. This is a very convenient feature which allows your projects dependencies to be neatly downloaded without having to include the source in your project. This has the disadvantage of having to clone the whole git repository which can take a long time and sometimes requires a large amount of disk space.

In my case the JSON for Modern C++ was taking 8 seconds to clone and took up 439 MB of space on my hard drive. JSON for Modern C++ is known to be a heavyweight for a JSON library, but it seemed like just too much for a library that hasto parse a little JSON. Fortunately JSON for C++ provides a link to the header only version of the library on their GitHub releases page.

CMake has a feature to download files from the Internet as part of the generation process. So basically we are going to use CMake to automatically download the single header file to satisfy the JSON for Modern C++ dependency instead of having to clone the entire repository.

the file(DOWNLOAD) CMake command can be used to download files from a URL to a directory. I've chosen the ${CMAKE_CURRENT_BINARY_DIR} since the header is a dependency and not actually considered part of the source of this project. A interface library can be added which allows the library to be recognized by CMake. The interface part is required because the library is a header only library and does not contain any source files. The target_include_directories() command is used to specify the include path for the library that has just been downloaded. Once this is done then the target_link_libraries can be used to "link" the header only library to whatever CMake targets that are available.

In this example I choose to download Catch2 unit testing framework and the JSON for Modern C++ framework in a minimal example. The CMakeLists.txt comes in at a 16 lines of code which is pretty short for automatically downloading the project dependencies and making them available to the rest of the CMake project.

# Project preamble
cmake_minimum_required(VERSION 3.15)
project(main)

# Install catch dependency.
file(DOWNLOAD https://github.com/catchorg/Catch2/releases/download/v2.11.0/catch.hpp ${CMAKE_CURRENT_BINARY_DIR}/catch.hpp)
add_library(catch INTERFACE)
target_include_directories(catch INTERFACE ${CMAKE_CURRENT_BINARY_DIR})

# Install json dependency.
file(DOWNLOAD https://github.com/nlohmann/json/releases/download/v3.7.3/json.hpp ${CMAKE_CURRENT_BINARY_DIR}/json.hpp)
add_library(json INTERFACE)
target_include_directories(json INTERFACE ${CMAKE_CURRENT_BINARY_DIR})

# Add main executable.
add_executable(main main.cpp)

# Link 3rd party dependencies.
target_link_libraries(main json catch)

#define CATCH_CONFIG_MAIN
#include <catch.hpp>
#include <json.hpp>

TEST_CASE("Factorials are computed", "[factorial]") {
    nlohmann::json j = "{ \"happy\": true, \"pi\": 3.141 }"_json;
    CHECK(j["happy"].get<bool>());
}

PC / Desk Setup

Wed, 25 Dec 2019 00:00:00 GMT

I spend a lot of time at my PC so I've put a lot of though into the parts that make the experience a pleasent one. I have a modest PC I build back in 2015 when I was still in high school. It's served me quite well over the years. Here are the major components that make up my system. I'm also planning on updating this post whenever I add new equipment to my desk setup.

ErgoDox EZ

I'm a self proclaimed keyboard enthusiast. I first discovered the benefits of a mechanical keyboard with my purchase of a Corsair K70. This was a modest entry to the mechanical keyboard world with Cherry MX Brown switches. Part of my journey to using mechanical keyboards has involved a detour into learning the Dvorak key layout. Dvorak positions the most commonly used keys on the home row, and places the less common keys in harder to reach areas of the keyboard. This means that it's easier on average to type words as the more common letters are easier to access. This decision has not necessarily made my typing faster, but it dose reduce the usage of some of the more awkward to reach button presses as often.

My current keyboard setup at the office is an ErgoDox EZ. This is a split keyboard with Cherry MX brown switches. I chose the key caps with no labels on them since they wouldn't line up anyways using the Dvorak layout. One of the best things about this keyboard is that it has ortho-linear keys. This means that the keys are not staggered and are lined straight up and down for each finger. It is shifted slightly to match the length of each individual finger. I cannot stand typing on regular keyboards anymore because of the staggered keys.

One thing that did take a while to get used to was the number of keys on the keyboard. Coming from a 10 key keyboard which has approximately 104 keys whereas the ErgoDox EZ only has 76 keys available. The ErgoDox EZ allows keys to be put on different layers that are accessible by pressing a key combination to make the hidden keys available.

Desk

I decided to "make" my own desk for home. It's really not that much work and I'm really happy with the outcome. I was going for the simple look so it's simply a piece of Acacia wood butcher block from Home Depot. I also wanted to try out the standing desk so I got adjustable legs which allow the desk to be raised to a standing position and lowered to a sitting position. The VIVO desk stand has a manual adjustment which reduced the prices by a couple hundred dollars over the electric motor version.

I splurged and bought two Dell 27 in 2560x1440 monitors. I got lucky and there was a sale where they were each $100 dollars off their regular price on Amazon. There is nothing better than haveing two identical monitors for the screen realistate it provides. The color profile and IPS viewing angles on these monitors is amazing. They are slightly lack-luster for gaming since they have a slow refresh rate of 60 Hz. This is not a problem for me though since I don't play any competitive games.

Optical Drive

I have a Plex Media Server setup at home and like to rip media from DVD and Blue-Ray discs to be viewed from my home server. MakeMKV is an excellent program to rip media from both DVD and Blu-Ray discs. Ripping Blu-Ray discs is only available if the software is purchased or used in a temporary BETA mode. The trick is that a BETA key is always provided in the MakeMKV fourm. I believe this is a way to get around licensing issues that come with providing a free Blu-Ray decoder. MakeMKV rips the DVD in a lossless format that can then be trans-coded into a more efficient into a more efficient codec like HVEC x265 using HandBrake.

Raspberry Pi IR Receiver

Thu, 12 Dec 2019 00:00:00 GMT

I recently got my hands on a IR Receiver and Transmitter module from a ELEGOO Upgraded 37 in 1 Sensor Modules Kit V2.0. I wanted to learn how the technology works and figure how to the IR transmitter with my Raspberry Pi 4.

EEVblog has an excellent video describing how the IR transmitter and receiver work in excellent detail. EEVblog #506 - IR Remote Control Arduino Protocol Tutorial

Signal Basics

The IR transmitter works by using OOK (on off keying) to modulate a carrier frequency. The carrier frequency is generated by producing a 32 kHz square wave that is transmitted by turning the transmitter LED on and off. A logical 1 is present when the carrier frequency is present, this is called a pulse. A logical 0 is present when the carrier frequency is absent, this is called a space. Data bits are encoded by a pulse of 600 us followed by either a space of 1600 us or 600 us. If a space of 1600 us follows the pulse then this is considered a 1. If a space of 600 us follows the pulse then this is considered a 0.

The key press from the IR transmitter has three parts: the header, pre-data, and payload. The header consists of a 9000 us pulse followed by a space of 4500 us. The pre-data is made up of 16 data bits that construct a value of 0x00ff. The payload is made up of 16 bits of payload data. This data is different for every button on the transmitter.

Raspberry Pi Setup

First the IR driver must be enabled. Un-comment the following line in the file /boot/config.txt

dtoverlay=gpio-ir,gpio_pin=17

Then make sure to install the lirc package. The Linux Infrared Remote Control. This is used to provide user space access to the IR driver and allow for easy integration into desktop tools.

The tool provides a debug executable to see raw data coming in from the infrared receiver. This program is called mode2 and output the values for all the pulses and spaces the driver receives.

mode2
...
pulse 9111
space 4438
pulse 637
space 521
pulse 605
space 527
pulse 582
space 549
pulse 581
space 551
pulse 577
space 555
pulse 581
space 550
pulse 605
space 527
pulse 579
space 553
pulse 605
...

LIRC provides a tool called irrecord which is used to make a configuration file for a remote transmitted and map all the payload values to a specific key value. This program unfortunately did not work for the several IR remotes I had lying around the apartment.

I instead created a python program to parse the output from mode2 to generate the config file. Here is the output of the python script. It decodes the payload for each button press.

Now that I have the payload of each key press from the IR transmitter I can create the configuartion script that lirc needs to process the raw payload into events that the rest of the OS can listen for. I copied a configuration script from another remote and modified the payload to match the keys. Here is the result.

begin remote

  name elegoo
  bits           16
  flags SPACE_ENC|CONST_LENGTH
  eps            30
  aeps          100

  header       9008  4466
  one           598  1645
  zero          598   550
  ptrail        611
  repeat       9005  2204
  pre_data_bits   16
  pre_data       0x00ff
  gap          107536
  toggle_bit      0


      begin codes
          KEY_POWER                0x000000000000a25d
          KEY_VOLUMEUP             0x000000000000629d
          KEY_STOP                 0x000000000000e21d
          KEY_PREVIOUS             0x00000000000022dd
          KEY_PLAYPAUSE            0x00000000000002fd
          KEY_NEXT                 0x000000000000c23d
          KEY_DOWN                 0x000000000000e01f
          KEY_VOLUMEDOWN           0x000000000000a857
          KEY_UP                   0x000000000000906f
          KEY_EQUAL                0x0000000000009867
          BTN_START                0x000000000000b04f
          KEY_1                    0x00000000000030cf
          KEY_2                    0x00000000000018e7
          KEY_3                    0x0000000000007a85
          KEY_4                    0x00000000000010ef
          KEY_5                    0x00000000000038c7
          KEY_6                    0x0000000000005aa5
          KEY_7                    0x00000000000042bd
          KEY_8                    0x0000000000004ab5
          KEY_9                    0x00000000000052ad
          KEY_0                    0x0000000000006897
      end codes

end remote

This config file then just needs to be copied to /etc/lirc/lircd.conf.d with a file name of elegoo.lircd.conf and then the irw executable can be used to process key press events.

irw
0000000000ff30cf 00 KEY_1 elegoo
0000000000ff30cf 01 KEY_1 elegoo
0000000000ff30cf 02 KEY_1 elegoo
0000000000ff30cf 03 KEY_1 elegoo
0000000000ff38c7 00 KEY_5 elegoo
0000000000ff38c7 01 KEY_5 elegoo
0000000000ff38c7 02 KEY_5 elegoo
0000000000ff5aa5 00 KEY_6 elegoo
0000000000ff5aa5 01 KEY_6 elegoo

Vim Plug Autoload

Tue, 24 Sep 2019 00:41:25 GMT

This post is about how to write some vim-script to autoload the plugin manager from the .vimrc. This makes the process of getting up and running on a new system easier because the plugin manager is automatically downloaded as well as the plugins are also automatically downloaded.

Using the commands from https://github.com/junegunn/vim-plug I was able to create a script to auto download VimPlug for vim and NeoVim on windows and on Linux.

" Expand the home directory to an absolute path.
let homeDir = expand('~')

" Find the desired VimPlug install location for different system configurations.
if(has('win32') || has('win64'))
    if has('nvim')
        let shareDir=homeDir.'\AppData\Local\nvim'
        let plugVim=shareDir.'\autoload\plug.vim'
    else
        let shareDir=homeDir.'\vimfiles'
        let plugVim=shareDir.'\autoload\plug.vim'
    endif
else
    if has('nvim')
        let shareDir=homeDir.'/.local/share/nvim/site'
        let plugVim=shareDir.'/autoload/plug.vim'
    else
        let shareDir=homeDir.'/.vim'
        let plugVim=shareDir.'/autoload/plug.vim'
    endif
endif

" Url of the VimPlug script.
let plugUri = 'https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim'
if empty(glob(expand(plugVim)))
    if has('win32') || has('win64')
    	" Make sure the autoload directory has been created.
        exec '!md '.shareDir.'\autoload'

        " Download VimPlug using PowerSHell.
        exec '!powershell -command Invoke-WebRequest -Uri "'.plugUri.'" -OutFile '.plugVim.'"'
    else
        " Download VimPlug using curl.
        exec '!curl -fLo '.plugVim.' --create-dirs '.plugUri
    endif

	# Automatically run PlugInstall command.
    autocmd VimEnter * PlugInstall --sync | source $MYVIMRC
endif

CMake Dependencies using Fetch Content

Sun, 27 Jan 2019 00:00:00 GMT

I recently have been wanted to play around a bit with learning the Vulkan graphics API. Before I was able to get started with that I needed a couple if libraries as dependencies to make things easier to work with cross platform.

GLFW is an Open Source, multi-platform library for OpenGL, OpenGL ES and Vulkan development on the desktop. It provides a simple API for creating windows, contexts and surfaces, receiving input and events.

It simply allows a window to be created on any supported platform that can be used to draw to using your graphics API of choice.

From the example website the following code can be used to draw a spinning triangle to the screen which is the hello world of graphics programming.

https://www.glfw.org/docs/latest/quick.html

#include <glad/glad.h>
#include <GLFW/glfw3.h>
#include "linmath.h"
#include <stdlib.h>
#include <stdio.h>
static const struct
{
    float x, y;
    float r, g, b;
} vertices[3] =
{
    { -0.6f, -0.4f, 1.f, 0.f, 0.f },
    {  0.6f, -0.4f, 0.f, 1.f, 0.f },
    {   0.f,  0.6f, 0.f, 0.f, 1.f }
};
static const char* vertex_shader_text =
"uniform mat4 MVP;\n"
"attribute vec3 vCol;\n"
"attribute vec2 vPos;\n"
"varying vec3 color;\n"
"void main()\n"
"{\n"
"    gl_Position = MVP * vec4(vPos, 0.0, 1.0);\n"
"    color = vCol;\n"
"}\n";
static const char* fragment_shader_text =
"varying vec3 color;\n"
"void main()\n"
"{\n"
"    gl_FragColor = vec4(color, 1.0);\n"
"}\n";
static void error_callback(int error, const char* description)
{
    fprintf(stderr, "Error: %s\n", description);
}
static void key_callback(GLFWwindow* window, int key, int scancode, int action, int mods)
{
    if (key == GLFW_KEY_ESCAPE && action == GLFW_PRESS)
        glfwSetWindowShouldClose(window, GLFW_TRUE);
}
int main(void)
{
    GLFWwindow* window;
    GLuint vertex_buffer, vertex_shader, fragment_shader, program;
    GLint mvp_location, vpos_location, vcol_location;
    glfwSetErrorCallback(error_callback);
    if (!glfwInit())
        exit(EXIT_FAILURE);
    glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 2);
    glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 0);
    window = glfwCreateWindow(640, 480, "Simple example", NULL, NULL);
    if (!window)
    {
        glfwTerminate();
        exit(EXIT_FAILURE);
    }
    glfwSetKeyCallback(window, key_callback);
    glfwMakeContextCurrent(window);
    gladLoadGLLoader((GLADloadproc) glfwGetProcAddress);
    glfwSwapInterval(1);
    // NOTE: OpenGL error checks have been omitted for brevity
    glGenBuffers(1, &vertex_buffer);
    glBindBuffer(GL_ARRAY_BUFFER, vertex_buffer);
    glBufferData(GL_ARRAY_BUFFER, sizeof(vertices), vertices, GL_STATIC_DRAW);
    vertex_shader = glCreateShader(GL_VERTEX_SHADER);
    glShaderSource(vertex_shader, 1, &vertex_shader_text, NULL);
    glCompileShader(vertex_shader);
    fragment_shader = glCreateShader(GL_FRAGMENT_SHADER);
    glShaderSource(fragment_shader, 1, &fragment_shader_text, NULL);
    glCompileShader(fragment_shader);
    program = glCreateProgram();
    glAttachShader(program, vertex_shader);
    glAttachShader(program, fragment_shader);
    glLinkProgram(program);
    mvp_location = glGetUniformLocation(program, "MVP");
    vpos_location = glGetAttribLocation(program, "vPos");
    vcol_location = glGetAttribLocation(program, "vCol");
    glEnableVertexAttribArray(vpos_location);
    glVertexAttribPointer(vpos_location, 2, GL_FLOAT, GL_FALSE,
                          sizeof(float) * 5, (void*) 0);
    glEnableVertexAttribArray(vcol_location);
    glVertexAttribPointer(vcol_location, 3, GL_FLOAT, GL_FALSE,
                          sizeof(float) * 5, (void*) (sizeof(float) * 2));
    while (!glfwWindowShouldClose(window))
    {
        float ratio;
        int width, height;
        mat4x4 m, p, mvp;
        glfwGetFramebufferSize(window, &width, &height);
        ratio = width / (float) height;
        glViewport(0, 0, width, height);
        glClear(GL_COLOR_BUFFER_BIT);
        mat4x4_identity(m);
        mat4x4_rotate_Z(m, m, (float) glfwGetTime());
        mat4x4_ortho(p, -ratio, ratio, -1.f, 1.f, 1.f, -1.f);
        mat4x4_mul(mvp, p, m);
        glUseProgram(program);
        glUniformMatrix4fv(mvp_location, 1, GL_FALSE, (const GLfloat*) mvp);
        glDrawArrays(GL_TRIANGLES, 0, 3);
        glfwSwapBuffers(window);
        glfwPollEvents();
    }
    glfwDestroyWindow(window);
    glfwTerminate();
    exit(EXIT_SUCCESS);
}

This library depends on two other external pieces of code. Something called glad which is generator for the OpenGL api calles and a loader. And also a linear algebra library called linmath.

As of now there are three external dependencies required to build this code.

glfw
glad
linmath

There are several common methods that could be used to download and build the external dependencies.

The dependencies could be manually downloaded and built manually by the programmer. This is very tedious and error prone and generally should not be done.
There dependencies could automaticlly be downloaded by a shell script simial to the process above, but automated. This is a little better but could be improved on by having better integration with CMake.
Packege Managers like Conon, VCPkg. These are nice systems, but they pull an additional step to the build process and require integrating with an additional piece of technology.

The method that I will propose is using FetchContent feature that is built into CMake. The fetch content is very much similar to the AddProject command which is already present in CMake, but with the distinction of being able configure the external dependencies at configuration time instead of build time. This allows for tighter integration with CMake because if the externalr dependenciy uses CMake our code may directly use the targets setup by the dependency.

cmake_minimum_required(VERSION 3.11)

include(FetchContent)

FetchContent_Declare(
        glfw
        GIT_REPOSITORY https://github.com/glfw/glfw
)

FetchContent_GetProperties(glfw)
if(NOT glfw_POPULATED)
    FetchContent_Populate(glfw)

    set(GLFW_BUILD_EXAMPLES OFF CACHE INTERNAL "Build the GLFW example programs")
    set(GLFW_BUILD_TESTS OFF CACHE INTERNAL "Build the GLFW test programs")
    set(GLFW_BUILD_DOCS OFF CACHE INTERNAL "Build the GLFW documentation")
    set(GLFW_INSTALL OFF CACHE INTERNAL "Generate installation target")

    add_subdirectory(${glfw_SOURCE_DIR} ${glfw_BINARY_DIR})
endif()

FetchContent_Declare(
        linmath
        GIT_REPOSITORY https://github.com/datenwolf/linmath.h.git
)

FetchContent_GetProperties(linmath)
if(NOT linmath_POPULATED)
    FetchContent_Populate(linmath)
    add_library(linmath INTERFACE)
    target_include_directories(linmath INTERFACE ${linmath_SOURCE_DIR})
endif()

FetchContent_Declare(
        glad
        GIT_REPOSITORY https://github.com/Dav1dde/glad.git
)

FetchContent_GetProperties(glad)
if(NOT glad_POPULATED)
    FetchContent_Populate(glad)
    set(GLAD_PROFILE "core" CACHE STRING "OpenGL profile")
    set(GLAD_API "gl=" CACHE STRING "API type/version pairs, like \"gl=3.2,gles=\", no version means latest")
    set(GLAD_GENERATOR "c" CACHE STRING "Language to generate the binding for")
    add_subdirectory(${glad_SOURCE_DIR} ${glad_BINARY_DIR})
endif()

add_executable(game
	src/main.cpp
	)

target_link_libraries(game glfw linmath glad)

Git Issue in Docker

Mon, 12 Nov 2018 00:00:00 GMT

Failed to Initialize Git Repo in Docker

http://blog.johngoulah.com/2016/03/running-strace-in-docker/

While working with Docker I was trying to automate the build of a project. The project automatically downloaded the source code for several dependencies using Git. When running this command on Linux everything worked flawlessly as expected. When I tested the same build setup on Docker for Windows on my PC things worked as expected there were no build issues.

Then as other people started to use the build script they ran into a weird error that I have never seen before that is shown below. The error message seems to indicate that somehow the git config file was incorrect suggesting that the core.filemode was unable to be set to false. I originally thought that there must be a file permission issue but the permissions were the same on both computers where one worked and the other didn't. And I noticed that files could still be created and written too so it wasn't a permission error. I thought the who point of docker and containers was to prevent issues like this and create a repeatable build environment.

error: invalid config file /data/wsl-directory/.git/config
fatal: could not set 'core.filemode' to 'false'

The first steps to debugging this was to find some way to reproduce the issue on my PC where everything worked as expected. My co-worker pointed my in the correct direction by demonstrating that when running the git init command in Docker in a directory that was created by WSL the same error message appeared that was the same as before. But if the folder was created in windows the git init command inside docker worked as expected.

To start debugging this issue I created a minimum reproducible example using the simple docker file below.

# Using Debian because it is small to download.
FROM debian

# Install git and strace which will be used later.
RUN apt-get update && apt-get install -y git strace

I created the file directory listed below.

docker-git
	windows-directory
	wsl-directory

# Build the Docker image and then run an interactive shell.
docker build -t debian-git .
docker run -v "%cd%":/data --security-opt seccomp:unconfined -it debian-git

# Run git init in the windows-directory created in Explorer.exe
root@872cca3eefb1:/ cd /data/windows-directory/                           root@872cca3eefb1:/data/windows-directory git init
Initialized empty Git repository in /data/windows-directory/.git/

# Run git init in the wsl-directory created in WSL.
root@872cca3eefb1:/data/windows-directory cd /data/wsl-directory/                                   root@872cca3eefb1:/data/wsl-directory git init
error: invalid config file /data/wsl-directory/.git/config
fatal: could not set 'core.filemode' to 'false'

I was trying to figure out why this error message was happening. All the Google search results seemed to be pointing to a different issue than I was having. They would talk about why you should use core.filemode not why the error was happening when the Git repo was being initialized. I needed to get a deeper understanding of what was causing the issue. I decided to give strace a try. strace is a utility that is used to log all the system calls that an executable is calling. For example it logs all file access, read, and writes. I ran the strace using the same commands as above and compared the differences between the two different log files.

First of all because strace is monitoring system calls which is potentially dangerous you must specifically tell docker to allow it with the --security-opt seccomp:unconfirmed command line argument to the Docker run command. You must manually compare the output of the good version to the bad version because the file handles are not deterministic between runs in docker. This means that I couldn't just use vimdiff to find the differences.

# This is the output from the git init that caused an error.
# Here we can see the code that writes the error message to the console.
# The system calls that preceded it are also included.
# The Correct output is included below.

open("/data/wsl-directory/.git/config", O_RDONLY) = 4
open("/data/wsl-directory/.git/config", O_RDONLY) = -1 ENOENT (No such file or directory)
fstat(2, {st_mode=S_IFREG|0755, st_size=20556, ...}) = 0
write(2, "error: invalid config file /data"..., 59error: invalid config file /data/wsl-directory/.git/config
) = 59
close(3)                                = 0
unlink("/data/wsl-directory/.git/config.lock") = 0
close(4)                                = 0
write(2, "fatal: could not set 'core.filem"..., 48fatal: could not set 'core.filemode' to 'false'
) = 48

# Here you can see that no error message haw been written.
open("/data/windows-directory/.git/config", O_RDONLY) = 4
open("/data/windows-directory/.git/config", O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0755, st_size=36, ...}) = 0
read(5, "[core]\n\trepositoryformatversion "..., 16384) = 36

Here we can see that the first difference between the two versions happens on the third call to open(). Also one thing to note is that the second and third call to open are in reference to the same file. In the git init with the error the second open() call fails with an error code where as the correct versions opens it without a problem. The problem seems to be when the git init is run in the folder that is created by WSL it cannot open the same file twice without throwing an error.

After doing some more directed research on Google with the newly found informating using strace I was able to find the problem.

TestDisk Saved My Data

Wed, 17 Oct 2018 00:00:00 GMT

It started a couple days ago when I was trying to format a flash drive on Linux.

I ran the following commands to format the flash drive to exfat so that I could use it to take some files to work that I needed.

mkfs.exfat /dev/sdc1

The only problem was that I didn't double check the device I was formatting. This is something that you should always check, when you are running a destructive command. It took me a couple minutes to realize that I did anything wrong. But when I realized that I accidentally formatted my 1 TB hard-drive filled with Movies, TV Shows, Personal Pictures, and Games. I had to take a minute to relax.

I spent the next two evenings researching how to recover data that has been lost. One fortunate thing that happened was is that the format operation I performed was a quick format operation. This means that it just rewrote the meta-data and didn't necessarily overwrite all my data like a complete format would have.

After my research it seemed that a program called TestDisk would be promising. TestDisk works by scanning every single sector on the hard drive partition in question. This is a time consuming process for my 1 TB hard drive, because every single sector had to be scanned to look for old file system meta data which could be used to recover the files from the formatted partition. After the process was done the TestDisk program was able to locate all the files left on the disk. No all that was left to do was copy them to a secondary hard drive that I had installed in my system.

The following images show TestDisk exhaustively scanning my hard drive for files that were referenced by the old file system.

In the end I was able to recover almost all my data. I was able to recover my personal pictures, movies, and Games, but for some reason I was not able to recover my TV Show folder. All in all I'm glad that I was able to get most of my data back. Now i know that there is software out there that can help you recever deleted data as long as a full format was not performed on the partition.

The moral of the story is that you should always double and even triple check the parameters to a destructive disk operation. It would have saved my a lot of time and lots of stress. Now I just need to look into a good solution for backing up large amount of data that don't fit on my OneDrive.

Wireguard VPN behind NAT

Wed, 17 Oct 2018 00:00:00 GMT

In this blog post I would like to show my setup on how I set a VPN to my work account without setting up port forwarding. Currently I work at a small company that does not have a large IT infrastructure. The building that we are leasing manages the network for us which means that we cannot customize the network to our needs. My goal was to be able to ssh into my work PC in case I forget to push some code and didn't want to wait until the next time I came in the office.

Wireguard

The VPN service I choose to use is called WireGuard. WireGuard is a new VPN software that is very small, modern, and simple to use. The actual implementation is under 5 kLOC. With WireGuard there is not necessarily a central server. There are many peers and any peer can connect to any other peer assuming they have the correct authentication credentials. Every peer has a private and public key used to identify its self.

Below I show how to generate the private and public keys that are used to connect to other WireGuard peers. A private and public key must be generated for every single peer in the network.

# Generate Private Key
wg genkey
# ANj/cYQOnkhFviLK70fvzK0f5s7IJSocANeTS11gwnE=

# Generate the Public Key from the Private Key.
echo "ANj/cYQOnkhFviLK70fvzK0f5s7IJSocANeTS11gwnE=" | wg pubkey
# xQhNm4o7P55RDuiF+rAcBhWdxKfVx0U/vC507ayvuT4=

# generate Private and Public Key at the same time using one command.
wg genkey | tee privatekey | wg pubkey > publickey

Peer A Configuration

Configuration files should be located in the /etc/wireguard/wg0.conf directory to be used by the wg-quick helper program.

[Interface]
# Address of the local interface on the PC.
Address = 10.0.0.1/32
# Disable overwritting wg0.conf when wg-quick is used to shut down the interface.
SaveConfig = false
# Port to listen for incoming connections.
ListenPort = 8040
# Private key of the server.
PrivateKey = $PRIVATE_KEY

# Laptop
[Peer]
# Public Key of the peer device.
PublicKey = $PUBLIC_KEY
# Only allow peers with the following IP address.
AllowedIPs = 10.0.0.2/32

# Work PC
[Peer]
PublicKey = $PUBLIC_KEY
AllowedIPs = 10.0.0.3/32
Endpoint = external-ip-of-peer-b:8040
PersistentKeepalive = 25

Peer B Configuration

[Interface]
Address = 10.0.0.3/32
SaveConfig = false
ListenPort = 8040
PrivateKey = $PRIVATE_KEY

[Peer]
PublicKey = $PUBLIC_KEY
AllowedIPs = 10.0.01/32
Endpoint = external-ip-of-peer-a:8040
PersistentKeepAlive = 25

All the information that is really needed to configure the WireGuard is the interface IPs, endpoint IPs, and keys. One important setting to take notice of is the PersistentKeepAlive. This setting will send a handshake to the endpoint every 25 seconds. This is important because if the keep alive packets are not send to the endpoint, the client will not be able to establish a connection to the WireGuard server. This is because the WireGuard is behind a Network Address Translation (NAT) table. This is what converts an external connection to a IP address to IP address of the PC in the internal private network. If the KeepAlive packets are sent out periodically, whenever a connection is established the route is configured in the NAT to the correct private IP address of the WireGuard server.

Network Interface Creation

To start using WireGuard the wg-quick command can be used to lead the configuration files and automatically create the Linux network interface. Just run the following command and then you can connect directly to the IP addresses specified in the network configuration files.

# Bring network interface up for the wg0.conf configuration file.
wg-quick up wg0

# Bring network interface down for the wg0.conf configuration file.
wg-quick down wg0

# Add wg-quick up command to run on start-up.
systemctl enable wg-quick@wg0

# Run the following command to allow ipv4 forward through the VPN.
# In file /etc/sysctl.conf uncommend the following line.
net.ipv4.ip_forward=1

# Run this command to make the change take effect.
sysctl -p

To show that you actually have a valid VPN connection try to connect to the remote peer using ssh. Now you have a simple easy to use private VPN connection to another computer on another network.

Decreasing Elf Size

Thu, 04 Oct 2018 00:00:00 GMT

Recently I had a problem of reducing the size of an ARM executable that was being loaded onto a embedded platform with only 640K of flash storage. I ran into an issue where the executable was much larger than it should have been.

For this post I'll create an example empty c file to demonstrate what is happening.

// main.c
int main() {};

I'll go ahead and compile the simple program using the arm cross compile. I must specify the specs because otherwise the newlib library expects an processor specific implementation of the _exit() function. I'll also strip the executable to remove extra debugging information from the executable.

# Compile the simple main.c file
arm-none-eabi-gcc --specs=nosys.specs main.c -o executable

# Get the size of the executable.
ls -lh executable
-rwxr-xr-x 1 jhamberg jhamberg 59K Oct  3 21:43 executable*

# Strip the ELF file.
arm-none-eabi-strip executable

# Get the size of the stripped executable.
ls -lh executable
-rwxr-xr-x 1 jhamberg jhamberg 36K Oct  3 21:44 executable*

Right off the bat 36K for the stripped executable seems very large for a program that only returns zero from the main function. There is a fair amount of overhead in the ELF file format to be expected, but not 36KB in overhead. I'll go ahead and use the handy program called bloaty that is provided by Google on GitHub to help diagnose what is going on. This tool is used to view the file and memory sizes of ELF executable files.

bloaty executable
     VM SIZE                       FILE SIZE
 --------------                 --------------
  91.4%  31.9Ki [LOAD [RX]]      31.9Ki  89.5%
   4.8%  1.67Ki .text            1.67Ki   4.7%
   3.0%  1.05Ki .data            1.05Ki   3.0%
   0.4%     148 [ELF Headers]       748   2.1%
   0.0%       0 .shstrtab           123   0.3%
   0.0%       0 .ARM.attributes      44   0.1%
   0.0%       0 .comment             43   0.1%
   0.1%      28 .bss                  0   0.0%
   0.1%      24 .fini                24   0.1%
   0.1%      24 .init                24   0.1%
   0.0%       8 .ARM.exidx            8   0.0%
   0.0%       8 .init_array           8   0.0%
   0.0%       8 .rodata               8   0.0%
   0.0%       4 .eh_frame             4   0.0%
   0.0%       4 .fini_array           4   0.0%
   0.0%       4 .jcr                  4   0.0%
   0.0%       4 [LOAD [RW]]           4   0.0%
   0.0%       0 [Unmapped]            2   0.0%
 100.0%  34.8Ki TOTAL            35.6Ki 100.0%

Here you can see what sections are taking the most amount of space in the ELF executable. Bloaty also shows a comparison of what takes up memory in the file and what takes up memory in the memory space of the application. For example the .bss section will take very little space in the executable file, but a large amount of space in the executable memory.

We can see that there is this [LOAD [RX]] section that is taking 31.9Ki of space in the executable. It doesn't seem to corrispond to anything in particular. Lets use the arm-none-eabi-readelf program to see if we can find any extra information about the executable.

# Read the section headers from the ELF executable.
arm-none-eabi-readelf -S executable
There are 15 section headers, starting at offset 0x8c14:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .init             PROGBITS        00008000 008000 000018 00  AX  0   0  4
  [ 2] .text             PROGBITS        00008018 008018 0006b0 00  AX  0   0  4
  [ 3] .fini             PROGBITS        000086c8 0086c8 000018 00  AX  0   0  4
  [ 4] .rodata           PROGBITS        000086e0 0086e0 000008 00   A  0   0  4
  [ 5] .ARM.exidx        ARM_EXIDX       000086e8 0086e8 000008 00  AL  2   0  4
  [ 6] .eh_frame         PROGBITS        000086f0 0086f0 000004 00   A  0   0  4
  [ 7] .init_array       INIT_ARRAY      000186f4 0086f4 000008 04  WA  0   0  4
  [ 8] .fini_array       FINI_ARRAY      000186fc 0086fc 000004 04  WA  0   0  4
  [ 9] .jcr              PROGBITS        00018700 008700 000004 00  WA  0   0  4
  [10] .data             PROGBITS        00018708 008708 000438 00  WA  0   0  8
  [11] .bss              NOBITS          00018b40 008b40 00001c 00  WA  0   0  4
  [12] .comment          PROGBITS        00000000 008b40 00002b 01  MS  0   0  1
  [13] .ARM.attributes   ARM_ATTRIBUTES  00000000 008b6b 00002c 00      0   0  1
  [14] .shstrtab         STRTAB          00000000 008b97 00007b 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  y (purecode), p (processor specific)

Here we can see all the individual sections that are included in the simple executable. For example the .text section contains the actual executable code. The .rodata contains read only constant that are used the the executable. The .data contains pre-initialized variables that contain values that are not zero. All of these seem all right. Non of the sizes are very large, at least nowhere near the 36K of the executable. First I'll explain some of the fied meanings. The Type field is the type of the section and what it will be sued for. The Addr field is the destination address in the virtual memory space. Off is the offset in the actual executable file it's self. And Size is the size of the section.

This seems to be all normal, except after some close examination you will notice that the .init section which is the first section starts at a file offset of 0x8000 which is 32K. This means that the first section in the ELF executable is located 32K into the file. There is some header information, but certainly not 32K of header information. This seems to be the problem.

The solution to this is to tell the linker what the page size of the sections should be. By default the linker assumes that the user want's a page size of 32K which means that the sections must me aligned to 32K which causes much empty space at the beginning of the program. This can be specified to the linker by using the -max-page-size linker flag.

# Compile with the max-page-size information.
arm-none-eabi-gcc --specs=nosys.specs -z max-page-size=0x04 main.c -o executable

# Print the size of the new executable.
 ls -lh executable :Q
-rwxr-xr-x 1 jhamberg jhamberg 28K Oct  3 22:00 executable*

# Print the size of the new stripped executable.
ls -lh executable
-rwxr-xr-x 1 jhamberg jhamberg 3.8K Oct  3 22:00 executable*

Here you can see that the size of executable is 3.8K which is much closer to what one would expect for such a simple program. I assume the linker defaults to such a large page file for performance reasons. When loading the executable into memory, presumably it would be faster to read data from the disk on page aligned memory offsets. This is fine for a host system, but when space is at a premium on a embedded system this can be detrimental.

Audiobook Ripping on Linux

Sat, 15 Sep 2018 00:00:00 GMT

In this blog post I would like to share my process for ripping audio-books for listening to on my commute to work. I will also talk about some of the challenges of ripping audio in a reliable way without many defects. I am also going to describe how to use some of the programs that I use to produce the output of the cdripping script.

CDPARANOIA

At the center of the audio CD ripping process is a program called cdparanoia. Cdparanoia extracts audio from compact discs directly as data, with no analog step between, and writes the data to a file in an uncompressed format. In addition to simple reading, cdparanoia adds extra-robust data verification, synchronization, error handling and scratch reconstruction capability.

Because of these additional error checking features, the ripping time can take a fair amount of time. It takes anywhere from 5 minutes to an hour to rip one audio CD depending on the number of errors on the disk. On average it takes about 10 - 15 minutes.

# RIP a cd by specifier the first track to the last track and the output file format.
cdparanoia 1- output.wav
cdparanoia III release 10.2 (September 11, 2008)

Ripping from sector       0 (track  1 [0:00.00])
	  to sector  298786 (track 11 [7:41.43])

outputting to cdda.wav

 (== PROGRESS == [+++ ++++++++  ++++++++++++>   | 260405 00 ] == :-P O ==)

Below is the legend for the symbols in the progress bar from cdparanoia. All of the errors should not affect the output unless specifically specified by the V symbol which means there was an uncorrected error/skip. All the other symbols were able to successfully be corrected

PROGRESS BAR SYMBOLS:
<space> No corrections needed
   -    Jitter correction required
   +    Unreported loss of streaming/other error in read
   !    Errors are getting through stage 1 but corrected in stage2
   e    SCSI/ATAPI transport error (corrected)
   V    Uncorrected error/skip

SOX

SoX is a program that is used to read, write, edit, and splice popular audio formats. It has a simple syntax to convert audio formats. Files can simply be converted to another format by specifying the source file and specifying the destination file with the appropriate file extension. Audio files can be combined into one file by specifying a list of files and the desired output file. this is useful if you want to combine multiple track of a disk back into one file for the disk. This swiss army knife of a tool can be used for many more things, but I simply use to transcode the resulting audio to compressed lossy and lossless formats. When cdparanoia is finished the result is a .wav file that is approximately 700 MB. SoX is used to convert the result .wav file to a lossless .flac file which is approximately 150 MB and then also a compressed Vorbis ogg file is produced which is approximately 50 MB large. The smaller compressed audio format is good for playing on a mobile phone to save space.

# Convert an uncompressed audio file into a compressed ogg vorbise file.
sox output.wav output.ogg

# Convert uncompressed audio file into a lossless flac file.
sox output.wav output.flac

# Concatinate many files into one.
sox part01.wav part02.wav part03.wav output.flac

# Support for new formats can be added by simply instaling the right packages.
sudo apt install libsox-fmt-mp3
sox output.wav output.mp3

SCRIPT USAGE

I wrote a simple shell script to expedite the process of ripping audio books to by desktop. The script is currently only around 150 lines of code with comments. Once on my desktop I can just copy the compressed output files to my phone to listen to on the bus. My script is available on here for reference.

The goals of the script were to automate the ripping process. Desired features

Automatic in order disk numbering.
Automatic output directory management.
Automatic audio transcoding of the audio disk.

Below is a sample invocation of the script. You must specify the album that is being ripped. This is used as the output directory of the script where audio end up. The first thing the script does is to search the output directory to see what disks are already present. It picks the next available disk and uses that as the filename for the output. This satisfies requirement one of the cdripper script. The file are automatically moved to the Ender_in_Exile directory when finished which satisfies requirement number two. After the audio disk has finished ripping the audio is automatically transcoded to Disk05.flac and Disk05.ogg. The flac file is for the lossless audio which can be used to transcode into any other format. The Vorbis ogg file is used to copy to my phone to listen to on the bus.

cdripper --album Ender_in_Exile
Ender_in_Exile/Disk05
cdparanoia III release 10.2 (September 11, 2008)

Ripping from sector       0 (track  1 [0:00.00])
	  to sector  298786 (track 11 [7:41.43])

outputting to Disk05.wav

 (== PROGRESS == [+++ +++  + +  ++++++++++++V+++| 298786 00 ] == :^D * ==)

Done.

Rip Time: 30:05m
Transcode Time: 59s