My ideal journalling app

    • WYSIWYG front-end editor
    • Data stored as light-weight markup (Markdown, etc) 1
    • Data stored as plain text files in a directory structure 2
    • Historical versions saved
    • Cross platform, including at least Linux and Android
    • Able to publish in various formats to various destinations
    • Scriptable, to allow automatic indexing, export, etc

    Actually, Zim seems to do all of this. I have used Zim in the past, and published a website <public.irons.nz> onto IPFS . But the IPFS node was a CPU and memory hog, so I stopped that experiment in 2022.

    I had been wondering how to get the articles in Zim onto my blog. Perhaps I should turn that around:

    • Zim becomes the main thing
    • Save to a local version control system; it does this automatically
    • Backup to a remote version control server
    • Publish to blogs, etc
    • Publish as a website

    And yet, somehow, it seems more of an effort to open a new program, rather than a new tab in a browser. Strange.


    1. This introduces compatibility issues, as there are many partly-compatible versions of almost every markup language ↩︎

    2. Easy to back up, version control, publish, edit externally ↩︎

    The things I like to write about

    I write sporadically about a few different things. Each of these fills a different purpose in my life, and I want to share them differently

    Category Examples Medium Reason
    Personal reflections Personal issues
    People problems
    Paper journal
    Face-to-face conversation
    If it is online, it will never remain private
    Family activities Snaps of the grandchildren
    Everyday activities
    Private WhatsApp group Common denominator among the group
    My interests Hobbies
    Photographs
    Blog Happy to share them widely
    Professional matters Software and system problems
    and their solutions
    Blog Want to share them widely

    Personal reflections are written in a paper journal; if they are shared at all, it is by face-to-face communication. My feeling is that if it is online anywhere, it will not stay private.

    Everyday activities are shared on a private WhatsApp group consisting of parents, children, siblings, etc.

    • It is hard to know who to include and who to exclude.
    • Sometimes, there is something that might be of interest to a few more remote relatives or friends, but I don’t want to add them to the group. I have so solution for this. Perhaps an unlisted public group of some sort. This makes it relatively private and secure, provided that it does not matter if anything posted here becomes really public.
    • I am not a big fan of WhatsApp, but it is the common denominator for this group of people, and I don’t have the time or motivation to get everyone to use something better. This WhatsApp group is linked to a self-hosted messaging server so that I am confident that I have a record of everything if WhatsApp turns ugly.

    Personal interests and professional matters that I feel can be made public, I write about on a blog linked to my own domain; the service is provided by a 3rd-party blog system.

    • I have configured the blog system to republish to various social networks.
    • I should also hook it up to my self-hosted messaging server so that I have control over the data.
    • At the moment, I use the blog provider’s web app to compose articles. This means that I need to be online; I have not found a suitable client application.

    I self-host a Matrix server. Initially, this was an experiment to see if it is something that grandma could use rather than WhatsApp 1. I subsequently set up gateways to various other messaging services (WhatsApp, Facebook Messenger, Signal, Google Groups, SMS)

    • to give me ownership of a copy my data
    • to give me a single application for all of my chats

    This is working pretty well, though I am not happy with my data backup solution.


    1. No. The server is reliable and stable. However, the client applications are confusing for non-geeks. ↩︎

    micro.blog markdown summary

    Structure

    • Heading 1: prefix with hash (#), or ==== on the next line
    • Heading 2: prefix with double hash (##) or ---- on the next line
    • Horizontal rule: four hyphens (----) on an isolated line
    • Block quote: prefix each line with greater-than sign (>)
    • Code blocks: three backticks with language (```python) is supported, but does not work in the preview. Alternatively, indent each line by 4 spaces.
    • Tables: use fairly obvious ASCII-art

    Inline text

    • Emphasis: surround with underscore (_)
    • Strong emphasis: surround with double asterisk (**)
    • Strike-out: surround with double tilde (~~); does not work in the preview
    • Code: surround with backtick (`). Use more backticks than you want to include in the code. (```)
    • Highlight: surround with mark tags (<mark>, </mark>)
    • Subscript: surround with sub tags (<sub>, </sub>)
    • Superscript: surround with sup tags (<sup>, </sup>)

    Lists

    • Unnumbered lists: prefix with asterisk (*)
    • Numbered list: prefix with a number, value not important (1)
    • Todo lists: use an unsorted list with square-brackets and square-brackets-x ([ ] and [x]) as the first text item. Does not show correctly in the preview.

    Links and images

    • Links:
      • URL in angle brackets (<http://url>)
      • Display text in square brackets, URL in round brackets ([display text](http://url))
      • Index/display text in square brackets, empty square brackets ([display text][]). Then, elsewhere in the document, index definition, colon, URL ([display text]: http://url)
    • Images:
      • Exclamation mark, alt-text in square brackets, URL in round brackets, optional title enclosed in double-quotes (![alt text](http://url "title text"))
      • Exclamation mark, alt-text in square brackets, index reference in square brackets (![alt text][index ref]). Then, elsewhere in the document, index definition, colon, URL, title text ([index ref]: http://url "Title text")
      • It seems that title text is not shown
      • If you want to suggest the size, you have to fall back to the HTML <img> tag with width and/or height properties.
    • Ensure that link and image references are unique in the document.

    Escapes

    • Markdown uses a number of special characters to indicate various markup. Examples are code spans (surrounded by backticks `) and emphasised text (surrounded by underscores _).
    • To include one of these special characters as themselves, escape it with a backslash \. So
      • This is an underscore ‘_’
      • This is a backtick ‘`’
      • This is an angle-bracket ‘<’
      • This is a backslash ‘\’
      • None of these are in a code-span.
    • The special symbols are
      • \ backslash
      • ` backtick
      • * asterisk
      • _ underscore
      • # hash symbol
      • + plus sign
      • - hyphen, minus sign
      • . dot, period
      • ! exclamation mark
      • { } curly brackets
      • [ ] square brackets
      • ( ) round brackets
      • < > angle brackets

    Examples

    Structure

    Heading 1

    Heading 2

    This is normal text

    This is a block-quote text of one line

    Back to normal text, followed by a code block

    code block
    more code
    

    Back to normal text. We can also use the syntax-highlighting style, though it does not show in the preview.

    • Python
      1
      2
      3
      4
      
      #! python3
      
      class Xdef:
          pass
      
    • Shell
      1
      2
      3
      
      #! /bin/sh
      
      i=0; while true; do echo $i; i=$(($i + 1)); done; then a lot more stuff; to show what happens when we overflow
      

    Inline

    • This is emphasis with underscores and with asterisks
    • This is strong emphasis with double underscores and with double asterisks
    • This is strike-out with double tilde
    • This is code with lots of backticks code with 4 backticks ````
    • This is highlight surrounded mark-tags
    • Subscript H2O
    • Superscript a x2

    Lists

    • Unordered list item 1
    • Unordered list item 2
      1. Ordered sub-list item 1
      2. Ordered sub-list item 2
    • Back to the original list
      • Unordered sub-list item 1
    • Back to the original list again

    Here is something

    • Todo list is not supported
    • Or maybe it is


    Images

    • Reverse tango hook alt text, with link and title

    • Reverse tango hook alt text, with reference

    • Reverse tango hook alt text, using img tag
    • Stephen Irons

    The link definitions are below this line in the source, but invisible in the presentation.

    Mechanical keyboard thoughts

    In August 2020, I bought my first mechanical keyboard: a Keydous NJ68 from Aliexpress

    • 68 keys(5 rows typewriter, with arrows and a few editing keys)
    • PBT keycaps, with nice, simple lettering, Cherry profile
    • Cherry Brown switches
    • RGB backlight with lots of patterns, etc
    • Hot-swapable switches
    • Bluetooth or USB-C, with big battery

    I am very pleased with the way it works. A few months later, I bought another, this time with Cherry Blue (noisy) switches. I use the noisy switches in my home office, and the Brown switches in my open-plan office at work.

    About annually, I do a scout around of mechanical keyboards to see if anything interests me. Then I realise that, for me, a keyboard is a tool, not a hobby. I have plenty of other hobbies to consume my money.

    Part of me would like to build a wooden housing for the keyboard. I have few nice chunks of wood (kwila, acacia, beech) and it would be an interesting winter project.

    These are my thoughts about the decisions I would make if I were to choose again

    • Mechanical keys are nice
    • Clicky (blue) keys are great, but tactile (brown) are less offensive for others trying to sleep
    • TKL (84 keys) might be better than 68 keys
    • Bluetooth is not worth it if you have a BIOS password; unique keyboard dongle is nice as it keeps the desk clear.
    • RGB backlighting is pointless
    • Don’t worry too much about programmability, it is easy to learn new muscle memory – unless you have different keyboard that you use simultaneously

    Details

    • I really do prefer the feel of the mechanical keys, and that is worth paying some money for.

    • I like both the clicky (Blue) and the feely (Brown) switches. I would probably get the Browns as the noise from the Blues does affect others at home.

    • It is nice and heavy, so it stays put on the desk.

    • I would probably choose a TKL (87 key) keyboard. This keyboard is big enough that there are not too many second-layer keys

      • I have learned where the most common editing keys are.
      • But the [`/~] key is on the right hand side so that the [Esc] key can be close to its normal position. This is unnatural.
      • The TKL layout is not much bigger, and I am not particularly limited in my desk space.
    • I would not choose Bluetooth.

      • I used to use a Logitech wireless keyboard, and do prefer to not have a cable lying on the desk.
      • I cannot use Bluetooth. For Various Reasons (TM), I have an encrypted disk and need to enter a password before the OS has booted. The BIOS on my PCs does not support Bluetooth, so I have to have either a wired keyboard, or one with a dongle that pretends to be a keyboard.
      • The battery lasts about a week, so I had to plug it in periodically anyway; I could get into the habit of hunting around for a USB cable to plug it in on Friday afternoons.
    • I really like the idea of swappable keycaps. However, in 2 years, I have not felt the need to change them.

      • I used to change keyboards when the bumps on [F] and [J] key wore off, usually after 3–4 years.
      • After 2 years, the [J] key is wearing down a bit; [F] is still ok.
      • I really do like some of the other colour options available.
      • I am not prepared to spend $100–$200 (or much, much more) on keycaps alone. This is not a hobby.
    • Don’t bother with hot-swap switches. It has been 2 years now, and I have not felt the need to change them. This is not a hobby.

    • Don’t bother with RGB backlighting.

      • I found the flashing patterns to be very distracting; I did rather like a dim cyan colour
      • I often accidentally hit the magic key-sequence to select another pattern, or to change the brightness, then I had to find the user guide to work out how to reset it
      • The light does not shine through the double-shot PBT keycaps; it only comes through the gaps between the keys
      • I have subsequently re-configured the keyboard to disable the pattern-change key-sequence.
    • Don’t bother with programmability (QMK, etc)

      • I keep the same keyboard long enough that it is easier to re-learn where the hidden keys are, rather than re-program the keyboard to suit my muscle memory.
      • This keyboard uses a proprietary, Windows-only application to re-program the keys.
      • I have only used it work around accidentally changing the backlighting; if there are no annoying colour patterns, and enough keys that second-layer keys are easy to learn, then there is no need.

    Nice keyboard colour schemes

    Despite having said that keycaps are too expensive, there are some really beautiful colour schemes out there that might be contenders when the [F] and [J] finally do wear out. My current favourites are

    • GMK Birch
    • GMK Green Tree

    NZ Post scam text

    I received a text today, purporting to come from NZ Post, saying that a parcel could not be delivered, with a link to click to arrange re-delivery.

    Unfortunately, I missed the indications that it might be a scam:

    • It said they would arrange re-delivery the same day; normally, re-delivery happens the following day
    • Phone number is Austrilian (+61, rather than +64)
    • The link to click points to bit.ly, a link-shortening service, which also acts as a link-obfuscation service.
    • Sent using RCS rather than normal SMS; this is just a bit unusual

    I clicked the link, and got a page that looked like a tracking page with a valid-looking tracking number, except

    • it had an unusual URL https://nztxpits.icu/?token=, nothing like NZ Post
    • the delivery events (pickup, out for delivery, etc) did not make any sense

    My experience with your standard re-delivery web pages is that it is easier to do with a keyboard rather than on a phone, so I transferred over to the laptop.

    Clicking the link again failed. Typing the tracking number into the NZ Post tracking service brought up a page with some data the same as the scam link, but slightly different details.

    It was only at this point that I realised something was not right. Searching NZ Post website for current scams showed that this is a thing.

    Lessons learned

    • I have stopped clicking on links in emails, unless I have explicitly requested something by logging in to the website, and the email appears in my inbox in the next few minutes.
    • I will have to stop clicking links on texts too.
    • Do not trust URL shorteners such as bit.ly, goo,gl, TinyURL, etc.
    • The bank says on its website that it no longer sends clickable links in emails, and requests that you log in manually; the same applies to all services.
    • If you do receive a link, even one that looks valid, do not click it. Instead, log in to the service manually, and type or copy-paste the identification number (tracking number, reference number, invoice number, etc) into the appropriate box on the website.
    • If a service provider does not provide enough information to do this, complain to the service provider.

    Outcome

    • I clicked the link, so the scammer knows that my phone number is valid.
    • I did not click the re-delivery options button, so I did not enter any important information.
    • But this was by good luck, not by good judgement.
    • My house is weatherboard too.
    • Painted the same colour.
    • My gumboots are on the deck.

    Move TLS certicates to a new host

    Use ssh to tar a file on the remote host and pipe it to tar on the new host. This avoids the risk of leaving a file around with this sensitive data.

    On the new host,

    ssh user@other.host sudo -S tar -C /etc/letsencrypt -zvc live archive | sudo tar -C /etc/letsencrypt -zvx
    

    Note that this runs sudo on the remote host; the -S argument uses stdin to read the remote user’s password. This means that the password is displayed on the terminal, but at least it is not in the command history. After running the command, consider clearing the terminal buffer using reset or similar.

    Access home router web UI from a remote location

    The home-router web UI is only accessible from the internal network, not from the outside world. This is to prevent the evil outsiders from try to break it, or break into it.

    But sometimes I need to access it remotely. This is a multi-stage process involving multiple ssh port forwards, and some DNS and HTTP-client trickery.

    Port forwarding

    This needs a PC on the remote network (local to the router) and a server that is world-visible.

    • ssh into a PC on the remote network, then set up a local port-forward from a local port to the router

        remote-pc:~$ ssh -N -L 8123:192.168.1.1:80 stephen@localhost
      
    • ssh into the same remote PC, then set up a remote port-forward from the world-visible server to the

        remote-pc:~$ ssh -N -R 8124:localhost:8123 stephen@world.visible.server
      
    • from the local workstation, set up a local port-forward to the world-visible server

        local-pc:~$ ssh -N -L 8125:localhost:8124 stephen@world.visible.server
      

    You can now use an HTTP client to connect to the local port (localhost:8125) and this is forwarded to the very remote router (192.168.1.1:80 on the remote network).

    However, there is still a bit of HTTP trickery to do. The HTTP request include a header ‘Host:’. If the remote host (the router) is doing its job, it will verify that this hostname is correct. Normally, the HTTP client uses the hostname from the URL (localhost) as the ‘Host’ header.

    In this case, the remote router accepts either the name configured in one of its settings, or one of its own IP address. In either case, either of those name does not resolve to the correct location.

    If using curl, you can use the ‘-H’ option to set the hostname

    curl -H 'Host: 192.168.1.1' http://localhost:8125
    

    Using firefox, however, this is more tricky. You need an extension such as ‘Modify Header Value’ to change the header on a per-URL basis. I found it best to do the following:

    • add a new host-name to IP mapping to /etc/hosts, something like

        127.0.0.1    remote-router
      
    • add a new header modification to the firefox extension, something like

        remote-router  modify    'Host'   192.168.1.1
      

    Self-hosted server: router/firewall configuration

    The internet router is a residential model from Vodafone. It seems to do the following

    • Supports IPv6, including

      • full support for outgoing IPv6 connections
      • stateless address autoconfiguration (SLAAC)
    • Allows IPv4 incoming connections only.

      • By default, blocks all incoming connections (IPv4 only).
      • Opens individual ports forwarded to specified host:port (standard NAT stuff, IPv4 only).
      • Can set up a DMZ server, where all other incoming connections are routed to a single host (IPv4 only).
      • No way to configure IPv6 incoming connections.
    • It seems fairly reliable, but something drops out periodically.

      • The router itself
      • The far-end of the fibre connection
      • The local Wi-Fi extender (TP Link Deco system)

    Setup

    • Ensure that IPv4 address remains fixed: requested from Vodafone, $5 per month

    • Ensure that IPv6 address remains fixed; might happen at the same time

    • Port configuration details

      • Leave the standard port-forwards as they are
      • Set up DMZ pointing to new server
      • Ensure that new server has firewall (ufw) enabled and stopping as much as possible

    Self-hosted server: software required

    Install and configure the following applications and services

    • Matrix homeserver

      • Server - synapse
      • Facebook messenger bridge - mautrix-facebook
      • Signal messaging bridge - mautrix-signal
      • Whatsapp bridge - mautrix-whatsapp
      • HTTP webhook - pushmatrix
      • Database - postgresql
      • HTTP server - nginx
      • TLS certificates - Let’s Encrypt
      • Move data from previous postgresql database
    • IoT platform

      • Server - Thingsboard
      • Database - postgresql
      • HTTP server - nginx
      • TLS certificates - Let’s Encrypt
      • Move data from previous postgresql database
    • Performance monitoring

      • Munin server
      • Munin node
      • HTTP server - nginx
      • TLS certficates - Let’s Encrypt
      • Move data from previous server – lots of RRD files
    • Backup storage - duplicity, rsync, etc

    • Data synchronisation - SyncThing

    Self-hosted server: server hardware

    I have an old HP EliteDesk G3 Mini PC

    • 4 core CPU
    • 8 GB RAM
    • 256 GB SSD (SSD)
    • Gigabit ethernet
    • Wi-Fi, etc

    This is twice the specs of both of my current virtual servers added together, so it should do the trick.

    Potential upgrades

    • RAM maxes out at 32 GB
    • Able to take 2 SSDs: one SATA and one NVMe

    Things to consider

    • Increase RAM to 32 GB

    • Install large (2 TB? 4 TB?) SATA and NVMe SSDs

    • Set up RAID1 – save data on 2 different drives, allowing for continued operation even if one of them fails.

    • Backup system – to local and remote devices

    • UPS to allow continued operation in case of power-failure. In this case, it has to provide power for

      • Server
      • Router
      • Switch
      • Backup server
    • Performance monitoring

      • CPU usage and performance
      • RAM usage and performance
      • SSD usage and performance
      • Network usage and performance
      • CPU health – voltages and temperatures
      • SSD health – SMART monitoring, RAID status
      • Network health – errors, retries, etc

    Self-hosted server

    It seems wasteful to rent two virtual servers from a cloudy-provider when the same money would buy a real server with twice of everything (CPUs, RAM, SSD, etc) in 12 months. However, moving it in-house comes with some additional costs.

    These are the things that run on the existing servers, in order of importance

    • Backup server for PCs and phones
    • Synchronisation server
    • Performance monitor
    • Matrix homeserver
    • IoT platform

    However, the biggest bang-for-the-buck will come from moving the IoT platform, Thingsboard, running as iot.irons.nz on iolanthe.irons.nz.

    • The only service on iolanthe.irons.nz, so once it is moved, we can shut down

    • Very few incoming network connections

      • one or two incoming webhooks (Swarm, weather-station)
      • one incoming MQTT connection (weather-station)
      • for testing, one or two LwM2M devices

    These are the things than need doing

    We will tackle these one at a time.

    Self-hosted Matrix server: move database to a new server

    • On both servers, create a ~/.pgpass file with the database usernames and passwords. Each line is the login credentials for one database, in the format hostname:port:database-name:username:password

    • On the old server, shut-down Synapse and the plugins

        sudo systemctl stop synapse
      
    • On the new server, shut-down Synapse and the plugins

         ssh stephen@constanza.irons.nz pg_dump -C -h localhost -U synapse synapse | psql -h localhost -U synapse synapse
      

      This took a few minutes; this is a small Synapse installation

      The databases are not exposed to the internet, so ssh is used to create a session on the remote side.

    Self-hosted Matrix server: configure nginx and Let's Encrypt

    Assuming that nxginx and Let’s Encrypt are installed

    • Create nginx configuration file /etc/nginx/sites-available/irons.nz

        server {
            listen 80;
            listen [::]:80;
      
            server_name irons.nz;
            return 301 https://$host$request_uri;
        }
      
        server {
            listen 443 ssl http2;
            listen [::]:443 ssl http2;
      
            listen 8448 ssl http2;
            listen [::]:8448 ssl http2;
      
            ssl_certificate /etc/letsencrypt/live/irons.nz/fullchain.pem; # managed by Certbot
            ssl_certificate_key /etc/letsencrypt/live/irons.nz/privkey.pem; # managed by Certbot
      
            server_name irons.nz;
      
            location / {
                return 404;
            }
      
            location /_matrix {
                proxy_pass http://localhost:8008;
                proxy_set_header X-Forwarded-For $remote_addr;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header Host $host;
      
                # Nginx by default only allows file uploads up to 1M in size
                # Increase client_max_body_size to match max_upload_size defined in homeserver.yaml
                client_max_body_size 50M;
            }
      
            location /_synapse/client {
                proxy_pass http://localhost:8008;
                proxy_set_header X-Forwarded-For $remote_addr;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header Host $host;
      
                # Nginx by default only allows file uploads up to 1M in size
                # Increase client_max_body_size to match max_upload_size defined in homeserver.yaml
                client_max_body_size 50M;
            }
        }
      
    • Temporarily comment-out the listen:443 lines and the certificate lines

    • Test the configuration, then restart nginx

      sudo nginx -t
      sudo systemctl restart nginx
      
    • Verify that it is running

      curl localhost:80
      
    • Update DNS record for https://irons.nz

    • Create certificate for https://irons.nz

        sudo certbot certonly --nginx
      

      This is an interactive command; there might be command-line options to specify the URL.

    Install Let's Encrypt -- for nginx, using pip

    Edit: this turned out to be too complicated. As much as I prefer to avoid a Snap package, I decided to go with it rather than jump through these hoops. When I have the system up and running, I might re-evaluation this decision.

    • Install dependencies

        sudo apt install python3 python3-pip libaugeas0
      
    • Remove any OS-installed version

        sudo apt remove cerbot
      
    • Set up a Python virtual environment with certbot

        sudo python3 -m venv /opt/certbot/
        sudo /opt/certbot/bin/pip install --upgrade pip
        sudo /opt/certbot/bin/pip install certbot certbot-nginx
      
    • Optionally, set up a symlink in the path

        sudo ln -s /opt/certbot/bin/certbot /usr/local/bin/certbot
      
    • Enable auto-renewal

        echo "0 0,12 * * * root /opt/certbot/bin/python -c 'import random; import time; time.sleep(random.random() * 3600)' && sudo certbot renew -q" | sudo tee -a /etc/crontab > /dev/null
      

    Derived from Let’s Encrypt - pip

    Self-hosted Matrix server: Install Synapse

    • Create a system-user

        SYNAPSE_USER_DIR=/var/lib/synapse
        sudo adduser --system synapse --home $SYNAPSE_USER_DIR/synapse
      
    • Install a Python3 virtual environment and Synapse

        SYNAPSE_DIR=$SYNAPSE_USER_DIR/synapse
        sudo -u synapse mkdir -p $SYNAPSE_DIR
        sudo -u synapse python3 -m venv $SYNAPSE_DIR/.venv
        sudo -u synapse $SYNAPSE_DIR/.venv/bin/pip install matrix-synapse
        sudo -u synapse $SYNAPSE_DIR/.venv/bin/pip install psycopg2-binary
      
    • Create and edit the configuration file. See https://matrix-org.github.io/synapse/develop/setup/installation.html for details.

        sudo -u synapse /.venv/bin/python3 -m synapse.app
      

      Incomplete

      Alternatively, copy the file from another installation and edit it

    • Create and install a systemd .service file

        (
        cat <<EOF
        [Unit]
        Description=Synapse Matrix server
        Documentation=https://www.matrix.org
        After=network.target
        #After=postgresql.service
      
        [Service]
        Type=notify
        NotifyAccess=main
        Restart=on-abort
      
        User=synapse
        Group=nogroup
        WorkingDirectory=$SYNAPSE_DIR
        ExecStart=$SYNAPSE_DIR/.venv/bin/python3 -m synapse.app.homeserver --config-path=$SYNAPSE_DIR/homeserver.yaml
        ExecReload=/bin/kill -HUP $MAINPID
        SyslogIdentifier=matrix-synapse
      
        [Install]
        WantedBy=multi-user.target
        EOF
        ) | sudo tee /etc/systemd/system/matrix-synapse.service          
      

      Derived from https://github.com/matrix-org/synapse/blob/master/contrib/systemd/matrix-synapse.service

    • Start service

        sudo systemctl start matrix-synapse
      
    • Verify that it has started

        systemctl status
        journalctl -u matrix-synapse
      
    • Set it to start automatically

        sudo systemctl enable matrix-synapse
      

    Self-hosted Matrix server: set up PostgreSQL for Synapse

    Assuming

    • PostgreSQL is installed.
    • Synapse and the various bridges will use the same database.

    Do the following

    • Create a user

        sudo -u postgres createuser --pwprompt synapse
      

      This will prompt for a password; record the password: it will need to go into the synapse configuration file.

    • Create databases owned by this user as needed for

      • synapse

      • mautrix-facebook

      • mautrix-signal

      • mautrix-whatsapp

            sudo -u postgres createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse synapse
            sudo -u postgres createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse mautrix-facebook
            sudo -u postgres createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse mautrix-signal
            sudo -u postgres createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse mautrix-whatsapp
        

    Yet another calendar-time special case

    In the beginning, time was simple for computers. We just did everything in our local time. Sure, calendars were tricky, what with months with a different number of days, months that did not always have the same number of days, and so on.

    But some places had this daylight saving thing, where the time changed twice a year, often by an hour in either direction. So, the intelligent among us decided to use UTC everywhere, and make adjustments to local time when needed.

    However, real people use local time. For example, if I ask a customer when a problem happened, they will use local time. That meant that to convert from the computer time to local time, we needed to know which timezone was in use. So we used the combination (UTC, timezone name). But that did not solve the problem everywhere, because we need an up-to-date set of timezone data to calculate the local time; this is not always available, or is not always correct or up-to-date.

    It then seemed that using (local time, timezone offset) or (UTC, timezone offset) would do it; these are semantically identical. I prefer, where possible, to use local time + timezone offset, for no justifiable reason.

    That does just fine for times where the timezone offsets are correct – that is, for times in the past where the timezone database is correct. However, timezones are a political thing, and there are errors in everything, so the timezone database can change, either because a government decides to change the timezone rules, or to fix bugs in the database.

    This means that you need to store 3 things about a timestamp

    • the actual time in either a local time or UTC
    • timezone offset from UTC at that time
    • the timezone name, in case the time is in the future, and the timezone offset changes, and we need to show the time in local time.

    Self-hosted Matrix server: Overview

    System

    • HP mini PC
    • 4 core, 3.1 GHz, 12 GB RAM, 128 GB SSD

    Other context

    • A few other services running on the server
    • Domain name irons.nz already set up
      • A few subdomains using A, AAAA and CNAME records

    OS setup

    • Ubuntu 22.04.2 LTS

    • postgresql (version 14)

    • nginx

    • Python3 (version

      • virtualenv
      • pip
    • git

    Installation

    • postgreql
    • synapse
    • nginx
    • letsencrypt
    • mautrix-facebook
    • mautrix-signal
    • mautrix-whatsapp
    • pushmatrix

    Using CANView on Ubuntu 22.04

    CANView is a tool for viewing CAN bus data. It can view live data when connected to a suitable tool, or can view saved log files in a few formats.

    It is a cross-platform tool, provided by Yacht Devices.

    The latest version is 1.40, released on 5 August 2021.

    On Ubuntu 22.04, it fails at startup, with the message

    ./CANView: error while loading shared libraries: libpng12.so.0: cannot open shared object file: No such file or directory
    

    Investigation shows that, indeed, libpng12 is not installed; libpng16 is installed; and libpng12 is not available for installation.

    A version of libpng12 is available for Ubuntu 22.04 from Launchpad, created by Linux Uprising.

    I could not install this using dpkg, as it complained about a conflict. However, a .deb file is really just an archive file of some sort, and can be opened using an archive viewer. I used file-roller to extract libpng12.so.0.54.0 and a symlink libpng12.so.0

    • Download CANView

    • Unzip CANVIEW.zip somewhere

    • Change the executable flag on CANVIEW/Linux/CANView

    • Download libpng12

    • Extract libpng12.so.0.54.0 and the symlink libpng12.s0.0 into the same directory as the Linux executable

    • Run CANView, pointing the library loader to the directory that has libpng12. Assuming that libpng12 and CANView are in the current directory

        LD_LIBRARY_PATH=. ./CANView
      

    Supported log file formats

    • .CAN,

    Credit-card single-board computers

    I have used the following credit-card sized single-board computers.

    • Raspberry Pi family
    • BeagleBone Black family
    • Rock64

    My preference is for the BeagleBone Black family, with the Green Wireless from SeeedStudio as my favourite.

    BeagleBone Green Wireless

    • All of the BeagleBone family have on-board eMMC for the main filesystem, rather than SD card
    • 4 USB ports
    • WLAN and Bluetooth, with on-board chip antenna and 2 u.FL connectors for external antennas
    • Connector for standard FTDI cable
    • Lots of GPIOs, various serial port, CAN, etc
    • No on-board HDMI framer (I thinik), so those pins can be used as GPIO; I don’t need video
    • Schematics, PCB layout and documentation for all of the devices available
    • No proprietary stuff
    • Modern Debian 11 (Bullseye), programmable via SD card socket

    Disadvantages

    • Old components
    • Rather slow
    • Small RAM
    • No on-board DC power connector. Power comes from the micro USB socket, or you can supply power from a piggy-back board (‘cape’) via the 5 V pin.
    • Very few piggy-back boards available
    • Small community

    I like the look of the BeagleBone Green Gateway (also from SeeedStudio), but I have not used it.

    • Similar to BBGW
    • On board DC power connector, with input range 5 – 28 V
    • Ethernet, WLAN, Bluetooth
    • 2 USB ports

    Raspberry Pi

    • Huge community
    • Lots of add-on boards
    • SD card is not reliable as the primary file system
    • Proprietary main process; no documentation available
    • Ethernet via USB

    Rock64

    • For its time, it was a very powerful processor
    • GB Ethernet
    • One ?? (fast) USB port
    • 4 standard USB ports
    • Proprietary processor; no documentation available
    • Small community, poor software support

    Home-made deodorant

    Tired of looking for my homemade deodorant recipe every few months, trying to remember which one I used, and how to modify it with the ingredients I have available.

    This is from www.treehugger.com/homemade-…

    • 1/4 cup shea butter
    • 2 tablespoons coconut oil
    • 3 tablespoons beeswax pellets
    • 3 tablespoons baking soda
    • 2 tablespoons arrowroot flour starch
    • 20 drops lavender essential oil
    • 10 drops tea tree essential oil

    My measurements were all approximate. This was enough to fill two reusable containers, with a tiny bit left over to top up a commercial stick of natural deodorant in a paper tube.

    Method

    Melt everything except the oils in a double-boiler. The beeswax takes a while to melt.

    While it is melting, prepare the containers

    • wind the screw to the bottom
    • remove the bottom plug
    • put on the lid
    • soften the remaining deodorant upside-down in the microwave (~30 s) so it drips into the lid

    When the batch is melted, add the oils. Pour into the prepared containers. Put them in the fridge, or just leave them overnight to set.

    Modifications

    • choose whatever scents you prefer
    • use different proportions
    • leave out the beeswax
    • substitute the arrowroot flour with cornflour
    • add kaolin clay. I tried NZ glacial clay, but it makes the deodorant brownish coloured, and leaves the deodorant with a faint river-bed-dust smell after the oils have evaporated.

    Device driver taxonomy

    • A controller is a piece of hardware that does something: a GPIO, a UART, an I2C controller, etc.

    • A naïve device (NDev) is one that just does what it has to when someone tells it to.

    • A shareable device (SDev) is a wrapper around a naïve device to allow it to be shared between different uses, but only for one purpose at a time.

      Some use-cases for shareable devices

      * A UART can be connected to two different external devices, with hardware to switch
        between the two. One of two different software modules will control the UART, depending on which
        external device it is connected to at the time
      
      • A bus device (BDev) is a driver that

        • An external I2C device needs to be powered down; the I2C pins need to be controlled in a way that does not conform to the I2C specification. The I2C controller module will talk to the UART, but only one at a time, depending on which
    • users at different times.

    There are two type of shareable device

    • Exclusive access shareable, where only one device can use it at at time. If another device is using it, any other device is refused access. Most hardware device can only be used by one thing at a time.

      To use an exclusive access shareable device

      • for each naïve device that must be shareable, the system creates a shareable device and links it to the naïve device

      • each user registers with the shareable device it needs; this gives the user a virtual device

      • to use a virtual device, the user

        • claims() the device; if the device is already in use, the claim is denied.
        • accesses the device, using any of the the methods and attributes of the underlying naïve device. If the user has not successfully claimed the device, the access is denied.
        • release() the device, to allow another use to use it. It it not an error to release a device that you have not claimed.
    • Bus shareable device, where any number of users can request access to the device. When no-one is using it, the device is ‘off’. Any number of users can request access and use it simulataneously. When the last user has finished, it turns ‘off’ again.

      This type of device is often used for power-buses or network interfaces.

      To use a bus shareable device

      • the system creates a bus shareable device

      • each user registers with the shareable device; this gives them each a virtual device

      • to use a bus shareable virtual device, the user

        • request() the bus shareable device; this will return the device status, which is one of

          • ‘wait’ (if the device needs to turn on) or
          • ‘on’ (if the device is already on and can be used immediately.)
        • if the bus shared device is off, it starts the turn-on process:

          • notifies all registrants that it is ‘turning on’
          • do whatever it needs
          • when the device is finally on, it notifies registrants that it is ‘on’.
        • if the request() was to wait, the user can either

          • request() again
          • wait for the ‘on’ notification
          • read the status
        • use the bus shareable device. If it is not ‘on’,

          • do nothing
          • block until on
          • access denied
        • when the user has finished, it release() the device

        • when the last user has release() the bus shareable device, it starts the power-off process

          • notify all registrants that it is ‘turning off’
          • does whatever it needs
          • when the device is off, it notifies registrants that it is ‘off’
        • every user should be prepared for the bus device to turn off unexpectedly; note that, the system will always do the power-off process

        • a user can use a device without request() it. However, if the device it not already on, or turns off, it will either do nothing, or block or deny access; the user must be prepared for this.

    On-line weather data

    I have one of those cheap (-ish) outside-the-house weather-stations that reports the weather data (windspeed, rainfall, etc) over a wireless link to an inside-the-house display unit. My inside-unit is a display only; there is no way to get the data into a PC or an online weather service.

    For a long time, I have wanted to build a device that decodes the wireless data and can then forward that data to an online service.

    A bit of reading showed that my system is a local re-brand of a common Chinese brand. The outside-unit transmit data over a 433 MHz link and it not encrypted or secured in any way.

    One night, I realised that I am unlikely to be the first person in the world to have this idea. A bit more searching showed that there are a number of open-source software-defined radio projects, and, in fact, one, rtl_433 that can decode data from almost all 433 MHz transmitters: garage door openers, alarm systems, and … weather stations. I happened to have a suitable SDR dongle lying around from a previous project.

    Welcome to rtl-weather.

    Still, there is a long way between decoding the weather-station data and getting it to appear on a graph on a website.

    It turns out that a single Linux pipeline can do it. It took a few hours with a bash-terminal to get each part to talk to the next, but we now have weather data

    The pipeline consists of

    • rtl_433, to sniff the 433 MHz band and decode any data it hears, reporting it in JSON format
    • jq, to modify the JSON data into a format suitable for the online service
    • mosquitto_pub, to connect to a remote MQTT broker that accepts the re-formatted JSON data

    I already had access to a Thingsboard IoT system. It would not be very hard to modify the script to also report the data to a service that uses an HTTP API, such as MetOffice Weather Observation Website or OpenWeatherMap.

    It turns out that I had to add two other things to get it to operate properly:

    • mosquitto_pub does not recover if the remote MQTT broker disappears then re-appears. I had to add a ‘watchdog’ process that looks for output from mosquitto_pub that it has actually sent data to the broker. If is does not see this message after a while, it kill the pipeline.
    • a system service (systemd, etc) to ensure that the weather-reporting service starts when the computer starts, and restarts it if it fails.

    I also found that calculating average wind-direction is not the trivial averaging operation – if you naively average east-of-north with west-of-north, you get south. Most IoT service offer only this trivial averaging operation.

    To calculate the average correctly, you need to decompose the direction into N- and E-vector components, average these components, then calculate the average direction from the averaged components.

    One more stage in the pipeline solves this problem, at the expense of additional storage on the server (2 direction components rather than 1 direction angle) and complexity on the server (calculating the direction from the components when display a graph, table, etc).

    On-hold to sort out an airline booking. They are playing terrible music, very loud. With the volume turned down to save my ears, it is hard to hear the voice that interrupts periodically to tell me that my call is important to them. Even turned down, the music is distorted – it must be distorting in their system, not my headphones.

    I assume this is a passive-aggressive, plausibly deniable-attempt to make me give up.

    Monthly computer system checklist

    Well, that was a major update session.

    It is easy to install something to test it. But after a few months, when the excitement has worn off, but the new ‘thing’ is part of your life, it becomes an effort to maintain it. It happens with everything: car, house, exciting new online service.

    The way to keep up it to acknowledge that it is a chore, like vacuuming the carpets, and make a checklist to do every month.

    Monthly checklist

    • Update laptop system software (4 laptops)
    • Update server system software (8 servers)
    • Verify backups are still being made, and can be restored
    • Update manually installed applications and services
      • Thingsboard
      • Matrix (synapse), and various bridges
        • mautrix-whatsapp
        • mautrix-facebook
    • Verify that service providers are up-to-date for the next month
      • Electricity and gas
      • Phone and internet
      • DNS service
      • Email service
      • Virtual-server provider
      • Office-app provider
      • Blog provider
      • Wiki provider
      • File and photo storage
Older Posts →