Reader

#music #radio #tipp #tools #software

I’ve been asked which tools and software I use to handle a large, well-tagged music library. Well, here’s the answer, with some links.

Note: This may appear a bit Linux-biased, but I use almost only Linux systems. Many, if not most, of the mentioned tools are also usable on MacOS or Windows.

Getting the music

• Buy CDs. Yes, still! You even sometimes find mixed lots cheaply and in good condition. Tip: When looking for older albums, the originals are often much better than the so-called “Re-Masters” that are mostly damaged by today’s “Loudness Wars”.
• If you are a known or at least serious DJ or web radio, you often get promo material (FLAC or CD) directly from contact with the bands (or their agents) and membership in promo portals.
• I also like Bandcamp for buying FLAC albums, since they pay the artists a much higher commission than Amazon & Co., and you can find rare pearls of music there.
• Fortunately, over the years a bunch of good music shops have appeared that sell albums in FLAC format. Always prefer lossless FLAC over MP3. If there’s no other choice, buy MP3s only in 320 kbit/s format.
• Don’t trust any “converters” or so-called “legal YouTube downloaders”. It might be tempting, but you’ll only get one thing: Junk!
• Never upscale music into a “better format”, like 128 kbit/s MP3 to 320 kbit/s, or MP3 to FLAC. It just makes garbage larger and doesn’t improve quality in any way. You can convert Apple’s ALAC .m4a, WAVE .wav, WavPak .wv, Monkey’s Audio .ape and OptimFROG .ofr files to FLAC, though—these are lossless conversions.

Ripping CDs

You need a PC with a CD drive. Most modern PCs don’t have one anymore, but a USB drive is fine. Get a good one. Some models have odd drive offsets, can’t switch off error correction, can’t read CD Text, etc., all resulting in possibly bad rips.

The CD Drive Offsets Table on the AccurateRip website can help deciding which drive to get. A low Correction Offset and a high number of Submitted By is preferable.

If you also plan to play or rip movies or music videos, a DVD or Blu-ray drive also usually works.

Regarding software, there are really only two choices here:

• On Linux, use Whipper. You must calibrate your drive once, and you’ll get perfect rips, matched up with MusicBrainz and the AccurateRip database.
• On Windows, use Exact Audio Copy (EAC) and set up a FLAC profile. Almost always perfect rips, also verifies with AccurateRip.

Pre-processing & organizing

This more or less also reflects my workflow, although I prefer FLAC over MP3.

• Become a member of MusicBrainz, the world’s best and largest open music encyclopedia, so you can add and edit album and song data. We all profit from that.
• MP3 Diags — Repair bad MP3 audio files.
• rmlyrics3 — Lyrics3 remover for MP3.
• loudgain — ReplayGain everything, correctly.
• Do not use mp3gain. Never. Period. It is outdated, and can—contrary to popular belief—damage files. It also uses APEv2 tags which nobody wants in their MP3 files.
• MusicBrainz Picard — Tagging, auto-organizing into folders; using enhanced script and extra tags (available). The only tagging software I really trust, and my last step before the audio file never gets touched again. Tip: Connect your Picard to your MusicBrainz account and get your own AcoustId API key.
• Clementine on Debian-based, Strawberry on Arch-based systems — Organize/find music.
• Quod Libet — Organize/find music, ultra-complex searches, preparation of radio playlists using my tools ql-playlists and pl-copyfiles. Very slow importing large amounts of music (like 160,000 songs), but the functionality outweighs that disadvantage.
• Notable mention: For Windows and MacOS users, foobar2000 (not FOSS) can also be a useful tool to organize/find/tag music. Highly customizable, many file formats and add-ons.
• Notable mention: cue_file, part of autocue — Analyses and optionally tags an audio file for cue-in, cue-out and “start next song” points and other loudness-related data, using the EBU R128 algorithms. Can also detect in-song silence (“hidden tracks”) and provide corresponding “early” cue-out and “start next song” points.

Backup!

Now that you have organized and tagged your valuable music, it’s time to set up a good backup strategy. Backup often—you don’t want to lose any of your music!

• Unfortunately, for Windows, I never found a 100% working incremental backup. You could maybe use rsync in WSL2.
• On Linux, anything that uses rsync is perfect for local or remote copies. If you want a GUI, grsync is quite nice, because you can have profiles for different tasks. Incremental backups can save enormous amounts of time and bandwidth!
• For synchronizing your collection between different machines, also remotely, Syncthing has proven to be a reliable tool. I use it to sync more than 240,000 files in 35,000 folders between 14 machines since about 12 years, and not one corruption or lost file since. Impressive.

Playout at home

• Logitech Media Server, now Lyrion, best for hi-end audio, especially when you still own the Slim Devices/Logitech hardware (Transporter, Receiver, Radio, …). Also great with the lightweight squeezelite software player. Squeezer is a good Android remote control for LMS.
• Jellyfin, also for video and eBooks. Apps (recommended) for almost all platforms, even smartphones, tablets, TVs.
• Almost any other audio player should work fine. On some, you have to manually enable the use of “ReplayGain Track Gain” to achieve a consistent loudness.

Radio Station Playout

• AzuraCast — A “radio station in a box”, very well done.
• Liquidsoap — The underlying audio & video streaming language in nearly all web radio server applications.

Live DJ-ing, Broadcasting

• Mixxx — DJ-ing software, also for live transmission. DJ controller support, vinyl record control and much more.
• IDJC — The Internet DJ Console, great for hosted live shows. Uses the JACK audio system, multiple inputs (guest microphones!), phone line support (VoIP/SIP). 3 players, playlist commands, cartwall, MIDI support (I use a Korg nanoPAD2 with it) and much more.
• Studio Display — Studio monitoring, telemetry and signal tower handling; works great with IDJC.
• Notable mention: mAirList (not FOSS, Windows only) — Fantastic radio automation by Torben Weibert, for more professional stations, supports hardware mixers.

That’s about it. Enjoy checking these out, and don’t forget to support the authors of these great Free and Open Source Software programs!

#azuracast #radio #linux #picotts

Many of us want (or are required to have) a periodic time and station ID announcement for our internet radio stations. We could make sound files for every hour and minute and write crazy long scripts to play those at the top of the hour (TOTH), but there is a much easier way:

How about having a simple MP3 file that you can use in your playlists any time and that always has the correct time and station ID?

I will show you how

We will use PicoTTS, a very lightweight and local Text-to-Speech engine, and sox to generate the MP3 file. Using a crontab job, we will regenerate the file every minute, so the time announcement is always current, whenever you play it.

PicoTTS supports British English, American English, German, French, Italian and Spanish. It runs offline and thus can be used on servers with no or restricted Internet access. With a little optimization in sox, the synthesized voices don’t sound too bad. They aren’t as good as online Google TTS using gTTS, but much better than eSpeak, Festival and pyttsx3, which I also tested.

All this will run in the host (so not within AzuraCast’s docker containers), and mount a folder into the station’s media folder that in turn contains the announcement MP3 files, ready to use.

You’ll need

• A working AzuraCast station in a virtual machine or cloud server, running under a Debian derivative, say an Ubuntu 22.04 server.
• SSH access to your server, the bash shell, and the possibility to install software and modify crontab entries on your server.

Prepare the host system

Install PicoTTS and sox

SSH into your server and become root (sudo su). Then install the following:

apt install libttspico-utils sox libsox-fmt-mp3

Install any needed server language packs

Note: For any other language than English, the correct system locale must be installed on your server and specified in the script! Example installation for the German language pack:

apt install language-pack-de

Install and modify the scripts

We will use small bash scripts for each language desired (PicoTTS supports British English, American English, German, French, Italian and Spanish).

Here are sample scripts saytime-en and saytime-de for British English and German, respectively. Modify these as desired, adapt the station ID message, and don’t forget to set the correct time zone for your station. If desired, you can also modify the silence padding (default: 3s of silence at beginning and end).

You can use nano as an editor on the server. Save the scripts into the /usr/local/bin folder.

saytime-en

#!/bin/bash
# saytime-en
# 2015-07-16 Moonbase
# 2023-03-09 Moonbase (update to PicoTTS; pico2wave & sox must be installed)
# Create /tmp/time.en-GB.wav for Liquidsoap to announce current time.
# Put this script into your ~/bin folder (or ~/.local/bin, or /usr/local/bin).
# 2023-05-28 Moonbase Adapted to AzuraCast
# On a Ubuntu Server, install required modules as follows:
# sudo apt install libttspico-utils sox libsox-fmt-mp3
# Install this file into /usr/local/bin and use root's crontab to run every minute.
# 2023-05-29 Moonbase Simplified silence generation
#
# crontab entry, to keep the file current, like so:
#   */1 *  *   *   *     /usr/local/bin/saytime-en > /dev/null

# Possible output languages (might require different sox post-processing):
# de-DE - German
# en-GB - British Englisch
# en-US - US Englisch
# fr-FR - French
# it-IT - Italian
# es-ES - Spanish
lang="en-GB" # language for PicoTTS
locale="en_GB.utf8" # (installed!) OS locale
tz='Europe/Berlin' # desired timezone
# In case you wish to copy the file from /tmp into your media folder, specify it here.
# Use a trailing slash for folder names ("/folder/").
# You can also specify a full MP3 file name ("/folder/time-signal.mp3")
media_folder="/var/azuracast/time/"
# silence to add before/after the spoken text, in seconds; empty string for none
pad="pad 3 3"

# English
AMPM="$(LC_ALL=$locale TZ=$tz date +'%p')"
AMPM="${AMPM^^}" # uppercase it
TIME="$(LC_ALL=$locale TZ=$tz date +'%l:%M ')${AMPM}"
TOTH="$(LC_ALL=$locale TZ=$tz date +"%l o'clock ")${AMPM}"
MIN="$(LC_ALL=$locale TZ=$tz date +'%-M')"
DAY="$(LC_ALL=$locale TZ=$tz date +'%A')"
ZERO="oh" # to pronounce the zero when minutes < 10 ("three oh two PM")

# German
#AMPM="$(LC_ALL=$locale TZ=$tz date +'%p')"
#AMPM="${AMPM^^}" # uppercase it
#TIME="$(LC_ALL=$locale TZ=$tz date +'%-H Uhr %-M')"
#TOTH="$(LC_ALL=$locale TZ=$tz date +'genau %-H Uhr')"
#MIN="$(LC_ALL=$locale TZ=$tz date +'%-M')"
#DAY="$(LC_ALL=$locale TZ=$tz date +'%A')"
#ZERO="Uhr" # to pronounce the zero when minutes < 10 ("three oh two PM")

if [ "$MIN" -lt 10 ]; then
  TIME="$(LC_ALL=$locale TZ=$tz date +'%l') $ZERO $MIN $AMPM"
fi

if [ "$MIN" -eq 0 ]; then
  TIME="$TOTH"
fi

# diag only
#echo $DAY, $TIME

# English
# build text (adapt volume level 0..100)
text='<volume level="100">'"It's $DAY, $TIME, and you're listening to Night Radio.</volume>"

# German
# build text (adapt volume level 0..100), 2s silence at beginning and end
#text='<volume level="100">'"Es ist $DAY, $TIME, und du hörst Nait Räidio.</volume>"


# create WAV audio file using PicoTTS
pico2wave -l=$lang -w=/tmp/time.$lang.wav "$text"

# use sox to make output better understandable (voices are rather muffled)
# we also add some silence at the beginning and end here, if so requested
# adding some treble in the range of +3 to +6 dB helps
# some voices might need a little bass reduction, use s/th like "bass -6 400"
# to avoid clipping, give headroom (gain -h) and reclaim afterwards (gain -r)
# skip the "compand" part if you don’t want compression/limiting.
sox /tmp/time.$lang.wav -c2 -r 44100 -C 128 /tmp/time.$lang.mp3 gain -h bass +2 treble +3 gain -r compand 0.008,0.024 -60,-60,-40,-35,0,-20 10 -60 $pad
# copy it into our media folder
cp /tmp/time.$lang.mp3 "$media_folder"
# diag only: play it loud
#play /tmp/time.$lang.mp3

saytime-de

#!/bin/bash
# saytime-de
# 2015-07-16 Moonbase
# 2023-03-09 Moonbase (update to PicoTTS; pico2wave & sox must be installed)
# Create /tmp/time.en-GB.wav for Liquidsoap to announce current time.
# Put this script into your ~/bin folder (or ~/.local/bin, or /usr/local/bin).
# 2023-05-28 Moonbase Adapted to AzuraCast
# On a Ubuntu Server, install required modules as follows:
# sudo apt install libttspico-utils sox libsox-fmt-mp3
# Install this file into /usr/local/bin and use root's crontab to run every minute.
# 2023-05-29 Moonbase Simplified silence generation
#
# crontab entry, to keep the file current, like so:
#   */1 *  *   *   *     /usr/local/bin/saytime-de > /dev/null

# Possible output languages (might require different sox post-processing):
# de-DE - German
# en-GB - British Englisch
# en-US - US Englisch
# fr-FR - French
# it-IT - Italian
# es-ES - Spanish
lang="de-DE" # language for PicoTTS
locale="de_DE.utf8" # (installed!) OS locale
tz='Europe/Berlin' # desired timezone
# In case you wish to copy the file from /tmp into your media folder, specify it here.
# Use a trailing slash for folder names ("/folder/").
# You can also specify a full MP3 file name ("/folder/time-signal.mp3")
media_folder="/var/azuracast/time/"
# silence to add before/after the spoken text, in seconds; empty string for none
pad="pad 3 3"

# English
#AMPM="$(LC_ALL=$locale TZ=$tz date +'%p')"
#AMPM="${AMPM^^}" # uppercase it
#TIME="$(LC_ALL=$locale TZ=$tz date +'%l:%M ')${AMPM}"
#TOTH="$(LC_ALL=$locale TZ=$tz date +"%l o'clock ")${AMPM}"
#MIN="$(LC_ALL=$locale TZ=$tz date +'%-M')"
#DAY="$(LC_ALL=$locale TZ=$tz date +'%A')"
#ZERO="oh" # to pronounce the zero when minutes < 10 ("three oh two PM")

# German
AMPM="$(LC_ALL=$locale TZ=$tz date +'%p')"
AMPM="${AMPM^^}" # uppercase it
TIME="$(LC_ALL=$locale TZ=$tz date +'%-H Uhr %-M')"
TOTH="$(LC_ALL=$locale TZ=$tz date +'genau %-H Uhr')"
MIN="$(LC_ALL=$locale TZ=$tz date +'%-M')"
DAY="$(LC_ALL=$locale TZ=$tz date +'%A')"
ZERO="Uhr" # to pronounce the zero when minutes < 10 ("three oh two PM")

if [ "$MIN" -lt 10 ]; then
  TIME="$(LC_ALL=$locale TZ=$tz date +'%-H') $ZERO $MIN $AMPM"
fi

if [ "$MIN" -eq 0 ]; then
  TIME="$TOTH"
fi

# diag only
#echo $DAY, $TIME

# English
# build text (adapt volume level 0..100)
#text='<volume level="100">'"It's $DAY, $TIME, and you're listening to Night Radio.</volume>"

# German
# build text (adapt volume level 0..100), 2s silence at beginning and end
text='<volume level="100">'"Es ist $DAY, $TIME, und du hörst Nait Räidio.</volume>"


# create WAV audio file using PicoTTS
pico2wave -l=$lang -w=/tmp/time.$lang.wav "$text"

# use sox to make output better understandable (voices are rather muffled)
# we also add some silence at the beginning and end here, if so requested
# adding some treble in the range of +3 to +6 dB helps
# some voices might need a little bass reduction, use s/th like "bass -6 400"
# to avoid clipping, give headroom (gain -h) and reclaim afterwards (gain -r)
# skip the "compand" part if you don’t want compression/limiting.
sox /tmp/time.$lang.wav -c2 -r 44100 -C 128 /tmp/time.$lang.mp3 gain -h bass +2 treble +3 gain -r compand 0.008,0.024 -60,-60,-40,-35,0,-20 10 -60 $pad
# copy it into our media folder
cp /tmp/time.$lang.mp3 "$media_folder"
# diag only: play it loud
#play /tmp/time.$lang.mp3

Prepare a media folder time

This folder will hold the auto-generated time announcements time.de-DE.mp3 and time.en-GB.mp3. It will later be mounted into your Docker container, as the station’s media/time folder.

mkdir /var/azuracast/time

Test if the scripts work

saytime-de
saytime-en

You should now have two new announcement files in your /var/azuracast/time folder:

ls -l /var/azuracast/time
total 468
-rw-r--r-- 1 root root 193515 Mai 28 16:24 time.de-DE.mp3
-rw-r--r-- 1 root root 180976 Mai 28 16:24 time.en-GB.mp3

Mount the time folder into your station’s media folder

Let’s assume your station is called “niteradio”. Edit the file /var/azuracast/docker-compose.override.yml to include the new volume (but don’t modify other entries that might already be there):

nano /var/azuracast/docker-compose.override.yml

and add the new time folder to the station’s media folder:

services:
  web:
    volumes:
      - /var/azuracast/time:/var/azuracast/stations/niteradio/media/time

Now, restart your station:

cd /var/azuracast
docker compose down
docker compose up -d

In your station, under Media → Music Files, you should now see a new folder time

Open it, and check the files by playing them

You should hear your time and station ID announcements (but not current yet).

Set up crontab to regenerate the announcement every minute

Time for automation! Set up crontab jobs to regenerate the announcement files every minute, so you can forget them and just play them whenever needed.

As root, edit your crontab file and set everything up:

crontab -e

At the end, add the following lines (modify to your needs), and save:

*/1 * * * * /usr/local/bin/saytime-de > /dev/null
*/1 * * * * /usr/local/bin/saytime-en > /dev/null

Don’t worry, PicoTTS is rather leightweight, and will not eat up much CPU, even when regenerating your announcements every minute. And it’s local, so no extra Internet accesses are needed.

Prepare your station to use the announcements

Check if the files play the current time

Check if the cron job works, by playing each “time” announcement a few times. It should play the current time and your station ID, in all languages you have set up.

If something is wrong, check your /usr/local/bin/saytime-* scripts. Use the correct time zone for your station, be sure the system has the needed language pack(s) installed, and you used the correct specifier, i.e. de_DE.utf8.

Set up correct fading

For the time & station ID announcement files, set up correct fading because they will later interrupt the AutoDJ stream. Under Media → Media Files → time, click on Edit for each “time” file, go to Advanced and enter the following values:

This assumes you haven’t changed the default 3s silence padding at the start and end of the files.

Hint: Don’t use the Visual Cue Editor to set up fading for these files, since their lengths change every minute!

Create a Top-of-the-Hour playlist

If you wish to play a time & station ID at the top of the hour (TOTH), you can create a playlist like the following:

Refine and adapt to your needs

You can always adapt and enhance the scripts, remove or add to the station IDs, and create wild playlists using the newly created announcements.

Since we installed PicoTTS, you could in theory now even use Liquidsoap’s pico2wave protocol and let it speak other things in your Liquidsoap script.

Further reading: SVOX Pico Manual.pdf

Enjoy!

Use, modify, steal, enhance, adapt to your heart’s content!

Don’t forget to share this, and possibly even add your great enhancements to the AzuraCast discussion thread.

Tags: #nextcloud #writefreely #linux

Writefreely is a superb, minimalistic blogging platform, but it lacks an easy means of integrating images (and other files). Imagine you could use simple Markdown to include images from your Nextcloud instance, like so:

![Frog](/files/USERNAME/Frog.jpg)

Well, I show you how to! The idea is to mount a special folder from your Nextcloud “Files” section—let’s call it Blog—into your Writefreely instance, under /var/www/writefreely/static/files/USERNAME, using the WebDAV protocol.

For simplicity’s sake, let’s assume that…

• your Nextcloud is at https://CLOUD.EXAMPLE.COM,
• has a valid SSL certificate,
• your username on both Nextcloud and Writefreely is USERNAME,
• you have control over your Writefreely server (i.e., have SSH access).
• you have installed Writefreely in the folder /var/www/writefreely,
• on Debian or a derivative like Ubuntu.

Adapt the following to your needs! Things you need to adapt I’ve noted in UPPERCASE.

Note: Should you run your Writefreely instance in a Proxmox container, you must check FUSE, under Options → Features!

1. Log in to your Writefreely server using SSH. It’s good practice not to log in as root, so we assume a normal user (with sudo rights) here. Just leave out the sudo in the commands listed if you are logged in as root.

2. We most probably need to install FUSE and the DAV file system first:

   sudo apt install fuse davfs2
   

3. Assuming you already have a working Writefreely installed at /var/www/writefreely, and are running it under the www-data user, we now create a mount point for our Nextcloud files:

   cd /var/www/writefreely/static/
   sudo -u www-data mkdir -p files/USERNAME
   sudo chmod -R 755 files/
   

   Note we create the new folders as user www-data (the user Writefreely runs as). Without need, never run anything as the root user! It adds an extra layer of protection.

4. For accessing our Nextcloud data via WebDAV, we need to give the system our Nextcloud credentials (Nextcloud username and password), hidden away in the file /etc/davfs2/secrets, which must have access rights of 600 (-rw-------) and thus be only available to the root user, for security.
   We use

   sudo nano /etc/davfs/secrets
   

   to edit this file, and under the section Credential Line add our mount point and our Nextcloud credentials:

   /var/www/writefreely/static/files/USERNAME USERNAME "MYSECRETPASSWORD"
   

   Using Ctrl+O and Enter we save the changes, then exit using Ctrl+X.

5. Time for testing:

   sudo mount -t davfs https://CLOUD.EXAMPLE.COM/remote.php/dav/files/USERNAME/Blog /var/www/writefreely/static/files/USERNAME -o ro,uid=www-data,gid=www-data
   ls -la /var/www/writefreely/static/files/USERNAME/
   

   All files from your Nextcloud “Blog” folder should appear with uid=www-data and gid=www-data, like so:

   -rw-r--r-- 1 www-data www-data 457744 Mar 27 13:07 Frog.jpg
   -rw-r--r-- 1 www-data www-data 474653 Mar 27 13:52 Gorilla.jpg
   

   (In Nextcloud, I simply copied two of the demo files from “Photos” into my “Blog” folder.)
   Unmount again, using:

   sudo umount /var/www/writefreely/static/files/USERNAME/
   

6. If step 5 went okay, it’s now time to set up your /etc/fstab, so your Nextcloud “Blog” folder will be automatically mounted into your Writefreely instance on every boot:

   sudo nano /etc/fstab
   

   Add the following and be sure not to make any typing mistakes—your Writefreely instance might not boot up anymore if you do:

   https://CLOUD.EXAMPLE.COM/remote.php/dav/files/USERNAME/Blog /var/www/writefreely/static/files/USERNAME davfs ro,uid=www-data,gid=www-data 0 0
   

   Before rebooting, try it out using:

   sudo mount -a
   ls -la /var/www/writefreely/static/files/USERNAME/
   

   The result should be as above in step 5.

7. Reboot and enjoy!
   Your files and images should now be available in Writefreely using simple Markdown commands. Let’s include the Frog.jpg here:

   ![Frog](/files/USERNAME/Frog.jpg)
   

   Result:

Note: Should you run both Nextcloud and Writefreely as separate VMs or containers on a Proxmox host, ýou must assure that Nextcloud is started before Writefreely! Just use Options → Start/Shutdown order and set to appropriate values. I have Nextcloud at 30 and Writefreely at 40, giving Nextcloud 30s to start up.

Note: Since Writefreely can be used as a multi-user blog, I showed you the (marginally) more complicated way using a USERNAME. This makes it easier adding more users and their blogs later on.

Note: The method described here keeps a WebDAV connection open to your Nextcloud instance. It is most appropriate in stable network conditions, but it will break if your Nextcloud sits behind a home router without a fixed IP address (i.e. one that periodically changes its outside IP address). Coping with such situations is a subject for another post, though. (Hint: Use autofs/automount.)

Tags: #selfhosting #nginx #npm #prosody

Let’s assume you run Prosody as your XMPP server, and use Converse.js as your web chat client. Using Prosody’s mod_conversejs, your base URL (say https://chat.example.com:5281/) will show just a boring “Prosody is running” page, and serve the web chat client under https://chat.example.com:5281/conversejs.

What you want instead is your main page directly show the Converse.js web chat under https://chat.example.com/ but still be able to access sub-paths and files in your web root directly (like maybe favicon.ico, or Prosody’s BOSH URL https://chat.example.com/http-bind). Let’s further assume you use the Nginx Proxy Manager (NPM) for increased safety and reverse-proxying to your local machine/container/VM called prosody (resolvable via local LAN DNS).

For your NPM Proxy Host, you’ll need to set up two custom locations—one for setting up some parameters and possibly extra headers, under location /, and another one to redirect only the base URL (without anything following it, like a path or file name) to your conversejs, as location = /. Yes, you read correctly: NPM actually allows using the same syntax you’d use in the Nginx web server configuration files!

Here is how that looks like:

(Instead of prosody as a simple host name—resolved by my local DNS—, you can also use an IP address or a FQDN.)

Now, when visiting https://chat.example.com/, your visitors will be greeted by the Converse.js login screen! And all the rest (like a personal favicon.ico or the BOSH URL) will still work.

NPM developers: Please never take that feature away!

Tags: #selfhosting #letsencrypt #nginx #npm #prosody #python

Especially when self-hosting, the wonderful Nginx Proxy Manager (NPM) is often used as a reverse proxy, and for generating and auto-renewing SSL certificates. But how to export the right certificate to another machine that needs it, like the Prosody XMPP server?

This can get rather complicated, since NPM internally stores its configuration and certificates in numbered files instead of using the domain names for filenames.

Well, here is a possible solution. I run Prosody as an XMPP chat server (not on the Nginx Proxy Manager machine) and need to transfer the SSL certifcates to Prosody. Here is a quick-and-dirty Python implementation of how this could be done.

1. Check all NPM’s numbered .conf files (1.conf, 2.conf, …) for the domain you need the certificate for. The domain name is on the server_name line.
2. Find the location where NPM stores the full chain certificate and the private key. These are on the ssl_certificate and ssl_certificate_key lines, respectively.
3. Open an SSH connection to your target machine (have an SSL key set up; using passwords in clear text is a security risk!) and use sftp to copy the certificate and key files over to your target machine. This will use readable filenames such as chat.mydomain.com.fullchain.pem and chat.mydomain.com.privkey.pem.
4. To avoid file permission problems, run this script once every night, via root’s crontab. Once a day should give enough safety margin, since NPM renews the certificates early enough before their expiry date.

Here is the code I used. It assumes you run NPM in a Docker container, and have your files in user admin’s npm folder. The domain we look for in this case is chat.mydomain.com, and we wish to copy the files to the machine prosody in the local network (resolvable by DNS, otherwise use an IP address), into user myusername’s certs folder. Adapt to your needs.

Note: You might have to install the paramiko and scp Python modules before:

sudo pip3 install paramiko scp

The code is far from perfect, it currently lacks error checking in case the domain or certificate locations aren’t found, but it has been working in daily use for some time now.

#!/usr/bin/python3
import re
import os
from pathlib import Path
from paramiko import SSHClient
from scp import SCPClient
from paramiko import AutoAddPolicy, RSAKey, SSHClient

proxy_hosts_dir = '/home/admin/npm/data/nginx/proxy_host'
archive_dir = '/home/admin/npm/letsencrypt/live'
target_host = 'chat.mydomain.net'
host_file = ''

for filename in os.listdir(proxy_hosts_dir):
    file = os.path.join(proxy_hosts_dir, filename)
    if os.path.isfile(file) and filename.endswith('.conf'):
        with open(file) as f:
            content = f.read();
            m = re.search(r'server_name\s+(.*);', content)
            if m:
              s = m.group(1).split()
              basename = ('_').join(s).replace("*", "x")
              if target_host in s:
                f = re.search(r'ssl_certificate\s+/etc/letsencrypt/live(.*);', content)
                if f:
                  fullchain = archive_dir + f.group(1)
                p = re.search(r'ssl_certificate_key\s+/etc/letsencrypt/live(.*);', content)
                if p:
                  privkey = archive_dir + p.group(1)

client = SSHClient()
client.load_system_host_keys()
client.set_missing_host_key_policy(AutoAddPolicy())
client.connect('prosody', username='myusername')
sftp = client.open_sftp()
sftp.put(fullchain, '/home/myusername/certs/' + target_host + '.fullchain.pem')
sftp.put(privkey, '/home/myusername/certs/' + target_host + '.privkey.pem')
sftp.close()

This script I run via root’s crontab on the NPM machine every night at 3:00 a.m. like so:

# send certificate and key to Prosody XMPP server
0 3 * * * /home/admin/bin/ssl-chat.py

And on the Prosody machine, at 3:10 a.m., also via root’s crontab:

# install the new cert+key sent over from Nginx proxy
10 3 * * * prosodyctl --root cert import chat.mydomain.net /home/myusername/certs/

I use Prosody’s prosodyctl cert update functionality here, since it’s the most reliable way to update certificates with Prosody.

Note: prosodyctl makes backups of both the certificate and key files, so you’ll eventually end up with hundreds of backup files in /etc/prosody/certs. To avoid that, I add an extra delete command in Prosody’s root crontab which will clean out all backups at 3:09 a.m., just before the installation of the new certificate and key:

# set bash as cron shell (below doesn't work with sh)
SHELL=/bin/bash
# remove old certificate and key backups
9 3 * * * rm /etc/prosody/certs/*.{crt,key}.bkp~* > /dev/null

This will leave us with the new certificate and key, plus one backup of each.

Works well so far. Maybe this can help others, too.

Tags: #tipp #tools

Gopher ist ein Internetprotokoll, das Anfang der 1990er-Jahre als Nachfolgedienst für FTP angedacht war – noch vor dem WWW und HTTP. Damit können Kataloge in Menü-Strukturen aufgelistet und Links verfolgt werden. Die Anzeige erfolgt im Textmodus.

Leider gibt es nahezu keine Suchmaschinen wie das damalige Veronica mehr, allerdings betreibt René Gabel noch den gopherspace.de.

Ich habe mich entschieden, unseren Blog-Server auch im Gopherspace bereitzustellen. Wer also etwas Nostalgie live erleben möchte, darf uns gerne auf gopher://blog.syvi.net/ besuchen!

Ich empfehle, lynx statt des üblichen gopher als Client zu benutzen, damit ist die Darstellung um einiges schöner. Aufruf mit:

lynx gopher://blog.syvi.net/

Viel Spaß beim Anschauen, wie es »damals« war!

Tags: #literatur #programmierung #tipp

A man is not dead while his name is still spoken.   ⸺ Going Postal, Chapter 4 prologue

About the Signal

X-Clacks-Overhead is a non-standardised HTTP header based upon the fictional work of the late, great, Sir Terry Pratchett.

In Terry Pratchett's science-fantasy Discworld series, “The Clacks” is a network infrastructure of Semaphore Towers, that operate in a similar fashion to telegraph – named “Clacks” because of the clicking sound the system makes as signals send.

In Sir Terry’s novel “Going Postal”, the story explains that the inventor of the Clacks – a man named Robert Dearheart, lost his only son in a suspicious workplace accident, and in order to keep the memory of his son alive, he transmitted his son’s name as a special operational signal through the Clacks to forever preserve his memory: GNU John Dearheart

G: Send the message onto the next Clacks Tower. N: Do not log the message. U: At the end of the line, return the message.

The nature of the 'GNU' code would keep the name of his son forever transmitting through The Clacks network so long as the network still existed.

GNU on the Internet

As a way to preserve the memory of Sir Terry Pratchett, the users of the SubReddit for the Discworld series came up with the idea behind the X-Clacks-Overhead HTTP Header. It allows web authors to silently commemorate someone through the use of a non-invasive header that can be transmitted from server to server, or server to client without operational interference.

At the time of writing, Mozilla.org (makers of the Firefox web browser), the makers of Debian (a popular Linux Operation System), and Xml.com (a major repository of standards information) are examples of some of the backbones of the Internet who transmit the Signal “GNU Terry Pratchett”. We do, too.

You would only know the header is present if you analysed the transmission headers of your content requests on web sites serving the header.

Visualise it in Firefox

Using the Firefox Extension X-Clacks-Overhead by Sixthhokage1, a bug-fixed fork of the original extension GNU_Terry_Pratchett by Rob Grundy, you can visualise the X-Clacks-Overhead content. The extension’s little icon will even flash the original Clacks code at you! If you mouse over the icon, the Clacks overhead message will be displayed.

Read more

• The X-Clacks-Overhead site shows sites carrying the signal, and allows to add your own.
• Visit GNU Terry Pratchett to find out how to include the signal on your site, blog, email, etc.
• Buy the “Going Postal” paperback at Amazon.de (affiliate link).

Keep the legacy of Sir Terry Pratchett and other beloved ones alive forever. For as long as their names are still passed along the Clacks, Death can’t have them.

Note to my children: I’d love to know you including my name in the X-Clacks-Overhead of your sites and blogs, whenever I can’t be with you anymore…

Wovon machen wir es abhängig, etwas mit »gut« oder »schlecht« zu bewerten? Ich denke, dieses Verhalten ist in jedem von uns tief verankert. Schon von Anbeginn der Zeit waren wir dazu gezwungen, Dinge in »gut« und »schlecht« zu kategorisieren. Ist es gut, der Frau einen mit der Keule überzuziehen und sie mitzunehmen? Ist es schlecht, vor dem Säbelzahntiger wegzulaufen?

Tags: #about #fediverse #opensource #pin

Dieses Blog wird mit der OpenSource-Software Writefreely betrieben und ist bei mir in Deutschland gehostet. Du kannst ein solches Blog ebenfalls anlegen: Frage mich per E-Mail oder auf cloud.syvi.net nach einem Einladungs-Link. Die Nutzung ist kostenlos, jedoch sind solche Projekte auf Spenden angewiesen.

Die Idee des Minimal Blogging ist, mit einer maximalen Konzentration auf die Arbeit am Text zu bloggen, ohne einen Überschuss an Funktionalität.

Tags: #lernen #radio #tipp #webdesign #writefreely

Lasst mich vorausschicken, dass ich Einbettungen fremder Inhalte per Iframe eigentlich nicht mag. Man muss immer darauf achten, dass diese auch verfügbar sind, keine gesetzwidrigen Inhalte beherbergen und einen nicht aus­spionieren. Okay, sind wir mal ehrlich: Das mit dem Aus­spionieren lässt sich heutzutage nicht mehr vermeiden, wenn man einbettet, DSGVO hin oder her.

Nichtsdestotrotz ergibt sich hin und wieder einfach die Notwendigkeit, etwas einzubetten, z.B. ein YouTube-Video – einfach, um den eigenen Blog anzureichern, auf eigene Inhalte zu verweisen oder überhaupt interessant genug zu bleiben.

Nun reichen leider oft die <iframe …>-Einbettungs-Codes der Video-Anbieter nicht aus, um eine ordentliche, responsive Darstellung im eigenen Blog zu ermöglichen. Hier ist die Lösung. Wir definieren einfach im CSS unseres Blogs folgende Ergänzung für einen Video-Container:

/* Responsive Video Container */
.video-container {
  overflow: hidden;
  padding-top: 56.25%; /* 16:9 */
  position: relative;
} 

.video-container iframe,
.video-container object,
.video-container embed {
  position: absolute;
  top: 0;
  left: 0;
  border: 0;
  width: 100%;
  height: 100%;
}

Im Beispiel-CSS gehen wir von Videos im 16:9-Format aus. Der notwendige Wert für padding-top errechnet sich aus

Wert = y / x * 100

also bspw. für 4:3-Videos als

Wert = 3 / 4 * 100
     = 75

Wir würden in diesem Fall oben einsetzen:

  padding-top: 75%; /* 4:3 */

Danach können wir die Einbettungs-Codes der Anbieter einfach in einem <div …> »einklammern«:

<div class="video-container"><iframe …></iframe></div>

Im Folgenden findet ihr ein paar Beispiele dazu.

YouTube-Video

Einbettungs-Code von YouTube, geklammert in unserem neuen <div>:

<div class="video-container"><iframe width="560" height="315" src="https://www.youtube.com/embed/P_jlzzzn8Rw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe></div>

Ergebnis

Vimeo-Video

Einbettungs-Code von Vimeo, geklammert in unserem neuen <div>:

<div class="video-container"><iframe src="https://player.vimeo.com/video/80836225?h=caeadc66bb&badge=0" width="640" height="360" frameborder="0" allow="autoplay; fullscreen; picture-in-picture" allowfullscreen></iframe></div>

Ergebnis

PeerTube-Video (aus dem Fediverse)

Einbettungs-Code von PeerTube, geklammert in unserem neuen <div>:

<div class="video-container"><iframe title="The Intelligent Plant – Scroll Website" src="https://tube.kh-berlin.de/videos/embed/0df53e9a-6e79-4a0a-b041-bb2449bdb35b" allowfullscreen="" sandbox="allow-same-origin allow-scripts allow-popups" width="560" height="315" frameborder="0"></iframe></div>

Ergebnis

Radio-Player

Zum Einbetten des Radio-Players einer AzuraCast-Station ist außer dem Einbettungscode des Anbieters nichts weiter nötig:

<iframe src="https://demo.azuracast.com/public/azuratest_radio/embed?theme=light" frameborder="0" allowtransparency="true" style="width: 100%; min-height: 150px; border: 0;"></iframe>

Ergebnis

Webseite am Beispiel eines Radio-Sendeplans

Bei AzuraCast-Stationen kann auch hier einfach der angebotene Einbettungs-Code verwendet werden:

<iframe src="https://demo.azuracast.com/public/azuratest_radio/schedule/embed?theme=light" frameborder="0" allowtransparency="true" style="width: 100%; min-height: 800px; border: 0;"></iframe>

Ergebnis

Für die ganz Wilden …

Man kann sogar die o.g. noch in ein <figure>-Element einpacken, um z.B. eine schicke Bildunterschrift und ggf. eine Quellenangabe hinzuzufügen!

<figure><div class="video-container"><iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/n_GFN3a0yj0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe></div><figcaption>AC/DC live at River Plate, Dezember 2009 (YouTube)</figcaption></figure>

Ergebnis

---

Viel Spaß beim Erstellen eines noch lebendigeren Blogs!

Ein paar Grundlagen

Dies ↑ ist eine Überschrift der Ebene 2.

Absätze

Wenn man einfach. Nur schreibt. Gibt das einen langen halbfetten Absatz!

Doppelte Zeilenschaltung hingegen macht einen neuen Absatz!

Aufzählungen (ungeordnet)

Das macht man einfach mit einem Divis (-):

• Liste eins
• Liste 2
  • Verschachtelt geht natürlich auch (a)
  • Verschachtelt (b)
• Längere Absätze werden korrekt umbrochen: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim.
  • Phasellus viverra nulla ut metus varius laoreet. Quisque rutrum. Aenean imperdiet. Etiam ultricies nisi vel augue. Curabitur ullamcorper ultricies nisi. Nam eget dui. Etiam rhoncus.
• usw.

Aufzählung (geordnet)

1. Liste eins
2. Liste 2
   1. Verschachtelt geht natürlich auch (1.1)
   2. Verschachtelt (1.2)
3. Mal gucken, was mit Buchstaben passiert! a. (3a) b. (3b)
4. usw.

Code Blocks

Benutzt man, um anderen zu zeigen »wie sowas programmiert wird«.

So geht **halbfett** in Markdown.

Angefixt?

Mehr lesen auf der Seite von Daring Fireball.

Einfache Einbindung eines Bildes mit Markdown

Einfache Einbindung einer »Figure« mit HTML

Die Welt ist am Ende, die Menschheit vor dem Abgrund, und mir geht's auch schon ganz schlecht. Aber was soll's?

Udo Jürgens sang mal: Wir haben alles im Griff auf dem sinkenden Schiff. Recht hatte der Mann. Viel richtig machen können wir eh nicht mehr, machen wir also ruhig alles falsch. Ändern werden wir nichts mehr.

Is it that time of year again? Mice seeking world domination?

#Watt #Würmer #Energie #Intellenz

Schön wär's ja! Noch gibt es von denen nämlich wahrscheinlich genügend. Hoffen wir es zumindest …

Aber wie das so ist: Ich hatte eigentlich (jajaja, ich soll nicht so oft 'eigentlich' verwenden!) vor, etwas Lustiges zu schreiben. Da kommt dann sowas bei raus!

Zur Sache, Schätz… (nein, genug der Zitate) … Ich schreibe also so meine Gedanken vor mich hin. Vielleicht zeigt sich hier irgendwann einmal – auch wenn ich selbst wenig Hoffnung habe – irgendetwas Intellentes! (Wer Schreibfehler findet, darf sie behalten.)

Nun also, allons-y, Alonso!

gez. Die linke Hand des Teufels

Tags: #fun #random

Nun, auf jeden Fall wäre es abends deutlich dunkler. Womöglich auch kälter. Immerhin bringt uns ja so ein Wurm ein Watt. Mit vierzig davon haben wir schon eine kleine Küchenfunzel.

Wenn wir ganz fleißig sammeln und die lieben Tierchen gut füttern, bekommen wir vielleicht sogar 1.200 zusammen – das reicht dann sogar zum Braten einiger Eierpfannkuchen. Die ausgebrannten Wattwürmer können wir gleich als Fleisch­beilage benutzen. Seit dem Vegan-Wahn gibt es ja außer zermahlener Pappe und Fleisch­ersatz sonst nichts mehr Echtes.

Wird es bald dunkel an der Nordsee?

Das könnte passieren. Ein durchschnittlicher 2–3–Personen-Haushalt kommt ja auf einen Durchschnittsverbrauch von rd. 3.500 Kilo­watt­stunden im Jahr. Das sind sozusagen 3.500 x 1.000 = 3.500.000 Wurm­stunden, also 3,5 Millionen Würmer, die eine Stunde lang richtig Gas geben müssen!

Wenn man davon ausgeht, dass man an einem guten Tag allein etwa 100 von den edlen Tierchen aus ihren Hügeln ausbuddeln kann, müssen für das Herbeischaffen dieser von nur einer Familie täglich benötigten Menge von 9.589 Wattwürmern bereits rund 96 Leute den ganzen Tag Würmer buddeln.

Ostfriesland – völlig zertorft?

Wenn man sich mal den Strand so anguckt, von Emden bis Harlinger Siel, dann kann das nicht lange dauern, bis zumindest die ostfriesischen Strand­gebiete völlig zertorft sind. Das schafft wohl nicht nur Bagger­fahrer Hannes Kabeltod.

Also auch kein guter Plan. Ich mein, klar, lieber Wattwürmer als Touristen, nech. Wat wär’ unser Ostfriesland doch schön, so ganz ohne Touris und dafür mit ein paar mehr Wattwurmhaufen. Gemütlich, nicht so überlaufen und nur für uns. Platt wird geschnackt, man kennt sich und die zugereisten Krimi-Autoren werfen wir auch gleich mit raus.

Aber was tun?

Klar, zurück auffe Hallig. Bei Kerzenschein. Ab und an ’ne Runde boßeln und den Lokalmatador feiern. Wunderschön für die, die das genießen können. Okay, okay, die meisten Halligen sind in Nordfriesland, und man ist sich ja nicht immer ganz grün. Trotzdem schön da.

Dumm ist nur, dass wir von etwas leben müssen. Der Rum mit Tee, das Schnäpschen mit dem Nachbarn und das Essen müssen ja auch bezahlt werden. Auto braucht man vielleicht nicht unbedingt – wer erst einmal in Friesland lebt, will da sowieso nicht mehr weg. Und bis zum Dorfkrug reicht auch das Fahrrad.

Aber Internet und Telefon brauchen wir. Vielleicht auch mal die Post, am besten ohne Inselzuschlag.

Mist. Schöner Traum und doch weiter wie bisher. Aber wenigstens im Radio läuft ab und zu noch das Lied von den Wattwürmern.

Tschüss!