Radio Channel Logger

From W9CR
Jump to navigation Jump to search

It seems to be a common requirement to make recordings of a radio or scanner output. I've not found much decent Free Software that can do this, and do it reliably in a headless configuration. What I've used is documented here, and will do 4 audio channels reliably. This soultion is unique as it uses OpenAI's API to trasnscribe the audio files to easily searchable text.

My concept is based around a RPI 5 with Cmedia USB audio dongles as pictured below. This is compact and low power, while a full Linux environment allows you to do many other things as needed.


Features

  • 4 audio channels
  • Date/time logging down to the seconds
  • Configuration of settings per channel
  • Recordings are 8000 samples per second 16 bit WAV
  • VOX activated recording with VOX level set per channel
  • Low Overhead on as no voice coding is taking place, should run on a pi0 for at least one channel (not tested)
  • 1tb ssd used for recording.

Hardware

RPI 5

Radio

Per channel

  • CDM radio
  • CDM radio interface cable

radio setup

ensure the low level expansion is set in the personality and that demphasis is selected.

Linux

The system is standard raspbian Linux

What is different is that we are booting off the nvme directly.

Disk Setup

This is not a complete howto, but rather enough to figure it out. You may want to do it differently.

GPT table

Part 1 is /boot/firmware

Part 2 is lvm Leave a little at the end free, I picked 11gb 

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1050623   512.0 MiB   0700  Microsoft basic data
   2         1050624      1930430463   920.0 GiB   8E00  Linux LVM

LVM config

From here I make that part 2 a LVM phsyical volume with 'pvcreate', then make the entire thing a volume group with vgcreate 

pvcreate /dev/nvme0n1p2
vgcreate vg0 /dev/nvme0n1p2

# pvs
  PV             VG  Fmt  Attr PSize    PFree
  /dev/nvme0n1p2 vg0 lvm2 a--  <920.00g    0

# vgs
  VG  #PV #LV #SN Attr   VSize    VFree
  vg0   1   4   0 wz--n- <920.00g    0

Next we need make the swap thick LV of 8gb 

lvcreate --name SWAP --size 8GiB vg0

Now make a the rest into a pool for the thin volumes 

lvcreate --type thin-pool --extents +100%FREE --name alberca vg0

Last make a root volume and then one for storing the recordings 

lvcreate --type thin --virtualsize 128GiB --name ROOT --thinpool alberca vg0
lvcreate --type thin --virtualsize 128GiB --name recordings --thinpool alberca vg0

# lvs
  LV         VG  Attr       LSize    Pool    Origin Data%  Meta%  Move Log Cpy%Sync Convert
  ROOT       vg0 Vwi-aotz--  128.00g alberca        7.18
  SWAP       vg0 -wi-ao----    8.00g
  alberca    vg0 twi-aotz-- <911.77g                1.14   10.75
  recordings vg0 Vwi-aotz--   50.00g alberca        2.37

Filesystem setup

Make some file systems on the root 

mkfs.ext4 /dev/mapper/vg0-ROOT
mkfs.ext4 /dev/mapper/vg0-recordings
mkswap /dev/mapper/SWAP

Copy the sdcard to the filesystems

so you can use cp or rsync to copy the file systems into the new place, but i used dd. dd is not ideal as you need to tune the file systems to change the UUID's. 

dd bs=4M conv=sparse if=/dev/mmblockroot of=/dev/mapper/ROOT
dd bs=4M  if=/dev/mmblockboot of=/dev/nvme0n1p1

Change the UUID's of the file systems 

tune2fs -U $(uuidgen) /dev/vg0/ROOT
mlabel -n -i /dev/nvme0n1p1
resize2fs /dev/vg0/ROOT

get the new UUID's 

lsblk -f

mount these in /mnt and chroot to them 

mount /dev/vg0/ROOT /mnt
mount /dev/nvme0n0p1 /mnt/boot/firmware
mount -t proc /proc /mnt/proc/
mount --rbind /sys /mnt/sys/
mount --rbind /dev /mnt/dev/
chroot /mnt

Change fstab to boot off nvme 

cat /etc/fstab
proc            /proc           proc    defaults          0       0
UUID=1B01-6969  /boot/firmware  vfat    defaults          0       2
UUID=d185ebc1-68df-4065-afe6-5ed288b28e32  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
UUID=01b68d2f-9577-4e63-b79f-6eaea8bbd61c       swap    swap    defaults        0       0
UUID=d5d62f0a-054b-4f9b-a615-4c1b02373fdc       /home/svar      ext4    defaults,noatime  0       1

disable swap in the chroot 

sudo systemctl disable dphys-swapfile.service

exit the chroot and reboot. remove the sdcard and confirm you're booting off the LVM.

Linux config

I set the timezone to local time and then configure the users/ssh keys. If this will be remote it's a good idea to setup the ssh remote service for remote access. I have a page on this here Secure Tunnel Service

The user I use for this is 'svar' and must be added to the audio group 

usermod svar -a -G audio 

# groups svar
svar : svar audio users

udev rules for ports

I identify the ports as upper and lower, left and right on the RPI. This should scale up to 32 devices max.

this will set the device properly using the name as below 

plughw:top-left,0

/etc/udev/rules.d/50-alsa.rules
SUBSYSTEM!="sound", GOTO="my_usb_audio_end"
ACTION!="add", GOTO="my_usb_audio_end"
# top right port 1-2:1.0
DEVPATH=="*/1-2:1.0/sound/card*", ATTR{id}="top-right"
# bottom right 3-2:1.0
DEVPATH=="*/3-2:1.0/sound/card*", ATTR{id}="bottom-right"
# top left port 3-1:1.0
DEVPATH=="*/3-1:1.0/sound/card*", ATTR{id}="top-left"
# bottom left port 1-1:1.0
DEVPATH=="*/1-1:1.0/sound/card*", ATTR{id}="bottom-left"

LABEL="my_usb_audio_end"

Software

This uses this great software 'svar' https://github.com/arkq/svar

compile and install it as /usr/loca/bin/svar

systemd scripts

This runs as a script under systemd which will set the mic gain and the then invoke svar on each channel per it's config file. if it crashes, it waits 5 seconds and restarts it. 

/etc/systemd/system/svar@.service
[Unit]
Description=Start recording from %i
After=network.target

[Service]
User=svar
EnvironmentFile=/etc/default/svar@%i
ExecStartPre=/usr/bin/amixer -D hw:${HW} -n sset Mic capture ${CAPLEVEL}
ExecStart=/usr/local/bin/svar -v --device=plughw:%i,0 -l${VOXLEVEL} --rate=${RATE} -s ${SEC} --fadeout-lag ${FOLAG} ${DIR}/${FNAME}
# Restart every >2 seconds to avoid StartLimitInterval failure
RestartSec=5
Restart=always

[Install]
WantedBy=multi-user.target

Config

directory config

as svar you will need to make the following directories 

mkdir -p /home/svar/rec/top-right/current
mkdir -p /home/svar/rec/top-left/current 
mkdir -p /home/svar/rec/bottom-right/current
mkdir -p /home/svar/rec/bottom-left/current

service config file

for each channel being recorded a separate service is used with it's own config 

/etc/default/svar@top-right
#audio level for capture
CAPLEVEL="0" 
#directory to put the files in
DIR="/home/svar/rec/top-right/current"
#filename BR/TR/TL/BL based on port
FNAME="TR-%Y-%m-%d-%H%M.%S"
#this sets the VOX level 
VOXLEVEL="1"
RATE="8000"
#seconds is the number of seconds to break up
#FOLAG is the lag in milliseconds and needs to match the seconds to ensure there is no
# clipping of silence in the output
SEC="10"
FOLAG="10000"

Start your services

systemctl enable svar@top-right.service
systemctl start  svar@top-right.service
systemctl status svar@top-right.service

systemctl enable svar@top-left.service
systemctl start  svar@top-left.service
systemctl status svar@top-left.service

systemctl enable svar@bottom-right.service
systemctl start  svar@bottom-right.service
systemctl status svar@bottom-right.service

systemctl enable svar@bottom-left.service
systemctl start  svar@bottom-left.service
systemctl status svar@bottom-left.service

you should see all the audio dongles blinking if svar is active. 

# service svar@* status
● svar@top-left.service - Start recording from top-left
     Loaded: loaded (/etc/systemd/system/svar@.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-02-12 18:24:38 CST; 1 day 5h ago
   Main PID: 6873 (svar)
      Tasks: 3 (limit: 9574)
        CPU: 17min 46.500s
     CGroup: /system.slice/system-svar.slice/svar@top-left.service
             └─6873 /usr/local/bin/svar -v --device=plughw:28,0 -l1 --rate=8000 -s 10 --fadeout-lag 10000 /home/svar/rec/top-left/current/TL-%Y-%m-%d-%H%M.%S

Feb 12 18:24:38 DonPie systemd[1]: Starting svar@top-left.service - Start recording from top-left...
Feb 12 18:24:38 DonPie amixer[6871]: Simple mixer control 'Mic',0
Feb 12 18:24:38 DonPie amixer[6871]:   Capabilities: pvolume pvolume-joined cvolume cvolume-joined pswitch pswitch-joined cswitch cswitch-joined
Feb 12 18:24:38 DonPie amixer[6871]:   Playback channels: Mono
Feb 12 18:24:38 DonPie amixer[6871]:   Capture channels: Mono
Feb 12 18:24:38 DonPie amixer[6871]:   Limits: Playback 0 - 127 Capture 0 - 16
Feb 12 18:24:38 DonPie amixer[6871]:   Mono: Playback 0 [0%] [0.00dB] [off] Capture 0 [0%] [0.00dB] [on]
Feb 12 18:24:38 DonPie systemd[1]: Started svar@top-left.service - Start recording from top-left.
Feb 13 09:50:00 DonPie svar[6873]: info: Creating new output file: /home/svar/rec/top-left/current/TL-2025-02-13-0950.00.wav

● svar@bottom-right.service - Start recording from bottom-right
     Loaded: loaded (/etc/systemd/system/svar@.service; enabled; preset: enabled)
     Active: active (running) since Thu 2025-02-13 12:18:21 CST; 11h ago
    Process: 12137 ExecStartPre=/usr/bin/amixer -D hw:${HW} -n sset Mic capture 0 (code=exited, status=0/SUCCESS)
   Main PID: 12138 (svar)
      Tasks: 3 (limit: 9574)
        CPU: 7min 3.715s
     CGroup: /system.slice/system-svar.slice/svar@bottom-right.service
             └─12138 /usr/local/bin/svar -v --device=plughw:31,0 -l1 --rate=8000 -s 10 --fadeout-lag 10000 /home/svar/rec/bottom-right/current/BR-%Y-%m-%d-%H%M.%S

Feb 13 12:18:21 DonPie systemd[1]: Starting svar@bottom-right.service - Start recording from bottom-right...
Feb 13 12:18:21 DonPie amixer[12137]: Simple mixer control 'Mic',0
Feb 13 12:18:21 DonPie amixer[12137]:   Capabilities: pvolume pvolume-joined cvolume cvolume-joined pswitch pswitch-joined cswitch cswitch-joined
Feb 13 12:18:21 DonPie amixer[12137]:   Playback channels: Mono
Feb 13 12:18:21 DonPie amixer[12137]:   Capture channels: Mono
Feb 13 12:18:21 DonPie amixer[12137]:   Limits: Playback 0 - 127 Capture 0 - 16
Feb 13 12:18:21 DonPie amixer[12137]:   Mono: Playback 0 [0%] [0.00dB] [off] Capture 0 [0%] [0.00dB] [on]
Feb 13 12:18:21 DonPie systemd[1]: Started svar@bottom-right.service - Start recording from bottom-right.

● svar@top-right.service - Start recording from top-right
     Loaded: loaded (/etc/systemd/system/svar@.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-02-12 18:24:38 CST; 1 day 5h ago
   Main PID: 6864 (svar)
      Tasks: 3 (limit: 9574)
        CPU: 18min 6.667s
     CGroup: /system.slice/system-svar.slice/svar@top-right.service
             └─6864 /usr/local/bin/svar -v --device=plughw:30,0 -l1 --rate=8000 -s 10 --fadeout-lag 10000 /home/svar/rec/top-right/current/TR-%Y-%m-%d-%H%M.%S

Feb 13 20:45:52 DonPie svar[6864]: info: Creating new output file: /home/svar/rec/top-right/current/TR-2025-02-13-2045.52.wav
Feb 13 20:54:56 DonPie svar[6864]: info: Creating new output file: /home/svar/rec/top-right/current/TR-2025-02-13-2054.56.wav
Feb 13 22:54:15 DonPie svar[6864]: info: Creating new output file: /home/svar/rec/top-right/current/TR-2025-02-13-2254.15.wav
Feb 13 22:56:06 DonPie svar[6864]: info: Creating new output file: /home/svar/rec/top-right/current/TR-2025-02-13-2256.06.wav
Feb 13 23:03:24 DonPie svar[6864]: info: Creating new output file: /home/svar/rec/top-right/current/TR-2025-02-13-2303.24.wav
Feb 13 23:11:22 DonPie svar[6864]: info: Creating new output file: /home/svar/rec/top-right/current/TR-2025-02-13-2311.22.wav
Feb 13 23:12:33 DonPie svar[6864]: info: Creating new output file: /home/svar/rec/top-right/current/TR-2025-02-13-2312.33.wav
Feb 13 23:16:16 DonPie svar[6864]: info: Creating new output file: /home/svar/rec/top-right/current/TR-2025-02-13-2316.16.wav
Feb 13 23:18:50 DonPie svar[6864]: info: Creating new output file: /home/svar/rec/top-right/current/TR-2025-02-13-2318.50.wav
Feb 13 23:21:41 DonPie svar[6864]: info: Creating new output file: /home/svar/rec/top-right/current/TR-2025-02-13-2321.41.wav

● svar@bottom-left.service - Start recording from bottom-left
     Loaded: loaded (/etc/systemd/system/svar@.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-02-12 18:24:38 CST; 1 day 5h ago
   Main PID: 6874 (svar)
      Tasks: 3 (limit: 9574)
        CPU: 16min 49.254s
     CGroup: /system.slice/system-svar.slice/svar@bottom-left.service
             └─6874 /usr/local/bin/svar -v --device=plughw:29,0 -l1 --rate=8000 -s 10 --fadeout-lag 10000 /home/svar/rec/bottom-left/current/BL-%Y-%m-%d-%H%M.%S

Feb 12 18:24:38 DonPie systemd[1]: Starting svar@bottom-left.service - Start recording from bottom-left...
Feb 12 18:24:38 DonPie amixer[6872]: Simple mixer control 'Mic',0
Feb 12 18:24:38 DonPie amixer[6872]:   Capabilities: pvolume pvolume-joined cvolume cvolume-joined pswitch pswitch-joined cswitch cswitch-joined
Feb 12 18:24:38 DonPie amixer[6872]:   Playback channels: Mono
Feb 12 18:24:38 DonPie amixer[6872]:   Capture channels: Mono
Feb 12 18:24:38 DonPie amixer[6872]:   Limits: Playback 0 - 127 Capture 0 - 16
Feb 12 18:24:38 DonPie amixer[6872]:   Mono: Playback 0 [0%] [0.00dB] [off] Capture 0 [0%] [0.00dB] [on]
Feb 12 18:24:38 DonPie systemd[1]: Started svar@bottom-left.service - Start recording from bottom-left.
Feb 13 09:49:39 DonPie svar[6874]: info: Creating new output file: /home/svar/rec/bottom-left/current/BL-2025-02-13-0949.39.wav

SAMBA Config

This makes a file share so that you can access the recordings on the system from another computer/windoze.

Install software 

apt-get install samba smbclient wsdd

Windoze 11 note If you're running windoze 11, you will need to enable this in powershell. M$ has fucked their shit up once again for no good reason. 

Set-SmbClientConfiguration -RequireSecuritySignature $false



/etc/samba/smb.conf

[global]
   workgroup = WORKGROUP
   security = user
   netbios name = RecPie
   server string = RECORDER
   client min protocol = SMB2 
   server signing = mandatory
   client signing = mandatory
   interfaces = 127.0.0.0/8 eth0
   bind interfaces only = yes
   log file = /var/log/samba/log.%m
   max log size = 1000
   logging = file
   panic action = /usr/share/samba/panic-action %d
   server role = standalone server
   obey pam restrictions = yes
   unix password sync = yes
   passwd program = /usr/bin/passwd %u
   passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
   pam password change = yes
   map to guest = Bad Password
   usershare allow guests = yes
[recorded]
    comment = recorder files
    path = /home/svar/rec
    read only = yes
    public = yes
    available = yes
    browsable = yes
    guest ok = yes

Transcription via OpenAI service

I've made a script that will use the wisperAI service to transcribe files into plain text and json format. It also sorts them into a year/month/day directory hierarchy, and converts the wave files to 24 kbit/s Opus format which is about 20% the size of the wave format. This allows up to a 2:22.22 long file to be automatically transcribed. Anything longer will fail, and there's no support to split them up, so it's suggested that the $SEC and $FOLAG be touched to break files up.

This is then automated via systemd scripts that monitor the directories and fire off the transcription script in the background. The only process that takes some CPU is opus encoding, but this is run with a 15 nice value to ensure there's no dropping of audio.

transcribe.sh

This is the script, it takes the basedir as it's only argument. It's designed so that the source files will not be moved until all operations complete.

https://github.com/W9CR/radiologger

Systemd config

There are two files needed, one to monitor the path and another to run when the path changes

transcribe@.path

cat >/etc/systemd/system/transcribe@.path <<'EOF' 
[Unit]
Description=Monitor folder %i/current for changes and transcribe files

[Path]
PathChanged=/home/svar/rec/%i/current/

[Install]
WantedBy=multi-user.target
EOF

transcribe@.service

cat >/etc/systemd/system/transcribe@.service <<'EOF'
[Unit]
Description=Transcribe recording from /home/svar/rec/%i/current/

[Service]
Type=oneshot
User=svar
#EnvironmentFile=/etc/default/
ExecStart=/bin/bash /home/svar/scripts/transcriber.sh /home/svar/rec/%i
EOF

Enable Service

systemctl enable transcribe@top-left.path
systemctl enable transcribe@top-right.path
systemctl enable transcribe@bottom-left.path
systemctl enable transcribe@bottom-right.path

systemctl start transcribe@top-left.path
systemctl start transcribe@top-right.path
systemctl start transcribe@bottom-left.path
systemctl start transcribe@bottom-right.path

journalctl -f -u transcribe@top-left.service

todo

  • opus support - opusenc --speech --bitrate 24 gives almost the same audio for 20% of the file size.
  • speech to text
  • moving files into other directories via cron
  • script to make searching text files easier
  • script to compute the number of seconds per day/week/month/year on a channel
Retrieved from "https://wiki.w9cr.net/index.php?title=Radio_Channel_Logger&oldid=8372"