Autodictating to self using Whisper to preserve privacy

Whisper is a very nice bit of code released by OpenAI, the kind people who brought us ChatGPT.  It’s a speech to text tool that can handle a huge array of languages and runs locally, as in on your hardware with your data.  There’s an API you can use on their servers, but only if you are sure the audio files and text can be released to the public.  Never put any data on anyone else’s hardware that you wouldn’t want to have leaked on pastebin or published in the New York Times; that goes for all services including gmail, Outlook, Office 365, etc.  Never, ever use someone else’s hardware to store proprietary or sensitive data.  It’s just mind-bogglingly stupid, and yet so many people fail to comprehend that “in the cloud” just means “on someone else’s computer.”

This is also true for most speech-to-text tools that (seemingly) kindly offer to translate your ramblings to text out of the goodness of the developer’s hearts.  Lots of people use this feature on their phones without realizing that, like Alexa, any voice command tool is an audio monitoring device you stupidly paid for and installed yourself on behalf of corporate spies who are all too happy to listen to whatever you have to say.  If you have an Alexa, get a hammer right now and smash it.  Go on, I’ll wait.  Good job.  Privacy restored.  Oh, smart TV too? Unplug that stupid thing from the internet.  Same for all your “smart” devices.  You thought “smart” meant you were smart for buying it?  Noooo… you’re a moron for buying it, the company was smart for convincing you to install monitoring devices in your house at your own expense.  Congrats. Own goal.  When you’re finished destroying all your corporate spyware here’s a way to get speech to text capability on your own hardware without the spying thanks to a very nice bit of FOSS code from OpenAI.

The workflow is to record some audio (speech probably) on your phone, store & forward that to your server (no synchronous connection required, unlike most spyware), (optionally) store and forward that to your desktop computer with a GPU to run AI text to speech, pop the results into an email queue to store & forward it back to you and all your searchable text archives. Speech is converted to accessible, indexed text easily and robustly and fairly legibly.

For the recording step, I use an Open Source app called Audio Recorder (available on F-Droid and other reliable repositories; if you need an app, try F-droid first and only use Play Store after deciding it is worth being spied on and having ads pushed to you).  Audio can be any length, seconds or hours.  I configured the settings to record to /storage/emulated/0/recordings and use 48khz, 16 bit, opus for speech; on my device the app supports up to 24bit/192khz, which vastly exceeds the S:N ratio and bandwidth of any microphone I’ll connect to a phone, but nice to know for audiophiles.

I also run NextCloud on my phone which connects to a NextCloud instance on my own server.  NextCloud is like a free, open source version of dropbox and provides directory sharing, calendar, password, etc – almost all services you want a server for on your own hardware so you actually retain possession and ownership of your data – amazing!  You do not have to give away your data to people you don’t know to use the internet.

The NextCloud client on my phone tries to sync the recording folder to my server so after I make a recording and hit the ✅ button, when the aether makes it possible the audio is uploaded (and, optionally, deleted from the mobile device). Nextcloud then syncs down to other clients, specifically one of my Linux clients for processing. It is entirely possible to do everything server side and the same scripts will work, but I don’t have a GPU on my server and Whisper has some dependencies that are easier to meet on a more frequently updated client, at least for now.

I’ve installed whisper on a Linux box, along with a NextCloud client and there I have a fairly simple script running as a cron job. Every 10 minutes it scans all the files in the locally synced “Recordings” directory and if there’s an audio file without a matching text “TSV” file, it calls whisper to convert the audio to text and then emails me the converted text.  That text is also synced back up to the server and to any other synced device and indexed both on the server and locally to make it easily discoverable (on clients I use the very awesome Recoll for indexing).

The whole process is very easy and any audio file like this:

is then automagically converted to text

test if we can record in Opus and then autoconvert the file back to text and
get that text as an email automatically this seems like quite a powerful tool
and should make it fairly easy to self take notes don’t we think yes

and then ends up in my inbox like this:

So what script does this good thing?  Just a few bash lines.  This version uses the time stamps in the TSV files to throw in fairly reasonable paragraph breaks. If the speaker pauses long enough that Whisper inserts a timing break, the script printfs in two newlines. There are a few other tricks below to try to infer or force reasonable paragraph breaks.

It also uses a slightly more robust construction to extract the subject of the email, which includes the first 60 characters of the text, minus any new lines (which make mailx barf).  The resulting text is flowed, pretty easy to copypasta into an email or document, and has moderately natural paragraph breaks.  It isn’t publication ready, but the accuracy seems quite good and it is hard to imagine an easier mechanism for making useful autodictations.  The process supports very long rambling diatribes, you should be able to talk for hours and get book’s worth of text in your inbox. I mean, maybe you shouldn’t be able to do that, but you can.

I put in a feature request with the Audio Recorder devs to add some metainfo to the files; what I’d really like is location data.  I can script up extracting that and (optionally) converting it to a place name, but aside from Nominatim or Gisography, there aren’t many options other than using big data APIs.  Anyway, seems like a reasonable bit of metadata to insert at the top or tail of the text: time+date+location the stream was recorded. If it is implemented, I’ll update to script to extract the metadata and create a dateline header.

Mailing flowed plain text

I found that mailx can’t handle long (flowed) text lines over ~1000 characters and inserts \n  at 998 or 997, which breaks up the pause to paragraphs code, so I switched the mailer to mpack (sudo apt install mpack) which simplifies the mail command and MIME encodes the text body and adds a checksum and a few other modern mail niceties and it now flows as desired without weird line breaks.

And then I found out that mpack thinks it is too good to send text files, it sets the MIME type to application/octet-stream and using the -c text/plain option yields the somewhat prissy error This program is not appropriate for encoding textual data oh my.  Thunderbird actually parses the attachment into a nicely flowed email, ignoring the quirks, but the best mobile client ever, FairEmail, does not and treats the attachment as something that it would prefer not to display inline (thanks for the details Marcel, you’re awesome!), given mailx isn’t very active any more changing that behavior is unlikely.  Next option: Mutt.  Mutt does something to a text attachment (using the -a option) that causes both TB and FairEmail to decline to display inline, but the body option -i yields a clean text-only email with the right flow, meaning no random line breaks inserted, so don’t install mpack, but sudo apt install mutt and create a /home/{user}/.muttrc file with at least the below (search engine around if you need to use a remote SMTP server to configure the server address, authentication, and encryption; mutt does the right things):

set realname = "{desired name}"
set from = "{your from email}"
set use_from = yes
set envelope_from = yes

And once that (and whisper) is working, the following script will convert your audio file to text and then mail it to you with paragraph breaks.

TextTiling

I didn’t plan to get into anything more complex, but long text conversions are kinda unreadable because Whisper doesn’t infer text.  There’s a whole science to inferring contextual shifts that should start new paragraphs using LSA/LDA/LSI that’s quite advanced mathematically and works sort of OK but is an awful lot of pipping modules and trying this or that.

I opted instead to go for a more brute force method, well three of them, really:

First: whisper has an experimental feature to compute word timings, which would normally be used to generate those unbelievably distracting and annoying and utterly horrible subtitles that are one word at a time or bouncing highlight word by word, but the feature can do more than create a miserable, distracting, utterly pretentious viewing experience: they seem to increase the frequency and possibly accuracy of gaps in the exported timing data. The first method of paragraph finding is detecting “long” gaps after a Whisper inferred sentence, effectively deriving speaker intent from cadence and AI content inference.  It works OK.

Second: I implemented a wake_word:command set that seds through the text and search-replaces the wake_word:command with the requested punctuation: .¶,:()…—?!“” There’s a whole theory behind wake words, but “insert” seems to be understood well and the command terms are ones that I tend to think of (e.g. “dots” not “ellipsis”), but that’s all obviously editable to preference.

Third: recommended paragraph length depends on the target and advice ranges from 3 sentence to 6.  I tend to be a bit long winded so I picked 5.  There’s an arbitrary script to look for any line that, after the timing inference and explicit breaks, still has more than 5 sentences and breaks it into multiple lines (meaning paragraph splits when the text is rendered). If that’s too long or too short, change the 5 in /usr/bin/sed -i "s/[.?!] /.\n\n/5;P;D" "$txt_file".

This all work fairly well, though there’s a known quirk with Whisper where it just randomly stops inserting punctuation after about 10 minutes and mechanisms 1 and 3 obviously also fail.  The way to deal with that is to break the audio into about 5 minute segments and then concatenate the results, but it’s a moderate chunk of code and debug and I’m assuming whisper will be updated.  If not and it gets annoying, I’ll work out that routine.

The script

Replace {user} and {domain} as appropriate to your system.  You may also have a different layout for commands, which bin (for example) is your friend.  I find full paths in cron execution provides better consistent reliability at the expense of portability.

#!/bin/bash 

watchdir="/home/username/Work/Recordings/"
to="email@domain.com"
stop_prev="0"
start=""
stop=""
text=""
wake_word="insert"

# Function to check if an audio file has a matching .txt file, then convert to text and email it
convert_to_text() {
    audio_file_file="$1"
    txt_file="${audio_file%.*}.txt"
    tsv_file="${audio_file%.*}.tsv"
    dir="$(/usr/bin/dirname "${audio_file}")"
    base_ext="$(/usr/bin/basename "${audio_file}")"
    base="${base_ext%.*}"


    if [ ! -e "$tsv_file" ]; then
        /home/gessel/.local/bin/whisper "$audio_file" -f tsv --model small.en -o $dir --word_timestamps True --prepend_punctuations True --append_punctuations True --initial_prompt "Hello."

        while IFS=$'\t' read -r start stop text; do
            # First line detection and skip checking it for gaps
            if [ $start == "start" ]; then
                /usr/bin/printf "" > "$txt_file"
                continue
            fi
            # Check if line ends in period or question mark for paragraph insertion
            if [[ $text =~ \.$|\?$ ]]; then
                # find natural pauses and insert paragraph breaks
                if [[ $stop_prev != $start ]]; then
                /usr/bin/printf "\n\n" >> "$txt_file"
                fi
            fi
            /usr/bin/printf "$text " >> "$txt_file"
            stop_prev=$stop
        done  < "$tsv_file"

        stop_prev="0"
        # search for explicit formatting commands and in-line replace them.
        /usr/bin/sed -i "s/[?,. ]*$wake_word period[?,. ]*/. /gI" "$txt_file"
        /usr/bin/sed -i "s/[?,. ]*$wake_word paragraph[?,. ]*/.\n\n/gI" "$txt_file"
        /usr/bin/sed -i "s/[?,. ]*$wake_word comma[?,. ]*/, /gI" "$txt_file"
        /usr/bin/sed -i "s/[?,. ]*$wake_word colon[?,. ]*/: /gI" "$txt_file"
        /usr/bin/sed -i "s/[?,. ]*$wake_word open paren[?,. ]*/ (/gI" "$txt_file"
        /usr/bin/sed -i "s/[?,. ]*$wake_word close paren[?,. ]*/) /gI" "$txt_file"
        /usr/bin/sed -i "s/[?,. ]*$wake_word dots[?,. ]*/… /gI" "$txt_file"
        /usr/bin/sed -i "s/[?,. ]*$wake_word long dash[?,. ]*/—/gI" "$txt_file"
        /usr/bin/sed -i "s/[?,. ]*$wake_word question[?,. ]*/? /gI" "$txt_file"
        /usr/bin/sed -i "s/[?,. ]*$wake_word exclamation[?,. ]*/? /gI" "$txt_file"
        /usr/bin/sed -i "s/[?,. ]*$wake_word open quote[?,. ]*/ “/gI" "$txt_file"
        /usr/bin/sed -i "s/[?,. ]*$wake_word close quote[?,. ]*/” /gI" "$txt_file"
        # brute force paragraphing: 5 sentences is enough, adjust for audience
        /usr/bin/sed -i "s/\([.?!]\) /\1\n\n/5;P;D" "$txt_file"
        # fix any sentence start/finish errors induced by the above edits
        /usr/bin/sed -i "s/^[a-z]/\U&/g" "$txt_file" # start with uppercase
        /usr/bin/sed -i "s/: [A-Z]/\L&/g" "$txt_file" # no uppercase after colon
        /usr/bin/sed -i 's/\s\+$//g' "$txt_file" # don't end with whitespace
        /usr/bin/sed -i "s/[,]$/./g" "$txt_file" # don't end with a comma, use .
        /usr/bin/sed -i '/[.?!]$/! s/$/./' "$txt_file" # if not ending with punctuation at all, add .
        /usr/bin/sed -i 's/^\.$//'  "$txt_file" # oops, no lines with just periods 
        /usr/bin/sed -i "s/\([a-z]\) \./\1./g" "$txt_file" # remove any spaces before periods
        /usr/bin/sed -i "s/  / /g" "$txt_file" # no double spaces
        /usr/bin/sed -i 's/\([0-9]\+\) \([FC]\) /\1°\2 /g' "$txt_file" # write temp to AMA, Chicago, Nat Geo, NOT APA or NIST
        # generate subject line from first sentence no longer than 80 char and remove any newlines
        subject=$(/usr/bin/head -n 1 -c 80 "$txt_file" | /usr/bin/sed 's/\(.*\)\..*/\1/')
        subject=$(/usr/bin/echo $subject | /usr/bin/tr -d '\n')
        subject=$(/usr/bin/echo $subject | /usr/bin/tr -d '\r')
        # send the cleaned up file as email
        /usr/bin/echo "" | /usr/bin/mutt  -F /home/gessel/.muttrc -s "AudioText - $base - $subject" -i "$txt_file" $to
    fi
}

# Main script scan the watch dir for unprocessed files (within the last 30 days)
/usr/bin/find "$watchdir" -mtime -30 -type f \( -iname \*.opus -o -iname \*.wav -o -iname \*.ogg -o -iname \*.mp3 \) | while read audio_file; do
    convert_to_text "$audio_file"
done

Note that Whisper has a lot of tricks not used here.  I’ve used it to add subtitles to lectures and it can do things like auto-translate one spoken language into another text language, and much more.