**** DONE Searching and Downloading YouTube Videos Via Shellscripts :blog:diy:software:cloud:services:pim:
CLOSED: [2024-02-24 Sat 15:27]
:PROPERTIES:
:ID: 2024-02-24-youtube-shell
:CREATED:  [2024-01-16 Tue 11:10]
:END:
:LOGBOOK:
- State "DONE"       from "DONE"       [2024-07-01 Mon 19:10]
- State "DONE"       from "DONE"       [2024-02-26 Mon 10:21]
- State "DONE"       from "STARTED"    [2024-02-24 Sat 15:27]
:END:

- Updates
  - 2024-02-26:
    - Somehow, many people seems to be fed up with the YouTube web
      interface in recent time and so multiple people are posting
      about the same issue. I introduced the "Similar Articles by
      Others" section. I'm using my method for years already.
    - Explanation on the semi-colon in the file names.
    - Further minor improvements happened after the initial
      publication as well: screenshot, better invocation examples, ...
  - 2024-07-01: I had to change my script because [[https://github.com/yt-dlp/yt-dlp/issues/10206][YouTube no longer provides download format 22 (720)]].

There are many reasons why someone would not want to use
https://YouTube.com in a web browser to search for and watch videos.

My most important reasons are:

- I never use an active Google account to log in to YouTube. This would violate my personal data privacy.
  - I happily accept the downside that I don't get personalized video
    recommendations. I would not use them anyway.
- I don't want to spend time clicking away all those useless cookie-windows - always reject all!
- I dislike the defaults for auto-play and such.
- I'm not interested in reading comments - most of the time. I never ever have written any YouTube comment myself.
- I prefer local video playback.
  - Omits any streaming issue because of bad bandwidth [[https://kevquirk.com/youtube-throttling][or similar]].
  - My local video player has so much more cool features accessible by easy-to-learn keyboard shortcuts.
- I don't enjoy watching advertisements.

Therefore, I created a way to search for YouTube videos, download
YouTube videos and watch them locally from my [[https://www.zsh.org/][zsh]] command line
interface.

If you don't want to use other solutions like [[https://invidious.io/][Individous]] or [[https://freetubeapp.io/][FreeTube]],
you might want to check out my workflow.

You don't have to use a shell if you want to use my method. You can
also wrap the shell scripts into easy to start temporary Terminal
windows to interact with them. I didn't bother so far. I'm very fine
with using the shell.

So here is my method which you can use and adapt to your personal
taste in case you're familiar with basic shell scripting and how to
invoke them.

***** My Basic Concept

I split up the method into several small shell scripts. This allows
for easy maintenance and modular re-use.

Furthermore, I differ between two different video sizes: low and high
resolutions. I use high resolutions when I want to watch the videos on
my desktop and I use low resolution videos for synchronizing them via
[[https://syncthing.net/][Syncthing]] to my mobile phone for watching when I'm away from my
desktop. Therefore, most scripts are available in two versions that
only differ with the output video resolution.

Since I'm using existing tools to do the heavy lifting, my shell
script method has some dependencies:

- [[https://github.com/yt-dlp/yt-dlp][yt-dlp]] is a fork of [[https://github.com/ytdl-org/youtube-dl][youtube-dl]]. This fork is better maintained and
  so it's recommended to migrate away from =youtube-dl= to =yt-dlp= as a
  proper replacement.

- [[https://github.com/pystardust/ytfzf][ytfzf]] offers a nice interface to find YouTube videos. As of
  2024-02-24 it's no longer maintained but it's still working on my
  side so far.

- [[https://github.com/novoid/guess-filename.py][guess-filename]] is one of my scripts from [[id:2014-05-09-managing-digital-photographs][my file management method]]
  which renames downloaded files like I want them. Instead of
  : ayo why you runnin [gFh4dAX5g-U].mp4=
  I get the file name
  : 2023-06-28 youtube - ayo why you runnin - gFh4dAX5g-U 1;07.mp4
  which contains the date of upload, the original title, the YouTube
  hash for re-finding it later and the duration of the video.

***** Searching for YouTube Videos

I search for videos via the wrapper scripts =yth= and =ytl= which
stand for "YouTube high resolution" and "YouTube low resolution" in my
head.

You invoke the scripts with a search string like in the following
example invocations:

: yth "trailer big lebowski"
: ytl "how to install newpipe on android"
: yth "review kaweco lilliput"

This will then show the videos matching this query:

#+CAPTION: Example output of ytfzf.
#+ATTR_HTML: :align center :width 630 :linked-image-width 1000
[[tsfile:2024-02-24T15.59.19 yth example query -- publicvoit screenshots.png][2024-02-24T15.59.19 yth example query -- publicvoit screenshots.png]]

In this interface, you can filter the results by typing your filter
strings. With the cursor keys, you can move among the result hits.
With the return key you invoke the download and close the window.

My =yth= and =ytl= scripts below are currently using the "view-count"
for sorting the results. You might as well change this to different
settings.

If you play around with =ytfzf= you could even get preview images. I
tried it for a couple of minutes, failed and did not care to re-try so
far. =ytfzf= does offer more stuff - just read its documentation.

This is my =yth=:

#+BEGIN_SRC sh
#!/usr/bin/env sh
URL=$(ytfzf -l -L --upload-sort=view-count "${1}")
[ "x${URL}" != "x" ] && yd "${URL}" 720
#end
#+END_SRC

Here is my =ytl=:

#+BEGIN_SRC sh
#!/usr/bin/env sh
URL=$(ytfzf -l -L --upload-sort=view-count "${1}")
[ "x${URL}" != "x" ] && yd "${URL}" 640
#end
#+END_SRC

The numbers 720 and 640 stand for the video width which then gets
translated into YouTube-specific download quality indicators which you
can see when you execute:

: yt-dlp -F 'https://www.youtube.com/watch?v=gFh4dAX5g-U'

At the moment, 720 is 720×720 mp4 using avc1.64001F codec and 640 is
6400×360 mp4 using the avc1.42001E codec.

If you want to use the same method for other video platforms supported
by =yt-dlp=, you only need to add functionality that chooses the
different quality indicators because they differ among different video
providers.

Please note that on YouTube not all videos are available in all
qualities. Especially older content might only be available in lower
resolutions.

Both wrapper scripts are using =yd= which is my general download script:

#+BEGIN_SRC sh
#!/usr/bin/env bash

URL="${1}"
FORMAT="${2}"

if hash yt-dlp 2>/dev/null; then
    YDBIN="yt-dlp"
    Y_QUERY_OPTIONS="--no-check-certificate --compat-options list-formats"
    Y_DL_OPTIONS="--no-mtime --write-info-json --no-check-certificate"

elif hash youtube-dl 2>/dev/null; then
    echo "WARNING: \"yt-dlp\" not found, using \"youtube-dl\" instead"
    YDBIN="youtube-dl"

else
    echo "ERROR: no youtube downloader tool found."
    exit 1
fi

if [ "${YDBIN}" = "youtube-dl" ]; then
    Y_QUERY_OPTIONS="--no-check-certificate -F"
    Y_DL_OPTIONS="--write-info-json --no-check-certificate"
fi

## FIXXME: doesn't work when URL is not lowercase or when URL contains "youtube" not in the domain part:
if [ -z ${FORMAT} ] || [[ "${URL}" != *"youtube"* ]]; then
    "${YDBIN}" ${Y_QUERY_OPTIONS} "${URL}" | grep -v only
    echo
    read -p 'Please enter the desired version to download: ' FORMAT
    echo
fi

## YouTube: translate download wish to actual formats:
if [ "x${FORMAT}" = "x720" ]; then
    FORMAT="-S vcodec:h264,fps,res:720,acodec:m4a"
elif [ "x${FORMAT}" = "x640" ]; then
    FORMAT="-f 18"
else
    FORMAT="-f ${FORMAT}"
fi

"${YDBIN}" ${Y_DL_OPTIONS} ${FORMAT} "${URL}"

# Optional, if you want to get file names like:
# "2023-07-04 youtube - Nix flakes explained - S3VBi6kHw5c 7;21.mp4"
guess-filename-for-info-json-mp4-files.sh

#end
#+END_SRC

Of course, if you don't want to use =yth= and =ytl= because you only
want high resolution videos, you could modify =yd= by replacing
=${FORMAT}= with the appropriate format string and invoke it instead
of =yth= and =ytl=.

The =yd= script could be further simplified by removing the
=youtube-dl= support. However, some people might find it useful when
it works for =yt-dlp= as well as with =youtube-dl=. I personally don't
use the latter any more.

Here is my wrapper script for invoking =guess-filename= after the
download: =guess-filename-for-info-json-mp4-files.sh=

#+BEGIN_SRC sh
#!/usr/bin/env sh

#- process
#   1. find latest (all?) file(s) in directory (with extension .info.json)
#   2. generate video file name by removing .info.json and replace with .mp4
#   3. invoke guess-filename on video file
#   4. remove json file

for jsonfile in *.info.json; do

    mp4name=$( echo ${jsonfile} | sed 's/.info.json/.mp4/')
    m4aname=$( echo ${jsonfile} | sed 's/.info.json/.m4a/')

    if [ -f "${mp4name}" ]; then
        guess-filename "${mp4name}"
        rm "${jsonfile}"
    elif [ -f "${m4aname}" ]; then
        guess-filename "${m4aname}"
    rm "${jsonfile}"
    else
        echo "I could not locate \"${mp4name}\" for the given \"${jsonfile}\" ... ignoring it."
    fi

done
#end
#+END_SRC

If you are fine with the default file names, you could remove
=guess-filename-for-info-json-mp4-files.sh= from =yd=. This would also
get rid of the =guess-filename= dependency.

***** Downloading YouTube Videos

If you do have a YouTube video URL from anywhere on the web, you can
use my wrapper scripts =ydh= and =ydl= just for downloading the
videos without the necessity for searching like above.

The script names =ydh= and =ydl= stand for "YouTube download high
quality" and "YouTube download low quality" in my head.

With the =yd= script from the previous section, those wrapper scripts are
pretty simple. Here is =ydh=:

#+BEGIN_SRC sh
#!/usr/bin/env sh
yd "${1}" 720
#end
#+END_SRC

And here is =ydl=:

#+BEGIN_SRC sh
#!/usr/bin/env sh
yd "${1}" 640
#end
#+END_SRC

You do invoke them like:

: ydh 'https://www.youtube.com/watch?v=gFh4dAX5g-U'

Alternatively, you can also invoke it with the YouTube hash only in
you want:

: ydh gFh4dAX5g-U

Both methods lead to a local video file like:

: 2023-06-28 youtube - ayo why you runnin - gFh4dAX5g-U 1;07.mp4

Don't you wonder about the ";" to separate minutes from seconds in the
last part of the file name. I had to switch from ":" because somehow,
Google decided that the Android file system of my Pixel 4a should
inherit the file name restrictions of MS-DOS/FAT by Microsoft. [[https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system#Directory_table][This
conflicts with many characters]] in cluding the colon. The semi-colon
worked somehow.

***** My Personal Usage Patterns

In practice, I rarely use =yth= or =ytl= for searching.

Most of the time, I come across a YouTube URL on Mastodon or on the
web. I copy the URL into my clipboard, [[id:2022-10-28-window-teleporting][switch to my =tmux= shell]],
switch to my default video download directory and invoke =ydh= or
=ydl= for downloading and renaming the video.

Using the =zsh= command =mpv *(.om[1])= from my command history, I open
the most recently downloaded video in [[id:apps-I-am-using][my favorite movie player]].

I can think of an automation method that watches for changes in the
clipboard and invokes =ydh= to my default download directory in case
the clipboard matches a YouTube URL. So far, I prefer the flexibility
to decide on the quality I want to download.

By the way, for Android YouTube consumption, I do recommend [[https://newpipe.net/][NewPipe]] as
a full replacement for the YouTube app by Google. It's a really good
mobile app that deserves all support you can give.


***** Similar Articles by Others
:PROPERTIES:
:END:

- 2024-02-25: [[https://kevquirk.com/ditching-youtube-kind-of][Ditching YouTube (Kind Of) | Kev Quirk]] ([[https://invidious.io/][Invidious]])
- 2024-02-25: [[https://hunden.linuxkompis.se/2024/02/25/how-i-make-youtube-on-the-web-less-annoying-and-distracting.html][How I make YouTube (on the web) less annoying and distracting | Hund]] ([[https://addons.mozilla.org/en-US/firefox/addon/youtube-recommended-videos/][Unhook]], [[https://addons.mozilla.org/en-US/firefox/addon/enhancer-for-youtube/][Enhancer for YouTube]])