How to Git Gud with Shell: tips, ideas, and resources
Without question, getting good with the shell is the most life-changing thing you can do if you’re a linux user. Read this post for tips on getting good, ideas for projects to accomplish with shell scripts, and resources for learning more.
by spitemim, 2022-08-14
Without question, shell (and shell script) is the most useful language to learn if you’re running Linux. Punching in a few lines at the shell prompt can save you hours of tedious, repetitive, manual work if you know what you’re doing. Even if you spend more time automating something than it would take to actually do the thing, writing a script to do something repetitive is more interesting than actually doing it. The more you practice with it, the more knowledge you gain that can be applied to other projects.
Getting good with the shell, just like getting good with any other computer language, can take a while especially if you’re not coding-oriented. In this post, I want to share some tips that I’ve picked up throughout my travels on the internet, as well as video and text-based resources that helped me progress, and project ideas for writing shell scripts.
(If you just want to skip my rants and get to the good stuff, click here.)
Shell vs. Bash/zsh
It’s important to make this distinction right off the bat. When I say shell, I’m talking about sh. Not bash, not zsh, ESPECIALLY not fish, just sh.
sh is a specification for how a shell should act, created by POSIX. It’s
essentially a set of rules and requirements that so-called POSIX
compliant shells should follow. Since it’s a specification and not an
/bin/sh is often a symbolic link to an implementation
of sh such as dash.
NOTE: On Arch and many other systems, sh is just a symlink to bash,
which will try its best to be POSIX-compliant when invoked as sh. Bash
is significantly slower than other sh implementations, like dash for
example, so you might wanna install another shell like dash and symlink
/bin/sh. Like this (run as root):
ln -sf "$(which dash)" /bin/sh
Bash and zsh support most of the features of the POSIX shell, but they also have their own features, known as bash-isms or zsh-isms. All these extra features cause bash and zsh to be slower and more bloated than plain sh implementations. Sometimes, POSIX sh scripts will behave differently when running in these shells due to the extended featuresets. I still use zsh as my main shell – the one I use when I’m just sitting at my terminal not scripting, because a lot of those extra features provide niceties like smart tab-completion. I don’t use zsh-isms when I’m just sitting at my terminal because I’m just not interested in learning features that I’m not going to use in scripts.
sh is a much simpler language, more portable across *NIX systems, and common implementations of it are faster too. If you’re writing software complex enough that you need the extra features of bash or zsh, you might be better off writing it in an actual language – see Python, Perl, Lua, C, Awk for that. This SO answer sums up my thoughts on sh pretty well, if you want a tl;dr.
That being said, bash and zsh are still powerful shell languges that can do a lot of stuff faster and easier than languages like Python. If you wanna write bash scripts, I’m not gonna fault you.
Shell vs. Shell Scripting
When I say you should get good with the shell, I mean getting good with the interactive shell AND with shell scripts. It might make sense to do most of your initial learning just by punching stuff into the terminal, since that just requires the core shell knowledge – how to use commands, how to use slightly more advanced syntax like subshells and stuff, and so on. Shell scripting is the art of taking that knowledge and wrapping it with extra logic to be used as a standalone program. A good UNIX tool should handle commandline args (if applicable to the problem at hand,) report errors, and generally not break down due to unexpected input. Taking one or more shell one-liners that accomplish a task, putting them together, and making the whole thing robust enough for use in other programs is the art of shell scripting. a For example, you could learn how to combine a still image and audio file into a video file with ffmpeg. That’s just shell knowledge. Shell scripting is figuring out how to put that into a script with extra logic for convenience, maybe it lets you pass image and audio filenames as arguments, does sanity checks like making sure the files exist, reports errors/usage statements, and whatever else.
Stuff to learn
- coreutils: All of them. You should already know file management utils like mv, cp, rm, mkdir, rmdir, and all that. Make sure to learn find, grep, sed, cut, tr, wc too.
- misc tools: Learning dmenu will give you so many ideas for shell scripts it’s not even funny. xclip is super useful and you can write a lot of convenient scripts with it. Learn ffmpeg and imagemagick commands like convert. Learn xdotool.
- shell features: If statements, loops, case statements, arguments (\$@, \$*, \$#, \$1, \$2, \$3 etc), weird variables like \$?, \$\$, \$-. Learn getopt.
- web scraping: jq and pup are super useful lesser known tools for parsing JSON and HTML, respectively. Learn how to change your user-agent with curl and you’re ready to do web scraping.
- 0x0.st is a super useful file sharing site that you can automate uploads to because uploading is just POSTing your data with curl. (Instructions can be found on the site)
- LEARN REGULAR EXPRESSIONS. Get good with regular expressions. Your respect from peers will increase tenfold. You will be truly superior to everyone else in the world. Attention from women will skyrocket. Attention from men will probably skyrocket too. Learn regular expressions.
People to watch
Bugswriter and Luke Smith are pretty much the only youtubers worth watching when it comes to shell script that I know of.
Bugswriter is probably hands-down the greatest youtuber you could possibly watch if you wanna learn the shell. His older thumbnails are cringe, but don’t let that fool you. His videos were super instructive when I was first learning shell scripting over a year ago, and I made a lot of really cool scripts that I still use today.
This is the number one thing you should take from this post: WATCH BUGSWRITER’S VIDEOS. If you listen to nothing else on this page, at least listen to this.
Luke smith doesn’t post much shell scripting related content anymore. His older videos on shell scripting and how to use basic commands are really useful for beginners though. He also has videos on regular expressions and stuff like that.
Stuff to Read
Everything from the Linux Documentation Project is pretty good. The Advanced Bash Scripting Guide is, as the title suggests, a bash-specific guide, but it looks pretty good for learning basic shell features. I don’t think it relies too heavily on bashisms.
According to /u/geirha on reddit, the Advanced Bash Scripting Guide is outdated and inaccurate. They recommend the BashGuide from the wooledge wiki, which I have not checked out but I assume is decent.
Read the dash manpage –
You will learn some useful and interesting stuff about shell syntax that
no one on Youtube will tell you because no one else is enough of a
depraved nerd to read the dash manpage.
Ideas for scripts
- Write a script that downloads all the comics from XKCD
- Write a script that gives you a dmenu prompt for clipboard actions (upload copied image to 0x0, upload copied text, download copied file URL, youtube-dl the copied URL, whatever. Be creative)
- Write a script that scrapes some informational site and gives back info in a terminal-friendly way (think lyrics sites, wallpaper sites, guitar tab sites, anything that has a search feature and content that can be downloaded or viewed on the terminal.)
- Write a script that compresses image files (you can find some command using
convertor ffmpeg to compress the images. The real task is handling arguments, reporting errors, all that good stuff. Maybe add a feature that can compress the image currently copied to the clipboard)
- Write an autoclicker with xdotool. Ideally, figure out a way to let the user toggle the clicker with a keyboard combo.