Bash practical

Wildcards and Symbols

Symbols	Meaning
*	Represents '0 or more' arbitrary characters
?	Represents 'exactly one' arbitrary character
[ ]	Represents 'exactly one character from within the brackets' (not arbitrary). For example, [abcd] means 'exactly one character that could be any of a, b, c, d'
[ - ]	When a hyphen is within brackets, it represents 'all characters within the specified range'. For example, [0-9] represents all digits between 0 and 9 because their encoding is consecutive!
[^ ]	If the first character inside the brackets is a caret (^), it means 'negation'. For example, [^abc] means 'exactly one character that is not a, b, or c'

Examples

# Example 1: Find files in /etc/ that start with "cron"
$ ls -d /etc/cron*  # The -d option is to display only directories

# Example 2: Find files in /etc/ that have exactly five letters
$ ls -d /etc/?????  # Using ? means exactly one character, so five ?s represent a five-letter filename

# Example 3: Find files in /etc/ that contain a digit
$ ls -d /etc/*[0-9]*  # Remember to place * on both sides of the brackets

# Example 4: Find files in /etc/ whose names do not start with a lowercase letter
$ ls -d /etc/[^a-z]*  # Note there is no * before the brackets

# Example 5: Copy the files found in Example 4 to /tmp/upper
$ mkdir /tmp/upper; cp -a /etc/[^a-z]* /tmp/upper

Command Redirection

In Linux/Unix, everything is treated as a file, including regular files, directories, and even devices. Each file has an associated number called a File Descriptor (FD).

Whenever you execute a program/command at the terminal, three special files are always open:

File	File Descriptor
Standard Input (stdin)	0
Standard Output (stdout)	1
Standard Error (stderr)	2

These file descriptors are essential for handling input and output operations. For example, when you run a command, its output is sent to the file descriptor for the screen (stdout), so you see the output on your monitor. If redirected to a printer, the output would be printed instead.

Redirection Operators

Redirection allows you to change the standard input/output devices to files or other commands. Here are the common redirection operators:

• >: Redirects standard output to a file, overwriting the file if it already exists.
• >>: Redirects standard output to a file, appending to the file if it already exists.
• <: Redirects standard input from a file.
• 2>: Redirects standard error to a file.
• 2>&1: Redirects standard error to the same location as standard output.
• &> or &>>: Redirects both standard output and standard error to a file.
• /dev/null: A special file that discards all data written to it (effectively a "black hole").

Examples of Redirection

# Redirect standard output to a file, overwriting if it exists
$ ls -al > list.txt

# Append standard output to a file
$ ls -al >> list.txt

# Redirect standard output and standard error to different files
$ ls -al 1> list.txt 2> list.err

# Redirect both standard output and standard error to the same file
$ ls -al 1> list.txt 2>&1

# Redirect standard output to a file and discard standard error
$ ls -al 1> list.txt 2> /dev/null

What happen if I don't discard standard error if an error occur?

In the above example ls -al 1> list.txt 2> /dev/null, consider a scenario where you run ls -al in a directory where some files or directories are not accessible due to permission issues.

If you do not discard the errors:

• You will see error messages ls: cannot access 'somefile': No such file or directory directly in your terminal. This can be helpful for debugging or for being aware of issues as they occur.
• The list.txt file will still contain the output of the ls -al command, but the terminal will also show any errors that occurred during the execution.

When to Use Redirection

• Saving important output: When the output displayed on the screen is important and needs to be saved.
• Background processes: When a program is running in the background and shouldn't interfere with normal screen output.
• System routine commands: When the execution results of some routine system commands need to be saved.
• Discarding known errors: When you want to discard known error messages using 2> /dev/null.
• Separating output types: When error messages and correct messages need to be output separately.

Summary

• Each file in Linux has a corresponding File Descriptor associated with it.
• The keyboard is the standard input device while your screen is the standard output device.
• > is used to redirect output to a file, overwriting it.
• >> is used to append output to an existing file.
• < is used to redirect input from a file.
• 2> redirects standard error to a file.
• 2>&1 redirects standard error to the same location as standard output.
• /dev/null is used to discard unwanted output.

By mastering command redirection, you can efficiently manage where your program outputs its data, making your work on Linux/Unix systems more flexible and powerful.

Process substitution

Many commands can accept input from a facility called standard input. By default, standard input gets its contents from the keyboard, but like standard output, it can be redirected. To redirect standard input from a file instead of the keyboard, the "<" character is used like this. 以最簡單的說法來說， 那就是『將原本需要由鍵盤輸入的資料，改由檔案內容來取代』的意思。:

$ sort < file_list.txt

In the example above, we used the sort command to process the contents of file_list.txt. The results are output on the display since the standard output was not redirected. We could redirect standard output to another file like this:

$ sort < file_list.txt > sorted_file_list.txt

As you can see, a command can have both its input and output redirected. Be aware that the order of the redirection does not matter. The only requirement is that the redirection operators (the "<" and ">") must appear after the other options and arguments in the command.

$? (指令回傳值) 與 && 或 ||

如同上面談到的，兩個指令之間有相依性，而這個相依性主要判斷的地方就在於前一個指令執行的結果是否正確。 還記得本章之前我們曾介紹過指令回傳值吧！嘿嘿！沒錯，您真聰明！就是透過這個回傳值啦！ 再複習一次『若前一個指令執行的結果為正確，在 Linux 底下會回傳一個 $? = 0 的值』。 那麼我們怎麼透過這個回傳值來判斷後續的指令是否要執行呢？這就得要藉由『 && 』及『 || 』的幫忙了！ 注意喔，兩個 & 之間是沒有空格的！那個 | 則是 [Shift]+[] 的按鍵結果。

指令下達情況	Meaning
cmd1 && cmd2	若 cmd1 執行完畢且正確執行($?=0)，則開始執行 cmd2。
	若 cmd1 執行完畢且為錯誤 ($?≠0)，則 cmd2 不執行。
cmd1 || cmd2	若 cmd1 執行完畢且正確執行($?=0)，則 cmd2 不執行。
	若 cmd1 執行完畢且為錯誤 ($?≠0)，則開始執行 cmd2。

範例：我不清楚 /tmp/abc 是否存在，但就是要建立 /tmp/abc/hehe 檔案
$ ls /tmp/abc || mkdir /tmp/abc && touch /tmp/abc/hehe

上面這張圖顯示的兩股資料中，上方的線段為不存在 /tmp/abc 時所進行的指令行為，下方的線段則是存在 /tmp/abc 所在的指令行為。如上所述，下方線段由於存在 /tmp/abc 所以導致 $?=0 ，讓中間的 mkdir 就不執行了！ 並將$?=0 繼續往後傳給後續的 touch 去利用啦！

$ echo $SHELL
/bin/bash              <==可順利顯示！沒有錯誤！

$ echo $?
0                      <==因為沒問題，所以回傳值為 0

$ 12name=VBird
bash: 12name=VBird: command not found...   <==發生錯誤了！bash回報有問題

$ echo $?
127                    <==因為有問題，回傳錯誤代碼(非為0)
# 錯誤代碼回傳值依據軟體而有不同，我們可以利用這個代碼來搜尋錯誤的原因喔！

$ echo $?
0
# 咦！怎麼又變成正確了？這是因為 "?" 只與『上一個執行指令』有關，
# 所以，我們上一個指令是執行『 echo $? 』，當然沒有錯誤，所以是 0 沒錯！

$? expands to the exit status of the most recently executed foreground pipeline.

Reference: Special Parameters section of the Bash manual

Pipe & Command substitution

Pipe

就如同前面所說的， bash 命令執行的時候有輸出的資料會出現！那麼如果這群資料必需要經過幾道手續之後才能得到我們所想要的格式，應該如何來設定？這就牽涉到管線命令的問題了（ pipe ），管線命令使用的是『 | 』這個界定符號！另外，管線命令與『連續下達命令』是不一樣的呦！這點底下我們會再說明。底下我們先舉一個例子來說明一下簡單的管線命令。 假設我們要讀取 last 這個指令中，那個 root 登入的『次數』應該怎麼作？注意呦！我們只需要『次數』。那麼我所進行的步驟是：

執行 last ，將所有這個月的所有人登入資料取出來；
使用 grep 將上面的輸出資料（stdout）當中的 root 擷取出來，其他的不要；
使用 wc 這個可以計算行數的指令將上一步的資料計算行數！
[test @test bin]# last
[test @test bin]# last | grep root
[test @test bin]# last | grep root | wc -l

這個管線命令『 | 』僅能處理經由前面一個指令傳來的正確資訊，也就是 standard output ( STDOUT ) 的資訊，對於 stdandard error 並沒有直接處理的能力!

Command substitution

"Command substitution" is the name of the feature of the shell language that allows you to execute a command and have the output of that command replace (substitute) the text of the command. These commands are executed in a subshell, and their data is what the substitution syntax expands to.

The command that the command substitution executes, is executed in a subshell, which means it has its own environment that will not affect the parent shell's environment.

在一串指令的執行中，還需要藉由其他額外的指令所提供的資訊時，可以使用反單引號『指令』或 『$(指令)』。

$ s=123
$ echo "hello $( s=world; echo "$s" )"
hello world
$ echo "$s"
123

xargs

xargs vs exec

Using xargs is far more efficient. In fact several benchmarks suggest using xargs over exec is six times more efficient.

The xargs command in UNIX is a command line utility for building an execution pipeline from standard input. Whilst tools like grep can accept standard input as a parameter, many other tools cannot. Using xargs allows tools like echo and rm and mkdir to accept standard input as arguments.

How to use xargs

By default xargs reads items from standard input as separated by blanks and executes a command once for each argument. In the following example standard input is piped to xargs and the mkdir command is run for each argument, creating three folders.

echo 'one two three' | xargs mkdir
ls
one two three

When filenames contains spaces you need to use -d option to change delimiter

ls
'one two three.txt' 'four.txt'
find . -name '*.txt' | xargs -d '\n' rm

How to use xargs with find

The most common usage of xargs is to use it with the find command. This uses find to search for files or directories and then uses xargs to operate on the results. Typical examples of this are changing the ownership of files or moving files.

find and xargs can be used together to operate on files that match certain attributes. In the following example files older than two weeks in the temp folder are found and then piped to the xargs command which runs the rm command on each file and removes them.

find /tmp -mtime +14 | xargs rm

Command redirection vs. Piping

TL;DR

Pipe is used to pass output to another program or utility. Redirect is used to pass output to either a file or stream.

• ls > log.txt sends the output to the log.txt file.
• ls | grep file.txt sends the output of the ls to grep command through the use of pipe (|), and the grep command searches for file.txt in the in the input provided to it by the previous command.

Piping redirects the output of one program to the input of another program.

Command substitution changes the command you enter by replacing the part with the command substitution with the output of the program.

Piping only works if the other program reads input from stdin (standard input channel).

Command substitution only works if the other program allows being called with an argument that can be fulfilled by the command substitution output.

$ echo hello | cat
hello

works because cat reads from stdin by default so cat just displays what it reads from there...the output of echo

$ cat $(echo hello)
cat: hello: No such file or directory

Doesn't work because the result of command subsitution is cat hello...what cat interprets as a filename to use.

Or the other way around...lets asume we have a test.txt file...

$ echo test.txt | cat
test.txt

Well..kind of works but probably not what you wanted...it outputs what you gave with echo..

$ cat $(echo test.txt)
<outputs the content of the test.txt file>

I hope that makes it more clear now.

End of options marker / option delimiter

An options marker (or end of options marker) is a command-line convention in Unix/Linux systems, represented by a double dash (--). Its primary purpose is to differentiate between options (also called flags) and positional arguments in commands.

Many command-line tools allow the use of options, which typically start with a dash (-) or double dash (--), to modify the behavior of the command. However, sometimes you may need to pass arguments to the command that begin with a dash but are not meant to be interpreted as options. In such cases, the options marker -- helps clarify to the command-line tool that all subsequent inputs should be treated as positional arguments (i.e., regular inputs), not options. Consider the following scenario:

You want to delete a file named -file.txt using the rm (remove) command. Normally, you might type:

rm -file.txt

But here, rm will interpret -file.txt as an option due to the leading dash (-), which would likely result in an error because there is no such option as -file.txt.

To avoid this, you can use the options marker (--):

rm -- -file.txt

Here, -- tells rm that everything following it is a positional argument (in this case, the file -file.txt) and not an option. This ensures that the file gets removed without rm misinterpreting it as an option.

Difference Between dotenv -- and dotenv && in package.json Scripts

"scripts": {
  ...
  "dev": "PORT=5174 dotenv -- node ./server.js",
  ...
},

When working with environment variables in Node.js, you might encounter two common ways to use the dotenv package in package.json scripts: dotenv -- node and dotenv && node. Though they look similar, they work differently, and understanding the distinction is important for managing environment variables effectively.

• 1. dotenv -- node ./server.js
  • This syntax runs the dotenv command-line tool to load environment variables from the .env file before executing node ./server.js.
  • The double dash (--) separates the dotenv command from the node command. Everything after the -- is passed to node, not dotenv.
  • dotenv and node run as part of the same process, which means the environment variables loaded by dotenv are immediately available to node when the server starts.
  • This is the preferred approach because it ensures that the environment variables are accessible to node from the start, without requiring any additional steps.
• 2. dotenv && node ./server.js
  • The && operator is a command chaining mechanism in Unix/Linux. It runs the first command (dotenv), and if it succeeds, it runs the second command (node ./server.js).
  • However, the dotenv command in this case runs as a separate process from node. It simply executes and exits, without persisting the environment variables in a way that the node process can access.
  • Since the environment variables loaded by dotenv do not carry over to the next command in the chain (node ./server.js), this method won’t work as expected. The environment variables won’t be available to node.

References

• Input Output Redirection in Linux/Unix Examples
• 第十章、認識與學習BASH
• $：(關於本 shell 的 PID)
• I/O Redirection
• Piping vs. command substitution--what's the difference?