shell编程基础

1. 命令运行

命令 描述
source hello_word.sh shell的内嵌命令,运行时没有启动子shell嵌套环境
. hello_word.sh .是一个命令,和./hello_word.sh中的.不同
sh hello_word.sh 启动了一个子shell进程,然后在子进程中执行脚本
./hello_word.sh 启动了子shell并执行脚本

只执行bash命令则是启动并进入了一个子shell,之后的命令会在这个子shell中进行。

2. 变量

所有变量搜默认为字符串类型。

系统预定义变量
系统预定义变量的变量名通常为大写,常用系统变量有$PWD,$HOME,$SHELL,$USER$PATH(保存命令的路径)。
执行命令env可以查看所有系统全局变量。执行命令set可以查看所有变量,包括系统定义和用户定义。

自定义变量

功能 命令
定义变量 a=2a="hello world",注意等号前后不能有空格
声明为全局变量 export a,此时可以在子shell中使用。但子shell对该变量的修改只在子shell中生效
只读变量 readonly b=5
删除变量 unset a,不能删除只读变量
命名 不能以数字开头,环境变量名建议大写

特殊变量
用于为脚本输入参数。

命令 描述
$n 使用时把“n”改为数字. $0为该脚本名称;$1-$9he ${10}表示对应位置的参数
$# 获取输入参数的个数
$* 命令行中的所有参数,把所有参数视作一个整体
$@ 命令行中的所有参数,所有参数构成一个集合
$? 最后一次执行命令返回的状态,0表示正确执行

3. 运算符

功能 命令
数值计算命令 expr 1 + 2, expr 5 \* 2,注意运算符和数字之间的空格,不能省略。星号需要转义符,加号不需要
数值计算 a=$((1+5))或者a=$[2+3],注意数字和运算符之间没有空格
数值赋值 a=$(expr 1 + 2)或者”a=`expr 1 + 2`“(中间的小点是反引号,反引号内的字符串当作shell命令来执行,返回值是命令的执行的结果,起到的是一个命令的替换作用)

4. 流程控制

条件判断
基本语法:test $a = hello 或者[ $a = hello ](满足条件时返回值为0;注意命令中的空格);

常用判断条件 描述
整数  
-eq equal缩写
-ne  
-lt less than缩写
-le less or equal缩写
-gt grater than
-ge great缩写
文件权限  
-r  
-w  
-x  
文件类型  
-e existence缩写,文件是否存在
-f 是否为常规文件
-d 是否为目录
-z  
``  

示例:

[ $a -lt 20] && echo "1" || echo "2" 为真输出1,否则输出2
[ $a -lt 10 -a $a -gt 5] -a为与,-o为或
[ -z $a ] zero缩写,字符串是否为空,或者数字值为0

if分支

Syntax Meaning / Purpose Example Notes
command (direct) Runs a command, true if exit code = 0 if grep -q foo file; then ... fi Most “Unix way.” No brackets.
[ ... ] POSIX test command (string, number, file checks) if [ -z "$x" ]; then ... fi Old, portable, must have spaces around operators.
[[ ... ]] Bash/Ksh/Zsh extended test command if [[ $x == foo* ]]; then ... fi Safer, supports regex, globbing, avoids word-splitting issues. Not POSIX.
(( ... )) Arithmetic evaluation (Bash/Ksh/Zsh) if (( x > 10 )); then ... fi Pure numeric contexts; no need for -gt, -lt.

单分支

if condition;then
  pass 
fi

or

if condition
then
  pass 
fi

多分支

if condition1
then
  pass1 
elif condition2
  pass2
else
  pass3
fi

case分支

case $var in
"var1")
  pass1
;;
"var")
  pass2
;;
*)
  pass
;;
esac

for循环

for((初始值;判断;变量变化)) # 因为(()),因此内部可以直接使用数学上的运算式
do
  pass
done

或者

# for var in {1..100} # 遍历从1到100这个序列
# for var in `ls` # 
for var in var1 var2 var3
do
  pass
done

while循环

while condition
do
  pass
done

读取控制台输入

基本语法:read [option] argument(选项 赋值的变量)

Option Description
-p prompt Display a prompt before reading input (bash only)
-s Silent mode (input not displayed on screen, useful for passwords)
-t sec Timeout in seconds
-n num Read only num characters
-r Do not allow backslash escaping (treats \ literally)
-a array Assign words to array elements
# reading from a File
while read line; do
    echo "$line"
done < filename.txt

函数

系统函数

basename

命令 描述
basename string 字符串剪切,只保留字符串最后一个/后的字符,通常用于从完整文件路径中获取文件名
basename string suffix 把字符串中的后缀删除

dirname

命令 描述
dirname string 字符串剪切,只保留字符串最后一个/前的字符,通常用于从完整文件路径中获取文件路径
user function

声明和定义函数后,才能调用函数。函数返回值只能通过命令$?获取,返回值为函数中return语句的返回值,返回值只能为0-255。如果没有return,则返回函数中最后一个命令的执行结果。实际上return的是一个状态码,因此取值范围很小,如果数值太大,不会报错,会返回溢出后的值。

function add()
{
  s=$1 # $1为传入的参数
  echo $s 
  # return 0 # 如果是返回状态码,建议使用return
}
sum=$(add $a)
echo $sum

正则表达式

正则表达式使用一些特殊字符来匹配对应字符串。部分工具引入了扩展的正则规则,这些规则并不通用,其它工具可能无法正确识别这些正则规则。

特殊字符 描述  
^sh 以“sh”开头  
end$ 以“end”结尾  
^$ 匹配空行  
r..t 匹配以r开头t结尾中间有两个字符的行  
ro*t *前的字符匹配至少一次,例如“rot”“root”“rooot”  
^a.*b 以a开始以b结束  
[a-f,g-p]or[3-5] 该字符只能处于这个范围  
[ab]*or[a,b]* 这个范围内的字符出现任意次  
'\$' 对$转义 #
# extract a string surround by "=> " and" (" from a long string

# Perl regex, available in GNU grep
# (?<==> ): positive lookbehind for =>
# .*?: minimal match
# (?= \(): positive lookahead for " ("
echo 'some text => desired_string (more text)' | grep -oP '(?<==> ).*?(?= \()'

# sed(more portable)
echo 'some text => desired_string (more text)' | sed -n 's/.*=> \(.*\) (.*/\1/p'

# awk
# -F'=> | \\(': splits by either => or (
# {print $2}: prints the part between them
echo 'some text => desired_string (more text)' | awk -F'=> | \\(' '{print $2}'

综合实践样例

根据[Linux笔记]中已经学习的命令,综合运用于实际实践中。

04 执行任务并定时关闭

首行是标记使用的脚本的类型,此处为bash脚本。
两个命令间用&连接,前面的命令执行后挂在后台并立即开始执行后面的命令。用&&连接,只有前面的命令执行完毕并退出后才会执行后面的命令。
第一行开启roscore后,计时两秒后结束roscorekill -2相当于Ctrl+C-2就是Ctrl+C发出的siginit

#!/bin/bash
roscore &
sleep 3 && kill -2

05 解压文件名有规律的系列文件

这一系列的文件名为“17.7z”、“18.7z”…“21.7z”。解压密码为“blue”。

#!/bin/bash

set -u

for i in {17..21}
do
  filename=$i".7z"
  7z x $filename -pblue
done

解压所在文件夹内的所有以“.7z”结尾的文件,解压密码为“blue”。

ls | while read line
do
  file=$line
  if echo $file | grep -q -E '\.7z$'
  then
    echo $file
    7z x $file -pblue
  fi
done

07 修改某文件夹下的特定文件

遍历所在指定文件夹及其所有子文件夹中的markdown文档,并把第四行修改为<日期+时间+时区>

#!/bin/bash

files=$(find "." -type f)

# 遍历文件
for file in $files; do
    # 检查文件名是否符合特定首尾条件
    if [[ "$file" == *_posts* && "$file" == *.md ]]; then
        # 获取文件的最近修改时间
        last_modified=$(stat -c "%Y" "$file")

        # 将时间戳转换为日月年时分秒的格式
        last_modified_formatted=$(date -d @"$last_modified" +"%Y-%m-%d %H:%M:%S %z")

        # 替换文件的第四行为最近修改时间
        sed -i "4s/.*/date:   $last_modified_formatted/" "$file"

        echo "Updated date in $file to $last_modified_formatted"
    fi
done

09 批量删除某个名称的文件夹

find . -type d -name .git -prune -exec rm -r {} \;
  • .: 指定查找的范围,look for in entire folder
  • -name: find命令的参数,后面接要查找的文件夹名称
  • .git: 我们想要删除的文件夹的名称,此处我想删除的文件夹为“.git”
  • -exec:
  • {}: can be read as “for each matching file/ folder”
  • \; is a terminator for the -exec clause

下面的这个命令也可以使用。

rm -rf `find . -type d -name a`

11 统计文件名

ls | while read line
do
  file=$line
  if echo $file | grep -q -E '\.bag$'
  then
    if [ 18 = `echo ${#file}` ] 
      then
      # time=`rosbag info $file | grep start`
      # str=`date -d @${time: 0-14: 13} "+%Y-%m-%d %H:%M:%S"`
      time=${file: 0: 14}
      echo $time >> date_t.txt
    fi
  fi
done

14 bind several processes via service

We can create several processes and kill them in one time via service. Here we create a user service, which is different from system service. First, # create a folder, which will contain service file and shell script. Assume username is “bs”.

mkdir -p /home/bs/.config/systemd/user/ 
cd /home/bs/.config/systemd/user/

Second, create a service file named myservice.service like below.

[Unit]
Description=describe your service here
# This service starts your custom script after the network is available. If this service id fully independent and does not rely on the state of any other services, you can delete this line
After=network.target

[Service]
Type=simple
# Use `ExecStart` to call a wrapper script to ensure that the script handles the termination of all its child processes upon receiving a signal to stop.
# shell script, it must be absolute path
ExecStart=/home/bs/.config/systemd/user/shell.sh
# ensures that all processes started by the service (including any child processes started by the script) will be terminated when the service itself is stopped.
KillMode=control-group
# on-failure: restart this service if this service dies; no: Don't restart the service automatically
Restart=on-failure

[Install]
WantedBy=default.target
# This service will be started at the default runlevel

Third, reload the user-level systemd configuration.

systemctl --user daemon-reload

Fourth, enable the service.

systemctl --user enable myservice.service

Fifth, don’t forget to create shell script. Now let’s create a script named shell.sh like below. Here I take ros as example.

#!/bin/bash

# Function to terminate all roslaunch processes
cleanup() {
    echo "Stopping all ROS nodes..."
    kill -SIGINT "$roslaunch1_pid" "$roslaunch2_pid" "$roslaunch3_pid"
}

# The trap command in the script sets up a handler for SIGTERM, which is the signal sent by systemd when the service is stopped. This handler function, cleanup, will explicitly kill all the background processes.
trap cleanup SIGTERM

# Start roslaunch commands in background
sleep 1 && roslaunch your_package launch_file1.launch & roslaunch1_pid=$!
sleep 2 && roslaunch your_package launch_file2.launch & roslaunch2_pid=$!
sleep 3 && roslaunch your_package launch_file3.launch & roslaunch3_pid=$!

# The wait command makes the main script wait for all the background processes, which means the main service process remains active as long as the background processes are running.
wait $roslaunch1_pid $roslaunch2_pid $roslaunch3_pid

We have completed service and shell file. We can use this service now.

systemctl --user start myservice.service # start service
systemctl --user status myservice.service # check service
systemctl --user restart myservice.service # restart service
systemctl --user stop myservice.service # stop service

Note that if you make some changes to service file, don’t forget to run this cmd systemctl --user daemon-reload to make it effect.

15 update date from local ntp server

create ntp server
Assume that ip of ntp server is 192.168.1.100 and host name is “bs”. Run cmds below:

sudo apt install ntp # install ntp
sudo systemctl start ntp # start ntp

set client
Run cmd below on client that need update date and time.

sudo nano /etc/systemd/timesyncd.conf # you can also use vim or gedit

Edit like below

[Time]
NTP=bs 192.168.1.100
FallbackNTP=bs 192.168.1.100

Then save and close the file. Restart the timesyncd service with cmd below:

sudo systemctl restart systemd-timesyncd

Check that the NTP synchronization is working correctly with:

timedatectl status

If everything is going well, you will see System lock synchronized is yes. You can also run date and check whether the result is right.

16 Creating a Script to Run at Boot with Cron

  1. Prepare Your Script: Ensure your script in /usr/local/func/ is executable. If it’s not, make it executable:
sudo chmod +x /usr/local/func/myscript
  1. Edit the Crontab:
crontab -e

Add the following line to schedule your script to run at every system boot:

@reboot /usr/local/func/myscript

This line tells cron to run /usr/local/func/myscript every time the system boots up.

  1. Save and Exit: After adding the line, save and exit the editor. cron will automatically install the new crontab.

  2. Verify: To ensure your crontab is set up correctly, you can list your current user’s crontab entries:

crontab -l

This command will show all scheduled cron jobs, including your new @reboot job.

  1. Additional Considerations Environment: Cron jobs run with a limited environment, meaning many of the environmental variables available in a full shell session (like those started by logging in) may not be available. If your script relies on certain environment variables, you might need to define them in the script or in the crontab entry. Or you can run source /home/${user}/.bashrc to make script run like in a terminal.

Logging: Since cron jobs don’t have a terminal, you might want to redirect your script’s output to a file for debugging:

@reboot /usr/local/func/myscript >> /var/log/myscript.log 2>&1

This setup ensures that your script runs at system boot without involving system-wide services or requiring systemd, fitting your preference for a simpler, less formal method.

17 split and convert string

#!/bin/bash

# colorful output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
NC='\033[0m'

if [ "$#" -eq 0 ]; then
  echo -e "${RED}Should contain project name.${NC}"
  exit
fi

# (absolute path (relative path))
SHELL_DIR=$(realpath $(dirname "${BASH_SOURCE[0]}"))

name=$1

# whether file exists
# if [ -e "$name" ]; then
#   echo -e "${BLUE}File ${name} exists, skip generation.${NC}"
#   exit
# fi

# if [ -d "$name" ]; then
#     echo -e "${YELLOW}Directory ${name} exists, skip creation.${NC}"
#     exit
# fi

# Internal Field Separator
IFS='_' 
# Convert string into an array
read -ra PARTS <<< "$name"
# Reset IFS if needed
unset IFS

# STRING="part1_part2_part3"
# # Get specific parts
# FIRST_PART=$(echo "$STRING" | cut -d '_' -f 1)
# SECOND_PART=$(echo "$STRING" | cut -d '_' -f 2)
# THIRD_PART=$(echo "$STRING" | cut -d '_' -f 3)

# traverse an array 
for PART in "${PARTS[@]}"; do
  # echo "$PART"
  # Convert first character to uppercase
  Name=${Name}$(echo "$PART" | sed 's/./\U&/') 
  # convert all character to uppercase
  NAME=${NAME}${PART^^} 
  NA_ME=${NA_ME}${PART^^}"_"
done

mkdir -p $name
cd $name

echo $Name
echo $NAME
echo $NA_ME

# "eval" allows you to construct and execute commands dynamically.
eval "${NAME}VERSION=1.0"
echo "${COVFERVERSION}"

date "+%Y-%m-%d %H:%M" # use date as separator

var_name="${NAME}VERSION"
# Create the variable dynamically
declare "$var_name=2.0"
# Access the newly created variable using indirect reference
echo "${!var_name}"  # Correct way to reference it