渡鴉之丘: 陽春FTP設定＆簡單拋檔script筆記

之前硬著頭皮找一些古早資料弄的，開一個簡單 FTP 服務拋檔（但沒有啟動 SSL 這種層級的設定）。這邊記錄下來。

然後卡了很久，硬著頭皮在整理放出來。

這個筆記分兩段：

第一部份是 FTP server 設定，會弄一個簡單的帳密檢查跟 virtual user

第二段則是一個簡單的 shell script，主要是紀錄檔名套日期的 shell script。

這邊一樣用 LXC container 來筆記

lab@lxdlab:~$ lxc launch images:centos/8-Stream/amd64 ftpsvr
Creating ftpsvr
Starting ftpsvr
lab@lxdlab:~$ lxc launch images:centos/8-Stream/amd64 ftpcli
Creating ftpcli
Starting ftpcli
lab@lxdlab:~$

以下在 LXC Container 裡面不需要，但實際環境需要，還是紀錄下來：分別是 OS 防火牆以及 SELinux 功能，不過這邊紀錄的是簡單作的方法

[root@ftpsvr ~]# firewall-cmd --zone=public --permanent --add-service=ftp
success
[root@ftpsvr ~]# firewall-cmd --reload
success
[root@ftpsvr ~]# getenforce
Permissive
[root@ftpsvr ~]#
[root@ftpsvr ~]# grep ^SELINUX /etc/selinux/config
SELINUX=permissive
SELINUXTYPE=targeted
[root@ftpsvr ~]#

下面不多廢話，直接把手上版本紀錄起來：這邊啟動的功能：

停用匿名連線
增加帳號登入
指定一個目錄給人放

安裝：

[root@ftpsvr ~]# yum install -y vsftpd
...略... 
[root@ftpsvr ~]# cp /etc/vsftpd/vsftpd.conf /etc/vsftpd/vsftpd.conf.orig
[root@ftpsvr ~]# ls /etc/vsftpd/
ftpusers  user_list  vsftpd.conf  vsftpd_conf_migrate.sh  vsftpd.conf.orig
[root@ftpsvr ~]#

[root@ftpsvr ~]# mkdir -p /data/ftpdir
[root@ftpsvr ~]# useradd ftpuser
[root@ftpsvr ~]# chown -R ftpuser. /data/ftpdir

主設定檔

[root@ftpsvr ~]# sed -e 's/anonymous_enable=YES/anonymous_enable=NO/g' -i /etc/vsftpd/vsftpd.conf
[root@ftpsvr ~]# grep 'local_enable=YES' /etc/vsftpd/vsftpd.conf
local_enable=YES
[root@ftpsvr ~]# grep '^write_enable=YES' /etc/vsftpd/vsftpd.conf
write_enable=YES
[root@ftpsvr ~]# sed -e 's/#anon_upload_enable=YES/anon_upload_enable=NO/g' -i /etc/vsftpd/vsftpd.conf
[root@ftpsvr ~]# sed -e 's/#anon_mkdir_write_enable=YES/anon_mkdir_write_enable=NO/g' -i /etc/vsftpd/vsftpd.conf
[root@ftpsvr ~]# sed -e 's/listen=NO/listen=YES/g' -i /etc/vsftpd/vsftpd.conf
[root@ftpsvr ~]# sed -e 's/listen_ipv6=YES/listen_ipv6=NO/g' -i /etc/vsftpd/vsftpd.conf
[root@ftpsvr ~]# sed -e 's/#chown_uploads=YES/chown_uploads=YES/g' -i /etc/vsftpd/vsftpd.conf
[root@ftpsvr ~]# sed -e 's/#chown_username=whoever/chown_username=ftpuser/g' -i /etc/vsftpd/vsftpd.conf
[root@ftpsvr ~]# 
[root@ftpsvr ~]# cat << EOF >> /etc/vsftpd/vsftpd.conf
##
## Additionally Added
##
# Virtual users will use the same privileges as local users.
virtual_use_local_privs=YES
# Specifies the directory vsftpd changes to after a local user logs in.
local_root=/data/ftpdir/
# Activates virtual users
guest_enable=YES
guest_username=ftpuser
# Chroot user and lock down to their home dirs
#chroot_local_user=YES
# Hide ids from user
hide_ids=YES

EOF
[root@ftpsvr ~]#

以下設定的 FTP 登入用的 virtual users：

這邊分別是設定檔 (僅用在生成以下 DB 檔)

virtual-user.txt

與生成的 Berkeley DB 資料庫檔

virtual-user.db

並在 PAM 裡面指定 (不帶副檔名)

[root@ftpsvr ~]# cat << EOF >> /etc/vsftpd/virtual-user.txt
fuser1
fuser1pwd
EOF
[root@ftpsvr ~]# db_load -T -t hash -f /etc/vsftpd/virtual-user.txt /etc/vsftpd/virtual-user.db
[root@ftpsvr ~]# chmod 600 /etc/vsftpd/virtual-user.db
[root@ftpsvr ~]# chmod 600 /etc/vsftpd/virtual-user.txt
[root@ftpsvr ~]# 
[root@ftpsvr ~]# mv /etc/pam.d/vsftpd /etc/pam.d/vsftpd.orig
[root@ftpsvr ~]# ls -l /etc/pam.d/vsftpd.orig
-rw-r--r-- 1 root root 335 Jan 27 07:36 /etc/pam.d/vsftpd.orig
[root@ftpsvr ~]# cat /etc/pam.d/vsftpd.orig
#%PAM-1.0
session    optional     pam_keyinit.so    force revoke
auth       required     pam_listfile.so item=user sense=deny file=/etc/vsftpd/ftpusers onerr=succeed
auth       required     pam_shells.so
auth       include      password-auth
account    include      password-auth
session    required     pam_loginuid.so
session    include      password-auth
[root@ftpsvr ~]# cat << EOF >> /etc/pam.d/vsftpd
#%PAM-1.0
auth       required     pam_userdb.so db=/etc/vsftpd/virtual-user
account    required     pam_userdb.so db=/etc/vsftpd/virtual-user
session    required     pam_loginuid.so

EOF
[root@ftpsvr ~]# chmod 644 /etc/pam.d/vsftpd
[root@ftpsvr ~]#

啟動 FTP 服務

[root@ftpsvr ~]# service vsftpd start
Redirecting to /bin/systemctl start vsftpd.service
[root@ftpsvr ~]# 
[root@ftpsvr ~]# service vsftpd status
Redirecting to /bin/systemctl status vsftpd.service
● vsftpd.service - Vsftpd ftp daemon
   Loaded: loaded (/usr/lib/systemd/system/vsftpd.service; disabled; vendor preset: disabled)
  Drop-In: /run/systemd/system/vsftpd.service.d
           └─zzz-lxc-service.conf
   Active: active (running) since Tue 2022-05-31 03:32:27 UTC; 4s ago
  Process: 1226 ExecStart=/usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf (code=exited, status=0/SUCCESS)
 Main PID: 1227 (vsftpd)
    Tasks: 1 (limit: 26213)
   Memory: 648.0K
   CGroup: /system.slice/vsftpd.service
           └─1227 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf

May 31 03:32:27 ftpsvr systemd[1]: Starting Vsftpd ftp daemon...
May 31 03:32:27 ftpsvr systemd[1]: Started Vsftpd ftp daemon.
[root@ftpsvr ~]#

接著是 FTP 登入指令跟一個陽春 script 的筆記這 script 主要是把檔案清單列進來，不過補上一個小功能：比對檔名的時戳

lab@lxdlab:~$ lxc shell ftpcli
[root@ftpcli ~]# dnf install -y epel-release
[root@ftpcli ~]# dnf install -y ftp
[root@ftpcli ~]# dnf install --enablerepo=powertools -y moreutils
[root@ftpcli ~]# 
[root@ftpcli ~]# cat << EOF >> /etc/hosts
10.18.0.186    ftpsvr
EOF
[root@ftpcli ~]#

先測試 FTP 會不會通

[root@ftpcli ~]# ftp -inv ftpsvr
Trying fd42:e1ce:ffc8:a5cd:216:3eff:fe2b:87ef...
ftp: connect to address fd42:e1ce:ffc8:a5cd:216:3eff:fe2b:87efConnection refused
Trying 10.18.0.186...
Connected to ftpsvr (10.18.0.186).
220 (vsFTPd 3.0.3)
ftp> user fuser1 fuser1pwd
331 Please specify the password.
230 Login successful.
ftp> ls
227 Entering Passive Mode (10,18,0,186,96,32).
150 Here comes the directory listing.
226 Directory send OK.
ftp> pwd
257 "/data/ftpdir" is the current directory
ftp> exit
221 Goodbye.
[root@ftpcli ~]#

Script 內容：

檔名：ftp_put_files.sh

#!/bin/bash
## ftp 拋檔
## 把連線資訊跟檔案清單都用設定檔控制
## Note: 
## * 參考樣板：https://ravenonhill.blogspot.com/2021/11/bash-shell-script-script.html
## * file_var_check() 只是用來管控有關的設定檔 (防止檔案誤刪卻沒注意到)
## * 沒有仔細檢查檔名清單
## * 檔名時戳的控制在環境變數設定檔中

## Get script filename
SCRIPTBN=$(basename $0)
SCRIPTFN="${SCRIPTBN%.*}"
SCRIPTDR=$(dirname $0)

## Execution Start Time
EXECUTEDATE=$(date +%Y%m%d)

## Dependent scripts/config files
DEPENVFL1="${SCRIPTDR}/copy_result_ftp.sh"
DEPENVFL2="${SCRIPTDR}/copy2_result_ftp.sh"
DEPENVFL3="${SCRIPTDR}/copy3_result_ftp.sh"
DEPFLIST1="${SCRIPTDR}/copy_result_filelist.txt"
DEPFLIST2="${SCRIPTDR}/copy2_result_filelist.txt"
DEPFLIST3="${SCRIPTDR}/copy3_result_filelist.txt"

## Log file path
LOGDIR=$SCRIPTDR/log/${EXECUTEDATE}/
LDRLOGFN=$LOGDIR/${SCRIPTFN}_${EXECUTEDATE}.log

file_var_check() {
  cat <<EOF
#########################
# Dependent Files Check #
#########################
EOF

  local varprefix=$1
  
  ## Check dependent script files (loop over variables)
  for ((i = 1; ; i++)); do
    local chkvar="${varprefix}$i"
    if [ -z ${!chkvar+x} ]; then
      echo "$chkvar is unset"
      break
    else
      echo "$chkvar is set to '${!chkvar}'"
    fi

    if [ ! -f "${!chkvar}" ]; then
      echo "[FATAL] File '${!chkvar}' not found!"
      exit 1
    fi
  done
}

function Usage()
{
   cat << EOF 
Usage : $SCRIPTBN -c <connection_env_file> -f <file_list>
        -c : Specify connection environment variable file
        -f : Specify file list
EOF
exit;
}

cmd_arg(){
## Command line options

  if [ $# -ne 0 ] ; then
    while getopts "c:f:" opts ; do
      case  "$opts" in
        c)
          ftpenv=${OPTARG}
          echo "FTP Connection Information file: ${ftpenv}"
          source ${ftpenv}
          ;;
        f)
          putlist=${OPTARG}
          echo "File list config: ${putlist}"
          ;;
        *)
          Usage
          ;;
      esac
    done
  else
    Usage
  fi

}


putftpfile(){
  echo "==FTP Connection Information=="
  echo "FTP Host IP: $FTPHOST"
  echo "FTP Login User: $FTPUSER"
  
  echo "==Reading File list=="
  while read fullpathstring; do
    #echo "Copying the file: ${fullpathstring/<datetime_month>/$(date -d '1 month ago' +%Y%m)}"
    echo "Copying the file: ${fullpathstring/$FILEPTN/$FILEYMD}"
    filefullpath="${fullpathstring/$FILEPTN/$FILEYMD}"
    filename="$(basename $filefullpath)"
#    ftp -inv $FTPHOST <<EOF
#    user $FTPUSER $FTPPSWD
#    cd $FTPPATH
#    mput $filefullpath
#    bye
#EOF
    # 我想縮排。。用 bash-specific one-word here-doc (here-string)
    # https://unix.stackexchange.com/questions/76481/cant-indent-heredoc-to-match-code-blocks-indentation
    ftp -inv $FTPHOST <<< "
    user $FTPUSER $FTPPSWD
    cd $FTPPATH
    put $filefullpath $filename
    quit"
  done < ${putlist}

}


do_main() {
  ##################
  ## Main process ##
  ##################
  
  ## Add log directory first
  mkdir -p ${LOGDIR}
  
  ## Redirect stderr/stdout to a process substitution that adds the prefix of choice
  ## And write all output to the log file
  ## 用 trap 抓 signal 並把 output STDOUT/STDERR 導向檔案，兩種要一起放括號裡面
  ## 此外 >& _file_descriptor_ 跟 &> _log_file_name_ 兩者功能不相同也不相關
  ## 額外從 EPEL 裝 moreutils 使用 ts 指令
  exec &> >( ts '[%Y-%m-%d.%H:%M:%S] ' > ${LDRLOGFN} )
  
  ## exit when any command fails
  set -e
  
  ## Script Start
  cmd_arg "$@"
  file_var_check DEPENVFL
  file_var_check DEPFLIST
  putftpfile

}

# Entry Point
do_main "$@"

以下為檔案複製的設定：

FTP 連線與路徑設定：copy_result_ftp.sh
本地要複製進 FTP 的路徑：copy_result_filelist.txt

右手邊因為我抄的迴圈不太好，要至少一個 Linux 底下的 newline（清單要超過一行），尤其是只有填一個檔要注意

## 檔名：copy_result_ftp.sh
# FTP connection info
FTPHOST='ftpsvr'
FTPUSER='fuser1'
FTPPSWD='fuser1pwd'
FTPPATH='/data/ftpdir/'
# File timestamp pattern
FILEPTN='<datetime_month>'
#FILEYMD=$(date -d '1 month ago' +%Y%m)
FILEYMD=$(date +%Y%m)

檔名：copy_result_filelist.txt

/path/to/result/file1_<datetime_month>.csv
/path/to/result/file2_<datetime_month>.csv
/path/to/result/file3_<datetime_month>.csv

排程

## FTP: ftpcli to ftpsvr
0 20 5 * *   ~/ftp_client_script/ftp_put_files.sh -c ~/ftp_client_script/copy_result_ftp.sh -f ~/ftp_client_script/copy1_result_filelist.txt
0 15 12 12 *   ~/ftp_client_script/ftp_put_files.sh -c ~/ftp_client_script/copy2_result_ftp.sh -f ~/ftp_client_script/copy2_result_filelist.txt
30 4 * * *   ~/ftp_client_script/ftp_put_files.sh -c ~/ftp_client_script/copy3_result_ftp.sh -f ~/ftp_client_script/copy3_result_filelist.txt

參考資料：

[FTP Server]

https://www.tecmint.com/install-ftp-server-in-centos-7/

http://blog.itist.tw/2016/08/build-ftp-server-with-vsftpd-on-centos-7.html

https://linux.vbird.org/linux_server/centos6/0410vsftpd.php

vsftpd_virtualuser_add

https://www.linuxcloudvps.com/blog/setup-virtual-users-in-vsftpd/

https://calvertyang.github.io/2017/02/06/how-to-setup-virtual-users-for-vsftpd-in-ubuntu/

virtual-user vsftpd

https://help.ubuntu.com/community/vsftpd#Configure_VSFTPD_for_virtual_user

local_root vsftpd

https://serverfault.com/questions/222906/vsftpd-local-root-var-www-sites-user-doesnt-get-interpreted