Zstandard 简介
Zstandard 也叫做 zstd,是一种快速的无损压缩算法,是针对 zlib 级别的实时压缩方案,以及更好的压缩比。它由一个非常快的熵阶段,由 Huff0 和 FSE 库提供。这个项目是作为开源的 BSD 许可收费的库,以及一个生成和解码 .zst 格式。Facebook 正在使用其快速无损压缩算法库,改进了 Facebook 的存储/备份的要求。
安装方法
升级 tar
安装前需要确认 tar
版本为 1.30 以上,验证版本号,如果低于 1.30 需要升级版本,1.30 以上可以忽略;
tar --version
tar (GNU tar) 1.34 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Written by John Gilmore and Jay Fenlason.
下载 tar
命令的 latest 版源码压缩包:
wget https://ftp.gnu.org/gnu/tar/tar-latest.tar.gz
解压
tar xvzf tar-latest.tar.gz
编译代码
cd tar-1.3.4
FORCE_UNSAFE_CONFIGURE=1
export FORCE_UNSAFE_CONFIGURE
./configure && make
复制到运行目录
cp src/tar /bin/tar
安装 Zstandard
可以使用 yum
或者 apt
安装
#Ubuntu/Debian
apt install zstd
#Centos
yum install zstd
但这里博主推荐使用编译安装,也就两行代码
git clone https://github.com/facebook/zstd.git
cd zstd && make && sudo make install
复制到运行目录
cp /usr/local/bin/zstd /usr/bin/
具体使用方法这里就不介绍了,请自行搜索。
备份脚本
这里博主使用的是军哥 LNMP 软件包里的备份脚本修改,给出 ftp 版和 lftp 版,自行选择
lftp 版
#!/usr/bin/env bash
#Funciont: Backup website and mysql database
#Author: licess
#Website: https://lnmp.org
#IMPORTANT!!!Please Setting the following Values!
Backup_Home="/home/backup/"
MySQL_Dump="/usr/local/mysql/bin/mysqldump"
######~Set Directory you want to backup~######
Backup_Dir=("/home/wwwroot/vpser.net" "/home/wwwroot/lnmp.org")
######~Set MySQL Database you want to backup~######
Backup_Database=("lnmp" "vpser")
######~Set MySQL UserName and password~######
MYSQL_UserName='root'
MYSQL_PassWord='yourrootpassword'
######~Enable Ftp Backup~######
Enable_FTP=0
# 0: enable; 1: disable
######~Set FTP Information~######
FTP_Host='1.2.3.4'
FTP_Username='vpser.net'
FTP_Password='yourftppassword'
FTP_Dir="backup"
#Values Setting END!
TodayWWWBackup=www-*-$(date +"%Y%m%d").tar.zst
TodayDBBackup=db-*-$(date +"%Y%m%d").sql.zst
OldWWWBackup=www-*-$(date -d -30day +"%Y%m%d").tar.zst
OldDBBackup=db-*-$(date -d -30day +"%Y%m%d").sql.zst
Backup_Dir()
{
Backup_Path=$1
Dir_Name=`echo ${Backup_Path##*/}`
Pre_Dir=`echo ${Backup_Path}|sed 's/'${Dir_Name}'//g'`
tar '-I zstd -9 -T0' -cf ${Backup_Home}www-${Dir_Name}-$(date +"%Y%m%d").tar.zst -C ${Pre_Dir} ${Dir_Name}
}
Backup_Sql()
{
${MySQL_Dump} -u$MYSQL_UserName -p$MYSQL_PassWord $1 | zstd -9 -T0 > ${Backup_Home}db-$1-$(date +"%Y%m%d").sql.zst
}
if [ ! -f ${MySQL_Dump} ]; then
echo "mysqldump command not found.please check your setting."
exit 1
fi
if [ ! -d ${Backup_Home} ]; then
mkdir -p ${Backup_Home}
fi
if [ ${Enable_FTP} = 0 ]; then
type lftp >/dev/null 2>&1 || { echo >&2 "lftp command not found. Install: centos:yum install lftp,debian/ubuntu:apt-get install lftp."; }
fi
echo "Backup website files..."
for dd in ${Backup_Dir[@]};do
Backup_Dir ${dd}
done
echo "Backup Databases..."
for db in ${Backup_Database[@]};do
Backup_Sql ${db}
done
echo "Delete old backup files..."
rm -f ${Backup_Home}${OldWWWBackup}
rm -f ${Backup_Home}${OldDBBackup}
if [ ${Enable_FTP} = 0 ]; then
echo "Uploading backup files to ftp..."
cd ${Backup_Home}
lftp ${FTP_Host} -u ${FTP_Username},${FTP_Password} << EOF
cd ${FTP_Dir}
mrm ${OldWWWBackup}
mrm ${OldDBBackup}
mput ${TodayWWWBackup}
mput ${TodayDBBackup}
bye
EOF
echo "complete."
fi
ftp 版
#!/usr/bin/env bash
#Funciont: Backup website and mysql database
#Author: licess
#Website: https://lnmp.org
#IMPORTANT!!!Please Setting the following Values!
Backup_Home="/home/backup/"
MySQL_Dump="/usr/local/mysql/bin/mysqldump"
######~Set Directory you want to backup~######
Backup_Dir=("/home/wwwroot/vpser.net" "/home/wwwroot/lnmp.org")
######~Set MySQL Database you want to backup~######
Backup_Database=("lnmp" "vpser")
######~Set MySQL UserName and password~######
MYSQL_UserName='root'
MYSQL_PassWord='yourrootpassword'
######~Enable Ftp Backup~######
Enable_FTP=0
# 0: enable; 1: disable
######~Set FTP Information~######
FTP_Host='1.2.3.4'
FTP_Username='vpser.net'
FTP_Password='yourftppassword'
FTP_Dir="backup"
#Values Setting END!
TodayWWWBackup=www-*-$(date +"%Y%m%d").tar.zst
TodayDBBackup=db-*-$(date +"%Y%m%d").sql.zst
OldWWWBackup=www-*-$(date -d -30day +"%Y%m%d").tar.zst
OldDBBackup=db-*-$(date -d -30day +"%Y%m%d").sql.zst
Backup_Dir()
{
Backup_Path=$1
Dir_Name=`echo ${Backup_Path##*/}`
Pre_Dir=`echo ${Backup_Path}|sed 's/'${Dir_Name}'//g'`
tar '-I zstd -9 -T0' -cf ${Backup_Home}www-${Dir_Name}-$(date +"%Y%m%d").tar.zst -C ${Pre_Dir} ${Dir_Name}
}
Backup_Sql()
{
${MySQL_Dump} -u$MYSQL_UserName -p$MYSQL_PassWord $1 | zstd -9 -T0 > ${Backup_Home}db-$1-$(date +"%Y%m%d").sql.zst
}
if [ ! -f ${MySQL_Dump} ]; then
echo "mysqldump command not found.please check your setting."
exit 1
fi
if [ ! -d ${Backup_Home} ]; then
mkdir -p ${Backup_Home}
fi
if [ ${Enable_FTP} = 0 ]; then
type ftp >/dev/null 2>&1 || { echo >&2 "ftp command not found. Install: centos:yum install ftp,debian/ubuntu:apt-get install lftp."; }
fi
echo "Backup website files..."
for dd in ${Backup_Dir[@]};do
Backup_Dir ${dd}
done
echo "Backup Databases..."
for db in ${Backup_Database[@]};do
Backup_Sql ${db}
done
echo "Delete old backup files..."
rm -f ${Backup_Home}${OldWWWBackup}
rm -f ${Backup_Home}${OldDBBackup}
if [ ${Enable_FTP} = 0 ]; then
echo "Uploading backup files to ftp..."
cd ${Backup_Home}
ftp -ivn ${FTP_Host} << END
user ${FTP_Username} ${FTP_Password}
type binary
cd ${FTP_Dir}
delete ${OldWWWBackup}
delete ${OldWWWBackup}
put ${TodayWWWBackup}
put ${TodayDBBackup}
bye
END
echo "complete."
fi
也可以直接使用 SFTP
协议备份到另一台服务器
if [ ${Enable_FTP} = 0 ]; then
echo "Uploading backup files to ftp..."
cd ${Backup_Home}
lftp -u ${FTP_Username},${FTP_Password} sftp://${FTP_Host} << EOF
cd ${FTP_Dir}
mrm ${OldWWWBackup}
mrm ${OldDBBackup}
mput ${TodayWWWBackup}
mput ${TodayDBBackup}
bye
EOF
echo "complete."
fi
如果换了默认端口的话在 ${FTP_Host}
里,把端口加上
FTP_Host='1.2.3.4:666'
WIN 解压缩
备份后缀为 .zst
在 WIN 系统 RAR 和 7ZIP 都是无法解压的,可以安装 7ZIP 的分支,也是免费开源,点这里官方下载
结语
使用军哥的 LNMP 网站备份脚本在备份的时候 CPU 很吃力,往往都飙到 70%以上,服务器负载持续飙升的罪魁祸首是备份时的压缩算法 gzip。在改用 Zstandard 压缩,测试后 CPU 仅在 8%左右,大大减少的服务器的负担。
最近在学习这方面的资料,需要对整个系统进行异地备份,然后找到了这篇文章,感谢作者的付出。本文原文地址:https://www.yunloc.com/2438.html
您也可以联系文章作者本人进行修改,若内容侵权或非法,可以联系我们进行处理。
任何个人或组织,转载、发布本站文章到任何网站、书籍等各类媒体平台,必须在文末署名文章出处并链接到本站相应文章的URL地址。
本站文章如转载自其他网站,会在文末署名原文出处及原文URL的跳转链接,如有遗漏,烦请告知修正。
如若本站文章侵犯了原著者的合法权益,亦可联系我们进行处理。
hi4个月前0
请问有详细一点的自己搭建的教程吗你好5个月前0
你好,可以再帮我看看吗? 我已经按照你的方法设定了,还是一样,wordpress后台的 Purge Varnish Cache 插件还是清除不到cache,依旧显示 the varnish control terminal is not responding at。谢谢 https://mjj.today/i/Srk2Tz https://mjj.today/i/Srkcoi你好5个月前0
对,你说的没错,我配置的时候改了一些东西,现在我按照你的教学,可以启动了,网页可以缓存了,不过wordpress 清除cache 那个插件没用的,我输入本地回环地址127.0.0.1 :6082 ,再输入API key ,插件显示the varnish control terminal is not responding at 127.0.0.1:6082,就你图片那样,然后试一下点击清除cache 那里,他显示error,研究了一天,还是没有不行。你好5个月前1
你好,为啥我按照你的方法,到第三部分,去到真正后源的服务器设定Varnish 部分,我填了真正后源的IP跟端口跟域名,然后重启 Varnish ,就出现这样了? 这是怎么回事? 谢谢 [Linux] AMH 7.1 https://amh.sh [varnish-6.6 start] ================================================== =========== [OK] varnish-6.6 is already installed. Could not delete 'vcl_boot.1713549650.959259/vgc.sym': No such file or directory Error: Message from VCC-compiler: VCL version declaration missing Update your VCL to Version 4 syntax, and add vcl 4.1; on the first line of the VCL files. ('/home/usrdata/varnish/default.conf' Line 1 Pos 1) ... #--- Running VCC-compiler failed, exited with 2 VCL compilation failedchu5个月前0
很完善的教程‘hu5个月前0
我用gmail EMAIL_SERVER="smtp://********@gmail.com:bpyfv*********[email protected]:587"叽喳5个月前0
MAIL_SERVER="smtp://[email protected]:[email protected]:587" 大佬 这个使用outlook 或者gmail 是什么样子的格式? 邮寄已经开启smtp了hu5个月前0
输入框的问题解决了,我没有设置反代,NEXTAUTH_URL改为域名+端口就好了