I will improve it in the near future after I have it restructured. It's then also easier to implement little changes
Automatic backup of config using a "pull method"
- Ersteller frater
- Erstellt am
I have restructured the script and tested it against the cluster of modems I am supervising.
It relies on 'httping' to be installed
The script can take command line parameters now...
- -c use current directory instead of default
- -d <folder> use <folder> instead of default
- comma delimited parameters for 1 modem (same syntax as the list)
If no parameters for a modem are given, it will search for a list containing modem parameters in the designated folder and parse that list and fetch both the Freetz config and the AVM-config.
The parameters for 1 modem can be as basic as its external address. It will then use the default values for the rest....
The AVM-config is password-protected with the external password.
The files will be placed in a folder and get a little check. If the config is (about) the same as the last one fetched, it will get discarded.
This way you can just create a symlink to the script in /etc/cron.daily and you will always have a folder with different configs.
You can easily see when a config has been changed.
I'm placing those files in a password protected website where I can immediately download one of the configs I need....
Maybe it will save my ass one day...
It can now also get the AVM-config if there's no freetz on it.
If it doesn't detect the Freetz interface it will do an AVM only.
If you don't want to even try the Freetz-port you can define it as port 0
The script could do with some better logging and maybe there are some bugs there.
If you use the script I would appreciate some feedback.
I also would like to know if I should bother posting updates here.
Thanks Daniel and MaxMuster for the URL's I needed to get..
Update May 18 17:30
Added better handling for "#"
You can put comments at the end of the modemline and of course in the beginning
Everything behind the # will be ignored...
Also added some code to prevent duplicate AVM-configs (in case Freetz is not used)
Update May 18 00:00
Not only remove the path, but also the extension from the name of the script to use as the name for modemlist
Update May 19 12:00
I should spend my holidays better, really...
Another improvement to make the detection for changes a bit better and more sensible.
If only an AVM-config is made, the script kept the config too often. I now check for big differences by ignoring the binary data in the config.
If, however, the latest config is older than 30 days the script does an exact comparison.....
Update May 20 13:15
The script still made some unnecessary backups, so I investigated it and improved the comparison. Added some extra logging...
If used as a cronjob, it will only output data if a changes is found....
Some examples:
Simple help
Code:
freetz_getconfigs -?
Code:
freetz_getconfigs
Code:
freetz_getconfigs -d /opt remote.domain.com
Code:
freetz_getconfigs -c remote.domain.com
Code:
freetz_getconfigs -c mymodem,192.168.178.1,,81,,,80
Code:
freetz_getconfigs -c mymodem,192.168.178.1,,0,,,80
Code:
freetz_getconfigs -c remote.domain.com,,,,,,451# cat /usr/local/sbin/freetz_getconfigs
Code:
#!/bin/sh
# Define constants
DATESTAMP=`date +%Y-%m-%d.%H-%M`
FOLDER=/var/www/vhosts/domain.com/freetz
USER_DEFAULT=admin
PASS_DEFAULT=secret
PORT_DEFAULT=6080
PORTAVM_DEFAULT=450
MINSIZE=20000
HEADLESS=
tty >/dev/null || HEADLESS=true
while getopts cd: opt
do
case $opt in
c) FOLDER="`pwd`" ;;
d) FOLDER="$OPTARG";;
?) printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM]\n" $0
exit 2;;
esac
done
shift `expr ${OPTIND} - 1`
# Subroutines
age_of_file ()
{
if [ ! -e "$1" ] ; then
echo 0
else
AGE_IN_SECONDS=$((`date +%s` - `date +%s -r "$1"` ))
echo $((${AGE_IN_SECONDS} / 86400))
fi
}
httpspeed ()
{
# Returns no value and errorlevel 1 if site is unreachable or too slow
# Returns the speed in milliseconds if it's below 2 seconds
_SPEED=`httping -c1 -t1 ${1} 2>/dev/null | \
egrep '.+/.+/.+/.+/.+' | tail -n1 | \
awk -F/ '{print $4}' | awk -F. '{print $1}'`
if [ -z "${_SPEED}" ] ; then
echo "${URL} is unreachable" >&2
elif [ ${_SPEED} -gt 2000 ] ; then
echo "${URL} is slow (${_SPEED} ms)" >&2
else
echo ${_SPEED}
return 0
fi
return 1
}
get_avm_config ()
{
[ ${PORTAVM} -eq 0 ] && return
TMPDIR2=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
if ! cd ${TMPDIR2} 2>/dev/null ; then
echo "Error changing current directory to ${TMPDIR2}. You should never see this message!" >&2
else
# It's intended to be called from WAN (https)
# If PORTAVM begins with 8 it assumes LAN and turns to http
PROTO=https
echo ${PORTAVM} | grep -q '^8' && PROTO=http
URL="${PROTO}://${IP}:${PORTAVM}"
if httpspeed ${URL} >/dev/null ; then
# get challenge key from FB
_CHALLENGE=`curl -s -k \
--user ${USER}:${PASSE} \
"${URL}/login.lua" | \
grep 'challenge' | egrep '[0-9a-f]{8}' | \
awk -F= '{print $2}' | tail -n1 | egrep -o '[0-9a-f]{8}'`
if [ -z "${_CHALLENGE}" ] ; then
echo "Error getting challenge for ${HOST} using ${URL}" >&2
else
# echo "Received Challenge \"${_CHALLENGE}\"" >&2
# build md5 from challenge key and password
_MD5=`echo -n ${_CHALLENGE}"-"${PASSE} | \
iconv -f ISO8859-1 -t UTF-16LE | \
md5sum -b | awk '{print substr($0,1,32)}'`
# assemble challenge key and md5
_RESPONSE=${_CHALLENGE}"-"${_MD5}
# get sid for later use
_SID=`curl -i -s -k \
--user ${USER}:${PASSE} \
-d 'response='${_RESPONSE} \
-d 'page=' \
-d 'username='${USER} \
${URL}/login.lua | \
grep "Location:" | awk -F'=' {' print $NF '}`
if ! curl -s -k \
--user ${USER}:${PASSE} \
--form 'sid='${_SID} \
--form 'ImportExportPassword='${PASSE} \
--form 'ConfigExport=' \
${URL}/cgi-bin/firmwarecfg >TMP1 ; then
echo "Error getting .export for ${HOST} (no data)" >&2
else
if [ `stat -c%s TMP1` -lt ${MINSIZE} ] ; then
echo "Error getting .export for ${HOST} using ${URL} (too small)" >&2
echo "SID: \"${_SID}\" CHALLENGE: \"${_CHALLENGE}\" MD5: \"${_MD5}\" RESPONSE: \"${_RESPONSE}\"" >&2
else
chown ${FUSER}:${FGROUP} TMP1
mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
grep "${IP}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
sort -rn | head -n1 | awk '{print $2}'`
if [ -z "${LATESTCONFIG}" ] ; then # No previous .export is found
echo "Saving \"${DEST_BASE}.export\" because no previous AVM-config has been found" >&2
mv TMP1 "${DEST_BASE}.export"
else
if [ ${TRIED_FREETZ} ] ; then # The Freetz comparison is better, so make that leading
if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
echo "Saving \"${DEST_BASE}.export\" because Freetz config has changed" >&2
mv TMP1 "${DEST_BASE}.export"
fi
else
if [ `age_of_file "${LATESTCONFIG}"` -gt 30 ] ; then # Compare EXACTLY if age is older than 30 days
if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is different" >&2
mv TMP1 "${DEST_BASE}.export"
fi
else
# remove BINARY data and timestamps before making a diff
sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d' "${LATESTCONFIG}" >TMP2
sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d' TMP1 >TMP3
if ! diff TMP2 TMP3 >/dev/null 2>&1 ; then # Compare the 2 stripped configs
echo "Saving \"${DEST_BASE}.export\" because AVM-config has changed" >&2
diff TMP2 TMP3
mv TMP1 "${DEST_BASE}.export"
fi
fi
fi
fi
fi
fi
fi
fi
fi
cd "${FOLDER}"
rm -rf ${TMPDIR2}
}
patch_files ()
{
# remove or patch some of the untarred files as they spoil the check for a changed config
if cd "$1" >/dev/null 2>&1 ; then
rm -f stat.cfg chrony.drift multid.leases 2>/dev/null
rm -f fx_cg fonctrl phonebook voipd_call_stat 2>/dev/null
sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' voip.cfg # remove datestamp in voip.cfg
sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' ar7.cfg # remove datestamp in ar7.cfg
sed -i 's/ *\/\*AVM\*\///g' ar7.cfg # remove the /*AVM*/ tag in ar7.cfg
sed -i 's/; *$/;/g' ar7.cfg # remove trailing spaces after a semicolon
cd - >/dev/null 2>&1
fi
}
get_freetz_config ()
{
# Fetch config-file with wget in background
# Somehow it doesn't listen to the 6 seconds timeout I'm giving it
wget -q --timeout=6 \
--http-user=${USER} --http-password=${PASS} \
${URL}/cgi-bin/backup/do_backup.cgi \
-O TMP_CONFIG 2>/dev/null &
# Little wait loop to give wget the time to fetch the file
n=1
while sleep 1 ; do
[ `stat -c%s TMP_CONFIG` -ge ${MINSIZE} ] && break
[ $n -gt 7 ] && break
let n+=1
done
if [ `stat -c%s TMP_CONFIG` -gt 300 ] ; then
# If I received at least some data, then wait some more and it might get everything
sleep 2
else
echo "wget didn't get any data from host \"${HOST}\"" >&2
exec 2>/dev/null
kill -9 %1 2>/dev/null
exec 2>&2
return 1
fi
return 0
}
keep_config_if_different ()
{
LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
sort -rn | head -n1 | awk '{print $2}'`
# If no older config is found
if [ -z "${LATESTCONFIG}" ] ; then
echo "Saving \"${DEST_BASE}.tgz\" because no previous Freetz config has been found" >&2
mv TMP_CONFIG "${DEST_BASE}.tgz"
get_avm_config
else
TRIED_FREETZ=1 # A flag for get_avm_config to know if the freetz config has been tried or not
mkdir f1 # folder to untar the current config
mkdir f2 # folder to untar the previous config
if ! tar xzf TMP_CONFIG -C f1 2>/dev/null ; then
echo "I just downloaded an invalid gzipped tar-file" >&2
rm -f TMP_CONFIG
else
if ! tar xzf ${LATESTCONFIG} -C f2 ; then
echo "A previously downloaded config turns out to be invalid... very strange... (${LATESTCONFIG})" >&2
else
# Remove files that are likely to be changed, but don't contain important config
patch_files f1/var_flash
patch_files f2/var_flash
if ! diff f1/var_flash f2/var_flash >/dev/null 2>&1 ; then
echo "Saving \"${DEST_BASE}.tgz\" because Freetz config changed" >&2
mv TMP_CONFIG "${DEST_BASE}.tgz"
# Now get the AVM config too
get_avm_config
elif [ `find "${DEST_FOLDER}" -maxdepth 1 -type f -name \*.export | wc -l` -lt 1 ] ; then
# No previous AVM config exists, so try and fetch one anyhow
get_avm_config
fi
fi
fi
fi
}
get_both_configs ()
{
PROTO=http
echo ${PORT} | grep -q '^4[456][0-9]' && PROTO=https
URL="${PROTO}://${IP}:${PORT}"
if ! httpspeed ${URL} >/dev/null ; then
# Freetz is not reacting, maybe it is a plain jane AVM Fritzbox
# I will not be able to see if a config has changed.......
echo "No reaction from Freetz.... will try AVM now.."
get_avm_config
else
TMPDIR1=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
if ! cd ${TMPDIR1} ; then
echo "Error changing folder to ${TMPDIR1}" >&2
else
touch TMP_CONFIG
get_freetz_config
if [ `stat -c%s TMP_CONFIG` -lt ${MINSIZE} ] ; then
echo "Error getting config from ${HOST} on ${IP}" >&2
else
# I think we're getting successful here, let's create a folder already
chown ${FUSER}:${FGROUP} TMP_CONFIG
mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
keep_config_if_different
fi
fi
cd "${FOLDER}"
rm -rf ${TMPDIR1}
fi
}
# Main Program starts here
if ! cd "${FOLDER}" ; then
echo "${FOLDER} does not exist" >&2
exit 1
fi
BASENAME="${0//*\/}" # Strip Path of script name
BASENAME="${BASENAME%.*}" # Strip extension
CONFIGSOURCE="${FOLDER}/${BASENAME}.list" # Construct modemlist from script name
CONFIGFILE=`mktemp` # temp file for sanitized modemlist
if [ -n "$*" ] ; then
echo "$*" >"${CONFIGFILE}" # push the commandline parameters in the configfile
elif [ -e "${CONFIGSOURCE}" ] ; then
awk -F# '{print $1}' "${CONFIGSOURCE}" | \
sed '/^ *$/d' >"${CONFIGFILE}" # Get all the data before # and remove empty lines
else
echo "Configfile: \"${CONFIGSOURCE}\" does not exist" >&2
exit 1
fi
if [ ! -s "${CONFIGFILE}" ] ; then
echo "No workable data in \"${CONFIGSOURCE}\"" >&2
exit 1
fi
# Main Loop
FGROUP=`stat -c%G .`
FUSER=`stat -c%U .`
while read MODEMLINE ; do
HOST="`echo "${MODEMLINE}" | awk -F, '{print $1}'`"
IP="`echo "${MODEMLINE}" | awk -F, '{print $2}'`"
PASS="`echo "${MODEMLINE}" | awk -F, '{print $3}'`"
PORT="`echo "${MODEMLINE}" | awk -F, '{print $4}' | tr -cd '0-9'`"
USER="`echo "${MODEMLINE}" | awk -F, '{print $5}'`"
PASSE="`echo "${MODEMLINE}" | awk -F, '{print $6}'`"
PORTAVM="`echo "${MODEMLINE}" | awk -F, '{print $7}' | tr -cd '0-9'`"
HOSTORG="${HOST}"
TRIED_FREETZ=
if [ -z "${IP}" ] ; then
HOST=`echo "${HOST}" | tr 'A-Z' 'a-z' | \
egrep -o '[a-z0-9.-]+\.[a-z0-9]+'` # Loosely Sanitize domain if no IP is given
fi
if [ -z "${HOST}" ] ; then
echo "${HOSTORG} is not a valid host" >&2
else
[ ${HEADLESS} ] || echo ${HOST}
if [ -z "${IP}" ] ; then
# If HOST is so clearly an IP-address then use that instead....
if echo "${HOST}" | egrep -q '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$' ; then
IP=${HOST}
else
IP=`host -t A ${HOST} 2>/dev/null | grep -o 'has address .*' | awk '{print $3}' | head -n1`
fi
fi
if [ -z "${IP}" ] ; then
echo "Unable to resolve ${HOST}" >&2
else
[ -z "${PORT}" ] && PORT=${PORT_DEFAULT} # port of Freetz interface
[ -z "${PASS}" ] && PASS=${PASS_DEFAULT} # internal password
[ -z "${USER}" ] && USER=${USER_DEFAULT} # external username
[ -z "${PASSE}" ] && PASSE=${PASS} # external password
[ -z "${PORTAVM}" ] && PORTAVM=${PORTAVM_DEFAULT} # port of AVM interface (https)
DEST_FOLDER="${FOLDER}/${HOST}"
DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}"
if [ ${PORT} -eq 0 ] ; then
get_avm_config # Fetch only AVM-config when the Freetz port is defined as 0
else
get_both_configs
fi
fi
fi
done<"${CONFIGFILE}"
rm -f "${CONFIGFILE}"
Zuletzt bearbeitet:
Great work! I'll try the new version in the next days.
I would like to see your updates here
BTW:
I would like to see a commandline for the list of modems since i'm using .sh on my shellscripts at the end and so my list would be named freetz_getconfigs.sh.list
I would like to see your updates here
BTW:
I would like to see a commandline for the list of modems since i'm using .sh on my shellscripts at the end and so my list would be named freetz_getconfigs.sh.list
@vice_pres...
I did some modifications over the weekend and I'm considering the script stable and full featured now.
I also added some code to get rid of the suffix (in your case .sh), but I just re-read your post and you wanted it in there...
A bit strange as it was using a "freetz_getconfigs.sh.list" in the old version of the script if the name of the script was "freetz_getconfigs.sh"
I now especially added code to get rid of the suffix if one exists and am going to leave it that way as it's much neater...
I tested the script to get the AVM-config from LAN-side. It works!
It needs to use http instead of https and it does this by looking at the port. If it starts with an "8" it will use http instead of https.
I enhanced the method to better determine a changed config. Dynamic data like phonebooks are deliberately being ignored.
I would get too many configs if I didn't (backups should be done automatically).
I'm keeping the oldest config, because you will then also have the info when that config has been changed.
The downside of this approach is that you may not have the latest phonebook data in your config.....
Maybe I will modify the script (by request) if someone would prefer to keep the newest config (and kill the older similar one)
I did some modifications over the weekend and I'm considering the script stable and full featured now.
I also added some code to get rid of the suffix (in your case .sh), but I just re-read your post and you wanted it in there...
A bit strange as it was using a "freetz_getconfigs.sh.list" in the old version of the script if the name of the script was "freetz_getconfigs.sh"
I now especially added code to get rid of the suffix if one exists and am going to leave it that way as it's much neater...
I tested the script to get the AVM-config from LAN-side. It works!
It needs to use http instead of https and it does this by looking at the port. If it starts with an "8" it will use http instead of https.
I enhanced the method to better determine a changed config. Dynamic data like phonebooks are deliberately being ignored.
I would get too many configs if I didn't (backups should be done automatically).
I'm keeping the oldest config, because you will then also have the info when that config has been changed.
The downside of this approach is that you may not have the latest phonebook data in your config.....
Maybe I will modify the script (by request) if someone would prefer to keep the newest config (and kill the older similar one)
Zuletzt bearbeitet:
Just a quick note - after upgrading my Ubuntu to 14.04 the script doesn't work because curl changed it behaviour. I had to add --cipher RC4-SHA to every curl call to get the script working again.
Thanks vice_pres for this feedback!
I have modified the code based on your findings.
I do make a difference between https / http connections (in case the program is run from the inside of the Fritzbox).
No idea what curl does when I supply these options on a plain http, but it is always better to keep code clean and supply parameters as they are expected... (garbage in => garbage out).
Can you confirm this code is working at your end?
~# cat /usr/local/sbin/freetz_getconfigs
I have modified the code based on your findings.
I do make a difference between https / http connections (in case the program is run from the inside of the Fritzbox).
No idea what curl does when I supply these options on a plain http, but it is always better to keep code clean and supply parameters as they are expected... (garbage in => garbage out).
Can you confirm this code is working at your end?
~# cat /usr/local/sbin/freetz_getconfigs
Code:
#!/bin/sh
# Define constants
DATESTAMP=`date +%Y-%m-%d.%H-%M`
FOLDER=/var/www/vhosts/freetz.yourdomain.com/httpdocs
USER_DEFAULT=admin
PASS_DEFAULT="defaultpass"
PORT_DEFAULT=61080
PORTAVM_DEFAULT=450
MINSIZE=20000
HTTP_OPTIONS=
HTTPS_OPTIONS='--cipher RC4-SHA'
HEADLESS=
tty >/dev/null || HEADLESS=true
while getopts cd: opt
do
case $opt in
c) FOLDER="`pwd`" ;;
d) FOLDER="$OPTARG";;
?) printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM]\n" $0
exit 2;;
esac
done
shift `expr ${OPTIND} - 1`
# Subroutines
age_of_file ()
{
if [ ! -e "$1" ] ; then
echo 0
else
AGE_IN_SECONDS=$((`date +%s` - `date +%s -r "$1"` ))
echo $((${AGE_IN_SECONDS} / 86400))
fi
}
httpspeed ()
{
# Returns no value and errorlevel 1 if site is unreachable or too slow
# Returns the speed in milliseconds if it's below 2 seconds
_SPEED=`httping -c1 -t1 ${1} 2>/dev/null | \
egrep '.+/.+/.+/.+/.+' | tail -n1 | \
awk -F/ '{print $4}' | awk -F. '{print $1}'`
if [ -z "${_SPEED}" ] ; then
echo -e "${URL} is unreachable\n" >&2
elif [ ${_SPEED} -gt 2000 ] ; then
echo "${URL} is slow (${_SPEED} ms)" >&2
else
echo ${_SPEED}
return 0
fi
return 1
}
get_avm_config ()
{
[ ${PORTAVM} -eq 0 ] && return
TMPDIR2=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
if ! cd ${TMPDIR2} 2>/dev/null ; then
echo "Error changing current directory to ${TMPDIR2}. You should never see this message!" >&2
else
# It's intended to be called from WAN (https)
# If PORTAVM begins with 8 it assumes LAN and turns to http
PROTO=https
echo ${PORTAVM} | grep -q '^80$' && PROTO=http
OPTIONS="${HTTP_OPTIONS}"
[ "${PROTO}" = 'https' ] && OPTIONS="${HTTPS_OPTIONS}"
URL="${PROTO}://${IP}:${PORTAVM}"
if httpspeed ${URL} >/dev/null ; then
# get challenge key from FB
_CHALLENGE=`curl -s -k ${OPTIONS} \
--user ${USER}:${PASSE} \
"${URL}/login.lua" | \
grep 'challenge' | egrep '[0-9a-f]{8}' | \
awk -F= '{print $2}' | tail -n1 | egrep -o '[0-9a-f]{8}'`
if [ -z "${_CHALLENGE}" ] ; then
echo "Error getting challenge for ${HOST} using ${URL}" >&2
else
# echo "Received Challenge \"${_CHALLENGE}\"" >&2
# build md5 from challenge key and password
_MD5=`echo -n ${_CHALLENGE}"-"${PASSE} | \
iconv -f ISO8859-1 -t UTF-16LE | \
md5sum -b | awk '{print substr($0,1,32)}'`
# assemble challenge key and md5
_RESPONSE=${_CHALLENGE}"-"${_MD5}
# get sid for later use
_SID=`curl -i -s -k ${OPTIONS} \
--user ${USER}:${PASSE} \
-d 'response='${_RESPONSE} \
-d 'page=' \
-d 'username='${USER} \
${URL}/login.lua | \
grep "Location:" | awk -F'=' {' print $NF '}`
if ! curl -s -k ${OPTIONS} \
--user ${USER}:${PASSE} \
--form 'sid='${_SID} \
--form 'ImportExportPassword='${PASSE} \
--form 'ConfigExport=' \
${URL}/cgi-bin/firmwarecfg >TMP1 ; then
echo "Error getting .export for ${HOST} (no data)" >&2
else
if [ `stat -c%s TMP1` -lt ${MINSIZE} ] ; then
echo "Error getting .export for ${HOST} using ${URL} (too small)" >&2
echo "SID: \"${_SID}\" CHALLENGE: \"${_CHALLENGE}\" MD5: \"${_MD5}\" RESPONSE: \"${_RESPONSE}\"" >&2
else
chown ${FUSER}:${FGROUP} TMP1
mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
grep "${IP}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
sort -rn | head -n1 | awk '{print $2}'`
if [ -z "${LATESTCONFIG}" ] ; then # No previous .export is found
echo "Saving \"${DEST_BASE}.export\" because no previous AVM-config has been found" >&2
mv TMP1 "${DEST_BASE}.export"
else
if [ ${TRIED_FREETZ} ] ; then # The Freetz comparison is better, so make that leading
if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
echo -e "Saving \"${DEST_BASE}.export\" because Freetz config has changed\n" >&2
mv TMP1 "${DEST_BASE}.export"
fi
else
if [ `age_of_file "${LATESTCONFIG}"` -gt 30 ] ; then # Compare EXACTLY if age is older than 30 days
if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is not EXACTLY the same" >&2
mv TMP1 "${DEST_BASE}.export"
fi
else
# remove BINARY data and timestamps before making a diff
sed '/ BINFILE:/,/ END OF FILE /d' "${LATESTCONFIG}" >TMP2
sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' TMP2
sed -i '/END OF EXPORT/d;/update_found/d' TMP2
sed -i '/END OF EXPORT/d;/running_version/d' TMP2
sed '/ BINFILE:/,/ END OF FILE /d' TMP1 >TMP3
sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' TMP3
sed -i '/END OF EXPORT/d;/update_found/d' TMP3
sed -i '/END OF EXPORT/d;/running_version/d' TMP3
if ! diff TMP2 TMP3 >/dev/null 2>&1 ; then # Compare the 2 stripped configs
echo -e "Saving \"${DEST_BASE}.export\" because AVM-config has changed\n" >&2
diff TMP2 TMP3 >&2
echo -e '\n---\n\n' >&2
mv TMP1 "${DEST_BASE}.export"
fi
fi
fi
fi
fi
fi
fi
fi
fi
cd "${FOLDER}"
rm -rf ${TMPDIR2}
}
patch_files ()
{
# remove or patch some of the untarred files as they spoil the check for a changed config
if cd "$1" >/dev/null 2>&1 ; then
rm -f stat.cfg chrony.drift multid.leases 2>/dev/null
rm -f fx_cg fonctrl phonebook voipd_call_stat 2>/dev/null
sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' voip.cfg # remove datestamp in voip.cfg
sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' ar7.cfg # remove datestamp in ar7.cfg
sed -i 's/ *\/\*AVM\*\///g' ar7.cfg # remove the /*AVM*/ tag in ar7.cfg
sed -i 's/; *$/;/g' ar7.cfg # remove trailing spaces after a semicolon
cd - >/dev/null 2>&1
fi
}
get_freetz_config ()
{
# Fetch config-file with wget in background
# Somehow it doesn't listen to the 6 seconds timeout I'm giving it
wget -q --timeout=6 \
--http-user=${USER} --http-password=${PASS} \
${URL}/cgi-bin/backup/do_backup.cgi \
-O TMP_CONFIG 2>/dev/null &
# Little wait loop to give wget the time to fetch the file
n=1
while sleep 1 ; do
[ `stat -c%s TMP_CONFIG` -ge ${MINSIZE} ] && break
[ $n -gt 7 ] && break
let n+=1
done
if [ `stat -c%s TMP_CONFIG` -gt 300 ] ; then
# If I received at least some data, then wait some more and it might get everything
sleep 2
else
echo "wget didn't get any data from host \"${HOST}\"" >&2
exec 2>/dev/null
kill -9 %1 2>/dev/null
exec 2>&2
return 1
fi
return 0
}
keep_config_if_different ()
{
LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
sort -rn | head -n1 | awk '{print $2}'`
# If no older config is found
if [ -z "${LATESTCONFIG}" ] ; then
echo "Saving \"${DEST_BASE}.tgz\" because no previous Freetz config has been found" >&2
mv TMP_CONFIG "${DEST_BASE}.tgz"
get_avm_config
else
TRIED_FREETZ=1 # A flag for get_avm_config to know if the freetz config has been tried or not
mkdir f1 # folder to untar the current config
mkdir f2 # folder to untar the previous config
if ! tar xzf TMP_CONFIG -C f1 2>/dev/null ; then
echo "I just downloaded an invalid gzipped tar-file" >&2
rm -f TMP_CONFIG
else
if ! tar xzf ${LATESTCONFIG} -C f2 ; then
echo "A previously downloaded config turns out to be invalid... very strange... (${LATESTCONFIG})" >&2
else
# Remove files that are likely to be changed, but don't contain important config
patch_files f1/var_flash
patch_files f2/var_flash
if ! diff f1/var_flash f2/var_flash >/dev/null 2>&1 ; then
echo -e "Saving \"${DEST_BASE}.tgz\" because Freetz config changed\n" >&2
mv TMP_CONFIG "${DEST_BASE}.tgz"
# Now get the AVM config too
get_avm_config
elif [ `find "${DEST_FOLDER}" -maxdepth 1 -type f -name \*.export | wc -l` -lt 1 ] ; then
# No previous AVM config exists, so try and fetch one anyhow
get_avm_config
fi
fi
fi
fi
}
get_both_configs ()
{
PROTO=http
echo ${PORT} | grep -q '^4[456][0-9]' && PROTO=https
URL="${PROTO}://${IP}:${PORT}"
if ! httpspeed ${URL} >/dev/null ; then
# Freetz is not reacting, maybe it is a plain jane AVM Fritzbox
# I will not be able to see if a config has changed.......
echo "No reaction from Freetz (\"${HOST}\").... will try AVM now.."
get_avm_config
else
TMPDIR1=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
if ! cd ${TMPDIR1} ; then
echo "Error changing folder to ${TMPDIR1}" >&2
else
touch TMP_CONFIG
get_freetz_config
if [ `stat -c%s TMP_CONFIG` -lt ${MINSIZE} ] ; then
echo "Error getting config from ${HOST} on ${IP}" >&2
else
# I think we're getting successful here, let's create a folder already
chown ${FUSER}:${FGROUP} TMP_CONFIG
mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
keep_config_if_different
fi
fi
cd "${FOLDER}"
rm -rf ${TMPDIR1}
fi
}
# Main Program starts here
if ! cd "${FOLDER}" ; then
echo "${FOLDER} does not exist" >&2
exit 1
fi
BASENAME="${0//*\/}" # Strip Path of script name
BASENAME="${BASENAME%.*}" # Strip extension
CONFIGSOURCE="${FOLDER}/${BASENAME}.list" # Construct modemlist from script name
CONFIGFILE=`mktemp` # temp file for sanitized modemlist
if [ -n "$*" ] ; then
echo "$*" >"${CONFIGFILE}" # push the commandline parameters in the configfile
elif [ -e "${CONFIGSOURCE}" ] ; then
sed '/^ *$/d;/^ *#/d' "${CONFIGSOURCE}" | \
awk -F'[\t ]#' '{print $1}' >"${CONFIGFILE}" # Get all the data before # and remove empty lines
else
echo "Configfile: \"${CONFIGSOURCE}\" does not exist" >&2
exit 1
fi
if [ ! -s "${CONFIGFILE}" ] ; then
echo "No workable data in \"${CONFIGSOURCE}\"" >&2
exit 1
fi
# Main Loop
FGROUP=`stat -c%G .`
FUSER=`stat -c%U .`
while read MODEMLINE ; do
HOST="`echo "${MODEMLINE}" | awk -F, '{print $1}' | awk '{print $1}'`"
IP="`echo "${MODEMLINE}" | awk -F, '{print $2}' | awk '{print $1}'`"
PASS="`echo "${MODEMLINE}" | awk -F, '{print $3}' | awk '{print $1}'`"
PORT="`echo "${MODEMLINE}" | awk -F, '{print $4}' | tr -cd '0-9'`"
USER="`echo "${MODEMLINE}" | awk -F, '{print $5}' | awk '{print $1}'`"
PASSE="`echo "${MODEMLINE}" | awk -F, '{print $6}' | awk '{print $1}'`"
PORTAVM="`echo "${MODEMLINE}" | awk -F, '{print $7}' | tr -cd '0-9'`"
HOSTORG="${HOST}"
TRIED_FREETZ=
if [ -z "${IP}" ] ; then
HOST=`echo "${HOST}" | tr 'A-Z' 'a-z' | \
egrep -o '[a-z0-9.-]+\.[a-z0-9]+'` # Loosely Sanitize domain if no IP is given
fi
if [ -z "${HOST}" ] ; then
echo "${HOSTORG} is not a valid host" >&2
else
[ ${HEADLESS} ] || echo ${HOST}
if [ -z "${IP}" ] ; then
# If HOST is so clearly an IP-address then use that instead....
if echo "${HOST}" | egrep -q '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$' ; then
IP=${HOST}
else
IP=`host -t A ${HOST} 2>/dev/null | grep -o 'has address .*' | awk '{print $3}' | head -n1`
fi
fi
if [ -z "${IP}" ] ; then
echo "Unable to resolve ${HOST}" >&2
else
[ -z "${PORT}" ] && PORT=${PORT_DEFAULT} # port of Freetz interface
[ -z "${PASS}" ] && PASS=${PASS_DEFAULT} # internal password
[ -z "${USER}" ] && USER=${USER_DEFAULT} # external username
[ -z "${PASSE}" ] && PASSE=${PASS} # external password
[ -z "${PORTAVM}" ] && PORTAVM=${PORTAVM_DEFAULT} # port of AVM interface (https)
DEST_FOLDER="${FOLDER}/${HOST}"
DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}"
if [ ${PORT} -eq 0 ] ; then
get_avm_config # Fetch only AVM-config when the Freetz port is defined as 0
else
get_both_configs
fi
fi
fi
done<"${CONFIGFILE}"
rm -f "${CONFIGFILE}"
Zuletzt bearbeitet:
Hi,
Code is working, I'm using a version I modified a bit (for example storing hostnames in the filename instead of ip because they are dynamic) etc - but tried your code in a new file and it's working
Edit:
I changed something to the sed when comparing the configs since the passwords seem to be rehashed everytime something is changed via webinterface - so there would always be a diff...
sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d'
Code is working, I'm using a version I modified a bit (for example storing hostnames in the filename instead of ip because they are dynamic) etc - but tried your code in a new file and it's working
Edit:
I changed something to the sed when comparing the configs since the passwords seem to be rehashed everytime something is changed via webinterface - so there would always be a diff...
sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d'
Zuletzt bearbeitet:
I will look into that sed later. I did notice some configs get an update when in fact nothing really changed.
I don't understand your remark about the dynamic host.
My script fully supports dynamic hosts. Just leave out the IP.
I don't understand your remark about the dynamic host.
My script fully supports dynamic hosts. Just leave out the IP.
Yeah, but the script stores the files with the ip-adress - and I find it more usefull if the script stores the hostname instead.
Because you set DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}" and I modified it to be DEST_BASE="${DEST_FOLDER}/${HOST}.${DATESTAMP}"
Code:
test.dyndns.org
Saving "/new/test.dyndns.org/83.20.222.212.2014-04-27.16-06.export" because no previous AVM-config has been foundBecause you set DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}" and I modified it to be DEST_BASE="${DEST_FOLDER}/${HOST}.${DATESTAMP}"
Hi,
Since 06.30 the '--cipher RC4-SHA' option is not needed anymore - in fact it stops the backup from working.
Since 06.30 the '--cipher RC4-SHA' option is not needed anymore - in fact it stops the backup from working.
I will check my script and publish the new one...
I will also check why I have a "new config" each day. Something must have been added that changes every day.
I need to add something more to the ignore part..
Will post something soon
I will also check why I have a "new config" each day. Something must have been added that changes every day.
I need to add something more to the ignore part..
Will post something soon
I modified the Script to have the HTTPS_Option as an extra option in the config file - since I have some Boxes that don't get 06.30 anymore (7270 for example) and need the cipher. But since I started on an older version of your script a diff is rather long - plus the modifications I wanted (storing in folder named as the hostname etc...). This version works fine for all boxes I'm backing up - the 06.30 Boxes get backed up almost daily (but not daily - some days there are no diffs. I assume it's the call log that changed and triggers an backup)
Code:
1c1
< #!/bin/bash
---
> #!/bin/sh
5c5
< FOLDER=/fritzbox/backup
---
> FOLDER=/var/www/vhosts/freetz.yourdomain.com/httpdocs
8,9c8,9
< PASS_DEFAULT=topsecret
< PORT_DEFAULT=6090
---
> PASS_DEFAULT="defaultpass"
> PORT_DEFAULT=61080
11a12,15
>
> HTTP_OPTIONS=
> HTTPS_OPTIONS='--cipher RC4-SHA'
>
20c24
< ?) printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM],[CURL_OPTS]\n" $0
---
> ?) printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM]\n" $0
48c52
< echo "${URL} is unreachable" >&2
---
> echo -e "${URL} is unreachable\n" >&2
71c75,79
< echo ${PORTAVM} | grep -q '^8' && PROTO=http
---
> echo ${PORTAVM} | grep -q '^80$' && PROTO=http
>
> OPTIONS="${HTTP_OPTIONS}"
> [ "${PROTO}" = 'https' ] && OPTIONS="${HTTPS_OPTIONS}"
>
74c82
< #if httpspeed ${URL} >/dev/null ; then
---
> if httpspeed ${URL} >/dev/null ; then
77c85
< _CHALLENGE=`curl ${CURLOPTS} -s -k \
---
> _CHALLENGE=`curl -s -k ${OPTIONS} \
96c104
< _SID=`curl ${CURLOPTS} -i -s -k \
---
> _SID=`curl -i -s -k ${OPTIONS} \
104c112
< if ! curl ${CURLOPTS} -s -k \
---
> if ! curl -s -k ${OPTIONS} \
118c126
< mkdir -p "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
---
> mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
120c128
< grep "${HOST}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
---
> grep "${IP}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
129c137
< echo "Saving \"${DEST_BASE}.export\" because Freetz config has changed" >&2
---
> echo -e "Saving \"${DEST_BASE}.export\" because Freetz config has changed\n" >&2
135c143
< echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is different" >&2
---
> echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is not EXACTLY the same" >&2
139,144c147,158
< # remove BINARY data, timestamps and passwords (since OS6.0 passwords are rehashed at every configchange through webif it seems) before making a diff
< #sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d' "${LATESTCONFIG}" >TMP2
< #sed '/ BINFILE:/,/ END OF FILE /d;/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d' TMP1 >TMP3
< # just remove timestamps and passwords (since OS6.0 passwords are rehashed at every configchange through webif it seems)
< sed '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d' "${LATESTCONFIG}" >TMP2
< sed '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d;/END OF EXPORT/d;/\$\$\$\$/d' TMP1 >TMP3
---
>
> # remove BINARY data and timestamps before making a diff
> sed '/ BINFILE:/,/ END OF FILE /d' "${LATESTCONFIG}" >TMP2
> sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' TMP2
> sed -i '/END OF EXPORT/d;/update_found/d' TMP2
> sed -i '/END OF EXPORT/d;/running_version/d' TMP2
>
> sed '/ BINFILE:/,/ END OF FILE /d' TMP1 >TMP3
> sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' TMP3
> sed -i '/END OF EXPORT/d;/update_found/d' TMP3
> sed -i '/END OF EXPORT/d;/running_version/d' TMP3
>
146,147c160,162
< echo "Saving \"${DEST_BASE}.export\" because AVM-config has changed" >&2
< #diff TMP2 TMP3
---
> echo -e "Saving \"${DEST_BASE}.export\" because AVM-config has changed\n" >&2
> diff TMP2 TMP3 >&2
> echo -e '\n---\n\n' >&2
156c171
< #fi
---
> fi
209c224
< grep "${HOST}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
---
> grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
220,221c235,236
< mkdir -p f1 # folder to untar the current config
< mkdir -p f2 # folder to untar the previous config
---
> mkdir f1 # folder to untar the current config
> mkdir f2 # folder to untar the previous config
234c249
< echo "Saving \"${DEST_BASE}.tgz\" because Freetz config changed" >&2
---
> echo -e "Saving \"${DEST_BASE}.tgz\" because Freetz config changed\n" >&2
256c271
< echo "No reaction from Freetz.... will try AVM now.."
---
> echo "No reaction from Freetz (\"${HOST}\").... will try AVM now.."
274c289
< mkdir -p "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
---
> mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
300,301c315,316
< awk -F# '{print $1}' "${CONFIGSOURCE}" | \
< sed '/^ *$/d' >"${CONFIGFILE}" # Get all the data before # and remove empty lines
---
> sed '/^ *$/d;/^ *#/d' "${CONFIGSOURCE}" | \
> awk -F'[\t ]#' '{print $1}' >"${CONFIGFILE}" # Get all the data before # and remove empty lines
317,319c332,334
< HOST="`echo "${MODEMLINE}" | awk -F, '{print $1}'`"
< IP="`echo "${MODEMLINE}" | awk -F, '{print $2}'`"
< PASS="`echo "${MODEMLINE}" | awk -F, '{print $3}'`"
---
> HOST="`echo "${MODEMLINE}" | awk -F, '{print $1}' | awk '{print $1}'`"
> IP="`echo "${MODEMLINE}" | awk -F, '{print $2}' | awk '{print $1}'`"
> PASS="`echo "${MODEMLINE}" | awk -F, '{print $3}' | awk '{print $1}'`"
321,322c336,337
< USER="`echo "${MODEMLINE}" | awk -F, '{print $5}'`"
< PASSE="`echo "${MODEMLINE}" | awk -F, '{print $6}'`"
---
> USER="`echo "${MODEMLINE}" | awk -F, '{print $5}' | awk '{print $1}'`"
> PASSE="`echo "${MODEMLINE}" | awk -F, '{print $6}' | awk '{print $1}'`"
324d338
< CURLOPTS="`echo "${MODEMLINE}" | awk -F, '{print $8}'`"
355c369
< DEST_BASE="${DEST_FOLDER}/${HOST}.${DATESTAMP}"
---
> DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}"My script is supposed to ignore the call logs as far as I can remember... If I have time I will take a look.
I haven't noticed that I wasn't getting any configs of some routers...
Didn't find the time to check. Recently bought my first 3d printer
I haven't noticed that I wasn't getting any configs of some routers...
Didn't find the time to check. Recently bought my first 3d printer
Just as a warning: The script can't backup 06.50 at the moment as it seems - and reboots the target (when running 06.50) after trying to get a backup. I'll try if I can take a look why this is happening on monday using my lab...
This is working from 06.06 (7270 V2) to 7490 (06.50) on all verisons I can test:
Since there is no login.lua anymore I changed it to login_sid.lua which is available on all 06 Versions I can test. But my testbox doesn't reboot like my production box when testing...
This is working from 06.06 (7270 V2) to 7490 (06.50) on all verisons I can test:
Code:
_CHALLENGE=`curl ${CURLOPTS} -s -k \
--user ${USER}:${PASSE} \
"${URL}/login_sid.lua" | \
grep Challenge | sed -e 's/.*<Challenge>//' | sed -e 's/<\/Challenge>.*//'`
Code:
_SID=`curl ${CURLOPTS} -i -s -k \
--user ${USER}:${PASSE} \
-d 'response='${_RESPONSE} \
-d 'page=' \
-d 'username='${USER} \
${URL}/login_sid.lua | \
grep SID | sed -e 's/.*<SID>//' | sed -e 's/<\/SID>.*//'`Since there is no login.lua anymore I changed it to login_sid.lua which is available on all 06 Versions I can test. But my testbox doesn't reboot like my production box when testing...
Zuletzt bearbeitet:
I am very very late implementing your fixes due to all kind of other stuff...
Here's the complete script again implementing your fixes.....
I don't know if it still can download the export of old fritzboxes....
The box on which I was testing it now works.....
# cat /usr/local/sbin/freetz_getconfigs
Here's the complete script again implementing your fixes.....
I don't know if it still can download the export of old fritzboxes....
The box on which I was testing it now works.....
# cat /usr/local/sbin/freetz_getconfigs
Code:
#!/bin/sh
# Define constants
DATESTAMP=`date +%Y-%m-%d.%H-%M`
FOLDER=/var/www/vhosts/freetz.mr-wolf.nl/httpdocs/courant
USER_DEFAULT=admin
PASS_DEFAULT=q1w2e3r4
PORT_DEFAULT=61080
PORTAVM_DEFAULT=450
MINSIZE=20000
HTTP_OPTIONS=''
HTTPS_OPTIONS=''
# HTTPS_OPTIONS='--cipher RC4-SHA'
HEADLESS=
tty >/dev/null || HEADLESS=true
while getopts cd: opt
do
case $opt in
c) FOLDER="`pwd`" ;;
d) FOLDER="$OPTARG";;
?) printf "Usage: %s [-c] [-d <folder>] [<HOST>],[<IP>],[PASS_FREETZ],[<PORT_FREETZ>],[<USER>],[PASS_EXTERNAL],[PORT_AVM]\n" $0
exit 2;;
esac
done
shift `expr ${OPTIND} - 1`
# Subroutines
age_of_file ()
{
if [ ! -e "$1" ] ; then
echo 0
else
AGE_IN_SECONDS=$((`date +%s` - `date +%s -r "$1"` ))
echo $((${AGE_IN_SECONDS} / 86400))
fi
}
httpspeed ()
{
# Returns no value and errorlevel 1 if site is unreachable or too slow
# Returns the speed in milliseconds if it's below 2 seconds
_SPEED=`httping -c1 -t1 ${1} 2>/dev/null | \
egrep '.+/.+/.+/.+/.+' | tail -n1 | \
awk -F/ '{print $4}' | awk -F. '{print $1}'`
if [ -z "${_SPEED}" ] ; then
echo -e "${URL} is unreachable\n" >&2
elif [ ${_SPEED} -gt 2000 ] ; then
echo "${URL} is slow (${_SPEED} ms)" >&2
else
echo ${_SPEED}
return 0
fi
return 1
}
get_avm_config ()
{
[ ${PORTAVM} -eq 0 ] && return
TMPDIR2=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
if ! cd ${TMPDIR2} 2>/dev/null ; then
echo "Error changing current directory to ${TMPDIR2}. You should never see this message!" >&2
else
# It's intended to be called from WAN (https)
# If PORTAVM begins with 8 it assumes LAN and turns to http
PROTO=https
echo ${PORTAVM} | grep -q '^80$' && PROTO=http
CURLOPTS="${HTTP_OPTIONS}"
[ "${PROTO}" = 'https' ] && CURLOPTS="${HTTPS_OPTIONS}"
URL="${PROTO}://${IP}:${PORTAVM}"
if httpspeed ${URL} >/dev/null ; then
# get challenge key from FB
_CHALLENGE=`curl ${CURLOPTS} -s -k \
--user ${USER}:${PASSE} \
"${URL}/login_sid.lua" | \
grep 'Challenge' | sed -e 's/.*<Challenge>//' | sed -e 's/<\/Challenge>.*//'`
if [ -z "${_CHALLENGE}" ] ; then
echo "Error getting challenge for ${HOST} using ${URL}/login_sid.lua" >&2
else
# echo "Received Challenge \"${_CHALLENGE}\"" >&2
# build md5 from challenge key and password
_MD5=`echo -n ${_CHALLENGE}"-"${PASSE} | \
iconv -f ISO8859-1 -t UTF-16LE | \
md5sum -b | awk '{print substr($0,1,32)}'`
# assemble challenge key and md5
_RESPONSE=${_CHALLENGE}"-"${_MD5}
# get sid for later use
_SID=`curl ${CURLOPTS} -i -s -k \
--user ${USER}:${PASSE} \
-d 'response='${_RESPONSE} \
-d 'page=' \
-d 'username='${USER} \
${URL}/login_sid.lua | \
grep SID | sed -e 's/.*<SID>//' | sed -e 's/<\/SID>.*//'`
if ! curl ${CURLOPTS} -s -k \
--user ${USER}:${PASSE} \
--form 'sid='${_SID} \
--form 'ImportExportPassword='${PASSE} \
--form 'ConfigExport=' \
${URL}/cgi-bin/firmwarecfg >TMP1 ; then
echo "Error getting .export for ${HOST} (no data)" >&2
else
if [ `stat -c%s TMP1` -lt ${MINSIZE} ] ; then
echo "Error getting .export for ${HOST} using ${URL} (too small)" >&2
echo "SID: \"${_SID}\" CHALLENGE: \"${_CHALLENGE}\" MD5: \"${_MD5}\" RESPONSE: \"${_RESPONSE}\"" >&2
else
chown ${FUSER}:${FGROUP} TMP1
mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
grep "${IP}.*\.export" | xargs -I{} stat -c '%Y %n' {} | \
sort -rn | head -n1 | awk '{print $2}'`
if [ -z "${LATESTCONFIG}" ] ; then # No previous .export is found
echo "Saving \"${DEST_BASE}.export\" because no previous AVM-config has been found" >&2
mv TMP1 "${DEST_BASE}.export"
else
if [ ${TRIED_FREETZ} ] ; then # The Freetz comparison is better, so make that leading
if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
echo -e "Saving \"${DEST_BASE}.export\" because Freetz config has changed\n" >&2
mv TMP1 "${DEST_BASE}.export"
fi
else
if [ `age_of_file "${LATESTCONFIG}"` -gt 30 ] ; then # Compare EXACTLY if age is older than 30 days
if ! diff TMP1 "${LATESTCONFIG}" >/dev/null 2>&1 ; then
echo "Saving \"${DEST_BASE}.export\" because latest AVM-config was older than 30 days and is not EXACTLY the same" >&2
mv TMP1 "${DEST_BASE}.export"
fi
else
# remove BINARY data and timestamps before making a diff
sed '/ BINFILE:/,/ END OF FILE /d' "${LATESTCONFIG}" >TMP2
sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' TMP2
sed -i '/END OF EXPORT/d;/update_found/d' TMP2
sed -i '/\$\$\$\$/d;/running_version/d' TMP2
sed '/ BINFILE:/,/ END OF FILE /d' TMP1 >TMP3
sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' TMP3
sed -i '/END OF EXPORT/d;/update_found/d' TMP3
sed -i '/\$\$\$\$/d;/running_version/d' TMP3
if ! diff TMP2 TMP3 >/dev/null 2>&1 ; then # Compare the 2 stripped configs
echo -e "Saving \"${DEST_BASE}.export\" because AVM-config has changed\n" >&2
diff TMP2 TMP3 >&2
echo -e '\n---\n\n' >&2
mv TMP1 "${DEST_BASE}.export"
fi
fi
fi
fi
fi
fi
fi
fi
fi
cd "${FOLDER}"
rm -rf ${TMPDIR2}
}
patch_files ()
{
# remove or patch some of the untarred files as they spoil the check for a changed config
if cd "$1" >/dev/null 2>&1 ; then
rm -f userstat.cfg stat.cfg chrony.drift multid.leases 2>/dev/null
rm -f fx_cg fonctrl phonebook voipd_call_stat 2>/dev/null
sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' voip.cfg # remove datestamp in voip.cfg
sed -i '/ [0-2][0-9]:[0-5][0-9]:[0-9][0-9]/d' ar7.cfg # remove datestamp in ar7.cfg
sed -i 's/ *\/\*AVM\*\///g' ar7.cfg # remove the /*AVM*/ tag in ar7.cfg
sed -i 's/; *$/;/g' ar7.cfg # remove trailing spaces after a semicolon
cd - >/dev/null 2>&1
fi
}
get_freetz_config ()
{
# Fetch config-file with wget in background
# Somehow it doesn't listen to the 6 seconds timeout I'm giving it
wget -q --timeout=6 \
--http-user=${USER} --http-password=${PASS} \
${URL}/cgi-bin/backup/do_backup.cgi \
-O TMP_CONFIG 2>/dev/null &
# Little wait loop to give wget the time to fetch the file
n=1
while sleep 1 ; do
[ `stat -c%s TMP_CONFIG` -ge ${MINSIZE} ] && break
[ $n -gt 7 ] && break
let n+=1
done
if [ `stat -c%s TMP_CONFIG` -gt 300 ] ; then
# If I received at least some data, then wait some more and it might get everything
sleep 2
else
echo "wget didn't get any data from host \"${HOST}\"" >&2
exec 2>/dev/null
kill -9 %1 2>/dev/null
exec 2>&2
return 1
fi
return 0
}
keep_config_if_different ()
{
LATESTCONFIG=`find "${DEST_FOLDER}" -maxdepth 1 -type f -size +10 | \
grep "${IP}.*\.tgz" | xargs -I{} stat -c '%Y %n' {} | \
sort -rn | head -n1 | awk '{print $2}'`
# If no older config is found
if [ -z "${LATESTCONFIG}" ] ; then
echo "Saving \"${DEST_BASE}.tgz\" because no previous Freetz config has been found" >&2
mv TMP_CONFIG "${DEST_BASE}.tgz"
get_avm_config
else
TRIED_FREETZ=1 # A flag for get_avm_config to know if the freetz config has been tried or not
mkdir f1 # folder to untar the current config
mkdir f2 # folder to untar the previous config
if ! tar xzf TMP_CONFIG -C f1 2>/dev/null ; then
echo "I just downloaded an invalid gzipped tar-file" >&2
rm -f TMP_CONFIG
else
if ! tar xzf ${LATESTCONFIG} -C f2 ; then
echo "A previously downloaded config turns out to be invalid... very strange... (${LATESTCONFIG})" >&2
else
# Remove files that are likely to be changed, but don't contain important config
patch_files f1/var_flash
patch_files f2/var_flash
if ! diff f1/var_flash f2/var_flash >/dev/null 2>&1 ; then
[ ${HEADLESS} ] || diff f1/var_flash f2/var_flash >&2
echo -e "Saving \"${DEST_BASE}.tgz\" because Freetz config changed\n" >&2
mv TMP_CONFIG "${DEST_BASE}.tgz"
# Now get the AVM config too
get_avm_config
elif [ `find "${DEST_FOLDER}" -maxdepth 1 -type f -name \*.export | wc -l` -lt 1 ] ; then
# No previous AVM config exists, so try and fetch one anyhow
get_avm_config
fi
fi
fi
fi
}
get_both_configs ()
{
PROTO=http
echo ${PORT} | grep -q '^4[456][0-9]' && PROTO=https
URL="${PROTO}://${IP}:${PORT}"
if ! httpspeed ${URL} >/dev/null ; then
# Freetz is not reacting, maybe it is a plain jane AVM Fritzbox
# I will not be able to see if a config has changed.......
echo "No reaction from Freetz (\"${HOST}\").... will try AVM now.."
get_avm_config
else
TMPDIR1=`mktemp -p /dev/shm -d ${0//*\/}.XXXXXXXXXX`
if ! cd ${TMPDIR1} ; then
echo "Error changing folder to ${TMPDIR1}" >&2
else
touch TMP_CONFIG
get_freetz_config
if [ `stat -c%s TMP_CONFIG` -lt ${MINSIZE} ] ; then
echo "Error getting config from ${HOST} on ${IP}" >&2
else
# I think we're getting successful here, let's create a folder already
chown ${FUSER}:${FGROUP} TMP_CONFIG
mkdir "${DEST_FOLDER}" 2>/dev/null && chown ${FUSER}:${FGROUP} "${DEST_FOLDER}"
keep_config_if_different
fi
fi
cd "${FOLDER}"
rm -rf ${TMPDIR1}
fi
}
# Main Program starts here
if ! cd "${FOLDER}" ; then
echo "${FOLDER} does not exist" >&2
exit 1
fi
BASENAME="${0//*\/}" # Strip Path of script name
BASENAME="${BASENAME%.*}" # Strip extension
CONFIGSOURCE="${FOLDER}/${BASENAME}.list" # Construct modemlist from script name
CONFIGFILE=`mktemp` # temp file for sanitized modemlist
if [ -n "$*" ] ; then
echo "$*" >"${CONFIGFILE}" # push the commandline parameters in the configfile
elif [ -e "${CONFIGSOURCE}" ] ; then
sed '/^ *$/d;/^ *#/d' "${CONFIGSOURCE}" | \
awk -F'[\t ]#' '{print $1}' >"${CONFIGFILE}" # Get all the data before # and remove empty lines
else
echo "Configfile: \"${CONFIGSOURCE}\" does not exist" >&2
exit 1
fi
if [ ! -s "${CONFIGFILE}" ] ; then
echo "No workable data in \"${CONFIGSOURCE}\"" >&2
exit 1
fi
# Main Loop
FGROUP=`stat -c%G .`
FUSER=`stat -c%U .`
while read MODEMLINE ; do
HOST="`echo "${MODEMLINE}" | awk -F, '{print $1}' | awk '{print $1}'`"
IP="`echo "${MODEMLINE}" | awk -F, '{print $2}' | awk '{print $1}'`"
PASS="`echo "${MODEMLINE}" | awk -F, '{print $3}' | awk '{print $1}'`"
PORT="`echo "${MODEMLINE}" | awk -F, '{print $4}' | tr -cd '0-9'`"
USER="`echo "${MODEMLINE}" | awk -F, '{print $5}' | awk '{print $1}'`"
PASSE="`echo "${MODEMLINE}" | awk -F, '{print $6}' | awk '{print $1}'`"
PORTAVM="`echo "${MODEMLINE}" | awk -F, '{print $7}' | tr -cd '0-9'`"
HOSTORG="${HOST}"
TRIED_FREETZ=
if [ -z "${IP}" ] ; then
HOST=`echo "${HOST}" | tr 'A-Z' 'a-z' | \
egrep -o '[a-z0-9.-]+\.[a-z0-9]+'` # Loosely Sanitize domain if no IP is given
fi
if [ -z "${HOST}" ] ; then
echo "${HOSTORG} is not a valid host" >&2
else
[ ${HEADLESS} ] || echo ${HOST}
if [ -z "${IP}" ] ; then
# If HOST is so clearly an IP-address then use that instead....
if echo "${HOST}" | egrep -q '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$' ; then
IP=${HOST}
else
IP=`host -t A ${HOST} 2>/dev/null | grep -o 'has address .*' | awk '{print $3}' | head -n1`
fi
fi
if [ -z "${IP}" ] ; then
echo "Unable to resolve ${HOST}" >&2
else
[ -z "${PORT}" ] && PORT=${PORT_DEFAULT} # port of Freetz interface
[ -z "${PASS}" ] && PASS=${PASS_DEFAULT} # internal password
[ -z "${USER}" ] && USER=${USER_DEFAULT} # external username
[ -z "${PASSE}" ] && PASSE=${PASS} # external password
[ -z "${PORTAVM}" ] && PORTAVM=${PORTAVM_DEFAULT} # port of AVM interface (https)
DEST_FOLDER="${FOLDER}/${HOST}"
DEST_BASE="${DEST_FOLDER}/${IP}.${DATESTAMP}"
if [ ${PORT} -eq 0 ] ; then
get_avm_config # Fetch only AVM-config when the Freetz port is defined as 0
else
get_both_configs
fi
fi
fi
done<"${CONFIGFILE}"
rm -f "${CONFIGFILE}"The download of AVM's config does not work with 6.83 anymore...
Anyone has a quick fix?
Somehow the challenge fails....
EDIT...
The root cause must lie somewhere else.
The script works fine on another 7490 with 6.83 (a higher version, but I will test some more)
EDIT
I found the root cause
It's that two factor challenge method that's spoiling it (where one needs to press a button).
I just tried a manual backup and that failed for these same reasons.
# Edit ar7.cfg and change two_factor_auth_enabled
nvi /var/flash/ar7.cfg
two_factor_auth_enabled = no;
# Stop ctlmgr
ctlmgr -s
# Start ctlmgr
ctlmgr
Anyone has a quick fix?
Somehow the challenge fails....
EDIT...
The root cause must lie somewhere else.
The script works fine on another 7490 with 6.83 (a higher version, but I will test some more)
EDIT
I found the root cause
It's that two factor challenge method that's spoiling it (where one needs to press a button).
I just tried a manual backup and that failed for these same reasons.
# Edit ar7.cfg and change two_factor_auth_enabled
nvi /var/flash/ar7.cfg
two_factor_auth_enabled = no;
# Stop ctlmgr
ctlmgr -s
# Start ctlmgr
ctlmgr
Zuletzt bearbeitet:
Neueste Beiträge
-
5690 Pro Datenraten Problem 5ghz und 6ghz
- Letzte: Björn Otto 1993
-
[Frage] Voip und VPN- Letzte: Jstessi
-
[Sammlung] Fritz!Box 5690Pro, Inhaus/Labor 7.90, Pfad 'Smart24P1FCS'
- Letzte: Scary674
-
[Sammlung] FRITZ!Box 4050, 5690 Pro, 6860 5G und 7690
- Letzte: Anfänger0815
-
[Sammlung] FRITZ!Smart Control 440 - Firmware 4.89 - 5.xx
- Letzte: zorro neu
-
Fritz!DECT 302 regelt nicht mehr
- Letzte: Jstessi
-
AVM und die vorgezogene Zwangstrennung
- Letzte: totalverrückteruser