grob run stalls on socat connection refused
it can happen that a grob run stalls because socat gets a connection refused, this is no detected and grob-client hangs forever
example log on server:
2020-08-31 07:01:01: starting the following create run at 2020-08-31.07:01:01:
2020-08-31 07:01:01: create_type=hourly
2020-08-31 07:01:01: custom_host=
2020-08-31 07:01:01: custom_drive=
2020-08-31 07:01:01: archive_stamp=2020-08-31.07
2020-08-31 07:01:02: checking how many hosts/drives to backup
2020-08-31 07:01:03: going to backup 2 hosts and 3 drives
2020-08-31 07:01:03: now processing host ldap.tilab.tuwien.ac.at drive var_lib_ldap (0/3)..
2020-08-31 07:01:03: starting create run for host Git.tilab.tuwien.ac.at (0/2) drive srv (1/3)
2020-08-31 07:01:03: logging into "/srv/grobdata/grobrepo-tilab-hourly/log/hourly/2020-08-31.07/2020-08-31.07:01:01_Git.tilab.tuwien.ac.at_srv.(grob|borg)"
2020-08-31 07:01:35: successfully completed create run for host Git.tilab.tuwien.ac.at (0/2) drive srv (1/3)
2020-08-31 07:01:35: now processing host Git.tilab.tuwien.ac.at drive srv (1/3)..
2020-08-31 07:01:35: starting create run for host Git.tilab.tuwien.ac.at (0/2) drive var_opt_gitlab (2/3)
2020-08-31 07:01:35: logging into "/srv/grobdata/grobrepo-tilab-hourly/log/hourly/2020-08-31.07/2020-08-31.07:01:01_Git.tilab.tuwien.ac.at_var_opt_gitlab.(grob|borg)"
and on client (skipped pre hook log):
[grob@backup 2020-08-31.07]$ tail 2020-08-31.07:01:01_Git.tilab.tuwien.ac.at_var_opt_gitlab.borg
done
done
done
done
Deleting old backups ... skipping
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
Backup task is done.
Remote: 2020/08/31 07:02:50 socat[30606] E connect(5, AF=2 127.0.0.1:40914, 16): Connection refused
[grob@backup 2020-08-31.07]$ cat 2020-08-31.07:01:01_Git.tilab.tuwien.ac.at_var_opt_gitlab.grob
grob: backing up Git.tilab.tuwien.ac.at:var_opt_gitlab drive_path=/var/opt/gitlab borg_repo_path=/srv/grobdata/grobrepo-tilab-hourly/borg_repo remote_user=grobclient ssh_tunnel_mode=tcp
grob: borg_pull: ssh err fifo already exists, removing it
grob-client: info: version 0.1.15.1 starting
grob-client: debug: declaring variable: declare -- client_type="tcp"
grob-client: debug: declaring variable: declare -a borg_args='([0]="create" [1]="--stats" [2]="--one-file-system" [3]="--exclude-caches" [4]="ssh://nop//srv/grobdata/grobrepo-tilab-hourly/borg_repo::hourly_2020-08-31.07_Git.tilab.tuwien.ac.at_var_opt_gitlab" [5]="/var/opt/gitlab")'
grob-client: debug: declaring variable: declare -- BORG_FILES_CACHE_TTL="20"
grob-client: debug: exporting variable: BORG_FILES_CACHE_TTL
grob-client: debug: declaring variable: declare -x GROB_OTP="f79e7bde-4b5e-43b1-a45f-6fcd5cb978ed"
grob-client: debug: exporting variable: GROB_OTP
grob-client: debug: declaring variable: declare -- GROB_TUNNEL_PORT="40914"
grob-client: debug: exporting variable: GROB_TUNNEL_PORT
grob-client: debug: declaring variable: declare -- grob_pre_hook="/usr/bin/sudo /usr/local/sbin/gitlab-backup"
grob-client: info: running pre hook: /usr/bin/sudo /usr/local/sbin/gitlab-backup
grob-client: debug: BORG_RSH=/usr/libexec/grob/grob-client-helper-tcp
grob-client: debug: now launching: borg create --stats --one-file-system --exclude-caches ssh://nop//srv/grobdata/grobrepo-tilab-hourly/borg_repo::hourly_2020-08-31.07_Git.tilab.tuwien.ac.at_var_opt_gitlab /var/opt/gitlab