rsync foo: bar:, with socat(1)

[Ivan Shmakov]

The other day I had to rsync files from host foo to host bar.
The problem is, the hosts are on the same "security tier,"
so I'd rather not to log into either one from another (even
with authorized_keys 'command' and "restricted" rrsync.)

There's, however, a third host, qux, from which I /do/ log into
both. Hence, my idea was to start two SSH sessions from qux to
both foo and bar, running Rsync as a server on one and as a
client on another, connecting them to one another with socat(1).

(Copying files from foo to qux and thence to bar was deemed
to be lacking a challenge.)

Here's how I got it working.

#!/bin/sh
set -C -e
# set -x
# dry=-n
dry=
## NB: see also --include=, --exclude below
from=foo.example.com:foo/
to=bar.example.net:bar/
## .
exec socat \
EXEC:"ssh -- ${from%%:*} \
rsync --server --sender ${dry} -u -rOtH \
--include=\\\\\\\*.text \
--exclude=\\\\\\\* -- ${from#*:}" \
SYSTEM:"ssh -- ${to%%:*} \
RSYNC_RSH=\\\\\\\"\"sh -c \\\\\'socat STDIO FD\:5\\\\\' dummy.sh \"\\\\\\\" \
socat STDIO EXEC\:\\\\\\\"\"\
rsync -v --msgs2stderr ${dry} -ub -rOtH \
--suffix=.~$(date +%s)~ -- NOWHERE\\\\\\\\: ${to#*:}/\\\\\\\"\"\\\
,fdin=5\\,fdout=5"

Some notes. First of all, I've found that at least some of the
'-urOtH' options (and perhaps more which I didn't use this time)
need to be the same for both the client and the server for
proper operation. Normally, as evidenced when run with
-e ssh\ -v, Rsync passes a bunch of them (including those that
don't seem to make any difference) to the remote via SSH.
Here, it becomes a user's responsibility.

While Rsync in --server mode uses stdin, stdout to communicate
with the remote party, the client insists on starting a child
process. So, I use two more socat(1) instances on bar: the
first one to divert stdio to fd 5, while another one, specified
as RSYNC_RSH and started by Rsync, diverts them to Rsync itself.

That is: the code above requires socat(1) to be available on
two hosts of the three involved. Note, however, that Rsync's
--server mode is orthogonal to --sender, so while I've used
--server on the sending host, and consequently had to use Socat
on the receiving one, it can be recoded the other way around.

The Socat instance in RSYNC_RSH is wrapped in sh -c with the
sole purpose of discarding the arguments passed to it by Rsync.

(Well, I bet that Rsync could easily be enhanced to support
--stdio in the client mode, obviating all this trickery.)

Curiously, both Socat and Rsync allow for "escaping" command
lines (in the EXEC and SYSTEM arguments for the first, and
RSYNC_RSH or -e for the second), yet the rules are markedly
different: Socat apparently implements only backslash as an
escape symbol (well, also, somewhat surprisingly, three kinds
of brackets; : and , are special and otherwise need escaping),
while Rsync only implements single and double quotes. In the
end, I've decided to use the Shell escapes via SYSTEM: for the
client command.

Finally, I wonder if using the Bash "coprocess" feature would've
been enough for the case?

[Ivan Shmakov]

Post by Ivan Shmakov
The other day I had to rsync files from host foo to host bar.
The problem is, the hosts are on the same "security tier," so I'd
rather not to log into either one from another (even with
authorized_keys 'command' and "restricted" rrsync.)
There's, however, a third host, qux, from which I /do/ log into both.
Hence, my idea was to start two SSH sessions from qux to both foo and
bar, running Rsync as a server on one and as a client on another,
connecting them to one another with socat(1).

Another option, especially suitable for repeated Rsync invocations,
initiated from the source or the target (whichever is decided to
be the server), is to forward a Unix socket via the third host.

Consider the following example.

1. Start an Rsync server for fromdir/, listening on a Unix
socket at foo:

foo $ socat UNIX-LISTEN:.rsync-server+fromdir.socket,fork \
SYSTEM:\''rsync --server --sender -u -rOtH \
--include="*".text --exclude="*" -- fromdir/ '\' &

The ,fork option there means that a new process will be
forked for each connection; the default is to listen for
a connection and "exec" the command (rsync in this case),
which obviously works just once.

2. Forward remote Unix socket from foo to one at qux:

qux $ rm -f -- .rsync-server-foo+fromdir.socket \
&& ssh -n -N -L ./.rsync-server-foo+fromdir.socket\
:/home/private/users/jrh/.rsync-server+fromdir.socket -- foo &

3. Forward local Unix socket from qux to one at bar:

qux $ ssh -n -- bar \
rm -f -- /home/private/users/jrh/.rsync-server-foo+fromdir.socket \
&& ssh -n -N \
-R /home/private/users/jrh/.rsync-server-foo+fromdir.socket\
:./.rsync-server-foo+fromdir.socket -- bar &

4. Invoke Rsync client at bar, copying data to todir:

bar $ RSYNC_RSH="sh -c 'exec socat STDIO UNIX:\"\$1\" ; ' dummy.sh " \
rsync -v -ub -rOtH --suffix=.~$(date +%s)~ \
-- .rsync-server-foo+fromdir.socket: todir/

Note that it seems that for some reason the remote socket
filenames in -L, -R have to be absolute. For the local ones
it suffices to use "./" to refer to the current directory, as
otherwise ssh(1) will try to interpret them as TCP endpoints.

The RSYNC_RSH code above may further be generalized to something
along the lines of:

#!/bin/sh

: "${REAL_RSYNC_RSH:=ssh}"

if [ -S "$1" ] ; then
## .
exec socat STDIO UNIX:"$1"
fi

## .
exec ${REAL_RSYNC_RSH} "$@"

I suppose that makes it fit for a permanent export RSYNC_RSH=.

Post by Ivan Shmakov
(Copying files from foo to qux and thence to bar was deemed to
be lacking a challenge.)

Not to mention that for repeated syncing that would imply either
keeping a copy of the data on the third host, or redownloading
it for each sync.

[...]