Re: [Pan-users] Problem with Pan and rsync

[Duncan]

Joe Zeff posted on Fri, 26 Aug 2011 21:29:44 -0700 as excerpted:

> Right now, I'm house sitting at Chaos Manor (Some of you might know what
> that is.) and using my laptop.  I have my own domain, and my desktop has
> an Internet-visible machine name with dynamic DNS.  Before using Pan
> here, I tried using rsync to get everything I needed over here with this
> command:
> 
> rsync -av address@hidden:~/.pan2 ~/.pan2
> 
> However, when I ran Pan, it showed every post since the last time I ran
> it, earlier this month, as unread and I had to mark them manually.  What
> did I do wrong?  (If it wasn't clear, this is on Linux.)

That rsync command got corrupted on gmane, as evidently it has an @ in 
it, and gmane took it as an email address and "obfuscated" it. =:^\  As 
such, this is of necessity a bit hand-wavy.

(Added later, right before sending:)  I hope the following doesn't come 
across as mocking.  It's not intended to, tho I'm aware it could come 
across that way, thus this note.  Rather, this is an important lesson a 
lot of users including me have had to learn the hard way.  In my case, my 
lesson on the importance of dry-run and confirming shell substitutions 
was a script that did an rm -r $variable/, only I typoed $variable so it 
ended up empty, and did an rm-r /, instead!  I made the further mistake 
of breaking the rules and testing the script "live as root", catching it 
only after the rm seemed to be taking longer, with way more disk 
activity, than it should have!  If you've never had that sinking feeling 
of realizing you just did a recursive rm / as root, consider yourself 
*VERY* lucky!  (/bin was gone, /boot, /etc, part of /home... when I hit 
^c.  Luckily I had a somewhat dated, but workable, backup to restore 
from.)  This experience is why I was SO careful to *CAREFULLY* debug my 
rsync scripts, making dry-run the default, putting in confirmations, 
etc.  If you've reversed your rsync direction, you've just staled (if 
you're lucky and didn't sync with an empty source dir!) everything you 
intended to update from, so rsync's --dry-run is a **VERY** good thing to 
have and for users to use.  (With a reasonable amount of system memory, 
dry-run isn't not really that bad, either, because the dry-run gets the 
files into cache so the LIVE run goes much quicker, without the level of 
disk thrashing the dry-run had, as the files are already cached on both 
sides.)

What I got from gmane's obfuscation:

rsync -av user- (user-something @ something) :~/.pan2 ~/.pan2

FWIW, when I use rsync for syncing between my netbook build-image on my 
main machine, and the netbook itself, I try using --dry-run (-n) first, 
just to see what it's syncing.  In my case, now that I've been using the 
scripted setup with all the correct switches for sometime, and am thus 
relatively confident I have /it/ right, sometimes I've changed a few 
files differently on each machine, between update sessions, and the dry-
run allows me to see which ones I might need to manually diff to remember 
what I changed, before I do the real pre-build-image-update.  After I've 
manually synced them, I do the rsync for real, so both sides are in the 
same pre-update state, then do the update, then sync again, without 
making changes to the netbook in the mean time, so the second sync can be 
one-way without danger of overwriting anything I might have changed on 
the netbook, since I haven't changed anything on it since the pre-update 
sync.

In your case, however, you'd use dry-run to make sure it's syncing what 
you intended, not something else.  (Before I got in the dry-run-first 
habit, I once synced without mounting a normally unmounted partition that 
I should have.  It filled up the existing mountpoint partition and errored 
out.  Fortunately, all I had to do was delete all the junk in the 
otherwise empty mount-point-only dir, but had I been syncing the other 
way, it would have deleted everything I wanted to sync, since I was 
syncing with an empty dir!  After that close-call, I *ALWAYS* do the dry-
run, first!)

Note that with --dry-run, you can reverse the sync from what you actually 
intend to do, as well, to make sure it's syncing the actual files that 
should be there, not something somewhere entirely unintended, due to a 
typo or forgetting to mount a partition first, or something.  Just be 
*ABSOLUTELY* *SURE* you have the --dry-run in there first.  (That's one 
benefit of the scripts.  If you make --dry-run the default as I did in my 
scripts, if there's a typo, it does the dry-run and doesn't harm 
anything.  I have to actually set a --LIVE parameter on my script, to get 
it not to do the dry-run, and the confirmation prompt shows me the 
command it'll run and specifically notes whether it is LIVE or not, 
before asking if I'm sure.)

The --dry-run first should take care of ensuring that you're doing what 
you intend to do, but what happened in this case?

I don't know for sure, but I can GUESS:  If you indeed used ~/ for 
/home/username/ in both cases, as the (part that I can make out of the) 
above indicates, that might have been the mistake, since the shell (on 
the machine you're running the rsync command from) expands that itself -- 
to the local value.  Quoting the bash (other POSIX shells should be 
similar but may not be identical, there's more to it, but this is the 
main bit) manpage:


Tilde Expansion
   If a word begins with an unquoted tilde character (`~'), all
   of the characters preceding the first unquoted slash (or all
   characters, if there is no unquoted slash) are considered a
   tilde-prefix.  If  none of the characters in the tilde-prefix
   are quoted, the characters in the tilde-prefix following the
   tilde are treated as a possible login name.  If this login
   name is the null string, the tilde is replaced with the value
   of the shell parameter HOME.  If HOME is unset, the home
   directory of the user executing the shell is  substituted
   instead.  Otherwise, the tilde-prefix is replaced with the
   home directory associated with the specified login name.

Basically, anything after a ~ but before the first / is considered a 
login name.  If it's null, the value of $HOME is substituted.  If $HOME 
is unset or the login name isn't null, the system queryies /etc/passwd 
(or whatever) for the home dir for that user, substituting the result.

Even more basically, under ordinary circumstances, the shell will 
substitute ~ with the location of the user's home dir, the user in 
question being the user under which bash is running.

My guess is thus that the value the shell substituted for those ~s was 
the wrong one for either the remote or local side.  If you were 
specifying the rsync user @ host (spaces added to avoid gmane 
obfuscation) information, it's VERY likely that the local shell's 
substitution for those ~s was NOT what you intended.

Once again, you'd have very likely caught this easily, had you used a 
--dry-run first, or even had you simply verified the output of what it 
actually did.  However, it would appear you didn't, so you didn't catch 
it.

Meanwhile, check your systems to see what DID get synced.  It's very 
possible you synced something between /root dirs, if you were running as 
root at the time, or something similarly not what you expected, but doing 
exactly what rsync was told to do after the shell finished doing its 
substitutions for those tildes. =:^)

Of course, that's another benefit of using a script that shows you the 
command as it's going to run it, with the same substitutions made, then 
asks you to confirm that's what you really intend to do, before it does 
it.  I very quickly learned the benefit of this myself, while debugging 
my own sync scripts, trying to get them setup so that what I was 
/telling/ rsync to do was exactly what I /intended/ to tell it to do!  
Had you been using such a script, you'd have seen the substitutions the 
shell made, and would have almost certainly noted if it was trying to 
sync the wrong path due to using substitutions other than what you 
intended.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman