Re: [Fab-user] Trouble using fabric with EC2

[Matt Calder]

Jeff,

On Thu, Jun 10, 2010 at 6:54 PM, Jeff Forcier <address@hidden> wrote:
> Hi Matt,
>
> Paramiko doesn't have a connection cache that I'm aware of, but Fabric
> itself does. However, from your description it sounds like you are
> creating a new instance and then connecting to it, so I'm not sure why
> a cache would present a problem.
>

I'm fairly certain fabric's cache is empty, because the code goes into
the network.py : connect function. The reason I suggested a "paramiko
cache" is that, while it is true that just after an instance goes from
"pending" to "running" there is a period when connections fail, but
that usually is very brief (< 10 sec). That is why I do a sleep(60)
after the startup, to give time for that to settle.

> If you're rebooting a remote system or doing anything to alter the
> networking of an already-connected system, then you can force a
> reconnect by manipulating fabric.state.connections. For example, see
> what the (master-only) reboot() operation does:
>
>    
> http://code.fabfile.org/repositories/entry/fabric/master/fabric/operations.py#L668
>

I will look at that.

> If the problem is as straightforward as it sounds, though, I'm
> honestly not sure what's up other than "possible Paramiko bug". Are
> you getting any prompts or anything when you connect to the new
> instance by hand?
>

I can log in by hand, completely and correctly, from a terminal. I can
do this after the instance is started but before fabric's first run
call. The funny thing is, if I do log in from a terminal, the fabric
run command will work. So, a pseudo code timeline:

# Version 1, this will fail, the run cannot connect to the instance.
startInstance()
sleep(60)
run("ls")

# Version 2, this will succeed in running "ls" on the instance.
startInstance()
sleep(60) # During this sleep, using a terminal, I log into the instance.
run("ls")

Another variation that works is:

# Version 3, this also succeeds.
startInstance()
sleep(60)
<Debugger breakpoint here> Using debugger, look at variables (no
changes), proceed
run("ls")

It is the examples that work that shout out "threading error" or
"caching error" to me.

> Another thing to try is to upgrade Paramiko to 1.7.6 if you're using
> the bundled 1.7.4.
>

I will try that. Thanks for taking the time to help!

Matt

> -Jeff
>
>
> On Thu, Jun 10, 2010 at 5:38 PM, Matt Calder <address@hidden> wrote:
>> Bruno,
>>
>> No it is in a good group. I can log in using fabric if I restart it
>> and the instance is already running. I can see that fabric is inside
>> network.py trying to make the connection. I get one of two errors:
>> either timeout or low level socket error. In debugging, I added
>> retries to network.connect and it will fail repeatedly. First it times
>> out a few times, then gives the "low level socket" error. While it
>> doing that, I can ssh into it from a terminal. I wonder does paramiko
>> have a connection cache ? Maybe it is not really retrying? Thanks for
>> any help.
>>
>>
>> Matt
>>
>> On Thu, Jun 10, 2010 at 5:23 PM, Bruno Clermont
>> <address@hidden> wrote:
>>> Is your instance in a security group that allow your IP and the port your
>>> trying to connect to?
>>> If it timeout, it's probably blocked by Amazon firewalls.
>>>
>>> On Thu, Jun 10, 2010 at 15:07, Matt Calder <address@hidden> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am having problems using fabric with EC2 instances. I am not
>>>> entirely sure fabric is even the source of the problem, but I am
>>>> hoping someone on this list can suggest a solution or a path to
>>>> investigate. Here is the problem. I start an EC2 instance using boto.
>>>> I wait for the instance to report its state as "running". I wait an
>>>> addition 60 seconds after that. Then I try to "run" things on the
>>>> instance through fabric. At that point I get:
>>>>
>>>> address@hidden run: ls
>>>>
>>>> Fatal error: Timed out trying to connect to
>>>> ec2-174-129-96-241.compute-1.amazonaws.com
>>>>
>>>> Aborting.
>>>>
>>>> Now, the interesting thing is this. During that additional 60 second
>>>> wait I can log into the instance from a separate terminal, moreover,
>>>> when I do that separate login, the fabric login succeeds.
>>>>
>>>> Obviously, there is not a lot to go on here, but I am not entirely
>>>> sure what additional information would be helpful. If anyone has a
>>>> suggestion of what I might try to do, I would greatly appreciate it.
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>> _______________________________________________
>>>> Fab-user mailing list
>>>> address@hidden
>>>> http://lists.nongnu.org/mailman/listinfo/fab-user
>>>
>>>
>>
>> _______________________________________________
>> Fab-user mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/fab-user
>>
>
>
>
> --
> Jeff Forcier
> Unix sysadmin; Python/Ruby developer
> http://bitprophet.org
>