Re: celery connection problems to RabbitMQ

[Ben Sturmfels]

Hi Fernando,

Thank you! I've merged your Celery/RabbitMQ fix and commented in the
spectrogram ticket.

Regards,
Ben

On Tue, 12 May 2020, Fernando Gutierrez wrote:

> PySoundFile and SoundFile are the same libraries. Originally it was
> called PySoundFile but in release 0.10 they renamed it to just
> SoundFile. There are a few small differences between both but the
> code I wrote can use any of them.
>
> I created this tickets and attached the proposed fixes:
>
> https://issues.mediagoblin.org/ticket/5609 : Proposed fix for celery
> RabbitMQ connection errors 
> https://issues.mediagoblin.org/ticket/5610 : Proposed patch to
> re-enable audio spectrograms
>
> Thanks
> Fernando
>
> On Mon, May 11, 2020 at 5:44 AM Ben Sturmfels
> <address@hidden> wrote:
>
>  Hi Fernando,
>
>  Please post a patch or a link to a remote branch to an issue on
>  the
>  issue tracker - ideally a separate change for celery and
>  spectrograms.
>
>  Regarding PySoundFile, the file you're after is setup.py.
>
>  For what it's worth, I see that the Python SoundFile library is
>  available in Debian, but PySoundFile doesn't appear to be. This
>  isn't a
>  complete showstopper, but it would help us when tackling distro
>  packaging in the near future.
>
>  Regards,
>  Ben
>
>  On Mon, 11 May 2020, Fernando Gutierrez wrote:
>
>  > Hi Ben
>  >
>  > Sorry I think I didn't explain clearly. I only fixed the connection
>  reset
>  > exceptions in celery but the bug with media changed to failed
>  state is
>  > not fixed.
>  >
>  > I will continue debugging but it may take some time. I don't
>  know
>  > why celery thinks a completed task needs to be run again.
>  >
>  > In the meantime I will submit a patch for the systemd file, the
>  > BROKER_HEARTBEAK issue and also a fix for the audio
>  spectrogram
>  > code as I mentioned in the IRC channel.
>  >
>  > I have a couple of questions:
>  >
>  > 1) I'm not familiar with the development process, I already
>  created an
>  > account in savannah.gnu.org but I don't see how to submit a
>  patch
>  > for review.
>  > 2) For the spectrogram I used the PySoundFile package. What
>  file do
>  > I need to modify so it gets pulled during setup?, in my setup I
>  > manually called ./bin/pip install PySoundFile
>  >
>  > Thanks
>  > Fernando
>  >
>  > On Sun, May 10, 2020 at 6:39 AM Ben Sturmfels
>  > <address@hidden> wrote:
>  >
>  >  Hi Fernando,
>  >
>  >  On Sun, 10 May 2020, Fernando Gutierrez wrote:
>  >
>  >  > I recently asked in the IRC channel about RabbitMQ
>  connection
>  >  reset
>  >  > errors in celeryd logs.
>  >  >
>  >  > I think there are two issues:
>  >  >
>  >  > 1) The example systemd file (mediagoblin-celeryd.service)
>  from
>  >  >
>  > 
>  https://mediagoblin.readthedocs.io/en/stable/siteadmin/deploying.html
>  
>  >  
>  >  > does not specify that celeryd must be started after
>  RabbitMQ,
>  >  so it is
>  >  > sometimes started before and fails because RabbitMQ is not
>  >  running
>  >  > yet.
>  >  >
>  >  > 2) In mediagoblin/mediagoblin/init/celery/__init__.py, it sets
>  >  > celery_settings['BROKER_HEARTBEAT'] = 1. In slower
>  systems
>  >  or
>  >  > under heavy load if the worker is too slow to respond in < 1
>  >  second it
>  >  > will miss the heartbeat and after a few missed heartbeats
>  the
>  >  > connection is considered dead and reset.
>  >  > I'm not sure what is the purpose of changing
>  >  BROKER_HEARTBEAT to
>  >  > 1 but the celery docs recommend not using such a small
>  >  value. In my
>  >  > install I changed it to 20 and I no longer see any connection
>  >  > problems.
>  >  >
>  >  > Are you willing to accept a patch for
>  >  > mediagoblin/docs/source/siteadmin/deployment.rst and
>  >  > mediagoblin/mediagoblin/init/celery/__init__.py to fix those
>  two
>  >  > problems?
>  >
>  >  Thank you very much for diving in and investigating the
>  issue.
>  >  We'd be
>  >  happy to take a patch on this. If you can add a comment to
>  >  explain the
>  >  new BROKER_HEARTBEAT value in the code, that would be
>  great.
>  >
>  >  I wonder if there there might still be a problem lurking here
>  >  though,
>  >  even if your system is now working properly Not being able to
>  >  connect to
>  >  RabbitMQ or an unresponsive celery worker probably shouldn't
>  >  change
>  >  existing processed media items to failed.
>  >
>  >  Thanks for your work on this!
>  >
>  >  Regards,
>  >  Ben