Airbrake deployment notification results in Jenkins job failure, but deployment gets recorded at Airbrake

Matt Chesler's Avatar

Matt Chesler

27 Jun, 2012 04:29 PM

We have capistrano configured to notify airbrake of deployments. We're doing our builds and deployments with Jenkins. I just had a deployment fail a couple of minutes ago, not because anything went wrong with the deployment, but because the airbrake notification returned a failure (see below). Despite the fact that the airbrake notification failed from Jenkins' perspective, the deployment was still recorded by airbrake. How can we prevent this from occurring in the future?

From Jenkins (some data redacted):

      triggering after callbacks for `deploy'
      * executing `airbrake:deploy'
      * executing "cat /XXXX/current/REVISION"
        servers: ["XXXX"]
        [XXXX] executing command
        [XXXX] rvm_path=/usr/local/rvm /usr/local/rvm/bin/rvm-shell 'ruby-1.9.3-p194@XXXX' -c 'cat /XXXX/current/REVISION'
        command finished in 2338ms
     ** Notifying Airbrake of Deploy (cd /XXXX; bundle exec rake RAILS_ENV=sandbox airbrake:deploy TO=sandbox REVISION=XXXX REPO=XXXX USER=XXXX)
      * executing "cd /XXXX; bundle exec rake RAILS_ENV=sandbox airbrake:deploy TO=sandbox REVISION=XXXX REPO=XXXX USER=XXXX"
        servers: ["XXXX"]
        [XXXX] executing command
        [XXXX] rvm_path=/usr/local/rvm /usr/local/rvm/bin/rvm-shell 'ruby-1.9.3-p194@XXXX' -c 'cd /XXXX; bundle exec rake RAILS_ENV=sandbox airbrake:deploy TO=sandbox REVISION=XXXX REPO=XXXX USER=XXXX'
        command finished in 94210ms
    failed: "rvm_path=/usr/local/rvm /usr/local/rvm/bin/rvm-shell 'ruby-1.9.3-p194@XXXX' -c 'cd /XXXX; bundle exec rake RAILS_ENV=sandbox airbrake:deploy TO=sandbox REVISION=XXXX REPO=XXXX USER=XXXX'" on XXXX
    Build step 'Conditional step (single)' marked build as failure
    Finished: FAILURE

  1. 2 Posted by Matt Chesler on 27 Jun, 2012 05:28 PM

    Matt Chesler's Avatar

    My gut tells me that this might be somewhat related to http://help.airbrake.io/discussions/problems/3388-capacity-issues. Note the time taken for the airbrake:deploy command to complete (94210ms => 94 seconds!)

  2. Support Staff 3 Posted by Morgan on 27 Jun, 2012 06:58 PM

    Morgan's Avatar

    Hello Matt,

    Thank you for raising this issue to our attention. I will add your email to the internal ticket to be notified. This issue may be related since your error may have been timeout related. I am surprised that a failure was issued when the job failed. I will report this to the development team. Thank you for the feedback, I will also reference this ticket for the data you have posted for us. I will keep you updated.

    From,
    Morgan

  3. 4 Posted by David Robertson on 03 Jul, 2012 09:48 PM

    David Robertson's Avatar

    Wouldn't mind knowing what happened with this

    command finished in 131445ms
    

    ** Airbrake Notification Complete.

    Our server is having the same issue

  4. Support Staff 5 Posted by Morgan on 03 Jul, 2012 11:49 PM

    Morgan's Avatar

    Hello Matt and David,

    We have done some major work on the gem and it would be a great idea to update to the latest stable version ( 3.1.2 ) and mimic this commit in your local code. Please let me know if problems persist after you have made those necessary changes. If that is the case I am happy to delve further into this issue.

    From,
    Morgan

  5. 6 Posted by David Robertson on 04 Jul, 2012 12:16 AM

    David Robertson's Avatar

    Just made it worse

    • executing `airbrake:deploy' ** Notifying Airbrake of Deploy (cd XXXXXXXXXXXX; bundle exec rake --trace RAILS_ENV=staging airbrake:deploy TO=staging REVISION=XXXXXXXXXXXX REPO=XXXXXXXXXXXX.git USER=XXXX)
    • executing "cd /XXXXXXXXXXXX bundle exec rake --trace RAILS_ENV=staging airbrake:deploy TO=staging REVISION=XXXXXXXXXXXX REPO=XXXXXXXXXXXX.git USER=XXXX" servers: [XXXXXXXXXXXX] [XXXXXXXXXXXX] executing command command finished in 236329ms ** Airbrake Notification Complete.

    Let me know if there is anything else I can do to help, this is causing our deploy downtime to be far to long. But we really like the features airbrake provides.

    Thanks Dave

  6. Support Staff 7 Posted by Morgan on 04 Jul, 2012 12:19 AM

    Morgan's Avatar

    Hello David,

    Please let me know what method you are using to deploy to airbrake.
    Also what is the output of Airbrake.configuration in the rails console?
    Also do you have config.host = 'api.airbrake.io' set in your airbrake.rb file?
    Please let me know, I want to make sure you are deploying to the right endpoint.

    From,
    Morgan

  7. 8 Posted by David Robertson on 04 Jul, 2012 12:30 AM

    David Robertson's Avatar

    We are using Capistrano
    require 'airbrake/capistrano'

    @api_key="XXXXXXXXXXXXXXXXXXXXXXXXXXX", @backtrace_filters= [#<Proc:0x000001033d1450@/Users/dave/.rvm/gems/ruby-1.9.2-p290@rails32/gems/airbrake-3.1.2/lib/airbrake/configuration.rb:106 (l #<Proc:0x000001033d1338@/Users/dave/.rvm/gems/ruby-1.9.2-p290@rails32/gems/airbrake-3.1.2/lib/airbrake/configuration.rb:113 (l #<Proc:0x000001033d1310@/Users/dave/.rvm/gems/ruby-1.9.2-p290@rails32/gems/airbrake-3.1.2/lib/airbrake/configuration.rb:114 (l #<Proc:0x000001033d12c0@/Users/dave/.rvm/gems/ruby-1.9.2-p290@rails32/gems/airbrake-3.1.2/lib/airbrake/configuration.rb:121 (l @development_environments=["development", "test", "cucumber"], @development_lookup=true, @environment_name="development", @framework="Rails: 3.2.2", @host="api.airbrake.io", @http_open_timeout=2, @http_read_timeout=5, @ignore= ["ActiveRecord::RecordNotFound", "ActionController::RoutingError", "ActionController::InvalidAuthenticityToken", "CGI::Session::CookieStore::TamperedWithCookie", "ActionController::UnknownAction", "AbstractController::ActionNotFound", "Mongoid::Errors::DocumentNotFound"], @ignore_by_filters=[], @ignore_user_agent=[], @logger= #<Logger:0x00000106cea7c8 @default_formatter=

    #<Logger::Formatter:0x00000106cea778 @datetime_format=nil>,
    

    @formatter=#<Logger::SimpleFormatter:0x00000106cea610 @datetime_format=nil>, @level=0, @logdev=

    #<Logger::LogDevice:0x00000106cea728
     @dev=#<IO:<STDOUT>>,
     @filename=nil,
     @mutex=
      #<Logger::LogDevice::LogDeviceMutex:0x00000106cea700
       @mon_count=0,
       @mon_mutex=#<Mutex:0x00000106cea6b0>,
       @mon_owner=nil>,
     @shift_age=nil,
     @shift_size=nil>,
    

    @progname=nil>, @notifier_name="Airbrake Notifier", @notifier_url="https://github.com/airbrake/airbrake", @notifier_version="3.1.2",

    We didnt have the config.host = 'api.airbrake.io' but it seems to be set in the configuration in the rails console so let me know if that maters and ill re deploy and see what happens this time.

    BTW Thanks for the very quick responses

  8. Support Staff 9 Posted by Morgan on 04 Jul, 2012 12:50 AM

    Morgan's Avatar

    Hi David,

    Thank you for the requested information! I am going to review this information and try to find out the problem behind this issue. I have also asked Herb on the development team to take a look into this one as well. Thank you for your patience!

    From,
    Morgan

  9. 10 Posted by DeveloperTown Administrator on 15 Aug, 2012 09:40 PM

    DeveloperTown Administrator's Avatar

    Is there any update to this issue? We're seeing this same behavior with apparently timedout requests. We're using 3.1.2, deploying through capistrano as well.

  10. Support Staff 11 Posted by Morgan on 20 Aug, 2012 05:18 PM

    Morgan's Avatar

    HI DeveloperTown,

    Thank you for re-raising this issue to our attention. Can you please supply me with a gist of your Airbrake.configuration as well as as much error output from the logs or stdout as is accessible. Thank you!

    From,
    Morgan

  11. 12 Posted by lancaster on 25 Aug, 2012 07:27 AM

    lancaster's Avatar

    My airbrake configuration only contains my api key, no other config at this point.

    Here's the relevant excerpt from our build system on a deploy that was successful, but marked as failure due to airbrake:deploy to one of our production environments:

     ** Notifying Airbrake of Deploy (cd /srv/www/xxxxx/releases/20120825064019; bundle exec rake RAILS_ENV=production airbrake:deploy TO=production REVISION=4e78983500bf8402d7943b7bd32d071677d132fe REPO=ssh://git@github.com/myorg/xxxxx-web.git USER=jenkins)
      * executing "cd /srv/www/xxxxx/releases/20120825064019; bundle exec rake RAILS_ENV=production airbrake:deploy TO=production REVISION=4e78983500bf8402d7943b7bd32d071677d132fe REPO=ssh://git@github.com/myorg/xxxxx-web.git USER=jenkins"
        servers: ["xxxxx-prodweb01.myorg.int"]
        [xxxxx-prodweb01.myorg.int] executing command
        [xxxxx-prodweb01.myorg.int] rvm_path=/usr/local/rvm /usr/share/ruby-rvm/bin/rvm-shell '1.9.3' -c 'cd /srv/www/xxxxx/releases/20120825064019; bundle exec rake RAILS_ENV=production airbrake:deploy TO=production REVISION=4e78983500bf8402d7943b7bd32d071677d132fe REPO=ssh://git@github.com/myorg/xxxxx-web.git USER=jenkins'
        command finished in 188437ms
    failed: "rvm_path=/usr/local/rvm /usr/share/ruby-rvm/bin/rvm-shell '1.9.3' -c 'cd /srv/www/xxxxx/releases/20120825064019; bundle exec rake RAILS_ENV=production airbrake:deploy TO=production REVISION=4e78983500bf8402d7943b7bd32d071677d132fe REPO=ssh://git@github.com/myorg/xxxxx-web.git USER=jenkins'" on xxxxx-prodweb01.myorg.int
    Build step 'Execute shell' marked build as failure
    Sending e-mails to: [email blocked]
    Finished: FAILURE
    
  12. Support Staff 13 Posted by Morgan on 25 Aug, 2012 09:23 PM

    Morgan's Avatar

    Hello,

    We have recently committed the following to stop loading of the entire environment for deploys. Please let me know if this does not solve the issues you are having!

    From,
    Morgan

  13. 14 Posted by David Robertson on 27 Aug, 2012 12:39 AM

    David Robertson's Avatar

    Sorry, I cant seem to update my airbrake bundle with the commit in it for this fix.

    Could you let me know which version to use?

  14. 15 Posted by lancaster on 27 Aug, 2012 01:32 AM

    lancaster's Avatar

    David - If you're using Bundler, it looks like you'll have to point to the git repo -- I set mine up like:

    gem 'airbrake', :git => "git://github.com/airbrake/airbrake.git"

    Then run bundle install to update it.

    Morgan: So far, so good, the first deploy took around 30 seconds for the airbrake plugin to do its work -- an improvement. I'll keep an eye on it. Thanks!

  15. 16 Posted by David Robertson on 27 Aug, 2012 04:16 AM

    David Robertson's Avatar

    Thanks Ian,

    * executing "cd /home/crichq/application/releases/*****************;
    bundle exec rake --trace RAILS_ENV=staging airbrake:deploy TO=staging REVISION=**********************REPO=git@github.com:********/*******.git USER=dave" servers: ["ec2-***-**-*-***.us-west-1.compute.amazonaws.com"][ec2-***-**-*-***.us-west-1.compute.amazonaws.com] executing command
    command finished in 158091ms
    

    Still takes ages for me...

  16. 17 Posted by Fabio Serra on 17 Oct, 2012 12:53 PM

    Fabio Serra's Avatar

    I am still getting this issue.
    Jenkins buid fails due do deploy notify to airbrake.
    How may I solve this?

  17. Support Staff 18 Posted by Morgan on 17 Oct, 2012 10:09 PM

    Morgan's Avatar

    Hello Fabio,

    Please make sure that your airbrake gem is the most current version.
    It would be helpful to have a private gist of the following to diagnose this issue:

    • Output from Airbrake.configuration run from inside of the rails console
    • The contents of your config/initializers/airbrake.rb file
    • Your gem list

    Update me when you get the chance, Happy to help you get this figured out!

    From,
    Morgan

  18. 19 Posted by Fabio Serra on 26 Nov, 2012 03:29 PM

    Fabio Serra's Avatar

    @Morgan

    here is the gist
    https://gist.github.com/08470838ea932175a8ae
    Sorry for delay

  19. Support Staff 20 Posted by Morgan on 02 Dec, 2012 07:20 PM

    Morgan's Avatar

    Hello Fabio,

    Thank you for getting us the information on your application's configuration.
    I have notified the development team to look into this issue.
    Thank you for your patience they will be in touch soon.

    From,
    Morgan

  20. 21 Posted by Ryan on 18 Feb, 2013 08:33 PM

    Ryan's Avatar

    Hello, we are currently revisiting tickets that seem that have been resolved. If you have any further questions let us know. Send an Email to
    [email blocked].

    Thanks,
    Ryan

  21. Ryan closed this discussion on 18 Feb, 2013 08:33 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac