Triggering Jenkins jobs from the SCM push to avoid the evil polling

Why?

If you have a continuous integration infrastructure with Jenkins you might have you jobs configured to make polling over your SCM in order to trigger a job when there are changes. But this has some problems when Jenkins has several nodes and there are a big amount of jobs.

In my case, i configured the Jenkins jobs to poll the SCM repository ( Mercurial ) every 5 minutes. What that means?

  • A very big load in the SCM repository server due to the big amount of pollings from every Jenkins job.
  • Unexpected behaviour of the polling due to Jenkins bugs or SCM plugin bugs ( at least in the Mercurial one )
    • Jobs launched with no changes because the polling couldn’t be done because maybe the workspace of the previous build is not available
    • Jobs not launched although there are changes because of the loss of threads that manage the channel connections between the master and the slaves (Jenkins bug)

How can we avoid these issues? Triggering the jobs only when there are changes in the SCM. How? With hooks

You can set a push hook to trigger the necessary jobs through the Jenkins CLI API.

In my case, the job names and the SCM branches are not the same, so i need to get a map of all jobs with their branches in order to launch the jobs that owns the branches obtained in the push. To get and launch them, also through the Jenkins CLI API.

These would be the steps:

    1. Create the push hook: this hook must analize the content of the push and detect the branches associated. This is how it looks like for a Mercurial repository:
      #!/bin/bash
      #$HG_NODE is an env variable set by Mercurial
      BASE_REV=$HG_NODE
      HEAD_REV=`hg tip --template '{rev}'`
      BRANCHES=`hg log -r $BASE_REV:$HEAD_REV --template '{branch}\n' | sort -u`
      
    2. Once you have the branches detected with the hook, you need to find which are the jobs that own those branches, to do that, use the Jenkins CLI API.

Brief tutorial of how to use it:

You need to download a .jar file that you can find in:

http://yourJenkinsDomain/cli

Test it’s working with:

java -jar jenkins-cli.jar -s http://yourJenkinsDomain/ help

It that worked, you need to configure the SSH keys to login through the API. Go to the configuration page of your Jenkins user ( create one if you don’t have )

http://yourJenkinsDomain/user/YourUser/configure

In the “SSH Public Keys” section, add the public SSH key of the user who is going to execute the hook. This section is the same as the file ~/.ssh/authorized_keys

Now, you should be able to login through the API with:

java -jar jenkins-cli.jar -s http://yourJenkinsDomain/ -i ~/.ssh/id_rsa login

(you must pass your private SSH key with the -i param)

Now you are able to connect to the API and use it. Then, obtain the jobs that own the branches detected with the hook above. How? With Groovy

    1. Create a Groovy script to get all the jobs with their branches. It could be something like:
hudson.model.Hudson.instance.getView('All').items.each() {
    if(it.scm.class.getSimpleName().equals("MercurialSCM")) {
        println it.fullDisplayName+"#"+it.scm.branch
    }
}

It will returns a line with the name and the branch per Jenkins job separated by “#”. Test it works with:

java -jar jenkins-cli.jar -s http://yourJenkinsDomain/ groovy /path/to/your/groovy/script
    1. Complete the hook to process the info gathered by the Groovy code and launch the jobs with something similar to:

#!/bin/bash
BASE_REV=$HG_NODE
HEAD_REV=`hg tip --template '{rev}'`
BRANCHES=`hg log -r $BASE_REV:$HEAD_REV --template '{branch}\n' | sort -u`
if [ ! -z "$BRANCHES" ]; then
    #Login
    java -jar jenkins-cli.jar -s http://yourJenkinsDomain/ -i ~/.ssh/id_rsa login
    #Get the Groovy info
    JOB_BRANCH_MAP=`java -jar jenkins-cli.jar -s http://yourJenkinsDomain/ groovy /path/to/your/groovy/script`
    #For each job found
    for i in $(echo $JOB_BRANCH_MAP | tr " " "\n")
    do
        #extract the first part of the line to get the job branch
        BRANCH_OF_THE_JOB=`expr match "$i" '.*#\(.*\)#'`
        for j in $(echo $BRANCHES | tr "\n" "\n")
        do
            if [ $BRANCH_OF_THE_JOB == $j ]; then
                #Extract the second part of the line to get the job name
                JOB_NAME=`expr match "$i" '\(.*\)#.*#.*'`
                # Launch the job
                java -jar jenkins-cli.jar -s http://yourJenkinsDomain/ build $JOB_NAME
                break
            fi
        done
     done
fi
Advertisements

8 thoughts on “Triggering Jenkins jobs from the SCM push to avoid the evil polling

    • You’re welcome!

      Just after i wrote this post i find what Kohsuke was doing with Git. Definitely, the Jenkins polling is something that should be avoided whenever is possible

    • The Groovy script is a new file places wherever you want and the Bash file is a Mercurial hook. If you don’t know about Mercurial hooks you should take a look to its documentation

  1. Can you elaborate on how to integrate cvs with jenkins for continuous integration and second to what extent integration of jira is possible with jenkins??
    Thanx in advance

    • Hi Nitish,

      I think you need to be more specific on what you want. cvs integration with jenkins is straigforward, you just need to install the git or mercurial plugin.
      And Jira, what do you need?

  2. Hi,

    I was trying to do polling on a CVS repository through Jenkins but it turns out to be a lengthy process.
    Besides, I don’t want the complete code to be checked out with each build being triggered. I just want the poll plugin to verify whether or not changes are pushed in the repository and trigger the build without checking out the code.
    is there any way I can achieve this ?

    • Hi,

      I don’t know how CVS repositories work, i know with Git you can do it using a bare repository where you can track the metadata but you don’t need to checkout the code.

      The question is, why are you using a CVS repository in 2017??

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s