New Infra Bootstrap

This document uses example.org as the domain for all examples. Please change to point to the intended systems for your project.

Jenkins

Steps

  1. Login to Jenkins at https://jenkins.example.org

  2. Navigate to https://jenkins.example.org/pluginManager/

  3. Update all plugins

  4. Install required plugins as documented in global-jjb install guide

  5. Install the following plugins:

  6. Navigate to https://jenkins.example.org/configure

  7. Configure Jenkins as follows:

    # of executors: 0
    Jenkins URL: https://jenkins.example.org
    System Admin e-mail address: Jenkins <jenkins-dontreply@example.org>
    Global Config user.name value: jenkins
    Global Config user.email value: jenkins@example.org
    

    If using the Message Injector plugin set Message to inject to Logs: https://logs.example.org/SILO/HOSTNAME/$JOB_NAME/$BUILD_NUMBER and replace SILO and HOSTNAME as appropriate.

  8. Click Save

  9. Configure Jenkins security as described in Jenkins Security

  10. Navigate to https://jenkins.example.org/configureSecurity/

  11. Configure the following permissions for Anonymous Users

    • Overall:Read

    • Job:ExtendedRead

    • Job:Read

    • View:Read

    Note

    If the project is not yet public, hold off on these permissions or adjust as necessary for the project’s case.

  12. Setup Jenkins global environment variables as described in the global-jjb install guide

    Note

    Skip the ci-management step in as we will be discussing that in the next section.

  13. Setup a jobbuilder account

  14. Setup global-jjb required Jenkins Files

Setup Job Builder account

The ci-jobs in global-jjb require a jobbuilder account which has permissions to login to Jenkins.

  1. Navigate to and create an account for jobbuilder https://identity.linuxfoundation.org/

    Note

    This step mainly applies to LF projects. Use the relevant identity system as it applies to your local configuration.

  2. Navigate to https://jenkins.example.org/configureSecurity and configure permissions for the jobbuilder account as follows:

    • Overall: Administer

    • Job: Configure

    • Job: Create

    • Job: Delete

    • Job: Discover

    • Job: Read

    • View: Configure

    • View: Create

    • View: Delete

    • View: Read

Setup Sandbox Access

To allow people access to the Jenkins Sandbox, we require an LDAP group to exist with the appropriate people added. Use lftools lfidapi create-group to create a group called $project-jenkins-sandbox-access and add any initial members you might need.

  1. Go to https://jenkins.example.org/configureSecurity and add the group with:

    • Overall: Read

    • Job: Build

    • Job: Cancel

    • Job: Configure

    • Job: Create

    • Job: Delete

    • Job: Discover

    • Job: Read

    • Job: Workspace

    • View: Read

ci-management repo

Once Jenkins is available we can initialize a new ci-management repo.

Setup administrative files

  1. Create ci-management repo in the project SCM system

  2. Create a README.md file explaining the purpose of the repo

    # ci-management
    
    This repo contains configuration files for Jenkins jobs for the EXAMPLE
    project.
    
  3. Setup tox/coala linting for jjb/ and packer/ directories

    .yamllint.conf

    extends: default
    
    rules:
      empty-lines:
        max-end: 1
      line-length:
        max: 120
    

    .coafile

    [Documentation]
    bears = WriteGoodLintBear
    files = *.md
    allow_so_beginning = False
    allow_there_is = False
    allow_cliche_phrases = False
    
    [GitCommit]
    bears = GitCommitBear
    ignore_length_regex = Signed-off-by,
        Also-by,
        Co-authored-by,
        http://,
        https://
    
    [JSON]
    bears = JSONFormatBear
    files = packer/**.json
    indent_size = 2
    
    [ShellCheck]
    bears = ShellCheckBear,
        SpaceConsistencyBear
    files = jjb/**.sh,
        packer/**.sh
    shell = bash
    indent_size = 4
    use_spaces = yeah
    
    [YAML]
    bears = YAMLLintBear
    files = jjb/**/*.yaml
    document_start = True
    yamllint_config = .yamllint.conf
    

    tox.ini

    [tox]
    minversion = 1.6
    envlist = coala
    skipsdist = true
    
    [testenv:coala]
    basepython = python3
    deps =
        coala==0.11
        coala-bears==0.11
        nodeenv~=1.3.0
    commands =
        nodeenv -p
        npm install --global write-good
        python3 -m nltk.downloader punkt maxent_treebank_pos_tagger averaged_perceptron_tagger
        coala --non-interactive
    
  4. Setup .gitignore

    .tox/
    archives/
    jenkins.ini
    
    # Packer
    .galaxy/
    *.retry
    cloud-env.json
    
  5. git commit -asm "Setup repo administrative files"

  6. git push files to the repository

  7. Run tox

    Note

    The jjb tox env will fail as the required jjb/ directory does not yet exist. This is fine and proves that tox is working before we continue in the next step.

Bootstrap common-packer and initial builder

Note

This section assumes the usage of an OpenStack cloud provider for Jenkins build nodes. Adjust as necessary if not using an OpenStack cloud.

  1. Navigate to the GIT_ROOT of the ci-management repo

  2. Install common-packer to GIT_ROOT/packer/common-packer

    git submodule add https://github.com/lfit/releng-common-packer.git packer/common-packer
    
  3. Follow common-packer doc to setup a template

  4. git commit -asm "Setup common-packer and initial builder"

  5. git push files to repository

  6. Upload a CentOS 7 cloudimg to use as a base for packer builds

    When uploading the cloudimg ensure it’s name matches the base_image name in common-packer/vars/centos-7.json.

  7. Run packer build -var-file=cloud-env.json -var-file=common-packer/vars/centos-7.json templates/builder.json

  8. Note down the image name from the packer build as we will need it later

  9. Navigate to https://jenkins.example.org/credentials/store/system/domain/_/newCredentials

  10. Configure the OpenStack cloud credential as follows:

    Kind: OpenStack auth v3
    Project Domain: Default
    Project Name: OPENSTACK_TENANT_ID
    User Domain: Default
    User Name: OPENSTACK_USERNAME
    Password: OPENSTACK_PASSWORD
    ID: os-cloud
    Description: openstack-cloud-credential
    

    Note

    Replace ALL_CAPS instances with your Cattle account credential.

  11. Configure an ssh keypair for the Jenkins <-> OpenStack connection

    1. Generate a new SSH Keypair

      ssh-keygen -t rsa -C jenkins-ssh -f /tmp/jenkins
      
    2. Navigate to https://jenkins.example.org/credentials/store/system/domain/_/newCredentials

    3. Configure the Jenkins SSH Key as follows:

      Kind: SSH Username and private key
      Scope: Global
      Username: jenkins
      Private Key: Enter directly
      Passphrase:
      ID: jenkins-ssh
      Description: jenkins-ssh
      

      Copy the contents of /tmp/jenkins into the Key field.

    4. Navigate to https://openstack-cloud.example.org/project/key_pairs

    5. Import the contents of /tmp/jenkins.pub into the OpenStack cloud provider account with the keypair name jenkins-ssh

  12. Navigate to https://jenkins.example.org/configfiles/selectProvider

  13. Create a OpenStack User Data file with the following specs:

    Type: OpenStack User Data
    ID: jenkins-init-script
    Name: jenkins-init-script
    Comment: jenkins-init-script
    

    With the contents (change the git clone URL as necessary for the project):

    #!/bin/bash
    until host gerrit.example.org &>/dev/null
    do
        echo "Waiting until gerrit.example.org is resolvable..."
    done
    
    git clone --recurse-submodules https://gerrit.example.org/r/ci-management /opt/ciman
    /opt/ciman/jjb/global-jjb/jenkins-init-scripts/init.sh
    

    For Windows:

    Type: OpenStack User Data
    ID: jenkins-init-script-windows
    Name: jenkins-init-script-windows
    Comment: jenkins-init-script-windows
    

    With the contents (change the git clone URL as necessary for the project):

    <powershell>
      # Resize first partition of first disk to maximum size
      Get-Partition -DiskNumber 0 -PartitionNumber 1
      $size = (Get-PartitionSupportedSize -DiskNumber 0 -PartitionNumber 1)
      Resize-Partition -DiskNumber 0 -PartitionNumber 1 -Size $size.SizeMax
    
      mkdir -Force "${SLAVE_JENKINS_HOME}"
      (new-object System.Net.WebClient).DownloadFile('${SLAVE_JAR_URL}','${SLAVE_JENKINS_HOME}\slave.jar')
      cd "${SLAVE_JENKINS_HOME}"
      java ${SLAVE_JVM_OPTIONS} -jar "slave.jar" -jnlpUrl "${SLAVE_JNLP_URL}" -secret "${SLAVE_JNLP_SECRET}"
    </powershell>
    
  14. Configure cattle cloud

    1. Create cloud config directory mkdir -p jenkins-config/clouds/openstack/cattle

    2. Configure the OpenStack cloud connection details in jenkins-config/clouds/openstack/cattle/cloud.cfg

      Replace <BUILD_IMAGE_NAME> and <NETWORK_ID> in the below file with the details for your cloud. Find <NETWORK_ID> at https://dashboard.vexxhost.net/project/networks/

      jenkins-config/clouds/openstack/cattle/cloud.cfg
      # Cloud Configuration
      CLOUD_CREDENTIAL_ID=os-cloud
      CLOUD_URL=https://auth.vexxhost.net/v3/
      CLOUD_IGNORE_SSL=false
      CLOUD_ZONE=ca-ymq-1
      
      # Default Template Configuration
      IMAGE_NAME=<BUILD_IMAGE_NAME>
      HARDWARE_ID=v3-standard-2
      NETWORK_ID=<NETWORK_ID>
      USER_DATA_ID=jenkins-init-script
      INSTANCE_CAP=10
      SANDBOX_CAP=4
      FLOATING_IP_POOL=
      SECURITY_GROUPS=default
      STARTUP_TIMEOUT=600000
      KEY_PAIR_NAME=jenkins-ssh
      NUM_EXECUTORS=1
      JVM_OPTIONS=
      FS_ROOT=/w
      RETENTION_TIME=0
      
    3. Create jenkins-config/clouds/openstack/cattle/centos7-builder-2c-1g.cfg

      IMAGE_NAME=ZZCI - CentOS 7 - builder - 20180604-1653
      HARDWARE_ID=v3-standard-2
      
    4. Run global-jjb jenkins-cfg script to update Jenkins cloud config

      Note

      This step requires crudini tool, install from your package manager to avoid python 2 vs 3 problems in your virtualenv.

      Note

      This step requires having lftools available on your path and a ~/.config/jenkins_jobs/jenkins_jobs.ini configured with Jenkins credentials.

      Set jenkins_silos to match the config section name in the jenkins_jobs.ini file.

      Run the following commands:

      export WORKSPACE=$(pwd)
      export jenkins_silos=production
      bash ./jjb/global-jjb/shell/jenkins-configure-clouds.sh
      # OPTIONAL: view the created script
      cat archives/groovy-inserts/production-cloud-cfg.groovy
      

      Then navigate to https://jenkins.example.org/script and copy the contents of archives/groovy-inserts/production-cloud-cfg.groovy into the script console. This will initialize the OpenStack cloud configuration.

    5. Commit the jenkins-config directory

      git add jenkins-config/
      git commit -sm "Add OpenStack cloud configuration"
      git push
      
  15. Navigate to https://jenkins.example.org/configure and verify the cloud configuration.

Setup global-jjb and ci-jobs

  1. Install global-jjb to GIT_ROOT/jjb/global-jjb

    git submodule add https://github.com/lfit/releng-global-jjb.git jjb/global-jjb
    
  2. Setup jjb/defaults.yaml

    - defaults:
        name: global
    
        gerrit-server-name: Primary
        git-url: 'ssh://jenkins-$SILO@gerrit.example.org:29418'
        jenkins-ssh-credential: jenkins-ssh
        lftools-version: '<1.0.0'
    
  3. Create the CI Jobs in jjb/ci-management/ci-jobs.yaml

    - project:
        name: ci-jobs
    
        jobs:
          - '{project-name}-ci-jobs'
    
        project: ci-management
        project-name: ci-management
        build-node: centos7-builder-2c-1g
    
  4. Manually push the initial ci-management jobs to Jenkins

    jenkins-jobs update jjb/
    
  5. Git commit the current files and push to Gerrit

    git commit -sm "Setup global-jjb and ci-jobs"
    git push
    
  6. Confirm verify jobs work

  7. Merge the patch and confirm merge job works

Setup packer jobs

  1. Create Initial CI Packer job in jjb/ci-management/ci-packer.yaml

    - project:
        name: packer-verify
        jobs:
          - gerrit-packer-verify
    
        project: ci-management
        project-name: ci-management
        build-node: centos7-builder-2c-1g
    
    - project:
        name: packer-builder-jobs
        jobs:
          - gerrit-packer-merge
    
        project: ci-management
        project-name: ci-management
        build-node: centos7-builder-2c-1g
    
        templates: builder
        platforms:
          - centos-7
          - ubuntu-16.04
    
  2. Git commit and push the patch to ci-management for review

    git commit -sm "Add packer builder job"
    git push ...
    
  3. Confirm packer verify job passes

  4. Merge patch and confirm merge job works

Nexus 2

Setup Server Config

  1. Navigate to https://nexus.example.org/#nexus-config

  2. SMTP Settings

    Hostname: localhost
    Port: 25
    Username:
    Password:
    Connection: Use plain SMTP
    System Email: noreply@example.org
    
  3. Application Server Settings

    Base URL: https://nexus.example.org/
    Force base URL: true
    UI Timeout: 120
    
  4. PGP Key Server Information

    Server 1: http://pool.sks-keyservers.net:11371
    Server 2: http://pgp.mit.edu:11371
    

Setup LDAP

  1. Navigate to https://nexus.example.org/#enterprise-ldap

  2. Click Add at the top menu bar

  3. Configure the LDAP connection as follows:

    Name: ldaps://ldap.example.org:636
    Protocol: ldaps
    Hostname: ldap.example.org
    Port: 636
    Search Base: dc=example,dc=org
    
    Authentication: Anonymous Authentication
    
  4. Click on the User & Group Settings tab

  5. Configure the User & Group Settings as follows:

    Base DN: ou=Users
    Object Class: inetOrgPerson
    User ID Attribute: uid
    Real Name Attribute: cn
    E-Mail Attribute: mail
    
    Group Type: Static Groups
    Base DN: ou=groups
    Object Class: groupOfNames
    Group ID Attribute: cn
    Group Member Attribute: member
    Group Member Format: ${dn}
    

Setup Admin role

  1. Navigate to https://nexus.example.org/#security-roles

  2. Click Add > External Role Mapping

  3. Configure mapping as follows:

    Realm: LDAP
    Role: lf-collab-admins
    

    Note

    If not an LF project replace lf-collab-admins with the relevant admin group for your case.

  4. Click Add and add the Nexus Administrator Role

From this point you should be able to login using your own account to administrate the server. Do that and then setup admin user email and deactivate the default deployment account as we will create separate project deployment accounts for each individual project.

  1. Navigate to https://nexus.example.org/#security-users

  2. Configure the admin user email to collab-it+PROJECT@linuxfoundation.org

    Note

    Replace email as necessary for your org.

  3. Set the default deployment user account Status to Disabled

Setup custom deployment role

LF projects use Nexus 2 as a server to host logs and requires the Nexus Unpack plugin configured. Since the default Nexus Deployment Role is not configurable, we will have to create our own custom one to ensure Unpack is available.

  1. Navigate to https://nexus.example.org/#security-roles

  2. Click Add > Nexus Role

  3. Configure the following settings:

    Role Id: lf-deployment
    Name: LF Deployment Role
    Description: LF modified deployment role
    
  4. Click Add and add the following roles:

    • Artifact Upload

    • Nexus Deployment Role

    • Unpack

Setup routing

  1. Navigate to https://nexus.example.org/#routes-config

  2. Clear all existing routes

  3. Click Add to add a new route

  4. Configure the route as follows:

    URL Pattern: ^/org/example/.*
    Rule Type: Inclusive
    Repository Group: All Repository Groups
    
    Ordered Route Repositories:
    
      * Releases
      * Snapshots
    

Nexus 3

Setup Server Config

  1. Navigate to https://nexus3.example.org/#admin/system/emailserver

  2. SMTP Settings

    Enabled: true
    Hostname: localhost
    Port: 25
    Username:
    Password:
    From address: noreply@example.org
    Subject prefix:
    

Setup LDAP

  1. Navigate to https://nexus3.example.org/#admin/security/ldap

  2. Click Create connection

  3. Configure the LDAP connection as follows

    Name: ldaps://ldap.example.org:636
    Protocol: ldaps
    Hostname: ldap.example.org
    Port: 636
    Search base: dc=example,dc=org
    Authentication method: Anonymous Authentication
    
  4. Click Verify connection and check that it works

  5. Click Next

  6. Configure the User & Group Settings as follows:

    Base DN: ou=Users
    Object Class: inetOrgPerson
    User ID Attribute: uid
    Real Name Attribute: cn
    E-Mail Attribute: mail
    Map LDAP groups as roles: true
    
    Group Type: Static Groups
    Base DN: ou=groups
    Object Class: groupOfNames
    Group ID Attribute: cn
    Group Member Attribute: member
    Group Member Format: ${dn}
    
  7. Click Verify user mapping and confirm it works

  8. Click Create

Setup Admin role

  1. Navigate to https://nexus3.example.org/#admin/security/roles

  2. Click Create role > External Role Mapping

  3. Configure mapping as follows:

    Mapped Role: lf-collab-admins
    Role Name: lf-collab-admins
    Role description: lf-collab-admins
    Privileges: nx-all
    

From this point you should be able to login using your own account to administrate the server. Do that and then setup admin user email and deactivate the default deployment account as we will create separate project deployment accounts for each individual project.

  1. Navigate to https://nexus3.example.org/#admin/security/users:admin

  2. Configure the admin user email to collab-it+PROJECT@linuxfoundation.org

    Note

    Replace email as necessary for your org.

Post bootstrap

With infrastructure bootstrapped, here is a list of tasks to consider that may be useful to setup.

GitHub

Nexus