Nexus

Nexus is an artifact repository typically used in Java / Maven projects. Stores Project artifacts, Javadocs, and Jenkins job logs.

File system layout

We recommend to configure the Nexus server storage for all artifacts and logs on separate file systems, preferably a file system that allows a large amount of inodes such as XFS for the logs storage.

/srv:

Contains Nexus install along with storage repositories.

/srv/sonatype-work/nexus/storage/logs:

Contains Jenkins server logs. Use a file system with a lot of inodes.

Note

OpenDaylight ran out of inodes before due to logs. Issue documented in Jira https://jira.linuxfoundation.org/browse/RELENG-773

Scheduled Tasks

We recommend configuring Nexus to clear out old SNAPSHOT artifacts as well as old Staging repositories. Some projects may have specific policies set by the TSC on how long artifacts need to stick around but below make a good starting point.

Purge old SNAPSHOTs

For purging SNAPSHOTs we should setup 2 jobs.

The first job to purge week old artifacts but keep 1 SNAPSHOT around in case the project has a broken merge job.

The second job to purge all 3 week old artifacts. This is necessary is to ensure that if a project removes a module from their build that downstream projects will notice by fact of their builds failing to find this artifact.

  1. LF: Purge week old SNAPSHOTs

    Name: LF Purge week old SNAPSHOTs
    Task Type: Remove Snapshots From Repository
    Repository/Group: Snapshots (Repo)
    Minimum snapshot count: 1
    Snapshot retention (days): 7
    Remove if released: True
    Grace period after release (days): 21
    Delete immediately: True
    Recurrence: Daily
    
  2. LF: Purge 3 week old SNAPSHOTs

    Name: LF Purge 3 week old SNAPSHOTs
    Task Type: Remove Snapshots From Repository
    Repository/Group: Snapshots (Repo)
    Minimum snapshot count: 0
    Snapshot retention (days): 21
    Remove if released: True
    Grace period after release (days): 21
    Delete immediately: True
    Recurrence: Daily
    

Purge old staging

Name: LF Purge old staging
Task Type: Drop Inactive Staging Repositories
Inactivity duration (days): 30
Scan open repositories: True
Scan closed repositories: True
Scan promoted repositories: True
Scan released repositories: True
Recurrence: Daily

Purge trash

Name: LF Purge trash
Task Type: Empty Trash
Repository/Group: All Repositories
Recurrence: Daily

Rebuild metadata

Name: LF Rebuild metadata
Task Type: Rebuild Maven Metadata Files
Repository/Group: All Repositories
Recurrence: Daily

Use Nexus as a log server

One use for a Nexus server is to be a log server for Jenkins. This is useful to offload logs from Jenkins and allow Nexus to store the longer term storage of the logs.

We suggest following advice from the File system layout <nexus-file-system> section before configuring the log server directory here.

Create log repository

  1. Navigate to https://nexus.example.org/#view-repositories

  2. Click Add > Hosted Repository

  3. Configure the repository as follows:

    Repository ID: logs
    Repository Name: logs
    Repository Type: hosted
    Provider: Site
    Format: site
    Repository Policy: Mixed
    
    Deployment Policy: Allow Redeploy
    Allow File Browsing: True
    Include in Search: False
    Publish URL: True
    
  1. Navigate to https://nexus.example.org/#security-privileges

  2. Click Add > Repository Target Privilege

  3. Configure the privilege as follows:

    Name: logs
    Description: logs
    Repository: All Repositories
    Repository Target: All (site)
    

Create log role

  1. Navigate to https://nexus.example.org/#security-roles

  2. Click Add > Nexus Role

  3. Configure the role as follows:

    Role Id: All logs repo
    Name: All logs repo
    Description:
    
  4. Click Add and add the following privileges:

    • logs - (create)

    • logs - (delete)

    • logs - (read)

    • logs - (update)

    • logs - (view)

    Note

    Be careful not to include the “Logs - (read)” (the one with the capitalized first letter) this one is for granting access to Nexus’ own logs.

  5. Click Save

Create log user

  1. Navigate to https://nexus.example.org/#security-users

  2. Click Add > Nexus User

  3. Configure the user as follows:

    User ID: logs
    First Name: logs
    Last Name: user
    Email: jenkins@example.org
    Status: Active
    
  4. Click Add and add the following roles:

    • All logs repo

    • LF Deployment Role

Configure log credential in Jenkins

  1. Navigate to https://jenkins.example.org/credentials/store/system/domain/_/newCredentials

  2. Configure the credential as follows:

    Kind: Username with password
    Scope: Global
    Username: logs
    Passowrd: <password>
    ID: jenkins-log-archives
    Description: jenkins-log-archives
    
  3. Navigate to https://jenkins.example.org/configfiles/editConfig?id=jenkins-log-archives-settings

  4. Click Add to add a new Server Credential

  5. Configure the credential as follows:

    ServerId: logs
    Credentials: jenkins-log-archives
    
  6. Click Submit

Configure global-var in ci-management

  1. Edit the file jenkins-config/global-vars-production.sh

  2. Add LOGS_SERVER=https://logs.example.org as a new global-var

  3. Repeat for all global-vars files as necessary

Refer to Jenkins CFG Global Variables for details on global-vars configuration.

Setup cron to cleanup old logs

We highly recommend setting up cron jobs to cleanup old logs periodically.

  1. Job to clean up files 6 months old on production path every day

  2. Job to clean up empty directories in the logs path every day

  3. Job to clean up all sandbox logs every week

The following example shows the puppet-cron configuration used by LF to manage logs following the Jenkins Sandbox rules defined in the Jenkins Sandbox Overview.

puppet-cron example
cron::daily:
  purge-logs-production:
    hour: 8
    user: 'nexus'
    # yamllint disable-line rule:line-length
    command: '/usr/bin/yes | /usr/bin/find /srv/sonatype-work/nexus/storage/logs/production -mtime +183 -delete 2>/dev/null'
  purge-empty-dirs:
    hour: 9
    user: 'nexus'
    # yamllint disable-line rule:line-length
    command: '/usr/bin/yes | /usr/bin/find /srv/sonatype-work/nexus/storage/logs -type d -empty -delete 2>/dev/null'
cron::weekly:
  purge-logs-sandbox:
    hour: 8
    weekday: 6
    user: 'nexus'
    # yamllint disable-line rule:line-length
    command: '/bin/rm -rf /srv/sonatype-work/nexus/storage/logs/sandbox/*'

Create Nexus2 repos with lftools

LF Tools provides an interface to Nexus 2 for creating resources or reordering staging repositories. More information on how to use the commands: LF Tools Nexus commands

The lftools nexus create repo command needs two files as parameters:

  • -c, –config Configuration file containing the repos and their tree structure.

    # Using ONAP as example
    
    base_groupId: 'org.onap'
    email_domain: 'onap.org'
    global_privs:
      - 'LF Deployment Role'
    repositories:
     appc:
       password: 'NjPAd1ZZ5RbDalZy4ROHaApb4Bk3buTU'
       extra_privs:
         - 'Staging: Deployer (autorelease)'
       repositories:
         cdt:
           password: 'NjPAd1ZZ5RbDalZy4ROHaApb4Bk3buTU'
           extra_privs:
             - 'Staging: Deployer (autorelease)'
     aaf:
       password: 'NjPAd1ZZ5RbDalZy4ROHaApb4Bk3buTU'
       extra_privs:
         - 'Staging: Deployer (autorelease)'
       repositories:
         sms:
           password: 'NjPAd1ZZ5RbDalZy4ROHaApb4Bk3buTU'
           extra_privs:
             - 'Staging: Deployer (autorelease)'
    

appc is the parent for cdt and aaf is the parent of sms. The projects created will be: appc, appc-cdt, aaf and aaf-sms.

Note

‘Staging: Deployer (autorelease)’ in the above example is in the extra_privs section as an example. If it applies to all repos, it can be part of the global_privs section.

  • -s, –settings Configuration file with all the admin settings

    # Using ONAP as example
    
    nexus: 'https://nexus.onap.org'
    
    user: 'admin'
    password: 'admin123'
    

After running lftools nexus create repo -c <the_repo_config> -s <your_settings_config>, the script will create all repos, users, roles and privileges. Also, the Repository Targets gets set with the patterns to set restrictions for projects and the location where they should post artifacts. These patterns should match the GroupId in the project’s pom.xml.

Troubleshooting

SSL certificate does not match due to SNI

When using the nexus-staging-maven-plugin and the build fails with the message below. This is due to Nexus 2 not supporting SNI and prevents the staging plugin from uploading artifacts to Nexus.

The workaround for this is to use another method to upload to Nexus such as cURL which is capable of ignoring the failure.

Error

[ERROR] Failed to execute goal org.sonatype.plugins:nexus-staging-maven-plugin:1.6.8:deploy-staged-repository (default-cli) on project standalone-pom: Execution default-cli of goal org.sonatype.plugins:nexus-staging-maven-plugin:1.6.8:deploy-staged-repository failed: Nexus connection problem to URL [https://nexus.opendaylight.org ]: com.sun.jersey.api.client.ClientHandlerException: javax.net.ssl.SSLException: hostname in certificate didn’t match: <nexus.opendaylight.org> != <logs.opendaylight.org> OR <logs.opendaylight.org> -> [Help 1]

Refer to https://jira.linuxfoundation.org/browse/RELENG-21 for further details.