Skip to main content

Security Vulnerability Handling

Introduction

Problem Statement

The Oracle Cloud Native Core software currently provides six planned releases per year (program increments), and each release will contain a mix of features and fixes.   In each release we typically move 3rd party software to the latest release in order to stay abreast of the latest security patches for our embedded 3rd party software.  This can lead to an two to three month lag for the release of critical security vulnerabilities, which isn't acceptable for critical fixes.     And as we typically only address vulnerabilities on the tip of the release under-development, a customer who has not yet moved to the latest release would need to upgrade to pick up a critical security fix.

Currently we only scan products that are under development; once released (posted to OSDC or MOS) we do not go back and rescan.   Because 3rd party vulnerabilities may be found after we have released our software,  a customer might be the first to discover that the Oracle provided software has a known vulnerability.

Finally, some customers are requiring that we agree to contract terms with very aggressive security patch turn-around times.   

In response, the CGBU has developed guidelines for all cloud native/on-prem products and in recent contract we have started to commit to providing patches with these very short turn around times.

Overview

This document describes the new vulnerability handling guidelines, the new process changes being introduced to support faster turn-around times, and discusses the new SW owner responsibilities.  In summary, this document discussed these process changes:

  1. We will introduce new tooling and processes to allow us to track and continuously scan all released products for new vulnerabilities
  2. The new tooling with come with improved knowledge bases which should help reduce vulnerability analysis costs
  3. Each NF must support a SW patch process allowing for the urgent release of critical security fixes on the latest published release

Vulnerability Handling Guidelines

Summary

This section summarizes the Oracle Communications Security Maintenance Obligations.


 Compliance

Compliance

 Vulnerability Handling

Vulnerability Handling

  • We will regularly scan:

    Asset
    Frequency
    Notes
    Native CodeEach PI

    Oracle developed native code each release cycle (using a wide range of scanners)

    Oracle StackContinuouslyOracle developed applications used in the solution (e.g., Oracle Linux, MySql, etc.)
    3rd Party CodeContinuously3rd party code integrated with or directly used by the native code
    External ReportsOn DemandVulnerabilities reported via support channel
  • We will assess scoring using CVSS V3.1 (or newer)
    • Oracle may adjust scoring to be higher or lower based on contextual risk assessment

  • We will provide vulnerability remediation public or externally reported vulnerabilities per this table:

    Vulnerability Base Score (CVSS 3.1)
    Time to Provide a temporary fix or workaround
    Time to provide official security patch
    7.0 - 10.0 (confirmed by Oracle Support)7 calendar days30 calendar days
    0.0 - 6.9 (confirmed by Oracle Support)not applicable6 calendar months
    • As soon a vulnerability is patched and publicized, the clock starts
    • In the case of a vulnerability being reported against our native code, the clock starts once support has reproduced the issue
 Oracle Guidelines

Oracle Guidelines

Oracle will (may):

  • not modify 3rd party code to patch 4th party dependencies
  • only disclose vulnerabilities once a patch is available
  • provide fixes on the tip of a release stream (e.g., when we make a fix on 1.4, you will get all of the post-release 1.4 fixes - no cherry picking)
  • disclose summary information on internal scans (pass / fail rates)
 Customer Guidelines

Customer Guidelines

Customers may:

  • Customers may audit our procedures
  • Customers may scan our distributed product

Process Assessment

What this means to us:

  • We must introduce continuous scanning of released products (to ensure that we identify triggering events in a timely fashion)
  • We must be efficient in our vulnerability assessment (as we are going to do them more often)
  • We must be ready to patch (a vulnerability in release 1.X will be fixed on the tip of 1.X - not on the release under development)

Release Cadence

The following diagram shows the new release cadence plan - a new alternate month urgent release (ur) is slated to support security patching:



With this new cadence, we will:

  • Report the last 3 releases in any given CPU notice
    • For example: the Jul CPU will include notification of security vulnerabilities fixed in PI-B, urPI-B, and PI-C
  • Have at most 1 patch release for any given release
    • For example: the PI-C release will only be patched in the urPI-C release
    • Any critical security bugs identified within 30 days of a planned PI release should be included in that PI release
    • If no critical vulnerabilities need fixing, there will be no patch release 


Vulnerability Handling Process Flows

The following diagram shows the process flow for vulnerability handling in Oracle Native Code.   This shows a periodic scan, but the flow is similar for an on-demand scan driven from the CI/CD pipeline.   Flows for non-containerized software (e.g., linux RPMs) are similar although different scanning tools might be employed.


Threat Detection

The first step of threat detection is to perform a structured design analysis in the form of an Architectural Risk Assessment.  The ARA will identify the threat landscape and highlight the interfaces than might need special scanning.  We typically try to detect potential threats in our native code through the use of static scanning (fortify) and dynamic scanning (fuzzers, port scanners).    We also scan to ensure that the 3rd party packages used in our solution are vulnerability free.  We check for 3rd party vulnerabilities both at build-time and at deployment time using a OWASP, OpenSCAP, and Anchore-Engine scanners.    

At the end of all of this, we end up with a list of possible vulnerabilities (some identifies by ARA and some identified via scanners).

Vulnerability Analysis

Once we have a list of possible vulnerabilities, we score the vulnerability using the CVSS 3.1 scoring system.  This gives the threat a numerical score.  Sometimes our vulnerabilities are pre-scored (3rd party vulnerabilities).  For these we look at the threat in the system context and we might adjust the CVSS scoring higher or lower.   For example,  if a vulnerability is only exploitable in the windows environment and we only deploy on Linux, we would rescore this to zero.  Similarly if the vulnerability is on exploitable at a certain kernel revision level, and we know the software is running on a patched kernel, we would rescore to zero.  Alternately,  if a minor vulnerability can have an overly large system impact, we would adjust the score higher. Regardless of the resulting score we will track the vulnerability and await a fix.  

Vulnerability Tracking

For any vulnerability that has been re-scored to a zero, we would open simple bug story to track that we are waiting on a third party update to resolve the vulnerability.  If the vulnerability was identifies in fortify scanning, we would adjust the fortify analysis to report "not and issue".   For any non-zero vulnerability we will open a BUGDB bug (regardless of severity).  The BUGDB bug should have the vulnerability flag set.

Software Patch Release

For serious vulnerabilities (CVSS score > 6.9) a patch release must be created for all release streams affected by the bug.   For example, if the 3rd party package Kubernetes is found to have a serious 7.5 CVSS vulnerability, we would need to patch OC-CNE to pick up the latest Kubernetes on the supported release stream.

3rd Party Out of Support Window

If there is no 3rd party release available on the older release stream that will address a known vulnerability (and you believe the vulnerability is present) contact your Security Spoc or Security Lead for guidance.   We might need to move existing customer to newer releases (via upgrade) where the fix appears, or move to newer 3rd party release streams in the released product (risky).


Oracle Critical Patch Update (CPU) Process

The CGBU security processing is for the most part more aggressive than the Oracle CPU requirements; we should be able to fully comply with the Oracle CPU guidelines.   The one area where we might have a gap is in the handling of 4th (5th, etc) party vulnerabilities.  The CGBU process focuses on providing fixes to 3rd party SW in a timely fashion.  If a vulnerability is found in 4th party software used by our up to date 3rd party software, we are under no time commitment to fix this independently of the 3rd party SW provided.  This could lead to a bit of friction if the 3rd party provider is slow to pick up fixes to their dependent software packages, and could lead us to become non-compliant to the Oracle CPU guidelines.  This doesn't happen very often, and we will accept this risk (and will work with 3rd party provider and with OSSA to resolve).

Comments

Post a Comment

Popular posts from this blog

Supporting OpenTracing jaeger in spring boot applications

This page describes code changes in a typical spring-boot based application to support OpenTracing and jaeger. Instrumenting a tracer If you are creating a simple spring boot application that uses  spring-boot-starter-web , by default, the application does not support writing traces to jaeger. To support jaeger tracing, the first thing is to modify the build.gradle to add dependency of jaeger: dependencies {      implementation  'org.springframework.boot:spring-boot-starter-web'      implementation  'io.opentracing.contrib:opentracing-spring-web-starter:3.0.1'      // support opentracing jaeger      implementation  'io.opentracing.contrib:opentracing-spring-jaeger-starter:3.1.2'      testImplementation( 'org.springframework.boot:spring-boot-starter-test' ) {          exclude group:  'org.junit.vintage' , module:  'junit...
Post a Comment
Read more

HOWTO on implementing scanning into a CI Pipeline

Introduction As a part of the Software Security Assurance guidelines, we are required to perform various types of security scanning.  Security scanning is a specialized kind of testing that attempts to identify potential software bugs that could make a deployed application vulnerable to attack and compromise.  As with any testing, the tests can be manual or automatic, and they wan be performed at various points in the development cycle.   We can classify security scanning into three general categories: Static Application Security Testing (SAST)   - performed against source code / configuration data a static scan looks for common logic errors that might lead to system compromise. Dy namic Application Security Testing  (DAST)  - Performed against a running system, a dynamic scan looks for vulnerable software / configurations that might lead to system compromise. Security Monitoring  - Deployed as a part of the system, a security monitor co...
Post a Comment
Read more

Fortify Tooling User Guide

  Introduction The fortify-tools container is located within a shared repository in OCIR and requires a JWT to be able to access.  The variable WF_JWT will need to be set to a valid MAT You will want to choose one of three ways to use the Fortify SCA tooling: Integration Description Using the Fortify Tools Standalone to Scan a Project This is for using the tooling without integration in GitLab CI or Jenkins CI. Using the Fortify Tools In GitLab CI to Scan a Project This is for using the tooling against a project that whose code is hosted in GitLab and whose CI engine is GitLab CI. Using the Fortify Tools In Jenkins CI to Scan a Project This is for using the tooling against a project that whose code is hosted in GitLab and whose CI engine is Jenkins CI. Using the Fortify Tools Standalone to Scan a Project Simple Usage Run the Fortify Tools in a container docker run -t --rm -v <path to project source root directory>:/var/fortify/src phx.ocir.io/oraclegbudevcorp/cn-shared/s...
Post a Comment
Read more