Alert
AL00343 - Notification of Data Breach at OSIsoft and Next Steps
2018-11-28
November 28, 2018

You may have seen a notice posted by OSIsoft in accordance with privacy norms for an internal data breach at OSIsoft. To OSIsoft’s knowledge, no customer accounts or information have been affected and this communication is primarily to provide transparency and information.

Stolen credentials were used to remotely access internal OSIsoft computers. Our security service provider recovered direct evidence of credential theft activity involving 29 computers and 135 accounts and we have concluded that all internal OSI domain accounts were affected. People with affected OSI Active Directory accounts have been notified and informed about what they should do. Note that customer accounts, such as those for the Technical Support Site and PI Square, are not affected by this incident.

Since learning of the breach, we are continuing to take steps to strengthen safeguards protecting our customers, the PI System community and OSIsoft. We have been engaged with threat detection and intelligence firms to review our systems for any unusual and unlawful activity. In this open-ended engagement, the firms will monitor activity and advise us on security strategies for the foreseeable future. We will also continue to disclose information to you per our Ethical Disclosure policy.

OSIsoft products are developed using a security development lifecycle process that includes human and automated code reviews in which products are checked for malicious code before digital signing. Based on what we have learned thus far, the attack did not affect our products or services, however, it did affect IT access safeguards. As such, we are performing a supplemental code review as an extra check to ensure the integrity our products and services.

At this time, there are no indicators of activity affecting customers and we are not recommending any action to our customers.

Security has always been and continues to be a high priority for our company and customers. We do not take incidents such as this lightly. We will strive to resolve this matter and keep you informed.

The official report can be obtained from the California State Attorney General at https://oag.ca.gov/ecrime/databreach/reports/sb24-141865. If you have concerns, we encourage you to contact OSIsoft Technical Support at techsupport@osisoft.com.

Security FAQ

What occurred?

A skilled threat group gained remote access to computers at an OSIsoft branch office, which expanded to the OSIsoft headquarter business network in mid-2018.  In response, intrusion detection sensor alerts were escalated and OSIsoft began a forensic investigation with our security service provider. By Q4 2018, we were able to characterize the activity as credential theft and filed a breach report in accordance with California privacy norms.

Who is responsible for this attack? 

OSIsoft does not know who the threat group is. Our security service provider and OSIsoft believe it is a skilled group seeking information.

How did the threat group get in?

We suspect phishing with a malicious URL or attachment, but we are not certain. Our uncertainty is because the unlawful activity traces to a computer that was already re-purposed before the attack was discovered.  Analysis provided by our email filtering service is consistent with targeted phishing as the initial attack vector.

What took so long to investigate?

Forensic analysis shows the threat group was largely dormant, surfacing every few months.  Additional instrumentation and traps were set to identify the source of intrusion and access mechanism. The threat group didn’t resurface during months of investigation. OSIsoft promptly filed a breach report on receipt of the forensic report. 

Was customer information compromised?

We do not have any evidence that information important to customer operations has been compromised. OSIsoft maintains a ‘data minimization’ policy with regard to customer operational data. Data important to customer operations is typically not sent to OSIsoft. OSIsoft has also seen no evidence of data dumps or abuse of OSIsoft accounts nor have we seen evidence that customer data was stored on any affected computers. Nonetheless, this is a legitimate concern and we will continue to look for any evidence of unlawful activity and continue to improve our safeguards.

Is anyone able to use the credentials to remotely access customer networks?

We have no evidence that this occurred and believe it is extremely unlikely.  OSIsoft primarily uses Bomgar, a secure remote access solution, for accessing customer networks.  The first step in establishing remote access is to authenticate to Bomgar using multifactor authentication. For attended sessions, customers present an access key to join the remote access session.  Some customers allow unattended sessions, in these cases customer network access credentials are stored in a separate vault unaffected by the credential theft activity.  Finally, at the end of each Bomgar session a comprehensive recording of the session is made available to you in support of audit requirements you may have. OSIsoft maintains the Bomgar audit trail for 90 days.
 
We also support remote access sessions using WebEx. Only attended mode sessions initiated by customers are supported when using WebEx.

Are OSIsoft websites safe to use?

We have not seen evidence of malicious activity on OSIsoft websites nor have we seen evidence of implants on OSIsoft web sites.

Has the source code of OSIsoft products been tampered?

We have no indications of threat activity with the code repository or our products. The possibility of implants in code evading our remaining safeguards is under investigation. These safeguards include project level access control; workflow enforcement policy, notifications on code changesets and other development activities. Supplemental reviews are underway. Preliminary results confirm all code changes are genuine.
 
Furthermore, OSIsoft products are developed using security development lifecycle process which includes human and automated code reviews.  Products are checked for malicious code before digital signing. OSIsoft is conducting further analysis and will inform customers in accordance with our Ethical Disclosure policy.

What is OSIsoft doing in response?

We are taking several actions:
  • All affected systems have been taken offline and replaced or upgraded
  • Access entitlements have been reviewed.  
  • Parties with affected Active Directory accounts have been notified and informed about what they should do.   
  • Multifactor authentication security solutions have been deployed to all remotely accessible services for all users to shield OSIsoft assets from unauthorized use.   
  • OSIsoft implemented additional email defenses for blocking and tracking of links. 
  • OSIsoft is reviewing incident response capability including expanded security monitoring across all global offices and continues to use security service providers to monitor internal and external endpoint security posture.
  • OSIsoft has engaged with threat detection and intelligence firms to review our systems for any unusual and unlawful activity. In this open-ended engagement, the firms will monitor activity and advise us on security strategies for the foreseeable future.
  • OSIsoft is also examining the underlying code of its products to confirm integrity and development safeguards remain intact.
  • We will update you on critical information in accordance with our Ethical Disclosure policy.

Is there any evidence that the compromised credentials of individuals were used to phish customers or falsify invoices?

There are no reports of scams targeting customers with OSIsoft credentials or appearing to do so.

Can you provide a method for us to validate the authenticity and integrity of OSIsoft software?

Software from OSIsoft is protected by an Authenticode digital signature. The Authenticode signature is applied to each installation package as well as each product module built by OSIsoft. You can verify the authenticity and integrity of a file in Windows by selecting ‘Details’ on the Digital Signature tab of the file properties dialog popup (right click on the file).  Look for ‘This digital signature is OK’.
 
While handy for a file you just downloaded, manually checking file properties can be tedious for files on a system that is already deployed. We recommend use of a Powershell script to verify files in bulk.

PS > Get-ChildItem C:\WorkingFolder\ | Where-Object {$_.Extension -in @(‘.exe’,’.dll’,’.msi’)} | Get-AuthenticodeSignature

Can we verify the hashes of the software that has been downloaded since the breach to make sure it wasn’t maliciously altered?

Software hashes are embedded within Authenticode signatures. Validation of the Authenticode digital signature as described above provides assurance a file is produced by OSIsoft and has not been altered.

Is it advisable to change our account passwords at OSIsoft we use to logon to tech support?

No, customer accounts such as those for the Technical Support Site and PI Square are not affected by this incident, so it is not necessary for you to change your password.
However, some customers have taken this opportunity to review entitlements to access their technical support cases and product downloads.  Unused or undesired customer accounts can be removed by request to privacy@osisoft.com.

Where can I find more information?

The official report can be obtained from the California State Attorney General. If you have concerns, we encourage you to contact OSIsoft Technical Support at techsupport@osisoft.com.
 
Official Breach Disclosure: https://oag.ca.gov/ecrime/databreach/reports/sb24-141865

OSIsoft also issued a Technical Support alert as an explicit clarification of the scope of the issue.
 
AL00343 - Notification of Data Breach at OSIsoft and Next Steps: https://techsupport.osisoft.com/Troubleshooting/Alerts/AL00343

OSIsoft also conducted a webinar with the Electricity Information and Sharing Agency (E-ISAC), a division of the North American Electric Reliability Corp. that gathers and analyzes security data, shares appropriate data with stakeholders, coordinates incident management, and communicates mitigation strategies with stakeholders. You can review the webinar by registering at E-ISAC's web site.

Can you provide Indicators of Compromise (IoCs)?

Yes, to assist the larger community in threat hunting, we are working with E-ISAC to vet and safely share IoCs from this incident. 

Revision History:

2018-11-28 – Initial posting.
2018-12-12 – Added Security FAQ section.
2019-01-03 – IOCs should now be available for E-ISAC members on E-ISAC portal.  
2019-01-24 – Next scheduled update.