• Our clients understandably do not wish for their IBM audit experiences to be shared with the world – even when a great result is achieved (we routinely help to mitigate 90%+ of risk, in some cases up to 100%).
• The worst-case scenario does not materialize, of course, in any audits where we provide support

Each IBM customer develops their own strategy in dealing with audits with varying levels of success. The worst-case outcomes usually occur when the IBM customer fundamentally misunderstands the audit process as well as the incentives for the stakeholders involved. It is rare that IBM audits end in court proceedings, however, a court case involving IBM, a whistleblower (Cimino) and the IRS makes allegations which paint a vivid picture of how an IBM audit process can unfold and the dynamics of any subsequent settlement negotiations. Details of this court case are publicly available, so you can reference it if your senior management need convincing about the necessity of managing IBM license compliance. Also, the case provides many useful lessons on dos and don’ts regarding IBM audits, of which we’ll list a few below. But first let’s summarize what allegedly happened:

(For the avoidance of doubt – ITAA was not involved in this audit in any way)

How an IBM audit led to a court case

In 2012 Deloitte performed an IBM license compliance audit at the Internal Revenue Service (IRS), otherwise known as the US tax authorities. After completion of the audit, IBM and the IRS agreed to a new enterprise license agreement (ELA) of $265 million. In 2013 a whistleblower who formerly worked at IBM decided to sue IBM on behalf of the US government (which is a right every US citizen has). He alleged that the audit findings were fabricated and that they were used to coerce the IRS into buying the new ELA. Following a period of investigation, the IRS decided not to join the whistleblower in progressing this court case. In 2019 the US District Court of Columbia dismissed the complaint. However, following an appeal, this decision has now been partly overturned allowing some of the original claims to proceed.

Given the recent events we felt it timely to examine this case in more detail.  To be clear, we are software license consultants, not lawyers, and it is not our place to speculate or comment on the legal merits of the claims.  Our main interest is the insight the allegations provide into the dynamics of the IBM audit process. The whistleblower (referred to as “Relator” in the quotations below) submitted many pieces of evidence, including copies of internal IBM e-mails, revealing details of what allegedly happened behind the scenes. We have selected a few notable statements to highlight below and have added our perspective to each one.  The quotations are taken directly from the 2019 court decision document but we have chosen to omit names of people whilst the case is still being considered:

An isolated incident?

“According to Relator, the IRS renewed the software license under the threat of a $91 million penalty, which was supported by the false audit findings. After a multi-year investigation, the United States declined to intervene. Relator nevertheless elected to prosecute the action.” (Case 1:13-cv-00907-APM Document 56 Filed 09/30/19 Page 1 of 19)

As we mentioned earlier, it is rare that IBM audits lead to court proceedings and, were it not for the IBM whistleblower, it is likely the alleged events in this case would never have been brought to light. However, a lack of examples in the public domain does not make this an isolated incident. The IRS’s experience of the IBM audit process, as presented in this case, is not dissimilar to the experiences of IBM customers we have worked with.

An audit trigger?

“The IRS was not using all of the products it had purchased from IBM, see id. ¶ 12, and it had begun migrating away from some IBM products to open-source software, see id. ¶ 54. The IRS nevertheless would need to continue using some of IBM’s software for the upcoming tax season, see id. ¶ 70, so it intended to negotiate an extension only for the software that it actually needed, id. ¶¶ 54–56. Relator alleges that “IBM stood to lose significant revenue if the IRS stopped purchasing the software” in the Initial License, a potential outcome which prompted IBM to formulate a plan to pressure the IRS into a new, long-term deal. Id. ¶¶ 57–59.” (Case 1:13-cv-00907-APM Document 56 Filed 09/30/19 Pages 2-3 of 19)

Whenever a customer decides to move away from IBM software it can trigger IBM to initiate a compliance audit. Some customers incorrectly assume that announcing to IBM that they plan to decommission IBM software will help to avoid an audit.

The following sections however are the most concerning from an audit perspective:

Can audits be impartial?

“IBM engaged Deloitte LLP to conduct the audit. Id. ¶ 17. After Deloitte’s audit showed only $500,000 in possible compliance charges—to Deloitte, a result almost unheard of with an entity as large as the IRS, see id. ¶ 74—“IBM suppressed these initial audit results and never released them to the IRS,” id. ¶ 20. Instead, IBM management requested that Deloitte manipulate the audit by basing it on assumptions “that were either without basis or . . . impossible” in order to create leverage over the IRS. Id. ¶¶ 76–77. One way that IBM purportedly drove up overage fees was to have the audit premised on the assumption that licenses deployed on discontinued servers, and thus never used, see id. ¶ 85, were in constant use, see id. ¶ 83. By September 2012, IBM’s changes to Deloitte’s audit assumptions resulted in approximately $18.9 million in overage fees. Id. ¶ 86.” (Case 1:13-cv-00907-APM Document 56 Filed 09/30/19 Page 4 of 19)

Full disclosure – I (the author of this article) formerly worked at Deloitte where I performed IBM license compliance reviews. I have never personally witnessed direct manipulation of audit results by IBM. However, the statements in this court case should, at minimum, prompt IBM customers to carefully consider the incentives for each party participating in the IBM audit process. It is naïve to assume that during an audit the customer perspective will be adequately represented either by IBM or the auditor. The customer must ensure they are confident in their compliance position and the information they are sharing. In almost all cases it is highly recommended to seek external expertise to guide you through the audit process.

Also, it is worth reflecting on the remark that a non-compliance risk of “only” half a million for a customer the size of the IRS would be considered very unusual – the implication being that non-compliance risk would usually be far higher in such a large organization. This aligns with our experiences at ITAA. It is an important caution for customers who feel very confident about their IBM compliance position, especially those with large and complex environments.

Can audit findings be unambiguous?

“In November 2012, IBM changed the audit assumptions yet again—this time, resulting in $292,000,000 in overage fees. Id. ¶ 91. Although [IBM Employee 1] considered the number “ridiculous” and [IBM Employee 2] “‘was not comfortable representing’ that number to the IRS[, . . . the number] was represented to the IRS anyway.” Id.¶¶ 91–92.” (Case 1:13-cv-00907-APM Document 56 Filed 09/30/19 Paged 4-5 of 19)

What is notable here is that the calculation of the findings changed significantly, using the same technical data collected throughout the audit process. In the sections quoted below it will change significantly a few more times. IBM customers often assume that IBM audit reports are unambiguous interpretations of technical data. In fact, audit reports are often based on many decisions and assumptions regarding aspects such as license interpretations, the selection of data sources, and calculation approaches. It is therefore essential to carefully review, and provide feedback on, any draft IBM audit report before it is submitted to IBM.

Can common sense prevail?

“IBM also created an internal audit team (of which Relator was a member) to validate Deloitte’s findings. Id. ¶ 94. Where Deloitte had found $27 million of Rational brand software
over-deployment, the internal audit team found at most $3 million of over-deployment. Id. ¶ 101; see also Am. Compl., Relator’s Exs. 1 & 2, ECF No. 35-1. Unsatisfied, Relator’s supervisor, [IBM Employee 3], and [IBM Employee 1] instructed the audit team to employ impossible assumptions. Am. Compl. ¶ 110. For example, although technically impossible, [IBM Employee 3]
instructed the team to assume that numerous IRS employees were using certain Rational brand “floating user” licenses concurrently—including employees who did not develop software and had no need to use the Rational brand. Id. Eventually, the team came up with $9.3 million in overage fees. Id. ¶ 116. Still too low to create leverage, IBM did not disclose these numbers to the IRS. Id. ¶¶ 117–18.” (Case 1:13-cv-00907-APM Document 56 Filed 09/30/19 Page 5 of 19)

Here again, it appears that very different conclusions are being reached regarding the same environment and data set. In some cases, common-sense observations can help to identify inaccuracies. In this case, the license shortfall being calculated assumed that a massive number of IRS employees used IBM software development software (Rational), whereas only a small subset of those employees were actual software developers. Refuting audit findings does not always involve technical analysis – to some extent you can also identify inaccuracies by applying common sense.

Is pushing back enough?

“On November 29, 2012, IBM presented $91 million in compliance charges to the IRS’s [IRS Employee 1]. Id. ¶ 121. The charges included both overutilized licenses and retroactive technical
support for those licenses. Id. ¶ 120. [IRS Employee 1] again rejected the audit findings. Id. ¶ 122.” (Case 1:13-cv-00907-APM Document 56 Filed 09/30/19 Page 5 of 19)

As this experience shows, it is not sufficient to reject audit findings by stating that they appear to be implausible, untrue, or unreasonable. Audit reports are carefully documented and directly reference supporting (technical) evidence, and even so the resulting conclusions can be incorrect. Any refutation of the audit findings must therefore be equally well-documented, and equally supported by technical evidence. Otherwise, IBM will maintain the stronger position in any settlement negotiations. This is why we consider it essential for IBM customers to have their own IBM audit experts to guide them through the audit process.

Can findings be taken at face value?

“When [IRS Employee 1] was out on vacation in early December 2012, IBM thought “this [was] a good time to keep the pressure on.” Id. ¶ 123. On December 11, 2012, Deloitte presented its inflated findings to [IRS Employee 1]’s superior, the IRS’s Deputy Chief Information Officer, [IRS Employee 2] and others. Id. ¶ 124. Deloitte’s presentation included a spreadsheet that contained a hidden column that, if revealed, would have showed that there was minimal to no usage of the products purportedly overutilized. Id” (Case 1:13-cv-00907-APM Document 56 Filed 09/30/19 Page 5 of 19)

When audits are finalized, IBM often keeps the pressure on to reach a settlement quickly. It is important for IBM customers to resist this pressure and calmly dissect the audit report with help from subject matter experts.

And general advice for everyone – double-check the hidden columns in your Excel sheets before sharing externally!

Is a good deal really a good deal?

“IBM told [IRS Employee 2] that it had retained lawyers to collect the $91 million overage payment, but that IBM would agree to waive the payment if the IRS entered a new contract. Id. ¶ 126. According to [IBM Employee 4], an IBM employee who attended the meeting with [IRS Employee 2], the IRS was “scared” of the Deloitte findings. Id. ¶ 127.” (Case 1:13-cv-00907-APM Document 56 Filed 09/30/19 Page 6 of 19)

Despite the judgement reached by the court to the contrary (see below) – audit reports and audit findings are often used as leverage by IBM to reach a favorable commercial agreement for them.

In 2019, the court decided that, despite the alleged shenanigans (our word choice) that occurred during the IBM audit, they did not believe that those events caused the IRS to sign a commercial deal for licenses they didn’t need. Two key arguments that led the court to this conclusion were that:

• The IRS never accepted the results of the audit. It is therefore not clear that the audit findings were material in their decision to sign a new commercial agreement with IBM.
• After this case was filed, the IRS didn’t participate in the court case on behalf of themselves, and they appeared to have made no effort to reclaim the nearly $90 million they had to pay.

Legal issues aside, the court in 2019 seemed somewhat uninformed on the dynamics of license compliance audits. In particular they did not appear to understand that:

• Organizations that are tricked into making very expensive and unnecessary licenses purchases are usually not keen on broadcasting this to the world.
• Any commercial agreement that is reached straight after an audit is very heavily influenced by the outcome of the audit, even if it does not say so on paper. Denying the link between the two is somewhat odd.

With the recent decision to overturn some of the 2019 rulings we are very interested to see how the next chapter in this on-going case plays out.

What can we learn about IBM audits from this court case?

Now we have reviewed what allegedly happened, what can we learn from this court case about IBM audits? There are countless lessons to be learned, below we have listed a few of our main takeaways:

Be realistic about motives and incentives

IBM is a for-profit company. IBM compliance audit firms are paid by IBM. It is reasonable to assume that financial motives and incentives drive compliance audits.

To be fair, over the years IBM has made attempts to develop a more customer-friendly approach in performing audits. The latest iteration of this is the IBM Authorized SAM Provider (IASP) program, where IBM customers can select one of four IBM partners to perform quarterly compliance assessments. There are potential benefits to this program (such as predictability). However, priorities at IBM can sometimes shift between customer satisfaction and profitability, and the four IBM partners have strong commercial ties with IBM. Our recommendation is therefore to always independently review your compliance position and any reports that are shared with IBM. And keep in mind that the “regular” IBM audits are still commonplace for those who do not sign up to IASP.

Don’t rely solely on legal defense

As the initial judgement in this court case shows, the legal outcome of court proceedings can be hard to predict. In this instance, the outcome doesn’t look good for any of the parties involved. IBM customers are sometimes quick to seek legal support when there is considerable audit risk. In itself there is nothing wrong with seeking legal advice. It is better to seek any support than none at all.

However, it is important to first consider the technical aspects of the audit. IBM compliance managers are well-prepared for any generic legal statements, for example with regard to the sub capacity requirements (which was surprisingly not an issue in this case). By negating audit findings before they reach IBM, a lot of financial risk can be mitigated before (legal) escalation takes place. It is unclear whether and to what extent the IRS performed their own analysis. However, an independent audit report may have revealed findings similar to the initial, undisclosed Deloitte findings worth $500,000 compliance risk. Such a report could have been used to challenge the findings eventually presented to the IRS. Once the audit-related discussions are settled and exhausted, the legal and commercial aspects of the audit results can be considered. By that point most of the financial risk will likely already have been mitigated.

Don’t focus on PVU licenses only

Many customers focus mainly, or only, on PVU-based licenses when measuring IBM compliance risk. This is understandable, as PVU licenses are usually indeed the highest-risk category of IBM licenses due to the requirements of sub capacity licensing. However, as this case shows, non-PVU licenses can also incur significant risk. In fact, Rational Floating Licenses are considered among the lowest-risk IBM license models, because license compliance can be enforced through technical restrictions (floating license key servers) which do not exist for most other IBM products. The $90 million compliance risk for Rational products in this case is very unusual, but for other product categories such large risks can be more common.

Proactively managing IBM compliance is essential

Every organization understands that introducing new software often requires supporting activities in the area of e.g. security, IT infrastructure, privacy. When purchasing IBM software, managing license compliance should be added to the list of essential daily support activities from day one.

Perform a devil’s advocate internal audit

Even though many customers measure IBM compliance as part of their internal Software Asset Management (SAM) processes, the people involved in doing the measurements are often inexperienced and reach conclusions that are too optimistic. When doing an internal review, try taking the “devil’s advocate” approach where you assess your compliance position harshly and uncompromisingly. This will then help to highlight areas for risk mitigation. Undertaking an internal compliance review is a service offered by ITAA.  Please reach out if this is an area where you would like support.

Make sure to fully meet the sub capacity requirements

Actually, this lesson doesn’t result from this court case at all. It appears that the requirements for sub capacity licensing and/or the implementation of the IBM license metric tool (ILMT) were not an issue in this case. However, this is very unusual for any IBM audit with significant compliance risk. If you have licenses based on the PVU metric you can usually assume that the sub capacity requirements are the number one IBM compliance risk area for your organization.

Be careful when retiring IBM software

Many customers consider retiring IBM software as an effective way to mitigate IBM costs and compliance risk. However, sometimes this may be a trigger for IBM to initiate an audit. If you are forced to buy new licenses after the audit, it will be harder to justify retiring IBM software. For this reason, it is usually best not to loudly communicate it to IBM when you intend to retire their software. And when you reduce your S&S fees, make sure you are prepared for the (almost) inevitable audit.

Don’t rush the audit process, especially at the end

Until you sign the final audit settlement deal, nothing is set in stone. The IRS was smart in not accepting the audit results given the disputed audit findings, although oddly it backfired in the 2019 court case decision as the judge considered it as evidence against causality. Ideally the IRS should have withstood the pressure to finalize the audit and spent more time to refute the findings and present their own perspective.

Ask for help

As you would expect from a company specialized in IBM audit defense, we strongly recommend asking for external support to manage any IBM audit. However, we do not intend to sell our services when they are not needed. We are always available to discuss your specific circumstances, in confidentiality and without obligations. Reach out to our IBM vertical lead Koen Dingjan today to discuss any IBM audit concerns you have, regardless of the stage of the audit you are in.

We have written a separate article about what to expect from an IBM Software License Review if you are interested in reading more on this subject.