Just a few miles down the road from Facebook’s headquarters, Schlumberger’s Software Technology Innovation Center in leafy Menlo Park feels like just another Silicon Valley business with dreams of changing the world.

Walls are covered with Post-it notes with ideas for product features and design principles, and the canteen has bowls of fruit and a shared pizza delivery every Friday lunchtime. Some of the engineers use standing desks and balance boards to exercise while they work. They are drawn from a diverse range of countries and industry backgrounds: one was at Nasa before coming here, another worked for HBO.

Like many successful engineers in northern California, Ashok Belani, Schlumberger’s executive vice-president for technology and the force behind the centre, drives a Tesla to work.

What Schlumberger is doing, however, is far from typical for Silicon Valley: it is working to increase output and cut costs for an activity at the heart of the old economy, oil and gas production.

The technology centre that the oilfield services group has created is a sign of the huge changes under way in oil and gas, as the industry begins to adopt the latest innovations in information technology. Techniques such as advanced data analytics, used by Google, Facebook, Amazon and others mainly to disrupt consumer-facing businesses, are now increasingly being applied to the energy industry. Many oil executives believe the results could be similarly dramatic.

The new opportunities being opened up include analysis of rocks to target wells more precisely in oil-bearing areas, reservoir models to enable production to be maximised through the lifetime of an oilfield, and automation that can make operations safer, more efficient and cheaper.

The increased production made possible by these innovations will put downward pressure on oil prices, creating headwinds for competing technologies including electric cars, and potential difficulties for producers in other countries that are not able to cut their costs the same way. It will also mean disruption for many in the oil industry, with job losses and changes in working patterns and culture.

Matt Rogers of McKinsey, the consultancy, says forecasters have failed to grasp fully the scale of the coming changes. “I don’t think we’ve built into our supply-side models just how much more oil this will provide,” he says. “The world in 10 years will feel very different . . . It’s going to feel like we’re in Star Wars compared to where we are now.”

The oil industry has for decades been at the cutting edge of advances in information technology. John Browne, the former chief executive of BP, began his career at the end of the 1960s mapping oil reservoirs in Alaska using a computer that was the state of the art at the time. On the Top 500 list of the world’s most powerful supercomputers today, the leading private sector owners include Total and Eni, the French and Italian oil groups, and Petroleum Geo Services, a reservoir imaging company.

The difference now is the rise of cloud computing services, which make it possible to store and analyse data at a relatively low cost, opening up possibilities for new applications for a much wider range of companies. The oil industry generates large volumes of data, both structured, such as temperature and pressure readings; and unstructured, such as video footage — and the quantity is growing all the time. The cost of sensors for collecting more data are falling and their sophistication rising, making it possible to monitor more aspects of an operation such as drilling a well.

Bill Braun, chief information officer of Chevron, the US oil group, says the volume of data the company handles has been doubling every 12-18 months. The expansion of its Tengiz oilfield in Kazakhstan, scheduled to start production in 2022, will include about 1m sensors.

Much of the industry’s data are never used, however. “A lot of data are collected, but a lot of it is very isolated,” says Binu Mathew, head of product management for digital at Baker Hughes, the oilfield services group majority owned by General Electric. “Only a small percentage of it is actually being analysed.”

Al Walker, chief executive of Anadarko Petroleum, a US independent oil producer, observed at the CERAWeek conference in Houston last year that although the company recorded a lot of data, “we don’t do that much with it”. He added: “I have terabytes and terabytes of seismic data, and I might use 5 per cent of that.”

Now that is changing. Data that have been collected on engineers’ laptops, or on paper, out in the field, and sometimes copied over by hand, can instead be transferred automatically back to the office. Old records stretching back decades are being converted into coherent data sets. A survey of oil executives last year by Accenture, another consultancy, found 70 per cent of them expected to invest more in digital technologies, with data storage and services the top priorities. It has become a cliché to say that “data are the new oil” — many companies are now looking for ways to turn data into oil.

There has been a spate of alliances between IT companies and oil companies. Microsoft last year signed strategic partnerships with Halliburton, another large oilfield services group, and with Chevron. Nvidia, which makes high-performance chips first used in the video game industry, has been working with Schlumberger, Halliburton and others to adapt its technology for viewing and interpreting seismic information. A year ago it announced a partnership with Baker Hughes on using artificial intelligence to help extract and process oil and gas.

The new technologies are creating business opportunities. Schlumberger, now the world’s largest listed oilfield services group, was founded by two French brothers who in 1926 launched a company to use electrical measurements to map rocks below the surface. Understanding what is happening in oil and gas reservoirs below the surface is still the company’s core competency, but it is increasing the range of services it can provide. Last year it launched a new software system called Delfi, which makes it possible to bring together and co-ordinate the way wells are designed, drilled and brought into production, to maximise output from an entire oilfield.

Mr Belani hopes that by the end of this year, oil companies will be using the technology “on a regular basis” in the US and around the world. In the shale oilfields of the US, the new system could cut production costs by 40 per cent within the next decade, he says.

Other estimates of the impact of digital technologies are more modest. The International Energy Agency, the habitually cautious government-backed watchdog, last year suggested it could cut oil and gas production costs by 10-20 per cent.

But Mr Belani says digitalisation represents a “step change” in the industry’s economics, just like the advances in horizontal drilling and hydraulic fracturing that first made production of shale gas commercially viable, about 15 years ago.

“It might help keep oil prices at the reasonable levels, meaning in the $60s, and let everyone produce economically,” he says.

Like Schlumberger, GE has chosen to put some of its digital operations in the San Francisco Bay area, to develop links with local IT businesses and with Stanford University, and to draw on the region’s pool of talent.

Across the bay from Silicon Valley, in San Ramon, Baker Hughes has a digital technology centre in a building shared with the software operations of its parent GE. It also shares GE’s software platform, Predix, which it uses for applications for oil and gas.

One of the parts of the industry that is moving fastest to grasp the potential of the new technologies has been the shale business in North America. Many thousands of wells are drilled there every year, making it possible to learn from past efforts and quickly apply those lessons. There have been spectacular increases in productivity, driven in recent years by refinements that are sometimes referred to as “shale 2.0”, including drilling longer horizontal wells. But progress so far has largely been a matter of trial and error: experimenting with a new technique and keeping going if it seems to work.

Data analytics offer a chance to make that experimentation more scientific and inject new life into productivity. Some early results have been promising: BP’s chief executive Bob Dudley said last week that it had worked with a Silicon Valley start-up to develop an optimisation model, and had been able to raise production at the 180 wells in its pilot project by 20 per cent.

The potential gains are huge. Companies are recovering only about 8-10 per cent of the oil in place in US shale, so if new systems could raise that rate by even a few percentage points, the results would be dramatic.

There is also a lot of scope for automation in shale. Today there might be 26 people in a drilling rig crew, Mr Belani says. In five years’ time there could be just five.

Mr Braun says that now, “humans are largely driving the drill bit. But we believe that will increasingly be automated and turned over to a computer, just like an autopilot in an aircraft.”

Automation is already on the rise. Schlumberger has started shifting its horizontal drilling experts from out in the field to its Houston offices, from where they can watch over operations for six wells at a time, instead of just one.

Like other sectors disrupted by the latest wave of innovation, the oil industry is facing radical changes in terms of job losses and transformed employment patterns. “It’s going to be uncomfortable for a while,” Mr Rogers says.

Realising the full potential of the new technologies will mean bringing in more people with skills in software and data science. It will also mean overhauling corporate structures. “There’s a lot of problems still to be solved,” says Kausar Qazilbash of Accenture. “It’s not simple: it’s a massive shift in management that requires technological change and operational change.”

If it can be accomplished successfully, the adoption of the new digital techniques will help defend oil and gas against the growing threat from renewable energy, battery storage and electric vehicles. Those new energy technologies are rapidly improving efficiency and cutting costs. But their old rivals in fossil fuels are not standing still.

“We are just scratching the surface of what can be done with artificial intelligence,” says Mr Mathew at Baker Hughes. “This is one of those uses that will change the world.”

New techniques: How oil groups are using computers to cut costs

The oil and gas industry is beginning to adopt new techniques made possible by modern computers that can store and process large and complex data sets. Often these techniques will employ “machine learning”, a form of artificial intelligence that uses algorithms to draw conclusions by studying large data sets. The applications include:

Seismic analysis

Building a picture of rock formations and the location of oil and gas miles below the surface using seismic and other surveys is a highly complex operation. Powerful computers enable companies to understand more about the geology of regions that are difficult to observe, and to predict more accurately where oil can be found.

Production optimisation

The flow of oil from a reservoir depends on a complex range of factors including the length and spacing of the wells and the types of fracturing used. Modern techniques allow companies to maximise the value from a field, sometimes throttling back initial production to achieve a higher ultimate return.

Predictive maintenance

Studying all the data associated with pieces of equipment such as pumps and valves can give indications as to when they are likely to fail, allowing them to be repaired or replaced before they break. The result is better safety and an ability to streamline maintenance schedules.

Automation

The spread of intelligent and connected devices means that processes that traditionally needed skilled staff at a wellhead or processing plant can increasingly be run remotely. The result will be lower labour costs, greater safety with fewer workers in hazardous environments, and increased efficiency.

Security

The energy industry is already a prime target for cyber attacks, such as the Shamoon and Shamoon 2 viruses that hit Saudi Aramco in 2012 and 2016. Increased mobile connectivity, the greater use of operational data and automation create new vulnerabilities, and oil companies need to deploy the most advanced technology to protect themselves.

Copyright The Financial Times Limited 2018. All rights reserved. Not to be redistributed, copied or modified in any way.