Automated drilling algorithms implementation on a laboratory drilling rig

Considering the state of art technology that exists today and the significant resources that are being invested into the technology of tomorrow, an idea of intelligent and fully automated machineries working on a drilling floor that is capable of consistently selecting best decisions or predictions based on realtime information available and providing drillers and operators with such recommendations, becomes closer to a reality every day. This work shows results of the research carried out on the topic of drilling automation. Its objectives are to design and test proof of concept technologies conducted on a laboratory-scale autonomous drilling rig developed at University of Stavanger, Norway. Main contribution of the study is on drilling speed (ROP) optimization with considering operational safety to personnel and environment (HSE) and drilling efficiency along with a digitized drilling program for directional drilling. The case studies are presented to show the different scenarios for drilling vertical wells and inclined wells.


Introduction
In recent years, the concept of drilling automation has advanced from primarily being automation of rig floor equipment to novel solutions that can be rapidly deployed to the rig environment and assist drillers in a variety of operations. Aside from providing an early warning to drillers, and if necessary, for instance performing a controlled shut-in procedure once a kick is migrating towards surface that gets detected, such intelligent systems could improve efficiency and reduce financial costs through continuous monitoring and interactions with drillers and supervisors. Smart and digital drilling systems could also be used to suggest operational parameters to drillers through correlating real-time drilling data with vast amounts of historic data stored in a virtual environment, popularly referred to as a digital twin, or even exert full control of all rig equipment if permissible (top drive, drawworks, mud pumps, elevator, rough neck and so on) leaving only major decision points to be determined by drillers. The latter automation level described for the drilling scene is likely still several years away from being deployed to fields. A timeline that highlights artificial intelligence applications in drilling practices is given in Bello (2015). In our opinion, short term advances in drilling automation lie in developing digital and robust tools for drillers to strengthen the understanding of the operations during critical phases, and if possible, automate some of the routine tasks that can be easily controlled by machines. In the same way that a Tesla can self-operate in known terrain, so should a drilling rig, enabling drillers to remain one step ahead, maintaining an overview of all on-going events. The study covers some of the topics including software framework design for drilling automation system, control framework implementation, autonomous search algorithms development that allows our control system to maximize the rate of penetration (ROP) while min-imizing the risk of running into drilling incidents and drilling with sub-optimal parameter setpoints, and a digital drilling algorithm proposed that has been implemented to autonomously drill a deviation well using a closed-loop steering system. Finally the results and discussions are given to show the performance of our autonomous drilling rig.

Drilling Rig
The drilling rig, see Figure 1, consists of in total four hardware systems, in addition to the control system. These are: rotation, hoisting, circulation and power systems that are introduced below. Rotation System: There exist two rotational systems: a conventional top drive, used for vertical drilling and a downhole motor used for directional drilling. This set-up allows us to circulate drilling fluid (either water, water-based mud or oil-based mud ) all the way from the top of derrick to the drill bit nozzles. The top drive provides rotational speeds up to 1500 RPM. By varying two analog voltage signals one can control the RPM and torque from the top drive. The signals are transmitted from programmable logic controllers (PLCs) to a driver for the top drive. Hoisting System: In order to simulate drilling operations, the system is equipped with a hoisting system that consists of three actuators, each equipped with a stepper motor. The motors raise/lower the top plate that resembles the drill floor where the top drive and other components are mounted. The three actuators are controlled by its own stepper motor with a stepangle of 1.8 degrees, where for each 1.8 degrees of stepping 10 micro-steps get transmitted (resulting in a total of 2000 steps/rev). High accuracy for the actuators is a key in order to ensure the required weight on bit (WOB) control. Such powerful stepper motors can provide an approximate maximum WOB up to 500 N. To address the challenge of vibrations in the system, a rigid coupling is used between the stepper motors and actuators to reduce the overall vibrations making the system more sturdy. More information and details are found in Løken and Trulsen (2017), Hagen et al. (2018), Hjelm and Nilsen (2018). Circulation System: The circulation system consists of two pumps alternating for 30 seconds each. This prevents the pumps from over-heating and should one pump fail, the other will kick in to ensure the required velocity margin for adequate hole cleaning and cooling of the bit. The pumps have a maximum flow rate of 19 L/min at a working pressure of 3.1 bar, see Hagen et al. (2018). The drill string assembly consists of: aluminium drill pipe, downhole sensor sub, knuckle joint, pneumatic   compressed (no bend), the mechanical angle of the top drive can be varied through pulsing to change the azimuth, if an offset is registered. The bit RPM is controlled by throttling the air flow coming from a compressor to the downhole pneumatic motor. The aluminium drill pipe used for drilling has the following dimension: Pipe limit calculations and destructive testing of the drill pipe are given in Hagen et al. (2018), which are summarized below: • critical slenderness ratio of the pipe is 70.2, and slenderness ratio of the drill pipe is approximately 280.2, • buckling limit is 280.5 N, • maximum torque before the pipe yields is approximately 19.0 Nm, while the maximum torque before the pipe shears is approximately 24.7 Nm, • maximum tensile load is approximately 10.331 kN and maximum compressive load is approximately 985.72 N.

Control System Architecture
The drilling system is a collection of several systems including mechanical, hydraulic, electrical, hardware, software, and data systems. For our system, the hardware architecture is comprised of three levels: PLCs on level 1, computers on level 2 and cloud-based storage of time-based data (optionally also depth-based data) on level 3. On level 1, the control aspect (PLCs) for the main systems on the rig are located to control rotational system, hoisting system and hydraulic mud system. Each system gets controlled through its own microcontroller (PLC), that both executes commands to the controllable systems on the rig (actuators, motors, pumps and so on) and gathers information from the equipped sensors. On level 2, PCs are located to handle the decisionmaking logic. This logic involves carrying out the digital detailed operating procedure (DDOP), see Section 6.3 for detailed information, ROP search algorithms (see Section 6.1), steering algorithm (see Section 6.2), detection of drilling incidents and carrying out remedial actions, as well as pushing the data to a configured database for post-analysis and data extraction (level 3). A key factor has been to ensure that all components that make up the drilling system are compatible for real-time control. The implemented control system dataflow with PLCs is illustrated in Figure A.1. The PID controller is used controller in the closed-loop systems as: where u 0 is the bias, u(t) is the error between true measurement and the setpoint, K p is the controller gain for proportional controller, K i is the controller gain for integral controller and K d is controller gain for derivative controller. When drilling on the laboratory rig, the hoisting system controls the WOB to drill the formation. It is important that the selected WOB setpoint is constant during drilling in order to keep a steady drill rate, and the use of a PID controller ensures that the error between the WOB setpoint and measured hook load is minimized, seeing that the WOB is calculated as the difference between free hanging weight and hook load. The K p , K i and K d gains can get defined and upper and lower thresholds allow the operator to adjust how prone the controller should be to axial vibrations. The rig operator can also select different distances and speeds for the actuators to move either to increase or decrease the WOB. The WOB data that is sampled in real-time is filtered using a moving median filter with a window-size of 5 samples, which combined with 60 Hz frequency has proven to be reliable for the WOB control.

Software Architecture
The software architecture is configured as a multithreading system, meaning that several client/server modules, referred to as threads that run on multiple cores on the central processing units (CPUs), run in parallel using a gRPC-API (see Section 4.1) to handle the communication between each module / thread. In addition, an API based on OPC UA (see Section 4.2) has been developed to support remote connectivity and remote event handling, allowing external partners to execute commands remotely to the control system, and receive data from the rig in real-time upon request.
The system is described further in Sand (2018) and Guggedal and Steinstø (2019).

gRPC
The gRPC is an open source Remote Procedure Call (RPC) developed by Google. RPC is a communication platform where the program performs remote operations on a server. The framework allows for communication between a variety of platforms and programming languages, and while traditional APIs (applicable programming interface) share data through formats readable by humans, the gRPC uses protocol buffers for transparent communication between client and server applications. Protocol buffers are communication platforms that are not readable by humans. The user creates the server client implementation. This kind of structure is beneficial for many reasons, but the main one is that both size of buffer files are smaller and communication times are much higher than traditional communication platforms, see Guggedal and Steinstø (2019).

API -OPC UA
An Open Platform Communications Unified Architecture (OPC UA) is used for the API implementation on the rig in order to integrate a plug-and-play concept on the drilling rig. This enables the option for the client to remotely execute two different types of interactions with the rig: request information such as drilling data or exert commands to control the rig, see Guggedal and Steinstø (2019).

Graphical User Interfaces (GUIs)
Two GUIs have been developed for this laboratory scale drilling rig. These are a visualization GUI to track the progress of the autonomous control system and drilling performance and a downhole well trajectory environment, see Figure A.2, also showing the real-time position, inclination and orientation of the bit.

Sensors
With the main task being developing an autonomous drilling rig, the sensor implementation is an area that has received a lot of attention. The computer relies on good data from sensors implemented to make decisions and provide users with valuable information. Our drilling rig system has a total of 17 sensors, including three load cells capable of providing measurements for x, y and z-axis; two torque sensors, one integrated in the top drive and one external for torque measurement at the top of the drill string. The design, manufacturing and implementation of a downhole measurement tool including accelerometer, gyroscope and magnetometer have also been conducted in our work. Drill string vibration is a complex phenomenon that often results in Non-Productive Time. The complexity lies in the coupled action of the three vibrational modes: axial, lateral and torsional. The modes excite one another, which makes it difficult to recognize a particular mode and prevent destructive vibrations. A downhole measurement system allows us to capture the moment when vibrations occur and adjust operational parameters (the WOB and the RPM) in real-time to mitigate vibrations. The other function of downhole measurement is to do he bit position tracking and control.

Database
During each drilling experiment, the sensors installed on the rig gather a large amount of time-stamped data.
There have been many drilling experiments conducted over the years and a lot of data has been collected, either in real-time or for post analysis using a highfrequency data acquisition (DAQ) module. TimescaleDB is a faster and more reliable solution. It is created by heavily modifying the architecture of PostgreSQL, a well-known RDBMS. TimescaleDB still preserves all the features and capabilities of a conventional RDBMS. Data is organized in tables with columns, rows, indexes, primary keys, foreign keys, constraints and all the desirable features usually reserved for relational databases. TimescaleDB satisfies our requirements where both leverages the features of relational database systems, and also offers great query optimization for time series data. This is convenient in order to not have to implement two different databases; one to store sensor data and another to store rig state data.
In order to easily manage the data that gets stored in the database, an application that serves as the interface between the users and the database is required. A separate small application is also required: responsible for pushing the data to the database. The sensors' signals are sampled at a specific frequency and these data are stored in text files on the computer.
To make the data in the database easily accessible, a user-friendly platform to access the data was created in the team. This is solved by writing a web application, where users can log in and manage the stored data. Users can access and download the data gathered from either specific experiments, or a particular subset of those data. They can then choose to visualize the data by generating graphs, query experiments using different parameters, manage which variables that the database shall contain, register the drilled rocks (label the rock types, experiments, add comments etc.), manage access to the database, and so on. In addition, the data can easily be down-sampled to for instance 10% of the raw data. When working with datasets of millions of rows (measurements) and between five and thirty columns (sensors or features), this immediately allows the user to select which data-frequency to work with, rather than having to import a large dataset, carry out either a linear down-sampling process and then save the data as a new file.

Autonomous Drilling
The concept is a closed-loop autonomous control system that combines several models with a digital detailed operational drilling procedure developed for directional drilling. A simplified flow chart illustrating the system is shown in Figure 2. In addition, a novel voting system has been designed to prioritize which commands that should receive priority if an event, or incident gets detected or if conflicting commands from models get received at the same time. The voting system has been configured so that critical incidents that can risk damage to either equipment or the well receive the highest priority. Maximizing the ROP and ensuring that downhole vibrations are low are located at the lowest priority since these tasks are not critical to handle immediately to meet the objective.
There are however several challenges to perform an optimal ROP search. In the drilling scene, there is lacking information with regards to the destination that the Figure 2: Simplified illustration of the logic behind the autonomous drilling system. machine needs to reach for the maximum ROP and the most optimal route to get there. With several local maximums in the state space, the machine needs to not only locate a peak, but the global peak to ensure that the highest ROP is achieved. Another challenge occurs when considering the route that the machine follows in the search. Perhaps would the shortest path in an operation cause the machine to drill with highly suboptimal controller set points; for instance a very high RPM combined with an unnaturally low WOB which can result in severe lateral vibrations and possibly also damage either the drill string, BHA or bit. A third challenge is the step size that the algorithm selects. By keeping the step size very small, the time to move from the start point to the goal state increases as the machine would need to not only vary the controller setpoint but also drill at those setpoints in order to measure the response. If the step size is too large however, this reduces the time required to reach the destination but increases the likelihood of possibly bypassing the destination (overshooting).

ROP Optimization
To optimize the ROP, one first needs to consider what ROP really is. ROP is a multi-variable vector func-tion that depends on various parameters such as WOB, torque on bit (TOB), RPM, flow rate (Q), rock formation strength (UCS), drill bit properties and so on. The machine exerts control over the bit side properties. The formation strength and drilling environment on the other hand is heterogeneous, and changes in the drilling environment must be responded to in realtime. The two most important control variables are WOB and RPM. By varying either of the control variables (that are coupled, meaning that if one is varied so becomes the other) a difference in ROP, drilling efficiency and system response can be expected. A gradient descent (multivariate) search algorithm to optimize the ROP has been developed in Geekiyanage et al. (2019). The search algorithm now calculates the ROP gradient from evaluating the change in the depth gradient (ROP) over the last five seconds, and a learning rate function can be used to determine the step size to move for each iteration. The algorithm is terminated when the following condition is no longer satisfied: Denoting the drilling parameters WOB = x 1 , TOB = x 2 , RPM = x 3 , etc., the ROP X = (x 1 , x 2 , ..., x n ) where n ∈ + . ROP can then be expressed as: The search algorithm will attempt to find a set of coordinates (x * 1 , x * 2 , ..., x * n ), so that the ROP becomes maximum. The operational state space of the drilling machine constitutes the search space of the algorithm, satisfying the condition X i,min < X < X i,max , for instance 0 < RP M < 2000 and 0 < W OB < 50. The depth gradient, referred to as instantaneous ROP, is calculated using linear regression as can be shown in Equation (4), where N represents the number of samples considered to obtain an instantaneous gradient and n is always ≥ 2.
An objective function J can be defined using Euclidean norm, so that: The ROP gradient vector can now get calculated from: At any given combination of the control parameters, the ROP gradient always points to the maximal increase of the ROP function, and the gradient is always perpendicular to the ROP hyper-surface contour f (X) = c, where the constant c is an arbitrary real number. The algorithm is now capable of determining the search direction from a previous operating point.
The optimization now is reduced to a univariate line search, moving along the local gradients. The gradient descent method can be shown as: where η represents the learning rate; i.e. the step size to move in each iteration, and k denotes the iteration number. The step size can be selected by the equation below, considering that we define φ(η k ) = f (X k + ηg k ) and 0 < < 1 2 , which satisfies the Wolfe condition: The termination criterion to abort the search if the ROP increases at k + 1 is less than k can be defined by: δ the tolerance specified for when gradient search shall terminate

Downhole Closed-loop Steering
The concept of closed-loop steering in the control system is currently limited to inclination steering through active management of knuckle joint bend in the BHA.
For every 1 mm measured depth (MD), the relationship between horizontal build and true vertical depth (TVD) gets calculated so that: A look up table has been created which contains the H/T V D planned . The look up table values are then called every mm, and the relationship between the actual build and planned build is checked by the equation: If the build status is ≥ 1, the system will be in a drop angle mode, and if the build status is ≤ 1, the machine will be in a build angle mode and needs to increase the build-rate by increasing the WOB. While the downhole RPM only depends on the gradient search algorithm and the different phases of the operation, the voting system in the control system will continuously evaluate whether or not the system is in a build or drop mode. If for example the ROP gradient search proposes 14 kg WOB, but the system should be in a drop mode, the WOB setpoint will be overwritten to 12 kg, which is the highest permissible WOB in the drop mode. While a higher WOB from experiments have yielded a higher ROP, another control algorithm exists that evaluates whether the recommended controller setpoint (within the range identified for build or drop mode) gets forwarded to the PID controller, or whether an even lower setpoint should be used in the event of severe downhole vibrations or if the system is in a critical phase.

Digital Detailed Operating Procedure
The following digital detailed operating procedure (DDOP) has been developed to autonomously execute the drilling of a directional well. The DDOP is arranged in eight phases, as visualized in Figure 4.

Rig Performance
In this section, three well logs of a short inclined section, a vertical pilot hole and a complete inclined wellbore section are presented. The plotted figures have been downsampled to 1 Hz (from a 60 Hz sampling frequency). To smoothen the plots for inclination, azimuth and weight on bit, a Kalman filter has been used.

Experiment 1: Inclined Well Section
The first well section with logging tools in the BHA was drilled from 355 mm MD to 440 mm MD in homoge- Figure 4: Eight phases constitute the digital directional drilling operation.
neous cement, using a 7 degree whipstock, see Figure 5. The WOB setpoint is kept constant at 5 kg, while the RPM setpoint is varied between 925 and 1075 revolutions per minute. The resulting ROP is approximately 0.4 to 0.5 m/hr, which is considered a good result given the increased wellbore friction and a low WOB setpoint. WOB setpoint has from experiments conducted in the past been assessed to be the most important drilling parameter affecting the ROP. The MSE varies between 10 and 40 MPa, and was calculated by hardcoding a constant torque value equal to 1 Nm. From the figure, it can be observed that even with high oscillations in the downhole sensor measurements, approximately 7.5 mm horizontal build is achieved, with an offset on the azimuth of approximately 1 mm. In 6 below, a cross-section of the first deviation well that was drilled successfully is shown after a waterjet cutting machine was used to split the rock sample. From inspection of the rock, approximately 55 mm horizontal build was achieved over 490 mm TVD. In the last 150 mm of drilling, the knuckle joint turned slightly due to a twist in the connection between the knuckle joint and sensor sub, causing an azimuth offset of approximately 10 mm. From evaluating the well integrity, it is probable that bit whirl has occurred, possibly caused by the undergauge BHA. It is however difficult to monitor and confirm bit whirl with the current configuration.

Experiment 2: Vertical Well Section
The second well log, is shown in Figure 7. The RPM is kept nearly constant at 1040, and the WOB is increased from first 5, to 10 and later 15 kg. While the increased WOB leads to an improvement in the ROP, a slight build can be observed (according to measurements and visual inspection), which is possibly caused by the knuckle joint starting to bend at high WOB. As can be recalled from Section 6.2, the range for WOB BuildInclination starts at approximately 12 kg (up to maximum 18 kg). While the MSE also for this well log is calculated with 1 Nm torque constantly, it can be observed that the MSE is reduced as the WOB increases, suggesting less energy is wasted, and a higher percentage of the combined energy usage goes into bitrock interaction.

Experiment 3: Deviation Well with WOB 5 to 20 kg
Continuing to vary the WOB, the same approach as before is used in an inclined section in Figure 8. For the first 235 mm, the bit is RIH, with a 5 kg WOB setpoint and an RPM of between 730 and 1040. The previously drilled well section that the whipstock had been landed in got drilled as shown in Experiment 2 above. Contrary to the last experiment, it is observable that while the ROP continues to increase with an increase in the WOB setpoint, the MSE does not decrease. A possible explanation to this is that increased friction between the non-rotating BHA and the wellbore in the inclined section leads to a lower ROP (approximately 1 m/hr) than expected. Horizontal build is calculated from measurements to approximately 100mm, with an offset on the azimuth of approximately 3 to 4 mm (occurs at the end of the inclined section). Upon inspection of the well, approximately 45 mm horizontal build was achieved. Possible explanations of both the MSE, over-estimated horizontal build and only a moderately high ROP can be that; either the increased wellbore friction reduces the overall efficiency of the drilling operation and that constant vibrations against the wellbore influence the measurements for inclination calculation. Another possible explanation can be that only half of the BHA is submerged inside of the well, leaving the BHA even more unconfined, which in turn affects the vibration cycle of the entire assembly.

Conclusions
The resulting control system is capable of executing a series of control algorithms and models in parallel, such as the directional drilling program, the ROP search (by gradient descent method), inclination steering, downhole vibrations, incident and stuck pipe detection (and handling), and so on. This is possible through the use of voting system that prioritizes pending commands from the control algorithms depending on which action is most critical to execute. While tuning off the PID controller and gradient descent method is difficult with a bendable BHA developed to be used in inclined well sections, the proof of concept has been obtained by the algorithms selecting optimal drilling setpoints based on measured rig performance and response to earlier changes executed. In terms of control algorithms, the suggested practice for a complete closed-loop system is that a database gets developed that contains best practice drilling parameters for the different formation types, so that when a new formation type gets confirmed, the optimal drilling parameters can immediately get selected. The rig can then make use of various search-algorithms in the reduced state-space to identify the point at which the Founder point of the system is found. This point describes the optimal performance point when drilling with optimal conditions related to WOB and RPM, and the point is a linear function between these two parameters so that when one of the two increases or decreases, equipment might get damaged. Once the Founder point has been located, for instance by gradient descent method , a series of drilling incident models can be used to not only classify the occurrence of a drilling incident, but also if possible, predict that an incident is imminent (unless measures get taken). While all drilling operations are unlikely to get automated in the near future, in our opinion the first step on the path is to provide the driller with powerful tools to increase safety, operational efficiency and reduce costs. It is therefore our recommendation to build a control system that promotes human-machine integration, where the machine for instance can suggest optimal strategies going forward, and the driller can select one of these or simply override the suggestion. Furthermore, ensuring high quality of data in all steps of the decision making process will increase reliability and ensure consistently high performance.  the University of Stavanger. We acknowledge Equinor academia program and Aker BP for funding the ongoing project. Sand, E. L. Design and implementation of a control system for a fully automated drilling rig. Bachelor thesis, University of Stavanger, 2018.
A. Appendix