I am at work and I’m doing an activity that I have done hundreds of times. There are no mistakes, because the case is extremely easy!
It was easy at that time, but the situation changed the next day :). I would like to tell you a story of how, due to my carelessness and error in one digit, I caused the breakdown of the object from a distance of about 350 km.
INTRODUCTION
Some time ago I’ve worked in a company and its specificity was based on providing customers with automation solutions in the water and sewage industry. In this company, among others, electrical cabinets and I&C are manufactures, which we (employees) had to set up in sophisticated efficiency and tested quite thoroughly. Then such a cabinet was sent to the customer by courier or packed in a company car and was immediately taken for the facility for the installation. The main advantage of this industry is a remote access to the PLC thanks to the GPRS modem and, consequently, providing the client with the SCADA visualization system. It is beautiful, isn’t it?
ROUTINE
One day I was supposed to configure another control cabinet. Such configuration consists of installing the program to the PLC controller and HMI panel, GPRS modem settings, security settings and any devices that can be configured. After configuration, before packing the control cabinet, it was still necessary to test it. These tests consist of checking as many functionalities as possible and situations that may occur during operation of the object, which is controlled by the control cabinet. After some time, these activities are performed automatically.
It you think that “my” breakdown was caused by the wrong testing and control cabinet was sent to the client, then you are wrong – such an oversights are common in every company. My breakdown was much more unusual 🙂
LAZINESS
The company headquarter is located in 2-floor building. My desk was on the second floor and control cabinets were assembled in special rooms in the basement. I had to use the stairs so many times. I had to go back so many times because I forgot something from my desk, ehh. Although those who know me will not believe me, because I am still, as it is gently said – thick.
2:45 pm – call from the basement:
- Marcin, control cabinet is ready, you can test it – it was a call from Rafał, and when he was the one who assembled the control cabinet, then it means that there will be no problems during testing. Cool 🙂
- I’m taking my stuff and I’ll be right there. – I answered.
I did exactly as I said. I took my laptop and went to my friend to take the SIM card for the GPRS modem. My friend was responsible for those cards and assigning the IP numbers to them. I went to the assembly room. I put my laptop next to the control cabinet. I inserted the SIM card to the modem and connected the power supply. I searched for communication cable which should connect the laptop and PLC controller. F*** I left it upstairs.
3:10 pm in assembly room:
- I forgot the cable for the controller, I’ll be right back.
- You are so stupid, you have remote access to the controller. We’ll have a break in a minute, take the laptop and upload the program from the upstairs and I will start configuring the control cabinet. – Rafał suggested, and it seemed to me, at the time, a brilliant idea.
I took my laptop and paper sheet with IP number. I went upstairs, put my laptop on the desk and started up special software. I entered the IP 50.4.10.7. I connected with GPRS modem in the control cabinet in the basement and remotely uploaded program to the PLC controller.
I called to Rafał – so he could finally test the control cabinet. I spend the rest of the day at work doing “something important”. 4:00 pm time to go home! Everyone act like like programmed robots – get up, high five, goodbye.
SEWAGE, EXCREMENT! SH** EVERYWHERE!
The next day at work:
8:00 am– coffee,
8:15 am – e-mails,
8:30 am – work, work, work!
10:00 am – supervisor picks up the phone:
- Good morning, I’m calling from waterworks in “350km from Marcin”. We have a problem with P-14 intermediate pumping station. The pumps stopped working yesterday after 3:30 pm, the emergency control does not respond at all and the sewage flows through the lid. – Our client called, operation data is from the SCADA visualization system.
- Ok, we will check it. – Supervisor answered and told me to check it.
Doesn’t work? It can happen that the level probe can break down or the shortcut can damage something. I checked which pumping station it is, logged in to the SCADA visualization server. Strange, all indications went crazy and showed nonsense. I was about to check it and connect remotely to the PLC via GPRS modem. From the database I took the IP number of the SIM card from this object and … IP is 50.4.10.7.
Humility?
HOW DID THIS HAPPEN?
How did this happen, that some housing estate, about 350km away, has been flooded by excrement?
After accurate analysis of this breakdown, I can bring a few facts:
- IP1 = 50.4.10.7 assigned to the SIM card of the object 350km away. This object is a sewage pumping station (it pumps over the sewage further, to the effluent treatment plant).
- IP2 = 50.4.10.1 assigned to the SIM card of the control cabinet located in the basement. This control cabinet was supposed to control the hydrophore (it keeps water pressure in your tap).
- Digit 7 is similar to digit 1 🙂
- PLC program of the hydrophore did not control the sewage pumping station.
- During remotely uploading a program to the PLC, always make sure it is uploaded into the proper object.
- Unfortunately for me, the breakdown of emergency control of the pumping station showed up – in normal operation, the pumps would start despite the wrong PLC program.
Yes, humility! This situation taught me the caution in my work.