Water Data Forum - Cybersecurity in Water: Protecting your data

[00:00:00] Elkin Hernandez: Good morning and good afternoon for some of you, depending on where you are. I welcome you to the Water Data Forum. The Water Data For  is a series of interactive web sessions that engage cross-sector experts in exploration of utility, industry, and research approaches to collecting, managing, and measuring water data for impact.

[00:00:41] Topics in this free virtual forum will range from new sensor and control technologies to broader application spaces, such as water data for environmental justice and climate resiliency. As well as, complex technology, topics as cybersecurity,  big data, ai, ml and,  analytics in general.

[00:01:05]  The water data forum has been running for a few years now.  Previous sessions are available, free of charge, on the water data web page, I encourage people to check them.  Today,  I'm very, very excited to present to the panelists that we have the luck of having a couple of experts from CISA that are willing to share their knowledge and their expertise with the group.

[00:01:42]  What you see on the screen is myself. My name is Elkin Hernandez. I'll be the facilitator of the discussion. I'm a director of maintenance service with DC water and I am the health chair of the intelligent water technology committee with WEF.  With that said, I'll move to introduce our panelists.

[00:02:11] Elkin Hernandez: Our panelists,  today are Lauren Wisniewski.  Lauren is the Water and Wastewater Sector Liaison, Cybersecurity and Infrastructure, with the Cybersecurity and Infrastructure Agency, CISA.  She is the nexus for CISA water and Wastewater sector efforts. She partners with the EPA and the Sector Risk Management Agency, as well as private and public stakeholders in the wastewater sector.

[00:02:41] In addition to supporting water efforts across CISA Ms Wisniewski is a CSA representative on the water government coordinating council. She works with EPA and other agencies, partners as well.  Ms. Wisniewski, has a bachelor's degree in engineering,  with a summa cum laude, from civil engineering from Duke University, and served as the engineering student government president during her senior year.

[00:03:18] Additionally, she has a master's in public health from GW University, George Washington.  Previously joining CISA, she spent a long tenure with the Environmental Protection Agency, EPA. She will be leading the discussion along with Matthew Rogers. Matthew is an ICS cybersecurity expert.  For the ones that are not familiar, ICS stands for Industrial Control Systems.

[00:03:48]  Matthew,  is a PhD and as an expert,  an expert that worked for the office of the technical director at CISA and the lead for the secure by design initiative for operational technology OT. He received his PhD in security legacy of the networks in vehicles from the University of Oxford on a Rhodes scholarship.

[00:04:13] Matthew worked as a founding engineer at a vehicle and weapon system cybersecurity startup before pursuing a broader ICS cybersecurity efforts. Matthew's focus at CISA is on how ICS research and development efforts can be transitioned to effective tools for critical infrastructure sectors.

[00:04:34] So these are two experts that we're lucky to have today and,  just want to, before I handed them I wanted to go over quick logistics here. We're going to have a presentation, at the end of the presentation, we're going to have an open discussion. For the people listening to,  participating in this, discussion in this session, if you want to send a, question to be shared with the rest of the public, please do it using the tool on the zoom application, on the Q&A, at the end of the presentation, we're going to go through the questions and try to, answer as many as we can.

[00:05:17]  With that said, I'll turn it to Matthew and Lauren, please. Welcome

[00:05:23] Lauren Wisniewski: Online now. So many of you probably already know this, but I just wanted to start off,  with some background on CISA. We're the nation's cyber defense agency, and the national coordinator of critical infrastructure, and kind of coordinate across all 16 critical infrastructure sectors, water and wastewater, being 1 of them. We were formed a little over 5 years ago as part of the Department of Homeland Security.

[00:05:50] And our vision is a secure and resilient infrastructure for the American people and to understand, manage and reduce risk to our nation's physical and cyber security.  Are you guys hearing me? Okay. I just got a note in the chat. Okay. 

[00:06:09] Matthew Rogers: we're good.

[00:06:10] Lauren Wisniewski: Physical and cyber infrastructure next slide. Before we get into some of the data protections and artificial intelligence, and, I wanted to give a little bit of background on some of our water and wastewater sector resources, because I understand we have a number of water systems on the line, EPA and SISA developed a toolkit at SISA.gov/water

[00:06:36] And this has a whole number of resources for drinking water and wastewater systems of all sizes and capabilities to help increase your cyber security, everything from vulnerability scanning, incident response, technical assistance, a whole host of resources that you can access there.

[00:06:56]  There's 1 in particular that I want to highlight on the next slide. And this is vulnerability scanning. This is a scan of the Internet accessible assets of a system. So if you think of your house, it's just going around the external network and letting, you know, your doors open and your windows open, and sharing those publicly facing vulnerabilities, letting, you know, every week.

[00:07:24] What the vulnerabilities is identified are how to mitigate them, you know, and from a data science perspective, the more water systems that we can have enrolled in this, the more data we get across the sector, the better we can identify vulnerabilities, particularly common ones to the water and wastewater system to tailor our messaging or to see, you know, what common devices or lack of updates there might be across the sector.  So when a system enrolls in this and it's completely free, they get weekly scans and these, weekly reports. Next slide please. And there's a huge benefit,  that we see that kind of across all 16 sectors, about a 40 percent reduction in vulnerabilities, for those systems who have enrolled.

[00:08:16] Usually within the 1st few months, you know that cyber incidents can be extremely expensive. And so there's a lot of benefit to enrolling in this service.  Even if you already may have a different vulnerability scanning service, this is another set of eyes that this is constantly updating.

[00:08:35] It's vulnerability scanning to include the latest vulnerabilities and we'll let you know what they find. Next slide,  for anyone who is not enrolled or wants to share this with other water systems, it's really simple to get started. You just email vulnerability@cisa.dhs.gov

[00:08:56] Asking to request vulnerability scanning services, providing some basic information,  once you fill out a couple of forms, then you'll be entered into the system,  usually within about 10 days. So that I will be handed over to Matt.

[00:09:21] Matthew Rogers: All right. Thanks so much. Warren. Okay, so you guys are going to get a bit of a sneak peek because, some CISA guidance is coming out for this broadly on operational technology, largely on how to integrate a lot of thoughts on AI and machine learning technologies into your general operational technology lifecycle, such that you can make some of the secure decisions, both from a procurement step, but also as you get into deployment and maintenance, so that we're actually considering, you know, safety and security when we're including any of these AI systems.

[00:09:49]  So to give you a little bit of a sneak peek, you know, a lot of the focus on AI cybersecurity is really centered around the security of the model and security of the data, but also on the more from an operational lens. What happens when you think about replacing one of your existing systems or augmenting some of your existing systems with a model that is now trained to learn, trained to be modified, trained to improve over time, but also is, you know, not a characteristic of your network that you have previously spent a lot of time with.

[00:10:25] Of a lot of experienced testing or is maybe from a new vendor you're not experienced with and perhaps trust less right so because of that just to highlight a couple of the questions that we encourage people to ask, you know, we have sections from procurement from deployment and maintenance here, you know, things like.

[00:10:43] Describing the expertise that an organization has when deploying and models for the water and wastewater water and wastewater system sector, because we've often found, particularly from a security vendor perspective, the actual water operators, people who work in these facilities.

[00:11:00] You all know far more about how the actual water system works, and so I'm much more capable of drawing insights from how that data can be used, or more importantly, how that data really should be used in an operational process, whereas sort of a generic machine learning based approach to data. Can provide some wins, the more you're capable of tailoring it to the actual water system that you have towards your common workflows, towards the data that you were able to actually get somebody access to, you know, the better the model is going to be.

[00:11:29] And some of that just simply comes down to domain expertise of understanding what the core tasks and functions of a water system are of a wastewater system are, but transit for water system is because a third party, especially coming from an AI space, there's a lot of new startups in this area. They may not just not have any of the domain expertise necessary, right?

[00:11:49] So just come in and saying, you know, they maybe don't have the expertise. That's not necessarily a deal breaker for working with anybody if you're looking to procure a solution, but it does mean that you're going to have to spend a lot more time and resources to make that resource actually good for you.

[00:12:04] And of course the marketing pitch for some of these products might say, it actually works just fine for the water sector, regardless of any expertise, you know, learned on the fly, but we've seen, you know, if you want to get the most bang for your buck out of anything that you purchase, right? Like, you're really going to need to work with them to understand how the AI model works within your particular domain.

[00:12:24] And similarly, trying to understand more of that dependency factor of, you know, when you're incorporating this into critical infrastructure, fundamentally, we're talking about this idea of resilience and critical infrastructure, right? Like how do you maintain uptime? Can't just deny water to folks. And so what dependencies does that generate from a third party vendor perspective, right?

[00:12:43] Like what are they doing with your data? Are they shipping that data off? Are they dependent on some third party company themselves that they don't have control over, you know, What risk are you taking on by including more people in your process? Again, not to say that it's always a wrong decision, but it's important to actually know that those risks are there so you can mitigate them appropriately.

[00:13:03] Right. And, you know, this last comment on operator domain expertise again gets this time component, right? Like, do you have the time and resources to really make the best use of that tool compared to maybe a more existing low tech or analog solution that you have right now,  in order to make sure that you're really making the most of the money that you're spending on some sort of AI ML solution.

[00:13:23] Because it's definitely not going to be a silver bullet out of the box.  From a deployment perspective, I think one of the main things that we're pushing at CISA is making sure that when you go back to that like instant response guide, any of your safety decisions or protocols, how are you actually factoring in the idea of this AI system failing?

[00:13:41] Now, it might look like any other system failing in terms of what that process looks like, but it definitely needs to be updated in the existing plan. About an AI system failing just straight up going offline or more subtly actually just giving wrong data, giving wrong results. Maybe there's a bit of a model drift problem where somebody is actually slowly having the model shift in one direction or another, maybe no fault of their own, but just because this model needs to be retuned. How do you know how often it needs to happen? What's your protocol for what goes wrong if that model drift starts to occur? These are all things that need to be considered when you're looking into actually deploying the system in the first place, right?

[00:14:26] And then, finally, from a maintenance perspective, again, like if you have an AI system in place, you need to be able to maintain it for long periods of time. A lot of these computers and their existing environment And especially in an operational technology sense are very much in a leave it and forget it mode.

[00:14:42] So how can we actually make sure that the system is still functioning properly over time? What are the upkeep measures? What are the maintenance measures that need to be particularly put into place, but also what security controls actually exist for who is doing that update? Who is doing that process of looking into these AI models?

[00:14:59]  You know especially in the era of attacks like we saw in last December, November timeframe with Iranian affiliated groups targeting open access on the public internet water systems. You know, it's a really important consideration to consider like a managed service provider or a vendor risk of if they have remote access capabilities into the network to update the AI model.

[00:15:23] Yes, there's the risk of them modifying the AI model and, you know, or stealing data, or, you know, if that data is being shipped up to the vendor, then that vendor is now like a very tantalizing target for attack. But also, if they have remote access into your system, that we have seen instances before, where a managed service provider gets hacked.

[00:15:46] That people, the hackers then use the access they get from that managed service provider, then have a bunch of different entities at scale, because that managed service provider themselves wasn't secure, right? So anytime you're adding any sort of new remote, any new device, particularly with remote connections into the network.

[00:16:02] You really have to think what are the access identity management policies that you can actually apply onto that system to make sure that they can get access whenever they wish. Now, one more advanced way that we see of people doing this is having a VPN that requires somebody to be on site to open up.

[00:16:19] So if you want remote access into the system, that's required for the AI model fine, but somebody on site needs to actually physically hit a button to enable that VPN. So the connection isn't always enabled. It's an on demand connection where the on demand portion is the will of the operator or the asset owner rather than the on demand whim of the vendor or the person selling you this AI product.

[00:16:41] That's a more advanced solution. And of course, in general, like segmentation controls and general best practices apply. And, you know, to get into that, right? You know, there's this whole life cycle to consider again, more guidance forthcoming, on, you know, the secure design operations and maintenance and development of these AI models themselves.

[00:17:01] So you can consider if you're buying, if procuring, or if you're modifying an open source model yourself, making sure that, you know, there's security solutions in place from a data perspective throughout the entire life cycle. But a lot of the cyber security specific considerations for AI do end up narrowing down to protecting the AI model and a lot of the cyber security features that you would honestly want to put in place anyway for a lot of these operational technology systems.

[00:17:29] Things like segmenting off access to make sure that you know, a limited proportion, a portion of traffic can actually hit your control network, role based access control to control any modification to that network, remote access security, particularly for that AI vendor management, like I was talking about earlier.

[00:17:46] But also things like securing data at rest, securing data in transit, you know, all good policies generally, but particularly in the context of AI are very important for making sure that somebody isn't able to really get a sense of what this AI model is capable of doing. But I will say, like, from an operational perspective, A lot of these security wins are really going to come more from the identity management, the segmentation, because once they're down low enough that they can access the root model, particularly if it's stored on like the operational network, you know, like there are, there are broader concerns due to the general trusting in nature of a lot of these OT networks.

[00:18:21] And so it's much more focused on making sure that those access paths are restricted or as controlled as possible. Including particularly for that remote access path. So with that, that's the main gist of the presentation.  We have a lot of time for questions, so I'm looking forward to answering questions from folks.

[00:18:39] And of course, you can always reach out via email to either Warren or myself,  if you have any broader questions about this initiative. Thanks.

[00:18:49] Elkin Hernandez: Thank you, Matthew. Right now, I would encourage participants to share questions with us so we can go over those as we use the remainder of time.

[00:19:00]  While we have questions, I'd like to myself ask a few ones. So, for both of you, and this might be good to Matthew,  what is the intersection of cyber security And AI and ML mean for Sisa.

[00:19:24] Matthew Rogers: I think the main focus is making sure that AI is treated very, you know is a particular software has very particular use cases for advancing the kind of our day to day operations.

[00:19:38] How we improve our general processes, but fundamentally we're trying to make sure that people are still treating it like software that needs to be secured and treated like any other software security process. And there's like some advanced features of what you would want to do to an AI ML system in order to do cybersecurity for it.

[00:19:54] But fundamentally, we're just trying to make sure that people are thinking of these solutions in a secure way and cybersecurity ramifications of adding them to their system, rather than it's just the new shiny thing that's going to solve a lot of problems. Because we've seen AI ML you know, broadly in the cybersecurity space for a good number of years now, right?

[00:20:12] Like generative AI is definitely caused a bit of a buzz in terms of how people are talking about these solutions. But, you know, things like using AI or ML systems for predictive maintenance, like a safety instrument system, maintaining a continuous process, or for things like anomaly detection or network monitoring.

[00:20:32] You know, these are not new processes and there's pros and cons to all of these approaches. But it's important to consider, you know, kind of the lessons learned from the past,  and incorporating cybersecurity into those to this day.

[00:20:47] Elkin Hernandez: A follow up question is, what do you think we are in the developing curve or, implementation of AI solutions on this water and wastewater sector? Are we in the early stage? In the middle? Are we advanced? But what do you think we are right now?

[00:21:07] Matthew Rogers: I mean, I would say that I think we're pretty low currently.

[00:21:10] Warren, you've seen way more water systems than I do on a day to day basis. Maybe you could Take a spin.

[00:21:16] Lauren Wisniewski: Yeah, no, I agree. I think it's fairly, early on, in the, you know, in the water and wastewater sector, you know, I've seen some smart metering and using it, but I think we're kind of just on the early adoption side and can may benefit from other sectors, like the energy sector who are a little bit further down that path.

[00:21:36] Matthew Rogers: And just an important addendum there, right? Like, a lot of the problems historically have come from getting access to this data in an operational setting, right? Like, trying to get. You know, network data communications data and an OT kind of setting is possible. We've seen a lot of vendors work in that space because it's less obtrusive.

[00:21:54] But if you're talking about enhancing water system processes. You really need a lot of that process data that actually comes from some of the core output from sensors from controllers to see what is actually happening in the system and getting that data has always been a bit of a just a practical challenge, right?

[00:22:11] And so we're seeing more manufacturers have a little bit more transparency with that data. We're seeing more people try to get access to that data or make conclusions based on process data, but I would say generally a lot of the critical infrastructure space. Has had more of a data access problem for getting down to these solutions.

[00:22:27] And then from a cybersecurity perspective, you know, that's kind of been a good thing. And that, like, it means that there's fewer connection paths into the operational network. So it's just worth noting, like, as people try to get more of this data out of the system, those are potential attack vectors if they're not done correctly, if they're not done securely.

[00:22:44] So it's just something to think through as you're trying to integrate more smart solutions into these systems.

[00:22:51] Elkin Hernandez: Thank you, Matthew. I see I'm starting to have some questions here, which is always a positive thing. Let me, one is very simple. If you can, post again your email addresses. Somebody probably wants to catch that I think they are in the presentation.

[00:23:16] Let me, let me see if I can do that.

[00:23:18] Matthew Rogers: Let's see. I just typed it out in the chat as well.

[00:23:26] Matthew Rogers: Yeah. So I answered on the chat.  yeah. So if you go to the answer tab, you should be able to see it. That's perfect. Awesome.

[00:23:35] Elkin Hernandez: That's perfect. Awesome Matthew. I have a question here.

[00:23:38] It says, When thinking of having AI make actual changes to live system parameters, where is the industry at with respect to these AI hands on controls opposed to advisory? I don't know if you guys have some chance to evaluate where we are there. That's an excellent question, actually.

[00:23:59] Matthew Rogers: Almost nobody I have seen has gone past the advisory stage.

[00:24:02] Yeah, it's very spooky to go past the advisory stage. So, you know, I've seen in like other sectors, some spaces are kind of a little bit more willing to go into some of that automated decision making, but beyond the parameters of a safety instrumented system where the entire goal is like maintaining a rigid process.

[00:24:22] I don't really see them making autonomous actions, right? It's either maintaining a stable, safe process, but not diverging from that. Diverging from that, I think is a little bit of a cliff currently in terms of operator sentimentality and safety.

[00:24:39] Elkin Hernandez: I share that feeling. That's, that's your view.

[00:24:41] In my case, it's a feeling. Yeah, it's not, not yet time to go there. Out of curiosity, which industry have you seen that gone into that direction?

[00:24:52] Matthew Rogers:  Well, I mean, the Department of Defense cares a lot about automated prevention. Transportation systems are also kind of interested in it it’s a bit spooky, but it becomes one of those, like time to react to questions.

[00:25:05]  Similarly, like, why would energy be curious about it, though they have so many safety concerns that it's, you know, kind of impossible. But from a, like, a Department of Defense perspective, like, if a big plane is in the air and the plane costs a billion dollars, like, if you can prevent something from costing you a billion dollars, you might do it.

[00:25:23] Even if the risks are a little high. So they're at least tinkering with the idea, but operationally, it's not happening currently.

[00:25:32] Elkin Hernandez: Thank you. And staying a bit with the topics, there's a question that, to me, it's an interesting one. It says, well, when we're talking here about AI and ML, and I think Lauren mentions, water metering.

[00:25:48] But besides that application, what system or features of applications are we seeing becoming prevalent in the industry or more popular?

[00:26:00] Matthew Rogers: Maybe I'll start and then Lauren, you can correct me here. I mean, the main use cases we see are either like a predictive modeling and maintenance perspective. So you might see it and you can kind of think of it like older telemetry sensors for vibration to identify if a component is like.

[00:26:15] Going to be failing relatively soon, you know, those sort of statistical models that people might now be labeling as AI models or ML models, either, you know, cynically, because they're just statistical models are more optimistically because they're improving over time with some of the more modern technology developments.

[00:26:32] So a lot of it is around predictive maintenance or maintaining a process space. And then, like, from a cybersecurity peer perspective. I mean, we are seeing like EDR products, like standard antivirus network monitoring tools, they've always used some level of statistical analysis and heuristics to try to check anomalies.

[00:26:52] They've been getting more advanced over time, just to make sure that if a piece of malware changes slightly, it still gets caught.  But, you know, we run the risk for a lot of that with, like, false positives, like, who's monitoring it, which was just always kind of a rabbit hole from an operational perspective.

[00:27:08]  Lauren, did I miss anything? Any water? 

[00:27:11] Lauren Wisniewski: No, that pretty much covers it with the predictive modeling, you know, potential water main breaks and, you know, how temperatures may influence that, but yeah, those are the big ones that I've seen too.

[00:27:24] Elkin Hernandez: Okay. Just changing a bit the topic and we may come back to this topic again later.

[00:27:30] Sure. It's a question that, I will see, comes at the level of the CIO or the executives. Well, they want to understand what will the national Security memorandum on critical infrastructure, security and resilience in as SM22 mean for the water and wastewater sector. How do you see that?

[00:27:53] Lauren Wisniewski: Right. So that's a great question.  and for those of you who may not be following the inside the beltway, activities on April 30th, national security memorandum  22 or N. S. M. 22,  replaced,  P. P. D. E. or presidential policy directive 21. And that kind of provides the framework for how the US government will protect and secure critical infrastructure, collaborating with both the public and private sector.

[00:28:20]  It kind of reaffirms CISA as the national coordinator of the 16 critical infrastructure sectors. EPA is the sector risk management agency or SRMA for the water and wastewater sector. So, you can continue to see those efforts. Right now, there's some, they're pretty stringent,  deadlines established by that memorandum including, the development for each of the 16 critical infrastructure sector risk assessments,  or in this case, EPA has a lead to identify and prioritize risks within the sector.

[00:28:56] And then following on the development of that plan, I will have to do a sector risk management plan and then these plans will have to be repeated every couple of years. So I think you'll see a more, you know, holistic, national, coordination, you know, across, you know, with CISA and looking at,  some of the, the cross sector impacts, but certainly, a lot of great work coming,  from EPA and, you know, identifying those, risks and coming up with, plans to address those.

[00:29:33] Elkin Hernandez: Thank you. And following going back to CISA's core functions.  What does CISA offer to the water and wastewater sector to ensure continued protection? I think you mentioned a few, but I'll let you expand that. Service.

[00:29:51] Lauren Wisniewski: Yeah, so I mentioned vulnerability scanning, which is a great 1. Another 1 that's really, I think, is our regional resources that CISA has cyber security advisors, protective security advisors, emergency communication coordinators.

[00:30:09] In all 50 states and 6 territories, their job is to help protect critical infrastructure entities like water and wastewater systems. They're in your states in your communities, and any water system that's interested could reach out to their system region. If you don't know how to get in touch, please contact me and I'll put you in touch and they can do an initial.

[00:30:32] Phone, you know, chat, kind of talk about some of the services,  they can come on site and do a cyber assessment ranging from a couple hours to a couple of days.  but really kind of tailor, you know, some risk mitigation strategies based on the entity and their needs and,  their cyber maturity and Matt.

[00:30:54] I don't know. Are there any others?  that, you want to mention?

[00:30:59] Matthew Rogers: Yeah, I mean, your, your regional cybersecurity advisors is a good bet for assessing your initial maturity. You know, they will go out and start conducting actual assessments. So if you're willing to let them, they'll walk through and I'll interview you and ask a bunch of questions to kind of model where your current cybersecurity is give you some recommendations going forward, but then potentially sign you up for more advanced services like we have one that is practically a red team for your systems.

[00:31:25] We have one called the Vader that will. Basically take a network capture of your O. T. traffic and then compare that to your documentation to see, like, are there any devices you don't know about that help you establish more of an asset inventory so you can actually see what you have going on, because that's a core problem that a lot of people have in these environments.

[00:31:46]  So we have a lot more advanced assessments, particularly as people are going up the maturity model.  But I would recommend starting with your regional advisor. Get a good sense. Regardless, they're a good person to know just in case there's ever an incident or you need to call somebody for a bit of advice or recommendation.

[00:32:01]  And then from there, you know, they'll link you with the best guidance and then put you along that maturity track for figuring out what resources or what assessments you should be doing in the future.

[00:32:13] Elkin Hernandez: And, following on those lines, what are the biggest challenges,  you guys have observed or seen? In getting utilities signing in and joining, this type of scanning and partnerships. What are the biggest barriers there?

[00:32:36] Lauren Wisniewski:  I've seen a couple. You know, when I was at AWW ACE in Anaheim, probably about 90 percent of the water systems that I talked to had never heard of CISA.

[00:32:48]  That, you know, we're part of DHS,  but, you know, if you don't know who CISA is and chances are, you probably don't want to send us your IP address to scan. And so I think creating,  those trusted partnerships and that's where, you know, the water environment federation, you know, and other water associations and webinars like this, you know, can help,  get our name out there.

[00:33:13]  And to understand that, you know, we're a resource for systems.  I think another 1 is that, you know, some water utilities just don't see themselves as, you know, having a threat for cyber that they don't recognize that they're at risk if they have things on the open Internet.  that, you know, adversaries from around the globe can see them.

[00:33:35] We, you know, some of the open source reporting from some of the recent attacks had water utility operators saying, you know, I, we're just a small system and this state, we never saw that, you know, the, you know, the Iranian group would be coming after us, or the pro Russian activists would be coming after us.

[00:33:52]  But that's just a reality and trying to, you know, convey, you know, the seriousness of the threat to water and wastewater systems of all sizes and all locations, you know, and I think a 3rd is just that, you know, water systems have a lot on their plate with, regulations, aging infrastructure, limiting budgets and this is, you know, 1 more thing.

[00:34:15]  And so really trying to find the space and to make it as easy as possible. You know, to implement basic, cybersecurity, you know, measures. That's something that we haven't talked about but in terms of just, you know, CISA has its secure world with, you know, strong passwords and enable multi factor authentication, be aware of phishing and update your software.

[00:34:40] And that for a lot of systems, some of those basic hygiene measures could, you know. Take away a lot of their threats and so trying to just build that culture of cyber security, you know, 1 step at a time.

[00:34:55] Elkin Hernandez: Well, thank you. And I would, not claiming any expertise in the topic, but I would see that attacks would only come from the state sponsored hackers. Sometimes it's just people trying to make some money, make lots of small utilities and make prey for them. You know, so it's not always a sophisticated attacker, it's just somebody who shared the passwords wrongly and somebody wants to make some money as Something is being seen in the sector.

[00:35:26] It's funny to add that part. This is a very interesting question here. So, you mentioned the resources that CISA puts, makes available free of charge to utilities to help protect themselves.  But are there, in addition to that, any federal funding available for water and wastewater utilities to respond to identified risks that require a budget for?

[00:35:51] So, if I need to buy equipment or hire people, is there any federal funding available for those type of efforts?

[00:35:59] Lauren Wisniewski: So I'm not sure about hiring people, but in terms of projects,  CISA has the, the state and local cybersecurity grant program, that's administered by each of the states and a little bit different.

[00:36:10] So you'd have to figure out, you know, what the options are in your state. And then,  the state,  revolving funds, both on the clean water and drinking water side, allow for cybersecurity, to be an eligible expense to apply for. So I know that the grant process can be a lot, but those are certainly some avenues for additional funding.

[00:36:38] Elkin Hernandez: All right. Thank you.  I'll jump back to the topic AI and ML topic,  what lessons can the water sector learn? from other sectors that have incorporated AI and ML into their operational processes.

[00:36:57] Matthew Rogers: I think one of the core mistakes that we saw early on is that, you know, a lot of the times because these models were being used for predictive maintenance, because they were being used to maintain a process, they hoovered up so much data that they themselves became a very valuable target.

[00:37:14] Both like understanding the network or understanding the system. Like one of the core ways that you can actually identify an attacker when they get to the OT network is, or any sort of threat actor is like, most of the time an operator doesn't really need to enumerate the water system because they kind of know what exists already, or at least what they.

[00:37:31]  You know, deal with on a regular basis and so having an AI system that aggregates all of that data for an adversary so they don't have to try and enumerate it and be kind of noisy on the network, you know, that's one potential attack factor, like a reason for securing this data. You know, we've also seen, you know, the light of all of the system advisories that would be put out on threats from Volt Typhoon from like PRC, Chinese affiliated actors.

[00:37:57] Like a lot of where their focus has been is basically getting onto a network and then exfiltrating data to use at a later point, right? So if they can get a password dump, figure out what all the passwords are in the OT environment, like you're not going to really change any sort of login or identity,  identity access to like an HMI very frequently.

[00:38:16] And so they can maybe come back six months later, Several years later, and then come back and use that information. Right. And so if you're aggregating a bunch of data that an AI model might use is potentially like a very nice target for somebody trying to just get all of that information very quietly or very quickly.

[00:38:33]  On top of that, like there's, that's the kind of data process model,  on the other side, like if you have an automated system that is doing any sort of control work, like a safety instrument system, the best practice, you can see this, if you look up the Triton attack that happened in 2017, please disconnect it from the rest of the network, like that safety process, whatever safety procedures you have that have some more of this like automated or statistical control measure, if they're part of the general network.

[00:39:02] That's a huge attack factor because now if an attacker can pivot to that system, they basically have the authority to execute task and execute commands, to kind of override the system. And usually the network, or like the water system generally assumes that that safety system is kind of a trusted actor and knows what it's doing.

[00:39:21] And so if somebody can compromise it at that point, like you're really in trouble. And so trying to make sure that those are disconnected from the rest of the network, kind of on their own separate thing, controlling the system and getting, data, either take a one way connection or in a way that is like heavily monitored,  is pretty key for making sure that we don't have some of the more disastrous attacks on the system.

[00:39:47] Elkin Hernandez: Thank you, Matthew. Another policy question to just keep jumping back on topics. I have a question here. It says, how does the built America buy American compliance? purchase of cybersecurity products.

[00:40:06] Matthew Rogers: Lauren, are you familiar with that?

[00:40:08] Lauren Wisniewski:  I'm familiar with it, but I'm not sure. Other than that may limit, you know, some of the foreign components that we probably don't want to have in our systems.

[00:40:21] Matthew Rogers: Yeah. I mean, in terms of pure cybersecurity products, right. Like differentiating cybersecure products versus cybersecurity products, right.

[00:40:30] Like if you're doing a monitoring thing. I mean, a lot of those companies are stationed overseas. There are a couple of American ones that are tailored to OT, but like several of them are Israeli, for example. And then I mean, same for like all of the automation controllers and everything that you'd find in a typical water system, right?

[00:40:46] I mean, like Siemens Schneider Electric, like those are European companies. Hitachi is a Japanese company. They all have US components, but like, you know, worth considering in no way saying don't buy from any of those controllers but I'm not actually sure how it works in terms of that particular policy measure.

[00:41:05] Elkin Hernandez: I would add a layer to that question. So I keep hearing, Oh, there's a lot of corner fed products. You think you're buying something and you're getting something else and you don't know if that something else is just a fake product or is a compromised product. So, we thought it was something that happens to other hunters.

[00:41:25] We went checking on some routers that we got,  and we bought them when we checked them. Authenticity. Probably we found that they were counterfeit. So if there's a take for anybody here is, for things that are related to your cyber systems, make sure that you're getting what you think you're getting in one way to do is to make sure there's the supply chain that has been verified.

[00:41:50] All right. I have a question. So I've seen an increase amount of platforms that provide analytics and machine learning for cyber security monitoring. What do you think we are in? And it's more generic, not only what was water, but on ISC system in general. In terms of developing those tools and being able to use them in a practical way where we can move from being overwhelmed by data and making sure we can actually see what's going on and take actions when they're granted.

[00:42:41] Matthew Rogers: I think the main utility from a lot of those security, you know, AI machine learning solutions that currently exist in the market.  from talking to some of the different customers across sectors, like the main utility folks seem to be getting from it is mostly from an asset inventory perspective of knowing what they have and basically doing some of that passive or in some cases like active monitoring to understand what exists on a pretty distributed system.

[00:43:05] In terms of the next level of that, but like doing the cybersecurity detections, dealing with like a flood of traffic of logs, and then like. You have like a secure operation center narrowing that all of that noise down to appear like this is the threat signal. I mean, it's definitely gotten better over the years, but like, we're not at a point where it wouldn't require some sort of cybersecurity team to actively monitor it to have it be effective, right?

[00:43:30] Like we're not at a point where you can necessarily integrate that viewpoint into an HMI or SCADA viewpoint. And then just kind of give a thumbs up and say, like, we have our little cybersecurity monitor in the corner, like, it's not at that point yet. Like, maturity isn't there. Maybe we'll eventually get there, but I'm not quite so optimistic, right?

[00:43:50] I think, like, about half of cybersecurity is really process, understanding regular workflows. Some of these cybersecurity tools are getting better at that. But fundamentally, you're probably always going to have to have somebody be. Having a rational process for like how this is how the organization works.

[00:44:05] So, just all of the say, like, you're not buying a silver bullet whenever you buy a cybersecurity product, but they are helpful in your organization, especially from like an ICS perspective, like doubling down on that comment we made for the AI stuff earlier, like make sure that they either have domain expertise in the system, like your particular sector, or if they like be willing to commit the time and resources.

[00:44:29] And also like if they don't have expertise in your sector, you should be getting it on sale because they don't actually understand what assets you're using, so be willing to work with them to understand, like what are the actual cybersecurity nuances that occur in your operational processes?

[00:44:44] Because that isn't intuitive or obvious to anybody, but the people in the sector.

[00:44:51] Elkin Hernandez: That was a very insightful question. And I wonder if that makes me feel better.

[00:44:58] So I have a follow up to the build American by American question. And I need more than a question. I think it's just a comment. I think this, attendees tell us that, bill American by American requires all components in a product, to have 50 percent of the cost be made in the, you know, from us products.

[00:45:16] So that means nothing is available for purchase. Most chips are made out of the country. So I think it's an interesting observation and I can see why he's made it. I just wanted to share that with the group. I have, this is a very interesting discussion. I have a question that is way more specific in terms of technical details.

[00:45:40]  We have one that says, what is your perspective on copying certain process data out of your OT data historian into a cloud data lake to enable management level reporting and data analysis? Can this be done safely?

[00:45:58] Matthew Rogers:  I think the answer is yes. So, one of the main concerns is that you can be pushing data out of the O. T. system, but data shouldn't really be coming into the operational technology bit of your network from external connection without you either asking for it from the O. T. side or having some sort of approved process.

[00:46:21] Right? So if you're just sending process data, out of the OT network and into some sort of data lake for like analytics or sort of some sort of data monitoring as long as you're following best practices in terms of like identity and access control for that data, making sure it's encrypted at rest and then encrypted into the transit into that system, like, Perfectly fine.

[00:46:41]  I mean, you're inherently accepting some sort of risk because you have a copy of process data that you otherwise probably don't want people to have access to. Also, like, if you want to make any conclusions out of it, you have to accept that risk. So I would just look up like a lot of the general cloud security guidance.

[00:46:58]  If you're using something more generic, like a Amazon Google Workspaces, Microsoft Azure kind of platform, then I would look up Sysa's scuba tool, which provides a lot of the best practices for cloud guidance and cloud configuration. So that you're actually just enabling the defaults that both like Sysa recommends, but also we've worked with those big cloud providers to actually recommend that guidance as well.

[00:47:22]  I would also recommend, and I'll put these, I'll put these links in the chat. So it's not just me naming things off.  There's a very good bit of guidance that actually comes from our partners in the United Kingdom called, like SCADA in the cloud, which is basically a long article from the national cybersecurity center on kind of the pros and cons of thinking about moving SCADA data into cloud environments.

[00:47:44] What are the risks there? A lot of that comes down to hybrid ownership, who's responsible for the data, but it's worth a read.  and so I'll link with those programs in the chat.

[00:47:58] Elkin Hernandez: Oh, thank you, Matthew.  I have another question here. And you know, one of the things we at least I've been hearing through these years when listening to cyber security experts is that,  for many utilities,  there's so little being done in terms of cyber security that even the governance is not in place.

[00:48:23] So the question is, does CISA offer any templates and guidance on producing incident response documentation? And governance in general.

[00:48:35] Matthew Rogers:  So we have the water incident response guide, which maybe Lauren, you were just about the reference. Do we have any other resources? Top of your head?

[00:48:46] Lauren Wisniewski: Right? So we have this incident response guide that we developed with EPA in the sector that kind of talks about, coordinating across the, the federal government and the various people that you would want to work with.

[00:48:59] Interact with and develop those relationships. We also, I think, have some overarching cyber training documents. Let me get at that trying to think of any others.

[00:49:18] Matthew Rogers: Yeah, there's also some checklists. If you look on the website for cyber security and part of this is incorporated into their incident response guide, but they have sort of a checklist based approach of, like, this is what you should do during an incident. And that can be a good foundation for actually developing an incident response guide yourself in terms of, like, the prior to incident, parts of approach.

[00:49:40] I believe the EPA checklist covers that as well. See if I can find a link,  but in terms of setting up that initial process, I don't know that we have quite as many links for that.

[00:49:54] Elkin Hernandez: All right. Well, thanks for that answered. 

[00:50:03] Besides, EPA, what other agencies is CISA,  interacting with when it comes to,  Water and wastewater sector, resources that you would suggest people to check and consult when it comes to,  cyber security in the water and wastewater sector. 

[00:50:32] Lauren Wisniewski:  So we work with our federal partners,  but, you know, we also work,  through the water sector coordinating council with groups like,  American water works association, the water ISAC water environment federation,  and a lot of those groups also have, cyber security, our resources, you know, some of what you're available at no cost.  You know, so as the federal government, we can't,  you know, recommend, you know, any endorse any particular,  outside groups,  resource, but, you know, they do have, specific, water sector resources that may be of use and value.

[00:51:11] Elkin Hernandez: Right. Do you guys have any work with NIST when it comes to documentation, technical documentation?

[00:51:18] Lauren Wisniewski: Yes.

[00:51:20] Elkin Hernandez: Okay. I think that's another source for anybody listening, where there's some, some, interesting, technical information that can support. And give guidance on, on cyber security. 

[00:51:32] Matthew Rogers: Yeah, I mean, in particular, the one for operational technology is NIST special publication 882.

[00:51:42] I'd recommend checking that out for a longer read about how some of the general NIST cybersecurity controls apply or don't apply to operational technology. Just due to some of the operational and legacy constraints. Otherwise, I would recommend assistive cyber performance goals, CPGs, which are mapped to the NIST cybersecurity framework.

[00:52:00] So the NIST cybersecurity framework generally will help you through some of the process of what to do prior to and after a cybersecurity incident to detect, identify, prepare, respond. I'm forgetting the fifth pillar and then actually establishing a governance system in this CSF 2. 0. and so a lot of those resources, I think it's just like NIST.gov/CSF,  will be particularly useful if you're trying to start a cybersecurity governance process or just thinking through what are some of the best practices, am I maybe missing something in my existing process, just to read about what some other people are doing.

[00:52:39] Elkin Hernandez: Well, thank you, and we're close to an end.

[00:52:42] We're close to the hour.  So before I close the session, is there anything you guys like to add?

[00:52:55] Lauren Wisniewski: Really appreciate the opportunity and thank you all for participating.

[00:53:02] Elkin Hernandez: Alright, let me, let me share my screen and I will, then thank everybody for attending the session. Remind you that this session will be uploaded on their website. The address is here. We're at you know, by the clevelandwateralliance.org/wdf.

[00:53:25]  There's upcoming sessions for the remainder of this year. And you can go back and check my memory works well, about 15 to 18 sessions that are available for consultation.  Thank you for your participation for sharing your knowledge with us and, thank you everybody for attending.

[00:53:46] Goodbye.

‍