While observability can provide many benefits to information technology departments, it also raises important confidentiality and ethical concerns, including data privacy, appropriate use of data, bias and discrimination, and transparency. Observability can tackle these issues by incorporating tools with strong security and privacy controls, using data responsibly, addressing bias and discrimination, and being transparent with users about data collection and use. By facing these concerns, organizations can build trust with users and ensure the benefits of observability are realized in a responsible and ethical manner.
The importance of observability in IT domains
Observability is a critical concept in IT domains because modern software systems have become increasingly complex, which require complex solutions to deal with potential problems. Observability helps to detect bugs in these intricate systems. Additionally, it allows issues to be resolved more quickly with its ability to collect and analyze data on everything from logs, metrics, and traces, to allowing IT teams to monitor system health, identify bottlenecks, and troubleshoot problems efficiently. This leads to greater collaboration between IT teams and other stakeholders, and ultimately produces better outcomes for organizations.
Beyond the basics
Observability tools aren’t simply capable of dealing with system issues, such as logging events and activities, collecting, tracking, capturing, and analyzing data and performance behaviors, or even sending alerts when system performances deviate from expected levels. Where these tools step things up a notch is in their ability to protect an organization’s all-important data security. Observability tools can monitor system logs and metrics for unusual activity that may indicate a security threat. For example, if users attempt to access a resource where they don’t have permission, observability can flag the behaviors. Also, observability tools can identify vulnerabilities that hackers could exploit. It can do so by monitoring network traffic to help detect insecure communication protocols or unencrypted data and tracking in-house user activity to ensure employees aren’t engaging in unauthorized or suspicious behavior. In the case of a potential security breach, observability tools enable companies to quickly respond thanks to real-time visibility into the potential incident.
Want More Tech News? Subscribe to ComputingEdge Newsletter Today!
Privacy and ethical concerns
While these make observability tools a must-have for IT companies in today’s fast-paced technological world, there are still some genuine issues that need to be addressed: namely the many privacy and ethical concerns, including data confidentiality. Observability tools often collect and store large amounts of data, including sensitive information such as user credentials, personal data, and other confidential information. This data needs to be protected from unauthorized access and comply with data privacy regulations such as the European Union’s General Data Protection Regulation (GDPR), which requires organizations to obtain user consent before collecting personal data and to implement measures to protect user privacy, and the Health Insurance Portability and Compliance Act (HIPAA).
Companies also need to be mindful of collecting data on employee behavior, including their use of corporate devices and applications. In fact, they need to get informed consent from users and employees before collecting or analyzing any data with observability tools. It is important for that consent to be clearly lay out exactly how the data will be collected, stored, and used.
Observability tools also can be programmed with biases that can lead to discriminatory outcomes. For example, if a tool identifies high-risk users based on past behavior, it may unfairly target certain groups or individuals based on factors such as race or gender. While these tools are excellent for collecting and saving data, there is also the issue of how long an organization intends to keep all this data and at what point (if ever) will it be deleted. It is critical for organizations to establish a clear policy that also complies with the relevant data privacy regulations.
Shivani G. Shukla, an assistant professor in the business department at the University of San Francisco who has a Ph.D. in Management Science from the University of Massachusetts, noted that observability can’t be the sole metric to be looked at with regulations. “Observability is mostly about metrics and tracing and what things are happening,” she said. “Monitoring, though, is how much it should happen, how much is available, and the performance capacity. Observability can tell you where the system is wrong, whereas monitoring is just noting that something is wrong. That’s why regulations need to actually be built on monitoring rather than observability, especially if you’re generating a large language model.” The questions that need to be addressed, she added, are, “Where is the bias? Where do the ethics need to be fixed? Can we build some sort of monitoring system around that on which the regulations can be built, which also creates some space for creativity?”
Addressing the complex issues
Until regulators catch up with this fast-paced industry, these rules will keep evolving. Nonetheless, there are still ways organizations can use observability tools by looking to others that have adapted to and addressed the pressing issues. One company is OpenTelemetry. The open-source observability platform has ethical guidelines, including transparency, respect for privacy, and the use of ethical data collection and processing practices.
Similarly, Datadog, a cloud-based monitoring and analytics platform, has implemented observability tools to monitor the performance of its systems. To address its privacy concerns, Datadog employs measures, such as data anonymization and user consent, prior to collecting data. It also has a privacy policy that outlines its commitment to protecting user privacy and ensuring the ethical use of data.
The Electronic Frontier Foundation (EFF), a nonprofit dedicated to defending civil liberties in the digital world, developed ethical guidelines for the use of observability in the IT domain. The EFF’s guidelines emphasize the importance of transparency, informed consent, and data minimization to protect user privacy and ensure ethical data collection and processing practices.
Embracing the technology with appropriate guardrails
There is growing concern about observability’s potential impact on privacy and ethics as it becomes more widespread. As the field continues to grow and organizations adapt, there are ways to ensure the benefits of observability are realized responsibly. As long as organizations define clear goals and metrics and always consider account privacy and ethical concerns, there is a path forward. By using data responsibly and ensuring transparency and fairness, companies will be able to take advantage of all observability has to offer within a framework that embraces the protection of personal data and the promotion of fair and ethical practices.
About the Author
Ajay Reddy Yeruva is a senior software engineer with more than a decade of experience in the Information Technology field across healthcare, networking and cybersecurity domains. He is a thought leader and a subject matter expert in DevSecOps, site reliability and platform engineering landscapes, specializing in observability using emerging technologies including artificial intelligence, machine learning, Internet of Things and deep learning. Ajay has a master’s degree in management information systems from the University of Illinois Springfield. For more information, contact Ajay at ajayr.yeruva@gmail.com or on LinkedIn.
Disclaimer: The author is completely responsible for the content of this article. The opinions expressed are their own and do not represent IEEE’s position nor that of the Computer Society nor its Leadership.
