Logging

Introduction In this tutorial, you will learn how to send logs from Fluent Bit installed on a self-managed MongoDB running on an Ubuntu Droplet to DigitalOcean’s managed OpenSearch database. OpenSearch is an open-source search and analytics suite that originated as a fork of Elasticsearch and Kibana. It supports real-time data ingestion, advanced querying, and robust security, making it ideal for e-commerce, IT monitoring, and finance applications. With features like SQL query support, machine learning, and alerting, OpenSearch continuously evolves through…

Introduction This tutorial demonstrates how to collect and forward logs from a DigitalOcean Kubernetes (DOKS) cluster to a DigitalOcean Managed OpenSearch instance using AxoSyslog, a scalable security data processor. By following this guide, you’ll learn how to set up a robust logging system that captures and analyzes logs from your Kubernetes applications, making it easier to monitor, troubleshoot, and secure your infrastructure. In this tutorial, you will use AxoSyslog to forward logs from a Kubernetes cluster to OpenSearch. Prerequisites Before…

Introduction Effective log management is essential for monitoring and maintaining applications in today’s data-driven world. OpenSearch, an open-source search and analytics engine, provides robust log aggregation and analysis capabilities. By combining it with Fluent Bit, a lightweight log forwarder, we can efficiently send Python logs to OpenSearch for real-time monitoring and analysis. This guide will walk you through setting up this powerful combination. Use Case This setup benefits developers and system administrators who need to monitor Python applications in real-time.…

Introduction Keeping track of web server logs is essential for running your website smoothly, solving problems, and understanding user behavior. If you’re using Nginx, it produces access and error logs full of valuable information. To manage and analyze these logs, you can use Logstash to process and forward them and DigitalOcean’s Managed OpenSearch to index and visualize the data. In this tutorial, we will walk you through installing Logstash on a Droplet, setting it up to collect your Nginx logs,…

Introduction OpenSearch is designed to offer highly scalable and flexible search capabilities for a wide range of applications. It supports full-text search, log analytics, and real-time monitoring, providing powerful tools for data exploration and visualization. OpenSearch’s efficient log management is crucial for troubleshooting and monitoring applications. DigitalOcean App Platform supports OpenSearch log forwarding. You can forward runtime logs from your App Platform apps to an OpenSearch cluster, where you can analyze and query your app’s logs using OpenSearch’s APIs and…

Introduction Effective web server log management is crucial for maintaining your website’s performance, troubleshooting issues, and gaining insights into user behavior. Apache is one of the most popular web servers. It generates access and error logs that contain valuable information. To efficiently manage and analyze these logs, you can use Logstash to process and forward them to DigitalOcean’s Managed OpenSearch for indexing and visualization. In this tutorial, we will guide you through installing Logstash on a Droplet, configuring it to…

Introduction Log shippers are essential tools in modern log management and observability ecosystems, enabling the collection, processing, and forwarding of log data from various sources to centralized logging systems like DigitalOcean Managed OpenSearch. Selecting the right log shipper is crucial for efficient log management, as it directly impacts the performance, scalability, and reliability of your logging infrastructure. This document compares four widely used log shippers—Logstash, Filebeat, Fluentd, and Fluent Bit—highlighting their primary uses, strengths, and considerations. Additionally, it outlines the…

Introduction An effective logging solution is crucial to the success of any application. Winston is a versatile logging library and a popular logging solution available for Node.js applications. Winston’s features include support for multiple storage options, log levels, log queries, and a built-in profiler. In this tutorial, you will use Winston to log a Node/Express application that you’ll create as part of this process. You’ll also see how to combine Winston with Morgan, another popular HTTP request middleware logger for…

Introduction Logrotate is a system utility that manages the automatic rotation and compression of log files. If log files were not rotated, compressed, and periodically pruned, they could eventually consume all available disk space on a system. Logrotate is installed by default on Ubuntu 22.04, and is set up to handle the log rotation needs of all installed packages, including rsyslog, the default system log processor. In this article, we will explore the default Logrotate configuration, then configure log rotation…