Fix Sentry ClickHouse DNS Error After Upgrade

by Admin 46 views
Application: Listen [::]:9009 failed: Poco::Exception – Troubleshooting Guide

Experiencing the Application: Listen [::]:9009 failed: Poco::Exception error after upgrading your self-hosted Sentry instance to version 25.10.0? This guide will walk you through the steps to resolve this issue, focusing on the Poco::Exception, the DNS error, and ensuring your ClickHouse configuration is correctly set up to handle IPv6 or IPv4 addresses. Let's dive in and get your Sentry instance back on track!

Understanding the Issue

After upgrading to version 25.10.0, you might encounter the following error in your ClickHouse logs:

Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 25.3.6.10034.altinitystable (altinity build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>

This error indicates that ClickHouse is failing to listen on the specified IPv6 address ([::]:9009). This typically occurs when IPv6 is disabled on your host machine, but ClickHouse is still configured to listen on IPv6 addresses. The error message itself suggests a solution: modify the <listen_host> element in the ClickHouse configuration file.

Prerequisites

Before we begin, make sure you have the following:

  • A self-hosted Sentry instance running version 25.10.0.
  • Docker and Docker Compose installed.
  • Access to the ClickHouse container's file system.
  • Basic knowledge of Docker and Docker Compose.

Steps to Resolve the Issue

1. Identify the ClickHouse Configuration File

The primary configuration file for ClickHouse is usually located at /etc/clickhouse-server/config.xml inside the ClickHouse container. However, in a Dockerized environment like Sentry self-hosted, the configuration might be composed of multiple files. These files are merged to form the final configuration.

Check the following locations inside the ClickHouse container:

  • /etc/clickhouse-server/config.xml
  • /etc/clickhouse-server/config.d/docker_related_config.xml
  • /etc/clickhouse-server/config.d/sentry.xml

The relevant configuration for the listening host is likely within one of these files.

2. Access the ClickHouse Container

To modify the configuration, you need to access the ClickHouse container. Use the following command to enter the container's shell:

docker exec -it sentry-self-hosted-clickhouse-1 bash

Replace sentry-self-hosted-clickhouse-1 with the actual name of your ClickHouse container if it's different.

3. Modify the <listen_host> Element

Now that you're inside the container, use a text editor like nano or vim to modify the configuration file. For example:

nano /etc/clickhouse-server/config.xml

Locate the <listen_host> element within the <tcp_port> section. If it's set to [::], it means ClickHouse is listening on all IPv6 addresses. To disable IPv6 listening, change it to 0.0.0.0:

<tcp_port>9000</tcp_port>
<listen_host>0.0.0.0</listen_host>

If the <listen_host> tag is missing, add it within the <tcp_port> section.

Repeat this process for other relevant ports, such as 9009, 8123, 9000, 9004, and 9005, if you see similar errors for those ports in the logs. Make sure that you are editing the correct file. It is important to review all the configuration files mentioned earlier to ensure the setting is applied correctly and not overridden by another configuration file.

4. Save the Changes and Exit

After making the necessary changes, save the file and exit the text editor. If you're using nano, press Ctrl+X, then Y to save, and Enter to confirm.

5. Restart the ClickHouse Container

For the changes to take effect, you need to restart the ClickHouse container. Exit the container's shell and use the following command:

docker restart sentry-self-hosted-clickhouse-1

6. Verify the Logs

After restarting the container, check the logs again to ensure the error is resolved:

docker logs sentry-self-hosted-clickhouse-1

If the Application: Listen [::]:9009 failed error is gone, you've successfully resolved the issue.

Additional Considerations

IPv6 vs. IPv4

  • IPv6 ([::]): Listens on all IPv6 addresses.
  • IPv4 (0.0.0.0): Listens on all IPv4 addresses.

If your server has IPv6 disabled, attempting to listen on an IPv6 address will result in the EAI: Address family for hostname not supported error. Ensure your ClickHouse configuration aligns with your server's network configuration.

Docker Compose Configuration

If you're managing your Sentry instance using Docker Compose, you can persist these changes by modifying the ClickHouse service configuration in your docker-compose.yml file. You can use a Dockerfile to override the default ClickHouse configuration.

Create a Dockerfile in the clickhouse directory with the following content:

FROM clickhouse/clickhouse-server:25.3.6

COPY config.xml /etc/clickhouse-server/config.xml

Place the modified config.xml in the same directory. In your docker-compose.yml file, reference this Dockerfile:

services:
  clickhouse:
    build:
      context: ./clickhouse
      dockerfile: Dockerfile
    # ... other configurations

Converting Ordinary Databases to Atomic

The logs also mention a warning about Ordinary databases being deprecated. To convert them to the Atomic engine, follow these steps inside the ClickHouse container:

sudo touch '/var/lib/clickhouse/flags/convert_ordinary_to_atomic'
sudo chmod 666 '/var/lib/clickhouse/flags/convert_ordinary_to_atomic'

Then, restart the ClickHouse container.

Conclusion

By following these steps, you should be able to resolve the Application: Listen [::]:9009 failed: Poco::Exception error in your self-hosted Sentry ClickHouse instance. Remember to align your ClickHouse configuration with your server's network settings and consider persisting your changes using Docker Compose for future updates. If you have any questions, feel free to ask!