Debugging KAFKA connectivity integration with Remote Application including Spring Boot, Spark, Console Consumer, Open SSL
Our downstream partners wanted to consume data from Kafka Topic. They did open network & firewall ports with respective zookeeper & broker servers.
But, Spring Boot application or Console Consumer failed to consume messages from Kafka topic. Refer log stack trace below -
[2024-01-10 13:33:34,759] DEBUG [Consumer clientId=consumer-o2_prism_group-1, groupId=o2_prism_group] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
[2024-01-10 13:33:34,762] WARN [Consumer clientId=consumer-o2_prism_group-1, groupId=o2_prism_group] Bootstrap broker ncxxx001.h.c.com:9093 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-01-10 13:33:34,860] DEBUG [Consumer clientId=consumer-o2_prism_group-1, groupId=o2_prism_group] Initialize connection to node ncxxx001.h.c.com:9093 (id: -1 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2024-01-10 13:33:34,861] DEBUG [Consumer clientId=consumer-o2_prism_group-1, groupId=o2_prism_group] Initiating connection to node ncxxx001.h.c.com:9093 (id: -1 rack: null) using address ncxxx001.h.c.com/192.168.32.1 (org.apache.kafka.clients.NetworkClient)
Using SSLEngineImpl.
We have security.protocol=sasl_ssl . There are 2 parts to debugging process -
First SASL ( Kerberos )
- Set following property to enable Kerberos debugging logs
- -Dsun.security.krb5.debug=true
- -Djavax.net.debug=ssl
- *** ClientHello, TLSv1.2
- *** ServerHello, TLSv1.2
- *** Certificate chain
- *** ECDH ServerKeyExchange
- *** ServerHelloDone
- *** ECDHClientKeyExchange
- *** Finished [Notifying client-side handshake finished]
- *** Finished [Notifying server-side handshake finished]
- One can use tcpdump to listen to network. For example, below command will listen to tun0 ethernet connection and save data to file ti-dump.pcap -
- sudo tcpdump -i tun0 -w ti-dump.pcap
- Now, one can install Wireshark to analyze tcpdump file.
- Reference following to get Cipher supported -
- https://help.mulesoft.com/s/article/How-to-list-the-Cipher-Suite-of-JVM-and-the-Cipher-used-on-a-handshake-with-endpoint
- https://superuser.com/questions/109213/how-do-i-list-the-ssl-tls-cipher-suites-a-particular-website-offers
- Refer below Open SSL command to check SSL connectivity with server -
- openssl s_client -msg -debug -state -connect kafkabroker.my.hadoop.com:9093 -CAfile <(/usr/java/jdk1.8.0_251/bin/keytool -list -rfc -keystore truststore.jks -storepass 'changeit')
Comments
Post a Comment