Azure HDInsights - Sudden Spark Job Failure & Exit - ERROR sender.RawSocketSender: org.fluentd.logger.sender.RawSocketSender
We observed that Spark Job suddenly exited without any Error when running long on Azure HDInsights.
But, we observed following error -
22/07/13 05:38:32 ERROR RawSocketSender [MdsLoggerSenderThread]: Log data 53245216 larger than remaining buffer size 10485760
22/07/13 05:59:54 ERROR
sender.RawSocketSender: org.fluentd.logger.sender.RawSocketSender
java.net.ConnectException: Connection
refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at org.fluentd.logger.sender.RawSocketSender.connect(RawSocketSender.java:85)
at org.fluentd.logger.sender.RawSocketSender.reconnect(RawSocke
On call and debugging with Microsoft, it was found that -
- It looks like peregrine is logging a lot to the point it is exceeding the buffer.
Comments
Post a Comment