QueryDB

  With Spring one can implement Thread Pool simply by org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor 1) Create  ThreadPoolConfig as below -  import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor; @Configuration public class ThreadPoolConfig { @Bean public ThreadPoolTaskExecutor threadPoolExecutor() { ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor(); executor .setCorePoolSize(50); executor .setMaxPoolSize(1000); executor .setAwaitTerminationSeconds(30); executor .setPrestartAllCoreThreads( true ); // executor.initialize(); return executor ; } } 2) Autowire  ThreadPoolTaskExecutor where needed -  @Autowired public ThreadPoolTaskExecutor taskExecutor ; logger .info( "Active Threads: {}, Pool Size: {}, Queue Size: {}" , taskE...

  We couldn't find any direct way to log Mongo Connection pool Size. So, we did implement an indirect way as below.  This may be incorrect at times when dealing with Sharded MongoDB having Primaty & Secondary nodes. Because, connection may be used based on read prefrence - Primary, primaryPreferred, Secondary, etc. But, this gives an understanding if connections are used efficiently and there is no wait to acquire connections from pool. This can be further enhanced to log correct connection pool statistics.  1) Implement  MyConnectionPoolListener  as below -  import java.util.concurrent.atomic.AtomicInteger; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import com.mongodb.event.ConnectionCheckOutFailedEvent; import com.mongodb.event.ConnectionCheckOutStartedEvent; import com.mongodb.event.ConnectionCheckedInEvent; import com.mongodb.event.ConnectionCheckedOutEvent; import com.mongodb.event.ConnectionClosedEvent; import com.mongodb.event.Conne...

  Experience with MongoDB and Optimizations Before reading below. I would like to point out that this  experience  is related to version  6.0.14-ent, having 6 shards, each shard having 3 machines, each machine is VM with 140 GB RAM and 2TB SSD. And, we had been hosting almost 36 TB of data. MongoDB is not good with Big Data Joins and/ or Big Data OLAP processing. It is mainly meant for OLTP purposes.  Instead of joining millions of keys between 2 collections. It is better to lookup data of one key from one collection then lookup it in other collection. Thus, merging data from 2 collection for same key. Its better to keep De-normalized data in one document.  Updating a document later is cumbersome.  MongoDB crash if data is overloaded. And, it has long downtime if crashed unlike other databases which fails write to database if disk space achieves certain limit. Thus, keeping database active and running for read traffic. MongoDB needs indexes for fast qu...

  Open File Descriptors w.r.t Kafka brokers relates with following -  number of file descriptors to just track log segment files. Additional file descriptors to communicate via network sockets with external parties (such as clients, other brokers, Zookeeper, and Kerberos). For # 1 this is formula -  (number of partitions)*(partition size / segment size) Reference -  https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/kafka-performance-tuning/topics/kafka-tune-broker-syslevel-file-descriptors.html For #2, every connection made my consumer or producer or zookeeper or  Kerberos  opens file descriptors. Note that each TCP connection creates 2 file descriptors. These connections can be for internal communication of heartbeat, or  security handshake , or data transfer to or from client (producer or consumer) When we run a Spark application integrating it with  Kafka . And, if it is not stable, meaning -  Streaming window for micro batches is les...