
A few years back, I was working at a company that is a global leader in retail. We had just successfully installed a **Strimzi-based Kafka and Kafka Connect cluster on Kubernetes**. The platform was up, stable, and ready. The next obvious step was integrations - one by one.

One of the first integrations assigned to me was with **Solace**.

At that time, Solace was not yet a central messaging platform. It was still in its early adoption phase, with a handful of teams publishing messages as part of ongoing experiments. Our goal was to bring selected streams from Solace into Kafka using a **Solace Source Connector** running on Kafka Connect.

## The Easy Part: PoC Success

I started by studying Solace fundamentals - topics, queues, subscriptions, and consumption models. Solace’s documentation and courses was decent, and it didn’t take long to get productive.

I set up a local PoC using dockerized Solace broker, Kafka, Kafka Connect and a Solace Source Connector.

Within a short time, I had messages flowing from a Solace subscription into Kafka topics.

I demoed the PoC to my team and manager. Everyone was happy. We communicated the success to other teams, and the **Solace–Kafka integration was officially considered "done."**

Or so we thought.

## Reality Arrives: Production

A few months later, the real requirements showed up.

The Solace team wanted to run proper integration tests against their **production-like Solace environment**. Their setup was still evolving, and my responsibility was to ensure that **Kafka Connect could reliably consume from Solace**.

We intentionally limited scope - one pipeline, one connector - pilot only.

I integrated the Solace Source Connector into our Kafka Connect Docker image, wired it into the CI/CD pipeline, and deployed it to Kubernetes.

Everything looked good. I configured the connector and started it.

And… it failed - right away.

## The Error That Changed Everything

After digging through the logs, one error stood out clearly:

```log
INFO Channel Closed (smfclient 6) (com.solacesystems.jcsmp.protocol.impl.TcpClientChannel) [task-thread-solaceSourceConnector-0]
INFO closeOutbound() : isSslDowngradeEnabled: false, mSslEngineClosed: false (com.solacesystems.jcsmp.protocol.smf.SSLSmfClient) [task-thread-solaceSourceConnector-0]
INFO Received Solace exception java.security.cert.CertificateException: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors, with the following: [com.solacesystems.jcsmp.protocol.impl.TcpChannel.executePostOnce(TcpChannel.java:257), com.solacesystems.jcsmp.protocol.impl.ChannelOpStrategyClient.performOpen(ChannelOpStrategyClient.java:90), com.solacesystems.jcsmp.protocol.impl.TcpClientChannel.performOpenSingle(TcpClientChannel.java:416), com.solacesystems.jcsmp.protocol.impl.TcpClientChannel.access$800(TcpClientChannel.java:106), com.solacesystems.jcsmp.protocol.impl.TcpClientChannel$ClientChannelConnect.call(TcpClientChannel.java:2390), com.solacesystems.jcsmp.protocol.impl.TcpClientChannel.open(TcpClientChannel.java:392), com.solacesystems.jcsmp.impl.JCSMPBasicSession.sniffRouter(JCSMPBasicSession.java:354), com.solacesystems.jcsmp.impl.JCSMPBasicSession.connect(JCSMPBasicSession.java:1191), com.solace.connector.kafka.connect.source.SolSessionHandler.connectSession(SolSessionHandler.java:176), com.solace.connector.kafka.connect.source.SolaceSourceTask.start(SolaceSourceTask.java:81), org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:208), org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177), org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227), java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511), java.util.concurrent.FutureTask.run(FutureTask.java:266), java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149), java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624), java.lang.Thread.run(Thread.java:748)]  (com.solace.connector.kafka.connect.source.SolaceSourceTask) [task-thread-solaceSourceConnector-0]
INFO ================ Failed to create JCSMPSession Session (com.solace.connector.kafka.connect.source.SolaceSourceTask) [task-thread-solaceSourceConnector-0]
INFO ================ Shutting down PubSub+ Source Connector (com.solace.connector.kafka.connect.source.SolaceSourceTask) [task-thread-solaceSourceConnector-0]
```

> `java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors`

This was not a Kafka error. Not a Solace error. Not a misconfiguration typo. This was **TLS**.

At that point in my career, my understanding of certificates was… shallow. Same for most of my team members. I reached out to the Solace team - they were equally unsure.

So I did what engineers eventually used to do: I stopped guessing and started learning.

## Who Is the Client and Who Is the Server?

Before going deeper, it’s important to be very precise about roles:

* **Solace broker** -> TLS server
* **Kafka Connect + Solace Source Connector** -> TLS client

The Solace source connector running in the Kafka Connect was trying to establish a **secure TCP (TCPS) connection** to the Solace broker. That connection failed during certificate validation.

So the problem had nothing to do with Kafka or Connect itself. It was about **trust**.

## How Secure Client–Server Communication Actually Works

TLS is often explained as **the server encrypting data with a key**, but that’s an oversimplification. It is actually a multi-stage negotiation.

### 1. The Trust Phase (Asymmetric Encryption)

When the Kafka Connect client hits the Solace broker, the broker presents its certificate.

The client (Kafka Connect) checks if the certificate is valid, not expired, and matches the hostname - basically an identity check.

Then client looks at the signature on the certificate. If it doesn't recognize the signer, it looks for the signer's certificate, and so on, until it finds a Root CA it trusts in its **truststore**.

Once trust is established, the client uses the server’s public key to encrypt a random **pre-master secret** and sends it back. Only the server can decrypt this because only the server has the matching Private Key. This is the key exchange phase.

### 2. The Data Phase (Symmetric Encryption)

Asymmetric encryption (using public/private keys) is computationally *expensive and slow*. Once both sides have safely shared the secret mentioned above, they generate Symmetric Keys - one single key to encrypt and decrypt data.

From this point forward, both the client and server use the same key to encrypt and decrypt the actual message payload.

This provides high-speed confidentiality and ensures that even if someone intercepts the traffic, it looks like gibberish.

## The Role of Certificates

A **TLS certificate** is primarily an *identity document*.

It contains:

* The server’s public key
* The server’s identity (DNS names)
* A digital signature from a Certificate Authority (CA)

The **private key** corresponding to that public key stays on the server and is never shared.

## Certificate Authorities and Trust

Here is the critical question:

> Why should a client trust a certificate presented by a server?

Because it is **signed by a Certificate Authority (CA)** that the client already trusts.

Operating systems and JVMs ship with a predefined set of trusted **root CA certificates**. These live in a file called a **truststore**.

It is worth noting the difference between the two stores often used in Java: while a `Truststore` is used to store certificates from identities we trust (like the Solace broker), a `Keystore` is used to store our own private keys and certificates to prove our identity to others.

Public web trust is dominated by a relatively small number of root CAs (DigiCert, Sectigo, GlobalSign, etc.). Enterprises often add their **own internal root CAs** on top of that.

## Certificate Chains (This Is Where Things Break)

In practice, certificates form a chain that contains:

1. **Root CA certificate** (self-signed, trusted implicitly)
2. **Intermediate CA certificate(s)** (signed by the root)
3. **Server certificate** (signed by an intermediate)

When a TLS handshake starts, the server sends its certificate (and usually intermediates) to the client. The client tries to build a chain. The chain must end at a root certificate present in the client’s truststore.

The client acts like a detective tracing a lineage. It starts with the server's certificate and follows the signatures upward. Each certificate must prove the identity of the one below it, eventually leading to a Root CA that the client explicitly trusts in its truststore. If any link is missing-or if the detective doesn't recognize the person at the top of the family tree - the investigation (and the connection) fails immediately.

![img](/images/tls/tls_img.svg)

## What `CertPathValidatorException` Really Means

In Java terms, this error means exactly one thing:

> "I cannot build a valid certification path from the server certificate to any trusted root certificate."

Common reasons could be at least one of the following or more:

* The root CA is missing from the truststore (one of our case)
* An intermediate CA is missing (one of our case)
* The certificate is expired
* The certificate is signed by an unknown internal CA
* The server sends an incomplete certificate chain

In Java terms, this error occurs strictly during phase 1 - trust phase. It means the Kafka Connect client reached the end of the *chain* provided by the Solace server and never found a *Trust Anchor* (a Root CA) that existed in its local cacerts file. Because it couldn't verify who the server was, it refused to move to phase two towards data exchange.

![img](/images/tls/tls_1_4.svg)

## Why It Worked Locally but Failed in Production

This part was painful but educational. When I looked back at my local PoC, I realized I hadn't actually tested the security layer at all.

My local Dockerized Solace broker was configured for plain text communication. I was connecting via a standard TCP port without TLS enabled. Because there was no *handshake*, there were no certificates to validate. It just worked because the security door was wide open.

The production-like Solace environment was **hardened**. It required a secure TCPS connection. Kafka Connect was now running inside a minimal Linux container. Unlike a corporate laptop, these containers don't come pre-loaded with any internal company certificates. The JVM truststore inside that container was empty of any *Enterprise* trust.

The code and the connector logic were identical in both environments. The difference was the infrastructure requirements: the local environment didn't ask for an identity, but the production environment demanded a passport (the certificate) that my container didn't know how to verify.

## The Fix (Conceptually Simple, Operationally Non-Trivial)

Once the root cause was clear, the solution itself was straightforward, but it required changes on both ends of the connection.

On the Solace side:

1. The broker certificate was updated to present a complete certificate chain, including the correct intermediate CA.

On the Kafka Connect side:

1. Obtain the **enterprise root (and intermediate) CA certificates** used by Solace
2. Convert them into a format usable by Java
3. Add them to Kafka Connect’s truststore
4. Mount that truststore securely into the Kafka Connect container
5. Reference it in the connector configuration

```bash
# Most corporate certificates are issued in binary (DER) format. We convert
# them to PEM (Base64) to make them compatible with Linux tools and easier to bundle.
openssl x509 -inform der -in certificate.cer -out certificate.pem

# Since Kafka Connect runs on the JVM, it cannot read the PEM file directly.
# We must import it into a JKS (Java KeyStore) file.
keytool -import -file certificate.pem -alias ent-root -keystore trustStoreFile -storepass verySecretPassword

# Finally, we store that truststore file as a Secret so Strimzi can mount it
# into our Kafka Connect pods.
kubectl create secret generic team-trust-store --from-file=trustStoreFile
```

In Kubernetes terms:

* Create a Java truststore containing the required CA certificates
* Store it as a Kubernetes Secret
* For Strimzi-managed clusters, this involves adding the secret to the spec.`externalConfiguration.container.volumeMounts` section of the `KafkaConnect` custom resource.
* Reference the mounted file path in the Solace Source Connector configuration (usually via `ssl.truststore.location`).

Source connector config:

```json
{
    "connector.class": "com.solace.connector.kafka.connect.source.SolaceSourceConnector",
    "errors.log.enable": "true",
    ...
    "sol.ssl_trust_store": "/opt/kafka/external-configuration/team-trust-store/trustStoreFile",
    "sol.ssl_trust_store_format": "JKS",
    "sol.ssl_trust_store_password": "verySecretPassword"
}
```

## The Result

Once the certificate chain on the Solace broker was corrected and the Kafka Connect truststore was properly configured the TLS handshake succeeded. The full certificate chain validated cleanly. Kafka Connect established a secure session with Solace. Messages started flowing into Kafka right away.

## What This Taught Me

This incident was one of my first real lessons in distributed systems:

* Security failures are often *configuration*, not *code*
* TLS errors are precise, even if they look cryptic
* Containers make trust explicit - nothing is inherited magically
* If you don’t understand certificates, production will force you to

That `CertPathValidatorException` marked the beginning of a much deeper journey into TLS, JVM internals, and distributed system reliability.

And honestly? I’m glad it failed. Because that failure taught me something I still use every day.


Kafka Connect TLS Certificate Error: A Real Production Story


A few days ago, I had the opportunity to attend an [**5-Day AI Agents Intensive Course with Google**](https://www.kaggle.com/learn-guide/5-day-agents) arranged by Kaggle in collaboration with Google. Today, I want to share what the course covered, what I learned, and why I believe agentic systems are going to shape the future of software development.

## The Course Structure

The course was organized into **five modules across five days**, each focused on one core pillar of agentic AI development. Every day included a rich set of learning materials: a summary podcast, a detailed whitepaper, hands-on codelabs in Python using ADK, and a one-hour YouTube livestream featuring managers, engineers, and researchers from Google discussing the day’s topic. This blend of theory, practice, and expert insight made the learning experience both deep and practical.

### **Day 1: Introduction to AI Agents**

The first day laid the foundation. We explored what AI agents are, why they matter, and how they differ from standalone LLM applications. Using the Agent Development Kit, we built our very first agent - and even a simple multi-agent system. It was surprisingly approachable: a bit of Python, some orchestration, and an LLM under the hood.

The accompanying whitepaper introduced a taxonomy of agent capabilities and emphasized the importance of **AgentOps**, reliability, governance, and secure interoperability using identity and constrained policies.

In the codelabs, we built a basic agent powered by Gemini, and a small multi-agent system capable of using Google Search for real-time information retrieval.

These early exercises highlighted the power of combining reasoning (LLMs) with tools and structured workflows.

### **Day 2: Building Tools for Agents**

Day two focused on the tool ecosystem that enables agents to act rather than just reason. We learned how [MCP](https://modelcontextprotocol.io/docs/getting-started/intro) servers and standardized protocols let agents call external APIs, retrieve real-time data, and perform actions that go beyond the LLM’s training data.

The whitepaper covered best practices for designing high-quality tools and introduced the architecture, communication layers, and enterprise-readiness considerations of MCP.

In the codelabs, we converted Python functions into tools agents could call, built custom MCP-based capabilities, and implemented long-running tool operations where an agent pauses execution and requests human approval before continuing.

### **Day 3: Context Engineering**

The third module was all about giving agents **memory**, continuity, and personalization. We explored how context engineering enables agents to maintain working memory within a session and persist long-term memory across sessions - crucial for multi-step or long-running tasks.

The whitepaper introduced “Sessions” as the container for immediate conversational history and “Memory” as the persistence layer for long-term state retention.

The codelabs demonstrated how to manage conversation history in ADK, enable multi-turn reasoning, and give agents long-term memory that survives across different interactions.

By the end of the day, agents felt more like persistent digital assistants rather than single-request responders.

### **Day 4: Reliability and Observability**

This module addressed one of the most important challenges in AI engineering: ensuring agents behave predictably and transparently. We explored logging, tracing, metrics, and evaluation methods to monitor agent behavior and manage costs over time.

The whitepaper presented a comprehensive evaluation framework built on:

* **Logs** (the diary),
* **Traces** (the narrative), and
* **Metrics** (the health report),

and discussed scalable evaluation techniques like LLM-as-a-Judge and Human-in-the-Loop(HITL) workflows.

In the codelabs, we learned to debug agent behavior using detailed logs and traces, understand why an agent made a particular decision, and evaluate the quality of responses and tool usage using structured scoring methods.

This day provided the essential “production mindset” needed for trustworthy deployment.

### **Day 5: Production-Grade Multi-Agent Systems**

The final module brought everything together. We moved from prototypes to production-ready architecture, learning how to design scalable systems composed of multiple cooperating agents.

The whitepaper explored the operational lifecycle of enterprise-grade agents, focusing on deployment, scaling, and the **Agent2Agent (A2A) protocol**, which enables agents to communicate and collaborate directly with other agents.

In the codelabs, we: built multi-agent systems using A2A, enabled agents to exchange messages and delegate tasks, and deployed an agent to the **Vertex AI Agent Engine** to run as a scalable service on Google Cloud.

This was the moment where all the concepts matured into a clear picture of how a team of agents can function like specialized human teams - each agent doing its part to achieve a complex collective goal.

## The Tech Stack Behind the Agents

From a software engineering perspective, the technology wasn’t entirely new.
The course relied on:

* Jupyter Notebooks
* Python
* MCP client–server architecture
* API and protocol design
* Logging, tracing, and metrics
* Databases for storing context and memory

What stood out as new was the ADK (Agent Development Kit). It functions much like the SDKs we’ve seen in other technology ecosystems - Android, iOS, cloud frameworks, and more. ADK provides a structured environment for defining agent behavior, attaching tools, and connecting agents to LLMs securely through API keys.

## How AI Agents Actually Work

LLMs, by themselves, can only reason over the information they’ve been trained on. They cannot take actions, fetch real-time data, or call external systems.
Agents bridge this gap.

They extend an LLM’s capabilities by:

* gathering fresh, real-world information through tools,
* executing functions or programs,
* performing calculations,
* calling APIs,
* searching the internet, and
* feeding the results back into the LLM for analysis.

In this model, the **LLM acts as the reasoner** - interpreting goals, understanding context, and making decisions - while the **agent functions as the executor**, carrying out steps, coordinating tools, and managing workflows.

This separation of roles is what allows complex, multi-step, and even multi-agent systems to function reliably and continuously, far beyond what a standalone LLM can achieve.

## A Practical Example: Content Creation With Agents

Consider something as common as producing a full article. Traditionally, this involves several steps:

1. researching the topic,
2. gathering reference material,
3. structuring and drafting the piece,
4. creating visuals,
5. editing and refining,
6. and finally publishing it.

A process like this usually demands hours of focused work - or the coordination of multiple people.

With agents, this workflow can be split into a team of specialists working together:

* a **research agent** gathering reliable contemporary information,
* a **title and SEO agent** generating headlines and keywords,
* a **drafting agent** turning findings into structured content,
* an **image-generation agent** creating visuals,
* an **editor agent** refining clarity and tone,
* and a **publishing agent** formatting and posting the final piece.

Each agent has a narrow responsibility, similar to microservices in software systems. This orchestration - driven by an LLM as the “reasoner” and powered by tools as the “executors” - lets machines automate large, multi-step creative workflows with a level of coordination that was never practical before.

What’s interesting is how this reflects a broader industry shift.

## My Final Project: Smart Property Discovery

For the course project, I built a multi-agent system that helps users find housing properties matching their preferences.

Here’s how it worked:

* One agent scraped property data from different sources.
* Specialized tools calculated features such as distance to the nearest station, water body, or bus stop.
* Another agent evaluated whether a property met the client’s requirements.
* If a property scored above a certain threshold, a notification agent emailed the user.

This solution saved significant time in manually filtering through endless property listings - a perfect example of real-world value from agentic systems.

## The Future of Work: What’s About to Shift?

Working through these agent-driven workflow patterns makes it obvious how quickly automated systems are moving from experimental to foundational. I think the shift isn’t about replacing individual tasks here and there - it’s about restructuring *how* work gets done.

### Where Automation Will Accelerate

The workflows denoted in the course - sequential pipelines, parallel research teams, refinement loops - highlight the kinds of tasks that are first-class for automation:

* processes built on repeatable, well-defined steps,
* information gathering that scales with volume,
* decisions shaped by consistent rules rather than deep judgment,
* work that benefits from predictable execution over creativity.

These are exactly the domains where agents outperform traditional manual approaches.

### Roles That Will Grow, Not Shrink

In my opinion, as systems will become increasingly agentic, entirely new categories of work will emerge around:

* designing and composing complex agents,
* building custom tools and function interfaces,
* maintaining long-running agent systems,
* orchestrating workflows across sequential, parallel, and loop structures,
* monitoring and debugging multi-agent pipelines.

We’re moving toward a world where “agent engineer” or “workflow orchestrator” becomes as common a title as “backend developer” is today.

Given the current trajectory, it's reasonable to expect broad adoption within the next **5–10 years**, not because AI is “taking over,” but because organizations will automate any process that can be reliably broken into agent-friendly steps. In fact, I am starting to see this happening in the organization I work for, right in front of my own eyes.

For anyone entering software development now, learning to architect and operate agent systems will be a career advantage for sure.

### A New Development Ecosystem Taking Shape

One of the clearest insights from the course is that we’re watching a new platform era begin. Major companies aren’t just releasing models - they’re introducing full ecosystems for agent construction.

Google’s stack is a prime example: the Gemini models, the [Agent Development Kit (ADK)](https://google.github.io/adk-docs/), workflow-specific agent types (Sequential, Parallel, Loop), custom tools, and orchestration layers. This ecosystemization is familiar in tech:

* Android → Android SDK
* iOS → Xcode and platform frameworks
* Cloud → AWS/GCP/Azure
* Web → React, Angular, Vue

Agentic AI is moving in the same direction. Each vendor is assembling its own “agent stack,” and engineers will eventually choose their agent platform the same way they choose a cloud provider or a frontend framework today.

And it’s not just cloud vendors—open-source ecosystems are emerging too. Frameworks like [AutoGen](https://microsoft.github.io/autogen/stable/index.html) are already enabling multi-agent coordination and tooling without being tied to any specific cloud platform. I’ll explore this more deeply in a separate article, but it’s clear that both proprietary and open-source stacks will shape how the agent landscape evolves.

We’re still early in this shift, but the competition and cross-pollination that follow will shape how agent systems are built for years to come.

### The Foundations Still Matter

Even as agents automate more of the surface-level work, the underlying engineering remains critical. Everything demonstrated in the course - passing state between agents, controlling execution order, managing tools, and handling refinement loops - depends on reliable systems beneath the surface.

Core skills still form the backbone - algorithms and data structures, database and API design distributed system fundamentals, reliability engineering, performance optimization etc.

Agents can coordinate workflows, but they still rely on well-architected infrastructure, predictable interfaces, and resilient systems. The fundamentals aren’t being replaced - they remain the bedrock that makes large-scale agentic automation possible.

## Final Thoughts

Overall, the Agentic AI course by Google on the Kaggle platform was incredibly insightful. It gave me a practical understanding of how agentic systems operate, how the ecosystem is evolving, and what software development might look like in the near future.

I walked away not just with new skills, but with a new perspective on how AI agents will transform workflows, industries, and our roles as developers.


A New Architecture Is Emerging: Inside the Agentic AI Shift in Software Engineering


Recent weeks have raised public concern in Bangladesh after several noticeable earthquakes shook the Dhaka–Narsingdi region. Many people are wondering: *Is this the start of something dangerous? Has seismic activity suddenly increased?* To answer these questions, we analyzed decades of earthquake data across Bangladesh and compared it with the events of 2025. Here is what the numbers — not rumors — actually show.

## **What the long-term data tells us**

A full historical catalog of 465 earthquakes (1976–2025) shows that Bangladesh experiences regular moderate seismicity. The recent activity sits within long-term patterns, but several features of **2025 stand out scientifically**:

### **Energy release in 2025 is higher than average**

This year’s events have released about **1.65× more seismic energy** than the historical yearly average. That increase is driven mainly by the **M5.4 earthquake on November 21**, the largest event of the year.

![img](/images/eqb/seismic_eng_per_yr.png)

### **Event counts are *not* unusually high**

Statistical tests show that **2025 has a normal number of earthquakes** — no spike in the *frequency* of events.

![img](/images/eqb/e_count.png)

### **Magnitudes remain in the typical range**

The distribution of earthquake magnitudes in 2025 does **not differ significantly** from previous decades.

![img](/images/eqb/mag_distribution.png)

![img](/images/eqb/mag_bin.png)

### **Depth is where things get interesting**

Multiple tests show that **2025 earthquakes are noticeably shallower** than in most previous years. Shallow earthquakes are felt more strongly at the surface, even if their magnitudes are moderate.

![img](/images/eqb/median_quake_depth.png)

This is one of the most important findings: **No unusual increase in the number of earthquakes — but the ones occurring are shallower, so they feel stronger.**

![img](/images/eqb/depth_dis.png)

## **Understanding the recent sequence near Dhaka / Narsingdi**

So far, four earthquakes attracted public attention in late November and early December:

| Date (UTC) | Mag | Depth | Location        |
|------------|-----|-------|-----------------|
| Nov 21     | 5.4 | 27 km | Narsingdi (SSW) |
| Nov 22     | 4.3 | 10 km | Narsingdi (W)   |
| Nov 27     | 4.0 | 10 km | Tungi (NNE)     |
| Dec 4      | 4.1 | 10 km | Dhaka (ESE)     |

The **21 November M5.4 earthquake** is almost certainly the **mainshock**, assuming no larger event occurred shortly before or after it. Terms like *mainshock* or *foreshock* are assigned **after** the fact, once the full sequence is known. The three subsequent earthquakes — magnitudes **4.0 to 4.3** over the following two weeks — fit the pattern of **aftershocks**, which is the normal and expected response of the crust after a moderate event like this.

Aftershock models and cluster analysis support this:

- Spatially close
- Shallow
- Occurred soon after the M5.4
- Magnitudes appropriate for aftershocks

There is **no scientific indication** that the sequence is an "ongoing mega-event" or a sign of an imminent major catastrophe.

![img](/images/eqb/quake_clusters.png)

## **Should people be worried?**

### **Short answer: No panic — but yes, be prepared.**

Based on the data:

- There is **no evidence** of a large earthquake building up in the immediate future.
- The pattern fits a **normal mainshock–aftershock sequence**.
- Bangladesh’s long-term seismic environment is active, but this year is not unprecedented.

That said, shallow earthquakes, even at moderate magnitudes, can **shake cities hard**, especially in cities like Dhaka with probably thousands of vulnerable buildings. This means:

- **Risk exists**
- **But it is manageable**
- **Preparedness is the key difference-maker**

## **Why shallow earthquakes matter more**

Our analysis found a statistically significant shift toward shallower depths in 2025. This is important because:

- A shallow M4–M5 can feel like a deeper M6 in terms of surface shaking.
- Dhaka has many old and unreinforced structures that are more sensitive to this kind of shaking.
- Even without a major earthquake, shallow moderate events can cause injuries, wall collapses, and infrastructure disruption.

So the message is not "a big earthquake is coming."
The message is: **a moderate earthquake can still be damaging if buildings are fragile — so strengthen preparedness now.**

## **What people and communities can do right now**

### **A. Stay informed without panic**

Follow official bulletins from:

- Bangladesh Meteorological Department (BMD)
- National Disaster Response agencies
- Universities or credible institutions

**Please avoid unverified social media rumors.**

### **B. Community organization**

- Form **local volunteer preparedness groups** in neighborhoods.
- Identify safe open spaces.
- Help elderly or disabled neighbors with emergency planning.
- Practice evacuation routes.

### **C. Training and drills**

Push local authorities and community leaders to:

- Organize **earthquake safety workshops**.
- Conduct **drop–cover–hold-on drills** in schools, offices, and markets.
- Teach people how to check homes for hazards (loose shelves, falling objects, unsecured gas cylinders).

### **D. Structural safety**

Citizens and journalists should push authorities to:

- Enforce building codes more strictly.
- Inspect vulnerable buildings, especially schools and hospitals.
- Provide public guidance on strengthening old structures.

### **E. Spread helpful information**

Local organizations can print simple, factual pamphlets covering:

- What to do during shaking
- How to prepare emergency kits
- Where to gather after an earthquake
- How aftershocks behave
- What NOT to believe on social media

Helping people understand what is normal in an earthquake sequence reduces fear — and that improves safety.

## **Final takeaway**

**No, Bangladesh is not currently experiencing a dramatic, unprecedented seismic crisis.**
But **yes**, 2025’s earthquakes — especially the shallow M5.4 near Narsingdi — remind us that Bangladesh sits in an active tectonic region where moderate earthquakes can be strongly felt.

- The number of earthquakes is normal.
- Magnitudes are normal.
- **Depths are shallower**, so shaking feels stronger.
- The recent events fit a **mainshock–aftershock pattern**.
- Preparedness, not panic, is the appropriate response.

Earthquakes cannot be stopped, but **knowledge, training, and readiness** can prevent injuries and save lives. The best time to prepare is **before** a larger earthquake happens — and these recent events give Bangladesh a valuable opportunity to do exactly that.

For those interested in the full analysis, code, and data, it is available on [this GitHub repository](https://github.com/mehdihasan/bd-earthquake-analysis-2025).

## Verified Sources Covering 2025 Earthquake

1. [No, Dhaka Is Not At Risk of a 9.0 Earthquake That Would Liquefy the City](https://counterpointbd.substack.com/p/no-dhaka-is-not-at-risk-of-a-90-earthquake) by Asifur Rahman Khan
2. [The Latest Earthquake Was a Warning Sign: Bangladesh Isn’t Ready](https://thediplomat.com/2025/12/the-latest-earthquake-was-a-warning-sign-bangladesh-isnt-ready) by Thomas L. Davis


Is Something Dramatic Happening Underground in Bangladesh? A Data-Based Look at Earthquakes


Connecting to a database from Python seems simple enough - you create a client, send queries, and handle results. But when we scaled that to thousands of concurrent requests under an async web server, things got a bit more interesting.

This post walks through how we hit a classic concurrency issue in our ClickHouse integration - how it appeared, what caused it, the quick fix we applied, and finally how we designed a long-term scalable solution using an async connection pool.

## The Setup

Our backend service is a data ingestion API that pushes concurrent large JSON payloads into ClickHouse. It runs on top of [**Uvicorn**](https://uvicorn.dev/), an ASGI web server that handles requests asynchronously.

We used the official [clickhouse-connect](https://github.com/ClickHouse/clickhouse-connect) library to communicate with the database. In the beginning, the design looked clean and simple:

```python
# Simplified version of the initial design

class CB:

    def __init__(self):
        self.client = AsyncClient(
            host="clickhouse_host",
            port=8123,
            database="DB_NAME",
            username="default",
            password="default"
        )

    async def post_raw(self, payload):
        await self.client.insert(table="m_data", data=payload)
```

We created **a single AsyncClient instance** when the backend started - a singleton pattern. Every request reused this same client to perform queries, inserts and health checks.

We leveraged [ClickHouse's server-side asynchronous insert](https://clickhouse.com/docs/optimize/asynchronous-inserts) feature by configuring the insertion settings to use ***async_insert=1***. This setting is key for high-throughput ingestion, as it tells the [ClickHouse server to buffer and batch data internally](https://clickhouse.com/blog/asynchronous-data-inserts-in-clickhouse). However, as we soon discovered, server-side data handling being asynchronous does not guarantee client-side session safety.

## The Problem Appears

Everything was smooth in development and staging environments. It looked fine - until we hit production and started ingesting thousands of rows per minute, the logs exploded with the following message:

> ***Attempt to execute concurrent queries within the same session. Please use a separate client instance per thread/process.***

Thirty percent of the inserts began failing randomly under load. Some succeeded, others didn't. It wasn't deterministic - but it was bad.

## The Role of Uvicorn and Async Concurrency

Our API application runs on **Uvicorn** web server, which means requests are handled *asynchronously* on a single [event loop](https://uvicorn.dev/concepts/event-loop/) by the [*uvloop* library](https://github.com/MagicStack/uvloop).

In async Python, multiple [coroutines](https://docs.python.org/3/library/asyncio-task.html) (i.e., concurrent requests) share the same event loop and can all run at once when waiting on I/O. So even though we weren't using multiple OS threads (no parallelism), we achieved high concurrency within the same single-threaded process by efficiently switching between waiting requests.

Here's what was happening under the hood:

![img](/images/ch1/image_1.jpg)

Because all requests used the **same Clickhouse AsyncClient instance**, multiple coroutines tried to send different queries over the same underlying TCP session - at the same time. This scenario is a classic example of shared mutable state in an asynchronous environment. Multiple requests raced to use the same underlying socket before the previous query had finished, violating the database client's contract.

ClickHouse rightfully rejected this pattern. Its client session protocol doesn't support multiple concurrent queries per connection. Each query must complete before another begins on the same connection.

It's important to distinguish between ClickHouse's server-side asynchronous insert feature (async_insert=1) and our client-side asynchronous concurrency (Uvicorn/coroutine model). The server-side feature efficiently buffers the data. However, the client driver still requires a non-conflicting network session to send the insert request and receive the response (even if that response is a quick acknowledgement). Our problem was that multiple coroutines were attempting to initiate their separate insert operations over the single shared client session at the exact same moment, which the client/session protocol disallowed. The fix was necessary to resolve the session-level concurrency issue, not the server's data-buffering mechanism.

That's why the error message said:

> ***Please use a separate client instance per thread/process.***

Even though we weren't using multiple threads, we had the same effect: concurrent use of a shared connection.

## Quick Fix: One Client per Request

When production fire alarms are ringing, elegance comes second.

As a **quick fix**, we changed our approach to create a **new ClickHouse client per request**.

```python
async def post_data(self, payload):
    async with AsyncClient(client=get_client(...)) as client:
        await client.insert(...)
```

That immediately solved the concurrency issue. Each request had its own isolated client - and therefore its own TCP session - avoiding shared-state problems completely.

But this (and probably most) quick solution came at a cost.

Creating and closing a ClickHouse client for every single request is expensive. At large ingestion volumes, it increases connection setup overhead, authentication latency, and CPU churn on both the client and server sides.

We needed something more efficient.

## The Final Fix: Connection Pooling

Once stability returned, we revisited the design.

Instead of one global client or one-per-request, we moved to a **connection pool** - maintaining a fixed number of ready-to-use ClickHouse clients and reusing them efficiently.

![img](/images/ch1/image_2.jpg)

### New approach

```python
class PooledClient:
    """Helper Context Manager Class"""

    def __init__(self, pool, client):
        self._pool = pool
        self.client = client

    async def __aenter__(self):
        """Called when 'async with' starts"""
        return self.client

    async def __aexit__(self, exc_type, exc_val, exc_tb):
        """
        Called when 'async with' block ends (even on error).
        For simplicity, we just return the client to the pool.
        (You could add logic here to replace broken clients.)
        """
        if not getattr(self.client, "is_closed", False):
            await self._pool.put(self.client)
        return False


class ClickhousePool:
    """Main Connection Pool Class"""

    def __init__(self):
        self._pool = asyncio.Queue(maxsize=SIZE)
        for _ in range(SIZE):
            client = AsyncClient(client=get_client(...))
            self._pool.put_nowait(client)

    async def get_client(self):
        """Returns a context manager object that handles the client lifecycle"""
        return PooledClient(self._pool, await self._pool.get())

    async def post_data(self, payload):
        """The Clean Consumption in the API"""
        rows = ...  # transformations of payload
        table_name = ...
        columns = ...

        async with self.get_client() as client:
            await client.insert(
                table=table_name,
                data=rows,
                column_names=columns,
                settings={
                    'async_insert': 1,
                    'wait_for_async_insert': 0,
                    # ... other settings
                }
            )
```

Once the connection pool is in place, usage of async API is straightforward:

```python
cp = ClickhousePool()
await cp.post_data(payload)
```

With the pooled design we make sure that, (1) each of the requests borrow a client from the pool, (2) they use it to execute a query or insert, (3) the client is released back into the pool afterward, (4) if a client breaks (due to a network or protocol error), it's replaced with a new one.

This hybrid approach provides safety by not sharing clients across concurrent requests, performance by avoiding recreate clients repeatedly, scalability by easily tune pool size to match workload.

```txt
Singleton        ->  One global AsyncClient
Per-request      ->  New client per request
Connection pool  ->  Limited shared pool, reused safely
```

## Validation and Results

Once deployed, the difference was clear.

We validated the new behavior in three ways. Error messages related to ***concurrent queries within the same session*** disappeared completely. Connection metrics stabilized - fewer reconnections, more consistent query latencies. The ClickHouse row count graphs started growing linearly again - confirming ingestion pipelines were back to full capacity.

## Lessons Learned

One of the key takeaways from this experience was realizing that asynchronous code isn't automatically thread-safe or concurrency-safe. Using async functions doesn't guarantee that shared resources can be accessed safely by multiple coroutines. In our case, reusing a single AsyncClient across concurrent requests seemed fine at first, but when multiple coroutines tried to use it simultaneously, it led to unexpected failures. The event loop can rapidly switch between these coroutines, and if the underlying resource isn't designed for concurrent access, it will eventually break - exactly as we saw with the ClickHouse client.

We also learned that Uvicorn's speed comes from concurrency, not parallelism. It multiplexes many requests on a single event loop in one process. That means even though the server feels lightweight and fast, all those requests are sharing the same runtime environment. Any global object - such as a database connection, HTTP session, or cache client - must be carefully managed to ensure it's concurrency-safe. The illusion of simplicity in async code can be deceptive; behind the scenes, coroutines are continuously yielding and resuming, which magnifies issues caused by shared mutable state.

When production problems strike, it's often tempting to apply quick fixes to restore stability. We did the same by creating one client per request - a brute-force but effective short-term solution. It isolated each operation and immediately eliminated concurrency conflicts. However, that approach didn't scale well. Each new client meant an additional connection handshake, authentication cycle, and network setup overhead. It worked to stop the bleeding, but it wasn't efficient enough for a high-throughput ingestion system. A connection pool, on the other hand, gave us the right balance - isolation without excessive overhead.

Finally, this journey reinforced that observability is critical. Without clear logs, metrics, and dashboards, it would have been nearly impossible to pinpoint the root cause or verify the effectiveness of our fix. Our Grafana dashboards, error logs, and ClickHouse ingestion metrics made it clear when the issue appeared, how severe it was, and how things stabilized after deploying the pooled solution. Good observability turned what could have been a long debugging marathon into a structured diagnosis and validation process.

## Closing Thoughts

This issue was a great reminder that even high-level async code hides some deep concurrency challenges underneath.

The evolution - from a single shared client, to per-request isolation, to an efficient async connection pool - illustrates how understanding the **runtime model** (Uvicorn + async I/O) is just as important as writing correct business logic.


Mehdi Hasan

Kafka Connect TLS Certificate Error: A Real Production Story

A New Architecture Is Emerging: Inside the Agentic AI Shift in Software Engineering

Is Something Dramatic Happening Underground in Bangladesh? A Data-Based Look at Earthquakes

Fixing Concurrent Queries Within the Same Session in ClickHouse Async Python Apps (Uvicorn + FastAPI)