Problem

Hyper-Q reports the error code and message:

DTM3116: Lost connection to underlying database: Error Message: [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: Error code 0x68[Microsoft][ODBC Driver 17 for SQL Server]Communication link failure SQLSTATE IS: 08S01, Backend Error Code: 104, SQL State: 08S01

Other error messages related to this can include:

DTM9034","ERROR","Client connection #1553404 lost",""

"DTM1018","ERROR","State machine <0.2237.1663> terminating. Reason: {sds_event,""DTM3116"",{sds_msg,3116,sds_msg_sev_fatal,""Lost connection to underlying database: ~ts"",""Lost connection to underlying database""},odbc_beta,848,undefined,[76,111|...],undefined}. Last event: [{'EXIT',<0.9640.1645>,{sds_event,""DTM3116"",{sds_msg,3116,...},odbc_beta,848,...}}]. State: waiting_for_request. Data: tdc_vr_state.",""

Cause

It is not uncommon to experience a dropped connection in Microsoft Azure during the execution of extract, transform, load (ETL) workloads from both an on-premise server to Azure cloud, where Datometry Hyper-Q and the Azure Synapse Analytics database are provisioned and running, or if the ETL server is in Azure as well.

The cause of this is most often:

• Network instability such as buffer size from the ETL server to Azure cloud.
• Workload management—or the lack thereof—in the Azure Synapse environment.
• Memory or I/O issues on the ETL server.

Solution

1. File a support request with Microsoft that includes the following information: date, timestamp, client tool being used, error message, and Azure Synapse resource consumption (I/O and memory usage).

2. Maintain a spreadsheet of all dropped connections and the above metrics to review with Microsoft Support, as this will occur frequently.

3. Look for patterns in ETL jobs and timestamps associated with dropped connections, including concurrency on the system.

4. If the patterns indicate dropped connections during periods of heavy loads on the system, put in place workload management or tune the existing workload management on Azure Synapse.

5. If the dropped connections occur during heavy memory and I/O consumption on the ETL server, adjust the workload and check for network and buffer issues from the on-premise server to Azure cloud. You may need to file a support request with the Microsoft network and infrastructure teams.