John Liu Blog

Every drop counts

Generate Self-signed Certificate

There are time we might need a certificate, for example, to set up Azure Service Principle without using secret. We can use PowerShell or OpenSSL to generate self-signed certificate. # 1. Config $certname = "SQLDataExportCert" # password for secure the private key file # use single quote to quote the string, to handle special character like $ $pwdstring = 'YourSecurePassword' # Define the expiration date (e.g., 5 years from today) $expiryDate = (Get-Date).

Posted by John Liu Tuesday, December 30, 2025

SQL Generate Parquet File

There are several ways to generate Parquet file from SQL. PolyBase can generate Parquet file to S3 or Azure but we can’t specify the file name generated. Refer to SQL Export to parquet file. Azure Data Factory (ADF) or Fabric pipe line Copy Activity can generate Parquet file with specific file name. Python script can generate Parquet file with specific name. Following are examples using Python script in SQL to generate the Parquet file.

Posted by John Liu Friday, December 26, 2025

SQL Read Parquet File

There are several ways to read Parquet file within SQL Server. 1. Using OPENROWSET (SQL2022+) Starting with SQL Server 2022, we can use OPENROWSET to query Parquet files directly from Azure Blob Storage, ADLS Gen2, or S3-compatible storage without creating a permanent table first. -- 1. Create credential CREATE DATABASE SCOPED CREDENTIAL [MyAzureCredential] WITH IDENTITY = 'SHARED ACCESS SIGNATURE', SECRET = 'sv=2022-11-02&ss=b&srt=sco&sp=rwdl&se=2025-12-31...'; -- 2. Create the Data Source pointing to your container CREATE EXTERNAL DATA SOURCE MyCloudLogs WITH ( LOCATION = 'abs://yourcontainer@yourstorageaccount.

Posted by John Liu Friday, December 26, 2025

SQL Query External Dta

Read text file contents as a single value: SELECT [BulkColumn] FROM OPENROWSET(BULK 'C:\TEMP\test.json', SINGLE_CLOB) AS Contents Read data from Excel: Option1: Using OPENROWSET. Note: if your column has data more than 255 characters, try to format the column as Text. Also try to change following registry key TypeGuessRows to 0 (default is 8), under path HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\16.0\Access Connectivity Engine\Engines\Excel (for 64-bit machine) or HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Office\16.0\Access Connectivity Engine\Engines\Excel (for 32-bit machine). Restart of SQL instance required if change registry setting.

Posted by John Liu Wednesday, June 4, 2025