Skip to main content

Part 3: Joining on Encrypted Data: Deterministic, Probabilistic and Homomorphic Encryption on AWS

Encrypted by Design: Securing Data in Modern Pipelines

Updated 20 min readMarkdown
Part of seriesEncrypted by Design: Securing Data in Modern Pipelines
Part 3: Joining on Encrypted Data: Deterministic, Probabilistic and Homomorphic Encryption on AWS

Where we left off

In the last post we covered encryption at rest and encryption in transit. Those two are the floor. Turn on SSE-KMS, force TLS, enforce aws:SecureTransport in your bucket policy, and you've handled the "someone walked off with a disk" and "someone sniffed the wire" problems.

But here's what that floor doesn't do. Once an analyst with read access to the bucket runs SELECT * FROM customers, S3 happily decrypts everything for them. Encryption at rest protects the storage. It does nothing for the data once someone is allowed to read it.

That's why I promised at the end of the last post that we'd look at encryption applied to the values themselves: deterministic, probabilistic and homomorphic. Back in Part 1 I also said we'd build something practical on AWS, so this post does both. We'll go through how each technique behaves, what it leaks (every one of them leaks something), and then write a working PySpark job that encrypts PII columns in a Glue pipeline using KMS.

The whole post really comes down to one question you have to answer for every PII column:

Do I need to compare this value with other values, or do I only need to hide it?

Hold on to that. Everything below is about how the answer changes your choice.

A quick refresher on field-level encryption

With field-level (or column-level) encryption, your pipeline encrypts specific columns before the data lands in the lake. The Parquet file in S3 contains ciphertext in the email column, not an email address. The bucket can still have SSE-KMS on top. Now you have two layers: the storage is encrypted, and the sensitive values inside are encrypted with a key that most of your users never get access to.

The catch is that analysts still need to do useful things with that data. They need to join customers to transactions on customer_id. They need to count distinct emails. And that's where the type of encryption starts to matter a lot.

Deterministic encryption: same input, same output

Deterministic encryption means that if you encrypt kishan@example.com today, and again next month, with the same key, you get the exact same ciphertext both times.

That one property is why it's so popular in data platforms:

  • You can join two tables on an encrypted column, because matching plaintexts produce matching ciphertexts.
  • You can run GROUP BY, COUNT(DISTINCT ...) and equality filters (WHERE email_enc = '...') without ever decrypting.
  • Authorized users or services can still decrypt it, because it's real encryption, not a hash.

Which algorithm?

Please don't build this yourself by taking AES and hardcoding the IV. People really do that, and it breaks badly. The algorithm built for this job is AES-SIV (RFC 5297). SIV stands for "synthetic initialization vector": instead of a random IV, the algorithm derives one from the plaintext itself, so the output is deterministic by design and still authenticated. If someone tampers with the ciphertext, decryption fails instead of quietly returning garbage.

You'll find AES-SIV in several places:

  • Python's cryptography library (AESSIV, available since version 37)
  • Google Tink, under the name "Deterministic AEAD"
  • BigQuery's DETERMINISTIC_ENCRYPT SQL function, which is built on Tink

What deterministic encryption leaks

This part is easy to skip, and it shouldn't be. Because equal values produce equal ciphertexts, an attacker who can see the encrypted column learns which rows share a value and how often each value appears. They don't learn the value itself, but frequency alone can give a lot away.

Picture a gender column encrypted deterministically. There are maybe three or four distinct ciphertexts. If an attacker knows the rough population split, they can guess which ciphertext is which in seconds. The same goes for state, country, marital_status, or any column with a small number of possible values. Researchers showed this back in 2015 with real hospital records ("Inference Attacks on Property-Preserving Encrypted Databases", Naveed, Kamara and Wright). They recovered a large share of deterministically encrypted attributes using nothing more than public statistics.

So the rule I follow:

Use deterministic encryption only on high-cardinality identifiers you actually need to join or match on. Customer IDs, emails, phone numbers, account numbers, PAN numbers. Not demographics.

This rule came out of a real config review. Our pipelines were metadata driven: each column had an entry saying whether it should be encrypted deterministically, probabilistically, or left alone, and the Glue job just followed the config. During one review I found an account type column marked deterministic, because someone downstream wanted to group by it. It had maybe five distinct values, and one of them covered most of the rows. Anyone who knew roughly how a retail bank's book is split could have mapped every ciphertext back to its value in a few minutes, and then that column became a hint for everything else in the row. We switched it to probabilistic and gave the reporting team a pre-aggregated view instead. Since then, "how many distinct values does this column have?" is one of the first questions I ask whenever a column gets marked deterministic.

Two more leaks are worth knowing about:

  1. Length. AES-SIV output is the plaintext length plus 16 bytes. A 3-character value and a 40-character value look different. If length matters for a column, pad the values before encrypting.
  2. Cross-column matching. If you use the same key for email and secondary_email, the same address in both columns produces the same ciphertext. Sometimes you want that. Often you don't. We'll fix this with associated data below.

Normalize before you encrypt

This one has caught more teams than any cryptographic weakness. Deterministic encryption is exact. Kishan@Example.com, kishan@example.com, and the same address with a stray space at the end are three different plaintexts, so they give different ciphertexts, and your join quietly drops rows.

I learned this one the hard way. On a migration for a banking client, we were moving customer data from a legacy system into S3, and the same customer ID came from two sources: a mainframe extract that zero padded it to 12 characters, and a CRM feed that didn't pad it at all. Both went through our masking framework and both got encrypted correctly. The pipeline ran green. It was only when our reconciliation job compared row counts after the join that we saw a big chunk of customers simply missing. Nothing was wrong with the encryption. 000045871203 and 45871203 are just different strings, and the ciphertexts were faithfully different too. The fix was boring: we added a normalization rule per column in the job's metadata config, so every source went through the same trim and pad step before encryption. But if we hadn't had reconciliation in place, those missing rows would have gone straight to the reporting layer.

Decide on a normalization rule for every deterministic column and apply it in the same place, every time:

  • Email: trim, lowercase
  • Phone: convert to E.164 (+919876543210)
  • IDs: trim, consistent zero padding, consistent type (don't encrypt 00123 in one table and 123 in another)

Once a value is encrypted you can't fix this later without decrypting everything, so get it right up front.

Probabilistic encryption: same input, different output every time

Probabilistic (or randomized) encryption mixes a fresh random value, a nonce, into every encryption. Encrypt kishan@example.com twice and you get two completely different ciphertexts. Both decrypt back to the same email.

This is what you should default to for anything you only need to store and retrieve: addresses, dates of birth, national ID numbers that nobody joins on, free-text notes, medical details. An attacker looking at the column learns nothing about duplicates or frequencies.

What you give up is every kind of comparison. No joins, no GROUP BY, no equality filter. The database can't tell that two ciphertexts hold the same value, because that's exactly what the scheme is designed to hide.

The standard choice is AES-256-GCM with a random 96-bit nonce. Two rules are non-negotiable:

  1. Never reuse a nonce with the same key. With GCM, a repeated nonce doesn't just leak a little. It can expose the XOR of the two plaintexts and let an attacker forge messages. Generate it with os.urandom(12) every single time.
  2. Don't encrypt forever under one key. NIST guidance for random nonces caps it at about 2^32 encryptions per key. For a big data lake that's a real number. Rotate data keys per dataset, per partition, or per job run.

This is also the scheme S3, RDS and EBS use under the hood for encryption at rest. We're just applying the same idea at the column level, with a key we control.

The pattern that gives you both: blind indexes

What if you need to join on a column and you're nervous about deterministic encryption? There's a well-known pattern for this, and AWS uses it in its own products.

Store two columns:

  • email_enc: probabilistic encryption (AES-GCM). Reversible, leaks nothing on its own.
  • email_idx: a keyed hash of the normalized email (HMAC-SHA256 with a separate key). Deterministic, so you join on it, but it can't be decrypted at all.

Analysts join and group on email_idx. Only the small set of services that truly need the real email get access to the key that decrypts email_enc.

The keyed part is important. A plain SHA-256 of an email can be brute-forced, because the space of plausible emails isn't that big. An HMAC with a secret key can't be precomputed by someone who doesn't have the key.

You've seen this pattern if you've used either of these:

  • AWS Clean Rooms, Cryptographic Computing (C3R). Columns you join on are "fingerprint columns" protected with HMAC. Columns you only select are "sealed columns" protected with AES-GCM. Same split as above.
  • AWS Database Encryption SDK for DynamoDB calls these "beacons": truncated HMACs that let you query encrypted attributes.

The blind index still has the frequency leak, since equal emails give equal HMACs. But you've separated "can match" from "can read", and that's a real improvement over handing everyone a decryptable deterministic column.

Building it on AWS: KMS, envelope encryption and a Glue job

Let's put this into an actual pipeline. The setup is:

  • A KMS key (alias/pii-field-encryption) that never leaves KMS.
  • A data key generated from it once. We store only the encrypted copy of the data key. This is envelope encryption, the same thing S3 does with SSE-KMS, which we walked through in the last post.
  • A Glue PySpark job that decrypts the data key through KMS once per task and encrypts columns locally. Calling KMS once per row would be painfully slow and would hit KMS request quotas almost immediately.

Step 1: Generate and wrap the data key (one-time)

import base64
import boto3

kms = boto3.client("kms")

CONTEXT = {"purpose": "pii-deterministic", "key_version": "v1"}

resp = kms.generate_data_key(
    KeyId="alias/pii-field-encryption",
    NumberOfBytes=64,            # AES-SIV with AES-256 needs a 512-bit key
    EncryptionContext=CONTEXT,
)

wrapped_key_b64 = base64.b64encode(resp["CiphertextBlob"]).decode()
print(wrapped_key_b64)

# resp["Plaintext"] is the raw key. Don't log it, don't save it, let it go out of scope.

The wrapped key is safe to store in SSM Parameter Store or pass as a Glue job argument. It's useless without kms:Decrypt on that KMS key and the exact same encryption context.

That encryption context matters more than it looks. KMS binds it to the ciphertext, so decryption fails unless the caller supplies the same key/value pairs. Trying to unwrap the key with a different purpose, or with no context at all, gets rejected:

unwrap with correct context: OK
unwrap with wrong context:   InvalidCiphertextException
unwrap with no context:      InvalidCiphertextException

It also shows up in CloudTrail, so every decrypt call tells you which purpose and key version it was for.

Step 2: Encrypt and decrypt helpers

import base64
import os
from cryptography.hazmat.primitives.ciphers.aead import AESSIV, AESGCM

KEY_VERSION = "v1"


def det_encrypt(siv: AESSIV, field: str, value: str) -> str:
    """Deterministic: same (field, value) always produces the same token."""
    ct = siv.encrypt(value.encode("utf-8"), [field.encode("utf-8")])
    return f"{KEY_VERSION}:{base64.urlsafe_b64encode(ct).decode()}"


def det_decrypt(siv: AESSIV, field: str, token: str) -> str:
    version, payload = token.split(":", 1)
    ct = base64.urlsafe_b64decode(payload)
    return siv.decrypt(ct, [field.encode("utf-8")]).decode("utf-8")


def prob_encrypt(gcm: AESGCM, field: str, value: str) -> str:
    """Probabilistic: a fresh nonce every call, so output differs every time."""
    nonce = os.urandom(12)
    ct = gcm.encrypt(nonce, value.encode("utf-8"), field.encode("utf-8"))
    return f"{KEY_VERSION}:{base64.urlsafe_b64encode(nonce + ct).decode()}"


def prob_decrypt(gcm: AESGCM, field: str, token: str) -> str:
    version, payload = token.split(":", 1)
    raw = base64.urlsafe_b64decode(payload)
    nonce, ct = raw[:12], raw[12:]
    return gcm.decrypt(nonce, ct, field.encode("utf-8")).decode("utf-8")

A few choices in there are deliberate:

The field name goes in as associated data. Associated data is authenticated but not encrypted. Here it ties a ciphertext to a logical field. Copy an encrypted email into the phone column and decryption fails. It also means the same value in two different fields produces different deterministic tokens, which fixes the cross-column leak from earlier.

One subtlety: for joins to work, both tables have to use the same associated data. So use the logical field name (customer_id), not the table name (crm.customers.customer_id). Otherwise the same customer ID encrypts differently in each table and the join returns nothing.

Every token carries a key version prefix. v1:... looks like a small thing until the day you rotate keys. With deterministic encryption, rotation is painful: tokens made with v2 won't match tokens made with v1, so a join across old and new data silently breaks. The prefix at least tells you which key made which token, so you can re-encrypt in a controlled way instead of guessing.

The deterministic and probabilistic keys are separate. AES-SIV wants 64 bytes and AES-GCM wants 32, so they couldn't share a key anyway. But even if they could, keep them apart. Separate keys mean separate IAM policies, which means you can give the analytics role "may match" without giving it "may read".

Quick sanity check:

siv = AESSIV(AESSIV.generate_key(bit_length=512))
gcm = AESGCM(AESGCM.generate_key(bit_length=256))

email = "kishan@example.com"

print(det_encrypt(siv, "email", email))
print(det_encrypt(siv, "email", email))   # identical to the line above
print(prob_encrypt(gcm, "email", email))
print(prob_encrypt(gcm, "email", email))  # different every time

print(det_encrypt(siv, "secondary_email", email))  # different from "email" field
v1:F3-fKOnskiobegoch33L2E2QSxl0-HM7m1Esc00bI3rvIg==
v1:F3-fKOnskiobegoch33L2E2QSxl0-HM7m1Esc00bI3rvIg==
v1:h7niPIMLorsr4btW-dBhTl1DHQcT5NxrFXdNVwpjyDmbocgh865mjMlM35aO_g==
v1:dEOqcLaHN0f-8VmgUbSJRBMCpgUz7GV_B70-Gx3wx3j4fwQYJpnbnhrz9jQXcg==
v1:ffHrFbt4raKeyOBvqvPtjOrnX3UIx5Et3yFYggFixwGXcw==

The first two lines are identical, the next two are different, and the last one is different from the first two even though it's the same email. And if you try to decrypt an email token as if it were a phone value, you get InvalidTag instead of garbage.

The zero padding problem from earlier looks like this in practice:

'000045871203' -> v1:CdWAAJdth6fKm_2sWo3yKdOAnnGL105GdS6L3w==
    '45871203' -> v1:jjCG5mnrNBkChpdfHYzJrRs8AmOs5oWx

Same customer, two completely unrelated tokens, and a join that quietly drops rows.

Step 3: The Glue job

import sys
import base64

import boto3
from awsglue.utils import getResolvedOptions
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.types import StringType

args = getResolvedOptions(
    sys.argv, ["wrapped_det_key", "source_path", "target_path"]
)

WRAPPED_KEY = args["wrapped_det_key"]
CONTEXT = {"purpose": "pii-deterministic", "key_version": "v1"}
KEY_VERSION = "v1"

# Which columns to encrypt, and how to normalize each one first.
# In a metadata-driven framework this would come from a config table.
PII_COLUMNS = {
    "email": lambda c: F.lower(F.trim(c)),
    "customer_id": lambda c: F.trim(c.cast("string")),
}

_cache = {}


def _siv():
    # Runs on the executors. The key is unwrapped once per task
    # (roughly once per partition), not once per row.
    if "siv" not in _cache:
        from cryptography.hazmat.primitives.ciphers.aead import AESSIV

        plaintext_key = boto3.client("kms").decrypt(
            CiphertextBlob=base64.b64decode(WRAPPED_KEY),
            EncryptionContext=CONTEXT,
        )["Plaintext"]
        _cache["siv"] = AESSIV(plaintext_key)
    return _cache["siv"]


def det_udf(field: str):
    aad = [field.encode("utf-8")]

    def _encrypt(value):
        if value is None:
            return None
        ct = _siv().encrypt(value.encode("utf-8"), aad)
        return f"{KEY_VERSION}:{base64.urlsafe_b64encode(ct).decode()}"

    return F.udf(_encrypt, StringType())


spark = SparkSession.builder.getOrCreate()
df = spark.read.parquet(args["source_path"])

encrypted = df
for column, normalize in PII_COLUMNS.items():
    if column in df.columns:
        encrypted = encrypted.withColumn(
            column, det_udf(column)(normalize(F.col(column)))
        )

encrypted.write.mode("overwrite").parquet(args["target_path"])

If your Glue version doesn't ship a recent enough cryptography, add --additional-python-modules cryptography to the job parameters.

The job only touches the PII columns that exist in the table it's given, so you can point the same job at customers and at transactions. Here's what came out when I ran it on a small sample. Note that the source had " C1001 " with spaces in one table and "C1001" in the other, and they still ended up with the same token:

== customers_enc ==
+-------------------------------+------+---------------------------------------------------+
|customer_id                    |name  |email                                              |
+-------------------------------+------+---------------------------------------------------+
|v1:tCKKHWyRA7EI1T7f5uaZ7LorOc7z|Kishan|v1:EHFDgnZY4OMr1iLdMYOKWQARazpG3xOgMQXALG59DLCEPA==|
|v1:9qJJmFxrj4MUmQXCIiQMu1YrUoJt|Priya |v1:Tv7OlFkiHUSmbFq3IuT3Y_f00e21IETofWJRkBy7C1rz    |
|v1:efY9rtFUxzBE44K5MucBfLZOLoiI|Arjun |NULL                                               |
|v1:Gs947E41voys6nQMqVz90R01DVUE|Meera |v1:nUm9Bev0n83X5L8mcdfPXFAw16y58FuDHLi5EmN-6e3j    |
+-------------------------------+------+---------------------------------------------------+

== transactions_enc ==
+------+-------------------------------+------+
|txn_id|customer_id                    |amount|
+------+-------------------------------+------+
|1     |v1:tCKKHWyRA7EI1T7f5uaZ7LorOc7z|1200.0|
|2     |v1:tCKKHWyRA7EI1T7f5uaZ7LorOc7z|350.5 |
|3     |v1:9qJJmFxrj4MUmQXCIiQMu1YrUoJt|999.0 |
|4     |v1:efY9rtFUxzBE44K5MucBfLZOLoiI|75.25 |
|5     |v1:Gs947E41voys6nQMqVz90R01DVUE|410.0 |
+------+-------------------------------+------+

With the same normalization and the same key on both tables, this works on encrypted data in Athena or Redshift Spectrum:

SELECT c.customer_id, SUM(t.amount) AS total_spend
FROM customers_enc c
JOIN transactions_enc t
  ON c.customer_id = t.customer_id
GROUP BY c.customer_id;
+-------------------------------+-----------+
|customer_id                    |total_spend|
+-------------------------------+-----------+
|v1:9qJJmFxrj4MUmQXCIiQMu1YrUoJt|999.0      |
|v1:Gs947E41voys6nQMqVz90R01DVUE|410.0      |
|v1:tCKKHWyRA7EI1T7f5uaZ7LorOc7z|1550.5     |
|v1:efY9rtFUxzBE44K5MucBfLZOLoiI|75.25      |
+-------------------------------+-----------+

Nobody running that query sees a real customer ID. The totals are right (C1001 had two transactions, 1200 + 350.5), and only a service holding the key can turn those tokens back into C1001, C1002 and so on.

A note on nulls: the UDF leaves nulls as nulls, so an observer can tell which rows have a value and which don't. For most columns that's fine. If "this customer has no phone number on file" is itself sensitive, encrypt a sentinel value instead.

Step 4: Lock down who can decrypt

The Glue role needs kms:Decrypt on the key, but only for this purpose. Condition the policy on the encryption context:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowDecryptForDeterministicPII",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:ap-south-1:111122223333:key/your-key-id",
      "Condition": {
        "StringEquals": {
          "kms:EncryptionContext:purpose": "pii-deterministic"
        }
      }
    }
  ]
}

Your analysts' role gets no permission on this key at all. They query ciphertext through Lake Formation like any other column. The small number of services that really need plaintext (say, the one that sends emails to customers) get their own role with decrypt access. Each of those decrypt calls then lands in CloudTrail with the encryption context attached.

That's what field-level encryption gets you over storage-level encryption. With SSE-KMS, read access to the bucket means read access to the PII. Here, read access to the bucket gets you ciphertext, and seeing the actual values is a separate permission that leaves an audit trail.

Homomorphic encryption: computing on data you can't read

Now for the one everyone finds exciting.

Homomorphic encryption lets you do math on ciphertexts. Add two encrypted numbers and you get an encrypted sum. Decrypt that sum and it's correct. The server that did the addition never saw the inputs or the output.

There are roughly three types:

  • Partially homomorphic supports one operation without limits. Paillier supports addition, which is enough for sums, counts and averages.
  • Somewhat homomorphic supports both addition and multiplication, but only up to a limited depth before noise makes the result unusable.
  • Fully homomorphic (FHE) supports any computation. Schemes like BFV and BGV work on integers, CKKS works on approximate real numbers (good for ML), and TFHE works on bits.

You can try the partially homomorphic case in a few lines with the phe library (python-paillier):

from phe import paillier

public_key, private_key = paillier.generate_paillier_keypair(n_length=2048)

# Branch offices encrypt their salary totals with the public key
encrypted = [public_key.encrypt(x) for x in [82000, 91000, 67500]]

# A central service adds them up without being able to read any of them
encrypted_total = sum(encrypted[1:], encrypted[0])

# Only the private key holder sees the result
print(private_key.decrypt(encrypted_total))  # 240500

It works, and it's honestly a little magical the first time you run it.

So why isn't everyone using it?

Cost. With a 2048-bit Paillier key, each ciphertext is 4096 bits, so a number that took 8 bytes now takes about 512 bytes. Every operation is big-number modular arithmetic instead of a single CPU instruction. Fully homomorphic schemes cost far more, often several orders of magnitude slower than the same computation on plaintext, although libraries like Microsoft SEAL, OpenFHE and Zama's TFHE-rs have closed a lot of the gap.

There's also no managed AWS service that runs your Athena or Redshift queries homomorphically. AWS Clean Rooms C3R sometimes gets mentioned in the same breath, but as we saw above, it uses HMAC fingerprints and AES-GCM sealed columns. That's the deterministic/probabilistic split from this post, not homomorphic encryption.

Where FHE actually makes sense today is narrow but real:

  • Private aggregation across organizations, like banks computing fraud statistics together without sharing customer records.
  • Private ML inference, where a client sends encrypted features to a model and gets back an encrypted prediction.
  • Private set intersection, where two parties find their common customers without revealing the rest of their lists.

In Part 1 I called homomorphic encryption a glimpse into the future, and for a typical data lake I’d still treat it that way: worth understanding, worth watching, and not what you reach for when you’re building next quarter’s pipeline.

Putting it together: what to use where

Deterministic (AES-SIV) Probabilistic (AES-GCM) Blind index (HMAC) Homomorphic
Same input, same output? Yes No Yes No
Joins / GROUP BY / equality Yes No Yes Limited, slow
Reversible Yes, with key Yes, with key No Yes, with private key
Leaks frequencies Yes No Yes No
Performance Fast Fast Very fast Slow to very slow
Good for IDs, emails, phones you join on Addresses, DOB, notes, anything you just store Join keys when you want "match" without "read" Aggregates across untrusted parties

And here's the column-by-column checklist I'd actually use:

  1. Does anyone need to join, group or filter on this value?
    • No → probabilistic. You're done.
  2. Yes. Is it high cardinality? (many distinct values, no single value dominating)
    • No → don't encrypt it deterministically. Generalize it, bucket it, or rethink whether it belongs in the analytics layer at all.
  3. Yes. Does anyone downstream need the original value back?
    • No → blind index (HMAC) only.
    • Yes → deterministic encryption, or blind index plus a probabilistic copy if you want to separate "match" from "read".
  4. Do you need math on the values across parties that don't trust each other?
    • That's when homomorphic encryption, or a clean room, belongs on the table.

What's next

We now have reversible, joinable, column-level encryption running in a Glue job. But we've quietly skipped the hardest part of all of this: the keys. How do you rotate a deterministic key without breaking every join in your warehouse? How do you handle "right to be forgotten" requests when the same customer ID shows up encrypted in 40 tables? And where does tokenization with a proper token vault start to beat encryption altogether?

That's the next post: key management, rotation and crypto shredding for PII pipelines on AWS. See you there.

If you've run deterministic encryption in production and hit a problem I didn't cover, tell me in the comments. I'd like to know what broke.

About the author

  • Kishan is a seasoned Data Engineer with strong expertise in building scalable, cloud-native data solutions on AWS. He brings over 4 years of hands-on experience delivering enterprise-grade data ingestion, transformation, masking, and reconciliation frameworks for top global clients in the banking and financial services sector. Currently working at Deloitte as a Consultant, he leads a high-impact team in designing metadata-driven data pipelines, automating end-to-end workflows using AWS Glue, Lambda, Step Functions, and enforcing granular RBAC policies via Lake Formation. His work has contributed to optimised data lake architectures, enabling 10–30 TB scale migrations from legacy systems to AWS S3, while ensuring compliance, security, and performance. Previously at Infosys, Kishan played a core engineering role in modernising data infrastructure for a leading US bank, delivering reusable ETL frameworks, QuickSight automation, and high-performance Python microservices. With deep knowledge of PySpark, Python, SQL, and tools like Airflow, Redshift, Aurora, and DBT, he has led the design of automated monitoring, alerting, and reconciliation systems with significant reductions in manual effort and latency. Certified across AWS and Microsoft Azure platforms, Kishan combines strong programming skills with architectural thinking, enabling seamless collaboration across business and technical teams. With a focus on scalability, automation, and data integrity, he thrives in fast-paced environments that demand innovation and execution.

No comments yet. Be the first to comment.

Encrypted by Design: Securing Data in Modern Pipelines

Part 3 of 3

A hands-on series for data engineers on protecting PII in cloud data platforms. It starts with a map of the main techniques, goes deep on encryption at rest and in transit on AWS, and then gets into encrypting individual columns so you can still join and analyze the data. Every part comes with working AWS examples.

Start from the beginningPart 1: A Data Engineer’s Map of PII Protection: Encryption, Masking, Tokenization and MoreEncrypted by Design: Securing Data in Modern Pipelines