How We Pushed CDC into Postgres

(snowflake.com)

133 points | by craigkerstiens 15 hours ago

11 comments

  • bastawhiz 13 hours ago
    Clickhouse really nailed this with the acquisition of peerdb. I used it with many terabyte databases and I essentially never thought about it. The only thing we really had to watch for was trying to replicate too much at once (because of the physical compute/io capacity of the postgres or clickhouse clusters).
  • gopalv 13 hours ago
    This was basically Vertica's party trick for quite a long time to have a WOS and ROS formats for the same row and anti-caching between those two.

    You could've built a similar system with dezebium and delta lake for quite some time but it would fail compactions, if you run it fast enough. I've seen Oracle GoldenGate 12c do this trick in 2014 or so, using Mysql as the cheap replica. But they are all fragile to schema updates in some direction.

    The closest batteries-included equivalent to this is the Aurora -> Redshift bridge[1].

    [1] - https://aws.amazon.com/rds/aurora/zero-etl/

    • bastawhiz 13 hours ago
      Aurora zero etl was a nightmare for us. Almost any schema changes require a VACUUM FULL for it to continue functioning. On a few occasions, it just stopped running without an obvious explanation, requiring slow and lengthy back and forth threads with AWS support. If it worked as advertised, it would be great, but I can't recommend it for any serious production system.
  • hasyimibhar 5 hours ago
    It's interesting to watch how different companies that offer both Postgres and warehousing solution under 1 roof approach the same problem:

    - ClickHouse focuses on traditional CDC (ClickPipes) and just make it blazingly fast

    - Databricks leans on their unified storage architecture (LTAP) to avoid copying data (though you can argue there is still a copy in the cache)

    - Snowflake uses a data mirroring CDC as extension so it runs directly on Postgres

    I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine.

    • mslot 4 hours ago
      The challenge is converting primary key updates/deletes to row offsets in a columnar table. That requires maintaining an expensive mapping or doing expensive scans, and is not something you want Postgres itself to do. It's also this bursty, memory-intensive workload that you'd rather not have a lot of dedicated infrastructure for.

      At Snowflake we use Snowflake to do the apply work. Hence end-to-end mirroring has more pieces than just Postgres, but the capturing of changes is cheap enough to do in Postgres directly.

      (Author)

      • hasyimibhar 3 hours ago
        Yes I’m not suggesting to do this inside Postgres. I’m hoping that a Postgres provider can provide this mirroring capability out of the box, similar to how they provide a connection-pooled endpoint out of the box so I don’t have to self-host pgBouncer. I just want to be able to check a box somewhere and have a table in Postgres automatically mirrored to Iceberg, with guarantee that no data is lost. They can charge more for it, I will gladly pay.
    • jbonatakis 5 hours ago
      > I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine.

      The issue that each of those providers above has recently adopted Postgres as a secondary product aimed at supporting their main product, an OLAP database or engine, so they don’t want you plugging in your own query engine.

      I’d bet you’re likely to see this from a Postgres-specific provider first, like Supabase.

      • hasyimibhar 3 hours ago
        I’m waiting for Cloudflare R2 to eventually support mirroring Postgres into R2 catalog. It seems like a nice fit because they already have R2 SQL.
    • pepperoni_pizza 3 hours ago
      > I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine.

      If you count AWS as Postgres provider, DMS into Kinesis into Firehose can do that.

      There was preview of just Firehose doing it directly, but AWS have pulled it because it was too unreliable. Maybe they rebuilt it since?

    • rockostrich 2 hours ago
      Google: Charges out the wazoo for a half baked product built on top of Dataflow
    • brightball 4 hours ago
      Snowflake/Crunchydata comes close to doing that with the pglake extension.

      There’s not a mirror function like what Snowflake offers directly but you can come close with a pgcron to upsert changes to the iceberg tables every so often.

      You can also purge the table put to the iceberg version every so often too depending on your data needs. Then you can create a query unions the results of both.

  • aboardRat4 1 hour ago
    What did Center for Disease Control use before? Mysql?
    • bjt 52 minutes ago
      In this case CDC = Changed Data Capture.
  • whateveracct 13 hours ago
    Snowflake is a really amazing product. It's been a delight using it the last few years.
    • arvyy 9 hours ago
      as much praise as some people give to it, I feel deeply uncomfortable with an idea of SaaS-only DB tech that you don't have an option to self host
      • cheema33 4 hours ago
        Agreed. I personally am very uncomfortable with the idea of SaaS-only DB tech. Databases I deal with are very large. And we are a small company. Self-hosted works for us. We cannot afford hosted database solutions as they charge by storage and some also by data transfer. From my point of view, 3rd party hosting of databases solves problems we don't have. Particularly with AI tools managing our services using ansible/terraform, I think we'd be worse off if we switched to a SaaS product.
    • p_l 7 hours ago
      Seeing Postgres articles from Snowflake surprises me a lot though given how there's zero relation between Snowflake the product and Postgres itself

      EDIT: I now see it's mainly to do with pushing data out of customer's postgres systems into snowflake

      • jbonatakis 4 hours ago
        They acquired Crunchy Data and now have some of the most prominent Postgres developers working there.
  • jauntywundrkind 12 hours ago
    Although pg_lake is open source, worth noting that it heavily refers to but is missing CDC capabilities.

    There's a bunch of comments/links to a closed https://github.com/snowflake-eng/sfpg-extension-pg_lake_repl...

  • datadrivenangel 3 hours ago
    Now fivetran has more competition. Good.
  • wlindley 3 hours ago
    Excellent that Control Data is contributing! Oh wait, I'm a few decades out of sync
    • kps 1 hour ago
      I'm afraid nobody (else) here remembers Control Data Corporation. But I'm happy to see the Centers for Disease Control using Postgres.
  • quotrend 3 hours ago
    [flagged]
  • holydementor 10 hours ago
    [flagged]