A practical guide to building a multi-layer deduplication system using PostgreSQL's native text-matching capabilities, Elixir for orchestration and scoring, and an LLM for fuzzy name resolution.
Deduplication works best as layered prevention, not a single pass.
| # You will need to install https://github.com/cpursley/html2markdown | |
| defmodule Webpage do | |
| @moduledoc false | |
| defstruct [:url, :title, :description, :summary, :page_age] | |
| end | |
| defmodule WebSearch do | |
| @moduledoc """ | |
| Web search summarization chain |
| See: https://github.com/cpursley/vecto |
See the new site: https://postgresisenough.dev
| { | |
| "status": { | |
| "code": 200, | |
| "text": "OK" | |
| }, | |
| "results": { | |
| "properties": [ | |
| { | |
| "_id": "00c72799d8c05b7158357c3bf4d0906b", | |
| "dateModified": "2023-07-07T03:40:17.731Z", |
| { | |
| "swagger": "2.0", | |
| "info": { | |
| "version": "1.0", | |
| "title": "QuickBooks Online V3API Collections - OAuth2", | |
| "description": "The QuickBooks Online Accounting API is a RESTful API that is used to access QuickBooks companies.\n\nDocs Ref - https://developer.intuit.com/docs/api/accounting" | |
| }, | |
| "host": "DefaultParameterValue", | |
| "basePath": "/v3/company/DefaultParameterValue", | |
| "securityDefinitions": { |
Do you really need Elastic Search?