Mathematical Foundations, Entity Resolution Cascades, and Empirical Survival Models for Gazette-Derived Commercial Intent Intelligence

1.0 Author & Research Standards

This monograph documents the algorithms implemented in the DACH Signals production pipeline. It is maintained alongside that code: every formula below is the one the production pipeline runs, and the text is updated when the implementation changes.

The motivation behind publishing this comprehensive compendium is to establish an open, mathematically verifiable benchmark for enterprise data intelligence. In an industry saturated with opaque, unaudited marketing lists, we enforce a strict Zero-Fabrication Policy: every statistical claim, prior probability, hazard decay curve, and entity resolution threshold must either be computed from real gazette data or declared as a stated modelling assumption, and must be reproducible from our production codebase.

2.0 The Epistemic Crisis in Commercial Data Brokering

Enterprise sales and procurement intelligence systems routinely suffer from catastrophic data decay. Commercial contact data decays continuously as organisations restructure, staff move on, and technographic crawls go stale. We do not publish a decay rate here: we have not measured one, and the figures usually quoted for it are vendor marketing rather than published research. What follows describes how this pipeline avoids depending on such an estimate at all, by anchoring every record to a dated statutory notice rather than to a crawl.

Furthermore, conventional data vendors employ generative LLMs or ungrounded statistical interpolation to impute missing corporate attributes (e.g., fabricating employee headcounts, synthetic tech stacks, or fictitious confidence scores). DACH Signals eliminates this failure mode by constructing a deterministic pipeline rooted exclusively in statutory public records where every data point is anchored in legal gazette notices.

2.1 Directory data and trigger data are different products

A commercial contact database answers “who exists, and how do I reach them”. Its value is depth — headcount, revenue, officers, technology in use — and depth is expensive to acquire and expensive to keep current, because each attribute decays on its own schedule. Vendors compete on breadth of attributes and freshness of crawls.

A statutory signal answers a narrower question: which organisations have just entered a commercial event that a public body was obliged to record. It carries almost none of the directory attributes. What it does carry is a dated, sourced, externally verifiable fact about a moment in that organisation's commercial life — that it won a contract, opened a tender, received a grant, or was incorporated.

These are not competing implementations of one product. Depth answers who to approach; a trigger answers when, and why now. This compendium documents a pipeline built entirely for the second question, which is why it treats the absence of headcount and revenue as a design boundary rather than a gap to be filled by imputation.

2.2 Provenance as the primary attribute

Because the product is a claim about an event, the citation for that event is not metadata; it is the deliverable. Every record carries the URL of the notice it came from, and that URL is included in every export. A buyer can open it, read the original, and confirm the claim without trusting the pipeline at all.

This constrains what may be built. A derived attribute that cannot be traced back to a notice cannot be shipped alongside ones that can, because the reader has no way to tell the two apart in a spreadsheet. It is the reason inference is confined to classification from official codes, and the reason a record with no code is left unsegmented rather than assigned a plausible one.

3.0 Statutory Source Acquisition

Every record originates in a notice that a public body was legally obliged to publish. That constraint is the design, not a convenience: it means each row can be traced to a document a third party can open, and it removes the need to trust any inference we make about a company.

3.1 The threshold problem, and why one source is never enough

European procurement law separates notices by contract value. Above the EU thresholds set in Directive 2014/24/EU, a contracting authority must publish to Tenders Electronic Daily. Below them, publication is governed nationally, and the notice never reaches TED at all. A dataset built on TED alone therefore sees the top of every market and none of its base, and the distortion is worst in countries whose economies are dominated by contracts under the threshold, and total in countries outside the Union. National sources are therefore read alongside it.

3.2 Both sides of a notice

An award notice names two parties, and they are different products. The buyer is the contracting authority that opened the budget; the supplier is the firm that won it. Most procurement services index the buyer, because their customers are bidders looking for work. This dataset records both, and every record states which side it represents.

3.3 Licences and updates

Each source is used under its own licence, checked before the source was added. Where a publisher requires a notice to accompany its data, that notice is delivered with every record from that source. Sources are checked every night, so a new notice normally appears in the dataset within a day of its publication.

4.0 Sector Classification from Official Procurement Codes

Every record is assigned to exactly one sector family. The assignment comes from the Common Procurement Vocabulary code that the contracting authority itself attached to the notice, never from text we interpret. This section explains why that restriction exists, what it costs, and what it buys.

4.1 What a CPV code actually asserts

CPV is a hierarchical vocabulary maintained by the European Commission[1]. Its first two digits are the division, and further digits narrow to group, class and category. Critically, the code describes what was procured, not what either party is. A code on an award notice tells you the winner supplied that category of goods or services; the same code on a call for tender tells you the buyer intends to purchase it. The code alone therefore cannot distinguish a software house from a town council buying software, and any interface that renders it as an industry label without stating the side is asserting something the source does not support.

This pipeline resolves that by pairing the code with the notice type and rendering the two together — supplying, buying, funded, or newly registered — so the reader is never shown a category without the verb that makes it true.

4.2 Families as a partition

Divisions are grouped into families because a division is too fine to shop by and too coarse to trust individually. The grouping obeys one rule without exception: every division belongs to exactly one family, and no division belongs to none. That makes the family set a partition of the corpus, and the partition property is load-bearing rather than aesthetic.

Because families partition, the count for a set of selected families equals the sum of their individual counts. A configurator that prices a segment can therefore be checked arithmetically: selecting two families must return exactly as many records as the two selected separately, and adding a family must never reduce the result. A division mapped into two families would break both properties silently, double-counting a segment a customer is paying for. These invariants are asserted automatically against the live API rather than reasoned about.

4.3 Why keyword classification was removed

An earlier generation of this pipeline inferred a sector from free text when no code was present, using hand-written keyword sets. It was removed, and the reasoning is worth recording because the approach is superficially attractive and its failures are not obvious in aggregate.

Keyword matching fails in a specific and damaging way: it produces confident labels from incidental substrings. A cultural association was routed to an automotive category on the “ev” inside the German legal form e.V.; a university was routed to clean energy because the word “solar” appeared in a photosynthesis abstract. Neither record looks wrong in a list — the company is real, the contact is real, only the segment is invented — which means the error survives casual review and reaches the customer as fact.

The sector is what a buyer filters and pays by, so a wrong one is not a cosmetic defect. The honest alternative is to leave the record unsegmented: it keeps every other field, stays in the dataset, remains searchable, and is simply not sold under a category we cannot evidence. Extending keyword sets to a broader family vocabulary would have meant inventing several hundred new word-to-sector associations with nothing behind any of them, which is why the tables were deleted rather than expanded.

4.4 The one non-code rule

Research grants carry no procurement code, because a grant is not a procurement and is never CPV-classified. Under a code-only rule every grant record falls to the unsegmented bucket and becomes unreachable from a sector catalogue, which is a real loss of a genuinely distinct signal type.

These records are therefore classified from their publishing programme. This is not the text inference described above: a record published by an EU research-funding programme is a research grant by definition of the programme, which is an authority making a statement about its own data in exactly the way a CPV code is. The distinction that matters is whether the classification comes from a party with standing to make it, not whether it happens to be a numeric code.

5.0 Entity Resolution Across Notices

One organisation appears in many notices, spelled differently each time: with and without its legal form, with umlauts transliterated, with a department appended. Resolving those into a single entity is the difference between a list of documents and a list of companies.

5.1 Strongest evidence first

Records are merged on the strongest evidence available: an official register identifier first, then a shared web domain, then names compared after legal forms and spelling variants are normalised away. Uncertain matches are left unmerged. That direction is the safe one: an unmerged duplicate is visible and annoying, while two different companies merged into one is a fabricated record carrying a contact that belongs to somebody else.

5.2 Merging without inventing

When two records merge, the surviving record keeps the notice URL, source and timestamp it came from, and only fills fields it lacks from the other record. One case is deliberately not merged: a consortium bid names several firms in one notice with one contact for the lead partner. Splitting it would require deciding which contact belongs to which firm, which the notice does not say, so those rows are flagged as joint bids and left intact.

6.0 Bayesian Beta Conjugate Intent Estimation

Purchasing intent is inherently uncertain. We formulate baseline commercial intent via a Beta-Binomial conjugate Bayesian framework[2][3]. Each signal category carries an informative prior distribution expressing our stated assumption about how likely the observed gazette event is to correspond to an active procurement budget. These priors are asserted, not estimated from outcome data.

Values are those in CATEGORY_PRIORS in pipeline/intent_scoring.py. A category with no entry there receives the neutral default \(\text{Beta}(5, 5)\). An earlier version of this table listed public_tender_notice at \(\text{Beta}(6, 4)\); the code has never defined that prior, so every tender has been scored from the default. It is corrected here rather than quietly dropped.

6.1 Why a conjugate prior rather than a fitted model

A supervised model would be preferable and is not available. Fitting one requires labelled outcomes — which contacted organisations responded, and which converted — and no such record exists for this pipeline. Training on a proxy would produce a model whose coefficients look empirical and are not.

The Beta–Bernoulli conjugate structure is used instead because it expresses a stated belief and updates it with observed evidence without requiring outcome labels. The prior encodes how likely a category of signal is to precede commercial activity; the evidence is the set of corroborating attributes present on the record. The posterior is a coherent probability rather than a weighted sum of arbitrary points, and the arithmetic is transparent enough to be reproduced by hand.

6.2 Chosen, not fitted — and labelled as such

The prior parameters are chosen. They are a documented judgement about which signals plausibly precede spending, not an estimate derived from data. This document states that explicitly wherever a parameter appears, because the alternative — describing a chosen constant as calibrated, empirical or fitted — is the specific dishonesty this project exists in opposition to. A reader who disagrees with a prior can see exactly which number to argue with.

The practical consequence is that the score is an ordering, not a probability of sale. It is legitimate to use it to decide what to look at first. It is not legitimate to present it as a conversion likelihood, and no surface in this system does.

6.3 Reading the score honestly

A bounded score invites misreading as a grade: a value in the middle of a nominal range looks mediocre even when it sits near the top of the observed distribution, because the observed distribution is nothing like uniform. The interface therefore shows a percentile alongside the raw value, computed over the whole corpus at load time. The raw score remains in exports so that a customer's own analysis is not silently reshaped by our presentation choice.

7.0 Weibull Temporal Survival Dynamics & Hazard Decay

Commercial buying windows are time-bounded. After a contract award or a grant, the organisation's suppliers are usually chosen early in the work, so the relevance of the signal fades with time. We model this temporal survival probability \( S(t) \) and instantaneous hazard rate \( h(t) \) using Weibull distributions[4][5].

7.1 Windows chosen per signal type

Each signal category carries its own \( (\lambda, k) \) pair. These parameters are chosen, not fitted. This project holds no outreach-outcome data — no reply rates, no conversion records — so there is nothing to estimate a survival curve against. They encode a stated prior about how long each kind of buying window plausibly stays open, and they are set out here so a reader can disagree with them.

The values below are the live contents of SURVIVAL_PARAMS in pipeline/intent_scoring.py, the only place they are defined. briefing/svg_charts.py imports them directly and the site’s offline chart receives them through the generator, so neither restates them.

A category with no entry in the table — open and planned tenders are the cases that occur in the data — is scored with the default \( \lambda = 90 \), \( k = 1 \).

Where \( k > 1 \) the instantaneous hazard \( h(t) \) increases with time, which is the intended reading for procurement: competition intensifies as an award moves toward delivery. tech_adoption is the one category set below 1, giving a decreasing hazard for a signal that persists rather than expires. registry_listing sits at \( k = 1 \) — memoryless and deliberately near-neutral, because it is excluded from scoring upstream and a stray record should not score at an extreme.

An earlier version of this section stated that “empirical sales telemetry yields shape parameter \( k = 1.85 \) and scale \( \lambda = 45 \) days” for tender award winners. Neither number appears in SURVIVAL_PARAMS, no such telemetry exists, and the real pair for public_contract_award is \( \lambda = 120 \), \( k = 1.3 \). It is corrected here rather than quietly dropped.

8.0 Multi-Factor Intent Score Synthesis

The final production Intent Score combines the Bayesian posterior expectation, Weibull temporal decay, a disclosed-funding bonus and a published-contact bonus into a bounded scalar \( \mathcal{I} \in [1.0, 10.0] \):

Here \( \mathbb{E}[\theta] \) is the posterior mean of Section 6 on the 0–10 scale, after \( m = 5 \) yes/no observations have updated the prior: a published website, a disclosed grant or funding amount, a detected technology footprint, an official register number, and a signal under 90 days old. \( t \) is the signal's age on the day the score is computed, and \( C \) is the disclosed grant or funding amount in euros (funding_amount_eur). A published contract value does not enter the score. The whole computation is composite_intent_score() in pipeline/intent_scoring.py, which both the pipeline and the briefing's per-account breakdown call.

One of the five observations is currently constant. No active source publishes a technology footprint, so tech_stack is empty on every record and the technology observation is a “no” for all of them. Because every prior here has \( \alpha_0 + \beta_0 = 10 \), it scales every posterior mean by the same factor, \( 14/15 \) of what it would be without the observation. It is stated here so the posterior means can be reproduced exactly.

8.1 Step-by-Step Analytical Example

Consider an organisation that received a €2,500,000 public R&D grant 30 days ago, publishes a website and a phone number, and has no detected technology footprint and no register number on its record. (An illustrative record, not a real one; every figure below is what composite_intent_score() returns for it.)

• Posterior Base (grant_funded prior \( \text{Beta}(7, 3) \); website, disclosed funding and a signal under 90 days are present, technology and register number are not, so \( k = 3 \) of \( m = 5 \)): \( \mathbb{E}[\theta] = \frac{7 + 3}{7 + 3 + 5} \times 10 = 6.67 \)
• Weibull Survival (grant_funded, \( \lambda=90,\ k=1.2 \)): \( S(30) = \exp\left(-\left(30/90\right)^{1.2}\right) = \exp(-0.268) \approx 0.765 \)
• Modulated Decay: \( \text{Decay}(30) = 0.35 + 0.65 \times 0.765 = 0.847 \)
• Time-Decayed Base: \( 6.67 \times 0.847 = 5.65 \)
• Capital Bonus (on the grant amount): \( \Delta_{\text{capital}} = 0.5 \times \log_{10}(1 + 25) = 0.5 \times 1.415 = 0.71 \)
• Contact Bonus: \( \Delta_{\text{contact}} = 0.50 \)
• Composite Intent Score: \( \mathcal{I} = \text{round}(5.65 + 0.71 + 0.50, 1) = \mathbf{6.9 / 10.0} \)

8.2 Why components are additive and bounded

The composite deliberately avoids multiplying independent bonuses. A multiplicative form lets a single strong component dominate, so a record with one large disclosed grant can outrank a recent, contactable, well-evidenced record on that basis alone. Additive components with individual caps keep any one factor from doing that, and keep the contribution of each factor legible when a customer asks why a record ranks where it does.

The funding bonus is logarithmic for the same reason: the difference between a small grant and a mid-sized one is informative, while the difference between two very large ones says more about how the programme is structured than about the recipient. A linear term would make the ranking a proxy for grant size. Contract values do not enter the score at all.

8.3 Signal age as a commercial axis, not only a score input

Age enters the score through the survival term, but it also governs price. The reasoning is that a procurement archive and a current signal are different goods sold from one dataset: an award from several years ago is market history, useful for understanding a sector, while a recent one is an actionable trigger. Charging a single flat rate for a segment prices the scarce, perishable part identically to the abundant, durable part.

Segment price is therefore computed per record from the age band that record falls into, and the resulting figure is returned with its own breakdown — how many records in each band, at what rate — so the price is explainable rather than asserted. The same computation runs in the offline layer, since a quote that changes when the network returns is not a quote.

9.0 Anti-Fabrication Invariants

The commercial risk in this product is not a missing field. It is a plausible one. A buyer who discovers a single invented figure has no way to tell which of the remaining figures are real, and correctly stops trusting all of them. Every mechanism in this section exists because the incentive to fill an empty surface is constant and the resulting errors are invisible in aggregate.

9.1 The failure has one shape

Fabrication in a data product does not arrive as an obvious lie. It arrives as a reasonable default: a fallback score for a record that has none, an illustrative constant that becomes a published parameter, a count computed once and pasted into markup, a persona invented to fill a column no source populates. Each looks like polish at the time of writing. Together they make the surface state things the data does not.

The unifying rule is therefore mechanical rather than a matter of judgement: no displayed quantity may be a literal. A count shown to a customer must be derived from the dataset at render time. A model parameter shown in documentation must be read from the code that executes it, not transcribed. A field with no source is removed from the interface rather than left blank, because a blank cell reads as “missing for this record” when the truth is “never available for any record”.

9.2 Absence is a value

Where a source does not publish a fact, the field is null and renders as an explicit statement that the notice did not disclose it. Deriving a plausible substitute — a domain guessed from a company name, a region inferred from a country's population distribution, a mid-range score for an unscored record — is prohibited even where the guess would usually be right, because a usually-right field cannot be distinguished from a sourced one by anyone downstream.

Sentinels receive the same treatment. Publishers signal “value not disclosed” with placeholder amounts at both extremes of the numeric range, and a sum that fails to exclude them reports a total dominated by records that disclosed nothing. Placeholders are nulled at ingestion so that no downstream surface has to know they exist.

9.3 Generated prose is verified, not trusted

Narrative sections of the customer briefing are written by a language model. The model is given a fact sheet computed in advance and instructed to introduce no figure of its own. Instruction alone is insufficient: a prompt is a request, and the output ships inside a paid document about named organisations.

Generated prose is therefore parsed before it is used. Every number in the text is extracted and checked against the fact sheet, allowing for legitimate re-expression — a value rounded to millions, a ratio stated as a percentage, either decimal convention. Ordinary counting words remain free. Any figure that cannot be traced to a computed value discards that section, and a deterministic template ships instead. The document is complete with no model available at all, which makes the narrative layer an enhancement rather than a dependency, and makes the choice of model irrelevant to correctness.

9.4 Enforcement outside the author's discipline

Rules that depend on the person writing the code are not enforcement. The invariants above are asserted by an automated suite that fails the build: that no customer-facing surface carries a hard-coded quantity, that no offline data layer holds a hand-written count, that every displayed model parameter matches the executing implementation, that the paid export applies exactly the filters the quote was priced with and the same contact restrictions as every other surface, and that arithmetic identities hold against the live API — partitions summing, narrowing never widening, prices equal to their own stated components.

These run against the committed sample as well as the full dataset, because a suite that validates a stale fixture is theatre: the vocabulary can be changed everywhere the developer is looking while the fixture continues to assert the old one.

10.0 Regulatory Framework: GDPR Art. 6(1)(f) Legitimate Interest Balancing Test

The processing of publicly gazetted corporate contract notices and institutional contact points operates strictly under the legal framework of Article 6(1)(f) of the General Data Protection Regulation (EU Regulation 2016/679)[6]:

• Purpose Test (Legitimate Interest): Facilitating transparency in public tender awards, enabling competitive B2B market intelligence, and supporting compliance with EU procurement disclosure rules.
• Necessity Test: Processing is restricted to official organizational data published by government contracting authorities under statutory procurement laws. No intrusive background profiling is conducted.
• Balancing Test: No consumer personal data, private residential information, or special category data (Art. 9) is processed. Data subjects are institutional representatives acting in their official commercial capacities. All subjects retain immediate right-to-object and opt-out privileges under GDPR Art. 21.

10.1 What is and is not personal data here

Most fields in this pipeline concern organisations, and organisational data is outside the scope of the GDPR. The boundary is crossed at exactly one point: a published contact address that identifies a natural person. A role address addressed to a function — procurement, information, a tender desk — is an organisational contact point. An address in the form of a person's given name and surname identifies an individual, whatever inbox it reaches.

The pipeline classifies addresses on that structural distinction rather than on where they were found, since a personal address published in an official notice is no less personal for being public.

10.2 The Article 14 duty and how it is handled

Where personal data is obtained from a source other than the individual, Article 14 places a duty on the controller to inform that person. That duty falls on whoever assembles the dataset, not on a downstream recipient, and it is not discharged by the data having been public.

Until a notification workflow exists, personal-name addresses are withheld from every API response, every export, every count and every paid download. They remain in the stored dataset so the position is reversible once notices are actually sent, behind an explicit configuration switch that defaults to withholding.

The cost of this policy is stated wherever coverage is reported, and stated correctly. It is tempting to quote only the records that lose their last remaining contact, which is a small number, and to omit the much larger number that lose an email while retaining a phone. The second figure is the one that describes what a buyer receives, so it is the one published.

10.3 Downstream obligations we cannot discharge

Lawful processing by this pipeline does not make any particular outreach lawful. German law imposes its own requirements on unsolicited commercial approaches by email and telephone, independent of the data protection basis on which an address was obtained. That obligation sits with the sender, and is disclosed rather than glossed, because a data supplier implying otherwise is selling reassurance it has no standing to give.

11.0 Academic References & Standards

1. European Commission. (2008). Regulation (EC) No 213/2008 on the Common Procurement Vocabulary (CPV). Official Journal of the European Union, L 74/1.
2. Murphy, K. P. (2012). Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge, MA.
3. Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer, New York.
4. Weibull, W. (1951). A Statistical Distribution Function of Wide Applicability. Journal of Applied Mechanics, 18(3), 293–297.
5. Kalbfleisch, J. D., & Prentice, R. L. (2002). The Statistical Analysis of Failure Time Data. John Wiley & Sons, Hoboken, NJ.
6. European Parliament and Council. (2016). Regulation (EU) 2016/679 (General Data Protection Regulation). Official Journal of the European Union, L 119/1.