The first result was too good, and that certainly meant something was wrong.

I wanted to study a model for Green Credit, and mine had gotten almost everything right, with an area under the ROC curve close to 1, and anyone who works with Machine Learning knows that can only mean one thing: the model is not good, something is wrong.

The context

I am finishing an MBA and decided to tackle a problem that interests me: why does a small company not get credit on better terms when sustainability is part of the picture? The answer is not simply that it is expensive, since this kind of subsidy requires hiring audits, technical reports, consultants, disclosures. What is a single line in a large company's budget is unaffordable for a bakery.

The question I asked myself: can you say anything useful about a company's environmental and social profile using only data that already exists, without asking the company for anything?

I took 2.7 million active companies from public records: tax registration, environmental penalties, integrity sanctions, employment relationship data. All of it published by government agencies, none of it submitted by the company.

The trap

My initial design was the obvious one (and far too simplistic): cluster the companies by environmental and social characteristics, use the clusters as a label, train a model to predict that label.

That is where the perfect result, and the big mistake, came from: I had created the label out of the very same variables I then handed to the model. It was not learning anything about the companies, it was rediscovering my own clustering rule. A mirror, not a model.

This kind of error does not show up in any metric. On the contrary, it usually makes an excellent disguise for a lazy result. In Statistics you learn from your first day of class that 90% of the work is cleaning data, looking at basic tables and simple statistics to decide which results matter. When you study regression models, you learn what spurious relationships and overfitting are. And those are exactly the basic precautions you have to keep when you put all of your data into the machine learning blender.

The fix: separate whoever sets the yardstick from whoever makes the prediction

The first correction was to drop the automatic clustering from the initial test. After all, I wanted to test a model, not whatever the great blender of Claude Code plus Python could hand me. So I built a classification yardstick anchored in official criteria, such as the taxonomy the banking sector uses to say which economic activities contribute to the green economy, combined with descriptive statistics of each company's observable behavior. It stopped being “the algorithm decides” and became “the criterion is replicable and documented, accelerated by the algorithms”.

In the second stage, the model came in to answer a different question: can that classification be anticipated using only what is known about the company at the counter, before any expensive lookup? And this is where I made what I consider the most important decision of the project: I forbade the model from seeing any variable used in the yardstick. It works only with size, share capital, time in operation and line of business. Four pieces of information any analyst has on screen within two minutes.

The metrics dropped and I celebrated

With the design corrected and the label richer, performance fell, and that was exactly what I expected. In the ML world, the near-perfect number meant only one thing: the model was reading the answer. An evaluation metric cannot rest solely on a high number, it has to rest on the overall context. Building a model requires it to be able to infer something while dealing with limited information and real noise, producing a useful approximation of reality.

In the context of my study, for the decision that matters in practice, namely whether the company is eligible for credit or not, the separation was strong enough to serve as a first filter, and that never replaces a full analysis, it only tries to cut the cost of screening by quickly pointing out which cases are actually worth a deeper look.

My advisor asked me to test against history

A credit model can reproduce regional inequality without anyone noticing, and algorithms, and Brazil, have a track record there. So I broke the results down by region and compared two things: the model's hit rate for companies that should be approved, and the error rate in each region.

Two findings.

  • First: the North and Northeast come out with higher approval than the Southeast, a result of the sector composition of those regions, not of any adjustment I made.
  • Second: where the model errs more in those regions, it errs upward, that is, it approves companies the yardstick would not. And starting from a principle of fairness, if there is going to be asymmetry, let it favor granting the subsidy.

This kind of test shows that a score that decides access to credit and is not tested against geographic bias should never leave the drawing board.

What is still unresolved

The model's probabilities rank well, but they still need to go through a more intensive calibration process before real use in credit. Improving the analysis requires greater depth on the data side, since part of the social indicators come from the sector rather than from a specific company, because public microdata is anonymized, and that is only a structural limitation of the available data, not a choice.

Another improvement is that the yardstick is a construction of my own, anchored in official criteria but without an external ground truth to validate it. I had no access to, if it even exists, a “truth” base on which small companies are eligible for green credit.

Why I am sharing this

No idea is born finished and nobody does research alone. Sharing findings is the most direct way to be challenged, to learn and to take the next step. I hope this account helps someone dealing with similar challenges, and that the exchanges here push me toward the next stage of this research, which is only just beginning.